time to experiment... gráinne conole, martin oliver and jane seale email: g.conole@bristol.ac.uk in the last edition of alt-j, the editorial team decided to try an experiment by creating a more interactive and critical debate on a particular article with the journal. readers were invited to submit comments on an article by davis and denning entitled: 'almost as helpful as good theory: some conceptual possibilities for the online classroom' (davis and denning, 2000). summarized below are some of the key comments raised about the paper. many thanks to everyone who took the time to send in these comments; individual acknowledgements have not been included for reasons of confidentiality. readers' comments comments fell into four main categories. the first addressed the paper's contribution to the learning technology field. some readers felt the paper to be quite difficult to understand, perhaps because it seems preliminary and speculative. one reader commented that in many respects davis and denning were exploring surprisingly uncharted waters. others added the following comments and observations: the paper is interesting in that it provides a framework for thinking about 'dimensions of practice' in virtual learning environments. this is timely and the area certainly lacks robust theoretical frameworks of the sort that this paper is trying to devise. this is an insightful categorization of typologies of 'group function in cyberspace'. the six conceptual areas, and the discussion of group function (and dysfunction) in each area will provide a useful framework for future researchers. some of the requests were simply for additional context: 5 grainne conole, martin oliver and jane seale time t o experiment... mention is made at several points in the paper of 'action science groups' this seems to be an important aspect of their work and would benefit from fuller exposition, as many readers will not be familiar with 'action science groups'. there are several places where claims are made which would benefit from further substantiation either by reference to the literature or their own data. the distinction between different basic assumption groups and work groups was not clear and needs to be made more explicit. others, however, raised questions of interpretation and application: the paper identifies that the theorizing it contains is based upon 'ongoing action research' over a three-year period. however, there is very little explicit connection made between the framework and the theoretical analysis that is provided with this underlying data. this is reflected by the comment in the conclusion that the model/framework that is presented is 'the product of inspiration and intuition based on our iterative analysis of data'. the relationship to the data should really be drawn out more strongly throughout the paper perhaps simply by including some illustrative quotes/examples. the paper describes 'two main theoretical sources' (group dynamics and situated learning). whilst the group dynamics thread does permeate the paper, the situated learning thread was much less clear. for example, there seems to be a leap from situated learning to learning dynamics in table 1. some more analysis of the interaction between 'conceptual areas and their corresponding states' and 'types' (based on the data rather than on supposition) is needed before the model can be thought to be robust. however, the authors seem to be aware of this and the framework does provide a useful starting point for analysis even prior to this work taking place. the authors use scaffolding to mean 'students building upon one another's comments'. this is a rather restricted way of using this term, which is more usually used (within the context of learning) to mean providing support for learners (through the means of channels) stemming from the work of vygotsky. the title of the paper suggests a playful reference to kurt lewin ('nothing so practical as a good theory') but there is little evidence in the paper of ways of evaluating the utility of the 'conceptual possibilities' alluded to in the title. indeed, it was rather hard to say exactly what these might be. some of the language was a little off-putting why use ersatz psychology when there are plenty of good and relevant constructs in scientific psychology? finally, there were some comments that challenged aspects of the paper, such as the following: there is concern about whether the 2x2 matrix is an appropriate extrapolation from the six conceptual areas. it appears that in fact there are already six dimensions to the matrix. from these six dimensions, four typologies are observed which are then reduced (back-mapped) to the two axes. a more common use of the 2x2 matrix is to characterize the points on the two scales and then build the representation of the quadrants from the 6 alt-j volume 10 number i positions of the respondents along the x and y axes respectively. this is not in any way to suggest that the four typologies are not both accurate and useful. it does, however question how the addition of two more dimensions 'depth' and 'time' will be achieved. one alternative to retaining the 2x2-matrix model and to testing and extending it would be to characterize each axis formally as 'learning dynamics' and 'group dynamics' (in cyberspace). alternatively, researchers might wish to work with the underlying 6dimension array. if a visual representation of outputs was required, spider diagrams could be used instead of 2x2 matrices! if researchers used consistent scales along the six dimensions, each of the typologies would have a characteristic 'shape'. other groups could then be analysed and their 'shapes' compared with the four typologies. these last questions and issues are, perhaps, the most interesting to focus upon this experiment was intended to promote debate, rather than consensus! in the following article, davis and denning respond to these comments. we would be very interested to hear readers' views on the value of this form of more interactive, in-depth discussion on one of the papers. please email us with your thoughts. references davis, m. and denning, k. (2000), 'almost as helpful as good theory: some conceptual possibilities for the online classroom', alt-j, 9 (2), 64-75. 7 a response to the critique of gender issues in computer-supported learning cathy gunn,* sheila french,** hamish macleod,*** mae mcsporran**** and grainne conole***** *university of auckland **manchester metropolitan university ***university of edinburgh ****unitec *****university of southampton email: ca.gunn@auckland.ac.nz we find the critique of our paper both interesting and informative. the author raises a number of points as a caution against oversimplification of the issues surrounding gender and computer-supported learning. we fully acknowledge the difficulty of reporting findings from a number of studies and of attempting to define or analyse the complex interaction of already complex concepts such as gender, identity and behaviour within computer-supported learning environments. in response we believe it is important to acknowledge the context in which the article was written: it summarized the research findings of several authors who together ran a discussion panel at the alt conference in 2001. the authors come from england, new zealand and scotland, have different professional backgrounds and work in different academic disciplines. we brought a variety of converging backgrounds with respect to theories of gender to this collaboration, including cognitive, developmental and experiential (siann, durndell, macleod and glissov, 1988; siann, macleod, glissov and durndell, 1990). the common interest we share is in the real business of working with students in technologically mediated learning environments. we have concerns, based on direct observations, which we would like others to engage with us about. gender issues, we would argue, have not been at the fore of either the alt journal or conference. our purpose with both the panel and the paper was to raise awareness and to bring issues that our own and others' research identify as important to the attention of a wider audience. the paper was neither an in-depth study of gender, nor an analysis of the social and cultural practices which constitute science and technology. this paper was a discussion of the panel's findings from their own experiences with reference to published research from other contexts. whilst the authors would have liked to explore the issues 80 alt-j volume 10 number 2 further, the length of the paper as well as the complexity of the issues prevented us from doing so. fuller reading of the case studies cited can be found in richardson and french (2001), mcsporran, dewstow and young (1999) and mcsporran and young (2001). in our paper, we have tended to universalize gender, particularly for the women in the studies. we have not discussed class, ethnicity, educational background or institutional culture, although our collective experience does make us aware of the impact of these factors. whilst we argue we have tried to acknowledge such differences, we may appear to have fallen into the trap of essentialism. the difficulties of exploring difference in students without losing the gendered subject, we agree, is complex and problematic. richardson and french realized in their study that there seemed to be a presence of 'woman', which, as reported, was a minority on the course. the experiences of these women compelled the authors to explore their experiences further. the authors were not blind to these differences but attempted to recognize them, although this was not covered in the short paper. however, the authors also recognize that to explore differences among women perhaps compels us to explore differences among men. this, we suggest, waters down the case of women and challenges the reasons we raised the issues in the first place. the authors acknowledge that this is a complex argument of feminist concern (alcoff, 1997). further difficulties and complexities arise when reporting findings from a group of international authors. we, the authors, are from diverse cultures and different genders. bringing together findings based on race, ethnicity, our own gendered experience and that of our students is almost impossible. we all have our own subjectivities and the cultural contexts we work in are not comparable. in our individual studies we did find differences in ethnicity and class, which again varied across countries and cultures. for example, richardson and french found their female students to be mainly under thirty, and of diverse cultures and class. mcsporran et al. found the females in their study were mainly working class, predominantly white but including some asian and other recent immigrant groups, as well as mature students returning to work. most of these women had seen the usefulness of computers in the workplace and were pragmatic in their choice of courses. however, our discussion revealed that there is no basis for comparison across the uk and new zealand cultures. social class and the perceived role of women within these cultures are different, as is the acceptable role of researchers studying cultural groups that they do not belong to. the ethnic mixes are very different, and so the complexity of definition and analysis increases. comparing or even discussing this in such a short piece of work was difficult, if not impossible. moving then to the educational aspects, the authors are aware that there is a body of work from authors such as clegg, mayfield and trayhurn (1999) who discuss the many discourses around the field of computing. we recognize that these are also complex and need further exploration, which some of us are pursuing. to conclude, the discussion raised by hughes from our original paper has been both stimulating and informative. the suggestions give the authors a further body of work to research should they so wish. we feel these discussions can only help to raise the profile of gender issues in computer-supported learning and hope they will be the start of a new and stimulating dialogue within the alt arena. we propose that bringing the various discourses of gender, identity and computer-supported learning to the attention of academics in all 81 cathy gunn et al a response t o the critique of gender issues in computer-supported learning disciplines will support a move towards the breadth of understanding required to address imbalances wherever these exist. a parallel to this is the evolution of the discipline of human computer interaction (hci) which brought multi-disciplinary perspectives to software design in order for its products to be suitable for, and accessible to non-specialists. references alcoff, l. (1997), 'cultural feminism versus post-structuralism: the identity crisis in feminist theory', in l. nicholson (ed.), the second wave: a reader in feminist theory, london: routledge, 330-55. clegg, s., mayfield, w. and trayhurn, d. (1999), 'disciplinary discourses: a case study of gender in information technology and design courses', gender and education, 11 (1), 43-55. mcsporran, m., dewstow, r. and young, s. (1999), 'who wants to learn on-line? identifying our flexible learners', proceedings of the world conference on educational multimedia, hypermedia and telecommunications conference, washington. mcsporran, m. and young, s. (2001), 'does gender matter in online learning?', alt-j, 9 (2), 3-11. richardson, h. j. and french, s. (2001), 'education on-line: what's in it for women?', in e. balka and r. smith (eds), women, work and computerisation: charting a course to the future, boston: kluwer. siann, g., durndell, a., macleod, h. and glissov, p. (1988), 'stereotyping in relation to the gender gap in participation in computing', educational research, 30, 98-103. siann, g., macleod, h., glissov, p. and durndell, a. (1990), 'the effect of computer use on gender differences in attitudes to computers', computers in education, 14 (2), 183-91. 82 calt12108.fm alt-j, research in learning technology vol. 12, no. 1, march 2004 discussion creating virtual communities of practice for learning technology in higher education: issues and experiences cathy gunn* university of auckland, new zealand taylor & francis ltdcalt1210810.1080/0968776...research in learning technology0968-7769 (print)/1741-1629 (online)research article2004association for learning technology121000000march 2004cathygunnuniversity of aucklandnew zealandca.gunn@auckland.ac.nz dempster et al.’s paper (2003) describes an example of a community of practice that fits well with the definition given in current literature—one of a few that does not stretch the definition beyond the limits. some would doubtless argue that communities evolve and cannot be created. this may point to a key element missing from the scenario described, i.e. a compelling reason for users to access the resources and participate in the communities provided for them by learning technologists. access to technology for educational use is a broader issue, as burbules and callister (2000) explain. in their definition, access involves having physical and operational capability, professional and cultural fluency and an appropriate conceptual model that allows competent interaction, innovation and creativity. this is still a long way off from many of the target users’ current access levels. the separation of the roles of expert, designer and intended user may be the root of the problem of mass participation identified by the authors. the disciplines of hci and computer system design needed the involvement of professionals from other disciplines to overcome the barriers created by expert driven software engineering. so too, the mainstream use of technology in education may be awaiting the arrival of the lay practitioner in system specifications. to label this a problem is perhaps unfair, though. experience tells us that designers of new technology rarely, if ever, foresee what their creation will eventually be used for or how long its infusion into practice will take. if history provides a useful model, then the designer’s role is to develop the opportunity for a critical mass of users to gain access and then devise their own useful * centre for professional development, university of auckland, private bag 92019, auckland, new zealand. email: ca.gunn@auckland.ac.nz issn 0968–7769 (print)/issn 1741–1629 (online)/04/010095–03 © 2004 association for learning technology doi: 10.1080/0968776042000211557 96 c. gunn purposes for the innovation. in this respect the results project team deserve high praise. their approach to needs analysis has been systematic, inclusive and responsive. however, since users never know what they don’t know, asking them what they want invariably yields results with limitations. what this questioning process does— extremely well in this case—is provide a sound basis for the development of a portal that reflects the needs and current capabilities of a range of practitioners in the field of learning technology. presumably not, though, of the entire potential target user group, because only those with some experience of using learning technology may have been included in the research process. these needs analysis limitations relate back to the broader issues of access mentioned in the opening paragraph, and unavoidably exclude those potential users who currently do not have this capacity. the debate about whether the results community was a core of specialists or a wider group is indeed a key question. if a ‘dictionary definition’ of a community of practice is applied, and particularly if the focus is on a learning community, it will surely involve a wider group so the less experienced can interact with and learn from the more experienced. radiation of knowledge outwards from a core group of experts is a key learning opportunity, while interaction with less experienced users can provide understanding of the novice perspective as an input to design specifications. the portal that was developed as a result of this research is one key element in an evolving landscape, where no one person or group can see the entire composition. it is another instance of the complex interaction of individually complex systems that inevitably leads to unexpected outcomes. as edward tenner (1996) reminds us, these outcomes are sometimes good, sometimes not, but they are usually unpredictable. so if the results developers see a long way to go for the portal to enter mainstream use, perhaps they need to accept that they can guide the situation so far but not all the way. they have taken some very important steps in the right direction. the rest is up to the users. they want to support, though any amount of training and development still cannot add the final variable to the equation. time is another essential ingredient in any significant culture shift, and that shift is what now needs to take place. there are many barriers to overcome—not all of them the responsibility of the developers. one aspect (perhaps aspiration is a better term in this context) of a functional community is that the discourse of the discipline uses shared language with common understandings. it may be safe to assume that semantic differences are just one of many cultural variations that a learning technology portal will have to accommodate. a useful alternative to specific targeting of a range of differences is to simply acknowledge and attempt to accommodate them. anything else may be too hard to achieve. in this respect, the paper does identify different categories of user and their preference for resource types. academics prefer to have specific needs addressed while intermediaries want adaptability. this seems a logical reflection of roles and confirmation of the need for a range of different materials to suit a variety of users. another barrier results seems to have managed to overcome is the technical one. most users do not and probably never will want to engage with the concepts of creating virtual communities of practice for learning technology in higher education: 97 metadata and interoperability standards. in this case, those that do can and those that don’t, don’t have to. if the simplicity of the technical description in the paper reflects the visibility of technology to users of the portal then another important hurdle has been crossed. it may even have demystified this aspect for some readers. in conclusion, the key remaining challenge underlies interpretation of the statement that ‘results has been less successful in securing a critical mass of user participation’. reasons are offered to explain the slow speed of uptake and possible remedies given—but here may be an important omission. the fact that ‘it is not yet understood how people use portals’ overlooks the complex nature of the evolving inter-relationship between expert and non-expert users and learning technology. it implies a need to understand existing practice better, although this does not reflect the reality of the innovation. the product of the synergy between current practice and new technology cannot be known at this stage of the integration process and is probably out with the control of the designers. while all the right steps have been taken to facilitate the process, there is a time when designers have to step back, be satisfied with their (excellent) creation and move on to the next task leaving users to make their own way. now may not be the time to remove the training wheels but it will surely come. the required outcome is nothing short of a culture shift, upon which the political dimension (so lightly mentioned but so significant) will doubtless impact. ultimately, it is up to the individuals who comprise the community to determine what the nature of this shift will be. references burbules, n. & callister, t. (2000) watch it: the risks and promises of information technologies for education (boulder, co, westview press). dempster, j., beetham, h., jackson, p. & richardson, s. (2003) creating virtual communities of practice for learning technology in higher education: issues, challenges and experiences, altj, 11(3), 103–117. tenner, e. (1996) why things bite back: technology and the revenge effect (london, fourth estate). discussion creating virtual communities of practice for learning technology in higher education: issues and experiences cathy gunn* the importance of cost-benefit analysis: a response david nicol and michael coen university of strathclyde email: d.j.nicol@strath.ac.uk the critique by draper raises some interesting points that we did not have space to discuss in our published paper. as he points out, taking a purely quantitative approach to the evaluation of ict investments in teaching and learning is wholly inappropriate. however, in this transitional period, where ict applications are new and the effects on operational processes within higher education institutions are unknown, it is not only qualitative issues that need to be investigated but also the potential changes to the scope and nature of the costs incurred by institutions. while the small-scale, and localized, introduction of ict in teaching might only affect the time and effort of a few individual academics, large-scale deployment of the same methodology may require substantial institutional investment (for example, in network infrastructure, hardware, licenses, support staff). the cba model encourages institutions to consider and record all the cost implications of their strategies, not in an attempt to quantify the outputs (benefits) of these new learning processes but to identify and quantify the inputs to these processes. these quantitative inputs can then be evaluated in the context of qualitative outputs. draper argues that the merit of our model lies in 'the value and accuracy of the categories or factors . . . [benefits and costs].. . used'. he then notes areas where the model fails to take into account some important factors (for example, staff stress, student time). while we agree the factors chosen for the cba are critical, we did not intend that users of the model would merely select from our example list of benefits. instead, we envisaged that a decision-making group, with representatives of all those with a stake in the cba, would discuss and formulate their own benefit types in relation to institutional strategies and context. a limited set of benefits, those that matter most to the institution, would then be used to evaluate the investment options. similarly, institutions would determine what costs to include within the costing framework. hence draper's examples of staff stress and student time could be accommodated. if, as a consequence of an investment, academic staff become unduly stressed then this would show up through some indicator of low-staff 122 au-j volume 11 number 3 satisfaction if that were one of the benefit categories. likewise, if student time is an important consideration, this could be incorporated into the model by including student time or student satisfaction as an evaluation criterion on the benefits side. as draper highlights, there are difficulties in the model when it comes to allocating cost information to activities. for example, time spent at a conference might benefit both teaching and research activities. although costing staff time is difficult (see rumble, 1997), we must attempt at least rough estimates of allocations for different activities. otherwise we might as well abandon any attempt to evaluate activities within higher education or to plan future investments. that said, the model advocates a light touch approach to estimating staff time, a much less rigorous methodology than that recommended by the he transparency review (joint costing and pricing steering group (jcpsg), 1999). this type of broad estimating has the inevitable consequence that the 'cost' side of the cba exercise cannot claim great accuracy. however, we believe that the process of investigating and debating the costs is, in itself, useful and that sensitivity analysis can be used to assess the potential impact that variations in the estimates of staff time will have on the final cba evaluation outcomes. just as the model does not limit what benefits can be included, the model does not proscribe what costs should be included in the evaluation. indeed, the model encourages institutions to identify all activities that have a bearing on the investment options under evaluation. draper is concerned about 'learning costs' such as the basic skills training of new students. the model represents this as a 'support activity'. a percentage of the total cost (made up of staff costs, revenue costs, capital costs, etc.) of 'learning' support can be allocated to the activities associated with each ict investment option; that percentage would be determined using a suitable metric (for example, number of students). it is true that the cba model attempts to reduce all factors to quantifiable inputs and measurable, but qualitative, outputs. users end up with a single figure for each implementation option, although choice is based on what trade-off between costs and benefits fits best with institutional priorities. the institution might decide on the lowest cost, the highest benefit or something in between. however, the process of getting to these inputs and outputs is much more important than the figures themselves. while defending the model we acknowledge that many problems remain. in the original paper we implicitly assumed that large-scale ict intervention in learning would be 'topdown', that is, a strategy would be defined and options could be evaluated through cba with reference to an institution's strategic objectives (potential benefits). while an individual's perspective of the benefits of an activity might vary over time (before, during and after), as draper notes, we assumed that the long-term strategic objectives expressed in an institution's learning or e-learning strategy would be stable over time and would provide an appropriate context for the evaluation of large-scale investments. moreover, we assumed that these strategic objectives would have been developed by an institution based on its understanding of educational processes not, as draper seems to suggest, that the cba evaluation would be used to investigate the learning processes themselves. however, further work in studying the risks associated with ict investments in teaching and learning, has led us to review some of these assumptions (nicol and coen, 2003). this type of 'top-down' approach to strategic management and investment is not the only 123 david nicol and michael caen the importance of cost-benefit analysis: a response approach that can be taken. many institutions adopt a 'bottom-up' approach funding a range of ict projects with the intention of gaining practical experience and exploring (rather than planning) benefits. with this 'thousand flowers bloom' approach institutions refine their strategies based on the outcomes of pilot implementations and a complex mix of factors (pedagogical, political, cultural, financial) determine which models are more widely adopted within the institution. as draper points out, we do not yet fully understand all the consequences for institutions of ict investment in teaching and learning. therefore, where 'bottom-up' experimentation is prevalent, cba, with its reliance on a set of planned objectives, may not be the most appropriate tool. other techniques such as the balanced scorecard (kaplan and norton, 1996) may provide a better tool for the management and evaluation of ict investments. in the balanced scorecard methodology a handful of critical perspectives are defined, such as a 'financial' perspective, a student perspective and a staff perspective. the methodology advocates the use of performance measurement to facilitate a process of continual performance review and improvement against each of these perspectives. this type of management tool may have particular applicability in early, small-scale implementations of ict in teaching and learning where costs and benefits are difficult to forecast and resources (staff time and cash expenditure) are not limited to, or defined by, a strict budgetary limit. this type of methodology might also address draper's call for an alternative management method that is less heavily reliant on accountancy and financial considerations. however, at some point each institution, armed with the knowledge gained from its experimentation, is forced to decide whether to invest potentially large sums of money in expanding and/or embedding successful models or, in rare cases, transforming the learning experience and associated institutional processes entirely. in this type of situation much more structured evaluation, of the type proposed by the insight cba model, is required. the model cannot guarantee that the information on costs and benefits that institutions feed into it is appropriate, but it does provide a structured, transparent framework through which institutions can insure that all relevant views have been reflected in the cba and in strategic investment decisions. references joint costing and pricing steering group (jcpsg) (1999), transparency review, manchester, http://www.jcpsg.ac.uk/transpar/index.htm kaplan, r. s. and norton, d. p. (1996), the balanced scorecard: translating strategy into action, cambridge, ma: harvard business school press. nicol, d. j. and coen, m. (2003), 'a framework for identifying the risks associated with elearning investments fe and he', university of strathclyde, unpublished manuscript. rumble, g. (1997), the costs and economics of open and distance learning, london: kogan page. 124 back to the future: team-centred, resourcebased learning as the antecedent of computer-based learning james e. cresswell department of biological sciences, university of exeter in this paper, i argue that gains can be made in both staff efficiency and educational value by replacing the traditional lecture/practical format of higher-education courses with a new format involving teamcentred, resource-based learning (rbl). under the new format, students are deployed in independent teams to tackle a series of tasks supported by various resources, including documentation, email access to a tutor and, of course, the team itself. the new format also fosters personal transferable skills (teamwork, time management), which are not directly addressed by the traditional lecture/practical format. i further argue that whereas computer-based learning applications typically have a minor role as an adjunct to the traditional lecture format, they provide a natural medium for the delivery of resources in the rbl format. i illustrate my contention by reference to bio. 1001, a course in fundamental science skills for first-year undergraduates. the results of student evaluations exemplify the success of the rbl format in bio. 1001. moreover, the new course returned substantial increases in staff efficiency. i describe the potential for computers to play a greater part in the course in future. in conclusion, i speculate that the team-based, rbl format can be usefully generalized to any course that can be converted to continuous assessment by serial, group tasks. introduction with respect to staff efficiency, it is hard to beat the traditional lecture as a format for a university course. with a folder of lecture notes, multiple-choice assessment and an optical mark reader, it is equally easy for the academic to process 10 or 100 or 500 students. indeed, the example of many american universities suggests that class size need be limited only by the capacity of the auditorium. the endurance of the lecture course is no accident. computer-based learning (cbl) technology can claim to introduce 'added value' to the student experience as an adjunct to an existing lecture course, or to be well suited to students with special needs, such as distance learners, but the acceptance of cbl by those at the sharp end of teaching will be slow unless cbl can contribute to increased staff efficiency. the traditional lecture course does, however, have an achilles' heel: it fosters in students few of the personal transferable skills (ptss) that are now widely recommended to 64 alt-j volume 6 number i universities, e.g. teamwork, independent judgement, organizational skills (dearing, 1997). moreover, traditional lecture delivery can sometimes favour certain kinds of learning (comprehension, information recall) over others (synthesis, evaluation). consequently, there is room to make improvements in educational efficiency if the cost in staff efficiency is not prohibitive. below, i will describe my experience in replacing a traditional lecture format with one where independent student teams undertake resource-based learning (rbl), which means that the course content was delivered through media other than the lecturer. the idea behind rbl is not new (noble, 1980; clarke, 1982; exley and gibbs, 1994), but i will argue that it makes a natural antecedent of cbl because there is an obvious role for the computers to play: that of delivering the learning resources. most importantly, the rbl format can actually increase staff efficiency by encouraging students to teach each other. overview bio. 1001, biological investigations, is a mandatory course for first-year undergraduates in the department of biological sciences at the university of exeter. the course caters for around 120 students and is supported by six postgraduate demonstrators (15 hours each) and one lecturer (35 contact hours). the it facility is available in business hours only, and offers 12 pcs with typical core software. in previous incarnations, the course took the traditional lecture/practical format with two lectures per week over 10 weeks and five practical sessions of three hours each. the lectures were generally unsuccessful because of the heterogeneity in student backgrounds. experienced students appeared to be bored by remedial teaching, and inexperienced students baffled by anything too advanced. accommodating all students in a single practical session was also unsatisfactory because of the requirement that students share a computer terminal, which did not encourage everyone to obtain hands-on experience, and some students became marginalized and unmotivated. the revised course retained its 10-week timetable position and credit value, but involved a redirection of the content away from pure statistical analysis, and a radical change in course format. the aim of the revised course was for students to be able to conduct a scientific experiment, to analyse the results, and to write a report on their experiment as a 'publication-quality' scientific paper. in this process, the students should acquire competence in the appropriate computer-based professional tools: spreadsheet, statistical package and word processor. learning environment the key to the new course is in its organization. approximately 120 students are grouped into 18 teams of 6 or 7, which are formed from the pastoral tutorial groups that the students join when they arrive in the department. each team is assigned a postgraduate demonstrator to act as a mentor. the mentor will advise and assess the team. each demonstrator acts as mentor to three teams. the cohesion of the course is maintained by email. students can contact each other, their mentor or the lecturer, although all transactions between teams and the staff are mediated initially by mentors. thus, mentors act as a filter between the students and the lecturer, to whom mentors report any problems either by email, or at a weekly meeting convened by the lecturer. there are several advantages to deploying students in teams: 65 james £ gesswell team-centred, resource-based learning as the antecedent of computer-based learning • resources can be distributed to teams instead of to individuals, which means that the resource budget is divided into 18 rather than 120 units. consequently, a higher quality of documentation can be afforded. • students are encouraged to develop team-related ptss. • the heterogeneity in student backgrounds can be put to work when the experienced students help the inexperienced students in their team. an important ingredient is that the students are assessed as a team for a major portion of the course marks. therefore, individuals should be concerned for their team's performance, and this could motivate them to share their skills. course work and its assessment the teams are set a series of 10 tasks, which involve either practice on software or, eventually, analysis of their own experimental results and the preparation of the scientific paper. the tasks are designed to build skills cumulatively towards the production of the scientific paper. therefore, students learn email first, then progress through exercises that use the spreadsheet, the statistics package and, finally, the word processor. each task is completed by each individual, but students can double their marks if the entire group completes the task by the deadline (typically the end of the week on which the task was issued). in this way, students are encouraged to help other group members. often, the completion of a task will involve sending output from an application (for example, the spreadsheet) to the mentor by email. by email, the student receives prompt acknowledgement and, when necessary, advice on how to repeat the task satisfactorily. some of the tasks involve a group assessment, which proceeds as follows: the entire group convenes at a time agreed with the mentor. only if the entire group is present does the mentor then 'randomly' select two or three students for assessment, in which they are observed in performing some simple computer-based tasks. again, all students must perform satisfactorily for the group to pass the assessment. the observational assessments help to ensure that all students have acquired competence, and reduce the possibility that some students are completing tasks for others. a final component of the assessment is the submission of the scientific paper itself, which is co-authored by each team except for the discussion section. each student writes and submits his or her own discussion, something which allows space for individuality. thus, each student receives a mixture of team marks and individual marks for the scientific paper. resources available to students in addition to providing a brief, the task sheets contain an appendix with hints and references to the course pack. in this instance, the course pack forms the heart of the rbl approach. each learning group receives a course pack, which contains rules and resources. rules specify the protocol for the assessments and the usage of resources (for example, a process for booking computer time, or access to mentors), and the rules for taking a group assessment. resources in the course pack include various instructions for computer applications, exercises in the use of statistical tests, and guidance in conducting experiments and writing reports. course pack documentation included material on 66 alt-j volume 6 number i software usage from it services at exeter and also copies of various student's guides (from software made simple, leicester). students are also encouraged to attend a weekly organizational meeting at the lecture theatre, where there are facilities for displaying computer output to the audience. here, they can watch a demonstration of the steps required to complete each task and they have a chance to ask questions in person. the most important resource is the student's own team, which acts as a learning group. in their team, students can pass on their own previous experience or collaborate in solving course-related problems. students can also question their mentor by email, or book an advisory meeting, although the teams are made aware of time limits (not more than 30 minutes mentor time per team per week). if necessary, students can contact the lecturer by email with the offer of an appointment if necessary. student performance in 1997 overall, the students' performance was very good, the mean mark for the course being 66 per cent, which was much higher than in preceding years. three quarters of students completed all 10 tasks and 96 per cent completed at least seven. the continuous assessment allowed early identification of wayward students, and their tutor was immediately notified. this procedure proved highly effective in re-directing under-performing students. without exception, the few team conflicts were resolved amicably by the rapid intervention of the pastoral tutor or lecturer (students were very quick to alert their mentor to difficulties that lost them points, which usually resulted from an inability to convene the entire group for observational assessments). the students' perception of their own competence in scientific skills (experimental design, statistics, reports) was improved significantly by their experience of the course (table 1), which is an important result because these are areas that, for undergraduates, are often surrounded by anxiety. with respect to it skills, the students' own perception of their abilities rose greatly across the course and, by the end, they showed widespread confidence (table 1). it is reasonable to assert that the course had provided satisfactory it training for the entire cohort (with the exception of www-related activities). skill area windows email word processor computers spreadsheets experimental design statistics reports file transfer statistical package before 1.5 i.i i.i 1.0 0.5 0.5 0.5 0.5 0.2 0.1 after .9 .9 .8 .5 .5 .1 .0 .1 .4 .0 table i: results ofself-assessment questionnaires completed by 62 students before and after participating in bio. 1001. students were asked to grade their competence in each skill area as either 0 (no familiarity), i (fair), or 2 (good).the numbers show the mean of the scores. 67 james e. cressweu team-centred, resource-based learning as the antecedent of computer-based learning staff efficiency and performance as lecturer, it took me about two weeks to design the course by conversion from the previous lecture format and to prepare the course materials. during the course, i met with students (one hour per week) and postgraduate mentors (also one hour per week). in addition, i spent some time in answering email, dealing directly with some groups and in notifying tutors of under-performing students. overall, i expect the new format to reduce my time input for the entire course from 35 contact hours to 25-30 hours. the postgraduate demonstrators were satisfied with the requirements made of them, and mostly preferred to work via email at their own convenience rather than having to attend a practical session for an entire afternoon. from my point of view, the performance of demonstrators was much more equitable, being less dictated by their inclination/ability to deal directly with students in a laboratory setting. possibilities for the development of cbl in bio. 1001 the course is now at a stage where further support could be provided for the teams in tackling the tasks set for them. computers provide a natural medium for delivering the supporting resources. for example, the steps consortium offers a variety of learning materials for the computer that relate to statistics and that are set in a biological context (mogey, 1996, 23). additionally, some form of computer-mediated self-assessment would offer opportunities for students to put their learning into practice. conclusions team-centred rbl appears to be an effective avenue for changing from lecture-centred delivery. in addition to improving staff efficiency, it also fosters a wide range of ptss in the students. moreover, the direct involvement of rbl provides a role for computers that are a natural choice for the delivery of the required resources to independent learners. it is possible to argue that the format of bio. 1001 will not generalize because many of the tasks are, in themselves, computer-based, and because the assessment of the tasks is pass/fail. however, these objections do not relate to the fundamental requirements for the adoption of the format, which is actually contingent on the ability to set teams to work on a series of tasks with adequate supporting resources. therefore, it may be possible to introduce serial, group tasks into any continuously assessed course in which the intended learning outcomes can be accurately and explicitly identified and then logically ordered. i thus contend that the potential advantages of team-centred rbl are sufficient to make its consideration worthwhile. references clarke, j. (1982), resource-based learning for higher and continuing education, london: croom helm. dealing, r. et al (1997), higher education in the learning society, report of the national committee of inquiry into higher education. london: ncihe publications (hmso), july. exley, k. and gibbs, g. (1994), course design for resource based learning oxford: oxford centre for staff development. 68 alt-j volume 6 number i mogey, n. (1996), learning technology dissemination initiative information directory, 9th edition, edinburgh: herriot watt university. noble, p. (1980), resource-based learning in post-compulsory education, london: kogan page. 69 computer-supported experiential learning (phase one staff development) alan staley and diana eastcott learning methods unit university of central england in birmingham, email: alan.staley@uce.ac.uk the computer-supported experiential learning project has been established to promote the use of communication and information technologies for teaching and learning within a vocational university. phase 1 has concentrated upon raising awareness and actively involving academic staff in experiencing these technologies. the project is curriculum-led, and considers how technology can be applied appropriately to an established curriculum model which links theory and practice (kolb, 1984). all academic staff were invited to take part by logging onto the university intranet, accessing information about teaching and learning, trying out ideas and emailing their online mentors with their plans and reflections. in addition, all staff could take part in discussion forums concerning a range of issues. the participation of academic staff is reported; which staff registered as having visited the site, which staff actively used the information to experiment with their teaching, and which staff took part in public online discussions. barriers which limited participation are also reported the outcome of phase 1 has been to encourage over 40 academic staff to embed the use of learning technologies in their own course modules in phase 2 with continued support from the learning methods unit. introduction in recent years the university of central england in birmingham has made considerable investments in developing computer networks. developments have been technology-led, and the major use of the network has been for administration. the computer-supported experiential learning project has been designed to refocus upon the curriculum, and to encourage academic staff to use the network technologies for teaching and learning. the broad aim of the project is to investigate and systematically evaluate the appropriate use of technology to improve the quality of learning. the curriculum model the theoretical framework for the research is a well-established curriculum model developed by kolb (1984) which links theory and practice (see figure 1). this model is relevant in a range of 39 man stale/ and diana eastcott computer-supported experiential learning (phase one staff development) concrete experience active reflective experimentation observation abstract conceptualisation figure l:the kolb learning cycle educational contexts from continuing professional development to vocationally orientated undergraduate courses. it has been used effectively by the authors of this paper on a wide range of staff development programmes. the crucial elements of this model are the links it makes between theory and practice; or, in the words of kolb, between 'concrete experience' and 'abstract conceptualization'. it is common for university courses to be described as either practical or theoretical. similarly, it is often said that theories are taught at university but 'real' learning is what happens out in the workplace. however, it is not sufficient just to have an experience in order to learn. equally, it is not sufficient simply to learn new concepts. this learning must be tested out in new situations. the learner needs to make the link between theory and action by planning for that action, carrying it out, and then reflecting upon it, relating what happens back to the theory. experiential learning in the context of the model designed by kolb involves a systematic, cyclical sequence of learning activities. the cycle may be entered by the learner at any point, but its stages must be followed in sequence. the interesting pedagogical issue is to develop appropriate learning and teaching strategies at each stage of the cycle. in other words, to provide structures which help learners to link theory and practice. the 'reflective observation' and 'active experimentation' stages of the cycle are crucial. reflective observation is the process of reflecting on an experience and making sense of it. based on the work of schon (1987) on the reflective practitioner, this stage of the learning cycle acknowledges the complexity of professional working practices where each decision is made in relation to the individual's own repertoire of examples, values, commitments and knowledge. in reality, reflection overlaps with the 'abstract conceptualization' stage of the cycle where new ideas and knowledge help the learner to have insights and to understand situations in a different 40 alt-} volume 7 number i way. the active experimentation stage of the cycle is the point where the learner plans how to make the link between theory and action. concrete experience is therefore the practice of trying out the ideas in the workplace. the stages of the cycle are by no means clear-cut. considerable overlap can occur, as noted above. work on learning styles (kolb, 1974; honey and mumford, 1986) indicates that just as there is seen to be a division between theory and practice in education, so individuals may have particular preferences in their learning. distinct styles of learning associated with the four stages of the experiential learning cycle have been identified, some learners being stronger at the 'practical' aspects, others at the 'theoretical' part of the cycle. application of technology to the curriculum model the application of kolb's model to phase 1 is described in terms of abstract conceptualization, concrete experience, active experimentation and reflective observation. abstract conceptualization academic staff spend considerable tune in lectures expounding facts, theories, and concepts to students that constitute the knowledge base of a subject. the effectiveness of lectures has often been questioned (bligh, 1972; gibbs, 1982), and it is suggested that precious time with students could be better spent elsewhere in the kolb cycle. a subject knowledge base can be constructed as a web (or intranet) site that students access in their independent learning time. that is not to suggest 'discovery' learning, where students discover things for themselves in a haphazard way (e.g. surfing the net), but rather a focused pursuit of knowledge. key to this pursuit of knowledge are the learning activities within the site that students must do for real in the concrete experience stage of the cycle. in this way, the students' learning is being 'driven' by activities and problems, as described by boud (1991). elton (1996) has suggested that we should not always expect students to be intrinsically motivated by our subject, and that students' intrinsic motivation will improve as they gain a sense of achievement or progress. therefore, it is suggested that this stage of the cycle could include some formative computer-assisted assessment (caa), so that students can check their understanding of key issues before putting theory into practice. concrete experience in the post-dearing age of lifelong learning it seems likely that work-based learning will grow, and links between theory and practice will need to be more explicit. experiencing 'real world' problems is crucial to this model, and it is envisaged that where possible learning activities embedded within the web site can be performed in a working environment. where this is not possible (as in many full-time undergraduate programmes) it is suggested that academic staff create 'substitute experiences' such as role-plays, mock courts, client briefings etc. in this way, class-contact time is shifting from conceptualizing to experiencing. whilst the authors acknowledge that some experiences can be simulated by computer, this is considered to be a last resort, used only wliere real experiences are too dangerous, expensive, or not practical. active experimentation and reflective observation it is at these stages of the cycle that computer-mediated communication (cmc) is of particular importance. in active experimentation students can email their online tutor or mentor concerning their plans for putting theory into practice. the online tutor can individually guide and 41 alan stale/ and diana eastcott computer-supported experiential teaming (phase one staff development) c m o m ibhcd) » die thon imaadowrt.aaniytaied'wiattopi. • t «pplyi«t i i k o i t 10 practice with tvtor by e-asall private 1:1 vhich allows m a y 10 oust? comnaaicatioa. (tabcti»bc«ik,elktnaic new has sad eoaospa andopod m www style pifes. todod«e aawnse tot camma* m d o n h wledse. rna.1velestbi.of i * i • r i figure 2: a model for applying learning technology to the experiential learning cycle support students, offering advice, warning of pitfalls, and generally helping those students that cannot see a way forward. reflective observation is well suited to online conferencing, where groups of students can work co-operatively to reflect upon how the theories worked in practice, and how the different contexts in which students have been working have influenced this. online conferencing is particularly suited to this, due to its asynchronous nature, which gives students time to think and make a considered contribution, unlike many face-to-face communications (mcconnell, 1994). the member of academic staff is now functioning as a group facilitator as opposed to an individual mentor. in summary, several technologies are being suggested, each having a specific purpose and place in the cycle. this is illustrated in figure 2. instigating organizational change rather than simply telling academic staff about this model, it was decided that all academic staff within the university should have opportunity to experience it for themselves. an intranet site was developed using microsoft front page, that contained ideas and theories about teaching, learning and learning technology. this included practical activities and small-scale experiments that staff could try out for real in their classes, i.e. putting theory into practice. the site also contained email links to a team of online mentors who could be contacted for help and support. all staff visiting the site were initially asked to register their name and faculty. staff were also given the opportunity of taking part in several online discussion forums, one designed to reflect upon their experiments, and several others such as 'rumours central', created purely to entice staff into using the technology. this phase of the project lasted fifteen weeks, after 42 au-j volume 7 number i which all staff not taking part were sent a questionnaire to establish the reasons for not participating, and their attitudes to learning technologies. participation in total 59 staff participated in the project to some extent for some, this simply meant browsing the site and registering. others went on to use the online mentoring facility or discussion forums. analysis of the questionnaire revealed that another 30 staff visited the site but did not register. the participants were from all faculties that have access to the network, indicating wideranging interest in using learning technologies. the vice-chancellor and one pro-vicechancellor participated, revealing senior management commitment however, only three managers (deans, associate deans, heads of departments, and heads of schools) participated. all other participants were lecturing staff. anecdotal evidence suggests that it is the younger staff that are attracted to learning technologies, and older staff demonstrate resistance to new technologies and changing work practices. this project did not substantiate this, as there was no significant difference between the ages of those staff that participated, and those that did not. a gender bias did emerge though, as only 18.6 per cent of the participants were female, in a target population that was 39.8 per cent female. very few staff used the online mentoring facility to assist putting theory into practice, although there was considerable dialogue with one member of staff who was redesigning a course module. many staff went straight to the discussion forums, and what follows is an analysis of these. it can be clearly seen from figure 3 that although staff started to participate as soon as the project started, this soon diminished, reaching a low during easter. readvertising the project with a sample of a 'rumours' discussion including a response from the vice-chancellor provoked more response, but without doubt the greatest level of participation occurred after 'learning technology week', a week of awareness-raising events that brought staff together face-to-face. this pattern of participation demonstrates the nature of online conferencing. clearly, interest wanes and, unless there is some personal value to be obtained, participation will drop. staff had no motive to participate other than curiosity; there was no assessment or certification to encourage participation. it is also worth noting how the face-to-face events generated interest and raised motivation. an analysis of discussion entries per member of staff revealed that the discussions were dominated by one of the authors and a small group of staff. this pattern is similar to face-toface discussions, with a small number of students frequently dominating events (gibbs, 1992). while most staff only posted one or two messages (a tentative foray into the online world), some staff really got involved. in several instances, these staff were not academic staff, but technical support staff who had found out about the project and instantly realized the application to their area. hence the 'technical queries' and 'intranet developments' discussion forums developed nicely. the discussion forum concerning teaching and learning issues was less spontaneous and needed considerable prompting from the authors. the rumours discussion forum was also slow to develop, with staff tending to read contributions but being reluctant to add new ones. the greatest discussion in the rumours forum followed an example given by the 43 nan staley and diana eastcott computer-supported experiential learning (phase one staff development) computer supported experiential learning project 50 45 40 35 20 15 10 5 n o . of staff registered for project -no. of discussion entries advertise project 1 lesmiflq technnjwilvwpjikj % n " t y " 1 re-advertise 1 \ ^ v l easter . si ••*. t ^ i---m--^^fz: -ii • " ^ h x-/--4 s, .jj^l \ .̂.̂ ^̂ v i ^ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 weeks figure 3: participation in the computer supported experiential learning project over time. vice-chancellor of what a rumour might look like as a way of introduction 'is it true that staff at perry barr will be charged £100.00 per year to use the car park from september?' clearly, whilst many of these discussions were light-hearted, they did entice staff to use the technology. of the staff that did not participate in the project, 26.7 per cent completed a questionnaire designed to establish their views on learning technology. the major reason for not taking part in the project was time (73.6 per cent of respondents). one respondent in particular expressed the time demands that are placed upon academic staff: too busy with full teaching, research, and admin load + hefc + course revalidation + new course developments. one barrier to the uptake of learning technology is therefore time, especially when one considers the 'up-front' investment in time that learning technology demands. other barriers included problems of access to, and reliability of, the technology. only 9 per cent of respondents indicated that they were not interested in the project, and the attitude to learning technology was very encouraging. few staff considered that their careers are threatened by the technology. most staff were interested in applying the technology to their teaching and learning and believed that learning technology would enhance the quality of student learning. the only other barrier that was identified was that of technical support, which , is aptly summed up by one respondent: . . . this creates a huge barrier to encouraging me to do anything involving technology in my teaching and learning. if the it environment improved, and the support culture existed, i would get more involved. at present, i bang my head against a brick wall. outcomes phase 1 of the project has raised awareness, and actively involved academic staff in using learning technology applied to an established curriculum model. it has encouraged 44 aa-j valume 7 number i approximately 40 staff to embed the use of learning technology in their own course modules in phase 2 of the project. training and support of the learning methods unit will continue during this phase. phase 2 will evaluate fifteen learning technology projects across most faculties in the university. references bligh, d. (1972), what's the use of lectures?, penguin. boud, d. (1991), the challenge of problem based learning, london: kogan page. elton, l. (1996), 'strategies to enhance student learning: a conceptual analysis', studies in higher education, 21, 57-68. gibbs, g. (1982), twenty terrible reasons for lecturing, standing conference on educational development. gibbs, g. (1985), learning by doing a guide to teaching and learning methods, london further education unit. gibbs, g. (1992), discussions with more students, the polytechnics and colleges funding council. honey, p. and mumford, a. (1986), the manual of learning styles, maidenhead: peter honey. kolb, d. a. (1984), experiential learning experience as the source of learning and development, new jersey: prentice hall. kolb, d. a., rubin, i. m and mcintyre, j. m. (1974), organisational psychology, an experiential approach, new jersey: prentice hall. mcconnell, d. (1994), implementing computer supported cooperative learning, london: kogan page. schon, d. a. (1987), educating the reflective practitioner, london: jossey bass. 45 alt-j, research in learning technology vol. 12, no. 3, september 2004 issn 0968–7769 (print)/issn 1741–1629 (online)/04/030301–08 © 2004 association for learning technology doi: 10.1080/0968776042000259609 discussion understanding and using technological affordances: a response to boyle and cook gráinne conole* & martin dyke university of southampton, uk taylor and francis ltdcalt120309.sgm10.1080/0968776042000259609alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2004association for learning technology123000000september 2004gráinneconoleschool of educationuniversity of southamptonso17 1bjg.c.conole@soton.ac.uk introduction our use of affordance draws on salomon (salomon, 1993) who takes the definition back to gibson and norman (gibson, 1977; norman, 1988). perhaps a key difference between the use of the term affordance in design is the emphasis on intended use, whereas our approach reflects salomon’s focus on ‘possible’ use. like gibson the approach taken in our paper is focused on the relationship between the infrastructure of information and communication technologies and people’s use of those technologies. we are interested in asking questions about what uses ict invites and facilitates, what it lends itself to and what it can do well. a potential difficulty with using a term so popular in the field of design is that ‘use’ tends to be focused on how something ‘should’ be used, what it is designed for. discussion about affordance can be limited to the intended, prescribed or designed function of technology. we are also interested in exploring the creative and innovative way people respond to technologies and perhaps adapt them for use in unforeseen circumstances. an affordance of the technology does not simply refer to the intended use but also to the unintended consequences. google’s use of hyper text links to drive the indexing of web searches might be an example of an affordance that is a consequence of creative engagement with technology, the adaptation rather than a feature of the original design related to hypertext. another example of this adaptive use of hypertext might include its use by teachers to provide a digital framework for formative assessment and support for student learning. class room teachers were quick to adapt presentation software and hyper text to present more interactive activity lessons. software affording transmission modes of delivery was thereby adapted to users needs and a more interactive affordance created. a tangible example might include the way teachers have used ict to engage students by re-creating quiz show themes in their * corresponding author: school of education, university of southampton, southampton so17 1bj, uk. email: g.c.conole@soton.ac.uk 302 g. conole & m. dyke classrooms and lecture theatres. these adaptations relate to the affordances of ict but not necessarily in a top down, structured designed ‘for’ way associated with narrow definitions of physical affordance. our use is therefore also wider than that suggested by norman’s term perceived affordance, which again could be construed as to represent a passive response to objects rather than a creative engagement with them. affordances are thereby focused on the relationship between people and object, their creative and adaptive interaction with the environment rather than any compliant response to any designed features of that environment. this is why we adopted salomon’s definition with emphasis on the possibilities of for use. is this an alternative articulation of affordance perhaps reflecting a more social and constructivist genealogy? we would like to begin by thanking boyle and cook on their detailed and insightful comments on our paper. in this article we attempt to address the issues they have raised. we would agree that the concept of affordances is ‘potentially rich and provocative’ which is why we wanted to explore its application in terms of providing a richer understanding of the nature of information and communication technologies (ict) and the ways in which they might be harnessed to support the development of pedagogically effective design of learning activities. since we wrote the paper sharpe has developed a useful typology of ‘effective interventions that support e-learning. her typology resents a number of characteristics of effective interventions and illustrates how these characteristics might operate in the context of working with resources, individuals and/or wider change (sharpe, 2004). it consists of five dimensions: ● accessibility—resources and tools need to be accessible to users. ● contextualisation—resources and tools need to be contextualized to the users needs and circumstances. ● professional learning—changing practice requires practitioners to learn through doing. ● communities—there are advantages to working with others and sharing experiences. ● learning design—practitioners need to be supported in engaging with a process that starts with the educational approach. ● adaptability—effective use is influenced by a practitioner’s ability to adapt and reuse the resource and these need to be easily repurposed. response to boyle and cook’s questions question 1: how valuable is the concept of affordances and does its application provide any really new insight into the inherent properties of technologies? boyle and cook provide a valuable review of the origin of the concept of affordances from the original definition development by gibson (gibson, 1977, 1979) and the subsequent critiques by norman (norman, 1998) and mcgrenere and ho (mcgrenere & ho, 2000) which we did not have time to discuss in the substance of our paper. we agree that mcgrenere and ho provide a useful clarification of the understanding and using technological affordances: a response 303 mis-use of the term and in particular think their idea of affordances as a framework for design, moving beyond gibson’s notion of affordances as binary (either there or no) to a concept of degrees of an affordance. they consider affordances in terms of two dimensions where one dimension describes the ease with which an affordance can be undertaken and the second dimension describes the clarity of the information that describes the existing affordance; each being a continuum. they state that the goal of design is to first maximize the necessary affordances and then maximum each of these dimensions (figure 1). figure 1. mcgrenere and ho’s diagram representing the affordance and the information that specifies the affordance as a continuum this follows our own thinking but was perhaps not clearly articulated in our paper. a more detailed analysis of specific instances of affordances against the taxonomy we describe is described in a recent review of the use and impact of tools which are changing practice (conole, 2004). the review categories tools by function, and considers tools for: ● manipulating text and data—word, excel and access; ● presentation and dissemination—the web, powerpoint and adobe; ● analysing data—spss, nvivo and stella; ● information seeking and handling—search engines, portals and gateways; ● storing and managing information—databases and journals; ● personal management—diary and to do lists; ● project management—microsoft project; ● communication—email, chat and bulletin boards; ● visualisation and brainstorming—mindmaps and visual tools; ● providing guidance and support—wizards, help systems and toolkits; ● evaluation and assessment—caa systems and web tracking. figure 1. mcgrenere and ho’s diagram representing the affordance and the information that specifies the affordance as a continuum 304 g. conole & m. dyke the critique in this review of the impact of word resonates well with the discussion by mcgrenere and ho: but word has also changed the nature of practice. the ability to type thoughts straight into a document, to cut and paste and move sections of text around a document, have changed the ways in which we create knowledge. writing with pen and paper required the user to think linearly, writing only when the text was near completion, in contrast the use of a word processor allows you to think non-linearly and to adapt and develop ideas as they emerge. … therefore word … forms part of our distributed cognition (salomon, 1993) or what perkins refers to as the concept of person-plus—i.e. that our ‘intelligence’ is distributed between our minds and past experience and a range of mediated tools and resources (perkins, 1993). … practitioners have become accustomed to building on and adapting previous material. this has enabled them to become more expert at reuse and repurposing and also offers the opportunities for them to improve materials based on evaluation of and reflection on the use of the materials in practice. (conole, 2004) question 2: how valid is the methodological approach suggested? our use of the term accessibility is deliberately broader than ‘access by everyone regardless of disability’ and is in line with sharpe’s definition (sharpe, 2004a). in terms of their comments on the ‘speed of change’ affordance we would argue that this is an inherent and important characteristic of technologies, understanding this and taking account of it can enable practitioners and students to development appropriate skills and strategies to cope with and address the inherent every changing nature of technologies. we agree with boyle and cook’s comment that we do not explore in enough detail on how these affordances can be used by students or practitioners, however this was not the central purpose of the paper which was to provide an initial taxonomy of affordances and associated characteristics. we are using this as a basis to develop a learning activity design toolkit which aims to provide a better linkage between good pedagogy and the design of learning activities and effective choice of appropriate tools and resources (conole & fill, in preparation). we have defined a specification for a learning activity which consists of the context (and rationale) within which the activity occurs, the learning and teaching approaches, the assessment and the series of tasks undertaken by the learner (figure 2). the toolkit will provide practitioners with layered guidance through the process of devising a learning activity and will link different pedagogical approaches to appropriate learning tasks and associated use of tools and resources. figure 2. specification of a learning activity question 3: what other approaches might be taken? we contend that adopting a social constructivist approach is valid and useful in particular because use of tools is critically dependent on context. again this aligns well with sharpe (2004b) and beetham (2004). boyle and cook contend that the authors articulate certain opportunities that the technology makes available to practitioners. they then balance these positive features with difficulties or challenge that are understanding and using technological affordances: a response 305 raised for users. if these barriers are not dealt with effectively then the opportunities may not be exploited. (boyle & cook, 2004) and argue ‘how does the concept of affordance enrich this approach?’ we believe that making the affordances (and in particular their benefits and limitations) more explicit can help practitioners make informed choice and as discussed above we are exploring this through the development of a learning activity design toolkit which aims to guide practitioners through the process of developing learning activities. question 4: the approach suggested reflects a particular interpretation of social theory. is this a limitation and might a broader analysis of other social theories yield new insights? kreijns et al.’s discussion of the sociability of cscl is a good example of using understanding of a particular affordance (in this case the affordance of communication and collaboration in our taxonomy) to tailor how a particular technological environment is used to maximize a particular type of interaction. they propose a framework which suggests incorporating properties into the cscl environment that facilitate the triggering of a new phase in which social interaction may take place. these properties—social affordances—ultimately create a social space amongst the members… (kreijns et al., 2002) this approach aligns completely with our own thinking and provides a good instance of the value of understanding ict affordances and then using this as a basis for mapping to the design and use of a particular tool or environment. as boyle and cook note they go on to state that: figure 2. specification of a learning activity 306 g. conole & m. dyke social affordances are properties of cscl environment that act as social-contextual facilitators relevant for the learner’s social interactions. when they are perceptible, they invite the learner to act in accordance with the perceived affordances, i.e. start a task or non-task related interaction or communication. (kreijns et al., 2002) they suggest that social affordances encompass two relationships: 1. the reciprocal relationship between the members and the cscl environment. 2. the perception-action coupling where once a member becomes salient (perception), the social affordances not only invite but also guide another member to initiate a task or non-task related interaction or communication with the salient member (action). they devise a taxonomy of elements affecting social interaction and group learning. the eight factors are: 1. appropriate teacher behaviour; 2. appropriate member behaviour; 3. nature of the learning tasks; 4. member roles; 5. task resources: knowledge or physical resources that enable execution of the task; 6. goal definition: describing the purpose of the collaboration; 7. formative evaluation with feedback from peers or from educators; 8. summative evaluation and reward structure. elements 3–8 map well to the elements defined for the learning activity specification described above. question 5: practitioners are still exploring the potential of new technologies, and the current uses of technologies often do not take full advantage of the medium. therefore, how can practice take full advantage of the affordances of ict? and question 6: does understanding of the affordances actually get us closer to improvement in practice and is this a useful framework? we agree that it is important to move beyond the development of a taxonomy towards developing new tools for learning but argue that the development of a taxonomy is a valuable first step in the process. their suggestion is timely and maps well with the current jisc e-pedagogy programme (jisc, 2004) which is attempting to address the following questions: ● how can we enhance current knowledge about what constitutes effective practice in e-learning? ● how can we support practitioners with their use and understanding of e-learning? ● how can we promote the development of terminology and frameworks that will understanding and using technological affordances: a response 307 improve understanding and sharing of practice in e-learning? ● what are the current approaches to the design of e-learning activities and how can these be developed in the future to ensure that we are using sound pedagogical models. an initial aspect of this programme has been a series of commissioned reports and reviews, including a review of e-learning models (beetham, 2004), a review of e-learning models and frameworks (de freitas & mayes, 2004), and a research study on the effectiveness of resources, tools and support services used by practitioners in designing and delivering e-learning activities (conole, 2004; franklin, 2004; littlejohn & mcgill, 2004; oliver, 2004; sharpe, 2004). we agree that: what is required in all approaches is serious attempts to include new empirical observations of learning in the evolution of new frameworks, tools and systems to support (i) tutors as they make decisions about the use of new technology, (ii) learners as they learn, and (i) systems designers and developers as they envisage new innovative tools to support learning. (boyle & cook, 2004) the second part of the research study on the effectiveness of resources, tools and support services used by practitioners in designing and delivering e-learning activities referenced above is taking the knowledge gained by the initial reviews and the typology developed by sharpe (2004b) and undertaking an empirical study to ascertain what actually happens in practice. the outcomes of this study will then help to inform and shape the future directions for the e-learning and pedagogy programme and in particular the development of innovative approaches and tools to transform practice and ultimately improve the student learning experience. references beetham (2004) review: developing e-learning models for the jisc practitioner communities. available at: http:///www.jisc.ac.uk/uploaded_documents/reviews2.1.doc (accessed 18 june 2004). conole, g. (2004) report on the effectiveness of tools for e-learning, commissioned review report as part of the jisc-funded e-pedagogy desk study reviewing resources, tools and services for e-learning. conole, g. & fill, k. (2004) specification for a learning activities design toolkit, paper in preparation. franklin, t. (2004) draft research study on national services used by practitioners in designing and delivering e-learning activities, commissioned review report as part of the jisc-funded e-pedagogy desk study reviewing resources, tools and services for e-learning. gibbson, j. j. (1977) the theory of affordances, in: r. shaw & j. bransford (eds) perceiving, acting, and knowing (hillsdale, nj, erlbaum). jisc (2004) e-learning and pedagogy programme. details associated paper available online at: http://www.jisc.ac.uk/index.cfm?name=elearning_pedagogy (accessed 18 june 2004). littlejohn, a. & mcgill, l. (2004) detailed report for e-learning and pedagogy research study: effective resources for e-learning, commissioned review report as part of the jisc-funded e-pedagogy desk study reviewing resources, tools and services for e-learning. norman, d. (1988) the psychology of everyday things (london, basic books). 308 g. conole & m. dyke oliver, m. (2004) effective support for e-learning within institutions, commissioned review report as part of the jisc-funded e-pedagogy desk study reviewing resources, tools and services for e-learning. salomon, g. (ed.) (1993) distributed cognitions—psychological and educational considerations (cambridge, cambridge university press). sharpe, r. (2004a) research study on the effectiveness of resources, tools and support services used by practitioners in designing and delivering e-learning activities—initial positioning report: effective resources and interventions for e-learning, commissioned review report as part of the jisc-funded e-pedagogy desk study reviewing resources, tools and services for e-learning. sharpe, r. (2004b) a typology of effective interventions that support e-learning practice, commissioned review report as part of the jisc-funded e-pedagogy desk study reviewing resources, tools and services for e-learning. learning technologies and the lifelong learner: armament or disarmament? jane seale school of occupational therapy and physiotherapy, university of southampton, email: jks i @soton.ac.uk educators at all levels are underpressure to produce 'lifelong learners'. their task is to 'arm' the student with knowledge and skills that will enable them to be creative and enterprising scholars. one possible way of arming the lifelong learner is through the use of learning technologies. learning technologies can offer armament by widening access and participation and offering flexible delivery. this paper will use the results of two evaluation studies to explore the argument that learning technologies have the capacity to both arm and disarm students. results from an evaluation of an email discussion list will be presented to highlight how the way a learning technology is used may arm a learner by giving them information but disarm them by promoting a lack of confidence and a low valuation of discussion. results from an evaluation of a microcosm application will be presented to highlight how the way a learning technology is used may arm a learner by helping them to apply knowledge but disarm them by placing restrictions on their selfdirected learning. these results will be discussed in order to argue that the 'disarmament' of students through the use of learning technologies may place obstacles in the way of lifelong learning. introduction in a general context, lifelong learning appears to be about increasing access to education and supporting individual development. in the specific context of the united kingdom, lifelong learning is about converting people to a culture of learning in order that the nation can produce creative, enterprising scholars (blunkett, 1998). in both contexts, it is considered that learning needs to be a lifelong commitment, in order that individuals can fulfil their potential and improve themselves. it is generally accepted that learning technologies can increase access to education through flexible delivery and innovative uses of open and distance learning (porritt, 1997; lovie, mclean and newlands, 1998). an examination of learning technology literature reveals strong consensus that the kind of learning to which people should be converted is more than just 61 jane seale learning technologies and the lifelong learner: armament or disarmament? knowledge acquisition. huge emphasis is placed on critical thinking and reflection. for example, hammond and trip (1998) argue: the computer is starting to be used not just as a delivery mechanism or surrogate calculator, but to encourage the learner to ask questions, explore and form active hypotheses, in short as a cognitive tool for learning, an augmenter of intelligent study, (p. 88) while mcloughlin and oliver (1998) see learning as a social, collaborative and conversational experience and therefore argue that: the conversation requires reciprocity and mutual understanding and this is achieved through talk, discussion, and negotiation. computer assisted learning can support interaction, dialogue, reflection and conversation if learning tasks are structured appropriately. (p-13) in considering the kind of learning that learning technologies should promote, the language used in the literature conjures up images of a 'battle' to 'equip' learners. for example steeples, unsworth, bryson, goodyear, riding, fowell, levey and duffy (1996) talk of how discursive technologies can be used to 'challenge conventional, didactic modes of education' (p. 78). hammond and tripp (1992) use the metaphor of a 'trojan horse' to describe how the computer is starting to be used in psychology education, while grabinger, dunlap and duffield (1997) talk of how learners need to be involved in a 'continual process of retooling their knowledge and skill base' (p. 5). the use of learning technologies may equip learners in a positive way, such as that described by grabinger et al. alternatively, the use of learning technologies may equip learners in a negative way. for example, carswell (1998) talks of how students who are ill-prepared to use the world wide web may be burdened with the 'extra cognitive load of developing new strategies for accessing internet-based material' (p. 77). in other words, instead of being armed and well-prepared for learning, students who use learning technologies may be disarmed and denied a rich learning experience. in this paper, two case examples of the use of learning technologies with university students will be described. the battle imagery highlighted in the literature will be utilized in order to develop the argument that the way learning technologies are used may serve to either arm or disarm learners. case example i: email discussion list an email discussion list was set up by the author to support a first-year module of behavioural sciences being taught in an occupational therapy and physiotherapy degree course at southampton university. in order to promote and instigate discussion, case studies, topical news items, questions and summaries of arguments were posted. in order to inform discussion and enable students to place the issues in a context, lecture notes {powerpoint slides) and reference lists were also posted. there was an implicit assumption that the students would reflect on the information contained in the lecture notes and use these reflections to contribute to a group discussion. this assumption reflects a common portrayal of email as an ideal means of providing opportunities for group discussion, tutor-student interaction and collaborative working (mcateer, tolmie, duffy, and corbett, 1997). the email list was evaluated through the use of both a paper-based and email-based questionnaire. the paper-based questionnaire was distributed amongst all 130 first-year 62 au-j vb/ume 7 number i students in december 1997 and invited comments on teaching and learning support, including the email list there was a 36 per cent response rate for this questionnaire. the email-based questionnaire included open questions about students' reasons for using the list, and more closed questions asking them to rate the learning experiences offered by the list from 1 (most valuable) to 5 (least valuable). there was a 15 per cent response rate for this questionnaire. finally, usage statistics were collated from majordomo, in order to assess numbers of subscriptions and postings to the list. case example 2: the cat microcosm application computer applications in therapy (cat) is a microcosm application, designed by the author to support a third-year option offered within the school of occupational therapy and physiotherapy at southampton university (seale, 1997). the basic design of the package is that students first choose one patient/client history and then browse through the resource information in order to find out as much as they can about whether or not the client in the case history would benefit from using a computer. students can enter and browse through the resource information at any level and in any order they wish. they can follow pre-determined links or create their own search strategies. the final outcome is that students can use the material in the resource information to write a report on whether and how they would use a computer with their chosen clients. the application uses case studies to encourage students to apply the knowledge they gained from the package and to think carefully about when and how they would use computers with clients. case studies are a popular method for facilitating self-directed learning, particularly when using cal. for example kahn and yip (1996), in considering the pedagogic principles of case-based cal, argue for case-based reasoning (cbr) whereby a set of examples or cases is employed in a structured network. a cbr network therefore offers students opportunities to use previous examples or cases to solve fresh problems. the student is helped to acquire an expert model through free or structured browsing. in order to evaluate the usefulness of the cat microcosm application a small evaluation study was conducted in february 1997. seven lecturers and eight third-year students were offered a 90-minute workshop in which the package was explained and demonstrated with opportunity for 'unstructured hands-on play'. at the end of the workshop they were asked to complete an evaluation questionnaire that contained a number of statements regarding the presentation, content and educational use of cat. respondents were asked to state whether they strongly agreed (score of 5) or strongly disagreed (score of 1) with each statement. in addition to answering the opinion questions, the students were asked to list the advantages and disadvantages of using the application. evidence for armament and disarmament the results from the evaluations of the email list and the cat microcosm application provide some evidence that students could potentially be both armed and disarmed by their use of these learning technologies. this evidence will now be presented in more detail. armed by information the data from the evaluation of the email discussion list revealed strong evidence that students highly valued the opportunity to obtain information in the form of lecture notes. for example, _ jane seole learning technologies and the lifelong learner: armament or disarmament? ranking positions summary of student rankings i st theory lecture notes 9 out of 11 of respondents gave a rank of i ( mean= 154) 2nd: applied lecture notes 8 out of 11 of respondents gave a rank of 2 (mean=z09) 3rd: reference lists 6 out of 11 of respondents gave a rank of 3 (mean=3.18) 4th: tutor-posted questions 3 out of 11 of respondents gave a rank of 4 (mean =4.09) 5th: student responses t o questions 6 out of 11 of respondents gave a rank of 5 (mean=5.45) 6th: student-posted questions 6 out of 11 of respondents gave a rank of 6 (mean=6.09) table i: student rankings of the facilities offered by the email discussion list in terms of being useful to learning the students ranked the learning experience offered by the lecture notes and reference lists higher than they ranked the posted discussion questions and their responses (see table 1). interestingly, the theory notes were ranked above the applied notes, and tutor-posted questions were ranked higher than student-posted questions. some of the comments made by the students may help to further our understanding of why the information obtained from the lecture notes was valued so highly (see table 2). information reinforced personal notes from lecture. allows you to pay more attention in class. you know there is always a back up if you dont understand things. enabled me not to worry about writing every single word down. i looked up the lecture notes to compensate for my notes, which were never comprehensive. confidence i sometimes feel i may lack confidence in answering some topics. i don't feel i have enough knowledge base yet to answer the questions. being unsure of some of the subject areas and not having experience of relevant areas inhibited me. i didn't understand some of the questions. table 2: example evaluation comments for the email discussion list disarmed by a low valuation of discussion the information in table 1 indicates that students using the email discussion list valued obtaining information above discussing the information. usage statistics from majordomo support this. so for example, statistics concerning the response to discussion questions revealed very poor response and contribution levels. of the eight debate issues posted to the list by the tutor, five failed to gain a single response by the students. of the three issues that elicited a response, only two or three students responded. armed with an opportunity t o apply knowledge the data from the evaluation of the cat microcosm application revealed evidence that a major advantage of using the application was the ability to obtain information that can be applied to other learning situations such as modules and placements (see table 3). interestingly, the students have listed more opportunities for applying knowledge than the lecturers. disarmed by a lack of confidence in trying to understand why the discussion element of the email list was not used or valued, the evidence suggests that some students appear to have been disarmed by a lack of confidence in 64 alt-j volume 7 number i third-year students lecturers applied knowledge useful for other modules useful for placements can be used in assignments useful for multi-disciplinary teaching useful for placement preparation self-directed learning learn in own time and learn students still need t o do their own research at own pace would be disappointed if students did not encourage self-directed study look further makes learning interesting weaker students may not look further saves you wading through books some students may need t o discuss their findings with others students'thinking may be limited by the linking table 3: example comments regarding the perceived advantages and disadvantages of using the cat microcosm application their individual knowledge and experience. this perhaps suggests that the discussion excluded those who for whatever reason did not feel 'qualified' to join in the discussion (see table 3). disarmed by restrictions on self-directed learning for the cat microcosm application, analysis of responses to open questions revealed disagreements between students and lecturers regarding self-directed learning. students felt the package would encourage self-directed learning because users can learn in their own time and pace. while lecturers expressed concern that students' thinking may be limited by the linking within the package and some students might not look further than the cat resource base and thus may not be as self-directed as they perhaps could be (see table 3). discussion for the email discussion list the results reveal that a lack of confidence and a low valuation of the discussion could disarm users. possible reasons why students felt unconfident may relate to the way the email list was set up. for example, the membership of the group was quite large (over 70) and there was no anonymity when posting messages to the list. these two things combined may have served to exclude those students who were afraid of exhibiting their 'ignorance' in such a public arena. these problems could be solved with a bit more creative thinking on behalf of the list owner. the apparent low valuation of the email discussion list may be explained by examining the role of the list owner. both laurillard (1993) and mcloughlin and oliver (1998) consider that the tutor plays a pivotal role in encouraging conversation and reflection. they suggest strategies for achieving such a role that include clarifying expectations, promoting verbal expressions of different perspectives and encouraging alternative solutions or approaches. perhaps the owner of the email discussion list failed to engage her students in an open dialogue about the objectives for using the email list and therefore there was no negotiation about the process or goals of learning. for the cat microcosm application the results reveal that users may be disarmed by a perceived restriction that the application places on self-directed learning. a possible reason why 65 jane seo/e learning technologies and the lifelong learner: armament or disarmament? the application may be seen to place restrictions on self-directed learning is the fact that it is a self-contained resource base. given that students are expected to search the resource base to find out more information about their chosen case study they may be given the impression that all the answers are held within the resource base. if all the answers were held within such an expert system there would seem to be no reason to direct learning to other sources of information. solutions to this disarmament problem may involve revisiting the design of the cat application and perhaps changing the links within the application so that students are encouraged to take paths out of the self-contained resource base. alternatively, the author of the package may need to make explicit to the students her learning goals and so for example 'sell' the package as the springboard to other sources of knowledge. implications for lifelong learning in the introduction an argument was made that focused on the idea that learning technologies can play a role in converting learners to a culture of critical thinking and reflection as opposed to a culture of knowledge acquisition. the two case examples described in this paper have served to illustrate how learning technologies can help students to acquire information and knowledge. however, it is debatable whether the use of these learning technologies has helped to produce lifelong learners that are capable of critical thinking and reflection. the way these technologies were used means that neither did particularly well in promoting such skills as discussion or self-directed learning. these skills would seem to be essential if students are to fulfil their potential and improve themselves. the two case studies described in this paper are preliminary investigations, which gathered information from only a small number of students. whilst the results from the two case studies suggest that the use of learning technologies may place obstacles in the way of lifelong learning, more detailed work needs to be undertaken in order to test the conclusion drawn here that learners will be disarmed unless educators are creative in their design and use of learning technologies and enterprising in how they negotiate and promote the objectives for using learning technologies with students. references blunkett, d. (1998), foreword to the learning age: a renaissance for a new britain, government green paper. carswell, l. (1998), 'possible versus desirable in instructional systems: who's driving?' alt-j, 6, 1, 70-80. grabinger, s., dunlap, j. c. and duffield, j. a. (1997), 'rich environments for active learning in action: problem-based learning', alt-j, 5 (2), 5-17. hammond, n. and tripp, a. (1992), 'cal as a trojan horse for educational change: the case of psychology', computers and education, 19 (1/2), 87-95. khan, t. and yip, y. j. (1996), 'pedagogic principles of case-based cal', journal of computer assisted learning, 12, 172-92. laurillard, d. (1993), rethinking university teaching: a framework for the effective use of educational technology, london: routledge. 66 alt-] volume 7 number i lovie, f., mclean, a. and newlands, d. (1996), 'access through technology: a comparison between conventional and distance learning students', active learning, 5, 30-5. mcateer, e., tolmie, a., duffy, c. and corbett, j. (1997), 'computer-mediated communication as a learning resource', journal of computer assisted learning, 13, 219-27. mcloughlin, c. and oliver, r. (1998), 'maximising the language and learning link in computer learning environments', british journal of educational technology, 29 (2), 125-36. porritt, n. (1997), 'managing to learn with technology', active learning, 7, 17-28. seale, j. (1997), 'computer applications in therapy: evaluation of a multimedia teaching package', in sinclair, k., wing fu, c. and king, b. (eds.), proceedings of the hong kong international occupational therapy congress, hong kong polytechnic university. steeples, c., unsworth, c., bryson, m., goodyear, p., riding, p., fowell, s., levey, p. and duffy, c. (1996), 'technological support for teaching and learning: computer-mediated communication in higher education (cmc in he)', computers and education, 26 (1-3), 71-80. 67 metadata in the changing learning environment: developing skills to achieve the blue skies juanita foster-jones and hazel beazleigh the open university email: j.foster-jones@open.ac.uk this short paper will examine the importance of metadata and its role in the changing learning environment, beginning with an introduction about what metadata is, and the benefits to be gained from applying it to all academic resources. two open university projects, portfolio and the reusable educational software library, will be described and used to illustrate how the ims learning resource metadata scheme is being applied, and the issues that have been encountered by the open university and how it is attempting to resolve them. the need for change in organizational culture so that metadata becomes part of the creation process, rather than an afterthought, will then be discussed the paper concludes with a glimpse into the blue skies of the future where all resources will have metadata as standard practice, and institutions can share and utilize their resources effectively. defining metadata it is perhaps a reflection of the increasing importance of metadata that the term is now commonplace amongst publishing, computing and library and information communities and, as such, subject to many attempts to define it properly. definitions range from simplistic attempts such as, 'stuff about stuff' (dovey, 2000), to more comprehensive efforts to capture its meaning and purpose fully. of these, the definition we feel is most enlightening states 'metadata describes an information resource. it is the internet-age term for information that librarians traditionally have to put into catalogues, and it most commonly refers to descriptive information about web resources' (hillman, 2000). the underlying concept of metadata is not an unfamiliar one. descriptive information has long been applied to resources to enrich their existence and more recent metadata developments simply reflect our need to manage the proliferation of electronic resources with which we now have to cope. 52 a/t-j volume 10 number i metadata in the changing learning environment as technology evolves, so the landscape of learning is clearly changing. no longer confined to the school, further education, and university education progression, learners can choose when and how they acquire knowledge, and lifelong and modular learning have become commonly sought after educational experiences. the open university (ou) has always offered its students a flexible approach to learning, so it is perhaps unsurprising that we are now also embracing ways to extend our expertise into new realms of information delivery. our challenge is to meet users' expectations of flexibility and ease of use, prompted by their experiences of many online environments; by facilitating access to information, so that their efforts may be concentrated on achieving individual objectives, rather than merely searching for the objects to achieve them. we are convinced at the open university that effective resource description could indeed provide the key to meeting this challenge by: • enabling users to discover and identify existing materials; • providing them with sufficient data to evaluate and distinguish between different resources; • providing the option to personalize information presented, via a learner's information profile. open university case studies we will use two ou projects to illustrate the issues raised while implementing metadata, portfolio and the reusable educational software library (resl),1 which, although they differ significantly in their implementation, share the following features: • the use of ims learning resource metadata specification vl .02; • the objective of facilitating reuse of existing materials; • a change in responsibility for metadata creation from the hands of experts (information professionals) to those of the practitioners (content creators). portfolio digital archive the ultimate function of the portfolio digital archive is to provide a repository for storing, locating and reusing print-masters of all ou course materials. to achieve this, we have adopted a phased metadata implementation strategy, selecting in the first instance a skeleton set of ims metadata elements (see figure 1) in order to test their impact on locating and evaluating records. this was driven by the size of the database (10,000 items initially with 500 items added per month) and the resource implications. we felt that it was more likely to be successful if a small number of elements were chosen. our test results will then be used to guide subsequent phases of development, which will focus on how users may select items for reuse on the basis of pedagogical rationale, using educational elements. currently, metadata is being added retrospectively to each item within the archive, which is a huge and possibly insurmountable task, as approximately 500 new items are being added to the archive every month. work is therefore in progress to develop tools that will facilitate 53 ]uanita foster-jones and hazel beazleigh metadata in the changing learning environment portfolio metadata educational el*tn*nts l«—wing k m m v c * typ* classification elements classification scheme keywords tor classificition troad subject citegoi) |eatsdnctt relation elements relation type figure /: portfolio metadata elements phase i: ims descriptive elements have been used for portfolio, in conjunction with other university projects to secure its future interoperability with other repositories 54 alt-] volume 10 number i concurrent, author-generated metadata a goal shared by other projects within the ou within the course production workflow. this will ensure that authors will not be burdened with additional work, but that metadata they already generate is systematically collected. resl the reusable educational software library (resl) is a strand of the source (software use, reuse and customization in education) tltp project.3 resl originally aimed to test the feasibility of a national software library through the creation of a prototype and encourage academics to add examples of software to the prototype. this concept was later expanded to encompass case studies created by source and other tltp projects in order to enhance the feasibility study, whilst facilitating the dissemination of information. potential users of resl were identified as software developers, academics and administrators, who would each have different needs. these were investigated in reslou-1 (available from resl), and resulted in the use of four different controlled vocabularies, with terminology aimed at satisfying the different information needs. it was assumed that academics would have little experience of metadata creation, so data entry templates were created. these were converted into data entry forms, presenting the metadata author with only the fields, required for a specific type of resource to ensure consistency and quality of metadata supplied. a final proofing stage was also built into the workflow as part of the quality control process. the use of controlled vocabularies, represented as 'pick lists' in the data entry templates, maintained authority control, whilst free text keywords allowed a degree of flexibility for the metadata submitter. the controlled subject vocabularies are also used as browse lists on the search interface, linking metadata to the information retrieval process. this recognizes the fact that searchers may not have a clear idea of what it is they are looking for, or be able to articulate it (belkin, 1982). by providing browse lists, searchers have a starting-point to guide them to the resources, which will, we hope, lead to a match for their needs. resl implements selected elements from all of the ims categories. additional projectspecific metadata elements have also been added to fulfil certain project needs, but only after much deliberation since they could impinge upon interoperability. interoperability was seen as highly desirable; if resl becomes a national database we would wish to share records with other sources. where ims was extended, it was only after we had checked that no other field was suitable for our purpose, and only if we felt that the field would give benefit to the users, or increase the functionality of the database. for example, we added a 'date last visited' field after the url, so that users had an idea of the currency of the information provided. we also added a project field to the metametadata screen, allowing projects to submit their project url and logo, thus identifying resources they had submitted. it was felt that this would encourage other tltp projects to add resources. it should also be noted that resl does not use xml binding for its metadata, due to time and resource constraints.4 if the project were to be scaled, then xml would need to be considered to ensure interoperability with other ims compliant systems. issues for both projects, the primary concern was to determine whether to use an existing metadata schema, or to create an ou-specific schema. in both instances we decided to 55 juanita foster-jones and hazel beazkigh metadata in the changing learning environment adopt an existing schema. this ensured that time was not wasted in duplicating effort, and would facilitate future interoperability. ims was chosen for its semantic density, in particular for the educational elements. the ou also has a responsibility as joint partners in cetis to contribute to ims development and thus, by implementing our metadata, we can then feed our experiences into future revisions. however, adoption of this schema has identified a number of issues. the ims metadata model consists of nine categories, with a total of sixty element tags for describing a learning resource, all of which are optional. having decided upon using ims, we then needed to select which elements to implement. different projects have different needs, and so will use a variety of element combinations. in addition, we also needed to decide on the depth of description to give. for portfolio, a staged approach was taken, starting with a minimum number of elements with the option to increase as required, whereas for resl a comprehensive approach was taken, implementing a large number of elements to test the ims metadata. portfolio used the staged approach for two reasons. firstly, due to the sheer size of the database, choosing a limited number of elements to complete would ensure more items were given metadata. secondly, portfolio was investigating the levels of granularity to which metadata should be added, that is, should metadata be added at the item, chapter or paragraph level. a small set of metadata would facilitate this. for resl we wanted to take advantage of the semantic density of the ims schema, to facilitate in resource identification and evaluation. for example, in the technical category, ims provides a number of elements which describe the hardware and software requirements of the resource. these were felt to be of use to the end user, as they would provide an indication of what equipment is required to access the learning resources described on the database. institutions will need to decide what level of metadata entry they can resource. selecting appropriate elements has proved to be no simple task. in some cases this has been hampered by the lack of examples provided with ims specifications. our experience has shown that descriptions within the guidelines can be complex and difficult to interpret, due to their terminology and technical bias. although we were approaching this as qualified librarians, and thus as experts in metadata, we have still experienced difficulties. to this end we have decided to look at all metadata elements within the ims specification, and give examples of how they may be used within the ou context. this will make adoption by other projects easier, and encourage a standardized implementation. where ims specifications refer to other documents for best practice it can be difficult to locate the relevant part, as you are referred to a main site. some of the documents referred to are also aimed at users with technical backgrounds, and can be complex to read and understand, often hindering progress. it should also be noted that ims did not meet all requirements of either project, although we recognize that it is unlikely that any standard would accomplish this. both portfolio and resl have had to make decisions about whether to add metadata elements to the schema. this was addressed with caution, on the understanding that any deviance would impinge on interoperability. where additional fields were added, it was only after all ims elements had been considered for their suitability. guidelines within the ims meta-data best 56 alt-j volume 10 number i practice and implementation guide (anderson and wason, 2000) on extending ims would have been welcome. finally there is the fact that ims is a specification, and hence susceptible to revision. a plan of action needs to be determined to review each new version, and decide to what extent any changes should be implemented. it may be that metadata is upgraded for new items, but those using the old schema are left unrevised due to lack of resources. lessons learnt the ou is still in the early stages of metadata implementation. as we become more experienced in our use of metadata, we shall be able to address some of the issues raised by these two projects. as further projects recognize the importance of metadata, it is likely that new issues will be raised. there is still a considerable amount of work to be done to ensure that metadata is created and exploited to its full potential, and that the issues identified above are overcome. experience from portfolio has shown that retrospective metadata application to existing, populated databanks is a herculean task. methods of creating metadata automatically at the point of item creation would help address this but would require working practices to change, to make metadata generation an integral aspect of authorship. tools such as dcdot5 and the sun microsystems developers toolkit6 can assist in this automatic generation of metadata, and work will need to be carried out to investigate their effectiveness, and then compare the quality of metadata generated by tools such as these with manually generated metadata. for any such change to be successful it requires authors to become stakeholders, and to share ownership of metadata. to do this we must first prove its value, by making resource location more effective and by removing fears of extra work by automating the process as much as possible. the ou is currently examining the use of xml style-sheets to see if they can provide a workable solution. metadata needs to be an integral part of an institution's information strategy. recognition and support for metadata is required from the top of an organization's structure. only with official recognition will it be possible to put measures in place for a standardized approach to metadata creation and quality control. within the ou a metadata working group has been created, to provide recommendations for a metadata strategy. this will be presented to ou boards for approval, and by so doing will set metadata firmly on the ou agenda. there is also a need for implementers to share implementation and best practice examples, and to build communities of practice. given that many metadata schemas are not yet standards, there is an opportunity for early adopters to influence their further development. blue skies it is too easy to become distracted by the obstacles and issues to be overcome, and to question the relevance of metadata. but it is crucial that we invest our efforts and resources in this developmental phase so that we can contribute to the evolution of metadata standards, using our experiences to create one that best suits our needs. 57 juanita foster-jones and hazel beazleigh metadata in the changing learning environment we should not lose sight of the ultimate objectives of this exercise, which are to locate, share and reuse the learning resources that are the assets of our organization. like information, a resource is only valuable if it can be found and utilized in time. 'the value of information often depends on how easily it can be found, retrieved, accessed, filtered and managed' (martinez, 2001). perhaps the way forward to achieving interoperability will be through the use of application profiles. application profiles 'consist of data elements drawn from one or more namespace schemas combined together by implementers and optimised for a particular local application' (heery and patel, 2000). in the blue skies of the future there will be a registry in which application profiles are stored. these will be accessible to implementers, so that they can benefit from work already completed, and to creators of metadata tools, to assist with the development of metadata creation and information retrieval processes that take into account this 'pick and mix' approach to metadata. utilization of the resource description framework (rdf)7 will contribute to this tailored metadata approach: the objective of rdf is to support the interoperability of metadata. rdf allows descriptions of web resources any object with a uniform resource identifier (uri) as its address to be made available in machine understandable form. this enables the semantics of objects to be expressible and exploitable. once highly deployed, this will enable services to develop processing rules for automated decision-making about web resources. (iannella, 1998) rdf allows a resource to be described using elements from multiple schemas, in effect providing the structure for application profiles. the aim of all metadata is to make resource location possible. we value libraries for structuring and managing books for research, therefore we should no less value metadata creators for structuring and managing our learning resource assets. at the ou we are making a commitment to metadata standards, to ensure the future accessibility of all our assets. notes 1 resl: http://www.resl.ac.uk/ (last accessed 17 december 2001). 2 ims: http://www. imsproject. orglindex. html (last accessed 2 january 2002). 3 source: http://www.source.ac.uk/ (last accessed 4 october 2001). 4 a full list of resl metadata elements, and associated rationale can be found in reslou-4 and resl-ou-5 on the source web site http://www.source.acuk/. 5 dc-dot: http://www.ukoln.ac.uk/metadataldcdot/ (last accessed 17 december 2001). 6 sun microsystems developers toolkit: http://www.imsproject.org/tools/sun.html (last accessed 17 december 2001). 7 rdf: http://www. w3. org/rdf/ (last accessed 2 january 2002). references anderson, t. and wason, t. (2000), 'ims meta-data best practice and implementation guide', version 1.1. available from: http://www.imsproject.org (last accessed 25 october 2001). 58 alt-j volume 10 number i belkin, n. i , oddy, r. n. and brooks, h. m. (1982), 'ask for information retrieval: part 1, background and theory', journal of documentation, 38 (2), 61-71. dovey, m. j. (2000),' "stuff" about " s t u f f the differing meanings of "metadata"', vine, 116, 6-13. heery, r. and patel, m. (2000), 'application profiles: mixing and matching metadata schemas', ariadne, 25, http://www.ariadne.ac.uklissue25/app-profiles/intro.html (last accessed 26 july 2001). hillman, d. (2000), 'using dublin core'. available from: http://dublincore.org/documentsl 2000/0716/usageguide/ (last accessed 18 april 2001). ianella, r. (1998), 'an idiot's guide to the resource description framework'. available from: http://archive.dtscedu.au/kdu/reports/rdf-idiotl (last accessed 19 december 2001). kim, y., clarke, s. and ramsden, a. (1999), 'baseline report setting the context for reusable educational software library (resl) of source project (resl-ou-1)'. available from: http://www.eres.ac.uk/source/docs/resl-ou-l.pdf (last accessed 17 december 2001). martinez, jose m. (2001), 'overview of the mpeg-7 standard', iso iiec jtci/ sc29/wg11 n4031. available from: http://www.cselt.it/mpeg/standards/mpeg-7/mpeg-7. htm (last accessed 16 may 2001). 59 alt-j, research in learning technology vol. 13, no. 3, october 2005, pp. 241–247 issn 0968-7769 (print)/issn 1741-1629 (online)/05/030241–07 © 2005 association for learning technology doi: 10.1080/09687760500376512 from open resources to educational opportunity m. s. vijay kumar* massachusetts institute of technology, usa taylor and francis ltdcalt_a_137634.sgm10.1080/09687760500376512alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)discussion piece2005taylor & francis ltd133000000october 2005m. s.kumars.porter@jisc.ac.uk since mit’s bold announcement of the opencourseware initiative in 2001, the content of over 700 of its courses have been published on the web and made available for free to the world. important infrastructure initiatives have also been launched recently with a view to enabling the sustainable implementation of these educational programmes, through strengthening organizational capacity as well as through building open, standards-based technology. each of these initiatives point to a rich palette of transformational possibilities for education; together with the growing open source movement, they offer glimpses of a sustainable ecology of substantial and quality educational resources. this discussion piece will highlight some of the educational opportunity presented by mit’s current information technology-enabled educational agenda and related initiatives, along with their strategic underpinnings and implications. it will address various dimensions of their impact on the form and function of education. it will examine how these ambitious programmes achieve a vision characterized by an abundance of sustainable, transformative educational opportunities, not merely pervasive technology. the internet, open content and active learning opencourseware looks counter-intuitive in a market-driven world. it goes against the grain of current material values. but it really is consistent with what i believe is the best about mit. it is innovative. it expresses our belief in the way education can be advanced— by constantly widening access to information and by inspiring others to participate. (charles m. vest, president emiritus, mit, september 2001) mit’s announcement to the world in april 2001, that it would be posting the content of some 2,000 mit courses on the web, is certainly a visible and voluble signal of the institute’s intent to extend technology-enabled opportunities to expand educational access and quality. no educational institution has ever proposed anything as revolutionary, or as daunting. * massachusetts institute of technology, 211 massachusetts avenue, room n42–253, cambridge, ma, usa. email: vkumar@mit.edu 242 m. s. vijay kumar the idea behind mit opencourseware (ocw) (http://web.mit.edu/oki) is to make course materials that are used in the teaching of almost all undergraduate and graduate subjects at mit available on the web free of charge, to any user anywhere in the world. ocw is a large-scale, web-based electronic publishing initiative funded jointly by the william and flora hewlett foundation, the andrew w. mellon foundation and mit. its goals are: 1. to provide free, searchable, coherent access to mit’s course materials for educators in the non-profit sector, students and individual learners around the world; and 2. to create an efficient, standards-based model that other universities may emulate to publish their own course materials. extensive metadata tagging, launching a comprehensive ongoing evaluation process, developing and enhancing our content management and publishing technologies, and evolving our internal staffing and workflow are important elements of this process. mit ocw will advance technology-enhanced education at mit, and will serve as a model for university dissemination of knowledge in the internet age. this venture continues the tradition at mit of open dissemination of educational materials, philosophy and modes of thought, and will help lead to fundamental changes in the way colleges and universities utilize the internet as a vehicle for education (diamond, 2003). according to miyagawa, an mit professor of linguistics, ‘ocw reflects the idea that, as scholars and teachers, we wish to share freely the knowledge we generate through our research and teaching’. he goes on to state that ‘while mit may be better known for our research, with ocw, we wish to showcase the quality of our teaching’. as of june 2005, 1,100 courses in areas ranging from computer science to political science have been published. the positive reaction and global impact of the initiative is evidenced through the extensive visits to the ocw site from people in countries across the world and also the comments on the value of the materials (http://ocw.mit.edu/ocwweb/global/aboutocw/worldreaction.htm). while mit’s initial efforts focused on meeting the original publishing target, it has also looked at ways to extend ocw’s reach through exploring translation efforts (such as spanish), through tools to facilitate the use and through the engagement of like-minded institutions. ocw has important implications for meeting the needs of education beyond providing extensive access to course content. ocw presents a snapshot of the transactions in a course—the lectures, assignments, and discussions, along with the structural relationship between these entities—thereby providing a view of the practices and pedagogies employed, and hence a useful model for educators everywhere. while ocw has been the keynote, it has not been an isolated event in mit’s agenda to employ technology for educational transformation. projects, such as those supported by the i-campus initiative (an alliance between mit and microsoft research) as well as the mit council on educational technology (mitcet), have led to innovations for developing content, tools and pedagogy with the potential to from open resources to educational opportunity 243 significantly impact the form and function of educational practice. the following examples illustrate this. transforming field studies mit students from the environmental studies programme have been traveling in australia and new zealand collecting data from soil and water samples through environmental and geo-positional sensors, storing the data, making computations in the field based on the data, and displaying the data. this group of students was part of the envit project at mit, whose goal was to create mobile field data collection software for the environmental professional. the central piece of technology employed here is a personal digital assistant integrated with other hardware and software modified for mobility; namely, teletype gps sensors, hydrolab multi-parameter water quality probes, esri arcpad mobil gis and windows ce development using embedded visual basic, c++ and access. this innovation has moved field sampling and measurements (still largely dependent upon pencil-and-paper field notebooks) to a new level of accuracy and efficiency. it has wonderful implications for mobile learners, field engineers and scientists who have hitherto lacked the appropriate technology to carry out their work in an efficient manner. ilab—first-hand experience over the internet over the past two years, students at globally distributed locations such as institutions in singapore, which are part of mit’s singapore mit alliance programme, as well as sweden among others, were able to use laboratory equipment located at mit. specifically, these students used network analyzer equipment employed in microelectronics courses for various experiments, such as measuring characterizations of transistors. the ilab project (http://swiss.csail.mit.edu/projects/icampus/projects/ilab.html) that makes this possible enables access to these instruments through a web interface. through ilab it is possible to deliver hands-on experimentation to students anywhere, at anytime. in addition to microelectronics, laboratories and experimental set-ups that are being web-enabled include reactors for polymerization, heat exchangers and shake tables for seismic measurements. the ilab ambition is to create the framework to extend access beyond these specific experiments to a variety of remote, geographically dispersed laboratories. educational value a significant characteristic of the technological innovations described here is that they support active learning and first-hand experience, both important elements of the mit’s educational programmes, in flexible, location-independent and contextually relevant ways that add richness to the learning experience. another notable example of promoting active learning is technology enabled active learning, which involves transforming large lecture courses to active, collaborative studio-based learning for physics. 244 m. s. vijay kumar an important educational characteristic of these technological innovations is also that they induce important proximities between learners, their peers and instructors as well as the community. the net result is an increased interaction of the learner with the content, context and community. the educational promise and strategic underpinnings the promise of new technologies in terms of new pedagogy, abundant content and new clientele for education are numerous and seductive. however, understanding and exploiting these opportunities requires educational institutions to have a ‘strategic focus’; that is, they should select a limited set of opportunities to explore and should pursue those opportunities in depth. institutions must identify specific targets of opportunity and need and particular areas of intrinsic value and potential uniqueness (kumar, 2000). mit’s approach leading to the launch of these innovations reflects (at least in hindsight) such a strategic focus. starting in 1999, the mitcet undertook a six-month study looking at how mit should respond to the opportunity presented by emerging technology; in particular, the internet. the visioning exercise led to a variety of potential scenarios of how mit could position itself. the starting point of this exercise was a reflection on and articulation of the unique aspects of the mit educational experience, to answer the question: ‘what is mit’s educational value proposition?’ the mitcet identified that a central dimension of mit’s educational value was the high bandwidth of interaction between excellent students and excellent faculty. another was the importance of first-hand experience in the educational process. mit’s educational technology strategy therefore was to launch experiments to: understand how information technology could amplify these value dimensions, extend it to other communities not typically served by mit, and address existing obstacles to achieving this value in current practice. mit’s educational technology initiatives over the past few years reflect this strategy and address the inherent opportunities and challenges. the ocw idea, that the best way to advance mit’s fundamental mission and exploit the internet would be to put all mit’s course materials on the web, open to everyone, and free, was, in retrospect, a logical conclusion to this strategic exercise. both ilabs, which builds on the values of hands-on experience in mit education, and technology enabled active learning, which addresses the problem of large lectures through studio-based learning in small collaborative groups, also reflect how the educational technology strategy was in alignment with the educational value proposition. the strategizing exercise by the mitcet set the stage for a series of large-scale educational experiments to promote innovative teaching and learning, as well as for the creation of an educational commons. infrastructure for sustainability important infrastructure initiatives were also launched with a view to enabling the sustainable implementation of these educational programmes, through building from open resources to educational opportunity 245 technology platforms and facilities as well as through strengthening organizational capacity and alignment. in the area of building technology platforms, three initiatives are important to mention in the context of this discussion: stellar, mit’s course management system; dspace, a project to develop a digital repository for educational materials; and the open knowledge initiative (oki), for developing an open architecture to support the portability and sustainability of educational applications and their easily integration into campus infrastructures. the long-term goals of these infrastructure efforts are to link the mit community worldwide, and to provide easy access to the full range of mit’s intellectual life and resources (library materials, research facilities, colloquia, etc.). mit’s infrastructure will make it easy to communicate, follow coursework and collaborate online. porting, sharing and integration through the oki the oki provides educational projects with a general framework that allows technology innovation to be safely incorporated into live learning environments. innovators’ work can also be easily shared with users and developers at other institutions that support the oki framework. for instance, oki will enable mit, the university of michigan and indiana university to offer leverage to their development efforts to share tool and applications in the evolution of their collaborative sakai platform (www.sakaiproject.edu). it will also enable institutions to provide a rich suite of educational functionality such as for assessment and concept mapping as well as for searching and presenting image collections. the visual understanding environment (http://vue.tccs.tufts.edu/) project at tufts university and the narravision application being developed at mit, to support the visualizing cultures class course, are examples. these tools use software provided by the oki that allows several repositories to be accessed simultaneously and offer leverages to the local institute’s infrastructure. oki enables research and development efforts to concentrate on core functionality, rather than spending time and money building simulated environments or managing complex system integrations. so how does oki do this? oki started with a somewhat modest ambition: to develop extensible and customizable applications that provide learning management systems functionality. early in the project, its leaders recognized that open, extensible and clearly defined architectural specifications are critical to realizing this ambition, as well as for other educational applications. the interface methods defined by oki support the ongoing integration of three general categories of software: ● learning applications ranging from individual quizzing, authoring and collaboration tools to suites of such tools that include course management and learning management capabilities. ● central administrative systems such as student information, human resources and directory management. ● academic systems including library information systems and digital repositories of research and educational materials. 246 m. s. vijay kumar at the time of launching oki we observed that while existing learning management systems were serving a very important function, in that they had created a very low barrier to entry for universities mounting course materials, they presented a high barrier to exit. technical design aspects limited the portability and interoperability of learning resources and consequently inhibited the kind of flexibility that leads to greater use and sharing of learning materials. for example, the tight coupling of the user interface to learning components (as in a monolithic learning system) constrains the ability for resources to be used in different educational settings and contexts. typically the solutions available were unsuitable because they were all-in-one packages with components offering a single level of functionality designed for widespread use. oki embraces a component-based approach that relies on loose, lightweight, but well-defined contracts to enable links to common infrastructure services. oki provides specifications for interfaces among components of a learning technology environment. the intent is to make it easier for campus and commercial developers to produce software that will work in a range of higher education infrastructures to satisfy diverse educational needs. by providing a stable, scalable base that supports the flexibility needed by higher education, oki offers an approach that may help us to produce and share high-quality courseware and to construct enterprise applications more cost-effectively and more efficiently, as learning technology is increasingly integrated into the education process. the oki has produced and made available as open source, a series of open service interface definitions informed by a broad architectural view of the educational technology landscape (http://www.okiproject.org). oki’s work in the educational systems is by no means the first. initial work in the interoperability area such as that of the ims global consortium led to some common definitions and standards for data interchange (sharable content object reference model). these are useful for copying content from one system to another. they primarily address the encoding of data for exchange among cooperating systems and address issues such as metadata and learning design that address the semantics of learning content. however, to get more complete real-time integration, service specifications are required in addition to data specifications. oki is still only a step in the path of component-based approaches to design and the evolution of global specifications and standards, crucial for advancing the use and interoperability of learning resources. oki’s open service interface definitions are available in open source mode to facilitate the large-scale adoption of this architectural model. this will allow institutions to easily take advantage of new technology and new learning management components as they become available. it will also allow components to be updated individually without destabilizing the overall environment. the end game: ecology of open educational opportunity mit’s approach for developing its educational technology strategy urges careful consideration of the core values of an institution and reflection of its value proposition. from open resources to educational opportunity 247 sustainability has been a key theme in mit’s efforts, and ensuring alignment with institutional and educational values was seen as a necessary first-step in this direction. selecting a limited set of educational experiments was an important aspect of our efforts, as was designing an infrastructure to enable the success and durability of these experiments. our efforts toward this end had to also take into account some ‘givens’ of the educational ecosystem: ● educational value is derived in multiple ways and modalities, requiring a heterogeneous set of tools. ● the technologies on which we build our infrastructure will necessarily change. ● the implementation of educational applications of scale and substance requires integration with the existing infrastructure and enterprise. the end goal is to be able to ensure that the quality educational content and tools that are being created through innovative uses of technology at mit elsewhere can indeed be leveraged to substantively impact educational opportunity and transformation in its form and function. in the period ahead we will be learning from the implementation of our efforts. the feedback we receive about the dynamics of adoption including contextual considerations will influence ongoing development of our initiatives and help deliver the promise of advancing learning through technology and progress in education. references diamond, d. (2003) mit everyware, wired magazine, 11 september. available online at: http:// www.wired.com/wired/archive/11.09/mit_pr.html (accessed 20 september 2003). kumar, v. (2000) choosing the right track for it’s transformation of teaching and learning, viewpoints, educause review, june. long, p. d., merriman, j., kumar, v. & walker, e. (2003) specifications and common services for interoperability: new landscape for learning technology, syllabus magazine, 1 july. available online at: http://www.syllabus.com/article.asp?id=78 (accessed 20 september 2003). merriman, j. (2002) what is oki?, white paper. available online at: http://web.mit.edu/oki. merriman, j. & kumar, v. (2001) building ‘open’ frameworks for education, new horizons, educause review, 36(6). tansey, f. (2003) the standard bearers close ranks, syllabus. available online at: http:// www.syllabus.com/article.asp?id=7359 (accessed 20 september 2003). alt-j, research in learning technology vol. 12, no. 3, september 2004 issn 0968–7769 (print)/issn 1741–1629 (online)/04/030205–09 © 2004 association for learning technology doi: 10.1080/0968776042000259537 putting interoperability to the test: building a large reusable assessment item bank niall sclatera* & mary macdonaldb auniversity of strathclyde, uk; bcolleges open learning exchange group (coleg) taylor and francis ltdcalt120302.sgm10.1080/0968776042000259537alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2004association for learning technology123000000september 2004niallsclaterniall.sclater@strath.ac.uk the cola project has been developing a large bank of assessment items for units across the scottish further education curriculum since may 2003. these will be made available to learners mainly via colleges’ virtual learning environments (vles). many people have been involved in the development of the cola assessment item bank to ensure a high level of technical and pedagogical quality. processes have included deciding on appropriate item types and subject areas, training authors, peer-reviewing and quality assuring the items and assessments, and ensuring they are tagged with appropriate metadata. one of the biggest challenges has been to ensure that the assessments are deliverable across the four main virtual learning environments in use in scottish colleges—and also through a stand-alone assessment system. cola is significant because no other large project appears to have successfully developed standards-compliant assessment content for delivery across multiple vles. this paper discusses how cola has dealt with the organizational, pedagogical and technical issues which arise when commissioning items from many authors for delivery across an educational sector. introduction various people have attempted to provide a definition of an item bank. these range from a simple: collection of text items that may be easily accessed for use in preparing exams (ward & murray-ward, 1994) to the more detailed but less generic: collection of test items that can be readily accessed for use in preparing examinations … normally computerized for ease of item storage and to facilitate the generation of new tests. each item … is coded according to competency area and instructional objective, as well as * corresponding author: learning services, 155 george street, university of strathclyde, glasgow, uk. email: niall.sclater@strath.ac.uk 206 n. sclater & m. macdonald empirically derived data such as measures of item difficulty and discrimination. (mccallon & schumacker, 2002) some of these definitions incorporate the concept of different but equivalent assessments being produced dynamically and automatically for each learner from a bank of items. others imply that the database will be used to store data about the usage of the items by learners. all the definitions suggest that items should be classified by descriptive data (metadata) of some sort to enable them to be located. the decisions taken over what type of metadata to use as well as the content and type of items will differentiate one item bank from another. while item banks have been around for many years and in a range of contexts, various factors are now coming together which suggest that their use is set to increase considerably. firstly the software is now available and the hardware ubiquitous enough to deliver assessments to learners either through virtual learning environments or bespoke online assessment systems. secondly there is an internationally-recognized format for the transfer of items between these systems (ims, 2002). this format can also be used to store items in a database separately from any proprietary assessment delivery system. thirdly there are now pressing economic and political imperatives for the development of national and international item banks. developing items and assessments across a subject area or sector can bring economies of scale in the development process and a considerable reduction in duplication of effort in different colleges and universities. the quality of items which are peer reviewed and validated centrally is likely to be higher than those developed on an ad-hoc basis in an individual institution. an increased adherence to technical standards should mean that the lifespan of items is prolonged and that items are more likely to be deliverable through a variety of assessment systems and virtual learning environments. there are already some successful examples of item banks under development. these tend to be either: (1) assessment-specific, e.g. the english as a foreign language item bank for the university of cambridge local examination syndicate; or (2) subject-specific, e.g. the electronics and electrical engineering assessment network (e3an), helping engineers learn mathematics (helm) and various initiatives taken by the learning and teaching support networks which are developing items in economics and computing. however, a third type of item bank is now emerging: sectoral. the cola (coleg online assessment) project is developing a large bank of items across the entire scottish further education (fe) curriculum. a simple definition of an item bank which incorporates these three types might be: a collection of items for a particular assessment, subject or educational sector, classified by metadata which facilitates searching or automated test creation. management of the cola project the cola project was established with funding from the scottish further education funding council (sfefc) which identified online assessment as a strategic priority. a large reusable assessment item bank 207 most scottish colleges had already deployed virtual learning environments (vles), providing new opportunities for online learning. feedback from the sector showed that the lack of a national database of assessment instruments was proving a barrier to widespread use of the vles for assessment purposes. in addition the scottish qualifications agency (sqa) had recently produced a set of guidelines for the use of online assessment (sqa, 2003) and was developing a strategy in this area. the funding council believed that the use of online assessment could reduce the burden on academic staff and encourage more of them to engage with information and communications technology for learning and teaching. cola’s aim was to develop a bank of high quality assessment instruments capable of being delivered through the four main vles in use in scottish colleges in a wide range of courses at all levels within further education (fe). the project is managed by the colleges open learning exchange group (coleg)—a partnership of 42 scottish colleges which undertakes collaborative projects to develop, exchange and promote open, flexible and online learning materials. coleg manages each project while college staff write, produce and peer review the materials and quality assurance staff check them through a rigorous quality assurance process before dissemination to the sector. coleg has used the same approach for the cola project. a steering group was formed to oversee the project, which includes representation from the various agencies, senior fe managers and fe practitioners with expertise and experience in online assessment, vles, interoperability issues and staff development. the project team includes a project manager, an administrator, a technical consultant with expertise in online assessment and interoperability, a technical advisor experienced in online assessment and a staff developer. a technical advisory group was also appointed from college staff with substantial experience in on-line assessment and expertise in vles and interoperability issues. this group has strong links with the cetis assessment special interest group and ims, the international body responsible for assessment interoperability specifications. selecting areas for assessment the prime aim of cola is to provide a bank of assessment instruments to encourage more widespread use of vles by college staff across the curriculum. awareness of the project was raised through local subject networks and staff were encouraged to put forward their suggestions for areas of the curriculum that would be appropriate for online assessment. as a starting point for selecting areas of the curriculum, staff were asked to focus on learning outcomes within sqa units that would be appropriate for objective testing. in practice subject specialists created assessments to meet the formative assessment requirements of complete outcomes, parts of outcomes (performance criteria) or a combination of topics (performance criteria) from several outcomes. it was recommended that an assessment should contain a maximum of twenty items in total. there was a general view among academic staff that twenty multiple 208 n. sclater & m. macdonald choice items would normally cover the formative assessment requirements of one sqa outcome. choosing item types the project has concentrated on developing pedagogically sound objective tests, using a limited number of item types. after consulting the e3an project team on the item types they had selected for their item bank in engineering, the types chosen were true/false, multiple choice, multiple response, fill in the blank and matching. there were a number of reasons for selecting a limited range of item types. the wide range and spread of assessments would be limited by specifying a small number of item types, allowing the assessment of a variety of skills and cognitive levels. a focused programme of staff development could be provided for writers. the assessments had to work in a range of vles and it was expected that the vles would accept these item types if they were marked up using the ims question and test interoperability v1.2 (qti) specification (ims, 2002). development of the templates to simplify the process of item creation, standard word templates were developed for the college writers. this approach had already been used successfully by the e3an project and it was expected that staff familiar with using word would be able to input content to the templates easily. a template was created for each item type. item templates allow authors to specify the stem of an item, the options and the correct answer, to incorporate graphics in the stem and the options and to provide feedback for each option. they also included a section for additional information such as the expected time to be taken, a description of the item, keywords and the subject topic. in addition an assessment template was developed to contain metadata about the assessment itself and to specify which items were contained in the assessment. metadata as the cola item bank grows it will become increasingly important to provide an adequate means of identifying items and assessments. the provision of appropriate and accurate metadata makes this possible. there is now an international standard for learning object metadata (lom) published by the ieee (ieee, 2003) which was chosen as the format in which to store cola metadata. if a cola item or assessment is uploaded to a vle or content repository, the metadata should be instantly recognized and allow users to search for the material on the metadata fields. ieee lom had never before been used to classify items and assessments. however, a group of uk experts has got together to produce an application profile (a kind of subset specifying mandatory and optional elements) of the lom for use within uk further and higher education. this is known as the uk lom core. it seemed appropriate to utilize this application profile for cola in order to maximize a large reusable assessment item bank 209 the chances of its metadata being understood by other systems. cola worked with experts in metadata and assessment to produce further application profiles of the uk lom core for items and assessments. work done for the cola project on metadata and content packaging has fed directly into v2.0 of the ims question and test interoperability specification. the cola templates allow authors to enter items and assessments and also to complete most of the metadata used to classify them. a template conversion tool which was built for the project ensures that metadata fields are transferred accurately and consistently from the templates to the lom format, while automatically completing some of the more esoteric fields which authors might have found difficult to understand. this is a much better solution than giving authors access to a tool which requires them to understand the lom format itself. it ensures that metadata across the entire collection of cola assessments and items has high levels of quality and consistency without creating an excessive burden on authors. each item is classified by the scottish credit and qualifications framework (scqf) level, a number from 1 to 8. assessment-level classification metadata is defined in a similar way to that of items. in addition to the level there are entries for the sqa outcome number, the performance criteria, the unit number and the unit title. identifying, training and supporting assessment writers coleg used its standard approach to recruit writers, working through its network of contacts in the colleges to disseminate information about the project to staff and to invite them to commit to the project. the project was launched with an awarenessraising event for college staff—curricular, technical and management—to explain the aims of the project, timescales and funding arrangements and to listen to their views on implementation. following the event, colleges were asked to confirm the services that they could provide to the project. standard levels of payment were set for writing and inputting of the assessments into the templates, for peer reviewing, for quality assurance and for project management. writers confirmed on a proforma the curriculum areas/ topics and peer reviewers of their assessments. at the same time technical staff with relevant experience of the different vles were invited to join the technical advisory group for the project and to advise the steering group on technical issues. thereafter a series of two workshops was organized for writers and peer reviewers. the workshops provided information about the project and clarified its focus on objective tests. the various item and assessment templates and the item types chosen were explained. a set of guidelines was created, including a writer’s/peer reviewer’s quality checklist for each item type and for the assessment information and a guide to completion of the templates. evaluation forms showed that the workshops were wellreceived by participants. technical and pedagogical quality of the items is likely to be higher than if they not been carried out. certainly there would have been confusion about the use of the templates. the writers also confirmed that the workshops helped 210 n. sclater & m. macdonald their understanding of the pedagogy of objective tests not just their understanding of the templates. a timescale of six weeks between may and june 2003 was set between the first writers’ workshop and the deadline for submission of the assessments by the writers. following the workshops, one-to-one guidance on pedagogy related issues was available. email and telephone support was also available for both pedagogical and technical issues. a further series of one day workshops was held for a second phase of development work between july and september 2003. in total 66 writers delivered 165 assessments (approximately 3000 items) in the first two phases of the project. only three writers withdrew from the project. quality assurance coleg implemented its standard quality assurance procedures in the project, including checking the quality of the items (from the subject specialist’s and the learner’s perspective), checking the quality of the production (grammar, typos) and checking the technical aspects (e.g. completion of template fields, use of standard file names). in checking the quality of the items from the learner’s perspective the quality assurance staff identified several key issues: ● what is to be assessed? ● why has a particular item type been selected? ● is the item or instruction (stem) clear? ● is contextualization necessary and appropriate? of the items created, 75% were considered to be of good quality. after further development work, it has been possible, with the exception of five assessments, to validate all the assessments in the first development phase. where there were questions over the quality of the assessments, the robustness of the peer review process was questioned, particularly where the wording of an item was inappropriate or the item type used was not suitable. it was felt that the quality of the feedback to the learner was particularly important for assessments which would primarily be used formatively. however, the online context and the vle technology sometimes limited the feedback that could be provided. for some item types it was stated in the guidance that only standard (no, this is incorrect or yes, this is correct) feedback could be provided because it would be impossible to predict the learner’s responses to the items. in practice some writers proved to be extremely creative with the additional general feedback that they provided. it was not possible to clarify some of the technical issues related to the templates at the time of the workshops. in addition further issues were identified at the later stage of testing of the exemplar assessments. in both cases these were addressed at the quality assurance stage. in the main it was felt that the writers had made a reasonable attempt to complete the fields in the templates. the general view was that it was important for the writers to gain skills in data input and that this would give them a better understanding of a large reusable assessment item bank 211 how the vles would handle the assessments. it was also established that it would be possible to standardize more of the content in the template such as feedback. this would reduce the potential for error. version control and file management has been an important issue during the quality assurance process. a file management system has been developed for the project that classifies the assessments into three categories: ● initial version: received from the writer following peer review; ● part-validated version: quality assured but checking or amendment required by the writer; ● validated version: approved by the quality assurance staff. a spreadsheet has been developed to record details of writer, peer reviewer, subject area and level, quality assurance process and administrative details. overall this system has worked well, though management and maintenance of the files has been time-consuming and requires a great deal of care and attention to detail. in a small number of cases writers changed items that had been validated and these needed to be rechecked. wherever practical, writers have been asked to notify the quality assurance staff of the amendments that they want to make rather than changing the templates themselves. transfer to qti and vle formats one of the primary aims of cola was to encourage colleges to use their vles by providing online assessments which could be run from the vles. in order to do this, the assessments had to be in a format which the vles would understand. the only international specification (not yet a standard) for the exchange of items and assessments is the ims question and test interoperability v1.2 (qti) specification. many vendors pay lip service to their products’ compliance with this specification but do not properly implement it. the four vles in use in scottish colleges all claim some level of compliance with the specification. cola took the decision to store all content in this platform independent format which is undoubtedly increasing in uptake worldwide. it was necessary to develop a program to convert the items and assessments from the word templates to the qti format. this task was carried out by the jisc-funded technologies for online interoperable assessment (toia) project which had the necessary expertise in qti in collaboration with an expert group representing the four main vles. there were many complications due to the different ways in which the vles interpreted the qti specification and their limited implementations of some of the item types. using a third party product called respondus which accepts qti it is now possible to transfer cola content into webct and blackboard. teknical now accepts cola content directly and it is still hoped that a solution can be found to put cola items into granada learnwise. having produced the items and assessments to the correct standard, they can also be uploaded and stored in some of the emerging learning content repositories with 212 n. sclater & m. macdonald ease, allowing items to be searched for on their metadata. in order for teachers to be able to search for items with ease the conversion tool creates two indexes which can be read using microsoft excel—one for items, the other for assessments. on each line of the spreadsheet is one item/assessment and all the metadata associated with that item such as author name and sqa unit number. distribution much discussion took place in the technical advisory group meetings as to how to distribute the items and assessments. while the distribution of cd-roms would have provided a further opportunity to disseminate the project to colleges, the technology is a backward one and it was considered to be simpler for colleges to download the latest versions of the item bank and install them in their vles directly from a central website. the coleg named contact in each college would be authenticated to do so. separate indexes of all items and assessments will be provided on the website, both searchable on any item of the metadata. staff will then be able to download the items and assessments required in ims qti format so that they can import them into their vle. conclusions the templates were developed in word for ease of use and overall writers have coped reasonably well with them. the development process has however highlighted a number of issues. there are limitations in the type of data input that the template will allow. it would be possible to standardize the feedback in some cases, thus reducing the potential for error. the filing protocol is cumbersome and a more simple referencing arrangement should be devised. a web-based development system, while requiring authors to be online, would remove the problems encountered with authors misnaming and misplacing the various item and assessment templates and graphic files. the workshops were well received by writers and the same format will be used in the future. the guidance and checklists will be reviewed and improved in light of feedback during the quality assurance process. this process has clarified where writers are likely to make errors. also, it will be possible to demonstrate real examples of the different item types and the creativity of writers using the items generated in the first phase of the project. the quality assurance process itself has worked well and reduced the burden on writers. the process has been resource intensive however and there has been a limited pool of staff to undertake the work. the process has also highlighted the importance of recruiting experienced quality assurance staff. care and attention to detail is crucial. improvements to the templates and to guidance and staff development for writers and peer reviewers should reduce the quality assurance work required in the future. a large reusable assessment item bank 213 the file management systems and version control again have worked well but are resource intensive. changes to the file protocol for the template should reduce the administrative burden. the development of a conversion tool which produces items which will render correctly in a range of vles has been a difficult process because the vles do not interpret the qti specification in the same way. in addition there has been no commonly agreed format for item metadata. however the toia-cola assessment metadata application profile provided a mechanism for storing the metadata required by the cola project in a standards-compliant format readable by a number of existing repositories. for the development of future assessments, cola is considering a web-based system which would remove some of the logistical problems currently being faced such as authors having to give items and assessments the correct name and having to zip up assessments and email them for validation and central collation. it would help considerably with version control of the templates, the conversion tool and the content itself and remove the dependence of the conversion tool on a particular version of word. a web-based system could act as a repository for the assessments themselves and a place where college staff could try out the assessments before loading them into a vle. it would allow instant viewing of the items and assessments by the authors themselves and provide much better control over the various administrative processes. it could also allow automated notification to validators by email when an assessment is ready to be validated. the next stage of the project will be to evaluate how easy colleges find it to import the cola items to their vles and to analyse the uptake of the materials by staff and students nationally. references cetis assessment special interest group. available online at: assessment.cetis.ac.uk coleg. available online at: www.coleg.org.uk e3an project. available online at: www.e3an.ac.uk helm project. available online at: www.lboro.ac.uk/research/helm/ ieee (2002) standard for learning object metadata, ieee 1484, 12.1-2002. ims (2002) question and test interoperability specification v1.2, ims global learning consortium. available online at: www.imsglobal.org mccallon, e. l. & schumacker, r. e. (2002) developing and maintaining an item bank, elm metrics inc. available online at: www.elmmetrics.com mcalpine, m. (2002) design requirements of a databank, caa centre. available online at: www.caacentre.ac.uk sclater, n (2003) toia-cola metadata application profile v1.2. available online at: www.cetis.ac.uk/profiles/uklomcore/ sqa (2003) guidelines on online assessment for further education. available online at: www.sqa.org.uk ward, a. w. & murray-ward, m. (1994) guidelines for the development of item banks, educational measurement: issues and practice (national council on measurement in education). reviews edited by philip barker p. race, 500 tips on group learning, london: kogan page, 2000. isbn: 0-7494-2884-8. softback, vii + 135 pages, £15.99. this is another in the long list of '500 tips' series, published by kogan page. phil race, the author, is a well-known name in learning and teaching and has published extensively. the presentation of advice in this way is clearly a well-tested formula and, judging by the copies from the series in my own institution's library, these are frequently borrowed books, but i wonder whether this format is now a little overused? advice from a guru can be very valuable, but in an era of evidence-based practice it is even better if the advice is backed up with references to facilitate the reader's further research in the area. unfortunately in this book references are few and far between. there is a short list of suggested further reading but, for example, in the section on being a group member, where the work of belbin is referred to, this is not referenced and belbin does not appear in the subsequent reading list. there are six chapters in the book. each chapter starts with a general introduction and then gives a list of tips together with an explanation for each one. the first three chapters, 'learning with others', 'getting groups going', and 'particular group learning contexts', give very general advice covering what group work means, how to prepare learners for group work and different sorts of groups. section 19 in chapter 3 purports to cover tutorials for adult distance learners, but only suggestions for tutorials when such distance learners attend the institution are given. strategies for holding successful virtual tutorials, such as computer conferences or web-based chat rooms, which are increasingly used in many programmes and courses, are not mentioned. although this might have been beyond the scope of this book, an acknowledgement of this sort of development, with indications for further reading could have been helpful. on the whole the tips given in the first three chapters are entirely common sense who could argue with, for example, in the section entitled 'group learning means learning by doing' that 'listening to each other has a lot to do with effective group learning'? the majority of tips are at this general level and not much new information is presented. the blurb on the outside back cover of the book states: 'packed with expert advice guidance and support, this inspirational book will provide educators and trainers with the tools needed to move from instruction to facilitation.' i am sorry to say that i did not find much inspiration in these three chapters. chapter 4 explores exercises and processes for groups and finishes by-giving a case study. here the tips were more focused and the case study one of the most useful parts of the whole book. the author presents a case study he has used when running training workshops on chemical hazards. this might sound a somewhat esoteric topic, but the format of the workshop could 75 reviews easily be translated to good effect into many other topic areas. chapter 5 considers groups behaving badly and gives tips for dealing with, among others, disruptive behaviour, the know-all, the dominator, the non-contributor and the group member who does not listen. there is also consideration of different behaviours of the group facilitator that can damage group work and this is also a relevant and helpful section. it is probably salutary for most of us to consider the sections on facilitators who talk too much and intimidate learners and those who favour clones of themselves! a section on institutional factors that can damage group work, including lack of provision of suitable spaces and timetable systems that militate against group learning will be familiar to most readers, i suspect. many of us might well have tried solutions such as 'use coffee bars' or 'take the group to the nearest pub' and had an interesting time, but found it harder to fulfil our anticipated outcomes for those particular group sessions! the suggestion to do a room usage audit to identify rooms which are regularly underused might evoke an ironic 'huh' who has time for that sort of task in our increasingly loaded timetables! chapter 6 is devoted to how group learning might be assessed. there are some helpful suggestions here that cover situations where not all members of a group might have contributed equally to a group assessment. advantages and disadvantages for each method are given which makes this a very useful chapter. the way in which the 500 tips are presented is somewhat confusing; it took me some time to work out that the sections in each chapter are numbered sequentially and cumulatively, but the actual tips in each section always start from one. this would make it harder to direct someone to a particularly useful tip. at the risk of seeming rather a sad person, i have to confess that i attempted to count the total number of tips presented and could find only 473 were we short-changed here? overall i am not sure that this topic area lends itself so readily to the '500 tips' approach. issues around group and collaborative learning merit a deeper and more considered presentation. a format that uses numbered statements to present suggestions and advice precludes the introduction of critical discussion and reasoning. 76 who is the book for? if it is aimed at trainee teachers or new entrants to higher education, then i am sure they will find much to help them. however, for the more experienced facilitators i do not think they will find this book offers very much that they do not already know. gillian jordan university of greenwich david kember, action learning and action research: improving the quality of teaching and learning, london: kogan page, 2000. isbn: 07494-3113-x. softback, ix + 243 pages, £19.99. several large-scale programmes to enhance the quality of teaching and learning in higher education have featured within the uk, for example tltp and fdtl. this book describes a similar programme, the action learning project (alp), which took place in hong kong between 1994 and 1998 with funding from the hong kong university grants commission. from its inception, the project was set within a framework of using action learning and action research to demonstrate that, as a quality assurance mechanism, this approach could be energizing for those involved and could lead to enhanced quality in teaching and learning in a cost-effective manner. the main aim of the book is to contribute to the quality assurance and enhancement literature rather than the research methodology literature, so it is not for those looking for a detailed, theoretical review of action research methodology. it will be of interest to those responsible for introducing or managing institution-wide teaching and learning schemes, and who are looking for an effective way of engaging participants and evaluating the outcomes within a research framework. the book is divided into five sections. part a sets the work within a framework of approaches to quality assurance; part b describes the projects and participants; part c describes the experiences of the participants, including their perceptions of the action research framework; part d describes the support provided for the projects; and part e provides concluding remarks on the effectiveness of the action research approach. the structure of the book reflects the cyclical stages of the action research approach. it also contains three appendices, a reference list and an index. in part a, the first three chapters set out the framework within which action research may be considered as a scheme for assuring the quality of a/t-j volume 9 number 3 teaching and learning. in chapter 1 there is an introduction to higher education quality schemes in general. a distinction is made between quality assurance and quality enhancement schemes and these are mapped against three research paradigms, positivist, interpretive and critical epistcmologies. the case is made for locating the project-based method of quality enhancement within the critical paradigm, which includes action research. chapter 2 continues to explore the rationale for considering quality enhancement schemes within an action research and action learning framework. the three key characteristics of action research are defined through a reference from carr and kemmis. it is concerned with improving social practice, there is a systematic, cyclical process of enquiry, and it is participative. action research is then compared with action learning, drawing on a definition from mcgill and beaty for the latter. action learning, rather than action research, was the preferred term for the project name because the funding body specifically excluded research. the chapter ends pragmatically by suggesting that the two terms will be used interchangeably throughout the rest of the book. chapter 3 provides more detail about methods of observation and evaluation that can be used by project teams to monitor and report on their work. overall, the treatment of so many complex research concepts in this short section is inevitably generalized but there are references to other more specific sources. part b contains chapters 4 to 7 and through describing the context of the alp, it addresses the steps necessary for implementing and managing a large project which itself is composed of fifty smaller sub-projects, spread across seven universities. chapter 4 discusses the ways in which academics can be persuaded to participate in projects, and chapter 5 describes the project management structure that had to involve all seven universities. chapter 6 gives an indication of the range of projects funded and chapter 7 outlines the methods used to disseminate the project outcomes, including a web site, http:llalp.polyu.edu.hk. in parts c and d, the author's associates contribute to chapters 8 to 15 and describe the perceptions and experiences of the participants. they draw on material from their own evaluation of the project. it is in these two parts of the book that the key characteristics of action research and action learning are most clearly illustrated in practice. chapter 8 focuses on the evaluation design and methods of data collection. chapter 9 considers the reasons why participants became involved in the project, the main one being a desire to improve the quality of their teaching and their students' learning. chapter 10 focuses on the importance of effective teamwork in the projects and highlights the collaborative nature of action research. chapter 11 describes how the project teams adopted the action research cycle. chapter 12 reports the participants' views on the outcomes of their projects. in part d, chapters 13 to 15 return to the organization and management of the project and focus on the various ways in which support was provided to the project teams. this involved a small co-ordinating team of facilitators, each of whom supported ten to twelve projects. they aimed to do this in conjunction with the internal educational development units of the universities. the nature of the support varied, but much of it centred on support for research design, methodology, evaluation techniques and writing up research. the facilitators found themselves to be most comfortable in the role of 'critical friend' to the project teams and chapter 14 describes in some detail the ways in which this role developed. in the final two chapters in part e, the author returns to the issue of quality assurance as he reflects on the overall outcomes of the alp. chapter 16 reflects on the high cost of ensuring quality in higher education and suggests that promoting quality through action learning projects is cost-effective. chapter 17 summarizes the overall outcomes of the project and concludes that action learning is an effective method of quality enhancement. in conclusion, those looking for an in-depth understanding of the philosophy and methodology of action research should consult other sources, for example, greenwood and levin (1998), and the quality of the book would have been increased if clear distinctions had been maintained throughout between action research and action learning, rather than using them interchangeably. nevertheless, the book is an easy read, and should be of interest to those involved in similar projects, both at the institutional as well as the national level. the frameworks and methods used will also be useful for individuals who wish to use action research and action learning as a framework for evaluating their own teaching activities. janet hanson bournemouth university 77 reviews references carr, w. and kemmis, s. (1986), becoming critical: education, knowledge and action research, brighton: falmer press. mcgill, i. and beaty, l. (1995), action learning: a guide for professional management and educational development, 2nd edn, london: kogan page. greenwood, d. and levin, m. (1998), introduction to action research: social research for social change, thousand oaks, ca: sage. e. parsloe and m. wray, coaching and mentoring practical methods to improve learning, london: kogan page, 2000. isbn: 07494-3118-0. softback, xiii +193 pages, £16.99. during the last decade the increasing pace of developments in industry and commerce, stimulated by technology, have resulted in significant changes in management and working practices. there is a realization that prosperity and economic competitiveness require an educated and trained workforce, and this is influencing education, both in its content and methods, with a widening access under a vision of lifelong learning. personal development of students in skills of communication, study methods, creativity, problem-solving and collaborative methods of learning, are becoming more prominent, and institutions are having to provide greater support for staff and students to achieve these wider objectives. it is here that coaching and mentoring are important, and this publication with its emphasis "on practical methods is both useful and timely. the orientation of the book is towards the workplace and community centres, but there is much that is relevant to educational institutions. with its direct approach and economical style, clear sectioning, summary points, checklists and charts, this book provides a useful and practical introduction to mentoring and coaching that is directed to improving learning. the first chapter sets out the development of coaching/mentoring that moved it from a marginal activity to a recognized component of training. although the links to sports coaching, workplace counselling, and professional training, are sketchy in their analyses within a relatively short chapter, they do illustrate the complexities in defining and circumscribing coaching/mentoring. a recurring comment is that these techniques are multi-faceted and have to adjust to differing institutional and cultural 78 environments. both coaching and mentoring are directed at individual group members; coaching typically addresses performance goals, whereas mentoring takes a wider view of personal development and growth. the chapter ends with a useful set of tables distinguishing their purposes and orientations. the second chapter, noting the global advances in information technology, sets out the concept of a learning organization in which coaching and mentoring are key components. the psychology on learning is selective and superficial, making few references to the research literature, but the authors make distinctions between the learning experience and reflection. these are loosely linked, without critical discussion, to learning styles and to development as 'the learning of new knowledge, understanding, skills and behaviours'. the authors' style is prescriptive but makes useful points on the perceptions and implementation of learning opportunities, and the motivation that is needed to exploit them. having outlined this view of the learning/ development process the next three chapters discuss the main theme of the book that the aim of coaching/mentoring is 'helping people learn how to learn'. coaching is conceived as a process of analysis, agreeing a personal development plan with the learner, then implementing and evaluating that plan. under this framework interesting comments are made on coaching style, that is, 'the manner in which coaching is performed' and which has to adapt to the experience and performance of the learner. examples are taken from the workplace with a useful discussion on coaching inexperienced learners using a 'practice spiral' and a 'skills framework' technique. these comments reflect the experience of the authors and are practically orientated. for the more experienced learners, the grow technique (establish goals, examine reality, consider options, and confirm the will to act) is outlined and the chapter ends with a selfassessment exercise to 'establish your current levels of competence as a coach'. in discussing mentoring, the authors point out that this one-to-one activity is employed in many different contexts, all with their varied demands and constraints. in consequence, an unambiguous definition of the mentoring process is hard to achieve, but four key stages are discussed which illustrate the principal features of mentoring, namely confirming the personal development alt-j volume 9 number 3 plan, encouraging self-management of learning, providing support and assisting in the evaluation of success. these stages are discussed using four examples to show the wide variations that exist in the theory and practice of mentoring. these examples are taken from business, institutes, social services and manufacturing, but all highlight the importance of preparation and training for mentors, agreeing a format for meetings, monitoring progress and evaluation. again the chapter ends with a self-assessment exercise. the third chapter in this section of the book addresses community mentoring. much practical illustration and discussion are provided in contexts that can require mentors to be guidance providers, pseudo-parents, caseworkers, as well as learning facilitators. the argument moves to the professional development of what are termed 'coach-mentors', notes recent government initiatives, but points to the care which will have to be exercised in designing training schemes which need to adapt to varying contexts. overall the authors provide a perceptive discussion of the differing requirements of community mentors and the need for empathy, and establishing rapport and a sense of trust with the person who is being mentored. in summary, readers wishing to know more of coaching/mentoring will find interesting topics for discussion, practical work schemes and checklists, and much sensible advice all written in a direct and pointed style. roger hartley university of leeds s. ryan, b. scott, h. freeman and d. patel, the virtual university the internet and resourcebased learning, london: kogan page, 2000. isbn: 0-7494-2508-3. softback, xiii + 204 pages, £19.99. in recent years there has been much talk about 'virtual organizations'; that is, the ability to create an entity that does not physically exist other than in cyberspace and in people's minds. virtual offices and virtual businesses, for example, have 'existed' for some considerable time. now we are seeing the application of the 'vi duality' paradigm within educational organizations. we are therefore now hearing much talk about virtual classrooms, virtual communities of learners, virtual learning environments and virtual university systems. this book deals with the use of the internet and resourcebased learning (rbl) in order to realize the electronic delivery of courses. naturally, the ultimate aim of this could be the realization of a virtual university facility. the material contained within the book is organized into an introduction, ten 'semiindependent' chapters, a reference section (containing the bibliographic details for both the conventional references and the on-line citations) and a glossary. i describe the book's chapters as 'semi-independent' since the authors claim that each chapter could be read (in a stand-alone fashion) independently of any of the others. there is a web site to accompany the book. this resides at http:llwestworld.dmu.ac.ukf vu-rbl and 'contains links to the web pages mentioned in the book as well as updates and new links'. this site is quite nicely designed echoing the colour scheme and design of the book's front cover. unfortunately, none of the web pages contain any dates and so it is not possible to see when, if at all, the site has been updated. the opening chapter of the book 'sets the scene' for the nine chapters that follow. it does this by describing many of the current developments that are taking place in higher education (he) and the impact that the internet and the web are having. the primary focus of the chapter is on developments that are taking place in the united kingdom, although some examples from other countries (such as america and australia) are also included. a fundamental premiss made by the authors (p. 2) is that 'rbl can only be done w e l l . . . if supported by appropriate models of learning and teaching and principles of course design'. this requirement is 'followed up' in chapter 2, which deals with the various theories of teaching and learning that are 'particularly relevant for understanding how to do rbl well' (p. 5). following the somewhat brief descriptions of learning theories (such as constructivism and conversationalism), types of learner and approaches to learning, the chapter concludes by presenting an outline framework for course design. the real 'meat' of the book starts in chapters 3, 4 and 5. the first of these examines the nature of the internet and focuses on a number of issues relevant to the use of this medium for teaching and learning. although some interesting concepts and examples are described, from a technical perspective the treatment presented in this chapter (and some of the ones that follow) is very superficial but then, i suppose the book is 79 reviews not really intended for 'technocrats'. chapter 4 builds on the material that was presented in chapters 2 and 3; it deals with using the internet and the web for course development and delivery. there are three main themes in this chapter, developing an online course (for which the authors advocate a team approach); tools for planning structuring and development; and delivery issues. as this chapter is quite a short one, the three topics (particularly, the last one) are only treated in a superficial way. chapter 5 continues the trilogy with a discussion of 'resources on the world wide web'. the discourse in this chapter covers a variety of different topics such as digital/on-line libraries, electronic publishing, discussion groups, newsgroups, museums and on-line laboratories. there is a 'page or two' on evaluation but the treatment given lacks depth and rigour. unfortunately, despite the many different types of resource that are described, very little is said in this chapter about the pedagogic aspects of using these resources to support teaching and learning. chapters 6 and 7 are probably two of the most interesting ones in the book. they deal with computer-mediated communication (cmc) for collaborative learning and computer-assisted assessment, respectively. the cmc chapter covers a wide range of techniques, such as realtime chat, muds and moos, email, discussion lists, newsgroups (again!) and conferencing. several types of conferencing technology are discussed. the discussion starts off with computer conferencing (using softarc's firstclass system as an example) and then goes on to consider a number of audio, video and multimedia conferencing systems. the chapter on assessment covers a wide range of techniques to facilitate both the automated and the semiautomated measurement of students' skills, knowledge and on-task performances. both formative and summative approaches to assessment are considered. in many ways, the 'climax' of the book comes in chapter 8. this is entitled 'integrated systems'. the overall theme of this chapter is the integration of electronic resources with a view to creating virtual learning environments. the three main topics that are addressed are: integrated course delivery systems (such as top class and webct); international standards to facilitate the exchange and integration of teaching and learning resources; and the integration and reuse of 'educational objects'. on the 'standards' theme, one of the interesting pieces of work 80 described in this chapter is the instructional management systems project. this uses metadata techniques in order to realize the exchange and inter-operability of pedagogic resources. some mention is also made of the european community's ariadne project which is also concerned with the reuse (and creation) of computer-based teaching and learning materials. the final two chapters in the book each deal with various aspects of 'change'. in chapter 9, the authors discuss 'changing cultures in he'. essentially, this chapter considers how conventional he organizations (and their staff) may need to adapt in order to accommodate ongoing developments in rbl and the competition that they might face from 'real' virtual universities operating in a global, world-wide context. chapter 10, similarly, looks at 'what's around the corner?'. two potentially important issues that the authors identify include technological developments and the role of he organizations in the information age in which we now exist. my own view is that no matter how much we automate our educational systems through the use of rbl, we will still need people (even though they may be virtual ones!). i waited a long time to get a copy of this book to review. having now read the book from cover to cover, i am disappointed that it did not live up to my expectations. i say this for two reasons. first, it does not really contain a great deal of information about virtual universities despite its title. second, i have the impression that the book was put together in a bit of a hurry. there are quite a few careless mistakes embedded within it and one or two factual errors too. nevertheless, despite these shortcomings, many people will undoubtedly find this book a useful source of information. however, those who already have some experience in rbl will probably find that it lacks depth in most of the areas that it attempts to cover. according to the authors (p. 3), the book is intended 'chiefly for teaching staff... who wish to be informed about the changes happening around them and to reflect on their implications critically and deeply'. bearing this in mind, the authors have done a fairly reasonable job in bringing together the diverse information 'sources that are contained within it. philip barker university of teesside editorial this issue of alt-j is dedicated to professor david squires, the previous editor of alt-j, who died tragically in august 2001. a previous editorial of this journal paid tribute to david's work and provided a review of some of his research interests (conole, 2001). the seven articles in this issue are from friends and colleagues who worked with david over the years. some worked with david for only a relatively short time; others are lifelong friends. the authors represent internationally recognized experts in their field and come from all corners of the earth. this in itself is a testament to david's significant contribution to learning technology research. the authors have submitted papers which in some way align with david's own research work or report on joint work done by the authors in partnership with him. the issue begins with a review by conole of the current status of learning technology, which benchmarks current developments against the visionary concept of a peripatetic electronic teacher outlined by squires only a few years ago (squires, 1999) and discusses the extent to which this vision has been realized. this article attempts to provide a taxonomy for learning technology and outlines current areas of research activity. the remaining articles in this issue provide an excellent sample of work across these areas and form a valuable snapshot of the current status of learning technology research. adaptive hypermedia is an area of increasing research interest and importance. the article by ng, hall et al. reports on the application and evaluation of adaptive hypermedia techniques, whilst kemp, davis et al. report on an evaluation of the shift from classroom tutor to hypertext adviser. developing engaging and interactive learning materials, particularly ones that provide interactive and authentic learning opportunities, is a major area of activity. a particular subset of this is the development and use of models and simulations in teaching and learning. riley reports on a pioneering initiative the computers in the curriculum 2 alt-j volume 10 number 3 project for which david squires was director of science in the 1980s, as well as more recent developments in systems modelling within geographic and business education. student assessment and evaluation are major areas of concern as we attempt to understand the benefits and impact of information and communication technologies (ict) in education and distil examples of good practice. macht and preece in their paper consider the issue of how accurately instructors judge students' attitudes online. finally, the remaining two papers consider aspects of associated student support mechanisms and the learning environment. mcdougal reports on the development of technology-supported environments for learning through cognitive conflict, whilst jameson reports on work that she did with david squires looking at graduate supervisors as advanced expert coaches. to conclude, we hope you feel, as we do, that this issue acts by way of a celebration of david's work to form a special tribute to him and his family. gráinne conole editor references conole, g. (2001), 'editorial a tribute to david squires', alt-j, 9 (3), 2-4. squires, d. (1999), 'peripatetic electronic teachers in higher education', alt-j, 7 (3), 52-63. 3 from classroom tutor to hypertext adviser: an evaluation michael kemp, h. c. davis, william roche and wendy hall the university of southampton email: hcd@ecs.soton.ac.uk this paper describes a three-year experiment to investigate the possibility of making economies by replacing practical laboratory sessions with courseware while attempting to ensure that the quality of the student learning experience did not suffer. pathology labs are a central component of the first-year medical undergraduate curriculum at southampton. activities in these labs had been carefully designed and they were supervised by lab demonstrators who were subject domain experts. the labs were successful in the eyes of both staff and students but were expensive to conduct, in terms of equipment and staffing. year by year evaluation of the introduction of courseware revealed that there was no measurable difference in student performance as a result of introducing the courseware, but that students were unhappy about the loss of interaction with the demonstrators. the final outcome of this experiment was a courseware replacement for six labs which included a software online hypertext adviser. the contribution of this work is that it adds to the body of empirical evidence in support of the importance of maintaining dialogue with students when introducing courseware, and it presents an example of how this interaction might be achieved in software. foreword the work in this paper was carried out by michael kemp as part of his ph.d. studies under the supervision of william roche in pathology and wendy hall in the department of electronics and computer science, and with assistance from hugh davis and the microcosm team. the ph.d. was examined in june 2000 by hugh davis (internal) and david squires (external), shortly before the latter's diagnosis of cancer. at the time david had encouraged us to publish the work in alt-j. we postponed the publication of this work so that it could appear in this memorial issue. 41 michael kemp et al from classroom tutor to hypertext adviser: an evaluation introduction in response to an initiative to improve student attendance and appreciation of pathology practicals, case-based teaching was introduced to a first-year undergraduate pathology course (mccullagh and roche, 1992). the practicals were designed as part of a total experience, to build upon material recently presented in lectures, and they typically included notes on a case study, slides and a microscope. students were required to answer a number of questions related to the case study and the practicals were followed by tutorial discussion groups in which the issues raised could be explored and reflected upon. the practicals were self-paced and the students could ask for assistance from demonstrators whenever they wished, although the majority of problems were sorted out by discussion between students. these labs were informal; there were generally around eighty students present in each lab session, and there was a background noise consistent with many students engaging in conversation. students arranged themselves informally into groups of between two and eight. these labs were perceived to be popular and effective. the problems were the difficulty in recruiting sufficient trained demonstrators and the potential for some students to take back seats. in some cases perhaps only one student would do the microscope work in spite of the possibility of using monitors. however, there was a need to economize and it was decided to conduct a two-part experiment: • to introduce some courseware to replace the practicals; • to investigate the possibility of doing without demonstrators for the courseware practicals. at the same time the pathology staff hoped that there might be educational gains from the introduction of courseware, since courseware may be used many times, allowing students the opportunity, after their tutorial discussion groups for example, to revisit the work and reconceptualize the material (mayes, 1993; mayes, 1995). in this paper we start by briefly describing the research context within which this work was conducted. we continue by describing the nature of the courseware that was designed to replace the practicals and by detailing the results of the first evaluation of this courseware, which demonstrated that student learning did not suffer from the use of the software while demonstrators were still present. encouraged by this result a second trial was conducted, this time without demonstrators present when the students used the courseware. again the results were encouraging, but the evaluation demonstrated that students had not always succeeded in getting answers to some of their questions. the final section describes a third trial in which a hypertext software adviser was used to provide students with additional help. the evaluation of this trial demonstrates that students found the adviser helpful and were more likely to accept the software and to find answers to their questions. context many evaluations of the adoption of courseware have a positive result in that they tend to demonstrate that student learning is either unaffected or improved by the intervention; for 42 alt-j volume 10 number 3 example, see the 'the no significant difference phenomenon' (russell, 1999; johnson, aragon, shaik and palma-rivas, 2000). however, there have been some more recent ethnographic studies (such as hara and kling, 1999) pointing out that students are not always pleased with the learning experience, and may be frustrated by the environment and their inability to talk with someone to solve their problems. in the more extreme cases they see such methods as attempts by universities to absolve themselves of their teaching duties (noble, 1998). as a result recent research has examined ways in which communication with and between students can be maintained within an online environment (wegerif, 1998; arvan, ory, bullock, burnaska and hanson, 1998). this work adds further evidence to the debate, and introduces a software adviser agent as part of the solution to the problem of maintaining dialogue. a particular feature of courseware that this work addresses is that of replacing practical laboratories with virtual practicals. there is a body of work in this area which aims at producing virtual or remote practicals (for example, colwell and scanlon, 2001) in order to make the experience available to distance students, to share expensive experiments more widely or to make dangerous experiments safer. the focus of this work was concerned with making economies in terms of laboratory equipment, laboratory space and demonstrator time. the scalpel courseware the general principle in building the scalpel (southampton computer assisted learning pathology education laboratory) courseware was to design an environment to replicate the same six case studies previously delivered in the traditional laboratory. the design constitutes a main window containing these case studies. hyperlinks were authored into the text to provide access to the same supplementary material previously provided in the lab, that is, pictures, videos and images of microscope work. additional online background material could easily be referenced and searched when required. a multiple choice question (mcq) engine was also integrated with the courseware in order to allow the students immediate feedback on whether they were getting the correct answers to the questions in the case study. the courseware was implemented using microcosm (hall, davis and hutchings, 1996). the rationale for this decision was mainly due to microcosm's facilities for integrating with third-party applications; in this experiment two-way integration with the mcq engine (provided by the stomp tltp project) was an important feature, and in the case where a student asked for help from background materials the engine had to be able to follow links back to content. integration with the media viewer was also important. two other features of microcosm that were useful were generic links (fountain, hall, heath and davis, 1990) which allowed the rapid authoring of links from all occurrences of keywords and phrases to appropriate materials, and the computer-link facility which used advanced text search features dynamically to locate suitable materials for the user. the initial design of scalpel was in 1996; at this stage the web was still in its infancy and browsers had support for little more than rendering of html. the web was seen as encouraging a didactic view of learning, rather than the student-controlled exploratory style (crook and webster, 1997) that was needed for these case studies. 43 michael kemp et al from classroom tutor to hypertext adviser: an evaluation ' --«i " ' . > » / . * • • • <! ? r r ^ a figure /: 7he scalpel interface figure 1 shows a screenshot of the scalpel interface. the case study notes are on the right. the user has followed a link to see two histology slides, and the mcq engine is ready for the student's answer to the question in the case study. it would be possible to create a fairly similar learning experience using any of the now widely available virtual learning environments (vles), but the choice of environment was fortuitous in making the courseware simple to deliver. evaluation methodology the evaluations were undertaken in three consecutive years. in each year there were around 160 first-year undergraduate medical students. the unchanged selection procedures and the collection of demographic data gave us confidence that from year to year these students formed a very similar selection of the population, and that it was therefore possible to generalize results from year to year. it is probably reasonable further to generalize the results for the whole body of undergraduate medical students but wider generalization to the population as a whole should only be done with caution. quantitative data were collected using questionnaires. these questionnaires asked for answers to factual and non-factual attitudinal information. the latter answers were collected on a five-point likert scale (strongly agree, agree, neutral, disagree, strongly disagree). the sense of the wording of some indicators was reversed to prevent the results being affected by, for example, students agreeing with everything. complex attitudes were measured using a number of facets of that attitude, and then those facets were averaged. questionnaires were pre-trialled with colleagues to highlight any problems before they were presented to students. all questionnaires were anonymous. in the first trial the practical sessions were timetabled, and questionnaires were distributed at the halfway point and at the end of the final practical, so that return rates were 100 per cent (n=157). in the second trial students completed the practicals in their own time, and were requested to return questionnaires when they had finished. this led to a much lower response rate (n=97/160). in the final trial, in spite of organizing a sweepstake offering two 44 an-) volume 10 number 3 £50 prizes (students could cut a corner off their questionnaire, write their name on it and add it to the sweepstake, thus maintaining anonymity), the return rate fell even further (n= 74/165). in addition to the quantitative data collection, focus groups were held and students were observed using the system. no attempt was made to quantify the data collected, but rather the results were used to add substance to and to help interpret the results gained from analysis of the questionnaires. those who wish to see greater depth of methodology are referred to michael kemp's ph.d. thesis (kemp, 2000) to see the hypotheses tested, the statistical tests applied and the full experimental results. online labs: the first evaluation the main purpose of the first evaluation was to discover whether there was any measurable change in the effectiveness of learning when lab practicals were replaced with scalpel courseware. clearly an important facet of the introduction of any new teaching method is to convince staff that students would not fare worse as the result of the change. as a subagenda we chose to investigate whether there were any groups of students who fared better or worse as the result of the change, where groups were defined by such concepts as demographics, preferred learning style and previous exposure to computers. for this part of the study twenty-three hypotheses were tested: the fundamental three were about pedagogic value. 1. the educational attainment of pathology on average differs between medical students who work through practicals in the traditional laboratory and those who use scalpel. 2. medical students find scalpel an acceptable pedagogic environment. 3. medical students' acceptance of scalpel influences educational attainment. hypotheses 4-9 looked at what aspects of students (demographics, learning styles, attitudes, and so on) influenced the pedagogic value of scalpel. hypotheses 10-13 looked at what student characteristics affected their need for demonstrators. hypotheses 14-18 looked at what characteristics might affect their attitude to computers and hypotheses 19-23 looked at how student characteristics affected learning style. for the purpose of this study the students were split into two groups. group a did the first three practicals using the traditional lab and the next three practicals using scalpel, whereas group b did the first three practicals using scalpel and the next three in the traditional lab. demonstrators were present in the lab and in the computer rooms where the scalpel practicals were held in the normal timetabled slots. educational attainment (ability to recall and apply subject domain knowledge) was measured by examination using mcq (one mark for right answer, minus one for wrong answer), and short-answer tests. these examinations were taken by all 159 students after the first .three practicals and after all practicals were completed. questionnaires were administered before the start of the first practical, at the end of their final practical using scalpel and after all practicals were completed. 45 michael kemp et at from classroom tutor to hypertext adviser: an evaluation the important outcomes of this evaluation were that there was no relationship between educational attainment as measured and the method used for study. both groups of students scored similar marks at the crossover and at the end, and the marks at the end were an improvement on the marks at the crossover point. acceptance of scalpel was distributed evenly around 'neutral' and no significant relationship was found between educational attainment and acceptance of scalpel. other important results were that the students expressed a strong preference for maintaining the presence of demonstrators in the labs when using scalpel, and that there was a weak negative relationship between those that expressed a need for demonstrators and expressed acceptance of scalpel (that is, those who did not enjoy scalpel were more inclined to need demonstrators). these results were further emphasized by the focus groups. no significant relationships were discovered between students grouped by demographics, learning styles or previous exposure to computers, and their acceptance of scalpel or their educational attainment. observations indicated that the dynamics of the lab sessions changed with the introduction of scalpel. the labs were quiet and the students worked mostly alone, although small ad-hoc groups of two or three students might form to discuss a problem occasionally. demonstrators were observed moving around the lab to deal with individual queries rather than groups, and as a consequence often answered the same question many times. few problems were observed with the software although in their first session it was noticed that some students took a while to learn the importance of closing a window after finishing with its contents. doing away with demonstrators: the second evaluation in spite of the strength of feeling expressed by the students in the first trial concerning the need for demonstrator assistance, the team were encouraged by the results showing no change in educational attainment, and decided to continue the experiment by using scalpel without demonstrators. for the second trial, all students were asked to complete the six case studies using scalpel. the work was self-paced, meaning they could do it when they liked as no demonstrators would be present in the computer rooms anyway, but tutorials would still be run at the set times and would cover the points that were raised from the last practical that they were expected to have completed. the important feature of this evaluation was the attempt to discover the methods that students used to solve problems they encountered during practicals. they were asked to note the number of times they would have liked to have consulted a demonstrator, why they wanted to consult a demonstrator, and how they eventually answered their question (whether they used the text book, lecture notes, resources within scalpel, asked a student, asked a member of staff, asked in tutorial, or whether they never did answer their question). figure 4 below shows the variation of expressed need for demonstrators during scalpel sessions, and it is clear that year by year the students showed less need for demonstrators, 46 alt-j volume 10 number 3 in spite of the fact, that in year 2 at least, nothing had been done to compensate for their absence. this effect is discussed more fully below. it was clear that the reasons for wishing to consult a demonstrator were nearly all concerned with the pathology content of the practicals and very rarely to do with procedural issues or operation of the software. figure 2 shows the methods students used to solve their problems. text books, tutorial dialogue and peer dialogue are clearly important. staff were rarely consulted, and the use of the scalpel courseware materials is small, indicating that students did not think it likely that answers to their questions would be discovered online. 60 i 50 5 40 | "s 30 20 10 i \ • 1 / \ : '/ v pathology histology aims and objectives mcq engine scalpel firebonality figure 2: methods used by students to solve problems in the absence of demonstrators students were also asked to suggest how scalpel might be improved by ranking the following and by making further suggestions: • glossary of terms; • online text books; • ability to ask questions online and get answers; • on line notepad; • more links to background material; • optional extra practicals; • more animations; • sound clips to introduce sections, such as patient histories. students were clear that their preferred improvements to the scalpel courseware would involve putting text books online, providing rich linking to the new online materials and the ability to ask questions online. these results were confirmed in focus groups; students 47 michael kemp et al from classroom tutor t o hypertext adviser: an evaluation showed little interest in the introduction of multimedia gizmos, in spite of the fact that a few high-quality examples had been created to demonstrate the sort of material that was envisaged. introducing a hypertext adviser: the third evaluation the importance of dialogue with peers and teachers is well documented (see, for example, laurillard, 2001; schank and clearly, 1995; mayes, 1995). in the earlier versions of scalpel the only way that the system entered into any dialogue with the student was via the mcq engine, which asked the student questions and gave them some feedback on their answers. clearly this is limited by the fact that the subject of the dialogue is decided by the system and there is no opportunity for the students to ask their own questions. the fact that students were allowed to carry out the scalpel practicals at the time of their choice ruled out the use of synchronous communication methods for online dialogue, as there were unlikely to be any staff or many students online at a convenient moment. on the other hand, there has been much work recently on the part that asynchronous communication can play in providing tutorial support and dialogue (such as masterton, 1998; stratfold, 1998). many of these approaches are based on work on 'answer gardens' (ackerman and malone, 1990). in this approach students pose questions, and the answers to these questions are collected in a kind of extended faq, organized by topic so that students can locate areas of similar questions to their own. as more questions are posed and answered, the answer garden grows. the final experiment set out to discover whether the introduction of a richer environment supported by an online 'demonstrator agent' would improve attitudes to scalpel. in this experiment all 165 students completed the same six practicals, but an experimental group of 40 volunteers were given additional access to an enhanced mcq engine which provided access to the demonstrator agent. of these, 28 students from the experimental group and 46 from the control group returned questionnaires. figure 3 shows a typical screen shot of the demonstrator agent in use. the student wishes to answer question 2, but has a question. the demonstrator agent is aware of the context (the question the student is trying to answer) and is therefore able to list all other questions that have previously been asked in this context (as well as offering the student the opportunity to consult other contexts). in the example the student has selected to see the answer to one of these questions. alternatively the student could have submitted a new query. this would have been answered in due course by a domain expert, and thus added to the list of previously asked questions. in this way the answer garden grows. in the example the student has read the answer, but is perhaps not clear of the exact meaning of one of the terms that is used in the answer, so has pressed the show links button. this has sent a query to microcosm asking it to show any links it has in its database which are anchored on terms which appear in the answer to the question. the screenshot shows microcosm returning with two links to background materials (in this case a glossary of terms, but it might have been an online text book) that might be useful. the results of the evaluation were encouraging. figure 4 shows the decreasing need that students as a whole show for consulting demonstrators. this was directly linked to a year by year improvement in the acceptance of scalpel. 48 alt-j volume 10 number 3 f/gure 3: screenshot o f scalpel's demonstrator agent in use figure 4: students' expressed need for demonstrators during scalpel sessions pathology histology aims and objectives mcq engine nality it is not clear why this year on year improvement occurred. one possible interpretation is that in the late 1990s computers were becoming increasingly widespread (this was confirmed in their responses to questions about previous exposure) and in particular the web was becoming a standard source of reference. a generation of students brought up with expectations of learning from and referring to online materials is more likely to find courseware acceptable. 49 michael kemp et at from classroom tutor to hypertext adviser: an evaluation there was little difference between the experimental group and the control group in the reasons expressed for wishing to consult a demonstrator. however, the control group expressed an average need of seven demonstrator consults per student compared with an average of five per student for the experiment group. ••pathology -histology •aims and objectives •-mco engine s c a l p e l functionality figure 5: how the control group solved problems pathology histology aims and objectives mcq engine scalpel functionality figure 6: how the group with access to the demonstrator agent solved problems 50 alt-} volume 10 number 3 the pattern for solving problems changed considerably as shown in figures 5 and 6. in particular we see that the scalpel courseware becomes much more used, questions to members of staff become used a little (by submitting questions via the agent) and the number of questions never solved is considerably reduced. there was a significant improvement in attitude to the use of scalpel recorded by those who had access to the demonstrator agent. this is attributed to a reduction in the average number of required demonstrator consults relative to the control group, and an increase in problems solved as a consequence of the demonstrator agent facilitating the resolution of misconceptions with the pathology material. conclusions this series of evaluation experiments has demonstrated that there was no significant change in student learning as the result of replacing traditional labs with virtual labs, even with the earliest group who were most averse to using courseware. an unexpected result of this work was the fact that, over the three years, the first-year students were increasingly accepting of the use of courseware and found it less important to consult demonstrators. this was independent of any changes we made in the courseware. it is possible that this effect could be in some way linked to the reducing percentage of students that answered the surveys, but is more probably a reflection of the way society is changing. over the period of this study in the late 1990s pcs, and in particular the web and email, became standard tools for most educated citizens of. the uk, and as a consequence the students were more accustomed to reading, learning from and interacting with computers. in spite of the general improvement in acceptance of courseware, there was a general preference for access to demonstrators. this is not surprising; models of learning based on constructivist views (brown, metz and campione, 1996) and conversational frameworks (laurillard, 2001) would lead one to expect that removing demonstrator assistance would lead to poorer learning, as reported by our students. the encouraging result was that the acceptance of the use of courseware was significantly improved by the reintroduction of a relatively simple software agent, which provided limited dialogue via an answer garden and dynamic access to suitable in-context links. it is testament to the success of this work that all pathology practicals at the university of southampton were subsequently delivered to students through the scalpel environment. references ackerman, m. s. and malone, t. w. (1990), 'answer garden: a tool for growing organizational memory', in the proceedings of the acm conference on office information systems, new york: acm, 51-9. arvan, l., ory, j. c., bullock, c. d., burnaska, k. k. and hanson, m. (1998), 'the scale efficiency projects', jaln, 2 (2), http://www. aln. org/alnweb/journal/jaln_vol2issue2.htm. brown, a. l., metz, k. e. and campione, j. c. (1996), 'social interaction and individual 51 michael kemp et 0/ from classroom tutor to hypertext adviser: an evaluation understanding in a community of learners: the influence of piaget and vygotsky', in a. tryphon, and j. vonèche (eds.), piaget-vygotsky: the social genesis of thought, brighton and new york: psychology press, 145-70. colwell, c and scanlon, e. (2001), 'the pearl project: remote, collaborative experimentation for science students', in the proceedings of cal 2001, warwick: warwick university. crook, c. and webster, d. s. (1997), 'designing for informal undergraduate computer mediated communication', active learning, 7. davis, h. c., hutchings, g. a. and hall, w. (1993), 'a framework for delivering large-scale hypermedia learning material', in h. maurer (ed.), educational multimedia and hypermedia annual 1993, proceedings of ed-media '93, orlando, fl: aace, 115-22. fountain, a. m., hall, w., heath, i. and davis, h. c. (1990), 'microcosm: an open model for hypermedia with dynamic linking', in a. rizk, n. streitz and j. andre (eds.), hypertext: concepts, systems and applications, the proceedings of the european conference on hypertext, inria, france, november 1990, cambridge: cambridge university press. hall, w., davis, h. c. and hutchings, g. a. (1996), rethinking hypermedia: the microcosm approach, vancouver, bc: kluwer academic publishers. hara, n. and kling, r. (1999), 'students' frustrations with a web-based distance education course', first monday, 4 (12), http://www.firstmonday.dk/issueslissue4_12/hara/index.html johnson, s. d., aragon, s. r., shaik, n. and palma-rivas, n. (2000), 'comparative analysis of learner satisfaction and learning outcomes in online and face-to-face learning environments', journal of interactive learning research, 11 (1), 29-49. kemp, m. w. (2000), 'from classroom tutor to hypermedia advisor: a case study in medical education', ph.d. thesis, university of southampton. laurillard, d. (2001), rethinking university teaching: a framework for the effective use of educational technology, 2nd edn, london: routledge. masterton, s. (1998), 'the virtual participant: a tutor's assistant for electronic conferencing', in m. eisenstadt and t. vincent (eds.), the knowledge web: learning and collaborating on the net, london: kogan page, 249-65. mayes, j. t. (1993), 'commentary: impact of cognitive theory on the practice of courseware authoring', journal of computer assisted learning, 9, 222-8. mayes, j. t. (1995), 'learning technology and groundhog day', in w. strang and d. slater (eds.), hypermedia at work: practice and theory in higher education, canterbury: university of kent press. mccullagh, p. and roche, w. r. (1992), 'first year pathology practicals: the southampton experience', journal of pathology, 169 (supplement 148a). noble, d. f. (1998), 'digital diploma mills: the automation of higher education', first monday, 3 (1), http://firstmonday.org/issues/issue3_1/noble/index.html 52 alt-j volume 10 number 3 russell, t. l. (1999), the no significant difference phenomenon, new branswick: idecc, and also see the accompanying website: http://teleeducation.nb.calnosignificantdifference/ schank, r. c and clearly, c. (1995), engines for education, hillsdale, nj: lawrence erlbaum associates. stratfold, m. (1998), 'promoting learner dialogues on the web', in m. eisenstadt and t. vincent (eds.), the knowledge web: learning and collaborating on the net, london: kogan page, 119-34. wegerif, r. (1998), 'the social dimension of asynchronous learning networks', journal of asynchronous learning networks, 2 (1), http://www.aln.org/alnweb/journallvol2_issue1/wegerif.htm 53 designing intelligent computer-based simulations: a pragmatic approach bernard m. garrett* and david callear** *the school of health care, oxford brookes university **department of information systems, university of portsmouth email: bmgarrett@brookes.ac.uk this paper examines the design of intelligent multimedia simulations. a case study is presented which uses an approach based in part on intelligent tutoring system design to integrate formative assessment into the learning of clinical decision-making skills for nursing students. the approach advocated uses a modular design with an integrated intelligent agent within a multimedia simulation. the application was created using an object-orientated programming language for the multimedia interface (delphi) and a logic-based interpreted language (prolog) to create an expert assessment system. domain knowledge is also encoded in a windows help file reducing some of the complexity of the expert system. this approach offers a method for simplifying the production of an intelligent simulation system. the problems developing intelligent tutoring systems are examined and an argument is made for a practical approach to developing intelligent multimedia simulation systems. introduction there has been great interest in the potential use of multimedia computer-based learning (cbl) packages within higher education. the effectiveness of such systems, however, remains controversial. there are suggestions that such multimedia applications may hold no advantage over traditional formats (barron and atkins, 1994; ellis, 1994; laurillard, 1995; simms, 1997; leibowitz, 1999). one area where multimedia cbl may still prove its value is in the simulation of activities where experiential learning is expensive, undesirable or even dangerous. simulation the aviation industry has successfully been using computer-based simulation to train pilots for a number of years, and the military use computer-generated simulations for battlefield 5 bernard m. garrett and david calkar designing intelligent computer-based simulations: a pragmatic approach tactical training (fletcher, 1999). computer-based simulation has also proved valuable in areas where experiential learning may not be possible due to time, cost or operational constraints such as in the training of police to respond to bomb threats (chung and huda, 1999). the use of computer-based simulations as learning tools has been developed in many areas of health-care education: intensive care (henry and waltmire, 1992), child health (lauri, 1992; krawczak and bersky, 1995), physical assessment (white, 1995), and midwifery (woodsbn, 1997; lyons, miller and milton, 1998). clinical decision-making skills are another area where the value of simulation has been recognized. during the training of nurses clinical decision-making skills are generally acquired in the practice setting under the guidance of a qualified clinical mentor. the practice of structured reflection is used to evaluate student performance (benner, 1984), but considerations of public safety make this method.of acquiring skills in clinical decision-making fraught with difficulties. there is usually some consolidation of practical decision-making experience with theory taught during management studies in the final year of training. this is designed to help students develop their deductive reasoning skills. paper-based simulation exercises are often used but these are complex, unwieldy and difficult to manage for large groups of students. the use of multimedia simulations for developing clinical decision-making skills in nursing students is undoubtedly an area where a computer-based tool would be useful. at least three systems are currently in development for teaching clinical decision-making skills to nurses (hjelm-karlson and stenbeck, 1997; oliver, 1999; garrett, 1997). these focus on different specific areas where clinical decision-making is applied and demonstrate the value of computers in modelling deductive processes. the last of these systems has been developed using an integrated intelligent agent in its design. this approach was taken in order to create a rich, effective multimedia learning tool, capable of developing deductive reasoning skills by giving individualized formative advice to the students. it is suggested that students learn most effectively with simulation when there is feedback immediately following an event (johannson and wertenberger, 1996; lyons et al, 1998; laurillard, 1995, 1997). in this way it is seen as relevant and becomes integrated into the learning experience. it also gives positive reinforcement by the immediate correction of misinterpretations. lester, stelling and stone (1999) suggest that feedback should be presented during problem-solving activity in such a manner that contributes to learning effectiveness and efficiency. the intelligent agent in the system described here was developed using the basic elements from intelligent tutoring system design but without the rigid framework imposed by this architecture. intelligent tutoring systems one of the beliefs behind the development of intelligent tutoring systems (its) in the late 1970s was that computers can provide effective individualized formative assessment (sleeman and brown, 1982; elsom-cook, 1987; wenger, 1990; nwana, 1990, 1993). computer systems were designed to emulate intelligent human behaviour, assessing student performance and giving guidance, as would a tutor with expert knowledge of the subject. this type of system was a radical departure from computer-assisted instruction (cat). prior to this, cai used programmed units of learning in computer programs to 6 alt-j volume 9 number 3 produce what were in effect programmed text-books, or branching programs which allowed a degree of student interaction, but gave no individualized feedback. the its developed from research into knowledge-based systems where knowledge is represented by a defined knowledge set, and a program that operates upon this knowledgebase (bench-capon, 1990). the use of heuristic search techniques by the program is also a key clement of its design. this allows a student to see the reasoning undertaken to reach the solution, but adds to the complexity of the system, as it must demonstrate the result and the proof. knowledge-based systems and expert systems (which use a knowledge base and inference engine to represent the knowledge of an expert in a particular domain) are one branch of 'artificial intelligence' research (jackson, 1999). the underlying paradigm on which they are based is that knowledge and every aspect of learning can be precisely described in some way. intelligent tutoring systems use an expert system for the representation of an expert's knowledge. therefore an its has the distilled knowledge from the experience of people who have years of expertise in one particular domain encoded in it. the encoding of this knowledge into expert systems has proceeded on the basis that computer-based problem-solvers should be built to operate within a specific domain, that of an expert's knowledge base (jackson, 1999). in this way the breadth of knowledge can be minimized and the depth of specific knowledge maximized and encoded. how to encode this knowledge and represent it within an application was originally identified as one of the major problems in developing intelligent tutoring systems (sleeman and brown* 1982) and it remains so today. encoding expert knowledge requires significant development time and is one of the more problematic features in moving intelligent tutoring systems from the laboratory into the practical world (seidel and park, 1994). in general an its has four main elements (nwana, 1993): • an expert knowledge module (the domain knowledge); • a student model module (assessment of the individual student's abilities and performance); • a tutoring module (structuring the learning appropriately within the system); and • a user interface module (between the system and the student or teacher). an its differs from traditional cbl applications in several ways: • it can provide a clear articulation of knowledge in a limited domain; • it uses a model of student performance to drive instruction; • knowledge and inference rules are derived from the designer whilst the tutoring sequence and content is derived from student performance; • it provides diagnosis of errors rather than drill and practice; and • students can pose questions to an its. one of the first examples of an expert system was mycin, which defined many of the standard characteristics of expert system design (clancy, 1983). it used backward chaining rules. these rules tested if certain conditions were satisfied and then led to an action. 7 bernard m. garrett and david callear designing intelligent computer-based simulations: a pragmatic approach mycin attempted to match the action rather than the conditions during its operation and in this way it worked from goals to facts, eliminating the need to solve every possible outcome for a given set of rules. a depth-first (rather than breadth-first) strategy was also used to search the knowledge base. clancy used mycin as a basis for developing an its (guidon) and went to great lengths to disentangle the pedagogic aspects of the its from the expert system. clancy used the ability to question the mycin system with 'how?' and 'why?' questions as a basis for developing his its. in this way guidon could demonstrate the way in which it made its decisions. the system stimulated the development of other intelligent tutoring systems and successfully addressed many of the problems in its design. intelligent tutoring systems developed today such as anatom-tutor, an intelligent anatomy tutoring system (beaumont, 1994) and atlas-tutor, a cerebral radiography tutor (garlatti, 1998), although more refined in their implementation, exhibit many of the basic design elements of guidon. the difficulties in creating these highly complex systems and proving their value, however, goes some way to explain why they have yet to be widely adopted in mainstream education. the prototype clinical decision simulator in nurse education and training there is a need to develop effective decision-making skills in students based upon nursing knowledge, in much the same way that medical staff take diagnostic decisions based on medical knowledge. an intelligent computer-based simulation was needed to facilitate the development of those skills in a safe environment allowing the student to explore various 'what if?' decisions and obtain appropriate advice. the underlying assumptions on which the system was designed were that: • successful clinical decision-making is based upon a hypothetico-deductive process involving inductive and deductive reasoning skills (gordon, 1980; putzier, padrick, westfall and tanner, 1985; padrick, tanner, putzier and westfall, 1987; ellis, 1997); • nurses pass through stages of development to achieve safe and competent practice. these have been described by benner (1984) as novice, advanced beginner, competent, proficient and expert. to realize this, a system was developed taking some elements from its design but using a simplified structure to facilitate the rapid development of an intelligent agent. this agent examines the results of students' decision-making processes at each stage in their progression through a computer-based simulation and provides appropriate individualized advice. designing a modular system an architecture for the system was chosen to provide an intuitive multimedia interface and to fulfil the basic requirements of an its whilst being relatively quick to produce. a design using three modules was developed. this consists of a multimedia simulation interface, an expert-assessment system and a searchable nursing knowledge base (see figure 1). the student accesses the system as a familiar microsoft windows application and uses a simple multimedia interface (with a web browser-style navigation bar) to move through the application (see figure 2). the design of this multimedia module was based upon the 8 alt-j volume 9 number 3 expert assessment module (prolog) formative assessment expert rule-based system generation of tutorial advice summatjve assessment generation system generation of final tutorial advice summary multimedia nursing knowledge module dstabsse search engine knowledge base (help file) multimedia problem based learning simulation "*"(,' student^) multimedia module (delphi) figure i: the three-module design. functional need to create a realistic simulation experience and to provide a stimulating interface that promoted interaction. digitized video is used extensively to play out the scenarios for the student. adding interactive video increases the realism of the simulations and provides a challenge to the student to solve the problems. the videos were filmed using a local group of actors in a hospital setting to add realism. the video was filmed with a digital video camera and edited into avi image format to retain quality, although this has the disadvantage of producing a rather large file size. this did not present a problem, as the final medium for the application was cd-rom, giving sufficient storage volume. audio was also used extensively with music and sound clips to accompany various elements of the program. the audio was used to give audible cues to certain system events, such as receiving advice or entering a chosen simulation. the student is introduced to the clinical decision simulator (cds) through the multimedia module and selects the appropriate simulation from this interface. an example of a pathway in the simulation is outlined in figure 1. this module shows the digitized video sequences, controls all of the branching of the simulation pathways and presents questions to the student following events in the simulation. simulation branching is dependent upon the decisions made by the student in answering these questions. this module also controls the sequencing of tutorial information, sending and retrieving data from the expert assessment module as required. the student can also access the help module directly via the multimedia interface and 9 bernard m. garrett and david collear designing intelligent computer-based simulations: a pragmatic approach the student is placed in the role of a nurse taking charge of a group of junior staff for a morning shift on a medical ward. 1. the student is asked to allocate patients to appropriate nursing staff. 2. the student makes a selection and allocates the patients unfairly. 3. another senior nurse complains that the allocation is unfair: 4. advice is given stating which nurse had the unfair allocation and why this was not good practice. it also suggests the nurse examine the section on managing workload in the help file, which becomes accessible at this point the solution is not given, however 5. the student is given another opportunity to allocate patients to appropriate nursing staff. 6. the student makes a selection and allocates the patients fairly and safely. 7. the nurses go out on to the ward to see their patients 8. workmen arrive unexpectedly asking if they can have access to carry out repairs to the ward floor 9. the student fails t o make a decision in the allocated time for the question and is prompted to make a decision. 10. the student allows the workmen in. 11. a patient falls over the workmen's tools and complains. 12. the student is asked to decide which nurses are sent on a coffee break first 13. the student sends t o o many staff off the ward at once. 14. a patient develops symptoms of haemorrhage. 15. the student fails to alert the doctor 16. a doctor arrives t o undertake a procedure and is annoyed no nurses are available. 17. the doctor is annoyed no one had informed him of the patient with the symptoms of haemorrhage. 18. advice is given suggesting the student has made several unsafe decisions in not noticing the seriousness of the haemorrhage symptoms, sending too many staff off the ward at one time and allowing workmen access inappropriately. the advice explains why this is unsafe practice. it also suggests the nurse examine the sections on signs and symptoms of haemorrhage, safe skill mix of staff and multi-professional team work in the help file (which becomes accessible at this point).the solutions t o the questions are not directly given in the help file, but the students can determine the reason they have made clinical errors.the advice also notes that the student took too long to make a decision in this situation (and also notes if this has occurred in other questions they have answered). the simulation continues until the shift has been completed and after the final question summative advice is offered to the student this lists all of the formative advice generated during the simulation and also gives advice categorizing the student's performance in each situation as unsafe (novice level), poor practice (beginners' level) or competent (competency expected for a qualified nurse). the student may print off this advice. the student has scored several unsafe and poor practice remarks so the advice suggests they examine the specific sections of the help file involved. it also suggests that they discuss the advice generated with their tutor or mentor and then run the simulation again (if they had scored consistently competent practice they would have been congratulated and offered an alternative simulation t o try). figure 2: example of a simulation pathway. 10 alt-j volume 9 number 3 search for information about clinical decision-making or particular aspects of the simulations. once the student has engaged in a simulation, however, access to the help file is not available and the module commences timing the student's responses to questions posed. help becomes available at each point the simulation is interrupted for the student to receive advice. information about the student's answers and the time taken to respond is recorded by the multimedia module and is then passed to the expert assessment module which builds up a model of the student's performance as they progress. an anthropomorphic representation of a human tutor is used to deliver advice from the expert assessment module to the student. a simple animated talking figure with accompanying text was produced to achieve this representation (see figure 3). the expert assessment module generates the actual feedback given by this figure. the agent was designed to take the role of a mentor and is represented as such. this style of pedagogic agent was chosen to facilitate the learning process. the expert assessment module can be seen as analogous to a human tutor in this context. the anthropomorphic representation is useful to contextualize the feedback from the system and enhance student interaction with the system (heitala and niemirepo, 1998; lester et ah, 1999). the expert assessment module was developed to give appropriate tutorial advice for all the possible outcomes from decision-making pathways. advice is generated in this module by a backward chaining expert system. this module uses rules concerning the outcomes of decisions, ways in which decisions are made during the simulation, and their relationship with defined competent nursing practice in this context. the student cannot directly access the expert assessment module but does so via the multimedia module. the advice itself clearly identifies why it was generated and makes reference to appropriate sections in a nursing knowledge base which the student can access and examine for further detailed information on the particular issue involved. in this way, a heuristic tutorial strategy is realized. the advice is also designed to promote a hypothetico-deductive approach to solving the problems in the simulation, rather than giving the student the complete answer. rather than attempt to codify the complete domain knowledge for the subject area and then model the student's performance on predetermined performance templates (as in a traditional its design), an alternative strategy was used. the expert knowledge base was limited to the domain knowledge essential for successful navigation through the simulations. the associated relevant clinical knowledge was simply represented in the third module using text and graphical material. this third module contains a clinical nursing knowledge base and was developed in the form of a microsoft windows help file. it also contains information on using the program and links to a web site with further resources and updates. reference to appropriate parts of this module is made by the advice from the intelligent agent, or it can be directly accessed and searched by the student (see figure 3). i i bernard m. garrett and david calkar designing intelligent computer-based simulations: a pragmatic approach r j clinical decision simulator filehlelp back next h0e3 " ennt help . exit 'some advice . . . assessment number 1 you failed to check mr howe's temperature during the physical assessment. you failed to check mr howe's pulse during the physical assessment. you failed to check mr howe's blood pressure during the physical assessment. you failed to check mr howe's respirations during the physical assessment. mr howe is dyspnoeic and the posture you have positioned him in will severely aggravate this situation. you should read the dyspnoea section in the help file for further information on positioning the patient before you continue. the concentration of oxygen you have chosen is ideal to help relieve mr howe's dyspnoea without compromising his respiratory drive. humidification will help prevent a sore mouth. the analgesia you requested was appropriate. however, a continuous infusion may maintain v j figure 3: screenshot of the student feedback screen. rapid application development the system was designed to enable a relatively quick development time. the initial stage of development consisted of identifying vignettes to use in the simulations. this was eventually narrowed down to two scenarios, one based upon managing a patient with a specific clinical condition (bronchial carcinoma), and the other upon the management of. the clinical environment. these were chosen to illustrate the potential for different types of simulation to be used. two decision-tree pathways were developed, one for each of these scenarios. these were then discussed with professional nurse educators and expert clinicians to ensure that they represented realistic professional situations, and modifications were made where necessary. flow diagrams were then created to represent these scenarios and all the decision pathways mapped out. simulation pathways are relatively quick to code from branching flow diagrams, and appropriate points for student feedback were integrated into these pathways. multiple-choice questions were used following events rather than open free-text questions as although this limits the realism of the simulation, it allows simulations to be developed and coded rapidly, and provides a simple way to assess student performance. following the development of the simulation module an expert assessment module was coded using an advice knowledge base, rules and an inference engine based upon all the possible decisionmaking outcomes for the scenarios. 12 alt-j volume 9 number 3 the multimedia interface module and simulation pathways were coded using an objectoriented language (borland delphi). this was chosen for ease of construction using event-driven programming. it provided a rapid application development environment to test the system as it was coded and gave the flexibility to link with an expert system created with another language (lpa winprolog). developing an intelligent agent with an objectoriented language, whilst feasible, would be cumbersome and time-consuming, so it was developed using prolog. this language is based upon the principles of logic, and use of rules is inherent in its structure; it is therefore well suited for the task of expert system construction. lpa winprolog was chosen as this implementation of the language provides an interface to link the expert system as a dynamic library with an object-coded microsoft windows application. finally the nursing knowledge-base module was written for each of the simulations and coded using a microsoft windows help-file creation tool (helpscribble). discussion intelligent tutoring systems, whilst promising, remain time-consuming and complex to develop. at present there has been little uptake of these systems in mainstream educational practice and they tend to be restricted to experimental work. justifying the development investment required for such systems is also difficult (seidel and park, 1994). there is little doubt that they offer great potential to harness the power of computers to provide intelligent individualized assessment, but unless techniques for domain knowledge representation become more accessible to the majority of teachers then it is difficult to see this situation changing. using the approach outlined in this paper, it is possible to construct and code a relatively simple model of student behaviour within the limited domain of the simulation. the advantages of this design are that it retains many of the important elements of an its without the time required for a full its development. the main disadvantage is that the agent does not model the complete domain knowledge for the student. however, it can guide the student to find this knowledge. the design has been developed to provide personalized formative and summative advice to the student with an intelligent agent, whilst allowing reasonably rapid application development. the clinical simulations offer greater realism than paper-based exercises and the system has the advantage over a traditional multimedia system in that it offers the student individualized advice, and promotes heuristic learning rather than didactic instruction. a comparative evaluation of the system against a similar system without the intelligent agent is currently being undertaken to establish the value of this approach. the modular design presents a more accessible method to create intelligent simulations by limiting specific domain knowledge to a help system knowledge base and by separating the tutorial advice and rules into a separate expert assessment module. the system itself is adaptable and easily extended. further simulations can be developed and added into the system and the expert assessment system can easily have advice added to its database, its rules modified or additional rules created. this is offered as a pragmatic approach to creating intelligent simulations, whilst offering many of the advantages of an intelligent tutoring system. there are now third-party expert-system components available for several 13 bernard m. gorrett and david callear designing intelligent computer-based simulations: a pragmatic approach object-oriented languages, making the development of this type of intelligent system even more straightforward. the potential for future development for this approach is to provide a simulation development environment whereby less technically skilled users can model and build their own simulations in much the same way as existing multimedia design applications are used. references barron, a. e. and atkins, d. (1994), 'audio instruction in multimedia education: is textual redundancy important?', journal of educational multimedia and hypermedia, 3 (3-4), 295-306. benner, p. (1984), from novice to expert: excellence and power in clinical nursing practice, menlo park, ca: addison wesley. beaumont, i. h. (1994), 'user modelling in the interactive anatomy tutoring system anatom-tutor', user modelling and user adapted interaction, 4 (1), 21-5. chung, c. a. and huda, a. (1999), 'an interactive multimedia-training simulator for responding to bomb threats', simulation, 1, 68-77. clancy, w. (1983), 'guidon', journal of computer based instruction, 10, 8-15. ellis, d. (1994), 'barefoot multimedia, or, all is not what it seems moriarty', ifip transactions a (computer science and technology), a-59, 151-4. ellis, p. (1997), 'processes used by nurses to make decisions in the clinical practice setting', nurse education today, 17, 325-32. elsom-cook, m. (1987), 'intelligent computer aided instruction at the open university', technical report no. 63: computer assisted learning research group, milton keynes: open university press. fletcher, j. d. (1999), 'using networked simulation to assess problem solving by tactical teams', computers in human behavior, may/july, 375-402. garlatti, s. (1998), 'the use of a computerized brain atlas to support knowledge-based training in radiology', artificial intelligence in medicine, 13 (3), 181-205. garrett, b. (1997), 'integrating formative assessment into multimedia simulations for nursing', proceedings of acendio 1997: the first european conference of the association for common european nursing diagnoses, interventions and outcomes, london: royal college of nursing. gordon, m. (1980), 'diagnostic strategies in diagnostic tasks', nursing research, 29, 39-45. heitala, p. and niemirepo, t. (1998), 'the competence of learning companion agents', the international journal of artificial intelligence in education, 9, 178-92. hjelm-karlson, k. and stenbeck, h. (1997), 'a simulation that teaches clinical decision making in nursing', nursing informatics 1997: international medical informatics association, ios press, 492-5. 14 alt-j volume 9 number 3 henry, s. b. and waltmire, d. (1992), 'computerised clinical simulations: a strategy for staff development in critical care', american journal of critical care nursing, 1 (2), 99-107. jackson, p. (1999), an introduction to expert systems, london: addison wesley. johannson, s. l. and wertenberger, d. h. (1996), 'using simulation to test critical thinking skills of nursing students', nurse education today, 16, 323-7. krawczak, j. and bersky, a. k. (1995), 'the development of automated client responses to computerized clinical simulation testing', computers in nursing, 13 (6), 295-300. lauri, s. (1992), 'using a computer simulation program to assess the decision making process in child health care', computers in nursing, 10 (40), 171-7. laurillard, d. (1995), 'multimedia and the changing experience of the learner', the british journal of educational technology, sept., 179-89. laurillard, d. (1997), 'learning formal representations in multimedia', in f. marton, d. hounsell and n. f. entwistle (eds), the experience of learning, 2nd edn, edinburgh: scottish academic press, 172-83. leibowitz, j. (1999), 'web based multimedia case studies: ahead of our time?', kybernetes, 1, 211-15. lester, j. c., stelling, g. d. and stone, b. a. (1999), 'lifelike pedagogic agents for mixedinitiative problem solving in constructivist learning environments', user modelling and user-adapted interaction (netherlands), 1, 1-43. lyons, j. (1999), 'reflective education for professional practice: discovering knowledge from experience', nurse education today, 19 (1), 29-34. lyons, j., miller, m. and milton, j. (1998), 'learning with technology: use of case based physical and computer simulations in professional education', contemporary nurse: a journal for the australian nursing profession, 7 (2), 98-102. nwana, h. s. (1990), 'intelligent tutoring systems: an overview', artificial intelligence review, 4, 251-77. nwana, h. s. (1993), mathematical intelligent learning environments, london: intellect. oliver, m. (1999), 'a self paced multimedia learning environment: facilitating the transition of graduating nurses into the workplace', australian journal of advance nursing, 16 (4), 42-4. padrick, k. p., tanner, c. a., putzier, d. j. and westfall, u. e. (1987), 'hypothesis evaluation: a component of diagnostic reasoning', classification of nursing diagnosis: proceedings from the seventh conference, st louis: mosby. putzier, d. j., padrick, k. p., westfall, u. e. and tanner, c. a. (1985), 'diagnostic reasoning in critical care nursing', heart and lung, 14, 430-5. seidcl, r. j. and park, o. c. (1994), 'an historical perspective and a model for evaluation of intelligent tutoring systems', journal of educational computing research, 1, 103-28. 15 bernard m. garrett and david callear designing intelligent computer-based simulations: a pragmatic approach simms, r. (1997), 'interactivity: a forgotten art?', computers in human behavior, may, 157-91. sleeman, d. and brown, j. s. (1982), intelligent tutoring systems, london: academic press inc. wenger, e. (1990), artificial intelligence and tutoring systems: computational and cognitive approaches to the communication of knowledge, london: academic press / morgan kaufmann. white, j. e. (1995), 'using interactive video to add physical assessment data to computerbased patient simulations in nursing', computers in nursing, 5, 233-5. woodson, s. (1997), clinical simulations in maternity nursing 2. 16 can virtual seminars be used cost-effectively to enhance student learning? peter halfpenny and sonja wellings university of manchester email: p.halfpenny@man.ac.uk this paper describes a virtual seminar initiative designed to investigate the extent to which computer-mediated communication (cmc) can cost-effectively strengthen staffstudent interaction and enhance student group discussion, and thereby improve collaborative learning. after setting the scene by means of a brief review of the discursive potential of cmc, the establishment of an asynchronous bulletin board system on three modules in the department of sociology at the university of manchester using industry standard software is described. detailed time diaries kept by all staff involved revealed that organizing and running the virtual seminars were very much less time-consuming than running face-to-face seminars. however, analysis of the students' access to and mage of the virtual seminars indicates that some of them were disadvantaged by cmc and that they favoured face-to-face contact with lecturers over virtual seminars. the latter should therefore be part of a portfolio of teaching techniques rather than the sole form of collaborative learning. the conclusion is that a significant obstacle to benefiting from cmc is the further demand on staff time that results from adding virtual seminars as a supplement to existing teaching practices. even though these extra demands may be modest, effectively deploying the discursive potential of cmc to enhance student learning increases staff effort rather than reducing it, as many have hoped or promised it would. introduction with the expansion of university student numbers over the past decade, deteriorating stait-student ratios have necessitated a re-evaluation of teaching and learning practices. in general, the amount of direct contact between staff and students has diminished. lecture audiences have grown larger and there is less opportunity within or immediately after lectures for interaction between students and lecturers. seminars, classes and tutorials have also grown and they often have fifteen or more students in them, allowing on average only four minutes or fewer of active participation by each member over the course of an hour's 43 peter halfpenny and sonja wettings can virtual seminars be used cost-effectively t o enhance student learning? session. these sessions are also increasingly taught by postgraduate teaching assistants, rather than the lecturers responsible for the modules, and some have become fortnightly rather than weekly. overall, the opportunities for students to engage actively in group discussions guided directly by academic staff have decreased substantially even though this is widely recognized as one of the most effective learning experiences (johnson and johnson, 1990). a potentially cost-effective way of reintroducing staff-guided student collaborative learning is through virtual seminars discussions that take place electronically over the campus network. virtual seminars have numerous virtues: they are independent of place and time in that students can contribute whenever they wish from wherever they can access a networked computer; they permit many-to-many communication and enable collaborative learning through text-based interactions; they promote thoughtful and reflective commentary and heighten critical awareness; they improve accessibility and learner control by making it difficult for one student to dominate the discussion and easier for shy students to contribute; and they encourage student-centred teaching (davis, 1997; duffy, arnold and henderson, 1995; harasim, 1993; pincas, 1997; stainfield, 1997). these features can give computermediated communication (cmc) advantages over traditional face-to-face tutorials. as the dearing report noted, 'computer-based programmes, such as tutorials . . . can be highly interactive and provide activities that students need to develop their understanding of others' ideas and the articulation of their own' (dearing, 1997, para 8.24). the coming revolution? rapid advances in computing technology and the emergence of industry standard software over recent years has made educational applications of cmc more viable (anderson and jackson, 2000; ehrmann, 1994). students come to university ever more computer literate and with prior experience of the web. staff have increasing knowledge of information and communications technologies and universities have been putting the infrastructure and support in place. following these developments, there have been growing numbers of higher education courses that are computer-assisted. in some cases the technology is still being used as a repository, merely storing course materials, or an administrative tool for organizing teaching tasks. for example, schneider (1998) describes how he set up a 'teaching home page' at the university of texas tech to provide course information which could be more easily accessed by students and readily updated to take account of current events and student feedback. similarly, browning and williams (1995) suggest that cmc can be used to make lecture notes available to' students for extra revision or to catch up if they have missed a lecture. in other cases, the communicative potential of the technology is used to introduce an innovative learning environment centred on asynchronous collaborative activities. for example, duffy et al (1995) describe how on-line undergraduate seminars in music history at the university of glasgow took place using netsem, a widely available asynchronous conferencing system. the reasons for introducing the system were primarily pedagogical: the aim was to improve the quality of seminars by encouraging more active debate and developing critical thinking skills. students particularly valued the flexibility and 44 alt-j volume 9 number 3 convenience that the system offered (many contributions being made late at night or over the weekend) and contributions were found to be relatively lengthy and well researched. the investigators reported that the seminars provided students with a 'dynamic' and 'enjoyable' experience and the staff involved concluded that seminar contributions produced interesting, informed and engaged argument. it was the discursive potential of cmc allied with concerns about the economics of online education that prompted the initiative described below. the manchester sociology department initiative in the 1998/9 academic year, the department of sociology at the university of manchester was awarded funding by the university-based enterprise centre for learning and curriculum innovation to introduce an asynchronous bulletin board system on selected modules. the aim was to investigate the extent to which cmc could cost-effectively strengthen staff-student interaction and enhance staff-guided student group discussion, on the assumption that this would improve learning. after taking the advice of technical staff and reviewing what systems had been used previously in similar initiatives, three virtual seminars were set up on the faculty web server by an information systems development officer using the microsoft frontpage bulletin board facility. the seminars could be accessed from any internet-connected computer using standard web browsers. overall provision by the university at the time was approximately one networked computer per 12.5 students in the faculty. participants and procedure one of the three selected modules was a final-year undergraduate course in social divisions (sd), taken by over fifty students. teaching comprised a weekly one-hour lecture, following which the students were divided into four groups, each of which normally met for weekly one-hour face-to-face tutorials. after forming the tutorial groups, the members of one were told that the traditional face-to-face seminar was to be substituted in its entirety by a virtual seminar. the thirteen students in this group were given the option of switching to one of the remaining three face-to-face tutorial groups, but none did. to prevent the virtual seminar from being perceived by students as merely optional something that could be ignored they were instructed to take part in discussing each week's lecture topic online. however, participation was not mandatory and students were not individually chided if they did not join the discussion. the second module was a postgraduate class of thirty-two students taking a course in qualitative research methods (qrm). in this case, the virtual seminar acted as a supplement to the weekly two-hour teaching period in which there was a lecture followed by face-to-face discussion between students and the lecturer. the students were encouraged to use the virtual seminar to continue the discussion or raise further issues, after first introducing themselves and their research interests in the forum. finally, one virtual seminar was set up as an adjunct to a postgraduate course on research design (rd) taken by forty students. in this case, the students were informed that the seminar was available, but it was not incorporated into the teaching, which included lectures, demonstrations and class exercises in a weekly three-hour period. 45 peter halfpenny and sonja weffings can virtual seminars be used cost-effectivel)' to enhance student learning? two control groups were identified in order that a comparison could be made of the staff effort involved in running and maintaining virtual seminars with the effort involved in running face-to-face tutorials. in the case of sd, the control was one of the other three tutorial groups for that module, which met weekly for a one-hour face-to-face discussion between the students and a postgraduate tutor. in the case of qrm, the control was another postgraduate course with the same format of a weekly two-hour session that included both a lecture and a discussion period. for convenience this will be referred to as the qrm face-to-face seminar, though it was a module on conversation analysis. there was no control for the adjunct virtual seminar in rd. all the virtual seminars took place in the second semester of the 1998/9 academic year for a period of ten teaching weeks. in each case the module lecturer acted as the moderator for the virtual seminar. at the beginning of the semester, all students attended a half-hour hands-on training session on how to access their virtual seminar, how to read contributions and make their own, and how to abide by the rules of 'netiquette'. evaluation methods in order to monitor staff effort, all those involved in virtual seminar activity kept diaries in which they recorded the length of time taken on all virtual seminar-related activities. staff responsible for the control groups also kept records of the time they spent preparing for and running their face-to-face seminars. a short pre-seminar questionnaire was administered to students to assess their levels of computer literacy and to determine their access to networked computers. students' use of seminars was monitored by a count of their contributions and simple content analysis of the discussion threads. student evaluation of the virtual seminars was obtained by means of a self-completion questionnaire administered at the end of the semester. the outcome of the investigation into student evaluation of the virtual seminars is reported in more detail elsewhere (halfpenny and wellings, 2000). after briefly considering student usage, this paper focuses primarily on staff effort involved in setting up and running the virtual seminars and their cost-effectiveness in enhancing staff-student interaction. access and usage low levels of computer literacy, problems in accessing on-line learning facilities and technical difficulties experienced during use are commonly acknowledged limitations of many computer-based teaching initiatives. however, in this study, computer literacy was high: on the preliminary questionnaire, 97 per cent of the students involved reported that they used a computer for word-processing on a regular basis (that is, a few times a month or more), only 7 per cent said that they had never used the web and 15 per cent that they had never used email. the technical support made available throughout the semester was called upon only once. responses to the student evaluation questionnaire indicate that access was not a major difficulty for most participants in this study: 78 per cent of postgraduates and 80 per cent of undergraduates reported no problems, even though they relied on heavily used campus computer clusters, with only 8 per cent of undergraduates having a network connection off-campus, although this figure rose to 40 per cent for the postgraduates. usage of the virtual seminars varied, with the postgraduate qrm seminar attracting most contributions: overall, 91 per cent of the thirty-two students in the group contributed and 46 alt-] volume 9 number 3 between them they made seventy-four contributions. the sd virtual seminar attracted eighteen contributions from 54 per cent of the thirteen students in the group. the quality of usage mirrored the quantity: compared with the undergraduate sd virtual seminar, in the qrm postgraduate virtual seminar contributions were longer, discussion threads contained more contributions, and more topics were student-initiated rather than tutorinitiated. that the supplement qrm virtual seminar was both more used and more productively used than the substitute sd virtual seminar was unexpected, given that the latter group had no alternative means of accessing the module lecturer whereas the former group had a weekly face-to-face discussion period with their lecturer. this unanticipated finding confirms the result of previous studies which have shown that pedagogic conventions, including the prior learning experience and expectations of students, influences their engagement with on-line learning (laurillard, 1993; light, colbourn and light, 1997; mcateer, tolmie, duffy and corbett, 1997). in this case, the postgraduates who are more likely to take responsibility for their own learning than undergraduates responded more fully to the opportunity to participate in the virtual seminar, even though they had more opportunities for face-to-face discussion than the undergraduates. the rd adjunct virtual seminar, which unlike the other two virtual seminars was not promoted as an integral part of the teaching, was little used: only eleven contributions from 13 per cent of its forty student members. because of the low usage, and the lack of a control group, the rd seminar is not discussed further. there was a noticeable gender disparity among contributors to the qrm virtual seminar (although not in the sd and rd virtual seminars). the group as a whole comprised nineteen women (59 per cent) and thirteen men (41 per cent). of the twenty-five known contributors (four others participated anonymously), ten were female (40 per cent) and fifteen were male (60 per cent). males were also more likely to participate in the debates. of the seventy-four contributions made, fifty-six came from male participants (76 per cent). others have been alert to gender disparities in the use of cmc (for example, light et al., 1997) and it might be that females perceive the technology as more suited to 'nerd' males. the findings on access and usage reveal that some students were (at least initially) far less experienced computer users than others, a minority did report that access was a problem, nearly half the undergraduates did not actively participate in their virtual seminar, and women contributed to the postgraduate virtual discussions less than men. in other words, some students are disadvantaged by cmc and it is important to ensure that this is not to the detriment of their learning. avoiding this entails integrating virtual seminars into a portfolio of teaching techniques and not using cmc as the sole form of collaborative learning. staff effort it required remarkably little staff time to set up the virtual seminars. because lecturers had course material readily available on disk, it took the is development officer (who had previous experience of frontpage) no more than fifteen minutes per course to set up each virtual seminar and transfer course material to the site. no further maintenance of the sites was needed. the development officer also delivered the thirty-minute training sessions to eacfi of the virtual seminar groups. only one student asked for assistance subsequently, making an email enquiry concerning access. no other students or members of staff sought assistance of any kind. 47 peter halfpenny and sonja wellings can virtual seminars be used cost-effectively t o enhance student learning? seminar type sd virtual sd face-to-face qrm virtual qrm face-to-face the seminar seminar preparation and administration 5 657 30 410 running the seminar 150 565 76 1200 communicating with individual students outside the seminar academic matters 5 110 172 205** nonacademic matters 65 41 other* 10 200 10 0 total time (minutes) 170 1597 329 1815 *for virtual seminars 'other' was replying to emails; for face-to-face seminars 'other' was mainly photocopying. **for the qmr face-to-face control group, the lecturer did not distinguish between the time spent on academic and non-academic matters. table i: time (in minutes) spent on seminar activities. staff diary records reveal that the time they spent organizing and running seminars, and communicating with seminar students, was considerably less for the virtual seminars than for the control face-to-face seminars (see table 1). this is clearest in the case of sd, where the virtual seminar was a substitute for a face-to-face one. the total time spent by the virtual seminar tutor on all seminar-related activities (170 minutes) was around a tenth of the time spent by the face-to-face tutor (1,597 minutes). substituting a face-to-face seminar by a virtual one resulted in a major saving of staff time. face-to-face sd seminar preparation involved spending time planning how to use the weekly discussion periods, which was not necessary for virtual seminars. face-to-face seminar administration involved keeping a register of attendance and writing to absentees, whereas there was less check on virtual seminar 'attendance' and messages were quickly transmitted to students via the bulletin board or by email. even if the time taken to set up the sd virtual seminar (less than fifteen minutes) and the student training (thirty minutes) is included, the total time devoted to the preparation for and administration of the substitute virtual seminar (fifty minutes) was substantially less than for the control face-toface seminar (657 minutes). running the substitute sd virtual seminar took much less time (150 minutes) than being present throughout the weekly face-to-face tutorials (565 minutes), although it should be noted that the time spent running the virtual seminar might have increased had students made more use of it. time spent talking with individual students outside the sd seminar about academic and non-academic matters was much greater for the face-to-face seminar (175 minutes) than the virtual seminar (five minutes), probably because the face-to-face tutor was on hand to be approached. to a modest extent face-to-face talk was replaced by email communication between individual students and the virtual seminar tutor (ten minutes). the face-to-face sd tutor spent over three hours (200 minutes) photocopying material for the seminar, a task which the virtual seminar tutor was spared because material on disk could be posted on the bulletin board in seconds. even had the virtual seminar tutor chosen to distribute material that was initially in print form, scanning and posting it on the 48 alt-j volume 9 number 3 bulletin board would have taken much less time than making multiple photocopies of it. conversely, if the photocopying had been done by central support services and therefore removed from the time cost of the face-to-face seminar, the total time of running the substitute sd virtual seminar (170 minutes) would still have been only about an eighth of the total time taken for the face-to-face seminar (1,397 minutes). moreover, arranging for the photocopying to be done centrally would have cost the face-to-face seminar tutor at least a little time. these findings contradict the common claim (for example, by timms, 1997) that computer-based teaching and learning is more costly in time than traditional methods. one reason is probably that often in the past custom software has been developed a timeconsuming approach whereas in this study a simple proprietary package was used. also, it is important to compare like with like. in this study, the time taken to prepare module outlines and lectures, and to set and mark essays and examinations, has been omitted because these activities are common to the course regardless of the form of the seminar. however, all seminar-related activities have been included in the comparison, not just the discussion face-to-face and on-line. nevertheless, a more thorough-going costing would be possible, that included the time needed to support and maintain the university computer clusters and the campus network (as opposed, perhaps, to supporting and maintaining the tutorial rooms). similarly, money costs have not been included. in the case of qrm, the figures in table 1 need to be interpreted carefully. the time spent preparing for, administering and running the supplement virtual seminar (106 minutes) was in addition to the time spent on preparing for, administering and running the weekly two-hour teaching session that included both a lecture and a face-to-face discussion period. the latter took 1,610 minutes in the control group. this indicates that a supplement virtual seminar adds only modestly to the time involved in running a postgraduate course only an additional 7 per cent. however, again it should be noted that the time spent running the virtual seminar might have increased had students made more use of it. time spent talking with individual students outside the qmr seminar about academic and non-academic matters was approximately the same for the virtual seminar (213 minutes) as for the face-to-face seminar (205 minutes). this indicates that if the lecturers are physically present, as both were in the weekly two-hour teaching periods, then students take the opportunity to talk to them. in the case of the qmr virtual seminar, face-to-face talk was supplemented by only a modest amount of email communication between individual students and the virtual seminar tutor (ten minutes). this suggests that, as seen earlier with sd, face-to-face communication with tutors is preferred by students over email. cost the study demonstrates that, as a substitute for face-to-face seminars, virtual seminars are much less costly in academic staff time: less than three hours for a virtual seminar compared with nearly twenty-seven hours for a weekly one-hour face-to-face seminar over a ten-week period. the time savings are realized in all activities related to the seminar: preparation, administration, running the seminar and communicating with individual students outside the seminar about academic and non-academic matters. the extra cost in support staff time in setting up the virtual seminar, posting teaching materials and training students in how to use the seminar is negligible, at less than one hour. 49 peter halfpenny and sonja wellings can virtual seminars be used cost-effectively t o enhance student learning? however, there are two caveats. first, the virtual seminars in this study were relatively little used. had the discussion threads been more numerous or more extended, then the staff time in contributing to and guiding them would have increased. nevertheless, it is unlikely that running even a very active virtual seminar would absorb anything close to the time taken to run a face-to-face seminar. the highest use of a virtual seminar, in the qrm module, amounted to 74 contributions over a ten-week period, at a cost in staff time of 4.4 minutes per student contribution on average. assuming that the time per contribution remained equal to this, there would need to be 408 contributions over the semester (equivalent to 12.8 contributions per student) before the total staff time running the virtual seminar surpassed that running the face-to-face seminar. second, for a combination of technical, pedagogic and equal opportunity reasons, virtual seminars are better integrated into teaching when used as a supplement to, rather than a substitute for, face-to-face seminars. given this, the time taken in running a virtual seminar would be additional to the time given to other means of teaching they represent a further demand on academic staff. suggestions that cmc could ease the burden on staff caused by falling staff/student ratios while maintaining the quality of teaching raise false hopes. effectiveness it was not an objective of the study to test the effectiveness of virtual seminars in terms of learning outcomes. instead, the principal aim was more narrowly focused: to assess their effectiveness in giving students greater access to lecturer-guided discussion, on the assumption that this enhances collaborative learning. some of the findings are positive: over all three virtual seminars, 44 per cent of the students who responded to the evaluation questionnaires said that they enjoyed being able to make contributions in their own time, 40 per cent reported that they found the quality of contributions from others to be of a high standard, and 30 per cent said that the contributions increased their understanding of the subject. all these percentages were higher for the more actively used virtual seminars. however, only 17 per cent overall thought that the virtual seminar gave them better access to the lecturer, the majority of this small percentage being students in the sd substitute virtual seminar who had no other means of direct access to their lecturer. moreover, 83 per cent of the members of the two actively used virtual seminars. (qrm and sd) said that they would have preferred to attend a face-to-face seminar. meeting a tutor in a face-toface seminar generates additional contact too: in the case of the undergraduate sd module, over the course of the semester the face-to-face seminar tutor spent 110 minutes communicating with individual students about academic matters and sixty-five minutes about non-academic matters, whereas the virtual seminar tutor spent just five minutes on the first of these and no time on the second, and only ten minutes on email communication with individual students by way of compensation for the lack of face-to-face contact. the findings demonstrate that whatever the benefits of incorporating virtual seminars into the range of teaching methods employed on a module, students still value face-to-face discussion with their lecturers and tutors, in class and individually. however, a caveat is appropriate here too. if virtual seminars become more prevalent, students might become more favourable towards them than indicated by the results above, which are based on relatively little used virtual seminars in an environment where the students had not experienced them before. although the present evidence suggests that face-to-face contact 50 alt-j volume 9 number 3 with lecturers is still greatly favoured by students over cmc, further study will be needed as virtual seminars become more prevalent. conclusion the case with which virtual seminars can be set up and the growing use of the internet make it likely that more and more lecturers will use cmc over the coming years. this was encouraged by the dearing report, which called for higher education institutions to develop suitable c&it strategies at national and local level, and to adopt these technologies as teaching and learning resources. however, the findings from this study, together with the results of past evaluations, suggest that cmc is not a cost-effective remedy to the erosion of staff-student ratios. many commentators see the obstacles to the effective integration of cmc purely in terms of the lack of resources to provide better technological infrastructure and access, and appropriate training and support for both staff and students. others take a broader view, raising concerns that the current culture of teaching and learning within universities is a more substantial barrier to the full potential of cmc being realized than lack of resources (crook, 1997; laurillard, 1993; light et al., 1997; timms, 1997; weller, 2000). this study suggests that there is a third problem, beyond the resources for c&it and the culture of teaching and learning. the constant search for cheaper ways of delivering higher education has generated widespread enthusiasm for computer assistance for several decades. yet this study, like others before it, reveals that even a shift from using computers to store information to employing them as aids to discussion misses a key component of quality education face-to-face interaction between teachers and learners. the collaborative learning opportunities provided by classroom-based tutorials can be usefully supplemented but not adequately substituted by cmc. supplementary virtual seminars require an extra input by academic staff which, though modest, is still an increase in the demands upon their time. the third problem is, then, that computers cannot replace academic staff, and that the effective integration of cmc into teaching and learning increases staff effort rather than reduces it. acknowledgements the authors wish to thank the enterprise centre for funding; shoba arun, nick crossley, fiona devine and rod watson for participating as seminar tutors; nick gould, the faculty information systems development officer, for support; and all the students involved for their participation. references anderson, m. and jackson, d. (2000), 'computer systems for distributed and distance learning', journal of computer assisted learning, 16, 213-28. browning, p. and williams, j. (1995), 'the geology@bristol experience', active learning, 2 (july), 34-8. crook, c. (1997), 'making hypertext lecture notes more interactive: undergraduate reactions', journal of computer assisted learning, 13, 236-44. si peter halfpenny and sonjo wellings can virtual seminars be used cost-effectively to enhance student learning? davis, n. (1997), 'do electronic communications offer a new learning opportunity in education?', in b. somekh and n. davis (eds), using information effectively in teaching and learning, london: routledge. dearing, r. (1997), the report of the national committee of inquiry into higher education, london: hmso. duffy, c., arnold, s. and henderson, f. (1995), 'netsem: electrifying undergraduate seminars', active learning, 2, 52-48. ehrmann, s. (1994), 'looking backwards: us efforts to use technology to transform undergraduate education', in j. martin, j. darby and b. kjollerstrom (eds), higher education 1998 transformed by learning technology, oxford: ctiss publications. halfpenny, p. and wellings, s. (2000), using virtual seminars to enhance student learning, manchester: centre for applied social research, university of manchester. harasim, l. (1993), global networks: computers and international communication, cambridge, mass.: mit press. johnson, d. w. and johnson, r. t. (1990), 'co-operative learning and achievement', in s. sharan (ed.), cooperative learning: theory and research, new york: praeger. laurillard, d. (1993), rethinking university teaching: a framework for the effective use of educational technology, london: routledge. light, p., colbourn, c. and light, v. (1997), 'computer mediated tutorial support for conventional university courses', journal of computer assisted learning, 13, 228-35. mcateer, e., tolmie, a., duffy, c. and corbett, j. (1997), 'computer-mediated communication as a learning resource', journal of computer assisted learning, 13, 219-27. pincas, a. (1997), 'institutional design for group-based learning by computerconferencing', in c. bell, m. boroden and a. trott (eds), independent flexible learning, london: kogan page. schneider, a. (1998), 'sociology: the internet as an extended classroom', social science computer review, 16, 40-52. stainfield, j. (1997), 'using it to manage a third year module', active learning, 6, 30-4. timms, d. (1997), 'c&it and the learning society some post-dearing thoughts', socinfo journal, 3 (december), 2-11. weller, m. (2000), 'implementing a cmc tutor group for an existing distance education course', journal of computer assisted learning, 16, 178-83. 52 inclusion and online learning opportunities: designing for accessibility efaine j. pearson* and tony k o p p i * * *special needs computing research unit, university of teesside **director, educational development and technology centre (edtec), university of new south wales email: e.pearson@tees.ac.uk, tkoppi@unsw.edu.au higher education institutions worldwide are adopting flexible learning methods and online technologies which increase the potential for widening the learning community to include people for whom participation may previously have been difficult or impossible. the development of courseware that is accessible, flexible and informative can benefit not only people with special needs, but such courseware provides a better educational experience for all students. this paper discusses an evaluation of webct in practice at university of new south wales (unsw). the objective was to assess the accessibility of webct for people with disabilities sensory, cognitive and physical. the guidelines provided by the world wide web consortium (w3c) (web content accessibility guidelines 1.0, http://www. w3.org/tr/wai-webcontent), checking mechanisms including the bobby web page analysis tool, developed by cast (http:llcast.org/bobbyl), and learner centred design methods provided the basis for the evaluation. the approach involved extensive evaluation of existing courses by the researcher, discussion with and survey of webct designers, and the involvement of students with disabilities through interview and focus groups. issues considered include: the provision of alternative information; structure, presentation and organization of content and navigational elements; and accessibility with assistive technologies. the results reveal that many barriers to accessibility can be overcome through awareness of design issues, and that there is a need for practical advice for academics. the aim is to present a case study of best practice in the design of accessible courseware incorporating design strategies that will enable students with various disabilities to take advantage of the 'just in time, just for me' approach. 17 elaine j. pearson andtony koppi inclusion and online opportunities: designing for accessibility introduction developments in online learning can present opportunities for a more inclusive education for students with disabilities. online learning facilitates real flexibility in the time and place for learning; independence is increased through the availability of online learning resources, and there is less reliance on the assistance of helpers such as library staff or note-takers. the ability to submit work electronically can release the students from the costs in time and resources in obtaining physical access. communication can be improved computer technology can give voice to those without, both literally and figuratively. communication devices such as chat rooms, discussion groups and email can enable students to participate without prejudice or embarrassment. the same developments, however, can also pose further barriers to those with disabilities. web-based learning materials tend to be heavily text-based, and students may have difficulty in reading or comprehending text or distinguishing colours. many have difficulty with organization, structure or memory, they may find material difficult to navigate, poorly structured or complex, or the interface may be inconsistent. students with mobility problems, limited motor control or vision impairment may not be able to use a keyboard or mouse. the increasing use of multimedia can be beneficial in providing an enriched environment, but it can also be a barrier to those who cannot see or hear audio, video and animations. furthermore, the growth in overseas students and distance learning means that there are increasing numbers who may not be fluent in the language of instruction. the trend towards development of courses and resources online with the support of a virtual learning environment (vle) such as webct and blackboard presents both opportunities and challenges in the creation of flexible, informative and accessible educational materials. part of the challenge is to provide opportunities to the individuals of a wide and varied community, some of whom might not otherwise be able to participate in higher education. people with disabilities face particular barriers to inclusion and the move to online learning may facilitate the lifting of some of these barriers. there is an increasing awareness of the need to make web sites accessible to everyone regardless of ability/disability, with a legal imperative coming from the disability discrimination acts in the united kingdom (http://www.disability.gov.uk/index.html) and australia (http://www.deet.gov.aularchiveliaelanalysislddaedcon.htm), and the disabilities act (http://www.wrightslaw.com/info/section504.ada.peer.htm) in the united states. guidelines, including web content accessibility guidelines 1.0 (w3c), and 'web accessibility resources: quick tips to make web sites accessible' (http://genasys.usm.mame. edulaccess.htm) exist to aid developers of web sites, and an increasing number of sites display the 'bobby approved' logo. however, learning environments and resources for online teaching have other requirements for accessibility beyond functional considerations and the use of 'alt' (alternative text) tags. care needs to be taken with the navigation, structure, content design and communication aspects, and learner-centred design is crucial to ensure that online learning is accessible to students with disabilities. most discipline-based academic course developers are stretched to keep abreast of technical developments. often online courseware development is carried out in addition to their normal workload and they have little time to learn programming skills or understand the technical terminology used in many of the guidelines, such as those provided by w3c. 18 alt-j volume 10 number 2 teaching with technology is not something that can easily be picked up along the way as something to be done off the side of the desk while engaged in more important or timeconsuming activities such as research. (bates, 2000) additionally, many of the existing guidelines (leung, owens, lamb, smith, shaw and hauff, 1999) lack practical examples for implementation and are themselves too technical to be easily followed and understood by academic courseware developers. furthermore, the guidelines tend to concentrate on the functional aspects of web design and the needs of users with sight or physical impairments, while there is a lack of attention to the needs of people with cognitive or learning disabilities. the guidelines also fail to address learnercentred design of online courseware. there is also a paucity of advice available specifically aimed at the design and development of accessible educational courseware for academic developers. a set of guidelines was required which are specifically tailored to learning and teaching in higher education and to web-based learning resources in particular. these guidelines needed to incorporate details of potential barriers to access, practical examples for implementation of accessible solutions and reference to the related w3c guidelines.. this paper presents a case study of best practice in the development of courses in webct (and also has generic application), taking account of the needs of people with a wide range of special needs. rather then focus on particular categories of disability, the approach is to consider the components and design issues concerned with the development of courseware in terms of the difficulties that students with disabilities might face in accessing them. addressing those issues has provided a blueprint for online courseware development that represents not only a design strategy for people with special needs but is equally applicable to learner-centred courseware design in general. the partners this project is a collaboration between the special needs computing research unit (sncru) of the university of teesside and the educational development and technology centre (edtec) of the university of new south wales. it has been supported by a study abroad fellowship from the leverhulme trust, uk. members of the sncru have carried out extensive research and development in the application of computer technology for people with physical and learning disabilities and the senior author of this paper provided the research expertise in this area. current research interests within sncru include development of a symbols-based managed learning environment for students with severe disabilities; assessment, remediation and development of language in children with special educational needs; and investigation of autistic spectrum disorders. the educational development and technology centre (http:llwww.edtecunsw.edu.au) provides services in the promotion and development of institution-wide support for online learning using webct. part of edtec's remit is to investigate means of making education more accessible, through technology, to people with disabilities. the centre's support enables teaching staff to develop online learning environments for themselves. as of march 2002 there were more than 600 courses online with more than 21,000 students enrolled on those courses. 19 elaine }. pearson and tony koppi inclusion and online opportunities: designing for accessibility some definitions accessible design in this context refers to the means by which the content and presentation of an online educational experience enables participation by every student regardless of the browser, browser settings or assistive technology they may be using. the goal should be to enable, as far as possible, the same educational experience for everyone. one should recognize, however, that it may be impossible to cater for all needs, all of the time, but care needs to be taken to provide alternatives when particular needs cannot be met. assistive technology can be defined as devices, tools, hardware or software that enable people with disabilities to use the computer. screen readers, screen magnifiers, alternative keyboards or input devices, voice recognition software and text-only browsers are some of the main examples of assistive technology. inclusion refers to the design of educational environments to take account of the needs of all learners, providing access to a learning experience at a pace and level to suit the learner. online learning presents opportunities for more inclusive education through the 'just in time, just for me' approach offering flexibility in the time, place, mode and pace of learning (collis and moonan, 2001). evaluation methodology as part of its mission to encourage inclusive practices in educational development, edtec aims to encourage staff developing courses in webct (designers) to attain, in the first instance, compliance with priority level one of the w3c web accessibility guidelines. priority one means that the designer is satisfying the basic requirements for access to web documents (w3c). this is in line with webcts own stated aim (webctand accessibility, http:llwww. webct. com/service/). in order to identify the accessibility problems related to the design of courses delivered through an online learning environment, and to propose solutions, an evaluation was carried out of existing courses at unsw, with the consent of the designers concerned. the process involved a combined qualitative and quantitative approach including: • extensive evaluation of webct courses developed at unsw; • discussion with and survey of designers; • interviews with students with disabilities; and • practical evaluation by students of the accessibility of courses using assistive technologies. the evaluation was based on the guidelines provided by w3c 'web content accessibility guidelines 1.0'; analysis of the courses using the bobby web site validation tool; testing with other available checking mechanisms (for example, graphics turned off and monitor set to high contrast); and also with regard to courseware design methods. the w3c guidelines are produced as part of the w3c web accessibility initiative and provide a guide to ensuring the accessibility of functional aspects of a web site. there are fourteen guidelines or general principles, each with a set of checkpoints, and the rationale 20 alt-] volume 10 number 2 behind each guideline. web bobby is a tool which provides a crude but indicative analysis, allowing the designer to check the extent to which their courseware has met the criteria of the w3c guidelines. however, evaluation of a learning environment which takes account of the requirements of those with sensory and learning disabilities also requires analysis of the structure, presentation, content and navigational aspects. for this the principles of learner-centred courseware design (figure 1) were employed. a model of courseware design that combines content design, navigational design and visual design but is informed at all stages by the learner model (pearson and green, 1999) is the one on which this evaluation is based. briefly, the model incorporates content (design of learning materials and activities), navigational design (flow of control and structure), visual modelling (user interface, icons, graphics) and, crucially, learner modelling (who the learner is, what the learning needs are, how the learner will learn, the aims and learning objectives). visual design / content design i \ r learner model \ \ \ navigation design figure i: a model ofleamer<entred courseware design (adapted from pearson and green, 1999) a set of evaluation criteria combining the w3c guidelines and aspects of learner-centred courseware design was developed. the evaluation criteria were also informed by the results of interviews with a group of students. they were asked about the assistive technology they use, their experiences in accessing the web and webct for learning resources, the difficulties they encountered and the benefits of online learning for them. the resulting set of seventeen checkpoints represents an instrument that is specifically aimed at evaluation of online learning environments as opposed to web sites in general. questions included: • is the content including page organization, headings and lists consistent? • are navigational elements consistent and well structured, do icons and pictorial images make sense? • is the content structured in usable chunks, avoiding excessive scrolling and are longer documents organized with links to sections and subsections? — elaine j. pearson and tony koppi inclusion and online opportunities: designing for accessibility • are learning materials presented in an accessible format, and if not, are alternatives provided? • is the material accessible to assistive technologies (screen readers, speech recognition software, text browsers, keyboard input, etc.)? course evaluation thirty webct courses were investigated. the evaluation was carried out to determine, in particular, the level of accessibility for students with sensory or cognitive disabilities. attention was paid to the structure, content, navigational design, communication facilities and compatibility of webct with assistive technologies such as screen readers, dictation software and browser settings. the issues raised from an evaluation of the courses investigated and from the interviews of students with disabilities enable the results to be presented in two categories: generic and webct-specif\c considerations. students' comments students' comments have been largely integrated into the results presented below, but some specific examples have been selected here to illustrate some of the problems faced by people with different disabilities in accessing courses. a student with dyslexia found the home page problematic on one course. he relies on the use of visual cues in remembering structure: the icons are too small, too complicated and the text links contain too many words. i don't know what these icons mean (a wand for evaluation, man at work for assignments), they are not memorable and with the long text links i wouldn't remember where elements of the course belong. also the background is too busy and interferes with the information i'm trying to look at. a student with rheumatoid arthritis and diabetes (which can cause eyesight problems) had problems with contrast, pdf documents and long scrolling documents: the welcome screen is orange/yellow background, red text; the colour contrast makes it difficult to read. the colours used in the course outline document are also difficult (orange/yellow with black text) it blurs when i try to read it. plain black text on white background is best. there are long scrolling text documents 1 would find it confusing and tedious to scroll through all this text to find a particular piece of information. i want to be able to home in on a particular section without going through it all every time. a student with severe nystagmus and very low reading vision had problems with pdf documents and with the discussion and chat functions: the online readings (pdf) are too small and the typeface is poor. it is too difficult to read. i would need dictation software to access the chat or discussion areas, but it doesn't seem to work with the chat tool, so it would be useless to me. a totally blind student found navigating through the webct interface tedious, but his main problem was with learning materials presented in an inaccessible format. 22 alt-j volume 10 number 2 as a law student, i find it particularly frustrating when many of the papers i need are in pdf format, also powerpoint slides are no use to me online, nor other read-only formats. i cannot use online chat easily because the conversation moves on too quickly for me to read. evaluation results generic the analysis of generic issues represents elements of courseware design which are a result of the methods, structure, design and presentation of materials by the designer that may pose difficulties in accessing the learning environment for students with disabilities. these generic issues are not because of any constraints imposed by webct itself. the following were the major problems encountered. inconsistent and inappropriate use of icons or graphics many students with learning disabilities rely on pictorial cues for navigation or understanding learning objects. no text equivalents for graphics, figures or other illustrations this can be a problem for people with vision impairments who rely on screen readers to interpret visual elements in a text format. poor organization of content there is a general design issue here in the tendency towards 'shovelware' content intended for traditional delivery is merely uploaded for online delivery. the result is large amounts of scrollable text which is difficult to manage and material which is not 'chunked' for manageability. there is also a tendency not to take advantage of some of the features of web ct for organization of content. the widespread use of pdf file formats for the presentation of text documents this is a major problem. pdf files cannot always be interpreted by screen reading software. for students with vision impairments for whom the traditional library is totally unusable, the presentation of documents online can be a huge benefit. the latest version of adobe acrobat reader (http://www.adobe.com/products/acrobatlsolutionsacc.html) provides an accessible solution, provided the document has been created in the first place in an accessible format. however, documents scanned and saved as graphic image files before conversion to pdf remain inaccessible. there are similar problems with other documents including powerpoint presentations or movies created in macromedia flash. text may not be correctly or accurately interpreted by screen readers this can present problems not just to vision-impaired students but also to those who have difficulty in reading, interpreting or understanding text or complex language. words in capitals may be read as individual letters, abbreviations may not be recognized, and unusual characters are interpreted literally; for example, the symbol ~ when used as a line separator, will be read as 'tilda, tilda, tilda, tilda...' for as many times as the symbol appears. the use of tables can be a problem for people with disabilities because screen reading software reads the information across cells, without appropriate table header descriptions and careful use of layout, the contents of the table makes no sense to the learner. poor interface design while designers may wish to customize their courses to reflect the individual academic and — elaine j. pearson and tony koppi inclusion and online opportunities: designing for accessibility course, inappropriate use of foreground and background colours and contrast can make an interface hard to distinguish, read or interpret for people with visual, perceptual or spatial difficulties. similarly the use of colour to convey important information and inconsistent use of icons can make access difficult for people with vision impairments or learning disabilities. evaluation resultswebctspecific whilst it is webct's stated aim to meet level one compliance with the w3c accessibility guidelines, some problems remain. the problems identified are not unique to this study {http:liwww.webct.com/wywlviewcontent7contentid-262793t) and the webct designers' online discussion forum (http://www.webct.com/webctlforum/topics?discussion=7816&topic =21849) has been helpful in highlighting the efforts of others in this regard. in seeking to provide designers with guidelines for inclusive design, it is important to distinguish between features that are subject to their own design decisions, and those factors that are attributable to the webct environment. the main problems encountered to date with features of the webct environment are: there is no text equivalent for some icons in the icon library; some titles and headings are graphics without text equivalents this is a problem for those using screen reading software. also the headings cannot be disabled, making it cumbersome for those using screen readers or with slow modem connections. the use of tables can be a problem just as outlined above, tables need to be properly defined to ensure that screen readers can interpret the information they contain. the overuse of frames in some instances, it has been reported that up to seven frames may be open on a screen at one time. once again, this can be confusing for those who rely on screen reading software. the use of applets and java for the implementation of some of the interactive elements of webct, including chat windows, discussion groups, the whiteboard and email this means that they cannot be accessed by students using dictation software, and presents a barrier to those for whom text entry and other forms of communication are difficult. synchronous chat is very difficult for people who have problems entering text or reading quickly enough. students complain that the conversation has moved on before they have the opportunity to respond. we would hope that these issues will be resolved in future versions of webct. in any case, many course designers will be using current and earlier versions for some time. staff perceptions a short survey was carried out of staff at unsw who are engaged in the development of online courses to determine their views on and awareness of issues of accessibility to online learning, and to confirm the results of informal discussions with six course developers. the questionnaire was distributed as part of a larger survey via webct to a hundred course designers at unsw, which generated twenty responses. 24 alt-] volume 10 number 2 academics at unsw build whole courses themselves, with support from edtec in instructional design, the use of webct, and specialist graphics, audio and video services. discussions with staff reveal that for those who have developed their first course in webct, the main concern has been to get material online with limited instructional design support and restricted timescales. some are not sure whether they have students with disabilities on their course, and are often not aware of the techniques and design methods that would make their courses inclusive. however, having successfully developed a course, the academics are keen to incorporate inclusive design into the new courses they are developing. the survey results support the anecdotal evidence. a total of 15 per cent of respondents could confirm that there were students with disabilities enrolled on their webct courses, while 45 per cent believed they had no students with disabilities and 30 per cent were not sure. there are a number of reasons why staff do not believe or are not sure if they have students with disabilities. it could be a lack of awareness of disability by academics; students may not be registered as having a disability especially if they have a learning difficulty; students may be reluctant to identify themselves as having a disability; or they may not be aware themselves that they have a disability. this final point is particularly true of some students with specific learning difficulties (such as dyslexia). students might identify themselves to the equity and diversity unit only if they believe they require particular assistance. staff were asked to indicate the type of disability their students have. this revealed that a range of disabilities is encountered: learning/physical disability, physical/ sensory disability and sensory disability. when asked if they had been able to take account of learners with disabilities in the design of their webct courses, 45 per cent believed that they had, 5 per cent said they had not, and 50 per cent did not know. however, 55 per cent of designers also expressed an interest in receiving training in the design of inclusive courses. discussion and outcomes guidelines ideally, the needs of learners with disabilities should be addressed at the design stage rather than attempting to retrofit solutions. while existing courses can be improved, through training, awareness and support, designers can create courses that meet the needs of most of their learners most of the time. a structured approach is required to help academics develop accessible courseware. at a general level, staff should be encouraged to use the available features of the learning environment effectively. this would help in the structuring of the environment and in making the transition from traditional to online learning. staff awareness needs to be raised of the available guidelines (such as w3c), checking facilities (bobby), and software development support (such as macromedia's accessibility extension for dreamweaver four and ultradev for checking the validity of web pages as they are created). above all, designers also need support in developing learner-centred courseware (harper and hedberg, 1997). the findings of the investigation have been distilled into a set of guidelines (pearson and koppi, 2001) for academic designers of courses in webct, but which are generally 25 elaine j. pearson andtony koppi inclusion and online opportunities: designing for accessibility applicable to all web-based learning environments. the guidelines take the form of checkpoints, with advice, hints, practical examples and techniques for implementation a 'how to' guide. the guidelines are available from http://www.edtecunsw.edu.au. this form of guidelines (which relate to those of w3c) is more useful to academic staff than the technical w3c guidelines which are not specifically aimed at the academic designer of online learning environments. the checkpoints employed by these guidelines are summarized as follows: • ensure consistent and appropriate use of graphics, icons and other visual cues; • ensure all graphics, figures, and other illustrations include a text equivalent; • organize content to take account of the transition to an online environment; • use the features provided by webct to organize and structure course content; • make pdf and other read-only file formats accessible; • be aware of the limitations of screen readers in interpreting unusual text, characters and abbreviations; • ensure that tables are carefully and appropriately used; • ensure appropriate use of colours and contrasts in screen design; • provide alternative sources of information for video or audio. each checkpoint includes: • an explanation of its relevance to accessibility; • advice and practical examples illustrating how to implement the checkpoints using webct and dreamweaver (the predominant web authoring software available to academics at unsw and uot); • referential links to further information where appropriate; • a reference for each checkpoint to the relevant w3c guideline(s). the document includes a second section which advises the designer on methods of checking their courses and resources for accessibility including using the available checking tools, simple manual checking methods and reference to edtec's 'quality teaching principles and guidelines' {http:llwww.edtecunsw.edu.au) for more advice on design issues. dissemination and future work these guidelines were developed for staff at unsw and others using the webct environment but they have universal application. there is clearly a demand for similar more generic guidelines across the academic community and there is scope for further related work. the university of teesside is beginning to implement blackboard as the environment to support online learning. since webct and blackboard are currently two of the most popular commercially available online learning environments worldwide, the guidelines will be extended to include specific advice for users of blackboard as well as webct. we intend to carry out a detailed evaluation of the guidelines and their effectiveness in 26 alt-j volume 10 number 2 assisting the academic in inclusive course design and development. other academics and institutions are encouraged to use the guidelines in the development of their own courses, and their feedback will be sought in developing and extending the guidelines. in response to the results of the survey, which indicated a need for training in accessible design, we have developed a series of training workshops to give staff hands on experience of assistive technologies and practice in accessible design. initially these workshops form part of the unsw suite of training workshops. they will then be extended to the university of teesside as they develop their support program for blackboard, and subsequently to other centres and universities. in addition, we have developed an online course, based on the guidelines, in accessible course design which is being used in the professional staff development program at unsw and also as a postgraduate module at uot. conclusion at the end of this stage of the project, we have been able to present an analysis of the problems encountered by students with disabilities in the use of web-based learning. as a result of this analysis, a set of proposals for the design of online courseware that is accessible not just to those with disabilities, but to all students, has been developed. these proposals have been distilled into practical guidelines presented in a 'user-friendly' manner for academics who simply want to get their courses online in the most efficient and accessible way. the next stage has been the development of workshops and an online course in accessible design which build on the guidelines and provide the opportunity for a better understanding of issues, techniques and tools in accessible design. the key to accessible courseware is to take a learner-centred design approach, one which takes account of the needs of the learner. although one cannot expect always to meet all the needs of every learner, reasonable steps can be taken to ensure the widest participation. even if we are not meeting the needs of all potential users we should be aware of exactly who we might be excluding. designing for inclusion provides the opportunity to develop accessible courses for all students, and the move to online learning prompts us to improve our teaching and learning practices for all our students. acknowledgements this project has been supported by a study abroad fellowship from the leverhulme trust, uk. references bates, a. w. (2000), managing technological change, san francisco: jossey-bass. collis, b. and moonan, j. (2001), flexible learning in a digital world, london: kogan page. harper, b. and hedberg, j. (1997), 'creating motivating interactive learning environments', paper presented at the ascilite conference, perth, 7-10 december. http://www.ascilite. org.au/conferences/perth97/papers/harper/harper.html, url last accessed jan. 2001. 27 elaine j. pearson and tony koppi inclusion and online opportunities: designing for accessibility leung, p., owens, j., lamb, g., smith, k., shaw, j., and hauff, r. (1999), assistive technology: meeting the needs of students with disabilities in post-secondary education, insititute of disability studies, deakin university, 99/6. pearson, e. j. and green, s. j. (1999), 'courseware engineering or design', association for learning technology conference (alt-c), bristol. pearson, e. j. and koppi, a. j. (2001), 'guidelines for accessible online courses', internal publication, university of new south wales, http://www.edtec.unsw.edu.au. 'adobe acrobat five', http://www.adobe.com/products/acrobat/solutionsacc.html, url last accessed 3 april 2002. cast 'bobby worldwide', http://www.cast.org/bobby/, url last accessed 3 april 2002. 'consultations on the draft disability standards in education', http://www.deet.gov.au/ archive/iae/analysis/ddaedcon.htm, url last accessed 3 april 2002. 'disability on the agenda', http://www.disability.gov.uk/index.html, last updated 6 july 2001, url last accessed 3 april 2002. 'quick tips to make accessible web sites', http://www.genasys.usm.maine.edu/access.htm, url last accessed 3 april 2002. 'section 504, the americans with disabilities act, and education reform', http://www.wrightslaw.com/info/section504.ada.peer.htm, peer, url last accessed 3 april 2002. 'some thoughts on accessibility and universal design', http://www.webct.com/wyw/ viewcontent?contentid=2627931, stephen rehberg, georgia state university, url last accessed 3 april 2002. 'web content accessibility guidelines 1.0', http://www.w3.org/wauresources/#gl, url last accessed 3 april 2002. 'webct discussion forum on accessibility', http://www.webct.com/webct/forum/ topics?discussion=7816&topic=21849, url last accessed 3 april 2002. 28 editorial history can be characterised as the overestimation of what can be accomplished immediately and the underestimation of long term consequences. (strassman, 1984) my first encounter with computer-based learning was in 1975 when i trialled packages produced by the chelsea science simulation project while on teaching practice. access to the minicomputer running the packages was via modem and teletype, and response times were measured in minutes rather than seconds. three of the four packages were little more than computer-based lookup tables but the fourth, despite the severe limitations of the technology, inspired me to take computers seriously as tools for learning. the application was a simulation of crossing different strains of fruit fly, a technique beloved of geneticists but impractical for school biology. i was impressed by the use of a computer to achieve something highly educational that was impractical by any other means. there have been many significant advances in computing since 1975: the advent of the eight-bit microcomputer, hypertext, graphical user interfaces, object technology, cdroms, the internet and the world wide web. prophets of what the latest technological developments will achieve for education have been many, and by and large false. progress in the short term has appeared painfully slow. pioneers have felt hurt by the failure of their colleagues to embrace the computer-based learning materials they have worked so hard to perfect, but perhaps they failed to recognize the extent to which they had embedded their own idiosyncratic approach to teaching into their products. the association for learning technology was founded by a group who believed that the long-term consequences of technology applied to education would be profound. they were keen to create a forum for learning technology practitioners as a way of identifying and sharing good practice. the annual alt conference has reflected a growing maturity among both technologists and teachers as they gain insight through shared experiences. 2 aa-j volume 7 number i the theme of the fifth alt conference, lifelong learning on a connected planet, was a challenge to delegates to think through the implications for education of a connected planet; one where all points on the global internet are equally accessible to all other points. five years ago the concept of the online course did not exist. now it is coming to be seen as the growth sector for tertiary education. many new online courses are conventional courses that have been 'ported' to the new medium. if it is done well the pedagogic qualities that made it a sound conventional course will also benefit the online version, but many inappropriate assumptions will also have been carried over. to be as good as, or better than, an equivalent conventional course, an online course needs to be rebuilt from the ground up. the course needs to integrate student interactions with the materials, with tutors and with fellow students, and the learning environment has to be flexible enough to recognize and support a diversity of student needs. the same fundamental pedagogic principles apply to all types of course but the peculiar strengths and weaknesses of the internet as a delivery and support medium require a wholesale reassessment of teaching methods. a consensus on effective online education has yet to emerge. in this journal you will find documented a rich vein of innovation, experimentation and research. there is imagination aplenty. not all approaches will prove to be successful but all are helping, to develop a better understanding of how to use technology to create more and better educational opportunities for more people. reference strassman, p. (1984), information payoff, cambridge mass: mit press. jonathan darby learning technology is inherently eclectic. professionals from many areas are involved: managers, courseware developers, teachers in universities and schools, trainers in industry, and, of course, learners. the range of the papers presented at alt-c98 bears witness to this diversity, and to the way in which the alt membership represents the uk learning technology constituency. likewise the papers in this special issue reflect the diversity of this constituency, with papers on the management and implementation of learning technology, discussions of the future role and impact of learning technology and reports of research studies. i think it is essential that alt-j continues to publish high-quality papers in these areas. the founding editor, gabriel jacobs, has established an excellent foundation for developing the journal's profile in all three areas. with grainne conole's help i look forward to working with you to continue this development. david squires 3 calt12205.fm alt-j, research in learning technology vol. 12, no. 2, june 2004 transactional distance in a blended learning environment jon dron*,a, catharine seidelb & gabrielle littenb auniversity of brighton, uk; bcrawley college, uk taylor and francis ltdcalt12205.sgm10.1080/0968776042000216219alt-t research in learing technology0968-7769 (print)/1741-1629 (online)original article2004taylor and francis ltd1220000002004jondronschool of computing, mathematical and information sciencesuniversity of brightonwatts building, lewes roadbrightonbn2 4gjukjon.dron@brighton.ac.uk this paper presents a case study that describes and discusses the problems encountered during the design and implementation of a blended learning course, largely taught online through a web-based learning environment. based on moore’s theory of transactional distance, the course was explicitly designed to have dialogue at its heart. however, the reality of systemic behaviours caused by delivering such a course within a group of conventional further and higher educational institutions has led to an entirely unanticipated reversion to structure, with unpleasant consequences for both quality and quantity of dialogue. the paper looks at some of the reasons for this drift, and suggests that some of the disappointing results (in particular in terms of the quality of the students’ experience and associated poor retention) can be attributed to the lack of dialogue, and consequent increase in transactional distance. it concludes with a description and evaluation of steps currently being taken to correct this behaviour. introduction this paper describes a blended learning course which has attempted to apply michael moore’s theory of transactional distance. this theory asserts that it is transactional rather than physical distance which is of greatest significance in any educational transaction (moore & kearsley, 1996). transactional distance is measured on a continuum of structure and dialogue. the more structure, the less dialogue and vice versa (figure 1). a highly structured course will give little opportunity to challenge concepts and explore congruent or tangential paths, while dialogue will inevitably lead to departures from planned outcomes and result in new, unanticipated learning outcomes. relationship between dialogue and structure *corresponding author. school of computing, mathematical and information sciences, university of brighton, watts building, lewes road, brighton bn2 4gj, uk. email: jon.dron@brighton.ac.uk issn 0968–7769 (print)/issn 1741–1629 (online)/04/020163–12 © 2004 association for learning technology doi: 10.1080/0968776042000216219 164 j. dron et al. moore does not suggest that either structure or dialogue is an inherently good thing. each may be appropriate in different circumstances and a typical educational event such as a conventional lecture will, at a micro-level, move constantly between the two. however, the reciprocal relationship between them at any given point is immutable. a third dimension of the theory suggests that more autonomous learners, being self-directed, are better able to cope with more structure while less autonomous learners benefit more from greater dialogue. moore equates less autonomy with more teacher control. experimental evidence (saba & shearer, 1994) based on predictions of a system dynamics model correlated with discourse analysis, appears to confirm that the theory is substantially correct. however, there may be some grounds for believing that there are rare exceptions (dron, 2004) and there is a fuzziness in his formulation that allows for different interpretations depending on whether transactional distance is considered as a two by two matrix, a single continuum or distinct clusters (garrison, 2000). in this paper we interpret structure and dialogue as broadly characterising two extremes of a single continuum with a reciprocal relationship between the two. beginnings in 2000, hefce (the higher education funding council for england) announced that it would fund a number of pilot projects to create foundation degrees. foundation degrees are strongly work-based or work-related, designed and managed in collaboration with employers and delivered mainly by further education (fe) colleges. equating to the first two years of an honours degree, they target students who might otherwise be excluded from higher education, notably those with unconventional qualifications and/or in work. their focus tends to be more practical than academic. led by the university of brighton, a consortium consisting of two universities, four fe colleges and various employers submitted a successful bid for hefce funding which led, in 2001, to the development of the esystems design & technology degree. the course was designed as an internet computing degree, to be delivered solely as a part-time, four-year course. for strategic reasons the plan was to deliver at least some of the course electronically, though the exact amount of and mechanism for this delivery was not made explicit. a course development team (cdt), consisting of representatives from each college and a university-based development leader, was appointed and given figure 1. relationship between dialogue and structure transactional distance in a blended learning environment 165 responsibility for course development, with quality control provided by the university. a course leader from the university was appointed with a research background in and practical experience of online collaborative learning. however, though playing a significant part in strategic planning, other commitments prevented the appointee (the first author of this paper) from playing more than a strategic role in course development. defining the course ethos decisions about the form of the course were made, grounded in theory and pragmatic choice. pragmatically, the internet was proposed as the main delivery mechanism so that students could access course materials from their workplace or from home. this was also seen as an effective means of monitoring and controlling the content and quality of the course across multiple colleges. lack of time and resources militated against the development of highly structured course materials. as the course was aiming to attract learners who were unwilling or unable to enter conventional higher education, there was a need to cater for a rich variety of prior knowledge and learning styles. wedermeyer (1971) noted that independent study of the early 1970s typically offered little freedom in goal determination and activity selection, an issue that has arisen frequently since then despite the ubiquity of computer-mediated forms of distance education. the need to cater for multiple learning styles implied alternative perspectives on the resources, multiple resources catering for different needs or an adaptive hypermedia approach (brusilovsky, 2001). furthermore, to cater sufficiently for a variety of learning styles it was felt important to make use of rich media which, as laurillard observes are generally too expensive to be produced by individuals (laurillard, 1993, p. 182). however, with little time to develop such resources, these approaches were not deemed possible. applying transactional distance theory, it was decided to define an ethos based on learning through dialogue. a dialogue-based approach would require a lower commitment of development time, albeit at the expense of time spent maintaining dialogue when the course was running (figure 2). most discussions would be facilitated through online asynchronous forums. synchronous approaches were felt to provide insufficient flexibility for students in full time work, as well as posing some technical obstacles. relationship of delivery to development time for structured/dialogic delivery modes the course was designed to be delivered in a blended mode. seely brown and duguid observe that ‘digital technologies are adept at maintaining communities already formed. they are less good at making them’. (seely brown & duguid, 2000, p. 206). for this reason, online discussions would be supplemented by occasional face-to-face meetings. they would also support the few parts of the course that required specialised laboratory facilities, which were kept to a minimum by basing practical work primarily on free and open source software. using computer mediated communication is a potentially hazardous approach as mediated communication is in some ways a pale shadow of face-to-face discussion 166 j. dron et al. and is not a simple replication of such discourse (peters, 2000). given the need to develop a learning community, course developers were urged to employ salmon’s five step model (salmon, 2000) and were given training in the approach. salmon outlines the five stages through which she believes an online learning community must pass in order to reach the stage at which learners are constructing knowledge through interaction—access & motivation, online socialization, information exchange, knowledge construction and finally, development. the model was adopted to provide teachers new to delivering distance learning with a reassuring and effective framework to assist them in coming to terms with this unfamiliar mode of teaching. with an explicit social constructivist ethos (vygotsky, 1978), it was hoped that, with time and suitable models of good practice, much of the learning might be facilitated by the students themselves. because students would come from relevant work backgrounds and (by virtue of enrolling on such a course) were expected to be mature, motivated learners, it was hoped that they might have much to offer in terms of experience and knowledge and form an effective community of practice. as wenger puts it, ‘the value of an institutionalized setting for learning often resides as much in its community-building potential as in the pedagogical intentions of its curriculum’ (wenger, 1998, p. 208). many authors have emphasized the importance of performance evaluation in education (bruner, 1966; gagne, 1985; laurillard, 1993). summative and formative assessment would be embedded throughout the course with a wide variety of approaches including portfolios, programming exercises, online tests, examinations, vivas, reports and problem-based assignments, in the belief that this would help figure 2. relationship of delivery to development time for structured/dialogic delivery modes transactional distance in a blended learning environment 167 maintain focus and motivation. the result of this was that students would seldom spend more than three weeks without the need to submit some form of assessed work. although efforts were made to make much of this assessment as authentic as possible, in particular by relating it to the students’ workplace, the course leader had some misgivings about the reliance on assessment to provide extrinsic motivation (whitehead, 1929; illich, 1971; holt, 1977). preparing for delivery the cdt met face-to-face approximately monthly, using an asynchronous discussion forum to facilitate the shared development of module specifications between meetings. this small group were able to engender a strong sense of shared responsibility and identity, which it was hoped would be replicated amongst student groups once the course was running. once the course outline had been approved and module specifications agreed, tutors from each of the four participating fe colleges were appointed to develop and to deliver the course. only two of the 20 or so tutors involved in preparation of the first year’s modules had any experience of distance learning, let alone online delivery, and only a handful had been involved in course development prior to this. most were excited by the challenge though a small but significant minority expressed scepticism about the wisdom of choosing this method of delivery, preferring a traditional faceto-face approach. all tutors were required to attend an online tutor training (ott) course created by one of the more experienced online tutors, based heavily on salmon’s five step model of e-moderating (salmon, 2000). an e-learning expert was employed to facilitate workshops in which module outlines were prepared and development work shared amongst the teaching team. this helped to encourage a strong sense of ownership of the modules by their development teams. these sessions had a strong emphasis on developing module structures and associated static materials, with, perhaps, insufficient emphasis on dialogue. for many tutors this was their first encounter with the course and its approach, so these workshops influenced them strongly. the cdt believed that they had anticipated some of the potential pitfalls of online education, but it soon became evident that this was not so. the huge change in teaching approach required and the cascade of systemic dependencies that it would set in motion is the main topic for the rest of this paper. the first two years twenty-eight students enrolled at the start of the first year. to compound this disappointing start, one college recruited only three students, one of whom dropped out almost immediately followed by a second a few months later. within a couple of months of starting, two of the colleges had amalgamated, with consequent upheavals and uncertainties for staff and students alike. 168 j. dron et al. in the rush to prepare for the start of the course, it went almost unnoticed that a small number of tutors had failed to attend the ott course and that, of those who had, only about half had successfully completed it by the time the course began. the main reason given was the lack of time, but the consequent lack of experience of dialogue-based learning proved problematic later. a poor student retention rate was matched by a high turnover rate for staff. although another instance of the ott course was run, this failed to keep pace with the rapid staff turnover which in turn meant that the already over-stretched staff were even less prepared for the processes of online teaching. to make matters worse, new tutors lacked the sense of ownership of the process that motivated the founding team. running the course: the tutor experience it became apparent early in the first year of the course that most tutors were relying on familiar teaching methods and using the discussion boards only as an occasional means of communication with the students. despite extensive online and offline encouragement this reliance on structure rather than dialogue did not improve throughout the first two years of the course. fe college tutors’ very full timetables contained discrete blocks of time when teaching was supposed to occur for this course (typically a single block of three hours each week, a mere thirteen percent of an average fe lecturer’s teaching time), militating heavily against its asynchronous delivery given that virtually all the students on the course are working, by far the majority of their study and communication occurred in the evenings or at weekends. although most tutors adapted to this pattern of working, it fitted very uneasily with the rest of their over-filled timetables and their expectations that evenings and weekends should be free from the demands of student contact. tutors found that dialogue-led delivery often took much longer than they had anticipated. to save time it was often simpler to rely on more traditional methods including a tendency to put off solving students’ problems until the scheduled faceto-face session. this was exacerbated by the fact that many of the early course modules involved the acquisition of specific skills and information (programming, database design, simple internet technologies and concepts). these topics were often perceived as lending themselves more readily to a structured one-way mode of delivery. communication was confined to clarifying misconceptions and discussing good practice, tending to be directive rather than discursive. running the course: the student experience the reason most often given for dropping out was and is students’ inability to cope with the demands of their work and the course. however, it soon became clear that some of their preconceptions about the course and their own learning processes were at odds with the course ethos. only a few were regular contributors to discussions, with many content to simply log in, do the set reading/tasks and log out again without transactional distance in a blended learning environment 169 any dialogue. the large amount of assessment spread throughout the course was over-demanding and demoralizing for many students. those hard-pressed for time got on with the essential reading and the assignments, and omitted the unassessed, and therefore apparently unnecessary dialogue. although some students were willing to help each other out with corrections to coding, good explanations and resource sharing, others saw helping each other as a diversion from their own studies, failing to appreciate the benefits that can ensue from teamwork and dialogue. some students lacked confidence, and so avoided posting questions or problems to the discussion forums, for fear of ridicule. perhaps because of their perceptions of education, combined with the demands of their employers, many students valued the acquisition of skills and factual information more highly than the surrounding processes of learning that can turn it into useful knowledge. lack of timely feedback discouraged students from using the medium. one student summed it up well: we do need more feedback to discussions and problems posted to the teamroom. some tutors are very good at this but others seem only to drop in once a week—and a week is a long time on this course! several students feel that, whatever the value of the classroom contact, it should be possible to do the course entirely remotely. the compound effect of all these factors was that in most modules, only staccato discourse occurred, rarely stepping higher than salmon’s third stage of information exchange (salmon, 2000). a vicious circle one tutor complained, ‘i have tried without much success to get students to have discussions in the teamroom. one or two students may join in, but no one else does— you can end up having a discussion with yourself.’ a student expressed the view, ‘teamroom activity breeds teamroom activity, and lethargy breeds lethargy.’ a vicious circle developed. a predominance of skills-based learning combined with tutors’ reticence to encourage dialogue online meant that the discussion areas were under-utilized. the fewer expectations placed on the students, the less they felt the need to contribute. at this point, moore’s theory of transactional distance became very apparent. as dialogue diminished, structure came to dominate. students, insufficiently supported by a learning community, came to rely increasingly on course notes. the more structured the course became, the fewer the opportunities for dialogue arose. the effects of this on less autonomous learners were devastating. few of the students had acquired the skill to be autonomous learners in an academic setting. the fact that most had responsible jobs in ‘white water’ environments (vaill, 1996, p. 4), where change is the norm and learning must be a way of life was not considered. for better or worse the course was built around a traditional academic framework, so students needed help and guidance to explore issues and 170 j. dron et al. clarify concepts. in the absence of an effective discussion forum, the brief face-to-face meetings, originally intended mainly to provide social cohesion, increasingly became the main occasions for learning on the course. one student noted: lessons are great, covering points missed and checking everything is as it should be, but as everyone seems to realize team room could be used more for discussion and covering points. torn between the demands of work and the need to participate in these contact sessions, with inadequate structured resources and no community to turn to in between, many students became disillusioned, and demotivated. by the end of the first year, half of the students had left the course and the college with a single remaining student left the programme. with only two fe colleges now involved in delivery, the second year began. although recruitment was a little lower than that of the first year, slightly better retention rates reflected refinements in delivery and design, especially a slight reduction in the assessment load. however, it was apparent that the course was still plagued by systemic problems, driven both by the tutors and the students. away from structure, returning to dialogue senge argues that, in any system, structure is a key determinant of behaviour (senge, 1993, pp. 40–41). the central problems of the course have been observed to be structural in nature, leading to systemic behaviours which hinged around a number of inter-related issues. these in turn led to a cycle of ever-decreasing returns, excessive workloads and a relative paucity of rich interactions in the virtual space. it was felt necessary to break that cycle. it was not possible to radically improve the allocation of time to tutors, and it was impractical to consider a major change in the surrounding educational environment and timetabling. there was still no time to develop effective structured learning materials. however, increasing dialogue seemed an achievable target. in collaboration with tutors, a simple ‘chunk and talk’ strategy was developed. students would be presented with a discussion forum as the entry point to each module, in which the tutor would outline the tasks for the week (or chunk of several weeks) and initiate a dialogue. by starting with a discussion, it was hoped that students would focus much more on process rather than absorption of content, and that there would be more of an emphasis on the learning community, reducing isolation for the learners. chunking topics into logical units provided students with more control over their own work patterns, although this was counterbalanced somewhat by the demands of dialogue, participation in which required continual attention. to help build a more dynamic learning community (and also because of the declining group sizes), the online teaching for both colleges was amalgamated, with a single tutor responsible for leading the online discussion for each chunk of learning. faceto-face meetings would, however, still be led at each college by the local staff member. transactional distance in a blended learning environment 171 it was hoped that this would engender more collaboration amongst the teaching team, and that as staff gained a greater understanding of the importance of online communication and collaboration, they would find it easier to facilitate the development of this sense of partnership amongst the student community. to help assist tutors lost within an unfamiliar online environment, strong advice was given to module designers to follow a pedagogy based on kolb’s experiential learning cycle (kolb, 1984). this cycle of abstract conceptualization, reflective observation, active experimentation and concrete experience, with its focus on situated learning, was seen to fit neatly with the work-related/work-based nature of the course as well as providing a simple framework and easy-to-follow formula. summative assessment would be significantly reduced and what was left would be embedded further into the practice of the course. specifically, some assignments required participation in online discussions, for example in designing the solution to a problem or exploring recent technological developments some early results the new approach has been implemented for one semester to the current cohort of 10–12 students per year group. modules that are running for the first time have been developed on the basis of this approach, while existing modules have been adapted to a greater or lesser extent. the results have not all been encouraging. staff changes and the limitations imposed in the hard-pressed fe sector have again resulted in insufficient time being allowed to do the approach full justice and not all staff have taken the advice to use the kolb cycle as a framework. similarly, although there is general recognition of the need for discussion, several tutors continue to use forums to ask simple, answerable questions rather than exploring the problems and issues that allow learners to explore concepts and engage in dialogue. just over 60% of all accesses to the system over the past semester were in the communication areas. analysis suggests a roughly similar proportion in previous years (60% discussion for the first year the course ran, 57% for the second) which, given potential inaccuracies and the fact that the systems in use are not directly comparable, is disappointing. however, the use of communication areas varies with year group, with rates of 64% for first year students, 61% for second year students and 57% for third year students. there are still very few deep conversations going on, with the majority of messages concerned with either problem solving or the process, such as giving assignment dates and schedules. worse, although it is difficult to draw direct comparisons due to the different organization of the sites, there seems little difference in the actual number of messages posted when compared with previous years. one positive outcome has been observed. this year, the most active students have posted around three times as many messages as the least active, whereas last year the ratio was greater than eighteen to one. this broadening of participation in dialogue 172 j. dron et al. will lead to a greater awareness of the presence of others, hence reducing the loneliness and isolation that less active participants experienced before. retention rates have improved when compared with the same period in previous years. discussion forums are not the only reason for this: contributory factors include a more thorough and honest interview process, a reduction in assessment load, greater experience of tutors, a more reliable online environment, more diverse discussion groups, better pedagogy and reduced dependence on tutorprepared materials. the move to a more discussion-oriented process has not been universally popular with our first cohort of students, now in their third year. it is directly responsible for the loss of one student from this group who commented: i’ll be back next year when, hopefully, the course will not discriminate against students who cannot make timely contributions to discussions (or the university pays my mortgage so i can give it priority over my employer!) i’m off now to read a few books over the coming months just to prove to myself, at least, that it is possible to learn without discussion. most students who joined in the first year of the course are of necessity autonomous learners, as any who were not have already left. for such learners it may seem an unnecessary imposition to require them to be involved in a collaborative learning process. it is also possible that insufficient attention has been paid to the significant benefits of peripheral involvement or ‘lurking’, which might still enable learners to enjoy the benefits of internalized dialogue. preece suggests that this kind of involvement may be highly immersive, leading to strong senses of belonging to the community despite lack of direct interaction (preece, 2000, pp. 87–90). conclusion the evolution of this course continues and this paper is just a snapshot capturing its progress so far from the authors’ highly situated perspective. there are few immutable conclusions to be drawn from the experiences presented here, but several issues seem worth emphasizing. firstly, the struggle for predominance between dialogue and structure has shifted throughout the course’s short life and shows no signs of abating, despite well-intentioned efforts to force it in a particular direction. this lends credence to moore’s theory of transactional distance: as structure increases, dialogue decreases and vice versa. secondly, it is difficult to implement something new when surrounded by selforganizing forces that tend towards equilibrium and reinforcement of the status quo. though by no means original, the following conclusions also bear repeating. firstly, the transition from face-to-face teaching to online learning requires a realistic (and large) allocation of resources. high structure requires a lot of up-front investment (laurillard, 1993), whereas high dialogue needs this to be ongoing. transactional distance in a blended learning environment 173 secondly, transforming teaching practice in the radical way required to implement a successful online course requires a thorough but sensitive approach to educating tutors as well as students. thirdly, in the absence of tacit social cues, weaknesses in pedagogy are accentuated in an online environment. those social cues are significant: ‘efficient communication relies not on how much can be said, but on how much can be unsaid’ (seely brown & duguid, 2000). to help overcome this, the authors believe that pedagogic theories should be explicitly applied as part of the strategic underpinning of a course. finally, maintaining regular face-to-face contact throughout the module encourages avoidance of online discussion. an alternative approach, using concentrated face-to-face periods at greater intervals, is currently being considered. despite the setbacks, the authors still believe in the importance of the principal strategy of building a successful learning community. if our aim is to produce independent, thinking and questioning graduates, we must be prepared to cultivate and nurture a collaborative online community in which geographical distance becomes less relevant as transactional distance shrinks. this paper has largely dwelt on the things that have gone wrong. it has made little mention of the fact that those students who remain are highly effective learners, achieving high commendations from internal and external examiners alike. it has said little of the tutors’ extraordinary dedication and care for students, nor of their high levels of skill and expertise that have been brought to bear on the course, nor the close-knit teams that make it all possible. the complex system that we recognize as a course is made of the interactions of its own parts and with other systems, including those of which it is a constituent. like many complex systems, it learns and develops with an internal dynamic very different from the sum of its components, not always in the ways that its leaders expect or intend. the movement from structure to dialogue and back again is one that can be strongly influenced, but seldom fully controlled. that control arises at least in part from the systemic interactions within the system itself. despite the inadequate recruitment, the high student drop-out rate, the rapid staff turnover, the unplanned disasters and the unintentional consequences of intentional change, the esystems course is in many ways a successful system. it has strong internal cohesion, yet it is fuelled by diversity. in adversity it has displayed an adaptive dynamic and continues to evolve, despite as much as because of those who try to control it. references bruner, j.s. (1966) toward a theory of instruction (cambridge, ma, the belknap press of harvard university press). brusilovsky, p. (2001) adaptive hypermedia, user modeling and user adapted interaction, 11(10), 87–110. dron, j. (2004) termites in the schoolhouse: stigmergy and transactional distance in an e-learning environment, paper presented to edmedia 2004 (forthcoming), lugano, switzerland. gagne, r. (1985) the conditions of learning (4th edn) (new york, holt, rhinehart & winston). 174 j. dron et al. garrison, r. (2000) theoretical challenges for distance education in the 21st century: a shift from structural to transactional issues, international review of research in open and distance learning, 1(1), 1–17. holt, j. (1977) instead of education: ways to help people do things better (harmondsworth, penguin). illich, i. (1971) deschooling society (new york, harper & row) kolb, d. a. (1984) experiential learning (englewood cliffs, nj, prentice hall). laurillard, d. (1993) rethinking university teachinga framework for the effective use of educational technology (london, routledge). moore, m. g. & kearsley, g. (1996) distance education: a systems view (belmont, wadsworth). peters, o. (2000) the transformation of the university into an institution of independent learning, in: t. evans & d. nation (eds) reflections on creating educational technologies (london, kogan page), 10–23. preece, j. (2000) online communities: designing usability, supporting sociability (chichester, wiley). saba, f. & shearer, r. l. (1994) verifying key theoretical concepts in a dynamic model of distance education, the american journal of distance education, 8(1), 36–59. salmon, g. (2000) e-moderating: the key to teaching and learning online (london, kogan page). seely brown, j. & duguid, p. (2000) the social life of information (boston, ma, harvard business school press). senge, p. m. (1993) the fifth discipline—the art and practice of the learning organisation (chatham, century business). vaill, p. (1996) learning as a way of being: strategies for survival in a world of permanent white water (san francisco, ca, jossey-bass). vygotsky, l. s. (1978) mind in society (london, harvard university press). wedemeyer, c. a. (1971) independent study, in: r. deighton (ed.) encyclopedia of education, vol. iv (new york, macmillan), 548–557. wenger, e. (1998) communities of practice: learning, meaning and identity (new york, cambridge university press). whitehead, a. n. (1929) the aims of education and other essays (new york, the free press). figure 1. relationship between dialogue and structure transactional distance in a blended learning environment jon dron*,a, catharine seidelb & gabrielle littenb figure 2. relationship of delivery to development time for structured/dialogic delivery modes gender issues in computer-supported learning cathy gunn,* sheila french,** hamish mcleod*** mae mcsporran**** and gráinne conole***** *university of auckland, **manchester metropolitan university, ***university of edinburgh, ****unitec, *****university of bristol email: ca.gunn@auckland.ac.nz contemporary research identifies significant gender-related differences in performance and interaction style in computer-supported learning (csl) environments. evidence suggests that initial perceptions of these environments as democratic and offering equal opportunities to all students were flawed because interactions that take place through electronic channels lose none of the sociocultural complexity or gender imbalance that already exists within society. this paper presents a summary of gender-related issues identified by international research and academic practice together with the opinions expressed by participants in a discussion forum staged at alt-c in 2001. two main questions were addressed during the conference forum. firstly, if computer access and literacy levels are assumed to be equalizing as the literature suggests, how can educational designers using csl technologies best serve all student groups? secondly, does the existence of gender-based differences in behaviour and interaction style in csl environments mean that any student group is disadvantaged? the paper concludes with suggestions about how educational designers might increase the flexibility of csl courses to offer equal opportunities to all students. a number of issues for further research are also identified. introduction in this paper, computer-supported learning (csl) environments are defined as educational • settings where some or all communications, learning activities and administrative tasks take place through the medium of computer technology. while much research in this area focuses specifically on computer-mediated communication (cmc), we adopt a broader 32 alt-j volume 10 number / definition that also includes the use of multimedia courseware, web resources and course management systems such as webct or blackboard. an emerging term in the australasian tertiary sector for this type of learning environment is mixed-mode education (mcnaught, 2001). the critical factors are that students work in a variety of ways with computer technology and exercise a degree of autonomy that is not commonly found in lecture-based courses. gender-based differences in performance and interaction style in csl environments have been recognized as an important focus for research (herring, 1993; weinman and cain, 1999; richardson and french, 2001). there are, however, conflicting views about the nature and impact of these differences and questions as to whether some student groups are disadvantaged by limited access and technology literacy remain inconclusively answered. two citations from published sources (american association of university women educational foundation, 1998; kleinfeld, 1998) illustrate the point of conflict: girls are under-represented and lower achievers in math, science and technology subjects. girls get higher grades, do better in standard tests and go on to achieve higher qualifications. the critical point here is that exactly the same data were used to support both these arguments, so clearly the matter is open to interpretation. most early research (circa 1990) in this area suggested that women and girls had less opportunity and ability to succeed in using technology for educational purposes (mcmahon, 1994; weinman and cain, 1999). more recently it has been found that technology access and literacy are disappearing problems and that young male students may be the group with most consistently low achievement levels in csl (siann, 1997; kleinfeld, 1998). evidence also suggests that theoretical conceptions of csl environments as democratic and offering equal opportunities are flawed because social and educational interaction that takes place through electronic channels loses none of the sociocultural complexity or gender imbalance that exists in more traditional learning environments (yates, 2001). research findings from the 1990s are fairly consistent in identifying gendertypical differences in interaction styles, and in approaches to and uses of csl technology. many studies conclude that, as well as having less access and ability to use technology, women are further disadvantaged by dominant and sometimes even aggressive male behaviour in cmc environments. however, deeper analysis of the situation raises questions about whether the existence of such differences actually supports the common conclusion that women are disadvantaged or simply that they use technology in different ways. this paper presents the collective experience of a small group of academics with experience of csl in an international, though predominantly western, world context, and summarizes gender-related issues identified by contemporary research and teaching practice. the case presented here is that research findings do not support the conception that women are disadvantaged. with new educational methods and the associated forms of interaction, we may also need to look for new ways of considering identity and binary concepts such as gender. 33 cathy gunn et al gender issues in computer-supported learning a collaborative investigation this exploration of gender issues in csl began as a collaborative venture to present a panel discussion session at the association of learning technology conference (alt-c) in the uk in september 2001. membership of the panel was invited on a voluntary basis and participation confirmed because of active research interests in the focus area. for the discussion session, each panel member presented a brief summary of their experience and supporting research, then concluded with a question about a particular aspect of gender difference and the implications for learning design. these questions provided the focus for small group discussions that allowed all participants to contribute their opinions and experience. the presenters' research summaries and the issues reported back by the groups form the basis of the case now presented. there is a particular focus on access and literacy issues, issues of student behaviour and cultural issues. access and literacy issues early csl research found that women were disadvantaged by lower levels of computer literacy, access to technology and confidence in its use, and that girls were both underrepresented and lower achievers in maths, science and technology subjects at school (weinman and cain, 1999). these findings were open to challenge at the time (kleinfeld, 1998), and more recent studies suggest that this is, to some extent, a disappearing problem, so that it should now be possible to examine differences that persist when imbalances due to access and literacy levels are removed. a ten-year survey of it literacy at university entrance data have been gathered about general levels of computer literacy among entry-level undergraduate students at the university of edinburgh over the past ten years. the method of collection is a brief, self-report questionnaire administered through the enrolment process, and thus relatively high response rates (60-80 per cent of intake) have been obtained. the survey is believed to provide a representative example of how levels of access, attitudes and expectations about computer use have shifted over the years. in the early years of the survey, gender was found to be a powerful predictor of the responses that students gave. in all areas of the questionnaire, gender differences were present and highly significant. women reported themselves as less likely to own computing equipment, believed themselves to be less experienced than their male colleagues in itrelated skills, and were generally less positive in the attitudes they expressed towards the importance and relevance of it to their academic studies and future careers. in the early 1990s, some of these differences were clearly related to patterns of recruitment into the various faculties of the university, and the responses were in some measure based on 'genuine' differences in the relevance of it to academic disciplines. for example, women were more likely to be joining the faculties of arts and social sciences than science and engineering. rightly or wrongly, at that time skills in computing and it were widely seen as being more relevant to the sciences than the humanities. over the passage of the decade, patterns of enrolment within the faculties and attitudes to it have changed in interesting ways. while gender differences have not disappeared entirely, a definite shift is discernible. questions about ownership of computing equipment in the 1999 survey (n = 4360 respondents, 75 per cent of intake) showed that women were still less likely than men to 34 aur-j volume 10 number i own a computer (see figure 1). the data show that ownership of a personal computer is still likely to be driven, to a significant extent, by its use for the playing of games, and this is still more likely to be the preserve of men than women. the most striking change in gender difference over the period can be seen in the responses to two questions that have been asked in identical form since the early days of the survey. newly arriving undergraduates are asked to speculate about the frequency with which they will be expected to make use of computing technologies, and the importance which these technologies will have for their studies. in the early 1990s these questions provoked marked and consistent differences in response between male and female students. over the decade these differences have gradually and completely disappeared (see figure 2). these questions call for objective judgements by the students about the importance and relevance of the technologies rather than asking them to rate their personal relationship with the technologies. male and female students have reached a point of agreement about the importance of computer technology in their work and study which augurs well for equity in applications of computers in teaching and learning. figure i: ownership of, access to, and plans to buy, computing equipment for studies computer use patterns over the period show a marked upturn in the number of students expressing themselves as 'very confident' in their ability to use electronic mail and webbrowsing applications. this may be explained by the changing significance of these two areas and the gender difference in confidence in these areas shows a closing gap between male and female students. the proportional increase in those describing themselves as 'very confident' proved to be greater among the women than among the men. ——— cathy gunn et a\ gender issues in computer-supported learning figure 2: percentage of the group responding either'important'or'vital'to the question 'how important do you think that computers will be to your university studies?" it may be argued that women's expressed interests in computers are increasingly positive as a result of the technology's increasing pragmatic significance. this point is summed up in a recent article in the new york times: grown-ups' angst over digital gender gap looks quaintly irrelevant now that teenage girls are addicted to instant messaging and the majority of internet users in the states are female. girls had no trouble adapting to computers once the machines did something that interested them. (new york times, 5 august 2001) the assumption that gender-based inequity in terms of access and computer literacy is a disappearing problem is borne out by the findings of this study along with other research and reports from various sources across the western world. the focus of attention now turns to the differences that persist in this 'more equal' environment and possible interpretations of their impact. disadvantaged or just different? once male and female students are more or less equal in terms of access and literacy, experience shows that gender-based social relationships, interaction styles and inequities that exist in traditional learning situations correspond fairly closely to those found in csl environments (herring, 1993). these evidence-based assumptions contradict more optimistic claims that technology is gender-neutral and that it provides a democratic and equal environment. 36 alt-j volume 10 number i a number of researchers have identified common differences in the behaviour of male and female students in csl environments (richardson and french, 2001; we, 1993; everts, 1998; barrett and lally, 1999). these differences include self-reported levels of confidence in ability to work successfully with technology, use of support systems and patterns of interaction. at a deeper level of analysis, studies of dialogue in cmc consistently reveal gender differences in styles of participation and contribution. generally speaking, women talk less, contribute less frequently, do not receive positive feedback to their contributions and do not appeal to the same sources of support as their male counterparts (richardson and french, 2001; herring, 1993). however, there is some debate over whether these issues constitute disadvantage or simply differences in interaction styles. there is evidence of male behaviour patterns that could be described as dominant in discussion areas (herring, 1994) and a notable tendency for men to espouse more confidence in their general capabilities in csl environments than do women. however, there is no evidence that this leads to better performance or learning outcomes and some research has found the opposite to be true (richardson and french, 2001; mcsporran and young, 2001). two case studies will now be presented in order to demonstrate how similar courses offered in different locations, the uk and new zealand, reached this conclusion. the case studies involve tertiary-level it modules with a significant degree of flexibility in study mode and method because of the mixed-mode teaching approach of csl environments. case study i:electronic information services this case study involves a cohort of information technology (it) undergraduates taking a first-year module called 'electronic information services' (eis) offered by the information systems institute at the university of salford in 1999 (see richardson and french, 2001 for a fuller report). whilst the course materials were presented mainly online, students attended some lectures and had access to paper-based materials. from a class of 140, 70 per cent of the students were male. the course designers were well aware of gender differences in attitudes to technology. recent research showed a decline in women taking it subjects at university (siann, 1997), and that the it industry in the uk was male-orientated with women representing just 22 per cent of those in the profession (ncet, 1996). the claims that men have the monopoly on technology (wajcman, 1991) and that women may even be excluded from its use by virtue of its design and function (griffiths, 1985) are important issues to confront when using technology in learning. given this background there was some concern that women might even shun the use of computers in learning. the research process the study was carried out using formative and summative questionnaires to collect quantitative and qualitative data. a number of meetings with focus groups also took place. further information was gleaned from the experiences of the lecturer and notes kept in the form of a diary. the research focused on gender issues in the following key areas: flexibility and access the study showed that all students appreciated the flexibility of the module and liked to be able to work at their own pace. all students used computers at the university. most also had access to computers in the home, although females reported more problems with access such as having to share the computer with other family members or friends. several 37 cathy gunn et al gender issues in computer-supported learning females reported that they did not have priority access to the computer in the home, whereas none of the males reported this. kirkup and abbot (1997) found that female students regarded a computer in the home as a family resource and did not see their studies as a reason for priority access. this situation is common to other studies and may warrant further investigation. confidence all students used a mix of online and paper-based materials. whereas males were very confident that they would enjoy using the online materials, several women stated they were apprehensive about using the materials and about their overall ability in the technical aspects of the course, that is, designing web pages. approaches to learning gender differences emerged in learning. the female students spent much more time preparing for the module. this could be because of their general lack of confidence. the frequency with which students studied at home differed according to gender; whilst most males reported mainly working between the hours of 4 and 8 p.m., the majority of women studied later in the evening, fitting study around other commitments such as family. support gender differences emerged in the support mechanisms utilized by students. women mainly emailed the lecturer and friends whilst males reported contact with their peers at university as well. no women seemed to use this method of support and this raised the issue of whether this type of learning isolated female students. there is conflicting evidence on this subject. some research has found that women, especially those with family commitments, embrace the flexibility in learning schedules and attendance requirements which online learning brings (von prummer, 1994). parallel research has produced conflicting evidence suggesting that women are 'social learners' and prefer traditional learning over the need to adapt to another imposed model such as online learning (von prummer, 1994; kirkup and abbot, 1997). the researchers in this case felt that this could be another important issue for further study. learning outcomes the confidence that the males expressed and the apprehension of many of the women were not substantially reflected in the grades they achieved. the males achieved slightly higher grades than the women as a whole, on what might be deemed the more technical aspect of the module (designing web pages). overall, however, the women's marks were concentrated at the top end of the grades for all of the .assessed work. implications for learning with technology from this research it can be concluded that the online learning experience may well be different for males and females. the females in this study seemed to have less confidence than their male counterparts and less guaranteed access to technology. in order to ensure equality, the needs are to improve access, make sure learners are not isolated and find ways of instilling confidence in some of the female students. social issues such as ownership of technology are at the forefront of research and need further discussion. the issues are complex and, as the other studies in this paper demonstrate, there is much conflicting evidence. 38 an-] volume 10 number i case study 2: interactive web design research conducted in a similar context in a new zealand tertiary institution found that female students performed consistently better than males in online courses (mcsporran, dewstow and young, 1999; mcsporran and young, 2001). a course in interactive web design enrols 70-100 students per semester and options are available to complete the course entirely online or to attend weekly classroom sessions in addition to independent study. the course structure and assessment have remained more or less the same over the three-year period of the study and involve: • a balance of theory and practical exercises; • self-test quizzes; • access to readings, urls and faqs; • constant feedback from tutors; • a research project requiring team collaboration and report preparation; • a website development assignment and a final exam. knowledge of contemporary research findings together with the benefits of past experience provided the basis for initial assumptions that female students might perform better on the research assignment because it involved communication and teamwork, while males may do better on the technical task of website development. however, these assumptions proved to be unfounded as the female students performed consistently better than males on both tasks. analysis of course-entry questionnaires showed that females rated themselves less competent with internet skills and experience at the outset, a situation that is reflected in the findings of other research (yates, 1992). analysis of participation in online activities recorded by the course management system server then raised the possibility that male students may be overconfident in their ability and think they do not have to put in as much effort to meet the course requirements (see table 1). this analysis revealed the following characteristics of male student behaviour: • greater non-submission of a complete assignment; • greater non-completion of all parts of each assignment; • lower number of online quiz attempts; • lower course website page views; • lower bulletin board participation. although these figures are relatively crude measures of cause and effect regarding performance levels, the tentative conclusion is that the course works well for women and mature students, while the younger male students appear to need more motivation and discipline than weekly classroom sessions currently provide. the reasons for better performance of women, and in this case mature students, are believed to include stronger motivation to succeed, greater ability to work independently and self-manage multi-tasking lives. these case-specific findings are further supported by a canadian study that reports opportunities for communication and participation in course-related activities being 39 cathy gunn et al gender issues in computer-supported learning semester female % male% quiz attempts n o of quizzes attempted 11.4 7.4 never attempted a quiz 4.0 20.0 bulletin board participation posts read 38.5 16.2 posts made 4.3 2.6 female (n) male (n) total page views 416.9 249.2table i: system data of male and female student participation in csl elements enhanced for women when they do not have to enter 'inhospitable' environments such as computer labs (chegwidden, 2000). it seems then, that although a significant body of literature suggests that suitable educational models are currently not widely used to promote women-friendly csl environments, some situations are not particularly male-friendly either. the challenge is to find solutions that offer equal opportunities to all students. the questions that this raises are what characteristics make learning environments friendly to both male and female students, and how can opportunities to participate and succeed be enhanced for all. before presenting the solutions proposed by the discussion groups, the impact of culture on gender differences is considered worthy of mention. cultural impact on gender relations a review of international literature (notably from the us, europe and australasia) on gender, identity and performance issues in csl environments raises questions about how far the differences are culturally determined (barrett and lally, 1999; eubanks, 2000). some variation may be attributable to the style of report or the nature of the gender imbalance in particular situations. however, it may also be the case that culturally based metaphors significantly influence perceptions of technology environments, their suitability for different groups, the values they reflect and the behaviour that is acceptable within their confines. the term 'cyberspace' itself has some significance in a discussion of cultural influence on gender issues. the concept of cyberspace has its origins in a science fiction work, neuromancer, published by gibson in 1984. gibson's cyberspace is a female region used and controlled by men; it is also highly sexualized. women in this region are simply resources, not active agents or users of technology. so the very metaphor that is used to define the online 'space' in which students now interact has its roots in a very unequal society. in contemporary cultures, there do appear to be differences that could be attributed to the sociocultural contexts in which educational experiences are studied and reported. 40 alt-j volume 10 number i published studies from the southern us identify dominant and sometimes even deviant behaviour among males (herring, 1994; blumenstyk, 1997; blum, 1999 herring, 1999). for example, blum found that males talk more and interpreted this as controlling behaviour, noted that they used different tones, ones of arrogance and certainty that she interpreted as intimidating to women, and asked fewer questions which implied that they had more knowledge. she concluded that interactions in cmc reflect those of face-to-face culture. although the findings of this study could be interpreted quite differently and the methodology questioned because of the small sample used for a study that claims generalizable results, the fact that the author made these interpretations may say something about the culture in which she works. herring reported construction and maintenance of asymmetrical gender/power dynamics and exposed blatant double standards in freedom of speech 'rhetoric' (herring, 1999). moving to a different cultural environment, barrett and lally (1999) present a case study from the uk in which male and female students played notably different roles: • males talk more; • meta/cognitive content was the same; • social content was higher for males; • interaction was higher for females; • only females had persistent access problems. although some findings are common with the us studies, the researchers do not draw conclusions about women being disadvantaged. rather they seek to explore the meaning of these differences in terms of perceived user friendliness of the environment and its impact on student performance. in the other cultural environments covered by this study, requests for participation in this research from australian academics resulted in expression of the opinion that 'gender is simply not an issue worthy of attention in this country'. in new zealand, the only research available suggests that women are performing better than their male counterparts, although this may reflect a lack of investigation (gunn and barnett, 2001) and published research rather than a difference in what appears to be the norm elsewhere. the conclusion here is that broader aspects of local culture may indeed be reflected in the interaction styles of students in csl environments. to a great extent, these issues go beyond the scope of influence of educational designers although they should be afforded due consideration in learning design. conclusions and recommendations consensus reached by the discussion groups at alt-c was that the differences identified by the presenters and supported by contemporary research exist in most western world educational environments and present significant challenges to educators using new technologies in teaching. however, a range of interpretations of both the underlying causes and the implications of these differences is possible. the critical issue is to develop widely applicable solutions rather than conducting more in-depth research into the causes. 41 cathy gunn et a\ gender issues in computer-supported learning it was agreed that inequality in levels of access and computer literacy among student populations are disappearing problems, and that male and female users generally take different approaches to the use of technology, that is, exploratory and developmental versus practical and instrumental. this inherent difference in approach sparked a debate about whether some aspects of gender are in fact attributable to sociocultural construction or to biological difference. regardless of the outcome of that particular debate, the implications for csl designers are that they have to accommodate a range of approaches to the use of technology and design learning environments that do not implicitly favour any particular approach. this is considered to require a degree of flexibility that many online environments currently do not include. for example, educationally focused games and challenges may be more appealing to a typically competitive male orientation, while communicative and collaborative activities may appeal more to typical female styles. the usefulness of these types of activities will depend on suitability to discipline and subjectrelated learning objectives as well as to gender-typical approaches. a critical issue in encouraging equal participation by both genders may be the levels and types of support provided for learners. for example, good moderation in discussions can help to ensure equal and appropriate levels of participation. inclusion of opportunities for self-assessment during a course can support self-monitoring of performance to ensure that espoused confidence is backed up by the ability to attain high performance levels with or without specifically requested support. the issue of confidence is perceived to be a particular challenge for male students who are considered less likely to ask for assistance as well as less likely to recognize the need to participate in all course-related activities and preparations for assessment. it was also noted, however, that in a context characterized by rapid change in so many areas technology, educational theories, design of learning environments, student demographics and professional requirements these issues are dynamic and in need of constant reappraisal. issues of gender should not be treated in isolation from other factors such as learning styles, educational backgrounds and capabilities. a more general point was also raised about computer-based course management systems in their current incarnation being predominantly designed to support didactic rather than constructive learning experiences. this is one more issue that needs to be addressed in the design of learning environments that offer equal opportunities to all students. it was generally concluded that gender is just one of many areas of focus for research and development of effective csl environments and activities. references american association of university women educational foundation (1998), gender gaps: where schools still fail our children, washington dc: aauwef. barrett, e. and lally, v. (1999), 'gender differences in an on-line learning environment', journal of computer assisted learning, 15, 48-60. blum, k. d. (1999), 'gender differences in asynchronous learning in higher education: learning styles, participation barriers and communication patterns', journal of asynchronous learning networks, 3 (1), 46-66. available from: www.aln.org/alnweb/ journal/jaln-vol3issuel.htm 42 alt-j volume 10 number i blumenstyk, g. (1997), 'a feminist scholar questions how women fare in distance education', the chronicle of higher education. available from: http://chronicle. com/colloquy/97/distance/background.htm. chegwidden, p. (2000), feminist pedagogy and the laptop computer. women, work and computerization: charting a course to the future, vancouver, bc: kluwer academic publishers, 293-9. eubanks, v. (2000), 'paradigms and perversions: a woman's place in cyberspace', pcsr newsletter, 18 (1). available from: http://www.cpsr.org/publications/news/etters/issues/2000/ winter2000/eubanks.html everts, s. (1998), gender and technology: empowering women, engendering development, london and new york: zed books. gibson, j. (1984), neuromancer, new york: ace books. gunn, c. and barnett, j. (2001), 'on-line learning: a quality experience', in p. roberts and m. chambers (eds), digital developments in higher education, los angeles, ca: taylor graham, 139-60. griffiths, d. (1985), 'the exclusion of women from technology', in w. faulkner and e. arnold (eds), smothered by invention, london: pluto press, 51-71. herring, s. (1999), 'the rhetorical dynamics of gender harassment on-line', the information society, 15 (3), 151-67. herring, s. (1994), 'gender differences in computer mediated communication: bringing familiar baggage to the new frontier', in making the net* work*: is there a z39.50 in gender communication?, miami: american library association. available from: http://cpsr.org/cpsr/gender/herring.txt. herring, s. (1993), 'gender and democracy in computer mediated communication', electronic journal of communication, 3 (2), 1-17. available from: http://www.cios.org/ www/ejc/v3n293.htm kirkup, g. and abbot, j. (1997), the gender gap. a gender analysis of the 1996 computing access survey, plum paper number 80. programme on learner use of media, milton keynes: open university. kleinfeld, j. (1998), the myth that schools shortchange girls: social science in the service of deception, washington dc: the women's freedom network. mcmahon, p. (1994), strategies to facilitate student computer usage: are attitudes the answer?, survey report, belfast: queen's university. mcnaught, c. (2001), 'quality assurance for online courses: from policy to process to improvement?', paper presented at the ascilite 2001: meeting at the crossroads, melbourne, vic. mcsporran, m., dewstow, r. and young, s. (1999), who wants to learn on-line? identifying our flexible learners, proceedings of the world conference on educational multimedia, hypermedia and telecommunications, washington: association for the advancement of computing in education. 43 cathy gunn et al gender issues in computer-supported learning mcsporran, m. and young, s. (2001), 'does gender matter in online learning?', association of learning technology journal, 9 (2), 3-11. ncet (1996), gender and it, coventry: ncet. richardson, h. and french, s. (2001), 'education on-line: what's in it for women?', in women, work and computerization: charting a course to the future, vancouver bc: kluwer academic publishers. siann, g. (1997), 'we can, we don't want to: factors influencing women's participation in computing', in r. lander and a. adam (eds), women in computing, exeter: intellect books. von prummer, c. (1994), 'women-friendly perspectives in distance education', open learning, 9 (1), 3-12. wajcman, j. (1991), feminism confronts technology, cambridge: polity press. we, g. (1993), cross-gender communication in cyberspace, department of communication, burnaby, bc: simon fraser university. weinman, j. and cain, l. (1999), 'technology the new gender gap', technos, 8 (1), 9-12. yates, s. j. (1992), gender and computer-mediated communication: an analysis of dt200 in 1990, cite report no 158, milton keynes: open university. yates, s. j. (2001), 'gender, language and cmc for education', learning and instruction, 11, 21-34. 44 almost as helpful as good theory: some conceptual possibilities for the online classroom mike davis* and kate denning** *learningtechnology research institute, university of north london **research and graduate school of education, university of manchester email: mike.davis@unl.ac.uk interest and activity in the use of c&it in higher education is growing, and while there is effort to understand the complexity of the transition to virtual space, aspects of development, particularly clarity about the nature of the learning community, may only be lightly theorized. based on an ongoing action research study involving postgraduate students studying in the uk and usa, this paper will identify some theoretical roots and derive from these six conceptual areas that seem to the authors to have relevance and significance for behaviour online. an exploration of these forms the basis for a twodimensional model which can account for what happens when groups come together to learn in cyberspace. in depicting this model, there is acknowledgement of the existence of third and fourth dimensions at work. however, the explanatory power of taking these extra dimensions into account is beyond the scope of the analysis thus far. introduction current interest in the use of communications and information technology (c&it) in higher education and other settings is at a high level (for example, framework v: towards a user-friendly information society; the esrc programme paccit) and while universities in response to the dearing report begin to exercise effort in restructuring course delivery methods to take advantage of the technology, there is a danger that the issue of the changes in the social dynamics that are a direct consequence of the transition from actual space to virtual space will be overlooked. some work has been undertaken in this area (notably rheingold, 1991; cook, 1995; mcconnell, hardy and hodgson, 1996) but there is need for additional research if we are to identify a clearer sense of how effective online learning might manifest itself. we agree with jones (1998), however, that the technology presents considerable challenges in establishing and maintaining communities: 64 alt-j volume 9 number 2 the learning process may bring people together insofar as such learning is often collaborative, but it is equally as often frustrating and off-putting. (jones, 1998: 8) this paper will explore an approach to postgraduate education which depends on online collaborative construction of knowledge, drawing on students' past experience and new understanding gained on the course. there are other models of practice employing computer-mediated communication (cmc) in postgraduate teaching, but it has been decided to limit the scope of this paper to action science groups (argyris, putnam and smith, 1986) because the nature of the work undertaken by these small groups is intense and personal, providing fertile ground for the creation of the learning community which is the focus of the research. action science has been described as 'the science of interpersonal action' (argyris, 1993) and is an approach to personal and organizational development. action science employs a number of methods by which participants examine their work-based defensive routines and look to change their underlying theories-in-use to produce more positive and effective ones. participants of all six cohorts under consideration here wrote case studies which were then interrogated online by their small group and the two facilitators. a case study in this context is a written record of a remembered conversation about which the case writer feels dissatisfied for some reason. it was expected that learning will occur at two levels. firstly, each case writer has the opportunity to gain insight into his or her own defensive routines as played out in the original dialogue. secondly, all group members learn the specific questioning skills required of an action scientist, enhance their ability to spot underlying inconsistencies in a conversation and gain practice in rewriting statements in a new format which is more likely to lead to mutual learning and win-win outcomes. it is felt that each group within the six cohorts that represents our data set has demonstrated some vivid characteristics. we consider that this rich environment is a source of real insights into the challenges offered by cmc in creating learning communities and this has enabled us to develop a model that goes some way to describing the characteristics of these communities. there have been almost fifty years of research and theorizing about the way in which groups of various sizes function in a variety of face-to-face settings, including some in higher education classrooms and lecture theatres. research into cmc has a much shorter history. an examination of the literature persuades us that it is easy to make incorrect assumptions about the characteristics of the online classroom. cmc is often seen simply as another process which can be incorporated into existing thinking and practice, rather than one requiring a shift in conceptualization: about teaching and learning; about groups; and about the effect of technology on their structure and function. as fernback noted, we know already that many of the assumptions we hold about the negotiation and formation of social relationships, and particularly about community, do not seem to apply in the complex realm of cmc. (fernback, 1999: 205) not surprisingly, the rhetoric and some of the practice of teaching and learning see cmc as a potential pedagogy for higher education as the capacity of the medium to deliver course materials and to generate interactivity between lecturers and students and among the student body becomes apparent. while the move towards the virtual campus cannot be ignored, there is the need to identify some of its characteristics and the theories underlying 65 mike davis and kate denning almost as helpful as good theory: some conceptual possibilities for the online classroom the practice. the purpose of this paper, therefore, is to explore some of the theory that contributes towards an effective understanding of what takes place when groups of students meet and learn online. in common with standard practice, students' names have been changed. their contributions to discussion, however, are uncorrected. some theoretical possibilities while not believing that activity in cyberspace has direct analogies with face-to-face experiences, our thinking is shaped by attention to two main theoretical sources: group dynamics both from a sociological (miller, 1993) and a psychoanalytical perspective (bion, 1961) and situated learning, particularly its focus on notions of communities of practice, socially constructed knowledge and authentic activity (mclellan, 1996). in the limited space we have available for this paper, we will go no further than claiming a relationship between the components of situated learning identified by mclellan, onto which we have mapped online experience of working with action science. key component online experience of action science stories the case studies reflection reflecting on and in action through case study interrogation and rewrites cognitive apprenticeship working with 'expert' facilitators collaboration working collaboratively to reveal defensive routines and to seek win-win alternatives coaching deliberate teaching and modelling of good action science interrogation multiple practice successive cases during which time participants become more skilled and confident articulation of learning skills the construction of a metadialogue to inform a group's practice and achievement technology making appropriate use of the available technology. table i: relationship between key components of situated learning and online experiences of action science our thinking, dominated at the time by consideration of group dynamics, led us to reflect on six conceptual areas which seem to be powerful ingredients in the online classroom: • social organization • orientation towards learning • orientation towards task and/or tutor • group work modality • emotional climate • group response to challenge. 66 alt-j volume 9 number 2 social organization while communication online can be task-based there is an expectation that other, non-task but socially essential communication will contribute to the growth and sustenance of the community. sociolinguists call this phatic communication (stubbs, 1983), and it is seen as essential to maintain effective social interaction. it has been argued (feenberg, 1989) that cmc is poor at fulfilling these phatic functions. he writes: 'ah such phatic signs are bypassed in computer conferencing. even standard codes for opening and closing conversations are discarded' (feenberg, 1989: 22). this is particularly problematic in online communication because there is only the single, textual cue available for inspection in contrast with the multiple cues that exist in face-toface communication. evidence from previous studies in cmc (davis, 1997; davis and holt, 1998) suggests that this form of interchange may not arise spontaneously and its absence can contribute towards social isolation and withdrawal. there are models that suggest that remediation of this is possible using a number of straightforward strategies, such as sending private email messages, directing comments at individuals and modelling responsiveness (harasim, hiltz, teles and turoff, 1995; midoro, 1999). the research is key both to identify and to refine appropriate facilitation strategies to maximize effective learner participation and interdependence. positive outcomes of well-timed and wellcrafted tutor interventions include independence and community. however, whilst independence can be a rewarding outcome of a learning experience, it can also work against effective community membership. orientation towards learning orientation towards learning has two related continua: individual >cooperative >collaborative dependent >independent >interdependent collaborative, interdependent learning, which is that discovered in a learning community, can be understood 'as a distributed, ongoing social process, where evidence that learning is occurring or has occurred must be found in understanding the ways in which people collaboratively do learning and do recognize learning as having occurred' (jordan, 1996: 42). collaborative learning occurs when participants mutually engage 'in a coordinated effort to solve [a] problem together' (roschelle and behrend, 1995: 70). whipple describes the benefits of collaborative learning: collaboration results in a level of knowledge within the group that is greater than the sum of the knowledge of the individual participants. collaborative activities lead to emergent knowledge, which is the result of interaction between (not summation of) the understandings of those who contribute to its formation. (whipple, 1987: 5). in contrast, hiltz, describing the software they use, writes: "this assignment was carried out using the 'activity branch' software. in a response branch, each student must answer the question before being able to read the answer of others' (hiltz, 1994: 59). this kind of structure may work counter to attempts at building a learning community. whilst the focus is on ensuring that the individual learner thinks, then an interactive 67 mike davis and kate denning almost as helpful as good theory: some conceptual possibilities for the online classroom building of ideas is absent and it becomes unlikely that collaborative learning can take place or a learning community emerge. orientation towards task/tutor a classic depiction (bennis and shepard, 1956) of group life is that it can be divided into two main phases: dependence-power relations and interdependence-personal relations, and it is this that determines the orientation towards the task and the tutor. in dependence-power relations, students engage in flight from the task through avoidance and reliance on social engagement to occupy time. counterdependency a metaphorical fight with authority can also emerge. the shift into interdependence-personal relations leads to conditions where there is considerably less dependence on external authority either in the shape of the task or the tutor and the development of a shared sense of responsibility for group performance. in online communities, this can create conditions in which members feel addicted to the process. in sharp contrast to this condition is the total separation of self from others: the condition of anomie 'when members of a superficially well-organised society feel disconnected and isolated' (reber, 1995: 39). group work modality group life can also be characterized from a more psychological perspective, originally modelled by wdfred bion (bion, 1961). fundamental to bion's thinking about groups is that membership of them is part of the human condition. as he wrote: 'no individual, however isolated in time and space, can be regarded as outside a group, or lacking in active manifestations of group psychology' (bion, 1961:132). bion proposed that groups work at two levels. work groups function effectively, engage with the task and with one another and attend to the needs of the group. according to bion, however, whilst a group is operating in 'work' mode it is also capable of being subverted at any one time by one of the three basic assumptions dependence, flight/fight or pairing. a group moves into basic assumption dependency whenever it is reliant on a leader and believes that the leader will control, make decisions and rein in any passions that are too threatening to the safety of the group. a group in conflict or under pressure will often move into denial manifest as flight (running away from a difficult issue to talk about 'safer' topics) or fight (usually a verbal struggle). a group is considered to be in basic assumption pairing when two members of the group are heavily involved in a discussion and the remainder of the group is silent but attentive. it is likely that a series of pairs will emerge, each dominating the discussion for a while. basic assumption groups are thought to be mutually exclusive: for example, a group in basic assumption pairing cannot demonstrate flight/fight or dependency-type behaviour. it is possible for a group to move readily from one basic assumption to another. emotional climate all of the above conditions represent a challenge to the group and this challenge contributes towards an emotional reaction, either shared or individual. among these, we have identified indifference (real or otherwise), frustration, off-task fascination, and 68 alt-j volume 9 number 2 anxiety. the first three are counterproductive in respect of the success of the group and/or its task completion. the latter can be productive or its opposite: too high anxiety invariably leads to ineffectiveness or, in the worst case, withdrawal; too low is insufficient to drive the motor of learning. group response to challenge groups respond to challenge at different times in different ways. our experience has led us to identify four responses: groups that become hostile to the task reveal passive resistance or aggression, often by showing little interest in the activity; others deny that there is a problem when attempts are made to establish dialogue about the events unfolding. others become fascinated with membership of the group and are seduced by the social aspects of their communication. successful groups, however, engage in risk-taking: challenging other members, indeed challenging themselves, to push the margins of what is possible. discussion in an attempt to gain some insight into how these conceptual areas might inter-relate, we chose a 2 x 2 matrix (see figure 1) to characterize four archetypal groups. high (9) learning dynamics low(l) type 9.1-"i'm ok, you're ok!" 1. independence 2. co-operative 3. counterdependency 4. flight 5. frustration 6. denial type 1.1"fragmented by technologies" 1. isolation 2. dependent 3. anomie 4. flight 5. indifference 6. passive resistance type 9 . 9 "tough love" 1. community 2. collaborative 3. addiction 4. work 5. anxiety 6. risk-taking type 1.9 "summer holiday" 1. displacement 2. individual 3. avoidance 4. pairing 5. off-task fascination 6. seduction • conceptual key 1. social organization 2. orientation towards learning 3. orientation towards task/tutor 4. group work modality 5. emotional climate 6. group response to challenge l o w ( l ) group dynamics high (9) figure /: group functioning in cyberspace (davis and denning 2000) fragmented by technologies ( i . i ) a group which is low on both learning and group dynamics may have very little activity and will not be concerned about the group processes nor will it be effective in its learning objectives. members will be isolated from one another and their approach to learning, where it exists, is individual. socially, group members are isolated and their basic assumption is flight from the task and any discussion about the task. this leads to public mike davis and kate denning almost as helpful as good theory some conceptual possibilities for the online classroom indifference (despite email messages that indicate private frustration and anger) and a group strategy of passive resistance or aggression. whilst successful groups showed themselves willing to build upon each other's ideas and create new levels of understanding, other groups never gained any momentum: hilary: i am not really enjoying our group interaction. it is very slow and uninvolved and the communication levels are very low. i am finding it hard to find questions to ask. i don't know why. a willingness to avoid the task and discussion about the task is well summed up by one case-writer who commented on his own case as follows: jack: i [sic] have reviewed your input, and appreciated your interest, the questions that were asked will help me focus on the situation. since this particular casewriter only made two interventions into his own case (the average for one of the cohorts was twenty-six) this represents 50 per cent of his output and clearly the experience has made little positive impact on him and the rest of the group. summer holiday (1.9) if a group is high on group dynamics but low on learning dynamics then group members may be having a lot of fun whilst achieving little learning. here, members are displaced from normal life and they demonstrate self-interest and individuality. work is avoided and the complex notion of basic assumption pairing is acted out. in this, the group waits for a magical event to emerge from possible pairing of other participants. accordingly, they can be high on social interaction invariably manifested through social 'conversation' at the expense of work. indeed, the social is the dominant theme in this type of group and this, of course, can be very satisfying for the members and is very seductive. the following example demonstrates a group being hampered by notions of the need to be inclusive whilst at the same time struggling with the process of making decisions online. laura: since nobody is taking the initiative but everybody seems to share the view (at least this is what i make out of it) i would like to see how many of you could make it for an on line session, real time sometime this satarday [sic] or sunday afternoon. this is the time that i can make it if you think that some other time is more convinient [sic] please suggest it. this comment came in week four of seven weeks when the group had already been discussing meeting synchronously (at the same time as each other) since the first week. it seems likely that the social element was so important for this group that the thought of meeting up at a time that didn't suit all of them was unthinkable. the group continued to resist moves by various members to experiment with synchronous communication until, in week six, the following intervention from one of the tutors coincided with the most innovative member being more proactive about her desires. jack: would 7pm on tonight and on wednesday night suit everyone? kate (facilitator): i think trying to get everyone may be a mistake. all you need is the casewriter and one or two others. more is obviously great, but not essential. i sense waiting for everyone to agree could mean that yet again you fail to meet. 70 alt-j volume 9 number 2 jack: that's true, kate . . . i had forgotten that that time would not be convenient for all . . . so who will meet me on line . . . i'll be here at 7.00pm. and she was. those that met up with her clearly enjoyed 'being' together, but continued to use the time in a largely social way. although the group had managed to overcome one of its difficulties (making decisions) it was unable to work against the by now well established norm of social activity dominating the work space. i'm ok, you're ok (9.1) if a group is high on learning dynamics but low on group dynamics then members will show little concern for each other personally and will tend to work independently rather than interdependently. one group whose group strategy we have characterized as denial had the following conversation: megan: i am aware that there are a number of things i have been thinking but not saying and i wonder if this is true for others also. rod: m, i too feel that perhaps we aren't as active as we could be. but i am ok with it. here rod refused to take up the gauntlet, preferring to work in her own way which megan later described as 'bullying'. such groups are capable of acting co-operatively rather than collaboratively. in the latter, understanding and insight grow from the social construction of knowledge. in the former, it is more competitive and individual understanding and insight is the desired outcome, possibly at the expense of others' learning. inevitably, groups who find themselves in this situation demonstrate counter-dependent behaviour with frequent (although invariably unsuccessful) appeals to authority to deal with the problematic group dynamics. equally inevitably, tutors are held responsible for their failure to make the groups work more effectively and members deny their collective and individual responsibilities for the difficulties the group is experiencing. below is an extract from a group who struggled and looked to the tutors to make the interaction more productive. sue: miked, kate, correct me if i'm wrong but i thought that in order for a case to be completed, it was necessary to provide the 'interrogated' with a tiu. this didn't happen in my case. kelly: yes, s. i am also interested in getting an answer to this question, you will remember that i raised a similar concern to kate and miked last time when my case was discussed, but i did not get any satisfactory response. as for miked, he did not even bother to comment on the issue. with kate it was better because, even if she did not answer my question, she at least asked me questions in relation to the issue. here we have two group members whose primary concern is that they gain from the experience without necessarily giving to the rest of the group. both are requesting a theoryin-use from members of the group, neither were very forthcoming in giving them to others. this demonstrates a group attempting to learn individually in an environment set up for collaborative learning. it is not possible to do action science alone. if it were possible they would have done it! 71 alike davis and kate denning almost as helpful as good theory: some conceptual possibilities for the online classroom tough love (9.9) groups who manage both group dynamics and learning dynamics get as close as is possible to becoming a learning community. they behave in a collaborative manner, in particular in checking out assumptions and perceptions with other group members. kate: i'm going to be really mean now rod and assume that 'ooh that's harsh' means i hit the nail on the head there. is my assumption correct? the work of the group takes on an incredible pace and volume: students report logging in at all times of the day and night and many independently have talked of how addictive the process is. jackie: i agree about this becoming somewhat addictive. i find myself wanting to checking [sic] several times daily. the consequence of volume and intensity is that considerable insight is developed into the interpersonal processes under exploration through the action science cases. jackie: what has been very revealing for me in these 2 days has been the emotion evoked at my keyboard. i have had more than a couple of moments where a light has come on, a bell clanged, or several ahas. i've been all teared up in several of my responses tonight. i've experienced many things sitting and reflecting [...] what i find so amazing and possibly the real truth is how all of you see through my coverup. action science, as an organizational intervention, is very powerful but it can raise high levels of anxiety given that the design is to explore participants' interpersonal incompetence. frequently people act out the characteristics of their case during interrogation, and that makes necessary risk-taking. one group had a difficult first week with only three of its members contributing actively to the discussion. after several days pete (following email correspondence with one of the facilitators) posted this message: pete: i have been following but rather in what to me is an awkward position. i don't know why i suddenly chose to be an observer in this discussion... he received a number of responses from the active members letting him know how hard they had been working to keep the group moving with such low 'attendance'. one of the replies indicated a broader awareness both of the effect his silence had on the group and of the reciprocal nature of the work. jackie: i realise that it isn't always easy [sic] to get to your computer but if we can at least check in at the beginning of the cases and if we are not going to be able to make comments for whatever reasons at least inform each other of that so that we are aware of what is going on. i think this would help us all to work more effectively as a group. we are all presenting cases here and i assume that we would all like all members of the group to contribute to this otherwise how can we be a group?? this marks a clear challenge to the 'observer' stance pete had been taking if you want us to work for you, then you have to work for us. pete took up the challenge, apologized to the group and began making effective interventions. this early friction was one of the key moments for this group who went on to produce some high-quality work. 72 alt-j volume 9 number 2 as has been mentioned earlier one of the elements of the group whose orientation towards learning is collaborative is that the knowledge is constructed together by the sharing of thoughts, feelings and knowledge. this building of ideas results in higher-order insights than might be gained individually. one participant commented on how the group seemed to be building individual lines of enquiry rather than working together, which, whilst it was true, was a comment which itself was part of the following scaffold: kelly: i find karen [facilitator] has made an intervention that has set me thinking . . . jackie: just to tag on to k's response... megan: good job k you've hit a big problem on the h e a d . . . maybe one solution is . . . our model, as it stands, assigns to each of the six conceptual areas four potential states to correspond to the creation of four ideal group types. this, however, is a misrepresentation of the complexity of the model we have created, as much as anything else for neatness of exposition. what we are aware of is a third dimension, not accommodated by the 2 by 2 matrix, which can indicate possible alternative alignments of the various characteristics of behaviour and their interrelationship. this we have designated depth (the fact that a conceptual area can be manifest in a number of ways) and see it as the third dimension of the model which as yet remains in its infancy. the danger, however, here is that we fall into the trap identified by aarseth: the race is on to conquer and colonise these [learning technologies] for our existing paradigms and theories, often in the form of 'the theoretical perspective of <fill in your favourite theory/theoretician here> is clearly really a prediction/description of <fill in your favourite digital medium here>.' this method is being used with permutational efficiency throughout the fields of digital technology and critical theory, two unlikely tango partners indeed. but the combinatorial process shows no sign of exhaustion yet. (aarseth, 1999: 31) at least we are aware of this risk, and we will remind ourselves of it from time to time. conclusion so, what, if anything, can we conclude? the model depicted in figure 1 was the product of inspiration and intuition based on our iterative analysis of data collected over a three-year period and it feels as if it has some explanatory power. we plan to re-examine our data in an attempt to confirm the accuracy of the conclusions we have drawn. we then want to examine other data from other online courses that are similar in nature to see if the model, as it currently stands, is robust. then we might be able to tackle the third dimension, depth, and the, as yet unmentioned, fourth dimension of time. references aarseth, e. (1999), 'aporia and epiphany in doom and the speaking clock: the temporality of ergodic art', in m. ryan (ed.), cyberspace textuality: computer technology and literary theory, bloomington: indiana university press. argyris, c. (1993), knowledge for action, san francisco: jossey-bass. 73 mike davis and kate denning almost as helpful as good theory: some conceptual possibilities for the online classroom argyris, c., putnam, r. and smith, d. (1986), action science, san francisco: jossey-bass. bennis, w. and shepard, h. (1956), 'a theory of group development', human relations, ix (4), 415-38. bion, w. (1961), experiences in groups and other papers, london: tavistock. cook, d. (1995), 'community and computer-generated distance learning environments', new directions for adult and continuing education, 67, 33-9. davis, m. (1997), 'fragmented by technologies: a community in cyberspace', interpersonal communication and technology journal, 5 (1/2), 7-18. davis, m. and holt, m. (1998), 'havingproblems@cm.com: new ways to miss the point', innovative higher education, 22, 4, 311-27. davis, m. and denning, k. (2000), 'online learning: frontiers in the creation of learning communities', proceedings of the networked learning 2000 conference, university of lancaster. de board, r. (1978), the psychoanalysis of organisations, london: tavistock. feenberg, a. (1989), 'the written world: on the theory and practice of computer conferencing', in r. mason and a. kaye (eds.), mindweave: communication, computers and distance education, leicester: pergamon. fernback, j. (1999), 'there is a there there: notes toward a definition of cybercommunity', in s. jones (ed.), doing internet research: critical issues and methods for examining the net, london: sage, 203-20. grinberg, l., sor, d. and tabak de bianchedi, e. (1975), introduction to the work of bion (trans. a. hahn), london: maresfield library. harasim, l., hiltz, s., teles, l. and turoff, m. (1995), learning networks: a field guide to teaching and learning online, cambridge, ma: mit press. hiltz, s. (1994), the virtual classroom: learning without limits via computer networks, norwood, nj: ablex publishing corporation. jones, s. (ed.) (1998), cybersociety 2.0: revisiting computer-mediated communication and community, thousand oaks: sage. jordan, b. (1996), 'ethnographic workplace studies and cscw', in d. shapiro, m. tauber and r. traunmuller (eds.), the design of computer supported cooperative work and groupware systems, amsterdam: elsevier. mcconnell, d., hardy, v. and hodgson, v. (1996), groupwork in educational computer conferences. final report to esrc, dace, university of sheffield, http://www.shef.ac.uk/ uni/projects/csnl/. mclellan, h. (ed.) (1996), situated learning perspectives, englewood cliffs, nj: educational technology publications. miller, n. (1993), 'how the t-group changed my life: a sociological perspective on experiential group work', in d. boud, r. cohen and d. walker (eds.), using experience for learning, buckingham: srhe/open university press. 74 alt-j volume 9 number 2 midoro, v. (1999), 'modelling online education', in communications and networking in education: learning in a networked society, proceedings of ifip working groups 3.1, 3.5 and 3.6 open conference, aulanko-hameenlinna, finland, 13-18 june 1999, 239-46. reber, a. (1995), the penguin dictionary of psychology, 2nd edition, harmondsworth: penguin. rheingold, h. (1991), virtual reality, london: mandarin. roschelle, j., and behrend, s. (1995), 'the construction of shared knowledge in collaborative problem solving', in c. o'malley (ed.), computer-supported collaborative learning, berlin: springer-verlag. stubbs, m. (1983), discourse analysis: the sociolinguistic analysis of natural language, oxford: basil blackwell. whipple, w. (1987), 'collaborative learning: recognizing it when we see it', bulletin of the american association for higher education, 40, 2, 3-7. 75 calt12202.fm alt-j, research in learning technology vol. 12, no. 2, june 2004 what are the affordances of information and communication technologies? grainne conole* & martin dyke university of southampton, uk taylor and francis ltdcalt12202.sgm10.1080/0968776042000216183alt-t research in learing technology0968-7769 (print)/1741-1629 (online)original article2004taylor and francis ltd1220000002004grainneconoleschool of educationuniversity of southamptonhighfieldsouthamptonso17 1bjukg.c.conole@soton.ac.uk the paper examines the notion that information and communication technologies (ict) have affordances that epitomize the features of our late modern age (giddens, 1991) and explores whether these affordances (salomon, 1993, p. 51) can be used to facilitate particular approaches to educational practice. it argues that a clear articulation of these affordances would enable us to understand how these technologies can be most effectively used to support learning and teaching. we believe that any one affordance can be considered to have both positive and negative connotations and the paper draws on social and educational theory to provide an initial taxonomy of these affordances. introduction there has been a rapid growth in online learning environments and associated tools to support learning and research. these include communication systems (email, discussion boards, synchronous chat), authoring and assessment tools, as well as integrated learning environments such as blackboard and webct. coupled with this, there is now a range of tools to facilitate information management including commercial products and subject-specific information gateways. despite this, there is evidence of slow uptake in the use of these technologies within education (jones, 1996). britain and liber (1999, p. 3) conclude that: amongst the factors that are slowing the uptake [is] … the lack of a coherent framework within which to evaluate both the pedagogical benefits and the organizational changes required to effectively implement it. furthermore evaluation research to date shows that it is difficult to encourage authentic virtual learning or collaboration. discussion board use, for example, often shows a pattern of peak use directly related to teacher intervention or responses to particular *corresponding author: school of education, university of southampton, highfield, southampton so17 1bj, uk. email: g.c.conole@soton.ac.uk issn 0968–7769 (print)/issn 1741–1629 (online)/04/020113–12 © 2004 association for learning technology doi: 10.1080/0968776042000216183 114 g. conole & m. dyke ‘hot’ topics. collaborative group work needs to be set up carefully and orchestrated to achieve desired results and, despite this, may still end up as a rather stilted collaboration exercise not comparable with direct face-to-face equivalent group work (jones, 1999). integrated learning environments are still predominantly used as ‘shells’ for displaying web pages and rarely get beyond basic information dissemination and administration (thomas & wyatt, 1999). with respect to information seeking and handling, the sheer volume available to learners/researchers is increasing exponentially, unmatched by the sophistication of the searching and handling tools (lawrence & lee giles, 1999). information overload, coupled with confusion of where to look, is increasingly problematic and, despite a growth in the range of searching tools and portals, it is not evident that the right information is being dispatched to the right users in a timely and quality assured fashion (conole, 2002). research into the use of ict has increased significantly in the last decade and there is evidence that learning technology as a research discipline is now beginning to mature (conole et al., 2003). however, research still tends to focus on particular case studies rather than the development of underpinning theories and approaches (conole, 2004). furthermore, little is understood about the ‘affordances’ of different technologies and, more specifically, how these properties might be exploited in particular learning and teaching contexts. we believe that a better understanding of the nature and properties of technologies will lead to a more systematic application of the use of ict for learning and teaching. this paper attempts to draw out these inherent properties building on gibson’s concept of affordances (gibson, 1979) and beck’s understanding of risk (beck, 1992). the paper outlines the current developments in ict research and the associated perspectives and views. the notion of affordances is described, as well as ways in which it might be used to develop a better understanding of the properties of different technologies. we argue that there is a need to explore a theoretical basis for the use of icts and present a conceptual framework as an attempt to further explore the notions of affordances in this respect. the research methodology used and the social theory on which the paper draws, are then described. the paper concludes with suggestions for further research. current perspectives on technologies—myths and realities the best example of the increased use and importance of ict is the now ubiquitous use of the pc as a work tool and the replacement of many traditional work modes of communication, such as memos, with online communication via email. there has been a commensurate growth in the use of technology to support learning, fuelled by the increased use of all-in-one software such as virtual learning environments. there has also been a growth in the amount and variety of resources to support learning and research, through specialized gateways and portals, and niche resource providers. the types of software tools, hardware systems and online environments have also increased in variety and complexity, with tools now available to support everything from research publication management to online assessment and monitoring. affordances of information and communication technologies 115 these developments have been met by a spectrum of views, ranging from wholesale acceptance and calls for pedagogical re-engineering and revolutionizing educational systems, through to significant dissent and cynicism about the use and value of these new tools. in addition, there are growing concerns about the unintended consequences of these changes, such as potential job losses, fears of increased surveillance methods and associated ethical questions (bonk, 2001; borgman, 2001; schuck, 2001; land & bayne, 2001; mckenna, 2002). the hegemony within this continuum of views has ebbed and flowed over the last decade, as outputs from research have accumulated to support different perspectives, and as the population in general becomes more e-literate and comfortable with using the technologies. the variety of ways in which technologies can be used to support learning and teaching, and the enabling tools and resources available to support this process, have increased significantly. however, evidence suggests that practitioners are still unclear about how to use technology appropriately, and its application is often based on common sense rather then being theoretically informed by pedagogical theory (conole et al., 2004). one reason for the lack of application of models and theories by e-learning practitioners may be that, as academics outside the field of education, they find the diverse array of theoretical perspectives alien and overwhelming (mcnaught, 2003). one important stakeholder in this respect are learning technologists, who work with academics and teaching practitioners to exploit new technologies. these ‘new professionals’ frequently have a very good grasp of current theories and models for e-learning and have the skills to support those academic staff who are less familiar with pedagogical theory. a taxonomy of ict affordances this section defines the notion of affordances and outlines a taxonomy of ict affordances with a description of each. salomon describes gibson’s concept of affordances as follows. ‘affordance’ refers to the perceived and actual properties of a thing, primarily those functional properties that determine just how the thing could possibly be used. (salomon, 1993, p. 51) salomon goes on to describe how norman has developed this concept and, in particular, the notion that the affordances of many objects (such as microwaves, car instruments, etc.) often restrict their accessibility to users, arguing that better design of artefacts would make it easier to accomplish certain functions. we argue that this could be developed further and, when applied to current ict applications, can have both positive and negative impacts on the users. pea builds on this in terms of the concept of ‘person plus’, which is the concept of cognition being distributed between the person and their environment, and associated artefacts (pea, 1998). application of the concept of affordances centres around three main research questions: ● how might the affordances of technologies be articulated? ● can these be developed into a taxonomy for ict? 116 g. conole & m. dyke ● how might a critical knowledge and understanding of such affordances be used to support learning and teaching? firstly, relevant current social theory and critique were analysed to establish the key features of modernity and, where relevant, to map this specifically to ict. in particular, we have drawn on the work of giddens (1990, 1991) on late modernity, castells’ work on the nature of the networked society (castells, 1996), and engestrom’s critique of activity theory (engestrom et al., 1999). secondly, the literature on the current use of technologies was analysed. from this analysis a list of the potential themes and commonalities was drawn up and distilled into a taxonomy of ict affordances, which is outlined below. accessibility the range of ict now available offers relatively easy access to vast amounts of information through a variety of different mechanisms. online access is possible via gateways, portals and websites as well as via knowledge networks and shared communities of users. however, the corollary of this is that there has been an exponential increase in the amount of information and the means of accessing it, which leads to issues of information overload, quality assurance and the need for more critical evaluation. this will require a change in the users’ learning and information analysing skills. the challenge is not in accessing material, but rather in knowing how to use what is available. perhaps the balance has shifted from ‘searching’ to ‘selecting’. speed of change an element of the late-modern perspective adopted here, is the information society discourse, where abundant and rapidly changing information is mediated through extensive communication technologies (castells, 1996). giddens acknowledges the importance of information in guiding decision-making: social reflexivity refers to a world increasingly constituted by information rather than pregiven modes of conduct. it is how we live after the retreat of tradition and nature, because we have to take so many forward orientated decisions. in that sense, we live in a much more reflexive way than previous generations have done. (giddens, 1999a, p. 15) continuous reassessment of experiences, and also of the information which is part of these experiences, is therefore central to living with the rapid change of late-modernity. earlier forms of modernity could more frequently use custom and tradition to guide action. the immediacy of access to rapidly changing information or events is a core feature of new technologies, enabling unprecedented speed of access to materials and world events as they happen. however, this speed can also raise issues about quality, lack of authority of sources and lack of reflection. the speed of change may also mitigate against reflective and critical thought, fostering surface approaches to learning. reflection as an affordance will be considered below. the issue here is that the speed of change in a world full of conflicting and changing information, is a challenge for the educational use of the new technologies. in other words, how can it affordances of information and communication technologies 117 be used to enable students to navigate their way through the myriad of changing information and make more informed decisions? diversity ict offers access to a vast range of diverse and different experiences that can inform learning such as overseas web sites, access to subject experts, or use of simulations to replicate complex behaviour. exposure to the experience of others is a key ingredient to effective learning and a potential affordance of ict. the work of boud (1993) and his colleagues defines learning as a holistic activity that needs to connect with people’s life experiences. learning is recognized as constructed in a social, economic and cultural context. boud emphasized the value of learning from the experience of others, of sharing experience and exposing ideas to the critical gaze of the ‘other’. he argued: we also need to be challenged so that we do not fool ourselves with our own distorted assumptions or fail to consider new information which is outside our present range of experience. (boud et al., 1993, p. 15) information technologies provide a means by which people can be exposed to experiences very different to their own and extend their experience beyond their own communities. experience of the ‘other’ through technology raises issues around authenticity and power in the ‘virtual reality’ that can be accessed. for example, there may be disjuncture between the mediated ‘reported’ experience and the reality of lived experience. it raises questions about how one distinguishes between what is real and what is rendered real via the technology. communication and collaboration the communication and collaborative abilities of technology present another key affordance that offers the potential for learning enriched by engagement with the ‘other’. new technologies have opened up the possibility of new forms of dialogue and communication. ict offers the potential to develop new forms of online communities and new means of communicating and sharing information, from signing up to particular mailing lists through to involvement in specialized discussion forums and chat rooms. the engagement with difference and the acknowledgement of diversity through communicative discourse connects with critical social theory and the habermasian idea of ideal speech situations (habermas, 1970). the importance to learning is documented in vygotsky’s approaches to learning (engestrom et al., 1999). this also aligns well with wenger’s notion of communities of practice (wenger, 1998). however, this can lead to issues in terms of individuals being ‘spread too thinly’ across communities, as well as issues of lack of identity and peripheral engagement. bauman presents a pessimistic view of modernity where: unable to stop treading on each others toes in the mega-community, we have stepped into our separate houses and closed the door, and then stepped into our separate rooms and closed the door. (bauman, 1995, p. 45) 118 g. conole & m. dyke but this can also raise questions about the communication and literacy skills of end users. issues and concerns about the e-literacy of both staff and students are currently being debated through a series of symposia organized by the joint information systems committee jisc.1 reflection asynchronous technologies (in particular) offer the potential for encouraging reflection and critique, with users engaging in discussions over a longer time frame than is possible in face-to-face discussions. in addition, users are able to access and build on archived material available from earlier discussion. asynchronous technologies have been used successfully, particularly in the use of discussion forums to support learning, but are also being used increasingly to support the research and teaching activities of academic staff. clearly, there is nothing inherent about ict that nurtures reflection—the key is how it is used. ict has the potential to enable reflection and criticality to be enhanced. it presents new opportunities for knowledge claims to be considered and subjected to the critical gaze of much wider and more diverse communities of practice. recently we have begun to look at this in more detail and have developed a map of learning theories against specific learning characteristics such as reflection and collaboration (conole et al., 2004). we are now using this as a basis to develop a learning design toolkit which aims to provide a more explicit and pedagogically grounded use of tools and resources, as part of the jisc/nsf-funded dialogplus project.2 there is, equally, a risk that the speed and pace of information change outlined above militates against reflection. it leaves no space for contemplation and considered judgement, and promotes a more pragmatic, reflexive immediate response to new information, as it is pixilated across our screens. this can be said to be particularly true of email, where nowadays users are bombarded with so much information that there is a tendency to skim read and adopt a surface approach in terms of reacting to responses and requests (gibbs, 1992). this view is presented in lash’s ‘critique of information’ (lash, 2002). perhaps new forms of reflection and critique will emerge in response to more transitory and digital text. though, as bauman’s (2000) analogy of ‘liquid modernity’ suggests, the shape and form might change. the issue is whether the essential ingredients of modernity, or in our case, learning through technology, remain the same? at a political level, there is evidence that globalization has nurtured, rather than stifled, critical reflection and resistance (klein, 2001; stiglitz, 2002; monbiot, 2003). the challenge, as laurillard (2002) notes, is how we can use ict to transform information into ‘knowledge’, a process which has parallels in dewey’s (1933) advocacy of critical reflection upon experience to create knowledge. multimodal and non-linear the non-linearity of the web (epitomized by hypertext and the use of powerful search engines) leads to the potential for different routes through, and forms of, learning. affordances of information and communication technologies 119 ict enables the learner to move beyond linear pathways of learning, characteristic of, but not exclusive to, behaviourist approaches, and to adopt more individualized strategies and pathways. dewey’s (1933) pragmatic approach to experiential learning is more characteristic of non-linear learning achievable with ict than that of the more prescriptive learning cycle of kolb (1984). another affordance of ict is the potential for multi-modal and non-linear approaches to learning. yet much current computerbased training material still appears to follow a linear, assembly line, mode of learning. many ‘e-learning’ packages are built on behaviourist principles of atomized experiences that need to be completed in a specified order before the individual is positively reinforced and permitted to move on—a form of electronic page-turning. the issue may be whether there is a stark contrast between the method of learning promoted with many computer-based tutorial packages, and the experimental and pragmatic way in which most people acquire for themselves skills in ict. multimodal and non-linear learning modes are perhaps an under-utilized affordance of ict. risk, fragility and uncertainty anthony giddens argues that the concept of risk is central to our understanding of contemporary modernity. giddens defines risk as being different from dangers or hazards in that they are essentially related to the impact of humanity on the world and relate to the consequences of our actions in our world. risk, therefore, is different from fate or ‘natural’ disasters, it relates to choices people make about avoiding risks or taking them. manufactured risk means human risk environments, or human environments of uncertainty, created by the very changes i discussed in the last lecture, created partly by the advance of science and technology. (giddens, 1999b, p. 5) connected to risk are the unintended consequences of actions (giddens, 1990, 1999a). these concepts are particularly pertinent to the context of information technologies, where history shows that there have been major unintended consequences, i.e. the technologies have not necessarily been taken up or used in the ways originally intended. furthermore, another aspect of the rapidly changing nature of ict is that there is an intrinsic level of fragility in digital technologies and networks. these complex systems are vulnerable to abuses, to disruption from viruses and spam, or simply to the servers ‘being down’. the increased use of technologies by different groups of users often gives rise to unintended consequences. for example, the increase in the volume of information available on the web has led to new forms of plagiarism. ubiquitous use of email has resulted in increasing commercial exploitation and unwelcome mail. there are risks and levels of uncertainty associated with a dependence and reliance on the instantaneous provision of information and communication. the intensification of work (pollert, 1991) and need for immediacy associated with the ‘flexibility’ presented by information and communication technologies, can result in levels of dependence whereby an individual’s success or failure in meeting commitments hangs by a tenuous digital thread. 120 g. conole & m. dyke immediacy the speed with which information can be exchanged via the web and email has led to a shift in user expectations in terms of response times to requests from other users. this, in turn, has led to a consequential intensification of working patterns, with users being increasingly required to respond almost immediately to requests which, in the past, would have been dealt with over longer timeframes. monopolization convergence and divergence of different technologies is increasingly important, leading to issues associated with scalability and globalization and the underpinning standards needed to support interoperability. a current focus is also on a critique of convergence verses standardization across the technical, pedagogical, human and organizational aspects. there is a tension between the benefits of diversification and sharing of developments, a central ethos of the open source community with that of monopolization and co-modification, such as the dominance of particular software products to support office applications. surveillance new technologies present new foucauldian means by which those with power can extend their gaze and secure greater knowledge and control over others (foucault, 1979). there is increasing concern about potential infringements on individuals which the infiltration of technological applications make possible. land and bayne (2001), for example, have critiqued the increased default inclusion of monitoring tools within virtual learning environments that mean teachers have the power to monitor student activities more closely than ever before. mckenna (2002) critiques the use of the blind copying function in email being used as a potential power tool unknown to recipients of the main email. similar concerns are being voiced about many of the new ‘smart’ devices and personal tags which are being included in commercial products that enable providers to target and personalize products more accurately. there are concerns about how these tracking devices might be used for other purposes (such as surveillance) or by other agents. application of the taxonomy the proposed taxonomy of ict affordances can then be used in a number of ways: ● to carry out further analysis, critique and development of the taxonomy framework against relevant theory and practical use. ● to establish a clearer understanding of the affordances which should help to inform practitioners in their use of particular technologies to achieve particular goals. affordances of information and communication technologies 121 ● to identify potential limitations and inappropriate uses of the technologies. ● to act as a discussion point for critique and further refinement. ● to offer a checklist to help practitioners understand costs and benefits of different technologies. ● to use as a mechanism for staff development and improving practice – for example, by providing a checklist of potential benefits and draw backs of different technologies which can be used to inform choice and the ways that practitioners might choose to use them. many of the affordances outlined above are known intuitively, but the development of an explicit taxonomy provides a basis for discussion, critique, and further refinement. this may involve questions such as: ● are these the only affordances? ● does their articulation make the benefits and potential drawbacks of technologies more explicit? ● can particular affordances be mapped to particular types of technologies? we believe that the taxonomy will be useful as a checklist for practitioners, to help them make informed decisions about the use of different technologies. it may also, in part, help to make them more aware of the properties of different tools and resources, and use this as part of their decision making process in developing learning activities and teaching plans. we are also considering how this might be used as part of a wider learning design process. this might involve integrating the use of technologies with other teaching methods, tools and resources, through the development of a learning design toolkit, which aims to enable practitioners to map particular pedagogical approaches to their learning design (conole et al., 2004). discussion this paper has considered whether application of the concept of affordances might enable better use and understanding of technologies and their application to learning and teaching. however, this approach leaves a number of unresolved questions. ● how valuable is the concept of affordances and does its application provide any really new insight into the inherent properties of technologies? ● how valid is the methodological approach suggested? ● what other approaches might be taken? ● the approach suggested reflects a particular interpretation of social theory. is this a limitation and might a broader analysis of other social theories yield new insights? ● practitioners are still exploring the potential of new technologies, and the current uses of technologies often do not take full advantage of the medium. therefore, how can practice take full advantage of the affordances of ict? ● does understanding of the affordances actually get us closer to improvement in practice and is this a useful framework? 122 g. conole & m. dyke a fundamental issue is the level of granularity at which this taxonomy might be appropriate; is the taxonomy useful as a philosophical critique of the inherent affordances of ict at a general level or can it be used more explicitly in terms of actually mapping to particular uses. our work in the development of the learning design toolkit as part of the dialogplus project is exploring this issue.2 to address these questions we intend to examine the issues further and discuss these ideas with researchers. in addition, we plan to test out the use of the taxonomy in the ways outlined above with practitioners. these activities should help practitioners to refine and develop the taxonomy and to give a clearer insight into its potential use. we believe that this approach is timely given that there has been little done in the application of the concept of affordances to date in learning technology and given the increased diversity and potential applications of learning technologies. by making the affordances explicit in the form of a taxonomy it will be possible for practitioners to make more informed choices about the ways in which different technologies can be used. as a next stage we plan to assign different affordances to specific technologies to provide a mapping of the potential applications of different technologies. this we hope will provide a mechanism for identifying the strengths and weaknesses of different technologies and the ways they can be used. for example a mapping of the communication and reflection affordances to asynchronous and synchronous communication mechanisms will reveal that both have strengths in terms of supporting communication, but that the latter may also have greater potential to nurture critical and reflective thinking. notes 1. www.jisc.ac.uk 2. www.dialogplus.org references bauman, z. (2000) liquid modernity (cambridge, polity). bauman, z. (1995) life in fragments (cambridge, polity). beck, (1992) risk society towards a new modernity (london, sage). bonk, c. j. (2001) online teaching in an online world, indiana university, available online at: http:// www.courseshare.com/reports.php. borgman, c. l. (2001) digital libraries and virtual universities, in: t. d. senta & t. tschang (eds) access to knowledge: new information technologies and the emergence of the virtual university (paris, international association of universities press), 207–240. boud, d., cohen, r. & walker, d. (1993) using experience for learning (buckingham, society for research in higher education/ open university press). britain, s. & liber, o (1999) a framework for pedagogical evaluation of virtual learning environments, jisc technology applications programme, report no. 41. available online at: http:// www.jisc.ac.uk/index.cfm?name=project_pedagogical_vle castells, e., (1996) the information age: economy, society and culture vol. i. the rise of the network society (oxford, blackwell). conole, g. (2002) systematising learning and research information, journal of interactive media in education, 7, available online at: http://www-jime.open.ac.uk/2002/7/ affordances of information and communication technologies 123 conole, g., ingraham, b. & cook, j. (2003) learning technology as a community of practice? in proceedings of 10th international conference for the association for learning technology (alt-c 2003), 11–13 september, sheffield. conole, g. (2004) research questions and methodological issues, in: j. seale (ed.) learning technology in transition: from individual enthusiasm to institutional implementation (lisse, swets and zeitlinger). conole, g., dyke, m., oliver, m. & seale, j. (2004) mapping pedagogy and tools for effective learning design, computers and education, 43, 17–33. dewey, j. (1933) how do we think? in: j. boyston (ed.) john dewey the later works, 1925–1953 (vol. 8) (carbondale, il, southern illinois university press). engestrom, y., miettinen, r. et al. (eds) (1999) perspectives on activity theory. learning in doing: social, cognitive and computational perspectives (cambridge, cambridge university press). foucault, m. (1979) discipline and punish: the birth of the prison (harmondsworth, penguin). gibbs, g. (1992) improving the quality of student learning (bristol, technical & educational services). gibson, j. j. (1979) the ecological approach to visual perception (boston, ma, houghton miflin). giddens, a. (1990) the consequences of modernity (cambridge, polity press). giddens, a. (1991) modernity and self identity (cambridge, polity press). giddens, a. (1999a) runaway world: how globalisation is reshaping our lives (london, profile). giddens, a. (1999b) the reith lectures revisited. directors lecture 2. london school of economics, 17 november 1999. available online at: http://www.lse.ac.uk/giddens/pdf/17-nov-99.pdf habermas, j. (1970) towards a theory of communicative competence, in: g. dreitzel (ed.) recent sociology: patterns of commmunicative behaviour (new york, macmillan). jones, a. (1995) constructivist learning theories and it, in: n. heap, r. thomas, g. eionon, r. mason & h. mackay (eds) information technology and society (london, sage), 249–265. jones, c. (1999) from the sage on the stage to what exactly? description and the place of the moderator in co-operative and collaborative learning, alt-j, 7(2), 27–36. kimball, l. (1998) managing distance learning—new challenges for faculty, in: r. hazemi, s. hailes & s. wilbur (eds) the digital university (berlin, springer), 25–38. klein, n. (2001) no logo: no space, no choice, no jobs (london, flamingo). kolb, d. (1984) experiential learning (upper saddle river, nj, prentice-hall). land, r. & bayne, s. (2001) screen or monitor? issues of surveillance and disciplinary power, in: ‘online learning environments’, 9th improving student learning symposium, improving student learning using learning technologies, 9–11 september, heriot watt university. lash, s. (2002) critique of information (london, sage). laurillard, d. (2002) rethinking university teaching—a framework for the effective use of educational technology (2nd edn) (london, routledge). lawrence, s. & lee giles, c. (1999) accessibility of information on the web, nature, 400, 107– 109. mackenzie, d. & wajcman, j. (eds) (1985) the social shaping of technology (buckingham, open university press). mckenna, c. (2002) what do we mean by electronic literacy? in: proceedings of the improving student learning using learning technology conference, heriot watt, 79–88. mcnaught, c. (2003) identifying the complexity of factors in the sharing and reuse of resources, in: a. littlejohn (ed.) reusing online resources—a sustainable approach to e-learning (london, kogan page). monbiot, g. (2003) the age of consent: a manifesto for a new world order (london, flamingo). pea, r. (1998) practices of distrbuted intelligence and designs for education, in: g. salomon (ed.) distributed cognition (cambridge, cambridge university press). pollert, a. (1991) farewell to flexibiltiy (london, basil blackwell). salomon, g. (ed.) (1993) distributed cognitions—psychological and educational considerations (cambridge, cambridge university press). 124 g. conole & m. dyke schuck, s. (2001) walking the electronic tightrope: questions surrounding infusion of it into education subjects, proceedings of the improving student learning conference, heriott watt. stiglitz, j. (2002) globalisation and its discontents (london, allen lane). thomas, g. & wyatt, s. (1999) shaping cyberspace—interpreting and transforming the internet, research policy, 28(6), 681–698. wenger, e. (1998) communities of practice—learning, meaning and identity (cambridge, cambridge university press). what are the affordances of information and communication technologies? grainne conole* & martin dyke 1. 2. reviews edited by philip barker alan clarke, designing computer-based learning materials, aldershot: gower, 2001. isbn: 0-566-08320-5. hardback, xviii+196 pages, £45.00. the utility of computers as learning resources is now well established. they can be used as standalone devices to explore various localized phenomena or they can be networked together to facilitate the creation of global learning communities. between these two extremes, a variety of possibilities exist for using computer and communications technologies for the support of teaching and learning processes. more often than not, some form of courseware material will usually be needed to facilitate these pedagogic applications of computers. the creation of this educational software is no simple task and, if it is to be effective, requires careful design. this book by alan clarke discusses some of the important issues that need to be considered when designing computer-based learning (cbl) materials for use in different contexts. the material in the book is organized into three basic sections. the early chapters in the book (1 to 4) make up the first of these. they cover some of the more fundamental issues of cbl. they deal, for example, with the nature of interaction and learning, communication styles, types of computer-based learning material and assessment methods. the second part of the book, (chapters 5 to 8) deal with the 'primitive' types of resource and techniques that designers might use to create a learning product. the topics covered include the use of text, colour, graphics and 90 multimedia. alan clarke is well known for his extensive work on 'screen design for computerbased learning'. needless to say, this work figures prominently in this second section of the book and also in the third part (chapters 9 to 12). the latter chapters of the book are less 'media-orientated' than the previous ones and are devoted to some of the more interesting design issues. the topics covered in these chapters include designing for online learning, screen layout, content, and evaluation. i enjoyed reading them all. however, the chapter on evaluation is the shortest and, in my view, the one that is covered in least depth. although this book is well written and easy to read, there are a few minor spelling and/or typographical errors. for example, the section of the 'periodic table' given on page 90 includes the metal calcium (ca) both in the alkali metals (group i) and in the alkaline earth metals (group ii). this confusion arises because the atomic symbol for cesium is cs and not ca! obviously, a simple typographical error? similarly, the symbol for the element zirconium is zr (and not ze) and the element with atomic number 104 is rutherfordium (rf). of course, simple errors of this sort reflect the need (especially within cbl) for sound proof-reading and the use of subjectmatter experts (a topic that is not extensively covered in this book although reference to 'subject expertise' is fleetingly made in a few places in chapter 11). the only major shortcoming of the book is the absence of any reference section and any useful alt-j volume 10 number i citations to the very large volume of related literature that is now available on designing and producing cbl materials. a 'selected bibliography' would therefore have been a useful addition to this otherwise outstanding book. unfortunately, within the 12 chapters that make up this volume, there is only one (rather dated 1987) reference. this appears at the end of chapter 7 (the 'graphics' chapter). as well as the distinct lack of references to conventional literature sources, there are also very few web references three in total! one of these (the gower web page) appears repeatedly in chapter 6 (referencing examples of the use of colour) and the other two appear in the chapter on evaluation. i think a few more references of both sorts (conventional and electronic) would have made a useful addition to the book. overall, this book provides • an excellent introduction to the world of designing materials for the support of computer-based learning. it is written in a clear, lucid style and contains lots and lots of good pictorial illustrations. of course, unlike electronic books, the use of colour in conventional paper-based books is often a difficult and costly process for many publishers. all the illustrations that appear in this book are therefore in black and white. naturally, writing a chapter on the use of colour (chapter 6) without actually being able to use it is not easy. the author overcomes this problem by placing his eight figures for chapter 6 on the gower website {http:llwww.gowerpub.com). but now, to get at these figures the reader has to navigate through this site to get to the entry for this book and then download a 'pdf' file. unfortunately, after i downloaded this file and opened it with adobe acrobat, i found that the numbering of the figures used by the publisher (figures 44 to 51) did not correspond with the numbering used by the author (figures 6.1 to 6.8)! of course, this is an error that could easily be rectified and reflects the tremendous flexibility of electronic (e-book) over paper-based publication. in my view, this is a book that is aimed at newcomers to the area of cbl design rather than established veterans like myself although i am sure many (like me) would find it a good read. undoubtedly, the book clearly identifies what is involved in designing cbl materials and what users of this technique can expect to achieve through its use. philip barker university of teesside helen edwards, brenda smith and graham webb (eds), lecturing case studies, experience and practice, london: kogan page, 2001. isbn 0-7494-3519-4. softback, 168 pages, £19.99. as the title suggests, this book is about lecturing, with a specific focus on higher education. it is one of the most useful and usable books on lecturing around. the target audience is anyone who has to prepare and deliver a single lecture or coordinate entire lecture courses. the main part of the book is a collection of seventeen real-life cases from universities in the uk, australia and the usa, illustrating the range of challenges facing university educators today. each case (of six to ten pages in length) follows the same format. it begins with two short sections: one providing an overview of the issue under discussion and the other giving an account of the context of the case. the contexts cover a range of types of higher education (theoretical, practitioner, professional) and disciplines including politics, physiotherapy, telecommunications engineering and contemporary poetry. the cases also deal with educators with very different levels of teaching experience, from the new teacher preparing to lecture for the first time through to experienced lecturers looking at ways of improving their current practice. each case account is divided into two or three parts. the first part expands on the issues being discussed, explaining what happened, how the teacher was feeling, how colleagues or students were responding to the emerging challenge. this is then followed by a small number of questions that lead the reader to think about the case, such as how they might have responded in the same situation and what they might do in order to get the class working effectively again. the questions depend on the nature of the case, but in each instance they provide a starting point for addressing the emerging problem. it is easy to envisage how this book could be used as part of a short course on university teaching or to help guide the discussions in teaching seminars. the second part continues the case account, developing it further by outlining the course of action the teacher followed in order to improve the situation. this is then followed by a more extensive series of questions and maybe a further development of the case. the case account ends with a discussion section and in some instances includes further questions for reflection and action. each case includes references to relevant education literature. 91 reviews the seventeen cases are divided into four sections. part 1 deals with key competencies in lecturing. it includes four cases that address topics such as dealing with disruptive students, taking over a course from a more experienced colleague, and that all too common situation these days being asked to teach a subject in which you have little expertise. whilst the emphasis in this section is primarily on new or inexperienced teachers, the case content will provide useful insights or points for discussion to educators of all experience levels. in part 2 ('orchestrating learning in lectures') there are four cases focusing on the learning that takes place during lectures, in particular large lecture settings. there is also an emphasis on individual learning in these cases, drawing the student out from the crowd and encouraging them to engage in discovery and interaction. there is also a useful case entitled 'just give us the right answer'. it deals with the situation facing many teachers where students prefer to be told the answer than spend time working on reasoning and constructing the answer for themselves. part 3 deals with feedback. this section includes cases that explore dealing with hostile feedback and responding to feedback in order to improve course content and the learning environment. the final section of the book (part 4: authenticity: living your values in lectures) covers cases that deal with carrying through our beliefs about learning and embraces various aspects of learning such as deep learning and the teaching of culturally sensitive material. there is a strong theme of experiential learning and valuing the students' lived experiences. i have already lent this book to a couple of colleagues and the feedback from them has been extremely positive. they had two main responses: firstly, they expressed a feeling of relief that other educators faced the same problems as they did and secondly, that the format of the cases provided a framework for thinking about ways of addressing the issues arising in each situation. this collection of cases is readable and thoughtprovoking. it addresses practical problems and challenges, and provides insights into possible solutions without being prescriptive or resorting to a 'tips and tricks' format. the inclusion of references provides further support for those not familiar with the literature. these cases will serve as an excellent resource for group discussions about teaching and learning it should be a part 92 of the library of every teaching and learning support unit. sue fowell university of new south wales p. schwartz, s. mennin and g. webb (eds), problem-based learning: case studies, experience and practice, london: kogan page, 2001. isbn: 0-7494-3492-9. softback, 182 pages, £19.99. in the dynamic world of educational practice, problem-based learning (pbl) is a hot topic. with its international origins and high profile application in a number of areas of professional education, pbl offers the tantalizing opportunity to support active, learner-centred, collaborative, interdisciplinary learning. however, in operation this opportunity is somewhat more challenging to grasp. for example, a review of the literature suggests a lack of consistency about what constitutes pbl, and questions whether its focus on the 'problem' as the stimulus for learning is in any real sense different from any other approach to learning (or indeed life itself!). there are definitional and terminological concerns and differences of emphasis and focus. and whilst the 'problem' is the common starting point for learning, the degree to which this needs to be situated into the everyday experience of the learner is contested. acknowledging this context, 'problem-based learning: case studies, experience and practice' is designed to help the reader steer a course through these issues and explore and exploit the potential of pbl. the book is intended to be a practical guide for both experienced pbl practitioners and those who 'do not yet understand it' (page 1). the book offers twenty-two short case studies designed to share experience and practice from a range of perspectives so that by . . . learning from the compelling accounts of their experiences you can develop helpful strategies for when you use (or suggest using) pbl in your own curriculum, (page 1) each case has a descriptive element written in the first person and a 'case reporter's discussion'. each description is split into two or more parts, each part concluding with a set of questions for the reader to prompt reflection and engagement with the text. readers will either love or hate this uncommon approach. certainly, prompts and questions of this kind can encourage the active engagement the editors seek. however, it can alt-] volume 10 number i also reinforce the perhaps inevitable lack of detail and substance in each of the short case studies, and highlight the questions not asked at least as strongly as those that are. whatever one's personal preferences the editors have, through this device, made a useful attempt at simulating a problem-based approach to the underlying strengths and pitfalls of pbl. the case studies are grouped. the first set of eight case studies explores political, administrative and resourcing issues. all these are located in medical or dentistry contexts, which may limit the extent to which the points raised in the discussion sections can be generalized to other educational and training contexts. the second set of six case studies explores issues relating to teachers. here medical school examples are enhanced by one located in an architectural department of an australian university and another located in an instructional innovation institute in the usa. the latter case study illustrates many of the strengths and issues of the collection as a whole. whilst it is an honest 'warts and all' account of a staff development workshop held in 1996 which raises some important issues about assessment in pbl, the format of the case study means that many of the issues are left hanging without a set of practical 'solutions' that might be transferred to other practice. this would be less of a concern had the promotion of the book not explicitly indicated that it had been designed to provide . . . an invaluable resource from which to draw exemplary lessons for those approaching teaching with pbl for the first time, this collection provides an inspiring and valuable guide to understanding its methods and developing successful strategies for using it in their own curriculum and teaching, (back cover) the third and final set of seven case studies addresses issues from a student perspective. again all but one are taken from a medical or dentistry context. however, the learner-focused nature of these contributions and the significance of assessment in many of them, helps to make this section the most rewarding of the three, particularly in terms of the transferability of the insights and 'lessons learned'. the final conclusion picks up on, and summarizes, the key 'lessons learned' from the case studies. the list of key points will not be a surprise to experienced practitioners and educational managers: • leadership is crucial to the successful introduction of pbl. • teaching staff need to 'buy in' and 'own' pbl as a legitimate approach to teaching, learning and assessment. • teaching staff need to be developed to deploy pbl successfully. • pbl requires effective communication and collaboration between key stakeholders. • conflict and uncertainty about power and control in teaching and learning will arise in the introduction and use of pbl. • assessment has to be consistent with the overall aims and intentions of pbl. • not all learners will 'play' the pbl 'game' (although whether this makes the individual or group 'dysfunctional' as suggested on page 175 is open to debate). those familiar with pbl will enjoy and empathize with many of the situations described in the case studies. however, i am less convinced that they will learn a great deal that is new from the book. rather, it offers a very accessible, thought-provoking and occasionally moving induction into the issues associated with pbl and indeed significant curriculum change more generally. as such it is a useful introduction to the topic which would have benefited from a wider range of case study contexts, and in particular case studies located in crossdisciplinary contexts or where so-called mixed or hybrid approaches to pbl were employed. peter funnell suffolk college john field, lifelong learning and the new educational order, stoke-on-trent, uk: trentham books, 2000. isbn: 1-8585-6199-x. softback, 200 pages, £13.95. john field's lifelong learning and the new educational order is an interesting, useful and scholarly view of the contemporary lifelong learning debate. written from what is largely a social science, rather than pedagogical, perspective, he draws together a range of social, political and economic perspectives on the issue of lifelong learning in the emergent 'knowledge economy'. although he doesn't especially engage with issues surrounding the application of learning technologies to lifelong learning, the 93 reviews book will nevertheless provide alt-j readers with a valuable perspective from which to view those technological and pedagogical issues that are perhaps their more immediate concern. of particular interest in this context is his success in distinguishing the reality from the rhetoric of lifelong learning. the book is relatively short. in the author's words, it is an 'extended essay'; but it is extremely thoroughly researched. the essay itself is about 150 pages and the bibliography almost 20 pages long. this care serves to situate his argument convincingly in a contemporary and international view of the issues he addresses in the book's five chapters. in brief, field argues that, like it or not, we currently inhabit both a knowledge economy and lifelong learning society. however, he goes on to argue that many of the commonplaces surrounding those ideas have little real foundation and that we need to be more reflective about what the implications of this emergent social order are and what we need to do in the face of this change. the first chapter spells out the case for recognizing that, at least in the developed world, lifelong learning is a genuine and significant phenomenon. however, rather than point at the institutional evidence for this, he reflects on the importance of what he calls 'informal learning' (p. xi). in this he includes such things as all the learning implicit in the pursuit of alternative medicines and leisure activities. in these he sees both evidence of a greater individualization of interest within society, but also of growing insecurity. in the second chapter he extends this view into the area of work-related learning and addresses many of those commonly expressed views about the ways in which changing patterns of employment both encourage and demand a more continuous approach to the acquisition of knowledge and skills. however, while acknowledging that essential truth, he is also careful to point out that this is less universally true than one might expect from the rhetoric. significant elements of the world's population as yet remain largely unaffected by these changes. still further in the third chapter, 'the learning economy', he argues that the knowledge economy is also not necessarily what policy and rhetoric might suggest. notably the realities of corporate economics do not necessarily support the often promoted view that well-organized and regular training is necessary to success. he offers several interesting case studies of major organiza94 tions that adopted worker-centred management and training strategies that, far from leading to corporate success, led to the sale of companies and the abolition of the training systems. in the fourth chapter, 'who is being left behind?', he examines the human cost of the changing environment. he observes that while humankind's capacity for learning throughout a lifetime empowers at least some people to cope with our rapidly changing contemporary world, others may be being positively excluded by the very mechanisms that are created to support the successful lifelong learners. in the final chapter he explores what he calls 'four key elements of a future strategy' (p. 134). these are: • rethinking the role of schooling in a learning society; • widening participation in adult learning; • building active citizenship in social capital; • pursuing the search for meaning. i have only one small reservation about what is, overall, a sound and scholarly perspective on the lifelong learning debate; and that is with the author's occasional 'pot-shots' at postmodernism. i describe his comments on the postmodern as 'pot-shots' because, while he dismisses post-modernism as 'a dead end' (p. 133), he never actually engages with it intellectually or presents any arguments to indicate why he views it as intellectually impoverished. reading between the lines it is clear that his problem with the postmodern is that it embraces uncertainty as a fundamental aspect of understanding. as indicated above, the last of his four elements of a future strategy is 'pursuing the search for meaning'. his quest throughout the book is an essenti-ally positivist one which sees lifelong learning as a route to a more certain understanding of one's role and place in an uncertain world. however, this is by no means incompatible with postmodernism. understanding that knowing is contingent, rather than certain, in no way undermines its ability to provide intellectually robust coping strategies. however, even if the author's pursuit of the philosophical underpinnings of his argument are a little less rigorous than might be ideal, it really takes away very little from this otherwise insightful essay. bruce ingraham university of teesside alt-j volume 10 number i d. squires, g. conole and g. jacobs (eds), the changing face of learning technology, cardiff: university of wales press, 2000. isbn: 0-7083-1681-6. softback, 182 pages, £7.99. selecting the papers to be included in this collection must have been an interesting task. the concept of illustrating the evolution of learning technology since 1993 when alt-j was first published, by synthesizing papers previously published in that journal, was well conceived, and this book provides a stimulating anthology. inevitably, the academic style and variety of authors means that this is not a book to read casually in a single sitting. it merits time, several return visits, and dipping in and out of sections as ideas emerge and re-emerge in different areas. there are thirteen papers in the collection. the earliest were first published in 1995, and the most recent in 2000. their authors are spread world-wide australia, new zealand, canada and usa as well as the uk, although the overall focus is very much on experiences from australia and the uk. the papers are presented under four headings: 'design and evaluation of learning technology'; 'institutional change'; 'learning technology in a networked infrastructure' and 'the future'. each paper is reproduced in full in its original form, but with an addendum to provide the author(s) with an opportunity to update their work, or comment on subsequent developments and experiences. the book begins with an introduction and overview of each of the sections, and could serve well in place of this review because it attempts to precis each paper and illustrate why they have been grouped into the sections chosen. in the first section, headed 'design and evaluation of learning technology', the emphasis seems to be on the need for, and the importance of, effective communication and interaction. two frameworks are presented, one to support the design of student-centred constructivist learning experiences (reals from grabinger and dunlap), and one to support evaluation by considering the whole context within which technology may be situated (secal from gunn). these are supported by fowler and mayes' discussion of learning relationships from a psychological and anthropological context. mitchell's wonderful paper on the outrageous abuse of statistical methods, and his challenge to the appropriateness of parametric approaches delighted me, but i did wonder whether it really fitted into the context both of this collection and this section. the section on 'institutional change' presents two papers from australia and two from the uk. reading the papers in close succession highlights a difference in scale and resource which is alarming, although there are still many common issues identified. unfortunately in the paper from mcnaught and kennedy, describing the 'substantial' (their word i think i might say breathtaking!) investment strategy at rmit, some of the tables have not printed with sufficient contrast, so much valuable information is lost to the reader. the three papers under the heading of 'learning technology in a networked infrastructure' return to the idea of learning communities, lifelong experiential learning, and the importance of supporting communication. they are generally ethnological and illustrated with specific examples, in contrast to the more theoretical papers in the opening section, but many ideas presented there can be seen echoed in this section. the final section is entitled 'the future' but acknowledges that the rate at which technology is evolving poses great challenges for anyone who wishes to speculate about or to forecast the future. it contains two papers, and again the ideas from the opening section are revisited here. one paper describes bt's real time interactive social environment (rise), which aims to present a user-centric portal to tools or modules, but via a sufficiently generic structure that it can be readily adapted and transported to different contexts. the other considers the notion of a peripatetic electronic teacher (pet) working in a virtual university, and the implications and requirements to make such a model effective. overall, i have enjoyed reading this book. because someone else has gone to the trouble of selecting the papers, i have read things i missed previously, as well as revisiting some wellthumbed pieces. perhaps i was a little disappointed to see so many familiar names amongst the authors, and i felt that the selection tended to present the views of staff in senior educational development or educational technology roles. the historical overview is good in terms of presenting frameworks and discussion of strategy, but i felt that perhaps it did not reflect the process which so many of us have been through where were the reflections on the mistakes and frustrations that we have all experienced? the editors express the hope that the publication will 'provide an interpretative framework for an understanding of the design and use 95 reviews of learning technology, and stimulate an appreciation of the underlying issues and their significance for supporting learning and teaching'. this a grand aim which i will not attempt to appraise, but in providing a single volume to draw together diverse papers representing a historical perspective on learning technology, the editors have succeeded in creating a readily accessible and thought-provoking collection of ideas and experiences. yes, we would all have made our own different selections of papers to represent the last eight years, but that does not make this particular collection less valuable. nora mogey university of edinburgh pat maier and adam warren, integrating technology in learning and teaching: a practical guide for educators, london: kogan page, london, 2000. isbn: 0-7494-3180-6. softback, vii + 162 pages, £19.99. to quote its authors, this book is first and foremost 'a resource book for academics engaging in c&it learning and teaching innovations'. the academics that the authors have in mind are mainly teachers in higher education within the uk and this narrows significantly the scope indicated by the title. it aims to bring together, and integrate within an educational framework, the dispersed and wide-ranging literature dealing with c&it implementation. this framework consists of three main elements: • the political agendas driving c&it implementation in the uk, together with support mechanisms; • the pedagogy of open and independent learning; • the technical issues involved in producing and delivering digital learning materials. the intended readership for this book is not just academics who use (or wish to use) c&it in their teaching but also educational developers who are 'devising staff development programmes'. in terms of coverage, there are five chapters: • agendas for change in higher education; • developing new teaching skills; • designing learning environments; • using communication technologies to facilitate learning; 96 • assessing student learning. each chapter follows the same format: exposition of a theme, with integrated references to relevant websites, followed by a set of activities, followed by exposition of another theme and so on. at the end of each chapter is a set of references for further reading. there is also a very useful, and well-designed, website at http:llwww.clt.soton.ac.uk/activeguide. this provides further web references and some useful learning materials relating to issues mentioned in the text. this formula works particularly well in those chapters which are strong on the practicalities of c&it implementation; the chapter on using communication technologies to facilitate learning is a case in point. almost two-thirds of the chapter deal with the broad educational issues of discussion, communication and collaboration within an online context, enabling readers to make informed choices about the methods they might employ in a given technical environment. referring to the available literature published to date on the web, it covers the possible structures for online group work and compares these with face-to-face group work, allowing the reader to make a balanced judgement. it also includes a section on the assessment of participation in online communication. the activities provide a clear and helpful framework, which should enable newcomers to this form of tutoring to make a confident start in managing online seminars, avoiding most of the well-known pitfalls. the last third of the chapter provides comprehensive coverage of the various technical options open to the teacher, evaluating their various strengths and weaknesses. given the importance of online communication in today's educational setting, this chapter alone is worth the price of the book. the chapter on developing new teaching skills is a useful starting point for any lecturer considering developing digital materials. it not only shows how easy it is to begin this type of development but also provides a great deal of realistic advice on the difficulties inherent in mastering some of the more complex technologies. it points out that a great deal of time and effort can be saved by exploiting materials that have already been created and are sitting on the web waiting to be discovered. the web references are especially valuable and the active guide provides further support materials. one of the most pleasing features is the authors' alt-j volume 10 number i concern to highlight the disadvantages as well as the advantages of digital materials. a resource book of only 160 pages is hardly likely to be able to address in any depth the complexities of the political agendas discussed in the first chapter or the pedagogy of instructional design which is dealt with at the beginning of the chapter on designing learning environments. in the case of chapter one, for instance, two paragraphs on the knowledge economy and just over a page on the globalization of higher education are clearly insufficient to provide a meaningful overview of the concepts involved. more worryingly, perhaps, the authors present teachers in he as being required to 'comply with the economic agenda of providing the right kind of work-force for the next century' and appear to believe that this new obligation is contemporaneous with' changes in educational theory' that are focusing on ' the learning process as well as the content'. in reality, of course, the nature of the learning process has been at the centre of educationalists' concerns for at least the last twenty years or so. this kind of superficial overview leads, perhaps inevitably, to activities which are far weaker than those of a more focused, practical nature to be found embedded in the other chapters. for example, in order to introduce the distinction between deep and surface approaches to learning, teachers in he are asked whether, when they are learning, they 'really want to understand what it is all about' or, alternatively, whether they 'prefer to collect facts, details and examples and then memorize them'. and with that the issue of approaches to learning disappears from view, never to resurface. despite these weaknesses, the authors were probably right to make the attempt to introduce some of the political and educational issues which inform c&it implementation, as all too often technological issues are discussed in a vacuum and educational priorities are pushed into the background. one could only wish that they had been more selective, however, and covered one or two of the more central issues in greater depth (approaches to learning and instructional design for example) rather than risk undermining their endeavour by surveying too wide an area. this said, the book does perform a valuable service not only to teachers in he and fe but also to teachers in primary and secondary schools, who, as the authors point out, are already grappling with c&it implementation within the context of the national grid for learning. terry goodison and john o'donoghue university of wolverhampton 97 using web-based support for campus-based open learning: lessons from a study in dental public health ruth holt* martin oliver** and claire mcavinia** *department of transcultural oral health, eastman dental institute, ucl **education and professional development ucl email: r.holt@eastman.ucl.ac.uk there has been much written about the use of the web in higher education, much of which advocates its use as an effective way of supporting learning, particularly in terms of the desirability of features such as flexibility and the value of online discussions. in this paper, a case study is described which calls some of this received wisdom into question. the study also explores wider issues of curriculum design, particularly in terms of the role of assessment and of self-assessment, both of which played a crucial role in the course. unlike many studies, then, the purpose of this paper is not to demonstrate the success of a particular approach or to advocate particular forms of practice, but instead to highlight the shortcomings of existing guidelines for curriculum development in this area. this suggests that further inquiry into this form of education is required — and in particular, inquiry that pays detailed attention to the backgrounds of learners, and involves close study of their experiences. introduction the use of the web to support courses in higher education has become commonplace in recent years. however, much of what has been written has concentrated on the features of the medium that make it suitable for distance learning; there has been less written about the influence of these features on campus-based courses. this paper presents a case study which explores this particular problem, and which questions received wisdom about some of the claims that the pedagogic features of web-based learning are desirable for learners. firstly, however, the literature on campus-based open learning, and on the use of the web to support dental education (which forms the focus for this study) will be reviewed. 51 ruth holt martin oliver and oaire mcavinia using web-based support for campus-based open learning background web-based support for campus-based open learning the use of web-based resources to support learners at a distance has been widely investigated in recent years, with many authors describing the benefits that it can bring. such benefits include increased flexibility for learners (nikolova and collis, 1998), its ability to support discussion (nixon and salmon, 1996) and its wider benefit for the institutions involved in terms of student recruitment (hanna, 1998). whilst some of the benefits that apply to distance learning will also accrue for students based at traditional institutions, it would be naive to assume that all will apply, or that they will all result in the same patterns of student engagement. however, the particular features of this niche form of web-based support remain ignored; even substantial investigations have grouped campus-based open learners alongside their distance counterparts as a contrast to students in traditional classes (for example, hiltz, coppola, rotter, turoff and benunan-fich, 2000). to date, much of the research on the use of web-based support for campus-based courses has concentrated on the implementation of systems (for example, shaikh and macaulay, 2001), or has focused on the results of a particular case study. few have attempted to draw out the wider implications of these for the design of web-based support for open learning on campus, although there are some notable exceptions (such as boyle and cook, 2001). in part, such caution is entirely appropriate, given the problems of generalizing from case studies of educational impact (oliver, 2000). however, one valuable way in which case studies can contribute to our general understanding of an area is by challenging received wisdom through the provision of counter-examples. there is thus a need to reconsider some of the benefits traditionally associated with online learning in the context of campusbased open learning. use of the web in dental education computer-assisted learning has been used as a part of undergraduate teaching in dentistry, but is perhaps more often seen as a resource suitable for postgraduate training. importantly, software packages (such as cd-roms) still seem to be the dominant form of computer-assisted learning in this area, with current research in dental education still coming to terms with now-familiar issues in learning technology research such as the reluctance of students to lose contact with their tutors and the difficulties of moving beyond affective responses to consider actual changes to learning (see, for example, lechner, thomas, bradshaw and lechner, 2001; welbury, hobson, stephenson and jepson, 2001). however, web-based courses are now being introduced for continuing professional development (cpd) purposes an area that is growing in importance for dental professionals (grigg and stephens, 1998). this is, perhaps, unsurprising given the pedagogic relevance of web-based courses for learners in work (bradley and oliver, 2002). one reason for the relatively low level of uptake is that the value of such approaches has been called into question for certain key topics such as clinical decision-making skills (for example, kay, silkstone and worthington, 2001). consequently, many educators in this field remain in doubt about the viability of web-based approaches to teaching. 52 alt-j volume 10 number 2 the context for this study against this backdrop, a decision was made to redevelop a programme in dental public health for the m.sc. of the university of london and the diploma of the royal college of surgeons. traditionally, this had been delivered through a one-year, full-time programme involving traditional face-to-face teaching methods. cohorts included uk and overseas graduates; there are usually around equal numbers of each on the course. in recent years cohorts have been small, rarely growing larger than ten students. the prospect of a partnership with an overseas institution led to questions about how the course could be redeveloped to support distance students. a distance-learning version of the course was available, but was not web-based; rather than relying on this, it was decided that the potential partnership presented an opportunity to explore whether a web-based format would be appropriate for this course. this exploration was carried out by piloting a web-based approach to teaching and learning on two modules of the course. the modules were delivered using webct, which is the institution's supported virtual learning environment software. in order to ensure that the inferences drawn from this pilot were appropriate for the target group of students, the pilot was carried out by offering the course to students currently studying on the traditionally taught master's degree. as a result, the students who participated in the pilot used a web-based course to support what was, for them, a campus-based education. the course that was developed drew on many of the standard features of webct, and the initial module made use of web pages, bulletin boards, assessment facilities (multiple choice, answer matching, etc.), essay submission and administration facilities, reference management resources, and so on. the students on the course were unfamiliar with webct, and so an introductory page was developed (figure 1) that introduced students to all of the features of the environment that they would need to use during the course. students worked through this, with tutor support, as part of an initial face-to-face workshop, and the materials remained available from the course home page as a point of reference. getting started with the dental public health course what this document is fir this documentwelcomesyoutothe course, tells you about webct and contains some exercises to help you gain confidence with working in webct. it will probably take you 30-60 tninutes to work through it. start by reading from the top, attempting the exercises as you go. remember that this document has been opened in a new window. this means that you can swap between this document and webct easily, without having to close either of them. this should help you when you're trying to follow the instructions for the exercises. .__ you have now arrived at the very first module of our course in dental public health! our course has been developed jointly by the eastman dental institute and education and professional development of university college, london and represents a new venture in the subject. for the figure i: an excerpt from the introductory materials for the course " s3 ruth holt martin oliver and claire mcavinio using web-based support for campus-based open learning in addition to the materials and resources provided for the students, collaboration and discussion was encouraged. structured discussion areas were set up, where students were asked to post answers to questions, to discuss other students' responses, and were encouraged to ask questions of the tutor and each other. several assessment features were used, including formative self-assessment and assignment management. from an early stage in the design process, it became apparent that the use of webct could support pedagogic approaches that were novel for this particular course, such as the introduction of interactive assignments, which involved making judgements about images (for example, figure 2) and analysing the resulting data using spreadsheets. these assignments closely resemble authentic problems facing practitioners. look at the first two sets of bitewing films and the completed example spreadsheet to make sure that you understand exactly what to do and how to enter your scores on the worksheet. (to look at the example spreadsheet, go to excel, and click on the tab at the bottom of the page that says, "example". this contains a worked example for the two set of bitewing films shown in the table below.) you may find it helpful to print this page off so that it is easier to compare the images with the numbers in the spreadsheet. figure 2: an image and excerpt from the assessment on judging bitewing radiographs another important development was the introduction of self-assessment activities in the second module. this reflects wider concerns within the field about practitioners' ability to assess their own ability, and the relevance of training students in self-assessment skills as a vital component of cpd (see, for example, evans, 2001). the students developed criteria for self-assessment at a second face-to-face session, based on their analysis of the strengths and weaknesses of a pre-prepared 'bad practice' document. students were then asked to apply these criteria to their own work, in order to assess their own strengths and weaknesses and to grade their performance. case study methodology to evaluate the pilot, several sources of data were considered. these included direct observation of students, monitoring their use of resources and participation in discussions using webcts student tracking facilities, and, most importantly, a focus group and interview. this range reflected both the small number of students on the course (three, together with additional participants who volunteered to be involved but were not formally assessed) and the emphasis on understanding students' experiences through considering both their accounts and our perceptions of the course. these data were analysed following the procedures described by kvale (1996) for ad hoc meaning generation. — an-] volume 10 number 2 results and discussion four main issues arose from this study: assessment, experiences of self-assessment, the role of flexibility and online discussion. each of these will be considered in turn, and will be illustrated by quotes from the focus group and interview with the students. assessment issues it was clear from tracking usage of webct, and from comments in the focus group, that the students were assessment-driven. this is hardly a surprising finding, echoing as it does other work on assessment (see, for example, the discussion of 'backwash' from assessment in biggs, 1999). however, the small cohort permitted a detailed level of observation and discussion that allowed the implications of this to become clear, revealing some important features of the students' engagement with the course. the direct implication of the influence of assessment was that non-assessed activities were neglected, and time was allocated to work in proportion to its perceived status as contributing towards a final score. with [this activity] having formal assessment assignments, deadlines and marking, you will almost instinctively give it a higher priority than you will other areas, because that is the way we have been brought up and educated. although they were assessment-driven, they did not particularly like being assessed. however, they saw it as being vital to their learning process not because of the feedback that they received (although they did recognize the value of this) but simply because it forced them to engage with the material they otherwise used superficially. i think, to be frank, who likes to do the assignment? who likes to do the deadline? you have to submit this assignment by the end of december, january. in the learning process, you have to do . . . you have to do the assignment, and by then only you learn, you learn something. so i think the assignment is very beneficial, because it makes you t o . . . it pushes you to do something, rather than [the alternative where] you are just left to do your own reading. because i think you won't do [the reading] until the end of course when the exam comes. this lack of engagement was systematic; it was not merely a symptom of this particular course. for the students, their various lecturers' use of assessment represented an important use of power that coerced them into committing time and effort to particular topics. i can get away with not doing the background reading for lecture x because i'm not likely to be found out. i can cover it. but if the assignment's not in by the computer time, then it's just black and white, isn't it. it's either there or it's not there. i think if you're going to put in something that's not covered anywhere else, then just by the structure of this, or the weighting of the grading of the other areas, you will ensure that we do that. although students discussed the competing demands of different parallel modules on their time, they made it clear that such coercion led primarily to the sacrifice of leisure time. i think the answer for us is, we like assignments because it keeps us thinking, or it makes us think about this course rather than going down the pub tonight. — ruth hoh. martin oliver and claire mcavinia using web-based support for campus-based open learning these responses seem to paint a cynical, calculating picture of the student experience. however, drawing on bourdieu's notion of different forms of capital (1986), they can be interpreted instead as representing a rational expenditure of precious personal capital their time in order to gain the maximum symbolic capital (qualifications and grades) in return for their investment. students' experience of self-assessment further assessment-related issues arose from the self-assessment exercise described above, which the students found to be extremely difficult. the creation of criteria for assessment was the first indication of the students' difficulties. as a group, the students were able to generate and agree criteria against which their work should be judged. however, they were not able to agree on the relative weightings each of these should be given. the weightings they chose for each group of criteria (layout, quality of analysis and use of evidence) appeared to reflect their understanding of the purpose of assessment, and so will be outlined briefly, under the headings suggested by the students as a rationale for their allocations. rationale for allocation of marks layout quality use of evidence 'substance not spin' 24 38 38 volume/length rewarding effort' 35 35 30 'masochism as motivation' 40 30 30 table i: students' percentage mark weightings for the three categories of criteria the first student weighted the marking scheme heavily towards the thoroughness and validity of the analysis, attributing relatively low importance to the structure and presentation of the documentation. (the weight he was willing to attach to this category arose almost exclusively from the need for a well-structured document.) this student defended his allocation on the basis that, if this were a real report for use in dental public health, the reliability of decisions based upon it would depend solely on the quality of the analysis. the second student spread the marks evenly across the three areas. she argued that this was fairer than over-emphasizing any one area, since a clever student who made no effort would score highly if analysis was allocated most of the marks. she explained that a recognition of effort, rather than an arbitrary measure of quality, was the most appropriate way to reward students. implicit in this is a positive motivational model of the role of assessment, encouraging those who try hard to greater lengths. the third student adopted a different scheme again, placing the greatest emphasis on layout and then evenly dividing the remaining marks. this would force him to take this aspect of his work which he knew that he was weak at more seriously, he explained. he described this as a 'punishment' and even a 'masochistic' model of assessment, intended as a deterrent to laziness. after discussing these different models, the third student agreed that his mark scheme was too personal to be appropriate for everyone in the group. (this illustrates another important distinction between the three; he viewed an ideal assessment scheme as being 56 alt-j volume 10 number 2 tailored to the individual being assessed, whilst the second believed that the allocation of marks should be tailored to the students' ability and the first felt that marks represented an objective set of criteria.) he was willing to adopt the mark scheme of the second student, but not the essentially positivist scheme of the first student. similarly, the first student refused to accept the appropriateness of the other two mark schemes on grounds of validity. consequently, it was agreed that each student should use all three mark schemes to grade their work, as would the course tutor, and that marks would be compared both across schemes and across markers in order to enquire into the ways in which these different value systems rewarded the same piece of work. eventually, however, only one of the students completed this process. the first student, refusing to accept that he (or any other non-expert) could validly mark the assignment, opted out of the exercise completely. the second student prepared a qualitative assessment of her work (which the tutor agreed was insightful and appropriate) but said that she felt it was inappropriate to award herself marks using any of the schemes. although the third student completed the exercise, he said that he did not take the marks he had given himself particularly seriously. interestingly, in spite of difficulties of grading, both the second and third students felt that the qualitative element of the self-assessment had been extremely valuable. clearly, the problem that arose in this case was not one of understanding or ability, but was instead a cultural issue. because this form of assessment places the responsibility for devising marking criteria and judging work with the student, it represents a radical challenge to traditional forms of power and hierarchy in education (rowland, 1993: 136-40). such a challenge left the students feeling uneasy. although they were all capable of completing the process, they could not accept it as an appropriate part of their course given their previous experiences of education. in effect, engagement in such self-evaluative activity simply was not part of the students' habitus (bourdieu, 1977). this suggests that attempts to introduce techniques such as self-assessment will need to take careful account of students' educational histories. it may be necessary to build up to the introduction of such an unfamiliar technique using less radical methods first in order to bridge the cultural gap between past experience and the desired form of learning. flexibility as outlined above, the importance of flexibility in the literature suggests that it would have a beneficial effect for learners. the course described here did make studying more flexible; for example, students used webct to access course materials from home, and from other sites, and at a range of times outside scheduled teaching hours. this helped balance the workload for the module against other course commitments. however, flexibility was not a straightforward virtue. firstly, although students valued the potential of webct'to support flexible study, they did not necessarily use it in this way. from my point of view, it's got to be [flexible] that's a big bonus. particularly if you're travelling. but on a practical level i've only accessed it when i'm in here. i mean, it still allows me some flexibility, i can do it when i've got a moment. i really should be doing it, for the sake of coming in. in fact, being able to do work 'when i've got a moment' was a problem for two of the three students, who would have preferred scheduled sessions: 57 ruth holt, martin oliver and claire mcavinia using web-based support for campus-based open learning 1: i think if we have a proper session for the webct, if we have [this], then we have, we can . . . [pauses] if we can do it whenever we are free, then some . . . we keep on doing something else rather than doing webct. so if we have a proper class for webct i think it's a bit better. 2: yes, it's better at pushing you, you know. do it, do it! 1: yeah. you have to do it. [laughs] with all three students, increasingly flexibility led to work being put off, which they acknowledged would cause problems in the longer term. assessment was seen as being particularly important as a way of counteracting this tendency. i personally prefer to be kicked up the backside to do something continually because then i know by the end i'll have done it. whereas most of us, out of human nature, would say, i don't have to do that now. and then you get into a mad panic come june or whatever. even with assignments, however, observation and discussion with the students showed that no matter how long the lead-time the work was completed as close to the deadline as possible. what these experiences illustrate is that there is a need for a balance between flexibility and structure in courses, that this may be difficult to achieve, and that it will vary according to the needs of individual students. online discussions in spite of drawing on a range of guidelines for practice (such as salmon, 2000), online discussions did not work well. the students felt that the structured format reduced the exchanges to a transmission of information. you know, there isn't enough discussion that goes on. there's too much . . . sometimes, you know, we might as well just have a handout. the students did acknowledge that this was, at least in part, a consequence of their own choices. i mean, the opportunity to discuss is there, but we as students don't always take up that challenge perhaps. in part, they viewed this as a consequence of the fact that their contributions to the discussion board were unassessed, and thus were viewed as being optional. that's why you see our postings up for these things is much poorer than the assignments. if they are not graded. yes, they are important, it's just that a lot of important points there, and you have to follow it up, but it's not graded, so . . . [pause; laughs]. that's the point of view, you know. you just put it as something for when you have time, and you want to have some fun, you have the mood to do it. another reason for this choice was that they felt that online discussion was inferior to faceto-face exchanges: i don't think it's the best way to discuss matters, i prefer to do it around a table. you see the answer that much quicker coming back, you get more from it, and if you don't 58 alt-j volume 10 number 2 understand something you can ask directly. that's my point. i don't see that as a strength of webct. i don't use it for discussion. i just don't. i use it for the learning tools, like the assignments. if i want to discuss, then i'll do it in here, because you get so much more out of it. it's a barrier for me to discuss something like that. given that this was a hybrid course it was simply easier and better to discuss issues when they met. however, they agreed that such discussion was better than nothing. i think that if it's the only option you've got for discussion then it is better than none at all, but as a replacement for sitting around a table, i think it's got a long way to go. i described it as a barrier because it's a barrier as opposed to just doing this sort of conversation. but it's a facilitator if there's no other way of conversing, isn't it. it just depends on what standpoint you're looking at it, and i'm absolutely, you know, sure that given the alternative between only having a once a month visit to your tutor and having the opportunity to communicate through a fairly instantaneous medium then that is a step in the right direction. but i don't think it replaces sitting round and discussing it in a group. importantly, this reaction against the online discussions seemed to represent an exercise of power by one of the students. of the three, the second and third (as described in the section on self-assessment) had contributed sporadically to the online discussions. in particular, the second student had posted several long, thoughtful messages. this contrasted sharply with her more hesitant, infrequent contributions in the face-to-face discussion. she explained that this was because she was able to reflect before writing, and was able carefully to compose and revise messages before sending them. this was something that she felt to be of great value given that english was not her first language. by way of contrast, the first student had avoided the online discussions wherever possible. he dominated traditional group discussions, partly because of his quick wit, but also because he was more fluent and articulate than the other two students. (he was the only one of the three for whom english was the first language.) he identified his dyslexia (which he had previously kept hidden from the group and the tutor) as one particular reason for preferring to speak than to write. thus the face-to-face discussions reinforced his position of control in the group, whereas the online discussions allowed the other two students and particularly student two to overshadow him. it would be naive to argue, on the basis of this case, that moving discussions online will thus eradicate or even diminish power structures within groups. however, what this study vividly illustrates is that such a move will change these structures in ways that are influenced by personality, background and fluency in writing and speech. as a result, these characteristics are also likely to inform students' preferences for (and the potential success of) a particular medium for discussion. other issues in addition to these main themes, several other points were noted. firstly, although students found some elements of webct frustrating, overall they were satisfied with their use of it. 59 ruth holt martin oliver and gaire mcavinia using web-based support for campus-based open learning it's not a problem it's just that when you don't know something, it's so annoying that it takes so long to work it o u t . . . you end up swearing at the screen, don't you, for some little thing like that. i think it's a pretty good sign for the system if between the three of us the only major concern is a little glitch like the type of file that has to be submitted. i mean, it does suggest that it's simple to use. moreover, they valued the novelty of this approach. i think that webctis another way to add to the variety of the learning process. methodologically, the focus group also highlighted the fact that logs of students' use of the system gives a flawed and partial account of their experience of the course, echoing the warnings of jones (1998). i tend to spend most of the time on the assignments. i tend to print the content anyway, so i've got a record of it. conclusions at a superficial level, the study demonstrated the success of using webct to deliver courses of this type. the students all completed the course, and were generally well disposed towards the use of the web to support their learning. however, closer examination shows that the course design, which had been carefully based on recommendations from the literature, caused a number of problems. what this study illustrates is that not all students are equally well suited to learning from web-based materials. thus the success of any such approach will be influenced by students' personalities and educational history as much as by the design of the course. the wider implication of this is that courses should, ideally, reflect the characteristics of the individual students on the course. the students described here (two non-native english speakers, and one dyslexic student) were not 'typical' but this is the point. no student is 'typical', even if their distinctiveness is not initially obvious (as was the case here with the dyslexic student); each will bring with them personal histories and dispositions that shape their interaction with the course, and will thus determine its effectiveness. in effect, then, the idea of 'student-centred learning' should be moved from being simply a set of pedagogic strategies applied to all students, to being an integral part of course design during which the suitability of all teaching methods, including those not traditionally considered to be student-centred, should be considered. perhaps more insidiously, this conclusion also implies that, given the increasing diversity of students (even within the context of specific courses), it may be impossible to create any single course that is well designed for each and every student in a cohort. also linked to the issue of individual educational history were the problems associated with the introduction of self-assessment. this was entirely new for the students on this course, and thus whilst it had the potential to provide considerable benefits, its success was limited because it was unfamiliar and challenged deeply held assumptions about the way education 'ought' to operate. 60 alt-] volume 10 number 2 flexibility, which is almost ubiquitously claimed as an advantage for courses involving the web, proved to be a mixed blessing. although students spoke of the value of studying when they liked, they also admitted that the reality was that they typically accessed materials on campus, and organized their work around deadlines. moreover, they indicated that more, not less, structure to the course would have helped them use their time more effectively. thus the potential of the web to support 'any time, any place' learning became not simply irrelevant but also detrimental to the students' management of their learning. this would imply that the current advocacy for flexible courses may have gone too far in its attempts to address the dilemma of structure identified by rowland (1993:45-6). the study also echoed other research which has emphasized the centrality of assessment in students' perceptions of courses. given that students organized their time around deadlines, the number and pacing of assignments will create the structure of the course, at least in terms of student work. inherent in all of these issues is one common problem: students' commitment of time to a course comes at the expense of time which might otherwise be spent elsewhere. this observation allows the often-criticized phenomenon of student instrumentalism to be reframed not as laziness or superficiality, but as a shrewd way of investing their personal resources (bourdieu, 1986). it is only rational to avoid making any more effort than necessary, by opting for the most familiar, convenient ways of completing the course. the short-term planning witnessed here illustrates that a lack of clear course structure can add to, rather than diminish, this tendency to avoid committing time to the course, at least for those students who do not find study intrinsically motivating. this has significant implications for the kind of hybrid course described here. where a web-based course is used to support campus-based education, students may well drop elements that they find difficult in favour of easier, more familiar options. unless they feel an inherent motivation to explore and experience these new teaching methods, it will be necessary to ensure that the innovative elements are either mandatory or easier for all the students involved. references biggs, j. (1999), teaching for quality learning at university, buckingham: open university press. bourdieu, p. (1977), outline of a theory of practice (2001 edition; translated by r. nice), cambridge: cambridge university press. bourdieu, p. (1986), 'the forms of capital', in j. g. richardson (ed.), handbook of theory and research for the sociology of education, new york: greenwood press, 241-58. boyle, t., and cook, j. (2001), 'online interactivity: best practice based on two case studies', alt-j, 9, 1, 94-102. bradley, c. and oliver, m. (2002), 'the evolution of pedagogic models for work-based learning within a virtual university', computers and education, 38, 37-52. evans, a. (2001), 'assessing competence in surgical dentistry', british dental journal, 190 (7), 343-6. 61 rutfi holt martin oliver and claire mcavinia using web-based support for campus-based open learning grigg, p. and stephens, c. (1998), 'computer-assisted learning in dentistry: a view from the uk', journal of dentistry, 26, 387-95. hanna, d. (1998), 'higher education in an era of digital competition: emerging organizational models', journal of asynchronous learning networks, 2 (1), 66-95, http://www.aln.org/alnweb/journal/vol2_issue1/hanna.htm, url last visited 5 april 2002. hiltz, s., coppola, n., rotter, n., turoff, m. and benbunan-fich, r. (2000), 'measuring the importance of collaborative learning for the effectiveness of aln: a multi-measure, multi-method approach', journal of asynchronous learning networks, 4 (2), http.//www. aln.org/alnweb/journal/vol4_issue2/lelhiltz/le-hiltz.htm, url last visited 5 april 2002. jones, c. (1998), 'evaluating a collaborative online learning environment', active learning, 9, 31-5. kay, e., silkstone, b. and worthington, h. (2001), 'evaluation of computer aided learning in developing clinical decision-making skills', british dental journal, 190, 554-7. kvale, s. (1996), interviews: an introduction to qualitative research interviewing, london: sage. lechner, s., thomas, g., bradshaw, m. and lechner, k. (2001), 'planning oral rehabilitation: case-based computer assisted learning in clinical dentistry', british dental journal, 191 (3), 152-6. nikolova, i. and collis, b. (1998), 'flexible learning and design of instruction', british journal of educational technology, 29 (1), 59-72. nixon, t. and salmon, g. (1996), 'computer-mediated learning and its potential', in r. mills and a. tait (eds), supporting the learner in open and distance learning, london: pitman publishing. oliver, m. (2000), 'evaluating online teaching and learning', information services and use, 20, 83-94. rowland, s. (1993), the enquiring tutor exploring the process of professional learning, london: falmer press. salmon, g. (2000), e-moderating: the key to teaching and learning online, london: kogan page. shaikh, a. and macaulay, l. (2001), 'integrating groupware technology into the learning environment', alt-j, 9 (2), 47-63. welbury, r., hobson, r., stephenson, j. and jepson, n. (2001), 'evaluation of a computer-assisted learning programme on the oro-facial signs of child physical abuse (non-accidental injury) by general dental practitioners', british dental journal, 190 (12), 668-70. 62 calt624997 179..189 talking back to theory: the missed opportunities in learning technology research sue bennetta and martin oliverb* afaculty of education, university of wollongong, wollongong, australia; blondon knowledge lab, institute of education, london, uk (received 31 january 2011; final version received 14 september 2011) research into learning technology has developed a reputation for being driven by rhetoric about the revolutionary nature of new developments, for paying scant attention to theories that might be used to frame and inform research, and for producing shallow analyses that do little to inform the practice of education. although there is theoretically-informed research in learning technology, this is in the minority, and has been actively marginalised by calls for applied design work. this limits opportunities to advance knowledge in the field. using three examples, alternative ways to engage with theory are identified. the paper concludes by calling for greater engagement with theory, and the development of a scholarship of learning technology, in order to enrich practice within the field and demonstrate its relevance to other fields of work. keywords: theory; design; learning technology introduction this paper poses the question: why should we be concerned with theory? to answer this, a review is provided of the ways in which theory has – and has not – been engaged with in learning technology research. this is followed by three cases, in which different ways of engaging with theory are offered. the paper concludes by identifying ways in which work in this field frequently fails to engage with theory, and how this situation could change, creating a more dynamic relationship between theory and practice. background: theory and pragmatics in learning technology research theory has had a relatively small role to play in learning technology research to date. mostly, research has focused on matters of practical implementation and design, largely driven by ‘common-sense’ assumptions about what technology can achieve, or – for many decades – by hype and excitement rather than evidence or theory (mayes 1995). for example, reviews (for example, conole, smith, and white 2007) have shown that visible, tangible investment – typically purchase of hardware or software – has been the priority for funding over a 45-year period, not educational principles. only later, if at all, has work followed that addresses the *corresponding author. email: m.oliver@ioe.ac.uk research in learning technology vol. 19, no. 3, november 2011, 179–189 issn 2156-7069 print/issn 2156-7077 online � 2011 association for learning technology http://dx.doi.org/10.1080/21567069.2011.624997 http://www.tandfonline.com patchy and inconsistent patterns of use that typically follow such investment. such reviews make it clear that fashion and markets (selwyn 2007), rather than principles or theory, remain the major driving forces behind much research, development and implementation work in this area: research has a tendency to follow policy directives and technological developments, rather than informing them [. . .] sadly there has been too much evidence of knee-jerk policy, which does not take account of evidence arising from research. (conole, smith, and white 2007, 53) this gap is not only visible in relation to policy. according to friesen (2009), research in this field generally treats theory and empirical work as separate, rather than as integral parts of the same endeavour. friesen offers a useful conceptual framework for this discussion, identifying three different traditions of research. drawing on habermas, he distinguishes between instrumental (concerned with technical interests associated with work or production), practical (concerned with interpretation or meaning) and emancipatory traditions. he goes on to argue that only the instrumental is well represented in this field, and that the conspicuous absence of theory is an important indication of this: to use the words of educational technologist rob koper [. . .] this research tends not to be ‘theory-oriented,’ but rather ‘technology-oriented’ in character. e-learning research, koper (2007) explains, is not focused on “predicting or understanding events [in] the world as it exists” (p. 356); it instead seeks to “change the world as it exists” (p. 356; emphasis added). e-learning or technology-oriented research, in other words, attempts “to develop new technological knowledge, methods, and artifacts” for practical ends or purposes (p. 356). it is this applied, practical, and technological research that koper (2007) says is ideally suited to e-learning. (friesen 2009, 7) another indication of this pragmatic orientation can be found in conole and oliver’s (2007) introduction to the field. they identify four groups of issues within learning technology research: pedagogic, technical, organisational and sociocultural. arguably, using friesen’s categories, the first three of these could be classified as instrumental, and even the fourth could be viewed as a means to instrumental ends. theory or even critique remains conspicuously absent. this current situation reflects a long-term preoccupation with practical problems. hawkridge’s historical account of the development of educational technology as a field (2002) illustrates this. the review draws together previous reviews, meta-studies and content analyses, grouped by continental tradition and stretching back to work in the late 1960s. he describes how a north american tradition grew from instructional design, which was founded on objectivism and linked to industrial and military uses of systems analysis. as this grew through the 1960s and 1970s, it established connections to broader educational work focused on the curriculum and on the science of teaching. the result of these links was the development of programmed instruction: a pragmatically oriented technique for enhancing learning outcomes using behaviourist principles. a consistent emphasis throughout this period was on the scientific improvement of the practice of teaching, largely through automation. (a fuller account of this is offered by saettler 1990.) research in other english-speaking countries was strongly influenced by the north american tradition, drawing heavily on us texts. systems-based 180 s. bennett and m. oliver approaches were also well represented in the united kingdom; for example, in the work of pask (for example, 1976). although the role of theory has broadened since this early period, hawkridge argues that the primary focus of learning technology research into this century has been pragmatic, concentrating on applications of new technologies and the pursuit of behavioural evidence of improved learning outcomes. in spite of the growing popularity of qualitative studies of constructivist inspired learning environments since the early 2000s, which could have offered other models of research, this emphasis on practical applications persists and indeed has been periodically reinforced by commentary such as that of reeves: to realize the fullest potential for online learning, our methods of research and development must be fundamentally changed, but additional changes are needed. first, we must shift from a position that views learning theory as something that stands apart from and above instructional practice to one that recognizes that learning theory is collaboratively shaped by educational researchers and practitioners in context. educational technology is a design field, and thus, our paramount goal of research should be solving teaching, learning, and performance problems, and deriving design principles that can inform future decisions. our goal should not be to develop esoteric theoretical knowledge that we expect practitioners to apply. this has not worked since the dawn of educational technology, and it won’t work in the future. (2005, 304) this continued focus on practical ‘use-inspired’ design research is promoted as ‘socially responsible’ (reeves, herrington, and oliver 2005), to be valued above and pursued in preference to other forms of research. this view, advocating the type of learning technology research that should be done, limits possibilities for advancing the field. while design research works well for investigating the effectiveness of a particular design in practice, and so helps to inform instructional design theory, it has little relevance to non-design problems. even the emergence of design research as a methodology – one with a “focus on advancing theory grounded in naturalistic contexts” (barab and squire 2004, 5) – has done little to change this, since its focus continues to be on “developing a profile or theory that characterizes the design in practice” (2004, 4), rather than on wider concerns. reigeluth and frick’s (1999) distinction between instructional design theory and descriptive theories of learning is helpful here, drawing attention to the scope (and hence the limitations) of a design research focus. seeing learning technology research as primarily – or even exclusively – occupied with developing and testing designs misrepresents the breadth of work actually being undertaken (czerniewicz 2010). to illustrate these wider concerns, thorpe’s review (2002) is helpful, since it is focused directly on theory and pedagogy. she describes how, over the previous decade, pedagogic thinking shifted from a focus on materials and instruction to social competence, collaboration and situated performance, mirrored by a shift in the theories used to justify the work from behaviourism to social constructivism. this account can be understood as raising some modest challenges to friesen’s critique, suggesting that practical and emancipatory research have in fact called instrumental ‘progress’ into question – albeit only for a few restricted areas of work – by suggesting that some changes may not always be seen as improvements. so, for example, thorpe describes how accounts (many of which are related to positions such as constructivism) that celebrate increased student independence and autonomy can be critiqued as excuses for leaving students isolated and unsupported. this is not the research in learning technology 181 kind of challenge that can be answered with more empirical data; instead it needs, we suggest, a position to be taken about what is desirable and why. in other words, it needs theorising. this account illustrates how research in the field can focus on questions other than design. as czerniewicz’s review (2010) demonstrates, while instructional design is sometimes positioned as the ‘core’ of work in learning technology, the field is really too diverse and fragmented for such a claim to be credible. she points instead to evidence that theory is brought in through links to numerous other fields, and development “takes the form of new languages which offer fresh perspectives and a new set of connections, rather than integration with existing theories and approaches” (czerniewicz 2010, 524). treating theory as if it was simply and solely a foundation for applied design fails to represent the richness of work that can, and sometimes is, undertaken within this field. it leaves theory unchallenged – and often unquestioned – so that empirical work supports or illustrates theory, but is not seen to develop or even provide a basis for rejecting it (cook 2002). such work is possible, however, as the cases in the next section will demonstrate. case studies of engagement with theory as argued above, much work in the field of learning technology either neglects theory or else operates in a derivative way, simply applying it. however, there are examples of kinds of work that show different forms of engagement with theory. three examples are given here, showing how empirical work can develop theory; how theories and evidence can undermine claims and redefine design problems; and how theory can change the way that a phenomenon is understood. case one: mutually informing theory and practice richard mayer’s body of work on multimedia learning exemplifies a particular branch of learning technology research based on the kinds of psychological theories and approaches that have been popular since the early years of research into computer-based or computer-assisted learning. over his career mayer has produced an extensive body of work that has drawn on traditional scientifically based experimental methods to advance understanding about how interactive multimedia influences learning. many well-recognised principles of multimedia design stem from mayer’s findings. mayer’s research illustrates how empirical work can define, rather than apply, theory. it builds on the desire “to understand how people integrate verbal and visual information” (mayer 1997, 4) as the basis for improving multimedia design; something that requires a twin focus on theory and practice. mayer, describing his theoretical approach (1997), explains how this work built on previously established theories of cognitive processes, which he applied to the new environment of multimedia. the literature contains a series of more than 40 studies that are informed by previous results and incrementally develop refinements and extensions to that theoretical approach (for example, mayer and chandler 2001; mayer and johnson 2008; mayer and moreno 2003). while, at one level, this work fits within the traditional of learning technology research as informing the design of instructional software, it does this in a dialogic way. theory does not simply inform design; mayer’s findings do generate design principles, but they also reshape the underlying theory. 182 s. bennett and m. oliver it would be easy to dismiss mayer’s work as ‘out of step’ with current thinking in learning technology research. for example, research into content-driven, direct instruction of the type with which mayer was concerned has been replaced by a focus on constructivist approaches. moreover, critics have labelled experimental methods in learning technology research misguided (clark 1983) and pseudoscientific (reeves 1993), arguing that learning is too complex to allow for the proper isolation of variables required by the scientific method. it is also claimed that experimental studies are conducted in environments so artificial that their findings have little relevance to real classrooms, and that researchers must over-reach in making claims about the wider applicability of their findings. such criticisms have been important in establishing qualitative research as a viable and valuable approach; however, it ignores the status of mayer’s design recommendations: they are provisional and tentative, rather than being nomothetic laws, created to guide design rather than prescribe it. problems that arise when using them in complex classrooms should therefore be fed back into the theory, allowing it to be refined, rather than being seen as evidence of its failure. mayer’s work thus has something important to offer a discussion of the role of theory in learning technology research. mayer has achieved a coherent, interconnected body of work that is informed in every way by the theory in which it is located, from its conception (building directly from unanswered questions raised by past research and under-developed theory), to execution (the methodologies used), and to interpretation (what it means in relation to the questions asked, earlier findings and future research). this work is narrow in its focus, and appropriately modest in its claims, limited to studies and applications of how people understand verbal and visual information in combination. however, mayer’s work demonstrates the kind of two-way conversation in which the empirical speaks to theory too, developing it. this kind of dialogue between empirical studies and theory is largely absent from learning technology research, according to the reviews provided above. through this reciprocity, mayer’s theory seeks to explain phenomena, and observation of phenomena is used to test and develop theory. through theory his studies connect to a wider body of related research on cognitive processes in computerbased learning. in sum, mayer’s work demonstrates how theory can inspire, frame and guide research that cumulatively builds knowledge in a highly focused area. case two: reframing design problems games and virtual worlds have become a popular topic within learning technology research, and much has been written about their educational potential (for example, mcfarlane, sparrowhawk, and heald 2002). it is claimed, for example, that digital games have developed powerful and effective pedagogies that education would benefit from adopting (gee 2005). in other words, games have developed design principles that could usefully be applied in educational contexts. however, there are two problems with this. the first can be revealed by using other theories to re-interpret the situation. for example, cultural theory suggests that it may be wrong to think of this as a design problem at all. this conception of the situation rests on an assumption that may simply be inappropriate: that education is failing students and that games are “a kind of remedy [. . . so that] the diseased, geriatric body of education can be treated through the rejuvenating, botox-like effect of educational game play” (pelletier 2009). if this kind of ‘treatment’ account fails research in learning technology 183 to hold – for example, because forced play is an oxymoron, and so requiring students to use games cannot credibly be seen as meaningful play – then the whole endeavour is misconceived. secondly, such ‘hopes’ are often perpetuated by claims about games’ potential. these claims are based on popular perception rather than theory or evidence arising from specific examples of play (squire 2002). in order to rethink the relationship between games and learning, and avoid the tendency to discuss both games and learning in the abstract, a situated approach was developed to studying what people learned when they played digital games. activity theory was used as the basis for this, since this allowed recorded performances to be characterised and analysed as purposeful, goal-oriented activities (oliver and pelletier 2006). the alternative – ‘common-sense’ accounts of ‘potential’ – was simply too vague to be studied and judged with any hope of credibility. the theory of expansive learning (engeström et al. 1996) positions learning as a creative, situated process in which social practices are adapted to cope with challenges or problems. building on this, evidence of learning during play was generated by identifying breakdowns, such as failed attempts to progress through a level, and then looking to see whether there was evidence that this problem had been overcome. where it had, the new approach to play was given a descriptive label and added to the expanded list of strategies used by the player. this resulted in specific, situated strategies (e.g. “spot unusual objects and click on them”) and three broader categories that were positioned as different kinds of learning: learning to use tools skilfully, learning about objects and spaces within the game, and learning the tactics and strategies needed to progress through the game (e.g. how to identify and respond to ‘set piece’ encounters, even when these differed representationally). subsequent work with a more complex game (oliver and pelletier 2005; pelletier and oliver 2006) provided evidence about transfer of strategies from previous play, but also about inappropriate transfer and of forgetting successful strategies. this meticulous building and categorisation of lists provided a credible and transparent evidence base for the claims that were made. unlike earlier claims, these were empirically grounded and culturally specific; they showed how learning happened when playing digital games, but also showed how situated it was. this helped to explain why games could not simply be ‘injected’ into curricula (de freitas and oliver 2006). moreover, these studies did not simply apply theory to inform design. conventionally, analyses using activity theory operate at the level of case studies, characterising the practices of a bounded group of actors over period of time, as (for example) in engeström’s change laboratories (engeström et al. 1996). in the cases referred to here, however, the focus is on individuals playing games for a period of hours – a relatively ‘micro’ focus, in which the influence of community (e.g. designers, other players) is either absent or mediated by the software and hardware. the result of this is a focus on evidence of learning through changed behaviour – something more akin to skinner’s behaviorism than to conventional socio-cultural accounts of learning. while oliver and pelletier’s work (2005, 2006) is case-study based, it uses this evidence to challenge widely-made claims; linking these to an established tradition of work by not only applying an existing approach in a new field, but also exploring, in a modest way, how the scope and utility of that approach could be developed. in this sense, it also provided an opportunity for dia184 s. bennett and m. oliver logue with the foundational tradition of work in which the theory had originally been developed. case three: theory to frame research within a wider context chen’s (2010) phd study into chinese international students’ experiences of online learning was prompted by a desire to frame a study of technology within a wider context. the research could have been conducted as a descriptive case study of the specific experiences of these learners of this type of online delivery, but theory allowed it to be conceptualised as a ‘case’ of something more general: a case of acculturation. this kind of reframing allows a study of technology use – something apparently only of interest within the field of educational technology – to be repositioned as relevant to an entirely new audience. current learning technology research includes many in-depth case studies of how a technology has been applied to education. findings from these cases studies often consist of detailed descriptions, sometimes concluding with a list of emerging themes or issues. these have obvious practical value: when presented appropriately, such research supports the kind of “naturalistic generalizations” that stake (1995) suggests can happen when readers compare a case study with their own experiences and draw conclusions. however, there is usually little attempt made in the analysis of such case studies to draw conclusions of the kind suggested by yin (2009, 43) that can “generalize a particular set of results to some broader theory”. this is because this type of casestudy research is driven by an interest in investigating a practical problem (e.g. how to use blogs in higher education); a pragmatic focus that often has little connection with theory. as a consequence, particular cases have no way to ‘speak’ to a wider body of research. they remain bound to the particularities of the case context. this final example demonstrates how theory can allow a case study to be positioned as part of a wider body of work, as well as bringing coherence to the study itself. in this case, the use of berry’s (1997, 2005) model of acculturation as an organising framework led chen to explore what chinese students ‘brought’ in terms of their assumptions about and past experiences of education in china (their heritage culture) to their experiences of australian higher education (their host culture), as enacted in the online learning environments they found themselves in (contact between cultures). this illustrates the use of theory to structure the study; theory provided an orienting framework to shape data collection and analysis by providing a lens that determines which aspects of the context should be attended to and why they are important to understanding the phenomenon. seen through berry’s model, the study focused on collecting data that could characterise the two cultures and the outcomes of their contact. in this way, the theoretical lens shaped the research design, and in so doing focused the researcher’s attention on particular aspects of the context. as part of her analysis, chen compared her results with the theory’s predictions to determine how well the theory explained the outcomes of students – an example of yin’s idea (2009) about how the empirical can ‘speak’ back to theory. chen further extended her analysis to draw on theoretical constructs from bernstein (1977) and maton (2006). the empirical work thus became a site in which previously unconnected theories were brought together; and by doing so demonstrated how theories from psychology and sociology can work in tandem to research in learning technology 185 conceptualise different aspects of a phenomenon and overcome limitations in each. furthermore, each construct came from a broader theory, although chen drew on only what was needed and useful for the scope of her study. this study thus demonstrates three things. the first is the value of theory in specifying and advancing understanding of a particular phenomenon. the second is the way that this individual study can be connect to broader concerns (in this example, acculturation of overseas students, how clashes in educational cultures occur, how different forms of knowledge and knowing are valued), not just remain of interest to those concerned with the specific context in which the study took place. thirdly, it emphasises the partial nature of theories: rather than merely applying one and expecting data to conform to it, theories were judged for their explanatory potential and combined or extended as necessary. chen (2010) demonstrates how this is possible within the scope of a phd, showing how the integration of theory is achievable and valuable even within a single study. discussion as the initial review showed, the use of theory in the field of learning technology has, primarily, been about its application to solve practical problems. where dialogues between studies and theory have occurred, these are mostly in the context of design – again, an instrumental end focused on pragmatics. there is nothing wrong with pragmatic research – however, the examples provided here show that more is possible. research in the field need not be defined by policy, technological developments or popular discourse. it is not always applied; and it is not always design oriented. the examples here echo and develop the points made in czerniewicz’s (2010) review of research in the field. research can draw on theories from psychology, sociology, cognitive science and education. more importantly, however, these examples show that theory is useful for more than just improving applied design practice. instead, it has a role to play in creating a coherent, bounded scope for the work; thinking through what would count as credible evidence; managing and interpreting data; rethinking whether the problem has been framed correctly in the first place; situating the study in relation to wider concerns and issues, such as work outside the immediate field; and developing the theories themselves. the examples have also highlighted that not all theory is ready to apply; some requires working through, adapting or selecting from in order to make it relevant and practical. as all three of the examples above show, use of theory requires taking a position: it requires judgement about which parts are useful, which are unnecessary, and which need development. in other words, the use of theory requires engagement, not just application. there are useful parallels to be drawn here with at least three of boyer’s categories of scholarship (1997): discovery; integration; and application. (the case for teaching is arguably less direct.) a scholarly approach to research in learning technology should build knowledge, including new theories; integrate work from different disciplines in a thoughtful way; and apply it to practical problems and concerns. calls to restrict research in the field to applied, design-based, problem-solving studies, in which theory is applied rather than challenged or developed, limit the kinds of work that are valued. as czerniewicz (2010) identifies, such calls are ideological: they legitimate particular research agendas at the expense of others. as 186 s. bennett and m. oliver friesen argues (2009), this misses out on other kinds of work that may also be important: work that asks whether we are tackling the right kind of problem in the first place, or who stands to benefit from these developments, for example. however, this does not mean that practical research is undesirable, or that all work should be theoretical. to suggest that all research in the field should engage with theory is just as ideological as proposing that it should not. rather, as freisen (2009) and czerniewicz (2010) both suggest, the field may be better served by valuing different kinds of work, and developing our capacity to engage with theoretical as well as pragmatic issues. work that is instrumental is useful, and has its place. so too, however, does work that is interpretive, critical or emancipatory, and these under-represented alternatives need valuing and encouraging so as to counterbalance the increasingly dominant position of theory-free or theory-applying work. conclusions research in learning technology has focused on practical, instrumental concerns, to the detriment of its ability to engage with theory. previous reviews have show that theory has been relatively neglected, with most of those examples that do use theory best characterised as applying it rather than engaging with it in a critical or scholarly way. this situation risks turning the field into a narrow and derivative area of work: at best, only able to draw from other areas; and at worst, only of relevance to those with a vested interest in the specific practical situation currently under study. this has resulted in many missed opportunities to ‘speak back’ to theory. learning technology, as a field, is about more than developing better instructional design processes, and this diversity needs to be recognised and valued. we need to be able to develop theory, question it and even reject it if necessary; we also need to develop the capacity to question whether work in the field serves particular interests rather than others, and whether this is appropriate. learning technology research could also be used to make connections with other fields. this could be achieved through the integration of theories drawn from other areas; it could also involve serious engagement with and development of the ideas and approaches that are used. such engagement would demonstrate the relevance of learning technology research to education, psychology and other social sciences, as well as enriching our understanding of the problems that we face. references barab, s., and k. squire. 2004. design-based research: putting a stake in the ground. journal of the learning sciences 13, no. 1: 1–14. bernstein, b. 1977. class, codes and control. volume 3. towards a theory of educational transmissions. london: routledge and kegan paul. berry, j. 1997. immigration, acculturation and adaptation. applied psychology: an international review 46, no. 1: 5–34. berry, j. 2005. acculturation: living successfully in two cultures. international journal of intercultural relations 29, no. 6: 697–712. boyer, e. 1997. scholarship reconsidered: priorities of the professoriate. san francisco: jossey-bass. chen, r. 2010. knowledge and knowers in online learning: investigating the effects of online flexible learning on student sojourners. doctoral diss., university of wollongong, australia. http://ro.uow.edu.au/theses/3099/ (accessed january 6, 2011). research in learning technology 187 clark, r.e. 1983. reconsidering research on learning from media. review of educational research 53, no. 4: 445–59. conole, g., and m. oliver. 2007. introduction. in contemporary perspectives on e-learning research, ed. g. conole and m. oliver, 1–18. london: routledgefalmer. conole, g., j. smith, and s. white. 2007. a critique of the impact of policy and funding. in contemporary perspectives on e-learning research, ed. g. conole and m. oliver, 38–54. london: routledgefalmer. cook, j. 2002 the role of dialogue in computer-based learning and observing learning: an evolutionary approach to theory. journal of interactive media in education 5. http:// jime.open.ac.uk/article/2002-5/88. czerniewicz, l. 2010. educational technology – mapping the terrain with bernstein as cartographer. journal of computer assisted learning 26, no. 6: 523–34. de freitas, s., and m. oliver. 2006. how can exploratory learning with games and simulations within the curriculum be most effectively evaluated? computers & education 46: 249–64. engeström, y., j. virkkunen, m. helle, j. pihlaja, and r. poikela. 1996. the change laboratory as a tool for transforming work. lifelong learning in europe 1, no. 2: 10–7. friesen, n. 2009. rethinking e-learning research. new york: peter lang. gee, j. 2005. learning by design: good video games as learning machines. e-learning 2, no. 1: 5–16. hawkridge, d. 2002. educational technology research in higher education. internal report, open university, milton keynes, uk. maton, k. 2006. on knowledge structures and knower structures. in knowledge, power and educational reform: applying the sociology of basil bernstein, ed. r. moore, m. arnot, j. beck, and h. daniels, 44–59. london: routledge. mayer, r. 1997. multimedia learning: are we asking the right questions? educational psychologist 32, no. 1: 1–19. mayer, r., and p. chandler. 2001. when learning is just a click away: does simple user interaction foster deep understanding of multimedia messages? journal of educational psychology 93, no. 4: 806–12. mayer, r., and c. johnson. 2008. revising the redundancy principle in multimedia learning. journal of educational psychology 100, no. 2: 380–6. mayer, r., and r. moreno. 2003. nine ways to reduce cognitive load in multimedia learning. educational psychologist 38, no. 1: 43–52. mayes, j.t. 1995. learning technology and groundhog day. in hypermedia at work: practice and theory in higher education, ed. w. strang, v.b. simpson, and d. slater. canterbury: university of kent press. mcfarlane, a., a. sparrowhawk, and y. heald. 2002. the role of games in education: a research report to the dfes. cambridge: teem. http://www.teem.org.uk. oliver, m., and c. pelletier. 2005. the things we learned on liberty island: designing games to help people become competent game players. paper presented at digra 2005: the digital games research association’s 2nd international conference, changing views: worlds in play, june 16–20, in vancouver, bc, canada. oliver, m., and c. pelletier. 2006. activity theory and learning from digital games: developing and analytical methodology. in digital generations: children, young people and new media, ed. d. buckingham and r. willett, 67–88. london: lawrence erlbaum. pask, g. 1976. conversation, cognition and learning: a cybernetic theory and methodology. amsterdam: elsevier. pelletier, c. 2009. games and learning: what’s the connection. international journal of learning and media 1, no. 1: 83–101. http://www.mitpressjournals.org/doi/abs/10.1162/ ijlm.2009.0006. pelletier, c., and m. oliver. 2006. learning to play in digital games. learning, media and technology 31, no. 4: 329–42. reeves, t.c. 1993. pseudoscience in computer-based instruction: the case of learner control research. journal of computer-based instruction 20, no. 2: 39–46. 188 s. bennett and m. oliver reeves, t.c. 2005. no significant differences revisited: a historical perspective on the research informing contemporary online learning. in online learning: personal reflections on the transformation of education, ed. g. kearsley, 299–308. englewood cliffs, nj: educational technology publications. reeves, t., j. herrington, and r. oliver. 2005. design research: a socially responsible approach to instructional technology research in higher education. journal of computing in higher education 16, no. 2: 97–116. reigeluth, c.m., and t.w. frick. 1999. formative research: a methodology for creating and improving design theories. in instructional theories and models, ed. c.m. reigluth, 2nd ed., 633–51. mahwah, nj: lawrence, erlbaum associates. saettler, p. 1990. the evolution of american educational technology. englewood, co: libraries unlimited. selwyn, n. 2007. the use of computer technology in university teaching and learning: a critical perspective. journal of computer assisted learning 23, no. 2: 83–94. squire, k. 2002. cultural framing of computer/video games. game studies: the international journal of computer game research 2, no. 1. stake, r. 1995. the art of case study research. thousand oaks, ca: sage publications. thorpe, m. 2002. from independent learning to collaborative learning: new communities of practice in open, distance and distributed learning. in distributed learning: social and cultural approaches to practice, ed. m. lea and k. nicoll, 131–51. london: routledgefalmer. yin, r. 2009. case study research: design and methods. 4th ed. thousand oaks, ca: sage publications. research in learning technology 189 alt-j, research in learning technology vol. 12, no. 3, september 2004 issn 0968–7769 (print)/issn 1741–1629 (online)/04/030249–12 © 2004 association for learning technology doi: 10.1080/0968776042000259564 inquire: a case study in evaluating the potential of online mcq tests in a discursive subject sophie clarkea*, katharine lindsaya, chris mckennab & steve newb aacademic computing development team, university of oxford, uk; bsaïd business school, university of oxford, uk taylor and francis ltdcalt120305.sgm10.1080/0968776042000259564alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2004association for learning technology123000000september 2004sophieclarkeacademic computing development teamoxford university computing services13 banbury rdoxfordox2 6nnacdt@oucs.ox.ac.uk there has been a wealth of investigation into the use of online multiple-choice questions as a means of summative assessment, however the research into the use of formative mcqs by the same mode of delivery still remains patchy. similarly, research and implementation has been largely concentrated within the sciences and medicine rather than the more discursive subjects within the humanities and social sciences. the inquire (interactive questions reinforcing education) evaluation project was jointly conducted by two groups at the university of oxford—the saïd business school and the academic computing development team to evaluate the use of online mcqs as a mechanism to reinforce and extend student learning. this initial study used a small set of highly focused mcq tests that were designed to complement an introductory series of first-year undergraduate management lectures. mcq is a simple and well-established technology, and hence the emphasis was very much on situating the tests within the student experience. the paper will cover how the online mcqs are intended to fit into the oxford undergraduate study agenda, and how a simple evaluation was executed and planned to investigate their usage and impact. the chosen method of evaluation was to combine focus groups with automated online methods of tracking, and the paper discusses the findings of both of these. introduction multiple choice question (mcq) tests are one of the most widely used teaching tools, and have translated very successfully into the online environment; there are countless free and commercial products, and nearly every vle has the built-in capacity to deliver online mcq tests. uptake has generally been dominated by the sciences and medicine, with less interest from the humanities and social sciences * corresponding author: academic computing development team, oxford university computing services, 13 banbury rd, oxford ox2 6nn, uk. email: acdt@oucs.ox.ac.uk 250 s. clarke et al. (mckenna, 2001a). this has been attributed to a belief that mcq tests are not well suited to discursive subjects, and furthermore that they propagate in students the mindset that their education is about the retention and regurgitation of facts (mckenna, 2001b). the type of opinions expressed in mckenna (2001b) are typical of the reservations academics in these subjects have about mcq tests. they are not convinced as to the suitability of mcq tests for their subject area, and if mcq tests can be of use then they are applicable only at a very unsophisticated level: ‘i do think that if you want to understand the basics of a course, it’s a good way of getting a basic answer’ (mckenna citing weldon, 2001b). this paper outlines a small-scale case study at the saïd business school, university of oxford, which aimed to investigate further the potential for using online mcq tests to support a discursive subject. the intention for these tests, or rather informal quizzes, was to look beyond the application of mcq quizzes for testing the retention of numerous facts to see if they could be used to deepen student understanding, and to see how they might complement the traditional teaching methods used within the undergraduate course. the evaluation carried out concentrated on the impact of the formative assessment on student learning, as well as the student and staff preferences and attitudes where substantial work has been carried out in other studies (charman, 1999). background undergraduate teaching at the saïd business school is very much in the model of the traditional oxford tutorial system. students have short periods of intense formal practitioner input in the form of tutorials, and these are complemented by lectures. outside of this small amount of high-intensity formal input, students are expected to manage their own studies, including a large amount of directed and also self-discovered reading, writing essays for tutorials, and preparing for examinations. there are two aspects to this system that influenced the decision to investigate the potential for using mcq tests further. firstly, in common with most uk higher education institutions, there is an increasing pressure from students to provide as many learning resources as possible. in particular, students had requested additional ways in which to learn and judge their progress during periods of low contact time with their tutors— especially in the lead up to examinations. a second factor in the decision to investigate mcq tests is the importance of the material covered in the tutorials and lectures. a great deal of emphasis is placed on the tutorials and lectures, and it is important that the ideas and concepts presented in them are grasped by the student. there is little or no opportunity for this to be done at a later date, as least not as part of the formal curriculum. tutorials are viewed as high priority by both students and staff, whereas lectures are accorded lower priority by students when other commitments interfere, and so it was deemed especially important that the mcq quizzes would support students who did not make the lecture. the reasons above highlight why the business school decided to trial online formative assessment as a means of providing students with an additional resource evaluating the potential of online mcq tests 251 to support the cycle of lectures, essay-writing and tutorials. the pilot implementation of mcq quizzes to support the lectures is discussed here, and in particular the results of the evaluation carried out to investigate their first implementation in the undergraduate curriculum. the scope of the pilot involved the use of online mcq quizzes to support the ‘introduction to management’ course, taken by all undergraduate students. the study evaluated the use of an online mcq test alongside the lectures to investigate students’ use of the resources. the aim was to develop further quizzes to support more of the undergraduate curriculum if they proved successful. the mcq quizzes had several key characteristics. the questions themselves were carefully crafted to address the educational objectives outlined below. each individual potential answer for each mcq question needed to be able to have its own feedback, and in both the questions and answers there was a requirement for the ability to include hyperlinks and images. a variety of mcq software was available at the university, and software choice was regarded as relatively insignificant at the start of the project. however, the feedback and html inclusion requirements dramatically limited the choice of suitable software. the solution finally chosen was quia, an online activity creation tool hosted on the service providers web site. quia fulfilled the aforementioned requirements, and was also suitably easy to use for non-technical question authors. aims and objectives there were four areas in which it was thought that the mcq quizzes could help students, and the pilot tests were carefully written with these aims in mind. investigating the success in fulfilling these aims was used as the basis for choosing an evaluation methodology. the four aims it was hoped that the mcq quizzes could address are outlined below. the reinforcement of learning from other pedagogic elements as previously discussed, the pressure for lectures and tutorials to deliver effectively is high, but there are many factors that can contribute to whether or not a student succeeds with a particular subject. lectures can vary in quality across the course, and are more likely to fall by the wayside completely in terms of student’s priorities when other commitments press, such as assignment deadline and extra-curricular activities. logistically the structure of the course means that there can be many months between the last formal teaching input on a particular topic and the exam. it was envisioned that the mcq quizzes would provide students with a further opportunity to cover the lecture material when and where it suited them. for students who did not make the original lecture, they would have extra help in comprehending the lecture handout, and a way of measuring their own level of understanding. 252 s. clarke et al. cementing students’ understanding mcq quizzes have a role in terms of testing that students have grasped the key facts and ideas. additionally, this is a practical safeguard as the lectures are designed to challenge the students rather than merely transmit information. in addition to covering the core concepts, mcq questions can be used to cement students’ understanding of the more subtle points in the lectures, particularly through the use of the carefully constructed feedback. deepening students’ knowledge the mcq quizzes provide an excellent opportunity to present entirely new material to students, leading them beyond their reading lists. for the more curious students, they can provide some framing to new areas that there might not be time to cover in the tutorials, and can guide the student towards credible resources that they may not otherwise have found. necessary to this end is the ability to link out to online resources (the links made ranged from company web sites to journal articles in the department electronic library provision). framing educational expectations mcq quizzes provide another way for course leaders to convey to students (explicitly and implicitly) what is expected of them. many first-year undergraduate students are surprised and daunted to find that they are expected to read complex journal articles and whole books. the mcq quizzes can reinforce the message of what they are expected to read by basing questions on those materials and by linking out to a wider variety of literature than is present on the reading lists. in general, students can be pointed in particular directions without being obviously spoon-fed. example question and answer with feedback in the lecture, you learned that even in recent years the shift to working in large organizations continues. what do you think that this has meant for the number of hours that people work? in other words, do you believe that it is true or false that working for larger organisations means that people work longer hours than they would have if they were working for smaller companies or for themselves? (hint: you might want to consult this study on long term employment trends: http:// phe.rockefeller.edu/work_less/) •true •false answer: false feedback: for a critical view of the conclusion that people are working fewer total hours, see juliet schor’s book the overworked american. there were several potential pitfalls identified with respect to the mcq quizzes, and attempts were made to avoid these, particularly in the writing of the questions. there evaluating the potential of online mcq tests 253 is a danger that by picking out particular areas (either deliberately or inadvertently), the quizzes could send misleading clues to students about what is and isn’t important. this is exacerbated by the students’ tendency to be very strategic and exam-focused when considering how best to spend their study time. a second potential problem to avoid was writing overly simple questions with black or white answers, or indeed the opposite issue of inadvertently adding subtle nuances that some students may wrongly pick-up. to avoid these pitfalls questions were authored collaboratively and tested before they were made available to the students. methodology this section outlines the evaluation methodology used to consider the usage of the six pilot mcq quizzes over the course of a term. the aim of using evaluation techniques was to assess if the quizzes delivered any or all of the four perceived aims outlined above. this type of subjective evaluation is notoriously difficult to design and carry out. the expectation for the evaluation was to get an initial indication of how well the pilot tests had worked. from there any particularly promising, or interesting, aspects could be identified to be investigated in greater depth and with more resources (possibly with a larger group of students over more than one course). a range of evaluation methods were considered, and each assessed against its potential to illuminate how well the mcq quizzes had addressed the four primary aims. a focus on methods with a proven track record in educational research and the evaluation of technology-based resources led to a shortlist of six possibilities: system logs, automated tracking of link following, confidence logs, 1-minute papers, video observation, focus groups (angelo & cross, 1993; harvey, 1998). the shortlist was then considered again to judge which methods could address multiple aims at once. between the chosen methods they would be required to address all of the four aims, and to provide a combination of qualitative and quantitative data. access logs, the tracking of link following and focus groups were chosen as the most appropriate. techniques that attempted to quantify student knowledge in some way (confidence logs and 1-minute papers) were discarded on the grounds of difficulty of application within the resources available, and also because of the problems inherent in using these methods to judge improvement against the stated aims. video observation of students was also decided against as the focus was more towards the cumulative result of accessing all six quizzes rather than in analysing the specific answers chosen in any one sitting. similarly, there was no particular concern with software usability at this stage, an area in which video footage would be particularly applicable. access logs provide an unobtrusive method of mapping the use patterns of individuals. they give a useful picture of how the quizzes were used in relation to the timing of other aspects of the course (lectures and tutorials), and also provide quantitative data to back up the focus groups. there are data protection issues surrounding this, 254 s. clarke et al. and students were asked to give their consent to tracking using anonymous but distinct usernames, i.e. student1, student2. the specifics logged by this method were the details as to the date, time and frequency with which each username accessed each test. tracking of link following was used to measure whether or not the students moved beyond the boundaries of their recommended reading and used the hyperlinks embedded in the questions, feedback and answers. code was embedded directly within each hyperlink, this took the student to the page advertised but via a script that recorded the following of that link in a database. tracking done in this way is extremely quick and is not visually obvious or intrusive to the student. however, ethics again came into play and student consent was necessary. finally, focus groups were chosen as the best method of getting the students opinions on the quizzes and judging the effect on their learning and understanding (morgan, 1988). the focus groups were chaired by a course leader familiar with the curriculum with some lines of investigation intended to assess student understanding of the lecture topics. to help ascertain to some degree the effects of the mcq quizzes as opposed to the learning bestowed by the course as a whole, it was decided to use three sample groups. because the quiz aims were as yet unproven, and because they would be made available to all students at a later date if beneficial, it was considered to be an ethically acceptable strategy. a self-selecting group of thirty student volunteers were involved in the evaluation. the groups are described below: the control group the control group students went through course in the same way as those not involved in the trial. they attended the lectures and tutorials in the usual way, but had no access to the mcq quizzes. the paper group this group of students completed the lectures and tutorials. in addition, after each lecture they were given a printed version of the mcq test, with answers provided on a separate sheet. online material referenced in the questions was given as printed urls, and references to journal articles and books were given in the form of traditional written citations. the online group this group of students completed the lectures and tutorials. after each lecture, an online mcq test was released. students were given immediate feedback after answering each question, urls were provided as hyperlinks and journal articles and books were also provided as hyperlinks wherever possible. each student had an individual but anonymous account as described earlier. evaluating the potential of online mcq tests 255 results analysis the investigation took place over six weeks, with the staggered release of six sets of quizzes coinciding with the six introductory lectures to the course. throughout this time the logs and tracking were analysed and at the end of the six-week period the students were brought together in their individual focus groups, not only to discuss the impact of the quizzes, but also for analysis of their level of knowledge and understanding concerning the concepts and ideas of the lectures that they had been to. the results that emerged gave a valuable insight into how a resource such as formative online multiple-choice questions can be used to reinforce lectures and enhance student learning. what follows is a results analysis of the focus groups, tracking and logs in relation to the four aims posed, and a discussion of other relevant findings. the reinforcement of learning from other pedagogic elements to monitor whether the quizzes were successful as a pedagogy to reinforce learning, the usage of the quizzes was analysed as well as students’ attitudes towards using them as an additional study resource. the paper group voiced their reluctance to use the quizzes. it was inconvenient to pick up the quiz sheets after their lectures and they were generally put to the bottom of the pile of paper accumulating on their desks. being delivered on paper was not appealing as the quizzes lacked a high level of interaction, students found them bland and boring and none admitted to going back to them after completing them the once, although some filed them away for ‘revision purposes’. for the online quizzes the collected logs were examined to show usage. whilst such an analytical method can only provide a broad overview of the students’ use patterns, the statistics collected provided some valuable information. in some individual cases students accessed all the quizzes for the first time a couple of days before the focus group session. here it could be argued that their behaviour was affected by the knowledge that they were being tracked and of the impending focus group, their attention drawn more to the act of evaluation rather than the act of learning (faulkner, 2000, p. 171). for the remainder of the students the statistics showed that the quizzes were subjected to light usage; 100% of these students accessed all of the quizzes, but only 40% accessed them more than once. during the focus group session the general consensus was that the quizzes were a valuable resource for many students: student c: … some people use different libraries, different readings and different newspapers, i think all of us—we all like choose how we study and stuff so it’s, i think, it’s valuable to have this as an option as well. cos as i said before, it’s not for everyone, but for sure it’s valuable for a lot [of students]. the lack of repeated use of the quizzes was largely due to the difficulty students had in perceiving how they would form a cemented part of their studies, as it did not fit into the rigorous essay writing and examination process. some students did go back 256 s. clarke et al. and redo certain quizzes where they felt they needed to improve. however, like the paper group, the majority left it at one attempt with the forethought that such a resource would be more useful in the period leading up to examinations. with this in mind students commented that they would like the quizzes to be ‘a little more based on the exam’ rather than being based on the lectures. the times at which the students accessed the tests showed a strong pattern of latenight quizzing. the majority of log-in times occurred between 8pm and 12am in the evening. on questioning the students in the focus group it became clear that the process of taking a short quiz provided a work-related distraction for them whilst they were engaged in another activity, such as preparing for a tutorial or waiting to meet friends: student a: it’s like something you can do [so] that you feel like you’re doing something useful … student b: it’s kind of like in-between time … after dinner doing work and maybe you go out later to meet your friends or something. cementing students’ understanding the understanding of the key concepts covered during the lectures differed between the three groups. the control group and the paper group had extremely patchy retention of the details of the lectures. they had yet to grasp the key concepts and facts that had been delivered and generally had to bounce off each other when recalling lecture topics: ‘i don’t really remember much of it but when people start saying stuff it starts coming back’. the paper group thought that the quizzes were ‘good as a memory trigger’, but there was little evidence within the focus group sessions that they had remembered significantly more than their peers. it was clear that the students’ knowledge was built primarily through essay writing and tutorials rather than analysing and going back to the lecture content. in the lectures they sought specific facts and ideas which would help them with that week’s essay: ‘he’s [the lecturer] probably going to be mentioning something that is very relevant to your essay … and then you want to capture certain points’. the online group were noticeably more forward in their focus group when discussing topics that specifically related to the lecture content and more readily recalled subtle points that had been addressed in the lectures, as well as key examples and facts. the group all agreed that the online quizzes had provided a way of helping them remember the lectures: student a: i think it’s a good thing to go back, sit down and answer questions, because if you answer questions you think about it more than when you just sit down and read the notes again. student b: and then sometimes you miss things, at lectures, you know you’re not always concentrating. so, you know, when you read the questions … they help you think about it more and, more in-depth. student c: yeah, i think it’s good because, it’s good to think about it more, and it’s the right kind of style, it is for learning like. i think he raises key points that he wants you to get out of the lecture, impressions. evaluating the potential of online mcq tests 257 student c: sometimes i felt that. student d: yeah, it kind of triggered your memory about, i waited about a week, two until after the lectures to do it … i remembered it better in my mind, doing that. cos, i think if i’d done it straight away, it would have kind of all been one thing and i think i would have just forgotten it all after a while. interestingly, the images used in the quizzes provided a particularly useful hook with the lecture material, jogging their memories and reinforcing ideas and facts that had been discussed. student a: i would use as many pictures, illustrations as you did, because they can sometimes just give you that little bit that makes you remember them. cos when you see a picture, a photograph of thomas edison, you think, oh hang on, yeah, general electrics, because it was on the slide. but when they weren’t on the slides, they sort of confuse you a bit, because you think i must have seen that before, but you haven’t. deepening students’ knowledge to achieve a deeper understanding of the course students were given links to journal articles and company websites. in the paper-based quizzes these links were written down as urls or references to hard copy books and journals. the students found typing out the urls ‘annoying’ and if they were not actually sat at a computer at the time of taking the quiz they were unlikely to follow them up. likewise, the references to hard copy books and journals were not pursued as they felt they were reading enough with the set tutorial texts they were given each week : ‘i don’t do any more reading than i have to’. as a result the students found it difficult linking the lecture topics that they could remember to relevant wider material outside of their reading lists. the link tracking showed that the students taking the online quizzes did follow the links to the outside resources. interestingly the students refuted this—denying that they had read the articles and company profiles: interviewer: … for the ones [questions] which had the links, which could take you off and give you something more, long to read, did anyone look at those? student e: no, i didn’t. student c: i read some, i just didn’t include them. student a: depends how long they were. sometimes i read through them but sometimes i thought, ooh that’s too long, for the moment. it is clear that many students just clicked-and-glanced at the wider reading materials. this, they considered, did not constitute learning as there was no depth of reading or interpretation. however, it was evident that this very mundane practice made the students acknowledge that there was a wider pool of resources that they could, and were expected to, consult within their study practices. links to scholarly articles that the students felt relevant to their essays received a greater number of hits than others and the company websites. many students commented that they had added these to their browser ‘favourites’ folder to read at a ‘later date’. 258 s. clarke et al. framing educational expectations through the focus group sessions it was clear that students were starting to take on the expectations of the department. there was little difference between the three groups indicating that the quizzes were a precarious factor in framing the students’ educational expectations. there was a slight difference in the online focus group session where there was a strong general consensus that they were expected to read much more than they were already doing, perhaps as a result of following the links placed to deepen their knowledge. further findings of the evaluation all three focus group sessions brought to light some interesting points concerning the nature of the students’ study patterns at the university. although not directly related to the research questions of this study, these highlighted some pedagogic concerns which are relevant to the development of online learning tools. firstly, it was clear that undergraduate students study habits were entirely governed by the tutorial and essay writing timetable—each academic week revolved around the process of reading and writing for the next tutorial. this rigid study routine gives students little incentive to read any wider than the set papers for their forthcoming tutorials, rarely consulting further reading lists or searching for wider materials themselves. secondly, it became evident that the lecture handout played a talismanic role in the students study process. the students were immensely grateful for the lecture handout. for them it holds a vitally important place in their resources for study, representing a walkthrough of the lecture. however, to the lecturer it only really offers an overview from which they expect their students to expand their knowledge through further reading and discussion. because the lectures were engaging and entertaining, students took few, if any, notes and tended to sit back and enjoy the class; they felt they could use the handout later as a reference point. interestingly, although the handout appears to be an important resource, students can remember very little of the content of their lectures when quizzed. filed away and retrieved nearer the examination period it provides an outline of what needs to be regurgitated for a particular topic. the rigorous structure of lecture-tutorial-examination presents to the advocator of e-learning a problem of materiality. when given a set reading list and lecture handout students are inclined to do little more than follow the lines of investigation that these both represent. this structure is not conducive to wider thinking and obtaining a deeper knowledge. additionally, learners want to make sure that the skills and capabilities they acquire through using study resources are relevant to their summative course marks and it is not clear to them how optional e-learning such as formative quizzes can contribute to their degree. conclusions lectures are a fundamental part of the university teaching system – they provide an overview of a course’s topics and key ideas, and guide students towards areas of evaluating the potential of online mcq tests 259 further research and investigation. however, in a university system that also includes high quality tutorials it is often the case that lectures are put aside when there are more pressing academic activities and events in a student’s life. from the inquire evaluation it is strongly indicated that reinforcing the content of the lectures through formative assessment can act to cement students’ understanding of key concepts and ideas. the value clearly lies in the ‘jogging’ effect that the quizzes had on the students’ memories, and fundamentally the mode of delivery influenced how effective this was. good practice in university teaching places a high emphasis on the entertainment value of a lecture with its quality of being engaging and thus memorable, the implication being that memorable lectures are more effective in terms of student learning. we need to realize that the same is true for learning resources. this evaluation project, like others before it (e.g. mansen & haak, 1996; martin et al., 1996; charman & elsmes, 1998a) concerned chiefly the attitudes and preferences of the students towards using formative online assessment. it also attempted to address the impact of this type of assessment on student learning through comparing the knowledge and understanding of the three student groups. the evaluation of the effectiveness of learning technology is particularly complex and is fraught with difficulties: the success will vary, depending on the scope of previous knowledge, attitudes, and conceptions which particular learners bring to the learning situation, and the larger context in which the learning situation is embedded (jackson, 1998). the qualitative results of this evaluation suggest that online mcqs can reinforce learning from other pedagogic elements and cement student understanding in subjects that were not traditionally thought to lend themselves to such a model (mckenna, 2001b; leon, 2002). however, to clearly establish the positive impact of this method of formative assessment there must be further investigation of a quantifiable nature. the most valid option in the current educational system is to chart examination performance, as charman states ‘the only real way to assess this quantitatively is to compare performances in summative assessments’ (charman, 1999). it will be highly interesting to observe over the next few years if there is a marked difference in the module examination results attributable to the online quizzes.1 the use of formative online mcqs could potentially offer measurable benefits in supporting humanities and social science students in their study. however, there is still much to do if this is to become fully integrated within the study experience. it is still widely perceived both by education establishments and students that all forms of assessment are a means to measure knowledge and are inextricable from certification. the aims of these quizzes, however was to compliment traditional teaching and deepen understanding rather than to assess competence in a subject. there is still much work to do if there is to be a shift in our assessment paradigms (birenbaum 1996; sambell et al., 1999) and the benefits of formative assessment can be realized and integrated into the rigid study structures posed by many higher education systems. although this was a small-scale evaluation it has laid some useful foundations for further research in this area. 260 s. clarke et al. notes 1. for similar studies that have compared exam performance, see boyle et al., 1997; paul & boyle, 1998; and charmen & elmes, 1998c,d. references angelo, t. a. & cross, k. p. (1993) classroom assessment techniques (2nd edn) (san francisco, ca, jossey-bass). birenbaum, m. (1996) assessment 2000: towards a pluralistic approach to assessment, in: m. birenbaum & f. douchy (eds) alternatives in assessment of achievements, learning processes and prior knowledge (london, kluwer). boyle, a., byron, d. & paul, c. (1997) computer-based learning and assessment: a palaeontological case study with outcomes and implications, computers and geosciences, 23, 573–580. charman, d. & elmes, a. (eds) (1998a) computer based assessment (vol 2): case studies in science and computing (university of plymouth, seed publications). charman, d. & elmes, a. (1998b) a computer-based formative a strategy for a basic statistics model, journal of geography in higher education, 22(3), 381–385. charman, d. & elmes, a. (1998c) formative assessment in basic geographical statistics module, in: d. charman & a. elmes (eds) computer based assessment (vol 2): case studies in science and computing (university of plymouth, seed publications). charman, d. (1999) using caa for formative assessment, in: s. brown, p. race & j. bull (eds) computer-assisted assessment in higher education (birmingham, seda). faulkner, x. (2000) usability engineering (new york, palgrave). harvey, j. (1998) the ltdi evaluation cookbook (glasgow, learning technology dissemination initiative). jackson, b. (1998) evaluation of learning technology implementation, in: n. mogey (ed.) evaluation studies (learning technology dissemination initiative). available online at: http:// www.icbl.hw.ac.uk/ltdi/evalstudies/esevalimp.pdf leon, p. (2002) log onto student motivation, the times higher educational supplement, 10 march 2002. mansen, t. j. & haank, s. w. (1996) evaluation of health assessment skills using a computer videodisc interactive programme, journal of nursing education, 35(8), 382–383. martin, j. b., mccaffery, k. & lloyd, d. (1996) the introduction of computer-based testing on an engineering technology course, assessment and evaluation in higher education, 21, 83–90. mckenna, c. (2001a) academic approaches and attitudes towards caa: a qualitative study, fifth international computer assisted assessment conference, loughborough university. mckenna, c. (2001b) introducing computers into the assessment process: what is the impact upon academic practice? higher education close up conference 2, lancaster university. morgan d. l. (1988) focus groups as qualitative research (london, sage). sambell, k., sambell, a. & graham, s. (1999) student perceptions of the learning benefits of computer-assisted assessment: a case study in electronic engineering, in: s. brown, p. race & j. bull (eds) computer-assisted assessment in higher education (birmingham, seda). silverman, d. (2001) interpreting qualitative data, methods for analysing talk, text, interaction (london, sage). doi:10.3402/rlt.v20i0.16150 perceptions of the effects of clicker technology on student learning and engagement: a study of freshmen chemistry students jenepher lennox terrion a * and victoria aceti b a department of communication, university of ottawa, ottawa, canada; b health informatics institute, algoma university, sault ste. marie, canada (received 17 august 2010; final version received 02 september 2011) while technology � in the form of laptops and cellphones � may be the cause of much of the distraction in university and college classrooms, some, including the personal or classroom response system (prs/crs) or clicker, also present pedagogical opportunities to enhance student engagement. the current study explored the reactions of students to clicker implementation in a large, introductory chemistry class. during the final class of the semester, 200 students in an introductory chemistry class responded to an attitudinal and informational student survey using both likert-type and non-likert type questions to evaluate their perception of the implementation of the clickers and their impact on student learning and engagement. the results demonstrated that, when implemented effectively, clickers contribute to greater student engagement and, ultimately, an opportunity for professors to enact best practices in higher education pedagogy. this study points to the importance of effective pedagogy in making clickers worthwhile. keywords: clicker; technology; student engagement; higher education; learning; pedagogy most of the students populating classrooms in today’s universities and colleges were born in 1982 or later and are the first university cohort to be so constantly wired and connected to friends, to the media and to the internet through cell phones, mp3 players and laptops. in today’s large lecture halls, students have access to wireless internet and are, to a great extent, unmonitored in terms of what is on their screens. with so much opportunity to check their email, text their friends, visit facebook or surf the web, it is not surprising, therefore, that many of these students are distracted during class (rice and bunz 2006). however, while technology may be the cause of much of the distraction in university and college classrooms, it also presents pedagogical opportunities to enhance the learning experience and, more specifically, student engagement. as defined by kuh (2003), student engagement refers to ‘‘the time and energy students devote to educationally sound activities inside and outside of the classroom’’ (p. 25). several major reviews of the literature conclude that student engagement is among the better predictors of learning and personal development (carini, kuh, and klein 2006; umback and wawrzynski 2005), academic success (page number not for citation purpose) *corresponding author. email: jlennoxt@uottawa.ca research in learning technology vol. 20, 2012 rlt 2012. # 2012 j. lennox terrion and v. aceti. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 16150 doi: 10.3402/rlt.v20i0.16150 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 (appleton et al. 2006; dunleavy and milton 2008; finn 1989; fredericks, blumenfeld, and paris 2004; marks 2000; pirot and de ketele 2000) and retention (kuh 2003). students become engaged when their academic experience is characterised by meaningful educational activities, including active learning, involvement in enriching educational experiences, seeking guidance from staff or working collaboratively with other students (carini, kuh, and klein 2006). however, as umback and wawrzynski (2005) suggest, little new knowledge has been generated about indicators of educational practice that predict student engagement. furthermore, understanding how students perceive these educational practices lends important insight into what actually contributes to this outcome. most universities and colleges have placed great emphasis on student engagement in the past decade, in part thanks to the national survey on student engagement (nsse), an instrument used by more than 1300 colleges and universities in the u.s. and canada since 2000. this instrument allows comparison of universities against each other as well as against a benchmark score. according to nsse, the ‘‘results provide an estimate of how undergraduates spend their time and what they gain from attending college’’ and the instrument provides a measure of ‘‘empirically confirmed ‘good practices’ in undergraduate education’’ (http://nsse.iub.edu/html/about.cfm). in terms of identifying these good practices, chickering and gamson (1987) are often looked to because of their model of the principles of effective undergraduate education. as umback and wawrzynski (2005) argue, most of these principles reflect pedagogical behaviours undertaken by instructors and thus this model focuses on the context created by faculty members on campus and the relationship of this learning context to student engagement. chickering and gamson suggest that good practice builds student engagement because it encourages contact between students and faculty, gives prompt feedback, develops reciprocity and cooperation among students, encourages active learning, emphasises time on task, communicates high expectations and respects diverse talents and ways of learning. student engagement is important to learning because unengaged students do not listen, process or attend to the learning process. while the influence of the instructor and his or her pedagogical choices and practices are primordial to the student experience (lane and shelton 2001), technology may offer a means to enhance student engagement. one technology that presents opportunities for student engagement is the personal or classroom response system (prs/crs), commonly referred to as the clicker. in classrooms equipped with a receiver and the appropriate software, students each have a handheld remote control, purchased along with their textbook for about $20 (cad). students ‘‘click’’ in their responses to multiple choice questions posed by the professor and projected on the screen from within a slide presentation. when all responses are received, the results are projected onto the screen for the entire class to see (either anonymously or with respondents identified). individual student and collective class data can be saved for each session, allowing responses to be recorded, analysed and graphed. as bruff (2009) points out in his book on teaching with classroom response systems, the consensus among studies of student perceptions of clickers is that attention, attendance and interest (and even learning, depending on the pedagogical approach taken) can be enhanced by using this technology. specifically, in their review of three decades of literature on clicker use, judson and sawada (2002) concluded that, j. lennox terrion and v. aceti 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 while the impact of clickers on learning outcomes was disappointing, use of this technology did seem to have a positive impact on student engagement and attitudes. caldwell (2007) and fies and marshall (2007), in their literature reviews of the impact of clickers in higher education settings, report that the technology has been found to have a generally positive impact on the classroom. numerous studies have concluded that, when implemented using appropriate and effective teaching methods, clickers can foster student engagement and overall student success in the course in which they are used (hoekstra 2008; kaleta and joosten 2007; rice and bunz 2006; twetten et al. 2007). one study, conducted at the university of wisconsin, involved 3500 students in 28 courses and 19 disciplines and found that 93.5% of faculty strongly agreed/agreed that students were more engaged and 72% strongly agreed/agreed that clickers benefited learning (ellozy 2007). similarly, rice and bunz (2006) found that clickers have led to ‘‘greater student engagement in the classroom, high satisfaction, and gains in student learning across the disciplines’’ (p. 3). in regard to creating the conditions for engagement and addressing the principles for effective undergraduate education, clicker technology presents interesting potential. to explore this potential, the current study measured student perceptions of clicker use and the pedagogical choices of the instructor in a large, introductory chemistry class. research questions: rq1: do students perceive that they are more engaged with course material and inclass discussions as a result of clicker use? rq2: do students believe that the clickers help them to more effectively learn the course material? rq3: do students who perceive a positive impact on their learning as a result of the clickers also perceive that they are more engaged in their learning experience? methodology in an effort to address student inattention and the distractions created by electronics in a large (200 students), first-year chemistry class, the instructor used einstruction’s classroom performance system (http://www.einstruction.com/) to build a number of clicker innovations into the lecture. the course was lecture based and used microsoft powerpoint slides with infusions of multimedia and online tools. each lecture incorporated many different clicker activities, including exercises, quizzes and teambased activities. for example, the instructor used quizzes at the beginning of the lecture to assess understanding of readings and at the end of challenging sections in lectures to check understanding. as well, the professor used ‘‘peer instruction’’ (mazur 1997) to help make lectures more interactive and to get students intellectually engaged with the material by having them think critically about the material and discuss their ideas and insights with their neighbours. specifically, after presenting a complex concept or process, the instructor would present a slide with a question, to which students would respond on their own using their clicker. the correct answer would not be displayed and, ideally, there would be a range of answers (demonstrating that the topic is challenging, that there is not universal understanding and, thus, that peer instruction is necessary). then, students would discuss with those research in learning technology citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 3 (page number not for citation purpose) http://www.einstruction.com/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 sitting next to them and answer again using their own clickers. at this point, the instructor would reveal the correct answer. participants as the chemistry class chosen for the study was an introductory course, the majority of the students were in their first year. specifically, 177 of students were in their first year of university (freshman), while 16 were in their second year (sophomore), 4 in their third year (junior) and 3 in their fourth year or higher (senior). participants’ age ranged from 17 to 34 years, with a mode of 18 years and a mean of 18.6 years. in terms of gender, 67 (33.5%) of the students were female, and 131 (65.5%) were male, with 2 participants leaving the gender question blank. questions of ethnicity, religion or socioeconomic status were not asked. given that 88.5% of students were in the first semester of their first year of studies, it is likely that the vast majority of participants had had no prior experience with the clickers and thus the threat to validity of prior knowledge and experience, or history, was removed. at the time of the study, only six professors (out of about 700 in total) at the university of ottawa were making use of the clicker technology, so this was a novel pedagogical approach for most students at this institution. procedures after receiving ethical clearance to conduct this study, a graduate research assistant unrelated to the course distributed a brief attitudinal and informational student survey to the 250 students registered in chm1301 principles of chemistry (fall 2009) during the last class of the semester. the professor was not involved in the distribution of the questionnaires and students were verbally informed that they were free to participate or not. they were collected by the research assistant and placed in a sealed envelope. the survey used both likert-type and non-likert type questions to evaluate the students’ perception of the clickers and their impact on student engagement. a total of 200 surveys (80%) were completed. measures the survey consisted of an introductory section of three questions in which students indicated demographic information and an attitudinal section consisting of 15 items to which students responded using a five-point likert-type scale (1 �strongly disagree; 2 �disagree; 3 �neutral; 4 �agree; and 5 �strongly agree). questions dealt with the perceived effect of the clickers on a range of variables, including learning and understanding, perceptions related to the use of the clickers during class time and the perceived effect of the clickers on attention, engagement, participation and enjoyment. these were later grouped to create three distinct variables: implementation six items were grouped to assess technical aspects of clicker use in the classroom such as the ease of operation of the clickers and the integration or fit of the technology within the course design. these questions centred on the students’ perspective, for both ease of use and the effectiveness of the professor’s integration of the technology j. lennox terrion and v. aceti 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 into the course. specifically, participants were asked to respond to the following: (1) the professor made good use of the clickers; (2) clickers were well integrated into the course; (3) the clickers were frustrating to use; (4) the clickers were easy to use; (5) the technology used in this course worked well; and (6) i would like to have the chance to use the clicker again in another class. question #3 (related to frustration with the technology) was recoded so that higher scores reflected stronger agreement with the statement. all five items (a�0.754) were combined to measure students’ perceptions of ease of use and integration of clickers in the classroom (mean �3.59, sd �0.74). student engagement seven items were combined to measure student engagement (a�0.908). using a fivepoint likert-type scale, items focused on students’ perceptions of the impact of the clicker on various aspects of student engagement. these items included questions about the relationship between the clicker and increased interest in course material, participation during lecture and interaction with other students and with the professor: (1) clickers contributed significantly to my interest in the course material; (2) as a result of the clicker, i felt more involved in the lecture; (3) as a result of using clickers in this course, i interacted more with other students; (4) the use of clickers in this course was an appropriate way to achieve course objectives; (5) as a result of using the clicker, i felt more engaged and involved; (6) as a result of using the clicker, i felt more inclined to participate in class discussions; and (7) clickers enabled the professor to respond to concepts that i might not have understood. none of the eight items was recoded (mean �3.12, sd �0.85). effect on learning two items were used to measure whether students perceived that clickers contributed positively to or detracted from their learning during lecture (a�0.779). using a fivepoint likert-type scale, students were asked to agree or disagree with the following statements: ‘‘clickers contributed significantly to my learning’’ and ‘‘clickers did not contribute to my learning experience.’’ the negative oriented item was recoded (mean �3.05, sd �1.04). data analysis the statistical software package spss was used to analyse the data. the aim of the study was to determine if there was a statistically significant relationship between the use of the clickers and student engagement. thus, the null hypothesis is that there is no significant relationship between the use of clickers and student engagement. depending on the variable to be assessed, a t-test, anova, or correlation was conducted in order to test the null hypothesis. results demographics the first questions focussed on the relationship between participant demographics and the three variables. no statistically significant differences between a student’s research in learning technology citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 year in university, gender or age and their perceived ease of use and implementation of the clicker in the class were observed. further, no significant difference was found between these variables and student engagement or these and impact on learning. these findings demonstrate that any significant relationships among the three variables (implementation, student engagement and impact on learning) cannot be attributed to demographic variance. findings in relation to the research questions rq1. do students perceive that they are more engaged with course material and in-class discussions as a result of clicker use? rq1 attempted to get at the relationship between student engagement and clicker use in the classroom. results found, as shown in table 1, a strong correlation between student engagement and clicker implementation (r �.0678, pb0.01), indicating that these two variables are collinear. this means that students perceive that clickers encourage engagement when implemented into lectures. rq2. do students believe that the clickers help them to more effectively learn the course material? the goal of rq2 was to understand the effect of clicker use and implementation in the classroom on students’ perceptions of their own learning experience. the correlation test on the variables integration and impact on learning showed, as seen in table 1, a moderate correlation (r �0.577, pb0.01). this demonstrates that students believe that using clickers as part of the class lecture had a positive effect on their ability to learn course material. rq3: do students who perceive a positive impact on their learning as a result of the clickers perceive that they are more engaged in their learning experience? finally, to assess whether student perception of learning correlated with reported engagement, correlation tests were performed on the variables student engagement and impact on learning of using the clickers. results showed, as seen in table 1, a very strong positive correlation (r �0.846, pb0.01), indicating that, if students perceived their learning to be enhanced by the clickers, then they also reported being more engaged in the learning experience. discussion this study has shown that students reacted positively to the professor’s integration of clickers into their chemistry class and perceived that this technology enhanced their table 1. correlation results. implementation student engagement impact on learning implementation 1.00 0.678 0.577 student engagement 0.678 1.00 0.846 impact on learning 0.577 0.846 1.00 j. lennox terrion and v. aceti 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 engagement and learning in this particular classroom. while it does not take into account directly the professor’s personal teaching style, it does take into account the learning context created through the use of clicker technology. further, this study assesses students’ perceptions of faculty behaviours and thus, as umback and wawrzynski (2005) suggest, places the emphasis on pedagogical choices made by instructors. these findings are supported by numerous studies (e.g., caldwell 2007; crossgrove and curran 2008; cue 1998; hoekstra 2008; jackson and trees 2003; kaleta and joosten 2007; preszler et al. 2007; rice and bunz 2006; trees and jackson 2007; twetten et al. 2007). as discussed above, in elaborating the seven principles of effective undergraduate education, chickering and gamson (1987) argue that good practise builds student engagement because it encourages contact between students and faculty, gives prompt feedback, develops reciprocity and cooperation among students, encourages active learning, emphasises time on task, communicates high expectations and respects diverse talents and ways of learning. the research findings will be discussed below in the context of each of these principles. in terms of increased contact between student and faculty, it could be suggested that the clicker enables a classroom of students � whether 50 or 500 � to communicate directly with the professor in a mediated format. in fact, clickers are designed specifically to be used in large classes, where it is difficult for students to get feedback, and clickers have been shown in numerous studies to be helpful in overcoming the challenges faced with student inattention and distractions in these large classes (boyle and nicol 2003). professors can respond to students by providing another example or a more detailed explanation to a question that the clicker results have shown is unclear to some or most students. our data showed that students perceived that the professor responded to concepts that were not understood when this lack of understanding was communicated by students through the clicker. large lectures may be intimidating for first year students who are not accustomed to the number of students, and thus students who do not understand something may be reluctant to ask questions. clickers afford students the opportunity to voice their misunderstandings anonymously and, as a result, the professor can respond to those students without centering them out when she sees incorrect responses on the screen. in a similar vein, clickers enable greater and more immediate feedback from professors and, thus, allow students to assess their knowledge of major concepts, as well as their level of understanding in comparison to their peers (kaleta and joosten 2007). when faced with a potential lack of knowledge or understanding (as shown through incorrect answers), the professor can encourage students to visit the instructor during office hours or to follow up with the teaching assistant or tutor in order to receive more tailored support. in fact, our data showed that students reported being not only more inclined to participate during lectures when using clickers but also, as discussed above, that the professor was more likely to respond to a lack of conceptual understandings demonstrated by students. in other words, twoway communication seemed to be enhanced by the clickers in the sense that students showed their level of understanding more effectively and the professor responded and adapted his/her lecture to address concepts that students might not have understood. in terms of increased reciprocity and cooperation among students, clickers provide the chance for students to work with each other when the clicker is used, as it was in this class, to enable ‘‘peer instruction,’’ a technique developed by eric mazur (1997) to help make lectures more interactive and to get students intellectually research in learning technology citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 engaged by having them think critically about the material and discuss their ideas and insights with their neighbours. as chickering and gamson (1987) argue, good learning is collaborative and social, not competitive and isolated. by requiring students to turn to their neighbours and discuss a question and possible answers before responding on their clickers, peer instruction fosters this goal. in our study, participants reported that the clickers allowed for increased interaction with other students during lecture and this, in a class of 200 students, is an important outcome where students could feel lost. as for active learning and diverse learning styles, the tangible act of answering a question using a clicker increases engagement, especially in students who are kinesthetic � or hands-on � learners (bruff 2009) because the clicker takes the student’s role from a passive observer to an active contributor. incorporating the needs of all learning styles increases student engagement, as each student is able to learn effectively through one of the delivery approaches � auditory, visual or kinesthetic � of the lecture. the data showed that there was a strong positive correlation between a perception of higher engagement and a positive impact on learning. this indicates that, when students report being more actively involved in the lecture, they are more inclined to report more positive perceptions of their own learning. the clicker also seems to enhance time on task, or the time and energy that students put into learning activities, by encouraging regular attendance. while some research has shown that students respond negatively to clickers when they are used only for attendance (trees and jackson 2007), many studies demonstrate the positive impact on attendance when clickers are used effectively. for example, burnstein and lederman (2001), in their study of clicker use in science classes, reported 80% to 90% attendance in classes where clickers were in use. in terms of communicating high expectations, kaleta and joosten (2007) found in their study of clicker use that students reported their ‘‘need to come to class ready to participate and pay attention’’ (p. 9) when the lecture incorporated clickers. likewise, as stated by trees and jackson (2007), the clicker ‘‘creates a learning environment with higher expectations for student preparation prior to class’’ (p. 25). as a result, students are more motivated to complete the required readings before class and, in turn, are more participatory in class discussions. kaleta and joosten agree, claiming that clickers create a more accountable student who is then able to engage with course material during lectures. furthermore, given the anonymity fostered by using the clicker, students likely experience less social anxiety about offering a wrong answer, so that students may be more likely to participate. while our data did not directly measure student preparation or completion of homework or required readings, our data do show that clickers positively contributed to students’ interest in course material and to their overall learning experience. one limitation of this study is that all data are based on students’ reported perceptions of their own engagement and learning. the authors were not involved in delivery of the course in any way and, thus, we do not have measures of attendance or final grades. we cannot, therefore, draw conclusions about the impact of clicker technology on these variables. future research could track attendance, grades and withdrawal and attempt to correlate these with clicker use. furthermore, triangulating the data through observations or qualitative interviews would enhance and deepen our understanding of the interaction between students and clicker technology and thus move beyond measuring reactions to the professor’s integration of the technology in the course. j. lennox terrion and v. aceti 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 conclusion this study has shown that clickers have the potential to be an effective learning tool that can be used to enhance student engagement through the enactment of chickering and gamson’s (1987) principles for good practices in higher education. as draper, cargill and cutts (2002) conclude in their study of clicker implementation, it is possible that this technology allows students to play an active and responsible role in the classroom, and, in so doing, increases their motivation and engagement. it is important to note, however, that while clickers appear in this study to have a significant and positive impact on learning and engagement, it is through thoughtful and purposeful implementation that they are most likely to be effective. as sutherland, robertson and john (2008) point out, professors may fear that, when they introduce a new technology into their classroom pedagogy, it will not work well or will use up class time while they struggle to get it working. further, as a result of the potentially negative impact of technological difficulties on professor credibility and course evaluations, professors may be reluctant to integrate innovations such as clickers in the classroom. thus, technical support must be available to quickly help professors during lectures, as well as to provide adequate training and troubleshooting prior to the start of the course. in addition to this support, educational materials must be created and made available to professors to assist in integrating clickers successfully. further, inconsequential use of the clickers in lecture could lead to negative student perceptions of the technology (trees and jackson 2007). for example, if clickers are used for only attendance purposes, students will feel that the clicker was a waste of money, may become more disengaged and may not be open to using clickers in the future when other professors employ the technology. thus, student support of clickers is imperative as the use of clickers creates a shift in the classroom environment where more active participation is required from students (trees and jackson 2007). for freshman students, this change to a more active classroom may not even be noticed; however, in upper year courses where students are used to taking a more passive role in the lecture, lecturers who employ clickers may face more resistance. furthermore, as lane and shelton (2001) point out, technology is no substitute for effective pedagogy and ‘‘the failure to place pedagogy prior to technology results in little or no net instructional gain’’ (p. 248). this contention is supported by judson and sawada (2002), who conclude in their extensive review of the literature on clicker use that it is the pedagogical practices of the instructor and not simply the incorporation of technology into the classroom that is essential to student learning. this means that instructors must carefully consider the learning objectives of their course as well as the pedagogical options they have, including lecture, small group discussion, in-class exercises and so on. many freshman college and university students are still teenagers, with the developmental limitations of this age group, and thus, as educators, we must develop strategies to engage students and enhance their retention in the critical first year of higher education. this research illustrates the perception of the clicker as a tool for student engagement not from a professor’s perspective, but the students’ perspective. it demonstrates how clickers can be incorporated in the classroom to create a culture of learning through increased communication and understanding between the students and professors. few pedagogical techniques allow students the opportunity to honestly answer questions in class without fear of ridicule or embarrassment while also providing the professor with a clear picture of learning in his/her classroom. research in learning technology citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 clickers, like all technologies in the classroom, present opportunities to continuously adapt to the realities of changing demographics in colleges and universities. when implemented effectively, they allow professors to connect with students in a way that complements the lecture format and thus may lead to greater student engagement. as clicker technology is made more accessible and less expensive, for example, by using a student’s own mobile phone as the clicker device (see tremblay, 2010), then the use of clickers may become more commonplace and, as a result, will require even more creative use of this learning tool. indeed, students themselves may have many ideas about ‘‘apps’’ that can enhance their learning experience and that can keep the implementations of the technology fresh and relevant. in this sense, pedagogical practice becomes a shared responsibility as professor and students participate in the joint venture of higher education. references appleton, j. j. et al. (2006) ‘measuring cognitive and psychological engagement: validation of the student engagement instrument’, journal of school psychology, vol. 44, pp. 427�445. boyle, j. t. & nicol, d. j. (2003) ‘using classroom communication systems to support interaction and discussion in large class settings’, association for learning technology journal, vol. 11, pp. 43�57. bruff, d. (2009) teaching with classroom response systems: creating active learning environments, jossey-bass, san francisco, ca. burnstein, r. a. & lederman, l. m. (2001) ‘using a wireless keypad in lecture classes’, the physics teacher, vol. 39, pp. 8�11. caldwell, j. e. (2007) ‘clickers in the large classroom: current research and best-practice tips’, cbe life sciences education, vol. 6, pp. 9�20. carini, r. m., kuh, g. d. & klein, s. p. (2006) ‘student engagement and student learning: testing the linkages’, research in higher education, vol. 47, pp. 1�32. chickering, a. w. & gamson, z. f. (1987) ‘seven principles for good practice in undergraduate education’, american association of higher education bulletin, march, pp. 3�7. crossgrove, k. & curran, k. l. (2008) ‘using clickers in nonmajorsand majors-level biology courses: student opinion, learning, and long-term retention of course material’, life sciences education, vol. 7, pp. 146�154. cue, n. (1998) ‘a universal learning tool for classrooms?’, proceedings of the first quality in teaching and learning conference, hong kong sar, china, december 10�12, 1998, [online] available at: http://celt.ust.hk/ideas/prs/pdf/nelsoncue.pdf, accessed 10 may 2010. draper, s. w., cargill, j. & cutts, q. (2002) ‘electronically enhanced classroom interaction’, australian journal of educational technology, vol. 18, pp. 13�23. dunleavy, j. & milton, p. (2008) ‘student engagement for effective teaching’, education canada, vol. 48, pp. 4�8. ellozy, a. (2007) ‘new initiative: ‘‘clickers’’ in our classrooms’, new chalk talk, vol. 6, pp. 8. fies, c. & marshall, j. (2006) ‘classroom response systems: a review of the literature’, journal of science education and technology, vol. 15, pp. 101�109. finn, j. d. (1989) ‘withdrawing from school’, review of educational research, vol. 59, no. 2, pp. 117�142. fredericks, j., blumenfeld, p. & paris, a. (2004) ‘school engagement: potential of the concept, state of the evidence’, review of educational research, vol. 74, pp. 59�109. hoekstra, a. (2008) ‘vibrant student voices: exploring effects of the use of clickers in large college courses’, learning, media and technology, vol. 33, pp. 329�341. jackson, m. & trees, a. (2003) clicker implementation and assessment, information and technology services and faculty teaching excellence program, university of colorado, boulder, co, [online] available at: http://comm.colorado.edu/mjackson/clickerreport.htm judson, e. & sawada, d. (2002) ‘learning from past and present: electronic response systems in college lecture halls’, journal of computers in mathematics and science teaching, vol. 21, pp. 167�181. j. lennox terrion and v. aceti 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 http://celt.ust.hk/ideas/prs/pdf/nelsoncue.pdf http://comm.colorado.edu/mjackson/clickerreport.htm http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 kaleta, r. & joosten, t. (2007) ‘student response systems: a university of wisconsin system study of clickers’, educause center for applied research: research bulletin, vol. 10, pp. 1�12. kuh, g. d. (2003) ‘what we’re learning about student engagement from nsse’, change, vol. 35, pp. 24�32. lane, d. r. & shelton, m. w. (2001) ‘the centrality of communication education in classroom computer-mediated communication: toward a practical and evaluative pedagogy’, communication education, vol. 50, pp. 241�255. marks, h. m. (2000) ‘student engagement in instructional activity: patterns in the elementary, middle, and high school years’, american educational research journal, vol. 37, pp. 153�84. mazur, e. (1997) peer instruction: a user’s manual, prentice hall, upper saddle river, nj. pirot, l. & de ketele, j.-m. (2000) ‘l’engagement académique de l’étudiant comme facteur de réussite à l’université: étude exploratoire menées dans deux facultés contrastées’, revue des sciences de l’ducation, vol. 26, no. 20, pp. 367�394. preszler, r. et al. (2007) ‘assessment of the effects of student response systems on student learning and attitudes over a broad range of biology courses’, cbe life sciences education, vol. 6, pp. 29�41. rice, r. & bunz, u. (2006) ‘evaluating a wireless course feedback system: the role of demographics, expertise, fluency, competency, and usage’, studies in media and information literacy education, vol. 6, no. 3, pp. 3. sutherland, r., robertson, s. & john, p. (2008) improving classroom learning with ict, routledge, new york, ny. trees, a. r. & jackson, m. h. (2007) ‘the learning environment in clicker classrooms: student processes of learning and involvement in large university-level courses using student response systems’, learning, media and technology, vol. 32, pp. 21�40. tremblay, e. (2010) ‘educating the mobile generation � using personal cell phones as audience response systems in post-secondary science teaching’, journal of computers in mathematics and science teaching, vol. 29, no. 2, pp. 217�227. tweeten, j. et al. (2007) ‘successful clicker standardization’, educause quarterly, vol. 4, pp. 63�67. umbach, p. d. & wawrzynski, m. r. (2005) ‘faculty do matter: the role of college faculty in student learning and engagement’, research in higher education, vol. 46, pp. 153�184. research in learning technology citation: research in learning technology 2012; 20: 16150 doi: 10.3402/rlt.v20i0.16150 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16150 editorial whitherält-j? issue 7.1 represented the annual special issue of selected papers presented at alt-c 98. this issue is the first time the new editorial team have edited an 'open' issue comprised of papers selected from those submitted as part of the normal refereeing process. the broad scope of the papers in this open issue courseware design, collaborative learning, evaluation, student use of electronic bibliographic databases, staff development bears witness to one of the most fascinating aspects of learning technology: its broad-based interdisciplinary nature. we feel strongly that alt-j should provide a forum for scholarly debate, which reflects the eclectic nature of learning technology. however, there is a danger that, with a diverse profile, the journal could lose a sense of direction. with this in mind we have identified four areas that we would like to encourage: • the application of learning technology is in its infancy. he institutions are trying out different approaches in an attempt to match the use of technology to local needs and demands. we can all learn from each other in this context, and we see alt-j as providing opportunities for strategic initiatives to be reported, leading to a discussion of issues such as staff training, resourcing and management. • our view is that alt-j is a scholarly journal. as such a prime rationale for the journal is the publication of the results of empirical research, and we wish to encourage research papers. • the youth of learning technology makes it very important that we think creatively about the future. 'brain-storming' has a role in thinking creatively about the educational use of technology, and we wish to encourage papers that reflect on future developments. 2 alt-] volume 7 number 2 • the journal must guard against being parochial. the boundaries in education are being eroded notably in the blurring of notions of higher and further education. likewise the interface between the commercial and industrial sectors and education is becoming more diffuse. education is becoming more international, in some part due to the possibilities afforded by information and communications technology. we intend to encourage papers that look to cross-phase developments, explore relationships with industry and commerce, and have international dimensions. in line with these aims we have decided that in addition to the annual conference edition, we will publish a special issue on a key topical theme or research area or work which reflects current developments in a particular sector. this will give an annual profile for the journal consisting of a special conference issue published in april, an 'open' issue published in july, and a special thematic issue published in november. in this way we hope that we will provide an opportunity for a wide range of papers to be published in a focused way, to reflect the four key themes of the journal. to launch this new format, issue 7.3 will develop the fourth aspect outlined above by focusing on work from the commercial sector and in particular will describe the educational work associated with the bt laboratories at martlesham heath. the work in the bt laboratories picks up on all four of the themes we have identified: interdisciplinary research and development, links between further and higher education, relationships with industry and commerce, and state-of-the-art research and development in learning technology. we would welcome suggestions for the next special edition to be published in november 2000. david squires and gráinne conole 3 editorial: the role and influence of key stakeholders in the learning technology community in this issue of alt-j we have six articles that address three broad topics of video, costbenefit analysis and the emergent learning technology community. the first two articles present case studies on the use of video in teaching and learning. shephard et al. describe their experiences of re-purposing a back 'care video' for video streaming, and use these experiences to highlight key challenges that others may face when attempting to re-purpose a video. blake and scanlon use their experiences of analysing video recordings of students, who were using computers to support their collaboration in solving statistical problems, to argue that such video analysis provides useful rich data with which to interpret and understand students' experiences. the third and fourth articles address the challenging issue of assessing the costs and benefits of both specific and general learning technologies. loewenberger and bull present the results of a survey that attempted to gather information about the costs and benefits of computer-based assessment; while nicol and coen describe and illustrate a model that has been developed to enable evaluations of the costs and benefits of ict. the final two articles debate issues raised by an article by lisewski and joyce, entitled 'examining the five-stage e-moderating model: designed and emergent practice in the learning technology profession', published in issue 11.1 of alt-j. tompsett and alsop present a commentary on this article and offer a reinterpretation of the research by taking what they call 'a stricter analysis' of the events described by lisewski and joyce. in a response to this reinterpretation joyce and lisewski argue that the approach taken by tompsett and alsop is too narrow and runs the risk of not taking into account the whole context in which learning technology as a profession is emerging. when i read the six articles, a quote by frank deruyter, an assistive technology researcher in america, came to my mind. in discussing methods for evaluating the success of assistive technologies he wrote: while it is reasonable to assume that each stakeholder desires a successful outcome, it is 2 alt-j volume 11 number 2 unreasonable to assume that each stakeholder desires the same outcome. (deruyter, 1995) picking up on this idea, it seems to me that one theme that draws all the articles in this issue of alt-j together is that of 'stakeholders' and the different views that different stakeholders may have on the costs, benefits, successes or failures of learning technologies. in the article by shephard et al. article the stakeholders can be seen to be the funders, the lecturers, the media producer, the learning technology coordinator as well as the actors and patients in the video. the different views and interests of all these stakeholders presented the authors with a number of challenges when trying to re-purpose a video for video streaming. for example, the ownership of copyright, creative and moral rights by various stakeholders in the video meant that the video had to be cut or re-edited in places and that restrictions were placed on where the video could be accessed. these and other challenges lead shephard et al. to conclude: v streaming existing videos is not for the faint-hearted; a wide range of issues will be encountered that must either be overcome or circumvented. deciding which of these two responses best fits each problem is probably the toughest task, and in each situation a rigorous cost-benefit analysis will be necessary. in the blake and scanlon article, the stakeholders can be seen to be the different members of the evaluation team: cal designers, course developers and educational technologists. but the use of video analysis to record detailed student behaviours introduces another stakeholder into the team the student. such an introduction is argued by blake and scanlon to enable a wider consideration of the potential benefits of cal programmes. in their article, loewenberger and bull report two examples in which key stakeholders influenced their survey. the first example is when staff from two faculties within the targeted institution resisted efforts to engage them in focus groups and interviews, which led to a change in methodology. the second example is that the majority of respondents in the final survey were reported to be those with high levels of seniority or experience. it is possible to suggest that more senior and experienced people returned the questionnaire because they had a bigger stake or interest in the relative costs and benefits of computerbased assessment. in both examples, stakeholders in the survey have exerted their power and influence in very different ways. nicol and coen identify two key groups of stakeholders in their article: those tasked with costs analysis and those tasked with benefits analysis. nicol and coen note: within institutions, those tasked with'cost analysis normally have little expertise in benefits evaluation and those with benefits expertise have little costing experience, and these groups rarely work together. the insight cost-benefit model described by nicol and coen can be seen as an attempt to draw the two sets of expertise together. nicol and coen conceptualize the model as a decision-support tool for senior managers in he tasked with making investment decisions about initiatives in technology-supported teaching. they also suggest that a key stakeholder that will benefit from the application of such a model is the higher education sector, which will be able to evaluate the benefits of ict from a wider institutional perspective. 3 editorial finally, the discussion papers by tompsett and alsop and by joyce and lisewski challenge us to think about the learning technology community and the different views that different stakeholders within that community may hold. joyce and lisewski also suggest that some of these stakeholders represent 'powerful interest groups' from managerial, academic and political arenas. stakeholder diversity within the learning technology means that for those working within that community life is never dull. perhaps we need to understand that diversity a little better in order to ensure that future developments can start from a springboard to success rather than a quagmire of 'nearly made it'. references deruyter, f. (1995), 'evaluating outcomes in assistive technology: do we understand the commitment?', assistive technology, 7 (1), 3-16. jane seale deputy editor 4 doi:10.3402/rlt.v20i0/14919 the effectiveness of educational podcasts for teaching music and visual arts in higher education cheung on tam* department of cultural and creative arts, hong kong institute of education, hong kong (received 31 january 2011; final version received 11 may 2011) podcasting is now being used frequently in the higher education sector. although research has been conducted into the use of podcasting in teaching business, engineering, sciences and languages, little has been done on its use in arts learning. this paper reports on a study that investigated the effectiveness of using podcasts to learn music and visual arts in a teacher-education institution. a total of 128 year-one to year-four students completed a survey and 24 students attended focus-group interviews and gave their feedback on the effectiveness of using podcasts in learning. generally speaking, the students found the podcasts useful and viewed podcasting as a method which could be used to supplement faceto-face teaching. results of the study indicated that demonstration of procedures or skills was the most effective type of podcast while reproduction of lecture materials was the least effective. the students suggested that the teachers needed to increase the relevance of the podcast content to what was taught during lessons. the paper concludes with a discussion of the production of effective podcasts and directions for future studies. keywords: podcasting; podcasts; teaching/learning strategy; arts education; teacher education background this paper reports on a study that investigated the effectiveness of customised educational materials in the form of podcast files created by teachers. these files included video presentations, demonstrations and edited lectures teaching particular aspects of the music and visual arts courses offered by the hong kong institute of education in the academic year 2009�2010. the institute is a tertiary institution dedicated to providing professional teacher education for those who wish to teach at early childhood, primary or secondary levels in hong kong. feedback collected from students by means of a survey and interviews offered information on the effectiveness of using podcasts to learn and learners’ views on what constitutes an effective educational podcast. the term ‘podcasting’ was first proposed by journalist ben hammersley in a february 2004 article of the guardian (copley 2007). it is a portmanteau word blending ‘ipod’ and ‘broadcast’ (sutton-brady et al. 2009). podcasting is a method of distributing a digital media file or a series of files over the internet for playback on portable media players such as ipods or personal computers (lazzari 2009). users may subscribe to a feed that links automatically to the most (page number not for citation purpose) *email: cotam@ied.edu.hk research in learning technology vol. 20, 2012 rlt 2012. # 2012 c.o. tam. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http:// www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 14919 doi: 10.3402/rlt.v20i0.14919 http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 recent materials, or download the files manually from the internet. all podcasts used in this study were movie files in mpeg-4 format, which is a multimedia container format that allows the storage and presentation of digital video, digital audio, texts and images. for the distribution of the podcasts in this study, it was carried out through the itunes program by providing really simple syndication (rss) feeds to students. first, a series of podcasts was saved on the distributor’s server. students then downloaded the podcasts using the ‘subscribe’ function of itunes by adding a uniform resource locator feed. teachers periodically uploaded newly produced podcasts to the server and students would get the updated files automatically each time they started itunes. files were stored locally on students’ computers or other devices and also could even be used offline. the itunes program also provided one special function*the ‘itunes u’*whereby students could search and download podcasts from universities, museums and educational institutions all over the world on topics related to the course. podcasting in higher education among the general public and in particular among young people, web-based applications of sharing information, images and videos through youtube, facebook or twitter have been increasingly used (lenhart et al. 2010). the launch of ‘itunes u’ in american universities including yale, mit, stanford and berkeley by apple computer, inc. also shows that digital learning is becoming important in higher education and that technologies such as podcasting are being used more often (lonn and teasley 2009). most studies on the use of podcasting in higher education have focused on describing the results of pilot projects (e.g. edirisingha et al. 2007) and the benefits and shortcomings of using podcasts from the perspectives of university lecturers and students (e.g. sutton-brady et al. 2009). according to tohill’s study (2008), podcasts have been used most frequently in teaching engineering and science (33.3%), computer and information technology (33.3%), and business and law (13.3%). the number of studies on applications of podcasting in arts learning is quite small. kerstetter (2009) suggests that the use of podcasts enriches the music classroom in the dissemination of course content, provision of supplementary course materials, recording of visiting lecturers and provision of campus updates in music courses. in researching how technology can be employed in art education, buffington (2008) asked students to create podcasts on specific works of art. he found that students not only learnt about the technology of podcasting, but also engaged in deep discussions about the meanings of the artworks. types of podcast currently in use educational podcasts at university level can be divided into four categories based on their purposes of production, namely, teaching-driven (e.g. record of lecture), servicedriven (e.g. library tour), marketing-driven (e.g. student recruiting advertisement) and technology-driven (e.g. support provided by university technology/media departments) (harris and park 2008). rosell-aguilar (2007) further classified teaching-driven podcasts into teacher initiated and student initiated. for the former, there are ‘lecture podcasts’ and ‘supplementary podcasts’. lecture podcasts are simply records of lectures (copley 2007) while supplementary podcasts are materials that support learning in relation to core learning materials (lee and chan 2007). c.o. tam 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 nie (2006) lists various types of supplementary podcast that are usually used in higher education, including pre-lecture materials, feedback to students on assignments, records of interviews, supplementary topics that have not been covered in lectures, explanation of difficult areas of courses, and information for field activities. examples of student-initiated podcasts include assignments, discussions between students, interviews of specialists, as well as records and reflections on learning activities (abt and barry 2009). effectiveness of podcast learning there are two major approaches to investigating the effectiveness of podcast learning. the first approach analyses the effects through user self-report while the second makes use of experimental designs and focuses on the examination and test results of students. in general, studies which have adopted the first approach suggest that podcasts can enhance learning. anzai (2007) studied the use of podcasts to improve english learning for english as a foreign language students. the study found that podcasts were beneficial to learning because of special features they possess: for instance, the accumulation of chronologically ordered contents with rss, mobile learning and free subscription. bongey, cizadlo, and kalnbach (2006) surveyed 246 students on a biology course and found that the students perceived podcasts to be useful in increasing their understanding of material covered in the lectures. lane (2006) examined health sciences students’ experiences of using podcasts; 70% of the participants agreed that the podcasts supported their learning and were helpful in preparation for examinations. in clark et al.’s (2007) survey of 30 postgraduate marketing students, 96% indicated that podcasting enhanced their learning and 89% stated that podcasting helped them actively engage in learning. however, the results of experimental studies on the effectiveness of podcasts are varied. kurtz, fenwick, and ellsworth (2007) found a significant improvement in the final project grades of those students who used podcasts. by contrast, some researchers found a similar effect on student learning when using other review materials and did not find that the use of podcasts had a significant effect in enhancing student grades (copley 2007; lazzari, 2009). abt and barry (2007) adopted an experimental research design in the delivery of a physiology course. they found no significant difference between the podcast group and the control group. lakhal, hager, and pascot (2007) compared the final marks of students who used podcasts and those who did not. the results suggested that podcasts have a positive effect on student satisfaction but no effect on student performance. the present study subscribed to the first approach and collected data basically through user self-report survey and user interview. the focus of the study was on evaluating students’ general learning experience of using podcasts. methodology the aims of this study were to investigate the effectiveness of using podcasts to learn music and visual arts and to identify the characteristics of an effective educational podcast from the perspective of students. a total of 184 students enrolled in eight music or visual arts courses participated in the study. an evaluation survey and focus-group interviews were used to collect students’ feedback. the survey was administered to all students in the last session of each research in learning technology citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 3 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 course and three students from each course were randomly selected to attend the focus-group interview. a total of 128 students completed the survey and 24 students attended the interview. the results of the survey provided a general picture of how the students evaluated the effectiveness of using podcasts to learn. the focus-group interview allowed the researcher to explore the data collected from the survey in an interactive way. during the interview, semi-structured questions were asked and students commented on the podcasts offered in their courses and provided reasons for their ratings. participants and procedures at the beginning of the study, teachers teaching the eight courses developed podcasts taking into account the content, schedule and assessment of the course. these courses focused on topics of music or visual arts, including pedagogy, arts making and appreciation, arts administration, use of computer software and research skills in the arts. the podcasts produced were mainly supplementary podcasts and could be categorised into three types: informational (i), demonstration (d), and activity or assignment related (aar). informational podcasts delivered references, handouts or powerpoint presentations in video format with audio commentary by the teachers. most of the content on the informational podcasts had already been covered by the teachers during the lessons and they were basically developed for student revision purposes. demonstration podcasts showed the procedures or skills involved in running computer software such as band in a box, spss and adobe indesign. teachers either captured the computer screen and their voice during demonstrations in lessons or developed a video specifically to show the steps involved in using the software. students watched these podcasts when they tried the software application after lessons. for podcasts related to learning activities and assignments, students were required to respond to questions asked in the podcast, complete worksheets after watching the podcast or provide feedback by producing their own podcasts. all participants were full-time students studying for a bachelor of education or a postgraduate diploma in education. they came from various years of study and over half of them were specialising in music or visual arts. table 1 shows the courses and related podcast types. table 1. podcast type and course title. podcast type course title informational . art management and culture policy (management) . music technology and creative music making (technology) . print-making (print) demonstration . foundation computer-based music technology (computer) . honours project (project) . visual communication (communication) activity or assignment related . pedagogical approaches to music teaching and learning (pedagogical) . sculpture: an evolution of body consciousness and aesthetics (sculpture) c.o. tam 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 survey statements and interview questions at the beginning of the survey, four general questions related to the ways in which students downloaded and watched the podcasts were asked. they are presented in table 2. in order to evaluate the effectiveness of the podcasts used on the course they attended, the students were asked to rate six statements according to a five-point likert-scale, with values ranging from one (strongly disagree) to five (strongly agree). the statements were constructed based on the findings of past studies (fernandez, simo, and sallan 2009; sutton-brady et al. 2009; evans 2008). the six statements are presented in table 3. three students from each course were randomly selected and attended separate focus-group interviews after the last lesson. nine questions were asked to elicit students’ feedback: (1) what do you think about the advantages and disadvantages of using podcasting in learning? (2) do you think using podcasts is an effective way of learning? why? (3) do you think using podcasts will help to enhance academic achievement? (4) which kinds of learning needs can podcasting address more effectively than face-to-face teaching? what is the main difference between learning from podcasts and face-to-face teaching? (5) how do podcasts assist and supplement face-to-face teaching? (6) do you expect podcasts to be used in more courses in the future? (7) will you use podcasts in your future teaching? (8) can you describe what constitutes an effective educational podcast? (9) what would you suggest to improve/assist/support podcast learning? data analysis the spss statistical computer software was used to analyse the results of the survey. a reliability test was conducted before the analysis. cronbach’s alpha (a�0.7), a coefficient measuring internal consistency of the scores obtained from the survey, was ensured. an anova was carried out after the reliability test to determine whether there were any significant differences between the three groups of students: those who received (1) informational, (2) demonstration and (3) learning activity or assignmentrelated podcasts. the scheffe test (the most conservative test) and the bonferroni test (a less conservative test) would be conducted as a follow-up procedure if significant differences were found. both tests are methods for adjusting significance levels so as to prevent data from being incorrectly interpreted as statistically significant. for the focus-group interviews, the students’ responses were audio recorded and a complete typed transcript was made. to ensure anonymity, the name of each student was replaced with a code. in order to identify and classify the target variables, table 2. general questions on podcast watching. g1 what device did you use for watching the podcasts? g2 on average, how many times did you watch each podcast? g3 where do you usually watch the podcasts? g4 an effective educational podcast should be___minutes long. research in learning technology citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 5 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 categories were developed and the related data were allocated to available typological categories before the generalisation of the data. the interviews were analysed in the following three areas: (1) students’ attitudes towards learning with podcasts; (2) comparing face-to-face learning with learning from podcasts; (3) students’ expectations of effective educational podcasting. findings a total of 128 (70%) out of 184 students completed the questionnaire. of the 128 respondents, 24 (19%) students replied that had not watched any of the podcasts. only feedback from students who had watched the podcasts at least once (n � 104) was used in the analysis of the results. three students from each course, totalling 24 students (n � 24), were interviewed. cronbach’s alpha was 0.87, which was higher than the level of acceptance. comparing the effectiveness of different types of podcast overall, the students rated podcast provision of demonstrations of procedures or skills as the most effective and considered informational podcasts as the least effective. podcasts related to learning activities or assignments were rated in the middle. the details are presented in table 4. a one-way anova, a statistical test used to compare means of different groups, was conducted to explore any differences in effectiveness rating among the three groups of podcasts. significant differences were found between the evaluation of the effectiveness of informational podcasts and the other two types, f (2, 101) � 22.297, pb0.05. however, there was no significant difference between the evaluation of the effectiveness of podcasts integrating information with learning activities and assignments and that of podcasts demonstrating procedures or skills. in the following discussion, the capital letters i, d and aar table 3. evaluative statements on the effectiveness of podcast learning. s1 podcast increases my interest in learning s2 i like podcasting because i can learn at my own pace s3 podcast is a convenient way to learn s4 podcast is an effective way to revise the course content s5 podcast aids in assignment preparation s6 the visual effects in the podcasting increase my interest in learning table 4. mean scores for the effectiveness of different types of podcast. types of podcast number of students number of podcasts mean sd informational 37 28 3.12 0.73 demonstration 23 17 3.88 0.47 activity or assignments related 44 8 3.85 0.38 note: rounded to 2 sig. fig. c.o. tam 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 are used to represent informational, demonstration, and activities or assignmentrelated podcasts, respectively. one word in smaller letters is selected from the course title and is used to refer to the course. s1, s2 or s3 refers to the student being interviewed. demonstrations of procedures or skills received the highest rating among the three types of podcast. students indicated how these podcasts helped them to learn: by following the steps in the podcast, i can run the [spss] test smoothly at home. (d, project, s1) although the lecturer has explained or demonstrated the skills during lessons, we have to try and see if we really understand them. we can . . . follow . . . and pause whenever we like. (d, computer, s2) students reported that podcasts were especially helpful on courses that taught something abstract and were not easy to explain with notes alone. the video clips with verbal explanations in the podcasts helped the students to comprehend the content more clearly: it is obvious that podcasts are more effective than notes if teachers are teaching something about lyrics or editing the lyrics of old songs, as podcasts can show both audio and visual information at the same time. (aar, pedagogical, s3) with regard to podcasts integrated with learning activities or assignments, students rated these as more effective than ‘‘pure’’ informational podcasts, as shown in their interview responses: i don’t think it is particularly effective if the lecturer only asks me to watch the podcasts on my own, but it will be better if the lecturer can provide me with some follow-up activities or discussions. (aar, sculpture, s2) although informational podcasts were generally rated as the least effective, students commented that they could use them to review lecture content, and they thought it was a time-saving and convenient way to learn: the advantage is that i can learn at any time and any place i want, so i can make better use of my free time. (i, print, s2) an effective educational podcast from students’ perspectives in the interviews, the students commented on the content, relevance, length, presentation and technical aspects of an effective educational podcast. considering the nature of the content the students highlighted the nature of the content as being an important factor in deciding whether podcasts should be used. in their opinion, not all kinds of course content should be produced in podcast form: it depends on the content . . . if it is something factual or something like historical knowledge, i don’t think podcast is a good choice. (d, computer, s3) this is dependent on the subject matter. if it requires a deep level of logical thinking, say like designing a [teaching] plan, i think it should be presented in notes, and podcasts are not so helpful in this case. (aar, pedagogical, s2) research in learning technology citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 7 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 showing the relevance to lesson learning students pointed out that podcasts would be less effective if they could not see the relevance to their lessons. short informational podcasts which summarised the main points of a lesson were suggested: the podcast should be related to the content taught in lessons and state clearly to which part it is related. (i, technology, s2) i think a podcast should be a further explanation of the lecture content; it should focus on the main points . . . (d, project, s3) dividing into small sections and adding instructions as students could not interact with their teachers directly when they were using podcasts, they stated that it was important to have clear instructions on how they have to respond and how to complete follow-up activities. they also thought that adding titles, subtitles, captions and sections in the podcast might help: the teacher should explain what we need to do clearly at the beginning of the podcast. short summaries could be used to highlight the points if the podcast contains more than one section. (d, communication, s3) titles and subtitles should also be added so that we can find what we need more easily, just like reading a book . . . (aar, sculpture, s3) optimal length a question was asked in the survey to investigate the optimal length of an educational podcast. ninety-five students (85.5%) preferred podcasts that lasted for less than 15 minutes (table 5). this finding was supported by the data obtained from the interviews: i think a podcast should not be too long and should be limited to 10 minutes. i feel lost if it is too long and i don’t know which part i should watch . . . (i, print, s2) presentation with audio and visual effects with regard to the presentation of podcasts, the students suggested that more audio and visual effects such as animation could be added. teachers also needed to pay attention to their voice and tone: i agree that learning materials will be more attractive if some visual effects are used. (i, management, s1) it is too boring to listen to the lecture again and i think more pictures and clips should be added. (aar, sculpture, s1) the tone! it will be even more boring if the teacher talks in a dull tone of voice. (aar, sculpture, s2) table 5. preferred length of podcasts. less than 5 minutes 5�15 minutes 16�30 minutes half an hour to an hour longer than an hour an effective educational podcast should be___ minutes long. 24% 61.5% 12.5% 1.9% 0% c.o. tam 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 dissemination and downloading the students thought that an effective podcast should be easy to access and download: i think it is a must to teach students how to use podcasts and how to access them. (aar, pedagogical, s1) . . . it would be good if students only had to access one website in order to find all the podcasts they needed. (d, communication, s3) discussion effective use of different types of podcast to supplement face-to-face teaching concerning the content of podcasts, the students’ rating of demonstrations as the most effective can be explained in several ways. an examination of those courses that provide demonstrations (graphic design, music creation and research methods) reveals that the application of software constituted an important part of the course. in learning these applications, students had to practise repeatedly in order to become familiar with the skills and apply them in assignments or graduation projects. in many of these podcasts, procedures were explained audibly and visually. this is supported by brittain et al.’s (2006) findings that information-dense course content with a heavy reliance on visual images is better podcasted. actively utilising the multimedia technology in presenting information, the podcasts have created a multisensory learning experience for students. the importance of using podcasts to review demonstrations was also highlighted by the students who participated in the present study, as well as by the students of biomedical subjects who took part in brittain et al.’s study (2006). the reviewing function of podcasts allows students to concentrate on what the lecturer said instead of trying to record all the details presented in lessons. with podcasts, students can trace information missed and catch up with the progress of teaching by self-learning. compared to demonstrations, students rated informational podcasts as less effective. the students’ responses suggest that if the material presented in the podcast is simply a repetition of that presented in the lecture, the podcast is not useful. for most of the courses in this study, informational podcasts were used to provide supplementary learning materials and students watched them only if they felt it necessary. some of these podcasts were records of individual student or group work while many of them were powerpoint or notes with audio commentary that had been gone through during lessons. in some cases, students preferred to have hard copies of handouts because they could make notes directly on the handouts and this made a deeper impression on them during revision. although previous studies have noted that the reproduction of lectures may lead to benefits (brittain et al. 2006) such as providing the lecture for those who did not attend and a record to support revision (evans 2008), the students who took part in the current study suggested that the reproduction of a lecture in the form of a podcast is not useful. in fact, researchers point out that the strategy of simply reproducing lectures in podcasts is in conflict with learning theories that advocate richer and more active learning experiences for learners (middleton 2009). another reason that may account for the low rating of informational podcasts is the time of conducting the survey and the interviews. as they were conducted in the last session of each course, the students by that time had not yet recognised the benefits of using podcasts as revision aids for final assignment preparation. research in learning technology citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 9 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 however, podcasts integrating information with learning activities or assignments obtained higher ratings when compared to ‘‘pure’’ informational podcasts. in these cases, students were able to see the connection between the podcasts and their learning or assessment. they felt involved and were motivated to watch the podcasts. the students also mentioned the importance of explicitly indicating the relevance of the podcasts to their studies. this demonstrates that the effectiveness of using podcasts to learn is determined not only by the unique advantages of podcasting itself, but also by teachers’ teaching strategies. an integration of podcasting with existing pedagogies will facilitate the effective use of podcasts. it is suggested that teachers can relate podcasts to classroom teaching with follow-up discussions, learning tasks and assessment. appropriate length and use of multimedia technology in podcasting in this study, we found that students were in favour of relatively short podcasts and they preferred podcasts to be an edited summary of lessons, further explanations of difficult concepts or a quick review of what was taught in lessons. a majority of the students suggested that an educational podcast should be at most 15 minutes long. this finding is in line with the findings of previous studies that the length of podcasts should range between 5 and 20 minutes (anzai 2007; chan and lee 2005; muppala and chan 2007). the increasing use of multimedia technology in education has allowed teachers to create learning resources that appeal to students (sutton-brady et al. 2009) and stimulate their learning motivation (belanger 2007). in the opinion of the students who participated in this study, audio and visual effects could be added to podcasts to make them more attractive. the students commented that the use of animation in podcasts increased their interest in learning and that in some cases they preferred watching clips to reading words. this supports evans’ (2008) findings that university students today are more receptive to learning using podcasts than traditional learning materials. this is also related to the environment in which students have grown up. campbell (2005) suggests that current university students are familiar with working in ‘‘rich media’’. students not only have a good understanding of multimedia but also use them as a ‘‘language’’ on a daily basis. in a study of 183 undergraduate students, howard, ellis, and rasmussen (2004) also note that multimedia with their video, audio and graphics effects can capitalise on the contemporary rich media environment and contribute significantly to learning. however, for teachers, creating podcasts with appealing visual effects or animations inevitably implies more workload. in this connection, technical and human resource support from the teaching institution becomes indispensable for the successful implementation of podcast learning. technical issues hew (2009) reviewed past studies on the barriers to using podcasts and found that unfamiliarity with podcasts and technical problems in accessing and downloading were common problems in podcast learning. during the interview conducted for the current study, a number of students expressed difficulties in subscribing to the podcast files, especially at the beginning stage of the study. this is the major disadvantage associated with podcast learning explicitly mentioned by students. the students also mentioned that they preferred to go to one website to access the podcasts instead of c.o. tam 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 navigating through many different sites. students nowadays have a better understanding and experience of using multimedia tools, but the use of podcasts or the rss feed is still relatively new to them. the findings also reflected the importance of choosing a platform for uploading and downloading and of technical support being provided throughout the learning process. the suggestions from the students supported past findings. to ensure the same learning opportunity for all students in using podcasts and to lessen the influence exerted by differences in levels of computer literacy, introductory sessions on what podcasting is, how to download podcasts from the selected platform, and how to fix common technical problems should be provided. conclusions and suggestions for further studies in general, the findings suggested that podcasts helped the students who took part in this study in achieving personalised learning. they perceived podcasting as a platform that provided supplementary learning materials to support face-to-face teaching. among the three types of podcast that have been identified, demonstration and activity or assignment-related podcasts are more effective than informational podcasts. podcasts that demonstrate the application of skills and content that requires audio and visual explanation are well received. simply converting lecture powerpoints or handouts into podcast files does not work well. students strongly suggested that instead of duplicating course content that has been taught in lessons in podcasts, teachers should integrate podcasts with lesson learning in the form of follow-up discussions, completion of assessment-related tasks or extension of lesson learning activities. the study has provided educators with an initial categorisation of podcasts according to their functions, purposes and their relative effectiveness in teaching music and visual arts courses. future podcast producers may work on this taxonomy and create finer classifications addressing the different nature of podcasts and their effectiveness. this paper has pointed out some of the factors that teachers may take into account when deciding whether and how podcasts should be used. further exploration of other factors, such as the relationship between podcast effectiveness and student educational levels, the use of podcasts in learning different types of subject matter, and the relationship between podcasts and student digital competency, is needed. there is still little description of how to combine podcasting successfully with existing pedagogies and there is no evidence-based model to follow, in either strategy or practice. there is also a lack of research on supporting podcast learning at an institutional level and there is a pressing need to develop a strategic plan for mobile learning. these aspects of podcast learning present possible directions for future studies. references abt, g. & barry, t. (2007) ‘the quantitative effect of students using podcast in a first year undergraduate exercise physiology module’, bioscience education e-journal, vol. 10, pp. 1�9. abt, g. & barry, t. (2009) a strategy for using podcasts for teaching and learning in the biosciences, centre for bioscience, the higher education academy, university of leeds, leeds, [online]. available at: http://www.bioscience.heacademy.ac.uk/resources/projects/ barry.aspx anzai, y. (2007) ‘empowering english learning utilizing podcasts’, in proceedings of world conference on e-learning in corporate, government, healthcare, and higher education 2007, ed. g. richard, aace, chesapeake, va, pp. 10�15. research in learning technology citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 11 (page number not for citation purpose) http://www.bioscience.heacademy.ac.uk/resources/projects/barry.aspx http://www.bioscience.heacademy.ac.uk/resources/projects/barry.aspx http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 belanger, y. (2007) summary of duke digital initiative instructional programs 2006�2007, duke university, durham, nc. bongey, s. b., cizadlo, g. & kalnbach, l. (2006) ‘explorations in course-casting: podcasts in higher education’, campus-wide information systems, vol. 23, no. 5, pp. 350�367. brittain, s., et al. (2006) ‘podcasting lectures: formative evaluation strategies helped identify a solution to a learning dilemma’, educause quarterly, vol. 29, no. 3, pp. 24�31. buffington, m. l. (2008) ‘creating and consuming web 2.0 in art education’, computers in the schools, vol. 25, no. 3�4, pp. 303�313. campbell, g. (2005) ‘there’s something in the air: podcasting in education’, educause review, vol. 40, no. 6 (novembr/december), pp. 32�46. chan, a. & lee, m. j. w. (2005) ‘an mp3 a day keeps the worries away: exploring the use of podcasting to address preconceptions and alleviate pre-class anxiety amongst undergraduate information technology students’, in good practice in practice. proceedings of the student experience conference 5�7th september’05, eds d. h. r. spennemann & l. burr, charles sturt university, wagga wagga, nsw, pp. 59�71. clark, s., sutton-brady, c., scott, k. m. & taylor., l. (2007) ‘short podcasts: the impact on learning and teaching’, in proceedings of mlearn conference 2007, eds a. austin & j. pearce, university of melbourne, australia, pp. 285�289. copley, j. (2007) ‘audio and video podcasts of lectures for campus-based students: production and evaluation of student use’, innovations in education and teaching international, vol. 44, no. 4, pp. 387�399. edirisingha, p., et al. (2007) ‘podcasting to provide teaching and learning support for an undergraduate module on english language and communication’, turkish online journal of distance education, vol. 8, no. 3, pp. 87�107. evans, c. (2008) ‘the effectiveness of m-learning in the form of podcast revision lectures in higher education’, computers and education, vol. 50, no. 2, pp. 491�498. fernandez, v., simo, p. & sallan, j. m. (2009) ‘podcasting: a new technological tool to facilitate good practice in higher education’, computers and education, vol. 53, no. 2, pp. 285�392. harris, h. & park, s. (2008) ‘educational usages of podcasting’’, british journal of educational technology, vol. 39, no. 8, pp. 548�551. hew, k. f. (2009) ‘use of audio podcast in k-12 and higher education: a review of research topics and methodologies’, education tech research dev, vol. 57, no. 3, pp. 333�357. howard, w., ellis, h. & rasmussen, k. (2004) ‘from the arcade to the classroom: capitalizing on students’ sensory rich media preferences in disciplined-based learning’, college student journal, vol. 38, no. 3, pp. 431�440. kerstetter, k. (2009) ‘educational applications of podcasting in the music classroom’, music educators journal, vol. 95, no. 4, pp. 23�26. kurtz, b. l., fenwick, j. b., jr. & ellsworth, c. c. (2007) using podcasts and tablet pcs in computer science, [online]. available at: http://portal.acm.org/citation.cfm?doid� 1233341.1233428 lakhal, s., hager, k. & pascot, d. (2007) ‘evaluation of the effectiveness of podcasting in teaching and learning’, in proceedings of world conference on e-learning in corporate, government, healthcare, and higher education 2007, eds g. richards & s. carliner, aace, chesapeake, va, pp. 6181�6188. lane, c. (2006) podcasting at the uw: an evaluation of current use, the office of learning technologies, university of washington, [online]. available at: http://www.washington. edu/lst/research_development/papers/2006/podcasting_report.pdf lazzari, m. (2009) ‘creative use of podcasting in higher education and its effect on competitive agency’, computers and education, vol. 52, no. 1, pp. 27�34. lee, m. j. w. & chan, a. (2007) ‘pervasive, lifestyle-integrated mobile learning for distance learners: an analysis and unexpected results from a podcasting study’, the journal of open and distance learning, vol. 22, no. 3, pp. 201�218. lenhart, a., et al. (2010) ‘social media and young adults’, pew internet and american life project, [online]. available at: http://www.pewinternet.org/reports/2010/socialmedia-and-young-adults.aspx lonn, s. & teasley, s. d. (2009) ‘podcasting in higher education: what are the implications for teaching and learning?’, the internet and higher education, vol. 12, no. 2, pp. 88�92. c.o. tam 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 http://portal.acm.org/citation.cfm?doid=1233341.1233428 http://portal.acm.org/citation.cfm?doid=1233341.1233428 http://portal.acm.org/citation.cfm?doid=1233341.1233428 http://portal.acm.org/citation.cfm?doid=1233341.1233428 http://portal.acm.org/citation.cfm?doid=1233341.1233428 http://www.washington.edu/lst/research_development/papers/2006/podcasting_report.pdf http://www.washington.edu/lst/research_development/papers/2006/podcasting_report.pdf http://www.pewinternet.org/reports/2010/social-media-and-young-adults.aspx http://www.pewinternet.org/reports/2010/social-media-and-young-adults.aspx http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 middleton, a. (2009) ‘beyond podcasting: creative approaches to designing educational audio’, research in learning technology, vol. 17, no. 2, pp. 143�155. muppala, j. k. & chan, k. k. (2007) ‘podcasting and its use in enhancing course content’, in cate’ 07 proceedings of the 10 th iasted international conference on computers and advanced technology in education, ed. v. uskov, acta press, anaheim, ca, pp. 492�495. nie, m. (2006) the pedagogical perspectives of mobile learning, [online]. available at: http:// hdl.handle.net/2381/407 rosell-aguilar, f. (2007) ‘top of the pods*in search of a podcasting ‘‘pedagogy’’ for language learning’, computer assisted language learning, vol. 20, no. 5, pp. 471�492. sutton-brady, c., et al. (2009) ‘the value of using short-format podcasts to enhance learning and teaching’, research in learning technology, vol. 17, no. 3, pp. 219�232. tohill, k. (2008) ‘i podcast, you podcast, together we podcast: podcasting as a learning tool in second language classrooms’, in proceedings of society for information technology and teacher education international conference 2008, eds k. mcferrin, et al., aace, chesapeake, va, pp. 3645�3650. research in learning technology citation: research in learning technology 2012; 20: 14919 doi: 10.3402/rlt.v20i0.14919 13 (page number not for citation purpose) http://hdl.handle.net/2381/407 http://hdl.handle.net/2381/407 http://researchinlearningtechnology.net/index.php/rlt/article/view/14919 supporting sustainable e-learning allison littlejohn centre for academic practice, university of strathclyde email: allison.littlejohn@strath.ac.uk this paper draws upon work carried out within phase one of a national forum for support staff, funded by the uk learning and teaching support network generic centre. it sets out themes in current learning technology research within the context of institutional practice. it reports the responses of a range of e-learning support staff to new developments in the reuse and sharing of learning objects. the article highlights tensions across support units, inconsistencies in support provision and confusion over issues concerning different modes of teaching. it also forewarns a growing gap between institutional practice and research in the development of approaches to sustainable elearning. issues in developing flexible and sustainable e-learning the united kingdom government recently published a consultation document 'towards a unified e-learning strategy' focusing on flexible and sustainable e-learning (department for education and skills (dfes), 2003a). this document envisions e-learning as an integral aspect of the learning and teaching environment, in which networked learners access richly structured, highly accessible, interactive settings (macfarlane, 1998; national committee of inquiry into higher education (ncihe), 1997; dfes, 2003b). this vision of flexible access to learning is shared by governments across the world who appreciate e-learning as being one way of providing education that can be adaptable and cost effective. huge investments have already been made to provide an infrastructure that can support the development of flexible curricula. however, the government is seeking to drive down development costs mainly through the promotion of sustainable approaches to e-learning (ncihe, 1997, appendix 2). central to this approach is attaining a balance between costs and strategic benefits (nicol and coen, 2003). evaluation of the benefits of e-learning is largely subjective and there is an elaborate interplay between costs and benefits. an additional complication is the fact that there are a variety of social, technological and 88 alt-] volume i i number 3 organizational factors that impact on the quality of teaching and learning, therefore the benefits arising from e-learning cannot be measured in isolation (draper, brown, henderson and mcateer, 1996). cost reduction is largely achieved through economies of ' scale, so many recent initiatives have focused upon reducing costs through the reuse and sharing of learning materials frequently referred to as learning objects (for example, the jlsc-funded exchange for learning programme, x4l). according to oliver and maclaughlin (2003), 'the more reuse is considered in design, the more sustainable online teaching and learning can become.' in an ideal world, these learning objects would be developed and shared by teachers and learners. they would be designed so that they could be adapted to fit different subject disciplines, levels of study and educational models (koper, 2001). this would potentially lead to educational benefits, with students profiting from enhanced quality of learning as well as improved access (dfes, 2003c). additional benefits that can arise from e-iearning include increased innovation, enhanced public profile, increased staff satisfaction and motivation and increased wealth creation (nicol and coen, 2003). despite these claims, many lecturers remain sceptical of the perceived benefits and have been slow to adopt e-learning (green, 1999; campbell, 2003). staff training in 'how to use' various types of supporting educational tools has not led to a dramatic increase in the use of these tools, for purposes other than the delivery of course materials. as crook (2002) observed: 'clearly academics do master other information technology tools routinely where they are perceived as powerful resources for their work.' why, therefore, has there been limited use of e-learning tools by academics? one reason is because designing and implementing a course is much more complex than simply sending e-mail, or using powerpoint for presentations. academics need clear support in how to perform difficult tasks such as designing, assembling and implementing courses supported by technology. this requires a coherent, supporting infrastructure comprising not only the required technologies and network systems, but also organizational structures (for example, local and national support mechanisms), and the people that make these systems work (including support staff and learning technologists) (dfes, 2003a, para 49). faced with the challenge of facilitating change across institutions, support staff are frequently confronted by tensions and conflicts that arise across academic departments as well as within support units themselves (mcnaught, 2002). this article outlines these tensions and describes ways in which they may inhibit the advancement of sustainable elearning. it describes the outcomes of phase 1 of a national forum for support staff, funded by the uk learning and teaching support network generic centre. the forum aimed to give support staff an opportunity to explore ways in which learning technology research, particularly in the area of interoperability, might impact upon future provision for elearning support. the unique aspect of this article is that it sets out current themes in learning technology research within the context of institutional practice, providing an insider view of the positive and negative aspects of existing support for e-learning. the article explores potential meanings of 'sustainable e-learning', examines current modes of support and investigates the implications of balancing costs and benefits. 89 allison litzlejohn supporting sustainable e-leaming implementing a uk national forum a variety of government-funded organizations already exist to support the development of sustainable e-learning. for example the learning and teaching support network (ltsn) is a national network for staff working in higher education, with twenty-four subject centres and a generic centre. each centre has provided considerable support for the development of e-learning in higher education, and in doing so has begun to question the sustainable characteristics of recent developments, in particular the drive toward content creation. the ltsn has sought to promote the sustainability of e-learning by encouraging its embedding within teaching practice and through working with support staff to tease out practical strategies towards this end (wiles and littlejohn, 2003). in 2003, the ltsn generic centre established the supporting sustainable elearning forum as an opportunity for the exchange of ideas by a wide range of support staff, educational developers and learning technologists from the united kingdom higher and further education communities. the aim of the forum was to devise practical strategies for supporting staff in the design, development and implementation of online courses; to disseminate good practice in supporting sustainable approaches to e-learning; and to contribute to the ongoing debate on the sharing and reuse of e-learning resources. through the forum, practitioners were invited to examine how to move e-learning on from project to embedded practice, and address questions around the scaleable nature of e-learning. the forum had four main themes, each of which was explored during four one-day sessions held in 2003 at the university of strathclyde (march), the ltsn generic centre in york (april), the open university (may) and the university of bristol (june). these themes were: designing sustainable online courses, implementing sustainable online courses, the design and development of reusable materials, and institutional change. each session was based on contributions, collaboration, construction and communication of new ideas: an approach used successfully by harris and higgison (2003). during each session key contributors were invited to present ideas, through draft commentaries and presentations. participants could share ideas and collaborate by brainstorming ideas during discussion fora. the ideas generated were drawn together to work towards and construct new models of staff development. these were communicated through the forum website and an electronic discussion list. the seventy-five forum members represented fifty different (mainly higher education) institutions from across the united kingdom. members were mostly support staff, including learning technologists, librarians, audio-visual staff, it support staff and directors of learning and teaching centres. their experiences of potential tensions in embedding sustainable e-learning and their responses to issues in current research and how this could impact upon learning and teaching practice are outlined in the following sections. what is 'sustainable e-learning'? e-learning is a term that is used in radically different ways by different people. key contributor martin oliver led an interactive session in which he asked forum delegates to write down their personal definition of 'e-learning'. most delegates did not find this task straightforward and a wide variety of definitions were drawn up. this is unsurprising, since 90 alt-] volume 11 number 3 there is similar ambiguity in the literature. kaplan-leiserson (2000) has a broad definition in which she asserts that e-learning is a 'term covering a wide set of applications and processes, such as web-based learning, computer-based learning, virtual classrooms and digital collaboration. it includes the delivery of content via internet, intranet/extranet (lan/wan), audio and video tape, satellite broadcast, interactive tv, cd-rom and more.' this definition alludes to the use of learning activities supported by technology; it may also refer to systems used for content delivery, for example, to distribute lecture notes, handouts and course information. jackson (2002) attempts to differentiate between these by defining e-learning in two ways: as technology-enhanced learning and technologydelivered learning, technology-enhanced learning is illustrated by the numerous modes of e-learning that exploit the communicative aspects of networked technology. many cases in point are from the field of open and distance learning, particularly within continuing professional development courses leading to professional accreditation (timms, booth, crompton, klein, bangali and schnuekel, 1999; ingraham, watson, mcdowell, brockett and fitzpatrick, 2001). this is partly influenced by the fact that students on distance learning courses often have limited scope for dialogue with tutors and peers. examples of technology-delivered learning are frequently found across campus-based institutions, where students' needs and expectations are different. a study by crook and barrowcliff (2001) presents a representative snapshot of the use of e-learning by campusbased undergraduate students. it is interesting to note that both lecturers and students advocate the use of e-learning systems primarily to supplement the face-to-face interactions of lectures and tutorials for the distribution of lecture notes, handouts and course information. therefore, it is important to recognize that the term 'e-learning' can describe a range of learning methodologies and also to draw distinctions across the application of e-learning for undergraduate, postgraduate, campus-based learning, distance learning and work-based education. frequently, in discussion on e-learning, these approaches to learning are grouped together, which is confusing for academics. delegates were also unclear on the notion of 'sustainable'. according to the national committee of inquiry into higher education (1997, appendix 2), sustainable e-learning is the adoption of technology to maintain teaching quality at reduced unit costs. a variety of initiatives are achieving this through economies of scale (collis and moonen, 2001). these programmes are focusing on the development of learning materials as learning objects that are described by metadata, classified and stored in digital repositories, where they can be easily accessed, recombined and reused within online courses (for example exchange for learning, x4l; resource discovery network, rdn). for the purposes of the forum 'sustainable elearning' was defined as 'the design and development of online courses that could easily be updated or scaled up'. central to this definition were notions of repurposing and reusing learning objects. this definition was not intended to be comprehensive, but rather served to help delegates and contributors discuss issues from a common perspective. how is e-learning development currently supported? common perspectives in terms of e-learning support are seldom achieved. this is partly because e-learning support structures vary widely across institutions (beetham, 2001). 91 allison utllejohn supporting sustainable e-leaming they range from centralized 'learning service', 'teaching and learning' and 'academic practice' units to decentralized support within faculties frequently located in it support units and libraries. the staff working within these units have a variety of responsibilities (oliver, 2003). staff with similar roles took part within the forum, representing their respective cultures, driving forces and agendas. therefore, it was unsurprising that forum delegates frequently had different views of e-learning and its focus. some delegates reported that the support and advice offered to an academic seeking assistance is frequently biased towards the perspective of an individual support staff member or unit. for example, advice given to an academic seeking help from an audio-visual unit may be very different from the advice that might be given to the same academic with a similar problem by a learning and teaching unit. while it is useful to be offered a range of opinions, much of the advice offered to academics tends to focus on content, rather than on underpinning educational issues. content issues are relatively easy to address, compared with complex pedagogical problems, and therefore may appear to offer a simple (and therefore attractive) solution. inconsistencies in approaches to staff support have arisen, due partly to the large number of support units existing within institutions. inevitably, tensions arise across and between these units, leading to contradictions in support provision. this is largely caused by the uncertainty of support staff roles and identities in a rapidly changing environment (oliver, 2002). an elearning strategy could tackle issues of poor communication between groups of support staff and with academics by aiming to promote better cohesion and collaboration. existing strategies that have proved to be successful are frequently based upon partnerships with academic 'champions', or round-table discussion methodology (ehrmann, 1998). feedback from participants indicated a variety of conflicts that hinder e-learning implementation. in an interactive session led by key contributor grainne conole, forum delegates were invited to reflect upon the variety of ways in which e-learning is currently supported within their institutions and explore these underlying tensions (sself, conole briefing paper, 2003). most institutions offer a wide range of staff development activities. these include awareness-raising events (workshops, institutional conferences, etc.) as well as tailored support, including departmental sessions, away days, one-to-one staff mentoring and institutional development projects. many institutions provide accredited, postgraduate courses leading to membership of the uk institute for learning and teaching in higher education. however, there is a noticeable move away from traditional workshops and courses towards developing partnership with academics (for example, on programme development, joint writing funding bids or working within formal accreditation programmes and learning and teaching committees). another form of partnership is through joint development of learning resources (for example, online guidelines, templates and case studies or portals linking to relevant external materials and support). institutions have diverse implementation strategies for staff support programmes. the structure of each programme is largely influenced by a range of context-specific driving forces that vary across institutions, schools and faculties. local drivers include research agendas, pressure from external professional organizations and partnerships with other institutions. there are additional drivers resulting from national government policy such as inclusion and quality enhancement. these influencing factors frequently contradict one 92 aa-] volume 11 number 3 another; however, staff support programmes have evolved that use these conflicts to their advantage. for example, programmes exploring the development of accessible learning resources can also be designed to take account of new content requirements stipulated by professional bodies. tensions arising between local and national agendas can result in national e-learning initiatives having limited impact at an institutional level, especially for short-term projects (littlejohn and peacock, 2003). funding bodies expect that project outcomes will be easily embedded within institutions, but there has been limited institutional support for staff developers in achieving this goal. this could reflect conflicting approaches to collaboration: externally funded projects are frequently based upon the premise that collaboration among institutions is mutually beneficial. however, institutions may have a more introspective and competitive culture, which does not always facilitate collaboration. this tension could be reduced through support staff being actively involved in the planning of learning and teaching strategies to ensure that the outcomes of national projects are linked to local initiatives (mcnaught, 2002). the proposed academy for the advancement of learning and teaching in higher education could help strengthen these links. how can costs be reduced? the dfes consultation document suggests that e-learning can reduce costs in a variety of ways. first, e-learning can improve the running of the organization by automating administrative functions (dfes, 2003a, para 71). second, it has the potential to increase teaching efficiency and may help address shortages of resources and teachers in certain subject areas (para 61). thirdly, resources can be reused and shared across courses and subject disciplines (para 56). over the past few years, cost-reduction strategies have largely focused on the reuse of content. however, other types of resources, such as student activities and course designs, can also be reused (koper, 2003). all these types of resources can be copied and reused using learning management systems (lms), sometimes referred to as virtual learning environments. an lms is a software environment integrating a series of tools which can support a range of teaching and learning activities (britain and liber, 2000). using these tools academic staff can set up online courses without the need for intensive it training. forum contributor, barbara newland, reported on her work in reusing course designs within a learning management system, blackboard (sself, briefing paper by newland, 2003). for several years newland has been developing strategies for supporting large numbers of academics in using blackboard at the university of durham. this support involves assisting, academics in designing courses made up of reusable student activities as well as reusable content resources. at durham reusable content sourced from academics, audio-visual staff and publishing houses has been redeployed across several different modules. a range of student activities has also been reused. this includes online discussion tasks, group project assignments, computer-marked and computer-managed assessments as well as interactive simulations. in addition, academics have redeployed course designs that have proved to be effective (for examples, see the leap website). these designs were duplicated within blackboard and repurposed by the original course designer, or other academics. according to newland, 'at the end of the academic year the course is ready for the next year with the same content and with a new set of students. the academic can just click radio buttons to make materials available or unavailable at certain dates.' 93 allison uttlejohn supporting sustainable e-leaming repurposing, rather than reusing resources, can feel more comfortable to academics and can help increase their teaching efficiency. most academics already have content resources that they use during lectures and tutorial sessions and content from a single module can be digitized, redesigned and repurposed for reuse in other modules. mary thorpe described a process of repurposing content that has been pioneered at the uk open university (thorpe, kubiak and thorpe, 2003). during this process, existing learning resources, such as text documents, images, etc., are transformed into free-standing learning objects that can be updated, 'without creating knock-on effects to any other part of a course'. there are two critical factors that are essential to this process: clear structuring of texts that can easily be adapted during the authoring process, and the application of metadata, enabling the resources to be searchable (sself, briefing paper by thorpe, 2003). the difference between repurposing and reusing content is important within the context of higher education, since most he lecturers prefer to reuse their own materials. however, within the primary, secondary and further education sectors, there is a longstanding, pervasive culture of reusing standardized learning materials. gerry graham, of 'learning and teaching scotland', presented his experiences in supporting teachers across scotland to share digital learning resources. these resources are stored within a central digital repository. teachers can search for learning objects by keying terms into a search tool. the search results return metadata information about each resource, such as the author, format and description of the resource. this information can help teachers quickly evaluate the suitability of each resource within their own teaching situation (sself, briefing paper by graham, 2003). learning objects can be submitted to a repository in a variety of formats. during the process of uploading, the resources will be described using metadata elements holding information about the author, educational intent, etc. this information is linked to the learning object as metadata. to complete the upload process to the digital repository, the learning object is classified using an educational taxonomy. metadata is crucial to this process for a variety of reasons. it enables searching and sourcing of content resources; it allows authorships and associated rights to be retained; it facilitates communication of educational intent; and it permits matching of learners' needs with appropriate content (sself, stiles briefing paper, 2003). until now, most institutions have implemented learning management systems that do not support these functions, restricting the reuse and sharing of resources and limiting opportunities to reduce costs. forum responses many forum delegates had attempted to reduce costs within their institution by encouraging the sharing and reuse of resources. however, this process was frequently complicated by a series of technological and social issues. delegates identified several technological problems that had inhibited resource sharing with academic staff. in their experience, several commercial systems (such as blackboard and webct) did not allow for easy resource sharing, resulting in duplication of materials within institutions. this shortcoming frequently led to further problems, since multiple copies of single resources had to be updated individually. temporary solutions had been tested, including storing resources at a location outwith the learning management system and creating links to these resources from online courses contained within the lms. however, this method had 94 alt-] volume i i number 3 limited success, since there was no means by which academics could source available resources. the best solution was to store resources within a searchable database or digital repository. delegates confirmed they had encountered a variety of social factors inhibiting resource sharing. these included academics' confidence in sharing self-generated materials, concerns over intellectual property rights (ipr), the time required by them to implement metadata tagging or to upload resources. three potential solutions were suggested to improve confidence in sharing and reusing materials. first, to build academics' confidence in reusing resources by making available 'off the shelf materials. second, to encourage lecturers to reuse their own materials, initially by themselves, then with close colleagues within their subject discipline, school or faculty. third, to advise academics to share students activities and course designs as well as content resources. while the use of metadata may reduce concerns in terms of ipr, the time required to describe resources with metadata is a major disincentive for most staff. a potential solution to this problem is to encourage lecturers to describe resources with small amounts of metadata. this would allow them to gauge the benefits of using metadata, without requiring a large, initial time input. a major factor critical to the successful reuse of learning objects is the degree of ownership felt by the academic. consequently resources are most likely to be reused by the original author. resources created by a support unit have a greater chance of being reused if developed in partnership with the academic. however, externally generated resources have much less chance of being reused in higher education. extensive funding has generated a variety of nationally available, reusable learning objects (for example x4l, rdn). in order to encourage the reuse of these resources closer links must be formed between these national initiatives and departments within institutions. once again, the proposed academy for the advancement of learning and teaching in higher education could strengthen these links. delegates reported that few institutions had reduced costs through the automation of administrative functions. all the universities and colleges represented had implemented computer-marked assessment software, though student records usually had to be transferred to record systems manually. some had linked computer-assessment software with student record systems. although there were no reports of fully integrated systems, some institutions were planning to implement these within the next academic year. these integrated structures will have a major impact on the development of student portfolios (an area that will be further explored in phase 2 of the sself forum). a large proportion of the 'reuse' debate has focused on content development. however, the strengths of higher education institutions lies in 'softer' areas. mark stiles summed this up by stressing that, 'educational institutions do not sell "content", but rather the provision of access to expertise and support, assessment and accreditation of learning.' therefore, it is important to explore ways in which educational benefits can be maximized. how can benefits be increased? the benefits of implementing sustainable e-learning can be viewed at three levels within an 95 allison utdejonn supporting sustainable e-learning institution: educational, organizational and external (nicol and coen, 2003). developments within the educational and organizational spheres will inevitably impact upon the external perception of an institution. if the technology is working well and students are being provided with better education, this will have a major, positive impact. any negative impact is likely to reduce the potential for income generation, so enhancement in the educational sphere will focus on learning and the learner. developments are likely to be influenced by the emergence of new systems that allow lecturers easily to design courses based on different educational models. one such approach, based on learning design, a notational system currently being developed by ims (www.ims.org), was presented by colin tattersall of the open university of the netherlands (sself, tattersall briefing paper, 2003). learning design involves course designs in which students are assigned activities (or tasks) and roles (for example group moderator, reporter, etc.). students have access to content appropriate to each task. the sustainability of courses is determined by the reuse of all these types of resources: content, activities and learning designs (koper, 2003). this approach requires a shift in the way elearning is implemented and supported. learning management systems that are being developed around learning design (see, for example, dalziel, 2002) encourage academics to design courses that are based on student activities linked to learning objectives. according to tattersall, 'it recognises that learning can happen without learning objects, that learning is different from content consumption and that. . . learning happens when learners cooperate to solve problems in social and work situations.' a critical factor affecting reuse is the size of a learning object. newland reported that smaller resources could be more easily adapted to specific learning contexts. these ideas were further explored by boyle (2002), who presented his research examining the relationship between the degree of reusability of a learning object and its size. boyle's aim was to ascertain whether or not a there is a favourable size that makes materials more reusable. he discovered that the size ultimately depends upon the 'level' of reuse. boyle classified reuse at four levels. at a basic level simple 'information objects' or 'learning objects' could be reused. at a more complex level, these objects could be aggregated to form larger 'topic based learning activity structures' or 'course based learning activity structures'. boyle's research has demonstrated that the reuse of learning objects at different levels can improve learning. however, many lecturers remain sceptical of the value of using learning objects, believing this approach leads to a fragmented, reductionist pedagogy. this issue was explored by robin mason, of the uk open university. mason believes that designing courses comprising learning objects (both student activity and information resources) can 'provide a rich learning environment and allow for easy updating and re-versioning' (sself, mason briefing paper). the four areas in which she believes the reuse of learning objects can promote a student-centred approach to learning include the accommodation of learner diversity; learner choice and selection; activity or problem-based learning (which resonates with the learning design approach); and interactivity and collaborative work. she considers that, in the future, there is likely to be greater emphasis on developing the interactive and communicative nature of the web. students will create their own learning objects, upload these resources to a repository and draw upon those of their peers. work is already under way to explore not only how to capture ideas, as learning objects, but the 96 alt-] volume 11 number 3 process by which these ideas were developed (didet). this may be achieved by capturing unwritten thoughts and courses of action as metadata. when teachers create resources, they naturally embed them within a particular context. however, to extend their potential for reuse, resources should be context-free and should not contain information specific to a particular subject discipline or course. this contradicts the way that lecturers normally modify and adapt resources to fit specific teaching situations, disciplines and abilities of students. patrick mcandrew addressed this critical issue by posing the following question: 'if some software, for example, is used in one situation to support learners working at a distance in their own time, and then the same software is used on campus in a fixed time tutorial, what is the most useful aspect to transfer across?' (sself, mcandrew briefing paper). mcandrew drew upon five potential ways in which these aspects could be addressed: metadata tagging, open source courseware, reusable learning objects, reusable learning activities and community building. he concluded that automatic metadata tagging is a potential time-reducing incentive to encourage the uploading of resources to shared repositories. however, mcandrew voiced his concern that focusing development in this area may 'design the people out of systems when the subtlety of education means that they should be very much designed in'. his suggestion is to link multiple methods in order to provide a more holistic approach. e-learning systems can enhance the organizational aspects of an institution by automating routine administrative processes. unfortunately, most learning management systems are not currently set up to allow lecturers to carry out these operations. integration of a series of complex functions across an institution requires a strategic approach linking a range of policies, procedures, roles and responsibilities. according to mark stiles, 'it includes such things as staff development as well as quality assurance, procedures for the validation and approval of new courses, academic monitoring, and the various procedures associated with the recruitment, enrolment, charging, assessment and accreditation of students' (sself, stiles briefing paper, 2003). stiles presented a case study of how this had been attempted at the university of staffordshire, highlighting the ways in which interoperability standards had helped and hindered the process. while metadata was vital to operating a policy of reusing learning objects, the standards also proved to be restrictive. for example, when relocating a course to a new system, ims content packaging allows the content to be packaged with a simple linear structure and transported. however, it does not preserve the more complex structure nor the educational intent (wilson and currier, 2002). therefore it is essential that teachers and learners are involved in the standards debate to help develop standards that will underpin future systems that meet academics' and students' requirements (sself, corley briefing paper, 2003). forum responses several delegates were concerned that the current focus on reusing content has distracted lecturers from course design. as a result academic staff often begin designing courses by developing content, afterwards tagging learning design around the available content resources. consequently, course designs tend to overemphasize the use of information resources, rather than learner engagement in activities. in general, the smaller or more granular a resource the greater the possibility that it will be reused in another educational context. however, larger resources may have greater educational value and may be less time97 allison utdejohn supporting sustainable e-leaming consuming for a lecturer to reuse a larger resource, such as a learning activity, rather than constructing a course from its most basic components. therefore, in terms of resource size, delegates identified a conflict between the educational value and the ease of interoperability. academics often have a clear idea about the content they want to use, but are most likely to require guidance and support in reusing course designs or activities. it is important that teaching staff are encouraged to design courses and develop student activities before deciding on the content resources. however, a significant number of support staff have limited understanding of pedagogical issues and might also tend to focus on content development. therefore, in order to shift the focus from content to course design issues, it is critical that support units work cooperatively. this is most likely to happen when support staff are assigned clear roles and feel secure while working within multi-disciplinary teams. delegates reported that academics often confuse technological and educational issues. this arises from the fact that current learning management systems do not allow easy development of courses based upon diverse educational models. most systems adopt specific pedagogical approaches that tend to be fixed. this only allows any one lms to be used within a limited range of contexts: use outwith this context usually requires a great deal of imagination and skill by the academic. learning management systems are equally uncompromising in allowing students actively to negotiate assessments, to set up online discussions, or to develop and upload their own learning resources. therefore, it is important that future learning management systems should allow more flexible development of courses based on a wide variety of educational models. a prototype learning activity management system (lams) is currently under development (dalziel, 2002). while lams is not yet based on ims learning design, it does use many of its elements. a surprisingly large proportion of delegates were unaware of developments in learning design. this reflects the gap that exists between educational 'research' and 'practice' cultures. many (though not all) of the support staff who are involved in current debates over the development of e-learning standards are involved in research. however, delegates perceived a mismatch between the ideas of the 'standards development community' and the practice of staff dealing with day-to-day learning and teaching issues. it is essential that the ideas being formulated by this research are fully tested in real situations if possible (with large numbers of students and within realistic timescales) and that the 'research' and 'practice' communities communicate more effectively. commentary this forum has highlighted major challenges that concern the development of a unified elearning strategy. many of these challenges are social and organizational issues, rather than pedagogical or technological concerns. some are related to the fast rate of change elearning has experienced in recent years. this rapid change has led to anxiety for support staff, as well as tensions-developing between support units, as mutual areas of engagement begin to overlap. if institutions are to take more responsibility for e-learning planning and provision (dfes, 2003a, para 59), the cooperative functioning of these units is an area deserving significant consideration. second, a related tension is the synergy between learning technology support staff and academic staff and the issue of deficiencies in communication between these groups. this 98 alt-] volume 11 number 3 issue is partly being resolved by the move away from a 'support' to a 'partnership' culture. these partnerships have, in some cases, extended to include work with students developing their own learning resources. these kinds of collaborations are likely to become increasingly important. third, there is a growing gap between institutional practice and research. this is likely to result in the benefits arising from research not being implemented at a local level. this problem is part of a wider issue in which national initiatives are not embedded within local institutions. national bodies, such as the learning and teaching support network, or the academy for the advancement of learning and teaching in higher education, could help forge stronger links. fourth, the continued use of imprecise terms without clear definition adds unnecessary intricacy to an already complex situation. the inconsistencies in, terminology amongst elearning support staff will confuse the academics they support. finally, viewing e-leaming as being somehow separate from other kinds of learning is not helpful when trying to devise new pedagogical designs (dfes, 2003a, para 56). it would be better to view e-learning as mainstream to learning and teaching practice, rather than as an add-on. this may help to disentangle the complexity of issues associated with different modes of teaching (distance learning, campus-based learning, continuing professional development, etc.). in summary, the development of a unified strategy to promote sustainable approaches to elearning requires major shifts in support and leadership. some of the greater challenges pertain to cultural and social issues, rather than technological concerns. these will be further explored in the next set of sself forum sessions which are due to take place early in 2004. acknowledgements the ideas within this paper have been amalgamated from a considerable body of work by the key contributors: gerry graham (lt scotland), lisa corley (cetis), dr colin tattersall (ounl), professor tom boyle (london metropolitan university), professor mark stiles (university of staffordshire), dr barbara newland (university of durham), professor robin mason, professor mary thorpe, dr patrick mcandrew (all from the open university, uk), dr martin oliver (university college london), professor grainne conole (university of southampton) and ros o'leary (university of bristol). i am indebted to them for their contributions and active involvement within the forum. thanks also to my colleagues at the ltsn generic centre for their support: kathy wiles, gill harrison and annita hirons. not least, i would like to thank the forum members, all of whom freely shared their thoughts and ideas. references beetham, h. (2001), 'career development of learning technology staff', jisc scoping study, http://www.sh.plym.ac.uk/eds/effects/ boyle, t. (2002), 'design principles for authoring dynamic, reusable learning objects', proceedings of ascilite 2002. http://www.unitec.ac.nz/ascilite/proceedings/papers/028.pdf 99 allison utdejohn supporting sustainable e-leaming britain, s. and liber, o. (2000), 'a summary of "a framework for pedagogical evaluation of virtual learning environments'", jtap report, http://www.jisc.ac.uk/jtap/htm/ jtap-041.html campbell, l. m. (2003), 'developing a learning object economy', chapter 4 in a. littlejohn (ed.), reusing online resources: a sustainable approach to e-learning, london: kogan page, 46-59. collis, b. and moonen, j. (2001), flexible learning in a digital world: experiences and expectations, london: kogan page. crook, c. (2002), 'the campus experience of networked learning', in c. steeples and c. jones (eds), networked learning: perspectives and issues, london: springer-verlag. crook, c. k. and barrowcliff, d. (2001), 'ubiquitous computing on campus: patterns of engagement by university students', international journal of human computer interaction, 13 (2), 245-58. dalziel, j., (2002), implementing learning design: the learning activity management system (lams), proceedings of ascilite 2002, http://www.adelaide.edu.au/ascilite2003/ docs/ws13.pdf department for education and skills (dfes) (2003a), towards a unified e-learning strategy, consultation document, july 2003, http://www.dfes.gov.uk/consultations2/16/ department for education and skills (dfes) (2003b), the future of higher education, white paper, january 2003, http://www.dfes.gov.uk/highereducation/hestrategy/foreword.shtml department for education and skills (dfes) (2003c), widening participation in higher education, consultation document, april 2003, http://www.dfes.gov.uk/highereducation/docs/wideningparticipation.pdf draper, s. w., brown, m. i., henderson, f. p. and mcateer, e. (1996), 'integrative evaluation: an emerging role for classroom studies of cal', computers and education, 26, 17-32. ehrmann, s. (1998), 'studying, teaching, learning and technology: a toolkit from the flashlight programme', active learning, 9, http://www.ilt.ac.uk/public/cti/activelearning/ al9.html green, k. c. (1999), 'the continuing challenge of instructional integration and user support', the campus computing project, http:llwww.campuscomputing.net/summariesl/999l harris, r. and higgison, c. (2003), 'the reuse of resources across communities of practice', chapter 19 in a. littlejohn (ed.), reusing online resources: a sustainable approach to elearning, london: kogan page, 234-47. ingraham, b., watson, b., mcdowell, l., brockett, a. and fitzpatrick, s. (2002), 'evaluating and implementing learning environments: a united kingdom experience', educational technology review, 10, 2, http://www.aace.org/pubs/etr/issue3/ingraham.cfm j a c k s o n , r . h . (2002), ' d e f i n i n g e-learning different s h a d e s of ' o n l i n e ' . . . a definitional protocol', http://www.hkwebsym.org.hk/2002/jackson_quote.html 100 alt-] volume 11 number 3 kaplan-leiserson, e. (2000), 'learning circuits e-learning glossary', http://www.learning circuits.org/glossary.html#e koper, r. (2001), 'modelling units of study from a pedagogical perspective: the pedagogical meta-model behind eml', ims learning design working group, http://eml.ou.nl/introduction/docs/ped-metamodel.pdf koper, r. (2003), 'combining reusable learning resources and services with pedagogically purposeful units of learning', chapter 5 in a. littlejohn (ed.), reusing online resources: a sustainable approach to e-learning, london: kogan page, 46-59. littlejohn, a. h. and peacock, s. (2003), 'from pioneers to partners: the changing voices of staff developers', chapter 6 in j. seale (ed.), from individual enthusiasm to institutional implementation: a review of learning technology in higher education, lisse, netherlands: swets & zeitlinger. macfarlane, a. (1998), working party of the committee of scottish university principals (csup), teaching and learning in an expanding higher education system, lasswade: polton house press. mcnaught, c. (2002), 'views on staff development for networked learning', in c. steeples and c. jones (eds), networked learning: perspectives and issues, london: springer, 111-24. national committee of inquiry into higher education (ncihe) (1997), higher education in the learning society, http://www.leeds.ac.uk/educol/ncihe/ nicol, d. and coen, m. (2003), 'a model for evaluating the institutional costs and benefits of ict initiatives in teaching and learning in higher education', alt-j, 11 (2), 46-60. oliver, m. (2002), 'what do learning technologists do?', innovations in education and teaching international, 39 (4), 245-52. oliver, m. (2003), 'rethinking the reuse of electronic resources: contexts, power and information literacy', journal of interactive media in education, 1, special issue on reusing online resources, http://www-jime.open.ac.uk/2003/1/ oliver, r. and maclaughlin, c. (2003), 'pedagogical designs for sustainable and scalable online learning', chapter 8 in a. littlejohn (ed.), reusing online resources: a sustainable approach to e-learning, london: kogan page, 46-59. thorpe, m., kubiak, c. and thorpe, k. (2003), 'design for reuse and versioning', chapter 6 in a. littlejohn (ed.), reusing online resources: a sustainable approach to e-learning, london: kogan page, 106-18. timms, d., booth, s., crompton, p., klein, b, bangali, l. and schnuekel, b (1999), 'review of telematics based open and distance learning', european commission educational multimedia taskforce schema, http://www.schema.stir.ac.uk/deliverables/d6.1.pdf wiles, k. and littlejohn, a. (2003), 'supporting sustainable e-learning: a uk national forum', proceedings of the ascilite conference, adelaide, australia. wilson, s. and currier, s. (2002), 'what is ims content packaging?', cetis standards briefings series, cetis 2002, http://www.cetis.ac.uk/lib/media/cpbrief.pdf 101 allison uttlejohn supporting sustainable e-learning websites didet, digital libraries for distributed innovative design education and teamwork http://dmeml.ds.strath.ac.uk/didet/ leap, learning^ environments and pedagogy http: i i www. itsn. ac uklgenericcentrelindex. asp?id=l 7729 rdn, resource discovery network http:llwww.jisc.ac.uklindex.cfm?name=services_rdn sself, supporting sustainable e-learning forum http: 11 www. itsn. ac. uklgenericcentrelindex. asp?id=18429 x4l, exchange for learning programme http:llwww.jisc.ac.uklindex.cfm?name=programme_x4l 102 doi:10.3402/rlt.v23.27197 using lecturetools to enhance student�instructor relations and student engagement in the large class jerie shaw, sofiya kominko and jenepher lennox terrion* department of communication, university of ottawa, ottawa, on, canada (received 7 january 2015; final version received 20 september 2015) positive student�instructor relationships are important for student engagement, motivation, retention and achievement. yet, as class sizes grow, these relationships can be increasingly difficult to develop. this study explores lecturetools � a webbased student response and learning platform that facilitates communication between instructors and students � as a possible solution to this issue by analysing survey data collected from students in a second-year communication class at a large canadian university. this study builds on previous evidence that using lecturetools results in an increase in student engagement, attentiveness and level of learning, while expanding on this work to include the concept of student instructor relationships. ultimately, the functionality of lecturetools was found to facilitate the development of student�instructor relationships in the large class while also enhancing student engagement. keywords: pedagogical design; undergraduate; mixed method; e-learning; technology the prevalence of large class sizes in the current post-secondary landscape presents a challenge for educators. of particular concern is the barrier that large classes present to the development of positive student�instructor relationships. these relationships have been found in numerous studies to contribute to student retention (evidence provided in jaasma and koper 1999) and improved student learning (evidence provided in tinto 1993), among other positive outcomes. given that profit-driven institutions are unlikely to eliminate large classes (australian universities teaching committee 2003), it is of great importance to understand how positive and supportive student�instructor relationships can be developed in this environment. lecturetools � a web-based student response and learning platform that facilitates communication between instructors and students � may offer a solution to this challenge. by harnessing the power of the personal laptop, lecturetools has the potential to enhance the student�instructor relationship through increased communication with the professor. the purpose of this article is to confirm existing results about the potential of lecturetools to produce positive effects in the post-secondary class, while also extending these results to consider the impact this tool may have on student�instructor relationships. *corresponding author. email: jlennoxt@uottawa.ca responsible editor: meg o’reilly, southern cross university, australia. research in learning technology vol. 23, 2015 research in learning technology 2015. # 2015 j. shaw et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 student�instructor relationships positive student�instructor relationships have been a topic of interest for communication and pedagogical researchers alike. according to frisby and martin (2010), researchers have found that instructors are better able to promote student learning if they foster positive interpersonal relationships with students. engaging in communicative behaviours designed to create these positive relationships generates affect, which in turn ‘enhances cognitive learning’ (frisby and martin 2010, p. 146). indeed, positive student�instructor relations have been found to result in increased student retention (evidence presented in jaasma and koper 1999), improved student learning (evidence presented in sher 2009; tinto 1993), accelerated academic achievement (battistich et al. 1997; eccles 2004; teven 2001) and increased feelings of confidence and self-directedness (ames 1992; midgley, feldlaufer, and eccles 1989; pintrich, roeser, and de groot 1994; ryan, gheen, and midgley 1998). while some researchers explicitly talk about student�instructor relationships, others use the construct of ‘rapport’ to make sense of the numerous positive outcomes associated with affect-generating communicative behaviours in the class. in order to understand what is meant by ‘positive student�instructor relationships’, it is useful to briefly explore the related concept of rapport. rapport is ‘an overall feeling between two people encompassing a mutual, trusting and prosocial bond’ (frisby and martin 2010, p. 147); the ability to generate rapport and the related feelings of trust and warmth has been found to be essential for excellent educators (catt, miller, and schallenkamp 2007; faranda and clark 2004; frisby and martin 2010). the precise relationship between the interrelated concepts of student� instructor relationships and rapport is not always clear in the literature and seems to be understood differently by different authors. suffice to say, our interest here is simply in exploring how the use of lecturetools in post-secondary classrooms may enhance positive student�instructor relationships (characterized by trust and rapport, as it is defined above). instructors can generate these positive relationships through interaction with students. positive student�instructor relationships are developed through communicative behaviours, often initiated by the professor (sher 2009). examples of behaviours which contribute to the development of these relationships include using a rapid feedback cycle (bergom et al. 2011), responding to questions (sher 2009), providing personalized feedback (sher 2009) and communicating with students outside of class (dobransky and frymier 2004). professors can also facilitate the development of student�instructor relationships through the environment they create in their class. for example, building a ‘collaborative and cooperative’ learning environment is thought to contribute to student�instructor relationship building (wurst, smarkola, and gaffney 2008, p. 1767), as it demonstrates that the professor cares about the input of students. similarly, in his study on student�instructor relationships, myers (2006) found that a professor’s attempt to demonstrate an understanding of students’ problems and needs contributes to positive closeness. difficulty of developing student�instructor relationships in the large class studies suggest that student�instructor relationships may be particularly difficult to develop in the large class, where lack of proximity and increased anonymity prevent the formation of meaningful connections (myers 2006). numerous factors � including j. shaw et al. 2 (page number not for citation purpose) citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 a growing perception of the necessity of post-secondary education and, in turn, a growing demand for increased access to colleges and universities � have put immense pressure on post-secondary institutions in canada and the united states (clark et al. 2009). these factors have led to undergraduate enrolment increases in degree-granting institutions of almost 50% over the past decade in ontario, canada (kerr 2011). in the u.s., enrolment increased by 37%, from 15.3 to 21.0 million between 2000 and 2010 (united states department of education 2012). one approach taken at both large research-intensive universities and small, primarily undergraduate schools is to increase class sizes. as kerr (2011) points out in her report of research done by the higher education quality council of ontario, about two thirds of ontario universities reported that at least 30% of first year courses had more than 100 students in 2009. this trend is unlikely to change. for profit-driven universities in particular, having a high student-to-instructor ratio is unavoidable (australian universities teaching committee 2003). given the prominence of large classes in today’s universities, understanding how these classes can be most effectively delivered is of significance to students, professors and administrators. this paper explores one way that student� instructor relationships can be facilitated in the context of the large, technologically advanced class. laptop use in the large class given the importance of student�instructor relationships and the barrier presented by large classes, it is important to explore whether technology can help facilitate the development of positive relationships. laptops are increasingly being used by students in post-secondary classrooms for note-taking, presentation viewing and online reference (cismaru and cismaru 2011). studies have demonstrated both negative and positive effects of laptop use in higher education. this discrepancy is perplexing, until one makes note of the variation in the nature of laptop use measured by these studies. according to kay and lauricella (2011) in their study of different types of laptop use in university classrooms, laptops can be used in either an unstructured or a structured way, with drastically different results. unstructured use of laptops occurs when students are permitted to use laptops in class, but the laptops are not incorporated into teaching practices in any meaningful way (samson 2010). with unstructured use, students may be using their laptops to take notes or for online reference, but the professor isn’t actively encouraging these � or any other � behaviours. in essence, the professor is lecturing as though the laptops do not exist, allowing students to use (or not use) laptops, as they wish. unstructured use of laptops is associated with a decrease in attentiveness and engagement in class (fried 2008; kraushaar and novak 2010; wurst et al. 2008), diminished performance on examinations (aguilar-roca, williams, and o’dowd 2012; fried 2008; grace-martin and gay 2001; sana, weston, and cepeda 2013), diminished classroom learning (ravizza, hambrick, and fenn 2014) and increased distraction in class (fried 2008; zhu et al. 2011). structured use, on the other hand, refers to ‘the use of technology in a deliberate and integrated manner [by professors] to affect learning goals’ (samson 2010, p. 2). structured use of laptops may be an effective way to encourage student engagement and attentiveness in university classrooms, as well as facilitate student�instructor relationship development. among other things, studies have found that structured research in learning technology citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 use of laptops in class can increase student engagement (zhu et al. 2011), improve academic achievement (kolar, sabatini, and fink 2002) and decrease off-task behaviour in class (kay and lauricella 2011). structured use of laptops can encompass myriad different tools, ranging from educational software, practical simulations, virtual assignments, videos, web pages and course-related tools such as lecturetools (kay and lauricella 2011, p. 34). lecturetools is a full suite of online tools, including auto-sync presentations, interactive polls and tests, and the ability for students to digitally ask questions, take notes on course material and flag concepts as confusing. lecturetools was developed by dr. perry samson at the university of michigan as part of a 2005 research project, and was later developed into a commercial product. the cloud-based suite of tools is licensed to higher learning institutions and, in turn, made available free for professors and students to use. the university of ottawa, where this study was conducted, purchased a volume license for lecturetools in 2013 and provides training and support to professors who wish to use the tool. several studies have specifically compared classes with unstructured laptop use (control group) to classes using lecturetools (experimental group), in order to gauge the effectiveness of the service (see bergom et al. 2011; samson 2010). for example, both bergom et al. (2011) and zhu et al. (2011) found that lecturetools was significantly correlated with an increase in attentiveness, engagement and level of learning compared to the control group (p. 23). in a similar study, samson (2010) found that students in the experimental group reported ‘a dramatic increase in engagement with the use of technology’ (p. 7) relative to the control group. these studies have led samson (2010) to conclude that ‘the benefits of lecturetools on student attentiveness and engagement (and self-reported learning) overcomes[sic] the potential risk from increased distraction’ (p. 7). lecturetools as a facilitator of student�instructor relationships in addition to the positive impact on engagement, learning and satisfaction associated with structured laptop use, there is ample evidence that laptops ‘can increase constructive discourse between students and between students and instructors’ (samson 2010, p. 2). lecturetools may specifically facilitate positive student�instructor relationships in several ways, as seen in table 1 and elaborated upon thereafter. in sum, lecturetools may facilitate student�instructor relationships in three main ways. first, through building a ‘collaborative and cooperative’ learning environment (wurst et al. 2008, p. 1767), lecturetools contributes to relationship building. as discussed, this behaviour enhances the development of student�instructor relationships by giving students a degree of control over their own learning and the direction of course content (wurst et al. 2008). in other words, the professor demonstrates concern for students by adapting the course content and direction to suit their needs. lecturetools is well situated to facilitate this behaviour. through polling � both multiple choice and short answer � professors can gather ideas from students and incorporate them into the lecture. professors may also poll students on the direction of course content, the preferred method of content delivery, or any other aspect of the course design or content. second, lecturetools may facilitate student�instructor relationships by enabling a rapid and personalized feedback cycle. through instant polling and flagging slides as confusing, students can provide their opinion to the professor in such a way that j. shaw et al. 4 (page number not for citation purpose) citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 enables the professor to understand students’ problems and needs. professors can then respond verbally, by text, or by adapting their behaviour in the class, thus completing the feedback cycle. the ‘ask a question’ function is a particularly powerful way of developing the student�instructor relationship. students can ask questions at any point in the lecture, and the professor is immediately notified. the professor (or teaching assistant) can respond immediately by text, or verbally in class. they may also choose to respond at a later time, such as after lecture. when students are logged into lecturetools, they are immediately notified when the professor responds to their question. in addition, the professor has easy access to students’ names and a complete record of their achievement on lecturetools. this includes data such as the previous questions that the student has asked, content that the student has flagged as confusing in the past and a record of his or her attendance. this easy access to data enables the professor to write a complete and personalized response to questions received. for all of these reasons, lecturetools has potential for facilitating the development of positive student�instructor relationships in large post-secondary classrooms where it might otherwise be exceedingly difficult without the use of technology. to examine this possible impact, lecturetools was made mandatory in a second-year communication course at a large canadian university for the duration of one semester. at the end of the semester, surveys were submitted to each student asking them to evaluate their experience using the tool, particularly as it related to table 1. behaviours which develop student�instructor relationships and associated lecturetools functions. behaviour which develops student� instructor relationships lecturetools function building a ‘collaborative and cooperative’ learning environment (wurst et al. 2008) instant polling, which enables the professor to incorporate student ideas into lectures, and adapt lectures to meet the real-time needs of students. using a rapid feedback cycle (bergom et al. 2011) instant polling, flagging slides as confusing and the ‘ask a question’ feature all facilitate a rapid feedback cycle in the class. understanding students’ problems and needs (myers 2006) flagging slides as confusing and the ‘ask a question’ function enable the professor to know (almost instantaneously) when students are experiencing a problem. during downtime in a lecture, or after class, professors may review the data and follow up individually with students who flagged slides as confusing or asked questions. responding to questions one-on-one (sher 2009) the ‘ask a question’ function gives the student an ‘instant line’ to ask the professor a question during class. providing personalized feedback (sher 2009) the professor has access to students’ names and can easily access their complete record of achievement on lecturetools. out of class communication (dobransky and frymier 2004; sheer and fung 2007) the ‘ask a question’ function allows students to ask questions during class, which can then be followed up by the professor outside of class. research in learning technology citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 the interconnected concepts of student satisfaction, student engagement and student�instructor relationships. method sample undergraduate students at a public canadian university completed a questionnaire during a regularly scheduled class period using lecturetools. a total of 92 surveys were submitted. the sample comprised students from a second-year organizational communication class (enrolment � 119). for privacy reasons, no demographic information such as participant age, gender, or program of study was collected. prior to completing the questionnaire, students were informed about the purpose of the study. the data collection period ran for roughly 15 minutes of class time during the last regularly scheduled lecture of the 12-week semester. students provided informed consent via lecturetools. in the event that a student did not respond to the informed consent question (or responded in the negative), that particular student’s data were removed prior to analysis. in several cases, students responded to all the questionnaire questions but neglected to respond to the informed consent question. in part, this accounts for the discrepancy in class enrolment (n �119) and the number of responses received (n �92). instrumentation to measure levels of student engagement and perception of the student�instructor relationship, students were required to respond to 10 statements using a five-point likert-type scale (1 � strongly disagree and 5 � strongly agree). statements targeting the topic of engagement in the large class asked participants to comment on their level of non-course related internet browsing and overall perception of feeling engaged as a result of lecturetools (e.g. ‘as a result of using lecturetools, i felt more engaged and involved in the lecture’). similarly, students were also asked about their perception of the student�instructor relationship, with questions such as ‘by using lecturetools, i felt more inclined to ask the professor or ta a question’, and ‘when using lecturetools, i felt like i could develop a more personal relationship with the prof and/or ta’. finally, students were asked to comment on the technical aspects of lecturetools by noting their level of frustration with the program as well as the use of all the tools offered. the goal here was to determine whether technical aspects such as the availability of tools or frustrations associated with the technology may have an impact on the relationship-building potential of lecturetools. a complete list of the exact survey questions appears in appendix 1. following these likert-type questions, students were presented with four openended questions. these questions allowed the students to make any additional comments about lecturetools in respect to the three categories � student engagement, student�instructor relationships and functions of the program. results and analysis frequency tables were generated in order to summarize the data received. a correlation analysis was conducted � the results have been presented in a correlation matrix table in j. shaw et al. 6 (page number not for citation purpose) citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 appendix 2. summative scales were also created by combining the variables within the two main themes (engagement and relationship building). a thematic coding process was applied to the qualitative data using the three themes � student engagement, student�instructor relationships and functions of the program � as a guide. results frequencies over 70% of responses to questions related to student engagement indicated either agreement (41%) or strong agreement (31%) that lecturetools facilitated increased engagement in-class. significantly, over 90% of respondents either agreed (43.5%) or strongly agreed (46.7%) with the statement ‘as a result of using lecturetools, i felt more engaged and involved in the lecture’. interestingly, approximately 57% of respondents said that using lecturetools caused them to browse the internet less often (compared to when they use their laptop in courses in an unstructured manner). approximately 60% of responses to questions related to student�instructor relationships indicated agreement that lecturetools facilitated the development of a positive student�instructor relationship. significantly, 60% of respondents either agreed (43.5%) or strongly agreed (16.3%) with the statement ‘when using lecturetools, i felt like i could develop a more personal relationship with the prof and/or ta’, with 24% neither agreeing nor disagreeing. finally, within the theme of lecturetools functionality, only 3% of respondents agreed or strongly agreed that lecturetools was frustrating to use. the frequency of responses for each question appears in figure 1. correlations three moderate, statistically significant relationships were found in the data, supporting the conclusion that student engagement and the development of student� instructor relationships are interconnected. first, there was a moderate, statistically significant correlation between responses to the question ‘by using lecturetools, 100% 95% 90% 80% 85% 75% 65% 60% 45% 35% 25% 20% 15% 10% 5% 0% rel1 rel2 rel3tec1 tec2 eng1 eng2 eng3 eng4 no data strongly agree agree neither agree nor disagree disagree strongly disagree rel4 30% 55% 50% 40% 70% figure 1. frequency of responses. research in learning technology citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 i felt more inclined to ask the professor or ta a question’, and responses to the question ‘as a result of using lt, i felt more engaged and involved in the lecture’. second, there was a moderate, statistically significant relationship between responses to the question ‘by asking the professor or the ta questions via lecturetools, i felt that i was contributing more to class discussion’ and responses to the question ‘as a result of using lecturetools, i felt more engaged in lecture’. both of these correlations suggest that students who used lecturetools to ask questions in class and interact one-on-one with the professor felt more engaged and involved in lecture. finally, there was a moderate, statistically significant relationship between responses to the question ‘when using lecturetools, i felt like i could develop a more personal relationship with the prof and/or ta’ and responses to the question ‘as a result of using lecturetools, i felt more engaged and involved in the lecture’. this suggests that students who felt that lecturetools was successful at facilitating student�instructor relationships also felt that the tool increased their level of engagement in class. analysis the purpose of this study was to explore the value of lecturetools in the large post-secondary class through the lens of student�instructor relationships. the theme of student engagement was also explored, as the literature suggested that the two constructs (student�instructor relationships and level of engagement) might have some bearing on one another. this is unsurprising given that the types of behaviours that enhance student�instructor relationships, such as responding to questions in class (sher 2009) and building a collaborative learning environment (wurst et al. 2008), would be expected to also have a positive bearing on student engagement. the data supports the conclusion that using lecturetools has a positive overall impact on student engagement and student�instructor relationship building. first, the theme of engagement will be explored. over 90% of respondents either agreed (43.5%) or strongly agreed (46.7%) with the statement ‘as a result of using lecturetools, i felt more engaged and involved in the lecture’. in response to the survey’s open-ended questions, students noted the value of the question feature of lecturetools � that is the ability to answer polls, respond to long answer questions and ask the professor questions in real time. one participant noted that ‘when i knew there was a lecturetools question coming up, i paid more attention and thought about the answer’. many students had similar responses, and mentioned that the most effective aspect of responding to long answer questions is that it increased their level of attention throughout the lecture. in other words, knowing they would have to respond to a question and demonstrate their understanding of the material required students to be more engaged and attentive. another participant commented on the effectiveness of multiple-choice polls in creating a more collaborative classroom environment, saying: polls and participation activities allowed me to understand how much of the class that doesn’t normally talk thinks. it would allow me to think of the concepts being taught in different ways, seeing how many others thought differently � trying to think of things from their point of view. j. shaw et al. 8 (page number not for citation purpose) citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 in addition to multiple-choice polling, students also commented on the ‘ask a question’ feature, noting its positive effect on attentiveness and participation. students found this feature useful because it allowed them to ask questions without ‘interrupting the whole class’ or ‘having to possibly humiliate [themselves] in front of the class’. these findings are consistent with the literature on lecturetools’ impact on student engagement. as noted, both bergom et al. (2011) and zhu et al. (2011) found that lecturetools was significantly correlated with an increase in attentiveness and engagement, relative to a control group (p. 23). this study has uniquely explored the particular features � such as the ‘ask a question’ function and multiple-choice polling � which students say resulted in this increased engagement. the next theme we explored � the focus of this study � was the effect of lecturetools on the formation of positive student�instructor relationships. approximately 60% of responses to questions related to student�instructor relationships indicated agreement that lecturetools facilitated the development of a positive student�instructor relationship, with 42% agreeing and 18% strongly agreeing. one student mentioned that lecturetools ‘created a sense of relationship with the ta and prof ’ while another noted that it ‘created a more hands-on ‘‘relationship’’ with the professor/ta’. one of the most prevalent themes that emerged from the qualitative data was that lecturetools facilitated a perceived one-on-one connection between the professor and students who would normally be too shy to develop such a connection, particularly in the large class. one student noted ‘it just made me more comfortable being able to speak to her and ask her questions’, while another said, ‘i’m usually shy in asking questions, but i was basically able to ask at least one every class’. this is consistent with sher’s (2009) finding that responding to questions one-on-one in the classroom improves student�instructor relationships. even students who didn’t note ‘shyness’ as a particular issue reported feeling more connected with the professor (and teaching assistant) than they normally would in a large class. one participant noted that, ‘especially in a large lecture, it is sometimes difficult to communicate with the professor but lecturetools made the professor more accessible’, while another said, ‘i felt like i had a closer line of communication with the ta/prof ’. this connectedness to the instructor seemed to extend beyond just simply being able to seek clarification. students described the connection being formed between student and professor as ‘personal’ and ‘close’. by having a direct, personal channel between the instructor and student through which communication is both synchronous and asynchronous, the instructor becomes more accessible. through such accessibility, student�instructor relationship development, even in the large class, becomes feasible. two other interesting themes emerged from the qualitative data around student� instructor relationships. several students noted that the opinion polling function (multiple choice and short answer questions asked in class) demonstrated that the professor ‘cared’ about the opinion of students. one student noted ‘i felt i could easily interact with the professor . . . by submitting my opinion’, while another said, ‘i think the occasional question allowed for better communication because it created feedback as a norm’. finally, one student noted that lecturetools facilitated the development of closer relationships between students in the large class: ‘it is really interesting to see how classmates and peers respond to questions as well. it puts a sort of positive relationship between us, creating a feeling of unity in the class’. it’s possible that in-class polling contributed to what wurst et al. (2008) call a research in learning technology citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 ‘collaborative and cooperative’ learning environment (p. 1767), which has been found to contribute to positive student�instructor relationships. finally, this study explored whether increased frustration with using the tool resulted in diminished returns of student engagement and student�instructor relationships. no such negative correlations were found. it would appear that the frustrations associated with adopting and learning a new technology, while present, did not prevent students from reaping the positive rewards associated with lecturetools use. in fact, in their open-ended responses, several students noted both technical concerns (e.g., that the ipad app needed to be improved), while also noting the excellent benefits of lecturetools on their level of engagement and connection to the professor. this is a promising finding for any professor who is concerned that technology frustrations may diminish the positive impact of using digital tools in the large class. while these findings were certainly promising, we did note some student concerns with the tool. approximately eight students expressed either neutrality (‘it didn’t really add or subtract anything’) or dissatisfaction with elements of the tool. those who expressed frustration or dissatisfaction noted a wide variety of issues, with no two students mentioning the same concern. these concerns included: technical issues with the ipad app; the lack of certain functionality that the student deemed to be important (in this case, the ability to copy and paste text from a slide); and finally, the student’s personal preference to record notes by hand. limitations the results of this study are based on data collected from one undergraduate course. the student experience in this course may be affected by other variables (such as the course material, the professor, or room), which may have influenced students’ overall positivity or negativity in responses. it would be of great interest and value to repeat the data collection with additional � and varied � groups of students. another limitation was created by the very polling technology this study evaluated. lecturetools permits multiple student responses to a single multiple-choice polling question. as a result, numerous students provided multiple responses to single questions (e.g., indicating both ‘agree’ and ‘strongly agree’ to a statement about student engagement). in these cases, responses to those questions had to be omitted from data analysis. conclusion this study confirms the results of previous studies, such as bergom et al. (2011), which found that lecturetools results in an increase in student engagement. uniquely, however, this study found that lecturetools also facilitates the development of positive student�instructor relationships in the large post-secondary class. students expressed overwhelming agreement that the tool increased their feelings of engagement with the lecture and facilitated a connection with the professor. this connection was described as ‘close’ and ‘personal’ suggesting a supportive relationship (myers 2006); such supportive student�teacher relationships have been found to result in numerous positive benefits for student learning, satisfaction, motivation and retention. it must be noted that instructors need to learn, and indeed master, technology in the classroom so that time is not wasted in setting it up or managing it, so that it is used productively and not trivially and so that students remain interested in its use (nielsen, hansen, j. shaw et al. 10 (page number not for citation purpose) citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 and stav 2013). nevertheless, large classes pose a barrier to the healthy development of positive student�instructor relationships. it is promising to find that technologies such as lecturetools have the potential to mitigate � perhaps even eliminate � the negative effects of large class size in this regard. references aguilar-roca, n. m., williams, a. e. & o’dowd, d. k. (2012) ‘the impact of laptop-free zones on student performance and attitudes in large lectures’, computers & education, vol. 59, no. 4, pp. 1300�1309. doi: http://dx.doi.org/10.1016/j.compedu.2012.05.002 ames, c. (1992) ‘classrooms: goals, structures and student motivation’, journal of educational psychology, vol. 84, no. 3, pp. 261�271. australian universities teaching committee. (2003) teaching large classes project 2001 final report. teaching and educational development institute, [online]. available at: www. cadad.edu.au/largeclasses/pdfs/autc_final_report.pdf battistich, v., et al., (1997) ‘caring school communities’, educational psychologist, vol. 32, pp. 137�151. doi: http://dx.doi.org/10.1207/s15326985ep3203_1 bergom, i., et al., (2011) ‘using laptops effectively in classrooms: lessons from a study of lecturetools’, centre for research on learning and teaching, [online]. available at: http://www.slideshare.net/lecturetools/university-of-michigan-crlt-study-of-lecturetoolsand-laptop-use-in-class catt, s., miller, d. & schallenkamp, k. (2007) ‘you are the key: communicate for learning effectiveness’, education, vol. 127, no. 3, pp. 369�377. cismaru, r. & cismaru, m. (2011) ‘laptop use during class: a review of canadian universities’, journal of college teaching and learning, vol. 8, no. 11, pp. 21�28. clark, i. d., et al., (2009) academic transformation: the forces reshaping higher education in ontario, mcgill-queen’s university press, kingston, on. dobransky, n. d. & frymier, a. b. (2004) ‘developing teacher�student relationships through out-of-class communication’, communication quarterly, vol. 52, pp. 211�223. eccles, j. (2004) ‘schools, academic motivation, and stage-environment fit’, in handbook of adolescent psychology. 2nd edn, eds. r. lerner & l. steinberg, wiley, new york, pp. 125�153. faranda, w. t. & clarke, i. (2004) ‘student observation of outstanding teaching: implications for marketing educators’, journal of marketing education, vol. 26, pp. 271�281. fried, c. b. (2008) ‘in-class laptop use and its effects on student learning’, computers & education, vol. 50, pp. 906�914. doi: http://dx.doi.org/10.1016/j.compedu.2006.09.006 frisby, b. n. & martin, m. m. (2010) ‘instructor�student and student�student rapport in the classroom’, communication education, vol. 59, no. 2, pp. 146�164. grace-martin, m. & gay, g. (2001) ‘web browsing, mobile computing, and academic performance’, education technology and society, vol. 4, no. 3, pp. 95�107. jaasma, m. a. & koper, r. j. (1999) ‘the relationship of student�faculty out-of-class communication to instructor immediacy and trust and to student motivation’, communication education, vol. 48, pp. 41�47. kay, r. h. & lauricella, s. (2011) ‘unstructured vs. structured use of laptops in higher education’, journal of information technology education: innovations in practice, vol. 10, pp. 33�42. kerr, a. (2011) ‘teaching and learning in large classes at ontario universities: an exploratory study’, higher education quality council of ontario, [online]. available at: http://www. heqco.ca/sitecollectiondocuments/teaching and learning in large classes eng.pdf kolar, r. l., sabatini, d. a. & fink, l. d. (2002) ‘laptops in the classroom: do they make a difference?’, journal of engineering education, vol. 91, no. 4, pp. 397�401. doi: http://dx. doi.org/10.1002/j.2168-9830.2002.tb00723.x kraushaar, j. m. & novak, d. c. (2010) ‘examining the effects of student multitasking with laptops during the lecture’, journal of information systems education, vol. 21, no. 2, pp. 241�251. midgley, c., feldlaufer, h. & eccles, j. (1989) ‘change in teacher efficacy and student selfand task-related beliefs in mathematics during the transition to junior high school’, journal of research in learning technology citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 11 (page number not for citation purpose) http://dx.doi.org/10.1016/j.compedu.2012.05.002 http://www.cadad.edu.au/largeclasses/pdfs/autc_final_report.pdf http://www.cadad.edu.au/largeclasses/pdfs/autc_final_report.pdf http://dx.doi.org/10.1207/s15326985ep3203_1 http://www.slideshare.net/lecturetools/university-of-michigan-crlt-study-of-lecturetools-and-laptop-use-in-class http://www.slideshare.net/lecturetools/university-of-michigan-crlt-study-of-lecturetools-and-laptop-use-in-class http://dx.doi.org/10.1016/j.compedu.2006.09.006 http://www.heqco.ca/sitecollectiondocuments/teaching%20and%20learning%20in%20large%20classes%20eng.pdf http://www.heqco.ca/sitecollectiondocuments/teaching%20and%20learning%20in%20large%20classes%20eng.pdf http://dx.doi.org/10.1002/j.2168-9830.2002.tb00723.x http://dx.doi.org/10.1002/j.2168-9830.2002.tb00723.x http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 educational psychology, vol. 81, no. 2, pp. 247�258. doi: http://dx.doi.org/10.1037/00220663.81.2.247 myers, s. (2006) ‘using leader-member exchange theory to explain students’ motives to communicate’, communication quarterly, vol. 54, no. 3, pp. 293�304. doi: http://dx.doi.org/ 10.1080/01463370600878008 nielsen, k. l., hansen, g. & stav, j. b. (2013) ‘teaching with student response systems (srs): teacher-centric aspects that can negatively affect students’ experience of using srs’, research in learning technology, vol. 21, [online]. doi: http://dx.doi.org/10.3402/rlt.v21i0. 18989. available at: http://www.researchinlearningtechnology.net/index.php/rlt/article/ view/18989 pintrich, p. r., roeser, r. & de groot, e. (1994) ‘classroom and individual differences in early adolescents’ motivation and self-regulated learning’, journal of early adolescence, vol. 14, pp. 139�161. doi: http://dx.doi.org/10.1177/027243169401400204 ravizza, s. m., hambrick, d. z. & fenn, k. m. (2014) ‘non-academic internet use in the classroom is negatively related to classroom learning regardless of intellectual ability’, computers & education, vol. 78, pp. 109�114. doi: http://dx.doi.org/10.1016/j.compedu. 2014.05.007 ryan, a. m., gheen, m. h. & midgley, c. (1998) ‘why do some students avoid asking for help? an examination of the interplay among students’ academic efficacy, teachers’ social�emotional role, and the classroom goal structure’, journal of educational psychology, vol. 90, pp. 528�535. samson, p. j. (2010) ‘deliberate engagement of laptops in large lecture classes to improve attentiveness and engagement’, computers in education, vol. 1, no. 2, pp. 1�19. sana, f., weston, t. & cepeda, n. j. (2013) ‘laptop multitasking hinders classroom learning for both users and nearby peers’, computers & education, vol. 62, pp. 24�31. doi: http://dx. doi.org/10.1016/j.compedu.2012.10.003 sheer, v. & fung, t. k. (2007) ‘can e-mail communication enhance professor�student relationship and student evaluation of professor? some empirical evidence’, journal of educational computing research, vol. 37, no. 3, pp. 289�306. doi: http://dx.doi.org/10. 2190/ec.37.3.d sher, a. (2009) ‘assessing the relationship of student�instructor and student�student interaction to student learning and satisfaction in web-based online learning environment’, journal of interactive online learning, vol. 8, pp. 102�120. teven, j. (2001) ‘the relationships among teacher characteristics and perceived caring’, communication education, vol. 50, pp. 159�169. tinto, v. (1993) leaving college: rethinking the causes and cures of student attrition. 2nd ed, university of chicago press, chicago, il. united states department of education. (2012) the condition of education 2012, national center for education statistics, [online]. available at: http://nces.ed.gov/pubs2012/ 2012045.pdf wurst, c., smarkola, c. & gaffney, m. a. (2008) ‘ubiquitous laptop usage in higher education: effects on student achievement, student satisfaction and constructivist measures in honors and traditional classrooms’, computers & education, vol. 51, pp. 1766�1783. doi: http://dx.doi.org/10.1016/j.compedu.2008.05.006 zhu, e., et al., (2011) ‘use of laptops in the classroom: research and best practices’, centre for research on learning and teaching, vol. 30, pp. 1�6. j. shaw et al. 12 (page number not for citation purpose) citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 http://dx.doi.org/10.1037/0022-0663.81.2.247 http://dx.doi.org/10.1037/0022-0663.81.2.247 http://dx.doi.org/10.1080/01463370600878008 http://dx.doi.org/10.1080/01463370600878008 http://dx.doi.org/10.3402/rlt.v21i0.18989 http://dx.doi.org/10.3402/rlt.v21i0.18989 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/18989 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/18989 http://dx.doi.org/10.1177/027243169401400204 http://dx.doi.org/10.1016/j.compedu.2014.05.007 http://dx.doi.org/10.1016/j.compedu.2014.05.007 http://dx.doi.org/10.1016/j.compedu.2012.10.003 http://dx.doi.org/10.1016/j.compedu.2012.10.003 http://dx.doi.org/10.2190/ec.37.3.d http://dx.doi.org/10.2190/ec.37.3.d http://nces.ed.gov/pubs2012/2012045.pdf http://nces.ed.gov/pubs2012/2012045.pdf http://dx.doi.org/10.1016/j.compedu.2008.05.006 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/27197 http://dx.doi.org/10.3402/rlt.v23.27197 appendix 1. instrument questions q # variable code question 1 a rel1 responding to short answer questions using lecturetools enabled the professor to get to know me better. 2 b tec1 other than submitting my reaction paper, i did not use any other lecturetools features (such as flagging slides as confusing or asking questions). 3 rel2 when using lecturetools i felt like i could develop a more personal relationship with the professor and/or ta. 4 rel3 by using lecturetools, i felt more inclined to ask the professor or ta a question. 5 tec2 lecturetools was frustrating to use. 6 c eng1 responding to short answer questions using lecturetools encouraged me to remain attentive during the lecture. 7 eng2 flagging unclear material enabled the professor to respond to concepts that i might have not understood otherwise. 8 rel4 by asking the professor or ta questions through lecturetools, i felt that i was contributing more to class discussions. 9 eng3 when using lecturetools i was less inclined to browse the internet (compared to laptop use in other classes that do not use lecturetools). 10 eng4 as a result of using lecturetools i felt more engaged and involved in the lecture. a variables beginning in ‘rel’ relate directly to student�instructor relationships; bvariables beginning in ‘tec’ relate to functions of the lecturetools program; c variables beginning in ‘eng’ relate to student engagement. research in learning technology citation: research in learning technology 2015, 23: 27197 http://dx.doi.org/10.3402/rlt.v23.27197 13 (page number not for citation purpose) appendix 2. correlation matrix rel1 tec1 rel2 rel3 tec2 eng1 eng2 rel4 eng3 eng4 rel1 pearson correlation 1 �0.157 0.121 0.103 0.024 �0.040 �0.061 0.065 0.043 0.168 tec1 pearson correlation �0.157 1 �0.103 0.041 0.135 0.113 �0.034 �0.052 0.115 0.129 rel2 pearson correlation 0.121 �0.103 1 0.387a �0.029 0.150 0.167 0.211b 0.215b 0.405a rel3 pearson correlation 0.103 0.041 0.387 a 1 0.103 0.313 a 0.279 a 0.535 a 0.139 0.437 a tec2 pearson correlation 0.024 0.135 �0.029 0.103 1 0.126 0.119 �0.063 �0.005 0.075 eng1 pearson correlation �0.040 0.113 0.150 0.313a 0.126 1 0.429a 0.482a 0.617a eng2 pearson correlation �0.061 �0.034 0.167 0.279a 0.119 0.335a 1 0.419a 0.378a 0.309a rel4 pearson correlation 0.065 �0.052 0.211b 0.535a �0.063 0.429a 0.419a 1 0.270a 0.415a eng3 pearson correlation 0.043 0.115 0.215 b 0.139 �0.005 0.482a 0.378a 0.270a 1 0.506a eng4 pearson correlation 0.168 0.129 0.405 a 0.437 a 0.075 0.617 a 0.309 a 0.415 a 0.506 a 1 a correlation is significant at the 0.01 level (2-tailed); b correlation is significant at the 0.05 level (2-tailed). j. s h a w e t a l. 1 4 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) c ita tio n : r e se a rc h in l e a rn in g t e c h n o lo g y 2 0 1 5 , 2 3 : 2 7 1 9 7 h ttp ://d x .d o i.o rg /1 0 .3 4 0 2 /rlt.v 2 3 .2 7 1 9 7 doi:10.3402/rlt.v24.29366 original research article fostering postgraduate student engagement: online resources supporting self-directed learning in a diverse cohort luciane v. mello* school of life sciences, university of liverpool, liverpool, united kingdom (received 6 august 2015; final version received 20 february 2016) the research question for this study was: ‘can the provision of online resources help to engage and motivate students to become self-directed learners?’ this study presents the results of an action research project to answer this question for a postgraduate module at a research-intensive university in the united kingdom. the analysis of results from the study was conducted dividing the students according to their programme degree � masters or phd � and according to their language skills. the study indicated that the online resources embedded in the module were consistently used, and that the measures put in place to support self-directed learning (sdl) were both perceived and valued by the students, irrespective of their programme or native language. nevertheless, a difference was observed in how students viewed sdl: doctoral students seemed to prefer the approach and were more receptive to it than students pursuing their masters degree. some students reported that the sdl activity helped them to achieve more independence than did traditional approaches to teaching. students who engaged with the online resources were rewarded with higher marks and claimed that they were all the more motivated within the module. despite the different learning experiences of the diverse cohort, the study found that the blended nature of the course and its resources in support of sdl created a learning environment which positively affected student learning. keywords: self-directed learning; online resources; postgraduate teaching; higher education; prior knowledge to access the supplementary material to this article, please see supplementary files under ‘article tools’. introduction postgraduate cohorts are diversifying as a result of changes in the uk higher education, shifting priorities of universities and changing training requirements of students. students’ prior knowledge is a key element to their success, particularly in taught modules. the prior knowledge of a diverse cohort naturally varies considerably, so that students’ preparedness for higher-level training in different areas is variable. the proposal explored here is that students will benefit more from, and engage more deeply with, their training if they are supported in the acquisition of any required prior knowledge for the modules. *corresponding author. email: lumello@liverpool.ac.uk responsible editor: peter reed, laureate online education and the university of liverpool online, uk. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 l.v. mello. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29366/0 http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29366/0 http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29366/0 http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29366/0 http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 the research question addressed here was: ‘can the provision of online resources help to engage and motivate students to become self-directed learners?’ first, selfdirected learning (sdl) was explored as one way to support the transition from undergraduate to postgraduate learning. the research subsequently uncovered the perceived value of online learning tools (online tests and podcasts) that facilitated sdl. this study also reports analyses of students’ prior knowledge, motivation for and engagement with learning. background handelsman et al. (2004) argued that scientific teaching is faithful to the process of discovery which is embedded in science. science teaching must therefore encompass the critical thinking, rigour and creativity that define research. the objectives are thus to help students improve their thinking skills, and to encourage students to achieve deep learning. learning based on knowledge acquisition through teacher transmission is therefore less compatible with scientific teaching, where a continuous stimulation of critical thinking about new problems and situations is essential. knowles (1975) analysed reasons for adopting an sdl approach, linking sdl with lifelong learning and learning within the education context. therefore, there is a clear link between sdl and scientific teaching, as to succeed in science, students must engage in learning. they must identify their goals, the strategies to be employed and the methods to be used, and must be able to evaluate their decisions and outcomes. these features are the basis of sdl (brookfield 1991). koohang and paliszkiewicz (2013) argued that e-learning promotes active learning, which in turn promotes knowledge construction. chung and davis (1995) showed that blended learning environments can help students to develop learning responsibilities in time management and searching for learning resources. these findings support a linkage between online resources, active learning and sdl. in the uk higher education, the provision of further training skills for phd students involved in stem subjects (sciences, technology, engineering and mathematics) is a requirement of the research councils funding agencies (rcuk). as a result, masters and phd students take the same modules at postgraduate (pg) level. for the purpose of this study, in the context of postgraduate teaching, diversity is conceptualised as encompassing various differences in students’ backgrounds. these differences dictate students’ prior knowledge and, consequently, their readiness to acquire new knowledge at postgraduate level. at the pg level a working knowledge of bioinformatics has become essential for students, as most research projects will generate data that require bioinformatics analysis (hingamp et al. 2008; via et al. 2013). after running a bioinformatics module at pg level for 2 years, differences in the students’ prior knowledge became increasingly visible, since the module received students from increasingly diverse undergraduate (ug) backgrounds. background information about the module the module was designed to give a broad overview of bioinformatics to students on a new masters programme (advanced biological sciences, abs). its design was based on the expertise of members of staff in different areas of bioinformatics. the module l.v. mello 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 consisted of a series of 1 hour lectures followed by 2-hour workshops, where the students had the opportunity to learn by doing (blumenfeld et al. 2006). from its second year, the module received students from other three masters programmes, the pg-level msc and mres programmes in post-genomic sciences (pgs) and the integrated masters (intmaster). numbers on the latter course were very small (table 1) but they were in the same m-level year as the msc and mres students and so, although intmaster students are undergraduates, they were combined in overall numbers for this analysis. from the academic year 2011�12, phd students were required by rcuk to take taught modules as part of their training and, when they too joined the module in that academic year, they represented half of the module’s students (table 1). further student diversity resulted from the enrolment in the module of both native british and international students, some english native speakers (ens), others speaking english as second language (esl). recognising the range of potential in the postgraduate student cohort, delivery was shifted from the ‘traditional’ lecture structure, to a more interactive model, aimed at engaging students better and thereby improving their learning. to achieve this, online resources in the form of podcasts and online tests were introduced, giving the module a blended structure combining face-to-face delivery with e-learning. the goal was to keep students engaged with the next topic to be discussed through an sdl approach. the intervention in the teaching approach was introduced over two consecutive academic years. the purpose of this research was to determine if the provision of online resources would help to engage and motivate students to become sd learners and thereby improve their preparedness for the bioinformatics postgraduate module. when analysing the effect of the intervention, two categories were used: programme of study and language. entry requirements vary for different postgraduate programmes (usually being higher for phd students); thus, the first categorisation evaluated if the students’ previously obtained degree classification influenced the degree for which they already had the required prior knowledge for the module, and their subsequent performance. the second category allowed an assessment of whether language could act as a barrier for learning. as cohort sizes were small in the masters programmes, they were combined with phd students to obtain meaningful statistics. research methods the study was designed in the form of an action research, described by ferrance (2000) as ‘a cycle of posing questions, gathering data, reflection, and deciding on a course of action’. research ethical approval was sought and obtained through the table 1. size and programme composition of the classes over 4 years of module delivery. masters abs masters pgs language cohort year total size msc mres msc mres intmaster phd ens esl 2009�10 16 10 2 0 0 0 4 7 9 2010�11 27 14 5 3 1 3 1 18 9 2011�12 45 15 5 4 2 1 18 27 18 2012�13 33 2 6 1 7 1 16 23 10 abs �advanced biological sciences, ens �english native speaker, esl �english second language, pgs �post-genomic sciences. research in learning technology citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 research ethics committee. statistical analyses were performed using spss-21.0 (nie 1975). significance was assessed at the pb0.05 level. likert scale data from the questionnaires were treated as ordinal data, whereas performance data were treated as scale data. comparisons between masters and phd students who completed more or less than 50% of the online tests were assessed using one-way anova, and specific group differences were tested by post-hoc tukey test. statistical differences in performance between the groups of students who completed more or less than 50% of the online tests were analysed using t-tests for independent samples. to evaluate whether there were any significant differences between the answers of students from different programmes or with different english language background the mann� whitney u test was chosen. the main study: methods and interventions the interventions: sdl and online resources this work covers a period of 4 years, from academic session 2009�10, when the module was delivered for the first time, to 2012�13, its fourth year (figure 1). the problems relating to student diversity were first identified in the session 2010�11. staff responses, intervention phases 1 and 2, followed in 2011�12 and 2012�13, respectively. in intervention phase 1, half of the face-to-face module activities were complemented by online resources activities (oras). intervention phase 2 saw the online activities extended to the whole module (table 2). the interventions implemented centred on enhancement of sdl and considered three key aspects: resources, formal support and clarity of expectation. resources, additional to the core module elements of face-to-face lectures and workshops, were provided in the form of podcasts and online tests. formal support was provided in an additional computer skills workshop in the first week of the module, ensuring that lack of it skills would not hold back students’ learning. regarding expectations, students received information about the online resources in the first lecture of the course, and were made aware of their own responsibility for their learning. this message was regularly reinforced during the course, reflecting the claim of volery and lord (2000) that ‘the level of interaction between the students and the lecturer appears predominant in online delivery’. a brief explanation of sdl was given to the students, discussing staff expectations and the reasons behind the introduction of the online resources: student diversity and variable prior knowledge. the online resources were directly connected with the lecture and workshop material, as explained below. as students’ prior knowledge can dictate their readiness to learn new material, the expectation was that through sdl the students would have the opportunity to achieve the required ‘starting point’ for each session in the module. figure 2a illustrates the questions which helped in the design of the blended activity to support sdl (figure 2b). the activity is described in detail in the next section. the interventions: blended learning activity the ora consisted of podcasts and online tests, complemented by reading material (figure 2b). the resources were made available to students a week before a module session (lecture plus workshop). in addition, students had access to lecture slides and l.v. mello 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 workshop handouts in advance through the university’s virtual learning environment (blackboard). podcasts were used to explain the theory (lecture) and exercises (workshop), drawing students’ attention to the prior knowledge required, and suggesting prereading. according to clark, taylor, and westcott (2012) ‘podcasting offers one means for the lecturer to support all members of the unit of study’s student population by supporting less experienced participants’. test questions varied from ‘testing prior knowledge’ to ‘testing new knowledge’, obtained from the podcasts and reading figure 1. diagram illustrating the action research project planning, execution and evaluation in a chronological manner: (i) module delivery 1, just after module creation; (ii) module delivery 2, when the problem identification and first thoughts occurred; and module deliveries 3 (iii) and 4 (iv), the two intervention’s phases. research in learning technology citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 material. formative feedback for the tests was provided, indicating the correctly given answers and the total final scores. for incorrect answers, students were guided to resources where the correct answer could be found, encouraging further reading. the same test was made available to the students after the session, with correct answers provided. in this way, the test was used not only as a formative assessment but also for feedback and self-monitoring (sadler 1989). overall, this blended delivery model aimed to foster knowledge transformation by keeping students engaged with the next topic to be discussed. the first intervention was a trial, covering half the module (table 2), and occurred during module delivery 3 (figure 1). in the first lecture of the module, students were made aware of the blended activity, and of the expectation that the online resources would be used by them for catch-up and/or reinforcement. the desirability of completion of the ora before the face-to-face sessions was emphasised to students, with the explanation that it would help prepare them to acquire new knowledge. students were told that the online resources would also be available after the face-to-face sessions so that tests could be retaken and used for self-monitoring. intervention phase 2 occurred in module delivery 4 (figure 1) and covered almost the whole module. six members of staff were involved in delivery of the module, five of whom participated in the preparation of the ora in phase 2. supplementary file describes the learning activities of the interventions in detail. the main study: methods and evaluation of the outcome evaluation of this study aimed to assess if and how the students used the online resources offered in the module, if they found the resources beneficial to their learning table 2. module structure. sdl/online resources academic year intervention number of lectures (1 hour) number of workshops (2 hours) podcast test 2011�12 phase 1 8 11 7 5 2012�13 phase 2 8 11 11 9 figure 2. (a) the key questions designed to support an sdl approach. (b) the blended learning activity designed to support sdl. l.v. mello 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29366/0 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 experience and if the structure adopted in this study supported sdl. a questionnaire and a focus group were used to collect student feedback after intervention phase 1, and a questionnaire only after phase 2. in addition, students were invited to comment on the module in a school-wide end-of-module evaluation. a structured focus group was used, allowing students to freely discuss the online resources, and encouraging them to think about sdl. free-text comments from questionnaires as well as the focus group transcript were analysed using thematic analysis (braun and clarke 2006). research findings students’ evaluation of the intervention phase 1 questionnaires analysis since blackboard proved incapable of recording whether students listened partially or fully to podcasts, podcast results are based on the half of the class who replied to the questionnaire designed to evaluate the intervention. of all ens and esl students, 54 and 43%, respectively, participated in the survey. for the online tests, a full analysis of students’ access, marks and number of attempts for each test was performed via blackboard. students’ perception of the influence of the tests on their learning was assessed using the questionnaire responses and compared with data from blackboard. figure 3a shows that 62% (ens) and 67% (esl) of the students watched all the podcasts, revealing no significant difference between the groups (p �0.944, z � �0.070). figure 3b shows a positive student perception of the effect of the podcasts on their lecture and workshop learning, irrespective of whether english was their first or second language (p �0.823, z ��0.224). although naturally varying in their english abilities, all esl students who listened to the podcasts claimed that the podcasts helped with the ‘language issue’ (figure 3c). except for test3 and test5, the majority of the students attempted the tests and most, 64%, completed them all (table 3). test5 was released in the same week as a summative assessment for a compulsory module, possibly explaining its low use. all students who responded to the questionnaire said the tests helped them with selfassessment, irrespective of their language background. figure 4 indicates the percentage of students in each category � language and programme � to take each test. no significant difference was found between ens and esl (t-test: t30,9 �0.26, p �0.80) or between masters and phd cohorts (t-test: t42,8 �0.53, p �0.60). the degrees of masters students can be awarded with ‘distinction’ or ‘merit’, depending on mark criteria, while marks are not recorded on the phd certificate. therefore, if the students were only concerned about their marks, we might expect that masters students would engage more with the online resources, aiming to achieve better marks, but this was not the case. in addition, all students who did the tests agreed that they helped with their performance in assessments, irrespective of their language background or programme degree. the tests could also be taken after the face-to-face interaction, for self-assessment and self-monitoring. all students agreed with the statements ‘the online tests helped me with self-assessment’ indicating a positive use of the tests for self-assessment. the self-monitoring aspect was quantitated as the number of times the students took the tests and when. ideally students would take the tests at least twice, as proposed in the blended activity cycle (figure 2) monitoring their own progress. however, research in learning technology citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 analysis showed that few students took the tests both before and after the face-to-face sessions. evidently, the test feedback provided the students with an opportunity for self-assessment, but few took the opportunity to self-monitor their progress. subsequently, many students highlighted their heavy pg timetable as the reason for not revisiting the tests, rather than a lack of interest in self-monitoring. students are more likely to engage with summative than formative assessment (voelkel 2013); therefore, making the tests scores part of the module marks would increase the number of students revisiting the tests. however, this would go against the aim of the ora in this module which was to provide students with an optional ‘extra’ learning activity. nevertheless, self-monitoring is directly involved in the process of sdl (brookfield 1991; knowles 1975), and a further study could investigate the lack of engagement with self-monitoring reported herein. however, it is possible that students used their marks on the continuous summative assessments to evaluate their own learning. forty-one per cent of the students replied to the module evaluation questionnaire. responses to the open questions from both questionnaires show students’ positive table 3. online tests 2011�12. online test ot1 ot2 ot3 ot4 ot5 mean9sd not finished 30 11 8 17 7 � completed 37 65 44 70 26 � total 67 76 52 87 33 63929 percentage of students who attempted each online test (ot) with or without completion. figure 3. students’ views on the podcasts (cohort 2011�12). number of students from each group (ens, english native speaker; esl, english as second language) who (a) watched the podcasts; (b) level of agreement with the questionnaire statement ‘the podcast helped with lecture and workshop content understanding. . .’; (c) level of agreement with the questionnaire statement ‘the podcast helped with my language issue. . .’. total number of students �22, ens �13, esl �9; likert scale. l.v. mello 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 comments reinforcing the findings from the quantitative analysis. typical comments are presented below: e-learning activities should be provided for all modules. as a part-time student they were particularly useful for when i had to miss lectures due to work. i would have used all the podcasts/online tests if i’d have had enough time. (student 1) i felt that the e-learning activities were very useful and made the learning process much easier, especially for someone who has not come across many of these topics and subjects before. (student 2) this method provides confidence to those who struggle to adapt to a more self-directed learning strategy, whilst creating an ideal environment for those who relish their independence. (student 3) focus group analysis the main aim of the focus group was to evaluate students’ views of the online resources with respect to important features of sdl. eleven students voluntarily attended the session, four msc, five mres and four phd. the focus group discussion revealed students’ clear understanding of the activity, flexibility in the way it was used (depending on the student’s needs and available time), enhancement of the students’ perception of learning and appreciation of the tutor support and time dedicated towards the implementation of sdl (table 4). thus, it is fair to conclude that important sdl features were achieved in the new blended learning structure of the module and appreciated by the students. students’ evaluation of the intervention phase_2 thirty students out of 33 (91%) replied to the questionnaire designed to evaluate the intervention. ninety-four per cent of all masters and 88% of all phd students responded to the questionnaire. grouped by language, 86% of all ens and 100% of the esl students responded. for analysis using data from blackboard, all students were included. figure 5a shows that 37% (ens) and 70% (esl) of the students watched all the podcasts. including students who watched only some of the podcasts, these figures increase to 89 and 91%, respectively. as in intervention phase 1, students considered that the podcasts had a positive impact on their understanding of lectures figure 4. students’ attempts at online tests (cohort 2011�12). percentage of students from each category: (a) ens (english native speaker) and esl (english as second language); and (b) masters and phd students who took each of the five online tests. n �the number of students in each group is indicated in parentheses. research in learning technology citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 and workshops, irrespective of language and programme differences (ens�esl: p �0.42, z � �0.803; and masters�phd: p � 0.29; z ��0.907) (figure 5b). however, a difference was seen with regard to the question of whether the podcasts helped with the language issue (figure 5c). some of the esl claimed not to have any language problem while one ens claimed the podcast has helped them regarding language issue. this particular student may have referred to problems regarding subject-specific jargon, or in comprehending the spoken words of a foreign staff member. table 5 summarises student attempts of the tests. test1 and test2 show a low percentage of test completions. from test3 an increased usage of the tests was observed (with the exception of test7 where students reported a clash with a summative assessment in the same week). figure 6a and b show the percentages of students in each category, language and programme, to take each test. as in phase 1, no statistically significant difference was found between ens and esl (t-test: t17,11 � �0.74, p �0.47) or between masters and phd students (t-test: t29,10 �0.66, p �0.52), indicating they used the tests with similar frequency. ens students (masters or phd) who responded to the questionnaire said the tests helped them with self-assessment. for the group esl, the difference between those who agreed with the statement and those who either disagreed or had no opinion is not as large as for the other groups (figure 6c). the self-monitoring analysis, based on the number of times the students took the tests and when, revealed that very few students took them both before and after face-toface sessions. this confirms the previous interpretation: students used the tests for selfassessment, but few took the opportunity to self-monitor their progress. students commented favourably on the immediate feedback from the tests. this exercise revealed that students can value formative feedback as a means of engagement rather than being simply driven by grades, as commonly reported (nicol and macfarlane-dick 2006). table 4. students’ views on the blended activity regarding sdl. students’ view sdl features students generally understood the structure for sdl � pre-lecture podcasts to introduce new topics/support they could expect from the tutor. structure different students used the sdl resources in different ways � students new to topics found it a good starting point/phd students used it as a check on topics they had already learnt/ other students used references in reading to deepen topics they were interested in. flexibility not all students used all of the sdl resources all the time � if time was short they sometimes did not use them. flexibility most of the students found the structure and process supportive for their learning � none thought it to be too structured or under-structured. enhance learning many commented that they would like more staff to adopt a similar approach but appreciated that it must take a lot of effort on the tutor’s part to set them up. enhance learning they valued and appreciated the apparent effort put in by the tutor and her responsiveness to problems or questions they were struggling with from the sdl resources. teacher as facilitator most used the online tests as a revision tool for assessments and appreciated the immediate feedback for self-assessment. structure and teacher as facilitator l.v. mello 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 this finding is supported by higgins, hartley, and skelton (2002) in their 3-year study showing that formative assessment feedback is essential to encourage deep learning. overall, the tests, as in the previous year, clearly positively affected students’ perception of their learning. fifty-two per cent of the students completed the final module evaluation. students’ comments reinforced the findings from the survey and highlighted other positive aspects of the blended structure that had not previously been considered: as a dyslexic phd student, your blended course removed the barriers found with traditional styles of teaching and made the material accessible and understandable not just for myself but for all students of all abilities. figure 5. students’ views on the podcasts (cohort 2012�13). number of students from each category: programme and language who (a) watched the podcasts; (b) level of agreement with the questionnaire statement ‘the podcast helped with lecture and workshop content understanding. . .’; (c) level of agreement with the questionnaire statement ‘the podcast helped with my language issue. . .’. total number of students �30, ens �19, esl �11; masters �16, phd �14; likert scale. ens: english native speaker; esl: english as second language. table 5. online tests 2012�13. online test ot1 ot2 ot3 ot4 ot5 ot6 ot7 ot8 ot9 mean/sd not finished 6 9 9 6 0 6 9 13 9 � completed 25 28 50 59 56 47 15 59 31 � total 31 37 59 65 56 53 24 72 40 50930 percentage of students who attempted each online test (ot) with or without completion. research in learning technology citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 table 6 shows the students’ opinion of sdl. ninety-seven per cent claimed they understood the meaning of sdl. of those, 50% of masters students said they liked it, against 77% of the phd students. bhat, rajashekar, and kamath (2007) in their work with first-year medical students claimed that students have a mix of abilities and questioned if all students would benefit from sdl. the authors compared the marks achieved by students from sdl and didactic lectures. the sdl activity consisted of individual research on a specific topic, followed by presentation and discussion. results showed that students achieving good results with didactic lectures scored even higher marks in sdl topics. however, for lower achievers no significant difference in marks was observed. if this study implies a correlation between academic level and readiness for sdl, then we would expect that higher performing students would appreciate the sdl approach more than others. indeed, this suggestion is in agreement with the higher appreciation of sdl by phd students (generally first class ug degrees) than by masters students (a mix including students achieving a 2.2 � according to the uk degree classification). inventories were not figure 6. students’ attempts at online tests (cohort 2012�13). percentage of students from each category: (a) ens (english native speaker) and esl (english as second language); and (b) masters and phd students who took each of the five online tests. n �the number of students in each group is indicated in parentheses. (c) number of students from each language and degree groups who agreed with the questionnaire statement ‘the online test helped with my self-assessment. . .’ table 6. students’ view of sdl (cohort 2012�13). know and like know and do not like no meaning masters 8 (50%) 7 (44%) 1 (6%) phd 10 (77%) 3 (23%) 0 (0%) know and like �the student knows the meaning of sdl and likes the approach; know and do not like �the student knows the meaning of sdl but does not like the approach; no meaning �the student does not know the meaning of sdl. l.v. mello 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 used to measure sdl readiness, as this would have required the students to fill in another long questionnaire. although some students, particularly msc students, were not positive about sdl, the same students could perceive the advantages of ora and, indeed, requested that other modules follow suit and provide them. this contradiction perhaps suggests that students did not fully appreciate the connection between the two. as in the focus group evaluation of the 2011�12 cohort, students said sdl activities should be supported and structured, as advocated by knowles (1975). but comments also showed that students prefer a combination of sdl and face-to-face activities to sdl only. overall, students’ satisfaction with the ora was clear and they requested that similar activities should be provided in other modules. did the blended activity influence students’ performance? this section evaluates whether the blended structure of the module truly had any tangible influence on the students’ marks, or if students merely perceived that it had helped them to learn. for both years, an in-year comparison between masters and phd students was done, dividing students into those who had completed more or less than 50% of the tests. statistical tests showed no significant difference between the four groups: for the group who completed �50% of the tests in 2011�12, the tukey p-value was 0.999. for the 2011�12 b50% group it was 0.628, and the corresponding 2012�13 values were 0.999 and 0.885. masters and phd students were therefore pooled within each year, after which a significant difference in performance was seen between those who took more or less than 50% of the tests. for the 2011�12 cohort, the average final marks of the module were 6997.2 and 7596.0, for students who took b50 or �50%, respectively (t-test: t40,2 ��3.044, p �0.004). for the 2012�13 cohort the corresponding mean marks were 67911.0 and 7495.0, respectively (t-test: t27,3 ��2.079, p �0.041). however, it is unlikely that the tests were the only influence on students’ marks. analysis of the questionnaire showed a correlation between the use of online tests and podcasts: most students used either both or neither. thus, it is perhaps more likely that the students who engaged more assiduously with the module in general were rewarded by higher marks. importantly, as the online tests were mainly designed to tell the students what to study for the next session, and no marks were attached to them, any difference in performance deriving from their use can be credited to student engagement with the sdl approach. the results of kibble (2007) also showed a significantly better exam performance in students who participated in voluntary, formative online quizzes, compared with those who did not. interestingly, no significant difference was found between the module final marks of masters and phd students. however, this finding should be treated with caution, due to the relatively small sample size available in this study. unfortunately, in the domain of biological sciences, large postgraduate classes, particularly with mixed programme cohorts, are rare making further analysis difficult. an extensive literature search revealed no similar published studies in other subject areas, emphasising the gap in the literature regarding postgraduate mixed programme cohorts. for the podcasts, although their direct impact on students’ marks cannot be measured, the findings of this study suggest an increase in students’ perception of learning. the work of clark, taylor, and westcott (2012) with postgraduate students also showed that students perceived learning benefits from podcasts when they were research in learning technology citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 introduced to address students’ prior knowledge. price et al. (2006) advocate that short podcasts are ‘within the attention span of an audience’, an aspect that could contribute to the higher use also observed in other studies (kukulska-hulme 2012; morrissey 2012). discussion and conclusion this study analysed whether the provision of online resources could help to engage and motivate students to become self-directed learners. sharpe et al. (2006) highlighted that the use of online resources as a way to provide immediate feedback helps with students’ engagement in out-of-class activities. analysis of the interventions revealed that the majority of students from the two cohorts used the resources and that their perception of a positive impact on their learning was accompanied by a tangible increase in their actual module marks as well as a reported higher motivation for the module. this result partially contradicts laird and kuh’s study (2005) that showed a strong positive relationship between the use of online resources and students’ engagement with active learning, despite the lack of a corresponding increase in marks. there was an expectation that different programme cohorts would make different use of the online activities. however, no difference was found in the use of resources between masters students, who can be awarded a final grade of ‘distinction’ or ‘merit’ and doctoral students, who may only pass or fail. furthermore, there was no difference in usage between ens and esl student groups, indicating that the resources embedded in the module were consistently used irrespective of student’s programme or language skills. the blended approach made available to support sdl was perceived and valued by all students. nevertheless, a difference was observed in how students perceive sdl: doctoral students seemed to prefer the approach and were more receptive to it than masters students. some students reported that the sdl activity helped them to achieve more independence compared with traditional approaches to teaching. apparently, the new activities implemented in this module contributed to the creation of a learning environment that not all students had necessarily experienced before. recognising this, different appreciation of the learning tools by students is only to be expected, as not all students welcome changes in their learning environment irrespective of their beneficial effect (entwistle and peterson 2004). self-monitoring is a key feature of sdl, and lack of familiarity with sdl could in turn explain students’ non-familiarity with self-monitoring. the use of online resources to support and enhance student learning and assessment is well documented in the literature. however, in terms of the use of the same online test to support and enhance self-monitoring, we found the literature to be limited, particularly when the test does not contribute to module marks. this study examined just one model of sdl: active learning through the use of online resources. although students appreciated the design of the sdl activities, a limitation of this study was the lack of an initial assessment of students’ prior experience of sdl. thus, the extent to which students developed their sdl skills cannot be quantified. one question that remains is whether doctoral students are indeed better prepared for an sdl approach through having the required skills. positive correlations between the sdl readiness scale (sdlrs, guglielmino 1978) l.v. mello 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 scores and academic performance are described by ting (1996) and shokar et al. (2002). in contrast, nepal and stewart (2010) and abraham et al. (2011) report no correspondence between sdl readiness and grade point average (gpa). the latter highlighted that although higher achievers presented high scores in the self-control category of the sdlrs scale, all students seemed to require support in their selfmanagement skills. this study provides further evidence that students require structured support prior to commencing sdl activities, until they achieve a greater degree of independence. in this regard, students’ satisfaction with the structure adopted in this study can be taken as a positive reflection of the care that went into the provision of that support. references abraham, r. r., et al., (2011) ‘exploring first-year undergraduate medical students’ selfdirected learning readiness to physiology’, advances in physiology education, vol. 35, no. 4, pp. 393�395. bhat, p. p., rajashekar, b. & kamath, u. (2007) ‘perspectives on self-directed learning � the importance of attitudes and skills’, bioscience education, vol. 10, [online] available at: http://www.heacademy.ac.uk/sites/default/files/beej.10.c3.pdf blumenfeld, p. c., et al., (2006) ‘learning with peers: from small group cooperation to collaborative communities’, educational research, vol. 25, no. 8, pp. 37�40. braun, v. & clarke, v. (2006) ‘using thematic analysis in psychology’, qualitative research in psychology, vol. 3, no. 2, pp. 77�102. brookfield, s. (1991) developing critical thinkers: challenging adults to explore alternative ways of thinking and acting, jossey-bass, san francisco, ca. chung, j. & davies, i. k. (1995) ‘an instructional theory for learner control: revisited’, [online] available at: http://www.editlib.org/p/78439 clark, s., taylor, l. & westcott, m. (2012) ‘using short podcasts to reinforce lectures’, proceedings of the australian conference on science and mathematics education, australia sydney, pp. 22�27. entwistle, n. j. & peterson, e. r. (2004) ‘conceptions of learning and knowledge in higher education: relationships with study behaviour and influences of learning environments’, international journal of educational research, vol. 41, no. 6, pp. 407�428. ferrance, e. (2000) ‘action research’, the education alliance, [online] available at: http://www. brown.edu/academics/education-alliance/sites/brown.edu.academics.education-alliance/files/ publications/act_research.pdf guglielmino, l. m. (1978) ‘development of the self-directed learning readiness scale’, dissertation abstracts international, vol. 38, no. 11�a, pp. 6567. handelsman, j., et al., (2004) ‘scientific teaching’, science, vol. 304, pp. 521�522. higgins, r., hartley, p. & skelton, a. (2002) ‘the conscientious consumer: reconsidering the role of assessment feedback in student learning’, studies in higher education, vol. 27, no. 1, pp. 53�64. hingamp, p., et al., (2008) ‘metagenome annotation using a distributed grid of undergraduate students’, plos biology, vol. 6, no. 11, pp. 2362�2367. kibble, j. (2007) ‘use of unsupervised online quizzes as a formative assessment in a medical physiology course: effects of incentives on student participation and performance’, advances in physiology education, vol. 31, no. 3, pp. 253�260. knowles, m. (1975) self-directed learning: a guide for learners and teachers, association press, new york. koohang, a. & paliszkiewicz, j. (2013) ‘knowledge construction in e-learning: an empirical validation of an active learning model’, journal of computer information systems, vol. 53, no. 3, pp. 109�114. kukulska-hulme, a. (2012) ‘how should the higher education workforce adapt to advancements in technology for teaching and learning?’, the internet and higher education, vol. 15, no. 4, pp. 247�254. research in learning technology citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 15 (page number not for citation purpose) http://www.heacademy.ac.uk/sites/default/files/beej.10.c3.pdf http://www.editlib.org/p/78439 http://www.brown.edu/academics/education-alliance/sites/brown.edu.academics.education-alliance/files/publications/act_research.pdf http://www.brown.edu/academics/education-alliance/sites/brown.edu.academics.education-alliance/files/publications/act_research.pdf http://www.brown.edu/academics/education-alliance/sites/brown.edu.academics.education-alliance/files/publications/act_research.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 laird, t.f. n. & kuh, g.d. (2005) ‘student experiences with information technology and their relationship to other aspects of student engagement’, research in higher education, vol. 46, no. 2, pp. 211�233. morrissey, j. (2012) ‘podcast steering of independent learning in higher’, aishe-j, vol. 4, no. 1, pp. 1�9. nepal, k. p. & stewart, r. a. (2010) ‘relationship between self-directed learning readiness factors and learning outcomes in third year project-based engineering design course’, 21st annual conference for the australasian association for engineering education, sydney, australia, pp. 496�503. nicol, d. j. & macfarlane-dick, d. (2006) ‘formative assessment and self-regulated learning: a model and seven principles of good feedback practice’, studies in higher education, vol. 31, no. 2, pp. 199�218. nie, n. h. (1975) spss: statistical package for the social sciences, mcgraw-hill, new york. price, a., et al., (2006) ‘a history and informal assessment of the slacker astronomy podcast’, astronomy education review, [online] available at: http://arxiv.org/ftp/astro-ph/papers/ 0606/0606326.pdf sadler, d. r. (1989) ‘formative assessment and the design of instructional systems’, institutional science, vol. 18, pp. 119�144. sharpe, r., benfield, g., roberts, g. & francis, r. (2006) the undergraduate experience of blended e-learning: a review of uk literature and practice, higher education academy, london. shokar, g. s., et al., (2002) ‘self-directed learning: looking at outcomes with medical students’, family medicine, vol. 34, no. 3, pp. 197�200. ting, t. m. (1996) the relationship between the self-directed learning readiness, attitudes towards computers and computer competencies. master’s thesis, national ching cheng university, taiwan. via, a., et al., (2013) ‘best practices in bioinformatics training for life scientists’, briefings in bioinformatics, vol. 14, no. 5, pp. 528�537. voelkel, s. (2013) ‘combining the formative with the summative: the development of a twostage-online test to encourage engagement and provide personal feedback in large classes’, research in learning technology, vol. 21, no. 1, pp. 19153. volery, t. & lord, d. (2000) ‘critical success factors in online education’, the international journal of educational management, vol. 4, no. 5, pp. 216�223. l.v. mello 16 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29366 http://dx.doi.org/10.3402/rlt.v24.29366 http://arxiv.org/ftp/astro-ph/papers/0606/0606326.pdf http://arxiv.org/ftp/astro-ph/papers/0606/0606326.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29366 http://dx.doi.org/10.3402/rlt.v24.29366 doi:10.3402/rlt.v20i0/14406 viewing mobile learning from a pedagogical perspective matthew kearney a, *, sandra schuck a , kevin burden b and peter aubusson a a centre for research in learning and change, university of technology, sydney (uts), australia; b the centre for educational studies, faculty of education, the university of hull, hull, uk (received 22 december 2010; final version received 15 august 2011) mobile learning is a relatively new phenomenon and the theoretical basis is currently under development. the paper presents a pedagogical perspective of mobile learning which highlights three central features of mobile learning: authenticity, collaboration and personalisation, embedded in the unique timespace contexts of mobile learning. a pedagogical framework was developed and tested through activities in two mobile learning projects located in teacher education communities: mobagogy, a project in which faculty staff in an australian university developed understanding of mobile learning; and the bird in the hand project, which explored the use of smartphones by student teachers and their mentors in the united kingdom. the framework is used to critique the pedagogy in a selection of reported mobile learning scenarios, enabling an assessment of mobile activities and pedagogical approaches, and consideration of their contributions to learning from a socio-cultural perspective. keywords: mobile learning; pedagogy; socio-cultural theory; framework; pedagogical features 1. introduction portable, handheld devices have increasingly powerful multimedia, social networking, communication and geo-location (gps) capabilities and consequently, mobile learning (m-learning) offers numerous opportunities as well as challenges in education. despite the ubiquity and flexibility of these devices, there has been minimal use of m-learning approaches in some education sectors and developments have tended to be more about the design of the tools than of the ensuing learning. there is an ongoing need to examine the pedagogies that are suitable for m-learning, and to conceptualise m-learning from the perspective of learners’ experiences rather than the affordances of the technology tools (traxler 2007). this paper investigates what a pedagogical framework for m-learning may look like from a socio-cultural perspective. this theoretical perspective suggests that learning is affected and modified by the tools used for learning, and that reciprocally the learning tools are modified by the ways that they are used for learning. central to our position here is the notion that learning is a situated, social endeavour, facilitated and developed through social interactions and conversations between people (vygotsky 1978), and mediated through tool use (wertsch 1991). (page number not for citation purpose) *corresponding author. email: matthew.kearney@uts.edu.au research in learning technology vol. 20, 2012 rlt 2012. # 2012 m. kearney et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 identifying specific, current features of m-learning and m-teaching from a sociocultural perspective provides a potentially useful lens for researchers’ analyses of pedagogical approaches; helps teachers to critique and reflect on their teaching activities and offers critical insights into the design of m-learning materials. our framework offers an examination of m-learning which foregrounds pedagogy rather than technology; a perspective in which the pedagogy is central and the technology is under investigation only for what may be distinctive about the learning afforded by that technology. although sophisticated theoretical models have been developed (laurillard 2007; pachler, bachmair, and cook 2009; sharples, taylor, and vavoula 2007), locating distinctive features of learning with mobile devices is an evolving process as devices and associated technologies mature. accordingly, informed both by current m-learning theory and by socio-cultural theory, this paper identifies three distinctive features of m-learning through our framework. the features are authenticity, collaboration and personalisation. the pedagogical framework was developed and extensively tested through a range of activities in two m-learning projects located in teacher education communities. mobagogy was a professional learning community of eight academics in an australian university, formed to investigate how to use mobile technologies in their own learning and teaching (schuck et al. 2010). the community met regularly over a period of 18 months to discuss emerging relevant teaching issues and applications. the bird in the hand project was a uk sponsored initiative supported by the teacher development agency and examined the experiences of a group of trainee and newly qualified teachers who were provided with smartphones (iphones) to use in their placement and first teaching schools. it explored how a group of eight trainee teachers and their mentors used smartphones to support and enhance their professional practice. extensive descriptions of activities within both these projects are available elsewhere (kearney, schuck, and burden 2010). 2. background m-learning is described in numerous ways, but these descriptions all consider the nexus between working with mobile devices and the occurrence of learning: the process of learning mediated by a mobile device. numerous characteristics of m-learning have been identified in the literature. koole’s (2009) frame model sits well with socio-cultural views of learning, taking into consideration both technical characteristics of mobile devices as well as social and personal learning processes. she refers especially to enhanced collaboration, access to information and deeper contextualisation of learning. our paper extends koole’s model, to include understandings of ‘‘mobile pedagogy’’ which draw on socio-cultural understandings presented in her model. danaher, gururajan, and hafeez-baig (2009) propose a framework based on three key principles: engagement, presence and flexibility (see figure 1). ‘‘presence’’ refers to the ‘‘simultaneous awareness and locatedness of self and others . . . encompassing the emotional element of being human’’ (26). they further breakdown ‘‘presence’’ into three sub-group ‘‘interaction types’’: cognitive (student-content), social (peer) and teaching (student-teacher). inherent in this model is implicit discussion of pedagogy; the aim of our paper is to make this discussion central and explicit. m. kearney et al. 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 other researchers have provided insights into different social aspects of m-learning. traxler (2009, 30) described m-learning as ‘‘noisy’’ and problematic, featuring three essential elements: the personal, contextual and situated; while klopfer, squire, and jenkins (2002) identified five features: portability, social interactivity, context sensitivity, connectivity and individuality. pachler, cook, and bachmair (2010) analysed the interrelationship of learners with the structures, agency and cultural practices of what the authors call the ‘‘mobile complex’’ (1). the identification of these sets of characteristics and relationships established core features that we had to ensure were addressed in the development of our framework. larger-scale, more complex conceptual frameworks for m-learning design and evaluation have been proposed. parsons, ryu, and cranshaw (2007) proposed a complex conceptual framework for m-learning with four perspectives: generic mobile environment issues, learning contexts, learning experiences and learning objectives. vavoula and sharples (2009) proposed a three-level framework for evaluating m-learning, comprising a micro-level concerned with usability, a meso level focusing on the learning experience (especially on communication in context) and a macro level dealing with integration within existing organisational contexts. our framework aims to further interrogate this ‘‘meso level’’ of learners’ experience. hence numerous frameworks have been proposed in the literature, ranging from complex multi-level models (e.g. parsons, ryu, and cranshaw 2007) to smaller frameworks that often omit important socio-cultural characteristics of learning or of pedagogy. common themes include portability of m-learning devices and mobility of learners; interactivity; control and communication. these descriptions acknowledge the prime importance of context, including spatial and temporal considerations, for analysing m-learning experiences. however, they typically attempt to merge affordances of mobile devices or characteristics of applications with features of the learners’ experience. while acknowledging that the features identified in other frameworks are important in characterising technology-mediated learning by mobile users, we propose a succinct framework highlighting a unique combination of distinctive characteristics of current mobile pedagogy to bring socio-cultural insights to the literature on m-learning. 3. time-space considerations formal learning is traditionally characterised by two constants or boundaries: time and space. learning places occupy fixed, physical spaces which are defined by relatively impermeable boundary objects such as walls, classrooms and school buildings. similarly, traditional learning is situated in permanent temporal slots such figure 1. from danaher, gururajan, and hafeez-baig (2009, 23). copyright: igi global reprinted by permission of the publisher. research in learning technology citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 3 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 as teaching periods (timetables or semesters) which are relatively immutable (traxler 2009). m-learning has the potential to transcend these spatial and temporal restrictions, overcoming ‘‘the need to tie particular activities to particular places or particular times’’ (traxler 2009, 7). with ‘‘space’’, m-learning offers a variety of alternatives including ‘‘virtual’’ or non-geographical spaces, such as virtual world environments created for mobile devices. in temporal terms, the requirement to learn in fixed, scheduled time spaces (which characterise current schooling) are also relaxed enabling the individual to be more flexible about when they learn. previously fixed engagements or appointments can now be readily rescheduled and fixed notions of linear time are increasingly making way for a softer version of what some authors have termed ‘‘socially negotiated time’’ in which each party to an event is able to create and rearrange their schedules without excessive detrimental effect to either side (ling and donner 2009). the implications of these two vectors in m-learning are beyond the scope of this particular paper but taken together they create what we term ‘‘malleable spatialtemporal contexts for learning’’. in blurring the physical and scheduled personality of institutional-based learning, time-space implications of m-learning open up opportunities for a wide variety of pedagogical patterns. mobile technologies thus enable learning to occur in a multiplicity of more informal (physical and virtual) settings situated in the context about which the learning is occurring. these informal scenarios range from structured, teacher-mediated experiences in semi-formal places like museums and libraries, to more self-regulated experiences in learner-generated contexts such as coffee shops and public transport settings (luckin 2010). we are not attempting to identify specific causal links between the level of formality of ‘‘time-space’’ and m-learning experiences. however, to discuss distinctive features of mobile pedagogy, we must firstly acknowledge that the organisation of ‘‘time-space’’ in any learning environment profoundly affects m-learning experiences (ling and donner 2009). from a socio-cultural standpoint, insights into the organisation of ‘‘time-space’’ in a given learning environment is an essential part of understanding the nature of a m-learning experience, as depicted in figure 2. figure 2. a two-way relationship between the organisation of time-space and m-learning experiences (socio-cultural perspective). m. kearney et al. 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 4. framework development and validation: locating distinctive mobile pedagogy features the current framework was developed through an iterative design-test-analyse-refine cycle, akin to that suggested by kemmis and mctaggart (1988), to address our key question: what does a pedagogical framework for m-learning look like from the perspective of socio-cultural theory? activities in both projects fed into this cycle and leveraged numerous opportunities to test and refine the framework and its representation. project activities contributing to the framework development included: exploring the socio-cultural characteristics underpinning m-learning; interrogating the literature on m-learning; investigating best practice approaches by interviewing global experts in the field; and initiating and testing selected m-learning pedagogies in the context of our own higher education subjects. a variety of strategies were used to promote collaborative critical reflection (ghaye and ghaye 1998) throughout the cycle, taking into account a range of perspectives from discipline, pedagogical and e-learning experts in our group. the framework was validated through four methods. firstly, inter-researcher validation was gained using feedback from m-learning researchers after presenting versions of the framework at four scholarly meetings: one internal teaching and learning university conference; a m-learning working group with scholars from around australia and beyond; an internal faculty presentation and an international m-learning conference (kearney, schuck, and burden 2010). secondly, intraresearcher validation was achieved through discussions amongst the designers of the framework. these discussions critiqued the framework from a pedagogical perspective and interrogated how well it aligned with the underlying socio-cultural theory. thirdly, each iteration of the framework was tested by using it to analyse existing m-learning initiatives in both the mobagogy and the bird in the hand projects, and also using it to guide the design of further m-learning experiences. fourthly, a critical friend � an expert in pedagogy from within the group � was invited to critique final iterations of the framework and subsequently, to become a fourth author of this paper. his feedback contributed to the current framework presented here. these methods involving the users’ perspective in the design process follow general design guidelines based on constructivist theory (willis 2000). informed by these processes and mindful of our quest to use socio-cultural theory to capture central pedagogical features of m-learning environments, a framework prototype was designed using four dimensions: place, connection, immediacy and activity. this early version of the framework integrated temporal and spatial considerations. this version was ‘‘tested’’ by using it to critique our student teachers’ use of mobile devices to vote on a controversial issue in a mass lecture (see upward thick arrows on each of the four scales in figure 3). in another example from our project trials, this version of the framework was used to critique the lack of interactivity in a group member’s trial of student teachers’ instructional use of podcasts. similar trials took place in the united kingdom where versions of the framework were used, for example, to gauge the extent to which trainee teachers could sustain a vibrant sense of community, which had characterised their face-to-face elements, whilst away from the university on their first teaching placements. further iterations of the framework emerged from our design and development cycle, as we tried to capture more succinctly the distinctive features of m-learners’ research in learning technology citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 5 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 experiences. a well-developed framework incorporating five ‘‘scales’’ and numerous sub-scales was presented at our university teaching and learning conference (see figure 4). a more succinct, penultimate version of the framework (see figure 5) was subsequently presented at mlearn2010 (kearney, schuck, and burden 2010). apart from a more succinct representation, a major development here was our treatment of ‘‘time-space’’ as a separate entity in the framework. figure 3. use of a prototype framework to analyse one of our project teaching trials. m. kearney et al. 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 further feedback from m-learning researchers at the conference, and from our critical friend, was valuable and informed refinement of the framework, in light of other data from the project activities. for example, one conference reviewer suggested we more closely examine critical features of games-based m-learning scenarios to help us further clarify the customisation section of our framework. descriptions of figure 4. another prototype framework presented at a university teaching conference, 2009. figure 5. penultimate framework presented at mlearn 2010 (kearney, schuck, and burden 2010). research in learning technology citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 7 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 this scale and other sections of the framework were subsequently refined. our critical friend critiqued our use of a ‘‘third space’’ theme (kearney, schuck, and burden 2010) and suggested that this might be a distraction to the main focus of presenting the three pedagogical constructs. also, two subsidiary sub-scales were developed for each section to more accurately pinpoint critical features of m-learning. as part of this final development, the customisation scale was changed to personalisation in the current framework, with sub-scales of agency and customisation. similarly, the social interactivity scale was changed to collaboration with sub-scales of communication and data sharing (see next section). also, it became evident that the ‘‘three circles’’ representation (see figure 5 above) caused confusion regarding ‘‘intersecting sections’’ and consequently, the three scales have been separated in the current visual representation (see figure 6). 5. current framework in this section, we describe a rationale for including personalisation, authenticity and collaboration as the three distinctive features of m-learning forming the basis of our current framework, working within our previously discussed conception of ‘‘time and space’’. we also have formulated two sub-scales for each of these three constructs, as depicted in figure 6 and described in the subsequent sub-sections. this current graphical representation now consists of circular layers, to show the close, connected relationship between the three constructs depicted in the inner ‘‘layer’’ and the six sub-scales in the outer layer. the bi-directional arrows in the representation depict the previously discussed symbiotic relationship between ‘‘time-space’’ and m-learning features. figure 6. current framework comprising three distinctive characteristics of m-learning experiences, with sub-scales. m. kearney et al. 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 5.1 personalisation personalisation, drawing on motivational theory (pintrich and schunk 1996) and socio-cultural theory (vygotsky 1978), has become a corner stone of e-learning. key features associated with personalisation include learner choice, agency and selfregulation as well as customisation (mcloughlin and lee 2008). learners can enjoy a high degree of agency in appropriately designed m-learning experiences (pachler, bachmair, and cook 2009). they may have control over the place (physical or virtual), pace and time they learn, and can enjoy autonomy over their learning content. goals are typically set by learners and their peers (e.g. some games). furthermore, the ‘‘just enough, just-in-time, just-for-me’’ nature of some m-learning activities can create a personalised, tailored learning journey. m-learning experiences can be customised at both a tool and activity level. users enjoy a sense of intimacy and convenience with their personal devices and the flexible, autonomous, often individually tailored activities lead to a strong sense of ownership of one’s learning (traxler 2007). in this sense, activities are customised for the learner to meet their different learning styles and approaches. hence, we used two sub-scales (agency and customisation) in our analysis of personalisation, as shown in (table 1). mobile users can use tools to record, organise and reflect on their customised m-learning experiences over time (naismith et al. 2004). emerging ‘‘context-aware capabilities’’ allow devices to acquire information about the user and their immediate environment (e.g. time, location, nearby people and objects), presenting unique opportunities to personalise learning experiences. also, emerging ‘‘augmented reality’’ applications and customised interactions with ‘‘the internet of things’’ (sundmaeker et al. 2010) offer promising ways for learners to select, manipulate and apply information to their own unique needs in a ‘‘pervasive learning environment’’ (laine et al. 2009). 5.2 authenticity there is general agreement that authentic tasks provide real world relevance and personal meaning to the learner (radinsky et al. 2001), although ultimately, authenticity ‘‘lies in the learner-perceived relations between the practices they are carrying out and the use value of these practices’’ (barab, squire, and dueber 2000, 38). ctgv (1990) delineate task, factual and process levels of authenticity. task authenticity refers to the extent to which tasks are realistic and offer problems encountered by real world practitioners. factual authenticity refers to how particular details of a task (such as characters, instruments etc.) are similar to the real world, while a process level of authenticity refers to how learner practices are similar to those practices carried out in the community or ‘‘real-world’’ of practice. radinsky table 1. two sub-scales of the personalisation construct used in our framework. scale sub-scale low activity is: high activity is: personalisation agency (pachler, bachmair, and cook 2009) externally controlled negotiated learning choices, for example, content, goals customisation (mcloughlin and lee 2008) uniformly structured, just-in-case tailored; just-enough, just-in-time, just-for-me research in learning technology citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 9 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 et al. (2001) espoused two models of authentic learning environments: a simulation model and participation model. tasks that fit a simulation model of authenticity use the learning space (e.g. classroom) as a ‘‘practice field’’ (separate from the ‘‘real community’’) but still provide contexts where learners can practise the kind of activities they might encounter outside of formal learning settings. alternatively, under a participation model of authenticity, students participate in the actual work of a professional community, engaging directly in the target community itself. hence, we used two sub-scales (contextualisation and situatedness) in our analysis of authenticity, as shown in (table 2). m-learning episodes potentially involve high degrees of ‘‘task and process authenticity’’ as learners participate in rich, contextual tasks (setting, characters, tools), involving ‘‘real-life’’ practices. learners can generate their own rich contexts (pachler, bachmair, and cook 2009) with or through their mobile devices. the deeper contextualisation of tasks in these physical or virtual spaces can be supported by geolocation and data capture facilities (brown 2010). 5.3 collaboration collaboration in socio-cultural theory is often emphasised in terms of learning interactions with more capable peers or adults and there is a pedagogical emphasis on scaffolding (trudge 1990). more broadly, social interaction, conversation and dialogue are fundamental to learning from a socio-cultural perspective as people engage in negotiating meaning (vygotsky 1978). recent pedagogical frameworks foreground the importance of these conversations in teaching and learning (e.g. laurillard 2007; sharples, taylor, and vavoula 2007), building on well-accepted vygotskian theory. shared conversational spaces mediated by mobile devices are conducive to timely, personally tailored feedback from instructors as well as rich peer interactions (e.g. multi-user mobile gaming environments). m-learners can enjoy a high degree of collaboration by making rich connections to other people and resources mediated by a mobile device. this often-reported high level of networking creates shared, socially interactive environments so m-learners can readily communicate multi-modally with peers, teachers and other experts, and exchange information. learners consume, produce and exchange an array of ‘‘content’’, sharing information and artefacts across time and place. exchanged data files are often ‘‘just-in-existence’’, enhancing the immediacy of the m-learning experience. indeed, the spontaneity of these communications and the currency of exchanged data are made possible by the accessibility and expectation of users being reachable at any time. we used two sub-scales (conversation and data sharing) in our analysis of collaboration as shown in (table 3). table 2. two sub-scales of the authenticity construct used in our framework. scale sub-scale low activity is: high activity is: authenticity contextualisation (e.g. ctgv 1990) contrived realistic/relevant to learner situatedness (e.g. radinsky et al. 2001) simulated participatory/embedded in real community of practice m. kearney et al. 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 6. interrogating examples of m-learning in this section, we use our current framework to critique a range of sample mlearning scenarios taken from recent, refereed literature in this field. the purpose of this analysis is to demonstrate how the framework can highlight important aspects of learning and pedagogy, as distinct from other analyses in the literature that typically focus on technical issues surrounding the affordances of mobile devices. we applied the framework to 30 scenarios chosen from recent m-learning conferences and other publications to capture the most innovative, contemporary activities flagged in the current literature. activities were analysed using the six sub-scales to rate the critical features of these m-learning activities. when group members’ ratings differed, differences were resolved through group consensus. from this analysis, we selected six examples (see table 4). these examples illustrate the use of the framework in a range of contexts that exhibit different levels of the constructs. informed by our time-space conceptions (see section 3), we also describe our interpretation of the time-space organisation for each activity to provide additional insights into the m-learning context of each scenario. the critical features of these m-learning activities were rated according to our framework using the scales and sub-scales described in section 5. hence, table 5 (below) provides an indication of the extent to which features of our m-learning framework are exploited. (these ratings do not determine the qualities of the activities per se. activities are designed for different purposes. some features of m-learning may be appropriate in some circumstances but not others.). despite the rhetoric around m-learning virtually guaranteeing contextualised learning, very few of these scenarios rated highly in the scales for authenticity. most activities involved either some form of contrived context (e.g. the high school maths ‘‘apps’’ example) or activities that were merely providing a simulation of reality (such as the game � they were not participating in a real-life ‘‘governance’’ scenario). interestingly, the twitter example rated highly in authenticity, despite being in a formal professional learning setting. the activity was relevant (task, process etc.) to participants who chose to contribute to the twitter feed. delegates were certainly engaged directly in the professional community � including networking with colleagues who were not physically at the conference � and in this way, they were following a participation model of authenticity. indeed, the process of twittering has an increasing level of factual authenticity, as teachers begin to take up this activity as a normal everyday part of their professional networking practices. similarly surprising were the generally low ratings in the personalisation scales. an exception was the game design scenario that allowed learners to enjoy high degrees of customisation and self-control over the learning process. in contrast, like most table 3. two sub-scales of the collaboration construct used in our framework. scale sub-scale low activity is: high activity is: collaboration conversation (e.g. laurillard 2007; sharple s, taylor, and vavoula 2007) unconnected/solitary rich/involves deep, dynamic dialogue data sharing (e.g. traxler 2010) isolated/emphasis on content acquisition & transmission networked/includes learner-generated content research in learning technology citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 11 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 table 4. brief description of examples of m-learning scenarios source/scenario brief description a ebner (2009). use of twitter at a conference delegates used a twitter ‘‘back-channel’’ at a professional learning conference. during keynote presentations, delegates tweeted brief comments and questions in reaction to the speakers (or other tweeters). twitter posts were projected in a cascading fashion on a screen behind the speaker. from the perspective of the delegates in this formal conference venue (rather than the ‘‘lurkers’’ online), time was bounded by temporal parameters of the keynote speech. use of time-space: fixed/scheduled/formal. b tangney et al. (2010). geometry in the field a second-level maths class studying trigonometry who were working in teams of four students, using their smartphone’s ‘‘angle tool’’ to measure the heights of three structures in their school grounds. follow-up discussion of concepts occurred in the classroom. c tangney et al. (2010). fractions smartphone ‘‘apps’’ maths students studying fractions use a cuisenaire rod ‘‘app’’ on their smartphone, allowing them to manipulate coloured cuisenaire-like rods on screen within a virtual ‘‘unit space’’. small groups were organised by configuring the allocation of rods such that learners ‘‘trade’’ or ‘‘swap’’ with peers to solve problems. the teacher controls level of difficulty and to avoid students guessing, both time taken to complete various challenges and number of moves made are recorded by the applications. use of time-space (b and c above): these maths learning activities take place ‘‘out-of-class’’ but nevertheless in a structured, teacher-mediated setting and are organised within fixed schedule of school temporal parameters. use of time-space: fixed/scheduled/formal. d buhagiar, montebello, and camilleri (2010) augmented learning in an art museum learners use an augmented reality application on their mobile devices that reacts to a user’s location in the display area of an art gallery. students’ augmented view consists of virtual information on their device screens, superimposed over the ‘‘real’’ object they are focused on. this learning activity takes place in an informal, albeit bounded setting but organised to a relatively unfixed schedule and pacing. use of time-space: ‘‘in between’’ fixed/scheduled/formal and malleable/negotiated/ informal. e gwee, chee, and tan (2010) games-based m-learning year 9 social studies students studying governorship using the game statecraft x on their iphones. in this multi-layer strategy game, students and teachers get involved in multiple role-play scenarios that ‘‘move’’ from the real world to the game world. other integral activities included online forums, reflective blogs, debates and whole-class discussions. this learning activity takes place in a hybrid of formal (school) and informal settings and is organised to a relatively unfixed schedule and pacing. use of time-space: malleable/negotiated/informal. f ng’ambi et al. (2010) podcasts of lectures recording device was used for lecture casting to an existing institutional lms. students download resources, including podcasts to low cost playback devices (mp3 players and/or mobile phones). queries that arose from listening to podcasts were sent as sms to an anonymous q&a tool within the lms. this learning activity takes place in an informal setting to a relatively unfixed schedule and pacing. use of time-space: malleable/negotiated/informal. m. kearney et al. 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 school-based tasks restrained by curriculum and learning space constraints, the podcast and maths examples lacked agency and customisation. the analysis highlighted a marked difference in the nature of collaboration in these scenarios. scenarios such as the augmented reality application in the museum and the podcast activity were solitary activities that lacked social interactivity. only the twitter and games scenarios (a and e) rated highly on these scales, due to the large network involved in the conference ‘‘twitterverse’’ (including ‘‘lurker’’ colleagues in cyberspace) and the multi-player nature of the game. these two mlearning experiences also elicited in-depth conversations in supplementary activities. given the text constraints of twitter, the face-to-face and virtual conversations elicited from the twitter display became a crucial part of the experience, at least from the perspective of the delegates present ‘‘live’’ at the conference. the face-to-face and virtual group activities structured by the social studies teachers enhanced collaboration for students during and after the game scenario (e.g. blogs, classroom-based discussions and role-plays). indeed, a point of interest is the way that teachers used hybrid, integrated approaches (dillenbourg 2006) to enhance pedagogically ‘‘weaker aspects’’ of these m-learning scenarios. for example, the supplementary, post-activity face-to-face class discussions used by teachers in the maths and games examples elicits further learning conversations. use of the framework to interrogate m-learning scenarios identified a potential problem with collaboration and authenticity in augmented reality scenarios in informal settings such as museums and science centres. cook (2010) addresses the problem of collaboration in a similar location-based, augmented learning museum activity by supplementing this experience with students working in pairs. students were also asked to create a collaborative video blog emerging from their discussions in the museum. these activities initiated further collaboration through both the collaborative nature of the video blog production and also the stimulus it provided for further verbal and blog-based conversation. laine et al. (2009) describe a similar system called lieksamyst that enhances the authenticity of the museum experience by creating a role-play scenario whereby users interact with the museum artefacts (focused on finland’s history and culture). authentic, albeit fictional, characters are table 5. ratings for each of the sample m-learning scenarios (described above in table 4). scenarios scales sub-scales a use of twitter at a conference b geometry in the field c fractions phone ‘‘apps’’ d augmented learning in an art museum e gamesbased mlearning f podcasts of lectures personalisation agency low low low low high low customisation medium low low medium high low authenticity contextualisation high low low medium medium low situatedness high low low medium low low collaboration conversation high medium medium low high low data sharing high medium medium low high medium research in learning technology citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 13 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 introduced through the system through a ‘‘story-based, role-play game’’ and users interact with these characters through the device. although this is done through the technology in this example, role-play could be introduced as a face-to-face teaching strategy to enhance the authenticity of these museum-based m-learning scenarios. in summary, the framework provides a renewed focus on important aspects of socio-cultural theory for educators and researchers working in and examining m-learning contexts. use of our current framework as a lens to analyse more than 30 scenarios from recent m-learning literature suggested to us that it was finally scalable for examining and critiquing the pedagogical impact of a wide range of m-learning contexts. some scenarios typically promoted in a positive light in this recent literature base did not necessarily rate highly in our three scales. for example, while listening to instructional podcasts on public transport may sound novel in terms of the informal context and control of task pacing, under closer inspection it mimics a transmission pedagogy with its roots in didactic teaching traditions of formal learning settings. a second use of the framework is as a guide for practitioners to interrogate their own m-learning designs. we recently examined our students’ use of hand-held devices to complete class-based polls. although the task elicited rich learning conversations and involved some networking activity, it was a relatively contrived, structured task with minimal flexibility. the insights gained from the use of the framework contributed to development and enhancement of our practice. these contributions arise from using the framework to make the relationships among elements of the learning explicit. in this way, the framework also serves as a developmental tool by focusing on the essential constructs of learning from a sociocultural perspective. importantly, the framework itself will continue to be revised and refined to enable it to represent the many varied manifestations of m-learning. 7. conclusion a succinct framework highlighting distinctive, current socio-cultural features of mobile pedagogy has emerged from our design and development procedures, leveraged by our project activities. three constructs characterising the pedagogy of m-learning have emerged: authenticity, collaboration and personalisation. the authenticity feature highlights opportunities for contextualised, participatory, situated learning; the collaboration feature captures the often-reported conversational, connected aspects of m-learning while the personalisation feature has strong implications for ownership, agency and autonomous learning. how learners ultimately experience these distinctive characteristics is strongly influenced by the organisation of spatial and temporal aspects of the m-learning environment, including face-to-face and virtual teaching strategies. the framework discussed in this paper is by no means prescriptive � while such a pedagogical framework provides a spotlight to illuminate and examine mlearning experiences, account still needs to be taken of learners’ specific characteristics and needs, the environments in which the learning could potentially take place and the preferences and characteristics of teachers, including their epistemological beliefs. teacher roles and the learning task design are further crucial factors. this paper did not set out to examine causal links between the use of ‘‘time-space’’ and m-learning experiences. however, we do advocate a need for researchers to explore in more detail the time-space continuum and how it might be organised to optimise learning mediated by mobile technologies. central to the idea of m-learning is that learning contexts can be generated by students; occurring in different places m. kearney et al. 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 and at different times and not confined to formal learning settings in institutions. informal learning environments characterised by fluid geographical boundaries and malleable, socially negotiated time frames need further investigation with these goals in mind. a framework specifying critical attributes of m-learning experiences provides a useful lens for this research agenda. as mobile technologies develop, our challenge as educational researchers is to probe new pedagogical opportunities that honour principles of authentic, collaborative, personalised learning, drawing on wellresearched socio-cultural tenets. the framework presented in this paper will also assist practitioners’ understanding and analysis of unique teaching challenges in emerging m-learning environments and facilitate critical insights supporting their design of m-learning experiences and resources. references barab, s. a., squire, k. d. & dueber, w. (2000) ‘a co-evolutionary model for supporting the emergence of authenticity’, educational technology research and development, vol. 48, no. 2, pp. 37�62. brown, e. (2010) ‘introduction to location-based mobile learning’, in education in the wild: contextual and location-based mobile learning in action, ed. e. brown, learning sciences research institute, university of nottingham, uk, pp. 7�9. buhagiar, a., montebello, m. & camilleri, v. (2010) ‘mobile augmented reality in an arts museum’, in proceedings of mlearn2010: 10th world conference on mobile and contextual learning, eds m. montebello, v. camilleri & a. dingli, university of malta, valetta, pp. 395�397. cook, j. (2010) ‘mobile phones as mediating tools within augmented contexts for development’, in education in the wild: contextual and location-based mobile learning in action, ed. e. brown, learning sciences research institute, university of nottingham, uk, pp. 23� 26. ctgv (cognition and technology group at vanderbilt). (1990) ‘technology and the design of generative learning environments’, educational technology, vol. 31, no. 5, pp. 34�40. danaher, p., gururajan, r. & hafeez-baig, a. (2009) ‘transforming the practice of mobile learning: promoting pedagogical innovation through educational principles and strategies that work’, in innovative mobile learning: techniques and technologies, eds h. ryu & d. parsons, igi global, hershey, pp. 21�46. dillenbourg, p. (2006) ‘orchestrating integrated learning scenarios’, in proceedings of the 23rd annual conference of the australasian society for computers in learning in tertiary education: who’s learning? whose technology? eds l. markauskaite, p. goodyear & p. reimann, sydney university press, sydney, pp. 955. ebner, m. (2009) ‘introducing live microblogging: how single presentations can be enhanced by the mass’, journal of research in innovative teaching, vol. 2, no. 1, pp. 91�100. ghaye, a. & ghaye, k. (1998) teaching and learning through critically reflective practice. david fulton, london. gwee, s., chee, y. & tan, e. (2010) ‘game play time and learning outcomes of boys and girls in a social studies mobile game-based learning curriculum’, in proceedings of mlearn2010: 10th world conference on mobile and contextual learning, eds m. montebello, v. camilleri & a. dingli, maltuniversity of malta, valetta, pp. 16�23. kearney, m., schuck, s. & burden, k. (2010) ‘locating mobile learning in the third space’, in proceedings of mlearn2010: 10th world conference on mobile and contextual learning, eds m. montebello, v. camilleri & a. dingli, university of malta, valetta, pp. 108�115. kemmis, s. & mctaggart, r. (1988) the action research planner. deakin university press, victoria. klopfer, e., squire, k. & jenkins, h. (2002) ‘environmental detectives: pdas as a window into a virtual simulated world’, in proceedings of ieee international workshop on wireless and mobile technologies in education, ieee computer society, vaxjo, sweden: pp. 95�98. research in learning technology citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 15 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 koole, m. l. (2009) ‘a model for framing mobile learning’, in empowering learners and educators with mobile learning, ed m. ally, athabasca university press, athabasca, canada, pp. 25�47. laurillard, d. (2007) ‘pedagogical forms of mobile learning: framing research questions’, in mobile learning: towards a research agenda, ed n. pachler, wle centre, institute of education, london, 153�176. laine, t., et al. (2009) ‘characteristics of pervasive learning environments in museum contexts’, in proceedings of 8th world conference on mobile and contextual learning, eds d. metcalf, a. hamilton & c. graffeo, university of central florida, orlando, pp. 26�34. ling, r. & donner, j. (2009) mobile communications. polity, london. luckin, r. (2010) re-designing learning contexts. routledge, london. mcloughlin, c. & lee, m. (2008) ‘the 3 p’s of pedagogy for the networked society: personalization, participation, and productivity’, international journal of teaching and learning in higher education, vol. 20, no. 1, pp. 10�27. naismith, l., et al. (2004) ‘report 11: literature review of mobile technologies in learning’, futurelab series, [online] available at: http://archive.futurelab.org.uk/resources/publicationsreports-articles/literature-reviews?custom:filteryear�2004 ng’ambi, d., et al. (2010) ‘podcasting for mobile learners: using ubiquitous technologies to enhance learning in large classes’, in proceedings of mlearn2010: 10th world conference on mobile and contextual learning, eds m. montebello, v. camilleri & a. dingli, university of malta, valetta, pp. 256�262. pachler, n., cook, j. & bachmair, b. (2010) ‘appropriation of mobile cultural resources for learning’, international journal of mobile and blended learning, vol. 2, no. 1, pp. 1�21. pachler, n., bachmair, b. & cook, j. (2009) mobile learning: structures, agency, practices. springer, new york. parsons, d., ryu, h. & cranshaw, m. (2007) ‘a design requirements framework for mobile learning environments’, journal of computers, vol. 2, no. 4, pp. 1�8. pintrich, p. & schunk, d. (1996) motivation in education: theory, research, and application. merrill, upper saddle river, nj. radinsky, j. et al. (2001) ‘mutual benefit partnership: a curricular design for authenticity’, journal of curriculum studies, vol. 33, no. 4, pp. 405�430. schuck, s., et al. (2010) ‘mobagogy: mobile learning for a higher education community’, in proceedings of the iadis mobile learning, 2010 conference, eds i. sánchez & p. isaı́as, iadis press, porto, portugal, pp. 69�76. sharples, m., taylor, j. & vavoula, g. (2007) ‘a theory of learning for the mobile age’, in the sage handbook of e-learning research, eds r. andrews & c. haythornthwaite, sage, london, pp. 221�224. sundmaeker, h., et al. (2010) vision and challenges for realising the internet of things. cerpiot, european commission, luxembourg. tangney, b., et al. (2010) ‘mobimaths: an approach to utilising smartphones in teaching mathematics’, in proceedings of mlearn2010: 10th world conference on mobile and contextual learning, eds m. montebello, v. camilleri & a. dingli, university of malta, valetta, pp. 9�15. traxler, j. (2007) ‘current state of mobile learning’, international review on research in open and distance learning, vol. 8, no. 2, pp. 1�10. traxler, j. (2009) ‘learning in a mobile age’, international journal of mobile and blended learning, vol. 1, no. 1, pp. 1�12. traxler, j. (2010) ‘the ‘learner experience’ of mobiles, mobility and connectedness’, background paper to presentation elesig symposium: digital futures, 21 jan ilab, innovation works, university of reading, [online] available at: http://cloudworks.ac.uk/cloud/view/ 3472 trudge, j. (1990) ‘vygotsky, the zone of proximal development and peer collaboration: implications for classroom practice’, in vygotsky and education: instructional implications and applications of sociohistorical psychology, ed. l. moll. london: cambridge university press, new york, pp. 155�172. m. kearney et al. 16 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 http://archive.futurelab.org.uk/resources/publications-reports-articles/literature-reviews?custom:filteryear=2004 http://archive.futurelab.org.uk/resources/publications-reports-articles/literature-reviews?custom:filteryear=2004 http://archive.futurelab.org.uk/resources/publications-reports-articles/literature-reviews?custom:filteryear=2004 http://archive.futurelab.org.uk/resources/publications-reports-articles/literature-reviews?custom:filteryear=2004 http://archive.futurelab.org.uk/resources/publications-reports-articles/literature-reviews?custom:filteryear=2004 http://archive.futurelab.org.uk/resources/publications-reports-articles/literature-reviews?custom:filteryear=2004 http://cloudworks.ac.uk/cloud/view/3472 http://cloudworks.ac.uk/cloud/view/3472 http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 vavoula, g. & sharples, m. (2009) ‘meeting the challenges in evaluating mobile learning: a 3-level evaluation framework’, international journal of mobile and blended learning, vol. 1, no. 2, pp. 54�75. vygotsky, l. s. (1978) mind in society. mit press, cambridge, ma wertsch, j. v. (1991) voices of the mind: a socio-cultural approach to mediated action. harvard university press, cambridge, ma. willis, j. (2000) ‘the maturing of constructivist instructional design: some basic principles that can guide practice’, educational technology, vol. 40, no. 1, pp. 5�16. research in learning technology citation: research in learning technology 2012; 20: 14406 doi: 10.3402/rlt.v20i0.14406 17 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14406 doi:10.3402/rlt.v20i0/16153 worth it? findings from a study of how academics assess students’ web 2.0 activities kathleen gray a, *, jenny waycott a , rosemary clerehan b , margaret hamilton c , joan richardson d , judithe sheard e and celia thompson f a school of medicine and department of information systems, the university of melbourne, melbourne, victoria, australia; b faculty of medicine, nursing and health sciences, monash university, melbourne, victoria, australia; c school of computer science and it, rmit university, melbourne, victoria, australia; d school of business information technology, rmit university, melbourne, victoria, australia; e faculty of information technology, monash university, melbourne, victoria, australia; f school of languages and linguistics, the university of melbourne, melbourne, victoria, australia (received 15 march 2011; final version received 28 july 2011) educational commentators have offered many pedagogical rationales for using web 2.0 to support learning in higher education, and academics are being encouraged to find ways for their students to use social web technologies. questions arise as to the value of these activities compared to more conventional assignments, and whether implementing such changes to student assessment is worth the effort. we conducted a survey of academics’ assessment of students’ web 2.0 activities in australian universities and found that this form of assessment is being conducted by a small number of academics, in a range of fields of study, but mainly in humanities and social sciences, with varying kinds of intended and actual learning outcomes. blogging and wiki-writing predominate, low and medium-stakes assessment are most common, and different methods of marking and feedback are in use. qualitative feedback from the survey and follow-up interviews gave further insights into benefits and challenges of web 2.0 assessment in relation to pedagogy, policy and practice. it appears that academics’ conservative approaches to conducting assessment and their novice approaches to utilising social web technologies are factors which seriously limit realising the potential of web 2.0 for medium or high-stakes assessment. keywords: assessment; assignments; higher education; social web; web 2.0 introduction when introducing a new assignment to university students, academics routinely encounter the question, ‘what’s it worth?’ as students calculate how much of their overall result for the subject is at stake. as they gauge the benefits and risks of adopting a new method to improve teaching or assessment, academics themselves may also wonder whether it will be worth the effort. this paper reports findings from a survey and interviews with australian academics undertaking summative assessment of students’ web 2.0 activities. these data provide (page number not for citation purpose) *corresponding author. email: kgray@unimelb.edu.au research in learning technology vol. 20, 2012 rlt 2012. # 2012 k. gray et al.. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 16153 doi: 10.3402/rlt.v20i0.16153 http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 insights into this aspect of web 2.0 pedagogy, including the investment and benefit at various stages of the assessment process, from planning through to evaluation. drawing on these findings, we discuss academics’ experiences and decisionmaking regarding the educational worth, assessable value and relative merits of assessing student learning using web 2.0. our research raises important considerations for individual academics, and also has implications for educational developers and technologists and for institutional leaders and policy-makers. background web 2.0, or social web, technologies allow co-creation of web content by large numbers of people simultaneously and over extended time spans. content is open for participants to create, manipulate or propagate by commenting, editing, rating and tagging. communication may be facilitated via avatars, fans, friends, locating, profiling and syndication. thus a group may ‘create a collective work whose value far exceeds that provided by any of the individual participants’ (o’reilly and battelle 2009, p. 2). web 2.0 can leverage content and communication not only on the open web but also in closed environments, for instance in the corporate sector (schneckenberg 2009) and in university learning management systems (e.g. conde et al. 2011). the idea that web 2.0 offers pedagogical affordances for collaborative learning, mobile learning and other aspects of learning and teaching is now established (e.g. kuswara 2011) and some of these affordances may have been taken up in practice (conole and alevizou 2010). web 2.0 innovations in higher education have appeared in recent years (for example, franklin and van harmelen 2007, lee and mcloughlin 2010), however, it is still not clear to what extent students’ web 2.0 activities are formally assessed, or what affordances web 2.0 offers specifically for assessment. most published accounts of web 2.0 pedagogies in higher education have described only formative assessment (i.e. providing feedback on work in progress so that a student can improve it before it is marked or graded) or low stakes assessment (i.e. earning marks that make very little impact on the student’s overall standing), according to gray et al. (2010). most of this literature has overlooked the issue of summative, high stakes assessment, or has described it theoretically � for example, ‘working with a set of circumstances rather than trying to control or alter them’ (hughes 2009, p. 30); and ‘capturing the visible evidence of invisible learning’ (bass and eynon 2009, p. 4). reasons can be inferred to explain the paucity of published examples of web 2.0 assessment practices that actually demonstrate how summative assessment is being done and how it can be done well. academics may be hesitant to reveal that they have deployed such novel technology for summative assessment. using web 2.0 to assess student work can pose challenges for institutional values and policies (johnson, levine and smith 2009, p. 5), including for the institutional reputation that hinges on assessment, and for the academic integrity of assessment. using web 2.0 for summative, high stakes assessment could have an impact for good or ill on the integrity and reputation of the field of study or the university where it is used (ehlers 2009). over and above the pressures on traditional forms of assessment (sadler 2009, boud 2000), web 2.0 assessment can raise concerns about major accreditation frameworks, employers’ and other external stakeholders’ expectations, endorsement of learning resources and activities, and questions of intellectual property (collis and k. gray et al. 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 moonen 2008, pp. 100�101). the visible and open nature of content and communication in web 2.0 is at odds with an academic culture that has traditionally supported the making of decisions about assessment alone and in private (hattie 2009). as yet, the academic policies of the authors’ own universities, and possibly elsewhere, are silent on web 2.0’s assessment implications, such as the identification, ownership, safety, recording, privacy and preservation of student work. using web 2.0 for assessment has significant implications for academic integrity, a term holding complex and contested meanings for academic authorship, textual ownership and the practices of acknowledging influential ideas and their media of expression (thompson 2008). it challenges extant referencing and citation protocols to embrace a style of academic writing that captures the dynamic, dialogic and multi-authored nature of texts and artefacts generated in web 2.0 (gray et al. 2008). the need to improve the practice of assessment in higher education is widely recognised (e.g. boud 2000). assessment’s impact on student learning is well established (e.g. rust 2002), including technology-based assessment (e.g. kirkwood and price 2008). if done well, assessment can support and reinforce the intended student learning; that is, assessment can work ‘for’ or ‘as’ learning (earl 2003). web 2.0 may well be able to offer affordances for every stage of assessment, from designing and implementing it, through supporting and grading it, to evaluating its effectiveness (waycott et al. 2010a). web 2.0 might offer ways to make assessment more authentic, personalised, engaging or problem-oriented (elliott 2008). there is a growing body of case-by-case accounts evaluating various approaches to assessing web 2.0 activities (e.g. farmer, yue, and brooks 2008, ng and hussain 2009, lai and ng 2010). whitelock (2010) analyses eight case studies, and waycott and sheard (2011) present six more, illustrating web 2.0 uses in assessment ‘for’ learning, assessment of learning processes, and overcoming discontinuity between formative and summative assessment. however, to date, there has been no systematic investigation into the aspects that academics find worthwhile � or conversely, not worth it � about assessment of student web 2.0 activities. the prospect of opportunities for improving assessment but equally significant challenges, gave rise to a national australian study in 2009. this study aimed to paint a detailed picture of where, why and how academics were using web 2.0 for summative, as well as formative, assessment of student learning. it aimed to fill gaps in our knowledge and provide a foundation for developing a more widely shared understanding of the pedagogical worth and assessable value of students’ web 2.0 activities in higher education. method academics were invited to participate in an anonymous online survey and an optional semi-structured interview during the period august to october 2009. human research ethics approval for this project was granted by each of the partner universities. we advertised the survey in national learning and teaching newsletters, inviting participation from any academic who had set assignments where their students used web 2.0 technologies to do work for which they earned subject marks. we also issued invitations to academics who had recently published papers about their use of web 2.0 tools in higher education. web 2.0 technologies were defined as research in learning technology citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 3 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 any of the following: blogging and microblogging, audio and video podcasting, social bookmarking, social networking, wiki writing or virtual worlds; using any tools, such as twitter, flickr, wordpress, second life, youtube, delicious, facebook, or tools bundled in learning management systems (e.g. blackboard’s wiki tool). survey respondents answered questions about one assignment: one graded task that had been required in one subject or unit of study. a total of 44 questions covered the way the assignment was designed and implemented, the marking process, student results, policy and procedure considerations, and background information about the subject. multiple responses were possible to many of the questions. the survey included three open-ended questions about the assignment. a copy of the survey instrument and detailed reporting of these data are available at http://web2 assessmentresources.wikispaces.com/4.�further�resources. semi-structured telephone interviews conducted by one researcher explored interviewees’ assessment practices and their perspectives on the use of web 2.0 for assessment an interview protocol (appendix 1) was used, although the precise questions varied from case to case, allowing conversation to develop organically. sixty-four academics responded to the survey, of whom 53 completed all or most questions, and 22 were interviewed. most survey respondents (81%) identified their role as ‘subject coordinator’, for example, having overall responsibility for learning, teaching and assessment in the unit of study. many respondents (37%) had first used the web 2.0 assignment they described in 2009; 48% had begun the assignment between 2006 and 2008. all open-ended survey responses and interviews were transcribed and analysed first separately and then together by four of the researchers, using open coding. care was taken to account for both opportunities and challenges, i.e. positive and negative reports of experiences. interview and survey findings were then categorised by two of the researchers using the organising principle of the different stages in the assessment cycle. we could have categorised these data in many ways, following the literature; however, many of these categories would have failed to provide insights into summative assessment practices as well as into the assessment process as a whole. franklin and van harmelen (2007) have proposed recommendations for writing assessments involving web 2.0; the majority of these are formative assessment, and hatzipanagos and warburton (2009) too make a case for exploring the links between formative assessment and social software. bartlett-bragg (2008) has proposed a framework for designing web 2.0 assessment which highlights how much effort is needed in the initial planning stages, while bower (2008), bower, hedberg, and kuswara (2010), and conole et al. (2004) have published methods for matching learning tasks and assignments with learning technologies which also emphasise the degree of preparatory effort required. to describe the experience of working with web 2.0 assessment more fully than this, we chose to analyse the data using the organising principle of the different stages in the assessment cycle: not only designing the assignment, but also implementing and supporting the assignment with students, marking the assignment and giving summative feedback to students, and quality management of the assessment processes (rust, o’donovan, and price 2005, bloxham and boyd 2007). k. gray et al. 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://web2assessmentresources.wikispaces.com/4.+further+resources http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 findings descriptive statistics and comments from interviews and open-ended survey responses illustrate key benefits and challenges that academics weighed up at each stage of the assessment cycle. designing the assignment findings about the design of web 2.0 assignments give a mixed picture of the extent to which academics are attributing value to such approaches. the 62 responses to a question about the type of web 2.0 tool used showed that wiki writing (52%) and blogging (50%) were most commonly used. uses of social networking (27%), audio or video podcasting (26%), virtual world activities (19%) and social bookmarking (18%) for student assignments were reported less often. web 2.0 assignments were designed for a range of settings, including all year levels in 35 undergraduate degrees and 16 postgraduate coursework degrees, and in both small and large classes. while 23% of respondents were working with classes of 30 students or fewer, 25% had classes of between 31 and 100 students, and 25% had more than 100 students (the largest class reported had over 500 students). differences were apparent in the value of these assignments for different fields of study. in almost half the 50 responses to the ‘field of study’ question, the assignment was part of a course in humanities or social sciences (42%). almost a third of assignments came from each of information technology (32%) and education (30%). mentioned less often were health and medicine (16%), management and commerce (12%), creative arts, and natural and physical sciences (6% each), architecture, engineering, and law (2% each). web 2.0 assignments appeared equally useful for assessing various types of learning outcomes: generic or graduate skills or attributes (55%), specialised knowledge or skills required in a discipline or profession (44%) and foundation knowledge or skills preparatory to a discipline or profession (45%). these assignments were valued not only for demonstrating outcomes but also for facilitating learning processes, for instance: . optimising peer review: ‘they get their [peer] feedback on the assignment and . . . improve it based on the feedback’ (interviewee 6). . extending the learning community: ‘it really reaches out their communication beyond the boundaries of the classroom . . . it’s not just my view anymore; they can go and look at other people’s views as well’ (interviewee 8). . encouraging ongoing engagement: ‘to make sure that [students] did something continuously through the semester’. (interviewee 20) interviewees cited a number of challenges in getting the assessment design right. some of these were common to any assignment, for instance the challenge of making room for student creativity while also providing sufficient structure: the bottom third of the class had difficulty thinking about what to post on [i.e. blog about] when it was left completely up to them . . . if you give too much guidance are you research in learning technology citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 5 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 constraining the best students? if you don’t give enough guidance you’re leaving the weaker students to flounder. (interviewee 11) other challenges arose from needing to learn by trial and error how a web 2.0 assignment would work: we didn’t understand how the interaction would unfold, how the tool would behave, a lot of downtime and template restrictions. (interviewee 18) implementing the assignment findings about how academics are introducing and supporting web 2.0 assignments with their students suggest that it is not a simple matter to do this well. compared to more conventional assignments, the extra efforts required to get students to work effectively with web 2.0 may detract from the perceived worth of the activity. students commenced work on most assignments either at the start of the study period (55%) or in the middle of the study period (34%). the time they were expected to spend on the assignment was most often 11�20 hours (33%), followed by 1�10 hours (27%), 21�30 hours (17%) and 31�40 hours (11%). students were allowed more than a month to complete most assignments (66%) or at least several weeks in a number of other cases (22%). students were expected to do the assignment using their own it facilities (88%) or it facilities provided by their university (78%). they were expected to work on it in various locations including: . on campus in class (25%). . on campus but out of class (39%). . off campus while undertaking fieldwork or workplace learning (11%). . off campus elsewhere than fieldwork or workplace (81%). academics interviewed spoke at length about their need to develop new strategies to help students interact online: ‘i would give them an example at the beginning in the first week or second week on how this would work and the sorts of comments that i would be expecting’. (interviewee 15) several interviewees described the work involved in encouraging students to draw on their personal experience to generate an online identity: ‘there’s a process that goes into them finding their different voices, how to share appropriately, how to write with authority’. (interviewee 8) academics reported coaching students in co-creation and collaboration: ‘i said it had to be [500 words] that you had to add and then you had to value-add to the content. so we talked about what we meant by that’. (interviewee 22) providing adequate support for students posed new workload challenges for some staff: [there is a lot of] work involved in setting up [the blog assignment] and making sure all the students know how to do it. if you ask them to write an essay they just go off and write it, you don’t have to spend the first three weeks of the course teaching them about essays. (interviewee 11) it’s hard yakka [work] for the teachers . . . because we often have to be on [online] almost all hours at times . . . you tend to do that to give the support. (interviewee 1) k. gray et al. 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 marking and giving feedback on the assignment findings about marking and giving feedback on web 2.0 assignments indicate that at this stage of the assessment cycle most academics are not taking up the open and collaborative approaches to feedback and assessment that the social web affords, but are exercising their individual authority and discretion as assessors. the value of web 2.0 assignments as a proportion of a student’s overall mark in the unit of study ranged from very little to 100%, with most assignments worth up to 50%, as shown in table 1. most students received marks plus other feedback, including confirmatory comments (67%), explanatory comments (64%), and corrective comments (61%). the majority of web 2.0 assignments were marked by one staff member (63%), with just over a quarter marked by more than one (27%). academics gave various responses to a free-text question asking them to identify strategies they used to brief other markers, including students who had to mark each other’s work, as shown in table 2. peer-assessment or self-assessment was factored into marking in only 14% and 11% of cases respectively. establishing a framework for evaluating the work that students produced was not straightforward. one interviewee found it difficult to set criteria that were appropriate to different levels of study, noting that the differences were ‘sort of formally in place with essays, you know, the number of words you write as a masters student is different to a grad dip but when it comes to creating [web 2.0 content], perhaps it’s a bit less clear’ (interviewee 7). in some assignments the diversity of content types that students could produce created another challenge for marking: ‘[students] are producing very different things, so i suppose the criteria need to capture that in some way’ (survey response). another survey respondent noted that the different style of writing in a web 2.0 assignment made marking difficult: ‘because the content does not have to be of an academic standard it can be hard (to) assess, e.g. marking sloppy or lazy prose’. in some instances calculating a final mark for individuals’ work was complicated by the need to assess how it had been influenced by class members’ running online exchange of feedback and interaction. twenty-three percent of respondents said that equal marks were allocated to everyone in a group project. interviewees reported that they had developed new marking techniques to respond to web 2.0 affordances in a variety of ways: we have a complete history of everything they do . . . so we can actually look in and see what this particular group has excelled at, which one gives us a wide variety of results so that we can separate the good students from the others. (interviewee 13) the assessor is not assessing a written document, they’re assessing a page which . . . is a whole labyrinth of choices and connections, so they’ve got to actually work their way through . . . decision paths and decision tree and things like that. (interviewee 10) table 1. web 2.0 assignment as a proportion of students’ overall mark. proportion of overall mark (%) percentage of responses (n�53) 1�20 34 21�50 45 51�80 10 81�100 11 research in learning technology citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 7 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 in particular, there were many instances where staff found merit in being able to combine formative and summative assessment, for example: i leave comments on the blogs as we go through but usually engaging with the content, not sort of marking it, but then i can add those sorts of comments against the rubric when i assess them. (interviewee 11) sometimes assessing process as well as product was problematic, for instance when marking co-created wiki content: how do you mark assignments when students can change/overwrite each other’s work! many students who contributed early, found that their work was completely lost. how do you manage this process of overwriting and still contributing to the same content? (survey response) some academics marked students’ work in stages (34%). however the multilateral peer-to-peer interactions and repeated student-to-staff interactions during a cumulative web 2.0 assignment could put pressure on staff responsible for marking large numbers of students: last year i had them doing a blog every week . . . the marking was killing me because you’ve got to mark them and get them back in a week. and then you’ve got four or five classes, that’s six or seven hours work a week in marking. and then you’ve got [other] assignments and exams. (interviewee 12) managing the quality of assessment processes in terms of overall assessment management, using web 2.0 for student assignments holds clear value for some academics but a level of risk for others. the survey addressed academic administration and quality management by asking participants to respond to a series of statements about their assignment in relation to major assessment policy issues. these were distilled from a review of australian universities’ table 2. measures taken to prepare assessors of student web 2.0 assignments. steps taken and illustrative quotes number of responses criteria/criterion referenced/graded marking sheet: ‘criteria and marking sheets agreed before term start, adhered to by all involved staff who meet to agree detail’. 8 rubric: ‘rubric provided to all prior to commencement of these units. discussion with students as they commence marking’. 7 experienced lecturers/tutors do the marking: ‘only experienced markers mark these assessments and they liaise before marking to ensure criteria are commonly understood’. 4 meeting: ‘meeting held between the tutors and course coordinators to standardise marking’. 3 peer review/panel review: ‘students engage in online peer review of their draft write ups on [a] peer review system’. 2 examples provided: ‘the lecturer provides their own answers to the questions posed, and discusses them with the markers before they mark the e-journals’. 2 inspection: ‘throughout the marking period the markers keep an eye on all the students’ blogs and comments made by other staff ’. 1 k. gray et al. 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 publicly available assessment policy and procedure documents. table 3 shows that most academics were confident that they were managing their assignment appropriately in most respects. however there were noteworthy levels of non-alignment or uncertainty in matters such as offering supplementary assessment, keeping students’ marked work on file, safeguarding students’ identity and privacy online, and recording students’ learning in ways that would allow it to be tendered as evidence of meeting prerequisites for further study (e.g. for recognition of prior learning). several interviewees described how web 2.0 affordances supported the quality management of assessment in terms of monitoring progress, reinforcing academic integrity, and protecting students’ moral rights. for example: students start off being able to see their own posting. so they can’t copy off one another. and then later on they see the other postings but then they can’t edit their own. so it truly is their own work because that first bit is done in their creative sense. (interviewee 12) one of the reasons why i chose to go with the blogs and make it an informal writing exercise (was) because then i could get a sense of their voice, get a sense of consistency as well . . . so when it comes to marking the formal assessment tasks that’s another plagiarism check. (interviewee 21) academics also noted key challenges for quality management of assessment, such as ‘being able to guarantee ongoing access to materials hosted on sites outside the university’ (survey response) and ‘all those issues of confidentiality and copyright’ (interviewee 20). as interviewee 13 stressed: i tell the students over and over again that it is on the world wide web, it’s not associated with the university, be careful what you put up there, make sure you are comfortable with this. discussion these findings show that academics have found many features that are worthwhile, and many ways to see value in designing, conducting, marking and managing student web 2.0 assignments. they report a range of challenges and a few real risks: some familiar to anyone changing their assessment practices but others unique to the assessment of student web 2.0 activities. it is noteworthy that a relatively small number of academics responded to the survey and many of them had used the web 2.0 assignment for the first time that year. we may speculate that other academics did not respond either because their use of this form of assessment was less well developed, or because they did not use this form of assessment at all. although the use of student web 2.0 activity in university education is rising perceptibly, our findings show that low and medium-stakes assignments are most common. this is consistent with our previous research, which indicated that web 2.0 assignments are offered chiefly for optional enrichment or as preparation for a more conventional written assignment or examination (gray et al. 2010). academics in our study, the majority of whom were from humanities and social sciences disciplines, were clearly finding value in the opportunities that web 2.0 afforded for assessment of openly published and collaborative student work. as these academics would appear to represent the prime essay-writing disciplines, it could be proposed that written text remains the most assessable of the many web 2.0 forms of activity. according with this, and consistent with the types of innovation most commonly reported in the literature, we found that web 2.0 assignments most research in learning technology citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 9 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 table 3. reported alignment with academic policies and procedures in assessment of student web 2.0 activities. policy areas strongly agree agree neither agree nor disagree disagree strongly disagree not sure students are provided with an extension of the due date for special consideration reasons. 52 37 0 4 4 4 students are provided with access to it services or equipment to complete this assignment. 50 27 14 6 2 2 guidelines on appropriate conduct and safeguards against inappropriate conduct in use of it facilities and services are in place. 49 37 12 2 0 0 copies of students’ marked work are available if there is a need to deal with appeals/ complaints. 48 35 8 2 4 4 students are provided with timely feedback on marked work for this assignment. 48 31 8 6 4 4 this assignment encourages academic honesty and integrity. 40 39 10 6 2 4 students’ moral right and copyright in work they produce are protected. 39 35 10 2 4 12 students whose work shows evidence of cheating or misconduct are formally disciplined. 37 37 12 2 0 14 copies of students’ marked work are kept on file for an agreed period of time. 35 27 6 10 8 15 this assignment provides for equitable assessment for students with a disability. 31 42 14 4 2 8 supplementary assessment is possible. 29 27 14 12 14 6 students’ identity and privacy in online environments are safeguarded. 23 37 21 12 4 4 this assignment allows for exemption or credit in recognition of prior learning. 8 4 4 23 45 17 k . g ra y e t a l. 1 0 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) c ita tio n : r e s e a rc h in l e a rn in g te c h n o lo g y 2 0 1 2 ; 2 0 : 1 6 1 5 3 d o i: 1 0 .3 4 0 2 /rlt.v 2 0 i0 .1 6 1 5 3 http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 frequently took the form of blogging and wiki writing. the academics still identified challenges with these affordances of open publishing and collaborative writing. even though they were willing to put effort into coaching students in the use of these new forms, they remained concerned, for instance, that students’ text-building efforts were not ‘academic’ enough. academic staff seemed to be trying to deal with these issues as isolated practitioners, innovating through trial and error. interestingly, most respondents were subject coordinators, potentially responsible for teams of tutors and markers, yet they volunteered little evidence of open or collaborative marking practices, and there was little suggestion of novel methods used to evaluate the effectiveness of their assessment practices. there would seem to be substantial room for more creative uses of social media technologies in assessment to automate or streamline marking; to devise new forms of feedback; and to incorporate multilateral assessment, making use of peers or even extending whitelock’s (2010, p. 322) concept of a ‘community’ of assessors beyond the classroom. the use of web 2.0 technologies may prove too disruptive for conventional assessment practices, shaking the foundations of two hitherto solid pillars of contemporary assessment thinking. the division between formative and summative assessment and the straightforward alignment of assessment tasks with objectives may ultimately be re-engineered by web 2.0 assessment forms. in the web 2.0 environment, formative and summative assessment can appear like a double helix � and perhaps become indistinguishable. as the process of content creation is transparent for the duration of the task and after, this makes it possible (and arguably raises student expectations) that summative assessment will attach value to the processes that students use to create their assessable products. it may be necessary to revisit constructive alignment between the learning activity required and the learning outcome intended (collis and moonen 2008). even when academics undertake solely summative assessment in these assignments, the affordances of web 2.0 mean that some formative feedback is most likely inbuilt. thus, the work of arriving at a student’s final mark can � and arguably must � explain how the student’s response to feedback has added value and accrued marks along the way to the later summative grade. a third pillar of assessment convention therefore may require re-engineering here: as well as the students’ content creation being made highly visible, the often very private evaluative work of giving formative and summative feedback and the detail of the actual feedback given can become more transparent than ever before, not only for a student cohort, but also for academic colleagues and others. this represents a shift from academics conducting assessment in a ‘black box’ (hattie 2009). thus, in using web 2.0 for assessing student learning, questions about which (and whose) ‘achievement’ is to be assessed may, at least in the short term, become more contested. is it the student who has been prepared to take individual risks (which may not, in the end, have been totally successful), or the student who has collaborated wholeheartedly with their peers? is it the student who has advanced in their progress towards a professional identity, or the student who has shown particular talent in using web 2.0 tools? and what of the academics? is it the one who has introduced a specific web 2.0 assessment form for learning a particular technique or concept, or the one who has redesigned assignments to be more engaging for students? is it the academic who has found a method to facilitate staged assessment, or the one who has devised a method for monitoring group work? an academic without a sound rationale for assessing students’ web 2.0 activities will struggle to research in learning technology citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 11 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 justify the added effort flowing from the assessment design � effort which may involve addressing critical questions about the allocation of extra attention and support for the duration of the task; about the calculation of results for students’ learning outcomes; and about the propriety of interactions between university students and others who are active in the wider social web. a limitation of this study is that it offers an overview of academics’ self-reported practices only. we cannot be certain whether there were others who chose not to respond and what their reasons might have been. still needed are in-depth comparative studies of individual academics’ decisions in using web 2.0 for assessment, as well as a fuller understanding of university students’ perspectives on these kinds of assessments. accordingly, a follow-on study from the survey reported here involves semester-long observation and document analysis in 17 diverse subject learning and teaching case studies of web 2.0 assessment, as well as interviews and focus groups with participating staff and students (waycott et al. 2010b). conclusion most advocates of using student web 2.0 activities in higher education offer little detailed advice on how to conduct assessment which comes to grips with the unique features of web 2.0, its difference from previous forms of student writing and staff marking or its academic administration. this study raises important considerations for teaching academics, for educational developers and technologists and for learning and teaching leaders and policy-makers. for individuals or groups who have heard enthusiastic commentaries about the assessment potential of web 2.0. our research offers new insights, derived from broadly based empirical data, into the chequered realities of current practice. it may be inadvisable for intending innovators to follow the example of our study participants in proceeding conservatively and applying web 2.0 affordances selectively to suit their pre-existing pedagogical context. nor does is seem to be a good idea to work on these pedagogical innovations as a sole practitioner of web 2.0 assessment in one’s academic department or discipline. in terms of educational efficiency and effectiveness, incremental and individualistic uptake of web 2.0 may not achieve sufficient scope and scale to deliver substantial improvement of academic assessment, and so may not be worth it in the face of workload and policy concerns. institutionally and collegially concerted shifts in this practice have more likelihood of adding the anticipated value to higher education. so far, although reform of student assessment in higher education is called for � ‘the revolution will encompass ‘feedback from assessment’ and the development of visible learning and visible teaching’ (hattie 2009) � and despite the potential of web 2.0 technologies to facilitate such change, web 2.0 cannot be seen to be transforming university assessment. on the contrary, its affordances are being appropriated, in some cases, to perpetuate conventional, invisible university assessment practices (even to the extent of constraining it inside online learning management systems). this trend is at odds with the view of some commentators that its very openness is what makes the social web most valuable. staff and students who are ideologically inclined towards this openness are increasingly likely to seek to disrupt the status quo of assessment standards and policy with ‘challenges concerning collaboration, evaluation, and the general aim and status of the material produced by students’ (dohn 2009). k. gray et al. 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 for those whose role is to encourage and support innovation in higher education, it is clear that the majority of teaching staff and students too will need broadly-based, well-informed advice and support to make the most of the assessment potential of web 2.0. there is much work to be done before we can be confident that using web 2.0 for assessment is safe, fair, engaging and worthwhile for students and staff. acknowledgements we are grateful to the 64 study participants from universities throughout australia who were willing to share their experiences with the wider academic community; and to anonymous reviewers for constructive feedback on this paper. support for this publication was provided by the australian learning and teaching council (altc) ltd., an initiative of the australian government department of education, employment and workplace relations. the views expressed in this paper do not necessarily reflect the views of the altc. references bartlett-bragg, a. (2008) ‘reframing assessment: using social software to collect and organise learning’, proceedings of world conference on educational multimedia, hypermedia and telecommunications 2008, chesapeake, va, usa. chesapeake, va: aace. pp. 4020�4024. bass, r. & eynon, b. (2009) ‘capturing the visible evidence of invisible learning (introduction and synthesis of findings)’, academic commons, (january), pp. 4�29. bloxham, s. & boyd, p. (2007) developing effective assessment in higher education, open university press, milton keynes. boud, d. (2000) ‘sustainable assessment: rethinking assessment for the learning society’, studies in continuing education, vol. 22, pp. 151�167. bower, m. (2008) ‘affordance analysis: matching learning tasks with learning technologies’, educational media international, vol. 45, pp. 3�15. bower, m., hedberg, j. g. & kuswara, a. (2010) ‘a framework for web 2.0 learning design’, educational media international, vol. 47, pp. 177�198. collis, b. & moonen, j. (2008) ‘web 2.0 tools and processes in higher education: quality perspectives’, educational media international, vol. 45, pp. 93�106. conde, m. et al. (2011) ‘web services layer for moodle 2.0: a new area of possibilities in web based learning’, international journal of technology enhanced learning, vol. 3, pp. 308� 321. conole, g. et al. (2004) ‘mapping pedagogy and tools for effective learning design’, computers and education, vol. 43, pp. 17�33. conole, g. & alevizou, p. (2010) a literature review of the use of web 2.0 tools in higher education. a report commissioned by the higher education academy, open university, milton keynes. dohn, n. (2009) ‘web 2.0: inherent tensions and evident challenges for education’, international journal of computer-supported collaborative learning, vol. 4, pp. 343�363. earl, l. (2003) assessment as learning: using classroom assessment to maximise student learning, corwin press, thousand oaks, ca. ehlers, u. d. (2009) ‘web 2.0 � e-learning 2.0 � quality 2.0? quality for new learning cultures’, quality assurance in education, vol. 17, pp. 296�314. elliott, b. (2008) ‘assessment 2.0’, international journal of emerging technologies in learning, vol. 3, pp. 66�70. farmer, b., yue, a. & brooks, c. (2008) ‘using blogging for higher order learning in large cohort university teaching: a case study’, australasian journal of educational technology, vol. 24, pp. 123�36. franklin, t. & van harmelen, m. (2007) web 2.0 for content for learning and teaching in higher education, jisc, bristol. gray, k. et al. (2008) ‘web 2.0 authorship: issues of referencing and citation for academic integrity’, the internet and higher education, vol. 11, pp. 112�118. research in learning technology citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 13 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 gray, k. et al. (2010) ‘students as web 2.0 authors: implications for assessment design and conduct’, australasian journal of educational technology, vol. 26, pp. 105�122. hattie, j. (2009) ‘the black box of tertiary assessment: an impending revolution’, in tertiary assessment & higher education student outcomes: policy, practice & research, ako aotearoa, wellington, new zealand, pp. 259�275. hatzipanagos, s. & warburton, s. (2009) ‘feedback as dialogue: exploring the links between formative assessment and social software in distance learning’, learning, media and technology, vol. 34, pp. 45�59. hughes, a. (2009) higher education in a web 2.0 world: report of an independent committee of inquiry into the impact on higher education of students, widespread use of web 2.0 technologies, becta/department of employment and learning/higher education academy and others, uk. johnson, l., levine, a. & smith, r. (2009) the 2009 horizon report, new media consortium, austin, tx. kirkwood, a. & price, l. (2008) ‘assessment and student learning: a fundamental relationship and the role of information and communication technologies’, open learning: the journal of open and distance learning, vol. 23, pp. 5�16. kuswara, a. (2011) ‘realising the potential of web 2.0 for collaborative learning using affordances’, journal of universal computer science, vol. 17, pp. 311�331. lai, y. c. & ng, e. m. w. (2010) ‘a case study of infusing web 2.0 tools for blended learning: virtual presentations as an alternative means of assessment’, in comparative blended learning practices and environments, ed. e. m. w. ng, information science reference, hershey, pa. lee, m. & mcloughlin, c., eds. (2010) web 2.0-based-e-learning: applying social informatics for tertiary teaching, igi global, hershey, pa. ng, h. z. & huzzain, r. m. r. (2009) ‘empowering learners as the owners of feedback while youtube-ing’, interactive technology and smart education, vol. 6, pp. 274�285. o’reilly, t. & battelle, j. (2009) web squared: web 2.0 five years on. special report [white paper] for the web 2.0 summit, san francisco, ca, [online] available at: http:// assets.en.oreilly.com/1/event/28/web2009_websquared-whitepaper.pdf rust, c. (2002) ‘the impact of assessment on student learning’, active learning in higher education, vol. 3, pp. 145�158. rust, c., o’donovan, b. & price, m. (2005) ‘a social constructivist assessment process model: how the research literature shows us this could be best practice’, assessment and evaluation in higher education, vol. 30, pp. 231�240. sadler, d. r. (2009) ‘fidelity as a precondition for integrity in grading academic achievement’, assessment and evaluation in higher education, vol. 34, pp. 1�17. schneckenberg, d. (2009) ‘web 2.0 and the empowerment of the knowledge worker’, journal of knowledge management, vol. 13, pp. 509�520. thompson, c. (2008), ‘dialogism and social computing: academic authorship in cyberspace’, in changing university learning and teaching: engaging and mobilising leadership, quality and technology, eds. j. mcconachie et al., post pressed, flaxton, australia, pp. 359�374. waycott, j. et al. (2010a) ‘implications for academic integrity of using web 2.0 for teaching, learning and assessment in higher education’, international journal for educational integrity, vol. 6, pp. 8�18. waycott, j., et al., (2010b) ‘transforming assessment in higher education: a participatory approach to the development of a good practice framework for assessing student learning through social web technologies’, in curriculum, technology & transformation for an unknown future, eds. c. h. steel, et al., proceedings ascilite sydney 2010, pp. 1040�1050. waycott, j. & sheard, j. (2011) ‘editorial 27(5): preface to the special issue � assessing students’ web 2.0 activities in higher education’, australasian journal of educational technology, vol. 27, pp. iii�ix. whitelock, d. (2010) ‘activating assessment for learning: are we on the way with web 2.0?’ in web 2.0-based-e-learning: applying social informatics for tertiary teaching, eds. m. lee, & c. mcloughlin, igi global, hershey, pa, pp. 319�342. k. gray et al. 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 http://assets.en.oreilly.com/1/event/28/web2009_websquared-whitepaper.pdf http://assets.en.oreilly.com/1/event/28/web2009_websquared-whitepaper.pdf http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 appendix 1. interview protocol hrec 0932125 thank you very much for completing the online survey. . can you please give me a brief description of the assignment you described in the survey? (technologies used, what students were expected to do, etc.) . what distinguishing features/particular functionalities of web 2.0 were most important for this assignment? . what was the reason/trigger/impetus/inspiration for introducing this assignment? . can you tell me a bit about the main benefits or worthwhile aspects of this assignment (e.g. pedagogical benefits, administrative benefits, etc)? . can you tell me a bit about the main challenges involved in running and assessing this assignment? . if you were running this assignment again, what specific changes would you like to make to the way this assignment works? . what advice would you give to another academic who was thinking of using this approach to assessment with his/her students? . what do students think/say about this assignment? . what do colleagues think/say about this approach to assessment? . do you think this assignment would work with other teachers or student groups? . we are trying to get an understanding of issues relating to academic standards, practices and reporting when using web 2.0 technologies in assessment. . . . can you tell me more about the process of marking this assignment? . have any issues arisen with this assignment in terms of student copyright and/or plagiarism? . have there been any issues relating to student privacy and identity online? . what happens to students’ work after the assignment is completed (e.g. is it publicly available on the web)? . do you have any other comments about factors relating to academic standards, practices and reporting when running assignments like this one? research in learning technology citation: research in learning technology 2012; 20: 16153 doi: 10.3402/rlt.v20i0.16153 15 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/16153 research in learning technology vol. 19, no. 1, march 2011, 19–38 issn 0968-7769 print/issn 1741-1629 online © 2011 association for learning technology doi: 10.1080/09687769.2010.548506 http://www.informaworld.com how do e-book readers enhance learning opportunities for distance work-based learners? ming niea*, alejandro armellinia, gabi witthausa and kelly barklambb abeyond distance research alliance, university of leicester, leicester, uk; bzeal solutions ltd, london, uk taylor and franciscalt_a_548506.sgm (received 17 september 2010; final version received 9 december 2010) 10.1080/09687769.2010.548506research in learning technology0968-7769 (print)/1741-1629 (online)original article2011taylor & francis191000000march 2011dr mingniemn79@le.ac.uk we report on the incorporation of e-book readers into the delivery of two distancetaught master’s programmes in occupational psychology (op) and one in education at the university of leicester, uk. the programmes attract work-based practitioners in op and teaching english to speakers of other languages, respectively. challenges in curriculum delivery included the need for more flexibility in the curricula, better access to essential readings and maximising the benefit of learners’ limited study time. as part of a suite of pilot changes to curriculum design and delivery, 28 sony prs-505™ e-book readers were preloaded with course materials and sent out to students. the evidence suggests that the students’ learning experiences improved as a result of four key benefits afforded by the e-book readers: enhanced flexibility in curriculum delivery to accommodate the mobile lifestyle of our learners, improved efficiency in the use of study time, especially short breaks during the working day, new strategies for reading course materials and cost. we discuss the opportunities and limitations associated with the e-book readers used and the challenges encountered in the study. keywords: e-book reader; distance learning; work-based learning; m-learning; flexibility; innovation introduction this paper reports a study of the incorporation of e-book readers into the curriculum delivery of two distance-taught master’s programmes in occupational psychology (op) and one in education at the university of leicester, uk. we discuss how e-book readers were used on work-based distance learning courses at the university of leicester to address some of the challenges faced by learners on the three programmes. an e-book reader is an electronic device designed primarily for reading digital files such as electronic books (e-books). it is smaller, lighter and cheaper than a small laptop computer or netbook and uses black-and-white e-ink technology, as opposed to an lcd or backlit screen. storage capacity and functionalities such as wireless connectivity and annotation features vary between models. the model used in this research was the sony prs-505™. other examples of e-book readers include the amazon kindle™ and the sony pocket™. *corresponding author. email: mn79@leicester.ac.uk 20 m. nie et al. in the light of earlier research, we evaluate the potential of e-book readers for work-based distance learners. we show how the e-book readers were integrated into the three curricula and describe the process that led to the production and loading of materials. we present our research methods and discuss our key findings in terms of the potential and limitations of e-book readers as a learning technology for delivering content for work-based distance learning programmes at master’s level. background mobile technology for work-based learning work-based learning plays an important role in meeting the demand for increasing the employability of university graduates (clamp and warr 2002; hills et al. 2003). most higher education institutions already have work-based learners as a vital segment of their target audience in their teaching and learning agendas. work-based learning also acts as a driver for innovation in the higher education system (nixon et al. 2006). evidence suggests that work-based learning can be more resource-intensive than other modes of learning (nixon et al. 2006). to provide cost-effective work-based learning solutions, higher education institutions need a more flexible approach (gallacher and reeve 2002) and more innovation in using new technologies in delivery (brennan 2005). this will enable students to have greater control over when and where their learning takes place and build their learning around other work and life commitments (fagerberg and rekkedal 2004). work-based learning has been defined as learning at higher education level carried out for work, through work and at work (gray 2001). in this article, we use the term work-based learners to refer to students registered on formal higher education programmes of study who meet one or more of the following criteria: ● the programme of study, which they follow while in employment, is directly related to and with immediate application to their jobs (andresen, boud, and cohen 1999). ● they do some or all of their learning while performing work-related activities, including travel. ● they do some or all of their learning at the workplace. work-based learners typically lead busy and demanding lives. they travel a lot and struggle to find enough time for studying. their mobile and time-poor lifestyles lead them to seek more flexibility and freedom to choose where and when to study. they require solutions that support learning on the move. the innovative applications of mobile technologies, including the use of pocket pcs, personal digital assistants and mobile phones have proved effective in extending and increasing the flexibility of learning programmes and the mobility of distance learners (rekkedal et al. 2005). access to e-libraries and other essential digital learning resources by learners in work placements is another challenge faced by work-based learners (walton, childs, and blenkinsopp 2005). some higher education institutions are experimenting with ways to use the affordances of mobile technologies to give these students the same level and quality of access as a computer on the university campus. at athabasca university, for example, a digital reading room was established for the delivery of course materials to personal digital assistants, mobile phones and other mobile devices research in learning technology 21 to increase the flexibility of accessing learning resources for mobile distance learners (mcgreal et al. 2005). the potential for e-books to support distance learning has also been investigated. shiratuddin et al. (2003) found that the use of e-books could improve access to teaching and learning materials for distance learners. a more recent study investigating the usability and usefulness of e-books in an m-learning environment indicated the following advantages for student learning: increasing access (e.g. more readings, multimedia and portable resources), enabling remote access, and optimising reading time (lam, lam, and mcnaught 2010). whalley (2010) argues that there will be an increasing use of e-books by students on and off campus, given the financial pressures faced by all uk higher education institutions and the tuition fees paid by students. mobile devices such as e-book readers might offer affordable solutions for course content delivery. this article reports on one of the possible ways in which e-book readers can be integrated into curriculum delivery at master’s level. context for the study the university of leicester is a sector leader in postgraduate distance education. since the early 1990s, over 18,000 students have earned degrees by studying at a distance. the university currently has over 7000 students on 40 distance learning programmes offered by 13 departments. over 50% of the distance students are outside the european union, with large numbers in africa, the caribbean and the united arab emirates. the vast majority of distance students are enrolled on work-based master’s programmes related to their professional lives. it is the university’s mission to continue investing in and expanding the delivery of its work-based distance learning programmes and to enhance the experience of these learners through learning technologies. print is still the main delivery technology in most of leicester’s distance learning programmes. however, an increasing number of departments have opted for appropriate blends of online and print delivery. six programmes are currently delivered solely or primarily online. this figure is likely to increase in the next 12 months. two master’s programmes in op and one in applied linguistics and teaching english to speakers of other languages (tesol) were targeted for specific innovations. almost all students on the three programmes are in employment. if successful, the innovations would be transferred to other programmes. the school of education at the university of leicester has been delivering a master’s programme in applied linguistics and tesol by distance learning since 1995. the programme is aimed at graduate teachers with at least two years of englishlanguage teaching experience, who want to further their academic and professional development. due to the nature of their work, most of these students are highly mobile, as they tend to travel internationally. the programme attracts about 90 parttime students per year, and can be completed within 2.5–5 years. it involves three 30credit modules, two option modules (30 credits in total) and a 60-credit dissertation. in september 2008, the education team began to transfer the delivery of course materials to the university’s blackboard virtual learning environment (vle) and gradually stopped sending printed materials to students. the school of psychology at the university of leicester has been delivering two distance programmes since 2000: the msc in occupational psychology and the 22 m. nie et al. diploma/msc in the psychology of work. between them, these programmes attract about 50 part-time students per year, and can be completed over two years. each programme involves six 20-credit modules and a 60-credit dissertation. in 2006, the op team began to use the university’s blackboard vle beyond a document repository to enhance the delivery of these programmes. online activities and tutor-moderated discussion forums provided guidance on module content and assignments and support to learners at all stages of their dissertations. course materials began to have their webbased versions and active links to course readings. blackboard became the central hub for all student resources. since 2007, students have submitted their assignments and dissertations through blackboard’s online submission system and marking by tutors is now fully electronic. the op team started enriching the students’ learning experience with additional interactive and portable online course materials in 2009. a two-year joint information systems committee-funded (http://www.jisc.ac.uk) research project called duckling (delivering university curricula: knowledge, learning and innovation gains; http://www.le.ac.uk/duckling) enabled both course teams to address curriculum delivery challenges through the appropriate incorporation of four technologies into learning design: podcasting, second life, wimba™ voice boards and e-book readers. we have previously reported on the successful incorporation of podcasting for work-based learners in op (nie et al. 2010), focusing on challenges in the areas of learner isolation and mobility. the primary purpose of the incorporation of e-book readers into the three programmes was to investigate whether these devices could enhance flexibility in curriculum delivery to better accommodate the needs and demands of highly mobile work-based learners. for example, could a stand-alone e-book reader, pre-loaded with course materials, help students to maximise their limited study time by increasing their access to the essential readings? in the context of the above challenges and opportunities, our central research question was: how does the incorporation of the e-book reader into curriculum delivery affect the experiences of the work-based distance learners on these three master’s programmes? the research was led by the beyond distance research alliance, an independent research, development and learning innovation unit at the university of leicester. the pilot between october 2009 and march 2010, 28 sony e-book readers were sent out to 17 education students and 11 op students for trial. twenty-seven of these e-book readers were sony prs-505™s. as this model was discontinued in 2010, one of the education students was given a sony prs-300™, the replacement model for the prs-505. both models are similar in size, weight, features and price. all 28 e-book readers were pre-loaded with course materials produced by the two course teams. the e-book readers given to 11 op students were pre-loaded with materials from three modules. materials from the remaining modules were subsequently made available via the vle for the students to download and transfer onto their e-book readers themselves, for which specific guidance was given. the e-book readers given to 17 education students were pre-loaded with materials from one module. all materials were pre-loaded in epub format. these materials were the same as those available, in multiple formats, via the vle. the education students also received a sociolinguistics textbook on their e-book readers, for which copyright clearance had research in learning technology 23 been obtained. the participating students were not required to return the e-book readers to the institution. the steps presented in table 1 were involved in preparing and sending out the e-book readers, and supporting students in the pilot. the six-step process described in table 1 was fully piloted as part of the duckling project. the guidance document for conversion of files into epub format is now available as an open educational resource (http://tinyurl.com/37sdzef) for other practitioners to use. figure 1. research methodology research questions the central research question aimed to establish the impact of the incorporation of ebook readers on work-based learners’ experiences: how does the incorporation of the e-book reader into curriculum delivery affect the experiences of the work-based distance learners on these three master’s programmes? it was approached through three sub-questions: (1) what does the e-book reader offer in terms of flexibility and opportunities for mobility when work-based learners need access to course materials? (2) what impact does the e-book reader have on how students study? (3) what are the cost implications of the incorporation of e-book readers into the delivery of these distance programmes? the questions were agreed on the basis of the challenges faced by the course teams, our collective interest in learning technology research and the institutional drive for learning innovation established in leicester’s learning innovation strategy, approved by the university senate in 2009. table 1. making course materials available on the e-book readers. step description 1: preparation the course teams updated and proofread all materials. 2: conversion a duckling project learning technologist converted the source materials into epub format. the formatting was supported by calibre software (http://calibre-ebook.com/), a free, cross-platform e-book reader management system. see how to turn a word document into epub format in figure 1. 3: pre-loading the re-formatted course materials (in epub format) – and in the case of the education students, the socio-linguistics textbook – were copied onto the e-book readers. 4: delivery the e-book readers were delivered to students. 5: updates in epub format via vle updated versions of the materials and new resources were uploaded to blackboard in epub format for students to add to their e-book readers. these files were also available in word or pdf formats for standard access. 6: guidance the duckling project team produced guidance documentation for students to follow and instructions on how to add new materials onto the devices, including journal articles from the digital library. 24 m. nie et al. action research the research was conducted using an action research methodology that involved the “action–reflection” cycle (mcniff and whitehead 2006, 9). as part of the duckling project, the action–reflection cycle started before the incorporation of the four technologies into curriculum delivery. through surveys and interviews, staff and students of both disciplines were consulted from the very start of the project on the challenges faced. their views were analysed and results were fed back to the two teams to inform course redesign. as the new technologies (including the e-book readers) were incorporated into the redesign, feedback from students and staff was regularly gathered and analysed. evidence was again fed back to the course teams to inform the next set of design changes. figure 1. turning a word document into epub format. research in learning technology 25 the survey to understand the impact of the e-book readers on learners’ experiences, data were first captured through a blackboard survey. the e-book reader survey was administered online to capture usage, usability and usefulness of the e-book reader. the full list of questions used in the survey is shown in appendix 1. a summary of the questions included in the survey, grouped by theme, is given in table 2. the survey was completed by all 28 students who had received the e-book readers, 17 from education and 11 from op. the 17 education students are all language teachers who teach english to speakers of other languages in a wide range of contexts in different countries around the world. the 11 op students hold a variety of posts, such as retail managers, human resource officers, recruitment managers, personal coaches and freelance consultants. table 3 summarises the number and gender distribution of the students who participated in the e-book reader research. quantitative data collected from the survey were transferred into excel spreadsheets and descriptive statistics were applied to closed questions. qualitative data gathered from the survey were coded using data-driven (inductive) coding (boyatzis 1998) and analysed using thematic analysis (boyatzis 1998; joffe and yardley 2004) to identify categories and combine categories into themes. the themes are reported in the results section. cognitive mapping to further understand the students’ experiences with their e-book readers, cognitive mapping interviews (bryson et al. 2004) were conducted with nine education students and three op students after the survey. cognitive mapping is a research method developed on the basis of kelly’s (1955) theory of personal constructs. people make sense of their lives and situations by constructing, elaborating, revisiting and re-revising a system of interconnected concepts (more formally called ‘constructs’). (bryson et al. 2004, 21) table 2. summary of the survey questions. category survey questions usage where, how often and for how long (each time) did you use the course materials on the e-book reader? what other documents (in addition to the pre-loaded course materials) did you use on the e-book reader? usability to what extent did you find the course materials on the e-book reader easy to read? how has the e-book reader changed the way you study? rank and justify your preferences for reading the course materials, given the choices of having them in print, on the e-book reader or on blackboard. how easy it is to transfer documents onto the e-book reader? usefulness how helpful did you find the e-book reader in your studies? to what extent were you satisfied with the e-book reader? what functions were you satisfied and not satisfied with? 26 m. nie et al. this method has been used in studies in psychology (curseu, schalk, and schruijer 2010), environment and planning (van kouwen et al. 2009), management and organisation (harris and woolley 2009), and strategy for adopting e-learning (russell 2009). in this study, the cognitive mapping process involved four steps: conducting the interview, creating a causal map, validation and analysis. the interviews were conducted via the telephone, as the students who participated in the study were all distance learners, and were digitally recorded. the interviews focused on two questions: what do students use their e-book reader for, and how do they use the device? the second step was to develop a causal map from each interview. the causal map was used to capture each student’s views, perceptions and experience of using the e-book reader. a causal map is a word-and-arrow diagram in which ideas and actions are causally linked with one another through the use of arrows. the arrows indicate how one idea or action leads to another. (bryson et al. 2004, 4) twelve causal maps (one per interview) were created using the decision explorer software (http://www.banxia.com/demain.html). figure 2 illustrates a fragment of a causal map created from an interview with an education student regarding his use of the e-book reader. the links from concept 3 to concept 4, and from concept 4 to concept 2 are interpreted as “the module material is put in short sections”, so “i can just read the material a little bit from the e-book reader, put it down, and pick it up and resume later”, and for that reason “i’ve been using the e-book reader mainly at work when i’ve got a spare 15–20 minutes”. figure 2. the third step is validation. in cognitive mapping, the causal map needs to be presented to the interviewee to ensure that the map is a true representation of his or her views and perceptions. as all participating students were distance learners, the validation was conducted electronically. each student was sent a causal map developed from his or her interview, plus a letter explaining how the map was developed, how to interpret the concepts and arrows, and guidance on how to provide feedback to the researcher. once all causal maps had been validated by the interviewees, the analysis began. the domain and central analyses provided by decision explorer were applied to the 12 students’ causal maps. in cognitive mapping, the domain analysis calculates “the total number of in arrows and out arrows from each node” (bryson et al. 2004, 324). the result of the domain analysis indicates the richness of meaning of each node. a node with the highest score indicates that it is the “nub of the issue” (bryson et al. 2004, 324) of that map. the central analysis calculates the centrality of a node within the map. a higher central analysis score implies that the node is of structural significance table 3. respondents’ profile. education students op students total instrument male female male female male female total survey 9 8 2 9 11 17 28 interview 6 3 1 2 7 5 12 research in learning technology 27 to the map. both domain and central analyses are used to detect the most important or “busiest” concepts. these concepts were compared and contrasted with the themes that emerged from the e-book reader survey. results the e-book reader, as used in the context of this research, improved the experience of work-based distance learners in a number of ways. the findings also highlighted the current limitations of the technology used in the study. these findings are presented under four themes, mapped against the initial challenges that the course team faced. flexibility to support the mobile lifestyles of learners students highly valued the portability and flexibility that the e-book reader offers. they used their e-book readers in various ways and in different locations. table 4 shows that 16 students from both disciplines used it in public places (such as a café) and 18 of them did so on the move (e.g. on a train, bus or aeroplane). figure 2. a fragment of a causal map. 28 m. nie et al. students from both disciplines emphasised the benefit in terms of flexibility in response to the open questions in the survey: the e-book reader is a lot more flexible in terms of where you study. i don’t feel like i am confined to my desk. (psy7) the e-reader has increased my ability to study in more places without needing to carry around a bulky folder. (edu4) for the same reasons, students found it easy to use their e-book readers in public places and on the move, as one of the students expressed in the survey: i loved the small size/weight and thought it would be practical to carry when i’m travelling by plane, car, etc., allowing me to study without transporting a lot of heavy books. i’m a bike commuter as well, so it would also be practical for daily trips to work, the library, or the beach or park, where i like to do a lot of my reading. (edu3) the readability of the e-book reader under different conditions makes it suitable for outdoor use. one student in malta stated in the interview: malta is very sunny. there is an issue with the readability in the sun. the visibility of a laptop is a problem when you use it outdoors. the e-book reader can be so readable in the sun. from this point of view, the e-book reader is excellent. it’s just like reading a paper. i take my e-book reader when i know that i’ll have a bit of time on the boat. the e-book reader has made things much easier for me. (psy3) many students felt that their reading experience from an e-book reader is similar to reading from a book. for students who do not enjoy reading from a computer screen, an e-book reader can be a welcome alternative, as one student stated in the survey: i find it very difficult to read for long periods on my computer due to the eye strain, and have always relied on printing out long chapters in order to read more comfortably. with the e-book reader i can avoid all of that and read anytime and anywhere i like. (edu10) accessing course materials from a single device without an internet connection is considered another advantage by the students, as one pointed out in the survey: i no longer worry about carrying so many papers in my bag or having an internet connection to access the modules material. i believe that it made my life easier. (psy8) long battery life, capacity to accommodate many readings and a user-friendly interface were also considered key advantages. table 4. where do students use their e-book readers? location/place education students op students total at home 9 8 17 in the office 6 3 9 at public places (e.g. library, café) 10 6 16 on the move (e.g. train, bus, plane) 11 7 18 research in learning technology 29 these findings suggest that the functionalities and portability of the e-book reader made the curricula more flexible, which catered for the mobility challenge faced by work-based distance learners. the device has made essential readings more accessible to students and their reading experience enjoyable. study time the portability of the e-book reader and its many functionalities made it easier for students to take all course materials with them anywhere and read whenever they had a gap in their diaries. many students pointed this out in the survey: it’s helpful from a practical point of view. the nature of distance learning means you have to squeeze in study whenever you have time. having textbooks and journal articles stored together so conveniently is much better than reams of paper on a packed commuter train in osaka. (edu4) i am using the e-reader to access the unit materials every day. this has increased the amount of course materials i have gotten through (i have finished reading through the part a on sociolinguistics). before this, i had been allocating weekends to spend on one or two units, and i would only study through the week if i had a few hours to spare. (edu5) i find that the reader has become an integral part of my day in particular situations. i have a load of eight hours a week of call (computer assisted language learning) classes here, during which there is pretty much nothing to do. but since i’ve received the reader, after i open the lab i sit and start reading the material i’ve loaded and take notes on what interests me. the reader, then, has given me more study time during the day. it’s nice to have the corpus of material i am working on loaded onto a single device. there’s something nice about being able to look at all the documents i’m interested in, and get a kind of “big picture” from doing this. i’ve also gotten into the habit of carrying my reader with me everywhere, and if i’m in a situation where i’m waiting for 20 minutes i will automatically open the reader. how i kill time has been transformed by this device, as previously i would have just spent the 20 minutes playing games on my iphone! (edu6) the bookmark and continue reading functions make the e-book reader extremely easy for students to turn off and restart where they left off. these functions increased the likelihood that students would use their e-book readers during their breaks (even short breaks) and on the move. the analysis of students’ cognitive maps revealed the same finding. a causal map developed from an interview with an education student (figure 3) shows how the device enabled him to make more effective use of his time. the e-book reader’s functions and the opportunity to have all the course materials on this stand-alone device made it very convenient for carrying around and reading a few pages whenever he had a gap. as a result of this, this student took the device with him wherever he went. the device enabled him to do more readings while on the move. figure 3. these findings suggest that the e-book reader was a welcome addition to the lifestyles of work-based distance learners. the device helped them to make better use of ‘dead time’ and enabled them to fill gaps in a purposeful way. learner study strategies some students reported that their study strategies changed as a result of having the e-book reader. table 5 shows that 11 students from both disciplines reported that the e-book reader changed the way they study. 30 m. nie et al. interviews with students revealed further detail on what changed as a result of having an e-book reader. an interview with an education student showed that the e-book reader enabled her to change strategies for keeping notes: the e-book reader changed my way of keeping notes and made my study more effective. before, i used my laptop to write a lot of notes because i feel i will forget the whole thing if i don’t take them down. but taking notes is time-consuming and not that effective because i never really use the notes. with the e-book reader, it’s not very inconvenient to go back to the material on the e-reader, and i can remember where the material was and go back to the module on the e-reader and look through it. as a result of that, i didn’t take a lot of notes and i don’t think that it [not taking notes] makes a difference to my study. (edu2) for some students, the e-book reader was not a substitute for other forms or means of reading. it offered them a choice about what to use in a given situation. a causal figure 3. effective use of time. table 5. has the e-book reader changed students’ study? has the e-book reader changed the way you study? education students op students total yes 8 3 11 no 9 8 17 research in learning technology 31 map developed from an interview with an education student (figure 4) illustrates how he chose different devices for reading to suit his situations and needs. figure 4. figure 4 shows reading habits that are highly situational. this student uses his laptop and books for conducting research because he considers cross-referencing speedy and intuitive (concept 31). he still prints out materials when he wants to make notes (concepts 9 and 11). he does not use his e-book reader on public transport or when travelling because he finds those environments too noisy to read (concept 20). he prefers to use his e-book reader at home when he has a quiet time (concept 27). he chooses to use his iphone™ for quick checks (concept 29). another interview with an op student indicated that the e-book reader enabled her to change strategies for reading and approaches to assignments: the e-book reader has changed my approach to how i do assignments to some extent. previously, i started my assignments after i’d done all the readings from print. i now start from the assignment and then read the material. i’m being more selective now in what i’m reading, and it’s saving me time in a way. before i just tried to read everything, i worked a lot harder before but i didn’t really get the grades. (psy1) on the one hand, the e-book reader enabled this student to become more selective in reading, resulting in saving time and being more effective in approaching assignments. on the other hand, this student is reading less and becoming more strategic in her study. the case of this student suggests that the increased access and convenience afforded by the e-book reader resulted in less reading but better grades. questions regarding breadth and depth of reading and the role of technology to promote better learning require further examination. figure 4. situational reading. 32 m. nie et al. cost-effectiveness some students became less dependent on printed materials or more selective in what they printed out as a result of having access to course materials on their e-book readers: … reading off the internet causes me eye strain. thus i would prefer to print off any reading to reading it online. however, reading from an e-book reader causes no strain and provides me with an equivalent reading experience, thus removing the necessity of print in my case. (edu12) i do like to have paper copies some of the time, but not all the time. i can read the notes on the e-book reader. if i think i need a copy i will print them out. if the memory was bigger i probably wouldn’t print out that much at all. (psy1) the causal maps reinforced this finding. for example, an education student explained how he was able to save can$100 from printing materials (figure 5), which suggests that delivering course materials on e-book readers can offer cost-saving benefits to students. figure 5. cost-cutting was not the motivation behind the introduction of e-book readers by either course team. instead, the move was part of a pedagogical innovation process designed to help enrich the students’ learning experience with interactive, flexible and portable course materials. table 6 summarises the costs involved in the incorporation of e-book readers into these teaching programmes in 2009. figure 5. cost saving for students. research in learning technology 33 limitations in functionality generally, students were unimpressed by the lack of a note-taking feature in the sony prs-505™. this drawback limited student use of the device. evidence showed that the e-book reader was considered appropriate for light reading, such as reading for a general overview. students went back to the printed materials, where they could underline and make notes, especially during revision and preparation for assignments or the dissertation. some considered e-book readers more useful and appropriate for reading for leisure than for study purposes. the e-book readers used in this study offered users one major benefit: a pleasant and efficient reading experience, providing some motivation for them to keep up the effort of studying alongside their busy work and home lives. higher-specification models, including the sony prs-600™ and other devices such as the amazon kindle™ and apple’s ipad™, offer highlighting and annotation functions. further research is needed to establish the extent to which this note-taking feature is usable and practical enough to change study habits and enable a better learning experience. another functionality that students were not satisfied with was the delay in pageturning, which causes ‘blinking’ or ‘flickering’. this adds about one or two seconds to page-turning, which some found “disappointing”, sometimes “disturbing” and “annoying”. however, evidence showed that the majority of the students put up with this discomfort and continued to use their e-book readers on a regular basis. only one student stated that she completely gave up on the device because of this limitation. students identified further areas for improvement and came up with the following wishlist: a bigger screen, a touchscreen, wireless connectivity, better presentation of tables and diagrams, and a more intuitive, straight-forward solution for converting word documents into epub format. newer devices, such as the third-generation kindle™ and the ipad™ do offer some of these benefits. the former is cheaper than the sony models we used in 2009 and offers a range of additional benefits, including a keyboard, wi-fi and optional 3g connectivity. discussion the evidence points to a range of advantages that an e-book reader can offer to workbased distance learners. these benefits suggest that the device can effectively address the initial challenges faced by the course teams on the duckling project: flexibility in curriculum delivery to accommodate the needs and demands of highly mobile learners, and maximising the benefit of limited and fragmented study time. the findings point to additional benefits of the e-book reader to work-based learners. table 7 summarises these points. table 6. cost of introducing e-book readers in 2009. cost category item costs per student device sony-prs-505™ (discontinued), replaced by sony prs-300™ £150 delivery secure dhl delivery £30 per parcel overseas materials formatting and converting module materials into epub averaged at £60 per device total cost per student £240 34 m. nie et al. access to other readings in addition to using the pre-loaded materials, 18 out of 28 students transferred journal articles and eight copied other internet materials onto the device for study purposes. thirteen transferred novels to their e-book readers for leisure purposes (table 8). this suggests that the device enabled students to read beyond the immediate requirements and materials of the course. technical experience and skills the evidence suggests that the e-book reader does not place high demands on the learner in terms of technical skills. twenty-one students from both disciplines found the e-book readers easy to use for reading course materials (table 9). twenty-two students from both disciplines found the process of transferring additional materials to the e-book readers straight-forward (table 10). copyright and intellectual property rights intellectual property rights and copyright are major issues encountered in this study. despite extensive help and collaboration from the university’s library and contacts table 7. summary of key advantages of e-book readers in curriculum delivery. finding key points enhanced flexibility in curriculum delivery to meet mobility demands students used their e-book readers in different places (at home, in the office, in public places and on the move) to access all essential course readings. better use of time students found it easy to take the e-book reader anywhere and read whenever an opportunity arises. they filled in the gaps during the day for study purposes. new study strategies some students changed their strategies for reading, keeping notes and approaching assignments. some students opted for the e-book reader for reading while on the move, the iphone™ for a quick check, and printed material for notetaking and highlighting. cost savings for students as a direct result of the availability of the e-book reader, some students were less dependent on printed material and more selective when printing materials out. table 8. content added by learners to their e-book readers. type of content education students op students total course-related material (i.e. journal articles) 11 7 18 novels 9 4 13 support material from other sources (i.e. the internet) 6 2 8 documents created by the student him/herself 4 0 4 images 1 1 2 research in learning technology 35 made at all levels, publishers gave blanket refusals to pre-loading copyrighted materials such as e-books and journal articles onto the e-book readers. a single education e-book on socio-linguistics was made available on the e-book readers after obtaining agreement from its publisher. the e-book readers could have been much more useful if all essential readings, including core textbooks and journal articles, had been preloaded on them. to sustain and scale up the use of the device, a possible way forward is the extensive use of open educational resources. further research is needed to establish the feasibility and impact of this approach in conjunction with e-book readers. conclusions this research has illustrated an innovation in the delivery of work-based distance learning programmes through the incorporation of e-book readers. the evidence showed that the students welcomed the innovation. the device enhanced the student learning experience by increasing flexibility in curriculum delivery to meet their mobile lifestyles, enabling relatively cheap and rapid access to essential readings, filling ‘dead’ time slots with useful reading and promoting changes to study strategies. it also reduced costs for some students, who became more selective in what they chose to print out. in summary, this innovation has increased the perceived quality of distance learning for students who value flexibility and learning on the move. with appropriate design and planning, the e-book reader made the work-based distance learner’s life easier. limitations in functionality, which have, at least to some extent, been overcome in later models, constituted a hurdle for some students. intellectual property rights and copyright restrictions have limited our ability to make other essential readings available table 9. to what extent do students find the e-book reader easy to use for reading materials? to what extent do you find the e-book reader easy to use for reading? education students op students total very easy 5 4 9 easy 7 5 12 neither easy nor difficult 4 2 6 difficult 1 0 1 very difficult 0 0 0 table 10. how easy it is to transfer documents onto the e-book reader? how easy it is to transfer documents onto the e-book reader? education students op students total very easy 3 1 4 relatively easy 11 7 18 relatively difficult 0 1 1 very difficult 1 0 1 i’ve never transferred any documents onto my e-book reader 2 2 4 36 m. nie et al. on the device, such as prescribed e-books and journal articles. to enable the benefits of e-book readers in distance education to be enjoyed on a wider scale, a possible way forward is to encourage students who have their own mobile reading devices (e.g. smartphones, netbooks, kindles™) to use these devices for study purposes, and to provide support by making course materials available in epub format, and linking to guidelines on downloading library resources to mobile devices. students with mobile reading devices could also be encouraged to explore relevant open educational resources, as a way of increasing access to quality material and keeping costs low. further research is needed to establish the feasibility and impact of these approaches to the use of e-book readers. further research several areas have emerged as suitable for further investigation. e-book readers have changed rapidly over the past two years and are very much in transition. more recent e-book reader models than the ones used in this research are not only cheaper and faster, but they offer a range of functions that our students would have benefited from, such as highlighting text, annotation, wi-fi and 3g connectivity. whether these additional features would contribute to better reading and learning experiences is worth researching. many advocates of e-books and e-book readers use the ‘green’ argument to make their case. this certainly merits proper research. at present, it is unclear whether the amounts of metal (including lithium), glass, rubber and other components involved in building and disposing of e-book readers make the device any ‘greener’ than printed books. how many paper books do we need to read for an e-book reader to become a ‘greener’ alternative? better access to course materials and new study strategies does not necessarily mean better reading comprehension or better learning. the case has also been made (richtel 2010) that we deprive our brains of much-needed ‘downtime’ by using every spare moment to read, where previously we might have used the opportunity to daydream or reflect. finally, it would be of interest to see how distance education providers respond to this type of technology in terms of the development of learning materials, technological infrastructure and cost models for their programmes. all these questions merit further investigation. acknowledgements figure 1 was created by emma davies and terese bird, university of leicester. references andresen, l., d. boud, and r. cohen. 1999. experience-based learning. in understanding adult education and training, ed. g. foley, 225–39. 2nd ed. sydney: allen & unwin. boyatzis, r.e. 1998. transforming qualitative information: thematic analysis and code development. london: sage. brennan, l. 2005. integrating work-based learning into higher education – a guide to good practice. bolton: university vocational awards council. http://www.uvac.ac.uk/ 0402.html. bryson, j.m., f. ackermann, c. eden, and c.b. finn. 2004. visible thinking: unlocking causal mapping for practical business results. chichester: john wiley & sons. research in learning technology 37 clamp, e., and e. warr. 2002. addressing employability through implementation of workrelated learning. paper presented at skills development in higher education: forging links, july 10–11, at the university of hertfordshire, hatfield, uk. curseu, p., r. schalk, and s. schruijer. 2010. the use of cognitive mapping in eliciting and evaluating group cognitions. journal of applied social psychology 40, no. 5: 1258–91. fagerberg, t. and t. rekkedal. 2004. enhancing the flexibility of distance education – designing and trying out a learning environment for mobile distance learners. paper presented at the 21st icde world conference on open learning & distance education, february 18–21, in hong kong. gallacher, j., and f. reeve. 2002. work-based learning: the implications for higher education and for supporting informal learning in the workplace. milton keynes: open university. http://www.open.ac.uk/lifelong-learning/papers/index.html. gray, d. 2001. a briefing on work-based learning. learning and teaching support network generic centre, assessment series no. 11. york: ltsn. http://www.bioscience.heacademy.ac.uk/ftp/resources/gc/assess11workbased.pdf. harris, e., and r. woolley. 2009. facilitating innovation through cognitive mapping of uncertainty. international studies of management & organisation 39, no. 1: 70–100. hills, j.m., g. robertson, r. walker, m. adey, and i. nixon. 2003. bridging the gap between degree curricula and employability through implementation of work-related learning. teaching in higher education 8, no. 2: 211–23. joffe, h., and l. yardley. 2004. content and thematic analysis. in research methods for clinical and health psychology, ed. d.f. marks and l. yardley, 56–68. london: sage. kelly, g. 1955. the psychology of personal constructs. new york: norton. lam, p., j. lam, and c. mcnaught. 2010. how usable are ebooks in an mlearning environment? international journal of continuing engineering education and life long learning 20, no. 1: 6–20. mcgreal, p., b. cheung, t. tin, and s. schafer. 2005. the athabasca university digital reading room: library resources for mobile students. proceedings of iadis international conference on mobile learning 2005, june 28–30, in qawra, malta, 47–54. lisbon: iadis. mcniff, j., and j. whitehead. 2006. all you need to know about action research. london: sage. nie, m., a. armellini, s. harrington, k. barklamb, and r. randall. 2010. the role of podcasting in effective curriculum renewal. alt-j, research in learning technology 18, no. 2: 105–18. nixon, i., k. smith, r. stafford, and s. camm. 2006. work-based learning: illuminating the higher education landscape. york: higher education academy. http://www.heacademy.ac.uk/assets/york/documents/ourwork/research/wbl_illuminating.pdf. rekkedal, t., a. dye, t. fagerberg, s. bredal, b. midtsveen, and j. russell. 2005. design, development and evaluation of mobile learning at nki distance education 2000–2005. norway: nki forlaget. http://www.dye.no/articles/mlearning/m_learning_2000_2005.pdf. richtel, m. 2010. digital devices deprive brain of needed downtime. the new york times, august 24, technology section. http://www.nytimes.com/2010/08/25/technology/ 25brain.html?pagewanted=1&_r=1. russell, c. 2009. a systemic framework for managing e-learning adoption in campus universities: individual strategies in context. alt-j, research in learning technology 17, no. 1: 3–19. shiratuddin, n., m. landoni, f. gibb, and s. hassan. 2003. e-book technology and its potential applications in distance education. journal of digital information 3, no. 4. http:// journals.tdl.org/jodi/article/view/90/89. van kouwen, f., c. dieperink, p. schot, and m. wassen. 2009. computer-supported cognitive mapping for participatory problem structuring. environment and planning 41: 63–81. walton, g., s. childs, and e. blenkinsopp. 2005. using mobile technologies to give health students access to learning resources in the uk community setting. health information and libraries journal 22: 51–65. whalley, b. 2010. learner acceptance of on-line learning and e-learning. association for learning technology. http://wiki.alt.ac.uk/index.php/learner_acceptance_of_on-line_ learning_and_e-learning. 38 m. nie et al. appendix 1. e-book reader survey questions 1. can you tell us your initial impression and thoughts about the e-book reader when you first received it and started to use it? 2. now you have been using the e-book reader for a while, have these thoughts and impressions changed at all, and in what way? 3. how often do you use the course module material on the e-book reader? 4. how often do you use the podcasts on the e-book reader? 5. how long do you use the e-book reader each time? 6. if you answered ‘it depends’ to question 5, can you please briefly explain? 7. where do you usually use the e-book reader? 8. to what extent do you find the course module material on the e-book reader easy to read? 9. to what extent do you find the podcasts on the e-book reader easy to use? 10. if you find the course materials or podcasts on the e-book reader difficult to use, can you briefly explain why? 11. how helpful do you find the e-book reader in your studies? 12. can you briefly explain your choice in question 11? 13. has the e-book reader changed the way that you study? 14. if you answered ‘yes’ to question 13, can you briefly explain what has been changed? 15. are there any functions that you wish the e-book had that would help in your studies? 16. given these three choices of having the course material in print, e-book reader or the blackboard online format, please rank your preferences for reading your course material. 17. can you briefly justify the order you chose in question 16? 18. giving these three choices of listening to the course podcasts on ipod/iphone/mp3 player, e-book reader or blackboard online, please rank your preferences for using the podcasts? 19. can you briefly justify the order your chose in question 18? 20. can you tell us what else you use the e-book reader for? 21. if you answered ‘others’ to question 20, can you please explain briefly what other activities you used the e-book reader for? 22. how easy it is to transfer documents onto the e-book reader? 23. if you find it difficult to transfer documents onto your e-book reader, can you briefly explain what the problem or difficulty is? 24. please rate your overall satisfaction with the e-book reader. 25. if you’re not satisfied with the e-book reader, can you briefly explain the reason? 25. overall, how would you summarise your experience with the e-book reader? editorial one of the things that makes alt-j distinctive is its diverse readership. there are people who consider themselves to be practitioners pure and simple; others who only research this area; a fair number who do both. because of this, there is a tension between timeliness and the need for deliberation in the articles that are printed. alt-n already provides a forum for articles where timeliness is the greatest priority; alt-j, therefore, must concentrate on articles where the emphasis is on identifying the wider context, providing considered debate or giving longer-term analyses. however, in a field as fast-moving as learning technology, there always remains the need to ensure that current issues are addressed. fortunately, the submissions for this issue have provided a good mix of timely discussion and consideration of long-term concerns. we have, for example, plewes and issroff's review paper, which will be particularly relevant to the uk-based projects soon to be funded under the jisc's exchange for learning call; pearson and koppi's analysis of inclusion and accessibility online, which is particularly timely for uk universities facing changes to the disability discrimination act; and waycott's article on the use of the increasingly popular personal digital assistants. at the same time, we have articles which address longstanding areas of concern, such as singh and dron's study of cross-cultural interaction, or holt et al.'s questioning of assumptions about the benefits of online learning. alsop and tompsett offer a piece that is as much about methodology as about findings. it is to be hoped that their concern for methodology will prompt other researchers in this area to adopt a similarly thoughtful and honest approach. this leads well into mention of an aspiration for future issues. alt-ts diversity is important to maintain, but as noted earlier, not at the expense of scholarship. indeed, this is an area that we are actively seeking to develop. to achieve this, we have begun to tighten 2 alt-j volume 10 number 2 up the submission and review process. in future, our handling of submissions should be smoother and swifter; at the same time, reviews of papers will become more focused. all authors will be expected to locate their paper within a wider research context, and a more consistent format will be adopted for articles, giving more importance to discussion of methodology and the paper's structure. finally, readers will notice that our 'experiment' from the last issue is featured again here. the feedback we have received has been universally positive, so we are planning to continue these discussions as a regular part of the journal. this issue sees gwyneth hughes replying to gunn, french, macleod, mcsporran and conole's article on gender issues. we hope, that this is merely the start of the debate on this topic, and on the others that we plan to feature. martin oliver deputy editor 3 supporting organizational change: fostering a more flexible approach to course delivery gail hart, yoni ryan and kerry bagdon teaching and learning support services, queensland university of technology, email: g.hart@qutedu.au queensland university of technology (qut) adopted a flexible delivery policy in 1996. the main objective of the policy was to develop a more student-centred approach to teaching and learning, since quts student population is predominantly part-time, 'mature age', already in employment, and very diverse in cultural and academic background for many staff, the policy was threatening: staff were uncertain where they might begin to adapt their traditional face-to-face teaching approaches to overcome the limitations associated with time and place, and they were fearful that their teaching role and academic expertise might be superseded by a technological alternative. they lacked confidence to incorporate appropriate and relevant technologies in an innovative and effective way to support student learning objectives. this paper focuses on the implementation of quts policy. it highlights the role of a central services department, teaching and learning support services (talss), in providing training and fostering cultural change across the university. the implementation was guided by a model of flexible education and a set of principles underpinning a 'whole of organization approach to flexible delivery. strategies for supporting innovators, sharing experience across disciplines, co-ordinating and focusing the support of educational developers, and embedding staff development processes are outlined the success and limitations of the organizational change strategy are summarized as 'lessons learned' to inform ongoing institutional policy and procedures. organizational contexts in australia, as in the uk, the political, economic and technological contexts within which higher education operates are shifting. many commentators (e.g. tiffin and rajasingham, 1995; oblinger and rush, 1997) have predicted that the outcomes for teaching and learning environments will be as shattering to the 'bricks and mortar' universities as the longpredicted tectonic plate shift which will swallow san francisco. 46 au-j vwume 7 number i our vision is less apocalyptic. indeed, we are convinced from various research projects in which we have participated (hart et al, 1998; cunningham et ai, 1998) that the move to student-centred education, if it is genuinely embraced by universities, will result in a slow evolution of existing university environments. however, there is no doubt that the last decade has already produced many changes within universities worldwide. classes are larger, student populations more diverse, both culturally and in terms of their preparation for independent learning. with the introduction of tuition fees and supplementary charges for support services such as computer access, students are becoming more discerning 'consumers'. they expect high-quality learning opportunities and better ancillary services. at the same time, public funding has been reduced as governments of mature economies seek to reduce their outlays by privatizing as many of their traditional services as possible. tertiary education has become a victim of its own success: in moving from an elite to a mass system, it has ceased to become a public investment by the community in its collective future, and has become a 'private good' (cf. saul, 1997) requiring economic decisions as to its worth to individual students. whether we view these changes with delight or dismay, the reality is that we must respond energetically and responsibly to ensure that we continue to foster optimum learning outcomes for our students. much of the burden of this work falls upon individual academics. but the organizational culture within which teaching and learning occurs plays an increasingly important role in teaching, for several reasons: • the increasing use of technology in the delivery of learning materials means that institutions have new challenges, e.g. in the provision and/or co-ordination of computer facilities, to ensure access and equity prevail; • individual academics cannot master alone the intricacies of technology-aided communication; • communication technologies are changing the nature of personal interactions, with less need for face-to-face communication to resolve course enquiries or provide individualized advice. analysis of qut's student population in the late 1990s revealed a changed demographic to predominantly part-time and mature-age students. students are attracted to qut because of its strong industry and professional links, high rates of graduate employment, and reputation for applied research. such a population seemed to demand more flexible administrative and teaching approaches that could accommodate their part-time status, and their need for highly relevant professional qualifications. qut is also a multi-campus institution with three campuses in the northern suburbs of brisbane, creating consequent travel difficulties for staff and students who operate across campuses. hence the university determined to investigate and if necessary 're-engineer' its structure to accommodate changing student needs and better prepare students for their professional futures. it was recognized that cultural change was integral to any systemic change, as both bates (1995) and mason (1998) argue. the term 'flexible delivery' is now widely used in australia to describe an educational approach that takes advantage of technology and new procedures (such as automated telephone or online enrolment) to allow students more choice in relation to the time and 47 gail hart et al supporting organizational change: fostering a more flexible approach t o course delivery place of their study experiences, and to bring greater efficiencies to an organization's administrative practices. qut introduced a flexible delivery policy in 1996. however, the launch of the policy prompted real concern amongst our staff. it coincided with the first round of job losses in australian universities and was linked with potential staff retrenchments. the emphasis on unfamiliar technologies alienated staff who were committed to face-to-face teaching, and who feared their hard-won discipline expertise would be de-valued in what they perceived to be an inevitable 'online university'. our staff are not alone: university of washington staff, usa, have protested strongly against the 'de-humanization' of teaching via online courses (the industry standard, 12 june 1998, at ttp://www.thestandard.net/articles/news_display/0,1270,670,00.html). however, changing practice was perceived as an imperative at qut, in order to maintain a competitive edge in the higher education sector and in response to the demands of an increasingly discerning student body. the task facing the institution's management was to develop: • an organizational climate which diminished fear and encouraged appropriate change; • a support structure which built on staff strengths, and involved staff in devising their own solutions to the problems of the external environment and the needs of their students; • a robust system, which could accommodate upgrades, software changes, and increasing bandwidth demands, as well as legal changes; • a support unit that provided the physical infrastructure and the initial/ongoing technical advice, and specialist academic development staff who could bridge the divide between the 'techos' and the teachers. a model of flexibility in tertiary education the first stage of the process was to develop a succinct definition of flexible delivery that might serve to allay the fears of staff by focusing on the goal of meeting student needs, thus aligning staff and organizational goals. the focus on flexible delivery was intended to signal to the university community that the strategy was underpinned by a commitment to incremental rather than radical change. hence the following definition was developed: flexible delivery refers to the use of a range of strategies and technologies to meet the diverse needs of students regarding the location and time of study. flexible delivery is applicable to both internal and external students (qut office of the deputy-vicechancellor, 3 july 1998). the deputy-vice-chancellor's policy adviser, in consultation with key senior staff in the university, prepared a discussion paper that was widely circulated and then tabled as a policy document at teaching and learning committee and then university academic board. a model of flexible delivery was subsequently developed as a framework within which policy and procedures could be determined. we reiterate that integral to the emphasis on flexible delivery was recognition that flexibility was merely the visible tip of a significant cultural change to the traditional university model. figure 1 represents the framework. 48 aa-j volume 7 number i euan* mnatiaetara ^^^m «ah m..<» a g a j a ^ ^ ^ ^ ^ ^ h ejdmkia ^ ^ " ^ ^ ^ ^ | ttnjaiil maadi ^ * ^ h carricala orsai •uatntctutv l i n a m n g strwc ^ n ttan •tow tur» kaaawaa ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ m | ^ ^ ^ otawaaauttnal haw figure /: the flexible delivery framework the structure signals that we see this model as requiring continuous adjustment, not to the extent of the changes of the last two years, but that the intersection of the ovals will always mean deliberate pressure on the organizational structure through management decisionmaking to support teaching and learning. we recognized that neither flexible learning nor flexible teaching can be mandated. flexible learning depends on student needs and abilities. flexible teaching rests on staff abilities such as discipline imperatives, commitment to quality teaching, knowledge of and confidence in using a range of technologies. however, we could, by 're-engineering' our systems and structures, nurture an organizational culture which valued 'learning about teaching and learning in a new context' as its primary goal. principles of an organizational approach to flexible delivery informing the model and the decisions consequent on the policy were several core principles: • 'flexibility' must be linked to specific learning objectives for each course and each subject, based on objective knowledge of student needs and the subject's rationale. while academic judgements regarding these objectives must be the final arbiter of the types of flexibility introduced, academics must also be fully informed of their students' emerging needs, and the potential of technology to assist them in enhancing learning. • student skills cannot be presumed in any pedagogic approach drawing on new technologies. therefore, unlike the mit management for the 1990s framework (scott morton, 1991), which focuses on strategy, structure, technology, management processes, and staff skills and roles, the qut approach includes student skills and roles as a primary component. • collaborative approaches drawing on the expertise of academics, technical and learning support staff result in the best possible learning materials and teaching strategies. • staff development is embedded in the experience of actually developing flexible options. • while individual academics must be encouraged to experiment and develop unique materials and strategies, sensible use of central resources means that investment must 49 gail hart et al supporting organizational change fostering a more flexible approach to course delivery be largely directed in university-wide or faculty-wide initiatives which contribute to the organization as a whole. • the 'learning organization' must demonstrate its commitment to whole-of-institution change by recognizing that university culture in particular is strongly individualistic and strongly decentralized. hence support structures must balance the strengths of centralized units of expertise with the strengths of localized and semi-autonomous units of discipline and teaching knowledge. • full evaluation must be integrated in the design and development of all flexible delivery strategies and materials. implementation once the initial policy on flexible delivery was disseminated throughout the university, a number of activities and processes were undertaken progressively (not always, we should add, in a thoroughly planned way). • a series of information sessions explicated the changing contexts of higher education, examples and demonstrations of flexible delivery within qut and in other institutions, and how the administration was proposing to assist staff. these sessions were sponsored by the senior executive, thus ensuring that they had a high profile within the university community. • the policy adviser to the deputy-vice-chancellor undertook a data collection exercise to determine the extent of existing operational flexibility within the institution, and to focus on the variety of ways 'flexibility' could be perceived and implemented. currently a more detailed pilot 'mapping' exercise is being undertaken within one faculty to understand better the impact of current initiatives and policy on the development of flexible practices, since we are convinced that information and staff consultation and input are vital to policy development. changed practice can only result from a sound knowledge base. • a series of commissioned divisional and departmental reviews over a two-year period from 1996 assessed the effectiveness of the central services supporting teaching and learning, identified potential cost savings and recommended structural changes to improve service provision. a three-staged restructuring process was implemented to forge synergies across previously separate units and ensure a more focused service. the resultant department is a large central unit, teaching and learning support services, providing a broad range of technical, professional and pedagogical expertise to staff and students. • a drop-in centre, staffed part-time, was established and featured learning resources that capitalized on technological advances such as data visualization and the range of commercial cd-roms available, in a threat-free informal environment outside the formal training programme. • since 1992, qut has provided funding for innovative teaching and learning projects. an internal evaluation of the grant scheme, following an earlier external evaluation, was designed to ensure a closer alignment between the projects funded and strategic so alt-j volume 7 number i university initiatives. the most important result has been our 'large grants', which provide funding up to $150,000 over two years, for projects which embed flexible delivery practices across a faculty, or between faculties. projects must have the support of the relevant dean or deans. • two extensive reports were commissioned from qut staff with a knowledge of the existing organizational culture; one examined the pedagogical dimensions of the new technologies (rossiter, 1997), the other evaluated existing computer-assisted learning at qut (wilss, 1997). the intention was to inform decision-making relating to the contribution that technology could make to increase flexibility. • the division invested in infrastructure and initiatives which demonstrated to staff the central commitment to extending flexibility. among these was a combined student help desk, the student computing guide and the continued development of datawarehouse as a teaching and learning vehicle as well as a management information system. • a web site, 'teaching and learning on the internet', dedicated to teaching with technology in various forms, was established with easy-to-use features, an inviting interface, and links to qut and international exemplars. • specialist positions at senior levels were identified and have progressively been filled with staff who combine both academic and technical expertise; the present authors are respectively at professorial and senior policy adviser (educational media and flexible delivery) levels. lessons learned the key lessons we have learned from the processes and initiatives described above might prove useful to other institutions seeking to avoid wasteful pilot projects and 'wheel reinvention'. • the visible and energetic support of senior management is critical to any strategic initiative involving cultural change and new approaches to teaching and learning. one of our successful faculties in 1998 admitted that had it not been for the criterion that their proposal for a grant must address flexibility, they would not have had to reexamine the nature of their curriculum and their students' expressed needs, and hence would not have realized how dated their teaching had become. • notwithstanding early resistance to the notion of uniting the academic and educational technology support units, it became obvious that unless management 'forced' the issue through amalgamation, the organizational culture would continue to be dichotomized between the 'techos' and the teachers. the resultant unit is a team of general and academic staff who bring their respective skills to catalyse flexible delivery initiatives across and within disciplines and faculty units. the renamed and restructured division (the division of information and academic services) has activated a high-profile promotion of its staff development activities to complement its existing reputation as a provider of technical and library services. • publicizing the successes (and failures) of initiatives through the university avoids ad hoc and sporadic small projects which are not sustainable. workshops, seminars, 51 gail hart et al supporting organizational change: fostering a more flexible approach t o course delivery newsletters, web discussion groups, training programmes are all vital in this process. the variety of information modes acknowledges the diverse learning approaches of staff and students, and models flexibility. • notwithstanding the energy and commitment of the part-time staff member appointed, the drop-in centre was not as successful as we had expected. we surmise that this reflected its part-time status, its confinement to one campus, and the initial attitude of staff. • the early data collection exercises revealed that 'flexibility' was not clearly and widely understood among staff: for example, many academics reported that they used presentation software such as powerpoint, with no apparent recognition that computer-developed overheads, of themselves, do not imply flexibility. in order to be used flexibly, overheads must be available on the web or placed in the library for student access independent of the lecture context in which they are delivered. such information has informed our staff development activities. • restructuring organizations, particularly universities, with their fierce defence of territories, is not without pain for many individuals. tempering fear and reworking job responsibilities, promoting team responsibility and collaboration while maintaining morale, demand time-consuming negotiation processes which require sensitivity on the part of senior management. accurate and timely information and communication, we have found, assist to dispel the atmosphere of distrust which can easily arise. conclusion there is no single template for organizational and cultural change within a university, anymore than there is a single template for online delivery. qut has taken the view that far from inducing cataclysmic change in the shape of universities, the new technologies offer an opportunity for evolutionary development in approaches to teaching and learning, which can accommodate student and staff needs and abilities. that view is informed by a belief that learning in organizations is a continuous and evolving process, and that focusing on the purpose of the university, learning itself, we can develop appropriate institutional responses. references bates, a. (1995), technology, open learning and distance education, london: routledge. cunningham, s., tapsall, s., ryan, y, stedman, l., bagdon, k. and flew, t. (1998), new media and borderless education, canberra: australian government publishing service. hart, g., ryan, y., williams, h. and lunney, p. (1998), improving the practice of mental health nursing in rural and remote communities, a report for the rural health support, education and training program of the commonwealth department of human services and health, canberra. mason, r. (1998), globalising education: trends and applications, london and new york: routledge. 52 alt-] volume 7 number i oblinger, d. and rush, s. (eds.) (1997), the learning revolution: the challenges of information technology in the academy, bolton, ma: anker. rossiter, d. (1997), the digital edge: teaching and learning in the knowledge age, brisbane: qut. saul, j. r. (1997), the unconscious civilization, ringwood: penguin. scott morton, s. (ed.) (1991), the corporation of the 1990s: information technology and organizational transformation, new york: oxford university press. 'teaching and learning on the internet', http://www.tals.dis.qut.edu.au/tlow/tlow.htm. tiffin, j. and rajasingham, l. (1995), in search of the virtual class: education in an information society, london: routledge. wilss, l. (1997), computer assisted learning at queensland university of technology: student learning process and outcomes, brisbane: qut. 53 reviews edited by philip barker anne brockbank, ian mcgill and nic beech, reflective learning in practice, aldershot: gower publishing, isbn: 0 566 08377 9. £49.50. the literature on independent learning suggests three sets of capacities are needed by the learner: strong motivation, self-management and reflection. of the three, most educators readily understand the need for motivation (though how to help learners with this remains problematic) and many resources, approaches and techniques exist to help learners develop self-management. the third requirement, capacity to reflect, is much less understood. so a book with the title 'reflective learning in practice' seems welcome. the book is organized in three parts: 'purpose and themes', 'on the journey learning through and around development programmes', and 'on the journey enabling effective learning'. readers might not see immediately the shape of the book from this, indicating a difficulty there is not an altogether clear distinction between its parts. part 1 is the most theoretical (and be warned it is not an easy read); parts 2 and 3 are collections of case studies of learning within organizations. part 2 differs from part 3 in that 'the learning processes' have 'some programmed element at their centre' (p. 3). the authors rightly make the point that learning is not limited to attending training courses; organizations need to be open to the full range of potential learning experiences. the book is about reflective learning within organizations. these are diverse, for example the following are featured: sterling bank, sainsbury's, the nhs, novartis, pearl assurance, thus pic, kodak, allied distillers and norwich union. yet the authors take a discouraging view of the potential for such learning in a corporate context, in which 'legal rights of proprietors take precedence over those of the employee' (p. 7). dominated by the 'rationalist tradition' in british management, the 'emotional content of learning' is resisted, yet this is 'essential for energising the learner, as well as being a critical component of deeply reflective learning' (p. 20). thus a gloomy backdrop is set for the case studies. these are a mixed bag some much more optimistic in tone than one might imagine from the first part of the book. in fact, some case studies come uncomfortably close to selfpublicity and corporate public relations. the tone of some of these strikes an uncomfortable contrast to the somewhat oppressively academic discourse of the earlier chapters. as does the cosy in-talk of the consultant in chapter 9. books comprising chapters from different hands (and many of the chapters are themselves coauthored) are invariably diverse in style and content. even admitting the difficulty of achieving consistency across so many authors, one nevertheless yearned for a stronger editorial hand. direct contributions from the organizations themselves would also have been interesting; the overwhelming voice is that of the management consultant/educator. what would 125 reviews sainsbury's have said, for example, about their relationship with manchester metropolitan university (chapter 7)? the intention of the book is 'to seek integration between useful theory and theoretically insightful practice' (p. 3). this balance proves, for the most part, elusive. there is a big gap between the theory of the framework set out in part 1 and the detail of the case studies. the authors define reflective learning as 'an intentional process, where social context and experience are acknowledged, in which learners are active individuals, wholly present, engaging with others, and open to challenge, and the outcome involves transformation as well as improvement for both individuals and their organisation' (p. 6). this is an ambitious definition. the book advances limits to what the learner can achieve on their own: 'being able to undertake reflection alone is necessary but not sufficient' (p. 7). other people have to be present to achieve 'transformation'. i have to question this, not only through my experiences as a tutor and learner working independently in open and distance learning but also when considering saints and other outstanding individuals, who withdraw from society either actually or in spirit, precisely to achieve transformation their own and that of others around them. all in all, this book represents a challenge to the reader. it is unnecessarily complex, both in its treatment of the topic and in the wearisome academic style to which it often resorts. the case studies, whilst infuriating at times, do contain insights though even these can be overpowered by complex frameworks or uneven style. roger lewis hefce, uk roger lewis and quentin whitlock, how to plan and manage an e-learning programme, aldershot: gower. isbn 0566 08424 4. £49.50. lewis and whitlock's how to plan and manage an e-learning programme is targeted at 'organisations wanting to make productive use of e-leaming'; and recognizes that 'the implications of adopting new learning strategies or delivery methods are far-reaching and usually require major development input.' written from the perspective of planning for and managing the 'far-reaching' implications of adopting elearning, the book provides a valuable guide to 126 this whole range of issues. it should also be noted that while the book is largely written from a commercial training perspective, the lessons are equally valid for staff managing these processes in further or higher education. however, it is a little surprising how relatively infrequently e-learning is actually mentioned, although, as the authors point out in the introduction, this is exactly as it should be. there is little about e-learning from this perspective that is actually new. most of the issues that arise with e-learning are the same as for other forms of open learning or for that matter from conventional classroom-based learning. what is perhaps new is that the potential of e-learning is leading many more organizations to engage with these issues and for organizations new to these issues this a clear and succinct guide. the book is divided into twelve chapters. the first two set out the authors' general approach which may be characterized as follows: begin with the learners' needs and then follow sound curriculum development procedures. this leads them to divide the planning and management process into six stages: analysis of the learners, programme content, design, develop, pilot and run. chapters 1 and 2 provide a general overview of this iterative process and the following chapters treat the issues arising from each of these stages in more detail. perhaps surprisingly the analysis of the learners begins in chapter 3 with the consideration of the business case and marketing strategy. to readers based in the education sector this may seem an unusual place to begin, but understanding a learner's needs begins with consideration of whether the learner needs the learning that you propose to offer. this in turn leads to the question of whether you can develop, deliver and support this learning in a cost-effective manner. chapter 3 explores these issues and provides useful guidance on how to cost and plan from this perspective. chapter 4 develops the analysis of learning needs from a more obviously pedagogical perspective while chapters 5, 6 and 7 look in detail at how the response to these needs can be articulated into a set of measurable and assessable learning outcomes that can underpin the planning and development of e-learning resources. alt-] volume 11 number 3 chapter 8 then addresses the issues involved in supporting e-learners at all stages of their progress from initial contact to completion. the chapter includes a brief but particularly useful consideration of the role of the tutor in the support process. chapters 9 and 10 then address the issues involved in selecting and/or developing resources. chapter 9 provides useful guidance about how to evaluate existing materials for use or adaptation within your programmes, while chapter 10 examines issues surrounding determining what sorts of materials may need to be developed. both chapters contain useful checklists of considerations and chapter 10 also has a number of useful case studies. indeed, overall the book makes good use of both checklists and case studies. chapter 11 considers the implications of developing in-house or outsourcing. among other things this includes a useful summary of the skill sets and personnel that are likely to be required to maintain an in-house development strategy. readers of alt-j may wish to call the resource implications of this perfectly reasonable analysis to the attention of their managers. the final chapter looks forward to the issues involved in embedding e-learning within wider virtual or managed learning environments. arguably, this is the weakest chapter in the book, but that is because this area is changing and developing so rapidly. overall the book has much to recommend it. it is clear and concise, but nonetheless covers the whole range of issues that need to be considered when adopting e-learaing as an organizational strategy. it will be most useful to those managers in both education and training organizations, who, as yet, have little experience of e-learning. this book can provide a quick and effective way of getting to grips with the scope of the issues that are involved. nonetheless, all but the most experienced managers of e-learning will benefit from lewis and whitlock's expertise. indeed, there is really only one drawback to this book and that is the price. according to amazon the list price is £49.50. at that price no selfrespecting reviewer could really recommend its purchase, even by libraries. one cannot help but feel that the publishers are actively discouraging readers from accessing the book; and that is a shame, because it is a good book. bruce ingraham university of teesside, uk d. murphy, r. walker and g. webb, online learning and teaching with technology: case studies, experience and practice, london: kogan page. isbn: 0 7494 3520 8 2001. £19.99. the use of online learning is becoming pervasive in education whether through studentor teacher-led demand, or via management insistence or peer pressure. this book does not attempt to provide instructions on how to put a course online, instead it chronicles the experiences of early adopters of online technologies in order to help others to avoid or at least understand how to deal with the problems and pitfalls these early adopters have faced and grappled with. the contributors from higher and further education institutions in australia, canada, hong kong, north america, the pacific, and the united kingdom relate their experiences in a number of situations. they identify the key issues, the problems they encountered, their attempts at resolution and reflect on their experience. the emphasis is not so much on the technology (the editors take a broad view of what educational technology is anyway) but rather on a critical incident that happened to the authors and how they learned and developed as an online teacher as a result of it. the book is divided into four sections dealing with issues of student interaction; teaching and assessment; planning and development; and policy. each case study comprises a summary of the issues raised and some background to the case, followed by the details of the case presented in two or more parts. each part is separated by a reflective break with questions posed for the reader to consider. the studies focusing on interaction examine cases where environments designed to be highly interactive have failed or where the interaction has lead to tension,, argument or discord. in 'flame war', robertson reports his experience of conflict when messages to a mailing list were interpreted as a personal attack and the discussion degenerated into a 'flame war'. the reporter discusses the differences in managing debate in a classroom situation and in an online environment, and provides an insight into the additional factors that need to be considered in online discussion. those cases dealing with teaching and assessment consider the issues that arise in online learning both by early adopters and by 127 reviews those who felt compelled to move toward online learning and found it more rewarding than they expected it to be. in 'it seemed like a good idea at the time', oliver discusses the unexpected outcomes arising from making what appeared to be a small and positive change to a course. the change was designed to benefit both students and teacher, and motivation was encouraged by the allocation of marks to that activity. oliver highlights the need to monitor closely the online course and the difference in interaction, feedback and communication between the faceto-face and online modes. the challenges to those involved in planning and development are presented in cases which describe the problems posed to those involved in planning and development issues in an atmosphere of rapid change in the type and availability of technologies; the consequent dangers of committing to particular hardware or software and the need for staff development not just for faculty but for all staff. somekh in "the great software gamble' reports on the experience of a large project designed to develop disciplinespecific computer-based courseware spread across a number of institutions. she describes the impact of inadequate technology and rapidly changing needs in higher education. somekh concludes that effective use of new technologies requires a cross-discipline, integrated approach with an institution-wide development plan; and that courses need to be redesigned to incorporate varied learning and teaching approaches. the case studies relating to policy discuss the problems and failures when staff have been obliged to develop online learning materials as a result of policy decisions, the way these shortcomings, have been handled and the impact on those teaching online. in 'who is leading whom?' selinger tells a positive story of how teacher trainers at warwick university addressed the disparity between the requirements of the department for education and employment (doee) for beginning teachers to know how, when and when not to use ict, and the gap in practice, provision, infrastructure and skills both in schools and among students. by adopting a student-led approach with just-in-time support and continuing access to support and facilities through the use of first class, a programme has been developed that meets the needs of students with differing levels of skills and experience working in schools with differing degrees of infrastructure and integration of ict. at the same time they are meeting the statutory 128 requirements of the doee. by making students the agents of change they have been able to strengthen the partnership between the institute of education at warwick and the schools, and move towards a role of support and development. the reader is encouraged to 'play the game' as they read each case study by reading only part one before pausing to reflect, note their impressions, predict what will happen next and consider what course of action they would take. at the end of the case study the reader is asked to reflect on how the situation was handled, the general issues raised and what the case and the issues mean to the reader. through being actively involved in the development of the case, the decisions made and the outcomes one is able to draw on and benefit from the experiences of these early adopters. this method of reading can be adopted informally as an individual, in discussion with colleagues, or formally as an activity for a group of students studying the practice of online learning and teaching. furthermore, the way in which these case studies are presented promotes and provides a means to practice that essential element in teaching reflection. the contributors to this book are to be congratulated in their willingness to share their experiences, difficulties, solutions and the consequences of the way they addressed the problems so that the rest of us can benefit from their experience and at least try to avoid the same pitfalls ourselves. elaine pearson university of teesside, uk phil race (ed.), 2000 tips for trainers and staff developers, london: kogan page, 2001. isbn 0749436883, 287 pages, £22.50. 2000 tips for training and staff developers is bursting with informative bullet points of useful tips. they appear shouting at you on every page and it is not always easy to see which tip is really important and which is a 'by the way', although the intention is not that you should read from cover to cover but dip into the book as and when the need arises. this is where the book excels in its-wide scope. it has been largely compiled from 500 tips on teaching, lectures, group learning, using computers and open and flexible learning. alt-j volume 11 number 3 contents and index pages are clearly set out over three pages each and can probably take you to whatever aspect of teaching and training you are interested in. race is famous for his use of post-its in training sessions and sure enough they appear in their own short chapter with the use of flip charts. if you have not done too many training sessions you could do well to look at sections on working with difficult participants and also the section on coping with the unexpected. the group-based training section has more subheaded bullets in light grey which are easy on the eye and seem to clarify the important points well. getting a group of people to relax and work with one another can sometimes be helped with ice-breakers. rather than take the approach of giving as many roles as possible for those in the group working with the facilitator, he polarizes the issue by using the terms 'leader' and 'fellowship'. with a very clear focus on the learning objectives of the training sessions and alignment with assessment, i ask, 'is the power play of the management boardroom of leader and follower a necessary component?' also instead of concentrating on small-group teaching methodologies such as those found in the jaques (2000) book which encourage positive interdependence getting learners to rely on each other to learn there are two long chapters on how adverse behaviours of small groups can be rectified. the tips then come into their own with useful suggestions. results-based training is concerned with getting trainees to learn by themselves and race comes up with an impressive list of the types of trainees who may be particularly helped including both high and low fliers, those using a second language and even those people who do not like being taught! he provides a very useful and extensive checklist for flexible learning materials, which include details of content, the learning outcomes, structure and layout, and self-assessment questions and activities. choosing and using computer-based resources is full of sensible common-sense ideas, which agonize over the frequently encountered problems of choice whether to use proprietary products, adapt, or create programmes from scratch. the later chapters on helping trainees to work together with computers and make use of email and email groups is still based on sensible common-sense ideas. advice is there on how to deal with the technophobe and well as the technophile. the practicalities of setting up computers at an unknown location are outlined followed by a computer glossary. the book ends with a separate chapter on evaluating your training. this is probably the most useful chapter in the book. it provides sound questions to ask on what it is you hope to achieve using a questionnaire, how to structure questions and advice on what to include and what to leave out! overall, a brilliant book written in a lively and common-sense style which is bound to inspire a trainer or staff developer to greater things. the book is well laid out but clearly cannot do everything. you still need something that outlines small group activities more clearly and a lot more on the technical side of computers but there are five pages on how you could retrieve this information from the internet and even what to do it if there is a power cut. john sweet uwcm otto peter, learning and teaching in distance education: pedagogical analyses and interpretations from an international perspective. london: kogan page, 2001. isbn: 0 7494 3594 10. paperback, viii + 279 pages, £19.99. government policies promoting widening access as well as an increasing focus on continued professional development and life-long learning have prompted universities to re-evaluate their educational provision. increasingly, providers (both traditional and commercial) turn to online learning as a means of providing greater flexibility in course delivery. in light of the recent growth of distance education programmes, learning and teaching in distance education offers a timely and much-needed exploration of distance education's special pedagogy, approaches and their implications for teaching and learning. the book, an updated version of a previous hardcover edition titled learning and teaching in distance education: analyses and perspectives from an international perspective, represents excellent value. case studies have been updated and conveniently consolidated in one chapter. the revised edition also contains new material on virtual universities and under 'further reading' a substantial compilation of the latest 129 reviews literature in distance education research (see review by pearson, alt-j1, (3), 106-8). like the first edition, the volume is structured into eight chapters, which are best approached sequentially, as later parts build on the theoretical concepts introduced in the beginning. a review of the state-of-play in distance education sets the scene in the initial chapter. firstgeneration distance education methods, such as correspondence courses using print and supplementary media (television, tapes and video), exist in parallel to secondand thirdgeneration methodology. the latter two methods employ teleconferencing and internet communication technologies, respectively, to overcome the geographical separation of teachers and learners and increase dialogical learning. the advantages and drawbacks of each of these approaches, however, only highlight the dilemma of distance education, which has to balance a high degree of accessibility for many with the quality of interaction in teaching and learning. further, peters introduces a framework of five principal factors as the means to differentiate conceptually between distance learning and conventional university education. these factors, namely, technical media, structural handicap, student type, forms of structuralization, and the special combination of forms of teaching and learning serve as criteria for the organization of the text and the analysis and interpretation of distance education pedagogy. in fact, the substantial contribution and strength of peters's book lies in its rigorous academic exploration of distance education. there are aspirations to develop a distance-learning science and theory. to this effect, chapter 2 examines different distance-teaching models employed by different countries and schools of thought. each of these models seeks to address the structural handicap of distance education, that is, the spatial and communicative or mental distance between teachers and students. interestingly, experts do not identify a clear preference for one model over others. rather, weighting three aspects, (1) the level of dialogue, (2) the level of predetermination and structure of the study course and programme, and (3) the level of learner autonomy (self-determination of learning goals), the optimal learning experience depends on the student type and his/her goals and expectations. the concepts of dialogue, structure and autonomy are further explored, first theoretically and then in their day-to-day application of distance 130 education (chapters 3 and 4). here, peters produces an exhaustive and meticulous list of the different dialogue-enhancing measures and structural approaches applied in the field. the plurality in approaches used is great and if anything, the trend is to intensify the dialogue between teachers and students and amongst students, as communication is seen as an important element of learning and knowledge verification. chapters 5 and 6 offer especially stimulating discussion. approaches in distance education are not only influenced by didactic and technological considerations, but also by economic, social, cultural and political thought. this is reflected in the fact that distance and open learning are increasingly used interchangeably. education and learning are not completed by the award of a degree but are supposed" to continue throughout life; they are supposed to be accessible and open to broad segments of society. drawing on the relationships between current societal trends, philosophical thought and distance education, peters evaluates the impact of digital information and communication and speculates on future developments. he outlines how digital communication and virtual learning environments could help provide the conditions to deliver the flexibility and customization that is stipulated by postfordist/post-industrial production if imbedded thoughtfully in suitable infrastructure. commendably, peters does not shy away from controversial issues. he suggests, or rather urges, that educationalists must consider the implications of the fundamental postmodern critique on the values and aims of education. the rising desire of students for self-development, fun and short-term satisfaction are perhaps not deliverable by time-tested models of distance education with centrally planned curricula and students working mostly in isolation; this probably requires new models and further development of the concept of the virtual university. in chapter 7, eight detailed institutional case studies illustrate different distance education models and structures. the compilation of a comprehensive and balanced set of examples, ranging from correspondence studies in south africa to telecommunication-centred set-ups in north america demonstrates the authors' expertise and broad knowledge of the subject. chapter 8 recaptures the main themes and provides a final analysis. alt-] volume 11 number 3 overall, the book is well written, except perhaps, for the oversight to provide translations for a range of german references, such as 'einsamkeit und freiheit' and the 'wissenschaftsrat'. frankly, this is an unfortunate omission, as the meaning of these terms contributes significantly to the understanding of the passages and arguments in which they are embedded. nonetheless, this second edition of learning and teaching in distance education is a worthwhile if not essential addition to the library of any academic involved in distance education. even for those not interested in distance education, the book is a valuable read. the pedagogic concepts and discussions shed new light on traditional university teaching and provide provocative thought for the engagement of universities in online teaching and virtual teaching environments. andrea i. frank cardiff university 131 key skills by design: adapting a central web resource to departmental contexts claire mcavinia university college london email: c.mcavinia@ucl.ac.uk web-based delivery of support materials for students has proved to be a popular way of helping small teams to implement key skills policies within universities. the development of 'key' or 'transferable' skills is now encouraged throughout education, but resources (both in terms of staffing and budget) tend to be limited. it is difficult for key skills teams to see learners face to face, and not feasible to print or distribute large amounts of paper-based material. web-based delivery presents a means of overcoming these problems but it can result in generic study skills material simply being published online without due consideration of the needs of different groups of learners within different subject disciplines. therefore, although a centralized website for skills provision can overcome logistical problems, it may be perceived as irrelevant or unusable by the student population. this paper presents a model for web-based delivery of support for key skills which incorporates two separate approaches to the design of these resources. the model was implemented as part of a wider key skills pilot project at university college london, over a period of one year. it includes a 'core' website, containing information and resources for staff and students. these can also be accessed via customized, departmental key skills homepages. this paper presents the basis for the design choices made in preparing these materials, and the evaluation of some of the pilot departments using them. it then draws some wider conclusions about the effectiveness of this design for supporting skills development. introduction: the key skills agenda key skills are referred to increasingly by government and employers in the uk as being essential to ensuring long-term economic prosperity and an adaptable workforce. the new prioritization of skills at government level is reflected in policy statements and strategy at i i oaire mcavinia key skills by design: adapting a central web resource to departmental contexts all levels of education and training (dfes, 2001), and indeed by the renaming of the department for education and employment as the department for education and skills. within higher education, key skills are represented as 'general' skills which employers expect graduates to bring to the workplace in addition to the specialist knowledge they have acquired in the course of their degree studies (atkins, 1999). they are usually defined as being separate from discipline-specific skills and assumed to be easily 'transferable' from one context to another. although there is a continuing debate about the nature of these skills, whether they can be transferred, and whether employers are in fact able to predict what kinds of skills they will need from future graduate cohorts (drew, 1998), the agenda has been steadily advanced by he policy in recent years. key skills have been grouped under a variety of labels: 'core', 'personal transferable' and 'generic' to name a few (bennett, dunne and carre, 1999). attempts have also been made on a national level to draw up a definitive list of key skills. the qualifications and curriculum authority (qca, 2000) has undertaken this for pre-university and national vocational qualifications, while dearing (dfee/ncihe, 1997) nominated four key skills which he recommended should be developed in the course of a student's higher education: communication, numeracy, information technology and learning to learn. there remain many differences between the institutions' and subject disciplines' definitions of key skills. institutions may define key skills in terms of 'graduateness', and the attributes they would like their graduates to have. for subject disciplines, the various categorizations of skills may stem from the difficulties in defining which skills are subjectspecific and which are 'general' or transferable (bennett, dunne and carre, 2000). this is reflected in the extensive lists of skills which have emerged from the subject benchmarking process instigated by the quality assurance agency for higher education (qaa, 2001). however, although the terminology and the lists themselves are not universally agreed, most will include communication, improving own learning, information technology skills, numeracy, teamwork and problem-solving. key skills delivery in higher education institutional approaches delivering provision for key skills to students in higher education is made more difficult by the fact that there is a wider debate about whether it is the role of higher education to prepare graduates for life after their studies and for the workplace (drew, 1998). if we accept that this is a significant responsibility of the university, then to what extent should it be resourced and should provision be made within the curriculum? a second issue here is that of support for any initiative once it is under way: university management may be reluctant to invest in long-term provision, the effects of which are insufficiently documented (hesketh, 2000), and towards which greater and lesser amounts of government interest are directed at various times (drew, 1998). as a consequence of these debates, there is an extensive range of approaches to the implementation of key skills from one university to another (fallows and steven, 2000). some universities undertake institution-wide audits for key skills and set firm targets for skills delivery and assessment. others may choose a project-based approach, whereby exploratory work is done with small groups of academic staff and a range of methods 12 alt-} volume 10 number i piloted. however, almost all of these approaches rely on centralized provision for skills development: this will either be in the form of centrally delivered, optional support for all students, or as self-contained generic units developed by a central team and 'bolted on' to individual subject curricula. practical issues for delivery the combined effects of the issues discussed above impact strongly on the practical delivery of key skills support to university students. day-to-day responsibility for provision tends to lie with a small centrally-located team which must address the needs of all students. sometimes this team may be part of an existing service division, in which case it has the benefit of that support structure, but it may also be a project team working in isolation. centrally delivered skills-related activities (whether optional and isolated from subject delivery or 'bolted on') may be well-intentioned but perceived as irrelevant by the students themselves. budget restraints mean that the printing and distributing of large quantities of paperbased material are not usually possible, or can only be done at critical times (for example, in the first term for first-year undergraduates). the nature of the support material linking as it does to study skills, problem-solving and even revision techniques means that it might be required quickly by students at any time. a small team would probably find it impossible to meet this need on a face-to-face basis, and so the web seems to present an ideal means of overcoming several difficulties: materials can be made available to students via a centralized website, and these will be available at all times and to all students with access to the internet. however, it raises a number of further issues, which will now be discussed. designing web-based resources for key skills a central university website for key skills will often form the baseline support for students' skills development, and many excellent generic websites have been produced to cover a variety of skills, both by teams in the uk and internationally.1 the resources found on these sites are wide-ranging and include study skills guidelines and materials, weblinks, cal packages, references to paper-based information or books, and opportunities for skills development in extra-curricular activities (such as student tutoring, student union activities or part-time work). the teams producing these kinds of sites may already have compiled large amounts of paper-based support materials or produced their own, based on the needs of their particular students. although there are some examples of where these have simply been made available online as heavily text-based webpages (perhaps due to staffing or resource constraints), in most cases they have been adapted for publication on the web. this kind of adaptation is important for usability, and it draws on elements of good practice in web design. guidelines for web usability (for example, nielsen, 2000) centre on ease of use. nielsen focuses particularly on fast access to web-based material: users should be given the easiest and quickest routes to the information they require. text should be 'chunked' into small sections so that it is easily scannable by the reader, and long documents divided logically into separate webpages which can then be interlinked. external weblinks should be selected carefully in terms of their relevance to the user. 13 oaire mcavinia key skills by design: adapting a central web resource to departmental contexts these design considerations are highlighted here because they are important in the context of key skills: learners may need to use this support as they try to complete assignments or job applications, or simply because they have a limited amount of time to give to skills development where it is taking place outside the curriculum. however, the design of an effective generic resource for key skills can also draw on instructional design approaches for the preparation of open and distance learning materials. the learners using a central key skills website are at a distance from the team, if not actually categorized as 'distance learners' by the university. although they may need to find information quickly for one circumstance, they may also seek an opportunity to develop or learn a particular skill in more depth. key skills teams will normally have few opportunities to meet with the students for whom they design provision, but these overlapping requirements in terms of information retrieval and learning objectives must be addressed by the resources they produce. whereas nielsen (2000) advocates design for quick and straightforward information retrieval, rowntree (1994) considers the learners' needs and provides a set of checklists for each stage of the preparation of materials. he discusses specific choices for the writer/designer: the medium, which learning objectives will be met by which materials, the management of text and the tone of the text, appropriate use of illustrations, and clear guidance around the materials for the learner. although his suggestions reflect some of the considerations for a well-designed website, there is one important difference and it is a difference which arguably needs much greater emphasis in the context of designing webbased resources for key skills: the materials should show an 'obvious awareness of different learners' needs' (rowntree, 1994). a key skills team is likely to have or to obtain some data on the student population, such as the distribution of age groups and home/overseas students between various departments, and technical information about how the students will access the web-based resources. all of this will help with the design of a generic central website, but a detailed investigation of the characteristics and needs of the learners raises the issue of different cultures within the university itself. although many key skills teams include members of staff who have taught (sometimes as lecturers within the same university) or worked with students over a period of time, this 'obvious awareness' may not be there for specific subject groups. in order to provide a useful web-based resource, the team may need to progress from nielsen's (2000) view of well-organized, speedily available information, to rowntree's (1994) model of a distance-learning resource designed to meet specific needs, and to be studied over a period of time. designing for different contexts if a web-based resource for key skills is truly to reflect the needs of its users, then it needs to take into account the cultures of the departments in which those learners are located. becher (1989) finds that subject disciplines function in different ways, and that staff identify with the subject community rather than with their respective institutions. this suggests that a 'contextualized' approach to supporting key skills may be more effective than a heavily centralized one. if we treat departments as cultures and design for each in a different way, we are contextualizing our approach to supporting learning. this is 14 alt-j volume 10 number i supported by theories of situated learning, but links also to participatory design, which seeks to include the users in the design and implementation of new technologies (schuler and namioka, 1993). by taking the situated learning and participatory design approaches into account, we might arguably be able to produce materials with greater perceived 'relevance' to the needs of the learners. mcloughlin and oliver (2000) have applied theories of situated learning and communities of practice to the field of instructional design. they argue: 'one of the limitations in current instructional design models is that they do not fully contextualize the learning experience, and are themselves the products of particular cultures.' they make a distinction between local sites, designed for one context and culture, and international sites that reach across many cultures. their use of local and international is not dependent on geographical locations, but rather on culture. in their case, the particular culture is the indigenous australian population, but their work suggests that it may be important to differentiate between subject cultures too. the reasons to design for an ethnic group, or for non-traditional or part-time learners may be much clearer than the reasons to design differently for particular subject disciplines. in designing a model for web-based skills resources at ucl, it was decided to try to establish whether contextualizing the resources for particular departments would make a difference to how the materials were received and used. the ucl model the ucl key skills website was developed to support a wider key skills pilot, which ran during the academic session 2000-1. this pilot stemmed from an initial audit of their curricula for key skills by departments, and the subsequent compilation of a list of skills which tutors felt were essential for their students to have on graduation. a paper-based information for staff information for undergraduate and graduate students ijclk skill / \ eysk llsgr d departmental skills webpages resource resource resource resource resource figure i: model of the ucl key skills website 15 claire mcavinio key skills by design: adapting a central web resource to departmental contexts profile was distributed to first-year undergraduate students from six departments, asking them to assess their own skills and to identify those needing further development. they were then encouraged to use the website to support this process. the design of the website (shown at figure 1) includes two approaches to the delivery of the resources: a centralized site for key skills and also customized webpages for departments. as can be seen from the diagram, the customized pages draw on the resources at the central site, but provide a different 'front end' from these: this has the aim of 'personalizing' and familiarizing the resource to enhance usability, so that the student can find his/her way around the materials without having to use the central 'anonymous' skills page. this was not a case of simply 'rebranding' the site with the department's house style for its own webpages: rather, it was designed to allow departments to prioritize certain skills at certain times in a course, and to allow students to take a particular route into the resources via a course topic or type of assignment. a number of wider potential benefits were also intended: the departmental skills pages might give ownership of skills development back to departments without adding to the workload of the academic staff from those departments, and potentially help skills development to become integrated as part of the department's provision. in terms of design, the two means of accessing the support material (generic and subjectspecific) reflect aspects of the two models discussed above: nielsen's (2000) idea of a clearly organized, 'just-in-time' resource is important in the design of the central website. rowntree's (1994) guidance for the design of distance-learning resources was also considered, but arguably to a greater extent in the case of the customized, subject-specific sites tailored to particular objectives. however, a variety of models might be called on in addressing the needs of different subject disciplines: mcloughlin and oliver's (2000) course design principles for 'culturally inclusive instructional design' suggest that a range of perspectives may have to be accommodated by the design of resources. this is important in avoiding a scenario where one culture appears to present another with a 'deficit model', that is, where a central team or potentially even a course team from a subject discipline may present one design of the resources as the ideal. mcloughlin and oliver (2000) refer to creating access to a variety of resources in order to ensure multiple perspectives: the tailored departmental pages provide an opportunity to pool supporting resources, be they subject-specific (for an assignment), generic (for the study skills or other skills used in completing that assignment), or a combination of both. in many discipline areas there are supporting subject-specific skills materials available. a customized webpage provides tutors with an opportunity to highlight these, and they can bridge the gap between generic and discipline-specific resources. a further issue relating to content arises in the context of the language and terminology of key skills. skills lists, and the materials designed to support the development of skills, may need to be adapted to the language of a subject discipline. the definition of a particular skill written by the key skills team may have quite a different meaning within a subject discipline, and the members of that discipline may perceive it as reductive. alternatively, such a definition may be irrelevant to the subject discipline where that skill encompasses the whole subject area. one example might be that of numeracy, where a key skills definition would essentially be meaningless to the members of a mathematics department. 16 alt-j volume 10 number i therefore, in approaching particular departments to discuss the customization of our resources for their students, we encouraged them not only to consider the design, organization and content of the webpages, but to define what they meant by each of the skills they hoped the sites would support. they were also asked to look for opportunities to provide authentic ways of using the key skills materials within the coursework, and although this proved difficult it yielded some positive results which are now discussed. examples of contextualized resources for key skills this section discusses the preparation of two sets of customized key skills pages: one each for the departments of geography and biology at ucl. both sets of pages were produced as part of the key skills pilot discussed above. department of geography for this department, the format and structure of the customized pages was discussed with academic tutors and an academic administrator. the pages were then designed in their departmental 'house' style but they were stored with the key skills website during the pilot phase. figure 2 shows the customized key skills home page for geography, and figure 3 the key skills homepage which conforms to the central ucl web design. (stiff & pasturejust: !t*erch" ; n ] p, h s s5t»1 * i s ] fee top «f fi« page to retsrs to the a a a • j * ; : • ] -"n?*.~f"r|^hil-u! ljhl.':^>'';-ftr-i to fea page at asy t=2f by clickir^ oa feis hsacing. r •"* ' " ^ r j ^ ^ » j ^ . ^ ^ ^ available shorsy; search all &• resources o s feis site. 5 * * * ^ m 3 5 s j ' ^-" *v-!tfit?*^ a selector of resources to si55port stadests a this area. 2.esources organised aroind &s ucl key skss grid. access the defeated sk£s package s-osi cere. iofor~aiio= for stadeats recording skjbs (boa fee isaia site). sor;e baci^roirid i f o r ^ a i o a aad resovrcss. figure 2: the geography key sk(7/s homepage 17 claire mcavinia key skills by design: adapting a central web resource t o departmental contexts par •efa £<s view £ o back i " . " 3 i befead hone s e a c h netscape si31 |gj st's boated picas* r>«* thai this site is currency under developtnen; &y gpp. . j 11 ^44 (£20 7c79 2000 © ucl 1588 2000 rii > figure 3: the main key skills homepage mentioned previously was the idea that the customized pages could be used by departments to prioritize particular skills at particular times. in this case, the department wished to focus on numeracy skills for one of their undergraduate courses. they felt that resources which would help students to revise some of their skills, and then lead into more complex skills such as data analysis, would be of help in supporting this course. the selection of appropriate resources to be included depended on discussion about what 'numeracy' meant for these courses and these students. the links included quality generic resources as well as the staff homepages of the course coordinator, and a range of material he had gathered or noted, but had not previously been able to present to the students in the context of supporting their coursework. the students were asked to use the site to prepare for a student-led session, and subsequently reported informally that it had been useful for this class. formal evaluation from a questionnaire, distributed to the year group as a whole, indicated greater usage of this site and the central site by these students than for students across five other key skills pilot departments. although this finding is based on questionnaires distributed to a relatively small group (25 students), it suggests that the inclusion of the contextualized resources within course activity had a positive impact on the learners. the tutors' feedback from the department suggested that they were giving further thought to the ways in which web-based resources might be used in this area, and it seems likely that the department will endeavour to continue this work independently. 18 alt-j volume 10 number i department of biology customized pages were designed for the department of biology in the latter half of the pilot phase, when the web development work was almost complete. these are being accessed by students from that department in the current academic session, and as a consequence, there is no formal evaluative data available. this case differed slightly from that of the geography department, in that there was no obvious course selected for which the pages could immediately be used as support. this highlights a difficulty in key skills delivery: tutors may have no previous experience of explicitly describing and supporting skills in their curricula, and therefore may be unsure of where such support can best be given. in this case, the tutor involved suggested that the work might be based on an existing audit of key skills within the courses offered at undergraduate level by the department. the department had produced a 'map' of key skills in the undergraduate curricula and represented this by means of a paper-based diagram of skills gained in each unit of the course. the paper-based diagram was converted to an 'image map' (shown at figure 4) around which the key skills resources were organized. students using the map online are directed from a course unit to details of where the key skills will be gained in that particular unit, with these in turn being mapped to supporting materials. fie view £ o an overview of key skills acquisition qaths page we jasst =chde<j a web-based tokos of 4 e chart oissssg biology csgtt courses, a d fee key zk3h acquired as each stage. tie 2*ap below is ceckahe: w5aa 70s mow ±z aosse over it, you w£ set ass the ecrsor <&scges shape to iterate a &&. 'wcert you tick oa fee tssoss =ksfo-^io2s. a sew vradow ws pop up gszisg a (iescrbioa of that part of &e course and suggesting some £&s froc fee mah key sk£s sb: sai could be of use for fcst aspect of your dtgrei cowsework. e r 1 forma! courses .' informal courses; j. i year 2 degree progfartsties • genaabstpatsj year 1 degree programmes core cobjses click here far the !:st of courses zni relevant links vear2 degree pragraranescare courses biology {ce8 and molecular biology pa»5 careers ta& i cere corses genescsznd hwraan gcnescs 2nd year i careers tafe" document; done figure 4: the biology key skills image map 19 claire mcavinia key skills by design: adapting a central web resource t o departmental contexts the department has expressed a positive response to the resource and it provides an interesting model in terms of the organization of the undergraduate curriculum around key skills. however, although it adopts the approach of customizing resources, it does not include them as part of course activity. without evaluative data it is impossible to assess whether this approach will be more or less successful than that adopted for the department of geography, and further evaluation with the tutors and students concerned is required. lessons from the pilot design considerations this pilot work involved two approaches to the design and delivery of web-based key skills resources: the central resource and the customized, individual sets of departmental pages. it has demonstrated that a central resource provided in tandem with tailored, customized resources for departments may be more effective in delivering support for skills development to undergraduate students than the purely centralized or 'bolt-on' provision of generic materials. the design of the central resource may benefit from a consideration of the principles of designing for distance learning, as well as those of good website design, although here it was difficult to gather large amounts of data to support this view. the adaptation of the resources to particular subject areas is not simply a question of changing the packaging or 'rebranding' them. it requires a meaningful dialogue with members of the department about the customization of the pages, and could even be considered in terms of a participatory approach. it is important to find and include authentic, contextually relevant material on these pages as well as links to the generic skills development material, and perhaps most important is to encourage tutors to find opportunities to incorporate the material within course activity. without these opportunities the customized resources risk having to be 'bolted on' as generic material has been in the past (fallows and steven, 2000). practical considerations work on the ucl key skills site was undertaken by a web developer (the author) whose time was exclusively devoted to the design and population of the central web resource and, over time, the customized key skills pages for departments. the outcomes of the pilot suggest that the approach has been effective, but in turn imply that this kind of approach needs a dedicated web developer on the key skills team. the location of the web developer within the team may also be beneficial in terms of awareness of the issues in key skills, skills terminology and the range of existing generic resources. future development the final phase of the pilot envisaged the departments taking responsibility for the pages after the initial pilot phase ended. this process is not complete at ucl, and the transfer is not something we have pressurized the pilot departments involved in this work to do. rather, they have been encouraged to think about who would be best placed to take responsibility for the pages and to expand the resources. not every department will have a dedicated web developer, and in some departments websites are maintained centrally or by an academic who already has a large number of other responsibilities. therefore, the transfer may not be something that can be done quickly. 20 alt-j volume 10 number i conclusions in spite of the varying extent to which skills policies have been implemented in he, and the many issues they raise, the evidence of the potential benefits to students of developing their key skills are now becoming clearer (fallows and steven, 2000). students are being encouraged to reflect on their learning, to take responsibility for it and to formulate their own objectives for university and life afterwards. they are also learning a language to describe their abilities and to record their achievements, and are effectively laying the foundations for their lifelong learning. although the delivery of skills support to students is problematic, patterns as to how it can best be done are starting to emerge. drew, shaw and mowthorpe (2000) suggest a continuum whereby generic key skills materials and activities are gradually integrated into coursework, and they document the increased usage of the resources by students where this has been done. this would seem to support the model presented here: a well-designed, generic web-based resource may be an effective (and cost-effective) form of baseline provision, but the benefits to students may be greater by adapting and eventually including such material within course activity. the short timescale of the project described here means that further research would be necessary to determine its long-term impact, and to gather additional evaluative data from the tutors and students involved. however, it does perhaps demonstrate that the nature of key skills support is evolving: if such support is to rely on the web as its keystone, then a 'decentralizing' approach may have to be built in to planning and budgeting processes. one of the most recent surveys of key skills development as a whole in higher education includes a number of accounts of where face-to-face skills development is now taking place with individual departments, and is considered to be essential in taking the initiative forward (fallows and steven, 2000). it may well be the case that web-based support will follow suit. a related issue is that of responsibility for the design and development of the website(s) for a key skills team. if this work is done by a member of the. team, then perhaps that role could increasingly be described as that of a learning technologist or 'new specialist' (beetham, 2001). this may have implications for the needs of such team members in terms of staff development and future responsibilities. where web development work is carried out centrally, there may be a need to examine the resource available for this if support is to be tailored to specific departments. the likelihood of future initiatives needing the same, or greater, degrees of technical support might also have to be examined by the institution. one example of this appears in the related initiative of student profiling, which has the potential to make heavy demands on a range of systems (including virtual learning environments and student records systems) once implemented online. the crossover between learning technologies and web-based support for key skills is again highlighted by the final report of the continuation project for key to key skills (drew, catto and anderson, 2001). in one institution the integration of the project's key skills online package was the catalyst for the institution to begin formulating a policy on online learning. in another case, the implementation of the package was a trigger to frame their key skills policy. therefore, it seems likely that the design of this kind of provision will continue to change and grow along with the technologies themselves. 21 claire mcavinia key skills by design: adapting a central web resource to departmental contexts notes 1 many examples have been included as hyperlinks from the ucl key skills site: http://www.ucl.ac.uk/keyskills/grid.html references atkins, m. (1999), 'oven-ready and self-basting: taking stock of employability skills', teaching in higher education, 4 (2), 267-80. becher, t. (1989), academic tribes and territories: intellectual enquiry and the culture of the disciplines, srhe/open university press. beetham, h. (2001), career development of learning technology staff: scoping study final report, http://www.sh.plymouth.acuk/eds/effects/jcalt-project/ bennett, n., dunne, e. and carre, c. (1999), 'patterns of core and generic skill provision in higher education', higher education, 37, 71-93. bennett, n., dunne, e. and carre, c. (2000), skills development in higher education and employment, milton keynes: open university press. dfee/ncihe (1997), report of the national committee of enquiry into higher education: higher education in the learning society, recommendation, 23. sheffield: crown copyright. dfes (2001), education and skills: delivering results, a strategy to 2006, suffolk: crown copyright. drew, s. (1998), key skills in higher education: background and rationale, seda special no. 6, birmingham: seda publications. drew, s., shaw, m. and mowthorpe, d. (2000), key to key skills project final report, http://www.shu.ac.uk/keytokey/finalrep4.pdf drew, s., catto, c. and anderson, v. (2001), tqef transferability and continuation funding for tltp3 key to key skills project: final report, http://www.shu.ac.uk/keytokey/ cont_final_rep.htm fallows, s. and steven, c. (eds) (2000), integrating key skills in higher education, london: kogan page. hesketh, a. (2000), 'recruiting an elite? employers' perceptions of graduate education and training', journal of education and work, 13 (3), 245-71. mcloughlin, c. and oliver, r. (2000), 'designing learning environments for cultural inclusivity: a case study of indigenous online learning at tertiary level', australian journal of educational technology, 16 (1), 58-72. nielsen, j. (2000), designing web usability: the practice of simplicity, indianapolis: new riders qaa (2001), benchmarking academic standards: benchmarking statements, http:// www.qaa.ac.uk/crntwork/benchmark/benchmarking.htm 22 alt-j volume 10 number i qca (2000), national qualifications: key skills specifications, http://www.qca.org.uk/nq/ks/main2.asp rowntree, d. (1994), preparing materials for open, distance and flexible learning, london: kogan page. schuler, d. and namioka, a. (eds) (1993), participatory design: principles and practices, mahwah, nj: l. erlbaum associates. 23 does the mode of delivery affect mathematics examination results? d.j. fiddes* a. a. korabinski* g. r. mcguire * m.ayoungson* and d. mcmillan** *heriot-watt university **scottish qualifications authority email: g.r.mcguire@hw.ac.uk at present most examinations are delivered on paper but there is a growing trend in many subjects to deliver some or part of these examinations by computer. it is therefore important to know whether there are any differences in the results obtained by candidates sitting examinations taken by computer compared with those obtained by candidates sitting conventional examinations using pen and paper. the purpose of this article is to describe the outcome of a pilot study designed to* investigate possible causes of any differences in results from the use of different modes of delivery in a mathematics examination. one outcome of this study was that the process of translating examination questions into a format required for use on the computer (but keeping this as a pen and paper test) can have a significant effect on examination results. however, the main conclusion is that changing the medium only has no effect on the results in mathematics examinations. introduction there is a growing trend in many subjects to deliver some or part of examinations by computer (bull and mckenna, 2001; lawson, 2001). indeed, in some areas banks of suitable questions are being assembled to allow future examinations in some undergraduate subjects to be taken on paper or by computer (white, 2001; sims-williams, 1999). very little is known, however, about the effects of the medium in testing basic skills. some studies have been conducted using multiple-choice tests (for example lee and weerakoon, 2001, in the area of health education) but mathematics examinations taken on paper and by computer normally consist of questions which the candidate attempts by giving their answer to each question as a number or more generally as a mathematical expression. an up-to-date review of computer-aided assessment in mathematics is presented in a chapter of a recently published book on effective teaching and learning in mathematics and its applications (beevers and paterson, 2002). 60 ak-j volume 10 number i examinations taken by computer have several advantages, not least of which is instantaneous marking and feedback. however, they are different from paper examinations in at least two ways: • the questions for a computer test require varying amounts of rewording and adjustment in layout compared with the same questions in a paper test in order to write them into a computer assessment package. we shall call this the rewording effect. • in a computer test, students read questions from the screen and answer these questions by typing in numbers or more general mathematical expressions at specific places (for example an answer box) on the screen. we shall call this the medium effect. these two effects can create differences in the results of the examination process. if any differences were found between the marks from a paper test and those from its computer version this might be due to either the rewording or the medium effects, or both. the main aim of our project was to devise an experiment to separate these two effects and investigate the significance of each. several other factors that might lead to differences between candidate performance in a paper examination and in one using computer delivery include the motivation of the candidate to do well in the examinations, the level of anxiety felt by the candidate when sitting the examinations and the familiarity of the candidate with the assessment software delivering the examinations on computer. the efforts made to deal with these factors through the timing of the experiment and in preparations for the actual tests of the experiment are outlined later in the section on running the experiment. setting up the experiment the scottish qualifications authority (sqa) higher mathematics examination has a high uptake in scotland with virtually all candidates having previously taken examinations from this board. for the experiment, three different test papers were supplied by the sqa, each containing short response questions typical of those found in paper 1 of higher mathematics. the questions used in this pilot project covered most topics in the syllabus with the answers to many questions requiring general mathematical expressions as well as numbers. the marking scheme provided by the sqa was that used in the higher mathematics examinations where credit is awarded for each key skill required to be shown by a candidate. in this experiment there was no intention to investigate the most appropriate key skills required in questions, as this would introduce more variables into the experiment. the intention was only to compare current examination practice with its possible computer replacement, and so the marking scheme provided by the sqa was adhered to throughout the experiment and no consideration given as to whether the most appropriate key skills were being examined. while the time allowed for each test was thirty minutes, it was expected that candidates would be able to complete the paper in less time than this so that, in general, time would not be an issue. we use the abbreviation 'p format' for this type of paper test. the questions in the p format tests were then converted into computer test questions as required by the cue assessment package (beevers, 2000). further details of the cue assessment system and online demonstrations of tests can be obtained at the calm project website (calm group, 2001). we use the abbreviation 'ict format' for this type of computer test. in order to separate the medium and rewording 61 d.j. fiddes et o\ does the mode of delivery affect mathematics examination results? effects, a third type of test was produced, called 'rt format', which is the reverse translation of the computer test and was basically a screen dump of the questions in ict format. examples of p format and ict format questions are shown in figures 1 and 2. paper question (p format) 1. if f[x) = '2x 3 and g{x) = 2a.-2 3 find an expression for g{f{j:)). write your answer in the form ax2 i fer-l-c . figure i: example of paper question (p format) if flc) = 2x 3 and g ( x ) = 2x2 3 then what are a , £> and c ? = ax2 + £x + c 1-1) what is a ? your currently accepted ansvrer: : submit 1-2) what is b ? : ; submit: your currently accepted answer: 1-3) what is c ? r your currently accepted answer: • submit figure 2: example of computer question (id format) 62 ak-j volume 10 number i although in the examples shown it would have been fairly easy to make the ict question the same as the p question we wanted to use questions in this project which were reworded so that the rewording effect could be investigated. since the rt format has exactly the same words in each question and exactly the same place to insert the answer, comparison of the marks between the ict and rt format (both marked in the same way) should determine the significance of the medium effect. as both the p and rt format tests are paper tests, there is no change in the medium. however in marking the p format, working is taken into account in giving partial credit for answers that are not correct but in which a candidate has shown some of the skills required to tackle the question. the same could be done with rt format. space was provided opposite each question for rough working so that in the marking process this working could be taken into account. therefore the rt paper was marked in two ways. the first was only to mark the answer, which we called rtc marking, since it was exactly how the answers in ict format tests were marked. the second was called rtw marking since it included giving partial credit for the rough working. hence comparison of ict with rtc marks investigates the medium effect and comparison of p with rtw marks investigates the rewording effect. running the experiment pupils from two schools, 18 from falkirk high school and 50 from queensferry high school were invited to participate in the project. all pupils were in their fifth or sixth year of secondary school. there were 40 males and 28 females. prior to carrying out the experiment, each school was visited, and the pupils were given details of what the project entailed and what would be expected of them. to give them some practice with inputting mathematical answers, a trial ict test with 5 questions was set up and the pupils were given the opportunity to do this test when help was available to answer their queries about any aspect of what was required to sit the test. the questions in this trial ict test were in general much easier than higher standard, but, unlike the questions involved in the project, random parameters were incorporated into the questions. the trial test was available to the pupils from the day of the visit until the day of the experiment so that they could practice as often as they wished beforehand. by introducing random parameters they would get a different test each time they ran the trial. practice with the trial test would allow pupils to minimize any navigational or inputting difficulties they might have during the computer test and help them gain some familiarity with this new type of test. some candidates took a lot of advantage of this practice, while others did not. to enable a paired statistical analysis to be performed on the results of the experiment three groups of pupils were set up at each of the schools in such a way that each group had roughly the same mixture of mathematical ability and gender. their mathematical ability was estimated from knowledge of their previous sqa examination and higher preliminary examination results. the actual tests took place during or just after school hours at each school in april 2001. this was in the small window of opportunity between the time when the pupils had covered enough of the material in the higher syllabus and before the period when their sqa examinations started. this was a time when the pupils were motivated to do well in 63 d.j. fiddes et al does the mode of delivery affect mathematics examination results? the tests, having been encouraged to regard the tests as good revision for their approaching higher examination. each candidate took each of the three tests, one in each of the formats, over a 90-minute period. this was done in such a way that each group sat the three tests in different orders and also that, at any time, no group was sitting the same test or taking a test in the same format as any other group. the computer tests were delivered over the web with the results being marked and stored at heriot-watt university as the pupils sat the tests. during the computer tests, the candidates did not experience either any difficulty in navigating within a test or any delays when submitting answers. pupil feedback the pupils were asked informally by questionnaire what they thought of the tests. out of a possible 68 pupils, 54 took this opportunity to express their views. the results give some indication about how the pupils felt about the tests they took in the project. • 47 per cent preferred the paper test, 25 per cent the reverse translation test, 4 per cent the computer test with 24 per cent expressing no preference. • 4 per cent found the paper test the most stressful, 4 per cent the reverse translation test, 69 per cent the computer test with 23 per cent finding no difference in stress levels between the tests. • 7 per cent found the computer tests a bit easier than the paper tests, 62 per cent found them a bit harder while 31 per cent found them to be much the same in terms of difficulty. • 20 per cent thought that using the computer was a better way to be tested in mathematics, 65 per cent thought it was not a better way and 15 per cent felt there was no difference. although these results were of some interest, they were not used in the subsequent analysis of the experiment. the pupils who took part in this project had been used to taking tests on paper throughout their school careers. considering how little experience they had with computer tests, perhaps these views on the computer tests are not surprising. marking and analysis after the pupils sat the tests they were marked as follows. the p format papers were marked by the sqa as though they had been higher examinations. the rtw examinations were marked at heriot-watt using the sqa marking scheme. the ict tests were marked (automatically) by computer, while the rtc examinations were marked at heriotwatt using the same marking scheme that was used by the computer. the statistical analysis involved two separate comparisons. the first was to investigate the rewording effect by comparing p marks with rtw marks. the second was to investigate the medium effect by comparing ict marks with rtc marks. within both schools each of the three tests was taken in each of its three forms by one of the three groups of pupils. with information available on the ability of the pupils, standard grade mathematics results for the falkirk pupils and higher mathematics preliminary marks for the queensferry pupils, matched pairs were constructed. for 64 alt-] volume 10 number i example, one falkirk pair consisted of two male pupils both with grade 1 in standard grade mathematics, such that one of the pair sat the p version of test 1 while the other sat the rtw version of the same test. whereas one queensferry pair consisted of two female pupils scoring 67 and 68, respectively, in their preliminaries, such that one sat the ict version of test 3 while the other sat the rtc version of the same test. in this way 14 matched pairs were created from the falkirk pupils, 11 being matched for ability and gender and 3 for ability but with different genders, and 48 from the queensferry pupils, all being matched for ability and gender. this resulted in a total of 62 matched pairs. due to absences of pupils on the day of the tests a very small number of test marks were not used. statistically a matched pair analysis provides the most efficient use of the data and the sample size of 62 was large enough to validate the analysis without the need for any assumptions such as requiring a normal distribution (mcghee, 1985). the two statistical analyses were based on the 62 differences in marks for the matched pairs of pupils, either (p rtw) or (itc rtc). in each case the null hypothesis was that the true underlying mean difference was zero against a two-sided alternative hypothesis. a onesample t-test on these differences was performed using the statistical package minitab with the following results. the rewording effect the mean of the 62 observed differences was -2.3 marks so that rtw marks were greater on average by 2.3 marks in tests, which were marked out of 21 or 22 (see table 1). n mean stdev se mean pmark rtw mark difference 62 62 62 8.484 10.774 -2.290 4.742 4.723 4.194 0.602 0.600 0.533 95% cl for mean difference: (-3.355, -1.225) t-test of mean difference = 0 (vs not = 0):t-value = -4.30 p-value = 0.0005 table i: paired t-test and confidence interval for p mark — rtw mark the observed t-statistic is -4.3 with a probability-value of less than 0.001 to 3 d.p., which is highly significant, giving very strong evidence of a difference between the p marks and the rtw marks. however, it was noted that the p format papers were marked by sqa while the rtw papers were marked at heriot-watt using the sqa marking scheme. therefore a potential source of variation between the two sets of marks could have been due to the different markers and not just the rewording effect. in order to eliminate any such marker effect and so isolate the rewording effect the p format papers were remarked at heriot-watt by the same marker as for the rtw papers, using the same marking scheme. this resulted in some small changes and the data were re-analysed to give the results in table 2. the set of p marks obtained from remarking is referred to as pmy mark in the output. 65 d.j. fiddes et al pmy mark rtw mark difference does the mode n 62 62 62 of delivery affect mathematics examination results? mean 9.065 10.774 -1.710 stdev 4.627 4.723 4.079 se mean 0.588 0.600 0.518 95% cl for mean difference: (-2.745, -0.674) t-test of mean difference = 0 (vs not = 0):t-value = -3.30 p-value = 0.002 table 2: paired t-test and confidence interval for pmy mark rtw mark so the mean difference is now -1.7 marks and this can be said to be due to the rewording effect. the probability value is 0.002, which still provides very strong evidence of a difference due to the rewording effect. these results are illustrated in figure 3, which gives a histogram of the 62 observed differences together with a 95 per cent confidence interval for the underlying mean and the null hypothesis mean presented below the histogram. the fact that the null hypothesis (ho) mean is well outside the confidence interval illustrates the very strong evidence of a difference due to the rewording. 1 0 f re qu en cy i i histogram of the (p rtw) differences (with ho and 95% t-confidence interval for the mean) i • •:••-•• x ho i i i i i i i i i i -12 -10 8 6 4 2 0 2 4 6 (p rtw) d fferences figure 3 the medium effect the mean of the 62 observed differences was only -0.2 marks so that rtc marks were greater on average by 0.2 marks in these tests. in this case this is not a significant difference (see table 3). the probability value is 0.57 showing that the observed differences could easily have occurred by chance. again this is illustrated in figure 4. this time the null hypothesis mean 66 is well inside medium effect cmark rtc mark difference the n 62 62 62 confidence interval, mean 7.479 7.065 0.415 indicating no stdev 5.654 4.534 5.738 evidence of se mean 0.718 0.576 0.729 alt-j volume 10 number 1 a difference due to the 95% cl for mean difference: (-1.043, 1.872) t-test of mean difference = 0 (vs not = 0):t-value = 0.57 p-value = 0.572 table 3: pairedt-test and confidence interval for c mark rtc mark f re qu en cy o <7 i o 1 1 1 histogram of (c rtc) differences (with ho and 95% t-confidence interval for the mean) r— — — t ] e——1 i i i i 20 -10 0 10 (c rtc) differences figure 4 additional comments the average mark over all the tests was 39 per cent, with the individual marks varying from 0 per cent to 100 per cent. although an investigation into a gender effect was not the main purpose of this investigation, a comparison was possible due to the fact that all but three of the 62 pairs were matched for gender. accordingly the above analyses were repeated for the male pairs and for the female pairs. the results showed that for both the rewording effect and the medium effect the conclusions were the same as above. conclusions and directions for further study the difference in marks between p format and rtw format may be due to the rewording, the formatting (in terms of the number of questions on a page) or the fact that the working for the answers in rtw format was not written in a linear way. in p format, the answers were written with one line following the other making it fairly clear where any error 67 d.j. fiddes et al does the mode of delivery affect mathematics examination results? occurred. if two answers were given in rtw format with one correct and one wrong, it was not as clear which answer the pupils had meant as their answer since, for example, both answers may have appeared side by side. accordingly, pupils may have been given the benefit of the doubt on occasions. this might be a reason why the rtw marks were higher than p marks. a more detailed study may be required to clarify this issue. candidates had little prior experience of computer tests. they were also required to use different mathematical symbols during the computer tests when typing answers into the computer from those with which they were familiar in paper tests. so even with the small amount of practice candidates may have obtained in the trial test beforehand, many would still have some anxiety when sitting the computer test. despite all of this, the major conclusion drawn from this project is that the medium has no effect on the marks for these tests. the cue assessment system has evolved following a number of educational experiments over the last fifteen years. one of its important pedagogic enhancements occurs with the introduction of steps in questions (beevers, mcguire, stirling and wild, 1995). steps are a possible way of providing partial credit in computer tests (beevers, youngson, mcguire, wild and fiddes, 1999; lawson, 2001). there were no steps in any of the questions in the current project. a second investigation is planned into the effect of steps on exam results. references beevers, c. e. (2000), 'computer aided assessment in mathematics at heriot-watt university', maths, stats and or newsletter, 1, 17-19. beevers, c. e., mcguire, g. r., stirling, g. and wild, d. g. (1995), 'mathematical ability assessed by computer', computers and education 25, 3, 123-32. beevers, c. e., youngson, m. a., mcguire, g. r., wild, d. g. and fiddes, d. j. (1999), 'issues of partial credit in mathematical assessment by computer', alt-j, 7, 26-32. beevers, c. e. and paterson, j. s. (2002), 'assessment in mathematics', in p. khan and j. kyle (eds), effective teaching and learning in mathematics and its applications, london: kogan page. bull, j. and mckenna, c. (2001), 'blueprint for computer-assisted assessment'. available from: http://www.caacentre/bp calm group (2001), 'cue assessment system'. available from: http://www.calm. hw.ac.uk/cue.html/ lawson, d. (2001), 'computer-aided assessment in relation to learning outcomes'. available from: http.//ltsn. mathstore. ac. uk/articles/maths-caa-series/ lee, g. and weerakoon, p. (2001), 'the role of computer-aided assessment in health professional education: a comparison of student performance in computer-based and paper-and-pen multiple-choice tests', medical teacher, 23, 152-7. mcghee, j. w. (1985), introductory statistics, st paul, mn: west publishing. 68 alt-j volume 10 number i sims-williams, j. (1999), 'open testing with a large databank of multiple choice questions', teaching mathematics and its applications, 18, 159-61. (also http://www.tal.bris.ac.uk/). white, s. (2001), 'electrical and electronic engineering assessment network'. available from: http://www.e3an.ac.uk/ 69 the importance of task appropriateness in computer-supported collaborative learning kathy buckner and kate morss queen margaret college, email: k.buckner@mail.qmced.ac.uk collaborative learning activities can be beneficial for exchanging ideas, sharing experiences, and developing shared understanding. it is our view that the task given to the student is central to the success or otherwise of the learning experience. in this paper, we discuss the need for the adaptation of traditional face-to-face tasks when these are incorporated in computer-supported collaborative learning (cscl) environments. we focus on critical issues in relation to the implementation of cscl tasks including: the appropriateness of the medium for the task, the role of individuals, the volume of work involved, the time allocated for tasks or sub-tasks, and, the assessment procedures. in this paper we describe and evaluate two case studies that illustrate the importance of the appropriateness of the task in computer-supported co-operative learning. both case studies (taught by one of the authors) involve final-year information management undergraduates in on-campus modules. our findings indicate that the task must allow the module outcomes to be achieved (and assessed if necessary), and must be supported by software tools which facilitate effective student learning. if it does, and the assessment mechanism is appropriate, the benefits include improved motivation to participate in discussion, improved student learning, and the ability to assess individual performance in group activities. introduction the study of learning in collaborative electronic environments is becoming established as computer supported collaborative learning (cscl) an emergent sub-discipline of the more established computer supported co-operative work (cscw) discipline (webb, 1995). using computers for the development of shared understanding through collaboration has been explored by crook who suggests that success may depend partly on having a clearly specified purpose or goal (crook, 1994). it is our view that the appropriateness of 33 kothy buckner and kate morss the importance of task appropriateness in computer-supported collaborative learning the task given to the student is central to the success or otherwise of the learning experience. however, the tasks that are given to facilitate collaborative learning in face-toface situations are not always suitable for direct transfer to the electronic medium. it may be necessary to consider redesigning these tasks in relation to the medium in which they are to be undertaken and the functionality of the electronic conferencing software used. the way in which cscl tasks are implemented in terms of assessment of learning outcomes (allocation of marks to individuals and/or to the entire group), and group process and operation (e.g. anonymity, role of individual, role of tutor) is also significant. the enforced requirement (or otherwise) to participate in assessed or non-assessed activities is likely to affect the amount and quality of contribution in any cscl activity (mcateer et ah, 1997). slavin (1996) examines in depth the theoretical perspectives which could help to explain student achievement in co-operative learning (not necessarily cscl) situations. he suggests that factors affecting achievement include group and individual accountability, social and group cohesion, and the ability to learn directly from peers and indirectly from supporting and instructing peers. these are issues that we consider in the work presented here. in this paper we describe and evaluate two campus-based case studies that illustrate the importance of task appropriateness in computer-supported collaborative learning. the case studies case study i: computer-assisted learning: electronic seminar presentation the first case study is one element of a final-year optional module in computer-assisted learning provided on the ba (hons) information management course at queen margaret college. as part of the assessment, students used cow software (conferencing on the web, http://thecity.sfsu.edu/cow2/, an asynchronous computer conferencing program, hosted by netlinks http://netways.shef.ac.uk/ at the university of sheffield) to deliver an electronic seminar presentation, lead and finally summarize the ensuing discussion of the topic. students were graded according to their ability both to lead and to contribute to the debate. each student was asked to select a topic for their seminar presentation and two or three students (in a class of 13) were assigned as conference leaders to each of six topics. in their role as conference leaders students were required to make an individual opening statement of 500 words incorporating appropriate references from print and electronic sources by a specific date. contributors had one week to make two contributions to each of the conferences they were assigned. they were required to take into account the originator's opening statement and contributions from other participants, incorporate additional evidence from print and/or electronic sources, and reflect on their own experience. the discussion was followed by a 500-word closing statement by each conference leader. the conference leaders were expected to be proactive in stimulating discussion. each student was assigned by the lecturer as a contributor to two additional conferences and was required to participate in one further conference of their own choice. this element of the module accounted for 33 per cent of the marks. no group marks were awarded for this activity. case study 2: human factors for information managers: problem-solving the second case study is one element of an undergraduate final-year compulsory module in human factors for information managers which operates within a networked learning 34 aii-j volume 7 number i environment (nle) (buckner and davenport, 1996). the collaborative aspect discussed here involved the students in a problem-solving activity that used netscape navigator news as a platform for asynchronous discussion and electronic submission of an agreed solution. each group of four students (in a class of 24) was allocated to a specific problem which was accessible via the nle. much of the information required to solve the problem was available within the nle, although traditional print material and other online resources were also utilized. students were informed that all discussion and debate should take place online in the newsgroup and that this electronic record would form the basis for the individual and group assessment. additionally they were allowed to use email to pass documents between each other provided that they copied any such transactions to the tutors. it was felt that this might be an easier mode of working as they neared a final outcome for the group solution to the problem. students were advised to work together as a team to solve the problem and it was suggested that they might want to assign different roles to specific individuals, e.g. moderator/team leader, or to take responsibility for gathering information on specific topics. the final solution was to be no more than 500 words long. a total of 30 per cent of the marks available for this module was allocated to the problem-solving activity, 15 per cent for the individual contribution and 15 per cent for the group solution to the problem. evaluation of the case studies a multi-instrument evaluation methodology was adopted for both case studies (draper et ah, 1994). questionnaires with a mix of open-ended and attitudinal questions were used to obtain feedback about the tasks undertaken, the appropriateness of the software provided for performing the task and operational issues. there were nine respondents from case study 1 (69 per cent), 18 respondents from case study 2 (75 per cent). a focus group discussion (involving three students who had participated in both case studies) was undertaken by an independent researcher to enable the issues to be explored in more depth. in addition, the students' assessments (electronic discussion, electronic conference presentations and debates) were examined. analysis and discussion in both case studies the students were quite clear about the nature of the exercise (the task). their roles within the conferences and newsgroup were understood, although these were more clearly defined by the tutor in the seminar presentation case study. in case study 1 (seminar presentation) more than three-quarters of the students thought that they were better infonned both as a presenter (or seminar leader) and as a contributor than in face-to-face discussions. there was less tutor intervention than in face-to-face seminars where student contributors are often reluctant to take an active part in discussion about another student's topic. the tutor and students thought that contributions to discussion were of a higher standard than in traditional seminars. students felt that: i) it was easier to recap on earlier parts of the discussion, ii) they were able to follow up, in an infonned way, on responses from other students, and iii) the medium encouraged more open participation (particularly by the more reserved students). 35 kathy buckner and kate morss the importance of task appropriateness in computer-supported collaborative learning referring to traditional seminars one student stated: . . . in the classroom situation . . . maybe two or three people speak all the time and people tend to sit back and not bother to say anything. a second student agreed: . . . or if somebody is actually saying what you are thinking there is no point. . . you coming out and saying it as well. in case study 2 (problem-solving), students felt that using newsgroups was beneficial for sharing research findings but made it harder to determine a final solution to the problem than if they had been using face-to-face communication. there was a feeling that they were 'waiting' for responses, that it was more time-consuming and less 'instant' than face-to-face debate. some groups (with the approval of the lecturer) resorted to meeting face-to-face to produce the final submission document. the quality and quantity of contributions was affected by two main factors: assessment and motivation. in both case studies the students' ongoing contribution to the discussion was assessed as well as the opening statement (in case study 1) and the final submission. because contributions were to be included in the assessment each posting was identified as being the work of a particular student. for case study 2, students appreciated being able to obtain an individual as well as a group mark. they found the transparency which the software provided (identifying them as authors of their contributions) was valuable in this respect as they could be confident that they would get credit for their own effort. whilst it was apparent that a few students contributed to the debate only because they were being assessed, others contributed over and above the required number of times. motivational factors, such as personal interest in the subject matter, pride in their contribution to the debate, the ability of the seminar leaders to generate discussion (by asking thought-provoking, challenging or probing questions) and feedback from their peers, were influential in this respect and also affected the quality of the submissions. in both case studies there were specific benefits in terms of learning. students learned from each other, were motivated because input from other students prompted further research, and spent considerable effort in finding evidence (both print and web-based) to support their arguments. almost all the students enjoyed having the freedom to decide when to contribute to the debates, although there was less agreement on whether it was easy or difficult to organize their time for regular interaction. those who had access to a computer and modem at home (67 per cent in case study 1,49 per cent in case study 2) enjoyed increased flexibility over choice of time and place to participate in the discussion, whilst students working from college also felt there was a benefit in not being constrained by a specific two-hour timetabled class. for example, one student stated: it suits me as a mum. i could put more in and research in my own time and then come in and put in my information when i'm free. furthermore, they felt that the length of time allotted to the debates (two/three weeks overall) led to the development of a more in-depth understanding of the topics. 36 alt-j vb/ume 7 number / conclusions there are several issues to be considered when defining and setting up tasks for computersupported collaborative learning, including the appropriateness of the medium for the task, the role of individuals, the volume of work, the time allocated for tasks or sub-tasks, and the assessment procedures. in the case studies outlined here it is our view that the task defined in case study 1 was entirely appropriate for the medium (a similar task utilizing different software is described by mcateer et al. (1997)). students were clear about the nature of the task, their role (see berge (1995) for a description of the role of facilitators in computer conferences) and the procedures to be followed. some criticism was made of the fact that cow did not allow for threading of messages, but this was a minor difficulty and on the whole the students found the interface easy to use. in terms of the task's comparability with traditional seminar presentations, greater emphasis was placed on the need for students to act as contributors to the debate (with this being assessed), although there was no verbal delivery of the paper. even when the task is appropriate for the medium, it is important that implementers get the expected workload right. in this example students were critical of the number of conferences to which they had to contribute (four, including the one they led). whilst this approach exposed them to a broad range of topics it also made it difficult for them to study any of the topics in sufficient depth. this will be reconsidered in future courses. whilst case study 1 provided a satisfactory outcome, case study 2 was on the whole less successful. although the nature of the task was clear, it was less appropriate for the software environment used. the use of newsgroups was satisfactory for facilitating the sharing of research findings, but the task required a consensual decision and the mechanism provided for sharing and agreeing the content of the final submission was rather cumbersome (a different piece of software might well provide the necessary tools). if this exercise is run again it will be necessary to revise the task, rework the assessment details (the length of the final submission was rather short for a group exercise) and reconsider the appropriateness of the software environment. to summarize: • the task set in a cscl environment must be appropriate for the medium. discussion and debate is appropriate: attempting to arrive at a consensual view may not be, unless appropriate support tools can be provided. • electronic conferencing systems provide an excellent mechanism for recording, monitoring and assessing individual contributions to group activities. • students have time to reflect on contributions to the debate, undertake further research and make further contributions, leading to improved quality in student learning. • the transparency of electronic discussion encourages participants to research topics thoroughly. if the task and assessment are appropriate, and software provides the right support tools, then the quality of the learning and the learning experience will be improved. 37 kathy buckner and kate morss the importance of task appropriateness in computer-supported collaborative learning references berge, z. l. (1995), 'facilitating computer conferencing: recommendations from the field', educational technology, 35 (1). buckner, k. and davenport, e. (1996), 'support issues for case-based learning in an undergraduate human factors class', education for information, 14 (4), 331-42. crook, c. (1994), computers and the collaborative experience of learning, london: routledge. draper, s. w., brown, m. i., edgerton, e., henderson, f. p., mcateer, e., smith, e. d., watt, h. d., (1994), observing and measuring the performance of educational technology, teaching and learning technology project tilt, university of glasgow. mcateer, e., tolmie, a., duffy, c. and corbett, j. (1997), 'computer-mediated communication as a learning resource', journal of computer assisted learning, 13 (4), 219-27. slavin, r. e. (1996), 'research on cooperative learning and achievement: what we know, what we need to know', contemporary educational psychology, 21(1), 43-69. webb, b. r., 1995, 'opinion: educational research and computer supported co-operative learning', innovations in education and training international, 32 (2), 139-46. 38 online mentoring and peer support: using learning technologies to facilitate entry into a community of practice lori lockyen john patterson, gregg rowland and doug hearne university of wollongong, wollongong, nsw, australia email: lori_lockyer@uow.edu.au ^ a vital aspect of any professional education is the opportunity for students to engage in meaningful practical experiences. in pre-service teacher education in australia, this vital teaching practice component has undergone challenges in recent years due to increasing student numbers (linked to the increasing demand for new teachers) and limited resources in university and school sectors. as such, initiatives to enhance the practical component of this professional degree have been sought. this paper details the methodology and outcomes associated with a pilot project that utilized asynchronous web-based communication tools to facilitate mentoring and peer support through the teaching practice experience. analysis of the online discussions and interviews with participants provides an indication of the nature of the interactions and the perceived value of the intervention, and informs the potential for larger-scale implementation. introduction a vital aspect of any professional education is the opportunity for students to engage in meaningful practical experiences. for pre-service teachers, the teaching practice experience provides the opportunity to observe the pedagogical strategies and classroom management practices of experienced teachers; to apply theoretical concepts and practical skills in reallife settings; to speak the language of the profession in a real setting; and to understand the concept of the school community and the professional role of the teacher within that community. teaching practice has been an example of how teacher educators and practising teachers work together to mentor newcomers into the community of practice. however, the 24 alt-j volume 10 number i effectiveness of the implementation of this partnership model has been the discussion of recent debate and calls for reform in australia (ministerial advisory council on the quality of teaching, 1998; ramsey, 2000). changing fiscal and social influences require new approaches to be investigated. the concerns of policy-makers are mirrored by the frustrations communicated by the active participants in the teaching practice experience (lecturers, pre-service and supervising teachers). university-based lecturers refer to limitations in discussing the application of theory into practice and providing counselling for pre-service teachers experiencing problems and anxieties when in the school setting. further needs analysis with pre-service and supervising teachers suggests requirements for greater and more consistent communication among participants, access to a variety of information (related to the programme specifically and to the teaching profession generally) and access to resources and tools to support the teaching and learning process. the affordances of information and communications technologies (icts) and their diffusion in primary, secondary and tertiary education setting allows for the exploration of innovative approaches to supporting the teaching practice experience. networked technology provides the opportunity for students to have continued and frequent contact with faculty staff and other students thus supporting a philosophy of collaborative learning (bruffee, 1993). the use of such technologies allows students to: • share their practical learning experiences to develop a collective body of knowledge; • discuss content material and their understanding of that content in the light of their practical experiences; • engage in reflective practice that is valued by mentors and peers; • develop a support network to facilitate ongoing information sharing; • combat isolation through the development of a networked community that includes access to peers, mentors and resources (naidu and olsen, 1996). this paper details the methodology and outcomes associated with a pilot project that utilized asynchronous web-based communication tools to facilitate mentoring and peer support through the teaching practice experience. content analysis provides an indication of the richness of the interactions possible in such technology-supported learning environments. in-depth interviews with lecturers and pre-service teachers provide a participant perspective on the value of the intervention and guidelines for large-scale implementation. the context the faculty of education in the university of wollongong provides undergraduate preservice teacher education and postgraduate continuing and professional development. a specialized bachelor of education degree prepares secondary school physical and health education teachers, and graduates of this programme have gone on to positions in community health and recreation centres. this initial teacher preparation in physical and health education involves the development of discipline-based knowledge, curriculum and pedagogy skills and professional ethos. in each year of the four-year programme, pre25 lori lock/er et al online mentoring and peer support service teachers engage in school-based teaching practice experiences. this teaching practice is designed to provide students with the opportunity to put theory into practice. professional and educational literature expounds the benefits of the use of information and communications technologies (icts) in support of the roles and responsibilities of the physical and health educator (milio, 1996; committee on enhancing the internet for health applications, 2000). in 1998, with a strategic commitment and structuring of lecturing teams to include an appropriate mix of technology and physical and health education expertise, the programme began to integrate ict use in discipline and pedagogybased subjects (lockyer and kerr, 2000; lockyer, patterson and harper, 2001). the success of such implementations suggested that extensions of that integration might address the aforementioned challenges to the teaching practice programme. of particular interest was how the online environment might facilitate the active process of reflection on the experience of professional practice through the interaction among the pre-service teachers and their university-based lecturers (knights, 1985; herrington, herrington and oliver, 1999). pilot project a small-scale trial of online peer support and mentoring was implemented for students engaged in their final five-week teaching practice experience in july/august 2000. ten fourth-year students volunteered to engage in the trial, with two lecturers acting as expert mentors. a website provided access to resources (such as links to sites related to syllabus topics and a lesson plan template) and a bulletin board, which facilitated the online interaction among students and lecturers. the bulletin board was structured to support specific topic exploration in order to provide: • a general forum where students could interact on a more social level not necessarily focusing on teaching practice issues; • an activities forum where pairs of students were rostered weekly to submit an activity undertaken during the teaching practice; • and a questions and problems forum where students could post any problems or queries they had in relation to their teaching practice or where lecturers could pose questions for the whole group. one week prior to their school-based teaching practice experience, a two-hour training session was held with students and lecturers, which included an introduction to using the bulletin board system. a key aspect of this session was to clarify the students' tasks and responsibilities related to their participation in the online discussion. similar attempts to support professional practice experience through online interaction have suggested use of an explicit structure to develop interaction (brehm, 1999). further, consideration was given to developing a structure that would facilitate the opportunity for all students to reflect upon and contribute their classroom experiences. given that many participants had limited familiarity with online discussion, a roster structure was established. each student was asked to contribute a description of one activity they had used or planned to use in the classroom during their teaching practice in an activities forum of the bulletin board during 26 alt-j volume 10 number / an assigned week. students were responsible for providing feedback to their peers regarding these posted contributions. all students were also given the opportunity to request advice from their peers and lecturers, as well as to suggest possible solutions to issues faced by other students during the teaching practice, on an ad hoc basis through the questions/problems forum of the bulletin board. lecturers were not given a defined task but were expected to monitor regularly all discussion forums and respond to students' questions and problems if it seemed that their peers could not help them resolve issues. lecturers were also expected, if necessary, to stimulate online discussion by posing practice-based questions. all postings to the three bulletin board forums were saved and analysed in order to understand the use patterns and nature of discussion that took place. further, students' and lecturers' perspectives on the experience were elicited through post-implementation interviews which also provided explanatory support to the analysis of the interaction data. interview questions covered issues related to participants' perceptions of the value of the online interaction in supporting the development of their practice-based skills and understanding, their discipline (that is, physical and health education) knowledge base, professional self-concept and teaching philosophy, and social/emotional encouragement as well as the technical considerations of using the online system. in addition to analysis of the pattern of postings, analysis of the nature of the online discussion was undertaken with consideration of these issues. analysis of the postings revealed that, while students had been assigned a week in which to post an example of teaching activities that they had used or intended to use in class, only four of the ten participants fulfilled this task. of those, one student's initial contribution was limited to a description of an activity used with children in class. probing questions posted by other students and a lecturer stimulated reflective responses from the initiator regarding the process and outcomes of engaging in the activity with the children. the other three students who initiated contributions during their assigned week focused on basic descriptions of an activity they planned to implement in class. questions to elicit more information about the planned activity and/or sharing of personal experiences with similar situations were posted by other students. this provided the initiators with an opportunity to clarify their lesson plans and spark ideas for possible activities among the other participants. for example, in response to a description of a ball game one student intended to implement, another participant posted: great game!... i will definitely be trying it with my year 7's next time we have prac. only thing is i'm not 100% clear on is what the zone in the middle is for?!?!? clarification please!!!!! the general and questions/problems forums were by far the most utilized by the participants (26 postings in the activities forum, 67 postings in the questions/problems forum, and 206 postings in the general forum). participants contributed practice-related issues and questions to both the general/problems forums. however, the majority of the postings in the general forum covered more personal or social topics (such as jokes, birthday greetings, organization of social outings, etc.). 27 lori lockyer et ol online mentoring and peer support the lecturers initiated discussion related to classroom practice (for example, strategies to engender respect from children) in the questions/problems forum within the first few days of the teaching practice in an attempt to welcome the students into this discussion space. however, students quickly took control of the discussion by sharing their classroom experiences, asking their peers and the lecturers for specific advice and providing their peers with practical suggestions and emotional encouragement. practice-based topics discussed in the questions/problems and general forums included aspects of classroom management, planning for activities, reflection on practice and the process of teaching practice assessment. questions posed by the lecturers and requests for advice from students engendered contributions of both conceptual understanding or philosophy of teaching practice as well as detailed description and reflection on specific strategies used in class by the students. one students' posting expressed emotional stress over the teaching practice situation: subject: soul searching how do you get yourself out of a rut? i don't want to be here. i know i should plan but for the life of me, it just isn't happening... there are soooo many things to think about and consider i simply can't be blessed starting... what do i do ?!? my life sux . . . signed: screwed up mind. four students responded to this posting with contributions of encouragement. for example, one student replied: subject: re: soul searching . . . we are all new at this we are trying to put the theory into practice, its tough man i understand the way your feelin bro . . . you think that your not cut out for this stuff . . . have a look around the staff room do you see any of them up till all hours thinking of what they have to teach tomorrow no. they have the experience, knowledge and resources and know how to use them efficiently and effectively. that's what we are here to learn and develop . . . just think about the good lessons or even just the good parts in the lessons, think of the information that the students have learnt from you!!! watch how their skills develop over the 5 weeks because of you!!! one of the lecturers used this set of postings to focus the discussion on the attributes of a professional teacher (an issue previously explored in lectures). the pattern and nature of contributions suggests that a more free-flowing interaction rather than structured assignment of roles was appropriate to support the students' needs while they were engaged in their practice teaching experience. this observation was further confirmed through post-implementation interviews with participants who noted that an imposed structure of assigned contribution in the activities forum confined the interaction process and that they felt more comfortable contributing to the more open general or questions/problems forums. extending the concept lessons learnt from this initial small-scale trial provided a basis on which to structure subsequent implementations of online peer support and mentoring through the teaching practice experience. the 2001 cohort of fourth-year students were given the opportunity to 28 alt-j volume 10 number i participate in the online intervention during their five-week (july/august) teaching practice experience. for this implementation one forum was utilized with two threads one for general or social issues and one for teaching issues. unlike the previous implementation, a structure or roster for contributions was not imposed. while far fewer contributions were posted by this group (n=106) than that from the previous cohort (n=299), the nature of the discussion was similar. the limited participation may have been a factor of technical difficulties (individual student access to the internet and university server downtime) combined with a smaller number of participants (seven versus ten). the concept was also extended to include second-year students who were involved in one week of secondary school teaching practice in each of two semesters of the academic year (one week in may and one week in october). the seven fourth-year students agreed to act as online mentors for fourteen second-year students. two second-year students were grouped with one fourth-year student acting as mentor. one week prior to the commencement of the first teaching practice experience for the second-year students, the participants attended a training and development exercise. this session was used for groups to meet and negotiate times at which they expected to access the bulletin board over the week. second-year students were advised that they could submit questions, problems and requests for advice. the fourth-year mentor was responsible for providing suggestions and feedback on the issues or questions raised by the second-year students undertaking their practice. students were advised that the participating lecturers would monitor the discussions on a regular basis to ensure that questions or problems were not going unanswered (particularly in the event that a fourthyear mentor experienced technical difficulty accessing the bulletin board). analysis of the online interactions found that the nature of the discussion focused on general behaviour management issues with some practical teaching strategies being suggested by the mentors. when second-year students returned to campus after their teaching practice, all participants were invited to attend a session to discuss the online experience. the teaching practice students agreed that they appreciated knowing that there was someone online with whom they could share their successes and frustrations. they said that they felt comfortable asking their online mentors for advice and suggestions. the online mentors felt that the experience allowed them to reflect on their previous practice teaching experiences. a comment by one mentor was wholeheartedly supported by the others: 'i was giving advice and i thought "where did that come from?" i can't believe how much i've learnt in the last three years!' the second teaching practice for the second-year students occurred after the fourth-year online mentors had engaged in their own final practice teaching experience (five weeks in july/august). preliminary analysis of the online interactions from the second teaching practice suggests that, in these discussions, the fourth-year mentors were far more explicit in their advice regarding teaching strategies. they made specific reference to examples of their experiences during their recent teaching practice and provided their analysis of the teaching strategies which they employed. whilst issues of technical stability of the bulletin board system and access to the site from schools needs further consideration, the opportunity to be involved in discussions both 29 lori lockyer et al online mentoring and peer support personal and professional over the duration of the teaching practice was rated highly and thought well worthwhile by all participants and therefore worthy of continuation. conclusions the success of these small-scale trials has highlighted the potential for learning technologies to enhance the practical component of a pre-service teacher education programme and provides a foundation for a future, larger-scale implementation. the following issues must be considered and detailed. structuring discussion and supporting reflection the initial attempt to elicit participation and facilitate reflective processes among the students via a roster of contributions regarding teaching activities was not successful. the pattern of discussion and feedback from students suggests a more free-flowing interaction to be beneficial. however, students' initial tendency to be descriptive (recounting what happened) needs to be extended by the other participants. questioning and contribution of their own experiences by lecturers and other students facilitates the engagement of students in a more reflective process, discussing why it happened and what they would do differently next time. preparing both the students and the lecturers to identify the difference between a descriptive and reflective posting and developing strategies to support reflection may need to become a focus of the pre-implementation training session. the lecturers' role has further importance in simulating discussion through practice-based questions, providing contributions that link the students' classroom experiences to campus-based lectures, and acting as a source of ideas or guidance. communication tools an asynchronous communication tool was selected based on benefits of flexibility of time for contribution and recording/reflection purposes. however, given that some groups of students choose to schedule their online meetings for a particular time, it may be appropriate to provide access to a synchronous tool such as a chat space which would better facilitate real-time interaction. future research associated with a large-scale implementation will provide empirical evidence for the extent to which this intervention prepares pre-service teachers for their future professional role. references brehm, b. (1999), 'factors for successful telementoring of preservice teachers', in collis, b. and oliver, r. (eds), proceedings of ed-media99 world conference on educational multimedia, hypermedia and telecommunications, charlottesville, va: association for the advancement of computing in education, 1300-1. bruffee, k. a. (1993), collaborative learning: higher education, interdependence, and the authority of knowledge, baltimore: the johns hopkins university press. committee on enhancing the internet for health applications: technical requirements and implementation strategies, computer science and telecommunications board, national research council (2000), networking health: prescriptions for the internet, washington: national academy of sciences. 30 alt-j volume 10 number i goodlad, j. i. (1994), educational renewal: better teachers, better schools, san francisco: jossey bass. herrington, t., herrington, j. and oliver. r. (1999), 'providing reflective online support for preservice teachers on professional practice in schools', in b. collis and r. oliver (eds), proceedings of ed-media99 world conference on educational multimedia, hypermedia and telecommunications, charlottesville, va: association for the advancement of computing in education, 161-71. knights, s. (1985), 'reflection and learning: the importance of a listener', in d. boud, r. keogh and d. walker (eds), reflection: turning experience into learning, london: kogan page, 85-90. lockyer, l. and kerr, y. (2000). 'learner as designer-producer: physical and health education students experience web-based learning resource development', in j. bourdeau and r. heller (eds), proceedings of ed-media2000 world conference on educational multimedia, hypermedia and telecommunications, charlottesville, va: association for the advancement of computing in education, 591-5. lockyer, l., patterson, j. and harper, b. (2001), 'ict in higher education: evaluating outcomes for health education', journal of computer assisted learning, 17 (3), 275-83. milio, n. (1996), engines of empowerment: using information technology to create healthy communities and challenge public policy, chicago: health administration press. ministerial advisory council on the quality of teaching (1998), towards greater professionalism: teacher educators, teaching and the curriculum, sydney: new south wales department of education and training. naidu, s. and olson, p. (1996), 'making the most of practical experience in teacher education with computer-supported collaborative learning', international journal of educational telecommunications, 2, 265-78. ramsey, g. (2000), quality matters: revitalising teaching: critical times critical choices; report on the review of teacher education, new south wales, sydney: new south wales department of education and training. 31 on-line student feedback: a pilot study liz barnettjane galbraith, paul gee, fran jennings and ron riley london school of economics and political science email: l.barnett@lse.ac.uk this paper reports on the outcomes of two experimental trials of the use of on-line questionnaires to assess student satisfaction with courses at the london school of economics and political science. in the first year, eighteen course modules were selected from three departments, surveying a total of 1,100 student places. students on ten of the courses were invited to complete the 'experimental' on-line survey and the remainder were invited to complete the paper-based questionnaires which have been in use for several years. in the second year, the scale of the experiment was increased, to include forty-six courses across seven departments. response rates were compared and possible barriers to completion of the on-line questionnaire were considered whilst electronic monitoring indicated that 95 per cent (first trial) and 80 per cent (second trial) of those contacted for the on-line survey opened the introductory email, only 23 per cent (first trial) and 27 per cent (second trial) completed the on-line survey, compared with a 60 per cent response rate on the paper-based survey. the on-line response is also slightly lower than that achieved by postal surveys of lse students (30-50 per cent response rates). whilst some technical difficulties could have acted as a barrier, motivation appeared to be the main barrier. initial results from the second trial, which included two reminder emails and some small incentives, show that it is possible to increase the response rate, but this may still be unacceptably low for staff whose promotion prospects may be affected by results. a third trial has been proposed, looking at ways in which the process as a whole could be amended, to overcome the problem of 'survey fatigue' that the current system faces. introduction student feedback on courses has been a standard feature of university life for several years, with a substantial research backing to it (see, for example, reviews by marsh, 1987; marsh and dunkin, 1992). most feedback systems endeavour to serve more than one purpose, with student feedback forming part of the evidence used by: • individual teachers to improve teaching; 17 uz barnett et 0/ on-line student feedback a pilot study • heads of departments to monitor and guide individual teachers, especially part-time teachers; • course leaders for course development; • senior managers to make decisions on probation and promotion related to teaching performance; consider changes to the wider learning environment; help with the marketing of programmes; • external agencies to assess quality of provision. there are a number of critiques questioning the validity of student response as a measure of educational quality. for example, greenwood and ramagli (1980) review a number of studies showing low correlations between student ratings and final course outcomes, thus questioning whether 'satisfied' students are necessarily 'effectively educated' students. kerridge and mathews (1998) revisit this debate, questioning the notion of 'student as client' versus 'student as customer', and again stressing that student satisfaction ratings do not equate with student outcomes. however, within the he sector, the increasing perception of 'students as customers' is significant (especially as personal financing of he is increasing). as such, student satisfaction is considered important, and measurement of it will no doubt continue to grow. many institutions use a combination of approaches, including standardized questionnaires (often combining both closed and open-ended questions), student discussion sessions, focus groups and meetings, and a range of other student-focused data collection methods (see partington, 1993 for examples). the emergence in the uk of the quality assurance agency (qaa) subject review process has encouraged many institutions to look at how systematic their feedback process is. qaa places importance on transparent and robust quality assurance systems, which provide evidence on course quality and are integrated into explicit quality feedback loops. in some institutions, central units take responsibility for the management of the student feedback process; see, for example, the work of the centre for research into quality at the university of central england (uce (http://www.uce.aauk/crq/)). this can be a major task with substantial time and resource implications. in the case of the london school of economics and political science, the teaching quality assurance and review office (tqaro) has been operating a centralized student feedback system since 1991 (husbands, 1997). up to now the system has been paper-based, with data subsequently scanned into computer and processed electronically. faced with the substantial costs needed to update/revise the existing processing operation, the teaching quality assurance committee (tqac), responsible for the survey, agreed to experiment with on-line delivery. the purpose of the experiment was to establish whether the technology can be used to maintain a robust quality assurance procedure which is less resourceand time-intensive and which can deliver results more quickly. research evidence on the use of on-line questionnaires is limited. a hefce-funded development project considering student feedback practices across a number of business schools in the uk makes reference to it, but does not cite any research evidence (http://www.lboro.ac.uk/departments/bs/fdtl.html). a search of the past ten years of the open learning journal included a number of reports of evaluations of distance learning programmes but interestingly these did not include any examples of studies using on-line 18 alt-j volume 9 number 3 surveys. taylor, woodman, sumner and blake (2000) used both on-line and paper postal questionnaires in evaluating an open university course. they found that the respondents to the two types of survey had similar profiles. they do not give response rates but report that, over successive on-line surveys, 'the response rate begins high then rapidly tails off'. there is a general impression from those working in the field that getting students to complete online questionnaires is likely to result in a low response rate, with motivation, access and it skill considered the three main barriers. this paper reports on a trial, at lse, comparing student response to on-line and paper-based surveys, to assess variations in response rate and possible explanations for them. the lse student feedback system at lse, student feedback is collected twice each year, with a separate survey (focused on part-time teachers) and the standard survey (for lecture courses completed in the first term) undertaken in november/december, and the main data collection (for all other lecture courses) in february/march. students are asked to comment on each course they attend (generally four courses per student per year). the questionnaire is in two parts. part a consists of a number of closed questions/rating scales. questions address both broad issues of course management and resources, and detailed responses from the students on up to three specified lecturers and three seminar/class teachers per course. part b is a series of open-ended questions for free response which ask students to detail aspects of the course they most/least appreciate. a central unit, using scanners, data analysis and interpretation software, processes the completed questionnaires. part a data is processed into a set of numerical results. part b is checked, any abusive comments removed, and the remainder presented verbatim, under the question headings. individualized reports are prepared for each member of staff and are delivered to them by hand. they are expected to follow up any issues arising and adjust their programmes as appropriate. details about individuals are also available to convenors (heads of departments) for management purposes, and play a part in decision-making about staff at the end of their probationary period, and at promotion. teachers are asked to arrange for the questionnaires to be circulated in class, and to organize for a student representative to collect the completed forms, and return them to the central office for processing. student anonymity is guaranteed and staff receive reports only on responses to items completed by five or more students. the process from sending questionnaires out to classes, to returning the results to individual members of staff takes around eight weeks. two members of staff work fulltime on this task. around 14,000 questionnaires are processed, with each student being expected to fill out up to four questionnaires, one for each course attended. key features of the paper-based system include: • a standardized approach, across the institution, achieving a consistently high response rate (of around 60 per cent); • contact with all students who turn up to classes when the questionnaire is being administered (there are no attempts to verify class lists for this purpose some students may be absent); • time set aside in classes for completion of the questionnaire; 19 liz barnett et al on-line student feedback a pilot study • a high degree of anonymity for the students; • a process which is consistent over time, enabling individual members of staff to build up a picture of their teaching over several years; • a confidential system for staff, who are aware of who has access to their individual data, and how that data may be used. the disadvantages of the system are: • the high cost of operating the system, including the vast quantities of paper involved; • wasting class time on what is seen to be an essentially administrative task; • processing time, and consequent delay in feedback to staff; • the lack of 'ownership' by academic staff of the process, lessening its potential as a development tool. in addition to these regular surveys of student opinion, the school undertakes more general surveys of the student experience. the purpose and format of these surveys is quite different from that of the annual course survey, but they provide a useful alternative for comparison on response rates, as, like the on-line questionnaire, they are left open to the student to decide whether or not to respond a very different scenario from the 'forced' situation where students are expected to complete surveys in a classroom situation. in looking at the development of an on-line alternative, the concern was to maintain the positive features of the existing system, whilst addressing some of the disadvantages. issues to be addressed in developing an on-line system from the start, there was concern about the response rate particularly given that results on individual teachers are subsequently used as evidence to judge teaching quality at probation/promotion. any reduction in overall response rate may be associated with a skewing of response (for example, a move away from a 'captive audience' may lead to response only from those who are dissatisfied, or possibly very enthusiastic, about a given course). in this context, worries were also expressed about differential response by course, based on the assumption that students in some subject areas may be less technically literate than others, and hence less likely to respond to an electronic questionnaire. the other main area of concern was with data protection and anonymity. would students feel confident that their anonymity would be maintained, given that they are well aware that email and internet access can be monitored? on the other hand, might students attempt to 'sabotage' the process, for example, by multiple 'malicious' responses from a single person? on this point, staff would need assurances, given that the evidence collected is used in the context of their career progression. trial one given some of the reservations expressed above, as well as lack of definitive evidence from elsewhere on the robustness of on-line course evaluation, a pilot study was set up, taking a limited range of courses in a small selection of departments. 20 alt-j volume 9 number 3 three departments were selected (government, philosophy and statistics) to give some indication as to whether different student groups have different it skills/preparedness to use an on-line format. six modules were selected from each participating department and structured randomization was used to allocate some to the control and some to the trial group. over 1,000 potential respondents were included in the pilot, with.530 being sent the electronic version (trial modules), and the rest being sent the traditional paper-based system (control modules). one difference with the on-line system was that circulation lists were based on central records of course members, whilst the paper version went only to those who happened to be in class on the appropriate day. these figures do not necessarily tally! given that each student attends four or more courses each year, it was quite possible that some students may have found themselves responding to one course using the paperbased questionnaire, and another using the on-line format. the actual questions in the paper-based and on-line formats were the same, but the presentation was not. in the paperbased version, students were presented with several questions per page. on-line, they were faced with a single question per screen. students attending the control courses received the paper-based questionnaires following the normal procedures. class teachers (provided with an ohp describing the new system) alerted students attending the trial courses to the on-line questionnaire. students were then individually emailed with a message describing the on-line survey pilot, and giving them direct access to the on-line site with a single mouse-click. they were sent a reminder three weeks later. emails were monitored to see how many students opened the initial message, and how quickly it was accessed. completed web forms were also monitored to check that each student only responded once per course. a follow-up survey was later emailed to all students on the trial courses to gauge their reactions. trial two trial two used the experiences of trial one to see if it would be possible to scale up the online survey without facing major technical difficulties. for trial two, 2,011 students from twenty-four courses in seven departments received the on-line survey, along with 2,347 students from twenty-two courses in the same selection of departments being included as a 'control', receiving the paper-based survey. the additional departments were law, social psychology, information systems and economics. trial two continued to look at ways of increasing response rate through improving the quality of presentation and ease of completion; increasing the number of 'reminder emails' sent to two (only sent to those who had not replied), and testing out the use of small incentives. on the last point, we ran an automated 'lottery', with students randomly allocated to different groups being told they might win photocopying cards worth £5 or £10 if they returned the survey. key questions addressed in evaluating the trials were: 1. do the response rates on-line compare favourably with the paper-based system? is there any obvious difference in uptake by department/student year group? is there any evidence to suggest that incentives and reminders improve the response rate? 2. how confident are students about the on-line version? are concerns about confidentiality, anonymity and access to the data set addressed to a degree that ensures confidence? 21 uz barrett et al on-line student feedback: a pilot study 3. has the system for monitoring responses (to ensure that only students registered for the module respond) worked, without allowing the identity of respondents to be revealed? 4. is it possible and useful to attempt to replicate the paper-based system on-line, or does the new delivery method require fundamental changes to the structure/form of the questionnaire? 5. how costly of staff time and effort are the two systems? results response rates in trial one, email monitoring indicated that 95 per cent of students contacted read the initial message within two days of it being sent out. however, only 23 per cent went on to complete the questionnaire. in contrast about 60 per cent of paper questionnaires were returned. there were no significant variations by discipline or by year of study. in trial two, 80 per cent of the on-line group opened the initial email message (the remainder deleted without reading (7 per cent), or did not open the message). a total of 48 per cent went on to open the survey applet, and 30 per cent completed and submitted the survey. the effects of the reminder were marked by peaking response rates in the two days following the emails, but there was a fall off over time, and the second reminder 'peak' was substantially lower than the first. given the study design, it was not possible to identify the relative effects of improvements in the technical delivery of the questionnaire, improved publicity, the use of an extra reminder, and the introduction of incentives in raising the response rate from 23 per cent in the first trial to 30 per cent in the second. student and staff confidence in trial one, 45 students (8.5 per cent) who were sent the on-line survey replied to the follow-up survey designed to elicit their reactions. most of these (35) had actually completed the on-line survey. respondents were generally appreciative of the experiment, being pleased that it saved class time, saved paper and was more private. there were concerns about time taken to complete the on-line version. as each question is a separate screen, it 'feels' much longer than the paper version. there were a few comments on technical difficulties, but this did not seem to be a major barrier to use. fifteen claimed that the briefing ohp had not been shown in class. twenty-three students thought arrangements for confidentiality were satisfactory but four still had some concerns. for staff, the main concern lies with the poor response rate. monitoring of response should be able to weed out any people who try to abuse the system by replying several times on the same course. there was no evidence of any attempts to do this on this occasion. another possible issue of concern for staff is that the low response rate using the on-line version might lead to quite a different set of results as compared with the paper-based version. for example, one might hypothesize that only the more motivated students would respond to the on-line survey. these might include students who are very concerned and/or students who are most enthusiastic about a given course, and this might lead to more extreme ratings and greater deviation on scoring. at lse, questionnaire scores play a part in probation/promotion decisions. hence any systematic difference in scoring would have serious implications for staff confidence. to explore this issue, six questions were selected from the total questionnaire, and variations in response from the two groups compared. 22 alt-j volume 9 number 3 questions on the library, overall course satisfaction, and the quality of lecturing showed no systematic differences in response from the two groups. however, on class teaching there does appear to be a slight tendency for on-line respondents to give a less favourable response to tutors than those completing the paper questionnaire. results here were consistent, but numbers of respondents were too small to draw any firm conclusions. anecdotal evidence from both staff and students suggests that in some cases students feel uncomfortable about completing questionnaires in the presence of class teachers; this might provide an explanation for this effect, and is in line with other studies (see, for example, wachtel, 1998: 195, reference to pulich who 'contends that even if the instructor leaves the room while the students are filling out the forms, some students may still be inhibited by the fact that the instructor himself/herself distributed them'). replicating the paper-based system it was technically possible to replicate the paper-based questionnaire for the on-line trial. however, several respondents did find the format off-putting, as each question was on a separate screen. the format was improved in trial two, but still maintained the match with the paper-based version. with the agreement of the staff, it may be possible to move to a different format for the on-line survey in the future, but this will require careful political negotiation, given the sensitive nature of student evaluation of staff. the move from paper to on-line presents a new set of options on customization, making it easier to design questionnaires more relevant to the needs of individual departments without at the same time seriously increasing costs of data processing and analysis. costs there are clearly costs in setting up any new system. however, the set-up costs for the online trials have been minor, compared with estimates for upgrading equipment/software needed for the paper-based system. there are some recurrent costs associated with the online survey, which do not apply to the paper-based system. these include sorting out class email listings, and monitoring of response. however, current estimates suggest that this is a much smaller task than the major data input and data-cleaning requirements of the paperbased system. with the latter, around 20 per cent of scanned data showed problems which required operator intervention. speed the processing of the on-line version was very quick, thus enabling faster feedback to the teachers. discussion in terms of cost, time, accuracy of data translation and speed, the on-line questionnaire is clearly better. trial two, involving many more students, demonstrated that technical issues in scaling up the process should not prove too problematic, although as yet, students cannot access the survey applet from outside the school network with ease (which may be another barrier to response for some students). on concerns about confidentiality of the data, and anonymity of respondents, it is important to be able to assure the integrity of the technical staff operating the system but this does not seem to be an issue of undue concern to students. 23 liz barnett et al on-line student feedback a pilot study the key factor considered in the lse on-line trial was response rate. on this point, the trial was considered a failure. anecdotal evidence from elsewhere suggests that motivation, access and it skill are the three main barriers to effective use of on-line feedback. our results tend to suggest that student it literacy and access to computers do not appear to be major barriers, although issues of screen design and time taken to complete the questionnaire will continue to be addressed in future trials. what of motivation? this perceived 'failure' requires further analysis. the high response rate of the paper-based version is in part due to it being delivered in a 'captive' situation in class. what is more, it totally fails to gauge the opinions of students who, for whatever reason, are not in class on the relevant day. as such, it may be biased against those who have ceased to attend because of problems with the quality and/or content of the teaching. a more appropriate comparison would be to look at questionnaires sent out to students, rather than delivered through the 'captive' classroom setting. at lse, a series of one-off postal surveys of the student experience achieved response rates of 30-48 per cent (stockdale, 1991,1993,1993a, 1994,1996). the most recent postal survey of lse students was undertaken by mori in spring 2000. this achieved a response rate of 33 per cent. at uce, a centrally managed annual student satisfaction survey achieved a response rate of just over 50 per cent in 1992, but this declined to around 38 per cent in 1997. various reports of postal surveys of distance learning students (for example, macdonald-ross and scott, 1997; webb, 1992) indicate response rates of 24-33 per cent. trial two, with a response rate of 30 per cent, is still not up to the level achieved by most of these postal surveys. however, one other feature of the on-line survey (and the chosen 'control') is the frequency and volume of surveys of this nature that students are expected to complete. work at uce has raised concern about student 'questionnaire fatigue'. the uce systems works on a sampling basis, and students will only receive a maximum of one questionnaire per year (although one reason for the fatigue is the growing number of departments doing their own surveys). at lse, each student faces around four surveys every year. at some stage, the issue of quality over quantity of response will need to be addressed. it may well be that if the on-line survey is to be perceived as successful in terms of improved response rate then more consideration will need to be given to the content and detail in the survey, and to the purposes for which the data is used. for now, we await a decision on whether to continue with the trials. if they do continue, it is likely that the next stage will again address the issue of 'scaling up' to ever greater numbers of students, to check the impact that this may have on our computer system. following this, we hope that it will be possible to revise the format, cutting down on the number of separate surveys students will be expected to complete (thus addressing the fatigue problem) and creating new survey designs more fitted to the on-line medium. ultimately, it will be important to get an appropriate balance between the demands of quality control and judgement centrally and the needs of staff for information and guidance on how to improve their teaching. references galbraith, i , gee, p. and riley r. (1999), 'interim report on the teaching quality assurance committee (tqac) pilot study in lent term 1999 of an on-line, web-based, student 24 alt-j volume 9 number 3 course evaluation questionnaire', internal report to the teaching quality assurance committee in lse. greenwood, g. e. and ramagli, h. j. (1980), 'alternatives to student ratings of college teaching', journal of higher education, 51 (6), 673-84. husbands, c. (1997), 'variations in students' evaluations of teachers' lecturing in different courses on which they lecture: a study at the london school of economics and political science', higher education, 33, 51-70. kerridge, j. r. and mathews, b. p. (1998), 'student rating of courses in he: further challenges and opportunities', assessment and evaluation in he, 23 (1), 71-82. macdonal-ross, m. and scott, b. (1997), 'a postal survey of ou students' reading skills', open learning, 12 (2), 29-40. marsh, h. w. (1987), 'students' evaluation of university teaching', international journal of educational research, 11 (3), 255-78. marsh, h. w. and dunkin, m. j. (1992), 'students' evaluation of university teaching: a multidimensional perspective', higher education: handbook of theory and research, 8, new york: agathon press, 143-233. partington, p. (ed.) (1993), student feedbackcontext issues and practice, usdu. stockdale, j. (ed.) (1991-6), a series of lse internal studies of the student experience. taylor, x, woodman, m., sumner, t. and blake, c. t. (2000), 'peering through a glass darkly: integrative evaluation of an on-line course', education, technology and society, 3 (4), available online at http:llifets.ieee.org/periodical/vol_4_2000/taylor.html. wachtel, h. k. (1998), 'student evaluation of college teaching effectiveness: a brief review', assessment and evaluation in higher education, 23 (2), 191-211. webb, r. (1992), 'a new survey of nec learners', open learning, 7 (1), 58-62. 25 networking: a study in planning and developing cross-cultural collaboration sanjeev singh* and jon dron** *institute of informatics and communications, university of delhi south campus **school of information management, university of brighton email: sanjeev@iic.ac.in, jon.dron@bton.ac.uk this paper reports on a collaboration between the authors at the university of brighton (uk) and the university of delhi, south campus. the collaboration came about as a result of the eu-india cross-cultural innovation network collaboration programme, a project involving several universities and organizations across europe and india. the authors of this paper both lecture in the area of computer networking. following meetings in delhi, they agreed to work together to produce a web-based networking resource to be generated by the students of both institutions. the first phase of development involved the mounting of web-based tutorials and documents produced by the students. the second phase will centre on the development of a knowledge base generated by the interaction of the students within an asynchronous forum. running alongside these phases will be a collaborative bookmarking system, a database in which the students will post urls of web-based resources that they find useful in their studies. this system incorporates a form of collaborative filtering, an evolutionary mechanism which seeks to embody the qualities that students value in resources to provide a dynamic set of ratings to assist in the selection of those of most use. the planning of such a system raises some unusual issues, not least in the process of collaboration itself, with concerns as diverse as technical compatibility, institutional and cultural differences, timezones and the reliability of email. limited bandwidth between our institutions causes special problems with the interactive elements of the resource. we present the methods we are investigating to reduce the impact of this. the fact that the students share an intellectual discipline but are otherwise separated by a cultural and geographical divide is expected to lead to fruitful diversity in thinking and approaches to problem-solving. introduction this paper discusses the theoretical and practical issues of implementing a web-based learning collaborative project across cultural borders. the overall purpose is to convey to 29 sanjeev singh and jon dron networking: a study in planning and developing cross-cultural collaboration the student community the benefits of incorporating a web-based cross-cultural learning environment into a conventional course. the criteria for successful applications are presented and issues relating to training and design, are discussed. the value of extending the classroom beyond national boundaries is only one of the many reasons for increased intercultural communication relating to the growing internationalization of education (brislin and yoshida, 1994). on graduation students will often be required to interact with others from different cultural backgrounds, whether they are well prepared to do so or not. in the hope of building strong international working relationships between people and organizations of different cultures, it is important that students be prepared for the emerging future where cross-cultural communication skills will be of great importance. one way of developing these abilities is to present students with authentic opportunities to work with people who possess specialized skills in a shared subject domain. the sociocultural context in the present model includes different values, learning environments and cultural/social perspectives of learners in two or more countries; these concerns are at the root of many problems that arise in international or cross-cultural programmes (conner, 1990). an analysis of cultural issues shows that the predominant trend in the area is to ignore the unique cultural contexts of diverse learners. occasionally, instructional adaptations and modifications are made to enhance cultural appropriateness, but cultural subtleties and deep-rooted value systems are often neglected in this process. it is nonetheless essential to take into account both the deep structure (values, learning styles) and the surface level (spoken language, forms of interaction) dimensions of a culture. each country manages its own educational system and establishes its own priorities for training and design. a 'not-invented-here' mentality has made many established institutions unwilling to use or recognize courses created by other institutions or in other countries. funding policies and mission clarity and articulation also become more complex when carried across national borders (murphy, 1991). thus the development of a crosscultural course must be sensitive to these issues in order to succeed. rationale web-based training does not merely provide a different medium for traditional classroom interactions, but creates an environment for students to take part in much richer and relevant educational activities. by incorporating a web-mediated cross-cultural learning activity into the course curriculum, students may be presented with new and greater challenges that extend beyond their traditional, and often passive, learning of theories and content. with respect to building cultural awareness and tolerance, web-mediated crosscultural learning activities have the potential of bringing about behavioural and attitudinal changes in people that otherwise would not occur. for example, students may develop a deeper sensitivity for culturally based differences in work habits by interacting with students of diverse cultures rather than reading about how one should be culturally aware of these differences in textbooks. likewise, pre-service teachers may acquire a better understanding of culturally based differences in learning styles, knowledge representation and cognition if they are given the opportunity to interact with students from diverse cultures. 30 alt-j volume 10 number 2 perhaps more important, however, is that providing students with the opportunity to participate in an authentic cross-cultural exchange releases them from the proverbial academic vacuum in which they typically build knowledge and understanding from readings and in-class discussions (dron, mitchell, siviter and boyne, 2000). computer communication technology has already changed the image of the classroom. these networks stretch across international borders to form cross-cultural learning environments where teachers and students of diverse cultures can learn with and from each other. with its capability of bringing innovative views to the classroom instantaneously, this collaborative project presents students with new opportunities to develop cultural awareness and build shared knowledge. some of the important criteria for implementing our web-based cross-cultural learning project are that the collaborative project must: • be relevant to the course and integrated into the curriculum goals; • include some aspect of work or learning that could not be accomplished without crosscultural computer mediation. student interest in, and commitment to, the virtual learning environment may diminish if they perceive that the learning activities could be done just as easily, if not better, outside the international forum. it is crucial that the cross-cultural model contains a unique element or dimension that otherwise would not be present; • have a clear and definite time-frame. students aware of the time-frame within which they are working may be more inclined to adopt proactive writing habits so that the virtual learning environment is not adversely affected by prolonged inactivity; • be as simple and uncluttered as possible. overly complex activities may lead to information overload, resulting in frustration, loss of interest and inactivity; • be preceded with greetings and introductions. students should be given the opportunity to 'meet' online before engaging in the structured learning activity; • encourage information-sharing and knowledge-building. one of the greatest potentials of a web-mediated cross-cultural exchange is to overcome the barriers that restrict the flow of ideas and the sharing of knowledge; • avoid compromising learning quality. although it may provide new and innovative learning opportunities, the academic standards and levels of achievement must be maintained. other advantages of this collaborative project are that, like many electronic methods of learning, it introduces new opportunities for collaborative learning, facilitates individualized feedback as well as contact with peers and faculty, promotes reflective and critical thinking due to its asynchronous nature, and permits students to work at a time and pace that is convenient to them. web mediation can bring these benefits to a learning environment; cross-cultural applications of this approach therefore have the potential to be extremely useful and relevant to students' learning. therefore, it is more than worthwhile to review current curriculum goals for viable opportunities for web-mediated crosscultural learning activities. 31 sanjeev singh and jon dron networking: a study in planning and developing cross-cultural collaboration context the indian context pedagogical value in india is directed towards acquiring knowledge in the form of information disseminated by the elders in general, and the teacher in particular (jadhav, 1999). mutual (student-to-student) interaction is less usual, typically being confined to exchange of notes and issues relating to interaction with the management of the teaching programme. students in india place a high value on the amount of effort one puts in acquiring knowledge through textbooks. emphasis on hard work is also reflected in heavy daily teaching schedules fixed by the system. students are always busy in finishing their assignments rather than looking for other resources. there is little relaxation time or flexibility in which to consult other resources because of the time constraints. with the increasing academic competition and pressure on the students to perform well, the education system today encourages earning grades. this is what students concentrate on: whatever is taught is simply absorbed by them, memorized and then reproduced at the time of examination. the university campuses generally do not provide adequate hostels or dormitory facilities. students are geographically dispersed. this prevents an educational institution from simply establishing its own internet and web service department and forces the educational institution to rely extensively upon isp. thus, it is very difficult to implement such networks in the majority of universities and higher education institutions. these are essentially maintained by individuals. the shortage of internet bandwidth, spurred by increased net usage, poses a major constraint in india. the current domestic traffic demands a minimum throughput of 4 gbps, which at present is serviced by an international pipe with a dismal capacity of around 420 mbps. already deficient by 88 per cent, the bandwidth used by institutions in india in general is of the order of 64 kbps to 128 kbps. this is one of the major technical problems in popularizing web-based learning models in india. power backup is also a problem, so the typical server downtime is greater than in the uk. the online discussion groups are also limited by the time constraints. although students may want to join these discussion groups, the institute may be closed and facilities do not exist at home. the uk context the course in delhi receives around 2,000-2,500 qualified applicants each year for 35 places. it is thus unsurprising that the quality of the students is exceptionally high when compared with the more modest ratios of the university of brighton. there are also significant differences in class size, with a modular system in brighton teaching up to 150 students in lectures and tutorials at undergraduate level as opposed to the small, focused cohort of 35 students in india. although the motivation of uk students is often very high (particularly in postgraduate courses) there is a higher proportion of less able and less willing students, as well as greater difficulties providing personal care and attention from faculty staff. partly as a result of this, as well as through the active efforts of teaching staff and a slightly less rigidly hierarchical social ethos, there is a greater tradition of spontaneous group working and mutual assistance amongst students in this institution. although technical provision is generally significantly better in the uk than in india, 32 alt-j volume 10 number 2 students must share resources with a far greater diversity and number of colleagues than their indian counterparts, who effectively have a dedicated pc for their own use. internet usage is also somewhat restricted due to constraints placed by janet and the conflicting needs of different groups of users within the university. for example, use of internet relay chat (irc) is explicitly banned. however, the vast majority of students of computing not only have a pc of their own but also a connection to the internet with a private isp, so it is far simpler for them to make use of the netand web-based resources to allow flexible working. this extra time helps to compensate for the somewhat lower innate abilities of the majority of students. the ethos in brighton is only slightly more relaxed than that in delhi with regard to timetabling and teaching methods, and similar constraints exist to those in india. there are, for example, few ways that provision is made for the creation and use of the online teaching in the allocation of staff resources. both institutions are still culturally based in a traditional model of lectures, tutorials and workshops. although this is changing as a result of pressures from within and without the universities, it provides a greater set of obstacles than would be the case if online learning were a more integrated part of the overall system. methodology a prime concern is that students be adequately trained to use the system so that any anxieties or apprehensions about using the technology are overcome by the time the crosscultural web-mediated learning activity begins. this entails building sufficient practice time into the course curriculum. a prototype website has been launched on one of the servers of the university of brighton (eu-india prototype website, 2000). this resource is built by and for students of the universities of delhi and brighton. its purpose is to provide a shared resource on the subject of networking for students in india and the uk. a brighton-delhi discussion group has also been created and students are strongly encouraged to interact with their collaborative institute and to comment on the resources. it was important to identify common ground without diminishing the importance of cultural differences. both cultural-general and cultural-specific approaches to designing the virtual space and learning activity were considered. various articles and tutorials were chosen to cover the common course curriculum of both the universities. these were uploaded and classified to help students structure their learning. online tests were also mounted for self-assessment. the students were asked to share their knowledge and queries using the mechanism presented on the site. students were also asked to prepare a database of resources (urls, notes and suitable links) based on their mutual interaction. this was enabled using a web application (cofind) developed by one of the authors (dron, mitchell, siviter and boyne, 1999). cofind is designed to discover what sorts of metadata are valued by its users. learners enter qualities, the things about a resource which are considered valuable, such as 'good for beginners', 'detailed', 'clear' and so on. these qualities are then used to rate resources that the learners also add to the system. as such, it presents a useful opportunity to identify the aspects of learning resources valued by groups of learners. — sanjeev singh and jon dron networking: a study in planning and developing cross-cultural collaboration findings pragmatic issues selecting groups of students with similar interests was not too difficult, as the basic curricula across the institutions are quite similar. however, this was far from being the end of the problem. students in india generally follow a direct path from undergraduate to postgraduate qualifications as a matter of course, whereas this is relatively rare in the uk. it was decided that the closest match of curricula and size/type of course was between the postgraduate part of the course in india and a conversion masters course (m.sc. in information systems) at the university of brighton, notably in the area of networking technologies. although the curricula for these courses are similar, they are at slightly different ends of the soft-hard divide, with indian students more concerned with the engineering side and british students more involved in the world of organizations and systems. this difference in emphasis was felt to be potentially valuable, allowing the strengths of each group of students to counterbalance the weaknesses of the other. both courses cater for a group of 30-40 postgraduate students. however, as the uk masters' course is a conversion degree, students are usually at a significantly lower technical level than their indian counterparts at the start of the course. to make matters worse, semesters in india start in july as opposed to october in the uk, exacerbating the gulf between knowledge levels. after much investigation of options, we were eventually able to identify a subject and time slot suitable for both groups of students near the start of the second semesters in each institution, which would allow us to apply our requirement for a clear time-frame. this left us with the problem of how to introduce students to each other and engage them in mutually beneficial interaction prior to sharing experience and understanding. the first route to achieving this was through the use of the collaborative bookmarking system. using this, students are able to contribute in an anonymous and beneficial way, with the ability to find resources only loosely related to computing ability. it is a system where clear benefits can be achieved through the efforts of a larger, more diverse group. we felt that it was important to start the socialization process earlier than any significant collaboration effort. as such, we created a simple discussion mechanism and relied on students' curiosity to start it moving. unfortunately, the difference in semester start dates resulted in disappointment for early adopters. a re-launch was planned to make sure students were informed of the resource in each location. however, even then little use was made of this. students in both institutions tend to organize their time according to the need to achieve good assessment results rather than out of pure curiosity. as there was little pressure applied by teachers, and as it was not clear to the students what the benefits would be, it was not heavily used. we were also unable to establish the clear and definite time-frame that we had originally identified as a requirement. technical issues the initial site that was set up to provide a basis for discussion was built using off-the-shelf components (such as a web-based discussion forum) available as part of microsoft's frontpage. although the tool contains some assistance for group working, bandwidth constraints meant that it was not feasible to work together effectively on a single live site. however, it provided a simple means to get a static, set of pages up quickly. both 34 alt-j volume 10 number 2 institutions employ a wide variety of technologies for web serving, from apache under solaris to microsoft's iis under nt to lotus domino. of these, domino seemed the logical choice for the discussion forum due to its ability easily to replicate databases, although bandwidth issues continue to make this a lengthy process. however, costs for extra client licences were high and our budgets were low so the current iteration of the environment makes use of talkltover (http://www.talkitover.com), an environment which provides the benefits of both bulletin boards and mailing lists in a single (albeit slightly slow) package, equally accessible to indian and british students. we are still considering opting for the tried and tested technology of usenet news which, despite its simplicity, provides most of the features needed in a discussion environment and is more usable due to its familiarity and built-in support for replication. however, technical limitations on propagation of newsgroups at the university of brighton have made this hard to implement. we had considered the possibility of synchronous discussions, but the time-zone difference meant that it would be difficult to find times that would fit the timetables of each group of students and we were keen to make it as simple as possible to communicate. cofind, the collaborative bookmarking system, was already written using microsoft's asp. initially two separate systems were installed at each location, but an unfortunate series of technical problems in delhi (where uninterruptible power supplies need to be the size of large freezers and temperature control is a serious issue) led to the loss of all our accumulated data. the bandwidth requirements of the application were not particularly high so it was felt to be unnecessarily difficult to replicate this across both servers. the system has now been implemented on a single server in the uk. however, it has mainly been used by brighton students, which is felt to be the result of the enthusiastic selling of the system by its author. working together after quite brief initial meetings in delhi, most of the time spent on discussing the process has involved exchanges of email between the authors of this paper. we met in the uk for further discussions later the same year, and again in delhi the following year. it is highly significant that most advances have been made as a result of our face-to-face meetings. a lot of the reason for this is to do with the fact that such meetings provide a focused period when all we need to do is to work on the system. however, it is interesting to observe the amount of time spent on getting to understand each other's motivation and expectations, as well as clarifying meanings. discussion the experience in india in spite of the common view of pedagogy as knowledge acquisition, the main problem encountered during the course was technical. it was observed that students were willing to use the mechanism set up in the project but they were restricted by the system. although students may want to join the discussion groups that were set up, for example, the institute was often closed and facilities do not exist at home. this certainly had some effect on lack of momentum sustained in discussions, requiring significant commitment to stay involved. given the students' other priorities, this commitment was generally not achieved. 35 sanjeev singh and jon dron networking: a study in planning and developing cross-cultural collaboration the experience in the uk it was noted at the outset that brighton's support for materials development was oriented to traditional forms of teaching and learning. this posed problems for the collaboration. as moore and kearsley note, development of distance learning in a piecemeal fashion is unsatisfactory from a student and a developer perspective (moore and kearsley, 1996). these problems are exacerbated when trying to integrate materials and resources from two culturally and geographically isolated institutions, with no shared systems. as with the indian students, another major problem was in making students' contributions to the joint effort worthwhile in terms of process and reward. our solution was twofold. firstly, we used the time-honoured trick of making participation a requirement to achieve marks. this is not as satisfying as it might be and much work is still needed to make it desirable to make use of the system as a means to achieving other ends, rather than as an end in itself. we therefore put a lot of effort into making the resources as useful as possible. tutorials unavailable elsewhere (many created by previous years' students as projects) were posted and a lot of emphasis was placed on the importance of finding web-based help and posting it to the collaborative bookmark system. this failed to reach the critical mass needed to make it self-sustaining but was used by some students to help them with individual projects and dissertations. conclusions we believe that whilst we have developed a workable model for collaboration, we are not yet in a position to report significant success in this venture. although we have been planning and running parts of the system for nearly two years, progress has been slow and fitful. part of the reason for this lies in attempting to add the collaboration to our already full timetables, but we have also encountered a host of problems from the technical (for example, delhi's mail server was unavailable for several weeks and an experimental bookmark database was lost to a rogue machine) to the practical (for example, the fact that students in delhi started just as students in brighton finished) to the cultural (coming to terms with different approaches and another ethos). it is significant that most progress has been made when the authors have been able to meet in person. partly this is due to the need to focus on the issues at hand in a limited timeframe, but it also highlights the need for constant checking of cultural cues when discussing a wide range of issues, and the inadequacy of electronic means of communication to achieve this. for reasons already cited, this attempt to collaborate has not been as successful as we had wished. technical issues will soon be less important as the reliability and speed of networked connections improves. however, maintaining the momentum of the project has proved to be an intractable problem. our latest solution is to provide students with assignments and projects requiring the development of tutorials which will be of use to other students in each country. we then collate this growing resource base, resulting in a form of collaboration without the students needing to try too hard. although this is far from our original intentions of enabling students to work together, it provides a low threshold process whereby the strengths of students in each institution may be shared and may provide the glue to enable more successful collaboration in future. this part of the 36 alt-j volume 10 number 2 collaborative project has motivated students to build their own web-based educational resources and now these are maintained by the students in india, including the content development, updating and so on (http:llservices.iicac.iri). acknowledgements authors gratefully acknowledge the support by the european commission under the 'euindia cross cultural innovation network' project. references brislin, r. and yoshida, t. (1994), an introduction to intercultural communication training, thousand oaks, ca: sage. conner, d. w. (1990), 'distance teaching across cultures: implications for instructional design', in proceedings of the 6th annual conference on distance teaching and learning, madison, wi: university of wisconsin. dron, j., mitchell, r., siviter, p. and boyne, c. (2000), 'website evaluation system: collaborative discovering what makes a website good', in proceedings of webnet 2000, october 2000, aace. dron, x, mitchell, r., siviter, p. and boyne, c. (1999), 'cofind an experiment in ndimensional collaborative filtering', in proceedings of webnet 99, hawaii, october 1999, aace. eu-india prototype website (2000), http:lllearntech.bton.acukleuindial, url last accessed 24 december 2001. jadhav, l. (1999), south asian immigrants living in two cultures (eric document reproduction service no. ed431337). moore, m. and kearsley, g. (1996), distance education: a systems view, belmont, ca: wadsworth. murphy, k. l. (1991), 'sociocultural context of turkish distance learning', in proceedings of the seventh annual conference on distance teaching and learning, madison, wi: university of wisconsin. 37 calt12102.fm alt-j, research in learning technology vol. 12, no. 1, march 2004 quality assurance for digital learning object repositories: issues for the metadata creation process sarah currier1*, jane barton1, rónán o’beirne2 & ben ryan3 1university of strathclyde, uk; 2city of bradford libraries, uk; 3university of huddersfield, uk taylor & francis ltdcalt1210210.1080/0968776...research in learning technology0968-7769 (print)/1741-1629 (online)research article2004association for learning technology121000000march 2004centre for academic practiceuniversity of stratchclydelivingstone tower26 richmond streetglasgowg1 1xhuksarah.currier@strath.ac.uk metadata enables users to find the resources they require, therefore it is an important component of any digital learning object repository. much work has already been done within the learning technology community to assure metadata quality, focused on the development of metadata standards, specifications and vocabularies and their implementation within repositories. the metadata creation process has thus far been largely overlooked. there has been an assumption that metadata creation will be straightforward and that where machines cannot generate metadata effectively, authors of learning materials will be the most appropriate metadata creators. however, repositories are reporting difficulties in obtaining good quality metadata from their contributors, and it is becoming apparent that the issue of metadata creation warrants attention. this paper surveys the growing body of evidence, including three uk-based case studies, scopes the issues surrounding human-generated metadata creation and identifies questions for further investigation. collaborative creation of metadata by resource authors and metadata specialists, and the design of tools and processes, are emerging as key areas for deeper research. research is also needed into how end users will search learning object repositories. introduction the emergence of the concept of reusable learning objects has been a major recent development in e-learning (littlejohn, 2003). much discussion and exploratory work has been undertaken, moving us towards what has been called “the learning object economy” (downes, 2001; campbell, 2003), where teachers, course developers and learners can share, reuse and re-purpose digital materials for incorporation into teaching and learning. some potential benefits of this ‘economy’ include: minimising duplication of effort for individual teachers across subject areas; reducing costs for * corresponding author. centre for academic practice, university of strathclyde, livingstone tower, 26 richmond street, glasgow g1 1qe, uk. email: sarah.currier@strath.ac.uk issn 0968–7769 (print)/issn 1741–1629 (online)/04/010005–16 © 2004 association for learning technology doi: 10.1080/0968776042000211494 6 s. currier et al. institutions (duncan, 2003b); and providing access to a wider variety of learning materials. in the past few years, various institutions and projects have been developing repositories for these reusable learning objects (downes, 2003) supported by international standardization work, notably the suite of specifications produced by the ims global learning consortium (ims). downes (2003) suggests that the next stage of development in this “economy of education” should be the development of a network of distributed learning object repositories. because metadata enables users to discover and select digital learning resources suitable to their requirements, it is a vital component of the learning object economy, any future distributed networks and the learning object repositories within them. extensive groundwork has been carried out in this area, mainly centred upon the development of the ieee learning object metadata standard, known as ‘the lom’ (ieee ltsc, 2002). ieee worked closely with the interoperability body ims in creating the lom; hence it is integral to such learning technology specifications as ims content packaging and ims digital repositories interoperability. the uk has played a central role worldwide in the ongoing development of good practice, common usage and appropriate vocabularies for the lom, including graham and campbell’s (2003) ‘uk lom core’ (originally known as the uk common metadata framework). so, given the existence of this work, why is there a need for further quality assurance? the key to answering this question involves distinguishing between the concepts of structure and content. the developments above deal primarily with the structure of the metadata; this paper is concerned with the creation of the content of the metadata fields. once a metadata standard has been implemented within a system, the specified fields must be filled out with real data about real resources; this process brings its own problems. for searchers, these manifest themselves in various ways, including poor recall of available resources and inconsistency of search results. they arise due to errors, omissions and ambiguities in the metadata, many of which are known and understood in other communities of practice with tried and tested solutions. within e-learning the problems of metadata creation have yet to be fully addressed. when this paper was first drafted for alt-c in february 2003, almost no formal research had been carried out into the process of filling in metadata fields describing learning objects. however, informal consultation via e-learning metadata forums revealed a significant number of colleagues who shared our concerns. all agreed, usually from personal experience, that the issue of who creates metadata and how has an important impact on the quality of collections of digital materials for resource discovery by end users. the scope of this paper this paper surveys the issue of metadata creation for digital learning object repositories with an emphasis on quality assurance, presenting three cases of repositories whose experiences have raised issues for debate and further investigation. we have quality assurance for digital learning object repositories: 7 limited our scope to the realm of “human metadata generation” (greenberg & robertson, 2002), wherein a “person intellectually manages the metadata generation”. the issue of machine-generated metadata continues to be the subject of extensive research within such disciplines as computer and information sciences, data mining and artificial intelligence, and there is much to be learned there for learning object repositories. however, we are not yet in a situation where machines can handle all metadata tasks, particularly in an area where many resources have limited textual content. those tasks that require human intelligence and creativity can include such areas as subject classification, educational attributes and determining the contributors to a resource. this is the domain with which we are concerned. we are also only investigating the creation of metadata necessary for resource discovery via searching and browsing, although such metadata can also be used for resource selection. there has been much recent discussion on what has been called secondary or conceptual metadata (mclean & lynch, 2003), usage data or “third party metadata” (downes, 2003). this refers primarily (but not exclusively) to metadata created about the use of a resource for teaching and learning, generally encompassing the idea of reviews or comments by users and intended to facilitate selection of appropriate resources. this issue is in the very early stages of investigation and the distinction may not be as clear-cut as we have stated here, but for clarity’s sake we have excluded it from this paper’s scope. however, there will no doubt be implications for quality assurance. it is worth noting that, although we focus on the creation of metadata necessary for resource discovery, none of the evidence we found in the e-learning domain included research into the ways in which users actually carry out resource discovery. this is a significant gap, perhaps arising from the paucity of working, well-populated repositories; however, other disciplines, such as library and information science, may give preliminary pointers on these issues. this area of research is vital for the formation of priorities and policies for metadata creation. we will return to this in our final section outlining future research questions. metadata is powerful although many learning resources are available on the web, searching the whole web using a search engine such as google can prove unsatisfactory. even with localised or advanced google-type searching within e-learning, learning objects come in a variety of formats; those which are images (including pdf text files), animations or simulations may have very limited textual content to search. browsing a directory of web resources which have been selected on the basis of some criteria, typically subject, can also be time-consuming, particularly when the sought-after material exists only as a small chunk embedded within a larger resource. one purpose of digital repositories is to overcome these problems by collecting good quality resources, preferably in small chunks (duncan, 2003a), together with detailed, consistent information about them, thereby enabling users to conduct precisely targeted searches and to retrieve relevant materials in an efficient and effective manner. 8 s. currier et al. this detailed information about resources, or metadata, is therefore key to unlocking their potential for reuse. at its best, “accurate, consistent, sufficient, and thus reliable” (greenberg & robertson, 2002) metadata is a powerful tool that enables the user to discover and retrieve relevant materials quickly and easily and to assess whether they may be suitable for reuse. at worst, poor quality metadata can mean that a resource is essentially invisible within the repository and remains unused. metadata within e-learning the development of interoperability standards and specifications within e-learning has involved, in the main, software and courseware developers, content developers and, to a lesser extent, teachers. information scientists and librarians, whose expertise lies precisely in the domain we are examining, were simultaneously developing metadata technologies (such as the z39.50 search protocol), standards (such as marc and dublin core metadata) and practice for web-based and other digital resources. for a long time these two fields remained largely separate (mclean & lynch, 2003) and opportunities to benefit from the experiences of the library and information science community were often missed. consequently, the metadata creation problem space has been elided within elearning. downes (2001) stated in his seminal paper on the necessity for a learning object economy: whatever the properties, the authoring of metadata itself will be straightforward for most course designers. because metadata files are machine-writable, authors will simply access a form into which they enter the appropriate metadata information. this statement encapsulates the (lack of) thinking in this area. ims and ieee, in their metadata specifications, have remained agnostic on the matter, offering no guidance on how good quality metadata creation may be ensured (ieee ltsc, 2002; ims, 2001; ims, 2003). we suggest that there are four erroneous assumptions behind the absence of inquiry into how metadata should best be created within e-learning: ● that, in the context of the culture of the internet, mediation by controlling authorities is detrimental and undesirable; ● that rigorous metadata creation is too time-consuming and costly, a barrier in an arena where the supposed benefits include savings in time, effort and cost; ● that only authors and/or users of learning materials have the necessary knowledge or expertise to create metadata that will be meaningful to their colleagues; and ● that, given a standard metadata structure, metadata content can be generated or resolved by machine. we would also put forward a fifth underlying reason, garnered from conversations with e-learning colleagues around the world: that for both technology and pedagogy experts, metadata creation is seen as a tedious chore rather than as a complex intellectual skill which is essential for unlocking access to resources. quality assurance for digital learning object repositories: 9 however, standards-based learning object repositories are now being more widely implemented and practical problems resulting from poor understanding of the metadata creation process are beginning to emerge. these experiences challenge the above assumptions and suggest that there is more to the creation of good metadata than simply filling in a form. three examples from the uk we now summarise relevant findings from three uk repositories. they are presented in chronological order, to illustrate the development of understanding in the area of metadata creation. the scottish electronic staff development library (sesdl) taxonomy evaluation sesdl was a seminal project that, from 2000 to 2001, investigated the development of a learning object repository based on ims specifications, including the ims learning resource meta-data specification (v1.1). funded by shefc’s scotcit programme and based at the universities of strathclyde, edinburgh and paisley, its purpose was to encourage the sharing and reuse of staff development materials within he. the main subject focus was the use of c&it in teaching and learning. in planning this early repository, employing an information specialist was not considered. however, when it was discovered that no appropriate, readily available subject classification scheme was available, a librarian was brought in to create the sesdl taxonomy. a small-scale peer evaluation of the taxonomy was carried out (currier, 2001). this evaluation was not designed to test the proficiency of resource authors and users in creating metadata, although it did point to potential problems in the specific area of subject metadata. the data gathered was complex and could no doubt yield more insight with further analysis; here we merely attempt to illustrate in a simple way the difficulties untrained users found in subject classification. six consultants drawn from the project’s user community were provided with eight learning objects to be classified using the taxonomy. the sesdl team agreed upon ‘ideal’ classifications for the eight objects, against which the consultants’ classifications would be compared. the team provided as much structure and guidance for the evaluation exercise as possible, while not providing the consultants with training which would bring them too far beyond the skill level of the intended users of sesdl. however, even with guidance notes, the ability of the consultants to understand and carry out the task varied considerably. one consultant commented in the postevaluation focus group: “the whole exercise has given me more admiration and respect for librarians” (currier, 2001). the sesdl team assigned a total of 35 classifications to the eight objects, averaging about four classifications per object. in only one instance did all six consultants agree with one of the ‘ideal’ classifications. for five of the eight objects, up to half of 10 s. currier et al. the ‘ideal’ classifications assigned were unused by any of the consultants. in all, only about 50% of the ‘ideal’ classifications had the agreement of more than half the consultants. a total of 71 ‘non-ideal’ classifications were assigned by the consultants, averaging about nine per object. in only 15% of these cases did more than one consultant agree upon the classification. only five classifications (7%) had the agreement of three or more consultants. out of a total of 106 classifications assigned (including ‘ideal’ classifications), only 39 (35%) had the agreement of more than one consultant. these figures indicate that users of sesdl will assign a wide variety of classifications to their objects and will do so inconsistently in comparison with each other. for example (currier, 2001), a learning object consisting of an html page defining the terms ‘vle’ and ‘mle’ was classified by one consultant as ‘student-centred learning’ and ‘collaborative learning’. this appears to reflect their belief that a vle or mle should be used in a student-centred way, for collaborative learning. the implication is that a repository user looking under ‘student-centred learning’ in the browse tree would expect to find a learning object defining the term ‘vle’ there. tables 1 and 2 show the variety of classifications assigned by all the consultants to this object. if the learning objects listed under a particular branch of the sesdl browse tree appear to be randomly or inconsistently classified, this may well influence users’ perception of the quality of the repository as a whole and their willingness to keep searching. the evaluation report (currier, 2001) concluded with a number of recommendations, the most pertinent of which relate to user support: ● explain what classification is for using simple, jargon-free language and examples. ● ensure users understand the availability of multiple classifications, with examples. ● suggest use of both the upload tool and a paper version of the taxonomy. ● provide a note-taking facility, or suggest that users take notes offline. ● suggest users look for other objects of a similar subject to theirs and note the classifications that have been assigned. ● provide a tutorial in classifying objects, designed to lessen the main barriers to effective classification as highlighted in the evaluation. ● make alternative terms visible within the upload tool, to assist with understanding the scope of the classifications. ● provide scope notes for classifications where appropriate. ● add more ‘see also’ notes and allow these to be seen within the upload tool. either include links or automatically bring up the terms referred to. sesdl was a project with a finite lifespan; no ongoing funding was available to implement any of these recommendations. however, its experiences have informed subsequent developments worldwide. perhaps the most pertinent to this enquiry is the final point made in the evaluation (currier, 2001): how can online resource provision services which expect their users to classify their own resources best support this so that future users will be able to find what they want? or is this approach ultimately inadvisable? quality assurance for digital learning object repositories: 11 the bolton woods local history project bolton woods community centre is a uk-online centre offering c&it facilities to the local community. since 1998 it has been a part of community networking project, shipley communities online, which is a partnership offering online learning, occupational guidance services and information and advice on training and work opportunities. in the centre’s bolton woods local history project, members of the community create digital resources, mainly family and local history materials, which are shared with their peers for use as informal learning resources. under the metadata for community content project, a network of community projects, including shipley communities online, were provided with a small repository so learning materials could be shared on a peer-to-peer basis with other communities. metadata for the resources was required to facilitate this sharing. table 2. overlap between classifications chosen by consultants classifications chosen by consultants no. of other consultants choosing this classification (out of 6) 1.3.4 educational development/approaches to teaching/student centred learning 0 1.5.1.2 educational development/educational environments/ electronic classrooms/web-based teaching 1 2. educational technology 0 2.9 educational technology/internet 0 2.16.2 educational technology/software packages/virtual learning environments (intended for resources about the use of specific packages, e.g. blackboard or webct) 0 table 1. number of consultants choosing ‘ideal’ classifications ‘ideal’ classifications no. of consultants who chose the classification (out of 6) 2.2. educational technology/virtual learning environments 4 2.2.3 educational technology/virtual learning environments/managed learning environments 4 1.5.1.1 educational development/educational environments/electronic classrooms/virtual learning environments 3 1.5.1.1.1 educational development/educational environments/electronic classrooms/virtual learning environments/managed learning environments 3 12 s. currier et al. faced with the challenge of creating metadata for a range of materials, the project decided to use the available labour. it became apparent early on that the process of metadata development was one that merited closer attention, so a small study was carried out to investigate whether the creators of resources could also create their own metadata and to compare their ability with information specialists. dublin core metadata was used, with some additional educational elements added for resources used to construct learning pathways (e.g. ‘audiencelevel’ and ‘typicallearningtime’). this proved to be particularly problematic as neither resource creators nor librarians had the pedagogical expertise either to create learning pathways or assign educational metadata. four local history enthusiasts with fairly high-level website design skills created community resources for the project and were given the task of creating metadata for their resources. two qualified librarians working on the project were asked to assign metadata to similar resources. brief guidance notes were provided, although a metadata tool was not used. instead, dreamweaver was employed, as this was familiar to the resource authors taking part. the study involved another librarian involved in the project observing the efforts of both groups in creating metadata and assigning a subjective score out of five in five key areas of managing metadata: understanding metadata; context of resources; choosing elements; assigning values; and subject classification. the findings were: ● resource creators did not have a good understanding of the purpose of metadata or an appreciation of its value; ● resource creators did understand the context of their resources and focused on these elements within the metadata; ● information specialists had a better understanding of the purpose of metadata and included a wider range of metadata elements; ● information specialists struggled with contextual aspects of the metadata; ● neither the resource creators nor the information specialists handled pedagogic aspects of the resources well. table 3 shows the scores gained by the two groups. the greatest difficulty arose from content creators’ lack of understanding of the rationale for assigning metadata (cf. case three, below). the metaphor of finding a book in a library was useful in explaining its purpose. table 3. comparative assessment of success in creating metadata activity information specialists score out of 5 content creators score out of 5 understanding 3 1 context of resources 1 4 choosing elements 4 1 assigning values 4 4 classification 4 2 quality assurance for digital learning object repositories: 13 specific issues included the content creators’ difficulty with the ‘relation’ element, which allows, for instance, a resource to be specified as part of another resource. the ‘rights’ element also highlighted problems in understanding ipr issues. subject classification was particularly difficult for content creators to understand, echoing the findings of the sesdl case study above. these findings suggest that a collaborative approach may yield the best results in terms of metadata quality, since it would engage the strengths of both groups. this small study resulted in an improved approach for the project, involving closer collaboration between content creators and metadata specialists. because neither the content authors nor the librarians were educational professionals, it was noted that further improvement might be facilitated through the involvement of this third group, and that a future study investigating this may be useful. the higher level skills for industry repository (hlsi) the hlsi project is developing a repository for digital learning objects to support the delivery of learning programmes over a wide curriculum area from gcse to higher education. an ongoing development, the project has both fed into and drawn from the development of this paper over the past year. as such, the issues raised and measures taken within hlsi to overcome problems are of great interest and represent a significant potential base for future research. based at the university of huddersfield and funded by local development agency yorkshire forward, the project involves 35 partner organisations, with over 300 members of staff actively participating. learning objects are uploaded to the repository by their authors (generally educational practitioners); they are intended to be shared and reused in e-learning environments across the partnership. by february 2003, the repository had gathered approximately 6500 learning objects from the initial 12 partner organizations, in a variety of sizes and file formats, together with author-generated ieee lom v.1.0 metadata records: the people who submitted resources also provide the metadata, which gives them some ownership over the records. the drawback is that the quality of metadata varies. (barker & ryan, 2003) clearly, in the early stages of the project, there was an assumption that authors who submit resources want ‘ownership’ of the metadata records. this is interesting in light of our initial assumptions and may warrant further investigation, although there was a shift around this issue later in the project. the problem of metadata quality is explained further (barker & ryan, 2003): the difficulty with this process is making sure the authors understand the purpose of the metadata and the methodology used to enter it. a balance had to be struck between getting high quality metadata and not going above the skill level of those entering the metadata. at the moment there is quite a large variation in the quality of metadata for the resources. for example some have spelling errors. this affects the performance [of the] repository so several steps are being taken to improve the process. 14 s. currier et al. as shown in table 4, an evaluation of the metadata records at that time showed that nearly half (46%) of the metadata records were of poor quality or unusable specific problems beyond spelling errors were specifically delineated by ryan & walmsley (2003): ● a single metadata record was duplicated, unchanged, for many or all components of a package of educational content. ● the terminology used by the metadata authors was not consistent. ● some metadata authors described the facets and characteristics of the educational object and not its content, e.g. describing a flash file about internal combustion as ‘flash file’ instead of ‘internal combustion’. ● the metadata tool allowed default values for certain fields and these were used inappropriately. steps for improving the process have now been under way for some months. initial measures involved providing more user support through education and documentation and employing a team of information science professionals to improve the existing metadata (ryan, 2003). ryan (2003) noted that, by june 2003, 2500 metadata records had been re-edited, taking about 550 hours and costing around £6500 (about £2.60 per record). subsequently, the partnership expanded and the project now has access to a large number of information science professionals who have adopted the metadata problem and are actively driving improvements forward. the process of metadata collection has now been split into two stages: 1. the educational practitioner is responsible for entering basic metadata, including title, description, contribution and any technical information they may be aware of. 2. the information scientist is responsible for reviewing the basic metadata and providing additional metadata for subject classification, educational attributes, etc. this process was created by a group of partners in one of the sub-regions covered by the project; it is now being adopted and actively promoted to the whole partnership, supported by continuous staff development and training. the information scientists also made a number of comments, suggestions and recommendations that have resulted in new development areas for the project. these include providing: table 4. metadata quality in the hlsi project prior to intervention metadata quality % of metadata records good 28 moderate 26 poor 32 unusable 14 quality assurance for digital learning object repositories: 15 ● spell-checking facilities within the metadata tool; ● functionality for browsing and searching authority lists (centralized coordination of forms of authors’ names, for instance) when reviewing and entering advanced metadata; ● a clear separation between basic and advanced metadata, with restricted access to the advanced metadata; ● an on-line thesaurus to support advanced metadata entry; ● updated terminology and option lists to reflect current practice; and ● functionality to record/report/track the review status of metadata records. these new development areas represent a significant amount of work and reflect the project’s change in emphasis from a purely technical implementation of specifications and standards to a more focused approach on addressing a significant problem. the creation of the two-stage process also led to a change in the conception of ownership of metadata records and the responsibility and authority for their quality, without removing the need for the resource author’s own expertise. specific metadata issues if the proposed learning object economy bears fruit, we may anticipate repositories and networks holding large collections of tens of thousands of learning objects or more. the above case studies highlight a number of areas where quality of the metadata may impact on the discovery of resources in this economy, which are expanded on below. error management the hlsi case study, with its large numbers of records, found that the issue of errors was significant in their repository. the following quote illustrates (amusingly) a potentially serious problem facing resource discovery. it touches on motivation and support for metadata creation by untrained resource authors, and on the necessity for checking of metadata, whoever it is created by: even when there’s a positive benefit to creating good metadata, people steadfastly refuse to exercise care and diligence in their metadata creation. take ebay: every seller there has a damned good reason for double-checking their listings for typos and misspellings. try searching for “plam” on ebay. right now, that turns up nine typoed listings for “plam pilots”. misspelled listings don’t show up in correctly spelled searches and hence garner fewer bids and lower sale-prices. you can almost always get a bargain on a plam pilot at ebay. (doctorow, 2002) authors’ and other contributors’ names: authority control an obvious example among a multitude of permutations is that of an author’s name changing when they get married. in this and similar cases, a search by name cannot retrieve everything by that person unless there is some kind of name management. 16 s. currier et al. libraries, archives and museums solve this problem by using centralised name authority records, a time consuming and costly exercise. this also applies to names of institutions, conferences, etc. however, bibliographic indexing services have traditionally not done this, instead relying on the author’s name and subject area to do a ‘good enough’ job. there is no evidence either way for a viable approach for learning object repositories. this is one area where research into how users will search would be useful. subject area this is one of the most complex areas of both metadata creation and resource discovery; there is insufficient space to cover it in depth here. all three case studies showed significant problems when untrained authors attempted to create subject metadata. there are two main ways in which subject access to resources is provided via metadata (as opposed to free text searching): indexing (e.g. key words) and classification. how may this difficult and complex task best be carried out for maximum resource discoverability by a heterogeneous population of searchers? should the resource author, who may know their subject area and its terminology well, create the subject metadata? or should it be a metadata specialist, who may know the specific area less well, but may be better placed to step back and think about all the potential users of a resource, and about consistency of key words and classifications across a repository or network? educational metadata it is commonly thought within e-learning that authors are best placed to create educational metadata. the bolton woods case study suggests that those with educational expertise should be involved in this area, where authors themselves do not have such expertise. nevertheless, there are many successful examples of professional cataloguing of specialised materials, such as music or photographs, so it is possible that a new sub-speciality of metadata experts could emerge in this area. accessibility metadata with the new senda (special educational needs and disability act, 2001) legislation in the uk there has been some interesting recent work around developing metadata to describe the accessibility properties of a resource. however, this may prove problematic for metadata creators who are not experts in accessibility. who creates metadata and how? there is much discussion in e-learning concerning the barriers to uptake of reusable learning objects, often focusing on teachers’ unwillingness to engage in the vaunted learning object economy. ipr and the ‘not invented here syndrome’ are often quoted quality assurance for digital learning object repositories: 17 as significant hurdles. however, how much of a barrier is the task of having to create metadata when uploading a resource, particularly with poorly designed tools? moreover, how much of a barrier will it be for teachers searching for resources, if metadata is of such poor quality that they cannot find what they need? in this section we outline some possible permutations of how metadata may be created and by whom. the task of creating metadata may be divided into two stages: the gathering and recording of the information and the expression of that information as conformant metadata. for example, in recording the names of contributors to a resource, a metadata creator may note illustrators, the authors of any text, any institutions that took part, and perhaps an editor. they may then check the cataloguing guidelines of their repository and enter this data in a way that conforms to the guidelines. for an experienced metadata creator, these stages may happen simultaneously, but it is an important distinction in deciding how metadata tasks may be allocated. we suggest the following three models for creating metadata: resource author or contributor only; metadata specialist only; and collaborative. in the first case, the design of metadata tools and user support and training take on a greater weight in terms of quality assurance. metadata quality in all five of the above-named specific metadata issues may be impacted by inadequate provision here. in the second case, the trained metadata specialist carrying out the task may be hampered by lack of knowledge about the pedagogical context, history or subject area of the resource, as shown in the bolton woods case study. within a collaborative model there are a number of possible scenarios. for instance, the author may enter data in certain fields (as in the hlsi case), such as their own name, resource title, institution, educational fields, etc. the metadata specialist may check these for accuracy and conformance and add other selected fields such as subject classification, keywords and accessibility information. this process may be truly collaborative, with the parties communicating directly, or they may work separately, perhaps with the specialist periodically checking records in bulk. in both the bolton woods case and the hlsi case, a collaborative method of metadata creation was chosen after practical experience of the difficulties in taking either the first or second approach. both repositories reported improvements as a result. there are various communities that have a body of research and experience to draw upon in examining some of these issues. the most obvious is library and information science, with an abundance of relevant peer reviewed journals and conferences. the archive and museum communities may also have something to contribute; for instance, the metadata in museum object records is considered to contain a large portion of a curator’s academic knowledge and research (zorich, 1991). commercial abstracting and indexing database services have long utilised author-generated metadata, requiring authors to submit abstracts and keywords with papers. research in this field has found that authors “may lack knowledge of indexing and cataloguing principles and practices, and are more likely to generate insufficient and poor quality metadata that may hamper resource discovery” (greenberg & robertson, 2002). there has been one formal information science study so far on the issue of author generated and collaborative metadata (greenberg & robertson, 2002), which looked 18 s. currier et al. at this with regard to dublin core metadata in support of the semantic web, a development which aims to bring structured knowledge representation to the web’s meaningful content. this study concluded that: … the integration of expert and author generated descriptive metadata can advance and improve the quality of metadata for web content, which in turn could provide useful data for intelligent web agents, ultimately supporting the development of the semantic web. […] if such partnerships are well planned and evaluated, they could make a significant contribution to achieving the semantic web. conclusions: issues for research analysis of the studies above enables several areas to be identified where focused investigation would produce useful information for decision making by developers and managers of repositories: ● what are the important cultural factors that may influence the e-learning community’s particular approach to metadata creation? for example, is ‘ownership’ of metadata by resource authors important? if so, how may this need best be met? in the hlsi case, there was a tradeoff between perceived ownership of metadata by resource authors and the quality of the completed metadata. further research as this repository progresses may shed more light on optimum management of this balancing act. ● what constitutes good quality metadata, within individual repositories and within the global networked environment? for example, to what extent does metadata that is ‘good enough’ for local purposes also support effective retrieval by remote users operating in a different contextual setting? can a set of ‘metadata metrics’ be agreed within communities and beyond? ● who is best placed to create the metadata in any given context? for example, to what extent does the type of metadata (subject metadata, educational metadata, etc.) have a bearing? is a collaborative approach to metadata creation the best way forward? since the evidence presented suggest that this is the case, how can this approach be managed effectively? ● how can tools be used to facilitate the metadata creation process and how much effect do they have? the hlsi repository improved their tool design as part of a programme of improving metadata quality, while the sesdl taxonomy evaluation stated that tool design will be vital if resource depositors are to create their own metadata. so how can the design of tools encourage the creation of good quality metadata, whoever is creating it? ● to what extent can the provision of guidelines and training improve metadata creation? for example, can information specialists provide adequate guidelines to enable non-specialists to use a taxonomy effectively? can librarians be trained to create good quality educational metadata? ● what are the costs and benefits associated with the various approaches to metadata creation? for example, are savings at the initial metadata creation stage eroded by subsequent costs such as data cleaning? the hlsi case would suggest this is the quality assurance for digital learning object repositories: 19 case. in addition, does reducing metadata costs within a repository simply increase the cost, in terms of time and effort, to the end user? ● how will users search for materials within learning object repositories and networks? for example, how important is it to have authority control over the names of authors and contributing institutions? what educational attributes will users search for and how? answers to these questions will have a profound impact on decisions about metadata creation. the evidence presented here suggests that a collaborative approach to metadata creation may be necessary, and that good design of processes and tools is important. however, further research needs to be done on specific implementation of these approaches. other issues have been raised, with no clear answers; particularly important is how end users will search repositories. what is clear is that there is work to be done before the e-learning community has a good understanding of the issues surrounding metadata creation, such that effective policies and practices can be put in place to assure the quality of metadata and hence the ability of teachers and learners to access resources. acknowledgements the authors would like to thank the following colleagues for their contributions to this paper: phil barker; lorna m. campbell; jackie carter; charles duncan; gordon dunsire; jane greenberg; jessie hey; neil mclean; david nicol; william nixon; andy powell; pauline simpson; stuart sutton; and steve walmsley. this paper is based on a research paper presented at alt-c 2003 and published in the conference’s research proceedings. references barker, e. & ryan, b. (2003) the higher level skills for industry repository, in: p. barker & e. barker (eds) case studies in implementing educational metadata standards (cetis). available online: http://metadata.cetis.ac.uk/guides/usage_survey/cs_hlsi.pdf (accessed 4 november 2003). campbell, l. (2003) engaging with the learning object economy, in: a. littlejohn (ed.) (2003) reusing online resources: a sustainable approach to e-learning (london, kogan page). currier, s. (2001) sesdl taxonomy evaluation report (glasgow, university of strathclyde). available online: http://www.sesdl.scotcit.ac.uk:8082/taxon_eval/sesdltaxfinrep.doc (accessed 4 november 2003). doctorow, c. (2002) metacrap: putting the torch to seven straw men of the meta-utopia, e-learning guru newsletter. available online: http://www.e-learningguru.com/articles/metacrap.htm (accessed 4 november 2003). downes, s. (2001) learning objects: resources for distance education worldwide, international review of research in open and distance learning. available online: http://www.irrodl.org/content/ v2.1/downes.html (accessed 4 november 2003). downes, s. (2003) design and reusability of learning objects in an academic context: a new economy of education? usdla journal 17(1). available online: http://www.usdla.org/html/ journal/jan03_issue/article01.html (accessed 4 november 2003). 20 s. currier et al. duncan, c. (2003a) granularization, in: a. littlejohn (ed.) reusing online resources: a sustainable approach to e-learning (london, kogan page). duncan, c. (2003b) the value of managing learning objects: an intrallect ‘white paper’ (livingston, intrallect ltd.). available online: http://www.intrallect.com/products/intralibrary/ loms_value.pdf (accessed 4 november 2003). graham, g. & campbell, l. (2003) uk learning object metadata core draft 0.1, july 2003. available online: http://www.cetis.ac.uk/profiles/uklomcore (accessed 4 november 2003). greenberg, j. & robertson, w. (2003) semantic web construction: an inquiry of authors’ views on collaborative metadata generation, proceedings of the international conference on dublin core and metadata for e-communities 2002, 45–52. available online: http://www.bncf.net/dc2002/ program/ft/paper5.pdf (accessed 4 november 2003). ieee ltsc (2002) draft standard for learning object metadata (new york, ieee ltsc). available online: http://ltsc.ieee.org/wg12/files/lom_1484_12_1_v1_final_draft.pdf (accessed 4 november 2003). ims global learning consortium, inc. (2001) ims learning resource meta-data best practice and implementation guide, version 1.2.1 final specification. available online: http://imsglobal.org/ metadata/imsmdv1p2p1/imsmd_bestv1p2p1.html (accessed 4 november 2003). ims global learning consortium, inc. (2003) ims digital repositories interoperability—core functions best practice guide, version 1.0 final specification. available online: http://imsglobal.org/ digitalrepositories/driv1p0/imsdri_bestv1p0.html (accessed 4 november 2003). littlejohn, a. (ed.) (2003) reusing online resources: a sustainable approach to e-learning (london, kogan page). mclean, n. & lynch, c. (2003) interoperability between information and learning environments: bridging the gaps: a joint white paper on behalf of the ims global learning consortium and the coalition for networked information (draft version, 28 june 2003). available online: http://www.imsglobal.org/tai3/mclean.pdf (accessed 25 november 2003). ryan, b. (2003) creating, using and re-using learning objects, presentation, (huddersfield, hlsi project). available online: http://www.cetis.ac.uk/groups/20010809144711/ fr20030807121739 (accessed 4 november 2003). ryan, b. & walmsley, s. (2003) implementing metadata collection: a project’s problems and solutions, learning technology, 5(1). available online: http://lttf.ieee.org/learn_tech/issues/ january2003/index.html#3 (accessed 4 november 2003). special educational needs and disabilities act (senda) (2001) (london, hmso). available online: http://www.hmso.gov.uk/acts/acts2001/20010010.htm (accessed 25 november 2003). zorich, d. (1991) library and museum information: beauty and the beast, spectra, 1991. table 1. number of consultants choosing ‘ideal’ classifications quality assurance for digital learning object repositories: issues for the metadata creation process sarah currier1*, jane barton1, rónán o’beirne2 & ben ryan3 2city of bradford libraries, uk; 3university of huddersfield, uk 1. the educational practitioner is responsible for entering basic metadata, including title, description, contribution and any technical information they may be aware of. 2. the information scientist is responsible for reviewing the basic metadata and providing additional metadata for subject classification, educational attributes, etc. table 2. overlap between classifications chosen by consultants table 3. comparative assessment of success in creating metadata table 4. metadata quality in the hlsi project prior to intervention time to experiment: a response mike davis* and kate denning** *learning technology research institute, university of north london **research and graduate school of education, university of manchester email: mike.davis@unl.ac.uk it is with some pleasure that we were given the opportunity to offer this paper for commentary and we are grateful for the efforts made by readers to help us to refine our thinking. given the constraints of space, we will respond to the main comments in turn. we plan to submit a more considered and elegant paper to a future edition when we have worked more on our model. as we are aware, the use of learning technologies is untheoretical, although informed by a number of theories from a variety of disciplines. part of our thinking was based on a desire to integrate some of these and work towards explanations for what we had seen going on in the courses that we were involved in teaching. that this is recognized in the commentary is much appreciated. our failure to describe action science in more detail was a product of pressure on space. briefly, action science, was developed as 'the science of interpersonal action' and is 'centrally concerned with the practice of intervention' (argyris, putnam and smith, 1986: 35) into frequently mismatched espoused theories and theories in use. the development of skill in action science relates to capacity for reflective action; not only reflection on action but also reflection on reflection in action. this is achieved by making theories of action explicit through the presentation and interrogation of case studies. the absence of examples from the data was the product of limitations on space. other papers we have written for conferences make more explicit the relationship between our thinking and the data (see for example davis and denning, 2000). 8 alt-j volume 10 number i the paper does little justice to the complexities of bion's thinking about basic assumption groups and work groups, but table 1 may help. bion's view was that groups had the capacity to function in any of the three basic assumption groups or to be a work, or sophisticated group, but that the two types are mutually exclusive. basic assumption groups based on fantasy impulsive dependent on leader uncritical rapid task-avoiding no desire to test consequences no patience with enquiry great insistence on feeling does not learn from experience resists change cliche-ridden anonymous ruthless ahistoric valency work groups in touch with reality task-specific shared leadership in service of the task critical reflective high level of commitment to the task conclusions tested with reality questions how t o achieve goal seeks for knowledge learns from experience seeks change dynamic ownership of responsibility collegiate and interdependent aware of passage of time cooperation of individuals table i: distinguishing between basic assumption and work groups situated learning was not pursued in anything like the depth it deserved and we are working on another paper that makes more explicit its relationship to computer-mediated learning, particularly in the context of action science. it was included, however, to demonstrate that we were aware of its potential implications for groups working collaboratively through personal case studies. the role of the online tutor here is of considerable importance and the paper failed to do justice to this and its relationship to the development of expertise. our use of the word 'scaffolding' is firmly based on the work of vygotsky and we did not mean to imply merely the restricted use of students building on one another's comments. indeed, this relates to our thinking about the role of the online tutor that we have explored elsewhere (see for example denning and davis, 2000). the title of the paper was deliberately playful, indicating the tentative nature of our conclusions. for us, at least, and perhaps for other readers, our thinking may inform future speculation about the nature of the online learning environment. neither of the authors is a psychologist but we regard some psychological literature as helpful in exploring the learning environments in which we have worked. we consider wilfred bion's work to be particularly important in thinking about the nature of groups and their impact on people's collaborative experiences. his theorizing led to the creation of the tavistock institute and to the considerable insights that this organization has given to social interaction in both the home and the workplace. 9 mike davis and kate denning time to experiment a response our use of the 2x2 matrix has clear limitations and we have struggled to make it work to our satisfaction. it did, however, enable us to characterize what we saw as some stereotypical outcomes of creating learning environments in cyberspace. this represents the provisional nature of our thinking. the suggestion of spider diagrams is extremely helpful and we will return to this in our future paper. the value of the model for us is that it helps to shape our, often muddled, reactions to the problem of why some groups were successful and others less so. it is not a definitive statement and some of the comments made by readers have indicated a way forward. references argyris, c., putnam, r. and smith, d. (1986), action science, san francisco: jossey-bass. bion, w. (1996), experiences in groups and other papers, london: tavistock. davis, m. and denning, k. (2000), 'online learning: frontiers in the creation of learning communities', proceedings of the networked learning 2000 conference, university of lancaster. denning, k. and davis, m. (2000), 'computer mediated communication in adult education', in d. watson and t. dowries, communications and networking in education, kluwer: amsterdam. 10 doi:10.3402/rlt.v20i0/14395 open educational resources: staff attitudes and awareness vivien rolfe* faculty of health and life sciences, de montfort university, leicester, uk (received 25 october 2010; final version received 27 april 2011) attitudes are changing in education globally to promote the open sharing of educational courses and resources. the aim of this study was to explore staff awareness and attitudes towards ‘‘open educational resources’’ (oer) as a benchmark for monitoring future progress. faculty staff (n �6) were invited to participate in semi-structured interviews that facilitated the development of a questionnaire. staff respondents (n �50) were not familiar with the term oer but had a clear notion of what it meant. they were familiar with open content repositories within the university but not externally. a culture of borrowing and sharing of resources exists between close colleagues, but not further a field, and whilst staff would obtain resources from the internet, they were reticent to place materials there. drivers for mobilising resources included a strong belief in open education, the ability of oer to enhance individual and institutional reputations and economic factors. barriers to oer included confusion over copyright and lack of it support. to conclude, there is a positive collegiate culture within the faculty, and overcoming the lack of awareness and dismantling the barriers to sharing will help advance the open educational practices, benefitting both faculty staff and the global community. keywords: open educational resources; staff attitudes; sustainability introduction the massachusetts institute of technology (mit) first talked about placing learning materials for free on the internet in 2001 (goldberg 2001), and soon after the term ‘‘open educational resources’’ (oer) emerged and was defined as: ‘‘educational resources, enabled by information and communication technologies, for consultation, use and adaptation by a community of users for non-commercial purposes’’ (unesco 2002). as the oer movement has gained momentum, so has the definition expanded to embrace the benefits of providing opportunities for knowledge transfer and to facilitate the sharing of best practices (the william and flora hewlett foundation 2010). central to the growth of the use of oers has been the technological developments supporting the production and retrieval of resources, and the emergence of a clear and simple licensing mechanism under creative commons (bissell 2009). since the first mit initiative, the number of repositories storing oer has grown in number and constantly shifts (ceri and oecd 2007; friesen 2009). the oer placed in these repositories range in granularity from whole courses such (page number not for citation purpose) *email: vrolfe@dmu.ac.uk research in learning technology vol. 20, 2012 rlt 2012. # 2012 v. rolfe. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http:// www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 14395 doi: 10.3402/rlt.v20i0/14395 http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 as those offered by the opencourseware consortium (ocwc 2011), to materials and learning objects such as those available on the multimedia educational resource for learning and online teaching (merlot 2011) and to raw asset files (jorum 2011). search for free open educational resources on jorum and this retrieves �12,000 items licensed for use worldwide under creative commons, giving users permission to reuse resources with a minimal restriction such as a requirement to attribute the author, to share alike or to prevent commercial use (creative commons organisation 2011). one of the major challenges that the oer movement faces if it is to flourish is that of long-term sustainability (atkins, brown, and hammond 2007; d’antoni 2008). as friesen notes (2009), the lifespan of a content repository can be fragile, and in his example he describes 11 repositories that were discontinued in the present decade, of which only two were over 5 years old. there are many factors that contribute to sustainability, central to which is the role that institutional staff play in producing and using the content, and many staff have already voluntarily chosen to champion and pioneer oer. downes (2007) describes the importance of staff, partnerships and communities in the oer movement, and also academic staff are fundamental to providing quality assurance and learner support that are important drivers of sustainability (d’antoni 2008). if institutional staff are to play such a critical role as indeed they have up to now, it is essential to understand their attitudes and behaviours towards oer to provide a benchmark to monitor progress and to identify strategies for training and support. a small number of studies have explored staff attitudes towards oer. in a global questionnaire of 193 institutional managers from 49 countries, the most significant barriers surrounding the use of oer included the lack of time and lack of a reward system (ceri and oecd 2007). the strongest perceived opportunity was the ability to gain access to the best quality materials. fundamental to the sustainability of oer is the notion of sharing, and a smaller survey of 109 faculty members investigated more deeply about staff perceptions of knowledge sharing in a korean institution and drivers for this activity were identified as having a clear purpose and necessity to share, along with relevant reward systems (kim and ju 2008). one further study evaluated the incentives and disincentives from the view of the general public in relation to the opencourseware initiative, where incentives to using oer included the fact there was no associated cost and that resources were available at any time to enable the user to pursue a subject of interest (arendt and shelton 2009). the aim of the current research was to explore staff awareness and behaviours surrounding oer with a focus on attitudes to sharing resources, to provide benchmarks for monitoring future progress of these new educational practices as they develop. in this study, the term ‘open educational resource’ was assumed to refer to high-quality, open-licensed materials available for global use (the william and flora hewlett foundation 2010). an abstract of this paper was presented at oer 2010 (rolfe 2010a). material and methods questionnaires and interviews in 2009, academic staff from the faculty of health and life sciences were invited to participate in preliminary interviews (n �6) that were semi-structured, recorded and v. rolfe 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 transcribed. the content was analysed by clustering the statements obtained into emerging themes, facilitated by the use of a spreadsheet (rolfe et al. 2008). the interview responses were used to form the basis of a questionnaire that contained questions on: (1) basic demographics, (2) awareness of the term oer and where to find oer and (3) an analysis of staff attitudes and experiences of open content and associated behaviours. staff attitudes were evaluated using a 5-point likert scale where 1 indicated ‘‘strongly disagree’’ and 5 represented ‘‘strongly agree’’. some questions were open ended. the questionnaires were distributed online via the tool survey monkey and by paper copy and were made available for 1 month. all staff were e-mailed regarding the survey and one reminder was issued. a prize incentive was offered. statistical analysis data were analysed using a chi-squared test and mann�whitney u test. significance was assumed at pb0.05. attitudinal data are presented as the mean likert response 9 standard deviation (sd). results staff demographics out of 307 academic staff (full time and part time), there were 50 survey responses (16% of staff), in which 66% of respondents were female and 34% were male. participation was evenly split across the four academic schools (table 1). awareness of oer and technologies in total, nine staff members (18%) had heard of the term oer (table 2). around half were aware of the university video library and half were aware of dora, an open research paper archive, of which the proportion of male colleagues aware was table 1. staff demographics. female male total no. of respondents age (years) 33 17 50 30�39 7 2 9 40�49 12 8 20 50�59 13 6 19 60� 1 1 2 time in faculty (years) 0�5 13 5 18 6�10 11 3 14 11�20 7 5 12 �21 2 4 6 school pharmacy 7 6 13 nursing and midwifery 10 4 14 allied health sciences 8 3 11 applied social sciences 8 4 12 research in learning technology citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 3 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 significantly higher than female staff (76% vs. 39%, respectively, pb0.05). the virtual analytical laboratory (an open laboratory skills resource) was familiar to 36% of staff. only 20% had heard of jorum, and fewer had heard of biosciencerelated repositories that reflected the subject profile of the staff population that also includes healthcare professions. despite the claimed lack of awareness of oer, the meaning of the term appeared reasonably self-explanatory and clear to staff, eliciting the following open responses: freedom to share learning and teaching resources created by any member of the university to improve the student experience. open access to resources and materials for learning. attitudes and behaviours surrounding borrowing and sharing resources openly the next section of the questionnaire explores current working practices in relation to sharing and borrowing resources. the results indicated that three quarters of staff currently share and borrow learning resources, mostly with close working colleagues (table 3). higher numbers of female staff members shared and borrowed compared to male colleagues (pb0.05). also, 50% of staff obtained resources from the internet, whilst only 12% placed resources there to make them globally available. staff attitudes to borrowing and sharing were then explored more deeply using a series of statements and a likert scale, where 1 represented strongly disagree and 5 represented strongly agree. overall, male and female staff shared similar attitudes to sharing and borrowing resources (p�0.05, mann�whitney u test, data not shown), so that the total data were pooled and are presented in table 4. when asked to indicate how they felt about sharing, strong drivers for sharing resources included a belief in open education and as a reputation enhancer both for the institution and the individual. strong drivers for borrowing were for economic reasons. staff shared a collegiate attitude, and statements regarding the idea that sharing gives colleagues an advantage, or that staff wanted to work individually, were weakly expressed. table 2. awareness of oer and associated technologies. tick the box(s) to indicate which of the following you have heard of ? female (n �33) male (n �17) total (n �50) total% open educational resources (oer) 4 5 9 18 dmu video library 18 8 26 52 dmu dora youtube edu 13 14 13* 11 26 25 52 50 itunesu 13 8 21 42 virtual analytical laboratory (val) 11 7 18 36 mit 11 2 13 26 alt open access repository 5 6 11 22 jorum 5 5 10 20 bioscience image bank 4 4 8 16 bristol biomed image archive 3 2 5 10 *pb0.05, chi-square test (female vs. male) v. rolfe 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 activity and attitudes in new staff versus longer standing staff members the data were divided into staff who had worked in the faculty 10 years or less (n �32) and longer standing members of staff who had been employed by the faculty for 11 years or more (n �18). data from tables 3 and 4 were reanalysed. there was no significant difference in the levels of activity between the two groups (p�0.05, chi-squared test and mann�whitney u tests), with the exception that the proportion of newer staff members borrowing from colleagues was higher than longer standing members (96% vs. 68%, respectively, pb0.05), and new staff were table 3. borrowing and sharing behaviour. statement female (n �33) male (n �17) total (n �50) total% do you make your materials available to others to share? yes 25 12 37 74 no 3 4 7 14 unanswered 5 1 6 12 to whom do you make your resources available? academic colleagues who you work closely with 24* 9 33 66 academic colleagues within the faculty 10 8 18 36 other academic colleagues, for example, through a subject network 9 4 13 26 academic colleagues outside the faculty but in dmu 5 6 11 22 openly available to anyone who wants them globally 4 2 6 12 unanswered 3 3 6 12 do you use resources that have originated elsewhere? yes 25 13 38 76 no 3 3 6 12 unanswered 5 1 6 12 where do you obtain resources? academic colleagues who you work closely with 21* 6 27 54 resources taken from the internet 16 9 25 50 resources supplied through a textbook manufacturer 13 9 22 44 other academic colleagues, for example, through a subject network 6 8 14 28 academic colleagues within the faculty 6 7 13 26 academic colleagues outside the faculty but in dmu 2 4 6 12 *pb0.05, chi-square test (female vs. male). research in learning technology citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 5 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 more likely to borrow from close colleagues than more experienced staff (75% of newer staff vs. 37% of longer standing staff, pb0.05). newer staff members had greater concerns over copyright and were more positive towards the notion that ‘‘borrowing resources saves time and money’’, whereas longer standing members of staff viewed more positively the statements ‘‘it is not easy to adapt resources to meet my needs’’ and ‘‘i prefer to work individually and use my own resources’’ (pb0.05, mann�whitney u test). activity and attitudes by age group the data were divided into age groups, 30�49 years (n �29) versus 50 plus (n �21). data from tables 3 and 4 were reanalysed. there was no significant difference in the attitudes and behaviours between the two groups (p�0.05, chi-square test and mann�whitney u tests). general comments regarding oer implementation the following preferences were offered as incentives to engaging in oer in the future (table 5), and the results show that the strongest preferences included the need for technical and it assistance, and clarity over copyright, whereas a lower priority was the need for incentives for people to produce resources. when asked about copyright ownership in more detail, just over half (54%) of the total staff thought that copyright for educational resources resided with the university, but 24% were unclear, table 4. staff attitudes towards using educational resources. statement mean sd indicate how do you feel about sharing your educational resources? i believe in open education 4.1 0.6 oer enhances the reputation of my institution 4.0 0.7 sharing resources enhances my reputation as a good teacher 3.9 0.8 i’d be happy for someone to adapt my resources 3.9 0.8 i would need to be acknowledged as the author 3.7 0.7 i have no support for making materials openly available 3.3 1.1 i have concerns over copyright 3.2 1.1 why give away resources to other universities 2.3 0.9 i don’t wish to give colleagues the advantage of using my resources 2.2 1.2 i don’t wish to share resources that i’ve spent a long time preparing 1.9 0.8 how do you feel about borrowing educational resources? borrowing resources saves time and money 4.0 0.6 i only use resources from someone i know and trust 3.5 1.0 i have trouble finding resources of high enough quality 3.3 1.0 i’d only use resources recommended by someone i know and trust 3.2 0.9 it is not easy to adapt resources to meet my needs 3.1 1.0 i don’t really know where to look for them 2.8 1.1 i prefer to work individually and use my own resources 2.7 0.9 likert scale data where 1 � strongly disagree and 5 � strongly agree. v. rolfe 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 and 16% felt that resources were copyright of the individual. in fact, the situation is complex with electronic resources copyright of the university, although other forms of teaching material are copyright of the individual member of staff. in a final section on the questionnaire, staff opinion of oer was expressed in an open comment section. staff comments raised the notion of needing a clear institutional strategy and a clear vision of the purpose for becoming involved in oer and how an institution could ensure that contributions were duly recognised: a clear purposeful reason why it is good for me personally and for the institution. a strong coherent strategy, not a half hearted, two or three people do all the work with no proper policy. danger that other people nick your ideas and get all the credit in an institute which may be seen to be lacking in oer. interview responses the initial interviews that led to the development of the questionnaire themes highlighted levels of awareness and understanding of oer by staff, which were similar to the survey results, with staff perhaps not having heard of the term but having a relatively clear understanding of what it meant. as reflected by the survey, the perception of sharing and borrowing activity going on within the institution was strong, and there were benefits to be gained: i would have thought it was something we should be doing more of because i would have thought there are a lot of people that continually re-invent the wheel. what i want to encourage is between us we can share material in a way that we can releases ourselves for sabbaticals. i would be quite happy with that, the ultimate aim is to get it as widely shared out as possible. if you develop a wonderful resource and only 3 people ever see it well it’s not much of a resource is it. the interviews identified barriers to sharing resources, which included time and it infrastructure and other barriers that were deeper rooted and more personal and that table 5. general comments regarding oer implementation. statement female male total total% what infrastructure, help and rewards would you like to see? it/technical assistance 24 8 32 64 clarity over copyright 22 10 32 64 staff development 22 7 29 58 incentives for people who develop the resources 17 7 24 48 the use of open source services, for example, youtube, picassa, (open to all) 13 8 21 42 an institutional repository (resources open to dmu staff & students) 14 5 19 38 blackboard (resources open to dmu staff & students) 11 7 18 36 wider use of external repository, for example, jorum (open to all) 9 6 15 30 research in learning technology citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 7 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 did not materialise through the questionnaire. whilst the idea that staff shared resources was clearly articulated in the initial interviews and the questionnaire, it is perceived that some staff do not participate, and perhaps this was linked to staff feeling insecure and, therefore, protecting their own work. the need for recognition was also expressed once more, and it was interesting to gain a perspective from staff who are asked to borrow resources and are then rebutted or those staff who produce materials yet do not have the opportunity to share them. these comments suggest how the activities of sharing and borrowing, central to the growth and sustainability of open educational practices, are entrenched in both professional and personal feelings and attitudes. ownership and security i think some people see it as their resources, they have spent their time doing it and they think therefore they are theirs. i think there is a feeling of well if it is available anyone can do it, you would be indispensable if you know what i mean. i think it’s to do with job security and things like that. confidence some people won’t do it so you can ask and you will get knocked back and that kind of puts you off. and also i suppose nobody asks really so you don’t really assume that anybody else would want your resources. recognition you wouldn’t want somebody who is not prepared any lecture notes to nick yours that you have spent all weekend doing. the scales have got to be equal rather than tipped in one persons favour. discussion the use of oer and understanding of open educational practices is advancing globally, and in the united kingdom, large numbers of individuals and institutions are contributing in particular through the higher education funding council for england (hefce) oer programme presently run by the higher education academy (hea) and the joint information systems committee (jisc), (jisc 2010). how to grow and sustain these open educational practices is a major challenge, and operational models have described financial, technical and contentbased solutions, but more current thinking is placing staff and communities at the heart of the sustainability of oer (d’antoni 2008; downes 2007). despite this, only one published study has looked at institutional staff awareness and attitudes to oer (ceri and oecd 2007), although there is a wealth of research ongoing as part of the oer programme, and much to be learned about sharing, use and reuse from previous work on reusable learning objects. the current study, therefore, contributes to the understanding of staff awareness, attitudes and behaviours surrounding oer v. rolfe 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 in a uk university and identifies both incentives and barriers to mobilising resources. the survey is openly available to use and repurpose (rolfe 2010b). in the current study, 50 staff members responded to the survey representing 16% of total academic staff, which although was a small sample, did equally represent all four schools. this is a reasonable response rate for a voluntary questionnaire, and these figures, perhaps, reflect the lack of awareness of oer at this moment in time of the study (staff interviewed and sampled in 2009). the present aim was to provide a snapshot of staff opinion and activity in time and to provide benchmarks for monitoring future progress and activity as oers are promoted and institutional strategy evolves. awareness in total, nine staff members (18%) had heard of the term ‘open educational resources’. no comparable published survey data exists, although the results of a recent survey of global health care professionals and scientists with an interest in sickle cell and thalassemia revealed that out of 226 respondents, 42% were aware of the term ‘open educational resources’ (rolfe, fowler, and dyson, forthcoming), potentially a higher figure due to increased awareness in the united states where many of the respondents were located. in the present survey, half of the respondents were aware of the institutional repositories for video and research publications, although only one fifth had heard of jorum, an external free online repository service developed for united kingdom further and higher education that promotes the sharing of oer globally. staff had a clear understanding of what the term oer meant, which was an important outcome because the clarity of terminology is essential for the positive engagement of people in new ideas and initiatives. in a previous survey of staff attitudes towards e-learning, academic staff were unclear as to what e-learning meant or the types of technologies involved, were less aware of the potential benefits and less likely to adopt e-learning strategies in their teaching (rolfe et al. 2008). our current study, therefore, endorses the term oer as one that, although the majority of staff had not heard of, the name does clearly relay the activities that it represents. activity and attitudes fundamental to the oer movement is an individual’s desire to borrow and share resources, and for institutions to cultivate collegiality. our results show a strong tendency of faculty staff to freely share resources with immediate colleagues, with female colleagues significantly more active than male staff. this concords with previous findings of a multi-centre study with 130 respondents, which looked at attitudes towards sharing e-learning resources (bell and rothery 2006). sharing occurred locally, but staff were apprehensive about making resources more openly available. in our survey, when considering oer more broadly, although half of the staff obtained resources from the internet, only 12% claimed to place resources there to make them globally available, and again a similar picture in bell and rothery’s study with only 6% of respondents making their educational materials openly available. as dholakia, king, and baraniuk (2006) suggested in their literature review in which they propose a model for oer sustainability, central to sustainability is the community and growth of a critical mass of interested individuals and it adopting technologies and approaches to create networks and discussion. then, a research in learning technology citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 9 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 vibrant community emerges. in our faculty, there is clearly a positive element on which to build on, but understanding the motivations and characteristics of potential users is important to develop strong and sustainable strategies and practices. it is vital to understand the drivers and motivational forces behind adopting oer and to identify barriers and challenges. in the current study, the strongest drivers for sharing and borrowing included a belief in open education, as a reputation enhancer both for the institution and the individual, and economic factors that were a significant reason for participating. staff shared a collegiate attitude, and statements that oer would give colleagues advantages and wanting to work individually were weakly expressed, although deeper rooted concerns were more readily expressed during the interviews. in the interviews, staff were concerned with job security and wanted clear recognition for time spent producing oer. some staff lacked the confidence to make their resources available. in a multi-national survey of institutional managers (ceri and oecd 2007), they perceived that staff barriers to the use of oer would include time and skills, and barriers to producing oers would be lack of reward systems, but this surveys none of the more personal attitudes such as lack of confidence or lack of job security. these motivational factors are different to those observed in other studies, reflecting the idiosyncratic nature of institutions and departments, and differences in research techniques. larsen and vincent-lancrin (2005) suggested a strong motivational force for people to make oer available was that material would be adopted and modified by others. kim and ju (2008) observed in a korean university that staff strongly viewed knowledge sharing as an important motivator, and similar to our study, a lack of trust, openness and it infrastructure were not expressed by staff as strong dissuading factors. this study selected staff randomly and therefore might have produced a fairer picture of opinions than the current study that was self-selecting. oer moving forwards in terms of oer moving forwards within the institution, staff expressed a need for it and technical assistance, clarity over copyright and staff development. copyright is a recurring barrier, and when questioned, staff were mixed as to their understanding of where the copyright of educational materials resided, with the university, the faculty or the individual? similar confusion in the ownership of digital materials was observed in bell and rothery’s study (2006). institutions need to provide clear and transparent copyright policies and guidance, and articles such as that by bissell (2009) that explain about open licences should be used as a starting point since it provides clear recommendations to educators and policy makers on how to proceed with copyright licensing for oer. other information sources, including jisc legal, explain the permissions that need to be obtained to mobilise material, and where these permissions might lie (jisc legal 2011). the role of individual pioneering staff is fundamental to the long-term sustainability of the oer movement, but institutions need to provide support and resources to embed oer into educational practices (atkins, brown, and hammond 2007). as with e-learning in general, a sustainable approach requires cultural and institutional change, where staff share visions and are supported by organisational structures (gunn 2010). in theory, the emergence of strong communities will provide a self-fulfilling prophecy, providing a community of producers, users and re-users (dholakia, king, and baraniuk 2006) and also providing a motivational force to v. rolfe 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 sharing work (larsen and vincent-lancrin 2005). many oer projects, such as the ‘‘humbox’’ repository, have achieved an end-user community and through this have demonstrated that user activity extends beyond initial project activity (borthwick and dicken 2010), and this addresses the view of downes (2007) describing the importance of all staff and partnerships in the building of oer communities. study limitations different institutions will have different cultural contexts. the current study was limited to one faculty within one higher education institution. the survey sample was small with 50 respondents, and larger scale studies are required in multiple centres to provide a rich body of data from which to draw conclusions. the interviews yielded a broader selection of views than the open questions could ascertain, so further research needs to incorporate appropriate methodologies for obtaining a richness of opinion. selecting participants on a voluntary basis is likely to engage more proactive staff, so responses may well be more positively biased. an alternative strategy would be the randomised selection of participants and employing alternative market research strategies for improving the response rate. conclusion as interest in oer advances in the united kingdom, it will be critical to understand staff attitudes and behaviours surrounding it, particularly in a climate of growing financial constraint. the current study provided an evaluation of staff perceptions of oer and associated behaviours and found that, although staff were unfamiliar with the term ‘open educational resources’, it was rather self-explanatory, and this should ease the promotion of further training and activities. there is already a culture of localised sharing and borrowing of material, but for staff to engage in open education practices beyond the institution, work needs to be done to explore the wider barriers. these might include some of the more deep-rooted concerns that emerged in the interviews, and these would need to be explored to successfully transform working practices. this study provides a benchmark against which to monitor future progress and also identifies pitfalls that need to be addressed to build a sustainable programme of oer production, use and reuse within the institution. acknowledgements the author thanks the uk subject centre for bioscience for supporting this project as part of their open educational resource phase 1 project ‘‘an interactive laboratory and fieldwork manual for the biosciences’’. references arendt, a. m. & shelton, b. (2009) ‘incentives and disincentives for the use of open course ware’, international review of research in open and distance learning, vol 10, no. 5, pp. 1� 26. atkins, d. e., brown, j. s. & hammond, a. l. (2007) ‘a review of the open educational resources (oer) movement: achievements, challenges, and new opportunities’, february 2007. http://www.oerderves.org/wp-content/uploads/2007/03/a-review-of-theopen-educational-resources-oer-movement_final.pdf [accessed 8 august 2011]. bell, v. & rothery, a. (2006) ‘e-sharing: developing use of e-repositories and e-libraries for learning and teaching, in is information technology shaping the future of higher research in learning technology citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 11 (page number not for citation purpose) http://www.oerderves.org/wp-content/uploads/2007/03/a-review-of-the-open-educational-resources-oer-movement_final.pdf http://www.oerderves.org/wp-content/uploads/2007/03/a-review-of-the-open-educational-resources-oer-movement_final.pdf http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 education?’, proceedings of the 12th international conference of european university information systems, estonia. 28�30 june 2006. bissell, a. n. (2009) ‘permission granted: open licensing for educational resources’, open learning, vol. 24, no. 1, pp. 97�106. borthwick, k. & dicken, a. (2010) ‘report 1: tracking the use of humbox oer’, [online]. available at: http://www.llas.ac.uk/resourcedownloads/3233/humbox_tracking_report1.pdf [accessed 8 august 2011]. centre for educational research and innovation and oecd organisation for economic cooperation and development (ceri). (2007) ‘giving knowledge for free’, the emergence of open educational resources, [online]. available at: http://www.oecd.org/document/41/ 0,3343,en_2649_35845581_38659497_1_1_1_1,00.html [accessed 8 august 2011]. creative commons organisation. (2011) http://creativecommons.org [accessed 8 august 2011]. d’antoni, s. (2008) ‘open educational resources the best way forward’, deliberations of an international community of interest, unesco report, [online]. available at: http:// learn.creativecommons.org/wp-content/uploads/2008/03/oer-way-forward-final-version.pdf [accessed 8 august 2011]. dholakia, u. m., king, w. j. & baraniuk, r. (2006) ‘what makes an open education programme sustainable?’, the case of connexions, [online]. available at. http://www.oecd. org/dataoecd/3/6/36781781.pdf [accessed 8 august 2011]. downes, s. (2007) ‘models for sustainable open educational resources’, interdisciplinary journal of knowledge and learning objects, vol. 3, pp. 29�44. friesen, n. (2009) ‘open educational resources: new possibilities for change and sustainability’, international review of research in open and distance learning, vol 10, no. 5, pp. 1�14. goldberg, c. (2001) ‘auditing classes at m.i.t, on the web and free’, new york times, 4 april, [online]. available at: http://web.mit.edu/ocwcom/mitocw/media/nytimes_040301_ mitocw.pdf [accessed 8 august 2011]. gunn, c. (2010) ‘sustainability factors for e-learning initiatives’, research in learning technology, vol 18, no. 2, pp. 89�103. joint information systems committee (jisc). (2010) ‘open educational resource programme’, [online]. available at: http://www.jisc.ac.uk/oer [accessed 8 august 2011]. jisc legal. (2011) ‘copyright law essentials (14/04/2011)’, [online]. available at: http:// www.jisclegal.ac.uk/linkclick.aspx?fileticket�j0cknzsyeg4%3d&tabid�243 [accessed 8 august 2011]. jorum. (2011) learning to share. http://www.jorum.ac.uk/ [accessed 8 august 2011]. kim, s., & ju, b. (2008) ‘an analysis of faculty perceptions: attitudes towards knowledge sharing and collaboration in an academic institution’, library and information science research, vol. 30, pp. 282�290. larsen, k. & vincent-lancrin, s. (2005) ‘the impact of ict on tertiary education: advances and promises’, paper presented at the oecd/nsf/u michigan conference ‘‘advancing knowledge and the knowledge economy’’, 10�11 january 2005, washington, dc. multimedia educational resource for learning and online teaching merlot. (2011) http:// www.merlot.org/merlot/index.htm [accessed 8 august 2011]. open course ware consortium (ocwc). (2011) ‘courses’. http://www.ocwconsortium.org/ [accessed 8 august 2011]. rolfe, v. (2010a) ‘faculty sharing of scientific educational materials: drivers and barriers’, abstract. oer10, [online]. available at: http://www.ucel.ac.uk/oer10/abstracts/1001.html [accessed 8 august 2011]. rolfe, v. (2010b) ‘oer staff attitude survey’, tag oerval, [online]. available at: http:// www.jorum.ac.uk/ [accessed 8 august 2011]. rolfe, v., alcocer, m., bentley, e., milne, d. & meyer-sahling, j. (2008) academic staff attitudes towards electronic learning in arts and sciences, european journal of distance learning, [online]. available at: http://www.eurodl.org/index.php?article�313 [accessed 8 august 2011]. rolfe, v., fowler, m. & dyson, s. (forthcoming) ‘global oer awareness survey of science and healthcare professionals’. diversity in health and care. v. rolfe 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 http://www.llas.ac.uk/resourcedownloads/3233/humbox_tracking_report1.pdf http://www.oecd.org/document/41/0,3343,en_2649_35845581_38659497_1_1_1_1,00.html http://www.oecd.org/document/41/0,3343,en_2649_35845581_38659497_1_1_1_1,00.html http://creativecommons.org http://learn.creativecommons.org/wp-content/uploads/2008/03/oer-way-forward-final-version.pdf http://learn.creativecommons.org/wp-content/uploads/2008/03/oer-way-forward-final-version.pdf http://www.oecd.org/dataoecd/3/6/36781781.pdf http://www.oecd.org/dataoecd/3/6/36781781.pdf http://web.mit.edu/ocwcom/mitocw/media/nytimes_040301_mitocw.pdf http://web.mit.edu/ocwcom/mitocw/media/nytimes_040301_mitocw.pdf http://www.jisc.ac.uk/oer http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jisclegal.ac.uk/linkclick.aspx?fileticket=j0cknzsyeg4%3dtabid=243 http://www.jorum.ac.uk/ http://www.merlot.org/merlot/index.htm http://www.merlot.org/merlot/index.htm http://www.ocwconsortium.org/ http://www.ucel.ac.uk/oer10/abstracts/1001.html http://www.jorum.ac.uk/ http://www.jorum.ac.uk/ http://www.eurodl.org/index.php?article=313 http://www.eurodl.org/index.php?article=313 http://www.eurodl.org/index.php?article=313 http://www.eurodl.org/index.php?article=313 http://www.eurodl.org/index.php?article=313 http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 the william and flora hewlett foundation. (2010) ‘open educational resources’, hewlett, [online]. available at: http://www.hewlett.org/programs/education-program/open-educational-resources [accessed 8 august 2011]. united nations educational, scientific and cultural organization (unesco). (2002) forum on the impact of open courseware for higher education in developing countries. final report, [online]. available at: http://unesdoc.unesco.org/images/0012/001285/128515e.pdf [accessed 8 august 2011]. research in learning technology citation: research in learning technology 2012; 20: 14395 doi: 10.3402/rlt.v20i0/14395 13 (page number not for citation purpose) http://www.hewlett.org/programs/education-program/open-educational-resources http://www.hewlett.org/programs/education-program/open-educational-resources http://unesdoc.unesco.org/images/0012/001285/128515e.pdf http://researchinlearningtechnology.net/index.php/rlt/article/view/14395 editorial letting go of the old and embracing the new change... is not a one-step process, for academics, but one that sees them in transition, as they let go of the old and prepare to embrace the new. a key question that needs to be asked is: how much of the past do they need to let go in order to embrace the future? (mcmurray,2001:80) a major focus of the research and practice in the field of learning technology is implementing and developing learning technologies in order to change or influence the student learning experience. we are all aware, however, that the process of developing or implementing learning technologies often requires adaptation or changes in staff roles and skills, in organizational culture and structures, in strategies and policies, in attitudes and perceptions as in well as the theories and models we use to explain and understand the learning and teaching experience. in this issue of alt-j we present ten papers. on first reading these papers represent quite an eclectic mixture of subjects and methods: a closer inspection, however, reveals a common theme that runs through all the papers, that of change. linked to this theme are notions of evolution, innovation and experimentation. the first two papers present a commentary on a paper published in a previous edition of alt-j. the following six papers were presented at the alt-c annual conference entitled 'changing learning environments' in edinburgh last year, while the final two papers report on the results of two different evaluation studies. the papers by conole et al., by davis and denning, and by wise represent an attempt to develop the relationship between alt-j authors and readers and facilitate a more interactive and perhaps reflective dialogue between the two parties. conole, oliver and seale summarize readers' comments on an article by davis and denning published in issue 9.2 of this journal entitled 'almost as helpful as good theory: some conceptual possibilities for the online classroom'. this is followed by a response from davis and denning who 2 alt-j volume 10 number i remark that 'some of the comments made by readers have indicated a way forward'. wise describes a program he has written using javascript that is designed to help life sciences students prepare a research protocol. in an attempt to engage with alt-j readers, he has made the program available on the web and invites comments and suggestions from alt-j readers for 'future collaborative research'. the notion of change through meaningful dialogue is also highlighted in the papers by mcavinia and by lockyer et al. mcavinia describes two approaches to the design and delivery of web-based key skills resources: the central resource and customized, individualized sets of departmental pages. mcavinia argues that the adaptation of the centralized key skills resources to departmental contexts is not simply a question of changing the packaging or 're-branding' them. it requires a meaningful dialogue with members of the department in order to produce a product that has contextual relevance for students. lockyer, patterson, rowland and hearne describe a small pilot study, which used asynchronous web-based communication tools with student teachers in australia to facilitate mentoring and peer support during teaching placements. the study found that while students seemed comfortable using the online peer support, they were not as 'reflective' as the authors had hoped they would be. in considering the implications of their findings lockyer et al. conclude that changing students' descriptions of teaching experience into reflections on teaching experience may be achieved by 'preparing both the students and the lecturers to identify the difference between a reflective and descriptive posting and developing strategies to support reflection'. the papers by riley and gunn et al. use knowledge gained from theory and practice in order to develop our understanding or use of learning technology. the paper by gunn, french, macleod, mcsporran and conole combines literature review, case studies and feedback from discussion groups held at alt-c to offer a range of interpretations for the underlying causes and implications of gender differences in computer-supported learning. the paper seeks to change and move our understanding of gender differences beyond considering differences in computer access and skills and challenges us to think about whether it is possible to change the computer-supported learning environments available to students so that they offer more 'widely applicable solutions' to the gender issues identified. the paper by riley outlines the concept and theoretical underpinnings of 'virtual culture' and explores how it might be developed to evaluate the 'innovative' potential of learning technologies in tertiary education. the notion of 'virtual culture' was first conceived to explain human cultural evolution, but riley develops and applies the concept to learning technologies by proposing four criteria by which they can be evaluated for their 'virtual culture potential'. foster-jones and beazleigh, in their paper, describe two open university metadata projects in order to illustrate how they have tried to apply the ims object metadata scheme. in discussing the implications of their experiences for future work they argue that 'it is crucial that we invest our efforts and resources in this development phase so that we can contribute to the evolution of metadata standards, using our experiences to create one that best suits our needs'. fiddes et al. and duffy et al. present two experimental studies that attempt to further our knowledge of how student performance in computer-based learning environments 3 editorial compares with that in non-computer-based environments. fiddes, korabinski, mcguire, youngson and mcmillan present results from an experiment that sets out to compare secondary students' mathematical performance in a computer-based examination with that in a paper-based examination. the experiment was specifically designed to examine the effect of two variables: rewording and medium. a major conclusion drawn from the project is that the assessment medium had no effect on student marks. duffy, gilbert, kennedy and poon wai kwong present a comparison of assessment results achieved by distance learning students and classroom-based students undertaking the same module in a health studies degree course. through a comparison of assessment results the authors found that students studying by distance learning obtained significantly higher end-of-module results than their classroom-based colleagues. the challenge faced by the authors of both papers is to use their results to build on the research of the previous decade in order to develop new ways of interpreting them. the ten papers in this issue have all addressed in some way issues of change, evolution, innovation or experimentation. whilst the papers offer us new ways of conceptualizing or understanding our practice they do not encourage us to let go of the old in order to embrace the new. they encourage us to use what we know or have experienced already in order to explore or investigate (with others where appropriate) what might be possible in the future. in answer to the question posed by mcmurray, 'how much of the past do they need to let go in order to embrace the future?', the answer is not all of it. whilst transition does involve a passing from one place to another, the bridge to the future needs a firm foundation in our past experiences. jane seale, deputy editor alt-j jeff haywood, alt-c 2001 organizing committee, university of edinburgh nora mogey, alt-c 2001 organizing committee, university of edinburgh reference mcmurray, d. w. (2001), 'the importance of "goodness of fit" between organizational culture and climate in the management of change: a case study in the development of online learning', association for learning technology journal, 9 (1), 73-83. 4 research in learning technology vol. 19, no. 1, march 2011, 1–4 issn 0968-7769 print/issn 1741-1629 online © 2011 association for learning technology doi: 10.1080/21567069.2011.555604 http://www.informaworld.com editorial evidence and evaluation in learning technology research taylor and franciscalt_a_555604.sgm10.1080/21567069.2011.555604research in learning technology0968-7769 (print)/1741-1629 (online)editorial2011taylor & francis191000000march 2011 this issue sees the change of name of the journal from alt-j, research in learning technology to research in learning technology – the journal of the association for learning technology. it might seem a small change in the reordering of the title and subtitle of the journal, but it will require a commitment from all of us involved in the journal to change our habits of referring to it simply as ‘alt-j’. the new title reflects our growing recognition of the importance of research in informing learning technology practice and the development of policy. the change in title also reflects our understanding of the community who produce and read such research. we hope that the self-explanatory research in learning technology title better represents the aim of the journal to publish papers from a broad inter-disciplinary field, which encompasses all sectors of education and industry. our intention is to be more inclusive to current and potential authors, reviewers and readers across the world. as we have debated and prepared for this title change, it is the needs of this broad readership that have preoccupied us. there is currently a focus on ‘evidence’ within this community. there is a plethora of evaluative studies becoming available as the field tries to respond to calls from governments, funders and institutions for evidenceinformed practice. learning technology researchers have an important role to play here and we are keen that the journal provides a forum where researchers can share their evidence, open it up to peer review and use the review process and subsequent publication, discussion and citation to develop their ideas. the journal plays a key role in the association for learning technology’s aim to raise the profile of research in learning technology. more widely, we hope that the journal, and the research community it represents, contributes to the effective use of learning technology. with this in mind, how does this issue help us to assess what kinds of evaluative evidence are needed and how we can go about producing it? clearly, we need to be able to produce convincing evidence arising from evaluative studies with explicitly stated research questions and appropriately selected methodology. in addition, evaluating learning technology often has its own challenges, from the common situation of working in multi-disciplinary and multi-national teams (mcandrew, taylor and clow 2010) to the perennial but inexplicable comparison of online to face-to-face teaching (bethel and bernard 2010). the first paper in this issue, zhen li’s study of e-learning community, demonstrates how we might respond to some of the challenges above. li uses archer’s (2007) account of reflexivity as the interplay between the causal powers of structural and cultural properties and the subjects themselves, to analyse how two groups of learners experienced the formation of learning communities within their online programs. there is a clear perspective for the data collection and analysis that “it should be kept in mind that it is their – the learners’ – communities that we hope will emerge and it is therefore important to examine their experiences from their own 2 editorial perspectives”. archer encourages us to look at people’s responses to environmental structures and we must do so through their eyes. what li finds is that there are striking differences between the student experiences in the two programs. contrary to expectations, learning communities emerged in the course underpinned by a didactic content-driven pedagogy, not the course led by constructive design principles. li shows that these findings can be understood by applying archer’s notions of structure and agency to an examination of the student responses to the different course pedagogies, set within the broader social and cultural structure of the chinese perspective. for those looking for ‘evidence’, this small-scale study may not seem relevant, as it involves just over 300 learners in two professional development programs and in a chinese context. however for me this paper demonstrates the importance of research which prioritizes the learner’s perspective; of theoretically informed research; and of careful, open-minded analysis by the researcher. reading this paper should change how we look at our own data and challenge us to think about how we interpret it. as a field, we need to be clear about the kinds of evaluative research we value and use the publication channels we have available to showcase them. the second paper is also a report of a small-scale evaluative study. ming nie and her colleagues report on how an innovative use of learning technology transformed the educational experience. they examined the impact of the provision of e-books preloaded with course materials on the study patterns of work-based, distance learners. taking an action research approach allowed them to consider the opportunities and challenges of the introduction of the technology as experienced by the staff and students involved over time. for anyone interested in, or who is being asked to provide evidence for “what works?”, this study exemplifies the benefit of having clear evaluation questions. the study aims to assess the extent to which “these devices could enhance flexibility in curriculum delivery to better accommodate the needs and demands of highly mobile work-based learners”. they found that students both valued and made use of the ability to have their course resources to hand, allowing these timepoor students to make better use of their study time by studying more in public places, on the move and without internet access. for colleagues looking for ways of improving the flexibility and convenience of study materials, this paper provides some evidence that e-books are at least worth a second look. next, macgregor, spiers and taylor have reported on what seems to be a similar evaluation of an e-learning innovation – audio feedback. students who received the audio feedback felt it was more detailed and easier to understand than students who were in the written feedback group. in addition, staff spent less time producing the audio than the written feedback. having established student perceptions the evaluation then tackles a more ambitious question. the evaluation is explicitly fore grounded with a statement about why it was conducted “anecdotal evidence gathered by a number of evaluations has hypothesised that audio feedback may be capable of enhancing student learning more than other approaches”. the aim here is to provide evidence that is more convincing than the anecdotes. as we so often see in educational research, despite positive student feedback, the quasi-experimental design provides no statistical support for an association between audio feedback and improvements in learning outcomes. these first three papers in this issue are more than just descriptions of practice. as norris and lefrere point out in their paper, such accounts of experiences are now more commonly shared on the web, and in the informal communities of practice that have emerged there, than in peer reviewed journals. rather, the papers here ask tough research in learning technology 3 evaluative questions – questions which preoccupy much of our current search for evidence. they ask “how can the use of technology transform the educational experience?” and “which technologies are useful in which contexts?” studies such as these add to a growing corpus of knowledge but are not, on their own, enough. a further challenge is how to pull together studies from a rapidly growing field. what do we need to make sense of all these evaluative studies? as bethel and bernard (2010) have proposed in relation to distance education, we need to understand how to choose models for synthesizing our research. they arranged existing models on a continuum from systematic to purposeful and give some guidance to researchers choosing a methodology suitable for their synthesis. synthesizing research evidence is particularly challenging in our field. learning technology research is interdisciplinary and that brings a wide range of research approaches and designs. the field is growing rapidly, with more studies being published and we are being asked to provide recommendations and guidance for practice and policy almost as soon as the latest technology has been made available and the first implementation studies conducted. alongside evaluative studies, and the synthesis and interpretation of them, i think we need challenging conversations. research in learning technology aims to be a place to provoke conversations within our community. for example, norris and lefrere in this issue challenge us to consider the state of the development of online practices, presenting us with a model and examples onto which we can map our own institution’s evolution of online learning. they are frank about the role that technology can play in reducing the costs of education to both institutions and their students. they present us with a range of ways of describing how institutions are using technology to transform their infrastructure, staffing and provision. they challenge us to reconsider how we conceive of higher education fundamentally – emphasizing the role of the learner in selecting their own learning experiences, resources and certification that meet their current needs from those available. this paper contributes to the ongoing conversation about how institutions can make use of technology to respond “nimbly” to a changing environment. in the final paper in this issue, borovik presents a provocative assessment of how the needs of the discipline are served – or not – by learning technology. borovik reminds us not to accept current wisdom on the adoption of ict in university level teaching, but to assess its suitability for our own context. using the example of the teaching of mathematics, borovik explains the ways in which e-learning has failed to meet the expectations of this community and provides a wish list of what is needed now. such a direct challenge to the use of virtual learning environments within a “one size fits all” approach is a welcome addition to the journal. it is written explicitly for an audience outside of mathematics and i hope it provokes conversations within your own disciplines about what such a paper might look like from your own perspective. this issue contributes a mix of practical evaluations and provoking conversations to our field in a way which, i think, suits research in learning technology well. this collection of papers contributes to our understanding of how to conduct and synthesize evaluative research in ways which help us to meet our aims of informing practice and policy. it also demonstrates the need for space within the journal in which we can debate how technology can serve our needs, whether that is the needs of the sector in difficult financial times or the needs of our discipline. it is good to have these conversations and i hope you all find something in this issue that not just informs, but provokes you. 4 editorial references archer, m. 2007. making our way through the world: human reflexivity and social mobility. new york: cambridge university press. bethal, e.c., and m. bernard. 2010. developments and trends in synthesizing diverse forms of evidence: beyond comparisons between distance education and classroom instruction. distance education 31, no 3: 231–56. mcandrew, p., j. taylor, and d. clow. 2010. facing the challenge in evaluating technology use in mobile environments. open learning 25, no: 3: 233–49. rhona sharpe oxford brookes university, uk calt12103.fm alt-j, research in learning technology vol. 12, no. 1, march 2004 building communities for the exchange of learning objects: theoretical foundations and requirements rob koper*, kees pannekeet, maaike hendriks & hans hummel educational technology expertise center (otec), open university of the netherlands taylor & francis ltdcalt1210310.1080/0968776...research in learning technology0968-7769 (print)/1741-1629 (online)research article2004association for learning technology121000000march 2004educational technology expertise center (otec)open university of the netherlandspo box 26906401 dl heerlenthe netherlandsrob.koper@ou.nl in order to reduce overall costs of developing high-quality digital courses (including both the content, and the learning and teaching activities), the exchange of learning objects has been recognized as a promising solution. this article makes an inventory of the issues involved in the exchange of learning objects within a community. it explores some basic theories, models and specifications and provides a theoretical framework containing the functional and non-functional requirements to establish an exchange system in the educational field. three levels of requirements are discussed. first, the non-functional requirements that deal with the technical conditions to make learning objects interoperable. second, some basic use cases (activities) are identified that must be facilitated to enable the technical exchange of learning objects, e.g. searching and adapting the objects. third, some basic use cases are identified that are required to establish the exchange of learning objects in a community, e.g. policy management, information and training. the implications of this framework are then discussed, including recommendations concerning the identification of reward systems, role changes and evaluation instruments. introduction in online education there is a common need to reduce the workload of course developers, to increase the quality of courses and—more generally—to reduce costs. one way of attaining this objective is to stimulate extensive reuse of learning objects when developing new courses. a learning object is defined as: any entity, digital or non-digital, that can be used, reused, or referenced during technology-supported learning (ieee ltsc, 2002). reuse is defined as: the integration of a learning object created by someone else in the context of a new course (koper, 2003). reuse can happen at different levels in a so-called learning object economy (e.g., campbell, 2003; downes, 2003). at the first level, a person reuses something that is self-created. at * corresponding author. educational technology expertise center (otec), open university of the netherlands, po box 2690, 6401 dl heerlen, the netherlands. email: rob.koper@ou.nl issn 0968–7769 (print)/issn 1741–1629 (online)/04/010021–15 © 2004 association for learning technology doi: 10.1080/0968776042000211502 22 r. koper et al. the second level, a person reuses something created by someone else within the boundaries of a specific community or organization. the third level of reuse is when a person buys something in the open market, e.g. a book from a publisher. the second level of reuse is the one that is mostly aimed at when reuse in education is discussed. we will refer to this as a ‘learning objects exchange community’. there is still a big gap to bridge between the worldwide need for reuse in education and actual reuse in practice. thousands of learning objects are already available for sharing purposes (e.g., through systems like merlot, see cafolla, 2003), but only a small portion is actually reused in educational practice. a course developer still has to cope with major problems when trying to find, retrieve, adapt or use materials that could be reused. for instance, in the context of open universities and euniversities in europe, we have experienced that the efforts required to reuse objects in most cases outweigh possible advantages. this is especially true when the objects are developed in different institutions. in many cases, course developers have decided that it is easier to (re-)create the materials themselves than reusing them from others. to tackle these problems, we felt a need to look at the underlying basic principles and requirements for effective reuse in education. the basic question we had was: can we define a theoretical framework to describe the requirements for the development of a successful learning object exchange community? this article presents the framework. it consists of a description of usable background theories, models and technologies and the set of use cases that capture the requirements for a learning object exchange community. theoretical and technological building blocks self-organization theory shaffer and anundsen (1993) define a community as a dynamic whole that emerges when a group of people share common practices, are interdependent, make decisions jointly, identify themselves with something larger than the sum of their individual relationships and make a long-term commitment to the well-being (of their own, others’ and the group as a whole). this vision is related to the principles of complexity theory (see waldrop, 1992), more specifically to self-organization theory (e.g., maturana & varela, 1992). this theory states that some characteristics of the social organization of communities emerge from the interactions of lower level agents. the social organization that emerges (e.g., grouping, specialisation of roles, coordination of actions, distribution of tasks and resources, conflict resolution, quality norms, and interaction standards) in its turn imposes behavioural constraints on the lower level agents by means of social objectives (ferber, 1999). for instance, an important emergent property in communities with lasting participation is ‘trust’. trust is the expectation that arises within a community of regular, honest, and cooperative behaviour, based on commonly shared norms on the part of the members of the community (fukuyama, 1995). when establishing a self-organized system one has to set up certain mechanisms, like initial policies that reflect the purpose of the community and the social behavioural building communities for the exchange of learning objects: 23 constraints (e.g., copyright, privacy, expected behaviour, grouping, user rights management) and mechanisms that facilitate interaction and allow policy change. another important factor is the creation of first and second order feedback mechanisms. first order means that people in the community know what comparable others are doing or have done. this provides information for navigation and provides example behaviour within the community. second order feedback is feedback about the emergent properties in the system: what is the performance of the community and how is it organized (gilbert, 1995)? self-organization research argues that a division of labour, with different activities performed simultaneously by (groups of) specialized individuals, is more efficient than a sequential execution of tasks by unspecialised individuals (bonabeau et al., 1999). however, the specialization of roles among members can change when other needs arise. this implies that a person holds a specialized role in a community, but that the role can change when needed, according to some policy principles. social exchange theory social exchange theory provides a theoretical framework with guidelines to increase active participation and decrease lurking. this theory (constant et al., 1994; thibaut & kelly, 1959), derived from rational choice theory in economics, suggests that there is a relationship between a person’s effect (satisfaction with a relationship) and his commitment to that relationship, i.e. his willingness to contribute and share knowledge. social exchange theory argues that individuals evaluate alternative courses of action so that they get best value at lowest cost from any transaction completed. literature (e.g., hall, 2001) suggests four main reasons that could underlie the motivation and commitment of community members to share knowledge: 1. personal need, or anticipated reciprocity: learner has a pre-existing expectation that he will receive actionable and useful information in return. 2. reputation: learner feels he can improve his visibility and influence to others in the community, e.g. leading to more work or status in the future. 3. altruism, or the perception of efficacy of the community in sharing knowledge as a ‘public good’, especially when contributions are seen as important, relevant and related to outcomes. 4. (tangible) reward: learners negotiate to get some kind of more tangible asset (financial reward, bond, etc.) in return. likewise, others make distinctions between individual (personal need, reputation, reward) and interpersonal factors (altruism) (deci & ryan, 1985), hard (e.g., access) and soft (e.g., satisfaction) rewards (hall, 2001), quantitative and qualitative gain, or intrinsic and extrinsic factors (hemetsberger, 2003). many researchers warn against introducing (more tangible) extrinsic rewards as incentives, since they might destroy the motivation to share for the ‘public good’. when knowledge is considered a public good (mclure wasko & faraj, 2000), knowledge exchange is motivated by moral obligation and community interest (altruism) rather than by narrow self-interest 24 r. koper et al. (personal need, reputation). a public good is a commodity that can be provided only if group members contribute something towards its provision; however all persons may use it (komorita & parks, 1995). greater self-interest reduces knowledge sharing (constant et al., 1994) and people are less likely to use collaborative technologies to share information perceived to be owned by the organization (jarvenpaa & sensiper, 1998). introducing tangible rewards in return for the provision of public goods promotes self-interested behaviour, reduces intrinsic motivation and destroys the public good. the danger is that individuals may appear to be contributing something, but what is not contributed is more significant. this would appear to be more significant when reward mechanisms are in operation (von krogh, 1998). incentives for knowledge sharing should match the spirit of what is to be achieved (sawyer et al., 2000). besides intrinsic and extrinsic factors that motivate individual community members to share knowledge, hemetsberger (2003) identified two other key elements of exchange processes and relations in a community: common goals and values. communities gather around a common interest, a passion (kozinets, 1999). values are important moderators for solidarity in relationships (anderson et al., 1999). communal relationships. on a micro level, intimate communal ties can be distinguished (rheingold, 2000). on a meso level there can be a web of personal relationships in cyberspace or trusting relationships between two or more partners (grönroos, 1999). all these elements of social exchange play a complex and often intertwined role in communities. the right combination of those elements can constitute a powerful mechanism for a self-sustaining system for the exchange and reuse of learning objects. all of them define the willingness (motivation) of the members to exchange learning objects in a community. models for community building revisiting shaffer and anundsen’s definition of communities (1993), summarized above, the features attributed to individuals can be extended by considering qualities attributed to the group as a whole. communities tend to be self-governed, selforganized and decentralized. common goals, values and communal relationships are important moderators in forming communities. communities have their own identity, which can change and evolve in time. a community of practice (cop) is a group of practitioners in a certain field of expertise who share knowledge and experience. the group members typically possess different levels of experience and novices are supported to become expert members of the group through some guidance (or scaffolding) mechanisms (see, for example, lave & wenger, 1991). within a cop, collaboration is stimulated; and the members of the community establish a group identity with common ideas about purpose and building communities for the exchange of learning objects: 25 values (implemented in so-called community policies) that are shared by the group members. in a cop, learning to become a (better) expert member is the major objective. wilson and ryder (1998) characterize learning communities as follows: 1. they have distributed control; 2. there is commitment to the generation and sharing of new knowledge; 3. learning activities are flexible and negotiated; 4. community members are autonomous; 5. there is a high level of dialogue, interaction and collaboration; and 6. there is a shared goal, problem or project that brings common focus and incentives to work together. when designing online communities, two aspects are of importance: usability and sociability. according to preece (2000, pp. 26–27), software with good usability supports rapid learning, high skill retention, low error rates and high productivity. it is consistent, controllable and predictable, making it pleasant and effective to use. paas and firssova (2003) developed a research-based framework of methods for the evaluation of usability in web-based learning environments. eight different attributes are identified: effectiveness, efficiency, learnability, memorability, flexibility, robustness, error minimization and satisfaction. sociability governs the social interactions in a community. it cannot be controlled directly, but it can be supported by carefully communicating the purpose and policies (values) of the community. preece (2000, pp. 95–96) identifies several different policies that can be addressed in a community: joining or leaving requirements; bylaws; codes of practice for communication; rules for moderation; issues of privacy and trust; practices for distinguishing professionally contributed information; rules for copyright; and democracy and free speech in the community. palloff and pratt (1999, p. 24) give a summary of the seven basic steps to follow when building a community. these steps can be considered as the sociability requirements to build successful learning object exchange communities: 1. clearly define the purpose of the group; 2. create a distinctive gathering place for the group; 3. promote effective leadership from within; 4. define norms (policies) and a clear code of conduct; 5. allow for a range of member roles; 6. allow and facilitate subgroups; and 7. allow members to resolve their own disputes. interoperability specifications for learning objects the concept of technical reuse of learning objects has its foundation in the objectoriented approach within software engineering (e.g. booch et al., 1999). in that domain, reusability is governed by certain general principles that can be applied to 26 r. koper et al. learning objects (koper, 2003). learning objects are technically reusable under the following four general conditions (the so-called non-functional requirements for exchange): 1. optimal granularity. the object must be as small as possible without loosing its internal consistency (south & monson, 2001). 2. encapsulation. the ideal learning object can stand on its own (encapsulated), have no side effects to e.g. dossiers of learners, enabling reuse without side effects (hamel & ryan-jones, 2002). 3. abstraction. reusable learning objects are abstracted towards the context of use (pedagogy, media, setting) as much as possible to allow for reuse within other pedagogical approaches, other settings and in other media (e.g., the reuse of paper resources in the web). 4. interoperability. in order to be reusable within a community, a learning object must be available in an interoperable way, i.e. it must conform to the standards that are defined within the community. a sensible approach to interoperability is to select and accept standards from the available open specifications (e.g. from ims and ieee) first and then to look at any further specifications that are needed to establish interoperability in the community. however, communities have to take care that the standards offer the functionality that they want. too many current standards are too restricted from a pedagogical perspective or too complex for the restricted use of a certain community. permanent monitoring and decisions about the standards (and versions of standards) to follow should be incorporated in any exchange community. we suggest considering the following interoperability standards for exchange communities (see table 1). there are several missing specifications in the field of binary resources (e.g., for simulations) and for most communication, search and collaboration services (although ims learning design defines an initial set of services). one has to decide within a community how to deal with the missing specifications. also one has to decide how to use the specification. most specifications allow for further customisation or have many optional elements one has to choose from. use cases for exchange one approach to capture functional requirements of the exchange community is to model so-called uml use cases (e.g., fowler, 2000). use cases are abstractions of scenarios in which the concrete behaviour of persons within a system, or using a system is described (cockburn, 2001). a use case model contains use cases, actors and relationships (armour & miller, 2001). use cases (ellipses in diagrams) are sequences of actions required of the system, which it must be able to support if it is to function properly. actors (stick figures in diagrams) are the persons or software agents that initiate the use cases, perform the use cases or benefit from the use cases. relationships (lines in diagrams) link two elements to show how these interact. derived from the theories and technologies described above, we modelled the use building communities for the exchange of learning objects: 27 case diagram represented in figure 1 to capture major functional requirements. in the next paragraphs we will explain the use case model. use cases in a learning object exchange community (uml use case diagram, all dotted lines are <<include>> dependencies) actors derived from the self-organization theory, we identify one type of actor that can have different roles. this actor is called a ‘member’ and is defined as any person who is actually or potentially creating learning objects within the domain of the exchange community. a member can be an individual or a group of individuals in any formal or informal organization. several roles for members can be specified, e.g. the producer, (re-)user, and group moderator. these types of roles search, use and add learning objects. we have drawn a specific role, called the community facilitator, who is responsible for communal aspects, like information, promotion, editing of the policies, monitoring and increase of usability. note that these roles are assigned to persons, and the same person can have multiple roles. the use case ‘change organization structure’ is responsible for the definition and allocation of roles and system rights. like any rule in the system, this process is governed by explicitly defined and collectively maintained community policies. having discussed (first level) requirements for technological interoperability in the previous section, we will now first describe the necessary (second level) actions (in the table 1. suggested use of interoperability specifications in exchange communities interoperability of: advised other alternatives non-binary resources xhtml (2003), using additional (namespaced in) schemas for formulas, images, etc. html or de facto standard formats like ms-word and rtf.html binary resources depends on data type, e.g. png, jpeg, mpeg, … de facto standards, like flash, java, gif, … resource descriptions for learning objects ieee (lom, 2003) with imslom xml binding (xml, 2003) dublin core metadata (with educational extensions) in rdf/ xml binding; or owl web ontology language (owl) resource descriptions for all other type of learning objects dublin core (dc, 2003) metadata in rdf/xml binding lom or any library format; or owl web ontology language packages ims content package (cp, 2003) zip files without predefined structure units of learning (e.g. course) ims learning design (ld, 2003) ims content package, or adl (scorm, 2003) testing ims question and test interoperability specification (qti, 2003) proprietary testing software (e.g. in lmss). learner data ims learner information package (lip, 2003) locally agreed learner dossier 28 r. koper et al. form of use cases) that enable the exchange of learning objects, followed by some basic (third level) actions that are required to establish exchange in the context of a community. use cases to realise exchange from a technological perspective find, get, edit and (dis-)aggregate. to be able to search and find a learning object, the common idea is that a search engine can search through the metadata descriptors (or full text index) of the objects. to get access to a learning object, the object itself should be available for a certain price or license. one way of getting access to an object is through an uri reference (for example merlot, see caloffa, 2003). another approach is to download a copy. users may want to adapt a learning object to the specific local context of use. in order to allow for this process, the learning object must be loaded into a suitable local or remote editing tool. this is only possible when the source of the learning object is available and when the source is in a format that available tools support. part of the editing process is to aggregate a collection of learning objects towards its aim: the building of a course. aggregation can be done in several ways: through a collection of links, through a collection of physical objects or a combination of both. typically a specification like ims content packaging, figure 1. use cases in a learning object exchange community (uml use case diagram, all dotted lines are <<include>> dependencies) building communities for the exchange of learning objects: 29 as used by ims learning design, will be used to for the aggregation of links and physical files. upload version, upload new and add metadata. one must be able to upload new or adapted learning objects. this can be done with standard available web protocols (e.g. ftp) or even with email. feedback and logging. feedback is needed to support self-organization, adequate object selection, quality control and navigation. it is important that users provide feedback about the use of a certain learning object, provide tips for other users and assess its quality. communicate and collaborate. besides feedback, other types of communication and collaboration are needed for communities to share experiences. one must be able to communicate members’ profiles, including information about the member’s role and past performance in the community. this provides a means to look at a member’s background and reputation, e.g. a feedback provider, to judge the relevance and quality of the contribution. it also provides information, e.g. an email address, for further (subgroup or private) discussion. meta-comments about the use of learning objects or the community facilities are required more generally, allowing the exchange of knowledge and experience at a more generic level (e.g., how do we reuse materials in our organization?) and supporting the establishment of a community of practice (lave & wenger, 1991). a typical approach in an online community is to organize moderated, asynchronous discussion forums connected to synchronous events like conferences, workshops, or invited addresses. tools that support collaboration between members who have some common interest and will typically establish a subgroup in the community are shared workspaces (e.g., bscw, 2003) and synchronous collaboration tools (e.g., shared whiteboard, version controlled joint editing of documents). use cases to realize exchange in a community needs assessment. the needs of the members must be assessed at a regular basis, to improve the usability and the overall functionality of the exchange system. manage and apply policies. one of the central use cases involves managing and applying policies. as the name suggests it contains three elements: keeping record of the policies in the community; providing mechanisms to apply these policies in the community; and providing policy-regulated mechanisms to change these policies. the policies include aspects as: intellectual property rights; terms of use; 30 r. koper et al. membership; joining or leaving requirements; the reward system; learning technology standards to be used in the community; quality control mechanisms; regulations for member participation (assessments, voting, peer-review); and regulations for subgroups and role changes (e.g., facilitator). to apply the policies, members must formally agree with the policies. the concrete regulations a person has to agree upon can differ depending on the user role and responsibilities. there must be a control mechanism that makes members behave according to the policies. this control mechanism can be implemented by assigning responsibilities to a member role, by establishing some social control mechanisms and/or a by technological measures, e.g., by setting user rights. there are several reasons to change policies, one of which is the monitoring of the emergent properties. when the frequency of exchange is low, exchanged learning objects have poor quality or, for example, there are systematic conflicts between participants (lack of trust), this can trigger a change in policies. changing policies is one of the major control instruments in self-organized systems. different policy aspects (quality, review) were already discussed in the theoretical part of this article. two more issues will be addressed here: the intellectual property rights and the user roles/rights. in the field of digital rights management (drm), currently two models are in force. from the recording and movie industries, the model of ‘enforced drm’ is promoted. the model of ‘attributed drm’ is promoted by educational organizations worldwide and is supported by the ieee (dr, 2003). in this model, which supports current practices, usage of material is permitted either by payment or by simply attributing through layers that are added to the material (metadata or digital rights expression language—drel). from an organizational point of view it is advisable to put some people in charge of tracking the information on the fast evolving fields of drm and the implementation of drm-policies. creating a greater ‘digital rights’ awareness amongst colleagues should also part of their job (see also cen/isss, 2003). another mechanism to control the monitoring of policies in an electronic environment is by setting user roles and rights. when a member’s role is changing, the assigned user rights can be set automatically in the system. these rights pertain to access to certain functions and data, and the ability to read, edit and upload. for instance, setting the rights for a community facilitator could pertain to editing the policies. given the fact that policies are the major control mechanisms for selforganization and that user rights can be seen as an implementation of the policies in electronic environments, it can be argued that the major control mechanism in an learning object exchange community is the definition and setting of user roles and connected rights. information, promotion and training. as stated in the section on background theories, the fact that a good exchange system is available does not guarantee its usage. for an organization the possibilities of sharing and reuse of resources should first and foremost lead to cost-reduction. such benefits are not always obvious to teachers who building communities for the exchange of learning objects: 31 have to do the job. if this change in (working) attitude is not managed and guided adequately, organizations will face the danger of reluctant teachers who will do anything to undermine these changes (mcnaught, 2003). this is a difficult motivation issue, as explained by social exchange theory. given this problem, it is necessary to give special attention to activities like: 1. attracting new members, mainly by communicating the purpose and policies (e.g., the reward system); 2. promoting use within the existing membership (e.g., promoting use of particular learning objects) and providing facilities to advertise personal productions; 3. providing training facilities for new and existing members and overcoming barriers at the personal level, like the ‘not-invented-here’ syndrome or the ‘free-rider’ effect. improving collaboration between actors is one of the most important challenges in establishing a learning object exchange community. to make a start with effective collaboration, a number of basic conditions need to be taken care of, including a common organizational structure, common tools for employers and common goals (wetterling & collis, 2003). a number of issues need to be addressed, recognized and guided alongside these basic conditions. for example, will a teacher be able to preserve his own teaching style, or does he need to conform to that of ‘the group’? will teachers be given enough time, support and guidance in order to make the (work) attitude change? it will be necessary for the organization to point out to the teachers that the challenge of reuse is exciting and not stressful. for some, this will require a change of attitude. with respect to the not-invented-here-syndrome, people still tend to think that something made elsewhere cannot be as good as that which is homemade. it may prove to be helpful for authors/developers to explain what they have made and how they use it (e.g., source, 2003). ‘free-riding’ (olson, 1965) or ‘lurking’ occurs when people never contribute or cease to contribute to a community. in most of these cases, knowledge is not acknowledged as a public good that is the responsibility of a group, but becomes a good that can be taken and used. the community thus suffers from a social dilemma. members of a community need to be motivated to contribute. as social exchange theory states, people make comparisons between two alternatives and select the alternative most beneficial for themselves, ensuring the lowest costs (thibaut & kelly, 1959). empirical evidence has proved that when group size increases, the proportion of volunteers that contribute decreases (fischer & ackerman, 1998). while this might be true for communities in environments other than the web, there is also empirical evidence that ‘virtual communities’ behave differently in this respect (hemetsberger, 2003). the linux and apache open-source communities are leading examples of communities that are successful in enabling a relatively large number of members to share knowledge and contribute to a growing array of software-products. these communities also substantiate research findings that communities should always gather around a common interest or goal (kozinets, 1999). 32 r. koper et al. manage usability. in the community it is necessary to set up mechanisms to monitor the usability of the system. a needs analysis can identify usability problems, but other approaches are also possible, such as a facility that counts errors or frequency of use of certain facilities. actions to increase usability almost always involve some (re-)programming activities and (re-)design of the user-interface. it is important to have capacity available to create the improvements needed. conclusion in this article we presented a framework that summarizes the functional and nonfunctional requirements for the exchange of learning objects in a community. the framework is based on self-organization theory, social-exchange theory, community building models, and interoperability specifications for learning objects. the nonfunctional requirements are presented in table 1, and the functional requirements are captured in a use case represented in figure 1. we propose recommendations on three consecutive levels: to use open standards and specifications to code, aggregate and identify the different learning objects; to build upon certain use cases to enable the actual exchange of the learning objects at a technological level; and to establish a community of practice based on self-organization principles to create an effective environment for exchange. above and beyond these recommendations, the most critical aspect in self-organized systems is to establish interactions between participants that are governed according to some rules, or policies as they are called in this article. these rules are comparable with the rules of games like chess, laws in a society or by-laws in an association. the article only provides some suggestions to establish these rules or policies, but one line of future research should focus on the optimization of policies to establish successful exchange systems. for instance, an experimental comparison of certain reward policies in a community might provide some useful insights to this optimization problem. working in communities might also be intrinsically rewarding. submitting a learning object, then ‘experiencing’ peer reviews and working further towards a final version of the object might work in ways similar to the reviewing process that guides the writing of journal articles. in domains like economics and social psychology, successful reward systems have appeared to be successfully applied, but we still have to search for the adequate combinations of intrinsic and extrinsic rewards that will establish a successful exchange community in the educational arena. besides the reward system the specific organization, e.g. the roles and responsibilities, can be the object of a second line of more detailed research. what type of organizational structure works best and under what conditions? for instance, there is a difference between the establishment of an exchange system within an existing organization (e.g., university, department, euniversity, company) based on the idea that only persons that work for the organisation will share there resources, as opposed to an exchange system with an open admission that can be used by everybody who wants to and is able to use it. as a third line of future research we envisage developing an evaluation instrument based on the requirements framework provided in this article. with such an instrument building communities for the exchange of learning objects: 33 we will be able to evaluate and compare concretely implemented initiatives and provide suggestions for their improvement. such an instrument would establish a minimal set of requirements, and weigh the different requirements for learning object exchange environments. and last but not least, it would provide a starting point for the development of new exchange communities. exchanging learning objects within the context of communities seems a potential step towards an affordable e-learning future. because of the potential quality and cost impact upon education, it is extremely important to continue further research, development and implementation this area. this research should combine a further definition of necessary standards, the development of sound, specific models and policies for exchange, and the building of facilitative tools and infrastructures that enable the exchange of learning objects in communities. references anderson, w., challagalla, g. & mcfarland, r. (1999) anatomy of exchange, journal of marketing theory and practice fall, 8–19. armour, f. & miller, g. (2001) advanced use case modeling (upper saddle river, nj, addisonwesley). bonabeau, e., dorigo, m. & theraulaz, g. (1999) introduction, in: swarm intelligence (oxford, oxford university press). booch, g., rumbaugh, j. & jacobson, i. (1999) the unified modeling language user guide (reading, addison-wesley). bscw (2003) basic support for cooperative work. available online: http://orgwis.gmd.de/projects/ bscw/ (accessed 25 november 2003). cafolla, r. (2003) project merlot: bringing peer review to web-based educational resources available online: http://www.aace.org/conf/site/pt3/paper_3008_721.pdf (accessed 25 november 2003). campbell, l. (2003) engaging with the learning object economy, in: a. littlejohn (ed.) reusing online resources: a sustainable approach to e-learning (london, kogan page). cen/isss (2003) european committee for standardization. available online: http://www.cenorm.be/ cenorm/index.htm (accessed 25 november 2003). cockburn, a. (2001) writing effective use cases (boston, ma, addison-wesley). constant, d., kiesler, s. & sproull, l. (1994) what is mine is ours, or is it?, information systems research, 5(4), 400–422. cp (2003) ims content packaging. available online: http://www.imsglobal.org/content/ packaging/index.cfm (accessed 25 november 2003). dc (2003) dublin core metadata initiative. available online: http://dublincore.org/ (accessed 25 november 2003). deci, e. & ryan, r. (1985) intrinsic motivation and self-determination in human behavior (new york, plenum press). downes, s. (2003) design and reusability of learning objects in an academic context: a new economy of education?, usdla journal, 17(1). available online: http://www.usdla.org/html/ journal/jan03_issue/article01.html (accessed 25 november 2003). dr (2003) ieee learning technology standardization committee wg4 digital rights expression language. available online: http://ltsc.ieee.org/wg4/drel_relevant_standards_survey-v4.doc (accessed 25 november 2003). ferber, j. (1999) multi-agent systems (reading, addison-wesley). 34 r. koper et al. fischer, r. s. & ackerman (1998) the effects of recognition and group need on volunteering: a social norm perspective, journal of consumer research, 25(december), 262–275. fowler, m. (2000) uml distilled, 2nd edn (upper saddle river, nj, addison-wesley). fukuyama, f. (1995) trust (new york, simon & schuster). gilbert, n. (1995) emergence in social simulation, in: n. gilbert & r. conte (eds) autonomous societies: the computer simulation of social life (london, ucl press), 144–156. grönroos, c. (1999) relationship marketing: challenges for the organization, journal of business research, 46, 327–335. hall, h. (2001) social exchanges for knowledge exchange paper presented at managing knowledge: conversations and critiques, university of leicester management centre, 10–11 april 2001. hamel, c. & ryan-jones, d. (2002) designing instruction with larning objects, international journal of educational technology, 3(1). available online: http://www.ao.uiuc.edu/ijet/v3n1/ hamel/index.html (accessed 25 november 2003). hemetsberger, a. (2003) fostering cooperation on the internet: social exchange processes in innovative virtual consumer communities. available online: http://ltsc.ieee.org/wg12/files/ lom_1484_12_1_v1_final_draft.pdf (accessed 25 november 2003). ieee ltsc (2002) ieee learning technology standardization committee draft standard for learning object metadata, version 6.4. available online: http://ltsc.ieee.org/doc/wg12/lom_wd6_4.pdf (accessed 25 november 2003). jarvenpaa, s. & sensiper, s. (1998) the role of tacit knowledge in group innovation, california management review, 40(3), 112–132. komorita, s. & parks, c. (1995) interpersonal relations: mixed-motive interaction, annual review of psychology, 46, 183–207. koper, e. (2003) combining reusable learning resources and services with pedagogical purposeful uols, in: a. littlejohn (ed.) reusing online resources: a sustainable approach to e-learning (london, kogan page), 46–59. kozinets, r. (1999) e-tribalized marketing? the strategic implications of virtual communities on consumption, european management journal, 17(3), 252–264. lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation (cambridge, cambridge university press). ld (2003) ims learning design specification. available online: http://www.imsglobal.org/ learningdesign/index.cfm (accessed 25 november 2003). lip (2003) ims learner information package specification. available online: http://www.imsglobal.org/profiles/index.cfm (accessed 25 november 2003). lom (2003) ieee learning technology standardization committee wg12 learning object metadata. available online: http://ltsc.ieee.org/wg12/index.html (accessed 25 november 2003). maturana, h. & varela, f. (1992) the tree of knowledge: the biological roots of human understanding, revised edition (boston, ma, shambhala/new science press). mclure wasko, m. & faraj, s. (2000) it is what one does: why people participate and help others in electronic communities of practice, journal of strategic information systems, 9, 155– 173. mcnaught, c. (2003) identifying the complexity of factors in the sharing and reuse of resources, in: a. littlejohn (ed.) reusing online resources: a sustainable approach to e-learning (london, kogan page). merlot (2003) peer review of merlot learning materials. available online: http://taste.merlot.org/ projects/peer_review/ (accessed 25 november 2003). olsen, m. (1965) the logic of collective action (cambridge, ma, harvard university press). paas, f. & firssova, o. (2003) usability evaluation of integrated e-learning, in: w. jochems, j. van merriënboer & e. koper (eds) integrated elearning (london, routledgefalmer). palloff, r. & pratt, k. (1999) building learning communities in cyberspace, (san francisco, ca, jossey-bass). preece, j. (2000) online communities: designing usability, supporting sociability (chichester, wiley). building communities for the exchange of learning objects: 35 qti (2003) ims question & test interoperability specification. available online: http://www.imsglobal.org/question/index.cfm (accessed 25 november 2003). rheingold, h. (2000) the virtual community: homesteading on the electronic frontier, revised edition (cambridge, ma, mit press). sawyer, s., eschenfelder, k. & hexkman, r. (2000) knowledge markets: cooperation among distributed technical specialists, in: t. srikantaiah & m. koenig (eds) knowledge management for the information professional (medford, nj, information today). scorm (2003) advanced distributed learning. sharable content object reference model. available online: http://www.adlnet.org/index.cfm?fuseaction=scormabt (accessed 25 november 2003). shaffer, c. & anundsen, k. (1993) creating community anywhere (new york, perigee books). source (2003) source: software use, re-use & customisation in education. available online: http://www.source.ac.uk/ (accessed 25 november 2003). south, j. & monson, d. (2001) a university wide system for creating, capturing and delivering learning objects. available online: http://reusability.org/read/chapters/south.doc (accessed 25 november 2003). thibaut, j. & kelly, h. (1959) the social psychology of groups (new york, wiley). von krogh, g. (1998) care in knowledge creation, california management review, 40(3), 13–153. waldrop, m. (1992) complexity: the emerging science at the edge of chaos (new york, simon & schuster). wetterling, j. & collis, b. (2003) sharing and re-use of learning resources across a trans-national network, in: a. littlejohn (ed.) reusing online resources: a sustainable approach to e-learning (london, kogan page). wilson, b. & ryder, m. (1998) distributed learning communities: an alternative to designed instructional systems, paper submitted to educational technology research and development. xhtml (2003) extensible hypertext markup language. available online: http://www.w3c.org/ markup/ (accessed 25 november 2003). xml (2003) extensible markup language. available online: http://www.w3c.org/xml/ (accessed 25 november 2003). table 1. suggested use of interoperability specifications in exchange communities building communities for the exchange of learning objects: theoretical foundations and requirements rob koper*, kees pannekeet, maaike hendriks & hans hummel 1. personal need, or anticipated reciprocity: learner has a pre-existing expectation that he will receive actionable and useful information in return. 2. reputation: learner feels he can improve his visibility and influence to others in the community, e.g. leading to more work or status in the future. 3. altruism, or the perception of efficacy of the community in sharing knowledge as a ‘public good’, especially when contributions are seen as important, relevant and related to outcomes. 4. (tangible) reward: learners negotiate to get some kind of more tangible asset (financial reward, bond, etc.) in return. common goals and values communal relationships 1. they have distributed control; 2. there is commitment to the generation and sharing of new knowledge; 3. learning activities are flexible and negotiated; 4. community members are autonomous; 5. there is a high level of dialogue, interaction and collaboration; and 6. there is a shared goal, problem or project that brings common focus and incentives to work together. 1. clearly define the purpose of the group; 2. create a distinctive gathering place for the group; 3. promote effective leadership from within; 4. define norms (policies) and a clear code of conduct; 5. allow for a range of member roles; 6. allow and facilitate subgroups; and 7. allow members to resolve their own disputes. 1. optimal granularity. the object must be as small as possible without loosing its internal consistency (south & monson, 2001). 2. encapsulation. the ideal learning object can stand on its own (encapsulated), have no side effects to e.g. dossiers of learners, enabling reuse without side effects (hamel & ryan-jones, 2002). 3. abstraction. reusable learning objects are abstracted towards the context of use (pedagogy, media, setting) as much as possible to allow for reuse within other pedagogical approaches, other settings and in other media (e.g., the reuse of p... 4. interoperability. in order to be reusable within a community, a learning object must be available in an interoperable way, i.e. it must conform to the standards that are defined within the community. find, get, edit and (dis-)aggregate upload version, upload new and add metadata feedback and logging communicate and collaborate needs assessment manage and apply policies information, promotion and training 1. attracting new members, mainly by communicating the purpose and policies (e.g., the reward system); 2. promoting use within the existing membership (e.g., promoting use of particular learning objects) and providing facilities to advertise personal productions; 3. providing training facilities for new and existing members and overcoming barriers at the personal level, like the ‘not-invented-here’ syndrome or the ‘free-rider’ effect. manage usability figure 1. use cases in a learning object exchange community (uml use case diagram, all dotted lines are <<include>> dependencies) developing cross-cultural communicative competence via computer-assisted language learning: the case of pre-service esl/efl teachers abdelilah salim sehlaoui department of foreign languages, emporia state university email: sehlaoua@emporia.edu based on a qualitative research project, this article presents a view on the use of computer technology to develop a critical cross-cultural communicative competence in english as a second language (esl) / english as a foreign language (efl) for preservice teachers. the article includes a brief critical theoretical framework, some classroom pedagogical implications, and a data-based discussion of pre-service teachers' views. these views included: (1) critical views and an awareness of cultural power relations in computer-assisted language learning (call), (2) uncritical views and a lack of awareness of cultural power relations in call, and (3) uses of metaphors in call. the powerful contribution of call can be found in its potential for providing ways to connect people and build communities, for offering opportunities for crosscultural communicative competence to be developed and used, and for improving processes of democratization via computer-mediated communication. however, a sociocultural criticism revealed that this powerful tool, like any other media, is non-neutral because it can serve to reinforce further the hegemonic aspects of education, that is, the dominant culture will be strengthened and protected via computer technology. computerbased technologies and software are increasingly incorporated into the curricula of esliefl teacher education programmes. however, this integration is often done in ways that seem to leave unquestioned the potential cultural and hegemonic ramifications of such technology. hence there is a need for a more critical technological competence. introduction a review of the relevant literature strongly suggests that many, if not most, english as a second or foreign language (esl/efl) teacher education programmes typically utilize 53 abdelilah salim sehlaoui developing cross-cultural communicative competence via computer-assisted language learning concepts of culture and strategies for teaching and learning that are open to some cogent, but constructive, critiques from the perspective of critical educational theory. in addition, according to wise (1995), throughout the last ten years the national council for accreditation of teacher education has required that colleges of teacher education must integrate technology into their curricula. in fact, esl/efl teacher education programmes are increasingly incorporating computer-based technologies and software into their curricula. however, this integration is often done in ways that seem to leave unquestioned the potential cultural and hegemonic ramifications of such technology (see, for example, bowers, 1988; jones, 1995; morton, 1996; murray, 1997; sehlaoui, 1999). the purpose of this article is to argue for the need for a critical computer literacy in the process of developing cross-cultural communicative competence in english as a second/ foreign language (esl/efl) pre-service teachers. to achieve this purpose, this article presents a brief theoretical framework, a data-based discussion of esl/efl pre-service teachers' views and some classroom pedagogical implications. background information the argument developed here is supported by the findings of a dissertation research project. the primary purpose of this research project was to describe and analyse current practices in an ma tesol (teachers of english to speakers of other languages) programme in terms of the programme's efforts to develop cross-cultural communicative-1 competence in students enrolled in the programme. the term cross-cultural communicative competence will be referred to in this article as cccc. the way in which cccc is defined and developed within the programme was thus an important, but broad domain of enquiry. within this overarching domain, the purpose of the research focused on the conceptualizations of culture utilized within the programme, the professional and cultural identity formation processes that appear to be occurring, the characterization of students' analyses of power relations and their own position within these relations, and the ways in which computer-based technologies are being increasingly incorporated as a part of these processes (sehlaoui, 1999). this article focuses on the last component of this research project, the role critical computer literacy plays in the process of developing cccc in esl/efl pre-service teachers. based on students' perspectives, beliefs and attitudes as described in sehlaoui (1999) data-based pedagogical suggestions will be made. methodology a qualitative research framework was chosen for this study. in order to examine how the development of cccc was being approached in the ma tesol programme at a state university in the usa, data were obtained by: (1) classroom observations in the three core courses for which the instructor's and students' permission had been obtained, (2) questionnaires given to graduate students who chose to participate in the study, (3) analysis of programme documents and materials such as syllabuses, software programmes used, etc., and (4) semi-structured interviews with ma tesol students who gave their consent. a domain/thematic analysis, based on the initial research questions, was performed by generating categories and then themes from the answers given by the responders, the classroom observations field notes data, and by examination of the written documents (ely, anzul, friedman, gamer and steinmetz, 1991; miles and huberman, 1994). 54 alt-j volume 9 number 3 participants the total number of participants in this study was twenty-seven: nine males and eighteen females. their age range was between twenty-three and forty-five. the participants were from ten different countries: china, taiwan, usa, belize, japan, korea, argentina, thailand, saudi arabia and jordan. the majority of participants (twenty-three) in this study came from a middle and upper-middle social class and only two from a low and lower-middle social class. setting the natural context in which the present research project was carried out is the english department at a state university in the usa. the target programme was the ma tesol programme, which focused on esl/efl pre-service teachers and esl/efl teacher educators' classroom learning/teaching practice. a critical theoretical framework the theoretical framework which guided the analyses and interpretations of data is grounded in critical educational theories that have emerged from the field of multicultural education and have often provided a basis for change in teaching and learning strategies relating to cultural dynamics (aronowitz and giroux, 1993; foley, 1990; giroux, 1981; sleeter and grant, 1999; shannon, 1992; and tidwell, 1993). in many ways, the proposed research was an attempt to draw together elements from the areas of multicultural education and tesol with an eye toward benefiting both of these fields of scholarship and practice. the special focus of this study on computer technology in the curriculum and' its implications for socio-cultural dynamics is a recognition of technology's increasing, and often unquestioned, role in socio-cultural processes. until quite recently, computer-assisted language learning (call) was a topic of relevance mostly to those with a highly specialized interest in that area. recently, though, computers have become so widespread in schools and homes of middleand upper-class families (lockard, abrams and many, 1997; poole, 1997) and their uses have expanded so dramatically that the majority of language teachers must now begin to think about the implications of computers for language learning and teaching (warschauer, 1996). from a critical perspective, it is important to recognize that computer use is non-neutral, in the most basic sense, because not all students in a society have equal access to computers (lockard et al., 1997). this will inevitably create a new category of 'haves' and 'have nots', usually along pre-existing lines of inequalities. according to mcgrath, thurston, mclellan, stone and tischhauser (1992: 468), inequities in computer access have two implications: 'first, the students affected will be educationally handicapped by the lack of powerful learning t o o l s . . . second, these students may find themselves at a disadvantage in their preparation for their future academic careers or jobs'. in a more globally cultural sense, computer programs, according to bowers (1988), carry with them the inherent mechanical mode of thinking that underlies the western approach to science and technology. while western languages, especially english, have developed in a complex historical relationship to a positivist world-view, the form and structure (and philosophical ethos) of many languages exist at odds with, or at least in a different relationship with, more mechanistic structures and forms. 55 abdelilah salim sehlaoui developing cross-cuttural communicative competence via computer-assisted language learning pennington (1996) points out that while computer use in language learning can promote positive learning experiences, it can also encourage a form of anti-social behaviour. like bowers and pennington, jones (1995) discusses the kind of culture that is being developed in what he calls 'cybersociety'. in other words, we need to be aware of the type of reality and world-view that computer-mediated communication (cmc) can bring with it. these authors point out that we must be aware that instead of developing students' cccc via email and internet, this computer use may be developing a new elite culture and fostering students' stereotypes and biases, especially when the focus is on cultural products rather than process. computer technology can hinder cross-cultural communication by exposing students and teachers to cultural material that can develop or strengthen stereotypes and prejudices rather than reduce or eliminate them. educators' unawareness of these forms of non-neutrality can serve to reinforce further the hegemonic aspects of education, that is to say, the dominant culture will be strengthened and protected via computer technology, since it is the dominant social classes and their cultural forms that will both shape and benefit most from this technology. in other words, structural inequalities will be perpetuated through both of these aspects. however, in spite of this sociocultural criticism, computer technology has offered the field of education a powerful tool for learning and teaching, and does hold some potential for the furtherance of processes of democratization. the powerful contribution of call can be found in its potential for providing ways to connect people and build communities, in offering opportunities for cccc to be developed and used, and for improving processes of democratization via cmc. however, a sociocultural criticism revealed that this powerful tool, like any other media, is nonneutral. the following section presents a data-based sociocultural analysis as it emerged from the research project by sehlaoui (1999). a data-based sociocultural analysis: computer technology, culture and power computer technology, culture and power emerged as a separate category in the research project. this category addressed the following research question: how is computer technology discussed and conceptualized among students as part of their socialization process in the programme? how is this technology integrated in the tesol programme? the data collected by means of observation and document analysis did not contain any reference to call. this means that during all observation sessions of three core courses in the tesol programme, call was not among the topics discussed or studied (apart from a few instances when one of the instructors mentioned to the students that he had sent an email message with some instructions and class announcements). furthermore, the textbooks used in all three courses did not integrate call into their content (see, for example, celce-maurcia, 1991; gebhard, 1996; ramirez, 1995; samovar and porter, 1994; ur, 1996). these observations about call in this particular programme are so significant because they corroborate the findings in the other two resources, namely, the interview and questionnaire data. three main themes emerged under this category in interview and questionnaire data: (1) critical views and an awareness of cultural power relations in call, (2) uncritical views and a lack of awareness of cultural power relations in call, and (3) uses of metaphors in 56 alt-j volume 9 number 3 call. these themes emerged in the data collected during the interviews with the participants, as well as their responses to the questionnaire. i. critical views and awareness of cultural power relations in call the following statement, by a student from thailand, illustrates the critical views held by some informants: now most of internet is used by rich people. people in schools and even now some colleges do not have internet. internet can change a lot. it's creating the change. it's influencing our culture. you know when i was in my country i didn't know about it, but i know that it has good and bad sides. i mean good in transferring knowledge and bad because it encourages nudism in my country. this echoes the attribute of 'double-edged sword' used by some participants in the study to describe the cultural changes in the world (see sehlaoui, 1999). the same informant added: 'ok, technology is one of the causes of this change. as a means of communication it helps a lot in those changes.' relating this to the hegemony of the english language and culture, another informant raised the question of whether all the people using internet should know english. he explained: the idea would be that they try to do this so that everything on the internet should be in english so that all the people should know english or should learn english. i would say that the point of arabizing or electronic arabizing is difficult because it is not desirable for the dominant culture . . . electronic arabizing would be ok and accurate. it's necessary to know the english language; therefore, the effort of arabizing is not encouraged because they want people to learn the american culture. if there is no translation, people will need to learn the english language and if they learn the language they must learn the culture too. this is another form of hegemony that supports the dominant culture, which has been described as linguistic imperialism (phillipson, 1992). such cultural criticism as it relates to computer technology should become part of the tesol curriculum. in another country, the use of the internet was explained by a participant who said that: the system, as far as i know, is not working yet on a wide range. it's something closed for university staff or something like that. the public cannot have access for using the internet and if they can use it, it's expensive for them (people who cannot afford it). and as far as i know there are regulations for displaying material there. the information will be filtered. it will be censored. things like pornography. i think it's hard to control political information, though. it's not something like sex. the system itself will be programmed not to allow something like this to get through. from the usa a student said that: the dominant culture has more access to it. like i said the middle-upper income class. those lower-income/poverty class don't have access and i have seen that with my own eyes . . . yeah. this closes the door to them, they can't improve socially and economically. i see that in the future, this is gonna be more and more important in our life, but the gap is gonna get larger and larger, especially when they don't teach technology and don't make it available. 57 abdelilah salim sehlaoui developing cross-cultural communicative competence via computer-assisted language learning this supports the implications made by mcgrath et al. (1992) about inequities in computer access. two different students (who apparently belong to two different socio-economic classes) explained their experiences with computer technology as follows. one saudi student said: for example, i was browsing the internet last night looking for some software for qualitative research and i found some of them that can code video-taped material and everything. i think this kind of software helps the researcher in doing all his/her research, for example, in months or weeks, you can do that in 90% less time which is good. in contrast, a us student explained that when the university is closed down that has been a problem for me because i don't have access to it at home. it's very frustrating. that was the one thing i was concerned about. i was still looking for a job and i was getting nearer and nearer to the end of the semester and i was communicating with a lot of people through email and i was 'oh my gosh! what am i gonna do?' and a friend offered that i could use her email but i have to drive twenty miles to get to her house. 2. uncritical views and a lack of awareness of cultural power relations in call although the above participant showed an awareness of the issue of equity and accessibility of computers, she was not that critical about computer use in the classroom and in relation to cccc and its cultural dynamics. the following statement illustrates this uncritical view: the students can use it to learn all about other cultures on the internet, type the name of the country and . . . i have not thought a lot about that. i would teach culture through internet like if that was a course based on culture or maybe if i give an assignment on culture i would recommend the use of internet maybe there might be a specific site where they can explore other cultures. i'm not really sure how i would use it. this statement also shows a lack of confidence in how to use computers in developing cccc in efl/esl contexts. this can be ascribed, partially, to the lack of training that esl/efl teachers have in this area (see, for example, morton, 1996; warschauer, 1996). the following statement illustrates this further, when the informants were asked to describe their experiences with call: not much. i have very little experience with computer technology. my abilities, that i developed on my own, include word-processing, some understanding of email and internet usage. my knowledge is very basic, though i would like to know more. another informant explained '[he is] learning a lot from computer technology on [his] own'. this informant's statement also shows that educational technology is still a weak component for most esl/efl educators, since most students said they were learning how to use computers on their own. this fact was confirmed by field note data and document analysis data as mentioned before. this triangulation of data analysis also helped in tracing the conceptualizations of culture used in the participants' responses to the use of call in developing cccc. 58 alt-} volume 9 number 3 statements about computer use in relation to cultural learning and teaching were also based on the conceptualization of culture as country, too. within this category the themes of computer neutrality and culture as country or artefact occurred. culture is equated to country as is shown in the following statements about call. one participant stated: 'it's wonderful you can communicate across the boundaryless world, not knowing or caring if the person is racially, ethnically, culturally the same or different from you. it gives students a lot of practice in writing and understanding.' to illustrate this general perception of the 'culture as country' and 'the neutrality of computers', one of the participants stated: well, for me, it provides people with a way to communicate with people in other countries or other cultures. beyond that, i am not really sure. i do think that it's easier and more comfortable talking/communicating with people who are different on the computer because you're not facing them, the differences can be ignored. this is what jones (1995) and bowers (1988) addressed in their cultural criticism of educational computing. the same uncritical views and perceptions were evidenced by the data collected during a pilot study that anticipated this dissertation research project. in their responses to the following statement, 'computer technology can be used to develop cccc because it is neutral', almost all participants (80 per cent) agreed that computer technology is neutral in the sense that has been discussed before in this article. only 5 per cent disagreed with the statement above, whereas 15 per cent had no opinion. 3. uses of metaphor in call another recurring theme in this study was that of metaphors used by participants to describe the role of computer technology in developing cccc. these metaphors included: 'the computer's ability to break down cultural walls', 'it [the computer] brings the world into a small global village', 'it [the computer] brings your culture to the tips of your finders', 'the internet acts as a window to information about the cultures of the languages studied'. the metaphors that were used or adopted by some informants while talking about technology, during interviews, included: 'the world becoming a small village', 'i think technology makes the world smaller as they call it "a global village"', 'computer and language are tools'. in support of murray's (1997) cultural criticism of the uncritical use of metaphor in call, one of the participants said: i believe that computers and the internet are changing our world and people from all around the global village are becoming closer. so intercultural communication is the field of the future. this statement is an example of the uncritical acceptance of 'the village' metaphor, that is, a village comprised only of those who have access to computers (murray, 1997). according to murray (1997), as language educators, we need to unravel the metaphors used to talk about computer technology because the metaphors used constantly in our everyday language shape our understanding and view of the world. she points out that many of the terms and metaphors used to describe the new technology, such as the ones used by the participants in the study reported in this article, lead us to ignore the social context in which the technology is introduced. murray states that 'what these metaphors hide is the historical fact that the introduction of all new technologies is not socially or morally neutral. it takes place in a social and cultural context that helps determine how it 59 abdelitah salim sewaoui developing cross-cultural communicative competence via computer-assisted language learning will be used' (murray, 1997: 4). one prevailing metaphor, discussed by murray, is that of the 'information superhighway'. this metaphor has become so common and so widespread that many people are fascinated by the notion of unlimited access to information and communications. however, this metaphor also invokes ethical issues in the form of questions such as: "who will build the highway? who will pay for its construction? who will have access to it? will it be a toll road or a freeway? will travel be restricted, controlled? who will police it?' (murray, 1997:2). beyond the language of critique what can be concluded from the above critiques is that, as people and as educators, we must ensure that we do not see a particular metaphor as describing the way all human beings think, but rather as determining how we view and use the technology. we must also be able to see power struggle as a struggle over meaning and representation, that is, who defines and assigns meaning. we must apply our knowledge and understanding about multiple intelligences and learning styles as much to decisions we make about computer use in our classrooms as we do to decisions about group work or visual aids. besides, we need to reflect on other metaphors used to describe the new technology, to deconstruct the metaphors for ourselves and ensure that our pedagogical decisions are not based on flawed or insidious metaphors. this means that the questions of 'how, where, why, and by whom' this computer technology is used are all legitimate to ask because, as noted by riel (1990), 'new tools alone do not create educational change. the power is not in the tool but in the community that can be brought together and the collective vision that they share for redefining classroom learning.' from a critical perspective on the data, it can be seen that the prevailing tendency among esl/efl pre-service teachers has often been to deal with computer technology as a neutral tool and to ignore its role in the socio-economic, and political dynamics of culture. the cultural strategy of appropriation is very useful in redefining and redirecting the use of computer technology for counter-hegemonic purposes (tidwell, 1993). data analysis showed that the predominant tendency among participants in the tesol programme was to reinforce further the hegemonic aspects of education via call. hence, the suggestions for improving the programme and the need for integrating a critical perspective (e.g. bowers, 1988; jones, 1995; morton, 1996; murray, 1997; pennington, 1996; and wise, 1995) into the use of call in tesol teacher education programmes. thus, esl/efl teachers need to be aware of the reality that computer technology brings with it. otherwise, instead of developing cccc in the learner via email and internet, this mode of computer use may foster students' stereotypes and biases, especially when the focus is on cultural products rather than processes, and enhance existing disparities in terms of class, ethnic and gender privileges. the data reported in this article reinforce the view that most of the students feel that computer technology is neutral without a critical consideration of its role in the socio-cultural dynamics of society. the following suggested recommendations and classroom implications aim at improving the quality of call in the tesol teacher education programmes in their attempt to develop pre-service teachers' cccc. 60 aa-j volume 9 number 3 recommendations and pedagogical implications to provide esl/efl learners with quality education on/via computers, computer systems 'should be viewed as structured learning environments with complex and comprehensive capabilities to access and manipulate information' (morton, 1996: 417). in addition to providing accessibility and availability of updated hardware and software equipment, the role of a supportive computer-based environment is to provide technical training and scaffolding for both esl/efl learners and esl/efl teachers to develop their critical and effective technological competence. the tesol programme does not yet provide such technical training for all students, as was recognized by all participants in the study. as one of them remarked: 'like i said i haven't had experience with technology in this program. i think there is a lot to be explored yet.' in fact, call courses do not figure on the list of core courses of this particular ma tesol programme or on the list of elective courses. this is why all participants (100 per cent) expressed their dissatisfaction with the way computer technology is being used in their programme. to develop effective critical technological competence, tesol teacher education programmes need to move beyond a view that embraces computer technology without questioning its non-neutrality and without understanding its role in the socio-cultural processes. they also need to move beyond a view of computers as add-ons which also encourage traditional instructional strategies and exclude computer-based learning and teaching from the curriculum (morton, 1996). one of the participants said: "they have to use pood technology, because i cannot imagine a program like that without having some introductory courses to call, for example.' however, simply adding a computer or a course to a classroom or programme in a school has not ensured that the teachers use the technology effectively for the purpose of teaching and learning (okolo, bahr and rieth, 1993). computer projects and classroom activities that democratize the educational process and help prospective teachers develop their cccc are highly recommended. for example, computer use in on-line communication can result in community building, in which individuals collectively solve problems and provide support for each other in a democratic and egalitarian manner (see, for example, bruce, peyton and batson, 1993; harris, 1993; rice-lively, 1994; and schrum, 1995). this type of communication, according to warschauer (1996), can be asynchronous through tools such as email, which allow each participant to compose messages at their own time and pace, or synchronous (real time), using programmes such as moos (multi-user object-oriented) on-line communication. it also allows not only one-to-one communication, but also one-to-many, allowing a teacher or student to share a message with a small group, the whole class, a partner class or an international discussion list of hundreds or thousands of people. computer-mediated communication (cmc) allows users to share not only brief messages, but lengthy documents as well. using the web, students can search through millions of files around the world within minutes to locate and access authentic materials and cultural information. virtual connections by warschauer (1995) offers esl/efl teachers and learners a rich resource book in this regard. the 'new ways in tesol' (boswood, 1997) series provides an excellent source of classroom activities on how to use the computer in the classroom for esl/efl learners/teachers. other activity and strategy resources for 61 abdelibh salim sehlaoui developing cross-cuftural communicative competence via computer-assisted language learning integrating call in the development of cccc can be found in heinich, molenda, russell and smaldino (1999), lockard et al. (1997), morrison, lowther and demeulle (1999), and poole (1997). however, esl/efl educators need to be cautious as to the selection of call software. warschauer (1996) points out several major problems that have surfaced in the use of call software in language teaching programmes: • first, there is the question of quality of available programs. while some teachers themselves can develop their own multimedia programs using authoring software such as hypercard (for macintosh) or toolbook (for pc), most classroom teachers lack the training or the time to make even simple programs. this has left the field to commercial software developers, who often do not base their programs on sound pedagogical principles, let alone consider critical emancipatory objectives related to call and its socio-cultural dynamics, as discussed in this article. • secondly, the cost for developing quality programs can price them out of the market of most english teaching programmes. • thirdly, some of today's computer programs are not yet intelligent enough to be truly interactive. for example, one particular program used in this area {dustiri) should ideally be able to understand a user's spoken input and evaluate it, not just for correctness but for appropriateness as well. finally it should be emphasized that efl/esl teachers should be educated to become not only critical intellectuals but transformative intellectuals as well (giroux, 1981, 1992; mclaren, 1998). in other words, they should base their classroom practice, in developing cccc via call in students, on a critical view of culture that starts where the learners are and focuses on unequal relations of power and means to address inequalities. besides, they should not only know 'what and how to teach' but also 'why' they teach 'what' they teach 'the way' they teach it (edge, 1996; crookes, 1997). in this way they will actively seek to empower themselves and their students to reconstruct society so that it better serves the needs and interests of all groups of people. this is an especially cogent framework in this historical period known as the information age. references aronowitz, s., and giroux, h. a. (1993), education still under siege (2nd edn), south hadley, ma.: bergin and garvey. boswood, t. (ed.) (1997), new ways of using computers in language teaching, new ways in tesol series ii', alexandria, va: tesol. bowers, c. a. (1988), the cultural dimensions of educational computing: understanding the non-neutrality of technology, new york: teachers college press. bruce, b., peyton, j. k. and batson, t. (1993), network-based classrooms: promises and realities, new york: cambridge university press. celce-maurcia, m. (1991), teaching english as a second or foreign language (2nd edn), boston, ma: heinle and heinle. 62 an-j volume 9 number 3 crookes, g. (1997), 'what influences what and how second and foreign language teachers teach?', the modem language journal, 18 (1), 67-79. edge, j. (1996), 'cross-cultural paradoxes in a profession of values', tesol quarterly, 30 (1), 9-30. ely, m., anzul, m., friedman, t., garner, d. and steinmetz, a. m. (1991), doing qualitative research: circles within circles, london: falmer. foley, d. (1990), learning capitalist culture: deep in the heart of tejas, philadelphia, pa: university of pennsylvania press. gebhard, j. g. (1996), teaching english as a foreign language: a self-development guide, ann arbor, mi: university of michigan press. giroux, h. a. (1981), ideology, culture and the process of schooling, barcombe, uk: falmer press. giroux, h. (1992), 'critical literacy and student experience: donald graves' approach to literacy', in p. shannon (ed.), becoming political: readings and writings in the politics of education, portsmouth, nh: heinemann. harris, j. b. (1993), 'an internet-based graduate telecomputing course: practicing what we preach', in d. carey, r. carey, d. a. willis, and j. willis (eds), technology and teacher education annual, charlottesville, va: association of advancement of computing in education, 641-5. heinich, r., molenda, m., russell, j. d. and smaldino, s. e. (1999), instructional media and technologies for learning (6th edn), upper saddle river, nj: prentice-hall. jones, j. g. (1995), cybersociety: computer-mediated communication and community, thousand oaks, ca: sage. lockard, i , abrams, p. d. and many, w. a. (1997), microcomputers for twenty-first century educators (4th edn), new york: longman. mcgrath, d., thurston, l. p., mclellan, h., stone, d. and tischhauser, m. (1992), 'sex differences in computer attitudes and beliefs among rural middle school children after a teacher training intervention', journal on computing in education, 24 (4), 468-85. mclaren, p. (1998), life in schools: an introduction of critical pedagogy in the foundations of education, new york: longman. miles, m. b., and huberman, a. m. (1994), qualitative data analysis: an expanded sourcebook (2nd edn), thousand oaks, ca: sage. morrison, g. r., lowther, d. l. and demeulle, l. (1999), integrating computer technology into the classroom, upper saddle river, nj: prentice-hall. morton, c. (1996), 'the modern land of laputa: where computers are used in education', phi delia kappa (february, 1996), 416-19. murray, d. e. (1997), language and society in cyberspace, available at http://www.tesol.edu/assoclarticles/news9808-03.html.. 63 abdelilah salim sehhoui developing cross-cultural communicative competence via computer-assisted language learning okolo, g, bahr, c. and rieth, h. (1993), 'a retrospective view of computer-based instruction', journal of special technology, 7 (2), 11-14. pennington, m. c. (ed.) (1996), the power of call, houston, tx: athelstan. phillipson, r. (1992), linguistic imperialism, oxford: oxford university press. poole, b. (1997), education for an information age: teaching in the computerized classroom (2nd edn), boston, ma: mcgraw-hill. ramirez, a. g. and hall, j. k. (1990), 'language and culture in secondary spanish textbooks', modern language journal, 74 (1), 48-65. rice-lively, m. l. (1994), 'wired warp and woof: an ethnographic study of a networking classroom', internet research, 4 (4), 20-35. riel, m. (1990), building a new foundation for global communities, the writing notebook (january/february), 35-7. samovar, l. a., and porter, r. e. (1997), intercultural communication: a reader, belmont, ca: wadsworth. schrum, l. (1995), 'educators and the internet: a case study of professional development', computers and education, 24 (3), 221-8. sehlaoui, a. s. (1999), 'developing cross-cultural communicative competence in esl/efl pre-service teachers: a critical perspective', doctoral dissertation, indiana university of pennsylvania, dissertation abstracts international, dai-a 60106, 2042, publication no. 99348338. shannon, p. (1992), becoming political: readings and writings in the politics of literacy education, portsmouth, nh: heinemann. sleeter, c. e. and grant, c. (1999), making choices for multicultural education: five approaches to race, class, and gender, englewood cliffs, nj: merrill prentice-hall. tidwell, m. (1993), 'education for a change: appropriation and control in the mayan refugees' school', unpublished doctoral dissertation, university of pittsburgh, pennsylvania. ur, p. (1996), a course in language teaching: practice and theory, cambridge: cambridge university press. warschauer, m. (1995), virtual connections: on-line activities for networking language learners, second language teaching curriculum center: university of hawai'i at manoa. warschauer, m. (1996), 'computer-assisted language learning: an introduction', in s. fotos (ed.), multimedia language teaching, tokyo: logos international, 3-20. wise, a. (1995), 'raising expectations for technology in teacher education', ncate quality teaching, 5 (1), 2-3. 64 putting theory into practice: the creation of reals in the context of today's universities sally ann kitts* and john t. hancock** *department of hispanic, portuguese and latin american studies, university of bristol, email: s.a.kitts@bristol.ac.uk **department of biological and biomedical sciences, university of the west of england, bristol rich environments for active learning (reals), as described by r. scott grabinger and joanna dunlap, are comprehensive educational systems based on constructivist principles that present an intellectual and practical challenge to university lecturers. as teachers and researchers, academics are concerned with improving the learning potential of teaching strategies and, to this end, the theory of the real provides inspiration and ideas based on sound theoretical principles. yet in the context of the current pressured climate, having the time and resources to put such an extensive, theory into practice can seem little more than a pipe-dream. it is argued that using a computer-based application such as the hypermedia learning tutorials (hlts) as the heart of a real allows lecturers to take positive steps towards the creation of comprehensive, flexible, integrated learning environments. the concept of the hlt is discussed and a practical application in the field of advanced second-language acquisition is described. based on conceptual analysis and the results of preliminary student evaluation, it is argued that the hlt encompasses both in theory and in practice the chief qualities of reals and can form the basis for their creation in a wide variety of disciplines. introduction bringing together theory and practice in the context of university teaching is no mean feat. on the one hand, lecturers are challenged and motivated intellectually by the theoretical arguments in the field of education of thinkers such as grabinger and dunlap, who have written extensively about comprehensive constructivist learning communities which they term euch environments for active learning (reals) (grabinger and dunlap, 1995; grabinger and dunlap, 1998; grabinger, dunlap and duffield, 1997). yet, on the other hand, they are also challenged and demotivated on a day-to-day basis with the practicalities of teaching increasing numbers of 4 ait-j volume 7 number 2 students with a decreasing unit of resource in institutions where competition for funding is fierce and where there is pressure from external reviews of research and teaching performance. grabinger and dunlap (1995) are careful to point out that reals, in the full sense of the acronym, are large-scale holistic communities encompassing all aspects of the teaching and learning experience, and reject 'delivery technologies', 'computer microworlds' and 'computerbased applications' as reals. the practical task of creating such complete, all-encompassing learning systems can seem daunting, even impossible given the pressures faced by lecturers in the present higher education climate. the present paper will argue that, far from being highly limited and inflexible, as grabinger and dunlap suggest, when fully integrated into the teaching programme, computer-based applications like the hypermedia learning tutorials (hlts) discussed below can lie at the heart of flexible learning environments. the paper will begin with a discussion of the concept of the real as a theoretical constructivist approach to teaching and learning. after initial consideration of the concept of an hlt, theory will be linked to practice in exploring the application of the concept to the area of advanced second-language acquisition with the creation and integration of the hlts for spanish, caminos a la expresividad (pathways to expressivity). the final part of the paper will show how the hlt, in spite of its essentially computer-based format, encompasses both in theory and in practice many of the qualities set out in the theory of reals. reals: learning communities founded on constructivist principles constructivism is a term describing a theory of learning which grew from a theory about knowledge, developed by piaget. his work on cognition from the 1920s onwards eschewed the idea that knowledge has an independent reality which will one day be attained. instead he emphasized the role of cognitive subjects actively involved in a continual process of building their own conceptual representations of the world: 'knowledge proceeds neither solely from the experience of objects nor from an innate programming performed in the subject but from successive constructions' (piaget, 1977, p. v). this view of knowledge as constructed by individuals from their experiences led naturally to a reconsideration of the way in which people learn and this in turn to the way in which they are taught. learning is seen as a dynamic process with the emphasis on the learner rather than the knowledge to be acquired. learners will encounter new experiences and ideas from idiosyncratic perspectives and will either adapt their personal constructs (schémas) or create new ones within the overall construct which is their knowledge about the world. with the rejection of the lockean view of the mind as a tabula rasa on to which the teacher aims to inscribe a fixed body of facts called knowledge, the focus is transferred on to the constructive mind and the role of the teacher is to facilitate the process of the creation, adaptation and refinement of schémas (desforges, 1997). in order to fulfil this facilitative function, teachers have to make use of strategies that present new information and ideas about the world in such a way as to enable, encourage and support their students in the process of constructing and modifying their knowledge, a process which will be different according to the experience of each individual. reals are comprehensive instructional systems that 'provide learning activities that engage the students in a continuous collaborative process of building and reshaping understanding as a 5 sally ann kitts and john t. hancock the creation of reals in the context of today's universities natural consequence of their experiences and interactions within learning environments that authentically reflect the world around them' (grabinger and dunlap, 1995, p. 5). premised on constructivist principles, they promote an integrative approach to teaching that provides students with appropriate and varied learning strategies that enable them to take control of and responsibility for their own learning in a fully supportive, contextualized and meaningful environment. the concept of the hypermedia learning tutorial the term hypermedia learning tutorial is used to describe a synthetic approach to student learning that centres on a highly flexible and adaptable computer application, created using asymetrix toolbook. the concept of the hlt is applicable to a wide variety of disciplines: the central computer-based learning application is fully authorable and the constructivist integrative approach can be adapted and tailored to subjects as wide-ranging as molecular cell biology or advanced second-language acquisition, two examples currently being developed at the university of the west of england, bristol, and the university of bristol respectively. introduction learning objectives menu [hlt1 i vocabulary] ^ i grammar | ^ 1 | comprehension | textt| í-iself assessment)| exercises | | vocabulary | future student input | i| composition] 4assessed student output essays tapes texts hlt 11 i figure i: a schematic representation of a serial approach to learning with the hlts. 6 au-] volume 7 number 2 introduction learning objectives menu il t 4' irt 1 choice of tutorials | ^ ¡contents i-4i *«• i choice of texts i t understanding aids within texts grammar vocabulary comprehension self assessment exercises i i uture student input! | composrrionl • > 1 grammar i i assessed student output essays tapes texts figure 2: a schematic representation of a holistic approach to learning with the hlts. caminos a la expresividad n ° 2 : contents texts exercises 1 . mujeres oprimidas yolanda aguflar 2. entrevista con cristina alberdi 3. las mujeres indígenas tejen sus sueno* 4. ii encuentro continental de mujeres indígenas 5. ellas rosa montero composition and i oral presentation | grammar 1 vocabulary web pages i 6. polémicas por una fumigación polémica 7. peru plan de prevención ; y control de drogas 8. las drogas enemigas -antonio gala 9. medio ambiente j la comunidad europea 10. turistas en masa 1 1 . al borde de la muerte first time? click on the first text back for more? click on any button exit j i back to introduction i figure 3: the clickable contents overview from hlt 2 in the caminos a la expresividad series. 7 sally ann kitts and john t. hancock the creation of reals in the context of today's universities text 3: nación, nacionalidades, nacionalismo (varios) ¿que e s una nación? ¿qué significa ser nacionalista? aquí y en las páginas que siguen hay una serie de textos sobre este y otros asuntos relacionados con el tema de la españa de las naciones. ofrecen definiciones y opiniones, ideas y propuestas, clarificaciones y deseos para el futuro. algunas definiciones de las palabras nación, nacional, nacionalidad, y nacionalismo, según el diccionario general ilustrado de la lengua española. nación. 1. sociedad natural de hombres, a los que la unidad de territorio, de origen e historia, de cultura, de costumbres o de idioma, indina a la comunidad de vida y crea la conciencia de un destino común. 2. conjunto de habitantes de un pais repdo por el mismo gobierno. 3. territorio de ese mismo país. 4. en bolivia, extranjero dicho de personas. sinónimo: pueblo sound recording q número 1 i^iwia j número 2 imiltlmi número 3 i » i it i m i .iii número 4 p ! h: i if i exil 3 i comprehention exercise i notepad ] i pige 7 of 1 í | i • • back lo contente i #•> 1 * • • 1 figure 4: a typical hlt screen from caminos a la expresividad showing hotwords, audio buttons, navigation buttons, notepad facility and button linking to self-assessment comprehension exercise. hlts are completely student-centred learning environments that place students firmly in control of exactly what, how much and how they learn. using hlts students focus on and explore the resources and support they need as individuals. they access the material in a form that suits their own particular learning preferences, an important aspect to consider if an application is to be successful (jones, jacobs and brown, 1997). the tutorials are constructed so as to offer both a guided 'serial* approach (figure 1) and an open, exploratory or 'holistic' approach (figure 2), or a mixture of the two. the terms 'serial' and 'holistic' were first used in this context by pask (1976). the tutorials are made available on a 24-hour open-access basis and in that sense students can make as much or as little use of them as they wish, whenever, they wish. however, the tutorials are also fully integrated into the structure of existing teaching programmes using a variety of links which serve both to encourage students to use them and to extend the learning potential of both the hlts themselves and the current teaching programme. this approach is consistent with the acknowledgement that student motivation and a successful integration strategy are key elements of a successful implementation programme (stoner, 1997; harvey and mogey, 1997). hlts are thematic and centre on a wide selection of meaningful material including straightforward written texts, audio/video recordings, animations, pictures and diagrams (hereafter referred to simply as texts). progress through the different texts can be linear, using the basic forward and backwards navigation buttons at the bottom of the screen, or exploratory via a clickable contents map or overview (figure 3). 8 alt-j volume 7 number 2 key elements of the texts are hyperlinked to provide instant clarifications or examples should the student wish to click on them. in the case of audio/video recordings this is effected via accompanying transcriptions. each text is linked via a screen button to an interactive comprehension exercise which enables students to assess their understanding of the key concepts contained within it, with instant feedback and a score provided (figure 4). the thematic nature of the hlts means that each tutorial as a whole provides information and ideas which can then be drawn upon and exploited by the student in a variety of assessment activities. assessment takes various forms. a series of hyperlinked interactive computer-based activities gives opportunities for self-assessment with general feedback. peer and lecturer assessment strategies are employed in a non-computer setting through links to other learning activities such as lectures, seminars, practicáis, workshops and discussion groups, and provide individual feedback. amongst the assessment strategies used are several that involve the students with the continued expansion and development of the hlts, for example a student could be required to locate an appropriate text either on a related subject or in an entirely new area which they would then analyse in order to provide a series of explanations for what they considered to be key elements of the text as well as devising a comprehension exercise with right and wrong answers. the inclusion of student-created material is an important strategy in motivating students, updating the material and making it relevant to students' needs (landow, 1994; harvey and mogey, 1997). a practical application of hlts in advanced second-language acquisition at the university of bristol, four hlts entitled caminos a la expresividad have been developed as part of the final-year language programme from september 1999 for single and joint honours students of spanish. they are written in asymetrix toolbook using the calscribe template as a starting-point (kitts and whittlestone, 1998). caminos a la expresividad centres on written texts and audio/video recordings on a variety of themes of contemporary relevance to students of hispanic and latin american culture. the basic aim of the tutorials is twofold. firstly, they provide information and ideas on contemporary topics of relevance and importance to students which they can use as starting-points for discussion as they develop and refine their own ideas. to support this aim caminos a la expresividad has lists of references for further reading, some of which are available directly from the tutorial via hyperlinks, and connections to an hlt website which provides additional ideas and links to relevant material. secondly, the tutorials give practical assistance with language work in terms of clarifications and explanations of complex or unusual aspects of syntax and lexis and through interactive self-assessment exercises on grammar, vocabulary, comprehension and composition. caminos a la expresividad is being integrated fully into the final-year language curriculum through links with oral presentations and written projects. students explore each tutorial on an open-access basis over a six-week period during which they also attend oral discussion classes and listen to, present and critique presentations by their peers. these presentations are based on tutorial topics and use texts and supplementary material to generate ideas. some of the oral presentations will be assessed by the lecturer in addition to the written project required in each twelve-week teaching block. the students work on their own, in pairs or in small groups. project work includes additional edited texts for inclusion in caminos a la expresividad, 9 sally ann kitts and john t. hancock the creation of reals ¡n the context of today's universities assessed oral presentations, for example representing contrasting arguments on a topic, or written essays expressing students' own views but informed by the different opinions and ideas contained in the various texts. the best student projects will subsequently be incorporated into existing tutorials or used as the basis for new ones. evaluation is an essential part of the development and introduction of any new pedagogic method (laurillard, 1993; draper, brown, edgerton, henderson, mcateer, smith and watt, 1994; draper, 1997) and formative evaluation has been carried out at various points in the development of caminos a la expresividad, from pre-program design through to target group assessment of two of the four final versions. the pilot tutorial, entitled 'la españa de las autonomías', was evaluated through observation and a user questionnaire in january/february 1998. response to this student evaluation led to greater diversity of themes and highlighted a number of technical problems with the software which have since been resolved. by far the most positive aspect was the overwhelming student support for the concept of the hlts for spanish as a new student-centred yet fully-integrated and responsive approach to final-year language teaching. ninety-three per cent of respondents to the questionnaire enjoyed the pilot hlt and considered the caminos a la expresividad series as a whole to be a valuable innovation to the language curriculum. a minimum of 71 per cent in each case approved of the language content, variety of exercises, planned integration and overall design of the courseware. further user evaluation of another two of the final four spanish hlts was carried out in march 1999 with the current final year whose successors will be the first full users. fortyfive students tried out the tutorials and discussed the integration plans. as far as language skills are concerned, 95 per cent agreed that the embedded metalinguistic points and exercises would help to improve their knowledge, understanding and ability to use the spanish language, while 98 per cent of those questioned favoured the integration plans, agreeing that caminos a la expresividad offers effective and valuable additional support to the process of oral and written expression in spanish. other evaluation results are discussed below. further evaluation will be carried out following the full integration of the tutorials into the degree programme from september 1999. are hlts really reals? grabinger and dunlap (1995, p. 5) give five key aspects of the holistic constructivist approach to learning which characterize reals. they maintain that reals: • promote study and investigation within authentic contexts; • encourage the growth of student responsibility, initiative, decision-making, and intentional learning; • cultivate collaboration among students and teachers; • utilize dynamic, interdisciplinary, generative learning activities that promote higher-order thinking processes to help students develop rich and complex knowledge structures; • assess student progress in content and learning-to-learn within authentic contexts using realistic tasks and performances. 10 alt-} volume 7 number 2 if hlts are to be considered as reals they should embrace these characteristics. an examination of the hlts both in conceptual terms and in their application to second-language acquisition show that to a large extent they do indeed have these characteristics. study within authentic contexts hlts make use of authentic materials grouped together under key themes. they are rich in information that is presented in relevant, meaningful contexts by virtue both of this thematic structure and through their integration into a wider teaching programme. in terms of the caminos a la expresividad series, this is achieved by providing texts in spanish from authentic peninsular and latin american sources on real-life topics of immediate relevance to today's students. they also support students in the study of the structure of the language itself in a rooted, contextual and authentic fashion. in the march 1999 evaluation, 90 per cent of the students found the topics covered by hlts 2 and 3 to be relevant and meaningful. while only 58 per cent agreed that a contextual approach was likely to be more effective for them than other more conventional strategies such as working through a grammar book or attending grammar classes, a large number (38 per cent) were unsure. classroom observation and informal feedback suggests that this uncertainty was simply because the students felt unable to give a definite response until they had tried out the new approach over several weeks. only 4 per cent felt that the new approach was unlikely to be more effective. student responsibility and intentional learning integration through assessment and the involvement of students in the continued growth and development of the hlts motivates students, thereby promoting and encouraging study and an increase in student responsibility, initiative and decision-making. students are also encouraged to take responsibility for their learning by the presentation of the hlts on an open-access basis, giving the students control about how much they use them. by focusing the students' minds on the final goal of the production of an item for assessment and presenting them with varied strategies within the hlt to achieve that final goal, the tutorials encourage intentional learning, as described by palincsar and klenk (1992) as 'an achievement resulting from the learner's purposeful, effortful, self-regulated, and active engagement' (p. 212). in relation to caminos a la expresividad, this applies both to the thematic work and the linguistic revision. the open presentation of the material allowing both serial and holistic approaches allows the hlts to respond to students' endogenous learning styles and to their existing framework of knowledge. sixty-two per cent of students in the march evaluation agreed that having the opportunity to contribute to existing or new hlts would encourage them to use the tutorials. again there were very few negative responses (7 per cent) with the remaining 31 per cent expressing uncertainty until they had experienced the hlts more fully. the hlt website was seen as a very positive and useful tool for the production of assignments by 97 per cent of students who considered that it would help them to develop and refine their ideas on the subjects discussed. collaboration among students and teachers the collaborative process of involving students in the continued expansion and development of the hlts is a motivating factor and a key to encouraging students to take responsibility for their learning. it gives students a share in the ownership of the learning resource and is also encouraged via additional learning activities linked to the hlts such as non-assessed oral presentations and discussion workshops to consider ideas raised by the texts and their ii solly ann kitts and john t. hancock the creation of reals in the context of today's universities hyperlinked computer-based activities. students have increased possibilities for collaborative work both with their peers and with their lecturers and just over 80 per cent of the students questioned in the march 1999 evaluation liked this idea. dynamic, interdisciplinary and generative learning activities both the hyperlinked and the non-computer-based activities involve students in generative learning in that, working with the ideas and concepts presented through the hlt texts, the different activities require that they 'engage in argumentation and reflection as they try to use and then refine their existing knowledge as they attempt to make sense of alternate points of view' (grabinger and dunlap, 1995, p. 19). the assessed work in particular involves them in relevant investigative projects which require them to develop and extend their knowledge structures, using higher-order critical and analytical skills. in caminos a la expresividad the fact that the assessed work is in spanish necessitates a high-level ability to use the language and this is supported by the metalinguistic aspects of the tutorials; for example, the explanations of items of syntax and lexis within the texts enable the students to observe the language in action and this in turn calls for the use of higher-order skills of analysis and synthesis to abstract the meaning behind the usage and incorporate the new structures into their own knowledge hierarchy. while 76 per cent of students found this a more attractive approach to the learning of language than other strategies they had experienced, only the results of the first series of assessments compared to those of previous years will be able to provide a clear indication as to whether this approach is more successful at developing and extending the students' higher-order critical and analytical skills compared with previous methods employed. assessment in authentic contexts students are assessed via a variety of different tasks and projects, both formative and summative, at different levels, involving self, peer, and lecturer assessment and with feedback always provided. it is not envisaged that the spanish hlts will be static creations, rather the individual tutorials will be extended and new ones created with the involvement of students in all aspects and in particular utilizing material prepared by the students in the various forms of assessment, thereby rendering that assessment more meaningful and authentic. in the march evaluation, 63 per cent of students felt that contributing directly to existing or new hlts would give added meaning to their work and help them to feel fully involved in and responsible for their own learning. again there were very few negative responses (7 per cent) with the remaining 30 per cent again expressing uncertainty until they had experienced the hlts more fully. conclusion the hlts, both conceptually and in their application to advanced second-language acquisition, encompass the chief qualities set out in the theory of reals, a view that is confirmed to a large extent by the evaluation carried out with final-year students in march 1999. students are encouraged to take responsibility for their learning through generative learning activities in authentic and realistic contexts and assessed on their learning in varied ways that encourage and demonstrate higher-order thinking and a collaborative approach, in terms of both student/ student and student/lecturer interactions. 12 alt-) volume 7 number 2 the concept of the hlt is not limited to a language application but rather can be applied across the disciplines: the computer-based application it revolves around is easily authored and texts can be written, audio/video, pictorial or diagrammatic. work at the university of the west of england, bristol, on an hlt for molecular cell biology and a literary application being developed at the university of bristol will demonstrate the interdisciplinary nature of the approach as both the programs and the integration strategies are developed and evaluated. the comprehensive integration into the degree programme of the computer application at the heart of the real, through a variety of complementary learning and assessment strategies, is clearly a vital factor, for without it the result would certainly be no more than useful but limited computer applications. planned further analysis of practical applications of hlts and detailed evaluation of the first cohort of students to graduate after the full integration of caminos a la expresividad into the degree programme will illuminate the extent to which reals have an effect on the final learning outcomes, both in terms of students' perceptions of their learning and their actual degree result. acknowledgements the hlts for spanish, caminos a la expresividad, have been supported in their development by three grants from the university of bristol teaching and learning group. the language material has been prepared by jaine beswick and sally-ann kitts, and the language consultants for the project are rogelio vallejo lozano and carmen brauning. references desforges, c. (1997), 'a theory of complex learning', perspectives 56, 17-22. draper, s. w. (1997, october), 'observing, measuring, or evaluating courseware: a conceptual introduction', http://www.icbl.hw.ac.uk/ltdi/implementing-it/measure.htm. draper, s. w., brown, m. i., edgerton, e., henderson, f. p., mcateer, e., smith, e. d. and watt, h. d. (1994), observing and measuring the performance of educational technology, tilt project report no.1, robert clarke centre, university of glasgow. fosnot, c. t. (1996), 'constructivism: a psychological theory of learning', in fosnot, c. t. (ed.), constructivism: theory, perspectives, and practice, new york: teachers college, columbia university, 8-33. fox, r. (1997), 'piaget on learning', perspectives 56, 23-37. grabinger, r. s., and dunlap, j. c. (1995), 'rich environments for active learning: a definition', alt-j 3 (2), 5-34. grabinger, r. s., and dunlap, j. c. (1998, september), 'rich environments for active learning: a definition', http://www.warwick.ac.uk/alt-e/rolling/123. grabinger, r. s., dunlap, j. c. and duffield, j. a. (1997), 'rich environments for active learning in action: problem-based learning', alt-j 5 (2), 3-17. harvey, j. and mogey, n. (1997, october), 'motivating students to use learning technology', http://www.icb1.hw.ac.uk/ltdi/implementing-it/motif.htm. 13 solly ann kitts and john t. hancock the creation of reals in the context of today's universities jones, p., jacobs, g. and brown, s. (1997), 'learning styles and cal design: a model for the future', active learning 7, 9-13. kitts, s. a., and whittlestone, k. d. (1998), 'calscribe: a multimedia template ideal for call development', recall, 10 (2) (november 1998), 4-11. landow, g. p. (1994), 'what's a critic to do?: critical theory in the age of hypertext', in landow, g. p. (ed.), hyper/text/theory, baltimore: john hopkins university press, 1-48. laurillard, d. (1993), rethinking university teaching, london: routledge. palincsar, a. s., and klenk, l. (1992), 'fostering literacy learning in supportive contexts', journal of learning disabilities, 25 (4), 211-25. pask, g. (1976), 'styles and strategies of learning', british journal of educational psychology 46, 128-48. piaget, j. (1954), the construction of reality in the child, new york: basic books. piaget, j. (1977), equilibration of cognitive structures, new york: viking. stoner, g. (1997, october), 'a conceptual framework for the integration of learning technology', http://www.icbl.hw.ac.uk/ltdi/implementing-it/frame.htm. von glasersfeld, e. (1995), radical constructivism: a way of knowing and learning, london: falmer. 14 reviews edited by philip barker danny saunders and nina smalley (eds.), the international simulation and gaming research yearbook — volume 8: simulations and games for transition and change, london: kogan page, 2000. isbn: 0-7494-3397-3. hardback, viii+271 pages, £40.00. the theme of this year's international simulation and gaming yearbook 'transition and change' addresses the topical discourse on the improvement of pedagogy by the introduction of active and student-centred learning in uk higher education. although the value of 'learning-by-doing' and experiential learning is well recognized by researchers in education, many subjects and university courses continue to employ teaching methods (such as lectures) that favour a passive reproduction of information rather than the fostering of student understanding. in this regard, simulations and games are powerful tools that not only promote an active, student-centred approach to learning but have the potential to help realize government policies to improve the quality of teaching and learning in higher education institutions in the uk. games and simulations, in fact, are becoming a steadfast methodology in certain subjects as well as in professional training. this eighth volume of a series on simulation and gaming research is vital proof of the persistent development and use of the methodology by dedicated teachers and trainers. the publication consists of a compilation of papers from well-known scholars and practitioners. theoretical topics are amply 76 complemented with descriptions of use and implementations of simulations and games in a variety of international educational settings. contributions are grouped thematically into sections on 'game design and development' (1), 'interpersonal relations' (2), 'business and management' (3), 'technology and computing' (4), 'practitioner's notes' (5) and 'yearbook information' (6). the book's overall structure follows a sequence that caters to readers who are unfamiliar with the topic area. the first section with theoretically orientated papers provides an introduction into current research and thought that places the application examples from the following sections in context. while most authors are enthusiastic about games and simulations, critical pieces are also included so that overall a very balanced view of the methodology is provided. the article by henry ellington, for example, reviews eloquently the categories and key characteristics of games and simulations in respect to pedagogy. depending on the context, games, simulations and case studies can be used to develop communication, interpersonal, research, analysis and cognitive skills. moreover, with an increasing trend toward computer-mediated self-study, online and distance learning, the need to balance studentcentred learning with interactive and face-toface learning opportunities is stressed. while games possess many strengths, such as the potential to integrate perspectives from different disciplines and to promote deep learning, the ak-j volume 9 number 2 author points also to weaknesses, such as the organizational challenges of accommodating games in a university schedule, the difficulties for assessment and the potential misuse of games as diversions. further papers in this section discuss simulation design, the implications of the emotional element in role-play and the use of simulations to develop and assess policy in health care. readers familiar with the field will not be disappointed. topical sections support a selective reading approach focusing on specific issues and applications. contributions cover a wide spectrum ranging, for example, from a playful imaginary classroom exercise on the diffusion of innovation and a training workshop for professionals in the catering industry to an interesting simulation on interpersonal relations which is designed to help participants gain an understanding of prejudice and discrimination. here, real world discrimination is 'rehearsed' in the simulation through deliberate segregation and information-channelling. the artificially created difference in information access and level leads to mistrust and prejudices in the interactions of the reunited group which are discussed in a structured manner at the conclusion of the exercise. much deeper understanding is achieved through this experiential approach than could be obtained through discussion alone. the final part, 'yearbook information', contains the most comprehensive and well-organized reference section on gaming and simulation work of which i am aware. a bibliography of recent articles categorized by major disciplines provides readers easy access to additional information and research on many different aspects of gaming and simulations. there is also an up-to-date list of national and international publications, such as books and journals as well as addresses of companies that design and distribute games. the simulation and gaming research yearbook dazzles with a great breadth of contributions from many fields and countries. however, depth and quality of individual articles vary. a number of essays from section 5, entitled 'practitioner's notes', are rather sketchy and eclectic. a lengthy description of a game design competition may have great entertainment value to the contributor's friends and acquaintances, but it is likely to be a cause for bewilderment for many others. on one hand, one may argue that more rigorous editorship might have helped to alleviate quality discrepancies, but the trade-off would almost certainly be less diversity in terms of authors and topics. this is a difficult choice to make, for there is a delicate balance between inclusion, breadth and diversity and inconsistency, lack of quality and eclecticism. on the other hand, a rigorous scholarly evaluation may be uncalled for here. in the preface of the fourth gaming and simulation yearbook, saunders asserted that reading passively about active learning is somewhat a contradiction of terms. games and simulations need to be experienced to evaluate their real quality. nonetheless, for lack of alternative media, the simulation and gaming yearbook series is, so far one of the best accounts of activities and research in the field of gaming and simulation available. this edition on transition and change complements earlier volumes with new and unique contributions that re-emphasize the wealth of possibilities for the use of games and simulations in different fields. this book, i believe, will proof useful for teachers and educators seeking to improve the learning experience of their students as the collection of theory, concepts, and practical examples provides an abundance of ideas and tools for use in various teaching and training situations. andrea frank cardiff university david mcconnell, implementing computer supported cooperative learning, 2nd edition, london: kogan page, 2000. isbn: 0-74943135-0. hardback, 264 pages, £19.99. in the six years since the first edition of this book was published much has changed. in 1994 the web was in its infancy and much of the educational use of computer-mediated communications (cmc) was of a pioneering or experimental nature. the use of cmc is not yet a mainstream activity but it is certainly commonplace. the world of text-based systems has been enriched by newer web-based systems and the prospect of full multimedia is now much closer as processing power and bandwidth are increasing rapidly. this book is not primarily about technology; it is aimed squarely at practitioners who are interested in co-operative learning and as such the pedagogical basis of the book is much more stable than technological change might imply. this book remains a valuable contribution because it is about the 77 reviews implementation of a pedagogical practice using technology rather than being simply technologydriven. the author describes the book as being divided into three sections. part i examines co-operative learning and the technologies available for supporting co-operative learning. it then goes on to draw comparisons between computer supported co-operative learning (cscl) and face-to-face environments. part ii (chapters 5 and 6) considers design issues and presents a case study of a long-running cscl programme. in part iii (chapters 7 and 8) the author concludes the book with a review of advanced learning technologies that can be applied to cscl and an examination of research into cscl. i have provided the indications as to which chapters fall into each section as the book has no guide to the sections that the author describes. chapter 1 of the book introduces the idea of cooperative learning. this chapter is the heart of the book, as it sets out the pedagogic framework around which the other chapters develop. it is updated by the inclusion of an additional section about socially orientated theories of learning that have become increasingly significant in the years between the two editions of this book. the chapter acknowledges the various uses of the terms 'collaborative' and 'cooperative' within the field of cscl.and wisely decides to use co-operative as an inclusive term to encompass the various approaches to the subject. the use of the term co-operative allows the author to draw support from the wider social theories of co-operation associated with axelrod and argyle. chapter 2 deals with technologies for cscl and is one of the parts of the book most affected by recent developments. the chapter outlines some of the currently available software systems for cscl in a more fully developed section than the first edition. these are divided into asynchronous and synchronous types, with the author expressing a clear preference for systems that support knowledge construction rather than systems with an underlying purpose of instructional delivery. the system reviews are quite comprehensive and they are informed by recent research at the university of sheffield. other sections of the chapter are not so thoroughly updated. in particular the structure remains much the same as the 1994 edition, though some of the technical specifications have 78 been removed, as they are now out of date. the problem with this chapter is that it does not look closely at current technical developments and their potential for educational applications. telecommunication companies are now supplying digital subscriber lines (dsl): bt for example currently offers adsl (asymmetrical dsl) in some regions and has plans to make the technology available across the uk. the capacity and possible uses of adsl and dsl lines in general are not mentioned. mobile web access is now available via wap telephones, and third-generation mobile technologies that will offer permanent connections and high data rates are on the horizon. some discussion of the affordances of these new systems would have been welcome, as audio and video capabilities may become widely available in the near future. chapter 3 examines tutoring and learning in cscl and face-to-face environments. in structure and content it remains largely the same as the first edition. an entirely new chapter that considers the dynamics of group work follows. the focus of this chapter is aspects of gender and the differences observed between male and female participants in cscl. it is based upon empirical research undertaken by the author in the early 1990s. the second section consists of two chapters, 'designing for cscl' and 'cscl in action a case study'. chapter 5 is a helpful chapter dealing with design issues and it has comprehensively updated references in the new edition. it also contains a short additional section that develops the socially orientated theories of learning introduced in chapter 1 by considering recent developments in the study of learning communities. the structure of this chapter and the one following remains largely the same as in the first edition. chapter 6 draws on further empirical research by the author to provide a detailed case study. the final section of the book consists of two chapters, chapter 7 (trends and developments') and chapter 8 ('researching cscl'). both chapters have been updated and chapter 7 is a complete reworking of the equivalent chapter in the first edition. overall this remains a valuable book. if you have not either read or bought a copy of the first edition, i recommend that you do not miss the revised second edition. if you have a copy of the first edition, i cannot in all honesty recommend that you purchase the new edition, as it remains largely the same book. you may wish to read a alt-j volume 9 number 2 copy of the new edition, though the book contains little new material beyond revisions and the new chapter 4 looking at group dynamics. specifically the revisions made in the area of technology and of advanced learning technologies are not sufficient to warrant an unqualified recommendation. chris jones lancaster university linda lau, distance learning technologies: issues, trends and opportunities, hershey pa (usa): ideas group publishing, 2000. isbn: 1-878-28980-2. softback, ii+252 pages, $69.50 us. with the growing attention being paid to the use of distance learning technology in education, this book is intended to promote and enhance knowledge and understanding of distance learning technologies for academics and practitioners. a collection of papers presented in sixteen chapters, the book deals with a diverse range of distance learning situations from a web-based learning environment for children, through instruction systems for higher education to webbased training for the network marketing industry. it is divided into three sections. the first concentrates on the theoretical foundations of distance learning. the second section describes the conceptual aspects of distance learning strategies for using distance learning technologies and elements to consider for successful development of distance learning programs. the final section presents case studies of practical implementation of distance learning technology. the first four chapters focus on the theoretical aspects of distance learning. chapter one provides reasonable coverage of constructivist learning theories and examples of constructivist approaches, but recommendations for implementing these theories in web-based learning environments are brief and rather superficial and do not offer any real proposals. the second chapter provides sound advice but again does not fully cover the area. for instance, it acknowledges that text-based learning can produce 'lurkers', but the authors fail to address the problem of how to ensure discourse takes place. chapter three discusses a framework for summative and formative evaluation of webbased teaching in higher education. the authors suggest that formative evaluation should justify the use of the teaching method; while in a formative evaluation one should analyse the needs of the student users and design an appropriate system to meet those needs. chapter four considers the implications of distance training programs in large businesses. it argues that a change in the way training is carried out requires a combined approach of strategic planning, change management and project management. this article argues that these skills should be utilized to ensure the smooth transition within a corporate structure to technology-based distance learning. while acknowledging the different focus of higher education and business, the author argues that both business and higher education can draw on the principles of strategy, change management and project planning to implement distance education. the middle chapters of the book describe conceptual aspects of distance learning technology. chapter six identifies and discusses the issues that need to be addressed by an institution when considering forming an alliance to provide distance learning. it highlights both the potential benefits and pitfalls and identifies the need for longitudinal research into the way such alliances evolve and the way they might affect the mission, identity and survival of an individual school. chapter seven presents a case study of dodea electronic school which was set up as a distance learning institution, and has moved to it-based delivery. the school uses lotus notes and this chapter examines what the authors regard to be the key elements of a successful distance learning program. the nine criteria appear to be a useful set of elements, though some are only possible in a particular type of learning environment. they may not be appropriate for some distance learning communities where students are, for instance, based at home. this is an enthusiastic and upbeat chapter designed to assist the instructional designer in developing existing courses, although it does tend to gloss over some of the difficulties and drawbacks of designing such an environment. online education from the perspective of the course developer is considered in chapter eight. it identifies and discusses three issues the author considers essential for the successful design and implementation of an online course: first, pedagogical concerns the instructional and pedagogical goals of the course for the students; secondly, organizational issues the extent of online delivery, type of assignment, group size, types of interaction, prerequisite skills; thirdly, 79 reviews i n s t i t u t i o n a l issues s u p p o r t for staff a n d innovative practices, credit for o n l i n e c o u r s e s a n d s t u d e n t e v a l u a t i o n of courses. two of t h e five c h a p t e r s ( c h a p t e r s 12 a n d 15) in t h e final section h i g h l i g h t t h e p o t e n t i a l for greater isolation in o n l i n e l e a r n i n g a s o p p o s e d t o t r a d i t i o n a l face-to-face e d u c a t i o n . t h e y discuss t h e a p p r o a c h e s t h e y h a v e t a k e n t o a d d r e s s a n d avoid this p r o b l e m . c h a p t e r twelve discusses t h e experience of s e t t i n g o u t specifically t o create a ' c o m m u n i t y of l e a r n i n g ' i n a p r o g r a m m e t h a t is 60 p e r c e n t face-to-face a n d 40 p e r c e n t o n l i n e ( t h o u g h s t u d e n t s t e n d t o s p e n d m u c h m o r e time u p t o 80 p e r c e n t o n l i n e ) . c h a p t e r f o u r t e e n e x a m i n e s w e b b a s e d i n s t r u c t i o n systems ( w b i s ) from t h e systems perspective, p r e s e n t i n g critical issues a n d p r o b l e m s r e l a t i n g t o w b i s . it p r e s e n t s a t a x o n o m y for classifying c u r r e n t t y p e s of w b i s , a n d suggests h o w t h e t a x o n o m y c a n b e used t o evaluate a n d select a p p r o p r i a t e t e c h n o logies t o d e v e l o p w b i s . t h e c h a p t e r e n d s by assessing t h e i m p a c t of c u r r e n t technological a n d s o c i o e c o n o m i c t r e n d s o n t h e future of w b i s . f o r m e , this w a s o n e of t h e m o s t interesting a n d f o r w a r d l o o k i n g c h a p t e r s of t h e b o o k . m y m a i n criticism of this b o o k is t h a t it w o u l d have benefited from closer p r o o f r e a d i n g a n d a m o r e n a r r o w l y defined focus. t h e p a p e r s m i g h t , however, b e useful r e s o u r c e m a t e r i a l for s t u d e n t s u n d e r t a k i n g a s t u d y of t h e a r e a . t h e r e a r e exceptions t o t h i s p a r t i c u l a r l y in t h e final section of t h e b o o k w h e r e t h e p a p e r s offer c o n t e n t of s u b s t a n c e t o t h e a c a d e m i c o r researcher. elaine pearson university of teesside chris morgan and meg o'reilly, assessing open and distance learners, london: kogan page, 1999. isbn: 0-7494-2878-3. softback, x+227 pages, £19.99. this volume is one in the 'open and distance learning series' published by kogan page. assessment is a controversial issue in any context and in open and distance learning (odl) there are many additional considerations which morgan and o'reilly identify and address. the book is divided into three parts. part a (chapters 1 to 4) deals with 'issues and themes in open and distance assessment', part b (chapters 5 to 11) covers 'assessing learners in 80 open and distance learning', while part c is devoted to 'case studies'. the authors suggest that readers may like to read sequentially or dip into sections of interest or need. for the purposes of this review i read sequentially but the layout of the book with its clear headings and subheadings, a full contents section and a comprehensive index facilitates serendipitous dipping. the first two chapters of part a provide a consideration of the positive and negative aspects of assessment and the purposes of assessment in general. in chapter 3 the focus of the discussion is assessment in an open learning context. a vignette describes how one distance learner took a strategic or pragmatic approach to her studies by focusing entirely on the assessment requirements and almost completely bypassing the carefully prepared study materials. this raises important questions about how effectively study materials support assessment and are linked to the objectives of the course. this is not only a problem in distance learning; we are all familiar with students in face-to-face contexts who appear to engage minimally with our carefully prepared sessions and whose only questions are about the assessment requirements! the authors engage us in interesting debate as to whether or not open learning really is as open and student-centred as its proponents assume. in relation to assessment they highlight students' concerns about lack of choice or variety in assessment methods, minimal opportunities for student self-assessment and inscrutable marking schemes. there are similar issues raised by tutors concerning standards and consistency, plagiarism and proof of authorship and the additional time that individualized learning activities require for marking. further discussion leads to the identification of the key qualities required for open and distance assessments. it would be difficult to argue with this list: • a clear rationale and consistent pedagogical approach; • explicit values, aims, criteria and standards; • authentic and holistic tasks; • a facilitative degree of structure; • sufficient and timely formative assessment; • awareness of the learning context and perceptions. alt-j volume 9 number 2 in chapter 4 the use of online technologies in open and distance assessment is considered. from the premise that, so far, most odl assessment activities have remained rather static, the authors discuss the opportunities and losses that online learning and assessment present. they also suggest that we need to be aware of the potential trap of harnessing new technologies as a medium for presenting old assessment methodologies. part b ('assessing learners in open and distance learning') covers the design and communication of assessment tasks, marking and grading, creating dialogue through assessment, time management, the development of assessment policies and evaluation. for me this is the most valuable section of the book. the authors request that readers read the materials critically rather than considering them as recipes for practice; they also hope that the information given will stimulate further ideas. chapter 5 considers aligning assessment with objectives, the selection of appropriate methods and the amount, length and timing of assessments. readers are asked questions on the validity, reliability, authenticity and openness of their chosen methods and the chapter ends with a useful checklist on the design of assessment tasks. in chapter 6 the important aspect of the communication of assessment tasks is discussed. the need for clear and sufficient information is considered and the impact on students when this is inadequate is highlighted. there is a very helpful section on the development of marking criteria and suggestions for the moderation of new assessment tasks before they can be used. chapter 7 gives straightforward and practical advice on marking and grading, and giving feedback. the issue of authentication is considered and, whilst stating that most adult open and distance learners are highly motivated and would consider cheating an anathema, it is suggested that we must ensure that the processes which ensure authentication must not also undermine the learning experiences of the students. some helpful ideas are offered from benson (1996) which suggests the following activities: • link assignments so that each builds on the former; • individualize topics as far as possible with the use of students' own workplaces, lives and values as the source of discussion; • use self-directed forms of learning such as learning contracts; • use work-based mentors, supervisors or assessors; • use videoand audio-based presentations as an alternative to print; • consider oral assessments via the telephone; • adapt and change the assessment topics regularly whilst maintaining the alignment between objectives, content and teaching and learning activities. chapters 8 and 9 give sensitive consideration on how to facilitate dialogue and communicate empathetically in an odl environment and how to deal with communication breakdown. the concept of learning communities is considered and strategies for enhancing self-management and time management are identified. in chapter 10 the link between odl assessment policies and institutional policies is discussed. the statement that 'policies reveal values' is illuminative in the context of how we value student learning and the degree to which we endeavour to put our students before institutional convenience. the last chapter in this section gives some timely advice on the variety of ways in which we can explore the effectiveness and efficiency of our assessment procedures. part c presents a selection of case studies that illustrate a variety of assessments. these have been organized to demonstrate the categories of learning outcomes developed by nightingale et al. (1996), assessing learning in universities: 1. thinking critically and making judgements. 2. solving problems and developing plans. 3. performing procedures and developing plans. 4. managing and developing oneself. 5. accessing and managing information. 6. demonstrating knowledge and understanding. 7. designing, creating, performing. 8. communicating. the thirty-one case studies presented are drawn from a wide range of disciplines, including education, nursing, business, anthropology, economics, philosophy, mathematics, agriculture and psychology. although i found it interesting to read what other people were doing, i did not find this section as helpful as parts a and b. using so 81 reviews many case studies inevitably means that each is given only brief coverage and it was not always easy to get a complete picture of the issues and strategies involved. nevertheless demonstrating the learning outcomes through the case studies was an innovative and appropriate way to draw the main themes of the book together. the authors are to be commended on producing a timely and readable book on this important topic which i recommend to all involved in the development and management of odl. it has much to offer to both those with experience in odl and those considering moving into this area of practice. a comprehensive reference list identifies many familiar sources but also a range of other authors and texts that will be worth following up. gillian jordan university of greenwich o. simpson, supporting students in open and distance learning, london: kogan page, 2000. isbn: 0-7494-3082-6. softback, vi+186 pages, £18.99. i was delighted to receive a review copy of ormond simpson's supporting students in open and distance learning. pressure of student numbers, competition with our neighbour universities and declining units of resource have meant that we need to look closely at the methods and tools of open and distance learning. we have our own institution-wide managed learning environment and supplement this with web-based resources and resourcebased learning. amongst other things, we are trying to support part-time students in full-time jobs and full-time students in part-time jobs and trying to optimize staffstudent contact by using as many other media as possible to provide and supplement student support. so to direct this book solely at academics working in open and distance learning is seriously to under-sell it. there are lots of ideas we could all look at. the coverage is thorough and very systematic, taking the reader through advising, supporting, tutoring and retaining students from a variety of backgrounds. the author's personal experience and topical anecdotes serve to underline the book's credibility. one area that might need more attention is that of the virtual or managed learning environments, such as webct, wolf or cose. whilst these systems may only combine existing functionality, 82 specifically conferencing, email and content transmission, the combination gives considerable synergy and these systems are set to grow. also some other topics, towards the end of the book, particularly 'structure, quality and staff development in student support' and "theories of student support' (the latter looks at models of interaction from counselling psychology) are less satisfying. whilst the brief introduction to counselling is useful, it does not do the subject justice. also there is actually an increasing interest in online counselling within the professional counselling community, for example in the british association for counselling and psychotherapy's journal counselling. this area could be reviewed in subsequent editions and used to build a better model of educational guidance based on online psychological counselling practice. as many institutions in the uk consider the adaptation and adoption of the techniques of open and distance learning, they are faced with the issue of quality. in particular they are faced with making sense of the all-embracing but rather abstract guidelines from the qaa on distance learning. the 'structure, quality and staff development in student support' chapter could have spent some time looking at the wider implications of 'quality' implied by bodies such as the qaa. having said that, i am tasked with supporting staff in my school as it moves towards more technology-supported learning and this is clearly the kind of book i could recommend to my colleagues systematic, comprehensive and practical. john traxler university of wolverhampton lyn pemberton and simon shurville (eds.), words on the web: computer-mediated communication, exeter intellect books, 2000. isbn: 1-87151656-0. softback, iv+124 pages, £14.95. new computer-mediated communication technologies, such as video-conferencing, email and the web have provided many different ways to interact with other people. the web alone has spawned chat-rooms, special interest discussion groups and marketing opportunities. this book considers the emergence and development of novel linguistic and social conventions for using these media. some of the papers in the book were specially commissioned, some were adapted ak-j volume 9 number 2 from presentations at a workshop in computermediated communication (cmc) organized by one of the editors (lp) at the sociolinguistics symposium in cardiff in 1997. others are developed from presentations at the writing and computers conference run by both editors in brighton, also in 1997. i found some of the papers slightly dated, given the exponential growth of web access and use in the public domain in the last two years with the advent of free internet service providers (isps) and unmetered web access. so, what are the objectives of this book? according to the introduction: "the aim has been to give coverage of as wide a range of approaches and phenomena possible at every level of interest to students of language, from the mechanics of replicating paralinguistic features in email to the pragmatics of multilingual communication and from the grammatical features of web page anchor text to the negotiation of meaning in an email discussion on traditional song.' the approach is certainly wide-ranging, spanning experimental psycholinguistics, systemic linguistics, social identity theory and conversation analysis. the book comprises twelve chapters, which are split into two distinct sections. the first section covers the issues of new language structures used in networked and computer-supported communication (such as the hypertext link mechanism that is discussed in chapter 1). the second section of the collection focuses on the languagebased behaviours of users of cmc technologies. jaime henriquez in chapter 1 analyses the differences between traditional print media and web communication. the focus of this analysis is via hypertext links. these links are the connections between a point on one web page and another. he concludes that this directional linking and the palimpsest display place a greater load on the reader to supply context. chapter 2 discusses knowledge content and narrative structure in terms of conversation theory. in chapter 3, einat amitay tries to identify emerging linguistic characteristics of web pages, concluding that consistent authoring choices are being made by web authors to explain context. chapter 4, by helmut gruber, is entitled 'scholarly email discussion list postings: a single new genre of academic communication?'. gruber analyses two lists, ethno and linguist, and concludes that scholarly email discussions do form a single genre, which is characterized by academic letter-writing as well as oral communication. chapter 5 (by pirkko raudaskowski) discusses the use of communicative resources in internet videoconferencing between groups of students in finland and sweden, using the cu-seeme system. in the final chapter of part one, fouser, inoue and lee discuss the pragmatics of orality in english, japanese and korean computermediated communication. they analyse synchronous chat-room data and asynchronous public newsgroup postings in all three languages. chapter 7 marks the start of part two of the book. in this chapter zazie todd and stephanie walker investigate multilingualism on the net, and the use of the web for the support of language. the following contribution by sandra harrison (chapter 8) then describes the use of politeness strategies in an email discussion group as a means of maintaining the virtual community. this paper uses brown and levinson's 1987 work on politeness and concludes that the framework can usefully be applied to email discourse. in chapter 9 heather matthews addresses the effects of group identity on discussions in public online forums, comparing a computer professionals' forum as a concrete focus forum and a new age discussion group as a forum with an abstract emotional focus. next, in chapter 10, sonja launspach zooms in to the negotiation of meaning in an internet discussion list. she focuses on a single discussion to demonstrate '. . . some of the difficulties that participants encounter in deriving the intended interpretation for utterances in the contextual environment of cyberspace'. in chapter 11, jacqueline taylor attempts a social psychological analysis of computer-mediated group interaction. finally, luiz perez-gonzalez writes on the interactional implications of computer mediation in emergency calls, concluding that electronic data entry forms are restrictive, and need a more flexible layout. however, computermediated interaction can assist in the detection of hoax calls. my overall conclusion about words on the web is that it offers a fascinating overview of the diversity of approaches to the analysis of ways in which language is used in computer-mediated communication. i felt that it lacks cohesion, but nevertheless provides an interesting introduction for the language student or web user. linda clark university of glasgow 83 calt12206.fm alt-j, research in learning technology vol. 12, no. 2, june 2004 implementing a learning technology strategy: top–down strategy meets bottom–up culture bernard lisewski* university of salford, uk taylor and francis ltdcalt12206.sgm10.1080/0968776042000216228alt-t research in learing technology0968-7769 (print)/1741-1629 (online)original article2004taylor and francis ltd1220000002004bernardlisewskieducation development unit, crescent house, the crescentuniversity of salfordgreater manchesterm5 4wtukb.lisewski@salford.ac.uk using interview-based ‘insider case study’ research, this paper outlines why the university of salford has adopted a learning technologies strategy and examines the factors which are likely to lead to its successful implementation. external reasons for the adoption focused on the need to: respond to ‘increased higher education (he) competition’, meet student expectations of learning technology use, provide more flexibility and access to the curriculum, address the possible determining effect of technology and establish a virtual learning environment (vle) presence in this ‘particular area of the he landscape’. internal drivers centred on the need to: continue a ‘bottom– up’ e-learning pilot project initiative, particularly given that a vle is a ‘complex tool’ which requires effective strategic implementation, and promote the idea that learning technology will play an important role in determining the type of he institution that the university of salford wishes to become. likely success factors highlighted the need to: create ‘time and space’ for innovation, maintain effective communication and consultation at all levels of the organization, emphasize the operational aspects of the strategy, establish a variety of staff development processes and recognize the negotiatory processes involved in understanding the term ‘web presence’ in local teaching cultures. fundamentally, the paper argues that policy makers should acknowledge the correct ‘cultural configuration’ of he institutions when seeking to manage and achieve organizational change. thus, it is not just a question of establishing ‘success factors’ per se but also whether they are contextualized appropriately within a ‘correct’ characterization of the organizational culture. introduction the recent chep report (collis & van der wende, 2002) which examined the current and future use of learning technology in he concluded that: change is slow and not radical but he institutions which have a clearer view on their mission with respect to serving different target groups (such as lifelong learners) with learning technology and on their position in those markets demonstrate higher levels of use of learning technology. *education development unit, crescent house, the crescent, university of salford, greater manchester m5 4wt, uk. email: b.lisewski@salford.ac.uk issn 0968–7769 (print)/issn 1741–1629 (online)/04/020175–14 © 2004 association for learning technology doi: 10.1080/0968776042000216228 176 b. lisewski the main challenge arising from these findings is for he institutions to develop more strategic policies on how learning technology can be used for different target groups especially given that as timmis (2003, p. 1) argues: learning technology is set to change the prevailing teaching paradigm as well as helping address other national drivers such as widening participation, increased student numbers and accessibility. this is not to exclaim that the call for more strategic approaches to learning technology implementation is a relatively new phenomenon. for example, the macfarlane report (1992, p. xi) recommended a more strategic and longer term approach to the development of learning and teaching in an expanding he system within which there should be a ‘vigorous programme of research and development in teaching methods and educational technology’. in the mid-1990s, daniel (1996) urged universities to adopt learning technology strategies to address the changing conception of the ‘university campus’ and the need to enhance curriculum accessibility, reduce costs and increase the flexibility of teaching and learning provision. on similar grounds, the dearing report (1997, section 13.17) emphasized that existing communications and information technology (cit) resources could only be used more effectively if ‘institutional managers developed and implemented a coherent and comprehensive cit strategy.’ that said, recent studies of the uptake of new technologies in the he sector (jenkins et al., 2001; smith, 2002; stiles, 2002) have highlighted the lack of institutional learning technology strategies as a barrier to their more widespread adoption in teaching and learning practice. however, even where it has been acknowledged that learning technology is moving to the centre of he institutional teaching and learning strategies, it is: marginal in terms of the practices and cultural values of most academic departments. a common theme in the audit of institutions was the presence of ‘pockets’ of activity and innovation, while senior managers still talked about ‘the enthusiasts’ as a small but precious minority. (timmis, 2003, p. 2) learning technology implementation and managing organizational cultural change timmis seems to be suggesting that the gap between policy, strategy and local practice needs to be bridged or what clegg et al. (2003, p. 51) refer to as the ‘need to keep track of the messiness on the ground’. within the learning technology literature, various authors have argued that the successful implementation of learning technology based teaching and learning practice will require changes in organizational culture. saunders (1998, p. 175) reports that it is the organizational culture and environment rather than the technology that ultimately determines the learning experience as well as being the principal influence on the use of the technology. however, implementation studies of learning technology within he settings have tended to display rather unsophisticated perspectives on the nature of the organizational culture and how to implementing a learning technology strategy 177 achieve effective cultural change. they have usually focused on the need for some combination of having a clear vision, strategic planning, technological infrastructure development and a strong pro-active institutional leadership (garrison & anderson, 2003, p. 112). mccartan and hare (1996) highlight the dynamic relationship between institutional culture and strategies for change led by the need for senior management support, integrative staff development policies, responsive central services and a mixture of funding arrangements. the dearing report (1997: section 13.57) however, placed less emphasis on financial factors as a possible barrier to successful implementation, in favour of the need for well-informed institutional leaders to consider both a ‘fundamental rethink of institutional priorities’ and ‘an equally essential change of culture’. littlejohn and cameron (1999) and mcnaught and kennedy (1999) focus on the importance of combining policy, culture and support alongside strong senior management direction in effecting strategic change involving the integration of new technologies within the academic curriculum. brown (2002) makes the case for ‘re-engineering the university’ in seeking to introduce more flexible learning technology based teaching and learning methods. this will involve not only a technical challenge but also a culture change, although he acknowledges that this will be harder to achieve especially in the democratic environment of a university. collis and moonen (2002) give little consideration to the nature of he organizational culture per se; instead they focus on what they regard as the four key components of technology based flexible learning: technology, pedagogy, implementation and institution. they assert that strategies that require the effective implementation of learning technology in educational settings revolve around the 4-e model: environment (institutional context), educational effectiveness (perceived or expected), ease of use and engagement (the person’s response to technology and to change). the individual’s likelihood of making use of learning technology innovation will be a function of these four factors. hanson (2003, p. 119) in examining the strategic implementation of e-learning in australian universities emphasizes the importance of senior management support, top–down budget allocation, the organization of central support—technical and pedagogic—and effective links across these and with faculties and schools, staff development opportunities and reward and recognition for teaching or involvement in e-learning. she regards the most critical factor as being the ‘winning of hearts and minds of lecturers’, who not only have to adapt their teaching methods but also change their conceptions of teaching in terms of generating a culture change. these learning technology implementation studies all identify the need for some form of organizational cultural change to occur alongside the existence of ‘key’ success parameters in driving this change but they tend to characterize university organizations as culturally rather simple and uniform. theorists on change and the ambiguity and complexity of organizational culture such as fullan (1993, 1999, 2003) and alvesson (1993, 2002) suggest that organizational change may not be such a straightforward process. alvesson (2002, p. 171) in particular warns against the ‘trivialization of managing culture’ by authors who offer ‘unitary and unique’ views of organizational culture that can be ‘shaped by managerial intentions’. 178 b. lisewski the university of salford and its learning technology strategy hannan and silver (2000, p. 87) describe the university of salford as an institution ‘disaggregated into interlocking subcultures’ within which a ‘number of competing cultures wax and wane’. given an ever changing he environment, they argue that teaching and learning cultural contexts are likely to be best explained by acknowledging ‘local’ complexity, conflict and confusion with regard to institutional policy developments. the university of salford learning and teaching strategy (university of salford, 2002a) is embedded within a strategic framework which emphasizes the institution as an ‘enterprise university’ alongside its two other core activities of teaching and research. the learning technologies strategy (lts) is viewed as being central to the institution’s programme delivery and learner support system. the principal aim with regard to integrating learning technologies within the university of salford curriculum is to: enhance the quality of, and access to, learning by supporting and developing the curriculum through the appropriate and effective use of learning technologies. (university of salford, 2002b, p. 2) two objectives are identified in relation to achieving this aim, that is, to: enable staff to identify where the appropriate and effective use of technology will add the greatest value to the curriculum and to support staff in the application and integration of learning technology; maximize the institution’s commitment to, investment in, and return on, the effective application of learning technologies. (university of salford, 2002b, pp. 2–3) driven by the formulation of these two objectives, the lts has adopted a staged top– down three year development plan whereby all modules within the university curriculum should establish a ‘web presence’ using the blackboard vle by august 2005. prior to the lts, blackboard-based curriculum innovations took the form of organic bottom–up developments, championed by enthusiasts and supported by an informal coalition of central support agencies as part of an evaluated ‘soft money’ e-learning pilot project initiative. for example, findings from the e-learning pilot project evaluation reported that in some faculties there were: one or two highly enthusiastic adopters of learning technologies, and that these individuals were the most effective agents of change … major barriers are ‘people issues’, including cultural traditions, risk aversion, lack of knowledge and user acceptance … compounded by the fact that the teaching staff involved had a wide range of cit literacy, varying attitudes to technology and differing levels of prior experience in collaborative projects. (keegan, 2003, p. 6) in making the transition from bottom–up experimentation to a more strategic top– down learning technology approach, the university of salford lts is overseen by a learning technology steering group. this reports to the university’s teaching and learning development sub-committee which is an adjunct of the university’s main teaching and learning committee. the lts is facilitated by a newly formed implementing a learning technology strategy 179 learning technologies centre (ltc) in collaboration with the university’s education development and academic enterprise units. the ltc also offers technical and pedagogic support for the blackboard vle alongside ongoing guidance for faculties and schools to identify and address learning technology priority areas in the context of their annual academic and business planning requirements. furthermore, the lts is underpinned by different staff development and learning technology support processes comprising of awareness raising, demonstrated ‘exemplar’ projects and blackboard training alongside a network of faculty based learning technology teaching fellows. research approach and the emerging cultural ‘narratives’ in order to examine why the university of salford has adopted a learning technologies strategy and the factors which are likely to lead to its successful implementation, seven semi-structured interviews were conducted with people involved in the development and implementation of the university of salford lts at different levels of the organization. the people interviewed were: three members of the learning technology strategy development group, the ltc manager, a learning technologist and two academics who use the blackboard vle in their educational practice. each interview was tape recorded and subsequently transcribed and anonymized for the purposes of interpretation and analysis. this approach was underpinned by the phenomenological assumption that the university organization exists as a ‘narrative’ of individual experiences. therefore, people’s individual experiences of the organization can be studied in relation to the organization and any findings can help to explain how these participants experienced this type of organizational culture with regard to the lts implementation. interview respondents were asked to distinguish between the external and internal reasons for the adoption of a lts at the university of salford and what factors were likely to lead to the successful implementation of the strategy. as this is an ‘insider case study’ and to maintain anonymity, respondents have been identified by capital letter rather than job title. external reasons typical responses on the prevailing external conditions driving the adoption of the lts were: competitors have really taken on board the use of learning technologies … we are becoming uncompetitive in that what we have to offer isn’t as flexible, isn’t as accessible and there is an expectation from the students that that’s what they are going to get … there is also a notion that the technology is there now and so some of the drivers are the technologies themselves… (respondent d) they have got to do it because other people are doing it, its become flavour of the month … but the university needs to and wants to address key developments and student expectations, for example, doing postgraduate courses … to be able to access components of that online, for undergraduate students, there is the need for more flexibility in delivery as 180 b. lisewski recent internal salford research indicated that students were working more and longer hours but they were struggling academically… (respondent g) smith (2002, p. 1), stiles (2002, p. 5), hanson (2003, p. 119), watson (2003, p. 106) and recently published e-learning strategy documents from the dfes (2003, p. 11) and the hefce (2003, p. 2) all emphasize the need for he institutions to: respond to the rising global competitive pressure of additional educational providers, promote the accessibility and flexibility of curriculum delivery and cater for the more sophisticated technology expectations of an increasingly diverse and growing student population. with regard to the isomorphic tendency of ‘everybody else’s doing it’, jenkins et al. (2001, p. 9) report that this may have ‘something to do with critical mass, i.e. as more institutions began to use these vles, other institutions felt that they needed to invest in this area also.’ that said, both saunders (1998, p. 178) and boys (2002, p. 2) emphasize that introducing new technologies into he challenges fundamentally conventional academic roles, practices and organizational assumptions. concerns were expressed in the interviews about how the strategic use of learning technology could impact on the role of academics and their practice: we should abandon our control over the curriculum at our peril, we are still academics who design, define and largely deliver our own curriculum … we should not allow the technology to make that decision for us … if an unanticipated outcome of introducing learning technology was that that changed without us having stopped to think about whether or not that was a change that we wanted to happen, then we would be failing in our duties, so we should have that dialogue about what the role of an academic is where technology is part of the context… (respondent a) clegg et al. (2003, p. 47) maintain that a crucial issue for academics in he is ‘who has control over curricula and teaching methodology?’ for example, becher and trowler (2001, p. 12) present evidence from rhoades (1997) who found that academics had been ‘professionally marginalized’ in important decisions about ‘whether, and how to, introduce and use’ learning technology. it is alleged that this is more likely to occur when top–down managerialist strategies are invoked to mainstream learning technology innovations; the inference being that managers are able to exert more control over the composition, design and delivery of the he curriculum. internal reasons the need to maintain the momentum of the previous bottom–up and localized elearning pilot project alongside a desire to develop the university of salford’s identity in learning technology innovation formed the centrepieces of the internal driver responses for the adoption of a more centralized and top–down lts. for example: we managed to bring together a whole host of people throughout the university and to co-ordinate them and to try and provide something, some sense of direction and what needed to be achieved, the fact that we now have a learning technology strategy has to be as a consequence of what we tried to do… (respondent d) implementing a learning technology strategy 181 there has been a drive from the bottom–up which has caused pressure and led to changes… (respondent g) given the history, nature and culture of this institution, it would expect of itself that it would have one, it has traditions of being a technologically based university, very vocational, those would appear to be part of the rationale for having such a strategy, the institution might think that its slipping down the table, getting more and more concerned about its place, and its position, about its competitiveness… (respondent f) within the literature, learning technology innovations in he organizations have been characterized as being part of a ‘transformational process’. jenkins, browne and armitage (2001, p. 29) suggest that vle implementation can be tracked as being ‘part of a continuum of development’ ranging from: small scale enthusiasts (individualized), to localized, to co-ordinated, to transforming, to embedded and finally to the institution wide ‘innovative’ stage. such processes are also reflected in the chep report (collis & van der wende, 2002, p. 8): institutions are now transferring from a period of rich and mostly bottom–up experimentation to a phase in which institution-wide use of cit is being encouraged. however, adjusting from bottom–up to more strategically placed and centralized top– down approaches is not without its potential pitfalls. brown (2002, p. 241) reports that although bottom–up approaches may lose out to competing initiatives and priorities within different parts of the university, a top–down management led strategy may be thwarted at middle management level where tough choices have to be made about the allocation of resources in the face of competing priorities at the departmental level of the organization. likely success factors interview respondents regarded the need for: creating ‘time and space’ for innovation, effective communication and consultation, more emphasis on the operational aspects of the strategy, staff development processes, negotiation as to the meaning of ‘web presence’ in local contexts and recognition by managers of the correct cultural configuration of the university as being the key success factors in the implementation of the lts. the following comments are examples of the need for room to innovate: its going to be difficult for them to do given workloads and the problems with time that they already have, using the vle isn’t going to solve their problems with time, its not going to be a means of reducing student contact… (respondent c) there is very little slack, flexibility, within the whole notion of how people develop their own programme, the teams that we have to develop, the whole approval systems and the work they are balancing, a lot of it is set in concrete at the beginning of the year, very little time given to development, you know we do not value the fact that in order to put together something that is really robust and innovative, takes time you know, its not just something that you can do overnight… (respondent d) 182 b. lisewski smith (2002, p. 1), saunders (1998, p. 176), and jenkins et al. (2001, p. 23) emphasize the importance of academics being given time to develop familiarity with different technologies and produce learning materials. steel and hudson (2001, p. 109) report that learning technology innovations must be recognised alongside other competing agendas and that the ‘notion of lack of space’ principally in terms of time may not be acknowledged by senior management. indeed as reynolds and saunders (1987, p. 207) illustrate in tracing curriculum planning formulation to changes in curriculum practice by using the metaphor of an ‘implementation staircase’: ‘making a response to policy requirements was only one amongst many other pressing concerns’ and that ‘what teachers thought and did about curriculum policy involved recognizing their shifting and precarious scales of priority over time’. steel and hudson (2001, p. 109) also highlight the importance of effective communication at all levels of the he organization in undertaking curriculum innovation and development. this is not just about delineating a clear vision but also how to develop and maintain a dialogue between senior management and teaching staff as to what management think is achievable using learning technology and what is actually achievable on the ground by academic practitioners. these issues were reflected in the salford context: it is all very well the strategy saying we will do x amount of things in the first year or whatever but that’s very much coming from the directors, the managers but its important to find out what the academic staff think is feasible, its hard to know exactly what information is feeding down but my perception is that perhaps not a lot is feeding down or feeding both ways… (respondent b) a common theme in the responses was the challenge of operationalizing the strategy between the top–down intentions and day to day academic practice at all levels of the organization: the strategy is never going to include everything which may happen but a lot more could have been included which would have made the strategy appear far more robust and in return would probably have gained greater support and trust from staff if they felt that all angles were covered… (respondent b) i am not convinced that the thinking has been properly done about how to operationalize the strategy and work with the deans, the schools and the individuals to do that, the mechanics haven’t properly been worked out … although the teaching fellows will be a really good link but in terms of operational detail there is an awful lot that needs to be done alongside the dissemination of that… (respondent d) the learning technology literature is awash with arguments that emphasize the importance of staff development when undertaking learning technology implementation (mccartan & hare, 1996; littlejohn & cameron, 1999; mcnaught & kennedy, 2000; brown, 2002; hanson, 2003; timmis, 2003) and this was reflected in the interview responses: people have got to feel equipped to do it, you just feel that you don’t have the time to do it and you don’t need to use it enough so its that need to do it and being given the time and the resources and support to be able to do it and to feel that its of genuine benefit, so the staff development is important… (respondent g) implementing a learning technology strategy 183 the key challenge here is how various staff development processes are able to engage with, change and improve local academic practice. the danger, however, is that the argument for staff development becomes a ‘catch all’ success factor; a seemingly quick fix solution for changing diverse work place practice and culture. oliver and dempster (2003) argue that given the complexity of supporting e-learning, no single model of staff development may be able to engage all staff. for example, centralized models of learning technology staff development maybe too separated from diverse ‘local practice’, accredited courses require staff time allowances for immersive participation and one to one support requires the provision of resource intensive activity. the importance of ‘bridging’ organizational level and individual level cultural signifiers and practices the university of salford lts makes explicit reference to establishing a ‘web presence’ via the vle in terms of pursuing an accessible, flexible and ‘value added’ curriculum. interview respondents displayed extreme variation as to what may transpire from this in terms of academic practice: you can make a strategy which says every individual shall have a web presence and the individual doesn’t have to do anything about it because they just have to supply the information, it might be something very simple… (respondent a) it could be anything from just having a bit of detail about the members of staff that are teaching on the module, maybe a few useful web links, some very minimal stuff it could be a few powerpoints, or it could be a significant part of the course supported or delivered through the vle, they are leaving it open to anybody’s individual interpretation, which gives them more flexibility to the way that it all develops… (respondent b) for staff, it’s enormously challenging for them, it’s potentially threatening, frightening, producing anxiety and fear… (respondent f) the term ‘web presence’ is the symbolic cultural mediator between the strategic goals of the lts and how these are translated into effective learning and teaching practice on the ground. on the one hand, it seemingly offers flexibility and room for interpretation within different cultural contexts but on the other may produce fear, anxiety and increased pressure on academic staff. knight and trowler (2001, p. 43) cite tierney (1989) who advises that university leaders need to ‘use symbols consistent with the local culture’ in seeking to change academic practice: rather than assume a functional view of symbols and a passive view of individuals, we need to reconceptualize culture as an interpretive dynamic whereby the leader’s symbols may or may not be interpreted the way he or she intended. the challenge is to understand how these symbolic forms exist within particular cultural contexts. when undertaking learning technology strategies, policy makers need to be cognizant of the different cultural perspectives and loyalties that exist at different levels of the university organization. alvesson (1993, p. 105) in examining organizational cultural practice within what he refers to as the different ‘social fields’ of the university argues that: 184 b. lisewski there is a discrepancy between the organizational level and the individual level which, however, through its connection with social fields, is collective in character … at the organizational level people’s values and ideals are expressed in a weak form, partly as a matter of compromise. deep values and ideals are more strongly expressed in forms of work, these being largely connected with specific social fields rather than collective/organizationrelated. similarly, trowler and turner (2002, p. 247) argue that developing a strong workgroup culture may mean ‘developing distance from, but interactive tension with, the wider university context’. respondents’ comments on the university of salford culture reflected this picture: it is an individualistic culture … my favourite definition of faculty is a group of people united by a common grievance on car parking … you can’t understand universities and academics by talking in terms of a collective … the analysis of culture at the organizational level would miss the diversity of the various perspectives that exist within and between different parts of the university as well as the even richer diversity of actual individuals… (respondent a) i don’t think that there is such a thing as a university culture, there are lots of sub-cultures … you perceive things from where you are and from where your immediate reference group are, it’s a whole rag bag of cultures ranging from out and out rebellion and anarchy, there is no concept of the university … they are quite complex, not all distinct, some people share things with other sub-cultures but not others… (respondent f) university strategists faced with the challenges of undertaking cultural change in the face of learning technology initiatives need to recognize their organizations as a fragmented ‘domain of factions’ or ‘collection of groups’ (silver, 2003, p. 165) where ‘conflict, uncertainty and the difficulties of response have penetrated the daily lives of academic staff’ and as dynamic ‘multiple cultural configurations’ (alvesson 2002, p. 190) where local practice and its associated meanings and symbols crucially affect how strategic change proposals are understood, received and appropriated within the local workgroup cultures. conclusions the university of salford has adopted a lts in response to external forces such as growing national and global he online provision, perceived student expectations of e-learning availability and the need to provide more accessible and flexible learning and teaching opportunities. however, the challenge remains as to how this more strategic approach will be received and translated into ‘local’ academic cultures and practice on the ‘ground’. the university of salford lts is a staged top–down strategy which emphasizes a ‘web presence’ vision and goal driven processes targeted at increasing curriculum flexibility and accessibility. such strategies have been characterized by trowler, saunders and knight (2003, p. 7) as being ‘technical-rational’ in nature where well-designed interventions aim to cause organizational change. control is directed from the top and mediated within seemingly tightly coupled systems. in such circumstances, knight and trowler (2001, p.14) state that the organization is assumed to act as a: implementing a learning technology strategy 185 co-ordinated unit with a common understanding of objectives, at least in the ideal situation. the assumption is that the outcomes of properly managed change processes are predictable. however, weick (1976, p. 6) describes educational organizations as ‘loosely coupled systems’ which are amenable to ‘localized adaptation’ without affecting the whole organization. thus, the ‘situated character of professional practice’ (trowler & knight, 2002, p. 153) is likely to determine local responses to centralized top–down approaches to the management of change. there may be an inherent contradiction between the university of salford lts that is principally technical-rational in design within an organization that is recognized both externally and internally as one which is ‘disaggregated’ in terms of its cultural form. there is a danger of ‘organizational schizophrenia’ in such circumstances manifesting itself in a mismatch between organizational goals and achievable practice on the ground. the achievement of the university of salford strategy will require a high level of co-operation and mutual understanding between the different central support units, and in relation to, locally based priorities and practice within the faculty and school structures of the university. this may produce a ‘contested space’ where the ‘web presence’ cultural symbolism of policy confronts the reality of practice on the ground. trowler and knight (2002, p. 158) suggest, that in such circumstances, educational development professionals will contribute most to policy interventions ‘by working as consultants with departments’ thereby engaging with specific work group practices’. furthermore, they argue: a university hoping to make a fundamental difference to teaching and learning quality by requiring all departments to introduce new learning environments should anticipate that there will be a considerable range of outcomes and recognize that learning quality might better be improved by encouraging a diversity of innovations. in effect, the challenge for the university of salford lts remains one of implementing a top–down strategic approach within a university characterized as ‘disaggregated into interlocking sub-cultures’. (hannan & silver, 2000, p. 87) this provides a ‘site for contestation’ (webb, 1996, p. 32) where top–down strategy meets bottom–up culture. the chep report (collis & van der wende 2002) cited at the start of this paper argued that increases in learning technology use in he were slow with little underlying change in pedagogic practice. this is not surprising as stigler and hiebert (1998, p. 6) describe teaching as a ‘cultural activity’ where such processes evolve over long periods of time ‘in ways that are consistent with the stable web of beliefs and assumptions that are part of that culture … and that these beliefs serve to maintain the stability of cultural systems over time’. because these teaching beliefs are cultural, then stigler and hiebert argue that they must be understood ‘in relation to the cultural beliefs and assumptions that surround them.’ with regard to learning technology implementation aimed at enhancing teaching and learning practice then, as trowler (2003, p. 146) asserts, if innovations are to be effective there must be ‘mutual understanding’ between 186 b. lisewski policy makers, implementers and practitioners. thus, it is not just a question of establishing ‘success factors’ per se but also whether they are contextualized appropriately within a ‘correct’ characterization of the examined organizational culture. therefore, it becomes essential for strategists to become fully aware of the ‘cultural configuration’ of their organization and the likely response of the practitioners to, in this particular case, strategic learning technology innovations. as kezar and eckel (2002, p. 457) are keen to point out, strategic change will only be successful if such initiatives are ‘culturally coherent or aligned with the culture.’ we need to have a more disfigured view of the policy implementation process within a more realistically characterized notion of he organizational culture. learning technologies cannot be unproblematically applied to improve learning and teaching practice. it is not just a question of putting in place the right ‘success factors’ but rather the need to have clear rationales which are effectively communicated throughout all levels of the organization in conjunction with strategies which correctly configure the cultural landscape and localized teaching and learning practice of the he organization. references alvesson, m. (2002) understanding organizational culture (london, sage). alvesson, m. (1993) cultural perspectives on organizations (cambridge, cambridge university press). becher, a. & trowler, p. (2002) academic tribes and territories (buckingham, srhe/open university press). boys, j. (2002) managed learning environments, joined up systems and the problems of organizational change, jisc report, april. brown, s. (2002) re-engineering the university, open learning, 17, 231–244. clegg, s., hudson, a. & steele, j. (2003) the emperor’s new clothes: globalization and e-learning in higher education, british journal of sociology of education, 24, 39–53. collis, b. & moonen, j. (2002) flexible learning in a digital world, open learning, 17, 217–230. collis, b. & van der wende, m. (2002) models of technology and change: an international comparative survey on the future use of ict in higher education. centre for higher education policy studies report. available online at: http://www.utwente.nl/cheps/documenten/ictrapport.pdf daniel, j. s. (1997) why universities need technology strategies, change, july/august 12–17. dearing report (1997) higher education in a learning society, national committee of inquiry into higher education (london, hmso). available online at: http://www.leeds.ac.uk/educol/ncihe dfes (2003) towards a unified e-learning strategy. consultation document. available online at: http://www.dfes.gov.uk/elearningstrategy/strategy.stm fullan, m. (1993) change forces (london, falmer press). fullan, m. (1999) change forces: the sequel (london, falmer press). fullan, m. (2003) change forces with a vengeance (london, routledgefalmer). garrison, d. r. & anderson, t. (2003) e-learning in the 21st century: a framework for research and practice (london, routledgefalmer). hannan, a. & silver, h. (2000) innovating in higher education (buckingham, srhe/open university press). hanson, j. (2003) perspectives on the strategic implementation of e-learning in australian universities, in: p. boezerooy (ed.), keeping up with your neighbours: ict developments in australian higher education. (ltsn generic centre, alt, surf), 119–126. implementing a learning technology strategy 187 hefce (2003) a consultation on hefce e-learning strategy. available online at: http:// www.hefce.ac.uk/pubs/circlets/2003/cl21_03/cl21_03a.pdf jenkins, m., browne, t. & armitage, s. (2001) management and implementation of vles: a ucisa funded survey, ucisa. available online at: http://www.ucisa.ac.uk/groups/tlig/vle/vlesurvey.pdf keegan, h. (2003) e-learning initiative final report, university of salford, april, internal document. kezar, a. & eckel, p.d. (2002) the effect of institutional culture on change strategies in higher education: universal principles or culturally responsive concepts, journal of higher education, 73 435–459. knight, p. t. & trowler, p. r. (2001) departmental leadership in higher education (buckingham, srhe/open university press). littlejohn, a. & cameron, s. (1999) supporting strategic cultural change: the strathclyde learning technology initiative as a model, alt-j, 7, 64–74. macfarlane report (1993) teaching and learning in an expanding higher education system (the committee of scottish university principals). mccartan, a. & hare, c. (1996) effective institutional change: the impact of some strategic issues in the integrative use of it in teaching and learning, alt-j, 4, 21–28. mcnaught, c. & kennedy, p. (2000) staff development at rmit: bottom–up work serviced by top–down investment and policy, alt-j, 8, 4–18. oliver, m. & dempster, j. (2003) embedding e-learning practices, in: r. blackwell & p. backmore (eds) towards strategic staff development in higher education (buckingham, srhe/open university press). reynolds, j. & saunders, m. (1987) teacher responses to curriculum policy: beyond the ‘delivery metaphor’, in: j. calderhead (ed.) exploring teachers’ thinking (london, cassell), 195–214. rhoades, g. (1997) managed professionals: restructuring academic labor in unionized institutions (albany, ny, university of new york press). saunders, m. (1998) organizational culture: electronic support for occupational learning, journal of computer assisted learning, 14, 170–182. silver, h. (2003) does a university have a culture? studies in higher education, 28, 157–169. smith, e. (2002) strategic factors affecting the uptake in higher education of new and emerging technologies, ltsn generic centre. available online at: http://www.technologiescentre.ac.uk/ newdocs/hedriversfinal.rtf steel, j. & hudson, a. (2001) educational technology in learning and teaching: the perceptions and experiences of teaching staff, innovations in education and teaching international, 38, 103– 111. stigler, j. w. & hiebert, j. (1998) teaching is a cultural activity, american educator, winter, 4–7. stiles, m. (2002) strategic and pedagogical requirements for virtual learning in the context of widening participation. available online at: http://www.inter-disciplinary.net/stiles%20paper.pdf tierney, w. (1989) symbolism and presidential perceptions, the review of higher education, 12, 153–166. timmis, s. (2003) embedding learning technology institutionally. senior management briefing paper key issues series: strategy and policy, joint information systems council (jisc) april. available online at: http://www.jisc.ac.uk/index.cfm?name=project_career trowler, p. r. (2003) education policy (london, routledge). trowler, p. r. & knight, p. t. (2002) exploring the implementation gap: theory and practices in change interventions, in: p. r. trowler (ed.) higher education policy and institutional change (buckingham, srhe/open university press), 142–163. trowler, p. r. & turner, g. h. (2002) exploring the hermeneutic foundations of university life: deaf academics in a hybrid ‘community of practice’, higher education, 43, 227–256. trowler, p., saunders, m. & knight, p. (2003) change thinking, change practices (york, ltsn generic centre). university of salford (2002a) learning and teaching strategy, internal policy document. 188 b. lisewski university of salford (2002b) learning technologies strategy, internal policy document. watson, b. (2003) perspectives on e-learning in australia, p. boezerooy, keeping up with your neighbours: ict developments in australian higher education. ltsn generic centre, alt, surf, 105–118. webb, g. (1996) understanding staff development (buckingham, srhe/open university press). weick, k. (1976) education organizations as loosely coupled systems, administrative science, 21, 1–19. implementing a learning technology strategy: top-down strategy meets bottom-up culture bernard lisewski* grounded theory as an approach to studying students' uses of learning management systems graham alsop and chris tompsett learning technology research group, kingston university email: g.alsop@kingston.ac.uk, c.p.tompsett@kingston.ac.uk this paper presents the first phase of a qualitative study of students' use of a learning management system (lms). a group of students at kingston university with experience of two different systems were afforded the opportunity to study the relationship between the interface to an lms and the usability of the system. a 'grounded theory' methodology (glaser and strauss, 1967) was selected to model the students' framework, as independently as possible from the views of other stakeholders in the lms. the approach still offered the potential for generalization despite working with a small group of students. in grounded theory, data collection and theory-building drive each other reciprocally until a stable 'theory' is reached. in an ideal world this research would be reported at the end of the whole process but it is considered that there is more value in producing less definitive results at this stage rather than a complete theory in one year's time (oliver, 2000). even at this stage of the research, it is clear that the students' conception of an lms is categorically different from that held by the other stakeholders responsible for implementing the system. identification of these differences and integration of some findings into future developments are critical even if, in the view of the researchers, further development of the 'theory' is required the findings from this phase validate the use of a grounded approach. they suggest that systemic issues regarding the integration of educational activities with other aspects of everyday life are fundamentally more important than issues in interface design. the students make effective use of remote access facilities but only where this offers a clear comparison with activities that already form part of studying at university. such comparisons are pragmatic and are seldom defined in terms of 'educational' benefit. introduction the study was designed to examine issues relevant to planning the full-scale implementation of the learning management system (lms) across the university. two aspects of 63 graham alsop and chris tompsett grounded theory for studying students' uses of learning management systems usability were of particular interest: whether the system could be improved through changes in the design of the interface and whether a student's ability to use an lms successfully was determined by technical abilities developed either at, or before, university. students from two courses had experience of using both webct and blackboard. a study using these two groups offered the opportunity to investigate how differences in the interface had affected the usability of the system for students with a relatively high degree of technical competence. an lms offers different benefits to the various stakeholders in the system. for management an lms can provide operational benefits, such as integration with other management systems (student records and finance, etc.) as well as strategic opportunities, for example to increase student numbers by decreasing the time spent by each student within the university. such benefits do not necessarily offer educational advantage to the students. for a student an lms would appear to offer few technical advantages beyond those that are available free through other internet providers offering support for more generic online communities. both webct and blackboard provided similar functions as learning environments accessed through a web portal. each offered links to a number of integrated functions supporting document management, both synchronous and asynchronous communication, and integration with email services. any educational advantages were essentially mediated by the design of the interface. the arrangement and labelling of icons and links to the various facilities controls the students' expectations of what activities they could be engaged in. if the learning management systems were substantially the same, then differences would only be realized if the students could anticipate the benefits that might be available and could successfully navigate to those different services. simple differences in the usability of the interface could provide significant educational gains. the researchers' own experiences with blackboard suggested that there was considerable potential for redesign. this research focused on students studying geographical information systems (gis) in their final year of two courses, either for a b.sc. or an hnd. these students had all used blackboard during 2001 and webct during 1999-2000. webct and blackboard were both offered with the normal suite of services from within the university. only blackboard was made available through remote access (with password protection). although the students were supported by an lms at each stage, the courses were still structured as conventional undergraduate courses with attendance required for lectures and other workshop activities. methodology the researchers were aware that their perception of what was critical in the design of an lms might not be seen as important by the students. the selection of a methodology that was relatively immune to the expectations of the researchers was critical. selection the first stage in the research was the selection of a methodology appropriate to the outcomes that were required and the constraints of the particular study. the most critical requirement was the importance of capturing the students' own perceptions and understanding of the systems that they were using. 64 alt-j volume 10 number 2 many research methodologies expect the research to be conducted within an objective, conceptual model that is framed by others external to the subject group. several stakeholders, other than the students, have an interest in the performance of the lms: university managers, system designers, system implementers, course lecturers and, of course, ourselves as researchers. research that is based on an external framework may well provide evidence that a system is not working but will seldom provide answers as to why or how the system fails to work. this requirement implied that a qualitative approach, grounded in the students' own conceptual frameworks, would be essential. the second requirement was that the methodology had to support the potential of building a theory about what was being studied. this would enable the results of the research to inform the next stage of the development in the use of the lms. this requirement would exclude any methodologies, such as ethnography, which did not conform to this. a third class of methodologies was excluded by the constraints of the study. some qualitative approaches use an initial subject-centred qualitative phase as grounds to generate relevant questions for a larger statistical analysis. the timing of the research in two phases would not produce any results over the time-scale available as all the students with experience of both systems would have left by the end of the year and the group would not have been large enough for a statistical analysis. our principle interest lay in providing a rich description of the range and variety of student perceptions and the factors that could be altered to produce different behaviours. the overall research model selected was grounded theory (glaser and strauss, 1967; glaser 1978; strauss and corbin, 1990; miles and huberman, 1994). a more detailed discussion of this selection process is provided elsewhere (alsop and tompsett, 2002). grounded theory the grounded theory method requires that the full research process is grounded in the data that is collected. as with most qualitative methods it accepts that the researcher cannot remain external to the collection and interpretation of data but is inherently bound up in the process of research. the process of collection and interpretation is cyclical, with low-level analysis of data providing grounds for simple levels of interpretation, which then provide the basis for reinterpretation of existing data and collection of additional data. it is an iterative process that focuses on the subject's perspective but also accepts that the researchers cannot approach the subject without bias or personal viewpoints. the implication that the results will be dependent on the researcher's implicit involvement is managed through careful recording of the process of interpretation, where the researcher's initial interests, subsequent comments and possible interpretations are treated not just as information external to the analysis, but are recorded as data within the project itself. the possibility that different researchers might not agree on results is as a logical consequence of a constructivist epistemology. the rigour of any resulting 'theory' is established by the ability of other researchers to audit the stepwise sequence of interpretation, whilst recognizing that individual bias and interpretation would not necessarily be the same. the degree to which the researchers share (or should share) the same values and frameworks will determine the extent to which different researchers would accept the same conclusions. 65 graham alsop and chris tompsett grounded theory for studying students' uses of learning management systems the development of a theory is cyclical too, using existing data to build a theory and using the developing theory to guide data collection, increasing the depth of interpretation and the coverage of the theory across subjects. the first stage is to generate a catalogue of the terms and concepts used by the subjects (termed 'open coding')this catalogue is then used to generate an 'axial model' (strauss and corbin, 1990: 96), a time-sequenced framework that establishes the interrelationships between the concepts within the open coding. this develops the interpretation of the subjects' perception of the problems that were being solved, for example the conditions controlling what was possible, the choices that were open to them and their assessment of what was successful and what was not. when this appears to be effective in structuring the data collected this model is then summarized by the identification of a single bounding concept, termed the 'core category'. this core category encompasses the framework within which all of the subjects' perceptions can be included. in reality there is a constant cycle between the last two stages; the intention is to develop a theory for which the core category provides a summary that allows all the data to be interpreted within the axial model and hence the core category. the axial model and core category are validated as a theory through a recursive process termed 'theoretical sampling'. in this phase concepts and issues that are already relevant in the current axial model are used to search for new cases to be integrated. the new data that is collected may well introduce new concepts or variables that enrich the open coding stage, provide changes and complexity to the axial model or even require a reconsideration of the core category. when this process stops, a process of uncertain length, the research is considered to be complete and the model is said to have reached 'theoretical saturation'. the outline of the process suggests that existing literature in the field is ignored but this would be illogical if the theory is, itself, to have predictive value. concepts and models from literature are treated here as frameworks that have been created by other stakeholders outside the immediate focus of the research. there is no direct implication that they should apply to the conceptual framework used by subjects. existing literature is treated as a further source of material to be used in developing theoretical saturation. awareness of possible frameworks from the literature, termed 'theoretical sensitivity', allows possible issues to suggest areas for data collection, as long as there is sufficient evidence from what has been collected already. the first phase this paper covers the first stages of the analysis: the open coding, early axial coding and considerations for the core category. whilst the research process is not complete, the convergence of the cyclical method to a stable theory offers the opportunity to comment on aspects of the developing theory at a stage at which those aspects become sufficiently well defined to be of significance. in this case the requirement to establish a definitive theory must be balanced against the potential utility of that information to stakeholders. even at this stage of the research, it is clear that the students' conceptions of an lms are categorically different from those held by the other stakeholders in the system. a delay in publication to develop and refine a 'theory' will have little value if decision-makers proceed to implement large-scale systems unaware that their expectations of how students will use the system is far from reality. 66 alt-} volume 10 number 2 data collection the particular method for generating data was derived from an earlier qualitative study (bliss and ogborn, 1977) that used semi-structured interviews to collect and analyse the experiences of first-year physics undergraduates. each session was carried out during time allocated to a normal lecture or workshop. an introduction was given to the purpose of the research and why the students had been chosen. the method was described as a technique for collecting their approaches to using the lms that was designed to avoid interference from the views of other stakeholders in the university's system. students were given details of the process by which their contributions would remain anonymous and were allowed the option of withdrawing from the process. students were then asked to write an account, with as much detail as they could remember, of the occasion on which the use of one particular lms had been 'the most rewarding educational experience'. as they completed their own individual account they were asked to summarize this account with a single word or short phrase. as four or five students finished they were formed into a group and completed the first phase by reviewing, without discussion, the accounts of the other students in the group, summarizing each in turn. they then discussed the accounts between themselves to agree on a common word or phrase to summarize the set of accounts within the group. this is less constrained than a semi-structured interview, but earlier experiments with a similar topic by one of the researchers has demonstrated that it can produce remarkable consistency between individuals and groups. each of the groups then repeated the cycle with a different starter question the occasion on which the use of one particular lms had been 'the worst educational experience'. in general the method worked as anticipated, though not all students confined themselves to single examples. if time allowed, students were then asked to contribute in two further ways to building the model. the first asked for a specific comparison of the different interfaces. the second asked them to identify any issues to which they or their group had failed to refer but which might indicate key issues that should be included in the model within the data collection process. this open structure to the collection of qualitative data was designed to elicit responses that would be framed within language understood by other students. descriptions written directly for us could be filtered or adapted to make them appropriate for lecturers. requesting a narrative account supported a simpler transfer to an axial model. the request for individual summaries checks that the accounts make sense to the students and, with a collective discussion, provides a starting point for developing a core category rooted in the subjects' perceptions. the intention was for students to produce an account that was acceptable or understandable by the student community. the data collection process was supplemented with field notes from the researchers throughout the process. it is not possible to collect data and interpret it in separate phases, as interpretation of what is collected cannot be 'turned off': for example the comment 'not exclusively about the technology' was recorded after the first session of data collection even before transcription began. contemporaneous notes on ideas that occur provide a mechanism for identifying these interpretations and, by recognizing the source, of tracking and auditing the influence of individual bias on the research. notes are also kept on any _ graham alsop and chris tompsett grounded theory for studying students' uses of learning management systems extraneous issues affecting data collection. in the first session, for example, it was noted that the offer of tea and coffee to students appeared to remove tension. the discussion below focuses on the language used by the students in these accounts. issues of 'theoretical sensitivity' were excluded at this stage. open coding the analysis of the full sample collected so far (49 accounts) was conducted using the software package nudist 4. this provided effective facilities for tracking the progressive searching and coding of accounts, the sorting and resorting of coding groups along with the parallel recording of coding notes. some of the students followed closely the request to describe a single event with some detail whilst others provided wider but shallower accounts. a typical account (transcribed with minor changes for legibility but with original spellings and expressions recorded) is presented in figure 1. when using blackboard, groups can be set up so you can communicate with people in group work. this involves talking to the people directly using chat or emailing them, but also being able to exchange files.this is very useful especially as people work in different places and times to others. it also enables people to communicate easily without having t o find the id numbers. it also means people can continue t o develop work without actually having to meet up to exchange files, which is very useful for people who don't need t o come into the building. personal summary: communication with others summaries of this account by other members of group: communication facilities, easy communication, communication tool group summary (combining all four accounts): communication figure i: a typical student account perhaps the best experience of blackboard i have had is that i can get my lecture notes /course materials from home (internet) which has been useful when i have not managed t o get a print-out of the materials when in university. this remote access proved useful when i needed some lecture notes to do a piece of work and i was able t o get hold of, and print the course materials out at home thus meaning i didn't have to go into university to do it personal summary: access from home summaries of this account by other members of group: remote access of notes, copy and print the notes, lecture notes group summary (combining all four accounts): communication figure 2: a further example of a student account 68 alt-j volume 10 number 2 the consistency within the summaries of the account in figure 1 was repeated in almost all cases and gave a clear indication that the individual accounts were well understood within the student community (see figure 2 for an additional example). the consistency of language and terminology used by the students in their individual accounts also provided a strong indication that the particular model of data collection was working effectively. even the simplest reading of the accounts reveals a terminology and language focused on being able to (or alternatively failing to) access resources in a remote-access mode when the students needed to or preferred to. one researcher specifically notes at this stage: 'stories seem to be about flexibility and choice students are working in semi-distance mode.' within this common theme, the different resources available seem equally significant (for example, lecture notes, assignment details, etc.), and the 'reasons' for access were varied. coding and notes interpretation (and recording of notes) inevitably occurs as soon as a researcher has access to the data (as with the comment above). open coding progresses from that point through a structured approach, interleaving analysis and coding of the data collected with recording of notes and commentary. interpretative comments were attributed to a specific point in the analysis and elaborated using explicit terminology from the cases (for example, 'several students use "choice" words need, have to, useful, unable, wasting, access (choice), repair (missing lectures)'). this generated, in turn, a further analysis of the cases for example a search through all the cases for examples where that specific terminology is used and extending the classification to a wider set of cases than those that triggered the initial comment. it was then possible to re-examine these cases in more detail, or juxtapose the new coding with a previous classification or to review the cases where 'choice' did not seem to play a role. absence of specific language, where the researcher's personal expectation was not met in reviewing text, was also considered a valid starting point for search, such as, 'no student mentions learning specifically why?' in such cases searches with no results were also recorded (as with: 'august 1st 2001 searched for "learning" nothing found; august 15th 2001 searched for "learn*" nothing found'). at this stage the study was most remarkable for the absence of significant categories of language. little comment or reference was made on specific issues that were assumed to be important in the original design, either interface issues or the relevance of technical skills. remote access, the result of a policy decision at kingston university, overrode any other consideration of implementation or design. what the data did not say as the students' accounts failed to produce anticipated commentary on either aspect of usability, careful consideration was given to whether this resulted from the design itself. as commented above, the positive and negative accounts (described in what follows as 'open' accounts), omitted almost all terms or issues relating directly to human-computer interface design; only one, triggered by the 'worst experience' question, could be interpreted as a lack of closure (commenting on a lack of confirmation after an assignment was delivered electronically through blackboard). the extent to which this could be a consequence of the questions that were asked was a consideration but was rejected after analysis of some of 69 graham a/sof> and chris tompsett grounded theory for studying students' uses of learning management systems the additional comments that were requested. when asked for comments comparing the two systems, some of the students readily demonstrated an understanding of humancomputer interface issues (see figure 3). blackboard is much better than webct, it has the modules separate in buttons, calendar t o show the submission date for the assignments or bb [blackboard] better than webct visually a mess bb very ordered figure 3: examples of students' comments addressing human<omputer interface issues clearly students are aware of the impact of an interface and can express preferences. there is, of course, indirect evidence from their accounts that they did use the interfaces successfully. the disregard of specific interface issues within the open accounts provides strong evidence that changes in interface design are of low priority for the students at this stage in the use of the lms. arguing that the research process has failed to capture the relevance of the interface would be to devalue what the students choose to report apparently devaluing a comment if it is not within a researcher's chosen category. the second aspect of usability, understanding the relationship between the students' technical abilities and their use of an lms, was also only reflected to a minor extent in the accounts and is not yet sufficient to feature in our models. the justification for this position follows the same structure as before but is too detailed to present in this paper the discussion of these two specific issues does not imply that technical ability and interface design do not have an influence on the usability of the system such a view would clearly contradict evidence from the general field of hci design. what is called into question is the relative significance of these issues, from the students' point of view. the open coding reveals a significant divergence in the issues and language used by the students, compared with the language anticipated in the original design of the research (usability) or the researchers' own personal interests in the potential of technology to improve the learning process. the openness of the research process to capturing students' views has redefined the focus of this research to what appears critical to the students, the decision process by which they choose to exploit the facilities offered by an lms. to view this as a weakness in the research design would be to ignore what is fundamental to the principles of grounded theory methodology. the immunity of the data collection process to the expectations of the researchers is valuable evidence of a well 'grounded' study.. this divergence allows the range of subjects available to the study to be increased and studied over a longer period of time, but does not imply a change in methodology. indeed, the effectiveness of a fully 'grounded' approach to capture the views of users of an lms seems clearly established. 70 alt-j volume 10 number 2 axial model a storyline is constructed to provide an account of how the individual in each case perceives the problems that they encounter. this includes the causal conditions that led to the problem, the particular context of each problem, the intervening events and behaviours that provide a dynamic sense to the case, the strategies applied by the individuals themselves and the eventual outcomes (strauss and corbin, 1990: 61-2). one possible formulation of the account from figure 1, represented here without a strict typology of concepts for simplicity, could be as in figure 4. figure 4: a formulation of the account in figure i with further analysis of similar accounts a model is developed that provides more subtlety to the analysis and an understanding of the consequences that result when options fail. figure 5, for example, includes an assessment to consider the possibility that a virtual meeting is not sufficient. even without a more rigorous approach there are evident gaps in the network that need to be explored (theoretical sampling). what, for example, determines the choice of mode for a virtual meeting? or what could be the consequence, to match with 'avoid coming into university', when a real meeting cannot be organized? completing an axial model to integrate all student accounts in the same, single model is the key step in 'understanding' the students' behaviour. there will be contexts in which students have little opportunity to make decisions that affect the consequences for themselves. equally there will be contexts and circumstances when students can make choices that they anticipate will produce greater benefits in their own terms. understanding the relationships between these circumstances, actions and benefits helps us to understand why students perceive some patterns of behaviour as the rational pattern to choose. understanding these relationships can, ideally, allow the university to make _ graham alsop and chris tompsett grounded theory for studying students' uses of learning management systems figure 5: a formulation of the account incorporating an assessment of sufficiency changes to the context or circumstances in which the students find themselves, so that they change their decision. core category a core category should provide an overriding conceptual framework within which the decisions taken in each case can appear rational to the individuals (strauss and corbin, 1990:116). the terms that were identified by students as summaries in the positive accounts were intended to provide a catalogue of concepts that could be considered as the candidate phenomena for a core category, with reinforcing evidence taken from the summaries of the negative accounts. this has not directly applied to the accounts that have been collected. within the positive accounts the concept of 'accessibility' (including variations such as working from home or remote access) was most consistently used and almost all the negative accounts reflected failures of the system to support this. the summary terms used by the students appear to be chosen in terms of their instrumental value, identifying the significant change in circumstances that has most influence on outcome (as also with 'communication' in the example used above). remote access did not confer direct benefits to the students. it was the way in which that access is exploited that was of benefit. this suggested that the summaries should primarily be treated as intervening conditions rather than as possible core categories. where the system works as students expect, the benefits cover a wide range of issues: avoiding 72 alt-} volume 10 number 2 journeys into university (as in the example), the direct saving of time, starting academic work sooner, catching up on study when behind, being aware of new and relevant information, etc. it was these stated benefits that became potential core categories. the benefits themselves, as taken from the accounts of the students, were too varied to provide a single 'core phenomenon' and generally could not be expected to show such consistency. a stated benefit, such as 'avoiding travel to the university', could be stated in a more general or specific form or be seen to have direct consequences that occur in other accounts, such as 'saving time' or 'having more time to work'. the development of an axial model and core category is not yet complete. it is clear that the relationships between the benefits can be constructed into a limited set of possible lattice structures which closely limit the range of what could act as a core category (see figure 6, for example). strauss and corbin (1990) suggest that it is more effective to produce alternative models where the balance between more than one type of benefit occurs, but this does not appear to be the case in this study. please note that some of the connections may not be obvious this is a representation of the students' perceptions and not our own interpretation. integrating study with life j choosing • where to work | avoiding i unproductive i i meetings j figure 6: a possible lattice structure catching up 1 73 graham alsop and chris tompsett grounded theory for studying students' uses of learning management systems as would be predicted from the minimal impact of usability in open coding, the network above makes little reference to either technical skills or to aspects of interface design. the actions and approaches to comparing alternative solutions are functional in terms of life management and ignore educational issues of effectiveness, learning style, etc. even where educational terms are used (such as revision, catching up) the benefits are realized in more functional life-style gains. an initial reaction to the students' use (or non-use) of the lms might be to suggest that students are merely using it as a passive filing or messaging system, but this would fail to reflect that the students are actively engaged in using (or attempting to use) the different systems to manage their learning. remote access, rather than on-campus access, provides them with a choice of how and where they can study. it is interesting to note from this analysis that these students have provided a more critical assessment than most students who work within a full distance-learning framework. these students choose to meet their objectives by working remotely, by following a traditional model of full attendance at the university, or by managing a combination of both. from the evidence in the accounts there is no simple solution; costs, benefits and reliability, rated in a number of different ways, have to be weighed against each other. continuing research this study is now continuing within the revised focus identified by the students. with the continuing promotion of the lms across the university, the range of courses to be included covers six faculties with some cohorts of more than 400. changes in the lms software and its use by staff should not invalidate the existing findings it should provide access to actual cases rather than hypothetical cases in the progress towards theoretical saturation. the overall consistency in the accounts that have been collected will now support consideration of 'theoretical sensitivity' to provide additional direction to this phase of the study. two aspects will need to be considered in introducing existing research, the issues that are considered and the findings themselves. issues that are of potential interest can be identified across a range of studies. bacsich, ash, boniwell, kaplan, mardell and caven-atack (1999) identify the same financial issues but suggest (p. 47) that students are not aware of the savings. our study suggests that students are actively balancing the finance and time in how they choose to study. the 4e model (collis and pals, 2000; sikkel, gommer and van der veen, 2000) also provides a useful summary of issues. although the model includes four 'dimensions', the resulting measures are not at the same level of granularity as used by students in the current study. the lack of similarity in findings, as opposed to issues, is unsurprising in view of the nature of the research. both survey techniques and mathematical modelling will cover the same aspects of a problem but produce results on a different scale that are relevant to other stakeholders but not to individual students. more challenging research comes, naturally, from activity theory (at) (nardi, 1996a), since the epistemological model is consistent, even if grounded theory is considered by those in at to lack 'underlying principles' (nardi, 1996b: 236). there are clear parallels 74 alt-j volume 10 number 2 between the fundamental concepts of at and the structure of an axial model. the difference arises, of course, in whether accounts of the students must fit with at, or whether they actually do. assuming that the accounts must fit appears to tackle the underlying principle of grounded theory. if the model from at fits, we may unwittingly have provided an investigative model (at this stage at least) for researchers who start with at as an underlying first principle. the role for at, perhaps more specifically as functional pragmatics (tetscher, meyer, wodack and vetter, 2000), will be to provide clarity to the potential meanings of goals, aims and actions where the narrative accounts appear unclear and where further discussion with the subjects is not possible. in this context there are other theoretical approaches for text analysis that claim equivalent respectability, even if from different principles. tetscher et al. (pp. 227-36) provide a useful summary with particular attention paid to underlying principles. conclusions the research from this study is published at this early stage for three reasons that are as relevant, or more relevant, at this stage as they will be at the end of the process. firstly, a research process that set out to be grounded in the perceptions of the student community has demonstrated, by confounding any expectations of the researchers, that it is able to achieve that objective. secondly, a grounded approach has revealed that the underlying decision process used by students to determine how they use an lms is considerably more complex than reported in positivist accounts. the categories used, and level of granularity with which students consider them, are not those that arise within the design of most educational technology. students make case-by-case decisions whether to replace existing university-based activities with equivalent activities based off-campus. if an lms is to support change in student behaviour, the models used by designers must change. such models must include an operational model of how students currently evaluate their choices in learning at university. it must also ensure that the educational benefits that are anticipated translate into benefits that can be recognized by the students. thirdly, decisions that extend the use of an lms in further and higher education are being taken daily, in a process that appears unstoppable. it is critical that we understand the impact that these changes will have on students' learning before the possibility of revoking any decision is overridden by the scale over which the change would need to take place. this argument appears similar to patton's utilization-focused evaluation (oliver, 2000) but there is a potential conflict between the design of this research and the values of the stakeholders who take the decision. will those who promote the use of an lms be prepared to accept research that suggests that their models are not valid? we hope so! entwistle provides a detailed analysis of some approaches to designing courses to 'encourage student learning directed towards personal understanding' (entwistle, 2000:1). the models and constructs that he reviews are significantly different from those that are used by these students. if we are to include both sets of models into the design of online courses then we have much to learn about how students achieve this. the answers on this aspect, however, must depend on the further stages in the research. 75 graham a/sop and chris tompsett grounded theory for studying students' uses of learning management systems references alsop, g. and tompsett, c. p. (2002), 'a grounded theory investigation of students' perceptions of learning management systems', in proceedings of the 9th improving student learning symposium, 2001, oxford: oxford centre for staff and learning development. bacsich, p., ash, c , boniwell, k., kaplan, l., mardell, j. and caven-atack, a. (1999), the costs of networked learning, sheffield hallam university, school of computing and management sciences, http://www.shu.ac.uk/cnl/report1.html. blackboard inc. (2001), http://company.blackboard.com. site last updated 4 january 2001. bliss, j. and ogborn, j. (1977), students' reactions to undergraduate science, london: heinemann. collis, b. and pals, n. (2000), a model for predicting and individual's use of a telematics application for a learning-related purpose', international journal of educational telecommunications, 6 (1), 63-103. entwistle n. (2000), 'constructive alignment to improve the quality of learning in higher education', paper presented at the dutch educational research conference, university of leiden, 24 may 2000. glaser, b. g. (1978), theoretical sensitivity, mill valley, ca: sociology press. glaser, b. g. and strauss a. l. (1967), the discovery of grounded theory: strategies for qualitative research, london: weidenfeld and nicholson. miles, m. b. and huberman a. m. (1994), qualitative data analysis: an expanded source book, london: sage. nardi, b. a. (ed.) (1996a), context and consciousness: activity theory and humancomputer interaction, cambridge, ma: mit press. nardi, b. a. (1996b), 'some reflections on the application of activity theory', in b. a. nardi (ed.), context and consciousness: activity theory and human-computer interaction, cambridge, ma: mit press, 235-46. oliver, m. (2000), an introduction to the evaluation of learning technology, educational technology and society, 3 (4), http://ifets.ieee.org/periodical/vol_4_2000/intro.html. sikkel, k., gommer, l. and van der veen, j. (2000), using shared workspaces in higher education, doc 00-30 dinkel, institute university of entchede, netherlands. strauss, a. l. and corbin j. (1990), basics of qualitative research: grounded theory procedures and techniques, london: sage. tetscher, s., meyer, m., wodack, r. and vetter, e. (2000), methods of text and discourse analysis, london: sage. webct, inc. (2002), http://www.webct.com/company. 76 simulation modelling: educational development roles for learning technologists david riley imperial college london email: d.riley@imperial.ac.uk simulation modelling was in the mainstream of cal development in the 1980s when the late david squires introduced this author to the dynamic modelling system. since those early days, it seems that simulation modelling has drifted into a learning technology backwater to become a member of laurillard's underutilized, 'adaptive and productive' media. referring to her conversational framework, laurillard constructs a pedagogic case for modelling as a productive student activity but provides few references to current practice and available resources. this paper seeks to complement her account by highlighting the pioneering initiatives of the computers in the curriculum project and more recent developments in systems modelling within geographic and business education. the latter include improvements to system dynamics modelling programs such as stella®, the publication of introductory textbooks, and the emergence of online resources. the paper indicates several ways in which modelling activities may be approached and identifies some educational development roles for learning technologists. the paper concludes by advocating simulation modelling as an exemplary use of learning technologies one that realizes their creative-transformative potential. foreword i first encountered david squires at hertfordshire's advisory unit for computer based education in 1981, little knowing that we were destined to share an office at chelsea college and to work for the pioneering computers in the curriculum project (cic). on joining chelsea's educational computing unit in 1983, david soon introduced me to his work with jon ogborn on the dynamic modelling system and their combined influences are apparent still, both in my educational research and in this paper. david was a pioneer at heart with a laid-back enthusiasm for innovative ideas and the educational opportunities afforded by microcomputers. during the 1980s, he managed and directed a variety of educational projects encompassing simulations, games, modelling 54 alt-j volume 10 number 3 programs, science microworlds and a collaborative environmental database pretty much the full gamut of what laurillard terms the adaptive and productive media. despite the range of david's interests and the ambition of his software development projects, he brought to his work a characteristic good humour, never being overwhelmed by his increasing responsibilities as he progressed from being a co-director of cic to become a lecturer and, eventually, a professor of king's college london. david's success may be attributable to his easy-going manner and a combination of a dayto-day pragmatism with a long-term educational vision. this proved to be a potent mix and one that endeared him to many of his colleagues and friends. today, we miss his personal and professional presence and can only wonder what he might have gone on to achieve. we look to a new generation of learning technologists for his natural successors. time for change educational games, simulations, microworlds and modelling programs were in the mainstream of 1980s computer-assisted learning (cal) when students were expected to engage and participate, to compete and collaborate, and to create and produce within their theoretical fields. since then, the technologies have improved by leaps and bounds whilst the adaptive and productive media seemingly have drifted into an educational backwater. these media receive only passing mention in the alt overview of learning technologies (seale and rius-riu, 2001), and laurillard (2002), suggests that they are underdeveloped and underutilized in higher education. it is as though the users of learning technologies have turned their backs on students as creators and producers, in favour of enhanced information delivery and ease of administration. to correct this apparent mission drift, it is suggested that learning technologists raise staff awareness of simulation modelling with programs like stella (see figure 1), that support both modelling activities by students and the development of educational simulations. figure i: a stella® simulation control panel assembled from drag and drop components, as is the underlying systems model ('population and resources' model provided with stella 7 by high performance systems') dashboard 2 : n a i r t resources 2013 t e n unfurl the model n 9*fhi aion n t i 1 ij / i '1 ! i » i -q-m view the model... 55 david riky simulation modelling: educational development roles for learning technologists a pedagogic case for the productive media is made by laurillard, who points to their unique potential for letting students create and produce their own mathematical and formal models of the world (laurillard, 2002). computer simulations and games allow students to interact with pre-defined dynamic models but only microworlds and modelling programs allow students readily to create such models for themselves. this paper seeks to illustrate this notion of students as the authors of models and seeks to complement laurillard's account by highlighting: • the 1980s modelling initiatives of david squires and the computers in the curriculum project; • subsequent developments in the learning and teaching of systems modelling, with special reference to geographic and business education; • educational development roles for learning technologists interested in promoting and supporting simulation modelling. this quick dash through time, and the world of modelling, is inevitably partial and illustrative rather than comprehensive and definitive. the paper introduces just two models and modelling programs prior to a discussion of approaches to classroom modelling and educational development roles for learning technologists. an extensive list of references has been provided to compensate for this selectivity, and for the lack of case studies or of evaluations of classroom use. in the latter respect, modelling provides particular challenges. modelling is a creative and relatively open-ended activity, there are no paperbased equivalents, the pedagogy of modelling is weakly developed across the curriculum, and modelling is still a comparatively rare educational activity. indeed, one hopes that this paper will attract learning technologists to these unexplored pedagogic frontiers and to seek out covert instances of modelling within their own institutions. modelling in the 1980s: a cic perspective one cannot write about modelling in the uk in the 1980s without mention of david squires. he spent much of the 1980s as a co-director of the computers in the curriculum project where he was an international figure in the development of games, simulations, microworlds and modelling software for science curricula (squires and mcdougall, 1986; watson, 1987). david championed the educational use of microworlds and modelling programs and, to take just one example, worked with jon ogborn to produce the dynamic modelling system (dms), which is described below. modelling with dms dms enabled students to create computer models without first having to learn calculus and a programming language such as fortran (lai, 1987; riley, 1986; wong, 1987). as with early spreadsheets, or predecessors such as modl (hartley and lewis, 1984; hartley, 1984), students could use dms to explore the theories of their subject fields, whilst the computer performed the repetitive numerical calculations. dms was significant because it widened student, and teacher, access to simulation modelling as a learning activity. providing access to a novel classroom activity was not sufficient however, and it was recognized that packs of illustrative models and guidance on their use would be needed. hence, david directed the publication of sets of illustrative 56 alt-j volume 10 number 3 models for sixth-form and undergraduate physics, chemistry, biology, economics and geography. an example of an elementary demographic model is shown in figure 2. such a model was intended to serve as a starting point for modelling activities in the life or social sciences where students were to investigate the growth of plant, insect, animal or human populations. the models were meant to be simple so that students could readily comprehend, criticize and improve upon them. some illustrative sequences of progressively more complex models were included in the dms packs to show how modelling activities might proceed through processes of elaboration and refinement. figure 2: an elementary demographic model as viewed in the four display pads of the dynamic modelling system: any two of the four pads could be displayed simultaneously on the low resolution screens of 8-bit microcomputers xndsbiktxs-semxs rant 1 ror list 10m l t m 112« 1150 lira 12 la 1241 12 t 1301 figure 2 reproduces the display pads of dms with a model of annual population growth resulting from constant numbers of births and deaths. dms calculated the results from the initial values and the model's difference equations, with each iteration of the model standing for the passage of a short time interval. models were expressed as mathematical 'difference equations' written in the basic programming language, thus a statement like: 'pop = pop + i n c was read as: 'let the current size of the population become equal to the previous size of the population plus the increase in population.' once students were familiar with dms, they could adapt and extend the models. for example, the above model could be improved by introducing crude birth and death rates, in place of the fixed annual increments. the inclusion of immigration and emigration rates could add further dimensions to a model of, say, a period of industrial growth in victorian manchester or 1980s hong kong. students could gather empirical data on critical rates of change, investigate their model's sensitivity to different rates of change, and relate their findings to known or anticipated trends in real-life populations. using learning technologies in this way was educationally attractive as students were engaged in researching, authoring and evaluating models. for instance, students using dms might have to: 57 david riley simulation modelling: educational development roles for learning technologists • locate suitable numeric data, whilst coping with different units and systems of measurement; • assess the relevance of available data and/or make reasonable estimates of values; • establish empirical or theoretical relationships between variables; • express these relationships logically/mathematically; • make sense of tables and graphs of results; • debug or refine their models until the results were satisfactory; • use the model to solve a problem or to arrive at a forecast. students typically worked in groups, and so modelling served as a stimulus for valuable communicative and collaborative activities. furthermore, many hours of productive learning could be stimulated by a well-designed activity whilst the development of starter models and model fragments was a relatively quick task for teachers. if time was pressing, teachers could 'model' or demonstrate the process of modelling and engage students in whole-class discussions as a model was created, tested and applied to a problem. modelling with stella the system dynamics modelling program stella® was launched on the apple macintosh in 1985 and it immediately impressed david squires and fellow cic staff. stella performed a similar function to dms but possessed a graphical interface based on a 'visual plumbing metaphor', as shown in figure 3. the cloud symbols indicate the system boundary and serve as infinite sources or sinks of the 'fluid' flowing through the system. the diagram is animated when the model is run, showing fluid levels rising and falling in the tank, whilst pointers on the dials of taps and the 'limit_to_growth_fraction' converter move back and forth in response to changing flows and numeric values. hence, the acronym stella: structural thinking experiential learning laboratory with animation. potential unit growth rate artual rate of growth limit to growth fraction carrying capacity figure 3: stella diagram of a simple 'limits to growth'model: with annotations identifying the positive (+ve) and negative (-ve) feedback loops 58 alt-j vo/ume / 0 number 3 system dynamics stories system dynamics models show how system behaviour is determined by system structure conceptualized as accumulations, rates of change, delays and closed loops exhibiting positive or negative feedback (forrester, 1961, 1968). for instance, the limits to growth model exhibits the classic pattern of 's-shaped growth' shown in figure 4, a dynamic perceivable in situations as diverse as the colonization by plants of a volcanic island or the saturation of the market for mobile telephones. according to the tenets of system dynamics, a system's behaviour is determined by the internal relationships or structure, rather than by events in the external environment, unless the system is pushed beyond the limits within which it can function. 1 : population size 1 : 10d0. gpoo 2 : actual rate o f growth 3 : a s t r a l rate of decay so1 3d s-shaped growth dynamic 45 so time figure 4: stella graph of the 'limits to growth' model figure 4 'tells the story' of s-shaped growth, an 'archetypal' pattern of behaviour in some systems (senge, 1990). an initially low population (for instance, of plants or mobile telephones) possesses a slow rate of growth but this begins to increase as 'success breeds success' and the reinforcing effects of positive feedback dominate. the population grows most quickly when the difference between the actual rates of growth and decay (death or obsolescence) is at its greatest. from that point onwards, population growth is inhibited as a biophysical or socio-economic limit is approached and the counterbalancing effects of negative feedback become stronger. the population reaches a high and steady level once the growth and decay rates balance and, in effect, there is a saturation of the ecological or market niche at a level approaching the potential carrying capacity (of, say, majority ownership of mobile telephones). constructing stella models stella models are assembled diagrammatically by arranging and connecting symbolic: pipes and taps (flows), tanks (stocks or accumulations), wires (carrying information about 59 david riley simulation modelling: educational development roles for learning technologists causal relationships) and converters (information or equation holders). this construction kit approach to model-building serves several purposes, it: • encourages users to think about the circular, cause-and-effect relationships within the closed loops of a system dynamics model; • automates the production of difference equations for the stocks; • constrains the writing of equations for the flows and converters these must include all the inputs shown in the diagram, and can include no other inputs. equations for the flows and converters are entered via an object's dialogue box and a complete set of equations can be inspected in the 'underlying' equation layer, as illustrated in the text box below. {initialization equations } init population_size = 2 unit_decay_rate = 0.05 potential_unit_growth_rate = 0.25 carrying_capacity = 1000 lim'rt_to_growth_fraction = (carrying_capacfty population_size)/ carrying_capacrty artualrate_of^rowth=popui^tion_size*potential_unit^rowth_rate*limit_to^rowth_fraction actual_rate_of_decay = population_size * unit_decay_rate { runtime equations } population_size(t) = population_size(t-dt) + (actual_rate_of_growth-actual_rate_of_decay) * dt limrt_to_growth_fraction = ( carrying_capadty population_size) / carrying_capacity actualrate_of^rowth=population_size*potential_unit_growth_rate* lim'rt_to_growth_fraction actual_rate_of_decay = population_size * unit_decay_rate notes: in the runtime stock equation:"population_size(t) = ..." population_size(t) is the value (in this case, between 2 and 1000) being calculated at time '(t)'. population_slze(t-dt) was the value one time interval before, at time '(t-dt)'. 'dt' is the short time interval between successive iterations of the model. figure 5: the 'limits to growth'model: equations in order of execution implications of stella to conclude this account of stella, david squires and fellow colleagues were impressed by this modelling program because it: • enabled students to see, modify and author models expressed in a formalism they could understand and that a computer could execute; • could be used to prototype models for use in cal games and simulations (squires and millwood, 1988), cic developed, trialled and published around 130 games, simulations and cal programs during the 1980s an average of one per month through the decade; • had the potential to evolve into a simulation authoring system suitable for teachers and lecturers with limited time, technical skills or financial resources. 60 alt-j volume 10 number 3 from mainstream to backwater the 1980s were notable for the high levels of interest in modelling as a form of computerassisted learning, and for several pioneering initiatives in schools and colleges. interest in the educational potential of modelling programs like stella was not restricted to cic. there was a widespread interest in modelling amongst cal developers and the wider educational community. for instance, classroom trials of stella led to the modus project (hassell and webb, 1990; cox and webb, 1994), and modelling featured in the it component of the newly launched national curriculum, whilst the london mental models group obtained esrc funding for research into the learning and teaching of modelling (bliss et al, 1992; mellar, bliss, boohan, ogborn and tompsett, 1994). these high levels of interest were not sustained during the 1990s and simulation modelling now seems to receive relatively little attention from learning technologists. this contrast with the 1980s is anomalous, given the: • pedagogic potential of simulation modelling for student engagement and authorship; • ease with which models and simulations can be assembled by students or teachers, using modelling programs like stella; • comparative abundance of personal and institutionally owned computers; • vastly improved capabilities of modern computers and networks. the reasons for this anomalous situation are worthy of reflection and, perhaps, of empirical research. for instance, there may be more simulation modelling taking place than is apparent, with the classroom modelling activities of individual lecturers being relatively invisible to learning technologists. one might hypothesize that, in general, learning technologists focus upon national and institutional initiatives rather than lecturers' unreported activities within departments. this is something learning technologists could investigate within their own institutions and share with the wider community, either on an informal basis or as a more organized form of empirical research. learning and teaching simulation modelling this section of the paper suggests ways forward and acknowledges some of the developments that have taken place over the past decade. if laurillard suggests the simulation modelling 'glass' is not full, this section suggests it is partially full and filling slowly. some approaches to simulation modelling are outlined, based on the growing pedagogic literature, and upon personal experience as a lecturer in physical geography at the former university of north london. making it happen students may first encounter specific models and the activity of modelling in their lectures, with practical work taking place in supervised 'laboratory' sessions where they gain handson experience of particular modelling programs. students, often working in small groups, may begin with simple ready-made models and learn to explore and modify these before constructing their own models based on provided materials or information gathered from library research. entirely free invention of models is less likely, although there is more scope for originality in students' projects and dissertations, especially at postgraduate level. 61 david rifey simulation modelling: educational development roles for learning technologists once students are familiar with the basic principles and practice in their subject field, lecturers can refer to models and modelling in other course modules, setting tasks to develop students' knowledge, skills and attitudes towards modelling. participation in a range of modelling activities can extend students' experiences of models as tools to 'think with' and to 'think about' (bliss, 1994a), and also of models as media for sharing ideas and explanations with others. reasons for modelling students may be unfamiliar with modelling as a professional or research practice and lecturers may need to describe the roles that modellers play in: • prediction and control (for example, in engineering design); • theory development (for example, in the biophysical and social sciences); • policy-making and decision-support (for example, in business and government); • education, training and entertainment (for example, in simulations and games). simulation modelling originated in the mathematical sciences and engineering as a means of predicting and controlling phenomena like the compression of gases, or the mechanical loading of structures. the extension of 'modelling to predict and control' from relatively well-theorized, predictable and universal domains, to the social and environmental sciences has been problematic (bloomfield, 1986; flood, 1999). here, phenomena tend to be less well theorized and are likely to be less predictable, as with weather and economic forecasting (ogborn and mellar, 1994). the notions of 'modelling to develop theory' (gilbert and troitzsch, 1999), or 'modelling to learn' (de geus, 1992) better suit business education and the context-sensitive, more 'situated' social sciences. social and economic models may also influence the phenomena they simulate, as people become aware of models and adjust their behaviour. on the one hand, this is a problem not faced by the modellers of biological, mechanical and electrical systems, where the principles of cybernetics and system dynamics were first formulated (forrester, 1968; lilienfeld, 1978). on the other hand, social responsiveness is turned to advantage where models are used to influence policy-makers and to inform public opinion. environmentalists are amongst those who have 'modelled to persuade', for instance, the club of rome report on global environmental futures and its sequels (meadows et al, 1972, 1992), and james lovelock's imaginary daisyworlds (watson and lovelock, 1983; lovelock, 2000). the latter leads in short order to the notion of 'modelling to entertain' as exemplified by the simearth and simcity type of computer games, which blur the boundaries between education and entertainment. approaches to modelling lecturers' reasons for including modelling in the curriculum will vary, depending on the subject field and course goals. this paper focuses on geographic and business education where social, economic, political and ethical issues are closely linked to the design and application of models in academic, personal, professional and public settings. it is desirable for students to understand what models are, how they are created and how they are used whether to inform and persuade or to confuse and mislead. it is notable that we teach a large number of university students about the use and abuse of statistics but teach 62 alt-j volume 10 number 3 very few about the use and misuse of computer models despite their widespread and significant applications. a common reason for including modelling in the curriculum is that it reveals, and may improve, students' formal reasoning (mandinach and cline, 1994; doerr, 1996; sweeney and sterman, 2000), with peer discussion and group-work playing supportive roles. the extent to which reasoning, communication or collaboration are prioritized is a matter for lecturers and course developers to determine along with other course goals. for instance, staff may choose to emphasize either: • cognitive or communicative learning outcomes; • individual or collaborative work; • convergent tasks or open-ended problem-solving; • the exploration of 'classic' models or the expression of students' models (mellar and bliss, 1994); • general mathematical methods or program-specific techniques and 'work-arounds'; • idealized theories of modelling or case studies of modelling in practice. options like these, and others, are not mutually exclusive and the balance between them may vary over a period of several weeks or longer. for instance, students' modelling activities within a degree programme may culminate in assessed projects or in final dissertations where students assume more responsibility for their own aims and approaches. introductory textbooks and resources lecturers and students can pick and choose amongst numerous introductory textbooks, of which there is a range in fields like environmental modelling (huggett, 1993; kirkby, naden burt and butcher, 1993; mcguffie and henderson-sellers, 1997). this range also includes program-specific textbooks with modelling activities based on excel (hardisty, taylor and metcalfe, 1993) and stella (ford, 1999). in addition, there are more general texts that address the concerns of human geographers (macmillan, 1989; robinson, 1998), which place greater emphasis on critiques of models and of modelling itself. books and resources on systems modelling are appearing too, in business and management education (coyle, 1996; pidd, 1998; senge, 1990; senge, kleiner, roberts, ross and smith, 1994), and many of these include sections on, or are based upon, system dynamics. a recent and comprehensive textbook on 'business dynamics' (sterman, 2000), is accompanied by a cd-rom with models and three system dynamics programs: vensim,2 powersim3 and 'ithink' the version of stella targeted at business markets (repenning, 1998; costanza and voinov, 2001). further tutorial materials and models are provided by sdep the system dynamics in education project4 on its mit-hosted website. the sdep road maps5 contain some 11mb of introductory and more advanced tutorials, mostly in 'pdf' format, that are 'quality assured' by the mit system dynamics group and can be reproduced for non-commercial purposes. these recent books and electronic resources succeed earlier texts for the modelling language dynamo (goodman, 1988; roberts, andersen, deal, garet and shaffer, 1983) and tend to be less formal in approach and to assume less knowledge of mathematics. peter 63 david riley simulation modelling: educational development roles for learning technologists senge's books, in particular, emphasize the concepts of system dynamics and reasoning with paper-based, causal loop diagrams, rather than quantitative modelling on a computer. this approach directly addresses students' conceptual understanding whilst avoiding the complications that accompany quantification and the use of mathematics. these are issues that arise elsewhere, for instance, with younger students (bliss, 1994b; brosnan, 1994), and with the teaching of environmental systems (inkpen, 1993). still, there are pitfalls with 'semi-quantitative' reasoning (richardson, 1986 and 1997), and it is a moot point whether students should be introduced to causal loop-diagramming before stella or whether the full stella formalism should be introduced first, with diagramming as a subsequent simplification. thus, a lecturer's choice of an introductory textbook or use of online materials may depend upon: • the backgrounds and needs of the students; • the curricular goals and reasons for teaching modelling; • available modelling programs; • the preferred formalism(s); • their sequence of introduction. also, lecturers can choose between a number of modelling programs and are by no means restricted to those based on system dynamics. gilbert and troitzsch, for instajice, suggest approaches based on the simulation of discrete events, cellular automata, intelligent agents or evolutionary processes may better appeal to social theoreticians. to return to the example of stella, lecturers will find this has improved considerably over the last fifteen years or so and it now supports windows computers. also, it provides an additional communications or interface layer 'above' the original diagram and equation layers. this new layer supports several features, including: • scene-setting introductions and tutorials with text, pictures and quicktime movies; • annotated step-by-step tours of a complete model or of selected parts to explain the underlying logic and behaviour; • simulation 'control panels', such as the one shown in figure 1, assembled from standard drag and drop components. in effect, stella has become the type of simulation authoring environment originally anticipated by david squires. it is an environment in which lecturers can rapidly produce models and simulations, without the need for conventional programming skills or the associated development resources and timescales. reflections there have therefore been several developments during the 1990s: the publication of introductory textbooks and online resources, and a growth in the number and quality of modelling programs. whilst laurillard may be correct in asserting the productive media are underutilized in higher education, the above are grounds for some optimism, at least with respect to systems modelling in geographic and business education. lecturers and learning technologists working in these fields can build upon broader and more substantial foundations than those available one or two decades ago. 64 alt-j volume 10 number 3 educational development roles for learning technologists this final part of the paper suggests learning technologists may be well placed to raise awareness of simulation modelling in higher education. many academic staff are appointed on the grounds of their research and income-generating potential rather than their pedagogic repertoires or use of learning technologies. although most lecturers new to higher education receive some training, they rarely experience anything equivalent to their lengthy apprenticeships as researchers. many rely on interested colleagues and support staff to suggest improvements to their course modules or to recommend 'new' approaches to learning and teaching. learning technologists, through their contacts with lecturers and course development teams, are well positioned to: • suggest simulation modelling as a learning activity when it otherwise may be overlooked; • draw attention to available modelling programs, online resources and introductory textbooks either in staff's own fields or in related ones; • assist in the design and production of multimedia support materials and tutorials for: simulations in programs like stella; resource-based learning materials on simulation modelling; • advise on the design, assessment and evaluation of modelling activities as instances of . the use of learning technologies; • contribute to the development of new learning environments supporting collaborative modelling activities. individual learning technologists may not be able to perform all of these roles but one hopes they will contribute to curriculum development both in terms of their technical and educational expertise. learning technologists benefit from meeting teaching staff from around an institution and can point to practice in different departments. they can help interested academics to network and to share their experiences of approaches outside the traditional repertoire of lectures, seminars, practicals and tutorials. once academic staff are familiar with programs like stella, and simulation modelling as a student activity, they may be self-sufficient or need support only to produce multimedia-rich introductions and tutorials. many academic staff are unused to developing student materials for resource-based or distance learning, whilst this is a more familiar activity to learning technologists. similarly, academics can benefit from introductions to the literature on the evaluation of learning technologies and the assessment of computer-assisted learning activities. finally, laurillard notes the potential for combining microworlds and modelling programs with communications technologies and points to developments in the commercial and entertainment sectors. this is the type of innovation that would have interested david squires and one looks for learning technologists to follow his example and develop new categories of educational applications. this will be neither cheap nor easy but it is a field in which learning technologists could make important contributions. 65 david riley simulation modelling: educational development roles for learning technologists conclusions this paper has drawn attention to simulation modelling and has sought to raise awareness amongst learning technologists of this distinctive, if relatively neglected, pedagogic activity. simulation modelling is advocated on the grounds that it: • makes cost-effective use of lecturers' and students' time. educational simulations, models or model fragments can be quickly retrieved, modified or created with software such as stella; • enables students to learn through modifying and creating simulation models within their theoretical field activities that only the productive media support; • introduces students to modelling as a significant academic and professional practice; • fosters cognitive, communicative and collaborative student activities; • facilitates a participative and productive approach to learning, one that realizes the creative-transformative potential of learning technologies. whilst laurillard may have exaggerated the underdevelopment and underutilization of simulation modelling, there clearly are opportunities for learning technologists to promote and support its 'rediscovery'. contributions may take many forms, including making suggestions to lecturers, assisting with the production of learning materials, and contributing to software development projects. simulation modelling provides many challenges for learning technologists, of the type that david squires used to seize with relish. now new champions are required, people who can once more bring simulation modelling into the mainstream of higher education and help develop curricula fit for the twenty-first century. notes and relevant websites the following urls were last accessed in august 2002. 1stella® and 'ithink®' are produced by high performance systems: http://www.hpsinc.com and distributed in the uk by cognitus, http://www.cognitus.co.uk/ 2vensim® is produced by ventana systems. vensim ple is free for educational use and downloadable from: http://www.vensim.com/ 3powersim© is produced by the powersim corporation: http://www.powersim.com/ 4system dynamics education project (sdep): http://sysdyn.mit.edu/home.asp 5sdep road maps (for stella and vensim ple): http://sysdyn.mit.edu/roadmaps/home.htm/ lists of online resources for systems dynamics and systems thinking: creative learning exchange, additional resource links: http://www.clexchange.org/links/ desert island dynamics an annotated survey of the essential system dynamics literature: http://web.mit.edu/jsterman/www/did.html günther ossimitz: system dynamics/systems thinking mega link list: http://www.uni-klu.ac.at/users/gossimit/links/bookmksd.htm 66 alt-} volume 10 number 3 system dynamics distance learning: http:llwww.albany.edulcprlsdsldistantjearning.htm references bliss, j., ogborn, j., boohan, r., briggs, j., brosnan, t., brough, d., mellar, h., miller, r., nash, c., rodgers, c. and sakonidis, b. (1992), 'reasoning supported by computational tools', computers and education, 18 (1), 1-9. bliss, j. (1994a), 'from mental models to modelling', in mellar et al., learning with artificial worlds: computer based modelling in the curriculum. bliss, j. (1994b), 'causality and common sense reasoning', in mellar et al., learning with artificial worlds: computer based modelling in the curriculum. bloomfield, b. r (1986), modelling the world: the social constructions of systems analysts, oxford: basil blackwell. brosnan, t. (1994), 'using spreadsheets to develop understanding in science', in mellar et al., learning with artificial worlds: computer based modelling in the curriculum. costanza, r. and voinov, a. (2001), 'modeling ecological and economic systems with stella: part iii', ecological modelling, 143 (1), 1-7. cox, m. and webb, m. (1994), 'developing software and curriculum materials: the modus project', in mellar et al., learning with artificial worlds: computer based modelling in the curriculum. coyle, r. g. (1996), system dynamics modelling: a practical approach, london: chapman and hall. de geus, a. p. (1992), 'modelling to predict or to learn?', european journal of operational research, 59 (1), 1-5. doerr, h. m. (1996), 'stella ten years later: a review of the literature', international journal of computers for mathematics learning, 1 (2), 201-24. flood, r. l. (1999), rethinking the fifth discipline: learning within the unknowable, london: routledge. ford, a. (1999), modeling the environment: an introduction to system dynamics modeling of environmental systems, washington dc: island press. forrester, j. w. (1961), industrial dynamics, cambridge, ma: wright-allen press. forrester, j. w. (1968), principles of systems, 2nd edn, cambridge, ma: wright-allen press. gilbert, n. and troitzsch, k. g. (1999), simulation for the social scientist, buckingham: open university press. goodman, m. r. (1988), study notes in system dynamics, cambridge, ma: the mit press (republication of 1974 original, wright-allen press). hardisty, j., taylor, d. m. and metcalfe, s. e. (1993), computerised environmental modelling: a practical introduction using excel, chichester: john wiley & sons. 67 david riley simulation modelling: educational development roles for learning technologists hartley, r. j. (1984), 'evaluation of a cal package for modelling', computers and education, 8 (1), 69-76. hartley, r. j. and lewis, r. (1984), 'a computer language system for model-building and experimentation', international journal of mathematical education in science and technology, 13 (4), 391-400. hassell, d. j. and webb, m. e. (1990), 'modus: the integrated modelling system', computers and education, 15 (3), 265-70. huggett, r. j. (1993), modelling the human impact on nature: systems analysis of environmental problems, oxford: oxford university press. inkpen, r. (1993), 'a low-tech approach to teaching physical systems', journal of geography in higher education, 17 (1), 37-46. kirkby, m. j., naden, p. s., burt, t. p. and butcher, d. p. (1993), computer simulation in physical geography, chichester: john wiley & sons. lal, s. (1987), 'two cheers for dms', journal of computer assisted learning, 3 (3), 176-80. laurillard, d. (2002), rethinking university teaching: a conversational framework for the effective use of learning technologies, 2nd edn, london: routledge. lilienfeld, r. (1978), the rise of systems theory: an ideological analysis, new york: john wiley & sons. lovelock, j. e. (2000), homage to gaia: the life of an independent scientist, oxford: oxford university press. mcguffie, k. and henderson-sellers, a. (1997), a climate modelling primer, 2nd edn, chichester: john wiley & sons. macmillan, b. (ed.) (1989), remodelling geography, oxford: basil blackwell. mandinach, e. b. and cline, h. f. (1994), classroom dynamics: implementing a technology-based learning environment, hillsdale nj: lawrence erlbaum associates. meadows, d. h., meadows, d. l., jørgen, r. and behrens, w. w. iii (1972), the limits to growth: a report for the club of rome's project on the predicament of mankind, new york: universe books. meadows, d. h., meadows, d. l. and jørgen, r. (1992), beyond the limits: global collapse or a sustainable future, london: earthscan. mellar, h., bliss, j., boohan, r., ogborn, j. and tompsett, c. (eds.) (1994), learning with artificial worlds: computer based modelling in the curriculum, london: falmer press. mellar, h. and bliss, j. (1994), 'introduction: modelling and education', in mellar et al., learning with artificial worlds: computer based modelling in the curriculum. ogborn, j. and mellar h. (1994), 'models: their makers, uses and problems', in mellar et al., learning with artificial worlds: computer based modelling in the curriculum. 68 alt-j volume 10 number 3 pidd, m. (1998), computer simulation in management science, chichester: john wiley & sons. repenning, n. (1998), formulating models of simple systems using vensim ple, available from: http://sysdyn.mit.edu/sdep/roadmaps/rm2/d-4697-2.pdf (last accessed august 2002). richardson, g. p. (1986), 'problems with causal-loop diagrams', system dynamics review, 2 (2), 158-70. richardson, g. p. (1997), 'problems in causal-loop diagrams revisited', system dynamics review, 13 (3), 247-52. riley, d. (1986), 'a case for classroom research into models, modelling and cal in secondary geography', in w. a. kent and r. lewis, computer assisted learning in the humanities and social sciences, oxford: blackwell scientific publications. roberts, n., andersen, d., deal, r., garet, m. and shaffer, w. (1983), introduction to computer simulation: a system dynamics modeling approach, reading, ma: addisonwesley. robinson, g. m. (1998), methods and techniques in human geography, chichester: john wiley & sons. seale, j. and rius-riu, m. (eds.) (2001), an introduction to learning technology within tertiary education in the uk, oxford: association for learning technology. senge, p. m. (1990), the fifth discipline: the art and practice of the learning organization, london: random house. senge, p. m., kleiner, a., roberts, c., ross, r. b. and smith, b. j. (1994), the fifth discipline fieldbook: strategies and tools for building a learning organization, london: nicholas brealey publishing. squires, d. j. and millwood, r. (1988), 'the influence of new software environments on cal development', computers and education, 12 (1), 67-71. squires, d. j. and mcdougall, a. (1986), 'computer-based microworlds a definition to aid design', computers and education, 10 (3), 375-8. sterman, j. d. (2000), business dynamics: systems thinking and modelling for a complex world, boston, ma: irwin mcgraw-hill. sweeney, l. b. and sterman, j. d. (2000), 'bathtub dynamics: initial results of a systems thinking inventory', system dynamics review, 16 (4), 249-86. watson, a. j. and lovelock, j. e. (1983), 'biological homeostasis of the global environment: the parable of daisyworld', tellus, 95 (b), 284-9. watson, d. (1987), developing cal: computers in the curriculum, london: harper & row. wong, d. (1987), 'teaching a-level physics through microcomputer dynamic modelling: i. teaching methods', journal of computer assisted learning, 3 (2), 105-16. 69 alt-j, research in learning technology vol. 12, no. 3, september 2004 issn 0968–7769 (print)/issn 1741–1629 (online)/04/030215–15 © 2004 association for learning technology doi: 10.1080/0968776042000259546 implementation of computer assisted assessment: lessons from the literature gavin sim*, phil holifield & martin brown university of central lancashire, uk taylor and francis ltdgalt120303.sgm10.1080/0968776042000259546alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2004association for learning technology123000000september 2004gavinsimdepartment of computing, computing and technology buildinguniversity of central lancashireprestonpr1 2hegrsim@uclan.ac.uk this paper draws attention to literature surrounding the subject of computer-assisted assessment (caa). a brief overview of traditional methods of assessment is presented, highlighting areas of concern in existing techniques. caa is then defined, and instances of its introduction in various educational spheres are identified, with the main focus of the paper concerning the implementation of caa. through referenced articles, evidence is offered to inform practitioners, and direct further research into caa from a technological and pedagogical perspective. this includes issues relating to interoperability of questions, security, test construction and testing higher cognitive skills. the paper concludes by suggesting that an institutional strategy for caa coupled with staff development in test construction for a caa environment can increase the chances of successful implementation. introduction this paper presents evidence that the more traditional methods of assessment within universities have their limitations. as a result of these limitations and also the continued increase in the use of technology to deliver curriculum, the gap between assessment methods and learning is widening. students entering higher education directly from schools and colleges are likely to have been exposed to information technology as part of the uk national curriculum. pilot studies conducted within schools for the delivery of summative assessment via the web (ashton et al., 2003; nugent, 2003) and for basic key skills tests in both learn direct and army centres (sealey et al., 2003) indicate that caa can successfully assess students and provide timely feedback regarding class and individual progress. there is also empirical evidence to suggest students find caa an acceptable assessment technique (sambell et al., 1999; croft et al., 2001; ricketts & wilks, 2002a). therefore it could be argued that for many students caa may become a * corresponding author: department of computing, computing and technology building, university of central lancashire, preston pr1 2he, uk. email: grsim@uclan.ac.uk 216 g. sim et al. more widely used method of assessment in schools, further education and universities. many universities are now using technology in their assessment strategies (stephens & mascia, 1997) and by examining the literature, lessons can be learned to facilitate the successful implementation of computer-assisted assessment. methodology a comprehensive literature review was conducted primarily using online resources, although library resources were also used. the searching centred on the databases ingenta, aace, science direct and conference proceedings such as the international computer assisted assessment conference. key word searching was problematic and time consuming, for example conducting a search using ‘computer assessment’ would produces a divergent array of articles in excess of one thousand. other terminology used in the search included ‘computer based testing, computer-based assessment, computer aided assessment and e-assessment’. browsing through the contents of entire journals such as ‘assessment and evaluation in higher education’ was also adopted. assessment in general academic assessment can be administered through various techniques. fifty varied techniques have been identified and used within higher education for assessment purposes (knight, 2001); the most commonly used are exams and essays (graham, 2004). however this does not include all the methods now available within caa packages for example incorporate questions that make use of multimedia. new assessment techniques will continue to emerge as technology and teaching methods change and develop, therefore continuing research will be required to determine the effectiveness and appropriateness of these methods. each form of assessment presents its own difficulties, whether computer based or traditional. essays present the problem of double marking, in one study both markers agreed only 52% of the time (powers et al., 2002). additionally there are the problems with cheating as internet sites offer custom-written and off the shelf essays (crisp, 2002). it has been suggested that exams tend to encourage surface learning (race, 1995) and may cause increased anxiety resulting in significantly lower scores (cassady & johnson, 2002). the multiple choice question (mcq) styles are used in both offline and caa exams and raise a number of concerns, for example, grade deflation by not enabling partial credit (baranchik & cherkas, 2000), poorly designed questions (paxton, 2000; jafarpur, 2003) and guessing (burton, 2001). however the advantages of using computers to deliver mcq for lecturers include automated marking (pollock et al., 2000) and for formative purposes the students have the opportunity to study at their own pace, repeat questions and receive instant feedback (loewenberger & bull, 2003). it is the potential advantages of caa that has driven research into ways to overcome the difficulties. ultimately in an academic environment the marks from summative assessment are accumulated to award an overall grade and there are concerns over comparability across implementation of computer assisted assessment 217 subject domains. it has been suggested that the scientific subjects produce more first class degrees than the humanities because of the nature of the marking criteria in using the full range of marks and subjectivity is eliminated from the equation where there is a predefined correct answer (yorke et al., 2002; horney, 2003). these findings would appear to be further corroborated by the higher education statistics agency (hesa) figures. of the students graduating from uk universities in 2001/02 in mathematical science 25.5% passed with a first class degree, compared to 10.4% in humanities (hesa, 2002) and this trend was also evident in other years for example, 1994/95 (hesa, 1995). caa, like mathematics and some science subjects, also tends to use the full range of marks therefore the trend towards a high proportion of first class degrees may occur in other subject domains adopting this technique in the future. there is pressure on lecturers not to fail students, and one study found that in professional subjects there is a tendency to leave the award of a fail to the next assessor (hawe, 2003). lecturers are confronted with emotional and ethical dilemmas when a close working relationship is formed, increasing their reluctance to award a fail (sabar, 2002). the emotional and subjectivity issues that are evident in human centred marking may be removed via automatic marking offered by caa software. it is important to recognize that some of these issues discussed are still prevalent in caa along with new challenges. adopting a diverse assessment strategy may lead to a fairer assessment of the student (race, 1995). computer-assisted assessment defined from the literature there is a lack of universal consent regarding the terminology and its definition, however, bull and mckenna (2001) argue that computer-assisted assessment is the common term for the use of computers in the assessment of students and the other terminology tend to focus on the activities. therefore the definition of caa used in this review will be that: caa encompasses the use of computers to deliver, mark or analyse assignments or exams. variations in caa within higher education institutions the application of caa has occurred in a number of varied ways, these include, adaptive testing (latu & chapman, 2002; mills et al., 2002), analysis of the content of discussion boards (macdonald & twining, 2002; wiltfelt et al., 2002), automated essay marking (christie, 1999; burstein et al., 2001), delivery of exam papers (sim et al., 2003) and objective testing (walker & thompson, 2001; pain & le heron, 2003). these methods vary considerably however the focus of this review of research will centre on the issues relating to implementing objective tests via caa. testing cognitive skills with caa there is concern in the literature relating to caa and its ability to test higher cognitive skills across subject domains (daly & waldron, 2002; paterson, 2002). the higher 218 g. sim et al. cognitive skills are often associated with ‘analysis, synthesis, and evaluation’ as defined in bloom’s taxonomy (bloom, 1956). however, a revised taxonomy takes into consideration the ‘knowledge dimension’ (anderson & krathwohl, 2001) and this has also been used in caa research for classification of questions (king & duke-williams, 2002; mayer, 2002). paterson (2002) indicated that it is not feasible to test the higher-level cognitive skills using caa within mathematics. bloom states that in the majority of instances synthesis and evaluation promote divergent thinking and answers cannot be determined in advance (bloom et al., 1971). heinrich and wang (2003) argue that objective testing is still not sophisticated enough to examine complex content and thinking patterns. however, other research in linguistics and computer programming concluded that the higher-level skills can be assessed via caa through innovative approaches (cox & clark, 1998; reid, 2002). in the study by reid (2002) a new language was devised and students were required to apply linguistic techniques in order to answer mcq. it has been suggested that caa tests of higher-level skills are more complex and costly to produce (dowsing, 1998) and this may be because more innovative approaches are needed. question styles objective testing has been used within assessment for over forty years (wood, 1960) and computer programs delivering mcq date back to the 1970s (morgan, 1979). more sophisticated question styles have emerged enabling more diverse assessment methods. the question styles delivered by the triads software developed at derby university are evidence of this evolution, offering 17 question styles in 1999 (mackenzie, 1999) and 39 in 2003 (ciad, 2003). however, staff at the university of liverpool using triads found that this presented an additional problem, as they were unfamiliar with the new question styles and lacked confidence in writing suitable questions (mclaughlin et al., 2004). staff development in writing suitable questions and guidelines can be used to overcome these problems. for example, generic guidelines developed by haladyna (1996), herd and clark (2002) present examples of the various questions styles used in further education whilst examples used within higher education can be found at http://www.caacentre.ac.uk. although there are a large number of possible formats for caa questions, it is possible to classify them into four distinct groups based on the human interaction technique required (ciad, 2003). these groups are defined as point and click, move object, text entry and draw object. point and click point and click questions include multiple choice (mcq) and multiple response (mrq) items, which have both been used within assessment practise for a considerable time and as a result are often transformed into caa (ricketts & wilks, 2002b). ebel (1972) suggests that any understanding or ability that can be tested by means of implementation of computer assisted assessment 219 any other technique, for instance essays, can also be tested by mcq. more complex mcq questions can be devised through assertion reasoning resulting in the testing of higher cognitive skills (bull & mckenna, 2001). both mcq and mrq have inherent problems, such as reliance on true and false style questions which students might perceive to be unfair (wood, 1960). davies also argues that the quality of mcq is dependent on the quality of the distracter and not the question (davies, 2002). move object move object style questions focus on the movement of objects to predetermined positions on the screen. they are a variation of the mcq format and are good for assessing students understanding of relationships (bull & mckenna, 2001). for example in computing they could be used for the labelling of entity relationship diagrams or in linguistics students could be presented with a poem and move the highlighted words to the appropriate word class. one problem is that when the number of moveable objects is equal to the number of targets, if a student knows all but one answer they will automatically get full marks (wood, 1960). text entry text entry questions consist of input of short predefined answers, such as factual knowledge or syntax in computer programming. an advantage of this format is that students must supply the correct answer removing the possibility of guessing (bull & mckenna, 2001) and this style has been found to be the most demanding format for students (reid, 2002). there are problems associated with text entry within some subject domains such as mathematics, as mathematical expressions cannot easily be included in most commercial software (croft et al., 2001; paterson, 2002). another problem associated with this question style is that the answer may be marked incorrect due to spelling mistakes and the time saving element may be reduced if lecturers need to manually check for spelling errors. draw object this is associated with drawing simple objects or lines. for example, students may be required to plot graphs which can be automatically marked. this style of question is a high discriminator between strong and weak candidates (mackenzie, 1999). there is little evidence in the literature concerning the effectiveness of this format, but this might be due to the fact that commercial software such as questionmark and i-assess do not have this style in their templates. interoperability and question banks question banks which are authored and peer reviewed by academics are emerging, such as the electrical and electronic engineering assessment network who 220 g. sim et al. developed a database of questions in electrical and electronic engineering (bull et al., 2002). one such bank will typically require 5000 questions making it unfeasible for a single institution to develop (maughan et al., 2001). constructing high quality questions is difficult, time consuming and expensive (sclater et al., 2003) and issues arise in the interoperability of questions between caa software (lay & sclater, 2001). there are several international standards established to enable interoperability of questions between software applications (herd & clark, 2003). these specifications are based on metadata structure for questions and their grouping together. unless these interoperability standards are developed and utilized question banks will have a limited life, as they cannot be used on a variety of delivery platforms (white & davis, 2000). systems are emerging that are ims-qti compliant (instructional management systems – question and test interoperability specification) to facilitate the exchange of questions (daly, 2002; bacon, 2003). the centre for educational technology interoperability standards (www.cetis.ac.uk) offers comprehensive resources and information on the issues concerning interoperability which may help direct further research. guessing a number of the question styles associated with caa can lead to artificially high marks through guessing (bush, 1999), which has implications for setting the pass mark of the test. for example, setting a pass mark of 40% based on assessment of true/false answers would be inappropriate, as guessing alone would give an average of 50% (harper, 2002). the problems of guessing may be addressed through various marking schemes, such as post test correction (bull & mckenna, 2001), negative marking (bush, 1999), increasing the number of questions or combining the results from several tests (burton & miller, 1999) or increasing the number of distracters and the pass mark (mackenzie & o’hare, 2002). it has been suggested that negative marking is not generally implemented in the uk (mcalpine, 2002) and that post test correction is only suitable with a single question style because the formulae would vary depending on the number of distracters (harper, 2003). statistical analysis has resulted in various methods being developed to assist in test construction in order to reduce the effects of guessing. an empirical marking simulator to assist in scoring and test construction based on a base level guess factor has been developed (mackenzie & o’hare, 2002), this program examines the mark distribution and measurement scale for a set of random answers, enabling tutors to establish the effects of guessing on their assessment. also statistics to award a score for partial credit through a formula based on a mean uneducated guessers score has been investigated (mccabe & barrett, 2003). this allows mcq to be unconstrained, similar to mrq styles, enabling students to provide more than one answer and their score is weighted depending on the number of choices. for example, an mcq with one correct answer, four possible options and a score of 3, if a student includes the correct answer by selecting 2 options they would only score 2 (2=3-1). davies used a combination of predetermining the students’ confidence in answering the question implementation of computer assisted assessment 221 prior to seeing the distracters and negative marking, resulting in students perceiving this to be a fairer test of their abilities (davies, 2002). there is lack of evidence that any one specific technique generates more accurate results than any other. it could be argued that these techniques are unnecessary if the tests are well constructed (bull & mckenna, 2001). accessibility uk institutions now have to comply with the special educational needs and disability act when preparing both teaching and assessment material (senda, 2001). the number of students in uk higher education registering a disability in 2000 was 22,290 and this has implications for caa (phipps & mccarthy, 2001). for example, a student with dyslexia may exert more cognitive resources in interpreting the question, therefore, ensuring the language is appropriate is a necessity (wiles & ball, 2003). in addition extra time may be required to complete the test which may necessitate the publishing of two different assessments, one with a longer duration. feedback from one dyslexic student regarding caa indicated that they thought it provided a more level playing field in which they can demonstrate their knowledge (jefferies et al., 2000). students with visual or physical impairment may struggle to answer move object and draw object style questions without the aid of assistive technology, they may need specially adapted input software and hardware such as, touch screens, eyegaze systems or speech browsers. there are guidelines for general teaching, however there is little evidence that guidelines for inclusive and accessible design in caa are emerging (wiles, 2002). for example, when multimedia elements, such as video are used within the assessment, it may necessitate the provision of an alternative paper-based version for students with sensory impairment. the introduction of an alternative, in this instance paper, poses the problem of ensuring comparability (bennett et al., 1999). when identical tests are presented on a computer and paper they are not comparable (clariana & wallace, 2002) because there are numerous variables that impact on student’s performance when questions are presented on a computer. these variables include the monitor (schenkman et al., 1999), the way text is displayed on screen (dyson and kipping, 1997), reading from a monitor is slower than paper (mayes et al., 2001) and the problems of obtaining a feel for the exam when only a single question is presented (liu et al., 2001). the web accessibility initiative (http://www.w3c.org/wai/) has produced useful guidelines for promoting online accessibility which may be applicable to caa but this initiative does not address the issue of comparability between questions. institutional strategies for the adoption of caa the greatest barrier to the adoption of caa by academics is lack of time, to both develop questions and learn the software (warburton & conole, 2003). this may have contributed to the fact that the adoption of caa has usually resulted from the 222 g. sim et al. impetus of enthusiastic individuals rather than strategic decisions (o’leary & cook, 2001; daly & waldron, 2002). the perceived benefits of caa of freeing lecturers’ time can be illusive if no institutional strategy or support is offered (stephens, 1994), successful implementation may be left to chance (stephens et al., 1998) and caa may be developed in an anarchic fashion (mckenna & bull, 2000). research conducted at the university of portsmouth indicate that there is no time saving benefit for courses with less than twenty students (callear & king, 1997). in order to utilize the features within software packages staff training and development is necessary (boyle & o’hare, 2003) and this may not be feasible without institutional support. institutions adopting caa are faced with the difficulty of evaluating and deciding upon the most appropriate caa software. without an institutional strategy, individual departments may adopt their own systems (o’leary & cook, 2001). this results in students having to cope with a number of different user interfaces and caa formats, increased licence costs and problems offering administrative and technical support. even if an institution has a clear strategy there are also problems in determining the selection criteria for software used to deliver assessment and there is a lack of analysis within the literature (valenti et al., 2002). sclater and howie (2003) contributed to this literature by defining the ultimate online assessment engine. this was achieved through a process of examining the user requirements of the system, establishing the stakeholders and their functional requirements. this research may aid institutions identify their needs and establish an appropriate evaluation methodology. the following guidelines for an institutional strategy have been formulated by loughborough university and the university of luton: establish a coordinated caa management policy for caa unit(s) and each discipline on campus; establish a caa unit; establish caa discipline groups/committees; provide funding; organize staff development programmes; establish evaluation procedures; identify technical issues; establish operational and administrative procedures (stephens et al., 1998). bs7988 is a new british standard code of practice that has been introduced governing the use of information technology in the delivery of assessments (bs7988, 2002). the guidelines have various implications for the delivery of assessments, for example, it is recommended that students take a break after 1.5 hours which has an impact on the invigilation process. if this recommendation is followed, procedures need to be established to prevent collusion between students during the break or the tests need to be split into two separate sections. one of the difficulties for many institutions using caa arises through the lack of resources to accommodate large cohorts of students sitting the exam simultaneously (mackenzie et al., 2004). this problem can be alleviated through institutional support and therefore, to fully utilize the benefits of caa an institutional strategy would appear necessary to increase the chance of successful implementation. these benefits are evident within a number of institutions with strategies, such as, ulster (stevenson et al., 2002), derby (mackenzie et al., 2002), coventry (lloyd et al., 1996) and loughborough (croft et al., 2001). implementation of computer assisted assessment 223 security the move from traditional teaching environments and examination settings presents additional issues relating to security. frohlich (2000) states that in traditional environments it is possible to ensure the security of the exam papers and scripts, this includes the transportation to and from the exam venue. however, even under this system breaches in security do occur, for example aqa had to replace 500,000 english and english literature exam papers after a box had been tampered with (curtis, 2003). tannenbaum (1999) defines security in computer systems as consisting of procedures to ensure that individuals cannot access material for which they do not have authorisation. this is essential within a caa environment as questions and student details are stored in a database and usually the test data is sent over a local network or the internet. before computers were connected to the internet it was relatively easy to have effective security measures (mason, 2003), but transmission of sensitive data over an insecure network requires additional security measure to be implemented. encryption techniques can be used to ensure the security of the questions and answers when transmitting data over the internet (sim et al., 2003). to increase security, examinations can be loaded on to the server at the last minute (whittington, 1999). if email is used to submit results there is a potential risk due to the lack of authentication (hatton et al., 2002). four security requirements have been identified by luck and joy, these being: all submissions must be logged, it must be verified that a stored document used for the assessment is the same as the one used by the student, a feedback mechanism must inform students that their submission has been received and the identity of the student must be established (luck & joy, 1999). with the majority of caa software students and administrators are required to have passwords which is often the weakest link in terms of protection (hindle, 2003). although an unlikely event, students could get access to the administrator password and change their results or gain access to the questions. other concerns include authentication and invigilation of the students, which can be are particularly problematic in remote locations (thomas et al., 2002). at present students enrolled on distance learning courses overseas need to sit exams in a specific location such as, the british council offices to enable authentication and invigilation. research is being conducted to overcome these problems but unless solutions are found, geographical barriers will remain as students need access to the test centres. during the test computers need to be locked down, removing the possibility of accessing other content and secure browsers have been developed to enable this such as, questionmark secure (kleeman & osborne, 2002). there are operational risks associated with caa that have security implications such as the server crashing and these risks need to be identified and procedures established to minimize them (zakrzewski & steven, 2003). there are software standards for security for example, the british standards on information security management bs7799, which has also been adopted as an 224 g. sim et al. international standard is17799. in addition, when data from the test has been collected institutions within the uk should abide by the data protection act 1998 (mason, 2003). if security measures are in place there is no evidence to suggest that the integrity of the examination is more compromised by delivery over the internet than by paper. conclusion the implementation of caa from a technical and pedagogical perspective is a complex process. the first, and perhaps the most important, lesson that can be learned is that an institutional strategy would seem to greatly increase the chances of success. there are recommendations that have been made to assist policy makers formulate an effective strategy. without institutional support implementing security procedures may be more problematic, such as locking down pcs. however, authentication and invigilation in remote locations is still an issue that has yet to be fully resolved. the other important lesson that can be learned is in relation to staff development and training in test construction within a caa environment. focused staff development may help alleviate a number of issues, such as guessing, testing various cognitive skills, using appropriate question styles and accessibility. the emergence of question banks may also address these issues depending on their level of interoperability. another issue is that whilst there are guidelines relating to accessible online content there are still no formal guidelines relating to caa. the reliance on a single method of assessment is problematic and a diverse assessment strategy is usually necessary. within an environment of increasing student numbers and a reduction of staff to student ratio, caa would appear to be a partial solution. this study has highlighted the issues surrounding the implementation of caa to both inform and direct further research in the field. references anderson, l. w. & krathwohl, d. r. (2001) a taxonomy for learning, teaching, and assessing. a revision of blooms taxonomy of educational objectives (new york, longman). ashton, h. s., schofield, d. k. & woodger, s. c. (2003) pilot summative web assessment in secondary education, proceedings of the 7th international computer assisted assessment conference (loughborough, loughborough university), 19–29. bacon, r. a. (2003) assessing the use of a new qti assessment tool within physics, proceedings of the 7th international computer assisted assessment conference (loughborough, loughborough university), 33–44. baranchik, a. & cherkas, b. (2000) correcting grade deflation caused by multiple-choice scoring, international journal of mathematical education in science and technology, 31(3), 371–380. bennett, r. e., goodman, m., hessinger, j., kahn, h., liggett, j., marshall, g. & zack, j. (1999) using multimedia in large-scale computer-based testing programs, computers in human behaviour, 15(3), 283–294. bloom, b. s. (1956) taxonomy of educational objectives: the classification of educational goals. handbook 1. cognitive domain (new york, longman). implementation of computer assisted assessment 225 bloom, b. s., hastings, j. t. & madaus, g. f. (1971) handbook on formative and summative evaluation of student learning (new york, mcgraw-hill books). boyle, a. & o’hare, d. (2003) finding appropriate methods to assure quality computer-based assessment development in uk higher education, proceedings of the 7th international computer assisted assessment conference, (loughborough, loughborough university), 67–82. bs7988 (2002) code of practice for the use of information technology (it) in the delivery of assessment bull, j., conole, g., davis, h. c., white, s., danson, m. & sclater, n. (2002) rethinking assessment through learning technologies, proceedings of the ascilite 2002 (auckland, unitec), 1–12. bull, j. & mckenna, c. (2001) blueprint for computer-assisted assessment (loughborough, loughborough university). burstein, j., leacock, c. & swartz, r. (2001) automated evaluation of essays and short answers, proceedings of the 5th international computer assisted assessment conference (loughborough, loughborough university). burton, r. f. (2001) quantifying the effects of chance in multiple choice and true/false tests: question selection and guessing of answers, assessment and evaluation in higher education, 26(1), 41–50. burton, r. f. & miller, d. j. (1999) statistical modelling of multiple-choice and true/false tests: ways of considering, and of reducing, the uncertainties attributed to guessing, assessment and evaluation in higher education, 24(4), 399–411. bush, m. (1999) alternative marking schemes for on-line multiple choice tests, proceedings of the 7th annual conference on the teaching of computing (belfast, elsevier). callear, d. & king, t. (1997) using computer-based tests, alt-j, 5(1), 27–32. cassady, j. c. & johnson, r. e. (2002) cognitive test anxiety and academic performance, contemporary educational psychology, 27(2), 270–295. christie, j. r. (1999) automated essay marking for both content and style, proceedings of the 3rd annual computer assisted assessment conference (loughborough, loughborough university). ciad (2003) summary of question styles. available online at: http://www.derby.ac.uk/ciad/ ciastyles.html (accessed 30 june 2003). clariana, r. & wallace, p. (2002) paper-based versus computer-based assessment: key factors associated with test mode effect, british journal of educational technology, 33(5), 593–602. cox, k. & clark, d. (1998) the use of formative quizzez for deep learning, computers and education, 30(3), 157–167. crisp, b. r. (2002) assessment methods in social work education: a review of the literature, social work education, 21(2), 259–269. croft, a. c., danson, m., dawson, b. r. & ward, j. p. (2001) experience of using computer assisted assessment in engineering mathematics, computers and education, 37(1), 53–66. curtis, p. (2003) missing paper sparks exam reprint (london, guardian). daly, c. & waldron, j. (2002) introductory programming, problem solving and computer assisted assessment, proceedings of the 6th international computer assisted assessment conference, (loughborough, loughborough university), 95–106. daly, j. (2002) an xml question bank using microsoft office, proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university), 107–118. davies, p. (2002) there’s no confidence in multiple-choice testing, proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university), 119–132. dowsing, r. d. (1998) flexibility and the technology of computer aided assessment, proceedings of the ascilite 1998 (wollongong, university of wollongong), 163–171. dyson, m. c. & kipping, g. j. (1997) the legibility of screen formats: are three columns better than one? computers and graphics, 21(6), 703–712. ebel, r. l. (1972) essentials of educational measurement (englewood cliffs, prentice-hall). 226 g. sim et al. frohlich, r. (2000) keeping the wolves from the door, wolves in sheep clothing, that is, proceedings of the 4th international computer assisted assessment conference (loughborough, loughborough university). graham, d. (2004) a survey of assessment methods employed in uk higher education programmes for hci courses, proceedings of the 7th hci educators workshop (preston, ltsn), 66–69. haladyna, t. m. (1996) writing test items to evaluate higher order thinking (needham heights, allyn & bacon). harper, r. (2002) allowing for guessing and for expectations from the learning outcomes in computer-based assessments, proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university), 139–150. harper, r. (2003) correcting computer-based assessment for guessing, journal of computer assisted learning, 19(1), 2–8. hatton, s., boyle, a., byrne, s. & wooff, c. (2002) the use of pgp to provide secure email delivery of caa results, proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university), 149–160. hawe, e. (2003) it’s pretty difficult to fail; the reluctance of lecturers to award a fail grade, assessment and evaluation in higher education, 28(4), 371–382. heinrich, e. & wang, y. (2003) online marking of essay-type assignments, proceedings of the world conference on educational multimedia hypermedia and telecommunications (hawaii, aace), 768–772. herd, g. & clark, g. (2002) computer assisted assessment implementing caa in fe sector in scotland: question types (glenrothes, glenrothes college). herd, g. & clark, g. (2003) caa implementation in the fe sector in scotland (glenrothes, glenrothes college). hesa (1995) students in higher education institutions 1994/95 (london, hmso). hesa (2002) students in higher education institutions 2001/02 (london, hmso). hindle, s. (2003) careless about privacy, computers and security, 22(4), 284–288. horney, w. (2003) assessing using grade-related criteria: a single currency for universities? assessment and evaluation in higher education, 28(4), 435–454. jafarpur, a. (2003) is the test constructor a facet? language testing, 20(1), 57–87. jefferies, p., constable, i., kiely, b., richardson, d. & abraham, a. (2000) computer aided assessment using webct, proceedings of the 4th international computer assisted assessment conference (loughborough, loughborough university). king, t. & duke-williams, e. (2002) using computer aided assessment to test higher level learning outcomes, proceedings of the 5th international computer assisted assessment conference (loughborough, loughborough university). kleeman, j. & osborne, c. (2002) a practical look at delivering assessment to bs7988 recommendations, proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university), 163–170. knight, p. (2001) a briefing on key concepts formative and summative, criterion and norm-referenced assessment (york, ltsn generic centre). latu, e. & chapman, e. (2002) computerised adaptive testing, british journal of educational technology, 33(5), 619–622. lay, s. & sclater, n. (2001) question and test interoperability: an update on national and international developments, proceedings of the 5th international computer assisted assessment conference (loughborough, loughborough university). liu, m., papathanasiou, e. & hao, y. (2001) exploring the use of multimedia examination formats in undergraduate teaching: results from the fielding testing, computers in human behaviour, 17(3), 225–248. lloyd, d., martin, j. g. & mccaffery, k. (1996) the introduction of computer based testing on an engineering technology course, assessment and evaluation in higher education, 21(1), 83–90. implementation of computer assisted assessment 227 loewenberger, p. & bull, j. (2003) cost-effectiveness analysis of computer-based assessment, alt-j, 11(2), 23–45. luck, m. and joy, m. (1999) a secure on-line submission system, software—practice and experience, 29(8), 721–740. macdonald, j. & twining, p. (2002) assessing activity-based learning for a networked course, british journal of educational technology, 33(5), 603–618. mackenzie, d. (1999) recent developments in the triartite interactive assessment delivery system (triads), proceedings of the 3rd annual computer assisted assessment conference (loughborough, loughborough university). mackenzie, d., hallam, b., baggott, g. & potts, j. (2002) triads experiences and developments. a panel discussion, proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university). mackenzie, d. & o’hare, d. (2002) empirical prediction of the measurement scale and base level ‘guess factor’ for advanced computer-based assessment, proceedings of the 6th international conference of computer aided assessment (loughborough, loughborough university), 187–201. mackenzie, d., o’hare, d., paul, c., boyle, a., edwards, d., willimas, d. & wilkins, h. (2004) assessment for learning: the triads assessment of learning outcomes project and the development of a pedagogically friendly computer-based assessment system, in: d. o’hare & d. mackenzie (eds) advances in computer aided assessment (birmingham, seda), 11–25. mason, s. (2003) electronic security is a continous process, computer fraud and security, 2003(1), 13–15. maughan, s., peet, d. & willmot, a. (2001) on-line formative assessment item banking and learning support, proceedings of the 5th international computer assisted assessment conference (loughborough, loughborough university). mayer, r. e. (2002) a taxonomy for computer-based assessment of problem solving, computers in human behaviour, 18(6), 623–632. mayes, d. k., sims, v. k. & koonce, j. m. (2001) comprehension and workload differences for vdt and paper-based reading, international journal of industrial ergonomics, 28(6), 367–378. mcalpine, m. (2002) principles of assessment (luton, caa centre). mccabe, m. & barrett, d. (2003) caa scoring strategies for partial credit and confidence levels, proceedings of the 7th international computer assisted assessment conference (loughborough, loughborough university), 209–219. mckenna, c. & bull, j. (2000) quality assurance of computer-assisted assessment: practical and strategic issues, quality assurance in education, 8(1), 24–31. mclaughlin, p. j., fowell, s. l., dangerfield, p. h., newton, d. j. & perry, s. e. (2004) development of computerised assessment (triads) in an undergraduate medical school, in: d. o’hare & d. mackenzie (eds) advances in computer aided assessment (birmingham, seda), 25–32. mills, c. n., potenza, m. t., fremer, j. j. & ward, w. c. (2002) computer-based testing. building the foundation for future assessments (mahwah, lawrence erlbaum associates). morgan, m. r. j. (1979) mcq: an interactive computer program for multiple-choice self testing, biochemical education, 7(3), 67–69. nugent, g. (2003) on-line multimedia assessment for k-4 students, proceedings of the world conference on educational multimedia, hypermedia and telecommunications (hawaii, aace), 1051–1057. o’leary, r. & cook, j. (2001) wading through treacle: caa at the university of bristol, proceedings of the 5th international computer assisted assessment conference (loughborough, loughborough university). pain, d. & le heron, j. (2003) webct and online assessment: the best thing since soap? educational technology and society, 6(2), 62–71. 228 g. sim et al. paterson, j. s. (2002) linking on-line assessment in mathematics to cognitive skills, proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university), 295–306. paxton, m. (2000) a linguistic perspective on multiple choice questioning, assessment and evaluation in higher education, 25(2), 109–119. phipps, l. & mccarthy, d. (2001) computer assisted assessment and disabilities, proceedings of the 5th international computer assisted assessment conference (loughborough, loughborough university). pollock, m. j., whittington, c. d. & doughty, g. f. (2000) evaluating the costs and benefits of changing to caa, proceedings of the 4th international computer assisted assessment conference (loughborough, loughborough university). powers, d. e., burstein, j. c., chodorow, m., fowles, m. e. & kukich, k. (2002) stumping e-rator: challenging the validity of automated essay scoring, computers in human behaviour, 18(2), 103–134. race, p. (1995) the art of assessing, the new academic, 4(3). reid, n. (2002) designing online quiz questions to assess a range of cognitive skills, proceedings of the world conference on educational multimedia, hypermedia and telecommunications (denver, aace), 1625–1630. ricketts, c. & wilks, s. (2002a) what factors affect students opinions of computer-assisted assessment? proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university), 307–316. ricketts, c. & wilks, s. (2002b) improving student performance through computer-based assessment: insights from recent research, assessment and evaluation in higher education, 27(5), 475–479. sabar, n. (2002) towards principled practice in evaluation: learning from instructors’ dilemmas in evaluating graduate students, studies in educational evaluation, 28(4), 329–345. sambell, k., sambell, a. & sexton, g. (1999) students’ perception of the learning benefits of computer-assisted assessment: a case study in electronic engineering, in: s. brown, j. bull & p. race (eds) computer-assisted assessment in higher education (birmingham, seda), 179–191. schenkman, b., fukuda, t. & persson, b. (1999) glare from monitors measured with subjective scales and eye movements, displays, 20, 11–21. sclater, n., davis, h. c., white, s. a., conole, g. c. & danson, m. (2003) technologies for online interoperable assessment, proceedings of the cal03 (belfast, elsevier). sclater, n. & howie, k. (2003) user requirements of the ‘ultimate’ online assessment engine, computers and education, 40(3), 285–306. sealey, c., humphries, p. & reppert, d. (2003) at the coal face: experience of computer based exams, proceedings of the 7th international computer assisted assessment conference (loughborough, loughborough university), 357–376. senda (2001), special educational needs and disability act 2001, http://www.hmso.gov.uk/ acts/acts2001/20010010.htm (accessed 10 march 2004) sim, g., malik, n. a. & holifield, p. (2003) strategies for large-scale assessment: an institutional analysis of research and practice in a virtual university, proceedings of the 7th international computer assisted assessment conference (loughborough, loughborough university), 379–390. stephens, d. (1994) using computer-assisted assessment: time saver or sophisticated distractor? active learning, 1, 11–15. stephens, d., bull, j. & wade, w. (1998) computer-assisted assessment: suggested guidelines for an institutional strategy, assessment and evaluation in higher education, 23(3), 283–294. stephens, d. & mascia, j. (1997) results of a survey into the use of computer-assisted assessment in institutions of higher education (loughborough university). available online at: http:// www.lboro.ac.uk/service/ltd/flicaa/downloads/survey.pdf (accessed 18 june 2004). implementation of computer assisted assessment 229 stevenson, a., sweeny, p., greenan, k. & alexander, s. (2002) integrating caa within the university of ulster, proceedings of the 6th international computer assisted assessment conference (loughborough, loughborough university), 329–340. tannenbaum, r. s. (1999) theoretical foundations of multimedia (new york, w.h. freeman). thomas, p., price, b., paine, c. & richards, m. (2002) remote electronic examinations: student experience, british journal of educational technology, 33(5), 537–549. valenti, s., cucchiarelli, a. & panti, m. (2002) computer based assessment systems evaluation via the iso90126 quality model, journal of information technology education, 1(3), 157–175. walker, d. m. & thompson, j. s. (2001) a note on multiple choice exams, with respect to students’ risk preference and confidence, assessment and evaluation in higher education, 26(3), 261–267. warburton, b. & conole, g. (2003) caa in uk heis—the state of the art? proceedings of the 7th international computer assisted assessment conference (loughborough, loughborough university), 433–441. white, s. & davis, h. c. (2000) creating large scale test banks: a briefing for participative discussion and agendas, proceedings of the 4th international computer assisted assessment conference (loughborough, loughborough university). whittington, d. (1999) technical and security issues, in: s. brown, j. bull & p. race (eds) computer assisted assessment in higher education (birmingham, seda), 21–28. wiles, k. (2002) accessibility and computer-based assessment: a whole new set of issues? in: l. phipps, a. sutherland & j. seale (eds) access all areas: disability, technology and learning (oxford and york, alt/techdis), 61–66. wiles, k. & ball, s. (2003) constructing accessible cba: minor works or major renovations? proceedings of the 7th international computer assisted assessment conference (loughborough, loughborough university), 445–451. wiltfelt, c., philipsen, p. e. & kaiser, b. (2002) chat as media in exams, education and information technologies, 7(4), 343–349. wood, d. a. (1960) test construction (columbus, charles e. merrill books). yorke, m., barnett, g., bridges, p., evanson, p., haines, c., jenkins, d., knight, p., scurry, d., stowell, m. & woolf, h. (2002) does grading method influence honours degree classification? assessment and evaluation in higher education, 27(3), 269–279. zakrzewski, s. & steven, s. (2003) computer-based assessment: quality assurance issues, the hub of the wheel, assessment and evaluation in higher education, 28(6), 609–623. staff edutainment on a corporate intranetlessons learned vlad lonesco royal institute of technology, stockholm, sweden, email: vladi@gtkth.se this paper discusses the results from a case study of an intranet-based staff education programme at the national swedish telecommunications corporation. the programme attempts to convey the environmental benefits of telecommunication services, aiming at increased staff awareness and new business opportunities. a multidisciplinary production team packaged the factual content in a rich mixture of audio and video clips, text, images, animations, quizzes, games, tests and polls. a standard web browser with downloadable plug-ins is used to view the material. the self-paced studies on the intranet are followed by a seminar, combining the advantages of individual and teacher-assisted education. the impact of the programme has been studied partly through a questionnaire distributed to a number of users, and partly through interviews. the results show that intranet-based 'edutainment' is a practicable and successful tool for organizational learning. ease of use, interactivity, ease of distribution and renewal, flexibility in time and space, are just some of the benefits offered. however, a number of pedagogical, organizational and technical requirements must be met in order to succeed context, pedagogical aims and standards, content, information structure, user characteristics, management commitment, technical platform, media types, the production team, user interface and user tests are just a few of the important factors that must be considered this paper attempts to identify some of the factors essential to successful staff education on the corporate intranet, discussing observations and lessons learned from the case study. web-based learning the educational level required of a successful member of today's society and working environments is becoming higher and higher. national and commercial education systems are struggling, as increasing numbers of people must be trained within harsh budget constraints. in this context, internet technology has emerged as a new tool in the field of education and training, and the number of applications of this technology is increasing steadily. in order to assess its full potential, it is important to study cases in both educational and business settings. the foremost advantage of internet technology is ease of distribution. once a web-site is launched, the material can be accessed by an audience of any size without dramatic cost 54 an-] vb/ume 7 number i increases, as no physical objects need be distributed. another advantage is ease of maintenance. changes in online material take effect instantly, compared with the much longer revision cycles of, say, printed material. online material can thus be kept up-to-date with reasonably little effort. a third advantage is ease of use, as the web interface is familiar to an increasingly large number of people, both at home and in the working environment convenience is also an advantage. students are no longer constrained in time or space. they can study at a time and place of their own choice, as long as a computer and the network are available. however, online learning also has some disadvantages. firstly, technology evolves very rapidly, making it difficult to create long-term, stable and flexible applications. secondly, it is difficult to cater for students' different learning styles. online material must be carefully structured, in order to offer the advantage of non-linearity without creating confusion. studies show that information structure has significant influence on learning performance (quentin-baxter, 1998; paolucci, 1998). thirdly, an increased burden of responsibility is placed on the students. they often have to study individually, without motivational support from fellow students, and with delayed support and feedback. despite these disadvantages, web-based learning may still be the answer to the increased educational needs of today. technology is still in its infancy, and further research and development will most likely lead to better results. the purpose of this paper is to describe the experiences from a multimedia staff education programme on the intranet of the swedish telecommunications corporation telia, and to discuss some factors that have been found important. telia corporation the swedish telecommunications corporation telia has approximately 26,000 employees. their mission statement is to 'enhance the quality of life, the environment and the competitive ability of individuals and organizations by bringing them together with easy-to-use telecom-based information services' (telia, 1998a). telia's ambitious environmental programme has been in progress for many years. apart from aiming at running its own business in an environmentally friendly way through the choice of materials and procedures, telia's ambition is to 'help its customers adapt their operations to environmental demands and demonstrate the environmental advantages of telecommunications' (telia, 1998b). in the light of this policy, telia decided to develop an educational programme for all employees. the goal of the programme is to increase awareness of the company's environmental policy, and of the environmental benefits of telecommunication services, hoping to motivate the staff both at work and at home. for instance, using phones, email, electronic commerce and digital documents can reduce transport and thus pollution. the decision was made to deploy the education programme on telia's intranet, in order to illustrate these benefits, and also to use and evaluate the latest technology available. the education programme the education programme consists of two parts: (i) the multimedia material on the intranet, which each employee goes through individually, and (ii) a subsequent traditional seminar, led by an expert on environmental issues. the seminar is meant to offer employees an opportunity to discuss issues with one another, ask questions and share ideas. a comprehensive description of the programme can be found in ionesco (1998). 55 vlod tonesco staff edutainment on a corporate intranet lessons learned the online material contains a large amount of facts and figures about the environment, related both to the company's business activities and to individual behaviour at work and home. the aim is to highlight the environmental benefits of telecommunications products and services, to arouse a sense of involvement in the employees and, in the long run, to create new business opportunities. in order to attract and retain the users' interest it was decided to give the website an informative, and yet entertaining profile. the decision was made to create an 'edutainment site', in which educational objectives can be realized in an entertaining manner. several of the technologies available on the web today have been used to present the information. frames, simple html and javascript code, a mixture of gif and jpeg images, shockwave and flash animations, streamed realmedia audio and video files, all contribute to a rich and varied design. further, a number of cgi-scripts are used for the navigation system, and to create games, polls, quizzes and tests. to facilitate navigation within the online material, the information has been divided into modules. the metaphor underlying the top level of the information structure is that of a city, with different areas and buildings, each linked to a module with a specific topic. all the modules have a similar tree structure, with a summary describing the module, a main scene (possibly made up from a chain of web pages) conveying the message, a list of internal links to relevant products and services, and a list of external links to successful implementations of the module topic. the current position in the information structure is always indicated on the web and a short cut to the top level of the module is always present a menu with short cuts to all the modules is located on the side of the screen, and it also indicates which modules have been visited by the user. this, of course, requires users to register the first time they enter the site, and then to log in for every new session. the individual progress of each user is saved on the server, enabling access from any computer on the network. the site contains a glossary covering most of the terminology used on the site. links to individual entries are abundant throughout the material, but the glossary can also be used as a separate module for browsing through all the entries. some entries are only short descriptions, while others consist of longer articles and images. a proposal box on the site provides everyone with the opportunity to submit comments and suggestions to the educators and to the staff responsible for the environmental programme. implementation some implementation issues will be briefly discussed here. for a more detailed discussion, see jonsson(1998). the production process was initiated by the environmental department at telia. business and learning goals were carefully defined and a specification was written. a virtual organization spanning over several smaller firms and consultants was formed, as no single company was found to have all the expertise required. the project team consisted of managers, environmental experts, content creators, web designers, artists, audio and video experts, and programmers. managing and co-ordinating a project team of such diversity can be a challenge. the terminology used by the different professionals differs, and insight into the practical limitations of others' domains is sometimes insufficient. it is of little use, for instance, when the design team thinks of a fanciful animation, if it cannot be implemented by the programming team. 56 aa-j volume 7 number i after a few such failures, co-operation was improved by enforcing a feedback mechanism, as well as detailed sub-goals, schedules and deadlines for all involved. form and content were given equal importance, taking cost, time and effort into consideration. the information content was structured and subdivided into modules, and scripts were written for each module. based on the script and on practical technical and economic considerations, different types of multimedia objects were developed for each module. a user-friendly interface was outlined and tested in co-operation with a sample of representative users. templates were developed to simplify the coding stage and to ensure consistency. as the project proceeded, user tests were performed regularly, applying iterative design until the final stage. method in the summer of 1998, six months after the initial installation, the programme was subject to a survey, partly based on the methodology described by maxwell (1996). the aim was to find both quantitative and qualitative measures of its outcome, and to identify critical factors. eighty-seven users were selected from the log files on the server and were asked to fill in a web-based questionnaire, containing both open and closed questions. forty-seven answers were received. the selection process was not entirely random, since email addresses were difficult to find for all users. in-depth interviews were carried out with a manager, a developer and two users. minutes from user tests during the implementation phase have also been used as data for this paper. the methodologies proposed by quentin-baxter and dewhurst (1992) or yildiz and atkins (1993) would have provided more data for the study. however, their level of detail is more suitable for fundamental research, and they could not be applied in this project due to limited resources. results low penetration during the first six months, 360 new users registered on the site, of which 150 completed the entire programme. the low penetration is accounted for by three factors. first, the programme is not compulsory. in the future, all telia employees will go through the programme, but for the moment it is voluntary. in today's stressed working situation it is difficult to motivate staff to find the three to four hours necessary to go through the material. second, difficulties with browser plug-ins on client computers were not resolved in time. normally at telia, installation and upgrading of software on desktop computers is handled remotely and automatically, at night, from a central server. thus, all computers have correct installations of the latest software version. productivity losses are avoided, and maintenance costs, which otherwise constitute a large part of the total cost of ownership for computer systems (gartner, 1994), are kept at a minimum. however, because of licence fees, some of the required browser plug-ins were not installed on all computers. instead, new users had the option to go through an introductory module. this module not only explains various features of the site, but also performs tests on the client computer in order to determine if the necessary plug-ins are present. if this is not the case, the 57 vlod lonesco staff edutainmerrt on a corporate intranet lessons learned user receives installation instructions. however, this proved too difficult an obstacle to users without a technical background. finally, the programme was not properly advertised on telia's internal information channels, many potential users remaining unaware of its existence. to improve user participation, telia took several measures. the required plug-ins were incorporated in the automated software installation scheme. heavy advertising on the intranet will bring people's attention to the project a contest with attractive prizes will hopefully be a sufficient incentive to complete the programme. high appreciation the low penetration is paired with high appreciation, as user reactions have been very positive. for instance, 97 per cent of the users in the survey rated the overall impression of the site as 'good' or 'very good', 98 per cent appreciated the mixture of media types, and all users found this instance of edutainment superior to, say, a traditional booklet. even if the users having completed the programme so far are lead users, with a special interest in environmental issues, the figures are still very encouraging. the interviews and answers to the open questions in the questionnaire also indicate that users found the site refreshing and inspiring. the varied media types conveyed the information and the underlying message, while retaining their interest. several users explicitly mentioned that the programme did not suffer from the kindergarten pace and mentality present in some other multimedia applications. the information content was perceived as useful and relevant by 96 per cent of the users. still, some users demanded a closer connection to their respective business areas, while others found the material too wordy, arguing that the three to four hours necessary to go through it are too many. either way the content is balanced, it will be difficult to find a solution that satisfies each and everyone. the best one can hope for is a reasonable compromise. the user interface was perceived as functional and easy to use, consistency and simplicity being identified as two key factors. navigation through the material was well supported, as the current location was always indicated together with clearly marked links. the introductory module contributed to the success, explaining various features of the site in detail, and giving the user a feeling of what to expect and how to act. an interesting point to note is that while the glossary was extensively used, the annotation tool was only used by a handful of users. apparently it added no value, and scribbling on a piece of paper was found superior. lessons learned during the survey process, it became increasingly obvious that it would have been useful to establish clear and measurable goals at the beginning of the project. however, the goal is not only immediate and cognitive, but also long-term and behavioural, making it difficult both to define and to achieve. an evaluation module should also have been incorporated in the on-line material, as immediate feedback from the users is more valuable than delayed feedback. better identification of users and logging of user behaviour should have been implemented, in order to find out which tools are used, how often, and which parts of the programme are in most need of improvement. 58 ati-j volume 7 number i as contact among students, and between students and educators enhances the learning process, communication should be improved, for instance by using listservs and newsgroups (gibson and rutherford, 1998). in the programme, only a simple email function was present in the navigation toolbar, meant to be used to submit questions and feedback to the educators. however, this was seldom used, and no interaction took place until the seminar. as telia relies heavily on its intranet for the information dissemination, most office workers regard themselves as experienced web users (88 per cent in the survey), and have their own desktop computers. still, user tests revealed that users have no in-depth understanding of how browsers and the web work. a simple and user-friendly interface is thus of utmost importance, especially as the long-term goal is to let all telia's employees go through the programme, not only the office workers. users should not be expected to install any software without the help of the support department. the best solution is automated centralized installation. management commitment is just as important as implementation issues. users must feel that there is a purpose with the programme, that it is not just a fad, or mere empty words. different projects and a high workload all compete for employees' time. to avoid conflicts of interest, priorities must be clarified and time must be allocated for education. follow-up and continuous updates must be undertaken to maintain credibility. the main technological difficulties were caused by somewhat capricious plug-ins, behaving differently in different browsers and on different platforms. unfortunately, plug-ins are essential if the site is to contain more than simple text. thus, it is important to specify exactly what hardand software is going to be used, and to perform extensive testing before the material is released. the negotiation and co-ordination required between the subsidiary it departments of a large corporation such as telia can be an unwieldy process. the corporate intranet offers a more stable environment and higher capacity than the internet. still, bandwidth proved to be an area of concern, as users in remote geographical locations complained of excessive download times. possible solutions are proxy servers, improved infrastructure, and optimization and compression of the multimedia objects. in conclusion, the telia case indicates that a multimedia staff education programme on the intranet is indeed a viable and successful solution, being appreciated by the users, and offering several advantages. hopefully, the concept can be further improved, based on the observations presented in this paper. references gartner (1994), the cost of lan computing: a working model, gartner group inc. gibson, j. and rutherford, p. (1998), 'learners are teachers too in our virtual classroom', proceedings of the seventh international www conference, 718-20. ionesco, v. (1998), 'edutainment on the intranet', proceedings of european multimedia, microprocessor systems and electronic commerce conference, 339-46. jonsson, a. (1998), 'producing distributed education for the corporate intranet', proceedings of european multimedia, microprocessor systems and electronic commerce conference, 294-300. mod tonesco staff edutainment on a corporate intranet lessons learned maxwell, j. (1996), qualitative research design an iterative approach, london: sage publications. paolucci, r. (1998), 'the effects of cognitive style and knowledge structure on performance using a hypermedia learning system', journal of educational multimedia and hypermedia, 7 (2), 123-50. quentin-baxter, m. (1998), 'hypermedia learning environments limit access to information', proceedings of the seventh international www conference, 587-90. quentin-baxter, m. and dewhurst, d. (1992), 'a method for evaluating the efficiency of presenting information in a hypermedia environment', computers and education, 18, 178-82. telia (1998a), 'telia business concept', http://www.telia.se/index_en.html. telia (1998b), 'telia environmental policy', http://hanl6ns.telia.se/telia/thk/thkjef01.nsf/ dockey/environment96en06. yildiz, r. and atkins, m. j. (1993), 'evaluating multimedia applications', computers and education, 21 (1), 133-9. 60 the evolving landscape of learning technology gráinne conole university of southampton email: g.c.conole@soton.ac.uk this paper provides an overview of the current and emerging issues in learning technology research, concentrating on structural issues such as infrastructure, policy and organizational context. it updates the vision of technology outlined by squires' (1999) concept of peripatetic electronic teachers (pets) where information and communication technologies (ict) provide an enabling medium to allow teachers to act as freelance agents in a virtual world and reflects to what extent this vision has been realized the paper begins with a survey of some of the key areas of ict development and provides a contextualizing framework for the area in terms of external agendas and policy drivers. it then focuses upon learning technology developments which have occurred in the last five years in the uk and offers a number of alternative taxonomies to describe this. the paper concludes with a discussion of the issues which arise from this work. foreword this article is dedicated to david squires and aims, along with the other articles in this issue, to celebrate david's research work and his substantial contribution to learning technology research. on a personal note, i knew david for a relatively short period of time but he had a significant influence on my own career; in particular he acted as a valuable sounding-board at a stage when i was making major career-changing decisions. he had a relaxed and easy-going approach, was always willing to work collaboratively with others and had a genuine enthusiasm for learning technology research. his support and friendship were much appreciated and he is very much missed. introduction in his paper 'peripatetic electronic teachers in higher education', squires explored the idea of information and communication technologies (ict) providing a medium to enable higher education teachers to act as virtual freelance agents (squires, 1999). in this futuristic 4 alt-j volume 10 number 3 vision, peripatetic electronic teachers (pets) existed as multiple telepresences (pedagogical, professional, managerial and commercial) in pet-worlds: global networked environments which supported advanced multimedia features. squires opens by stating that most people agree that the role of teachers will change with the advent of ict. the article explores the possibility that ict could break exclusive links between educators and a single institution. squires states: just as future learners may be seen as clients contracting to receive educational provision from a range of providers, educators may be seen as independent workers offering their services to learners on demand. these educators will not be confined to the classroom, rather they will be electronic workers providing a virtual presence in public spaces, the workplace and the home. a new type of peripatetic electronic teacher (pet) will emerge. this article provides an overview of learning technologies and considers the extent to which this vision has been achieved. a review of pedagogic developments in this area has already been undertaken (thorpe, 2002), charting the shift from focusing on individuals to the social context of learning. thorpe concentrated on the effects of changes in practice which have arisen as the use of ict has become more widespread. she maps the changes to relevant pedagogical theories, in particular drawing on the notion of communities of practice (lave and wenger, 1990) and the shift from independent to collaborative learning. this article is complementary to thorpe's review, concentrating on the following aspects: • the impact of national and international policy agendas and drivers; • an outline and categorization of key funded technological and related development programmes (focusing particularly on those in the uk); • an overview of some of the resultant issues; • a synthesis of key emergent themes. background over the past decade a considerable wealth of digital resources and information and communication technology (ict) tools have been developed to support learning and teaching. these have been funded from a variety of sources such as the national science foundation (nsf) and the sloane foundation in the states, and the joint information systems committee (jisc) and the new opportunities fund (nof) in the uk. these research and development programmes have allowed academics to experiment with the use of new technologies and integrate digital resources with more traditional teaching methods. this has raised a host of pedagogical and technical issues, such as: • how effective are these new approaches? • what are the underpinning technical, managerial and infrastructural requirements? • what protocols and standards are needed to ensure interoperability? • how do we deal with issues such as copyright and the rise in plagiarism? this article aims to provide an overview of these activities and some of the emergent 5 grainne conole the evolving landscape of learning technology themes and focuses primarily on higher education; guile and hayton have published a review of the impact of ict in further education, summarizing key implications (guile and hayton, 1999). a considerable body of research has been carried out on the development and use of learning technologies. the literature on the area has been expanding exponentially in the last five years, including the emergence of a range of specialized journals (such as alt-j, education, technology and society, the journal of interactive multimedia education, the asynchronous learning network journal, and more recently educational, information and communication) and a series of notable summaries for the area (bates, 1995; hiltz, 1995; mason, 1998; collis and moonen, 2001; laurillard, 2001). in parallel specialized conferences have arisen (such as alt-c, ascilite, edmedia, icalt, icce, ideas in cyberspace education, and the networked learning conference, to name but a few). finally there are a number of useful introductory guides to the area (forsyth, 1996; barnett, brunne et al, 1997; seale and rius-riu, 2001). there is a growing trend towards taking stock of what has been achieved. squires et al. have collated a representative sample of articles published in alt-j over the last decade (squires, conole et al, 2000). lockwood and gooley have co-edited a collection of essays on successful developments in online and web-based learning (lockwood and gooley, 2001). in addition several books have focused on networked learning, including two excellent overviews of its foundations and theoretical perspectives (mcconnell, 2000; steeples and jones, 2002). there have been several recent reviews of the area. williams reviewed the literature on learning online, concentrating on the problems involved in transferring courses online (williams, lock, crisp and longstaffe, 1995; williams, 2000; williams, 2002), whilst bonk provides a survey of the use of ict in teaching in the states (bonk, 2001). an analysis and mapping of ict developments to underpinning learning theories have been carried out independently by ravenscroft and thorpe (ravenscroft, 2002; thorpe, 2002). since no article could adequately cover the breadth and scope of this area, this article instead aims to provide a snapshot of recent and ongoing activities and provide some coherent categorization of these. it will begin by focusing on the context in which this work arises. the wider context drivers and initiatives national strategies and policies for the higher education sector and, in particular, the role of information and communication technology (ict) to support learning and research, sit within a context of rapid technological change (conole, 2001). this provides immense opportunities, whilst at the same time making firm commitment to specific systems or developments very difficult. over the past decade, there is evidence that this area is maturing; it is becoming integrated into the core business of institutions and is being considered as part of a wider, more generic learning and teaching debate. as might be expected, the technical research aspects become dated very quickly. in contrast the organizational and pedagogical issues remain pertinent and a number of common themes emerge. one of the most influential policy reports, addressing many of these issues, was the dealing report (ncihe, 1997). this was the culmination of the systematic review of higher education, and included at least fifteen recommendations 6 alt-] volume 10 number 3 which make explicit reference to ict. these have significantly influenced the focus and direction of many ict projects in the last five years, as was evident from a recent in-depth analysis of the teaching and learning projects of the jlsc-funded jtap programme (conole, 2002). similar effects have arisen from the fe sector (kennedy, 1998); other notable reports include the fryer report (nagcell, 1997), the booth report (booth, 1998), and associated papers on lifelong learning (dfee, 1998). the dearing report has had a significant influence over the focus and direction of many of the funding programmes and initiatives. several strategic factors have been of particular importance in the last five years. the first is that the higher education funding council for england (hefce) now requires all higher education institutions to have a clear and demonstrable learning and teaching strategy as a condition of funding; closely related to this are institutional information strategies and more recently human resource strategies. taken together, these strategies are an attempt to ensure the use of ict is appropriately embedded into the institution (mcnaught and kennedy, 2000). secondly, one of the most prominent features of the impact of ict on teaching and learning is the way in which ict has acted as a catalyst for opening the debate on learning and teaching issues more generally (littlejohn and cameron, 1999). furthermore, the scaling-up of ict use has had a profound influence on the infrastructure of institutions, raising debates about staff and student training needs, the integration of strategies and future planning (the jisc has funded a range of projects which have addressed these issues and also explored the associated sociocultural issues, in particular see www.jisc.ac.uk/mle). this has now begun to mature as most institutions move towards developing institution-wide managed learning environments. the above indicates that ict is moving from being associated with peripheral innovations and developments to affecting all aspects of learning and teaching, although currently this is probably more true at the strategic than at the operational level. it is clear that the 'ictdebate' can no longer be addressed in isolation, but must be considered' across all institutional and national strategies and policies. the latter point has profound implications on funding mechanisms and national initiatives. initiatives over the past decade reflect these trends. there has been an increased focus on ict in hefce and jisc calls, but also in research-focused programmes through the epsrc and esrc. in addition, the national grid for learning (ngfl) and the university for industry (ufl) have sought to increase the base-level of ict skills within the community and provide a solid technological infrastructure for education from primary through to tertiary level. the shift towards embedding ict is well illustrated by the teaching and learning technology programme (tltp), where the last phase of funding for the programme clearly shifted from development of materials to their integration (hefce 1997). within jisc calls, there has been a greater focus on 'joined-up' technologies and the provision of a solid technical infrastructure with a critical mass of materials through the development of an information environment (ie) of resources, tools and initiatives focusing on the development of managed learning environments. technological developments are not the only influence on higher education; the sector has also been changed by developments such as the government's widening participation agenda and the consequential expansion of student numbers, the collaborative 7 gra/nne conole the evolving landscape of learning technology opportunities and potential threats of increasing globalization, the impact of increased monitoring and quality assurance processes, the increased drive towards professionalization and accreditation, the changing nature and expectations of the student population and the rise of the 'nintendo generation' of computer-literate students (morice, 2000). overview of current ict developments this section provides a broad overview of areas of current interest in the uk. it is a synthesis of a number of recent research studies, reviews and evaluations (bull, conole et al, 2002; conole and bull, 2002; conole, smith and franklin, 2002). learning technology taxonomies learning technology is an inherently multidisciplinary field involving researchers from a wide range of backgrounds (educational research, cognitive psychology, instructional design, computer science, and so on) as well as teaching subject-experts who engage with it as 'end users' or 'consumers'. this multi-disciplinarity is a common feature of emergent research areas, but makes the task of providing an overview of the key research activities difficult. this diversity also makes the task of creating a definitive taxonomy, for learning technologies difficult. this section highlights some of the alternative classification systems that have been developed. beetham has suggested that ict can be classified in three main ways: as a tool, as a resource, or as a medium for communication (beetham, 2001). these can range from simple office tools through specialized databases and applications, to subject-specific tools (chemical molecule drawing packages or cad software for engineers). tools can be used to produce a range of digital resources ranging from simple documents through animated presentations to complex modelling tools. finally ict can be used as a communication mechanism (asynchronous and synchronous). examples of the former include email and discussion boards where the interaction does not occur in real time. in contrast, examples of synchronous communication includes live chat groups and video conferencing. ict tools can be combined in a variety of ways to meet specific purposes and functions, or for use within a specific context. alternatively, ict can be classified by research and development activities (conole 2002), using the following categories: • the development and use of digital resources, tools and software; • the development of supporting infrastructure, underpinning standards and protocols; • research into related pedagogical and sociocultural issues (the use and uptake of ict); • the development of learning environments to support learning, teaching, research and administration. finally a taxonomy can be used which illustrates the breadth of the area spanning the development and use of specific technologies (such as video conferencing), underpinning technologies such as networking and infrastructural issues, holistic and integrative environments such as virtual learning environments (vles) and managed learning environments (mles), as well as associated 'softer' issues such as impact on organizational change and associated staff development needs (conole, 2002). classification then includes: 8 alt-j volume 10 number 3 • experimentation and reporting on novel uses of ict; • review of technologies; • evaluation of the use of ict in teaching and learning; • development of resources and tools; • exploration of associated issues; • synthesis of key findings and production of good practice guidelines, support tools and toolkits, and templates; • methodological issues about the use and impact of evaluation; • critiques on the broader socio-cultural issues; • evaluation of the use of communication technologies; • evaluation of the use of virtual learning environments; • multimedia and adaptive interactive environments. in terms of simplicity the remainder of this section will outline some key developments under the following broad categories which cut across each of the taxonomies described above. some of the features described in the above lists have been omitted as they have already been covered by thorpe (2002). this paper focuses on: • learning environments focusing on vles and mles; • digital libraries, gateways and portals; • resources and tools for learning; • underpinning architecture and standards development; • associated research and impact (changing practice and professional identities; accessibility, ipr, copyright and plagiarism; organizational change; and strategic and policy issues). learning environments the development of robust technical infrastructures for institutions has become a major area of activity in the last five years. these learning environments include enclosed virtual learning environments (vles) as well as more encompassing managed learning environments (mles) which attempt to join up vles with institutional administrative systems. vles are systems that integrate communication tools (email, discussion boards, and so on) with learning resources and assessment tools. there is a host of activities that vles are purported to support, such as, 'encouraging active learning', 'shifting from didactic to facilitative teaching' or 'building online communities' (britain and liber, 1999; armitage, brown et al, 2001). the reality is more complex, involving the context within which these systems are used and how well they are adapted to specific student needs. making a discussion board available on a course will not automatically create a motivated virtual 9 grdinne conole the evolving landscape of learning technology community of collaborating students with a benevolent smiling 'tutor on the side'. therefore although vles can be used to significant effect for the delivery of materials, for supporting learning and as a management tool, this is not an inherent property of the technologies themselves. vles have three main applications: (i) to support learning and teaching, (ii) as an administrative tool, and (iii) for storing information. vles can be used to support learning and teaching; through discussion boards to encourage debate, by extending face-to-face contact, through preand postworkshop topics, peer support groups, live chats around topical issues, or access to subject experts. administratively, communication functions include the use of 'virtual notice boards', preand post-meeting agendas, minutes and papers, the setting up of tailored email groups and mailing lists and the use of discussion boards for debates on meetings or issues. management functionality includes the setting up of staff and student details, collaborative document writing, monitoring/tracking the use of resources, and as a staff development tool to explore the potential of the online environment and ways of sharing expertise. information management includes the development of shared virtual filing cabinets, mechanisms for storing materials, shared repositories of course information (syllabi, timetables, diaries), and so on. knowledge management applications include staff expertise databases, course templates, shared resources, questionbanks, faqs, bookmarks and reference repositories. one of the key properties of vles is the fact that they have the potential to provide a holistic allencompassing platform for supporting both teaching and learning and the associated administration and management. recent interest has shifted to thinking about how these functions can be 'joined up' with other university management systems, such as student records and personnel data. a managed learning environment (mle) will include administrative information about courses, resources, support and guidance, collaborative information, assessment and feedback essentially linking up to back-end office systems and databases. jisc is currently undertaking a large programme of work on the development of managed learning environments (jisc, 2000). one of the key advantages of an mle is the ability to have common authentication between vles and other systems such as email and also enable transfer of results from vles or online assessment systems to the mis. digital libraries, gateways and portals the second major area of development is specialized information gateways. these are increasingly important because of the need to tailor information for specific audiences (currier, brown et al, 2002). a number of niche information gateways have developed. in the uk the main gateway for higher education is the resource discovery network (www.rdttac.uk); similar gateways exist in europe and elsewhere (renardus, 2001). the term 'gateway' is used to describe a range of internet sites that in some way provide access to other, predominantly internet-accessible, resources. gateways are intended to facilitate resource discovery by their target audience, to help their users find what they need via the internet. 'information gateway' is a generic term that refers to a whole range of web-based resource guides pointing to internet information resources, whereas 'subject gateways' are subject-based instances of these (renardus, 2001). resource description and subject classifications are the most important characteristics of such guides. in addition, it is becoming more common for publishers to make journals, books, data archives and 10 alt-j volume 10 number 3 collections available online. considerable thought is being given to how users can access and use these resources in a coherent way; a good example of this is a recent definition of a portal architecture by powell. in his paper, he defines a subject portal as something that ' "brokers" end-user access to a range of services, relevant to a particular subject area, over a number of application protocols (ldapp, whois++, z39.50, ill, ftp) [and] . . . provides "discover", "locate", "request" and embedded "use" . . . functionality' (powell, 2000). powell goes on to describe the architecture for a portal. he defines six types of 'player'; the end-user, presenters, coordinators, mediators, communicators and providers, and describes the roles and relationship between them. at the 'bottom' of the architecture are the information providers, such as subject gateways, data archives, service providers and collections. clearly, these systems represent a rich variety of structures, supporting a range of applications. it is also evident that the boundaries between vles for learning and portals for research are blurring and that many of these systems are starting to provide additional functionality beyond their original scope. for example, the social science research gateway, sosig, primarily focuses on the development of a quality-assured set of resources of relevance to the social science research community. however it also includes a facility to develop personal profiles, 'my sosig', and to network with 'likeminds' using its grapevine facility. sosig is also linked closely to a set of online tutorials which provide an introduction to the internet (virtual training suite, available online at http:llwww.vts.rdn.ac.uk). similarly vles are often now used as gateways or portals to specific types of resources of relevance to a particular learning programme. these developments suggest that evolution is in process, namely that an information source, once established, starts to adapt according to user needs. resources and tools for learning there has been significant investment in the development of resources and tools for learning in the last five years. jisc and the new opportunities fund (nof) have funded a number of large-scale programmes to enable the development of resources for learning and teaching. some examples are described here. the development of computer-aided learning materials to support learning and teaching is nothing new, but perhaps began to accelerate with the emergence of easy-to-use authoring tools (such as authorware and toolbook) that did not require extensive programme skills. and during the late 1980s and early 1990s the teaching and learning technology programme (tltp) produced a range of courseware resources, using these kinds of tools. a more recent example of a resource of this type is the highly successful virtual training suite which is a set of over 50 tutorials which provide a clear subject-specific introduction to the internet. subject areas span fe and he and include subjects as diverse as 'internet for chemists' and 'internet for hairdressers'. the guides sit alongside the relevant subject gateway within the resource discovery network (place, conole et al, 2000a; place, conole et al, 2000b). a second area of significant activity has been the development of models, simulations and virtual worlds to support subject disciplines. for example the virtual economy is a model of the uk economy which very closely mirrors the real model used by the chancellor of the exchequer. students can change a range of variables and study the impact at microand macro-economic levels. the model is linked to a variety of resources and teaching guides and is part of the larger biz/ed business gateway and portal. other projects have grainne conole the evolving landscape of learning technology attempted to simulate practical teaching such as the development of virtual laboratories and virtual fieldtrips, see for example http://www.ch.icac.uk/vchemlab/ and http://www.geog. le. ac uklvfcl. in terms of assessment there have been two main areas of focus: the development of assessment tools for authoring, the creation of subject-specific question banks and the delivery of tests. an example of the former is the castle toolkit which is an online authoring tool that allows course tutors and managers quickly to create interactive quizzes for use on the web (http://www.ie.acuk/castle/). the creation of question banks has been of particular interest in the science and engineering subject domains. an fdtl-funded project e3an is developing a network of expertise in assessment issues within electrical and electronic engineering. a major focus of this project is the development of a test bank of peer-reviewed questions for diagnostic, formative and summative assessment (white, dickens et al, 2002). the technologies for online interoperable assessment (toia) project is developing administrative procedures and question and test interoperability (qti)-based systems and databases to facilitate and promote the uptake of online assessment across the sector (toia, 2002). video conferencing has developed significantly in the last decade and most higher education institutions now have one or more central video conferencing systems in place. these are most commonly used for administrative and research meetings, but they are also used in institutions with multiple sites or where programmes are delivered across more than one institution. scotland has been at the forefront of these developments. the mans initiative and talisman project encourage the use of the atm video conference suites available at all scottish heis, see www.umi.ac.uk for further details. an important example of the use of video conferencing for staff development is a seminar series where institutions across scotland link into monthly video conference discussions and lectures (sesdl, 2002). however desktop video conferencing systems are not yet so commonplace, perhaps because they are still perceived as being unreliable. the rapid growth of multimedia and visualization technologies has led to the exploration of the increased use of images and different visual representations of data (conole, evans et al., forthcoming). there are a number of trends emerging in terms of the use of images. firstly, traditional disciplines are using the technologies to create remote classrooms and laboratories. secondly, library and archive services are finding new ways of storing, indexing and making material available. furthermore, the ability to create, collect, store and retrieve images and multimedia objects of all types will be central to the successful exploitation of ict in the future. underpinning architecture and standards developments there has been considerable work on the development of networking technologies, specifications and associated standards. much of this is being orchestrated internationally through the ieee (www.ieee.org). the standards that underpin the use of networks to deliver educational content include: transport protocols, text, images, sound and moving images, 3-d environments, metadata and rights management. in some areas standards are relatively mature, whilst others are still being contested. one of the most important aspects of this work with respect to learning technologies and education is the development of a standard for learning object metadata (lom). the standard aims to specify the syntax 12 alt-j volume 10 number 3 and semantics of learning object metadata (ltsc, 2001). learning objects are defined as any entity, digital or non-digital, which can be used, reused or referenced during technology-supported learning (littlejohn, forthcoming). examples of technology-supported learning include computer-based training systems, interactive learning environments, intelligent computer-aided instruction systems, distance-learning systems, and collaborative learning environments. examples of learning objects include multimedia content, instructional content, learning objectives, instructional software and software tools, and persons, organizations or events referenced during technology-supported learning. the learning object metadata standards focus on the minimal set of attributes needed to allow these learning objects to be managed, located and evaluated. however work in this area is still been contested; in particular, claims such as 'where applicable, learning object metadata may also include pedagogical attributes such as teaching or interaction style, grade level, mastery level, and prerequisites' (littlejohn, forthcoming) is highly contentious. discussion as outlined above, ict research covers a broad spectrum of activities ranging from a focus on particular technologies or their application to associated organizational, cultural or infrastructural issues. it is evident, however, that there are a number of cross-themes involved in differing degrees: the software and hardware requirements, human resource and skills issues, and associated cultural and organizational aspects. the software research tends to focus on the development and application of new forms of software to support learning and teaching. hardware research is primarily concerned with the development of a robust technical infrastructure to support this; but also investigation of the nature and potential role of peripheral devices and their use. a large area of research in this area is interested in the investigation of the human resource implications of increased use of ict and in particular associated staff and student needs and skills and the type of training, support and development they will need to achieve maximum benefit out of the technologies. research is also exploring the associated attitudes and perceptions of different key stakeholders and the resultant issues of organizational change. other organizational issues include evaluation of the impact on infrastructure and the integration within existing practices. this links closely to institutional strategic and policy debates and the impact on and effects of relevant strategy and policy and the influence of national level agendas and drivers. a consistent issue across many ict-developments is sustainability, including who should maintain and 'own' resources once produced. in recent years funders have begun to address the problem by improving central programme support through the provision of generic workshops, guidance and support, via the creation of networks across common projects and through the introduction of named funding support liaison staff. the teaching and learning technology programme (tltp), for example, provided a comprehensive project management handbook, and the joint information systems committee (jisc) through its 5/99 and 7/99 programmes, has in place a variety of valuable support mechanisms. in addition, the sector now has a developed network of expertise and 'project professionals' in the management of these kinds of programmes (beetham, jones et al, 2001). there has also been a growth in research and development centres, where a 13 grainne conole the evolving landscape of learning technology number of initiatives of this kind co-locate and are then provided with a local support and infrastructure. importantly these kinds of centres provide a degree of stability to projects and associated staff and enable the provision of staff development and sustainability beyond an individual project's lifespan. examples of centres of this kind include the institute for learning and research technology (www.ilrt.bris.ac.uk), ukoln (www.ukoln.aauk), and the learning technology research institute (www.unl.aauk/ltri). how far do the above go towards enabling the vision of squires' peripatetic electronic teacher (pet)? certainly there is now a fairly robust technical infrastructure in place and a rapidly growing number of ways of getting at resources along with learning object metadata to enable teachers to pick and choose courses. there are also tools for assessment and a variety of mechanisms for supporting communication and collaboration online. despite this, the take-up of ict in teaching is still fairly low. this can be attributed to a number of factors. it is evident, for example, that staff and students often still lack the necessary ict skills, as recent reports' on literacy skills and user behaviour highlight (harris, mackenzie et al, 2002; rowlings, 2002). other barriers to large-scale uptake include resistance to change, lack of time for experimentation and implementation, dedicated resources and already adequate mechanisms for ensuring sustainability (conole, 2001; bull, 2002). conclusion this paper has provided an overview of learning technology research over the last five years, focusing on the uk. it has described the issues associated with the increased use of learning technologies and its impact within institutions at both strategic and operational levels. throughout these technology-focused programmes a common aim has been to explore and report on the range of technologies and their uses in learning and teaching. perceived benefits include the potential ict has in terms of offering access to resources, enhancing skills, improving curricula, raising standards, providing mechanisms for sharing practice, accessing a range of online materials' and advice, linking up isolated learners, accessing authentic learning (for example use of native speakers in language learning), accessing a wide range of quality materials, creating opportunities to tailor materials to specific interests and abilities of the individual and overall enabling the potential to make learning more attractive. however, research has also uncovered related issues, such as the need to develop robust underlying infrastructures to support these activities, staff and student training needs, associated cultural and organizational issues, integration with other learning and teaching processes, associated accessibility and legal issues, and the need for transparency over human and technical costs. there is also concern about over-hyping the potential of technology and the fact that fictional vignettes which aim to give example scenarios of the potential opportunities are often technology-centred and date quickly. the paper set out to consider to what extent the infrastructure and context required for squires' vision of a peripatetic electronic teacher has been achieved. it is clear that learning technologies are now no longer marginal activities within institutions. they are a significant part of strategic thinking, as senior managers recognize that a robust technical infrastructure is mission-critical and essential for future success. furthermore technology 14 alt-] volume 10 number 3 tools and resources are increasingly part of the working practice of teachers and students; email is now probably the main communication vehicle, and students and researchers alike increasingly expect resources and papers to be available online through digital libraries and other content repositories. however, there is also a recognition of the limitations of technologies and the value of many traditional practices. increasingly users look to integrate real with virtual practices using each to its particular strengths. so in one sense we are a long way from squires' visionary peripatetic electronic teacher (and may never actually achieve it) but in another sense we could argue that we have moved beyond this, with technologies becoming simply a part of the fabric of higher education, an essential tool, used as and when appropriate. the next few years will see ever increasing changes as applications such as mobile and wireless network impact and it is likely that a similar pattern will arise with innovations giving way to more embedded uses alongside existing practices. acknowledgements my thanks go to martin dyke and martin oliver for all their helpful comments, and to the two referees who both gave extensive and very useful feedback; as a result, the paper is much improved. references armitage, s., brown, t. and jenkins, m. (2001), 'management and implementation of virtual learning environments: a ucisa funded survey', ucisa. barnett, l., brunne, d., maier, p. and warren, a. (1997), using technology in teaching and learning, southampton: interactive learning centre, university of southampton. bates, a. w. (1995), technology, open learning and distance education, london and new york: routledge. beetham, h. (2001), 'using c&it for learning and teaching a handout for the lthe embedding learning technologies programme', plymouth: university of plymouth. beetham, h., jones, s. and gornall, l. (2001), career development of learning technology staff: scoping study, a final report for the jisc jcalt, bristol: university of bristol and university of plymouth, 72. bonk, c. j. (2001), online teaching in an online world, bloomington, in: courseshare.com, available online at http://www.publicationshare.com/, last accessed november 2002. booth, c. (1998), accreditation and teaching in higher education: final report to the committee of vice-chancellors and principals of the universities of the united kingdom. britain, s. and liber, o. (1999), a framework for pedagogical evaluation of virtual learning environments, bangor: university of wales, bangor. bull, j. (2002), implementation and evaluation of computer-assisted assessment final report, luton: university of luton, 28. 15 grainne conole the evolving landscape of learning technology bull, j., conole, g., danson, m., davis, h., sclater, n. and white, s. (2002), rethinking assessment through learning technologies, ascilite 2002, auckland. collis, b. and moonen, j. (2001), flexible learning in a digital world, london: kogan page. conole, g. (2001), drivers and barriers to utilising information and communication technologies, london: ieee conference on teaching and learning. conole, g. (2002), review of jtap projects: perspectives on teaching and learning, commissioned report for techlearn service of the jisc-funded technologies centre, southampton: university of southampton, 40. conole, g. and bull, j. (2002), 'pebbles in the pond evaluation of the caa centre', sixth international computer assisted assessment conference, loughborough. conole, g., evans, j. and simms, e. (forthcoming), 'use and reuse of digital images in teaching and learning', in a. littlejohn (ed.), reusing online resources. conole, g., smith, t. and franklin, t. (2002), a review of the technical applications programme perspectives on teaching and learning, ascilite, auckland. currier, s., brown, s. and ekmekioglu, f. c. (2002), inspiral: linking digital libraries and vles, glasgow: university of strathclyde. dfee (1998), the learning age, london: department for education and employment. forsyth, i. (1996), teaching and learning materials and the internet, london: kogan page. guile, d. and hayton, a. (1999), 'information and learning technology: the implications for teaching and learning in further education', in a. green and n. lucas (eds), fe and lifelong learning: realigning the sector for the twenty-first century, london: institute for education, 113-38. harris, c., mackenzie, a. and davies, h. (2002), 'the big blue: information skills for students: a survey of present practice and a blueprint for the future', final project report for the jisc jcalt. hefce (1997), teaching and learning technology programme phase 3: invitation to bid, hefce, 14/97, bristol: hefce, tltp. hiltz, s. r. (1995), the virtual classroom learning without limits via computer networks, norwood, nj: ablex publishing corporation. jisc (2000), 'mle information pack', www.jisc.ac.uk/mle kennedy, h. (1998), learning works, further education funding council's committee on widening participation in further education. laurillard, d. (2001), rethinking university teaching: a framework for the effective use of educational technology, london: routledge. lave, j. and wenger, e. (1990), situated learning: legitimate peripheral participation, cambridge: cambridge university press. littlejohn, a. (forthcoming), reusing online resources: a sustainable approach to elearning, london: kogan page. 16 alt-j volume 10 number 3 littlejohn, a. and cameron, s. (1999), 'supporting strategic cultural change: the strathclyde learning technology initiative as a model', alt-j, 7 (3), 64-74. lockwood, f. and gooley, a. (2001), innovation in open and distance learning: successful development of online and web-based learning, london: kogan page. ltsc, iltsc (2001), 'standard for information technology education and training systems learning objects and metadata', ltsc. mason, r. (1998), globalising education trends and applications, london and new york: routledge. mcconnell, d. (2000), implementing computer supported cooperative learning, london: kogan page. mcnaught, c. and kennedy, p. (2000), 'staff development at rmit: bottom-up worked serviced by top-down investment and policy', alt-j, 8 (1), 4-18. morice, j. (2000), skills and preferences: learning from the nintendo generation, international workshop on advanced learning technologies: design and development issues, palmerston north, nz, ieee. nagcell (1997), learning for the 21st century, london: national advisory group for continuing education and lifelong learning (nagcell). ncihe (1997), higher education in the learning society (dearing report) (national committee of inquiry into higher education), hmso/ncihe. place, e., conole, g. and sharp, k. (2000a), a 'virtual training suite' for the uk's resource discovery network: enhancing the value of internet resources for learners and teachers, learning to choose, choosing to learn, ascilite 2000, coffs harbour, australia: southern cross university press. place, e., conole, g. and sharp, k. (2000b), a virtual training suite to enhance the value of internet resources. international workshop on advanced learning technologies, palmerston north, nz, ieee. powell, a. and lyon, l. (2000), 'dner portal architecture: positional paper for the distributed network of electronic resources (dner)', http://www.ukoln.ac.uk/distributedsystemsljisc-ielarchldner-arch.html ravenscroft, a. (2002), communities, communication and cognitive change: social processes and designing engaging e-learning discourse, montreal: aace e-learning. renardus (2001), 'gateways defined', positional paper from the european consortium. rowlings, j. (2002), 'monitoring and evaluation framework', jisc jcalt. seale, j. and rius-riu, m. (eds) (2001), an introduction to learning technology within tertiary education in the uk, oxford: association of learning technology. sesdl (2002), the scottish electronic staff development library (sesdl) video conferencing seminar series. squires, d. (1999), 'peripatetic electronic teachers in higher education', alt-j, 7 (3), 52-63. 17 gra/nne cono/e the evolving landscape of learning technology squires, d., conole, g. and jacobs, g. (eds.) (2000), the changing face of learning technology: selected papers from alt-j, cardiff: university of wales press. steeples, c. and jones, c. (2002), networked learning: perspectives and issues, london: springer. thorpe, m. (2002), 'from independent learning to collaborative learning: new communities of practice in open, distance and distributed learning', in m. and n. lea (eds), distributed learning: social and cultural perspectives on practice, london: routledge falmer, 131-51. toia (2002), 'the technologies for online interoperable assessment (toia) project', http://www.strath.ac.uk/projects/toia/, last accessed november 2002. white, s., dickens, k. and davis, h. (2002), engineering: a testbed of engineering questions, auckland, ieee. williams, c. (2000), distance education and ict: teaching and learning online, brighton: university of brighton. williams, c. (2002), 'learning on-line: a review of recent literature in a rapidly expanding field', journal of further and higher education, 26 (3), 263-72. williams, j., lock, a., crisp, j. and longstaffe, a. (1995), the use and capture of images for computer-based learning ii, bristol: university of bristol. 18 doi:10.3402/rlt.v23.21193 critical connections: personal learning environments and information literacy alison hicks* and caroline sinkinson department of arts & humanities, university libraries, university of colorado, boulder, co, usa (received 22 april 2013; final version received 30 october 2014) personal learning environments (ples) and critical information literacies (cils) are two concepts that have been presented as responses to the challenges of the rich and complex information landscape. while both approaches support learners’ critical engagement with new information environments, each was developed within a different field. this paper connects and contrasts ples and cils in order to explore the design of pedagogical responses to the information environment. through a careful examination of ple and cil literature, the paper demonstrates that information literacy education intersects with the concepts and goals of ples. as such, the authors suggest that ple scholarship informed by cil scholarship, and vice versa, will yield a deeper understanding of modern learning contexts as well as a more holistic and responsive learner framework. the example of the research assignment will be used to demonstrate the viability of this approach. with these propositions, the authors invite educators, librarians and information technologists to engage in a dialogue about these concepts and the potential for pedagogical change. keywords: personal learning environments; information literacy; information landscape; self-directed learning introduction educational theorists have long been calling for new pedagogies that are responsive to changing digital information landscapes, that position learners in active and participatory roles and that afford authentic learning opportunities (brown and adler 2008; jenkins 2009; mcloughlin and lee 2008; siemens 2005). these changes mark a significant shift in how scholars understand learning in higher education. the need for alternative educational solutions is amplified by the nature of developing information landscapes, which can be described as the ‘intersubjectively created spaces that have resulted from human interaction, in which information is created and shared and eventually sediments as knowledge’ (lloyd 2010, p. 9). advances in technology, which include lower barriers to participation, have greatly increased the amount of information available on the open web. this explosion in content is forcing educators to reshape their understanding of information, particularly in terms of traditional conceptions of knowledge, authority and validity. as an example, these changes can be seen in shifting practices of scholarship, or ‘scholars’ use of *corresponding author. email: alison.hicks@colorado.edu research in learning technology vol. 23, 2015 research in learning technology 2015. # 2015 a. hicks and c. sinkinson. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons cc-by 4.0 license (http:// creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 participatory technologies and online social networks to share, reflect upon, critique, improve, validate, and further their scholarship’ (veletsianos and kimmons 2012b, p. 768). this also means that educators must re-examine the capacities that learners need to engage in inquiry as well as the way that they meet these needs. personal learning environments (ples) and critical information literacies (cils) are approaches to learning and inquiry that are responsive to these new information landscapes. conceptualised as complexes of activities, tools and values, they were developed within different fields, that is, educational technology and information studies. despite their similar underlying motivations and goals, they have never been compared nor contrasted, probably due to their origins in different areas of study. as such, the first aim of this paper is to explore ple and cil literature to demonstrate how an approach to inquiry that is grounded in ple and cil concepts is better equipped to meet learner demands in emerging information landscapes than previous models. through this examination, the authors will highlight the similarities and differences between each approach as well as the ways in which the two concepts complement each other. this leads to the secondary goal of the paper, which is to contend that ple scholarship informed by cil scholarship, and vice versa, will position educators to respond holistically to modern learning environments. with these propositions, the authors invite educators, librarians and information technologists to engage in a dialogue about these concepts and the potential for pedagogical change. throughout the paper, the authors will use the concept of a research assignment to illustrate the need for alternative approaches to the teaching of inquiry. the research assignment is typically defined as a paper that ‘evaluates sources of information, relates the accounts of information to one another, frames an argument that ties them together, and either reveals something important about the sources themselves or develops into a new contribution on the same topic’ (bizzell and herzberg 1987, p. 303). while some educators have experimented with multi-modal versions of the research assignment, printand text-based versions that are based around traditional academic information practices and norms remain the standard in north american undergraduate education. this means that it serves as an ideal way to explore learner needs, new information landscapes and the current teaching of inquiry in higher education. by redesigning an assignment around ple and cil concepts, the authors will illustrate how to model and engage students in a more holistic conception of inquiry within digital information landscapes. this paper will start by providing a brief overview of historical approaches to learning and inquiry before further defining both ples and cils. the paper will then explore how ple and cil approaches respond to modern learning environments through an examination of first, the information landscapes, and second, the approaches to learning within them. the paper will finish by exploring how ple and cil concepts could be used in the redesign of a research assignment. history and context the growing critique of learning management system (lms) and information literacy (il) standards demonstrates the shift from behaviourist to constructivist understandings of learning within the fields of educational technology and information studies. defined as a technology solution to respond to and ‘leverage’ the internet, the lms, which is also known as a course management system (cms) or virtual learning environment (vle), was designed to manage online learning (mott and a. hicks and c. sinkinson 2 (page number not for citation purpose) citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 wiley 2013, p. 4). comparably, the association of college and research libraries (acrl) information literacy standards for higher education (2000), which define il as the ability to identify an information need, and to locate, evaluate and use information, were developed in order to structure learner engagement in complex information landscapes. both of these solutions were early attempts to meet changing educational paradigms and information realities, but are now seen to fall short of enabling transformative learner experiences. in terms of the lms, educators commonly criticise the closed and hierarchical structure that privileges an institutionally managed and teacher-driven approach to learning (dalsgaard 2006; johnson et al. 2006; littlejohn 2011; mott 2010; mott and wiley 2013). in the case of the il standards, educators censure the way that the standards reduce information seeking and interaction to linear and simplistic steps without acknowledging the learner’s active role in diverse contexts within the knowledge economy (coonan 2011; elmborg 2006; jacobs 2008; kapitzke 2003; luke and kapitzke 1999; swanson 2005; tuominen, savolainen, and talja 2005). accordingly, both the lms and the il standards are increasingly seen as being in conflict with the learner-centred, active and reflective approaches to learning that are necessary for complex digital information landscapes. these criticisms can be seen more concretely through an examination of how inquiry is taught, and more specifically the research paper assignment. teaching research and inquiry has always been especially important within higher education, but traditional methods, which tend to centre around text-based realities and traditional modes of scholarly discourse, no longer match information practices. more specifically, the rigid and procedural nature of the traditional research assignment is seen to distance students from inquiry. furthermore, its inability to engage students authentically in conversation with other thinkers and writers means students fail to understand their own voice in inquiry, or see inquiry as a conversation they can enter, whether they are in civic, social or scholarly contexts (detmering and johnson, 2012; fister, 1993; leckie, 1996; lee 2013; nelson 1994). in this way, and like the lms and the il standards, the traditional research assignment is seen as increasingly anachronistic in these new information landscapes. definitions an examination of the historical context shows that ples and cils grew out of educational technologists’ and librarians’ increasing dissatisfaction with the ability of the lms and il standards to meet both learning and learner needs. this section will define each of these concepts. personal learning environment while there are numerous competing definitions and conceptions of ples, the authors align with attwell’s (2010) definition, which summarises the essential characteristics: ples are made-up of a collection of loosely coupled tools, including web 2.0 technologies, used for working, learning, reflection and collaboration with others. ples can be seen as the spaces in which people interact and communicate and whose ultimate result is learning and the development of collective know-how. a ple can use social research in learning technology citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 software for informal learning which is learner driven, problem-based and motivated by interest � not as a process triggered by a single learning provider, but as a continuing activity. ples are spaces for the modern learner to create, explore and communicate (dalsgaard 2006, p. 2). while they include and respond to new technology tools, ples are characterised as an approach to learning rather than a set of applications (eli 2009, p. 1). critical information literacy cil is a complex set of behaviours, attitudes and interactions that a learner adopts to engage critically in information landscapes. it is founded in critical pedagogy and critical information studies and reframes il as a culturally and socially situated phenomenon (luke and kapitzke 1999, p. 5). the cil approach to inquiry encourages an individual who is not passive, but one who poses questions and interrogates information systems; who explores, plays, internalises and contributes; one who reflects and revises information practices and beliefs (accardi, drabinski, and kumbier 2010, p. xii). information landscapes the paper will now turn to explore the changing context of modern learning environments, starting with an examination of information landscapes, which are characterised as open, broad and participatory. this section will explore how ple and cil approaches respond to these emerging contexts in a highly complementary manner. openness openness, defined as ‘both a technical feature and what might be called a ‘state of mind’’, is a central characteristic of information landscapes (weller 2011). open and free software, which allows anyone to create, edit and disseminate shared content, is an example of openness. however, the culture of openness found in modern information landscapes extends beyond these technical affordances to assume that by default, information should be freely accessible by all (stewart 2013). this culture of openness has had a significant impact on the educational sphere, as shown by the rise of open educational resources (oer) and open access (oa). while a complete transition to information openness has not yet been realised, educational practices that are entirely focused around traditional, closed and proprietary knowledge systems are in tension with these changing information landscapes. in contrast, ple and cil approaches match the values underpinning the culture of openness. the ples, for example, are structured to engage the learner with information from the wider community, rather than just information that is sanctioned by the teacher or institution (eli 2009). in this way, the learner engages with information that is ‘liberally licensed for re-use . . . free from restrictions to modify, combine and re-purpose’ (schaffert and hilzensauer 2008, p. 6). furthermore, the learner is invited to engage in content creation, thereby mirroring the values of open culture. likewise, the underlying values of cil approaches encourage the learner a. hicks and c. sinkinson 4 (page number not for citation purpose) citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 to question customs of knowledge production, including ‘‘what should count as knowledge,’ ‘for whom’ and ‘in whose interests’’ (luke and kapitzke 1999, p. 484). by questioning the traditional hierarchical systems ‘wherein expert authorities determine what counts as ‘knowledge’’, learners are positioned to engage in decentralised and open information exchange (pawley 2003, p. 426). source requirements in the traditional research assignment illustrate the tension between inherited educational practices and the open values of modern information landscapes. for example, assignment guidelines frequently direct students to use local subscription-based library resources in lieu of open web or non-library sources (holliday and rogers 2013, p. 264). this is problematic because it suggests that sources of knowledge found on a library shelf are ‘hard and fast truth’ (fister 1990, p. 506). it simultaneously devalues open content and new systems of dissemination and means that learners are not encouraged to question authority, significance and validity. the focus on subscription library resources that are not available after graduation further complicates the transferability of research skills. broad modern information landscapes can also be characterised by a rich tapestry of elements that include ‘tools, artefacts, processes and physical connections’ (couros 2010, p. 125). with the wider availability of free and open communication tools, the processes and networks involved in knowledge creation are made transparent, thereby adding more depth and breadth to information landscapes. in this way, information landscapes cannot solely be described as comprising formal or textual information products but must also include the connections and relationships therein. within academia, scholars’ use of new tools and technologies has made the scholarly research process more visible. scholars increasingly conduct dialogue through social media and blogs, for example, or exchange information through social bookmarking or tagging (veletsianos and kimmons 2012a). this content and conversation contribute to the broadening of information landscapes. unlike traditional methods for teaching inquiry, ple and cil approaches accommodate the dynamic nature of these new scholarly processes. ples, for example, help learners to build, manage and adjust networks as they navigate through information landscapes (chatti, jarke and specht 2010, p. 80). this includes understanding what littlejohn (2011) refers to as the ‘cumulative actions of others’ such as recommendations and tag clouds as well as the relationships that form between people, teams, organisations and communities. similarly, cil approaches do not privilege the textual; instead they acknowledge the role of social, procedural and physical information within information landscapes (lloyd 2007). as talja (1997) points out, knowledge consists of ‘a mix of scientific or expert knowledge, and unconscious, selective or culture-specific background assumptions’ (p. 73). it is legitimised by communities of practice, which may see these assumptions as valid in some contexts, while questionable in others. in this way, cil shifts the focus from information seeking and consumption to ‘a way of knowing the many environments that constitute an individual’s being in the world’ (lloyd 2007). traditional research assignments fail to capture these broad and lived experiences of inquiry within modern information landscapes. learners are rewarded for their final essay, with little attention given to the intermediary research processes (nelson 1990, p. 365). at the same time, success is commonly judged by the inclusion research in learning technology citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 of formal textual or codified knowledge in the works cited page, rather than authentic dialogue with information sources and scholars. participation the proliferation of freely available content creation and sharing tools means that modern information landscapes can also be distinguished by contributions from a broad base of participants. low barriers to authoring and sharing enable participants to engage in dialogue with other thinkers and writers, as well as to experiment with personal creations (jenkins 2009). if individuals engage as active participants instead of passive consumers they are able to circumvent traditional models of editorial control, thereby increasing the flow of information. within higher education, low barriers to participation broaden contributions to scholarly discourse by giving voice to a range of individuals, whether they are acclaimed senior faculty or novice undergraduates. this can be seen in the rise of alternative publication venues such as blogs or micro publishing sites as well as in the rise of new avenues for scholarly contributions such as curation and content sharing (veletsianos and kimmons 2012a). ple and cil approaches invite participation by encouraging learners to recognise the social and networked elements of inquiry and scholarship. within ples, tools enable a learner to build a network of people and content, and establish new modes of communication (chatti, jarke, and specht 2010). at the same time, by learning to be part of someone else’s network, for example, by editing, organising and sharing content, learners come to appreciate the social components of their community and interactions, as well as starting to understand their own agency in this setting (fisch 2007). similarly, principles of cil emphasise that inquiry is a process built on participation and the meeting of various voices in dialogue, rather than the passive consumption of sources (jacobs 2008, p. 259). in this way, a cil approach encourages the learner to recognise the role of context and social interactions as much as information sources in the development of new knowledge and meaning making. learning approaches the growing complexities of information landscapes demand new approaches to learning that are learner-centred and self-regulated. this section will explore how ple and cil approaches to learning complement each other as they match the needs of modern information landscapes. while this section focuses on the individual learner’s process, the authors recognise that the information landscape is networked and connected and that learning occurs neither in isolation nor in an entirely internal state (siemens 2005). learner-centred in an information landscape that is marked by broad participation, a learner’s contributions and expressions are tied to his/her prior knowledge as well as to his/her new experiences (swanson 2004, p. 261). in addition, changing modes and methods of expression mean that the learner has to take responsibility to decide which tools to use and what contributions and connections to make on an on-going basis. this means that static, imposed frameworks designed to manage inquiry and a. hicks and c. sinkinson 6 (page number not for citation purpose) citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 information are no longer viable solutions. instead, a model of learning that is learner-centred will need to be adaptable, flexible and more personally meaningful in the face of changing landscapes. within ples, a learner gains control as he/she makes individual choices about the technologies he/she uses in alignment with his/her personal needs, style of learning, preferences, context or changing interests (attwell 2007, p. 3; downes 2012, p. 27; johnson and liber 2008, p. 8; weller 2000, p. 183). by engaging with components of his/her ples, the learner participates and engages in ‘purposeful tinkerings that often form the basis of situated understanding’ (brown and adler 2008, p. 30). in this way, a learner constructs and manages an individually designed and self-reflective learning environment rather than one that is modelled around the teacher or institution. the underlying values of cil emphasise that meaningful knowledge construction occurs when a learner is encouraged to draw from prior information experiences and to reform them in light of new and emerging contexts (swanson 2004, p. 265). the learner is not generic nor is the learning. this means that educators cannot establish an exemplar that directs all lines of inquiry and by which all learners’ habits may be measured (elmborg 2006, p. 196). the establishment of norms such as these also run the risk of disregarding ‘variables of gender, class, religion, culture and ethnicity’ and privileging learners who match ‘those socially preselected for academic success’ (elmborg 2006, p. 194; kapitzke 2003, p. 58). the traditional research assignment is often driven by standards that favour imposed or institutional values over learner values and beliefs. the evaluation of information sources provides a good example. imposed or transmitted value rubrics that are checklists for establishing the validity and credibility of an information source distance students from their interaction with information even though ‘the only judge of ‘aboutness’ is the person who seeks to be informed ‘about’ something’ (elmborg 2012, p. 84). this may also contribute to a student’s tendency to remain passive, with little ownership in the inquiry process (coonan 2011, p. 18). self-regulated learner within new information landscapes, learners are continually challenged to manage their own learning. the speed of change, as well as the growth of contextual information needs, means that learners are required to regulate the direction and scope of their learning throughout their lives (buchem, attwell, and torres 2011, p. 11). this includes reflecting on strategies and practices that are effective as well as being able to identify gaps and needs for learning. ples enable a learner to pool all of his/her learning experiences, whether these are workplace, academic or civic experiences. this brings ‘together sources and contexts for learning hitherto separate’ to form a comprehensive record that he/she may use to find patterns, connections or contradictions in his/her learning experiences (attwell 2007, p. 7). in turn, this can be employed to make more informed and selfdirected decisions about learning needs. similarly, a cil approach to learning encourages students to adopt a selfreflective stance that examines one’s creation, dissemination and use of information (swanson 2004, p. 264; ward 2006, p. 396). in doing so, learners are encouraged to take the time to explore and adjust information strategies and practices as they learn more about their world. this reflection means that learners develop the capacity to research in learning technology citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 identify the questions that are meaningful to them as well as being able to understand their orientation and trajectory within information landscapes (elmborg 2006, p. 193). reflection is seldom a component of traditional research assignments. the completion of the assignment marks the end of the learner’s engagement with the topic of inquiry as well as with their own process of investigation. the learner is not afforded the space to reflect, which inhibits the adaptation and revision of strategies for future acts of inquiry (ward 2006, p. 398). research assignment the previous sections have explored how approaches to learning that are based on concepts of ples and cils may be more suited to meet learner needs in modern information landscapes. this section looks at how the research assignment can be used and adapted to provide a more meaningful approach to the teaching of inquiry. research assignments feature prominently in the undergraduate curriculum. educators may use the research assignment to firstly, assess students’ skills of inquiry and writing and secondly, to acculturate students into academic discourse and disciplinary content. in this way, these assignments are often miniature replications of the formal scholarly activity that academics enact throughout their professional careers (elmborg 2006, p. 196). despite being designed to model expert habits for the research novice, students often exhibit only superficial interaction with the process and the content of study (fister 1993; leckie 1996; lee 2013; nelson 1994). they may consequently experience research as a decontextualised, procedural task rather than engaging their own questions, knowledge or interests. components of the research assignment may also exist in tension with modern information landscapes, as discussed above. this includes the privileging of specific information sources, an emphasis on research product rather than research process and the imposition of an inauthentic audience. traditional methods of teaching the research assignment may also conflict with the approaches to learning that are necessary in modern information landscapes. this includes the reliance on artificial evaluation mechanisms, the isolation of inquiry within the academic context, and the failure to recognise the importance of reflection. educators are challenged to adjust the research assignment in order to avoid these potential pitfalls in the teaching of inquiry. redesigning the research assignment ple and cil approaches to learning could be used to redesign the research assignment and frame inquiry as a process that is social and connected as well as tolerant of new and emerging modes of communication. due to the participatory and open characteristics of both approaches, students are positioned to enter into conversation with a specific community or context while also remaining poised to scrutinise the specific information landscapes. the nature of modern information landscapes means that the teaching and learning of inquiry should acknowledge the social and connected nature of knowledge construction. ple and cil approaches meet this need by revealing the social connections that lead to new knowledge and understanding. within the academic context, this is exemplified through the academic’s reliance on personal contacts and citation trails during their inquiry process (leckie 1996, p. 202). the social and a. hicks and c. sinkinson 8 (page number not for citation purpose) citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 communicative aspects of this experience are less obvious to learners who are new to the process. by breaking down and unveiling the human interaction and dialogue in the process, students may more readily draw connections between these interactions and formal publications. in a redesigned assignment, this could involve asking students to identify aspects of informal scholarship, such as experts, key journals or conferences in the field in order to help them see patterns within the various streams of information and understand the collaborative nature of research (sinkinson and hicks 2013). this integrates students more widely into the discipline and the research process because the student learns to participate with the authentic audience beyond the classroom. the breadth of information sources within modern information landscapes further demand a shift in the approach to teaching and learning. approaches to inquiry based on ples and cils are expansive enough to incorporate multiple modalities and forms of information, as well as diverse and chaotic information interactions. for example, a ple approach supports the ‘entire set of resources that a learner uses to answer questions, provide context, and illustrate processes’ (eli 2009, p. 1). a cil approach helps to underscore this process-oriented understanding of inquiry. a result of these approaches would be an assignment in which students are free to pull from a variety of modalities and formats, including dominant and emerging sources of information. rather than promoting one modality over another, students will use the information that is the most appropriate and useful for their specific questions. in this way, the students may develop a critical stance and learn to engage in contextual evaluation of sources, rather than merely subscribing to prescribed values. learners will pursue inquiry within digital information landscapes throughout their lives (jacobs 2008). despite this fact, current research assignments do not stress the real-world application of inquiry. approaches to learning based on ples and cils allow students to contextualise inquiry within a specific knowledge community. in ples, for example, a learner can start using the tools and the resources that are used in the practice of a field. in this way, ples can be seen as ‘a tool intended to immerse yourself into the workings of a community’ (downes 2010). cil approaches extend this idea by encouraging students to recognise and consequently shape the wider conventions and characteristics of the discipline or community, rather than remaining an onlooker. within an assignment, learners might be asked to engage a specialist, either by commenting on a blog or through an interview. by viewing inquiry as entering a conversation, students learn not only how to ‘research and write like the specialists who inhabit these communities’ but also to ‘participate in a world of already articulated ideas’ (elmborg 2003, p. 73). these examples demonstrate the highly complementary nature of ple and cil approaches to learning. at the same time, cils bring a unique critical lens to the teaching of inquiry that is not currently emphasised in ple literature. the need for critical scrutiny is linked to broader questions about the role of technology in society as well as, more specifically, the fact that modes of communication are constantly changing and unpredictable (hall 2011). for that reason, it is important that learners adopt a critical stance to information and technology in order to expose underlying values, dynamics and relationships of power. as such, while individually both the ples and cils form useful approaches to inquiry, together they would offer a new approach to guide learner inquiry. more explicitly, the ples provide a practical structure for managing inquiry within modern information landscapes. a cil approach, on the contrary, encourages the development of learners’ critical dispositions towards research in learning technology citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 and within information landscapes. accordingly research assignments founded on ples and cils encourage learners to examine entire systems of information from production through to distribution and dissemination. conclusion in conclusion, this paper explores two approaches to learning that are responsive to changing information landscapes; ples and cils. as mcloughlin and lee (2008) note, the challenge for the new information environment is ‘to facilitate learning, be less prescriptive, and be open to new media, tools, and strategies, while nurturing the skills of information evaluation as well as the blending and remixing, and recombination of ideas to reach creative solutions’ (p. 20). this paper demonstrates that the interlocking of ple and cil approaches will offer strategies to help educators be more responsive in these ways. while this paper focuses on a theoretical exploration, it is clear that a practical implementation of these ideas warrants more dialogue. in order to enact the ideas outlined here, a teacher will need to work as a facilitator, motivator, co-learner or coach rather than an authoritative figure at the front of the classroom. students will need to work with self-direction and autonomy, which requires that they come to value informal, adaptive and flexible learning. these changing roles mark a significant transition from traditional learning and present questions such as: � what capacities and practices will learners and teachers need to develop? � how will the learner and teacher role be defined within these approaches? � what barriers will inhibit the exploration of these approaches? � what components of the traditional approach should be protected and maintained? working collaboratively, librarians and educational technologists can begin to address these questions as they design more meaningful and creative learning opportunities. references accardi, m., drabinski, e. & kumbier, a. (2010) critical library instruction: theories and methods, library juice press, duluth, mn. association of college and research libraries (acrl). (2000) ‘information literacy competency standards for higher education’, [online] available at: http://www.ala.org/ acrl/standards/informationliteracycompetency attwell, g. (2007) ‘personal learning environments � the future of elearning?’, lifelong learning, vol. 2, pp. 1�8. attwell, g. (2010) ‘personal learning environments and vygotsky’, [online] available at: http:// www.pontydysgu.org/2010/04/personal-learning-environments-and-vygotsky bizzell, p. & herzberg, b. (1987) ‘research as a social act’, the clearing house, vol. 60, no. 7, pp. 303�306. brown, j. s. & adler, r. (2008) ‘minds on fire: open education, the long tail, and learning 2.0’, educause review, vol. 43, no. 1, pp. 16�32. buchem, i., attwell, g. & torres, r. (2011) ‘understanding personal learning environments: literature review and synthesis through the activity theory lens’, learning, vol. 54, no. 2, pp. 1�33. a. hicks and c. sinkinson 10 (page number not for citation purpose) citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 http://www.ala.org/acrl/standards/informationliteracycompetency http://www.ala.org/acrl/standards/informationliteracycompetency http://www.pontydysgu.org/2010/04/personal-learning-environments-and-vygotsky http://www.pontydysgu.org/2010/04/personal-learning-environments-and-vygotsky http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 chatti, m. a., jarke, m. & specht, m. (2010) ‘the 3p learning model’, journal of educational technology & society, vol. 13, no. 4, pp. 74�85. coonan, e. (2011) ‘a new curriculum for information literacy: theoretical background’, [online] available at: http://arcadiaproject.lib.cam.ac.uk/docs/theory.pdf couros, a. (2010) ‘developing personal learning networks for open and social learning’, in emerging technologies in distance education, ed. g. veletsianos, athabasca university press, athabasca, pp. 109�128. dalsgaard, c. (2006) ‘social software: e-learning beyond learning management systems’, european journal of open, distance, and e-learning, [online] available at: http://www. eurodl.org/materials/contrib/2006/christian_dalsgaard.htm detmering, r. & johnson, a. m. (2012) ‘research papers have always seemed very daunting: information literacy narratives and the student research experience’, portal: libraries and the academy, vol. 12, no. 1, pp. 5�22. downes, s. (2010) ‘pedagogical foundations for personal learning’, [online] available at: http:// www.slideshare.net/downes/pedagogical-foundations-for-personal-learning downes, s. (2012) connectivism and connective knowledge, [online] available at: http://www. downes.ca/files/books/connective_knowledge-19may2012.pdf educause learning initiative (eli). (2009) ‘seven things you should know about personal learning environments’, [online] available at: http://net.educause.edu/ir/library/pdf/eli7049. pdf elmborg, j. (2003) ‘information literacy and writing across the curriculum: sharing the vision’, reference services review, vol. 31, no. 1, pp. 68�80. elmborg, j. (2006) ‘critical information literacy: implications for instructional practice’, the journal of academic librarianship, vol. 32, no. 2, pp. 192�199. elmborg, j. (2012) ‘critical information literacy: definitions and challenges’, in transforming information literacy programs: intersecting frontiers of self, library culture, and campus community, eds. c.w. wilkinson & c. bruch, acrl, chicago, il, pp. 75�95. fisch, k. (2007) ‘creating personal learning networks’, [online] available at: http:// thefischbowl.blogspot.com/2007/08/creating-personal-learning-networks.html fister, b. (1990) ‘teaching research as a social act: collaborative learning and the library’, reference quarterly, vol. 29, pp. 505�509. fister, b. (1993) ‘teaching the rhetorical dimensions of research’, research strategies, vol. 11, pp. 211�219. hall, r. (2011) ‘revealing the transformatory moment of learning technology: the place of critical social theory’, research in learning technology, vol. 19, no. 3, pp. 273�284. holliday, w. & rogers, j. (2013) ‘talking about information literacy: the mediating role of discourse in a college writing classroom’, portal: libraries and the academy, vol. 13, no. 3, pp. 257�271. jacobs, h. l. (2008) ‘information literacy and reflective pedagogical praxis’, the journal of academic librarianship, vol. 34, no. 3, pp. 256�262. jenkins, h. (2009) confronting the challenges of participatory culture: media education for the 21st century, mit press, cambridge, ma. johnson, m., et al., (2006) ‘towards a reference model for the personal learning environment’, who’s learning? whose technology? proceedings of the 23rd annual conference of the australasian society for computers in learning in tertiary education, sydney university press, sydney, [online] available at: http://www.ascilite.org.au/conferences/sydney06/proceeding/ pdf_papers/p141.pdf johnson, m. & liber, o. (2008) ‘the personal learning environment and the human condition: from theory to teaching practice’, interactive learning environments, vol. 16, no. 1, pp. 3�15. kapitzke, c. (2003) ‘information literacy: a review and poststructural critique’, australian journal of language and literacy, vol. 26, no. 1, pp. 53�66. leckie, g. (1996) ‘desperately seeking citations: uncovering faculty assumptions about the undergraduate research process’, journal of academic librarianship, vol. 22, no. 3, pp. 201�208. lee, k. k. (2013) ‘the research paper project in the undergraduate writing course’, in the new digital scholar: exploring and enriching the research and writing practices of nextgen students, eds. r. mcclure & j. purdy, asist, medford, nj, pp. 41�63. research in learning technology citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 11 (page number not for citation purpose) http://arcadiaproject.lib.cam.ac.uk/docs/theory.pdf http://www.eurodl.org/materials/contrib/2006/christian_dalsgaard.htm http://www.eurodl.org/materials/contrib/2006/christian_dalsgaard.htm http://www.slideshare.net/downes/pedagogical-foundations-for-personal-learning http://www.slideshare.net/downes/pedagogical-foundations-for-personal-learning http://www.downes.ca/files/books/connective_knowledge-19may2012.pdf http://www.downes.ca/files/books/connective_knowledge-19may2012.pdf http://net.educause.edu/ir/library/pdf/eli7049.pdf http://net.educause.edu/ir/library/pdf/eli7049.pdf http://thefischbowl.blogspot.com/2007/08/creating-personal-learning-networks.html http://thefischbowl.blogspot.com/2007/08/creating-personal-learning-networks.html http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p141.pdf http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p141.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 littlejohn, a. (2011) ‘charting’ [online] available at: http://littlebylittlejohn.com/charting/ lloyd, a. (2007) ‘recasting information literacy as socio-cultural practice: implications for library and information science researchers’, information research, vol. 12, no. 4, [online] available at: http://www.informationr.net/ir/12-4/colis/colis34.html lloyd, a. (2010) information literacy landscapes: information literacy in education, workplace and everyday contexts, chandos publishing, oxford, uk. luke, a. & kapitzke, c. (1999) ‘literacies and libraries: archives and cybraries’, pedagogy, culture and society, vol. 7, no. 3, pp. 467�491. mott, j. (2010) ‘envisioning the post-lms era: the open learning network’, educause quarterly, vol. 33, [online] available at: http://www.educause.edu/ero/article/envisioningpost-lms-era-open-learning-network mott, j. & wiley, d. (2013) ‘open for learning: the cms and the open learning network’, in education, vol. 15, no. 2, [online] available at: http://ineducation.couros.ca/index.php/ ineducation/article/view/53 mcloughlin, c. & lee, m. (2008) ‘the three p’s of pedagogy for the networked society: personalization, participation, and productivity’, international journal of teaching and learning in higher education, vol. 20, no. 1, pp. 10�27. nelson, j. (1990) ‘‘this was an easy assignment’: examining how students interpret academic writing tasks’, research in the teaching of english, vol. 24, no. 4, pp. 362�396. nelson, j. (1994) ‘the research paper: a ‘rhetoric of doing’ or a ‘rhetoric of the finished word?’’, composition studies/freshman english news, vol. 22, no. 2, pp. 65�75. pawley, c. (2003) ‘information literacy: a contradictory coupling’, library quarterly, vol. 73, no. 4, pp. 422�452. schaffert, r. & hilzensauer, w. (2008) ‘on the way towards personal learning environments: seven crucial aspects’, elearning papers, [online] available at: http://www.elearningeuropa. info/files/media/media15971.pdf siemens, g. (2005) ‘connectivism: a learning theory for the digital age’, international journal of instructional technology and distance learning, vol. 2, no. 1, pp. 3�10. sinkinson, c. & hicks, a. (2013) ‘unraveling the research process: social bookmarking and collaborative learning’, in the plugged-in professor: tips and techniques for teaching with social media, eds. s. ferris & h. wilder, chandos publishing, oxford, uk, pp. 49�60. stewart, b. (2013) ‘massiveness�openness �new literacies of participation?’, journal of online learning and teaching, vol. 9, no. 2, pp. 228�238. swanson, t. (2004) ‘a radical step: implementing a critical information literacy model’, portal: libraries and the academy, vol. 4, no. 2, pp. 259�273. swanson, t. (2005) ‘applying a critical pedagogical perspective to information literacy standards’, community and junior college libraries, vol. 12, no. 4, pp. 65�77. talja, s. (1997) ‘constituting ‘information’ and ‘user’ as research objects: a theory of knowledge formations as alternative to the information man theory’, information seeking in context: proceedings of an international conference on research in information needs, seeking and use in different contexts, taylor graham, london, pp. 67�80. tuominen, k., savolainen, r. & talja, s. (2005) ‘information literacy as a sociotechnical practice’, the library quarterly, vol. 75, no. 3, pp. 329�345. veletsianos, g. & kimmons, r. (2012a) ‘assumptions and challenges of open scholarship’, international review of research in open & distance learning, vol. 13, no. 4, [online] available at: http://www.irrodl.org/index.php/irrodl/article/view/1313 veletsianos, g. & kimmons, r. (2012b) ‘networked participatory scholarship: emergent techno-cultural pressures toward open and digital scholarship in online networks’, computers & education, vol. 58, no. 2, pp. 766�774. ward, d. (2006) ‘revisioning information literacy for lifelong meaning’, journal of academic librarianship, vol. 32, no. 4, pp. 396�402. weller, m. (2000) ‘using learning environments as a metaphor for educational change’, on the horizon, vol. 17, no. 3, pp. 181�189. weller, m. (2011) the digital scholar. the digital scholar: how technology is transforming scholarly practice, london, bloomsbury. a. hicks and c. sinkinson 12 (page number not for citation purpose) citation: research in learning technology 2015, 23: 21193 http://dx.doi.org/10.3402/rlt.v23.21193 http://littlebylittlejohn.com/charting/ http://www.informationr.net/ir/12-4/colis/colis34.html http://www.educause.edu/ero/article/envisioning-post-lms-era-open-learning-network http://www.educause.edu/ero/article/envisioning-post-lms-era-open-learning-network http://ineducation.couros.ca/index.php/ineducation/article/view/53 http://ineducation.couros.ca/index.php/ineducation/article/view/53 http://www.elearningeuropa.info/files/media/media15971.pdf http://www.elearningeuropa.info/files/media/media15971.pdf http://www.irrodl.org/index.php/irrodl/article/view/1313 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21193 http://dx.doi.org/10.3402/rlt.v23.21193 research in learning technology vol. 19, no. 1, march 2011, 73–85 issn 0968-7769 print/issn 1741-1629 online © 2011 association for learning technology doi: 10.1080/09687769.2010.548504 http://www.informaworld.com information technology in university-level mathematics teaching and learning: a mathematician’s point of view alexandre borovik* school of mathematics, university of manchester, manchester, uk taylor and franciscalt_a_548504.sgm (received 26 july 2010; final version received 18 november 2010) 10.1080/09687769.2010.548504research in learning technology0968-7769 (print)/1741-1629 (online)original article2011taylor & francis191000000march 2011professor alexandreborovikborovik@manchester.ac.uk although mathematicians frequently use specialist software in direct teaching of mathematics, as a means of delivery e-learning technologies have so far been less widely used. we (mathematicians) insist that teaching methods should be subjectspecific and content-driven, not delivery-driven. we oppose generic approaches to teaching, including excessively generalist, content-free, one-size-fits-all promotion of information and communications technology. this stance is fully expressed, for example, in the recent teaching position statement from the london mathematical society (2010) and is supported by a recent report from the national union of students (2010, 5): “not every area of study needed or was compatible with e-learning, and so to assume it would grant blanket advantages was not accurate”. this paper is an attempt to explain mathematicians’ selectivity in use of information and communications technology and its guiding principles. the paper is addressed to our non-mathematician colleagues and is not intended to be a survey of the existing software and courseware for mathematics teaching – the corpus of existing solutions is enormous and its discussion inevitably involves hardcore mathematics. keywords: mathematics; university; information technology selectivity: why? i argue that mathematics deserves special treatment not only because of its highly specific cognitive nature, but also because the mathematics community has accumulated much more experience of using computers and information technology (it) in teaching, learning, research and communication than many of our colleagues outside stem (science, technology, engineering and mathematics) disciplines have attained in their considerably shorter exposure to it. we are not special: computer science, physics, and many engineering disciplines are in a similar position and should be trusted to use their own tried and tested approaches to it. historically, mathematicians (and computer scientists) were the first to use it in teaching. even in the era of mainframe computers, green displays and dot-matrix printers, some serious work was done in that area (e.g. mass generation of random problems of controlled level of difficulty in linear algebra and differential equations). university mathematicians (and not only researchers, but all mathematicians who teach all kinds of mathematics in universities) form a professional community; it is global and transcends national boundaries. yes, mathematicians were some of the first *email: borovik@manchester.ac.uk 74 a. borovik to use email, too – at the time when the internet existed as a set of ftp sites and was unknown outside mathematics, physics and computer science departments. in short, the mathematics community has experience and knowledge of what can and cannot be done with computers. in that respect, we differ from, say, our colleagues in the humanities who are still on a path of discovery. i sincerely hope that it solutions that do not work in mathematics teaching can be happily used, say, in the humanities or in the social sciences – but please do not force on us solutions unsuitable for, and unworkable in, mathematics learning and teaching. a case study: tex in the late 1970s, the great mathematician and computer scientist donald knuth launched a revolution in scientific communication by creating tex, a cross-platform computer language for typesetting mathematical texts (knuth 1979, 1986a). in one step, he brought mundane mathematical scribbles – not only research papers, but also lecture notes, exercise sheets, seminar handouts – to the highest reaches of typographic art. since the early 1990s, tex and its dialect, latex (lamport 1985), have been international standards for mathematical typesetting. however, their routine everyday use in teaching in every mathematics department remains unnoticed and unappreciated by the wider education community. this is unfortunate because tex/ latex is a pedagogical success story: it allows the presentation of even the most complicated mathematical formulae as structured and logically justified shapes, optimised for visual processing by the human eye and brain. after all: typography may be defined as the craft of rightly disposing printing material in accordance with specific purpose; of so controlling the type as to aid to the maximum the reader’s comprehension of the text. (morison 1951; quoted from lawrence 2003, 165) tex succeeded in part because donald knuth spent years studying the tradition of calligraphy (worth remembering – it is millennia long) and the art of typesetting (five centuries old) (knuth 1999). in the process, knuth produced such masterpieces as 3:16 bible texts illuminated (knuth 1990) with 60 original illustrations by many of the world’s leading calligraphers. speaking about software systems for teaching and learning that are currently commercially available in the higher education market, we can safely conclude that in 95% of it products the presentation of mathematics lags 20 years behind tex; their developers have not done their homework with the same care as donald knuth did his. too frequently, it developers and promoters of e-learning invite mathematicians back to the stone age. figure 1. a sample of latex output optimised for viewing on an iphone screen. for those who have never seen how latex typesets mathematical formulae, figure 1 contains a few random examples set in font type, size, and column width suitable for viewing on narrow screens of smart phones (some of my students indeed use their smart phones for access to bite-sized learning materials like exercise sheets – although smart phones are not very convenient for reading more substantial pieces of text like lecture notes). figure 2 shows how latex handles difficult spatial arrangements of type. figure 2. latex: handling of nested roots and continued fractions. for dyslexic students, one can easily meet disability consultants’ recommendations by setting up lecture notes in landscape mode, double-line spacing and a huge sans serif font (figure 3). figure 3. an example of typesetting of formulae adjusted for needs of dyslexic readers. research in learning technology 75 students matter i have taught computer-based courses since 1995 – and i had a chance to observe my students in pedagogical and psychological mise en scènes of information and communications technology (ict). studies of students’ attitudes to it already exist, and the association for learning technology (2010) provides a useful survey. a recent report from the national union of students (2010) expresses a summarised students’ opinion in a very direct and unambiguous way. i quote two points that match my personal observations. one point is that students want to have choice and want to be in control: figure 1. a sample of latex output optimised for viewing on a smart phone screen. 76 a. borovik students prefer a choice in how they learn – ict is seen as one of many possibilities, alongside part-time and traditional full-time learning, and face-to-face teaching. (national union of students 2010, 3) students could see some advantages to an e-learning approach. if it were presented as an option, as opposed to an obligation, it would avoid onerous undertones. (national union of students 2010, 5) figure 2. latex: handling of nested roots and continued fractions. figure 3. an example of typesetting of formulae adjusted for needs of dyslexic readers. research in learning technology 77 students who preferred or were most familiar with traditional lecturing methods also might feel cheated if the course was provided mainly online. (national union of students 2010, 26) another point is that content matters for students more than delivery: participants expressed concerns over ‘surface learning’ whereby a student only learns the bare minimum to meet module requirements – this behaviour was thought to be encouraged by ict: students can easily skim-read material online, focusing on key terms rather than a broader base of understanding. (national union of students 2010, 5) students matter, and their opinion should matter, too. what we want: windows in mathematical world let me formulate in one word the shared key feature of it that finds modern uses in direct teaching of mathematics: this word is virtualisation. a computer is useful if it creates a new (virtual) reality that cannot be created by other means. in mathematics, the word ‘reality’ includes the ideal platonic world of mathematical objects and structures. matlab,1 maple, mathematica – commercial (and expensive) mathematics software packages widely used in undergraduate teaching – are windows into this platonic world. as a rule, software that provides such windows needs a powerful mathematical engine. matlab, maple, mathematica and statistics packages such as spss (commercial) and r (open source and free) are not just toys for learning – they are professional research tools; mastering them is a valuable transferable skill for graduates seeking employment in mathematics-intensive industries. i can give less known and more specialised examples, like tarski’s world – an expertly crafted courseware package for learning mathematical logic (barwise and etchemendy 1993); an improved version is now available under the same title (barker-plummer, barwise, and etchemendy 2008) and the wonderful visualisation and experimentation tools for elementary geometry, cinderella2 and geogebra.3 assessment of mathematics learning software inevitably involves a mathematical characterisation of its built-in mathematical world. for example, it matters that the interface language of tarski’s world is ‘interpreted’ (in some specific meaning of this word as used in logic) – and serious implications of this fact for teaching logic with tarski’s world had been pointed out in one of the first reviews of the package (hodges 1989) – written in 1989! if the reader wishes to get hands-on experience, i invite him/her to have a look at cinderella and geogebra – they are available for free, and their uncluttered minimalistic interfaces provide for an immediate usability; because of space limitations, i can only mention in passing commercial dynamic geometry packages (cabri, geometer’s sketchpad). the next two paragraphs use cinderella and geogebra to give a sample of a mathematician’s approach to probing and testing the software; they can be skipped in the first reading. it is interesting to compare the behaviour, in cinderella and geogebra, of a simple interactive diagram: two intersecting circles of varying radii and the straight line determined by their points of intersection (see figure 4a for a cinderella snapshot). in geogebra, when you vary the radii or move the centres of the circles and make the circles non-intersecting, the line through the points of intersection disappears – exactly as one should expect (figure 5). in cinderella, the line does not disappear, it moves 78 a. borovik figure 4. cinderella: (a) a line through the intersection points of two circles; (b) the line through the intersection points of two circles does not disappear after the circles are disengaged. research in learning technology 79 following the movements of the circles, always separating them (figure 4b); when circles touch each other and start to intersect again, the line turns to be, again, the common tangent line of two circles or, in the case of two intersecting circles, the line through the points of intersection. (the line is called the radical axis of the two circles.) to a mathematician, the behaviour of this diagram suggests that the underlying mathematical structure of geogebra is the real euclidean plane. in cinderella, the underlying structure is the complex projective plane; what we see on the screen is just a tiny fragment of it, a real affine part. the radical axis of two non-intersecting circles is the real part of the complex line through two complex points of intersection. the intersection points of two real circles are complex conjugate, the line is invariant under complex conjugation and therefore is real and shows up on the real euclidean plane. for a mathematician, this is a strong hint that cinderella could work better than geogebra in accommodating non-euclidean geometries: elliptic and hyperbolic (the lobachevsky plane) since they happily live in the complex projective plane. but perhaps there may be pedagogic situations where the real euclidean plane might be a safer tool for some students than the complex projective plane, even if the difference is undetectable at the first glance. figure 4. cinderella: (a) a line through the intersection points of two circles; (b) the line through the intersection points of two circles does not disappear after the circles are disengaged.figure 5. geogebra: (a) a line through the intersection points of two circles; (b) the line through the intersection points of two circles disappeared after the circles are disengaged. the unity of research and teaching one interesting feature of matlab, maple, mathematica and spss is that they were originally designed and developed for research purposes and only later fed into university teaching – mostly by mathematicians who transferred to their teaching the figure 5. geogebra: (a) a line through the intersection points of two circles; (b) the line through the intersection points of two circles disappeared after the circles are disengaged. 80 a. borovik skills developed in their research. it was the mathematics research community who acted as a driver of technological change in mathematics teaching. this example, even taken on its own, demonstrates the futility of erecting a fence between mathematics research and mathematics teaching. the situation with specialised teaching-only software packages is even more instructive. returning to one of my case studies, tarski’s world, i wish to comment that one of its authors – and the initiator of the project – was kenneth jon barwise, a prominent mathematician, philosopher and logician. development of tarski’s world and other programs that became part of the courseware package language, proof and logic (barwise and etchemendy 2003): fitch, boole, grade grinder, required not only a pioneering re-assessment of methodology of teaching mathematical logic (barwise and etchemendy 1998), but also the creation of a new direction in mathematical logic itself, heterogeneous reasoning, which formed the core of the computer algorithms implemented as courseware (shin 2004). the first reviewers of tarski’s world (boolos 1990; goldson and reeds 1993) were fully aware of mathematical difficulties that its authors had to overcome. the work of jon barwise and his collaborators is a manifestation of a phenomenon specific to mathematics: the central role of didactic transformation; that is, mathematical reworking of teaching material into a form suitable for students’ consumption. the term transformation didactique was coined in 1852 by french philosopher auguste comte (1852) and is well known in french education studies (chevallard 1985), but remains unused in english-language literature on education. hyman bass, a prominent mathematician and a champion of mathematics education, picked up from his french colleague jean-pierre kahane (bass 2005) the term ‘didactic transformation’ together with an explanation of its role in relations between mathematics and mathematics education: in no other living science is the part of presentation, of the transformation of disciplinary knowledge to knowledge as it is to be taught (transformation didactique) so important at a research level. in no other discipline, however, is the distance between the taught and the new so large. in no other science has teaching and learning such social importance. in no other science is there such an old tradition of scientists’ commitment to educational questions. (bass 2005) in barwise’s case, didactic transformation took the form of mathematical research in logic that was then fed into software development. even the roots of dynamic geometry packages aimed at the teaching of elementary euclidean geometry can be traced back to the hard core research. this is a testimony from judah schwartz, creator (in early 1980s) of one of the first interactive geometry packages, the geometric supposer: in the mid-1960s, while on the research staff of the lawrence radiation laboratory in california, two colleagues and i developed a series of computer-generated motion picture films that depicted graphically the collisions of subatomic particles as described by formal mathematical machinery of quantum mechanics. i became interested in the potential of the computer to make accessible representations of spatial and temporal whose natural distance and time scale lay well outside the ken of human sensory apparatus. my colleagues and i pursued these efforts for a while but soon concluded that the true utility of computers to help people with mathematical abstraction would lie in the interactivity and control that the soon-to-come microcomputer would offer. (schwartz, yerushalmy, and wilson 1993, 4) research in learning technology 81 paradoxical economics of education so mathematicians have developed, and systematically use, specialist software in direct teaching of mathematics – and find it very useful. however, as means of delivery of mathematics teaching, it and e-learning have so far been unable to meet our expectations. there are several reasons for this. one reason is that our expectations are high. due to the level of sophistication already achieved, say, in matlab/mathematica/maple or in tex/latex, mathematicians’ demands for functionality of it are high and are not met by many software packages and virtual learning environments (vles) currently promoted in british universities. at a neurophysiological level, teaching/learning mathematics is a communication between two brains. it is best done one-to-one, or in a small group. large class lectures are an unhappy compromise with economic necessity. from a pedagogical point of view, the right alternative to a large class lecture is not streaming-on-demand of video recordings; the true alternative is a small class lecture. unfortunately, this alternative in most cases is financially infeasible. collaborative online small groups provide some interesting possibilities, but students themselves insist that ict should be a supplement to, not a replacement of, face-to-face teaching: motion 306, passed at the april 2010 nus national conference states that: [—] 4. the provision of e-learning should be utilised as a tool for learning, in all institutions, but that should not merely be used as a method of reducing costs and should be in conjunction with, not instead of, other face-to-face teaching methods. 5. technology should complement good teaching, allowing students to benefit from the additional value of e-learning but should not be used as a substitute for face-to-face contact and good teaching. (national union of students, 2010, 18) there is a need to assess the efficiency of particular methods of teaching not only from the pedagogical, but also from a socio-economic point of view. of course, ‘generic’ technologies are very tempting to policy-makers because of their promise (mostly unrealistic) of economies of scale. it would be wrong to reduce the all-important discussion of learning and teaching to deciding the choice of the cheapest variety of margarine as a substitution for butter. last but not least, at the socio-economic level relations between mathematics and information technology are also paradoxical. mathematics, by its nature, is an open source phenomenon. a powerful formulation of this principle belongs to joachim neubüser, the initiator and leader of the gap project,4 perhaps the most successful community project in experimental mathematics. you can read sylow’s theorem and its proof in huppert’s book in the library without even buying the book and then you can use sylow’s theorem for the rest of your life free of charge, but – and for understandable reasons […] – for many computer algebra systems license fees have to be paid regularly for the total time of their use. in order to protect what you pay for, you do not get the source, but only an executable, i.e. a black box. you can press buttons and you get answers in the same way as you get the bright pictures from your television set but you cannot control how they were made in either case. with this situation two of the most basic rules of conduct in mathematics are violated: in mathematics information is passed on free of charge and every thing is laid open for 82 a. borovik checking. not applying these rules to computer algebra systems […] means moving in a most undesirable direction. most important: can we expect somebody to believe a result of a program that he is not allowed to see? (neubüser 1993) it is almost a rule that open source software systems are friendlier to mathematics; perhaps this could be explained by the social and cultural background of the open source movement; neubüser’s words “can we expect somebody to believe a result of a program that he is not allowed to see?” is a war cry and a cultural paradigm shared by mathematicians (who apply it, in the first instance, to proofs of their theorems) and by open source warriors. a good illustration of this principle can be found in a comparison between moodle, a free open source vle (it provides for a decent rendering of latex), and proprietary vles, some of which are completely unfit for use in mathematics courses. developers of quality proprietary software for mathematics and statistics (like matlab/mathematica/maple, spss) have taken reasonable care to allow the users a sufficient degree of freedom in tinkering with the interface (and, at least in the case of matlab – with the computational core, too – matlab smoothly incorporates bespoke fortran and c code). also, matlab/mathematica/maple allows export of results (both symbolic and graphic) in formats directly usable in tex/latex documents. r, a very popular statistics package, is a gnu licensed open source product. tex, the true and unsurpassed masterpiece of the art of computer programming, is faced with a strange fate: it somehow does not show up on the radar of promoters of it for higher education. i believe this has a very simple explanation: tex is free – thanks to the generosity of donald knuth – and open source (moreover, the source code, with detailed comments and explanation, is published as a book; knuth 1986b). it exists like the air that we breathe. for that reason tex remains unadvertised and not promoted, and therefore unnoticed by university administrators who make decisions about the acquisition of it products. paradoxically, these are the same administrators who hold the purse strings and are apparently on the quest for the cheapest it solutions. i conjecture that moodle is also disadvantaged by being free, not promoted by vested commercial interests, and therefore invisible in the market. we have to make free open source options visible – this will allow them to compete with commercial for-profit products. the problem is wider and concerns not only software and it, but also textbooks. one of the referees of this paper suggested that an open-source model supports ‘niche’ disciplines better than a commercial model. this assessment is echoed by gary hall (2008, 42): many publishers have decided to focus on introductions and readers for the relatively large (and so more profitable for their shareholders) first year undergraduate ‘core course’ markets, and hardly produce books for second-and-third-year students, let alone research monographs or even edited collections of original scholarship aimed at postgraduates and other researchers, at all. from the next semester, i will be teaching one of my lecture courses using a gnu licensed textbook (hefferon 2010). besides pedagogical reasons, my decision is motivated by new functionality provided by open source textbooks: availability of source latex files gives, for example, a possibility of global changes in the text, say a uniform change of notation over the entire textbook. in a properly written latex source file, it is achieved by changing one line in the preamble. the need for change in notation rarely arises in most disciplines, but is quite common in mathematics. research in learning technology 83 availability of the source file resolves of number of issues of disability support – one line change allows a lecturer to change size, type, colour of fonts used, switch to the landscape mode, and so forth. it is a social imperative of our challenging times: open source teaching suits publicly funded universities best. because it is not promoted by commercial interests, it needs its champions. the textbook that i am planning to use finds an unexpected champion in arnold schwarzenegger. the book is endorsed by the free digital textbook initiative run by the california learning resources network (http:// www.clrn.org/fdti/). the california learning resources network is funded by the state of california. this state is experiencing financial difficulties, and the webpage of the free digital textbook initiative proudly displays a message from governor of california arnold schwarzenegger: this initiative will ensure our schools know which digital textbooks stand up to california’s academic content standards – so these cost-effective resources can be used in our schools to help ensure each and every student has access to a world-class education. some of the existing repositories of open education resources (such as http:// www.oercommons.org/) and programmes for their development (such as the one run by hefce and jisc: http://www.jisc.ac.uk/oer) are still not sufficiently representative and not sufficiently selective. a systematic peer review is badly needed; perhaps it can be provided by greater involvement of subject specialists and their learned societies. shopping list mathematicians do not want to work in isolation from the rest of the it learning community; there are a number of issues (like support to users with disabilities) that need a coordinated effort. here is a brief list of our concrete wishes. it was suggested by my colleagues who read earlier versions of my notes. any help and advice from the it learning community would be warmly appreciated. in vles, we need support for, and interfacing with, matlab, mathematica, maple, spss, r: symbolic input and output in matlab, mathematica, maple; import of mathematics graphics produced by these packages; in spss and r – input and output of data files, import of tables. as already explained, vles are unusable in mathematics learning and teaching if they do not support tex/latex. one of the benefits cited for vles is the ability for students to engage in discussions. they cannot do this if we have barriers to getting mathematics into a machine. provision for visually impaired students remains a serious problem. screen readers do not work with mathematics. online computer-aided assessment is a big issue for those of us that want to formatively assess 350+ size classes without using very bland questions. in particular, assessment systems should allow easy and unconstrained entry of mathematical formulae and be able to interpret their meaning. some obstacles to that are discussed in sangwin (2006). last but not least – the role and status of free and/or open source software, courseware and textbooks deserve a thorough discussion. 84 a. borovik acknowledgements needless to say, all opinions expressed here are those of the author and no-one else. the author is grateful to the anonymous reviewers for their feedback. this article was first published in the association for learning technology online newsletter (issue 20, 11 august 2010, issn 1748-3603), reproduced here with minor amendments. copyright of this paper remains vested with the individual authors or their employer and is licensed for use with a creative commons ‘attribution non-commercial-no derivative works 2.0 uk: england & wales licence’. notes 1. a list of available learning resources based on matlab – and far from being complete – can be found online (http://www.mathworks.com/matlabcentral/linkexchange/?term=tag: “mathematics”). 2. an earlier version of cinderella can be downloaded for free (http://cinderella.de/tikiindex.php?page=download+cinderella+1.4&bl). 3. geogebra is free and open source (http://www.geogebra.org/cms/). 4. gap – groups, algorithms, programming – a system for computational discrete algebra (http://www.gap-system.org//). references association for learning technology. 2010. learner acceptance of on-line learning and e-learning. http://wiki.alt.ac.uk/index.php/learner_acceptance_of_on-line_learning_and_ e-learning. barker-plummer, d., j. barwise, and j. etchemendy. 2008. tarski’s world. chicago: university of chicago press. barwise, j., and j. etchemendy. 1993. the language of first-order logic: including the ibmcompatible windows version of tarski’s world 4.0. cambridge: cambridge university press. barwise, j., and j. etchemendy. 1998. computers, visualization, and the nature of reasoning. in the digital phoenix: how computers are changing philosophy, ed. t.w. bynum and j.h. moor, 93–116. london: blackwell. barwise, j., and j. etchemendy. 2003. language, proof and logic. stanford: csli publications. distributed by university of chicago press. bass, h. 2005. mathematics, mathematicians, and mathematics education. bulletin of the american mathenatics society 42, no. 4: 417–30. boolos, g. 1990. review of jon barwise and john etchemendy, turing’s world and tarski’s world. journal of symbolic logic 55: 370–1. chevallard, y. 1985. la transposition didactique – du savoir savant au savoir enseigné. grenoble: la pensée sauvage. comte, a. 1852. catéchisme positivist. goldson, d., and s. reeds. 1993. using programs to teach logic to computer scientists. notices of the american mathematics society 40, no. 2: 143–8. hall, g. 2008. digitize this book! the politics of new media, or why we need open access now. minneapolis: university of minnesota press. hefferon, j. 2010. linear algebra. ftp://joshua.smcvt.edu/pub/hefferon/book/book.pdf. hodges, w. 1989. review of j. barwise and j. etchemendy, tarski’s world and turing’s world. computerised logic teaching bulletin 2, no. 1: 36–50. knuth, d.e. 1979. tex and metafont: new directions in typesetting. stanford: digital press, and american mathematical society. knuth, d.e. 1986a. the texbook. reading, ma: addison-wesley. knuth, d.e. 1986b. tex: the program. reading, ma: addison-wesley. knuth, d.e. 1990. 3:16 bible texts illuminated. middleton, wi: a-r editions. knuth, d.e. 1999. digital typography. reissue ed. cambridge: cambridge university press. lamport, l. 1985. latex – a document preparation system – user’s guide and reference manual. reading, ma: addison-wesley. lawrence, r. 2003. maths = typography? tugboat 24, no. 2: 165–8. research in learning technology 85 london mathematical society. 2010. teaching position statement on mathematics degrees, their teaching and assessment. http://www.lms.ac.uk/policy/2010/teaching_position_ statement.pdf. morison, e. 1951. first principles of typography. cambridge: cambridge university press. national union of students. 2010. student perspectives on technology – demand, perceptions and training needs. report to hefce by nus. http://www.hefce.ac.uk/pubs/rdreports/ 2010/rd18_10/rd18_10.pdf. neubüser, j. 1993. an invitation to computational group theory. invited talk at the conference ‘groups st andrews’ at galway 1993. http://www.gap-system.org/doc/talks/cgt.dvi and postscript: http://www.gap-system.org/doc/talks/cgt.ps. sangwin, c. 2006. assessing elementary algebra with stack. http://www.open.ac.uk/cetlworkspace/cetlcontent/documents/4607d31d634fd.pdf. schwartz, j.l., m. yerushalmy, and b. wilson, eds. 1993. the geometric supposer: what is it a case of? hillsdale, nj: lawrence erlbaum associates. shin, s.-j. 2004. heterogeneous reasoning and its logic. the bulletin of symbolic logic 10, no. 1: 86–106. calt12203.fm alt-j, research in learning technology vol. 12, no. 2, june 2004 evaluating a virtual learning environment in the context of its community of practice rachel ellaway*, david dewhurst & hamish mcleod university of edinburgh, uk taylor and francis ltdcalt12203.sgm10.1080/0968776042000216192alt-t research in learing technology0968-7769 (print)/1741-1629 (online)original article2004taylor and francis ltd1220000002004rachelellawaylearning technology sectioncollege of medicine and veterinary medicineuniversity of edinburgh15 george squareedinburgheh8 9xdukrachel.ellaway@ed.ac.uk the evaluation of virtual learning environments (vles) and similar applications has, to date, largely consisted of checklists of system features, phenomenological studies or measures of specific forms of educational efficacy. although these approaches offer some value, they are unable to capture the complex and holistic nature of a group of individuals using a common system to support the wide range of activities that make up a course or programme of study over time. this paper employs wenger’s theories of ‘communities of practice’ to provide a formal structure for looking at how a vle supports a pre-existing course community. wenger proposes a learning architecture framework for a learning community of practice, which the authors have taken to provide an evaluation framework. this approach is complementary to both the holistic and complex natures of course environments, in that particular vle affordances are less important than the activities of the course community in respect of the system. thus, the vle’s efficacy in its context of use is the prime area of investigation rather than a reductionist analysis of its tools and components. an example of this approach in use is presented, evaluating the vle that supports the undergraduate medical course at the university of edinburgh. the paper provides a theoretical grounding, derives an evaluation instrument, analyses the efficacy and validity of the instrument in practice and draws conclusions as to how and where it may best be used. introduction virtual learning environments (vles), and systems like them, provide ‘the ‘online’ interactions of various kinds which can take place between learners and tutors, including online learning’ (jisc and ucisa, 2003). these systems all share a common thread; they can take on many roles and they can support a wide range of *corresponding author. learning technology section, college of medicine and veterinary medicine, university of edinburgh, 15 george square, edinburgh eh8 9xd, uk. email: rachel.ellaway@ed.ac.uk issn 0968–7769 (print)/issn 1741–1629 (online)/04/020125–21 © 2004 association for learning technology doi: 10.1080/0968776042000216192 126 r. ellaway et al. educational, administrative and logistical processes, each of which is able to interact and integrate with the others. because of this integration, because vles can be used in many different ways, and because much that was implicit in the traditional learning environment becomes explicit in its online equivalent, the evaluation of vles has proved to be a particularly complex problem. furthermore, because of the sheer scale, complexity and cost of vles, their adoption and use is increasingly undertaken at an institutional level and any subsequent evaluation, if it is not done at the level of the individual learner, is most often also undertaken at this institutional level. between the micro and macro approaches are levels that remain relatively disregarded, those of the programme of study or cognate discipline area, which can, in some cases, be usefully modelled as a distinct ‘community of practice’. this paper proposes a holistic approach to evaluating vles in the context of the community of practice in which they are used, where such a community already exists. this is based around the ‘learning architecture framework’ (laf) proposed by etienne wenger (1998). this approach is practitioner oriented and was developed in response to requirements for evaluating programme-wide vles in integrated subjects such as medicine and veterinary medicine. evaluating vles although there has been much published on evaluative work on vles, this has until recently rarely gone further than analysing their various features and functions (see chest mle/vle comparison grid;1 edutools comparison grid;2 or jenkins et al., 2001) or the phenomenology of vles in use (lee & thompson, 1999; barajas & owen, 2000; richardson & turner, 2000). occasionally more sophisticated approaches to evaluating vles have emerged (britain & liber, 1999; koper, 2000) which take a more grounded and pedagogically orientated approach, but which continue to orientate towards predictive and intrinsic properties of the vle. in presuming that a vle has intrinsic properties, that the context into which a vle will be deployed is neutral and that any given vle will automatically deliver predictable benefits (or otherwise) into that context, the predictive approach is significantly limited in providing a useful perspective of a vle in a grounded course context. it is important to note that most of these approaches have been directed towards a novitiate audience looking for the best evidence or advice available to help them select a suitable system to meet their needs. a different approach to prospective and predictive models is the evaluation of a ‘vle-in-use’, which investigates the unique properties and dynamics of a coursevle instance. this is a variation on ‘situated’ or ‘holistic’ evaluation (bruce et al., 1993). such approaches are now beginning to appear in the literature, for instance using grounded theory (alsop & tompsett, 2002). in introducing a typology of approaches to evaluating learning technologies oliver (1998) identifies ‘holistic evaluations’ as starting from the position that introducing technology to educational settings will tend to alter learning outcomes rather than just evaluating a virtual learning environment 127 the quantity or quality of what is learnt. the holistic approach seeks to encompass broader aspects of learning technology such as its social or organizational dimensions as well the immediate focus of interest. oliver describes this methodology as seeking: to identify the positive and negative aspects of technology use, as compared with traditional teaching methods, to build categories from these data, and then to statistically analyse the response patterns in order to arrive at generalizable findings. a situated and/or holistic ‘vle-in-use’ approach is therefore likely to be a more appropriate methodology to adopt to unlock the nature and value of our vles in their complex and multifactorial relationships with their course contexts. asking the right questions whether used for distance learning or deployed in support of on-campus courses, and whether a commercial ‘off-the-shelf’ or a ‘home-grown’ system is being used, vles share two essential characteristics; multiple systems integration and a course or module focus. it might be expected that any given vle might be employed in different ways within different course contexts. in the same way, a particular course could be expected to use different vles in different ways. thus, the pairing of a course and a vle should be understood as a unique instance, with a unique set of characteristics, rather than as two separate entities. the effectiveness and value of a vle to a course is therefore not an inherent property of the vle software but depends on its use in facilitating and mediating the needs and activities of a particular course. extending this argument further it might be concluded that all vle functions exist in a ‘blended’ relationship with human activities, independent of whether they are the primary delivery medium (e.g. distance courses) or one among many (e.g. oncampus courses). this is of course true of any technology, but it is of key importance when considering complex situations where the permutations of how different kinds of affordances can be taken up far exceed the prime designed affordances of any given technology. added to this is a need to consider the adaptive nature of human practices to the affordances of the available technologies. if a thing can be done, not only will the direct affordances it offers tend to change our practices, but, in doing so, it will most likely change what we want to do, and what value we attach to these different activities and functions (graham, 1999). the defining aspect is therefore how such complex technologies are used in specific circumstances. the question which should be asked about a vle is not ‘what can it do?’ but rather ‘what is it doing?’, thereby focusing on its function and role in the situated educational context. it is often the case, however, even for the best managed courses, that over time their ethos and procedures become blurred, sometimes to the point where there is no clear or common understanding of what the course is about and, more importantly, exactly how it works. furthermore the deployment of a vle into any course context initiates 128 r. ellaway et al. a complex set of exchanges, each component shaping the other in a cycle of mutual adaptation. the individual organization is neither merely a passive receiver of predetermined technological artefacts nor an autonomous controller of technological change. rather, in organizing the flows of knowledge and resources within and between groups, organizations shape the technology process at the same time as it shapes them. (scarborough & corbett, 1992, p. 10) there may therefore be no single reliable or meaningful way to independently measure a course’s dynamics or the relevant contextualized affordances of a vle. however, given a common set of defining criteria or factors, a vle may be evaluated within the specific course context in which it is used, avoiding the problems of independent analysis and benefiting from the situatedness of such an approach. because of the complex nature of this vle problem space it is unlikely that any one technique will be effective in evaluating a vle fully. however, a triangulation approach (breen et al., 1998), employing a range of techniques and perspectives is likely to provide a better solution. the rest of this paper describes the creation and use of an evaluation instrument based on wenger’s theories on ‘communities of practice’ that can provide a strong contributing dimension to this kind of analysis. communities of practice and vles originally focusing on ideas of apprenticeship, the development of theories of ‘communities of practice’ coalesced around the concepts of legitimate peripheral participation in a community of practice identified by lave and wenger (1991). legitimate peripheral participation in a community of practice is centred on the notion that: learners inevitably participate in communities of practitioners and that mastery of knowledge and skill requires newcomers to move toward full participation in the sociocultural practices of a community. (lave & wenger, 1991, p. 29) in ‘communities of practice’, wenger (1998) proceeded to argue that current orthodox approaches to learning that are based on concepts of individual learners learning in prescribed ways causally linked to teaching are redundant. wenger proposed instead a ‘social theory of learning’, which is based upon learning as individual engagement and participation in a community of practice. wenger’s theories can be particularly relevant in modelling learning environments where they encompass a pre-existing learning community of students, teachers/ tutors, support staff and potentially many other roles and groups. furthermore, any participant may adopt or change roles; students may be involved in teaching each other, teachers may become learners, support and administration responsibilities may fall to different participants at different times and so on. all of this activity is in turn informed by the socio-cultural norms and values inherent in the practice and the related social contexts in which it is situated. if this is the case then such a course may be modelled as a community of practice, and indeed, its component parts (such as evaluating a virtual learning environment 129 modules of study or groupings such as ‘students’) may themselves constitute subsidiary communities of practice. the relevance of this model depends on the degree and coherence of shared purpose, meaning, activity and identity across the course community. for instance in a modular arts or science context the model may be expected to be weak as students pursue individual patterns of cross-disciplinary study while in an integrated vocational context such as law or teaching the model would be expected to be more relevant. there are existing studies that have applied wenger’s theories in the context of vles (e.g. rogers, 2000; chalk, 2001). these have, however, tended to take wenger’s general topics of ‘mutual engagement’, ‘joint enterprise’ and ‘shared repertoire’ as the basis for their work rather than anything more structured. their focus has also tended to fall uneasily between the learner and the environment without clearly identifying one from the other. however, following his discussions of the general dynamics and characteristics of communities of practice, wenger goes on to formalize these dynamics in his ‘learning architecture framework’ (laf) for a learning community of practice. this framework has the following defining characteristics or properties (wenger, 1998, pp. 237–239): facilities of engagement ● mutuality—interactions, joint tasks, help, encounters across boundaries, degrees of belonging. ● competence—opportunities to develop and test competences, devise solutions, make decisions. ● continuity—repositories, documentation, tracking, ‘participative memory’, storytelling, ‘paradigmatic trajectories’. facilities of imagination ● orientation—location in space, time, meaning and power. ● reflection—models and patterns, opportunities for engaging with other practices or break rhythm with the community mainstream. ● exploration—trying things out, simulations, play. facilities of alignment ● convergence—common focus or cause, direction, vision, values, principles. ● coordination—procedures, plans, schedules, deadlines, communication channels, boundary encounters and brokers. ● jurisdiction—policies, contracts, rules, authority, arbitration, mediation. by accepting these as key properties of a learning community of practice, this framework can be used as the basis of a more structured evaluation methodology, 130 r. ellaway et al. evaluating a vle in its context of use. it is proposed that by using this framework a vle can be evaluated in terms of its success and value in supporting these nine properties in the context of the community of practice that employs it. a note of caution should be added at this point. the approach advocated in this paper is a descriptive post-hoc evaluation model and is not intended as a template for designing a vle. as schwen and hara observe: while wenger’s work is a provocative ideal to achieve and useful as a dialogue between the designers and client systems, it is not a recipe for construction of such phenomena (schwen & hara, 2003, p. 262) this approach is also intended for use where a vle is one medium for course delivery amongst many. some approaches that draw upon the principles of communities of practice are predicated on the community of practice being either fully or predominantly online (notess & plaskoff, 2004). it is more usual in higher education for a vle to provide scaffolding and support within a multi-modal environment (ellaway et al., 2003). method: developing a vle evaluation tool based on the laf the first stage of the development process was, starting with wenger’s laf, to move from the general component factors of the laf to increasingly specific questions aimed at evaluating vles. the first three steps of this process are shown in table 1 and were at this stage fully derived from wenger. the second stage was to extend wenger’s theories to develop a pool of vleoriented questions based on the ‘specific aspects’ column of table 1. this pool of questions was piloted with a variety of members of the target learning community and, as a result, a number of questions were combined, rephrased or omitted. a particular outcome of this piloting was the development of a three-stem structure for each question, based on general effectiveness, personal utility and personal value. the questions were then rephrased as value statements and participant response option was structured against likert scales. rather than creating new scales, the likert scales were selected from those available in the online evaluation system that was to be used in delivering the instrument. the instrument was piloted again and further refinements and adjustments made. in addition to questions derived from the laf, a number of questions were included to verify responses against other data (such as server logs) and to act as ‘consistency traps’. when completed the evaluation instrument comprised of 60 items (see table 2). the instrument was designed to be administered to all of the vle’s users, irrespective of their role in the learning community supported by the vle; whether student, academic or support staff. this is, as was noted earlier, an inherently ‘situated’ and faceted approach; the nine-point laf has no inherent hierarchy or ranking of importance or relevance. these can only be judged or evaluated in the context at hand. indeed, any given course context may well contain distinct constituent communities of practice (such as evaluating a virtual learning environment 131 staff or students) that themselves hold contrasting and conflicting perspectives and value systems within the broader course community of practice. in taking this approach the authors are making the following assumptions: ● that wenger’s theories adequately model a community of practice; ● that the subject area or discipline has a strong identity as a community of practice; ● that the community of practice encompasses the whole course. this approach does not seek to test theories of communities of practice. rather it assumes a pre-existing course community of practice as a given reference point and thereby evaluates a vle by its ability to support that course community of practice. table 1. first steps in developing wenger’s learning architecture into an evaluation instrument (after wenger, 1998, pp. 237–239) general factors general questions specific aspects mutuality does/will the system support and facilitate the required mutuality and how important is this? interaction joint tasks peripherality competence does/will the system support and facilitate the required competences and how important is this? initiative and knowledgeability accountability tools continuity does/will the system support and facilitate the required continuity and how important is this? reificative memory participative memory orientation does/will the system support and facilitate the required orientation and how important is this? location in space location in time location in meaning location in power reflection does/will the system support and facilitate the required reflection and how important is this? models and representations comparisons time off exploration does/will the system support and facilitate the required exploration and how important is this? scenarios simulations practicum convergence does/will the system support and facilitate the required convergence and how important is this? focus, vision and values leadership coordination does/will the system support and facilitate the required coordination and how important is this? standards and methods communication boundary facilities feedback facilities jurisdiction does/will the system support and facilitate the required jurisdiction and how important is this? policies, mediation, arbitration and authority 132 r. ellaway et al. table 2. learning design questionnaire—generic format showing response type and mapping to the learning architecture framework no. root type laf mapping 1 how effective is the system’s engagement with the course in general? qa 2 how useful is the system at engaging you with the course? qa general 3 the system’s support of my engagement with the course is important to me … sa 4 how effective is the system in general at supporting interactions with students and staff on the course? qa 5 how useful is the system in supporting your interactions with students and staff on the course? qa mutuality, competence, continuity, coordination 6 the system’s support of my interactions with students and staff is important to me … sa 7 how effective in general is the system at supporting collaborative activities required by the course? qa 8 how effective is the system at supporting the collaborative activities you are involved in? qa mutuality, competence, continuity, exploration 9 the system’s support of the courses’ collaborative activities is important to me … sa 10 how effective is the system in general at providing the course information and help required for the course? qa 11 how useful is the system at providing the course information and help that you require to participate fully in the course? qa competence, continuity, orientation, convergence, 12 the system’s provision of course information and help is important to me … sa coordination 13 how useful is the system in general at interacting with university services and systems beyond the course? qa 14 how useful is the system at supporting your interactions with university services and systems beyond the course? qa mutuality, reflection, coordination 15 the system’s support of my interactions with university services and systems beyond the course is important to me … sa 16 how effective is the system in general at supporting assessment in the course? qa competence, continuity, 17 how useful is the system at supporting your assessment needs in the course? qa orientation, coordination, jurisdiction 18 the system’s support of assessment is important to me … sa 19 how effective is the system at providing the courses’ guidelines, rules and regulations? qa 20 21 how useful is the system’s provision of guidelines, rules and regulations you require? the provision of guidelines, rules and regulations by the qa competence, continuity, convergence, coordination, jurisdiction system is important to me … sa 22 23 24 how effective are the tools provided by the system? how useful to you are the tools provided by the system? the provision of tools by the system is important to me … qa qa sa mutuality, competence, continuity evaluating a virtual learning environment 133 table 2. continued no. root type laf mapping 25 how effectively in general does the system support progression through the course? qa 26 how useful is the system’s support of your progression through the course? qa continuity, orientation, convergence 27 the system’s support of my progression through the course is important to me … sa 28 how effective in general is the system at supporting out of hours working? qa 29 how useful is the system at supporting your need to work out of hours? qa continuity, orientation, reflection 30 the system’s support of my work out of hours is important to me … sa 31 how effective is the system in general at supporting teaching and learning activities at different locations? qa 32 how useful is the system at supporting your teaching and learning activities at different locations? qa continuity, orientation, reflection 33 the system’s support of my teaching and learning activities at different locations is important to me … sa 34 how effective is the system at providing timetabling and scheduling information? qa 35 how useful is the system at providing the timetabling and scheduling information you require? qa continuity, orientation, coordination 36 the system’s provision of timetabling and scheduling information is important to me … sa 37 how effective is the system in general at providing secondary learning materials? qa 38 how effective is the system at providing you with secondary learning materials? qa competence, orientation, exploration 39 the system’s provision of secondary learning materials is important to me … sa 40 how effective is the system at providing access to materials and resources that help with the reflective aspects of the course? qa 41 how useful is the system at providing materials and resources that help you with reflective aspects of the course? qa competence, orientation, reflection, exploration 42 the system’s provision of access to materials and resources that help with reflective aspects of the course is important to me … sa 43 to what degree does the system embody the focus, vision and values inherent in the course? qa 44 how useful is the system’s embodiment of the focus, vision and values inherent in the? qa orientation, convergence, coordination, jurisdiction 45 the system’s embodiment of the focus, vision and values inherent in the course is important to me … sa 46 how effective is the system at supporting the educational practices and methods of the course? qa 47 how useful is the system at supporting the educational practices and methods of the course? qa mutuality, competence, convergence, coordination 48 the system’s support of the educational practices and methods of the course is important to me … sa 134 r. ellaway et al. using the laf evaluation instrument the laf evaluation instrument was used to evaluate the ‘edinburgh electronic medical curriculum’ (eemec),3 a purpose-built vle system supporting the undergraduate medical course at the university of edinburgh. it was considered that this course had a strong existing course community of practice: students followed a common and integrated programme of study that was not shared with any other students; the programme of study was intrinsically focused on inducting students into medical practice; it was taught by practicing clinicians (often in real clinical contexts), it had a strong socializing agenda, and it had a continuous history going back over more than two centuries.. eemec was already well established across the course supporting approximately 1,200 students and approximately 700 staff across all five years of the programme. the development and characteristics of eemec have been described elsewhere (warren et al., 2002; ellaway et al., 2003). table 2. continued no. root type laf mapping 49 how effective is the system at supporting feedback and evaluation within the course? qa 50 how useful to you is the system at supporting feedback and evaluation within the course? qa mutuality, continuity, coordination 51 the system’s support of feedback and evaluation within the course is important to me … sa 52 how effective is the system at tracking student and staff use of the system? qa 53 how useful to you is the system’s ability to track student and staff use of the system? qa competence, continuity, jurisdiction 54 the ability to track student and staff use of the system is important to me … sa 55 overall i think the system is a very useful system in helping me engage with the course … sa 56 overall i think the system is a very valuable system in helping me engage with the course … sa general 57 overall i think the system is a reliable system in helping me engage with the course … sa 58 how often do you use the system? on 59 how responsive to requests for help and/or support is the system? qa 60 are there any aspects of the system that you think should be added to, improved or changed to make the system more useful to you? ft response types: qa = excellent to awful; sa = strongly agree to strongly disagree; on = all the time to never; ft = free text; yn = yes/no. evaluating a virtual learning environment 135 the survey instrument was deployed using eemec’s own ‘evaluation engine’ which allows staff to create, schedule, deliver, record and analyse questionnaires online (wylde et al., 2003). this is done by mapping different copies of the instrument in the vle database to the groups that would receive them. by setting a start and end time, when a user logged in to eemec with the period set, the system would check to see that an uncompleted questionnaire was set for the user’s group and, if this was the case, would present the questionnaire to them in a pop-up window. although a log of who had completed questionnaires was kept, this was separated from the responses so that they were anonymized at the point of storage and only one response per individual was permitted. the period set for delivery was from 10 april to 30 april 2003, a period that mapped on to different years based on their term or rotation schedule: year 1—on vacation to 15th april then starting term 3. year 2—on vacation to 15th april then starting term 3. year 3—new clinical rotation started 10th april. year 4—in middle of clinical rotation (24/2 to 30/5). year 5—in middle of clinical rotation (31/3 to 23/5). it is important to note that this was the first time that staff had been surveyed in this fashion. the response rates (shown in detail in table 3) were high overall although the staff responses were particularly low. the figure of 699 staff includes 50 or so guest logins and a large number of clinical and related staff who have a relatively peripheral engagement with the course. it is a peculiarity of medical education that a large number of clinical staff will be involved in teaching but only for a fraction of the working year. thus, despite the high potential numbers of staff in the course, at any given time only a relatively low number are actively engaged in teaching. this explains to a major degree the relatively poor responses in the staff cohort. there is also a notable lower response rate in year 3 with only just over half of the year responding relative to that in year 4. in the 2002–3 academic session, eemec was progressively less important to the students’ engagement with the course in later years, as there were less tools and materials provided, and it was expected that there table 3. responses for all cohorts course role population returns % returns year 1 students 236 207 87 year 2 students 214 186 87 year 3 students 258 142 55 year 4 students 221 192 87 year 5 students 176 120 68 all staff 699 45 6 136 r. ellaway et al. would be a gradual decrease in the response rates in later years. in this respect, the relatively high response rate in year 4 is more atypical than the low response rate in year 3. this may be interpreted as being due to year 3 students focusing on orienting themselves within their new clinical attachments and not engaging with eemec to a great extent while year 4 students, already established in their attachments, were using eemec to research and submit coursework as well as to link back to their peers and tutors. it is acknowledged that using eemec as the medium to evaluate itself could have introduced bias to the returns, in that only eemec users can have responded. however, the high response rates achieved, the near-mandatory requirement for students to access eemec regularly and the high commitment to evaluative activity across the course in general are considered to have ameliorated the effects of any such bias. analysis and interpretation triad analysis (effectiveness/utility/importance) questions 1–54 were delivered in triads framing the same question in terms of general effectiveness, personal utility and personal importance. a mean for each respondent for each of the three stem variants was taken. this was then analysed for internal reliability by calculating a cronbach’s alpha reliability coefficient, which showed a high degree of internal reliability (α=0.8652). non-parametric correlation analysis (spearman’s ρ) was then carried out between each of the three pairings (respondent means) with the following results: ● there was a significant positive correlation between general effectiveness and personal utility (ρ=0.914, n=821, p<0.0005). this indicates that there are fairly balanced feelings as regards eemec; respondents did not rate eemec as particularly good or bad in general relative to eemec’s usefulness to them. analysis of this correlation would be expected to indicate whether a vle had a particular subjective reputation-bias relative to its objective evaluation. in the case of eemec effectiveness and utility were essentially equivalent in the respondents’ minds. ● there was a significant positive correlation between general effectiveness and personal importance (ρ=0.429, n=818, p<0.0005) and a significant positive correlation between personal utility and personal importance (ρ=0.448, n=818, p<0.0005). respondents considered eemec’s importance to them was less than its perceived general effectiveness or personal utility. these pairings are taken to indicate the degree to which the vle is the medium for course business. high positive correlations would indicate that the vle was the principal medium for the course, no correlation that the vle was no more or less important than other media for the course and high negative correlations that the vle was of little or no relevance to the course. the data reflects the situation that eemec is a significant medium for course business but not the largest or most important one. evaluating a virtual learning environment 137 laf validation since the questions had originally been generated from the laf, it was important to verify that the predicted mapping between questions and the laf was statistically valid. factor analysis and inter-item reliability and correlation tests were performed. the factor analysis identified fourteen underlying significant factors with eigenvalues higher than 1.0, the first 13 of which were interpretable with the dominant factor being that of ‘personal importance’ (these are shown in table 4). although the factor analysis was interesting there was no strong equivalence between it and table 4. factor analysis and interpretations (extraction method: principal component analysis, rotation method: varimax with kaiser normalization) factor rotation sums of squared loadings questions with eigenvalues over 0.3 interpreted factor description total % of variance cumulativ e % 1 9.296 15.756 15.756 3,6,9,12,15,18,21,24,27,30,33 ,36,39,42,45,48,51,55,56,57,5 8 personal importance and relevance 2 3.875 6.567 22.323 1,2,7,8,25,26,46,47 general course participation 3 3.781 6.409 28.732 13,14,15,34,35,37,38,43,44 external connectivity 4 3.589 6.083 34.816 17,22,23,28,34,35,41 support of activities 5 3.380 5.729 40.544 17,31,32,40,41,42,43,44,45,4 6 educational support 6 3.109 5.270 45.814 26,28,29,31,32,59 personal logistics 7 2.204 3.735 49.549 49,50,52,59 feedback 8 2.187 3.707 53.256 19,20,25 authority 9 1.978 3.352 56.608 45,52,53,54 tracking and protection 10 1.906 3.230 59.838 4,5 communication 11 1.887 3.199 63.037 16,17,52 assessment 12 1.848 3.132 66.169 2,10,11 provision of information 13 1.720 2.915 69.084 25,46,47,59 general support 138 r. ellaway et al. the laf map although there was some congruence. however, as the responses were overall very positive and therefore heavily negatively skewed, there was a low level of variance and therefore a factor analysis would not be expected to be a particularly illuminative tool. an inter-item test of reliability (cronbach’s alpha) was performed for each set of mapped responses to the laf (as shown in table 5). this was performed on the personal utility component of each triad (the previous section displayed that there was a very high correlation between effectiveness and utility and a reasonably high correlation between utility and importance). the results show a strong level of consistency across the question groups and therefore an acceptable level of reliability for the question-laf map (none of the reliability coefficients were <0.8). a non-parametric correlation analysis (spearman’s ρ) was also carried out for all item pairs. there was no significant difference between the mean correlation for the laf mappings and the overall mean correlation. this indicates that, although a reasonable level of reliability has been established, the overall mapping is not very strong and further work needs to be done to refine this part of the instrument. overall laf analysis having established the question-laf map, each factor was analysed for each respondent group. the results of this are shown in figure 1. from this a ranking of factors was plotted as shown in figure 2. learning architecture framework scores for eemecranked learning architecture framework factors for eemec table 5. questionnaire-to-laf map and inter-item reliability question triad mutuality competence continuity orientation reflection exploration convergence coordination jurisdiction 4, 5, 6 x x x x 7, 8, 9 x x x x 10, 11, 12 x x x x x 13, 14, 15 x x x 16, 17, 18 x x x x 19,20, 21 x x x x x 22, 23, 24 x x x 25, 26, 27 x x x 28, 29, 30 x x x 31, 32, 33 x x x 34, 35, 36 x x x 37, 38, 39 x x x 40, 41, 42 x x x x 43, 44, 45 x x x x x 46, 47, 48 x x x x 49, 50, 51 x x x 52, 53,54 x x x cronbach’s inter-item alpha reliability coefficient 0.886 0.917 0.926 0.920 0.838 0.818 0.876 0.914 0.823 evaluating a virtual learning environment 139 figure 1. learning architecture framework scores for eemec figure 2. ranked learning architecture framework factors for eemec 140 r. ellaway et al. individual question triad analysis each of the question triads was then analysed across respondent groups and in comparison to the triad average and the overall average. a graph for each triad provides a useful illustration of the dynamics within the course community for each of the 18 issues addressed. a few examples are shown in figures 3 and 4. involvement with the mbchb course—triad average sd = 0.17, overall average sd = 0.30). interpretation: eemec is considered to provide a highly effective and useful service to the mbchb community of practice in engaging it with the course. however, because the course is not predominantly delivered or mediated online, eemec is considered to be less important by the community than it is effective or useful. although there is deviation between cohorts, this is relativelysmall and indicates a relative consensus response. the staff cohort shows atypically higher importance rating while year 4 shows an atypically low importance rating. conclusion: eemec is successfully mediating members of the mbchb community engagement with the course. there is a gradual decrease in score across the course with staff scoring somewhere in the middle.providing timetabling and scheduling information question average sd=0.43, overall average sd=0.30. possible interpretation: eemec’s provision of timetabling and scheduling information, although scoring reasonably well, shows little consensus between cohorts. most rated the importance of this factor higher than eemec’s effectiveness and utility in this area. personal utility was generally rated higher than general effectiveness. conclusion: although not entirely lacking,eemec’s provision of timetabling and scheduling information to the mbchb community could be significantly better. this appears to be a priority for the community that is currently insufficiently well supported. free-text analysis question 60 was a free-text response allowing any free-text comments to be added. about 30% of respondents took up this option. their comments largely echoed the findings from the rest of the questionnaire or made suggestions for specific changes or developments in the system. these were fed back to the course community as the basis for discussion within eemec’s steering and user groups. these comments will also contribute to the overall triangulated vle evaluation but fall outside the immediate scope of this paper. figure 3. involvement with the mbchb course:triad average sd = 0.17, overall average sd = 0.30). interpretation: eemec is considered to provide a highly effective and useful service to the mbchb community of practice in engaging it with the course. however, because the course is not predominantly delivered or mediated online, eemec is considered to be less important by the community than it is effective or useful. although there is deviation between cohorts, this is relatively small and indicates a relative consensus response. the staff cohort shows atypically higher importance rating while year 4 shows an atypically low importance rating. conclusion: eemec is successfully mediating members of the mbchb community engagement with the course. there is a gradual decrease in score across the course with staff scoring somewhere in the middle. evaluating a virtual learning environment 141 longitudinal and parallel measures the edinburgh mbchb is heavily evaluated (like most other medical courses). evaluation fatigue is a major concern and, as the instrument is relatively large, a repeat survey will not take place until the autumn of 2004 (establishing an 18 month cycle). this evaluation has recently been used on a vle supporting undergraduate medical education in another university. a full data analysis is not yet available but provisional analysis indicates a similar ranking of laf factors to those shown in figure 2. respondent group bias no accommodation has been made in the analysis for cohort bias as each academic year in the course has the approximately the same make up regarding gender, ethnicity etc and there is a significant mixing up between cohorts mid-course as about 40% of each year’s students take a year out to do an intercalated honours course. figure 4. providing timetabling and scheduling information:question average sd=0.43, overall average sd=0.30. possible interpretation: eemec’s provision of timetabling and scheduling information, although scoring reasonably well, shows little consensus between cohorts. most rated the importance of this factor higher than eemec’s effectiveness and utility in this area. personal utility was generally rated higher than general effectiveness. conclusion: although not entirely lacking, eemec’s provision of timetabling and scheduling information to the mbchb community could be significantly better. this appears to be a priority for the community that currently considers itself to be insufficiently well supported. 142 r. ellaway et al. delivery bias as the vle was the medium for delivery of the questionnaire, it is reasonable to anticipate a degree of bias from the fact that eemec users were the only ones who could have responded. the fact that use of eemec is mandatory for a number of course activities and the high student response rates indicate a fairly comprehensive coverage of the overall population however. discussion the development and use of an evaluation instrument (based on wenger’s learning architecture framework) to investigate the utility of a vle in the context of a specific course context has proved to be useful in capturing many of the dynamics of a vlein-use and thereby contributing to a broader holistic evaluation process. validation of the instrument has however pointed out limitations in the mapping between the instrument and the laf. further work therefore needs to be done in refining this tool both in terms of the mapping between questionnaire items and the laf and in carrying out longitudinal and parallel studies using this instrument. the application of the evaluation instrument to an undergraduate medical vle (eemec) indicates that there are aspects of the vle that can be improved, particularly in the areas of course coordination (e.g. timetables), jurisdiction (e.g. rules and authority) and exploration (e.g. secondary learning materials) while other aspects are relatively strong. analysis of its effectiveness, utility and importance show that the community using eemec has a reasonably realistic view of this vle, and analysis of the individual questions provides feedback on specific aspects of how the system relates to its community of practice. it is important to emphasise that this is a theory-based approach, which is predicated on a pre-existing course community of practice. in those situations where this is a valid assumption, for instance in subjects such as medicine, then it has immediate relevance and utility. for other situations, for instance in modular programmes of study, where communities of practice may not equate to a course (or even exist coherently at all) then there may be less relevance in such a study, although a module may in some cases retain a degree of internal coherence as a community of practice. this approach to evaluating vles does not provide information on what the vle can do, nor what its features are or even how it is used. these can be obtained from other sources and indeed these are usually already reasonably well known within the vle’s user community. what it does help to provide is a perspective of how successfully the vle is serving the communities of practice involved with the course in question, and thereby is able to provide pointers to areas in which it could be improved to the benefit of that community. it is important to restate that this approach is intended to contribute to a multidimensional ‘triangulated’ approach to vle evaluation; other components may include log file analysis, use case expressions and analysis of impact although elaboration of these approaches are outwith the scope of this paper. evaluating a virtual learning environment 143 the proposal is that the laf is not presented as a prescriptive framework around which a vle should be designed and built; such prescriptive frameworks have been identified as inimical to professional practice (lisewski & joyce, 2003). it is descriptive rather than prescriptive; the themes used and the insights gained are recommended as tools to inform the reflective practices of those responsible for the design and delivery of vle systems within coherent community of practice contexts. as schwen and hara point out: a rich descriptive theory is not a warrant or recipe for the construction of certain phenomena, and a useful prescriptive theory may not provide a full understanding of the phenomena but rather a perspective on the conditions or circumstances of its applied use. (schwen & hara, 2003, pp. 261–262) eemec has been developed organically over a number of years in response to the course’s needs and wishes. this process was not directly based on the laf although it was developed with the intent to support the whole 5-year programme. the laf evaluation is therefore not intended to be a formal validation of a particular vle design but rather as a contribution to the ongoing and iterative development of a vle in support of its course and course community. it is also intended as a technique for practitioner-researchers who need immediate and reflexive insights into vle design, deployment and support for specific educational contexts. conclusion as oliver (2000) observes: ‘evaluation forms a unique meeting point between policy, theory and practice, and as a consequence, it seems unlikely that its practice will ever be uncontentious’. however, the approach presented here has been grounded in theory, is based around a holistic view of course-vle instances and has provided significant utility to the authors in the evaluation of their own work. oliver (2001) identifies a potential weakness in this kind of approach when others seek to use it— ‘the purpose of theory may not be fixed but may depend on the way users appropriate it’. thus the level to which other users may find utility in this work may depend on the degree of agreement and alignment in approach and philosophy with that of the authors, and the contexts they are working in. in situations where vles are used in programme settings where there is a strong and coherent community of practice then there is particular benefit in carrying out this kind of evaluation. there is often a major investment in, and dependence on, vles in these contexts (cook, 2001), yet little in the way of appropriate evaluation instrumentation to supply those making this investment with feedback as to their efficacy and outcome. it is hoped that the laf framework can also provide a common language with which to compare different systems or ways of working with vles. even if a vle is only supposed to provide one or two aspects of support to a course it is still valid to apply this approach to investigate, for instance, the ‘functionality creep’ as the community using it tends to adopt originally unintended affordances from the system. all of these factors and issues will need further attention. 144 r. ellaway et al. the development of approaches to evaluating vles as described in this paper can contribute insights on the multifactorial interactions between communities of practice and technology-mediated extensions to their learning environments and make a strong and valid contribution to broader holistic and triangulated evaluation programs. notes 1. http://www.chest.ac.uk/datasets/vle/checklist.html 2. http://www.edutools.info/course/compare/all.jsp 3. eemec-online at www.eemec.med.ed.ac.uk references alsop, g. & tompsett, c. (2002) grounded theory as an approach to studying students’ use of learning management systems, alt-j, 10(2), 63–76. barajas, m. & owen, m. (2000) implementing virtual learning environments: looking for holistic approach, educational technology and society, 3(3). available online at: http://ifets.ieee.org/ periodical/vol_3_2000/barajas.html/ breen, r., jenkins, a., lindsay, r., & smith, p. (1998) insights through triangulation: combining research methods to enhance the evaluation of it based learning methods, in: m. oliver (ed.) innovation in the evaluation of learning technology (london, university of north london). britain, s. & liber, o. (1999) a framework for the pedagogical evaluation of virtual learning environments. jtap report 41. available online at: http://www.jtap.ac.uk/reports/htm/jtap041.html bruce, b. c., peyton, j. k. & batson, t. w. (1993) electronic quills: a situated evaluation of using computers for writing in classrooms. available online at: http://alexia.lis.uiuc.edu/∼ chip/pubs/ equills/siteval/index.shtml chalk, p. d. (2001) learning software engineering in a community of practice—a case study. cal2001 conference, warwick. available online at: http://www.ics.ltsn.ac.uk/pub/conf2001/papers/ chalk.htm cook, j. (2001) the role of virtual learning environments in uk medical education. jtap report 623. available online at: http://www.ltss.bris.ac.uk/jules/jtap-623.pdf ellaway, r., dewhurst, d. & cumming, a. (2003) managing and supporting medical education with a virtual learning environment—the edinburgh electronic medical curriculum, medical teacher, 25(4), 372–380. graham, g. (1999) the internet: a philosophical enquiry (london, routledge). jenkins, m., browne, t. & armitage, s. (2001) management and implementation of virtual learning environments: a ucisa funded survey uk, ucisa. available online at: http://www.ucisa.ac.uk/ groups/tlig/vle/vlesurvey.pdf jisc and ucisa (2003) managed learning environment activity in further and higher education in the uk (prepared by siru (university of brighton), education for change ltd & the research partnership). koper, r. (2000) from change to renewal: educational technology foundations of electronic learning environments (open university of the netherlands). available online at: http://eml.ou.nl/introduction/docs/koper-inaugural-address.pdf lave, j. & wenger e. (1991) situated learning: legitimate peripheral participation (cambridge, cambridge university press). lee, m. & thompson, r. (1999) teaching at a distance: building a virtual learning environment (jtap 033) (jtap, uk). available online at: http://www.jisc.ac.uk/uploaded_documents/ jtap-033.doc evaluating a virtual learning environment 145 lisewski, b. & joyce, p. (2003) examining the five-stage e-moderating model: designed and emergent practice in the learning technology profession, alt-j, 11(1), 55–66. notess, m. & plaskoff, j. (2004) preliminary heuristics for the design and evaluation of online communities of practice systems, elearn magazine. oliver, m. (ed.) (1998) innovation in the evaluation of learning technology (london, university of north london). oliver, m. (2000) an introduction to the evaluation of learning technology, educational technology and society, 3(4). available online at: http://ifets.ieee.org/periodical/vol_4_2000/intro.html/ oliver, m. (2001) what’s the purpose of theory in learning technology? (alt special interest group for theory and learning technology positional paper). available online at: http://homepages.unl.ac.uk/%7ecookj/alt_lt/oliver.htm richardson, j. a. & turner, a. (2000) a large-scale ‘local’ evaluation of students’ learning experiences using virtual learning environments, educational technology and society, 3, 4. available online at: http://ifets.gmd.de/periodical/vol_4_2000/richardson.html rogers, j. (2000) communities of practice: a framework for fostering coherence in virtual learning communities, educational technology and society, 3(3). available online at: http:// ifets.ieee.org/periodical/vol_3_2000/e01.html/ scarborough, h. & corbett, j. m. (1992) technology and organization: power, meaning and design (london, routledge). schwen, t. m. & hara n. (2003) community of practice: a metaphor for online design? the information society, 19, 257–270. warren, p., ellaway, r. & evans, p. (2002) meet george … using learning technology in a ‘blended’ approach to enhance the integration of knowledge and understanding across a 5year medical course, asme 2002 conference proceedings, norwich, uk. wenger, e. (1998) communities of practice: learning, meaning and identity (cambridge, cambridge university press). wylde, k., ellaway, r., cumming, a. & cameron, h. (2003) electronic submission and delivery of student feedback, amee 2003 conference proceedings, bern, switzerland. table 1. first steps in developing wenger’s learning architecture into an evaluation instrument (after wenger, 1998, pp. 237-239) evaluating a virtual learning environment in the context of its community of practice rachel ellaway*, david dewhurst & hamish mcleod 1. 2. 3. table 2. learning design questionnaire-generic format showing response type and mapping to the learning architecture framework table 2. continued table 2. continued table 3. responses for all cohorts table 4. factor analysis and interpretations (extraction method: principal component analysis, rotation method: varimax with kaiser normalization) table 5. questionnaire-to-laf map and inter-item reliability figure 1. learning architecture framework scores for eemec figure 2. ranked learning architecture framework factors for eemec figure 3. involvement with the mbchb course:triad average sd = 0.17, overall average sd = 0.30). interpretation: eemec is considered to provide a highly effective and useful service to the mbchb community of practice in engaging it with the c... figure 4. providing timetabling and scheduling information:question average sd=0.43, overall average sd=0.30. possible interpretation: eemec’s provision of timetabling and scheduling infor mation, although scoring reasonably well, shows littl... doi:10.3402/rlt.v20i0.16151 making science real: photo-sharing in biology and chemistry jenny waycott a *, barney dalgarno b , gregor kennedy a and andrea bishop b a health informatics and virtual environments, faculty of medicine, dentistry and health sciences, the university of melbourne, melbourne, australia; b school of education, charles sturt university, wagga wagga, australia (received 23 march 2011; final version received 19 august 2011) in this paper, we examine students’ reflections about the value of two photosharing activities that were implemented in undergraduate biology and chemistry subjects. both activities aimed, broadly, to provide support for authentic and meaningful learning experiences in undergraduate science. although the activities were similar � both required students to capture and share images as part of an independent inquiry activity � students in the biology case study were more positive, overall, than the chemistry students in their evaluation of the activity. in this paper, we examine the findings from the two case studies in parallel to provide insight into our understanding of meaningful learning in undergraduate science. the results suggest that, for meaningful learning to occur, the learning activity needs to be well aligned with students’ individual learning goals and with the objectives and characteristics of the course. in the two case studies examined in this paper, this alignment was successful for the biology case study but less successful in the chemistry case study. keywords: science education; photo-sharing; meaningful learning introduction the challenge of creating meaningful learning experiences for students is an enduring concern for higher education in general and undergraduate science in particular. learning science typically involves developing knowledge about complex systems and sometimes difficult and abstract concepts that are at odds with the (mis)conceptions students bring to the science classroom (laurillard 2004; linn, eylon, and davis 2004). making science accessible to students, then, is a key concern for science educators (linn, davis, and eylon 2004). we conducted evaluations of two technology-mediated learning activities that aimed, broadly, to make science learning more meaningful for students. in the first case, undergraduate biology students took photographs of beetle specimens collected during independent fieldwork and loaded the images to a shared website. in the second case, first-year chemistry students took photographs of everyday objects or experiences that illustrated chemistry principles in action and shared these with classmates on the flickr website. in this paper, we examine students’ reflections about the value of these activities, comparing the two cases to provide insight into our understanding of meaningful learning in undergraduate science. (page number not for citation purpose) *corresponding author. email: jwaycott@unimelb.edu.au research in learning technology vol. 20, 2012 rlt 2012. # 2012 j. waycott et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 16151 doi: 10.3402/rlt.v20i0.16151 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 background perspectives on authentic learning one of the prominent approaches to fostering meaningful science learning � that is, learning that is personally relevant and engaging for students � is to create authentic or ‘‘real world’’ learning experiences. however, authentic learning has been interpreted in a number of ways. one interpretation suggests that when learning is situated in the context in which it will ultimately be applied, students will see its relevance and importance, will be more motivated to learn and will ultimately have a better chance of transferring their knowledge outside the classroom. this is based on brown, collins and duguid’s (1989) concept of situated cognition and aspects of lave and wenger’s (1991) ideas about student participation in communities of practice. key to this interpretation is genuine participation within the community of practice, in which learners engage in the ‘‘ordinary practices of the culture’’ of the profession (brown, collins, and duguid 1989, p. 34). however, it can be challenging to implement these kinds of authentic learning experiences in formal settings (stein, isaacs, and andrews 2004). in university settings, many undergraduate subjects service students from a range of courses, creating numerous possible professional contexts and making it unfeasible to situate learning in a genuine community of practice. a second interpretation of authentic learning focuses on the tasks themselves (as distinct from the context in which they are carried out). herrington et al. (2004) argued that investigation activities that mirror the ill-structured and often openended nature of real-life tasks can be authentic even if not necessarily situated in a professional context. the idea of generative learning environments proposed by the cognition and technology group at vanderbilt (1990) draws on this aspect of authenticity, as does lebow and wager’s (1994) idea of cognitive apprenticeship. in undergraduate science education, one approach might be to design tasks that involve authentic scientific inquiry; for example, conducting experiments or undertaking fieldwork to collect data for an investigation that emulates the sort of scientific inquiry conducted by researchers in that discipline. however, students who do not see themselves as ultimately becoming scientists (for example, those studying a service subject towards a professional qualification) may not be engaged by simulated research tasks. a further interpretation takes a more learner-centred approach and relates more closely to the concept of making learning personally meaningful to students. stein, isaacs and andrews (2004) argued that learning experiences may be authentic if they ‘‘engage students’ lived experiences’’ and help them to ‘‘find meaningful connections with their current views, understandings and experiences’’ without necessarily being situated within a community of practice (p. 240). an alternative way to make science more meaningful and accessible to students then is to create learning activities through which students build connections between their formal knowledge and their personal experiences (linn, davis, and eylon 2004; vavoula et al. 2007). such activities might be situated within students’ ‘‘real lives’’ outside of university rather than within a particular professional context. in biology, for example, it is common practice to ask students to go into their everyday environments and collect biological specimens. leonardo da vinci’s experiments where he dropped objects of various sizes and masses from the leaning tower of pisa is an age-old example from physics, and many teachers have asked students to carry out similar experiments themselves. j. waycott et al. 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 tasks requiring students to undertake simple chemistry experiments in their kitchens at home provide another example (see, for example, boschmann 2003). tasks such as these share with laboratory experiments a fundamental sequence of activities: observing, recording and analysing/interpreting. elements of this sequence can be supported and enhanced by the use of new technologies. mobile technologies, in particular, can be useful for capturing and recording data in inquiry activities that take place outside the classroom (scanlon, jones, and waycott 2005). in recent years, mobile devices that have digital cameras embedded in them have become pervasive everyday technologies and offer the potential to enhance the data capture (or recording) aspect of scientific inquiry activities. in addition, photoand video-sharing web tools, such as flickr and youtube, can be used to share the digital images students have captured. by sharing resources in this way, students can learn from their peers’ experiences and see the relevance of a wide range of scientific concepts and ideas, extending the analysing/interpreting component of scientific inquiry. this points to a further interpretation of authentic learning that draws on sociocultural perspectives (e.g., vygotsky 1978) to emphasise the social and discursive aspects of learning. in the context of science education, undertaking collaborative scientific investigations or sharing and discussing findings can help students reach common understandings (rule 2006). rule argued that one of the key components of authentic learning is that ‘‘students engage in discourse and social learning in a community of learners’’ (p. 2). while the concept of ‘‘learning community’’ has been used widely, and it has been argued, problematically in recent years (see draper 2008, for a review), it is nevertheless useful for distinguishing peer learning from the notion of communities of practice. in this paper, we refer to a learning community as a group of peers who are engaged in similar learning tasks and goals; for example, a cohort of students enrolled in a subject. there are numerous possibilities for enhancing or extending learning communities through the use of new technologies. photo-sharing appears to offer particular possibilities; sites such as flickr, which enable users to ‘‘tag’’ photographs with keywords and connect to other users who share similar interests, have been shown to foster communication within informal learning communities (see, for example, burgess, foth, and klaebe 2006; miller and edwards 2007). recent research has also shown that photo-sharing can be useful in formal learning settings, as outlined below. learning through photograph taking and sharing there is a growing body of work suggesting that capturing and sharing images could add value in a range of learning settings. in one example, primary school students used handheld computers with plug-in digital cameras to record images of plants while undertaking an environmental science investigation (lai et al. 2007). in another example, university students learning about environmental issues used their own camera-enabled mobile phones to record images of local environmental blights. students sent the photographs to a moderator who uploaded selected photographs to the project website. participants could then review other students’ photographs, which became the focus of group discussions with classmates and the moderator (uzunboylu, cavus, and ercag 2009). in other fields, learners have used camera phones to illustrate progress made on major assignments, providing visual representations that aided self-reflection and facilitated communication between lecturers and students (cochrane and bateman research in learning technology citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 2009). camera phones have also been shown to be useful for students on work placement. for instance, vocational students have used camera phones and multimedia messaging to record their progress and communicate with teachers while working on a building site (hartnell-young and vetere 2005). education students working in school settings have used mobile phones to capture artefacts such as audio recordings and digital photographs of student work (ferry 2009). these were then shared with classmates in face-to-face discussions, providing a focal point for students’ reflections on the learning experience. as these examples show, capturing and sharing images can be particularly valuable for engaging learning communities and for supporting learning that is situated outside the classroom. however, while these technologies appear to offer valuable support for learning, educators should not assume that they can be easily incorporated into all learning activities. in some cases, students may not use the tools in the way educators had intended (e.g., petersen, divitini, and chabert 2008). recent research also suggests that students may resist using personal or everyday technologies in a formal learning setting (jones et al. 2010). further, the examples outlined here were typically small-scale trials that highlighted the potential benefits of learning through photograph taking and sharing but provided limited empirical evidence that these activities foster meaningful learning. in this paper, we examine findings from two case studies in which undergraduate science students captured and shared photographs to support their learning. the findings reported here are based on students’ perceptions about the value of the photo-sharing activity, particularly their reflections on the relevance of the activity to their learning and the value of sharing knowledge with their peers in this way. this evaluation aims to identify lessons about the challenges and learning possibilities of introducing photo-sharing activities to create meaningful learning experiences in undergraduate science. case studies the two case studies reported here were undertaken as part of a larger project, in which technology-mediated activities were implemented in eight different learning and teaching settings (see kennedy et al. 2009). this project aimed to examine the administrative, technical and pedagogical issues that emerged when social technologies were used as learning tools in different higher education contexts. the two case studies examined in this paper share many similarities. in both cases, students used digital cameras or camera-enabled mobile phones to capture digital photographs in independent activities that took place outside the classroom. both activities involved peer review and knowledge sharing, and both aimed to foster meaningful learning in an undergraduate science setting. however, there were also key differences. in the biology case study (beetle gallery), the photo-sharing activity was used to enhance an assessable fieldwork activity (collecting beetle specimens). in the chemistry case study (chemistry around us), the photo-sharing activity was a hurdle requirement: while students were not given a graded mark, they needed to complete it to pass the course. this activity also differed from the biology beetle collection exercise in that it did not emulate the sort of scientific inquiry that a professional scientist might undertake but was designed to encourage students to build connections between their formal learning about chemistry and their day-to-day experiences. that is, each activity represented a different interpretation of authentic learning but both aimed to facilitate knowledge-sharing to enhance meaningful learning. j. waycott et al. 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 beetle gallery the beetle gallery activity was implemented in a second-year biology subject at charles sturt university, a large multicampus regional provider of both on-campus and distance education. the activity was part of a larger assignment (worth 30% of the subject mark) in which students collected and described six beetle specimens. as well as physically collecting and displaying the specimens, students photographed each beetle from at least two angles, highlighting morphological characteristics such as antennae type, leg type, thorax type, the pronotum, abdomen shape, thorax colour, wing colour and wing patterns. students then uploaded their images, along with a document describing the beetle (and, optionally, video footage and/or audio recordings of the beetles) to an online beetle gallery, a file sharing site hosted on the university’s learning management system (lms). students created a new folder for each beetle specimen they uploaded, using a unique descriptor to label each beetle folder, including the antennae, leg and thorax type. students were also required to compare and contrast their beetles to other specimens in the beetle gallery. students’ contributions to the beetle gallery were worth 20% of the mark for the overall assignment. initially, 44 students were enrolled in this subject (25 on-campus and 19 distance education students), and they all participated in the beetle gallery exercise. however, two on-campus students did not complete the unit and did not submit the final assessment item. chemistry around us for this case study, first-year chemistry students at the university of melbourne, a large metropolitan university, used digital cameras to capture chemistry-related images from everyday life that they then published and shared with other students on the flickr website. students were given two topics each (from nine topics covered in lectures) and asked to take photographs, drawing on their everyday experiences that illustrated those topics. students were asked to publish at least two photographs on flickr (to a private group, set up by the project team) and to write textbook-style captions that outlined how their images illustrated that topic. students also reviewed other students’ photographs and captions and nominated the two best photographs for a specific topic. while students uploaded and viewed the photographs in flickr, they ‘‘voted’’ for the best images on the university’s lms. this activity was introduced for the first time in this implementation and was one of four independent learning tasks in this subject. during this case study, there were 920 students enrolled in first-year chemistry; 799 students completed the task. at the end of semester, 1894 photographs had been published and almost all photographs (97.6%) were accompanied by a text description (see waycott and kennedy 2009, for more information about this case study). method in both case studies, students were asked to complete an evaluation questionnaire that was used in each of the eight case studies for the project (see gray et al. 2009). the overall questionnaire was designed to address the aims of the larger study; for the purposes of this paper, we have chosen to focus on those items that can give us research in learning technology citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 some insight into whether students found the activity to be meaningful or personally relevant to their learning. the survey asked students to indicate whether they felt the activity had helped them in their studies this semester (‘‘yes,’’ ‘‘no’’ or ‘‘maybe’’) and to rate, on a seven-point scale, how relevant, interesting, useful and enjoyable they felt the activity was. in addition, five-point likert scales were used to gauge levels of agreement in response to several statements regarding how well the activity supported various aspects of students’ learning (e.g., ‘‘[this activity] helped me better understand the material i was studying’’). qualitative data were elicited through open-ended questions asking students to indicate, for example, the best and worst things about the activity, how they felt it could have been improved, and to elaborate on why they felt the activity did/did not support their learning. we also held focus group discussions and interviews with students at the end of the semester, eliciting more information about students’ perceptions of the photo-sharing activities. for the purposes of this paper, we have examined the qualitative data to identify key themes that provide an indication of students’ views on the relevance and usefulness of the activity. in the biology case study, 19 on-campus and four distance students completed the survey (55% response rate). a further four on-campus students participated in a focus group discussion and four distance students were interviewed individually by telephone. in the chemistry case study, the response rate to the online questionnaire was very low, with only 44 students completing the voluntary survey (in addition, some respondents did not complete all the questions). the low response rate highlights the need to interpret the quantitative data, in particular, with caution. however, given that, in this descriptive analysis, limited inferential statistics will be used, we feel it is appropriate to include these data. ten chemistry students participated in a focus group discussion. results students’ evaluation of the learning value of the activity when asked ‘‘did this activity help you in your studies this semester?,’’ most of the biology students who completed the survey responded positively (17 students, or 74%, said ‘‘yes’’ while 3 students, or 13%, said ‘‘no’’). in contrast, most of the chemistry students who took part in the evaluation said the activity did not help them in their studies (24 students, or 67%, said ‘‘no’’). a similar pattern can be seen in the levels of agreement/disagreement to statements describing the potential learning benefits of the activity (table 1). overall, students in the biology case study felt the activity supported their learning. in contrast, most chemistry respondents disagreed with many of the learning benefit statements. there were slightly more favourable responses from chemistry students to the statements ‘‘the activity improved my ability to share my knowledge and/or opinions with other students’’ (10 students, or 26% of respondents, agreed/strongly agreed) and it ‘‘helped me develop my thinking skills by writing or producing study related material’’ (10 students, or 26%, agreed/strongly agreed). overall, however, most chemistry respondents disagreed with the statements proposing the activity supported their learning. as shown in table 2, a t-test comparing the mean response on each item relating to the way the photo-sharing activity supported learning indicates that biology students were significantly stronger in their agreement on all items. j. waycott et al. 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 table 1. degree of agreement with statements about how the photo-sharing activity supported learning.* biology (n �23) chemistry (n �38) the photo-sharing activity . . . strongly agree agree neutral disagree strongly disagree strongly agree agree neutral disagree strongly disagree helped me better understand the material i was studying 3 13 6 1 0 1 3 8 11 15 improved my ability to reflect on what i was learning 3 12 6 1 1 1 6 7 12 11 improved my ability to share my knowledge and/or opinions with other students 5 12 4 2 0 2 8 8 14 6 helped me develop my thinking skills by writing or producing study-related material 3 12 7 1 0 2 8 9 10 9 helped me develop my independent research skills 2 12 7 0 1 1 6 11 10 9 *figures show the number of students who gave each response. some survey respondents skipped these questions. r e se a rch in l e a rn in g t e ch n o lo g y c ita tio n : r e s e a rc h in l e a rn in g te c h n o lo g y 2 0 1 2 ; 2 0 : 1 6 1 5 1 d o i: 1 0 .3 4 0 2 /rlt.v 2 0 i0 .1 6 1 5 1 7 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 students’ general perceptions of the activity in order to establish students’ broad perceptions of the learning tasks, we asked them to provide ratings of several single-word descriptors (e.g., ‘‘interesting’’ and ‘‘useful’’), using a 7-point scale (1 �‘‘not at all’’; 7 �‘‘extremely’’). overall, the students in the biology case study gave higher ratings to positive items (‘‘interesting,’’ ‘‘easy to use,’’ ‘‘useful’’ and ‘‘enjoyable’’) than they did to negative items (‘‘irrelevant’’ and ‘‘boring’’). in contrast, the mean ratings from the chemistry students who responded to the survey were generally lower for the positive items and higher for the negative items (see figure 1). as shown in table 3, a t-test comparing the mean response on each item indicates that biology students were significantly stronger in their agreement on the four positive items (interesting, easy to use, useful and table 2. comparisons between biology and chemistry students in degree of agreement with statements about how the photo-sharing activity supported learning (where 1 �‘‘strongly agree’’ and 5 �‘‘strongly disagree’’). biology chemistry the photo-sharing activity . . . mean standard deviation mean standard deviation t p helped me better understand the material i was studying 2.2 0.7 4.0 1.1 6.73 b0.001 improved my ability to reflect on what i was learning 2.4 0.9 3.7 1.2 4.75 b0.001 improved my ability to share my knowledge and/or opinions with other students 2.1 0.9 3.4 1.1 4.45 b0.001 helped me develop my thinking skills by writing or producing study-related material 2.3 0.8 3.4 1.2 4.10 b0.001 helped me develop my independent research skills 2.4 0.8 3.5 1.1 4.40 b0.001 figure 1. biology and chemistry students’ mean ratings of the learning task. j. waycott et al. 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 enjoyable) and chemistry students were significantly stronger in their agreement on the two negative items (boring and irrelevant). the qualitative data, elicited from survey responses and interview and focus group discussions, provide further information about these differences. a number of themes about learning and engagement emerged in the biology case study; these were not limited to the online aspect of the exercise. indeed, students frequently noted that the ‘‘best aspect’’ of the exercise was that they were required to physically collect beetles from their habitats, although some students also expressed frustration at the time and effort involved in sourcing beetles. one student described collecting and handling specimens as ‘‘opening your eyes to a lot of things that we might have covered in the text already but until you see a [beetle] under a microscope, it doesn’t stick in your memory nearly as much.’’ another noted: ‘‘on a page in a book [the beetle] looks quite defined . . . but actually looking at live specimens it is totally different.’’ many students indicated that they would like to see this kind of activity used in other areas of their studies. however, as one student clarified, this would only be considered valuable ‘‘if it was beneficial to the study [and] not [just] for the sake of it.’’ in contrast, chemistry students’ responses to the open-ended survey items were often negative, with comments such as: ‘‘it didn’t really aid any understanding of the course’’ and ‘‘i learnt about one little topic, not all the others.’’ a number of students clearly felt the task lacked relevance and interfered with other higher priority tasks, such as preparing for examinations. for example: i unfortunately failed to see the point [of this activity]. after discussing it with other students, i found that they thought similarly of the exercise. it seems an unnecessary and unhelpful hurdle that will not actually teach us anything practical. i thought i could have used my time more effectively, for example, i could’ve been revising the examinable material rather than mucking around trying to upload some photos. these students did not see the photo-sharing activity as relevant to their learning needs and goals in first-year chemistry. similarly, when asked what they believed the purpose of the activity was, a small number of students appeared to not understand that the primary aim of the activity was to encourage them to think about the role chemistry plays in everyday life. instead, they suggested that the purpose of the activity was to ‘‘make us use technology’’ or to take photographs. some students expressed exasperation at having to undertake a task that they felt was not relevant to this subject: ‘‘i signed up for chemistry, not photography!!!’’ table 3. comparisons between biology and chemistry students in degree of agreement with single word descriptors of the activity (where 1 � ‘‘not at all’’ and 7 �‘‘extremely’’). biology chemistry mean standard deviation mean standard deviation t p interesting 4.8 1.8 3.2 1.7 �3.56 0.001 easy to use 4.9 1.6 3.5 1.8 �3.12 0.003 useful 4.7 1.6 2.8 1.6 �4.35 b0.001 enjoyable 4.7 1.3 2.9 1.7 �4.3 b0.001 boring 2.2 1.4 4.1 2.0 4.1 b0.001 irrelevant 2.2 1.6 4.4 2.2 4.2 b0.001 research in learning technology citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 the students who took part in the focus group, however, were generally more positive in their reflections. they noted that the activity successfully enabled them to link their formal learning about chemistry to the ‘‘real world’’ around them. for instance, ‘‘it was good to actually put it [chemistry] into reality’’ and ‘‘i think learning by analogy is very, very helpful. so if you go out there and find an example of something . . . you’re always going to be able to refer to that later.’’ the focus group session also included some discussion about the value of using mobile technologies to capture and share images. these students ‘‘liked the fact you could [do the activity] from anywhere.’’ one student related how she had taken a photograph while out hiking, and another said she had taken one of her photographs while on the train. in these cases, students were using technologies that they carried about with them, making it easier to incorporate this learning activity into their everyday life. students’ perceptions of the knowledge sharing component of the activity as shown in table 1, 17 biology students who responded to the survey felt the activity improved their ability to share knowledge or opinions with other students (74% of respondents). this positive response was also reflected in a number of student comments from the qualitative data. the knowledge-sharing aspect appeared to be particularly beneficial because the cohort was geographically dispersed. one student noted: ‘‘i think it’s amazing that we’re all over the state, we’re all over the country and we can all look at the different beetles that we’ve selected. that’s amazing to me.’’ many biology students, both on-campus and distance, commented that the photo-sharing aspect of the task made them feel connected to the group, which appeared to improve engagement with the activity: ‘‘it made me feel involved in the subject . . . like i was part of a group. i guess for me that was the main thing.’’ in the chemistry case study, we did not find strong evidence that students felt more connected to the learning community by taking part in this activity. however, there were chemistry students who found it useful to share knowledge and review other students’ photographs. when asked to indicate the best thing about the activity, typical comments included: ‘‘there were some good photos in my review topic, and some explanations that were clear and concise on the topic and helped me to understand the concept’’ and ‘‘[it was] interesting to see what other people thought.’’ similarly, in the biology case study, students said they found the photo-sharing aspect of the exercise beneficial for their learning. they liked having the opportunity to self-assess and evaluate their progress by comparing and contrasting their work to other students’ work (e.g., ‘‘[it] helps being able to plot where you’re at’’). in addition, the photo-sharing activity exposed biology students to a broader range of beetle specimens and possible interpretations: going through everyone’s photos and saying ‘well that’s the same as my beetle but i’ve actually labelled it differently’ . . . it was interesting to see how different people interpreted those slight differences in those characteristics. having the opportunity to view a number of different beetles and their features gave a broader overview than if i had just focused on my own, some of which were quite similar. students in the chemistry focus group also noted this benefit: ‘‘it’s interesting to see other students’ perspectives on your topic . . . . you sort of looked at it and you’re j. waycott et al. 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 like ‘oh, so that’s what they thought’ whereas i might have thought something completely different.’’ however, a number of students in the chemistry case study felt there was limited value in having access to and viewing other students’ work. students noted that, in some cases, their peers appeared to misunderstand the instructions and made errors in their captions: ‘‘there were wrong/misleading descriptions of a concept.’’ in addition, students raised concerns about the quality of images they were asked to review. for example: ‘‘they’re usually non-professional photos and can be dull’’ and ‘‘a lot of [the photos] are very blurry . . . . you can’t see what they’re taking a photo of ’’. the repetition of photographs and the volume of pictures produced by such a large cohort also made the peer review component of the activity challenging (e.g., ‘‘everyone ended up doing similar pictures’’). perceptions about the quality of student-created material and the repetition in images led a number of students to question the value of reviewing other students’ work and the usefulness of a collaborative photo-sharing activity as a whole. discussion the two photo-sharing activities in these case studies both aimed, broadly, to provide support for authentic and meaningful learning experiences in undergraduate science. the two activities appeared, in many ways, to be quite similar: both involved, for instance, capturing and sharing photographs as part of an independent inquiry activity in undergraduate science. however, analysing the two case studies in parallel has revealed strong differences in the way students responded to the photo-sharing tasks in each setting. overall, the biology students who participated in the evaluation were more positive and rated the activity more highly than did the chemistry participants. most of the chemistry students who responded to the survey did not see the activity as relevant and meaningful for their learning. these results question our understanding of ‘‘meaningful learning’’ in undergraduate science, suggesting that the authenticity of the task is central to meaningful learning. the biology case study provided an example of a more authentic learning activity: the photo-sharing task contributed to a fieldwork activity that emulated the sort of task a professional biologist might undertake. students in this case study had no difficulty in recognising the value of the activity for their learning. in contrast, many of the chemistry students who responded to the survey questioned the relevance of the activity, sometimes expressing exasperation that they were being asked to spend time on an activity that they felt was ‘‘pointless.’’ however, it appears that numerous features of the learning setting and the activity design may have contributed to chemistry students’ negative evaluations of this activity, particularly how well the task aligned with individual students’ strategic approaches to learning and studying. despite the learning task having a clear and considered educational rationale � that is, to encourage students to consider how chemistry principles are embedded in everyday experiences � students did not see the activity as well aligned with their own, individual learning needs and goals. it seems that many students had clear views of the most important components of the firstyear chemistry curriculum and the most appropriate activities that needed to be completed; and, for many students, the photo-sharing activity was not one of them. the fact that the activity was not assessed but was a hurdle requirement would have contributed to students’ perceptions about the relative importance of the different research in learning technology citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 activities included in the curriculum. preparing and revising for an examination was considered important; taking photographs of chemistry principles in action was not. in contrast, the biology case study was part of a larger assessment activity. the fieldwork task of collecting and describing beetles had a clear place in the curriculum. the photo-sharing task extended this activity, making it possible for students to share their fieldwork experiences. it appears, then, that learning activities that are aligned with the course curriculum and assessment � in line with biggs’ (1996) notion of constructive alignment � may be viewed more favourably as relevant and meaningful for students. furthermore, knowledge-sharing appeared to have particular benefits for students in the biology case study because of the features of the setting in which this activity took place. first, some of the students were studying off-campus and found that being able to share and view other students’ work in this way made them feel more connected to the learning community. second, it was useful for students to be able to see other students’ beetle specimens to extend their learning. normally, in a fieldwork exercise such as this, students would only be exposed to the samples they had been able to source and observe. the photo-sharing task meant students could learn from a larger and more varied data set. third, this was a smaller group of students than in the first-year chemistry setting. chemistry students complained about the repetition of photographs, the quality of images and the volume of photographs that they needed to look at in order to complete the peer review component of the task. it appears that introducing photo-sharing in a large cohort such as this will not necessarily create opportunities for meaningful learning through knowledge-sharing. in contrast, students may find reviewing large volumes of student-created content to be a difficult and disengaging task. conclusion fostering meaningful learning in undergraduate science is a complex endeavour. designing authentic learning tasks, encouraging students to build connections between their formal learning and everyday experiences and facilitating knowledge-sharing within learning communities may help individual students in certain settings become more engaged as learners. however, these approaches may not be successful with all students in all contexts. for meaningful learning to occur, the task needs to align well with students’ individual learning goals and with the objectives and characteristics of the course. in designing authentic tasks for university students, alignment with the subject goals and the likely learning and achievement goals of the student may be more important than personal meaningfulness. moreover, the fostering of a learning community will only work if students have a personal reason for engaging with the task and if features of the learning community (e.g., its size) are conducive to knowledge-sharing and peer learning. photo-sharing in science may be a useful way of fostering knowledge-sharing and enriching the learning experience, but its relevance as a learning activity is dependent on the context in which it is used and how it contributes to the other learning activities students are engaged in. acknowledgements we would like to thank the students and staff who participated in these case studies, especially the lecturers in the department of chemistry at the university of melbourne and the school of environmental sciences at charles sturt university who implemented the case study activities. j. waycott et al. 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 particular acknowledgements are due to peter tregloan, carmel abraham, rosemary chang and andrea wilson. these studies were undertaken as part of the educating the net generation project, conducted in collaboration with sue bennett, rosemary chang, kathleen gray, terry judd, kerri-lee krause and karl maton. support for this project was provided by the australian learning and teaching council, ltd, an initiative of the australian government department of education, employment and workplace relations. the views expressed in this paper do not necessarily reflect the views of the australian learning and teaching council. references biggs, j. (1996) ‘enhancing teaching through constructive alignment’, higher education, vol. 32, pp. 347�364. boschmann, e. (2003) ‘teaching chemistry via distance education’, journal of chemical education, vol. 80, pp. 704�708. brown, j. s., collins, a. & duguid, p. (1989) ‘situated cognition and the culture of learning’, educational researcher, vol. 18, pp. 32�42. burgess, j., foth, m. & klaebe, h. (2006) ‘everyday creativity as civic engagement: a cultural citizenship view of new media’, proceedings of communications policy & research forum, sydney, [online] available at: http://eprints.qut.edu.au/ (accessed 27 june 2009). cochrane, t. & bateman, r. (2009) ‘smartphones give you wings: pedagogical affordances of mobile web 2.0’, in same places, different spaces. proceedings of the australasian society for computers in learning in tertiary education 2009, eds. r. atkinson and c. mcbeath, ascilite, auckland, pp. 142�152. cognition and technology group at vanderbilt. (1990) ‘technology and the design of generative learning environments’, educational technology, vol.31, pp. 34�40. draper, s. w. (2008) ‘learning and community’, in learners in the co-creation of knowledge: proceedings of the lick 2008 symposium, edinburgh 30 october 2008, eds a. comrie et al., napier universitypp, edinburgh, pp. 132�144. ferry, b. (2009) ‘using mobile phones to enhance teacher learning in environmental education’, in new technologies, new pedagogies: mobile learning in higher education, eds j. herrington et al., university of wollongong, wollongong, pp. 45�55. gray, k. et al. (2009) educating the net generation: a toolkit of resources for educators in australian universities, australian learning and teaching council, strawberry hills, nsw. hartnell-young, e. & vetere, f. (2005) ‘lifeblog: a new concept in mobile learning?’ proceedings of the 2005 ieee international workshop on wireless and mobile technologies in education, ieee, los alamitos, ca, pp. 174�178. herrington, j. et al. (2004) ‘designing authentic activities in web-based courses’, journal of computing in higher education, vol. 16, pp. 3�29. jones, n. et al. (2010) ‘get out of myspace!’, computers & education, vol. 54, pp. 776�782. kennedy, g. et al. (2009) educating the net generation handbook: a handbook of findings for practice and policy, australian learning and teaching council, strawberry hills, nsw. lai, c.-h. et al. (2007) ‘affordances of mobile technologies for experiential learning: the interplay of technology and pedagogical practices’, journal of computer assisted learning, vol. 23, pp. 326�337. laurillard, d. (2004) ‘rethinking the teaching of science’, in mediating science learning through information and communications technology, eds r. holliman & e. scanlon, routledge falmer, london and new york, pp. 27�50. lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation, cambridge university press, cambridge, england. lebow, d. g. & wager, w. w. (1994) ‘authentic activity as a model for appropriate learning activity: implications for design of computer-based simulations’, proceedings of selected research and development presentations, 1994 national convention of the association for educational communications and technology, nashville, tn. linn, m. c., davis, e. a. & eylon, b.-s. (2004) ‘the scaffolded knowledge integration framework for instruction’, in internet environments for science education, eds m. c. linn, e. a. davis & p. bell, lawrence erlbaum associates, mahwah, nj, pp. 47�72. research in learning technology citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 13 (page number not for citation purpose) http://eprints.qut.edu.au/ http://eprints.qut.edu.au/ http://eprints.qut.edu.au/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 linn, m. c., eylon, b.-s. & davis, e. a. (2004) ‘the knowledge integration perspective on learning’, in internet environments for science education, eds m. c. linn, e. a. davis & p. bell, lawrence erlbaum associates, mahwah, nj, pp. 29�46. miller, a. d. & edwards, w. k. (2007) ‘give and take: a study of consumer photo-sharing culture and practice’ proceedings of the sigchi conference on human factors in computing systems, eds. b. begole et al., acm press, new york, pp. 347�356. petersen, s. a., divitini, m. & chabert, g. (2008) ‘identity, sense of community and connectedness in a community of mobile language learners’, recall, vol. 20, pp. 361�379. rule, a. c. (2006) ‘the components of authentic learning’, journal of authentic learning, vol. 3, pp. 1�10. scanlon, e., jones, a. & waycott, j. (2005) ‘mobile technologies: prospects for their use in learning in informal science settings’, journal of interactive media in education, vol. 25, pp. 1�17. stein, s. j., issacs, g. & andrews, t. (2004) ‘incorporating authentic learning experiences within a university course’, studies in higher education, vol. 29, pp. 239�258. uzunboylu, h., cavus, n. & ercag, e. (2009) ‘using mobile learning to increase environmental awareness’, computers & education, vol. 52, pp. 381�389. vavoula, g., et al. (2007) ‘learning bridges: a role for mobile technologies in education’, educational technology, vol. xlvii, pp. 33�36. vygotsky, l. s. (1978) mind in society: the development of higher psychological processes, translation m. cole, et al., harvard university press, cambridge, ma and london. waycott, j. & kennedy, g. (2009) ‘mobile and web 2.0 technologies in undergraduate science: situating learning in everyday experience’ in same places, different spaces. proceedings of the australasian society for computers in learning in tertiary education 2009, eds. r. atkinson and c. mcbeath, ascilite, auckland, pp. 1085�1095. j. waycott et al. 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16151 doi: 10.3402/rlt.v20i0.16151 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16151 does distance e-learning work? a comparison between distance and face-to-face learners using e-learning materials sara de freitas and gerald r roberts birkbeck, university of london email: s.defreitas@bbk.ac.uk; gerald.roberts@ud.pc.uk this study compares continual assessment data, intake numbers, retention numbers and final examination grades of a mixed cohort of face-to-face and distance learners against similar data from previous years where e-learning materials were not used in order to test whether e-learning materials can support the same quality and quantity of teaching and learning for both face-to-face and distance learners. the results for this cohort of learners demonstrate that: (i) distance e-learners score as well and sometimes better than face-to-face learners; (ii) face-to-face student numbers have increased; (iii) overall, student retention and student attendance have been maintained; (iv) final examination results have been maintained or in some cases improved; (v) lecturer workload was high, but not unmanageable, and it is clear how manageability can be improved. introduction birkbeck college was an early advocate of distance learning and the use of ict (information and communications technology) to support student learning, provide distance learning courses (fourteen courses are currently offered for distance learners including four through the university of london external programme), provide online learning (moss and sansom, 2000), and use e-learning in courses more generally (school of crystallography; school of biological and chemical sciences and school of computer science and information systems). providing e-learning materials are embedded into course work with due consideration of pedagogy, context and learner requirements, the benefits of using e-learning to support 69 sara de freitas and gerald p. roberts does distance e-learning work? distance learners or distance e-learning in this article may provide greater flexibility for the learner (cullen, hadjivassiliou, hamilton, kelleher, sommerlad and stern, 2002; reynolds, caley and mason, 2002). in particular, electronic course materials may offer benefits including ease of access by students 'anywhere and anytime' (dfes, 2003). there is a range of literature that explores the factors such as motivation levels and retention that affect learning at a distance or for online learning (see inglis, 2000; collis and moonen, 2001; eisenstadt and vincent, 2000; stephenson, 2001). adding to this work, this paper focuses upon whether greater flexibility is compatible with high quality of learning provision in the absence of face-to-face contact between tutor and students with a particular reference to retention and completion rates. in the face of the growth in the numbers of distance learners in the united kingdom from 150,000 in 1996/7 to 200,000 in 2000/1 (source: hesa), and indeed the pressure which many institutions are under to increase student numbers, many are turning to e-learning materials to support distance learning. while there is a significant body of work conducted in the united states that has found 'no significant difference' between 'technology-based instruction' and face-to-face learning in terms of learner outcomes (for example, russell, 2001), there have been few similar studies in the united kingdom. this article attempts to test the hypothesis that distance e-learning can maintain or increase the quality and quantity of teaching in a united kingdom institution. the paper provides new quantitative data to allow comparison of the two forms of learning (distance and face-toface) with a single cohort of students, providing a comparison between: (1) numbers attending, (2) retention numbers, (3) continual assessment grades, and (4) final examination grades. in order to provide a control for the study, wherever possible the paper compares this single cohort with cohorts from previous years where no electronic teaching material was used. the e-content used for distance e-learning at the school of earth sciences included a range of digital materials: • full-text documents; • supporting materials; • high-quality graphical materials (including photos, diagrams and movies with sound); • interactive links to course schedules; • links to online resources; and • links to other lecture notes in the series. the distance e-learning materials were created using adobe acrobat writer™ which facilitated authoring of pdf files from other electronic media. in addition, whilst connected to the web, students can access the electronic library, email and other web resources using buttons placed on the pdf files. due to the central mission of birkbeck to reach people who work during the day, accessibility was a key issue in the mode of delivery of the materials; therefore e-learning materials were made available on cd-roms. this facilitated inclusion of high-quality photographs allowing the student to magnify images to study relevant details, just as a face-to-face student might magnify a specimen using a microscope or climb a hill to gain an overview of a geological scene. 70 alt-] volume 11 number 3 the course material was provided to both face-to-face and distance learning students. to ensure that students received weekly feedback, a feature that was thought to be particularly relevant for distance learning students, the final examination requirements for the year 2001-2 were changed from previous years. previously, final examinations consisted of a theoretical and practical examination. in 2001-2 the practical examinations were replaced by weekly-assessed practical tests the marks from these were returned within three weeks (except in the case of one course to allow evaluation of the effect of this facility; see below). three half-course units and one full-course unit from the b.sc. geology and b.sc. environmental geology were offered with cd-rom lecture material for the first time in 2001-2; the courses were also offered for distance learners for the first time. these four courses have been evaluated in this project in order to provide data to find out whether distance e-learning can maintain high-quality learning and whether e-learning materials are beneficial to both distance learners and face-to-face learners. the following courses were examined. • introduction to geology; (1.0 course unit: 20 lectures and practicals) complete on five cd-roms. assessed work was marked each week and returned to the students with a provisional mark. • structural geology 1; (0.5 course unit: 10 lectures and practicals) complete on one cdrom. assessed work was marked each week and returned to the students with a provisional mark. • structural geology 2; (0.5 course unit: 10 lectures and practicals) complete on two cdroms. assessed work was marked each week and returned to the students with a provisional mark. • global tectonics; (0.5 course unit: 10 lectures and practicals) complete on one cdrom. assessed work was returned at the end of the course. the same students studied both this course and structural geology 2. available data for these courses consists of entry numbers, attendance records and marks achieved by students. method the method used to evaluate the mixed cohort of learners consisted of a graphical analysis of trends in the data. (1) to investigate the quality and quantity of teaching and learning across the cohort, the following is presented: • the number of students for several years of intake is compared; • the retention rates of students for several years of intake are compared; • a general survey of student entry comments made at admissions interview for a sub-set of the 2001-2 cohort is presented and analysed; • average marks for several years of final examination results are compared; 71 sara de freitas and gerald p. roberts does distance e-leaming work? • the distribution of marks for several years of final examinations is compared; • mark accumulation during assessed course work is analysed including differentiation between (a) students who had previous degree qualifications at intake, (b) those who did not, and (c) distance learners; • a comparison of mark accumulation during assessed work and final examination results for two course units, where one course included weekly feedback and one did not; the same sub-set of the 2001-2 cohort took both courses. (2) to investigate whether there would be a decrease in face-to-face attendance if highquality lecture notes and graphics are provided, the following is presented: • attendance records for several years of students are presented and analysed; • attendance records for students admitted in 2001-2 who stated at interview that they may only attend sporadically (using the lecture notes provided on cd-rom for distance learning) are presented and analysed; • finally, an analysis of lecturer workload will be presented, commenting on preparation of the e-learning materials and provision of feedback to the learners. distance learning in earth sciences and the adopted method learning in earth sciences specifically requires student access to high-quality visualizations of the processes and products of geological evolution. it also requires the development of practical skills, which allow students to recognize and interact with varied geological materials and geological scenarios. traditionally, these have been provided by fieldwork and laboratory studies. of these, fieldwork rightly remains the primary focus of geological teaching (as specified by the benchmark statement for earth sciences, environmental sciences and environmental studies, and the need for 100 days of fieldwork during a b.sc. geology course to achieve accreditation and chartered geologist status within the geological society of london). traditional laboratory studies have always been regarded as essential because of the difficulty of delivering interactive practical scenarios with high-quality visualizations by any other method. however, the web is increasingly providing a viable alternative for the delivery of highquality materials, for example, the university of leeds has delivered high-quality materials over the web (see: http:llearth.leeds.ac.uk/peoplelbutlerlcontents.htm). due to the nature of these large files, for example, a single photograph that is of sufficient quality to allow examination at a variety of scales (as would be the case with a student increasing the magnification of a microscope or climbing a hill to gain a different perspective on a geological field scene) is at least 10mb in size when converted to a digital form. in the past, due to the slow download times for 56k modems, we considered that web delivery impeded attempts at delivering geological laboratory-style studies for distance learning. however, in the near future the web may provide a valid alternative for delivering earth science materials; as the numbers of students with broadband connections at home increase, the materials that are currently offered via cd-rom can be easily uploaded to the internet, when ease of access for all students can be assured. 72 alt-] volume i i number 3 a method has been developed whereby large file sizes can be delivered via cd-rom using a simple and freely available software package. most computers can use adobe acrobat reader7*1 to read pdf files. this reader software is present on most computers at the time of purchase or can be downloaded free from the adobe website. the user does not have to be connected to the web to access lecture material and could be using a laptop anywhere. however, when connected, links allow easy access to the web. the file sizes can be as large a several hundred megabytes for multipage lecture notes (the size of cd-rom) yet access speeds are a few seconds for a page containing four 10mb photos plus text. adobe acrobat writer™ allows an author to add 'interactive buttons' to lecture notes which permits the student to open web-links to electronic libraries or staff homepages, open files, go to specific pages in files, play movies and zoom in on photos and diagrams, conduct searches for words and display thumbnails to the pages of lecture notes. when printed the page format is controlled by the author rather than by the particular browser software chosen by the student. this allows the lecturer to control the exact format of printed pages seen by the student. the software allows inclusion of full colour graphics, is easy to use and compresses file-sizes to a minimum. a standard but individual 'house style' can be developed with ease to standardize lecture note styles from individual departments/institutions. the files have been placed onto cd-roms using a cd multicopier purchased for this purpose by the school of earth sciences. the cd-roms are given to distance learning and face-to-face students as part of the normal course with no extra fee. the practical material for the four courses studied in the project was developed over a considerable period. in total the materials took about 175 days to develop; this was spread over a period of about four years. this broke down into: • twenty days to research and write the text for each 0.5 course unit; • thirty days to draw appropriate diagrams on computer for each 0.5 course unit; • five days to scan appropriate photographs for each 0.5 course unit; • fifteen days to paste the text, diagrams and photographs into the house format pdf file and to make the electronic links, buttons and interactive features for 0.5 course unit. results comparison of students intake for the years 1993-2001 to assess whether the introduction of e-learning materials influences student numbers, the number of students entering single honours b.sc. geology is plotted in figure 1. after a decrease in student numbers following 1993 when many oil companies moved offices out of the london area following a change in taxation on oil exploration in the north sea, student numbers enrolling for introduction to geology were between nine and seventeen. cd-rom lecture material was advertised and offered for the first time in 2001. intake rose in october 2001 to thirty-six, a number which included seven distance learning students. the total number of students entering the school of earth sciences in the year 2001-2 was fifty-three, a number which included students studying for joint honours degrees. 73 sara de freitas and gerald p. roberts does distance e-leaming work? : stuctent intake to slngfe honoura geology 4 0 30 4—r 20 10 -ft 1923 1934 [995 1035 iss7 1398 1933 2000 2001 year ris^is dl«jblei sftsi cd_r0m \&ctut£t rl'sstei ml retention rates of students for the years 1993-1994 to 2001-2002 to assess whether the introduction of e-learning material influences student retention rates, the number of students not taking the final examination for introduction to geology in the years 1994-2002 is compared with the number that enrolled for the course (see figure 2). there is no clear pattern of increased or decreased retention over the period studied. however, it is noted that retention was worse in the year following introduction of e-learning material than in the previous four years, but similar to that of five years previously. figure 2: percentage net taking intio to geology bam c u © q. 40 20 0 n n" 1 d m 1 1993 1994 1995 1935 1997 1993 1959 2cq0 2001 year ma clear paitafn emerges ccncairing stutlsrr.3 who took std fnal •3eok:-s'y s x for the intake of 2001 into introduction to geology, the percentage of students who took the final examination was similar for face-to-face and distance learners (see figures 3a and b). however, the percentage of face-to-face learners who left the college was about double that of the distance learners (see figure 3c). thus, although both face-to-face and distance learners enrol and then do not take the final examination, a larger proportion of distance learning students in this sample remained with the college, deferring examinations to a later time compared with face-to-face students. 74 o it a n c3 . 0 ? rt = j-' ; o o 1 u 1.1 j u. u. a m w> n c 11. o r j ccoooo o co i£) t fj c! s d q q o on a t n § 1 . is o o q o o o o u! irt ^r rn cm »— u •zz-zz lm2 9681 561=: £86eco2 7hv? lcus •'x-zz 6661assi • 9661 56sl .1? r n q a o o o o a o »— ~ ii ' ^ .-^ t i w) -r ri -\j — •s 5 •< = 665!. 366'.. i66'. 966i s66'. *-ss(. £66'. 2c-o2 tcos ccffi 6661 3ss!. s661. rest £661 111 "i f ' f — ̂ i ~i ii r i a ??. ill ji3 § 3 5 = 55 •a 1u i j a 3 ••y 8 a j il ts ••y sara de freitas and gerald p. roberts does distance e-leaming work? survey of student entry comments made at admissions interview students were questioned at interview to assess their backgrounds and reasons for enrolling. the students include occasional, non-degree students and joint honours students in addition to the mixed cohort of single honours face-to-face and distance-learners described above. the results are presented in the table 1. course intake number bsc geology . 29 distance learning 10 bsc environmental science/physical sciences/ geology and physics 8 non degree/occasional 4 bsc environmental geology 2 53 in total table i: intake number across the courses summaries of the findings are as follows. • seventeen students already had degrees out of fifty-three (32 per cent). • the average age of students was thirty-five. • twenty-five of the students said they had never heard of birkbeck, but found the college on the web. • ten students (who are not distance learners) said they were especially attracted by the use of e-learning material; nine of these would not have come to the school of earth sciences without cd-rom material and all nine of these enrolled for the b.sc. geology. the most notable finding is that nine students indicated that they would not have enrolled without the provision of e-learning material. this was due to the expectation that they would use the distance learning option as a safety net if they could not attend all the lectures. thus, these nine students together with the ten students who enrolled for distance learning, increased the student numbers by nineteen (36 per cent of the total intake of fifty-three) as a direct result of provision of e-learning material. comparison of average marks at final examination for the years 1994-2002 to assess whether the e-learning material helps to maintain the quality of teaching and learning, as evidenced by final examination results, data for the years 1994-2002 were compared (the results for the year 1995 are not available and structural geology 2 and global tectonics run every second year, see figure 4). the examination results for 2002 in introduction to geology showed the highest mean percentage on record. the examination results for 2002 in structural geology 1 showed the second highest mean percentage on record. the examination results for 2002 in structural geology 2 showed the second highest mean percentage on record. marks for global tectonics were similar to previous years. external examiners confirmed there was no difference in the level of examinationss for these years. 76 alt-j volume 11 number 3 distribution of average marks at final examination for the years 1994-2002 to assess whether the e-learning material helps to maintain the quality of teaching and learning, as evidenced by distribution of final examination results, data for the years 1994-2002 were compared (the results for the year 1995 are not available and structural geology 2 and global tectonics run every second year, see figure 5). it was judged that too little data was available for analysis using histograms so the standard deviation of the results was analysed. the examination results for all four courses show no clear pattern of variation in standard deviation. mark accumulation through time during assessed course work to assess whether continual assessment associated with e-learning material influences achievement as evidenced by accumulated marks, the cumulative marks scored by students through time are presented (see figure 6). overall findings of the study on the introduction to geology showed: • there was no obvious difference in achievement between those with or without a degree upon entry; • distance learners have a high drop-out rate, but can achieve first-class marks; • the single first-class student improved with time evidenced by a better fit to the data using a second-order polynomial than with a linear equation. however, this highlights one of the few examples of clear improvement associated with continual assessment and weekly feedback; • most students who do not drop out follow linear learning curves, in some cases interrupted by flat portions to the curve caused by not handing in some assessments; • the lack of upwardly concave curves, which would indicate improvement in achievement through time, may be due to several factors: i. the early practicals may be too easy compared with abilities of the students (although external examiners state that the assignments are appropriate for this stage of a b.sc. geology); ii. the later practicals may be too difficult compared with abilities of the students (although external examiners state that the assignments are appropriate for this stage of a b.sc. geology); iii. students were not trying as hard in later practicals; iv. upward concavity may be hidden in stepped curves associated with not handing in practicals. the results of the structural geology 1 course are presented in figure 7. the findings show that: • there is no obvious difference between distance learning and face-to-face students; • distance learners can achieve first-class marks; • most students who do not drop out follow linear learning curves, in some cases interrupted by flat portions to the curve caused by not handing in some assessments; 77 1—:— l "—""^ " .zocz cgsz 3s6l 3sel s-s6l — 1 1 l —t—i— i koz 8651 i xir i y o 'o o zooz i.oo2 'ccoz 8661 .3661 s66l eegl i! a § / < _ i ' ' * ; " * ' " " * > ' • ' • _ . ' i i ^ ^ tooz ••cooz •66 6 1 • 8 6 6 1 . •iggl .9661 .s66c kg l £66c 2000 1800 1600 koo 1200 1q00 eoo 600 400 200 0 jr m *™ 1*1**" r jj g o * 16 i t or •lisjl'is&'lv.'ci t figure c b inlrodutlbn toceclog1,* siuclenls 'aillionl iicvj !•>:• liilr,1 iniklljjiicelc-jiii.r'im m 1800 •jf 1c0d g 1400 ~ 1200 > icon •3 boo coo 400 —y-r• " . > « . ... • j+ 10 fkjuie mnoducibn 1000 ,„ 1b00 -k 1600 | moo qj 1200 •£ 1m0 s mo p eoo u 200 5 10 15 ixc-s or a i i « s a | viva k there is no otwious different^ in achievemem between those with or uiihout a degres upen entry. (black symbols shew i st class marks; bls:k crosses shew p3ss marks. cistoncq lesrners have a high drop-cut rate but can achieve 1st class msil.-s. thosin^lo 1st oass student improved with time. other symbols :hdw nia-k zzcumulation for individual students! f l y u i * &\ and e: inl iociik 1 tan to gcclogy: c;i uwlhc un» foi si i i : t n l x, wii li p.w dlffci enl hesii fii func i v:mi to li r.t-sl fcjjilo •aiic-ll ier tl>? sludimit ;iu: unui!;it a i mai b in o llne.'ii c< .u"t;doi:ii ing p;il! i. > 101x1 • ^ gou • "3 ^ijli l> d. v a a i 7 a « ' j 5a.»3r«-9.5.*ca uj n ^iifi? j f c * * ^ e ^ "'••ca v 7s.3o73i12'x.11 p* n qd î7 * * * * * * '11 i 111 1'j i'o 0 i 10 l i 30 the ainglo isc class distance learner iniproved ftrcucjri trot: evidenced by 2 better fit to tho data using a 2nd cdor polynomial than tuth a linear ecjjition. hotvever, this is cub of the few examples of clear iir.prcvoment associated with continual assessment and weekly feec&sck. most students who do net drop-oui follcw linear learning curves, in some cases interuptcd by flat portions to tho curve caused by not handng in some assessments. the lack of upwardly-concave curves, which would indicate improvcnicrit in adiid'/etnent through time, may bo duo to several factcs: i f tho early practical? may be too easy: 2} the later practic3ls may he too difficult; 3) students 3ro not trying as hard in later pra-ticals; 4 i upward concavity may be hidden in stepped curves issaciatcd with not handing in practicsla. sara de freitas and gerald p. roberts does distance e-learning work? figu re 7a: structural geolcgy 1 : distance learning students 900 aoo 700 600 500 400 3ao zao ico 0 ; • : * y i ! • • • • • • • " . « i t r * • • • • : • • • • • •• * ••: < : : • • ' : ' • : t • 1 i 5 10 practials (symbols represent individ ual srudenls) figure 7b: structural gsolcgy 1: face to face students 1200 ^ 1000 c aoo > n 6qc § 400 u 200 -•'-'; ••> t.. . . " ! t ' . ! i ' : " " . . c " : \ • ' " r " • ' • • • • ' " . . •c t • ' • ' ' • :'-:;r | 1 • • • • i f -1 2 4 6 practials (s)tn bcls leptean t indhidual sudaitsi 10 there is no obvious difference between the achievements of face to faceand distance learning students. distance learning students cart achieve 1 st class marls. the lack of upwardly-concave curves, which would indicate improvement in achievement through time, may tie due to several factors: 1) the early practical may be too easy: 2) the later practical may be too difficutt: 3) students ate not tiying as ha rd in later pratticals; 4) upward concavity may be hidden in stepped curves associated with not handing in practieals. (black circles show i st class marks: black erases shew pass marks) • the lack of upwardly concave curves, which would indicate improvement in achievement through time, may be due to several factors, as mentioned above. comparison of mark accumulation during assessed work and final examination the same students took the courses: structural geology 2 and global tectonics. however, structural geology 2 assessed work each week, which was marked and marks were handed back each week. global tectonics assessed work each week, but the marks were only returned at the end of the course. the results (figures 4 and 8) show that: • ten out of eleven students achieved better marks in structural geology 2 during assessed work; • in terms of final marks, the mean mark in 2002 for structural geology 2 was 60 per cent (standard deviation 14 per cent) with a mean of 57 per cent for global tectonics (standard deviation 20 per cent); (see figure 4). the above may indicate that handing back marks each week improves learning evidenced by marks achieved during assessed work. alternatively, it may indicate that the practieals in structural geology 2 were easier than in global tectonics although external examiners did not confirm this. the final examination results were not distinguishable for the two courses, indicating that marking and returning assessed work each week has not had an obvious effect in terms of increasing final marks. attendance records for the years 2000-2001 and 2001-2002 to test whether provision of e-learning material affected attendance, records for the years 2000-1 and 2001-2 are compared in detail for the course introduction to geology (see figure 9); less detailed analysis of the other three courses is also included (see figure 10). 80 figure aa si 800 % 600 i 4t>0 "§ 200 3 o • strut 2 a global tectonics 5 praclfcal 10 flguie bb 4 0 0 zqo £l • etruc 2 « global tectonics 0 s piactlcal 10 flguta be 8 0 0 goo 400 200 0 ! & • • • • strue 2 t) global tectonics 10 practical flguiefid 'fe 800 | 600 i 400 00 0 struc 2 tectonics 0 5 practical 10 figure se flguie bf figure figure fill j * struc 2 s gicbst tectonics 600 400 200 0 • struc 2 u global tectonics boo 600 400 200 0 • struc 2 ei global tectonics • struc 2 5 practical 10 5 piactlcal h 1000 e 800 | 600 j! 400 1 200 3 0 a • struc 2 h global tectonics 800 600 400 200 0 • • • struc 2 h global tectonics 0 5 practical 10 s practical 10 ft] lire sk | 800 « 600 s 400 1 200 3 o • struc 2 h global tectonics 5 practical the aa/ne students took structural geaiogy 2 and glcfcil teetuijes. structural cijulcvjry 2 had assessed work eadi meek wtilcli was marked and tiaiiderj b i t k nltli a n-iirk eatfi meek. qoliat tectaiilfs juul assessed noik eaai week but the maiks were only returned &i the end of the course. 10 out ot 11 5tu<tata actileved better marks in stiuciural geology 2. th£ may indcaie that llantltig back niaiks thdi week impruves learning. alternatively, it may hmtcate ihat the practitali in structural geology 2 were easier titan in global teutonics althciu(fi this via* not uancinntd by external: exarvjuas. sara de freitas and gerald p. roberts does distance e-learning work? figure 9a: introduction to geology: patterns of individual student attendence2co0-2001 25 20 | 1 5 1 10 3 5 <2 0 + * * «: w * * li * *•* k 1 a a 3 * , 4 j i >. mi3 _ _ u, r ' ^*t k 3z 3 jc • ^ 2pi ***' v "£ tt n j » ' • <» ib i i ,u ••> 4& vtf a • * * 10 * * week 15 20 25 (symbols represent indn'idlql stlctents) figure 9b: introduction to geology: patterns of individual student attendance 2301-20q2 25 20 4 3 15 1 3 2 sit'* r. ; : • f -*e 3 -x-il i k e fe f • • * • x pi • 'm •jfrsr r • • —-+—: • c • • -m-a 10 15 week 20 25 (s>mbds represent individual students) there is little difference in the pattern of cumulative weeks attended by individual students between 2000-2001 and 2001-2002. the average week's attendance is 12.2 (61 per cent) for 2000-2001 and 12.5 (63 per cent) for 2001 -2002. 82 alt-] volume 11 number 3 figure 100 -r a 9 0 " 01 80 f eo<u 50 y 40flj 30-1201 0 • 0 • ioa:structural geology l:'?.j attendance • " ' . : v * ' ' . • > • ' ' • • : -: -"*:. '•'•'': tj 2000-2001 2001-2002 year figure lgc:globaltectonics;'?i attendance • ' 9 0 si bos' 7 0 rsoft 5 0 ' h! 4 0 £ 3 0 2 0 10n ;»a:4? r 2000-2001 2001-2002 year figure 9 0 « 80 10figur9 0 a) 80 ra 7 0 ^ 6 0 <u 5 0 y 4 0 £ 3 0 2 0 10c'b: structu i^ 1 geo log*/ 2: ':':j attendance . , ; . . . • . . . 2000-2001 2001-2002 yaar ! iod: introdu'rtion to geology: % attendance : ' ; . . • : • . • ; • • • • • • . • ' : • ' j . • • . • . . • ' • , . ' ' • ••'-' 2000-2001 2001-2002 year i in terms of the percentage of weeks attended there is little difference between the same courses run in the years 2000-2001 and 2001 -2002, with two courses showing an increased percentage and two courses showing a decreased percentage. for introduction to geology, there is little difference in the pattern of cumulative weeks attended for the two years in question. the average weekly attendance is 12.2 (61 per cent) for 2000-1 and 12.5 (63 per cent) for 2001-2. a comparison for the years 2000-1 and 2001-2 for all four courses shows that in terms of percentage weeks attended there is little difference between the same courses run in the years 2000-1 and 2001-2, with two courses showing an increased percentage and two courses showing a decreased percentage (see figure 10). attendance records for students with sporadic attendance an assessment has been made of whether students who anticipated at interview that they would use the distance learning option in part actually behaved in this manner (see figure 11). for the eight students who anticipated using the distance learning option in part and attended at least part of the course, two of the four who completed the course show flat portions to their curves indicating non-attendance. one student attended seventeen out of twenty lectures with the other attending fourteen out of twenty lectures. four students did not complete the course. the drop-out rate of 50 per cent is similar to that of the course 83 sara de freitas and gerald p. roberts does distance e-leaming work? figure 11 attendance record for face to face students who anticipated using the distance learning option in part s cd 20 15 1 0 5 0 . , • ' i . ' . ; • , • ' j " * * " ^ ' • " • ' " • • ' " ; -\j,-j> i . ' y ' ^ . ' >,-.•...̂ -_.'.:,.-.'-..;.-?, *i:iis-i.l;~y,''\ ''•'•: }'•''•''-;''-:--''•.•• : ^ r ; v " v . ; ' ' > ' u ; v ^ " ' ; : • / ; > • • ' • • • • • " • • ; > f ' ^ j v > • • • * • * ! • • • : • " • " ' . • * . " ' • • • • . . ' • . ' " . ; ' • • . : ; • • " • * • • . ! ' * m ' " " " • • • • 4 4 » • • • > : ; ; t j ; . • • ; ; ; . : u:;,f« • • 4—4• : •.',' ' i . . " • • ' • " • • " • • • • » , , , . . . • , • ; . . « • « —4—4' • ' ' . ' • • ' z ' • ' \ ' * • ' ' • ' ' : ; ' a ' ' . " • . ' * ; % * • • • " : • ; . . • • • • . . • . • . • • ^ : • . ; • . • ; ; ; • • h :^ : ' • : • \ . c •• • ? " . • • " , • • --.-. ' • •-•' ( • . • " " > • • • . . . : " , • • — • 10 15 20 25 weeks for the 8 students who anticipated using the distance learning option in part and attended at least part of the course, 2 of the 4 who completed the course show fiat portions to their curves indicating non-attendence. one student attended 17 out of 20 lectures with the other attending 14 out of zo lectures. 4 students did not complete the course. the drop-out rate of 50% is similar to that of the course overall (42% for face to face; 50% for distance learning). 2 of the 8 students attended all the lectures. overall (42 per cent for face-to-face; 50 per cent for distance learning). two of the eight students attended all the lectures. thus, half of the students who indicated at interview that they would attend sporadically due to provision of e-learning material, attended every week. lecturer workload in the october-december term the following courses ran: introduction to geology; global tectonics; structural geology 2. in the january-march term the following courses ran: introduction to geology; structural geology 1. grading the assessment provided a large proportion of the allocated teaching time. it was found that grading and providing student feedback took about one day per week for each course. thus three days per week were spent marking in the first term and two days per week in the second term. further time was spent on the enrolment and administration of the distance learners, including setting up a website for advertising purposes. an email, feedback service for students was run and widely advertised. although it was expected that an email feedback service would provide the main method of feedback to students and would be very timeconsuming, there were only two email queries in the whole year. 84 alt-] volume 11 number 3 summary of results and discussion the provision of teaching and learning material in an electronic format produced the following features: • an increased intake of students into the b.sc. geology course as it both allowed for the provision of distance learners and increased the face-to-face student intake due to increased flexibility as perceived by the students; • no noticeable effect on the percentage of students who took the final examinations through the years 1994-2002; • sufficient quality ensuring that similar percentages of face-to-face and distance learners took the final examination in the year 2001-2; • sufficient quality ensuring that a higher percentage of distance learning students were retained by the college to enter the following year compared with face-to-face students; • maintenance and in some cases an increase in the achievement of students at final examination. this is supported by the fact that these materials were introduced in the year prior to the highest examination mean percentage on record for introduction to geology and second highest for structural geology 1 and structural geology 2; but clearly longer-term data are desirable; • no noticeable effect on the distribution of final examination results around the mean percentage as evidenced by standard deviations (see appendix 1); • no noticeable effect on weekly or total attendance evidenced by a comparison of cumulative attendance growth curves for introduction to geology and percentage attendance levels for the three other courses (see appendix 1); • very limited non-attendance in a subset of students who indicated at the initial interview that they anticipated utilizing the distance learning option in part even though they were registered as face-to-face students; • a high workload, as it took 175 days over four years of a lecturers' time to author the elearning material; • assessed work was marked and returned on a weekly basis; findings from this showed no noticeable difference in terms of final assessed work marks or weekly accumulation rates for marks compared across students who (i) already had a degree prior to entry, (ii) no prior degree on entry, (iii) distance learners; • results from the assessment did not commonly produce concave upwards cumulative mark accumulation curves indicative of weekly improvement in performance: most students followed linear mark accumulation curves although several other external factors that have not been analysed may have contributed to this; • higher marks and steeper mark accumulation curves compared to a course where marks were not returned each week although several other external factors that have not been analysed may have contributed to this final examination results were very similar for the two courses; 85 saru de freitos and gerald p. roberts does distance e-learning work? • assessment conducted on a weekly basis took up fifty days of a lecturers' time out of a hundred days available (two terms of ten weeks) for research, administration and teaching; • overall, the results are consistent with the hypothesis that the provision of electronic materials to support teaching and learning can maintain the quality and quantity of teaching. most notably, findings of the pilot project run showed that: • overall student numbers have increased; • student retention and student attendance have been maintained; • final examination results have been maintained or in some cases improved. other benefits of providing electronic materials for the lecturer included: • better organization of the teaching materials; • time-saving in the long run for lecture preparation time, although assessed work produced a high to unmanageable workload. other notable aspects of the comparison include: • the benefits of weekly assessment were not borne out by any significant effect on student marks; • although email support was offered to the distance learners this was rarely used and did not provide a significant aspect of the course tutors' interactions. it may be that the large amount of feedback given to students each week cut down on the feedback they required via email. it may be more time-effective to provide email feedback with answers to the assignments provided on a separate cd-rom. as a consequence practical examinations were reintroduced in the year 2002-3; practical work will now be self-assessed with the answers provided on a separate cd-rom. the email feedback service will also be made available. conclusions this study has shown 'no significant difference' in the quality of teaching and learning in the cohort regardless of whether the learner was a face-to-face or a distance learner (russell, 2001). the introduction of distance learning material has increased student numbers. our work provides a positive test of the hypothesis that distance e-learning can maintain or increase the quality and quantity of teaching in a united kingdom institution, providing e-learning materials are well-considered, fit into the tutor's pedagogic approach and are clearly presented. references collis, b. and moonen, j. (2001), flexible learning in a digital world: experiences and expectations, london: kogan page. cullen, x, hadjivassiliou, k., hamilton, e., kelleher, j., sommerlad, e. and stern, e. 86 alt-] volume 11 number 3 (2002), review of current pedagogic research and practice in the fields of post-compulsory education and lifelong learning, london: tavistock institute. department for education and skills (2003), towards an e-learning strategy: consultation document, nottingham: dfes publications. eisenstadt, m. and vincent, t. (eds) (2000), the knowledge web: learning and collaborating on the net, london: kogan page. inglis, a., ling, p. and joosten, v. (2002), delivering digitally: managing the transition to the knowledge media, london: kogan page. moss, d. and sansom, c. (2000), 'principles of protein structure: an established internetbased course in structural biology', aly-j, 8, 2, 29-39. reynolds, 1, caley, l. and mason, r. (2002), how do people learn?, london: chartered institute of personnel development. russell, t. (ed.) (2001), the 'no significant difference' phenomenon: a comparative research annotated bibliography on technology for distance education, north carolina state university, international distance education certification center. school of earth sciences, university of leeds web pages, see http://earth.leeds.ac.uk/ people/but/er/contents.htm, last accessed 13 november 2003. stephenson, j. (ed.) (2001), teaching and learning online: new pedagogies for new technologies, london: kogan page. 87 alt-j, research in learning technology vol. 12, no. 3, september 2004 issn 0968–7769 (print)/issn 1741–1629 (online)/04/030261–17 © 2004 association for learning technology doi: 10.1080/0968776042000259573 don’t write, just mark: the validity of assessing student ability via their computerized peer-marking of an essay rather than their creation of an essay phil davies* university of glamorgan, uk taylor and francis ltdcalt120306.sgm10.1080/0968776042000259573alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2004association for learning technology123000000september 2004phildaviesschool of computinguniversity of glamorganpontypriddmid-glamorgan wouth walescf37 1dlpdavies@glam.ac.uk this paper reports on a case study that evaluates the validity of assessing students via a computerized peer-marking process, rather than on their production of an essay in a particular subject area. the study assesses the higher-order skills shown by a student in marking and providing consistent feedback on an essay. in order to evaluate the suitability of this method of assessment in judging a student’s ability, their results in performing this peer-marking process are correlated against their results in a number of computerized multiple-choice exercises and also the production of an essay in a cognate area of the subject being undertaken. the results overall show a correlation of the expected results in all three areas of assessment being undertaken, rated by the final grades of the students undertaking the assessment. the results produced by quantifying the quality of the marking and commenting of the students is found to map well to the overall expectations of the results produced for the cohort of students. it is also shown that the higher performing students achieve a greater improvement in their overall marks by performing the marking process than those students of a lower quality. this appears to support previous claims that awarding a ‘mark for marking’ rewards the demonstration of higher order skills of assessment. finally, note is made of the impact that such an assessment method can have upon eradicating the possibility of plagiarism. introduction davies (davies, 2002a) reported a student comment ‘i learnt more from marking the work of others than from writing my own report’. if this were to be true then does it negate the need for students to create their own essays? this forms the basis of this study in making use of a computerized peer-assessment environment in order to * school of computing, university of glamorgan, pontypridd, mid-glamorgan cf37 1dl, uk. email: pdavies@glam.ac.uk 262 p. davies assess a students’ knowledge in a particular subject area, rather than making them write an essay in this area. the benefits of peer-assessment are well documented with respect to student and staff benefits (falchikov, 1995b; boud et al., 1999). part of the negativity that surrounds peer-assessment is the need for ‘proof’ that the end mark produced by the peers is equitable with respect to the mark that would have been provided by a tutor if they had performed the marking process (falchikov & goldfinch, 2000). numerous studies have reported back favorable results with regard to the equity of student and tutor marking (stefani, 1994). the majority of the research previously carried out in the area of peer-assessment has tended to concentrate on the marking aspect of the process. this generation of marks is only part of an assessment process with feedback being of utmost importance (collis et al., 2001). some previous studies have attempted to concentrate on the comments provided by the peers rather than merely just the marks (falchikov, 1995a). assessing whether the quantification of the quality of peer-comments can also be used as a method for assessing the quality of an essay is a relatively new area of research (davies, 2003a). in recent years computerized systems have been introduced in order to provide managed environments to support the use of peer-assessment systems (davies, 2000; bhalero & ward, 2001; bostock, 2001; lin et al., 2001; parsons, 2003) aimed at assessing the quality of essays and reports. systems are now also being created that are attempting to utilize the inherent benefits of peer-assessment in other areas such as computer programming (sitthiworachart & joy, 2003; lewis & davies, 2004). the higher-order skill of evaluation (bloom, 1956) is an area that would be expected to be assessed and used to identify the ‘better’ students. the method of peer-assessment detailed in this paper attempts to address this issue, and facilitate a method whereby these ‘better’ students will be able to demonstrate these skills and score higher. lin et al. (2001) noted that ‘high executive students contributed substantially better feedback than the low executive thinkers’. it will be interesting to ascertain whether this maps to their ability in the marking of the work. robinson (2002) suggests that as much as a third of the feedback provided by students was ‘inadequate’. therefore students producing such feedback will receive fewer marks than those students who show consistency within both their marking and commenting. rada and hu (2002) suggest that students should receive credit for doing good comments. this was also taken account of in the study reported here. essays are considered as a means of assessing the subjective skills of a student. however, lecturer marking can be highly subjective, ‘essays are demonstrably the form of assessment where the dangers of subjective marking are greatest’ (race, 1995). if the attribution of marks for peer-marking as presented in this study is to be of any value, then the student’s overall subjectivity should be removed (high and low marking and commenting), and their ability in showing consistency of evaluative skills should be what is rewarded. due to the experimental nature of this exercise, other assessments were included within the overall summative assessment process, namely objective tests (a series of mcq tests including confidence testing) (davies, 2002b) and also the creation and assessing student ability via peer-marking of an essay 263 submission of an essay of their choice (accompanied by a presentation) within the subject area of the module (networks and internet architectures). this ensured that the assessment of the learning outcomes of the module was fully covered. it was decided that in order to attain full student engagement within all aspects of the assessment process, there would be an equal contribution of marks from each of the assessment methods that contribute to the final summative grade. background one of the aims of this work was to replace essay writing with peer assessment. the students were directed to research a particular area of study within a module, and then used this to evaluate the work of a previous student. part of the study was to investigate how a mark for the demonstration of evaluative skills can be mapped to an actual standard marking scale? the study was undertaken during the autumn term 2003, within the school of computing at the university of glamorgan. the assessment was aimed at a cohort of 34 students studying on the post higher national diploma (hnd) course of study. this course is designed for students who have previously attained a hnd and are using this post hnd course as a bridging year to increase their credits to a level whereby they are able to enter into the final year of studies on the bachelor of science honours course (bsc hons). the students undertaking the module networks and internet architectures (nia) were from a broad range of previous named hnd awards varying from network administration, computer studies to information systems and business. at the start of the module students were presented with a series of questions on computer networking. from this, the area of n-tier architectures was highlighted as one which they had not covered. this area is traditionally covered within the final year module in distributed systems and enterprise networks (dsanden). in the academic year 2002–2003, an essay had been set to the final year group in this area of n-tier architectures. from this, 39 essays were available, each on average had been marked by their peers six times and a compensated peer-mark having been previously derived. rada et al note (rada et al., 2002) that ‘students (as well as teachers) may manifest bias, and a student may unfairly evaluate another student’s work’. by having a number of markings this bias is removed. also the compensation processes employed minimizes this aspect of the peer-generated grades. the nia students were provided with web references on the basics of n-tier architectures. they were also provided with the same assessment pro-forma that had been given to the dsanden students. to ensure that they perform the peer-marking process as closely to the dsanden students, they were able to use the cap menu driven marking tool (figure 1). the dsanden students had also made use of the anonymous communications facilities of the cap system (davies, 2003), however as these students had already graduated the previous year, this facility was not available to the nia students. figure 1. cap marking via pull-down menu 264 p. davies methodology the essays to be marked were from the last year’s final year. the compensated peergenerated marks for each of these essays had already been recorded. the marking of these essays had been undertaken making use of the pull-down menu driven cap system (figure 1). lin et al. (2001) note that ‘some students complain that holistic peer feedback was often too vague or useless’. by using this menu-driven system for marking, greater specificity can be generated within the marking process. also students make use of the menu system as a scaffold for their own commenting (davies, 2003a). assessment via this tool provides both the ability to mark and comment upon an essay. in this way two quantified values can be generated and compared, i.e. the peer-mark and the feedback-index. analyses of previous student commenting indicates that while doing the marking, students include free-text comments as well as using the menu driven marking system, therefore the feedback-index used included these comments as well as those generated by the menu system. use was made of the ‘mark-up’ tool (figure 2) in order to create feedback-indexes for each essay marking. figure 2. mark-up tool for quantifying comments the nia students as part of their requirements within this assessment were asked to mark six essays each. the average feedback-index was created for each essay previously marked by the dsanden students for comparative purposes (figure 3). figure 3. creation of feedback index for dsanden essays a compensation process was included to take into account of both high and low commenting/marking, as there is not necessarily a direct correlation between the mark produced and the comments. on one occasion a mark of 90% was allocated by figure 1. cap marking via pull-down menu assessing student ability via peer-marking of an essay 265 a student, who then went on to heavily criticize almost every aspect of the report (davies, 2000). in order to assess the validity of using these feedback indexes, a comparison between the average peer-marks and feedback indexes was performed for these figure 2. mark-up tool for quantifying comments figure 3. creation of feedback index for dsanden essays 266 p. davies essays. if there is validity within these ‘scores’ for an essay, then they are used as a measure against the peer-marking and commenting produced by each of the posthnd students. by producing average differences within the markings of a post hnd student and those of the essays, a measure of a student’s ability in evaluating the essays is produced (figure 4). these deviations in marking and commenting are used to produce a grade for the post hnd student both representing their marking and commenting ability. figure 4. quantification of differences in nia student markings to produce an average difference for the marking, the peer mark produced by the dsanden students has the mark given by the nia student taken away from it + or − x. the average of the differences can then be calculated, i.e. (x1) + (x2) … and then subsequently be divided by the total number of markings giving av. this figure represents the average differences from the perfect 0. however, in doing this no account is taken of the consistency of a student’s marking. therefore, the absolute differences are calculated from this average and divided by the total markings, i.e. σ(av – [difference for each marking]) / number of markings. this numerical value now represents the consistency of the marker. a similar process is performed in order to produce a numerical value to represent the consistency of the marker with regard to the differences of the feedback indexes (associated with comments). these differences are then mapped via some form of linear structure in order to generate a valid grade for a marker. how the allocation of this absolute grade will be produced from this consistency mark will be presented later in this paper. figure 4. quantification of differences in nia student markings assessing student ability via peer-marking of an essay 267 results and analysis to use the dsanden results as a ‘marker’ for the nia students marking, there must be a confidence that these initial results are valid. of the original essays 10% had been cross-marked and there was a maximum of 5% variation in any of the compensated peer-marks awarded which was considered acceptable. the qualitative assessment of the comments generated within the marking process was not previously generated. making use of the markup tool (figure 2), a feedback index was created for each essay (figure 3) as marked by the dsanden students. the correlation of these grades to the marks is shown in table 1. however, some students tend to overor under-mark. this is addressed by using the compensated peer-mark. table 2 shows the effect of using the compensated feedback indexes for the dsanden student markings. there is a significant positive correlation between the average feedback indexes and the average peer marks awarded for the essays (tables 1 and 2). this improves by performing the compensation process (positive correlation of over 0.9). more importantly the average standard deviations produced for each average reduces from 4.33 to 3.61. this shows that the range of marks within each category table 1. mapping of non-compensated dsanden marks to comments f/index −5 −4 −3 −2 −1 0 0 1 2 3 4 5 6 7 39 46 58 60 60 69 63 67 63 70 73 76 80 54 45 59 58 68 67 68 69 68 73 60 65 67 60 69 70 59 60 62 76 50 62 75 62 68 71 avg 39 50 0 51.5 57.6 60 64 63.7 64 66.7 71.3 70.5 74.5 80 st dev 5.66 9.19 4.28 5.57 3.14 3.56 3.21 3.04 3.54 2.12 table 2. mapping of compensated dsanden marks to comments f/in −5 −4 −3 −2 −1 0 0 1 2 3 4 5 6 7 8 9 39 54 58 50 60 60 68 63 63 71 76 73 80 46 58 59 62 69 67 67 69 68 73 60 45 65 60 68 71 68 76 59 60 67 62 70 69 62 75 avg 42.5 0 54 58.7 53.3 60 61.8 65.2 65 67.8 70.4 72 74 0 0 80 st d 4.95 1.15 6.95 2.36 3.96 2.94 3.59 2.79 5.66 1.73 268 p. davies of the feedback indexes is reduced. also noticeable (table 3) is the fact that the top essay mark has moved along the linear scale significantly with regard to the feedback received by performing the compensation process. this positive correlation maps well with the previous results (davies, 2003a). due to this correlation it is fair to assume that the feedback indexes and the peer-marks produced for the dsanden essays are true measures of the quality of the work produced, and can be used as controls for the assessment of the nia students. it was decided to permit the nia students to perform their marking and commenting of the essays making use of the cap marking system (with the same menu driven commentings) as had been used by the previous year’s dsanden students. the average mark for the essays produced and marked by the dsanden students was 63.52%, with a standard deviation of 8.69. the average mark produced for the same essays, as marked by the nia students was 58.75%, with a standard deviation of 12.71 (only 5 out of 34 markers on average over-marked). a positive correlation of 0.77 existed between the average compensated marks generated for the essays by the dsanden students, with the average marks produced by the nia students. looking at the feedback indexes generated by the nia students, on average their feedback was −1.37 (only 8 out of the 34 markers over-commented) compared with the feedback produced by the dsanden students. table 4 shows the results of the nia markings and comments. on examining the use of the menu-driven comments and the free-text comments, a number of students tended to make use of both facilities integrated together. it was therefore decided that the feedback index produced for each marking should also include these free text responses by again making use of the markup (figure 2) application. also included in table 4 are the gradings (0–5) for this and other assessments within the study. table 4 shows the average absolute differences (as discussed previously) produced by the students for both their marking and commenting. to allocate a final grade, three assessments were specified. therefore the differences produced via the peer-marking process by the nia students needed to be quantified. in trying to allocate marks in a linear manner, the following grading was decided upon (table 5). in previous years of this module the average mark produced for the students has been between 55–60%. therefore it was assumed reasonable, as there were no indicators that this year’s nia students were any different to previous years, that a similar average would be produced within this cohort. a range of 0-5 marks was decided upon, and linear scales were determined for each aspect of the assessment based on these previous year’s expectations. table 3. comparison of averages of non-compensated and compensated average feedback indexes f/index −5 −4 −3 −2 −1 0 0 1 2 3 4 5 6 7 8 9 avg t1 39 50 51.5 57.6 60 64 63.7 64 66.7 71.3 70.5 74.5 80 avg t2 42.5 54 58.7 53.3 60 61.8 65.2 65 67.8 70.4 72 74 80 assessing student ability via peer-marking of an essay 269 t ab le 4 . r es u lt s f o r n ia s tu d en ts i n a ll a ss es sm en ts ( m ap p ed t o l in ea r sc al e 0 –5 ) s tu d en t a ve ra ge co m p en sa te d d if fe re n ce in m ar ki n g g ra d e 0 –5 f o r m ar ki n g a ve ra ge d if fe re n ce (j u st u si n g m en u d ri ve n ) g ra d e 0 –5 fo r m en u co m m en ts a ve ra ge d if fe re n ce (u si n g fr ee te xt + m en u ) g ra d e 0 –5 fo r f re ete xt co m m en ts d if fe re n ce in fe ed b ac k m en u /f re e te xt c o m b in ed gr ad e fo r m ar ki n g an d fr ee -t ex t co m m en ti n g / 2 0 c o m b in ed gr ad e fo r m ar ki n g an d fr ee -t ex t co m m en ti n g % m c q ra n ge 0 –5 e ss ay p ro d u ce d 0 –5 t o ta l b as ed u p o n /5 eq u al r es u lt s b as ed o n m c q a n d es sa y a ve ra ge ti m e ta ke n to m ar k an es sa y (m in u te s) 1 4 .8 4 2 .5 4 4 2 .8 4 −0 .2 1 6 8 0 3 .4 3 .3 7 0 .9 6 6 .3 3 5 .5 2 1 8 .2 1 9 .5 1 1 0 .3 0 −0 .8 2 1 0 1 .6 3 .5 3 7 .4 5 1 .1 3 2 .7 3 8 .3 3 4 .5 3 5 .0 3 −0 .6 1 2 6 0 3 .6 2 .1 5 8 .4 5 7 .7 5 4 .5 4 6 .3 4 4 .3 3 5 .0 3 −0 .8 1 4 7 0 2 .6 2 .6 5 8 .1 5 2 .2 3 8 .3 5 5 .7 4 2 .1 4 4 .0 4 −1 .9 1 6 8 0 3 .4 2 .6 6 6 .9 6 0 .4 3 3 .2 6 4 .3 4 2 .6 4 3 .6 4 −1 .0 1 6 8 0 3 .7 3 7 1 .0 6 6 .5 3 0 .2 7 1 6 .0 1 4 .2 3 7 .3 2 −3 .1 6 3 0 2 .6 2 .1 4 1 .2 4 6 .8 5 5 .8 8 3 .3 5 1 .7 5 2 .7 4 −1 .0 1 8 9 0 1 .9 2 5 6 .3 3 9 .5 6 1 .0 9 4 .2 4 2 .5 4 6 .1 2 −3 .6 1 2 6 0 3 .4 2 .6 6 0 .4 6 0 .6 4 9 .6 1 0 1 1 .5 3 5 .1 3 5 .8 3 −0 .7 1 2 6 0 3 .3 2 .8 6 0 .5 6 0 .8 3 5 .2 1 1 1 0 .3 3 3 .5 4 4 .5 3 −1 .0 1 2 6 0 3 .3 3 .1 6 2 .8 6 4 .2 3 6 .3 1 2 1 1 .9 3 3 .2 4 2 .8 4 0 .4 1 4 7 0 2 .2 3 5 7 .9 5 1 .9 1 7 .0 1 3 1 6 .5 1 8 .3 1 9 .3 1 −1 .0 4 2 0 2 .6 2 .8 4 2 .3 5 3 .5 4 1 .8 1 4 8 .9 3 5 .7 3 6 .2 2 −0 .6 1 0 5 0 1 .5 2 .1 4 1 .2 3 6 .7 6 8 .6 1 5 1 9 .5 1 9 .1 1 8 .9 1 0 .2 4 2 0 3 .6 1 .8 4 2 .4 5 3 .6 4 7 .3 1 6 9 .7 3 4 .7 3 3 .9 4 0 .8 1 4 7 0 1 .3 2 .3 4 7 .0 3 5 .4 1 6 .0 1 7 1 1 .0 3 4 .6 3 7 .6 2 −3 .0 1 0 5 0 2 .6 2 .8 5 2 .0 5 3 .1 6 5 .2 1 8 1 2 .2 2 2 .4 4 5 .5 3 −3 .2 1 0 5 0 3 .4 3 .1 5 9 .6 6 4 .4 3 3 .7 1 9 9 .3 3 5 .7 3 8 .4 1 −2 .8 8 4 0 2 .1 2 .3 4 2 .2 4 3 .2 4 0 .3 2 0 1 5 .2 2 2 .2 4 2 .8 4 −0 .7 1 2 6 0 2 .8 2 .6 5 6 .5 5 4 .7 5 8 .0 2 1 1 5 .1 2 3 .3 4 5 .4 3 −2 .0 1 0 5 0 3 .4 2 .6 5 7 .2 6 0 .7 4 0 .2 2 2 9 .6 3 4 .1 3 5 .9 3 −1 .7 1 2 6 0 2 .3 2 .9 5 4 .8 5 2 .1 4 8 .5 2 3 2 .7 5 4 .8 3 7 .9 2 −3 .1 1 4 7 0 3 .4 2 .5 6 2 .5 5 8 .7 3 4 .5 2 4 7 .6 4 4 .7 3 3 .7 4 1 .0 1 6 8 0 3 .4 1 .9 6 1 .5 5 2 .3 3 4 .3 2 5 1 1 .0 3 3 .3 4 4 .6 3 −1 .3 1 2 6 0 3 .0 2 .6 5 7 .5 5 6 .2 3 3 .4 2 6 1 0 .3 3 3 .9 4 4 .7 3 −0 .8 1 2 6 0 1 .4 0 2 9 .2 1 3 .8 2 1 .2 270 p. davies t ab le 4 . c on ti n u ed s tu d en t a ve ra ge co m p en sa te d d if fe re n ce i n m ar ki n g g ra d e 0 –5 f o r m ar ki n g a ve ra ge d if fe re n ce (j u st u si n g m en u d ri ve n ) g ra d e 0 –5 fo r m en u co m m en ts a ve ra ge d if fe re n ce (u si n g fr ee te xt + m en u ) g ra d e 0 –5 fo r f re ete xt co m m en ts d if fe re n ce in fe ed b ac k m en u /f re e te xt c o m b in ed gr ad e fo r m ar ki n g an d fr ee -t ex t co m m en ti n g / 2 0 c o m b in ed gr ad e fo r m ar ki n g an d fr ee -t ex t co m m en ti n g % m c q ra n ge 0 –5 e ss ay p ro d u ce d 0 –5 t o ta l b as ed u p o n /5 eq u al r es u lt s b as ed o n m c q a n d es sa y a ve ra ge ti m e ta ke n to m ar k an es sa y (m in u te s) 2 7 8 .2 3 2 .1 4 2 .2 4 −0 .2 1 4 7 0 2 .6 3 .4 6 3 .3 6 0 .0 3 5 .7 2 8 1 0 .3 3 3 .0 4 3 .5 4 −0 .6 1 4 7 0 2 .6 2 .9 5 9 .6 5 4 .4 4 4 .0 2 9 9 .2 3 4 .2 3 5 .9 3 −1 .7 1 2 6 0 4 .0 3 .4 6 9 .4 7 4 .1 4 3 .0 3 0 1 7 .6 1 4 .3 3 6 .3 2 −2 .0 6 3 0 3 .2 3 .1 5 2 .4 6 3 .6 4 1 .3 3 1 6 .5 4 7 .2 2 6 .2 2 1 .0 1 2 6 0 1 .9 0 3 2 .7 1 9 .1 4 9 .5 3 2 4 .7 4 2 .2 4 2 .0 4 0 .2 1 6 8 0 1 .7 2 .1 5 2 .2 3 8 .4 4 2 .7 3 3 4 .0 4 1 .8 5 6 .2 2 −4 .3 1 2 6 0 3 .7 3 .1 6 5 .4 6 8 .0 2 8 .0 0 3 4 1 1 .3 3 7 .0 2 8 .8 1 −1 .7 5 8 4 0 3 .6 2 .9 5 6 .5 6 4 .7 8 1 .6 a vg 9 .9 3 4 .2 3 .3 5 .5 2 .8 −1 .2 1 1 .5 5 7 .6 2 .8 2 .5 5 4 .7 5 3 .2 4 2 .0 s t d ev 4 .6 0 .7 3 .2 1 .4 4 .3 2 .1 1 .0 5 .7 2 8 .3 0 .2 0 .3 1 0 .1 1 .1 3 2 .6 assessing student ability via peer-marking of an essay 271 a comparison of average feedback differences produced the results shown in table 6. the overall mark for the peer-marking aspect of the coursework was 2.88/5 with a standard deviation of 1.41 (using menu + free text as the feedback score). as a percentage this evaluates to 57.6% (within expectations from previous years). the mark for the essay and presentation was slightly lower than was expected (50%). however, during the presentations it was noted that the nia students’ ability in developing an essay of their own was in general quite poor. this skill may be one that is developed throughout the course of the post hnd course. in the past it was considered that the more time taken to mark an essay results in a more detailed and precise marking and commenting. the average time taken to mark the essays was 42 minutes (table 4). this varied considerably between students with a standard deviation of 32.6 for the times taken. the times taken to mark an individual essay ranged from 10 to 104 minutes. the student with the highest average time for marking was 81 minutes (number 34) yet he only received 1 for his quality of commenting, whereas the student with the lowest average time for marking 16 minutes (number 12), received 2.8 for commenting, with both receiving 3 for consistency of marking. from viewing the figures there was no significant correlation between the time taken and the grade awarded for the peer-marking processes. for a true comparison to be made between the various assessment methods, then they needed to be graded in a consistent manner (table 5). a composite final grade was produced on the basis of equal weightings of the three methods of assessment. correlation between essay/mcq combined grade and the final assessment grade including the peer-assessment was 0.80. this would suggest a good match of the table 5. mapping to common assessment scale 0–5 marks marking difference feedback difference mapping for mcq and essays 5 <4 <2 90–100% 4 <8 <4 80–89% 3 <12 <6 60–79% 2 <16 <8 40–59% 1 <20 <10 20–39% 0 20 or more 10 or more 0–19% table 6. average results for assessments (0–5) method of assessment average /5 std dev avg as % score for peer marking 3.0 0.71 score for feedback just menu driven 3.29 1.41 score for feedback with menu + free text 2.76 2.12 score for mcq 2.8 0.15 56% score for essay + presentation 2.52 0.27 50% 272 p. davies students within the grade awarded for the peer-marking process. table 7 shows the frequency distributions of the number of students within each grading. in ordering the students via their final grade awarded there is on average a consistency of performance against the three methods of assessment used (table 8). if based only on mcq and essay (50/50 split) and removing the peer-marking process, the final average would have been 53.20% (with a standard deviation of 1.13). this would have had an effect of increasing the final overall average by 1.48%. table 7. frequency distributions of results 0 1 2 3 4 5 marking differences 0 5 3 15 9 2 feedback on menu only 0 3 2 13 14 2 feedback on menu + free text 1 4 8 10 11 0 final mark (based on menu) 0 3 3 20 7 1 final mark (based on text + menu) 1 4 6 17 6 0 mcq tests 0 7 10 16 1 0 produce own essay and presentation 2 2 20 10 0 0 final based on 50/50 mcq and essay only 2 4 16 12 0 0 assessing student ability via peer-marking of an essay 273 based upon the final results produced by all three assessment processes, it is interesting to note which students performed best at the peer-marking process, i.e. displayed the greatest improvement by demonstrating evaluative skills. table 9 indicates that the claim at the outset of this study that the higher order students will be rewarded appears to be vindicated. the results are affected by the fact that one of the two students in the 30–39% group did not submit an essay. without this student it would have been −13.69%. also the student in the 20–29% category did not submit an essay. the results indicate on average a correlation between the results produces across the three methods of assessment and also the rewarding of higher order skills as demonstrated by the ‘brighter’ students. student feedback seventeen responses were received from the 34 students (50%) who undertook this assessment process. of the students who replied they all stated that it was the first time that any of them had used any form of peer-assessment. the students were asked whether they had prepared in a different way prior to the marking than if they were going to write an essay themselves. 14 out of the 17 stated they had prepared in exactly the same way as they would have normally. those who felt that they had prepared in a different manner generally made the point that they had performed table 9. improvement based on performing peer-marking process based on all three results effect on student’s final % by including peer-marking process 70% + +4.53% 60–69% +1.51% 50–59% +1.82% 40–49% −2.16% 30–39% −0.03% 20–29% +15.45 table 8. frequency distribution of nia student results (final grades) final overall % mark number of students average mcq average essay average peer-marking 70% + 2 3.52 3.13 4.0 60–69% 9 3.39 2.82 3.33 50–59% 14 2.78 2.67 3.00 40–49% 6 2.28 2.21 1.92 30–39% 2 1.75 1.75 1.75 20–29% 1 1.38 0 3.00 note: student 20–29% did not submit an essay. 274 p. davies much more research. they felt that a much better understanding of the subject area was required prior to the actual assessment process taking place. one student made note of the fact that their research continued throughout the marking process and they had often looked up areas that they didn’t understand whilst progressing through the marking process. this might result in a problem for this student being able to maintain a standard throughout their marking process. the students were asked how they felt about marking their peers. a number highlighted ‘an apprehension’ in undertaking this method of assessment initially, but this lessened throughout the course of the marking process, and a number commented ‘it really became enjoyable’. a number of the students expressed their concerns that they ‘didn’t know how many marks to give or take away’. this feeling of uncertainty was expressed by most of the respondents. since these post hnd students are deemed to be at level two of a degree scheme, it was felt that it would be interesting to assess their thoughts on the quality of work produced by a level three student. the common thread of comments related to ‘amazed at the wide range of work i had to mark’. the feedback reported back that there were few ‘average’ essays but they tended to be ‘either excellent or poor’. the students were questioned as to whether there were particular areas of the essays that they had found to be good or bad. the areas that were reported to be good were the use of examples, good grammar throughout, and where students had attempted to provide significant personal conclusions. the main areas that were reported as being poor concerned the referencing of the material. the markers commented on how difficult it had been in many essays ‘to find out where they’ve got their information from’. on being questioned on whether they the marker had a good understanding of the subject area, 15 out of the 17 respondents answered in the positive. comments such as ‘i’ve a much better understanding than if i’d written my own’ and ‘i now have the confidence to explain this subject to my classmates’ were very positive. the two students who did not feel they had a good understanding both said that their knowledge was still ‘very general’. on seeking improvements to the method used, the common comment from the students was that they required better initial guidance concerning such matters as the marking scheme, how to ‘judge a good from a bad piece of work’ and what to look for when marking. with regard to the cap system one student suggested ‘the pull down menu comments are inserted in alphabetical order’. two students suggested that a good essay be provided, with the marks given and also the comments. if this were the case then the subjectivity of the peer-marker may well be weakened. with regard to not doing an essay of their own, no student felt they’d been disadvantaged in any way. a number felt that they’d improved their knowledge significantly in the subject area because ‘really had to learn the stuff, not just write about it’. two students were pleased that ‘the hassle of writing an essay has been removed’. overall the feedback was very positive towards this method of learning through assessing. a general comment made concerned the students not knowing ‘how will i get my marks’. even though it was explained up front that they would be judged on their assessing student ability via peer-marking of an essay 275 consistency of feedback and marking, the visualisation of what was considered ‘good’ and ‘bad’ was difficult for the students to judge. a fact that has been noted in the past as a possible negative against peer-assessment is the problem of students not knowing or following the university rules associated with the identification of plagiarism. this is identified in the following ‘quite a lot of copy and pasting (plagiarism), but it was referenced’? conclusions this study aimed to assess whether students could be judged on their knowledge in a particular subject area by marking rather than creating an essay. the results indicate that this is possible and that student feedback is positive. from the analysis of the results it has been noticeable that not all students perform at the same level in the various assessment types undertaken (though the average shows good mapping). this indicates that it is important to assess via a variety of methods to ensure students are assessed fairly. removing the ability to include free-text responses within the peer-assessment process removes the need for employing the markup utility, and hence speeds up the provision of awarding a ‘mark for marking’. however, from student feedback and the tutor’s analysis of the feedback, there is an indication that the inclusion of free-text responses as well as the menu-driven system encourages students to include emphasis and further subjectivity within their comments. it has also been noted that the interpretation of a comment may differ from student to student. on the menu-driven system the comment ‘overall a fair report’ might be deemed to be a negative comment and this interpretation problem is currently being researched, students are now able to develop an augmented database to include their own comments as well as the standard comments. each comment is given a weighting by the student thus producing a greater degree of subjective based feedback. the results from this research will be reported upon in the future. one of the main advantages of this form of assessment is the elimination of plagiarism. the students are not able to copy material off the web (as they do not produce their own essays) and can not copy off their peers in the marking process as the essays are randomly selected. an interesting question arises is what procedures should be undertaken if a student in peer-marking identifies plagiarism within a student’s essay, when that student may have already been awarded their qualification. from the essays produced and presentations of post hnd students, the author noted that the students lacked the level of skills of expression and referencing that would be expected of a level three student. these skills may well be collected via further experience at a higher level of study and augmented through the duration of this post hnd year of study. if the nia students had been required to have written an essay themselves in this area of study they may well have received a lower grade. would this have been a fair reflection on their knowledge in the subject area? in producing the mappings in table 5, it is noted that the linear scale 0–5 is used to represent a student’s achievement in a particular item of assessment. this is based 276 p. davies upon expectations based upon previous student cohorts. it is questionable whether these mappings could be used across all modules, years, courses, etc. also from the author’s experience it has been noted that different ethnic backgrounds have varying expectations, e.g. a student from one culture may consider 80% poor, whilst another might considered this good. what this study has shown is the knowledge acquisition process need not be limited to the development of an essay, but can be enhanced via peer assessment. this study highlights the need for a compensated grade. this is illustrated by the example of the original marking and commenting for one dsanden student, who was awarded a mark for their own work of 71%, but who consistently under-marked by an average of 2% per essay. however, on looking at his commenting, noting the feedback index scales shown previously of −5 to +9, his average under-commenting was −5.16. this means that just looking at his comments would have a very negative response. if their essay was only to be judged on a numeric representation of the commenting then possibly a good essay would receive a very low mark. for these methods of peerassessment to be utilized then it is clear that the need for the computerisation of the processes is essential for management purposes. to summarize this study has produced positive results, both statistically and from student feedback. the results do support the fact that the ‘brighter’ students achieve more by utilizing their higher order skills. the method of assessment reported is capable of augmenting the assessment process, however the author would not make any claims concerning replacing the need for student essay production completely within a course of study. a final question that arises from this study is at what stage of the educational process or at what age group could this method of assessment be deemed acceptable? this is an area that will provide future work before any assumptions may be formulated. references bhalerao, a. & ward, a. (2001) towards electronically assisted peer assessment: a case study, alt-j, 9(1), 26–37. bloom, b. (1956) a taxonomy of educational objectives: handbook of cognitive domain (new york, mckay). bostock, s. (2001) peer assessment: principles, a case, and computer support, paper presented at the ltsn-ics workshop in computer assisted assessment, warwick university, 5–6 april. available online at: http://www.keele.ac.uk/depts/cd/stephen_bostock/docs/ltsnicsapril 2001.pdf. boud, d., cohen, r. & sampson, j. (1999) peer learning and assessment, assessment and evaluation in higher education, 24(4), 413–426. collis, b., de boer, w. & slotman, k. (2001) feedback for web-based assignments, journal of computer assisted learning, 17, 306–313. davies, p. (2000) computerized peer assessment, innovations in education and training international, 37(4), 346–355. davies, p. (2002a) using student reflective self-assessment for awarding degree classifications, innovations in education and teaching international, 39(4), 307–319. davies, p. (2002b) there’s no confidence in multiple choice testing, in: m. danson & c. eabry (eds) proceedings of the 6th annual caa conference, loughborough, july, 119–130. assessing student ability via peer-marking of an essay 277 davies, p. (2003) closing the communications loop on the computerized peer-assessment of essays, alt-j, 11(1), 41–54. davies, p (2003a) peer-assessment: no marks required just feedback?—evaluating the quality of computerized peer-feedback compared with computerized peer-marking, in: j. cook & d. mcconnell (eds) communities of practice, research proceedings of the 10th association for learning technology conference (alt-c 2003), university of sheffield and sheffield hallam university, 8–10 september. falchikov, n. (1995) improving feedback to and from students, in: p. knight (ed.) assessment for learning in higher education (london, kogan page). falchikov, n. (1995a) peer feedback marking: developing peer assessment, innovations in education and training international, 32(2), 175–187. falchikov, n. & goldfinch, j. (2000) student peer assessment in higher education: a meta-analysis comparing peer and teacher marks, review of educational research, 70(3), 287–322. lewis, s. f. & davies, p. (2004) the automated peer-assisted assessment of programming skills, in: p. chalk (ed.) proceedings of the 8th java and the internet in the computing curriculum conference jicc8, january. lin, s. s. j., liu, e. z. f. & yuan, s. m. (2001) web-based peer assessment: feedback for students with various thinking-styles, journal of computer assisted learning, 17, 430–432. parsons, r. (2003) self, peer and tutor assessment of text online: design, delivery and analysis, in: j. christie (ed.) proceedings of 7th international caa conference, july, 315–326. race, p. (1995) the art of assessing, the new academic, 4(3) and 5(1). rada, r. & hu, k. (2002) patterns in student–student commenting, ieee transactions on education, 45(3), 262–267. robinson, j. (2002) in search of fairness: an application of multi-reviewer anonymous peer review in a large class, journal of further and higher education, 26(2), 183–192. sitthiworachart, j. & joy, m. (2003) deepening computer programming skills by using web-based peer assessment, 4th annual ltsn-ics conference, galway. stefani, l. a. j. (1994) peer, self and tutor assessment: relative reliabilities, studies in higher education, 19(1), 69–75. doi:10.3402/rlt.v20i0.16148 linking theory to practice in learning technology research cathy gunn a * and caroline steel b a centre for academic development, university of auckland, auckland, new zealand; b school of languages and comparative cultural studies, faculty of arts, university of queensland, brisbane, queensland, australia (received 31 january 2011; final version received 5 october 2011) we present a case to reposition theory so that it plays a pivotal role in learning technology research and helps to build an ecology of learning. to support the case, we present a critique of current practice based on a review of articles published in two leading international journals from 2005 to 2010. our study reveals that theory features only incidentally or not at all in many cases. we propose theory development as a unifying theme for learning technology research study design and reporting. the use of learning design as a strategy to develop and test theories in practice is integral to our argument. we conclude by supporting other researchers who recommend educational design research as a theory focused methodology to move the field forward in productive and consistent ways. the challenge of changing common practice will be involved. however, the potential to raise the profile of learning technology research and improve educational outcomes justifies the effort required. keywords: research methodology; educational design research; educational theory; evaluation despite a reasonably large body of knowledge built up over recent decades, an association for learning technology (alt) sponsored report on technology in learning notes that research typically does not address the problem of building an ecology of learning or take important factors related to integration of learning innovations into account (alt 2010, 5). the authors of that report believe it is impossible to separate theory from evidence in this area, reflecting a narrowing gap between educational theory and practice. the implication is that a body of reliable evidence from research into practice provides an ideal basis for theory generation and testing. such evidence is also required to attend to theoretical questions about the nature of learning in context (collins, joseph, and bielaczyc 2004). the question that remains unanswered is, what forms of evidence are sufficiently detailed, robust and reliable to be acceptable for these purposes? reeves, mckenney and herrington (2010) present a plausible answer by proposing educational design research as one effective way to advance theoretical knowledge, and to provide sector-wide confidence in the outcomes of learning technology research. other advocates of this approach concur, for example: (page number not for citation purpose) *corresponding author. email: ca.gunn@auckland.ac.nz research in learning technology vol. 20, 2012 rlt 2012. # 2012 c. gunn and c. steel. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 16148 doi: 10.3402/rlt.v20i0.16148 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 historically, design in educational research has served as a way to implement theories for testing. the emerging design research paradigm treats design as a strategy for developing and refining theories. three types of theories can be developed: domain theories, design frameworks and design methodologies. i argue for design research . . . because [it] engages researchers in the direct improvement of education practice. (edelson 2002, p. 105). educational design research is supported by an existing knowledge base, in a methodological approach that blends theory with practice and context in a way that is sufficiently structured to present meaningful evidence to various stakeholders. the ‘‘continuous process’’ element has potential to address the ‘‘ecology of learning’’ challenge identified in a recent report published by alt (2010, p. 5). the proposal by reeves, mckenney and herrington (2010) is not new. it is a recent iteration of a call to action that has been repeated for more than a decade. special editions promoting educational design research (with variations on the name but not the methodology) include journal of the learning sciences 10(1 and 2) 2001 and 13(1) 2004; educational researcher 32(1) 2003; and journal of computing in higher education 16(2) 2005. books include educational design research (van den akker et al. 2006) and the handbook of design research methods in education (kelly, lesh, and baek 2008). sector-wide engagement with the methodology is progressing slowly however, for reasons as complex as the contexts that researchers aim to explain through its processes. reeves, mckenney and herrington (2010) point to weaknesses in programs designed to prepare the next generation of educational researchers and the preference of many journal editors for purely empirical studies. equally challenging is the funding body preference for ‘‘established’’ methodologies and the requirement for researchers to work under pressures of time to produce results that show impact within an immediate study context rather than contributing to the generation or refinement of theories. with these influences in mind, a study of articles published in two leading international journals between 2005 and 2010 allowed us to see if the issues had changed since earlier investigations were undertaken, and if similar or different influences were now presenting. a review of learning technology research articles: methodology we chose ajet (australasian journal of educational technology) and alt-j (since renamed research in learning technology) as the source of articles to review because these journals represent leading professional societies and practitioner communities that aim to promote best practice in the field of learning technologies in higher education (ascilite in australasia and alt in the uk). our decision to conduct a review was motivated by concern that theory in the field was not advancing at the rate that might be expected with the increased volume of research publications and reports being produced. we believed that insufficient attention was being paid to the role of theory in this area of research and that the critical link between theory in educational design and practice was too often ignored. we also suspected that questions posed and actions proposed as a result of earlier critiques of research (e.g., bannan-ritland 2003; barab and squire 2004; kelly 2003; mitchell 2000; reeves 1995) had not been widely addressed. we began with three simple criteria for selection, which we applied to a total of 318 articles published in the two journals from 2005 to 2010. this preliminary selection process allowed us to identify studies that focused on practice in a higher c. gunn and c. steel 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 education context and presented data on the impact of a technology enhanced learning design, i.e., (1) the research was conducted to evaluate some aspect of technology supported learning design or resources in use, or the use of technology to solve an educational problem; (2) the research involved collection and analysis of data; and (3) the research was conducted in a higher education context. this screening process returned 100 articles, which we analyzed on 12 dimensions, developed to reflect the elements of educational design research and qualitative methodology in general. we refined the criteria after a trial run with a sample of articles that also served to verify common interpretation of the criteria by the two researchers. each paper was assessed on the following dimensions: (1) what educational problem or issue does the design/innovation attempt to address? (2) is the purpose of the evaluation to test or improve design of learning resources, a learning design or a technology solution? (3) is theoretical grounding of the educational design concept described? (4) does the evaluation use an appropriate methodology, and is it rigorously applied? (5) was evidence collected systematically from different sources and using different methods throughout the implementation of the elearning initiative? if not, [how] does this affect the scope of the findings? (6) are the limitations of methods used, e.g., self-report or sole use of objective or subjective data, noted in the paper? (7) was the context of implementation acknowledged in the evaluation design? (8) did the evidence clearly show the impact of the initiative on student learning and teacher behaviours? (9) to what extent was the study longitudinal and what stage of development or implementation was in focus? (10) what were the outcomes of the study? (11) were the findings informative for the study and possibly for other [higher education] contexts? (12) is this case an exemplar of any kind and why or why not? while educational design research principles provided a guiding framework for analysis, we did not judge the overall quality of articles on their application. rather, we considered the variety of types of research questions the articles addressed and acknowledged where the chosen methodology was suitable to the circumstances. we did, however, consider the broader implications that choosing a particular methodology would have on study findings and on authors’ claims of impact. as well as context, integration and timeframes, two of our dimensions (i.e., items 3 and 11 in the list above) related directly to the role of theory. the first looked at whether theory was cited as a basis for learning resource or activity design, while the second considered the contribution that study findings made to production of general guidelines, theory development and testing or the general body of knowledge. while these two dimensions are the main focus in this paper, others � such as research in learning technology citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 consideration of context and integration � are reported because they relate directly to the case we present. pedagogical context (item 7 in the list) is an important aspect of integration that is often overlooked (barab and squire 2004; gunn 2000). pedagogical context is defined as ‘‘the relationship between a setting and how participants interpret that setting, including the meaning of practices’’ (moschkovich and brenner 2000, 463). contextual detail should, therefore, be included as an integral feature of descriptions of evolving and successful innovations in published accounts. this also impacts on the question of whether findings will generalize to other educational contexts. without knowledge of contextual influences and a strong theoretical connection, the question can prove difficult to answer. in order to address the role of theory in sufficient depth and detail, findings related to other dimensions will be reported in separate articles. results we found a wide variety of study designs and purposes in the sample of 100 articles. in some cases, technology was used to solve particular educational problems. in others, the affordances of new technology were being explored. the majority was case studies of research into practice, and while investigation of authentic educational settings is generally recommended, the ‘‘one off ’’ nature of many studies does limit its utility. snapshots vs longitudinal views figure 1 shows that more than half (n �53) the selected articles featured studies that were snapshots at a point in time. less than one quarter (n �23) reported research that authors described as part of an ongoing study, while just 24 were longitudinal according to the fairly generous definition of a study that involved more than two iterations of an innovative learning design. without some evidence that findings could be replicated in similar contexts and possibly others, or that they build on or will be used to inform future research, there is little scope to make any comment on underlying theory. figure 1. longitudinal study or snapshot in time? c. gunn and c. steel 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 study objectives results on the dimension of explicit theory related aims, shown in figure 2, were similarly patchy, i.e., . 21 explicitly sought to develop or extend theory, to explore its application to practice or to define design principles based on study findings; . 23 were exploring the affordances of technology as support for a particular pedagogical approach; . 17 were testing a particular, pedagogically driven approach; and . 6 aimed to refine a learning design to enhance student learning. the remaining 33, and the largest proportion of articles in each journal that shared a common purpose, reported studies designed to test the affordances of a technology and the impact this would have on learning. some made broad reference to theory and how it related to the perceived affordance of the particular technology. however, many articles either failed to provide an adequate description of the pedagogical context or appeared to be so context specific that it was virtually impossible to consider the work as a contribution to theory or a body of knowledge. while such studies may be useful in the immediate context and sometimes beyond, they do little to address the need for a body of reliable evidence and generally applicable theories of learning with technology. this leads to questions about editorial policies and how the aims and scope of journals relate to broader theory generation objectives, as well as the motives of researchers in various professional practice areas. inconsistent reporting format the variable levels of reporting we found in our sample did little to serve the broader goals of the learning technology research community. while many articles offered a full description of objectives, context, theoretical grounding of designs, evaluation methodology, data collection methods and findings, others included few contextual figure 2. aims of research in published articles. research in learning technology citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 details, sketchy descriptions and drew conclusions without presenting any actual data. from the 100 articles, 65 did include some level of description of the theoretical grounding of the learning design, however; 19 had incomplete descriptions, and the remaining 16 provided minimal detail or none at all. many authors made no serious attempt to reflect on or extend existing theories based on the study findings. this echoes what kanuka (2011) and gibbs (2010 cited in kanuka 2011) say in their critiques of the tendency for scholarship of teaching and learning research to make scant reference to theory or previous studies and to assume, with little or no evidence, that findings will generalize to other contexts. such articles add no value to our theoretical understanding of learning with technology, and incomplete descriptions do not reflect the principles of good practice in qualitative research design (figure 3). comments on methodology two key principles of qualitative research design are to gather data from different sources to support triangulation as a form of verification and to present all available evidence so readers can verify (or contest) interpretations and conclusions offered by authors. neither principle was well addressed in many of the articles in our sample. too many relied on one source of data and failed to acknowledge the limitations this would impose on the outcomes of the study. we found the practice of relying solely on attitude surveys far too common in situations where evidence of the impact on learning would be a real measure of success. a variable range of methodologies combined with incomplete descriptions resulted in a lack of clarity that limited the ability of readers to learn from the study or to consider broader implications of the findings. some authors cited references without explaining, even briefly, why a particular methodology was chosen and how it worked for the study. others provided enough detail for experienced readers to find similarities between the process applied in the study and existing methodologies. without sufficient detail on the methodology it became necessary, but not ideal, to analyze the subtle differences between essentially similar approaches (figure 4). figure 3. is theoretical grounding of design described? c. gunn and c. steel 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 the value of study findings on the question of whether findings were useful in the study context and more generally, in worst-case scenarios, the findings were of questionable value even in the immediate context. some articles presented insufficient data to support the conclusions drawn and were, therefore, considered to have produced no results of any real value for readers. some were so context specific that nothing general could be claimed or derived, while others revealed potential but noted that further study was required to develop this into meaningful findings. while some authors acknowledged this need, they did not always indicate that further research was planned. the majority of studies did show results that could potentially be considered useful. however, in many cases, the predictive value was limited either by poor research design or the limited scope of data collected. less than half the papers featured sound research design and sufficient evidence to support claims about general application of findings (figure 5). figure 4. research methodology applied to study design. figure 5. were findings useful in context and / or more generally? research in learning technology citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 while we do not wish to underrate the local value of research findings that apply only to a specific context or lack explicit theoretical linkages, we are concerned about the difficulties this presents to the learning technology research community as a whole. in a restrictive fiscal climate, this combination of issues does little to advance credibility in a field of research that already suffers from low esteem or to advance theoretical understanding of the nature of learning where technologies are involved. to address these limitations, we propose theory development as a link that must be forged to guide learning design and, as a key, unifying theme for learning technology research methodology and reporting. theory, methodology and educational research educational research methodologies have been a focus for debate for many years, and researchers such as cobb (2001) have proposed using learning design as a strategy to develop and test theories. in this context, theory can be defined as an organizing framework that brings an additional layer of understanding to concrete experience by implying relationship, consistency and a degree of predictability and testability. it provides a way to move between concrete and abstract explanations of the same phenomenon. for theory development in the field of learning technology to progress, the challenge of balancing the predictive and innovative aspects must be addressed. in a seminal article on this theory development challenge, brown (1992, p. 147) identified the need for new and complex methodologies to capture the systematic nature of learning and teaching. her article put educational design research into perspective by outlining the increasingly complex task of studying a field characterized by evolving and sophisticated beliefs about learning. unlike other approaches, educational design research provides direct and dynamic links between theory, research and practice. these links create opportunities to produce useful learning design frameworks and reliable evidence of impact on learning, thus helping researchers to meet the general criteria for what constitutes good theory. the weight of evidence evidence is clearly a critical factor in the development and refinement of theory. however, our analysis of articles showed that many researchers paid insufficient attention to it in reporting and in supporting claims made as a result of their inquiry. questions about form and weight of evidence in relation to claims are integral to theory development and testing. brown (1992) identified a fairly common problem, which we also encountered, in the ‘‘tendency to romanticize research, and base claims of success on a few engaging anecdotes or particularly exciting transcripts.’’ the challenge here is to find the means to convey what brown calls the ‘‘selective and seductive’’ aspects, as well as the representative, reliable and repeatable findings. mitchell (2000, p. 51) describes another angle on the same problem: much published research about education and the impact of technology is pseudoscientific; it draws unwarranted conclusions based on conceptual blunders, inadequate design, so called measuring instruments that do not measure, and/or use of inappropriate statistical tests. while research design has clearly evolved in recent years, some studies show it is still a work in progress (e.g., barab and kirshner 2001; fishman et al. 2004; reeves 1995). c. gunn and c. steel 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 opportunities and methods to collect rich contextual data have expanded to meet the growing complexity of requirements. what remains to be addressed is the critical mindset that promotes the selection of appropriate data collection and analysis tools for the task at hand and goes beyond the immediate and interesting to contribute to theory and design principles in learning technology. this requires methodologies that are generative and transformative, directed at understanding learning and teaching processes, and that aim to both explore and confirm, using methods selected to fit the purpose and circumstances. it also requires a broad perspective, to acknowledge the importance of pedagogical context and ecologies of learning. the case for longitudinal and theoretically grounded studies theory development is an organic process of exploration, discovery, confirmation (through cycles or iterations) and dissemination. the process needs to be theoretically grounded and support testing over time and in different contexts. according to our analysis, timeframe in research design remains problematic. many studies continue to take a ‘‘snapshot at a point in time’’ to suit researchers’ immediate aims and funding body or other external requirements. while these are valid parameters, they should not be the only ones. the short-term nature of many inquiries meant that few authors explicitly or adequately grounded their research in relevant theory and then attempted to reflect on or extend that theory based on their study findings. this seriously limits the potential to generalize findings to other contexts. to address the issue of adequate grounding in theory, we reiterate edelson’s (2002) advice to make design a useful part of theory development by keeping it research driven. edelson argues that in order for design research to yield productive findings, it must be informed by prior research and linked to both research findings and research perspectives. further, he implores researchers to draw on available theories (even if incomplete) and empirical results so that their work is ‘‘guided by an informed understanding of the gaps in current understanding in order to . . . make a useful contribution to understanding.’’ (p. 116). the case for educational design research the strengths of design studies lie in testing theories in the crucible of practice . . . in recognizing the limits of theory, and capturing the specifics of practice and the potential advantages from iteratively adapting and sharpening theory in its context. (shavelson et al. 2003, p. 25). as a practical way to address problems identified by other researchers and confirmed by our own study, we reiterate the proposal to adopt educational design research as a systematic and holistic methodology for research in learning technology. we believe this offers the greatest potential to reposition theory in the pivotal role it warrants. to serve the broad aims of learning technology research and theory development, kelly (2006) and bannan-ritland (2003) both situate educational design research studies within a larger scientific cycle of exploratory or developmental and validation stages. exploratory studies can produce well-designed innovations that are worthy of scaling up. in the final chapter of educational design research, nieveen, mckenney and van den akker (2006) note the importance of a developmental stage of learning research in learning technology citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 design that is guided by ‘‘deep factual and theoretical understanding.’’ edelson (2006) defines this level of understanding through context theory, which focuses on a design setting and might include, for example, an analysis of the needs of particular groups of learners, the nature of subject matter and learning in a discipline. as well as theoretically grounded designs, this stage can produce design principles for use in solving broader educational problems. for this, edelson (2006) uses the term outcomes theory, as a way to describe the effects of interactions between design setting and design elements. outcomes theory explains why a designer would choose certain elements for use in one context and different elements for another. it also supports a third kind of theory, which is the design framework and methodology that rests on context and outcomes theories. our study suggests that few learning designs are grounded in such deep consideration of context or theoretical factors. validation studies are used to test the impact of an innovation in use and to provide input for future exploratory and development work. these studies contribute most to advancing domain specific instructional theories, echoing the alt report’s (2010) call to focus on an ecology of learning and the claim that evidence cannot be separated from theory. thus, a body of reliable evidence from research into practice at the validation stage provides an ideal basis for this stage of theory generation. while the literature underlines the importance of both exploratory and validation studies, nieven, mckenney and van den akker (2006) note the trend, which we observed in our own research, of emphasizing the final stage of the educational design research cycle. thus, aims to test claims of causality through studies of learning designs in use are more common than those that focus on the grounding or rationale for designs. this trend is partly fuelled by traditions in educational research and the emergent nature of educational design research as a methodology. the parameters set by funding bodies and editorial standards define norms that both limit what researchers can do and, in the former case, set somewhat arbitrary timeframes they must work within. another powerful driver is the tendency to focus on producing results that are immediately and practically useful in the study context. while this is understandable from the perspective of individual researchers, these factors combine to limit progress on important aspects of theory development, particularly in the rapidly expanding and high stakes field of learning technology. educational design research offers a viable opportunity to remove these limiting parameters from the research environment. the pivotal role of theory in educational design research reeves, mckenney and herrington (2010) join a growing number of researchers (barab and squire 2004; reeves, herrington and oliver 2005; van den akker 2006; wang and hannafin 2005) in promoting educational design research as a ‘‘socially responsible’’ methodology. the opportunity that design offers . . . is the possibility of using the lessons learned in constructing design procedures, problem analyses and design solutions to develop useful theories (paraphrased from edelson 2006, p. 101) the question that remains is, what does adopting this approach, with its objectives of sustainable educational innovation and theory generation mean in practical terms? first and foremost, it means a shift in thinking around acceptable methodology for c. gunn and c. steel 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 published work, so that cases for theoretical grounding and systematic exploration become acceptable along with (well designed and thoroughly reported) validation studies. it requires longer timeframes for investigations and broader collaboration than is currently the norm. the collaborative nature of investigations involving learning designers, educational researchers, teachers, technologists and academic support staff is described in the study of elearning sustainability by gunn (2010). nieven, mckenney and van den akker (2006, p. 156) assert that distributed expertise and working relationships within interdisciplinary teams are critical factors to bring credibility to educational design research as a valid form of scientific enquiry and, thus, to broad acceptance of the theories it produces. the iterative cycles of design, development, implementation and evaluation that are involved require long-term commitment from all parties. this can be hard to maintain in the face of competing demands and priorities, although the potential to improve learning outcomes and advance theoretical understanding of the field should be sufficient incentive to overcome any barriers. strong, tested and connected evidence that theory informed, technologyenhanced designs can improve learning outcomes is required to further reduce the historical gap between educational research and practice. researchers have linked this gap directly to the field of educational technology for over 20 years (e.g., gunn, woodgate and o’grady 2005; hammond et al. 1992; rickard 1999; zemsky and massy 2004). during that time, the volume of learning technology research studies published in educational journals has expanded considerably. however, further steps are needed to turn the findings presented in published papers into reliable evidence that can inform design and provide plausible reports on the impact of technology on teaching and learning. while some learning technology researchers do gather solid evidence of such improvements, the challenge to produce and present evidence in a form that is sufficiently robust, standardized and acceptable to the wider educational community persists. furthermore, theory needs to be used in an explicit role to inform research studies and learning designs, and the outcomes of studies of these designs should extend or enhance our theoretical knowledge-in-action (practice). conclusions the literature reviewed and the study findings presented in this paper support our case to reposition theory to play a pivotal role in learning technology research. without theory, there is no solid grounding for learning designs and no generally applicable aspect to findings. yet, our analysis of articles published in two leading journals found the same situation as earlier studies of a similar nature; well-grounded designs and systematic evaluation approaches reported side by side with poorly conceived or poorly applied methodologies, limited reference to theory, weak results, incomplete descriptions, uneven presentation of data and overblown and unsupported claims of impact and importance. while this is an extreme statement in relation to most of the articles we reviewed, the incidence remain unacceptably high and is, therefore, detrimental to advancing the field of research in learning technology. we echo the call from other researchers who recommend educational design research, with its solid grounding and broad theory extending aims, as a suitable and holistic approach to the field of study. this is not a new approach, but one that has evolved over a number of years in response to changing conditions, beliefs and research in learning technology citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 observations about learning. in some respects, it has developed in a similar way to a theory, i.e., by drawing on that which is already known, accommodating changing beliefs and conditions and proving consistent and predictive as a basis for organizing investigations in a complex and challenging professional field. brown (1992) sums the matter up by acknowledging the importance of seeing current research goals and associated methodologies as part of the continuum they actually are. changes in understanding of learning require changes in the methodologies used to study it. kelly (2003) describes the discourses that come together in educational design research as those that support exploration and produce rich descriptions to illuminate arguments about process, with those that focus on confirmation and feature randomized trials of educational variables. the discourse that emerges from this combined approach is generative and transformative, directed at deep understanding of teaching and learning processes. its aim is to support arguments constructed around the results of active innovation and interventions in teaching and learning. both its basis and its outcomes are theories of learning with technology, hence our claim of a pivotal role for theory and educational design research as the reliable method for developing, testing and extending that theory. if the accumulation of knowledge takes time and requires a continuous process of reasoning, supported by the dynamic interplay of methods, theories and findings as bloch (2004) advises, then what stage of maturity is learning technology currently at as a field of enquiry? synthesis across studies is needed to discover, test and explain the diversity of findings. yet, our own recent study suggests that the sector has yet to establish a set of norms that will allow researchers to reflect on that interplay, and to develop broad understanding, reliable theories and realistic expectations through that reflection. we acknowledge also that the relationship between research and practice is complex. while sound research methods and reliable evidence can make a significant contribution, it is unrealistic to expect that these alone will create all the conditions necessary for teachers and policy makers to apply the findings of research in learning technology to practice. however, a useful step towards that goal would be to apply the ‘‘six guiding principles’’ that bloch (2004, p. 99) identifies as underlying all scientific enquiry, including educational research: . pose significant questions that can be investigated empirically; . link research to relevant theory; . use methods that permit direct investigation of the question; . provide a coherent and explicit chain of reasoning; . replicate and generalize across studies; and . disclose research to encourage professional scrutiny and critique. our study suggests that there is work still to be done on all these points, except perhaps the first one, if good theories and useable design frameworks are to result. a more even balance between exploratory and validation focused studies is also implied. references association of learning technology. (2010) technology in learning: a response to some questions from the department of business innovation and skills, association of learning technology, oxford. c. gunn and c. steel 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 bannan-ritland, b. (2003) ‘the role of design in research: the integrative learning design framework’, educational researcher, vol. 32, no. 1, pp. 21�24. barab, s. & kirshner, d. (2001) ‘rethinking methodology in the learning sciences’, the journal of the learning sciences, vol. 10, no. 1�2, pp. 5�15. barab, s. & squire, k. (2004) ‘design-based research: putting a stake in the ground’, the journal of the learning sciences, vol. 13, no. 1, pp. 1�14. bloch, m. (2004) ‘a discourse that disciplines, governs and regulates: the national research council’s report on scientific research in education’, qualitative inquiry, vol. 10, no. 1, pp. 96�110. brown, a. (1992) ‘design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings’, journal of the learning sciences, vol. 2, no. 2, pp. 141�178. cobb, p. (2001) ‘supporting the improvement of learning and teaching in social and institutional contexts’, in cognition and instruction: 25 years of progress, eds. s. carver and d. klahr, lawrence erlbaum associates, mahwah, nj, pp. 455�478. collins, a., joseph, d. & beilaczyc, k. (2004) ‘design research: theoretical and methodological issues’, the journal of the learning sciences, vol. 13, no. 1, pp. 15�42. edelson, d. c. (2002) ‘design research: what we learn when we engage in design’, the journal of the learning sciences, vol. 11, no. 1, pp. 105�121. edelson, d. c. (2006) ‘balancing innovation and risk: assessing design research proposals’, in educational design research, eds. j. van den akker, k. gravemeijer, s. mckenney and n. nieveen, routledge, london and new york, pp. 100�106. fishman, b., marx, r. w., blumenfeld, p. & krajcik, j. (2004) ‘creating a framework for research on systemic technology innovations’, the journal of the learning sciences, vol. 13, no. 1, pp. 43�76. gunn, c. (2000) ‘cal evaluation: future directions’, in the changing face of learning technology, eds. g. jacobs, g. conole and d squires, university of wales press, cardiff, pp. 59�67. gunn, c. (2010) ‘sustainability factors for elearning initiatives’, alt-j, vol. 18, no. 2, pp. 89� 103. gunn, c., woodgate, s. & o’grady, w. (2005) ‘repurposing learning objects: a sustainable alternative?’, alt-j, vol. 13, no. 3, pp. 189�200. hammond, n., et al. (1992) ‘blocks to the effective use of information technology in higher education’, computers and education, vol. 18, no. 1�3, pp. 155�162. kanuka, h. (2011) ‘keeping the scholarship in the scholarship of teaching and learning’, international journal for the scholarship of teaching and learning, vol. 5, no. 1. kelly, a. (2006) ‘quality criteria for design research: evidence and commitments’, in educational design research, eds. j. van den akker, k. gravemeijer, s. mckenney and n. nieveen, routledge, london and new york, pp. 107�118. kelly, m. (2003) ‘research as design: the role of design in educational research’, educational researcher, vol. 32, no. 1, pp. 3�4. kelly, m., lesh, r. & baek, j. (2008) handbook of design research methods in education, routledge, new york and london. mitchell, d. (2000) ‘the impact of educational technology: a radical reappraisal of research methods’, in the changing face of learning technology, eds. d. squires, g. conole and g. jacobs, university of wales press, cardiff, pp. 51�58. moschkovich, j. & brenner, m. (2000) ‘integrating a naturalistic paradigm into research on mathematics and science cognition and learning’, in handbook of research design in mathematics and science education, eds. a. kelly and r. lesh, lawrence erlbaum associates, mahwah, nj. nieveen, n., mckenney, s. & van den akker, j. (2006) ‘educational design research: the value of variety’, in educational design research, eds. j. van den akker, j. k. gravemeijer, s. mckenney and n. nieveen, routledge, london and new york, pp. 151�159. reeves, t. c. (1995) ‘questioning the questions of instructional technology research’, proceedings of the annual conference of the association for educational communications and technology, research and theory division, eds. m. simonson and m. anderson, anaheim, ca, pp. 459�470. research in learning technology citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 reeves, t. c., herrington, j. & oliver, r. (2005) ‘design research: a socially responsible approach to instructional technology research in higher education’, journal of computing in higher education, vol. 16, no. 2, pp. 97�116. reeves, t. c., mckenney, s. & herrington, j. (2010) ‘publishing and perishing: the critical importance of educational design research’, ascilite 2010, curriculum, technology and transformation for an unknown future, eds. c. steel, m. keppell, p. gerbic and s. housego, ascilite 2010, sydney, pp. 787�794. rickard, w. (1999) ‘technology, education, and the changing nature of resistance’, educom review, vol. 34, no. 1, pp. 42�45. shavelston, r. et al. (2003) ‘on the science of education design studies’, educational researcher, vol. 32, no. 1, pp. 25�28. van den akker, j., gravemeijer, k., mckenney, s. & nieveen, n. (2006) educational design research, routledge, london and new york. wang, f. & hannafin, m. (2005) ‘design-based research and technology � enhanced learning environments’, educational technology research and development, vol. 53, no. 4, pp. 5�23. zemsky, r. & massy, w. (2004) thwarted innovation: what happened to e-learning and why? final report of the weatherstation project, the learning alliance, university of pennsylvania. c. gunn and c. steel 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16148 doi: 10.3402/rlt.v20i0.16148 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16148 creating virtual communities of practice for learning technology in higher education: issues, challenges and experiences jacqueline a. dempster,* helen beetham,** peter jackson* and steven richardson*** *university of warwick, **university of bristol, ***university of manchester email: jay.dempster@warwick.ac.uk the need for a web portal to support the rapidly growing field of learning technology has been well established through a number of national surveys and scoping studies over recent years. the overarching vision has been the provision of a virtual environment to assist in informing and developing professional practice in the use of learning technologies. this paper outlines the issues and challenges in creating such a portal through the experiences of developing the results network. in the paper, design and participation issues are considered within the wider context of online and networked approaches to supporting practice and professional development. user participation methodologies and technical developments for results are described in relation to a review of existing representations of practice and a comprehensive survey amongst the learning technology users' community. an outline of key achievements and experiences is presented, followed by some conclusions regarding the cultural and political issues in creating a viable and sustainable facility and suggestions for possible future direction in national provision. introduction most higher education institutions have taken seriously the recommendations in the dearing report concerning greater use of technology for learning and teaching (national committee of inquiry into higher education (ncihe), 1997). each year, more teaching staff are making use of information and communication technology (ict) tools and environments with their students, or integrating digital resources into their courses. it is increasingly evident (armitage, rothery and jenkins, 1999; beetham, 2000; beetham, jones and gornall, 2001) that nearly all universities and colleges now provide some form of specific support for learning technology, indicated in the number of new posts and support units created for this area. the result is a rapidly growing group of multi-skilled 103 jacqueline a. dempster et al creating virtual communities of practice for learning technology in higher education professionals, employed both within institutions and/or nationally funded learning technology programmes and services, which have been such a key feature of the united kingdom experience. as well as 'learning technologists', other categories of staff, such as librarians and it developers, teaching support staff, educational developers and institutional managers, are developing new capabilities and understanding in learning technology (gornall, 1999; gibbs, 2000; beetham et al., 2001). many such staff also have responsibility for brokerage of information and networking (oliver, 2002). it is notable that in a growing number of institutions and in national programmes, a major theme appears to be to capture lessons learned on the use of ict and, in particular, to improve the dissemination and transfer of good practice. the abundance of new academic conferences and journals for learning technology is testimony to the high value placed on communication with others working in the field. however, learning technology roles and practices are diverse and poorly differentiated from one another and both the professional practice and the academic field of knowledge remain emergent (conole, 2003). supporting a learning technology community of practice a wide range of strategies is being used to support learning technology practice, including national support agencies, institutional initiatives and via professional development programmes (see review by lally and mcconnell, undated). there remains a feeling, however, that learning technology support staff, and the academics they work with, do not • have easy online access to high-quality information and resources which, thanks to the jisc and other initiatives, are now available. a number of national studies (notably talisman, 1999; armitage et al., 1999; gibbs, 2000) have identified significant gaps in provision of resources to assist staff involved with promoting and supporting the use of ict in teaching and learning. access to information and resources is felt to be patchy and finding materials is serendipitous and time-consuming. those working in the field of learning technology benefit enormously from collaboration and exchange of expertise with colleagues in similar roles in other institutions. it was therefore believed that the creation of a national portal to facilitate sharing and discussion of materials and methods used in learning technology practice would be beneficial. this requires interoperability across the relevant professional and subject networks, but this takes several forms technical, semantic, cultural, educational and political, and any portal developed would need to address this. while this paper focuses on users of learning technologies, most of the factors, debates and outcomes overlap with other disciplines and are equally applicable to any educational portal supporting diverse communities of practice. creating an online resource exchange and support network the results network project was devised by jay dempster and clive young under the jlsc-funded 5/99 programme to provide a learning technology portal for united kingdom higher education. the portal aimed to offer a focus for professional exchange and discussion and to fill the gap in information, resources and communication identified by 104 alt-] volume \ i number 3 the sector. results is the portal (www.resiilts.ac.iik) and the results network its wider community of users. providing a portal service that will support and help transform practice must be subject to an understanding and appreciation of the variety of identities and roles that practitioners take on, and the ways in which professional learning and teaching practice is changing. the research underpinning much of the theoretical and overview materials was commissioned early on and carried out by beetham. many of the key foundations of results stems from the findings of this user needs study (beetham, 2001a) which included a literature review, an online survey, structured interviews and focus groups. the research instruments, survey methodology and study findings are described in detail in the full report (beetham, 2001a). the study set out to identify the potential users of a learning technology portal, the nature of their practice, and the kinds of resource and service that can support them in developing their practice effectively. factors motivating participation were explored in depth. the research findings strongly indicated that the users wanted a community-orientated approach to contribution and annotation of resources. results was built on the vision of a rich information environment for user-defined resource submission, access and discovery coupled with facilities that encourage exchange and networking amongst communities of learning technology practice. the project overall was led by jay dempster, and essentially has meant the development of a database-driven web architecture, undertaken by steven richardson using specifications identified from the user participation studies and drawn up by peter jackson. the results portal enables users to link to, categorize (and interlink) resources on the web, directly integrated with discussion areas, as well as providing personalization features and user profiling. no such facility previously existed in any single and coherent sense. the portal aimed to tackle the multidisciplinary nature of learning technology. it enables and encourages cross-fertilization across diverse categories of users in the sector and supports a community approach to categorization and, as such, organic development of a consensusdriven taxonomy for learning technology. representations of practice for a virtual community the theoretical and methodological touchstones for the user needs study and in turn for the results network development arise from a number of existing educational frameworks for developing virtual communities of practice (beetham, 2002). lave and wenger (1991) are the authors most closely identified with the idea of 'community of practice', which they describe as 'a set of relationships among persons, activity and world, over time, and in relation with other tangential and overlapping communities of practice'. it is evident from experiences of knowledge modelling (stefik, 1995) that procedural and tacit knowledge of practice and expertise can be difficult to capture and are strongly biased towards subject knowledge and traditional modes of pedagogical practice in particular disciplines. the aster project, for example, has found that terms such as 'seminar' or 'tutorial' are open to interpretation and can relate to completely different sets of pedagogical practices in different subject areas (aster, 1999). 105 jacqueline a. dempster et al creating virtual communities of practice for learning technology in higher education in analysing online professional development networks, foster, bowskill, lally and mcconnell (2000) found that they have several prerequisites: facilities for asynchronous discussion, shared information resources, tools (for collaboration itself, and to support aspects of off-line practice), and investment of time and resource, both from the 'centre' and from participants. action learning (beaty and mcgill, 1995) is likely to be a factor in motivating participation in such networks, since all members will be novices in some aspects of learning technology work but experts in others, and all will benefit from peer support in dealing with the 'real problems' of their practice. the idea of organizing a professional network around specific shared activities brings in another series of possible models from the literature on computer-supported collaborative (or cooperative) work (cscw: for example mcconnell, 2000; kollock and smith, 2000; baecker, 1992) and the investigation of groupware systems. cscw is usually seen as involving two interlinked activities: peer-to-peer communication and the development of shared artefacts. the virtual learning space (http://www.vls.scotcit.ac.uk), a scottish universities service, has used frequent real-time chat sessions and access to subject experts as a successful drive for participation. another example is the virtual teacher centre (http:llvtc.ngfl.gov.uk), a professional practice community, which includes a teacher resource exchange (http:llcontribitte.ngfl.gov.uk) designed to help teachers develop and share ideas for activities in the classroom. finally, online professional development networks will have some features of an academic research community or subject portal as learning technologies become an object of academic study in their own right. examples of good practice in united kingdom higher education include sosig(http://www.sosig.ac.uk) and the learning and teaching support network (ltsn) engineering portal (http://www.ltsneng.ac.uk). sosig offers a one-stop gateway to scholarly resources for students and academics in the social sciences. sosig uses 'harvester' technology to automatically gather new resources from the web, based on urls from the existing sosig catalogue. the ltsn engineering portal exists to enhance learning and teaching practice and is based not around content resources but orientated far more towards engagement with their end users. user engagement the results user needs study (beetham, 2001a) suggested that on visiting the portal, academic teaching staff will have different needs from intermediaries such as educational developers, learning technologists, it developers, librarians, managers, project staff or researchers. one way to view the needs of potential users is to identify questions that the various types of user are likely to ask. three original areas of interest for developing results were addressed in the study. key findings are presented here for each area. developing artefacts for sharing practice knowledge artefacts that are most effective in helping to transfer practice knowledge from one context to another were explored with a view to identifying resource types and availability of these within the community. the user study distinguished resources that informed practice from those that were most likely to be adopted in practice. the gap between encountering an idea and making practical use of it was smallest in the case of interactive resources, such as frameworks, 106 alt-] volume 11 number 3 guidelines and tools (beetham, 2001b). while academic staff preferred materials with an exact fit to their own needs, intermediaries wanted adaptable resources. in theory, many of the stated requirements could be met by materials currently used in practice, re-purposed for local support needs, such as guidance notes and workshops. this was precisely the approach taken by a scotcit project, sesdl {http:llwww.sesdl.scotcit.ac.uk), in developing a web-based resource centre to encourage sharing and reuse of staff development materials. in this case, it might be argued that the most effective materials are those that are most readily adaptable and usable by other staff and educational developers, rather than those which are most directly accessible for academics (beetham, 2001b). in reality, the difficulties in terms of user participation are with regard to metadata description and the conflict of interest between user needs for rich descriptions and provider requirements, including expertise and the time to create informative metadata. developing forums for discussion forms of online dialogue that can support effective sharing of practice and the role and integration of local (offline) dialogue were explored. learning technology specialists are required to undertake continuous learning as their area of expertise moves on. normal practice could be described as 'peer supported experimentation'. collaborative online communities of practice and virtual play spaces are seen as valuable opportunities to engage in this. academic staff also needed dialogues, but favoured face-to-face relationships in their own context of practice. all those surveyed emphasized the value of examples and experiences from practice and opportunities for communication: discussion, reflection, networking and peer learning, from their desktop, and as needed. networking is regarded as a high priority by all staff, though given the choice they prioritize contacts, events and discussion forums over access to materials and resources. users emphasized the need for easy-to-use tools for engaging with the online community: finding, contributing and commenting on resources and for locating other people with relevant expertise. good examples include the sosig 'grapevine' facility for contacting other people with similar interests and the amazon intelligent search results with links to reviews. users valued push forms of information dispersal for current information providing it could be filtered for relevance. developing networks and communities of practice it was important to identify the potential users of, and contributors to, the portal and their requirements and motivations to participate. previous user needs analyses for the resource discovery network (rdn) portals have been able to focus on a specific subject community, and its need for disciplineand content-based resources in relation to a relatively well-established set of academic practices teaching and learning. learning technology users were likely to be found from both this set and from those in a multitude of roles across discipline and institutional communities, engaged in poorly defined, yet often innovative, practices, and requiring support that is practicerather than content-focused. there were debates over whether a potential results community was really a fairly small inner core of learning technology specialists, or the large periphery of casual and potential learning technology users, including academic staff and educational researchers. we explored how practitioners collaborate with one another offline and how active online participations can be encouraged and supported so that it is integrated and self-sustaining 107 jacqueline a. dempster et al creating virtual communities of practice for learning technology in higher education within an overall professional development activity. the study revealed an apparent conflict between the problem-solving culture of much learning and teaching practice the 'instant gratification' of a tip or trick discovered from the academic desktop, or the more considered solutions arrived at by collective brainstorming and the long-term habits of reflective practice and professional scholarship. a portal that focused only on this problem-solving culture would be doing little to raise the status and scholarship of teaching. those surveyed felt that a national service to support communities of practice in learning technology might best focus on direct, personal participation by its core community, but work towards indirect participation (for example, via ltsn subject portals or institutional vles) by a much broader community of users. it is likely that many users will continue to seek peer exchange, services and benefits through offline networks, and many people are likely to turn to online support only when avenues of face-to-face help have been unproductive. the survey identified a desire for national updates via a briefing service, calendar of events, news service and integration of face-to-face workshops. a directory of expertise and various types of discussion areas were also specified. development of the results network bringing together the above elements was a key aim of results. three main structural lists were identified for use in results: user types, resource types and interaction types (beetham, 2001a, 2001b; but see also goodyear and steeples, 1998, and oliver and conole, 2001). functionally, professional development networks might usefully be grouped into two broad areas: facilities for interaction with artefacts; facilities for interaction with individuals/groups; with a third hybrid area of facilities for developing or enriching artefacts (mulholland, domingue, zdrahal and hatala, 2000), which is in itself a form of indirect communication with others. for results, it was felt that these different facilities needed to be seamlessly integrated to support the full representations of practice identified. they also needed to be capable of evolution as users discover new ways of interacting with one another and with their shared resources. the results environment comprises three areas: 'myresults', 'resources' and 'communities'. visually, these appear separate on the site (figure 1) but are functionally integrated through associations between resources, users and practice. however, the distinction serves as a reminder that results is not concerned principally with resources but with resources that relate to practice. resource metadata is kept to a minimum taking into account the conflict between the needs of contributors and users identified in the study area concerned with developing artefacts for sharing practice knowledge. the focus of description is on their use in practice. it is intended that resources are constantly commented upon, updated, contextualized, organically developed in fact, resources should simply be contingent traces of the community process, rather than ends in themselves. the results portal is based on a lamp (linux, apache, mysql, php) architecture and demonstrates a wide range of user-orientated, practice-centred functionality and interoperable web technologies (richardson and powell, 2003; richardson, childs and dempster, in preparation; for more information, see the cetis website) and an integrated discussion board (phomm) on which a narrative of results design and development can be found.) 108 alt-] volume 11 number 3 ' 5 result home page • microsoft internet ehplorcr piavidcd by the uriivcnsil? df warwick fie i : i : vfe™ hefc ::• : ,-!.< » s-.,i i • 2 in nl s i-j >• • ^ h i in s l t i l i l hi-l w*lco in « jay d*m p »t« r i:l. • flil ui ii-lnli i n " l_i m ,ri,. m .l .c-i yf i i > . i l las \1vf £e1 , ili "f*1 < pi ulti: ^ . i l t ik. i l l i . i ••. . • _ , . 3 resources: .whc li r-t ".-•'=• . b l c t u m r i..k fjt-nlt tllji i-*ii r.n * :••-• o • i i i t '1 " '•'j d l 1 ! ' jk4, conimuniti<is; • i f f p i r j b :<tt t. lrn->'.-'j c . p y r i j l i ci200q pesultv s«l\i-w i tei in-nw* figure i: the results welcome page (arrived at after registration or login) user features the visual environment resources and the category tree the category tree (figure 2) forms only an initial taxonomy for learning technology (browse by topic), comprised of a 'learning and teaching' classification suggested by the ilthe/ltsn (sally brown and brenda smith) and a 'learning technology' classification suggested by beetham (2001a). a guided tour (figure 1) provides a section of introductory materials, case studies and frameworks: written by expert practitioners for new practitioners, commissioned by national agencies or developed by research projects. results demonstrates the integration of a number of key areas of need identified in the research study (beetham, 2001a). figure 3 shows the relationship between user 109 jacqueline a. dempster etol creating virtual communities of practice for learning technology in higher education 511 wnep'-aqcuicioscft imtcmh ucplorerpiovided by t t e uniw'ersitp ol w-sr j fie ed: vt~ ' f;«ios "331 via learning and teaching technology strategies viii am linns cdtvuui we; fivrtvr; m o r i if 1 m-tlt h l y r r f r t . r r f r v*re*i:ni j j h r v * j mirny l c < u i i i>:i ris-hi: hi *iti n :• -rii if," il ii nva.iwii t.l vra i :=•«• riictlnv ft pr^n-ft-—i r i frrvunviff itts^rtft^vj 1" rt-1.-*!^ ?«•-(* date: . pijttmtf ,,m»i.,i.l ?nt> im ins ancle idtm>?-i^ nj !•:•» i i:ii-, i.n 1 i n !'•• i f iii.i j : :i iti 1: v mris attc*ii\r n1-irfltinj t^rr"i*;rfi;rnjfrt-f ,tfi''j''.t| 11 bjj.<.ji:: -vj.<a figure 2: browsing the results category tree. top left location in the category tree; middle left resource copied from elsewhere; top right add new resources or categories; bottom left resource-associated functions. requirements and design features in terms of mapping user engagement and participation activities to the online functions and facilities in results. linking resources to people and communities of practice results enables meaningful associations to be made by the users between resources as they relate to practice and to their own interests through facilities to: • copy/cut/paste resources around the category tree; • link resources (including that linked resources appear immediately after linking); • shortlist resources to a personal collection in the user's my results area; and • start discussions from resources (function available from the details of the resource record). mo presentation collection interaction production communication requirements request resources create categories, move resources, collate resources use resources or interact with others author and develop new resources submit resources and discuss practice activities browse, search shortlist analyse, evaluate, discuss synthesise editorials disseminate, share, review, debate alt-] functions browse by topic, list by subject, type, target audience add new category, shortlist comments metadata field; copy/cut/paste edit; link, profiles directory add new resource commissioned edit profile, start discussion; (rate/review/ forward) volume 11 number 3 facilities category tree mycontributions (mypreferences) myshortlist clipboard myprofile (myactivitylog) mycontributions guided tour myprofile discussion forums tips, issues, innovations special interest groups figure 3: passive to active engagement within results (functions/facilities shown in brackets were those that could not be implemented within the project timescale, but which the system can support straightforwardly) personal development profiling is an emergent feature. from the myresidts area, users develop a personal profile of expertise (myprofile: initially created on registration) and can list their own contributions (mycontributions). the system can also support production of a record of activities on results (myactivitylog). under communities (figure 1), a 'directory of expertise' is available in the collective profiles directory providing a source of contacts for finding expertise or common areas of interest. users can also share approaches in practice: by contributing to tips and tricks, issues and problems, inspirations and innovations. they can start or contribute to discussion forums: associated with a resource or independently. special interest groups are available for sharing resources amongst specific collaborative groupings. technical developments behind the scenes metadata submission and forms handling although user demands for searchable and well-contextualized resources suggest the use of standard templates for submission of metadata, this is not a straightforward task. first, jacqueline a dempster et al creating virtual communities of practice for learning technology in higher education the learning object metadata standard does not extend to describing resources for which the primary intended user is not the learner. second, potential end-users had concerns about the time involved in submitting metadata. results developed a form handling application and a lightweight approach to resource description to cope with the demands of submission process and user interaction. exporting taxonomy using ims metadata and the oa1-pmh results operates as a fully functional oai-pmh (open archives initiative protocol for metadata harvesting) repository which means that any compliant oai harvester can retrieve metadata records in xml format from the results database. these records are available as minimal dublin core or as extensive ims records. instructions on how to use the protocol may be found on the open archives website (http://www.openarchives.org). the categorization route(s) of a resource may be extracted as taxonpath elements (ims metadata 9.2). this provides a mechanism by which resources can be programmatically imported from one database into another (via oai-pmh) and automatically placed in the right location in the category or directory structure of the importing database system creating new categories as required should taxons (categories) from a taxonpath not exist. in this way, we can merge taxonomies very easily, starting with an empty or existing structure (richardson et al, in preparation). bearing in mind that taxonomies (even established ones) are continually evolving, this is crucial. rss feeds the ten most recently submitted results resources are available over an rss feed (http:llwww.resiilts.ac.ukltoplo_rss.php). this is demonstratedon the ltsn materials education website (http://www.materials.ac.iiklfeatureslresiiltsproject.asp). an innovative use of rss provides basic remote search capabilities to results (http:llwww. materials.ac.uk/technical/results/search.asp). a description of the rss service is available on the results development website (http://www.residts.ac.uk/aar/presentations.php). z39.50 results also runs a z39.50 server connected directly to the relational database which can return dublin core xml records. this server provides boolean/multi-field search functionality for results and given more time and resources it could be opened to the world in general as a full z39.50 service. at present access to the server directly is blocked by a firewall. key achievements and experiences in addressing the needs of users identified within the learning technology communities of practice, there have been two major successes in creating the results network. the first is in the selection and development of the technologies that drive it. the second is in the proof of concept. however, results has been less successful in securing a critical mass of user participation. this is most likely to do with the need for integration and continuity within existing services offering regular face-to-face activities and off-line networks. this is discussed here as a third cultural and political issue for user participation. technologies in terms of the technologies, significant success rested on development work in new and untested territories. the technologies selected as fit-for-purpose and implemented within 112 alt-] volume 11 number 3 the results architecture are novel (at least to higher education and the education sectors). it was necessary to resolve technical as well conceptual problems in clean and uncluttered ways and, given the relatively small nature of the results project, to do so with a single web developer. results uses open source technologies and is not based on any proprietary software or code. it means that we have been able to draw upon and contribute to a large and developing body of knowledge and expertise. while database-driven websites and portals are far from new, we believe that results has been constructed and managed, from a technical point of view, in quite unique ways. all resources are submitted and managed online through the actual portal itself, and not through some intermediary content management system or human moderator. we know of no other examples of such an approach in portal systems and it works exceptionally well. interoperability has also been a high priority. results has gone a very long way and now represents a full complement of protocols: oai-pmh (protocol for metadata harvesting), rss feeds and z39.50 searching (richardson and powell, 2003). resource records are all dublin core and ims conformant and can be exchanged using the corresponding xml schemas. we believe this represents the forefront of thinking and development in this area in the education sector. finally results is fast. code has been specifically developed and rebuilt and graphical images kept to a minimum to enable the results portal to work at lightning speeds. it loads quickly, even on a machine with a standard modem connection; navigation around the portal and submissions to the portal are to all intents and purposes instantaneous. proof of concept the second major success of results has been in the 'proof of concept'. the concept focused on how a dynamic, database-driven portal one that has a clear target audience and has undertaken specific user needs analysis can work. we have successfully linked resources and resource submission to people, working on the key metaphor of 'communities of practice'. results offers a portal that is largely owned and designed by its users. it is community driven and has the significant potential to reflect, support and integrate a complex group of user communities. one implementation approach that was perhaps rather brave is the unmoderated resource submission. we trust our users in their professional conduct as it is open to misuse, even abuse. to our knowledge no other educational portal does this. all other services are moderated in some way, by some form of editorial or other quality control process. indeed, the approach supports the concept of reflecting on one's own professional practices, as it requires users to exercise a high degree of reflection and judgement on the purpose and value of the resource to practice. importantly, it also avoids resources being dumped into the database just to populate it with all and sundry. the results environment supports learning technology communities of practice by offering an organic means for development of a taxonomy supporting the diverse subcommunities of learning technology practice to become transparent with consensus reached over a period of time. a substantial step in terms of interoperability of this concept was taken by making resources, their 'taxonpath', and the tree 'transportable', allowing embedding within other compatible databases. 113 jacqueline a dempster et al creating virtual communities of practice for learning technology in higher education user participation given the short time scale of the project, the usage of the results portal site is currently modest and a critical mass is far from being reached. the reasons for the slow progression of uptake are complex and overlapping but include the innovation of user-driven and technical approaches and the pervasive 'not invented here' culture of higher education. this could have been overcome had a national agency driven and marketed it. it seems unlikely that the permanent revolution in individual and collective practice which is demanded by educational technology will ever be sustained without the support of local, face-to-face dialogues such as those offered by the more permanent national services and support bodies. despite early measures to ensure that such partnerships were forged, particularly with the ltsn and alt who were partners in the project from the outset, various political developments delayed any definitive commitment to host and further develop or drive the results network. a scoping study undertaken by results (jackson, 2002) with additional funding from the jisc focused upon identifying key issues and requirements in the development of a learning and teaching portal as a joint venture between the ltsn generic centre and the jisc. the study draws heavily on the conceptual framework for results, the user participation findings, technical developments and project experiences and in so doing is a major legacy for results in shaping this and other future portal developments. through collaboration with angel (authenticated networked guided environment for learning), a jisc 5/99 dner project at the lse concerned with creating middleware services {http:ll angel www.angel.ac.uk), results is currently hosted by lse. conclusions and future directions there is still much vagueness concerning the nature of portals. it is not yet understood how people make use of portals, or whether personalization is useful. furthermore, there needs to be a greater clarity in the relationship between national and institutional portals or between information-based portals and learning-based portals (edner, 2002). what is clear from both the user participation survey and our own experiences, at least for professional development networks, is the need to build links between virtual 'portal' activities and face-to-face activities such as workshops and conferences. the portal must endeavour to become a part of the wider communities and networks which means the portal must become the online part of the communities and networks that users already belong to. the notion of roving facilitators to bridge online and offline communities of practice is one possible mechanism for managing continual engagement of practitioners. directed and scaffolded support is likely to motivate participation in both online and offline professional development, such as that provided through the structured and formal framework of activities and opportunities within cpd accredited programmes. (indeed, seda, alt and the ilthe are exploring routes specifically for learning technologists.) as the portal is seen to offer facilities that are valuable, timely, engaging, fruitful and inspiring, word will consequently pass around those networks, and users may then start to turn to the portal as a first priority. there is a strong indication that results will be sustainable in the long term only as part of more general learning and teaching portals and only providing that it is cross-searchable from the subject portals which are becoming the recognized repositories of practice-based 114 aw-j volume 11 number 3 information for united kingdom higher education. integration of learning technology communities of practice would thus incorporate users who may utilize more specific or local entry points. there is a danger that a focus on access to information and resources per se will allow portals to become perceived as administrative repositories simply affording access to the 'knowledge' that the user needs, rather than offering tools for educational annotation and thus supporting shared professional scholarship and growth. on balance, results has been successful in developing a portal system that offers a valuable range of professional exchange functionality and interoperability. given the collective accomplishments within learning technology communities in united kingdom higher education and further education, the only constraining factors remain with the political dimension of joined-up thinking in the aforementioned portal developments. effective collaboration between national agencies is a key requirement, coupled with the need for sustainable development in project-initiated services: a matter for the funding councils. acknowledgements results has been a worthwhile project and we are grateful to the jisc for its initial funding and on-going support and enthusiasm for the work. the authors wish to thank the wide range of learning technologists, educational and staff developers, academic staff and many other researchers and practitioners for their involvement and commitment to exploring the conceptual and technical parameters on which portal development relies. the journey towards providing a one-for-all, lively and effective online community of practice for learning technology has really only just begun. references armitage, s., rothery, j. and jenkins, m. (1999), 'report of a ucisa survey of support provided in universities and colleges for the use of technology in teaching and learning', universities and colleges information systems association: teaching, learning and information group. also available on the web: http://www.ucisa.ac.uk/tlig/teach/docs/ survey99/ aster project (1999), 'a survey of current practice in the use of c and it to support small-group teaching activities in disciplines associated with the humanities, physics and psychology', accessed on the web 5 march 2003, http://cti-psy.york.ac.uk/aster/resources/ publications/report_2/report2.pdf baecker, r. (1992), readings in computer-supported co-operative work, morgan kaufmann publishers, inc. beaty, e. and mcgill, i. (1995), action learning: a guide for professional, management and educational development, london: kogan page. beetham, h. (2000), 'learning technology scoping study: lessons for educational developers', educational developments, 1, 4. beetham, h. (2001a), 'user needs study for the results project', accessed on the web 28 february 2003, http://www.results.ac.uk/aar/documents/user_needs.doc 115 jacqueline a dempster et al creating virtual communities of practice for learning technology in higher education beetham, h. (2001b), 'how do representations of our practice enable change to happen?', educational developments, 2, 4, 19-22, accessed on the web 7 march 2003: http:llwww.eres. ac.uk/source/docs/pub-ou-47.pdf beetham, h. (2002), 'developing learning technology networks through shared representations of practice', in c. rust (ed.), improving student learning through learning technologies, oxford: centre for staff and learning development, accessed on the web 7 march 2003, http://www.eres.ac.uk/source/docs/pub-ou-53.pdf beetham, h., jones, s. and gornall, l. (2001), career development of learning technology staff: scoping study final report, accessed on the web 3 march 2003: http://sh.plym.ac.uk/eds/effects/jcalt-project/final_report_v8.doc cetis, centre for educational technology interoperability standards, http://www.cetis.ac.uk/ conole, g. c. (2003), 'the evolving landscape of learning technology', alt-j, 10 (3), 4-18. edner (formative evaluation of the distributed national electronic resource) project (2002), portals in higher and further education, issues paper 5, accessed on the web: 27 march 2003, http://www.cerlim.ac.uk/edner/ip/ip05.rtf foster, j., bowskill, n., lally, v. and mcconnell, d. (2000), 'researching community: developing a design for a virtual professional development center in computer based collaborative group work', paper presented to international consortium for educational development in higher education, 22-6 july, bielefeld, germany. gibbs, g. (2000), 'learning and teaching strategies: the implications for educational development', educational development, 1.1 (1), 3-5, http://www.seda.ac.uk/ed_devs/voli /eddevs11.htm goodyear, p. and steeples, c. (1998), 'creating shareable representations of practice', alt-j, 6 (3), 16-23. gornall, l. (1999) 'new professionals: change and occupational roles in higher education', in perspectives: policy and practice in higher education, 3 (2), 44-9. jackson, p. (2002), 'scoping study for a learning and teaching portal for he', unpublished report to the jisc/ltsn. kollock, p. and smith, m. (2000), 'communities in cyberspace', in m. smith and p. kollocks (eds), communities in cyberspace, london: routledge. lally, v. and mcconnell, d. (undated), 'networked professional development: a review of current practice', institute of education research topic papers, university of sheffield, accessed on the web 3 october 2003 at, http://www.shef.ac.uk/education/research/ rtplandmc.shtml lave, j. and wenger, e. (1991), situated learning: legitimate, peripheral participation, cambridge: cambridge university press. mcconnell, d. (2000), implementing computer supported cooperative learning (2nd edn), london: kogan page. mulholland, p., domingue, j., zdrahal, z. and hatala, m. (2000), 'supporting 116 alt-j volume 11 number 3 organisational learning: an overview of the enrich approach', journal of information services and use, 20, 1, 9-23, accessed on the web 7 march 2003 at, http://kmi.open. ac.uk/projects/enrich/isu-enrich.pdf national committee of inquiry into higher education (ncihe) (1997), higher education in the learning society, bristol, higher education funding council for england, accessed on the web 7 march 2003 at, http://www.leeds.ac.uk/educol/ncihe/ oliver, m. (2002), 'what do learning technologists do?', innovations in education and teaching international, 39, 4, 1-8. oliver, m. and conole, g. c. (2001), 'supporting structured change: toolkits for design and evaluation', chapter 6 in r. macdonald (ed.), academic and educational development: research, evaluation and changing practice in higher education, seda research series, london: kogan page, 62-75. richardson, s., childs, m. and dempster, j. a. (in preparation), 'developing interoperable taxonomies for sharing resources within multidisciplinary communities of practice', submitted october 2003 to international journal of interactive technology and smart education (new journal: http://www.troubador.co.uk/itsel) richardson, s. and powell, a. (2003), 'exposing information resources for e-leaming harvesting and searching ims metadata using the oai protocol for metadata harvesting and z39.50 ariadne 34', accessed on the web 28 february 2003 at www.ariadne.ac.uk/ issue34/powell/ stefik, m. (1995), introduction to knowledge systems, morgan kaufmann publishers. talisman (1999), 'review of staff development courses and materials for c and it in teaching, learning and assessment', http://www.talisman.hw.ac.uk/ 117 understanding the development of teaching and learning resources: a review louise plewes and kim issroff university college london (ucl) email: k.issroff@ucl.ac.uk, louise_plewes@hotmail.com this paper is a literature review of research concerned with the production of learning resources in higher education (he). it forms part of a larger research project in progress. we identify and discuss six models which may be used to understand the development and reuse of learning resources. we focus specifically on the level at which the models operate, their purpose, structure and content and critically analyse the advantages and disadvantages associated with their application. aspects which the models do not address, for instance actual staff practices and the cultural context of development, are then discussed. we focus on understanding academic staff practices and behaviour in learning resources production. staff time is a major cost element in the production of learning resources. despite recent international work on costing methodologies for he and their application to networked learning resources (for example, bacsich and ash, 2000; detya, ernst and young, 2000; flashlight, 2002; jones, 2001; nbeet, 1994), there is little documentation of the use of staff time in the production of learning resources for faceto-face campus-based teaching. this paper is not concerned with costing methodologies, rather we are interested in the non-economic factors which influence the everyday practice of staff in the learning resources production process. we argue that the latter will enable us to understand better the needs of staff for support in the production of learning resources. a phenomenographic approach is needed to understand the everyday practices of staff and factors influencing the learning resources production process. introduction in the next five to ten years there will be a sector-wide need in he to produce a wider range of teaching and learning materials, for example, more resource-based learning (rbl) materials for self-directed study. the context for this is rapid technological advancement, 4 alt-] volume 10 number 2 increasing numbers and diversity of students and a decline in the unit of resource (ncihe, 1997). in particular, networked learning resources are increasingly presented as a cost-efficient means of maintaining teaching quality (ncihe, 1997). however, the relationship between cost-effectiveness and quality is contested. this future scenario has significant implications for university staff, whose time represents the largest cost element in the production of learning resources (chiddick, laurillard, quigley and wolf, 1997). some commentators (such as noble, 1998) present a pessimistic view of the future status of university staff given the increasing automation of teaching and intrusion of commercial interests. the increase in student-centred learning and use of technology may also lead to a significant change in the role of the teacher, from the 'sage on the stage' to 'guide on the side' (jones, 1999; salmon, 2000). an extensive survey of the literature has identified few studies which specifically investigate the production of learning resources in campus-based higher education institutions (heis). we suggest that the production of teaching and learning resources has not been considered a legitimate research topic, since it is an implicit and routine aspect of academic staff duties. therefore, there have been few in-depth studies which record everyday accounts of learning resources production and the factors affecting this. firstly we introduce the background to the research project, of which this paper is one part. we then critically review six models of the learning resources production process and its organization. finally, we suggest that a phenomenographic account of the everyday practices of learning resources production is needed. background to the research project the review presented here is part of a research project funded by the higher education funding council for england (hefce)'s teaching quality enhancement fund (tqef). the project aims to support the implementation of the institution's learning and teaching strategy, which aims to promote effective student learning. for a national overview of learning and teaching strategies see gibbs, habeshaw and yorke (2000) and hefce (2001a; 2001b). the research project will develop an understanding of the needs of academic staff for support in the production of a variety of teaching and learning resources, including networked learning resources, for use in undergraduate teaching across all subject disciplines within the university (plewes and issroff, 2002). the project will make recommendations about possible future structures for learning resources production at the university, including changes to the support and training infrastructure. literature review in this section we critically review six prominent models which document and cost the learning resources production process. we focus specifically on two aspects: firstly, the level at which the analysis operates for example, institution, course, faculty or department, member of staff, or student; secondly, the levels at which the costs of production of course materials are experienced. we also consider the purpose, structure and content of the models, critically analyse the advantages and disadvantages of their applications and issues arising from this. we discuss: 5 louise plewes and kim issroff understanding the development of teaching and learning resources: a review • open university (ou) course materials production models (rumble, 1976; bates, 1994); . • costs of networked learning course lifecycle model (bacsich, ash, boniwell, kaplan, mardell and caven-atack, 1999); • the pedagogic toolkit model (oliver and conole, 1999); • cost structures of teaching methods model (ncihe / chiddick et al., 1997); • student preferences / consumption of learning resources model (hobbs and boucher, 1997; boucher, 1998), and • the course resource appraisal model (laurillard, 1999). table 1 presents summary information for the models reviewed. model level of analysis level at which costs are experienced process of application and variables analysed open university (ou) faculty and course materials course production models institution (specifically distance learning) course team, specialized division of labour: staff time. costs of networked learning course lifecycle model course over full lifecycle. activity basedcosting can be applied at multiple levels. pedagogictoolkit model multiple levels, institution e.g., session, week, term, course cost structures of course institution teaching methods model student preferences / course institution consumption of learning resources model institution. also identifies accounting method activityhidden costs absorbed by based costing, stages of course staff and students. lifecycle. input teaching activities / media, allocate student time between media, look at staff time costs and resources needed. based on laurillard's (1993) conversational model. teaching methods (large and small group, external and in-house rbl), student time between these, staff time to prepare and present for each of these methods. economic cost curve analysis (graphical). amount of teaching delivered for fixed cost while incorporating student preferences. 6 alt-j volume 10 number 2 model level of analysis level at which process of application and costs are experienced variables analysed course resource course institution activities map on to media and appraisal model technologies, student time split between media, workload of three groups of o u staff to prepare and deliver teaching. table i: summary information for the models reviewed open university (ou) course materials production models as described previously, there is a lack of published literature describing the course materials production process in campus-based heis. however, there is an established literature on the course material production process for distance learning, in particular the uk open university (ou) (rumble, 1976; 1992; 1997; bates, 1994; 1995; 2000). recent developments in online distance education and virtual or 'mega-universities' (daniel, 1996) also build upon this ou model. we therefore use this model as a starting point for our review. however, since the ou is not representative of most heis, the conclusions may not be readily transferable to other contexts. ou course materials are generally developed by a course team of thirty people, who meet regularly. the team includes subject experts, educational technologists, a bbc producer, designers, editors and a course assistant. since there can be several drafts, the development and approval of course materials is time-consuming (for example, 18-20 months; rumble, 1976), and therefore also expensive (£450-500 for one hour of printed study materials in 1988; bates, 1994). such a course production system can only be justified by producing materials for large numbers of students (some ou science foundation courses have student numbers of 30,000; bates, 2000), and/or by using the same materials for many years. typical course life is 4-8 years (rumble, 1976). course materials are maintained by one to two members of the course team over the course life, at the end of which they are remade or replaced with new materials (rumble, 1997). this type of mass, industrialized production with a highly specialized division of labour (lewis, 1971a, 1971b; rumble, 1992; fames, 1993) is specific to the ou and is not common in campus-based heis where learning resources production is more of an ad hoc 'cottage industry' (fames, 1993; bates, 1997; peters, 1998). oliver, bradley and boyle (2001) have recently described the organizational and pedagogical difficulties in a collaborative, iterative approach to the distributed authoring of course materials for a virtual university. this builds upon the ou course materials production process described above, and again includes a highly specialized division of labour (peters, 1998). rumble (1997) has developed a weighting technique which expresses the staff time required to develop ou course materials. the unit of analysis is the faculty, but there is some interfaculty variation which reflects disciplinary differences in the types of learning resources developed (e.g. becher, 1989; smeby, 1996; neumann, 2001). sufficient course materials to keep the average learner studying for 10-12 hours per week are termed 1 unit. the ratio of course development time to course maintenance time is 10:1. that is, staff could maintain 7 louise pfewes and kim issroff understanding the development of teaching and learning resources: a review ten units of existing materials in the time taken to develop one new unit from scratch. where staff act as consultants and contribute to the development of units by other people this is weighted at 0.5 of a new unit. rumble (1976) and wagner (1977) have also conducted macro-analyses of staff time taken to develop course materials at the level of the whole institution. rumble and wagner have produced formulae to calculate the total cost of course material production to the institution. this is a function of the number of courses in development and presentation, student numbers, average cost of a course in development and presentation, average delivery cost per student, institutional overhead costs (fixed) and average course life (wagner, 1977). the purpose of this analysis was to inform an ongoing debate about the comparative cost of educating one student by the ou rather than a traditional hei. course lifecycle model the jlsc-funded 'costs of networked learning' project (bacsich, ash and heginbotham, 2001; bacsich et al, 1999) developed a three-stage cyclic course lifecycle model and advocated the use of activity-based costing (abc) methods to investigate the costs of networked learning. the purpose of this model is to make bidden costs explicit and to promote the use of a standard costing methodology for the he sector. the three stages of the course model are: planning and development; production and delivery; maintenance and evaluation. the model is based on course planning frameworks from the distance education sector (see previous section), designed to be equally applicable to both traditional and electronic learning resources. the model considers the course in the full context of pre-course (r & d) and post-course (evaluation) activities. abc is an accounting method where the cost of a product or service is determined by the activities involved in its production, not by volume-related allocation of overheads such as staff hours. abc methods may be applied at a number of different levels, such as department, faculty or institution (bacsich et al., 2001); application of the model involves use of a nested set of spreadsheets to examine costs at a variety of levels and to identify which of three stakeholders (the institution, staff or students) meets these costs. the student perspective is rarely considered, although the hobbs and boucher (1997) model considers student preferences (see later section). however, this model does not investigate the non-economic issues surrounding learning resources production. also, specialist training and software are required before the abc methods promoted may be used. therefore, this approach may not be accessible to those without training in abc methods. the results, however, should be comprehensible to all university staff and the standard costing methodology should enable comparisons to be made. the pedagogic toolkit model this project produced an online course design toolkit (oliver and conole, 1999), designed to operate flexibly at a variety of levels session, week, term and course. the toolkit enables teachers to input teaching activities and educational processes and to plan and cost the course development process. although targeted at implementation of learning technology into courses, the toolkit can also be used for traditional teaching methods and resources. the toolkit has three elements. firstly, 'media rater' looks at the teaching activities to be used. 8 alt-j volume 10 number 2 secondly, 'course modeller' allocates student time between these teaching activities and finally, 'media selector' considers the costs of staff time and resources required to support the course. estimates of the time taken to develop various types of materials from scratch are presented, although there is no indication of how these data were derived. the toolkit is based on laurillard's (1993) 'conversational framework' which also informs two other models discussed here (chiddick et al, 1997; laurillard, 1999). the toolkit is easy to use, encourages staff to reflect on the mix of media and teaching activities in their courses and to consider alternatives in terms of potential implications for their workload. cost structures of teaching methods model ncihe (1997) noted the recent shift in uk heis from a mixture of small-group teaching and lectures towards increased use of rbl materials for self-directed study. the implications of this scenario for staff workloads and costs to institutions were examined in appendix 2 of ncihe (1997) by chiddick et al. (1997). chiddick et al. (1997) analyse how a student's time might be distributed across three combinations of teaching methods each based on different ratios of small group teaching, lectures and rbl. assumptions are made about staff time for preparation and presentation required to produce one hour of student learning for each method (table 2). however there is no indication of the source of these time estimates or how it has been established that these would produce 'one hour of student learning'. teaching method preparation time (hours) presentation time (hours) total time (hours) i 1.5 0 2 0 20 table 2: assumptions of preparation and presentation staff time required to produce one hour of student learning for four different teaching methods (chiddick et al., 1997) the effect of increasing student numbers on staff time spent on learning resources production is analysed graphically as cost curves for each of the three scenarios. the report recommends the use of fixed-cost (for example, rbl) rather than variable-cost (for example, small-group) teaching methods. fixed-cost teaching methods are insensitive to increased student numbers and the high production costs can be amortized over large numbers of students. externally developed rbl material which can be adapted quickly is a more cost-effective option than developing rbl materials in-house, and a shift towards this is promoted (table 3). methods traditional study hours lectures (max. 100 students) groups (max. 10 students) rbl (external) rbl (in-house) 3 0 2 20 lectures groups rbl (external) rbl (in-house) 30 50 15 5 current study hours 60 5 15 20 future study hours 10 30 50 10 table 3: alternative combinations of teaching methods (chiddick et al., 1997,table i) 9 louise plewes and kim issroff understanding the development of teaching and learning resources: a review chiddick et al. (1997) developed their model in the context of continuing rises in student numbers. however, recent data (hefce, 2001c) suggests that large increases in student numbers between 1988 and 1994 (a 67 per cent increase) were followed by a levelling off of demand for full-time undergraduate he places between 1995 and 2001 (a 6 per cent increase). also, in practice, the shift away from small-group teaching has not occurred in all heis. some have continued to offer small-group teaching while reducing costs by employing lower unit cost staff (such as hourly-paid postgraduate students). student preferences / consumption of learning resources model hobbs and boucher (1997) suggest that student attitudes towards teaching methods should be considered, especially given the increasingly 'consumer-led' nature of he. therefore, they develop an economic analysis, similar to that of chiddick et al, using cost curves but incorporating 'end user preferences', that is, the consumption of learning resources by students. they suggest that according to ncihe (1997) students prefer traditional, nontechnology-based teaching methods such as lectures and small-group teaching, therefore the lowest cost option (rbl) may be unacceptable to students. however, relatively little is known about student preferences and this assumption may be anecdotal. the literature on learning resources in general concentrates on the production of resources by staff, rather than consumption by students, although student feedback on teaching may form part of revised quality assurance procedures. further research into and explanation of student preferences would therefore be useful. the model examines the amount of teaching that can be delivered for a fixed cost output, rather than considering input (for example, the staff time to deliver a fixed amount of teaching). the analysis focuses on capital rather than labour as the key factor in learning resources production. increasing use of networked learning resources relies upon the substitution of capital for labour, but it is debatable whether it reduces total costs. boucher (1998) argues that chiddick et al. do not adequately define what type of costs, total or average, are considered, and neglect other fundamental economic concepts such as economies of scale, marginal costs and diminishing returns. this model makes no distinction between internally and externally developed rbl materials which both chiddick et al. (1997) and laurillard (1999) consider to be fundamental. hobbs and boucher suggest that the optimal solution of student preferences coincident with lowest cost requires improvements in the quality of rbl materials. course resource appraisal model laurillard's (1999) course resource appraisal model is a simple framework which tabulates the allocation of student study hours between various learning activities. these relate to different media forms, which in turn map on to particular technologies as outlined in table 4. technologies printtv.video, dvd library, dvd, web seminar; online group laboratory, simulation essay, product model table 4: course resource appraisal model, after laurillard (2001) 10 learning activities attending investigating discussing practising articulating media forms narrative interactive communicative adaptive productive alt-j volume 10 number 2 the model looks at how student workload is distributed across various learning activities and media and assesses the implications for the workload of three groups of ou staff, termed academic, production and presentation. in this model the radical change in the role, workload and relations between technical and academic staff when producing electronic learning resources are significant, as discussed below. the production of electronic learning resources is especially labour-intensive in terms of technical staff time used in development. this is the main factor in their expense. for instance, at the ou, changing 20 per cent of course material to electronic learning resources increases academic staff time by 40 per cent but doubles technical staff time compared with production of traditional learning resources (laurillard, 2001), assuming that all electronic learning resources are created from scratch. this is a specific example for laurillard (2001); in general the degree of change in workload is dependent on the nature of material developed. an alternative option to creating electronic learning resources from scratch is to use generic, customizable resources. if 60 per cent generic, customizable resources are used, academic staff time increases by only 10 per cent and technical staff time increases by 20 per cent. therefore it is not a cost-effective use of staff time for an individual to produce their own materials, as they do for traditional learning resources (laurillard, 1999). in traditional campus heis, where staff are unlikely to work with dedicated teams of production staff, the entire burden of resource development and production would probably fall on academic staff. this is confirmed by a national survey of the production of electronic learning resources (hefce, 1999), which found that staff were mostly developing their own materials for their own needs. in addition, there are complex cultural reasons why generic, customizable learning resources are rarely used by academics in traditional, campus-based heis. while the review so far has mainly considered individual models, we now discuss some common features of the models. all of the models, by definition, are highly abstract and diagrammatic. several have a strong accounting or economic focus and are concerned with costs. few of the models explicitly consider staff except as passive variables, and even fewer consider students, with the exception of bacsich et al. (1999). most of the models are based on ou course production team methods which, although similar to team production of online distance teaching materials, do not correspond well to learning resources production practices in traditional campus-based teaching. laurillard's (1993) 'conversational framework' provides the theoretical basis underlying several of the models (chiddick et al.,\991; oliver and conole, 1999; laurillard, 1999). discussion: cultural factors influencing development and reuse of learning resources there are two ways in which staff may acquire learning resources without producing them themselves. these are (i) inter-institutional collaboration in the production of resources and (ii) the use or adaptation of generic, customizable resources (laurillard, 1999; jisc, 2002). neither of these options is perceived as attractive by academic staff (hefce, 1999), indicating a relatively widespread and entrenched autonomous and decentralized model of resource production. louise plewes and kim issroff understanding the development of teaching and learning resources: a review bates (2000) has coined the term the 'lone ranger and tonto' approach to describe actual practice in the development of learning resources in north american he. here the academic ('lone ranger') works with or employs an it-literate graduate student ('tonto') to develop electronic learning resources (bates, 1997). however, this practice does not often produce a usable end product as constant revisions to keep pace with technological change are necessary (bates, 1997). in contrast to the six models reviewed, this approach describes actual practice and should, we argue, be developed further. if, as it seems from the examples given above, academic staff are mostly developing learning resources themselves, why is this the case? hefce (1999) identifies a series of well-known 'constraining and enabling factors' to the development of electronic learning resources. these include lack of it skills, lack of time for it training, conflicting priorities, management pressure, incentives and rewards, and lack of suitable examples (tait and mills, 1999). the opportunity cost of the use of academic staff time is a key issue. hefce (1999) suggests that it is not cost-effective to institutions for staff to develop electronic materials. however, inter-institutional collaborative production can reduce development costs to a single institution and spread the risk (laurillard, 1999). especially at first-year undergraduate level, where the core curriculum is common to many institutions, there is clear potential for collaboration (laurillard, 1999). however, at higher levels where courses are research-led, difficulties arise which explain the lack of interest in this activity. the use of commercial resources is also possible. purchasing generic commercial material is cheaper than in-house development or commissioning (hunt and clarke, 1997; laurillard, 1999), but has tended to be a victim of the 'not-invented-here syndrome' where staff are reluctant to use materials which they feel do not bear their own personal stamp (laurillard, 1999). however, hammond et al. (1992; 160) suggest that the 'not-inventedhere' label is flawed as it underestimates and undervalues the conventional process by which lecturers prepare and update their material. they refer to textbooks, monographs, the research literature, and their own research and experience for sources of factual information, ideas, representations, and organisation during the preparation of lecture notes, seminars and tutorials. while use of commercial materials saves development time, additional time is needed to evaluate and adapt resources (laurillard, 1999). also, there are few commercial resources developed for or targeted specifically at he, and, conversely, there is little interest by commercial publishers in resources developed by heis (hefce, 1999). these issues relate to teaching staff reluctance to cede autonomy over teaching content and delivery (ryan, scott, freeman and patel, 2000). electronic learning resources in particular raise these issues, given their potential to change radically the nature of academic work and university organization (see noble, 1998). for example, the trend towards automation and commodification of he (noble, 1998) and publicly available online materials may result in increased scrutiny of these materials, and potentially the substitution of materials for teachers (jones, 1999). 12 alt-j volume / 0 number 2 conclusions this paper has identified and discussed six models which may be used to understand the development and re-use of learning resources. we have focused specifically on aspects which these models do not address such as actual staff practices and the cultural context of development. there are few in-depth empirical studies of the everyday practices of academic staff in terms of learning resources production. since these are rarely articulated, recorded or documented there has been little systematic investigation of the ways in which learning resources production is conceptualized. unless we explore and document these conventional forms of work in the production of traditional learning resources, it is difficult to understand the value attached to them and how the increased use of electronic learning resources represents a threat to these traditional resource production practices. as a result of these concerns, there is a need to develop a phenomenographic account of staff everyday practices in the production of a variety of teaching and learning resources. phenomenography is an emerging qualitative research method which is increasingly applied in educational research (such as marton, 1981; marton, hounsell and entwistle, 1984; entwistle, 1997; brew, 2001a; 2001b) to study empirically descriptive conceptions of the world around us. this includes the different ways in which people experience, perceive, understand and conceptualize the same phenomena. references bacsich, p., ash, c., boniwell, k., kaplan, l., mardell, j. and caven-atack, a. (1999), the costs of networked learning, sheffield hallam university, school of computing and management sciences, http://www.shu.ac.uk/cnl/report1.html. bacsich, p. and ash, c. (2000), 'costing the lifecycle of networked learning: documenting the costs from conception to evaluation', active learning, 8 (1), 92-102. bacsich, p., ash, c. and heginbotham, s. (2001), the costs of networked learningphase two, telematics in education research group on behalf of the school of computing and management science, sheffield hallam university, http://www.shu.ac.uk/cnl/report2.html. bates, a. w. (1994), costing distance education technologies: developing a methodology, canada, open learning agency. bates, a. w. (1995), technology, open learning and distance education, london: routledge. bates, a. w. (1997), 'the impact of technological change on open and distance learning', distance education, 18 (1), 93-109. bates, a. w. (2000), managing technological change: strategies for college and university leaders, san francisco: jossey-bass. becher, t. (1989), academic tribes and territories: intellectual enquiry and the cultures of disciplines, buckingham: srhe/open university press. boucher, a. (1998), 'information technology-based teaching and learning in higher 13 louise plewes and kim issroff understanding the development of teaching and learning resources: a review education: a view of the economic issues', journal of information technology for teacher education, 7(1), 87-111. brew, a. (2001a), 'conceptions of research: a phenomenographic study', studies in higher education, 26 (3), 271-85. brew, a. (2001b), the nature of research: inquiry in academic contexts, london: routledge/falmer. chiddick, d., laurillard, d., quigley, g. and wolf, d. (1997), 'new approaches to teaching: comparing cost structures of teaching methods', in ncihe, higher education in the learning society, report of the national committee of inquiry into higher education, 'the dearing report', london: hmso, http://www.leeds.ac.uk/educollncihe/a2_001.html. daniel, j. s. (1996), mega-universities and knowledge media: technology strategies for higher education, london: kogan page. detya and ernst and young (2000), 'a study to develop a costing methodology for the australia higher education sector', http:llwww.detya.gov.au/highered/otherpub/costing/costing.pdf entwistle, n. (1997), 'introduction: phenomenography in higher education', higher education research and development, 16 (2), 127-34. fames, n. (1993), 'modes of production: fordism and distance education', open learning, 8 (1), 10-20. flashlight (2002), http://www.tltgroup.org/programs/flashlight.html gibbs, g., habeshaw, t. and yorke, m. (2000), 'institutional learning and teaching strategies in english higher education', higher education, 40, 351-72. hammond, n., gardner, n., heath, s., kibby, m., mayes, t., mcaleese, r., mullings, c. and trapp, a. (1992), 'blocks to the effective use of information technology in higher education', computers and education, 18, 155-62. haywood, j., anderson, c., coyle, h., day, k., haywood, d. and macleod, h. (2000), 'learning technology in scottish higher education a survey of the views of senior managers, academic staff and "experts"', alt-j, 8 (2), 18-28. hefce (1999), communications and information technology materials for learning and teaching in uk higher and further education, bristol: hefce. hefce (2001a), analysis of strategies for learning and teaching: research report by professor graham gibbs, bristol: hefce, http://www.hefce.ac.uk/pubs/hefce/2001/ 01 _3 7a..htm. hefce (2001b), strategies for learning and teaching in higher education: a guide to good practice, bristol: hefce, http://www.hefce.ac.uk/pubs/hefce/2001/01_37.htm. hefce (2001c), supply and demand in higher education, bristol: hefce, http://www.hefce.ac.uk/pubslhefcel2001/01-62.htm. hobbs, p. j. and boucher, a. c. (1997), 'an analysis of teaching methods, costs and student preferences', active learning, 7, vii-x. 14 alt-j volume 10 number 2 hunt, m. and clarke, a. (1997), a guide to the cost effectiveness of technology-based training, coventry: national council for educational technology and sheffield: dfee. joint information systems committee (jisc) (2002), exchange for learning (x4l) programme, circular 2/02 january 2002, http://www.jisc.ac.uk/pub02/c02_02.html. jones, c. (1999), 'from the sage on the stage to what exactly? description and the place of the moderator in co-operative and collaborative learning', alt-j, 1 (2), 27-36. jones, d. (2001), 'technology costing methodology handbook', version 1.0, nchems/ wcet, http://www.wiche.edultelecom/projects/tcm/proj-projects.html. laurillard, d. (1993), rethinking university teaching: a framework for the effective use of educational technology, london: routledge/falmer. laurillard, d. (1999), 'investing in information technology pays big dividends', planning for higher education, 27, 1-8. laurillard, d. (2001), 'exploring the principles of online learning', presentation given in cambridge, may 2001, http://www2.open.ac.uk/ltto/lttoteam/diana/cambridge/index.htm. lewis, b. n. (1971a), 'course production at the open university i: some basic problems', british journal of educational technology, 1 (2), 4-13. lewis, b. n. (1971b), 'course production at the open university ii: activities and activity networks', british journal of educational technology, 2 (2), 111-23. marton, f. (1981), 'phenomenography: describing conceptions of the world around us', instructional science, 10, 177-200. marton, f., hounsell, d. and entwistle, n. j. (eds.) (1984), the experience of learning, edinburgh: scottish academic press. national committee of enquiry into higher education (ncihe) (1997), higher education in the learning society, 'the dearing report', london: hmso, http://www. leeds. ac. uk/educol/ncihel. national board of employment, education and training (nbeet) (1994), costs and quality in resource-based learning onand off-campus, commissioned report no. 33, canberra: australian government publishing service, http://www.detya.gov.au.nbeet/. neumann, r. (2001), 'disciplinary differences and university teaching', studies in higher education, 26 (2), 135-46. noble, d. (1998), 'digital diploma mills: the automation of higher education', first monday: peer-reviewed journal on the internet, 3 (1), http://www.firstmonday.dk/issues/ issue3_1/noble/index.html. oliver, m. and conole, g. (1999), assessing and enhancing quality using toolkits, effects report no. 14, london: university of north london. oliver, m., bradley, c. and boyle, t. (2001), 'the distributed development of quality courses for a virtual university', alt-j, 9 (2), 16-27. peters, o. (1998), learning and teaching in distance education: analyses and interpretations from an international perspective, london: kogan page. 15 louise plewes and kim issroff understanding the development of teaching and learning resources: a review plewes, l. and issroff, k. (2002), 'academic staff attitudes towards the use and production of networked learning resources', in proceedings of the third international conference on networked learning 2002, university of sheffield, 489-96. rumble, g. (1976), the economics of the open university of the united kingdom, open university planning office. rumble, g. (1992), 'the competitive vulnerability of distance teaching universities', open learning, 7, 31-45. rumble, g. (1997), the costs and economics of open and distance learning, kogan page: london. ryan, s., scott, b., freeman, h. and patel, d. (2000), the virtual university: the internet and resource-based learning, london: kogan page. salmon, g. (2000), e-moderating: the key to teaching and learning online, london: kogan page. smeby, j.-c. (1996), 'disciplinary differences in university teaching', studies in higher education, 21 (1), 69-79. tait, a. and mills, r. (eds.) (1999), the convergence of distance and conventional education: patterns of flexibility for the individual learner, london: routledge. wagner, l. (1977), 'the economics of the open university revisited', higher education, 6, 359-81. 16 educational innovation, learning technologies and virtual culture potential' david riley imperial college centre for educational development imperial college of science, technology and medicine email: d.riley@ic.ac.uk learning technologies are regularly associated with innovative teaching but will they contribute to profound innovations in education itself? this paper addresses the question by building upon merlin.donald's co-evolutionary theory of mind, cognition and culture. he claimed that the invention of technologies for storing and sharing external symbol systems, such as writing, gave rise to a 'theoretic culture' with rich symbolic representations and a resultant need for formal education. more recently, shaffer and kaput have claimed that the development of external and shared symbol-processing technologies is giving rise to an emerging 'virtual culture'. they argue that mathematics curricula are grounded in theoretic culture and should change to meet the novel demands of 'virtual culture' for symbol-processing and representational fluency. the generic character of their cultural claim is noted in this paper and it is suggested that equivalent pedagogic arguments are applicable across the educational spectrum. hence, four general characteristics of virtual culture are proposed, against which applications of learning technologies can be evaluated for their innovative potential. two illustrative uses of learning technologies are evaluated in terms of their 'virtual culture potential' and some anticipated questions about this approach are discussed towards the end of the paper. the challenge of recognizing and anticipating educational innovation innovative teaching is one of the key themes of alt-j and a matter of considerable practical concern to lecturers embracing learning technologies. many innovations, however, are digital variations on previous practices that date back to the times when numbers and letters were first inscribed, when literacy became a concern, and formal education was invented as the solution. this paper explores the notion that learning technologies could 45 david riky educational innovation, learning technologies and 'virtual culture potential' require us to be innovative in a more fundamental sense and will force us to review our cultural assumptions about education itself. the question addressed here is how we might recognize, or better still anticipate, profound innovations that could result from the use of learning technologies. a low-pass cognitive-cultural filter a premiss of this paper is that we need to filter out the high frequency 'noise' of technological and educational change, whilst attending to the lower frequencies and themes running through human cognitive and cultural evolution. it is assumed that our ability to appreciate and compose across the full dynamic range of educational innovation will be enhanced by an awareness of these underlying themes. a theory of human evolution that is admirably suited to the purpose is outlined next. the theory begins with an account of hominid evolution several million years ago and culminates in a cultural critique of education as we know it today. this theory has been further elaborated by two educationists concerned with the impact of computational technologies on innovation in mathematics curricula. their conception of an emergent 'virtual culture' provides the inspiration for this paper and a device that may prove fruitful in the search for innovation in education. a theory of cognitive and cultural co-evolution the psychologist merlin donald claimed in his landmark book (donald, 1991) and subsequent precis (donald, 1997) that the human mind co-evolved in close interaction with both brain and culture over millions of years. he postulated three major cognitive transitions with each cognitive step-change being marked by the development of novel and increasingly hybrid forms of representation and by the subsequent rise of more complex human cultures. episodic culture according to donald, early hominids possessed a chimpanzee-like brain that was brilliant at event-perception, capable of subtle social interactions, and sensitive to the significance of environmental events. he claims, however, that episodic recall was poor and environmentally driven rather than readily self-triggered at will. he describes this culture as 'episodic', with limited scope for the voluntary recall of events, for the rehearsal and purposive refinement of skills and the expression of knowledge. mimetic culture donald's first biophysical and cognitive transition was a breakthrough in motor and mimetic skills, permitting the development of body language as a form of representation and communication. this change, he says, occurred around one and a half million years ago as homo erectus appeared. the resulting mimetic culture was based on the new-found capability for self-triggered rehearsal and for voluntary representational acts that facilitated sociocultural development. skilled tool-making, gesture, dance, mime and ritual became possible through embodied performance with the capacity for iconic representation leading to more abstract forms of symbolic but pre-linguistic representation. mythic culture donald's second transition resulted from changes to human biological and cognitive structures affecting motor skills, mimesis and symbolic representation. they enabled 46 an-] volume 10 number i lexical invention and contributed to the development of oral language. he argues that speech and narrative forms emerged some 300,000 years ago, following physical and cognitive adaptations of the brain, the nervous system and the human vocal apparatus. the resulting mythic culture was a hybrid, interweaving inherited episodic and mimetic capabilities with the new ones of storytelling, more flexible symbolization and the potential for abstraction away from specific events and performances. theoretic culture instead of a biophysical change in memory and cognition, donald's third transition marked a technological change around fifty to thirty thousand years ago, in the upper palaeolithic. people produced inscriptions for the first time and externalized symbolic memory, initially in images and later in numerical and written systems. the resulting theoretic culture was founded on external memory records that were created, interpreted and applied by a 'literate class' with responsibilities for commerce, governance, legal codes, religion and health. he suggests the demands of numeracy, literacy and the maintenance of specialist discourse communities were such that formal education became a necessary component of the emerging theoretic culture. over time, education enabled the sharing and dissemination of these literacies whilst reinforcing theoretic culture through the invention of sophisticated and specialized symbol systems. contemporary mathematics, science and technology are fruits of this culture, as are the arts and humanities. technologies for externally storing and sharing symbol systems proved to be remarkably potent as homo sapiens took culture beyond the biological limits of individual memory, mental reasoning and oral transmission. with respect to modern information and communication technologies, donald treats these as significant developments falling within the scope of theoretic culture. he notes that they enhance the storage, communication and display of symbol systems and symbolic representations and discusses the cognitive and cultural impact of a 'global electronic information environment'. a critique of theoretic culture donald's inclusion of modern technologies within an established theoretic culture has been challenged by david shaffer and james kaput (1999), who believe there has been a fourth cognitive transition triggering the emergence of a 'virtual culture'. they suggest that donald overemphasized the importance of memory and external symbol storage functions in theoretic culture and underestimated the significance of external symbol processing. shaffer and kaput argue that the advent of computational technologies constitutes a major cognitive transition bringing culturally unprecedented forms of dynamic and interactive representation, and giving rise to the qualitatively new virtual culture. the educational implications of a virtual culture shaffer and kaput used their conception of an emergent virtual culture to argue that it places additional and novel demands on people's representational abilities, and hence on education. they argue that mathematics curricula will need to be changed if they are to satisfy the symbol-processing needs of a virtual culture. 47 david riley educational innovation, learning technologies and virtual culture potential' in their view, the aims of mathematics curricula should be to: • promote 'generative fluency', that is, students' ability to learn varieties of representational systems (see also kaput, 1998); • provide opportunities to create and modify representational forms; • develop skills in making and exploring virtual environments; • emphasize mathematics as a fundamental way of making sense of the world; • offer a more embedded and situated mathematics, that builds on meaning and places less emphasis on exact computation and formal proof except for students who require these specialized skills. in other words, shaffer and kaput say that learning to read and write symbols is no longer sufficient for students of mathematics. a virtual culture requires people who are able to 'read and write' symbol-processing for representational purposes. thus, an education system that developed to support the internal symbol-processing needs of theoretic culture should adapt and grow to support the external symbol-processing needs of virtual culture. virtual culture and the challenge of educational innovation the above account of human biological and cultural evolution is stimulating but surely a new and different stage in human culture should be universal in its reach and impact? if shaffer and kaput's claim of a virtual culture is reasonable, their arguments about mathematics curricula should be generalizable across most, if not all, curricula. readers are invited to judge for themselves whether they believe computationally based technologies are having such a deep impact in their own fields and whether curricula are changing or whether they should change as a result. this paper proceeds on the assumption that shaffer and kaput's conception of virtual culture is insightful and therefore could be of value in evaluating the innovative potential of learning technologies. i will regard the use of learning technologies for symbol storage and display to be characteristic of a theoretic culture and unlikely to be culturally innovative. in contrast, i regard the use of learning technologies for autonomous symbol processing and interactive representations to be characteristic of a virtual culture and more likely to be culturally innovative. for instance, students posting an essay on an intranet or sending emails to a tutor are using the digital equivalents of earlier systems. these practices are digital in their implementation but rely on the communication and sharing of symbols and hence are typical of theoretic culture. the use of online multiple-choice tests with automated feedback for tutors and students takes advantage of machine computation and of hybrid, interactive representations. this is indicative of an educational practice belonging to virtual culture. virtual culture potential to structure and take forward these ideas, i introduce the notion of 'virtual culture potential' and provide a working definition of virtual culture as an assemblage of four diagnostic practices: 48 alt-j volume 10 number i 1. the processing of external symbol systems on interactive, computational devices; 2. the use of these devices for representational purposes in human communications; 3. the representation of entities, events and environments that are material, imaginary or abstract; 4. the use of multiple and hybrid forms of representation the juxtaposition and interweaving of representational forms typical of episodic through to virtual cultures. this assemblage of practices makes it possible to characterize uses of learning technologies in terms of their alignment with, and potential to further, virtual culture. applications of learning technologies that entail all four practices should be ascribed a high virtual culture potential whilst lower potentials are indicated if these practices are absent or less well developed. for instance, both the applications of learning technologies shown in table 1 evaluation criteria for virtual culture potential of applications of learning technologies learning technologies, educational applications and match with criteria geographic information dynamic modelling system [gis] system spatial variability of nitrogen fluxes soil nitrate-n in an agro-ecosystem 1i) the processing of external symbol systems on yes interactive, computational devices (2) the use of these devices for representational yes purposes in human communication (3) the representation of entities, events and yes environments that are material, imaginary or abstract (4) the use of multiple and hybrid forms yes of representation: the juxtaposition and interweaving of representational forms typical of episodic through t o virtual cultures yes yes yes yes table /: evaluation of the virtual culture potential of two applications of learning technologies illustrate high levels of virtual culture potential. this potential would be reinforced by the integration of the two learning activities using a composite spatial and temporal modelling system (maceachren, 1995). learning activities such as those illustrated in table 1 would have been impracticable before computers became available and few environmental scientists or geographers would have imagined educational projects of this kind. yet, in the wider world, it no longer seems strange that decision-makers should consult epidemiological models or global climate modellers (bloomfield, 1986; macmillan, 1989). nor does it seem exceptional that 49 david riley educational innovation, learning technologies and 'virtual culture potential' interactive computer simulations should seamlessly represent the past, the present and a range of imaginary futures whilst serving the worlds of science and politics as twenty-firstcentury oracles. reflections on virtual culture potential this paper has strayed close to the abstract, theoretical and speculative limits of respectability for a journal such as alt-j. the time has come to acknowledge and address a few anticipated criticisms of the notion of 'virtual culture potential' as it has been advanced in this paper. firstly, whilst donald's theory is widely acknowledged, it is merely one of several theories of the evolution of the human mind (mithen, 1996; plotkin, 1997), and i can neither support nor refute his particular account. instead, it is regarded as a valuable source of inspiration for myself and for shaffer and kaput who made the crucial distinction between the importance of symbol-storing and symbol-processing technologies. this distinction and their notion of virtual culture could prove to be fruitful in identifying opportunities for educational innovation even if donald's theory is found to be wanting in some respects. secondly, the distinction between symbol storage and symbol processing is simplistic, in that running an email or intranet system entails considerable information-processing. however, i believe the computations that are embedded within such systems and that are transparent to users are not germane to the educational issues and approach expressed in this paper. thirdly, what is new in this paper? the privileging of computation is not new; disessa (2000) discusses at length his ideas about computers, learning and literacy, whilst acknowledging the early influence of seymour papert. it seems to me, however, that donald's ideas and their derivatives have a broader cultural sweep than those of disessa, who restricts much of his thinking to the notion of programming in the mathematical sciences and related fields. the challenge of combining or integrating interactive and narrative representational forms is, for instance, an issue gaining attention in the humanities (igcc, 2001) and sits more comfortably within shaffer and kaput's notion of a virtual culture than within disessa's frame of reference. the notion of computing to represent and communicate similarly differs from the approach explored by brna, baker and stenning (1999) in a conference on communicative interactions and the development of students' modelling skills in mathematics and science. virtual culture affords the wider educational sweep and highlights the activity of modelling to represent and to communicate, rather than the activity of communicating in order to model. conclusions this paper has developed a notion of 'virtual culture potential' by building on the earlier work of donald, shaffer and kaput. the notion is advanced for its heuristic possibilities in recognizing the potential of learning technologies to trigger profound educational innovations. acknowledgements this paper is dedicated to the memory of the late professor david squires of king's college london. i had looked forward to discussing the implications of virtual culture 50 au-j volume 10 number i with david at alt-c 2001 and it was with great sadness that i learnt of his death two weeks before the conference. i am deeply indebted to merlin donald, david williamson shaffer and james kaput, whose ideas have been recast and inevitably caricatured here. finally, my thanks go to the anonymous referees and colleagues who provided feedback on drafts of this paper, in particular, to dr benita cox of the management school of imperial college. references bloomfield, b. p. (1986), modelling the world: the social constructions of systems analysts, oxford: basil blackwell. brna, p., baker, m. and stenning, k. (eds) (1999), 'the roles of communicative interaction in learning to model in mathematics and science', unpublished proceedings of the clemmas conference, ajaccio, corsica, april 1999. disessa, a. a. (2000), changing minds: computers, learning, and literacy, cambridge, ma: mit press. donald, m. (1991), origins of the modern mind: three stages in the evolution of culture and cognition, cambridge, ma: harvard. donald, m. (1997), 'precis of origins of the modern mind: three stages in the evolution of culture and cognition', behavioural and brain sciences, 16 (4), 737-91. also: ftp://ftp.princeton.edu/pub/harnadlbbs/. wwwibbs.donald.html (accessed october 2001). igcc (2001), 'the international game cultures conference 2001', university of the west of england, http:llhumanities.uwe.acuk/gamesculturelindex.htm (accessed october 2001). kaput, j. j. (1998), 'representations, inscriptions, descriptions and learning: a kaleidoscope of windows', journal of mathematical behaviour, 17 (2), 265-81. maceachren, a. m. (1995), how maps work, new york: guilford press. macmillan, b. (ed.) (1989), remodelling geography, oxford: basil blackwell. mithen, s. (1996), the prehistory of the mind: a search for the origins of art, religion and science, london: phoenix. plotkin, h. (1997), evolution in mind: an introduction to evolutionary psychology, london: penguin books. shaffer, d. w. and kaput, j. k. (1999), 'mathematics and virtual culture: an evolutionary perspective on technology and mathematics education', educational studies in mathematics, 37 (2), 97-119. 51 alt-j, research in learning technology vol. 12, no. 3, september 2004 issn 0968–7769 (print)/issn 1741–1629 (online)/04/030203–02 © 2004 association for learning technology doi: 10.1080/0968776042000259528 editorial taylor and francis ltdcalt120301.sgm10.1080/0968776042000259528alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2004association for learning technology123000000september 2004gráinneconoleschool of educationuniversity of southamptonso17 1bjg.c.conole@soton.ac.uk this issue forms a special collection on computer assisted assessment (caa) and has been produced in conjunction with the international computer assisted assessment conference (www.caaconference.com offers a free downloadable archive of all previous caa conference papers. it includes selected papers from the caa conference, along with other caa papers from across further and higher education. the variety of different papers, and the issues they explore, illustrates that caa is now an active and established research area, which is testing both the technical and pedagogical challenges associated with developing and delivering assessment online. this special issue is dedicated to a colleague and good friend, dr joanna bull, who sadly passed away in june this year. her death has come as a great shock to us all. jo was a leading light in the field of caa, being well known and respected by the community. she directed the highly successful tltp-funded national caa centre, and through this co-authored the blueprint for caa which is still widely cited and used. jo produced a wealth of publications, material, and most recently had set up an independent consultancy through her own business ‘eduology’. she will be very, very sadly missed, but not forgotten. the issue consists of six papers: five focus specifically on caa, and one more broadly on the pedagogies of e-learning. the first article by sclater, describes an ambitious cross-sector initiative in scotland which is developing a large bank of assessment items for units across the scottish further education sector. the paper highlights the issues involved with the initiative such as decisions about appropriate item types and subject areas, the need for training and guidelines for authors, the nature of the peer-review and quality assurance processes and associated metadata and interoperability issues. the second article by sim et al. provides a review of the literature on caa and a critique of associated issues, such as interoperability, security, test construction and testing higher cognitive skills. the third article by mcalpine reports on the development of an evaluation framework to facilitate the translation of paper-based to computer-based assessments and in particular the framework aims to assess and enhance the practicability of largescale caa migration for existing items and assessments. the forth article by clarke et al. provides an insightful case study to evaluate the use of online mcqs in humanities as a means of reinforcing and extending student learning at oxford university. in particular the research focused on a series of themes, 204 editorial reinforcement of learning, cementing students’ understanding, deepening students’ knowledge and framing educational expectations. however they conclude that ‘there is still much work to do if there is to be a shift in our assessment paradigms … and the benefits of formative assessment can be realized and integrated into the rigid study structures posed by many higher education systems’. the fifth article by davies provides a different kind of case study that evaluated the validity of assessing students via a computerized peer-marking process. the study concluded that the higher performing students achieve a greater improvement in their overall marks. the sixth article by mehanna on the pedagogies of e-learning explores the different pedagogic techniques employed to establish effective e-learning practice. she carried out a detailed study of a computer mediated conference over a year, and derived nine clusters of pedagogies consisting of 29 pedagogic behaviours, and concludes that blending different learning theories and pedagogies is not only possible but beneficial. we continue our discussion series and in this issue boyle and cook provide an insightful commentary on conole and dykes’ affordances of ict paper published in the last issue, along with a response from the authors. the notion of affordances in relation to e-learning is a hotly contested one and we hope that these responses provide a useful starting point for a more detailed discussion on this topic. gráinne conole, editor myles danson, director of the international caa conference and caa manager at loughborough university doi:10.3402/rlt.v23.28451 a blended learning framework for curriculum design and professional development negin mirriahi*, dennis alonzo and bob fox school of education & learning and teaching unit, university of new south wales, sydney, australia (received 5 may 2015; final version received 27 september 2015) the need for flexibility in learning and the affordances of technology provided the impetus for the rise of blended learning (bl) globally across higher education institutions. however, the adoption of bl practices continues at a low pace due to academics’ low digital fluency, various views and bl definitions, and limited standards-based tools to guide academic practice. to address these issues, this paper introduces a bl framework, based on one definition and with criteria and standards of practice to support the evaluation and advancement of bl in higher education. the framework is theoretically underpinned by the extant literature and supported by focus group discussions. the evidence supporting the criteria and standards are discussed with suggestions for how they can be used to guide course design, academic practice, and professional development. keywords: digital literacy; criteria; standards; academic development; curriculum design introduction the demand for flexibility in learning and the affordances of technology provided the impetus for the rise of blended learning (bl) across the higher education sector. since the early 1990s, its popularity has increased, and recently, it has received more attention due to education institutions attempting to offer more personalised learning experiences. bl has the capability to deliver personalised learning when designed with a strong focus for meeting the needs of individual students (gaeta, orciuoli, and ritrovato 2009) and provided with strong institutional support and policy to enable more effective learning and teaching (hargreaves 2006). the popularity of bl, particularly in higher education contexts, however, does not necessarily translate into advancement of academic practice due to three key challenges. first, digital fluency or academics’ confidence and skills in using online technologies remain low (johnson et al. 2014) despite the availability and affordances of digital technologies. the low digital skills of academics compromise appropriate technology integration, limiting the facilitation of more effective student learning (torrisi-steele and drew 2013). at the moment, the use of technology for instruction is mostly for management and administrative purposes rather than for facilitating learning (palak and walls 2009). *corresponding author. email: negin.mirriahi@unsw.edu.au responsible editor: carlo perrotta, school of education, university of leeds, united kingdom. research in learning technology vol. 23, 2015 research in learning technology 2015. # 2015 n. mirriahi et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 second, there are various views and definitions of bl, which according to oliver and trigwell (2005) ‘is ill-defined and inconsistently used’ (p. 24). consequently, there is no uniform understanding of bl, and hence, academic practice is often underpinned by individuals’ own interpretations of the term rather than a consistent approach across an institution (hinrichsen and coombs 2013). the inconsistencies revolve around the design, pedagogical approaches, portion of online versus face-to-face time, purpose of blending, and the role of technology. for example, garrison and kanuka (2004) posit that the integration of differing modalities requires the combination of the most desirable aspects of face-to-face and online environments. further, bl can address access, convenience, and cost-effectiveness, enabling students to save a considerable amount of time and resources from commuting and institutions to reduce the cost for additional buildings and facilities (bleed 2001). however, procter (2003) critiques the view that bl addresses the challenge of distance, arguing that it has a different design and delivery approach than fully distance learning. in particular, procter (2003) emphasises that bl requires the ‘effective combination of different modes of delivery, models of teaching and styles of learning’ (p. 3). this is based on the assumption that the achievement of learning outcomes is dependent on the quality of learning and teaching experiences. third, the tools available to guide and evaluate bl course designs are limited (smythe 2012). though, there are available frameworks to design and evaluate bl practices both from the perspectives of learning and teaching and it infrastructure, these frameworks are problematic either in their design or in the criteria and standards, or lack thereof. for example, some frameworks have identified the criteria needed but take the form of a likert scale with no description of standards. smythe’s (2012) framework has five levels of performance and claims to be standards-based, but it lacks the descriptions of standards for each level. this is problematic as it allows academics to have their own judgement on what is considered appropriate for each level. oliver (2003) provides benchmarks with criteria and standards, but it is largely an adaptation of the principles of face-to-face teaching rather than considering the criteria for effective bl practices. parsell and collaborators’ (2013) framework includes criteria, but they are generic with emphasis on the elements of learning and teaching with technology appearing as an additional component and not as interweaved together. the use of explicit criteria and standards in bl will facilitate more effective learning and teaching activities as the criteria can be used to benchmark academic practice (reed 2014). the three issues discussed above are critical for bl implementation for enhancing academics’ skills and confidence using technologies, formulating a consistent definition to inform academics’ practice, and providing frameworks for objective evaluation of bl practice. we propose a standards-based bl framework based on one definition that reconciles the discrepancies in the literature discussed under issue two above and informed by the literature and supported by qualitative data gathered from focus groups. the framework provides a consistent understanding of bl practice and engages academics in self-assessment of their own practice to help them identify areas of expertise and areas requiring further development. the framework is introduced in the next section to provide a conceptual overview of the criteria and standards and how they have been informed. the introduction of this proposed framework provides academics, researchers, and others interested in bl with an opportunity to trial and/or adapt the framework to their own contexts. n. mirriahi et al. 2 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 the blended learning framework after investigating the various interpretations and purposes of bl in the literature, the following consolidating definition forms the basis of the proposed bl framework: a process of integrating the most appropriate learning and teaching strategies, technology and/or media to provide meaningful, flexible learning experiences to achieve learning outcomes (based on dick, carey, and carey 2009; holden and westfall 2010). institutions can adapt this definition to fulfil their own strategic bl directions and provide learning experiences as appropriate for their contexts. the framework, therefore, guided by this definition, addresses the three issues identified. the standardsbased framework proposed in this paper can: (1) guide course design to ensure consistent high quality bl practice across an institution; (2) be used by academics as a self-assessment instrument to identify their strengths and weaknesses in bl practice; and (3) inform professional development programs. method a combination of logical, rational and theoretical approaches (brown 1983; friedenberg 1995) was used in the development of the framework. the criteria and standards were established based primarily on literature on bl frameworks and tools for bl evaluation and with support from focus groups at one higher education institution. the development process started with a literature search from online databases and search engines to identify existing bl frameworks and evaluation instruments used in higher education institutions. two focus groups were conducted with a total of eight participants representing various disciplines and roles to collect information regarding current bl practice, the challenges faced in terms of implementation, and what they require to advance bl in their disciplines. focus group discussions were audio recorded, transcribed, and any identifiable information was removed prior to analysis. table 1 below shows the distribution of the focus group participants. while the number of participants is limited, it provides an initial understanding of whether the participants’ views support or refute the findings in the literature that inform the proposed bl framework. the proposed framework the proposed bl framework contains criteria, which are the indicators of the ability of academics in designing and delivering a bl course, and standards that define the quality of practice. the use of criteria and standards has been proven by research to effectively support practice and skill development (wolf and stevens 2007). table 1. distribution of focus group participants based on role. role number senior administrator (associate dean/director) 2 lecturer 2 educational developer/instructional designer 3 research in learning technology citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 the criteria and standards will guide academics to determine their current level of bl practice and subsequent levels of practice that they should aspire towards to further improve their practice (inbar-lourie 2008). griffin’s (2000) research revealed that using criteria and standards can support various professionals in their skill development and using a skills assessment instrument with criteria and standards can be linked to identifying professionals’ zone of proximal development (zpd), as defined by vygotsky (1986) as the area of opportunity for skill development. further, the results of a skills assessment with explicitly defined standards can inform needs-based professional development programs and resources. this is consistent with bruner’s (1996) learning theory where the progression of learning should be built upon the learner’s current level of ability. the criteria the criteria and standards in the proposed framework are organised around the rase learning design model that supports a student-centred, technology-rich environment suitable for bl. this particular model was chosen due to its suitability for bl in technology-rich environments. central to the rase model is identifying activities for students to work on, to gain multiple skills, literacies, knowledge and content competencies needed to address meeting the learning outcomes defined by a bl course. the resources in the rase offer students well-structured exercises and discipline content, that enables students to work successfully through the activities set. support in the rase provides students with assistance in technical, peer and tutor support needed to understand how best to work through the tasks set and to offer advice in areas students might find difficult. evaluation (assessment) in the rase offers a structured guide to help students understand how well they are doing as they work their way through the course. the evaluation (assessment) stage in a blended environment also enables teachers to monitor how well individual and groups of students are doing as they progress through the course. the evaluation (assessment) stage can offer a traffic light signal to teachers, identifying students in need of further tutorial support (churchill et al. 2013). in the following section, each criterion is discussed with evidence from the literature and supplemented with comments made by focus group participants where available. resources criteria related to resources focuses on the availability of resources in general and of those related to formative assessment. (1) availability of course resources. the availability of digital resources ensures equitable and sustainable access (bath and bourke 2010) thus, giving students equal opportunity to engage with course resources. similarly, these digital resources can be easily updated and re-organised for future use. further, academics should allow students to develop their own resources at their own level of understanding and interest, when appropriate to do so, so they are actively engaged in resource development as a way to support their learning. the creation of material informs the higher standard for this criterion (standard c on table 2) where not only are resources available online but students are also creating them. one of the n. mirriahi et al. 4 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 table 2. the blended learning framework. standards criteria standard a standard b standard c facultydetermined resources availability of course resources course resources when possible are provided online. standard a plus resources are organised according to course topics. standard b plus resources are created or co-created by students. availability of formative assessment resources links to online formative assessment are provided. standard a plus the assessments are accessed by students to better achieve learning outcomes. standard b plus assessments are personalised for students based on their competency to better achieve learning outcomes. activities selection of activities for online and faceto-face delivery both face-to-face and online activities are aligned to course learning outcomes. standard a plus the online activities offer flexible learning opportunities. standard b plus students have the opportunity to design activities or personalize them to help achieve their goals. technology to meet students’ diverse needs and preferences technology and/or media are used to deliver uniform activities for all students. technology and/or media are used to deliver differentiated teaching and learning activities. standard b plus learning activities are based on the unique learning needs and preferences of students. online activities to support student learning online tools to support learning activities are provided to students. standard a plus the online tools are used by students for interactive and collaborative learning activities. standard b plus students use online tools to further facilitate self-directed, interactive and collaborative learning activities. support support for students’ digital literacy skills question and answer forum for technical support is facilitated by the academic. standard a plus online support resources are provided. standard b plus students’ technical capability is determined and personalised training and support are provided based on their needs. opportunity for interaction both face-to-face and online interaction opportunities between students and the academic are provided standard a plus a range of online interaction opportunities between peers are provided. level b plus the online interaction opportunities are based on the diverse needs of students. r e se a rch in l e a rn in g t e ch n o lo g y c ita tio n : r e se a rc h in l e a rn in g t e c h n o lo g y 2 0 1 5 , 2 3 : 2 8 4 5 1 h ttp ://d x .d o i.o rg /1 0 .3 4 0 2 /rlt.v 2 3 .2 8 4 5 1 5 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 table 2 (continued ) standards criteria standard a standard b standard c facultydetermined feedback to support learning online technologies are used to provide feedback on student progress. standard a plus a range of online feedback methods are used to provide feedback throughout the semester (e.g. audio/video, overall written, or in-text comments). level b plus there is opportunity for students to receive peer feedback (e.g. comments on blogs or discussion forums or reviewing each other’s work) assessment (evaluation) design of assessment tasks where appropriate, technology/media is used to submit assessment tasks. standard a plus students have opportunity to use online tools (e.g. wikis, blogs) to complete assessment tasks. standard b plus students can use different online tools to complete their assessment that best demonstrates their learning (e.g. recorded presentation, wiki, blog) student access to their progress and achievement technology (e.g. online assessment marks, quizzes) is used to inform students about their learning achievement. standard a plus students have continuous access to an online grade book or progress bar to monitor activities completed or marks to date. standard b plus students are encouraged to identify their own learning goals to track back against their progress. self and peer assessment both face-to-face and online technologies are used to engage students in self and peer assessment activities. standard a plus standardsbased rubrics are used by students to self-assess and peer-assess. standard b plus continuous self-assessment is encouraged to help students achieve their own intended goals. n . m irria h i e t a l. 6(p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) c ita tio n : r e se a rc h in l e a rn in g t e c h n o lo g y 2 0 1 5 , 2 3 : 2 8 4 5 1 h ttp ://d x .d o i.o rg /1 0 .3 4 0 2 /rlt.v 2 3 .2 8 4 5 1 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 focus group participants, p1fg2, describes how students are actively involved in creating and designing resources: we use students a lot for developing our resources, they are constantly designing our resources, and most of our resources are designed by students themselves. (2) availability of formative assessment resources. formative assessment activities have significant impact on improving student learning (hattie 2008) providing opportunities for students to practice, determine their level of performance, and set their own learning goals. such assessments may include self-tests, peer review, and adaptive lessons that can be facilitated through online technologies supporting students’ achievement (yam and rossini 2011). activities both face-to-face and online activities should facilitate learning experiences that help students achieve the intended learning outcomes of the course. (1) selection of activities for online and face-to-face delivery. once the availability of resources has been determined, focus should turn to the selection of activities. matching the design of online and face-to-face activities to the learning outcomes enables the best possible blend (garrison and kanuka 2004) as focus group participant, p1fg2, discusses: . . . it’s really looking at the outcomes and how the technology can support students developing those outcomes, and there might be particular issues such as you might have students at a distance. that means you need technology to enable things. it might be that you need to be able to put students into the field, in such a virtual way, through simulations. the thoughtful process of designing face-to-face and online activities will ensure that the activities in both mediums support one another rather than overlap. to achieve this, the two environments should be integrated, as focus group participant, p2fg1, emphasises, ‘how you integrate what you do with one with the other so that then the two are not separate things, you are not repeating, you are not replacing, but you are integrating’. (2) technology to meet students’ diverse needs and preferences. the flexibility of bl helps meet students’ educational needs especially for those who have work and other responsibilities (wingard 2004). integrating online technologies allows students to have greater flexibility since learning does not have to be tied to a particular time or physical space (garrison and kanuka 2004). further, the use of an online platform could address students’ learning preferences (mohr, holtbrügge, and berg 2011). identifying and addressing students’ particular needs have informed many of the higher standards on the proposed framework (standard c on table 2). focus group participant, p2fg1, explains that students’ preference for learning, together with other factors such as distance to the campus impacts attendance at face-to-face sessions: i think different students will approach it differently as well, if they’ve got the options. i mean, some students would still love coming to lecture, whereas some students shy away from coming to lectures but they listen research in learning technology citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 to their [video or audio] recordings . . . so i think, as well as discipline specific context, there is individual student context. (3) online activities to support student learning. the features and functionality afforded by various technologies can significantly impact the choice of learning activities and teaching strategies adopted. selection of technology and media should consider the appropriateness for achieving the learning outcomes within given contexts (holden and westfall 2010) as summarised by focus group participant, p1fg2: . . . and this is part of digital literacy too, knowing that different tools work differently and have different affordances. so, it’s being able to take a step back and evaluate the new opportunities. in addition, a number of online tools are readily available through learning management systems or through the internet due to the advancement of web-based technologies such as wikis, blogs, discussion forums, simulations, and synchronous webinars that can facilitate interactive and collaborative learning enhancing student engagement (chen, lambert, and guidry 2010). this informs the higher standards for this criterion (standards b and c on table 2). support these criteria focus on supporting students’ digital literacy skills, responding to queries, and providing on-going feedback. (1) support for students’ digital literacy skills. we cannot assume that all students are familiar with existing learning tools as focus group participant, p3fg2, argues: . . . yes, some of them are, but this is an assumption . . . academics assume that students are so much more advanced in terms of digital literacy . . . but the research shows that they are literate in pockets, they are really good at using facebook perhaps, or using their phones to send sms messages but as far as using technologies for learning or for professional purposes maybe they’re not so literate. thus, strong support for students’ development of digital literacy is necessary to ensure their effective and meaningful engagement with bl (garrison and kanuka 2004). kennedy et al. (2010) argue that students’ technological experiences and expectations need to be managed. focus group participant, p2fg1, further elaborates, ‘. . . what support is available for students . . . what training and development is there, these are the things that probably will make a better blended environment’. further, apart from the use of technology, students need to be trained in their role as bl learners (cheung and hew 2011). hence, as the higher standard for this criterion (standard c on table 2) states, students’ technical capabilities should be identified in order for the appropriate training resources to be provided for using the online tools effectively. (2) opportunity for interaction. wagner (2006) identifies interaction as one essential element of bl, which cuts across the two modes of delivery. in both online and face-to-face learning, interactions play significant roles as students construct their own learning. interactions in bl can take the form of transactions, as outcomes or as experience. however, the sustainability and level of n. mirriahi et al. 8 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 interaction in online environments can be challenging (wang 2010). hence, bl course design should make use of a variety of interaction tools to provide a range of opportunities for students to interact with each other, academics, and other external experts (dziuban et al. 2006). (3) feedback to support learning. central to enhancing students’ learning is the on-going provision of feedback (hattie and timperley 2007). in an online environment, technology can support the delivery of timely and effective feedback. one benefit of online feedback is that it can contain specific links to learning resources (velan et al. 2002) that are specific to the needs of each student. similarly, with the affordances of technology for assessment, feedback can be delivered through a combination of mediums: audio, video, rubrics, in-text comments, and overall written feedback, which inform the higher standards for this criterion (standards b and c on table 2). in addition, technology can help facilitate peer feedback providing opportunity for students to enhance their work (li, lu, and steckelberg 2010). such peer feedback can be facilitated through comments on blogs or discussion forums or peer review tools. this informs the third standard for this criterion (standard c on table 2). assessment (evaluation) these criteria focus on the design of assessment tasks, student access, and self and peer assessment. (1) design of assessment tasks. in a recent study by mirriahi and alonzo (2015), it was discovered that students want to engage more in all aspects of assessment using online technologies. the use of technology in assessment allows students to engage in several authentic and complex tasks (means et al. 1993). for instance, in clinical teaching, technology can facilitate learning through patient simulations mitigating ethical challenges often faced with practicing with real patients (lateef 2010). focus group participant, p1fg1, describes how technology can assist with innovative assessment for students: . . . there are new opportunities for assessment online . . . looking at sort of deductive assessment for students, we can do that quite well in an online environment that we can’t do on paper. equally important is the use of technology to develop differentiated assessment and provide students with the option to choose how to best demonstrate their learning through a variety of assessment submission types such as videos, posters, presentations, and essays. students’ autonomy to choose how they demonstrate their learning informs the higher standard of this criterion (standards b and c on table 2). (2) student access to their progress and achievement. students’ access to their progress and achievement enables them to monitor their learning development over time, which consequently enhances their self-regulation (sadler 1989). for example, progress bars and grade books in online learning environments can help students monitor the number of activities or assessments they have completed, and their scores along the way. the ability for students to have continued access to monitor their progress and achievement informs the higher research in learning technology citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 standards for this criterion (standards b and c on table 2). further, the provision for students to identify their own learning goals and track against their achievement of them informs the third standard for this criterion (standard c on table 2). (3) use of self and peer assessment. the use of self and peer assessment to improve learning is supported by a range of literature (dochy, segers, and sluijsmans 1999). brown, smith, and henderson (2007) have revealed that more experienced learners have stronger perceptions that collaborative work is more effective than traditional assessment practices. integrating peer assessment in bl course design should help develop students’ skills ‘such as critical reflection, listening to and acting on feedback, sensitively assessing and providing feedback on the works of others’ (liu and carless 2006, p. 289). further, the use of standards-based rubrics for self and peer assessment provides an objective approach to help students monitor and gauge their achievement of the intended learning outcomes prior to submitting their work. in the case of peer assessment, such rubrics can provide an objective way for peers to offer feedback that is focused on criteria and standards (sadler and good 2006). the organisation of standards the academic digital literacy required in bl environments can be seen as a hierarchy of skills and attributes (bennett 2014). underpinned by this view, the standards established in the proposed framework include three pre-defined levels building upon one another (standards a through c) and with an additional fourth standard (faculty-determined standard). this highest level of standard allows individual faculties to account for the breadth and diversity of bl and online practices across an institution. as one of the focus group participants, p2fg1, explains: . . . what works in one faculty is not going work for another faculty, the science and the practical subjects in medicine, have face-to-face lab work, yes you can have simulations, and yes you can have all the ipad apps you’ve got, but the physical working with chemicals or whatever it is, that’s a whole different experience to something like journalism where a lot of the work will now be in the online environment . . .. the first level or standard requires academics to integrate technology to offer online teaching and learning activities. technology use is evident but it is more academiccontrolled. however, the second level or standard requires academics to use technology to deliver flexible learning opportunities, which are constructive, authentic and collaborative, while the third level or standard moves towards a more enriched and higher degree of student collaboration. in addition, in this third level, technology allows students to have greater participation in terms of creating the resources allowing them to learn with technology rather than just from technology (churchill et al. 2013). this is consistent with darling-hammond, zielezinski, and goldman’s (2014) view that ‘the curriculum and instructional plan should enable students to create content as well as to learn material’ (p. 15). table 2 presents the complete proposed bl framework. n. mirriahi et al. 10 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 implications for academic development to realise its full potential in transforming academic practice, the following should be considered when using the bl framework: (1) the framework is meant to be a formative tool, and hence should not be used rigidly to define bl practice. rather, all academics are encouraged to further innovate beyond the described standards. the emerging nature and affordances of technologies, together with the various characteristics of the student cohort, should be considered to offer more flexible and effective learning experiences. (2) the framework is intended to be a self-assessment instrument for academic staff to identify their current level of practice to determine areas of expertise and areas that can be further developed. academic development programs should align with the criteria and standards in order to provide the necessary support and resources for academics to enhance their practice and move towards the higher standards. (3) the framework should form the discussion around bl across a higher education institution. this can help ensure that all stakeholders at a system level have a common understanding and expectation to provide the necessary support mechanisms such as technology infrastructure, resources, and support for academics to redesign their courses to bl. implications for research there are several implications for research related to the proposed framework. first, the framework should be trialled with academic staff across different disciplines and at different stages of their bl practice to identify how they engage with it and whether they find it useful for gauging their current bl practice and identifying enhancements. while the authors intend to pilot the framework at their institution, they hope that by introducing it as a proposed conceptual framework largely underpinned by the literature, they provide an opportunity for other researchers and educational developers to trial the framework with their own staff or adapt it for their own contexts. second, as the number of focus group participants was limited, future work should include gathering the views and perceptions of both academics and students in order to further refine the criteria and standards. the views of students, in particular, would help identify whether the standards associated with their engagement of online technologies would support their learning. finally, future research can include the inclusion of professional development resources associated with the framework’s standards and criteria in order to investigate how academics engage with them and how academic development units can best support the development of bl practice across their institutions. conclusion the proposed bl framework, theoretically informed by the extant literature and supported by qualitative data collected from focus groups, addresses the three issues identified in the bl literature: (1) lack of academics’ digital fluency, (2) multiple definitions of bl, and (3) lack of standards in existing bl frameworks. first, as a self-assessment research in learning technology citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 instrument, the framework can help academic staff identify their current standard of bl practice and the changes required to progress to the higher standards. consequently, this could lead academics to enhance their practice or seek professional development, both, which will develop their digital literacy skills addressing the first issue. second, a consolidated definition of bl based on the literature forms the basis of the framework and can be adapted by institutions to inform their own bl definitions and policies. third, the proposed framework provides three levels of descriptive standards with a fourth level for faculties to identify for themselves to address their own disciplinary and contextual needs. references bath, d. & bourke, j. (2010) getting started with blended learning, griffith university, australia, [online] available at: https://www.griffith.edu.au/__data/assets/pdf_file/0004/ 267178/getting_started_with_blended_learning_guide.pdf bennett, l. (2014) ‘learning from the early adopters: developing the digital practitioner’, research in learning technology, vol. 22, 21453, doi: http://dx.doi.org/10.3402/rlt.v22. 21453 bleed, r. (2001) ‘a hybrid campus for the new millennium’, educause review, jan/feb, pp. 17�24. available at: http://er.educause.edu/~/media/files/article-downloads/erm0110.pdf brown, f. g. (1983) principles of educational design and psychological testing, 3rd edn, hoilt, rinehart, and winston, new york. brown, g., smith, t. & henderson, t. (2007) ‘student perceptions of assessment efficacy in online and blended classes’, in blended learning: research perspectives, eds a. g. picciano and c. dziuban, sloan consortium, needham, ma, pp. 145�160. bruner, j. (1996) the culture of education, harvard university press, cambridge, ma. chen, p.-s. d., lambert, a. d. & guidry, k. r. (2010) ‘engaging online learners: the impact of web-based learning technology on college student engagement’, computers and education, vol. 54, no. 4, pp. 1222�1232. cheung, w. s. & hew, k. f. (2011) ‘design and evaluation of two blended learning approaches: lessons learned’, australasian journal of educational technology, vol. 27, no. 8, pp. 1319�1337. churchill, d., king, m., webster, b. & fox, b. (2013) ‘integrating learning design, interactivity, and technology’, in proceedings of the 30th ascilite conference, sydney, pp. 139�143. available at: http://www.ascilite.org/conferences/sydney13/program/papers/churchill.pdf darling-hammond, l., zielezinski, m. b. & goldman, s. (2014) using technology to support at-risk students’ learning, alliance for excellent education and stanford center for opportunity policy in education, [online] available at: https://edpolicy.stanford.edu/sites/ default/files/scope-pub-using-technology-report.pdf dick, w., carey, l. & carey, j. o. (2009) the systematic design of instruction. 7th edn, pearson merrill, upper saddle river, nj. dochy, f., segers, m. & sluijsmans, d. (1999) ‘the use of self-, peer and co-assessment in higher education: a review’, studies in higher education, vol. 24, no. 3, pp. 331�350. dziuban, c., et al., (2006) ‘blended learning enters the mainstream’, in the handbook of blended learning, eds c. j. bonk & c. graham, wiley, san francisco, ca, pp. 195�208. friedenberg, l. (1995) psychological testing: design, analysis and use, allyn and bacon, boston, ma. gaeta, m., orciuoli, f. & ritrovato, p. (2009) ‘advanced ontology management system for personalised e-learning’, knowledge-based systems, vol. 22, no. 4, pp. 292�301. garrison, d. r. & kanuka, h. (2004) ‘blended learning: uncovering its transformative potential in higher education’, the internet and higher education, vol. 7, no. 2, pp. 95�105. griffin, p. (2000) competency based assessment of higher order competencies, nsw state conference of the australian council for educational administration, mudgee, nsw. n. mirriahi et al. 12 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 https://www.griffith.edu.au/__data/assets/pdf_file/0004/267178/getting_started_with_blended_learning_guide.pdf https://www.griffith.edu.au/__data/assets/pdf_file/0004/267178/getting_started_with_blended_learning_guide.pdf http://dx.doi.org/10.3402/rlt.v22.21453 http://dx.doi.org/10.3402/rlt.v22.21453 http://er.educause.edu/~/media/files/article-downloads/erm0110.pdf http://www.ascilite.org/conferences/sydney13/program/papers/churchill.pdf https://edpolicy.stanford.edu/sites/default/files/scope-pub-using-technology-report.pdf https://edpolicy.stanford.edu/sites/default/files/scope-pub-using-technology-report.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 hargreaves, d. (2006) a new shape for schooling, specialist schools and academies trust, london. hattie, j. (2008) visible learning: a synthesis of over 800 meta-analyses relating to achievement, routledge, hoboken. hattie, j. & timperley, h. (2007) ‘the power of feedback’, review of educational research, vol. 77, no. 1, pp. 81�112. hinrichsen, j. & coombs, a. (2013) ‘the five resources of critical digital literacy: a framework for curriculum integration’, research in learning technology, vol. 21, 21334, doi: http://dx. doi.org/10.3402/rlt.v21.21334 holden, j. & westfall, p. (2010) an instructional media selection guide for distance learning: implications for blended learning, 2nd edn, united states distance learning association, [online] available at: https://www.usdla.org/wp-content/uploads/2015/05/aimsgdl_2nd_ ed_styled_010311.pdf inbar-lourie, o. (2008) ‘constructing a language assessment knowledge base: a focus on language assessment courses’, language testing, vol. 25, no. 3, pp. 385�402. johnson, l., et al., (2014) horizon report: 2014 higher education, new media consortium, austin, tx. kennedy, g., et al., (2010) ‘beyond natives and immigrants: exploring types of net generation students’, journal of computer assisted learning, vol. 26, no. 5, pp. 332�343. lateef, f. (2010) ‘simulation-based learning: just like the real thing’, journal of emergencies, trauma, and shock, vol. 3, no. 4, pp. 348�352. li, l., lu, x. & steckelberg, a. (2010) ‘assessor or assessee: how student learning improves by giving and receiving feedback’, british journal of educational technology, vol. 41, no. 3, pp. 525�536. liu, n. f. & carless, d. (2006) ‘peer feedback: the learning element of peer assessment’, teaching in higher education, vol. 11, no. 3, pp. 279�290. means, b., et al., (1993) using technology to support education reform, u.s. government printing office, washington, dc. mohr, a. t., holtbrügge, d. & berg, n. (2011) ‘learning style preferences and the perceived usefulness of e-learning’, teaching in higher education, vol. 17, no. 3, pp. 309�322. mirriahi, n. & alonzo, d. (2015) ‘shedding light on students’ technology preferences: implications for academic development’, journal of university teaching & learning practice, vol. 12, no. 1, pp. 1�14. doi: http://ro.uow.edu.au/jutlp/vol12/iss1/6/ oliver, m. & trigwell, k. (2005) ‘can learning be redeemed?’, e-learning, vol. 2, no. 1, pp. 17�26. oliver, r. (2003) ‘exploring benchmarks and standards for assuring quality online teaching and learning in higher education’, proceedings of the 16th open and distance learning association of australia biennial forum, canberra, act, pp. 79�90. palak, d. & walls, r.t. (2009) ‘teachers’ belief and technology practices: a mixedmethods approach’, journal of research on technology in education, vol. 41, no. 4, pp. 417�441. parsell, m. & collaborators. (2013) standards online education framework, [online] available at: http://www.onlinestandards.net/standards/ procter, c. (2003) ‘blended leaning in practice’, proceedings of the education in a changing environment conference, [online] available at: http://usir.salford.ac.uk/27428/2/blended learninginpractice.pdf reed, p. (2014) ‘staff experience and attitudes towards technology-enhanced learning initiatives in one faculty of health and life sciences’, research in learning technology, vol. 22, 22770, doi: http://dx.doi.org/10.3402/rlt.v22.22770 sadler, d. r. (1989). ‘formative assessment and the design of instructional systems’. instructional science, vol. 18, no. 2, pp. 119�144. sadler, p. & good, d. (2006) ‘the impact of selfand peer-grading on student learning’, educational assessment, vol. 11, no. 1, pp. 1�31. smythe, m. (2012) ‘toward a framework for evaluating blended learning’, future challenges, sustainable futures, proceedings ascilite, wellington, new zealand, pp. 854�858. torrisi-steele, g. & drew, s. (2013) ‘the literature landscape of blended learning in higher education: the need for better understanding of academic blended practice’, international journal for academic development, vol. 18, no. 4, pp. 371�383. research in learning technology citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 13 (page number not for citation purpose) http://dx.doi.org/10.3402/rlt.v21.21334 http://dx.doi.org/10.3402/rlt.v21.21334 https://www.usdla.org/wp-content/uploads/2015/05/aimsgdl_2nd_ed_styled_010311.pdf https://www.usdla.org/wp-content/uploads/2015/05/aimsgdl_2nd_ed_styled_010311.pdf http://ro.uow.edu.au/jutlp/vol12/iss1/6/ http://www.onlinestandards.net/standards/ http://usir.salford.ac.uk/27428/2/blendedlearninginpractice.pdf http://usir.salford.ac.uk/27428/2/blendedlearninginpractice.pdf http://dx.doi.org/10.3402/rlt.v22.22770 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 velan, g. m., et al., (2002) ‘web-based assessments in pathology with question mark perception’, pathology, vol. 34, no. 3, pp. 282�284. vygotsky, l. (1986) thought and language, mit press, boston, ma. wagner, e. d. (2006) ‘on designing interaction experiences for the next generation of blended learning’, in the handbook of blended learning, eds c. j. bonk & c. graham, wiley, san francisco, ca, pp. 41�55. wang, m. (2010) ‘online collaboration and offline interaction between students using asynchronous tools in blended learning’. australasian journal of educational technology, vol. 26, no. 6, pp. 830�846. wingard, r. g. (2004) ‘classroom teaching changes in web-based enhanced courses: a multiinstitutional study’, educause quarterly, vol. 27, no. 1, pp. 26�35. wolf, k. & stevens, e. (2007) ‘the role of rubrics in advancing and assessing student learning’, the journal of effective teaching, vol. 7, no. 1, pp. 3�14. yam, s. & rossini, p. (2011) ‘online learning and blended learning: which is more effective’, 17th pacific rim real estate society conference, gold coast, australia, [online] available at: http://www.prres.net/papers/yam_online_learning_and_blended_learning.pdf n. mirriahi et al. 14 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28451 http://dx.doi.org/10.3402/rlt.v23.28451 http://www.prres.net/papers/yam_online_learning_and_blended_learning.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28451 http://dx.doi.org/10.3402/rlt.v23.28451 comment: the coming tornado? ray mcaleese centre for combined studies, heriot-watt university. email: r.mcaleese@hw.ac.uk an extended review of john s. daniel's mega-universities and the knowledge media: technology strategies for higher education (london: kogan page, 1996; 212 pages, isbn: 0-7494-2119-3). i will start with a declaration of interest and prejudice. i have been for some thirty years a 'learning technologist', or what used to be unfashionably called an 'educational technologist'. i have observed and read books, chapters and papers by john daniel, the author of this work, for about twenty years, since his days in concordia university, montreal. professor daniel is without peer as a judge of continuing, open and distance education. as vice-chancellor of the open university, he has seen his institution grow to a leading position in the uk open-learning market with hundreds of courses offered under its inimitable course-design system. he has ensured that the open university is a beacon of what is best in innovative teaching and programme delivery. he has steered an adept course, gaining widespread international approval for the prototypical university of the air. his views are canvassed by the great and the good. one can discern his hand in the recent dealing (1997) recommendations. why, then, do i feel uncomfortable with this book? i started this review almost a year ago, just in time for the publication of the dealing report in the summer of 1997.1 thought, as i read more and more of the book, that i wanted to say yes. but i also wanted to argue. i wanted to contest, to say: 'it's not like that. i don't like the way you describe what you claim we do.' the thrust of daniel's argument is that there is a global crisis in the old world of higher education. universities have a window of opportunity to take advantage of developments in information and communications technology. knowledge media have become a commodity that is attractive to entrepreneurs and students alike. sufficient is known of competitive advantage between organizations to draw up strategic guidelines for innovative institutional plans. further, there are tried and tested implementations strategies for 68 alt-j volume 6 number 2 technology-based teaching drawn from organizational change and development. all these developments will allow the old-world universities to become, like the open university, successful mega-universities. the argument is simple. the evidence is well documented (the book's origin is in a doctoral dissertation written during a sabbatical leave). the writing is easy. so what, again, is it that makes this book uncomfortable reading for me? if one takes daniel's logical analysis of competitive advantage, the reason is soon apparent. he starts his argument with the premise that rational economic analysis is appropriate for the output of universities. he asserts that students are outputs and the buyer is the state. further, and by implication, students themselves buy universities as consumers. some (for example, hobbs, 1998) have doubted if such a rational cost-sensitive demand exists, and if it does, how it is characterized. interestingly, this recursive consumerism is currently exhibiting a decline in demand because the price of the product has just been gazumped by the government. the introduction of the £1,000 fee is highlighting the complex supply and demand system that operates. this effect suggests that universities may have a product ('knowledge') and that students will value and pay for it, just as they would for any other commodity. daniel's market-based economic analysis goes on to suggest that students, in turn, acquire 'skills' through just-in-time delivery of courses in what tom king et al. in another irritating book, so inelegantly call 'learn-places' (ford et al, 1996; see reviews of this book in alt-j, 5 (2), 64-74). the fact is that student demand has crossed a cusp. perhaps there really are market forces at work. not only is there a demographic-led increase, but perhaps also a rational market-led decrease. in 1998, the most recent figures from ucas suggest that there is a definite decline in the number of applicants for university places in uk. daniel is able to conclude that universities should focus on five core activities, namely: 'inbound logistics, operations, outbound logistics, marketing and sales' (p.73). the issue i sense here is the use of a commodity-oriented framework that seems and feels wrong for my understanding of a university. my orientation to universities is more in keeping with bruce truscot's (1943) much maligned red brick university, or the collegiality of the pre-robbins community of scholars, rather than the consumerism of the modern 'mega-university' system. to me, the language of description and analysis seems to ignore what universities do best. i feel that universities should concentrate on educating young people. it is as simple as that. universities should provide a scholarly and rigorous framework for the creation and adoption of scientific knowledge. universities, after all, consist of scholar-teachers who engage with learners in a shared activity teaching and learning based on post-newman scholarship. now, it is logical and feasible to analyse any system using a different framework. such a device is often used by social scientists who suggest, for example, an epidemiological model for the adoption of new ideas. it is as if ideas propagate, disseminate and are curtailed in a way that is similar to the growth of a biological epidemic. they.use this socio-medical framework to suggest interesting new insights. however, unlike my reading of daniel, they do not go on from a useful analytical framework to a deterministic projection of how things should be. normative determinism, based on an interesting but inappropriate framework is a characteristic i sense in daniel's book. the analysis of the 'value chain' logical and appropriate to such as the electronics industry seems a wholly inappropriate way to understand how universities can make the 69 ray mcaleese comment the coming tornado? most of technological opportunities to facilitate learning and scholarship. it does not seem like that to me. the reality of being a student or a teacher in higher education may of course be different from the noble aspirations i allude to above. teaching is a job just like many others in a service industry. there are clients who have expectations, and the teachers meet these expectations in a professional way. they share their expertise through well-tried teaching methods, while at the same time keeping abreast of their subject discipline and contributing to the success of their institution through research and consultancy. likewise, students have a need to acquire paper qualifications that certify them as being competent and employable. their 'graduateness', while comprising both transferable skills and subject-based skills, is judged by their ability to perform traditional tests of recall, evaluation and communication. and the reality, as seen from the shop floor, is of many teaching staff and researchers disaffected with the inexorable drive to research output and efficiency. the workforce is under-skilled and sometimes untrained. its ability to make the most of communications and information technology hangs on initiatives that aim to disseminate best practice, and on painfully short induction courses to the complexities of 'teaching over the web' and 'automatic assessment using self-marked tests'. most teachers want to help learners understand what they need in order to be successful, not to acquire a new set of professional skills for which they will receive no direct reward. the experience of being a student is no less confusing. many students up to a third in some instances come to university with almost no communications or it skills. they have experienced traditional or neo-traditional teaching methods in schools, and their only ability to reflect on their competencies has been in preparing a short cv for a summer job. when they are invited to take responsibility for their own learning and to seek out information for themselves in order to satisfy the requirements of distant teachers, they cry for help and consume a disproportionate amount of tutoring time. often, the teacher and the taught are both on a voyage of discovery into the complexities of a computersupported collaborative learning environment. the reality is that while there is much to be said for communications and information technology in higher education, there are few listening and few with the minimum set of skills to achieve 'take-off. if this were not true, why should the funding councils support initiatives such as tltp and ltdi? there is a crisis with under-skilled staff and students. there are solutions to problems that do not exist and no solutions for problems that do exist. a reasonable view might be to concentrate our efforts on a better understanding of the process of learning using the existing technologies before starting on a head-long rush to re-skill, re-learn and reorganize. there is a respectable argument that suggests: do what you do best well, before your do hew things. i only wish john daniel had placed more weight in chapters 2 and 3 on the book by his colleague diana laurillard (laurillard, 1993). when she asked what students bring to learning, and examined the 'complexity of coming to know', she touched on the essential ingredient for a successful higher-education business: an understanding of the customer. in a chapter entitled making technology attractive (chapter 6), daniel uses moore's (1995) 'technology life-cycle' to argue the case for capitalizing on technology to 'dominate the niche' (p. 90). daniel asserts: 'the organisation that develops whole products successfully in a series of niches may create a tornado as the mass market takes off' (p. 93). the 70 alt-j volume 6 number 2 tornado concept an unstoppable advance through synergistic alliances allows, as daniel paraphrases moore, 'large numbers of customers to enter the market simultaneously, swamping the supply system and creating a frenzy' (p. 94). the implication is clear for those who wish to see it. technology is going to support learners and institutions, but only those who move forward using the technology paradigm will survive. only those institutions who see their competitive advantage being served by buying into distance learning, open systems and communications and information technology will be the real players. it is almost like the tornado that blows the smaller football clubs away when the premier clubs strike deals with the media to take them to a state of higher turnover and profitability. the comparison between soccer and scholarship is not only close; it offers an opportunity to consider the impact of competitive advantage on those who still enjoy standing on the terraces supporting local heroes. now, this is where one can see the full impact of daniel's' argument. in the real world, technology-supported infra-structures are differentiating between those that are keeping up and those that will have to change or be left behind. the higher-education system is infected with the competitive spirit. attracting punters (sic) is now an every-day job for each and every one of us. to offer better opportunities to learn through metropolitan area networks; to share applications over distributed systems, and to be a member of a virtual community of learners is what we are told should be attracting more students . john daniel may be right, but i do not like it. i feel exposed. i may have to change my cost stance. i may have to reflect on what i should do, not what i like doing. turning to the core issue, 'knowledge-media', may present an attractive proposition as far as learning technology is concerned. daniel draws heavily on his well-informed open university colleagues diana laurillard and mark eisenstadt. it is eisenstadt's 'knowledgemedia' that offers such an attractive term in the title of the book. the observed convergence of telecommunications and computing, along with the cognitive and learning sciences, has suggested this term to eisenstadt. the key assumption by eisenstadt is the belief that the increased 'bandwidth' of what dealing was later to call the communications and information technologies, gives rise to qualitative differences in what can be learned and what is learned. because it is possible to push terabytes out over a network, it is possible now to create new virtual learning environments so the argument goes. no longer do students have to wait at their modems for a trickle of bytes at 1200 baud to download the tutor's comments on their assignments. students can make the most of distant tutors who can see, hear and almost touch the cyber-learners. of course, there is a lot of sense in this chapter (chapter 7). diana laurillard who herself made a highly significant contribution to the dearing committee and report has previously offered a 'conversational framework' for understanding how learning is facilitated. this conversation facilitates the social aspect of learning. learning, according to laurillard is discursive, adaptive, interactive and reflective. her analysis, first widely publicised in rethinking university teaching (laurillard, 1993) is clearly referred to by daniel, but i feel he does not make the most of her analysis. perhaps in a similar way, the conclusions of the macfarlane report (1992) paid lip-service to the thorough analysis of entwistle et al who provided the pedagogy for the report's intensely supportive learning environments (macfarlane, 1995). macfarlane (1992) himself placed a high premium on making learning 'more productive and efficient by harnessing modern technology' (p. 13). on a 71 ray mcaleese comment the conning tornado? larger canvas, daniel seems to prefer the artefacts that emanate from 'knowledge-media' to the cybernetic negotiation of understandings between teachers and students mediated through communications media. in summary, daniel concludes: 'universities need to do considerable development work if they are to derive full advantage from opportunities such as on-line tutoring and the internet' (p. 135). here, again, i can see the road ahead like . the bill gates vision. the potential to place the cart (the technology artefacts) before the horse (the instructional technology of laurillard's 'conversational system') is all too apparent. so what is it in this book that has made me read it, re-read it, think about it, talk about it, cite it, hate it? it is the logic of daniel's argument. i fear he is right. i may have been left behind. the world out there that allows me to support my students through my mail system while i write this passage at home. the ability of high-performance workstations to model complex process and allow learners to engage in their own learning process is here now. i suspect the real impact of john daniel's' book will be that it brings out in each of us in a cathartic exorcism the deepest, darkest feeling that we have been dinosaurs in a park, owned and run by the entrepreneurial knowledge-scientists. this reminds me of the way barbara falk, the grand lady of australian staff development in the 1970s, alluded to the role of the successful staff developer as being like 'a professional irritant'. the successful staff developer makes others 'scratch'. it is perhaps daniel's role in writing this book, from the authority of his long experience, to irritate the rest of us into new ways of seeing the world. if the reader wants to know: how to initiate change; what the new language of innovative teaching is; how to get one over on the next department/college/ system, then this book will be essential reading. if the reader wants to understand what is known about learning and scholarship under communications and information technology, there are pointers galore. if the reader wants to impress the senior-management team with an awareness of the next window of opportunity, this book is required reading. i still want to know more of the 'ethnographic' studies daniel refers to that explicate the process of learning. i want to know what will happen when the current wave of communications and information technology is superseded by new and as yet unthought of artefacts. i want to know how i am to compete for attention as a learning technologist, without selling cosy scholarly principles to the grim technology reaper. but before i change my name to faust, i have to acknowledge a debt to professor daniel. i journeyed through his book and wriggled on every page. yet i have now read it cover to cover on three occasions. i feel i will do so again. for more down-to-earth readers, the book has eight well written chapters, a fascinating set of biographies of existing 'mega-universities' in the appendix, and a useful bibliography to set them off verifying, corroborating, testing daniel's assertions. may their journey be interesting. references dearing et al (1997), higher education in the learning society: report of the national committee of inquiry into higher education, london: hmso and ncihe publications. ford, p. et al (1996), managing change in higher education: a learning environment architecture, london: open university press. 72 alt-j volume 6 number 2 hobbs, p. (1998), 'an analysis of teaching methods, costs and student preferences', active learning, 7, 8-10. laurillard, d. (1993) rethinking university teaching, london: routledge. macfarlane report (1992), teaching and learning in an expanding higher education system, committee of scottish university principals, isbn: 0-9519377-1-5. macfarlane, a.g.j. (1995) 'future pattern of teaching and learning' in schuller, t. (ed), the changing university?, london: open university press. moore, g.a. (1995), inside the tornado, new york: harper business. porter, m.e. (1985), competitive advantage: creating and sustaining superior performance, new york: free press. truscot, b. (1943) [pseudonym of e.a. peters], red brick university, london: faber & faber. 73 comparing distance education and conventional education: observations from a comparative study of post-registration nurses tim duffy,* lain gilbert,* david kennedy* and poon wai kwong** *university of paisley **online education, hong kong email: david.kennedy@paisley.ac.uk this retrospective study presents a comparison of assessment results achieved by distancelearning students and classroom-based students undertaking the same module in a degree course. the purpose of the comparison is to provide some objective measurement of the quality of distance education in relation to conventional classroom-based education. the authors have selected three groups of students, who have all undertaken the same module in the b. sc health studies degree programme offered by the university of paisley. one group (in paisley) undertook their studies by means of conventional classroom-based education, the second (in hong kong) by supported distance learning with face-to-face contact in the form of tutorials, and the third (in a geographically dispersed group in the united kingdom and other countries) by supported distance learning with no face-to-face contact. the results obtained by these three groups of students were analysed. because of the differences in the size of the groups, the kruskalwallis 1way anova test was applied to validate the face value findings. the authors include findings from the literature comparing distance education with conventional education and from cross-cultural studies to present their data in context. analysis of the assessment results showed that students from all three groups were successful in their studies, but the students studying by distance learning obtained significantly higher end-of-module results than their classroom-based colleagues. this latter finding reflects the conclusion that other investigators have reached in their discussion the authors identify educational, cultural and personal factors that may help to explain their findings. a limitation in the study is that it concerns only one module in the degree programme. the research now moves on to comparing students who have undertaken the whole degree programme by the means described introduction in the united kingdom, the university with the highest number of students is a distance education university (hesa, 2000). the higher education sector can clearly deliver 70 alt-j volume 10 number i distance education. however, in the current ideological and financial drive towards increasing the distance education component of higher education, one of the key questions is this: is the quality of the distance education offered in higher education equivalent to the quality of its classroom-based education? this paper reports a study carried out in the university of paisley, which attempts to shed some light on this question. it makes a specific comparison of the results achieved by classroom-based students and distance-learning students undertaking the same degree course. all the students under comparison study the same module material and undertake identical assessments. but they study in different modes and in different locations in the world: some based on the campus in the town of paisley and undertaking the course by conventional classroom-based learning; others geographically dispersed with the majority living in hong kong and accessing the course by distance learning. one indicator of the quality of learning and teaching is the results that students achieve in assessable coursework. accordingly we have phrased our research question in the following terms: do post-registration nurses studying by distance learning achieve the same assessment results as colleagues undertaking the same module by conventional classroom-based study? comparing distance education and classroom education peterson (2001) states that scholars of distance learning fall into two main camps. the first camp argues that distance learning is an opportunity for those that could not usually attend university to participate in tertiary education; that it promotes lifelong learning; and that 'the experience of an online course can be as rich and fulfilling as the experience of a traditional course'. they argue that more diverse populations of students with different learning styles can benefit from online courses. the second camp argues that in distance learning the focus is on teaching rather than on learning, and that the claims of flexibility are not reflected in present distance learning courses. they argue that students are frustrated, not just by the feelings of isolation but by lack of feedback from their teachers and difficulty in understanding written instructions given to them (peterson, 2001). ng (2001) highlights a number of issues relating to problems of delivering distance learning. he noted that social cohesion with the group of students was not good and that communication between students was limited and focused mainly on personal rather than academic issues. several students in his study felt anxiety about using electronic (email) communication. edwards, hugo, cragg and peterson (1999) assert that 'research generally shows no significant difference in educational outcomes [that is, course achievement and satisfaction with instruction] between courses taught through distance education and those offered in the regular classroom'. such a conclusion is a typical one from the research looking at differences in educational outcomes associated with conventional classroom-based education and distance education. the issue of 'significant difference' is one that russell (1999 and 2002) and mcgreal (2002) have both tackled by assembling annotated bibliographies of comparative research studies. the authors of this present study have carried out a meta-analysis of thirty-seven studies that specifically deal with online education and which are reported in either russell's no 71 tim duffy et a! comparing distance education and conventional education significant difference or mcgreal's significant difference annotated bibliographies. the meta-analysis is restricted to research reported in the decade 1992-2002. the findings from this meta-analysis are presented in table 1. this meta-analysis shows that most studies have focused on academic performance as their means of comparing the effectiveness of online distance education with conventional classroom-based education. the majority (16 studies out of 31) have found that there is no significant difference in the academic performance of online students when compared with on-campus students. however, a significant number of studies (13 out of 31) find that students in online courses outperform conventional students in terms of academic performance measured by examinations. a majority of studies found that the quality of learning and the participation in learning is also higher in distance education. in the area of student satisfaction, the picture is reversed. a significant number of studies (four out of seven) report that the on-campus students are more satisfied with their courses than the online students. focus of the research reported academic performance measured by examinations, tests or grades student satisfaction with the course quality of learning and learning outcomes achieved student participation in the course retention rates pass rates faculty evaluation of the course number of studies 31 7 5 2 2 2 2 number of studies reporting no significant difference 16 3 2 0 0 0 1 number of studies finding in favour of online education 13 0 3 2 1 1 1 number of studies finding in favour of conventional education 2 4 0 0 1 1 0 table i: meta-analysis of research studies reported in the 'no significant difference' and 'significant difference' annotated bibliographies we might, then, summarize a decade of literature comparing the effectiveness of classroom-based education and distance education in the following terms: distance education involving online support is at least as effective as classroom-based education in terms of academic performance measured by examination, and in several cases more effective. it seems to promote a higher level of student participation. but it does not obtain such high student satisfaction ratings as classroom-based education. this summary needs to be qualified. in 1999, phipps and merisotis conducted a review of the contemporary literature on the effectiveness of distance learning for the american institute of higher education policy (ihep). their overall conclusion agrees with the meta-analysis presented above, although they report more positive studies on student 72 alt-j volume 10 number i satisfaction. but they add a caveat: 'the most important problem is that the overall quality of the original research is questionable and thereby renders many of the findings inconclusive' (phipps and merisotis, 1999: 18). one of the limitations of the research that they identify, for example, is that many of the studies focus on part of a course, rather than a whole educational programme. that is a charge that the authors of this paper recognize may be laid against their own study. educational culture in hong kong and scotland it is appropriate to set the study reported in this paper in its educational and cultural context. there are two main educational backgrounds to be aware of: the british (that is, the distinct scottish and english) educational backgrounds of the on-campus students and of the distance-learning students who are supported directly from the university with no face-to-face teacher contact; and the chinese educational background of the distance learning students who live in hong kong and who have some face-to-face contact with their teachers. because of its colonial history, the educational culture in hong kong is derived from two distinct traditions, chinese and english. the chinese tradition holds a 'deep reverence for learning, a predilection for the integration of knowledge and delight in formal reasoning' (university grants committee, 1996). the english tradition is concerned with a far more pragmatic approach to learning. both systems have moved away from elitist traditions toward more inclusive, mass higher education systems. li yixian (1992) observes that most students in hong kong study solely for the purpose of obtaining a degree and, consequently, to improve their chances in the labour market. this aligns with the assertion made by sumner (2000) that distance education is frequently 'instrumental' and 'serves the system'. the scottish dimension of the degree programme probably spans both the english and the chinese philosophies. in the report of the scottish committee to the national inquiry into higher education, garrick detected a 'strong belief in scotland in the value of education, partly for its own sake and partly to overcome economic or social disadvantage' (garrick, 1997). background to the study the university of paisley first offered its b.sc. in health studies in 1991. this course was specifically designed to meet the demand from health care personnel with a desire to upgrade their professional qualifications to degree status. it aimed to do this by offering a flexible, part-time mode of study while offering credit for prior accredited learning. to date more than one thousand students have graduated through the conventional on-campus delivery programme. these students have been health professionals, mostly nurses, working in the west and central belt of scotland. in 1995 further flexibility was built into the delivery when this programme was also made available by computer-supported distance learning. at the time of writing (january 2002), over 800 students have graduated through the distance-learning programme. those 73 tim duffy et a\ comparing distance education and conventional education students have been health professionals, mostly nurses, and the bulk of them are working in hong kong. with the advent of the new hong kong hospital authority in 1992, the bachelor's degree became the preferred route for a registered nurse to advance to the status of nursing officer. large numbers began to seek opportunities to further their studies. the majority of them preferred distance learning, from either australian or united kingdom universities. paisley university was the only uk university with a presence in hong kong that offered a degree in health studies. the main overall distinction we draw in this paper is between the conventional classroombased delivery and the distance-learning delivery. however, within the distance-learning delivery, there are two main variations in the support given to students. one version incorporates some face-to-face support from an on-site tutor. this is the pattern of delivery for our students in hong kong. the other version involves no face-to-face contact but relies solely on computer-mediated support by teachers based in the university. this is the pattern of delivery for our students who are dispersed throughout the united kingdom or situated in norway or switzerland. in summary, the three forms of course delivery are: 1. on-campus delivery, in conventional classroom-based format; 2. supported distance learning using networked learning, with optional face-to-face contact; 3. supported distance learning using networked learning, with no face-to-face contact. this paper focuses on one module within the b.sc. health studies programme 'professional practice and teaching'. this is a module that enables health professionals to study, and apply in practice, educational principles which are applicable in their clinical practice when teaching patients or junior colleagues. we have selected this module because all the students in the groups that we will consider have undertaken the same generic assignment. we can use the mean scores achieved by each group as an outcome measure to make reliable comparisons between the groups. the different forms of course delivery this section gives an outline of the three modes of delivery of the module 'professional practice and teaching' with a particular focus on the following aspects: • the actual mode of delivery; • the processes of student support; and • the assessments which the students undertake. mode of delivery the conventional on-campus delivery of the module is taught over a period of twelve weeks (one semester). the class meets for two hours one evening per week. the main approach to teaching the module is the modified lecture. within the lecture there are exercises focused on the development of specific teaching skills. the distance-learning students receive core material in the form of printed booklets. they also receive that same material as an electronic document, in hypertext format. it is 74 alt-j volume 10 number i important to note that the course content for the distance-learning students is identical to the course content for the classroom-based students. the distance-learning version of the course is a development of the lecture-based version, and is not an alternative course. the distancelearning material incorporates interactive features, exercises and self-assessment questions. student support support for the on-campus students within the timetable for the conventional on-campus delivery there are embedded two classes, which are entitled 'tutorial support'. these classes occur the week before the assignments are due. in all, four hours in the module are designated for student support. ongoing tutorial support was usually given immediately after the taught classes, lasting anything up to an hour. the learners were informed of the availability of individual tutorial support during the day, but this service is seldom used. occasionally telephone tutorial support was requested and given. support for all distance-learning students within the distance-learning approach there are two different modes for supporting the student, one in which there is face-to-face contact, and one without. in both modes, communication between the students and the teacher and between the students and each other is maintained by an 'electronic campus'. this is a virtual learning environment that supports email, asynchronous (delayed time) and synchronous (real-time) discussion and a repository of relevant documents and articles. students have access to their tutor throughout the module, and should have an answer within twenty-four hours of making an enquiry. support for distance-learning students who have only online contact in the no-face-to-face contact mode, as exemplified by the united kingdom mode of delivery, the support the students received and gave to each other was carried out online. email correspondence between the teacher and the students during the module consisted of both general messages addressed to the whole student group and individual messages for individual students. the teacher initiated 24 general messages in the eight weeks of the module. these messages communicated information that all the students required. the number of messages sent by the teacher to individual students ranged from 14 to 24 messages. these messages concerned specific issues about which the students wished to consult their teacher. the average number of exchanges was 43 exchanges, or 86 separate messages over the eight weeks of the module (each exchange involves a communication and a response). students and the teacher participated in five discussion forums and six teleconferences. in the forums, key topics related to the module and requiring in-depth exploration were discussed online. participants were free to contribute to this discussion at any time within the week and participation was voluntary. at the end of the week, the discussion was summarized by the teacher and the summary archived as an educational resource. in the teleconferences, which are real-time discussion events, a topic was agreed in advance and participants shared their views by keyboard conferencing. the major difference between the hong kong and the united kingdom-based distance learning groups was the greater use of electronically mediated discussion by the united kingdom students. 75 tim duffy et al comparing distance education and conventional education support for distance-learning students who have online contact plus additional face-to-face contact distance-learning students in hong kong had the benefit of face-to-face contact in the form of tutorials. attendance at these was variable. these students also used email to send questions or messages once they had identified an issue. this obviated the need to wait until the next available face-to-face tutorial. on average, the teacher received ten emails per week. messages typically concerned administrative information such as the dates for course commencement and assignment release, as well as information on the scheduling of face-to-face tutorials, topics for discussion and any preparation that was necessary for participation in these. as most of the students were nurses working shifts, it was often difficult to arrange suitable times to be online together. an attempt was made to arrange teleconferences for the students, but only three students responded online. forum discussions are not dependent on synchronicity, but the hong kong students were noted to be reluctant to respond to the question set for the forum. ng (2001) similarly found hong kong students to lack social cohesion as a group, to demonstrate limited communication with each other and to show anxiety about computer-mediated communication. there was also a tendency for the students to demand answers from the teacher rather than engage in discussion, the 'instrumental' approach that we have alluded to already from sumner's (2000) study. assessments the assignments used in the assessment process were identical for all students taking this module. the assignments involve the students in carrying out a learning needs assessment in their places of work and, on the basis of their analysis of learning needs, delivering an appropriate lesson. they must also evaluate the lesson that they have delivered. students made use of different support mechanisms in preparing to submit their assignments. in the distance-learning mode that included face-to-face support, students participated in tutorials allowing them to raise questions and receive guidance. emails to the teachers increased in frequency in the run-up to the submission date. some students preferred to page the teacher, allowing them to talk directly with the teacher on the phone and receive an immediate answer to their enquiries. in the distance-learning mode that excluded face-to-face contact, discussion forums were suspended in the period when assignments were due. teleconferences continued, as they are a good means by which students can access teachers for last-minute guidance. some students were entirely independent in the preparation and submission of their assignments. others made full use of the tutor, by submitting one-page drafts of their ideas fot comment, or sections of their proposed assignments for comment. although these means of support are different in the forms of communication they employ, they are equivalent in effect. analysis of the outcomes from assessments this study seeks to identify what difference, if any, exists between student performance, as measured by the marks awarded to students from their assignments, and the different delivery modes that have been described. 76 alt-] volume 10 number i method earlier in this paper we identified three different modes of delivery of this course. for the purposes of this analysis, four student groups are identified and their marks presented. the reason for this increase of groups from three to four for analysis is that one of the groups (the hong kong distance-learning student group) comprises two types of student: indigenous chinese students and expatriate british students. treating the marks gained by these students separately may enable us to elucidate the influence of culture. the four groups identified for the purpose of comparison are: 1. chinese students in hong kong studying by distance learning and supported from hong kong (group 1, n=85); 2. british students in hong kong studying by distance learning and supported from hong kong (group 2, n= 15); 3. british students in the united kingdom studying by distance learning and supported from paisley (group 3, n=17); 4. british students studying in a classroom-based environment and supported on campus at paisley (group 4, n=37). results during the module, students submit two pieces of coursework for assessment a learning needs assessment that they have carried out and an account and evaluation of a teaching group number mean score group i (chinese, distance learning, hong kong) group 2 (british, distance learning, hk) group 3 (british, distance learning, uk) group 4 (british, classroom-based, uk) table 2: assessment results for the four groups of students: mean scores for the module 'professional practice andteaching' session that they have conducted. for each of these groups, the average mark for the two pieces of assessed coursework was calculated. the comparison of the mean scores achieved by each of these groups is presented in table 2: it is clear, from the scores presented in table 2, that these students are successfully meeting the demands of the assessments in this module. the pass mark is 40 per cent and the mean score for each group is well in excess of that. at face value, the scores presented in table 2 indicate that british students studying by distance learning, either in hong kong (group 2) or in the uk (group 3), are achieving the highest scores in this module. their average scores are almost the same. british students studying on campus (group 4) and chinese students studying by distance learning in hong kong and supported from hong kong (group 1) have achieved identical scores in this module. 77 85 15 17 37 61 74 72 61 77m duffy et al comparing distance education and conventional education because of the disparate sizes of the groups from which we have derived these mean scores, we then subjected our data to analysis using the kruskal-wallis test. when all four groups were considered together, this test confirmed the statistical significance of the difference across group means (h = 45.4163; significance, p < .01). the differences between the pairs groups compared 1 and 2 1 and 3 1 and 4 2 and 3 2 and 4 3 and 4 h-value from kruskal-wallis 26.6573* 19.1488* 0.3869 0.6012 24.2715* 18.7099* table 3: comparisons of the mean scores achieved by the four groups of students, using kruskal-wallis. *p<.0l of groups were then analysed. application of the kruskal-wallis test allowed us to distinguish those face value comparisons between groups which can be substantiated statistically from those which cannot. these comparisons are presented in table 3. from the analysis presented in table 3, we can state the significant findings as follows: • british students studying by distance learning in hong kong achieved a significantly higher average mark than their chinese counterparts. • british students studying by distance learning in the united kingdom achieved a significantly higher average mark than chinese students studying by distance learning in hong kong. • british distance-learning students, whether based in hong kong or in the united kingdom, achieved a significantly higher average mark than british classroom-based students. discussion it is important to avoid simplistic cause and effect conclusions about these findings. there are several factors to consider, and we outline these here. the language of instruction the language of instruction for this degree programme is english. this is the first language of the british students in each of the modes of delivery of this course. it is, however, a second language for the chinese students. for most of them their first language is cantonese or mandarin. so, in our comparison of results, we have to bear in mind that our chinese students are studying the module material and constructing their assignments in a foreign language, while the british students are studying in their mother tongue. however, we cannot jump to conclusions about the effect of studying in a language that is not the student's first language. some of the hong kong students completed their basic nursing training in hong kong government schools of nursing where the language of instruction was english. moreover, the hong kong-based students have to possess an equivalent level of competence in english as the on-campus students in the united kingdom. their competence in english is further attested by the internal moderators of their assessable coursework and by the external examiners. 78 alt-j volume 10 number i cultural differences between hong kong and the united kingdom the pattern of social interaction in a society and particularly the interaction between students and teachers is an aspect of culture that impinges on the educational experience. we have already noted that hong kong students were reluctant to raise questions and to engage in discussion in the public domain of the online forum. ip, lee, lee, chou, wootton and chang (2000) have detected a negative disposition towards critical thinking among chinese undergraduate nursing students. however, this reluctance is not unique to the chinese students. the united kingdom-based students also showed reluctance at times to participate in online discussions. this reticence is, in fact, a widespread and common aspect of online education. there is, perhaps, among the chinese students a greater respect for the status and knowledge of the teacher. however, our discussions with other teachers in hong kong indicate that this is fading fast. moreover, fung and carr (2000) found that the image of the hong kong students as passive and tutor-directed is not borne out. hong kong students are rapidly becoming the same as students in the united kingdom and usa in terms of treating their teachers as sources of information who are open to challenge. this approach was evident in the response of the chinese students to their teachers when they required help with preparing assignments. the chinese students made heavy demands on their teachers as the date of submission approached, contacting their teachers by phone, pager and email. this matches exactly the pattern of communication that occurs on campus when the submission of an assignment draws near. the role of personal ambition and drive the prospect of promotion by means of obtaining a degree is a strong motivator for the chinese students in hong kong. our discussions with the students and their tutors in hong kong make it clear that that is their major rationale for undertaking the degree. the appeal of a british degree is also a significant factor in their choice of programme. however, the british students in hong kong are working within the same health care system. they benefit in the same way as their chinese counterparts from the possession of a relevant degree. their motivation is likely to be the same. but their marks are significantly higher. something more than personal ambition and drive lies behind the differences in results that we have found. different values placed on higher education there is a contradiction in what we have found in terms of the different values placed on higher education in britain and hong kong. on the one hand, we have reported the traditional chinese reverence for learning, compared with the pragmatic british attitude to learning. on the other hand, we have identified that our chinese students are more likely to be motivated by the pragmatic prospects of promotion and enhanced salary than our british students. in fact there is probably a levelling factor to consider the influence of the profession in which these students are working. it is mandatory for nurses in the united kingdom to keep their knowledge up to date and to provide evidence of this. in hong kong, such updating is not mandatory, but most large hong kong hospitals require their nursing personnel to study for continuing nursing education (cne) points and to achieve a 79 tim duffy et al comparing distance education and conventional education minimum number of cne points each year. it is likely that these mechanisms to ensure continuing professional education override the different values placed on higher education by the two cultural groups that we are considering. variations in mode of delivery there is a significant difference in the results obtained by students undertaking this module in its different modes of delivery. the average marks obtained by students in the distance learning programme are 61 (group 1), 74 (group 2) and 72 (group 3). this gives an average mark over these groups of 69. this is significantly higher numerically than the average mark of 61 for the classroom-based students. but it is not significantly higher in terms of grade. both of these marks lie within the b+ grade in the university of paisley grading scheme; in fact, they are the lower and upper boundaries of that grade. we can say that students studying this module by distance learning obtain marks that are significantly higher than the marks obtained by students studying the module by attendance on campus, but that the difference does not extend as far as a difference in grade. variations in teacher input due to variations in class numbers the last factor we consider in our discussion is the input of teachers to these groups, particularly in the light of the different sizes of the groups the teachers are working with. on campus, the average class size in the b.sc. health studies programme is twenty students. the mean score these students are achieving in their assignments for this module is 61 per cent (see table 1). in the hong kong distance-learning mode, the average number of students per tutor group is 20 students. when we aggregate the marks gained by all the students studying by distance learning in hong kong (that is, both chinese and british students), we see that these students are achieving a mean score of 68 per cent in their assignments. the average class size for the united kingdom-based distance-learning students in this study is nine students. the mean score these students are achieving in their assignments is 72 per cent. these results would at least indicate that the increased contact that students have with their teacher due to both the means of delivery of their course and their class numbers leads to increased scores in assignments. conclusions both our distance-learning and our classroom-based modes of delivery are enabling students to complete the module 'professional practice and teaching' successfully. when we compare the average marks obtained by those undertaking the degree by distance learning and those undertaking it on campus, we find that the distance-learning students are obtaining significantly higher marks than the classroom-based students. that difference, though numerically of statistical significance, does not stretch to a grade difference. on average, then, our distance-learning students and our classroom-based students in this module are obtaining the same overall grades. the term 'distance learning' is no more a unitary term than the term 'classroom-based learning'. in both these modes of delivery there are widely different practices. in the module we have considered here, distance learning encompasses independent learning enhanced by face-to-face tutorials and computer-mediated distance learning with no faceto-face tutorials. we have tried to identify these differences using the average scores 80 alt-j volume 10 number i achieved and the results indicate that the entirely online form of computer-mediated distance learning is associated with the highest scores in the module under consideration. this is the first stage of our research. it has been a retrospective study. the next step is a prospective study. this forthcoming study will focus on only two modes of delivery, classroom-based study and supported distance learning with no face-to-face contact. once again the independent variable will be the common assessment, which all students in the study will undertake. however, the role of culture, existing professional background and motivation in the lives of the students who are undertaking this degree programme will continue to influence our findings. acknowledgement the help of mr david mcvey, writer of distance-learning material, in preparing this manuscript is acknowledged by the authors. references edwards, n., hugo, k., cragg, b. and peterson, j. (1999), 'the integration of problembased learning strategies in distance education', nurse educator, 24 (1), 36-41. fung, y. and carr, r. (2000), 'face-to-face tutorials in a distance learning system: meeting student needs', open learning, 15 (1), 35-46. garrick, r. (1997), 'the national committee of inquiry into higher education: report of the scottish committee, chapter 2, section 2.4'. available from: http:llwww.keds. ac.uk/ncihel higher education statistics agency. (2000), higher education statistics for the united kingdom, 1998/99, cheltenham: hesa, table 1, 8. ip, w. y, lee, d., lee, i., chau, j., wootton, y and chang, a. (2000), 'disposition towards critical thinking: a study of chinese undergraduate nursing students', journal of advanced nursing, 32 (1), 84-90. li yixian (1992) 'on the characteristics, strong points and shortcomings of education in hong kong: a mainland chinese educator's view of education in hong kong', in g. a. postiglione (ed.), education and society in hong kong: toward one country and two systems, hong kong: hong kong university press. mcgreal, r. (2002), 'significant difference'. available from: new brunswick: teleeducation, http://teleeducation.nb.calsignificantdifferencelindex. cfm ng, k-c. (2001), 'using e-mail to foster collaboration in distance learning', open learning, 16 (2), 191-200. peterson, p. w. (2001), 'the debate about online learning: key issues for writing teachers', computers and composition, 18, 357-70. phipps, r. and merisotis, j. (1999), what's the difference? a review of contemporary 81 tim duffy et al comparing distance education and conventional education research on the effectiveness of distance learning in higher education, washington, institute for higher education policy. available from: http:llwww.ihep.org/pubs/pdf/difference.pdf russell, t. (1999), the no significant difference phenomenon, chapel hill, nc: office of instructional telecommunications, north carolina state university. russell, t. (2002), 'the "no significant difference" phenomenon'. available from: http://teleeducation. nb.ca/nosignificantdifferencelindex.cfm sumner, j. (2000), 'serving the system: a critical history of distance education', open learning, 15 (3), 267-85. university grants committee (1996), 'higher education in hong kong; a report'. available from: http://www.ugc.edu.hk/hervw/chapte16.html 82 editorial in this issue the editorial space has been given over to a tribute to david squires. the review below gives an outline of some of his research interests and his substantial contributions to learning technology. in the last issue, as an experiment, we invited comments on the paper by davies and denning. the idea behind this was to try and encourage more discussion around papers and to look at introducing a more interactive element to the journal. we will postpone this now until the next issue. if you would like to contribute to this debate, please email comments to g.conole@bristol.ac.uk by friday, 14 december. finally, i would like to welcome dr jane seale and dr martin oliver to the editorial team. jane is based at king's college, london and is also editor for alt-n. martin is based at university college, london and has extensive editorial experience. i am looking forward to working with them both over the next few years. david squires, 1948-2001 professor david squires took over as editor of alt-j in 1999. his recent tragic death is a true loss to learning technology. david's career spanned the educational spectrum. in the uk, he was one of the original pioneers in work on considering how computers could be used to enhance learning. he began his career as a physics teacher and was based for one year at the latymer school in enfield. he moved in the late 1970s into an advisory role and was adviser for computers in education for devon local education authority in the early 1980s. he went on to become director of sciences for the seminal project 'computers in the curriculum' (one of the three major posts on the project) based at king's college. this was one of the first major computers-in-education initiatives in the country. at about the same time he also took up a post at king's as a lecturer in educational computing within the school of education. he remained at king's and was promoted to senior lecturer in 1994 and then to a chair in 2000. 2 alt-j volume 9 number 3 his research history shows a commitment over twenty years to scholarly work in the use of multimedia and information and communication technologies (ict), in educational settings at both university and school level. particular interests included theories of learning and the design and evaluation of the use of multimedia and ict in educational settings, and the use of ict in academic research. his research was strongly linked to curriculum development through the design of multimedia learning materials for use in higher education and schools. a major, thrust of his research was to articulate a theoretical rationale for the design and use of educational multimedia. this work featured a synthesis of ideas from a range of disciplines (software design, human-computer interaction, cognitive development and theories of learning, curriculum theory and pedagogical practice) with the aim of identifying synergies between these ideas at a theoretical level, to inform design and evaluation. in 1994 he co-authored a book which addressed the multi-faceted nature of the use of multimedia in teaching and learning, and proposed a paradigm for educational software evaluation, which he took broadly to include theoretically grounded analyses of validity and practice-orientated assessments of function. the contribution to theory made by his work was much praised. for example, the following editorial appeared in the journal of information technology for teacher education: many papers in this international journal discuss and develop concepts in relation to the professional development of teachers. in particular, the journal considers the professional development required by beginning and practising teachers in order to be able to use it to encourage learning, to use it to teach with and to manage the application of it in the learning environment. it was therefore with delight that we received a paper which provided a clear framework. anne mcdougall and david squires's paper provides the keystone to this issue of the journal and this editorial. the framework's title is 'perspectives interactions paradigm'. within it are five commonly observed foci for teacher professional development [... ]. i think that many in the field have already written about aspects of these foci, but this may be the first time they have been brought together in a clear framework and i suspect that it may apply well beyond it to any innovation in education. (davis, 1997) he was also particularly interested in the relationship between human computer interaction and teaching and learning: the use of direct manipulation educational software. was the focus of his ph.d. thesis. in december 1996 he hosted a british computer society human computer interaction special interest group meeting on 'usability and educational software design'. he also undertook a research fellowship with bt in 1999 on the design of an interface to a web-based teachers' toolkit for open and distance learning. another strand to his research was an exploration of the use of ict in academic research, which included longitudinal analyses and case studies of the use of ict by academics. from its inception this work was at the cutting edge: notably the use of state of the art ict tools, including on-line bibliographic sources, networked cd-roms, bibliographic management software, web browsers, email and pre-print bulletin boards. methodologies for researching the use of ict in a research context were ill formed in 1992, and the work david directed at king's has been acknowledged as making a significant 3 editorial contribution to this area. his research highlighted that many assumptions about how ict is used in research have been shown to be simplistic; for example, beliefs that user support can be primarily reactive to users' needs rather than proactive in setting an agenda for user support. these findings have contributed to our understanding of the complex interactions that characterize the use of ict in a research context. much of his experience of applying ict in higher education was gained in a european context. he was a member of the 'aqua' tempus project based on a collaboration between king's college, the institute for science education, kiel and the university of hradec in the czech republic. this project aimed to develop the use of ict in environmental education, with an emphasis on establishing quality-assurance procedures for curriculum development. during the period 1994-7 he was a managing partner of the 'site' tempus project principally involving collaboration with german and bulgarian partners. this project focused on educational multimedia development, the analysis of satellite images using state-of-the-art software applications, and the development of internationally accredited masters degree programmes in ict and environmental studies. prior to this, from 1991 to 1994, he was a managing partner for the 'cobes' tempus project, which aimed to revise the environmental studies curriculum offered to engineering students at the technical university in sofia. this outline of some of david's key research interests only touches on the range and breadth of his work. in addition, he was an active member of the university and the higher education community, serving on innumerable committees, steering groups and advisory boards, as well as undertaking significant undergraduate and postgraduate teaching and research supervision. he will be sadly missed, by me personally and by everyone else who was fortunate enough to have worked with him. gráinne conole editor reference davis, n. (1997), 'framing teacher professional development', journal of information technology for teacher education, 6 (2), 109-13. 4 discussion section gender issues in computer-supported learning: what we can learn from the gender; science and technology literature gwyneth hughes centre for learning technologies manager, university of east london email: gwyneth@uel.ac.uk this paper is a response to the article, 'gender issues in computer-supported learning', in alt-j 10 (1). i argue that the studies presented in the original paper could be enhanced by a more rigorous approach to gender that avoids universalizing identity, recognizes gender as a construction and which builds on previous research from gender, science and technology studies. after the surge in research and literature on gender, science and technology in the 1980s and early 1990s, this paper brings us up to date with the growth in computer-supported learning (csl), adding an extra dimension since computer technology is now a mode of delivery as well as a discipline under study. the evidence presented in this paper that a gender gap in access and it literacy has been steadily closing during the 1990s seems to follow the closing of gender gaps in access and achievement in science and technology subjects in the uk (arnot, david and weiner, 1996). however, a closer look suggests that discriminatory practices that disadvantage females and males in csl persist, and i suggest that there are some lessons which can be learned from the earlier gender, science and technology literature to avoid over-simplifying gender and assuming that undesirable gender effects can be easily overcome with good practice recommendations. firstly, research that universalizes gender by assuming that all females (or males) have common experiences is problematic. the authors acknowledge in their conclusion that 'issues of gender should not be treated in isolation from other factors', including educational backgrounds, but most of the studies reviewed in the paper did the opposite. a range of identities intersect with gender, such as ethnicity, class (hughes, 2001), as well as sexuality and disability. age (generation) was briefly mentioned in the article and is likely to be significant too. the case studies presented here are of it and web design courses and, consistent with european union gender segregation statistics in computing and it at 77 gwyneth hughes gender issues in computer-supported learning higher education level (rees, 2001), we know that the it course is male-dominated. it would be useful to have this information for the web design case study too. despite the gender bias, the females performed well, but this is not necessarily a straightforward gender effect. studies of other 'masculine' constructed disciplines such as physics and engineering have found that the women succeed, although they may have a difficult time, but this is because they are middle-class and academically advantaged female students (thomas, 1990). meanwhile, male underperformance in education in general has been linked to particular forms of (often classand age-differentiated) masculinity which reject or play down academic study (connell, 1995). ethnicity may add more complexity as there is evidence that black females are more willing to persevere with masculine areas of study than their white counterparts because of the career opportunities in an inequitable job market (mirza, 1992; hughes, 2001). because of such differences between females, the two statements about girls' achievement given in the introduction are not necessarily contradictory it depends on which girls are counted. secondly, and to add further complexity, there is a body of theory that views gender as a constructed rather than essential characteristic (for example, davies, 1989), and this too is missing from the discussion. rowan (2000) explains that it is not only differences between women, but differences, or conflicting identities, within women that produce the apparent gender differences in behaviour in csl environments. it is not easy to construct a feminine identity while undertaking masculine activities, and female reporting of lack of confidence with internet technology could be one way of dealing with internal gender conflict in a symbolically and culturally masculine environment of cyberspace. thus, examining the context in which a gender identity is produced is very important. essentialist approaches to gender have focused on learner deficiencies and tend to underplay the importance of educational and social factors on reproducing gender effects. the authors do go some way towards acknowledging the importance of social context by stating that 'social issues such as ownership of technology are at the forefront of research and need further discussion'. wacjman (1991) has pointed out that technology does not transform, but tends to reproduce, social relations. the authors implicitly support this approach in evidence from herring that gender interactions and inequities in csl reproduce those of traditional environments, and in their concern that the design of course management systems reproduces conventional didactic, rather than constructivist, learning environments. however, other links between gender identities and context are not made. it is not clear how far the gender effects reported in the paper arise from the use of technology as a learning tool, or because the technology is the subject of study, or indeed an inter-relationship between the two. do the gender associations of the discipline make a difference; for example, are the gender issues of cmc (computer-mediated communication) different in sociology from computer studies? finally, recommendations for designers to construct well-moderated, flexible learning environments that are appropriate for a wide range of learners are valuable. however, the limited impact of initiatives such as wise (women into science and engineering) (henwood, 1996) would suggest that there is no guarantee that gender divisions will readily disappear. care must be taken that recommendations do not merely reinforce gender binaries by highlighting gender difference (as in the suggestions that males prefer games and females more collaborative activities). although the audience for the alt-c panel 78 alt-j volume 10 number 2 wanted solutions rather than more research on causes of gender inequalities, i would argue that quick fixes are unlikely to be found. challenging the restrictions of gender binaries, without ignoring gender completely, is a tricky business and requires a sound appreciation of the complexities. fortunately, the foundations for further exploration of gender and csl already exist in an established body of related work. references arnot, m., david, m. and weiner, g. (1996), educational reforms and gender equality in schools, manchester: equal opportunities commission. connell, r. (1995), masculinities, cambridge: polity in association with blackwell. davies, b. (1989), frogs and snails and feminist tales, sydney: allen & unwin. henwood, f. (1996), 'wise choices? understanding occupational decision-making in a climate of equal opportunities for women in science and technology', gender and education, 8 (2), 119-214. hughes, g. (2001), 'exploring the availability of student scientist identities within curriculum discourse: an anti-essentialist approach to gender inclusive science', gender and education, 13 (3), 275-90. mirza, h. (1992), young female and black, london: routledge. rees, t. (2001), 'mainstreaming gender equality in science in the european union: the etan report', gender and education, 13 (3), 243-60. rowan, l. (2000), 'human resource management, "flexible" learning and difference', in v. jakupec and j. garrick (eds), flexible learning human resource and organisational development: putting theory to work, london: routledge. thomas, k. (1990), gender and subject in higher education, milton keynes: open university press. wajcman, j. (1991), feminism confronts technology, cambridge: polity. 79 doi:10.3402/rlt.v24.29369 the educational problem that moocs could solve: professional development for teachers of disadvantaged students diana laurillard* london knowledge lab, ucl institute of education, london, united kingdom (received 6 august 2015; final version received 31 january 2016) the demographics of massive open online course (mooc) analytics show that the great majority of learners are highly qualified professionals, and not, as originally envisaged, the global community of disadvantaged learners who have no access to good higher education. mooc pedagogy fits well with the combination of instruction and peer community learning found in most professional development. a unesco study therefore set out to test the efficacy of an experimental course for teachers who need but do not receive high-quality continuing professional development, as a way of exploiting what moocs can do indirectly to serve disadvantaged students. the course was based on case studies around the world of information and communication technology (ict) in primary education and was carried out to contribute to the unesco ‘‘education for all’’ goal. it used a co-learning approach to engage the primary teaching community in exploring ways of using ict in primary education. course analytics, forums and participant surveys demonstrated that it worked well. the paper concludes by arguing that this technology has the power to tackle the large-scale educational problem of developing the primary-level teachers needed to meet the goal of universal education. keywords: moocs; blended learning; disadvantaged students; teaching designers; peer learning; learning community; professional development introduction digital technologies can do much for education, but it is suboptimal to ask ‘what problem can we solve with this technology?’ this is what happened when the first free online courses from top us universities attracted massive student numbers. big numbers attract investment, and this led to the development of learning platforms to manage teaching activities on the large scale. then the rhetoric needed the big problems to match the invention of the massive open online course (mooc), and ‘‘education for all’’ was the obvious candidate. access to higher education is one of the very big problems in education. for example, by 2025 the global demand for higher education is predicted to double to �200 m per year, mostly from emerging economies (nafsa 2010). the early excitement about moocs coalesced around the idea that they could potentially be a solution to this massive educational problem (laurillard 2014). *email: d.laurillard@ioe.ac.uk responsible editor: carlo perrotta, university of leeds, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 d. laurillard. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 the problems moocs could solve however, the demographics of moocs are not moving in that direction. by far the most popular on the coursera platform are those in the it and finance areas, which update the already highly skilled professionals in those areas. the harvard and mit platforms, offering a wide range of subject areas, attract participants of whom 72% and 63%, respectively, already have a bachelor degree or more (ho et al. 2014). the equivalent coursera figures are 85% (coursera 2014). so the massive numbers are not the intended pre-undergraduate population, but highly qualified professionals. of the harvard�mit participants, only 2.7% are from the un ‘least developed countries’. in the context of massive courses this is �20,000 people, which seems an impressive advance, except that these are not degree awarding courses, and even 20,000 is a drop in the ocean of unmet need (daniel, 2012). the problem that moocs are actually solving, therefore, is how to provide free education to highly qualified professionals � not a problem anyone had previously thought to identify. if, instead of starting with the technology, we take a major education problem, such as education for all, 1 or raising learning outcomes in the poorest countries, 2 and ask ‘how can technology help?’, what follows has to be an analysis of what it would take to achieve that. unesco estimates that universal primary education requires �1.6 m new teachers in the poorest countries. similarly, the raising of learning outcomes is being tackled through research that will ‘provide policymakers and practitioners with concrete ideas on how to improve learning, and understanding of how these will translate to their specific contexts and institutions’ (esrc-dfid call 2015). 2 ask how technology can help at this stage in analysing the problem, and one answer could well take the form of an mooc. the nature of professional development for teachers one of the critical issues identified in many studies of continuing professional development (cpd) for teachers, especially with respect to the adoption of learning technology, is lack of time to innovate, closely followed by lack of support or leadership (dowker 2009; griffin 2004; knight, tait, and yorke 2006). another more general study identified an even more important factor: the lack of collaboration among educators within their department, across subject matter groups, and with educational science itself (hökkä and eteläpelto 2014). the authors conclude that there are two requirements for professional development that: a. individual teacher educators can learn, renegotiate their professional identities and maintain their enthusiasm and commitment b. teacher education communities and organisations can create innovative and collaborative ways to implement teacher education in the future (hökkä and eteläpelto 2014, p. 49). this analysis matches the findings of a recent study of reports on teacher professional development programmes that have a more reflective and iterative approach. it shows that the nature of cpd for teachers is beginning to change. the traditional in-service training (inset) approach that delivers information and guidance to teachers is shifting towards a co-learning approach. it documents the recognition that teachers’ d. laurillard 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 own knowledge and experience must be at the heart of professional development, to make it something done with them, not done to them. co-learning involves: networking and interchanges among schools and situations and is strengthened in formalised experiences such as courses and workshops that introduce peer coaching or support collaboration and joint projects. . . the lesson learned is that teachers naturally talk to each other, and that such talk can take on an educational purpose. (avalos 2011, p. 18) from both studies it seems evident that unless teachers are the ‘prime actors’ in their own development, it will be impossible for them to keep up with the rapid changes in the environment, political, cultural, economic and, especially, technological. the mooc as co-learning for professional development this paper gives an account of a problem for which an mooc offered a real solution. faced with the problem of how best to communicate a report on the findings of a unesco study on ict in primary education, which documented the innovative practice of teachers across 19 different countries, the report authors decided to investigate the use of an mooc format. the aim was to promote the use of information and communication technology (ict) in primary education, especially in those countries where this is uncommon, or even resisted. publishing a book is unquestionably an instructivist approach to professional development; we wanted a more collaborative and constructivist engagement with teachers. what works in one school does not necessarily work in another, so the basic principles of using learning technology must be debated and contextualised by the professionals themselves, if adoption is to succeed. this teacher-centred, co-learning approach follows exactly the ideas proposed by avalos, and hökkä and eteläpelto. the affordances of the main mooc platforms focus on presentation formats, such as video lectures and reading resources, supported by automated testing (mostly forms of multiple choice questions), discussion forums, and peer-reviewed assignments. clearly, the opportunities for co-learning were limited, but could an mooc be viable, nonetheless? in their analysis of the current evidence of institutional approaches to moocs, hollands and tirthali (2014) found that the forums in moocs tend to be used for question-and-answer (q&a), rather than peer discussion. since peer discussion is the main opportunity for co-learning in an mooc, this is not promising. however, bayne and ross, in their review of five uk moocs, made the important point that ‘mooc pedagogy is not embedded in mooc platforms, but is negotiated and emergent’ (bayne and ross 2014, p. 8), that is, the prevalence of q&a is not a property of the platform: the way forums are used depends on the pedagogic design. both discussions and assignments could be used to support co-learning. mooc pedagogy has the potential to go beyond the assumed affordances of the mooc platform and find ways of making connections between learners and their peers, and between learners and facilitators (kop, fournier, and mak 2011). recent reviews make the point that moocs for professional teacher development have been poorly researched (jobe, östlund, and svensson 2014), and that the pedagogy of moocs has not received sufficient focus and needs greater attention (bayne and ross 2014). this paper is intended as a contribution to fill that gap. research in learning technology citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 the aim of our project was to design an mooc that would promote a form of colearning to provide the type of teacher professional development that researchers argue is now needed. the mooc as professional co-learning: a case study this paper documents the results of an investigation into the mooc as cpd, carried out in collaboration with the unesco institute for it in education (iite). the main mission of the iite is to bridge the digital divide in education: building inclusive knowledge societies, reinforcing national capacities, promoting e-environments and increasing access to education and lifelong learning. 3 in the context of the unesco ‘‘education for all’’ programme, the value of an mooc in particular was its potential to reach teachers in the emerging economies. our initial question was: to what extent could such an open online course succeed in bringing a co-learning model of professional development to large numbers of teachers, especially in the emerging economies, as a contribution to solving the educational challenge of integrating ict effectively into primary education? course design and development the course was developed on the coursera platform, hosted by the university of london international programmes. the course team was formed of the authors of three volumes of research studies, funded by the unesco institute of it in education, 4 to bring an understanding of ict in primary education to a wide audience (kalaš et al. 2012, 2014; lim et al. 2014). the material in the three volumes were extracted for the readings, along with videos and texts provided by the 32 case studies from schools around the world. the course team was formed by the eight researchers who had developed the three books on which it was based, with one member (the author of this paper) acting as lead instructor and chair of the team. the design process with eight members in different countries and time zones, there were few opportunities for face-to-face collaboration, so we used two skype meetings to discuss and agree overall planning and scheduling, took advantage of members’ international travel for occasional small group meetings and used one�one skype meetings between the lead instructor and other members to fine tune designs. the team used a shared googledoc to plan the teaching�learning activities. the overall pedagogic approach was designed for the first week by the lead instructor, which provided the model for the remaining weeks. as the design for each week was completed, the technical designer transferred it to the coursera platform, so that instructors could edit the study guide, add further resources and forum activities, and edit the assignment instructions and rubrics. an experienced primary training educator was employed to carry out detailed developmental testing and give feedback on problems that would need redesign. the first 2 weeks of the course were submitted to a quality assurance panel at the university of london for final approval. d. laurillard 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 the intended learning outcomes for the course given the overall aim to bring an understanding of ict in primary education to a wide audience, the course was targeted at primary education leaders, teachers and policymakers in all countries, and had the following intended learning outcomes: (1) the primary school teaching community to be able to contribute to the integration of a range of effective ict-based practices and pedagogies. (2) school leaders to be able to develop, improve and share the strategies and mechanisms that optimise the development of ict-based teaching and learning in their school. (3) policymakers and schools-related stakeholders, agencies and companies to be able to support head teachers and specialist staff in developing pedagogy-led and problem-led uses of ict. (4) for all participants, a greater awareness of the range of pedagogic innovation that uses ict, and the many free technology tools now available, to enhance primary education. teaching and learning activities the standard mooc pedagogy is to provide a mix of presentations (videos, digital resources), automated assessment, peer-assessed assignments and peer discussions conole 2013). the format is a good fit with the pedagogy of the standard cpd course, which provides presentations to update participants on the latest research and developments, along with peer discussion and a non-assessed certificate of attendance. cpd requirements make it possible to run moocs on a very large scale, as they require only the fixed teaching costs of preparation, and none of the variable teaching costs of student support while the course is running. like other mooc platforms, coursera does not easily support the kind of collaborative learning we intended. in addition to the presentations and discussions, we wanted participants to be able to work in peer groups, sharing experiences, ideas and expertise. this is in line with the co-learning approach to cpd that recognises the great value of the experience and expertise that teachers can offer each other. the pedagogy of this cpd mooc combined curated digital resources with orchestrated peer collaboration. curating the resources of videos, texts, case studies and research findings is straightforward on an mooc platform. collaboration is more difficult and typically has to be done off-platform. ‘orchestrating’ collaboration is essential. the process has to be carefully designed; it does not just happen by providing forums. to enable participants to share ideas and experiences, they were invited to provide links to ‘show-and-tell’ case studies of their ict-based pedagogies, and examples of technology tools and resources they used in their teaching. we did this by setting up external sites such as padlet and diigo, where they could share and comment on each other’s contributions. it was important then to bring them back to the course on the mooc platform, so we set up discussion forums that did not rely on participants to organise their own knowledge-building (de waard 2015) but guided them to focus on explicit issues relating to their contributions on each of the external sites. through these activities, participants were able to make contacts and create professional connections, and learn about and discover technologies for their own use, building on each other’s contributions. research in learning technology citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 a study guide for activities the typical mooc format for a week of study begins with an overview (in the announcement) and access to the video lectures, quizzes, forums and other resources, with guidance on how to use each one. instead of this, given our strong focus on contextualised practice and collaborative learning, the team designed each week as a clear study guide in the form of a sequence of resources and activities linked by tutorial text (see figure 1). the main distinctive pedagogic features were: (1) indication of activities that were core for passing the course, or optional (2) expected learning time for each activity (3) external resources and tools to support course resources (4) guidance on how to learn from each activity (5) issue-focused forums to focus discussion (6) links to external sites for sharing experiences, ideas and resources (7) advice on building a course journal for use after the course. the first four features were designed to support the participants in independent but guided learning; the next two were to support co-learning using the external sites to support participants in learning from each other; and the course journal was to ensure they would be able to take something of use from the course into their normal teaching. supporting collaborative learning collaborative learning goes beyond discussion learning by bringing ‘constructionism’ (kolb 1984; papert 1980) to combine with the social learning of ‘social constructivism’ figure 1. showing some of the key features of the pedagogic style. d. laurillard 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 (vygotsky 1978; wertsch 1985), and thereby adds to discussion learning the requirement to produce a joint construction. this could be as simple as a definition or summary, and as complex as a design or a research finding. without the joint output, discussion can be relatively unchallenging and unresolved, whereas in order to produce a joint output it is essential for students to debate and negotiate until the single shared product can be delivered. this is a much richer learning experience because it gives focus to the discussion and enables students to learn from and build on the outputs of their peers and to share their reflections and interpretations of what happened within their practice (laurillard 2009). it also needs more guidance than is typical for moocs, in which participants are often encouraged to use social media to develop their own knowledge-building communities, without any guidance (de waard 2015; liyanagunawardena, adams, and williams 2013). for teachers learning from each other about the optimal use of technology, the shared output could be, for example, a comparison between different technologies to use for a particular learning outcome, or an analysis of student performance under different conditions, or a learning design to improve the impact of a technology. however, this would mean being able not just to discuss and debate online but also to create and share their practice outputs as well. mooc platforms do not easily support this. table 1 sets out the requirements for truly collaborative learning (laurillard 2012), the ways in which digital technology could support them, and the extent to which moocs succeed in doing so. it is clear, therefore, that although it is possible to extend the community learning capability of moocs by linking to external sites and off-platform collaboration tools, moocs tend to support only discussion learning, not the more challenging and richer form of collaborative learning that includes the joint construction of a shared output. assessment in line with standard cpd design, there was no formal assessment of the ict in primary education course. the automated assessment methods using quizzes were not appropriate for evaluating the qualitative descriptions and localised interpretations of theory that were the main outputs from participants. peer assessment was the only form of assessment used. studies of peer assessment methods used in moocs have emphasised the importance of using qualitative assessment which recipients can use as guidance for improving their work (carless et al. 2010), and of guiding their assessment by defining criteria and scoring rubrics to improve the validity (luo, robinson, and park 2014). for our teacher participants, the assignments asked them to develop documents, plans or resources for their future work beyond the course. each assignment had a set of criteria, which participants could use to assess their peers, giving scores and constructive qualitative feedback. the four peer-assessed assignments were to: � create and describe a learning object � develop a ‘technology decision’ for their school or class � prepare a suggestion for tackling one of the key challenges identified in the literature � design a two-page brochure or newsletter to inform policymakers about the issues of implementation research in learning technology citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 in each case they could use within their assignment what they had produced during prior activities on the course. the requirement to review the outputs of their peers was also an opportunity for participants to use this experience to improve on their own outputs. summary of the course design approach in summary the pedagogic principles governing the design of the course were: � to curate the most useful evidence and resources for teachers, heads and policymakers � to orchestrate the teaching community’s co-learning that would build their knowledge of using digital technology � to guide participants’ study planning by providing core and optional activities, and recommended timings for each one � to engage participants in making best use of learning technologies through guided activities, issue-focused discussions and both independent and collaborative learning � to provide the tools and activities that enable participants to build their learning on the course into their working practices the most difficult design challenge was to meet the full requirements of collaborative learning as outlined in table 1. collaboration is only viable for small groups, and on coursera it is not yet possible to group the students registered on an mooc. the coursera platform can orchestrate the inter-student activities involved in peer assessment, that is, the successive stages of: table 1. the extent to which digital technologies, in general, and mooc platforms, in particular, support the optimal conditions for collaborative learning. requirements what technology could offer what mooc platforms offer italics indicate ‘off-platform’ 1. a shared task goal this is for the teacher to specify instructions for the learning task 2. resources relevant to the task websites, digital libraries, oers, collaboration tools course videos and texts links to external resources and tools 3. the means to discuss the task online discussion environment, synchronous and asynchronous, recording all discussions open asynchronous forums links to external tools for recording webinars or hangouts 4. guidance on managing the discussion�solution process study guide, roles, scripts, worksheets, teacher intervention orchestration of the learning task in the study guide 5. the means to construct and revise representations of meeting the task goal applications for creating visual representations, usergenerated content links to external tools for user-generated content, designs, representations 6. the means to share representations of meeting the task goal design environment, file exchange, file management links to external tools for exchanging user-generated content 7. the means to test ideas and solutions for meeting the task goal a modelling environment that gives intrinsic feedback on the learner’s solution links to external modelling environments for testing ideas d. laurillard 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 checking a student has submitted an assignment, allocating it for review by other students who have already submitted, recording and calculating scores, and advising each student of their score. this is a valuable reciprocal learning activity, but it is not collaboration. the mooc as community learning: an evaluation our empirical study evaluated an instantiation of professional community learning in a particular course to answer the general question: to what extent could an open online course succeed in bringing a co-learning model of professional development to large numbers of teachers, especially in the emerging economies, as a contribution to solving the educational challenge of integrating ict effectively into primary education? this course focused on learning technology, which must play an important part in supporting primary education if only because of its potential to support independent adaptive learning for many topics in the primary curriculum (kalaš et al. 2014; butterworth and laurillard, 2010; robertson and howells, 2008; smeets 2005). this was a basic and unquestioned premise of the course, so the research questions we had to answer are: � can it reach large numbers of teachers in the emerging economies? � can an mooc support meaningful professional development? � can one course be of value to such a diverse group of teachers, in terms of the practical impact in their local educational context? we might succeed in the first two by the simple expedient of placing a course on an mooc platform, so the third point is the real challenge. the data reported here is collected either by the extensive analytics provided by coursera, or from the preand post-course surveys run by the course team. reaching teachers in the emerging economies the course was clearly advertised as cpd for teachers and education professionals, and indeed this was the audience it reached. over the duration of the course �9000 teachers registered, from 174 countries. the average registration on a coursera course is �11,000, so for our very niche topic this was a reasonable number. at the start of the course 5891 had registered, of whom 3230 (54%) were active in the first week, also fairly typical. the pre-course survey had 1150 respondents, that is, 36% of active participants, which is relatively high for such surveys. the demographic data it produced showed that the course reached its intended audience of teachers and policymakers in primary education, as well as academics from further and higher education (see figure 2). the educational level of the participants was high at 89% with degrees, as would be expected of a cpd course, but this was not much higher than the coursera average of �80% for general courses. research in learning technology citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 more interesting for our first research question was the reach to emerging economies. participation reached 174 countries, and the registration percentage from africa and oceania was 7% and 4%, respectively, almost double that of the coursera averages of 4% and 2%, respectively. we also looked at participation from the emerging economies by investigating the 60 ‘countries of focus’ identified by dfid 5 as those with the most challenging educational problems. coursera collects data on participation from 47 of these countries: 20 registered no participation in our course; 27 registered a total of 1282 participants, 14% of the total registration. on the one hand, there is a very long way to go to reach all countries in greatest need; on the other hand, to have reached nearly 1300 participants from 27 of them at the first attempt is remarkable, and it encourages us to think that it is viable to use moocs as a means of reaching some of these countries in focus. completion rates for the ‘countries of focus’ were less good than for the others: 19% against the 28% average for this course. interestingly, the completion rates were highest for the higher status employment groups of policymakers, school managers and lecturers at 50%, 40% and 48%, respectively, whereas class teachers were at 20%. completion may also be determined by working time spent online, as even head teachers achieved 27% completion. the value of the course as professional development the measures of the success of the course come from the post-course survey, which was sent to all registered participants, and elicited 174 responses (21% of week 6 active users). the results for the overall experience and the perceived improvement in subject understanding are shown in figure 3. the course received the resounding appreciation of the respondents, with a pleasing 99% rating of excellent, very good or good. figure 3b shows one reason for this: the significant improvement in participants’ understanding of ict in primary education. it seems to have broadly achieved the intended learning outcomes in terms of satisfying participants’ own expectations, as figure 4 shows. the agree/strongly agree scores total more than 90% on every statement and also reveal a strong appetite for a follow-on course. there is good evidence therefore that the course was highly valued and achieved its main objectives. figure 2. showing that the course attracted the intended audience. d. laurillard 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 supporting community learning evidence of its success as a form of community learning comes from the post-course survey, the course analytics and the content of forum posts. engagement in the discussion forums is an important measure of how participants value this form of peer community learning. low participation rates in forums are a common feature of moocs, partly because posts are diverse comments, rather than conversations, which are difficult to manage for the very large numbers present in moocs. our pedagogic design sought to overcome this by creating issue-focused forums that reduced the wide variation in points raised, and elicited more conversational discussions. the evidence shows that engagement in the ict in primary education forums was much higher than for the average mooc, and was maintained throughout the course. figure 5 shows this in comparison with the first four moocs run by the university of london, all of which reported successful results: of participants active in week 6 around 60% watched a video, whereas only 2�3% posted in forums. by contrast, 39% of our course participants still active in week 6 were posting in forums. the posts to some of the later discussion forums show that teachers enjoy the issue-focused discussion forums because they value highly the experience and contributions of their peers: figure 3. (a) overall experience, and (b) change in subject understanding. figure 4. participants were asked to rate their agreement or not with the above statements. research in learning technology citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 thanks to whole team of this course and peers. it was an interesting journey, learned and shared so much. a community in practice, i really enjoyed participating. the peer assessments for the assignments were very interesting, allowing me to learn from the practice of others in education and to see the different priorities from around the world. we all have a common goal and that is to make a positive influence on the lives we inspire. thanks for a great time of learning and sharing. i have had an amazing time learning so much from all the teachers around the world!!! comparing the popularity of forums, those embedded in optional activities, which had an average of 275 posts, were less popular than the forums in core activities, which had an average of 345 posts. making an activity optional therefore risked a 20% drop in engagement. in addition to the issue-focused discussions set up by the course team, there were also discussions relating to participants’ outputs, such as lesson plans, or case studies. these were not, strictly, collaborative activities, because the platform does not lend itself easily to collaboration. we could not ask participants to work together to produce an agreed joint output as it is not possible to set up small groupings of students. the forum activity is highly appropriate for discussing issues and interpretations among peer professionals, but there is no form of shared user-generated production environment in the platform, even something as simple as a googledoc. the closest we could get to this was to use external peer resource exchange environments such as diigo and scoop-it, or external googledocs and similar tools, and link these to a forum. it remains difficult in this platform environment to engage participants fully in collaborative learning activities that enable them to negotiate a joint output. this is not because they are unwilling to share with each other. there are over 400 shared digital resources now sourced from these participants on the diigo site set up for the course, creating a valuable repository of tagged and recommended resources from the peer community. we also saw evidence of a willingness to share designs in the responses to the learning designer tool. 6 in the forum discussing their initial use of the tool, participants often commented on the value of being able to share their learning designs: i love the fact that we can share, co-create and better our practice in an organised and efficient way. i am going to recommend this to my student teachers especially. figure 5. of those still active in week 6, video access was similar, but the ict in primary education mooc showed a significant difference in the forum activity. d. laurillard 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 i have found it to be a fantastic tool. it helps teachers to share good knowledge and practice in teaching ict. the learning designer tools [sic] enables teachers to share their good teaching ideas which is fantastic! . . . what a wonderful � simple but effective teacher tool! i love the idea of how it is all shared and adapted to suit each different context! it is a great tool for teachers to share. i plan to use it this summer and share this site with colleagues when i return to school in the fall. such comments are encouraging for the prospect of using moocs to promote the building of community knowledge among teachers about the optimal use of learning technology. for a professional development course to be truly effective, teachers must be able to use their new learning in their local practice. the return to normal practice can too easily seem unmanageably distant from the inspirational course. by providing issue-focused forums that related the course resources to their normal practice, and with the reminders of the course journal, we hoped that teachers would end the course with a sense of empowerment to innovate. there was evidence that their enthusiasm for innovation would continue beyond the 6-week course in many of the final forum comments: i can’t express enough, how much i’ve gained from this course... i can’t wait to go back and share all this with my co-workers. i was asked to do a presentation of my learning for my colleagues. our school is on the verge of ‘stepping it up a notch’, thanks to things learnt through this course. merci beaucoup. through this course i’ve been able to order my thoughts and find some resources to take my school to the next level with ict integration. there is so much to do, and i’m finding that i am the person to do it in my context. i have learnt sooo much and have even been able to use this new knowledge to give advise to my colleagues and guide the school into a more ict focused environment! i am truly grateful for this time to gain knowledge and experience of information technology. i will surely use these ideas with my class and share with my colleagues. all registered participants continue to have access to the resources and activities after the formal end of the course. indeed, a year after its end it is still being used. there is evidence from this experience, therefore, that although it does not support fully collaborative learning, the mooc format fits well with the objective of supporting effective co-learning for professionals, who appreciate this form of highquality learning, value each other’s experience and knowledge and are willing to share their experiences to learn together. this is not the by-product of providing video lectures and open discussion forums for general education. the distinctive pedagogic design features of this course, listed above in the section ‘a study guide’, were explicitly intended to support a relatively homogeneous group of professionals in guided colearning, and they seem to have succeeded. concluding discussion if the mooc format is to be an effective mechanism for promoting educational innovation it must be able to support a co-learning model of professional development for the community of teachers. this paper has demonstrated that by utilising the current functionality of moocs, it is possible to provide some of the critical features of a co-learning approach: research in learning technology citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 � issue-focused discussion forums that elicit valuable community discussions � peer-assessed assignments that enable teachers to learn from each other � discussion forums linked to off-platform tools for sharing resources and ideas collaborative learning in the form of small groups negotiating to develop a shared output is not supported within the current functionality of moocs, but participants still gained value from discussion of the key issues and from sharing ideas and resources in the off-platform locations, which seemed to motivate innovation and provide a sense of community exploration and shared endeavour. being able to collaborate on constructing a shared output is an important area for future mooc pedagogy development because of the high value of collaborative learning for all types of student, and as a ‘necessary foundation’ for individual learning (ludvigsen et al. 2015, p. 2). the functionality needed is to be able to: � form small groups to work together online synchronously and asynchronously � share a collaborative space for constructing a shared output, similar to a googledoc � link asynchronous text comments to specific parts of the output to iterate improvements � link the output to a forum conversation, including synchronous audio and chat, for negotiating a final version of the output these activities include learning through practice (construction and responding to feedback), discussion (comments and conversation) and production (negotiating an output for evaluation by others), making it a complex and valuable learning process. the rapid development of other digital tools and resources useful for education will certainly continue, but the few that are designed explicitly for education rarely test their effectiveness properly with target students. the responsibility for the quality of technology-based learning therefore rests entirely with teachers � both to discover the innovations that have educational potential and to develop the best ways of using them. they deserve some specialised tools for collaboration. the paper began by asking whether an open online course could succeed in bringing a co-learning model of professional development to large numbers of teachers in emerging economies. we have demonstrated that the free and open mooc platform can reach thousands of teachers, even those at primary level and even those in emerging economies. the significant improvement in participants’ understanding of ict in primary education establishes the course as a clear contribution to solving the educational challenge of integrating ict effectively into primary education the co-learning model is important because: a. this is now a growing trend in the professional development of teachers. b. it is an effective way of enabling the teaching community to collaborate on keeping abreast of the potential of new learning technologies. c. it enables them to build community pedagogic knowledge of how best to use technology. returning to the greatest educational challenge, could moocs make a contribution to achieving universal primary education? the recent unesco ‘‘education for all’’ report (unesco 2014) points out that: d. laurillard 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 education quality is undermined by the need for additional teachers, with 1.6 million required to achieve universal primary education by 2015 and 5.1 million to achieve universal lower secondary education by 2030. (p. 186) there is enough evidence from this carefully designed cpd mooc to see that there is genuine potential for this technology to engage adults in the emerging economies in a form of professional development that would be commensurate with the immense challenge of capacity building on this scale for the teaching profession across the range of skills they need. adults needing training as teachers are a different group from the professionals we were engaging with, but in many of the most challenging communities in the emerging economies there are informal and unofficial private schools being developed to fill the gaps left by the public service (oketch et al. 2010). if the teachers on an mooc could be given the support they need to run local training for these community leaders, not necessarily involving technology at the local level, then we can begin to see how online courses with 10,000 participants could match the scale of the 1.6 m teachers needed for universal primary education. if we were to address this major educational problem and ask the question ‘how can technology help?’ then mooc technology could be a solution. it is not yet good enough to provide education directly to disadvantaged students, but, as we have shown, we should now be exploiting it to develop the teachers who can. acknowledgements the books that formed the basis of the course were funded by the unesco institute of it in education. the course development was funded by the university of london international programme and the institute of education. the members of the course team were: author, professor ivan kalaš, comenius university, slovak republic; dr ernesto laval, visiting research associate, ucl, and tide s.a., chile; professor cher ping lim, the hong kong institute of education; professor florian meyer, université de sherbrooke, canada; lynn roberts, ucl institute of education, uk; professor alain senteni hamdan bin mohammed smart university, uae; natalia tokareva, unesco institute for it in education (iite), russian federation; professor márta turcsányi-szabó, eötvös loránd university, hungary. notes 1. http://www.unesco.org/new/en/education/themes/leading-the-international-agenda/education-for-all/the-efa-movement/ 2. http://www.esrc.ac.uk/funding-and-guidance/funding-opportunities/33605/esrc-dfid-raising-learning-outcomes-in-education-systems-call-2015-%2814-may%29.aspx 3. http://iite.unesco.org/ 4. http://iite.unesco.org/ 5. department for international development, uk government. 6. http://learningdesigner.org � an open online tool for creating and sharing learning designs. references avalos, b. (2011) ‘teacher professional development in teaching and teacher education over ten years’, teaching and teacher education, vol. 27, pp. 10�20. bayne, s. & ross, j. (2014) the pedagogy of the massive open online course: the uk view, higher education academy, york. butterworth, b. & laurillard, d. (2010) low numeracy and dyscalculia: identification and intervention, zdm mathematics education, special issue on cognitive neuroscience and research in learning technology citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 15 (page number not for citation purpose) http://www.unesco.org/new/en/education/themes/leading-the-international-agenda/education-for-all/the-efa-movement/ http://www.unesco.org/new/en/education/themes/leading-the-international-agenda/education-for-all/the-efa-movement/ http://www.esrc.ac.uk/funding-and-guidance/funding-opportunities/33605/esrc-dfid-raising-learning-outcomes-in-education-systems-call-2015-%2814-may%29.aspx http://www.esrc.ac.uk/funding-and-guidance/funding-opportunities/33605/esrc-dfid-raising-learning-outcomes-in-education-systems-call-2015-%2814-may%29.aspx http://iite.unesco.org/ http://iite.unesco.org/ http://learningdesigner.org http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 mathematics learning, vol. 42, no. 6, pp. 527�539. doi: http://dx.doi.org/10.1007/s11858010-0267-4 carless, d., et al., (2010) ‘developing sustainable feedback practices’, studies in higher education, vol. 36, no. 4, pp. 395�407. conole, g. (2013) moocs as disruptive technologies: strategies for enhancing the learner experience and quality of moocs, revista de educación a distancia, vol. 39, [online] available at: http://www.um.es/ead/red/39/ coursera. (2014) ‘coursera conference data’, paper presented at the coursera partners’ conference, university of london, london, uk. pp. 1�3. daniel, j. s. (2012) ‘moocs: making sense of musings in a maze of myth, paradox and possibility’, journal of interactive media in education, [online] available at: http://jime. open.ac.uk/2012/18 de waard, i. (2015) ‘mooc factors influencing teachers in formal education’, revista mexicana de bachillerato a distancia’, vol. 7, no. 13, pp. 1�8. dowker, a. d. (2009) what works for children with mathematical difficulties? the effectiveness of intervention schemes (vol. 00086-2009bkt-en), department for children, schools and families. dcsf publications, nottingham, uk. griffin, s. (2004) ‘building number sense with number worlds: a mathematics program for young children’, early childhood research quarterly, vol. 19, no. 2, pp. 173�180. ho, a. d., et al., (2014) harvardx and mitx: the first year of open online courses harvardx and mitx working paper no. 1, mit office of digital learning, boston, usa. hökkä, p. & eteläpelto, a. (2014) ‘seeking new perspectives on the development of teacher education: a study of the finnish context’, journal of teacher education, vol. 65, no. 1, pp. 39�52. hollands, f. & tirthali, d. (2014) moocs: expectations and reality, teachers college, columbia university, new york. jobe, w., östlund, c. & svensson, l. (2014) ‘moocs for professional teacher development’, in proceedings of society for information technology & teacher education international conference 2014, eds m. searson & m. ochoa, aace, chesapeake, va. pp. 1580�1586. kop, r., fournier, h. & mak, j. (2011) ‘a pedagogy of abundance or a pedagogy to support human beings? participant support on massive open online courses’, international review of research in open and distance learning, vol. 12, no. 7, pp. 74�93. kalaš, i., et al., (eds) (2012) ict in primary education. analytical study. volume 1: exploring the origins, settings and initiatives, unesco institute for information technology in education, moscow, russian federation. kalaš, i. et al., (2014) ict in primary education: volume 2 policy, practices, and recommendations, unesco institute for information technologies in education, moscow, russian federation. knight, p., tait, j. & yorke, m. (2006) ‘the professional learning of teachers in higher education’, studies in higher education, vol. 31, no. 3, pp. 319�339. kolb, d. a. (1984) experiential learning: experience as the source of learning and development, prentice-hall, englewood cliffs, nj. laurillard, d. (2009) ‘the pedagogical challenges to collaborative technologies’, international journal of computer-supported collaborative learning, vol. 4, no. 1, pp. 5�20. laurillard, d. (2012) teaching as a design science: building pedagogical patterns for learning and technology, routledge, new york. laurillard, d. (2014) ‘hits and myths: moocs may be a wonderful idea, but they’re not viable’, times higher education, 16 jan. p. 28. lim, c. p. et al., (2014) ict in primary education. analytical survey volume 3: collective case study of promising practices, unesco institute for information technologies in education, moscow, russian federation. liyanagunawardena, t., adams, a. & williams, s. (2013) ‘moocs: a systematic study of the published literature 2008�2012’, the international review of research in open and distance learning, vol. 14, no. 3, pp. 203�227. ludvigsen, s., et al., (2015) ‘collaboration and the formation of new knowledge artifacts’, international journal of computer-supported collaborative learning, vol. 10, no. 1, pp. 1�6. luo, h., robinson, a. c. & park, j. y. (2014) ‘peer grading in a mooc: reliability, validity, and perceived effects’, journal of asynchronous learning network, vol. 18, no. 2, pp. 1�14. d. laurillard 16 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 http://dx.doi.org/10.1007/s11858-010-0267-4 http://dx.doi.org/10.1007/s11858-010-0267-4 http://www.um.es/ead/red/39/ http://jime.open.ac.uk/2012/18 http://jime.open.ac.uk/2012/18 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 nafsa. (2010) the changing landscape of global higher education, association of international educators, washington, dc. oketch, m., et al., (2010) ‘why are there proportionately more poor pupils enrolled in nonstate schools in urban kenya in spite of fpe policy? international journal of educational development, vol. 30, pp. 23�32. papert, s. (1980) mindstorms: children, computers, and powerful ideas, the harvester press, brighton. robertson, j. & howells, c. (2008) ‘computer game design: opportunities for successful learning’, computers & education vol. 50, pp. 559�578. smeets, e. (2005) ‘does ict contribute to powerful learning environments in primary education? computers & education, vol. 44, pp. 343�355. unesco. (2014) efa global monitoring report: teaching and learning � achieving quality for all efa global monitoring reports, unesco, paris, france. vygotsky, l. s. (1978) mind in society: the development of higher psychological processes, harvard university press, cambridge, ma. wertsch, j. v. (1985) cultural, communication, and cognition: vygotskian perspectives, cambridge university press, cambridge. research in learning technology citation: research in learning technology 2016, 24: 29369 http://dx.doi.org/10.3402/rlt.v24.29369 17 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29369 http://dx.doi.org/10.3402/rlt.v24.29369 doi:10.3402/rlt.v20i0/14430 cross-discipline investigation of the relationship between academic performance and online resource access by distance education students andrea crampton a, *, angela t. ragusa b and heather cavanagh a a faculty of science, charles sturt university, wagga, wagga, australia; b school of humanities and social science, charles sturt university, wagga, wagga, australia (received 14 march 2011; final version received 30 june 2011) educational technology implementation often owes more to the technical proficiency of the teaching staff and/or the capacity of the institution than to a student outcome-centred design process. creation of online resources takes considerable time and involves significant cost to both the institution, for devices and platforms, and to students for devices and internet connectivity charges. here, we present a cross-discipline investigation of student engagement with a range of simple resources. our aim was to determining if the provision of such resources had an impact on student academic performance regardless of the students’ level of academic proficiency. this research focused on students studying first-year introductory subjects at a distance (off campus) from two different faculties, arts and science. analysis of the web access data from the learning management system (sakai) demonstrated that students who accessed the most resources in terms of diversity and percentage of available resources achieved higher grades. we postulate that the resources prompted students to spend more ‘‘time-on-task’’ and facilitate more active styles of learning. we suggest, however, that students need to be made aware of the value of the resources and how they are best used to enhance academic performance. keywords: distance education; online resources; academic performance; virtual learning environments; e-learning introduction the shift from an industrial to an information-driven society (castells 2000) has substantially changed education delivery, often facilitated by the use of internetdriven communications that are a recent, yet foundational, component of many educational systems (schifter 2004). the globalisation of world economies and internationalisation of curricula (barjis 2003) fostered an environment, whereby universities increasingly are internet dependent. as e-learning supplements, and even supplants, traditional classroom learning environments (ragusa 2009), universities compete to lead in flexible delivery, to achieve market dominance and to meet government policy objectives (ragusa 2007). in response, higher education educators are encouraged to augment online delivery resources and align their teaching practice with the needs and expectations of twenty-first century learners and markets. (page number not for citation purpose) *corresponding author. email: acrampton@csu.edu.au research in learning technology vol. 20, 2012 rlt 2012. # 2012 a. crampton et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 14430 doi: 10.3402/rlt.v20i0.14430 http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 technology-infused distance education is the fastest growing sector in education (tennant, mcmullen, and kaczynski 2009). a range of socio-economic factors is responsible for this growth. distance education requires less physical infrastructure and resources and can offer students access to higher education for a reduced living cost (betts, hartman, and oxholm 2009). distance education enables institutions to access a global student market (tennant, mcmullen, and kaczynski 2009) that extends to non-traditional populations, particularly those with family and/or work obligations that prevent attendance at university campuses (betts et al. 2009). furthermore, distance education is material to the heightened global interest in higher education and the growth in international students (ragusa and steinke 2011). these economic realities, coupled with the growth in size and acceptance of online education, and the introduction of widely available, public access, educationassociated social media (e.g. itunesu 1 ) clearly signal that technology-dependent learning is as much a part of twenty-first century education as the chalk board was in classrooms of yesteryear. the changes fuelled by these economic realities, however, are underscored by a deeper socio-historical shift. in contrast with past societies’ reliance on face-to-face communication, rapid changes in how we interact today, ‘‘for the first time in human history’’ (giddens 1984, 185), made possible by technology led us to shift from a local to inherently global social organisation. giddens’ later works (1991, 1994, 2002) helped us to theoretically frame how education pedagogies, informed by communicative technologies, exist in a social order characterised by different traditions, identities, expertise and global relations. epistemic and technical shifts in how individuals relate to one another have, however, raised some concerns in the higher education sector, including how the mechanism for social interaction and communication affects learning outcomes. questions regarding the effects such changes have on the educational experience of students are but one example of issues facing contemporary consumers of knowledge in societies, where specialised expert knowledge replaces traditions to grant or deny power and authority (giddens 1991). alongside, macro level social changes are relentless micro level, everyday shifts that determine our technical proficiency. in the case of education, the focus on technical solutions can translate into poorly designed or executed resources that waste both educators’ and students’ time, the latter of which might have been better spent on traditional learning tasks, such as textbook reading. conversely, educators who are less enthusiastic about educational technologies may feel that their expertise and authority are threatened by a lack of technical command in an increasingly technical environment. given the ever-evolving plethora of educational technology platforms, tools and resources (‘‘bells and whistles’’) presented to the academic community, it is easy to lose sight of core educational goals. a specific educational goal that we investigate in this article is the enhancement of learning environments to facilitate students’ acquisition of substantive and tacit knowledge. a growing body of educational technology research, however, is beginning to question whether the twenty-first century bells and whistles enabled by educational technologies drown, or at least distract from, sound pedagogy. according to mcfarlin (2008), for educational technology to be effective, student learning and outcomes must be the focus of its implementation and use. pedagogies with technologies driven by learning outcomes are increasingly noted in the literature (see tennant, mcmullen, and kaczynski 2009) with larson and chung-hsien (2009) a. crampton et al. 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 concluding that the delivery mode is irrelevant so long as the instructor applies ‘‘best practices’’. the present study seeks to fill knowledge gaps in the research literature by examining how and if the use of pedagogically driven technologies enhances or detracts from higher education delivery in two large and diverse introductory subjects at an australian university. by providing an evaluation of online communication and resources used by distance students in a range of natural and social science courses, as well as the humanities, our research sample includes students with widely varying pre-tertiary performance abilities that we term ‘perceived academic capacity’. background the resources and communication tools used to teach in two subjects, introductory microbiology (mic) and introductory sociology (soc), provided students with dynamic learning environments not traditionally available to geographically dispersed distance students. the design and application of learning resources mimicking those of traditional face-to-face classrooms, we argue, represents real progress, not the step sideways that those critiquing blended learning approaches claim (morrison 2003; tremblay 2006). traditionally, students studying by distance education (the ‘‘de cohort’’ of each course) receive a paper-based subject outline containing details of assessment procedures, a recommended study pattern and a study guide with key explanations from the textbook and some additional figures and explanations written by the teaching staff. here, we examine the effectiveness of the use of virtual lectures, access to lecture handouts and student use of asynchronous forums on the education outcomes of students as measured by their final grade and analysed by measuring activity logs from the learning platform for the entire teaching session. due to large enrolment numbers (soc n �145 and mic n �221), the online space mirrored a more traditional didactic classroom, where students passively received information, albeit via virtual lectures (narrated slides) or electronic handouts, rather than live lectures or tutorials. this structure is commensurable with other initial forays into online education (tennant, mcmullen, and kaczynski 2009) that rely on familiar, normative pedagogies (i.e. we start with what we know) and are based on materials developed for face-to-face environments (salter, pang, and shrama 2009). both subjects used an electronic forum structured similarly to the ‘‘question and answer’’ sessions that often accompany face-to-face lectures and/or tutorials. unlike many other studies of online or blended learning, we examine subjects previously taught only by correspondence (see schifter 2004), where students had no opportunity to attend lectures, with the exception of a compulsory 4-day residential school on campus for the microbiology students to learn practical laboratory skills. while the provision of online resources replicating traditional classrooms may not initially seem conducive to increasing student engagement, the platform context (i.e. resource hosting plus synchronous and asynchronous communication tools) enhances opportunities for student interaction with the content and with each other beyond those possible in conventional classrooms. the online platform also allows distance education students to interact with on-campus students. therefore, provision of online resources, with their capacity to facilitate interactions, can be viewed as an intermediate step between lecture-based delivery and the proposed ideal, student-centred teaching model designed around active or problem-based research in learning technology citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 3 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 learning (exeter et al. 2010). this transitional step is, perhaps, a suitable option to engage first-year students, most of whom enter university environments from the more traditionally didactic background of secondary schooling, although this is likely to change in the future as online learning becomes more prominent in the high school system (drabinski, clark, and roberts 2011). with universities’ increased reliance on self-directed learning, this transitional step may assume even greater importance, particularly for students studying by distance. clearly structured higher education learning environments, with resources such as audio lectures or lecture notes, can provide students clear signposts likely to further facilitate transition towards self-directed learning. this transition step will no doubt evolve as pedagogic practices of secondary schools and other educational institutes that increasingly adopt the use of blended learning. movement from an online environment replicating traditional classrooms towards an effective blended environment with active learning and engagement requires professional development for teaching staff and training for students (salter, pang, and sharma 2009). thus, the re-socialisation of staff and students is a necessity and likely to be affected by familiarity and use of blended learning in prior organisational settings. the learning environments examined were not open learning environments as defined by clarebout and elen (2008); students were neither able to manipulate their learning environment nor presented with meaningful problems to solve. as distance students, a high degree of self-regulation was assumed and considered necessary for academic success (barnard-brak, lan, and paton 2010). the degree of selfdirectedness, or self-regulation, required from distance learners is a major risk factor for attrition (rovai and downey 2010). while direct measures of selfregulation have been applied in educational settings (biesinger and crippen 2010), sociological studies focused on how interventions, such as the structure and design of learning environments, affect learning outcomes, are required to complement studies of psychological characteristics such as those noted above. robinson’s (2007) sociological study, for example, noted how giddens’ structuration theory can inform e-learning design and the value of providing multiple communication tools to students as a way of enhancing students’ agency in affecting their academic goals. most modern learning management systems (lms) provide at least one simple measure of student access to, and use of, learning resources. use of learning resources offers an indicator of students’ capacity for self-regulation accessible by an audit file, most commonly in the form of a log tool. web logs, as indicators of students’ study practices, have been used in a variety of ways. our research builds on prior research, such as judd and kennedy’s (2010), 5 year analysis of web use at dedicated computers, which found that students predominantly accessed the university’s core systems and learning resources posted through the internal system rather than external sites and tools. this behaviour, however, was exhibited only by advanced students because most courses utilised the university’s lms (judd and kennedy 2010). log files were also used to investigate the pace of 6112 students’ learning in a fully online environment to examine resource type and use in relation to time of day and duration of access (hershkovitz and nachmias 2009) as well as to map embedded versus non-embedded support tool use and student self-regulation (clarebout, et al. 2010). generally, the use of log files allows researchers to examine ‘‘real world’’ practices (schrader and lawless 2007), instead of conducting lab-based scenarios and other a. crampton et al. 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 more intrusive instruments such as surveys. hence, this technique is ideal for exploratory analyses of students’ lms use, permitting creation of appropriate interventions or variations in design that can improve learning resource effectiveness. log files matched with assessment of learning outcomes, such as grades, are useful for examining students’ initial resource use to see how and if this affects academic performance as well to explore the impact of intervention strategies applied during the teaching session. moreover, the collection of log data is suitable to large classes for which surveys, focus groups and other more intrusive or intensive methodologies are less feasible. methods the primary research aim was to examine the effect of providing online learning resources on students’ academic performance. to minimise the impact of subject type and student cohort, we chose subjects from two different faculties containing students with varied levels of pre-tertiary academic performance and preparation for university study. secondary data, available from university subject site logs, student grades and learning forum logs, were compiled to reflect student utilisation of online learning resources and quantitatively analysed. sample and design the research sample consisted of two large, first-year introductory subjects taught between august and december 2009. the subjects spanned two divergent faculties, science and arts. the first subject, soc, was taught by distance only, contained 145 students and was primarily taught by a teaching assistant under the management of a senior academic who designed and produced the subject content and learning resources. the second subject, mic, was taught both internally (on campus) at two different locations (400 km apart) and simultaneously by distance education, containing a total of 593 students. mic was taught by a single academic at one campus, and by nine academics at the other. all cohorts were supported by a single online platform managed by the academic staff. the 221 students studying mic by distance travelled to one campus for a 4-day residential school to complete the laboratory component of the subject, whilst the soc students were not required to attend any on-campus sessions. the assessment requirements varied with each subject. soc students were required to complete three assessment items: an online mid-semester multiple choice exam, a 1000 word essay and an invigilated paper-based exam at the end of the semester. mic students were assessed by weekly quizzes on laboratory topics, a midsemester and an end-of-semester exam, both of which contained multiple choice and short answer questions. this investigation examines the distance students enrolled in both of these subjects. online platform and learning resources both mic and soc used a customised version of the sakai platform, termed ‘interact’, plus an additional forum tool for asynchronous communication. in both subjects, the forums were divided into a ‘‘general’’ forum and ‘‘sub-forums’’ that related to each topic taught (9 for mic and 11 for soc). administrative tasks, such research in learning technology citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 5 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 as the organisation of residential schools (mic) and general queries (soc), were managed utilising the general subject forum. students in soc were provided with powerpoint slides summarising the chapters of the textbook and providing additional sociological information. these slides were linked to separate virtual lectures produced using adobe captivate # (v3). the virtual lectures were provided in both shockwave and.exe formats and contained the lecturer’s narration of key introductory concepts. mic students were also provided powerpoint slides. their slides, however, were the same as those presented to the subject’s internal students during campus lectures and textbook-based tutorial exercises. both subjects also provided the students with a printed subject outline and study guides (figure 1). analysis of resource use the sakai platform included a log tool (sitestats) used to generate data about which resources were accessed by each student. it is important to note that sitestats only records access to the resources and not use of the resource. hence, some resources may have been accessed but not viewed. actual reported use of the resources was not assessed by a student survey because of the typically low student survey response rates at the university being sampled. use of secondary data permitted reliable analysis of the entire sample’s performance and each student’s resource access, rather than perceived use from social surveys. students’ forum participation was analysed by counting the number of messages each student posted on each forum. furthermore, overall forum use was calculated using an inbuilt statistics tool for forum evaluation. analysis of individual student use of forums and resources was conducted after final grades were announced; a procedure approved by the school of biomedical science’s human ethics committee, approval 406/2009/12. student resource use and grades were entered into cohort-specific spss data files and analysed separately by subject to produce descriptive statistics and correlations. t-tests were conducted with significantly correlated (p �b0.05) variables using the independent sample t-test with levene’s test for equality of variances, where equal variances were assumed. resource type mic soc virtual lectures (voice over powerpoint slides created by lecturer) x pdf of lecture slides x x pdf or word files with tutorial activities (i.e. quizzes, games, key questions) x general discussion forum x x content-specific forums x (nine) x (eleven) figure 1. types of resources provided to each cohort of students. a. crampton et al. 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 results one hundred and nine soc students and 181 mic distance students completed all the requisite tasks to receive a final grade. descriptive statistics revealed that 51% of mic and 46% of soc students posted messages on the general forums. only 8% of the soc students and 20% of the mic students posted messages on topically related sub-forums. at least one virtual lecture was downloaded by 71% of the soc students and 12% downloaded all virtual lectures. the powerpoint slide files were accessed by 81% of the soc students, with 9% downloading all of them. five students (5%) accessed at least one slide file but none of the virtual lectures. conversely, 14 students (13%) accessed at least one virtual lecture but none of the slide files. lecture slides were downloaded by 88% of the mic students, with most of those (84%) also accessing at least one tutorial resource. sixteen percent of the mic students accessed all available tutorial resources. in mic, weak yet significant positive correlations (i.e. the higher a student’s grade, the more resources he or she accessed) were found between a student’s final grade and his or her downloading lecture slides (r �0.28; p �0.00) or accessing tutorial resources (r �0.31; p �0.00) and/or posting to the content forums (r �0.34; p �0.05). in soc, a moderately strong and significant correlation existed between students accessing the virtual lectures and their final grade (r �0.40; p �0.00). in other words, those students in both subjects who downloaded electronic learning resources were more likely to receive higher grades. no significant correlations were observed between students’ contribution to subject forums and final grades in soc, indicating that lecture slides may be of greater utility to learning that subject’s content than forum communication. it is important, however, to note that correlations do not determine causality. therefore, to test if the correlation between learning material usage and final grades is statistically meaningful, parametric tests of significance were conducted. after tests for normality confirmed that the samples were normally distributed, two-tailed independent groups t-tests were conducted to test the means between the two subjects related to resource use and final grades. findings revealed a significant difference in the final grades between soc students who received at least a pass (m � 9.40, sd �4.59) and those who failed (m �5.24, sd �4.72) related to the access of virtual lectures t(133) �5.22, p �0.00. in other words, the more virtual lectures accessed, the higher soc students’ final grades. these significant relationships are evident in figures 2, 3 and 5. similar analyses of resource use by the mic students revealed four significant relationships, the strongest related to students’ access of tutorial resources and general forum usage, the latter in contrast to the soc students. students who failed in the subject accessed fewer lecture slides (m �8.75, sd �3.83), t(156) �2.020, p � 0.05, than students who received at least a passing final grade(m �10.45, sd �4.09). students with at least a passing final grade (m �9.98, sd �12.98) posted more messages on the general forum, t(89) �1.95, p �0.00, than students who failed (m � 3.14, sd �3.23). students who failed in the subject also accessed (m �1.00, sd � 0.67) the subject content forums less frequently, t(40) �2.05, p �0.05, than students with a passing grade or better (m �2.34, sd �2.03). finally, students with at least a passing final grade accessed more of the tutorial resources (m �12.95, sd �6.77), t(150) �2829, p �0.01, than students who failed (m �8.75, sd �3.83). research in learning technology citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 7 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 discussion our research reveals that the provision of online resources to distance education students supports learning outcomes and improves academic performance. the finding that the majority of students accessed lecture slides was anticipated figure 3. percentage of students who posted a message on either the general or contentspecific forums, relevant to their final grade. fl, fail; ps, pass; cr, credit; di, distinction; hd, high distinction. figure 2. average percentage of an available resource type accessed by students relative to their final grade. fl, fail; ps, pass; cr, credit; di, distinction; hd, high distinction. a. crampton et al. 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 because such activity replicates face-to-face classroom practices, such as note taking. figure 4, however, presents an unanticipated finding: soc students eventually stopped downloading lecture slides but continued to access the virtual lectures that represent a much larger commitment in terms of time and download time/cost. students’ preference to use virtual lectures alone, particularly as audio files in a mp3 format, rather than powerpoint slides coupled with audio, reveals a shift in learning preference made possible by recent advances in communication technology. student comments from subject evaluations revealed that mp3 files were preferred because of their transportability and use in multi-tasking scenarios such as while driving, commuting, doing household tasks, etc. given the poverty of time experienced by many distance education students, it is possible that those who accessed the mp3 files were more likely to use them, than those who only accessed the powerpoint slides. accessing virtual lectures fostered better comprehension of subject content and concepts as evidenced by higher academic achievement (see figures 2 and 5) than accessing powerpoint slides alone. the virtual lectures ensured students who are audio learners, or potentially just busy, could engage with the content in a manner that fitted their learning style and/or lifestyle; a delivery mode unavailable to distance students because institutions stopped posting correspondence courses on tape. the degree to which students access online learning opportunities is a potential indicator of how prepared they are for self-regulated education fostered by online platforms. for some students, however, the transition from seeking knowledge out of casual interest to taking a disciplined approach towards reaching a concrete learning goal, such as a final grade, may be less readily adopted during their first year of university study if left entirely up to their independent pursuit. help-seeking behaviour is regarded as a significant component of effective self-regulated learning. figure 4. download patterns for virtual lectures and slides by 101 soc students during the 12 weeks of the semester. research in learning technology citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 9 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 students who choose to study via distance education are expected to have a high degree of self-efficiency (barnard-brak, lan, and paton 2010) and be able to regulate their learning environment around other life issues. indeed, barnard-brak, lan and paton (2010) noted even competent self-regulators who ‘‘do what it takes . . ..to achieve well in their learning but they don’t do much more’’ (barnard-brak, lan, and paton 2010, 72) are able to achieve academic success. our results indicate students who actively sought help and/or clarification for issues, by way of forum postings, achieved higher levels of academic success (figure 3). that is not to say students who read forums without posting contributions, otherwise known as ‘‘lurkers’’ (bax and pegrum 2010), are less effective at self-regulated learning; ‘‘lurking’’, as prior sociocultural research found, may simply reveal that help-seeking needs can be met by passive engagement. our focus on first-year subjects demonstrates that students may be affected by their level of academic socialisation. this supports prior research findings that first-year students may not understand the value of support mechanisms provided or have the capacity to use them adequately (azevedo, cromley, and seilbert 2004). identification of key attributes used by skilled self-regulators is one way universities can develop better support and training systems, particularly for nonself-regulators. enhanced support and training may lead to better outcomes such as academic performance and completion rates. many current learning environments that give students control over access to support devices require them to be good selfregulators and aware of the full potential of the provided resources, before they can adequately access or identify any support they may need (see clarebout et al. 2010). enhancing the support provided to help students become better self-regulated learners from the time they enter university is crucial to address the long-term nature of goal orientation and self-regulation, which requires more than one teaching session to affect (biesinger and crippen 2010). the role of the educator is changing figure 5. percentage of students who accessed at least one of the available resources relevant to their final grade. fl, fail; ps, pass; cr, credit; di, distinction; hd, high distinction. a. crampton et al. 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 from one who simply possesses and imparts knowledge to one who facilitates the development of learning skills that show students how to obtain and interpret knowledge (i.e. the shift from didactic styles of early e-learning to c-learning 2 ) (ledzinska and postek 2010; tan, wang, and xiao 2010). we need to consider the implications of changing roles when examining the development and use of nontraditional learning tools. the overestimation of students’ capacity to use the technology provided appropriately (kvavik 2005; nasah et al. 2010) is an area of particular concern and a possible confounding factor in how the resources in our study were applied. furthermore, lack of appropriate technical support for distance students studying online is a major contributor to attrition (rovai and downey 2010) and a common criticism noted by distance students at our institution. accessing tutorial resources (mic) or virtual lectures (soc) was positively correlated with improved academic performance as illustrated in figure 2. both of these resource types simulate the same key practice, namely time on task (i.e. the amount of time a student spends engaged with a resource or activity). grimstad and grabe (2004) noted that students who accessed online tutorial exercises (in their case quizzes) performed better academically than those who did not. they concluded that although quizzes gave students experience with the types of questions in the examination, it was the time spent on the quizzes that possibly explained their correlation with academic success (grimstad and grabe 2004). similarly, kuh (2004) noted, in relation to student development, that the amount of time devoted to ‘‘educationally purposeful activities’’ was the best predictor of students’ learning outcomes. in regards to microbiology, johnson (2008) found a significant correlation between academic performance and use of online resources (again quizzes), attributing that time-on-task may be a key component of the association. furthermore, a meta-analysis of online learning studies conducted by the us department of education concluded that time-on-task was the only statistically significant aspect of online learning affecting student performance, noting that online students who spent more time-on-task fared better than their counterparts studying in the traditional face-to-face mode (us department of education 2009). our research adds further support that purposive online learning activities positively affect student performance. the online tutorials and virtual lectures provided students in our study a vehicle to interact with each subject’s content in more than one way, encouraging active learning via completing tutorial exercises and discussion questions. use of these resources required students either to work through problems or to answer questions that involved connecting content and concepts in a manner not presented in the core materials provided. active learning promotes better engagement with subject matter (windham 2005), an important factor in student success and retention (sheard, carbone, and hurst 2010). nevertheless, the value of active learning, and its effect on student outcomes, remains a contentious, unresolved issue with the perceived benefits potentially attributable to a time-on-task effect as previously noted (us department of education 2009). conclusion our research evidenced that the provision of low-cost and easy-to-create online resources quantitatively enhanced students’ academic performance: students who accessed the online resources achieved greater academic success. this is notable given research in learning technology citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 11 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 that the institutional cost of providing the supplemental electronic resources was limited to the purchase price of the software (adobe captivate # v.3) and a limited time investment for staff training with minimal impact on staff time. for example, apart from up-skilling in software use, mic staff required no additional preparation time, as they simply uploaded online versions of resources already created for the face-to-face class. conversely, soc staff spent an additional hour per week to create the lecture slides, record them and upload them; however, these resources could then be reused for multiple sessions. the relationship between resource access and academic success was related to both the diversity of available resources and the proportion of each type accessed by the student. such findings may be informed by the sociological theory of giddens, specifically his work on culture and structuration theory that, as contextualised by robinson (2007), suggests that the provision of a range of resources enhances an educator’s capacity to provide a diverse range of learners with adequate agency to devise their most effective learning environments. to make the most use of such agency, however, the users must first be able to adequately engage with the resource/ technology and be aware of its potential value. therefore, we suggest that course developers would be well advised to ensure that first-year students are adequately trained, and directed towards the use of key learning resources as well as informed of the relationship between resource use and grades. furthermore, as resource utilisation may be linked to students’ proficiency to self-regulate their learning, as suggested by prior research (azevedo, cromley, and seilbert 2004; barnard-brak, lan, and paton 2010;). therefore, mechanisms for assisting students with their development of self-directed study skills should be included in learning environment design. future studies would do well to look at students’ demographic attributes, such as age, gender, cultural background and socio-economic status, as well as their time constraints (i.e. combining study with employment or caring responsibilities). additionally, student and educators’ familiarity with, and preference for engaging with, online resources should be identified. such knowledge would further enhance educators’ capacity to develop, implement and promote the most suitable resources that are both relevant and costand time-effective. adequate awareness of the digital literacy levels of academic staff would also help institutions develop professional development programmes that address all levels of staff capacity while maximising the effective use of virtual learning environments. the ‘‘bells and whistles’’ provided music to some soc students’ ears and visual guidance to many mic students’ eyes, yet regardless of the tool implemented, it was utilisation of resources, rather than technology type that positively affected many students’ academic aspirations and fostered engagement in the learning process. notes 1. a subsection of itunes with free recorded lectures from many prestigious universities. 2. ‘‘c-learning is most often defined as a learning process happening in virtual surrounding, in which the teacher assumes the role of not a source of knowledge, but rather a source of metaskills needed to operate on that knowledge’’ (ledzinska and postek 2010, 2). a. crampton et al. 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 references azevedo, r., cromley, j. g. & seibert, d. (2004) ‘does adaptive scaffolding facilitate students’ ability to regulate their learning with hypermedia?’, contemporary educational psychology, vol. 29, no. 3, pp. 344�370. barjis, j. (2003) ‘an overview of virtual university studies’, in virtual education, ed f. albalooshi. irm, hershey, pa, pp. 1�15. barnard-brak, l., lan, w. y. & paton, v. o. (2010) ‘profiles in self-regulated learning in the online learning environment’, international review of research in open and distance learning, vol. 11, no. 1, pp. 61�79. bax, s. & pegrum, m. (2010) ‘lurking in multicultural online educational forums: ‘‘i wasn’t invited to the party’’’, in interaction in communication technologies and virtual learning environments:human factors, ed t. ragusa. information science reference, hershey, pa, pp. 145�159. betts, k., hartman, k. & oxholm, c. iii. (2009) ‘re-examining and repositioning higher education: twenty economic and demographic factors driving online and blended program enrolments’, journal of asynchronous learning networks, vol. 13, no. 4, pp. 3�23. biesinger, k. & crippen, k. (2010) ‘the effects of feedback protocol on self-regulated learning in a web-based worked example learning environment’, computers and education, vol. 55, no. 4, pp. 1470�1482. castells, m. (2000) the information age. 3 vols. blackwell, oxford. clarebout, g. & elen, j. (2008) ‘tool use in open learning environments: in search of learner related determinants’, learning environment research, vol. 11, no. 2, pp. 163�178. clarebout, g. et al. (2010) ‘the relationship between self-regulation and embedding of support in learning environments’, education technology research development, vol. 58, no. 5, pp. 573�587. drabinski, e., elizabeth clark, j. & sarah, t. r. (2011) ‘introduction: shaped or shaping? the role for radical teachers in teaching with technology’, radical teacher, vol. 90, pp. 3�8. exeter, d. j. et al. (2010) ‘student engagement in very large classes: the teachers’ perspective’, studies in higher education, vol. 35, no. 7, pp. 761�7005. giddens, a. (1984) the constitution of society: outline of the theory of structuration, polity press, cambridge. giddens, a. (1991) modernity and self-identity: self and society in the late modern age, polity press, cambridge. giddens, a. (1994) ‘living in a post-traditional society’, in reflexive modernization: politics, tradition and aesthetics in the modern social order, eds u. beck, a. giddens & s. lash polity press: cambridge, pp. 56�109. giddens, a. (2002) runaway world: how globalisation is reshaping our lives, 2nd edn. profile books, london. grimstad, k. & grabe, m. (2004) ‘are online study questions beneficial?’, teaching in psychology, vol. 31, no. 2, pp. 143�146. hershkovitz, a. & nachmias, r. (2009). ‘consistency of students pace in online learning’, in educational data mining 2009: 2nd international conference on educational data mining, proceedings, eds t. barnes, m. desmarais, c. romero, & s. ventura, pp. 71�80. cordoba, spain. johnson, m. t. (2008) ‘impact of online learning modules on medical student microbiology examinations scores’, journal of microbiology and biology education, vol. 9, pp. 91. judd, t. & kennedy, g. (2010) ‘a five-year study of on-campus internet use by undergraduate biomedical students’, computers and education, vol. 55, no. 4, pp. 1564�1571. kuh, g. d. (2004) ‘the national survey of student engagement: conceptual framework and overview of psychometric properties (national survey of student engagement)’, available at: http://nsse.iub.edu/2004_annual_report/pdf/2004_conceptual_framwork.pdf kvavik, r. (2005). convenience, communications and control: how students use technology, in educating the net generation, eds d. g. oblinger & j. l. oblinger, educause. larson, d. k. & chung.-hsien, s. (2009) ‘comparing student performance: online versus blended versus face-to-face’, journal of asynchronous learning networks, vol. 13, no. 1, pp. 31�42. research in learning technology citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 13 (page number not for citation purpose) http://nsse.iub.edu/2004_annual_report/pdf/2004_conceptual_framwork.pdf http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 ledzinska, m. & postek, s. (2010) ‘the role of metacognition in c-learning curriculum’, problems of education in the 21st century, vol. 23, pp. 137�145. mcfarlin, b. k. (2008) ‘hybrid lecture-online format increases student grades in an undergraduate exercise physiology course at a large urban university’, advanced physiology education, vol. 32, no. 1, pp. 86�91. morrison, d. (2003) e-learning strategies: how to get implementation and delivery right first time, john wiley & sons, london. nasah, a., boaventura, d., carolyn, k., and soonhwa, s. (2010). ‘the digital literacy debate: investigation of digital propensity and information and communication technology’, education technology research development, vol. 58, no. 5, pp. 531�555. ragusa, a. t. & steinke, e. (2012) ‘studying locally, interacting globally: demographic change and international students in australian higher education’, in computer-mediated communication across cultures: international interactions in online environments, eds k. st amant & s. kelseyigi, hershey, pa, pp. 344�368. ragusa, a. t. (2009) ‘sociological insights in structuring australian distance education’, in encyclopaedia of information science and technology, 2nd edn, ed m. khosrow-pour, information science reference, london, pp. 3513�3519. ragusa, a. t. (2007) ‘the impact of socio-cultural factors in multi-cultural virtual communication environments: a case example from an australian university’s provision of distance education in the global classroom’, in linguistic and cultural online communication issues in the global age, ed. k. st amant, idea group, hershey, pa, pp. 306�327. robinson, k. (2007) ‘configuring virtual learning environments to support diversity and intercultural learning’, paper presented at education in a changing environment conference, september 12�14, in manchester, uk. rovai, a. p. & downey, j. r. (2010) ‘why some distance education programs fail while others succeed in a global environment’, internet and higher education, vol. 13, no. 3, pp. 141� 147. salter, d., pang, m. y. c., & sharma, p. (2009) ‘active tasks to chains use of class time within and outcomes-based approach to curriculum design’, journal of university teaching and learning practice, vol. 6, no. 1: article 4. available at: http://ro.uow.edu.au/jutlp/vol6/iss1/4 schrader, p. g. & lawless, k. a. (2007) ‘dribble files: methodologies to evaluate learning and performance in complex environments’, performance improvement, vol. 46, no. 1, pp. 40� 48. schifter, c. (2004) ‘faculty participation in de programs: practices and plans’, in the distance education evolution: issues and case studies, eds d. monolescu, c. c. schifter & l. greenwood, information science publishing, hershey, pa, pp. 1�21. sheard, j., carbone, a. & hurst, a. j. (2010) ‘student engagement in first year of an ict degree: staff and student perceptions’, computer science education, vol. 20, no. 1, pp. 1�16. tan, l., wang, m. & xiao, j. (2010) ‘best practices in teaching online or hybrid courses: a synthesis of principles’, lecture notes in computer science, vol. 6248, pp. 117�126. tennant, m., mcmullen, c., & kaczynski, d. (2009) ‘learning in the digital age’, in teaching, learning and research in higher education, routledge, hoboken. tremblay, r. (2006) ‘‘‘best practices’’ and collaborative software in online teaching’, the international review of research in open and distance learning, vol. 7, no. 1. available at: http://www.irrodl.org/index.php/irrodl/article/view/309/486 u.s. department of education, office of planning, evaluation, and policy development. (2009) evaluation of evidence-based practices in online learning: a meta-analysis and review of online learning studies. us department of education, washington, dc. available at: http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf windham, c. (2005) ‘the students’ perspective’, in educating the net generation, eds d. oblinger & j. oblinger, e-educause, washington, dc, pp. 5.1�5.16. a. crampton et al. 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14430 doi: 10.3402/rlt.v20i0.14430 http://ro.uow.edu.au/jutlp/vol6/iss1/4 http://www.irrodl.org/index.php/irrodl/article/view/309/486 http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf http://researchinlearningtechnology.net/index.php/rlt/article/view/14430 reading with new tools: an evaluation of personal digital assistants as tools for reading course materials jenny waycott institute of educational technology, the open university email: j.lwaycott@open.ac.uk lightweight, palmtop devices such as personal digital assistants (pdas) can now be used for reading electronic text, opening up their potential as learning tools. this paper reports a study that evaluated the use of pdas for reading course materials by students on an open university master's course. the research is grounded in activity theory, which provides a useful framework for examining how the introduction of a new tool changes an existing activity. student perceptions of the possibilities and constraints of the pda, as determined by questionnaires and interviews, reveal the impact the new tool had upon reading. the pda constrained reading with limitations such as the small screen size, new requirements for navigating through the text and awkward methods for taking notes. these conditions made it difficult for students to skim-read the text, to move back and forth within the document and to interact with the text as easily as they could with paper. nevertheless, students welcomed the opportunity to have the course materials on a portable, lightweight device that could be used at any time and in any place. this made it easier to fit the reading activity around the various other activities in which students were involved in addition, the pda was used in conjunction with existing tools, such as the printed version of the course materials and the desktop computer. therefore, it was not seen to replace paper but rather to extend and complement it. the findings are discussed using concepts from activity theory to interpret how the new tool modified the reading activity. introduction eleven years ago, tom stonier predicted that personal computers would soon be 'pocketbook sized and contain a wide range of features and capabilities', which would become 'the ultimate in electronic books' (stonier, 1991: 364). today, it appears that stonier's prediction has come true. people now have access to highly personal, portable computers, such as palmtop computers or personal digital assistants (pdas), which can be used for 38 an-} volume 10 number 2 various functions, including supporting time and information management, recording and storing notes, and accessing internet content. furthermore, the availability of e-book reading software for many pdas means that they can now also be used to read electronic documents, opening up more possibilities for their potential as learning tools. several authors have argued that pdas and similar devices could be usefully exploited in educational contexts (fung, hennessy and o'shea, 1998; hennessy, 2000; sharpies, 2000; soloway, hennessy and o'shea, 2001). sharpies (2000), for example, proposes that palmtop computers could be valuable lifelong learning tools, used to input data or access information whenever it is needed. portable devices could also be useful for distance education students, who typically have to fit their self-managed learning activities around other tasks, such as work and family commitments (jones, kirkup and kirkwood, 1992). providing access to learning resources anytime and anywhere, pdas could enable students to make more effective use of time while away from the home or office environment. using pdas, however, would also impose some limitations. as carroll and colleagues argue, the possibilities and constraints introduced by a new tool will change the activity that the tool is used to support, a process they term the 'task-artifact cycle': 'an artifact suggests possibilities and introduces constraints that often radically redefine the task for which the artifact was originally developed' (carroll, kellogg and rosson, 1991: 79). the primary possibility introduced by a pda is, of course, portability and, consequently, the ability to have easy access to information stored electronically. the constraints are equally apparent. the portability of the pda compromises other features such as screen size, which has to be small enough for the device to fit in the palm of the hand. as dillon (1994) argues, a small screen isolates the text on display, removing it from its context and making it difficult to scan-read. furthermore, pdas do not benefit from the same data input capabilities as desktop and laptop computers, which typically have a full-size keyboard and mouse. most pdas have touch-sensitive screens that enable direct manipulation of the onscreen interface through the touch of a stylus or finger. while such an interface is useful for quickly accessing and viewing applications, it is not so beneficial for entering text. typically, users would enter text on a pda using a small on-screen keyboard or by writing on the screen in a special handwriting script, which the pda then converts into type. handwriting recognition systems, however, lack speed and accuracy, and data input methods on pdas are generally considered problematic (pascoe, ryan and morse, 2000). the purpose of the research reported in this paper is to elucidate how the possibilities and constraints introduced by a pda change the activity of reading course materials in a distance education context, as determined by students' perceptions of this experience. the research is grounded in activity theory, which provides a useful means for describing how the introduction of a new tool changes an existing activity. the theoretical background of the research is elaborated further below, followed by a description of the study undertaken. examining new tool use: activity theory activity theory (leont'ev, 1978; vygotsky, 1978) emphasizes the sociocultural context in which activity takes place. one of the central tenets of activity theory is the notion that all human activity is mediated by the use of tools, both conceptual tools such as language, and physical tools such as technology. these tools are one aspect of the sociocultural context of the activity. engestrom (1996) has extended this notion of tool mediation to provide a representation of the wider social context of activity, as shown in figure 1. according to 39 jenny waycotz reading with new tools: an evaluation of personal digital assistants as tools for reading course materials this model, activity takes place within a community of people who share the same object, or goal, of activity. there is a division of labour among members of the community and this mediates the relationship between the community and the object. in addition, the community is governed by a set of rules, that is, norms and standards that regulate the activity, and these mediate the relationship between the subject and the community (kuutti, 1996). rules / subject / / / tool \ \ community \ \ object division of labour figure i: activity triangle model (from engestrom, 1996) the emphasis on the mediational role of tools in activity means that activity theory has gained popularity for use in research examining the impact of new technologies on learning and workplace cultures (for example, mwanza, 2001; halloran, rogers and scaife, 2002). activity theory also provides tools for understanding the impact of new technologies at the operational level of activity. leont'ev (1979) divided activity into hierarchical layers, consisting of actions and operations. operations are routinized processes that are carried out to perform the action. the action is a conscious process with a specific goal that helps to meet the overall objective of the activity. for example, the activity of reading with the objective of learning might consist of such actions as reading specific course texts, taking notes, looking up cited texts, discussing the course materials with other students, and so on. the operations through which these actions are performed would include turning pages in the document, using a pen and paper to take notes, and flicking through the document to move back and forth between sections. the actions and operations that contribute to an activity will change as the activity evolves. actions may become routinized over time and consequently be transformed into operations. meanwhile, if the conditions of the activity change, an operation may become an action, requiring conscious effort directed at a specific goal (kuutti, 1996). the introduction of a new mediating tool such as a new computer program would change the conditions of the activity. processes that were formerly operations when a more familiar tool was used would become individual actions, and over time, as familiarity with the new tool increases, these actions may become operationalized (bodker, 1991). the — alt-} volume 10 number 2 introduction of a new tool such as a pda, then, would change the relationship between the actions and operations that contribute to the activity of reading. reading mediated through computer tools past research has examined how the use of new computer tools impacts upon reading. for example, dillon (1994) reviewed early research that compared reading from a computer screen with reading from paper. he concluded that familiarity with paper materials, and the presence of contextual clues such as paper size and document weight, means readers are generally more efficient at navigating through paper documents than electronic text. similarly, o'hara and sellen (1997) compared the two media and found that paper offered several advantages over computers. participants in their study were assigned to either an 'online' condition or a 'paper' condition, and were given a four-page article to read and summarize. those in the paper condition were able to navigate efficiently through the document, using their familiarity with the page layout to flick through the pages, simultaneously recording notes on separate paper. in other words, these processes were operationalized. meanwhile, those using the electronic version found it cumbersome to switch between viewing the document and taking notes (which was also done on the computer), and they were only able to perform these actions serially, rather than simultaneously. thus, using the new tool constrained the activity of reading by introducing new conditions for the activity, which made it difficult for participants to utilize the operations they had developed from past experience with paper documents. it is to be expected, therefore, that a new tool such as a pda would create new conditions for the reading activity which may inhibit the use of operations previously developed and require new tune-intensive actions. the study described below aimed to identify such changes, as well as to examine the impact of the pda on the wider social context of the activity. the study the study context the study centred on the uk open university course applications of information technology in open and distance education', one of three courses that make up a master's programme in open and distance education. students are encouraged to critique and evaluate their own experiences of using information technology on the course. for this reason, the course provided a valuable forum in which to conduct this study, as students were generally eager to try out a new medium for reading course materials, and interviewees and questionnaire respondents offered extensive feedback on their experience of using the pda for reading. the course is undertaken part-time and students are involved in various other professional activities. all 65 students enrolled in the course were supplied with palm ml05 pdas at the end of july 2001. however, participation in the evaluation of pdas was voluntary. students on this course were primarily novice pda users, although nine students said they already owned a pda or had used palmtop computers in the past. the course is primarily delivered online, making use of web resources and first class conferencing software. however, during the final block of the course, students use printbased reading materials, and it was during this block that the study was conducted. three 41 jenny waycott reading with new tools: an evaluation of personal digital assistants as tools for reading course materials sections of the study guide used during this block were provided for access through the pda (67 a4 pages). these sections make up nearly half of the study guide, which is a discursive guide to the readings for the block, written by academics associated with the course. during the study, students had access to both the printed version of the study guide and the electronic version on the pda. the remaining sections of the study guide and other reading materials were provided on paper only. the documents on the pda were entirely text-based and were supplied as word documents, not reformatted for presentation on the pda. the default font size of the text was 12 points (this could be changed by the user). an example screen view of the text on the pda is shown in figure 2. figure 2: an example screen view of the study guide on the pda equipment pdas the pahn ml05 is a relatively affordable model that offers typical pda features, such as note-taking facilities, an address book, calendar and to-do list. it has 8mb of random access memory, sufficient for storing a large amount of electronic text. it has a touchsensitive screen and text can be entered by tapping letters on an onscreen keyboard, or by using the graffiti handwriting recognition software. this involves learning the specific graffiti alphabet characters and writing these on the screen with the stylus, which the pda then converts into type. the palm ml05 is 11.8 cm x 7.9 cm (screen area is 5cm x 5cm) and it weighs 125g with batteries installed. it operates on two aaa batteries, which should last for up to two months. it has a monochrome display, with black writing on a green background. document reader software wordsmith, a commercially available document editor and viewer, was used to present course materials on the pda. this was chosen because it is a flexible program that can be 42 alt-j volume 10 number 2 used for a variety of purposes, including to record notes, edit documents and read text. users can move through the text in a variety of ways: by using the scroll bar on the side of the screen; using the buttons at the bottom of the device; or by using the additional scrolling facilities of the software, such as the 'teleprompter' facility which moves the text automatically. the software has search facilities, such as 'find', which seeks out instances of the specified word, and 'go to paragraph', which allows users to jump to a particular paragraph. instructions students were given the manufacturer instruction manuals for the pda and wordsmith software, as well as additional instructions developed by the researcher. these included hints about using the pda for reading course materials for example, suggestions about how to take notes on the pda, how to underline or embolden sections of text, and how to use other features of wordsmith, such as the 'find' and 'go to paragraph' facilities. data collection interviews approximately two months after students received the pda, telephone interviews were conducted with ten students who were randomly selected from the entire cohort of students on the course. only a small sample of students could be interviewed due to time constraints. nevertheless, the interviews elicited in-depth information about the selected students' perceptions of the benefits and limitations imposed by the pda, and how the pda changed the reading activity. questionnaires online questionnaires were administered at the end of the course; thirty-five responses were received. a mixture of closed and open-ended questions asked students what features of the pda they had used, whether they had found the pda to be a useful tool for supporting their studies, the sort of reading strategies they adopted when using the pda, and what the benefits and limitations were of using the pda to read course materials. responses to open-ended questions regarding the latter are reported here. a full report of the findings can be found in waycott and kukulska-hulme (2001). how the pda changed reading this section identifies students' perceptions of the possibilities and constraints associated with reading from the pda. the impact of these possibilities and constraints is discussed with reference to concepts from activity theory, particularly with regard to how the pda changed the relationship between actions and operations and modified the wider social context of the reading activity. portability when asked what was the main benefit of using the pda to read course materials, most questionnaire respondents (63 per cent) referred to the portability of the device. portability also emerged as a common theme in the interviews. being able to carry around course materials in a small, lightweight device appeared to change the activity of reading. in particular, it made it easier for students to fit their reading around other activities. one interviewee said: 43 jenny waycott reading with new tools: an evaluation of personal digital assistants as tools for reading course materials it was so much easier to do the work, you could do it everywhere and when you're trying to fit it in around work and kids and everything, that was an enormous advantage. the pda enabled students to read course materials in various situations, even while they were carrying out other activities. for example, one interviewee said he used the pda while stuck in traffic. another interviewee used the pda to read course materials while in work meetings when the topic being discussed was not relevant to her. the pda was an 'unobtrusive and neat device that could always be to hand'. some students used the pda to aid the process of revision, re-reading course materials when away from the desk or office environment. the possibility of accessing course materials anytime anywhere, then, had the potential to change the way students undertook the reading activity, particularly with regards to how reading course materials was structured around other activities. screen display students also identified limitations of using the pda for reading. in particular, the small screen size and poor screen quality of the pda caused some constraints. the limited amount of text visible meant that students had to adopt a 'line by line' reading strategy and this made it difficult to scan-read through the text and to gain an overview of the documents. reading text on the small screen was also slower than on paper, as this questionnaire respondent suggests: the size of the text was relatively small for me. plus the continual scrolling down put me off. i couldn't seem to be able to scroll as fast as i read and i kept losing my place as i did scroll. in the end it was too much of a stop start situation. it could be that the act of scrolling through the text was not operationalized and instead required the focused, goal-directed effort of an action, as this interview comment suggests: you're having to concentrate that little bit more because you're only seeing so many characters in vision at the same time. it's more a conscious effort to scroll down. the disruption caused by the pda, then, may have been due to the relative unfamiliarity of reading from a small screen and this meant that students could no longer employ the same strategies they used when reading from a4 size printed materials ('only being able to see a small amount of text at any one time, you have to adopt a different reading strategy.'). moreover, the limitations caused by the size and quality of the screen might not be easily overcome with practice. some students, for example, said they experienced headaches and eyestrain when using the pda for too long, suggesting that the poor screen quality is an important issue that may detract from the user experience. navigation difficulties closely related to the issue of small screen size was the problem of navigating through the documents, which some students found difficult on the pda. the software's navigational tools, such as the 'find' and 'go to paragraph' features, were considered 'irksome' and 'finicky'. it was also difficult to pick up on visual clues, such as headings, and some contextual clues, such as page numbers, were not available on the pda: 'i found i got lost in the text as [there were] no visual cues to page number etc' one interviewee suggested that more information about her location within the document space would have been beneficial: i know it has a little bar on the side to say how you've progressed through the reading, 44 alt-j volume 10 number 2 but i'd still like something a little bit more . . . i've seen one somewhere on some computer screen that has percentages how far you've got through the reading. i think something a little bit more obvious like that would be more helpful. i sort of like to know where i am in the reading. therefore, students had to learn to use new contextual clues to aid the process of navigation, and this caused some disruption to the reading activity. one student suggested that navigational difficulties might lessen over time: 'i found i could lose the sense of where i was quite easily with such a small screen but this was as much a question of practice as anything.' some students also suggested that the documents on the pda would have benefited from more extensive use of formatting in order to emulate printed materials, for example: "there was no break in the text. reading printed materials, the page is set out to show areas, categories, etc. demarcation was not obvious.' it appears, then, that students would have liked to be able to use the familiar navigation strategies they had developed from past experience with printed text. however, features of the pda, such as the small screen and scroll bar, meant these familiar strategies or operations could not be utilized, making it difficult to navigate through documents on the pda. note-taking students also experienced difficulties when taking notes about the course materials on the pda. the pda introduced new methods for taking notes and as these methods were not yet operationalized, they were difficult and time-consuming to use. typing on a full-size keyboard or writing on paper was deemed to be much faster and superior. using the graffiti handwriting system was slow and error-prone, as illustrated by this interview comment: i experimented with the writing p a r t . . . and of course one of the problems was the letters weren't always coming out as i intended but i think that would have been a question of practice... so i got to the stage of being able to write and produce letters in print form on the screen but as it took a lot of time to do it i gave up. however, there was recognition that the speed and accuracy of entering text may improve with practice. one interviewee who already owned a palm pda said: 'i have to say probably my previous experience with a palm pilot helped. i'm very, very aware of that [graffiti\ and have gotten much more skilful in it in the time i've had my palm pilot.' over time, then, it is possible that the actions required to take notes on the pda could become operationalized. despite the difficulties of entering text on the pda, some students persevered and did use it for this purpose. for example, one interviewee described how the note-taking facility on the pda opened up more possibilities for her to make use of time while away from her normal study environment: it was nice to be able to take notes anywhere . . . i usually take notes on a computer. if i lugged the course materials to the kids' band practice, or were to do that and handwrite the notes, then i'd have to go and i'd have to type them into the computer. but taking notes on the palm, from that point of view, was wonderful. notes recorded on the palm were already in electronic format and could be easily transferred to the desktop computer. this allowed the student to devise strategies for 45 jenny waycott reading with new tools: an evaluation of personal digital assistants as tools for reading course materials overcoming limitations imposed by note-taking methods on the pda: 'the key thing was to get the notes down. if the odd character was wrong it was more time-consuming to go back and correct it than it would be just to sort it out when you've got it on the main * machine.' one benefit of using the pda for this purpose, then, is that it can be used in conjunction with other tools, in particular the desktop or laptop computer. use of existing tools the pda did not replace the tools students already used to support reading. instead, it was used in conjunction with existing tools, and for students who used the pda extensively it modified the way those tools were used. having course materials on the pda complemented the printed documents: 'having the choice between paper and palm meant that i felt like i could process more of the course materials, and consolidate my understanding of it; reading in one domain reinforced reading in the other domain.' students could use the pda to read or revise the materials while on the move and revert to using the printed documents when in their normal study environment. therefore, the pda became an extension of the printed materials, providing more opportunities for students to read and review the course materials. the pda could also be used in conjunction with the desktop computer. one questionnaire respondent used the pda to 'cut and paste important points/quotes' from the course materials and then 'hotsynced' this to her pc. this meant that notes recorded on the pda could be incorporated into more extensive documents on the pc. for some students, the pda reduced reliance upon other tools. one student commented: 'now i use it on the move instead of pen and paper.' another said she uses the pda to support the initial stages of essay-writing. the advantages of the new tool being instantly available highlighted limitations of other tools: great for initial essay-writing (getting ideas jotted down as they are thought of, rather than trying to get to a computer, where you have forgotten by the time you've turned it on. or even to paper sometimes when you've forgotten your thought by the time you have found a working biro . . . ) . introducing a new tool into an activity, then, has the potential to change the way existing tools are used. in this context, the pda provided an extension of existing tools, such as the printed study guide and the desktop computer. in some cases, as in the example above, the pda could also be used to overcome limitations of existing tools. discussion these findings suggest that the introduction of a new mediating tool a pda does modify the activity of reading course materials in a distance education context. the new tool presents new possibilities and constraints; as carroll et al. (1991) argued, these have the potential to change dramatically the activity the tool is used to support. in activity theory terms, these changes may affect both the social context of activity and the distinction between the actions and operations that make up the activity (kuutti, 1996). in this study the changes are most apparent at the action/operation level. the new tool introduced new ways of doing things new methods of entering text, new requirements for navigation and a new display size for the text and this meant students could not apply the operations they had developed from past experience of using paper documents and 46 alt-j volume 10 number 2 desktop computers. this had the effect of disrupting the reading activity, making it slower and more difficult than reading on paper. these findings, therefore, are similar to earlier studies comparing paper-based reading with onscreen reading (dillon, 1994; o'hara and sellen, 1997): existing tools, with which the user is familiar, are viewed more favourably than new tools, which disrupt the activity. using the pda was different from using both paper materials and desktop or laptop computers. the screen size was the most notable difference as this severely limited the amount of text on display at a time. therefore, scrolling through the text was a necessary, but somewhat arduous, process. furthermore, data manipulation techniques on the pda were different from familiar techniques used with both paper and desktop computers for example, flicking through paper documents, using a pen to record notes, or using a mouse and keyboard on a desktop computer. therefore, in general, students found it difficult to record notes on the pda or to interact with the text in the same way as they would when using more familiar tools. for some students, however, the benefits of using the pda were great. the tool was considered more lightweight and portable than the 'cumbersome' a4-size printed materials. this meant the course materials could be carried around and used 'anytime possibilities: portability, electronic text constraints: small i screen, navigation, notei taking methods j i operations become actions j i (e.g., scrolling, looking for j ! navigation clues, taking j notes) i existing tools the pda may change the way existing tools are used j student on j subject • course _̂ i reading course { object i i ™lel!?!i i rules community division of labour i rules governing open \ j university students; i i informal regulations \ ; governing the community j other students; tutors \ student roles; tutor \ j roles j figure 3: the introduction of a new tool (pda) on the activity of reading course materials 47 jenny waycott, reading with new tools: an evaluation of personal digital assistants as tools for reading course materials anywhere' which had the effect of modifying the wider context of the activity: with the pda, the activity of reading course materials could be structured more effectively around other activities in which the student was involved. in addition, the pda had the potential to change the way students used existing tools. some students who used the pda for notetaking, for example, found they no longer used pen and paper, and they were able to use the pda in conjunction with the desktop computer to extend and modify their notes. figure 3 extends the activity system model developed by engestrom (1996) to represent the effect that the pda had upon the activity of reading course materials. the new tool is shown to introduce new possibilities and constraints, which interact with, and change, the actions and operations of the activity. thus, most of the effects of the new tool can be seen at the top of the model, whereas engestrom's wider social context of activity (the community, rules and division of labour) remain unchanged. this may be because the activity in question is an individual activity. although students are members of a community consisting of other students and tutors and they take part in community activities such as online conferencing during other parts of the course, the activity of reading course materials is primarily carried out alone. the notions of community, rules and division of labour, therefore, would be more useful for research examining activities that are specifically collaborative in nature (for example, mwanza, 2001). however, halloran et al. (2002) did examine a collaborative learning situation and they found that 'rules' and 'division of labour' were not useful constructs in the context of their research. one limitation of the model for illustrating the findings from this study is that it does not allow a representation of the multiple activities in which a subject is involved. this means that the effect of the portability of the pda which changed the way the activity of reading was structured around other5 activities cannot be represented. there is insufficient scope in this paper to provide a solution to this limitation. nevertheless, it is an issue that could be elaborated in further research using activity theory to examine the effect of new technologies. conclusion the study showed that students had mixed responses to the use of pdas for reading course materials. the pda disrupted and constrained the reading activity through its introduction of new conditions, which made it difficult for students to apply the operations they had developed from experience using paper and desktop computers; instead they had to formulate new time-intensive actions. limitations such as the small screen size, navigational difficulties and unfamiliar note-taking methods changed the way students read course materials. they had to adopt a concentrated, line-by-line reading strategy, and were unable to skim-read or move backwards and forwards within the documents as easily as they could with paper. however, the ability to have 'anytime, anywhere' access to the course materials was welcomed by students. in this respect the pda was a valuable tool, suggesting that the provision of course materials for access on a portable device is useful for distance education students. however, for some students the constraints of using the pda, in particular the small screen size, meant they were reluctant to use it instead of paper. while it would be premature to consider replacing printed documents, then, this study does show that there may be some value in using pdas to extend existing tools. 48 alt-j volume 10 number 2 acknowledgements this study forms part of my ph.d. research, supervised by professor eileen scanlon and dr ann jones. the project was conducted in collaboration with dr agnes kukulskahulme, chair of the course team that runs the open university course 'applications of information technology in open and distance education'. references bodker, s. (1991), through the interface: a human activity approach to user interface design, hillsdale, nj: lawrence erlbaum associates. carroll, j. m., kellogg, w. a. and rosson, m. b. (1991), 'the task-artifact cycle', in j. m. carroll (ed.), designing interaction: psychology at the human-computer interface, cambridge: cambridge university press. dillon, a. (1994), designing usable electronic text: ergonomic aspects of human information usage, london: taylor & francis. engestrom, y. (1996), 'toward overcoming the encapsulation of school learning', in h. daniels (ed.), an introduction to vygotsky, london and new york: routledge. fung, p., hennessy, s. and o'shea, t. (1998), 'pocketbook computing: a paradigm shift?', computers in the schools, 14 (3/4), 109-18. halloran, j., rogers, y. and scaife, m. (2002), 'taking the "no" out of lotus notes: activity theory, groupware, and student groupwork', in g. stahl (ed.), computer support for collaborative learning: foundations for a cscl community proceedings of cscl 2002 (boulder, colorado, usa january 7-11), hillsdale, nj: lawrence erlbaum associates. hennessy, s. (2000), 'graphing investigations using portable (palmtop) technology', journal of computer assisted learning, 16, 243-58. jones, a., kirkup, g. and kirkwood, a. (1992), personal computers for distance education: the study of an educational innovation, london: paul chapman publishing. kuutti, k. (1996), 'activity theory as a potential framework for human-computer interaction research', in b. a. nardi (ed.), context and consciousness: activity theory and human-computer interaction, cambridge, ma and london: mit press. leont'ev, a. n. (1978), activity, consciousness, and personality, englewood cliffs: prentice hall. leont'ev, a. n. (1979), 'the problem of activity in psychology', in j. v. wertsch (ed.), the concept of activity in soviet psychology, armonk, ny: m. e. sharpe, inc. mwanza, d. (2001), changing tools, changing attitudes: effects of introducing a cscl system to promote learning at work, milton keynes: knowledge media institute, open university. o'hara, k. and sellen, a. (1997), 'a comparison of reading paper and on-line documents', proceedings of chi 97, atlanta, ga: acm. 49 jenny waycott reading with new tools: an evaluation of personal digital assistants as tools for reading course materials pascoe, j., ryan, n. and morse, d. (2000), 'using while moving: hci issues in fieldwork environments', acm transactions on computer-human interaction, 1 (3), 417-37. sharpies, m. (2000), 'the design of personal mobile technologies for lifelong learning', computers and education, 34 (3-4), 177-93. soloway, e., norris, c., blumenfeld, p., fishman, b., krajcik, i and marx, r. (2001), 'log on education: handheld devices are ready-at-hand', communications of the acm, 44 (6), 15-20. stonier, t. (1991), 'futures: the personal electronic pocket-book', educational and training technology international, 28 (4), 364-8. vygotsky, l. s. (1978), mind in society: the development of higher psychological processes, cambridge, ma and london: harvard university press. waycott, j. and kukulska-hulme, a. (2001), an evaluation of the use of personal digital assistants for reading course materials on h802: final report, milton keynes: institute of educational technology, open university. 50 the importance of cost-benefit analysis (cba) a response steve draper university of glasgow email: s.draper@psy.gla.ac.uk in issue 11.2, nicol and coen (2003) argue for developing a cost-benefit analysis model for learning technology. however, there are three substantive issues with this approach. first, it is problematic because it involves comparing different things; second, because by understanding the purpose of the analysis, it may be possible to justify the necessary simplifications for special cases and limited applications while being explicit about the limitations; and third (and conversely), this can expose errors and omissions in the analysis even after accepting its general principles. the need for cost-benefit analysis is not about ignoring distinctions of quality and value and reducing things to a monetary value. it comes from the need to make decisions. in making a decision alternatives are ordered to enable selection, but these are not always equivalent to each other, which makes this problematic. ict (information and communication technology) research and development activities generally entail a trade-off between time, money and quality. because budgets are fixed, decision-making tends to weigh money over quality and timeliness, whereas decisions should be made on appropriate educational factors. is a model for evaluating the costs and benefits of the use of ict worthwhile? yes, because it forces attention on the factors that determine educational outcomes. the aim of the model is to calculate quantitatively in order to make decisions between alternatives. the actual but implicit benefit, it is argued, lies in identifying costs and benefits and then measuring them. but the advantage of this approach is not actually quantitative (being able to make close decisions more accurately) but qualitative (identifying what the key issues are). proponents argue that the use of their model lies in the value and accuracy of the categories or factors used: however, if these do not correspond to sensible and useful divisions in the first place neither will results. furthermore, their approach does lead to the generation of a list of important factors for 118 alt-j volume / / number 3 users of their model to consider applying. an aspect that could usefully be added would be to enable searching for important factors. after all, as they assert, this is one of the main benefits of the approach. pessimism about the value of the quantitative aspect of this type of approach is reinforced by landauer (1995). he argues that there is little evidence of the benefits of it to the united states economy, except in a few special technical areas, which is in contrast to the kind of clear economic indicators of the benefits of other inventions. if nearly fifty years of it investment in the usa do not show clear quantitative benefits, then the prospects of measuring these in an ict project in one university seem remote. my general view is that we need above all to understand the factors involved: we are far from identifying, let alone quantifying, these. cba forces us to uncover and consider these factors. is their approach focused enough on this? i criticize their approach only for not going far enough. the general problem with the accountancy categories used in their model is that they try to assign costs under headings, whereas given costs often produce benefits under several headings. thus the categories used for costs, far from being 'hard', are questionable. the model considers that costs are 'hard' measures (count the money) and benefits are 'soft', therefore difficult to measure. but this fails to bring out the difficulty of inventing and applying categories that provide useful analysis. the model attempts to classify staff time into either teaching or research, whereas in reality many activities contribute to both. we can measure time (like money) accurately, but this should be assigned solely to either teaching or research. similarly consider, for example, classifying the cost of a student learning to use email. is this a skill of benefit to their learning needs, relevant for future employment or something of value to their social life? in reality it is valuable for all of these. these examples bring out not only the inappropriateness of apparently obvious cost categories, but also an issue not faced by nicol and coen: about how the same action can be classified differently at different times. a student signing up for email training, or an academic going to a conference will very often see the benefit differently before or after going or again at some time later. so too would an analyst, since more information about the true beneficial consequences is available later; yet decisions have to be made on the basis of advance information. thus these measures are not objective or stable over time. furthermore, it suggests that cba done purely to support management decisions may be rather poor at understanding what the real benefits are, and therefore poor at developing understanding of education and at supporting management decisions in the longer term. overall there are three problems with the approach adopted. first, accountancy is about applying categories that are general and fixed (for example, by tax or laws governing accounts for public companies): these legal requirements in turn stem from forcing companies to apply the same categories in order to allow investors to compare (unlike) companies. because of the primitive state of cba of education, we should build cautiously from small cases, embracing new categories as appropriate and acknowledging the deficiencies and time limitations of each. generalization may only become possible later. instead the focus should be on identifying the usefulness of each category at a given time. the preference for standard categories, so important in accountancy, is a liability in an educational context. 119 steve draper the importance of cost-benefit analysis (cba) a response second, though they say the paper comes from trying to combine an accountant's approach to costs with an educational evaluator's approach to benefits, in fact the accountant too is identifying benefits not costs. it is not difficult to measure how much money has been spent: the difficulty is in identifying what benefits have been bought by it. categories such as research vs teaching, perhaps even infrastructure vs value-added activity, are categories of benefit bought by money. third, money is the only cost (in the sense of negative benefit or limited resource) being analysed in their model. if accountants can only measure money, then other cost experts must be brought in to identify and measure the other costs such as staff stress (soon to become a legal liability to organizations), and student time (not paid for, yet a fixed and limited resource, around which the organization's business is structured). my final point considers the issue of learning costs, which is paramount as ict projects often fail to become embedded; the costs of addressing associated staff development needs were not included. in addition, learning costs are particularly tricky because they change over time. in the usa, there is now an assumption that most students have a basic familiarity with computers. this means that training in basic computer literacy has moved to being a remedial activity for a minority. learning costs are crucial to analyse because they have a major effect on when a given project may become worthwhile; which disciplines it will and will not work for; and economies of scale: it may work only for large-scale change where a single learning cost gives benefits on repeated occasions. there is no space for more detailed discussion: i have concentrated on the main issues. to support decision-making and so to do anything of practical usefulness to learners, we must be able to reduce all alternatives to a single order of preference. this does not necessarily require expressing their merits in terms of money or any other numerical score; but it does require comparing qualitatively unlike things. nicol and coen are tackling one of the most important issues in the education field and one with enormous potential benefits to learners. however the benefits may be a long way off, because the difficulties are substantial. i thus fully endorse their overall aim, but is progress possible? we need to uncover the associated factors and issues. their method should be modified to stress an open-ended enquiry phase of active search for the important factors in each case and learning costs should be included in the classification. furthermore it is important to consider whether the model can reason about the cost-time-(learning) quality triangle that implicitly dominates most practical educational decision-making in both research projects and daily practice; or alternatively if you follow phillips (1996) and reeves (1992) then you will require it to reason about the quantity-quality-cost trade off. a longer version of this commentary is available at: http://www.psy.gla.ac.ukhstevelqqa.html references nicol, d. and coen, m. (2003), 'a model for evaluating the institutional costs and benefits of ict initiatives in teaching and learning in higher education', alt-j, 11 (2), 46-60. landauer, t. k. (1995), the trouble with computers: usefulness, usability, and productivity, cambridge, ma: mit press. phillips, r. (ed.) (1996), developer's guide to interactive multimedia: a methodology for 120 alt-] volume 11 number 3 educational applications, perth, western australia: computing centre, curtin university of technology. reeves, t. c. (1992), 'evaluating interactive multimedia', educational technology, may, 47-52. 121 taking over someone else's design: implications for the tutor's role in networked learning mireia asensio,* janice whatley** and chris jones* *csalt, lancaster university, **information systems institute, university of salford email: m.asensio@lancaster.ac.uk the experience of taking over an already designed web-based course helps us to investigate the claims in the literature about the role that tutors have more generally in networked learning. this paper addresses this issue through a case study and brings together the tutor's experience and her reflective diary, as well as the interview data from a jisc/calt phenomenographic study of tutors' and students' experiences. this particular case study raises issues about the tutors' role, teaching activity, design and the value of content resources and knowledge representation. finally the paper reflects on the implications for the tutor in this situation and provides suggestions for future practice. introduction this paper reflects on the experiences of a particular tutor taking over a specific course. it is an example of a common situation found in higher education (he) institutions, when newly appointed or temporary part-time or full-time staff are hired to teach courses that have been designed and developed by other members of teaching staff. the experience of this particular tutor may help us to illuminate the role that tutors have more generally in networked learning environments. for the purpose of clarification, we define networked learning as 'learning in which c&it is used to promote connections between one learner and other learners, between learners and tutors, between a learning community and its learning resources'. a large part of this particular course was the focus on the connection between learners and the on-line web learning resources, but there was also a'significant element of interaction between learners and tutor and between the learners themselves, occurring on-line and via face-to-face events. in this paper we examine claims in the literature as to whether the tutor's 65 mireia asensio et al taking over someone else's design: implications for the tutor's role in networked learning role in a networked learning environment is qualitatively different from that in conventional teaching. the paper investigates some current literature in this area and specifically collins and berge's (1996) classification of tutor activity. collins and berge identify four main areas to which tutors in networked learning need to pay attention. these are pedagogical, social, managerial and technical teaching activities. our research findings help us to explore those main areas, and the implications for the tutor's role when this involves taking over someone else's design of a mainly web-based course. as a consequence other issues are raised on the value of content resources and knowledge representation. our research is based mainly on a phenomenographic approach to research (marton, 1994; marton and booth, 1997) and utilizes both the tutor's experience and the use of a reflective diary. the study was carried out as part of a jisc/calt networked learning project in he, uk. the tutor's diary detailed the tasks that arose, the reaction of the students to the module week by week, and the time spent by the tutor on amendments to the module. the diary was a 'vehicle for reflection' as suggested by moon (1999), and was kept for the purpose of evaluating the module prior to redesigning it, as necessary, for the next academic year. this paper uses some edited extracts of the diary and includes a section on the tutor's experiences and suggestions for future practice. the jisc/calt project has been looking at the students' and tutors' experiences across different uses of communication and information technologies for networked learning. the collaborative research between the tutor and the researchers of the jisc/calt project has involved bringing together the tutor's diary and her general experiences about the course, and interviews with the original designer and the current tutor of the course. the tutor's role in networked learning this section examines the staff role in networked learning as it is portrayed in current literature. in broad outline, the role of the educator in networked learning environments has been summarized by the idea that the educator needs to move from 'the sage on the stage to the guide on the side'. the role of the tutor in a networked learning environment tends to be that of a facilitator or moderator. early accounts of computer conferencing emphasized this shift (see, for example, kerr, 1986; feenberg, 1989). in more recent years the idea has become part of what has been described as a new paradigm of teaching and learning (mason and kaye, 1990; harasim, hiltz, teles and turoff, 1995; koschmann, 1996; reigeluth, 1999). the new role for the moderator/facilitator means that they 'must change from the purveyor of knowledge to instigator, promoter, coach, helper, model and guide of knowledge construction' (jonassen, 1996: 261). compared with other media, networked environments are said to emphasize social skills in teaching at the expense of content and personal delivery. collins and berge (1996), consolidating a wide range of literature, classify the tutor's role under four headings: pedagogical (intellectual; task) some of the most important roles of on-line discussion instructor/moderator/tutor revolve around their duties as an educational facilitator. the instructor contributes their special knowledge and insights and uses questions and probes for student responses that focus discussions on critical concepts, principles and skills. by modelling appropriate on-line behaviour, the instructor can prepare students, alone or in groups, to experience moderating the conference for themselves. 66 alt-] volume 9 number 3 social creating a friendly, social environment in which learning is promoted is also essential for successful on-line teaching. this suggests promoting human relationships (affirming and recognizing students' input, providing opportunities for students to develop a sense of group cohesiveness, maintaining the group as a unit, and in other ways helping members to work together in a mutual cause) are all critical to the success of any conferencing activities. managerial (organizational; procedural; administrative) this role involves setting the agenda and pacing for the conference: the objectives of the discussion, the timetable, procedural rules and decision-making norms. meta-comments can be used to remedy problems in context, norms or agenda, clarity or irrelevance, and help participants deal with information overload. unobtrusively managing the flow and direction of the conference discussion without stifling the participants is a sine qua non of successful conference facilitation. technical the instructor must first become comfortable and proficient with the technology, and then must ensure that participants are comfortable with the system and the software that the conference is using. the ultimate technical goal for the instructor is to make the technology transparent. when this is done, the learner may concentrate on the academic task at hand. introduction to the case study the case study was a third-year course entitled 'developing systems for teaching and learning' taught in the information systems institute (isi) at the university of salford. this module was concerned with exploring the development of information systems for teaching and learning, and it aimed to provide the theoretical underpinning and practical experiences needed for the development of education and training systems. the original designers of the course have written a report (nicholson and bird, 1998), which outlines reasons for adopting their particular approach to web-based teaching and learning to the course design. they described the course as an answer to growing student numbers. it was also designed to cope with a changing student population, involving increasing numbers of non-traditional students (nicholson and bird, 1998). the web site was intended to provide the main medium for teaching and learning, and the face-to-face lectures and seminars were intended to supplement or complement the main web-based material. prior to the case study being undertaken, but after access had been agreed for the researchers to interview the students, both members of staff responsible for the course obtained jobs elsewhere. the new member of staff, who had been appointed on a temporary full-time contract, agreed to allow the jisc/calt study to continue. this case study has, as a consequence of this change, the added dimension of being taught by a lecturer not responsible for the initial design. the new tutor was also new to the centre and the course subject; she had access to the on-line material, some other course resources and some of the journal papers written by the previous tutors. however she did not have the opportunity to talk to these tutors about the course and the design itself, or about their reflections on any problems encountered throughout the course. she had limited support 67 mireia asensio et al taking over someone else's design: implications for the tutor's role in networked learning and little time to become familiar with the web site, resources, aims, objectives and assessment procedures for the course. the module comprised web-based material covering educational theory and design principles. the web site contained a freeware conferencing utility, the course administration, syllabus and assignment details, learning resources exercises and virtual spaces for the students to interact with each other and the tutor. there was a space for informal chat and a virtual seminar to discuss the course content. the virtual seminar was structured around the course into themes and one seminar space was set up for teams to give feedback to each other about their project work. the web site also contained an archive allowing students to look at previous years' design projects. the course had a team work component that was informed by ideas about co-operative learning. the students' experiences of this were marked by the general context of the degree structure, as the entire degree was built around teamwork. the students had broad experience of working in this way within isi and also occasionally undertook 'commissions' for outside bodies, in teams composed of first-, secondand third-year students. thus the students on this third-year course were very experienced in team working, in a variety of settings with a number of different team members. the assessment consisted of five different components; an individual element comprised 20 per cent and a teamwork element comprised 80 per cent of the final mark. the project was the major component of the course assessment, in which student teams of five were expected to build a prototype for a web site on a topic of their choice, as a learning and instructional system. project titles included topics such as astrology for beginners, the basics of guitar-playing, how to upgrade and maintain a pc, etc. each team was expected to describe the design, purpose and intention of their web site and to provide feedback to the other teams on their project proposals. the marking scheme contained an element of self and peer assessment. within teams, individuals were peer-assessed by the members of their own team. team projects were assessed anonymously by the other teams and the tutor collated these marks, which were used to moderate the final marks determined by the tutor. implications for the networked learning tutor in the context of this particular course and situation we explore how collins and berge's (1996) classification of the different activities that a tutor should take into account, that is, pedagogical, social, managerial and technical, apply to this tutor's experience. managerial and technical issues when the tutor described her experience her main focus was on the managerial and technical issues that she had to deal with before the course started. she commented that, in the same way that developing an on-line course requires considerable input 'up front', updating an existing course also requires a great deal of input before the course is used. compared with a lecture-based course, she did not have to write or adapt lecture material, but needed to familiarize herself with the published content material. the following text is an edited version of the tutor's thoughts and reflections from her diary: the structure of the course in terms of assessment dates and progress through the material needed to be sorted before going live. unlike a lecture-based course there is no 68 alj-j volume 9 number 3 option to sort out dates and material as the course progressed. this took sixteen hours before it started plus the reading to familiarize myself with the course material. this was quite time-consuming as i read it on-line as the students will need to. at the beginning i assumed that i would be able to read up again on each section just before the students were programmed to get there. the 35 hours were, on top of the 2 hours allocated each week for lectures, tutorials or seeing students about their work. this time probably represents about the same time as would be spent on a traditional lecture-based course, so there was little saving through the module being on-line. a crucial element we have identified, as indicated below in the tutor's quotation from the interview, is the fact that time pressure was a main constraint to redesigning the course content before the course started. this means that the tutor had to take the course content at face value, and try to amend/change it throughout the course. so as well as doing two full modules i was then going to do six weeks of this which brought me right up to easter so the time pressure got worse actually, i wasn't able to look at the site, spend any time on it basically, i was fire-fighting from then on. if we think of taking over a conventional teaching course, we find that the tutor has more flexibility to change the content of the course before each class: the tutor can prepare a new set of slides the night before if they wish to do so. in a web-based course, in which all the learning resources are available to the students from day one, the tutor lacks flexibility. in the case described here, she had no time to make any major changes to the course content and design, apart from updating the sections on course administration. she took the pedagogical design at face value and hoped to be able to redesign it as the course progressed over the weeks. interestingly the tutor had also taken over a conventional lecturing course on systems design, a comparison with which shed light on the differences in tutor role in this medium. as the tutor said during the interview: um it's probably easier to pick up a load of acetates and be able to deliver a fifty-minute lecture than it is to read through web materials and think of some tutorial sessions that you should prepare with t h e m . . . by contrast we have here something that all the course materials were there at the beginning. actually yes, the big difference is that i prepared material as i went along for the hour lectures, so at the beginning i'd only done three weeks' lecture notes, then i sort of carried.on and i was preparing stuff as i went along . . . it was easier because nothing was set in stone from the beginning. the 'set in stone' analogy is revealing, as it points out that taking over the networked learning course implied taking over, at least temporarily, the previous tutors' representation of knowledge and their personal understanding of the students' intended learning experience. interestingly as the course progressed the tutor became more aware of how the material had been presented and she perceived mismatches between aims, objectives and assessment. this was also exacerbated by the lack of course design documentation and specification up front. in addition she saw herself having to understand and internalize the content at the same time. not having done the course or known what it was all about before, i was actually trying to read the material on-line at the same time as trying to update things... but there was 69 mireia asensio et al taking over someone else's design: implications for the tutor's role in networked learning no material there that directed you to what was on the site; i had to map it out myself and that's one criticism of the site that i don't think it's very clearly mapped and the students have said that when they've done the personal reflection at the end of the course, they thought in cases it was very messy and they didn't know where they were. so that's one thing that needs altering, (tutor's interview) this indicates that because in a web-based course the content is 'set in stone', the on-line tutor has less flexibility, at least at the beginning, to readapt the web site. moreover, as the tutor was trying to make sense herself of how the course content had been presented, given the time constraints and other work pressures, this also caused some anxiety. she described the whole experience as feeling like 'fire-fighting from then on' once the courses started, trying to catch up particularly with the redesign of the networked learning course. this emphasizes the importance of well-structured web pages and smooth navigation to aid both the tutor and the students. as a teacher i felt that i was perhaps letting the students down a little bit in that i wasn't always able to get back to them immediately with an answer to a question, assignments took me a little bit longer to mark because i was trying to figure out how to mark them, so i do feel i let the students down a little bit in a way, but at the end of the day most of them appreciated that i was under pressure and taking the course on face value they actually enjoyed what they did in the course, so that negated any criticism of my management of it. so in the end it wasn't too bad but i feel i could have done it a lot better given a little bit more time, (tutor's interview) this experience seems to indicate that because of time pressure, the tutor could only pay attention to two immediate priorities: the managerial and the technical. this is not to say that she did not consider, think or care about the pedagogical and social implications of the course. pedagogical and social issues: the importance of face-to-face tutorials the tutor found that regular face-to-face tutorials could compensate for the lack of time that she had to amend the site and resource material, in a way that would better represent her educational view of the course and teaching style. the only way i could alter anything is to actually put in a tutorial, actually prepare some material that filled in the gaps and present it at a tutorial session/face-to-face session as an extra which i did . . . in terms of the degree qualification you are trying to get the students to think about things, compare, contrast and delve a little deeper into some of the things, whereas the material on-line is a very superficial look at theory. you either take it that it's there so it must be fact, a bit like reading it from a b o o k . . . so the face-to-face tutorial is the only opportunity to actually discuss any of those 'now do you think it is right or are there not occasions where this theory might be slightly wrong?' (tutor's interview) this raises the issue that the course content and delivery style was very personal and unique to the previous tutor. in other words the web site and the choice of content represents the tutor's design and representation of knowledge and, arguably, not that of the student. conversely we could suggest that in a text-based computer conferencing environment, the opportunities for sharing and building on the students' as well as the 70 alt-j volume 9 number 3 tutor's previous experiences and knowledge, do not as clearly overemphasize the tutor's view of the world and representation of knowledge. it appears that in this type of course the role of the tutor is also perceived by the students to be that of the 'subject expert'. not surprisingly the new tutor found herself in a situation of both trying to comprehend someone else's design rationale and become acquainted with the course content. it appears to us that the face-to-face tutorials provided the means to bridge the gap of mutual understanding between the interpretation of the tutor of the course content and that of the students. however it is worth noting that a truly on-line module would pose difficulties in this respect. the ability to generate reflection from students is an important part of learning, which as salmon (2000) notes is a different tutor role on-line from that for a faceto-face environment. a virtual seminar potentially enables this sort of discussion, but it requires tutorial facilitation to elicit meaningful reflection. a true on-line course would of course have no tutorials, and some students were happy with this arrangement, whereas others liked to have a regular weekly contact time. 'it would be wrong to present material which those who did not want to come would miss out on, as attendance was not stipulated on this module, except for certain sessions' (tutor's diary). indeed the difficulty was to offer equal access to content material and information on the course to all the students, given that on-line provision might offer a choice for students. however, the majority of students attended the tutorials, though these were not explicitly compulsory. hence our interpretation that the face-to-face tutorials provided the means to check understanding with the students. they were a way of making sense of the content and the course objectives. we can argue that they also provided an opportunity for the students to socialize as a class, to see each other and discuss issues as a whole group. this could have provided the way to include the pedagogical and the social activities. the use of the virtual seminar was so patchy that the tutor found it difficult to check understanding from the postings made. the questions posed to the virtual seminar needed to have clear objectives for them to form the basis of a useful discussion. in addition, both the face-toface tutorials and the fact that students saw each other regularly for their project work meetings hindered the students' active participation in the virtual seminar. tutor's experience and suggestions for future practice in detailing such an account of progress regularly using a reflective diary it is possible to have a more objective view of the module. it was possible to identify patterns in the need to perform some tasks and to list these tasks in the form of a checklist for completion the next time the module is run. the diary also proved invaluable for comparing the work involved in running this on-line module with that involved in running a traditional lecture-based module. the problems encountered whilst updating the module were: • learning the web development tool dreamweaver; • understanding the rationale for the assessments and the marking schemes; • finding the individual web pages for alteration; 71 mireia asensio et al taking over someone else's design: implications for the tutor's rote in networked learning • planning appropriate topics for tutorials. the tasks that needed completing before the module commenced included: • altering all references to dates for lectures and tutorials; • changing module leader contact details; • clearing out the previous cohort's virtual conference contributions; • moving the previous cohort's team work to an archive; • becoming familiar with the learning material. the technical difficulties meant that the tutor was unable to devote as much attention to the pedagogical and social areas of the module. it was found that more than sixteen hours were devoted to making technical alterations before the module started, followed by over thirty-five hours as the module was running. these times were in addition to the allotted tutorial times each week. ideally the module should be technically complete prior to commencement to enable the tutor to concentrate on pedagogical and social issues as the module is running. as a result of the tutor's experiences a list of suggestions for designing online courses is presented: • keep timetable elements in a limited number of pages, which are documented; • ensure there is good documentation with answers to exercises and mcqs (multiple choice questions); • there should be a clear map or outline with navigation between pages; • ensure discussion topics have clear objectives; • be rigorous when defining assignment specifications with marking schemes. discussion and conclusion if we think of the implications for the tutor's role in this situation, it is important to be aware of time pressure and lack of course design documentation and specification, and how this may affect the order of the priorities that we need to pay attention to. in this particular case the order of activities that the tutor had to take into account given the" time pressures clearly prioritized the technical and managerial side over the pedagogical and social responsibilities, which, according to collins and berge's (1996) classification of tutor activity, should be the first two priorities in the networked learning environment. it appears that the literature on the tutor's role and activities in a networked learning environment refer more widely to cmc (computer-mediated communication). however, when addressing a web resource-based course, we have found that other implications for the tutor arise. the study we have made of a web-based course might suggest that the role of the tutor is not only that of the moderator/facilitator/guide, but also that of the subject expert. this is clearly reflected on the content materials and resources developed by the tutor and in turn in the overall design of the course. 72 alt-j volume 9 number 3 this case study illustrates that design encapsulates an individual or individuals' view of knowledge and understanding of the students' learning experience. we have explored elsewhere the practitioners' accounts of design in networked learning and how the design process and outcome is informed by assumptions about the nature of learning and the learning process (jones and asensio, 2001). this case study brings about an added understanding of design itself. taking over someone else's design implies taking over someone else's representation of knowledge: this is exemplified here by the tutor's experience of having to familiarize herself with the course design and content. in this situation it also appears that the face-to-face tutorials were a way to embed pedagogy and social interaction. interestingly, the students in the course did not report that they had been directly affected by the tutor being new to the course and found the tutorials useful. this supports the belief that the value did not lie in the content alone, but in how it was managed by the tutor and used by the students. this raises the issue of whether the added value of the course is not the stated content in itself (which is only a partial and personal representation) but the input of the teacher who is taking over the course, and the students' approach to the course material. clearly face-to-face tutorials are not always possible in the networked learning environment and the increasing demand to provide flexible and open access to a global student market may hinder face-to-face contact. the danger is that increasing numbers of tutors in he may be in a similar situation in the future, and that time constraints may adversely affect the more pedagogical and social aspects of the teaching activity. acknowledgement this work was part-funded by a grant from the committee on awareness, liaison and training of jisc (the joint information systems committee of the uk higher education funding councils). the views expressed here are not necessarily those of jisc calt. further information about the project can be obtained from the project's web site (http://csalt.lancs.acuk/jiscf). we would like to thank the previous tutor in the course and all those students that took part in the interviews. we would like to acknowledge the contributions of other members of the project team: peter goodyear, vivien hodgson, christine steeples, susan armitage, mark bryson, michael o'donoghue and david hutchison. finally we would like to thank the two anonymous reviewers and clive young for their helpful comments. references collins, m. and berge, z. (1996), 'facilitating interaction in computer mediated on-line courses', on-line at http://emoderators.com/moderators/flcc.html. feenberg, a. (1989), 'the written world: on the theory and practice of computer conferencing', in r. mason, r. kaye and a. kaye (eds), mindweave: communications, computers and distance education, oxford: pergamon press. out of print but available online at http://www-icdl.open.ac.uk/mindweave/mindweave.html. harasim, l., hiltz, s. r., teles, l. and turoff, m. (1995), learning networks: a field guide to teaching and learning online, cambridge, ma.: mit press. 73 mireia asensio et at taking over someone else's design: implications for the tutor's role in networked learning jonassen, d. h. (1996), computers in the classroom: mindtools for critical thinking, englewood cliffs, nj: merrill, prentice hall. jones, c. and asensio, m (forthcoming, 2001), 'designs for networked learning: a phenomenographic investigation of practitioners' accounts of design', to be published in an edited collection, springer verlag. kerr, e. (1986), 'electronic leadership: a guide to moderating on-line conferences', ieee transactions on professional communications, pc, 29 (1), 12-18. koschmann, t. (ed.) (1996), cscl: theory and practice of an emerging paradigm, mahwah, nj: lawrence erlbaum associates. marton, f. (1994). 'phenomenography', in t. husen and t. n. postlethwaite (eds), the international encyclopedia of education (2nd edn), oxford: pergamon, 4424-9. marton, f., and booth, s. (1997), learning and awareness, mahwah, nj: lawrence erlbaum associates. mason, r. and kaye, a. (1990), 'towards a new paradigm for distance education', in l. harasim (ed.), online education: perspectives on a new environment, new york: praeger. moon, j. (1999), learning journals handbook for academics, students and professional development, london: kogan page. nicholson, b. and bird, d. (1998), 'using the world wide web and computer conferencing to improve campus based teaching and learning', http://salford.ac.uk/iti/papers/ m320eval.htm [last accessed 5 july 2000]. reigeluth, c. (ed.) (1999), instructional-design theories and models volume ii. a new paradigm of instructional design, london: lea. salmon, g. (2000), e-moderating: the key to teaching and learning online, london: kogan page. 74 alt-j, research in learning technology vol. 12, no. 3, september 2004 issn 0968–7769 (print)/issn 1741–1629 (online)/04/030295–05 © 2004 association for learning technology doi: 10.1080/0968776042000259591 discussion understanding and using technological affordances: a commentary on conole and dyke tom boyle* & john cook london metropolitan university, uk taylor and francis ltdcalt120308.sgm10.1080/0968776042000259591alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2004association for learning technology123000000september 2004tomboylelearning technology research institutelondon metropolitan universitylondont.boyle@londonmet.ac.uk introduction the paper by conole and dyke sets the context by pointing to a number of problems that inhibit the widespread, effective use of information and communication technologies (ict) to support learning. they argue that this situation highlights the need to explore a theoretical basis for the use of ict to support learning. the central argument of the paper is that the notion of affordances can make a significant contribution to this endeavour. the paper aims to articulate the potential impact of these affordances primarily through the development of a taxonomy. it draws on social constructivist theory to help understand and articulate the impact of these affordances. the concept of affordances is potentially both rich and provocative. conole and dyke provide a refreshing and diverse look at the theoretical basis for the use of ict to support learning. we have structured our commentary around six questions that are provided by conole and dyke in their discussion section. whilst examining these questions we highlight various issues raised by the paper, which we believe, need further consideration and clarification. commentary question 1: how valuable is the concept of affordances and does its application provide any really new insight into the inherent properties of technologies? there are really two questions here. this section will concentrate on the first question, which involves clarifying the central concept of affordances. this central concept in the paper is only briefly explored before the paper moves to building a taxonomy. mcgrenere and ho (2000) point out, however, that the concept of * corresponding author: learning technology research institute, london metropolitan university, uk. email: t.boyle@londonmet.ac.uk 296 t. boyle & j. cook affordance is not well understood, and that there is considerable ambiguity and confusion in the use of the term. a brief exploration of how the concept has been used in the psychological and human computer interaction (hci) literature helps to clarify the important issues surrounding the different uses of this term. the concept of affordance was developed by j. j. gibson. gibson (1977, 1979) argued for an ecological approach to understanding human perception. he argued that the environment and animals have co-evolved. there are features of the environment that afford (i.e. enable) perception and action in that environment. they are not constructed by the person. they exist independently in the environment, and are discovered rather that constructed by the human (or animal) actor. thus a rigid surface stretching to the horizon under our feet affords locomotion; an object of a certain size affords grasping and so on. gibson’s concept has to be understood within this ecological approach. he argues, ‘the world of physical reality does not consist of meaningful things. the world of ecological reality … does’ and ‘their meanings can be discovered’ (gibson, 1979, p. 33). the idea of affordances was popularized in the hci community by donald norman in his book ‘the psychology of everyday things’ (norman, 1988). norman, however, introduced a number of changes, which led to subsequent confusion and variability in the use of the concept. the most fundamental difference relates to norman’s concern, as a designer, with making affordances salient so that users can easily perceive them. norman conflates two important but different things—designing the utility of an object and designing the way in which that utility is conveyed to the user. in his later work norman acknowledges the confusion and seeks to distinguish ‘real from perceived affordances’ (norman, 1998, p. 123). mcgrenere and ho (2000) provide an informative analysis to clarify the situation. they distinguish the utility of an object, the actions it affords for the user, from the usability of an object, which is related to the perceptual information that signals the affordances. they propose that there can be degrees of affordance (warren, 1995), and that affordances can be related in sequential or nested structures. this analysis and clarity in the use of the term is missing in the article by conole and dyke which moves rapidly to using the term before analysing and articulating it. the danger is that it falls into the problem identified by mcgrenere and ho, ‘ as the concept of affordances is used currently, it has marginal value because it lacks specific meaning’ (mcgrenere & ho, 2000, p. 8). question 2: how valid is the methodological approach suggested? another important issue is a critical consideration of the ‘taxonomic’ approach to capturing and representing the implications of the concept. conole and dyke argue that practitioners are often unclear about how to use the technology appropriately. they then present a taxonomy of ict affordances as a contribution to solving this problem. this taxonomy is grounded in the base of ‘current social theory and critique’. understanding and using technological affordances: a commentary 297 the affordances listed are various. the first is called ‘accessibility’—the internet ‘affords’ opportunities for accessing information and knowledge in a new way. the authors, however, use accessibility in a non-standard way. they argue for a shift in the primary challenge from finding to selecting relevant information. this diverges from the standard concern with accessibility as ‘access by everyone regardless of disability.’ (web accessibility initiative: http://www.w3.org/wai/). another affordance proposed by the authors is ‘speed of change’, which refers to attempts to understand the question of how technology can ‘be used to enable students to navigate their way through the myriad of changing information and make more informed decisions’. however, it is left unclear how this can act as affordance. a large list of ‘affordances’ then follows in the paper; which includes diversity, communication and collaboration, reflection, multi-modal and non-linear learning, and so on. rather than elaborating on how any one of these ‘affordances’ could be relevant to a learner or a practitioner the authors tend to indulge in a certain amount of hopeful expectation that affordances and abilities will simply emerge; for example in the context of reflective affordances: ‘perhaps new forms of reflection and critique will emerge in response to more transitory and digital text.’ question 3: what other approaches might be taken? the third question points to the theoretical tension introduced by the use of this concept—the slotting into a social constructivist approach ignores the quite different theoretical framework within which gibson developed the concept of affordances. the concept of affordances arises from an (biological) ecological approach to human cognition and perception. the natural home for an approach based on gibsonian affordances is not social theory. the concept might have been used to challenge the limits of social theory rather than be simply being assimilated into it. a key question is—does the concept of affordances really help? the authors articulate certain opportunities that the technology makes available to practitioners. they then balance these positive features with difficulties or challenge that are raised for users. if these barriers are not dealt with effectively then the opportunities may not be exploited. how does the concept of affordance enrich this approach? if we stick with gibson’s original concept, then one approach is to articulate the ‘ecological’ approach that is at the base of gibson’s concept of affordances, and to contrast it with the predominant social constructivist approach. this theoretical tension could be uncomfortably productive, as gibson’s approach is (a) not constructivist, and (b) not social, though it is, in its own way, interactionist. the technology changes the ‘habitat’ in which we live. this new habitat affords new ways of doing things. an articulation of the affordances of this new environment enables us to adapt more effectively to the (new) opportunities, and avoid the (new) pitfall where ‘new’ here may indicate a significant degree of change that has marked implications for adaptive behaviour. the transition from the natural habitat that concerned gibson to the technological habitat of ict supported learning provides ample opportunities for theoretical elaboration. 298 t. boyle & j. cook question 4: the approach suggested reflects a particular interpretation of social theory. is this a limitation and might a broader analysis of other social theories yield new insights? if the term is going to be used in a post-gibsonian way to focus on social factors, which could be productive, then there is an existing relevant body of work (e.g. laurillard et al., 2000; kreijns & kirschner, 2001; kreijns et al., 2002). kreijns et al. (2002), for example, have proposed a theoretical framework for ‘social affordances’ by suggesting the incorporation of certain properties in computer supported collaborative leaning (cscl) environments. these properties are meant to act as social contextual facilitators, i.e. social affordances, to initiate and sustain learner’s social interactions. ultimately these properties are intended to create a social space amongst the members of distributed learning groups. kreijns et al. (2002) base their environment on a theoretical framework that suggests the embedding of certain properties in cscl environments: that act as social-contextual facilitators relevant for the learner’s social interactions. when they are perceptible, they invite the learner to act in accordance with the perceived affordances, i.e., start a task or non-task related interaction or communication. (kreijns et al., 2002) furthermore kreijns et al. (2002) include a software tool that aims to provide the learner group with awareness about the others in the task and in the non-task context. to merely provide affordances for group interaction may be insufficient for the development of a true knowledge-building community. hewitt et al. (1997) have suggest that a set of tools that permit the individual to monitor communal activity and their own participation in the activity is also required. this a productive direction that is currently being explored by garnett and cook (submitted). however, we note that the authors briefly mention as a footnote a planned dialogplus tool. question 5: practitioners are still exploring the potential of new technologies, and the current uses of technologies often do not take full advantage of the medium. therefore, how can practice take full advantage of the affordances of ict? and question 6: does understanding of the affordances actually get us closer to improvement in practice and is this a useful framework? finally, we briefly comment on the potential limits in the scope of the application of this concept to e-learning and present an alternative. developing taxonomies that make affordances explicit so that practitioners can make informed choice about existing technology may prove useful. however, perhaps another productive goal would be to develop a framework that allows us to envisage new tools for learning and to describe the communicative contexts in which these tools could be used. for example, garnett and cook (submitted) envisage the provision of various computerbased tools to encourage learners to frame their activities in light of a problem (rather understanding and using technological affordances: a commentary 299 than a topic), e.g. a tool to create discussion notes; a tool to author with (e.g. to develop a personal web page), a tool to structure learners’ interactions in light of statements of proposed problem solutions and information needed to advance that solution. located at the heart of these learning tools is a community knowledge map. this map would be used to identify hot-spots of community activity and clusters of related work, and to depict the work of the entire community rather than the work of an individual author. what is required in all approaches is serious attempts to include new empirical observations of learning in the evolution of new frameworks, tools and systems to support (i) tutors as they make decisions about the use of new technology, (ii) learners as they learn, and (iii) systems designers and developers as they envisage new innovative tools to support learning. conclusion conole and dyke have provided a useful examination of the use of affordances to provide a theoretical basis for ict learning support. we hope some of the issues raised above lead to a productive debate in this area. references garnett, f. & cook, j. (submitted) models of learning with icts in community settings, technology, pedagogy and education. gibson, j. j. (1977) the theory of affordance, in: r. shaw & j. bransford (eds) perceiving, acting and knowing (hillsdale, nj, erlbaum). gibson, j. j. (1979) the ecological approach to visual perception (hillsdale, nj, lawrence erlbaum). hewitt, j., scardamalia, m. & webb, j. (1997) situative design issues for interactive learning environments: the problem of group coherence, paper presented at the annual meeting of the aera, april. laurillard, d., stratfold, m., luckin, r., plowman, l. & taylor, j. (2000) affordances for learning in a non-linear narrative medium, journal of interactive media in education, 2. available online at: www-jime.open.ac.uk/00/2 kreijns, k. & kirschner, p. a. (2001) the social affordance of computer-supported collaborative learning environments, 31st asee/ieee frontiers in education conference, 10–13 october, reno. kreijns, k., kirschner, p. a. & jochems, w. (2002) the sociability of computer-supported collaborative learning environments, educational technology & society, 5(1). available online at: http://ifets.ieee.org/periodical/vol_1_2002/kreijns.html mcgrenere, j. & ho, w. (2000) affordances: clarifying and evolving a concept, proceedings of graphics interface, may, montreal. available online at: http://www.dgp.utoronto.ca/∼ joanna/ papers/gi_2000_affordances.pdf norman, d. (1988) the psychology of everyday things (london, basic books). norman, d. (1998) the invisible computer (boston, mit press). warren (1995) constructing an econiche, in: p. flach, j. hancock, j. caird & k. j. vicente (eds) global perspectives on the ecology of human-machine systems (hillsdale, nj, lawrence erlbaum). calt632491 173..177 editorial theory in learning technology this special issue is being published at a significant point in time in relation to simultaneous changes in higher education, in technology and in the field of learning technology itself. as the 2011 alt c conference themes clearly state, learning technology needs to learn to thrive in a colder and more challenging climate. in this difficult political and economic environment technological trends continue to develop in terms of mobility, cloud computing, ubiquity and the emergence of what has been called big data. e-learning has become mainstream and the field of learning technology itself is beginning to stabilise as a profession. profession here is understood as a knowledge-based occupation and a form of cultural work where the tasks addressed are human problems amenable to expert advice and distinguishable from other kinds of work by the fact that it is underpinned by abstract knowledge (macdonald, 1995). abstract knowledge is the focus of this special issue, in the form of theory. the hallmarks of a profession include both a codified body of practices (embodied knowledge including competence) and abstract knowledge in the form of theory which is expressed in a codified body of principles; in conceptual knowledge (denning 20011). indeed learning technology, like other applied technology fields, has a distinct relationship to explicit, codified, systematic and scientific knowledge (jones 2004). czerniewicz (2010) has shown how learning technology as an applied field takes the form of a horizontal knowledge structure, segmented and expressed as co-existent parallel specialist ‘languages’, with a tendency to repeat and proliferate. a bernsteinian framework shows how theory is manifest in a “general approach discursive plane”, where metalanguages are produced to legitimize how phenomena are understood and interpreted, and to determine what counts as a proper description. it is important that the theories expressed through these metalanguages are developed and made explicit. coherent theoretical frameworks are needed to enable integration across the segmented clusters so that generalizations can be made, lessons learnt across multiple sites, and a community of researchers enabled to share a common language to build knowledge together. in addition, because of the emergent nature of the field and the particular nature of learning technology in terms of practice, several related issues pertaining to theory and its role arise. these include: � the place of theory in an applied practice-based field; � the legitimacy of specific theories; � the related contestation between theoretical positions and claims; and � the tensions and disjunctures between the empirical and the theoretical in research. research in learning technology vol. 19, no. 3, november 2011, 173–177 issn 2156-7069 print/issn 2156-7077 online � 2011 association for learning technology http://dx.doi.org/10.1080/21567069.2011.632491 http://www.tandfonline.com the need for theory development in learning technology echoes arguments made in educational research more broadly, where for example ashwin (2009) has argued that research in higher education has been about exemplification rather than about theory development. the case for theory and for theory development is unsurprisingly one that weaves through all the papers in this special issue. in an extended and articulate case, bennett and oliver (this issue) argue that the lack of theoretical perspective in learning technology risks making the field at best one that is “narrow and derivative”, and that the focus on applied design work in learning technology is too easily driven by common sense assumptions associated with matters of practical implementation. both bennett and oliver and howard and maton (this issue) make the case for the need to build theory and howard and maton argue that without theory the danger will be that knowledge remains at the level of empirical description, rather than resting on the principles underlying the data. we believe that the papers in this special issue provide an antidote to these concerns, providing numerous examples of the opportunities for exploring the role of theory in learning technology research. in the first instance, authors in this special collection theorise the applied nature of the field itself by conceptualising different aspects of use and of practice. cram et al. (this issue) consider the practices of problem solving as understood by theories of expertise while johri (this issue), drawing on orlikowski, makes the case for practicebased bricolage. such practice-based theorisations are welcome additions to learning technology, because they demonstrate in different ways how knowingin-action and reflective practice are intimately associated with theoretical knowledge. the field’s traditional focus on design is not neglected, and we value paavola et al. (this issue) for rigorously problematising the challenges associated with theorising design principles. it is worth repeating their conclusions that “conceptualisations can be more resistant to change than expected” and that “both practical applications and theoretical development require conscious development and cannot be taken for granted as a result of design-based research”. this kind of honesty is essential for meaningful progress to be made in theory development in the field. also essential is a reminder of the inextricable relationship between the empirical and the conceptual, especially relevant in the current climate of big data and learning analytics. we find sayer’s (1992) three versions of theory helpful as a starting point, as he considers the three dimensions of theory as follows: (1) theory as an ordering framework which permits observational data to be used for predicting and explaining empirical events (2) theory as conceptualisation, prescribes a particular way of conceptualising something (3) theory as hypothesis or explanation (sayer 1992, 50) sayer criticises the ordering framework (1) and hypothesis and explanatory views of theory (3) for their failure to deal with the indirect nature of our knowledge of the world. in his view conceptualisation lies at the heart of observation, a point of contact made famous in einstein’s comment that it is theory that decides what we observe2. data is not neutral, its very selection indicates a theoretical perspective. data does not produce facts that are direct reflections of the world, rather facts about the world, like theory, are constructs of human cultural engagement with the world and are never direct. in our view theory can generalise across a variety of 174 editorial contexts and across time and at its best can compress experience into robust, considered and usable forms. because we do not believe that facts provide direct access to the world we stand against the tradition (perhaps especially strong in england) to distrust theory and to emphasise the empirical. we stand firmly against this trend and argue that a sound theoretical basis for learning technology in conceptualising the research object is essential and that empirical work that fails to engage with theory has a very limited ability to develop and inform the field. currently the anti-theoretical stance has taken on a new importance with the development of the idea of a “fourth paradigm” (hey, tansley, and tolle 2009). famously anderson has claimed: this is a world where massive amounts of data and applied mathematics replace every other tool that might be brought to bear. out with every theory of human behavior, from linguistics to sociology. forget taxonomy, ontology, and psychology. who knows why people do what they do? the point is they do it, and we can track and measure it with unprecedented fidelity. with enough data, the numbers speak for themselves. (anderson 2008) in a world of big data the idea that theory is at best irrelevant and at worst counterproductive is extremely dangerous. in educational technology the idea of learner analytics could easily be developed to suggest that if we simply collect enough raw data on students then theory will become irrelevant. such a stance in relation to theory ignores all the known weakness of numerical data and the weaknesses of ‘big data’ in particular (see for example boyd and crawford 2011). in a world in which there is a data flood in which new data sources impact on all the disciplines that learning technology draws on, a renewed engagement with theory will be essential to ensure that the data we collect for analysis is not garbage so that the answers derived from it are not just as poor. it is notable that the papers in this issue are largely drawn from the social sciences, even when these are situated in relation to technology and information sciences. this suggests either a narrowing or a resolution and the beginning of some coherence with regard to the multi-disciplinary nature of the field. learning technology has of necessity drawn on a range of disciplinary areas that often have contrasting and even conflicting assumptions about the nature of knowledge and how theoretical understanding can be achieved. apart from education and educational theory, learning technology has drawn significantly from a range of disciplines including psychology, sociology and anthropology, computer and information sciences etc. a consequence of this diversity is that the claims to knowledge are often based on deeply ingrained philosophical stances, disciplinary paradigms and entrenched research methodologies. a striking example of this range is the positivist influenced paradigm of research commonly found in psychology, involving largely quantitative methods and experimentation, contrasted with the research paradigm in the cultural and social sciences which adopts a largely interpretivist approach and conducts research in real-life settings. the sheer range of disciplinary sources presents difficulties for anyone trying to outline or develop a coherent theoretical stance applying to learning technology as a whole. it is interesting that the call for papers for this special issue on theory in learning technology drew few papers with an emphasis on learning theory and that there is little evidence of a strong psychological, computer science or educational research influence. it is also of note that the dominance of learning research in learning technology 175 theory especially in terms of the instructivist/constructivist see-saw noted in recent years (see czerniewicz 2010) is barely evident here. there is a sociological bent indicative of what we perceive as a general shift to the socio-cultural context of learning. we note too an emphasis on and an interest in the persistent problem for learning technology: the relationship between technology and learning and the co-construction of education and technology. three of the papers in this edition draw on actor network theory (ant) (johri, edwards et al., and wright and parchoma this issue) while a third makes use of social studies of science and technology (scot) (jones and bissell this issue). these papers contribute to exploring the relationship between new technologies and learning. all three suggest an interpretive flexibility in relation to technology that is not fixed by design and emerges in interaction and use. the articles in this issue also demonstrate how theory can locate discussion and research in a broader academic and scholarly discourse (see for example johri in this issue and his consideration of four dimensions of theory). equally theory can be criticised for occluding aspects of experience, making researchers and practitioners blind to some aspects of their own context, or the context of their work, and making the simple appear both complex and obscure. two papers in this special issue interrogate dominant discourses (edwards, tracy, and jordan; wright and parchoma). wright and parchoma question the discourse surrounding the use of mobile devices in education while edwards, tracy, and jordan suggest that framing education as spatial orderings raises questions for those dominant discourses that focus on learning and teaching in education. the article by hall (this issue) makes a general call for placing critique, based on a deep understanding of the socio-cultural contexts within which technology is deployed, at the core of learning technology research. the current major crisis in neo-liberal capitalism suggests that critique is likely to emerge as a major theme in the near future. as governments cut public expenditure and look for quick fixes to fundamental problems, learning technology may become a battleground in which economic constraints and a consumerist philosophy applied to education come into conflict with popular movements focused on the public role of the university. in this harsh environment original responses are essential. the collection of articles published in this special issue provides a rich diversity of sites and an array of conceptual approaches to learning technology research and innovation. the cases provided are evidence of proactive engagements with challenging conditions through robust research which is slowly closing the spaces between fragmented thinking and isolationist approaches. this is a contribution to both community building and cumulative knowledge building in the field. notes 1. see an elaboration of this in hodgkinson-williams and czerniewicz, (2007). 2. this quote was related by heisenberg and quoted in unification of fundamental forces (1990) by abdus salam according to http://en.wikiquote.org/wiki/albert_einstein. references anderson, c. 2008. the end of theory: the data deluge makes the scientific method obsolete. wired magazine 16, no. 7 [online]. http://www.wired.com/science/discoveries/ magazine/16-07/pb_theory (accessed september 27, 2011). 176 editorial ashwin, p. 2009. relations between theory and data in research in higher education. education research seminar, october 7, lancaster university. http://www.lancs.ac.uk/fass/ doc_library/edres/09seminars/ashwin_07.10.09.pdf (accessed september 27, 2011). boyd, d., and k. crawford. 2011. six provocations for big data. paper presented at oxford internet institute’s “a decade in internet time: symposium on the dynamics of the internet and society”, september 21. ssrn. http://ssrn.com/abstract=1926431 (accessed september 27, 2011). czerniewicz, l. 2010. educational technology – mapping the terrain with bernstein as cartographer. journal of computer assisted learning (jcal) 26, no. 6: 523–34. denning, p.j. 2001. the profession of it: who are we? communications of the acm 44, no. 2: 15–9. hey, t., s. tansley, and k. tolle, eds. 2009.the fourth paradigm data-intensive scientific discovery. microsoft research, redmond, wa. http://research.microsoft.com/en-us/collaboration/fourthparadigm/ (accessed september 27, 2011). hodgkinson-williams, h., and l. czerniewicz. 2007. educational technologists in higher education institutions in south africa: moving beyond random acts of progress. paper presented at research based elearning, hogsback. http://www.cet.uct.ac.za/files/hodgkinson-williams%20&%20czerniewicz%20full%20paper.pdf. jones, c. 2004. theory and the practices of learning technology. in networked learning 2004: proceedings of the fourth international conference on networked learning 2004, ed. s. banks, p. goodyear, v. hodgson, c. jones, v. lally, d. mcconnell, and c. steeples. lancaster: lancaster university. macdonald, k. 1995. the sociology of the professions. london: sage publications. sayer, a. 1992. method in social science: a realist approach. 2nd ed. london: routledge. chris jones the open university, uk c.r.jones@open.ac.uk laura czerniewicz university of cape town, south africa laura.czerniewicz@uct.ac.za research in learning technology 177 research in learning technology vol. 19, no. 1, march 2011, 5–17 issn 0968-7769 print/issn 1741-1629 online © 2011 association for learning technology doi: 10.1080/09687769.2010.548505 http://www.informaworld.com learners’ reflexivity and the development of an e-learning community among students in china zhen li* school of education, university of southampton, southampton, uk taylor and franciscalt_a_548505.sgm (received 15 august 2010; final version received 5 december 2010) 10.1080/09687769.2010.548505research in learning technology0968-7769 (print)/1741-1629 (online)original article2011taylor & francis191000000march 2011dr zhenlizl@soton.ac.uk the experiences of chinese learners on two e-learning programmes in china were investigated, focusing particularly on the formation of learning communities. data were collected using a range of instruments to access the learners’ perspectives in depth and detail. archer’s account of reflexivity as the mediating power between structure and agency is applied to understanding how learners succeeded in one programme in forming a learning community, through their negotiated responses to the existing structural and cultural conditions, whereas little evidence was found of the emergence of learning community in the other case. further understanding emerges from reappraisal of confucian philosophy of learning and social relationships, how these influenced the participants’ prior learning experiences and how they play a part in their responses to the e-learning experience. keywords: learning community; reflexivity; archer; chinese learners; e-learning introduction the focus of learning theorisation has moved in recent years from individual behaviour and psychology to social interaction and practice, recognising in doing so the significant role of context in the learning process. the unit of analysis has moved from learners as bounded biological organisms, independent of context, to socially located “communities of practice” (lave and wenger 1991) or “activity systems” (cole and engeström 1993). in studies of e-learning a similar shift in focus has widely occurred, from seeing technology as an individualised instructional tool to portraying it as a more discursive and communicative medium (conole, smith, and white 2007). attention has been drawn to pedagogical innovations such as computer-supported collaborative learning that reflect the contemporary dominance of socio-constructivist learning approaches. e-learning design has increasingly focused on the capacity of digital networks to support collaborative knowledge-building and a learning environment to facilitate learners’ collaboration, online communication and reflection, to promote the formation of a learning community. conole (2007) argues, however, that many of these innovations remain as ideas only and may not have appeared as actual changes in practice. despite the fact that the term “community of practice” (wenger 1998) has been widely used and discussed in e-learning design and practice, the nature and develop*email: zl@soton.ac.uk 6 z. li ment of learning communities through the actual experiences of e-learners, from their own perspectives, are still under-explored. some initiatives have been taken in the united kingdom to understand learners’ use and experience of technology across programmes and subject disciplines (for example, creanor et al. 2006; conole et al. 2006) but more attention is still needed from the perspective of learners’ experiences if we are aiming to facilitate the emergence of learning communities in e-learning. it should be kept in mind that it is their – the learners’ – communities that we hope will emerge and it is therefore important to examine their experiences from their own perspectives. this paper uses two e-learning programmes in china as case studies to explore this issue; in each of these, a range of methods was used to collect data on the learners’ e-learning experiences. theoretical framework this study rejects technological determinism in e-learning research and analyses the actual experiences of learners in developing a learning community. in doing so, it draws on margaret archer’s (2000, 2007) work on reflexivity as the mediating power between structure and agency, defined as “the regular exercise of the mental ability … [of people] … to consider themselves in relation to their (social) context and vice versa” (archer 2007, 4). the pursuit of any social practice entails interplay between the causal powers of subjects themselves (agency) and those of relevant structural and cultural properties (structure). through agential reflexivity, these two sets of causal power interact and an active role is thus restored to the individual agent. in this section, key elements of archer’s analytical framework will be outlined, with a more detailed engagement appearing in the later discussion of the study’s findings. archer distinguishes between the existence of structural and cultural properties and subjects’ responses to them. herein lies the important part played by reflexivity: people can “design and determine their responses to the structured circumstances in which they found themselves, in the light of what they personally care about most” (archer 2007, 11). the causal power of structural and social properties has to be activated by agents through social action. archer (2007, 13) argues that the “deliberative process involved … is emotionally charged rather than being a simple exercise in instrumental rationality, because … our emotions (as distinct from moods) are commentaries on our concerns”. by employing archer’s theoretical framework in e-learning contexts and recognising reflexivity as a distinct human power, this research treats learners as active rather than passive agents, who can exercise some governance in their own lives and learning if we are to understand the interactions between the learners and the e-learning programmes in this study as forms of socio-cultural action, then, following archer (1996, xxi) , we must acknowledge that any such action takes place in a “context of innumerable interrelated theories, beliefs and ideas which had developed prior to it, and … exert a conditional influence on it”. the cultural context within which the learners in the study are working and to which their previous learning experience belongs is that of contemporary china. in order to understand better this context and these experiences, the study reappraises the nature and processes of learning in the philosophical tradition of confucianism, in recognition of its tremendous influence on education in china. it challenges over-simplified (and often erroneous) interpretations of confucian writings on learning and their miss-application in research in learning technology 7 practice. in the confucian tradition education is understood as a whole person constructing and transforming his/her life experience. practice lies at the heart of this process and confucian learning is a process of constantly practicing ‘ren’ ( , benevolence). this significance of action gives the confucian educational tradition a close connection with western thinkers, such as archer, who claim the centrality of practice in the emergence and development of our sense of self and all forms of knowledge. the study two e-learning programmes were chosen as case studies: case one is a sino-uk collaborative ma e-learning programmed signed for in-service teachers; case two is a part-time ba e-learning programme for adult learners, delivered through a chinese university. in case one, 87 in-service teachers based at six different schools participated in the programme and each of them was assigned to study one of the three modules in the programme. in case two, 250 students from a range of locations across the country responded to an initial survey and became the subjects of the study. each of these students was studying one of five different subjects in the programme. the cases were chosen on two grounds: there is a striking difference in the course design origins – one mainly due to western practitioners inspired by social constructivist approaches, the other based on a traditional didactic model; and they target similar populations. learners’ responses to the different designs are therefore highlighted through the comparison; at the same time the similarity in the learners’ social and cultural backgrounds allows a degree of generalisability, which is often problematic in case studies. data were collected through five main tools: questionnaires, individual and focus group interviews, digital audio learning diaries, informal discussions and tutor reports. quantitative data gathered by questionnaires provided key characteristics of the learners and their backgrounds, and identified appropriate individuals to participate in the qualitative phases of the research. in this paper, the evidence is mainly drawn from the detailed accounts of learners’ experiences obtained from the instruments other than questionnaires: there are four focus groups interviews with 87 learners conducted in case one, and two focus groups plus 22 individual interviews with 35 learners together in case two, combined with tutor reports, informal discussions with course designers and administrators on the sites in both cases. in addition to holding interviews, a students’ digital audio diary approach was used in case two to capture learners’ experiences and interactions with greater immediacy and in their natural settings. for adult learners with a full-time job commitment, keeping a written diary is time-consuming (timms et al. 2004), while an audio diary allowed the capture of day-to-day events that were significant to the learners without imposing a serious burden. data from the various qualitative instruments were organised and coded according to emerging patterns. archer’s account of reflexivity and emergence of social identity enabled the difference between the ‘existence’ of cultural and structural properties and the learners’ ‘responses’ to them to emerge from the data, revealing the process of learners’ reflexively interacting with the e-learning environment. there were striking differences in learners’ experiences in responding to the different learning models in these two cases. 8 z. li findings and discussions the constructivist design in case one was inspired by the desire to build a shared community of practice, with learners learning collaboratively. all the materials were transferred to an online format and were trialled in a virtual learning environment. there was a mix of resources and activities in every section of each study module: case studies; reading and reflective writing (via a journal); forum discussions; and three assignments. learners were expected to finish all the learning materials within a certain time, but the form of assessment was not made clear from the outset, since it was not considered to be a necessary part of the piloting process. far from building a shared community, however, learning was passive and chaotic. most learners did not fully appreciate the instructional design in the way that the designers had expected. for instance, there was generally a lack of understanding of the purpose and use of the online discussion forum that was a part of the programme design. sometimes i logged in to the course website, but didn’t know what to do, so i logged out again. since we really haven’t got much spare time after the whole day’s work, it is impossible to spend so much time to find out what to do instead of actually doing it. (case one: student focus group 1) i tried to post my question or respond to others in the forum, but it’s very frustrating because i’m not sure how it works. does it only allow commenting on a thread once or you have to comment at least a couple of times? (case one: student focus group 2) consequently, neither sufficient in-depth discussion nor effective collaboration developed. most learners were seeking synchronous communication during their study but the chatting facility in the virtual learning environment was reported to be “poorly functioning” since it was not valued in the programme design. a learning community failed to emerge in this case. in contrast, the learning model in case two was more content-driven and didactic. the learning objectives, course demand and assessment procedures were very explicit and very similar to the conventional programmes delivered on campus. learning materials were designed for individual use. online discussion forums were mainly designed for teaching assistants to answer learners’ questions and solve technical problems. no collaborative learning activity was promoted by the course designers and tutors. interestingly here, in compensating for the shortage of teacher–student interaction in the formal e-learning settings in the course, a high level of self-help and spontaneous collaboration emerged, both online and offline, amongst the learners. learners made intensive use of the communicative tools they used in everyday life as well as the forum provided by the school to build up an intensive network amongst themselves. they organised learning groups to exchange information, solve problems, tackle assignments and prepare for examinations. learners from the same city also organised social events for people to meet with each other from time to time. learners’ interviews and diary entries illuminate this phenomenon: the best thing about coming to this course is that i got to know so many people who are doing the same course. we are varied in age, profession and location. but we help each other and learn from each other. many of us became very good friends. this is something really wonderful (case two: student interview 1). research in learning technology 9 learning depends on good understanding, understanding depends on good communication. sometimes participating in communication seems more important than giving the right answers (case two: student diary entries 5:2) learning communities emerged in this case despite a lack of intention in the course design. learners created their own social networks to support their learning, tailored to their particular needs rather than being constrained within those provided by the courses. they had strong personal views on how technologies may or may not support their learning activities. archer’s theorisation of reflexivity and social identity can help us to understand the contrasting findings in these two cases and the very nature of the emergence of learning community among students. a brief summary of archer’s (2000) theorising of social identity archer (2000) introduces a stratified view of ‘the subject’ whose different properties and powers (personal emergent power) emerge at each level: the self, the person, the agent and the actor. among these, the self – the continuous sense of being one and the same subject – emerges early in life and is the source of reflexive selfconsciousness that lasts throughout life. the person refers to our unique personal identity. our ‘social selves’ – the agent and the actor – emerge respectively through our involuntary embroilment in society’s role-array and they are dependent upon “the prior emergence of a continuous sense of self and are co-dependent with the emergence of personal identity” (archer 2000, 255).the process through which agents transform themselves in pursuing social change and obtaining a social identity is summarised in figure 1. figure 1. realism’s account of the development of the stratified human being.source: archer (2000, 260). primary agents are defined as “collectivities sharing the same life-chances” (archer 2000, 263) and therefore everyone is necessarily a primary agent by being assigned to positions on society’s distributions of scarce resources (archer 2000). the pre-existence of those structures, which shape the situation in which we find ourselves, impinges upon us without our compliance, consent or complicity, leading to the aggregate status of primary agents. in order to reshape or retain the structural and/or cultural features, collective action that has systemic consequences has to be developed. ‘corporate agents’ are those “who articulate shared interests, organise for collective action, generate social movements and exercise corporate influence in decision-making” (archer 2000, 266). archer distinguishes primary agents from corporate agents as primary agents neither express interests nor organise for their strategic pursuit, either in society or in a given institutional sector. however, for primary agents, archer emphasises that “to lack of a say in systematic organisation and re-organisation is not the same as have no effect upon it, but the effects are unarticulated in both senses of the word – un-coordinated in action and unstated in aim” (archer 2000, 266). therefore, primary agents are not deemed intrinsically passive but their passivity represents a suspension of immediate action, often a deliberate suspension (archer 2000, 266). by contrast, corporate agents are active in the sense that they are attempting to bring about certain outcomes, rather than being objects to whom things happen. the social and structural transformations depicted in this process are not the focus in this study given the scale and time involved for these to emerge in practice. however, the process of socialisation of primary agency, in particular from quadrants 2 and 3 in figure 1, can be employed to analyse the data in order to reach 10 z. li an understanding of the emergence of an e-learning community as revealed in this study. in figure 2, the upper part of the figure depicts archer’s model of the acquisition of social identity as represented in figure 1, showing the process of socialisation of primary agents undergoing transformation into corporate agents. corresponding to that, the lower part shows the formation of a learning community in an e-learning context. figure 1. realism’s account of the development of the stratified human being. source: archer (2000, 260). research in learning technology 11 figure 2. the emergence of the (e-)learning community.note: cep, cultural emergent power; pep, personal emergent power; sep, structural emergent power. in an e-learning environment, learners come to study on an online course as individuals. the pre-existence of course content, structure, pedagogy, teaching method, communicative means and technology adopted, plus the institutional environment and culture, have shaped the course context in which e-learners are going to learn. individual learners, in this sense, can be seen in this context as primary agents, who are assigned to similar positions in the course and share similar resources available. however, because of learners’ personal power – reflexivity – they act back on these structural settings. in forming a learning community, individual learners may (or may not) undergo ‘re-grouping’ at some precise point by articulating their shared interests or organising collective actions with others to achieve common aims. wenger (1998) characterises three dimensions of a community of practice: mutual engagement, a joint enterprise and a shared repertoire. common features can be found between these characteristics and archer’s theorisation. by organising their strategic pursuit, corporate agents obtain their membership in a community of practice. this mutual engagement does not require homogeneity but does require being included in what matters (wenger 1998). the things that matter are referred to as a joint enterprise, which in turn can create a source of community coherence; and for archer (2000) the thing that matters is an articulated shared interest. both authors believe that this joint enterprise results from a negotiated response to the situations in which people find themselves. moreover, they both recognise the pre-existence of structural and cultural factors (structural emergent powers and cultural emergent powers ) in shaping the situations, plus specific resources being available and the existence of constraints. archer (2000), in particular, specifies that the distribution of those available resources can include both the material and the ideational. the power to form collective action (archer 2000) or achieve mutual engagement towards a joint enterprise (wenger 1998) resides with the participants. for archer, the nature of our human figure 2. the emergence of the (e-)learning community. note: cep, cultural emergent power; pep, personal emergent power; sep, structural emergent power. 12 z. li powers – reflexivity – means that “the actual outcomes are matters of secondary determination, governed by our inner deliberations about such obstructions and facilitations, under our own descriptions. we often have the capacity to suspend both” (archer 2007, 8). in a similar vein, wenger argues that: yet even when the practice of a community is profoundly shaped by conditions outside the control of its members, as it always is in some respects, its day-to-day reality is nevertheless produced by participants within the resources and constraints of their situations. it is their response to their conditions, and therefore their enterprise. (1998, 79) attention has thus been drawn to this “indigenous enterprise of community of practice” (wenger 1998, 79) or “human power in mediating structural factors” (archer 2000, 8) without ignoring the influence of structural and cultural factors (structural emergent powers and cultural emergent powers ). therefore, examining the interplay of these two set of powers in the e-learning contexts, with the aid of this diagram, can assist with interpreting and understanding the findings of the study regarding the emergence of a learning community in the two cases, as is unfolded in the following discussion. the emergence of an e-learning community according to the discussion above, a learning community emerges when a group of individual learners has articulated its shared interest or joint enterprise through negotiation, and organised strategic pursuit to achieve this stated target, and its collective actions in turn have exerted effects to bring about situational change. there was no lack of common interest in the course among the learners in case one, such as interest in the subject knowledge and improving their own working efficacy. moreover, learners were also evidently eager to share their learning and working experience, exchange ideas and solve problems together. the face-to-face informal discussions during the course study with their peers in the same school were very intensive. as this participant indicated in a focus group interview: i found the discussions with colleagues in the school who are also in the programme have been much more helpful, rather than trying to find someone you don’t know to talk to from the [online] forum, even these colleagues might be taking different modules. (case one: students focus group 4) unfortunately, the number of participants taking the same module in the same school was small (less than five on average). therefore, some problems can be highlighted, such as the poorly functioning communicative tools promoted by the course designers, and too limited a number of participants taking the same module so that these common interests failed to be articulated and developed into a workable joint enterprise. consequently, actions remained un-coordinated and individual learners did not progress beyond being primary agents. in contrast, in case two, despite the more content-focused course design, the negotiation and achievement of the joint enterprise was enabled through intensive networking with the aid of different communicative means both through and outside the course venue, giving rise to coherence of practice in a learning community. extensive activities, including information sharing and exchanging, dealing with assignments and examinations and solving problems, were in evidence in the online forums and group space. geographical proximity also helped group members interact with research in learning technology 13 each other more directly. some of them set up face-to-face meetings and study groups with peers in the same city. going beyond the face-to-face meetings and discussions of the course, the class committee also organised some social events and activities from time to time to gather people together. one of the most enjoyable and positive experiences reported by many learners in this case was meeting their peer students and developing relationships with them. wenger (1998) has argued that what it takes for a community of practice to cohere enough to function can be very subtle and delicate, and this coherence also requires work. in this case, the course provided a platform, although unintentionally, where learners were able to link up with each other, and share their common interests and concerns as a means of mutual support to help them successfully navigate the course. archer (2000) argues that the influence of structural factors is mediated by the human power of reflexivity. this can explain the gap found between the course designers’ intentions and learners’ actual experiences in both cases. similarly, wenger (1998) argues that an enterprise can never be fully determined by an outside mandate, by a prescription, or by any individual participant, but can only be negotiated by the community that conditions and resources practice, and whose demands shape that practice. it is ‘their’ (learners’) learning community to emerge where ‘their’ (learners’) negotiated joint enterprise becomes necessary as a source of community of coherence. on the other hand, individual agents cannot make whatever they want from the situation but are constrained by the ideational and material resources available (archer 2000). wenger (1998) has argued that whatever it takes to make mutual engagement possible is an essential component of any practice. the limited number of students taking the same module made it difficult for learners in case one to find peers to ‘share information’ with. consequently, the process of negotiation and discussion was inhibited, particularly since this limitation was combined with the short duration of the programme, weakly defined learning objectives and assessment methods, unfamiliarity with the pedagogy and communicative tools employed, as well as language barriers. what we observe here, therefore, is an example of agential power constrained by a combination of structural powers. to engage with these constraints and attempt to overcome them would have required a greater commitment to the programme than the students felt was justified when balancing its potential professional value against other demands in their daily lives (cf. archer 2000, 297). the significance of user numbers in increasing engagement as identified in this study is in line with crook’s (2008) observation that the growth in the sheer number of internet users contributes to the popularity of social networking where people share and exchange information and interact with each other. furthermore, learners who gained their membership in the learning community were apparently those who had transformed from being primary agents to corporate agents: “who know what they want, can articulate it to themselves and to others, and have organised in order to obtain it, can engage in concerted action to reshape or retain the structural and/or cultural features in question” (archer 2000, 265). the learners’ experiences in case two have illustrated such agential power in articulating shared interests, organising for collective actions and exercising corporate influence in decision-making, such as their collaborative efforts and success in getting the poorly delivered lectures re-organised. learners in this case have thus been strategically involved in the modelling or re-modelling of the course structure and culture by making themselves heard in the decision-making arena. 14 z. li structural constraints and enablements in the emergence of a learning community also manifested themselves in a more subtle way. research conducted in conventional classroom settings with chinese learners, such as that of lee (1999) and zhou, knnoke, and sakamoto (2005), suggests that acquaintance and personal familiarity with peer students influence students’ interactions with each other. this study shows that this applies equally to an e-learning environment. the development of warm and trusting interpersonal relationships among learners evidently helped the emergence of the learning community to a great extent. as one interviewed teacher expressed it: “i found they actually know each other very well, even the personal life and family issues”. this familiarity with their peer students might have been gained from their informal chatting using the online instant messenger, social activities organised by the class committee, and so forth: ‘sharing meaning’ has become motivation for learners to collaborate, as crook (2002) similarly found in his study. learners share information, from the course assignments to the latest gossip. wenger argues that: what it takes for a community of practice to cohere enough to function can be very subtle and delicate … it is difficult to distinguish between the value of a specific piece of information and the value of the atmosphere of friendliness they create, or between bits of talk about work and the personal exchanges that are woven into their conversations. (1998, 74) the social aspects of interaction are thus crucial to building up trusting relationships, and knowledge of other participants is a central part of these aspects. however, i argue that the preference for communicating with people whom they know should not be seen as a feature exclusive to chinese learners, as some, such as bond (1991), have assumed. research has already found that in a computer-conferencing setting, people might be put off by the impression of a large lurking, anonymous audience, who could read their contributions (grint 1989). similarly, wegerif (1998) found in his study that the social dimension is important to the effectiveness of asynchronous learning networks and needs to be taken into account in the design of courses. to understand this from an archerian analysis of learners’ reflexivity it was also necessary to draw on key concepts within chinese ‘traditional’ philosophies and experiences of learning and social engagement. as discussed previously, the central concept in confucian philosophy, ‘ren’ ( , benevolence), represents balanced and harmonised human relationships, and trust is the critical ingredient of all dependent social interactions in order to achieve this balanced and harmonised human relationship. ‘ren’ ( ) is seen as being manifested in chinese educational tradition and practice in many different aspects: learning is a cooperative process between teacher and student; there is a strong sense of trust and responsibility within the teacher–student relationship; students’ spontaneous collaborations outside the classroom, small group work involving collaboration and cooperation seems to be a ‘natural’ way to structure learning among ethnic chinese learners (li 2009). unfortunately, a warm and close relationship between teacher and students failed to be achieved in both cases in this study. in case one, it was found more difficult to achieve such trusting interpersonal interaction through computer-conferencing communication. one-to-one communications with the tutors were sought by the learners but discouraged by tutors. in case two, the one-way teaching process left learners’ desire to know and interact with their teacher unfulfilled. the online learning environment made it difficult for learners to make any informal contacts with teachers as they might have been used to doing in their previous learning experience. research in learning technology 15 moreover, the findings suggest that developing a trusting human relationship also entails mutual understanding, which was not achieved well between teachers and learners in both cases. the origins of the gap were found to be in the diverse understandings held by the course designers, teachers and learners about learners’ needs, and their preferences with regard to learning strategy and technology usage. in particular, in case one, rather than examining the structural constraints in the course design and other factors in the e-learning settings, learners were harshly judged to “lack autonomy” because “the traditional ways of chinese pedagogic approaches … are typically teacher-driven and teacher-centred” (case one: etutor report i). this stereotyping of the learners and misunderstanding of their learning behaviours inhibited the giving of truly valuable support, thus impeding the building of a trusting relationship between tutors and learners in the e-learning environment. in case two, the limited interaction between teacher and learners also prevented such mutual understanding from developing. by contrast, the most enjoyable experiences reported in both cases were the warm and trusting relationships developed with peer learners, which in case two formed the basis of a learning community. conclusions the forming of a learning community is a socialisation process for a group of individual learners, during which they express their interests and organise the strategic pursuit of their common target. to facilitate such a socialisation process, there should be resources available, both ideational and material, in the e-learning settings. for instance, an e-learning course can be used as a platform where learners can engage with those who have similar interests to themselves. as has been found in classroom settings, spontaneous collaboration and small group work involving collaboration and cooperation seemed to a natural way to structure learning among ethnic chinese learners. at least, it is seen as inappropriate to attribute an unsatisfactory implementation of a collaborative course design as due to a default ‘learners’ predisposition’. when there are teaching innovations that are fresh to learners, the right question to ask might be what kind of extra support should be given to them. moreover, the development of trusting human relationships is a key to the emergence of the learning community because relationships are identified in this study as a driving force in the learning process. technologies expand the availability of such relationships across the traditional boundaries of time and space. at the same time, they set up barriers to be overcome by removing more direct human contacts during the interactions. therefore, bringing moral support into the e-learning environment should be seen as a crucial issue. the findings in this study have also indicated the significance of human relationships in an e-learning environment influencing learners’ learning experiences; in particular, in forming a learning community. the strong sense of responsibility and personal care marked in teacher–student relationships in chinese cultural and educational tradition is difficult to achieve in an e-learning environment where the opportunities for informal social interaction are reduced. any change in the balance of power in relationships between teacher and learner does not come automatically as a result of adopting a new pedagogical approach in e-learning design but depends on trusting human relationships being established. this concept of ‘ren’ in chinese philosophical tradition thus has significant meaning in an e-learning environment. the building up of ‘ren’ ( ) in e-learning settings calls for effective 16 z. li interpersonal interactions, with the aid of communication technologies. ‘social presence’ in communication is perceived as important for shortening the interpersonal distance and achieving mutual understanding. therefore, human contact cannot, and should not, be replaced by contact with or through a machine. moreover, a trusting relationship can only be achieved in the absence of prejudice and stereotyping but with an understanding of the possible different preferences in learning approaches between different cultures and social contexts. references archer, m. 1996. culture and agency: the place of culture in social theory. cambridge: cambridge university press. archer, m. 2000. being human: the problem of human agency. cambridge: cambridge university press. archer, m. 2007. making our way through the world: human reflexivity and social mobility. new york: cambridge university press. bond, m.h. 1991. beyond the chinese face: insights from psychology. hong kong: oxford university press. cole, m., and y. engeström. 1993. a cultural–historical approach to distributed cognition. in distributed cognitions: psychological and educational considerations, ed. g. saloman, 1–46. cambridge: cambridge university press. conole, g. 2007. an international comparison of the relationship between policy and practice in e-learning. in handbook of e-learning research, ed. r. andrews and c. haythornthwaite, 286–310. london: sage. conole, g., j. smith, and s. white. 2007. a critique of the impact of policy and funding. in contemporary perspectives in e-learning research: themes, methods and impacts on practice, ed. g. conole and m. oliver, 38–54. london: routledge. conole, g., m. de laat, t. dillon, and j. darby. 2006. jisc lxp: student experiences of technologies. final report of jisc-funded lxp project. southampton: university of southampton. http://www.jisc.ac.uk/elp_learneroutcomes.html. creanor, l., d. gowan, c. howells, and k. trinder. 2006. lex: the learner experience of elearning – final project report. glasgow: glasgow caledonian university. http:// www.jisc.ac.uk/uploaded_documents/lex%20final%20report_august06.pdf. crook, c. 2002. the campus experience of networked learning. in networked learning: perspectives and issues, ed. c. steeples and c. jones, 293–308. london: springer-verlag. crook, c. 2008. what are web 2.0 technologies, and why do they matter? in education 2.0? designing the web for teaching and learning: a commentary by the technology enhanced learning phase of the teaching and learning research programme (tlrptel), ed. n. selwyn, 6–9. http://www.tlrp.org/pub/commentaries.html. grint, k. 1989. accounting for failure: participation and non-participation in cmc. in mindweave: communication, computers and distance education, ed. r. mason and a. kaye, 182–92. oxford: pergamon press. lave, j., and e. wenger. 1991. situated learning: legitimate peripheral participation. cambridge: cambridge university press. lee, c. 1999. learning through tutorial discussion and learners’ culture: some preliminary observations from the views of hong kong chinese tertiary students. language, culture and curriculum 12, no. 3: 255–64. li, z. 2009. making our way through the e-world: chinese adult e-learners’ learning experiences. phd diss., university of southampton, uk. timms, s., r. o’leary, e. weedon, and k. martin. 2004. a multi-disciplinary, holistic approach to networked learning research: a critique of a large-scale empirical study into student online learning experiences. paper presented at networked learning conference, april 5–7, in lancaster university, uk. wegerif, r. 1998. the social dimension of asynchronous learning networks. journal of asynchronous learning networks 2, no. 1. http://www.aln.org/alnweb/journal/vol2_issue1/ wegerif.pdf. research in learning technology 17 wenger, e. 1998. communities of practice: learning, meaning and identity. cambridge: cambridge university press. zhou, y.r., d. knoke, and i. sakamoto. 2005. rethinking silence in the classroom: chinese students’ experiences of sharing indigenous knowledge. international journal of inclusive education 9, no. 3: 287–311. doi:10.3402/rlt.v24.30630 research article perceptions of online tutorials for distance learning in mathematics and computing tim lowe*, ben mestel and gareth williams department of mathematics and statistics, the open university, milton keynes, uk (received 4 december 2015; final version received 29 june 2016) we report on student and staff perceptions of synchronous online teaching and learning sessions in mathematics and computing. the study is based on two surveys of students and tutors conducted 5 years apart, and focusses on the educational experience as well as societal and accessibility dimensions. key conclusions are that both staff and students value online sessions, to supplement face-to-face sessions, mainly for their convenience, but interaction within the sessions is limited. students find the recording of sessions particularly helpful in their studies. keywords: elearning; etutorials; synchronous online learning; stem introduction mathematics and computing educators have long understood the importance of active student participation to facilitate deep learning of core ideas and techniques. for part-time distance-learning students studying with the uk’s open university, the traditional teaching medium has been highly structured, professionally produced notes and support from an individual tutor, specifically feedback on tutor-marked assignments and face-to-face tutorials. the latter have provided an important means of student support, but some students find it hard to attend because of work and social commitments. students with additional requirements, those residing far from tutorial venues, overseas students, and those with family commitments have found it particularly challenging to attend face-to-face events. these problems have naturally led to the growth of online teaching, which provides its own challenges for symbolically-rich disciplines. over the last 25 years, there has been significant research in distance learning and, since the internet era, in elearning, both as a component of distance education and as a significant research field in its own right (moore 2013; zawacki-richter and anderson 2014). according to moore’s model of distance education, transactional distance (a function of course structure, dialogue and learner autonomy) is a key factor in students’ learning (moore 2013, chapter 5). consequently, effective distance-learning course design should reduce or mitigate the effects of transactional distance. as reported by mcbrien, jones, and cheng (2009) in their study at the *corresponding author. email: tim.lowe@open.ac.uk responsible editor: carlo perrotta, school of education, university of leeds, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 t. lowe et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 university of south florida, virtual classrooms have been found to have a positive effect on each of the three key components of transactional distance. other studies of synchronous learning tools have tested the community of enquiry framework (garrison, anderson, and archer 1999). for example, ling (2007) studied organisational informatics tutorials conducted in online chat-rooms at murdoch university. she found evidence of all three elements of the model, which are (evans and haughey 2014): ‘teaching presence (instructional activities required to facilitate learning), social presence (activities that support discussion and dialogue for learning), and cognitive presence (the learning resulting from the interactions in the community)’. other research has focussed less on theoretical models underpinning distance education. kear et al. (2012) report on online conferencing for the open university module networked living: exploring information and communication technologies. their work follows on from earlier studies by chetwynd et al. (2009), lissaman, de pomerai, and tripconey (2009), loch and mcdonald (2007), mestel et al. (2011), ng (2007), and reushle and loch (2008). in the context of mathematics, computing and other stem (science, technology, engineering and mathematics) disciplines, building a community of enquiry and/or reducing or mitigating the effects of transactional distance are complicated by disciplinespecific requirements: interactive presentation of visual images, symbolically-rich notation, complex calculations, either by hand or through software packages, and even scientific experiments. many stem educators have enthusiastically embraced the opportunities provided by the internet, despite their clear challenges. since 2009 the open university has used an education-oriented synchronous communication tool, sometimes known as a virtual classroom. consequently, a significant amount of experience and knowledge on the use of this medium has accumulated, from both the student and tutor perspectives. this study aims to capitalize on this wealth of experience to investigate whether familiarity with such systems has changed attitudes towards their use by considering the following research questions: (1) have tutor and student attitudes to the use of virtual classrooms changed over the 5 years since their introduction, and if so, how? (2) has the use of virtual classrooms reduced or mitigated the transactional distance in distance education and helped build a community of enquiry, especially for disabled students? (3) what are student/tutor attitudes to the recording of online sessions? background and methodology the open university (ou) is the uk’s premier distance-learning university, with around 170,000 mainly part-time students, studying courses ranging from preparatory access modules to masters and beyond. it has a world-wide reputation based on its blended open learning methodology, combining quality learning material (both printed and online), correspondence tuition, and online and face-to-face support provided by a network of approximately 6,000 tutors. since its early days it has continually updated its teaching and learning to incorporate technological developments, including full use of the internet to deliver both asynchronous and synchronous learning opportunities. t. lowe et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 in 2008�2009, as part of a project to investigate the possibilities offered by the internet to enhance the distance learning of mathematics, the authors conducted a trial of synchronous online tutorials using the elluminate live! software system (mestel et al. 2011). the trial covered principally mathematics, but also some computer science, technology and science modules. subsequently, the ou adopted elluminate live! as its platform for synchronous online teaching. when the software was merged into blackboard collaborate, the ou adopted the revised product. the current study combines the 2008 survey with a further survey conducted in 2014, and focusses on student and tutor perceptions of synchronous online learning events within stem higher education, principally mathematics, and how these have changed during the first 5 years of use. although the specific names of the software systems used were mentioned in the surveys (since they were familiar to tutors and students), here we refer to the sessions conducted in the virtual classrooms provided by these systems as ‘esessions’. there are technical and aesthetic differences between the different versions of the software used, and between other products providing similar functionality. the main technical improvement in the software used over the period of this study was related to the transmission of audio, and in particular reductions to both delay and echo. the aesthetic differences between elluminate live! and blackboard collaborate are illustrated in figure 1 and mainly relate to styling and icons. for this study, however, it is the underlying functionality of the products to facilitate online teaching and learning which is the focus, and this is not specific to the particular software system. the 2008 trial and survey comprised 29 tutors conducting esessions. after each event, all the participants (students and tutors) were asked to rate the esession both from a technical standpoint, and as a learning experience. at the end of the trial, each participant was asked to complete a survey that addressed wider questions including societal and accessibility dimensions. the trial largely focussed on perceptions of technical features of the environment used and student and staff perspectives of these aspects were reported by mestel et al. (2011). all tutors in the trial were self-selecting and were paid a fee for their participation. only the students of the participating tutors were invited to participate in the 2008 trial. in 2014 most tutors who participated in the 2008 trial had continued to hold esessions throughout the 5 year period, and were willing to share their perspectives: 21 of the original 29 participated on a voluntary, unpaid basis. since students are assigned to a tutor for only a single module, and students have many varied study pathways, it was not possible to poll the same students in 2014 as in 2008. instead figure 1. the elluminate live! (left) and blackboard collaborate (right) user-interfaces. research in learning technology citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 student participation was solicited via an open invitation in module online forums. consequently care must be taken when comparing the results in the two studies. in both surveys, participants were asked to rate aspects of the system and their learning experience on a five-point likert scale (with 1 the most negative, 3 neutral, 5 the most positive) and given the opportunity to provide free-form comments. the 2008 survey included questions regarding participants’ internet connection and provider, and specific questions on audio quality (which was a particular concern at the time) which were not included in the 2014 survey. the surveys did not consider demographic information on students, their previous background or study intentions. headline results in 2014, 16 responding students reported non-participation in esessions, due to a lack of: awareness, interest, internet access, and confidence in the software, as well as the limited number of esessions offered and a preference for face-to-face tutorials. two students indicated that they had accessed recordings but had not attended a live esession. anecdotal evidence suggests that there is a significant group of students in this category. in total 106 student participants provided feedback on their experiences (three students provided no feedback). over 80 modules were covered; these were principally mathematics, mathematics education, computing, science and technology, but also included a small number of business and languages modules. of the different types of esession attended by students, tutorial sessions (etutorials) formed the bulk, with electures also featuring strongly. interestingly, students did not report using virtual classrooms for self-help groups and few have had one-to-one individual support (esupport) sessions with a tutor. figure 2 shows the responses, from both surveys, for the perceived effectiveness of several features of the virtual classroom in supporting learning. note that in the 2008 survey, students and tutors rated the features after each individual esession, leading to higher overall response numbers. the figure shows a diverging stacked bar chart [following robbins and heiberger (2011)] with neutral responses centred on zero, positive responses on the right, and negative responses on the left. it shows widespread agreement between staff and students in 2008 and 2014. interestingly, the student perception of text-chat to help learning decreased slightly between the surveys, despite anecdotal reports that it is widely used. perhaps multiple simultaneous conversations on many topics, not all entirely related to the teaching aim of the session, contribute to this. staff perceptions of the effectiveness of application sharing increases slightly over the study period, perhaps as they became more familiar with the possibilities it offers, whilst students’ perceptions decreased. both staff and student views on the effectiveness of breakout rooms and webcams decreased over the period of study. these conclusions must, however, be viewed in the light of the smaller number of responses in 2014. comparison with face-to-face tuition to understand student and tutor views on esessions compared to traditional face-toface teaching, there were several questions in the survey to test opinion on the ‘learning experience’ (figure 3). although this term was not explicitly defined in the surveys, we take it to be a holistic view of the effectiveness of the esessions for t. lowe et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 teaching and learning, be that transfer of knowledge or academic discourse. it is apparent that there is widespread consistency between staff and students, and across the 2008 and 2014 surveys. esessions are highly rated for the overall learning experience, although there is a slight decrease in the rating given by students between the two surveys. this may be due to heightened student expectations as a result of figure 2. responses on a five-point likert scale for perceived effectiveness of various aspects of the virtual classroom. items marked * were not used in all sessions, and data were only collected for sessions in which they were used. research in learning technology citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 changes to higher education in the uk between 2008 and 2014. there is a very small overall view that esessions are as effective for learning as face-to-face teaching, but esessions rate poorly for interaction compared to face-to-face sessions from both students and staff. within the virtual classroom, interaction could take place between students and tutor or amongst students using text or audio communication, on the shared whiteboard or within a shared application. once again, student rating on this have slightly decreased over the study period. this relative low score is somewhat concerning, especially from the perspectives of the community of enquiry and transactional distance frameworks. despite this, overall views on interaction of a social nature within esessions were neutral. interaction in general also featured as a strong theme in open comments by students and tutors, with both positive and negative experiences. negative comments included: it is rather passive (little interaction). lack of/quality of interaction with other students (there is some but not a huge amount). almost impossible to interact and ask effective questions. the tutor is unable to read the mood of the group and judge pace and content. with no face-to-face tutorials in[ter]action with fellow students massively reduced. figure 3. evaluation of esessions over several criteria, including comparison with face-toface, interaction and overall learning experience. t. lowe et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 one student commented: these sessions are no substitute at all for face-to-face teaching. interaction is stilted and slow, particular[ly] with many participants. in general, ‘tutorials’ tend to become lectures. of course, this can happen in face-to-face settings as well, but the nature of the medium makes it harder to avoid with online presentation. on the other hand, there were significant positive comments on interaction amongst students: ‘i like the real time interaction with my classmates . . . and tutor’. some thought esessions compared favourably with other modes of communication: ‘better way to meet/chat to other students and tutor than in [asynchronous] forums’, and liked ‘interaction and asking questions without having to take time to try and annotate it in an email to our tutor’. another commented: ‘i like being put on the spot with poll questions and getting that instant feedback’, and noted that ‘my course . . . is very solitary (which mostly suits me) but i do like the little bit of live interaction that i get in these sessions with my fellow students primarily in the im [instant messenger] section’. it is not surprising that those students who do not have access to face-to-face tutorials (as in the msc in mathematics programme) welcome the interaction that esessions provide, while those with face-to-face opportunities were more circumspect. staff comments were similarly mixed. ‘i enjoy taking online tutorials and have much bigger numbers attending than at face-to-face tutorials. but i don’t think that online tutorials will ever replace the magic of a good face-to-face tutorial where interaction, group work and individual contact with students are all easier’. another remarked: one of the most difficult aspects for me has been encouraging students to interact with the session � this is much easier in a face-to-face session. this extends to the fact that as a tutor, you have no visual clues to how students are reacting to the content being covered, and other techniques need to be used to try and gauge this, i.e. polling. some students commented on the specific difficulties in using the system with mathematics and scientific subjects, in particular the difficulty in writing equations, the fact it is hard to write neatly on the whiteboard without specialist equipment, and the restrictions imposed by the size of the whiteboard. similar difficulties are not unknown in face-to-face teaching. societal and accessibility dimensions convenience and optimal times figure 4 shows the staff and student responses for questions relating to convenience and efficiency of esessions. there is a slight decrease in convenience over time reported by both staff and students. in regard to efficiency, there is a slight decrease in the ratings given by students between the two surveys, and a slight increase in the tutors’ ratings. staff and student comments highlighted the convenience of the medium. as one student put it: ‘no need to travel/able to get to tutorials that i could not travel to; saving time and money’. for some, esessions are the only option: ‘since i don’t live in the uk, [an esession] is the only way to attend a tutorial. so i guess i will make do research in learning technology citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 with the technical shortcomings and will of course attend all tutorials on offer’. and ‘i just wouldn’t be able to get to any in person’. convenience was also emphasised by tutors: ‘the convenience of [esessions] is a real bonus, particularly when the weather is bad and may impede travelling to the tutorial venue (for both students and tutors)’ and one student reported: ‘for the present course, the . . . etutorial was far more convenient than having to travel a 70 mile return journey for a face-to-face tutorial’, but, for one student, convenience was the only advantage: ‘i don’t have to leave the house. that’s about it. otherwise i much prefer physical face-to-face classes’. however, it seems that convenience does not necessarily lead to big attendance. as one tutor remarked: ‘in the early days of using [esessions], i expected a larger number of students to attend etutorials, due to the convenience, but over the years, whilst it may be the case that a small number of additional students attend, it generally tends to be the same students who would normally attend a face-to-face tutorial. the numbers of students attending [online] tutorials has not been significantly larger, in my experience’. accessibility respondents self-identifying as disabled were asked to comment on their experiences of esessions and on any accessibility issues. in 2008, 9 of the 105 students (8.6%) responded, as did 2 of the 29 tutors (6.9%). in 2014 these figures were 6 out of 106 students (5.7%) and 2 out of 21 tutors (9.5%). in 2014, the two disabled tutors raised no disability issues. although the small number of respondents demands caution and any conclusions must necessarily be tentative, it is interesting that the comments divide into two groups: a. visual/aural/cognitive disabilities (including dyslexia) b. physical/psychological disabilities. group a respondents experienced serious difficulties with aspects of the virtual classroom. a dyslexic student found the ‘size of text on white[board] was sometimes too small’ and ‘struggled to cope with all the different areas of the screen that were figure 4. responses for convenience of esessions for learning (students) and teaching (staff) and whether esessions were an efficient use of time. t. lowe et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 operating at the same time’. a partially sighted student likewise ‘had difficulty following what was going on, mainly because there were so many different activities simultaneously, and [the] screen reader can only focus on one at a time’. a second dyslexic student was not keen on esessions, commenting: ‘i get a lot more out of personal contact, and explaining things over the phone is just as bad’. another commented: ‘i have a hearing impairment and need to lipread � this was not possible with [esessions] so i feel disadvantaged’, a view not limited to those with auditory impairments: ‘being visually handicapped, i had difficulty keeping track of what was going on’. however, one student commented ‘i am hearing impaired but it doesn’t cause too many problems as i can control the audio on my computer. in addition if i miss something i can listen to the recording afterwards. the tutorials i have attended do not have too much audio interaction with students which does help me. i have also found that the tutors speak clearly which is a huge help’. however, group b respondents were much more positive on their experience. one student described esessions as ‘one the highlights of the course’. several further quotations suffice to illustrate student viewpoints: i suffer from severe chronic clinical depression and have never attended a tutorial because of problems related to confidence, etc. the online tutorial was excellent for me, and i thoroughly recommend its use for those consistently unable to attend tutorials. yes, i am disabled and without the [esessions] i would not have had any tutorials. it saved my travelling and having to face people if i am too ill. i could ‘leave the room’ if i wanted and to [be] able to playback at my own time was brilliant. as a wheelchair user not having to get into a crappy inaccessible building was a bonus. it is notable that for some disabled students the disadvantages are at least ameliorated by the possibility to playback a recording of an esession. recordings and privacy the virtual classroom has a recording facility, providing an almost complete record of the esession for future playback. many tutors take advantage of this facility to provide a resource for students, although this varies between tutors and module. the recording is generally made available to the tutor’s own students, including those who had not attended the live esession. if required, a recording can also be made available more widely. in the 2008 trial, 24 out of 29 tutors (82.8%) recorded at least one esession and 66 out of 105 students (62.9%) attended an esession which was recorded. however, only 33 (31.4%) actually accessed a recording. in 2014, 78 out of 106 students (73.6%) had accessed a recording, including sessions they did not attend; a significant increase. of the remainder 12 (11.3%) answered ‘not applicable’, most likely because their tutors had not recorded sessions, with 16 (15.1%) not having accessed a recording. in addition, there were several students who were not captured in detail by the survey who indicated that they had accessed recordings without having ever attended a live esession. students are typically warned in advance that the esession might be recorded and the assent of all students attending is required for the session to be recorded. these precautions are necessary because the recordings are not easy to edit and the whole session ‘warts and all’ is available to students for replay, although not for permanent download without the use of screen capture software. any mistakes or slips by tutor research in learning technology citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 or students are faithfully preserved, potentially causing embarrassment to those affected. indeed, there is a concern that the act of recording itself might have a detrimental effect on the esession, with students unwilling to participate and tutors sticking too carefully to prepared scripts. table 1 summarises the responses on whether recordings affected the model of participation. three students reported that they would be inhibited from speaking or asking questions. however, one student said they were ‘more relaxed’ because they ‘did not bother taking notes, but instead concentrated on what the tutor said’. a second reported: ‘if you lose track of some complicated reasoning, but felt that all necessary explanation was covered by the tutor, you would not feel the need to ask to go over it again if it was recorded’. some students noted that they were ‘more selfconscious, less likely to answer if unsure of [the] answer’ and ‘holding back a little on writing comments/questions’. one student commented: ‘the group and i appeared slower to offer opinions and answers, obviously conscious of the recording, but that said, i’d still vote for it due to the value of accessing the recordings of the sessions which i was unable to attend’. another wrote: ‘it can be quite difficult to get interaction going in [esessions] and this may be more so if students feel that a wrong answer will be recorded for posterity!’ tutor opinion was similar. two noted that they had been affected initially but the effect had been transient, although one of them added: ‘my concern is that if tutors are routinely recorded they will be less inclined to experiment and stick to things that will go smoothly’. a third commented: ‘it made me watch what i said a lot more! and made me think about what i was saying so that those listening later would have a full understanding of what was happening. recording certainly made me try to do a more professional job, and enhanced the actual live tutorial too, therefore’. other tutors’ comments included: ‘it makes me less informal’, ‘it adds to the pressure of the session’, ‘i find recording inhibiting and so do the students i have asked’ and ‘students are affected by the fact that they are being recorded and are distracted from learning’. recordings have a potentially unlimited lifetime and it was useful to gauge student and tutor opinion on the appropriate length of time a recording should be kept. we also probed opinion on the types of use to which a recording might be put. figure 5 shows opinions on the retention period of a recorded esession. staff views on this issue were broadly the same in 2008 and 2014, with a small group that has no objection to recordings being kept indefinitely. the majority opted for a retention period of 2 years or less. student opinion differed somewhat between 2008 and 2014: 81.9% of students opted for 2 years or less in 2008, while this was reduced to 64.2% in 2014, with a significant group (30.5%) favouring indefinite retention. this change most likely reflects greater student awareness in 2014 of the utility of recordings. table 1. the effect of recording on mode of participation. did the fact that the session was recorded affect the way in which you participated in the [esession]? yes no total student 2008 6 (7%) 78 (93%) 84 (100%) staff 2008 3 (12%) 21 (88%) 24 (100%) student 2014 11 (20%) 43 (80%) 54 (100%) staff 2014 4 (22%) 14 (78%) 18 (100%) t. lowe et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 the quality of the recordings was a concern. one respondent commented: ‘this is a live almost amateur presentation and would not compare well with broadcast quality media with professional recording editing and effects’. there is considerable concern that increased use of technology (especially the internet) has had a negative impact on the privacy of individuals, and, with this in mind, both staff and students were asked to comment on any issues that concerned them with regard to the recording of esessions. in 2008, several students and tutors raised privacy and data protection as concerns, with one suggesting: ‘anonymity of students participating should be confirmed’, and another suggesting that it would be difficult to ‘manage issues where a participant makes a statement they later regret’. one student strongly disagreed with recording a session in which a webcam was used. a tutor suggested only first names should be displayed by the system; currently the full name of each participant is visible to all. in 2014, attitudes were somewhat more relaxed. one student commented: ‘i don’t know what the fuss is about recording’ although another suggested ‘names should be hidden to give privacy’. it is hard to reconcile all of these concerns raised by respondents. indeed, as one tutor commented: ‘i like students to participate actively in sessions as far as possible. i want them to have a go at answering questions even if they get them wrong. i think that they are less likely to do this if the session is recorded, so it will become more of a lecture from me. i am also concerned that fewer students would attend the ‘‘live’’ sessions. on the other hand, i am aware that students cannot always attend tutorials at a particular time and would benefit from a recording. if i was given the choice, i might opt for a halfway house with some of the tutorial being recorded and some not, but this would involve more organisation’. discussion and conclusions this study reports on student and staff perceptions of synchronous online learning esessions, focussing on surveys conducted in 2008 and 2014. in this section, we return to the three research questions posed in the introduction. figure 5. student and staff opinion on how long an esession recording should be retained. research in learning technology citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 1) have tutor and student attitudes to the use of virtual classrooms changed over the 5 years since their introduction, and if so, how? in both surveys, students generally liked the convenience of online tutorials, and especially appreciated them when no alternatives were possible, although face-to-face was considered better if available. the ability to interact with other participants was praised, but many also felt the online sessions could be made more engaging and more interaction could be encouraged. this view was more marked in the 2014 survey. student experience varied greatly with tutor and/or module. there was some feeling tutors might be better trained in the use of the medium. over the period of study, students perceptions of the usefulness of text-chat has decreased slightly, as has their view of application sharing, in contrast to the staff whose perception of application sharing has slightly improved over time. the perceptions of the effectiveness of breakout rooms and the use of webcams decreased in both the staff and student groups over the period of study. we note that students’ ability to use virtual rooms and expectations of the software may have been influenced the increased use of systems such as skype and facetime in the general population. 2) has the use of virtual classrooms reduced or mitigated the transactional distance in distance education and helped build a community of enquiry, especially for disabled students? transactional distance is a function of three aspects: dialogue, structure and learner autonomy (moore 2013), while, according to evans and haughey (2014), the three elements needed to build a community of enquiry are: teaching, and social and cognitive presences. in the studies presented here, the lack of interaction reported by students and tutors suggests a reduction in dialogue, with the result that esession often becomes more of a one-way lecture, hence increasing the structure of the esession. both of these factors adversely affect the transactional distance. whilst this overarching view appears to contradict the results of mcbrien, jones, and cheng (2009), many of the individual student comments they report reflect those obtained in the current study. as reported above, students did appreciate the esessions and transactional distance is certainly mitigated by virtual classrooms when there is no face-to-face alternative. these seemingly contradictory results may be a consequence of the domain of study. here, the subject material (mathematics and computing) often requires the use of technical notation that is currently difficult to communicate within the virtual classroom, where as mcbrien et al. considered students of the social sciences. wang, mattick, and dunne (2010) also reported interaction difficulties with video-linked and streamed medical lectures. for the community of enquiry model, the teaching element is certainly present within the scheduled esessions although the limited social interaction (figure 3) and the unidirectional nature of esessions referred to above present problems for social and cognitive presence. it appears that virtual classrooms may help in building a community of enquiry, but are not sufficient by themselves. for some disabled students, this difficulty of communication is considerably less than the difficulties encountered when attempting to attend a face-to-face session, and thus transactional distance is reduced for these students. for others, principally those t. lowe et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 with auditory and visual disabilities, the difficulties presented by the synchronous online teaching software increases considerably the transactional distance and decreases significantly social and cognitive presence, at least over face-to-face events. of course, without the option of a face-to-face alternative, the provision of online esessions can be an asset for all students, provided no disabled student feels disadvantaged and/or marginalised by the difficulties associated with synchronous online teaching. 3) what are student/tutor attitudes to the recording of online sessions? students expressed strong support for the recording of esessions, finding the ability to review an esession at their convenience particularly valuable. in general, other issues associated with recordings provoked mixed views. students and staff alike did express privacy concerns, although this was not a major worry. there were requests that recordings be well catalogued and that the interface for viewing them should be sufficiently sophisticated for the purpose. there was some concern that the nature of an esession was changed by the act of recording, but others thought the additional care taken by both the tutor and students in a recorded session improved the standard. we conclude by reporting on 2014 staff and students’ reported intentions regarding future use of esessions. of 21 staff, 17 would be highly likely to give an esession in future and 80 of 100 students would similarly be highly likely to attend an esession if offered in the future. perhaps more surprisingly, the availability of esessions in addition to other tutorial provision may incentivise students to register for study. these facts in themselves suggest that online tutorials will continue to be a component part of distance learning in the future. as one enthusiastic 2014 student put it: ‘i will definitely be requesting [esessions] for my next module and really hope that i get them!’ recommendations a number of recommendations regarding virtual rooms and their use for distance learning are apparent from this study. (1) distance-learning courses should include the use of esessions to support students; however, they should be used as a supplement to, not as a replacement for, face-to-face sessions. (2) developers of virtual classrooms should include better support for the communication of mathematics and other symbolically-rich disciplines. (3) developers of virtual classrooms should better consider the needs of disabled users in future designs of their software. (4) future studies of this nature should consider the demographics and the backgrounds of students surveyed, including prior experiences of synchronous internet communication tools. (5) where appropriate to the content, esessions should be recorded, subject to obtaining the permission of all participants. in general such recordings should be kept for a maximum of 2 years. participants should be given advanced notice of the possible uses to which the recordings might be put. research in learning technology citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 (6) for large student groups, it may be more effective to record shorter esessions without students present rather than to record a live tutorial. (7) tutors leading esessions should share best practice in the use of virtual classrooms. acknowledgements the authors thank gaynor arrowsmith (for her help with the 2008 study), and pat bailey, frances chetwynd, graham eaton, sarah chyriwsky, noel eastham, christine leach, mike hay, andrew mascord, peter mitchell, liz ross, richard walker and other ou tutors and students for their contributions. references chetwynd, f., et al. (2009) ‘the use of synchronous online tutorials to provide numeracy support for technology students’, cetl-msor conference 2009, milton keynes, uk. evans, t. d. & haughey, m. (2014) ‘online distance education models and research implications’, in online distance education: towards a research agenda, eds. o. zawackirichter & t. anderson, athabasca university press, edmonton, pp. 131�149. garrison, d. r., anderson, t. & archer, w. (1999) ‘critical inquiry in a text-based environment: computer conferencing in higher education’, the internet and higher education, vol. 2, no. 2�3, pp. 87�105. kear, k., et al., (2012) ‘web conferencing for synchronous online tutorials: perspectives of tutors using a new medium’, computers & education, vol. 58, no. 3, pp. 953�963. ling, l. h. (2007) ‘community of inquiry in an online undergraduate information technology course’, journal of information technology education, vol. 6, pp. 153�168. lissaman, r., de pomerai, s. & tripconey, s. (2009) ‘using live, online tutoring to inspire post 16 students to engage with higher level mathematics’, teaching mathematics applications, vol. 28, no. 4, pp. 216�221. loch, b. i. & mcdonald, c. (2007) ‘synchronous chat and electronic ink for distance support in mathematics’, innovate: journal of online education, vol. 3, no. 3, article 6. mcbrien, j. l., jones, p. & cheng, r. (2009) ‘virtual spaces: employing a synchronous online classroom to facilitate student engagement in online learning’, the international review of research in open and distance learning, vol.10, no. 3, pp. 78�94. mestel, b., et al., (2011) ‘teaching mathematics with online tutorials’, msor connections, vol. 11, no. 1, pp. 13�17. moore, m. g., (ed.) (2013) handbook of distance education, 3rd edn, routledge, new york. ng, k. c. (2007) ‘replacing face-to-face tutorials by synchronous online technologies: challenges and pedagogical implications’, the international review of research in open and distance learning, vol. 8, no. 1, pp. 67�81. reushle, s. & loch, b. i. (2008) ‘conducting a trial of web conferencing software: why, how, and perceptions from the coalface’, turkish online journal of distance education, vol. 9, no. 3, pp. 19�28. robbins, n. b. & heiberger, r. (2011) ‘plotting likert and other rating scales’, proceedings of the survey research methods section, american statistical association, [online] available at: http://www.amstat.org/sections/srms/proceedings/y2011/files/300784_64164.pdf wang, r., mattick, k. & dunne, k. (2010) ‘medical students’ perceptions of videolinked lectures and video-streaming’, alt-j, research in learning technology, vol. 18, no. 1, pp. 19�27. zawacki-richter, o. & anderson, t., (eds.) (2014) online distance education: towards a research agenda, athabasca university press, edmonton. t. lowe et al. 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30630 http://dx.doi.org/10.3402/rlt.v24.30630 http://www.amstat.org/sections/srms/proceedings/y2011/files/300784_64164.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30630 http://dx.doi.org/10.3402/rlt.v24.30630 doi:10.3402/rlt.v24.29927 original research article participant association and emergent curriculum in a mooc: can the community be the curriculum? frances bell a *, jenny mackness b and mariana funes c a itinerant scholar, macclesfield, uk; b independent education consultant and researcher, cumbria, uk; c chartered research psychologist horsham, sussex (received 1 october 2015; final version received 3 june 2016) we investigated how participants associated with each other and developed community in a massive open online course (mooc) about rhizomatic learning (rhizo14). we compared learner experiences in two social networking sites (snss), facebook and twitter. our combination of thematic analysis of qualitative survey data with analysis of participant observation, activity data, archives and visualisation of sns data enabled us to reach a deeper understanding of participant perspectives and explore sns use. community was present in the course title and understood differently by participants. in the absence of explanation or discussion about community early in the mooc, a controversy between participants about course expectations emerged that created oppositional discourse. fall off in activity in moocs is common and was evident in rhizo14. as the course progressed, fewer participants were active in facebook and some participants reported feelings of exclusion. despite this, activity in facebook increased overall. the top 10 most active participants were responsible for 47% of total activity. in the rhizo14 mooc, both community and curriculum were expected to emerge within the course. we suggest that there are tensions and even contradictions between ‘community is the curriculum’ and deleuze and guattari’s principles of the rhizome, mainly focussed on an absence of heterogeneity. these tensions may be exacerbated by snss that use algorithmic streams. we propose the use of networking approaches that enable negotiation and exchange to encourage heterogeneity rather than emergent definition of community. keywords: rhizomatic learning; rhizo14; mooc; social network analysis; facebook introduction in this paper, we explore the concept ‘community is the curriculum’ in a 2014 massive open online course (mooc) titled ‘rhizomatic learning: the community is the curriculum’ (rhizo14). recent years have seen a growing interest in how moocs promote the development of online communities. bates (2015) suggests that ‘moocs are an ideal way to bring together specialists scattered around the world to focus on a common interest or domain’, but that they ‘do not always incorporate what research indicates are best practices for developing communities of practice’. higher education institutions aspire *corresponding author. email: frabell@gmail.com responsible editor: carlo perrotta, university of leeds, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 f. bell et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 to promote social learning within and beyond the formal institution, and support is needed to understand how this can happen (de laat and prinsen 2014). social learning is also researched outside of education, for example, connected learning that addresses the gap between in-school and out-of-school learning (ito et al. 2014); yet whether or not community is congruent with social learning in moocs is still an open research question. in relation to curriculum, watters’ experience of a mooc caused her to question the academic rigour of mooc curricula, worrying that they might become increasingly academically ‘lite’ and ‘intellectually bland’ (watters 2015). rhizo14 focussed on both community and curriculum development. the questions posed at the beginning of the course were: what happens when we approach a learning experience and we don’t know what we are going to learn? where each student can learn something a little bit different � together? (cormier 2014a). these questions reflect how cormier, the course convener, interpreted deleuze and guattari’s six principles for rhizomatic thinking (figure 1) in the rhizo14 mooc (deleuze and guattari 1987). figure 1. deleuze and guattari’s six principles for rhizomatic thinking (mackness, bell and funes 2015). f. bell et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 the authors were participants in rhizo14, and found the course to be different in its pedagogical approach to their prior mooc experience, thus stimulating their interest in researching the learner experience in this mooc, particularly in relation to the formation of community and curriculum development. this is the third paper resulting from this research. each paper addresses a significant aspect of the course. in our first paper, we explored positive and negative experiences of learning in rhizo14 and the possible reasons for these (mackness and bell 2015). in our second paper (mackness, bell, and funes 2015), we discussed the implications of using the concept of the rhizome as a metaphor for designing an open online teaching and learning space. in this final paper, we focus on how a course designed on rhizomatic principles (see figure 1) affects the formation of community and curriculum development. the three papers contribute to the need for more research on learner experience, social interaction and community building in moocs and other potentially innovative learning ventures, as identified by veletsianos (2013a). here the question we ask is � can the community be the curriculum? we investigate how the community formed in rhizo14, what curriculum developed and the relationship between the concept of ‘community is the curriculum’ and rhizo14 participant learner experiences in two social networking sites (snss). rhizo14 � the mooc rhizo14 attracted an estimated 500� participants. it was scheduled to run for 6 weeks (the official course in what follows). it ran for a further 6 weeks (the unofficial course in what follows) organised by a small group of enthusiastic participants (mackness and bell 2015). the mooc extended not only in time, but also ‘in place’ beyond its original p2pu platform (cormier 2014a) to multiple snss and participant blogs. although it was not billed as a continuing professional development course, many active participants were (or had been) employed in some professional capacity in education. this was not a mooc with detailed pre-planned content but one where the learning outcomes were not prescribed. those wishing to know what the course was about were reliant on the title, the course convener’s identity, blog posts and the p2pu site to help gauge their interest. in a post announcing rhizo14 (cormier 2013), learners were encouraged to orient, declare, network, cluster and focus: � orient � find and organise materials, links, and sessions for each week. � declare � declare yourself and your place for reflection � this could be your own blog, a forum, a hashtag. � network � follow and connect to other people and to their work. � cluster � a few weeks in you will find a cluster of people with similar interest, a community. � focus � work on a project for you to use what you have learned (cormier 2010). the only formal content for the official course consisted of a weekly provocative statement for discussion and a video produced by the convener, posted in the p2pu space to introduce each week as follows: 14-jan-14 cheating as learning 21-jan-14 enforcing independence research in learning technology citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 28-jan-14 embracing uncertainty 04-feb-14 is books making us stupid? 11-feb-14 community as curriculum 18-feb-14 planned obsolescence there was no assessment or formalised teaching support for participants. the course was based on the idea of decentring the role of the teacher and encouraging a ‘student as producer’ ethos (neary and winn 2009). as was stated earlier, the course continued beyond the first 6 weeks for a second period we have called here ‘the unofficial course’ because it was organised by a subset of participants and not the course convener. although these were nominally ‘weeks’, they varied slightly in length. these six extra weeks were titled as follows: 25-feb-14 the lunatics are taking over the asylum 04-mar-14 demobbing soldiers 11-mar-14 why do we need lurkers? 22-mar-14 creativity: the art of thriving in arid environments 31-mar-14 powerful thoughts 10-apr-14 mooc missionaries literature review in this review, we explore the concepts curriculum, community and ‘community is the curriculum’ in different contexts in order to inform our understanding of the data collected from rhizo14 participants. curriculum curriculum theory is a substantive body of knowledge (e.g. pinar et al. 1995). here we offer only a few highlights to help situate our research. curriculum is a term usually associated with formal education and is variously expressed as, � a written syllabus: a body of knowledge to be transmitted, � the characteristics of the product: for example, learning outcomes or objectives, or � a process that can help to plan a formal educational activity (smith 2000). curriculum can be seen as praxis, being embodied in values, thinking abilities and intended actions. smith (2008) sees the centre of praxis as informed, committed action, a practice that does not rely on individuals but rather on collective understandings. curriculum can be developed within an institution or negotiated with students on a course (williams, karousou, and mackness 2011, p. 51). for example, a negotiated curriculum is key to self-managed learning (cunningham 1999) where the creation of a learning contract by the student determines the curriculum to be covered in the taught elements of a course of study. self-directed learning, of which self-managed learning is a subset, can be said to have ‘existed even from classical antiquity’ (hiemstra 1994). it is within this history that we situate modern day moocs and curriculum setting. in moocs such as rhizo14, the curriculum can be co-created by tutors and f. bell et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 participants as an on-going process through the course, leading to emergent learning outcomes. the curriculum does not exist a priori and may not even be known by either the tutors or participants. there is also the ‘hidden curriculum’, the unintended curriculum, the unseen influences on learning, such as norms, values and beliefs, as well as ‘out of sight’ activities and discussions. for example, the discourse of participation can be used to manage out dissent within a course rather than challenge power and inequality in a societal context (crowther 2000). in moocs where learning takes place across distributed platforms, the curriculum can also be influenced by hidden knowledge infrastructures, software and associated practices (edwards 2015b). edwards has argued that ‘the broader practices of knowledge infrastructures result in increasing inscrutability in the curriculum’ (edwards 2015a, p. 11). in a course like rhizo14, the process of curriculum emergence, both vis-à-vis platforms used, and social interaction is of particular significance precisely because the curriculum is not identified or made explicit in advance. community and other forms of association community learning is an ancient phenomenon, but the definition of community is constantly changing (yuan 2012). a number of different terms have been used when discussing social learning in communities, for example, communities of inquiry (garrison, anderson, and archer 2000), communities of interest, learning communities (palloff and pratt 1999) and communities of practice (wenger 1998). community as a concept remains confusing (cox 2005) and this is exacerbated by the growth of online networks and virtual communities as sites for social learning. wenger has said that all communities are networks, but not all networks are communities (e. wenger, personal communication, 2008). wenger, trayner, and de laat (2011) see networks and communities as ‘two aspects of social structures in which learning takes place’. the internet has brought new ways to associate, communicate, organise and learn through bulletin boards, email, web sites and virtual or online communities. the impact of the internet on the meaning of community has also been discussed by yuan (2012) who writes that there is ‘a mismatch between ‘‘community’’ as a traditional form of society and the internet as a new form of materiality on liberal narratives’ (p. 666) and ‘a mismatch between empirical findings about online social interaction patterns and what is symbolized by the concept of community’ (p. 667). brint (2001, p. 8) critiqued the community concept, writing that tönnies’ (1957) belief that community is based on neighbourhood, locality and strong ties, invites confusion and causes people to either romanticise or debunk community. brint explored communities in relation to virtues and vices. he identified two groups which he named ‘communitarians’ and ‘liberals’. communitarians believe the virtues lie in fraternalism and mutual support, low levels of stratification and power, and informal settlement of disputes (brint 2001, p. 14). liberals believe the vices lie in limits to human freedom, hostility to change, authoritarian thought, hostility to innovation and creativity, inequalities of power, exclusionary behaviour, enforced conformity and illiberalism (brint 2001, p. 19). brint concluded that for the modern world to benefit from the concept of community, communities need to be loosely connected activity groups, which demonstrate the virtues and avoid the vices. in order to do this, they need consistent support for intergroup tolerance, trust and respect. research in learning technology citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 yuan (2012) describes a networked approach to online community as an inadequate conceptualization of culture; wenger (1998) writes of dysfunctional communities and the problems of idealizing communities of practice; and brint (2001) writes that communities are rife with power and division and that ‘it is perhaps unnecessary to add that not all communal social relations are amicable’. in the context of higher education, kogan (2000) asserts that community is ‘a warm glow word’ implying good relations that may neither exist nor be necessary; the use of the term could interfere with the hard work of showing and making effective connections. he recommends an alternative to community that suggests that institutions, subjects and individuals could give ‘each other support not by the assumed sharing of values implied by ‘‘community’’ but through negotiation and exchange’ (kogan 2000, p. 216). permitting different values could sustain the diversity needed for fruitful knowledge construction (kirschner 2015). media used and strength of ties can constitute each other. haythornthwaite (2002) identifies the need for communication media that can reach all members to maintain strong and weak ties at low effort; and media that can support subgroups that are focussed on specific activities. at first view, snss do offer support both for maintaining connections, and focussing on specific activities by use of groups and hashtags. streams such as in twitter and facebook timelines appear to offer the chance to maintain strong and weak ties, but the application of algorithms to the timelines seem to make weak ties weaker. facebook rewards increased interaction with visibility and threaten invisibility to those who are less interactive (bucher 2012). community is the curriculum linking the curriculum to the community can be seen as part of a broader, historical project to make education more relevant, ground it in its local context and/or to make a community within the classroom (dewey 2013). service learning, which links learning to service in the community, is seen as a way of enriching learning, renewing communities and giving dignity to a scholarship of community (bringle and hatcher 1996); and as having an impact on the community in university�community partnerships (cruz and giles, jr. 2000). service learning emerges from and is embedded in a pre-existing local community, but in the rhizo14 mooc both community and curriculum were expected to emerge within the course (cormier 2008). methodology rhizo14 was an experimental course which made use of a range of snss and emerging technologies. veletsianos (2013b, p. 641) has pointed out that the potential of emerging technologies remains undecided. they are in a state of becoming and are neither fully understood nor fully researched. it is within this context that we have tried to make sense of community and curriculum formation in rhizo14. having been participants in rhizo14 we were aware of the dangers of projecting our experiences onto those of others. we did not want to speak ‘for’ others but rather to engage with what was said in survey responses, twitter, facebook and other spaces. this was done through participant observation and the retrospective study of course archives. we see ourselves as both insiders and outsiders (dwyer and buckle 2009) to this research process. f. bell et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 the ethical protocol for use of data was developed in consultation with rhizo14 participants. like veletsianos (2013b, p. 643), we do not claim to be conducting an ethnography, but rather to have used ethnographic data collection methods, alongside surveys, social network analysis (sna) and review of twitter and facebook archives. the survey attracted 47 responses and more than 30,000 words of data. the follow-up email interviews sent to 35 participants yielded more than 15,000 words of data. our own responses as the authors of this paper were discounted. the survey data collection for the study allowed respondents to elect for anonymity. full details of the survey and emails interviews have been fully reported in the first two papers (mackness and bell 2015; mackness, bell, and funes 2015), and the research process is shown in figure 2. we started by independently inductively coding the 47 survey responses (thomas 2006). as new codes emerged, we rechecked the previous data to see if new occurrences could be found and created an initial clustering of codes. we checked and amended the coding where errors and inconsistencies emerged, and reviewed the clustering, revising and collapsing codes where appropriate. we then independently coded the follow-up interview data identifying core themes that emerged from our inductive analysis of survey responses and follow-up interviews. we developed narratives for these core themes. in order to gain an overview of what was happening across the different platforms and services, we collected activity data on the p2pu site, the g� rhizo14 community and participant blogs manually, and were able to extract data automatically from facebook and twitter. activity data from different platforms are not directly comparable as the concept of a post and a comment is context-dependent but the data does offer a descriptive view. two summaries of activity data are presented in figure 2. organic research process (updated from (mackness and bell 2015)). research in learning technology citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 tables 1 and 2. these show that the most active areas on the course were the facebook group and the twitter hashtag and this determined the decision to explore both these platforms in greater depth. we discuss these tables in the next section. twitter and facebook activity data were analysed using sna tools. data from the twitter rhizo14 hash tag was exported from the openly available tagsexplorer archive into netlytic. data from the rhizo14 facebook group was downloaded using the netvizz app that creates networks and tabular files for user activity around posts (rieder 2013), exported as a.gdf file that can then be visualised and explored further using the software gephi (bastian, heymann, and jacomy 2009). we used the visualisations and data from netlytic and gephi, to identify peaks of activity that could then be explored with the qualitative data from facebook and twitter. the core theme narratives were considered in conjunction with a review of the facebook and twitter archives, directed by visualisations of activity data to guide the discussion below. discussion introduction in this discussion, we explore, � network and community formation � curriculum development and emergence � the impact on the course of the most active course platforms there were three forms of participant association highlighted in the title and introduction to rhizo14: networking, clustering and community is the curriculum. we explored frequency of participant association by means of rhizo14 activity data. there was a pattern of decline in participation across most of the rhizo14 platforms (g�, twitter, blogs and p2pu) between the official and unofficial courses as shown in tables 1 and 2. these patterns of activity reflect the decline in participation and low completion rates recorded in other moocs (brinton et al. 2014; jordan 2014). in rhizo14, twitter and facebook were the most active spaces that supported participant observation. what stood out in the activity data was that in facebook, unlike on the other platforms used in rhizo14, there was an increase in the number of posts, and an increase in engagement with posts and by users (see table 2). however, whilst activity in facebook increased overall, it was focussed on a small minority of participants. in the official course, the top 10 of 239 active participants accounted for 47% of the engagement in facebook increasing to 67% during the unofficial course for the top 10 of 198 active participants (see table 2). networking and community formation in rhizo14 although our main focus was on twitter and facebook, we also observed events happening on participant blogs and other rich activities as links to them cropped up in the archives. our analysis of twitter and facebook confirmed our event-based perspective (see figures 3�6). networking was evident in all of the platforms in the busy early weeks of rhizo14. there was also a networking activity, promoted by the course leader for f. bell et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 table 1. summary of activity on official and unofficial courses for p2pu, g� and blogs. official course no. of p2pu comments g� posts g� comments no. of unique posters on g � blog posts blog comments no. of posters on blogs 14-jan-14 cheating as learning 184 73 254 44 28 111 22 21-jan-14 enforcing independence 103 87 203 38 18 128 14 28-jan-14 embracing uncertainty 47 40 102 19 12 63 9 04-feb-14 is books making us stupid? 38 45 136 18 17 133 12 11-feb-14 community as curriculum 14 35 89 15 12 54 8 18-feb-14 planned obsolescence 28 64* 162* 18* 10 40 9 total official course 414 ** 97 529 ** unofficial course # p2pu comments 25-feb-14 the lunatics are taking over the asylum 31 3 9 3 04-mar-14 demobbing soldiers 31 4 12 4 11-mar-14 why do we need lurkers? 37 3 16 2 22-mar-14 creativity: the art of thriving in arid environments 7 4 8 3 31-mar-14 powerful thoughts 27 2 2 1 10-apr-14 mooc missionaries 11 1 0 1 total unofficial course 144 ** 17 47 ** overall total 558 344 946 ** 114 576 ** *because the g� space was not organised by week in the unofficial course, week 6, w/c 18-feb-14, contained all posts from then until the end of the unofficial course. **summing total no. of posters on g� and blogs doesn’t make sense because unique number of posters were summed across a week and could overlap between weeks. r e se a rch in l e a rn in g t e ch n o lo g y c ita tio n : r e se a rc h in l e a rn in g t e c h n o lo g y 2 0 1 6 , 2 4 : 2 9 9 2 7 h ttp ://d x .d o i.o rg /1 0 .3 4 0 2 /rlt.v 2 4 .2 9 9 2 7 9 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 community building. this used tags explorer which provided a visualisation of connections on the twitter hash tag (cormier 2014b). twitter users were encouraged to ‘click on lonely dots’ to draw participants into interactions, and this was generally welcomed by participants. this resulted in a flurry of activity, but it was short-lived (figure 3), and there was no corresponding activity detectable in facebook (figure 4). there were other participant initiatives that enabled participants to find each other’s blogs. lockhart (2014) shared the comment scraper that posted outlines of mooc blog posts and comments on one page so that readers could see which were attracting comments or not, and visit and comment on those with no or few comments. another participant shared a rhizo14 blogger feed list that could be used in an rss reader (melcher 2014) that helped participants keep track of blog postings. these initiatives supported the visibility of weak ties in contrast with the facebook algorithmic stream that tended to make less active participants invisible (see later section, facebook and the algorithmic stream, for further discussion of this point). clustering activities were also evident as participants instigated collaborative activities, such as the emergent collaborative poem facilitated through blogs and twitter (lau 2014). other sites for clustering included a diigo group for collaborative curation of resources. blogs could also become clusters as participants commented and linked from their own blog posts. rhizo14 demonstrated that people can come together on a course with minimal structure of resources or process; connect, engage, make multimedia artefacts together and, in some cases, make deep and lasting friendships. as one participant said learning is made more potent when it is fuelled by a sense of community and belonging but others were unable to come together or form meaningful associations, seeming to stand on the sidelines looking in . . .. i never found my tribe mostly i have a sense of looking in from the outside or far away at what are they doing � i seldom had a feeling of belonging to this community i wish i had not felt so peripheral table 2. activity data summary for rhizo14 twitter hash tag and facebook group. official course 8 jan 2014 to 24 feb 2014 unofficial course 25 feb 2014 to 16 apr 2014 total number of tweets 5089 1514 total number of facebook (fb) group posts 401 415 total engagements with fb posts (likes, comments, comment likes and shares) 6248 8096 total number of engagements by fb users (likes, comments but not comment likes or shares) 4059 4672 number of active participants on fb group 239 198 engagement (likes, comments) by top 10 most active participants as percentage of total engagement by active participants in fb group 47% 67% f. bell et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 one participant described the enthusiasm about community as an obstacle to connection as the prevailing narrative was too divisive. it seemed to this participant that in order to belong one had to buy into the idea that, we are great. figure 3. tweets (posts on twitter) by date in official course via tags explorer and netlytics.org. figure 4. engagement with facebook posts during official course via netvizz, gephi and excel. research in learning technology citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 we have no way of knowing where and how those standing on the sidelines, or turned off by an enthusiasm for community, were engaging with rhizo14. there was no designated main community space. they could have been posters of facebook posts that attracted no engagement (14 in the official course, and 13 in the unofficial course). there was a substantial thread of discussion about community in the facebook group at the beginning of week 5, whose topic was ‘community is the curriculum’ (figure 4). whether rhizo14 was a community or a network was discussed, along with how it was framed and in which spaces it was performed. some thought subcommunities aligned with platform, while others did not. this confirms cox’s assertion that community can be a confusing concept (cox 2005). the discussion on this thread figure 5. tweets (posts on twitter) by date in unofficial course via tags explorer and netlytics.org. cae is collaborative auto ethnography. figure 6. engagement with facebook posts during unofficial course via netvizz, gephi and excel. f. bell et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 approached a consensus on feelings of closeness to others in a rhizo14 community, analogous to kogan’s characterisation of community as a ‘warm glow word’ (kogan 2000). in the absence of any shared understanding of community, the notion of community may have been ‘romanticised’ as proposed by brint (2001). as we can see from figure 4, by week 5 activity was sustained on facebook whilst it was falling away on twitter (figure 3) and p2pu, g� and blogs (table 1), with facebook becoming the main community space towards the end of the official course, and this was reinforced during the unofficial course. curriculum development in rhizo14 in rhizo14, the curriculum was a praxis (smith 2008) in which participants shared and created multimedia such as the collaborative poem already discussed. the weekly provocative statements (see earlier description of course) gave a broad steer to topics, and hinted at what was the preferred view of the course convener who shared his perspective on the topic in the weekly video. activity related to some topics (e.g. ‘cheating as learning’) was clearly observable on twitter (see figure 3), but links to further topics became less clear as activity declined. on facebook there were several very long threads only loosely associated with the weekly topics. as there were no stated objectives for the course, expectations for the curriculum relied initially on the title and introductory blog posts. the implicit expectation was that participants would discuss and model rhizomatic learning, and form a community through which the curriculum would emerge. in the official course, a defining early event that related to ideas of community and influenced curriculum development was a controversy in week 2 played out mainly on facebook. the controversy was about whether or not deleuze and guattari’s ideas about rhizomatic thinking should be discussed. a blogger wrote a post expressing the personal view that some engagement with the theory that underpins rhizomatic learning was necessary. another participant took exception to this post, feeling that discussion of theory would exclude non-academics. this controversy led to the longest thread in facebook during the official course (figure 4), in which the discussion was rich, although opportunities for exploring how disagreements and misunderstanding in online spaces can be resolved were not realised. as the course progressed, the influence of this controversy became more evident as theory came to be associated with privilege. theory and practice were discussed in terms of academics and non-academics, theorists and pragmatists: in other words categorisations of participants rather than of concepts, and the theorists/academics came to be associated with privilege. although the main protagonists in the theory controversy had disappeared from rhizo14 within a couple of weeks, the legacy of this incident was the emergence of a kind of discourse that set a binary opposition of ‘academics vs pragmatists’ and this led to theory being largely off the curriculum. some representative quotes from these themes show the oppositional nature of the discourse that emerged, i could have done with a lot less of the theoretical and lot more of the practical application of the principles in the classroom. . . . i almost quit when the theoretical bloviating reached a fever pitch and just about pegged my bullshit-0-meter, but there were always enough pragmatists out there to reel me back in. research in learning technology citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 in the normal course of things, i would have gone back to first principles and read ‘the’ theory, but i was so hacked off by academic posturing around d&g on rhizo14 i’ve set my face against them. another participant noted that the theory controversy led to discussion that focussed on people rather than ideas: . . . it seemed to me that theoretical explorations and advocacy for the sake of discussion were being unconsciously conflated with underlying belief systems about how people ‘should’ react/behave/respond to the rhizomatic theory we present these quotes to show the discomfiture of some participants with exposure to other participants’ learning goals and styles of communication. heterogeneity is one of the principles of the rhizome (figure 1), a principle that expects difference, and does not require a ‘homogeneous linguistic community’. in rhizo14, a convergence of one community emerged rather than heterogeneous subcommunities, with facebook as the main space for sharing and communication. we discuss this further in the following section. facebook and the algorithmic stream a further significant influence on curriculum development was the tendency for rhizo14 participants to discuss blog posts within the facebook group as well as at the blog itself. when someone posts a link within a facebook post, content from the link is pulled through into the facebook site. in the case of blog posts, this seemed to invite comment within the facebook thread, possibly without commenters even visiting the blog and without the blogger knowing that their post was being discussed. thus facebook’s emergence as the main rhizo14 discussion space significantly affected the formation of community and curriculum development. some participants in the rhizo14 facebook group experienced close emotional connection, and even lasting friendship. facebook was favoured by many participants as a congenial environment for sharing ‘creative expression, play, and multi-modal formats of communication’ (survey respondent). one participant noted that rhizo14 gave her the space ‘to talk to other like-minded people’. kirschner (2015) has confirmed this capacity of facebook for creating networks of friends based on similarity and suggests that, as such, facebook is a poor environment for fruitful argumentation and discussion: it connects ‘likeminded people who discuss, confirm, validate and strengthen the group’s position’ (p. 622). whilst we cannot know the specific details of the operation of the algorithm at a given time for a given user, we can assume that in facebook we are more likely to see posts with a high engagement score and from people to whom we are more connected (backstrom 2013). if posts with no engagement are ‘competing’ with posts engaged with by participants who are friends with each other on facebook, the more engaged posts will tend to come to the top of more streams. in turn, they attract more engagement, particularly from more closely connected participants, and the posts with no engagement become less noticeable, as they disappear from busy streams. this threat of ‘invisibility’ wielded by the stream and algorithms in facebook was noted by bucher (2012) and helps to explain the wide disparity in engagement that we f. bell et al. 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 observed between participants, and the ‘more by fewer’ effect between the official and unofficial courses (table 2). in rhizo14, groups of learners who only participated on one platform could become invisible to other groups participating in different platforms and within facebook itself posts that were not engaged with by the most active participants disappeared to the bottom of the stream supported by the facebook algorithm. facebook becoming established as the main space led to clustering around ties that were already strong, thereby limiting the potential benefit of weak and latent ties (haythornthwaite 2002). it was not a universally popular location for discussion, even with those who ultimately became active members of the facebook group. one such participant wrote, ‘i really hate facebook and i really like google�’ and of having to adapt to facebook in order to join the discussion. there were others who were not even aware of the discussions in facebook. another participant who was not a member of facebook wrote: what i found most interesting for me, anyway, is that i never was part of the facebook arm (hand? fingers?) of rhizo14, and in fact, i didn’t know the fb element of rhizo14 existed until a few months after rhizo14 ended (if it ever really ended). . . . mostly, i am ok with that. but realizing later that there were discussions unfolding and people connecting outside of my own realm of rhizo14 leaves me feeling a little lost, as if maybe what i thought was happening didn’t quite happen the way i thought it had. what was notable was that choice of platform was generally observed rather than discussed in a critical fashion. conclusions in this paper, we asked the question: can the community be the curriculum? this paper has explored how community formed during rhizo14, the curriculum that developed and the relationship between the concept of ‘community is the curriculum’ and rhizo14 participant learner experiences in two snss in particular. a key insight from our work here is the need to critically assess the hidden impact of the software platforms used in determining both curriculum and community. we spoke earlier about facebook being a less than ideal environment for fruitful dialogue because in its design it connects those who already think alike (kirschner 2015). if we add to this the ‘more by fewer’ effect found in our data (see table 2 for details) and the stated design principle of both community and curriculum emerging within the course in rhizo14, we can see how tensions and contradictions can remain hidden from view as those who are not ‘likeminded’ may be silenced by default, if not design. below we sum up key issues the paper raises and conclude with what we feel is this paper’s contribution to mooc research overall. the formation of community advice to participants at the beginning of rhizo14 encouraged them to network and then cluster to find those with shared interests � a community. the idea of community was also embodied in the title of the course ‘community is the curriculum’. research in learning technology citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 15 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 there was confusion over what community meant, and where and how to perform it in rhizo14, as participants brought different tacit understandings of the term to the course. the ‘warm glow’ communitarian notion of community emerged as a shared meaning that often defined both interaction and curriculum. the alternative approach, as recommended by kogan (2000), that support could be offered through negotiation and exchange rather than the assumption of shared values was not available. for participants, the move from networking to clustering became a process of convergence, and this we see as in contradiction with the rhizomatic principle of heterogeneity. the development of curriculum in a community collaborative use of multimedia was evident and participants enjoyed these selforganised activities, contributing to the curriculum in concrete ways by producing digital artefacts. as facebook became the main space, social learning guided loosely by the weekly provocative statements became the curriculum � an emergent curriculum of praxis. the controversy over links between deleuzian theory and rhizomatic learning we discussed earlier revealed the influence of community formation in facebook on what became an acceptable curriculum. the reduction of links between facebook and other spaces over time rendered alternative curricula less acceptable. the ephemeral nature of the algorithmic facebook stream militated against the creation of a negotiated curriculum, and relied more on shared interests and activities. as the social links in facebook deepened, the rhizomatic principle of heterogeneity was lost as difference was filtered out either through explicit commenting on appropriateness or through the algorithm demoting posts that lacked engagement. our contribution to theory and practice we found that choice of snss impacts on community and curriculum formation. teachers, facilitators and learners should reflect on their choice of platforms and how these are used to avoid creating learning environments that contradict the chosen pedagogy. this requires understanding of sns platform design and what it is designed to encourage or discourage. further work can explore the digital and media literacies required by educators and learners in innovative learning ventures. tensions between the lack of agreed objectives, minimal curriculum and the need to form community impacted on the experiences of learners. this may have been an intentional element in the course design, yet from a theoretical perspective rhizomatic learning is intended to encourage heterogeneity rather than convergence to the discourse acceptable to the most active participants amongst hundreds. if educators wish to achieve a more heterogeneous participation and curriculum, in the absence of explicit learning goals, we have argued that it may be more suitable to consider encouraging kogan’s approach of negotiation and exchange in networks, in place of an emphasis on a term like community with its inherent conceptual challenges. this seems particularly important in a course that is based on an emergent approach to both community and curriculum. finally, we believe that other research in moocs that include facebook groups can benefit from adopting the methodological approach this paper illustrates to make greater sense of participants’ experiences and responses to social learning. f. bell et al. 16 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 acknowledgements we would like to thank rhizo14 participants especially those who gave their time to complete the survey and answer our questions. we would also like to thank anonymous reviewers, catherine cronin and george veletsianos whose comments helped us to improve the originally submitted paper. references backstrom, l. (2013) ‘news feed fyi: a window into news feed. facebook’, [online] available at: https://www.facebook.com/business/news/news-feed-fyi-a-window-into-news-feed bastian, m., heymann, s. & jacomy, m. (2009) ‘gephi: an open source software for exploring and manipulating networks’, bt international aaai conference on weblogs and social. third international icwsm conference, san jose, ca, pp. 361�362. bates, a. w. (2015) ‘teaching in a digital age’, open.bccampus.ca, [online] available at: http:// opentextbc.ca/teachinginadigitalage/ bringle, r. g. & hatcher, j. a. (1996) ‘implementing service learning in higher education’, the journal of higher education, vol. 67, no. 2, pp. 221�239. brint, s. (2001) ‘gemeinschaft revisited: a critique and reconstruction of the community concept’, sociological theory, vol. 19, pp. 1�23. brinton, c. g., et al., (2014) ‘learning about social learning in moocs: from statistical analysis to generative model’, ieee transactions on learning technologies, vol. 7, no. 4, pp. 346�359. bucher, t. (2012) ‘want to be on the top? algorithmic power and the threat of invisibility on facebook’, new media & society, vol. 14, no. 7, pp. 1164�1180, [online] available at: http://nms.sagepub.com/cgi/doi/10.1177/1461444812440159 cormier, d. (2008) ‘rhizomatic education: community as curriculum’, innovate, vol. 4. [online] available at: http://tinyurl.com/6axkez cormier, d. (2010) ‘success in a mooc’, [online] available at: https://www.youtube.com/ watch?v�r8avyq5zqm0 cormier, d. (2013) ‘unravelling -� a model for an open course?’, [online] available at: http:// davecormier.com/edblog/2013/12/29/unravelling-a-model-for-an-open-course/ cormier, d. (2014a) ‘rhizomatic learning � the community is the curriculum’, p2pu, [online] available at: https://p2pu.org/en/courses/882/rhizomatic-learning-the-communityis-the-curriculum/ cormier, d. (2014b) ‘connection activity � how you can help make more community’, [online] available at: http://davecormier.com/edblog/2014/01/19/connection-activity-how-you-canhelp-make-more-community/ cox, a. m. (2005) ‘what are communities of practice? a comparative review of four seminal works’, journal of information science, vol. 31, no. 6, pp. 527�540, [online] doi: http://dx. doi.org/10.1177/0165551505057016 crowther, j. (2000) ‘participation in adult and community education: a discourse of diminishing returns’, international journal of lifelong education, vol. 19, no. 6, pp. 479�492. cruz, n. & giles, d. e., jr. ‘where’s the community in service-learning research?’, michigan, journal of community service learning, vol. 7, no. 1, pp. 28�34. cunningham, i. (1999) the wisdom of strategic learning: the self managed learning solution, gower publishing, ltd, aldershot, england. de laat, m. & prinsen, f. (2014) ‘social learning analytics: navigating the changing settings of higher education’, research & practice in assessment, vol. 9, pp. 51�60. deleuze, g. & guattari, f. (1987) a thousand plateaus: capitalism and schizophrenia, university of minnesota press, minneapolis; london. dewey, j. (2013) the school and society and the child and the curriculum (google ebook), university of chicago press, [online] available at: http://books.google.com/books? hl�en&lr�&id�1f0taaaaqbaj&pgis�1 dwyer, s. c. & buckle, j. l. (2009) ‘the space between: on being an insider-outsider in qualitative research’, international journal of qualitative methods, vol. 8, no. 1, [online] available at: http://ejournals.library.ualberta.ca/index.php/ijqm/article/view/2981 research in learning technology citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 17 (page number not for citation purpose) https://www.facebook.com/business/news/news-feed-fyi-a-window-into-news-feed http://opentextbc.ca/teachinginadigitalage/ http://opentextbc.ca/teachinginadigitalage/ http://nms.sagepub.com/cgi/doi/10.1177/1461444812440159 http://tinyurl.com/6axkez https://www.youtube.com/watch?v=r8avyq5zqm0 https://www.youtube.com/watch?v=r8avyq5zqm0 https://www.youtube.com/watch?v=r8avyq5zqm0 https://www.youtube.com/watch?v=r8avyq5zqm0 https://www.youtube.com/watch?v=r8avyq5zqm0 http://davecormier.com/edblog/2013/12/29/unravelling-a-model-for-an-open-course/ http://davecormier.com/edblog/2013/12/29/unravelling-a-model-for-an-open-course/ https://p2pu.org/en/courses/882/rhizomatic-learning-the-community-is-the-curriculum/ https://p2pu.org/en/courses/882/rhizomatic-learning-the-community-is-the-curriculum/ http://davecormier.com/edblog/2014/01/19/connection-activity-how-you-can-help-make-more-community/ http://davecormier.com/edblog/2014/01/19/connection-activity-how-you-can-help-make-more-community/ http://dx.doi.org/10.1177/0165551505057016 http://dx.doi.org/10.1177/0165551505057016 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://books.google.com/books?hl=en&lr=&id=1f0taaaaqbaj&pgis=1 http://ejournals.library.ualberta.ca/index.php/ijqm/article/view/2981 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 edwards, r. (2015a) ‘software and the hidden curriculum in digital education’, pedagogy, culture & society, vol. 23, no. 2, pp. 265�279, [online] available at: http://www.tandfonline. com/doi/full/10.1080/14681366.2014.977809 edwards, r. (2015b) ‘knowledge infrastructures and the inscrutability of openness in education’, learning, media and technology, vol. 40, no. 3, pp. 251�264, [online] available at: http://www.tandfonline.com/doi/abs/10.1080/17439884.2015.1006131 garrison, d. r., anderson, t. & archer, w. (2000) ‘critical inquiry in a text-based environment: computer conferencing in higher education’, the internet and higher education, vol. 2, no. 2�3, pp. 87�105. haythornthwaite, c. (2002) ‘strong, weak, and latent ties and the impact of new media’, information society, vol. 18, no. 5, pp. 385�401. hiemstra, r. (1994) ‘self-directed learning’, reprinted here with permission, in the international encyclopedia of education (second.), pp 1-11 eds. t. husen & t. n. postlethwaite, pergamon press, oxford. ito, m., et al., (2014) ‘connected learning: an agenda for research and design’, [online] available at: http://dmlhub.net/publications/connected-learning-agenda-for-research-anddesign/ jordan, k. (2014) ‘initial trends in enrolment and completion of massive open online courses massive open online courses’, international review of research in open and distance learning, vol. 15, no. 1, pp. 133�160. kirschner, p. a. (2015) ‘facebook as learning platform: argumentation superhighway or deadend street?’, computers in human behavior, vol. 53, pp. 621�625. doi: http://dx.doi.org/10. 1016/j.chb.2015.03.011 kogan, m. (2000) ‘higher education communities and academic identity’, higher education quarterly, vol. 54, no. 32, pp. 207�216. lau, t. (2014) ‘engagement or alienation? reflections on mooc design, facilitator role, and context. special issue on moocs’, journal of global literacies, technologies, and emerging pedagogies, vol. 2, no. 3, pp. 236�240. lockhart, g. (2014) ‘mooc comment scraper output � #rhizo14’, [online] available at https:// gbl55.wordpress.com/mooc-comment-scraper-output-rhizo14 mackness, j. & bell, f. (2015) ‘rhizo14: a rhizomatic learning cmooc in sunlight and in shade’, open praxis, vol. 7, no. 1, pp. 25�38. mackness, j., bell, f. & funes, m. (2015) ‘the rhizome: a problematic metaphor for teaching and learning in a mooc’, australasian journal of educational technology, vol. 32, no. 1, pp. 78�91. melcher, m. (2014) ‘#rhizo14 opml feedlist’, [online] available at https://x28newblog. wordpress.com/2014/01/12/rhizo14-opml-feedlist/ neary, m. & winn, j. (2009) ‘the student as producer: reinventing the student experience in higher education’, in the future of higher education: policy, pedagogy and the student experience, eds. l. bell, h.stevenson, m. neary, continuum, london, pp. 192�210. palloff, r. & pratt, k. (1999) building learning communities in cyberspace, jossey-bass publishers, san francisco, ca. pinar, w. f., et al., (1995) understanding curriculum. an introduction to the study of historical and contemporary curriculum discourses. counterpoints: studies in the postmodern theory of education, vol. 17, peter lang publishing, inc., new york. rieder, b. (2013) ‘studying facebook via data extraction: the netvizz application’, proceedings of websci’13, the 5th annual acm web science conference, paris, france, pp. 346�355. smith, m. k. (2000) ‘curriculum theory and practice’, in the encyclopaedia of informal education, [online] available at: http://www.infed.org/biblio/b-curric.htm smith, m. k. (2008) ‘informal learning’, in the encyclopaedia of informal education, [online] available at: http://infed.org/mobi/informal-learning-theory-practice-and-experience/ thomas, d. r. (2006) ‘a general inductive approach for analyzing qualitative evaluation data’, american journal of evaluation, vol. 27, no. 2, pp. 237�246. tönnies, f. (1957) gemeinschaft und gesellschaft [community and society], michigan state university press, oxford, england. veletsianos, g. (2013a) ‘how do learners experience open online learning?’, hybrid pedagogy, [online] available at: http://www.hybridpedagogy.com/journal/how-do-learners-experienceopen-online-learning/ f. bell et al. 18 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 http://www.tandfonline.com/doi/full/10.1080/14681366.2014.977809 http://www.tandfonline.com/doi/full/10.1080/14681366.2014.977809 http://www.tandfonline.com/doi/abs/10.1080/17439884.2015.1006131 http://dmlhub.net/publications/connected-learning-agenda-for-research-and-design/ http://dmlhub.net/publications/connected-learning-agenda-for-research-and-design/ http://dx.doi.org/10.1016/j.chb.2015.03.011 http://dx.doi.org/10.1016/j.chb.2015.03.011 https://gbl55.wordpress.com/mooc-comment-scraper-output-rhizo14 https://gbl55.wordpress.com/mooc-comment-scraper-output-rhizo14 https://x28newblog.wordpress.com/2014/01/12/rhizo14-opml-feedlist/ https://x28newblog.wordpress.com/2014/01/12/rhizo14-opml-feedlist/ http://www.infed.org/biblio/b-curric.htm http://infed.org/mobi/informal-learning-theory-practice-and-experience/ http://www.hybridpedagogy.com/journal/how-do-learners-experience-open-online-learning/ http://www.hybridpedagogy.com/journal/how-do-learners-experience-open-online-learning/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 veletsianos, g. (2013b) ‘open practices and identity: evidence from researchers and educators’ social media participation’, british journal of educational technology, vol. 44, no. 4, pp. 639�651. watters, a. (2015) ‘challenging moocs’, hack education, [online] available at: http:// hackeducation.com/2015/06/23/superhero-mooc/ wenger, e. (1998) communities of practice: learning, meaning and identity, university press, cambridge. wenger, e., trayner, b. & de laat, m. (2011) ‘promoting and assessing value creation in communities and networks: a conceptual framework’, centrum, vol. 18, pp. 1�60. williams, r., karousou, r. & mackness, j. (2011) ‘emergent learning and learning ecologies in web 2.0’, the international review of research in open and distance learning, vol. 12, no. 3, [online] available at: http://www.irrodl.org/index.php/irrodl/article/view/883 yuan, e. j. (2012) ‘a culturalist critique of ‘‘online community’’ in new media studies’, new media & society, vol. 15, no. 5, pp. 665�679, [online] available at: http://nms.sagepub.com/ cgi/doi/10.1177/1461444812462847 research in learning technology citation: research in learning technology 2016, 24: 29927 http://dx.doi.org/10.3402/rlt.v24.29927 19 (page number not for citation purpose) http://hackeducation.com/2015/06/23/superhero-mooc/ http://hackeducation.com/2015/06/23/superhero-mooc/ http://www.irrodl.org/index.php/irrodl/article/view/883 http://nms.sagepub.com/cgi/doi/10.1177/1461444812462847 http://nms.sagepub.com/cgi/doi/10.1177/1461444812462847 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29927 http://dx.doi.org/10.3402/rlt.v24.29927 calt12107.fm alt-j, research in learning technology vol. 12, no. 1, march 2004 networks and learning: communities, practices and the metaphor of networks chris jones* lancaster university, uk taylor & francis ltdcalt1210710.1080/0968776...research in learning technology0968-7769 (print)/1741-1629 (online)research article2004association for learning technology121000000march 2004centre for studies in advanced learning technologylancaster universitycartmel collegelancasterla1 4ylukc.r.jones@lancaster.ac.uk this article argues that the use of the network metaphor can link together various different aspects of research into the use of advanced learning technologies based on computer networks. the idea of networked learning has become commonplace as an alternative to e-learning that stresses the interaction of learners, tutors and resources through networks. the arguments put in this article are firstly that learning technology needs to take account of the wider debate about networks and secondly that research in this field needs to address the theoretical and practical issues raised by advances in the field of networks. a third point is that the idea of the network acts as a powerful metaphor even if we are able to discount any particular theory generated in its support. the network metaphor can act as a unifying concept allowing us to bring together apparently disparate elements of the field. networks are an important issue in the study of learning using advanced technologies and they speak to some of the central issues in learning theory such as virtual communities and communities of practice. introduction the growth of the internet and specifically the web has opened up an apparently limitless field for communication and the searching out of resources. however, theoretical approaches to computer networks have hardly kept pace with the spectacular adoption of networks in fields such as education and learning. at the same time we are increasingly aware of the ‘small world phenomenon’, those strange connections that seem to emerge and make this vast expanse feel much closer and more limited (watts & strogatz, 1998). castells has recently written about the way in which the possibilities and limits of the internet play out in relation to community (castells, 2001). he claims that the internet is especially effective at maintaining weak ties (p. 129) as well as strong ties at a distance. in education the focus has often been on the latter, with an interest in community and communities of practice being strongly * centre for studies in advanced learning technology, lancaster university, cartmel college, lancaster, la1 4yl, uk. email: c.r.jones@lancaster.ac.uk issn 0968–7769 (print)/issn 1741–1629 (online)/04/010081–13 © 2004 association for learning technology doi: 10.1080/0968776042000211548 82 c. jones related to the introduction of networked technologies. the 2003 association for learning technology conference had as its main theme communities of practice. the definition offered in the introductory thoughts for delegates noted that, in this approach, learning involved a process of engagement in a community of practice and that the distinctive feature of a community of practice was sharing practice. shared practice in turn requires members to have the time and space to collaborate (lave & wenger, 1991; wenger, 1998). by contrast, castells suggests that the pattern of the internet is one of ‘networked individualism’. castells also makes the point that on-line communities may emerge but that they rely on networks built out of an interaction between off-line and on-line sociability. this stress on the interconnection between off-line and on-line elements is a corrective to the often-overhyped claims for virtual universities and the notion that online worlds are somehow sequestered from real life concerns and processes. networks networks are providing an interdisciplinary framework for understanding a wide range of phenomena. the use of the term is not neutral and john law has noted that the ubiquitous nature of the metaphor may be due to more than the emergence of new social forms. indeed, it is possible: that we are in the process of uncritically reproducing some kind of dominant ideology. we are reproducing the ways in which the current orderings of the world like to represent themselves. (law, draft: para 1) as such, networked learning is part of a hegemonic discourse not simply in educational terms but as part of wider debates concerning the nature of social processes, power and culture. law notes that when we analyse in terms of networks, we also help to perform networks into being. the danger he identifies is that when we write as network analysts we may be buying into and adding strength to a managerialist agenda. i give this point prominence here as a corrective to any suggestion that might arise in what follows that networks can be adopted uncritically as a technical term solving a number of practical problems. networks have within them a latent politics that must be considered in our work. at a simple level, mathematical modelling of networks concerns itself with the description of phenomena in terms of nodes and the links between them. basic techniques focus on the ways in which transfers can take place across a network, for example whether the network is traversable or not. the importance of this field of study is that it holds out the prospect of developing mathematical laws of networks that may prove to be robust in describing a broad range of phenomena. in recent years a number of texts aimed at lay readers, originating in mathematical and physical science traditions, have begun to examine phenomena from a wide range of areas, including social and biological domains (barabasi, 2002; buchanan, 2002). network analysis examines systems through the links between nodes in a web-like structure. the nodes can be web documents, individuals, groups, publications or language. networks and learning: 83 within this paper i make no proper assessment of these broad claims; rather my intention is to draw attention to the literature, its relevance to the field of networked learning and a number of conceptualisations that arise from this work. work reported by barabasi has argued that advances in the characterisation of complex networks that focus on scale-free and hierarchical architectures demonstrate that a wide range of complex networks, including the web and internet, share these properties (2002). without attempting to develop the maths involved in a proper description of scale-free networks, it is important to state that these networks differ from random networks in which nodes are connected without any organising principle. scale-free networks show a degree of organisation; in particular they display a power-law distribution. those nodes with only a few links are numerous, but a few nodes have a very large number of links. the power law distribution is radically different from the familiar bell curve distribution with the distribution tapering off towards both margins. the rationale behind this kind of distribution rests on some simple propositions. firstly networks grow through the addition of new nodes and these new nodes link to pre-existing nodes. secondly there are preferential attachments within the network such that the probability of linking to a pre-existing node is higher if that node already has a large number of attachments. as well as being scale-free, most networks also display a high degree of clustering. this clustering is consistent with the predictions of scale-free descriptions of networks with a power law distribution, if a hierarchical organisation is introduced into the network model. the presence of the hierarchical structure helps to re-interpret the role of hubs in complex networks. the theory predicts a modular topology with self-nesting groups of nodes in clusters with dense interconnections. the overall suggestion is that networks are: … far from random, but they evolve following robust self-organising principles and evolutionary laws that cross discipline boundaries. (barabasi et al., 2002) the interesting nature of these networks retains some of the advantages of random networks (such as few steps between any two nodes) with the features of organisation. mark buchanan (2002) concentrates on recent developments in a well-known set of mathematical problems known as ‘small world phenomena’. the most popularised form of this problem is in the idea of ‘six-degrees of separation’: that any one person is likely to be connected to any other through only six steps. one of the key elements in the description of the networking structures and patterns that lie behind smallworld phenomena is the idea of the strength of weak ties. strong links, for example between close family members, do not reach out very far. it is often the weaker links that can bridge between quite different clusters in networked worlds. buchanan reports a light-hearted challenge in a german newspaper to connect a turkish kebabshop owner in frankfurt with his favourite actor, marlon brando. this was done using no more than six links of personal acquaintance using a bridging weak link, a friend of the kebab-shop owner living in california. the small world phenomenon draws attention to the need for weak links and bridging contacts to the organisation of networks. these links are central to the dissemination and propagation of ideas and 84 c. jones are of particular interest in education. together the idea of networks as having a selfnested topology and the idea of small worlds begins to map networks as self-organising structures that lie somewhere between order and chaos. this positioning of networks on the border of order and chaos offers a tantalising link with theories that have suggested chaos and complexity theory as potentially rich sources of ideas for analysing education (e.g. barnett, 2000). network society networks and the development of the internet have profoundly affected our views about how computers have impacted social life, education and learning. social theories that postulated a new form of society in which industry and factory production were replaced by service industry pre-dated the development of networks by some years. this point is worth noting, as theories of the ‘information’ society, developed well before networked computing, are often presented as outcomes of the application of computer technology. daniel bell’s idea that just as industrial society had replaced agrarian society so a new post-industrial society based on services would replace industrial society (1974) was later popularised in alvin toffler’s writings as the third wave (1980). steve wosniak, one of the originators of the personal computer at apple has been quoted as saying that the inventors of the pc were self-consciously exploring the idea of a technological revolution reshaping society (lyon, 1995). whilst the pc was an invention tangential to the development of networks, this comment implies a much more sophisticated relationship between technology and society than simple technological determinism—one in which the design of technical artefacts was informed by the very debates that claim technology as their source. the information society debate still influences discussions of how networks affect education and society. policy is often based upon the idea of computer networks being the prime lever for social change; for example a recent minister of education in the uk remarked: when you look back at the development of our schools it has been very strangely evolutionary. one of the reasons for that is there has not been a development, there has not been an invention which has brought about a transformation which has signalled the revolution and if you look at health it was maybe antibiotics or it was maybe the discovery of dna. if you look at transport it was maybe the internal combustion engine and once those discoveries were made, once those changes happened nothing was ever the same again. i think ict is our dna, it’s our internal combustion engine it is the trigger that can introduce a revolution in how we teach and in how we learn. (morris, 2002) christine steeples and i have discussed the persistence of the technological determinist approach in networked learning and its effect on policy elsewhere (jones & steeples 2002). at this point i simply want to stress the idea that the debates about the information society (and more recently networked society) inform policy makers who in turn set the parameters for work in the field of networked learning. advanced learning technology is introduced in relation to policy frameworks and funding decisions that are closely associated with claims that society is fundamentally transformed networks and learning: 85 by the use of computer networks. recently, wolf (2002) has provided an illustration of how the view of a changing workforce, central to the idea of an information society, misinforms a major plank of government policy. the economic case for widening participation and the massification of higher education is closely related to the idea that services are replacing industries and that knowledge workers are replacing unskilled and manual work. in her conclusion, wolf comments: for the last quarter-century education policy has been driven consistently by preoccupation with growth and with workplace skills … the results have been frequently disastrous. (wolf, 2002, p. 244) it is not necessary to agree with wolf’s anti-government agenda to agree with this observation, which links failing policies with a particular view of the needs of an information society. a second connection with the wider debate about network society is through the idea of virtual community. rheingold’s highly influential book the virtual community (1993) generalised from the well, a virtual community based in the san fransisco bay area in the mid-1980s. this book argued that the internet opened up the possibility of a new form of community on-line, formed around shared values and interests. the accounts of community facilitated by computer networks, provided by rheingold, have been aligned by others with social and situated views of learning and the idea of communities of practice in computer supported collaborative (or cooperative) learning (cscl). this claim about a technological basis for a new sociability formed a natural alliance with educational theories that placed an emphasis on communities of practice. accounts of situated learning (e.g. brown et al., 1989; lave & wenger, 1991; wenger, 1998) have had a particular influence. a community of practice has been defined by wenger using three dimensions: ● what it is about—as a joint enterprise as it is understood and continually renegotiated by its members; ● how it functions—as a mutual engagement that binds members together into a social entity; ● what capability it has produced—the shared repertoire of communal resources members have developed over time, e.g. routines, sensibilities, artefacts and vocabulary. this emphasis on collaboration and community stands in sharp contrast to the notion of networked individualism identified as a characteristic of networked society by castells (1996, 2001). the idea of communities of practice has recently come under critical review (fox, 2002a; hodgson & reynolds, 2002). fox articulates a view of communities of practice as sub-units within wider actor-networks that include both animate and inanimate elements, described as actants. he argues that communities of practice rely on a romanticism derived from anthropological studies set outside of modern societies. he goes on to suggest that its application within modern bureaucratised organisations focuses on sub-cultures, non-canonical groups in the interstices of organisations. 86 c. jones in summarising learning as peripheral participation in such communities of practice, fox comments: it is a community which is specialised around some specific practice or other, a single task, occupation, interest or profession. it may rely on a certain amount of formal education as a pre-requisite for joining (e.g. the us naval quartermasters often had to attend training school), but the real learning comes from the triadic communal relations between oldtimers, young journey-men and/or women and newcomers, i.e. real learning comes from the social relations between people with more knowledge and skill interacting with people with less knowledge and skill in the pursuit of a common practice. (fox, 2002a, p. 111) as an alternative to communities of practice fox suggests actor–network theory (2002a, 2002b). in his view communities of practice may be seen as a special case in terms of actor networks as they have specific practices within them such as mimicry, demonstrating and working together. beyond a certain scale fox argues that a community of practice becomes an imagined community in which the actor-networks link and connect multiple communities of practice. the view fox outlines is one that has a strong resonance with the observations by castells in relation to networked individualism and community in networked society. together they imply that communities and communities of practice are not the necessary or inevitable outcomes of networked society or networked technology; rather, they are special cases of more general network phenomena that rely on a particular form of individualisation. hodgson and reynolds’ critique of community focuses on power and democracy. they argue that networked learning offers a promising medium for supporting participative approaches to learning and that participative and democratic values are facilitated because it allows for the emergence of ‘splinter’ groups based on differences or differences of interests. they claim that traditional approaches tend to reflect notions of collaboration that discourage recognition of differences and that practices in networked learning still conform to that approach. in practical terms trehan and reynolds (2002) have noted that within peer assessed groups aspects of power relations, associated with wider social networks in education, impinge on the practices of a learning community intended to foster a cooperative and collaborative style of working. the comments of hodgson, reynolds and trehan touch on some central questions in relation to networks and communities when they are applied to educational settings. network metaphors can be used in such a way that power relations become diminished or invisible. the different access to resources held by actors in the network can be ignored by a focus on connections between nodes rather than on which nodes act as attractors in the flows across the network. in education the gloss of the ‘guide on the side’ has often ignored the power of the tutor in setting assessment tasks and in grading those assessments (jones, 2000). as trehan and reynolds make clear, these power relationships are also evident in the interactions of collaborative and peer assessment. networked learning in the network society the claim of this paper is that the use of the network metaphor can link the different aspects of research into the use of advanced learning technologies based on computer networks and learning: 87 networks. in this section i want to draw attention to the connections at various levels and how the network metaphor might be applied. the aspects i will cover are not exhaustive but are intended to give a flavour of the range of issues to which this metaphor can be applied. policy and politics higher education is currently suffering a policy overload in the uk. the government white paper issued in the spring of 2003 may either be on its way to implementation or dead in the water by the time this article appears in print. the entire process of consultation and negotiation of this policy will illustrate a powerful application of network theory in terms of government and governance. the idea of policy networks has developed in relation to a broad change in the relationship of government to society. the notion of governance has been used to separate out government action and actions that may include government and its agencies but which incorporate a network of bodies, some within and some outwith the boundaries of government itself. governance in this sense refers to the management of ‘self-organising, interorganisational networks with the following characteristics’ (rhodes, 1997): 1. interdependence between organisations; 2. continuing interaction between network members; 3. game-like interactions, rooted in trust and regulated by rules; and 4. no sovereign authority, self-organising, with a limited state steer. politically, this structure is an organisational third way, standing between the hierarchy associated with the bureaucratic active state and the anarchy of the market (rhodes, 1997). there are potential concerns that this form of governance is not as loose as it might appear at first glance. in managerial terms the autonomy of teams, brought together either for specific projects or as self-managing units, is circumscribed. an early commentator on new public sector management described such arrangements as ‘freedom within boundaries’ (hoggart, 1991). in other words, networks can concentrate as well as disperse power. power can be exercised in the setting of the limits within which component parts can operate. at another level, governance can concentrate its efforts on performance, on products not process. traditional management might be concerned with the way that work was conducted, with process issues. networked forms of governance shift concerns to outputs and performance, to what is done, not how it was achieved. the education system has been at the sharp end of reforms based on this model. the network metaphor can also help evaluate novel policy initiatives. the evaluation of the uk’s learning and teaching support network (ltsn) pointed to the network nature of the organisation and how this structure affected policy that flowed through it and its influence on policy formation. the subject centres had a high degree of autonomy in their dealings with subject communities and this in turn had a strong influence on how they developed. the overall structure of the ltsn was of a 88 c. jones managed network and policy was introduced into this system through the executive who had a close eye on government policy and the funding councils. the flow of policy initiatives was negotiated through this complex system in which power relations were often expressed through financial constraints. the ltsn provided an interesting example of how networks can provide a unique responsiveness to diverse interests and retain a degree of structure and control that makes them of interest to policy makers. the ltsn is a policy initiative that relies upon remaining close to its constituency. to be effective that strand of the initiative is critical. yet that requirement sets up a tension with the natural desire of the funding bodies and beyond them the government, to make such initiatives responsible for carrying through major items on the policy agenda (jones, 2002). network metaphors are also useful in relation to new and developing forms of management. a dutch school of political studies has moved from using network analysis to describe government to setting out some basic guidance on how networks can be managed (kickert et al., 1997). this type of analysis may be especially appropriate in higher education, which has multiple centres of power and influence. kickert et al. comment that “management in networks is about creating strategic consensus for joint action within a given setting” (p. 167). finding a common purpose for higher education is one of the great challenges in the midst of current social and technological changes, and for this an analysis of how managed networks operate is invaluable. networked learning a first wave of writing about the use of computer networks in education took place in the late 1980s and early 1990s with the emergence of such canonical texts as mindweave, the ‘virtual classroom’ and online education (mason & kaye, 1989; hiltz, 1994; harasim, 1990). the focus of these approaches was largely textual, emphasising the interactive nature of the technology and exploring how these could be used for distance education in particular. mason (1994, p. 25) commented that: no concept so characterises educational thinking in the 1990s as does interactivity… so embedded in the spirit of the age is it that there is relatively little questioning of its value, much less evaluation of its effects. interactivity altered its focus during this period from interacting with computers to interacting through computers (crook, 1994). in 1996, koschmann tried to periodise the developments in the context of research programmes (koschmann, 1996), identifying an ‘emerging paradigm’ that he called cscl. this emerging set of research questions was contrasted with previous outlooks, computer assisted instruction, intelligent tutoring systems and the ‘logo-as-latin’ paradigm associated with papert’s seminal text mindstorms. koschmann related the research within the cscl paradigm to socially oriented constructivist viewpoints such as the soviet sociocultural theories and theories of situated cognition. likening the paradigm shifts networks and learning: 89 he identified to a gestalt change in view, he claimed that the new paradigm drew attention to the social and cultural context of education as the object for research. in interviews with uk academics using networked learning, the paradigm identified by koschmann has been found to provide a high level philosophy informing the thinking of practitioners in uk higher education (jones et al., 2000; jones & asensio, 2002). in 2001 koschmann returned to this theme and examined the emergent paradigm of cscl in more detail: cscl research has the advantage of studying learning in settings in which learning is observably and accountably embedded in collaborative activity. our concern, therefore, is with the unfolding process of meaning-making within these settings, not so-called ‘learning outcomes’. it is in this way that cscl research represents a distinctive paradigm within it. by this standard, a study that attempted to explicate how learners jointly accomplished some form of new learning would be a case of cscl research, even if they were working in a setting that did not involve technological augmentation. on the other hand, a study that measured the effects of introducing some sort of cscl application on learning (defined in traditional ways) would not. i find this definition of the field unsatisfactory precisely because it fails to relate the research area to the particular features or affordances of the technology. i would suggest that the technological change surrounding the web, only in its infancy in 1996, has been the spur to generating another set of research questions in an area that i would describe as networked learning. the centre for studies of advanced learning technology (csalt) group at lancaster university and i have been associated with the following definition of networked learning: networked learning is learning in which information and communication technology (c&it) is used to promote connections: between one learner and other learners, between learners and tutors; between a learning community and its learning resources. this definition of networked learning is, like cscl, related to social theories of learning. it develops beyond cscl in moving away from defining the new paradigm simply in terms of the local processes of social learning. networked learning remains concerned with social process but it sets itself the task of understanding the links between different nodes, the learners, the tutors and the resources that make up a networked learning setting. the growth of the web and the drive to digitise and make accessible a range of reified ‘learning objects’ has made the research focus on social process, understood as group collaboration, too narrow. it is important now to understand the relationship of individuals and groups to artefacts as a sense making activity. the move from interaction with computers to interaction through computers has now moved on to interaction in relation to computer networks. the computer itself is no longer the centre of research attention; instead it is the network and network resources made available through the computer. indeed if mobile and ubiquitous computing develop strongly, the device through which the network is available may well cease to be recognisable as a computer. networked learning can take account of policy, organisational and whole institution issues that arise with the embedding of the new network technologies. networked learning is still concerned with meaning 90 c. jones making and the social process of learning but it is also concerned with the social and organisational dynamics within which those processes take place. research into networked learning as networked learning develops it will be increasingly important to develop an understanding of how traditional research approaches might transfer to networked environments. in my own field of interests there has been a growing interest in ‘virtual’ or ‘on-line’ ethnography. hine (2000) comments on some of the ways that the virtual is distinct from face to face settings, noting that networked technologies show “a high degree of interpretive flexibility” (p. 64). she argues that this flexibility derives from the way the technology is dependent upon different contexts and the way that the technology has to be “acquired, learnt, interpreted and incorporated into context” (p. 64). hine’s conclusion is that: virtual ethnography is adequate for the practical purpose of exploring the relations of mediated interaction, even if not quite the real thing in methodologically purist terms. (p. 65) wittel has also explored the move ‘from field to net to internet’ and has suggested a move to an ethnography of networks (2000). networks are still strongly related to geographical space—like field. unlike field, a network is an open structure, able to expand almost without limits and highly dynamic. and even more important: a network does not merely consist of a set of nodes but also a set of connections between nodes. as such, networks contain as much movement and flow as they contain residence and localities. an ethnography of networks would contain the examination of nodes of a net and the examination of the connections and flows (money, objects, people, ideas etc) between these nodes.[5] his conclusion is similar to hine’s, recommending a reshaping of the concept of field site to one focused on flow and connectivity. by continuing to place an emphasis on the node within a network, wittel’s formulation has the benefit of retaining some aspects of the idea of location within the expansion of the concept to cover flow and connectivity. conclusions i have argued that using the metaphor of networks can assist us to theorise the broad context in which learning and education take place in a society reliant on computer networks. i have not outlined a research programme associated with this view but there are within this article strong hints that such a research programme could and possibly should be elaborated. networks provide a useful focus for analysing the patterns of growth and interaction in a wide range of fields. the use of the network metaphor in learning technology helps us to connect ourselves to that wider debate and to theorising about the fundamental nature of the network and patterns of activity associated with it. the use of the network metaphor also links our work in learning technology with that concerning the network society. this has direct connections to networks and learning: 91 learning technology because it informs debates in higher education and training in terms of the type of workforce that awaits our current students and the type of skills they are likely to need. the networked society debate directly relates to the ideas of virtual communities and communities of practice. network theory suggests that the strong notions of community within these theoretical approaches might ignore the importance of the ‘strength of weak links’. networks can be structured in ways that combine both the benefits of random and organised patterns. the educational focus on strong links and community may have made less visible those necessary but weak connections that make the network idea so powerful. the idea of online communities also tends to separate out and privilege the virtual against the real. the ideas of online and offline communities need to merge in the activity of real people who are both simultaneously on and offline when they are engaged with computer networks. in relating networked learning to the networked society this paper has identified three areas that may be open to investigation using network analysis. policy, politics and management is one area in which network analysis is already well developed in ways that may be of direct use for researchers in learning technology. this is also an area in which the idea of networks can help social practice theories of social and situated learning connect themselves to the policy debate. the idea of clusters and hubs in hierarchical networks may allow us to develop a rich picture of just how it is that communities of practice relate to the wider policy networks in education. in terms of the current dominant paradigms influencing learning technologists, the network metaphor assists in shifting the emphasis towards the reifications deployed in networks alongside a focus on meaning-making. finally the paper raises the question of research methods. it raises the possibility that researchers may need to adjust and amplify their methods to take advantage of new possibilities and to translate classic research methods into the network setting. acknowledgements this paper is based on a research paper presented at alt-c 2003 and published in the conference’s research proceedings. references barabasi, a-l. (2002) linked: the new science of networks (cambridge, ma, perseus publishing). barabasi, a-l., dezso, z., ravasz, e., yook, s-h. & oltvai, z. (2002) scale-free and hierarchical structures in complex networks. available online: http://www.nd.edu/∼ networks/pdf/proceeding%20sitges2004.pdf (accessed 25 november 2003). barnett, r. (2000) realizing the university in an age of supercomplexity (buckingham, srhe/open university press). bell, d. (1974) the coming of post-industrial society: a venture in social forecasting (hamondsworth, penguin). brown, j., collins, a. & duguid, p. (1989) situated cognition and the culture of learning, educational researcher, 18(1), 32–42. 92 c. jones buchanan, m. (2002) nexus: small worlds and the groundbreaking science of networks (london, w.w. norton & co). castells, m. (1996) the information age: economy, society and culture. volume 1: the rise of the network society (oxford, blackwell). castells, m. (2001) the internet galaxy: reflections on the internet, business, and society (oxford, oxford university press). crook, c. (1994) computers and the collaborative experience of learning (london, routledge). fox, s. (2002a) networks and communities: an actor–network critique of ideas on community and implications for networked learning, in: s. banks, p. goodyear, v. hodgson & d. mcconnell (eds) networked learning 2002: a research based conference on e-learning in higher education and lifelong learning (sheffield, university of sheffield), 110–118. fox, s. (2002b) studying networked learning: some implications from socially situated learning theory and actor–network theory, in: c. steeples & c. jones (eds) networked learning: issues and perspectives (london, springer). harasim, l. (ed.). (1990) online education; perspectives on a new environment (new york, praeger). hiltz, s. (1994) the virtual classroom: learning without limits via computer networks (norwood, nj, ablex). hine, c. (2000). virtual ethnography (london, sage). hodgson, v. & reynolds, m. (2002) networked learning and ideas of community, in: s. banks, p. goodyear, v. hodgson & d. mcconnell (eds) networked learning 2002: a research-based conference on e-learning in higher education and lifelong learning (sheffield, university of sheffield), 119–127. hoggart, p. (1991) a new management in the public sector? policy and politics 19(4), 243–256. jones, c. & steeples, c. (2002) perspectives and issues in networked learning, in: c. steeples & c. jones (eds.) networked learning: perspectives and issues (london, springer). jones, c. (2000) from the sage on the stage to what exactly? description and the place of the moderator in cooperative and collaborative learning, in: d. squires, g. jacobs & g. conole (eds) the changing face of learning technology—selected papers from alt-j (cardiff, university of wales press). jones, c. (2002) is there a policy for networked learning? in: s. banks, p. goodyear, v. hodgson & d. mcconnell (eds) networked learning 2002: a research-based conference on e-learning in higher education and lifelong learning (sheffield, university of sheffield), 175–182. jones, c. & asensio, m. (2002) designs for networked learning in higher education: a phenomenographic investigation of practitioners’ accounts of design, in: c. steeples & c. jones (eds) networked learning: perspectives and issues (london, springer). jones, c., asensio, m. & goodyear, p. (2000) networked learning in higher education: practitioners’ perspectives, alt-j, 8(2), 18–28. kickert, w., klijn, e-h. & koppenjan, j. (1997) managing complex networks (london, sage). koschmann, t. (2001) revisiting the paradigms of instructional technology, in: g. kennedy, m. keppell, c. mcnaught & t. petrovic (eds) meeting at the crossroads: proceedings of the 18th annual conference of the australian society for computers in learning in tertiary education (melbourne, the university of melbourne), 15–22. available online: http:// www.medfac.unimelb.edu.au/ascilite2001/pdf/papers/koschmannt.pdf (accessed 28 february 2003). koschmann, t. (ed.) (1996) cscl: theory and practice of an emerging paradigm (mahwah, nj, lawrence erlbaum associates). lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation (cambridge, cambridge university press). law, j. (draft) networks, relations, cyborgs: on the social study of technology, centre for science studies and the department of sociology, lancaster university. available online: http:// www.comp.lancs.ac.uk/sociology/soc042jl.html (accessed 24 february 2003). networks and learning: 93 lyon, d. (1995) the roots of the information society idea, in: n. heap, r. thomas, g. einon, r. & h. mackay (eds) information technology and society: a reader (london, sage). mason, r., kaye, a. (eds) (1989) mindweave: communication, computers and distance education (oxford, pergamon). mason, r. (1994) using communications media in open and flexible learning (london, kogan page). morris, e. (2002) speech to bett conference available online: http://www.dfes.gov.uk/speeches/ index-shtml (accessed 12 march 2004). rheingold, h. (1993) the virtual community: homesteading on the electronic frontier (reading, ma, addison-wesley). rhodes, r. (1997) understanding governance: policy networks, governance, reflexivity and accountability (buckingham, open university press). toffler, a. (1980) the third wave (london, pan). trehan, k. & reynolds, m. (2002) online collaborative assessment: power relations and ‘critical learning’, in: c. steeples & c. jones (eds) networked learning: perspectives and issues (london, springer). watts, d. & strogatz, s. (1998) collective dynamics of ‘small world’ networks, nature, 393, 440–442. wenger, e. (1998) communities of practice: learning, meaning, and identity (cambridge, cambridge university press). wittel, (2000) ethnography on the move: from field to net to internet [23 paragraphs]. forum qualitative socialforschung/forum: qualitative research, 1(1). available online: http://qualitative-research.net/fqs (accessed 12 march 2004)l wolf, a. (2002) does education matter? myths about education and economic growth (london, penguin books). networks and learning: communities, practices and the metaphor of networks chris jones* 1. interdependence between organisations; 2. continuing interaction between network members; 3. game-like interactions, rooted in trust and regulated by rules; and 4. no sovereign authority, self-organising, with a limited state steer. the management of schools’ websites in cantabria, spain full terms & conditions of access and use can be found at http://www.tandfonline.com/action/journalinformation?journalcode=zrlt20 download by: [81.170.141.0] date: 07 june 2017, at: 05:25 research in learning technology issn: 2156-7069 (print) 2156-7077 (online) journal homepage: http://www.tandfonline.com/loi/zrlt20 the management of schools’ websites in cantabria, spain carmen álvarez álvarez & josé inés-garcía to cite this article: carmen álvarez álvarez & josé inés-garcía (2017) the management of schools’ websites in cantabria, spain, research in learning technology, 25:1, 1270579, doi: 10.1080/21567069.2017.1270579 to link to this article: http://dx.doi.org/10.1080/21567069.2017.1270579 © 2017 the author(s). published by informa uk limited, trading as taylor & francis group. published online: 13 jan 2017. submit your article to this journal article views: 2309 view related articles view crossmark data http://www.tandfonline.com/action/journalinformation?journalcode=zrlt20 http://www.tandfonline.com/loi/zrlt20 http://www.tandfonline.com/action/showcitformats?doi=10.1080/21567069.2017.1270579 http://dx.doi.org/10.1080/21567069.2017.1270579 http://www.tandfonline.com/action/authorsubmission?journalcode=zrlt20&show=instructions http://www.tandfonline.com/action/authorsubmission?journalcode=zrlt20&show=instructions http://www.tandfonline.com/doi/mlt/10.1080/21567069.2017.1270579 http://www.tandfonline.com/doi/mlt/10.1080/21567069.2017.1270579 http://crossmark.crossref.org/dialog/?doi=10.1080/21567069.2017.1270579&domain=pdf&date_stamp=2017-01-13 http://crossmark.crossref.org/dialog/?doi=10.1080/21567069.2017.1270579&domain=pdf&date_stamp=2017-01-13 research article the management of schools’ websites in cantabria, spain carmen álvarez álvarez and josé inés-garcía department of education, university of cantabria, santander, spain abstract school websites are not only sources of information and documentation, but also means of communication that are increasingly being accessed by members of society, especially by the different educational communities, thanks to the democratisation of internet access. this paper deals with a qualitative research project in which information and communication technology coordinators of infant and primary publicly funded private schools (concertados) and state schools in urban, rural and semi-urban/rural areas within the region of cantabria (spain) were interviewed in order to understand how school websites are managed. the findings show a range of views and practices in the schools’ management of their websites, some positive developments in their uses and functions, and numerous opportunities for improvement. school websites are essential communication tools, which need to be continuously improved/updated by their managers, with the support of the education authorities. article history received 23 february 2016 revised 24 october 2016 accepted 3 december 2016 responsible editor carmen álvarez álvarez, university of cantabria, spain keywords school website; ict coordinator; educational community; school improvement; spain 1. introduction in the information and communication society that we live in, it has been internalised that the internet provides the answer to almost all of our questions (castells, 2006). schools therefore face the challenge of satisfying the curiosity of online visitors through their institutional website (du preez, 2007; hartshorne, friedman, algozzine, & isibor, 2006; hartshorne, friedman, algozzine, & kaur, 2008; honiges, 2013; hu & soong, 2007; poock, 2005; stewart, jacob, & jensen, 2012; tamatea, hardy, & ninnes, 2008; tubin & klein, 2007). international studies have shown that school websites are gaining more and more interest, although further research into the subject is required (hartshorne et al., 2008; hu & soong, 2007; tubin & klein, 2007). school websites are being visited ever-more frequently, and in the future this trend will increase due to the greater levels of internet connectivity of stakeholders in education processes and society as a whole (hartshorne et al., 2006, 2008; hu & soong, 2007; tubin & klein, 2007). the person responsible for the school website is the information and communication technology (ict) coordinator (balas, mirea, & mirea, 2014; mcgarr & mcdonagh, 2013; rodríguez-miranda, pozuelos-estrada, & león-jariego, 2014). in the coordinator’s absence, that responsibility may be taken on by a member of the management team, whose main task is to ensure the use of the new technologies in the school. a study in contact carmen álvarez álvarez mcalvarezalvarez@gmail.com research in learning technology, 2017 vol. 25, no. 1, 1270579 http://dx.doi.org/10.1080/21567069.2017.1270579 © 2017 the author(s). published by informa uk limited, trading as taylor & francis group. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/ by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. http://orcid.org/0000-0002-8160-2286 http://www.tandfonline.com ireland conducted by mcgarr and mcdonagh (2013) showed that the ict coordinator plays an important role in tasks such as the implementation of ict in the curriculum and the maintenance of computer equipment within the school. a study carried out in spain by rodríguez-miranda et al. (2014), based on 101 ict coordinators in the andalusia region, revealed that their main roles are working with and guiding teachers in the use of ict, and enabling the use of digital content in the school, with the administration of the school website being their least important function. perhaps this explains why management of the school website has been the least researched of all the tasks performed by ict coordinators. however, this website management is an important task, as the site is visited both by stakeholders who are internal to the school (staff, families and students) and by those who are external (potential education professionals, potential families and students, and society in general), and it serves as the school’s showcase. in fact, a study by honiges (2013) on the image of schools in their communities showed that one of the strategies that promoted a positive image of these centres was the school website. the reason was that it enabled visitors to see if the school used technology in an innovative way, if it had a two-way relationship with the community, if it had links with social networks, and if it engaged in educational marketing. before families choose a school for their children, they are increasingly researching the different institutions, communicating with other families or teachers, and accessing the schools’ official websites. they consider the websites to be the most objective comparison tool between schools (cucchiara & horvat, 2014). in addition, if the sites are regularly updated, members of the school community regularly visit them, because they are useful tools for obtaining information on events, downloading forms, dealing with administrative procedures, answering questions, getting insights into teachers’ work on a regular basis, finding out about specific school events, etc. (hu & soong, 2007). while school websites have to achieve an undetermined range of roles and objectives (tubin & klein, 2007), they all have the challenge of providing truthful, useful, comprehensive, relevant and up-to-date information (du preez, 2007) for all kinds of people, including those with disabilities, in order to prevent informational exclusion (lomce, 2013)1. this challenge falls at the feet of the ict coordinator. however, as revealed by a study by rodrigo rodrigo, chacón peirats, san martín alonso, and gallardo fernández (2014), there is often a lack of prior training and poor recognition of the ict coordinator’s work, and they have little support from the management team and teachers in the school. in fact, for the design, development, implementation and evaluation of the website to be of high quality, the ict coordinator’s work should be based on leadership, trust and the distribution of responsibilities (de paula rodríguez & pozuelos estrada, 2009). updating and constantly improving a website can be an arduous task if it falls to a single person, especially if there is no feedback to help identify ways of improving it. today there are three major trends in official school sites: (1) a greater focus on the content, and a desire for thoroughness; (2) a focus on marketing, with the downside that a biased image of the school is sometimes provided (honiges, 2013; tamatea et al., 2008); and (3) a greater focus on communicating with visitors, which is the least developed trend so far (du preez, 2007; hartshorne et al., 2008; hu & soong, 2007). the ideal site should provide accurate, useful, comprehensive, relevant and up-todate information to potential visitors, both internal and external to the school (du preez, 2 c. álvarez álvarez and j. inés-garcía 2007; hu & soong, 2007); facilitate communication with them to the greatest extent through the web; and not overindulge in marketing. in the cantabria region, all publicly funded schools have at least one official web portal (generated by the educantabria platform, which is under the aegis of the spanish ministry of education, culture and sports), and sometimes another portal on a proprietary domain, or a blog that is managed independently. this illustrates the importance given by them to the internet: the schools in the region are in the process of opening up to society as a whole through their web presence. however, there are notable differences between the websites of the schools in the region. this study seeks to explore how these sites are being managed by schools and public authorities today. 2. method this paper describes a study on the management of websites of primary education schools supported by public funding in the cantabria region (known in spanish as comunidad autónoma or autonomous community). the objective was to obtain first-hand knowledge as to how websites were managed on a day-to-day basis by schools in the cantabria region, in order to ascertain the importance accorded to them. it seemed appropriate to design a study to become acquainted with some of the people responsible for managing school websites on a regular basis. this was intended to meet the overall objective of the study, as well as to discover what their training was, what importance they gave to websites as a means of communication, how they assessed the website in their particular school, and the resources provided by the comunidad autónoma. the lack of any prior qualitative studies of this kind served as an incentive to choose a methodology based on in-depth interviews conducted with key informants – that is, ict school coordinators and managers of educantabria, the educational portal whereby the regional authority provides virtual support to, and maintenance of, the official websites of schools that want to use it. potential participants were identified, contacted and asked to take part in, and consent to, a personal interview. all of the websites in the region were analysed (which resulted in another study still pending publication) and a number of relevant schools were identified and contacted. basically, this study consisted in identifying all of the comunidad autónoma websites (202 schools) in order to analyse their layout, organisation of information, content, usefulness and the role given to the educational community. the purpose was to see whether it was possible to assess the schools’ teaching methods and internal dynamics. as noted by hernández (2014), interview-based studies can be crucial to obtain further knowledge about issues on which little previous research has been conducted, such as the present one. the results obtained from interviews (shown in the next section) can contribute to generate academic knowledge, and are in line with the basic principles of grounded theory (charmaz, 2014). an emergent design was employed in the study. initially, six schools were selected as potential participants (and later contacted), with three requirements being taken into account: (1) that both public and publicly funded private (concertados) schools – primary and infant education – were represented; (2) that they served urban, rural and semiurban/rural environments, in order to better understand the role played by the research in learning technology 3 management of school websites in all types of schools in cantabria; and (3) that their school websites were diverse, in terms of both their quality and the extent to which they were kept up to date, so that various ict coordinators with different degrees of involvement in the development of their school website could be contacted. the researchers believed that six interviews would be sufficient to encompass all the issues of interest and have a highly diverse range of participants. some of the schools did not respond to our request for collaboration and therefore another school with the same characteristics was found to replace the one originally selected. once this data collection stage was completed, additional ict coordinators were contacted in order to provide further triangulation and saturation of data. in-depth interviews were therefore ultimately held with nine ict coordinators or managers and two managers of the educantabria portal. the ict coordinators and managers interviewed were distributed as follows: four worked in public schools (two urban, one semi-urban/rural and one rural) and five in publicly funded private schools (concertados) (two urban, two semi-urban/rural, 1 rural). the following abbreviations are used to distinguish between them: (p) public, (pfp) publicly funded private schools (concertados); (u) urban, (s) semi-urban/rural; (r) rural. information related to the identity of the participants and their respective schools cannot be disclosed in the study for ethical reasons, as this was a commitment made to the respondents. key codes were used to identify participants. the themes dealt with in the interviews were: (1) the people involved in the daily management of the school website, their training and their role. (2) the history of the school’s official website, its updating, how it currently works, and challenges for the future. (3) the school’s ict resources and its assessment of the educantabria portal. (4) sources of documentation for improving the school’s website, reports received from visitors and the school’s evaluation of its own website. in-depth interviews were carried out to exhaust all the themes under discussion. each of the interviews lasted approximately one hour, and consequently each of the relevant aspects was investigated to a considerable extent. all the interviews were recorded, and later transcribed and analysed using content analysis procedures to ensure that the participants’ statements would be accurately represented and achieve a high level of rigour. all the participants were informed of the ethical considerations of the study, in particular regarding the anonymous use of their data in both the analysis and dissemination stages of the study. the results were largely consistent from the first to the last interview. therefore, triangulation of statements, individuals and schools was achieved. once the first six interviews had been conducted, the results started to become repetitive. it was decided that three additional interviews would be carried out to ensure the consistency and reliability of the results and provide saturated data. after the nine interviews had been completed, it was considered that no additional ones were required. the transcripts were analysed using content analysis procedures. an original system of categories was thus created, based on the four major questions asked in the interviews. 4 c. álvarez álvarez and j. inés-garcía 3. results the data collected shows that there are significant differences in the management of school websites in cantabria. in order to account for the situation, the data was organised around the following themes: (1) managers of school websites, who deal with the different situations concerning the website management as noted by the ict coordinators; (2) the websites at the initial stages and their current status, focusing on their development and, particularly, their current status; and (3) the improvement of school websites, with an emphasis on the main challenges faced by schools in optimising their websites. 3.1. managers of school websites the website managers are usually the ict coordinators or managers, depending on the school’s internal dynamics. however, while management of the school website often falls to them, most schools try to seek support among teachers (and sometimes students) to update, maintain and provide answers to all members of the educational community: i keep it up to date, but all of the tutors send me things they do on a daily basis. they take photos, write news items and send them to me for uploading. everybody helps, and now we have children from year 6 helping. they also help. i upload something every day. (ict coordinator, p.r.) this does not always happen, and one individual may end up assuming the onerous duty of managing a website, which is perceived as something tedious that entails great responsibility and can cause problems, such as replacing the person who performs these tasks: i’m trying to keep things going collaboratively and to let go of it a bit. i have six years to go until i retire. every so often i say, ‘if anyone wants to take this on, i won’t object, i’ll help this person as much as needed and i’ll take a step back to concentrate on teaching.’ but nobody wants to get into this quagmire. (ict coordinator, p.u.1) there are also cases of schools where, as no teacher would take care of the ict coordinator role, the head teacher had to take it on, despite lacking the training and the time to do it, with negative consequences for the development of the website. there is a website that is there, but it is not up-to-date or anything. it is not managed by anyone on a daily basis. if i were the head teacher of another school and had time to develop the website, i would manage it, but i have many more things to do, so i don’t. the school doesn’t have an ict coordinator because these are volunteer positions that require training, and you cannot force people to be a coordinator of anything. (ict coordinator, p. u.2) management of the website may also fall to a group of teachers or a company, especially at the beginning, due to a lack of teachers with the training to develop websites. this involves a financial expense for the school until they can manage it independently. a change towards independent management has been made possible thanks to the ease of new website structures and formats, similar to blog posts, which require less knowledge and simplify the work of keeping the website up to date. ‘we set research in learning technology 5 the website up earlier this year. two teachers who are the most knowledgeable about ict and are very creative were very keen and spent a long time with the ict expert’ (ict coordinator, p.s.). in the case of publicly funded private schools, the website is not always the responsibility of the school, but of a religious congregation. it is updated more sporadically, which causes systematic obsolescence and a lack of response to the current needs of the educational communities, who are accustomed to accessing online information. our website is not ours, we do not manage it; it belongs to the congregation. every year we send what we think should be on the website to the web coordinator and it is uploaded. it is a joint website for all the schools in the congregation worldwide. the last thing we uploaded was from the end of last year. we sent it to the coordinator and he put it up. i am the ict coordinator of the school, but i take care of other things (ict coordinator, pfp.r.). all the participants noted that no special skills or knowledge are required for managing a website. today, school websites are very intuitive to update, as internally they are very similar to blogs. for schools that use the educantabria portal to host their websites, training courses are on offer to ict coordinators, and there are tutorials available on the website. since 2013 there has been training for schools that wish to experiment with the new platform, and on the website itself we have a lot of videos where we have been doing the processes, and we’ve been recording them so people can teach themselves how to put a blog up, how to design a website, how the different areas are changed on the page, how it is divided into sections … (educantabria advisor 1) however, websites are often a secondary concern for those responsible for their upkeep. i think the ict coordinator gives a lot of importance to the application of technology in the classroom and has very few hours available to do it. for the ict coordinator, school website work is a minor task, compared to other duties. because he has to ensure that students have their computers ready when they go to the classroom, they have the relevant educational applications available, advise a colleague on how to make their class webpage, etc. (educantabria advisor 2) 3.2. websites at the initial stages and their current status the first school website in cantabria began in 1984 in an urban state school. however, websites only became popular later. between 2005 and 2007, the department of education of cantabria launched the first tool to enable all schools to create their own website. this has been under continuous development and culminated in the educantabria portal, which is a much more advanced, comprehensive, complex and secure tool. the initial stages of school websites were difficult in all cases: it tools were not advanced and programming languages required users to be trained, processes were much slower, etc. when we started to work on websites i had no idea. it was when hardly anyone had seen a computer, they were connected to a basic telephone line. at the time there was a lot of awareness in the school about ict and when the first websites came out we said ‘let’s see 6 c. álvarez álvarez and j. inés-garcía how his goes’, and we did one that was very static, with poor interaction options…, a bit like a website 1.0. (ict coordinator, p.u.1) the first web portals made schools very dependent on companies to maintain them, but this is not the case anymore. before this website we have now there was one made in flash and a company was hired. it was completely static and also every time you had to make a change, it had to be made by the company, we had to talk to the company, and pay, it wasn’t practical… this was the type of websites people used in the past, but now we want them to be more practical. (ict coordinator, pfp.s.2) currently, all the schools in cantabria have their own school website in the educantabria portal and only need to keep it updated; alternatively, they may choose to create one on a private domain or use a blog. maintaining this type of website format is fairly intuitive, so there is no need to be a computer expert or rely on a company. having a full school website today is very simple. you enter by clicking on ‘access’ with your username and password and edit what you want. it is very easy. it’s like a blog. it is very simple. it isn’t complicated at all. (ict coordinator, pfp.u.1). however, there are many challenges for the improvement of school websites, as will be discussed in the next section. currently, websites primarily serve two types of purposes: information and marketing. all respondents stated that the main role of official websites is to inform. our website is mainly intended for information purposes, although we do a lot that we do not show on it. the website could be much more dynamic, it could have more news about everything we do. sometimes the department of education releases projects we are already working on and then we realise that we don’t really display everything we do, and we should do it. (ict coordinator, pfp.u.1) nevertheless, the ict coordinators of the two publicly funded private schools (concertados) who were interviewed made it clear that the website is an excellent showcase, and looking after it ensures that new families will be keen to enrol their children in the school. it is intended to make the school known, to ensure that parents send their children here and not to another school, to promote enrolment. we also want to introduce people to the vocational training courses, because almost everyone here knows that this is a school, but few people know that there is vocational training. we always try to release information about the vocational courses to inform people about them. (ict coordinator, cs2) those schools that update their websites frequently are enthusiastic about their websites’ potential, and are motivated to continue to improve them. during the enrolment period, school websites become particularly important, because they are used to obtain information, which can be decisive in terms of decisionmaking together with discussions with other families that have students enrolled in the school. one way to know is word-of-mouth and another one is reviewing websites. then of course, they look at the website and say ‘your website is nothing like the others i’ve seen around, research in learning technology 7 this one provided proper information’. and indeed, there are new students who surprise you with the knowledge they already have of the school. (ict coordinator, p.u.1) students also use websites when they provide information that interests them; for example, curriculum content, learning proposals and/or complementary activities or photographs and contributions from other students. ‘the students mainly visit the classroom blogs that are linked to [the website]. they very often look at this. for example, [they use] the english blogs … in class and at home’ (ict coordinator, pfp. u.1). the same applies to the authority. ‘the inspector has also been reviewing it regularly because she always tells us, “you have put this up, that’s great, you have put the other thing up, you can put this up…”’ (ict coordinator, p.r.). ‘many families have no idea what is being done, and now as they access the website and see it, they like it. many people are visiting it, and that’s something we did not expect’ (ict coordinator, pfp.u.2). 3.3. improvement of school websites most schools face many challenges in managing their websites. one of these challenges pertains to providing information about the school’s activities in broad terms, to meet any user’s demands for information. ‘i think we could promote some aspects, some new plans or projects that we launched this year. we are developing department of education plans and projects and it is good that people are a little more aware of this’ (ict coordinator, c.u.2). similarly, the websites must provide services to users that are as broad as possible, and that go beyond simply informing: the challenge is to enable two-way communication. we have created an app that gives information about our educational project, location, etc. it enables telephone calls to the school to be made from the app, [and allows people] to see a map to get to the school, to report incidents or absences, etc. we also have many online forms that are very useful. for example, we have a robinson list for families who want to sign up to receive information by e-mail only, not on paper. so we save paper and gain immediacy, and 65% of families are already benefiting from the service. (ict coordinator, p.u.1) a continuing challenge for any school is to update their website frequently. one of the things that i was very clear about when i became a head teacher was that i wanted to renew the website, i wanted it to be up-to-date and dynamic. the fact that you access a website and see last summer’s trips makes you feel that it is not cared for, it is not monitored and it is useless; that you have it up just because you have to… since we have had a couple of people involved in ict and the website, it is constantly updated. (ict coordinator, pfp.u.2) although most schools are far from achieving continuous updating, the most forwardlooking ones are clearly aware that progress has to be made, as they see new advantages, such as the usefulness of conveying the activities undertaken each term and making education more visible. the news section is updated constantly. not the contents menu, as we’re involved in a restructuring process. that is where we take the information for the annual magazine. what 8 c. álvarez álvarez and j. inés-garcía is not published doesn’t exist, and we want everything we do to be visible to the educational community and all those who have come in contact with our website. (ict coordinator, c.s.1) another challenge related to websites relates to the need to strongly promote interaction mechanisms with users, to enable the website to be a cutting-edge communicative tool that goes beyond being a mere one-way channel of information from schools to society. however, both the schools and the education authority have doubts about the best way to move forward in this regard without causing problems. we could have forums on the websites, but in a forum anyone can write the most outrageous things, and unless you have someone constantly moderating the forum, it reaches everyone and that’s a problem. the interactive part is good because everyone can see, but not controlling it is a risk. that is why i put an email contact on the website, and not a forum. (educantabria advisor 1) an increasing number of educational institutions prefer to use social networks such as facebook, twitter and instagram, which are perceived to open up many opportunities for schools. i think that facebook is very effective. many people do not visit the school website, but follow us on facebook, and having an instagram account means that we are able to show the photography contests or the library initiatives. now the next step is to for parents and students who follow us on facebook to make comments and contributions. (ict coordinator, p.s.) however, an aspect of great significance in the improvement of school websites, partly as a result of the lack of research on the subject in spain, is the lack of reference points for schools to look to in terms of website design. during the design process, ict coordinators tend to look at other sites, generally randomly, for inspiration. but once the basic structure has been designed, all the ict coordinators said that it is then left to intuition and improvisation. the lack of scientifically substantiated references from national or international publications on how to further the improvement of websites reduces their enhancement opportunities. when we designed the website, we looked at a lot of pages to see what buttons we wanted, and we chose from among several options, but now improving the website is something of an intuitive process. we upload what we think best. we are not inspired by anything. it is all intuition, we upload what we consider appropriate using our own judgement. (ict coordinator, pfp.s. 2) in this sense, only self-criticism and reflection on the progress made, as well as requests for information made by external agents to the schools, serve to provide some guidelines, which are inevitably blurred, for action in the improvement of this tool. 4. conclusions the results lead to the following conclusions about managers of school websites, the general development of websites and the challenges of school websites. first, it was found that the managers of school websites vary greatly in their levels of involvement, which has an impact on their quality. overall, participants expressed a research in learning technology 9 slightly higher interest in the use of websites in publicly funded private urban schools, as websites are deemed to be used for informative and marketing purposes, and play a part in attracting new students (honiges, 2013). all of the collected statements showed that managing a school website today is easy and does not require specific sophisticated training, although it does take time and dedication. a barrier in the development of this tool could be the very identity of ict coordinators, who see themselves more as support for their colleagues in the implementation of the ict curriculum than as the institution’s website managers, with the latter task taking second place (rodríguez-miranda et al., 2014). additionally, very different management practices were found depending on the people who were responsible for the website; that is, whether there was a single leader, several leaders, collaboration with a company, dependence on a larger home website such as that of a religious congregation, etc. all of this suggests that for a website to remain up to date and thorough, and provide useful information, the collaboration and cooperation of everyone in the school’s educational community is needed; that is, the management team, teaching staff, parents’ association, partner organisations, etc. it is very difficult to maintain an up-to-date and comprehensive website if a single individual is responsible for it (ict coordinator or manager). in this sense, it is necessary to give more recognition to the work of ict coordinators and encourage schools not to entrust management of the website to a single individual (rodrigo rodrigo et al., 2014). it would also be useful for schools to increase the hours spent on developing their website and coordinating the ict curriculum. it must be noted that the general development of websites has been mainly positive. the initial stages were hard: there was large dependence on computer experts and the resulting websites were static. however, today’s websites are much easier to create and maintain and the educational authority – with its educantabria format – provides technological and training support. this does not mean that managers of the websites do not need to continue working to ensure that the websites are thorough and fully up to date, but that the current situation, compared with the past, is highly positive (du preez, 2007). the various public administrations, therefore, need to continue to generate systems for the training and support of schools. schools are benefiting from the information and marketing opportunities offered by websites, but they should also begin to explore becoming more interactive, for which forums or social networks could be of help. at present there are few schools that use these networks; however, it is becoming increasingly common for citizens to have social networking profiles and to look at them daily. the use of these tools could contribute greatly to the improvement of communication in schools, not only with the educational community, but also with others who are connected to the internet. some pioneering schools have already created profiles on facebook, twitter and instagram, which they use to post up-to-date information about school life, and their number of followers increases on a daily basis. additionally, as website managers themselves have said, when they create these profiles, or update their websites, they gather new followers, which motivates ict coordinators to further improve their web tools and implement new ideas and content, creating a positive feedback loop that benefits all users. it can be concluded that the management of school websites is fraught with challenges and opportunities for improvement, namely giving full information about the 10 c. álvarez álvarez and j. inés-garcía school’s activities in order to fulfil the information needs of all users; providing services to users beyond the merely informative purposes; providing regular updates; generating new avenues of communication with the community of visitors, etc. from all these challenges, one seems particularly problematic and relevant: conducting further research on school websites in spain. the lack of national reference points, and the difficulty in accessing international models, which are often in english, leave little room for schools and ict coordinators to professionally improve this tool, which means they miss out on opportunities for enhancement, leaving them over-reliant on intuition and improvisation. the current information society demands that schools deliver quality not only academically, but also on their web portals. in the future, it would be of interest to carry out research specifically on the best websites in our region, by selecting 10–20 of them and conducting interviews with ict coordinators to identify how they managed to create high-quality websites. note 1. spanish general act of parliament on the improvement of quality in education. disclosure statement no potential conflict of interest was reported by the authors. orcid carmen álvarez álvarez http://orcid.org/0000-0002-8160-2286 references balas, e., mirea, d., & mirea, l. (2014). the ict coordinator in school a xxi century education. journal plus education, 10(1), 312–317. castells, m. (2006). la sociedad red [the network society]. madrid: alianza. charmaz, k. (2014). constructing grounded theory. london: sage. cucchiara, m., & horvat, e.m. (2014). choosing selves: the salience of parental identity in the school choice process. journal of education policy, 29(4), 486–509. doi:10.1080/ 02680939.2013.849760 de paula rodríguez, f., & pozuelos estrada, f.j. (2009). contributions on the development of teachers’ training in ict schools. case study. pixel-bit: media and education journal, 35, 33–43. du preez, h. (2007). issues to consider during the development and promotion of a primary school web site. pretoria, south africa: university of pretoria. hartshorne, r., friedman, a., algozzine, b., & isibor, t. (2006). secondary schools online: are high school web sites effective? american secondary education, 34(2), 50–66. hartshorne, r., friedman, a., algozzine, b., & kaur, d. (2008). analysis of elementary school web sites. educational technology & society, 11(1), 291–303. hernández, r.m. (2014). qualitative research through interviews. its analysis by grounded theory. pedagogical issues:journal of educational sciences, 23, 187–210. honiges, a. (2013). school image in the context of new communication technology. knowledge horizons-economics, 5(2), 162–166. hu, c., & soong, a.k. (2007). beyond electronic brochures: an analysis of singapore primary school web sites. educational media international, 44(1), 33–42. doi:10.1080/09523980600922761 research in learning technology 11 http://dx.doi.org/10.1080/02680939.2013.849760 http://dx.doi.org/10.1080/02680939.2013.849760 http://dx.doi.org/10.1080/09523980600922761 lomce. (2013). organic act 8/2013, of december 19th, for quality improvement of education, boe of december 10th, 2013. mcgarr, o., & mcdonagh, a. (2013). examining the role of the ict coordinator in irish post-primary schools. technology, pedagogy and education, 22(2), 267–282. doi:10.1080/ 1475939x.2012.755132 poock, m. (2005). determining the design of effective graduate school web sites. college and university journal, 80(3), 23–26. rodrigo rodrigo, m., chacón peirats, j., san, martín, &, á., & gallardo fernández, i. (2014). perceptions around the ict coordinator in smart schools. a case study. to educate, 50(1),167– 184. rodríguez-miranda, f., pozuelos-estrada, f.j., & león-jariego, j. (2014). the role of ict coordinator. priority and time dedicated to professional functions. computers & education, 72, 262–270. doi:10.1016/j.compedu.2013.11.009 stewart, t., jacob, a., & jensen, l. (2012). school site visits: what can we learn from choice schools in milwaukee. milwaukee evaluation report, 34. tamatea, l., hardy, j., & ninnes, p. (2008). paradoxical inscriptions of global subjects: critical discourse analysis of international schools’ websites in the asia-pacific region. critical studies in education, 49(2), 157–170. doi:10.1080/17508480802040241 tubin, d., & klein, s. (2007). designing a school website: contents, structure, and responsiveness. planning and changing, 38(3–4), 191–207. 12 c. álvarez álvarez and j. inés-garcía http://dx.doi.org/10.1080/1475939x.2012.755132 http://dx.doi.org/10.1080/1475939x.2012.755132 http://dx.doi.org/10.1016/j.compedu.2013.11.009 http://dx.doi.org/10.1080/17508480802040241 abstract 1. introduction 2. method 3. results 3.1. managers of school websites 3.2. websites at the initial stages and their current status 3.3. improvement of school websites 4. conclusions note disclosure statement references alt-j, research in learning technology vol. 12, no. 3, september 2004 issn 0968–7769 (print)/issn 1741–1629 (online)/04/030231–18 © 2004 association for learning technology doi: 10.1080/0968776042000259555 easing the transition from paper to screen: an evaluatory framework for caa migration mhairi mcalpine* scottish qualifications authority, glasgow, uk taylor and francis ltdcalt120304.sgm10.1080/0968776042000259555alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2004association for learning technology123000000september 2004mhairimcalpinescottish qualifications agencyglasgowmhairi.mcalpine@sqa.org.uk computer assisted assessment is becoming more and more common through further and higher education. there is some debate about how easy it will be to migrate current assessment practice to a computer enhanced format and how items which are currently re-used for formative purposes may be adapted to be presented online. this paper proposes an evaluatory framework to assess and enhance the practicability of large-scale caa migration for existing items and assessments. the framework can also be used as a tool for exposing compromises between delivery mechanism and validity—exposing the limits of validity of modified paper based assessments and highlighting the crucial areas for transformative assessments. background all holders of assessment materials are currently investigating what the impact of ict will be on their future practice. there is an acknowledgement that the traditional manner of assessment will change, however as yet no clear vision of what will replace it. bennett (2001) outlined the major changes that assessment would undergo in response to the changing technology. ripley (2003) has built on this idea to refine the three models of change which will dominate in the migration from paper-based to screen-based assessment. figure 1 gives an illustrated summary of the ripley model. figure 1. the ripley model there is, however, an issue in how we get from a mass system of testing to an individualized assessment structure, quite apart from the changes that the introduction of ict will create. while the introduction of ict is, for many, a time to rip up the rule book and start again—there is a need to take the practitioners along with the technology, starting from where we are now and introducing change slowly and incrementally * scottish qualifications authority, glasgow, uk. email: mhairi.mcalpine@sqa.org.uk 232 m. mcalpine f ig u re 1 . t h e r ip le y m o d el an evaluatory framework for caa migration 233 and ensuring that the cultural shift between paper-based and computer-assisted assessment is supported—and above all that the confidence in assessment systems is retained. this paper looks at how existing systems of assessment and collections of items can be put online and how these can be evaluated to appreciate the difficulties and challenges that this transition presents. by acknowledging where compromises have been made and differences created, there is an awareness of the limitations of the technology. this can expose where validity risks are being compromised for the sake of the assessment format. this proposed framework also can be used as a means to prioritize developments and make explicit where the likely challenges in certain types of developments may be. used in this manner, it can be an important tool for implementers of large-scale caa developments to manage change from paper-based to on-screen delivery. most uk developments are currently in the first or second stages of the transition process, looking at how they can adapt their current assessment practice to an onscreen delivery format. although this might seem a little un-ambitious, especially when compared with the more radical online assessment methodologies being developed, it is a necessary evolutionary step to engender confidence in computer delivery without too much of a radical change in the assessment format. the scottish qualifications authority is currently in the process of exploring the potential of caa (mcalpine & ware, 2003). it is anxious to avoid the fragmented approach that mckenna and bull (2000) report has characterized the development of caa in higher education in the uk and the resulting difficulty in achieving sustained systematic innovation across the education system. to that end it has actively sought partnership in its caa activities with its stakeholders, and is involved in setting up the infrastructure which underpins caa systems, and putting in place the processes of change which will ease the transition for all involved. we work in partnership with our centres and the rest of the scottish educational community and are keen that we are aware of what we are expecting of them through this time of innovation, and are doing all that we can to support them as they make this transition with us. introduction six subjects were chosen to form the medium of this pilot. these were english, maths, a science (chemistry), a humanity (history), an art (music) and a modern foreign language (french). all were chosen from the external assessment component of the higher. these are summative terminal assessments which are typically done by more able students in their fifth year of high school in what is considered the main determinant of entry to scottish universities. the question papers varied in a number of respects: the time allocated to the examination; the number of questions; the response that the questions demanded; the appearance of items, and the supporting material associated with the papers. it was considered that the major change issues with a move from paper based to on-screen 234 m. mcalpine assessment would be the response type that an item in the paper expected, and the inclusion of any stimulus material which is currently given on paper. each item in each of the question papers was considered—looking at any stimulus associated with the items, the type of response input that the item required, and the type of marking required by the item. classification of ease of migration in order to facilitate decisions of which types of items should be considered suitable for straightforward translation, which for modification and which should be considered from a transformative standpoint, a coding was established to identify which types of responses; input mechanisms and stimulus material were able to be directly translated into an computer based format. a classification scheme was developed to identify the extent of the challenge (table 1). as can be seen from table 1, classes one and two are most suitable for direct onscreen migration, classes three and four were possible to implement with some consideration, but may well be more suited to a modified form, while classes five and six were not available for direct translation and may require the kind of third stage transformative work. table 1. classification of ease of migration 1 currently widely available there should be only trivial issues to resolve. immediate implementation is feasible. 2 currently available, but requires refinement some minor decisions may have to be taken about how exactly it is implemented. immediate implementation is possible, however small amounts of work, or consideration of issues may have to be given to ensure long-term success. 3 currently available, but needs development for operational use substantial decisions may have to be made about the technology used or the manner in which it is implemented. it may require investment to ensure that it is of the standard which we would require. 4 limited availability and requires development decisions would have to be made about how it is developed and to what extent. implementation will not be possible until the development is complete. it will require some investment for operational use. 5 potential availability with commitment to development this technology may well be at the beta stage or only available as a trial version. substantial decision would have to be made about how exactly it is implemented and in what form, it will require investment both to finalize the technology and to make it operationally available. 6 not currently available without significant commitment and investment there is no reliable method of doing this at the present time. experimental projects are at an early stage or have not reached satisfactory conclusions. in order to implement this operationally, significant resources would have to be deployed to ensure its success. an evaluatory framework for caa migration 235 description and classification of items by attribute stimulus from the papers selected, five types of stimulus material were identified: diagram/ graphs; photo/drawing; quote; aural cue and formula. each question was classified according to which type of stimulus was associated with it, the most numerous minority (43.7%) of the items had no stimulus material associated with it, while only one question had more than one associated stimulus (table 2). each stimulus code was taken in turn to consider how difficult it would be to migrate that type of stimulus to a computer format. table 3 details this classification together with some analysis of how it was reached. issues which were considered included how difficult it would be to present through computer, how difficult it would be to access it, how candidates with special needs might be affected by this method of delivery, any special pieces of software which might be required to enable this, what the ‘industry’ tended to use for the delivery of this type of material, and alternative ways that it might be presented, including some evaluation of these methods.1 to construct this classification, a number of approaches were used, including a literature review, consideration of software known to the author and consultations with those involved with practical projects involving caa. it does not claim however to be a definitive account of all available technologies. response type from the papers selected, six types of response type were identified: numeric responses; algebraic responses; lexical responses; diagrammatic responses; closed responses and selected responses. each question was classified according to which type of response was expected from it. the most numerous grouping of responses were lexical responses (class 3), accounting for 59.6% of the responses required. categories two and three were further divided by the length of response expected, as in these response types that is a significant factor influencing the ability of the computer to automatically mark the item (table 4). as with stimulus, each response type was taken in turn to consider how difficult it would be to migrate that type to a computer format. two issues were considered, how difficult the response type was for a candidate to enter an answer into table 2. count of stimulus by code stimulus code 0 1 2 3 4 5 total counta 131 41 6 84 31 8 300 % of items 43.7 13.7 2.0 28.0 10.3 2.7 100 % of items with stimulus 24.2 3.6 49.7 18.3 4.7 100 a note that one question had both a photo and quote stimulus. 236 m. mcalpine table 3. description and classification of stimulus types code and description stimulus classification code 1: diagram/graph items with images where the information in the stimulus was essential to the answering of the question—thus an accurate reproduction of the image would need to be rendered on computer in order not to disadvantage candidates. class 2 there are a number of standard image formats available that diagrams and graphs can be rendered in. most caa engines and vles will accept these forms, however the display mechanism may cause subtle variations, which may affect question quality. furthermore the capabilities of the machines which the candidates attempt the question may affect the rendering. code 2: photo/drawing items with images where the information in the stimulus was impressionistic on the answering of the question. thus so long as the image was visible and retained its meaning it would be an acceptable rendering. class 1 there are a number of standard image formats available that photos and drawings can be rendered in. most caa engines and vles will accept these forms. again the capabilities of the machines which the candidates attempt the question on may affect the rendering—and although this might not be so critical as in the above example, it may bias results towards better resourced centres and candidates. code 3: quote items with text stimulus of a few words to a sentence or two. as with the above this could be rendered as an image, however it is assumed that the underlying coding style is textual. class 1 most caa engines and vles will accept textual stimulus material. code 4: aural cue items which required candidates to listen to something before responding. class 3 there are a number of rendering mechanisms for aural stimulus material, however the implementation of this into live examinations is surrounded by technical and practical issues which would have to be resolved prior to live use. some of these issues include ● the sound quality which, as with the image rendering, may be affected by the specification of the machine on which the candidate is being examined; ● the candidates control over the music playing – can they play it themselves, or would the computer play it for them in the manner that the invigilator currently does; an evaluatory framework for caa migration 237 the computer, and how difficult it was to enable automatic marking of that type of question. table 4 details these classifications together with some analysis of how they were reached. issues which were considered for input purposes included special characters, free-input, specialist notation and some consideration of accessibility issues. issues which were considered for marking purposes included the availability of technology that could enable computer-based marking of these types of questions, any minor changes that could be made to the questions to make them easier to mark on computer, and the reliability of the marking. as with stimulus, it does not claim to be a definitive account of all available technologies, but based on existing practice, known issues, currently available software and published literature. visualizing the papers using the stimulus class and the highest of the marking class and input class, a view of how easy each of the papers would be to migrate to caa were established. a short consultation exercise in which method of visualisation could most easily communicate the essential information indicated that bubble graphs were favoured over the two alternative methods put forward (stacked area graphs and luminosity squares). table 3. continued code and description stimulus classification ● how candidates might have access to the aural cues at different times without disturbing other candidates in the room; ● may a computer be able to get round some of the problems that candidates with sen may have in accessing certain part of the examination? (e.g. through increased amplification etc.). code 8: formula this code was used to demarcate stimulus which was presented in a standard subject specific form (in this case using mathematical notation and chemical notation). these could, in theory at least, be presented as an image and fall into code 1, however essential information would be lost which should be retained to maximize the usage of the question (not least in question generation). class 5 there are a number of ways that chemical and mathematical formulae can be represented on computer. these include latex; mathml and chemml as well as a variety of plug-ins. none of these, (except latex, which is an imperfect partial solution), can be adequately rendered on the majority of caa engines or vles—this would cause a significant problem should the meaning behind the formulae have to be retained. 238 m. mcalpine table 4. description and classification of response types code and descriptions input classification marking classification code 1: numeric items where a numerical answer was required. class 1 in the case of most numbers, input is fairly straightforward and can be done using standard notation on a standard keyboard—additional characters which would be required in addition to digits would be ‘-’ (negative numbers) ‘/’ (fractions), ‘.’ (decimals), ‘i’ (imaginary and complex numbers), ‘e’ (2.11) and ‘π’ (3.14). the only one which causes significant issues and is not found on standard keyboard is ‘π’. most caa engines accept numeric data, input issues should be minimal. class 2 there are a variety of ways that numerical questions may be marked. sometimes a precise answer is required (e.g. what is 1/2 expressed as a decimal? ans = 0.5 only) and other times more than one representation may be acceptable (e.g. if 8 cakes are shared among 10 people how many do each get? ans = 0.8 or 4/5 or 8/10) thus there would have to be the facilities available for the evaluation of the answer and an understanding of numerically equivalent forms as well as the facility to limit the acceptance of equivalent forms in certain cases. there are a number of caa systems which have both of these capabilities and although tweaking them to the precise requirements may require some work, this should not be a significant limiting factor. code 2: algebraic items where an algebraic answer (i.e. one including unknowns, typically represented by letters) was required. for marking purposes it is separated into sections 2a which requires only one line of input (the answer expected class 3 where unknowns are represented by letters as is common—this should not pose a problem as they are found on a standard keyboard. at lower levels unknowns may be represented in other forms (e.g. stars or question marks) which may prove more challenging. algebraic answers can quickly become complex and may require the whole range of algebraic notation available. this might include (but not limited to) complicated fractions, integrals, logs and powers. there are a number of ways of 2a—simple algebraic: class 3 as with numeric answers, algebraic answers may have a number of acceptable equivalent representations however the question may limit the number of acceptable forms. accepting too many equivalents may compromise the question, especially where the question is designed to test their ability to manipulate algebraic expressions. thus, as with the numeric questions, there would have to be the facilities available for evaluation of the answer and an understanding of numerically equivalent forms as well as the facility an evaluatory framework for caa migration 239 table 4. continued code and descriptions input classification marking classification would be a formula) and 2b those which would require more than one line of input (the expected answer would be a proof). inputting these as well as standard text input, such as selection from a menu (as in ms word equation editor); special codes (as in latex); or typing in characters in an appropriate order. many caa engines are able to accept algebraic data, however the quality of their input mechanisms vary (especially for more complex expressions) and input issues may challenge validity if they are not adequately considered. to limit the acceptance of equivalent forms in certain cases. where input issues were affecting the quality of the answers, there would have to be recognition of common errors which had been caused by input difficulties. this would best be recognized at the input stage – thus it might require an intermediate evaluation of the answer given checking for common input errors (e.g. x2 computer asks if that is x2; 2x or ‘times 2’). class 2b—proof: class 4 there are a variety of packages available which allow for input of algebraic expressions longer than one line, in some cases however this would have to be coded as separate answers. the cue system in use with the pass-it trials, allows supporting ‘steps’ to be accessed and used when candidates request them (this would be of particular relevance in the case of codes 1a and 2a where the answer itself is of a different form, however the proof may allow access to the partial credit available)—although it could be effectively insisted upon. there would be a number of difficulties associated with proofs, particularly as there may well be no one correct proof, but a variety of answers which may legitimately gain the available marks. mccabe (2001) has suggested the objectification of proof questions to assess this type of learning and suggests a variety of ways which various caa engines have approached this. all in all, this would be a problematic area and one which would require further consideration. 240 m. mcalpine table 4. continued code and descriptions input classification marking classification code 3 textual response items where the response expected was textual. this may also include non-lexical answers (such as a telephone number or date). the code is divided up into five subsections dependant on the length of response expected for analysing the potential for computer enhanced marking. code 2a examines single word responses (which may on occasions cover a short phrase or nonlexical responses such as a date), code 2b covers short responses, ranging from a few words to a sentence; code 2c covers expected responses ranging from a sentence to a paragraph. code 2d, codes extended responses where between one and three would be expected and code 3e covers class 1 the input mechanisms for this response type would be fairly straightforward—involving the standard characters on the keyboard—although this may include numbers and other characters (such as ‘£’; ‘/’ etc.). this would be accepted by all caa engines—although any unusual characters which were likely to be used in an assessment would have to be flagged and considered. as the length of response demanded grows it would have to be considered whether there were sufficient elements in place in the case of a systems failure and whether there could be checks built into the system to ensure that no input was lost. code 2a: single word: class 2 almost all standard caa packages mark single word responses. problems may well occur, especially with less able candidates where spelling is poor, compromising the computer’s ability to recognize the answer, or where there are a number of synonyms which would be equally acceptable. these can be circumvented by entering a variety of alternative answers, including common spelling mistakes, which should also be marked as correct. alternatively, for spelling errors, a formulaic interpretation can catch unusual errors, however this must be monitored carefully to ensure that the net is not being cast too wide. changing the format of the question may be an option worth consideration in some cases—there are questions would lend themselves to objectification (perhaps through pulldown menus or drag and drop). realtime spell checks may also assist candidates enter a response which was recognisable to the computer, with mis-spelt words being highlighted to the candidate for revision – suggesting alternatives may however compromise the validity of the test. ultimately it should be fairly straightforward to computer mark single word response, however there may need to be an element of human marking back up, or question redesign to ensure the reliability of the system. these types of items could probably be migrated with minimal difficulty. code 3b: short response: class 3 these responses tend to be relatively factually based—where the mark key an evaluatory framework for caa migration 241 table 4. continued code and descriptions input classification marking classification essay responses where a response over two paragraphs would be demanded. is determined very much by the content of the response, rather than by its construction or style. these should thus not present too much of a challenge to most caa systems although the methods of obtaining a reliable marking schema which can be entirely computer driven may be laborious and time consuming. for small entry subjects, the process of creating the algorithms for computer marking may negate the benefits of on-line marking unless progress is made in this area. the technology to make this possible is certainly available, however some advances would help to make it a desirable innovation. code 3c—short answer: class 3 most caa packages accept short answer responses, however the accuracy of the marking varies in its reliability. michell et al. (2003) have suggested that there are systems available which after human moderation can mark at 99.4% accuracy overall. in a trial all items were marked at over 93% accuracy and with 98.1% of items over 95% accuracy. for more problematic items these could be redesigned to ease marking. they would still require a level of human moderation (figs above are post moderation) but this is a hopeful development. as with the above there is some issue with the time which may be needed to create and moderate the marking scheme, however as these question-types are more difficult to mark by human markers, this may not be such an issue in this instance. there are a few issues still to be resolved with these items however it looks as if reliable marking of short response questions may appear soon. 242 m. mcalpine table 4. continued code and descriptions input classification marking classification code 3d—extended response: class 4 this type of response would be best marked in a manner similar to that described below and would have similar problems and challenges associated with it—although it might be imagined that the problems associated with essays would be reduced as the size of the material was reduced, this may not be the case and further investigation into the technologies available would have to be performed. code 3e: essay—class 4 there are packages on the market which are designed to automatically mark essay responses. the most widely known and used is the e-rater system from education testing services (ets). these have problems associated with them, and it is not clear whether they would be accepted by markers and teachers. the developments in this area tend to come from the us, and are heavily influenced by us assessment practices, which may create challenges when migrating the technology to a scottish context. code 9: diagram items which require the candidate to draw something which is then evaluated. class 5 this would be a difficult one to implement without significantly changing the question or providing very specialist hardware, although questions did vary in their input computerisation difficulty. much though would have to be given to what the question was actually designed to assess, and how this could be put onto computer without compromising the nature of the assessment. some methods of input are already available—such as the class 5 much of the possibility of cam would be determined by the input mechanism used. where specific tools were used, there might be less difficulty in establishing a marking mechanism, however where a more generic input mechanism was used this may prove more challenging. there are already caa packages which do allow some element of automatic marking of questions requiring a diagrammatic response, however most significantly change the an evaluatory framework for caa migration 243 table 4. continued code and descriptions input classification marking classification demand of the questions in doing so. the pass-it trials included one such questions, however it is unclear whether the question was indeed equivalent to the paper based form or an alternative way of assessing the same skills. a full review and evaluation of the area would have to be undertaken. code 10: cloze response items where there is a body of material with missing pieces together with a choice of pieces to complete the material. candidates are asked to insert these pieces at the appropriate points. class 2 although the fundamental computerisation of a cloze response is quite unproblematic, there are a number of minor issues. the format of answering may be changed on a computer—including say drag and drop or scrolling and there may also be other methods of input. this may indeed add to the questions reliability by slightly altering the input mechanism to get rid of externalities such as spelling ability. class 1 although there will be trivial issues around spelling (where candidates are expected to type their response in)— these marking issues suffer from the same kind of difficulties as single word response items (code 3a). most other marking issues will be similar to those with paper based clozes. code 11: selected response items which required candidates to make a choice out of a number of possible given answers. class 1 input for these types of questions should not pose any particular problem and could be implemented in a number of ways. indeed computerisation of selected response items can add to the validity in a number of cases by changing the input mechanism (e.g. to a hotspot) rather than the traditional a/b/c/d response. class 1 techniques for marking selected response items on computer are well established. this should pose no particular problems. there may be issues where these items migrate from there traditional form to newer less well tested input mechanisms (e.g. hotspots) however these can be avoided until confidence in their reliability can be ensured. table 5. count of response type by code response type 1 2a 2b 3a 3b 3d 3e 3f 4 5 6 total counta 8 3 36 11 45 25 38 71 13 3 48 300 % of items 2.7 1.0 12.0 3.7 15.0 8.3 12.7 23.7 4.3 1.0 16.0 100 a note that one question was classified as both 1and 4, one as both 3 and 5 and one as both 5 and 9. 244 m. mcalpine 1. papers suitable for direct migration chemistry paper 1 only chemistry paper 1 was really suitable for direct migration. this was a multiple choice paper and, as can be seen from the graph above, very few of the questions pose any difficulty at all in migration to a computer based format. this examination could migrate practically instantly with very little modification. 2. papers which may be suitable for modification music paper 2 this paper has some challenges surrounding the stimulus material that it uses and indeed the stimulus is the biggest problem. the response types used for the majority of questions do not pose significant issues and the difficulties inherent in some of them could be circumvented. english paper 1 in this english paper, the response type is causing difficulties in migration although there are no stimulus issues. the questions are all fairly similar, suggesting that technical developments are needed to overcome the difficulties faced. changing the response type may challenge validity unless carefully studied. french paper 1 this paper has some technical challenges associated with the response type used, although these are not insurmountable. given the uniformity of the response types, hastening migration through adaptation may pose challenges to the validity of the assessment. an evaluatory framework for caa migration 245 french paper 2 there are some issues to be overcome in this paper both through the stimulus used and the response types used, although again these are not insurmountable. whether these can be done whilst maintaining the quality of the assessment needs to be viewed in a wider context. 3. papers where some of the paper may have to be rethought for caa chemistry paper 2 the variety of difficulty levels and technical difficulties which need to be overcome, mean that this might not be a suitable paper for migration until a significant number of the technical challenges have been overcome. should early migration be seen as desirable, compromises and adaptations to the response types used may be necessary. migration of some of the items may challenge validity. maths paper 1 this paper poses significant challenges to migrate. there may be some room to adapt the response types in particular and certain parts of stimulus could be presented in a more migrateable format, however there are a number of items which would have to change significantly in order to be computerized. 4. papers that may require a transformative approach english paper 2 as with the above, the response type used in this paper is not conducive to migration. technical developments are needed to enable migration. this may be an area in which the efforts involved in paper to screen migration may not be worth the transitional results. 246 m. mcalpine maths paper 2 this paper also has significant challenges to migration. there are few issues with the stimulus required, however the response types used are not conducive to migration. the response types used would have to be reconsidered should migration be desired in the short term. history paper 1 as with the english papers this history paper has significant problems associated with the response types used. using more migrateable response types would significantly alter the character of the exam. history paper 2 as with paper 1, considerable technical developments are needed to enable these types of items, and adaptation and changing response types would significantly alter the character of the exam. conclusions this methodology suggests a mechanism whereby papers and subjects which are being considered for migration to caa can be compared in their suitability for online delivery taking into account the wide variations in response types and stimuli which are found across papers and subjects. furthermore it also gives an indication to what extent the migration to computer-assisted formats for assessment may pose challenges to both reliability and validity, and at the same time open up opportunities for re-thinking the methods of assessment in those areas. the subjects which cannot easily be migrated to a caa format are perhaps the ones most amenable to the type of transformative assessment talked about by ripley and bennett, and the most promising for emerging technologies – however it must be considered to what extent this an evaluatory framework for caa migration 247 is indicative of sqa’s assessment practices and how can be allocated to the curricular areas themselves. while the conclusions that can be drawn from only six subjects, at one level are limited, the classification framework is now in place to rapidly construct comparable indicators for other subjects and other levels. this will give us an indication of what issues need to be tackled in each subject, how significant a problem they are and (for institutions which hold large numbers of paper-based items which they would wish to move to an online format) suggest an order in which migration can commence. one of the weaknesses of this project was that there was insufficient review of the emerging item marking and display technologies. it was outside the scope of this study to perform the type of comprehensive review which would be required and the classifications are given to the items should be interpreted with that caveat. a comprehensive study of emerging caa technologies is long overdue and would greatly inform the sector, not least by ensuring that anyone using this methodology for migrating paper based items to a caa format has a robust system in place by ensuring that the classification was as accurate as possible. notes 1. it should be noted that these codings are given on the basis of current knowledge of the author and are not based on any external categorisation. this may lead to inaccuracies in classification where technology has progressed beyond the author’s awareness. where an accurate classification is required it is recommended that a thorough review is undertaken and that further progress and development in the area is monitored. references ashton, schofield & woodger (2003) piloting summative web assessment in secondary education, 7th international computer assisted assessment conference, loughborough, 8–9 july. ripley (2003) the e-assessment tail: new opportunities? new challenges? the uses of ict in assessment, beyond the exam conference, bristol, 19 november. ripley (2004) assessment using the qca ‘rules base’ with process-based data, advisory council for new technologies in education, cambridge, 10 february. bennett, jenkins, persky & weiss (2004) assessing complex problem-solving performances: the naep problem solving in technology-rich environments project, advisory council for new technologies in education, cambridge, 10 february. bennett (2001) how the internet will help large-scale assessment reinvent itself, education policy analysis archives, 9(5). nisbet (2002) special arrangements for using ict in sqa assessments. project report to scottish qualifications authority. call centre, university of edinburgh. nisbet (2003) accessibility of sqa assessments in portable document format. project report to scottish qualifications authority. call centre, university of edinburgh mcalpine & ware (2003) introducing computer assisted assessment in scotland: laying the foundations for an integrated approach, 29th international association for educational assessment conference, manchester, 5–10 october. mccabe (2001) computer assisted assessment (caa) of proof = proof of caa—new approaches to computer assessment for higher level learning, 6th international conference on technology in mathematics teaching, ictmt-6, klagenfurt, austria. 248 m. mcalpine mckenna & bull (2000) quality assurance of computer assisted assessment: practical and strategic issues, quality assurance in education, 8(1), 24–31. mitchell, aldridge, williamson & broomhead (2003) computer based testing of medical knowledge, 7th international computer assisted assessment conference, loughborough, 8–9 july. doi:10.3402/rlt.v20i0/14404 by design: negotiating flexible learning in the built environment discipline richard tucker a, * and gayle morris b a school of architecture and building, waterfront campus, deakin university, geelong, australia; b centre for the enhancement of teaching and learning, the university of hong kong, hong kong (received 19 december 2010; final version received 9 may 2011) the term ‘flexible education’ is now firmly entrenched within australian higher education discourse, yet the term is a contested one imbued with a multiplicity of meanings. this paper describes a process designed to elucidate how the idea of flexible education can be translated into teaching models that are informed by the specific demands of disciplinary contexts. the process uses a flexible learning ‘matching’ tool to articulate the understandings and preferences of students and academics of the built environment to bridge the gap between student expectations of flexibility and their teacher’s willingness and ability to provide that flexibility within the limits of the pedagogical context and teaching resources. the findings suggest an informed starting point for educators in the built environment and other creative disciplines from which to traverse the complexities inherent in negotiating flexibility in an increasingly digital world. keywords: flexible learning; flexible delivery introduction in australia the term ‘flexible education’ is commonly used to incorporate flexible teaching, flexible learning and other related terms with which it is often used synonymously (e.g., e-learning, open learning, recourse-based learning, distance learning and self-directed learning). most australian universities claim flexible provision as a strategic teaching and learning objective, and the pervasiveness of the idea of ‘flexibility’ has only been increased by the most recent report to influence australian higher education policy and discourse � the 2008 bradley review (bradley et al. 2008). yet there is no universally accepted definition of what is meant by flexible education (casey and wilson 2005; kirkpatrick 1997; ling et al. 2001; morrison and pitfield 2006; nicoll 1998; normand, littlejohn, and falconer 2008; nunan 1996; sappey 2005). nor is there agreement on how flexibility (no matter how it is defined) as an institutional objective should be implemented at the teacher/student interface; where teachers have to provide for flexibility within budgetary limits often informed by the requirements of more traditional teaching. (page number not for citation purpose) *corresponding author: email: rtucker@deakin.edu research in learning technology vol. 20, 2012 rlt 2012. # 2012 r. tucker and g. morris. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 the project described in this paper aimed to articulate the meaning of ‘flexible’ learning for students and teachers at an australian school of architecture and building, where traditionally the dominant teaching form has been studio-based and thus face-to-face, by developing a ‘matching’ process for negotiating student and teacher competing demands. the matching process aimed to bridge the gap between student expectations of flexibility and their teacher’s willingness and ability to provide that flexibility within the limits of pedagogical context and available teaching resources (finances, technology, staff � see palmer on these difficulties �(palmer 2001)). the process in normand and littlejohn’s (2006, 22) terms is illustrative of a bottom-up initiative, enabling a ‘teaching�learning’ discourse rather than a ‘managerialist discourse,’ and thus providing the foundation for more robust course development and design. the project had three primary goals; namely, to: 1. articulate a context related understanding of ‘flexible’ learning that it is contained within manageable and meaningful boundaries; 2. advance a negotiation tool for matching students’ expectations of flexible learning to teacher attributes, recourses and pedagogical intent; 3. create two distinct models of flexible learning to be used as a basis for informing appropriate flexibility for (1) theory-lecture-based and (2) designstudio-based learning. these goals address a need to bridge two distant extremes; those of traditional modes of higher education delivery, which provide the basis of most teachers’ experience, in contrast to what the modern student demands, which is (as van den brande defines flexible learning (1993, 22)) being able to ‘learn when they want (frequency, timing, duration), how they want (modes of learning), and what they want (that is learners can define what constitutes learning to them).’ taking a lead from the central importance of context in making meaning of flexibility (casey and wilson 2005; kirkpatrick 1997; ling et al. 2001; morrison and pitfield 2006; sappey 2005), and from the primacy of the individual agency of teachers and students over institutional rhetoric and policy (bigum and rowan 2004; errington 2004; nicoll and chappell 1998; normand, littlejohn and falconer 2008), our study is underpinned by the argument that the precise meaning and value of flexible education can only be found in the details of the experience of teachers and students engaged in their specific discipline context. to articulate student/teachers experiences we have used a model of flexible learning based on the work of nikolova and collis (1998) and collis and moonen (2004), who discuss five basic ‘categories’ of flexibility (time, content, access/entry requirements, instructional approach and delivery), which can be further split into 19 ‘dimensions.’ the model supports bottom-up analysis enabling discourse among those providing and receiving learning flexibility. this is consistent with kennedy et al. (2008) who suggest that rather ‘than making assumptions about what students like � and are like � universitites and their staff must look to the evidence to inform both policy and practice’ (118). the 19 flexibility dimensions are grouped into the five operationalising categories as follows: r. tucker and g. morris 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 � aspects of time 1. time and date at which module starts and finishes 2. periods of time students are able to participate 3. pace of learning 4. time when assessment occurs 5. sequence in which topics are covered � aspects of content 6. choice of topics covered 7. amount of learning activities expected to be completed 8. level of difficulty of module content 9. assessment standards � aspects of access/entry requirements 10. pre-requisites for module participation � aspects of instructional approach/design 11. social organisation of learning (group or individual) 12. times available for support 13. choice of who decides what modes of flexible learning are available 14. language for communication � aspects of delivery 15. time and place where support is available 16. methods of obtaining support 17. types of support available 18. places for studying 19. delivery channels (i.e., lectures, tutorials, internet, podcasts) before we consider in detail our own analysis of how these 19 dimensions can be used to identify appropriate pedagogic models, we shall briefly consider the literature on how the implementation of institutional flexible education policy has been received by learners and teachers. background the implementation of flexibility there has been limited research on how flexible education can be appropriately translated into teaching models informed by the specific demands of other disciplinary contexts, including the focus of this paper; the built environment. within the wide range of telescopia, a project involving the application of a flexible delivery platform for trans-european tele-learning, collis, vingerhoets and moonen (1997) suggest that moving from fixed to flexible is more difficult to implement for some of the 19 dimensions than for others. thus, timeand place-flexibility are easiest to implement, but offering flexibility on the other dimensions is difficult because; (1) the costs of realising flexibility are high and, (2) increased flexibility on a number of the dimensions leads to conflicts between institutional management, teachers and learners. collis, vingerhoets, and moonen see that ‘the transition between offering a well-designed and well-supported course, and offering more of a ‘cafeteria’ of options will require conceptual changes not only for course providers but also for the broader society’ (collis et al. 1997). research in learning technology citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 3 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 within work that focuses on student experiences, only a few studies engage with the disciplinary contexts. for example, mcshane, peat, and masters (2007) have shown that chemistry undergraduates enroled in a traditional, research-focussed university continued to expect face-to-face lectures when given online options (26). they also found that undergraduates, particularly with little previous exposure to online instruction, preferred on-campus timetable structures and that half of the surveyed group never accessed the online material made available. their evidence suggested that a sudden move to web-based learning alienated less technically competent students or those without access to adequate computers. mcshane et al. concluded that flexible practices should be introduced from the first year via a staged and coordinated process (26). in line with these findings, dobozy (2008) has shown that first year students do not cope with flexible access provision and suggests that universities therefore have an ethical obligation to help early students to improve their engagement levels. similarly, samarawickrema (2005) demonstrated that first-year design students were highly teacher dependent, required better access to academic staff and that all categories of learners experienced difficulties related to online study in their first semester. pillay, irving, and tones (2007) investigated a diagnostic tool for assessing tertiary students’ readiness for online learning. they found (221) that ‘online students rated flexible pacing, time of study and management of conflicts between study time and other commitments as more important, and social interaction as less important, than classroom students.’ it is commonly suggested that online learning should augment, rather than replace, experiential face-to-face learning (jones and richardson 2002). similarly, poindexter (2003) warns against incorporating technology in isolation, advocating a more holistic approach that uses multiple strategies and takes into account the changing student generation. as with all instructional tools and approaches, the use of technology needs to grow out of sound learning objectives and resonate with an evidence base about how students’ learn. given the focus of this study, we might add that this evidence base ought to include the disciplinary contexts in which we teach � for different disciplinary cultures have their own discursive practices and their own ways of thinking, relating and being. others, such as shaffer (2004), offer a slightly different starting point by exploring how the psychological implications of new media may require a new way of understanding ‘how tools and thoughts contribute equivalently (though perhaps still not equally) to educational and practical outcomes’ (1). shaffer raises the possibility that it is the relationship between technology and cognitive activity that informs social and pedagogical choices in the context of rapidly emerging technologies. teaching/learning flexibility nexus: a matching process as nikolova and collis have explained (1998, 67), at one extreme of the flexibility continuum are traditional courses with module dates that are fixed, content that is predetermined, instructional approaches chosen, learning materials prepared in advance and course organisation that is pre-defined. the other extreme of the continuum ‘is a just-in-time, workplace-based, problem-induced learning, about which the learner makes key choices and which occurs life-long.’ we argue that it is somewhere between the two extremes that appropriately flexible learning exists � at a point that matches student expectations with teacher willingness, ability and resources. the method used by us to identify this nexus builds on a process first posited by nikolova and collis r. tucker and g. morris 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 (1998) that maps the 19 flexibility dimensions to generic flexible module profiles to inform the adaptation for flexibility of a course’s teaching modules. our method is also in line with the work of de boer and collis (2005) who aimed to operationalise flexibility by examining the extent to which the 19 dimensions were being offered in practice by instructors. only 12 dimensions were considered in the de boer study; that is, those seen to represent aspects of flexibility that could be determined by instructors. the 12 excluded dimension numbers 5, 7, 13 and 16 (although de boer and collis have varied their dimension descriptions from the 19 listed in our introduction). in common with our own project, a five-stage continuum of flexibility (see figure 1) was used by de boer and collis to numerate each dimension. this continuum progresses from a traditional, ‘closed’ course tied down in terms of place, time and content towards more flexible, ‘open,’ learning where the design of the course is shared by students and academics. once each dimension was given a flexibility value (1 being the most fixed and 5 being the most flexible), the mean was calculated. in the de boer and collis study, the greatest flexibility was offered in: pace of learning (3.06), delivery channels (3.40), places for studying (3.28) and time and place where support is available (3.18). de boer and collis suggest the study indicates two types of flexibility can be operationalised by instructors (2005, 46): planning flexibility, which maintains more or less the same pedagogy and teaching and learning programme while offering more flexibility in terms of delivery; and interpersonal flexibility, which implies a change in pedagogy to more ‘studentcentred contributions that relate to the experiences of the individual students and can be re-used by others as learning resources.’ they concluded that the change to interpersonal flexibility is more difficult ‘because instructors need to rethink their courses in terms of the activities within the course and also the assessment of those new activities.’ methodology in our project, as reported elsewhere (tucker and morris 2011), for each of the 19 dimensions a ‘flexible learning nexus’ was calculated for the two types of teaching model that dominate built environment education � design-studio-based and theorylecture-based modules. the nexus matched student and teacher expectations at a half-way point of flexibility. the 19 nexus were then collapsed to the five operationalising categories of time, content, access/entry requirements, instructional approach and delivery. the project utilised online questionnaires to profile learners and teachers and to identify what dimensions of flexibility both groups felt were appropriate to the two different learning contexts. once each dimension was given a flexibility value, the mean was calculated between what students desired and what teachers felt able to provide. the online questionnaire was in six sections: the first two profiled the participants by determining demographics and learning styles, the third considered attitudes to figure 1. five-sage continuum of flexibility. research in learning technology citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 5 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 and learner/teacher confidence in the modules being studied/taught, the fourth considered awareness and understanding of what flexible education is, and the final two sections determined for each of the 19 dimensions what degree of flexibility both groups felt were appropriate to the two different learning contexts. in the fifth section, questions asked participants to determine an order of preference, for each dimension, between: 1. inflexible, 2. �intermediate and 3. flexible (see table 1). in the sixth section, participants were asked to rate, on a five-point likert scale, six different learning materials, five avenues of obtaining feedback and six learning spaces, for example, physical locations of study. after the data from the questionnaire were analysed, the findings were translated into the design of two generic modules � one lecture-based (delivering theory) and one studio-based (applying that theory to design practice). the generic modules were used as the basis for redesigning two specific modules that the students who had answered the entry questionnaires then participated in. further research will see these specific modules evaluated through exit questionnaires and student evaluations of teaching (set). participants overall 78 students participated in the study. there was an approximately even split between male (40, 51.3%) and female (38, 48.7%) participants, with their ages ranging from 18 to 44 years of age with a mean age of 22. the majority of students were domestic (77, 98.7%) compared to international students (1, 1.3%). students were enroled in three degree programmes: 31 (39.7%) were enroled in architecture, eight (10.3%) were enroled in a construction management (cm) degree and 39 (50%) of students were enroled in a double degree (architecture and cm). at the unit level, students were enroled in a total of 12 undergraduate units in the architecture, cm and archi/cm double degree courses. there were four, 1st year units, four, 2nd year units, two, 3rd year units and two, 4th year units. the majority of students were part-time, but all students, regardless of status attended classes on campus. one of the authors was directly involved in teaching one of the units, however, any perception of undue influence was mitigated by ensuring strict adherence to a high standard of research ethics. participation was voluntary and the data deidentified to ensure anonymity. it should also be noted in terms of the participants’ background, that the survey did not set out to specifically map students’ previous experiencs of flexible learning, however, consistent with other research, for example, kennedy et al. (2008) we would expect that in relation to experience of technologies, as one manifestation of flexible learning, patterns of access and use of a range of technologies would be widely disparate (108). table 1. example of order-of-preference question for flexibility dimension. who should decide the level of difficulty of unit content? 1st choice 2nd choice 3rd choice 1) lecturer decides 2) you, the learner, negotiates difficulty level with lecturer 3) you, the learner, decides between basic, intermediate or advanced r. tucker and g. morris 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 thirteen lecturers participated in the study, approximately half (7, 53.8%) were male and their ages ranged between 29 and 60 with the mean age being 47. the lecturers ranged in experience from early career researchers in their 30s to senior academics late in their careers. seven lecturers taught in all three degree programmes, one only in architecture and five only in cm. their units were selected to cover a range of online engagement, with some having the most basic level of online presence to those that had the majority of materials online. results awareness and understanding of flexible learning let us now consider the results of the fourth section of the questionnaire; that is, awareness and understanding of what flexible education is. there was an even split in our study between students who were aware that flexible learning was a core educational goal of their university and those who were not. all of the lecturers who responded were aware that flexible learning was a core educational goal. a single sample t-test examined students understanding of flexible learning with results indicated that students felt they had a significant understanding of flexible learning (m 2.50, sd .82) ([m�3] t (77) �5.29, p�0001). the majority of lecturers who responded indicated that they understood what flexible learning was (10, 76.9%) and that flexible learning was a familiar term to them (12, 92.3%). two questions examined the general importance of flexible learning; (1) how important to you is it that your education is flexible? and (2) how flexible should your education be? an average of these items in a single sample t-test indicated that students felt flexible learning was significantly important (m 1.92, sd .61) ([m�3] t (77) �15.44, p� 0001). a simple regression analysis revealed that ‘understanding’ of flexible learning significantly predicted ‘importance’ of flexible learning, (b�0.439, t(76)�4.25, pb 0.0001). understanding of flexible learning also explained a significant proportion (19.20%) of the variance in importance of flexible learning, (r 2�0.192, f(1, 77)� 18.11, pb 0.001). students felt that flexible learning was important for three reasons; educational, personal and economical reasons, and that this flexibility should allow them to learn how, when and what they wanted (see table 2). pearson correlation analyses (table 3) indicated that while the importance of flexibility in ‘how’ and ‘what’ students learnt significantly correlated, the importance of ‘when’ they learnt was not correlated with the other flexibility dimensions. in terms of what motivates flexibility, economy, education and personal reasons all correlated with one another. however, only economy and personal reasons correlated with ‘when,’ only educational and personal correlated with ‘how,’ and none of the flexibility motivators correlated with ‘what.’ simple regression analyses examined how the importance of each of the flexibility motivators predicted the importance of the three different areas of flexibility � when, how and what. the first simple regression examined the ability of the three motivators (economy, personal and education) to predict the importance of flexibility in ‘when’ students learnt. results revealed that the model significantly predicted 50.8% of the variance in the importance of ‘when’ (r 2� 0.508, f(3, 74)� 25.42, pb 0.0001). however, this finding was driven by the ability of economy (b� .285, t (74)�2.89, pb0.005) and personal (b�.534, t(74)�5. 220, pb 0.0001) to research in learning technology citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 7 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 predict ‘when.’ education as a motivator for flexibility in ‘when’ students could learn remained non-significant (b��.063, t(76)��.717, pb .475). the second simple regression examined the ability of the three motivators to predict the importance of ‘how’ student’s learnt. results revealed that the model significantly predicted 39.1% of the variance in the importance of flexibility in ‘how’ students learnt (r 2�0.391, f(3, 74)�15.81, pb0.0001). however, this finding was driven only by the ability of education (b�.611, t(74)�6.254, pb 0.0001) to predict flexibility in ‘how’ students learnt. both personal and economical remained nonsignificant in predicting ‘how’ (both pb0.53). the final simple regression examined the ability of the three motivators to predict the importance of flexibility in ‘what’ students learnt. results revealed that the model did not significantly predict variance in the importance of ‘what’ (r 2�0.052, f(3, 74)�1.36, pb 0.260). the following findings can be interpreted from the above analyses of student understanding of flexible learning: . students felt that flexible learning was significantly important. . as might be expected, students understanding of flexible learning predicted how important they felt flexible learning to be. . flexible learning was important to students for educational, economical and personal reasons. . while economic and personal reasons predicted the importance to students of flexibility in when they learnt, educational reasons predicted the importance to students of flexibility in how they learnt. table 2. one sample t-test. mean (sd) [u�3] t (df 77) important when 1.73 (.80) �14.003** important how 1.71 (.79) �14.452** important what 2.10 (.96) �8.244** important economically 1.72(1.03) �10.986** important educationally 1.82 (.89) �11.657** important personally 2.15(1.04) �7.148** table 3. pearson correlations. important when important how important what important economically important educationally important personally important when 1 0.201 0.019 0.568(**) 0.204 0.671(**) important how 0.201 1 0.279(*) 0.215 0.622(**) 0.244(*) important what 0.019 0.279(*) 1 �.088 0.128 0.100 important economically 0.568(**) 0.215 �0.088 1 0.255(*) 0.559(**) important educationally 0.204 0.622(**) 0.128 0.255(*) 1 0.364(**) important personally 0.671(**) 0.244(*) 0.100 0.559(**) 0.364(**) 1 r. tucker and g. morris 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 . students felt that it was important that what, how and when they learnt was flexible. however, as we shall see now, these findings were not consistent with what students told us in detail in the final two sections of the questionnaire about their preferences regarding the 19 flexibility dimensions. for although this data also showed a desire for choice in ‘when’ and ‘how’ students learned, it suggested that students did not desire to choose ‘what’ they learned. students’ expectations of flexible learning matched to teacher attributes, recourses and pedagogical intent the five collapsed nexus explained in the method section are recreated below (figure 2 white representing matched flexibility for the learning context of designstudio-based teaching, and grey representing matched flexibility for lecture-based teaching). figure 2. the five collapsed nexus for time, content, entry requirements, instructional approaches and delivery. research in learning technology citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 9 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 although students rated at over 3.0 (i.e., ‘flexible’) six dimensions of flexibility in design-studio-based modules and three dimensions in theory-lecture-based modules, the collapsed-nexus demonstrated that for both staff and students, in neither learning context was flexibility in time, content or access/entry requirements seen as appropriate or desirable. in the fourth operationalising category � instructional approaches � if we remove the dimension of language (for this in most cases was not a possible variable) the collapsed nexus returns means of: studio�3.57, lecture� 3.37. in the fifth operationalising category � delivery � all five dimensions were desired as flexible by students, and the collapsed nexus for both learning contexts are again rated as ‘flexible,’ with means of: studio�3.65, lecture�3.59. the overall interpretation for the above five delivery dimensions is straightforward; students wanted flexibility in delivery and social organisation for both theory-lecture-based and studio/design modules. these findings are largely consistent with what de boer and collis found to be offered in practice by instructors. once the nexus were determined through survey, the flexibility ratings were translated into learning design implications or frameworks for each of the 19 dimensions � for both lecture-based and studio teaching (table 4). the outcomes were arrived at by identifying for each dimension what the most fixed and flexible outcome might be and then scaling towards the dimension rating. thus, for example, for dimension 1 (time and date module starts and finishes), a 1.0 rating equated to fixed module start and finish and 5.0 to an elective that could be enrolled in at any time during the degree programme. with many of the dimensions it was not possible to provide choice due to the constraints of professional accreditation � which often dictates the order in which modules can be studied, prerequisites, assessment standards, and module and course content. table 4 sets out what might be usefully considered as prototypes of two of the main modes of teaching in the built environment, studio design and theory-lecturebased modules. at a glance it provides academics with a more nuanced reading of each of the flexibility dimensions. the table tries to illustrate through concrete examples what it might mean in practice. in doing so, it may expose dimensions hitherto not recognised, and therefore open flexibility doors where appropriate to do so. as prototypes, the objective is for academics to see possibilities emerging from the research findings, which could be used, but are of course open to negotiation and other interpretations. findings in relation to other categorisations of flexible education there are, of course, others ways of characterising models of online delivery than the 19-dimension/five-operationalising categorisation of nikolova and collis. for example, roberts (2002) describes a four model categorisation of online delivery � the naı̈ve model (where only lecture notes are provided online), the standard model (with an email list and other online resources), the evolutionary model (which scores most highly for flexibility) and the radical model (which is largely based on group work) (5�6). using this categorisation it can be seen that the two modules described in table 4 are variations on an ‘evolutionary’ model, with the lecture/theory module representing a standard/evolutionary hybrid. how ever, it should be stressed that both of the two modules are intended to have significant face-to-face components. the u.s. department of education (2009) use an alternative conceptual framework which identifies three key components when considering online delivery, either r. tucker and g. morris 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 table 4. flexibility dimensions set out against general module characteristics for designand theory-lecture-based modules. nexus learning design implications flexible learning dimension studio/ design model lecture/ theory model studio/design model lecture/theory model time 1. time and date module starts and finishes 2.11 1.60 fixed fixed 2. periods of time students to participate 3.17 2.25 lectures fixed tutorial times chosen by students and varying as assignments required lectures fixed tutorial times chosen by students at the beginning of semester 3. pace of learning 3.17 2.69 project-based assignment and formative assessment allows learning pace to develop within fixed semester duration limits project-based assignment allows learning pace to develop within fixed 4-weekly duration limits 4. time when assessment occurs 2.15 2.08 fixed fixed 5. sequence in which topics are covered 2.49 2.22 malleable within assignment limits fixed content 6. choice of topics covered 3.34 2.59 students choose between assignments student choose specialisation 7. amount of learning activities completed 2.92 2.12 fixed fixed 8. level of difficulty of module content 2.89 2.5 students determine project complexity fixed 9. assessment standards 2.19 1.93 fixed fixed access/entry requirements 10. pre-requisites for module participation 2.31 2.40 fixed fixed pedagogy 11. social organization (group or individual) 3.47 3.39 student has choice unless ‘teamwork’ skills are a focus options for those students that have to work alone student has choice unless ‘teamwork’ skills are a focus options for those students that have to work alone 12. times available for support 3.18 2.85 face-to-face fixed within 6-hour studio sessions face-to-face fixed to hourly tutorials but feedback available regularly via online discussion boards 13. who chooses modes of flexible learning 2.84 2.67 predetermined by matching process predetermined by matching process 14. language for communication 1.77 1.46 fixed fixed research in learning technology citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 11 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 table 4 (continued) nexus learning design implications flexible learning dimension studio/ design model lecture/ theory model studio/design model lecture/theory model delivery 15. time and place where support is available 2.93 2.70 face-to-face fixed within 6-hour studio sessions face-to-face fixed to hourly tutorials but feedback available regularly via online discussion boards 16. methods of obtaining support 3.46 3.23 predominantly face-toface ‘sign-up’ tutorials with some on-line feedback heavy online support available via unit webpage discussion thread email supporting face-to-face 17. types of support available for 3.52 3.26 predominantly face-toface peer-support and peerfeedback opportunities online email face-to-face discussion board for reflective discussions 18. places for studying 3.15 2.96 studio lecture theatre lecture theatre tutorial rooms online via unit webpage mobile devices 19. delivery channels 3.81 3.62 a range of electronic resources linked to from the course home page studio workshops paper course notes lecture slides in power point format assignment marking guidelines full contact details of all instructors pre-recorded audio lectures available from the web for key lectures ‘live’ lectures a range of electronic resources linked to from the course home page electronic copies of all printed course materials lecture slides in powerpoint format assignment marking guidelines full contact details of all instructors copies of past examinations for the cours hints and tips for the current examination model answers an electronic course discussion list pre-recorded audio lectures available from the unit webpage for all lectures animations to explain many of the concepts ‘live’ lectures web-based archives of mailing list discussions from previous semesters electronic assignments submission, recording marking and return r. tucker and g. morris 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 ‘wholly online’ or as part of a blended approach, ‘(1) whether the activity served as a replacement for or an enhancement to conventional face-to-face instruction, (2) the type of learning experience (pedagogical approach) and (3) whether communication was primarily synchronous or asynchronous’ (3). each component represents what might be conceived of as a key decision point where the response opens up or limits the way in which technology might be incorporated. discussion the generic modules represented in table 4 are not intended to be prescriptions or templates but, rather, to offer academics an informed starting point from which to traverse the complexities inherent in negotiating flexibility. in that sense they represent one way of representing what typical design-studio-based and theorylecture-based fexible learning modules might look like. both types of flexible module could be said to be consistent with de boer and collis’s planning flexibility mode of flexible teaching in that they maintain more or less the same pedagogy, while offering more flexibility in terms of delivery. academics are often unclear about how to incorporate aspects of flexibility and the appropriation of technology is often more by ‘feel’ rather than an theoretically informed perspective. the generic modules may ease that gap by providing academics a template from which to consider the pedagogical benefits of and the changes required to provide for flexible learning. the modules also provide an organisational framework that brings coherence to elements of flexible learning and conventional models of delivery. one of the limitations of the tool presented here is that it may appear to offer up a prescription, or a kind of ‘one size fits all’ solution that is driven purely by student want and not a considered position grounded in robust instructional design or learning theory. as noted previously, the intention is to offer academics a way forward in terms of brokering several key dimensions of flexibility based on local evidence. the key here is in ‘brokering;’ for as with any mode of delivery, it is incumbent to be guided by an evidence base about how students learn, and the corresponding kinds of learning experiences we wish our students to have. but, as shaffer’s work reveals, these positions too are contested as new theories emerge on cognition and technology that challenge traditional notions of how we learn. conclusions using a flexible learning ‘matching’ tool, this project was able to articulate the understandings and preferences of students and academics of the built environment to bridge the gap between student expectations of flexibility and their teacher’s willingness and ability to provide that flexibility within the limits of the pedagogical context and available teaching resources. the findings suggest an informed starting point from which to navigate the pedagogical complexities inherent in negotiating flexibility in an increasingly digital world. with that in mind, there are a number of implications for thinking about flexible learning within the built environment disciplines and a number of conclusions that can be drawn about teachers and learners in this specificic discipliniary context. these implications and conclusions are: research in learning technology citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 13 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 . for studio modules, only in content did students demand less flexibility than teachers offered, whereas for lecture modules students in all cases demanded greater flexibility than teachers. this demonstrates that, for built environment students, and perhaps too for students in other creative disciplines, learning can be more flexible than it is at present. in particular, the demand by students for more flexibility than their teachers are providing is greatest for the operationalising category of content delivery. thus, as we have argued elsewhere (tucker and morris 2011) what our built environment students desired of teaching, and what their ‘teachers were able to provide, were multiple mediums of knowledge delivery that allowed students flexibility in when and where they could learn.’ . multiple mediums of delivery (types of learning material) were seen by students as more important for theory-lecture-based modules than for designstudio-based modules. this we suggest is largely due to the preference for faceto-face feedback in design. this preference is because much of design learning is experiential and thus feedback is usually individualised to the artefact created as a demonstration of that learning � artefact which, in the case of design is different for every student. . for the categories of time, content and acess/entry requirements, students did not demand flexibilty. this is perhaps because in such an amorphous, creative learning context as design, students desire a learning structure that is clear and unshifting. in other words, when designing artefacts (such as buildings), students may not wish to be designing their education. . thus, the desire for fixidity in most dimensions, in a context where learning outcomes are less fixed, might be said to be related to the paradox of choice that sees learners in general experience a decrease in confidence when there is an increase in options. indeed, this effect on learner confidence might be even greater where, as is the case for students learning how to design, there is uncertainty in the learning outcome and where assignment solutions are infinite. the research that underpins this project was a direct attempt to engage with students and academics about the meaning and value of flexible education � as a lived practice in their local discipline contexts. it is not without limitations. the sample size was limited and was restricted to the built environment disciplines. moreover, it is likely that each new cohort of students’ preferences will be shaped by their particular experiences and so the extent to which our findings can be extrapolated to future cohorts and different disciplinary contexts requires further investigation. future research will endeavour to map students’ learning preferences to those expressed in the flexible learning survey to explore any significant patterns. as the growth in flexible learning continues within higher education, there is a corresponding need to find new ways of understanding and measuring qualitatively different learning experiences of students in specific learning contexts. disciplines by their nature emphasise some skills, ways of knowing and being over others which, as our findings suggest, leads to appropriating technologies for different purposes. it is our hope that research of this nature will add to our understandings of student learning and enable a more authentic basis from which to negotiate flexible learning in an increasingly digital world. r. tucker and g. morris 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 acknowledgements input from our diligent research assistant, catherine reynolds is acknowledged. references bigum, c. & rowan, l. (2004) ‘flexible learning in teacher education: myths, muddles and models’’, asia-pacific journal of teacher education, vol. 32, no. 3, pp. 213�226. bradley, d., noonan, p., nugent, h. & scales, b. (2008) review of australian higher education � final report. commonwealth of australia, canberra. casey, j. & wilson, p. (2005) a practical guide to providing flexible learning in further and higher education. quality assurance agency for higher education scotland, glasgow. available online at: http//www.enhancementthemes.ac.uk/documents/flexibledelivery/fd_flexible_ learning_jcaseyfinalweb.pdf [accessed 15 november 2010]. collis, b. & moonen, j. (2004) flexible learning in a digital world, 2nd edn, routledge and falmer, london. collis, b., vingerhoets, j. & moonen, j. (1997) ‘flexibility as a key construct in european training: experiences from the telescopia project’, british journal of educational technology, vol 28 no. 3, pp. 199�217. de boer, d. & collis, b. (2005) ‘becoming more systematic about flexible learning: beyond time and distance’, alt-j’, research in learning technology, vol. 13, no. 1, pp. 33�48. dobozy, e. (2008) ‘plan your study around your life, not the other way around: how are semiengaged students coping with flexible access?’ paper presented at the edu-com conference, khon kaen, thailand. errington, e. (2004) ‘the impact of teacher beliefs on flexible learning innovation: some practices and possibilities for academic developers’ innovations in education and teaching international, vol. 411, no. 1, pp. 39 � 47. jones, s. & richardson, j. (2002) ‘designing an it-augmented student-centred learning environment’, paper presentat at the herdsa conference, perth, australia. kennedy, g. et al. (2008) ‘first year students’ experiences with technology: are they really digital natives?’, educational technology, vol. 24, no. 1, pp. 108�122. kirkpatrick, d. (1997) ‘becoming flexible: contested territory’, studies in continuing education, vol. 19, no. 2, pp. 160 � 173. ling, p., et al. (2001) the effectiveness of models of flexible provision of higher education � 01/9, department of education, training and youth affairs, canberra. available online at: http://www.dest.gov.au/sectors/higher_education/publications_resources/profiles/archives/ models_of_flexible_provision_higher_education.htm [accessed 15 november 2010]. mcshane, k., peat, m. & masters, a. f. (2007) ‘playing it safe? students’ study preferences in a flexible chemistry module’, australian. journal. of education. in chemistry, no. 67: 24�30. morrison, l. & pitfield, m. (2006) ‘flexibility in initial teacher education: implications for pedagogy and practice’’, journal of education for teaching, vol. 32, no. 2, pp. 185�196. nicoll, k. (1998) ‘"fixing" the "facts": flexible learning as policy invention’, higher education research & development, vol. 17, no. 3, pp. 291 � 304. nicoll, k. & chappell, c. (1998) ‘policy effects: "flexible learning" in higher education and the "de-differentiation" of the vocational sector’, studies in continuing education, vol. 20, no. 1, pp. 39�50. nikolova, i. & collis, b. (1998) ‘flexible learning and design of instruction’, british journal of educational technology, vol. 29, no. 1, pp. 59�72. normand, c. & littlejohn, a. (2006) enhancing practice � flexible delivery � a model for analysis and implementation of flexible programme delivery, the quality assurance agency for higher education, gloucester. available online at: www.enhancementthemes.ac.uk/ documents/flexibledelivery/flexible_delivery_qaa_124.pdf [accessed 15 november 2010]. normand, c., littlejohn, a. & falconer, i. (2008) ‘a model for effective implementation of flexible programme delivery’’, innovations in education and teaching international, vol. 45, no. 1, pp. 25�36. nunan, t. (1996) ‘flexible delivery � what is it and why is it a part of current educational debate? ’, in higher education research and development society of australasia annual conference, perth, pp. 595�600. research in learning technology citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 15 (page number not for citation purpose) http://http//www.enhancementthemes.ac.uk/documents/flexibledelivery/fd_flexible_learning_jcaseyfinalweb.pdf http://http//www.enhancementthemes.ac.uk/documents/flexibledelivery/fd_flexible_learning_jcaseyfinalweb.pdf http://www.dest.gov.au/sectors/higher_education/publications_resources/profiles/archives/models_of_flexible_provision_higher_education.htm http://www.dest.gov.au/sectors/higher_education/publications_resources/profiles/archives/models_of_flexible_provision_higher_education.htm http://www.enhancementthemes.ac.uk/documents/flexibledelivery/flexible_delivery_qaa_124.pdf http://www.enhancementthemes.ac.uk/documents/flexibledelivery/flexible_delivery_qaa_124.pdf http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 palmer, s. (2001) ‘engineering flexible teaching and learning in engineering education’, european journal of engineering education, vol. 26, no. 1, pp. 1�13. pillay, h., irving, k. & tones, m. (2007) ‘validation of the diagnostic tool for assessing" tertiary students’, higer education research and development, vol. 26, no. 2, pp. 217�34. poindexter, s. (2003) ‘the case for holistic learning’, change, vol. 35, no. 1, pp. 24�31. roberts, t. s. (2002) ‘flexible learning: how can we get there from here?’, ascilite 2002, auckland, new zealand, pp. 8�11, december 2002. samarawickrema, g. r. (2005) ‘determinants of student readiness for flexible learning: some preliminary findings’, distance education, vol. 26, no. 1, pp. 49�66. sappey, j. (2005) ‘the commodification of higher education: flexible delivery and its implications for the academic labour process’, in reworking work-airaanz 19th conference, eds m. baird, r. cooper & m. westcott, the association of industrial relations academics of australia and new zealand, sydney, pp. 495�502. shaffer, d. (2004) ‘pedagogical praxis: the professions as models for post-industrial education’, teachers college record, vol. 106, no. 7, pp. 1401�1421. tucker, r. & morris, g. (2011) ‘anytime, anywhere, anyplace: articulating the meaning of flexible delivery in built environment education, british journal of educational technology, vol. 41, pp. 904�915. u.s. department of education. (2009) evaluation of evidence-based practices in online learning:a meta-analysis and review of online learning studies. available online at: http// www.ed.gov/about/offices/list/opepd/ppss/reports.html [accessed 15 november 2010]. van den brande, l. (1993) flexible and distance learning, john wiley & sons, new york. r. tucker and g. morris 16 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14404 doi: 10.3402/rlt.v20i0/14404 http://http//www.ed.gov/about/offices/list/opepd/ppss/reports.html http://http//www.ed.gov/about/offices/list/opepd/ppss/reports.html http://researchinlearningtechnology.net/index.php/rlt/article/view/14404 courseware in academic library user education: a literature review from the gaels joint electronic library project nick joint* bob kemp* and susan ashworth** *centre for digital library research, university of strathclyde **glasgow university library email: n.c.joint@strath.ac.uk the use of courseware for information skills teaching in academic libraries has been growing for a number of years. in order to create effective courseware packages to support joint electronic library activity at glasgow and strathclyde universities, the gaels project conducted a literature review of the subject. this review discovered a range of factors common to successful library courseware implementations, such as the need for practitioners to feel a sense of ownership of the medium, a need for courseware customization to local information environments, and an emphasis on training packages for large bodies of undergraduates. however, we also noted underdeveloped aspects worthy of further attention, such as treatment of pedagogic issues in library computer-aided learning (cal) implementations and use of hypertextual learning materials for more advanced information skills training. we describe how these findings shaped the packages produced by the project and suggest ways forward for similar types of implementation. introduction the gaels project (glasgow allied electronically with strathclyde) is a library-based project intended to promote a culture shift among engineering researchers at glasgow and strathclyde universities. our intention is to decrease researchers' dependence on separately held local print collections in favour of collaboratively held networked electronic resources. to support this aim, gaels (1999) has created a courseware package to teach researchers the different information retrieval skills required to use such networked resources. in the course of this work, we undertook a review of the literature relating to the use of computer-aided learning materials in academic libraries. we wanted to create a hypermedia-based package delivered over a web platform, but we knew that writers on 26 alt-j volume 9 number 3 courseware development and educational theory (laurillard, 1993; benyon and woodruffe, 1997) had expressed caution about the effectiveness of hypermedia as a way of supporting genuine educational outcomes. moreover, well-regarded library user education programmes using both traditional and cal-based approaches had hitherto failed to disseminate the necessary level of networked information skills amongst academic staff. there appeared to be no consensus on how to offer effective networked information skills training, nor on the larger issues of modifying existing patterns of research information use. before we committed ourselves to a particular approach, we wished to examine what previous projects had discovered in using a variety of learning technologies for library user education. user education in libraries librarians interested in exploring a courseware-based approach to library user education have available to them a well-developed tradition of thinking about information skills teaching. this has been summarized elsewhere (tiefel, 1995), and it is not the intention of this paper to re-present this material. however, it is worth noting how this tradition has shaped applications of educational technology to information skills teaching. user education in libraries evolved at the end of the nineteenth century. it appeared that library users were failing to make best use of library and other information resources because of a number of factors (salony, 1995). firstly, users did not possess the practical skills needed to exploit libraries. but beyond this, an intelligent information user needed a more complex set of intellectual skills, habits and attitudes. tucker (1979) summarizes these as 'the art of discrimination', together with independent or life-long learning skills. these higher level skills have been codified under the banner of 'information literacy' (snavely and cooper, 1997). this thinking has developed along a dual path. there is a tradition of theoretical deliberation about the pedagogy of information skills teaching, and alongside it there is a practitioner tradition of documenting practice which shows how far library professionals have been able to embody good instructional models in their teaching. thus, to cite one example from many, the practitioner hanson (1985) took instructional models from bruner and gagne and applied them within the sphere of library user education. such models emphasized important facets of the learning process. hanson's work took from bruner (1966) the need to acknowledge the nature of the learner and the way in which the learner obtains knowledge. drawing on gagne (1977), hanson also tried to give learners opportunities to demonstrate how they have understood the rules of information systems while receiving feedback on their performance during practice in 'spaced reviews'. one commentator (piette, 1995) noted that such educationally well-founded work tends to be frustrated by the shortcomings of the library instruction environment. since user education takes place outside the regular instructional schedule, it takes place in a vacuum, leaving teacher librarians with few opportunities to create an enriching process of on-going review, drill or practice. academics or course tutors, in support of whose teaching library user education is offered, are far better placed to create such a learning experience. it is the academic who sees how information is used to inform a student's essay, and who is best positioned to give feedback and advice to the student on improving such use. librarian27 nick)oint et al • courseware in academic library user education tutors can only really describe the mechanics of library use in occasional decontextualized 'information skills' sessions. moreover, it is only rarely that lecturers, together with library staff, share the role of examining and improving patterns of information exploitation within a course, although the literature does recognize the existence and importance of such collaborations (carlson and miller, 1984). nevertheless, there is a long history in user education practice of elaborating the concept of 'information literacy' as a subject in its own right, that is, as a subject which can be taught as a part of the standard academic curriculum outside the courses which information skills teaching normally supports (rader, 1990). the conceiving of user education as more than a simple set of practical skills, rather as a larger philosophy of information use, is an important attempt to overcome the shortcomings of the library instruction environment and has had a significant impact on the content of user education programmes. broadly speaking, this ambitious user education syllabus has moved away from teaching skills that are based on simple, mechanical tools, in favour of generic and rather abstract searching principles that apply equally to any information tool (on-line catalogue, bibliographic database, internet search engine). tiefel (1995) points out that this syllabus is comprehensive and broadly applicable, and it also facilitates the larger ambition of leading the student towards the development of critical thinking skills. however, in response to this, there is a practitioner school of thought which is hostile to the over-elaboration of the information skills programme. this school argues that the content of the user education syllabus should be essentially practical and modest, and that the very existence of user education as an activity in its own right may be more a reflection of the inability of librarians to make their libraries and database services usable (eadie, 1990; pacey, 1995). if libraries and information systems were made easier to use, then the practical skills component of information literacy courses would effectively disappear, leaving an inflated array of philosophical teaching aims that could only make sense within the framework of proper, mainstream academic subject teaching. the potential of information technology to make libraries much easier to use suggests that the elimination of the practical skills barrier to information use may be nearer than ever before, and with it the need for most library user education (savenije, 1999). the gaels project the courseware development aspect of the gaels project, in common with most library user education activity, was brought into existence by the perception of an information skills training need (salony, 1995). it is the established practice of most academics and researchers to depend on well-resourced library provision of hard copy journals in science, technology and medicine for current awareness and updating purposes rather than electronic sources. user education programmes have failed to modify this pattern of information use (hallmark, 1994). this restricts scientific research communities to the physical environs of a university campus library and is extremely resource-intensive. where similar research communities exist in close proximity to each other, for example within a metropolitan area such as the city of glasgow, there is inevitably a highly expensive duplication of printed library materials across a small geographical area. more 28 alt-] volume 9 number 3 importantly, it seemed likely that failure to exploit networked information resources was in itself an undesirable state of affairs. given the well developed nature of modern electronic library services, and the potential for data-sharing across the west of scotland metropolitan area network, it seemed that a courseware-based attempt to re-educate the distributed engineering research communities of the universities of glasgow and strathclyde into using networked access to library resources might be effective. however, library skills courseware implementations at glasgow university such as the tilt project (creanor, duradell and primrose, 1996) had had little real impact on user education practice despite being acknowledged as successful examples of educational technology. thus, they seemed to show that a courseware-based approach to modifying user behaviour would have little effect in practice in re-educating research engineers in their pattern of library use, unless the reasons behind the lack of adoption of these previous packages were understood. the fact that we were aiming at a sophisticated, research-level group of library users meant that the package should aspire to more than simple skills-based learning if it was to gain acceptance. since our intended users were highly intelligent and well-motivated, they were hardly incapable of grasping the basic search techniques required by a number of wellproduced database services. it was at least arguable that the reasons for their failure to adopt networked information retrieval practices might be a higher-level educational issue, possibly one that could be understood within the framework of information literacy (joint, kemp and ashworth, 2000). it would be important to find case studies which genuinely managed to teach the core skills of information retrieval and achieve meaningful levels of effective implementation in practice. but also, bearing in mind critics of the concept of information literacy, it would be valuable for us to find library skills courseware case studies which managed to create enriching learning experiences without falling prey to charges of irrelevant educational pretension. non-web case studies (a) computer-aided instruction via hypermedia early courseware development successes the first well-documented uses of computer-aided instruction (cai) technologies in academic library user education occurred in north american libraries during the mid1980s. the technologies used may be presumed to be 'hypermedia' only in a loose sense, since these studies used now obsolete packages to produce results more readily achievable by modern software. a number of successful outcomes were noted, with the courseware approach emerging favourably in comparison with traditional teaching methods (nipp and straub, 1986; lawson, 1989). developing a more ambitious library skills teaching package these optimistic early pre-web case studies were picked up by a later case study conducted between 1994 and 1996 (kaplowitz and contini, 1998). this study noted the work mentioned above (lawson, 1989; nipp and straub, 1986) together with other optimistic studies of cai in libraries (pask, 1988; piette and smith, 1991; cherry, 1991). however, the authors were cautious about the existing cai evaluations in the literature, noting, for example, that neither pask nor piette and smith assessed students' skills levels after taking 29 nick joint et al courseware in academic library user education the cai. moreover, kaplowitz and contini cite other writers (harrington, 1989; kulik, 1989; welsch and loomis, 1990) who criticized the potential of cai for real interactivity with students. kaplowitz and contini used their insight into the literature of library cai to conduct their own methodologically sound and very well-resourced cai case study, successfully creating a library skills package for undergraduate biologists at ucla which aimed at teaching more than simple information skills. despite their initial pessimism, their study did create a courseware package capable of producing results at least equivalent to those of traditional user education methods. resource-intensive nature of sophisticated library skills teaching although this study was optimistic about learning outcomes for library cai, it was pessimistic about resourcing. the package was very expensive and labour-intensive to develop, so that it was unclear whether it would be revised and used in future. a pattern was developing. in the 1980s, library courseware seemed to work well in teaching simple tasks where drill is required for mastery (turner, 1990). effective packages were created with simple interactive activities and sustainable resourcing which could be maintained in future. resourcing the development of more sophisticated learning materials was problematic (kaplowitz and contini, 1998). thus, if more ambitious outcomes are achievable in library skills training, the resourcing required to create such packages appears to be excessive. however, the limited emphasis on sophisticated learning in simple library skills training means that hypermedia packages have a valuable, sustainable role in basic library instruction. subsequent pre-web studies (dixon, garrett and smith, 1995; van der meer and galen, 1996) confirm this, teaching basic library skills with sound evaluations and good learning outcomes. (b) computer-mediated communication methods a number of interesting cmc-based distance-learning course case studies were documented in the 1990s. these studies described teaching innovations based on email communication with dispersed groups of students. they often taught the technologies that were delivering the teaching the internet, usenet, listservs, and the like and were delivered as correspondence courses (burke, 1996; vishwanatham, wilkins and jevec, 1997). more library-specific courses focused on simple information tools, such as a database interface (kelsey, 1999; jensen and sih, 1995; butros, 1997). initially, hypertext courseware technologies were not used with cmc. this was inevitable, given the incompatibility of the technologies at that time. interactive pre-web stand-alone hypermedia courseware was offered to students by in-person workshops while distance learning was synonymous with cmc-based courseware. by contrast, once web platforms became available to libraries, networking of hypertextual packages became easier and courseware developers could allow themselves more flexibility of approach (mazoue, 1999). the drive to combine a cmc-approach to courseware with other technologies derives from perceived limitations of the medium. exposition of material via cmc technologies such as email is deemed to be labour-intensive, and so is the subsequent interaction and feedback (herther, 1997). the labour-saving automated exercises and feedback of the noncmc-dependent hypermedia-based library packages are a great practical benefit. the 30 alt-] volume 9 number 3 activity and feedback functions of such packages require less support from tutors, even though their drill-based exercises could be seen as simplistic. ixitcr cmc case studies later successful case studies based on synchronous rather than asynchronous cmc technologies such as conferencing packages and chat rooms confirm the resource-intensive nature of this medium (pival and tunon, 1998; simoneaux, d'angelo and jung, 1999). as previously, problems of distance were dealt with, but staffing resources remained an issue. moreover, the idea of replicating contact lecturing via conferencing software for library skills teaching raises general questions about the suitability of the contact lecture for basic library education. because the lecture is a narrative educational format, it suits some topics well, for example, chronological historical subjects, or the step-by-step building of philosophical argument, whereas simple information skills teaching tends to consist of self-contained descriptions of tools, concepts or processes, none of which combine together in a linear sequence of cumulative argument or incremental drama. pival and tunon (1998) advise keeping the training segment relatively short because of the difficulty of sustaining lecture-type expositions. however, subsequent studies indicate that cmc-based solutions are suitable for more ambitious types of information-related teaching, such as teaching students of information management or postgraduate information scientists, either when used on their own (mcmurdo and meadows, 1996) or with other approaches (van brakel, 1999). resource-intensive nature of sophisticated library skills teaching in summary, there is an impressive body of library cmc case studies with adequate evaluative data which show that this is a valuable technology with positive learning outcomes. if used to the exclusion of other technologies, it requires good staff resourcing to succeed. it displays no resourcing advantages over non-web-based hypermedia in delivering more advanced skills training. library skills tutors with the requisite support who feel that this is a suitable approach may choose it for their cal implementations with a fair degree of confidence. case studies of web-based courseware the move from packages using software such as hypercard and toolbook to htmlgenerated hypermedia resources delivered via the web is marked by one simple but powerful trade-off. the more rigorous analytic approaches of earlier computer-aided instruction projects are relaxed as courseware implementations become more prevalent in everyday library teaching practice. practitioners' sense of ownership of the courseware medium in the united states, the widespread adoption of html/web technologies democratized the courseware development process. the mystique was removed from package authoring, and in some cases librarian teachers were empowered to become courseware developers themselves (laverty, 1997; murphy, 1998). this is less true of the united kingdom, where there is a longer-lived tendency for courseware development to remain the preserve of supra-institutional showcase projects outside the practitioner mainstream. the glasgow university-based tilt library cal project (creanor et al, 1996) created a toolbook package (later moved to a web platform). as with pre-web american _ nick joint et al courseware in academic library user education implementations, the pedagogy is rigorous and the evaluation exemplary. but the tilt remit required that the package had to be generic, not site-specific. however, this generic quality did not lead to the package being extensively adopted at other sites. this generic approach is typical of uk library courseware projects, and may explain why courseware approaches have spread less widely in uk libraries (sconul, 1998) as opposed to us libraries (bevilacqua, 1993). if generic british packages, such as glasgow's, or the into info learning materials created by the educate project (thomasson and fjallbrant, 1996; fjallbrant, 1996), have had little documented impact in the uk outside the institutions which created them, perhaps uk courseware implementations will grow by following the american model of locally authored cal packages. informing versus educating (a) web hypertextualpublishing as effective teaching as learning technologies have become increasingly the tool of the practitioner, a loss of focus on certain principles has brought disadvantages. firstly, the distinction between an information resource and an educational resource has become blurred. this can be seen in the second half of the 1990s in articles about direct web conversions of 'user education' handouts or in articles which explore the web as a 'passive' hypertext publishing medium without using drill-based interactive routines (halvorsen, 1997; mosley, 1998). as noted previously, benyon and woodruffe dispute that there is educational benefit from simply hypertextualizing existing material as web/html resources. halvorsen's and mosley's hypertext web publications certainly succeed as information resources, but it is not clear in what sense this success has an instructional outcome. this is unfortunate, because case studies which demonstrate the educational value of hypertext learning resources without drill-based routines do offer a way of escaping from simplistic multiple-choice-style on-line activities. there are, however, examples where 'passive' hypertext pages are claimed by the developer to be bibliographic instruction. the educate project has been viewed as an internet publishing case study (o'riordan, 1996) and as an interactive educational project (fjallbrant, 1995). this case is made more clearly in the teaching of legislative processes as an aid to retrieving legal information (hoffman, 1997). for hoffman, 'simply hypertextualizing the material' (benyon's phrase) is educationally beneficial because the resultant overview of the legislative web on a hypertext map brings the student fully into the mechanism of creating new laws. as laurillard points out (1993: 130), hypertext learning occurs readily 'when the world is itself textual'. to summarize, the blurring of the distinction between passive web/html publishing and genuine educational resources can sometimes be seen as a category error. on the other hand, a resource can genuinely work as a set of teaching materials while remaining largely a form of hypertext publishing without simplistic, drill-based exercises. a direct html conversion of a high-quality correspondence course textbook, or a richly hyperlinked learning environment (hoffman, 1997) are both good examples. (b) virtual library tours as educational activities the claims for the instructional value of virtual library tours are similarly complex, since these web resources are essentially informative, comprising exposition of material without activity and feedback. mosley (1996) describes a rich web user education resource 32 alt-j volume 9 number 3 presented as a tour through a library. other uk web-based courseware investigations (gorman and lees, 1995; biddiscombe, rnowles and upton, 1997) also prefer informational rather than educational outcomes, as do similar us case studies (borah, 1997; murphy, 1998). all these papers describe excellent library web pages, but their usefulness may be informational not educational. descriptions of them do not emphasize interactive teaching and learning achieved by drill-based routines, nor by hypermedia features. educating rather than informing: true web/html library learning materials library web resources become more demonstrably learning materials when they leave behind the metaphor of the instructional handout and the library tour. both metaphors imply a user-librarian relationship, where the user passively receives information from an information provider. once user education web resources are created within the framework of the taught class, the interactive relationship of the tutor-student supplants the passive user-librarian relationship. the level of educational interactivity often remains elementary, however. a pattern for web/html library learning materials a typical pattern emerges. a package is created to support a subject-specific class (orians and sabol, 1996); or alternatively, a course in communications, professional or study skills provides a curriculum context for the library skills training (parise, 1998; cribb and woodall, 1997). the academic level is undergraduate not research. occasionally courseware focuses on a free-standing library information retrieval tool, such as a single database or database host, but this is rare (prestamo, 1998; hawkins, 1997). more often a range of external networked services and local library facilities are described. such library courseware consists of one or more modules, which offer a linear path through a set of web pages. the student follows a primarily textual exposition of material, enriched with graphics, screenshots and/or hyperlinks to information resources. a drillbased quiz often concludes each module; and a more ambitious final assignment based on the whole package may be given. satisfactory completion of the quizzes and/or larger piece of work is integrated into passing the departmental course supported by the package. email interaction with library staff, peers or faculty often occurs, but is not the only means of interactivity and feedback. occasionally a particular feature of these packages is highlighted, for example the automation of assessment (niemeyer, 1999). however, scholz, kerr and brown (1996), greenwood and frisbie (1998), lehner and jacobson (1997) and nozero, whitton and heaton (1998) are more typical, giving equal emphasis to most features of this model. evaluation ofweb/html library learning materials if interactive features are prominent here, the interest in learning outcome evaluation is less pronounced. occasionally evaluation of learning outcomes becomes as prominent as in pre-web case studies. sabol (1998) expands earlier work (orians and sabol, 1999) into a separate paper on evaluation. this is not typical. later studies also abandon evaluated comparisons of learning outcomes between traditional methods and courseware approaches. one purpose of such comparative evaluation is to convince a sceptical audience of the merits of learning technologies in library training, and its decline is regrettable. 33 nick joint et al courseware in academic library user education web/html library learning materials: learning styles and interface issues two other aspects of courseware implementation without significant treatment are the effective accommodation of differing learning styles and interface issues. although adjusting teaching content to learning styles formed an important strand in the uk edulib project (1999), we found little emphasis on this approach in the literature (wood, ford and miller, 1996). to be usable, learning materials need to have a good interface, and they need to be found easily within a web environment. laverty (1997), describes the effective use of html tables as interfaces to individual sets of materials, while fjallbrant, bostrom and ekman (1997) describe pathfinders, a task-based interface. however, there appears to be no published work dealing with the integration of library learning materials into surrounding local web pages. 'conclusions about web/html case studies the wider adoption of courseware approaches to library skills teaching in the us is an encouraging trend, though wider adoption has been accompanied by less rigorous attention to pedagogic issues. this can cause problems. more recent library courseware projects have failed due to confusions in educational design untypical of earlier projects (sonntag, 1999). however, lack of rigour in using pedagogic techniques such as evaluation in library teaching practice may be caused by lack of resources rather than a lack of awareness (bober, poulin and vileno, 1995). these later case studies also show that the true web/html on-line library skills class resembles earlier case studies. information skills are taught by simple but effective interactive exercise-and-feedback models, but generally at undergraduate level. using a generic package for research students can be problematic (creanor et al., 1996). thus, despite successes with simpler skills training for undergraduate populations, the use of web/html technologies in libraries has not yet demonstrated simple, resource-friendly ways of creating more ambitious learning outcomes. (3) non-case study literature and analyses of library courseware collaboration expert reviews and practitioner advice literature there is a recent trend not to document library courseware implementations, presumably because there is little incentive to report what appears increasingly to be accepted practice. expert reviews of courseware, which look at packages themselves rather than at literature about them, thus have a role in creating an overview of otherwise undocumented activity (cox, 1997). dewald (1999a and 1999b) provides a combination of expert review, pedagogical theory and advice on how to create good courseware. dewald's papers show that, when a new technology comes into professional practice, there is a role, not just for expert review, but also for 'how-to-do-it' literature. certain early case studies are sometimes used as general exemplars (pask, 1988; kaplowitz and contini, 1998), while others (burke, 1996; jensen and sih, 1995) are used as exemplars for cmcbased approaches. practitioner librarians also look to the intelligent magazine-style advice article (ardis, 1998; bell, 1998), or a synthesis of expert review and theory (nipp, 1998). 34 alt-j volume 9 number 3 practitioner librarians may sometimes refer to non-library material, ranging from the introductory (mazoue, 1999; driscoll, 1998) to the more theoretical (benyon and woodruffe, 1997; laurillard, 1993). it is striking however, that much of the best general writing on courseware development and instructional research is not cited by most case studies. thus, the work of influential organizations such as the cti centre for library and information studies (ctilis, 2000; hopkins, 1996; hopkins, 1999), and projects such as netlinks (1998) at the university of sheffield (levy, bowskill and worsfold, 1998) have yet to be fully exploited by the information practitioner. it is likely that the quality of current information skills courseware is compromised by failure to exploit such educational research. studies of collaboration in library courseware development there is little discussion in the literature of collaboration between different institutions in producing library courseware. while collaboration between dispersed campuses of a single institution is common (pival and tunon, 1998, and vishwanatham et al, 1997), there is little cross-institutional collaboration reported. as noted above, the establishment of a sense of ownership of a package limits most courseware implementations to single institutions. early optimistic reports of collaborative economies of scale in creating library learning materials (clark, 1981) should be viewed cautiously. wood and agogino (1996) have described cross-institutional engineering courseware but faulhaber (1996) notes that, although networking learning materials creates economies of scale in multi-site language teaching, it does not do so in its library aspects. statements that libraries can import generic packages without alteration are problematic (lepge and reid, 1998). us practice shows how packages tend to be local, not bought in. higher education information environments are often highly individualized, generating site-specific information skills syllabuses. this is not the case with courseware for many mainstream academic subjects, where the undergraduate science and engineering syllabus is broadly similar between institutions. legge and reid may be assuming similarities between library skills and mainstream subjects that do not exist. moreover, barriers to library collaboration may also lie in the organizational context, as analysis of an internet/library instruction course for the utah western governors virtual university has shown (hansen and lombardo, 1997). summary and conclusions the american literature shows that library computer-aided learning packages have entered mainstream practice to a greater extent than in the united kingdom, and that they have done so due to an emphasis on certain factors: • the practitioner sense of ownership of the courseware medium and materials; • sustainable resourcing in terms of skills levels to maximize maintainability; • tailoring to a local information environment rather than generic solutions; • concentrating on simple skills training for large bodies of undergraduates. 35 nick joint et 0/ courseware in academic library user education recommendations for library user education (in practice) • us library courseware projects can afford to create learning packages purely for local use. because british projects need to spread costs over many sites, we recommend creating institutional courseware packages that can then be reduced to generic web/html templates. such templates can be shared with other uk sites who, in turn, tailor them to local requirements. this combines the economies of scale of generic multi-site courseware with the need for local institutional identity. • early library courseware implementations combined a thoughtful approach based on wide background reading with a strong emphasis on evaluation of learning outcomes. courseware development today would benefit from adopting similar approaches, drawing on equivalent contemporary research (ctilis, 2000; netlinks, 1998). implications for the gaels project our response in the gaels project has been to make evaluation a central part of the project and to follow the template idea through, obtaining a solution that can be copied and transferred between institutions. although such template-based approaches have found little favour in non-library applications, librarian practitioners at glasgow and strathclyde have independently authored a biomedical courseware information skills package (gaels biomedical, 2000) using templates generated from the original engineering package created by a specialist courseware designer. this package has been successful in evaluated workshops with life scientists. because there is little evidence that drill-based interactive packages are suitable for research-level information skills students, we have adopted a bypertextual approach to interactive learning (hutchings, hall and briggs, 1992; mckendree, reader and hammond, 1995). this hypertextual approach has a number of benefits: • templates derived from hypertextual resources are open, flexible and easily reconfigured, avoiding the inhibiting constraints of other template-based approaches. this approach avoids the technical complications of heavily scripted routines, thus facilitating practitioner authoring of courseware. • practitioners feel that they have ownership of the courseware-authoring medium and can make their packages institution-specific. • the need for expensive technical support is also dispensed with, thus avoiding the resourcing problems of otherwise successful library courseware implementations which ultimately prevent the packages being widely adopted once project funding is withdrawn. • a rich hypertextual learning environment is suitable for engaging the learner in the higher-level information usage issues that underpin researchers' information-gathering practice. the mechanistic scripted routines pervading most information skills courseware are not suited to this, and if adopted by us, would not offer an effective means for modifying researchers' dependence on browsing local hard copy research collections. we believe that at the time of writing this is the first example in the uk of librarian practitioners with no previous background in courseware design successfully authoring 36 alt-j volume 9 number 3 their own information skills courseware packages along the lines of us case studies. we thus believe that gaels offers a good model of courseware implementation for other uk libraries to emulate, and suggest that our strategies for involving practitioners directly in the process of courseware development may be useful in other non-library subjects where practitioner involvement is required. references ardis, s. (1998), 'creating internet-based tutorials for bibliographic instruction', information outlook, 2 (10), 17-20. bell, h. a. (1998), 'the librarian as trainer: internet training lessons learned', information outlook, 2 (4), 17-20. benyon, d. and woodruffe, m. (1997), 'experience with developing multimedia courseware for the world wide web: the real need for better tools and clear pedagogy', international journal of human-computer studies, 47 (1), 197-218. bevilacqua, a. (1993), 'computer-assisted bibliographic instruction', in l. hardesty (ed.), bibliographic instruction in practice: a tribute to the legacy of evan farber, ann arbor: pierian press, 67-76. biddiscombe, r., knowles, k. and upton, i. (1997), 'developing a web library guide for an academic library: problems, solutions and future possibilities at the university of birmingham library', program, 31, 59-73. bober, c., poulin, s. and vileno, l. (1995), 'evaluating library instruction in academic libraries: a critical review of the literature, 1980-1993', the reference librarian, 51/2, 53-71. borah, e. g. (1997), 'a virtual 'hello': a web-based orientation to the library', journal of business and finance librarianship, 3 (2), 73-8. bruner, j. (1966), towards a theory of instruction, cambridge: harvard university press. burke, j. j. (1996), 'using e-mail to teach; expanding the reach of bi', research strategies, 14 (1), 36-43. butros, a. (1997), 'using electronic mail to teach melvyl medline at the university of california biomedical library', medical reference services quarterly, 16, 69-75. carlson, d. and miller, r. h. (1984), 'librarians and teaching faculty: partners in bibliographic instruction', college and research libraries, 45 (6), 483-91. cherry, j. m. (1991), 'an experimental investigation of two types of instruction for opac users', canadian journal of information science, 16, 2-22. cox, a. (1997), 'using the world wide web for library user education: a review article', journal of librarianship and information science, 29, 39-43. creanor, l., durndell, h. and primrose, c. (1996), 'library and study skills using hypertext: the tilt experience', the new review of hypermedia and multimedia, 2, 121-47. cribb, g. and woodall, l. (1997), 'webbook for engineers: an interactive information skills programme', new review of information networking, 3, 245-53. 37 nick joint et al courseware in academic library user education ctilis, the cti centre for library and information studies (2000), url: http://lboro.ac.uk/departments/dils/cti/ [last visited 28 july 2000]. dewald, n. h. (1999a), 'web-based library instruction: what is good pedagogy?', information technology and libraries, 18 (1), 26-31. dewald, n. h. (1999b), 'transporting good library instruction practices into the web environment: an analysis of online tutorials', the journal of academic librarianship, 25 (1), 26-32. dixon, l., garrett, m. and smith, r. (1995), 'building library skills: computer-aided instruction for undergraduates', research strategies, 13 (4), 196-208. driscoll, m. (1998), 'how to pilot web-based training', training and development, 52 (11), 44-51. eadie, t. (1990), 'immodest proposals: user instruction for students does not work', library journal, 115(17), 42-5. educate (1996), the educate project into info courseware www site, http://educate1.lib. chalmers.se/eduinfo.html [last visited 16 january 2000]. edulib (1999), the edulib project web site, http://www.tay.ac.uk/edulib/index.html [last visited 2 march 2000]. faulhaber, c. (1996), 'distance learning and digital libraries: two sides of a single coin, experience teaching a graduate class in old catalan language and literature from uc berkeley to uc irvine and uc san diego', journal of the american society for information science, 47, 854-6. fjällbrant, n. (1995), 'educate: a user education program for information retrieval and handling', new review of information networking, 1, 119-27. fjällbrant, n. (1996), 'educate: a networked user education project in europe', ifla journal, 22 (1), 31-4. fjallbrant, n., bostrom, r. m. and ekman, m. (1997), 'into info (educate) www based programs for information education, training and access', 63rd ifla general conference conference programme and proceedings, 31 august-5 september 1997, http://ifla.inist.fr/iviifla63/63fjan.htm [last visited 10 february 2000]. gaels biomedical (2000), http://gaels.lib.strath.aauk/biomed/ [last visited 28 july 2000]. gaels project courseware (1999), http://gaels.lib.gla.ac.uk/infoskills1/ [last visited 21 february 2000]. gagne, r. m. (1977), the conditions of learning, austin: holt, rinehart & winston. gorman, p and lees, r. (1995), 'designing infopoint: a multi-purpose multimedia library guide at liverpool john moores university', new review of academic librarianship, 1, 85-94. greenwood, t. e. and frisbie, j. (1998), 'library instruction that's out of this world! information literacy at fort lewis college', reference services review, 24 (4), 45-50. hallmark, j. (1994), 'scientists' access and retrieval of references cited in their recent journal articles', college and research libraries, 55 (3), 200-8. 38 alt-j volume 9 number 3 halvorsen, a. l. (1997), 'the two instructional faces of the web: information resource and publishing tool', internet reference services quarterly, 2 (2-3), 67-76. hansen, c. and lombardo, n. (1997), toward the virtual university: collaborative development of a web-based course', research strategies, 15(2), 68-79. hanson, j. r. (1985), 'teaching information sources in business studies: an application of the theories of j. bruner and r. m. gagne', journal of librarianship, 17 (3), 185-99. harrington, j. p. (1989), 'computer-assisted instruction for end-users: our pandora's box', library software review, 8, 152-4. hawkins, d. t. (1997), 'web-based training for online retrieval: some examples', online, 21 (5), 73-5. herther, n. k. (1997), 'education over the web: distance learning and the information professional', online, 21 (5), 63-6. hoffman, e. (1997), 'untangling the web: a web page for bibliographic instruction on the legislative process', internet reference services quarterly, 2 (2-3), 115-32. hopkins, t. (1996), 'use of cbt packages for user education', infocus, 1 (2), http://www.lboro.acuk/departments/dils/cti/infocus2/cbt.htm [last visited 6 june 2000)]. hopkins, t. (1999), 'user education and librarians' attitudes to cbt infocus', 3 (2), http://www.lboro.ac.uk/departments/dilslcti/infocus8/resue.htm [last visited 6 june 2000]. hutchings, g. a., hall, w. and briggs, j. (1992), 'authoring and evaluation of hypermedia for education', computers and education, 18 (1-3), 171-7. jensen, a. and sih, j. (1995), 'using e-mail and the internet to teach users at their desktops', online, 19 (5), 82-6. joint, n., kemp, b. and ashworth, s. (2000), 'information skills training in support of a joint electronic library in glasgow: the gaels project approach to library courseware development', aslib proceedings, 52 (8), 301-12. kaplowitz, j. and contini, j. (1998), 'computer-assisted instruction: is it an option for bibliographic instruction in large undergraduate survey classes?', college and research libraries, 59 (1), 19-27. an accompanying bibliography with authorial commentary is available at http://www.library.ucla.edu/libraries/biomed/staff/people/caibib.html [last visited 14 february 2000]. kelsey, s. (1999), 'library user education: implementing an e-mail bibliographic instruction course at a college library', lla bulletin, 61 (4), 222-5. kemp, r. (1999), 'training materials for advanced information skills in engineering and architecture: a web-based approach to advanced information skills', in candida fenton (ed.), building on the past, proceedings of the 12th arclib international conference, 21-23 july 1999, university of strathclyde and glasgow school of art, glasgow: arclib, 10-16. kulik, j. a. and chen-lin, c. (1989), 'effectiveness of computer-based instruction', school library media quarterly, 17 (3), 156-9. 39 nick joint et al courseware in academic library user education laurillard, d. (1993), rethinking university teaching: a framework for the effective use of educational technology, london: routledge. laverty, c. y. c. (1997), 'library instruction on the web: inventing options and opportunities', internet reference services quarterly, 2 (2-3), 133-2. lawson, l. v. (1989), 'using a computer-assisted instruction program to replace the traditional library tour: an experimental study', rq, 29 (1), 71-9. legge, t. and reid, b. (1998), 'user education on the web', library association record, 100 (8), 413-14. lehner, j. a. and jacobson, t. e. (1997), 'instructional opportunities of a subject-oriented (law) web page', internet reference services quarterly, 2 (2-3), 133-42. levy, p., bowskill, n. and worsfold, e. (1998). netlinks research report, section 2.62, http://neiways.shef.ac.uk/rbase/reports/section2.htm#tsnap [last visited 16 june 2000]. loven, b., morgan, k. and shaw-kokot, j. (1998), 'information skills for distance learning at the university of north carolina at chapel hill', medical reference services quarterly, 17 (3), 71-5. mazoue, l. (1999), 'the essentials of effective online instruction', campus wide information systems, 16 (3), 104-10. mckendree, j., reader, w. and hammond, n. (1995), 'the homeopathic fallacy in learning from hypertext', interactions, 2 (3), 74-82. mcmurdo, g. and meadows, a. j. (1996), 'acceptance and use of computer-mediated communication by information students', journal of information science, 22 (5), 335-48. mosley, p. a. (1996), 'touring the campus library from the world wide web', reference services review, 24 (4), 7-14. mosley, p. a. (1998), 'making library instructional handouts accessible through the world wide web', research strategies, 16 (2), 153-61. murphy, d. a. (1998), 'building the library starter kit: developing new technologies for instruction', reference services review, 26 (3-4), 109-16. netlinks(1998), 'project web site', http://www.shef.ac.uk/~np/ [last visited 16 june 2000]. niemeyer, c. (1999), 'a computerized final exam for a library skills course', reference services review, 27 (1), 90-106. nipp, d. (1998), 'innovative use of the home page for library instruction'; research strategies, 16 (1), 93-102. nipp, d. and straub, r. (1986), 'the design and implementation of a microcomputer program for library orientation', research strategies, 4 (2), 60-7. nozero, v. a., whitten, p. a. and heaton, s. (1998), 'a flexible instruction programme: how demographics drive the design', reference services review, 26 (3-4), 61-70. orians, c. and sabol, l. (1999), 'using the web to teach library research skills in 40 alt-] volume 9 number 3 introductory biology: a collaboration between faculty and librarians', issues in science and technology librarianship, 23, http://www.library.ucsb.edu/ist//99-summer/article2.html [last visited 29 february 2000]. o'riordan, g. (1996), 'internet publishing case study: ulix and educate', iatul proceedings (ns), 5, 53-67. pacey, p. (1995), 'teaching user education, learning information skills; or, towards the selfexplanatory library', the new review of academic librarianship, 1, 95-103. parise, p. (1998), 'information power goes online: teaching information literacy to distance learners', reference services review, 26 (3-4), 51-2, 60. pask, j. m. (1988), 'computer-assisted instruction for basic library skills', library software review, 7(1), 6-11. piette, m. i. and smith, n. m. (1991), 'evaluating a hypercard library instruction program', research strategies, 9 (2), 87-94. piette, m. i. (1995), 'library instruction: principles, theories, connections, and challenges', the reference librarian, 51/52, 77-88. pival, p. r. and tunon, j. (1998), 'netmeeting: a new and inexpensive alternative for delivering library instruction to distance students', college and research libraries news, 59 (10), 758-60. prestamo, a. m. (1998), 'development of web-based tutorials for online databases', issues in science and technology librarianship, 17, http://www.library.ucsb.edu/istll98-winter/ article3.html [last visited 29 february 2000]. rader, h. b. (1990), 'bringing information literacy into the academic curriculum', college and research libraries news, 51 (9), 879-80. sabol, l. (1998), 'the value of student evaluation of a web site', research strategies, 16 (1), 79-84. salony, m. (1995), 'the history of bibliographic instruction: changing trends from books to the electronic world', the reference librarian, 51/2, 31-51. savenije, b. (1999), 'integrating library instruction into student education', education libraries journal, 42 (1), 5-10. scholz, a. m., kerr, r. c. and brown, s. k. (1996), 'pluto: interactive instruction on the web: purdue libraries undergraduate tutorial online', college and research libraries news, 57 (6), 346-9. sconul (standing conference of national and university libraries) (1998), sconul annual library statistics 1996-97, london: sconul. sih, j. and hightower, c. (1994), 'when in-person classes aren't the answer: teaching inspec electronically', dra bulletin, 14 (1), 6-10. simoneaux, l., d'angelo, b. j. and jung, k. (1999), 'the library and you: curriculum integrated library instruction via the www, campus wide information systems, 16 (3), 89-99. 4? nick joint et at courseware in academic library user education snavely, j. and cooper, n. (1997), 'the information literacy debate', journal of academic librarianship, 23 (1), 9-14. sonntag, g. (1999), 'using technology in a first year experience course', college and undergraduate libraries, 6 (1), 1-16. thomasson, g. and fjallbrant, n. (1996), 'educate: the design and development of a networked end-user education program', education for information, 14 (4), 295-304. tiefel, v. m. (1995), 'library user education: examining its past, projecting its future', library trends, 44 (2), 318-38. tucker, j. m. (1979), 'the origins of bibliographic instruction in academic libraries, 1876-1914', in r. d. stuart and r. d. johnson (eds.), new horizons for academic libraries, new york: k. g. saur, 268-76. turner, a. (1990), 'computer-assisted instruction in academic libraries', journal of academic librarianship, 15 (6), 352-4. van brakel, p. (1999), 'teaching information management via a web-based course', the electronic library, 17 (6), 389-94. van der meer, p. f. and galen, e. r. (1996), 'multimedia: meeting the demand for user education with a self-instruction tutorial', research strategies, 14 (3), 145-58. vishwanatham, r., wilkins, w. and jevec, t. (1997), 'the internet as a medium for online instruction', college and research libraries, 58 (5), 433-44. welsch, e. k. and loomis, a. (1990), 'research assistant: a hypercard approach to library instruction', oclc micro, 6, 20-5. wood, f., ford, n. and miller, d. (1996), 'information skills, searching behaviour and cognitive styles for student-centred learning: a computer-assisted learning approach', journal of information science, 22 (2), 79-92. wood, w. h. and agogino, a. m. (1996), 'engineering courseware content and delivery: the needs infrastructure for distance independent education', journal of the american society for information science, 47 (11), 863-9. doi:10.3402/rlt.v24.30146 original research article shaping the future of learning using the student voice: we’re listening but are we hearing clearly? chris meadows*, kate soper, rod cullen, catherine wasiuk, colin mcallister-gibson and phil danby learning innovation, manchester metropolitan university, manchester, england (received 23 october 2015; final version received 20 september 2016) student voice data is a key factor as manchester metropolitan university strives to continually improve institutional technology enhanced learning (tel) infrastructure. a bi-annual institutional student survey enables students to communicate their experience of learning, teaching and assessment on programmes and specific units studied. each cycle of the survey contains approximately 40�50,000 free text comments from students pertaining to what they appreciate and what they would like to see improved. a detailed thematic analysis of this data has identified 18 themes, arranged into six categories relating to the ‘best’ aspects of courses, and 25 themes, arranged in seven categories in relation to aspects of courses considered to be ‘in need of improvement’. this student data was then used as a basis for semi-structured interviews with staff. anecdotally, evidence suggested that student expectations and staff expectations around tel and the virtual learning environment (vle) differed. on-going evaluation of this work has highlighted a disconnect. in significant instances, academic colleagues seemingly misinterpret the student voice analysis and consequently struggle to respond effectively. in response to the analysis, the learning technologist’s role has been to re-interpret the analysis and redevelop tel staff development and training activities. the changes implemented have focused on: contextualising resources in vle; making lectures more interactive; enriching the curriculum with audio�visual resources; and setting expectations around communications. keywords: student engagement; student voice; analytics; technology enhanced learning; student feedback introduction and context student engagement data and feedback is now a key strategic driver for universities (see alderman, towers, and bannah 2012). recruitment of students is predicated on accurate information about potential university destinations to enable decisionmaking. academics and university administrators need accurate data to help them plan, monitor and improve. in order to facilitate benchmarking and indicate market performance university’s require timely and accurate data. at a higher level, governments and sector agencies need this information to plan funding and policy development and to highlight accountability. much of the data needed to satisfy these requirements centres on the performance indicators supposedly surrounding the concept of student engagement (see harvey 2011; reed and whatmough 2015; ritchie and spencer 2002). *corresponding author. email: c.meadows@mmu.ac.uk research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 c. meadows et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 conceptually, ‘student engagement’ is not a cohesive or universally interchangeable term. it has theoretical roots in astin’s student involvement theory (astin 1999) and indeed in tinto’s student departure theory (tinto 1975). as demonstrated by trowler’s (2010) comprehensive literature review, there are distinct character differences on either side of the atlantic. student engagement is perhaps more comprehensively defined in north america and australia as the result of specific engagement questionnaires. the national survey of student engagement (nsse) and the australasian survey of student engagement (ausse) are grounded in a refined definition student engagement predicated on ‘a body of knowledge built up since the mid-1980s establishing correlation between student’s investment of time, effort and interest in a range of educationally-orientated activities, and favourable outcomes’ (trowler and trowler 2010, p. 8). both the nsse and ausse seek to ‘stimulate evidence-focused conversations about student’s engagement in university study’ (ausse 2009, p. vii). the uk, in contrast, has an absence of student engagement surveys resulting in much of the student engagement data being gleaned from other ‘traditions such as student feedback surveys (such as the national student survey, nss), student representation and student approaches to learning’ (trowler 2010, p. 3). there is a coalescing conceptualisation of student engagement in the united kingdom. the higher education funding council for england (hefce) views student engagement as ‘the process whereby institutions and sector bodies make deliberate attempts to involve and empower students in the process of shaping the learning experience’ (hefce 2008, p. 8) subsequently the hea began piloting a student engagement survey with nine institutions in 2013, rising to 32 institutions with 24,000 respondents the following year (buckley 2014). it brings the uk picture more in line with the us and australian surveys as the ukes comprises 50 questions � 39 of which are derived from the nsse. consequently, the future research into the state of student engagement is likely to be comparable. at the institutional level, manchester metropolitan university (mmu) has invested significantly in developing a technical infrastructure to support teaching, learning and assessment (stubbs 2014a). this development has paved a way for mmu to inform its activity. by developing core quality assurance mechanisms such as the institutional student survey (iss), the university has been able to capture student feedback as never before. two large-scale institutional projects have worked in sequence to improve the student experience and capture the student feedback in driving institutional change. the enhancing the quality of assessment for learning (eqal) (see bird et al. 2015) and the traffic project (see traffic 2015) brought about a tremendous amount of synchronicity between various student data systems to facilitate the quick and easy interchange of data. aligned with a series of web service apis and a core vle (moodle), this has produced a technical infrastructure that has ‘wrapped the institution around the learner’ (stubbs 2014b, p. 1). these technical developments revolutionised the approach to technology enhanced learning (tel) at mmu and laid the foundations for a quality enhancement and assurance drive. this drive has been able to leverage the student voice through feedback and utilised these insights to inform the institutional strategy from a firm evidence base. these developments have been made possible by the introduction of the iss. this survey operates during key windows (unit/programme end points) during the academic calendar. from 2008 to 2015 the survey ran bi-annually in december and march but since 2015, it runs tri-annually in december, march and june. the survey is c. meadows et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 presented to all students on taught undergraduate and post-graduate programmes primarily as a series of quantitative ‘likert’ scale questions (see appendix 1). initially on a bi-annual basis but later on a tri-annual basis students are invited to comment quantitatively and qualitatively on their experiences of teaching, learning and assessment at the university. the iss is a critical student feedback mechanism into the continuous monitoring and improvement process. a key component in this process is a sophisticated digital dashboard presenting a whole range of quantitative and longitudinal metrics on performance to unit and programme leaders to institute change within programmes of study. indeed the process aims to: . . . support the maintenance of standards, to assure the consistency of learning opportunities and to enhance the quality of the learning experience for students by continually reviewing provision, identifying areas for improvement and taking appropriate and timely actions. http://www.mmu.ac.uk/academic/casqe/experience/monitoring-improvement.php quantitative data from the iss is augmented with additional student engagement data including vle usage stats, assessment submission and attendance data where available. all of this data provides a quantitative view of student engagement. a large volume of qualitative student feedback is also gained about the student experience from the iss through two free text questions that ask students to respond to the following statements about the course as a whole and individual units: the best thing about my course/unit is; in need of improvement on this course/unit is; the december 2014 data set returned 47,800 free text comments across all levels and areas of study relating to the student experience of teaching, learning and assessment. although there are obvious uses of this data for module and programmelevel enhancement, there is also an opportunity to mine this large data set to investigate student voice and experience in relation to specific issues. this paper draws together two related pieces of work. the first is a thematic analysis of free text comments relating to the students’ experience of tel and the use of this analysis by a team of technology enhanced learning advisors (telas) to develop a targeted staff development and training programme in response. the second is an investigation of the interpretation of the tel themes identified in the analysis by academic staff colleagues supported by the telas. consequently, this paper aims to: � analyse how free text comments extracted from the main iss data set have been used to investigate student engagement and student voice data relating to the experience of tel. � demonstrate key engagement themes for tel to inform staff development and training practice by a team of telas. � examine the interpretation of identified tel themes in the student comments by the academic staff colleagues that the telas support. research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 3 (page number not for citation purpose) http://www.mmu.ac.uk/academic/casqe/experience/monitoring-improvement.php http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 methodology the approach methodology was that of bricolage (see kincheloe 2005; lincoln 2001; rogers 2012). this was a blend comprising characteristics of grounded theory (see birks and mills 2011; charmaz 2006) and derivatives of phenomenology (see o’leary 2004). it involved two stages � investigating both the student voice and the staff voice. the research involved several sequential stages mapping the narrative of the collection, analysis, interpretation and utilisation of the student voice data. this was done through a series of iterative phases: phase 1 � student voice data collection phase 2 � student data analysis phase 3 � student data visualisation/interpretation phase 4 � staff data collection phase 5 � staff data interpretation primarily the research was opportunistic as data was collected from surveys already being conducted. this frames the theoretical approach as primarily grounded theory. however, as initial outputs from the thematic analysis informed and framed the approach to interviews conducted with staff, it could be argued that there are overtones of a phenomenological inquiry. such complexity of methods in social research has been discussed by lincoln (2001), mclaren (2001), kincheloe (2001, 2005) and rogers (2012) and they have developed the ‘bricolage’ concept. bricolage is a multimethod mode of research typically understood to ‘involve the process of employing these methodological strategies as they are needed in the unfolding context of the research situation’ (kincheloe 2001, p. 324). this is not to be ‘wantonly eclectic with the critical tradition’ but moreover to ‘make the point that any attempts to delineate critical theory as discrete schools of analysis will fail to capture the evolving hybridity endemic to contemporary critical analysis’ (kincheloe, mclaren, and steinberg 2011). phase 1 � data collection as previously discussed, the data was derived from the iss and comprised of a filtered search of over 47,000 free text comments. this filtering was achieved by means of a keyword search of 24 keywords (moodle; wifi; wireless; print; pc; pcs; mac; macs; software; drop in; mobile; twitter; phone; video; ipad; tablet; facebook; podcast; email; matlab; mymmu; app; mmutube; padlet; it zone). the key words were selected during an initial scan of a subset of 1000 comments derived from the main data set of over 47,000 free text comments. in an initial familiarisation process the keywords were selected as they were considered to characterise comments pertaining to tel-related issues. this extracted 2072 comments relating to the student experience of the institutional vle and other aspects of tel; 4.5% of the total comments received in the iss pertained to tel or aspects of technology related to teaching and learning. this provided a data set of 746 comments relating to good practice; 1326 related to areas for development. phase 2 � student data analysis these data sets were interrogated using a framework approach thematic analysis (see ritchie and spencer 1994, 2002; stubbs 1994a). this involved five interconnected stages spanning several of the phases of investigation. c. meadows et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 (1) familiarisation (2) identifying a thematic framework (3) indexing (4) charting (5) mapping and interpreting familiarisation the familiarisation process began with two members of the team independently immersing themselves in the data and reading for items of interest. this familiarisation phase involved ‘text segmentation’ (see guest, macqueen, and namey 2012). this is an iterative process of defining text boundaries around identified features of interest. this process involved the analysts continually questioning where meaning begins, ends, intersects and overlaps in the text (see guest, macqueen, and namey 2012). whilst this phase is presented as the first sequentially, this process of familiarisation permeates all subsequent sequences as the analysts continually question and improve the themes identified refining their application of them. this allowed the researchers to compare bounded, segmented text fragments (guest, macqueen, and namey 2012) in order to: (1) evaluate and document the overall utility of the data (2) aid the exploration of thematic elements in terms of similarity, dissimilarity and relationships thematic framework the thematic framework was developed through the two independent analysts from the familiarisation phase comparing segmented text to tag arising themes or codes. at the most arbitrary level, this involved the application of what guest, macqueen, and namey (2012), termed ‘key-word-in-context’ (kwic). this was the identification of a word as the locus for the theme or concept being tagged. saldaña defines it as ‘a phrase or sentence that identifies what a unit of data is about and/or what it means’ (saldaña 2009, p. 139). metaphorically this is the placing of the pin in a map. the benefit of using kwic in a large data set is that it allows rapid searching as you can quickly search for these kwic words. also the analyst is able to quickly search for synonyms allowing rapid tagging of schema and development of a codebook for application. by embedding the synonym searching in the method, this strengthens the process expanding a literal search into a broader concept search. the process culminated in the production of the codebooks in tables 1 and 2. indexing a team of eight telas analysed the data using the codebooks derived from the framework setting stage. during the indexing phase several analysts revisited the familiarisation phase clarifying and improving the coding which could be more formally built into the method. macqueen et al. (2008) present an exemplified codebook template which contains a code label which is a short descriptive mnemonic. this is further expanded by a short definition. the codebook utilised here in the indexing phase comprised a hybrid version encapsulating a blend of the code label and short definition. however, the scale of the thematic analysis here perhaps justifies the condensing of the codebook. research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 table 1. best aspects of programme/unit themes code effective online communications (prompt response to online communications e.g., email, discussion postings, regular/helpful moodle announcements) b1 use of alternatives to moodle e.g., facebook for communications b4 use of discussions for communication between students e.g., those on placements b16 consistent/clear instructions on projects and assessments available (timely) b2 variety of assignment types, e.g., podcast b17 prompt/high quality feedback b18 moodle use general comments that moodle is a good thing b5 induction in how moodle will be used b6 clear instruction on self-directed work and how to use general resources in moodle provided b12 moodle resources available in advance (of teaching)/kept up to date b3 ability to print consistent/high quality learning resources from moodle b13 well organised/consistent/high quality learning resources on moodle b11 provision of content overviews (e.g., providing shorter synthesis of book chapters, papers/lecture notes) b7 provision/use of audio/video resources (in class, via youtube, mmu tube and moodle) b8 provision of lecture recordings (audio/video) b9 access to library resources b14 use of mmu app for convenience b10 interactive lectures incorporating classroom technologies e.g., text wall, ipads/apps b15 table 2. aspects of programme/unit in need of improvement themes code general issues with poor organisation of progammes/units i1 poor communication by lecturers (inaccurate info/slow response) i2 inconsistencies in information provided between tutors i3 timetables (inaccurate, subject to change, complicated to understand, difficult to keep to) i4 stop using facebook and other social media for communications i23 assessment deadline bunching i5 assessment briefs lacking/inaccurate i6 assessment criteria not available on moodle/not soon enough i7 moodle flakiness/downtime in relation to assessment submission i9 moodle flakiness/downtime in general i8 access to appropriate computers i10 poor wifi/internet issues i11 printing problems/expense i12 search facility / sort in date order function in moodle would help i24 better induction on how moodle will be used / clearer steer on how staff will use moodle as students often unaware of need to keep checking in i13 no clear instruction on self-directed work/no indication of how to use general resources in moodle i17 materials and resources not in moodle i14 materials and resources not available in moodle in advance i15 materials and resources of poor quality/poorly organised in moodle i16 powerpoint slides alone in moodle are not useful i19 slides used in lectures � not the same as those in moodle i18 insufficient use of video in teaching/moodle i20 lectures should be recorded and made available afterwards i22 inappropriate/over use of video 125 lecture/seminars should be more interactive and not just reading/repeating what is in moodle i21 c. meadows et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 phase 3 �student data visualisation/interpretation charting the charting phase involved a higher level of abstraction and organisation. in searching for connective meanings and interpretation, the analytical methodology introduced categorisation. this involved the introduction of structure onto the codes or themes through the use of meta-themes. the grouping of themes into linked categories involved the application of a higher level of meaning not directly visible in the data. the meta-themes were derived from the original codebooks and are shown in table 3. this codification is demonstrated in adaptations of tables 1 and 2 to produce tables 4 and 5, respectively. the revised codes were charted to produce initial visualisations of the data at an institutional and faculty level. initially, these were bar charts as shown in figures 1 and 2. here the occurrence of the codes was plotted across the meta-themes and themes. firstly the ‘best’ elements of the programme/unit were plotted. secondly the areas for ‘improvement’ were plotted as shown in figure 2. interpretation of these initial visualisations indicated a degree of ‘mirroring’ of certain meta-themes and themes. for example, ‘communication’ was reported a ‘positive’ facet with 127 reports of ‘communication’ as a strength of the unit. yet it was also reported as a ‘negative’ with 155 incidents where students indicated perhaps a quicker response or clearer communications were required. this ‘mirroring’ was not a contrary report by the students but indicated an inconsistent experience across programmes. used as a comparator, ‘communication’ could be reported positively in one unit that was utilised as a benchmark and to report a ‘poor’ experience in a different unit. so students are not commenting differently about the same unit but are highlighting that communications in some units/ programmes are perceived as effective and in others are not. students are positive about a fast response to questions and highly frustrated where this is not the case. in investigating this concept of ‘mirroring’ further, subsequent visualisations were produced using tableau as an image medium. these included hotspot analysis to drill into the data at a departmental level to inform the faculty reports that were produced. also ‘mirrored’ charts were visualised to investigate the mirroring concept further. mapping and interpreting the mapping of hotspots across the themes and departments generated visualisations that gave a clear perception of where issues or strengths lie. they also clearly indicate table 3. meta-themes applied to the codebook meta-themes � best thing about my program / unit meta-themes � things to improve in my program / unit communication / collaboration communciation / collaboration assessment assessment moodle use technology issues moodle organisation moodle use moodle content moodle organisation classroom technology moodle content classroom technology research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 departments that can be exercised for sharing good practice and for identifying ones that perhaps need greater levels of support. in figure 3, well performing departments can be seen by strong vertical colour bands that indicate departments performing well across numerous facets and could be leveraged by faculties to lead in good practice sharing. strong horizontal banding shows strong performance in a particular theme across numerous areas, which perhaps is more pertinent for institutional level attention. similarly, the representation of areas for improvement indicates areas of practice that need development at an institutional level by strong colour banding in horizontal themes. these may indicate areas of focus for institutional concern. strong vertical colour banding indicates particular departments that may need further support or interventions at a faculty level to support development. this can be seen in figure 4, here several themes show particularly high colour strength indicating a high degree of student comment. for example, areas of concern in figure 4 are communications and the use of powerpoint slides in the vle. the researchers wished to avoid alienating cooperative academic colleagues and the reports produced utilising the hot spot analysis where produced on a per faculty basis with anonymised results shown for other faculties. this allowed the institutional view but moved away from departments being able to compare with each other outside of their own faculties. the hotspot analysis formulated part of the reports table 4. meta-themes and themes � best aspects of programme/unit themes code communication/collaboration effective online communications (prompt response to online communications e.g., email, discussion postings, regular/helpful moodle announcements) b1 use of alternatives to moodle e.g., facebook for communications b4 use of discussions for communication between students e.g., those on placements b16 assessment consistent/clear instructions on projects and assessments available (timely) b2 variety of assignment types, e.g., podcast b17 prompt/high quality feedback b18 moodle use general comments that moodle is a good thing b5 induction in how moodle will be used b6 clear instruction on self-directed work and how to use general resources in moodle provided b12 moodle organisation moodle resources available in advance (of teaching)/kept up to date b3 ability to print consistent/high quality learning resources from moodle b13 well organised/consistent/high quality learning resources on moodle b11 moodle content provision of content overviews (e.g., providing shorter synthesis of book chapters, papers/lecture notes) b7 provision/use of audio/video resources (in class, via youtube, mmu tube and moodle) b8 provision of lecture recordings (audio/video) b9 access to library resources b14 use of mmu app for convenience b10 classroom technology interactive lectures incorporating classroom technologies e.g. text wall, ipads/apps b15 c. meadows et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 they were sent to faculty executive groups for action and comment. some of the results of which are discussed later. further mappings of the ‘mirroring’ effect were also produced to investigate further which themes were prevalent. the early visualisations as displayed in figures 1 and 2 were considered too visually busy to be easily interpreted. the categories were compared in an adapted area graph with positive results displayed on the top and negative results presented underneath. as seen from table 3, the meta-themes across both the ‘best’ and ‘to improve’ elements were not uniform, so outlier elements were removed from the analysis. the plot shown in figure 5 shows the resulting comparison between the meta-themes that remained for comparison. the resulting visualisation clearly identified five significant areas that the students commented upon in terms of their experience. the distinct yellow plot referred to ‘general positive comments about moodle’ which the analysts discarded as it table 5. meta-themes and themes � aspects to improve on the programme/unit themes code communication/collaboration general issues with poor organisation of progammes/units i1 poor communication by lecturers (inaccurate info/slow response) i2 inconsistencies in information provided between tutors i3 timetables (inaccurate, subject to change, complicated to understand, difficult to keep to) i4 stop using facebook and other social media for communications i23 assessment assessment deadline bunching i5 assessment briefs lacking/inaccurate i6 assessment criteria not available on moodle/not soon enough i7 technology issues moodle flakiness/downtime in relation to assessment submission i9 moodle flakiness/downtime in general i8 access to appropriate computers i10 poor wifi/internet issues i11 printing problems/expense i12 search facility / sort in date order function in moodle would help i24 moodle use, organization and content use better induction on how moodle will be used / clearer steer on how staff will use moodle as students often unaware of need to keep checking in i13 no clear instruction on self-directed work/no indication of how to use general resources in moodle i17 organisation materials and resources not in moodle i14 materials and resources not available in moodle in advance i15 materials and resources of poor quality/poorly organised in moodle i16 content powerpoint slides alone in moodle are not useful i19 slides used in lectures � not the same as those in moodle i18 insufficient use of video in teaching/moodle i20 lectures should be recorded and made available afterwards i22 inappropriate/over use of video 125 in class lecture/seminars should be more interactive and not just reading/repeating what is in moodle i21 research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 appeared a catchall rather than a meaningful research proposition. the remaining four characteristics that received significant student comment focused on: (1) effective communications (2) moodle organisation and high quality materials (3) use of audio visual resources (4) desire for ‘interactive’ lectures. (5) moodle a ‘good’ thing (yellow area discounted as discussed above) the subsequent phases sought to explore the staff perception in relation to the outcomes in order to attempt to determine if there was awareness of these issues amongst academic colleagues and to suggest possible interventions. at this juncture, the role of the telas was perhaps seen as being a lens through which the data could be interpreted on behalf of academic colleagues. it was felt by the researchers that the expressions of the student voice centred on some key concepts with regard to the student experience within the environs of technology. for example, ‘effective communications’: there is a plethora of possibilities with regard to what these means on practical level. does it refer to speedy response to email queries? does it refer to meaningful communication rather than being time specific? does it refer to effective contextualising of resources made available in the vle for students to use by using effective and efficient instructions? there are a figure 1. institutional overview of ‘best’ themes. c. meadows et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 variety of possibilities. as such the researchers felt that there may be a role for them in positioning the student voice feedback. where this interpretation should appear in the sequence of the research is still a moot point. in this study, it was positioned following staff investigations but perhaps in subsequent iterations of this research the action research cycle might position this interpretive phase between student data analysis and the staff data collection. these results highlighted target areas for intervention. from a tel perspective, interpretation of these results pointed towards clear intervention strategies. effective communications was interpreted as a need to develop a mutual expectation among staff and students around clarity of communication and setting expectations around the immediacy of communication. also, it pointed towards strategies for minimising communication need by providing greater context and instructional notes round resources made available. much of the perceived need for communication stemmed from assessment materials and deadlines that emphasised the need for academic colleagues to focus more attention on to the provision of meaningful communication in relation to assessment. in a similar vein, feedback pertaining to moodle use, emphasised the need for academic colleagues to provide greater contextualisation and instructional direction to resources in the vle to drive the usage and utility value of them. students cited further, and more targeted use of audio�visual resources as a desirable outcome. often the use of these resources were sporadic and seemingly figure 2. institutional overview themes ‘to improve’. research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 unrelated to content either in the sessions or in the vle, echoing the earlier concern around setting the context and providing instructional guidance. yet these interpretations derived from the telas were often not expressed by academic colleagues. indeed, it became apparent through anecdotal evidence that the figure 3. departmental hotspot analysis � areas of best themes � horizontal correlation demonstrates persistently positive reporting of a specific theme across multiple departments. vertical correlation demonstrates persistently positive reporting of multiple themes within a single department. figure 4. institutional hotspot analysis areas for improvement � horizontal correlation demonstrates persistently negative reporting of a specific theme across multiple departments. vertical correlation demonstrates persistently negative reporting of multiple themes within a single department. c. meadows et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 interpretations of these student results by academic colleagues deviated dramatically from the telas. why and how this was the case became the next area of focus. the data derived from the student data analysis was utilised as foci for questions to be put to academic colleagues in investigating this disconnect. phase 4 �staff data collection the data derived from the ‘student engagement’ analysis pointed to four areas for investigation with academic colleagues. (1) effective communications (2) moodle organisation and high quality materials (3) use of audio visual resources (4) desire for ‘interactive’ lectures. the approach taken involved semi-structured interviews around these themes. analysts from the research team conducted these interviews across the various faculties. an open research call was placed to the academic body. forty academic colleagues indicated a willingness to participate. four per faculty was set as an initial sample. due to timing and various logistical issues 15 interviews were conducted and transcribed verbatim. this constituted 13 hours of transcribed interviews. the interviews were semi-structured around four key areas derived from the student feedback. five prompt questions were utilised in the interviews as foci for discussion. these were: (1) what does the data indicate about learning technology and the student experience? (2) what is the student expectation around communication and how do we achieve it? (3) what constitutes a well organised/resourced moodle area and how do we achieve it? (4) what issues arise from either using or planning to use audio/visual resources? (5) what is meant by interactive lectures? and what challenges do they present? the original data was digitally recorded and transcribed verbatim. this was then analysed in relation to the meta-themes from the student study. one of the objectives figure 5. mirrored meta-themes. research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 was to determine how student perception deviated from staff perception in relation to these themes. yet from the outset, it became abundantly clear that other issues pervaded the academic psyche and significant mismatches in perception existed around these four key areas. phase 5 � staff results and interpretation the staff transcripts were analysed for themes relating to the earlier foci and excerpts highlighted to see if the staff narrative reflected the student feedback analysed thus far. communications some examples as shown in table 6 demonstrate the varying opinions around communications. comparing these insights underlines the incongruence between student expectation and staff perception. staff appear to recognise the expectation for a ‘quick accurate response’ but that ‘responding at different times. . .creates an expectation that colleagues will do this as well’. in contrast, students indicate some staff refuse to engage at all in email communication and some staff seemingly engage as a useful way of ‘covering yourself ’. these typologies of response expand the sense of disconnection ostensibly strengthening the view expressed earlier that telas are perhaps critical filters in determining meaningful interpretations of the student data. audio�visual the use of audio�visual resources demonstrated a disparity of views. students emphasised the positive use that enhanced learning by allowing them to ‘gain a full understanding of topics’ but also criticised practices that were poor when they were ‘rather pointless videos’. in contrast, staff perception of audio�visual resources suggested a fear of being replaced (see table 7). also further recognition that table 6. staff and students’ views relating to communications student quotes staff quotes the support from the tutors. 90% of the tutors have said to me do not email me??? !! we pay for their advice and support. consistency needs to be there but people are responding at different times and i’m guilty . . .. but i just think i’ll get that done . . . but then that creates an expectation on colleagues to do this as well. moodle space for communicating with the cohort is especially cherished as it keeps support and contact with the uni during placements. i think their expectation, again if we are talking about fee-paying students, is for a very quick accurate response, that can solve my problem, and off i go. . . . they expect 24/7 [communication] . . . when you go on to moodle you can say, well i did, and you can see the announcements that i sent out and if you go back to moodle you can see the information that was sent out that all the other students got, so it is good for covering yourself as well. c. meadows et al. 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 colleagues using these resources would then create an expectation among students and potentially create a mismatch. therefore, a programme approach should be taken either all in or all out (see table 7). also prevalent were performance fears whereby technical difficulty or unfamiliarity could change ‘the dynamic, you have to regain your dignity unfortunately’ (see table 7). interactive lectures performance issues permeated the staff psyche in relation to interactive lectures. here staff perceived a sense of a ‘west end show’ and difficulty for staff to ‘live up to that expectation’ (see table 8). further staff concerns centred around control whereby enabling students to use technology led to a sense of ‘i’ve lost them and i need to try and get them, back in’ (see table 8). conversely students viewed this approach as enabling for them with the ability to ‘engage more verbally in class through the texting app’ (see table 8). the absence of interaction was seen as an impediment to learning as students found it ‘hard to concentrate when the lecturer is simply reading’ (see table 8). moodle use the moodle usage comments from staff appeared to emphasise the quantity of material and the ‘temptation to put loads on there’ (see table 9). the predominant feeling from students was that material was ‘hard to understand when it was out of context’. summary extracting and thematically analysing student voice data in the form of free text comments from the internal student survey has proved to be a valuable tool in identifying key aspects of students’ engagement and experience of tel. furthermore, the table 7. staff and student views of audio�visual resource use student quotes staff quotes the business lectures are too long (3 hours). some of the lectures i don’t think have been very useful, e.g., a lot of youtube videos. we could have been given the links and then we could have the choice to watch them in our spare time. the business plan should have been briefed a lot earlier instead of showing us rather pointless videos. . . . like when you go and see a play and in the first few minutes if it’s well acted then you buy into it. if for the first few minutes the lecturer is presenting this narrative, is stuck, if the student has to say ‘sir you don’t do it that way, you only have to press ctrl + enter’, it changes the dynamic, you have to regain your dignity unfortunately. he is attentive to the needs of the students with podcasts that allow us to listen to our lecture again to gain a full understanding of the topic (which is necessary in some instances) as well as ensuring that there is an overall good teaching standard in all the tutorials. [re podcasting] . . . such and such does it why don’t you do it? it has to be consistent. . .you have to have an agreement that all of you are going to do it or none of you are going to do it. . . . they won’t come to lectures anymore even though i know it doesn’t necessarily happen because at a private college i’ve done them and they do still come. research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 15 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 approach has enabled telas to develop targeted staff development and training provision in response to key themes identified. experience of delivering staff development and training and exploration of academic staff interpretations of the key tel themes highlighted a sense of fear in academic staff around some of the technologies and related approaches to learning and teaching. as a consequence of this work, telas have made several changes to their practice and approach to staff development and training for academic staff colleagues. there has been a redesign of the training provision to create greater table 9. staff and student views of moodle use student quotes staff quotes i struggle to find things on moodle menus all over the place secondary page? pedagogy page? subject page? and there is no decent search facility . . . or even a way to view resources in date order would help. . . . when you’re putting together your moodle area, it makes sense to you, but not necessarily to them. it would be good if the student and staff attitudes to moodle could be brought into line. staff use it while students don’t necessarily know they have to. i try and really use that [moodle] student view, i’m still trying to move stuff off, because i’m looking at what it looks like to them. i want to make my life easier, so that they can find things. information that is uploaded to moodle is sometimes not explained or even mentioned in class . . . although this is good extra reading it’s hard to understand when it’s out of context. . . . there is a temptation to put loads on there and almost provide lots, because there is that expectation that you only see them for 3 hours per week and that they can carry on their own private learning . . . but then, is that when they start to think that it’s unorganised and not relevant to the exam? table 8. staff and student views of interactive lectures student quotes staff quotes i know this may be difficult within this subject but i would prefer more interaction within seminar/lectures. despite loving this area i find it hard to concentrate when the lecturer is simply reading from the board. makes lecture very hard to concentrate. i’d have liked to have been able to use, not mobile technology but other things for them to use in the class, but that’s the thing after they have found that bit of information, you don’t know that they are looking at, so that’s scary. my fear is impacting in them i think, i’ve lost them then i need to try and get them back in. i feel that in the lecture we are able to engage more verbally in class through the texting app which helps create discussion about the topic rather than just reading from a board and feel this technique helps us understand each unit better. . . . it becomes ‘come on entertain me, it does need to entertain, but it’s not a west end show. what’s really difficult is when . . . you’ve got a course that’s technologically driven or uses technology as part and parcel of what they do . . . then it’s really easy for the students to expect that . . . what’s very difficult then for other members of staff is where that’s not their normal practice to live up to that expectation. c. meadows et al. 16 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 synergies with the institutional learning teaching and assessment strategy (lta strategy). this redesign has resulted in an alignment of training sessions along the six principles expressed in the lta strategy (see mmu lta 2015) the first of which were presented in the summer of 2015. these sessions embedded technology much more discretely into workshops rather than had previously been the case where technology had been the headline element. initial anecdotal evidence suggests that this had positive impact in terms of attendance, attrition rates and feedback though this requires further evaluation. in trying to address cohesion between staff interpretation and the student voice data, a vle template has been developed as starting point for building online unit areas. in this respect, perhaps the main outcome relates to the need to contextualise resources and provide instructional guidance on the use of the vle as a teaching platform. the vle template has attempted to scaffold this for academic colleagues and has been trailed in three faculties across the institution. further iterations of the thematic analysis should be able to determine whether these practical interventions have had a significant impact on the conundrum expressed in one of the staff comments. to consider that ‘when you’re putting together your moodle area, it makes sense to you, but not necessarily to them’. references alderman, l., towers, s. & bannah, s. (2012) ‘student feedback systems in higher education: a focused literature review and environmental scan’, quality in higher education, vol. 18, no. 3, pp. 261�280. astin, a. (1999) ‘student involvement: a developmental theory for higher education’, journal of college student development, vol. 40, no. 5, pp. 518�529. ausse (2009) ‘engaging for success: australasian student engagement report australasian survey of student engagement’, hamish coates published by acer, [online] available at: http://research.acer.edu.au/cgi/viewcontent.cgi?article=1017&context=higher_education bird, p., forsyth, r., stubbs, m. & whitton, n. (2015) ‘eqal to the task: stakeholder responses to a university-wide transformation project’, journal of educational innovation, partnership and change, vol. 1, no. 2. issn 2055-4990. available at: https://journals.gre.ac. uk/index.php/studentchangeagents/article/view/177/244 [accessed 16 november 2016]. birks, m. & mills, j., (eds) (2011) ‘essentials of grounded theory’, in grounded theory: a practical guide, sage, london, pp. 11�26. buckley, a. (2014) uk engagement survey 2014 � the second pilot year. higher education academy, york, uk, [online] available at: https://www.heacademy.ac.uk/system/files/ resources/ukes_report_2014_v2.pdf charmaz, k. (2006) constructing grounded theory: a practical guide through qualitative analysis, sage, london. guest, g., macqueen, k. m. & namey, e. e. (2012) applied thematic analysis, sage, thousand oaks, ca. harvey, l. (2011) ‘the nexus of feedback and improvement’, in student feedback: the cornerstone to an effective quality assurance system in higher education, eds. c. s. nair & p. mertova, chandos, oxford, pp. 11�26. hefce (2008) ‘tender for a study into student engagement, call for tenders edn, higher education funding council for england, bristol’, [online] available at: http:// webarchive.nationalarchives.gov.uk/20120118164921/http://www.hefce.ac.uk/pubs/hefce/ 2008/ kincheloe, j. (2001) ‘describing the bricolage: conceptualizing a new rigor in qualitative research’, qualitative inquiry, vol. 7, no. 6, pp. 679�692. kincheloe, j. (2005) ‘on to the next level: continuing the conceptualisation of the bricolage’, qualitative inquiry, vol. 11, no. 3, pp. 323�350. research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 17 (page number not for citation purpose) http://research.acer.edu.au/cgi/viewcontent.cgi?article=1017&context=higher_education https://journals.gre.ac.uk/index.php/studentchangeagents/article/view/177/244 https://journals.gre.ac.uk/index.php/studentchangeagents/article/view/177/244 https://www.heacademy.ac.uk/system/files/resources/ukes_report_2014_v2.pdf https://www.heacademy.ac.uk/system/files/resources/ukes_report_2014_v2.pdf http://webarchive.nationalarchives.gov.uk/20120118164921/http://www.hefce.ac.uk/pubs/hefce/2008/ http://webarchive.nationalarchives.gov.uk/20120118164921/http://www.hefce.ac.uk/pubs/hefce/2008/ http://webarchive.nationalarchives.gov.uk/20120118164921/http://www.hefce.ac.uk/pubs/hefce/2008/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 kincheloe, j. l., mclaren, p. & steinberg, s. (2011) ‘critical pedagogy and qualitative research: moving to the bricolage’, in the sage handbook of qualitative research, 4th edn, eds. n. k. denzin & y. s. lincoln, sage, thousand oaks, ca, pp. 163�178. lincoln, y. (2001) ‘an emerging new bricoleur: promises and possibilities � a reaction to joe kincheloe’s ‘‘describing the bricolage’’’, qualitative inquiry, vol. 7, no. 6, pp. 693�696. macqueen, k. m., et al., (2008) ‘team-based codebook development: structure, process, and agreement’, in handbook for team based qualitative research, eds. g. guest & k. m. macqueen, altamira, lanham, md, pp. 119�135. mclaren, p. (2001) ‘bricklayers and bricoleurs: a marxist addendum’, qualitative inquiry, vol. 7, no. 6, pp. 700�705. mmu lta (2015) ‘the mmu strategy for learning, teaching and assessment’, [online] available at: http://www.celt.mmu.ac.uk/ltastrategy/ o’leary, z. (2004) the essential guide to doing research, sage, london. reed, p. & whatmough, s. (2015) ‘hygiene factors: using vle minimum standards to avoid student dissatisfaction’, e-learning and digital media, vol. 12, no. 1, pp. 68�89. ritchie, j. & spencer, l. (1994) ‘qualitative data analysis for applied policy research’, in analyzing qualitative data, eds. a. bryman & r. g. burgess, routledge, london, pp. 173�194. ritchie, j. & spencer, l. (2002) ‘quantitative data analysis for applied policy research’, in the qualitative researcher’s companion, eds. m. huberman & m. b. miles, sage, london, pp. 305�331. rogers, m. (2012) ‘contextualising theories and practices of bricolage research’, the qualitative report, vol. 17, no. 48, pp. 1�17. saldaña, j. (2009) the coding manual for qualitative researchers, sage, thousand oaks, ca. stubbs, m. (2014a) ‘keynote: enhancing the student experience at mmu: clearing a path and interpreting digital footprints’, european first year experience network (efye) 2014 conference 9�11 june, nottingham university, nottingham, [online] available at: http:// www.ntu.ac.uk/apps/events/9/home.aspx/event/151843/multimedia stubbs, m. (2014b) ‘transforming the student experience: manchester metropolitan university’s eqal project’, european university information systems conference 2014, umea university, umea, pp. 1�6. tinto, v. (1975) ‘dropout from higher education: a theoretical synthesis of recent research’, review of educational research, vol. 45, pp. 89�125. traffic (2015) ‘jisc transforming assessment and feedback for institutional change’, [online] available at: http://lrt.mmu.ac.uk/traffic/ trowler, v. (2010) student engagement literature review, hea, york. trowler, v. & trowler, p. (2010) student engagement evidence summary, higher education academy, york, [online] available at: https://www.heacademy.ac.uk/studentengagement/ research_and_evidence_base_for_student_engagement c. meadows et al. 18 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 http://www.celt.mmu.ac.uk/ltastrategy/ http://www.ntu.ac.uk/apps/events/9/home.aspx/event/151843/multimedia http://www.ntu.ac.uk/apps/events/9/home.aspx/event/151843/multimedia http://lrt.mmu.ac.uk/traffic/ http://https://www.heacademy.ac.uk/studentengagement/research_and_evidence_base_for_student_engagement http://https://www.heacademy.ac.uk/studentengagement/research_and_evidence_base_for_student_engagement http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 appendix appendix 1. internal student survey questions. questions about the course s. disagree b-� s. agree staff on my course are good at explaining things feedback on my work helped to clarify things i did not understand i have received sufficient advice and support with my studies the course is well organised and running smoothly university resources are appropriate to my learning needs the course has helped me to develop confidence and skills to succeed overall i am satisfied with the quality of my course best things about my course things i would most like improved on my course 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 free text free text questions repeated for each unit s. disagree b-� s. agree this unit was taught well best things about this unit things 1 would most like improved on this unit 1 2 3 4 5 free text free text research in learning technology citation: research in learning technology 2016, 24: 30146 http://dx.doi.org/10.3402/rlt.v24.30146 19 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30146 http://dx.doi.org/10.3402/rlt.v24.30146 rlt1869 1..14 research in learning technology vol. 25, 2017 original research article telfest: an approach to encouraging the adoption of educational technologies farzana latif* university of sheffield, sheffield, uk (received 16 may 2016; final version received 11 december 2016) barriers to technology adoption in teaching and learning are well documented, with a corresponding body of research focused on how these can be addressed. as a way to combine a variety of these adoption strategies, the university of sheffield developed a technology enhanced learning festival, telfest. this annual, week-long event, emphasises the role technology can play through an engaging learning experience which combines expert-led practical workshops, sharing of practice, discussions and presentations by practitioners. as the popularity of the event has grown and the range of topics expanded, a community of practice has organically coalesced among attendees, supporting the mainstream adoption of several technologies and helping to broaden educational innovation beyond isolated pockets. this paper situates telfest within the technology adoption literature by providing details about telfest, outlining the results of an investigation into the impact that it has had on attendees' teaching practice and summarising some of the limitations of the method along with reflections on how to address these limitations in the future. keywords: academic adoption; barriers; challenges; technology enhanced learning; staff development; faculty development; teaching methods; higher education; staff adoption; engagement techniques introduction the role of technology to enhance the delivery of teaching is one that is acknowledged to have the potential, where its use is well thought out, to transform the learning and teaching experience and support radical change (keppell, suddaby, and hardy 2015). however, despite the affordances and opportunities that exist, as well as huge investments in technology that institutions have put in place to support this (buabeng-andoh 2012), barriers towards the adoption of technology are acknowledged throughout the literature. a lack of uniform understanding of key terms such as ‘technology enhanced learning’ (tel) (kirkwood and price 2014) can lead to differences in academic practice (mirriahi, alonzo, and fox 2015). therefore, it is important that tel is unambiguously defined. here, we adopt the u.k.’ s higher education academy (hea) definition that describes tel as ‘a synonym for e-learning but can also be used to refer to technology enhanced classrooms and learning with technology, rather than just through technology e-learning’ (hea 2016). in this context, the term tel should therefore be considered to encompass terms such as ‘blended’, ‘digital’, ‘online’, and ‘distance learning’. *corresponding author. email: farzana.latif@sheffield.ac.uk 1research in learning technology 2017. @ 2017 f. latif. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 (page number not for citation purpose) mailto:farzana.latif@sheffield.ac.uk http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v25.1869 this paper focuses on telfest, a technology enhanced learning festival, which is an approach used at the university of sheffield to encourage the adoption of educational technologies amongst teaching staff (i.e. academics). it investigates the impact of this approach, alongside barriers that are preventing changes from taking place, whilst further exploring ways in which such an approach can be improved. the paper begins by setting the scene and exploring the barriers to adoption, alongside different adoption strategies. barriers to adoption birch and burnett (2009) carried out a literature review into the barriers related to the adoption of tel and categorised the results into the following three themes: institutional, pedagogical and individual. these categories are further exemplified as follows: • institutional barriers can include a lack of institutional direction (buchanan, sainter, and saunders 2013), and it is asserted that institutions should identify their own definitions and develop a clear vision of tel in order to address the lack of common understanding. lack of support may include limited access to suitable educational software and ict equipment (buabeng-andoh 2012). whilst not acknowledged by birch and burnett (2009), restrictions due to the curriculum (e.g. rigid structure) are also considered as part of this category; these restrictions may be a result of external factors such as collaborations with professional bodies (ertmer and ottenbreit-leftwich 2010). • pedagogical barriers can include scepticism or the inability of teaching staff to see the value of technology within learning (buabeng-andoh 2012; buchanan, sainter, and saunders 2013; walker et al. 2016). arguably, it can be difficult for teaching staff to contextualise the role of technology, as the individual’s own learning experiences are likely to involve learning by ‘traditional’ teaching methods, and this will influence their own teaching methods (reilly et al. 2012). • individual factors can include a perceived lack of time, resistance to change, lack of rewards and recognition, lack of skills and lack of confidence. every 2 years, the universities and colleges information systems association (ucisa) surveys uk institutions on their adoption of tel and for the last six iterations (published between 2005 and 2016) ‘time’ has consistently been cited as the biggest barrier to tel development (walker et al. 2016). this is unsurprising considering the assertion of vaughan (2007) in alammary, sheard, and carbone (2014), who suggests that developing a blended course takes up to three times longer than if the equivalent course were to be delivered in a traditional format. adoption strategies rogers’ (2003) diffusion of innovation theory describes the way in which cultures adopt an innovation, and it is a model commonly used when exploring the adoption of tel (wilson and stacey 2004). rogers categorises adopters as follows: (1) innovators, (2) early adopters, (3) early majority, (4) late majority and (5) laggards, where innovators are the first to adopt and laggards the last. rogers explains, and other f. latif 2 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 http://dx.doi.org/10.25304/rlt.v25.1869 research has confirmed (latif 2011), that innovators and early adopters can act as a positive force to promote and encourage adoption of technologies by the mainstream majority. however, porter et al. (2016) draw on the work of moore (2002) who warns that there is a ‘chasm’ between the early adopters and innovators and that the strategies used to engage these groups with an innovation will be different to the mainstream adopters. this outlines the importance of adopting different strategies to engage different groups of adopters (robinson 2009). the work of rogers has been contested as having little relevance to the adoption of learning technologies, with arguments suggesting that using this model encourages typecasting and that adoption can and should vary depending on the educational context and the relevance of the technology within that context (eisenhauer and keisch 2016). communities of practice is a social learning theory based around the formation of usually informal, non-hierarchical, cross-institutional communities where members have shared learning needs (lave and wenger 1991). these communities develop over time; early stages involve members ‘finding others’ whilst more active communities can involve members working on joint activities and developing shared understandings (wenger 1998). the value of communities of practice and the importance of the peer support that they can offer has been identified as a way of supporting the adoption of tel within an institution (austen et al. 2016; nicolle and lou 2008; reilly et al. 2012). early adopters and innovators can have a fundamental role to play in sharing their practice (austen et al. 2016) and new expertise will grow from others within the community. however, to prevent a community from becoming too ‘insular’ in its practice, it is important to ensure that there is enough access to activity from ‘outside’ the communities’ ‘boundaries’ to renew learning (wenger 1998). formal staff development is a clear theme that acts as an enabling factor to the adoption of tel, whether this is one-to-one, online, ad hoc workshops, or part of a structured programme (alammary, sheard, and carbone 2014). the value of this is further supported by the 2016 educause learning initiative’s ‘key issues in learning and teaching’ results, where ‘faculty development’, with a focus on enhancing the ‘strengths and abilities’ of teaching staff, was identified as one of the topics of most interest to universities (educause 2016). there is a strong consensus that it is important to focus on ‘why to’, (i.e. pedagogy and added value as part of staff development) and not just on ‘how to’ (alammary, sheard, and carbone 2014; johnson et al. 2012; glover et al. 2012). in fact, some argue that it is more important to focus on the ‘why’, as this is where the real learning gain is with regards to developing a better understanding of tel pedagogy (johnson et al. 2012). on the other hand, it is being increasingly recognised that assumptions about the ‘digital skills’ possessed by both the teaching staff and students are frequently ill-founded (jisc 2015), further emphasising the importance of combining the ‘why’ with the ‘how’. as well as this, it is considered important to develop teacher ‘self-efficacy’, that is, teacher confidence in using their skills and knowledge to facilitate student learning (ertmer and ottenbreit-leftwich 2010). alammary, sheard, and carbone (2014) reviewed different blended learning approaches and proposed three distinct design approaches for the implementation of tel; low impact (adding extra technology-enhanced activities to an existing course), medium impact (replacing existing activities in an existing course with technology-enhanced activities) and high impact (building a course from scratch). they argue that the higher the impact, the larger the potential change will be to a research in learning technology citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 3 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1869 course and to the student learning experience. the model has been adopted as a method to encourage staff to think more deeply about their adoption of tel (learning development centre 2015), with the ultimate goal being that teaching staff will implement a high impact blend. the importance of deploying different strategies to support the adoption of innovations is supported by a project to introduce a new virtual learning environment (vle) integrated with other learning technologies at city university of london, where alternative methods were used depending on the culture and previous exposure to tel within each of the schools of the university (glover et al. 2012). strategies employed included staff development (group workshops, one-to-one, online), student training, employing champions and the introduction of minimum standards for vle-supported courses. the ucisa 2016 tel survey’s top three enabling approaches for the adoption and use of tel tools across institutions are: (1) providing support and training to academic staff, (2) providing platforms for sharing good practice and (3) delivery of accredited programmes for academic staff (walker et al. 2016). some institutions have put in place a strategy of offering staff cash incentives, yet it has been argued that such initiatives offer no correlation with motivating teaching staff to adopt tel (johnson et al. 2012). background to telfest as described, it is important for institutions to integrate different strategies in order to encourage and support the adoption of tel. telfest is one approach that has been put in place to complement the current tel support already available across the university of sheffield, a research-intensive university in northern england, which has approximately 3500 academic and learning and teaching support staff. this annual, week-long festival, introduced in september 2014, is organised by the institution’s central technology enhanced learning team in collaboration with other teams that support tel activity (e.g. library services) and is championed by the institution’s vice-president for education, thereby offering institutional support at the highest level. the event was primarily aimed at staff involved in teaching students has also attracted academic support staff (e.g. librarians and tel support staff such as learning technologists). collectively, these attendees have the potential to form a community of practice consisting of those who wish to reflect on their own practice or support the practice of others. prior to its inception, staff development opportunities related to tel were seen as disjointed, and the festival offered an opportunity to showcase the type of support available across the institution and surface practice already taking place (e.g. from the early adopters and innovators), in order to support the wider scale adoption of tel. the event is partly inspired by similar initiatives such as blackboard festival, delivered by nui galway to create a sense of ‘excitement and enthusiasm’ around staff development (flynn 2014). a deliberately varied agenda allows staff to select the sessions that are of most interest to them, and unlike a conference, it is considered important that staff leave the event with the skills and actionable ideas that will allow them to change their practice immediately. in addition, sessions are devised to encourage staff to generate new ideas that could enhance their delivery of teaching and develop the confidence in expanding these. a variety of adoption strategies, such as those outlined in the previous section, have been employed to achieve this: f. latif 4 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 http://dx.doi.org/10.25304/rlt.v25.1869 • show and tell sessions comprising of case studies of practice from teaching staff (e.g. early adopters and innovators) across the university, mobile app share, innovation corner (enabling attendees to consider how cutting edge research taking place within the institution could be valuable to learning). • panel discussions and debates, usually consisting of cross-faculty panels, to offer wide perspectives; examples of sessions that have run include: ‘opportunities and challenges with adopting tel’ and ‘new opportunities for e-assessment’. • workshops (the majority of sessions fell into this category, exploring both ‘how’ and ‘why’); these sessions were designed to offer a place to learn and make mistakes (to develop self-efficacy). these sessions varied from: more basic sessions (introduction to the vle) through to emerging educational technology areas (e.g. gamification) in teaching and learning. the sessions were often delivered by a member of the professional staff (e.g. educational developer, learning technologist) or co-delivered with a teaching colleague who is using a technology as part of their practice. in order to develop the community of practice, networking opportunities are integrated within the agenda and a number of workshops encourage group activities, to further develop ideas and understanding of concepts collaboratively. alongside this, one-to-one drop-in sessions run throughout the week, to support staff with specific questions or those who want to explore an idea further after they have attended a session. the sessions are delivered via a variety of different methods in order to model the practice being discussed. for instance, sessions on flipped learning and mobile learning use these same techniques to enable attendees to experience the method as students, supporting a situated method of learning (lave and wenger 1991). attending telfest is entirely voluntary and there is no formal accreditation linked to the sessions. however, digital badges are given to attendees to outline how participation at telfest relates to the hea’s professional standards framework for teaching and learning in higher education (hea 2011). approach/methodology at the time of carrying out this research, telfest had run in september 2014, january 2015 and september 2016, and there had been a significant increase in the number of people that had attended, such that more than 400 staff within the institution have attended at least one of these festivals. the events now act as a core part of telrelated staff development at the university of sheffield. whilst the events are attracting more people and receiving positive informal feedback, little is known about the impact of telfest on the implementation of tel at the university. therefore, this investigation sought to answer the following questions: (1) what is the perceived impact of telfest? (with an emphasis on changes to practice), (2) what are the ongoing barriers that are preventing changes to practice?, and (3) how could the long-term impact of telfest be improved and sustained? data collection to gain insight into these research questions, a web-based questionnaire was e-mailed to all staff that had previously attended telfest, excluding those that had a key role research in learning technology citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 5 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1869 in planning the festival (i.e. anyone involved in organising the entire event or involved in delivering more than one session). the questionnaire consisted of multiple choice and open-ended questions and was tested for reliability by other organisers of telfest. those that completed the form were asked whether they were willing to have their experiences explored further as part of a 45–60 min focus group. ethical consent was obtained through the university of sheffield’s ethical review process. data analysis in order to ensure integrity of the results, all data were anonymised, retaining only the role of the participant (e.g. teaching staff, support staff), and then aggregated before any analysis took place. the web-based platform used for the survey had its own analytical tools built-in and this was used to support the analysis of the quantitative portions of the responses. the open text responses, alongside transcription from the focus groups, were thematically analysed (guest, macqueen, and namey 2011) in line with the research questions. findings a total of 58 out of a possible 398 participants completed the initial questionnaire, a 14% response rate. out of those that responded, 37(64%) of respondents were teaching staff and 21(36%) were professional/administrative/support staff (principally learning technologists and academic developers). this split is typical of the demographic of telfest attendees. four focus groups were held, with between 2 and 5 participants in each, and in total there were 17 focus group participants. perceived impact of telfest (1) what is the perceived impact of telfest? (with an emphasis on changes to practice) changes to practice as part of the questionnaire, participants were asked to consider whether telfest had helped them to generate new ideas, with a majority of participants (40; 69%) agreeing with this. in total, 47 participants (81%) said that they had already made or intended to make changes to their practice, and that these changes and plans were influenced by attending telfest. a total of 31 participants (53%) had made changes, whilst 16 participants (28%) had planned but not yet implemented the changes. respondents reported barriers that correlated to the barriers highlighted within the literature. these are outlined in subsequent findings. respondents who had made changes (n = 31) were further asked to specifically comment on the degree of impact of their changes, in line with the previously described impact model by alammary, sheard, and carbone (2014). these questions were asked using a likert scale. cumulative responses alongside the mean and standard deviation for each of these sets of responses are provided in table 1. responses stating ‘not applicable’ and those with no response have been removed from this calculation. f. latif 6 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 http://dx.doi.org/10.25304/rlt.v25.1869 table 1. impact of changes (n = 31). strongly disagree strongly agree standard deviationquestions indicative examples provided 1 2 3 4 5 unanswered mean low impact: attending telfest has resulted in me [adding additional activities to a module/course] enhancing design of vle, adding discussion boards 0 1 2 20 5 2 4.03 0.64 medium impact: attending telfest has resulted in me [changing existing learning activities] adding a twitter back channel to large classroom teaching; introducing mobile learning activities to replace traditional methods of teaching; incorporating game-based learning activities 0 1 5 18 6 1 3.85 0.66 high impact: attending telfest has resulted in me [redesigning a module or course] redesign of face to face teaching; move to a flipped learning model of delivery; redesign of assessment activities 3 3 11 7 3 4 3.14 1.13 r esearch in l earning t echnology c itatio n : r esearch in l earn in g t ech n o lo gy 2017, 25: 1869 h ttp ://d x.d o i.o rg/10.25304/rlt.v25.1869 7 (page num ber not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1869 examples of specific changes that have been made to practice are outlined in column 2 of this table 1. the results indicate that most changes made are in the lowand medium-impact range, with fewer reaching levels of high impact. the examples of changes provided were closely aligned to the sessions delivered throughout telfest, and several respondents outlined multiple changes to practice, highlighting the importance of ensuring that sessions are relevant and resonate with participants. developing lowand medium-impact activities offer opportunities to implement new technologies and develop confidence in them, whereas high-impact changes have more risk associated with them (alammary, sheard and carbone 2014). all of those that had made changes indicated that they planned to make further changes, implying that they saw the integration of tel as an ongoing process. therefore, this could lead to higher impact change in the future as confidence increases. additionally, those who are yet to make changes indicated that they plan to make changes when appropriate, highlighting that telfest has made people think pedagogically rather than technologically about making changes: ‘i haven’t gone back and done things straight away. i’ve thought, “that’s another tool for the toolbox.”. when a good opportunity to use it comes along, then i will use it, rather than using it for the sake of using it’. respondents were further asked to consider how they planned to or how they have evaluated changes made to practice. based on those that had made or intended to make changes, a minority (16 out of 47; 34%) had considered this. those who had considered ways to evaluate practice relied on standardised feedback (e.g. module/ course evaluation, although there was an acknowledgment that this might not necessarily surface feedback related to specific changes), changes to grades, direct informal feedback, no longer receiving bad feedback, analysing analytics related to online access of resources/completion of activities, and developing specific questionnaires for students to complete. one focus group participant said, ‘i made changes to my course. it made me feel better, but i’m not sure if it offered value to my students’. this comment, alongside the minority that had considered ways to evaluate changes, suggests a lack of consideration towards evaluating practice, despite this being a key aspect of scholarly practice. however, lack of evaluation appears to be a sector-wide trend, as demonstrated by kirkwood and price (2013) in their literature review of tel practices, which found that a scholarly approach to enhancing teaching and learning is seldom taken by teachers and policymakers. additionally, at an institutional level, the 2016 ucisa tel report (walker et al. 2016) asked institutions whether they had ‘evaluated the impact of technology enhanced learning tools and systems on the student learning experience across the institution’ (walker et al. 2016, p. 45) and found that less than half (40 of 99; 40%) of the institutions that responded had done so within the last 2 years. community of practice responses indicated that the community of practice that has developed through telfest has influenced changes to practice. in line with the diffusion of innovation theory, those already adopting a technology (e.g. early adopters and innovators) have played a key role in influencing these changes. learning from the experiences of innovators and earlier adopters has enabled attendees to generate new ideas and consider how similar technologies can be used f. latif 8 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 http://dx.doi.org/10.25304/rlt.v25.1869 within their own context. respondents explained how understanding the successes and failures of others has further enabled them to identify what good practice looks like within their context, ‘from telfest we encounter the issues that others have had and have tried to ensure that we don’t make the same mistakes’. this implies that delegates benefit from the experiences of peers in other disciplines, and this was reinforced by an academic describing how they had invited a speaker from another department to share practice at their departmental meeting. these examples are indicative of many others, where less experienced colleagues have drawn on the experience of innovators and early adopters to help to mainstream innovations. there were also examples of long-term sustained relationships and activities beginning to form. an early adopter of flipped learning described being asked to peer review a session on flipped learning for another colleague after having presented at telfest ‘i had someone i met at telfest contact me to peer review their use of flipped learning’. others described how the community is developing through social media: ‘[i] sometimes correspond with people [i have connected with through telfest] on twitter for ideas’. it was explained by one academic, how these relationships are important as they often work on their own when developing their modules/courses ‘often you can feel quite isolated when teaching a module, you feel like an island, there are other islands that you can go to nearby, but you would have to get the boat out’. there was also evidence of new projects being formed with colleagues across disciplines, who they may not have otherwise met, ‘new projects have formed as a direct result of attending’. as well as teaching and learning projects being formed, there was mention of a research project being identified, resulting in a cross disciplinary joint application. initially, telfest was developed with the aim of creating a community of practice for academics. however, the combined community with both academic and support staff that arose has offered greater value. for academic staff, telfest has enabled them to identify support staff, both at a departmental/faculty level and central level, who could help them to make changes to their practice after telfest. support staff outlined that the event gave them the opportunity to identify academic staff that were ‘interested, just by their presence there’, enabling them to connect with staff who might be willing to pilot new tools and techniques or share their practice with others. support staff also talked about how they found telfest a useful way to connect with other professional staff from different parts of the university and identified ways in which they could better work with each other to provide improved support to teaching staff, such as through collaborations across central teams (e.g. library and it services) and between faculties. ongoing barriers/challenges (2) what are the ongoing barriers that are preventing changes to practice? barriers reported by respondents aligned to birch and burnett’s (2009) ‘institutional’ and ‘individual’ categories rather than those of a ‘pedagogic’ nature. this shows that the barriers are related more to external factors rather than a lack of knowledge, and are outlined below. raising the profile of tel (institutional barrier) there were strong views within the focus groups that the institution needed to do more to promote the importance of developing teaching as ‘there has never been any proper research in learning technology citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 9 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1869 value over teaching, not like there should be’ and that this is reinforced through perceived limited promotion routes that exist for staff based on teaching, rather than research excellence. however, feedback has indicated that telfest has helped to raise the profile of tel and the understanding of the breadth of activities that it encompasses ‘the agenda offers a great overview of all the things i still need to learn’. respondents were encouraged that the events are both supported and attended by staff at a senior level, highlighting the importance of senior buy-in for such initiatives. it was proposed that more still needed to be done to raise the understanding of tel and that they would have liked to see more colleagues attending the event, with one delegate’s explanation being ‘i thought it was for it managers, but next time i hope to go to more events’ highlighting that there would be others that had this misconception. it was proposed in three of the focus groups that a better understanding of tel and related staff development events such as telfest could be achieved by creating clearer links within accredited routes (such as the hea’s uk professional framework and internally accredited programmes such as the postgraduate level certificate in learning and teaching). departmental culture (institutional barrier) several focus group attendees described how telfest had inspired them and enabled them to generate new ideas, but when they returned to their department it was difficult to motivate change beyond their course—‘i wanted to get our department to mark online, colleagues did not like it, they said that they prefer paper’. in addition, some respondents felt that their colleagues were not invested in enhancing their teaching practice—‘it feels like my colleagues are behind the curve, things are almost stuck to where they were 30 years ago’. such views imply that for some, teaching practice is a low priority and could provide insight as to why people don’t engage with staff development events such as telfest. there were further observations that there is not always enough encouragement to change practice at a departmental level and that when this encouragement did take place, it was as the result of bad feedback from students, and was therefore reactive rather than proactive change. however, it was clear that a lack of department support was not a view shared by all participants, and others spoke favourably about the support offered by the departments within which they work: ‘i think that we have a very supportive environment’, further showing that a single approach to staff development for the whole institution would likely be met with limited success. individual barriers in line with the literature, ‘lack of time’ was the most cited reason to describe why changes had not yet been made. participants explained how, having had time to contemplate their practice and possible alternatives, they walked away with ideas and new skills, but the ongoing pressures of their duties meant that once away from telfest, they struggled to find the time to put in place the changes they wanted to make into place. this meant that people have not been able to find ‘time to sit and focus on making the course as good as it could be’ and in some cases has led to no changes being made, ‘i had a couple of really good ideas but i haven’t done anything’. f. latif 10 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 http://dx.doi.org/10.25304/rlt.v25.1869 participants also described how other priorities, including research, administrative tasks, and other planned developmental work, prevented them from making the integration of tel a priority. suggestions to make time included blocking time out to make changes and having access to a list of support staff. one delegate proposed the idea of an organised ‘tel retreat’, lasting a period of 2–5 days, where participants could bring their ideas with them and develop their work with peers who also want to make changes and specialists, such as learning technologists, educational developers, and technology specialists. such an event would help to further develop the community of practice by strengthening the support networks comprising of peers and support staff. on the other hand, there were also concerns as to whether, given the limitations to time and changes to technology, the invested time would be valuable: ‘how long before i need to change what i do again?’ improving telfest (3) how could the long-term impact of telfest be improved and sustained? the research has shown that whilst telfest has resulted in significant change to practice, the potential long-term and ongoing impact can be increased in a number of ways and these are explored. catering to an audience with a widening digital divide whilst a significant proportion of telfest attendees are returning participants, there has been an ever-increasing number of first-time attendees at each event. the importance of maintaining a balance between sessions for those new to tel and those who have been making use of technology in their teaching for some time was recommended: ‘there is a danger that it is just the same kind of topics each year, … you need to keep it fresh if you want the same people to keep coming’. as was the need to appeal to new audiences ‘… you have an impossible job of catering for people that don’t have a clue and others that are nuanced … you need a mix of sessions’. it was also recommended that future sessions should clearly outline required prior knowledge and experience in order to support delegates in making informed choices when they select a session. continuing to cater to a wide audience through delivering a wide variety of sessions will therefore continue to be essential when designing future telfest agendas in order to ensure that colleagues regardless of their level of adoption are catered for. furthermore, inviting external speakers could prevent the community from becoming too ‘insular’ (wenger 1998). timing telfest is scheduled to run at the start of the academic semester, and this was specifically chosen to assist staff in developing their modules just before the start of the term. the majority of attendees indicated that the times the festivals run are appropriate and allowed them to make small changes and start to think about larger changes for the following semester, describing the festival as a ‘good thing to come back to after the break’. others expressed frustrations that there was not enough time to make significant changes, as in some cases teaching started the week after telfest. unfortunately, the nature of the institution and the varied programmes being taught research in learning technology citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 11 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1869 means that the teaching cycle varies significantly and there is a clear consensus that there is no right time to arrange such events. as a result, there are plans to change the dates annually, in order to attract different audiences and provide staff with more time to make changes. student digital literacy in all of the focus groups, the importance of supporting students and developing their skills was recognised. concerns were also expressed that by not providing students with adequate support, they may use this as an excuse for not completing work ‘my internet is broken ...[has become synonymous with]… the dog ate my homework’. respondents were also conscious of the importance of parity and equity ‘…[the institution is] apple-centric … [and i] don’t want to exclude people’. it was proposed that telfest could help raise awareness of the support that teaching staff should offer students when introducing a new tel innovation. ongoing support it was suggested that providing delegates with contact lists of those that support and are interested in developments in certain areas would assist in developing these relationships. a number of respondents highlighted that more opportunities to network and discuss ideas would be valuable during telfest, as these relationships take time to develop. developing events specifically for telfest attendees after the event could also help to develop this community further. ongoing promotion it was recommended that more should be done to promote the festivals at departmental meetings and by departmental learning and teaching leads, as there were still many people that were unaware of the event. if more people attended from a department, it was proposed that this would help to develop localised networks. research limitations the research undertaken has helped to provide both a quantitative and qualitative insight into the research questions. however, from the small sample of participants, it is difficult to know whether the participants, who were self-selected, represent a sample of typical telfest attendees. excluded from participation were those that were heavily involved in shaping telfest. in some cases, though, they attended the festival and may have been able to offer a different perspective. additionally, widening the call for participation to those who had not previously attended could have provided more insight into ways that the long-term impact of telfest could be improved. on the other hand, telfest is aimed primarily at teaching staff, it also attracts wider audiences, so some of the questions asked were not applicable to all research participants. differentiating teaching staff and support staff responses or asking each group different questions would have helped to better understand the different needs of each group. furthermore, research related to the impact of telfest f. latif 12 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 http://dx.doi.org/10.25304/rlt.v25.1869 could have been further strengthened by asking students about the perceived impact of the changes that teaching staff had made or planned to make. conclusion this research has demonstrated that there have been positive benefits for those that have attended telfest. these include generating new ideas, changing practice, developing support networks and forming collaborations. the majority of changes made to practice have been described as low and medium impact. however, there is an indication that high impact levels will be attained with time, as teaching staff become more confident and familiar with technology. therefore there would be value in carrying out research in subsequent years to gauge whether the impact of change develops over time. since carrying out this research, a further telfest ran in june 2016 and july 2017 (and will continue to run for the foreseeable future). these events saw an increase in the number of attendees, and in line with the findings, many changes have been put in place to ensure growth and long-term value. however, this research has highlighted barriers that exist at an institutional level; these issues are complex and lack uniformity. telfest is just one of the methods used to support the academic adoption of tel. the findings will feed into wider institutional initiatives that are taking place to address such issues and further support the adoption of tel within the institution. conflict of interest and funding no funding has been received for this study. references alammary, a., sheard, j. & carbone, a. (2014) ‘blended learning in higher education: three different design approaches’, australasian journal of educational technology, vol. 30, no. 4, pp. 440–454. austen, l., parkin, h., jones-devitt, s., mcdonald, k., & irwin, b. (2016) digital capability and teaching excellence: an integrative review exploring what infrastructure and strategies are necessary to support effective use of technology enabled learning (tel). [online], retrieved december 2, 2016, from http://www.qaa.ac.uk/en/publications/documents/ digital-capability-and-teaching-excellence-2016.pdf birch, d. & burnett, b. (2009) ‘bringing academics on board: encouraging institution-wide diffusion of e-learning environments’, australasian journal of educational technology, vol. 25, no. 1, pp. 117–134. buabeng-andoh, c. (2012) ‘factors influencing teachers’ adoption and integration of information and communication technology into teaching: a review of the literature’, international journal of education and development using information and communication technology, vol. 8, no. 1, pp. 136–155. buchanan, t., sainter, p. & saunders, g. (2013) ‘factors affecting faculty use of learning technologies: implications for models of technology adoption’, journal of computing in higher education, vol. 25, no. 1, pp. 1–11. educause. (2016) key issues in teaching and learning, [online] available at: http://www. educause.edu/eli/initiatives/key-issues-in-teaching-and-learning eisenhauer, y. c. & keisch, d. k. (2016) disrupting the technology adoption spectrum: thinking creatively about faculty technology use, [online] available at: http://www.educause.edu/ events/eli16/sess25 ertmer, p. a. & ottenbreit-leftwich, a. t. (2010) ‘teacher technology change: how knowledge, confidence, beliefs, and culture intersect’, journal of research on technology in education, vol. 42, no. 3, pp. 255–284. research in learning technology citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 13 (page number not for citation purpose) http://www.qaa.ac.uk/en/publications/documents/digital-capability-and-teaching-excellence-2016.pdf http://www.qaa.ac.uk/en/publications/documents/digital-capability-and-teaching-excellence-2016.pdf http://www.educause.edu/eli/initiatives/key-issues-in-teaching-and-learning http://www.educause.edu/eli/initiatives/key-issues-in-teaching-and-learning http://www.educause.edu/events/eli16/sess25 http://www.educause.edu/events/eli16/sess25 http://dx.doi.org/10.25304/rlt.v25.1869 flynn, s. (2014) the blackboard festival, [online] available at: https://prezi.com/vbfrmtljfaqu/ the-blackboard-festival/ glover, i., et al., (2012) ‘a tale of one city: intra-institutional variations in migrating vle platform’, research in learning technology, vol. 20. guest, g., macqueen, k.m. & namey, e.e. (2011) applied thematic analysis, sage, ca, usa. hea. (2011) the uk professional standards framework for teaching and supporting learning in higher education, [online] available at: https://www.heacademy.ac.uk/sites/default/files/ downloads/ukpsf_2011_english.pdf hea. (2016) technology enhanced learning (tel) toolkit | higher education academy, [online] available at: https://www.heacademy.ac.uk/enhancement/toolkits/technology-enhancedlearning-tel-toolkit jisc. (2015) developing digital literacies, [online] available at: https://www.jisc.ac.uk/guides/ developing-digital-literacies johnson, t., et al., (2012) ‘technology adoption in higher education: overcoming anxiety through faculty bootcamp’, journal of asynchronous learning networks, vol. 16, no. 2, pp. 63–72. keppell, m., suddaby, g. & hard, n. (2015) ‘assuring best practice in technology-enhanced learning environments’, research in learning technology, vol. 23. kirkwood, a. & price, l. (2013) ‘missing: evidence of a scholarly approach to teaching and learning with technology in higher education’, teaching in higher education, vol. 18, no. 3, pp. 327–337. kirkwood, a. & price, l. (2014) ‘technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? a critical literature review’, learning, media and technology, vol. 39, no. 1, pp. 6–36. latif, f. (2011) designing, planning and implementing learning technologies through engaging teachers, [online] available at: http://archive.alt.ac.uk/conference/altc2011.alt.ac.uk/talks/ 22233.html lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation, cambridge university press, ca, usa. learning development centre. (2015) the blended learning design framework, [online] available at: https://sleguidance.atlassian.net/wiki/display/bl/the+blended+learning+design+framework mirriahi, n., alonzo, d. & fox, b. (2015) ‘a blended learning framework for curriculum design and professional development’, research in learning technology, vol. 23. moore, g. a. (2002) crossing the chasm: marketing and selling disruptive products to mainstream customers, harpercollins, ny. nicolle, pam, s. & yiping, lou (2008) technology adoption into teaching and learning by mainstream university faculty: a mixed methodology study revealing the “how, when, why, and why not”, journal of educational computing research, vol. 39, no. 3, pp. 235–265. porter, w.w., graham, c.r., bodily, r.g. & sandberg, d.s. (2016) ‘a qualitative analysis of institutional drivers and barriers to blended learning adoption in higher education’, the internet and higher education, vol. 28, pp. 17–27. reilly, j.r., vandenhouten, c., gallagher-lepak, s. & ralston-berg, p. (2012) ‘faculty development for e-learning: a multi-campus community of practice (cop) approach’, journal of asynchronous learning networks, vol. 16, no. 2, pp. 99–110. robinson, l. (2009) changeology, [online] available at: http://www.enablingchange.com.au/ summary_diffusion_theory.pdf rogers, e. m. (2003) diffusion of innovations, 5th edn, free press, ny, ca, usa. williams, s. (2014) ‘embedding technology enhanced learning at universities: a collaboration between university of cumbria and newcastle university’, journal of learning development in higher education. vaughan, n. (2007) perspectives on blended learning in higher education, international journal on elearning (ijel), vol. 6, no. 81, pp. 81–94. walker, r., et al., (2016) survey of technology enhanced learning for higher education in the uk, universities and colleges information systems association, oxford. wenger, e. (1998) communities of practice, cambridge university press, ca, usa. wilson, g. & stacey, e. (2004) ‘online interaction impacts on learning: teaching the teachers to teach online’, australasian journal of educational technology, vol. 20, no. 1, pp. 33–48. f. latif 14 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1869 http://dx.doi.org/10.25304/rlt.v25.1869 https://prezi.com/vbfrmtljfaqu/the-blackboard-festival/ https://prezi.com/vbfrmtljfaqu/the-blackboard-festival/ https://www.heacademy.ac.uk/sites/default/files/downloads/ukpsf_2011_english.pdf https://www.heacademy.ac.uk/sites/default/files/downloads/ukpsf_2011_english.pdf https://www.heacademy.ac.uk/enhancement/toolkits/technology-enhanced-learning-tel-toolkit https://www.heacademy.ac.uk/enhancement/toolkits/technology-enhanced-learning-tel-toolkit https://www.jisc.ac.uk/guides/developing-digital-literacies https://www.jisc.ac.uk/guides/developing-digital-literacies http://archive.alt.ac.uk/conference/altc2011.alt.ac.uk/talks/22233.html http://archive.alt.ac.uk/conference/altc2011.alt.ac.uk/talks/22233.html https://sleguidance.atlassian.net/wiki/display/bl/the+blended+learning+design+framework http://www.enablingchange.com.au/summary_diffusion_theory.pdf http://www.enablingchange.com.au/summary_diffusion_theory.pdf http://dx.doi.org/10.25304/rlt.v25.1869 doi:10.3402/rlt.v24.31856 original research article a systemic model for differentiating school technology integration tel amiel a *, luis claudio kubota b and willian washington wives b a university of campinas (unicamp), nucleus of informatics applied to education, campinas, sp brazil; b institute for applied economic research (ipea), social policies and studies, brası́lia, df brazil (received 6 april 2016; final version received 14 june 2016) school technology integration rarely begins with school or educator choice. it is part of a wider context where external and internal factors have direct influence on the goals and tools that are adopted over time. the objective of this study is to investigate the systemic conditions that contribute or inhibit the development of different activities by teachers making use of new media. we compiled a list of wellknown conditions for technology integration success and mapped these in the historical and culturally bound perspective of activity theory (cultural historical activity theory). we conducted a multiple case study analysis of four schools, public and private. the results point to unique and distinctive scenarios even when homogeneity would be expected, reinforcing the argument that material conditions do not determine pedagogical outcomes nor do they determine changes in practice. beyond this, the study proposes a methodology that can help elicit tensions in technology integration, pointing to avenues for school development. keywords: school; new media; technology; systemic introduction valuable projects have aimed to integrate educational technology in schools around the globe, in the name of innovation and change. there is always hope that these designs and interventions will flourish and positively influence school activities and practices. many projects do not attempt to modify and effectively challenge the grammar and operation of schooling, beyond what is needed for the temporary implementation of the project (tyack and tobin 1994). there is no doubt that sustaining successful interventions is a challenge, but one which researchers cannot ignore if they aim to do more than add to the long list of ‘what works’, ephemerally. fishman and colleagues have appropriately incorporated some of these ideas into systemic design-based research: we understand that not all researchers who work with technology will find work in systemic reform contexts appealing or appropriate . . . but the field, as a whole, would benefit from engaging in systemic design-based research on technology innovations . . . if we do not make progress in this area, we will have missed an opportunity to bring some of the most potentially valuable technological innovations from the research community to bear on the ongoing challenge of systemic school reform. (fishman et al. 2004, p. 70) *corresponding author. email: tamiel@unicamp.br responsible editor: amanda jefferies, university of hertfordshire, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 t. amiel et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 though we are generally interested in what happens during interventions, there seems to be less interest in what happens on the day-to-day operations of schooling (cf. cuban 2001). it has become a truism to assert that artefacts are neither neutral tools nor do they determine in social activities. still, their influence in social settings is being recognised as an essential component in the development of human activity, in varying degrees. within educational contexts, we know different artefacts can contribute but do not determine pedagogical orientation (salomon 2002). in other words, an activity might be extremely successful without the use of novel artefacts. it is also quite difficult to judge the quality of the activity simply by the novelty of gadgetry or pedagogical perspective. it can be argued that teachers are, generally speaking, not deterministic about their practice. teachers facilitate numerous types of activities that are more or less innovative, and make more or less use of novel artefacts without the support or interventions of external actors. as an example of these influences, one could cite educational policy. because educational technology deployment in school programmes generally follows the top� down approach of educational policy in general, artefacts reach schools charged with a substantial amount of predetermined guidelines for use, training, support and other pertinent elements of the system. looking at educational technology interventions through a wider lens from the beginning allows us to make sounder decisions in identifying configurations of media and methods that can address political, economic, ethical and equity constraints. beyond the common condemnation of the public school as antiquated and the teacher as defiant in their use of technology, there is a need to better understand how context impacts, restrains and inclines on new media use in educational activities. school teachers, in brazil and abroad, are increasingly making use of media to prepare and mediate educational experiences (cetic.br 2015). in this study, we follow an increasing body of research aimed at a more complex and systemic understanding of the web of relations that influence on teachers’ use of educational technology. we make use of cultural historical activity theory (chat) to identify tensions in technology integration within school systems. this methodology was applied in a cross-case study based on four cases with both private and public schools in brazil. systems, ecologies and activities lim (2002) proposes that ‘many [. . .] studies lack detailed investigation of what actually takes place in the information and communication technology (ict) learning environment and its sociocultural context. ict does not exist in isolation; it is interwoven with the rest of the tools and participants in the learning environment’ (p. 411). a systemic investigation is a methodological necessity, even for those whose aim is to create generalisations: ‘[w]ithout observations of the whole system of interrelated events, hypotheses to be tested could easily pertain to the educationally least significant and pertinent aspects, a not too infrequent occurrence’ (salomon 1991, p. 17). nardi and o’day (1999) propose looking at these settings as information ecologies, ‘a system of people, practices, values, and technologies in a particular local environment. in information ecologies, the spotlight is not on technology, but on human activities that are served by technology’ (p. 49). zhao and frank (2003) t. amiel et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 investigated the differential use of computers in 19 schools by examining schools as ecosystems, computer uses as living systems and teachers as keystone species. the ecological perspective is also systemic: ‘an ecological systems thinker proceeds with an eye to the relationships between any given system and its superordinate, coordinate, and subordinate systems, for those relationships strongly influence the success of any change effort’ (squire and reigeluth 2000, p. 145). the investigation of technology integration through a ‘systems’ lens demands a framework that will consider multiple levels, entities and their relationships over time. within this framework, there is increasing interest in theories that go beyond institutions and people in order to incorporate objects and non-humans as essential components (or participants) in how humans and collectives make sense, organise and develop. these frameworks include actor-network theory, organisational semiotics and activity theory (hornung and baranauskas 2013). within the realm of education, activity theory has become a powerful theoretical perspective to study goal-directed activities. in chat, multiple actors engage towards an object and produce a shared, historically determined outcome that is mediated by tools. the original concept of activity theory stems from the work of vygotsky on tool mediation (particularly language) and has been extended to include both psychological and cultural (including physical) tools (engeström 1987). methodology we began the investigation by gathering a number of studies that discuss factors which hamper or facilitate the integration of educational technology in schools (cuban, kirkpatrick, and peck 2001; groff and mouza 2008; hew and brush 2007; iste 2009; lowther et al. 2008; pelgrum 2001; sorj and lissovsky 2011; zhao et al. 2002). we avoided adding ‘new factors to the ‘‘laundry list’’ of factors associated with technology uses’ (zhao and frank 2003, p. 801). directly or indirectly (through mediation) these factors impact the use and adoption of new media by teachers in the classroom (table 1). we suggest that different systems might produce different conditions, which in turn lead to different sorts of activities in schools. to researchers and practitioners in the field, most of these factors will be quite familiar. what we do not generally do is treat them in a systemic fashion (park et al. 2013). these factors were aligned with the systemic model proposed by engeström (1987, figure 1), which is based on chat. with this, a tentative working model of the relations between an activity system and a school was created (examples will be provided in the results section). the activity diagrams permit a clear visualisation between different elements of the school activity system and the tensions (evidenced by factors) that might develop through the integration of technology. we see the table 1. factor list. access to tools peer relations and support assessment alignment professional learning community support technical/pedagogical support culture � school student competencies culture � subject teacher competencies curriculum connection time constraints leadership tools and infrastructure educational regulation/legislation vision and plan financial sustainability research in learning technology citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 factors as reducing or amplifying contradictions or tensions within the system. they might also contribute to expanding or reducing the margin of manoeuvre teachers have in regard to their use of new media. chat has been used to investigate different educational contexts, from higher education (amory 2014) to basic education, the object of our study. lim (2002) expanded the activity diagram to include a series of concentric circles which involve the course of study (e.g. assessment), the school (e.g. type), the educational system (e.g. policies) and society at large (e.g. employers). these factors and actors clearly impact school and teacher activities and can lead to a substantial misalignment between what more distant participants envision and what actually takes place in schools (karasavvidis 2009). lim and hang (2003) demonstrate the use of chat during a case study using interviews with key actors to identify contradictions that emerge between different levels of the school system. a similar methodology was used by demiraslan and usluel (2008), who investigated ict integration in two schools using the perspective of teachers as a starting point in identifying relations and contradictions in different school activity systems. one application of chat is to look at how tensions emerge as individuals (such as school administrators and teachers) work in a specific context (a school or a classroom) permeated by regulations, division of labour, tools and other factors (paula and moreira 2014). the analysis of contradictions and tensions is particularly useful in studies involving educational technology given the (usually purposeful) disturbances these projects have in organisations, such as schools. the analysis of the tensions that emerge in educational technology projects from the perspective of teachers can help understand how changes in educational practice might take place in school contexts (murphy and rodriguez-manzanares 2008). in this study, we considered the object of analysis as technology integration, in broad terms. the analysis of the object is meant to understand a purpose or a latent figure 1. activity diagram (amory 2014). table 2. school interviews conducted. school years teachers principals coordinators private a 1�12 4 1 1 private b 1�12 3 1 1 public c 1�5 2 1 0 public d 6�9 2 1 1 total � 11 4 3 t. amiel et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 quality to a collective activity: ‘[t]he object is a heterogeneous and internally contradictory, yet enduring, constantly reproduced purpose of a collective activity system that motivates and defines the horizon of possible goals and actions’ (engeström 2004). we believe that technology integration has been an enduring collective object and an underlying focus of collective action in schools (cuban 1986). through the guise of pedagogical innovation, digital inclusion or economic gains, most school personnel have a latent understanding that it is a collective expectation. tools were defined narrowly, focused on the hardware and systems made use by teachers for educational purposes, excluding more managerial uses of software. we were interested in investigating technology as new media, or computational media (for details, see manovich 2001), which in schools amounts to tablets, digital white boards, computers and the systems that run on, or support them (here, we use ‘new media’ and ‘technology’ interchangeably). actors were defined as teachers and school administrators who were interviewed and observed in this study. rules were focused on legislation and regulation that is defined by or forced upon the school (state, administration, etc.). the division of labour can help identify how relevant actors distribute responsibilities within school (such as technical support). community is identified with all relevant actors mentioned by participants, usually including parents, co-workers, volunteers and the like. we made use of a case study method, taking the school as a case. we created interview and observation protocols deriving questions from our factor list, as they pertained to the relations in the activity diagram. these included protocols for (1) school principal interviews, (2) pedagogical coordinator interviews, (3) teacher interviews, (4) classroom observations and (5) post-observations debriefing interview with teachers (table 2). interviews with principals were done to understand current school context and practices, and a short (5 years) history related to educational technology projects. interviews with pedagogical coordinators were aimed at understanding how the perspective of administrators is passed on to teachers and how concerns and problems reported by teachers are dealt with by administration. teacher interviews focused on personal trajectories and practices, factors related to their day-to-day involvement with making use of educational technology in teaching, collaborative practices, their knowledge of project and policies and equipment access and use. teachers were selected based on recommendations by school principals. all protocols were semi-structured with follow-up questions, allowing for detours and deviations. the interviews were transcribed and (1) analysed to create a diagram for each actor, (2) joining the most common and prevalent elements into a single diagram for the school. this process was done in multiple iterations, until agreement between the three researchers was reached. the diagrams do not include every element presented by individual actors. we aim for this to be a diagnostic tool that would also allow us to typify and compare activity systems, highlighting essential tensions. four schools from the city of brasilia participated in this study. two private schools were chosen because of their open publicity on the use of educational technology. two public schools were chosen, selected from a list of six provided by the state education secretariat. the selection criteria were not opened to us, but we assume that (1) the units were selected from the outskirts of the city, and (2) the schools demonstrated experiences with educational technology that were considered particularly interesting by the secretariat. research in learning technology citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 identifying differences in activity systems brazil is a federative republic in which states and municipalities share responsibility with the federal government. the federal district, where brasilia is located, shares the responsibility of both state and municipality. for public schools, we can identify two strong spheres of influence: the ministry of education (mec) and the government of the federal district (gdf) that directly impact schools. mec influences schools through a myriad of programmes that have historically promoted icts through the purchase of equipment, content and teacher professional development (ipea 2015). for private schools, the influence of the federal and state government can be felt on regulations regarding curricula. both schools are part of larger conglomerates. one is part of a collective local like-minded schools, which in turn, as a group, adhered to the teaching methodology and philosophy of a larger nationwide conglomerate. this might entail actions like adopting a particular perspective on curricula, purchasing of educational materials, management systems, evaluation tools and the like. the second school is part of a large, nationwide private conglomerate, responding directly to a central office, sharing in the brand, values, curriculum and methodologies of the larger organisation. below we present each school individually with their corresponding diagrams and details concerning the most relevant tensions and contradictions. private school a we interviewed the principal, a coordinator and four teachers, and observed two classrooms. the instructors that we interviewed teach high school subjects. private school a is a traditional and large institution in brası́lia, which includes kindergarten, elementary and high school students of high socio-economic status (ses). it is a unit of a larger educational group. it offers paid activities � including educational robotics � as well as a computer lab, projectors in the classrooms and interactive white boards. the central unit of the educational group to which the school belongs maintains a portal with information and activities for students and teachers (figure 2). the school offers a week for pedagogical training for teachers, but it does not focus on the use of technologies in classrooms. this may explain why some teachers have difficulties using these technologies. moreover, there is no formal pedagogical support for the use of technologies. some teachers reported asking for assistance from colleagues in case of necessity (tension actors-tools). the school offers robotics as an activity apart from the main curriculum. as it is optional, there is no formal student evaluation (tension rules-object). school management does not seem so sure about the success of initiatives that involve the implementation of the use of new media. there is no explicit policy to encourage the experimentation by the teachers (tension actors-community). the school curriculum is defined, in part, by the central educational group, which is located in another state. general planning is created and pushed by headquarters, and teachers must plan lessons and adapt activities to fall within this framework. some teachers mentioned that part of the content needed adaptation to be used locally. moreover, there are only two paid hours per week to prepare classes. these factors might contribute to the low level of flexibility in class planning and to a low level of experimentation � regardless of the use of specific technologies � by the teachers (tension rules-actors). t. amiel et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 according to those interviewed, the main motivation for the use of technologies in the school seems to be to attract students’ attention, and not within a larger consideration of its integration as a pedagogical tool. as a result, the resources available are used especially in extracurricular activities and as an audiovisual tool (tension intrinsic to the object). private school b we interviewed the principal, a coordinator and three teachers, and made two classroom observations. the instructors that we interviewed teach in the elementary school. private school b is also a large institution, catering to high ses students. it is also a unit of a larger educational group, from which it uses school contents but with local autonomy. in comparison with private school a, private school b has a greater variety of programmes integrated to the curriculum and greater drive for pedagogical experimentation, using technology in manners that are both constructive and informational. technical training is offered to help teachers in making use of technologies in classrooms. the school also invests in the training of the management team. the use of technologies in school b is characterised by the existence of coherent projects, determined cooperatively by the principals, coordinators and teachers. managers and some teachers of private school b have undergraduate or graduate degrees in areas related to educational technology. there is a recent trend to integrate technologies inside the classroom, which are all equipped with projectors and interactive white boards. there are also four figure 2. school a diagram. research in learning technology citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 differentiated learning environments built to cater to the needs of specific knowledge areas (such as humanities). the school offers digital and board games that students use in a specific room and that are also meant to be used at home with the parents. the robotics programme is part of the compulsory curriculum. the school offers technological and pedagogical support for the activities that use technologies. similarly to school a, school b pays only 2 hours per week for class preparation, which might inhibit experimentation in the classroom by the teachers (figure 3). there is a gradual transition in the use of technologies in the school. the computer lab � a model considered outdated � is giving place to the use of tools in the classroom and in the differentiated learning environments, which leads to a shift in how school personnel, particularly the laboratory teacher, perceive themselves (tensions toolsdivision of labour). students and parents have high expectations concerning the infrastructure and tools at school, having their personal electronics as a reference point. this might lead, in turn, to frustration with outdated computer lab equipment. the school constantly seeks updates and novelties to satisfy the expectations of students and parents, as well as to promote its pedagogical orientation, but teacher competencies and training do not always follow pace (tension actors-tools). even though there is a conscious integration between the activities involving technologies and the demands of the curriculum based on competencies, only part of the teachers use the technological resources provided by the school. for example, the interactive white board is frequently used as equipment for projection of images, not as an interactive tool. there is indication that some teachers are resistant to experimentation (tension actors-tools). curricular demands both by the government and by the headquarters have a great impact in the activities developed at the school. high school creates its own demands, teachers competencies equipment underuse first years teachers/more difficulties professional learning limited for the teacher (overloaded) community support parents don’t follow up on activities at home infrastructure old equipment clash with students expectations infrastructure wi-fi being implemented computer lab has less computers than students tools projector and interactive whiteboard in every room computer lab learning and content management system “learning rooms” educative games tablets for the teachers educational robotics class object technology integration new curriculam with focus in competencies and skills “extra” projects integrated with the curriculam teachers need to constantly “run” and learn. high-end schools need to update technology frequently computer classes became “instrumental” robotics uses another model (business/crafts) division of labor companies offer professional development in technology education and it support tools and infrastructure conflict between the computer lab and classroom model students parents project based pedagogy adapt to the student reality engagement with the problem situations high-end schools need to be up-tp-date on technologies there’s no “market” for teachers that dont’t know icts computers help content “stick” and “increment” class seduction for the “new” and different actors managers teachers time constraints curricular and hq demands time available for class planning culture school hq determines guidelines that are at times distant from the teachers reality vision and plan technology: more than instrumental moving away from the computer room model regulation/legislation curricular contents defined by mec and the hq figure 3. school b diagram. t. amiel et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 with a focus on entry to university. this in turn reduces activities that are more constructive and/or make creative use of technologies. as an example, more advanced classes underuse the computer lab � as its use was, but no longer is, compulsory (tension actors-rules). the school offers activities that require the involvement of the parents, especially educational games to be played at home. some parents show resistance to this practice, failing to understand the pedagogical potential or simply because they don’t have time to participate (tension community-tools). public school c we interviewed the principal and two elementary school teachers. the school does not have a pedagogical coordinator. public school c is a small rural elementary school. it attends primarily the children of local workers. about a third of the students take part in the full-time education programme. unlike the other three schools, this school does not offer a robotics programme. even though it is a rural school, it has a good technological infrastructure, with a computer lab and classrooms equipped with projectors and computers. the individual initiative of the principal is the main explanation for the quality of the infrastructure. by means of partnerships, she/he managed to gather multiple resources and services for the maintenance and enlargement of the school. it is also evident that she/he played a key role in building a motivated team prizing the quality of the education in the institution. classes make intense use of the available technological resources. still, internet access is limited and unstable, and classroom activities do not benefit from it (figure 4). figure 4. school c diagram. research in learning technology citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 the school’s pedagogical programme does not comprehensively contemplate the use of technology in education. uses are defined on a case-by-case basis, without an overarching plan that connects them with a greater educational goal (tension rulesobject). the interviewed educators point to legislation � characterised by curricular goals � which also do not deal with the integrated use of technologies in educational activities. goals are defined based on content to be covered. there is a school culture that incentivises the use of technologies, which leads to the expectation of its use by educators. the teacher does not need to schedule the computer lab in order to use the equipment, as there are projectors and computers in each classroom. on the contrary, the computer is used only in combination with the projector, and there is little student engagement. in case of technical problems, equipment was available for immediate replacement. many teachers do not have sufficient technical training to use the available tools, which limits the creative possibilities for the use of new media (tension actors-tools). educators suggested that the interest in the use of new media differs markedly from teacher to teacher. the interest of the teacher does not necessarily have a direct correspondence to his or her technical competence, which is aggravated by the lack of specialised technical support. activities involving computers and projectors usually depend largely on the access to the internet. the schools internet link is insufficient for the use of audio or video with students, which again limits possible activities and contributes to an underuse of the resources (tension tools-object). the principal usually calls on the community for help in tasks like the building of furniture. on the one hand, this is positive, as it denotes community engagement. on the other hand, it points to the limited sustainability of the technology plan. there is neither a maintenance contract for the equipment nor available technical staff. this leads to instability in the activities that depend on technology. there is also no technical support for the activities themselves, a function that is answered for by the principal (tension division of labour-tools). finally, the object showed some contradictions. even though school actors say it follows a constructivist approach, the object of the technological integration seems more devoted to motivate students in the school activities, to attract ‘digital natives’. student motivation was a fundamental drive for the use of new technologies. public school d we interviewed the principal, a coordinator and two teachers, and made two classroom observations. the instructors that we interviewed teach in the elementary school. public school d is a midsize institution, with students from middle and low ses. about 10% of the students take part in the full-time education programme. the school has a programme for gifted youngsters, which serves students from public and private schools in the region. the school does not have computers, projectors or interactive white boards in the classrooms. it has rooms for projections and a computer lab, which need to be scheduled in advance by the teachers. the school has a robotics programme that was initially available only for the gifted students programme, but that is being opened to include all students from the full-time education programme. t. amiel et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 the school has a radio project, which includes web radios, and equipment used to edit sound and for transmission. a volunteer initially developed this initiative and the students and parents provide the necessary resources (figure 5). the principal does not have a personal interest in the use of technologies. however, she/he supports the activities that are developed by the school coordinator. there is no evidence of a culture of use of technologies by the teachers (tensions actors-tools). legislation and central planning create serious limits on how the school deals with equipment, both novel and old, and what it can purchase with available financial resources. there are difficulties in equipment maintenance: much of what is available is not used and cannot be legally discarded. the result is that the school has to keep piles of old equipment. when the school receives new computer equipment, there are operational difficulties in rejecting those that are dead on arrival (tension and rulesactors). moreover, there are no technical support personnel (tension division of labour-tools). some equipment is bought with resources donated by the parents or local companies. part of the students does not have access to equipment at home, which may affect their instrumental competence with the tools available at the school (tension tools-community). teachers have to teach an excessive number of classes, which makes it difficult to experiment and implement innovative or creative practices (tension actor-rules). the lack of resources contributes to limit the scope of activities that make use of the technologies in the school. the limited-bandwidth internet access provided by the government is not sufficient for activities involving student participation. in order to remedy this, the school collects community funds to pay for a faster internet connection. figure 5. school d diagram. research in learning technology citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 classrooms do not have projectors or computers, and when teachers need these, they have to schedule a specific room in advance. the students have to physically move there, which is recognised as a substantial waste of instructional time (tension tools-object). despite the above-mentioned tensions, school practices point to the possibility of digital inclusion for many students, as well as the increase in the participation by teachers in projects involving new media. some evidence for this can be found in the positive results achieved by school students in external robotics competitions. projects involving new media focused on the gifted students attract the attention of the other students. the school plans to include a great number of the students in these projects as it expands the full-time curriculum. there is, for now, integration between the gifted students and full-time education programmes, as some of the gifted students participate in activities along with the full-time students. analysis the data presented here suggest that the use of new media is strongly influenced by planning and administrative leadership. this in turn generates an environment more favourable to experimentation and use by teachers. when these conditions are not present, the pressure is on an individual actor, usually a teacher, who becomes an isolated agent � an exception to the rule. public schools demonstrate a more creative vein in the use of their limited resources. interesting and novel activities were identified, such as radio programme in one school and teaching about agriculture involving icts in another. the individual initiative determines the existence and survival of these programmes. there is greater teacher autonomy in public schools and also more frail sustainability. these actions are not part of a larger school culture and do not find systemic support. consequently, they might vanish if its champion disappears or momentary support dissipates. the distance between a distant headquarters and the school for day-to-day schooling can create an asymmetry in expectations. reduced time for planning a centralised curriculum and other determinations can make it hard for administrators and teachers to experiment. on the contrary, a well-defined structure can contribute to more cohesive action, one that envisions not only equipment but also a clear alignment between curriculum, practices, architecture and devices, as evidenced in one of the schools. these external actors provide a team that can examine curricula and legal demands, promote opportunities for professional development and mediate external agents (e.g. providers), which in turn can lead to a more sustained and cohesive transition. this might never occur were schools left to their own devices (and resources). in public schools, educators have to worry about routine material questions, such as maintenance and acquisition of equipment. in public schools, there is a clear effort by administration in the constant search for alternative ways to build a better school and technology infrastructure. this may generate interesting learning opportunities despite the lack of resources. though the private schools were similar in terms of student ses, prestige, location and espoused teaching philosophy, the analysis portrays markedly different roles for new media. this helps negate pedagogical determinism of technology richness. in one school, there is a preoccupation with the relationship between technology and curriculum, administration, access to resources and so on, as part of a process of t. amiel et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 change for the school as an organisation. the other school displays a more tenuous integration of technology. we identified tensions in both, but in very different terms. some comparisons across the public/private divide are possible. when comparing private school b with public school c, we found similarities in the way new media are integrated in their pedagogical proposals. school b focuses on projects and school c has a constructivist approach, perspectives which are usually associated with each other. in both schools, new media are integrated into subject areas and are used as a complement to classroom activities. on the contrary, private school a and public school d use technology primarily in extracurricular activities, apart from the regular subject areas. there are similarities between private school b and public school c, where equipment available in the classrooms is replacing the use of computer labs and technology outside the classrooms. in all four cases, classes follow a traditional teacher�student relationship, except in the case of robotics classes. robotics is a separate course and follows its own grammar. the robotics courses, which show a more participatory and active engagement by the students, are outliers in terms of method, content, mentors and goals. classes follow an entrepreneurial model, part of a methodology that is sold to schools as a package that goes beyond the hardware itself; this moves the experience closer to a shop or maker movement. the goals, for at least one school, are also external: participating successfully in one of many competitive robotics championships promoted outside of school. we identified cases where robotics activities are integrated with curricular objectives. unless these types of synergies can expand to promote structural change, it is doubtful that they will contribute to any meaningful change to other activities. as pressures from external examinations and community expectations for college entry increase, high school has also become, more than a level of schooling, a ‘school within a school’. here, the integration of technology is likely to be limited by the pressures of time and excessive content, and is not necessarily a function of the availability of resources or competence. regarding schools that have a high school level, systemic pressures to cover extensive content (or competencies), more so in private, but also in public schools, are evident and pervasive. in this context, any initiative that aims to increase experimentation, greater engagement and depth � where new media could be of interesting use � tend to be neglected. this represents a tension between the purported objectives of the school and the overarching rules, materialised in the form of college entrance exams. conclusion in this study, we propose a chat-based methodology for the systemic investigation of tensions and contradictions with a focus on technology integration in schools. while we recognise that to typify is to reduce, we believe there is value in modelling patterns of tensions. this study supports the idea that educational actors see the integration of technology in schools as a vector for ‘modernisation’, aiming to increase motivation and engagement of students. but external pressures and contingencies manifest themselves in pre-defined margins of manoeuvre and unique expectations by different actors (sun 2006). therefore, there is often a conflict between the purported objectives and the actual results. we did not identify a relationship between a robust infrastructure and pedagogical orientation. in spite of decades of discussions on the limited power of infrastructure and artefacts in directing change to school culture and teacher practices, we find research in learning technology citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 ourselves once again obligated to emphasise this point. we have evidenced that tools make themselves present and available in very different terms and constitute a significant part of how schools envision change in practice. brazil, like many other countries, continues to promote a failed model of technology integration propelled and focused on the large purchase of devices as agents of change (cornils 2011). some caveats are called for. it is not trivial to get private, prestigious schools to open their doors to scrutiny. teachers who were initially interviewed at the end of one semester were no longer part of the school when we returned to observe their classes. for public schools, the gdf’s school recommendations might have promoted a biased view on technology integration. in either case, observations were done on the suggestion of the principal, which points us in the direction of teachers and activities with exemplary or heavy ict usage. still, particularly in earlier years (1�5, 6�9), icts seem to be a tool for teachers’ use or extracurricular activity. if a bias did exist, our observations point to a more traditional and teacher-centred approach to integration. there is no evidence of administrator or teacher hesitance in presenting the dilemmas and tensions in their practice. this study was not aimed at evaluating the quality of teaching or school administration. our focus was exclusively directed at the object of this study. the results of the study will be presented to each school in hope that contradictions and tensions may lead to reflection. we hope that these insights and methods can help schools build more systemic and realistic strategies for technology integration, whatever their goal or scope may be. acknowledgements we would like to thank the participating school personnel, and the lemann center at stanford university, particularly dr. paulo blikstein, for support in the form of a visiting fellowship, where research into this methodology began. references amory, a. (2014) ‘tool-mediated authentic learning in an educational technology course: a designed-based innovation’, interactive learning environments, vol. 22, no. 4, pp. 497�513. cetic.br. (2015) tic educação 2014: pesquisa sobre o usa das tecnologias de informação e comunicação nas escolas brasileiras, nic.br, brası́lia. cornils, p. (2011) ‘um computador por aluno. quando? onde? como?’, arede, vol. 7, no. 74, pp. 10�17. cuban, l. (1986) teachers and machines: the classroom use of technology since 1920, teachers college press, new york. cuban, l. (2001) oversold and underused: computers in the classroom, harvard university press, cambridge, ma. cuban, l., kirkpatrick, h. & peck, c. (2001) ‘high access and low use of technologies in high school classrooms: explaining an apparent paradox’, american educational research journal, vol. 38, no. 4, pp. 813�834. demiraslan, y. & usluel, y. k. (2008) ‘ict integration processes in turkish schools: using activity theory to study issues and contradictions’, australasian journal of educational technology, vol. 24, no. 4, pp. 458�474. engeström, y. (1987) learning by expanding: an activity-theoretical approach to developmental research, orienta � konsulti, helsinki. engeström, y. (2004) ‘new forms of learning in co-configuration work’, journal of workplace learning, vol. 16, no. 1/2, pp. 11�21. t. amiel et al. 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 fishman, b., et al., (2004) ‘creating a framework for research on systemic technology innovations’, the journal of the learning sciences, vol. 13, no. 1, pp. 43�76. groff, j. & mouza, c. (2008) ‘a framework for addressing challenges to classroom technology use’, aace journal, vol. 16, no. 1, pp. 21�46. hew, k. f. & brush, t. (2007) ‘integrating technology into k-12 teaching and learning: current knowledge gaps and recommendations for future research’, educational technology research and development, vol. 55, no. 3, pp. 223�252. hornung, h. & baranauskas, m. c. c. (2013) conceptual frameworks for interaction design: analysing activity theory and organizational semiotics contributions, in proceedings of the fourteenth international conference on informatics and semiotics in organisations, eds k. liu, w. li & s. r. gulliver, informatics research centre, stockholm, sweden, pp. 138�146. ipea. (2015) polı́ticas sociais: acompanhamento e análise, ipea, brası́lia, brazil. iste. (2009) essential conditions: necessary conditions to effectively leverage technology for learning, [online] available at: http://iste.org/nets karasavvidis, i. (2009) ‘activity theory as a conceptual framework for understanding teacher approaches to information and communication technologies’, computers & education, vol. 53, no. 2, pp. 436�444. lim, c. p. (2002) ‘a theoretical framework for the study of ict in schools: a proposal’, british journal of educational technology, vol. 33, no. 4, pp. 411�421. lim, c. p. & hang, d. (2003) ‘an activity theory approach to research of ict integration in singapore schools’, computers & education, vol. 41, no. 1, pp. 49�63. lowther, d. l., et al., (2008) ‘does technology integration ‘‘work’’ when key barriers are removed?’, education media international, vol. 45, no. 3, pp. 195�213. manovich, l. (2001) the language of new media, mit press, cambridge, ma. murphy, e. & rodriguez-manzanares, m. a. (2008) ‘using activity theory and its principle of contradictions to guide research in educational technology’, australasian journal of educational technology, vol. 24, no. 4, pp. 442�457. nardi, b. & o’day, v. l. (1999) information ecologies: using technology with heart, mit press, cambridge, ma. park, j., et al., (2013) ‘a systems approach to facilitating ict-pedagogy interaction’, in tic & educação 2012, cetic.br, brası́lia, pp. 219�230. paula, h, de f.e. & moreira, a. f. (2014) ‘atividade, ação mediada e avaliação escolar’, educação em revista, vol. 30, no. 1, pp. 17�36. pelgrum, w. j. (2001) ‘obstacles to the integration of ict in education: results from a worldwide educational assessment’, computers & education, vol. 37, no. 2, pp. 163�178. salomon, g. (1991) ‘transcending the qualitative-quantitative debate: the analytic and systemic approach to educational research’, educational researcher, vol. 20, no. 10, pp. 10�18. salomon, g. (2002) ‘technology and pedagogy: why don’t we see the promised revolution?’, educational technology, vol. 42, no. 2, pp. 71�75. sorj, b. & lissovsky, m. (2011) internet nas escolas públicas: polı́tica além da polı́tica, working paper 6, centro edelstein de pesquisas sociais, rio de janeiro. squire, k. d. & reigeluth, c. m. (2000) ‘the many faces of systemic change’, educational horizons (spring), vol. 78, no. 3, pp. 143�152. sun, h. (2006) ‘the triumph of users: achieving cultural usability goals with user localization’, technical communication quarterly, vol. 15, no. 4, pp. 457�481. tyack, d. & tobin, w. (1994) ‘the grammar of schooling: why has it been so hard to change?’, american educational research journal, vol. 31, no. 3, pp. 453�479. zhao, y. & frank, k. a. (2003) ‘factors affecting technology uses in schools: an ecological perspective’, american educational research journal, vol. 40, no. 4, pp. 807�840. zhao, y., et al., (2002) ‘conditions for technology innovations’, teachers college record, vol. 104, no. 3, pp. 482�515. research in learning technology citation: research in learning technology 2016, 24: 31856 http://dx.doi.org/10.3402/rlt.v24.31856 15 (page number not for citation purpose) http://iste.org/nets http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31856 http://dx.doi.org/10.3402/rlt.v24.31856 calt12204.fm alt-j, research in learning technology vol. 12, no. 2, june 2004 course developers as students: a designer perspective of the experience of learning online iain mcalpine*,a, tony koppia, jan mcleana & elaine pearsonb athe university of new south wales, sydney, australia; buniversity of teeside, middlesbrough, uk taylor and francis ltdcalt12204.sgm10.1080/0968776042000216200alt-t research in learing technology0968-7769 (print)/1741-1629 (online)original article2004taylor and francis ltd1220000002004iainmcalpineeducation and development & technology centre, level 4 mathews buildinguniversity of new south walessydney 2052australiai.mcalpine@unsw.edu.au academic developers of online courses may not have experienced this mode of learning and teaching from the learner perspective. this article makes a comparison between suggestions for online course design from research literature and user perspectives from a focus group, responses to questions on the most and least effective aspects of online study and lasting impressions, and from reflective diaries kept by two of the authors while they were engaged in study from online courses. this direct evidence is used to highlight key issues in the literature from the viewpoint of the learner. introduction academics developing courses have their own experiences as students to draw upon. the experience of online learning, however, is still new to many people and many developers of online courses have no experience of learning online before trying to teach in this mode. although developers of online courses can draw on their experiences in other modes of teaching and learning, lack of experience as an online learner makes it particularly difficult for people to design and teach in this mode. this lack of student experience means they may not know what practices to avoid and therefore they may inadvertently cause unforseen difficulties for the students. this paper considers online learning from the perspectives of the teacher/designers, and the students, of courses that are entirely online for off-campus students. a major issue for teachers and designers is to attain a level of student engagement with the course, to keep students on track and making satisfactory progress, and to ensure that students are developing effective levels of knowledge and skill. in many instances, this means using online discussions as a substitute for on-campus tutorial *corresponding author. educational development & technology centre, level 4 mathews building, university of new south wales, sydney 2052, australia. email: i.mcalpine@unsw.edu.au issn 0968–7769 (print)/issn 1741–1629 (online)/04/020147–16 © 2004 association for learning technology doi: 10.1080/0968776042000216200 148 i. mcalpine groups, to maintain a dialogue among students and with the tutor on issues relating to course topics. these discussions present interesting challenges for both teachers and students who may be relatively inexperienced with this mode of learning and teaching. research literature on the constructivist approach to learning and teaching and the implications for online course design, is considered to identify some key design issues for online courses. this is considered in relation to student feedback from surveys, focus groups and reflective diaries, to gain student perspectives on online courses. these include feedback from the authors, reflecting on their own recent experiences as online students. student perspectives are compared with recommendations from research literature to suggest some practices to avoid, and some tips for better practice, from the perspective of the online student. online learning and teaching a key point identified from research is that teaching and learning online requires an approach to instructional design that embeds effective learning strategies. current approaches to curriculum design demonstrate a move from content-centred curricula to designs that focus on learning outcomes (oliver & omari, 1999). this change reflects the adoption of constructivist approaches to teaching and learning. active learning by problem solving is a key aspect of any constructivist environment (reeves & okey, 1996). oliver and omari (1999) suggest that effective university learning involves: ● high levels of activity and engagement; ● a motivating environment based on a well structured knowledge base; ● finding solutions to real-world problems; ● collaborative problem-solving; ● active communication among students and with the tutor. these learning activities are seen to support higher-order thinking and learning. courseware designed to incorporate these forms of constructivist learning activities is likely to support effective learning. many approaches to online courseware show a tendency to ‘reformat on campus course materials and learning strategies into an online form’ (parker, cited in oliver & omari, 1999, p. 2). this approach is considered to be an underutilization of the technology. some approaches to online material for off-campus students have been identified as using a ‘write/text’ paradigm. this is characterized by a didactic approach in which ‘learners were presented with screens of information followed by assessment activities to demonstrate recall and understanding’ (brunetto et al., 1999, p. 64). these approaches, while they achieve the important goal of enabling offcampus students to study online, do not apply the constructivist approaches developed to attain higher-order learning outcomes. course developers as students 149 constructivist approach the constructivist theoretical position has led to a range of approaches to instructional design of online courseware that exemplify specific approaches such as problem-based or case-based learning or collaborative learning. problem or case-based approaches immerse the learner in complex tasks that reflect the type of real-world issue that the learners will have to deal with as a working professional. online technologies are used to present the problem scenario and support the student with guidance, access to information, resources, and learning tools. problem solving activities may be individual or group activities (e.g. romiszowski, 1990; oliver & omari, 1999; minasian-batmanian et al., 2000). collaborative learning approaches develop skills in teamwork, negotiation, taking account of other perspectives, information seeking and analysis, and critical thinking. these approaches are also likely to be based around case-based scenarios, however the instructional process is specifically designed to be student focused and to develop skills in problem-solving as a group (milter & stinson, 1995; mcalpine, 2000). these approaches have been specifically developed to address deficiencies in typical graduate skill profiles by fostering higher-order cognitive skills and personal communication, problem-solving and teamwork skills. central to aiding knowledge construction is raising the learner’s awareness and understanding of the process of learning—that is to develop their metacognitive skills or ability to learn how to learn. this may involve, for instance, helping learners develop awareness of how they go about learning, their learning goals, the ability to plan and evaluate learning strategies, and to monitor progress (jonassen et al., 1993; reeves, 1999). this is also a crucial element in allowing students to gain control over their learning, and to become self-directed learners, which is widely accepted as a goal of student-centred learning (brookfield, 1986; laurillard, 2002). learning communities learning communities, or communities of practice, are considered to have evolved from the constructivist approach as a way of addressing authentic problems (johnson, 2001). their purpose and value are closely related to the key concepts of constructivism, including: ● open-ended problems; ● learning from real-world problems; ● having shared goals; ● using cognitive tools; and ● a tutor’s role as facilitator or coach (johnson, 2001). in a course structured on these concepts, the process of dealing with problems in a collaborative way is considered to foster, and be aided by, a sense of community among learners. processes that open up the learning process to facilitate participation, and to enable learners to move from peripheral to central status as quickly as possible are valuable in this context (wegerif, 1998). the role of the tutor in facilitating this 150 i. mcalpine process by providing guidance, and the course design in providing access and support, are important contributors to creating the appropriate level of collaboration (oliver et al., 1998). online discussion appropriate and effective use of online discussion forums is a critical part of the instructional design process for online learning, as this enables the communication and tutor facilitation processes. discussion forums, both synchronous and asynchronous, may be the main form of communication between the tutor and the students, and among students when the students are off-campus. as the online discussions play a critical role in the teaching/learning process, the use of discussion forums needs to be planned to integrate into the overall learning process. the vital part of using online discussion in any format is to keep the students actively engaged. palloff and pratt (1999) suggest a range of activities that foster active engagement, including: ● posting instructions and learning expectations; ● forming teams and posting guidelines for their performance; ● encouraging a search for real life examples; ● using dialogue as enquiry by encouraging thought provoking expansive questioning; ● sharing responsibility for facilitation among group members; ● promoting constructive feedback. methods for facilitating dialogue as enquiry may include using open-ended questions (e.g. ‘what do you consider to be the ethics of a computing professional?’ mcloughlin & o’sullivan, 2000, p. 47) or using statements that are designed to promote discussion (palloff & pratt, 1999). the tutor’s role the role of the online tutor is critical in attaining effective learning outcomes. tutors need to set appropriate activities and provide the right kind of guidance. feedback from experienced tutors indicates that online tutoring can be more demanding than tutoring in a face-to-face situation. some students show a high level of appreciation for online conferences, but students are likely to be dissatisfied if there is a lack of participation by others (mason & weller, 2000). for online discussion to be successful the topics must be clearly related and relevant to the learning situation, and the online discussion needs to be an integral part of the learning process. students need to be motivated through appropriate activities (salmon, 2002) and participation should be directly assessed or associated with assessment tasks. those who are not active in the course should be contacted and monitored by the tutor. without that support, even the most experienced learner can fall behind and out of a course (roffett, 2000). course developers as students 151 guidance and scaffolding traditional approaches to instructional design emphasize the importance of student and instructor guides as components of a learning package (e.g. dick & carey, 1985). online courses may not use this design element, which can lead to problems for students using online courseware. in an evaluation of a range of online courseware, brunetto et al. (1999) identify learning supports, or scaffolding, as an essential element of online learning (with learning activities and content). they also found that scaffolding was not a common design aspect in many of the online courses and that the means of doing this varied widely due to the lack of standard approaches to design for online courses. they suggest several approaches to this, including: ● learners can be encouraged to scaffold learners; ● teachers/trainers/facilitators can act to scaffold learners; ● learning systems can be designed to track learners’ progress and to provide intelligent tutoring to guide further activity based on previous learning success; ● electronic support systems can be created with the capacity to provide assistance at the point of need (brunetto et al., 1999, p. 67). scaffolding can be considered as guidance and support needed at the beginning of a course or activity that can be withdrawn or ignored once the student is competent to manage the activity. hannafin et al. (1999, p.131) describe a range of guidance or scaffolding approaches that may be necessary in courseware to assist the learner towards the most effective learning. these are: procedural guides the learner in using the technology in order to engage in the learning activities. conceptual guides learner in what to consider; considerations when problem task is defined. metacognitive guides how to think during learning: ways to think about a problem under study and/or possible strategies to consider; initial role in finding and framing problems, and ongoing role during resolution. strategic guides in analysing and approaching learning tasks or problem; provided initially as macrostrategy or ongoing as needs or requests arise. student guidance, focusing on these aspects, may be incorporated into the course in different ways. procedural guides would focus on finding a way around the course materials and how to do some of the basic organizational aspects. other guides might relate to study skills, such as managing the learning process and working with other students on group tasks. these guidelines may be embedded in the courseware or actions by the tutor, or both. 152 i. mcalpine review of courseware design issues the constructivist approach, with its emphasis on open-ended problem solving and collaboration, is considered by many to be the most effective design model for an online course. this approach is based on models for learning and teaching that have been developed to achieve high-level learning outcomes. critical factors for online course design are to create learning activities based on constructivist principles, and to design online processes that enable these learning activities even though the students are off-campus. online discussions play a key role in enabling group problem-solving, the formation of a learning community, the facilitation of knowledge construction by the tutor and provision of guidance in the most effective approaches to the learning task. online course design also involves the provision of access to relevant information, and to an appropriate level of guidance, through the tutor and other means such as guides to process and technical support, to enable the student to focus on the learning task and to carry it out effectively. the following section of the paper describes the authors’ approach to gathering feedback from student experiences of online study, taking account of the desirable online course features considered above. method the aim of the project was to gather data that reflected the experiences of the authors and close colleagues in relation to being a student on a course conducted entirely online, and to consider how these data reflected on approaches to online course design. awareness of the desirable attributes of an online course, outlined above, was an important consideration in the study. by taking account of the student perspective, an examination of the implementation of course design could be conducted. this reflects some of the outcomes of the design approach, and leads to some suggestions for providing effective support to students in the design and implementation of online courses. research questions the following questions were used as a focus for this study: ● what elements of the design of online learning courses contribute to effective outcomes? ● what can the experience of online learning contribute to the improvement of online course design? data gathering the following data sources were drawn upon: ● reflective diaries kept by two of the authors while they were students in an online course on the topic of online learning, studied in 2001. course developers as students 153 ● a survey that included some authors and close associates who were also developers. the survey had three open-ended questions, asking about positive experiences of online study, negative experiences, and lasting impressions. surveys were sent to five people and four were completed. ● a focus group, conducted by one of the authors, with a group of students who were experienced in online learning. this group had completed a year of part time study for an online masters program. one of the authors was involved in the design of the course, and was known to the students. twelve students participated in the focus group at the end of the 2000 academic year. the method of analysis was to group the responses thematically, so that a range of comparative and contrasting perspectives were available on the key issues. data from all sources have been aggregated to enable a thematic grouping, using the actual words from the data sources as much as possible, to reflect the student experiential perspective (seidman, 1991). results student comments from the focus group and the diaries reflect the major issues for course design outlined in the literature review: ● course design around open-ended, real-world problems; ● group problem-solving; ● developing a learning community; ● the tutor’s role as a facilitator; and ● course design for guidance and student support. the student perspective on each of these is considered below, with the addition of some lasting impressions of the overall experience of online learning taken from the diaries and the survey. as the inclusion of all responses would take up too much space, a selection indicating the overall directions of the comments is included. for each comment the source, the reflective diaries, the survey, or the focus group, is indicated in brackets. course design around open-ended, real-world problems while this is an important issue, it did not elicit a wide range of student comments. it is likely that the students commented mainly on the experience of dealing with the problems, which is covered under the next topic of group problem-solving, rather than the nature of the problem tasks. some comments on the nature of the learning tasks are: courses i enjoyed the most were those that included regular amounts of small assessment or regular activities, this ensured that i logged into the course regularly. this may not be a fault of the course but does help keep students involved with the online course. (survey) 154 i. mcalpine the assignments … were interesting and motivating—due to being ‘real life’ authentic tasks, with real contexts. (survey) the tasks all promoted learning and i felt i learned a lot both pedagogically and technically. (survey) have just completed the first assignment which was a well-drafted, authentic task—quite enjoyed it and feel at least i have done something! (reflective diary) it appears that the use of real-world problems as a focus of the learning activities was valued. additional perspectives on the nature of the experience are given in the next section. group problem-solving the process of learning from group tasks drew a wide range of comments. this issue, more than any other, seemed to attract the students’ attention. opinions were polarized, with the students tending to either like it or hate it, depending upon the experience. a student from the focus group, who had completed several subjects online said: yes it was good—nothing will ever replace sitting in a room with people and talking to them but to do the group work … (online)—you certainly got to know the group members much better and you interacted with their thoughts and ideas and that’s the way you learned. some comments indicate positive experiences: i teamed up with a chap from new zealand and i found that enjoyable as we worked well together. it was good to have that level of collaboration and i was pleased with the assignment tasks we did. (survey) once we worked out who could do things during the week and who could do things on weekends then everyone was comfortable with that you could then share the work so that it was easy and it worked well once you worked that out but working out the logistics often took the time—once the logistics were worked out then it flowed pretty well. (focus group) collaboration was useful for accomplishing a task—discussions were actually carried out by three one-hour telephone conferences (focus group) we tended to find though that a couple of telephone calls we’d organize it quicker than with the chat systems—our group got particularly well organized for that particular session … it then proceeded to run extremely well in pushing our assignments forward. (focus group) one respondent commenced a course using a collaborative approach and was asked, during the course, to change to individual work. the reflective diary response was: i think the course has lost a lot because of the loss of collaboration. it actually took an extra effort to work like that but the learning should have been better. it is now easier just to do my own thing and not worry about any body else or have to incorporate anyone else’s points of view. course developers as students 155 the respondent’s solution was to substitute work colleagues for fellow students: i actually talk about the topic tasks with people here at work and receive comments and suggestions from them prior to and whilst doing the task. i find it very useful to consult with my colleagues to help consolidate and formulate my ideas. it’s good reflection stuff. i am using my work colleagues in lieu of my classmates to discuss the coursework. in general, the responses relating to collaborative work were positive. one student, however, points out some associated stresses: it’s quite positive the experience but you can overdo the group stuff too because i found the group stuff quite time consuming so whilst i agree that yes it’s valuable and yes we did work out how to do it more effectively as we went along, having to do group stuff so often with so many units in a year when a lot of us travel away from our places of work, etc., etc., my judgement is ‘yes if you ask me i’d say continue with it but let’s not do it at every opportunity—let’s be strategic about when you choose for us to do group work … (focus group) collaborative tasks can cause major problems if students are not motivated to work together, as these responses indicate: the collaborative task isn’t going well. the person has disappeared. he sent me an email yesterday (four days after my initial one) which shows he isn’t clear about the task and clearly hasn’t started any work on it. i really need to get on with it as i can’t extend my deadline … so i feel a real sense of urgency. i don’t think this is a great task to collaborate on. i find collaborating with someone i don’t know really uncomfortable. mason and weller (2000) suggest i do task 2 alone given the problems with collaborating… (reflective diary) there’s a major problem with group work if you can’t get the cooperation of one person … (focus group) the courses that are the focus of these comments clearly made use of group tasks as an important part of the course design. when this process works well, students appear to find it rewarding, leading to the intended outcome of learning to work as a team, sharing ideas among the group, and a deeper understanding of the topic. this outcome certainly isn’t guaranteed however, as collaboration can go awry very easily if some members of a group do not contribute effectively to the process. group processes may also place additional stress on the group members, and could be overdone in some situations. developing a learning community while some students appeared to enjoy the process of group work online more than others, the sense of being a part of a learning community did not figure in the comments. if the group process worked that was fine, but the community aspect or sense of belonging to a group only elicited comments when its absence was particularly felt, as is seen by the following: i would have preferred ‘traditional’ face -to-face interaction. i found the lack of being part of a ‘real’ learning community frustrating and the on-line set up didn’t compensate. (survey) 156 i. mcalpine … the feelings of isolation and not really feeling part of a class—i never got to know anyone other that the tutor (whom i got to know very well)—while we were invited to contribute to discussion forums etc, i felt we needed more structure to ‘force’ us to engage with each other. (survey) to work in a team when you don’t know what people look like is a bit strange to start with. (focus group) … still no attempt to ‘pull together a class’—i have no feeling whatsoever of actually studying a subject. (reflective diary) students who experienced success with group processes may have felt a sense of a learning community, evidenced by their satisfaction in working together. this could be considered to be a part of the group learning process rather than a separate issue. it has been considered separately in this study because of some literature that focuses on this issue directly, and also because of the strength of the students’ comments. this issue may also overlap with the next sections on the role of the tutor and the course design, as these may influence the perception of belonging to a learning community. the tutor’s role as a facilitator the role of the tutor in an online course may be critical to the degree of success. as students are both isolated and probably coping with a new approach to learning and new technology, they naturally look to the tutor to provide a positive input. it appears that if this doesn’t happen, perhaps because the tutor holds the belief that the students will work things out for themselves, then the students may experience some concerns. some comments reflect this: … there was never any real ‘start’ to the course where we all had to do something together and therefore to find out who else was studying. (survey) the person running the course did not make much of an effective contribution. (survey) my worst experience was my first online course, the lecturer was very absent from the course, email did not get answered for a considerable period of time, the bulletin board was empty and many students complained about a feeling of being lost in the course. (survey) the tutor’s role, however, can be seen to be very effective: … the level of tutor support, feedback and responsiveness (and flexibility) … was extremely high. (survey) very much appreciated [the tutor’s] response (via email). a positive aspect of the work so far is (the tutor’s) openness to feedback and responsiveness to issues raised. i’ve realized that one advantage of online learning is that the teacher can respond rapidly to issues/ concerns and let everyone know outcomes immediately. (reflective diary) the critical issue for the tutor appears to be the degree of involvement in the online discussions. moderating these discussions can be seen as a new skill, about which a body of research is emerging that is leading to a range of guidelines such as those in salmon (2000, 2002). course developers as students 157 course design for guidance and student support respondents were strongly negative about situations in which there did not appear to be an appropriate level of guidance, or ‘scaffolding’. when the way to use the online materials was not clear, this led to high levels of frustration and confusion: there is no place to go to ‘get started’ which tells us how we are going to work together, what we will do, etc. (reflective diary) interface design—irrelevant metaphors; distributed information on any one topic, especially assessment; spurious labelling, e.g. weekly tasks under either ‘computer laboratory’ or ‘lecture theatre’. (reflective diary) what does a discussion forum need to be student centred? it needs to be easily accessible, easy to read messages in a threaded format, easy to post replies to any particular author, and easy to post a new question (i.e. start a new thread). several of these student centred attributes are missing from the current forum. (reflective diary) the final assessment was poorly ‘scaffolded’ for students … i.e. no preparation … and seemed to be on a different topic to the course. (survey) it came as a big shock in about week 5 to discover a large assessment task worth 30% of the course was due imminently. i was working on the assumption that the submissions of the 2 weekly tasks … was the assessment for the course. i now understand that these tasks (many of which involve considerable time and effort) are only worth 30% of the course. this is disproportionate to the effort required … i should have been more strategic: work out where the big value tasks are and work on those. (reflective diary) the data do not contain balancing comments of a positive nature about course design, scaffolding and guidance issues. perhaps it can be considered that if the students are satisfied then the course design is good as it does not confront the student with any problems so they can concentrate upon the learning tasks. when the design leaves students confused or with a sense of no direction, as the above comments seem to indicate, then there appear to be shortcomings that need to be addressed. the above comments are mostly from reflective diaries, indicating a sense of frustration emerging from course design that did not seem to provide a clear path for the students to follow. lasting impressions what was the verdict at the end of the course? some students were left with a bad impression of online study, because a particular course was poorly managed or didn’t work for them: remains nearly 6 months later overwhelmingly negative. both of the course, the ‘flexible delivery’ mode and the school involved. (survey) for one respondent, the lasting impression is: what i learned from the tasks and also that scaffolding is very important in the implementation of an online course. its importance is paramount. (survey) 158 i. mcalpine fortunately, despite the inevitable problems or hurdles, most of the respondents had positive comments about the overall experience of studying online. an example of lasting impressions tells a more favourable story: overall i’ve enjoyed studying online, i have now completed five masters level papers online over the past three years. not surprisingly, online courses are greatly effected by the abilities of the teaching staff running the course, perhaps even more so than face-toface classes. flexibility! since starting my online masters course i have shifted countries, if i had started a traditional masters program it would have been much more difficult to continue studying. i was also able to continue with my studying while travelling. the other thing i enjoy about online courses are lecturers who are responsive, i’ve experience some very good online lecturers who have been very responsive but have also let students time to interact with each other and discuss issues. feedback from the online teacher was perhaps the biggest factor effecting my reaction to the course, as well as the amount of interaction happening in the bulletin board. (survey) discussion the project reported here, set out to answer the following questions: ● what elements of the design of online learning courses contribute to effective outcomes? ● what can the experience of online learning contribute to the improvement of online course design? in considering the extent to which these questions have been answered, the discussion will focus on the five themes of: ● course design around open-ended, real-world problems. ● group problem-solving. ● developing a learning community. ● the tutor’s role as a facilitator. ● course design for guidance and student support. course design around open-ended, real-world problems the first key course design element is a constructivist approach of using problem or case based, real-world scenarios as a focus for the learning process (oliver & omari, 1999). this approach was used to some extent by most of the online courses included in the data gathering process. several courses made group problem-solving a central part of the course design. while the nature of the task did not elicit many comments (unlike the actual group process) the nature of the comments indicated that the students appreciated the authentic tasks. course developers as students 159 group problem-solving the next key design element is that learning from problem or case based scenarios should be a group process (oliver & omari, 1999; mcalpine, 2000; minasianbatmanian et al., 2000). this aspect of design elicited mostly positive comments, although there were situations in which the group process didn’t work, which led to high levels of frustration. one important factor evident from the focus group is that it takes time to develop the necessary capability to work as a group online. the students had completed several courses online and knew most of the other participants well, and had learned to work this way. the respondents who experienced frustration were new to the experience of online group work, and often found that the people they were supposed to work with were not active in the process. comments indicate that the respondents regretted losing the opportunity to collaborate online. it appears that some students had enrolled in the course but failed to participate in an effective way, despite this being a course requirement. this suggests that additional management processes may be required to kickstart the group process. salmon (2000) suggests a range of activities, including easy introductory activities, to involve the students in tasks that are easily manageable before asking them to engage in a group task online, which is complex and demanding. the students in the focus group were brought together for an intensive week’s study before starting the online courses, so they would know each other and clearly understand the importance of online group work. it is likely that tutor follow-up is needed if individuals do not contribute effectively. developing a learning community the third key design element is the development of learning community. it is suggested that a learning community or community of practice is critical in dealing with complex open-ended tasks using a constructivist approach (johnson, 2001). the respondents do not use language that would indicate they experienced a sense of community; however, negative comments made it clear that some students were looking for this and didn’t find it. one comment made clear that the provision of a discussion forum did not mean that people would use it, and using the discussion should be a requirement of the course. the student feedback indicates specific needs, to have a sense of belonging to a group, that were not met. this raises questions about how this can be done. clearly the use of the online discussion should be a requirement for course participants, possibly as an assessed activity. an optional activity may not engage most of the students, who will naturally focus on what they are required to do. the answers to this issue may overlap with the previous aspect of group work and the following aspect related to the tutor’s role. the tutor’s role as a facilitator the role of the online tutor is the next key design element. palloff and pratt (1999) suggest a range of proactive tutor activities including posting instructions and expectations and using dialogue as a process of inquiry. a broader range of tutor activities 160 i. mcalpine is provided by salmon (2000). feedback on this issue is clear cut. some comments indicate that the tutor was largely absent from the online course and so no activities were undertaken that would facilitate engagement by the students. these comments indicate considerable frustration at the lack of tutor involvement. by contrast other comments indicate strong appreciation of the tutor’s willingness to respond and provide support. this suggests that tutors need to learn appropriate techniques for moderating online discussions. course design for guidance and student support the final design element, course design for guidance and student support, seemed to create many issues for the students. this is a critical element that is a mix of the way the online courseware is designed to provide navigation, the presence of appropriate student guides within the coursework, and the way that the course is managed by the tutor. brunetto et al. (1999) suggest that this element is not well done in many online courses. hannafin et al. (1999) propose several aspects of guidance or scaffolding required. these include procedural, conceptual, metacognitive and strategic aspects relating to the learning tasks, for which different forms of guidance may be needed. student feedback indicates some dissatisfaction with this aspect of course design. most of the issues seem to revolve around procedural matters rather than the conceptual material. this referred to difficulties in navigating the coursework and knowing what the assessment tasks were and where they were to be found. confusion through lack of guidance or difficult navigation caused apparent frustration that may have influenced the student’s attitude to the course overall. this may involve a careful consideration of basic aspects of online courseware design to avoid these problems. these design elements all appear to be important elements that contribute to an effective online course. the student perspective highlights the major issues for developers to focus on, to ensure that the course is effective. conclusions in this study the student perspective on course design has been used to illuminate some of the strengths and weaknesses of learning and teaching online. the constructive approach to course delivery was used as a tool to interpret and analyse the student experience and perspective. this approach has revealed that for the range of online courses included in this project, the most notable weaknesses were support for group processes, building a learning community, and courseware design to facilitate student access to all relevant components and information. these can be seen as issues relating to a fundamental aspect of course design, how learning and teaching is going to work for the learner, and secondly to aspects of scaffolding and student support to reinforce the learning and teaching process. the five key design elements all seem to be critical to the development and implementation of an online course that will foster high-level learning outcomes. course developers as students 161 furthermore, by enrolling in online courses as students, course designers have gained a useful insight into where the emphasis needs to be placed to improve the effectiveness of online courses. references brookfield, s. (1986) understanding and facilitating adult learning (san francisco, ca, josseybass). brunetto, y., wharton, r. f., oliver, r., skippington, p. & towers, s. (1999) toolboxes for online learning: evaluation report of the anta toolbox initiative (brisbane, australian national training authority). available online at: http://www.anta.gov.au/toolbox (accessed july 2000). dick, w. & carey, l. (1985) the systematic design of instruction (2nd edn) (london, scott foresman). hannafin, m. j., land, s. & oliver, k. (1999) open learning environments: foundations, methods, and models, in: c. m. reigeluth (ed.) instructional-design theories and models, vol. ii (mahwah, nj, lawrence erlbaum), 115–140. johnson, c. m. (2001) a survey of current practice in online communities of practice, the internet and higher education, 4, 45–60. jonassen, d., mayes, t. & mcaleese, r. (1993) a manifesto for a constructivist approach to uses of technology in higher education, in: t. m. duffy, j. lowyck & d. h. jonassen (eds) designing environments for constructive learning (berlin, springer-verlag), 231–247. laurillard, d. m. (2002) rethinking university teaching: a conversational framework for the effective use of learning technologies (2nd edn) (london, routledgefalmer). mason, r. & weller, m. (2000) factors affecting students’ satisfaction on a web course, australian journal of educational technology, 16(2), 173–200. mcalpine, i. (2000) collaborative learning online, distance education, 21(1), 66–80. mcloughlin, c. & o’sullivan, m. (2000) educating software engineers for the 21st century: assisting students to develop communication and team skills for success in the it profession using the www, open learning 2000: proceedings of the 4th international conference on open learning, brisbane, december 6–8 (brisbane, learning network queensland). milter, r. g. & stinson, j. e. (1995) using lotus notes to facilitate action learning, paper presented at the sixth annual business/economics teaching conference, chicago il, 10 november. available online at: http://mbawb.cob.ohiou.edu/paper1.html (accessed 19 august 1998). minasian-batmanian, l. c., koppi, a. j. & pearson, e. j. (2000) developing lifelong learners: a novel online problem-based ultrasonography subject, in: d. squires & g. conole & g. jacobs (eds) the changing face of learning technology (cardiff, university of wales press), 137–148. oliver, r. & omari, a. (1999) using online technologies to support problem based learning: learners’ responses and perceptions, ajet, 15(1), 58–79. available online at: http://www.ascilite.org.au/ajet/ajet15/oliver.html (accessed april 2003). oliver, r., omari, a. & herrington, j. (1998) exploring student interactions in collaborative world wide web computer-based learning environments, journal of educational multimedia and hypermedia, 7(2/3), 263–287. palloff, r. m. & pratt, k. (1999) building learning communities in cyberspace: effective strategies for the online classroom (san francisco, ca, jossey-bass). reeves, t. c. (1999) teaching and learning online: opportunities and responsibilities. keynote address to net*working ‘99 conference, melbourne, 2 september. available online at: http:// www.nw99.net.au/papers/reeves.html (accessed may 2001). reeves, t. c. & okey, j. r. (1996) alternative assessment for constructivist learning environments, in: b. g. wilson (ed.) constructivist learning environments: case studies in instructional design (englewood cliffs, nj, educational technology publications), 191–202. 162 i. mcalpine roffett, a. (2000) confessions of an e-dropout, it forum paper #51. originally in training magazine, august 2000. available online at: http://www.trainingmag.com. romiszowski, a. j. (1990) the case study methodology: interactive media and instructional design, in: h. e. klein (ed.) problem solving with cases and simulations (world association for case method research and application (wacra), waltham, ma, bentley college press). salmon, g. (2000) e-moderating: the key to teaching and learning online (london, kogan page). salmon, g. (2002) e-tivities: the key to active learning (london, kogan page). seidman, i. e. (1991) interviewing as qualitative research (new york, teachers college press, columbia university). wegerif, r. (1998) the social dimension of asynchronous learning networks, journal of asynchronous learning networks, 2(1), 34–49. course developers as students: a designer perspective of the experience of learning online iain mcalpine*,a, tony koppia, jan mcleana & elaine pearsonb doi:10.3402/rlt.v24.26497 original research article nurturing global collaboration and networked learning in higher education catherine cronin a *, thomas cochrane b and averill gordon c a centre for excellence in learning and teaching, national university of ireland, galway, ireland; b centre for learning and teaching, aut university, auckland, new zealand; c school of communication studies, aut university, auckland, new zealand we consider the principles of communities of practice (cop) and networked learning in higher education, illustrated with a case study. icollab has grown from an international community of practice connecting students and lecturers in seven modules across seven higher education institutions in six countries, to a global network supporting the exploration and evaluation of mobile web tools to engage in participatory curriculum development and supporting students in developing international collaboration and cooperation skills. this article explores the interplay of collaboration and cooperation, cop and networked learning; describes how this interplay has operated in icollab; and highlights opportunities and challenges of learning, teaching and interacting with students in networked publics in higher education. keywords: communities of practice; networked learning; collaboration; cooperation; social media; mobile social media introduction the communities of practice (cop) concept, rooted in social learning theory, emphasises collaboration across strong ties and the process of membership from initial peripheral participation to fuller participation as members of a community engaged in a process of collective learning (lave and wenger 1991; wenger 1998; wenger, mcdermott, and snyder 2002). with the growing prevalence of networked individualism and the ubiquity of social media, cop in higher education must be considered in a wider context of networked communication, participatory culture and networked learning. icollab is a community of practice that intentionally operates within and across networks, building upon the principles of cop and networked learning as well as connectivism (downes 2007; mcconnell, hodgson, and dirckinck-holmfeld 2012; siemens 2005). icollab connects students and lecturers in seven modules across seven higher education institutions in six countries. the ethos of icollab is open, connected and democratic, seeking to build a sense of trust and mutually beneficial relationships among and between students and lecturers. the interplay and complementarity between collaboration and cooperation, communities and networks, and strong ties and weak ties lies at the heart of the icollab project. this article begins with an exploration of cop and networked learning, highlighting the potential tension between them as well as their complementarity. *corresponding author. email: catherine.cronin@nuigalway.ie research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 c. cronin et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 then, the rationale, inception and evolution of the icollab project is described, with a focus on its evolution into a global network of educators and students. the article concludes with reflections of the authors and icollab practitioners on the opportunities and challenges of learning, teaching and interacting with students in networked publics in higher education. collaboration and networked learning the concept of cop describes how people ‘engage in a process of collective learning in a shared domain of human endeavor’ (wenger 2006). the concept can be, and has been, used to describe and theorise learning communities in higher education (hodgkinsonwilliams, slay and siebörger 2008). higher education now takes place within a sociotechnical context that is changing rapidly. this changing context is characterised by ubiquitous connectivity, a shift from knowledge scarcity to knowledge abundance and a move from hierarchical towards networked forms of social organisation. as networked individuals, people move within, between and beyond various cop throughout their lives (castells 2010; rainie and wellman 2012; ryberg and larsen 2008). thus, while the cop concept focuses on strong ties and collaboration, more loosely tied cooperative modes of learning are increasingly evident (ryberg, buus, and georgsen 2012). whereas a cop is sustained via strong ties of a shared domain of interest of the participants, a network tends to function around more flexible links between participants on a more ad hoc basis. theoretical frameworks such as connectivism and networked learning seek to understand learning in this broader networked context. connectivism is the thesis that knowledge is distributed across a network of connections; learning consists of the ability to construct and traverse those networks (downes 2007; siemens 2005). in the connectivist model, a learning community is considered a node in a larger network. networked learning theory is rooted in an understanding of learning as a social, relational phenomenon, specifically locating learning and knowledge construction in the connections and interactions between learners, teachers and resources (mcconnell, hodgson, and dirckinck-holmfeld 2012). jones and esnault (2004) highlighted tensions that exist between the metaphor of networks and cop. while cops blend the individual and collective in a shared commitment and a common domain, networks have no collective dimension. access to networked information flows and exchanges may be direct or indirect, intentional or serendipitous (wenger, traynor, and de laat 2011). however, the two poles are not necessarily opposed; they may be complementary (p. 15): social learning is enhanced by a dynamic interplay of both community and network processes. such interplay combines focus and fluidity as it braids individual and collective learning. the work of fostering learning needs to take advantage of this complementarity. to make the most of such complementarity, social media plays an important role in both nurturing a community of practice and facilitating wider network interactions. nurturing cop is identified as a critical element in their sustainability (wenger, white, and smith 2009). social media, particularly mobile social media (msm), can be an enabling platform for collaboration and cooperation across temporal and c. cronin et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 spatial boundaries. the interplay and complementarity � between collaboration and cooperation, communities and networks, and strong ties and weak ties � lies at the heart of the icollab project. icollab is a community of practice that intentionally operates within and across networks. while social media usage has been prevalent among higher education students for a decade, only in recent years has its viability as a learning medium been considered by a growing number of educators (arteaga sánchez, cortijo, and javed 2014; czerniewicz and brown 2013; junco, elavsky, and heiberger 2013; reed 2013; sugimoto et al. 2015; tess 2013; timmis 2012; waycott et al. 2013). selwyn (2010) noted three factors that tend to motivate the use of social media in higher education: recognition of students as connected, creative and networked individuals; the changing relationship between learners and knowledge, that is, construction versus consumption; and the rise of informal, user-driven learning. social media can be used to facilitate synchronous and asynchronous connections between and among students and lecturers � not only within courses and institutions, but also across multiple institutions, countries, time zones and academic terms. open sharing of media on personal mobile devices, outwith institutional virtual learning environments (vles), can support pedagogies which aim to challenge the conventional one-to-many power relationship between lecturers and students (mott and wiley 2009). cochrane et al. (2013) note that use of msm within icollab aims to support ‘shifts from teacher-directed pedagogy to student-generated content and student-generated contexts’, explored further below. icollab inception and evolution icollab is an international community of practice connecting students and lecturers in seven modules across seven higher education institutions in six countries: ireland, the uk, spain, germany, australia and new zealand. the goal of icollab is to explore and evaluate mobile web tools, engage in participatory curriculum development and create opportunities for students to develop international collaboration and cooperation skills (cochrane et al. 2013; cochrane and keegan 2012). icollab challenges the typical format of modules, courses and programmes, which run for a set period of time with a focus on discrete units of assessment; icollab students work across cohorts, levels, institutions, countries, academic terms and time zones, using social media and other mobile, web-based tools for multimedia production, collaboration, cooperation and reflective practice � as well as exploring the creation and negotiation of digital identities and personal learning networks. icollab has developed through five stages (see figure 1) based on the cop concepts of boundary crossing and brokering (wenger 1998). the first four stages were establishing a core community of practice, brokering participation, nurturing participation and brokering practice. these were defined and elaborated in an earlier article (cochrane et al. 2013) and are only briefly summarised here. in this article, we focus on the fifth stage: the evolution of icollab into a global network. msm has been utilised at all stages for scaffolding and sustaining interaction, both locally and globally. stage 1. establish a core community of practice the global core group of icollab participants was established over the course of four years (2011�2015) with the initial group of four growing to eight educators across seven institutions in six countries. research in learning technology citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 stage 2. broker participation in the second stage, students and lecturers, in their respective course cohorts, explored together the concepts of cop, networks, collaboration, cooperation, social media, digital identity and personal learning networks (boyd 2010; downes 2010; rajagopal, et al. 2012; rheingold 2012; wenger 1998). lecturers and students together identified activities relevant to their respective modules that would be good candidates for collaboration with a global cop. stage 3. nurture participation in the nurturing participation stage, lecturers intentionally and explicitly modelled participation within the cop. lecturers occasionally participated as remote contributors on selected topics in one other’s classes. students were encouraged to choose their own topics, develop their own research questions and create crossinstitutional and international teams to investigate and collaborate on these. students presented and shared their work via synchronous and asynchronous mobile video streaming. msm was leveraged for communication, curation and open sharing of student-generated content. stage 4. broker practice in the fourth stage, the focus was on sharing reflections and learning as outcomes of the cop through collaborative reflective scholarship shared via conference and journal publications (see icollab bibliography) as well as a range of social media tools. projects across consecutive years of courses and student cohorts are visible in student blogs and curated collections of student artefacts, accessible via the #icollab hashtag. these collections comprise a record of pedagogical change and students’ learning over time (cochrane et al. 2013). although not the focus of this article, reflections by students emerged in students’ own web spaces, for example: ‘i was stage 1 stage 2 stage 3 stage 4 stage 5 global cop mediated by mobile social media the reified activity of the icollab cop mediated by mobile social media evolution of icollab into a global network nurtured & sustained by mobile social media cops brokered by lecturers to their studentsestalishment of global lecturer cop global lecturer community of practice nz nz nz uk uk uk interested peripheral participants de de de es es es ie ie ie icollab global community of practice au au au nz students uk students de students es students ie students au students figure 1. five stages of icollab development (2011�2015). c. cronin et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 exposed to a broad range of technologies including apps, hardware, and social networking and marketing tools. the course provides a good platform for enhancing an online presence, which is vital to creative professionals in a competitive market’. (student blog post 2011); ‘thanks to this project i have transformed my understanding of social media usage, i understood how powerful it is’. (student blog post 2012); ‘i have learnt that social media/social networks are not just to be used as a distraction for not getting work done but can be used as an aid to get the work done. social media/social networks can provide useful tools to help with academic learning’. (student google� post 2012). icollab has used the collective activity of the global cop for brokering the concept to a wider audience of interested peripheral participants, including some within our own institutions and local networks, resulting in the development of a fifth evolutionary stage of the cop. stage 5. evolution of icollab into a global network as the icollab cop has grown and matured, it has created a foundation for an everevolving global network of educators, students and graduates. members of the lecturer cop continue to collaborate with one another and with others, beyond the icollab project. the activity of the icollab cop from 2011 to 2015 is briefly summarised in table 1 (for more details on the 2011 to 2013 icollab cop activity, see cochrane et al. 2013). as shown in table 1, beginning in 2013 the icollab cop began growing beyond a single collaborative project to encompass several different activities by 2015. the icollab network now provides a supporting framework for innovation in education across a variety of projects and course contexts. this network includes the 2013 mobile augmented reality movie workshop (marm workshop) facilitated at aut university and at the 2013 ascilite conference by icollab members, both face to face and virtually, several elective projects in 2014 and 2015 (cochrane and antonczak 2014, 2015a, 2015b; cochrane, antonczak, and guinibert 2014), a two-year national project across six new zealand higher education institutions (npf14lmd) (frielick et al. 2014), development of a lecturer professional development cmooc (cochrane, narayan, and burcio-martin 2015) and moco360 mobile film-making collaboration (keegan 2014). thus the fifth stage of the icollab cop has resulted in the development of a model of an ecology of msm resources for supporting a global educational network (figure 2). table 1. evolution of icollab iterations (2011�2015). 2011 2012 2013 2014 2015 project title icollab11 icollab12 icollab and marmw icollab, npf14lmd and autmsm npf14lmd, mosomelt cmoocand mobime project hub wikispaces wordpress wordpress google � google � wordpress google � wordpress participants 70 students 7 lecturers 5 courses 4 countries 70 students 5 lecturers 4 courses 4 countries 340 students/ participants 6 lecturers 5 courses 5 countries 1060 students 40 lecturers 7 courses 6 countries 140 students 45 lecturers 1 course 4 countries research in learning technology citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 icollab in action the icollab project began and grew by creating a flexible environment for collaboration and cooperation between several groups of lecturers and students, all involved in higher education courses in different contexts, but wanting to engage in open, creative and collaborative learning. the formal structures of seven different courses with seven different term schedules meant that activities could be synchronised at only four or five points in a given year. we used the ability of social media to attach the #icollab hashtag to all project-related content and communication, thus creating and curating a stream of icollab resources and conversations. each course group dipped in and out and contributed to the project stream as it fitted their time schedules and course priorities. a wide range of social media tools was used, including an open google� community, wordpress, twitter, google maps, wikitude, youtube, vine and bambuser. we also used several mobile-friendly hashtag curation tools to curate project-related activity, including tagsexplorer, flipboard, scoopit and tagboard. collectively, use of the #icollab hashtag across multiple tools and platforms helped to establish a sense of virtual community. all streams were linked within the project public archive wordpress open access publications google maps communication asynchronous twitter synchronous g+ hangouts community network hub & admin g+ community google docs participant eportfolios wordpress researchgate orcid curated social media channels #hashtag tagsexplorer youtube/vimeo/vine playlists figure 2. an ecology of msm resources to support a global educational network. c. cronin et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 google� community, which served as a project hub. figure 3 is a snapshot of a map of the 2013�2014 icollab participants. � icollab google� community: www.bit.ly/1ft1ww3 � icollab blog: www.icollab.wordpress.com � icollab participant map: www.goo.gl/maps/rklja � icollab contributor map: www.bit.ly/1fuq4t6 � #icollab network activity (tagsexplorer): www.bit.ly/19o0cpp � #icollab tagboard: www.tagboard.com/icollab/15606 students and lecturers were collaborative editors in this map, linking their online profiles to a geographical context. the map included 138 collaborators and attracted 6791 views in 2014, illustrating the breadth of interest in the project. one of the most important tools used by our global asynchronous cop has been twitter. in 2013�2014, there were 153 twitter nodes (active participants) and 756 edges (conversations) associated with the #icollab project. figure 4 illustrates twitter activity using the #icollab hashtag during november 2013, generated using tagsexplorer 1 . the establishment of significant nodes of conversation can be observed in this network diagram. while initially these were predominantly icollab lecturers, over time several students gained confidence and a significant voice within the virtual community, thus providing role models and mentorship for other students. the project also attracted a number of lecturers, researchers and others, not directly members of the cop but interested in the conversations, media artefacts and/or impact of the project. this provided multiple opportunities for focused twitter chats during which students could engage with academic staff 2 , authors whose work we were reading 3 and more. example icollab project collaboration within the icollab community of practice typically takes place among two or more cohorts for specific, purposeful learning activities. students use various figure 3. icollab participant map 2014. research in learning technology citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 7 (page number not for citation purpose) http://www.bit.ly/1ft1ww3 http://www.icollab.wordpress.com http://www.goo.gl/maps/rklja http://www.bit.ly/1fuq4t6 http://www.bit.ly/19o0cpp http://www.tagboard.com/icollab/15606 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 social media tools to share and communicate their work, ensuring both real-time and persistent visibility across multiple online spaces and audiences. use of the #icollab hashtag facilitates curation of resources and the development of a body of work � and identity � over time. as an example, during 2013�2014, students in ireland and new zealand collaborated within icollab by sharing their work, in various formats, across several social media platforms. second-year computer science and information technology students in ireland designed, created, delivered and shared presentations based on it-related topics of their choice. this activity followed class sessions during which topics such as digital identity, presentation skills, curation, copyright and creative commons were explored. students created and shared their work using a range of social media applications � all using the #icollab hashtag as well as the class hashtag #ct231 4 . at this stage, postgraduate public relations students in new zealand entered the #icollab activity stream, enabling peer-to-peer interaction and feedback. the public relations students were asked to identify, record and analyse the value of msm activity as an integral part of a public relations campaign for a particular organisation. students explored the affordances of msm such as twitter, vine and google� � illustrated by examples of the irish #ct231 student projects within #icollab. projects were presented in class, live streamed, collated via tagboard and figure 4. tagsexplorer twitter #icollab analysis. c. cronin et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 linked to the #icollab google� community where they received feedback from students and lecturers in ireland. this collaborative exercise was a way of not only teaching students about mobile media spaces and communication essential for both it and pr, but immersing students in those spaces with current tools and modelling the use of social media and msm for sharing and collaboration. the connections between students and lecturers in ireland and new zealand, via #icollab on various social media platforms, provided opportunities for peer-to-peer interactions, sharing and feedback across the two countries, exemplified via the sharing of student social media explorations using twitter, storify, vine and bambuser 5 . example icollab network activity based upon our icollab experiences, the mosomelt (mobile social media learning technologies) cmooc (cochrane et al. 2015) was designed in 2015 as a framework for upscaling lecturer professional development. the initiative was based on creating a global network of cop across universities utilising a connectivist massive open online course (cmooc) as a framework. the mosomelt cmooc framework links icollab participants as global experts into an authentic professional development experience via the integration of a range of msm learning technologies, including a webinar series and linking to the icollab cop core participants’ online social media profiles and activities as examples. the framework is designed as a series of triggering events over 24 weeks to support the development of participants’ personal eportfolios and pedagogical practice that then can be optionally validated by external certified membership of the association for learning technology (cmalt accreditation). mosomelt models a critical scholarship of technology-enhanced learning (sotel) through open access publications and social scholarship practices. mosomelt also models the use of an msm ecology of resources to support a global network of lecturer cops 6 . discussion the icollab lecturers who initiated and facilitate the icollab cop share a common understanding of higher education students, in all their diversity. we recognise that students, as networked individuals, enter higher education with existing identities, networks and practices � both digital and embodied. we do not ask students to leave these at the door (or the virtual door, in the case of vles). instead, we invite students to join a community of practice that is itself networked, to reflect on and develop their identities, networks and practices within the icollab cop and to the extent that they wish, in wider networks to which the icollab cop provides visibility and access. icollab is an example of a type of a pedagogical design described by ryberg and larsen (2008, p. 113): we wonder how networked learning systems would look if they were genuinely based on the metaphor of networks and intersections of weak and strong ties. for instance, one could imagine learning environments that took their departure in students’ and learners’ networks, interest groups and research projects rather than solely being constructed around subject matter and courses. social media can be a democratising platform, but democratic practices are not inherent in any technology or media. however, as part of a refocus upon students as research in learning technology citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 content producers and partners in negotiated learning outcomes, use of social media � particularly msm � can help in empowering students with a sense of personal agency in the learning process (cochrane et al. 2013; dabbagh and kitsantas 2012). this is particularly so if complex issues such as digital identity, privacy and data ownership are explored. if learner agency and empowerment is a goal of any pedagogy, those values must be established at the start, as was the case with icollab. since the inception of the icollab project in 2011, increasing numbers of educators are using social media to engage with students as collaborators in open educational experiences, to broaden learning activities beyond the classroom and to model and practice networked learning (costa 2014; keegan and bell 2011). newer pedagogical models, using open networked technologies and collaborative web tools, perceive education in radically different ways, for example, beginning with the premise that education is a ‘social and relational process that is distributed across and between physical and online spaces’ (van mourik broekman et al. 2015, p. 29). while many icollab students enjoyed international communication and peer feedback, building trust across networks is a complex process. creating a real sense of connection between seven disparate groups of students around the globe and building a platform of trust was one of the key challenges associated with the icollab project. while the icollab lecturers model this process, a lack of student experience of building professional communities can be a significant cultural shift for students (beetham and white 2014; helsper and eynon 2010). students require support, specific feedback and opportunities for reflection on the artefacts they produce for assessment, of course, but also on the processes of collaborative and cooperative learning, building trust across networks and building personal learning networks. without a formal external influence mandating project milestones or set outcomes, the icollab cop tends to ebb and flow as participants’ time pressures and interests allow. however, we observed positive outcomes and impact on student learning throughout the successive iterations of icollab (cochrane et al. 2013; cochrane and keegan 2012; cronin and cochrane 2014). as one lecturer in the icollab cop observed (keegan 2012): traditionally, we deliver modules/courses, neatly chunked into 12 weeks, with units of assessment, leading to grades, etc. and that’s the way things are (generally) done. i’m not saying scrap all of that, but i do think that modules are best served as springboards to other things. increasingly, students are connecting across levels and cohorts through twitter and now we have ex-students getting together with current students, undergrads coming to postgrad classes (and vice versa) as they’ve connected online and have a genuine interest in getting involved in other groups/further curricula outside of their taught modules. since the inception of icollab in 2011, the project has grown and evolved considerably. as shown in table 1, there was no new icollab project in 2015; however, successive collaborative activities that originated within icollab continue to emerge and evolve. at the heart of all of these international collaborative projects is a focus on student-directed content and pedagogy; creation and sharing via social media, particularly msm; and maximising the benefits of both communities and networks. over the past five years, the icollab cop/network has produced nine international conference contributions, five journal articles and one book chapter describing, analysing and reflecting on icollab pedagogical goals, designs and activities. the evolution of the icollab cop into a global network is reified in a number of c. cronin et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 professional development strategies including the mosomelt cmooc and an explicit focus upon developing a global research network around the sotel. conclusions veletsianos (2015) makes a strong case for the creation of curricula that helps scholars to make sense of networked identities, societies and cultures. we agree with this and would argue further that all students require such curricula and pedagogy, regardless of their future paths. in networked publics, and through the use of social media, students and educators can move beyond the rigid role definitions often prescribed for them within higher education. the teacher�student relationship can become open to change, with students and lecturers becoming co-learners in networked learning spaces (cochrane et al. 2013; rheingold 2012; stewart 2013). in networked cop, it is possible for students and lecturers to have more equal roles in creating content, sharing resources, participating in conversations and starting conversations. although the technologies themselves do not create democratic environments, educators who choose to engage with students in networked publics through the use of social media and other open tools and who engage in and model democratic practices can build cops and create spaces for powerful, student-driven learning. nurturing global collaboration and networked learning in higher education requires significant effort and commitment. educators who make a commitment to this endeavour are, in many senses, ‘building the raft while swimming’ (floridi 2014, p. 8). however, the mutual support of fellow networked educators helps when dealing with challenges that may arise. as icollab participants, we have found that the benefits of becoming part of a networked community of practice have enabled a new level of creativity and the potential for authentic global and cultural learning experiences � for our students, for ourselves and for a widening global network of educators. notes 1. see live tagsexplorer #icollab link for dynamic snapshot: www.bit.ly/19o0cpp 2. student-staff twitter chat: www.bit.ly/1ms8nsl 3. digital identities twitter chat: www.ct231.wordpress.com/2012/10/09/week-6/ 4. ct231 student showcase: www.scoop.it/t/ct231-student-showcase 5. students created vine videos documenting their exploration of the city (www.vinebox.co/ tag/icollab) and used storify to share their experiences (www.storify.com/4foursquare aut/auckland-via-foursquare) 6. www.mosomelt.wordpress.com/about/ references arteaga sánchez, r., cortijo, v. & javed, u. (2014) ‘students’ perceptions of facebook for academic purposes’, computers and education, vol. 70, pp. 138�149. beetham, h. & white, d. (2014) students expectations and experiences of the digital environment, jisc, bristol, uk, available at: http://www.jisc.ac.uk/publications/reports/ 2014/student-expectations-and-experiences.aspx boyd, d. (2010) ‘social network sites as networked publics: affordances, dynamics, and implications’, in self: identity, community, and culture on social network sites, ed. z. papacharissi, routledge, new york. research in learning technology citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 11 (page number not for citation purpose) http://www.bit.ly/19o0cpp http://www.bit.ly/1ms8nsl http://www.ct231.wordpress.com/2012/10/09/week-6/ http://www.scoop.it/t/ct231-student-showcase http://www.vinebox.co/tag/icollab http://www.vinebox.co/tag/icollab http://www.storify.com/4foursquareaut/auckland-via-foursquare http://www.storify.com/4foursquareaut/auckland-via-foursquare http://www.mosomelt.wordpress.com/about/ http://www.jisc.ac.uk/publications/reports/2014/student-expectations-and-experiences.aspx http://www.jisc.ac.uk/publications/reports/2014/student-expectations-and-experiences.aspx http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 castells, m. (2010) the rise of the network society, the information age: economy, society, and culture. 2nd edn, wiley-blackwell, oxford, uk. cochrane, t., et al., (2013) ‘building global learning communities’, research in learning technology, vol. 21, pp. 1�13. cochrane, t., et al., (2015) ‘designing an authentic professional development cmooc’, in globally connected, digitally enabled, proceedings the 32nd ascilite conference, eds t. reiners, et al., curtin university, perth, australia, 29 november�2 december. 53-64 fp:41-fp:52. cochrane, t. & keegan, h. (2012) ‘new global learning cultures: interdisciplinarity through networked technologies’, selected papers of internet research, pp. 1�31. available at: http://spir.aoir.org/index.php/spir/article/view/15 costa, c. (2014) ‘the habitus of digital scholars’, research in learning technology, vol. 21. available at: http://www.researchinlearningtechnology.net/index.php/rlt/article/ view/21274 cronin, c. & cochrane, t. (2014) ‘nurturing global collaboration’, edtech 2014: the 15th educational technology conference of the irish learning technology association (ilta), university college dublin, ireland, 29�30 may. czerniewicz, l. & brown, c. (2013) ‘the habitus of digital ‘‘strangers’’ in higher education’, british journal of educational technology, vol. 44, pp. 44�53. dabbagh, n. & kitsantas, a. (2012) ‘personal learning environments, social media, and selfregulated learning: a natural formula for connecting formal and informal learning’, the internet and higher education, vol. 15, no. 1, pp. 3�8. downes, s. (2007) ‘what connectivism is’, half an hour, 3 feb. available at: http://halfanhour. blogspot.com/2007/02/what-connectivism-is.html downes, s. (2010) ‘collaboration and cooperation’, half an hour, 12 april. available at: http:// halfanhour.blogspot.ie/2010/04/collaboration-and-cooperation.html floridi, l. (ed) (2014) the onlife manifesto: being human in a hyperconnected era, springer, london. available at: https://ec.europa.eu/digital-agenda/sites/digital-agenda/files/mani festo.pdf frielick, s., et al., (2014) learners and mobile devices (#npf14lmd): a framework for enhanced learning and institutional change. ako aotearoa, national centre for tertiary teaching excellence [online]. available at: https://akoaotearoa.ac.nz/learner-mobiledevices helsper, e. j. & eynon, r. (2010) ‘digital natives: where is the evidence?’, british educational research journal, vol. 36, no. 3, pp. 503�520. hodgkinson-williams, c., slay, h. & siebörger, i. (2008) ‘developing communities of practice within and outside higher education institutions’, british journal of educational technology, vol. 39, pp. 433�442. jones, c. & esnault, l. (2004) ‘the metaphor of networks in learning: communities, collaboration and practice’, in proceedings of the 4 th international conference on networked learning, eds s. banks et al., lancaster university, lancaster, pp. 317�323. junco, r., elavsky, c. m. & heiberger, g. (2013) ‘putting twitter to the test: assessing outcomes for student collaboration, engagement and success’, british journal of educational technology, vol. 44, pp. 273�287. keegan, h. (2012) ‘a new academic year: global, connected, creative � and not (quite) a mooc’, heloukee: edtech and digital culture, available at: http://heloukee.wordpress. com/2012/09/20/not-a-mooc/ keegan, h. (2014) ‘#moco360 kicks off!’, heloukee: edtech and digital culture, available at: https://heloukee.wordpress.com/tag/moco360/ keegan, h. & bell, f. (2011) ‘youtube as a repository: the creative practice of students as producers of open educational resources’, european journal of open, distance and e-learning (eurodl), special issue: creativity and open educational resources, vol. 14, no. 2, pp. 149�168. lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation, cambridge university press, cambridge. mcconnell, d., hodgson, v. & dirckinck-holmfeld, l. (2012) ‘networked learning: a brief history and new trends’, in exploring the theory, pedagogy and practice of c. cronin et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 http://spir.aoir.org/index.php/spir/article/view/15 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21274 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21274 http://halfanhour.blogspot.com/2007/02/what-connectivism-is.html http://halfanhour.blogspot.com/2007/02/what-connectivism-is.html http://halfanhour.blogspot.ie/2010/04/collaboration-and-cooperation.html http://halfanhour.blogspot.ie/2010/04/collaboration-and-cooperation.html https://ec.europa.eu/digital-agenda/sites/digital-agenda/files/manifesto.pdf https://ec.europa.eu/digital-agenda/sites/digital-agenda/files/manifesto.pdf https://akoaotearoa.ac.nz/learner-mobile-devices https://akoaotearoa.ac.nz/learner-mobile-devices http://heloukee.wordpress.com/2012/09/20/not-a-mooc/ http://heloukee.wordpress.com/2012/09/20/not-a-mooc/ https://heloukee.wordpress.com/tag/moco360/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 networked learning, eds l. dirckinck-holmfeld, v. hodgson & d. mcconnell, springer, new york, pp. 3�24. mott, j. & wiley, d. (2009) ‘open for learning: the cms and the open learning network’, in education, vol. 15, no. 2. rainie, l. & wellman, b. (2012) networked: the new social operating system, mit press, cambridge, ma. rajagopal, k., brinke, d. j., van bruggen, j. & sloep, p. b. (2012) ‘understanding personal learning networks: their structure, content and the networking skills needed to optimally use them’, first monday, vol. 17, no. 1�2. available at: http://journals.uic.edu/ojs/index. php/fm/article/view/3559/3131 reed, p. (2013) ‘hashtags and retweets: using twitter to aid community, communication and casual (informal) learning’, research in learning technology, vol. 21. available at: http:// www.researchinlearningtechnology.net/index.php/rlt/article/view/19692 rheingold, h. (2012) net smart: how to thrive online, mit press, cambridge, ma. ryberg, t., buus, l. & georgsen, m. (2012) ‘differences in understandings of networked learning theory: connectivity or collaboration?’, in exploring the theory, pedagogy and practice of networked learning, eds l. dirckinck-holmfeld, v. hodgson & d. mcconnell, springer, new york, pp. 43�58. ryberg, t. & larsen, m. (2008) ‘networked identities: understanding relationships between strong and weak ties in networked environments’, journal of computer assisted learning, vol. 24, pp. 103�115. selwyn, n. (2010) ‘looking beyond learning: notes towards the critical study of educational technology’, journal of computer assisted learning, vol. 26, no. 1, pp. 65�73. siemens, g. (2005) ‘connectivism: learning as network-creation’, elearnspace. available at: http://www.elearnspace.org/articles/networks.htm stewart, b. (2013) ‘massiveness � openness � new literacies of participation?’, merlot journal of online teaching and learning, vol. 9, no. 2. available at: http://jolt.merlot.org/ vol9no2/stewart_bonnie_0613.htm sugimoto, c., hank, c., bowman, t. & pomerantz, j. (2015) ‘friend or faculty: social networking sites, dual relationships, and context collapse in higher education’, first monday, vol. 20. available at: http://www.firstmonday.org/ojs/index.php/fm/rt/printer friendly/5387/4409 tess, p. (2013) ‘the role of social media in higher education classes (real and virtual) � a literature review’, computers in human behavior, vol. 29, no. 5, pp. 60�68. timmis, s. (2012) ‘constant companions: instant messaging conversations as sustainable supportive study structures amongst undergraduate peers’, computers and education, vol. 59, pp. 3�18. van mourik broekman, p., et al., (2015) open education: a study in disruption uk, rowman & littlefield international, london. veletsianos, g. (2015) ‘university curricula should include the teaching of networked scholarship’, george veletsianos blog. available at: http://www.veletsianos.com/2015/07/ 21/teach-neatworked-digital-scholarship-curricula/ waycott, j., sheard, j., thompson, c. & clerehan, r. (2013) ‘making students’ work visible on the social web: a blessing or a curse?’, computers and education, vol. 68, pp. 86�95. wenger, e. (1998) communities of practice: learning, meaning, and identity, cambridge university press, cambridge. wenger, e. (2006) communities of practice: a brief introduction. available at: http://wengertraynor.com wenger, e., mcdermott, r. & snyder, w. (2002) cultivating communities of practice: a guide to managing knowledge, harvard business school press, boston, ma. wenger, e., traynor, b. & de laat, m. (2011) promoting and assessing value creation in communities and networks: a conceptual framework, ruud de moor centrum, open universiteit, heerlen, netherlands. wenger, e., white, n. & smith, j. (2009) digital habitats: stewarding technology for communities, cpsquare, portland. research in learning technology citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 13 (page number not for citation purpose) http://journals.uic.edu/ojs/index.php/fm/article/view/3559/3131 http://journals.uic.edu/ojs/index.php/fm/article/view/3559/3131 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/19692 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/19692 http://www.elearnspace.org/articles/networks.htm http://jolt.merlot.org/vol9no2/stewart_bonnie_0613.htm http://jolt.merlot.org/vol9no2/stewart_bonnie_0613.htm http://www.firstmonday.org/ojs/index.php/fm/rt/printerfriendly/5387/4409 http://www.firstmonday.org/ojs/index.php/fm/rt/printerfriendly/5387/4409 http://www.veletsianos.com/2015/07/21/teach-neatworked-digital-scholarship-curricula/ http://www.veletsianos.com/2015/07/21/teach-neatworked-digital-scholarship-curricula/ http://wenger-traynor.com http://wenger-traynor.com http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 icollab bibliography buchem, i., et al., (2012) ‘mlearning 2.0: the potential and challenges of collaborative mobile learning in participatory curriculum development in higher education’, in proceedings of the iadis international conference on mobile learning 2012, eds i. a. sánchez & p. isaias, iadis international association for development of the information society, berlin, germany, pp. 311�314. cochrane, t., et al., (2013a) ‘augmenting mobile movie production’, electric dreams: 30th ascilite conference, macquarie university, sydney, australia, 1�4 december. cochrane, t., et al., (2013b) ‘building global learning communities’, research in learning technology, vol. 21, alt-c 2013 conference proceedings � building new cultures of learning, pp. 1�13. available at: http://www.researchinlearningtechnology.net/index.php/ rlt/article/view/21955 cochrane, t., et al., (2014) ‘riding the wave of byod: developing a framework for creative pedagogies’, research in learning technology, vol. 22. available at: http://www.researchinlearningtechnology.net/index.php/rlt/article/view/24637 cochrane, t. & antonczak, l. (2014) ‘implementing a mobile social media framework for designing creative pedagogies’, social sciences, vol. 3, no. 3, pp. 359�377. cochrane, t. & antonczak, l. (2015a) ‘designing creative learning environments’, interaction design and architecture(s) journal � ixd&a, no, 24, pp. 125�144. cochrane, t. & antonczak, l. (2015b) ‘developing students’ professional digital identity’, 11th international conference on mobile learning, madeira, portugal, 14�16 march. cochrane, t., antonczak, l. & guinibert, m. (2014) ‘designing transformative learning environments’, rhetoric and reality: critical perspectives on educational technology: 31st ascilite conference, otago university, dunedin, 24�26 november. cochrane, t. & keegan, h. (2012) ‘new global learning cultures: interdisciplinarity through networked technologie’, selected papers of internet research, pp. 1�31. cochrane, t., narayan, v. & burcio-martin, v. (2015) ‘designing a cmooc for lecturer professional development in the 21st century’, in handbook of research on active learning and the flipped classroom model in the digital age, ed s. keengwee, igi global, hershey, pa. cronin, c. & cochrane, t. (2014) ‘nurturing global collaboration’, edtech 2014: 15th educational technology conference of the irish learning technology association (ilta), university college dublin, ireland, 29�30 may. c. cronin et al. 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 26497 http://dx.doi.org/10.3402/rlt.v24.26497 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21955 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21955 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/24637 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/24637 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/26497 http://dx.doi.org/10.3402/rlt.v24.26497 doi:10.3402/rlt.v20i0.16152 applying a framework to evaluate assignment marking software: a case study on lightwork eva heinrich* and john milne massey university, palmerston north, new zealand (received 27 october 2010; final version received 18 august 2011) this article presents the findings of a qualitative evaluation on the effect of a specialised software tool on the efficiency and quality of assignment marking. the software, lightwork, combines with the moodle learning management system and provides support through marking rubrics and marker allocations. to enable the evaluation a framework has been developed that is based on an extensive literature review and interviews with academics in tertiary settings. the framework introduces key factors that are crucial to educationally sound and efficient assignment marking. the use of lightwork is compared to the prior experiences of participants who used either electronicor paper-based approaches. the findings are analysed using the framework. the study indicates that lightwork is well suited to support efficient, high quality assignment marking. it is suggested that the evaluation framework can be used for future studies in this area. keywords: assignments; assessment; evaluation; lightwork; case study; marking; quality; efficiency; e-learning introduction assessment via assignments is common practice in tertiary education. students create work that is assessed both summatively and formatively by teachers and markers. marking includes a wide range of tasks, ranging from administrative steps to the application of pedagogical principles. for a number of years, learning management systems (lms) have provided rudimentary support for assignment marking. this support is lacking in terms of both practical administration tasks and educational theories. in response, a new software application called lightwork (http://lightworkmarking.org) has been developed. lightwork is an open source application that complements the assignment functionality already provided by the moodle lms (http://moodle.org). it is installed on the user’s personal computer (windows, macintosh or linux), and its operations are tightly integrated with moodle’s assignment drop box and grade book. the teacher defines the assignment specifications in moodle; students submit their work to moodle and receive their marking results via moodle. lightwork takes information on courses and student lists from moodle. the organisation of marking, the marking itself and the quality control of marking are conducted in lightwork (figure 1). lightwork supports marking teams, by capturing who is marking whom (page number not for citation purpose) *correspondence author. email: e.heinrich@massey.ac.nz research in learning technology vol. 20, 2012 rlt 2012. # 2012 e. heinrich and j. milne. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 16152 doi: 10.3402/rlt.v20i0.16152 http://moodle.org http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 as the basis for progress checking and quality control. among the reasons for developing lightwork as a separate application instead as a web-based plug-in were the desire of academics to mark offline and the needs around handling and potentially annotating student work in a variety of application formats. the decision to interface lightwork with moodle was based on the popularity of moodle (well above 50,000 registered installations in june 2011 according to http:// moodle.org/stats/) and its open source license. the level of support moodle offers for assignments is comparable to that of other widely used lms. the functionality required (and provided by lightwork) to complement existing moodle functionality would conceptually be very similar for other lms. this article presents a framework for structuring the evaluation of assignment marking tools and then uses this framework to report on the evaluation of lightwork. particular attention is paid to the effects on efficiency and quality of assignment marking. this work is part of a wider research effort that looks at how academics can be supported in conducting assessment with assignments that facilitates student learning and is manageable within the constraints of the tertiary sector. a strength of the framework is that it builds both on education theories and on experiences from practice. it utilises the significant knowledge available around assessment and assignment marking to provide focus to an evaluation of marking approaches and tools. key members of the lightwork team, who are experienced academics and e-learning researchers, have conducted the lightwork case study. while great care has been taken to remain objective, this was designer-led rather than an independent evaluation. further independent studies would be valuable to explore how lightwork could be used in different contexts. evaluation framework a comprehensive review of the literature was conducted to establish the nature of the support that an e-learning solution for assignment marking needs to provide (heinrich et al. 2007). this was complemented with research involving 90 lecturers in tertiary education in new zealand (milne et al. 2007) who were asked about their practical needs for assignment marking. from there, key issues were identified that could be addressed by a software solution. these key issues (table 1) form the framework for evaluation. the following sections outline the importance of each section of the framework and the focus of the evaluation. figure 1. screenshot of lightwork showing marking sheet using a mock-up course. e. heinrich and j. milne 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 http://moodle.org/stats/ http://moodle.org/stats/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 support for marking rubrics the importance of marking rubrics is stated in literature (gronlund 2006; hanna and dettmer 2004; lambert and lines 2000; linn and miller 2005; nitko 2004). rubrics support learning and instruction by making expectations and criteria explicit (jonnson and svingby 2007). rubrics should be developed together with assignment tasks and should be made available together with the tasks. rubrics help students understand targeted learning and quality standards (reddy and andrade 2010). rubrics help markers to focus on assessing the learning outcomes targeted and prevent being side-tracked by issues like presentation. scoring with a rubric is likely to be more reliable than scoring without (jonnson and svingby 2007). the 2006 interviews confirmed that lecturers use marking rubrics. this evaluation looked at how rubrics are supported. support for providing feedback aligned to marking rubrics every student should receive feedback outlining strengths and weaknesses in their work (linn and miller 2005; nitko 2004). individualised feedback is required, regardless of type of marking rubric used (nitko 2004). marking is a time-consuming task (linn and miller 2005) that can be made more efficient with the provision of comment banks (mclachlan-smith and irons 1998). students highly appreciate feedback on their work (margrain et al. 2009). they often regard feedback they get as too vague (campton and young 2005). they request overall guidance as well as specific comments placed directly into their work (orsmond, merry, and reiling 2005). a finding from the 2006 interviews has shown that time is seen as the biggest obstacle to providing high quality feedback. it was also indicated that teachers tend to give more feedback when supported by electronic systems, as doing so is more efficient. this evaluation examined how feedback can be aligned to marking rubrics and what support is available for creating high quality feedback efficiently. support for administration of student submissions and marking results while administration issues are seldom discussed in the education literature, they were prominent in the interviews with lecturers in 2006. lecturers wanted to reduce time spent on administration in order to focus on what contributes to learning. table 1. evaluation framework for marking and management of assignments. key issues focus of evaluation marking rubrics what are the issues around devising marking rubrics? marking feedback what are quality and efficiency issues of providing marking feedback? administration of student submissions and marking results what impact does lightwork have on traditionally timeconsuming tasks (as good pedagogy cannot be implemented if practicalities are not taken care of)? marking consistency and reliability if and how does lightwork facilitate achieving consistency and reliability? reflection on marking process and outcomes does using lightwork facilitate reflection of teachers? additional aspects around electronic support what are marker experiences of using lightwork with regard to pedagogy and practicalities? research in learning technology citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 current electronic systems help but need improvements, such as the elimination of manual steps like downloading, unpacking, renaming and converting of files or the replacement of manual marker allocation and manual distribution of student work and marking results. keeping results in electronic form is useful as it simplifies record keeping and allows for analysis. yet, there is concern about the current need for double-handling between systems. this evaluation analysed the support provided for administration issues. support for achieving marking consistency and reliability students are concerned about fairness of assessment (nesbit and burton 2006). achieving reliability is challenging (linn and miller 2005; nitko 2004), and moderation is critical to successful marking (brown 2009; gronlund 2006). the use of appropriate rubrics enhances the reliability of marking (gronlund 2006). the 2006 interviews have suggested that having assignments available electronically facilitates moderation and quality control. teachers tend to feel more confident in their marking and assessment processes as electronic tools support reliability and transparency. this evaluation checked if features are provided that which support the consistency and reliability of marking. support for reflection on marking processes and outcomes the analysis of all feedback provided to students identifies strengths and weaknesses across the class and can guide further teaching (nitko 2004). teachers can and should learn from this feedback (hattie 2009). the 2006 interviews have shown that teachers who keep historic records of marked assignments see benefits for marking and moderation for subsequent classes. lecturers reflect on assignment experiences to inform future teaching. doing so is aided by electronic assignments and marking. this evaluation examined if reflection is supported. additional aspects around electronic support many additional aspects regarding electronic tool support in general, and assessment support more specifically, are addressed in the literature. electronic systems should offer a complete approach, covering the actual marking process in addition to management aspects (jones et al. 2005). electronic systems can positively influence links between assessment and teaching and learning (buzzetto-more and alade 2006) and can increase the awareness of staff regarding assessment approaches (aller et al. 2005). electronic systems can contribute to staff development by facilitating sharing of marking comments among colleagues (mckenzie 2004). staff members need strong support networks to support adoption of systems (freeman and mckenzie 2002). lecturers in the 2006 interviews emphasized that they need strong institutional support in selection of electronic systems and in learning to use these systems effectively. this evaluation assessed whether the system supports both practical issues and pedagogy. e. heinrich and j. milne 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 methodology a total of 22 semi-structured interviews were conducted with participants from four new zealand tertiary institutions. appendix a provides the schedule of questions used to guide the interviews. fifteen participants were teachers in charge of the assignments, two participants had the role of providing administrative and marking support, and five participants were employed as assistants to mark assignments under guidance. an invitation to participate in the evaluation study was sent to known lightwork users at the participating institutions. all individuals who agreed to participate were interviewed, with the exception of two individuals who were unable to participate in the timeframe available. massey university’s ethics committee approved the evaluation project, and all participants’ institutions gave permission to interview staff. the interviews were audio recorded and transcribed. the analysis of the transcripts was guided by the criteria of current context; previous methods of marking; perceptions of efficiency and marking quality. first, the task and marking contexts of the participants were considered. data were extracted that relate to the subject areas of the assignments, the class sizes, the type of student work requested for the assignment and the file formats submitted. the composition of the marking teams was examined in terms of the number of team members and their roles in the marking process. next, the marking approaches practised before using lightwork were analysed, as knowledge of these was essential for interpreting responses on the effect of using lightwork. information on the approaches was extracted and sorted into categories. shortcomings and advantages of these pre-lightwork approaches were summarised. the focus of the analysis then turned to addressing the criteria of the evaluation framework. any statements had to be set against the specific assignment circumstances of the participants and, in particular, their reference points to paper or electronic assessment. results those interviewed taught in the following subject areas: accountancy, computer science, education, healthcare, information systems, information technology, languages, law, management, physiology, psychology, sociology and sports sciences. students submitted work in the form of essays, reports, presentation material, calculations and computer programs. the files types of student work included word processing documents, pdf documents, presentation slides, spreadsheets and compressed programming project file collections. the assignments investigated belonged to courses that contribute to certificate, diploma and university preparation levels, as well as undergraduate and postgraduate degree programmes. overall, assignments for 17 courses with class sizes between 10 and 1000 students were discussed. in some cases, lightwork had been used for several assignments of the same course. the experience of using lightwork was analysed in the context of the approaches used previously by the participants. some lecturers had already used moodle for electronic assignment submission. for these lecturers, the transition focused on research in learning technology citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 learning the application lightwork. some lecturers had not yet worked with electronic assignment submission, and for these lecturers, using both the moodle assignment module and lightwork were new approaches. the following sections link the findings of the evaluation to the evaluation framework identified earlier. findings related to marking rubrics participants confirmed that lightwork provides the structures for creating marking rubrics and found that creating rubrics in an electronic tool requires more precision than just sketching out a rubric on a piece of paper. it seems that using the tool influenced the thinking of lecturers: l8: ‘‘the fact that you have to state what it is and its funny because i mean i have always done that, but probably more fudged, i think it’s made me, i think it’s made me more explicit.’’ participants stated that structuring a marking rubric requires lots of thought and is best done in conjunction with designing the assignment task, with the rubric structure following the structure of the assignment. participants spoke about the challenges they experienced when constructing their rubrics. there is tension in not wanting to restrict too much while still having enough structure for marking, especially in context of assignments on higher conceptual levels or with high levels of freedom of content or creativity. more research will be required to explore issues around marking rubric creation to be able to distinguish more clearly between the effects caused by an electronic tool as compared to the questions that arise conceptually around rubric creation. the indications from this case study are that lightwork had a positive impact. it allowed all participants to create rubrics in the way they wanted. the mechanics of the software ensured that the rubric creation was well completed before marking started. using lightwork triggered reflections on the usefulness of the rubrics used. findings related to marking feedback the rubric structure that is mirrored in the marking sheets lightwork creates for each student helped to focus marking. the ability to create comment banks was regarded as very useful for providing valuable feedback efficiently. these comment banks can contain extensive explanations that can be quickly inserted by teaching assistants. this led to more feedback than would have been given if all comments had to be typed for each student individually: l11: ‘‘ . . . i think they ended up getting more feedback and probably more useful feedback.’’ using the prepared comments reduced the number of errors contained in the feedback to students. the support provided by comment banks can be particularly helpful to marking assistants and students working and studying in a second language. completeness of feedback reassures students that markers have considered all aspects of work and consequently led to fewer enquiries after marking had been released. e. heinrich and j. milne 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 it seems clear that lightwork has had a positive impact on providing feedback to students. the software made it faster to provide feedback than was possible with the tools or approaches used previously by the participants. this impacted on feedback being more complete and comprehensive. the organisation provided by the marking rubrics was seen as valuable for the structure and transparency of the feedback given to students. findings related to handling administration of student submissions and marking results participants appreciated that lightwork has all the information required for marking in one place. this saved time locating documents and other details during marking. manual copy-and-paste steps are eliminated, which again saved time and lowered the risk of making mistakes. participants agreed that allocating student work to markers could be done easily and efficiently in lightwork. having marker allocations and the marking status for each student within easy reach made it feasible to check on marking progress, contributing to faster return times. it was perceived that more quality checking was likely to happen based on the ease with which this is possible. participants appreciated how lightwork simplifies the return of marking to students by uploading marks, marking sheets and annotated student work: t1: ‘‘ . . . before we used to have to upload each individual mark sheet for each assignment that was marked. so we used to go into [moodle] and we used to then browse to locate a mark sheet that we had input information on and saved in that student’s name and then uploaded that and then saved that and then adjusted the mark as well. so that was quite time consuming and lightwork completely removed the need to do all of that, so that was huge.’’ the evaluation clearly showed that lightwork helps with the administration issues around assignment marking. participants confirmed that these administration issues are very important. time saved can be invested into more productive aspects of marking. ease of access to information enables quality control steps to be carried out and automation reduces the chance of human error. findings related to marking consistency and reliability there was strong support among participants that marking with lightwork can improve reliability of marking and consistency between multiple markers. the features that support this were the marking rubrics, the comment banks and the instructions to markers that can be integrated into the marking rubrics. lightwork provides the lecturers in charge with easy access to marking that has been completed by the teaching assistants. lecturers appreciated this feature as it enabled quality checking. this facilitated timely feedback to the teaching assistants, reassuring them if they are on the right track or pointing them in the right direction if required: l8: ‘‘ . . . i can pop in and see easily, locate the mark sheet, which i did on a regular basis . . . .’’ participants stated that lightwork also had an effect on consistency in instances where there was just a single lecturer marking: research in learning technology citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 l13: ‘‘ . . . it gives you immediately a good view of how you have marked other students.’’ in lightwork, the lecturer in charge of the assignment releases all marking to students. a teaching assistant with lower access rights cannot perform this task. this allows the lecturer to conduct quality checking and determine the point of time all students get access to their marking results. this feature was appreciated. in summary, participants gave a clear indication that lightwork facilitates checking for consistency and reliability. the tools that lightwork provides at this stage are fairly simple and are basically limited to providing structured, timely and flexible access to information concerned with the marking process and marking results. many extensions to lightwork are possible that would further improve support for this important area of assignment marking. findings related to marking reflection nearly all lecturers interviewed had many years of teaching and marking practice. despite this level of experience lecturers stated that using lightwork encouraged them to review their practice: l13: ‘‘ . . . when i started using lightwork i all of a sudden became aware of, oh well okay i’ve been doing this for some time now but maybe if i structured these things accordingly then that might help me a lot, . . . that actually worked.’’ lecturers reflected how they moved forward from their experience of marking the first assignment with lightwork to the second of the same course, how in future they would be more specific in formulating the work requested from students and how it would be better to have requirements and feedback more closely aligned. using lightwork had the effect of encouraging thinking beyond the current assignment to the bigger picture of assessment and course design. lightwork seemed to have had a strong effect on encouraging lecturers to reflect on their assignment practices. this is positive, as reflection is an important step in the improvement processes. findings related to additional aspects around electronic support lightwork satisfied the requirements of the participant for assignment marking. while additional features, for example, for audio in addition to typed comments, would be welcomed, all core steps of the assignment marking process are covered. as lightwork eliminates most of the manual handling steps related to marking, it lowers the risk of making mistakes. lightwork encourages much more detailed recording of marking than many lecturers would do otherwise. this was seen as an advantage when there are enquiries about the marking: l11: ‘‘ . . . last year we got questions around would you please clarify why this happened and this year i think they seemed to feel much more comfortable with why we gave them the mark we did.’’ participants acknowledged that lightwork can be a catalyst for reviewing and changing assessment practices. participants also rightly expressed that software tools by themselves cannot improve marking but that success will always depend on the person using the tools: e. heinrich and j. milne 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 l13: ‘‘so you know from that perspective looking at it overall i’d say that it probably is not a blanket statement that lightwork will make feedback better, it totally depends on who is doing the feedback.’’ lightwork had just been developed when the participants agreed to trail it. some of the participants moved directly from paper-based assignment submission to use of moodle and lightwork. several participants commented on the importance of having good support available to them. discussion the participants in this study have shown a solid understanding of what constitutes good marking and how important marking, and assessment more generally, is to student learning. the exploration of using lightwork inadvertently led to comparisons with other tools and approaches used previously by the participants. this confirmed findings from the literature and the previous 2006 study that neither manual, paper-based approaches nor combinations of current lms with generic tools provide sufficient support to academics in tertiary settings. as a consequence, carrying out all steps required for providing valuable formative feedback and for monitoring the quality of marking is very time consuming and not supported conceptually. this leads to academics either spending large amounts of time on the manual execution of steps or to academics not performing these steps to a level suggested by the education literature. a lms like moodle is good at handling some aspects, such as student assignment submission and provision of access to marking. the case study shows that a tool like lightwork, which is designed based on assessment theories, can fill the gap that otherwise exists with regard to marking support for academics and their marking teams. of specific importance is the impact on formative feedback, with case study participants confirming that lightwork helped providing more complete feedback in better-structured form. as well-designed software should, lightwork supports users in carrying out educationally sound practices. lightwork supports the important steps recommended in the literature for assignment marking. as such, the tool fits in with established good practice, instead of forcing the user to adapt their practice to the tool: l2: ‘‘it is a lot of the practices we do already; it is just consolidated so it talks back to that efficiency and to maintaining consistent quality between different markers.’’ in addition, using lightwork has had the welcome effect of encouraging reflection. table 2 summarises the key findings linked to the criteria of the evaluation framework. conclusions the case study indicates that lightwork has a positive effect on the efficiency and quality of assignment marking. the participants, who showed significant knowledge of the marking practices recommended in the literature, stated that lightwork provides efficient support for good marking approaches. the evaluation framework outlined in this article has not only justified the findings but can also be used to research in learning technology citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 table 2. key findings. key issues focus of evaluation key findings marking rubrics what are the issues around devising marking rubrics? lightwork allowed the construction of marking rubrics. constructing rubrics in lightwork helped identify conceptual challenges in designing suitable rubrics that could be discussed and addressed. marking feedback what are quality and efficiency issues of providing marking feedback? compared to tools used previously by participants, lightwork: . makes providing feedback more efficient; . is likely to result in more complete feedback; and . encourages markers to give positive feedback in addition to constructive feedback. quality and efficiency of providing feedback are tightly linked and cannot be looked at in isolation. administration of student submissions and marking results what impact does lightwork have on traditionally time-consuming tasks (as good pedagogy cannot be implemented if practicalities are not taken care of)? lightwork successfully automates mundane and repetitive tasks.this saves time and reduces human error. marking consistency and reliability if and how does lightwork facilitate achieving consistency and reliability? lightwork successfully addresses consistency and reliability issues by: . linking the marking rubrics with the student feedback sheets and . making marking data accessible in a structured and efficient way, and so lowering the barriers towards carrying out quality assurance steps. reflection on marking process and outcomes does using lightwork facilitate reflection of teachers? the use of lightwork triggers reflection on the marking process, as well as marking and assignment task specification.the reason seems to be that using software (instead of paper-based approaches) brings more structure and enforces consistent application. e . h e in rich a n d j. m iln e 1 0 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) c ita tio n : r e s e a rc h in l e a rn in g te c h n o lo g y 2 0 1 2 ; 2 0 : 1 6 1 5 2 d o i: 1 0 .3 4 0 2 /rlt.v 2 0 i0 .1 6 1 5 2 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 table 2 (continued) key issues focus of evaluation key findings additional aspects around electronic support what are marker experiences of using lightwork with regard to pedagogy and practicalities? lightwork successfully addresses two important areas: . it incorporates the pedagogically important concepts (marking rubrics, feedback sheets, comment banks and marker allocations). . it addresses practicalities and saves time that can be invested better into pedagogically valuable tasks. as with any software tool, the capacity of lightwork to improve assignment marking rests on the abilities of its users. r e se a rch in l e a rn in g t e ch n o lo g y c ita tio n : r e s e a rc h in l e a rn in g te c h n o lo g y 2 0 1 2 ; 2 0 : 1 6 1 5 2 d o i: 1 0 .3 4 0 2 /rlt.v 2 0 i0 .1 6 1 5 2 1 1 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 evaluate other electronic tools for assignment assessment, the results of which could then be compared with the lightwork results. the case study collected qualitative data that captured the richness of participants’ experiences. the experiences of the case study participants varied greatly, depending on their personal frameworks for dealing with assignments and the previous experiences with which they compare lightwork. these qualitative data shed insight on how the participants’ used lightwork and the impact it had on their practice. future studies could complement these data with quantitative measures. this was not the purpose of this study. participants used an early version of lightwork in the case study. lightwork continues to evolve, and comments by participants alerted to general challenges in the design of the software. an application needs to facilitate good educational practice without restricting the user: it should cater for a variety of circumstances without becoming too complex by providing too many options. the participants in this case study were early adopters who were highly knowledgeable in assignment marking practices and highly motivated. institutionwide implementation of an application like lightwork could reveal different issues where academic participants had diversity in motivation and in their knowledge of assessment practices. excellent support structures on technical and pedagogical levels will be required to enable the software to be used in support of effective assessment practices. in conclusion, it is important to acknowledge that this research did not test the validity of the marking rubrics developed by case study participants. the focus of the study was on the application of these rubrics in reliable and efficient ways. further research, in conjunction with lightwork or in more general terms, should be conducted to examine the validity of marking rubrics. references aller, b. et al. (2005) ‘webal: a web-based assessment library to enhance teaching and learning in engineering’, ieee transactions on education, vol. 48, no. 4, pp. 764�771. brown, g. (2009) ‘the reliability of essay scores: the necessity of rubrics and moderation’, in tertiary assessment & higher education student outcomes: policy, practice & research, eds l. meyer et al., ako aotearoa, wellington, new zealand, pp 43�50. buzzetto-more, n. a. & alade, a. j. (2006) ‘best practices in e-assessment’’, journal of information technology education, vol. 5, pp. 251�269. campton, p. & young, j. (2005) ‘please sir, may i have some more? a comparative study on student satisfaction with assessment feedback methods in an undergraduate unit’, paper presented at the balance, fidelity, mobility: maintaining the momentum? the 22nd annual conference of the australasian society for computers in learning in tertiary education (ascilite), brisbane, australia. freeman, m. & mckenzie, j. (2002) ‘spark, a confidential web-based template for self and peer assessment of student teamwork: benefits of evaluating across different subjects’, british journal of educational technology, vol. 33, no. 5, pp. 551�569. gronlund, n. e. (2006) assessment of student achievement, pearson, boston. hanna, g. s. & dettmer, p. a. (2004) assessment for effective teaching using contextadaptive planning, pearson, new york. hattie, j. (2009) ‘the black box of tertiary assessment: an impending revolution’, in tertiary assessment & higher education student outcomes: policy, practice & research, eds l. meyer et al., ako aotearoa, wellington, new zealand, pp. 259�275. heinrich, e. et al. (2007) ‘literature review on the use of e-learning tools for formative essaytype assessment’, [online] available at: http://etools.massey.ac.nz/research.htm, retrieved on 6 september 2010. e. heinrich and j. milne 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 http://etools.massey.ac.nz/research.htm http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 jones, d. et al. (2005, 9�11 november) ‘what makes ict implementation successful: a case study of online assignment submission’, paper presented at the open learning and distance learning association of australasia (odlaa), university of south australia. jonsson, a. & svingby, g. (2007) ‘the use of scoring rubrics: reliability, validity and educational consequences’, educational research review, vol. 2, pp. 130�144. lambert, d. & lines, d. (2000) understanding assessment: purposes, perceptions, practice, tj international, padstow. linn, r. l. & miller, m. d. (2005) measurement and assessment in teaching, pearson merrill prentice hall, columbus. margrain, v., et al. (2009) ‘‘‘ka pai � well done’’: student teacher perceptions of assessment feedback in distance learning’, in tertiary assessment & higher education student outcomes: policy, practice & research, ed. l. meyer, et al., ako aotearoa, wellington, new zealand, pp 129�139. mckenzie, s. (2004) ‘assessing quality of feedback in online marking databases: an opportunity for academic professional development or just big brother?’ beyond the comfort zone: proceedings of the 21st ascilite conference, perth. mclachlan-smith, c. & irons, b. (1998) ideas to share: examples of succesful extramural study guide design, massey university, palmerston north. milne, j., et al. (2007) ‘survey report on the use of e-learning tools for formative essay-type assessment’, [online] available at: http://etools.massey.ac.nz/research.htm, retrieved on 6 september 2010. nesbit, p. & burton, s. (2006) ‘student justice perceptions following assignment feedback’, assessment & evaluation in higher education, vol. 31, no. 6, pp. 655�670. nitko, a. j. (2004) educational assessment of students, 4th edn, pearson education upper saddle river, nj. orsmond, p., merry, s. & reiling, k. (2005) ‘biology student’s utilization of tutor’s formative feedback: a qualitative interview study’, assessment & evaluation in higher education, vol. 30, no. 4, pp. 369�386. reddy, y. & andrade, h. (2010) ‘a review of rubric use in higher education’, assessment & evaluation in higher education, vol. 35, no. 4, 435�448, first published on: 07 august 2009 (ifirst). to link to this article: doi: 10.1080/02602930902862859. appendix a: interview questions for lightwork users asignment context what was the subject area of the assignment? how many students were in the course? what was the level of the course? what type of work did the students submit (essay, report, calculation, . . .)? how many files of what file types did the students submit? did students submit in groups? if yes, how many submitted together? if yes, did all team members receive the same mark and feedback? marking context did you mark by yourself ? did you have a marking team? if yes, how was this team composed? (markers, admin staff, . . . ) did you/your markers annotate student work directly? if yes, in which program did you do this? (word track changes, pdf annotation, . . . ) efficiency of marking do you find that using lightwork has helped with the efficiency of the assignment marking? if yes, in which ways? research in learning technology citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 13 (page number not for citation purpose) http://etools.massey.ac.nz/research.htm http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 quality of marking do you find that using lightwork has helped with the quality of the assignment marking? if yes, in which ways? feedback from marking team members (if applicable) have you had any feedback from your marking team members on marking with lightwork? if yes, what kind of feedback? feedback from students have you had any feedback from your students in regard to the assignment marking? if yes, what kind of feedback? knowledge about assessment has working with lightwork had an effect on your knowledge about assessment? if yes, what kind? closing would you provide us with a copy of your marking rubric? could i have a copy of your comments to markers and your frequently used comments? would you allow me to publish your marking rubric in a document/website to help others with the development of marking rubrics? why did you structure or write your marking rubric in this way? what worked well about it? what did not work so well? what would you change for next time? would you provide us with some examples of marking sheets and annotated student work? do you have any comments? e. heinrich and j. milne 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16152 doi: 10.3402/rlt.v20i0.16152 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16152 (page number not for citation purpose) 1 *corresponding author. email: maria.espada@upm.es research in learning technology 2020. © 2020 m. espada et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 research in learning technology vol. 28, 2020 original research article development of the learning to learn competence in the university context: flipped classroom or traditional method? maría espada*, josé antonio navia, patricia rocu and maite gómez-lópez faculty of physical activity and sport sciences, polytechnic university of madrid, madrid, spain (received: 29 march 2019;revised: 21 november 2019; accepted: 3 december 2019; published: 31 january 2020) this study analyses the use of a flipped classroom to develop the ‘learning to learn’ competence in the university context. this research was conducted on a subject about applied teaching methodology included in the physical activity and sports science degree at the polytechnic university of madrid (spain). a total of 110 university students (mean age 21.6 ± 3.0 years) participated in the research and were divided into two groups: one group (44 students) received an intervention based on the traditional method (with theoretical classes and resolved questions) and the other group (66 students) received an intervention using the flipped classroom method; self-perception of the level of development of the ‘learning to learn’ competence was analysed before and after the intervention. the design involved two groups that followed different types of teaching (traditional vs. flipped classroom) × two moments in time (before and after). this study did not find any significant differences between the traditional and flipped classroom method, in the perception of the development of the ‘learning to learn’ competence. keywords: flipped classroom; ‘learning to learn’ competence; self-management of learning; self-evaluation of learning; self-knowledge as an apprenticeship introduction education based on competences over the last three decades, the development of competence systems, accompanied by the escalation in information and communication technologies, and the globalisation process, has had multiple repercussions in the world of education (climént 2010). in the early 2000s, more and more european countries made efforts to shift from solely knowledge to a broader competence approach in national curricula. the adoption of the 2006 recommendation on key competences for lifelong learning and its annexed european reference framework of key competences for lifelong learning was therefore very timely to support this process and provide a common reference document (european commission 2018). mailto:maria.espada@upm.es http://www.alt.ac.uk/� http://creativecommons.org/licenses/by/4.0/� http://dx.doi.org/10.25304/rlt.v28.2251� m. espada et al. 2 citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 (page number not for citation purpose) this tendency for education based on competences has led to the establishment of education laws in the different european countries to foment its development. thus, in spain, with regard to the pre-university level, the organic education law and royal decree 15/13/2006 of 7th december stipulates the need to develop competences, defining this term as ‘the capacity to use knowledge and abilities, in a transversal and interactive manner, in contexts and situations which require the intervention of knowledge linked to different learning, which implies understanding, reflection and discernment bearing in mind the social dimension of each situation’. (p. 43058). similarly, the present organic law 8/2013 of 9th december on the improvement of educational quality insists on the importance of developing basic competences determining that ‘the simplification of curricular development is an essential element for the transformation of the educational system, a simplification which, according to the european union guidelines, should provide solid knowledge of the content that guarantees the effective acquisition of the basic competences’. (p. 8) the organisation for economic cooperation and development (oecd), definition and selection of competences (deseco) project had quite a significant influence on the work on competence frameworks in the european union, both in numerous member states and in the debates and collaboration at the european level (rychen and salganik 2001). it defined the aim of key competencies as contributing to a successful life for individuals and a well-functioning society and the project classified key competences into three broad categories (using tools interactively, interacting in heterogeneous groups and acting autonomously). the key competences that have been established include that of ‘ learning to learn’. this competence implies possessing the abilities to determine one’s own learning and be able to learn in an increasingly efficient and autonomous way depending on one’s own objectives and needs (carretero and fuentes 2010). hofmann (2008) refers to ‘learning to learn’ as a competence that allows people to become more effective, flexible and self-organised learners in a variety of contexts. an overview of competence definitions in the international competence frameworks shows that the ‘learning to learn’ competence means ‘ability to use knowledge and information interactively’, ‘analytical and critical thinking’, ‘autonomous learning skills’ and ‘problem solving’ (european commission 2018). currently, most students still come from environments in our educational system that are characterised by teaching using a vertical and hierarchical discourse which demands that the students reproduce its academic knowledge. this concept of learning and teaching is incongruent with both current social demands and the intended aims of the new educational paradigm (salmerón and gutierrez-braojos 2012). with regard to the university context, the educational activity proposed by the european higher education area (ehea) makes the student the protagonist, emphasising active, autonomous, critical and reflexive learning (herrero 2014). the implementation of these competences implies an enormous challenge for teachers as they are expected to find strategies to integrate these competences into their different curricular subjects (trier 2003; voogt and roblin 2012). all of which has led http://dx.doi.org/10.25304/rlt.v28.2251� research in learning technology citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 3 (page number not for citation purpose) to the appearance of new methodologies, resources and material, new areas and time periods from those traditionally used until now in the classroom (herrero 2014). in designing active methodologies to favour training in competences, the challenge is to extend the methodological repertory by attempting to thoroughly understand the possibilities offered by the different strategies and to proceed by experimenting with their application in teaching practice, thus adopting and adapting these methods to circumstances and possibilities (fernández-march 2006). likewise, learning methods based on teaching using technological means have been used with high expectations and the idea that they would increase students’ motivation and facilitate learning the content presented (lee, lim, and grabowski 2010). flipped classroom the flipped classroom method is one of these new methods and consists of inverting the order of the traditional class in such a way that the students can see the content of a topic in the virtual classroom and then carry out activities and practical exercises later on in class to consolidate this content (bergmann and sams 2014). bergmann and sams (2014) define this new concept by explaining that what traditionally was done in class is now done at home and what traditionally was done as homework is now done during the classes. chis et al. (2018) explained that the flipped classroom is a student-centred pedagogy in which students complete pre-class work (e.g. watching a video clip, searching online for certain information) to gain basic knowledge, and class time is dedicated to activities that promote the application and mastery of this knowledge. in this respect, this technology helps to improve student engagement in terms of the time spent on a task, quality of effort and student involvement (shah and barkas 2018). in addition, the flipped classroom could improve students’ motivation and help them to manage the cognitive load (abeysekera and dawson 2015). however, despite the fact that a priori this may seem a simple model, it is important to devise a suitable design and correctly implement the method for it to be effective (khanova et al. 2015). this new method has aroused great interest in the educational community as it combines the use of technological material on the one hand and the development of autonomy on the other. in particular, in the university context, advancing digital technologies within the higher education sector are challenging both the pedagogical stance of traditional didactic teaching seen for decades within universities and at the same time offering dynamic and innovative opportunities for student learning (o’flaherty and phillips 2015). in the traditional didactic approach where academics transmit knowledge as expert teachers to students, the latter have been treated as empty vessels passively absorbing information, wherein their interests are diminished and diverse learning styles are disregarded (betihavas et al. 2016; lage, platt, and treglia 2000). in this way, the strategy of flipped learning requires the autonomy of the students because they have to watch video lectures outside the classroom. research conducted on university students showed better management of their time with the flipped classroom method (kim 2018). in this respect, some research revealed several barriers to the  full adoption of the flipped model, such as the students’ passive learning habits, and the busy schedules of part-time students who were not always able to properly review the study materials in advance of the synchronous http://dx.doi.org/10.25304/rlt.v28.2251� m. espada et al. 4 citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 (page number not for citation purpose) class (chen et al. 2014; zainuddin and attaran 2016). this model then puts more responsibility for learning on the students so that they can work towards mastery of the material (o'flaherty and phillips 2015). in this sense, flipped learning enables an individual to develop the competence of ‘learning to learn’. however, despite the strong willingness of academics to engage in the redesign of their students’ learning experiences using the flipped classroom (o'flaherty and phillips 2015) and the popularity of this new method, there are currently not enough scientific investigations that permit the suitability of the flipped classroom to be confirmed as a method for all areas and educational levels (abeysekera and dawson 2015; kashada, li, and su 2017). thus, it is necessary to carry out research that makes it possible to analyse the development of competences using active methodologies. the aim of this study was to estimate the relationship between using the flipped classroom methodology and the traditional method and the perception of the development of the ‘learning to learn’ competence in higher education. the hypothesis is that ‘students who learned through the flipped classroom methodology increase their perception of the development of the “learning to learn” competence’. method participants the study involved a compulsory third-year subject on applied teaching methodology from the degree in physical activity and sport sciences, during six classroom sessions (one unit) of the first semester of the 2017–2018 course. a total sample of 169 students filled out the questionnaire; however, 59 of them filled out only one of the parts (either before or after the intervention) and therefore they were excluded from the analysis. of the 110 remaining students, 66 (16 women and 50 men) received the flipped classroom methodology and 44 (9 women and 35 men) received the traditional methodology. the mean age of the students was 21.6 years [standard deviation (sd) = 3.0]. design and procedure in many situations in educational research, it is not possible to randomly assign subjects to a treatment group (ary et al. 2010). in the present study, it was not possible to reorganise the classes to accommodate the research study. for this reason, the study had a quasi-experimental design in which one group received a traditional pedagogical intervention (control group) and the other an intervention using a flipped classroom approach (experimental group). there were two teachers. each teacher had two groups (one control and one experimental group). the students were distributed into two groups, one of which received the content using a traditional classroom method, in which they were given theoretical classes and resolved questions. the other group received the content using the flipped learning strategy so that the students had to watch video lectures outside the classroom that had been uploaded to the subject’s web space and read powerpoint material. then they received practical classes with activities in which they worked on the content using problem solving, quizzes and a discussion group. the students had never used the flipped classroom method before this research; therefore, http://dx.doi.org/10.25304/rlt.v28.2251� research in learning technology citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 5 (page number not for citation purpose) the experimental group received guidance on how to be a flipped classroom-student before the intervention through video lectures containing theoretical content on flipped classroom methodology. each of the groups was measured using the instrument (see below) before and after the intervention. therefore, the design is of two independent groups (the two traditional groups were merged and the two flipped classroom groups were also merged for the statistical analysis) that followed the different types of teaching × two moments in time (before and after). instrument the scientific community already knows and assumes that the adaptation of tests is not merely a linguistic matter, and that it requires the conjunction of cultural, conceptual, linguistic and metrical aspects that have to be analysed from both analytical– rational and empirical perspectives (muñiz, elosua, and hambleton 2013). for this reason, to comply with the aims of the present research and to measure the self-perception of the level of development of the ‘learning to learn’ competence in the context of the spanish university, an instrument was used that had been developed and validated for this population: the questionnaire ‘self-perceived level of development of the “learning to learn” competence in the university context: a proposed measuring instrument’ drawn up by muñoz-san roque et al. (2016). the instrument has been applied to a sample of 458 college students. it has three factors (learning management, self-assessment and the self-learning process as a learner) and nine items: items 1–4 for learning management; items 5 and 6 for self-assessment; and items 7–9 for the self-learning process as a learner. the answers referred to a likert-type scale from 1 to 6, where 1 was ‘little developed’ and 6 was ‘highly developed’. cronbach’s alpha reliability coefficient was 0.88. exploratory and confirmatory analyses were conducted to validate the theoretical model and reliability indices, and the suitability of the structural model was determined to justify the validity of the instrument (muñoz san roque et al. 2016). after the participants gave their consent to participate in the study, we proceeded to administer the questionnaire. data analysis the mann–whitney u test was used to analyse the scores on the questionnaire (ordinal variables: likert scale from 1 to 6). cohen’s d was used to express effect size, with values of 0.2, 0.5 and 0.8 to express small, medium and large effects, respectively. the values of the three resulting factors (see muñoz-san roque et al. 2016) were submitted to a 2 (within groups, time: before and after) by 2 (between groups, pedagogical intervention: traditional and flipped classroom) repeated measure analysis of variance (rm anova). shapiro–wilk’s and levene’s tests showed normal distribution and homogeneity of variances, respectively. partial eta-squared depicted the effect size, with values of 0.01, 0.06 and 0.14 for small, medium and large effects, respectively (cohen 1977). significance level for all analyses was set at alpha = 0.05. spss v. 24 and jamovi v.0.9.5.16 programs were used to perform the statistical analysis. http://dx.doi.org/10.25304/rlt.v28.2251� m. espada et al. 6 citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 (page number not for citation purpose) results the values of the three factors on the questionnaire were similar across the types of pedagogical interventions (figure 1). accordingly, rm anova confirmed no significant effect for pedagogical intervention on learning management [f(1,108)  =  2.98, p  =  0.087, η p 2= 0.03], self-evaluation of the process [f(1,108) = 2.13, p = 0.147, η p 2= 0.02] or self-knowledge as a learner [f(1,108) = 0.34, p = 0.564, η p 2= 0.00]. as shown in figure 1, pupils reported higher values of learning management after the intervention, an rm anova revealed the main effect of time [f(1,108) = 6.66, p = 0.011, η p 2= 0.06]. yet, these effects of time were not found for self-evaluation [f(1,108) = 1.27, p = 0.263, η p 2= 0.01], or self-knowledge measures [f(1,108) = 1.02, p = 0.314, η p 2= 0.01]. finally, the rm anova did not reveal an interaction between the type of pedagogical intervention and time for any of the factors on the questionnaire (figure 1). table 1 presents the scores of the participants in the different items on the questionnaire. before the intervention, differences between traditional and flipped classroom were found in ‘i am able to have a certain control on how i learn’ (u = 1081, p = 0.016, d = 0.44) and ‘i organise my studying setting realistic objectives’ (u = 1125, p = 0.037, d = 0.44). after the classes, differences remained only in ‘i organise my studying setting realistic objectives’, with a smaller effect size (u = 1138, p = 0.047, d = 0.40). figure 1. values of the questionnaire factors by type of pedagogical intervention, before and after. error bars represent standard error of the mean. http://dx.doi.org/10.25304/rlt.v28.2251� research in learning technology citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 7 (page number not for citation purpose) to examine the actual effect of the pedagogical intervention on the participants’ scores, we calculated the difference between the before and after scores for each item for every participant, and then compared those scores by pedagogical intervention (traditional vs. flipped). the mann–whitney u test did not reveal a significant difference across groups for any of the items, with p-values ranging from 0.137 for ‘i am able to identify my study habits and can suitably describe them’ to 0.913 for ‘i am able to use different strategies’. discussion we aimed to compare traditional versus flipped classroom methodology in the perception of the development of the ‘learning to learn’ competence in higher education. to this end, we measured the management of the learning process, self-evaluation of the process and self-knowledge as a learner (muñoz-san roque et al. 2016) before and after the pedagogical intervention. the development of key competences needs to start at an early age and to continue throughout life. measures to improve competence development need to reach all learners, independently of their background and individual capacities (european commission 2018). table 1. mean (m) and standard deviation (sd) of the scores relative to perceived development of the ‘learning to learn’ competence by type of pedagogical intervention (flipped vs. traditional) before and after the intervention. items before after flipped traditional flipped traditional m sd m sd m sd m sd 1. i am able to have a certain control on how i learn 4.33 0.92 * 4.70 0.73 4.52 1.01 4.77 0.72 2. i organise my studying setting realistic objectives 4.03 1.10 * 4.48 0.88 4.21 1.16 * 4.64 0.92 3. i establish times for study 4.30 1.25 4.23 1.12 4.44 1.23 4.30 0.93 4. i am able to use different strategies 3.91 1.20 4.19 1.20 4.09 1.18 4.40 1.14 5. i can check during the task if it is responding to the objectives i have set 4.15 1.13 4.41 0.97 4.26 1.04 4.52 1.09 6. i can check if the results correspond to the objectives i set at the beginning of the task 4.33 0.90 4.48 1.07 4.38 1.08 4.64 1.04 7. i am aware of my level of knowledge and my particular method of processing information 4.80 0.92 5.00 0.78 4.82 0.90 5.05 0.71 8. i am able to identify my study habits and can suitably describe them 4.83 0.95 4.64 0.92 4.76 1.16 4.89 0.78 9. i am aware of my virtues and difficulties when i am studying or learning 5.05 0.83 5.02 0.73 5.03 1.05 5.16 0.75 note: *p < 0.05 http://dx.doi.org/10.25304/rlt.v28.2251� m. espada et al. 8 citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 (page number not for citation purpose) different authors and institutions are recommended to incorporate self-assessment of students in this process to know their opinions about the development of the competences (salcines et al. 2018). students did report higher values of learning management after the intervention, but regardless of the methodology. in fact, results indicated that none of those factors showed a clear variation as a function of pedagogical intervention. likewise, while the traditional group reported higher scores in some of the items (i.e. ‘i am able to have a certain control on how i learn’ and ‘i organise my studying setting realistic objectives’) either before and/or after, it did not show systematic variations in learning rate across time for any of the items composing the questionnaire. in other words, the values of items and factors suggest that both methods (flipped classroom vs. traditional) might produce similar results in acquiring ‘learning to learn’ competence. the present findings are in line with the previous research on flipped class interventions among university students. for instance, the study by morgan et al. (2015) found no significant differences in the self-evaluation of knowledge between the traditional method and flipped classroom. kim (2018) did not find differences of a flipped learning course with regard to student satisfaction. in another recent study employing epidemiology master students, performance between flipped and traditional methods was similar with regard to quantitative assessment of the course (shiau et al. 2018). some of the authors have provided some explanation of the absence of differences among methodologies. despite some of the advantages of adopting the flipped classroom methodology such as students’ general positive feelings and greater satisfaction and engagement, this methodology is not exempt from some problems, such as passive learning habits, heavier workloads and lack of time to review the videos at home, particularly among part-time students (chen et al. 2014; kim 2018; zainuddin and attaran 2016). results of our experiment and aforementioned studies can be understood in the framework of non-significant difference (russell 1999). this well-documented phenomenon raises doubts about the (usually overestimated) benefits of the alternative methods of education based on the use of new technology in conjunction with blended or online instruction over the traditional classroom. research suggests that score grades and student satisfaction are similar between traditional and alternative methods, and that they are mediated by other exogenous factors such as type of technology, available time of students, course design and so on (russell 1999). in this sense, we highlight the need for the publication of educational works finding nonsignificant differences between instructional methods in order to avoid the publication bias. otherwise, published works in favour of either traditional or alternative technology-based classrooms would lead to misguided conclusions. limitations this study is not exempt from shortcomings. first, while we measured the perceptions of the students in acquiring the competence of ‘learning to learn’, we failed to record the actual scores of the participants during the intervention. collecting both perceptions and scores would provide a better picture of the teaching–learning process. the study was also limited by the short time available for the intervention because it was only performed during six classroom sessions. http://dx.doi.org/10.25304/rlt.v28.2251� research in learning technology citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 9 (page number not for citation purpose) for further research on this subject, future studies should be conducted during one complete year and should increase the sample size (involving students from the first to the last years). this might be enough time to see a difference between students using flipped classroom and traditional lectures. it would also make it possible to analyse the difference between freshmen and experienced students as more experienced students probably have developed study habits that might be difficult to change later, especially if these habits were developed by having only traditional lectures. conclusion we can conclude that this study did not find any significant differences between the traditional and flipped classroom methods, in the perception of the development of the ‘learning to learn’ competence. however, given the controversy that exists between the results of the present study and the results of different investigations, it is necessary to carry out more research using the flipped classroom method in order to be able to present scientific evidence to corroborate the influence of the flipped model in the university context. references abeysekera, l. & dawson, p. (2015) ‘motivation and cognitive load in the flipped classroom: definition, rationale and a call for research’, higher education research & development, vol. 34, no. 1, pp. 1–14. doi: 10.1080/07294360.2014.934336 ary, c., et al., (2010) introduction to research in education, 8th edn., wadsworth, belmont, ca. bergmann, j. & sams, a. (2014) dale la vuelta a tu clase: lleva tu clase a cada estudiante, en cualquier momento y cualquier lugar (flip your class: take your class to each student, at any time and in any place), fundación santa maría-ediciones sm, madrid. betihavas, v., et al., (2016) ‘the evidence for “flipping out”: a systematic review of the flipped classroom in nursing education’, nurse education today, vol. 36, pp. 15–21. doi: 10.1016/j. nedt.2015.12.010 carretero, m. r. & fuentes, m. (2010) ‘la competencia de aprender a aprender (the “learning to learn” competence)’, aula de innovación educativa, vol. 192, pp. 7–10. chen, y., et al., (2014) ‘is flip enough? or should we use the flipped model instead?’, computers & education, vol. 79, pp. 16–27. doi: 10.1016/j.compedu.2014.07.004 chis, a. e., et al., (2018) ‘investigating flipped classroom and problem-based learning in a programming module for computing conversion course’, educational technology & society, vol. 21, no. 4, pp. 232–247. climént, j. b. (2010) ‘reflexiones sobre la educación basada en competencias (reflections on competence-based education)’, revista complutense de educación, vol. 21, no. 1, pp. 91–106. cohen, j. (1977) statistical power analysis for the behavioral sciences, academic press, new york. european commission (2018) council recommendation on key competences for lifelong learning, official journal of the european union, brussels. fernández-march, a. (2006) ‘metodologías activas para la formación de competencias (active methodologies for the training of competences)’, educatio siglo xxi, vol. 24, pp. 35–56. herrero, r. m. (2014) ‘el papel de las tic en el aula universitaria para la formación en competencias del alumnado (the role of ict in the university classroom for training the students in competences)’, pixel-bit. revista de medios y educación, vol. 45, pp. 173–188. doi: 10.12795/pixelbit.2014.i45.12 http://dx.doi.org/10.25304/rlt.v28.2251� http://dx.doi.org/10.1080/07294360.2014.934336� http://dx.doi.org/10.1016/j.nedt.2015.12.010� http://dx.doi.org/10.1016/j.nedt.2015.12.010� http://dx.doi.org/10.1016/j.compedu.2014.07.004 http://dx.doi.org/10.12795/pixelbit.2014.i45.12 m. espada et al. 10 citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 (page number not for citation purpose) hofmann, p. (2008) ‘learning to learn: a key-competence for all adults?’, convergence, vol. 41, no. 2–3, pp. 173–181. kashada, a., li, h. & su, c. (2017) ‘adoption of flipped classrooms in k-12 education in  developing countries: challenges and obstacles’, international journal of emerging technologies in learning, vol. 12, no. 10, pp. 147–157. doi: 10.3991/ijet.v12i10.7308 khanova, j., et al., (2015) ‘student experiences across multiple flipped courses in a single curriculum’, medical education, vol. 49, no. 10, pp. 1038–1048. doi: 10.1111/medu.12807 kim, j. y. (2018) ‘a study of students’ perspectives on a flipped learning model and associations among personality, learning styles and satisfaction’, innovations in education and teaching international, vol. 55, no. 3, pp. 314–324. doi: 10.1080/14703297.2017.1286998 lage, m. j., platt, g. j. & treglia, m. (2000) ‘inverting the classroom: a gateway to creating  an inclusive learning environment, journal of economic education, vol. 31, no. 1, pp. 30–43. doi: 10.1080/00220480009596759 lee, h. w., lim, k. y. & grabowski, b. l. (2010) ‘improving self-regulation, learning strategy use, and achievement with metacognitive feedback’, educational technology research and development, vol. 58, no. 6, pp. 629–648. doi: 10.1007/s11423-010-9153-6 morgan, h., et al., (2015) ‘the flipped classroom for medical students’, the clinical teacher, vol. 12, no. 3, pp. 155–160. doi: 10.1111/tct.12328 muñiz, j., elosua, p. & hambleton, r. (2013) ‘international test commission guidelines for test translation and adaptation: second edition’, psicothema, vol. 25, no. 2, pp. 151–157. doi: 10.7334/psicothema2013.24 muñoz-san roque, i., et al., (2016) ‘autopercepción del nivel de desarrollo de la competencia de aprender a aprender en el contexto universitario: propuesta de un instrumento de evaluación (self-perceived level of development of the “learning to learn” competence in the university context: a proposed measuring instrument)’, revista de investigación educativa, vol. 34, no. 2, pp. 369–383. doi: 10.6018/rie.34.2.235881 o'flaherty, j. & phillips, c. (2015) ‘the use of flipped classrooms in higher education: a scoping review’, internet and higher education, vol. 25, pp. 85–95. doi: 10.1016/j. iheduc.2015.02.002 russell, t. l. (1999) the no significant difference phenomenon: as reported in 355 research reports, summaries and papers, north carolina state university, raleigh, nc. rychen, d. s. & salganik, l. h. (eds.) (2001) defining and selecting key competencies, hogrefe & huber publishers, ashland, oh. salcines, i., et al., (2018) ‘validación de la escala de autopercepción de competencias transversales y profesionales de estudiantes de educación superior. (validation of selfperception scale of transversal and professional competences of higher education students)’, profesorado. revista de currículum y formación de profesorado, vol. 22, no. 3, pp. 31–51. doi: 10.30827/profesorado.v22i3.7989 salmerón, h. & gutierrez-braojos, c. (2012) ‘la competencia de aprender a aprender y el aprendizaje autorregulado. posicionamientos teóricos. (the learning to learn competence and self-regulated learning. positioning theorists)’, profesorado. revista de currículum y formación de profesorado, vol. 16, no. 1, pp. 5–13. shah, r. k. & barkas, l. a. (2018) ‘analysing the impact of e-learning technology on students’ engagement, attendance and performance’, research in learning technology, vol. 26, no. 210. p. 2070. doi: 10.25304/rlt.v26.2070 shiau, s., et al., (2018) ‘evaluation of a flipped classroom approach to learning introductory epidemiology’, bmc medical education, vol. 18, no. 1, p. 63. doi: 10.1186/ s12909-018-1150-1 trier, u. p. (2003) ‘twelve countries contributing to deseco: a summary report’, in definition and selection of key competences. contributions to the second deseco symposium, eds, d. rychen, l. salganik, & m. mclaughlin, swiss federal statistical office, neuchâtel, pp. 27–64. http://dx.doi.org/10.25304/rlt.v28.2251� http://dx.doi.org/10.3991/ijet.v12i10.7308� http://dx.doi.org/10.1111/medu.12807� http://dx.doi.org/10.1080/14703297.2017.1286998� http://dx.doi.org/10.1080/00220480009596759 http://dx.doi.org/10.1007/s11423-010-9153-6� http://dx.doi.org/10.1111/tct.12328� http://dx.doi.org/10.7334/psicothema2013.24 http://dx.doi.org/10.6018/rie.34.2.235881 http://dx.doi.org/10.1016/j.iheduc.2015.02.002 http://dx.doi.org/10.1016/j.iheduc.2015.02.002 http://dx.doi.org/10.30827/profesorado.v22i3.7989 http://dx.doi.org/10.25304/rlt.v26.2070 http://dx.doi.org/10.1186/s12909-018-1150-1 http://dx.doi.org/10.1186/s12909-018-1150-1 research in learning technology citation: research in learning technology 2020, 28: 2251 http://dx.doi.org/10.25304/rlt.v28.2251 11 (page number not for citation purpose) voogt, j. & roblin, n. p. (2012) ‘a comparative analysis of international frameworks for 21st century competences: implications for national curriculum policies’, journal of curriculum studies, vol. 44, no. 3, pp. 299–321. doi: 10.1080/00220272.2012.668938 zainuddin, z. & attaran, m. (2016) ‘malaysian students’ perceptions of flipped classroom: a case study’, innovations in education and teaching international, vol. 53, no. 6, pp. 660–670. doi: 10.1080/14703297.2015.1102079 http://dx.doi.org/10.25304/rlt.v28.2251� http://dx.doi.org/10.1080/00220272.2012.668938 http://dx.doi.org/10.1080/14703297.2015.1102079 (page number not for citation purpose) 1 *corresponding author. email: alessio.surian@unipd.it research in learning technology 2019. © 2019 a. surian and a. sciandra. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 research in learning technology vol. 27, 2019 original research article digital divide: addressing internet skills. educational implications in the validation of a scale alessio suriana* and andrea sciandrab auniversity of padova, padova, italy; buniversity of modena and reggio emilia, reggio emilia, italy (received: 8 september 2018; revised: 28 june 2019; accepted: 25 july 2019; published: 23 august 2019) recent studies indicate that internet skills have a positive impact on academic achievement. this article presents a national study that seeks to validate an internet skills scale that was already tested in other eu countries (the netherlands and the united kingdom) to understand the competence level of the population as a whole as well as across population sectors. the scale questionnaire was completed by a sample of the italian population stratified by gender, age and geographical area. the result is globally consistent at the empirical level as well as at the cross-national level. all the five scales showed excellent internal consistency. keywords: digital inclusion; internet skills; skill recognition; survey research; reliability; confirmatory factor analysis introduction, aims and objectives this italian national study seeks to validate an internet skills scale (iss) (van deursen et al. 2017) that has previously been tested in other eu countries (the netherlands and the united kingdom) with a view to determining internet competence level of the population as a whole as well as across population sectors. the debate and the definition of competencies and skills are of fundamental importance to educational policies. both 21st-century skills and digital skills are considered crucial to future socio-economic and educational scenarios, albeit this combination has been neither sufficiently addressed nor defined. to this purpose, van laar et al. (2017, p. 578) introduce the concept of ‘21st-century core and contextual digital skills’. their framework includes seven core skills (technical, information management, communication, collaboration, creativity, critical thinking and problem-solving) and five contextual skills (ethical awareness, cultural awareness, flexibility, self-direction and lifelong learning). such skills are defined as critical for both people and organisations because they determine technological developments and innovating products and processes. utilising van laar et al. (2017) framework, the present study is concerned with the specific skills that relate to a competent use of the internet. gui, fasoli, and carradore (2017, p. 169) claim that individual ‘digital well-being’ is not a function of personal ‘digital well-being skills’ alone, but it is also dependent on the norms, values and expectations of the social context. mailto:alessio.surian@unipd.it http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2155 a. surian and a. sciandra 2 citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 (page number not for citation purpose) the specific skills that relate to a competent use of the internet need to respond to the development of internet and digital technologies. such development is having an impact on the demand for skills and the design and implementation of key policies to foster skills development that take into account the digital world and related educational and vocational policies. the internet requires specific facilities and competencies (van deursen, helsper, and eynon 2016; weinert et al. 2011, p. 82), including the identification and management of the appropriate tools and information, the ability to access contextual operations, the ability to assess and to communicate data and information according to specific contexts. a better awareness of these issues by educational environments enables such organisations to improve the way they address inequalities in terms of internet access (first-level digital divide); internet skills and use (second-level digital divide and the focus of this paper) and third-level digital divide which highlights the tangible outcomes of internet use (scheerder, van deursen, and van dijk 2017). this is particularly relevant in the learning and education domains where online learning management systems, massive online open courses and open educational resources expand opportunities for learning. in addition, online educational games offer opportunities to develop collaborative and communication skills. recent studies indicate that internet skills have a positive impact on academic achievement, with school performance being positively associated with longer internet use for study (kim et al. 2017; leung and lee 2012). this effect is found to be stronger for students with low academic performance or poor socio-economic background and, conversely, for students in technical or vocational schools (pagani et al. 2016). according to oecd (2016), the increasing use of online digital technologies is reshaping everyday activities as well as the business models and organisational policy, suggesting that digital literacy skills are becoming increasingly important. based on these observations, the european union has defined a digital competence framework (carretero, vuorikari, and punie 2017). the framework is articulated in five competence areas each of which incorporates internet skills. the present article focuses on one eu country, italy. an interesting aspect of the study is the validation of the iss in a country that, according to the italian digital agency (oecd 2017), lies behind eu average in terms of digital skills, as well as in the number of internet users. specific aims are twofold: (1) an exploration into how digital exclusion may be affecting vulnerable groups in society (helsper and reisdorf 2017), and (2) to determine the types of information that might be relevant to inform learning policies and actions. it is argued that this is especially important should it be necessary to address biases in the provision of learning and training opportunities for low-skilled adults (oecd 2017, p. 108). the internet skills scale based on the principle that internet skills represent a fundamental element of digital inclusion, van deursen, helsper, and eynon (2014, 2016) proposed a definition centred on individual abilities because internet skills are required to access the key features of the information society. such skills relate to a socially accepted and effective use of the internet in accordance with established standards such as netiquette. internet skills are distinct from computer skills because internet use implies particular actions such as searching for information, online communication and creating online content. van deursen, helsper, and enyon (2014, 2016) include theoretically informed http://dx.doi.org/10.25304/rlt.v27.2155 research in learning technology citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 3 (page number not for citation purpose) survey measures of individual’s digital skills, as their conceptualisation accounts for technical aspects (medium-related skills) and substantial aspects (content-related skills). in this way, a technologically deterministic interpretation of internet skills is avoided by enhancing the role of content-related skills along with technology (i.e. medium)-related skills. through the development of operationalisation and validation of the iss in many studies conducted, particularly in the netherlands, van deursen, helsper, and eynon (2014, 2016) determined a framework consisting of five skill sets: (1) operational skills are the basic technical skills required to access on the internet (e.g. opening a website, navigating forward and backward by using browser buttons, opening, sending or receiving email, using a search engine, managing different file formats); (2) information navigation skills relate to searching, selecting and evaluating information on the internet or, in other words, to maintain the orientation in navigation; (3) social skills include several abilities among which communicating and interacting online to understand and exchange meaning; managing contacts online; attracting attention online and profiling; (4) creative skills are essential to create content to be published or shared with others on the internet (e.g. text, photo, video) and (5) mobile skills relate to the ability to install and to monitor mobile devices newer applications. consistently with the work by van deursen et al. (2017), the italian validation of iss provides an overall framework including both web 1.0 activities implying operational and information navigation skills, and web 2.0 activities implying social and creative skills. finally, as the skills in using mobile devices are currently considered very crucial, it was decided to keep mobile skills as a separate scale. the five-dimension iss uses the likert format for each response item, ranging from ‘not at all true of me’ to ‘very true of me’ in terms of the respondents’ behaviour related to internet skills truth claims. in this way participants indicate the extent to which they believe each item to be true of them, or they can choose the ‘i do not understand what this means’ option. the complete iss includes 35 items, while a shorter version includes 23 items only. we elected to validate the 35-item scale for the italian population. the higher number of items provides a higher probability to identify cultural diversity issues and therefore to control for potential cross-cultural issues as well. the iss implementation in the united kingdom and the netherlands indicated promising results in terms of reliability and external, convergent and discriminant validity (van deursen, helsper, and eynon 2014, 2016), so we chose to validate the questionnaire1 through a representative italian sample, as explained below. moreover, we added one new internet skills item addressing open source software (oss) to allow further investigations and questions about respondents’ profiles to check potential groups comparison variables. research design the study has been structured into three main phases: pre-test, pilot survey and full test validation. 1. the iss items are available at http://alexandervandeursen.nl/joomla/articles/journal/ 2015%20-%20iss.pdf. http://dx.doi.org/10.25304/rlt.v27.2155 http://alexandervandeursen.nl/joomla/articles/journal/2015%20-%20iss.pdf http://alexandervandeursen.nl/joomla/articles/journal/2015%20-%20iss.pdf a. surian and a. sciandra 4 citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 (page number not for citation purpose) during the pre-test phase, our research team translated the 35 items from english to italian. the first step of the process of cross-national validation began with a group discussion to verify the understanding of the italian translation, especially to detect issues of cultural diversity, that is, items that could not be understood as intended by the survey developers. before starting the pilot survey, we conducted interviews aimed at checking item understanding with subjects representing different segments of the italian population according to age, gender and education. in this way we explored whether there was any problem regarding item understanding and question formulation. we did not detect any issue of cultural adaptation. two minor changes were required to achieve better comprehension: we replaced tumblr with instagram in an example within the social subscale; smartphone and tablet were added as examples of ‘mobility’ within the mobile subscale. as recommended by van deursen, helsper, and eynon (2016), the information navigation skill scores were reversed because this scale contained negatively worded items. the study’s second phase included a pilot survey to validate the italian translation of the instrument, in particular by testing the reliability of the five scales. in the last phase of the study, we conducted a full test validation through a representative sample survey of italian internet users, including reliability analysis and general fit of the model to the data (cronbach’s alphas, confirmatory factor analysis [cfa]); external validity analysis (checking the averages across the scales for different socio-demographic groups; convergent and discriminant validity of the scales) and scale consistency (stability of the factor solution; correlations analysis between factors within different socio-demographic groups). pilot survey after the pre-test, we conducted pilot tests with an online questionnaire in may 2017. the aim of the pilot tests was to verify the italian translation and to check the reliability of the five iss scales. a total of 61 respondents completed the online survey. from a demographic perspective, we determined the respondents’ age, gender, diversity of professional and academic backgrounds. the operational, social and mobile scales showed significantly high means, but these values are very close to the ones of the dutch survey (van deursen, helsper, and eynon 2016). as shown in table 1, out of five alpha coefficients only one (social) was below 0.80, the other four scales show alphas equal to 0.85. only three items, if deleted, would increase their scale, cronbach’s alphas. only one of these three items belongs to the questionnaire short form, so our decision to use the full-scale questionnaire is corroborated by this result. these results made us very confident about the italian translation and the five iss scales. table 1. scale characteristics in pilot survey. mean variance a operational 4.52 1.05 0.85 information navigation 2.01 1.25 0.85 social 4.35 1.27 0.78 creative 2.95 2.15 0.85 mobile 4.32 1.40 0.85 http://dx.doi.org/10.25304/rlt.v27.2155 research in learning technology citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 5 (page number not for citation purpose) full test results participants – samples and settings this section explores whether the scales constructed and validated in the netherlands show reliability, internal and external validity across the italian population of internet users. as suggested by van deursen, helsper, and eynon (2016), by including other cultures in further investigations it should be possible to truly test the cross cultural validity of the instrument. therefore we examined the consistency of the iss with a representative sample survey of italian internet users. for this purpose, we used the professional services of swg, a research institute that created a 60 000-people online panel that is believed to be a largely representative sample of italian population. as members respond to some surveys, they accumulate points that allow them to receive rewards. swg panel includes users with different internet skill levels, so it should fit for our cross-national validation framework. on july 2017, a questionnaire comprising iss was completed by a sample stratified by gender, age and geographical area (data taken from the national institute of statistics – istat) of 1067 adults, for a confidence level of 95% and a margin of error of 3%. each of the participants took approximately 8–9 min to complete the questionnaire. table 2 shows the characteristics of the respondents. the following statistical analyses recall those carried out by van deursen, helsper, and eynon (2016). when necessary, we added more analyses to explore critical results further. cronbach’s alphas and confirmatory factor analysis we tested iss reliability and fit on the italian population survey by looking at cronbach’s alphas and cfa. table 3 shows that all five scales have excellent internal consistency because no alpha is below 0.90 and each of them is systematically higher than the equivalent scale of the dutch sample (van deursen, helsper, and eynon 2016). a cfa was carried out on the whole sample to test the validity of the five-factor model. mobile scale is not analysed separately in table 4 because it is made up of table 2. demographic profile of the italian sample (n = 1067). n % gender male 516 48.4 female 551 51.6 age 18–30 153 14.3 31–45 277 26.0 46–60 298 27.9 61+ 339 31.8 education primary 129 12.1 secondary 545 51.1 tertiary 393 36.8 primary = 8-year schooling degree; secondary = 13-year schooling degree; tertiary = bachelor or above. http://dx.doi.org/10.25304/rlt.v27.2155 a. surian and a. sciandra 6 citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 (page number not for citation purpose) three items, thus this model is saturated. fit assessment was based on several indices. as the χ2 statistic is dependent on sample size, we considered also the comparative fit index (cfi), as it performs well with large samples. values above 0.95 are usually considered satisfactory (schermelleh-engel, moosbrugger, and muller 2003). the root mean square error of approximation (rmsea) was also used. this is an absolute fit index assessing approximation of parameter estimates to true parameters in the population. values below 0.08 reflect good fit (schermelleh-engel, moosbrugger, and muller 2003). the theoretical model provided an excellent fit to the observed data: all the subscales and the overall scales model show optimal cfi and rmsea values, very similar to those deriving from dutch survey. furthermore, all estimated factor loadings were significant and generally high (>0.50) confirming very good levels of internal factor structure validity. correlations of the subscales are presented in table 5. all the correlations between the scale constructs were significant in the italian population sample and higher than the dutch sample. we must note that the large samples boost levels of significance. external validity in this section we examined the averages across the scales for different socio demographic groups and tested for convergent and discriminant validity of the scales. table 3. scale characteristics in italian sample. mean variance a operational 4.53 0.93 0.92 information navigation 4.38 0.89 0.90 social 4.46 0.95 0.91 creative 3.43 1.90 0.92 mobile 4.25 1.36 0.91 table 4. cfa fit (n = 1067). c2 df p cfi rmsea 90% ci p aic operational 15.29 16 0.50 1.00 0.00 0.0–0.027 1.00 93.29 information navigation 9.42 10 0.49 1.00 0.00 0.0–0.032 1.00 61.42 social 3.61 5 0.61 1.00 0.00 0.0–0.036 0.99 35.61 creative 15.82 9 0.07 1.00 0.03 0.0–0.048 0.97 69.82 overall scales 1758.27 495 0.00 0.99 0.049 0.046–0.051 0.76 2028.27 table 5. correlations between scales. operational information navigation social creative information navigation 0.754 1 social 0.637 0.655 1 creative 0.542 0.564 0.534 1 mobile 0.688 0.632 0.571 0.580 http://dx.doi.org/10.25304/rlt.v27.2155 research in learning technology citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 7 (page number not for citation purpose) next, we tested the convergent and discriminant validity of the scales and whether the scale characteristics would be consistent through random resamples of the population using bootstrapping techniques. reliability analysis shows that all the scales have rather similar alphas across different socio-demographic groups (table 6). most of the cronbach’s alphas are above 0.90, while the lowest values were found in the student group. we must stress that there were only 58 students in our adults’ sample, their ages ranging from 18 to 36 (with 23.04 as mean age). all the differences between groups were significant, as predicted by the literature and confirmed in the dutch sample: males estimated their own skills higher than females; younger generations estimated their skills higher than older generations; those with higher educational levels were more confident and the employed; and students were more confident having the highest skill levels. convergent and discriminant validity composite reliability (cr), average variance extracted (ave), maximum shared variance (msv) and average shared variance (asv) tests were run to understand whether the factor models show convergent and discriminant validity. in other words, we wanted to test whether the factors were internally coherent and could be externally distinguished from other factors. the analyses reported in tables 7 and 8 provided evidence for the convergent validity because all reported cr values are above 0.70 and ave values for the variables are above 0.50 (fornell and larcker 1981; hair et al. 2010). the above analyses suggest that the latent variables have high convergent validity, suggesting that the items on the scales highly correlate with each other, based on their factor loadings. the social, creative and mobile dimensions exhibited discriminant validity, that is, the ave exceeded the squared correlation for the specific paired constructs. therefore, in these three scales the items correlated more highly with each other than they did with items that were on other factors. yet, the ave for operational and information navigation scales is smaller than maximum shared squared variance (msv) and average shared squared variance (asv). moreover, the square root of ave for table 6. reliability (a) in different groups. operational information navigation social creative mobile men 0.91 0.88 0.92 0.92 0.91 women 0.92 0.91 0.90 0.92 0.90 18–30 0.93 0.84 0.79 0.86 0.77 31–45 0.91 0.88 0.93 0.90 0.88 46–60 0.88 0.88 0.89 0.92 0.85 61+ 0.91 0.91 0.91 0.92 0.91 primary 0.94 0.93 0.93 0.94 0.90 secondary 0.90 0.89 0.91 0.92 0.91 tertiary 0.90 0.89 0.88 0.91 0.91 employed 0.91 0.89 0.91 0.92 0.88 unemployed 0.92 0.90 0.90 0.92 0.90 retired 0.90 0.91 0.91 0.91 0.91 student 0.82 0.80 0.59 0.88 0.83 http://dx.doi.org/10.25304/rlt.v27.2155 a. surian and a. sciandra 8 citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 (page number not for citation purpose) operational and information navigation scales is less than the absolute value of the correlations with another factor. these results show that there is a discriminant concern for the first two scales, so further testing was undertaken, as we conducted a χ2 difference test for the overall scales fitted model with a model with four factors (operational and information navigation scales had been merged in a unique scale). by using a scaled difference χ2 (satorra and bentler 2001), we assumed that a large change in χ2 compared to the difference in degrees of freedom indicates that the freed parameters constitute a real improvement. the χ2 difference was significant (χ2=406.19, df=4, p<0.01), indicating that the five-factor solution was significantly more accurate in its reconstruction of the observed data than the four-factor solution (bagozzi and yi 1988) and supporting the discriminant validity hypothesis. scale consistency to examine whether the factor solution was stable, we conducted a bollen and stine (1992) test for the overall scales model with the ‘lavaan’ package of r software (rosseel 2012). with this package, the standard p-value of the χ2 test statistic is supplemented with a bootstrap probability value, obtained by computing the proportion of test statistics from the bootstrap samples that exceed the value of the test statistic from the original sample. we chose 2000 bootstrap draws and in all samples the fit was better than in the original model (p <0.001), so we can be confident that the overall scales model is a stable solution for the italian population. reliability comparison between socio-demographic groups following the results of the discriminant analysis for operational and information navigation scales, we focused on examining correlations between these two factors within the different socio-demographic groups, using fisher’s r-to-z transformation test. in summary, in comparing the correlations among the different skills scales across age groups, the operational and information navigation skills scales seem to imply table 7 convergent and discriminant validity indicators skills scales. cr ave msv asv operational 0.91 0.55 0.74 0.58 information navigation 0.90 0.52 0.74 0.57 social 0.90 0.60 0.53 0.52 creative 0.90 0.54 0.52 0.47 mobile 0.90 0.76 0.59 0.47 table 8 factor correlation and ave2 (on diagonal). operational information navigation social creative mobile operational 0.74 information navigation 0.86 0.72 social 0.72 0.73 0.78 creative 0.68 0.70 0.65 0.73 mobile 0.77 0.73 0.65 0.72 0.87 http://dx.doi.org/10.25304/rlt.v27.2155 research in learning technology citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 9 (page number not for citation purpose) a ‘digital exclusion’ dimension as they present higher correlations with the older, the primary education and the retired people. discussion in this article we examined the consistency of the iss when measured in a representative sample survey of italian internet users. the result is globally consistent at the empirical and cross-national levels. all the five scales showed excellent internal consistency, with cronbach’s alpha higher than 0.90. theoretically, the subscales and the overall scales model proved to have optimal fit indexes in the cfa. the analyses provided evidences for external and convergent validity as well. some concerns appeared in the discriminant validity for the operational and the information navigation scales, even though the five-factor solution fit was significantly more accurate than the four-factor solution. van deursen, helsper, and eynon (2016) stress that several items which are often used in information navigation scale loaded on different factors and recommended to include additional items for further investigation. in the italian survey we included an additional item concerning oss, that is, ‘i am able to identify proprietary software vs open source software’. this responded to a specific concern. in the first instance, because we noticed that the dutch survey and the italian pilot survey showed significantly high mean scores in three scales, the new item was introduced as a more difficult benchmark to achieve for an internet user. in fact, the oss question mean score was lower than each scale mean score, except for the creative scale. similarly, the italian survey identified a significantly high mean score for the isss, as shown in table 3. this suggests that in future studies it might be useful to include updated items (litt 2013) such as those that address relevant social issues. moreover, reliability comparison between socio-demographic groups indicated higher correlations between the operational and information navigation skills for the older class (61+), with primary education and retired people. this result is consistent with previous studies that highlight that digital exclusion is becoming more concentrated in (economically and socially) vulnerable groups, as it was found in sweden and the united kingdom (helsper and reisdorf 2017). in the netherlands, the elderly and those with lower levels of education had fewer skills, engaged less and achieved fewer outcomes (van deursen et al. 2017). litt (2013) reviewed several studies finding strong evidence of a positive relationship between education level and internet skills, while mixed (positive and negative) associations were identified in the relationship between age and internet skills. in future studies it would be important to address litt’s (2013, p. 621) distinction between technical versus evaluation internet skills. this should allow a better understanding of the relationship between technology (i.e. medium) related skills versus content-related skills and their evolution not just in terms of age but in terms of educational dynamics as well. according to helsper and reisdorf (2017), patterns of digital exclusion persist even when access is almost universal and many services are only available online. thus, further research would be important to understand why specific groups of people are not taking advantages of existing digital opportunities and what type of learning opportunities could be made available to support them. looking at the whole sample, the five average scale scores are systematically higher in the italian sample than in the dutch one. the possible explanations may concern a more generous self-perception of their skills by the italians and/or the possibility http://dx.doi.org/10.25304/rlt.v27.2155 a. surian and a. sciandra 10 citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 (page number not for citation purpose) that after 3 years since the dutch survey, some skills have improved in the general population. the misalignment between self-perceived skills and actual performances is typical in proxy survey measures because they represent one of the most prevalent ways of measuring internet skills (litt 2013). however, survey measures are usually less timeand budget-consuming and more suitable for researching large samples. therefore, we argue that a regular assessment and improvement of the iss would be essential for the instrument durability. the misalignment between self-perceived skills and performance-based skills indicates a limitation of this study. in performance-based and observation internet skills studies, participants are asked to complete a set of tasks while the researcher observes and evaluates their behaviours. clearly, performance/observation methods present higher validity, despite the known issues concerning reliability and generalisability (litt 2013). we believe that a combination of self-reported data and performance-based measures (gui and argentin 2011) would be a possible improvement of the internet skills collection, as the diffusion of smartphones and related apps could make it easier to record performances. another limitation refers to the sample population because we used an online panel. arguably, however, we were seeking to determine internet skills and would have utilised an alternative research design and instrument of inquiry if internet non-users were to be included. globally, in the italian validation we did not detect any issue of cultural diversity because only minor changes were required to achieve better comprehension. the scores of the information navigation scale were reversed because these items were all negatively formulated. as recommended by van deursen, helsper and eynon (2016), future research should use positively formulated items to measure these skills. no other item needed to be adjusted for cultural difference, so we suggest that future italian research about internet skills should use a faithful translation of the original iss, including some updated items. conclusion tools such as the iss can be instrumental in monitoring key aspects of digital inclusion and to provide potential benchmarks for educational and inclusion policies. pagani et al. (2016) suggest that an increase in digital skills can be specifically helpful in the case of population groups lacking significant cultural stimuli from their family and social context. their rationale is that digital skills can act as a substitute for family background when the latter is poor and they should be a key element in designing both digital inclusion and broader educational policies. iss data offer an opportunity to design activities in at least three areas that are considered crucial in digital inclusion policies by both the european commission (cedefop 2017) and oecd (2017): (1) skills assessment to enable adults to identify their existing skills and ways to upskill; (2) learning offer, aiming at boosting digital skills and/or at allowing progress towards higher qualifications and (3) validation and recognition of adults’ skills. further research is needed to determine the development of internet skills and digital competencies not just as skills that are functional to the job market but also as overall personal growth opportunities, that is, in relation to the role that education plays in supporting the ‘socialisation’ and ‘subjectification’ of individuals in the broadband society (biesta 2015; unesco 2017). http://dx.doi.org/10.25304/rlt.v27.2155 research in learning technology citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 11 (page number not for citation purpose) references bagozzi, r.p. & yi, y. (1988) ‘on the evaluation of structural equation models’, journal of the academy of marketing science, vol. 16, no. 1, pp. 74–94. biesta, g. (2015) ‘what is education for?’, european journal of education, vol. 50, no. 1, pp. 75–87. https://doi.org/10.1111/ejed.12109. bollen, k.a. & stine, r.a. (1992) ‘bootstrapping goodness-of-fit measures in structural equation models’, sociological methods & research, vol. 21, no. 2, pp. 205–229. https://doi.org/ 10.1177/0049124192021002004. carretero, s., vuorikari, r. & punie, y. (2017) digcomp 2.1: the digital competence framework for citizens with eight proficiency levels and examples of use (no. jrc106281), joint research centre (seville site), eur 28558 en, publications office of the european union, luxembourg. https://doi.org/10.2760/38842. cedefop. (2017) on the way to 2020: data for vocational education and training policies. country statistical overviews – 2016 update, publications office, luxembourg, cedefop research paper; no 61. https://dx.doi.org/10.2801/414017 fornell, c. & larcker, f.d. (1981) ‘evaluating structural equation models with unobservable variables and measurement error’, journal of marketing research, vol. 18, no. 1, pp. 39–50. gui, m. & argentin, g. (2011) ‘digital skills of internet natives. different forms of digital literacy in a random sample of northern italian high school students’, new media & society, vol. 13, no. 6, pp. 963–980. https://doi.org/10.1177/1461444810389751. gui, m., fasoli, m. & carradore, r. (2017) ‘“digital well-being”. developing a new theoretical tool for media literacy research’, italian journal of sociology of education, vol. 9, no. 1, pp. 155–173. https://doi.org/10.14658/pupj-ijse-2017-1-8. hair, j., et al., (2010) multivariate data analysis: a global perspective, pearson education, singapore. helsper, e. j. & reisdorf, b. c. (2017) ‘the emergence of a “digital underclass” in great britain and sweden: changing reasons for digital exclusion’, new media & society, vol. 19, no. 8, pp. 1253–1270. https://doi.org/10.1177/1461444816634676. kim, s.y., et al., (2017) ‘the associations between internet use time and school performance among korean adolescents differ according to the purpose of internet use’, plos one, vol. 12, no. 4, pp. e0174878. https://doi.org/10.1371/journal.pone.0174878 leung, l. & lee., p. (2012) ‘impact of internet literacy, internet addiction symptoms, and internet activities on academic performance’, social science computer review, vol. 30, no. 4, pp. 403–418. https://doi.org/10.1177/0894439311435217. litt, e. (2013) ‘measuring users’ internet skills: a review of past assessments and a look toward the future’, new media & society, vol. 15, no. 4, pp. 612–630. https://doi. org/10.1177/1461444813475424. oecd. (2016) skills for a digital world, panel 4.2 background document, ministerial meeting, the digital economy: innovation, growth and social prosperity, cancún, mexico, 21–23 june 2016. oecd. (2017) skills strategy diagnostic report: italy, oecd, paris. pagani, l., et al., (2016) ‘the impact of digital skills on educational outcomes: evidence from performance tests’, educational studies, vol. 42, no. 2, pp. 137–162. https://doi.org/10.1080/ 03055698.2016.1148588. rosseel, y. (2012) ‘lavaan: an r package for structural equation modeling’, journal of statistical software, vol. 48, no. 2, pp. 1–36. https://doi.org/10.18637/jss.v048.i02. satorra, a. & bentler, p. m. (2001) ‘a scaled difference chi-square test statistic for moment structure analysis’, psychometrika, vol. 66, no. 4, pp. 507–514. scheerder, a., van deursen, a.j.a.m. & van dijk, j.a.g.m. (2017) ‘determinants of internet skills, uses and outcomes. a systematic review of the secondand third-level digital divide’, telematics and informatics, vol. 34, pp. 1607–1624. https://doi.org/10.1016/j. tele.2017.07.007. http://dx.doi.org/10.25304/rlt.v27.2155 https://doi.org/10.1111/ejed.12109 https://doi.org/10.1177/0049124192021002004 https://doi.org/10.1177/0049124192021002004 https://doi.org/10.2760/38842 https://dx.doi.org/10.2801/414017 https://doi.org/10.1177/1461444810389751 https://doi.org/10.14658/pupj-ijse-2017-1-8 https://doi.org/10.1177/1461444816634676 https://doi.org/10.1371/journal.pone.0174878 https://doi.org/10.1177/0894439311435217 https://doi.org/10.1177/1461444813475424 https://doi.org/10.1177/1461444813475424 https://doi.org/10.1080/​03055698.2016.1148588 https://doi.org/10.1080/​03055698.2016.1148588 https://doi.org/10.18637/jss.v048.i02 https://doi.org/10.1016/j.tele.2017.07.007 https://doi.org/10.1016/j.tele.2017.07.007 a. surian and a. sciandra 12 citation: research in learning technology 2019, 27: 2155 http://dx.doi.org/10.25304/rlt.v27.2155 (page number not for citation purpose) schermelleh-engel, k., moosbrugger, h. & muller, h. (2003) ‘evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures’, methods of psychological research online, vol. 8, no. 2, pp. 23–74. unesco broadband commission. (2017) working group on the digitalization scorecard: which policies and regulations can help advance digitalization, paris, available at: http:// www.broadbandcommission.org/documents/publications/wg-digitalization-score-cardreport2017.pdf van deursen, a.j.a.m., helsper, e.j. & eynon, r. (2014) measuring digital skills. from digital skills to tangible outcomes project report. available at: www.oii.ox.ac.uk/research/ projects/?id=112 van deursen, a.j.a.m., helsper, e.j. & eynon, r. (2016) ‘development and validation of the internet skills scale (iss)’, information, communication & society, vol. 19, no. 6, pp. 804–823. https://doi.org/10.1080/1369118x.2015.1078834. van deursen, a.j.a.m et al., (2017) ‘the compoundness and sequentiality of digital inequality’, international journal of communication, vol. 11, pp. 452–473. van laar, e., et al., (2017) ‘the relation between 21st-century skills and digital skills or literacy: a systematic literature review’, computers in human behavior, vol. 72, pp. 577–588. https:// doi.org/10.1016/j.chb.2017.03.010. weinert, s., et al., (2011) ‘development of competencies across the life span’, zeitschrift für erziehungswissenschaft, vol. 14, pp. 67–86. https://doi.org/10.1007/s11618-011-0182-7 http://dx.doi.org/10.25304/rlt.v27.2155 http://www.broadbandcommission.org/documents/publications/wg-digitalization-score-card-report2017.pdf http://www.broadbandcommission.org/documents/publications/wg-digitalization-score-card-report2017.pdf http://www.broadbandcommission.org/documents/publications/wg-digitalization-score-card-report2017.pdf http://www.oii.ox.ac.uk/research/projects/?id=112 http://www.oii.ox.ac.uk/research/projects/?id=112 https://doi.org/10.1080/1369118x.2015.1078834 https://doi.org/10.1016/j.chb.2017.03.010 https://doi.org/10.1016/j.chb.2017.03.010 https://doi.org/10.1007/s11618-011-0182-7 reviews edited by philip barker brian clegg, mining the internet — information gathering and research on the net, kogan page: london, 1999. isbn: 0-7494-3025-7. paperback, 147 pages, £9.99. nowadays, in this current era of the web, searching the internet is probably as important (if not more so) as using conventional libraries and the facilities that they contain in order to locate sources of sought-after information. indeed, the internet (as an information resource) is now so important that virtually all students (both in schools and colleges) are given some exposure to it at some stage during their education. because of its importance, there is a 'veritable multitude' of books on searching the internet. for me, this book is a particularly attractive one because of its low cost, its scope and coverage and, most importantly, the style in which it is written. the author has organized the eleven chapters contained in this book into two basic types: skills and focus. the seven 'skills chapters' (namely, 1 to 6 and 11) are intended to provide helpful information on how to find things on the internet. the four 'focus chapters' (7 to 10) are then used to offer help with specific information topics such as quick reference, locating people, expertise, archives, and so on. throughout the book the author employs four different types of display box (these are labelled zoom in, try it, sample sites and miner's tales) in order to provide useful additional, sometimes anecdotal, material to accompany (and amplify) that contained in the main textual narrative. at the 80 end of the book there is a useful appendix entitled 'getting onto the internet'. this discusses connection techniques (modems, isdn, and so on) and the type of software that is needed in order to use the internet. because of their introductory nature, the first three chapters of the book are aimed at newcomers to the internet and the web. these chapters describe what the internet is (and is not), how and when to use it (using a web browser, ftp and telnet) and some of its important attractions and limitations. the real substance of the book commences in chapter 4 (entitled 'search engines and indexes'). this chapter discusses the basic ideas underlying web-based information retrieval using, both portals and specific search tools. as well as outlining techniques for improving the quality of a search, the author describes many of the pitfalls to be wary of. chapter 5 ('agents and ferrets') discusses meta-searching and various tools to support this activity (such as dogpile, metacrawler and ask jeeves) while chapter 6 ('information from people') homes in to the use of electronic mail, newsgroups, conferences and online chatting for gleaning information from other people. the final 'skills chapter' (chapter 11 'getting launched') acts as a sort of 'super recap' which pulls together the various key aspects of the material that was presented in previous parts of the book. the 'focus chapters' commence at chapter 7. the first of these 'quick reference and news' describes the web as the world's 'largest alt-j volume 8 number 2 encyclopaedia' and then goes on to discuss how to obtain reference information from various sources such as electronic books, virtual newspapers, e-magazines ('e-mags') and broadcasters' sites (containing tv and radio listings). for some reason, even though e-mags are described, the use of electronic journals is not considered either in this section of the book or in the later part that deals with education. in the following chapter the author focuses his attention on finding people (by looking for contact details such as email addresses), locating places (by using online maps) and obtaining details of organizations (primarily, businesses and educational establishments). the focus of chapter 9 is on locating and using webbased expertise and archives. this chapter contains a useful section on 'digital libraries' as sources of multimedia information such as photographs (and other forms of static image), sound files and video clips. the section concludes with a cautionary reminder about the complexity of copyright law with respect to web-based resources. chapter 10 (entitled 'buy, buy, buy') is the final focus chapter. here the author discusses electronic commerce that is, buying and selling on the web. some of the important issues discussed in this chapter include: techniques for locating items of interest (for purchase); security issues relating to using credit cards with a web browser, buying software packages from the net; and, of course, virus protection. overall, this is a very useful book, which i commend very highly. however, i do have some minor reservations about the limited depth to which some of the topics are covered. indeed, i believe that many readers, having been through this book, would probably want to graduate on to a more advanced treatment of web searching that could build on the basic introductory material that clegg's book presents. therefore, for anyone who might want to seek out some 'follow up' reading on internet searching, i would strongly recommend the 'web search engines' book by randolph hock. hock's text would be a very useful supplement to this book as it goes into the underlying mechanisms of search engines in considerable depth. details of hock's book can be found at http:llwvnv.onstrat.com. philip barker, university of teesside danny saunders and jackie severn (eds.), the international simulation and gaming research yearbook, volume 7: simulations and games for strategy and policy planning, london: kogan page, london, 1999. hardback, isbn: 0-74942956-9, xiii+284 pages, £35.00. this is the latest in a series of yearbooks supported by sagset, the society for interactive learning. sagset is a professional organization, which aims to develop the use of games and simulations in education and training. the editors' introduction provides essential information on a variety of definitions for the terms 'simulations' and 'games' and an overview of the book's contents. this book contains six sections with twenty-five articles called chapters. each chapter has accompanying biographical information and either references or recommendations for further reading. section 1 deals with theory and strategy; it consists of five chapters, the first of which looks specifically at strategic planning in higher education. the use of simulations to deal with real-life scenarios is dealt with in a simple straightforward manner. the second chapter looks at the use of games and simulations in education, training and research whilst providing a theoretical model and framework for a taxonomy on gaming. it covers a breadth of information and places the use of games and simulations into a structured framework. chapter 3 focuses on a case study to link the theory and practice of simulation, whilst chapter 4 uses the case study to develop hypotheses regarding the contribution that gaming and simulation can offer in a research context. chapters 3 and 4 use examples from educational settings to provide believable, practical articles. the final chapter in this section, in contrast to the first four, looks at an interactive procedure used by the gaming machine industry called hilo. the results of a survey on the probability of winning using either a rational or irrational play strategy are given. section 2 looks at business and commerce within five chapters, numbered six to ten. chapter 6 is concerned with student expectations of a computer-simulated business exercise in the service management industry. it is easy to read, using real-life case study examples. it looks at the realism of the simulation exercise and how students made use of their past theoretical knowledge. chapter 7 provides an evaluation of 81 reviews a specific expert system to assist in high-level strategic decision-making. the expert system is thoroughly reviewed through the development of a prototype and feedback received from students using it. chapter 8 focuses on the design and authoring of two cd-roms for business education. it considers the needs of the learners and the appropriate context mix required, and then states the experiences of the author in its implementation. chapter 9 outlines a game designed for the licensed trade industry. it describes how the system was developed and its practical uses. a list of the type of team decisions available and past participants' responses are given. chapter 10 outlines the use of an activity called a 'war game'. this mechanism is used to pre-test a product prior to its launch to assess the potential effects that the product or service will have in the market place. students, working in groups, undertook a range of roles and became involved in the 'war' either to attack or to defend the company. specific case studies are illustrated and evaluated in terms of the game's usefulness as a learning activity. section 3 looks at communication and culture, subdivided into three chapters numbered eleven to thirteen. chapter 11 outlines a game that attempts to cross language barriers. during the game players cannot use known languages as the game emphasizes communication issues to create a neutral language platform. an outline of the game's philosophy, direction and development provides intensely interesting food for thought. chapter 12 looks at the teaching of psychology using the internet, not as an information store, but as a tool for developing critical questioning and discussion skills. it concludes that the internet is a valuable tool for student reflection as it encourages discussion. chapter 13 provides instructions for a card game and an insight into the benefits that may be achieved by playing it. the setting is politics but the game could be used effectively in other situations. the chapters in this section form an extremely important part of the yearbook as they describe and analyse real games and simulations in context. section 4, covering chapters 14 to 18, looks at management and group processes. chapter 14 describes the rationale and the results of the use of a game within a large organization wishing to effect change. chapter 15 provides a critical evaluation of how simulations can be used by the business community to consider organizational change and effectiveness strategies. in chapters 16 and 17 simulations are used to assist 82 in problem-solving and decision-making situations. both chapters highlight the use of a project management plan to facilitate group problem-solving activities in organizations. chapter 18 examines how different role-play experiences can be de-briefed. it uses particular theories and practices taken from a theatre arts background. section 5 covers chapters 19 to 23 and looks at the areas of science, engineering and technology. these chapters describe the application of games and simulations within specific environments. chapter 19 looks at a business environment using case studies to enable the acquisition of skills and knowledge to improve communication, numeracy and learning how to learn. chapter 20 uses simulations within a nursing context where simulations are a safe and effective training vehicle. chapter 21 outlines the use of games and simulations for management decision-making in a microcomputer-based information system to support teaching and team-building in an interactive scenario. the tv show 'countdown' is the main focus for chapter 22 where, using artificial intelligence, a mental arithmetic game is modelled. in chapter 23 two approaches to address the need for a changing curriculum, where usable professional and social skills are required for graduate construction managers, are • outlined. these approaches involved integrating simulations into project work and introducing problem-based learning techniques. the final section of the book provides yearbook information in two chapters. the first details relevant recent articles and the second provides contact addresses and sources for further information. this yearbook provides a breadth of information on games and simulations. it will appeal to those currently researching this area and to organizations wishing to use effective learning tools. educational researchers and lecturers will find much of interest in this easy to read and practical handbook. anne m. brown, heriot-watt university tim russell, effective feedback skills, second edition, london: kogan page, 1998. isbn 07494-2569-5. paperback, 155 pages, £16.99. assessment remains one of the least researched areas of further and higher education practice. assessment might be seen to have a number of functions, for example to: alt-] volume 8 number 2 • provide evidence of academic quality at a range of levels; • provide data to support judgements about an individuals academic attainment or vocational competence; • support learners understand and apply relevant academic or vocational standards, knowledge or competence; • establish a context for, and evidence to support, meaningful feedback to encourage subsequent learning. feedback from tutors, peers and through mechanisms of self-review and reflection are essential if effective learning is to facilitated. this view of the importance of feedback is explicit in tim russell's informative and practice-orientated book: 'feedback is an essential skill for learning knowledge and skills and there are many different circumstances when feedback will be given within a training and teaching context' (p. 33). this quote also offers a concise insight into the specific orientation and focus of the text. the book itself is part of the kogan page 'practical trainer series' and offers a practical review of 'feedback' as something the 'trainer' provides to the 'trainee'. as such, it is an essentially unidirectional process designed to support the acquisition of competence: 'probably the single most important skill for any trainer is being able to give feedback to trainees on their performance. unless trainees can find out what they are doing right, they will not know how to do it again. unless they can find out what they are doing wrong, they will see no need to change. unless they are shown the correct way to operate, they will not know how to change... if we give too much of the wrong sort of feedback, we can sour the relationship between trainee and trainer, if not the whole concept of training for life'(p. 10). within this context the book offers a number of short chapters focusing on aspects of feedback and debriefing, a more problematic concept for russell and intriguingly employed in 'areas of training where there is no right or wrong', the purpose of debriefing being 'to explore options rather than confirm any particular course of action' (p. 11). specifically the book explores: • issues and terms, with a particular focus on how people can and do learn and on defining the 'feedback' and 'debriefing'; • summary analysis based on experience of the contexts for feedback, how to give feedback, how people might receive such feedback, and how feedback might be turned into a debriefing; • summary analysis of the contexts for and the means of delivering a debriefing session; • and finally, practical advice on reporting back. all of this is presented in 153 pages of text, checklists for trainers, figures and illustrative training documentation. the one-side summary on page 154 repeats and reinforces the focus and rationale for the text. specifically, that feedback 'is probably the most important of all trainer skills... the content of the message might affect the rest of a person's life'. the list of suggestions for further reading identifies a number of the better-known texts in the field but lacks a contemporary feel, the most recent text identified being another from the kogan page stable published in 1993. there is much of value in this book. it explains its subject matter with commendable clarity and is clearly based on extensive practical training experience. it is focused, practical and concise. there are clearly complexities within learning and assessment on which the author is silent. the uniqueness of, and the idiosyncrasies typified by, any individual leaning interaction tend in the writing to be subsumed into the established wisdom of training practice. feedback, for russell, is either 'right' or 'wrong' no post-modern ambiguities here! certainly, the reader seeking a more critical and learnerfocused exploration of the topic will need to look elsewhere. however, those readers who are seeking a short, introductory text designed to 'train the trainer' could do far worse than start with tim russell's readable guide. peter funnell, suffolk college graham chapman (ed.), computer-based learning in science proceedings of the cblis 99 international conference, university of ostrava, czech republic, 1999. isbn: 80-7042-144-4. hardback, 600 pages, £20.00. computer-based learning (cbl) is by no means a new topic, however, its scope and complexity have ensured the subject remains at the forefront of academic discussion. it is expected, therefore, that this extensive edited 83 reviews work, comprising 63 fully refereed conference papers, will be of interest to many within the sciences. contributors are drawn from fifteen nations and although the majority are from the czech republic and central europe, there are a number from as far afield as china, australia, usa and canada. thus, the volume's contents can be considered as a reasonably wide representation of academic research and opinion in the field of cbl. the papers were presented at the fourth international conference for computer-based learning in science, cblis '99, held at the university of twente, enschede, the netherlands. focusing on 'developments in computer technology to assist learning in science', the articles within this publication cover many aspects of this subject. reflecting this, computer-based learning in science is divided into a general section and eight topical sections, namely teaching and learning, intelligent software, teaching packages, simulation packages, virtual reality and laboratory in science teaching, multimedia techniques, distance learning, and evaluation and monitoring of performance. "the computer promises my students an endless supply of information, but what good will that do if they can't make sense of any of it? technology promises to help my students express their ideas better, but what good is that if they don't have any ideas to express?' (lowell monke). this quote is given by jana kapounova, in an approach to information education of teachers. this and the paper by peter j. wright are the two that perhaps best set the scene and reflect the title of this book. in total there are four articles that discuss background issues contained within the initial, general section, which also contains two more specifically directed papers. the teaching and learning chapter contains eight papers. those more general in nature discuss the integration of cbl within the polish education system, the practical adaptation of traditional courseware into web-based instruction to afford multifunction teaching in china and the development of automatic auditing within viable learning systems. the more directed papers report on the implementation of specific cbl examples, including problem-solving, databases and computer knowledge. these tend to focus rather more on the practical aspects of teaching than on the evaluation of use and outcome. a further three papers can be found in the related intelligent software section. 84 indicative of the ever-growing influence of cbl is the chapter on teaching packages that contains 16 papers. not only are many nationalities represented, but so too are many fields. topics are indeed wide-ranging, covering, for example, geometry, soil mechanics, stock price forecasting, biology and digital photogrammetry. some packages discussed are now in educational use, while others are still in the developmental stage. the reports vary in scope from those that are purely descriptive regarding the nature of implementation of a particular package, to specific case studies providing examples of use and those reporting on usability and evaluations of learning. findings by and large, suggest students responded favourably to the introduction and use of these packages, especially for those that offered students active engagement within simulations and models. in addition, there were some indications that learning could be enhanced, albeit where measurement was in terms of pass rates and times to complete for short questions. however, such extraneous effects as motivation levels and increased study time were not eliminated from the results. four simulation packages are next described. each focus on the construction of interactive applications with a high degree of 'real-world' topical relevance, for example air pollutant dispersion, optics and logistics. in a similar vein the following section describes four case studies using virtual reality or virtual laboratories. the chapter on multimedia techniques features seven articles. wide-ranging in nature, they discuss both background issues and individual case studies. the use of the web for material delivery and assessment is reviewed and the design implications of large file sizes, common in multimedia, are considered. case studies report on the use of computer graphics, transferral of a toolbook development to the web, computer-based instruction in matlab, the integration of subjectspecific features within a general framework and a tool for implementing objectorientated models. as well as a general review of information technology in social sciences, the distance-learning section includes papers on the development of design frameworks for situational and/or networked learning, the use of cbl for seminar preparation, a database course and an evaluation of teamwork and video-conferencing within two international projects. the final chapter contains seven papers regarding evaluation and performance monitoring. most alt-] volume 8 number 2 studies used questionnaires to obtain student feedback; however, a few provided evaluations based upon increases in learning as determined by pre and post test results. perhaps of most interest to lecturers are the findings relating to comparisons between computer and paper-based materials. gosper and love reported higher response levels for online questionnaires, but content was not the same as that for the paper version. learning using the esp programme versus paper-based instruction was reported by oliver and oliphant a slight advantage (statistical significance not given) was recorded for esp. educational content was deemed the same; however, details regarding such aspects as course structure and integrated questions and answer sections were not provided. any publication aiming to provide comprehensive coverage of computer-based learning requires consideration of, and reflection on, not only the computer/technological tools used to facilitate learning, but also the cognitive processes of learning. that this work does not neglect the latter creates a rather more balanced view of this topic than is frequently achieved through conference compilations. aspects that detract to some extent from the overall presentation, are the lack of consistency in text formatting and style of referencing, the rather low print quality for many of the figures and some observed difficulties in translation. the sixtythree papers not only suggest a variety of ways in which computer-based learning can be used effectively, but also highlight areas of developmental overlap and indicate many future directions and enhancements to teaching and learning. the proceedings comprise a mixture of background or topic review papers, evaluation studies, empirical research reports and instructive reports; consequently the strength of theoretical underpinning varies considerably from paper to paper. notwithstanding this diversity, there is much to interest the computers-in-teaching enthusiast and much to encourage those still not totally convinced of the benefits of using computer technology in science teaching. kate garland, university of bristol leslie rae, using evaluation in training and development, london: kogan page, 1999. isbn: 0-7494-2805-8. paperback, xi+243 pages, £19.99. leslie rae's latest handbook aims to provide everything a trainer needs to evaluate training, from the design of the evaluation process to the use of particular types of test. in typically readable fashion, rae produces practical guidelines for the busy professional which quietly broaden perspectives and challenge assumptions. his style is succinct and his text liberally interspersed with boxed lists, diagrams and summaries. rae's main concern is to bring a more considered and professional approach to evaluation, which he believes is one of the least practised subjects in training. despite the potential cost and time of his proposals, he regards evaluation as essential to demonstrating the effectiveness of training and the amount invested in it. his audience are the trainers and managers he believes have a stake in evaluation. he also caters for those working towards investors in people or training and development nvqs. in an era of accountability, anybody preparing and presenting data about training should find using evaluation in training and development a valuable source of ideas and evidence. the book offers a range of practical ways for embedding evaluation in the training cycle. rae's focus is broad and the frameworks he presents build on the work of influential authors like kirkpatrick, honey and newby. he considers evaluative measures that take place before, during and after training, including long-term follow-up. he offers guidelines and practical resources for the wide range of people responsible for the success of the training process. he deals with direct training, selfinstruction, open, distance and programmed learning. although he is not concerned with the evaluation of education programmes, many of the techniques rae outlines are adaptable for education or training. the opening chapters discuss the place and function of evaluation and rehearse familiar reasons for its neglect. these range from an overreliance on end of course 'happiness sheets', to a failure of stakeholders to take responsibility for their role. to address this, rae underlines the need actively to involve key players in all stages of the cycle. line managers, for instance, should help establish learning outcomes and play more part in briefing learners before and after programmes. chapters 4, 5 and 6 are some of the most interesting in the book. they begin by introducing non-specialists to current approaches to testing. rae outlines a variety of techniques 85 reviews including the use of likert and thurstone scales in multiple-choice tests. complementary methods of assessing knowledge and application are explored. his insights often strike a chord. his advice against using experts to set practical tests will be familiar to those who have tried to elicit knowledge from experts who suggest short cuts which do not correspond to complete and agreed practice. once or twice his touch deserts him experienced educators may cringe, for instance, at the advice to avoid using notice boards to publish league tables of test results. he regains his poise as he considers tools for observing interpersonal behaviour in groups. by devoting almost a chapter to behaviour analysis, he introduces tools for the quantitative and qualitative profiling of individual activity in groups. he covers simple contribution scoring, directional sociograms and reflective, self-reporting techniques. he handles the area well and, though he avoids being drawn into conceptual debate, his approach to group interaction is lively and provides useful background for those keen to research it. it is a chapter to return to. chapters 7,8 and 9 cover evaluations which take place during and after training programmes. i enjoyed reading about thermometers, speedometers and progressive blob reviews. the limitations of end-of-course reactionnaires which fail to illuminate the extent of learning or its transfer to the workplace are underlined well. the chapter on interim evaluations contains the sensible advice to avoid undertaking them unless you are prepared to address the issues they raise. it also presents some thoughtful variations on learning logs. educational developers will probably find the chapter on evaluating open and distance learning disappointing. it runs to just ten pages and focuses on instructional packages. although the absence of frameworks for evaluating networked learning or the quality of dialogues for iterative design is understandable, it is perhaps surprising that rae's earlier interest in group dynamics does not prompt him to look at asynchronous, collaborative work. the short section on media-based learning offers less to the learning technologist than the dedicated tools we now have available through ltdi and tltp. even at ten years old, the reliable meda materials offer a richer framework for evaluating educational software. despite this, like most of leslie rae's books. using evaluation in training and development is 86 very readable. it offers those new to training a framework for evaluation and experienced trainers practical solutions, using tested resources. it supplies staff or educational developers with materials that can supplement increasingly specialized resources. it provides a sound evidence base for those putting together portfolios for accreditation. its great strength is to suggest ways of evaluating staff development interventions more strategically by involving a wider number of stake-holders. in a busy world, including all members of rae's training quintet (learner, trainer, line manager, training manager and senior manager) may be unrealistic but, for organizations with the time, he provides the tools to do so. using evaluation in training and development is a useful addition to the trainer's bookshelf. paul haslam, king alfred's, winchester alistair inglis, peter ling and vera joosten, delivering digitally: managing the transition to the knowledge media, london: kogan page, 1999. isbn: 0-7494-2933-x. hardback, xiv+210 pages, £35-00. delivering digitally 'is designed for those with responsibilities for managing the transition from traditional approaches to education and training whether face to face or at a distance to forms of digital delivery. this includes everyone from the chief executive officers of educational and training organisations to heads of departments, course co-ordinators and technical and educational specialists' (p. xii). the book is divided into four sections: 'context', 'implementation', 'quality assurance' and "the future'. each of these sections is intended to stand on it own; and, indeed, most of the book's thirteen chapters are reasonably self-contained and each provides valuable insights into managing this process of change. in an industry that is changing as rapidly as that of digital educational media, it is not surprising that the first section of the book, 'context', is not its strongest section. none the less the initial three chapters do contain some useful material. in these chapters the authors seek to contextualize the issues and provide a foundation upon which policy can be built. in the first chapter, "the origins of the knowledge media', they set forth a definition of digital media that is perhaps a little too computer-centric. the phrase 'knowledge media' was coined by marc eisenstadt at alt-] volume 8 number 2 the uk's open university (p. xiii). they, like eisentstadt, use it to refer to web and cd-rom based material. given the convergence of digital broadcasting with web and other computerbased media, this perhaps needlessly narrows the book's perspective, but doesn't detract too much from its central focus on managing the process of change. as the title suggests, the second chapter, 'social forces driving educational change', examines the role of the university in modern society and considers how that role has evolved from cardinal newman's rather elitist view in the nineteenthth century to the more populist, marketdriven model of the present the chapter also contains a series of useful tables that set forth a series of premises and policy implications with respect to managing the change to digital media. indeed, one of the book's greatest strengths is the inclusion of a range of tables, flow diagrams, checklists, and so on, that can serve as valuable models to inform decisionmaking. the final chapter in the opening section, 'learning in an electronic environment', unsurprisingly, offers a fairly 'constructivist' view of education and considers what an ideal virtual learning environment (vle) should be like. significantly and, to this reviewer's mind, rightly, they argue that a vle should include a full range of teaching, learning and administrative functions. they list these as: 'enrolment/ registration of students; provision of course advice; provision of course materials; provision of feedback; provision of responses to administrative and academic queries; conduct of assessment; [and] provision of assessment results' (p. 37). this list alone should suggest to managers just how great is the scale of change implicit in the digital delivery of higher education. the second section, 'implementation' is by the far the largest. its eight chapters occupy more than half the book and in their titles provide a reasonably comprehensive list of the issues facing management in this period of change. the first two chapters, 'counting the cost' and 'a cost or an investment?' make a number of valuable points about the cost implications of digital delivery. by no means the least important of these points are that managers need to be careful to count the hidden costs as well as the more obvious ones and that the case for 'economies of scale' is by no means transparent. however, the most important point in these two chapters is that financial management in higher education needs to become more adept at recognizing when an expenditure is a 'cost' and when it is an 'investment'. the expense involved in migrating to digital delivery is as much an investment in the future of higher education as is the creation and maintenance of the building stock; and will similarly require continuous maintenance in the future. the next four chapters, 're-engineering technological infrastructure', 'reskilling and supporting staff', 'reorienting the teaching programme' and 'redesigning learner support services', address in detail some of the more obvious things that need to be considered. the first emphasizes the importance of establishing clear and manageable targets for the reimplementation of the technical infrastructure. for example, it is pointless adopting a vle, however good, that cannot run over key systems. the crucial issues of staff development and support are addressed in the next of these chapters. digital delivery is novel and staff will need to be trained to exploit it; and, even when trained, they will require new levels of support. still further this staff development process might be described as a 'bootstrapping' one. as the authors put it, 'you can't recruit experience in tomorrow's technology' (p. 85). the next of these four chapters addresses the issue of how far institutions should engage in courseware development and provides some useful guidance about how to choose material that is not created in-house. although this is a very useful chapter in many ways, it should perhaps be noted that it takes little account of the good work in this area that has been done in the uk in recent years. however, it does make one very important point that harks back to the issue mentioned in the last paragraph. this is to do with the corporate memory. not only are universities going to need to create the experience and expertise to exploit digital delivery, they are also going to need to find ways of keeping that expertise within the university community. the last of these four chapters considers the wider impact that digital delivery will have on learners and on those, from librarians to student counsellors, who provide them with supportive services. the impact on these services is easily overlooked, but by no means any less important. the last two chapters of this section as well as the single chapter in section 3, 'quality 87 reviews assurance', address the issues of evaluation, change management and quality control. both 'developing an evaluation strategy' and 'managing the transition' provide a range of checklists and other instruments that can be used by managers (and others) to measure how effective the process has been and how far an institution has progressed along the path from traditional to digital modes of delivery. the third of these chapters looks at how one maintains standards during this process and offers, as the title suggests, 'a quality assurance framework for achieving best practice in new learning technologies'. the single chapter in the final section, 'anticipating tomorrow's innovations', is, like the opening chapter and for much the same reasons, a little disappointing. however, this does not take away from the book's real strengths. 'delivering digitally' provides an insightful overview of the managerial issues involved in meeting the challenges of the digital age; and, still more importantly, it provides a range of practical instruments to support managers engaged in meeting those challenges. bruce ingraham, university of teesside a. h. miller, b. w. imrie and k. cox, student assessment in higher education a handbook for assessing performance, london: kogan page, 1998. isbn: 0-7494-2797-3. paperback, 288 pages, £19.99. assessment is such a ubiquitous concept. it is helpful to know that this book only addresses 'student' assessment in 'higher education'. of course, if we are to understand what assessment is and how it can be used, we really need to generalize to some extent. higher education does have its peculiar attributes. standards and progression vie with subject mastery and transferable skills. the isolation of 'students', with regard to assessment, is illusory. higher education is a complex community where teachers help learners assess their own and each others' performance, and students provide useful evidence for staff who need to show their developing professionalism for accreditation. this applies throughout education. a final thought, as a prologue the book suggests it is a 'handbook' for assessing 'performance'. i will deliberately ignore the definitive and practical nature of a 'handbook' and comment that 'performance' is probably the most misunderstood concept, apart from 'competence', used in 88 higher education. i still find that the assessment of what some call intellectual skills is excluded from the ambit of 'performance'. students perform activities, tasks, and even duties but seldom is performance attributed to intellectual skills. it seems to me from what i hear, that subject knowledge is not 'performed' and is therefore invisible to scrutiny by assessment. perhaps the book will help clarify this point if so, it will serve this reviewer well as i have my suspicions. what is the book about? the authors, part of the 'aussie mafia' who have made such an important contribution to curriculum design, professional development and communications and information technology (cit) over the last two decades, are well known for their writing in innovation, curriculum matters and student learning. the aim of the book is to provide an authoritative reference to relevant assessment literature. it is also for those reflective teachers who are dissatisfied with their current practices. maybe it is part of the reading list for a 'fast track* to institute of learning and teaching accreditation. the book falls into three parts a broad brush pedagogic background; a more practical 'how it is done' section that includes a good chapter on theses; and an 'issues' section that even includes matter such as ethics and plagiarism (see below). so the cloth is broad but, is it well-cut? the best way to test this is twofold. first, ask a question and see if the book provides an answer that is authoritative, useful and that generates reflection in the reader. second, allow the authors to offer something that the reader did not know he/ she did not know they did not know! in other words a proactive question and a reflective response. the question first: 'do examinations hinder student learning?' if one is to be honest, this is not a question that is often heard in common rooms and 'quality assurance' visits. rather, it is implicit in many of the comments made sotto voce on these occasions. one hears of the efforts that go into making sure that grades are as reliable as possible; that self-assessment is an 'off the wall idea' best consigned to the 'soft sciences'; or that the use of cit will revolutionize marking by removing the tyranny of the examination paper and freeing up so much precious academic time! the trouble with books is that authors do not necessarily think of answers in the way readers pose questions. i think the answer to my alt-j volume 8 number 2 question is there. chapter 3 offers some guidance. apart from the usual citations to the marton et al. work along with entwistle, the authors offer a brief guide to a serialist vs. holist dichotomy they infer from gordon pask. while accurately reporting the essence of this oft cited distinction, they miss the range of approaches to learning that pask recorded for example, 'globetrotting as a pathological' approach. i do not think there is too much mileage in this analysis in a book on assessment. they need to consider the tasks that bring about the response identified. i don't think the authors suggest that approaches or styles are stable which would be a fallacy; rather they tell me about a classification related to learning (e.g. bloom again!) without explaining how this is effected by assessment tasks. on the other hand they drew my attention to the work of another australian writer john bowden. the citation on page 43 which reads 'unfortunately there is a good deal of recent evidence to suggest that the quality of student learning . . . is adversely affected by inappropriate assessment methods' rings a bell! it suggests that the issue is about the 'quality' of learning as opposed to a fixed or static style, and the interaction of this with different methods. as a reader, i will follow up the bowden surveys and analysis and i take seriously the advice offered by the authors (p. 45): 'involving students in discussions of learning style and learning strategy would provide a sound basis for both students' intellectual development and quality assurance of the teaching program'. so, i am encouraged by the book's response to my question. i reason that there is some interaction between the methods used in assessment ('examinations') and the quality of the learning that occurs. a good start. now, what can i learn from the authors and what does it mean to me? i am going to take the issue of 'academic (dis)honesty' (chapter 15, pp. 241-52). i really had not thought too much about this until the authors stopped me in my tracks and made me wonder how i would answer a quality assessment style of question ' . . . umm . . . can you tell us . . . umm . . . what procedures you have in place to make sure that the judgement you make of student abilities a r e . . . u m m . . . indeed judgement of abilities rather than judgements on what the students have found out?' we need to consider the ethics and codes of conduct and procedures by which those involved in assessment come to a valid conclusion on performance. cit and the ubiquitous internet when it is working! are forcing us to consider 'what does an assignment handed in by a student tell us about that student's cognitive abilities rather than his/her abilities to find an appropriate essay and pay for it?' the authors are right to identify the 'qualified trust' (cf. dealing) that needs to operate between society and institutions with regard to student abilities. i take away from the book an idea and some useful citations plus a nice twist on the interesting idea 'copy from one, it's plagiarism; copy from two, it's research' (citing wilson mizner (1876 -1933)). i judge the book to be useful and stimulating. it gave me answers to questions i posed and it offers ideas i had not thought about. so, this is a book to be recommended. thinking of assessment, i would offer kogan page some advice on the index. too short with insufficient cross-indexing a beta minus on my personal utility scale! as a final question i wonder what criteria the authors set themselves in order that they could evaluate the success of this book? any selfassessment question is at the heart of effective learning and assessment. if students are unable to know the criteria that are being used in order to make a judgement or if teachers are not explicit in what they value so that students know what is valued we have a system that will not develop or improve. self-assessment must be one of the under valued issues in higher education and what is now called personal development. now, where in the book did the authors make reference to this issue.. .umm . . . ? ray mcaleese, heriott watt university david lowe and wendy hall, hypermedia and the web an engineering approach, chichester: john wiley & sons, 1999. isbn: 0-471-98312-8. paperback, xxiv+597 pages, £27.50. hypermedia authoring software such as toolbook, director or frontpage, can appear all too easy to use. the temptation to design and develop, using a visual interface, at the computer monitor, can prove too strong for many. this book emphasizes the need for a systematic approach to development, particularly of largescale hypermedia. it suggests that there is a crisis in the development of hypermedia, which like the 'software crisis' identified in 1968 requires the formulation of 'engineering' approaches. concepts from software engineering are brought 89 reviews together with several abstract reference models specifically for hypermedia. three sections deal in turn with hypermedia fundamentals, practice and research. hypermedia development is assumed to be so distinct from software development that software engineering is referred to analogously, as something that, serendipitously, may hold lessons for the 'emerging discipline' of "hypermedia engineering'. while the requirements of hypermedia information models are acknowledged to be 'similar' to those of general application development, hypermedia is apparently different enough to spark off its own 'hypermedia crisis', and to necessitate the new discipline of hypermedia engineering. a substantial amount of the material covered, however, reflects standard software development, and aspires towards software engineering qualities and characteristics of maintainability, modularity, adaptability, portability and abstraction. the combination of visually driven development environments, often inherently irregular and exploratory structures, and disparate and dynamic media types, would certainly appear to require new and different approaches. the earlier chapters contain some illuminating taxonomies and nuanced definitions in relation to hypermedia. and the most cogent parts of the book mainly, but not exclusively, located in the substantial research section are those that set out design methods and reference models that are specific to hypermedia. the overview of several key reference models is both representative and clearly summarized, as is the relationship management methodology (which adapts entityrelationship design from software engineering, and does not necessarily lend itself to educational hypermedia). among others, the dexter reference model and the amsterdam model are covered, along with valuable insights from the authors' own microcosm and matilda.projects. each of these research development chapters adopts the same structure, following up the product and process issues identified in the preceding section. this helps to sharpen the focus on these issues, and allows the reader to compare and synthesize the different approaches. the material presented constitutes a useful theoretical resource, particularly given the rapidly expanding role of hypermedia in the form of the web and of computer-based learning. rigorous models and methods are necessitated by the size of projects. in the taxonomy of what 90 is large and not large, educational material is generally considered to be 'large'. as such, ad hoc approaches are rejected in favour of abstract models of node and link structures. also sidelined are less formal constructivist approaches that so often inform the development of interactive educational material. there is little recognition of a need to 'engineer' correct algorithmic content for user activity and other functionality within nodes or components. it is the links that are said to give functionality to hypermedia, rather than anything inside the nodes, which are seen essentially as information units within a browsable document database. an indication of how a document-based psychology might constrain expectations of the analytical functionality of non-textual media, is perhaps given in the use of image-maps as an example of an operation that analyses pictures (rather than as a means of interactive presentation). for educational material where the functional and interactive richness of media should ideally take particular precedence over their presentational role a hierarchical structure, enhanced by 'links to a glossary and references', is said to be suitable. in spite of the acknowledged debt to software engineering, assumptions are made about the prevalence of hacking in software development that appear to overlook thirty years of applied method: where the 'product model' is based on a programming language, it is claimed that there is no means of visualizing the information structure; that it would be difficult to maintain; and that it will be coded 'from scratch' by isolated individuals. the implications of hypermedia production paradigms that combine screendriven authoring with scripting and programming could be explored. with visual authoring software packages, the comments regarding their inability to separate information from interface implementation are not strictly applicable to macromedia director. and while not capable of microcosm's full decoupling of link information from data, with a little scripting effort at a high level and a program-engineering approach toolbook, for example, can facilitate a significant degree of generic associative and navigational linking. the book is, at 600 pages, long. there is a narrative tendency to ask questions (what is hypermedia? what is hypermedia engineering? how is material to be structured and accessed?) on several occasions, and to defer answering them until later. indeed, several important chunks of text are alt-j volume 8 number 2 repeated verbatim at different points. this and the numerous apostrophe and typo errors is a reflection on the copy-editing rather than the authoring of the book. this book is visionary rather than prescriptive pointing in directions rather than giving directions. concerned with large-scale hypermedia development principles in general (rather than hypermedia-assisted learning in particular), it will be particularly useful for academics teaching hypermedia design; for researchers seeking an introduction to the theory of hypermedia development methodology; and for developers including web developers who wish to gain a theoretical understanding of high-level development issues in order to move towards more systematic methods for the design and development of hypermedia. peter mckenna, liverpool hope university college brian tucker (ed.), handbook of technologybased training, aldershot: gower, 1997. isbn: 0566-07809-0. hardback, x+437 pages, £75.00. the use of technology in training (and learning) is a constantly changing field continuously affected by developments in various applicable fields of technology. this handbook provides both a description of the history and development of technology-based training (tbt) and a compendium of available courses. the handbook consists of two parts. the first of these addresses technology-based training and the second comprises cross-indexed listings of tbt courses available. the nine chapters in part 1 contain a description of tucker's view of technology-based training and some real-life examples of its use. the case studies are useful in defining the context and main thrust of the handbook training staff using self-paced and automated computer-based techniques. chapter 1 introduces and defines tbt, chapter 2 discusses possible organizational benefits of utilizing tbt, and chapter 3 comments on the possibilities for implementing tbt. how to use tbt in distance learning, pre-course work, course work, and post-course work is outlined in chapter 4. it is not possible to generalize effectively in relation to tbt; indeed almost all applications in this field have their own specialities and require careful consideration with respect to customization of pre-packaged (generic) solutions. generic training is covered in chapter 5, whilst chapter 6 addresses bespoke training issues such as in-house versus outsourcing. chapter 7 provides guidance for the evaluation of tbt, and includes considerations of other factors that may be considered in course appraisal. one way to gauge the suitability of tbt solutions may be to analyse carefully the requirements, leading to the generation of a requirements specification. this process would benefit markedly from the inclusion of factors described in part 1 of the tbt handbook. the future of training utilizing technology is the subject of chapter 8, while chapter 9 sets out case studies from large organizations which have implemented successful tbt programmes. i found the case studies most useful for a number of reasons. firstly, some measure of confidence is gained in reading about successful implementations of tbt; and secondly, actual cases serve to put the handbook into a real-life perspective. by comparing case studies to a particular need, the reader may be able to access their own needs and include relevant factors that may have otherwise been overlooked. part 1 is, in places, a little thin on the discussion of issues, but does provide an excellent overview to tbt by listing readily categorized areas that must be considered. part 2 lists some 700 generic courseware titles indexed by category, alphabetically, by delivery method and by supplier. each course is summarized in a template format, providing information under the following headings: title, purpose, suitable for, prerequisites, description, delivery method, minimum hardware requirement, price and supplier. this is an excellent format, as it provides a basis for comparing possible outcomes from each course. an address is included to allow tbt producers to add their courseware to the handbook's database for future releases. with the type of material contained in this handbook being so dependent on currently available courseware, it is apparent that regular updates will be essential to ensure its ongoing appeal. overall the book is a good starting point for someone in the market for pre-packaged tbt courses. by addressing some of the best points, and some of the limitations, the handbook prepares the reader to evaluate the courseware titles listed in the directory section of the book. most of the commercial packages listed would 91 reviews appeal to larger organizations, and in some cases, they could provide flexible cost-effective tbt in broad areas such as management, customer services, induction training and the like. the reader would at the very least be able to ascertain the availability of material, and be at a point to commence negotiations with suppliers for courseware to suit their particular needs. lindsay offer, anglia polytechnic university erratum for volume 8, number 1 we apologize for not including an appropriate acknowledgement to the publisher of activstats in the paper entitled 'active learning of statistics: a case study', by erica morris and eileen scanlon, which appeared in volume 8, number 1 of this journal. activstats is published by longman software publishing. the activstats screen shots shown in figure 2, figure 3 and figure 4 of that article are reproduced by permission of longman software publishing. 92 (page number not for citation purpose) 1 *corresponding author. email: gkrull@uoc.edu research in learning technology 2019. © 2019 g. krull and j. m. duart. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 research in learning technology vol. 27, 2019 original research article supporting seamless learners: exploring patterns of multiple device use in an open and distance learning context greig krulla* and josep m. duartb adepartment of business management, iq academy, east london, south africa; bdepartment of education, universitat oberta de catalunya, barcelona, spain (received 18 december 2018; revised: 22 july 2019; accepted: 22 july 2019; published: 13 august 2019) educators need to be aware of not just what their students are learning, but how and why as well. this study investigates how open and distance learning (odl) students make use of multiple devices for learning, particularly how students use their devices together. this study is situated in the context of ‘seamless learning’, where odl students learn at different times, in different locations and with the use of different technologies. understanding the needs and learning practices of students can help to improve the design of learning experiences and support offered to students. data were collected through semi-structured interviews at two odl universities, one in spain and one in south africa. the results show that while students mainly use one digital device at a time, they sometimes make use of two or more devices together. this usage can be characterised as sequential (moving from one device to another) or simultaneous (using two or more devices together at a time). this article describes the study patterns associated with sequential and simultaneous use of multiple devices, facilitated by the use of different devices and synchronisation tools. a continuum of seamless learners is proposed that can be used to help identify levels of support required by odl students. keywords: seamless learning; online learning; multiple devices; higher education; mobile learning introduction in the past, a common goal for educators was to ensure that each student had access to a personal device for learning (chan et al. 2006). however, now students utilise one or more digital devices for learning (galanek, gierdowski and brooks 2018; newman, beetham and knight 2018). the use of personal devices impacts the study habits of students, yet the nature of and motivation for technology use for learning is not always well understood (margaryan, littlejohn and vojt 2011). it is known that students have access to different digital devices, but it is not well understood how students use these devices together or move between fixed and handheld devices. this is of particular relevance in open and distance learning (odl), where students do not learn at fixed times or at a specific location. odl students learn at different times and locations with the aid of different technologies (cross, sharples and healing 2016). mailto:gkrull@uoc.edu http://www.alt.ac.uk/� http://creativecommons.org/licenses/by/4.0/� http://dx.doi.org/10.25304/rlt.v27.2215� g. krull and j.m. duart 2 citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 (page number not for citation purpose) this article will explore how students make use of different devices for learning, either separately or together. the research question that guides this study is ‘how do undergraduate odl students make use of handheld devices together with fixed devices to perform their learning activities?’. the study is framed within the context of seamless leaning, where students can learn in a variety of scenarios, using technology to move between scenarios (chan et al. 2006). the findings are important because they show emerging learning patterns as students make use of different devices together (either sequentially or simultaneously), supported by cloud technologies, to achieve their learning goals. these learning patterns can be characterised as sequential (moving from one device to another) or simultaneous (using two or more devices together at a time). this emphasises the changing role of educators in supporting students by knowing how and where their students are learning. educators need to understand how their teaching methods and course design align with the flexible, mobile needs of their students (awadhiya & miglani 2016). furthermore, educators need to consider whether they are able to deliver seamless learning experiences to students who want to learn in this way (cross et al. 2019). review of the literature the use of various digital technologies is now a common feature in higher education, with the widespread use of learning management systems (lms), online library services and online communication tools (henderson et al. 2015). universities are therefore interested in how technologies are used for learning. this requires an understanding of what technologies students have access to, why and where they study and their levels of digital literacy (cross, sharples and healing 2016). odl students may have multiple commitments (work and personal) and so may need to study at different times and in different locations (cross et al. 2019). various studies have shown that on-campus university students have access to various personal digital devices, particularly mobile devices such as tablets (green et al. 2016; reid and pechenkina 2016). digital devices include both fixed and handheld devices. fixed devices include desktop computers and laptops, while handheld or mobile devices refer to smartphones, tablets or e-readers (cross, sharples and healing 2016). while odl universities may be aware of what technologies students have access to, there may not be a similar awareness of how students make use of different personal technologies for learning. it is important to address this gap because universities need to meet the changing needs of their students in a digital age. understanding the learning patterns of students enables educators to address students’ current needs and anticipate future ones, particularly as technologies are seen as essential for academic success (galanek, gierdowski and brooks 2018). the role use of technologies by university students is of interest to many academics. in several countries, annual quantitative surveys of students’ use of technologies for learning are undertaken to determine insights and trends (galanek, gierdowski and brooks 2018; newman, beetham and knight 2018). these general studies show an increase in the ownership of technological devices and that many students make use of laptops and smartphones for academic purposes. other studies similarly show higher education students’ uptake of technologies for learning, particularly mobile devices and instant messaging services (dolch and zawacki-richter 2018; krull and duart 2017). however, there is a need to investigate patterns of ownership and use in more detail through a qualitative approach (cross et al. 2019; newman and beetham 2017). http://dx.doi.org/10.25304/rlt.v27.2215� research in learning technology citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 3 (page number not for citation purpose) several previous studies have highlighted students’ use of different devices for learning in distance or dual mode universities (cross, sharples and healing 2016; farley et al. 2015). however, these studies did not consider how students make use of their different devices together or only considered the use of handheld devices specifically. this study uses the concept of seamless learning to frame the research. seamless learning means that ‘a student can learn whenever they are curious in a variety of scenarios and that they can switch from one scenario to another easily and quickly using the personal device as a mediator’ (chan et al. 2006, p. 6). students can move between physical and virtual locations such as homes, buses, libraries and lmss (barden and bygroves 2017). a blurring of the seams occurs when students learn in formal and informal settings, at various times of the day, with different resources and technologies (sharples et al. 2012). for example, students can make use of cloud computing to access previously created learning artefacts wherever they are situated and not worry about losing any work (barden and bygroves 2017). although there are many benefits for learners able to learn in a seamless manner, there may be challenges related to network access or device affordances (sharples et al. 2012). the approach of seamless learning guides this research by investigating how learning occurs across time and locations through the use of different devices (wong 2012). learners need support in knowing how to navigate their learning across multiple times and locations when completing different learning activities. not all students know how to positively adapt their study habits when confronted with new technologies (cross et al. 2019). google (2012) found that consumers spent their time using a range of devices in a day, determined by their available time, the goal to accomplish and their location. using a classification proposed by google (2012) for consumer usage of devices, being either simultaneous or sequential, this research aims to explore these two patterns of device use for learning. most commonly, studies of student multitasking behaviour tend to focus on students using two or more digital devices at the same time for unrelated tasks (patterson 2017). however, there are few studies that explore the usage of devices for related learning activities, across different locations and times. methodology this study made use of a qualitative approach through the collection of data via semi-structured interviews (creswell 2009). a case study approach was used to investigate the use of different technologies for learning by undergraduate students at two odl universities. the two universities were the universitat oberta de catalunya (uoc) in spain and the university of south africa (unisa) in south africa. both of these institutions are single mode (distance) institutions. these cases were selected on their basis of representativeness of odl students in each country and because the researchers obtained approval to collect data at these institutions. two cases were selected to understand any similarities or differences between the cases and provide a more robust and reliable study (stake 2006). a semi-structured interview protocol was designed for students to provide explanations and examples of how they use digital devices and other technologies for learning. appendix a contains the interview protocol. the data collection instrument was piloted with a small group of students at uoc for validation purposes. a random sample of students who had previously completed an electronic survey about their use of technology for learning at both institutions were invited via email http://dx.doi.org/10.25304/rlt.v27.2215� g. krull and j.m. duart 4 citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 (page number not for citation purpose) to participate in a telephonic or online semi-structured interview. the results of the surveys have not been published yet. the results of the surveys were used to develop the interview protocol and expand on the experiences of students using devices for learning. thirty-four interviews were conducted between july and december 2016. prospective interviewees were asked to sign consent forms before the interview. eighteen interviews were conducted with uoc students (case 1) and 16 interviews were conducted with unisa students (case 2). interviews were recorded and transcribed. interview analysis was done using a grounded theory approach (corbin and strauss 2015). a three-stage iterative process was followed for analysis: (1) an initial or open coding of interview data to identify relevant and frequent concepts, (2) the generation of conceptual categories and themes and (3) analysis of theoretical patterns and details. the researchers made use of nvivo pro version 11 to store and organise the data and compare codes. each case was first analysed separately before comparing the results of both cases. results the results from both universities are combined in this section. this is because although the two universities operate in different cultural and geographic contexts, the results of each case in this study were broadly similar. the major difference in results was related to internet access and technology adoption. while uoc is a completely online university, many programmes at unisa are offered via traditional distance methods (paper-based) with supplementary online support. the first section of the results analyses the data obtained regarding devices, learning locations and learning activities. the next section analyses the data regarding how students make use of multiple devices for learning. while each student makes use of the technologies they have access to for learning in a way that suits them, certain patterns of learning emerged. devices uoc students (n=18) had access to between three and five digital devices, while unisa students (n=16) had access to between two and four devices. in total, each interviewee had a smartphone, while all but one interviewee had access to a laptop. other devices included desktops, tablets and e-readers. all interviewees used at least one device regularly for learning, with the average being three devices used for learning. learning locations and activities interviewee data showed that digital devices were used in multiple locations for learning purposes, thus demonstrating the potential for seamless learning. these included private locations (such as at home) and public locations (such as at work, or intransit). the use of different devices means that students are able to study in different locations and at different times. the portability of the device influences the locations where it is used for learning: i use the laptop because of the convenience. i'm usually in the dining room [at home], but if my daughter wants to watch the tv, i go to the kitchen. that is, with the laptop i can easily change my location of where i am learning. (interview 1-10) http://dx.doi.org/10.25304/rlt.v27.2215� research in learning technology citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 5 (page number not for citation purpose) i use the smartphone when i'm out on public transport, during work breaks, at lunchtime, etc. i can continue studying on the public transport... for example, i can continue at the same point i stopped reading when i was at home. the mobile phone allows me to study anywhere anytime. (interview 1-13) when i’m somewhere waiting for the bus, i start reading on the ipad… sometimes when i’m travelling, i want to use my iphone, instead of ipad so i can receive a call, i can be on the internet, and also i can save some files, downloaded from the internet to my iphone and transfer it back to my computer or to my ipad. (interview 2-14). interviewee data also showed that students use their devices for similar and different learning activities. fixed devices, such as laptops and desktops, tend to be used for a wider variety of learning activities, such as reading study materials, watching videos, conducting research and doing assignments, whereas handheld devices, such as smartphones, tend to be used more for quickly checking information (such as results), although some students do use their phones to read materials or watch videos. these study habits are demonstrated in the following quotations: i do practically everything with the laptop and the pc, i do the assignments, i check the material and i do internet searches. i use them to print material for reading the material in depth. (interview 1-13) i basically use [the laptop] for doing my assignments mostly, because it’s easier to type on, it’s a lot easier to use. i use it for doing assignments, for going to the online portal and to obtain our textbooks. i use it for things like that. also looking at interactive videos that our lecturer sends us. (interview 2-02) using my phone, i check my results now and then on myunisa [the lms]. as well as download any other pdf files should i require them, to read them at any given time, so i have them on my phone. (interview 2-03) multiple device usage although students indicated that they mainly use one particular device for learning, they do sometimes use multiple devices together. students use devices together in one of two ways, either sequentially or simultaneously. sequential use refers to students starting a learning activity on one device and continuing or finishing the learning activity on a different device. simultaneous use refers to students using two or more devices at the same time to perform a learning activity. sequential use interviewees were asked to explain how they used their devices sequentially for learning. sixteen (89%) uoc students and 13 (81%) unisa students interviewed described how they move from one device to another. the main types of sequential usage were found to be students moving between different locations, students moving to a different (but related) learning activity, continuing the same learning activity at a later stage and moving between devices because of device features. table 1 illustrates these usage patterns together with student interview examples. http://dx.doi.org/10.25304/rlt.v27.2215� g. krull and j.m. duart 6 citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 (page number not for citation purpose) interviewees acknowledged that the reason they move from one device to another is for practical reasons, to make use of devices they have access to at that time or to take advantage of available time. occasionally, students may have to study without internet access and then later submit assignments when they have internet access again: when i am at the library working on my assignments without internet, what i do is work on my assignment at the library, afterwards, when i arrive home and make some corrections (if needed), i then submit my assignment through the uoc campus. (interview 1-03) table 1. sequential use patterns and examples. sequential patterns devices used interview examples move from one location to another, such as from work to home tablet/laptop -> laptop/desktop if i am working with my laptop or ipad outside home, once i arrive home and switch on the laptop or the home mac i would be able to see the same page in which i was working on when i was outside home. (interview 1-03) desktop -> laptop i’m looking at the time, maybe my assignment is due, i have to finish, so i will do part of it at work and as soon as i get home i continue where i left off. i do what i need to at home then i continue. (interview 2-01) move from one related learning activity to another, such as reading materials and then starting to write laptop/desktop -> tablet in my case i study law (court rulings and similar issues) so these documents are long to read, so what i usually do is i find it on the laptop or desktop pc and then i read it on the sofa with the tablet. (interview 1-07) smartphone -> laptop with, say something a little more complex to understand, i would move from my smartphone to my laptop. to research it further, to be able to open a lot of tabs or whatever. (interview 2-12) pause and continue the same learning activity at a later stage desktop -> laptop sometimes i come here to the desktop computer, work for a while… half an hour, three quarters of hour and then see that the progress has been almost none… and then i take the laptop and go to a different room. (interview 1-12) laptop -> tablet or smartphone having to work full-time and having a family, there are things you will need to do. i will always start with the mac, but i could finish on the ipad or iphone. i pretty much don’t have a preference between the two. (interview 2-09) move from one device to another because of device features (lack of battery or internet access) laptop -> tablet because usually i’m not using the computer for connection to the internet…to submit the assignment, i can take it from my laptop and transfer it to my ipad. from my ipad i can attach and send to unisa for electronic assignment submission. (interview 2-14) http://dx.doi.org/10.25304/rlt.v27.2215� research in learning technology citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 7 (page number not for citation purpose) simultaneous use interviewees were asked to explain how they used their devices simultaneously for learning. eleven (61%) uoc students and 11 (69%) unisa students interviewed described how they use two or more devices at the same time. the main activities performed simultaneously were found to be writing assignments together with searching for information or reading materials, watching a video and taking notes, writing an assignment and communicating with others and reading materials and searching for information. table 2 illustrates these usage patterns together with student interview examples. interviewees acknowledged that the reason they use different devices for learning at the same time is because of efficiency. students did not want to have to toggle between two windows on one computer, but wanted to be able to see the information on two screens at the same time. this means that students are able to undertake complementary activities in an easier way, as shown in the following quote: it’s the ability to be able to multitask. i can have the library online, i can have the textbook i need to work from, either in pdf or in paper in front of me, and then the assignment or the paper i am working on, on the computers in front of me. (interview 2-08) table 2. simultaneous use patterns and examples. simultaneous patterns devices used interview examples write assignment and search for information/read materials tablet + laptop an example would be to use the tablet as a device for reading (the subjects’ pdfs) and being able to answer the different questions in the assignments on the laptop. (interview 1-16) laptop + tablet + smartphone i would use the laptop and i would be typing up an essay. and i would use my tablet or my phone to either watch a video or look at the source documents. (interview 2-02) watch a video and take notes tablet + desktop if i'm watching a video i usually watch it on the tablet, then i use the computer for taking notes. (interview 1-04) laptop + tablet it’s mostly during exam periods. i actually sit and make the notes on the mac, while watching videos and tutorials on the ipad. (interview 2-09) write assignment and communicate with others tablet + desktop i have the forum open on the tablet, the messages from teachers or information that the teachers send us through the forum while i am doing the work with the computer. (interview 1-04) laptop + smartphone when i use them at the same time most of the time is when i’m doing assignments. when i have to look up something on the internet or i have to gather some information or i need to ask someone to help me explain what i don’t understand over the social media. then i do call them or i just send them a text and they help me out. (interview 2-16) read materials and search for information tablet + smartphone if i’m busy with the tablet, and i need an extra screen and i want to just cross-reference something that’s on the article that i’m currently reading, i can do it off the cell phone. it’s just to verify the data that’s on the one. (interview 2-11) http://dx.doi.org/10.25304/rlt.v27.2215� g. krull and j.m. duart 8 citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 (page number not for citation purpose) cloud services and synchronisation interviewees affirmed that moving between devices is facilitated by the synchronisation of files using cloud services. the cloud is the key enabler for switching between devices as synchronisation is automatic and students can continue exactly from where they left off. students make use of cloud services such as google drive or dropbox, as shown in the following quote: but always taking into account that i have everything in the cloud, the books are in the cloud, the continuous assessments. as i have it in google drive, i don’t mind from which device i’m accessing because everything is always at the point at which i left it before. (interview 1-06) however, for some students in south africa (the unisa case), internet access and cloud services are expensive or inaccessible. internet accessibility remains a challenge for some students who rely on mobile data. similarly, some students rely on older technologies such as memory drives or email to manually synchronise their work across different devices. these patterns are illustrated in the following quotes: i don’t have any other ways to access the internet. just on my phone. i just buy airtime bundles and convert them to data. it’s quite expensive. because now and then you need to download the files as well as watch youtube videos. (interview 2-03) so i’ll be running two simultaneous projects and i’ll either copy my data onto google drive or i’ll carry the data on a memory stick to the home computer. ( interview 2-15) discussion while students at the two different odl universities study in different contexts, several commonalities in the use of technologies for learning existed across both interview sets. the only real difference in study patterns was the limited access to (and relatively high cost of) quality internet services for several students at the south african university. although the use of one device at a time is more common, students at both universities make use of different devices at different times or locations to complete learning activities. more students used their devices sequentially than those who used multiple devices simultaneously. students move between devices for practical reasons, to make use of devices they have access to at that time or at that location to take advantage of available time for study. they also move between portable and fixed devices, depending on their location. this is because students can use a larger device when in a stationary environment such as home, but usually have their smartphone with them at all times to use in various locations. four patterns of sequential use for learning were identified: (1) students moving between different locations; (2) students moving to a different (but related) learning activity; (3) continuing the same learning activity at a later stage and (4) moving between devices because of device features. students also sometimes make use of multiple devices at the same time for learning to be more efficient. they are able to view ‘two screens’ at the same time to perform complementary activities. four patterns of simultaneous use for learning were identified: (1) writing assignments together with searching for information or reading materials; (2) watching a video and taking notes; (3)  writing an assignment and communicating with others and (4) reading materials and searching http://dx.doi.org/10.25304/rlt.v27.2215� research in learning technology citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 9 (page number not for citation purpose) for information. while few studies have explored students’ use of multiple devices, these findings match an australian campus university study about the use of tablets (reid and pechenkina 2016), which found that students tend to use their devices simultaneously and in a complementary manner, rather than using one device for all learning activities. a similar campus study of tablet usage found that students move between laptops and mobile phones based on the type of task that they want to perform (green et al. 2016). the movement between devices is facilitated by cloud services that facilitate automatic synchronisation. students are able to act on an idea with immediacy, make adjustments to assignments or record or share ideas wherever they are located (barden and bygroves 2017). useful tools for accessing and storing resources are the institutional lms, google drive and dropbox. while cloud services are useful for students, because of internet accessibility or cost issues, they are not used by all students, particularly in the south african context. some students rely on older technologies such as email or memory drives to move information between devices. the use of multiple devices (both fixed and handheld devices), supported by cloud services, has made it possible to achieve the aspiration of seamless learning. students are able to learn across physical places and virtual spaces. many students are able to learn in different contexts, but, more importantly, are able to maintain or continue their learning across contexts. however, this does mean a minority of students currently cannot yet learn seamlessly. the results from this study have led to the proposal of a continuum for seamless learners, as shown in figure 1. individual students may place themselves at a particular point along the continuum, rather than at one end. some students are able to more easily achieve seamless learning (seamless learners), while the learning of other students has clear ‘seams’ (discontinuous learners). while the use of multiple devices has positively affected the study habits of some students, the study habits of some students have been affected to a lesser extent. students who are able to learn more seamlessly typically make use of multiple devices for study and make use of automatic synchronisation and cloud services to easily move across devices, locations, times and learning activities. at this end of the continuum, quality and affordable internet access can be taken as a given for students. in addition to device access and connectivity, learner characteristics and preferences affect the degree of seamless learning. some students feel more comfortable using many different types of technologies for learning, while other students figure 1. continuum of seamless learners. http://dx.doi.org/10.25304/rlt.v27.2215� g. krull and j.m. duart 10 citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 (page number not for citation purpose) prefer to stick to the technologies they know. furthermore, seamless learners tend to study in a wider variety of locations and may undertake many different types of learning activities in these locations. at the other end of the continuum, discontinuous students are not able to learn seamlessly and move with great difficulty across devices, locations, times and learning activities. these learners typically have access to only one or two devices and internet access is intermittent or expensive. these students rely on older technologies (like memory drives) to manually synchronise across devices. they may also study in fewer locations or use their devices only for specific learning activities. however, more representative of odl learners is that their learning patterns are likely to exist at various points between the two ends of the continuum. furthermore, the position of each student will move along the continuum, depending on their location, the learning objective they intend to achieve and the device(s) and tool(s) they are using at a particular point in time. in supporting students, educators and odl universities need to recognise that students in a particular course may exist across this continuum and that students need to be supported in different ways. for example, some students may struggle with access to devices and connectivity. in supporting students, educators and odl universities need to recognise that students learn in different contexts and with various levels of technology use and need to be supported in different and personalised ways. these contexts include a variety of learning activities, devices, locations and times. therefore, educators may need to develop more flexible and effective pedagogies to support ‘anytime, anywhere’ learning (cross et al. 2019). conclusion this study explored how students’ use of multiple devices affects their learning patterns. universities are seeking insights into the study habits of students using different devices, particularly handheld devices, and the impact on learning (cross, sharples and healing 2016). the aim of this study was to explore how undergraduate odl students make use of different devices together, both fixed and handheld devices. the research study made use of a qualitative approach to interview students at two odl universities in two different contexts. odl students were found to sometimes use their digital devices together to be more efficient or take advantage of available devices or time. two patterns of using devices together for learning were explored, either sequential or simultaneous use. a continuum of seamless learners was proposed that can be used to determine how easily students are able to learn seamlessly, based on their contexts and practices. an area for future research is how educators can better support students using multiple devices and how to reduce any potential ‘seams’ in their learning experiences. this study has contributed towards the understanding of the educational value of technologies and how they support learning, specifically how students use multiple devices, together with supporting technologies, to support their learning. a greater understanding can inform improved educational practices (margaryan et al. 2011). references awadhiya, a. k. & miglani, a. (2016) ‘mobile learning: challenges for teachers of indian open universities’, journal of learning for development, vol. 3, no. 2, pp. 35–46. http://dx.doi.org/10.25304/rlt.v27.2215� research in learning technology citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 11 (page number not for citation purpose) barden, o. & bygroves, m. (2017) ‘“i wouldn’t be able to graduate if it wasn’t for my mobile phone.” the affordances of mobile devices in the construction of complex academic texts’, innovations in education and teaching international, vol. 55, no. 5, pp. 1–11. http://doi.org/ 10.1080/14703297.2017.1322996 chan, t.-w., et al., (2006) ‘one-to-one technology-enhanced learning: an opportunity for global research collaboration’, research and practice in technology enhanced learning, vol. 1, no. 1, pp. 3–29. http://doi.org/10.1142/s1793206806000032 corbin, j. & strauss, a. (2015) basics of qualitative research: techniques and procedures for developing grounded theory, 4th edn., sage, thousand oaks, ca. creswell, j. w. (2009) research design: qualitative, quantitative and mixed methods approaches, 3rd edn., sage, thousand oaks, ca. cross, s., sharples, m. & healing, g. (2016) ‘learning with mobile devices: the changing place and space of distance learners’ study’, edulearn16: proceedings of the 8th international conference on education and new learning, iated, barcelona, pp. 5385–5393. cross, s., et al., (2019) ‘distance learners’ use of handheld technologies’, the international review of research in open and distributed learning, vol. 20, no. 2, pp. 223–241. https:// doi.org/10.19173/irrodl.v20i2.4040 dolch, c. & zawacki-richter, o. (2018) ‘are students getting used to learning technology? changing media usage patterns of traditional and non-traditional students in higher education’, research in learning technology, vol. 26. https://doi.org/10.25304/rlt.v26.2038 farley, h., et al., (2015) ‘how do students use their mobile devices to support learning? a case study from an australian regional university’, journal of interactive media in education, vol. 2015, no. 1, pp. 1–13. http://doi.org/10.5334/jime.ar galanek, j. d., gierdowski, d. c. & brooks, d. c. (2018) ecar study of undergraduate students and information technology, ecar, louisville, co [online]. available at: https:// library.educause.edu/resources/2018/10/2018-students-and-technology-research-study google. (2012) ‘the new multi-screen world study’, google think insights [online]. available at: http://www.thinkwithgoogle.com/research-studies/the-new-multi-screen-world-study.html green, d., et al., (2016) ‘tablets@university: the ownership and use of tablet devices by students’, australasian journal of educational technology, vol. 32, no. 3, pp. 50–64. http:// doi.org/10.14742/ajet.2195 henderson, m., et al., (2015) ‘students’ everyday engagement with digital technology in university: exploring patterns of use and “usefulness”’, journal of higher education policy and management, vol. 37, no. 3, pp. 308–319. http://doi.org/10.1080/1360080x.2015.1034424 krull, g. e. & duart, j. m. (2017) ‘research trends in mobile learning in higher education: a systematic review of articles (2011 – 2015)’, the international review of open and distributed learning, vol. 18, no. 7, pp. 1–23. http://dx.doi.org/10.19173/irrodl.v18i7.2893. margaryan, a., littlejohn, a. & vojt, g. (2011) ‘are digital natives a myth or reality? university students’ use of digital technologies’, computers and education, vol. 56, no. 2, pp. 429–440. http://doi.org/10.1016/j.compedu.2010.09.004 newman, t. & beetham, h. (2017) student digital experience tracker 2017: the voice of 22,000 uk learners, jisc [online]. available at: http://repository.jisc.ac.uk/6662/1/ jiscdigitalstudenttracker2017.pdf newman, t., beetham, h. & knight, s. (2018) digital experience insights survey 2018: findings from students in uk further and higher education, jisc [online]. available at: https:// digitalinsights.jisc.ac.uk/news/digital-experience-insights-survey-2018-findingsstudentsuk-further-and-higher-education/ patterson, m.c. (2017) ‘a naturalistic investigation of media multitasking while studying and the effects on exam performance’, teaching of psychology, vol. 44, no. 1, pp. 51–57. http:// doi.org/10.1177/0098628316677913 reid, d. & pechenkina, e. (2016) ‘bring-your-own-device or prescribed mobile technology? investigating student device preferences for mobile learning’, in mobile learning futures: sustaining quality research and practice in mobile learning, 15th world conference on http://dx.doi.org/10.25304/rlt.v27.2215� http://doi.org/10.1080/14703297.2017.1322996 http://doi.org/10.1080/14703297.2017.1322996 http://doi.org/10.1142/s1793206806000032� https://doi.org/10.19173/irrodl.v20i2.4040 https://doi.org/10.19173/irrodl.v20i2.4040 https://doi.org/10.25304/rlt.v26.2038 http://doi.org/10.5334/jime.ar� https://library.educause.edu/resources/2018/10/2018-students-and-technology-research-study� https://library.educause.edu/resources/2018/10/2018-students-and-technology-research-study� http://www.thinkwithgoogle.com/research-studies/the-new-multi-screen-world-study.html� http://doi.org/10.14742/ajet.2195� http://doi.org/10.14742/ajet.2195� http://doi.org/10.1080/1360080x.2015.1034424� http://dx.doi.org/10.19173/irrodl.v18i7.2893 http://doi.org/10.1016/j.compedu.2010.09.004� http://repository.jisc.ac.uk/6662/1/jiscdigitalstudenttracker2017.pdf� http://repository.jisc.ac.uk/6662/1/jiscdigitalstudenttracker2017.pdf� https://digitalinsights.jisc.ac.uk/news/digital-experience-insights-survey-2018-findings-students-uk-further-and-higher-education/� https://digitalinsights.jisc.ac.uk/news/digital-experience-insights-survey-2018-findings-students-uk-further-and-higher-education/� https://digitalinsights.jisc.ac.uk/news/digital-experience-insights-survey-2018-findings-students-uk-further-and-higher-education/� http://doi.org/10.1177/0098628316677913� http://doi.org/10.1177/0098628316677913� g. krull and j.m. duart 12 citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 (page number not for citation purpose) mobile and contextual learning, eds l. e. dyson, j. fergusson & w. ng, iamlearn and the university of technology, sydney, pp. 218–225. sharples, m., et al., (2012) innovating pedagogy 2012: open university innovation report 1, the open university, milton keynes. stake, r.e. (2006) multiple case study analysis, guilford press, new york. wong, l.-h. (2012) ‘a learner-centric view of mobile seamless learning’, british journal of educational technology, vol. 43, no. 1, pp. 19–23. http://doi.org/10.1111/j.1467-8535.2011. 01245.x http://dx.doi.org/10.25304/rlt.v27.2215� http://doi.org/10.1111/j.1467-8535.2011.01245.x� http://doi.org/10.1111/j.1467-8535.2011.01245.x� research in learning technology citation: research in learning technology 2019, 27: 2215 http://dx.doi.org/10.25304/rlt.v27.2215 13 (page number not for citation purpose) appendix a: interview protocol student code: interview date/time: interviewer: interview tool: 1. which digital devices do you have access to? 2. use of a desktop or laptop: a. if you learn with a desktop or laptop, please give me one or more examples of how you use it for your learning? b. if you have it, but do not use it for learning, why not? 3. use of a mobile phone or tablet: a. if you learn with a mobile phone or tablet, please give me one or more examples of how you use it for your learning/? b. if you have it, but do not use it for learning, why not? 4. in what contexts (places) do you use your different devices? 5. how do you access the internet for your studies? 6. if you use more than one device for learning, can you explain why you choose a specific device to perform learning tasks? 7. do you ever move from one device to a different to complete a learning activity? if so, please provide examples. 8. do you ever complete a learning activity using two or more devices at the same time? if so, please provide examples. 9. when you have a challenge or issue completing a learning task or while studying, what do you do? 10. when you have a technical problem related to your studies, what do you do? 11. what do you see as the most important tools, technologies or services for your learning? 12. if you think about your study habits, how have they changed since you started using the different devices that you have available to you? 13. any general comments? http://dx.doi.org/10.25304/rlt.v27.2215� doi:10.3402/rlt.v24.31485 letter to editor responding to ‘nurturing global collaboration and networked learning in higher education’ sir, i read with great interest the case study on icollab reported by cronin, cochrane and gordon (2016), which was recently published in your esteemed journal. the authors created the icollab for the purpose of networked learning and they claimed that icollab is based on the principles of community of practice (cop) proposed by lave and wenger (1991), wenger (1998) and wenger, mcdermott and snyder (2002). in associating cop with icollab, the authors talk about the ‘concepts of boundary crossing and brokering’ (p. 4). other important characteristics of cop, especially the ‘membership’, the roles of ‘old-timer and newcomers’ and the ‘legitimate peripheral participation (lpp)’, are missing in the discussion. in fact, these are amongst the defining characteristics of a cop which distinguishes it from other sorts of affiliations. as highlighted by gee (2005, p. 214), even though wenger (1998) ‘has tried to be careful in delineating just what is and is not a community of practice’, it has been used by others ‘to cover such a wide array of social forms that [they] may be missing the trees for the forest’. whilst i acknowledge that icollab is an innovative approach to learning and a practical model to be applied in other parts of the world, especially in southeast asia, i was wondering whether icollab holds the true spirit of a cop as the paper does not discuss how icollab meets the defining characteristics of cop. there are several studies which reveal that adopting the notion of cop to conceptualise a group of networked users is problematic (e.g. harris and shelswell 2005; pellicone and ahn 2014; taylor 2014). taylor for instance suggests that in networked environment, learning does not always involve newcomers observing and interacting with old-timers but could be the other way round, as it is common to have expert newcomers and novice old-timers amongst the many networked learners. my own phd research carried out at the university of nottingham, uk, on teachers’ informal learning on social networking sites (see rashid 2015) supports taylor’s argument where i found that when teachers introduce topics to seek teaching-related knowledge, they pose their questions to all the community members, including newcomers and old-timers (however defined), and even if they specifically address their questions to expert old-timers or expert newcomers, other members of the community still respond to them, which reflects that learning on social networking sites occurs in a more complex way than lpp. gee’s (2005) concept of affinity space, which emphasises different levels of involvement and flexible forms of participation amongst a group of networked individuals in a particular space seems to be more useful than lave and wenger’s (1991) research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 r. ab rashid. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 31485 http://dx.doi.org/10.3402/rlt.v24.31485 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31485 http://dx.doi.org/10.3402/rlt.v24.31485 cop for explaining the icollab project. this is not surprising as lave and wenger put forward the concept of cop based on their observations of offline communities, whereas gee’s concept of affinity space is based on his observations of online activities. unlike lave and wenger who give structure to how learning takes place in the community through lpp, gee is of the view that learning can occur in any way ‘through the joint action with advanced peers’ (p. 216) who might be newcomers or old-timers in the space. since the focus of the icollab project is the collaboration amongst the users instead of the ‘membership’ of the users, i strongly recommend that the authors consider gee’s contribution of affinity space in making sense of the activities that take place in icollab. radzuwan ab rashid school of english language studies faculty of languages and communication universiti sultan zainal abidin (unisza), malaysia email: radzuwanrashid@unisza.edu.my biography radzuwan ab rashid is a senior lecturer in school of english language studies, faculty of languages and communication, universiti sultan zainal abidin (unisza), malaysia. he is also the coordinator for master’s programmes at unisza graduate school. he has an educational background in teaching english as a second language (tesl) and has received his phd degree from the university of nottingham, uk (2015). his doctoral dissertation exploring teachers’ co-construction of social support on a social networking site was a discourse analytic exploration of displays of support in social interaction. he has published papers related to tesl and discourse analysis in several reputable journals including discourse studies, international journal of technology enhanced learning, mextesol journal and english language teaching. references cronin, c., cochrane, t. & gordon, a. (2016) ‘nurturing global collaboration and networked learning in higher education’, research in learning technology, vol. 24, 26497, doi: http:// dx.doi.org/10.3402/rlt.v24.26497 gee, j. p. (2005) ‘semiotic social spaces and affinity spaces: from the age of mythology to today’s schools’, in beyond communities of practice: language, power and social context, eds. b. dave & k. tusting, cambridge university press, cambridge, pp. 214�232. harris, s. & shelswell, n. (2005) ‘moving beyond communities of practice in adult basic education’, in beyond communities of practice: language, power and social context, eds. b. dave & k. tusting, cambridge university press, cambridge, pp. 158�179. lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation, cambridge university press, cambridge. pellicone, a. & ahn, j. (2014) ‘construction and community: investigating interaction in a minecraft affinity space’, 10th gls conference proceedings, [online] available at: http:// ahnjune.com/wp-content/uploads/2014/05/pellicone-ahn-gls-final.pdf rashid, r. a. (2015) teachers’ co-construction of social support on a social networking site, phd thesis, the university of nottingham, uk. r. ab rashid 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31485 http://dx.doi.org/10.3402/rlt.v24.31485 http://dx.doi.org/10.3402/rlt.v24.26497 http://dx.doi.org/10.3402/rlt.v24.26497 http://ahnjune.com/wp-content/uploads/2014/05/pellicone-ahn-gls-final.pdf http://ahnjune.com/wp-content/uploads/2014/05/pellicone-ahn-gls-final.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31485 http://dx.doi.org/10.3402/rlt.v24.31485 taylor, p. (2014) ‘mother tongue and identity in a thai esp classroom: a communities-ofpractice perspective’, language education and acquisition research network (learn) journal, vol. 7, no. 1, pp. 76�90. wenger, e. (1998) communities of practice: learning, meaning, and identity, cambridge university press, cambridge. wenger, e., mcdermott, r. & snyder, w. (2002) cultivating communities of practice: a guide to managing knowledge, harvard business school press, boston, ma. research in learning technology citation: research in learning technology 2016, 24: 31485 http://dx.doi.org/10.3402/rlt.v24.31485 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31485 http://dx.doi.org/10.3402/rlt.v24.31485 doi:10.3402/rlt.v20i0.16149 teaching with wikis: improving staff development through action research robyn benson a *, charlotte brack a , and gayani samarwickrema b a faculty of medicine, nursing and health sciences, monash university, melbourne, australia; b faculty of health, deakin university, melbourne, australia (received 17 may 2011; final version received 30 september 2011) this paper reports on the use of action research in a case study involving two iterations of an online workshop implemented at two universities in late 2007 and early 2009 to prepare teaching staff for using wikis for student group work and assessment. workshop participants were immersed in the experience of collaborating in a wiki as learners and then reflected on this experience as teachers. experience of the pilot workshop suggested a need for more orientation, potentially by introducing a blended learning design. the second iteration highlighted a need to develop the orientation session further and increase support strategies throughout the workshop, suggesting the value of offering it at faculty or department level if no ‘‘reward’’ is available for participation. outcomes from the two cycles illustrate the value of action research for iterative improvement of this staff development model and for implementing the scholarship of teaching and learning to develop and share professional knowledge in this emerging area. this paper outlines a staff development approach involving web 2.0 applications on which others can build. keywords: action research; scholarship of teaching and learning; staff development; social software; web 2.0 introduction this paper draws on the idea of the scholarship of teaching and learning as an action research process with rigour achieved through responsiveness to practice (schön 1995). we used this process to drive the design, implementation and improvement of an online staff development model to prepare teaching staff at two australian universities for using wikis for student group work and assessment. our aim was to use action research to refine, iteratively develop and evaluate the workshop model and make our findings available for use by others. this was based on the assumption that identifying approaches for assisting staff to teach with emerging technologies is an important aspect of enquiry to counter the typically slow adoption of educational technologies by academics (birch and burnett 2009). it also supports the concept of teachers becoming reflective practitioners to capture and reuse the pedagogic forms they create when using learning technologies, so that exploration of innovative pedagogy ‘‘take[s] on the character of research � a more rigorous way of developing our knowledge in this critically important field’’ (laurillard 2008, p. 153). (page number not for citation purpose) *corresponding author. email: robyn.benson@monash.edu research in learning technology vol. 20, 2012 rlt 2012. # 2012 r. benson et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 our specific research questions were the following: (1) how effective is this online staff development model in helping staff to use wikis for student group work and assessment? (2) what implications for improving the model emerged over two action research cycles? (3) is action research a useful approach for the iterative development of this workshop? a further aim of this paper is to demonstrate how promoting change in specific situations through action research (kember and kelly 1993) can contribute more broadly to building knowledge through the scholarship of teaching and learning. while understandings of what the scholarship of teaching and learning involves vary (e.g., boshier 2009; brew 2006; norton 2009), we characterise it as a form of enquiry (hutchings and shulman 1999) where knowledge about teaching is renewed when teachers’ work ‘‘becomes public, peer-reviewed and critiqued, and exchanged with other members of our professional communities so they, in turn, can build on our work’’ (shulman 2000, p. 49). norton (2009) suggests that pedagogical action research is a compelling way of engaging with the scholarship of teaching and learning. the emergent nature of knowledge about e-learning appears to lend itself to the rapid sharing of experiences captured through such processes, so that advances in pedagogy do not lag behind advances in technology. we begin by outlining the potential implications for teaching and learning of web 2.0 applications such as wikis. we then consider these in the context of staff development approaches for online teaching, before focusing on the use of action research for informing staff development. we explain how action research was used iteratively in our case study to prepare staff for teaching with wikis. we then report on the knowledge gained from this experience and discuss its implications. conceptual background web 2.0 applications the ability of web 2.0 applications to support online group interaction offers important benefits for teaching and learning. as one of these applications, wikis provide an online space where students can write, edit and construct knowledge together, taking greater responsibility for and control of their learning. these features present new challenges in designing learning and assessment tasks that accommodate the ‘‘wisdom of crowds’’ (surowiecki 2004). this university 2.0 environment (barnes and tynan 2007) requires teachers to relinquish control and permit organic and emergent structures to arise from group interactions (dron 2007). it also offers potential for enquiry into teaching and learning, which focuses on how teaching staff can support learning in this setting and on how students build knowledge through virtual collaboration (e.g., guo and stevens 2011; ruth and houghton 2009; su and beaumont 2010). staff development for online teaching an underlying challenge for teaching staff is to learn to use these new tools and apply them in pedagogically effective ways. wilson and stacey (2004) reviewed programs for r. benson et al. 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 preparing staff for online teaching, emphasising the importance of situating learning activities in authentic contexts and offering opportunities for staff to share experiences, ideas and reflections as they engage as learners. doherty (2011) supports the value of a project-based approach for web 2.0 staff development. situated learning allows participants to experience the complexity and ambiguity of real world challenges, embedding learning in the social context in which it will be used (brown, collins, and duguid 1989). using the social context to reflect on learning highlights the value of communities of practice where groups of people with common interests ‘‘deepen their knowledge and expertise . . . by interacting on an ongoing basis’’ (wenger, mcdermott, and snyder 2002, p. 4). this approach draws on social constructivism (vygotsky 1978) for explaining social engagement online. garrison and anderson (2003) describe how teachers and learners may engage in an educational community of enquiry, encouraging both cognitive independence and social interdependence. these ideas underpinned our workshop design while action research guided its continuing improvement. action research action research offers many benefits for enquiry into teaching and is an established model for staff development in higher education (grundy 1995; kember and gow 1992; webb 1996). its cyclical character, involving planning, action, observation and reflection, supports critically reflective thinking about one’s own practice, is grounded in the use of collaboration to forge new meanings from experience and offers a clear framework for action (brookfield 1995; carr and kemmis 1986; kemmis and mctaggart 2005). its iterative nature, facilitated by reflection, appears appropriate to the evolving state of knowledge in the area, allowing for incremental accumulation and sharing of new ways to enhance student learning. action research typically begins with a pedagogical problem, paradox, issue or difficulty (norton 2009), which invites enquiry and action by practitioners to address a concern that relates immediately to their teaching practice. it is usually associated with interpretive research methods and informed by a strong emphasis on the social, reflective and emancipatory aspects of critical social theory (carr and kemmis 1986). implementation of the process is seen in different ways. for example, norton (2009) regards the identification of the issue to be investigated as observation, which is then followed by planning, action and reflection. in contrast, kember and kelly (1993) describe the plan-act-observe-reflect sequence based on the seminal work of lewin (1946), providing examples that begin with initial reflection to commence the planning process. we adopted the latter approach but began with the planning phase. the pedagogical issue we were addressing was the need to explore ways to assist academic staff to use wikis for teaching and assessment. we planned to use an interpretive method, not only observing teaching staff engage in an immersive experience working in a wiki as students (the action) and recording our observations as an aspect of research but also including evaluation by participants as a research component. critical reflection occurred initially with participants and then among ourselves as workshop designers and developers. hence, the participatory nature of action research offered a way of conceptualising the investigative aspects of this project, informing ongoing staff development and, potentially, establishing a basis for future collaboration with workshop participants to generate new understandings about teaching with wikis. research in learning technology citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 the following sections outline how we used action research for planning, implementation and improvement. the method and results for each cycle are presented separately to demonstrate how the first cycle influenced the second in a process of iterative development. we then discuss implications from our experience of these two cycles. method (cycle 1) a description of the first cycle (samarawickrema, benson, and brack 2008, 2010) is summarised here to provide a basis for explaining the modifications made during cycle 2. planning a key intention was that participants would work in two groups, each consisting of participants from both universities. as staff members of these universities we worked together in this project to address a pedagogical issue that was relevant to both institutions in exploring approaches for facilitating staff development in the use of new technologies. since participants would be immersed in the experience of collaborating in a wiki as learners and then reflect on this experience as teachers, the workshop design consisted of two main stages: task completion (two weeks) and a debriefing phase over one week representing the reflection part of the action cycle. a primary workshop wiki called wikis in higher education would provide the following: (1) resources about using wikis in higher education; (2) a workshop task; (3) guidelines on working in a wiki; and (4) links to two workshop wikis that participants would develop as ‘‘students.’’ the workshop task, designed to resemble a student group project, required an outcome that would allow assessment of both a group product and individual contributions. the task required participants to identify a student group project that could be undertaken in a wiki and then collaboratively develop a set of guidelines for their students, using the features of the wiki to present these guidelines. during the reflection phase, participants would be given access to the wiki of the other group and be asked to assess the group product and individual contributions to it. the wikis in higher education site was available for interaction between participants and facilitators, but the workshop wikis were to ‘‘belong’’ to group members (dron 2007) and have no facilitator involvement. participants would not have access to the wiki belonging to the other group until debriefing. our role as facilitators during task completion would be to observe, answer questions and offer guidance via discussion on the wikis in higher education site. we would then take a more proactive role during debriefing to facilitate feedback and reflection, summarise participant responses and share facilitator reflections. action at university a (ua), invitations to participate were sent to staff from a single faculty selected for their prior involvement or interest in online learning, whereas an r. benson et al. 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 open invitation was sent to all staff at university b (ub). thirteen volunteered (six from ua and seven from ub). we divided participants into two groups, giving group 1 access to workshop wiki 1 and group 2 access to workshop wiki 2. we began the workshop by emailing participants with instructions for logging into the wikis in higher education site, inviting them to introduce themselves to other participants, identify their groups and locate their workshop wiki to complete the task over the following two weeks. there were delays in group formation and task completion, and because participants did not return to the wikis in higher education site for support as anticipated, we took a more proactive role, clarifying requirements on the wikis in higher education site, sending group and individual emails to encourage and support participants and offering some guidance in the workshop wikis. observation we monitored the sites daily, observing task progress, which resulted in the unplanned actions outlined above. we recorded our daily findings in an online diary. reflection in the final week, we opened the workshop wikis to all participants and asked them to return to wikis in higher education for debriefing, requesting them to: (1) critique the other group’s wiki and comment on the content, group effort and individual efforts; (2) reflect and comment on the experience of working collaboratively in their wiki, identifying what they had learnt and ideas for using wikis in teaching and learning; and (3) evaluate the workshop itself. results (cycle 1) task progress task progress was slow. by the end of week 1, four participants had accessed each workshop wiki but minimal progress had been made in workshop wiki 2. at the end of week 2, five participants had accessed workshop wiki 1 and six had accessed workshop wiki 2. workshop wiki 1 was more advanced with three wiki pages (two of them incomplete). group 2 created only one page. the debrief critique workshop wiki 1 participants commented that workshop wiki 2 was poorly proof read, was confined to the main page, lacked focus on the task and required better content organisation, noting that participants spent too much time on the task context rather than the task itself. in contrast, workshop wiki 2 participants were impressed with the group effort, content and navigation in workshop wiki 1. research in learning technology citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 reflection eight participants contributed reflections, suggesting that the workshop helped them to contextualise how to use wikis and introduce them to students. they felt they needed more time for discussion, planning, familiarisation with wiki functions, group formation and editing others’ work. evaluation five participants contributed evaluation comments. the experience of participation was regarded as the most useful aspect but they expressed concerns about technical issues related to logging in, lack of familiarity with the wiki environment, insufficient time for the task and for group formation and a sense of needing more guidance. our reflections we considered the basic workshop design to be appropriate, including the aims, objectives and structure. however, collaborative engagement between participants was inhibited by their previous level of technology adoption, unfamiliarity with the wiki environment, lack of time and technical issues, indicating a need for more orientation and support in both mastering the wiki tools and conceptualising engagement with others to create a group output. we also noted that the lack of ‘‘reward’’ for participation reduced the imperative to complete the task, unlike assessable student group work. method (cycle 2) planning based on our experience, we planned a second workshop with a similar design but modified as follows: (1) a face-to-face orientation session was added to clarify workshop aims and expectations and fast track group formation and interaction in the wiki. (2) the wikis in higher education site was replaced by a blackboard site containing resources, instructions, discussion spaces and entry to the workshop wikis. as blackboard was a familiar environment for participants, we expected improved access to and engagement in the wikis. (3) two parallel workshops were organised (one at each university), each with two wiki groups (i.e., a total of four groups). we expected participants to feel more comfortable working with colleagues from their own university. we planned to give participants access to all four wikis during debriefing and ask them to select one to critique, using an assessment rubric, which we would provide. (4) workshop duration was shortened to 10 days: eight for task completion and two for debriefing. although pilot workshop participants had expressed the need for more time, the challenge of competing work pressures suggested that task completion might be assisted by maintaining momentum over a shorter time. we expected that the orientation session would address some of the issues that prompted a sense of insufficient time in the first workshop. r. benson et al. 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 (5) we advertised the workshops at both universities, offering a maximum of 12 places at each, with six participants per workshop group representing a manageable size. although there would still be no reward for participation, we expected greater motivation to complete the workshop from self-initiated involvement. again (because of the facilitators’ institutional roles), participants were from a single faculty at ua but across the faculties at ub. action nine participants registered at ua and 12 at ub. at the face-to-face orientation sessions at each university, participants practised using wiki discussion, history and editing functions and began group formation by planning roles and responsibilities, communication strategies and timelines and brainstorming project ideas. as facilitators, we participated in both sessions. following each session, we asked participants to complete an evaluation survey on their blackboard site. participants then engaged in the online workshop task using similar instructions to those provided in the pilot workshop. we used the blackboard mail and discussion tools to contact participants, answer queries, provide advice and offer reminders. at ua, two wiki server malfunctions interrupted progress, requiring explanatory communication via blackboard. ua group 1 members also used the discussion tool to contact each other and schedule their wiki contributions. at ub, group emails were sent to encourage and support participants, prompt completion of orientation session feedback, remind them of the impending workshop closure date, announce the start of the debriefing phase and reiterate debrief expectations. observation as in the pilot workshop, we observed task progress daily throughout the action phase. reflection for the final two days, we gave participants access to all four wikis and asked them to critique a wiki from either university (using the provided rubric for assessment of individual and group contributions and content). this was followed by reflection and evaluation as before, using tools provided on their blackboard sites. results (cycle 2) orientation session all ua participants attended an orientation session, except one who was located at an overseas campus and supported by telephone. two ua participants attended the ub session because of scheduling issues, joining nine of the 12 ub participants. table 1 summarises participants’ survey responses, with multiple choice items scored from strongly agree (5) to strongly disagree (1) on a 5-point scale. these items were completed by the six participants at the ua session and the overseas-based participant. seven of the 11 ub attendees completed the survey, two of whom could have been the ua participants attending that session. research in learning technology citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 results indicated moderate to high levels of satisfaction with the orientation session. ua participants were most confident about accessing support (4.57) while ub participants were most confident in recognising how a group project in a wiki could be assessed (4.00). participants were also asked about changes that would have improved their preparation for the online workshop and for comments about any other aspects of the session. ua comments indicated interest in the potential of wikis and the logistics of setting them up and suggested an increased timeframe for the session. ub suggestions included running a prior technical workshop on wikis, providing printed (rather than online) help on creating wiki pages and having only one orientation session presenter with other facilitators assisting. however, comments indicated that ‘‘[i]t was well set out’’ and that ‘‘[t]he session was very enjoyable and . . . [has] great possibilities.’’ task progress eight of the ua participants contributed to wiki development, four in each group. each group created eight wiki pages, but while group 2 populated all, group 1 populated five of their pages. ub participants were divided into two groups of six. five were active in one wiki, creating six pages, and three in the other, creating three pages. table 2 summarises the number of page edits and discussion contributions by each participant. table 1. responses to multiple choice items from the orientation session evaluation surveys. multiple choice items ua mean n �7 ub mean n �7 (1) i am clear about what is expected of me in the online workshop project. 4.14 3.14 (2) i am confident about using a wiki for project discussion, editing and viewing the history of the project. 3.14 3.71 (3) i am confident about working with members of my group in the workshop project. 4.14 3.71 (4) i can recognise how a group project in a wiki could be assessed (including assessment of process and product, and of individual and group contributions). 3.71 4.00 (5) i know how to access support during the project if i need it. 4.57 3.71 table 2. edits and discussion contributions per participant. university a university b group 1 group 2 group 1 group 2 participant edits discussion edits discussion edits discussion edits discussion 1 69 06 36 13 03 00 05 03 2 51 03 30 08 18 06 11 04 3 02 01 65 10 12 04 01 05 4 15 02 05 06 02 02 00 03 5 36 09 00 01 6 00 02 total 137 12 136 37 71 23 17 16 r. benson et al. 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 figures 1 and 2 show concentrated activity at the start and end of the workshop period. discussions for all groups were most active on day 1 (figure 1), although ua group 2 made an equal number on day 8. editing activity was generally low to moderate in the early days, increasing on day 8 (figure 2). there was no ua activity on day 4 because of the server malfunction. figure 1. distribution of discussion posts over time. figure 2. distribution of edits over time. research in learning technology citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 the debrief critique though the critique phase at ua was impeded by another server malfunction, two assessments of the group 1 wiki and one of the group 2 wiki were received, using the rubric provided. comments on the group 1 wiki acknowledged contributions to group organisation and output but critiqued it for the incomplete effort. structure, navigation and layout were commended. the single group 2 wiki assessment noted that two people had emerged as leaders and praised the communication effort and examples of individuals supporting each other. accuracy, completeness and presentation were commended, with reservations expressed about lack of succinctness and use of animated graphics. all three assessments competently graded both the group effort and individual contributions. at ub, two assessments of the group 2 wiki were received but group 1 received no assessment from group 2. as indicated in table 2, the ub group 2 wiki had three contributors and was the least developed. consequently, remarks on group work commended its early discussion and planning and critiqued it for not following plans through. the specific efforts of the two main group 2 contributors were recognised. reflection seven ua and four ub participants contributed reflections. overall, ua responses were more positive, though limitations of using wikis were expressed by participants from both universities. participants’ comments on the orientation, the task and benefits and limitations of wikis for teaching and learning are summarised below. orientation. all ua respondents regarded the orientation session as important, valuing and enjoying the experience for understanding online possibilities and initiating group work. while ub participants found the session useful, they noted the need for more time to get to know each other, ‘‘overcome the getting started barriers,’’ and become ‘‘conversant with the technicalities of wikis.’’ the task. three ua participants commented positively on the wiki task, using terms such as: ‘‘very enlightening’’ and ‘‘interesting.’’ one found it an ‘‘interesting, irritating, enjoyable, boring, creative activity (almost all at once . . .).’’ further comments included: ‘‘i was amazed at the different (not better or worse just different) ways that the . . . [task was] conceptualized’’ and ‘‘i would like to try using wikis with my students, but feel i am still in need of some support and advice.’’ participants identified potential uses of wikis for: (1) ‘‘[providing] a repository of information which they [students] could access when out on fieldwork’’; (2) ‘‘distance education, allowing group members to collaborate and learn’’; (3) facilitating problem-based learning; (4) ‘‘around the workplace’’; and (5) ‘‘working with academics/clinicians to collaborate.’’ all four ub participants who commented stated that the limited workshop time affected their level of contribution. suggestions included providing a technical session as a workshop prerequisite and appointing a leader to ensure ‘‘that all r. benson et al. 10 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 members participated appropriately.’’ one suggested that ‘‘a range of exemplars would be useful’’ and perhaps some wiki templates for outputs such as research reports and presentations and group assignments. benefits and limitations. participants identified more limitations than benefits of using wikis for teaching and learning (table 3), three ua respondents referring to their age as an inhibiting factor. evaluation an evaluation survey on each blackboard site asked participants what they found most useful about the workshop, what they would change and provided space for additional comments. six ua and four ub participants responded. most useful aspects. all ua respondents appreciated the opportunity to work with others as students in a wiki and learn about their potential for collaboration. also appreciated were the supportive, continual input of the facilitators, the opportunity to become familiar with another online system, the support resources and the faceto-face session. experiencing technical glitches was viewed as a positive learning experience. ub participants appreciated understanding the difficulties of working as a group online under time pressure, learning to use design components within the wiki and ‘‘[g]etting a clear view of a working wiki.’’ they also mentioned the support resources and workshop task in helping to develop conceptions of how wikis can be useful in learning and teaching. suggested changes. two ua participants advised extending the face-to-face session for practising while another recommended allowing more time to get to know group members and set roles and responsibilities. other ideas included having fewer presenters and starting the session by explaining the potential and limitations of table 3. some perceived benefits and limitations of using wikis in teaching and learning. benefits limitations (1) enhancing student ownership of learning objectives (ua) (2) reducing (or eliminating) the need to circulate emails (ua) (3) monitoring and evaluating student input into group work (ua) (4) avoiding use of track changes for collaboration (ua) (5) sharing information (ua) (6) facilitating tracking of individual contributions (ua, ub) (7) collaborative possibilities (ub) (1) losing contributions when group members are editing simultaneously (ua) (2) technology failures (ua) (3) trawling through pages for assessment cumbersome (ua) (4) managing multiple groups potentially difficult (ua) (5) engaging students potentially problematic (ua) (6) software lacks user-friendliness � technical knowledge required (ua and ub) (7) working collaboratively with unknown people difficult (ua and ub) (8) learning new skills requires effort (ua and ub) (9) making a regular time commitment (ua and ub) (10) relying upon collaboration of others (ub) (11) varying levels of participation (ub) research in learning technology citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 wikis, followed by hands-on help in operating the application. to accommodate participants based at several campuses, videoconferencing the session was suggested. there was one request for tips to set up and manage student accounts and monitor multiple groups and edits. ub participants also wanted a longer face-to-face session that incorporated the technicalities of operating wikis. suggestions included having one instructor present the training while the others gave hands-on help (reiterating a point made in the orientation session evaluation) and commencing the workshop task at the face-toface session ‘‘so that the projects run smoother during the online components.’’ the short workshop duration gave little opportunity ‘‘to do the norming, storming, and performing parts of teamwork, and if people are busy . . . it’s difficult to get into the task.’’ they requested a longer (three week) workshop starting earlier in the year, fast and functional pc lab computers for the orientation, and the ability for participants to send email pop-ups to group members to advise site changes. additional comments. ua participants enjoyed the experience and appreciated the work involved and the difficulty of meeting the varied agendas of participants. one commented that it was a ‘‘great way to learn,’’ including recognising the potential of technical problems and the time commitment required. ub participants reiterated that more time was needed to respond to the task and provide worthwhile feedback to others. one suggested that potential workshop participants be told that success depends on collaboration of all participants and that they should not register unless they had the time required. participants from both universities were interested to see the wikis from the other university. our reflections we noted the orientation session as a valuable design addition that facilitated group formation and technology familiarisation but recognised that both aspects required further refinement. the additional support in group formation allowed participants to be more active in their wikis earlier and more effectively than in the first cycle, despite the hesitant participation by some members. nevertheless, after initial orientation difficulties, the collaboration of those who participated did not appear to be inhibited by the technology but the requests for more technical help indicated their need for greater confidence. through critical reflection and dialogue (carr and kemmis 1986), we concluded that lengthening the workshop would not increase participation, given the variability of contributions by participants. the importance of technical stability was illustrated by the experience of ua participants, with adverse experiences undermining the momentum of the workshop, their perceptions about the potential of wikis and their willingness to use them for teaching. discussion group interaction, task progress and group output showed marked improvement over the pilot workshop in the second cycle (despite variations between groups), suggesting the value of introducing the orientation session. using the blackboard sites for running the workshop was more successful than using another wiki because it distinguished between support via blackboard and the sites for the workshop task. however, shortening the workshop and recruiting participants by advertising did r. benson et al. 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 not appear to assist in increasing commitment to the workshop by all participants. this highlighted the perceived lack of reward for participation, which also affected the pilot workshop and seemed to be a particular problem for ub participants. some key implications for further development of the program are outlined below. the orientation session evaluations (table 1) indicated moderate to high levels of satisfaction but evaluation comments on this session and the workshop itself expressed concerns about the technical aspects of using wikis. these concerns could be addressed with more face-to-face practice, but there are also implications for ongoing support while the workshop task is in progress. this suggests a need for trialling different support options (e.g., printed support materials, online chat sessions, peer mentoring). participant concerns also suggest a tendency for technology anxiety to undermine focus on pedagogical issues. this has implications for contextualising the introduction to the technology as part of the session to maintain the pedagogical emphasis (rather than treating it separately as requested by some participants), clearly articulating the pedagogical purpose of the workshop. increasing the length of the orientation session would probably assist in this. workshop task progress showed an early phase of high discussion activity and moderate activity in wiki editing, followed by a lull, with a late phase of high editing activity but moderate to low discussion activity. this is consistent with collaborative group work in other contexts, reflecting an early task-negotiating phase leading to a co-operative phase where participants execute the tasks. caution in analysing the data further is needed as the nature and significance of the contributions are not identified. variations between groups (figures 1 and 2) may reflect common interests and ease in facilitation of the faculty-based ua workshops. this suggests that faculty or department workshops may be preferable to university-wide recruitment if tangible reward for participation (e.g., inclusion in a graduate certificate course) is unavailable. it may also help address evaluation comments, which emphasised the value of the experiential, collaborative nature of the workshop but expressed frustration with nonparticipating colleagues. providing increased support for group work, fast tracking group formation and helping to establish a community of practice (wenger, mcdermott, and snyder 2002) in the extended orientation session may also assist. despite the intended role of the orientation in supporting group formation, the required levels of bonding and trust did not always develop sufficiently, particularly in ub group 2. when participants fail to form functional groups in their wikis, their ability to engage with the task and to form a community of enquiry (garrison and anderson 2003) among themselves (and with us) is impaired. the reflection and evaluation components of the debriefing provided useful information but the critique component appears to need greater emphasis if using a rubric to model wiki assessment is retained as an important aspect of the workshop. while the small number of ua responses (three) was probably influenced by the server malfunction, there were also only two ub critiques. this may indicate participants’ discomfort in reviewing colleagues’ novice wiki work. in summary, key points in planning a third iteration of the workshop will be to: (1) extend the orientation session to integrate technology familiarisation with the pedagogical purpose of the workshop, while reducing the number of presenters; (2) improve group formation and increase participant responsibility to create a community of practice; research in learning technology citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 (3) emphasise the role of critique during debriefing and its relevance to assessing group work; (4) facilitate deeper reflection on the affordances of wikis for student group work; (5) develop ongoing support strategies in these areas; and (6) offer the workshop at faculty or department level if no reward is available for participation. thus, our experience has suggested that this online staff development model is potentially effective in helping staff to use wikis for student group work and assessment (research question 1). the main implications for improving the model that emerged over two action research cycles (research question 2) are outlined above. they indicate that action research was a useful approach for the iterative development of the workshop (research question 3), illustrating its practical advantages in informing this staff development approach (kember and kelly 1993; norton 2009) and exploring innovative pedagogy (laurillard 2008). development of the model has contributed to addressing the need to prepare staff for online teaching (wilson and stacey 2004), relating to the particular opportunities and challenges that web 2.0 technologies offer (barnes and tynan 2007; dron 2007). our findings support the value of a project-based approach for web 2.0 staff development (doherty 2011), by embedding it in an authentic, online context (wilson and stacey 2004). conclusion this exploration of the experiences of a small number of groups at two australian universities contributes to the more general need to explore ways to assist academic staff to use web 2.0 applications for teaching and assessment. the case provides a model for the investigation of innovative staff development that is widely applicable and adaptable to other contexts through action research. making our enquiry public and available for peer-review, critique and exchange with others so that they can build on our work, as in our description of the first cycle of the development of this model (samarawickrema, benson, and brack 2008, 2010), meets the definition by shulman (2000) of a contribution to the scholarship of teaching and learning. by developing and investigating the effectiveness of this staff development model with its inbuilt capacity for improvement and adaptation through action research, our aim was to contribute to this field of enquiry, share the model itself so that others could use or modify it and, also, to illustrate the particular value of action research for engaging with the scholarship of teaching and learning (norton 2009) by iteratively developing professional knowledge in this emerging area. acknowledgements this paper was developed from a presentation made at the issotl10 conference, liverpool, uk, 19�22 october 2010. we are grateful for an outstanding paper award from the ascilite auckland 2009 conference for our description of the first iteration of this staff development model based on its relevance and wide applicability. the resulting invited journal publication (samarawickrema, benson, and brack 2010), provided us with evidence of the contribution of this approach to the scholarship of teaching and learning. r. benson et al. 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 references barnes, c. & tynan, b. (2007) ‘the adventures of miranda in the brave new world: learning in a web 2.0 millennium’, alt-j, vol. 15, no. 3, pp. 189�200. birch, d. & burnett, b. (2009) ‘bringing academics on board: encouraging institution-wide diffusion of e-learning environments’, australasian journal of educational technology, vol. 25, no. 1, pp. 117�134, [online] available at: http://www.ascilite.org.au/ajet/ajet25/ birch.html boshier, r. (2009) ‘why is the scholarship of teaching and learning such a hard sell?’, higher education research and development, vol. 28, no. 1, pp. 1�15. brew, a. (2006) research and teaching: beyond the divide, palgrave macmillan, houndmills, uk. brookfield, s. (1995) becoming a critically reflective teacher, jossey-bass, san francisco. brown, j. s., collins, a. & duguid, p. (1989) ‘situated cognition and the culture of learning’, educational researcher, vol. 18, no. 1, pp. 32�42. carr, w. & kemmis, s. (1986) becoming critical: education, knowledge and action research, the falmer press, lewes, east sussex. doherty, i. (2011) ‘evaluating the impact of educational technology professional development upon adoption of web 2.0 tools in teaching’, australasian journal of educational technology, vol. 27, no. 3, pp. 381�396, [online] available at: http://www.ascilite.org.au/ ajet/ajet27/doherty.pdf dron, j. (2007) control and constraint in e-learning: choosing when to choose, information science publishing, hershey, pa. garrison, r. & anderson, t. (2003) e-learning in the 21st century: a framework for research and practice, routledgefalmer, london. grundy, s. (1995) action research as ongoing professional development, arts accord affiliation of arts educators, perth, wa. guo, z. & stevens, k. j. (2011) ‘factors influencing perceived usefulness of wikis for group collaborative learning by first year students’, australasian journal of educational technology, vol. 27, no. 2, pp. 221�242, [online] available at: http://www.ascilite.org.au/ ajet/ajet27/guo.html hutchings, p. & shulman, l. s. (1999) ‘the scholarship of teaching’, change, vol. 31, no. 5, pp. 10�15. kember, d. & gow, l. (1992) ‘action research as a form of staff development in higher education’, higher education, vol. 23, no. 3, pp. 297�310. kember, d. & kelly, m. e. (1993) improving teaching through action research, herdsa green guide no. 14, nsw. kemmis, s. & mctaggart, r. (2005) ‘participatory action research: communicative action and the public sphere’, in the sage handbook of qualitative research, eds n. k. denzin & y. s. lincoln, 3rd edn, sage, thousand oaks, ca, pp. 559�603. laurillard, d. (2008) ‘the teacher as action researcher: using technology to capture pedagogic form’, studies in higher education, vol. 33, no. 2, pp. 139�154. lewin, k. (1946) ‘action research and minority problems’, journal of social issues, vol. 2, no. 4, pp. 34�46. norton, l. s. (2009) action research in teaching and learning, routledge, london. ruth, a. & houghton, l. (2009) ‘the wiki way of learning’, australasian journal of educational technology, vol. 25, no. 2, pp. 135�152, [online] available at: http:// www.ascilite.org.au/ajet/ajet25/ruth.html samarawickrema, g., benson, r. & brack, c. (2008) ‘teaching with wikis: addressing the digital divide’, in alt-c 2008: rethinking the digital divide, eds n. whitton & m. mcpherson, association for learning technology, leeds, uk, pp. 84�94. samarawickrema, g., benson, r. & brack, c. (2010) ‘different spaces: staff development for web 2.0’, australasian journal of educational technology, vol. 26, no. 1, pp. 44�49, [online] available at: http://www.ascilite.org.au/ajet/ajet26/samarawickrema.html schön, d. (1995) ‘the new scholarship requires a new epistemology’, change, vol. 27, no. 6, pp. 26�34. shulman, l. s. (2000) ‘from minsk to pinsk: why a scholarship of teaching and learning?’, the journal of scholarship of teaching and learning, vol. 1, no. 1, pp. 48�52. research in learning technology citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 15 (page number not for citation purpose) http://www.ascilite.org.au/ajet/ajet25/birch.html http://www.ascilite.org.au/ajet/ajet25/birch.html http://www.ascilite.org.au/ajet/ajet27/doherty.pdf http://www.ascilite.org.au/ajet/ajet27/doherty.pdf http://www.ascilite.org.au/ajet/ajet27/guo.html http://www.ascilite.org.au/ajet/ajet27/guo.html http://www.ascilite.org.au/ajet/ajet25/ruth.html http://www.ascilite.org.au/ajet/ajet25/ruth.html http://www.ascilite.org.au/ajet/ajet26/samarawickrema.html http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 su, f. & beaumont, c. (2010) ‘evaluating the use of a wiki for collaborative learning’, innovations in education and teaching international, vol. 47, no. 4, pp. 417�431. surowiecki, k. (2004) the wisdom of crowds, doubleday, new york. vygotsky, l. (1978) mind in society, harvard university press, cambridge, ma. webb, g. (1996) understanding staff development, srhe and open university press, buckingham. wenger, e., mcdermott, r. & snyder, w. m. (2002) cultivating communities of practice, harvard business school press, boston, ma. wilson, g. & stacey, e. (2004) ‘online interaction impacts on learning: teaching the teachers to teach online’, australasian journal of educational technology, vol. 20, no. 1, pp. 33�48. r. benson et al. 16 (page number not for citation purpose) citation: research in learning technology 2012; 20: 16149 doi: 10.3402/rlt.v20i0.16149 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/16149 (page number not for citation purpose) 1 *corresponding author. email: panos.vlachopoulos@mq.edu.au research in learning technology 2019. © 2019 p. vlachopoulos et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 research in learning technology vol. 27, 2019 original research article a comparative study on the traditional and intensive delivery of an online course: design and facilitation recommendations panos vlachopoulos*a, shazia k. janb, lori lockyerc afaculty of arts, macquarie university, sydney, australia; bfaculty of business & economics, macquarie university, sydney, australia cgraduate research school, university of technology sydney (received: 14 november 2018; revised: 12 february 2019; accepted: 13 february 2019; published: 7 march 2019) in this paper, we present findings from a comparative study on a fully online postgraduate course offered in traditional (i.e. 13-week academic session) and intensive (i.e. 6-week academic session) delivery formats. keeping the course curriculum, structure and quality consistent in both delivery modes, the study investigated student participation and academic performance given different facilitation techniques applied to the discussion forums. using data from the learning management system and students’ final marks, we conducted quantitative and qualitative analysis and found no difference in the academic performance of students in both courses; however, there was a statistically significant relationship between student participation and academic performance in the intensive delivery format but not in the traditional delivery format. we also found differences in the type of interactions in the different delivery formats. two key takeaways emerge from our study. firstly, intensive online courses can be as effective as traditional courses in terms of achievement of learning outcomes with variations in learning design, in this case, the facilitation approach used. secondly, considering the level and nature of interactions, student-centred discussion forums that allow students to assume different roles work well in the intensive delivery format especially in open discussions. these are important findings for academics and practitioners who wish to offer intensive courses without compromising on course quality and student success. keywords: online; intensive; facilitation; discussion; interactions; performance introduction a large number of universities are offering intensive courses across a variety of disciplines (harwood et  al. 2018; mcdonald 2003). intensive courses typically offer the  same learning objectives as a traditional course but over a shorter duration of study. such courses are being offered to meet the demands of changing demographics of students who wish to partake in higher education. an increasing number of mature aged students are studying part-time due to work or family commitments (chao,  derocco, and flynn 2009). therefore, the students require the flexibility mailto:panos.vlachopoulos@mq.edu.au http://www.alt.ac.uk http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2196 p. vlachopoulos et al. 2 citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 (page number not for citation purpose) to compress or fast-track their studies depending on their needs (ellis and sawyer 2009; harwood et al. 2018). the duration of traditional and intensive courses varies between institutions. traditional courses run over 12–15 weeks with an expectation that students will devote around 12 h per week to their studies in the course, whereas an intensive course could range from 4 to 6 weeks with an expectation that students would devote around 22–25 h per week. although it is clear that intensive courses offer course flexibility and convenience, the impact of the compressed format on learning outcomes is less obvious, especially in purely online courses. majority of the literature comparing traditional with intensive courses is in the face-to-face context and therefore, findings are not transferrable to online learning. regardless of the mode of delivery (face-to-face or online), keeping the quality of the course and curriculum content consistent across intensive and traditional modes is a challenge (chen 2007) but a requirement considering the goal of equivalent learning outcomes for students. there is typically little or no flexibility in the curriculum content; therefore, the onus rests on adaptation of teaching methods and course design in different delivery formats (mcdonald 2003; wlodkowski 2008; wlodkowski and ginsberg 2010). in the context of online learning, given common limitations of time and resources faced by academics, modification to the online facilitation approach used presents as the most achievable adaptation to teaching methods. considering the need for and dearth of literature on the effectiveness and design of intensive online courses, we conducted a comparative study on an intensive and traditional offering of a purely online graduate level course at a large metropolitan university in australia. the main objective of the study was to compare student participation and academic performance in the traditional and intensive delivery modes which only differed in duration and the facilitation approach used for the online discussion activities. specifically, the study was guided by the following research questions: is there a difference in students’ academic performance in the traditional and intensive offering of the course? do students participate differently in the online discussion forums of the traditional and intensive course? what is the relationship between participation in discussion forums and academic performance in traditional and intensive course? does the nature of participation in discussion forums in the traditional and intensive course differ? in view of the increasing popularity of intensive online courses and shortage of research in the area, findings from our study provide valuable guidelines to academics and practitioners for the design and facilitation of online discussion forums in traditional and intensive online courses. literature review there is plenty of literature reporting favourable learning outcomes for students taking face-to-face courses in intensive and traditional delivery formats. research studies report either no difference in learning outcomes (anastasi 2007; lovett, meyer, and thille 2008) or an improvement in the outcomes (hall, wilson, and sanger 2012; scott 2003) for students in the intensive delivery mode (daniel 2000; davies 2006; scott and conrad 1992). findings of comparative studies on traditional versus intensive delivery of courses in the face-to-face context are not applicable to online courses due to the significantly different pedagogical practices in and affordances of ( gibson 1979) the online space (harasim 1990). however, in the face-to-face context, the equivalency or perhaps even superiority of intensive courses as compared to traditional courses in terms of students’ academic performance prompts the question http://dx.doi.org/10.25304/rlt.v27.2196 research in learning technology citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 3 (page number not for citation purpose) of whether the same holds true in the online context. the three comparative studies (ferguson and de felice 2010; gao 2013; harwood et al. 2018) we have identified on purely online courses corroborate the positive relationship between intensive delivery on student performance. ferguson and de felice (2010) examined both academic performance and satisfaction of students who completed an intensive and traditional course for pre-service teachers. in the study, the only difference between the courses was the duration while the content and teaching methods remained the same. they found that students in the intensive course performed better; however, there was no difference in student satisfaction between the traditional and intensive delivery. likewise, gao (2013) examined the academic performance of students in a traditional and intensive course first-year introductory course in computer technology and reported higher grades of students in the intensive format. most recently, harwood et al. (2018) compared student outcomes and satisfaction in a postgraduate level traditionally delivered course with an intensive online course in the health sciences and found no significant differences in student performance and satisfaction. in this study the two courses differed in pedagogy and curriculum. to date, we have been unable to identify additional comparative studies on online traditional versus intensive courses, a gap in literature reported by ferguson and de felice (2010) almost a decade ago, gao (2013) and most recently by harwood et al. (2018) as well. again, borrowing from literature in the face-to-face context, research studies suggest that typically lecturers of intensive courses modify or adjust teaching methods or techniques to allow for the shorter duration (daniel 2000; kretovics, crowe, and hyun 2005). furthermore, it is reported that instructional methodology plays an important role in the success of intensive courses (grady 2013). in a review of research studies on intensive and traditional courses, davies (2006) puts the onus of the success of a course, intensive or traditional, on the appropriateness of the teaching methods used. therefore, adaptation of teaching methods according to the mode of delivery appears critical for course quality and student success. kuiper, solomonides, and hardy (2015) also emphasised that organising teaching and learning through intensive modes of delivery may require different approaches to curriculum development and pedagogy compared to traditional course planning and delivery, especially when the intensive delivery utilises online technologies. the key components of an online course comprise of the learning management system, the course content or curriculum, and the learning design. broadly, the learning design includes the course structure, learning activities, tools, and facilitation techniques used for online engagement in, for instance, discussion forums. given that lecturers are often strapped for time, altering the overall pedagogy, structure and activities of courses for traditional and intensive delivery is not a desirable undertaking. therefore, the most achievable adjustment between the two delivery formats is adaptation to the facilitation approach adopted. the impact of different online facilitation approaches on student engagement or participation within discussion forums has been the subject of research since the early 1990s (vlachopoulos and cowan 2010a). we define facilitation ‘to be an activity in which someone (usually a tutor) makes interventions to encourage students to engage with, and achieve, their overall learning outcomes’ (vlachopoulos and cowan 2010a, p. 214). facilitation of online discussion forums can be divided into two broad categories: hierarchical or staged models in which students’ transition from one stage to the next is facilitated by the tutor and; student-centred non-stratified models that allow students the freedom to assume different roles. staged models typically focus http://dx.doi.org/10.25304/rlt.v27.2196 p. vlachopoulos et al. 4 citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 (page number not for citation purpose) on the role of the instructor or tutor who is primarily responsible for ensuring meaningful student engagement. examples of such models are the five-stage moderation model (salmon 2000) and the reeal model of online facilitation (bedford 2011). in salmon’s (2000) model, the facilitator engages those students who progress from one stage to the next until they reach the last two stages of knowledge construction and development. student-centred models of facilitation put the onus of meaningful engagement on students who use different strategies to encourage peer participation (baran and correia 2009). aviv, erlich, and ravid (2005) reported a clear relationship between students’ level of student responsibility in structured online discussion with higher levels of involvement and critical thinking. an example of one such model is the ring-fence model (vlachopoulos and cowan 2010a) in which the role of the facilitator is located outside a ring-fence which is also referred to as the enclosed learning arena. within the ring-fence, students direct and facilitate their own learning. vlachopoulos and cowan (2010b) conceived a range of different roles that a teacher can play in student facilitated discussions, which varied from being a critical friend to that of a summariser and even a discussion rescuer. findings of research on both types of facilitation approaches are mixed. for instance, research shows that active participation of the instructor enhances student engagement (fidalgo and thormann 2012; tsiotakis and jimoyiannis 2016). on the other hand there is evidence that instructor presence does not increase student participation (zhao and sullivan 2017). student-led facilitation strategies are shown to foster student involvement, a greater sense of community, and learning outcomes (baran and correia 2009). it is important to note, though, that in online discussions, motivation to participate has been shown to be strongly associated with ‘how learners experience and perceive social interaction while they relate to individuals’ (yang et al. 2006, p. 278). yang et al. (2006) concluded that student-centred facilitation may lead to marginalisation and dis-engagement of low-achieving students who become passive because their goal is not to learn from each other but to satisfy their tutor’s priorities with their responses. what we do know is that the design and facilitation of discussion forums impacts student participation which in turn plays a critical role in fostering learning which communities (jan 2018; jan and vlachopoulos 2018; palloff and pratt 2007) have a positive bearing on the learning outcomes (yang et  al. 2016). due to time lags, asynchronous online interactions take time to establish. in discussion forums that include facilitation approaches based on staged models, students rely on the instructor or tutor to drive the discussion. the challenge of such models is that they take time to work because it is not immediately apparent what the students are supposed to do. in student-centred discussions where there is minimal expectation of intervention or direction from a tutor, students take on different roles and assume ownership of the forum in a relatively shorter period of time. considering the ultimate goal of equivalent learning outcomes for students in intensive and traditional delivery formats of a course, the aim should be to maximise student participation as higher engagement leads to community formation which is linked to better learning outcomes. context of the study we conducted our study on two offerings of a fully online postgraduate course in education leadership and management at a large metropolitan university in australia. http://dx.doi.org/10.25304/rlt.v27.2196 research in learning technology citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 5 (page number not for citation purpose) the course was offered in semester three (s3) (intensive delivery) 2013 and semester one (s1) (traditional delivery) 2014. the curriculum and design of the course in the learning management system in s3 and s1 were identical. the differences between the courses in the two semesters were the duration of the course, facilitation approach used in the discussion forums and the time gap between each discussion activity. for our purposes, we refer to the different facilitation approaches as student-moderated (sm) and tutor-moderated (tm). s3 ran for a period of 6 weeks and comprised backto-back sm discussion forums spanning 2 weeks each. s1 ran over 13 weeks and included tm discussion forums which ran over 2 weeks each but were interspersed with other activities. a total of 27 students saw the course to completion in s3 as compared to 21 students in s1. in both semesters, one tutor facilitated the course. table 1 summarises the differences between the course in s1 and s2. the cohort includes those students who completed the course. the three discussion forums were identical in design in s3 and s1 but were different from one another. discussion (d1) was a guided discussion in which students were provided with a reading and a set of questions. in discussion 2 (d2), the students were given a discussion topic but were required to contribute questions for discussion. discussion 3 (d3) was an open discussion in which students were asked to discuss anything that needed to be addressed from the course syllabus and was of interest to them or was directly linked with the assessments. all discussions were threaded. in both semesters, online participation constituted 10% of the finals marks, and the remaining 90% was allocated to written assignments. methodology we used a mixed-method approach to address the research questions. data for quantitative and qualitative analyses were obtained from the learning management system (moodle) at the end of s3 and s1. data included: (1) the online messages posted by the students and the online tutor in the online discussion forums and (2) the final student marks in the courses under scrutiny. statistical and social network analysis (sna) was used to examine the difference in overall academic performance, participation and the relationship between participation and performance. sna was conducted in ucinet 6.0 and spss was used for statistical analysis. qualitative analysis of the content of posts made in the discussion forums was used to determine the nature of interactions. qualitative analysis was conducted using nvivo. the research was conducted throughout in accordance with the regulations and ethical codes of the concerned university. appropriate precautions were taken to protect the confidentiality of participants. the researcher obtained written permission from the participants to use the data for research purposes and publication. table 1. differences between the intensive (s3) and traditional (s1) courses. differences semester 3 (s3), 2013 semester 1 (s1), 2014 cohort 27 students 1 tutor 21 students 1 tutor duration 6 weeks 13 weeks discussion (3 forums) 2 weeks each consecutive 2 weeks each gaps in-between facilitation approach student-moderated tutor-moderated http://dx.doi.org/10.25304/rlt.v27.2196 p. vlachopoulos et al. 6 citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 (page number not for citation purpose) quantitative analysis sna is an analytical technique used for investigating relationships or ties between nodes in a network. the nodes in our study were the students and the tutors. the relationships were the interactions within the discussion forums between the nodes. note that we use the terms ties and interactions interchangeably. therefore, students who only posted to the discussion forums but did not interact with others (isolates) were not included in the sna measures calculated. the networks were weighted and directed meaning the number of times two nodes interacted and original posts versus responses to posts were identified. sna measures of density and individual degree centralities were calculated for networks from d1, d2 and d3 in both semesters and for all nodes. the density of a network is calculated by the total number of interactions or ties divided by the total number of possible ties. the inand out-degree centrality of a node is the number of incoming and outgoing posts for each node, that is, the number of posts made and responses received. given the number of participants in the study, non-parametric measures of correlations, specifically the mann–whitney u test and spearman rankorder correlation, were used as tests of significance. qualitative analysis qualitative content analysis was conducted on all posts (including isolates) in the three discussion forums in s3 and s1. the illocutionary unit (howell-richardson and miller 1996) was used as the unit of analysis. the illocutionary unit focuses on the linguistic properties of the messages and the individual to whom the message is directed. all messages were coded for the type of interactions using the coding scheme given in table 2. findings quantitative analysis the final mark obtained in the courses was taken as a measure of overall academic performance. table 3 shows summary statistics on the final marks in both courses in table 2. interaction coding scheme. type of interaction code criteria group proactive gpi student or tutor looks for a response from someone in the group – anyone group reactive gri student or tutor responds to one of the above, or some other message, playing reply back to group monologue m a new thread. no evidence of interaction with any other participant individual proactive pi student or tutor looks for a response from a specific contributor, or even asks for it quasi interactive qi threaded (follow-up) message where tutor or student acknowledges previous message but continues with a new idea/concept individual reactive ri student of tutor responds to one of the above, or some other message, from and then to a specific contributor note: for detailed indicators of criteria, refer to vlachopoulos (2012). http://dx.doi.org/10.25304/rlt.v27.2196 research in learning technology citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 7 (page number not for citation purpose) s3 and s1. interestingly, the average marks in both semesters are the same as that of the highest marks in s3 and lowest in s1. mann–whitney u test conducted on the final marks of students in s3 and s1did not indicate any statistically significant (p < 0.001) difference in the academic performance of students in the traditional and intensive cohorts. table 4 shows the network size, total number of posts and ties or interactions, and density for each of the discussion forums in s3 and s1. the density of the networks in d1 and d2 in s1 (tm) is slightly higher than in s3 (sm). however, the density of the network from d3 in s1 (tm) is substantially less than in s3 (sm). to test for significance of differences in densities, we conducted mann–whitney u tests on the inand out-degree centralities (inand outgoing posts) for all nodes. we found no statistically significant difference in the centralities for d1 and d2. however, for d3, the mann–whitney u test indicated that the in-degree centralities were significantly greater for d3 in s3 (mdn = 6.00) than d3 in s1 (mdn = 3.00), u = 130, p = 0.007. similarly, the out-degree centralities (outgoing posts) were greater for d3 in s3 (mdn = 5.00) than for d3 in s1 (mdn = 2.00), u = 95, p < 0.001. this leads to the conclusion that the sm, an open-ended discussion forum (d3) in the intensive course (s3), was most effective in engaging students. to ascertain the relationship between interactions or participation and academic performance, we conducted spearman rank-order correlations between inand out-degree centralities in the discussion forums and the final marks of the students in s3 and s1. the correlations obtained are shown in table 5. as can be seen, in s3, the inand out-degree centralities are positively correlated with the final marks in all except one case, that is, d1 and in-degree centralities. on the other hand, in s1, only one correlation is significantly positive, that is, d1 and out degree centralities. this is an important finding as it indicates that in the sm discussion forums of s3, regardless of the design of the discussion activity, students’ level of participation is positively linked to academic performance, meaning that those students who interacted more performed better. in s1, while the overall academic performance of students is not statistically different from the students in s3 (as discussed above), participation in the discussion activities does not have any statistically significant relationship with academic performance. given that online participation constituted only 10% of the final marks, this finding has important implications but does not support a conclusive relationship between participation and academic performance. summarising findings from the quantitative analysis, we conclude that the duration over which the course was delivered had no impact on the learning outcomes of students as they performed comparably. furthermore, our findings show that in the table 3. summary statistics of final marks. statistic s3 s1 no. of students 27 21 mean 70.46 70.73 median 69.50 72.50 std. deviation 7.909 10.986 range 33 52 minimum 55 33 maximum 88 85 http://dx.doi.org/10.25304/rlt.v27.2196 p. vlachopoulos et al. 8 citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 (page number not for citation purpose) intensive course, higher participation in the sm discussion forums is linked with overall higher performance in the course. finally, we find that, at least at the postgraduate level, open-ended discussions invite greater participation. the next section presents findings from qualitative content analysis of all posts in s3 and s1 discussion forums conducted to determine if there is a relationship between higher participation and the type of interactions. qualitative analysis all posts from s3 and s1 were extracted from the learning management system and coded using the coding scheme given in table 2. each post or part thereof was assigned a code. posts or parts of it that belonged to more than one category were assigned to multiple categories. two researchers independently performed the coding and achieved a cohen’s (1960) kappa interrater reliability of 73%. results of the coding are shown in table 6. table 5. spearman rho’s correlations between degree centralities and performance. centrality s3 marks s1 marks d1 in-degree correlations 0.371 0.300 sig. (2-tailed) 0.057 0.186 n 27 21 d1 out-degree correlations 0.403* 0.567** sig. (2-tailed) 0.037 0.007 n 27 21 d2 in-degree correlations 0.592** 0.260 sig. (2-tailed) 0.001 0.254 n 27 21 d2 out-degree correlations 0.417* 0.326 sig. (2-tailed) 0.030 0.149 n 27 21 d3 in-degree correlations 0.432* 0.017 sig. (2-tailed) 0.028 0.943 n 27 21 d3 out-degree correlations 0.500** 0.118 sig. (2-tailed) 0.009 0.632 n 27 21 notes: *p ≤ 0.05, **p ≤ 0.01 table 4. network size, ties and density. measure s3 (sm) s1 (tm) d1 d2 d3 d1 d2 d3 no. of students 27 27 27 21 21 21 no. of tutors 1 1 1 1 1 1 network size1 28 28 28 23 22 20 no. of posts 204 185 235 112 115 86 no. of ties 120 109 154 91 71 54 density 0.32 0.29 0.41 0.36 0.31 0.28 notes: 1network size includes only those students who participated in the discussions, that is, isolates are excluded. http://dx.doi.org/10.25304/rlt.v27.2196 research in learning technology citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 9 (page number not for citation purpose) as shown in table 6, the sm discussion forums of s3 had a substantially higher number of codes as compared to the tm discussion in s1 in line with the differences in the total number of posts made (see table 4). the largest differential of 63% occurs in the open-ended d3 which is not surprising since d3 in s3 had the highest participation rate overall and d3 in s1, the lowest. charts 1, 2 and 3 depict the type of posts in each of the discussion forums in s3 and s1. overall, the charts clearly show that the higher interactions in the sm discussion forums in s3 are linked with the number of posts or parts of a post that were coded as individual reactive. if we consider each of the discussion activities individually, chart 1 shows that the largest number of codes in the sm-guided discussion (d1) in s3 is individual reactive as compared to monologues in the tm discussion in s1. this indicates reciprocal exchange amongst students. chart 2 shows that in d2 in which students are provided the topic of discussion but expected to raise their own questions, the number of individual reactive and individual proactive codes in s3 (sm) are again more than s1 (tm). again, this points to a greater mutual exchange and discourse between students. in the open-ended discussion (d3), a larger total number of codes in s3 (sm) are expected since the overall participation is significantly higher; however, what is interesting to note in chart 3 is that the majority of the codes are individual reactive followed by individual proactive. this clearly shows the high level of interactions and mutual engagement of students which points to not only the effectiveness of the sm approach to facilitation but also to the effectiveness of open-ended discussion forums in terms of fostering engagement and mutual exchange – requisites for formation of learning communities. thus, the qualitative analysis supports and further elaborates on our quantitative findings. discussion the impetus for conducting the study came from the increasing demand for intensive online courses and lack of research on the impact of such courses on the learning outcomes of students. the key objective of the study was to compare overall academic performance and participation in online discussion activities of students in the traditional and intensive delivery modes of an online course. we wanted to ascertain whether modification of the facilitation approach used for the online discussion activities is an effective modification to the design of the course for compressed delivery. we used quantitative and qualitative analyses to investigate the differences in students’ academic performance, differences in student participation in the online discussion activities, the relationship between participation and academic performance and differences in the nature of participation in the online discussion forums. table 6. number of codes. discussion forums s3 (sm) s1 (tm) s3 – s1 difference discussion 1 (d1) 227 121 −47% discussion 2 (d2) 181 115 −36% discussion 3 (d3) 229 85 −63% chart 1. discussion 1. chart 2. discussion 2. chart 3. discussion 3. http://dx.doi.org/10.25304/rlt.v27.2196 p. vlachopoulos et al. 10 citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 (page number not for citation purpose) the course comprised three differently designed discussion activities. d1 was a guided discussion, d2 was topic driven but without guiding questions and d3 was open-ended. the discussion activities in the intensive version of the course in s3 were sm and those in the traditional course (s1) were tm. we found no statistically significant difference in the academic performance of students in s3 and s1. from this we conclude that, given a different facilitation approach, that is, sm discussions, the compressed duration of the course did not have any impact on learning outcomes of students. therefore, our study corroborates findings of studies that report equivalency of learning outcomes in intensive and traditional modes of delivery in the face-to-face (e.g. lovett, meyer, and thille 2008) and online context (e.g. harwood et al. 2018). we found no statistically significant difference in participation in d1 and d2 in s3 and s1; however, we did find a statistically significant difference in participation in d3 in s3 and s1. the finding of the lack of a significant relationship between the facilitation approach used and participation in d1 and d2 is in line with findings in previous research (e.g. fidalgo and thormann 2012; zhao and sullivan 2017). however, our findings of the statistically significant difference in participation in d3 in s3 and s1 indicates that a sm facilitation approach may be most effective if the discussion activity is open-ended which allows students the freedom to direct their own learning. we also found a statistically significant positive relationship between participation and academic performance in the intensive delivery mode (s3) but not in the traditional delivery (s1) mode. we cannot arrive at a definitive conclusion based on this finding since, in the course, participation only constituted 10% of the final grade. however, the finding does raise the question of whether assessment allocations truly reflect the learning that takes place via participation, engagement and discourse in online courses (mcdonald 2003). one cannot help but wonder, had participation been allocated a higher percentage to the final grades in this course, given the significant differences in interactions in d3 between the two offerings, would the learning outcomes have been equivalent? the qualitative analysis showed the dominance of individual reactive posts in the discussion forum with the highest participation rate, that is, d3 in s3. from this, we conclude mutual or reciprocal engagement of students. on the other hand, in the discussion forum with the lowest participation rate, that is, d3 in s1, even though the number of individual reactive posts is the highest, there are a sizeable number of monologues as well which is indicative of disengaged students. thus, our findings from the qualitative analysis provide further support to the effectiveness of sm open-ended discussion activities in fostering student participation. findings of our study are encouraging for those academics and practitioners who are faced with the challenge of compressing courses for intensive delivery without compromising on content and quality. it is apparent that by manipulating the design of a course, in this case, the facilitation approach used, one can achieve similar learning outcomes in compressed courses. limitations of our study include the lack of control of confounding factors that might have impacted student participation and academic performance, for instance, other personal and professional commitments of students during the course. given the global demand in online intensive courses and a serious gap in research on the design and effectiveness of such courses, we believe that findings from our study make a valuable contribution to the sparse literature in the area. http://dx.doi.org/10.25304/rlt.v27.2196 research in learning technology citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 11 (page number not for citation purpose) future research there may be other factors that influence participation, and exploring these potential factors could be part of further research. other factors may relate to what activities were scheduled or due concurrently in the course or other opportunities for students to have their questions answered outside the online discussion forum. further research could also look into the factors that appear to influence the way in which online facilitators decide when and how to intervene in sm discussions in short timeframes, such as in intensive online courses. in particular, the field might benefit by a study which explores the extent to which these factors may influence the facilitators to operate in a proactive or a reactive way, and the impact of differences in their practice on students’ learning and development. this would establish wider applicability for this study’s suggestions. acknowledgements this research was funded by macquarie university’s strategic priority grant. references anastasi, j. s. (2007) ‘full-semester and abbreviated summer courses: an evaluation of student performance’, teaching psychology, vol. 34, no. 1, pp. 19–22. https://doi.org/ 10.1177/ 009862830703400103 aviv, r., erlich, z. & ravid, g. (2005) ‘reciprocity analysis of online learning networks’, journal for asynchronous learning, vol. 9, no. 4, pp. 3–13. baran, e. & correia, a. (2009) ‘student-led facilitation strategies in online discussions’, distance education, vol. 30, no. 3, pp. 339–361. https://doi.org/10.1080/01587910903236510 bedford, l. (2011) ‘the adjunct’s role in delivering quality online instruction’, in cases on innovations in educational marketing: transnational and technological strategies, eds p.  tripathi & s. mukerji, igi global, hershey, pa. https://doi.org/10.4018/978-1-60960599-5.ch004 chao, e. l., derocco, e. s. & flynn, m. k. (2009) adult learners in higher education; barriers to success and strategies to improve results, u.s. department of labour, washington, dc. chen, s. (2007) ‘instructional design strategies for intensive online courses: an objectivistconstructivist blended approach’, journal of interactive online learning, vol. 6, no. 1, pp. 72–86. cohen, j. (1960) ‘a coefficient of agreement for nominal scales’, educational and psychology measurement, vol. 20, pp. 37–46. https://doi.org/10.1177/001316446002000104 daniel, e. l. (2000) ‘a review of time-shortened courses across disciplines’, college student journal, vol. 34, no. 2, pp. 298–308. davies, m. (2006) ‘intensive teaching formats: a review’, issues in educational research, vol. 16, no. 1, pp. 1–20. ellis, b. & sawyer, j. (2009) ‘regional summer schools: widening learning opportunities through intensive courses’, education in rural australia, vol. 19, pp. 35–52, [online] available at: http://www.spera.asn.au/ ferguson, j. m. & defelice, a. e. (2010) ‘length of online course and student satisfaction, perceived learning, and academic performance’, international review of research in open and distance learning vol. 11, no. 2, pp. 73–84. https://doi.org/10.19173/irrodl.v11i2.772 fidalgo, p. & thormann, j. (2012) ‘a social network analysis comparison of an experienced and a novice instructor in online teaching’, european journal of open, distance and e-learning, vol. 1, pp. 1–15. http://dx.doi.org/10.25304/rlt.v27.2196 https://doi.org/​10.1177/​009862830703400103 https://doi.org/​10.1177/​009862830703400103 https://doi.org/10.1080/01587910903236510 https://doi.org/10.4018/978-1-60960-599-5.ch004 https://doi.org/10.4018/978-1-60960-599-5.ch004 https://doi.org/10.1177/001316446002000104 http://www.spera.asn.au https://doi.org/10.19173/irrodl.v11i2.772 p. vlachopoulos et al. 12 citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 (page number not for citation purpose) gao, q. (2013) ‘effects of online course length on undergraduate student performance’, international journal of instructional technology and distance learning, vol. 10, no. 11, pp. 25–34. gibson, j. j. (1979) the theory of affordances. the ecological approach to visual perception, houghton mifflin, boston, ma. grady, j. r. (2013) ‘improving student satisfaction with large-scale, compressed timeline online courses’, quarterly review of distance education, vol. 14, no. 4, pp. 195–208. hall, m. v., wilson, l. a. & sanger, m. j. (2012) ‘student success in intensive versus traditional introductory chemistry courses’, journal of chemical education, vol. 89, no. 9, pp. 1109–1113. https://doi.org/10.1021/ed100535d harasim, l. m. (1990) ‘online education: an environment for collaboration and intellectual amplification’, in online education: perspectives on a new environment, eds l. m. harasim, praeger, new york, pp. 39–64. harwood, k. j., et al., (2018) ‘comparing student outcomes in traditional vs intensive, online graduate programs in health professional education’, bmc medical education, vol. 18, no. 240, pp. 1–9. https://doi.org/10.1186/s12909-018-1343-7 howell-richardson, c. & mellar, h. (1996) ‘a methodology for the analysis of patterns of participation within computer mediated communication courses’, instructional science, vol. 24, pp. 47–69. https://doi.org/10.1007/bf00156003 jan, s. k. (2018) ‘identifying online communities of inquiry in higher education using social network analysis’, research in learning technology, vol. 26, pp. 231–241. https://doi. org/10.25304/rlt.v26.2064 jan, s. k. & vlachopoulos, p. (2018) ‘influence of learning design on the formation of online communities of learning’, international review of research on open and distributed learning, vol. 19, no. 4, pp. 1–16. kretovics, m. a., crowe, a. r. & hyun, e. (2005) ‘a study of faculty perceptions of summer compressed course teaching’, innovative higher education, vol. 30, no. 1, pp. 37–51. https:// doi.org/10.1007/s10755-005-3295-1 kuiper, a., solomonides, i. & hardy, l. (2015) ‘time on task in intensive modes of delivery’, distance education, vol. 36, no. 2. https://doi.org/10.1080/01587919.2015.1055058 lovett, m., meyer, o. & thille, c. (2008) ‘the open learning initiative: measuring the effectiveness of the oli statistics course in accelerating student learning’, journal of interactive media in education, [online] available at: http://oli.cmu.edu/wp-oli/wpcontent/uploads/2012/05/ lovett_2008_statistics_accelerated_learning_study .pdf. https://doi.org/10.5334/2008-14 mcdonald, j. (2003) ‘assessing online collaborative learning: process and product’, computers & education, vol. 40, no. 4, pp. 377–391. https://doi.org/10.1016/s0360-1315(02)00168-9 palloff, r. & pratt, k. (2007) building online learning communities: effective strategies for the virtual classroom, eds r. m. palloff & k. pratt, 2nd edn., jossey-bass, san francisco, ca. salmon, g. (2000). e-moderating: the key to teaching and learning online, london: kogan page. scott, p. (2003) ‘attributes of high-quality intensive courses’, new directions for adult and continuing education, vol. 97, pp. 29–38. scott, p. a. & conrad, c. f. (1992) ‘a critique of intensive courses and an agenda for research’, higher education: handbook of theory and research, vol. 8, pp. 411–459. tsiotakis, p. & jimoyiannis, a. (2016) ‘critical factors towards analysing teachers’ presence in on-line learning communities’, the internet and higher education, vol. 28, pp. 45–58. https://doi.org/10.1016/j.iheduc.2015.09.002 vlachopoulos, p. (2012) ‘the importance of power dynamics in the development of asynchronous online learning communities’, in proceedings of ascilite – australian society for computers in learning in tertiary education annual conference 2012, eds m. brown, m. hartnett & t. stewart, australasian society for computers in learning in tertiary education, [online] available at: https://www.learntechlib.org/p/42718/ http://dx.doi.org/10.25304/rlt.v27.2196 https://doi.org/10.1021/ed100535d https://doi.org/10.1186/s12909-018-1343-7 https://doi.org/10.1007/bf00156003 https://doi.org/10.25304/rlt.v26.2064 https://doi.org/10.25304/rlt.v26.2064 https://doi.org/10.1007/s10755-005-3295-1 https://doi.org/10.1007/s10755-005-3295-1 https://doi.org/10.1080/01587919.2015.1055058 http://oli.cmu.edu/wp-oli/wpcontent/uploads/2012/05/lovett_2008_statistics_accelerated_learning_study http://oli.cmu.edu/wp-oli/wpcontent/uploads/2012/05/lovett_2008_statistics_accelerated_learning_study https://doi.org/10.5334/2008-14 https://doi.org/10.1016/s0360-1315(02)00168-9 https://doi.org/10.1016/j.iheduc.2015.09.002 https://www.learntechlib.org/p/42718 research in learning technology citation: research in learning technology 2019, 27: 2196 http://dx.doi.org/10.25304/rlt.v27.2196 13 (page number not for citation purpose) vlachopoulos, p. & cowan, j. (2010a) ‘choices of approaches in e-moderation: conclusions from a grounded theory study’, active learning in higher education, vol. 11, no. 3, pp. 213–224. https://doi.org/10.1177/1469787410379684 vlachopoulos, p. & cowan, j. (2010b) ‘reconceptualising moderation in asynchronous online discussions using grounded theory’, distance education, vol. 31, no. 1, pp. 23–36. https:// doi.org/10.1080/01587911003724611 wlodkowski, r. j. (2008) enhancing adult motivation to learn: comprehensive guide for teaching all adults, 3rd edn., jossey-bass, san francisco, ca. wlodkowski, r. j. & ginsberg, m. b. (2010) teaching intensive and accelerated courses: instruction that motivates learning, jossey-bass, san francisco, ca. yang, c.-c., et al., (2006) ‘exploring the relationships between students’ academic motivation and social ability in online learning environments’, internet and higher education, vol. 9, pp. 277–286. https://doi.org/10.1016/j.iheduc.2006.08.002 yang, j. c., et  al., (2016) ‘effects of online presence on learning performance in blog-based online course’, the internet and higher education, vol. 30, pp. 11–20. https://doi.org/ 10.1016/j.iheduc.2016.04.002 zhao, h. & sullivan, k. p. (2017) ‘teaching presence in computer conferencing learning environments: effects on interaction, cognition and learning uptake’, british journal of educational technology, vol. 48, no. 2, pp. 538–555. https://doi.org/10.1111/bjet.12383 http://dx.doi.org/10.25304/rlt.v27.2196 https://doi.org/10.1177/1469787410379684 https://doi.org/10.1080/01587911003724611 https://doi.org/10.1080/01587911003724611 https://doi.org/10.1016/j.iheduc.2006.08.002 https://doi.org/​10.1016/j.iheduc.2016.04.002 https://doi.org/​10.1016/j.iheduc.2016.04.002 https://doi.org/10.1111/bjet.12383 (page number not for citation purpose) 1 *corresponding author. email: candace.nolan-grant@durham.ac.uk research in learning technology 2019. © 2019 c.r. nolan-grant. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 research in learning technology vol. 27, 2019 original research article the community of inquiry framework as learning design model: a case study in postgraduate online education candace r. nolan-grant durham centre for academic development, durham university, lower mountjoy, durham, united kingdom (received: 22 february 2019; revised: 11 june 2019; accepted: 24 june 2019; published: 18 july 2019) interaction within online educational environments has long been advocated as conducive to learning, whether interaction between the learner and their teacher, the learner and online resources or the learner and their peers. the relationship among these three types of interaction is also receiving increasing attention, with the community of inquiry (coi) framework providing a method of interpreting this relationship in terms of the interplay of teaching presence, cognitive presence and social presence. this case study investigates the use of the coi framework as a learning design model, showing how it was used to address specific issues in a postgraduate online module. specifically, the framework informed decisions to strongly link together interaction with the video content, activity on discussion boards and release of new learning materials. using discussion board posts and video analytics as the primary evidence of learner engagement, the findings show how learner activity significantly increased in both ‘social’ contexts and ‘cognitive’ contexts. more importantly, analysis also revealed strong correlations among participation in discussions, video viewing and module completion. the study suggests that the coi framework is a robust model for learning design in online environments. keywords: online learning; postgraduate; academic professional development; interactive video; conditional release introduction the importance of learner–learner and learner–teacher interaction in online education is well documented (cherney, fetherston, and johnsen 2018; hrastinski 2008; salmon 2004). in particular, the formation and cultivation of a community is often explored in work around interactions between learners (zydney, denoyelles, and kyeong-ju seo 2012). interactivity in learning objects has also emerged as a key contributor to engagement and learning gain (delen, liew, and willson 2014). but more recently the community-building impact of all three of these interactions – with peers, with teachers and with learning materials – has received more attention (costley, hughes, and lange 2017; luo, zhang, and qi 2017). this ties directly into core elements of the community of inquiry (coi) framework, namely, social presence, teaching presence and cognitive presence (garrison, anderson, and archer 2010). while coi is often seen as a framework through which to investigate online learning mailto:candace.nolan-grant@durham.ac.uk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2240 c.r. nolan-grant 2 citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 (page number not for citation purpose) events or contexts, the current study argues that it can also be used as a learning design model to develop online learning experiences. examining how coi influenced changes to videos, to discussions with peers and tutors, and to the interplay between the two in an online postgraduate module, this study provides quantitative evidence that coi learning design can positively affect learner engagement. background aimed at staff who are new to teaching, foundations of learning and teaching in higher education is a well-established postgraduate module at durham university, successful completion of which leads to associate fellowship of advance higher education (he), the uk’s professional recognition body for teaching in higher education. the module content, discussions and assessment engage participants in theory-informed reflection on their teaching practice, culminating in a portfolio addressing a number of aspects of the uk professional standards framework, upon which advance he fellowship is based. in the 2016–2017 academic year, it was decided that this module would be expanded to include a fully online delivery as well as the face-to-face version to give staff who taught on a uk-wide programme validated by durham university to have the opportunity for development and to achieve associate fellow status. for this new online version, the module leaders and a tutor created video resources, mostly recording their voices over presentations that had been designed for the faceto-face sessions. the videos were captioned and added to a copy of the face-to-face virtual learning environment (vle) site, where the other resources were rearranged for online delivery. the face-to-face timetable was retained, with each session lasting 1 week. participants were assigned to two tutor groups at random for the purposes of interaction. within the resources for each session was a textual prompt to contribute to the group discussion board and to share a draft of the assignment section associated with the session on the group blog. once all of the sessions were complete, participants had nearly 5 months to compile and submit their summative assessments. out of a total of 45 participants, 19 submitted summatives, eight did not submit, five deferred, four chose to audit and nine withdrew. this first delivery of the module in a fully online format was evaluated by means of a module evaluation questionnaire, interviews with participants, data collected from the vle and tutor observation. coi was used as a framework against which to test the findings. the issues that emerged as a result of the evaluation are common in online education, as the literature review will show. briefly, however, the three main issues were as follows. participants averaged less than one blog or discussion board post per person per session, although everyone was expected to contribute at least once to the each weekly. this was considered to be a lack of social presence. participant feedback in module evaluation questionnaires and interviews strongly indicated that the minimalist approach to tutor discussion board participation could have been one cause of the paucity of interaction. this seemed to demonstrate a lack of teaching presence. the module evaluation questionnaires and interviews revealed that many participants felt that the videos were too long and unengaging. viewing analytics for all participants showed that, on average, those who started watching a video only watched 50%, and even those who completed the module viewed an average of just 59% of videos they began watching. this suggested a lack of cognitive presence. http://dx.doi.org/10.25304/rlt.v27.2240 research in learning technology citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 3 (page number not for citation purpose) it is the intention of this study to show how common issues in online learning like these (often culminating in just such failures to enable a true coi) can be addressed by employing coi as a learning design model. literature review conceptual framework developed in the context of asynchronous online discussions in higher education, the coi framework posits that the three key presences that sustain such a community – teaching presence, social presence and cognitive presence – are interrelated, working dynamically and in tandem throughout the educational experience. so, while teaching presence originates in what the teacher does to structure the learning, it interacts with cognitive presence in selection of learning content and social presence in setting the community climate. cognitive presence and social presence meanwhile determine the community’s discourse. this is illustrated in figure 1. engagement in online discussion past research mirrors many behaviours that were observed during the module pilot. for example, the steady decline in discussion board activity after the first session could have represented a vicious circle in which participants were discouraged from contributing because of the lack of contribution from others (kurucay and inan 2017; lee, kim, and hackney 2011; luo, zhang, and qi 2017; nerantzi and gossman 2015). this was very likely compounded by minimalist tutor engagement. some research does support tutor restraint (kahn et al. 2017; zydney, denoyelles, and kyeong-ju seo 2012), but there is also evidence to the contrary (chen and jang 2010; garrison, anderson, and archer 2010; jones and issroff 2005; parks-stamm, zafonte, and palenque 2017). finally, learners need to perceive concrete reasons for participating (jones and issroff 2005); figure 1. community of inquiry framework. http://dx.doi.org/10.25304/rlt.v27.2240 c.r. nolan-grant 4 citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 (page number not for citation purpose) perhaps the relatively intangible benefit of a better understanding of the course material through discussion and the reward of receiving feedback on drafts of an assignment that was not due for 5 months were not sufficient motivations for many participants. how to demonstrate the immediate and long-term benefits of interaction to learners is a difficult question. going on the basis that, fundamentally, discussion is beneficial (salmon 2004; zydney, denoyelles, and kyeong-ju seo 2012), the question then becomes how to encourage learners to experience the benefits and thus create a positive perception. perhaps an obvious method is to associate marks with quantity and/or quality of discussion board contributions. while this has been considered to be too invasive (gulati 2008) and to some degree counter-productive (lee, kim, and hackney 2011), it has proven successful in some cases. gafni and geri (2010) found that postgraduate students who participated in graded online forum assignments performed significantly better in the examination at the end of the module versus those for whom forum participation was optional. when comparing undergraduates in an online course, some of whom were assigned to work in groups to produce summative assessments and some of whom worked independently, kurucay and inan (2017) found that the former achieved significantly higher marks. luo, zhang and qi (2017), in their study of online discussion boards in a professional education context, concluded that some kind of incentivisation to participate could contribute towards creating a sense of community. and kovanović et al. (2015) invoked the coi model to look specifically at the cognitive presence aspect of online discussion board engagement within a group of postgraduates, concluding by advocating a flexible approach to accommodate different types of learners, including motivational support – -such as working in groups – for those less inclined to participate. thus, for a context involving mature learners’ summative marking would not necessarily be appropriate, but a light-touch method of compulsory contribution could be effective. in reference to tutor activity in discussions, it has been shown that teaching presence can have a positive effect on both cognitive and social presence (garrison, anderson, and archer 2010). however, it is often difficult to determine how and how many tutors should participate in online interactions with students (moule 2007; parks-stamm, zafonte, and palenque 2017; zydney, denoyelles, and kyeong-ju seo 2012). there is some evidence that points towards a direct correlation between tutor and student engagement (parks-stamm, zafonte, and palenque 2017), and to the positive effect of personalised tutor–student interaction on need satisfaction and thus learning outcomes (chen and jang 2010). it is also clear that the type of tutor engagement is critical. in terms of content, palincsar (1998) found that tutor participation is beneficial when the talk is interpretive rather than descriptive and pushes the discussion forward. in terms of timing, ouyang and scharber (2017) advocate a dynamic balance among tutor roles rather than simply assuming that ‘more is better’, while moule (2007) supports an increase in tutor facilitation as the discussion activities move from instructivist to constructivist. these findings therefore pointed towards increased tutor activity on the discussion boards, not for the sake of quantity alone but to respond to participant threads in a context-appropriate, interpretive way. engagement with videos a number of common bad practices in the implementation of online educational video emerge in the literature, and the videos in the pilot were guilty of several. firstly, they were much longer than is generally recommended for online educational videos. there is a large amount of evidence in the literature to suggest that the longer http://dx.doi.org/10.25304/rlt.v27.2240 research in learning technology citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 5 (page number not for citation purpose) the online video, the more likely viewers are to stop watching (costley, hughes, and lange 2017; geri, winer, and zaks 2017; guo, kim, and rubin 2014). guo, kim and rubin (2014) concluded that online educational videos should be no longer than 6 min, as drop-off tends to increase after the 3-min mark. this  study looked at videos on massive open online courses (moocs), however, which are informal, noncredit-bearing courses. costley, hughes and lange (2017) found much better drop-off rates for videos on credit-bearing higher education modules, in contrast to much of the literature, but do not report the average length of the videos provided. as the videos in the current study were an average of 30 min long, with the most lengthy at 55 min, and as students had commented negatively on the video length, the evidence still seemed to support reducing the length of the videos in some way. it has also been shown that even very basic interactivity options in video can have a significant effect on learning gain and student satisfaction (zhang et al. 2006). delen et al. (2014) demonstrated that students who were provided with interactive video retained learning better and were more likely to remain engaged, and geri, winer and zaks (2017) found that interactivity had the potential to increase learners’ attention spans (and thus decrease drop-off rates). these studies all looked at simple forms of interaction, many of which are achievable on standard video streaming platforms. presence through the interplay of content and discussion just as the three different types of presence in the coi model work in tandem to create a community (garrison, anderson, and archer 2010), so do the different elements involved in what the community does. costley, hughes and lange (2017) suggest that the students in their study may have engaged more with video content because of an awareness ‘of their responsibilities as part of a community of inquiry’ (p. 201). they conclude that clear expectations around when videos will be watched and which activities (e.g. discussions) will follow on from the videos increase engagement with both the videos and the course as a whole. and while they did not invoke the coi model per se, luo, zhang and qi (2017) found that student–student and student–instructor interaction had a significantly stronger effect on students’ sense of community when they also showed higher engagement with the content. thus, much literature does successfully employ the coi framework to analyse a variety of online educational scenarios. there is less evidence, however, of the framework being used as a design model to connect discrete elements of learning together in a structured, dynamic way to foster community in the first place. this study therefore attempts to show how specific interventions were designed to work together within the coi framework to bring about observable improvements in student engagement. methodology learning design four key changes were made to the design of the module in light of the evaluation and literature review. the first was fairly obvious: the online module would no longer follow the face-to-face timetable, but rather each session would run for 3 weeks instead of one. secondly, a low-stakes way of signalling to participants that, while the discussions were not summatively assessed, they were a core part of the learning was devised. http://dx.doi.org/10.25304/rlt.v27.2240 c.r. nolan-grant 6 citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 (page number not for citation purpose) this took the form of a ‘reward’ system: participants needed to contribute to the current session’s discussion board at least once to see the next session’s content on the date it was released (or immediately, if they contributed after the next session had begun). this method avoided issues of both quantitative surveillance and qualitative judgement (gulati 2008) – in fact there was no manual gatekeeping at all. the type or length of the post did not matter; there was no time constraint on posting; and working ahead was not an option regardless of discussion board activity (which also encouraged participants to work through the module as a group rather than independently). it was recognised that one danger of this method would be ‘monologue’ posting (i.e. participants posting their own work without engaging with previous or subsequent discussion), as gafni and geri (2010) found. however, their study also revealed that students for whom contribution was compulsory still evidenced significantly more learning gain than their peers for whom discussion contribution was optional, even when they did not engage in the discussions. the third change was to increase tutor involvement. participants were prompted to post everything on the discussion board and the tutors aimed not only to increase their engagement as far as number of posts was concerned, but also to take an adaptive approach, supporting, encouraging, challenging or hanging back as needed. the final change was to the video content. there was neither time nor resource to record new, bespoke videos and to produce captions. but there was capacity to divide each video into two or three shorter videos, interspersed with other course materials. then, on the basis that learners are more likely to engage when prompted, but not forced (delen, liew, and willson 2014), short questions were inserted into the videos at intervals of 10 min or less. the questions were all open-ended, requiring the participants to think, reflect and make notes and it was explicitly stated that the notes would later be used to contribute to the discussion boards. participants were then given the option to click ‘done’ if they had completed the task or ‘i’ll do it later’, after which the video would continue to play. this allowed them to take as little or as much time as they wanted, and also gave an indication of whether participants were engaging with the questions. the discussion prompts which followed the videos then made direct reference to one or more of the embedded questions, encouraging participants to use the notes that they had made during the videos to write their discussion board posts. figure 2 shows how this completed the loop for each session. this cycle also illustrates how, at the start of the module, it was possible to roughly align each type of coi presence with the main activities, as shown in figure 3. as the cycle continues over time, however, we would expect these categorisations to blur – both because the activities are designed to be interrelated and because the types of presence themselves are interrelated. for example, video interaction involves teaching presence and cognitive presence, but increasingly also social presence as the participants begin to anticipate sharing their responses on the discussion board. the timeand activity-dependent release of module content certainly begins with teaching presence, but encourages social presence by creating a shared rhythm of learning, which, in turn, cultivates cognitive presence. participants data associated with a total of 77 participants were analysed, 45 from the 2016 to 2017 delivery of the module (cohort 1) and 32 from the 2017 to 2018 delivery (cohort 2). this represents all registered participants who logged into the vle at least once. participation in the module was optional, although some participants http://dx.doi.org/10.25304/rlt.v27.2240 research in learning technology citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 7 (page number not for citation purpose) were required or encouraged to complete the module by their employers. as only a limited amount of demographic data was available, and as the small sample sizes made it unlikely that demographic correlations would be statistically significant, demographic data were not collected. tutor activity on discussion boards was also included in the analysis. one tutor facilitated group a in cohort 1 and another group a in cohort 2; a third tutor facilitated group b in both 2016–2017 and 2017–2018. figure 2. cycle of session activity. figure 3. initial mapping of the three coi presences onto the cycle of session activity. http://dx.doi.org/10.25304/rlt.v27.2240 c.r. nolan-grant 8 citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 (page number not for citation purpose) frame of analysis because the coi framework was used as a high-level design model (rather than a prescriptive formula), it also provided a theoretical framework for analysis without the risk of asking circular questions. thus, the research questions are: • did the changes that were introduced have an impact on cognitive and social presence? • was the impact (if any) evident across presences? measures of presence several quantitative and qualitative measures were used as indicators of teaching, social and cognitive presence. while these measures by no means tell the whole story of individual or group presence, they were chosen to reflect the original issues that were identified in the 2016–2017 pilot, namely, lack of activity in discussions and video format. the first measure, linked to social presence, was number of online posts (kurucay and inan 2017; parks-stamm, zafonte, and palenque 2017), which was then broken into new posts (answering a prompt without reference to others’ posts) and replies (direct responses or references to others’ posts). for cohort 1, both discussion board posts and blog posts and comments were counted (as the tasks that were done on the blog in 2016–2017 and moved to the discussion board the next year had not changed, this was deemed to be a consistent comparison). to ensure that the change in tutor engagement (a measurable form of teaching presence) had been significant, number of tutor posts, as well as numbers new tutor posts and replies, were also measured. similarly, several quantitative measures were used as indicators of a form of cognitive presences, namely, engagement with the videos (geri, winer, and zaks 2017; guo, kim, and rubin 2014), including average proportion of videos watched and whether participants reported completing the interactive elements in 2017–2018. one advantage to using the same video content was that the only differences between the first and second module deliveries were the changes to the learning design described above, making it more likely that any significant differences in watching behaviour were due to these changes. the third measure was that of correlation between number of discussion board posts, proportion of videos watched and submission of the summative assessment. this follows garrison, anderson and archer (2010) in postulating interconnections between teaching, social and cognitive presence, resulting in increased participant engagement. as shown above, it was anticipated that changes to one element of the module could very well influence engagement with another element. therefore, it was not deemed appropriate to attempt to measure whether discussion changes affected social presence, or whether video changes affected cognitive presence, but whether all of the interventions taken together had an impact on these aspects of module engagement. data collection and analysis discussion board and blog data were collected from the vle (blackboard) via the built-in analytics tools and by manual categorisation of posts. video data were collected from the video platform (kaltura) analytics tool. there was no missing data, although analytical data for two videos were omitted from this study completely as http://dx.doi.org/10.25304/rlt.v27.2240 research in learning technology citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 9 (page number not for citation purpose) the videos were shared with students taking another module. all data were anonymised upon download and stored securely throughout analysis. excel was used to organise the data and produce descriptive statistics and spss for analysis. as the data were not normally distributed (as per the kolmogorov–smirnova test for normality), non-parametric independent samples tests (mann–whitney u and kruskal–wallis) were run to compare the two cohorts. likewise non-parametric correlation tests (spearman) were used to identify possible relationships. results and discussion completion and attrition withdrawals, deferrals and non-submissions were not significantly different between the two cohorts. it should be noted that, as the population in both cases was taken to be all participants who had logged into the vle, calculated averages are potentially misrepresentative. (for example, participants who withdrew early would not be expected to continue contributing to the discussion board or watching videos.) as far as comparing the two cohorts is concerned, however, the data are consistent. discussion boards tutor engagement it was first necessary to ensure that tutor interaction with participants had indeed increased as per the intended changes to the learning design. a weighted average of tutor replies was therefore compared to the total number of participant posts. a significant increase (p < 0.001) in tutor replies in 2017–2018 was found, as shown in table 1. number of total posts there was a significant increase (p = 0.016) in the number of participant posts from 2016–2017 to 2017–2018, with cohort 1 averaging 6.33 posts per participant, while cohort 2 averaged 12.75. figure 4 shows how number of posts per participant compared session by session. this suggests that the changes made did have an effect on discussion engagement. combined with the significant increase in tutor replies to participant posts, these findings are similar to those of parks-stamm, zafonte and palenque (2017) in terms of a relationship between tutor and student participation in small groups. new posts and replies when posts were divided into two categories, new posts and replies to others’ posts, a significant increase in both new posts (p = 0.022) and replies (p = 0.031) was also table 1. tutor replies to participant posts. total participant posts tutor replies average tutor replies per participant post cohort 1 285 37 0.13 cohort 2 408 172 0.42 http://dx.doi.org/10.25304/rlt.v27.2240 c.r. nolan-grant 10 citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 (page number not for citation purpose) found, as shown in table 2. the greater number of replies in cohort 2, both compared to cohort 1 and to cohort 2’s new posts, indicates that participants did not appear to be ‘monologue’ posting (gafni and geri 2010), which would have been evident if there were a larger number of new posts than replies. the fact that, on average, participants contributed more posts than they needed to reveal new content also indicates engagement with the discussion beyond the minimum requirements. forum structure one unexpected finding emerged when comparing the two tutor groups within each cohort. in 2016–2017, there was a significant difference between the two tutor groups’ levels of engagement: group a had significantly more total posts (p = 0.002), new posts (p = 0.002) and replies (p = 0.002) than group b, even though group membership numbers were comparable. a possible explanation for the differences was that the group a tutor only created one starter thread, meaning that participants took it figure 4. average number of posts per participant by session. table 2. average number of new posts, replies and total posts per participant by session. cohort 1 averages per participant cohort 2 averages per participant session new posts (3.31 total) replies (3.07 total) total posts (6.33 total) new posts (5.78 total) replies (6.97 total) total posts (12.75 total) 1 1.80 0.49 2.29 1.71 1.29 3.00 2 0.54 0.39 0.93 0.90 0.94 1.84 3 0.39 0.54 0.93 0.90 1.29 2.19 4 0.34 0.56 0.90 0.68 1.16 1.84 5 0.27 0.61 0.88 0.45 0.71 1.16 6 0.27 0.22 0.49 0.68 1.23 1.90 7 0.24 0.29 0.54 0.65 0.58 1.23 http://dx.doi.org/10.25304/rlt.v27.2240 research in learning technology citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 11 (page number not for citation purpose) upon themselves to create new threads within each session’s forum, whereas the group b tutor created starter threads for each session, and participants limited their posts to these threads. thus, group a also had significantly more discussion board threads (p < 0.001). in 2017–2018, the two tutors took the same approaches as the year before, and while the difference in number of posts was not statistically significant, group a again created significantly more threads than group b (p = 0.019). a summary of the results is shown in table 3. this inverse relationship between tutor-created discussion threads and participant engagement goes against some previous findings (salmon 2004; zydney, denoyelles, and kyeong-ju seo 2012). one explanation could be that mature learners who teach mature learners themselves are confident in taking control of discussions early on: the requirement to post on the discussion board is sufficient scaffolding, and so autonomy on the board itself is more conducive to discussion (gulati 2008). video proportion of videos watched ninety-one per cent of participants in cohort 2 began watching at least one core video, a relative increase over cohort 1 – only 71% of whom started watching any videos. video drop-off rates (the proportion of video content watched once the video is played) increased significantly for core videos in 2017–2018 (p = 0.001). while cohort 1, on average, watched 50% of a core video once started, cohort 2 had an average drop-off rate of 63%. figure 5 shows a comparison of drop-off rates in four clusters. this is a strong indication that the changes made to the module design did encourage the participants to remain engaged for longer. this also supports geri, winer and zaks (2017) finding that interactivity may increase learners’ attention spans and delen, liew and willson (2014) conclusion that interactive options could help learners’ persistence and engagement. interactivity cohort 2’s responses to the questions that appeared during the videos showed that they reported engaging with the prompts 77% of the time, as detailed in table 4. this increased dramatically from the first video to the second. this could have been caused by the wording of the prompts: the session 1 prompt asked participants to post on the discussion board, whereas the subsequent prompts asked participants to make note of their thoughts to post on the discussion board later. removing the outlying first video, reported completion of the rest of the questions was 83%. this is another indication that the changes made to the module had a real effect on participants’ engagement. table 3. total threads and posts by cohort and group. cohort 1 cohort 2 group a group b group a group b total threads 70 10 58 14 total posts 191 34 287 124 average posts per thread 2.73 3.4 4.95 8.86 http://dx.doi.org/10.25304/rlt.v27.2240 c.r. nolan-grant 12 citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 (page number not for citation purpose) taken together, these results indicate that the first research question can be answered in the affirmative: the changes that were introduced did have an impact on cognitive and social presence as measured by engagement with discussion boards and videos. interrelatedness of discussion, video viewing and module completion correlations were found among discussion board and blog posting, video viewing and summative assignment submission for both cohorts. total video viewing time and total number of posts were strongly correlated (p < 0.001), as were total video viewing time and module completion (p < 0.001). number of discussion board and blog posts was also strongly correlated to submission of the final assessment (p < 0.001). an overview is provided in table 5. while these correlations reveal that discussion and video engagement were related to each other and to module completion just as strongly in 2016–2017 as in 2017–2018, they also confirm the value of encouraging engagement with these two dimensions of the module. this is especially valid as, in both cases, it was possible to submit a summative assignment without table 4. proportion of viewers reporting whether they completed activities by session. session (video number) ‘done’ (77% overall) (%) ‘later’ (23% overall) (%) 1 (3) 21 79 2 (1) 74 26 3 (1) 94 6 3 (2) 97 3 4 (1) 73 27 4 (2) 82 18 5 (1) 88 12 5 (2) 86 14 6 (1) 73 27 7 (2) 78 22 figure 5. average proportion of videos watched once played (drop-off rate). http://dx.doi.org/10.25304/rlt.v27.2240 research in learning technology citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 13 (page number not for citation purpose) having contributed to the discussions or watched the videos at all. these significant relationships between module activity and assignment completion support kurucay and inan’s (2017) results, indicating that frequency of online interaction is related to achievement, and to zhang et al.’s (2006) findings that video with interaction is linked to learning gain. moreover, they point to a certain degree of success in iteratively developing social and cognitive presences through teaching presence. the second research question can therefore be addressed: the impact of the changes was evident across teaching, social and cognitive presence. conclusion this study showed how the coi framework was used to address issues with engagement in a postgraduate online module. teaching presence in the form of an enhanced learning design linked together input elements (video), discursive elements (discussion boards) and access to new materials to encourage cognitive and social presence. the impact of these changes was measured using discussion and video analytics, on the basis that increased activity would be indicative of increased presence. findings confirmed that there were significant increases in number of discussion posts and in proportion of videos watched, and showed that putting the impetus on participants to create their own discussion threads was linked to increased activity. correlations between discussion activity and watching videos, and between both of these and submission of the summative assessment, were also identified. this showed that these module elements were indeed closely related, thus suggesting a strong interplay among teaching, social and cognitive presence. limitations this study looked at a small population which may not be representative of students in online modules generally, and which made it difficult to confirm statistical significance when comparing two cohorts or groups. while it was unlikely that, given the small population size, demographic data would have revealed significant differences, it could have better situated the participants in this study within their cohorts and within wider population of students in online courses. there were also limitations in data collection. cohort 2 participants were not available for interviews or focus groups which could have provided a more qualitative, nuanced perspective on their approach to online video and discussion. coding or otherwise qualitatively analysing discussion posts might also have given a deeper picture of how participants were interacting online. table 5. correlations among key metrics. correlations between total video viewing time and total posts correlations between total video viewing time and module completion correlations between total posts and module completion cohort 1 cohort 2 cohort 1 cohort 2 cohort 1 cohort 2 correlation coefficient (spearman's rho) 0.787 0.810 0.560 0.707 0.560 0.707 sig. (2-tailed) 0.000 0.000 0.000 0.000 0.000 0.000 n 45 32 45 32 45 32 http://dx.doi.org/10.25304/rlt.v27.2240 c.r. nolan-grant 14 citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 (page number not for citation purpose) further research this study demonstrated the potential benefits of further research into conditional release of online content based on student contribution – there is evidence that this is an effective way to increase engagement, but it has not often been addressed in the literature. as interaction within video becomes increasingly ubiquitous in higher education settings, broader evaluation of types of interaction and learner engagement would also be valuable. finally, survey instruments to explore the coi framework in online and blended contexts have emerged in recent years (stenbom 2018), which could provide a fuller picture of student experience through the coi lens. references chen, k.-c. & jang, s.-j. (2010) ‘motivation in online learning: testing a model of self determination theory', computers in human behavior, vol. 26, no. 4, pp. 741–752. doi: 10.1016/j.chb.2010.01.011 cherney, m. r., fetherston, m. & johnsen, l. j. (2018) ‘online course student collaboration literature: a review and critique’, small group research, vol. 49, no. 1, pp. 98–128. doi: 10.1177/1046496417721627 costley, j., hughes, c. & lange, c. (2017) ‘the effects of instructional design on student engagement with video lectures at cyber universities’, journal of information technology education: research, vol. 16, pp. 189–207. doi: 10.28945/3728 delen, e., liew, j. & willson, v. (2014) ‘effects of interactivity and instructional scaffolding on learning: self-regulation in online video-based environments’, computers & education, vol. 78, pp. 312–320. doi: 10.1016/j.compedu.2014.06.018 gafni, r. & geri, n. (2010) ‘the value of collaborative e-learning: compulsory versus optional online forum assignments’, interdisciplinary journal of e-learning and learning objects, vol. 6, pp. 335–339. garrison, d. r., anderson, t. & archer, w. (2010) ‘the first decade of the community of  inquiry framework: a retrospective’, internet and higher education, vol. 13, no. 1–2, pp. 5–9. doi: 10.1016/j.iheduc.2009.10.003 geri, n., winer, a. & zaks, b. (2017) ‘a learning analytics approach for evaluating the impact of interactivity in online video lectures on the attention span of students’, interdisciplinary journal of e-skills and lifelong learning, vol. 13, pp. 215–228. gulati, s. (2008) ‘compulsory participation in online discussions: is this constructivism or normalisation of learning?’, innovations in education and teaching international, vol. 45, no. 2, pp. 183–192. doi: 10.1080/14703290801950427 guo, p. j., kim, j. & rubin, r. (2014) ‘how video production affects student engagement: an empirical study of mooc videos’, first acm conference on learning@ scale conference, atlanta, ga, pp. 41–50. hrastinski, s. (2008) ‘what is online learner participation? a literature review’, computers & education, vol. 51, pp. 1755–1765. doi: 10.1016/j.compedu.2008.05.005 jones, a. & issroff, k. (2005) ‘learning technologies: affective and social issues in computer-supported collaborative learning’, computers & education, vol. 44, pp. 395–408. doi: 10.1016/j.compedu.2004.04.004 kahn, p., et al., (2017) ‘understanding student engagement in online learning environments: the role of reflexivity | springerlink’, educational technology research and development, vol. 65, no. 1, pp. 15. doi: 10.1007/s11423-016-9484-z kovanović, v., et al., (2015) ‘analytics of communities of inquiry: effects of learning technology use on cognitive presence in asynchronous online discussions’, internet and higher education, vol. 27, pp. 74–89. doi: 10.1016/j.iheduc.2015.06.002 http://dx.doi.org/10.25304/rlt.v27.2240 research in learning technology citation: research in learning technology 2019, 27: 2240 http://dx.doi.org/10.25304/rlt.v27.2240 15 (page number not for citation purpose) kurucay, m. & inan, f. a. (2017) ‘examining the effects of learner-learner interactions on satisfaction and learning in an online undergraduate course’, computers & education, vol. 115, pp. 20–37. doi: 10.1016/j.compedu.2017.06.010 lee, h., kim, j. w. & hackney, r. (2011) ‘knowledge hoarding and user acceptance of online discussion board systems in elearning: a case study’, computers in human behavior, vol. 27, no. 4, pp. 1431–1437. doi: 10.1016/j.chb.2010.07.047 luo, n., zhang, m. & qi, d. (2017) ‘effects of different interactions on students’ sense of community in e-learning environment’, computers & education, vol. 115, pp. 153–160. doi: 10.1016/j.compedu.2017.08.006 moule, p. (2007) ‘challenging the five-stage model for e-learning: a new approach’, research in learning technology, vol. 15, pp. 30–50. doi: 10.1080/09687760601129588 nerantzi, c. & gossman, p. (2015) ‘towards collaboration as learning: evaluation of an open cpd opportunity for he teachers’, research in learning technology, vol. 23. doi: 10.3402/ rlt.v23.26967 ouyang, f. & scharber, c. (2017) ‘the influences of an experienced instructor’s discussion design and facilitation on an online learning community development: a social network analysis study’, the internet and higher education, vol. 35, pp. 34–47. doi: 10.1016/j. iheduc.2017.07.002 palincsar, a. s. (1998) ‘social constructivist perspectives on teaching and learning’, annual review of psychology, vol. 49, pp. 345–376. doi: 10.1146/annurev.psych.49.1.345 parks-stamm, e. j., zafonte, m. & palenque, s. m. (2017) ‘the effects of instructor participation and class size on student participation in an online class discussion forum’, british journal of educational technology, vol. 48, no. 6, pp. 1250–1259. doi: 10.1111/bjet.12512 salmon, g. (2004) e-moderating, 2 edn, routledge, london. stenbom, s. (2018) ‘a systematic review of the community of inquiry survey’, the internet and higher education, vol. 39, pp. 22–32. doi: 10.1016/j.iheduc.2018.06.001 zhang, d., et al., (2006) ‘instructional video in e-learning: assessing the impact of interactive video on learning effectiveness’, information & management, vol. 43, no. 1, pp. 15–27. doi: 10.1016/j.im.2005.01.004 zydney, j. m., denoyelles, a. & kyeong-ju seo, k. (2012) ‘creating a community of inquiry in online environments: an exploratory study on the effect of a protocol on interactions within asynchronous discussions’, computers & education, vol. 58, no. 1, pp. 77–87. doi: 10.1016/j. compedu.2011.07.009 http://dx.doi.org/10.25304/rlt.v27.2240 doi:10.3402/rlt.v23.28507 the space for social media in structured online learning gilly salmon a , bella ross b *, ekaterina pechenkina c and anne-marie chase d a office of the pro vice-chancellor (education innovation), the university of western australia, crawley, wa, australia; b student academic support unit, monash university, monash, caulfield, vic, australia; c learning transformations unit, swinburne university of technology, hawthorn, vic, australia; d australian council for educational research, camberwell, vic, australia (received 12 may 2015; final version received 23 november 2015) in this paper, we explore the benefits of using social media in an online educational setting, with a particular focus on the use of facebook and twitter by participants in a massive open online course (mooc) developed to enable educators to learn about the carpe diem learning design process. we define social media as digital social tools and environments located outside of the provision of a formal university-provided learning management system. we use data collected via interviews and surveys with the mooc participants as well as social media postings made by the participants throughout the mooc to offer insights into how participants’ usage and perception of social media in their online learning experiences differed and why. we identified that, although some participants benefitted from social media by crediting it, for example, with networking and knowledge-sharing opportunities, others objected or refused to engage with social media, perceiving it as a waste of their time. we make recommendations for the usage of social media for educational purposes within moocs and formal digital learning environments. keywords: moocs; social media; learning design; online identity; facebook; twitter introduction: social media in learning in this paper, we explore the role played by social media in a massive open online course (mooc) offered by an australian university. social media sites are increasingly being used for educational purposes and a range of benefits and drawbacks have been documented in the research. we examine how the usage of social media in the mooc enhanced participants’ overall learning experience and how it led to increased networking and knowledge sharing with peers. we also report on the negatives of social media usage as perceived by the participants. these lessons may inform future design choices of the inclusion of social media in moocs and other structured digital learning. social media in higher education has been found to enhance learning outcomes and academic achievement (junco, heiberger, and loken 2010; özmen and atici 2014), and to contribute to knowledge construction (dewitt et al. 2014; kassens-noor 2012). social media can assist students to share administrative information with peers, such as meeting times and locations, and assessment requirements (bosch 2009; selwyn 2009), and also to network and promote peer feedback (davies et al. 2010). *corresponding author. email: bella.ross@monash.edu responsible editor: carlo perrotta, school of education, university of leeds, united kingdom. research in learning technology vol. 23, 2015 research in learning technology 2015. # 2015 g. salmon et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 when deployed for learning, social media can facilitate the development of online communities, allowing for collaborative and participatory engagement by placing emphasis on collective knowledge and social interaction (maloney 2007; wodzicki, schwämmlein, and moskaliuk 2012). social media can help strengthen the social relationships among students, heighten students’ self-esteem, and boost their learning performance (llorens and capdeferro 2011; yu et al. 2010). students may be more willing to voice their opinions or disagreement with peers in an online discussion rather than in a face-to-face setting (ellison, steinfield, and lampe 2007; kim 2008; oblinger and oblinger 2005; sullivan 2002; thompson and ku 2005). further, online social interaction allows shy students to contribute and be ‘heard’ by the group (davies et al. 2010) and thus may provide benefits to those with lower self-esteem (ellison, steinfield, and lampe 2007). although university-utilised learning management systems (lmss) and social media platforms both enable file sharing, collaboration and discussion (gray, annabell, and kennedy 2010), social media platforms tend to be more popular with students for peer-to-peer interactions (davies et al. 2010; veletsianos and navarrete 2012) due to their familiarity and flexibility. peer learning communities (off and online) have been identified as a way to foster the development of higher order thinking skills and increase student and academic engagement, interaction, retention, and satisfaction (brownell and swaner 2010; dodge and kendall 2004; yuan and kim 2014). in moocs, where engagement and motivation tend to be low (de freitas, morgan, and gibson 2015; yang et al. 2013), social media may be beneficial in fostering online learning communities, which, in the context of a mooc, are necessarily located online, enabled by an lms (also referred to as virtual learning environment) or social networking site. online learning communities have been found to engage students in collaborative learning and reflective practice (oliver et al. 2007). the research into online learning communities should, however, be viewed with caution, as the term is often used without clear or common definitions or rigorous theoretical underpinnings (henderson 2015). the term is often used uncritically to describe collaboration with overly positive overtones of social support. some researchers offer definitions for the online learning community (lai 2015): the participants have a shared goal, support one another, produce material collaboratively, show a sense of belonging to the community, and are interested in the welfare of its members. excellent as these qualities are, current research indicates that they are neither aspired to nor achieved by the design of online communities. consequently, this paper aims to examine how participants leveraged social media to network with others to exchange information and knowledge. many of the positives of deploying social media in digital education, outlined above, have also been challenged, with some studies highlighting that there is no adequate evidence that social networks provide an arena for all students to develop critical and independent thinking skills (henderson, selwyn, and aston 2015; ziegler 2007). other studies show that students feel that social media reduces student collaboration because students work separately on different parts of a project, which minimises opportunities for collaborative learning (hrastinski and aghaee 2012). students also report that using social media in learning may lead to misunderstandings, less knowledge sharing, and less creative thinking (hrastinski and aghaee 2012). furthermore, the quality and accuracy of the information shared in collaborative social media spaces varies greatly (laird 2014). g. salmon et al. 2 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 some students report seeing the use of social media such as facebook as time wasting or a form of procrastination (davies et al. 2010). studies such as madge et al. (2009) argue that students are not keen to use facebook for formal discussions relating to teaching and learning, or to liaise with instructors. students may feel that using social media in education distorts the boundaries between their online social and educational worlds, between social and informal and formal spaces, and between public and private spaces (land and bayne 2008). both students and academics may feel that social media for educational purposes interferes with their private lives, which raises concerns about representing identities in the online environment (aghili et al. 2014; tess 2013; veletsianos and navarrete 2012). particular to moocs, participants may feel overwhelmed and as just one ‘faceless student’ out of many (knox 2014). participation in moocs can provoke anxiety in a learner about their presence and orientation in relation to large-scale activity (kop, fournier, and mak 2011), and can cause a sense of loss of identity and individuality (knox 2014). participants have described moocs as depersonalised and lacking a meaningful student-teacher relationship (levinson 2013). students in moocs may feel anxious that they are missing out on important information and discussions; the many online opportunities and ‘spaces’ of the course can cause students to feel confused and overwhelmed (knox 2014). despite a large body of scholarly work on moocs and digital education, with broader and quickly growing scholarship focusing on the educational use of social media, there is still a scarcity of studies specifically looking into the usage of social media in moocs. this paper seeks to contribute to this gap of knowledge and offer some insights into how social media can be successfully deployed in a mooc. ‘how and to what extent does social media enhance or complement participants’ overall learning processes and outcomes?’ is the key question that drives us to be particularly interested in investigating two outcomes. first, why did the cd mooc participants chose to spend time in social media platforms in addition to the structured lms provided? second, what was the impact, if any, on their learning, experience and engagement? the context for the study: the carpe diem mooc the carpe diem massive open online course (cd mooc) was designed and delivered by swinburne university of technology (australia) in 2014. the mooc was designed to attract educators from all sectors interested in designing their own courses. the goal of the cd mooc was to equip participants with the knowledge and skills to develop their own online and blended teaching materials and processes through a practical design approach. just over 1,000 participants accessed the cd mooc. they were mainly in the 46-year-plus age bracket; employed in the education sector; and the majority were female (salmon et al. 2015). participants were allocated into small online groups, and tasked with designing and planning their own courses with a focus on student-centred learning, and using innovative learning technologies. there was a high level of engagement and completion. the pedagogy of the carpe diem process is a socially mediated constructivist approach that uses rapid project development and creative problem-solving techniques for quick and collaborative design (salmon 2013, 2015; salmon and wright 2014; salmon et al. 2015). the cd mooc aimed to provide participants with an experience research in learning technology citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 of working through the six stages of the carpe diem learning design process: write a blueprint, design a storyboard, build a prototype, check reality, review and adjust, and build an action plan. the carpe diem process, which usually takes place over 2 days with co-located participants, was adjusted to suit an asynchronous digital open course. hosted on the free coursesites (provided by blackboardtm) lms, the cd mooc process was defined by extensive group work and peer review. at the beginning of the cd mooc, participants were randomly enrolled into groups of approximately 30 participants, and were guided and supported by a mooc moderator over the 6-week period. in addition to the structured lms, more informal social media elements were offered. the cd mooc designers selected facebook and twitter as social media platforms for participants to use, based on the studies that social media contributes to collaborative learning experiences (balakrishnan 2013; blaschke 2014; llorens and capdeferro 2011; maloney 2007; yu et al. 2010). the facebook group and the twitter hashtag (#cdmooc) were created in february 2014 to coincide with the beginning of the cd mooc. both social media outlets were open to all cd mooc participants. the facebook group was a closed group that required participants to request to be added by the carpe diem facebook page moderators. both the cd mooc facebook group and the twitter hashtag #cdmooc were extensively advertised to participants through general announcements on the cd mooc’s coursesites page, tweets via the cd mooc’s official twitter account and through the text-based online and printed literature advertising the mooc. twitter was also linked directly to the cd mooc facebook group by including the twitter hashtag in facebook posts. on the facebook group, moderators guided participants to ask questions about the cd mooc, seek practical help, communicate and discuss issues around work tasks, and share links to online group work and resources. twitter was used by both the cd mooc team and participants to share practical information and resources, while also encouraging participants to share their thoughts and experiences. methodology the data for this study were collected via an anonymous online survey, interviews with a randomly selected group of survey respondents and twitter and facebook postings by the cd mooc participants throughout the duration of the mooc. these methods were selected as they are mutually enriching by allowing for an in-depth investigation of topics of inquiry initially revealed by the survey responses. the survey consisted of six likert-scale questions, two write-in general feedback questions and six demographic questions. the survey questions covered aspects such as the students’ evaluation of the usefulness of the cd mooc and their engagement with its various elements, including its length, structure, the use of videos, resources and digital badges. the write-in questions directed the respondents to identify the most useful aspects of the cd mooc and explain why the features were useful. the survey was sent out to the entire cd mooc participant cohort and was completed by 155 respondents. no survey questions were specifically designed to gauge students’ engagement with social media elements (facebook and twitter), however, the theme of social media naturally emerged from the write-in responses data. to further explore how social g. salmon et al. 4 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 media in the cd mooc influenced students’ educational experiences, a question about social media was included in the interview schedule, with the ultimate goal of correlating interviewees’ responses on the usage of social media in the cd mooc with the survey findings. after the survey completion, 29 of the survey respondents consented to be interviewed. semi-structured telephone interviews were conducted, transcribed and de-identified. in regards to the interview respondents’ use and experiences of social media in the cd mooc, we asked whether interviewees felt motivated to engage in informal interaction with their peers in online spaces outside of the cd mooc structured environment, and whether or not, or to what extent, they felt social media helped their learning. interviewees were also asked whether they interacted with other cd mooc participants on facebook or twitter, and whether they thought that social media improved their learning outcomes. we utilised the analytical software dedoose to conduct a thematic analysis of the interview transcripts, open-ended survey responses and social media postings, in which social media was identified as one of the frequently discussed topics. the analysis of the data collected from the three sources was guided by the mixed-methods approach (bazeley 2012), in which we triangulated the findings to facilitate the process of concept formation and identify frequently mentioned concepts and causal inferences (strauss and corbin 1998), though the latter was problematic considering the small size of the samples. another level of triangulation was achieved by having data systematically (co)analysed and (co)checked by the authors. the processes of data triangulation produced high levels of inter-reliability through comparisons, moderation and thorough discussions of the data and any inconsistencies that arose between the authors. throughout the reporting of findings, we treat the triangulated data as one body of knowledge. the three key themes which emerged from the body of data are discussed next. findings participation in social media having first emerged as a major theme from the survey’s write-in responses data, social media then became one of the main topics of inquiry investigated via the interviews. the social media postings data revealed that facebook was most frequently used by the cd mooc participants, compared to the microblogging site twitter. participation in the facebook group continued actively for around 3 months (up to may 2014) with some participants ‘signing off ’ by displaying their certificate of completion. the twitter hashtag attracted 664 tweets over a 9-month period. initially, twitter was used by participants to announce their intention to participate in the cd mooc, and then during the mooc to gather, provide and exchange resources and information. interview participants were specifically asked whether they used social media throughout the cd mooc and if so, whether they used facebook or twitter, or both, and how they engaged with either platform. the usage data quantified on the basis of interview transcripts (table 1) shows that approximately half of all interviewees used at least one of the social media platforms in the cd mooc. the most used form of social media was facebook (used by 31%), a small minority (3%) used twitter only, while 14% of interviewees utilised both facebook and twitter. research in learning technology citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 confusing the findings from the tess’s (2013) study, the interviews clarified that the different social media platforms were used for different purposes by the cd mooc participants. the facebook group was used for discussions, while twitter was used for gathering and providing information. emerging themes analysis of the survey’s write-in responses and the interview narratives helped us identify three key themes that emerged in relation to participants’ use of and engagement with social media in the cd mooc. the first theme revealed that the online collaboration through social media assisted participants by enhancing their learning. the second theme showcased that many participants enjoyed and benefitted from using social media to engage with a diverse range of people with whom to network and exchange knowledge. finally, the third theme illuminated the reasons behind participants’ objections or reluctance to use social media within their structured online learning experience. theme 1: improving learning approximately half (14) of the interviewed participants used the cd mooc facebook group during the duration of the mooc, mainly with the intention of improving their learning outcomes. these participants reported various positive social aspects of their learning process enhanced by using facebook, confirming other studies (brownell and swaner 2010; dodge and kendall 2004; kassens-noor 2012). participants who used the cd mooc facebook group, but did not use twitter, thought that social media was useful for posting and gathering information, and that facebook improved their learning outcomes by facilitating discussion with peers and moderators. facebook was also seen as an easy aid to increase learning and encourage discussions with peers as it facilitated the sharing of videos and images, and hence allowed participants to directly see others’ work. facebook also served as a spark for a discussion, which could then be built upon within the more structured mooc environment. one participant noted: for me, the most useful activities were designing and giving feedback on the e-tivity, an opportunity to use the templates provided. aside from that, [the most useful aspects were] the enormous amount of discussion generated through various media both within and outside the course environment. a number of interview participants stated that they were unable to use social media as much as they would have liked, as a result of either a lack of time or lack of interest table 1. interview responses: did cd mooc participants use social media? did participants use social media during the cd mooc? percentage (%) of interview cohort (n �29) no 41 yes, all � twitter and facebook 14 yes � facebook only 31 yes � twitter only 3 question not asked 10 g. salmon et al. 6 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 among other cd mooc group members in participating in the facebook group. despite their lack of engagement with facebook, these participants perceived social media as being helpful to their collaborative learning efforts whether they engaged with either facebook or twitter. another perceived benefit of participants engaging in social media was more knowledge sharing � it was suggested that the large amount of discussion generated on facebook was one of the most useful aspects of the cd mooc experience. theme 2: networking both survey and interview participants spoke of the benefits of networking with people from different backgrounds, locations and professional affiliations when using social media platforms to connect and further their learning outcomes in the cd mooc. one participant stated: i love the fact that . . . you’re networking with people all over the world and just such an incredibly empowering thing. a lot of my colleagues feel that because they work in . . . a government education department they should only really talk to people . . . in their own area and i just love that whole knowledge of being able to . . . communicate with someone in ireland and . . . someone in israel and it’s very empowering i think. and it also made me feel good because they know lots of stuff, so i felt really confident and comfortable. another participant mirrored the sentiment above: ‘collaborating with participants worldwide made it a great cultural experience, particularly in seeing how social media is taking on the world’. another confirmation of this sentiment came from a tweet: ‘[our] team are networking as well #cdmooc finding interesting connections from the online world!’ finally, as another participant stated: learning from peers and using learning platforms such as discussion boards, wiki, the cd mooc and others enhanced this process and helped make this happen! i personally found this experience, although challenging at times, useful and interesting. it kept me engaged and motivated to explore and engage with others. i particularly enjoyed freely sharing knowledge on a wide and global scale!! these responses, provided by both interview participants and survey respondents as well as interspersed throughout the facebook and twitter feeds, exemplify how strongly the cd mooc participants felt that they were part of a larger online learning cohort with shared goals, and in a collaboration with peers on social media that aided their learning and in general enriched the cd mooc experience. theme 3: objections to using social media despite the positive comments outlined in the previous section, a significant segment of the cd mooc participants who took part in interviews (41%) did not use any forms of social media as part of the cd mooc experience. the main objections to using social media can be divided into three categories: a belief that social media might be a waste of time; the perception of social media platforms as confusing or intimidating; and concerns about blurring social and professional identities. these objections are discussed in the following sections. research in learning technology citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 time to engage with social media among reasons for not engaging with social media in the cd mooc, a lack of time emerged with a systematic frequency: these participants believed that social media would ‘take time away’ from what could otherwise have been spent on the structured parts of the cd mooc, a concern that is reflected in other studies (davies et al. 2010). within this group, some were negative towards social media in general, seeing it as a waste of time, both inside and outside of the cd mooc. this sentiment among participants is exemplified by the interview excerpt below in which a participant did not engage with social media for lack of time; he thought that time spent on social media would detract from the time he could spend on coursework. it might be that more time is required if the platform is unfamiliar. it was important to many participants that they were comfortable using social media, however, and they avoided unfamiliar platforms. i made a couple of tweets myself but they were just generally looking forward to doing a cd mooc and everything. i didn’t actually respond to anybody else’s tweets on there but i did have a look every so often. but again that’s another time issue. you know you’re looking on the community and the twitter and all of the social things and you end up tearing off your actual course work [emphasis added]. meanwhile, a number of participants wanted to use the social media with which they were familiar and comfortable. some of these participants were notably less enthusiastic about the use of coursesites lms of the cd mooc, instead preferring to spend time on social media. this finding fits in with the literature emphasising that learners often prefer to use the social media to which they are already accustomed (kaeomanee, rias, and perveen 2012; veletsianos and navarrete 2012). such learners find social media easier to use, or more ‘advanced’ than the institutionally provided lms (davies et al. 2010). a participant who did not interact with others via facebook or twitter attributed this indifference to time constraints and multiple other commitments, and ‘preferred to do [use social media] because it would have given [the cd mooc] a little more structure’. the same participant also stated that it would have been simpler to use facebook, which was familiar, instead of coursesites, which this person described as ‘cumbersome’. participants generally reported being more comfortable using platforms that they were more accustomed to � typically facebook. the participant cited below used facebook to gain knowledge of the cd mooc’s coursework through online peer discussions, however, did not feel comfortable using twitter: i tried to get in touch with the course participants to get informed about the discussion around this course so i registered on facebook and also twitter. the use of facebook is quite helpful but twitter did not work because it was not the social media i am accustomed with. this apparent trend of participants preferring to engage with the cd mooc via social media rather than lms can be explained by the existence of navigational issues with the cd mooc’s lms, another common theme in both surveys and interviews. we conclude that a significant segment of participants did not see the value of social media in the cd mooc, mostly because they were concerned about losing the time they had available to spend on coursework. g. salmon et al. 8 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 confusion or intimidation another objection by participants related to their lack of knowledge about social media platforms and their resulting confusion. a participant voiced strong opinions about the use of facebook and twitter for educational purposes: ‘i absolutely disagree with that [facebook and twitter]. i hate facebook, i don’t think it’s an appropriate forum. i dislike it’. she believed that in order to ‘get as much bang for your buck in as short a time as possible’, all information should be in one place, and information spread over several social media sites can get lost: ‘you want to say it once, say it clearly and keep it all in one place and that way it’s easy to get onto the points’. other participants held similar ideas � they expressed their confusion at the range of online communication channels available during the cd mooc: ‘#cdmooc to tweet or fb? don’t feel i can do both. plus wiki plus group discussion board. collaboration galore!’ this participant echoes this conflict between social media platforms: while it is great to encourage using other forums [outside the coursesites] to collaborate, it created more confusion as we did not know where to go to stay in touch. encouraging a set place as a common thread would have been helpful at the start. in practice, we noted that the cd mooc groups had some difficulty reaching consensus about which platforms to use for their work because of the variety of opinions within the groups about which social media sites were appropriate. however, those participants in groups who used both facebook and twitter found the experience of using both platforms extremely helpful, though not without initial disagreements. for example: we had a bit of a problem in our group in the beginning because instead of thinking about the process we had a couple of people who were arguing about whether social media was any good, so it kind of got a bit out of hand . . ., ‘no, we’re not using facebook because of this, this, this and this’ but they weren’t kind of thinking about what the process was of the actual carpe diem design methodology. rather they were thinking of, ‘do i like social media or don’t i?’ so that kind of overtook that a little bit. such comments are not unusual in the literature on moocs. students may feel that they are ‘missing out’ if they do not follow every thread or post on the connected social media platforms (knox 2014). other research (henderson, selwyn, and aston 2015) tells us that students prefer having all of their course content and materials located in one place, such as the official lms. our study demonstrates, however, that those participants who made a decision to engage with the social media elements of the cd mooc found the experience rewarding. blurring of social and professional identities a third key objection of participants to engaging with the social media elements of the cd mooc dealt with reluctance about what was perceived as a potential merging of professional and social identities. one of these participants, who felt uneasy interacting with other participants using social media, preferred to separate his networking and more formal learning approaches: i did not use twitter or facebook. those are social sites. for professional work, i prefer it to be on a professional platform . . . i don’t use facebook and twitter anymore. this participant saw the cd mooc as an adequate source of course information, stating that: ‘the collaboration messages inside the tools on blackboard are more research in learning technology citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 than sufficient’. comments like these demonstrate that some participants discounted social media as a valuable way to gain knowledge and engage with peers in an online community. another participant stated that using social media for the purposes of the cd mooc felt too intimating to her: ‘i don’t do twitter, facebook . . . i didn’t want to get on facebook with them . . . [it] was a little too intimate’. the reluctance to blur social and professional identities is a common theme emerging from the literature on online learners and mooc participants (aghili et al. 2014; jones et al. 2010; land and bayne 2008; veletsianos and navarrete 2012). based on their findings, jones et al. (2010) recommend that learning design address individual student preferences to either combine or separate their online identities. summary of objections those who reported being uncomfortable using social media cited three main reasons for not wanting to engage in social media for their formal learning. first, they did not believe that there was any learning value in using social media sites, and that it was a waste of their time. second, they found the social media landscape too confusing and intimidating. third, they did not want to blur what they perceived as the socialprofessional divide. conclusions the experience of delivering the cd mooc allowed us to study how participants engaged with the social media within the mooc along with the structured learning provided. our findings show that, although participants enjoyed using social media to support their mooc experience and thought that they had benefitted from using it, some had objections. the benefits included enhancing learning through the social and informal interaction with their peers, and from the online communities that formed around the cd mooc. connecting with peers using social media also strengthened participants’ sense of belonging to the cd mooc cohort. non-institutionally provided digital environments bring both opportunities and challenges to digital learning. amongst the opportunities is the ability for online students to learn anywhere, on an array of devices, on multiple online platforms, and in their own time, allowing for personal flexibility and choice. the challenges of online learning include becoming skillful in and comfortable with new technology, developing ways to relate to and communicate with other learners online, and becoming comfortable about having an online presence and digital identity. most notable was the wide variety of differing views and experiences by the cd mooc participants. the cd mooc participants held contrasting views that highlighted how individual preferences for online learning differ considerably within the same learning cohort. this also highlighted the willingness and benefits for those who did venture outside the structured and provided learning environment. recommendations when designing for moocs or online learning, participants’ preferences for social media use should be taken into account. the design of content and delivery methods g. salmon et al. 10 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 for the intended target audience should consider the cohort’s demographics and background, and use moocs-centred research. such predictors can help in the selection of platforms that are the most appropriate, based on the demographic data of social media usage, for example. social media platforms such as facebook and twitter should be viewed by mooc designers as an addition to a lms rather than a substitution. social media should ideally serve complementary purposes to a lms, ensuring that material is not duplicated. participants should be clearly informed that information communicated on social media does not duplicate anything else already given within the course/official platform, in order to persuade students that social media is not a waste of time. nevertheless, online designers need to ensure that learners are not given too many social media options, as more is not always better. instead, more options may lead to confusion, intimidation, and learners who log off altogether. our study demonstrated that cd mooc participants found too many options of engagement and different online platforms overwhelming. they stated they were often unable to keep up to date with every post or discussion thread due to time restraints, and they worried about falling behind. as mooc participants have busy lives and jobs, and often embark on a mooc in their spare time, moocs should be simplified wherever possible. for this reason, the social media options should be limited in order to avoid information overload and minimise confusion for learners. it may be worth, for example, incorporating a twitter feed directly into the lms site, thereby opening access up to all participants and avoiding the requirement of logging into yet another social media site. several participants cited a lack of time for not engaging with social media in the cd mooc, while others did not believe that social media contributed any value to their mooc experience at all. as a result, it may be useful to outline in detail to students the contributions that social learning can bring to a mooc and, indeed, to any online learning environment. those who believe that conversations on social media are a waste of time may view things differently if they understand how conversations and knowledge sharing with their peers can support their learning experience. online learning should be designed around multiple learner preferences, if possible, to tailor the learning experience to each participant. some learners prefer to separate their social and professional identities, while others are unconcerned about merging their social and professional lives, perhaps because they do not distinguish between the two. mooc designers need to take this into account. one solution is to offer a few different platforms, in addition to the lms, but not require that learners use them if they feel uncomfortable. alternatively, ask learners to create professional identities on social media for all formal learning and professional development uses. we have shown that despite the objections cited by participants, social media can be a powerful tool deployed in moocs and other digital learning: it can boost collaboration, enhance networking efforts, and improve learning experiences and outcomes. ethics this project was approved for human research ethics by swinburne’s human research ethics committee (suhrec) and follows the australian government’s national statement on ethical conduct in human research (2007). the data used in this study was de-identified prior to researchers accessing the primary data to ensure research in learning technology citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 confidential treatment of participants’ data. conflicts of interest were minimal and resolved by employing researchers who were not involved in the deployment of the cd mooc. acknowledgements the authors thank lee partridge and sid nair of the university of western australia for their valuable comments on earlier versions of this paper. references aghili, m., et al., (2014) ‘unifying informal and formal learning environments: educational use of social network sites through implementing community of inquiry framework’, international journal of e-education, e-business, e-management and e-learning, vol. 4, no. 3, pp. 191�196. doi: http://dx.doi.org/10.7763/ijeeee.2014.v4.329 balakrishnan, v. (2013) ‘using social networks to enhance teaching and learning experiences in higher learning institutions’, innovations in education and teaching international, vol. 51, no. 6, pp. 595�606. doi: http://dx.doi.org/10.1080/14703297.2013.863735 bazeley, p. (2012) ‘integrative analysis strategies for mixed data sources’, american behavioral scientist, vol. 56, no. 6, pp. 814�828. blaschke, l. m. (2014) ‘using social media to engage and develop the online learner in selfdetermined learning’, research in learning technology, vol. 22, 21635, doi: http://dx.doi. org/10.3402/rlt.v22.21635 bosch, t. e. (2009) ‘using online social networking for teaching and learning: facebook use at the university of cape town’, communication: south african journal for communication theory and research, vol. 35, no. 2, pp. 185�200. doi: http://dx.doi.org/10.1080/ 02500160903250648 brownell, j. e. & swaner, l. e. (2010) five high-impact practices: research on learning outcomes, completion, and quality, association of american colleges and universities, washington, dc. davies, c., et al., (2010) ‘research and development to support the next stage of the harnessing technology strategy’, the learner and their context. the technology-based experiences of learners as they approach and enter the world of work. a report for becta, university of oxford, oxford. de freitas, s. i., morgan, j. & gibson, d. (2015) ‘will moocs transform learning and teaching in higher education? engagement and course retention in online learning provision’, british journal of educational technology, vol. 46, no. 3, pp. 455�471. doi: http://dx.doi.org/10.1111/bjet.12268 dewitt, d., et al., (2014) ‘interactions in online forums: a case study among first year undergraduate students’, frontiers in education (fe), vol. 2, no. 1, pp. 6�13. dodge, l. & kendall, m. e. (2004) ‘learning communities’, college teaching, vol. 52, no. 4, pp. 150�155. doi: http://dx.doi.org/10.3200/ctch.52.4.150-155 ellison, n. b., steinfield, c. & lampe, c. (2007) ‘the benefits of facebook ‘‘friends’’: social capital and college students’ use of online social network sites’, journal of computermediated communication, vol. 12, no. 4, pp. 1143�1168. doi: http://dx.doi.org/10.1111/j. 1083-6101.2007.00367.x gray, k., annabell, l. & kennedy, g. (2010) ‘medical students’ use of facebook to support learning: insights from four case studies’, medical teacher, vol. 32, no. 12, pp. 971�976. doi: http://dx.doi.org/10.3109/0142159x.2010.497826 henderson, m. (2015) ‘the (mis) use of community of practice: delusion, confusion, and instrumentalism in educational technology research’, in critical perspectives on technology and education, eds. s. bulfin, n. f. johnson & c. bigum, palgrave macmillan, new york, ny, pp. 127�140. henderson, m., selwyn, n. & aston, r. (2015) ‘what works and why? student perceptions of ‘‘useful’’ digital technology in university teaching and learning’, studies in higher education, pp. 1�13. doi: http://dx.doi.org/10.1080/03075079.2015.1007946 g. salmon et al. 12 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 http://dx.doi.org/10.7763/ijeeee.2014.v4.329 http://dx.doi.org/10.1080/14703297.2013.863735 http://dx.doi.org/10.3402/rlt.v22.21635 http://dx.doi.org/10.3402/rlt.v22.21635 http://dx.doi.org/10.1080/02500160903250648 http://dx.doi.org/10.1080/02500160903250648 http://dx.doi.org/10.1111/bjet.12268 http://dx.doi.org/10.3200/ctch.52.4.150-155 http://dx.doi.org/10.1111/j.1083-6101.2007.00367.x http://dx.doi.org/10.1111/j.1083-6101.2007.00367.x http://dx.doi.org/10.3109/0142159x.2010.497826 http://dx.doi.org/10.1080/03075079.2015.1007946 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 hrastinski, s. & aghaee, n. m. (2012) ‘how are campus students using social media to support their studies? an explorative interview study’, education and information technologies, vol. 17, no. 4, pp. 451�464. jones, n., et al., (2010) ‘get out of myspace!’, computers in education, vol. 54, no. 3, pp. 776�782. junco, r., heiberger, g. & loken, e. (2010) ‘the effect of twitter on college student engagement and grades’, journal of computer assisted learning, vol. 27, no. 2, pp. 119�132. doi: http://dx.doi.org/10.1111/j.1365-2729.2010.00387.x kaeomanee, y., rias, b. m. & perveen, r. (2012) ‘social software enhanced e-learning system to support knowledge sharing among students: malaysian higher education institutions perspectives’, paper presented at the 2012 international conference of computer & information science (iccis), kuala lumpur. kassens-noor, e. (2012) ‘twitter as a teaching practice to enhance active and informal learning in higher education: the case of sustainable tweets’, active learning in higher education, vol. 13, no. 1, pp. 9�21. doi: http://dx.doi.org/10.1177/1469787411429190 kim, h. n. (2008) ‘the phenomenon of blogs and theoretical model of blog use in educational contexts’, computers & education, vol. 51, no. 3, pp. 1342�1352. doi: http://dx.doi.org/10. 1016/j.compedu.2007.12.005 knox, j. (2014) ‘digital culture clash: ‘‘massive’’ education in the e-learning and digital cultures mooc’, distance education, vol. 35, no. 2, pp. 164�177. doi: http://dx.doi.org/10. 1080/01587919.2014.917704 kop, r., fournier, h. & mak, j. s. f. (2011) ‘a pedagogy of abundance or a pedagogy to support human beings? participant support on massive open online courses’, international review of research in open and distance learning, vol. 12, pp. 74�93. lai, k.-w. (2015) ‘knowledge construction in online learning communities: a case study of a doctoral course’, studies in higher education, vol. 40, no. 4, pp. 561�579. doi: http://dx. doi.org/10.1080/03075079.2013.831402 laird, t. (2014) using social media in education for classroom teaching, student learning, and educator professional development, queen’s university, kingston, ontario, canada. land, r. & bayne, s. (2008) ‘social technologies in higher education: authorship, subjectivity and temporality’, paper presented at the sixth international conference on networked learning, halkidiki, greece. levinson, m. (2013) where moocs miss the mark: the student-teacher relationship [web log post], available at: http://www.edutopia.org/blog/where-moocs-miss-the-markmatt-levinson llorens, f. & capdeferro, n. (2011) ‘facebook’s potential for collaborative e-learning’, revista de universidad y sociedad del conocimiento (rusc), vol. 8, no. 2, pp. 197�210. doi: http:// dx.doi.org/10.7238/rusc.v8i2.963 madge, c., et al., (2009) ‘facebook, social integration and informal learning at university: ‘‘it is more for socialising and talking to friends about work than for actually doing work’’’, learning, media and technology, vol. 34, no. 2, pp. 141�155. doi: http://dx.doi.org/10.1080/ 17439880902923606 maloney, e. j. (2007) ‘what web 2.0 can teach us about learning’, the chronicle of higher education, vol. 53, no. 18, p. b26. oblinger, d. & oblinger, j. (2005) ‘is it age or it: first steps toward understanding the net generation’, educating the net generation, vol. 2, no. 1�2, p. 20. oliver, r., et al., (2007) ‘representing authentic learning designs supporting the development of online communities of learners’, journal of learning design, vol. 2, no. 2, pp. 1�21. özmen, b. & atici, b. (2014) ‘the effects of social networking sites in distance learning on learners’ academic achievements’, european journal of open, distance and e-learning, vol. 17, no. 2, pp. 61�75. doi: http://dx.doi.org/10.2478/eurodl-2014-0019 salmon, g. (2013) e-tivities: the key to active online learning, 2nd edn, routledge, new york. salmon, g. (2015) carpe diem learning design: preparation & workshop, available at: http:// www.gillysalmon.com/carpe-diem.html salmon, g., et al., (2015) ‘experiential online development for educators: the example of the carpe diem mooc’, british journal of educational technology, vol. 46, no. 3, pp. 542�556. doi: http://dx.doi.org/10.1111/bjet.12256 salmon, g. & wright, p. (2014) ‘transforming future teaching through ‘‘carpe diem’’ learning design’, education sciences, vol. 4, no. 1, pp. 52�63. research in learning technology citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 13 (page number not for citation purpose) http://dx.doi.org/10.1111/j.1365-2729.2010.00387.x http://dx.doi.org/10.1177/1469787411429190 http://dx.doi.org/10.1016/j.compedu.2007.12.005 http://dx.doi.org/10.1016/j.compedu.2007.12.005 http://dx.doi.org/10.1080/01587919.2014.917704 http://dx.doi.org/10.1080/01587919.2014.917704 http://dx.doi.org/10.1080/03075079.2013.831402 http://dx.doi.org/10.1080/03075079.2013.831402 http://www.edutopia.org/blog/where-moocs-miss-the-mark-matt-levinson http://www.edutopia.org/blog/where-moocs-miss-the-mark-matt-levinson http://dx.doi.org/10.7238/rusc.v8i2.963 http://dx.doi.org/10.7238/rusc.v8i2.963 http://dx.doi.org/10.1080/17439880902923606 http://dx.doi.org/10.1080/17439880902923606 http://dx.doi.org/10.2478/eurodl-2014-0019 http://www.gillysalmon.com/carpe-diem.html http://www.gillysalmon.com/carpe-diem.html http://dx.doi.org/10.1111/bjet.12256 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 selwyn, n. (2009) ‘faceworking: exploring students’ education-related use of facebook’, learning, media and technology, vol. 34, no. 2, pp. 157�174. doi: http://dx.doi.org/10.1080/ 17439880902923622 strauss, a. & corbin, j. (1998) basics of qualitative research: techniques and procedures for developing grounded theory, 2nd edn, sage publications, inc., thousand oaks. sullivan, p. (2002) ‘it’s easier to be yourself when you are invisible: female college students discuss their online classroom experiences’, innovative higher education, vol. 27, no. 2, pp. 129�144. doi: http://dx.doi.org/10.1023/a:1021109410893 tess, p. a. (2013) ‘the role of social media in higher education classes (real and virtual) � a literature review’, computers in human behavior, vol. 29, no. 5, pp. a60�a68. doi: http:// dx.doi.org/10.1016/j.chb.2012.12.032 thompson, l. & ku, h. y. (2005) ‘chinese graduate students’ experiences and attitudes toward online learning’, educational media international, vol. 42, no. 1, pp. 33�47. doi: http://dx.doi.org/10.1080/09523980500116878 veletsianos, g. & navarrete, c. (2012) ‘online social networks as formal learning environments: learner experiences and activities’, the international review of research in open and distance learning, vol. 13, no. 1, pp. 144�166. wodzicki, k., schwämmlein, e. & moskaliuk, j. (2012) ‘‘‘actually, i wanted to learn’’: studyrelated knowledge exchange on social networking sites’, the internet and higher education, vol. 15, no. 1, pp. 9�14. doi: http://dx.doi.org/10.1016/j.iheduc.2011.05.008 yang, d., et al., (2013) ‘turn on, tune in, drop out: anticipating student dropouts in massive open online courses’, paper presented at the 2013 nips data-driven education workshop, lake tahoe, nv. yu, a. y., et al., (2010) ‘can learning be virtually boosted? an investigation of online social networking impacts’, computers & education, vol. 55, no. 4, pp. 1494�1503. doi: http://dx. doi.org/10.1016/j.compedu.2010.06.015 yuan, j. & kim, c. (2014) ‘guidelines for facilitating the development of learning communities in online courses’, journal of computer assisted learning, vol. 30, no. 3, pp. 220�232. doi: http://dx.doi.org/10.1111/jcal.12042 ziegler, s. g. (2007) ‘the (mis)education of generation m’, learning, media and technology, vol. 32, no. 1, pp. 69�81. doi: http://dx.doi.org/10.1080/17439880601141302 g. salmon et al. 14 (page number not for citation purpose) citation: research in learning technology 2015, 23: 28507 http://dx.doi.org/10.3402/rlt.v23.28507 http://dx.doi.org/10.1080/17439880902923622 http://dx.doi.org/10.1080/17439880902923622 http://dx.doi.org/10.1023/a:1021109410893 http://dx.doi.org/10.1016/j.chb.2012.12.032 http://dx.doi.org/10.1016/j.chb.2012.12.032 http://dx.doi.org/10.1080/09523980500116878 http://dx.doi.org/10.1016/j.iheduc.2011.05.008 http://dx.doi.org/10.1016/j.compedu.2010.06.015 http://dx.doi.org/10.1016/j.compedu.2010.06.015 http://dx.doi.org/10.1111/jcal.12042 http://dx.doi.org/10.1080/17439880601141302 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/28507 http://dx.doi.org/10.3402/rlt.v23.28507 doi:10.3402/rlt.v24.29920 original research article university students and faculty have positive perceptions of open/ alternative resources and their utilization in a textbook replacement initiative nicole delimont a , elizabeth c. turtle b , andrew bennett c , koushik adhikari d and brian l. lindshield a * a department of food, nutrition, dietetics and health, college of human ecology, kansas state university, manhattan, ks, usa; b kansas state university libraries, kansas state university, manhattan, ks, usa; c department of mathematics, college of arts and sciences, kansas state university, manhattan, ks, usa; d department of food science and technology, college of agricultural & environmental sciences, university of georgia, griffin, ga, usa (received 30 september 2015; final version received 6 may 2016) this is contribution no. 16-114-j from the kansas agricultural experiment station. the kansas state university open/alternative textbook initiative provides grants to faculty members to replace textbooks with open/alternative educational resources (oaers) that are available at no cost to students. open educational resources are available for anyone to access, while alternative educational resources are not open. the objective of this study was to determine the perceptions towards oaers and the initiative, of students enrolled in, and faculty members teaching, courses using oaers. a survey was sent out to 2,074 students in 13 courses using the oaers. a total of 524 (25.3%) students completed the survey and a faculty member from each of the 13 courses using oaers was interviewed. students rated the oaers as good quality, preferred using them instead of buying textbooks for their courses, and agreed that they would like oaers used in other courses. faculty felt that student learning was somewhat better and it was somewhat easier to teach using oaers than when they used the traditional textbooks. nearly all faculty members preferred teaching with oaers and planned to continue to do so after the funding period. these results, combined with the tremendous savings to students, support the continued funding of the initiative and similar approaches at other institutions. keywords: e-textbook; college; open educational resources; kansas state; open textbook to access the supplementary material to this article, please see supplementary files under ‘article tools’. introduction textbooks are a big expense for college students, it is estimated that the average student spends $1,200 on books and supplies each year (the college board 2015). to combat this, there are multiple initiatives aimed at replacing textbooks with *corresponding author. email: blindsh@k-state.edu research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 n. delimont et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29920/0 http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29920/0 http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29920/0 http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29920/0 http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 open/alternative educational resources (oaers) at the institutional (bell 2012; billings et al. 2012; oberlander 2015; sutton and chadwell 2014) and system levels (caswell 2012). there are also a number of more focused initiatives/projects that have replaced textbooks in a course or similar courses (bliss et al. 2013a, 2013b; feldstein et al. 2012; hilton et al. 2013; morris-babb and henderson 2011; petrides 2011; watson, clouser, and domizi 2014). open educational resources are ‘educational materials that are either (1) licensed under an open copyright license (e.g., creative commons), or (2) in the public domain (wiley and green 2012)’. open education resources (oers) can be accessed for free and revised, reused, remixed and redistributed by others (wiley and green 2012). while oers have tremendous potential to replace textbooks, there are not always oers suitable for replacing textbooks, and faculty may not be able to find relevant open content to utilize. nonopen resources that are used to replace textbooks are referred to as alternative educational resources. alternative educational resources may be licensed or copyrighted, or have not been made openly available. the term resource is used because the resources may not be a text, and could have different formats or mediums than the textbook they replaced. to provide support for replacing textbooks, there are a number of initiatives that provide grants to faculty members to replace textbooks with oaers that students can use for free. the alternative textbook project at temple university provides grants up to $1,000 (bell 2012), and the open education initiative at the university of massachusetts at amherst provides grants of $1,000�2,500 for faculty members to replace textbooks with oaers (billings et al. 2012). while these initiatives have been successful in attracting faculty to adopt, adapt, or create oaers, they have been less successful in reaching high-enrollment, lower level courses that would benefit the most students. faculty members at the university of massachusetts indicated that this may be because faculty in high-enrollment, lower level courses want larger grants to make a change (billings et al. 2012). at kansas state university, 2 faculty members, who had created and used oers and advocated for their use on campus, collaborated with a libraries’ faculty member to pursue funding to start an initiative similar to those at temple university and the university of massachusetts during the 2013�2014 academic year. kansas state libraries committed $10,000, and a proposal funded by the student government association awarded $50,000 to be used to provide grants of up to $5,000 for faculty members to replace their textbook with an oaer that was available at no cost to students. requests for applications to the initiative were sent out in the 2013�2014 fall and spring semesters through a university newsletter and email announcements. fourteen applications were received, 12 were funded. awardees received half their award up front, the other half was awarded when they began teaching with their oaer instead of a textbook. awardees met once with an initiative faculty member to discuss their vision for the resource, and to establish a timeline for completing it. in the latter part of the spring 2014 semester, there was a meeting of all the awardees to discuss their experience with the initiative up to that point in time. for those who had not completed their resources over the summer, an initiative faculty member followed up with awardees, in person and through email. while some faculty members taught with oaers during the 2013�2014 academic year, all awardees taught with their oaers during the fall 2014 semester. there were a variety of resources used, including adapting or adopting an open textbook, compiling resources on the university’s learning management system, writing an n. delimont et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 online text for the course, and producing video-centered resources. the design of two awardees’ collaboratively developed e-texts has been described recently (rhodes and rozell 2015). an estimated 4,231 students enrolled in courses using oaers during the 2014�2015 academic year, saving these students approximately $415,000. the savings were calculated using the actual cost for texts that were priced below $100, and an average cost of $100 for textbooks when the new cost exceeded $100. this corrected calculation was used to give a more realistic estimate of savings that accounted for renting, buying used textbooks, and purchasing access codes for e-textbooks instead of purchasing a new textbook. community college students have found oaers to be high quality (bliss et al. 2013a, 2013b; caswell 2012), and something that they would like to continue using (petrides 2011). despite a number of similar initiatives at 4-year institutions, there are limited data regarding students’ perceptions on taking a course, and faculty members’ perception on teaching, with oaers instead of textbooks. the lack of data from current initiatives may be limiting their impact and preventing institutions without similar initiatives from undertaking them. methods student survey administration during the latter part of the fall 2014 semester, email addresses of students enrolled in courses using oaers were collected, and a course specific link to an anonymous survey in k-state survey (qualtrics, provo, ut) was emailed to students enrolled in each course. the courses, enrollments, response rates, and a brief resource description are summarized in table 1. the email content was the same for each course except that the instructors helped develop a description of the oaers used in course for the email. the intent of the description was to assist students in understanding what they were asked to complete a survey about. the survey was available for 2.5 weeks, with weekly email reminders sent to students. instructors were encouraged to make their students aware of the survey, and were sent survey completion rates for their course, and other initiative courses when reminder emails were sent out. this was done to inform them of their students’ completion rates compared with students in other courses. survey branching logic was used in the survey; survey questions can be found in the supplementary material. all students were asked about how frequently they used the oaers during the semester using a 7-point likert scale. if they answered never, they were directed to a question asking why they did not use the oaer. students that indicated that they used oaers were directed to 5 likert scale questions asking about their perceptions and use of the oaers. all students were then directed to 3 likert scale questions asking whether they would like to enroll in courses that use oaers, and whether they supported continued funding for the open/alternative textbook initiative. finally, there were two open-ended questions asking why they supported, or did not support, the open/alternative textbook initiative and other similar textbook replacement initiatives/programs. the second question asked why they supported, or did not support, use of donor or university monies to fund the research in learning technology citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29920/0 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 open/alternative textbook initiative. answers to non-open-ended questions were required to submit the survey. for open-ended questions, responses were categorized based on answer themes, some answers were categorized multiple times because they had multiple relevant themes. faculty interview administration open/alternative textbook initiative faculty awardees were emailed to explain the purpose of the interview, and requested to indicate their availability to be interviewed. reminder emails were sent to awardees until times were scheduled with at least one faculty awardee from all 13 courses. interviews were conducted by a graduate student not associated with the initiative (n.d.), in person or via video conferencing, and were recorded; with the exception of one interview, which was transcribed. during interviews, faculty were encouraged to add comments or context where they felt appropriate. interviews were conducted during a 2-month period, during summer 2015. results from interviews were coded and collated to protect the anonymity of awardees before sharing with coauthors associated with the initiative. for open-ended questions, responses were categorized based on answer themes, some answers were categorized multiple times because they had multiple relevant themes. table 1. courses, enrollments, response rates, and resource description. course enrollment responses (%) resource biol 198 � principles of biology 744 153 (20.6) open, adapted openstax resource (cc by 4.0) biol 340 � human body 139 49 (35.3) alternative, produced in ibooks chm 371 � chemical analysis 41 10 (24.4) open, already available edcep 851 � multicultural aspects of academic advising (global campus) 60 37 (61.7) alternative, produced in microsoft word edel 320 � core teaching skills and lab 89 26 (29.2) alternative, produced in ibooks edsec 376 � core teaching skills and field experience/lab 44 20 (45.5) alternative, produced in ibooks edsp 324 � exceptional child in the regular classroom 41 17 (41.5) alternative, modules of resources in learning management system imse 201 � introduction to industrial engineering 90 29 (32.2) alternative, produced in microsoft word math 100 � college algebra studio 25 8 (32.0) open, already available book used, problem sets produced math 100 � college algebra (global campus) 381 67 (17.6) alternative, videos/website math 150 � plane trigonometry 183 38 (20.8) alternative, produced in latex mc 180 � fundamentals of public relations 191 46 (24.1) alternative, videos/content psych 470 � psychobiology 46 24 (52.2) alternative, produced in microsoft word n. delimont et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 faculty interview questions faculty interview questions can be found in supplementary material. faculty were initially asked about the student experience in their courses using their oaers, followed by questions focusing on their experience using the oaers. survey questions consisted of 7-point likert scale questions, and 11 open-ended questions, 3 of which had prompted examples. data analysis descriptive statistics (means9standard error of the mean) and percentages were calculated from student and faculty likert question responses. individual response scores were compared to mean course response scores, to determine whether the larger courses were skewing outcomes. the means calculated from mean course responses and the individual responses were nearly identical, so individual response means are presented. due to the breadth of oers and a wide variety of experiences from faculty utilizing them, qualitative data have been included from interviews, and individual level data is presented. the final 3 likert scale survey questions from student surveys were analyzed by student’s t-tests using sas 9.3 (sas institute inc., cary, nc) with p b0.05 considered significant. results student survey results the survey was sent to 2,074 students in 13 courses using the oaers. in total, 524 (25.3%) students completed the survey. frequency of use students indicated that they used the oaers on a weekly basis (table 2). interestingly, this score (4.7) aligns well with a similar question on a survey of table 2. frequency of use and student support for open/alternative educational resources and the open/alternative textbook initiative (mean9sem). question all (n �524) used (n � 471) did not use (n �53) how frequently have you used the open/alternative resource this semester (1 � never, 7 � more than 3 times a week). 4.890.1 5.290.1 1 i would like open/alternative resources used in other courses that i take (1 � strongly disagree, 7 � strongly agree). 5.690.1 5.790.1 4.690.3* continued funds should be provided for the kansas state university open/alternative textbook initiative (1 � strongly disagree, 7 � strongly agree). 6.090.1 6.190.1 4.690.2* i support donor or university funds being used to support the kansas state university open/alternative textbook initiative (1 � strongly disagree, 7 � strongly agree). 5.890.1 5.990.1 4.890.2* *p b 0.001 vs. used. research in learning technology citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/rt/suppfiles/29920/0 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 campus students done previously using an oer in a course at the same institution (4.8) (lindshield and adhikari 2013). fifty-three students (10.1%) indicated that they never used oaers. it is worth noting that 30 (56.6%) of these students were in a large mathematics course, in which 44% of survey respondents indicated that they never used the oaer. that course created online problem sets and adopted an open textbook to support what was taught in class, and thus, some students may not have understood that the problem sets were part of the oaer. among the students that answered never, 38 (71.7%) answered the open-ended question about why they did not use oaers. sixteen (42.1%) students indicated that they did not know about the oaer or know where to find it; 16 (42.1%) students indicated that they did not need, did not think to use, or that the subject matter was taught differently than the oaer; and 3 (7.9%) students indicated that they thought or heard that the oaer was poor quality or was hard to understand. two (5.2%) students mentioned using the oaer in their explanations, so they must not have understood the first question. one (2.6%) student indicated that the oaer was inconvenient to access. preference of oaers students indicated that they were somewhat satisfied taking courses using oaers and used them somewhat more to more than a normal textbook (table 3). students rated the oaers as good quality and indicated that they were somewhat easy to use. students agreed that they preferred using oaers instead of buying textbooks for their courses. it is interesting to note that these outcomes are similar to findings in a survey of students using an open educational resource in a course at this institution (lindshield and adhikari 2013). open/alternative resources and open/alternative textbook initiative support overall, students agreed that they would like oaers used in other courses (table 1). they also agreed that funds should be provided for the kansas state university open/alternative textbook initiative and supported donor and university funds used to do so. interestingly, students who used oaers had significantly higher scores on table 3. student perceptions of the open/alternative educational resource using a 7-point likert scale (mean9sem). question used (n �471) rate your level of satisfaction of taking this course using an open/alternative resource (1 �completely dissatisfied, 7 �completely satisfied). 4.890.1 compared to your experience with normal textbooks, i use the open/alternative resource: (1 �much less, 7 �much more) 5.590.1 rate the level of quality of the open/alternative resource (1 �very bad, 7 �very good). 5.790.1 rate the level of difficulty of using the open/alternative resource (1 �very difficult, 7 �very easy). 5.390.1 i prefer using the open/alternative educational resource instead of buying a textbook for this course (1 �strongly disagree, 7 �strongly agree). 5.790.1 n. delimont et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 these questions than students who did not use them (p b0.001). while students that did not use the oaers were still supportive, they were significantly less supportive than those who did use oaers. reason for supporting or not supporting the open/alternative textbook initiative given the differences in student responses to the previous three questions, responses to the open-ended questions were analyzed separately for students who used and did not use the oaers. among students that used the oaers, 313 (66.5%) answered the first question asking why they supported or did not support the initiative. of these, 194 (62%) students indicated they were supportive because of financial benefit, 72 (23%) students were supportive because oaers were easy to access, and 59 (18.8%) students valued that oaers were customized to the course. it is worth noting that 2 of the 13 courses (15.4%) did not use customized resources. if all had used customized resources then the number of students that indicated this as a reason for being supportive likely would have been higher. twenty-six (8.3%) students liked not having to carry around the textbook or having a heavy book bag, and 10 (3.2%) students were supportive because they felt that the oaers were more interactive than traditional textbooks. it was interesting that students valued this factor because only a limited number of the resources are active or interactive. ten (3.2%) students were supportive because it was easy to find content in oaers, while 9 (2.9%) students liked that the resources were online. eight (2.6%) students valued that oaers conserved resources, or were better for the environment. among students that used the oaers but were not supportive of the initiative, 19 (6.1%) students indicated wanting a physical book/hard copy. nine (2.9%) students liked features of the oaers, but did not like screen reading and preferred having a physical book or copy. seven (2.2%) students were not supportive because they did not like the oaers or had problems using them, and 5 (1.6%) students did not think the oaers were as good of quality as a traditional textbook. four (1.3%) students were cautious about supporting the initiative because they wanted university funds to be used for other projects, were concerned about whether it would be successful in other courses, or wanted to make sure that the resource would be of equivalent quality to the oaer in their current course. among students that did not use oaers, 35 (66%) provided an answer to this open-ended question. among these, 11 (31.4%) were supportive of the initiative because of the financial benefit, and 2 (5.7%) students were supportive because they thought it was a good idea. one (2.9%) student was supportive, but wanted the oaers customized to the course; another student (2.9%) was supportive because he/ she liked having an online book. one student (2.9%) liked not having a heavy book bag, another student (2.9%) was supportive because of the environmental benefit. four (11.4%) students felt neutrally because they either did not need the oaers, or were torn between financial benefit versus wanting a physical book. two (5.7%) students were not supportive because they wanted a physical book, another 2 (5.7%) students were not supportive because the oaers were poor quality, and 1 (2.9%) student was not supportive because he/she thought the initiative took away learning opportunities. the last open-ended question asked why students did or did not support donor or university funds being used for the open/alternative textbook initiative. because all reasons that students were supportive of the initiative were gathered in the previous research in learning technology citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 question, the focus was on categorizing responses related to funds used to support the initiative. fifty percent (263) of the students provided a response to the open-ended question that did more than direct the reader to their answer to the previous question. eighty (30.4%) students thought the initiative was a productive use of donor and/or university funds. seventy (26.6%) students did not specifically identify which funds they supported using, but indicated they were supportive because of the financial benefit to students. nineteen (7.2%) students indicated donor funds, or funds not coming from students/university, should be used to support the initiative. ten (3.8%) students thought that donor and university funds could be used for better purposes than the open/alternative textbook initiative; one (0.4%) student thought that the program would attract students to the university and another student (0.4%) supported university funds being used, but wanted donor funds used for scholarships or other earmarked purposes. faculty perceptions of student use and experience with open/alternative resources faculty indicated that they perceived that students liked oaers better (4/13), that students had course materials in class more often (3/13), that students read the resource more closely, more often, or knew content more broadly (6/13), and that students were able to better access video, web links, and blogs relevant to the course with the oaers compared with traditional textbooks (3/13). four out of 13 indicated that they did not perceive a difference. faculty answered that student performance using the oaers was neutral to somewhat better. of those that did not see improved performance with the oaers, large class size (2/13), or lack of data (1/13) were the most commonly cited reasons. faculty who felt that performance was better cited customized content (2/13), more flexibility in course structure (faculty could change the oaer to reflect lecture material and vice-versa) (4/13), or enhanced grasp of complex concepts due to the ability to include video, websites, or audio (2/13). faculty also felt that student learning was somewhat better using oaers than when they used traditional textbooks. reasons they believed learning was improved were that oaers were more up to date (6/13), or that faculty felt confident referring students to the oaers to learn outside of class time (3/13). one instructor felt that their resource did not enhance student learning because the oaer was too in depth for the course content. faculty believed that the student experience was better than with a traditional textbook, which they attributed oaers’ easy access (6/13) and customization (3/13). most faculty did not perceive a difference in students’ perception of the course, or the instructor using the oaers compared to when it was taught with a traditional textbook (8/13). faculty who did note differences indicated that students connected to the instructor better because of the content (1/13, the faculty member felt that they were humanized due to the content presented); that students connected better to the content (5/13); or that students were able to connect to more complex or broad concepts either within, or outside of course structure, due to the content (2/13). one faculty member noted that students did not change their perception of the course with the oaer, but that it may have been due to the fact that students were not enthusiastic about the course. n. delimont et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 faculty experiences creating/adopting open/alternative resources faculty felt that creating, adopting, or adapting their oaer was somewhat difficult (table 4). some of those who felt that the resource creation or adoption was somewhat to very difficult noted that they needed or acquired special knowledge to develop the resource (4/13). it is worth noting that all of these faculty members created oaers. other issues faculty encountered included questions about copyright or licensing of materials, and difficulty accessing materials to use in the resource (3/13). faculty members (6/13) felt that the time required to develop the resource contributed to the difficulty of the process and indicated that it took somewhat more time than they anticipated to create, adopt, or adapt their oaers. faculty felt it was somewhat easier to teach with oaers compared to a traditional textbook. reasons provided included customization and enhanced applicability to the course (5/13), ease of supplementing lecture content with the resource (2/13), improved course organization (1/13), and decreased financial reasons for not having access to materials allowing for full utilization of the resource, and thus enhanced ease of teaching (1/13). utilization and continuance nearly all (12/13) faculty members indicated that they preferred teaching their course with the oaers instead of a traditional textbook. several (11/13) indicated that customization was a reason for this preference. other reasons included easier format to use (2/13), flexibility in teaching content (5/13), ability to collaborate with other instructors more seamlessly (1/13), ease of teaching (1/13), and congruency in learning materials (faculty and text) (1/13). the faculty member who indicated not preferring the oaers indicated that he/she liked the oaer, but felt that a traditional textbook was just as, or easier to use. the primary reason for preferring a traditional text was that he/she had taught the course for many years, refining it over time, and so it was table 4. faculty interview responses to 7-point likert scale questions (mean9sem). question average score how did student performance in the course using the open/alternative resource compare to a traditional textbook (1 �much worse, 7 �much better)? 4.590.3 how did you perceive student learning in the course taught the open/alternative resource compares to the course taught using a traditional textbook (1 �much worse, 7 �much better)? 4.890.4 how did the student experience in the course using the open/alternative resource compare to a traditional textbook (1 �much worse, 7 �much better)? 5.990.3 how would rate the level of ease/difficulty in creating/adopting/adapting the open/ alternative resource (1 �very difficult, 7 �very easy)? 3.690.5 how would you rate the amount of time it took you to create/adopt/adapt the open/alternative resource compared to what you thought it would take before you did so (1 �much less, 7 �much more)? 4.790.4 how would rate the level of ease/difficulty in teaching with the open/alternative resource compared to teaching with a traditional textbook? (1 �very difficult, 7 �very easy)? 5.390.3 how would you rate your experience in the open/alternative textbook initiative (1 �very bad, 7 very good)? 6.590.2 research in learning technology citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 not difficult to teach with the textbook. this was the only awardee that did not plan to utilize the oaer in the future. similar to preference, nearly all (12/13) faculty members planned to continue using oaers beyond the funding period. several (6/13) indicated that this was due to adaptability of materials that they had created over time, while customization (5/13), and flexibility (4/13) in teaching content were the next most highly cited reasons. two respondents indicated that they felt oaers enhanced the overall class experience. the respondent who did not plan to continue to utilize the resource indicated that it was because they did not plan to teach the course in the future. that faculty member indicated that had they continued to teach the course, they would have likely used the oaer. reasons motivating faculty to use oaers included financial benefit to students (12/13), financial support from the initiative (9/13), consistency with their teaching philosophy (8/13), lack of suitable materials for the course (6/13), customization of resource to teaching (6/13), and 1 respondent noted that this idea directly coincided with professional development goals. challenges, resources, and post-initiative perspective the most frequently cited challenges in creating, adopting, or adapting oaers were time required (6/13), technology issues (6/13), and concerns about copyright/licensing (4/13). one faculty member stated that it was difficult to coordinate the use of the oaer with other faculty teaching their course, and that the work was somewhat tedious. others said that it was difficult to create an innovative resource (1/13), or that it was difficult to get students to use the resource (1/13). two faculty cited no problems or challenges in creating their oaers. when asked what areas of support would have helped faculty when creating their oaers, several faculty (8/13) indicated that more credit for development, or consideration of development of oaers in tenure or professional advancement would have been important to them. faculty (7/13) also felt that more support from the department, college, or university as a whole (most often department heads), and support from peers using oaers would have been valuable (6/13). faculty also noted that more training (5/13), time (3/13), or financial resources (3/13) would have enhanced their project. two faculty members did not feel that they needed additional support from the initiative. faculty indicated that if they could go back through the initiative again, they would have wanted to know about overall availability of platforms to create and adopt oaers and best sources to obtain content from (4/13). additionally, technological or authoring assistance during the process of creating their resources (3/13), and more support with copyright or licensing issues (2/13) would have been helpful. others noted that they would have liked to have a better idea of the amount of time to create the oaer (2/13). one faculty member noted that he/she would have liked to have known about opportunities to collaborate before undertaking the project, and another stated that they would have liked to have known that students in his/her course were likely to print the materials rather than use them electronically. faculty experiences with the initiative itself overwhelmingly, faculty indicated that they had a good to very good experience in the open/alternative textbook initiative (table 4). faculty appreciated that the n. delimont et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 initiative was transparent, and ‘said it was going to do what it did’ (7/13). some faculty (4/13) reported that they enjoyed the process of creating their oaer, a few (2/13) enjoyed receiving the award, or appreciated that the initiative prompted college or departmental change (2/13). one said that students liked the oaer better, and thus had a good experience with the initiative. the faculty member who rated their experience as somewhat good noted that challenges included a somewhat disorganized start to the initiative, and that they did not quite understand what an oaer could really be. when asked about ideas that could encourage more faculty to create, adopt, or adapt oaers, faculty (7/13) answered that oaers would be better adopted if faculty had an idea of what peers had done before. training, financial support, and time beyond the funding period or reduced teaching load were also mentioned (3/13). two faculty proposed directly targeting large lecture classes where the most financial benefit could be gained, and another two noted enhanced use of accolades, including tenure considerations and promotion would likely support further initiative growth. one faculty member stated that collaborating with other faculty could be helpful to reduce the work load to individual awardees. overall, 7/13 faculty independently responded that they endorsed university wide support or expansion of the project. three other faculty members indicated that although their overall experiences were positive, the project was either time consuming (2/13) or that they needed a reduced work load to create their resource (1/13). discussion to the best of our knowledge, this manuscript is the first to assess both student and faculty perceptions at a 4-year institution where an initiative replaces textbooks with oaers. while a higher survey completion rate would have been desirable, the completion rate was towards the top of the range (5.9�33%) reported for online surveys of other initiatives (bliss et al. 2013a; feldstein et al. 2012). overall, the findings indicate that students and faculty had positive experiences in courses that used oaers, which is similar to mathematics students at a community college using an open educational resource (hilton et al. 2013). salient findings that may interest institutions adopting oaers have been summarized in table 5. the frequency that students reported using the oaers was similar to 2�3 times per week that community college students in project kaleidoscope reported table 5. initiative findings and practical implications for institutions adopting oaers. finding practical relevance 1. overall student and faculty preference oaers may be acceptable in a wide variety of educational settings 2. limited student preference for physical or hard copy of material pushback from faculty or student preference for hard copy vs. oaer may have limited implication in practice 3. preference for customizable nature of oaers institutions promoting oaers adoption may benefit from faculty led or course driven material development 4. financial benefit institutions promoting oaers may consider financial incentive proportional to course need 5. preference for credit through tenure/promotion faculty encouraged to develop and maintain oaers may be incentivized by institutional support by way of promotion and tenure credit research in learning technology citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 (bliss et al. 2013a). students’ preference for the oaers over traditional textbook is similar to business students indicating that they preferred the content in oers at a 4-year institution (feldstein et al. 2012). it is worth pointing out that the findings from likert questions are similar to those from a survey of students using an oer in a course at the same institution (lindshield and adhikari 2013). this consistency may mean that similar results may be expected to these questions on future surveys of initiative students. it is also interesting that the percentage of students that were not supportive of the initiative because they preferred a physical or hard copy was similar between students that used or did not use oaers (�6%). given this consistency, this may be an indication that the resistance for this reason is relatively limited. it was also notable that faculty and students both indicated that financial benefit and customization were reasons that they supported oaers. these results may indicate that initiatives should strive to have faculty do more than simply adopting an oaer, they should be encouraged to adapt or customize it for their course. the responses from faculty on what they wish they would have known before developing their oaer are invaluable and can help guide initiatives regarding the necessary training and resources to provide faculty awardees. the recommendations from faculty members suggest that institutional support and professional credit in tenure and promotion would spur more faculty adopting/adapting/creating oaers. the initiative was by and large successful in reaching larger enrollment courses, 1 of its primary goals, but it is not known how much of this was due to the larger monetary awards that were offered (as opposed to the awards offered at the university of massachusetts or temple university initiatives). it is also difficult to determine what the impact would be if all resources were required to be oers. many awardees are using alternative educational resources, some of which will hopefully become oers eventually. although the initiative has been fairly successful, larger monetary awards and more active recruitment, rather than simply requesting applications, may be needed to entice some large enrollment courses to replace their textbooks with oaers as suggested by some faculty interviewees. in addition, if the initiative is scaled up, more funding will be required for administrative functions, mentoring, resource revisions and updates, and training to support its continued success. it is important to note some of the limitations of this research. this research was conducted at a single institution using a survey that has not been validated. while the survey response rate was higher than some other surveys of students in similar initiatives, we cannot rule out the potential that the students who responded had a better experience and/or were more supportive of the initiative than those who did not. it is also important to note that there was a time difference between when the surveys were administered to students and when faculty interviews were conducted. in conclusion, these results combined with the tremendous savings to students, support the funding of this initiative and similar initiatives at other institutions. further research is needed to better understand how to maximize the positive impact to students and faculty in similar initiatives. references bell, s. j. (2012) ‘coming in the back door: leveraging open textbooks to promote scholarly communications on campus’, journal of librarianship and scholarly communication, vol. 1, no. 1, p. ep1040. doi: http://dx.doi.org/10.7710/2162-3309.1040 n. delimont et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 http://dx.doi.org/10.7710/2162-3309.1040 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 billings, m., et al., (2012) ‘open educational resources as learning materials: prospects and strategies for university libraries’, research library issues, vol. 280, pp. 2�10. bliss, t. j., et al., (2013a) ‘an oer coup: college teacher and student perceptions of open educational resources’, journal of interactive media in education, vol. 1, pp art. 4. doi: http://dx.doi.org/10.5334/2013-04 bliss, t. j., et al., (2013b). ‘the cost and quality of online open textbooks: perceptions of community college faculty and students’, first monday, vol. 18, no. 1�7. doi: http://dx.doi. org/10.5210/fm.v18i1.3972 caswell, t. (2012) ‘the open course library of the washington state colleges’, in game changers: education and information technologies, ed d. g. oblinger, educause, louisville, co, pp. 259�262. feldstein, a., et al., (2012) ‘open textbooks and increased student access and outcomes’, european journal of open, distance and e-learning. vol. ii. http://www.eurodl.org/? p=archives&year=2012&halfyear=2&article=533 hilton, j. l., iii, et al., (2013) ‘the adoption of open educational resources by one community college math department’, the international review of research in open and distributed learning, vol. 14, no. 4. http://www.irrodl.org/index.php/irrodl/article/view/1523/2652 lindshield, b.l. & adhikari k. (2013) ‘online and campus college students like using an open educational resource instead of a traditional textbook’, journal of online learning and teaching, vol. 9, no. 1: pp. 26�38. morris-babb, m. & henderson, s. (2011) ‘an experiment in open-access textbook publishing: changing the world one textbook at a time’, journal of scholarly publishing, vol. 43, no. 2, pp. 148�155. doi: http://dx.doi.org/10.3138/jsp.43.2.148 oberlander, c. (2015) ‘textbooks and educational resources in library-based publishing’, in getting the word out, eds m. bonn & m. furlough, the association of college & research libraries, chicago, il, pp. 177�192. petrides, l. (2011) ‘open textbook adoption and use: implications for teachers and learners’, open learning, vol. 26, no. 1, pp. 39�49. doi: http://dx.doi.org/10.1080/02680513.2011. 538563 rhodes, a. e. & rozell, t. g. (2015) ‘a constructivist approach to e-text design for use in undergraduate physiology courses’, advances in physiology education, vol. 39, no. 3, pp. 172�180. doi: http://dx.doi.org/10.1152/advan.00011.2015 sutton, s. c. & chadwell, f. a. (2014) ‘open textbooks at oregon state university: a case study of new opportunities for academic libraries and university presses’, journal of librarianship and scholarly communication, vol. 2, no. 4, pp. 34�48. doi: http://dx.doi.org/ 10.7710/2162-3309.1174 the college board. (2015) ‘average estimated undergraduate budgets, 2014�2015. trends in higher education’, [online] available at: http://trends.collegeboard.org/college-pricing/ figures-tables/average-estimated-undergraduate-budgets-2014-15#key points watson, c. e., clouser, s. & domizi, d. (2014) improving the quality of instruction and increasing the affordability of higher education through the adoption of open education resources (oers), [online] available at: http://www.cideronline.org/conference/presentation1.cfm? pid�1652 wiley, d. & green, c. (2012) ‘why openness in education?’, in game changers: education and information technologies, ed d. g. oblinger, educause, louisville, co, pp. 81�89. research in learning technology citation: research in learning technology 2016, 24: 29920 http://dx.doi.org/10.3402/rlt.v24.29920 13 (page number not for citation purpose) http://dx.doi.org/10.5334/2013-04 http://dx.doi.org/10.5210/fm.v18i1.3972 http://dx.doi.org/10.5210/fm.v18i1.3972 http://www.eurodl.org/?p=archives&year=2012&halfyear=2&article=533 http://www.eurodl.org/?p=archives&year=2012&halfyear=2&article=533 http://www.irrodl.org/index.php/irrodl/article/view/1523/2652 http://dx.doi.org/10.3138/jsp.43.2.148 http://dx.doi.org/10.1080/02680513.2011.538563 http://dx.doi.org/10.1080/02680513.2011.538563 http://dx.doi.org/10.1152/advan.00011.2015 http://dx.doi.org/10.7710/2162-3309.1174 http://dx.doi.org/10.7710/2162-3309.1174 http://trends.collegeboard.org/college-pricing/figures-tables/average-estimated-undergraduate-budgets-2014-15#key points http://trends.collegeboard.org/college-pricing/figures-tables/average-estimated-undergraduate-budgets-2014-15#key points http://www.cideronline.org/conference/presentation1.cfm?pid=1652 http://www.cideronline.org/conference/presentation1.cfm?pid=1652 http://www.cideronline.org/conference/presentation1.cfm?pid=1652 http://www.cideronline.org/conference/presentation1.cfm?pid=1652 http://www.cideronline.org/conference/presentation1.cfm?pid=1652 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/29920 http://dx.doi.org/10.3402/rlt.v24.29920 (page number not for citation purpose) 1 *corresponding author. email: l.drumm@napier.ac.uk research in learning technology 2019. © 2019 l. drumm. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 research in learning technology vol. 27, 2019 original research article folk pedagogies and pseudo-theories: how lecturers rationalise their digital teaching louise drumm* department of learning and teaching enhancement, edinburgh napier university, edinburgh, scotland, uk (received 6 june 2018; final version received 10 january 2019) the gap in knowledge about how learning theories relate to everyday digital teaching practices in universities inhibits scholarly and practical developments in this area. this article reports on part of a qualitative research project which identified patterns across teaching modes, descriptions and accompanying rationales. it found that learning theories played a minor role in educators’ rationales, even though many of their teaching practices could be described as pedagogically ‘sound’. although social constructivist approaches were strongly represented in the data, the most widespread rationales for technology uses were folk pedagogies and pseudo-educational theories. this contradicts much of what scholarship and ‘edtech’ culture espouses as pedagogically led technology use. such educational technology orthodoxies hinder the progress of theory use in this area and fail to address the realities of how lecturers use digital technologies. while it may come as no surprise that educators did not articulate their practices referencing learning theories, the dominance of pseudo-theories in this research represents a threat to the criticality of scholarship and practice in this area. this article recommends that critical and scholarly approaches to digital teaching are encouraged, and that folk and pseudo-theories are acknowledged and leveraged in the support and development of digital teaching. keywords: pedagogy; learning theories; digital pedagogy; rhizoanalysis; criticality introduction ‘pedagogy before technology’ is something of a mantra in the support and development of teaching with digital technologies. however, the extent to which educational theory relates to the use of digital technologies for teaching, specifically the pedagogical application of learning theories, is rarely examined in light of existing teaching practices (selwyn 2010). furthermore, learning theories (concerned with how learners learn) and pedagogy (the intersection between teaching decisions and learning activities) are often conflated and unproblematised (jaffer 2010).what constitutes ‘theory’ is also troublesome, particularly as demand for pragmatic approaches often results in frameworks and models being assumed to be theoretically grounded ( bulfin, henderson, and johnson 2013). the first decade of 21st century has witnessed substantial research conducted in this area(e.g. beetham and sharpe 2007; conole et  al. 2004; kanuka 2008; mayes and de freitas 2007), with debates on whether mailto:l.drumm@napier.ac.uk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2094 l. drumm 2 citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) theory should be appropriated from other domains (howard and maton 2011; johri, 2011; oliver, 2011, 2012; selwyn 2010) sparring with the idea that something fundamentally new is occurring with digital teaching and learning, and new theories are required (andrews 2011; franklin and harmelen 2007; garrison 2011; harasim 2012; haythornthwaite and andrew 2011; masterman and vogel 2007). the purpose of this article is to revisit a selection of commonly referenced theories of learning and use them as a means to explore how, and if, they relate to how lecturers discuss their digital teaching practices. when it comes to scholarship in the area of education and technology, the field has been dominated by specific affordances of technology and proofs of ‘success’ (selwyn 2010, 2013). yet, when taking the long view of the history of research of education and technology, such as that taken by mayes (1995) in a paper now over 20 years old, there appears to be much that has been forgotten, with nuance of thought and debate drowned out by a need for certainties and unquestioned orthodoxies of ‘best practice’. as identified by kanuka (2008), there is a need to examine the choices made by educators about pedagogical and technology in light of their underlying belief systems. this article is an attempt to draw together some established concepts of learning and map them against practice so as to contribute to our ‘understanding the often uneven, contested and contradictory realities of technology use within educational settings’ (selwyn 2010, p. 67). in doing so, it furthers knowledge about theory and practice in this area and also addresses the perceived wedge between theory and practice for educators and those, such as academic developers, who support them. literature selected learning theories and learning myths the following theories of learning (behaviourism, cognitivism and social constructivism) are frequently used to conceptualise, model and even proselytise the use of digital technologies in education. they are a subset of theoretical models of teaching and learning, and have been selected because they are often used as shorthand for theoretically informed pedagogies and are often implicitly implied whenever pedagogy is discussed. there are, of course, many other theoretical approaches to understanding learning which overlap, complement and contradict these theories. myths around learning are also addressed here, as many of these have roots in the more established learning theories. behaviourism as a theory of learning, behaviourism is broadly understood to be where a teacher, or technological proxy, uses stimuli to elicit a response or change in behaviour in the learner, which can be measured externally (anderson and dron 2011). behaviourism’s manipulative undertones, positivist assumptions and non-student-centred objectives have, most likely, led to discomfort in acknowledging that technology, through automation and interactivity, can easily facilitate such behaviourist approaches in teaching. this may account for the scarcity of explicit references to behaviourism in teaching practice despite obvious influences, such as the gamification and ‘badging’ of learning (watters 2016). this was reflected in wu et al.’s (2012) research on studies http://dx.doi.org/10.25304/rlt.v27.2094 research in learning technology citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 3 (page number not for citation purpose) on games-based learning where they found a tendency for researchers to theorise their practice with contemporary ‘fashionable’ theories rather than with behaviourist-based theory. conole et al. (2004) found a similar re-framing of behaviourist teaching as ‘constructivist’. in an alternative assessment, mayes and de freitas (2007) critique crude interpretations of behaviourism and make the case for it as more closely aligned to active ‘learning by doing’ methods and, as such, should not be dismissed. behaviourism has had a profound influence on popular strategies, in particular on instructional design and similar procedural approaches to teaching (kanuka, 2006). cognitivism in cognitivist learning approaches, internal cognitive processes such as motivation, reflection and pre-existing knowledge are taken into account (ally 2008). emerging from psychology, cognitivism frames learning as retention of, and access to, knowledge in working and long-term memory, and the importance of existing cognitive structures to aid these processes (ibid.). this focus on the individual learner has contributed to the promotion of certain orthodoxies with digital technologies that have become widespread, to the point they are seen as self-evident ‘common sense’. for example, cognitivism appears to underpin efforts to personalise learning through the use of digital technologies (wheeler 2015), whereby educators can selectively deliver digital content to individual students. however, the view that learners can be viewed as ‘self-educators’ due to digital technologies has been identified by kirschner and van merriënboer (2013) as an ‘urban myth’ and chimes with warnings about the idealised autonomous learner (biesta 2012). indeed, some have labelled the ‘personalised learning’ touted by massive open online courses (moocs) a myth (bates 2012), arguing that understanding of the learner as an individual is required for true personalisation rather than generic ‘types’ of learners. furthermore, this facilitation of personalised needs of the learner via technology’s different modes has found pedagogical justification in ‘learning styles’ (ally 2008), a controversial but persistent conceptualisation of learning addressed below. social constructivism constructivism is based on the principle that knowledge is not passively received by the learners but something they actively construct. the role of the educator in constructivist learning shifts from that of an architect to a facilitator who supports the learners in building their own knowledge structures. this is often aligned with a shift from ‘teacher-focused’ to ‘student-centred’ teaching (kemp 2013), which is itself problematised according to biesta (2005, 2012) who laments the disappearance of the central relationship at the heart of teaching and learning (2012). under social constructivism, digital content creation and communication tools are repurposed as sites for student-led knowledge construction(conole and alevizou 2010; kinchin 2012). consequently, social constructivism and collaborative digital technologies have become enmeshed in educational technology discourse, each enabling and reifying the other. social constructivist learning theories have been elevated to ‘best practice’ when using digital technologies for teaching(selwyn 2009). that is not to say they are not appropriate for some teaching practices, but the recent literature tends to over-report social constructivism and under-report on other theories. http://dx.doi.org/10.25304/rlt.v27.2094 l. drumm 4 citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) folk pedagogies and pseudo-theories of learning ‘folk pedagogies’, coined by olson and bruner (1996), refer to theory of mind held by teachers about learning. here, i extend folk pedagogies to include those that are gained through personal experience as both a learner and teacher, and through cultural norms about teaching (e.g. from a disciplinary teaching culture). such naïve ideas of teaching have been pathologised in favour of change, specifically conceptual change on the part of the educator to ‘better’ beliefs(ho, watkins and kelly 2001; torff 1999). however, as apparent by olson and bruner’s (1996) examination of theories of mind, all teachers hold personalised mental models of teaching which could be readily described as folk pedagogies, yet can all teachers be ‘wrong’? closely related to folk pedagogies are popularly held concepts misconstrued as theories of learning, ‘pseudo-theories’. examples abound, such as ‘urban myths’ of digital nativism, now widely debunked (bayne and ross, 2011; bullen and morgan 2016; jones et al. 2010), and ‘intuitively appealing’ learning styles (kirschner and van merriënboer 2013). the veracity of the claims behind learning styles has been tested in systematic reviews (coffield et al. 2004; cuevas 2015; pashler et al. 2008), which concluded that there was no evidence that learners inhabit a single learning style exclusively, nor is catering for learning styles an effective practice to improve learning. however, the persistence of the learning styles ‘neuromyth’ (newton, 2015) highlights the need by educators to have some kind of model of learning to frame their teaching. it aligns with intuitive understandings of learners as individuals. these add fuel to the argument that this field needs a greater theoretical examination of its practices. purpose the purpose of the research project was to explore current teaching practices with digital technologies through the lens of educators’ experiences and beliefs concerning both teaching and technology. the doctoral research project, from which this article partially draws upon, addressed the following question: ‘what is the role of theory in teaching with digital technologies in universities?’ specifically, it aimed to explore explicit and implicit theorisations, rationales and principles of teaching as presented by educators themselves and mapped these against a selection of educational and technological theories. this article focusses on the relationship between these educators’ teaching practices and the paradigms discussed above, and asks the following question: ‘what is the role of theories of learning in digital teaching in universities?’ methodology the research design was based on qualitative methods comprising semi-structured interviews with participants who taught in higher education (the interview schedule is presented in appendix 1). two universities in scotland and ireland were chosen, not for comparative reasons but for a richer dataset from contexts where the professionalisation of teaching in higher education was being addressed at institutional and national levels (hunt and higher education strategy group 2011; moir 2012; the scottish government 2010). recruitment of participants was done through gatekeepers in central academic development units and continued in tandem with interviews until the point of data saturation (bryman 2012). purposeful sampling (patton 2002) http://dx.doi.org/10.25304/rlt.v27.2094 research in learning technology citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 5 (page number not for citation purpose) was used to fulfil the criteria to ensure a representation across disciplines, age and gender. during recruitment, effort was made to include participants who had a range of experience and confidence with technology, including those with little or no experience. a small number of participants were recruited, which included those who had limited or narrow experience of technology and those who classified their experience as ‘negative’. in total, 25 educators from seven disciplines were interviewed. a list of the participants with their disciplines and ages is presented in table 1 and more information on the breakdown of institutions by faculty, subjects, levels and modes of teaching is presented in appendix 2. data analysis was performed in two phases: the first phase was a sweep of the data with coding for emergent themes, and the second phase comprised an a priori search for themes addressing what the educators do with technology, and why. this article addresses the findings from the second phase of analysis which employed a ‘light touch’ rhizoanalysis (masny 2013). rhizoanalysis has been considered as a means to disrupt the limitations of qualitative research (cumming 2015) and is based on deleuze and guattari’s rhizome theory (1988). rhizoanalysis entails mapping multiple connections between different parts of the data, without resorting to hierarchical structures which keep data within silos. any point can connect to any other. through building a map of these connections it was possible to construct overlapping layers of teaching practices, theories (including pseudo-theories) and rationales. an example of this can be seen in figure 1,which displays a subsection of a spreadsheet displaying table 1. list of participants and their discipline and age range. pseudonym discipline age range (years) abigail business 40–49 alice business 40–49 andrea nursing 40–49 avril health sciences 50–59 brendan engineering and technical communication 40–49 darren computer science 50–59 emma nursing 40–49 esme computer science 40–49 gary engineering 40–49 gerard biomedical and manufacturing science 30–39 ivan engineering 50–59 james business 40–49 jane nursing 50–59 janice business 30–39 judy business 50–59 kevin environmental sciences 30–39 lynn sociology 50–59 mike engineering 40–49 nuala social sciences 40–49 pam business 40–49 paul history 50–59 pearce physics 40–49 philip medicine 40–49 steve politics 30–39 victor engineering 50–59 http://dx.doi.org/10.25304/rlt.v27.2094 l. drumm 6 citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) name teaching practices theories/concepts/pseudo-theories rationales c o re m at er ia ls c o re a ct iv it y a lt er n at iv e ro u te s u p p le m en ta ry a ct iv e e xp er ie n ti al in d ep en d en t s o ci al l ea rn in g st yl es d ig it al n at iv es a u th en ti c c o nv en ie n ce e ffi ci en t abigail false true false true false true false false false true false true true alice false true false false true true true true false false true false false andrea true true true false true false true false false false false false false avril true false true false false true false false false false false false false brendan true false false false false false false false false false false false false darren false true false false false false false false false true false true true emma false true false false true true false true false false true false false esme false true true false false false false true false false false true false gary true true true false false false false false false false false false false figure 1. example section of cross-reference spreadsheet for data analysis. the presence (‘true’)or absence (‘false’) of meta-themes for each participant and cross references these against their teaching practices, their theorisations and rationales. thus, four simple categories of teaching practice – core teaching, core student activity, alternative routes to teaching and supplementary teaching – could be analysed visually according to colour and contrast for patterns in rationales and mapped to theories. as a qualitative study, this project had rich data but limited number of participants (n = 25). as such, the findings are presented as rich ‘snap shots’ of beliefs and practices of a selection of lecturers and are not intended to be generalisable across all higher education settings. results as described in the previous section, there were four distinct teaching practices reported by the participants. figure 2 demonstrates the rhizoanalytic approach where multiple connections build rich layers between individuals and their teaching as each practice was mapped to participants’ pseudonyms, and, as visible by the number of ribbons spanning from right to left, the majority of the participants described more than one practice. i grouped each teaching practice under the following headings: ‘pragmatic’ (core teaching), ‘pedagogic’ (core student activities), ‘broadcasting’ (alternative routes to teaching) and ‘augmenting’ (supplementary materials). these are not to be seen as labels for the individuals, but for the teaching practices they describe and were not unique behaviours. a synopsis of each subgroup is available in figure 3,which maps technology uses, rationales, theories and pseudo-theories, along with digital tools employed for each practice. http://dx.doi.org/10.25304/rlt.v27.2094 research in learning technology citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 7 (page number not for citation purpose) pragmatic digital teaching participants whose practices included ‘pragmatic’ approaches used technologies for the practicalities of core teaching, such as online lectures and materials for distance learning. their reported rationale was flexibility for students, particularly ‘anytime, anywhere’ access, prioritising functional benefits of technology over pedagogical ones. source: image created with circos.ca. figure 2. participants’ connections with multiple teaching practices. figure 3. four teaching practices with digital technologies with purposes, tools, rationales and theorisations/pseudo-theorisations. http://dx.doi.org/10.25304/rlt.v27.2094 l. drumm 8 citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) they used technology to solve the ‘problem’ of off-campus learning. when probed about pedagogy, these educators frequently used phrases such as ‘self-directed’ and ‘independent learning’. however, rather than the generally accepted interpretation of these terms within education as giving learners a choice over direction and content of their learning (meyer, 2010), these educators were using them as a shorthand to giving students materials and wrap-around instructions for working without requiring ongoing support from the lecturer. this reframing of ‘independent learning’ could therefore be described as a ‘folk theory’ built on these educators’ intuited knowledge and experience. there was an underlying thread of control, which could be framed as behaviourist, in some of the pragmatic teaching descriptions, such as restricting learners’ access to content so as to elicit particular behaviours such as post responses on an online forum, for example, ‘they’re given a week, a window of a week to view the class, […] so they’re on a timer’ (andrea, nursing). many of those who practised pragmatic teaching worked in isolation, developing their teaching methods without consulting scholarship or colleagues. these folk pedagogies were often influenced by their own experiences of learning: my eureka moment was if i needed to change a tyre on a car and didn’t [know] how to do it i’d just go youtube it and…i just thought you could make a course built out of lots of video clips. (brendan, engineering) rather than viewing these educators’ actions as ‘reinventing the wheel’, it could be argued that they are, in fact, intuitively forging new pathways, such as those described above, between their discipline and methods of teaching. pedagogic digital teaching the pedagogic subgroup, as the name implies, demonstrated the significant influence of learning theories upon their teaching. their use of technology was driven by a desire to get learners to ‘do’ things such as complete multiple-choice question (mcq) quizzes or collaborate on a wiki, which often contributed to a summative assessment. although these two different activities could be filed, respectively, under behaviourist and constructivist/social constructivist theories of learning, the context of these activities frequently displayed a nuanced blend of pedagogies. for example, nuala (social work) used classroom-based tasks with students constructing frameworks which they later used individually during an ‘open book’ mcq. effectively, she shifted her teaching to encompass social constructivist methods while incorporating the use of an mcq in a non-behaviourist manner. however, at no point did she frame her teaching with anything more pedagogically detailed than ‘it’s all about the learning rather than the assessment’. when asked why she used digital technologies for teaching, she answered: ... [i]t’s a good way of communicating with the students and it means that they can do things in their own time and space. they don’t have to be in the classroom doing it. em, and it works. (nuala, social work) her instrumental answer mirrored answers from other pedagogic educators who used myths like ‘digital natives’ as rationales for their use of digital technologies, http://dx.doi.org/10.25304/rlt.v27.2094 research in learning technology citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 9 (page number not for citation purpose) while never referring to the pedagogical affordances which, from their own descriptions, they appeared to be employing. in contrast to other pedagogic educators, alice (business) referenced social constructivist learning, amongst other theories, when describing her use of a wiki for group work with entrepreneurship students. she articulated its pedagogic principles that emanated from the discipline: ...[e]ntrepreneurship education is built on a lot of experiential learning techniques, em and...social learning. (alice, business) whether they had visible or invisible pedagogies, this subgroup also had a higher propensity than others to use technology myths about their teaching. cultural clichés about technology such as ‘digital natives’ may have been reinforced by their own experiences in the classroom, as most of these educators were using technology in a blended mode with students who were generally more homogenous in age. broadcasting digital teaching this subgroup centred their digital teaching on delivering content online to students as an alternative to existing face-to-face teaching. for example, a face-to-face lecture would be recorded for later viewing by absent students. this communication was a one-way process, with no requirements for students to reciprocate. they saw their use of technology as solving a number of perceived problems: bringing students up to the same ‘level’ of knowledge, providing students with ‘choice’ in learning and accommodating different ‘learning styles’. for example, gerard provided optional materials on a website: i let them use it in their own way if they wish. […]they can dip in a dip out in another way of learning what they’re, what they’re being taught in class as well. (gerard, biomedical sciences) here digital tools and content provided alternative means for students to access materials. teaching practices described by those who broadcast rarely aligned with the three paradigmatic learning theories described above. many rationalised their use of technology with the uncritical adoption of problematic concepts such as learning styles, for example: …[i]n a lecture you don’t get to hit all of them. but if you can give people some video material, something to read, something to do, other skills come into play to support their learning (pearce, physics) the idea that technology can deliver personalised learning based on cognitive differences is evident here, yet none of this group indicated that they had assessed students individually for their needs. learning theories such as cognitivism and constructivism could account for many of the reported teaching practices of those who used broadcasting techniques, but instead these educators rationalised their teaching with folk pedagogies and neuromyths. it would appear that when it comes to using digital technologies for teaching and learning, some educators who do not have http://dx.doi.org/10.25304/rlt.v27.2094 l. drumm 10 citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) access to educational theories will fill the theoretical vacuum with, at best, their own intuited pedagogy, or at worst, highly contested concepts. augmenting digital teaching the final subgroup, those who augmented, used digital technologies experimentally to ‘add value’ to their existing teaching, such as providing preparatory ‘icebreakers’ or short videos as revision aids before exams. some used digital materials to stimulate engagement or contextualise core learning. paul (history), for example, used primary sources during lectures: i think this kind of thing gets them into it, you know, this kind of imagery, they like that sort of thing, […] it’s good to mix in a little more traditional social history imagery from punch. (paul, history) although this subgroup had barely any references, explicit or implicit, to learning theories, there was a consistent theme of encouraging learners to expand their thinking beyond the requirements of assessment to the ‘bigger picture’ of their subject area: i wanted them to know that it’s not only what’s been taught in the class, [it] is the whole world, the whole world is wider than this for any module. (mike, engineering) as such, their teaching with technology, while having a clear intended outcome for their students, did not fall neatly into one of the paradigmatic learning theories, nor did they give any account of how they supported the students to achieve this outcome. this explanatory vacuum may also account for the pronounced use by this group of ill-defined terms like ‘engagement’ or references to technology as entertaining or inherently stimulating, itself something of a cliché. discussion the results of this study indicate that learning theory paradigms play a minor role in teaching with digital technologies, and of them, only social constructivism is employed explicitly by some participants. this study also demonstrates that not only do other, more unfashionable, theories of learning (such as behaviourism) have a part to play, but also pseudo-theories and folk pedagogies make significant contributions to educators’ conceptualisations of their digital teaching. furthermore, it can be seen from the data that these educators employed a range of pedagogical approaches, sometimes within an intricately interdependent framework, echoing the highly connected rhizome (deleuze and guattari 1988). these blended, sometimes invisible, pedagogies challenge dichotomies of ‘good’ or ‘bad’ teaching practices, for example, behaviourism and teacher-focused as ‘bad’ versus social constructivism and student-centred as ‘good’. in effect, these educators were utilising the functionality of digital tools and forging learning experiences through implicitly employing more than one learning theory. http://dx.doi.org/10.25304/rlt.v27.2094 research in learning technology citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 11 (page number not for citation purpose) the ease with which many of these educators spoke at length about their digital teaching practices with scant reference to theories of teaching or learning should not be taken as evidence of the irrelevance of theory to teaching practices. nor should it be viewed as a worrying indictment of teaching, and the support of teaching, in universities.. as demonstrated, digging deeper into their reports, the selected learning theories still relate to practice. however, it would appear that theory vacuums, such as those who taught by ‘broadcasting’, are filled by proxies such as ‘learning styles’. the dominance of folk pedagogies and pseudo-theories presents a credible threat to critical perspectives on digital teaching and learning as they are sticky concepts, often presented as common-sense ‘edtech’ orthodoxies beyond challenge. on the other hand, folk pedagogies respond and change according to personal experiences of teaching, giving educators an agential role in developing their beliefs of what works, in a way that rigid understandings of learning theories do not. there may be a role for such folk pedagogies in improving our understanding and support of teaching with digital technology at universities. while it is perhaps not surprising that implicitly held knowledge about teaching is a struggle for educators to articulate(elias and merriam 1980), the lack of shared language makes discussing theory challenging across disciplines. it may be the case that academic developers and learning technologists, themselves inhabiting precarious and powerless positions(clegg 2009) and operating with limited resources, use clichés and simplified maxims to help educators ‘across the line’ when supporting their use of technology. however, the problem of technology clichés and non-critical understandings of digital teaching is widespread (bayne 2015), reflecting the lack of seriousness with which critical educational research is viewed from policy-makers and institutional leaders down. ideally, educators should critique and adapt ‘best practices’, taking charge of their own pathways of teaching. indeed, as demonstrated in the data, many of these lecturers do this, but there is a block in articulating, reflecting and sharing these pathways. a solution could be to frame academic development and teaching qualifications as a medium for educators to explore their own voices and communicate about their teaching, without requiring them to fit into prescribed orthodoxies. rather than setting folk pedagogies and pseudo-theories as ‘incorrect’, they could be acknowledged and used as starting points for conversations about teaching. conclusion this article addressed the question: ‘what is the role of theories of learning in digital teaching in universities?’ adopting a rhizoanalytical approach to map connections within the data, this study found that, while educators rarely explicitly referenced learning theories, their practices reflected a complex blend of pedagogies. future research which could be undertaken relating to this work includes an examination of theories not accounted for within the learning theory paradigms, such as those which address functional and technological concerns. additionally, rhizome theory in the form of rhizoanalysis has proven to be a productive means to cross reference within the data and undercut qualitative tendencies to present data as hierarchical,thereby allowing for a more holistic view of the realities of teaching with digital technologies. future work should explore in greater detail the theoretical and practical applications of such an approach. http://dx.doi.org/10.25304/rlt.v27.2094 l. drumm 12 citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) teaching practices can, and should, ‘talk back to theory’ (bennett and oliver 2011) as much as theory to practice. there is a need for institutions to adopt strategic approaches that encourage and support critical engagement by educators on their use of digital technologies, bypassing the easy-to-grasp clichés and embracing the complexity of human-technological entanglement. at a practical level this can be supported through academic development and the encouragement of thoughtful, evidence-informed use of educational technologies. the narrative of orthodoxies and pseudo theories can be countered through a renewed focus on professional development of academic developers and learning technologists, specifically in scholarly and research-based skills, and the recognition of these skills in such roles. through identifying connections between learning theories, folk pedagogies and teaching practices, this article provides a contribution to knowledge which can be taken forward, not only by other researchers, but also by academic staff who teach, those who support them and senior staff in higher education. thus, a positive impact on student learning could be attained through explicit discourse and transparency of purpose when digital technologies are employed for teaching and learning in universities. acknowledgements the author wishes to thank the interview participants for their time and contribution to this project. she is also indebted to two anonymous reviewers for providing criticism on an earlier version of this article. references ally, m. (2008) ‘foundations of educational theory for online learning’, in the theory and practice of online learning, ed t. anderson, athabasca university press, edmonton, ab, p. 472. anderson, t. & dron, j. (2011) ‘three generations of distance education pedagogy’, the international review of research in open and distributed learning, vol. 12, no. 3, pp. 80–97. doi: 10.19173/irrodl.v12i3.890. andrews, r. (2011) ‘does e-learning require a new theory of learning? some initial thoughts’, journal for educational research online, vol. 3, no. 1, pp. 104–121. bates, t. (2012) what’s right and what’s wrong about courserastyle moocs, [online] available at: http://www.tonybates.ca/2012/08/05/whats-right-and-whats-wrong-aboutcoursera-style-moocs/ bayne, s. (2015) ‘what’s the matter with “technology-enhanced learning”?’, learning, media and technology, vol. 40, no. 1, pp. 5–20. doi: 10.1080/17439884.2014.915851. bayne, s. & ross, j. (2011) ‘“digital native”and “digital immigrant” discourses’, in digital differences: perspectives on online education, eds s. bayne & r. land, pp. 159–169, [online] available at: http://link.springer.com/chapter/10.1007/978-94-6091-580-2_12 beetham, h. & sharpe, r. (2007) ‘an introduction to rethinking pedagogy’, in rethinking pedagogy for a digital age, eds h. beetham & r. sharpe, routledge, oxon, ox, pp. 41–61. bennett, s. & oliver, m. (2011) ‘talking back to theory: the missed opportunities in learning technology research’, research in learning technology, vol. 19, no. 3, pp. 179–189. doi: 10.1080/21567069.2011.624997. biesta, g. (2005) ‘against learning. reclaiming a language for education in an age of learning’, nordisk pedagogik, vol. 23, pp. 70–82. biesta, g. g. (2012) ‘giving teaching back to education: responding to the disappearance of the teacher’, phenomenology & practice, vol. 6, no. 2, pp. 35–49. http://dx.doi.org/10.25304/rlt.v27.2094 http://www.tonybates.ca/2012/08/05/whats-right-and-whats-wrong-about-coursera-style-moocs/ http://www.tonybates.ca/2012/08/05/whats-right-and-whats-wrong-about-coursera-style-moocs/ http://link.springer.com/chapter/10.1007/978-94-6091-580-2_12 research in learning technology citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 13 (page number not for citation purpose) bryman, a. (2012) social research methods, 4th edn, oxford university press, oxford. bulfin, s., henderson, m. & johnson, n. (2013) ‘examining the use of theory within educational technology and media research’, learning, media and technology, vol. 38, no. 3, pp. 337–344. doi: 10.1080/17439884.2013.790315. bullen, m. & morgan, t. (2016) ‘digital learners not digital natives’, la cuestión universitaria, vol. 7, pp. 60–68. clegg, s. (2009) ‘forms of knowing and academic development practice’, studies in higher education, vol. 34, no. 4, pp. 403–416. doi: 10.1080/03075070902771937. coffield, f., et al., (2004) learning styles and pedagogy in post-16 learning. a systematic and critical review, learning and skills research centre, london. conole, g. & alevizou, p. (2010) a literature review of the use of web 2.0 tools in higher education table of contents, the open university, milton keynes. conole, g., et al., (2004) ‘mapping pedagogy and tools for effective learning design’, computers & education, vol. 43, no. 1–2, pp. 17–33. doi: 10.1016/j.compedu.2003.12.018. cuevas, j. (2015) ‘is learning styles-based instruction effective? a comprehensive analysis of  recent research on learning styles’, theory and research in education, vol. 13, no. 3, pp. 308–333. doi: 10.1177/1477878515606621. cumming, t. (2015) ‘challenges of “thinking differently” with rhizoanalytic approaches: a reflexive account’, international journal of research & method in education, vol. 28, no. 2, pp. 137–148. doi: 10.1080/1743727x.2014.896892. deleuze, g. & guattari, f. (1988) a thousand plateaus: capitalism and schizophrenia, bloomsbury publishing, london. elias, j. l. & merriam, s. (1980) philosophical foundations of adult education, robert e. krieger, malabar, fl. franklin, t. & harmelen, m. (2007) ‘web 2.0 for content for learning and teaching in higher education’, teaching in higher education, vol. 2008, may, pp. 1–29. doi: 10.1080/1356251032000052348. garrison, d. r. (2011) e-learning in the 21st century: a framework for research and practice, taylor & francis. doi: 10.4324/9780203166093. harasim, l. (2012) learning theory and online technologies, new york, routledge. haythornthwaite, c. & andrew, r. (2011) e-learning: theory & practice, london, sage. ho, a., watkins, d. & kelly, m. (2001) ‘the conceptual change approach to improving teaching and learning: an evaluation of a hong kong staff development programme’, higher education, vol. 42, pp. 143–169. doi: 10.1023/a:1017546216800. howard, s. & maton, k. (2011) ‘theorising knowledge practices: a missing piece of the educational technology puzzle’, research in learning technology, vol. 19, no. 3, pp. 191–206. doi: 10.3402/rlt.v19i3.17109. hunt, c. and higher education strategy group (2011) national strategy for higher education to 2030, department of education and skills, dublin, ireland, [online] available at: http:// www.hea.ie/files/files/des_higher_ed_main_report.pdf jaffer, s. (2010) ‘educational technology pedagogy: a looseness of fit between learning theories and pedagogy’, education as change, vol. 14, no. 2, pp. 273–287. doi: 10.1080/ 16823206.2010.522066. johri, a. (2011) ‘the socio-materiality of learning practices and implications for the field of learning technology’, research in learning technology, vol. 19, no. 3, pp. 207–217. doi: 10.1080/21567069.2011.624169. jones, c., et al., (2010) ‘net generation or digital natives: is there a distinct new generation entering university?’, computers and education, vol. 54, no. 3, pp. 722–732. doi: 10.1016/j. compedu.2009.09.022. kanuka, h. (2006) ‘instructional design and e-learning: a discussion of pedagogical content knowledge as a missing construct’, e-journal of instructional science and technology, vol. 9, no. 2, pp. 1–17. kanuka, h. (2008) ‘understanding e-learning technologies-in-practice through philosophies-in practice’, in theory and practice of online learning, ed t. anderson, athabasca university press, edmonton, ab, pp. 91–118. doi: 10.1007/978-1-59745-324-0. http://dx.doi.org/10.25304/rlt.v27.2094 http://www.hea.ie/files/files/des_higher_ed_main_report.pdf http://www.hea.ie/files/files/des_higher_ed_main_report.pdf l. drumm 14 citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) kemp, s. j. (2013) ‘exploring the use of learner-focused teaching approaches in different academic disciplines’, journal of further and higher education, vol. 37, no. 6, pp. 804–818. doi: 10.1080/0309877x.2012.684041. kinchin, i. (2012) ‘avoiding technology-enhanced non-learning’, british journal of educational technology, vol. 43, no. 2, pp. 43–48. doi: 10.1111/j.1467-8535.2011.01264.x. kirschner, p. & van merriënboer, j. j. g. (2013) ‘do learners really know best? urban legends in education’, educational psychologist, vol. 48, no. 3, pp. 169–183. doi: 10.1080/ 00461520.2013.804395. masny, d. (2013) ‘rhizoanalytic pathways in qualitative research’, qualitative inquiry, vol. 19, no. 5, pp. 339–348. doi: 10.1177/1077800413479559. masterman, l. & vogel, m. (2007) ‘practices and processes of design for learning’, in rethinking pedagogy for a digital age, eds h. beetham and r. sharpe, routledge, oxon, ox, pp. 52–63. mayes, t. (1995) ‘learning technology and groundhog day’, hypermedia at work: practice and theory in higher education, (january), university of kent, pp. 28–37. mayes, t. & de freitas, s. (2007) ‘learning and e-learning: the role of theory’, in rethinking pedagogy for a digital age, eds h. beetham & r. sharpe, routledge, london, pp. 13–25. meyer, k. a. (2010) ‘the role of disruptive technology in the future of higher education’, educause quarterly magazine, vol. 33, no. 1, pp. 1–7. moir, j. (2012) developing and supporting the curriculum: challenges and changes for staff, the quality assurance agency for higher education, glasgow, [online] available at: http://www.enhancementthemes.ac.uk/pages/docdetail/docs/publications/developing and-supporting-the-curriculum-challenges-and-changes-for-staff newton, p. m. (2015) ‘the learning styles myth is thriving in higher education’, frontiers in psychology, vol. 6, no. 1908, pp. 1–5. doi: 10.3389/fpsyg.2015.01908. oliver, m. (2011) ‘technological determinism in educational technology research: some alternative ways of thinking about the relationship between learning and technology’, journal of computer assisted learning, vol. 27, no. 5, pp. 373–384. doi: 10.1111 /j.1365-2729.2011.00406.x. oliver, m. (2012) ‘learning technology: theorising the tools we study’, british journal of educational technology, vol. 44, no. 1, pp. 31–43. doi: 10.1111/j.1467-8535.2011.01283.x. olson, d. r. & bruner, j. s. (1996) ‘folk psychology and folk pedagogy’, in the handbook of education and human development: new models of learning teaching and schooling, eds d. r. olson and n. torrance, blackwell, oxford, pp. 9–27. pashler, h., et al., (2008) ‘learning styles: concepts and evidence’, psychological science in the public interest, vol. 9, no. 3, pp. 105–119. doi: 10.1111/j.1539-6053.2009.01038.x. patton, m. q. (2002) qualitative research & evaluation methods, london, sage. selwyn, n. (2009) ‘challenging educational expectations of the social web: a web 2.0 far?’, nordic journal of digital literacy, vol. 4, pp. 72–85. selwyn, n. (2010) ‘looking beyond learning: notes towards the critical study of educational technology’, journal of computer assisted learning, vol. 26, no. 1, pp. 65–73. doi: 10.1111/j.1365-2729.2009.00338.x. selwyn, n. (2013) distrusting educational technology: critical questions for changing times, london, routledge. the scottish government (2010) building a smarter future: towards a sustainable scottish solution for the future of higher education, scottish government, edinburgh, [online] available at: http://www.scotland.gov.uk/resource/doc/335256/0109656.pdf torff, b. (1999) ‘tacit knowledge in teaching: folk pedagogy and teacher education’, in tacit knowledge in professional practice, eds r. j. sternberg and j. a. horvath, mahwah, nj, lawrence erlbaum, pp. 195–214. watters, a. (2016) re·con·figures: the pigeons of ed-tech, hack education, [online] available at: http://hackeducation.com/2016/09/22/pigeon http://dx.doi.org/10.25304/rlt.v27.2094 http://www.enhancementthemes.ac.uk/pages/docdetail/docs/publications/developing-and-supporting-the-curriculum-challenges-and-changes-for-staff http://www.enhancementthemes.ac.uk/pages/docdetail/docs/publications/developing-and-supporting-the-curriculum-challenges-and-changes-for-staff http://www.scotland.gov.uk/resource/doc/335256/0109656.pdf http://hackeducation.com/2016/09/22/pigeon research in learning technology citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 15 (page number not for citation purpose) wheeler, s. (2015) personal technologies in education: issues, theories and debates, [online] available at: https://www.scribd.com/document/264371745/dr-s-wheeler-personal technologies-in-education-issues-theories-and-debates wu, w. h., et al., (2012) ‘investigating the learning-theory foundations of game-based learning: a meta-analysis’, journal of computer assisted learning, vol. 28, no. 3, pp. 265–279. doi: 10.1111/j.1365-2729.2011.00437.x. http://dx.doi.org/10.25304/rlt.v27.2094 https://www.scribd.com/document/264371745/dr-s-wheeler-personal-technologies-in-education-issues-theories-and-debates https://www.scribd.com/document/264371745/dr-s-wheeler-personal-technologies-in-education-issues-theories-and-debates l. drumm 16 citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) appendices: folk pedagogies and pseudo-theories appendix 1: data collection interview schedule warm-up questions a. tell me about your teaching commitments. who/what do you teach and how often? b. what has been your career pathway to this point? part 1: discussion about teaching artefact 1. can you tell me about an example of digital technology used in your teaching? prompts: a. what is its context: the module and programme(s)? b. how did you go about choosing and/or designing it? c. what happened when you used it? d. has it been changed? part 2: general discussion 2. looking at this module overall, what are the key ‘take-aways’ for students in terms of what they learn? prompts: a. how do you go about teaching these? what do you do? b. what do students do to learn them? c. you’ve been teaching for [x from questionnaire] years, would you say you have developed (or are developing) a set of principles or a philosophy for what works for you as a teacher? could you describe them? 3. what is the teaching culture of your department? 4. why do you use digital technologies in your teaching? where do you get your ideas from? 5. are there aspects of your teaching which you believe could not be done using digital technologies? 6. does using technology change your role as an educator, for example, the way you interact with students? does it give you more or less control? 7. are there situations where you would like to do more with digital technologies but there are barriers? what are they? 8. looking at digital technologies from a wider perspective – both personal and professional – what role does it play in your life? prompts: a. are you comfortable with it having that role? b. and for society in general, do you have any hopes or concerns arising from the large-scale use of digital technologies? c. in terms of the use of digital technology in education, do you have any hopes or concerns for students? http://dx.doi.org/10.25304/rlt.v27.2094 research in learning technology citation: research in learning technology 2019, 27: 2094 http://dx.doi.org/10.25304/rlt.v27.2094 17 (page number not for citation purpose) a p p en d ix 2 : s am p le b re ak d o w n b y in st it u ti o n , fa cu lt y, s u b je ct s ta u gh t, l ev el s an d m o d e o f te ac h in g ir is h u n iv e rs it y ( t o ta l p a r ci p a n ts n = 1 2 ) t h o se w h o h e ld t e a ch in g q u a lifi ca � o n = 3 . t h o se w o rk in g t o w a rd s te a ch in g q u a lifi ca � o n = 1 . s ch o o l o r fa cu lt y s u b je ct a re a (n u m b e r o f p a r ci p a n ts ) u n d e rg ra d u a te /p o st g ra d u a te / a d u lt e d u ca o n m o d e (d is ta n ce , b le n d e d o r fa ce -t o -f a ce ) e n g in e e ri n g (2 ) m e ch a n ic a l e n g in e e ri n g ( 1 ) u g & u g , a d u lt e d b le n d e d sc ie n ce c o m m u n ic a � o n ( 1 ) u g f2 -f & d is ta n ce c o m p u � n g (1 ) c o m p u te r sc ie n ce ( 1 ) u g & p g b le n d e d & d is ta n ce p h ys ic s (1 ) p h ys ic s (1 ) u g b le n d e d b u si n e ss sc h o o l ( 3 ) m a n a g e m e n t in fo rm a � o n sy st e m s (1 ) u g & p g b le n d e d in d u st ri a l r e la � o n s & h r ( 1 ) u g b le n d e d a cc o u n ta n cy ( 1 ) u g b le n d e d so ci a l sc ie n ce s (1 ) so ci a l p o lic y & s o ci a l w o rk (1 ) u g & p g b le n d e d n u rs in g , m e d ic in e , b io m e d ic a l & h e a lt h sc ie n ce s (4 ) n u rs in g ( 2 ) u g & p g d is ta n ce & b le n d e d h e a lt h e d u ca � o n ( 1 ) u g & a d u lt e d d is ta n ce b io -m e d ic a l & m a n u fa ct u ri n g ( 1 ) u g & a d u lt e d b le n d e d & d is ta n ce s co � sh u n iv e rs it y ( t o ta l p a r� ci p a n ts n = 1 3 ) t h o se w h o h e ld t e a ch in g q u a lifi ca � o n = 6 . t h o se w o rk in g t o w a rd s te a ch in g q u a lifi ca � o n = 2 . s ch o o l o r fa cu lt y s u b je ct a re a (n u m b e r o f p a r� ci p a n ts ) u n d e rg ra d u a te /p o st g ra d u a te / a d u lt e d u ca � o n m o d e ( d is ta n ce , b le n d e d o r fa ce -t o fa ce ) e n g in e e ri n g , co m p u � n g & e n vi ro n m e n t (5 ) e n g in e e ri n g ( 3 ) u g b le n d e d u g b le n d e d u g & p g b le n d e d & d is ta n ce c o m p u � n g ( 1 ) u g b le n d e d su st a in a b le e n vi ro n m e n ts ( 1 ) u g & p g b le n d e d b u si n e ss & so ci e ty ( 6 ) b u si n e ss m a n a g e m e n t (3 ) u g & p g f2 -f & b le n d e d u g b le n d e d u g b le n d e d so ci a l s ci e n ce h is to ry ( 1 ) u g f2 -f & b le n d e d so ci a l s ci e n ce p o li� cs ( 1 ) u g b le n d e d so ci a l s ci e n ce so ci o lo g y & cr im in o lo g y (1 ) u g b le n d e d h e a lt h & li fe sc ie n ce s (2 ) n u rs in g ( 2 ) u g & p g b le n d e d u g = u n d er gr ad u at e p g = p o st gr ad u at e f -2 -f = f ac eto -f ac e te ac h in g o f st u d en ts b le n d ed = t ea ch in g w h ic h i n co rp o ra te s b o th f ac eto -f ac e an d o n li n e te ac h in g ap p ro ac h es d is ta n ce = d is ta n ce l ea rn in g w h er e st u d en ts c an f u lfi l m o st o f th ei r le ar n in g an d a ss es sm en t at a d is ta n ce f ro m a p hy si ca l ca m p u s http://dx.doi.org/10.25304/rlt.v27.2094 (page number not for citation purpose) 1 *corresponding author. email: s.c.allan@hw.ac.uk research in learning technology 2020. © 2020 s. allan. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 research in learning technology vol. 28, 2020 original research article migration and transformation: a sociomaterial analysis of practitioners’ experiences with online exams stuart allan* director of online learning, edinburgh business school, heriot-watt university, edinburgh, scotland (received: 6 june 2019; revised: 19 october 2019; accepted: 30 november 2019; published: 27 january 2020) many institutions are making the move from pen and paper to online examinations, but the literature offers relatively few critical reflections on the ramifications of such a shift. this research presents evidence of the ways in which the social and human practices of online exams are deeply entangled with the material and technological, and cautions against the reinscribing of essentialist or instrumentalist assumptions about technology in assessment practices. through semi-structured interviews with eight practitioners in norway, the netherlands, the uk and ireland, it analyses the impact, dimensions and limitations of two main discourses: migration, whereby exam technologies are assumed to be neutral instruments used independently by humans to realise their preordained intentions; and transformation, whereby the essential and inalienable qualities of technologies can be released to ‘transform’ or ‘enhance’ assessment. its findings indicate that: (1)  exam technologies are neither inherently neutral nor essentially transformational; (2)  implementation projects underpinned by the migration discourse can be much more complex and resource-intensive than anticipated; and (3) ‘transformative’ change may be value-laden and driven by assumptions. given the complex and entangled nature of online exams, practitioners are encouraged to think creatively about how assessment strategies align with educational goals, to consider the limitations of current discourses and to analyse critically the relational and performative roles of digital technologies. keywords: assessment; digital exams; online assessment; online exams; sociomateriality introduction the emergence and adoption of digital technologies presents educators with new opportunities to think creatively about assessment and to increase its alignment with long-term educational goals (boud and soler 2016; hepplestone et al. 2011; jisc 2018; o’shea and fawns 2014). given the range of skills and capabilities that could feasibly be demonstrated in digital contexts – including new modes of research, meaning making and collaboration – the apparent popularity of online exams1 (ferrell  2014) 1 ‘online exams’ are understood here as high-stakes summative assessment events, mediated by digital technologies, often taking place in a defined place or time and under secure conditions (e.g. invigilation, restrictions on access to course materials, notes or communication). mailto:s.c.allan@hw.ac.uk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v28.2279 s. allan 2 citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 (page number not for citation purpose) may be viewed with surprise by those who have long critiqued the shortcomings of the examination (e.g. gibbs and simpson 2004). while some online exams are computer-based but otherwise largely traditional (i.e. physically invigilated, closed-book), others have attempted to broaden the scope by, for example, pre-releasing a case study or research task, allowing candidates to sit the exam at the time or place of their choosing, or facilitating access to online resources (khare and lam 2008; myyry and joutsenvirta 2015; williams and wong 2009). although the literature offers many insights on the intricate relationship between assessment and learning (boud and soler 2016; carless 2007; rust 2007), in digital contexts assessment practice is sometimes critiqued for adopting a relatively uncritical stance. ferrell (2014), for example, voiced concern that digital technologies are too often ‘bolted on’ (p. 15) to existing assessment processes, with little consideration of underpinning educational values or goals. hillier and fluck (2015), meanwhile, decried many online exam technologies as ‘armoured word processors’ or ‘glorified multiple-choice quiz tools’ (p. 463), conceived of mainly as a means of extending the scale of education while mitigating the risk of students cheating. based on interviews with higher-education staff, i will argue that prominent discourses around practice with online exams embed particular assumptions about technologies and the educational purposes of assessment. further, i will argue that these assumptions, if left unchallenged, could lead to practice that reinscribes the limitations of existing assessment paradigms or fails to meet institutions’ ambitions. i will critically analyse the impact of the entanglement of social and material, human and non-human in practitioners’ experiences with online exams, and will conclude by proposing future directions for research and practice. migration and transformation ripley (2009) outlined two main motivations among institutions adopting online exams. in the first approach, ‘migration’, technology is regarded primarily as an instrument: a vehicle to ‘move traditional paper-based tests to screen versions’ in order to generate ‘administrative gains and service improvements’ (p. 92). in the second approach, ‘transformation’, technology is positioned as an essentially disruptive, almost revolutionary force: transformation ‘sets out to redefine assessment and testing approaches in order to lead educational change’ (p. 92). ‘migration’ and ‘transformation’ are understood here not only as key terms but also as discourses: discursive practices that constrain and enable what can be said and that define ‘what counts as meaningful statements’ (barad 2003, p. 819). within the migration discourse, technologies are conceptualised instrumentally: they are regarded as ‘neutral means employed for ends determined independently by their users’ (hamilton and friesen 2013, p. 3), with their role being ‘to enhance pre-existing personal and societal educational objectives’ (bayne 2015, p. 9). in contrast, in the transformation discourse, technologies are positioned in a largely essentialist way: ‘independent forces for the realisation of pedagogical aims that are intrinsic to them prior to any actual use’ (hamilton and friesen 2013, p. 3), meaning that ‘“learning” can be transformed by the immanent pedagogical value of certain technologies simply by allowing itself to be open to them’ (bayne 2015, p. 9). crucially, both the essentialist and the instrumentalist positions assume the social (human) and material (non-human and technological) to be discrete and http://dx.doi.org/10.25304/rlt.v28.2279 research in learning technology citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 3 (page number not for citation purpose) separate entities. i will challenge this assumption in relation to online exams, drawing on sociomaterial theory. theoretical context and research questions sociomaterial theory problematises the idea that the material/technological is separate from, or subordinate to, the social/human. instead, humans and technologies are seen to be interwoven and co-constitutive in complex, immanent assemblages (fenwick, edwards, and sawchuk 2011; orlikowski 2010). meaning emerges dynamically in and through these assemblages, in which all elements (including technologies) take on a relational role: ‘material things are performative and not inert; they are matter and they matter. they act together with other types of things and forces to exclude, invite and regulate particular forms of participation in enactments’ ( fenwick, edwards, and sawchuk 2011, p. 4). this research seeks to answer the following questions: • how do the migration and transformation discourses manifest themselves in practitioners’ reflections on their practice with online exams? • what are the dimensions, implications and limitations of these discourses? • to what extent are the relationships between social and material dimensions interwoven, relational and co-constitutive in practitioners’ experiences of online exams? • what possibilities exist to expand the horizons of future research on, and professional practice in, online exams in higher education? literature review the literature on online exams in higher education reflects a preoccupation with how this form of assessment can be administered most efficiently and securely. there is an emphasis on operational or technological concerns such as how to prevent cheating and maintain academic integrity (d’souza and siegfeldt 2017; hylton, levy, and dringus 2016; milone et al. 2017; ullah, ziao, and barker 2019) and how to operate exam software most effectively (karim and shukur 2016). there is also some concern with how online exams are perceived by students: for instance, berggren, fili and nordberg (2015) reported students’ apparent enthusiasm for typing as opposed to writing exam responses, while others (cramp et al. 2019; james 2016; karaman 2011) have reported anxiety among students regarding the use of digital technologies in high-stakes exams. while the literature offers insights on both students’ and teachers’ experiences of online assessment more generally (boud and soler 2016; carless 2007; crisp, guàrdia, and hillier 2016; o’shea and fawns 2014), there are relatively few studies that specifically analyse the implications of online exams for the university staff who design or deliver them. among those that can be found, some traces of the migration discourse can be detected: schmidt, ralph and buskirk (2009), for example, referred to digital technologies as tools that can be used to ‘convert’ existing examination practices (p. 1), while escudier et al. (2011) focused on possible efficiency gains while ‘ maintaining the reliability and robustness of traditional methods’ (p. 441). http://dx.doi.org/10.25304/rlt.v28.2279 s. allan 4 citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 (page number not for citation purpose) of the relatively few published studies that provide detailed analysis of the educational implications of online exams in higher education, many highlight the potential benefits of using innovative question types. williams and wong (2009), for example, claimed that setting up wide-ranging, largely unscaffolded questions for students ‘fosters understanding of learning processes in terms of real-life performance as opposed to a display of inert knowledge … [and provides] an effective bridge between a learner’s education and the social context of their professional practice’ (p. 229). others have identified the potential for specifically digital (again largely unstructured, often collaborative) question types to increase constructive alignment and assessment authenticity (khare and lam 2008; myyry and joutsenvirta 2015; newhouse 2011) and facilitate peer-to-peer feedback discussions (karaman 2011). however, cramp et al. (2019) warned that in the absence of thoughtful question design underpinned by detailed engagement with the affordances of digital technologies, online exams might have a negative impact on students’ performance by increasing cognitive load. this article extends the current literature by: providing a new theoretical perspective on practice with online exams by analysing it through a sociomaterial lens; gathering data on practices across multiple contexts2 via in-depth semi-structured interviews; and focusing its analysis on educational rather than administrative dimensions of practice. while not claiming to produce generalisable findings due to its small scale, this article identifies significant themes that can help to guide future practice and research in this area. methods i recruited interviewees using a mixture of purposeful and snowball sampling. at first, through contacts from my professional network, i purposefully recruited six academics and practitioners from the netherlands, norway, ireland and the uk on the basis that they had recent and direct experience of designing, coordinating, supporting and/or grading summative online exams. my intention was to seek out participants whose experiences were well aligned with the research questions and could offer rich information about practices ‘on the ground’. in order to yield further insights, i then asked the initial interviewees to suggest other potential participants. this ‘snowball’ stage led to the recruitment of a further two participants, both from the netherlands. while this meant that half of the final sample (four out of eight interviewees) was located in the netherlands, the dutch participants had experience across a range of educational contexts, and the initial purposeful sample provided insights across several different countries. after the final interview, i felt that the main themes raised by participants were starting to overlap and that any improvements yielded by further interviews would be marginal. therefore, i concluded that a sufficiently rich set of data had been collected to address the research questions (tracy 2013). participants were either academic or technical leads on online exam projects across six organisations, with most working for universities (from a small university college in norway to a large public university in the uk). table 1 provides an overview of the research participants and the practices they described. 2 myyry and joutsenvirta (2015) acknowledged that ‘the specific context and tradition of university examinations in finland’ (p. 129) might have influenced their results. http://dx.doi.org/10.25304/rlt.v28.2279 research in learning technology citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 5 (page number not for citation purpose) i conducted semi-structured interviews via videoconferencing between february and april 2016, with each interview lasting 60−90 min. i prepared a short list of broad questions to gather participants’ reflections on, and future aspirations for, online exams; i then investigated emergent themes via follow-up questioning (tracy 2013). participants were sent full transcripts for feedback and approval before detailed analysis began. participants’ names and any details that might disclose their identity (including the names of institutions and specific projects) were removed prior to coding and analysis. all participants were identified using pseudonyms throughout. using a grounded theory methodology, i performed initial and focused coding before checking and integrating theoretical categories, interrogating emergent themes via an iterative analytical process (glaser and strauss 1967). this research project was approved in line with the moray house school of education (university of edinburgh) ethics processes. all participants provided informed consent. results as they reflected on their experiences, participants outlined some of the contours, dimensions and limitations of the migration and transformation discourses. starting with migration, there was some evidence of instrumentality in participants’ accounts of their motivations for pursuing online exams: from the academic staff perspective, marking time; so much could be saved … and then also the student experience side of things, there was a drive to provide a quick turnaround on marks and feedback. (sandra) i think an online exam works best when the technology is almost not thought of at all … the technology is almost in the background for them. it just works. (martin) table 1. research participants (identified by pseudonyms). participant name(s) institution main role in online exams practices described maria and john large vocational university, the netherlands joint technical leads large-scale, physically invigilated, closed-book online exams anna large modern university, ireland academic lead small-scale, closed-book online exams, mainly in the health sciences field. both physical invigilation and online proctoring. julia national organisation for higher education, the netherlands project lead overview of practice across several dutch institutions lucas small state university college, norway technical lead various, ranging from invigilated, closed-book online exams to collaborative home research papers. martin and sandra large public university, uk joint technical leads large-scale, physically invigilated, closed-book online exams kim large university teaching hospital, the netherlands academic lead assessment of student radiologists via the interpretation of images from volumetric scans http://dx.doi.org/10.25304/rlt.v28.2279 s. allan 6 citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 (page number not for citation purpose) similarly, lucas claimed that in his experience, norwegian students, some of whom were lobbying universities to offer online exams, often viewed exam technologies as an instrument by which to increase ease of use: my impression of the student movement was they just wanted to get this a little bit digitalised, to plug in the old examination model, to make it a little bit easier [to use]. i didn’t see so much innovative thinking in the student movement, where they really wanted to look at [the] use of [the] internet and applying knowledge. (lucas, emphasis added) julia added that the transition to online exams in her context was motivated both by potential efficiency gains and by a more essentialist belief that technologies could enhance learning: if you take an exam for 500 students in a digital way it's much cheaper than when you do it with a paper exam – but on the other hand there is also a growing awareness about … the possibilities of making use of digital forms of assessment to enhance learning. (julia) likewise maria described how the use of analytics in online exams would improve the quality of future exam questions: afterwards, every test is analysed and the questions that are bad will be thrown away … so the quality has improved a lot […] but when the questions are done on paper, we don’t have data on it. … those [paper-based] exams were a lot worse than they are now because i’ve seen them. (maria) some participants advanced the argument that the migration of traditional exams to digital contexts was a logical and pragmatic first step towards more wide-ranging changes at some point in the future. for example, maria described migration online as a forerunner for more authentic summative assessment in the future: it’s a process of growth, and you can’t do everything at the same time … i think in about five years the exams with multiple-choice questions … will become formative and we will need to test in [summative] exams only the ability to do professional things. (maria) however, lucas saw such an approach as inherently flawed. because resources were being absorbed by solving the technical and administrative problems of migration projects, he argued, some institutions were left with little time, money or energy to use digital technologies more creatively: lucas: the problems that arise and the requirements in terms of technical resources, personnel etc. are so great that there’s an enormous amount of time and energy and money that will have to go into that problem, and that aspect of assessment is what’s using up 95% of our resources now. interviewer: ok. and do you think that’s a necessary step to take? lucas: no, i don’t. i see it as what’s happening, but it’s not a necessary step. http://dx.doi.org/10.25304/rlt.v28.2279 research in learning technology citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 7 (page number not for citation purpose) lucas anticipated that for as long as universities continued to pursue an approach driven by the migration discourse, creative practice with summative assessment would continue to be a secondary concern: when it comes to innovative exams, that’s going much more slowly [than the move from handwritten to typed exams]. i can imagine that within 10 years we’re completely upscaled in terms of using computers to take exams, but still we’re at 50% old-style exams and maybe by the end of this century or maybe by the end of this millennium we’ll finally be approaching really creative use of exams. (lucas) while adopting instrumentalist and essentialist positions, respectively, the migration and the transformation discourses are underpinned by a common assumption: a division between the social (human) and the material (technological). for bayne (2015), this separation has a reductive effect on the scholarship of digital education, arguing that it ‘robs the field of its complexity and richness, reducing our capacity to understand it as a domain of genuine social significance’ (pp. 9−10). in this research, participants made some observations that pointed to a more entangled relationship between humans and technologies. for example, anna said that the extent to which an online exam was appropriate in her context depended as much on the students’ confidence with, and prior experience of, digital technologies as it did on the technology itself: i think it’s really important to map up the individual students themselves and where they’re coming from to the actual assessment that you decide to work with. … we definitely have shades of grey in there. i mean it’s not a one-size-fits-all, for sure. (anna) meanwhile, kim described how digital technologies were deeply intertwined with practice, education and assessment within her discipline, radiology: at the beginning of the 21st century, the whole of clinical practice gradually changed from analogue viewing of images to digital viewing. and that's why also the task just changed, and so i think then you have to change the education as well as the testing. (kim) others identified the potentially constraining influence of wider socioeconomic factors. describing his home country (norway) as ‘a stinking rich country’, lucas argued that its relative wealth was having a negative impact on creativity and was reinforcing practices consistent with the migration discourse: people have so much that they really aren’t forced to be creative […] we’re giving lip service to creative digital assessment but largely [we’re] trying to, as quickly as possible, take this old style of exams over to a digital style. (lucas) some participants were particularly interested in the idea of universities collaborating on large question banks in order to create individually randomised question papers. julia, for example, saw this as potentially increasing the validity and academic rigour of online exams across multiple institutions in the netherlands. however, http://dx.doi.org/10.25304/rlt.v28.2279 s. allan 8 citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 (page number not for citation purpose) she indicated that a combination of human, financial and technological factors was constraining this collaboration, including incompatibility between platforms, lack of funding and lack of adequately trained staff. maria voiced similar concerns, while also highlighting the impact of technology vendors’ resistance towards cross-platform collaboration: it’s a technical thing. it’s ‘how you get the questions you make in system one, how do you get them in system two?’ [there is] a standard way to do that, but it doesn’t always work and … it’s also a bit cultural because the supplier … [doesn’t] want you to get the questions out; they want you to stay in [their] system. (maria) meanwhile, lucas highlighted the dialogical relationship between technologies and assessment practices and argued that truly innovative practice with online exams would problematise the term itself: for me the notion of ‘online exams’, i’m not sure that even describes what i would like to do in terms of assessment […] if our assessment structure could reflect the use of co-operative activity between the students and somehow assessing the product of their co-operation to a much greater extent … then that’s where i’d land. and this has a digital and an online aspect, but that's not the primary aspect. it’s not that it's digital, it's just that it allows for the use of digital resources when appropriate. it’s still the human resources that are vital. (lucas) likewise, sandra perceived a need to shift towards more authentic assessment, which appeared to trouble aspects of traditional examination processes: in the real world, when students graduate they’re going to go out and have access to all of this information and we should be setting assessments that are curating that information rather than saying ‘you need to know this stuff without being able to go and access the information’. (sandra) discussion although the migration and transformation discourses position the use of technology in different ways (instrumentalism and essentialism, respectively), both are underpinned by the same assumption: that the technological and the human are discrete and separate. there are examples of both perspectives here, such as martin positioning technology instrumentally (‘in the background … it just works’) and julia describing technologies’ essential ‘possibilities’ in terms of ‘enhanc[ing] learning’. the strict separation of human and non-human that is embedded in migration and transformation discourses is arguably over-represented in the educational literature and elides significant complexities (bayne 2015; hamilton and friesen 2013; orlikowski 2010). in contrast to instrumentalist and essentialist perspectives, this research reveals some of the ways in which technologies and humans are ‘entangled in cultural, material, political and economic assemblages of great complexity’ (bayne 2015, p. 18) as well as some of the implications of technological entailments for organisational culture (orlikowski and scott 2008). maria and john, for example, identified multiple (technological, social and cultural) dimensions to changing assessment practices. http://dx.doi.org/10.25304/rlt.v28.2279 research in learning technology citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 9 (page number not for citation purpose) in the migration discourse, the process of transplanting traditional exams to digital environments is positioned as what marshall (2010) calls a ‘sustaining’ change; that is, one that ‘improve[s] the function of the organisation in ways that are consistent with previous activities’ (p. 180). decision-makers may be convinced that migration is a first step towards transforming assessment practices over the longer term. however, to borrow anna’s phrase, ‘it’s not a one-size-fits-all’: in reality, a two-step process appears highly challenging. migration may require the resolution of a range of expensive and labour-intensive technological, pedagogical and human dilemmas that mean anticipated short-term benefits fail to materialise. over the longer term, the compromises and resources required to get through the migration process may be so large that more wide-reaching change is shelved indefinitely − as lucas says, such issues can absorb ‘95% of our resources ...[but] it’s not a necessary step’. meanwhile, the essentialism of the transformation discourse, should also be approached with some caution. hamilton and friesen (2013) critique educational research that is ‘framed by assertions of the inevitable and pervasive changes’ (p. 3) resulting from technologies, while o’keeffe (2016) argued that the analysis of data from online exams is often framed in administrative terms, is highly value-laden and ‘promote[s] a very specific and normative vision of how sociomaterial relations in the world should be configured’ (p. 101). johnston, macneill and smyth (2018) go further, critiquing ‘the myth of digital transformation’ (p. 63) and drawing attention to the ways in which a largely transactional transformation narrative builds on pre-existing practices (including pedagogies) without challenging them or proposing alternatives. we should consider what is left unchallenged by transformation and migration discourses, as exemplified by lucas’s description of paying ‘lip service’ to creativity while migrating exams across to ‘a digital style’. the discursive influence of terminology also deserves critical consideration. a range of terms is used in the literature to describe large-scale assessment events such as these, including ‘digital exams’, ‘e-assessment’ and ‘computer-based testing’. ‘online exams’ seems to be the most widely used, although the term itself is rarely defined. terminology can reveal stakeholders’ interests and represent an attempt to define the criteria by which practices and technologies are judged (gillespie 2010). therefore, we might ask ourselves: does the way we describe examinations in digital contexts embed instrumentalist and essentialist ideas about technology while tacitly reinscribing pre-existing assumptions about the educational purposes of assessment? kim urges educators ‘to change the education as well as the testing’, but does the bolting on of a digital-era prefix (whether ‘online’, ‘digital’, ‘computer-based’ or something else) to a pre-digital stem (‘exam’) act to legitimise discourses that have significant limitations under the veneer of practices being, to quote lucas, ‘a little bit digitalised’? moreover, if institutions wish to organise assessment events that are high-stakes and take place under secure conditions, yet align with complex educational aims, then is a new term required? if so, my suggestion would be ‘online assessment under exam conditions’. ‘exam conditions’ would respect operational concern with scale and security, while ‘online assessment’ could loosen ties between assessment design and delivery mode, thereby creating a space for educators to push creatively at the boundaries of traditional exams (e.g. through the prior release of materials, communicating via modes other than text or setting collaborative tasks). the participants in this research offer a more complex picture of practices with  online exams than the migration or transformation discourses would suggest. http://dx.doi.org/10.25304/rlt.v28.2279 s. allan 10 citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 (page number not for citation purpose) they provide evidence that materiality can take a performative role in facilitating, altering or even proscribing assessment practices. at the same time, social dimensions are enacted with and through technologies, resulting in what hannon (2013) terms ‘significant unintended consequences’ (p. 168). these findings provide evidence that the material realities of online exam technologies are intertwined with the human and the social; as such, they problematise the instrumentalist and essentialist assumptions that underpin the migration and transformation discourses. conclusions while small in scale, this research provides a detailed analysis of practitioners’ experiences with online exams across multiple national and institutional contexts. these findings surface and problematise both the migration and the transformation discourses: they illustrate some of the ways in which technology is not neutral and show how essentialism elides potentially significant human and cultural dimensions. this research indicates that the challenges being negotiated in the use of online exams run much deeper than concerns with exam administration and security; to this end, i have argued that the assumptions underpinning practice should be surfaced and analysed critically. building online exams onto unquestioned ideological foundations (such as prior assumptions about the role of technology, the educational outcomes that can or should be assessed, and even the language used to describe these events) risks reinscribing long-standing assumptions that may in fact work contrary to many of the professed aims of contemporary higher education (e.g. the development of skills and dispositions whose benefits extend beyond graduation; boud and soler 2016; johnston, macneill, and smyth 2018). meanwhile, there are examples of university staff struggling to reconcile educational goals with the materiality of particular systems, and of the social and human practices of online exams being enmeshed with, constituted by, and in dialogue with material and non-human dimensions. at a pragmatic level, any expectation of a two-step change process – whereby profound educational change follows on from the (apparently) more prosaic matter of migrating traditional exams online – appears to unravel as universities attempt to solve sometimes complex and unexpected problems. in the absence of critical reflection, the implementation of online exams may incur significant short-term demands that potentially jeopardise anticipated long-term developments. meanwhile, those who would take a transformative approach must subject essentialist claims about technologies’ inherent capabilities to critical scrutiny and consider the ways in which technologies are interwoven with practices. as such, this study endorses hillier and fluck’s (2015) suggestion that skilful handling of ‘embedded cultured attitudes’ (p. 465) is a key requirement of future developments in online exams. practitioners are encouraged to ask the following questions when considering current or future practice with online exams: • what motivations exist for pursuing online exams, understood particularly in  terms of their potential relationships with student-defined or institutional educational goals? http://dx.doi.org/10.25304/rlt.v28.2279 research in learning technology citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 11 (page number not for citation purpose) • in what ways might digital technologies facilitate, proscribe or otherwise mediate the achievement of these goals in online exams? • if a secure event is required to protect assessment validity at scale, how can future practice transcend the traditional limitations of exams – for example, by pre-releasing a research task and/or allowing students to bring notes or access other resources in invigilated spaces? • what opportunities exist to complement online exams (reframed here as ‘online assessment under exam conditions’) with other forms of assessment, for example, multimodal assignments, portfolios or peer-to-peer evaluation? to enhance the existing literature, future research on online exams could seek to stimulate debate; challenge assumptions about the roles of digital technologies; and articulate multiple, bold visions for the future. for example, future studies could: • investigate in detail the relationships between the materiality of particular digital environments and the practices of students, academics and professional staff. • advance understanding of the contexts within which the use of online exams is feasible and/or desirable. there seems a particular need for more case studies considering how the educational purposes of exams are understood and translated into practice, and how institutions might nurture innovative practice. • provide more detailed analyses of student and staff experiences of non-traditional online exams in order to help university decision-makers resolve concerns about exam equivalence, security and validity. as an alternative to migration and transformation discourses, critical engagement with technology and the complexities of institutional contexts is required in order to drive a productive dialogue around practice (orlikowski 2010). the analysis of online exams through the theoretical lens of sociomateriality can help practitioners to challenge instrumentalist and essentialist discourses and to ask new questions about what types of assessment could or should be offered securely and at scale. in resisting these discourses, educators can strive to cultivate ‘a potentially fruitful dialogue between pedagogical values, educational philosophy and technological design’ (hamilton and friesen 2013, p. 14) based on a shared understanding of exam technologies as being deeply entangled with, and co-constitutive of, educational practices. acknowledgements this research was originally conducted for a masters dissertation in digital education at the university of edinburgh. i thank my supervisor there, dr jen ross, for her feedback and guidance during the research and in the preparation of this article. my thanks also go to the anonymous reviewers for their constructive comments and to gill ferrell and martha gibson for their help in recruiting participants. competing interests the author declares no conflict of interest with any organisation regarding the material discussed in this article. http://dx.doi.org/10.25304/rlt.v28.2279 s. allan 12 citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 (page number not for citation purpose) references barad, k. (2003) ‘posthumanist performativity: toward an understanding of how matter comes to matter’, signs, vol. 28, pp. 801–831. doi: 10.1086/345321 bayne, s. (2015) ‘what’s the matter with ‘technology-enhanced learning?’, learning, media and technology, vol. 40, pp. 5–20. doi: 10.1080/17439884.2014.915851 berggren, b., fili, a. & nordberg, o. (2015) ‘digital examination in higher education – experiences from three different perspectives’, international journal of education and development using information and communication technology, vol. 11, pp. 100–108. boud, d. & soler, r. (2016) ‘sustainable assessment revisited’, assessment and evaluation in higher education, vol. 41, pp. 400–413. doi: 10.1080/02602938.2015.1018133 carless, d. (2007) ‘learning-oriented assessment: conceptual bases and practical implications’, innovations in education and teaching international, vol. 44, pp. 57–66. doi: 10.1080/14703290601081332 cramp, j., et al., (2019) ‘lessons learned from implementing remotely invigilated online exams’, journal of university teaching and learning practice, vol. 16, no. 1., [online] available at: https://ro.uow.edu.au/jutlp/vol16/iss1/10 crisp, g., guàrdia, l. & hillier, m. (2016) ‘using e-assessment to enhance student learning and evidence learning outcomes’, international journal of educational technology in higher education, vol. 13. doi: 10.1186/s41239-016-0020-3 d’souza, k. a. & siegfeldt, d. v. (2017) ‘a conceptual framework for detecting cheating in online and take-home exams’, decision sciences: journal of innovative education, vol. 15, pp. 370–391. doi: 10.1111/dsji.12140 escudier, m. p., et al., (2011) ‘university students’ attainment and perceptions of computer-based and traditional tests in a high-stakes examination’, journal of computer assisted learning, vol. 27, pp. 440–447. doi: 10.1111/j.1365-2729.2011.00409.x fenwick, t., edwards, r. & sawchuk, p. (2011) emerging approaches to educational research: tracing the sociomaterial. routledge, london. ferrell, g. (2014) electronic management of assessment (ema): a landscape review, jisc, bristol, [online] available at: http://www.eunis.org/wp-content/uploads/2015/05/ema_ report.pdf gibbs, g. & simpson, c. (2004) ‘conditions under which assessment supports students’ learning’, learning and teaching in higher education, vol. 1, pp. 3–33, [online] available at: http://eprints.glos.ac.uk/3609/ gillespie, t. (2010) ‘the politics of ‘platforms”’, new media and society, vol. 12, pp. 347–364. doi: 10.1177/1461444809342738 glaser, b. b. & strauss, a. l. (1967) the discovery of grounded theory: strategies for qualitative research, transaction, new brunswick, nj. hamilton, e. & friesen, n. (2013) ‘online education: a science and technology studies perspective/éducation en ligne: perspective des études en science et technologie’, canadian journal of learning and technology/la revue canadienne de l’apprentissage et de la technologie, vol. 39, [online] available at: https://www.learntechlib.org/p/54417/. doi: 10.21432/t2001c hannon, j. (2013) ‘incommensurate practices: sociomaterial entanglements of learning technology implementation’, journal of computer assisted learning, vol. 29, pp. 168–178. doi: 10.1111/j.1365-2729.2012.00480.x hepplestone, s., et al., (2011) ‘using technology to encourage student engagement with feedback: a literature review’, research in learning technology, vol. 19, pp. 117–127. doi: 10.3402/rlt.v19i2.10347 hillier, m. & fluck, a. (2015) ‘a pedagogical end game for exams: a look 10 years into the future of high stakes assessment’, in proceedings of the australasian society for  computers in learning in tertiary education (ascilite), eds t. reiners et al.,  perth,  australia, 29  november – 2 december 2015, pp. 463–470, [online] available at: http://www.2015conference.ascilite.org/wp-content/uploads/2015/11/ ascilite2015-proceedings.pdf http://dx.doi.org/10.25304/rlt.v28.2279 http://dx.doi.org/10.1086/345321 http://dx.doi.org/10.1080/17439884.2014.915851 http://dx.doi.org/10.1080/02602938.2015.1018133 http://dx.doi.org/10.1080/14703290601081332 https://ro.uow.edu.au/jutlp/vol16/iss1/10 http://dx.doi.org/10.1186/s41239-016-0020-3 http://dx.doi.org/10.1111/dsji.12140 http://dx.doi.org/10.1111/j.1365-2729.2011.00409.x http://www.eunis.org/wp-content/uploads/2015/05/ema_report.pdf http://www.eunis.org/wp-content/uploads/2015/05/ema_report.pdf http://eprints.glos.ac.uk/3609/ http://dx.doi.org/10.1177/1461444809342738 https://www.learntechlib.org/p/54417/ http://dx.doi.org/10.21432/t2001c http://dx.doi.org/10.1111/j.1365-2729.2012.00480.x http://dx.doi.org/10.3402/rlt.v19i2.10347 http://www.2015conference.ascilite.org/wp-content/uploads/2015/11/ascilite-­2015-proceedings.pdf http://www.2015conference.ascilite.org/wp-content/uploads/2015/11/ascilite-­2015-proceedings.pdf research in learning technology citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 13 (page number not for citation purpose) hylton, k., levy, y. & dringus, l. p. (2016) ‘utilizing webcam-based proctoring to deter misconduct in online exams’, computers and education, vol. 92–93, pp. 53–63. doi: 10.1016/j. compedu.2015.10.002 james, r. (2016) ‘tertiary student attitudes to invigilated, online summative examinations’, international journal of educational technology in higher education, vol. 13. doi: 10.1186/ s41239-016-0015-0 johnston, b., macneill, s. & smyth, k. (2018) conceptualising the digital university: the intersection of policy, pedagogy and practice. palgrave macmillan, london. joint information systems committee (jisc) (2018) designing learning and assessment in a digital age. jisc, bristol, [online] available at: https://www.jisc.ac.uk/guides/ designing-learning-and-assessment-in-a-digital-age karaman, s. (2011) ‘examining the effects of flexible online exams on students’ engagement in e-learning’, educational research and reviews, vol. 6, pp. 259–264. karim, n. a. & shukur, z. (2016) ‘proposed features of an online examination interface design and its optimal values’, computers in human behaviour, vol. 64, pp. 414–422. doi: 10.1016/j. chb.2016.07.013 khare, a. & lam, h. (2008) ‘assessing student achievement and progress with online examinations: some pedagogical and technical issues’, international journal on e-learning, vol.  7, pp. 383–402. marshall, s. (2010) ‘change, technology and higher education: are universities capable of organisational change?’, alt-j, research in learning technology, vol. 18, pp. 179–192. doi: 10.1080/09687769.2010.529107 milone, a. s., et al., (2017) ‘the impact of proctored online exams on the educational experience’, currents in pharmacy teaching and learning, vol. 9, pp. 108–114. doi: 10.1016/j. cptl.2016.08.037 myyry, l. & joutsenvirta, t. (2015) ‘open-book, open-web online examinations: developing examination practices to support university students’ learning and self-efficacy’, active learning in higher education, vol. 16, pp. 119–132. doi: 10.1177/1469787415574053 newhouse, c. p. (2011) ‘using it to assess it: towards greater authenticity in summative performance assessment’, computers and education, vol. 56, pp. 388–402. doi: 10.1016/j. compedu.2010.08.023 o’keeffe, c. (2016) ‘producing data through e-assessment: a trace ethnographic investigation into e-assessment events’, european educational research journal, vol. 15, pp. 99–116. doi: 10.1177/1474904115612486 orlikowski, w. (2010) ‘the sociomateriality of organisational life: considering technology in management research’, cambridge journal of economics, vol. 34, pp. 125–141. doi: 10.1093/ cje/bep058 orlikowski, w. j. & scott, s. v. (2008) ‘sociomateriality: challenging the separation of technology, work and organization’, academy of management annals, vol. 2, no. 1, pp. 433–474. doi: 10.1080/19416520802211644 o’shea, c. & fawns, t. (2014) ‘disruptions and dialogues: supporting collaborative connoisseurship in digital environments’, in advances and innovations in university assessment and feedback, eds c. kreber et al., edinburgh university press, edinburgh, pp. 259–273. ripley, m. (2009) ‘transformational computer-based testing’, in the transition to computerbased assessment, eds f. scheuermann & j. bjornsson, european commission joint research centre, ispra, italy, pp. 92–98. rust, c. (2007) ‘towards a scholarship of assessment’, assessment and evaluation in higher education, vol. 32, pp. 229–237. doi: 10.1080/02602930600805192 schmidt, s. m. p., ralph, d. l. & buskirk, b. (2009) ‘utilizing online exams: a case study’, journal of college teaching and learning, vol. 6, pp. 1–8. doi: 10.19030/tlc.v6i8.1108 tracy, s. j. (2013) qualitative research methods: collecting evidence, crafting analysis, communicating impact, blackwell publishing, oxford. http://dx.doi.org/10.25304/rlt.v28.2279 http://dx.doi.org/10.1016/j.compedu.2015.10.002 http://dx.doi.org/10.1016/j.compedu.2015.10.002 http://dx.doi.org/10.1186/s41239-016-0015-0 http://dx.doi.org/10.1186/s41239-016-0015-0 https://www.jisc.ac.uk/guides/designing-learning-and-assessment-in-a-digital-age https://www.jisc.ac.uk/guides/designing-learning-and-assessment-in-a-digital-age http://dx.doi.org/10.1016/j.chb.2016.07.013 http://dx.doi.org/10.1016/j.chb.2016.07.013 http://dx.doi.org/10.1080/09687769.2010.529107 http://dx.doi.org/10.1016/j.cptl.2016.08.037 http://dx.doi.org/10.1016/j.cptl.2016.08.037 http://dx.doi.org/10.1177/1469787415574053 http://dx.doi.org/10.1016/j.compedu.2010.08.023 http://dx.doi.org/10.1016/j.compedu.2010.08.023 http://dx.doi.org/10.1177/1474904115612486 http://dx.doi.org/10.1093/cje/bep058 http://dx.doi.org/10.1093/cje/bep058 http://dx.doi.org/10.1080/19416520802211644 http://dx.doi.org/10.1080/02602930600805192 http://dx.doi.org/10.19030/tlc.v6i8.1108 s. allan 14 citation: research in learning technology 2020, 28: 2279 http://dx.doi.org/10.25304/rlt.v28.2279 (page number not for citation purpose) ullah, a., ziao, h. & barker, t. (2019) ‘a study into the usability and security implications of text and image based challenge questions in the context of online examination’, education and information technologies, vol. 24, pp. 13–39. doi: 10.1007/s10639-018-9758-7 williams, j. b. & wong, a. (2009) ‘the efficacy of final exams: a comparative study of closedbook, invigilated exams and open-book, open-web exams’, british journal of educational technology, vol. 40, pp. 227–236. doi: 10.1111/j.1467-8535.2008.00929.x http://dx.doi.org/10.25304/rlt.v28.2279 http://dx.doi.org/10.1007/s10639-018-9758-7 http://dx.doi.org/10.1111/j.1467-8535.2008.00929.x (page number not for citation purpose) 1 *corresponding author. email: lindsayrichardson@cunet.carleton.ca research in learning technology 2023. © 2023 l.d. richardson. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 original research article the effects of interactive mini-lessons on students’ educational experience lindsay d. richardson* department of psychology, carleton university, ottawa, on, canada (received: 19 september 2022; revised: 2 february 2023; accepted: 4 march 2023; published: 6 april 2023) with the shift to online learning, many instructors have been forced into course delivery that involves educational lecture videos. there are a number of different elements that impact the quality of educational videos and overall student experience (e.g. instructor eye gaze, audio levels, screen sizing). more specifically, research has demonstrated that segmented videos have educational benefits over the traditional didactic ones. the present experiment aimed to examine whether interspersed interactive content could increase post-secondary students’ retention and engagement above simple segmentation. as such, young adults experienced one of four lesson types: didactic video, segmented videos, segmented videos with interactive content, and a condensed version of the interactive segmented videos. then, they were asked to complete an engagement scale, an online learning experience questionnaire, and a surprise test. the results demonstrated a performance benefit to segmented videos for post-secondary students who prefer to learn in person as opposed to online. keywords: education; online learning; lectures; lessons; interactive introduction online learning is hardly a new concept, as students have had the choice to learn from a distance for many years (beatty 2002). however, the onset of a global pandemic (i.e. march 2020) revoked students’ choice to learn from home. similarly, instructors were forced to teach online courses and many of them had little-to-no experience or guidance in doing so. some instructors found themselves teaching through web-conferencing by using tools such as zoom and microsoft teams. others turned to creating asynchronous courses, wherein most of the course material was delivered via pre-recorded lecture videos. hence, all instructors were required to learn a potentially new skill: media production and multimedia presentation. unfortunately, much of the research on multimedia (e.g. mayer 2009) comes from the late 1990s and early 2000s, coinciding with the surge of technology within the classroom. moreover, much of it pertains to technology within a face-to-face (f2f) classroom and may not always be applicable to an e-learning environment. for research in learning technology vol. 31, 2023 the appendices for this paper are available as supplementary material mailto:lindsayrichardson@cunet.carleton.ca http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2900 http://dx.doi.org/10.25304/rlt.v31.2900 http://dx.doi.org/10.25304/rlt.v31.2900 l.d. richardson 2 citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 (page number not for citation purpose) research that pertains to e-learning, the literature remains quite sparse. this might be, at least in part, due to the challenge of managing courses with different requirements. bates’ (2015) book, entitled teaching in a digital age, exemplifies the difficulty of disseminating best practices in online teaching. bates discussed many general guidelines for teaching and learning online and instructors might find this to be a great starting point. still, many questions have not been answered concretely, likely because there are many nuances related to teaching online and, therefore, the topic itself is so extensive. while all of these nuances are important factors related to online teaching and learning, the present research aimed to investigate one key component: creating engaging educational videos for asynchronous dissemination. while media production practices, such as lighting techniques and sound checks, can vastly increase students’ attention, there are other factors that influence cognitive engagement and learner retention in online lecture videos. in his influential work, mayer (2009) outlined 12 principles of multimedia use in learning that are meant to reduce students’ cognitive load (e.g. excluding extraneous material, highlighting main points, and adding relevant pictures). this present article is interested in learner-centred post-secondary education, wherein instructors are meant to facilitate the learning of young adults. this area of research, interested in adult learning, has been termed andragogy (knowles 1984). andragogy, or adult learning theory, posits that many factors influence the learner’s ability to effectively incorporate new information in a meaningful way: working memory, learner engagement, and motivation to listen. knowles argued that the adoption of the term andragogy acknowledges the difference between teaching children and adults, that is, where pedagogy refers to teacher-centred learning, andragogy encompasses learner-centred designs, wherein the teacher acts as a facilitator of learning. this approach adds another layer of complexity to the issue: creating asynchronous lecture videos that encompass learner-centred course design. creating engaging lecture videos that help learners meet the learning objectives for the course without the presence of the learning facilitator (i.e. instructor) is likely to be a difficult task. to add to this intricacy, the literature is mixed with regard to researchers’ understanding of empirically based practices surrounding online lecture video creation. for instance, some researchers have advocated for shorter lecture videos due to learners’ relatively short attention span (e.g. cooper and richards 2017; jeffries 2010; mayer et al. 1996). jeffries argued for a lecture model in which lectures should comprise 15to 20-min segments. however, bradbury (2016) argued that evidence suggesting a short attention span during lectures is lacking in the primary literature. in fact, bradbury argued that adhering to the 15-min lecture model, with no empirical evidence to support it, implies that we, as instructors of science, ‘don’t really care about evidence’ (p. 513). bradbury’s stance highlights the importance of conducting empirical investigations to determine whether shorter and segmented videos can lead to academic benefits over the typical 90-min didactic lecture model. recently, humphries and clarke (2021) aimed to examine the effect of video length on learner satisfaction and retention. the researchers hypothesised that segmented videos would be perceived as more satisfying by learners compared with the typical didactic lecturing. in addition, they posited that segmented videos would lead to performance benefits such as higher grades. humphries and clark measured students’ preference for didactic videos versus segmented ones over the course of an academic term and collected students’ final grades. their results demonstrated a strong preference towards segmented recordings compared with didactic lecture videos. moreover, learners who watched segmented video lectures outperformed their peers http://dx.doi.org/10.25304/rlt.v31.2900 research in learning technology citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 3 (page number not for citation purpose) academically in the course. this study demonstrates an empirical benefit to shorter lecture videos on measures of student preference and performance. while this research offers a great contribution to the literature, humphries and clark (2021) do not differentiate between active and passive learning in their model. that is, all participants acted as passive learners, as opposed to active ones, in both the segmented and didactic lecture videos. one method for allowing learners to be active in their learning, while watching lecture videos, is to add a component of interactivity to them; that is, allow students a chance to do something with what they are learning. in a f2f lecture, it is not uncommon for an instructor to stop lecturing and pose a question to the class. this simple change in teaching style can allow for students in the classroom to become active learners rather than passive consumers of knowledge. while the focus may still be on the content, this teaching strategy allows the instructor to shift from an instructor-centred approach to a learner-centred one. unfortunately, research on interactive components within educational lecture videos is non-existent. however, some researchers have investigated interactivity in the context of online textbooks. for instance, sommers et al. (2018) investigated the efficacy of interactive textbooks used in post-secondary education. participants were randomly assigned to one of three different textbook conditions: print book, on-screen portable document format (pdf), or interactive ebook. they were then surveyed on satisfaction and were tested on retention. the results demonstrated that students reported increased levels of satisfaction with ebooks compared with both print and pdf documents. additionally, students retained more information from the ebooks compared with print and pdf. this suggests that interactive content may be superior to static content either in print or online. while it has not yet been applied to online lecture videos, it is plausible to expect that the addition of interactive components to an educational video might increase retention and cognitive engagement as well. creating a series of segmented lecture videos with interspersed interactive components might be a challenge, as there are many components that influence students’ engagement and retention. for example, research on conducting online surveys has highlighted the importance of minimising scrolling, zooming, and pinching. more specifically, bacon et al. (2017) found that these actions were related to lower levels of participant engagement in online surveys. since the platform for online surveys is similar to online learning systems in look and feel, this notion can be reasonably applied to online learning as well. accordingly, it would be cognitively taxing to have several lecture components on one page, ultimately forcing students to continue scrolling to finish the lecture. just as powerpoint animations can be either engaging or distracting in lectures (schmaltz and enström 2014), it is likely that other factors contribute to the engaging or distracting nature of an interactive lesson. for instance, the number and type of interactive components between content videos might either engage students or distract from consolidating the information from the previous video. this should be taken into consideration when designing interactive lessons. the goal of this experiment was to investigate the influence of interactivity and video length on students’ engagement and retention. as such, participants experienced one of four different types of lessons, depending on condition: a 20-min didactic lecture (lecture); a 20-min series of segmented videos (series); the 20-min segmented videos interspersed with interactive content (lesson); and a 16-min compact version of the interactive lesson (compact lesson). they then completed a modified version of o’brien et al.’s (2018) engagement scale. after completing a survey about http://dx.doi.org/10.25304/rlt.v31.2900 l.d. richardson 4 citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 (page number not for citation purpose) their recent online learning experiences, participants were asked to complete a surprise retention test. it was hypothesised that participants would report higher levels of engagement for segmented and interactive lessons compared with the typical didactic lecture model. it was also expected that participants in the interactive and segmented lesson would outperform their peers on the test. methods participants participants were 440 undergraduate students from carleton university enrolled in introductory level psychology courses. they were recruited via carleton’s sona system. as such, they were compensated with partial course credit for participation. training the stimuli for this experiment were mini lessons. participants experienced one of four lessons, depending on condition: 20-min video (lecture), five 4-min videos (series), five 4-min videos with interactive content interspersed (interactive lesson), or five 3-min videos with interactive content interspersed (compact interactive lesson). the content for all lessons was the same; that is, the topic for the lesson was the neural basis of addiction. lecture the lecture video was 20 min and 18 s in length and comprised a narrated powerpoint presentation with the occasional appearance of the instructor on screen. the average pace of the lecture was 140 words per minute. for a transcript of the lecture, please see appendix a (supplementary material). series the series was constructed by cutting the lecture video into five videos of approximately equal length. videos were 3:57, 4:01, 4:30, 3:37, and 4:11 min in length. videos varied slightly in length as it was ensured that they were segmented at a natural stopping point (e.g. at the end of a sentence). between each video, participants needed to press a ‘next’ button to advance the lesson. interactive lesson the interactive lesson was created using the same five videos from the series. this was done to eliminate the possibility that video length or modularity could confound the results. just like the series, participants needed to press ‘next’ to advance the lesson. however, for this condition, participants were given tasks (e.g. ‘[…] rank order the following items by how often you would experience that “dopamine dump” in the nucleus accumbens’) between each lecture videos. for a full list of the interactive content (and where they appeared within the lesson), please see appendix b (supplementary material). http://dx.doi.org/10.25304/rlt.v31.2900 research in learning technology citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 5 (page number not for citation purpose) necessarily, the interactive lesson would take more time to complete than the other two lessons. hence, it would be plausible to argue that time-on-task could drive any effects that were found in the interactive lesson condition. in an attempt to rule out that possibility, a fourth condition was created: compact interactive lesson. compact interactive lesson this lesson was an exact replicate of the interactive lesson but with extraneous information cut from the videos. thus, the total video length was 16 min rather than 20. note that no content was eliminated and all interactive materials were kept as well. filler tasks to provide participants with some time in between training and testing, all questionnaires, including demographic information, were administered after the lesson and before the test. firstly, participants were asked to provide the following demographic information: sex, gender, age, ethnicity, first language, english fluency, program major, year of study, and gpa. they were also asked to list their favourite course and the reasoning behind that decision. after providing this information, participants were asked to complete two surveys: an engagement scale and an online lecture experience survey. engagement scale o’brien et al.’s (2018) engagement scale was used to assess participants’ engagement with the simulated lesson. participants were presented with 29 statements (e.g. ‘i lost myself in this experience’; ‘i felt frustrated while watching this lesson’; ‘this lesson was attractive’) and asked to rate them on a 5-point likert scale (i.e. strongly disagree to strongly agree). the entire 29-item scale is presented in appendix c (supplementary material). online lecture experience survey participants were asked about their recent online lecture experiences in the online lecture experience survey. firstly, they were asked how many courses they had taken online. then, they were instructed to think about their recent online learning experiences when answering two sets of statements. for the first set, participants indicated their level of agreement on a 5-point likert scale with statements such as ‘i lack the self-regulatory skills needed to succeed in online learning’ and ‘i appreciate the freedom that online learning provides’. for the second set, they were asked to rate their level of satisfaction with certain learning experiences such as ‘video quality’, ‘level of engagement’, and ‘access to internet’. again, a 5-point likert scale was implemented. the full survey is presented in appendix d (supplementary material). testing participants completed a 15-item retention test that comprised multiple-choice questions. the questions were randomly selected from a pool of 30 questions. the presentation order of the questions was randomised. thus, each participant received a unique version of the test. all 30 questions are presented in appendix e (supplementary material). http://dx.doi.org/10.25304/rlt.v31.2900 l.d. richardson 6 citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 (page number not for citation purpose) general procedure this experiment was conducted online. qualtrics (qualtrics, provo, ut) was used to present all stimuli and instructions and to record all responses and response times. after first providing informed consent, participants immediately experienced one of four lessons, depending on condition: lecture, series, interactive lesson, or compact interactive lesson. after the lesson, participants were asked demographic questions before completing the engagement scale and the online lecture experience survey. participants were then surprised with a 15-item multiple-choice test before viewing the debriefing form. in total, the experiment lasted about 65 min. results firstly, participant compliance was investigated. since the length of the video was 20 min, participants who increased the speed to the maximum (i.e. 2.5x speed) could have finished the lesson in 480 s. participants who did not take at least 480 s to complete the lesson were removed from the dataset due to non-compliance (n = 130, 29.5%). furthermore, 27 participants’ durations were more than two standard deviations above the mean. thus, their data were also eliminated from the dataset (8.7%). next, participants who were expected to complete interactive components were removed if it was found that they failed to do so (n = 5). remaining were 278 participants with unequal sample sizes among conditions: lesson (n = 62), series (n = 63), lesson (n = 74), compact lesson (n = 79). therefore, participants’ data were randomly selected to remain in the dataset to achieve equal sample sizes (n = 60) across conditions. time on task recall that participants completed four different conditions, whereby the compact lesson was created to account for the extended amount of time participants might have spent completing the interactive components of the lesson compared with both the series and the lecture. so, the correction for time on task was examined. in other words, the time on task (i.e. total time completing the lesson) was compared in all conditions. the results are presented in figure 1 and demonstrated that there was a difference in the amount of time it took to complete the different lessons, f(3, 236) = 18.61, p < 0.001. the lesson took the longest to complete (m = 1845.35, sd = 518.20) compared with the other conditions: lecture (m = 1242.33, sd = 513.88), series (m = 1346.99, sd = 523.91), compact lesson (m = 1650.97, sd = 426.64). the compact lesson took significantly longer to complete than the lecture, ∆ = 408.65, p < 0.001 and the series, ∆ = 303.99, p = 0.001. importantly, the lesson did take significantly longer to complete when compared with the compact lesson, ∆ = 90.79, p = 0.03. still, it is impossible to rule out the explanation that any benefit to retention or engagement can be explained by time on task. this means that duration was further considered in all further analyses. learner engagement next, participants’ scores on the engagement scale were analysed. the results found no significant correlation between duration and engagement. thus, duration was not considered as part of the engagement analyses. results also demonstrated that http://dx.doi.org/10.25304/rlt.v31.2900 research in learning technology citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 7 (page number not for citation purpose) participants were generally quite engaged (m = 3.15, sd = 0.68). however, lesson type did not have a significant influence on participants’ self-reported engagement with the lesson, f(3,326) = 0.64, p = 0.59. learner retention before analysing participants’ scores on the surprise test, the test itself was analysed because it had never been validated. firstly, questions 2, 5, 14, 15, 18, 14, and 30 were removed, as the majority of participants chose the wrong answer. secondly, each question’s validity was analysed using a discrimination index, which is one method for analysing assessment quality. specifically, this index evaluates the validity of multiple-choice questions by examining the question’s ability to discriminate between high and low-performers on the same test. as a result, question 28 was removed from further analyses. finally, participants’ scores on the surprise test were analysed. the results found that participants generally performed well (m = 60.66, sd = 18.98). it was also found that duration was not significantly correlated with retention. then, an anova was conducted on score with condition as the independent variable with four levels: lecture, series, lesson, compact lesson. the results were not significant, f(35,2) = 0.86, p = 0.46. a priori predictions were that participants in the series would outperform their peers from the lecture condition, as per humphries and clark’s (2021) results. however, this difference was not significant, ∆ = 5.14, p = 0.14. similarly, even though participants in the lesson condition scored higher on the test (m = 62.31, sd = 18.57) time on task figure 1. the time participants spent completing the different lessons: 20-min video (lecture), five 4-min videos (series), five 4-min videos with interactive content interspersed (interactive lesson), or five 3-min videos with interactive content interspersed (compact interactive lesson). error bars represent ± 2 standard error. the difference between lecture and lesson is significant. so too is the difference between series and lesson. http://dx.doi.org/10.25304/rlt.v31.2900 l.d. richardson 8 citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 (page number not for citation purpose) compared with those from the lecture condition (m = 57.43, sd = 18.97), this difference was not significant, ∆ = 4.88, p = 0.16. online learning preferences to explore the data more granularly, participants’ responses on the online learning questionnaire were examined. the two subscales (i.e. preference towards online learning and satisfaction with their recent experience) were analysed separately. while the majority of participants reported that they would rather take part in f2f courses (m = 4.00, sd = 1.18), 40% of them reported that they enjoyed learning online. moreover, as seen in figure 2, some participants reported that they appreciate the freedom that online learning provides while others reported that they lack the self-regulatory skills need to succeed in online learning. these two points were negatively correlated, r = −0.30, p > 0.001. in terms of feeling equipped and satisfied, the majority of participants reported they were satisfied with video quality, audio quality, instructors’ ability to use technology to teach online, presentation quality, level of understanding, access to internet, and access to a learning space. however, figure 3 shows that participants satisfaction with their own level of engagement was mixed (m = 2.83, sd = 1.17). subsequently, a composite variable was created to denote participants’ relative preference towards online learning. this was done using questions from the online learning experience survey (presented in appendix d). specifically, questions 2, 3, 5, 8, 9, 10 and reverse-coded questions 4 and 7 were averaged to provide one variable to represent a preference for online learning. this new variable suggested that there were some students who preferred online learning and others who did not (m = 2.78, sd = 0.84). thus, a median split was conducted that separated the sample into two distinct groups: online learners and non-online learners. this median split had a significant impact on the results of the effects of lesson type (condition) on learner retention (score). more specifically, an anova was conducted to determine whether there was an overall difference in participants’ score based on condition (lecture, series, lesson, compact lesson) and group (online learners vs. non-online learners). the results of the omnibus test were found to be significant, f(7,232) = 2.46, p = 0.02. thus, the dataset was split based on group and two separate anovas were conducted. the results are shown in figure 4. while there were no significant differences across conditions for online learners, f(3,116) = 0.77, p = 0.52, there were significant differences for non-online learners, f(3,116) = 5.03, p = 0.003. bonferroni-adjusted post hoc comparisons demonstrated that the only significant pairwise difference was between lecture and series, ∆ = 18.02, p = 0.001. no other differences reached significance (all p-values > 0.06). in sum, for students who preferred f2f learning (as opposed to online learning), their score on the test was better when they had engaged in a series of short videos compared with when they had watched one long lecture. discussion recently, many instructors have been required to create more intensive educational media, including asynchronous lecture videos. there are several ways to engage students in online synchronous lectures that might mirror f2f delivery (e.g. creating http://dx.doi.org/10.25304/rlt.v31.2900 research in learning technology citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 9 (page number not for citation purpose) smaller discussion groups, allowing for live polling, responding to student questions in real-time). however, for asynchronous lectures, ensuring that videos are engaging and informative can be a difficult feat. the purpose of this experiment was to determine whether there are engagement and retention benefits to interactive segmented lessons when compared with didactic lecture videos. as such, participants experienced one of four types of lessons, depending on condition: lecture, series, interactive lesson, and compact interactive lesson. after completing a modified version of o’brien et al.’s (2018) engagement scale and an online learning experience survey, participants were asked to complete a surprise test. students’ learning preferences figure 2. participants’ self-reported beliefs toward their recent online learning experience. a rating of 5 indicated that they extremely agreed while a rating of 1 indicated that they extremely disagreed with the statement. http://dx.doi.org/10.25304/rlt.v31.2900 l.d. richardson 10 citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 (page number not for citation purpose) two hypotheses were made. firstly, it was expected that participants would report being more engaged in an interactive lesson compared with a full video lecture. additionally, it was expected that those who experienced the interactive lesson would outperform those who watched the full video lecture on the surprise multiple-choice test. these hypotheses were not fully supported, however. students reported similar levels of engagement for all four conditions. similarly, it did not seem as though interactivity and segmentation had an effect on learner retention. however, once the sample was split based on participants’ preference for online learning (i.e. online learners and non-online learners), an effect of segmentation emerged. it seems as though students who had a tendency to prefer some online learning aspects (e.g. flexibility in learning) were unaffected by interactivity and segmentation of online lecture videos. conversely, students who reported a general dislike for online learning were affected by segmentation. the results demonstrated that non-online learners from the series outperformed their peers from the lecture. this means that the simple act of hitting ‘next’ caused an increase in learning but not engagement. this effect cannot be explained by duration because there was no difference in the amount of time it took participants to complete the series compared with the lecture. limitations and future research even though the results demonstrated an effect of segmentation on learning for students who report a general dislike for the online learning environment, no effect of interactivity was found. moreover, interactivity and segmentation did not affect students’ self-reported level of engagement. a ceiling effect might explain the latter. more specifically, it is plausible to argue that students were too engaged in an interesting topic (i.e. the neural basis of addiction). future research might investigate par�cipants’ level of engagement in their recent online learning experience figure 3. participants’ self-reported level of engagement in their most recent online learning experiences. a rating of 1 indicated that they were extremely dissatisfied while a rating of five indicated they were extremely satisfied. http://dx.doi.org/10.25304/rlt.v31.2900 research in learning technology citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 11 (page number not for citation purpose) topics that are more variable in level of interest. another explanation might be the length of the lessons. regardless of segmentation and interactivity, the stimuli used in this experiment were lessons that took approximately 20 min to complete. it could be that students sustain engagement for this short period of time. future research might consider increasing the amount of time participants spend learning a subject. figure 4. non-online learners’ and online learners’ test score (as a percentage) after completing the different lessons: 20-min video (lecture), five 4-min videos (series), five 4-min videos with interactive content interspersed (interactive lesson), or five 3-min videos with interactive content interspersed (compact interactive lesson). error bars represent ± 2 standard error. the difference between lecture and series is significant only for non-online learners. http://dx.doi.org/10.25304/rlt.v31.2900 l.d. richardson 12 citation: research in learning technology 2023, 31: 2900 http://dx.doi.org/10.25304/rlt.v31.2900 (page number not for citation purpose) segmentation might become more meaningful when the learning phase becomes long enough to disengage students at certain points throughout learning. this present research has provided an initial basis upon which to further examine the effectiveness of interactivity and segmentation. even though no support has been found for a benefit to interactive lessons, there is some indication that segmentation can increase learner retention. references bacon, c., et al., (2017) ‘how effective are emojis in surveys taken on mobile devices? dataquality implications and the potential to improve mobile-survey engagement and experience’, journal of advertising research, vol. 57, no. 4, pp. 462–470. doi: 10.2501/jar-2017-053 bates, a. w. (2015) teaching in a digital age: guidelines for designing teaching and learning, tony bates associates, vancouver, bc. beatty, b. j. (2002) social interaction in online learning: a situationalities framework for choosing instructional methods (order no. 3054431), proquest dissertations & theses global: social sciences, (305512502), [online] available at: https://proxy.library.carleton. ca/login?url=https://www-proquest-com.proxy.library.carleton.ca/dissertations-theses/ social-interaction-online-learning/docview/305512502/se-2?accountid=9894 bradbury, n. a. (2016) ‘attention span during lectures: 8 seconds, 10 minutes, or more?’, advanced physiological education, vol. 40, pp. 509–513. doi: 10.1152/advan.00109.2016 cooper, a. & richards, j. (2017) ‘lectures for adult learners: breaking old habits in graduate medical education’, the american journal of medicine, vol. 130, no. 3, pp. 376–381. doi: 10.1016/j.amjmed.2016.11.009 humphries, b. & clark, d. (2021) ‘an examination of student preference for traditional didactic or chunking teaching strategies in an online learning environment’, research in learning technology, vol. 29, pp. 1–12. doi: 10.25304/rlt.v29.2405 jeffries, w. b. (2010) ‘teaching large groups’, in an introduction to medical teaching, eds w. jeffries & k. huggett, springer, dordrecht, pp. 11–26. knowles, m. (1984) the adult learner: a neglected species, 3rd edn, gulf pub. co., book division, houston, tx. mayer, r. e. (2009) multimedia learning, 2nd edn, cambridge university press, cambridge. mayer, r. e., et al., (1996) ‘when less is more: meaningful learning from visual and verbal summaries of science textbook lessons’, journal of educational psychology, vol. 88, no. 1, pp. 64–73. doi: 10.1037/0022-0663.88.1.64 o’brien, h. l., cairns, p. & hall, m. (2018) ‘a practical approach to measuring user engagement with the refined user engagement scale (ues) and new ues short form’, international journal of human-computer studies, vol. 112, pp. 28–39. doi: 10.1016/j.ijhcs.2018.01.004 qualtrics. (2020) qualtrics (march, 2020) [computer software], provo, ut, [online] available at: https://www.qualtrics.com schmaltz, r. m. & enström, r. (2014) ‘death to weak powerpoint: strategies to create effective visual presentations’, frontiers in psychology, vol. 5, p. 1138. doi: 10.3389/fpsyg.2014.01138 sommers, s. r., et al., (2018) ‘quasi-experimental and experimental assessment of electronic textbook experiences: student perceptions and test performance’, scholarship of teaching and learning in psychology, vol. 4, no. 1, pp. 11–22. doi: 10.1037/stl0000129 http://dx.doi.org/10.25304/rlt.v31.2900 https://doi.org/10.2501/jar-2017-053 https://proxy.library.carleton.ca/login?url=https https://proxy.library.carleton.ca/login?url=https http://www-proquest-com.proxy.library.carleton.ca/dissertations-theses/social-interaction-online-learning/docview/305512502/se-2?accountid=9894 http://www-proquest-com.proxy.library.carleton.ca/dissertations-theses/social-interaction-online-learning/docview/305512502/se-2?accountid=9894 https://doi.org/10.1152/advan.00109.2016 https://doi.org/10.1016/j.amjmed.2016.11.009 https://doi.org/10.25304/rlt.v29.2405 https://doi.org/10.1037/0022-0663.88.1.64 https://doi.org/10.1016/j.ijhcs.2018.01.004 https://www.qualtrics.com https://doi.org/10.3389/fpsyg.2014.01138 https://doi.org/10.1037/stl0000129 doi:10.3402/rlt.v20i0/14431 the implementation and evaluation of a new learning space: a pilot study gail wilson a * and marcus randall b a teaching and learning centre, southern cross university, gold coast, australia; b faculty of business, bond university, gold coast, australia (received 22 april 2011; final version received 14 october 2011) a dramatic, pedagogical shift has occurred in recent years in educational environments in higher education, supported largely by the use of ubiquitous technologies. increasingly, emphasis is being placed on the design of new learning spaces, often referred to as ‘‘next generation learning spaces’’ (ngls) and their impact on pedagogy. the traditional idea of ‘‘classroom’’ now incorporates the use of both physical and virtual space. increasing availability of digital technologies has enabled access by teachers and students to a wider range of communication and information that can now be incorporated into the formal learning process. this change has meant a greater focus on the design and use of flexible learning spaces, more use of blended learning approaches and more personalised, individualised learning opportunities for students. while many such classrooms have been built and used in universities globally, only a few formal studies have been reported on how these spaces are used by both teachers and students. this article focuses on a pilot study of the use by academic staff and students of a next generation learning space � the pod room � and makes recommendations for further research into the effectiveness of new learning spaces in universities. keywords: next generation learning spaces; blended learning; case study introduction bond university is a private, single campus university located on the gold coast in australia, which celebrated its 20th anniversary in 2009. the university operates across three academic semesters a year allowing students to complete their undergraduate degrees in two years. the student cohort at bond comprises australian students and overseas students from 47 different countries. the university’s focus is on face-to-face small group teaching and individualised attention given to students. in 2007 the university revamped one of its traditional classrooms into a next generation learning space called the pod room, taking its name from the kidneyshaped, group work desks that are a feature of the room. this space was designed to facilitate collaborative group work activities across a range of disciplines. the reconstruction of this space was part of a move in other universities across australia to create new learning spaces spurred by the emphasis of the australian learning and teaching council (altc)’s funding of the next generation learning spaces (ngls) project (radcliffe et al. 2009). both authors were contributors to this project. (page number not for citation purpose) *corresponding author. email: gail.wilson@scu.edu.au research in learning technology vol. 20, 2012 rlt 2012. # 2012 g. wilson and m. randall. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons "attribution 3.0 unported (cc by 3.0)" license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited. 1 citation: research in learning technology 2012, 20: 14431 doi: 10.3402/rlt.v20i0.14431 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 in this paper, learning spaces are defined as ‘‘those spaces which encompass the full range of places in which learning occurs, from real to virtual, from classroom to chat room’’ (brown 2005). the paper begins with a literature review summarising the emerging literature in this area, followed by a description of the pilot study and the methodology used to gather data. both qualitative and quantitative analyses of the data obtained in the study are then presented, followed by conclusions and future research directions of this work. literature review the literature relating to learning spaces is diverse and ranges on a continuum from general to more detailed in conceptualisation and analysis with various perspectives explored � architecture, space design, pedagogy, staff and student needs, and stakeholder involvement in the design process. across this literature there is a consistent view that universities should be more innovative and creative in the ways that they use, reconfigure and/or build new learning spaces to meet the expectations of tomorrow’s students. there is broad agreement that learning spaces should be student-centred rather than teacher-centred; have the necessary technology and furnishings to meet student and ‘‘subject’’ needs; support pedagogic, multidisciplinary, multimedia formats that engage the student; and be flexible, ergonomically comfortable, functional and multi-usable. importantly, embedding technology into teaching and learning spaces is ‘‘more of an evolutionary process than a revolutionary one’’ (joint information systems committee 2006, 6). learning spaces are part of the ‘‘new learning paradigm’’ and institutions wanting to be leaders in this area need to implement ‘‘new paths for teaching, learning and space utilisation’’ (johnson and lomas 2005, 14). the emerging area of learning space design is the link that integrates the pedagogy of learning spaces with the technology that is used within them. this section of the paper draws on a sample from a larger number of publications used to frame this study from the us and the uk as well as some australian literature arising from the ngls project mentioned above, and a more recent altc project (lee and tan 2011) which supports the direction of the findings of this paper. this literature was subsequently categorised by the authors as (1) principles or guidelines for practice, (2) descriptions of practice and (3) research-oriented, with some of the literature providing a blend of (1) and (2). principles or guidelines for practice one strand of the literature focused on principles or guidelines for practice addresses the issues surrounding learning space design principles for new as well as repurposed learning spaces within an institution, or anticipating what the future of learning spaces may hold (brown 2009; brown and lippincott 2003; francis and raftery 2005; jamieson, dane, and lippman 2005; johnson and lomas 2005; long and ehrmann 2005; oblinger 2005). while in some of this literature the style of writing can be idealised and uncritical there is value in the guidance provided on a range of issues connected with learning spaces. these issues include the relationship between new spaces for learning and student learning theory; key approaches to the design of learning spaces; the need for linkage between learning spaces and institutional approaches to teaching and g. wilson and m. randall 2 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 learning; and attention to the need for integration of spaces for learning � formal and informal, internal and external � across the institution. the convergence of pedagogy, space and technology provides a framework with which to address a host of issues associated with the design of learning spaces (oblinger 2005): the need for an institutional vision to guide planning, which can be expressed as a set of ‘‘learning principles’’ to guide the design process; the types of analysis and information gathering an institution should do prior to the design of the space; major design principles to guide the design of such spaces; and the importance of ongoing assessment of new learning spaces to inform improvements to current spaces and frame the design of future ones (brown and lippincott 2003). new learning spaces are described in different ways across the literature. in the case of one uk university its institutional framework for blended learning has helped define a ‘‘blended learning landscape’’ for the campus which focuses on the physical spaces of a blended learning environment that promote student collaboration, flexibility of access and use of reusable learning resources in a technology-enhanced learning environment (francis and raftery 2005). the concept of ‘‘learning precincts’’ (jamieson, dane, and lippman 2005, 21), formal/informal and internal/ external areas of a campus, captures the imagination in the visualisation of the entire campus space within which links are formed between spaces that would normally be independent of each other. learning spaces are also seen of as a series of overlapping, cross institutional networks, a single ‘‘ecology’’ (brown 2009, 66) to guide the planning, construction and maintenance of spaces, facilitate standardisation of technology across the campus, link formal, informal and virtual spaces, and facilitate flexibility of use for the campus community. in this perspective all members of the community have a role to play in ensuring the success of these spaces. some of the learning spaces literature is futuristic in style, where, in answer to the question ‘‘what will the classroom of the future look like?’’ authors attempt to forecast what they see as the future for learners, learning and learning spaces. two researchers (long and ehrmann 2005, 56) speculate that the well-designed classroom of the future will need to be comfortable and inviting, accommodating of specific types of learning activities and of different technologies brought to the space. they also need to afford flexible and variegated use across a range of disciplines and offer 24/7 access. descriptions of practice the literature on practice focuses on descriptions of innovation in relation to new learning spaces, particularly emphasising the planning of these spaces before the architects commence their work. the joint information systems committee (jisc) (2006) in the united kingdom offers a comprehensive and visual guide to those planning and designing 21 st century learning spaces, combining text, visual representations and photographs of specific spaces in higher and further education institutions in the united kingdom. these spaces include entrances to buildings, teaching precincts, learning centres and social spaces. in a number of case studies from universities in singapore, australia and the united states, new learning spaces have been deliberately planned to promote an inquiry approach to learning, quite different from traditional teaching and learning spaces in universities (long and holeton 2009). both physical and virtual learning spaces are considered as part of the planning process. planning for evaluation of new learning spaces that links learning space performance to student learning outcomes also needs to occur (dugdale 2009). research in learning technology citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 3 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 research-oriented literature the research literature on new learning spaces is a developing field. some research findings have emerged from learning space projects that have been formally evaluated in the united states (hunley and schaller 2009; tom, voss, and scheetz 2008), and a broad review of the learning spaces research literature (temple 2008) has noted areas for further research. in this section the discussion on evaluation of learning spaces is limited to two studies supported by the altc � the case studies of evaluation of learning spaces within the ngls project report mentioned earlier in this paper (radcliffe et al. 2009), and a more recent project (lee and tan 2011) which has published results from the trials of a learning space evaluation model. a key output of the ngls project is the pedagogy-space-technology (pst) design and evaluation framework (radcliffe 2009, 11�16) by which new learning spaces can be both designed and evaluated. this framework is designed for a range of users across a wide span of projects and institutions. within the framework, the three areas of pedagogy space and technology influence each other and key questions for evaluating learning spaces are framed within each of these three areas. the pst framework is useful for all institutions wishing to evaluate the impact of their spaces in these three areas. in this study it has been is used to frame the discussion of findings found later in this paper. figure 1 represents the framework diagrammatically. user surveys are the most popular way of evaluating teaching spaces across the cases that were part of the ngls project, followed by observational studies. no empirical measures of gathering evaluation data were found in any of the cases, although the project report acknowledged that new learning spaces are more ‘‘space hungry’’ than traditional lecture and tutorial rooms and given this fact, it will be increasingly important ‘‘to gather evidence of positive outcomes from these spaces to counter the inevitable. . . questions of space efficiency’’ (radcliffe et al. 2009, 30). findings from the individual case studies highlighted the benefits of a planned, collaborative approach to the design of learning spaces (huon and sharpe 2009); the need for timely, pedagogical advice for teaching staff in the use of these spaces to gain optimal use of their features (andrews and powell 2009); and the requirements for responsive technical support for the space (reushle et al. 2009). a more recent altc project (lee and tan 2011) has focused on a learning evaluation model trialled across three australian universities. each institution’s trial evaluation focused on different types of learning spaces: a 24/7 learning space, a learning commons, and an advanced concept teaching space (acts). figure 1. the pst framework (adapted from radcliffe 2009, 13). g. wilson and m. randall 4 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 of immediate interest to the pod room study are some of the findings from the evaluation of the acts space � the need for teachers to ‘‘re-imagine’’ their teaching; the learning design of subjects and courses taught in these new spaces; and a need for integrated but broad-based approach to learn in these new spaces; and an ‘‘integrated but broad-based approach to imagining future learning landscapes’’ (lee and tan 2011, 15). pod room pilot study this study of the pod room was conducted in 2008 across two teaching semesters. the following section presents details about the characteristics of the room and the rationale for conducting the pilot study. characteristics of the pod room the pod room comprises a number of integrated components as described in table 1 � five student pods, a single master pod, an informal breakout area and whiteboards. figure 2 shows the layout of the pod room. rationale for the pilot study the rationale for our study was to inform innovation at the university and contribute to the research literature on learning spaces through its formal evaluation of a grouporiented new learning space. the focus of the pilot study was to investigate and analyse the user-centred design and flexibility of the pod room space in relation to the three areas of technology, space and pedagogy. through dissemination such as this paper, we also intended our findings and the investigative framework in figure 1 to inform the design and use of such spaces, not only at our home institution, but further afield as well. it was also necessary to develop a greater understanding of what constituted ‘‘blended learning’’ for the bond campus overall. littlejohn and pegler (2007, 26) see blended learning as ‘‘a hybrid model of e-learning that allows coexistence of conventional face-to-face teaching methods and newer e-learning activities and resources together in a course.’’ at the time of this study, blended learning was in its early stages of adoption across the university. the blackboard tm learning management system (lms) had been introduced in 2006. emphasis was placed on maintaining high quality, face-to-face staff-student contact, while also encouraging technology-enhanced blended learning environments that fostered further innovation in the use of technologies in teaching and learning across disciplines. a 2009 survey of teachers’ use of this system conducted by the university’s teaching and learning centre recommended the need for the university to articulate clearly a vision for blended learning across the campus. methodology a total of seven academic staff comprised the sample for this pilot, four women and three men, selected from a list of academics who had expressed interest in participating in the pilot, and who also represented a spread of disciplines across research in learning technology citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 5 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 the university. five of the seven teachers in the pilot taught undergraduate (ug) students and two of the staff in teacher education taught postgraduate (pg) students only. table 2 shows a list of these staff and their discipline. ethical clearance was gained from the university to conduct this research. an initial meeting was arranged with each teacher to explain the nature of the pilot and to obtain their agreement to participate. a pseudonym was used for each teacher in the study. all teacher and student participants were asked to read an information statement about the study and sign a consent form indicating their willingness to participate in the study. table 1. pod room integrated components. components description student pods these are designed to seat a maximum of six students to form a class working group. each pod has a networked computer that is connected to dual screens. given the width of the pod two screens are necessary but they display only a single image. the facility is also available to allow students to connect their own notebook computer to the pod. additionally, a lighting control is mounted in each desk which allows the students to change the level of lighting directly over their pod. master pod the teacher’s workspace used to control the entire room. it has the ability to override any of the students’ settings such as lighting control. all controls are inbuilt into a single touch panel. technological features of the master pod include: m all lighting � there are front and back house lights, as well as down lights for each pod. each light can be controlled individually from the master pod. m dual projectors � these are wide screen projectors that are capable of displaying computer output as well as dvd movies. m image switching � images from each pod (including the master pod) can be switched to any other pod or projector. this flexibility gives the room many of its educational advantages. the teacher may also choose to preview the image on their screen first before putting it to another device, such as a projector. m document camera � this allows the facilitator to demonstrate real world objects. for example, a marketing academic may wish to display a commercial product. m dvd player � a range of instructional movies and real-world case studies can be displayed this way. m touch pen � the pen acts as a mouse and an annotation tool. often this is used to mark-up a document for illustrative purposes. m external av and computer input � the teacher can use their own notebook computer and/or video equipment (such as an mp3 player) instead of the one inbuilt. informal breakout area this area contains several ottoman furniture pieces. the furniture can be pushed into any configuration and serves to allow groups to talk with one another away from the pod area. teachers can use this area to bring students together at the end of a class for a final discussion and overview of the learning that has taken place in that particular class. whiteboards four whiteboards are located on the side walls to allow student groups to explore ideas for problem solving. as the facilitator has electronic annotation facilities available, there is no whiteboard at the front of the room. g. wilson and m. randall 6 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 the principal author observed one class in the pod room for each of the teachers and took notes on what was observed in each class. a 30-minute interview about the teaching observation scheduled as close as possible to the observed class was then organised with each teacher. all interviews were transcribed and a copy returned to each teacher for checking. each interview focused on one to two issues that had emerged from the observation of the classes beforehand � a particular pedagogical approach used by the teacher, the mix of technologies used in the class, etc. finally, an emailed survey was administered to each academic at the end of the semester. teachers in the study were also asked on a weekly basis to record their ideas, thoughts and concerns as they used the pod room on a personal blog space created in a blackboard tm community site created to support the study. the authors of this paper had access to this blog space for the duration of the study. all seven staff made more than one posting to the blog, with three staff making multiple postings over the duration of the study. information gained from analysis of the blog space comments, the interviews and observation data were used to support and amplify findings from the staff survey. figure 2. pod room layout. table 2. teachers and disciplines involved in the pilot study. teacher discipline 1 languages 2 sustainable development 3 teacher education (pg) 4 teacher education (pg) 5 marketing 6 health sciences 7 information technology research in learning technology citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 7 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 the staff survey questions are provided below: (1) how has the use of the pod room impacted on the way you teach this subject? (2) what types of learning activities do you find this space has facilitated well? (3) what have been the differences, if any, between the way you used this space and other spaces where you have taught this subject? (4) how has the use of this space impacted on your role as a teacher? (5) what do you perceive as the impact this space has had on the student’s experience of learning? (6) what would you add/change/remove from the current pod room and why? a total of 56 students participated in the pilot by completing a survey in class time about their experiences of the pod room at the end of the semester. surveys were distributed by individual teachers and collected and returned by internal mail to the principal author. six questions were included in the student survey, with the sixth question the same as that asked of the teachers: (1) how has the design of the pod room space impacted on the way you approach your learning in this subject? (2) what differences, if any, did you notice this past semester about the way you used this space in your subject, in comparison to the way you engage in small group activities in other group teaching spaces? (3) have you noticed any change in the way you interacted with your teacher? (4) have you noticed any change in the way you interacted with other students? (5) has any aspect of the way you worked as a student in this room really stood out above all others? if so, please describe. (6) what would you add/change/remove from the current pod room and why? analysis of data staff survey data teacher responses to the survey questions are summarised in table 3 below. all teachers responded to all six survey questions. for the last question, the responses belong neither to the ‘‘positive summary’’ nor the ‘‘negative summary’’ categories. the survey responses, together with the interview data and comments from the blog spaces were further analysed using the pst framework mentioned earlier in this paper. these results are presented in the findings section of this paper. student survey data table 4 provides a summary of the results from the student surveys. column 2 in the table shows the overall response rate to the question and the number of responses coded as positive, negative or no difference to each question. note that for the last question responses belong neither to the ‘‘positive summary’’ nor the ‘‘negative summary’’ categories. the survey responses were further analysed using the pst framework mentioned earlier in this paper. g. wilson and m. randall 8 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 table 3. analysis of staff survey questions. questions positive summary negative summary how has the use of the pod room impact on the way you teach this subject? students able to work collectively and effectively on pbl (problem-based leaning) scenarios; students can access the internet and share work with the class; allows activities that encourage student autonomy; flexible learning space; promoted awareness of student learning; allows extended learning activities to be introduced; allows student centred teaching what types of learning activities do you find this space has facilitated well? sharing work of individual groups to the whole class; enabled collaborative learning; enabled pbl activities; easy transition from desk-based activities to class discussion; allows relaxed group discussion; the breakout area facilitates discussion what have been the differences, if any, between the way you used this space and other spaces where you have taught this subject? greater integration of online work using two screens offers greater sharing potential; internet access encourages autonomous learning; this room makes group work possible; flexible seating allows for more academic and social environments students locked into groups, though rearrangement was possible how has the use of this space impacted on your role as a teacher? group work activities, enabled by the pods, engaged students; the room encourages independent learning and peer support what do you perceive as the impact this space has had on the student’s experience of learning? students liked appearance of the room and technology available; the breakout area encouraged student discussion and participation; more effective use of class time; students liked the elevated level of interaction that the room encouraged; greater ownership of their learning; positive group work experiences frustration when equipment not working; visual interference to the front of the room; positioning of whiteboards at the side of the room; lack of instructions in the room as to how to operate the touch panel in the master pod. what would you add, change or remove from the pod room and why? we need a remote control for slides; independent sound systems required for each pod; need simpler controls for the console; technology in the room should be checked regularly; having the lectern can be inconvenient; whiteboards on the sides of the room can be hard to utilise; having a movable, group work table r e se a rch in l e a rn in g t e ch n o lo g y c ita tio n : r e s e a rc h in l e a rn in g te c h n o lo g y 2 0 1 2 ; 2 0 : 1 4 4 3 1 d o i: 1 0 .3 4 0 2 /rlt.v 2 0 i0 .1 4 4 3 1 9 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 table 4. analysis of student survey data. question discussion positive summary negative summary 1. how has the design of the pod room space impacted on the way you approached your learning in this subject? 96% response rate. the majority of responses were positive towards the impact the pod room had on their learning. thirteen students responded negatively to this question. five students recorded no impact. clear access to content and resources; more interactivity; flexible layouts; design facilitated expression of ideas unhelpful technology; physical obstructions in the room; this subject did not suit this space 2. what differences you noticed about the way you used this space this past semester in this subject from the way you engage in small group activities in other small group teaching spaces? 82% percent response rate. the majority of responses were positive. nine students responded negatively to this question. seven students recorded no difference. good computer group work; more and better group and class discussions; relaxed communications computers can lead to an impersonal class; computer monitors are in the way; desks are too distant from each other 3. have you noticed any change in the way you interacted with your teacher in this space? 62% response rate. the majority of responses were positive. seven students responded negatively to the question. fourteen respondents indicated no change. more visible presentations; increased relaxed communications; easy to see computer screens affected the communication negatively; design of the master pod made it difficult to have eye contact with the teacher; desks too far away and spread out to have eye contact with the teacher 4. have you noticed any change in the way you interacted with other students? 60% response rate. the majority of responses were positive. eleven students responded negatively to the question. ten respondents reported no difference. more frequent and effective communication; facilitated sharing written work across computers; sitting two at a table makes partners more accessible; interaction and sharing of ideas with everyone at the same time; sharing with others made learning more interesting less activity with students and more with computers; only interacted with those at my desk; no extended friendships because other students too far away g . w ilso n a n d m . r a n d a ll 1 0 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) c ita tio n : r e s e a rc h in l e a rn in g te c h n o lo g y 2 0 1 2 ; 2 0 : 1 4 4 3 1 d o i: 1 0 .3 4 0 2 /rlt.v 2 0 i0 .1 4 4 3 1 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 table 4 (continued) question discussion positive summary negative summary 5. has any aspect of the way you worked as a student in this room really stood out above all others? if so, please describe 52% response rate. screens on both sides of room facilitated group work, could see the work of others; able to communicate better and with more confidence as there was individual access to information to support my ideas; communication easier with teacher and class; more comfortable space; more engaged; more effective group work and more enjoyable; easier to facilitate class activities and discussions; aided learning feeling of isolation 6. what would you add, change or remove from the pod room and why? 51% response rate. poor quality sound systems; all computers need to be working properly and consistently; pods that fit three people and computers; provide instructions in the room as to how to use it; fewer chairs and more space to move around; screen on desk is a distraction and blocks eye contact with lecturer; lights on master pod distracting; separated the class too much r e se a rch in l e a rn in g t e ch n o lo g y c ita tio n : r e s e a rc h in l e a rn in g te c h n o lo g y 2 0 1 2 ; 2 0 : 1 4 4 3 1 d o i: 1 0 .3 4 0 2 /rlt.v 2 0 i0 .1 4 4 3 1 1 1 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 findings findings from the analysis of both staff and student data gathered during the project are presented below using the pedagogy � space � technology framework. pedagogy (teaching and learning) a large number of participants, both staff and students, reported positive improvements in interactivity and engagement as a result of their experiences in the pod room, arising from the increased opportunities the space afforded for communicative, collaborative, role-playing and problem-based learning style tasks. a few of the staff commented on the ability to facilitate exploratory/discovery style tasks that led to greater individual and group autonomy. the tasks and topics could be ‘‘bigger’’ without ‘‘pedantic little exercises’’. more than half of the staff commented on a reduction in ‘‘teaching’’, in favour of facilitating and guiding more learning-centred teaching. a few staff commented on an increase in selfreflection about their teaching and that teaching in the space over the period of the pilot study had widened their options in terms of choice of teaching strategies a couple of staff made comments about the need to redesign their subjects to better suit the capabilities of the room, and that they needed time for experimentation within the space to maximise the space’s full potential. a single staff member commented on the ‘‘off-the-cuff ’’ capabilities of the room, while another highlighted the flexible curriculum that can develop from having a flexible learning space like the pod room. from observations of the teachers’ classes during data collection, most used a blend of learning activities that combined the use of technologies, individual work and face-to-face discussion. one example of this occurred in a language class in which the teacher made use of websites for key word searches, online games for new vocabulary acquisition and other learning activities that moved smaller groups into larger ones. she also shifted students from their pod desks to the informal learning space at the front of the room for an end-of-class conversation. positive student responses focused overwhelmingly on increased interaction with their classmates in the form of discussion, sharing of ideas, greater participation and the opportunities for peer learning. more than a few students commented on the improved relationship and increased approachability of their teachers, which some attributed to extended communication afforded by the space. one student commented that ‘‘the pod room is perfect for group work and presentations’’. another highlighted the space allowed ‘‘open discussion on an even platform as opposed to not really engaging by commenting and reflecting on what others say’’. another stated ‘‘i asked more questions and got concepts clarified if i didn’t understand’’ and ‘‘i was able to hear and understand the teacher better’’. other students’ comments pointed to the teachers’ use of resources found on computers to support learning activities in the class space. two students undertaking pbl activities in the space noted that these classes were more successful than classes in more traditional spaces because of the ability to do research in class on computers to solve the problem presented by the facilitator. however, a couple of students found the space ‘‘more difficult’’. one student wrote ‘‘i stuck to the same group of people . . . situated on my pod. i didn’t really interact with other students’’, while one student stated ‘‘i felt more distant from the students and didn’t establish the same relationships’’ [as in other classes]. g. wilson and m. randall 12 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 it should also be noted that that more than half of the student survey comments to questions one to four relevant to this element of the pst framework reported ‘‘no difference’’. space (master pod, pod desks, ottoman furniture and whiteboards) most comments about the lectern in the master pod were from the teachers. half of these comments suggested that it was not required, or that it should be moved to the edge of a desk, or the edge of the room. the other half of the comments noted the barrier created by the lectern between the teacher and the students. a very large number of participants responded positively to the pod-shaped desks and their contribution to effective group work. half of the teachers commented on the organic group memberships that arose from the layout of the pod space. the tightness and reliance on these micro-communities stimulated students to consider their membership preferences, more so than in a regular classroom. half of the negative comments with respect to the physicality of the pods were related to the obtrusiveness of the computer monitors. teachers acknowledged it was difficult to see students, and students reported the distraction between them and the teacher, or the presentation space at the front of the room. it was suggested by many that the monitors could be recessed in some way or stored flat in the desks. the other half of the negative comments in this category related to the distance between the five pods and these comments were mainly from the students who claimed little interaction between the pods due to the physical distance. this feeling was heightened with subjects with small class sizes in the pod room, where the distance between students and the teacher was seen as ‘‘too far’’. a very large number of participants responded positively to the ottoman furniture and the informal/breakout learning space closer to the front of the pod room, in the sense that it created an inclusive forum for sharing ideas in full class discussion. it also had the effect of creating a sense of closer relationships between students stemming from the informal tone of the space overall. a single teacher suggested that the ‘‘huddle’’ in the ottoman area was much more productive than trying to share ideas on the class wiki. on the negative side, a couple of students commented on the difficulty in using computers or writing materials while in the ottoman space. a single teacher suggested that the space could be much larger. while it was recognised amongst the teachers in the study that whiteboards are always useful, most teachers commented on the difficulty of transitioning between the ‘‘front’’ of the pod room and the whiteboards which were located on the side walls of the space. even seeing the whiteboards from regular locations around the room (e.g. from the master pod, the ottoman space and the far pods) was difficult, and this was confirmed during the class observations conducted during the study. the solution proposed by some teachers was for the whiteboards to be on wheels. it appears that the teachers were reluctant to provide the virtual whiteboard space afforded by the touch panel and its associated software at the master pod. as noted in table 1, the original intent of the whiteboards was to aid student groups for problem solving exercises. a very large number of participants responded positively to the general feeling created by the room. descriptors such as ‘‘ambience’’, ‘‘happy’’, ‘‘ultra-modern’’, ‘‘comfortable’’, ‘‘relaxed’’ and ‘‘informal’’ were used by respondents. many of these comments were expanded to explain benefits to the learning process such as less fear research in learning technology citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 13 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 and more ownership of the learning environment and the overall positive learning process afforded by the space. interestingly, some students suggested that this space was not suited to their particular subjects. technology (computers at individual pods, technology located within the master pod and projection facilities) a large number of participants responded positively to the projection capabilities of the space, including the replication of projection at the front of the room across all pod monitors. the auxiliary laptop connection at each pod was also acknowledged. teachers in the study spoke of perceived benefits of this projection feature in increased class discussion, a feeling of inclusion, a sharing of work, collaboration, enhanced competition amongst groups in class exercises and a reduction in passive behaviours by students. some of the teachers reported the lack of a presentation remote control as restricting their movements in the pod room, and a few also reported a level of difficulty in operating the central control system with requests for further training and a ‘‘cheat sheet’’ located at the master pod. participants also responded positively to the computer and internet access for research and activities that engaged learners more than traditional worksheets. teachers and students alike recognised the extended potential the technology afforded for pbl research, blogging, wiki creation, use of podcasts, presentation preparation and a discovery approach to learning that could be facilitated by the teacher with access to such a wide array of resources and tools. a single student reported increased confidence from the fast access to information provided by the technology, and another reported improved it skills. the most frequent negative comment with respect to pod technology was that a single computer per pod is inefficient. suggestions were made by students that multiple monitors at each pod would improve the workability of group computer use, but this comment was balanced by at least six comments from students related to the supportive team environment that arose when having only a single computer at each pod, which was the original intent of the design. in summary, more than half of the negative comments relating to technology in general were in regards to reliability. one student stated that both students and teachers needed a ‘‘how to’’ lesson on using the technology because of difficulties in being able to correct any malfunctioning that occurred with technology in the room. a few general comments were made by teachers with regards to the overall availability of technologies, and the placement of this room somewhere between a classroom and a computer lab. there were also a few requests from teachers and students for improved audio capabilities at each pod, as the single earphone port was deemed impractical for multiple users. conclusions and future research the pod room space in this study was an institutional response to the challenge of building a ngls designed to foster collaborative and small group learning and facilitate the use of new technologies across the disciplines. the data arising from this pilot demonstrated the learning benefits achievable by the use of a space like the pod room. it is clear from the survey responses, particularly those of the teachers that the ‘‘next generation’’ orientation of the pod room contributed to enhanced learning g. wilson and m. randall 14 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 experiences and opportunities for students. through observation of and interviews with teachers, a variety of learning ‘‘blends’’ were being used that involved both activity, media and resource blends (littlejohn and pegler 2007), which provided some insights into the nature of blended learning made possible by this space, but also prompting ideas for future more focused research into the teachers’ approaches to blended learning within this space. it is also important to highlight that the academic staff in this study were ‘‘learning by doing’’ in determining the most appropriate teaching approaches to use in this space. as reported in the literature review, teachers working in new learning spaces like the pod room require initial and ongoing support in relation to pedagogy and the appropriate use of technologies to enhance the student learning experience. there was no strong evidence gathered in this study as to how consistent teachers were in briefing students about their use of the space in particular subjects and any ongoing adjustments they may have had to make to the arrangements for group work in the space. given that this was a pilot study, there were a number of limitations that must be borne in mind while considering the findings presented in this article. an important consideration is the need for a greater depth of measuring user response to the pod room. for example, the use of video footage and student focus groups may reveal details of use and potential improvement of the space. more observations of individual teacher’s use of the space would be needed to draw definitive distinctions between the differences in disciplinary use of the room and to tease out the relationship between a teacher’s approach to teaching and the impact of that approach on the design of learning for the space. however, the larger question is whether learning was significantly improved as a direct result of the space. this would be a large undertaking requiring the qualitative and quantitative comparison of learning outcomes of students in the pod room as opposed to a standard classroom. additionally, this study has not considered the effects of having some activities (particularly group and pbl) in the pod room while others are in more conventional settings. the findings from the study also pointed to the need for changes to some areas related to technology and physical space in the design of future learning spaces at the university. aspects of technology in the pod room which hindered what long and ehrmann (2005) called the ‘‘capabilities’’ of new learning spaces such as monitors obstructing views of the front of the room clearly need to be addressed when next an ngls space is created. compiling these points in the form presented in this article effectively adds to the design input process. in fact, many of the negative points raised by both teachers and students were technical and maintenance issues that have been rectified since this study was conducted. most recently, the university has completed a technology review of the pod room, as much of the equipment was due for renewal. in consultation with the second author and users of the room, it was decided to address the issue of technical stability and robustness. as a result, some of the switching capabilities (such as from desk to desk) have been removed. however, software solutions that can more efficiently affect image switching are currently being investigated. this study yielded a number of areas spanning all three dimensions of the pst framework discussed earlier in this paper that warrant further investigation � areas of learner interactivity, learner engagement, teacher and student use of specific technologies within the space, the impact of furniture provided in the space on learning design and discipline-specific approaches to use of this space. also evident in research in learning technology citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 15 (page number not for citation purpose) http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 this study was a need for exploration of the nature and type of professional learning opportunities for academic staff working in new learning spaces, as well as an optimum way of delivering such support. suggestions have been made in the literature that prototype spaces opened up to staff prior to timetabling of teaching in a new learning space is way in which staff understanding of the space and its capabilities can then inform the design of the subject and specifically the nature and type of learning activities that are possible within the space. furthermore, this study has identified areas of enhancement and change that could be presented in a solid business case to university management. acknowledgements the authors acknowledge the financial support provided by the bond university vice chancellor’s research grant scheme used to fund research assistance for this work. the work of the two research assistants � helga noakes and jason daniels � employed in the literature review and data analysis stage of this project is also recognised. a previous version of this paper was published in the peer-reviewed proceedings of the 2010 conference of the australasian society of computers in learning in tertiary education (ascilite). references andrews, t. & powell, d. (2009) ‘collaborative teaching and learning centres at the university of queensland’, in learning spaces in higher education: positive outcomes by design. proceedings of the next generation learning spaces 2008 colloquium, university of queensland, brisbane, eds d. radcliffe, h. wilson, d. powell & b. tibbetts, university of queensland and the australian learning and teaching council, brisbane, pp. 38�39. available at: http://www.uq.edu.au/nextgenerationlearningspace/proceedings brown, m. (2005) ‘learning spaces’, in educating the net generation, eds d. oblinger & j. oblinger, educause e-book, chapter 12. available at: http://net.educause.edu/ir/library/ pdf/pub7101l.pdf brown, m. (2009) ‘inversions’, educause, vol. 44, no. 2, pp. 65�70. available at: http:// www.educause.edu/educause%2breview/ervolume442009/educausereview magazinevolume44/163794 brown, m. & lippincott, j. (2003) ‘learning spaces: more than meets the eye’, educause quarterly, vol. 26, no. 1, pp. 14�16. available at: http://www.educause.edu/educause� quarterly/educausequarterlymagazinevolum/learningspacesmorethanmeetsthe/157 222 dugdale, s. (2009) ‘space strategies for the new learning landscape’, educause, vol. 44, no. 2, pp. 51�62. available at: http://www.educause.edu/educause%2breview/ervolume 442009/educausereviewmagazinevolume44/163794 francis, r. & raftery, j. (2005) ‘blended learning landscapes’, brookes ejournal of learning and teaching, vol. 1, no. 3, pp. 1�5. available at http://bejlt.brookes.ac.uk/vol1/volume 1issue3/perspective/francis_raftery.html hunley, s. & schaller, m. (2009) ‘assessment. the key to creating spaces that promote learning’, educause, vol. 44, no. 2, pp. 26�34. available at: http://www.educause. edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/ 163794 huon, g. & sharpe h. (2009) ‘centre for teaching and learning seminar room’, in learning spaces in higher education: positive outcomes by design. proceedings of the next generation learning spaces 2008 colloquium, university of queensland, brisbane, eds d. radcliffe, h. wilson, d. powell & b. tibbetts, university of queensland and the australian learning and teaching council, brisbane, pp. 73�76. available at: http://www.uq.edu.au/ nextgenerationlearningspace/proceedings jamieson, p., dane, j. & lippman, p. (2005) ‘moving beyond the classroom: accommodating the changing pedagogy of higher education’, refereed forum proceedings of the australian association for institutional research, pp. 17�23. available at: http://www.aair.org.au/app/ webroot/media/pdf/aair%20fora/forum2005/jamieson.pdf g. wilson and m. randall 16 (page number not for citation purpose) citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 http://www.uq.edu.au/nextgenerationlearningspace/proceedings http://net.educause.edu/ir/library/pdf/pub7101l.pdf http://net.educause.edu/ir/library/pdf/pub7101l.pdf http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/learningspacesmorethanmeetsthe/157222 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/learningspacesmorethanmeetsthe/157222 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/learningspacesmorethanmeetsthe/157222 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/learningspacesmorethanmeetsthe/157222 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/learningspacesmorethanmeetsthe/157222 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/learningspacesmorethanmeetsthe/157222 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://bejlt.brookes.ac.uk/vol1/volume1issue3/perspective/francis_raftery.html http://bejlt.brookes.ac.uk/vol1/volume1issue3/perspective/francis_raftery.html http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.uq.edu.au/nextgenerationlearningspace/proceedings http://www.uq.edu.au/nextgenerationlearningspace/proceedings http://www.aair.org.au/app/webroot/media/pdf/aair%20fora/forum2005/jamieson.pdf http://www.aair.org.au/app/webroot/media/pdf/aair%20fora/forum2005/jamieson.pdf http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 johnson, c. & lomas, c. (2005) ‘design of the learning space. learning and design principles’, educause review, vol. 40, no. 4, pp. 16�28. available at: http://www.educause. edu/educause�review/educausereviewmagazinevolume40/designofthelearning spacelearni/157984 joint information systems committee. (2006) designing spaces for effective learning. a guide to 21st century learning space design, joint information systems committee, bristol, uk. available at: http://www.jisc.ac.uk/media/documents/publications/learningspaces.pdf lee, n. & tan, s. (2011) a comprehensive learning space evaluation model. final report, the australian learning and teaching council, sydney. available at: http://www.altc.edu.au/ resource-learning-space-evaluation-swinburne-2011 littlejohn, a. & pegler, c. (2007) preparing for blended e-learning, routledge, london. long, p. & ehrmann, s. (2005) ‘future of the learning space. breaking out of the box’, educause review, vol. 40, no. 4, pp. 42�58. available at: http://www.educause.edu/ educause�review/educausereviewmagazinevolume40/futureofthelearning spacebreaki/157992 long, p. & holeton, r. (2009) ‘signposts of the revolution? what we talk about when we talk about learning spaces’, educause, vol. 44, no. 2, pp. 36�48. available at: http:// www.educause.edu/educause%2breview/ervolume442009/educausereview magazinevolume44/163794 oblinger, d. (2005) ‘leading the transition from classrooms to learning spaces’, educause quarterly, vol. 1, pp. 7�12. available at: http://net.educause.edu/ir/library/pdf/eqm0512.pdf radcliffe, d. (2009) ‘a pedagogy-space-technology (pst) framework for designing and evaluating learning places’, in learning spaces in higher education: positive outcomes by design. proceedings of the next generation learning spaces 2008 colloquium, university of queensland, brisbane, eds d radcliffe, h. wilson, d. powell & b tibbetts, university of queensland and the australian learning and teaching council, brisbane, pp. 11�16. available at: http://www.uq.edu.au/nextgenerationlearningspace/proceedings radcliffe, d., et al. (eds) (2009) learning spaces in higher education: positive outcomes by design. proceedings of the next generation learning spaces 2008 colloquium, university of queensland, brisbane. university of queensland and the australian learning and teaching council, brisbane, available at: http://www.uq.edu.au/nextgenerationlearningspace/ proceedings reushle, s., et al. (2009) ‘tell us about it: establishment of a technology-enhanced learning laboratory’, in learning spaces in higher education: positive outcomes by design. proceedings of the next generation learning spaces 2008 colloquium, university of queensland, brisbane, eds d. radcliffe, h. wilson, d. powell & b. tibbetts, university of queensland and the australian learning and teaching council, brisbane, pp. 111�116. available at: http://www.uq.edu.au/nextgenerationlearningspace/proceedings temple, p. (2008) ‘learning spaces in higher education: an under-researched topic’, london review of education, vol. 6, no. 3, pp. 229�241. tom, j., voss, k. & scheetz, c. (2008) ‘the space is the message. first assessment of a learning studio’, educause quarterly, vol. 2, pp. 42�52. available at: http://www.educause. edu/educause�quarterly/educausequarterlymagazinevolum/thespaceisthe messagefirstasses/162874 research in learning technology citation: research in learning technology 2012; 20: 14431 doi: 10.3402/rlt.v20i0.14431 17 (page number not for citation purpose) http://www.educause.edu/educause+review/educausereviewmagazinevolume40/designofthelearningspacelearni/157984 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/designofthelearningspacelearni/157984 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/designofthelearningspacelearni/157984 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/designofthelearningspacelearni/157984 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/designofthelearningspacelearni/157984 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/designofthelearningspacelearni/157984 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/designofthelearningspacelearni/157984 http://www.jisc.ac.uk/media/documents/publications/learningspaces.pdf http://www.altc.edu.au/resource-learning-space-evaluation-swinburne-2011 http://www.altc.edu.au/resource-learning-space-evaluation-swinburne-2011 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/futureofthelearningspacebreaki/157992 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/futureofthelearningspacebreaki/157992 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/futureofthelearningspacebreaki/157992 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/futureofthelearningspacebreaki/157992 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/futureofthelearningspacebreaki/157992 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/futureofthelearningspacebreaki/157992 http://www.educause.edu/educause+review/educausereviewmagazinevolume40/futureofthelearningspacebreaki/157992 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://www.educause.edu/educause%2breview/ervolume442009/educausereviewmagazinevolume44/163794 http://net.educause.edu/ir/library/pdf/eqm0512.pdf http://www.uq.edu.au/nextgenerationlearningspace/proceedings http://www.uq.edu.au/nextgenerationlearningspace/proceedings http://www.uq.edu.au/nextgenerationlearningspace/proceedings http://www.uq.edu.au/nextgenerationlearningspace/proceedings http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/thespaceisthemessagefirstasses/162874 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/thespaceisthemessagefirstasses/162874 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/thespaceisthemessagefirstasses/162874 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/thespaceisthemessagefirstasses/162874 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/thespaceisthemessagefirstasses/162874 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/thespaceisthemessagefirstasses/162874 http://www.educause.edu/educause+quarterly/educausequarterlymagazinevolum/thespaceisthemessagefirstasses/162874 http://researchinlearningtechnology.net/index.php/rlt/article/view/14431 (page number not for citation purpose) 1 *corresponding author. email: khristin.fabian.perth@uhi.ac.uk research in learning technology 2019. © 2019 k. fabian et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2296 http://dx.doi.org/10.25304/rlt.v27.2296 research in learning technology vol. 27, 2019 original research article putting design into practice: an investigation of tpack scores of lecturers in a networked institution khristin fabian*, emma clayes and laura kelly perth college, university of highlands and islands, perth, united kingdom (received: 13 july 2019; revised: 24 october 2019; accepted: 25 october 2019; published: 18 december 2019) demand for blended and online learning environments is increasing and concurrent with this is the changing competencies required for teachers to be able to facilitate learning in both face-to-face and virtual space. the technological pedagogical content knowledge (tpack) is a measure of teachers pedagogical, content and technical knowledge and their skill to embed technology in practice. using the tpack framework, this study explores the relationship between technical skills, learning design and how these relate to pedagogy. the study also investigates how tpack varies by subject area, teaching qualification, and employment. a survey of 112 lecturers from a multi-campus university was conducted. we found that lecturers who have high tpack tend to use more varied and interactive learning activities. tpack did not significantly vary by subject area. it did not also vary between those who received a few trainings over the past year in comparison to those who didn’t. however, significant differences in tpack were found in terms of nature of employment and teaching qualification in higher education. these findings suggest that there is a need to provide a varied approach to develop staff competencies. keywords: blended learning; online learning; tpack; learning design; technology integration; teachers introduction universities have been moving aggressively to expand their online offerings over the past few years either through massive open online courses (moocs), distance learning courses delivered via virtual learning environments (vles), or through a mixture of traditional face-to-face contact and online learning. concurrent with this change are the shifting competencies required for teachers to be able to facilitate learning in both virtual and face-to-face environments. the range of learning activities that can be carried out with technology varies. conole (2007) provided a taxonomy of e-learning activities or learning mediated with technologies: assimilative (such as reading, viewing and listening), information handling, communicative, productive (such as creating, writing and synthesizing) and experiential (such as investigating and performing). these different tasks require a range of tools and strategies that vary by learning space. for example, to simulate the assimilative activities in the classroom in an online learning environment, a lecturer could either use the vle as an online repository or use the virtual classroom software to simulate the classroom experience. mailto:khristin.fabian.perth@uhi.ac.uk http://www.alt.ac.uk/� http://creativecommons.org/licenses/by/4.0/� http://dx.doi.org/10.25304/rlt.v27.2296 k. fabian et al. 2 citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 (page number not for citation purpose) for experiential learning activities, activities could either be in the form of inquiry-based learning such as webquests or the use of virtual labs that simulate science experiments. the shift into blended and online learning requires lecturers to be able to design learning activities that can be delivered in both virtual and physical learning spaces. anderson, barham and northcote (2013) added that ‘a lecturer’s knowledge of how to select and appropriate different technologies to facilitate a variety of learning tasks influences the quality of learner experience in an online course’ (p. 550). teaching expertise goes beyond subject knowledge and now includes having technological competency (benson and ward 2013). however, previous research has shown that there is still a gap in lecturers’ knowledge of the use of learning technologies. jisc (2017) noted that ‘there are huge differences among teaching staff use of digital technologies to support learning, even within the same college departments’. to address the gap in lecturers’ information and communications technology (ict) skills, educational institutions typically roll out technology training in the form of workshops/tutorials (e.g. workshop on how to use the vle). but while these trainings provide staff with the skills for using various software, training on how to align the use of technology with content is essential to allow teachers to appropriate technology in their learning design (foulger, graziano, and schmidt-crawford 2017; rienties et al. 2013). in a systematic review of strategies for teacher training on technology, one of the recommendations was to use an integrated approach to training – one that models how technology can be used in teaching activities (kay 2006). previous research has shown that a successful integration of ict in education follows a strategy that aligns technology use with a pedagogical approach (mishra and koehler 2006; rienties et al. 2013). meadows and henry (2008) noted that ‘because the online world is a categorically different environment, a particular blend of skills and knowledge is necessary’ (p. 6). benson and ward (2013) added that ‘effective online teaching and learning requires an understanding of the unique ways in which technology interacts with subject matter expertise and pedagogical skills to promote student learning’ (p. 154). the technological pedagogical content knowledge (tpack) framework (mishra and koehler 2006) is one strategy used to map lecturers’ use of educational technology. tpack is built-on schulman’s (1986) argument that content knowledge (ck) and pedagogical knowledge (pk) are needed to deliver effective teaching. tpack extends schulman’s model by adding the technology domain, such that it is the intersection of these three knowledge domains that facilitate effective use of technology for learning and teaching. jaikaran-doe and doe (2017) summarise the tpack model as follows: ck which covers knowledge of a the subject area; pk refers to knowledge of teaching methods; technological knowledge (tk) refers to knowledge of various technology; pedagogical content knowledge (pck) refers to the knowledge of teaching methods with respect to the subject matter; technological content knowledge (tck) refers to the knowledge of how technology can be integrated to facilitate learning; technological pedagogical knowledge (tpk) refers to knowledge of how teaching changes as a result of using various technologies. an illustration of the tpack model is shown in figure 1. several instruments have been developed to analyse the tpack framework (archambault and crippen 2009; schmidt et al. 2009). of particular interest to the current study is archambault and crippen’s (2009) survey due to its focus on online learning. however, criticisms of tpack surveys highlight the fuzzy boundaries between the seven domains (archambault and barnett 2010). for example, in http://dx.doi.org/10.25304/rlt.v27.2296� research in learning technology citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 3 (page number not for citation purpose) archambault and barnett’s factor analysis of archambault and crippen’s (2009) 24-item tpack instrument, only three factors emerged in their analysis: pck, technological-curricular content knowledge (tcck) and tk. cubeles and riu’s (2018) study of tpack profiles of spanish university professors using an adopted version of schimdt’s instrument also resulted in three factors. in both cases, ck was integrated into pk, whereas tpk and tck were combined into a single factor. only the original tk remained as a separate factor. cubeles and riu simplify these groupings as: knowledge on teaching without technology, knowledge on teaching with technology and tk (p. 346). these groupings map to archambault and barnett’s three factors. as there are several studies that point to the three factors rather than the original seven, and archambault’s instrument is more aligned to online learning, we carried out this analysis using archambault and barnett’s three-factor instrument. using the tpack framework, the goal of this article is to investigate lecturers’ skills in embedding technology into their design of online learning environments. by looking at the relationship between the various components of tpack, the study aims to explore if there is a relationship between technical skills and design and how this relates to content and pedagogy. lastly, this article aims to explore differences in tpack scores of lecturing staff based on different demographic factors such as nature of employment and subject area. specifically, this study aims to answer the following research questions: 1. to what extent do lecturers demonstrate tpack knowledge? 2. what is the relationship between tpack and design of learning activities? 3. how does tpack vary by nature of employment, teaching qualification and subject area? figure 1. tpack model (mishra & koehler 2006) (reproduced by permission of the publisher © 2012 by tpack.org). http://dx.doi.org/10.25304/rlt.v27.2296� http://tpack.org k. fabian et al. 4 citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 (page number not for citation purpose) literature review studies published on tpack tend to focus on pre-service teachers or teachers of kindergarten to grade 12 (k-12). few studies have unpacked tpack in the higher education (he) sector (anderson et al. 2013; benson and ward 2013; cubeles and riu 2018). in a case study with three he professors, benson and ward (2013) illustrated different tpack profiles and how lecturers operationalise the framework, showing the different degrees of technological, pedagogical and content knowledge. they found that pk is the enabling force in developing higher levels of technology integration skills, more so than tk on its own. this raises questions about the standalone technology training delivered as workshops/seminars to improve lecturers’ capacity to use technology. however, this does not mean that tk is not important. anderson et al.’s (2013) study of tpack profile of lecturers found tk as an enabling force in delivering online programmes. the challenge for lecturers was how to emulate their teaching strategies in face-to-face modules to the online environment. so, while pk precedes tk, this shows that online learning requires a standard level of tk to be able to provide an engaging online learning environment. lecturers also vary in their beliefs about the role technology plays in the classroom, which correlates to their use of technology and perceived competence to use  technology (lee and tsai 2010; marcelo and yot-domínguez 2018). marcelo and yot-dominguez’s study found that lecturers’ use of technology is associated with their confidence in using technology. those who do not have the confidence in their ability to use technologies veer from using them as they feel this affects their  ability to offer a student-focused teaching. reyes et al. (2017) noted that technology integration is more than a dichotomy of lecturers who use technology and those who do not. instead, profiles can be categorised into three: those who are able to integrate technology into their teaching; those who have the technological skills but not the confidence to teach with it; and those who are hesitant towards technology. therefore, competence alone is not the decisive factor, rather it is confidence, both in terms of being able to utilise existing skills and being able to acquire those skills. support is an important factor in developing lecturers’ capacity to use technology. studies that implemented training interventions to improve lecturers’ skills in teaching with technology found significant improvement in the lecturers’ tpack scores before and after the intervention (brinkley-etzkorn 2018; rienties et al. 2013). in koh, chai and tsai’s (2014) study, it was found that teachers who are supported by an educational technologist developed higher levels of tpack in comparison to lecturers who were not. continuous technology training has been consistently highlighted in teacher training literature, but supporting teachers in the use of technology is not only about providing training on how to use technology, but also includes providing continued support and guidance as lecturers navigate the use of technology in the classroom. studies that investigated tpack found that scores tend to vary based on various demographic variables such as teaching experience and subject area. several studies have found that teachers with more experience in using technology had higher measures of self-efficacy in using technology for teaching (corry and stella 2018; cubeles and riu 2018). these findings are not surprising as there is an overwhelming anecdotal evidence that when it comes to technological skills, experience http://dx.doi.org/10.25304/rlt.v27.2296� research in learning technology citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 5 (page number not for citation purpose) facilitates competence. jang and tsai’s (2012) study of more than 800 teachers found that tpack subscale scores except for tk varied by years of teaching experience, with these scores positively correlated to years of teaching service. in cheng and xie’s (2018) study, age and experience were positively correlated with pck, but  negatively correlated with tk. several other studies noted that experienced teachers were less confident about integrating technology (koh et al. 2014; lee and tsai 2010). in jang and tsai’s (2012) study, tk was lowest in comparison to the other subscales and did not vary by years of teaching experience. the tk scores were also lower in comparison to the other subscale scores. several studies echo this finding. scores for pck and tcck (in reference to either the original subscale or the combined subscales) were higher than tk (benson and ward 2013; cubeles and riu 2018). this suggests that both novice and experienced teachers do not feel as confident with their tk as they do with their pck. the demand to use technology and the nature of technology use tend to vary by  subject area. for example, in an investigation of the different ways he lecturers use technology in their teaching, and it was found that social science lecturers tend to use technology to present information, engineering lecturers use experiential learning activities and health science lecturers use it more for communicative learning activities (marcelo and yot-domínguez 2018). cubeles and riu’s (2018) study of he lecturers did not find significant differences in pck and tcck of lecturers from business administration, engineering and architecture lecturers. however, there was a significant difference in the tk scores of engineering lecturers in comparison to architecture lecturers. he lecturers have achieved a certain degree of subject expertise in their field, which could account for the parity of pck and tcck scores. however, different subject areas would vary in their use of technology, hence the significant tk scores between the two faculties. two studies on primary/secondary teachers also found that subject area has a moderating effect to tpack scores (jang and tsai 2012; nelson, voithofer, and cheng 2019). development support has been shown to improve tpack and support can be in the form of technology training, providing guidance on the use of technology, or development support in terms of improving pedagogy. lecturers would need different levels of support based on various factors. lecturers coming into he teaching from a commercial background would likely need pedagogical training. jaikarandoe and doe’s (2017) comparison of tpack scores of he engineering lecturers was significantly lower than pre-service teachers’ tpack scores. nature of employment (whether full-time or part-time) also dictates the support needs (beaton and gilbert 2013). studies in relation to tpack on this regard are few and point to contrary findings (chukwuemeka and iscioglu 2017; voithofer et al. 2019). voithofer et al. found that tpack did not vary by tenure, whereas chukwuemeka and iscioglu found tk and tpk to be higher for part-time staff. this section has drawn together findings on tpackand he-focused studies as well as differentiation of tpack scores based on demographic variables. research that assesses tpack in higher and further education (fe) is few. findings on differentiation of tpack scores by various factors such as subject area and experience have also shown contrasting results. in addressing this gap in the literature, the http://dx.doi.org/10.25304/rlt.v27.2296� k. fabian et al. 6 citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 (page number not for citation purpose) current study investigates the tpack scores of lecturers in a dual-nature institution and will explore how these scores relate to the design of learning activities. methodology setting the study was carried out in a multi-campus university that offers both fe and he programmes. due to the dual nature of the institution, some lecturers teach both fe and he subjects ranging from level 4 to level 12 in the scottish credit and qualification framework. the university consists of different academic partners (fe colleges and research centres) dispersed across the region. there are two faculties within the university: faculty of arts, humanities and business, under which subjects on arts, social sciences and business are contained; whereas science-related disciplines and engineering are under the faculty of science, health and engineering. degree programmes are delivered across the network, and as such students who are part of the same cohort are not necessarily from the same campus. in such cases, teaching is delivered synchronously in the different campuses via videoconferencing or similar virtual classroom technology. data were collected via online survey. a total of 112 lecturers participated in the survey. of the respondents, 58 (52%) were teaching he only modules, while the rest (n = 54; 48%) had dual roles both as an fe and he lecturer. of the lecturers, 65 (59%) were from the arts, humanities and social sciences background, while the rest (n = 45; 41%) were from the science and engineering disciplines. sixty-one of the lecturers (55.5%) were employed full time, while the other lecturers were on part-time employment contracts (n = 49; 44.5%). instrument the survey was divided into three parts: (1) tpack inventory, (2) design of online learning and (3) relevant training. demographic information was also asked in the survey. the tpack inventory was adopted from archambault and barnett (2010) and uses the combined three factors rather than mishra and koehler’s (2006) original seven factors. pck combines the items from pk and ck into one subscale. tcck is the combination of the tck and tpk into one subscale. tk reflects all the items in the original archambault and crippen’s (2009) instrument. computed internal reliability cronbach’s alpha values for the three constructs were: 0.927 for pck, 0.936 for tpck and 0.910 for tk. sample tpack items and corresponding subscales are listed in table 1. lecturers rated their ability in doing the tasks listed in table 1 using a five-point likert scale, with a higher score reflecting higher skill. lecturers were also asked to identity strategies and the type of media they use in the vle. responses to these open-text questions were used to categorise the level of interactivity of their learning design. staff were also asked to identify the training they received in the last 12 months. finally, demographic information was sought, including subject area, subject-level taught (he, fe or both) and membership to professional bodies such as the higher education academy. the membership to higher education academy (now advanced he) was used as a proxy measure for identifying he teaching qualification. http://dx.doi.org/10.25304/rlt.v27.2296� research in learning technology citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 7 (page number not for citation purpose) procedure and data analysis ethical permission was sought prior to the start of the survey. consent was also sought from the developers of the tpack instrument. the survey was administered using an online survey and participants completed the survey within the first 6 weeks of the semester. tpack scores for each subscale were computed using the average score of the items corresponding to each subscale (table 1). the level of interactivity was analysed using the responses on the questions that asked what type of activities and strategies lecturers implement in their online/blended learning environments. conole’s (2007) taxonomy of e-learning tasks was used to categorise the level of interactivity into low or high. responses that identified assimilative and communicative tasks were categorised as low-level interactivity (e.g. use of word/pdf documents concurrent with the use of discussion board). responses that combine the use of assimilative, communicative and productive tasks (synchronous and asynchronous communication tools, interactive quizzes and online activities) were categorised as high-level interactivity. to analyse the relationship of the different tpack subscales, pearson’s correlation was used. relationship between tpack and level of interactivity was tested using kendall’s tau (table 4). multivariate analysis of variance (manova) was used to compare the lecturers’ tpack scores based on subject area, teaching qualification, nature of employment, recent training and the level of interactivity of the blended/online learning module (tables 2 and 3). table 1. descriptive statistics of tpack scores. tpack item and subscales mean sd pedagogical content knowledge (pck) 3.67 0.71 my ability to comfortably produce lesson plans with an appreciation for the topic. 4.02 0.87 my ability to use a variety of teaching strategies to relate various concepts to students. 3.66 0.86 my ability to determine a particular strategy best suited to teach a specific concept. 3.6 0.9 technological curricular-content knowledge (tcck) 3.11 0.84 my ability to implement the curriculum in an online environment. 3.43 1.05 my ability to implement different methods of teaching with technology. 3.22 1.01 my ability to create an online/blended learning environment which allows students to build new knowledge and skills. 3.21 1.09 technical knowledge (tk) 2.58 1.14 my ability to address various computer issues related to software (e.g. downloading appropriate plug-ins, installing programmes 2.71 1.28 my ability to assist students with troubleshooting technical problems with their personal computers. 2.56 1.2 my ability to troubleshoot technical problems associated with hardware (e.g. network connections) 2.46 1.22 http://dx.doi.org/10.25304/rlt.v27.2296� k. fabian et al. 8 citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 (page number not for citation purpose) results tpack scores descriptive statistics for the tpack subscales were computed by averaging the individual item scores for each subscale. lecturers’ scores on the tpack subscales of pck, tcck and tk were highest for pck [mean = 3.67; standard deviation (sd) = 0.71] followed by tcck (mean = 3.11; sd = 0.84) and lowest for the tk (mean = 2.58, sd = 1.14). the scores demonstrate that lecturers have a good to very good pck, good tcck and fair to good tk, based on a five-point likert scale. at item level, lecturers were mostly confident about their ability to produce lesson plans (mean = 4.02; sd = 0.87) but least confident on their skills to troubleshoot technical problems (mean = 2.46; sd = 1.22). table 1 shows the item and subscale-level descriptive statistics of the tpack scores. table 2. descriptive statistics of tpack scores between the two faculties. dependent variable faculty he teaching qualification mean se pck arts, humanities and business no 3.386 0.239 yes 3.903 0.122 science, health and engineering no 3.320 0.175 yes 4.220 0.200 tcck arts, humanities and business no 2.610 0.285 yes 3.304 0.145 science, health and engineering no 2.760 0.209 yes 3.700 0.239 tk arts, humanities and business no 2.190 0.416 yes 2.623 0.212 science, health and engineering no 2.295 0.305 yes 2.567 0.348 table 3. descriptive statistics and p-values by demographic variables. tpack subscales manova pck tcck tk f p nature of employment full time 3.88 (0.64) 3.28 (0.76) 2.52 (1.16) 7.619 0.000 part time 3.36 (0.64) 2.78 (0.75) 2.53 (1.05) training training 3.72 (0.65) 3.03 (0.81) 2.34 (1.16) 0.896 0.446 no training 3.62 (0.72) 3.09 (0.82) 2.63 (1.07) table 4. correlation between tpack variables and learning design. 1 2 3 4 1. pck 1.00 0.657** 0.167 0.210* 2. tcck 1 0.594** 0.314** 3. tk 1 0.211* 4. interactivity (low, high) 1 *p < 0.05, **p < 0.01. http://dx.doi.org/10.25304/rlt.v27.2296� research in learning technology citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 9 (page number not for citation purpose) a two-way manova was run between the faculty groupings and teaching qualification as the independent variables and pck, tcck and tk as dependent variables. the combined tpack subscales were used to assess lecturers’ tpack scores. assumptions for running manova were met. there was no statistically significant interaction effect between faculty grouping and teaching qualification on the combined dependent variables: f(3, 51) = 0.469, p = 0.705, wilks’ λ = 0.973, partial η2 = 0.027. there was a statistically significant main effect for he teaching qualification on the combined dependent variables: f(3, 51) = 5.975, p < 0.001, wilks' λ = 0.740, partial η2 = 0.260. follow-up univariate test was conducted, and the main effect of teaching qualification was considered. there was a statistically significant main effect of teaching qualification for pck, f(1, 53) = 14.097, p < 0.0005, partial η2 = 0.210, and for tcck, f(1, 53) = 13.136, p = 0.001, partial η2 = 0.199 but not for tk, f(1, 53) = 1.392, p = 0.289, partial η2 = 0.021. pck was 0.517 [95% confidence interval (ci), −0.021 to 1.054], higher for those with a teaching qualification in he in the faculty of arts, business and humanities lecturers, but this difference was not statistically significant, p = 0.896. for the lecturers in faculty of science, health and engineering, pck was higher by 0.900 (95% ci, 3.67 to 1.433), p = 0.001. tcck was 0.694 higher for those with teaching qualification in he for the lecturers in the faculty of arts, business and humanities, p = 0.035, and 0.940 higher for science, health and engineering lecturers, p = 0.005. table 2 shows the descriptive statistics of the tpack scores of the two faculties, further grouped by those with teaching qualification in he and those who do not. overall, these results suggest that subject area/faculty does not influence tpack scores but teaching qualification does, particularly for pk and ck. interestingly, both faculty and teaching qualification have little impact on tk, in contrast to cubeles and riu’s (2018) study which found tk scores varied by subject area. a likely reason for this relates to the university’s faculty grouping being only two, which means that the groupings were not sensitive enough to measure and compare distinct subject areas that may use technology differently (i.e. anatomy and literature). a comparison of tpack was also considered for attendance to training and by nature of employment (full time vs. part time). this was initially considered to be included in the manova above, but as the assumptions for adequate sample size and multi-collinearity were not met, multiple manova was run instead. there was no statistically significant difference between staff who attended training in the previous academic year and staff who did not on the tpack subscales, f(3, 107) = 0.896, p = 0.446 (see table 3). there was a statistically significant difference between full-time and part-time lecturing staff on combined tpack subscales, f(3, 105) = 7.619, p < 0.0005. this difference was statistically significant for the subscales pck (p < 0.0005) and tcck (p = 0.001) with lecturers in full-time employment having higher tpack scores for pck and tcck subscales (see table 3) but not for tk. tpack and learning design correlations between the different tpack subscales are shown in table 4. pck was correlated to all other variables except tk, ranging from low to strong correlation. tcck was correlated to all other variables except for the pedagogical-layer score, with a strong correlation with tk. tk was only significantly correlated to tcck and level of interactivity. table 4 further shows the correlation of the tpack scores http://dx.doi.org/10.25304/rlt.v27.2296� k. fabian et al. 10 citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 (page number not for citation purpose) to the level of interactivity provided in the course design. this variable was measured as a dichotomous variable derived from coding the responses to the open-ended question asking lecturers what strategies and resources they use in the design of their vle. the data suggest that these subscales are tapping into a similar overall ability of teaching proficiency, and that lecturers who have higher tpack scores tend to use more varied and interactive learning activities (such as the use of synchronous and asynchronous communication tools, in addition to active learning activities) in their design of the vle. to further explore the relationship of tpack with learning design, a manova was run on the combined tpack subscales. there was a significant difference in the combined tpack scores of staff who have higher levels of interactivity in their learning design versus staff who incorporate learning designs with low levels of interactivity, f(3, 84) = 4.511, p = 0.006 (see table 5). this difference was statistically significant for pck (p = 0.027), tcck (p = 0.036) and tk (p = 0.041), suggesting that those who employ more interactivity in their learning designs also have higher tpack. discussion technology adoption is facilitated by various factors such as lecturer’s ability, the demands of the subject to use technology and support. using tpack as a way to measure how lecturers assess their own ability to integrate technology into their practice, we found that lecturers in our study had high pck, but fair to average tk. this pattern is consistent with other tpack studies (cubeles and riu 2018; jang and tsai 2012). however, we found that pck and tcck vary between those with a teaching qualification in he and those who do not. a similar study that compared tpack scores of pre-service teachers and he lecturers attributed the higher tpack scores of pre-service teachers to the pedagogical training they received (jaikaran-doe and doe 2017). this links back to the importance of pck driving tpack development. effective teaching in online learning environments is characterised by studentcentred learning activities, interactivity and social connectivity (pelz 2003). in our study, we found that lecturers with higher tpack were able to promote these learning activities with more interactivity in their learning design. previous research on tpack noted that pedagogical skill is core in the development of tpack (benson and ward 2013). the characteristics of effective online learning environments highlighted by pelz illustrate the importance of putting pedagogy before technology. puentadura (2006) categorised the use of technology into two main themes: the use of technology to enhance learning activities, mostly to substitute or augment existing forms of practice, and the other is the transformative use of technology where technology allows task redesign or creating new task that was previously inconceivable. table 5. levels of interactivity and tpack subscales. level of interactivity tpack subscales manova pck tcck tk f p low 3.52 (0.63) 2.77 (0.60) 2.30 (1.00) 4.511 0.006 high 3.86 (0.70) 3.40 (0.85) 2.79 (1.11) http://dx.doi.org/10.25304/rlt.v27.2296� research in learning technology citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 11 (page number not for citation purpose) effective  learning design starts with the learning objective, followed by an exercise to match these strategies with a suitable technology. knowledge of the different capacities of technology and how it can be integrated into learning and teaching allow lecturers to realise design of learning activities that move technology use from enhancement to transformative. tpack did not significantly vary by attendance to training (with participants grouped into those with 1–3 training or those without). this shows that a few training sessions are not enough to raise tpack overall. studies that incorporated training to improve tpack have found significant increases in the lecturers’ preand postintervention tpack scores (brinkley-etzkorn 2018). it is worth noting that these studies incorporated a longer period of training over time. the training received by staff in the current study were mostly workshops that lasted only for a few hours and varied in terms of offerings (i.e. content-specific training or sessions on how to use a software), so this contrary finding is not surprising. therefore, while formal training seems to have an impact on tpack (particularly for pck and tcck), this benefit does not translate from shorter, perhaps more informal courses. rienties et al. (2013) noted that an integrated approach to training is needed to facilitate improvement. we found that the tpack of full-time lecturers was significantly higher than that of part-time lecturers. a possible reason is the opportunities and support made available to part-time staff. for example, it might be that some professional development days are scheduled outside working hours of part-time lecturers, and so those in parttime employment tend to miss these opportunities. the difference in tpack scores of part-time and full-time staff suggests a need to revisit the support offered to part-time staff in terms of training and development and this is supported by nelson et al.’s (2010) study that found that perceived institutional support contributes to tpack. knowledge of the different system functionalities of the vle allows lecturers to adopt a wider range of tools for the learning environment. anderson et al.’s (2013) study discussed the importance of tk in facilitating the transition from traditional face-to-face teaching to the online environment. in assessing the relationship between the different tpack subscales to the level of interactivity used in online learning environments, we found a correlation between tpack and level of interactivity, for all three subscales. a higher level of interactivity in the design of learning materials was implemented by staff with higher tpack and vice versa. this reiterates the need to provide sustained developmental support to lecturing staff to facilitate design of interactive learning activities. conclusion this study is a positive response to the gap in tpack studies at he level. our study set out to identify the skills of lecturers in integrating technology into their learning and teaching as measured by the tpack instrument. we found that those with higher tpack scores tend to apply more interactive learning designs into their modules. we investigated how tpack varies by subject area, employment status, training attendance and teaching qualification and found nature of employment and teaching qualification in he to be moderating factors. several limitations can be found in this study. the dual nature of the institution means that we have surveyed lecturers’ teaching in both he and fe courses, which is likely to have a confounding effect in how lecturers embed technology into their practice. our use of hea status as a http://dx.doi.org/10.25304/rlt.v27.2296� k. fabian et al. 12 citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 (page number not for citation purpose) proxy measure to teaching qualification in he is another limitation as there would be staff who obtained their he teaching qualification via other routes. we have found teaching qualification and employment status to influence tpack scores, although it may be worth exploring the nature of training and experience further to ensure that these results are reliable. these two variables are indirectly related to years of teaching experience which we have not accounted for in our design, so it would be worthwhile looking into this further. in our study, we found that tk scores of lecturers are lower overall, but their pck and tpk scores were higher. this was regardless of the demographic factors investigated. it is thus important to offer technology training that will help align their tk scores with pck and tcck, building on what they know and are confident with. for example, aligning technology training with a specific teaching strategy, providing context and examples of how technology can be used for their respective subject areas. support to align tpack of part-time lecturers and those who transition into teaching is also needed to bridge the skills gap. knowing the tpack profile of lecturers can facilitate the design of training that would support the development of staff competencies. references anderson, a., barham, n. & northcote, m. (2013) ‘using the tpack framework to unite disciplines in online learning’, australasian journal of educational technology, vol. 29, no. 4, pp. 549–565. doi: 10.14742/ajet.24 archambault, l. & barnett, j. (2010) ‘revisiting technological pedagogical content knowledge: exploring the tpack framework’, computers & education, vol. 55 no. 4, pp. 1656–1662. doi: 10.1016/j.compedu.2010.07.009 archambault, l. & crippen, k. (2009) ‘examining tpack among k-12 online distance educators in the united states’, contemporary issues in technology and teacher education, vol. 9, no. 1, pp. 71–88. beaton, f. & gilbert, a. (2013) developing effective part-time teachers in higher education: new approaches to professional development, routledge, oxford. benson, s. & ward, c. (2013) ‘teaching with technology: using tpack to understand teaching expertise in online higher education’, journal of educational computing research, vol. 48, no. 2, pp. 153–172. doi: 10.2190/ec.48.2.c brinkley-etzkorn, k. (2018) ‘learning to teach online: measuring the influence of faculty development training on teaching effectiveness through a tpack lens’, internet and higher education, vol. 38, pp. 28–35. doi: 10.1016/j.iheduc.2018.04.004 cheng, s. l. & xie, k. (2018) ‘the relations among teacher value beliefs, personal characteristics, and tpack in intervention and non-intervention settings’, teaching and teacher education, vol. 74, pp. 98–113. doi: 10.1016/j.tate.2018.04.014 chukwuemeka, e. & iscioglu, e. (2017) ‘an examination of lecturers’ technological pedagogical content knowledge: perceptions at the faculty of education in emu in cyprus’, croatian journal of education, vol. 18, no. 4, pp. 999–1034. doi: 10.15516/cje.v18i4.1845 conole, g. h. (2007) ‘describing learning activities: tools and resources to guide practice’, in  rethinking pedagogy for a digital age: designing and delivering e-learning, eds beetham & r. sharpe, routledge, london, pp. 81–91. doi: 10.4324/9780203961681 corry, m. & stella, j. (2018) ‘teacher self-efficacy in online education: a review of the literature’, research in learning technology, vol. 26, pp. 1–12. doi: 10.25304/rlt.v26.2047 cubeles, a. & riu, d. (2018) ‘the effective integration of icts in universities: the role of knowledge and academic experience of professors’, technology, pedagogy and education, vol. 27, no. 3, pp. 339–349. doi: 10.1080/1475939x.2018.1457978 http://dx.doi.org/10.25304/rlt.v27.2296� https://doi.org/10.14742/ajet.24 http://dx.doi.org/10.1016/j.compedu.2010.07.009� http://dx.doi.org/10.2190/ec.48.2.c� http://dx.doi.org/10.1016/j.iheduc.2018.04.004� http://dx.doi.org/10.1016/j.tate.2018.04.014� http://dx.doi.org/10.15516/cje.v18i4.1845� http://dx.doi.org/10.25304/rlt.v26.2047� http://dx.doi.org/10.1080/1475939x.2018.1457978� research in learning technology citation: research in learning technology 2019, 27: 2296 10.25304/rlt.v27.2296 13 (page number not for citation purpose) foulger, t., graziano, k. & schmidt-crawford, d. (2017) ‘educator technology competencies’, journal of technology and teacher education, vol. 25 no. 4, pp. 413–448. jaikaran-doe, s. & doe, p. (2017) ‘assessing technological pedagogical content knowledge of engineering academics in an australian regional university academics in an australian regional university’, australasian journal of engineering education, vol. 20, no. 2, pp. 157–167. doi: 10.1080/22054952.2015.1133515 jang, s. j. & tsai, m. f. (2012) ‘exploring the tpack of taiwanese elementary mathematics and science teachers with respect to use of interactive whiteboards’, computers & education, vol. 59, no. 2, pp. 327–338. doi: 10.1016/j.compedu.2012.02.003 jisc. (2017) ‘embedding blended learning in further education and skills’, [online] available at: https://www.jisc.ac.uk/guides/embedding-blended-learning-in-further-education-and-skills. kay, r. h. (2006) ‘evaluating strategies used to incorporate technology into preservice education: a review of the literature’, journal of research on technology in education, vol. 38, no. 4, pp. 385–410. doi: 10.1080/15391523.2006.10782466 koh, j., chai, c. & tsai, c. (2014) ‘demographic factors, tpack constructs, and teachers’ perceptions of constructivist-oriented tpack’, educational technology & society, vol. 17, no. 1, pp. 185–196. lee, m. h. & tsai, c. c. (2010) ‘exploring teachers’ perceived self efficacy and technological pedagogical content knowledge with respect to educational use of the world wide web’, instructional science, vol. 38, no. 1, pp. 1–21. doi: 10.1007/s11251-008-9075-4 marcelo, c. & yot-domínguez, c. (2018) ‘from chalk to keyboard in higher education classrooms: changes and coherence when integrating technological knowledge into pedagogical content knowledge’, journal of further and higher education, vol. 43, no. 7, pp. 975–988. doi: 10.1080/0309877x.2018.1429584 meadows, j. & henry, j. (2008) ‘an absolutely riveting online course: nine principles for excellence in web-based teaching’, canadian journal of learning and technology, vol. 34, pp. 75–90. doi: 10.21432/t20c7f mishra, p. & koehler, m. j. (2006) ‘technological pedagogical content knowledge: a framework for teacher knowledge’, teachers college record, vol. 108, no. 6, pp. 1017–1054. doi: 10.1111/j.1467-9620.2006.00684.x nelson, m. j., voithofer, r. & cheng, s. l. (2019) ‘mediating factors that influence the technology integration practices of teacher educators’, computers & education, vol. 128, pp. 330–344. doi: 10.1016/j.compedu.2018.09.023 pelz, b. (2003) ‘(my) three principles of effective online pedagogy’, journal of asynchronous learning networks, vol. 14, no. 1, pp. 103–116. puentadura, r. (2006) ‘transformation, technology and education’, [online] available at: http:// hippasus.com/resources/tte/ reyes, v. c., et al., (2017) ‘integrating ict into teacher education programs from a tpack perspective: exploring perceptions of university lecturers’, computers & education, vol. 115, pp. 1–19. doi: 10.1016/j.compedu.2017.07.009 rienties, b., et al., (2013) ‘online training of tpack skills of higher education scholars: a cross-institutional impact study’, european journal of teacher education, vol. 36, no. 4, pp. 480–495. doi: 10.1080/02619768.2013.801073 schmidt, d., et al., (2009) ‘technological pedagogical content knowledge (tpack): the development and validation of an assessment instrument for preservice teachers’, journal of research on technology in education, vol. 42, no. 2, pp. 123–149. doi: 10.1080/15391523.2009.10782544 shulman, l. (1986) ‘those who understand: knowledge growth in teaching’, educational researcher, vol. 15, no. 2, pp. 4–14. doi: 10.3102/0013189x015002004 voithofer, r., et al., (2019) ‘factors that influence tpack adoption by teacher educators in the us’, educational technology research and development. doi: 10.1007/s11423-019-09652-9 http://dx.doi.org/10.25304/rlt.v27.2296� http://dx.doi.org/10.1080/22054952.2015.1133515� http://dx.doi.org/10.1016/j.compedu.2012.02.003� https://www.jisc.ac.uk/guides/embedding-blended-learning-in-further-education-and-skills� https://doi.org/10.1080/15391523.2006.10782466 http://dx.doi.org/10.1007/s11251-008-9075-4� https://doi.org/10.1080/0309877x.2018.1429584 https://doi.org/10.21432/t20c7f http://dx.doi.org/10.1111/j.1467-9620.2006.00684.x� http://dx.doi.org/10.1016/j.compedu.2018.09.023� http://hippasus.com/resources/tte/ http://hippasus.com/resources/tte/ http://dx.doi.org/10.1016/j.compedu.2017.07.009� http://dx.doi.org/10.1080/02619768.2013.801073� http://dx.doi.org/10.1080/15391523.2009.10782544� http://dx.doi.org/10.3102/0013189x015002004� http://dx.doi.org/10.1007/s11423-019-09652-9� doi:10.3402/rlt.v19s1/7816 editorial for proceedings papers here are the proceedings of the 2011 alt conference ‘‘thriving in a colder and more challenging climate’’. proceedings papers report on a piece of research, possibly in its early stages, or they can be ‘‘thoughtpieces’’ which state a point of view or summarise an area of work, perhaps giving new insights. the conference has six themes: . research and rigour: creating, marshalling and making effective use of evidence . making things happen: systematic design, planning and implementation . broad tents and strange bedfellows: collaborating, scavenging and sharing to increase value . at the sharp end: enabling organisations and their managers to solve business, pedagogic and technical challenges . teachers of the future: understanding and influencing the future role and practices of teachers . preparing for a thaw: looking ahead to a time beyond the disruptive discontinuities of the next few years. interestingly, there were very few proposals for the conference as a whole against the sixth theme: and no proceedings papers. perhaps the thaw is still perceived as being too far away to warrant any preparation yet! unsurprisingly, research and rigour is well represented in the proceedings but there is a problem-solving practical flavour relevant to the title of the conference. garnett and ecclesfield (2011) look at boyer’s often cited model of scholarship and update it for the current more open and networked environment. jones and kennedy (2011) argue for a more problem-driven approach to social science and specifically educational research using a pluralist approach combining quantitative and qualitative methodologies and tools. bain (2011) revisits the role of online discussion in learning for all learners and produces a framework. kerrigan et al. (2011) report on the successful use of a tool to support and indeed require student reflection on the feedback they have been given. it is now in use in several universities and colleges. these four general papers are balanced by two that are specifically subject-based. abadi and alsop (2011) use a formal combination of activity theory and grounded theory in a practical way to improve learning of initial java programming, alt-c 2011 conference proceedings isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # d. hawkridge et al. this is an open access article distributed 1 under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7816 a high-volume area with considerable payoff. bacon, windall, and mackinnon (2011) discuss using emotional effects in the multimedia training of executives responsible for safety critical services such as fire or police, a lower volume area but again with high payoff in the light of the important role for which training is being given, and the expense of traditional methods. making things happen through problem solving with significant actual or potential payoff is also important, especially in the current environment. collins et al. (2011) report on the successful use of short ‘‘photomarathons’’ in engaging primary school children in learning on a visit to an historical site. kohl (2011) discusses the deployment of a voluntary plagiarism checking system which addresses this serious problem through prevention rather than post hoc detection. smith (2011) reports on the use of a ‘‘virtual internet’’ to solve some of the security problems of group work in learning computer networking concepts and techniques. gorjian (2011) describes and discusses a very carefully designed experiment that suggests that hypertext annotations in second language teaching, while improving vocabulary learning, so do primarily in the short term only. this result is sufficiently disturbing to generate an interesting discussion of the context of the experiment and why the result might not generalise. several papers involve collaborations but only tan and pearce (2011) major in scavenging in their use of youtube videos in teaching sociology. their discussion of the barriers to and advantages of such use has wide applicability. at the sharp end, strachan et al. (2011) discuss the results of a questionnaire that looked at workplace-based learning from the points of view of not just learners and teachers but also from those of the workplace and external stakeholders. the results are of wide relevance in the current environment. stewart, mckee, and porteous (2011) discuss an integrated system for lecture capture that does not require heavy hardware or software and can be accessed from simple hand held devices. finally, what of the teachers of the future? two papers offer very different approaches to teacher professional development. cochrane and naryan (2011) describe a new approach at a single site, for lecturers through a ‘‘social learning technologies’’ course which aims to produce a community of practice in the use of mobile web2.0 technologies. in contrast, arati, todorova, and merrett (2011) are looking to bring together the strange if not completely incompatible, bedfellows of government in germany, france and the uk, in public/private partnerships with intel. the project supports the professional development of teachers to help them use technology more in their teaching and to enable them to facilitate the acquisition of twenty-first century skills by their students. we thus see the effects of the cold. to thrive in the current climate, research needs to be more focused than in the past on solving problems, preferably in a scalable way: that is where the payoff lies. nevertheless, the papers remain solidly grounded in alt-c 2011 conference proceedings 2 theory, properly constructed and well argued. alt’s field overall has an important role to play in ‘‘bringing on the thaw’’. david hawkridge kia ng steven verjans references abadi, m.k., and g. alsop. 2011. studying the learning of programming using grounded theory to support activity theory. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7801 arati, d., a. todorova, and r. merrett. 2011. implementation and sustainability of a global ict company’s programme to help teachers integrate technology into learning and teaching in germany, france and the uk. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7802 bacon, l., g. windall, and l. mackinnon. 2011. the development of a rich multimedia training environment for crisis management: using emotional affect to enhance learning. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7780 bain, y.c. 2011. learning through online discussion: a framework evidenced in learners’ interactions. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8. in september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7779 cochrane, t., and v. narayan. 2011. defrosting professional development: reconceptualising teaching using social learning technologies. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7796 collins, t., r. joiner, a. white, and l. braidley. 2011. school trip photomarathons: engaging primary school visitors using a topic focused photo competition. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7797 garnett, f., and n. ecclesfield. 2011. towards a framework for co-creating open scholarship. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7795 gorjian, b. 2011. using hypermedia annotations to teach vocabulary on the web. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7799 jones, c., and g. kennedy. 2011. stepping beyond the paradigm wars: pluralist methods for research in learning technology. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7798 kerrigan, m.j.p., s. walker, m. gamble, s. lindsay, k. reader, m. papaefthimiou, l. newman-ford, m. clements, and g. saunders. 2001. the making assessment count (mac) consortium � maximizing assessment and feedback design by working together. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7782 kohl, k.e. 2011. fostering academic competence or putting students under general suspicion? voluntary plagiarism check of academic papers by means of a web-based plagiarism detection system. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7611 smith, a. 2011. development of a simulated internet for education. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7777 alt-c 2011 conference proceedings 3 stewart, i., w. mckee, and k. porteous. 2011. towards a personalised learning mesh: the implementation of a low overhead, multipath learning tool. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7815 strachan, r., l. liyanage, b. casselden, and r. penlington. 2011. effectiveness of technology to support work based learning: the stakeholders’ perspective. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7800 tan, e., and n. pearce. 2011. open education videos in the classroom: exploring the opportunities and barriers to the use of youtube in teaching introductory sociology. paper presented at the 18th association for learning technology conference (alt-c 2011), september 6�8, at the university of leeds, uk. doi: 10.3402/rlt.v19s1/7783 alt-c 2011 conference proceedings 4 (page number not for citation purpose) 1 *email: ekrem.bahcekapili@ktu.edu.tr research in learning technology 2023. © 2023 e. bahçekapılı. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 research in learning technology vol. 31, 2023 original research article predicting the secondary school students’ intention to use e-learning technologies ekrem bahçekapılı* department of management information systems, karadeniz technical university, trabzon, türkiye (received: 26 august 2022; revised: 24 october 2022; accepted: 11 november 2022; published: 04 january 2023) technology acceptance studies are interesting because they are practical and theoretically helpful in explaining the adoption and intention to use a particular technology. there is a large amount of research on e-learning and other technologies in the literature, but there is limited evidence to explain why secondary school students’ intention to use e-learning. this study explains secondary school students’ intentions to use e-learning with an extended technology acceptance model (tam). tam is a useful theory to explain how people adopt new technologies in different fields. data were collected from 2739 secondary school students in turkey (mage = 11.95). confirmatory factor analysis (cfa) and structural equation modelling (sem) were used to test the conceptual model. the results are consistent with the original tam model. the most critical variable affecting secondary school students’ intention to use e-learning technologies is enjoyment. the results show that there may be differences in the intention to use e-learning technologies for secondary school students in different cultures and contexts. keywords: e-learning; tam; technology acceptance; perceived enjoyment; secondary school students; k–12 introduction with the covid-19 pandemic, popularity of e-learning has increased even more. because face-to-face education, which was interrupted during the pandemic, was supported by online learning methods in higher education (almaiah, al-khasawneh, and althunibat 2020) and k–12 education (asvial, mayangsari, and yudistriansyah 2021). when e-learning research trends are examined, it can be said that most of the studies have been carried out at the higher education level, and studies at the k–12 level are limited (arnesen et al. 2019). the increasing use of e-learning is emerging as a new concept, sometimes as technology and sometimes as a method, especially for younger students (bhaumik and priyadarshini 2020). for this reason, it is crucial to understand how students in the younger age group perceive e-learning and what factors affect their adoption of e-learning. in this way, better e-learning environments and applications can be designed. mailto:ekrem.bahcekapili@ktu.edu.tr http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2881 e. bahçekapılı 2 citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 (page number not for citation purpose) this study aims to understand secondary school students’ intentions to use e-learning in turkey. for this purpose, an extended theoretical model based on tam is proposed. the tam argues that user acceptance of technology may occur depending on the variables of perceived usefulness (pu) and perceived ease of use (peu) (davis 1989). tam is a powerful theory that contributes to understanding the factors that affect individuals’ adoption and acceptance of new technologies in different fields (marangunić and granić 2015). also, many studies based on tam have been carried out to understand the factors affecting the adoption of e-learning-related technologies by students (abdullah and ward 2016; nguyen et al. 2020). these studies have shown that tam is also a viable and valid model for e-learning applications. when the e-learning and tam-based researches are examined, although much research has been carried out at the higher education level, studies at the k–12 level are pretty limited (cheng and yuen 2018, 2019; teo et al. 2019; zhao et al. 2021). on the other hand, studies based on tam emphasise that cultural differences can affect the acceptance and adoption of e-learning (zhao et al. 2021). no study up to date has been found that deals with the variables that affect the adoption of e-learning by students studying at the k–12 level in turkey within the framework of tam. online learning applications to support face-to-face teaching at the k–12 level have been used for a long time in turkey. examples of these are the educational information network (eba) (aktay and keskin 2016) and tv broadcasts (bozkurt 2017) created by the ministry of national education. however, there is no regular distance education practice in turkey, where students carry out their main education processes online until the pandemic (fiş erümit 2021). with the covid-19 pandemic, students continued their interrupted teaching processes with online live lessons and previously available e-learning resources (can 2020). therefore, it can be said that students at the k–12 level in turkey have encountered e-learning applications in real terms during the pandemic period. in recent years, the increasing popularity of e-learning applications and worldwide health concerns have increased the importance of the factors that will impact the use of these technologies in secondary schools. however, knowing little about the factors affecting k–12 students’ intention to use e-learning creates a significant gap, both theoretically and practically. this research is significant as it is the first study to determine the intention of secondary school students in turkey to use e-learning technologies. in addition, considering the limited number of studies in the literature explaining the intention of students at this level to use e-learning technologies, it is thought to contribute to the literature. research model and hypotheses in this section, the research model and hypothesis are explained. the basis of this study is tam, and the model has been expanded with two external (perceived enjoyment and computer self efficacy) variables to explain students’ adoption of e-learning. figure 1 presents the research model used in this study. in the study, 7 hypotheses were tested to explain the secondary school student’s behavioural intention (bi) from using the e-learning system in turkey. technology acceptance model technology acceptance model (tam) initially developed by fred davis (1989). tam is the most known model for user adoption and acceptance of information http://dx.doi.org/10.25304/rlt.v31.2881 research in learning technology citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 3 (page number not for citation purpose) technologies (davis and venkatesh 1996). tam emphasises two key individual beliefs to understand a user’s bi to use technology. these are peu and pu. pu is defined as ‘the prospective user’s subjective probability that using a specific application system will increase his or her job performance’ and peu refers to ‘the degree to which the prospective user expects the target system to be free of effort’ (davis 1989). bi is expressed as an important indicator of an individual’s exhibiting a given behaviour (ajzen & fishbein 1980). attitude construct included in original tam excluded from the final tam model. because empirical evidence showed that attitude did not fully mediate the effect of peu on bi (davis and venkatesh 1996). tam is a valid and useful model for various information technologies, work environments and individuals (legris, ingham, and collerette 2003). similarly, tam reveals that it is an effective model in explaining students’ acceptance and adoption of e-learning in different educational contexts (cheng 2011; purnomo and lee 2013; salloum et al. 2019). previous studies show that, as indicated in the main tam model, pu affects bi positively (cheng and yuen 2018, 2019; teo et al. 2019; zhao et al. 2021), peu affects bi positively (cheng and yuen 2018, 2019) and peu also affects pu positively (cheng and yuen 2018, 2019; teo et al. 2019; zhao et al. 2021). based on tam’s theory, the following research hypotheses included: h1: perceived usefulness (pu) is positively related to one’s intention to use (bi) e-learning h2: perceived ease of use (peu) is positively related to the pu of e-learning h3: perceived ease of use (peu) positively affects one’s intention to use (bi) e-learning perceived enjoyment perceived enjoyment (enj) is an indicator of intrinsic motivation and is the degree to which the user’s computer use is perceived as enjoyable (davis, bagozzi, and warshaw 1992). intrinsic motivation means that an individual does an activity for its own sake because she/he finds it interesting or satisfying for her/him (lee, cheung, and chen 2005). prior studies have found enj has a direct influence on individual’s peu (cicha figure 1. research model. http://dx.doi.org/10.25304/rlt.v31.2881 e. bahçekapılı 4 citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 (page number not for citation purpose) et al. 2021; findik-coşkunçay, alkiş, and özkan-yildirim 2018; rizun and strzelecki 2020), pus (abdullah and ward 2016; chang, hajiyev, and su 2017; teo et al. 2019) and bi (findik-coşkunçay, alkiş, and özkan-yildirim 2018; teo et al. 2019; vladova et al. 2021). thus, it is hypothesised that: h4: enjoyment (enj) is positively related to one’s intention to use (bi) e-learning h5: enjoyment (enj) is positively related to the pu of e-learning h6: enjoyment (enj) is positively related to the peu of e-learning computer self-efficacy self-efficacy is the belief that an individual can do a job that has an impact on his/her life at the specified level of success (gallagher 2012). efficacy beliefs affect people’s feelings, thoughts, self-motivation and behaviours (bandura 1993). computer self-efficacy (cse) also refers to an individuals’ beliefs about their abilities to efficiently use computer technologies (compeau and higgins 1995). studies in the literature generally emphasise that cse has a positive direct and significant effect on peu (kanwal and rehman 2017; salloum et al. 2019; teo et al. 2019). although there are studies reporting a significant positive effect of cse on pu (abdullah and ward 2016; cicha et al. 2021; rizun and strzelecki 2020), this effect was generally insignificant (chang, hajiyev, and su 2017; kanwal and rehman 2017; salloum et al. 2019). therefore, in this study, it is hypothesised as follows: h7: computer self-efficacy (cse) is positively related to the peu of e-learning method participants and data collection convenience sampling method was used as the sampling method due to the ease of access to the participants due to the pandemic conditions. with the convenience sampling method, data can be obtained easily and quickly from appropriate data sources (lavrakas 2008). before the data collection process, necessary permissions were obtained from the national education directorate and the university’s ethics committee. the participant population of the research consists of secondary school students studying in the 2019–2020 academic year. necessary information was given to the families of the students through the directorate of national education, and the questionnaire was delivered in the online form. the data were collected anonymously. a total of 2890 students responded voluntarily to the study. in preliminary analysis, all of the answers with missing data, the participants who entered the same answer to all the questions, the records where the same answer was entered consecutively were excluded. a total of 2739 (female = 56.7% and male = 43.3%) valid responses were collected and adopted for further data analysis. the ages of the students range from 9 to 15, with a mean age of 11.95 (standard deviation [sd] = 1.2). students’ e-learning experiences the distance education process was carried out in a similar way throughout the country. this process is carried out by the ministry of national education and its affiliated http://dx.doi.org/10.25304/rlt.v31.2881 research in learning technology citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 5 (page number not for citation purpose) table 1. instrument and resources. constructs items measures resources behavioural intention to use (bi) bi1 i intent to frequently use of the e-learning system now. (salloum et al. 2019; venkatesh and bala 2008) bi2 i will recommend using an e-learning system in the future. bi3 i will use an e-learning system in the future. perceived ease of use (peu) peu1 it is easy for me to use the e-learning system. (teo et al. 2019; venkatesh and bala 2008) peu2 it is easy for me to attend the e-learning courses. peu3 i find the e-learning system to be easy to use. peu4 using the e-learning system does not require a lot of my effort. perceived usefulness (pu) pu1 using the e-learning system improves my learning performance. (teo et al. 2019; venkatesh and bala 2008)pu2 using the e-learning system makes easier my learning. pu3 using the e-learning system enhances my effectiveness pu4 i find using the e-learning system to be useful. perceived enjoyment (enj) enj1 i find using the e-learning system to be enjoyable. (teo et al. 2019; venkatesh and bala 2008) enj2 the e-learning system is a pleasant environment. enj3 using the e-learning system increases my curiosity. enj4 i like the lessons in the e-learning system. computer self-efficacy (cse) cse1 i could use the e-learning system even if there was no one around to tell me what to do. (salloum et al. 2019; venkatesh and bala 2008)cse2 i have sufficient skills to attend the e-learning courses. cse3 i could use the features of the e-learning system by on my own. units. students experienced e-learning for the first time with the interruption of formal face-to-face education during the covid-19 pandemic. e-learning activities continued in the form of synchronous online learning. weekly and daily lesson plans were created for online education like face-to-face education. in this way, they have been provided to learn various lessons online live at different times of the day. synchronous lessons were carried out on the eba live course application (atmaca 2021), which is similar to zoom application. instrument the questionnaire for the study distributed to respondents in turkish language and consisted of two parts. the first part contains information about demographic variables – gender and age. the part two contains 18 survey items to test the research hypotheses. the items were developed by adapting from the existing literature to make them consistent with the requirements of the current study. in order to make sure that the students understood the adapted questionnaire items correctly, each question was evaluated together with a questioner and a student by the researcher, and necessary adjustments were made. the items and references were given in table 1. data were analysed by exploratory factor analysis and confirmatory factor analysis (cfa). the final version of the instrument comprised 18 items using a five-point likert scale (1 = strongly disagree to 5 = strongly agree) for five constructs. http://dx.doi.org/10.25304/rlt.v31.2881 e. bahçekapılı 6 citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 (page number not for citation purpose) data analysis the data were analysed using microsoft excel, spss 22 and amos 22. to examine the hypotheses stated in the research model, a two-stage approach proposed by anderson and gerbin (1988) was adopted in this study. accordingly, first, the measurement model was tested, and then the structural model was analysed. results descriptive statistics the descriptive statistics of each construct are given in table 2. all means of constructs are above 3.00, and the sds range from 0.91 to 1.32. the kurtosis (from −1.164 to 3.359) and skewness (from −1.915 to −0.156) values are within the acceptable limits suggested by kline (kline and rex 2011). testing the measurement model cfa with maximum likelihood estimation was used to evaluate the measurement model. for assessing multivariate normality, the mardia coefficient was calculated. the value of 132  244 obtained in this study was lower than the suggested value n(n + 2) (where n is the total observed number of items) (raykov and marcoulides 2008). this value shows that the multivariate normal distribution criterion was obtained for cfa. to examine the reliability and validity of the model; construct reliability, and convergent and discriminant validity were used. construct reliability was tested using cronbach’s α (>0.7), and composite reliability (cr > 0.7) and the internal consistency of the items in each construct were assumed valid (fornell and larcker 1981). teo and noyes (2014) stated that average variance extracted (ave) and cr values of 0.50 or above can demonstrate the acceptable reliability of the measurement model. also, the standardised estimate (se) of each item was tested. it determines whether an item contributes significantly to explaining its underlying construct. the values calculated for each item were found to be higher than 0.50. this value shows that the contribution is sufficient (hara, takemura, and yoshida 2010). table 3 lists the factor loadings of the items – cr, ave and cronbach’s α for various constructs. to evaluate the discriminant validity, first, the square root of the ave of each construct is calculated. if these values are greater than the correlation coefficients between the construct and other constructs, it is accepted that discriminant validity table 2. descriptive statistics of the study constructs. construct item mean sd skewness kurtosis bi 3 3.27 1.27 −0.292 −1.061 pu 4 3.13 1.30 −0.156 −1.161 peu 4 3.87 1.08 −0.871 −0.065 enj 4 3.15 1.32 −0.187 −1.164 cse 3 4.42 0.91 −1.915 3.359 note: sd = standard deviation; bi = behavioural intention to use; cse = computer self-efficacy; enj = perceived enjoyment; peu = perceived ease of use; pu = perceived usefulness. http://dx.doi.org/10.25304/rlt.v31.2881 research in learning technology citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 7 (page number not for citation purpose) is provided (fornell and larcker 1981). as shown in table 4, the value calculated for each construct in this study meets the relevant criteria. to analyse the fit of the measurement model, several fit indices were used: the chisquare test (χ2) with degrees of freedom (df), comparative fit index (cfi), tucker-lewis index (tli), root mean square residual (srmr) and the root mean square error of approximation (rmsea). according to the limits specified by hair et al. (hair, page, and brunsveld 2019), the model fit indices used for the measurement model, the ratio of the minimum fit function (χ2) and the minimum fit function to χ2 degrees of freedom (χ2/df) are less than 5, the cfi and tli, with values greater than 90, which is acceptable. in addition, it is desirable that the srmr and rmsea values are less than 0.08. based on these thresholds, the cfa results confirmed that the measurement model has an acceptable fit to the sample data (χ2 = 559.366, df = 119, χ2/df = 4.701, cfi = 0.986, tli = 0.982, srmr = 0.0308, rmsea = 0.037 [0.034, 0.040]). table 3. analysis summary of measurement model. variables item se use t-value cr ave cronbach’s α enj enj1 0.826 1   0.893 0.675 0.900 enj2 0.838 1.007 62.074     enj3 0.818 0.991 48.328     enj4 0.804 0.955 47.276 pu pu1 0.835 1 0.941 0.800 0.916 pu2 0.900 1.084 65.847     pu3 0.887 1.051 51.111     pu4 0.952 1.16 44.825     peu peu1 0.827 1 0.841 0.576 0.836 peu2 0.840 0.998 47.675     peu3 0.792 0.885 44.612     peu4 0.535 0.711 27.478     bi bi1 0.739 1 0.821 0.606 0.823 bi2 0.812 1.146 39.701     bi3 0.782 1.043 38.458     cse cse1 0.733 1 0.797 0.567 0.790 cse2 0.793 0.891 34.168     cse3 0.731 0.915 32.856     note: bi = behavioural intention to use; cse = computer self-efficacy; enj = perceived enjoyment; peu = perceived ease of use; pu = perceived usefulness; use = unstandardised estimates; se = standardised estimates; cr = composite reliability; ave = average variance extracted. table 4. square root of ave and correlations between the constructs. bi enj pu peu cse bi 0.778         enj 0.771 0.822       pu 0.737 0.786 0.894     peu 0.560 0.550 0.489 0.759   cse 0.372 0.367 0.293 0.592 0.753 note: bi = behavioural intention to use; cse = computer self-efficacy; enj = perceived enjoyment; peu = perceived ease of use; pu = perceived usefulness. all correlations are significant at the 0.001 level. values in bold are the constructs’ square root of the aves. http://dx.doi.org/10.25304/rlt.v31.2881 e. bahçekapılı 8 citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 (page number not for citation purpose) test of the structural model the structural model results are shown in figure 2, together with the path coefficients (i.e. standardised beta coefficients) and their significance. the structure model received an acceptable fit (χ2 = 567.582, df = 121, χ2/df = 4.691, cfi = 0.986, tli = 0.982, srmr = 0.0312, rmsea = 0.037 [0.034, 0.040]). all the hypothesised relationships were supported (see table 5). for the three endogenous constructs in this study, pu was significantly influenced by peu and enj, and they explained 62% of the variance. peu was significantly influenced by enj and cse, with 48% of variance explained. finally, 66% of the total variance in bi was explained by pu, pe and enj. discussion and conclusion e-learning applications are widely used, especially in higher education. e-learning applications have played a critical role in the sustainability of teaching, especially table 5. path coefficients and hypothesis results. path path coefficient (β) p support h1. pu→bi 0.318 *** yes h2. peu→pu 0.077 *** yes h3. peu→bi 0.171 *** yes h4. enj→bi 0.428 *** yes h5. enj→pu 0.744 *** yes h6. enj→peu 0.387 *** yes h7. cse→peu 0.452 *** yes note: bi = behavioural intention to use; cse = computer self-efficacy; enj = perceived enjoyment; peu = perceived ease of use; pu = perceived usefulness. ***p < 0.001. figure 2. the structural model results. http://dx.doi.org/10.25304/rlt.v31.2881 research in learning technology citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 9 (page number not for citation purpose) during the pandemic period. there is a lot of work to be done to understand the adoption and intentions to use e-learning applications in higher education. however, little is known about the factors that influence k–12 students’ intentions to use e-learning. this creates a crucial knowledge gap. understanding how k–12 students intend to use e-learning will help fill in the gaps in research and lead to better ways of teaching. in this study, a tam-based model is proposed to fill this gap. this study expanded tam with the exogenous variables of enjoyment and cse and investigated the factors that affect the intention to use e-learning technologies of secondary school students in turkey. all hypotheses of the research were supported. the research results showed that tam is a powerful model in predicting secondary school students’ intention to use e-learning technologies in turkey, as revealed in previous studies (cheng and yuen 2018; kuliya and usman 2021). together with the exogenous variables, the model explained 66% of secondary school students’ intention to use e-learning technologies. students’ intentions (bi) are most strongly influenced by enjoyment (β = 0.428, p < 0.001), compared to pu (β = 0.318, p < 0.001) and peu (β = 0.171, p < 0.001). when students perceive e-learning technologies as enjoyable, this situation increases their intention to use these technologies more than peu and pu. similarly, in their study with nepali students, teo et al. (2019) stated that enj is the variable that most affects students’ intention to use it. on the other hand, in the study of hwang et al. (2021), it was reported that the relationship between enj and the intention to use computer-based educational technologies of malaysian students was not significant. compared to similar studies in higher education, the enj→bi relationship is consistent with this study and teo et al. (2019). this situation once again reveals the importance of cultural differences in adopting new technologies. different results suggest that student characteristics and related technology are critical in revealing the intention to use a technology (mccoy, galletta, and king 2007; teo, luan, and sing 2008). however, these results differ from the results of teo et al. (2019), who worked with a similar age group. in their study, only the effect of pu on intention was hypothesised, and no significant relationship was found (teo et al. 2019). pu (β = 0.318, p < 0.001) and peu (β = 0.171, p < 0.001) also positively affect students’ intention to use e-learning technologies. it shows that secondary school students in turkey increase their intention to use e-learning technologies when they perceive e-learning technologies as requiring little or no effort and when they perceive using e-learning technologies as beneficial. the findings are consistent with the original tam model (davis 1989) and higher education level e-learning studies (cheng and yuen 2018; kuliya and usman 2021). enj (β = 0.744, p < 0.001) explained the pu variable better than peu (β = 0.077, p < 0.001). in other words, the fact that students perceive the use of e-learning technologies as easy and especially enjoyable increases them to find these technologies useful. these findings are consistent with similar studies in the same age group (teo et al. 2019) and at the university level (al-rahmi et al. 2019; chang, hajiyev, and su 2017; salloum et al. 2019). considering the findings in terms of the peu variable, it was found that both cse and enj variables had a positive and significant relationship with the peu variable. in other words, as students’ self-efficacy towards e-learning technologies and their perceived fun in these technologies increase, their belief that they can use these technologies more easily or effortlessly also increases. in terms of the effects of the variables, cse (β = 0.452, p < 0.001) explained pu better than enj (β = 0.387, p < 0.001). in a http://dx.doi.org/10.25304/rlt.v31.2881 e. bahçekapılı 10 citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 (page number not for citation purpose) comprehensive literature review, including the e-learning context, the essential variables that affect peu were identified as cse and enj, consistent with this study (castiblanco jimenez et al. 2020). however, studies examining the effect of enj on peu show sometimes negative (salloum et al. 2019) and sometimes insignificant (teo et al. 2019) results. in summary, this study shows that the tam model successfully explains secondary school students’ intention to use e-learning technologies in turkey. the most critical variable affecting students’ intention to use e-learning technologies is perceived enjoyment, which is included in the research as an exogenous variable. finally, the results show that there may be differences in the intention to use e-learning technologies in different cultures and contexts. limitations and future work this study aims to determine students’ intentions at the k–12 level to use e-learning environments. there is a limited number of studies in the literature on this subject. the study is significant in this respect. however, the fact that it was carried out with the convenience sampling method due to the covid-19 pandemic can be considered lacking in generalisability. studies to be carried out with appropriate sampling approaches will contribute to the generalisability of the results. in addition, studies show that cultural differences can affect individuals’ technology acceptance. this study was carried out in turkey. and this only provides evidence for students in this culture. in the future, with cross-cultural studies, the factors that may affect the adoption of e-learning technologies by students at the k–12 level can be culturally compared. references abdullah, f. & ward, r. (2016) ‘developing a general extended technology acceptance model for e-learning (getamel) by analysing commonly used external factors’, computers in human behavior, vol. 56, pp. 238–256. doi: 10.1016/j.chb.2015.11.036 ajzen, i. & fishbein, m. (1980) understanding attitudes and predicting social behavior, prentice hall, englewood cliffs, nj. aktay, s. & keskin, t. (2016) ‘eğitim bilişim ağı (eba) incelemesi’, eğitim kuram ve uygulama araştırmaları dergisi, vol. 2, no. 3, pp. 238–256. available at: https://dergipark.org.tr/en/ pub/ekuad/issue/28248/300311 almaiah, m. a., al-khasawneh, a. & althunibat, a. (2020) ‘exploring the critical challenges and factors influencing the e-learning system usage during covid-19 pandemic’, education and information technologies, vol. 25, pp. 5261–5280. doi: 10.1007/s10639-020-10219-y al-rahmi, w. m. et al., (2019) ‘integrating technology acceptance model with innovation diffusion theory: an empirical investigation on students’ intention to use e-learning systems’, ieee access, vol. 7, pp. 26797–26809. doi: 10.1109/access.2019.2899368 anderson, j. c. & gerbing, d. w. (1988) ‘structural equation modeling in practice: a review and recommended two-step approach’, psychological bulletin, vol. 103, no. 3, pp. 411–423. doi: 10.1037/0033-2909.103.3.411 arnesen, k. t. et al., (2019) ‘k-12 online learning journal articles: trends from two decades of scholarship’, distance education, vol. 40, no. 1, pp. 32–53. doi: 10.1080/ 01587919.2018.1553566 asvial, m., mayangsari, j. & yudistriansyah, a. (2021) ‘behavioral intention of e-learning: a case study of distance learning at a junior high school in indonesia due to the covid-19 pandemic’, international journal of technology, vol. 12, no. 1, pp. 54–64. doi: 10.14716/ijtech.v12i1.4281 atmaca, d. (2021) ‘a study on the educational informatics network (eba) live course application in the “covid-19” pandemic process’, journal of educational theory and practice http://dx.doi.org/10.25304/rlt.v31.2881 http://dx.doi.org/10.1016/j.chb.2015.11.036 https://dergipark.org.tr/en/pub/ekuad/issue/28248/300311 https://dergipark.org.tr/en/pub/ekuad/issue/28248/300311 http://dx.doi.org/10.1007/s10639-020-10219-y http://dx.doi.org/10.1109/access.2019.2899368 http://dx.doi.org/10.1037/0033-2909.103.3.411 http://dx.doi.org/10.1080/01587919.2018.1553566 http://dx.doi.org/10.1080/01587919.2018.1553566 http://dx.doi.org/10.14716/ijtech.v12i1.4281 research in learning technology citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 11 (page number not for citation purpose) research, vol. 7, no. 3, pp. 301–313. available at: https://dergipark.org.tr/tr/pub/ekuad/ issue/67798/1051908 bandura, a. (1993) ‘perceived self-efficacy in cognitive development and functioning’, educational psychologist, vol. 28, no. 2, pp. 117–148. doi: 10.1207/s15326985ep2802_3 bhaumik, r. & priyadarshini, a. (2020) ‘e-readiness of senior secondary school learners to online learning transition amid covid-19 lockdown’, asian journal of distance education, vol. 15, no. 1, pp. 244–256. doi: 10.5281/zenodo.3891822 bozkurt, a. (2017) ‘türkiye’de uzaktan eğitimin dünü, bugünü ve yarını’, açıköğretim uygulamaları ve araştırmaları dergisi, vol. 3, no. 2, pp. 85–124. available at: https://dergipark.org.tr/en/pub/auad/issue/34117/378446 can, e. (2020) ‘coronavirüs (covid-19) pandemisi ve pedagojik yansımaları: türkiye’de açık ve uzaktan eğitim uygulamaları’, açıköğretim uygulamaları ve araştırmaları dergisi, vol. 6, no. 2, pp. 11–53. available at: https://dergipark.org.tr/en/pub/auad/issue/55662/761354 castiblanco jimenez, i. a. et al., (2020) ‘commonly used external tam variables in e-learning, agriculture and virtual reality applications’, future internet, vol. 13, no. 1, p. 7. doi: 10.3390/fi13010007 chang, c.-t., hajiyev, j. & su, c.-r. (2017) ‘examining the students’ behavioral intention to use e-learning in azerbaijan? the general extended technology acceptance model for e-learning approach’, computers & education, vol. 111, pp. 128–143. doi: 10.1016/j. compedu.2017.04.010 cheng, m. & yuen, a. h. k. (2018) ‘student continuance of learning management system use: a longitudinal exploration’, computers & education, vol. 120, pp. 241–253. doi: 10.1016/j. compedu.2018.02.004 cheng, m. & yuen, a. h. k. (2019) ‘cultural divides in acceptance and continuance of learning management system use: a longitudinal study of teenagers’, educational technology research and development, vol. 67, no. 6, pp. 1613–1637. doi: 10.1007/ s11423-019-09680-5 cheng, y.-m. (2011) ‘antecedents and consequences of e-learning acceptance’, information systems journal, vol. 21, no. 3, pp. 269–299. doi: 10.1111/j.1365-2575.2010.00356.x cicha, k. et al., (2021) ‘covid-19 and higher education: first-year students’ expectations toward distance learning’, sustainability, vol. 13, no. 4, p. 1889. doi: 10.3390/su13041889 compeau, d. r. & higgins, c. a. (1995) ‘computer self-efficacy: development of a measure and initial test’, mis quarterly, vol. 19, no. 2, p. 189. doi: 10.2307/249688 davis, f. d. (1989) ‘perceived usefulness, perceived ease of use, and user acceptance of information technology’, mis quarterly, vol. 13, no. 3, pp. 319–340. doi: 10.2307/249008 davis, f. d., bagozzi, r. p. & warshaw, p. r. (1992) ‘extrinsic and intrinsic motivation to use computers in the workplace1’, journal of applied social psychology, vol. 22, no. 14, pp. 1111–1132. doi: 10.1111/j.1559-1816.1992.tb00945.x davis, f. d. & venkatesh, v. (1996) ‘a critical assessment of potential measurement biases in the technology acceptance model: three experiments’, international journal of humancomputer studies, vol. 45, no. 1, pp. 19–45. doi: 10.1006/ijhc.1996.0040 findik-coşkunçay, d., alkiş, n. & özkan-yildirim, s. (2018) ‘a structural model for students’ adoption of learning management systems: an empirical investigation in the higher education context’, educational technology and society, vol. 21, no. 2, pp. 13–27. available at: https://www.jstor.org/stable/26388376 fiş erümit, s. (2021) ‘the distance education process in k–12 schools during the pandemic period: evaluation of implementations in turkey from the student perspective’, technology, pedagogy and education, vol. 30, no. 1, pp. 75–94. doi: 10.1080/1475939x.2020.1856178 fornell, c. & larcker, d. f. (1981) ‘evaluating structural equation models with unobservable variables and measurement error’, journal of marketing research, vol. 18, no. 1, pp. 39–50. doi: 10.2307/3151312 gallagher, m. w. (2012) ‘self-efficacy’, in encyclopedia of human behavior, v.s. ramachandran, (ed.), 2nd edn, elsevier, new york, pp. 314–320. doi: 10.1016/b978-0-12-375000-6.00312-8 http://dx.doi.org/10.25304/rlt.v31.2881 https://dergipark.org.tr/tr/pub/ekuad/issue/67798/1051908 https://dergipark.org.tr/tr/pub/ekuad/issue/67798/1051908 http://dx.doi.org/10.1207/s15326985ep2802_3 http://dx.doi.org/10.5281/zenodo.3891822 https://dergipark.org.tr/en/pub/auad/issue/34117/378446 https://dergipark.org.tr/en/pub/auad/issue/34117/378446 https://dergipark.org.tr/en/pub/auad/issue/55662/761354 http://dx.doi.org/10.3390/fi13010007 http://dx.doi.org/10.1016/j.compedu.2017.04.010 http://dx.doi.org/10.1016/j.compedu.2017.04.010 http://dx.doi.org/10.1016/j.compedu.2018.02.004 http://dx.doi.org/10.1016/j.compedu.2018.02.004 http://dx.doi.org/10.1007/s11423-019-09680-5 http://dx.doi.org/10.1007/s11423-019-09680-5 http://dx.doi.org/10.1111/j.1365-2575.2010.00356.x http://dx.doi.org/10.3390/su13041889 http://dx.doi.org/10.2307/249688 http://dx.doi.org/10.2307/249008 http://dx.doi.org/10.1111/j.1559-1816.1992.tb00945.x http://dx.doi.org/10.1006/ijhc.1996.0040 https://www.jstor.org/stable/26388376 http://dx.doi.org/10.1080/1475939x.2020.1856178 http://dx.doi.org/10.2307/3151312 http://dx.doi.org/10.1016/b978-0-12-375000-6.00312-8 e. bahçekapılı 12 citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 (page number not for citation purpose) hair, j. f., page, m. & brunsveld, n. (2019) essentials of business research methods, 4th edn, routledge, new york. doi: 10.4324/9780429203374 hara, h., takemura, a. & yoshida, r. (2010) ‘on connectivity of fibers with positive marginals in multiple logistic regression’, journal of multivariate analysis, vol. 101, no. 4, pp. 909–925. doi: 10.1016/j.jmva.2009.12.014 hwang, l.-a. et al., (2021) ‘nurturing academic enthusiasm and creativity among children from vulnerable communities: the role of computers’, behaviour & information technology, vol. 41, pp. 1–20. doi: 10.1080/0144929x.2021.1938227 kanwal, f. & rehman, m. (2017) ‘factors affecting e-learning adoption in developing countries–empirical evidence from pakistan’s higher education sector’, ieee access, vol. 5, pp. 10968–10978. doi: 10.1109/access.2017.2714379 kline and rex, b. (2011) principles and practice of structural equation modeling, 3rd edn, the guilford press, new york. kuliya, m. & usman, s. (2021) ‘perceptions of e-learning among undergraduates and academic staff of higher educational institutions in north-eastern nigeria’, education and information technologies, vol. 26, no. 2, pp. 1787–1811. doi: 10.1007/s10639-020-10325-x lavrakas, p. (2008) encyclopedia of survey research methods, sage publications, inc., thousand oaks, ca. doi: 10.4135/9781412963947 lee, m. k. o., cheung, c. m. k. & chen, z. (2005) ‘acceptance of internet-based learning medium: the role of extrinsic and intrinsic motivation’, information & management, vol. 42, no. 8, pp. 1095–1104. doi: 10.1016/j.im.2003.10.007 legris, p., ingham, j. & collerette, p. (2003) ‘why do people use information technology? a critical review of the technology acceptance model’, information & management, vol. 40, no. 3, pp. 191–204. doi: 10.1016/s0378-7206(01)00143-4 marangunić, n. & granić, a. (2015) ‘technology acceptance model: a literature review from 1986 to 2013’, universal access in the information society, vol. 14, no. 1, pp. 81–95. doi: 10.1007/s10209-014-0348-1 mccoy, s., galletta, d. f. & king, w. r. (2007) ‘applying tam across cultures: the need for caution’, european journal of information systems, vol. 16, no. 1, pp. 81–90. doi: 10.1057/ palgrave.ejis.3000659 nguyen, h. t. h. et al., (2020) ‘factors influencing students’ intention to use e-learning system: a case study conducted in vietnam’, international journal of emerging technologies in learning (ijet), vol. 15, no. 18, pp. 165–182. doi: 10.3991/ijet.v15i18.15441 purnomo, s. h. & lee, y.-h. (2013) ‘e-learning adoption in the banking workplace in indonesia’, information development, vol. 29, no. 2, pp. 138–153. doi: 10.1177/0266666912448258 raykov, t. & marcoulides, g. a. (2008) an introduction to applied multivariate analysis, an introduction to applied multivariate analysis, routledge, new york. doi: 10.4324/9780203809532 rizun, m. & strzelecki, a. (2020) ‘students’ acceptance of the covid-19 impact on shifting higher education to distance learning in poland’, international journal of environmental research and public health, vol. 17, no. 18, p. 6468. doi: 10.3390/ijerph17186468 salloum, s. a. et al., (2019) ‘exploring students’ acceptance of e-learning through the development of a comprehensive technology acceptance model’, ieee access, vol. 7, pp. 128445– 128462. doi: 10.1109/access.2019.2939467 teo, t. et al., (2019) ‘exploring the drivers of technology acceptance: a study of nepali school students’, educational technology research and development, vol. 67, no. 2, pp. 495–517. doi: 10.1007/s11423-019-09654-7 teo, t., luan, w. s. & sing, c. c. (2008) ‘a cross-cultural examination of the intention to use technology between singaporean and malaysian pre-service teachers: an application of the technology acceptance model (tam)’, educational technology and society, vol. 11, no. 4, pp. 265–280. available at: https://drive.google.com/ file/d/185e7oftp_adiluskqs9c34cupndy-mjl/view http://dx.doi.org/10.25304/rlt.v31.2881 http://dx.doi.org/10.4324/9780429203374 http://dx.doi.org/10.1016/j.jmva.2009.12.014 http://dx.doi.org/10.1080/0144929x.2021.1938227 http://dx.doi.org/10.1109/access.2017.2714379 http://dx.doi.org/10.1007/s10639-020-10325-x http://dx.doi.org/10.4135/9781412963947 http://dx.doi.org/10.1016/j.im.2003.10.007 http://dx.doi.org/10.1016/s0378-7206(01)00143-4 http://dx.doi.org/10.1007/s10209-014-0348-1 http://dx.doi.org/10.1057/palgrave.ejis.3000659 http://dx.doi.org/10.1057/palgrave.ejis.3000659 http://dx.doi.org/10.3991/ijet.v15i18.15441 http://dx.doi.org/10.1177/0266666912448258 http://dx.doi.org/10.4324/9780203809532 http://dx.doi.org/10.3390/ijerph17186468 http://dx.doi.org/10.1109/access.2019.2939467 http://dx.doi.org/10.1007/s11423-019-09654-7 https://drive.google.com/file/d/185e7oftp_adiluskqs9c34cupndy-mjl/view https://drive.google.com/file/d/185e7oftp_adiluskqs9c34cupndy-mjl/view research in learning technology citation: research in learning technology 2023, 31: 2881 http://dx.doi.org/10.25304/rlt.v31.2881 13 (page number not for citation purpose) teo, t. & noyes, j. (2014) ‘explaining the intention to use technology among pre-service teachers: a multi-group analysis of the unified theory of acceptance and use of technology’, interactive learning environments, vol. 22, no. 1, pp. 51–66. doi: 10.1080/10494820.2011.641674 venkatesh, v. & bala, h. (2008) ‘technology acceptance model 3 and a research agenda on interventions’, decision sciences, vol. 39, no. 2, pp. 273–315. doi: 10.1111/j.1540-5915.2008.00192.x vladova, g. et al., (2021) ‘students’ acceptance of technology-mediated teaching – how it was influenced during the covid-19 pandemic in 2020: a study from germany’, frontiers in psychology, vol. 12, p. 636086. doi: 10.3389/fpsyg.2021.636086 zhao, y. et al., (2021) ‘do cultural differences affect users’ e-learning adoption? a meta-analysis’, british journal of educational technology, vol. 52, no. 1, pp. 20–41. doi: 10.1111/ bjet.13002 http://dx.doi.org/10.25304/rlt.v31.2881 http://dx.doi.org/10.1080/10494820.2011.641674 http://dx.doi.org/10.1111/j.1540-5915.2008.00192.x http://dx.doi.org/10.3389/fpsyg.2021.636086 http://dx.doi.org/10.1111/bjet.13002 http://dx.doi.org/10.1111/bjet.13002 doi:10.3402/rlt.v19s1/7611 fostering academic competence or putting students under general suspicion? voluntary plagiarism check of academic papers by means of a web-based plagiarism detection system kerstin eleonora kohl* pädagogische hochschule freiburg, university of education, freiburg, germany (received 17 may 2011; final version received 11 june 2011) in view of the increasing number of cases of plagiarism and the ease of use of online published texts, universities are faced with a considerable challenge to prevent and take action against plagiarism in academic student papers. in reaction to plagiarism, web-based plagiarism detection systems (pdss) are increasingly used to check submitted papers � this checking entails various problems, for example the percentage of plagiarism found is only an indication of the actual extent of plagiarism and not all types of plagiarism can be identified. to cope with this problematic situation the voluntary plagiarism check (vpc), an alternative preventive university didactic concept, was developed at the university of education, freiburg (germany). it focused on the development of individual skills. students were able to submit their academic papers (e.g. an undergraduate paper, final thesis) anonymously. these were then tested with the pds ephorus. following interpretation and summary of the findings by the project team � plagiarism as well as referencing mistakes � we advised the students on a suitable approach to academic writing based on their own typical mistakes. the vpc was conducted as a three-semester research project and was later evaluated. about 500 academic papers were tested. in 90% of the undergraduates’ work incorrect and/or missing citations were found. this high percentage decreased among students in later semesters. instances of plagiarism were detected in about 40% of the papers when the texts of advanced students (]6th semester) were tested. at the same time the length of the plagiarised texts decreased. around half of the students stated that it was acceptable to copy single sentences or short passages from other sources without citation; they did not consider plagiarising on a limited scale as cheating. a similar number of students admitted to having doubts about whether they could write a good paper without plagiarising. almost all students said that they had experienced considerable uncertainty, stress and fear while writing academic papers. the project results offer new insights into internet-focused working strategies, on student justification for plagiarism and attitudes to literary property and on frequent mistakes. in addition to showing that there is broad acceptance among students of the vpc, the results can be taken into consideration in curriculum development and in developing courses to meet the needs of students. keywords: voluntary plagiarism check; universitiy didactic concept; plagiarism check of academic papers *corresponding author. email: kohl@ph-freiburg.de alt-c 2011 conference proceedings 0079 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # k.e. kohl. this is an open access article distributed under 90 the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http:// creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7611 the setting: the prefix ‘e’ changes academic learning and work plagiarism is dishonest and as such is academic misconduct. the intellectual property of third persons is ignored in practising plagiarism. academic writing requires the ability to use the correct academic style for the citation and acknowledging of sources. students should be able to apply what they have learned in the teaching context to their own research and publication practice. the increase in incorrect documentation practice goes hand in hand with the increase in use of on-line material � in fact we are dealing with the problem of ‘e’-plagiarism. our daily experience shows that many students use the internet to borrow texts and ideas without mentioning the sources. they claim this on-line material as their own work. weber (2009) refers to this phenomenon as the ‘‘copy & paste culture’’. studies show that 25�60% of students plagiarise (e.g., greubel 2009; mccabe 2005; scanlon and neumann 2002; szabo and underwood 2004; weber 2009). for a current literature review on the academic (dis)honesty of college students, see payan, reardon, and mccorkle (2010). in the context of academic learning, the internet serves as a source of information and a means of communication, participation, collaboration and social networking. the addition of the prefix ‘e’ necessitates changes in teaching practices. it is not just a question of applying existing learning skills to an electronic medium. with the establishment of e-learning contexts, universities expect students to become more active knowledge-designers in individualised, self-determined learning processes. this ‘‘shift from teaching to learning’’ (welbers and gaus 2005) raises the expectation that university didactic concepts will support such a change of the entire teaching and learning culture. in order to achieve a move towards learnercentred teaching and learning, on-line-based didactic concepts have to cover the whole qualification process, including the resulting academic papers. given that when plagiarism occurs, it is mostly a case of copying & pasting internet content (weber 2009); this (mal)practice must be viewed and treated as common’ e’plagiarism. therefore, the detection of such plagiarism is usually carried out by means of a text comparison with on-line sources. this means we are dealing with a challenge in the field of media-supported teaching and learning and their didactics. in order to find concepts for prevention, the focus must be on finding out which specific features of on-line-culture encourage plagiarism. attention should be paid to the variety of reasons for plagiarism, ranging from deliberate to unintentional acts (evans 2006; kohl 2010). in figure 1, an overview of the possible causes is given (summarised from bowman 2004; greubel 2009; mccabe, klebe treviño, and butterfield 2005; weber 2009). the findings show that in many papers lack of knowledge, poor academic sensitisation and false perceptions of the purpose and readership of the paper of the writing cause the errors and omissions. in view of our particular university didactic perspective we focused on a concept of prevention: the aspects in figure 1 with a white background. there are some didactic approaches which can help prevent unintended plagiarism such as giving students the opportunity to self-check their academic papers by using a plagiarism detection system (pds). students themselves can use the technical systems to check their papers. such procedures receive positive feedback from the students themselves (e.g. dahl 2007). however, it is questionable whether students can acquire honest academic working strategies and learn correct citation by alt-c 2011 conference proceedings 91 merely receiving a technical detection report. in the report, errors are not self-explanatory and there is no opportunity to explain or to discuss inappropriate attitudes. however, feedback and advice from academic staff seems to result in changes in the writing process and the acquisition of ethical norms (engler and landau 2011). [. . .] if a campus decides to address academic dishonesty by creating a social norms campaign, then the campaign must use the most credible source of information. [. . .] our results indicate that students are attentive to the source of these messages and may use this information, in part, to determine their behavioral response to the normative message they receive. efforts to identify and use credible sources (perhaps professors) to share messages about the true levels of academic dishonesty will contribute to more successful social norms campaign outcomes. (engler and landau 2011, 48) bringing the subject of plagiarism up for discussion and evaluating different types of plagiarism consistently is a difficult undertaking for colleges. there is disagreement among academic staff regarding what constitutes academic honesty with regard to using and acknowledging sources � in the evaluation of student performance as well as in their own writing behaviour (bennett, behrendt, and boothby 2011). this hinders the development of a common teaching concept across different faculties. the university didactic research project voluntary plagiarism check plagiarisms are mostly produced out of ignorance and only rarely intentionally. measures to avoid plagiarisms should rather be aimed at explaining and instructing the correct action than at repression. (dannenberg 2009, 133, translated by author) with this project we achieved a change of perspective: from general suspicion to an opportunity to develop skills. our central aim was to raise awareness and allow students to gain additional qualifications, so that they take responsibility and carry out future academic work fairly. intentional plagiarism unintentional plagiarism deliberate cheating, translation, full-/partial plagiarism reaction to deindividualization and/or regimented studies culture of the easiest way, limited epistemology cultural assumptions, culture of samplings reaction to disappointment, e.g. lack of mentoring oversights, careless mistakes lack of knowledge regarding different text types lack of awareness, becoming an academic vpc-strategy: offering information for the correct style of work vpc-strategy: discussion of attitudes to academic writing figure 1. possible reasons for plagiarism. k.e. kohl 92 our main questions while carrying out and evaluating the research project (using descriptive statistics as well as formative and summative evaluation methods) were: (1) is a facility such as the vpc system suitable to support the transfer from the teaching context to personal academic practice? is there a demand for this among students? (2) which types of plagiarism and/or mistakes in referencing and the acknowledgement of sources exist at our college, and to what extent? how do students explain their academic writing process? which factors do students list as affecting their behaviour? a general atmosphere of trust on the part of university lecturers in the honest working methods of their students as well as responsibility on the part of students cannot be achieved in an atmosphere of general suspicion or by means of a routine test of all academic papers by using a pds. the concept of the project voluntary plagiarism check (vpc) should be viewed as one possible alternative or a complement to other didactic concepts with the aim of prevention. in the three semesters up to february 2011, a vpc of students’ academic papers (term paper, undergraduate or final thesis) was implemented and evaluated at the university of education in freiburg, germany (www.ph-freiburg.de). the university offers programmes of study in education. eighty per cent of the students are undertaking initial teacher training. ba/ma degree courses in health education and promotion, economics education, media education, early years education and adult education are also available. undergraduate degrees typically take three to four years to complete. graduates of all courses will most likely work in the field of education. this means that they may act as role models when it comes to handling intellectual property. the project was based at the centre for media literacy (medienjkompetenzjzentrum, www.ph-freiburg.de/mkz), and run by the author. using the internet-based pds ephorus (www.ephorus.com), students were able to have their work checked anonymously. they could either send the project team their paper by email or bring a file (*.doc, *.rtf, *.pdf) stored on a usb memory stick during office hours. after plagiarism detection and interpretation of the pds-report they receive personal feedback and, if necessary, useful concrete advice for the practical application in academic communication and writing as well as background information concerning ethical and epistemic background issues. examples drawn from students’ texts in the pds report were used to generate feedback. the students were able to choose between a personal consultation and summarised feedback by mail. the students did not receive the full pds report. this was to prevent the pds reports from being used to check whether intentional plagiarism would be detected when the paper was submitted to faculty. at the same time, we wanted to encourage students to revise the paper on their own with the help of the examples provided. it became clear that certain mistakes appeared repeatedly. it therefore proved useful to give advice based on well-chosen examples. in our study, about 500 academic papers were checked for plagiarism and mistakes in citing and this was followed by a consultation. directly after the consultation, the vpc service was evaluated by the students. an on-line questionnaire with standardised and open questions was used (https://www.soscisurvey.de/plagiatskontrolle). alt-c 2011 conference proceedings 93 with the aim of statistical capture, the text characteristics of every submitted paper were recorded by the project team (e.g. type of paper submitted, mistakes found, types of plagiarism, subject area). descriptive results could be derived from this data, for example, frequency of plagiarism, type of quoting mistakes (see research question (1)). from analysis of this questionnaire we obtained information about: a. students’ motives for cheating, for using the vpc, for their writing process, b. self-assessment of writing skills, habits in the use of on-line material and their underlying epistemologies, c. course of study, year of study (see research question (2)) and d. assessment of the didactic concept and the actual performance of vpc, their suggestions for improvement (see research question (1)). we established that students from all courses and all departments used vpc. this is shown in figure 2. depending on the size of individual departments, the number of papers submitted varied. it was interesting to establish that plagiarism and/or citing mistakes did in fact take place in subject areas where academic staff had denied this would happen due to subject-specific circumstances. we were able to show that the incorrect use of sources is present in all departments and that students from all departments have a need for advice. regarding the kind of academic papers that were submitted (figure 3) one can see that mainly final theses were tested with vpc (60%). term papers were 37% and 3% were dissertations as well as other types of texts we could not identify. web-based plagiarism control � detection is not proof it is difficult to identify and prove plagiarism without technical support. comparison is often time-consuming and, therefore, cost-intensive. noticeable changes in writing style, unusual choice of words or ‘brilliant ideas’ sometimes indicate plagiarism � but the original sources are often not identifiable or traceable, as in the case of translated texts, for example. a pds allows for quick and easy detection � by comparison with on-line material and with a bank of previously checked files � with no extra effort. the pds compares a paper with millions of others on the internet. the document is 0 2 4 6 8 10 12 14 16 18 20 disciplines % educational science psychology german mathematic economics/technics media education theology art geology sports french english history biology others/n.s. figure 2. disciplines using the vpc. k.e. kohl 94 simultaneously checked for matches with any papers submitted to that system in the past. the results are displayed in a summarised report. the ease of use of this technical search solution makes it of particular interest to universities. the software can be integrated in existing learning management systems. we have to consider what level of reliability can be achieved by a pds. the result of plagiarism checking contains information about correspondences with other text sources. the percentage of agreement between the detected source and the original source is shown along with the associated url (figure 4 and 5). every match has to type of papers final thesis (diploma, master) term paper others/n.s. 0 10 20 30 40 50 60 70 % figure 3. type of papers submitted. figure 4. screenshot, detailed results. alt-c 2011 conference proceedings 95 be evaluated � not every match is a plagiarism. in fact, in many cases corresponding parts were in fact not plagiarism, e.g. matches due to careless mistakes in quoting sources, definitions or legitimate quotes from classical literature, as shown in figure 5. the pds indicates plagiarism, but cannot prove it. the detected text parts become plagiarism only by the interpretation of the lecturer. when both plagiarism and mistakes are present, as in the example, it is not easy to decide. only a small number of plagiarism types can be identified by a pds � it is limited to verbatim plagiarism of more than 10 consecutive words. these detected sentences mostly originate from on-line material, available free, the extracts rarely come from printed or fee-based publications. it is not possible to detect plagiarism based on translations, mosaic plagiarism, adopting the pattern or structure of arguments and ideas, alterations, paraphrasing and restructured parts of a text. self-plagiarism or submitting someone else’s work cannot be identified either. in our experience, mostly mixed forms of plagiarism appeared in the papers checked. in the event of detection of ‘e’-plagiarism it is very likely that other types of plagiarism may occur too and/or that text extracts from printed publications have been borrowed as well. to sum up, a pds is not a miracle solution to detect plagiarism, but it is extremely helpful as long as teachers and lecturers know how to interpret the findings. they also have to be aware that if no plagiarism was found by the pds, this does not automatically mean that there was no plagiarism at all in the paper submitted. precisely, because it is not a miracle solution, but it was available for use in the context of the preventive didactic methods shown in this article. the work in figure 5 was an original part of a checked undergraduate thesis. underlined passages correspond to sections in source documents, urls are given at the top. sub-section 1: real plagiarism with some words substituted; parts were borrowed from a ba thesis. sub-section 2: citation (‘kirsten peters said ‘‘sehr viele . . .’’’) without date or page reference. perhaps this was a mistake because the corresponding percentage 8% checked: 22.12.2010, 13:30 corresponding documents: 4% http://www.e-text.org/text/goethe%20-%20faust.pdf 3% http://de.wikisource.org/wiki/faust_i 3% http://www.digbib.org/johann_wolfgang_von_goethe_1749/faust_i2% http://www.diplom.de/bachelorarbeit-10006/kindsmords_im_deutschen_drama.html […] wegweisend ist dabei das herausarbeiten der unterschiedlichen umsetzungen des motivs in den beiden dramen. zunächst geht es um einen historischen abriss des themas, um diskrepanzen zwischen der historischen realität und der literarischen umsetzung aufzuzeigen. hier werden die rechtlichen sanktionierungen, unter anderem die peinliche gerichtsordnung aus dem jahr 1532, die lange zeit maßgebend für die verurteilung der kindsmörderinnen war und das soziale milieu zum gegenstand der untersuchung. mit dem fall der kindsmörderin susanna margareta brandt, die im jahre 1772 in frankfurt hingerichtet worden ist, möchte ich die zuvor genannten themen noch einmal kurz aufzeigen. heute werden kindsmörder/-innen genauso wie diejenigen “verbrecher” bestraft, die einen menschen getötet haben. in der literatur des 18. jahrhundert werden in der regel unschuldige bürgermädchen von adligen verführt und dann verlassen. […] so bemerkt kirsten peters: “sehr viele frauen hätten ihre kinder nicht getötet, wenn die äußeren bedingungen sie nicht dazu gezwungen hätten, wobei neben der materiellen situation die einflussnahme der umwelt nicht übersehen werden darf.” […] schaut man wieder in goethes drama, so werden die bewertungen zu unehelichen kindern deutlich. „wenn erst die schande wird geboren, wird sie heimlich zur welt gebracht, und man zieht den schleyer der nacht, ihr über kopf und ohren; ja, man möchte sie gern ermorden“. so beschreibt valentin im vers 3537 von goethes faust die situation nach einem vorehelichen geschlechtsverkehr mit folgender schwangerschaft und geburt. 1 2 3 figure 5. printable result of plagiarism detection via pds ephorus. k.e. kohl 96 source is listed in the reference list. or it might be borrowed from the ba thesis too, because the citation itself appears in the thesis as well. in this case, the student would not have actually read the article written by author kirsten peters. sub-section 3: a correct textual component, a verse from johann wolfgang von goethe’s faust. results: thoughtlessness, carelessness, ignorance but rarely cheating personally, i used the vpc to make sure that i made no mistakes, even though i really feel relatively secure in citation. the theme of correct academic writing is rarely discussed in courses. so, the vpc is a good way to make up to for this. at the end of the long writing-process i was so confused. i really didn’t know if i had overlooked errors in some kind of inaccurate work and i didn’t remember the sources of some passages either. this was not deliberate plagiarism and the vpc gives me the chance to eliminate these errors before submitting my paper. with the feedback i got i feel more confident now. i spent a lot of time on literature review and research. after a certain point, it is difficult to distinguish between your own thoughts and the intellectual property of others. so the vpc is very helpful! (student, mail feedback 2009; translated by author) at this point it should be emphasised that the following results may not be generalised to the general occurrence of plagiarisms at our university or to the overall attitudes, writing habits and knowledge of all students. it is assumed that a sub-group of students with a background of particular interests and knowledge used vpc. we reached about 10% of the students at our university with our service. for this subpopulation of students we can determine that the vpc was suitable for their aims and useful as a complementary qualification. it was rated as very helpful and graded very highly. the most commonly cited reasons for using vpc were (1) fear of negative consequences in the case of mistakes, (2) awareness of gaps in one’s own knowledge, (3) insecurity in the face of inconsistent information for correct academic writing (e.g. citing styles), (4) high level of conscientiousness because of planned academic career and (5) compensation for the lack of input on correct academic writing in the academic disciplines. summary of results (1) occurrence of plagiarism and quoting mistakes: plagiarism, intentional as well as accidental, exists in all text forms and appears in every discipline. in the academic papers of students in their first three semesters of study, we found a large number of plagiarism and quoting mistakes (90% of examined papers). mostly, longer excerpts were copied from sources, which were not cited at all or cited in an incomplete/ erroneous way. this high instance of plagiarism decreased as students progressed through their studies. from the 6th semester to degree-level, quoting mistakes and plagiarism ‘only’ occurred in about 40% of the papers. also, the length of the excerpts used clearly decreased. this might be seen as progress in learning. or it may be that with the increasing differentiation in studies and specialisation in research questions, the probability of discovering text conformities may decrease: it was not possible to integrate as much internet content as in the academic papers written by students in early semesters. alt-c 2011 conference proceedings 97 (2) self-assessment of academic writing: there is great uncertainty about correct academic writing and considerable pressure when composing one’s own texts. ninetyone per cent of students said that they had had problems, ranging from fear and stress (not being good enough at the subject, writing something wrong, difficulties in expressing their own ideas, time pressure/cognitive overload, unforeseen mistakes in form and content type) to loss of motivation (lack of interest in the topic, lack of mentoring by lecturers, lack of explanations about honest academic practices). about 30% of the students assumed the submitted papers were not read thoroughly by lecturers. about 50% stated ‘it is acceptable to borrow single sentences or shorter passages from texts without citing’, about 25% admitted to plagiarising in this way and to doing this regularly. two-thirds of students learned the correct use of sources on their own, some learnt correct academic writing from peers, only 20% said that they received explicit instruction about honest academic practices in lectures or seminars at university. one-third stated that they acquired knowledge about citing incidentally. (3) evaluation of the vpc: 95% of students who had used vpc indicated having used it to reduce insecurity and in particular the fear of poor marks (50%). twenty-one per cent wanted to compensate deficits in their studies by utilising the consultation � this type of motive can be labelled as ‘self-determined prevention’. these students were motivated to learn the correct method of citing. in addition, we interpreted intended plagiarism in the occurrence of several large matching text passages (�15% of the content) in single papers. sometimes students presented plagiarism with text correlations �85%, mainly borrowed from on-line-buyable sources for academic theses. three per cent of the students presenting such papers tried to use vpc to check whether the deliberately plagiarised paper would be identified when submitted to faculty. (in fact, they were disappointed that we could not give them this assurance. perhaps this is the reason why in the last semester of the project not a single paper of this kind was presented to the team.) we found there was broad acceptance of the vpc. it was evaluated very positively with the grade 1, sehr gut (equivalent of a grade a). all students were keen that vpc should be set up as a regular service in the future. in particular, the personal feedback given with examples from their own text was considered extremely helpful. in addition, it was suggested by students that feedback and advice should be available more quickly � it sometimes took two or three days until the results were available due to technical issues. summary and perspective the frequent cases of plagiarism at universities were a convincing reason for implementing the university didactic concept of the voluntary plagiarism control. we viewed ‘e’-plagiarism as an opportunity to discuss attitudes, to gain knowledge as well as to establish a basis for curricular development in teaching academic integrity and, in particular, correct citing and honest use of sources. with the vpc authentic and significant learning becomes possible. this kind of situated learning ties in with the concept of autonomous learning and thus reflects the idea of on-line based teaching at universities in general. the findings of other authors cited above on the various causes of plagiarism (figure 1) � of which a large number do not lead to deliberate k.e. kohl 98 plagiarism � were confirmed for the sub-group of students using vpc. we assume that students in general often have difficulties in academic writing and many of them plagiarise intentionally or unintentionally. to address this issue the vpc has been available as a permanent service at the centre for media literacy since march 2011. currently, a detailed artefact analysis of the submitted papers is in preparation. we would like to explore aspects of the ontogeny of academic writing competences (pohl 2007) concerning the prevalence of plagiarism, also in correlation with research competences and epistemic relief (pohl and steinhoff 2010). references bennett, k.k., l.s. behrendt, and j.l. boothby. 2011. instructor perceptions of plagiarism: are we finding common ground? teaching of psychology 38, no. 1: 29�35. bowman, v., ed. 2004. the plagiarism plague � a resource guide and cd-rom-tutorial for educators and librarians. new york: neal-schuman. dahl, s. 2007. turnitin†: the student perspective on using plagiarism detection software. active learning in higher education 8: 173�91. dannenberg, d. 2009. »das kurze leben des s.b. preuss« oder: zitieren und belegen in bibliothekskursen. in wissensklau, unvermögen oder paradigmenwechsel? plagiate als herausforderung für lehre, forschung und bibliothek [cheating, nescience or paradigm shift? plagiarism as accepted challange at universities, research and librarianship], ed. r. barth, n. böller et al., 133�42. chur: arbeitsgemeinschaft informationswissenschaft. engler, j.n., and j.d. landau. 2011. source is important when developing a social norms campaign to combat academic dishonesty. teaching of psychology 38, no. 1: 46�8. evans, r. 2006. evaluating an electronic plagiarism detection service. the importance of trust and the difficulty of proving students don’t cheat. active learning in higher education 7, no. 1: 87�99. greubel, r. 2009. vom fehlverhalten zum plagiator � fördert das internet den wissensklau? in wissensklau, unvermögen oder paradigmenwechsel? plagiate als herausforderung für lehre, forschung und bibliothek [cheating, nescience or paradigm shift? plagiarism as accepted challange at universities, research and librarianship], ed. r. barth, n. böller et al., 1�11. chur: arbeitsgemeinschaft informationswissenschaft. kohl, k.e. 2010. im zweifel für die lernchance? freiwillige plagiatskontrolle wissenschaftlicher arbeiten. in digitale medien für lehre und forschung [to give an opportunity to learn? voluntary plagiarism check of students academic papers], ed. s. mandel, m. rutishauser and e. seiler schiedt, 415�27. waxmann. mccabe, d.l., l. klebe treviño, and k. butterfield. 2005. cheating in academic institutions: a decade of research. ethics & behavior 11, no. 3: 219�32. mccabe, d.l. 2005. cheating among college and university students: a north american perspective. international journal for educational integrity 1: 1. payan, j., j. reardon, and d.e. mccorkle. 2010. the effect of culture on the academic honesty of marketing and business students. journal of marketing education 32: 275�91. pohl, t. 2007. studien zur ontogenese wissenschaftlichen schreibens [ontogenesis of academic writing]. tübingen: niemeyer. pohl, t., and t. steinhoff. 2010. textformen als lernformen. kölner beiträge zur sprachdidaktik (köbes) 7. scanlon, p.m., and d.r. neumann. 2002. internet plagiarism among college students. journal of college student development 43, no. 3: 374�85. szabo, a., and j. underwood. 2004. cybercheats. is information and communication technology fuelling academic dishonesty? active learning in higher education 5, no. 2: 180�99. weber, stefan. 2009. das google-copy-paste-syndrom. wie netzplagiate ausbildung und wissen gefährden [the google-copy-and-paste-syndrome. internetplagiarism endanger the quality of study and apprenticeship]. hannover: heise. welbers, u., and o. gaus. 2005. the shift from teaching to learning. konstruktionsbedingungen eines ideals. bielefeld: w. bertelsmann. alt-c 2011 conference proceedings 99 doi:10.3402/rlt.v24.30083 original research article designing for learner engagement with computer-based testing richard walker a * and zoe handley b a e-learning development teamuniversity of york, heslington, york, uk; b department of education, university of york, heslington, york, uk (received 20 october 2015; final version received 23 november 2016) the issues influencing student engagement with high-stakes computer-based exams were investigated, drawing on feedback from two cohorts of international ma education students encountering this assessment method for the first time. qualitative data from surveys and focus groups on the students’ examination experience were analysed, leading to the identification of engagement issues in the delivery of high-stakes computer-based assessments.the exam combined shortanswer open-response questions with multiple-choice-style items to assess knowledge and understanding of research methods. the findings suggest that engagement with computer-based testing depends, to a lesser extent, on students’ general levels of digital literacy and, to a greater extent, on their information technology (it) proficiency for assessment and their ability to adapt their test-taking strategies, including organisational and cognitive strategies, to the online assessment environment. the socialisation and preparation of students for computer-based testing therefore emerge as key responsibilities for instructors to address, with students requesting increased opportunities for practice and training to develop the it skills and test-taking strategies necessary to succeed in computer-based examinations. these findings and their implications in terms of instructional responsibilities form the basis of a proposal for a framework for learner engagement with e-assessment practices. keywords: computer-based testing; summative assessment; participant-informed design; digital skills; it proficiency for assessment introduction high-stakes computer-based testing is growing in importance within uk higher education and has developed from computer-based formative assessment practices which are already well established across the sector (jenkins, walker, and voce 2014; jisc 2007). the most recent universities and colleges information systems association (ucisa) technology enhanced learning survey (walker et al. 2016) reported that 10 out of the 110 institutions surveyed have implemented defined response (multiple choice) summative tests for over 50% of their courses. we observe that whilst there has been strong institutional uptake of high-stakes e-assessment, there has been little corresponding research conducted on students’ attitudes towards and experiences of computer-based testing. a review of the literature reveals that most research to date has focused on investigating the differential impact of *corresponding author. email: richard.walker@york.ac.uk responsible editor: peter reed, university of liverpool, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 r. walker and z. handley. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 computer-based versus pen-and-paper assessments on student achievement (kingston 2008; leeson 2006; mead and drasgow 1993) and the relationship between individual differences including gender, race, socio-economic status, digital literacy, computer anxiety and performance on computer-based assessments (leeson 2006). only a smaller number of studies have investigated students’ attitudes towards computerbased assessment (e.g. dermo 2009; deutsch et al. 2012; hillier 2014; walker, topping, and rodrigues 2008), and these studies have tended to focus on undergraduate students and report on a single experience of computer-based testing, with few studies exploring engagement over a series of assessments � deutsch et al. (2012) is a notable exception. research studies comparing performance on computer-based and pen-and-paper examinations have found that differences are generally small and not of practical significance � in particular, for objective and multiple-choice assessments (kingston 2008; leeson 2006; mead and drasgow 1993). results of studies investigating the relationship between learner variables and performance on computer-based assessments are mixed, and where effects of learner variables are observed, these tend to be small (leeson 2006). whilst these studies are important and serve to allay concerns about equity that have been raised in explorations of students’ attitudes towards and experiences of computer-based testing (e.g. hillier 2014), the few post-examination surveys and focus groups that have been conducted have identified a wider range of concerns which merit further exploration. concerns frequently cited in these studies include computer anxiety and perceptions of computer self-efficacy, concerns relating to test security and the potential for cheating, and technical difficulties (deutsch et al. 2012; hillier 2014; ozden, erturk, and sanli 2004). this paper attempts to fill the gap in the literature on the postgraduate student experience with computer-based assessment. postgraduate students are important subjects for investigation because they present unique challenges with respect to the introduction of computer-based assessment; they are often mature students who may have less experience of using technology in education, and the short duration of ma programmes can limit the time available to socialise and prepare them for different examination formats and procedures. our study investigates student engagement issues for a computer-based examination of research methods, drawing on research from two cohorts of international ma students who were encountering this assessment method for the first time in their programme. it combines preand post-examination surveys and follow-up focus group interviews to provide insights into students’ concerns and the steps tutors ought to take to prepare students for computer-based examinations. these insights have informed the development of a framework for learner engagement with e-assessment practices (lee-ap), which is presented in this paper. context and exam design in 2010, the university of york introduced the requirement that all taught content on a study programme should be assessed. consequently, we were now required to assess our master’s level research methods module. by the time this was introduced, our research methods module was taken by more than 150 international students, mainly chinese students, and the number of students on the programmes was increasing year by year. given the size of the cohort and the need to complete marking and delivery of feedback to students within 6 weeks, the decision was taken to introduce a predominantly multiple-choice-style examination. r. walker and z. handley 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 in academic years 2011�2012 and 2012�2013, the ma research methods module was therefore assessed through a 2-hour pen-and-paper examination, comprising two-thirds multiple-choice-style questions and one-third longer answer questions. multiple-choice questions were used to assess knowledge and understanding of a wide range of methods for carrying out research. longer answer questions were employed to assess students’ ability to critically examine research reports in terms of the suitability of the methods used and the implications for the substantive claims that are made (see the assessment design section). the module team, however, still found that marking the paper-based multiplechoice examination in the required 6-week period was challenging when busy teaching other modules. the decision was therefore taken to switch to an e-assessment format in 2013�2014, which would permit the automation of the marking of the multiplechoice questions and consequently reduce the turn-around time in the delivery of marks and feedback to students. the 161 students who sat the module in 2013�2014 completed a 1-hour formative exam in their first term (november 2013) and a 2-hour summative exam at the beginning of the second term (january 2014). multiple-choice items accounted for the main component (70%) of the summative examination, with open questions employed to assess critical thinking and higher order skills. the former were worth between 1 and 6 marks depending on the number of parts and difficulty; the latter required students to write around five sentences and were worth 10 marks each. they were presented in a randomised order within a locked down and secure version of the institutional blackboard learn virtual learning environment (vle). the 191 students of the 2014�2015 cohort followed a similar procedure. however, in response to the previous cohort’s concerns over typing proficiency and noise levels in the completion of open-response questions, adjustments were made to the summative exam design, with a reduction in the number of both openand multiple-choice-style question items � as discussed in the assessment design section, adjustments were also informed by difficulty and discrimination analyses. as in 2013�2014, the order of presentation of the questions was randomised in both the formative and summative examinations because this was essential to maintaining the integrity of the examination. privacy screens were not available in all the personal computer (pc) rooms which were being used as test centres, and in this context there remained the possibility that students could look at each other’s screens � hence the randomised design. in response to feedback on the organisation of the 2013�2014 summative exam, questions were however grouped by weighting; items worth the highest number of marks were presented first, and randomisation was applied within those groups (as opposed to across the examination as a whole). research methods we recognise students as key stakeholders in the assessment process, and feedback was actively sought from each cohort and used to guide improvements to the examination as part of our participant-informed design approach along with analyses of pitch of the examination (see the assessment design section). feedback on students’ assessment experience was solicited through questionnaires and focus groups. in 2013�2014, questionnaire and focus group data were collected after the summative examination. in 2014�2015, questionnaire data were collected after the formative examination and again after the summative examination, with focus group interviews also conducted after the summative examination. the questionnaires and research in learning technology citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 focus group instruments were informed by dermo’s (2009) student perceptions of e-assessment questionnaire (speaq) and a review of previous instruments and research on students’ attitudes towards and experiences of computer-based testing (deutsch et al. 2012; ferrao 2010; frein 2011; hillier 2014; williams and wong 2009). students’ prior exposure to computer-based testing and their reflections on their preparation for this exam were explored. students were also invited to reflect on their experiences of taking the exam, comment on the suitability of the assessment method and make recommendations for how it could be improved in the future. the study was approved by the departmental ethics committee, and information sheets and consent forms were distributed to all participants. the 2013�2014 survey was completed by 48 of the 161 students who sat the end of module examination. the 2014�2015 preand postexamination surveys were completed by 42 of the 191 students who sat the module. most of these participants were aged 20�24, and, reflecting the demographics of the cohort as a whole, the majority of respondents were chinese females (see table 1). this paper focuses on the data collected from the 36 chinese females who completed the survey in 2013�2014 and the 28 chinese females who completed it in 2014�2015. a total of 18 chinese female students from the 2014�2015 cohort volunteered to participate in three separate focus group interviews in january 2015, with five students from the 2013�2014 cohort participating in february 2014. transcripts from the focus groups were generated, and a qualitative content analysis was performed (hsieh and shannon 2005). the unit of analysis was a line in the transcript, which, in turn, could address multiple units of meaning. comments were categorised and then mapped against an evaluative framework based on dermo’s (2009) key themes, addressing affective variables, validity, practicality, reliability, security and pedagogical issues related to the assessment method. the categorisation and mapping processes were then repeated for the open comments from the surveys, and the outputs from these analyses were then compared to form a rich picture of student experiences with computer-based testing, from which common themes in the student assessment experience were derived. findings and discussion the findings of the questionnaires and interviews highlight a range of issues, which appear to have influenced our students’ reception of computer-based testing. these issues are discussed below along with recommended actions for instructors to consider when introducing computer-based testing. together they form a proposal for a framework for lee-ap, (see table 2). table 1. demographics of the cohort and sample. cohort sample academic year overall no. no. chinese females % chinese females overall no. no. chinese females % chinese females 2013�2014 161 140 87 42 36 86 2014�2015 191 170 89 31 28 90 r. walker and z. handley 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 table 2. learner engagement with e-assessment practices (lee-ap) framework. theme issues recommended actions 1. socialisation of learners orientation of learners to assessment methods (i) prior exposure to e-assessment/ familiarisation offer comprehensive induction procedure addressing: (ii) timing within study programme -rationale for e-assessment (iii) rationale for e-assessment: perceived fit with curriculum & teaching methods and perceived benefits -suitability of the methods; their fit within the study programme and complementarity to discipline and teaching methods. (iv) perceived fairness and equity in assessment methods reassurance to students on exam procedure and equity of methods establish e-assessment as the norm from the start of the programme 2. preparation of students for assessment guidance to and preparation of students for e-assessment (i) keyboard proficiency under exam conditions (ii) reading from screen prior exposure to exam hardware or byod approach, based on students using their own devices (locked down). (a) digital skills (iii) familiarisation with exam environment & controls for navigation/question selection/writing authentic practice opportunities under timed conditions: frequent opportunities for students new to e-assessment (b) exam technique: test-taking strategies for online exams (i) familiarisation with exam format and question types video tutorials and class-based guidance and feedback on formative tests. (ii) ‘online exam craft’: addressing organisational (question selection; time management) and cognitive strategies (reviewing & editing answers) provision of multiple practice opportunities, enabling students to review and adjust paper-based test-taking strategies to the computer-based testing environment 3. assessment design & interface organisation and presentation of question set, preparation of user interface and assessment venue (i) balance (open/closed items) and volume of questions per allotted time (ii) sequencing and order of questions perform iterative question review (difficulty & discrimination analyses) (a) assessment design ensure consistency in ordering of questions (question value) to individual workstations when randomised question selection methods are employed. r e se a rch in l e a rn in g t e ch n o lo g y c ita tio n : r e se a rc h in l e a rn in g t e c h n o lo g y 2 0 1 6 , 2 4 : 3 0 0 8 3 h ttp ://d x .d o i.o rg /1 0 .3 4 0 2 /rlt.v 2 4 .3 0 0 8 3 5 (p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 table 2 (continued ) theme issues recommended actions (b) design of assessment interface (i) item presentation (ii) intuitive controls (e.g. flagging & navigation) comprehensive induction and guidance on exam environment and controls (c) preparation and management of assessment centre(s) (i) consistency in presentation of assessment venues (set-up & invigilation procedures) exam management ensuring consistency in presentation and management of exam venues (ii) noise management: keyboards review of supported hardware (iii) specialist support & contingency measures (technical failures) contingency policy & procedures in place r . w a lk e r a n d z . h a n d ley 6(p a g e n u m b e r n o t fo r c ita tio n p u rp o se ) c ita tio n : r e se a rc h in l e a rn in g t e c h n o lo g y 2 0 1 6 , 2 4 : 3 0 0 8 3 h ttp ://d x .d o i.o rg /1 0 .3 4 0 2 /rlt.v 2 4 .3 0 0 8 3 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 socialisation of learners feedback indicated a need for the instructor to articulate the rationale and the suitability of the assessment methods to the discipline being assessed, with some students unconvinced by the need for computer-based testing methods. i think this exam could also be paper based because in research methods we always use computers to analyse some data, always computer but in this exam only a few questions is about how to analyse data. the others could also be shown on paper, so for me, i think, it’s not really necessary to be done with computer. (post-test focus group 2014� 2015) this resonates with findings from hillier’s cross-disciplinary review of undergraduate students’ attitudes towards e-assessment and the perceived ‘readiness’ of disciplines to match a computerised assessment approach (hillier 2014). when introducing e-assessment in disciplines and module areas where there is a less obvious fit with approach to assessment, it will be necessary to provide students with greater support in managing their anxiety levels. the rationale for e-assessment should also address the benefits of this approach. our findings confirm previous research (e.g. noubandegani 2012) in highlighting the perceived presentational benefits of typed open responses as opposed to poorly handwritten answers and the favourable impression that students believe this will have on markers: if we are bad at writing, yes especially in china, we lose marks because of that. (posttest focus group 2014�2015) notwithstanding these benefits, some students felt that there was a generational bias in favour of more technically literate students (i.e. those who had proceeded directly to graduate study and not spent time working) and an equity issue affecting mature students returning to education, who might take longer to adjust to computer-based exams: some of our classmates after they have had some experience . . . they return to school to get more experience in teaching. it could be some difficulty for them to use a computer in typing when they attend the examination, so it could take them longer time to get used to the system, so i think it could unfair for them. (post-test focus group 2014�2015) this reflects a commonplace perspective on the existence of net generation (tapscott 2008) or digital natives (palfrey and gasser 2008; prensky 2001a, b) with distinctive skills and aptitudes for digital learning, which still persists and perhaps is more keenly felt by mature and returning students to full-time education, despite recent studies in the uk rejecting these claims (jones et al. 2010; margaryan, littlejohn, and vojt 2011). whilst research on computer-based testing (e.g. frein 2011; leeson 2006) has found no difference in test results between highly experienced and less experienced users of computers, our findings align with previous studies of student perceptions of e-assessment (fluck 2013; hillier 2014) in underlining the need to address students’ concerns about fairness in the orientation phase and to offer reassurance from the outset. the orientation of learners to a new assessment method � particularly when this method departs from established assessment norms within a study programme � emerges as a key responsibility for instructors to address. this is consistent with research in learning technology citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 previous studies (e.g. deutsch et al. 2012; zakrzewski and steven 2003) which have identified the need for instructors to integrate the assessment method at an early stage into the curriculum and in this way address and reduce student anxiety levels, which may stem from a variety of factors touching on computer self-efficacy, concerns over security and cheating, to fears over computers crashing during an online examination (mogey and sarab 2006). preparation of students for assessment computer aversion or anxiety as a barrier to the adoption of computer-based assessment is a well-researched concept (e.g. durndell and lightbody 1993; meier 1985, 1988). meier defines this as a negative affective and cognitive state which often occurs when individuals have low expectations about the rewards of using computers or low confidence in their ability to use computers effectively. adequate preparation appears key to counteracting this mindset, but there is less agreement on the nature of the interventions that are required to address student anxiety levels. student feedback in our study highlighted two areas where preparation may be needed: digital skills and examination technique. digital skills zakrzewski and steven (2003) have highlighted the enhancement of students’ information technology (it) skills as a prerequisite for student preparation, but it is far from clear what this actually entails. leeson’s (2006) review suggests that there is no established relationship between examinees’ level of computer familiarity and performance on computer-based tests. similarly, we found no association between any of the indicators of digital literacy that we surveyed, which included previous experience of computer-based testing and keyboarding skills and preferences for computer-based versus paper-based assessment. indeed, whilst keyboarding proficiency was highlighted by some individuals as a differentiating factor in exam performance, more commonly concerns were expressed over the use of universityprovided hardware and software in the exam: i’m not used to using the keyboard because it’s different from laptop keyboard. (post-test focus group 2014�2015) i find that when i type really faster and quickly, there are typing errors i won’t recognise because there are no red lines on the line of words that i typed wrong. (post-test focus group 2014�2015) these findings suggest that we need to draw a distinction between digital proficiency � reflected in the effective day-to-day use of technology for learning (e.g. from email to essay writing) � and it proficiency for assessment, reflected in the capability to use unfamiliar technology under time pressure in computer-based exams. in other words, our research suggests that we need to go beyond equipping students with basic it skills and familiarising them with the assessment environment and provide them with opportunities to develop proficiency in typing under time pressure, including use of unfamiliar hardware and software under authentic examination conditions. this might be achieved through the creation of computer-based formative assessments which align with the format of the summative examination. r. walker and z. handley 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 exam technique: test-taking strategies for online examinations another important determinant of performance in examinations which students identified was effective test-taking strategies. like the students in hong, sas, and sas’ (2006) investigation of the test-taking strategies of high school students in a paperbased mathematics examination and walker, topping, and rodrigues’ (2008) investigation of firstand second-year university students’ expectations and perceptions of a computer-based science examination, the students in our study attempted to deploy a range of organisational (e.g. time management and sequencing) and cognitive (e.g. checking, eliminating and using memory aids) strategies. transferring strategies developed for paper-based examinations to the computer-based examination was, however, not always straightforward for our students. for example, difficulties with time management appeared to be associated with the fact that questions were presented one at a time and students were not provided with an overview of the exam content: we can’t see the exam as a whole. reading the exam as a whole by clicking may waste time. (2014�2015, pre-test questionnaire) in terms of time management, i think we, when we are doing the handwriting exam, i know what questions i have, but in e-exam i just didn’t know what i am currently facing and i don’t know what kind of questions, you know closed or open question is coming next. (2014�2015, post-test focus group) suggestions provided by the students to address this and other issues raised here are discussed in the next section which focuses on assessment design. the possibility to annotate the examination and externalise their knowledge of topics were among the most frequently noted cognitive strategies that students expected to be able to apply in the online environment: i do not like not having the ability to circle questions i am unsure about or make notes to myself about which questions to come back to. during written assessments, i often write all over my test questions with arrows, circles, and other brainstorming sketches and it is difficult to work through the online assessment without these techniques. (2014�2015 pre-test questionnaire) similarly, some students felt that the lack of annotation facilities made it difficult for them to check their work: easy-go-back and check approach. don’t just provide question numbers and the function of flagging as in some online exams. provide key terms of the question for later reminding or shown only in checking process. students could even have a chance to make notes for reminding. (2014�2015 pre-test questionnaire) as the students suggest, some of the above concerns about the appropriation of test-taking strategies might be addressed through modifications to the design of the test and the platform on which it is delivered. however, it is also clear from the feedback that one assessment design will not fit all students’ test-taking strategies. as recommended by hillier (2014) and zakrzewski and steven (2003), the best solution may be to provide students with opportunities to familiarise themselves with the computer-based assessment environment and adjust the test-taking strategies research in learning technology citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 that they have developed in paper-based contexts to suit the computer-based environment: more practice to help the students be familiar with the system as well as question type. (2014�2015 pre-test questionnaire) assessment design and interface assessment design our research � like much before it � suggests that appropriately pitching a multiplechoice examination requires careful planning and iterative cycles of question review (crisp and palmer 2007; haladyna 1999). analyses of students’ performance on the 2013�2014 examination suggested that the balance of open-response to multiple-choice questions was not appropriate. despite careful review of the 2012�2013 paper version of the examination including content and cognitive behaviour reviews and subsequent revisions, students’ marks on the 2013�2014 were skewed towards the lower end of the university scale. the automatic difficulty and discrimination analyses generated by blackboard learn’s assessment engine suggested that students found the open-response questions particularly difficult. the balance between open-response and multiplechoice questions was therefore adjusted for 2014�2015. careful planning and reviewing of questions before administration and annual reviews including difficulty and discrimination analyses are therefore recommended until an appropriate and stable pitch is achieved. randomisation of the order of the questions was also necessary to prevent cheating because there was only one pc lab at the university equipped with privacy screens. whilst some students were positive about this aspect of the design of the assessment, others expressed concerns about the equity of randomisation: random questions for each student don’t represent the level of difficulties, for some students could encounter long answer question at q1 which gives little confidence of students to move on. more it could also waste time in trying to answer that question and therefore time is not enough. (2013�2014 post-test questionnaire) furthermore, randomisation caused distraction and appeared to raise some students’ levels of anxiety � the noise associated with typing meant that students were aware of the progress their neighbours were making with the questions: maybe the first three questions should all be typing, maybe, so all the people are typing maybe, so they were not just distracted. (post-test focus group 2014�2015) . . . i will tidy so much words but other students they finish one by one and very very quickly because the person needs a multiple choice item and i was ‘oh my god’ maybe they have written number 7 or number question, but i still do the first. (2014�2015 posttest focus group) ideally, all pc labs used for examination purposes should be equipped with privacy screens between work stations. this, however, may not be feasible in ‘greenfield’ e-assessment sites like our own where dedicated computer-based testing venues have not been established on campus. where this is not feasible and randomisation of questions is necessary to prevent cheating, consistency in the presentation of questions to students is essential to promote fairness, avoid distractions and avoid increasing r. walker and z. handley 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 students’ levels of anxiety. furthermore, consideration of possible test-taking strategies ought to be taken into account. it is, however, acknowledged that students may have individual preferences with respect to test-taking strategies and the presentation of the examination. for example: it’s no sense to put an open question for ten points at the beginning, so because our brain doesn’t work at the beginning to write/type so much. (post-test focus group 2014�2015) it is unlikely to be possible to design the examination to accommodate all students’ preferences and therefore essential that the assessment platform is flexible and students have ample opportunities for ‘practice’ to develop dedicated test-taking strategies. such opportunities for ‘practice’ might be provided through formative assessments. design of assessment interface our research also suggests that the design of the assessment interface requires careful consideration in order to accommodate students’ varying test-taking strategies. as discussed previously, students reported using a range of organisational and cognitive strategies. in relation to organisational strategies, in order to help students get an overview of the examination and allocate time accordingly, examinees wanted to be able to see all the questions on one screen first � a protocol which has been reported on in previous studies (e.g. frein 2011): maybe one page with all the questions on the first page and then one by one. (post-test focus group 2014�2015) as exemplified above, there was no consensus on item presentation thereafter. leeson’s (2006) summary of the research literature, however, suggests that multiple items on screen may have a facilitating effect in allowing examinees to skip, scan and build off previous item information and may counter the effects of single-item presentation on screen, which may encourage hurried responses and increase errors. easy navigation of the examination is also essential, given that it is unlikely to be possible to accommodate students’ differing preferences for the sequencing of questions. to support navigation, students made a number of suggestions including tagging questions with keywords and grouping them into categories: easy-go-back and check approach. don’t just provide question numbers and the function of flagging as in some online exams. provide key tests of the question for later reminding or shown only in checking process. students could even have the chance to make notes for reminding. (2014�2015 pre-test questionnaire) yes, also, i don’t think it’s necessary to just complete one question one time because maybe two or three that’s ok, because one and you have to click, click, click. how about one hundred questions . . . categorise the options, which one appears here to go back and which one to out of that, and which to submit. (2014�2015 post-test focus group) preparation and management of assessment centres our research also has implications for the preparation of multi-purpose pc labs which are to be used as assessment centres. feedback suggests that invigilators should research in learning technology citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 ensure all technical issues have been resolved before students enter assessment centres: it is better not to let the students enter into the it room until confirmed all the computers work as well.(2014�2015 post-test questionnaire) students also highlighted the importance of ensuring equity in conditions across assessment centres. noise levels were a particular concern as in previous research (fluck 2013; hillier 2014), and our data suggest that the impact on noise levels of the size of assessment centres as well as the quality of keyboards ought to be considered: there may be too many people at one computer room which leads the noise of typing (2014�2015 post-test questionnaire) keyboard in [named] college is much quieter than in library which minimises the noise during the exam. (2014�2015 post-test questionnaire) conclusions in drawing conclusions from this study, we should acknowledge the limitations of the research evidence that has been generated. the cohorts under review predominantly comprised female chinese students with limited prior exposure to e-assessment practices in uk higher education. moreover, they comprised international master’s students who had only been at the university for 3 months prior to the examination. as a result, the opportunities for tutors to normalise online assessment procedures prior to the delivery of the research methods module were considerably limited. we acknowledge the distinctive features of this study, which may impact the generalisability of the findings and recognise that the assessment context for other institutions and cohorts will differ greatly. notably, the findings are likely to differ from those for undergraduate students on multi-year programmes of study for whom there are likely to be greater opportunities to embed and normalise e-assessment practices. notwithstanding these limitations, the study offers an insight into the factors influencing postgraduate students’ reception of computer-based testing when encountering these methods for the first time. building on the existing literature (dermo 2009; hillier 2014), our research suggests that the issues influencing students’ acceptance of e-assessment methods can be grouped into three key categories focusing on: (1) socialisation of learners to the assessment method, (2) preparation of students for the computer-based assessment and (3) assessment design and infrastructure. of the factors identified, we observed that the socialisation of learners is a key step in the process of learner engagement and that it requires an upfront investment of time to explain to students the rationale and value of computer-based assessment methods, particularly when it involves a departure from established assessment practices. this appears to be particularly important for mature students and those returning to education who may not have been previously exposed to e-assessment methods and may share heightened concerns over the equity and reliability of computer-based testing. furthermore, we observed that students will need adequate preparation to negotiate the transition from formative to summative computer-based assessments. specifically, our findings suggest that the provision of opportunities for students to familiarise themselves with the exam format under authentic conditions (i.e. timed conditions within the test environment) is as important as orientation to r. walker and z. handley 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 the question types that students will encounter in the exam and outweighs issues of digital literacy and keyboarding skills. attention should instead be directed to fostering dedicated it skills for assessment, and in particular ‘online exam craft’. this will require students to adapt rather than directly transfer paper-based exam techniques to the online context, addressing both organisational and cognitive strategies. references crisp, g. & palmer, e. (2007) ‘engaging academics with a simplified analysis of their multiplechoice question (mcq) assessment results’, journal of university teaching and learning practice, vol. 4, no. 2, pp. 88�106. dermo, j. (2009) ‘e-assessment and the student learning experience: a survey of student perceptions of e-assessment’, british journal of educational technology, vol. 40, no. 2, pp. 203�214. deutsch, t., et al., (2012) ‘implementing computer-based assessment � a web-based mock examination changes attitudes’, computers and education, vol. 58, no. 4, pp. 1068�1075. durndell, a. & lightbody, p. (1993) ‘gender and computing: change over time? computers and education, vol. 21, pp. 331�336. ferrao, m. (2010) ‘e-assessment within the bologna paradigm: evidence from portugal’’, assessment and evaluation in higher education, vol. 35, no. 7, pp. 819�830. fluck, a. (2013) ‘implementation of on-campus digital examination practices’, transforming assessment in a digital era, racv city club, melbourne, [online] available at: http:// www.eexams.org/stories/documents/ta in a digital era fluck.pdf frein, s. t. (2011) ‘comparing in-class and out-of-class computer-based tests to traditional paper-and-pencil tests in introductory psychology courses’, teaching of psychology, vol. 38, pp. 282�287. haladyna, t. (1999) developing and validating multiple-choice test item, lawrence erlbaum, mahwah, nj. hillier, m. (2014) ‘the very idea of e-exams: student (pre) conceptions’, proceedings ascilite 2014, dunedin, new zealand, pp. 77�88, [online] available at: http://ascilite.org/ conferences/dunedin2014/files/fullpapers/91-hillier.pdf hong, e., sas, m. & sas, j. c. (2006) ‘test-taking strategies of high and low mathematics achievers’, the journal of educational research, vol. 99, no. 3, pp. 144�155. hsieh, h.-f. & shannon, s. e. (2005) ‘three approaches to qualitative content analysis’, qualitative health research, vol. 15, no. 9, pp. 1277�1288. jenkins, m., walker, r. & voce, j. (2014) ‘achieving flexibility? the rhetoric and reality of the role of learning technologies in uk higher education’, proceedings ascilite 2014, dunedin, new zealand, pp. 544�548, [online] available at: http://www.ascilite.org/conferences/ dunedin2014/files/concisepapers/161-jenkins.pdf jisc. (2007) effective practice with e-assessment: an overview of technologies, policies and practices in further and higher education, [online] available at: http://www.jisc.ac.uk/ media/documents/themes/elearning/effpraceassess.pdf jones, c., et al., (2010) ‘net generation or digital natives: is there a distinct new generation entering university? computers and education, vol. 54, no. 3, pp. 722�732. kingston, n. m. (2008) ‘comparability of computerand paperadministered multiple-choice tests for k-12 populations: a synthesis’, applied measurement in education, vol. 22, no. 1, pp. 22�37. leeson, h. v. (2006) ‘the mode effect: a literature review of human and technological issues in computerized testing’, international journal of testing, vol. 6, no. 1, pp. 1�24. margaryan, a., littlejohn, a. & vojt, g. (2011) ‘are digital natives a myth or reality? university students’ use of digital technologies’, computers and education, vol. 56, pp. 429�440. mead, a. d. & drasgow, f. (1993) ‘equivalence of computerized and paper and pencil cognitive ability tests: a meta-analysis’, psychological bulletin, vol. 114, pp. 449�458. meier, s. t. (1985) ‘computer aversion’, computers in human behavior, vol. 1, pp. 171�179. research in learning technology citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 13 (page number not for citation purpose) http://www.eexams.org/stories/documents/ta%20in%20a%20digital%20era%20-%20fluck.pdf http://www.eexams.org/stories/documents/ta%20in%20a%20digital%20era%20-%20fluck.pdf http://ascilite.org/conferences/dunedin2014/files/fullpapers/91-hillier.pdf http://ascilite.org/conferences/dunedin2014/files/fullpapers/91-hillier.pdf http://www.ascilite.org/conferences/dunedin2014/files/concisepapers/161-jenkins.pdf http://www.ascilite.org/conferences/dunedin2014/files/concisepapers/161-jenkins.pdf http://www.jisc.ac.uk/media/documents/themes/elearning/effpraceassess.pdf http://www.jisc.ac.uk/media/documents/themes/elearning/effpraceassess.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 meier, s. t. (1988) ‘predicting individual differences in performance on computer-administered tests and tasks: development of the computer aversion scale’, computers in human behavior, vol. 4, pp. 175�187. mogey, n. & sarab, g. (2006) ‘exams, essays and tablet computers � trying to make the pill more palatable’, proceedings of the 10th international computer assisted assessment conference, loughborough, pp. 313�320, [online] available at: https://dspace.lboro.ac.uk/ dspace-jspui/bitstream/2134/4414/1/mogey_n_sarab_g_h2.pdf noubandegani, p. (2012) ‘students’ perceptions of computerized toefl test’, language testing in asia, 2’, no., vol. 2, pp. 73�101. ozden, m. y., erturk, i. & sanli, r. (2004) ‘students’ perceptions of online assessment: a case study’’, journal of distance education, vol. 19, no. 2, pp. 77�92. palfrey, j. & gasser, u. (2008) born digital: understanding the first generation of digital natives, basic books, new york. prensky, m. (2001a) ‘digital natives, digital immigrants’’, on the horizon, vol. 9, no. 5, pp. 1�6. prensky, m. (2001b) ‘digital natives, digital immigrants part ii: do they really think differently? on the horizon, vol. 9, no. 6, pp. 1�6. tapscott, d. (2008) grown up digital: how the net generation is changing your world, mcgraw-hill, new york. walker, d. j., topping, k. & rodrigues, s. (2008) ‘student reflections on formative e-assessment: expectations and perceptions’, learning’, media and technology, vol. 33, no. 3, pp. 221�234. walker, r., et al., (2016) 2016 survey of technology enhanced learning for higher education in the uk, ucisa report, [online] available at: https://www.ucisa.ac.uk/bestpractice/ surveys/tel/tel williams, j. b. & wong, a. (2009) ‘the efficacy of final examinations: a comparative study of closed-book, invigilated exams and open-book, open-web exams’, british journal of educational technology, vol. 40, no. 2, pp. 227�236. zakrzewski, s. & steven, c. (2003) ‘computer-based assessment: quality assurance issues, the hub of the wheel’, assessment and evaluation in higher education, vol. 28, no. 6, pp. 609�662. r. walker and z. handley 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30083 http://dx.doi.org/10.3402/rlt.v24.30083 https://dspace.lboro.ac.uk/dspace-jspui/bitstream/2134/4414/1/mogey_n_sarab_g_h2.pdf https://dspace.lboro.ac.uk/dspace-jspui/bitstream/2134/4414/1/mogey_n_sarab_g_h2.pdf https://www.ucisa.ac.uk/bestpractice/surveys/tel/tel https://www.ucisa.ac.uk/bestpractice/surveys/tel/tel http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083 http://dx.doi.org/10.3402/rlt.v24.30083 (page number not for citation purpose) 1 *corresponding author. email: sadala@utu.fi research in learning technology 2019. © 2019 s. laato et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 research in learning technology vol. 27, 2019 original research article affordances of music composing software for learning mathematics at primary schools samuli laatoa*, teemu laineb and erkki sutinena adepartment of future technologies, university of turku, turku, finland; bdepartment of computer science, electrical and space engineering, luleå university of technology, skellefteå, sweden (received: 16 april 2019; revised: 19 june 2019; accepted: 5 august 2019; published: 5 september 2019) music composing is associated with various positive learning outcomes, but in several countries, such as finland, it is not part of the primary school music curriculum. there are several issues as to why music composing is not taught at schools, such as beliefs that composing requires extensive knowledge of music theory, lack of teachers’ confidence, lack of evidence on the method’s effectiveness and difficulty of assessment. composing software has the potential of solving some of these issues, as they are connected to mathematics via music theory and technology, and with practical opportunities arising from adopting phenomenon-based learning at schools, the affordances of music composing technologies for learning mathematics are investigated in this study. for this purpose, 57 music composing software were categorised and reviewed. our analysis identified eight types of music visualisations and five types of note input methods. the music visualisations were compared to the mathematics content in the finnish primary school curriculum and the note input methods were evaluated based on their relationship to the music visualisations. the coordinate grid-based piano roll was the most common visualisation and the tracker visualisation had the most affordances for learning primary school math. music composing software were found to have affordances for teaching mathematical concepts, notations and basic calculus skills, among others. composing methods involving direct interaction with visualisations support the experiential learning of music theory, and consequently, the learning of mathematics. based on the findings of this study, we concluded that music composing is a promising activity through which mathematics and music theory can be learned at primary schools. keywords: music technology; composing; mathematics; primary education; software review introduction music in general has been found to, for example, reduce stress and anxiety (nilsson 2008), support literacy development (paquette and rieg 2008), enhance motivation to exercise (edworthy and waring 2006) and affect feelings (habibi and damasio 2014), which have holistic positive effects on a persons’ life (thompson 2015). music compositions are the kind of permanent creations that give a person a sense of mailto:sadala@utu.fi http://www.alt.ac.uk/� http://creativecommons.org/licenses/by/4.0/� http://dx.doi.org/10.25304/rlt.v27.2259� s. laato et al. 2 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) self-accomplishment (runco, ebersole, and mraz 1991). just as playing an instrument creatively or painting a picture can be forms of self-expression, so can a musical composition. creating music and listening to music spark positive emotions, which consequently lead to increased motivation and more effective learning (sylwester 1995). music has also been a medium for conveying ideas and influencing people and cultures. it is an integral and undeniable component of being a human, and it is thus no wonder that music is being taught at finnish (finnish ministry of education 2014) and british primary schools (uk government 2013) and elsewhere throughout the entire primary school curriculum. primary school music education covers various aspects of music from playing instruments and collaborative music making to understanding music-related culture. however, the aspect of music which is seldom covered is music composing (byo 1999; strand 2016). phenomenon-based learning was introduced to the finnish national educational curriculum in 2014 (lonka et al. 2018; symeonidis and schwarz 2016). phenomenon-based learning focuses on teaching traditional subjects and curriculum content via phenomenon, for example, mathematics and music via music composing, instead of teaching subjects separately (lonka et al. 2018; mattila and silander 2015). several different subjects can thus be learned during a single lesson (mattila and silander 2015). phenomenon-based education calls for novel teaching methods. a promising area is music composing, which naturally blends together mathematics, music and logical thinking, and, when digitised, information technology (it). the possibilities of integrating music with mathematics have already been explored in previous studies (an and capraro 2011; an and tillman 2015; bamberger 2003; lim, lee, and ke 2018; tossavainen and juvonen 2015). in addition, skills in problem-solving and it as well as scientific thinking have also been taught through music composing (berkley 2014; manaris and kohn 2016; rogers 2016; ruokonen and ruismäki 2016). figure 1 illustrates how music composing can be seen as a phenomenon through which it skills, mathematics and music can be learned. the current study focuses on the intersection of music and mathematics education, and the affordances music composing technologies have for combining the two subjects in primary school education. a related study was conducted in 2005, where figure 1. mathematics, it and music education can all be combined through the phenomenon of digital music composing. http://dx.doi.org/10.25304/rlt.v27.2259� research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 3 (page number not for citation purpose) the multimodal affordances of technology for teaching music composing were investigated (gall and breeze 2005). the study utilised empirical work with children as a research method, and came up with promising results, highlighting the affordances composing technologies offer for education (gall and breeze 2005). however, instead of general affordances of software for music education, the current study focuses on the affordances of the composing software for mathematics education, as the musical benefits of composing are such an integral part of the composing process that they can be assumed to be present in all solutions. mathematics, on the other contrary, manifests differently depending on the type of composing activity and technology used (mazzola, mannone, and pang 2016). in the realm of human–computer interaction (hci), the term affordance has been reported to have some ambiguity, with two definitions circulating in research, one by the original discoverer of the term james gibson, and the other by psychologist donald norman, who popularised the term in the realm of hci (mcgrenere and ho 2000; norman 1988). the so-called norman’s affordances are defined to be perceived properties of an object that may or may not exist, which can be dependent on the experience, knowledge or culture of the actor (mcgrenere and ho 2000). in comparison, in gibson’s affordances, the properties of an object exist independently of the observers’ ability to identify them (greeno 1994). in the current study, norman’s definition was adopted, as the potential and the supporting properties music composing technologies have on mathematics and music education are observed and analysed. moreover, previous studies on the affordances of music composing technologies have also used norman’s definition (gall and breeze 2005). for clarification, the current study makes a difference between composing music, as opposed to creating sounds, playing music and listening to music. obviously, these four are all intertwined in music creation, and for most people effective composing is impossible without, for example, being able to hear what the piece in progress sounds like. approaches to composing vary from soundscape composing (fourney and fels 2009; martin 2018) and context-dependent composing (koutsomichalis 2018) to theory-based composing (bunting 1987), with each having their own artistic effects. in the current setting, the traditional music theory-based composing is explored. visualising compositions with the help of music theory provides the most suitable platform for mathematical abstraction, and hence the best tools for learning logic and mathematics in comparison to alternative composing techniques. the term “music composing software” is in some contexts used to describe algorithmic composing tools, but in this study, the term is used explicitly for describing software that aids the user in composing music, but does not generate music on its own. background music composing has historically not been taught to others than exceptionally gifted children, at least partly due to the presumption that it is an activity that requires extensive knowledge of music theory, mastery of western music notation (sometimes also referred to as sheet music or simply musical notation [smith and williams 1997]) and sufficient skill on at least one instrument (beckstead 2001; pope et al. 1995). modern music technology removes the two later arguments, as mastery of western music notation and mastery of an instrument can now be replaced by mastery of composing software. the argument for the need for knowledge of music theory still http://dx.doi.org/10.25304/rlt.v27.2259� s. laato et al. 4 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) stands; however, a composing software can be seen as an object through which learning of music theory can happen. it can, however, be argued that instead of seeing music theory as a requirement to start composing, music theory can be learned by composing. for example, piaget’s theory of cognitive development argues for learning through personal experiences, and heavily emphasises the importance of inventing things as a way of learning compared to being thought the things directly (piaget 1970). other theories including empirical modelling (beynon 2012) and experiential learning, focusing on learning from experiences (kolb 2014), echo piaget’s theory, and in the spirit of radical empiricism (james 1976) argue for learning through personal experiences. it is therefore reasonable to presume that with sufficient tools to learn with, that is, construals at disposal (beynon 2012), music theory can be learned through composing music, instead of seeing quite vast music theory knowledge as a prerequisite for starting to compose music. music composing is not currently part of the finnish educational curriculum (finnish ministry of education 2014). in some other countries, such as the united kingdom, composing is being taught as part of music education, and it is embedded in music education in the national educational primary school curriculum (uk government 2013). however, how music is being taught in practice at primary schools heavily depends on the music teacher. composing has been perceived to be the most difficult aspect of music to be effectively taught in a primary school setting (byo 1999), and techniques and tools that are available for teaching it vary greatly. the raw numbers of to what degree music composing is being taught at primary school level have been measured by some studies. for example, in the state of indiana where composing is not a compulsory part of the primary school music education (indiana state 2016), 89% of educators said they sometimes still use some sort of music composing tasks in their teaching. however, only 6% of indiana teachers reported to use composing activities regularly as part of their teaching (strand 2016). challenges of music composing as a classroom activity composing as a classroom activity has been explored in several studies (i.e. macgill 1988; pitts 2000; sætre 2011; savage 2005; selwyn 1993; strand 2016; winters 2012), but it has not yet established itself as a part of the finnish primary school music education curriculum (finnish ministry of education 2014). studies focusing on technology as a solution for helping the teaching of music composing argue for its usefulness (beckstead 2011; finney and burnard 2010), but overall research on practical technological solutions for classroom composing education is lacking. one of the earliest studies of composing technologies for classroom use is peters’ article from 1992 where he lists 27 unique music software from four development generations (peters 1992). the study shows how the software improve with each generation and are better suited for educational purposes. sætre (2011), on the contrary, highlighted the role of the teacher as the most important factor in teaching music composing instead of technology. general reasons found in the literature of why music composing is often not thought at primary and secondary school level include teachers’ lack of confidence in their abilities to teach music (atkinson 2017; winters 2012), the lack of evidence that composing as an activity produces useful tangible learning outcomes http://dx.doi.org/10.25304/rlt.v27.2259� research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 5 (page number not for citation purpose) compared to other teaching methods (savage 2005), the difficulty to assess compositions and the learning outcomes (savage 2005), and the lack of practical technologies, or educators’ knowledge of such technologies, that would allow students to focus on their composing during class in a limited amount of time (winters 2012). a more practical unproven hypothetical reason for not including composing music as part of the primary school music education is a historical reason, more specifically the lack of technology. without the help from technology all practical composing activities would either be too difficult or produce too much sound and noise to be carried out in a standard classroom setting. the reasons why composing music is not being taught in primary and secondary schools can thus be summarised as follows: • composing music is only seen as something that gifted musicians who know music theory are capable of. • music teachers lack the confidence to teach music composing. • lack of evidence that composing music is an effective teaching method for music theory and mathematics in comparison to alternative methods. • the difficulty to assess music compositions. • a composer needs to be able to hear what the composition sounds like, and without proper software and headphones this would result in cacophony in a classroom setting. the role of technology in music composing the rise and development of digital audio processing has had several positive effects on music composing. midi technology enables songwriters to immediately hear what even the most complex compositions sound like, which has had a great impact on music education and composing (beery 1995). software can create sheet music straight from playing a midi keyboard using automatic melody dictation (cambouropoulos 2000), which makes the process of composing fluent. in addition, with technology, revisions and rewrites to the composition are fast to do, and, for example, the entire order of sections in a song can be quickly turned around (rabiner and gold 1975). converting analogical instruments to digital representations for editing is a craft, that is still being developed, but some software is already quite good at it (melodyne 2019). algorithmic composing refers to music composing procedures that utilise software for automating the song writing process (cope 2015). algorithmic composing has become possible through software that, for example, automatically generates bass lines or harmonies to an existing melody (beckstead 2001). these kinds of composing software can differ vastly from each other, and consequently, the programme authors can be seen as the composers. machine learning techniques also allow for previously unobtainable ways to compose music. for example, a study by malmi et al. (2016) demonstrates how rap lyrics can be generated automatically through a machine learning powered programme. cope (2015) argues that using algorithms or at least some level of automation in composing is natural, and that  algorithms simply serve the purpose of informing someone how to proceed (with music). http://dx.doi.org/10.25304/rlt.v27.2259� s. laato et al. 6 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) visualisation of music can guide the user to think about music in a certain way, and it has the potential of making music more memorable, even assisting in understanding it (foote 1999; fourney and fels 2009; hiraga, watanabe, and fujishiro 2002). for a composer and for a learner, the way to visualise music is equally important. technology allows for previously unobtainable or completely impractical visualisations, for example, visualisations that require constant movement or animation (hope 2017; miller et al. 2018). furthermore, technology allows the switching between various visual representations of a single musical score, making it easier to establish a connection between the music and its potentially endless visual representations. for a composer this can be helpful, as different visualisations draw out and highlight different parts of the musical score. the influence of technology on music education at primary and secondary school level has been discussed in several studies (bauer, reese, and mcallister 2003; beckstead 2001; finney and burnard 2010), which provide examples of how various hardware and software can be used in a classroom setting and outside. technology has now evolved to a point (holmes and holmes 2002; taylor 2014) where it allows composing to be integrated into music education in previously unattainable ways (gall and breeze 2015; pitts and kwami 2002). new technologies and ideas for music composing emerge constantly, a few recent examples being multiplayer music making (wejam 2018), a distributed programmable computer music system (shapiro et al. 2017) and a mathematical composing tools with gamified elements (laato et al. 2017; lim, lee, and ke 2017). even early studies where students composed music using midi technology show promising results (airy and parr 2001). recently also serious games for learning music have been proposed (hendradjaya 2018). research design to address the discussed research problem of what affordances current music composing software offer for mathematics education, the following research question was formulated: what kinds of music visualizations and note input methods exist in contemporary music composing software, and what affordances do these have for teaching mathematics and music theory in late primary and early secondary school? the age groups of 10–15 years were chosen for this study because mathematics in music composing should be introduced as early on as possible, but students should be old enough to have the required skills and capabilities to get started with composing. to find the optimal target audience, the curricula of finland (finnish ministry of education 2014) and the united kingdom (uk government 2013) were analysed and based on the mathematics and music contents in the existing curricula, the target age of 10 years was determined to be an absolute minimum requirement. the semantic fields of music notation and the visual representation of music also overlap; however, the visual representation of music can be more than just notation (hope 2017). several notations exist, for example, the guido notation format (hoos et al. 1998), the graphics colour music notation (holcombe 2016) and the haskore music notation (hudak 1996), but this study http://dx.doi.org/10.25304/rlt.v27.2259� research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 7 (page number not for citation purpose) aims to look beyond the notations to the concrete visualisations present in available composing software. the research design of this study has two stages, which are explained in detail in the following sections: 1. systematic search of music composing software 2. analysis • identification of the genre of the composing software based on existing classification. • analysis of the composing software in each genre to identify all unique visualisations and note input methods. • going through the mathematics education curriculum of finland for the chosen age groups, and finding mathematics concepts that could be supported by identified music visualisations. • determine which kinds of affordances each note input method provides for education by looking at the magnitude of the connection to the abstraction of the music. search of music composing software the prisma literature review method by moher et al. (2009) was adopted and translated to the realm of software to conduct a systematic search for composing applications. a composing software was defined as a computer application with which users can create original music and listen to it. for the initial collection of composing software, existing lists of composing software, research databases and prominent available search engines were used. existing non-scholarly lists of composing software were also used, for example, wikipedia’s list of composing software (2018). the covered research databases were the springer database, ieee xplore, scitepress and the research database search engine google scholar. the search engines used were duckduckgo and google. for making queries in the databases and search engines, the search terms “music composing software” and “composing software” were used, and the first 30 results were checked. the list of software was also supplemented with information found on news articles and forums. the platform was narrowed to desktop only, as the composing software of which available studies were found were primarily made for desktop operating systems, for example, work by pejrolo (2012), farbood et al. (2004) and mccoid et al. (2013). a certain quality consideration had to be made when choosing which software to include in the final data set. the software had to either be available on latest desktop operating systems, or academic research had to be published of it. remixing software, non-musical audio editing tools and other software which did not allow the user to freely composer music were excluded. thus, the following items were excluded: 1. items that did not fit the chosen definition of a composing software. 2. items that were never published or made available to the public. 3. items that were only available for mobile platforms (android/ios/windows phone). http://dx.doi.org/10.25304/rlt.v27.2259� s. laato et al. 8 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) identifying the genre of the composing software firstly, the software was sorted into genres based on the existing literature. the following categories were used: digital audio workstation (daw) (leider 2004), tracker (obarski 1987), midi sequencer, notation software, educational software, game, beat machine and sound editing software. this way of categorising composing software is ubiquitously present in music software discussions and studies (leider 2004; peters 1992; walzer 2016). the genre was determined by the features of the software, and found and determined by looking at external sources, aka manuals, websites and marketing material related to the software. most of the time the genre was defined by the developers themselves, but in some cases the features of the software had to be looked to determine the genre. categories for music visualisations and note input methods from each genre, all unique music visualisations and note input methods were recorded. the categories were formed by observing the logic behind the solutions. for each unique logic, a new category was created. music visualisations and note input methods are sometimes put into a bundle and called an editing interface or just an interface of a music editing tool (marrington 2010). in the current study, the two were separated, as it is evident that several programmes offer the possibility to visualise the same song in multiple ways, and also offer a variety of ways to input notes (hosken 2014). affordances of the visualisations for education the mathematics content in national educational curricula of the united kingdom (uk government 2013) and finland (finnish ministry of education 2014) was observed for the chosen target age groups of 10–15 years. the found areas of mathematics were compared to the music visualisations in the composing software. there were two ways the visualisations could support the mathematics content: direct and indirect. in case the visualisation contained direct references to specific mathematics content, for example, geometric shapes, it was recorded to have the affordance for teaching geometry. however, the visualisation could also indirectly support the learning of some mathematics content, by, for example, containing a logical structure, which taught processes needed in mathematics. an example of this would be visualisations that rely on an xand y-axis to represent aspects of the music. these visualisations might not directly show the coordinate grid, but essentially working which such visualisations would still have the affordance of teaching how a coordinate grid functions (gall and breeze 2005). affordances of note input methods for education in addition to visualisations, note input methods were looked at. based on the theories of cognitive development (piaget 1970), experiential learning (kolb 2014) and empirical modelling (beynon 2012), it was assumed that the more direct correlation between the input and the feedback, the better the learner is able to grasp the connection between their actions and the output. it was also presumed that in case a visualisation was found to have the affordance of teaching mathematics, interacting directly http://dx.doi.org/10.25304/rlt.v27.2259� research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 9 (page number not for citation purpose) with the visualisation would boost the learning effect (kolb 2014). in comparison, note input methods that were not related to the visualisation, and that had a delayed feedback loop, were seen as suboptimal from the standpoint of effective immediate learning outcomes. results the initial list of programmes gathered through search engine results, research databases and supplemented with additional information comprised 115 programmes. after filtering out items that were not composing software as defined in this article, 57 programmes remained (see appendix a). before looking into the visualisations and note input methods present, the programmes were sorted into eight genres based on their features. the results of this categorisation are seen in figure 2. each software was sorted into the primary category it belongs to based on its features, meaning that each software is presented only once, even though some software might have features that could place them into several categories. software from each genre was then analysed to identify categories of unique visualisations and note input methods. many of the programmes contained multiple visualisations of music as well as several note input methods. for example, in programmes classified as daws, a certain visualisation and composing logic might be prominent or encouraged, but other options are almost always included (leider 2004; marrington 2010). the user was able to switch between various visualisations for the same composition, and also input notes in a multitude of ways. besides being the most popular genre of composing software, daws on average contained the most versatile selection of note input methods and music visualisations. the second most popular category, midi sequencers, could also be found embedded in daws. trackers were the third most popular in terms of quantity; however, lately they have had only minimal use in the industry (marquez 2014). figure 2. analysed software sorted into categories based on their features. http://dx.doi.org/10.25304/rlt.v27.2259� s. laato et al. 10 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) visualisations of music some of the software had different visualisations for single melodies and whole orchestras, but for visualisation of the three components of a note: pitch, duration and instrument, the following two categories with four subcategories were discovered: 1. coordinate grid-based visualisation: • piano roll • tablature • western music notation • waveform view 2. other visualisations • musicxml (code) • tracker visualisation • pattern/grid view • instrument emulations coordinate grid-based visualisations the most popular way of displaying music among the analysed programmes was found to be a coordinate-grid based visualisation, a two-dimensional (2d) coordinate system where the x-axis represents time and the y-axis represents note pitch. examples of this kind of a solution are piano roll and western music notation. as only the pitch and duration are determined by the coordinate grid, the blocks that are placed on the grid have metadata on them that determines one or more of the following: the velocity of the note, the instrument the note is played on and any pitch bends done or additional effects on the note. for example, the instrument is often not assigned to a single block, but rather the whole track, an example of which one can find in western music notation. in figure 3, the composing software mario paint composer is displayed, where the type of note determines the sound of the note. the song in question is bonetrousle aka papyrus theme from the game undertale by toby fox (fox 2015), arranged by youtuber “pokesonicddrninja/ deepfreeze757.” some coordinate grid-based solutions where the duration of the note was not determined by the coordinate grid at all were identified. this approach is mainly used in western music notation, which is featured in several composing software, for example, sibelius and guitar pro 7. figure 4 shows the mit laboratories composing software hyperscore (farbood, pasztor, and jennings 2004), which combines the approach of mario paint composer and western music notation, displaying at once both a unique looking note like in mario paint composer and western music notation, and then allowing the length of that unique looking note to be manipulated. a coordinate grid is also used to visualise patterns, verses and other bigger blocks of composed music in most analysed programmes. even though multiple solutions exist for displaying melodies and chords, the bigger blocks of music are almost exclusively presented in a coordinate grid, where the x-axis shows time and the y-axis shows all individual tracks. http://dx.doi.org/10.25304/rlt.v27.2259� research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 11 (page number not for citation purpose) figure 3. the visualisation of music in mario paint composer is based on western sheet music. figure 4. the visualisation of music in the hyperscore programme, developed by mit laboatories (farbood et al. 2004). http://dx.doi.org/10.25304/rlt.v27.2259� s. laato et al. 12 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) other visualisations a close relative to the coordinate grid visualisation of music is the so-called tracker visualisation. the name tracker comes from the composing software, created by karsten obarski in 1987, and called the ultimate soundtracker. the tracker visualisation features a discrete 2d matrix to visualise notes. the notes are written with plain text to the grid, for example, the f note in the 5th octave could be written as f-5. this kind of a 2d grid visualisation is not only present in trackers, but also seen in more recent software like the song maker (2018) by chrome experiments music laboratory. furthermore, the western music notation can be seen to be divided into grids, with bar lines as the dividing factor. virtual instruments in the analysed software could have their own visualisations when inputting notes, but after the notes are input, the visualisation shifts back to a coordinate grid. for example, audiotool and flstudio provide virtual beat creation synth user interfaces that have their own visual representations of the beat in a pattern/grid view, but once the beat is complete, it is presented as a single block in a higher level abstraction of the whole song. the tablature visualisation is identified to be an instrument-based visualisation, as it is based on the neck of a six-string guitar. the instrument-based visualisations are arguably more useful for those musicians who are familiar with a specific instrument, and want to utilise their knowledge of that instrument in composing. music can be looked at and edited in code, and of that musicxml is a good example. pitch, duration, velocity, instrument and so on are all defined in code using string, numbers, arrays and variables. however, this kind of visualisation is not prominent in almost any software. rather, the software only uses this kind of data structure underneath and provides a more user-friendly interface for the composer to work on. note input methods in the software and their affordances for education note input methods can be split into two categories: (1) internal methods where the user directly interacts with the visual representation of the music to manipulate it, and (2) external methods where the user uses another way to input a note, which then appears on screen. most daws and complicated software, for example, guitar pro, cakewalk sonar, fl studio, cubase and garageband, host a variety of note input methods (leider 2004; marrington 2010). the categories for note input methods found in the software were as follows: • interacting with or manipulating visualisations (internal) • playing a real instrument or singing (external) • playing a virtual instrument (external) • typing in notes or code (internal) • using ai to generate music (external) the first category, interacting with or manipulating visualisations, can occur in multiple ways. a note can be typed in, dragged and dropped, or it can be placed by touch controls, for example, drawn like in the western music notation. these types of note input methods are present in the majority of the software. the other direct or internal note input method category was typing in notes, which is present in, for example, when composing with the musicxml programming language or trackers http://dx.doi.org/10.25304/rlt.v27.2259� research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 13 (page number not for citation purpose) (obarski 1987). internal methods were found to be always dependent on a visualisation, and could not exist on their own. external methods, on the contrary, could be things like singing, playing a real or virtual instrument or generating music with the help of artificial intelligence. these categories represent note input methods that are not linked to a specific visualisation. in terms of affordances for mathematics education, it was concluded that based on the theories of cognitive development (piaget 1970) and empirical modelling (beynon 2012), directly linking action into empirical experiences and feedback strengthens the learning process. therefore, note input methods classified as internal offer better affordances for mathematics education and learning music theory, as they provide instant visual feedback to the composer, and are thus better tools for experiential learning (kolb 2014). external methods, such as playing a virtual instrument to input notes, require an already existing knowledge structure of music theory for the composer to be in total control of the composition. the external note input methods can, however, support understanding music as a whole, as the composer will learn to draw links between various abstractions of the same musical score. affordances of music visualisations for mathematics education the mathematics concepts and goals found in the finnish and british primary and secondary school curricula were almost identical. this is in line with reports from meyer, kamens and benavot (2017), who analysed a wide range of national educational curricula and found them to be, in most cases, similar with each other. the finnish national curriculum (finnish ministry of education 2014) lists 14 learning goals in mathematics for the age groups 9–12 and further 20 learning goals for the age groups 13–15. the uk national curriculum (uk government 2013) is not as precise, and leaves more room for teachers to adapt their teaching to their own and to their pupils’ preferences. therefore, the finnish educational curriculum was used in the comparison analysis between mathematics concepts in the educational curriculum and the visualisations found in the composing software. out of the 14 learning goals of the finnish national curriculum for age groups 9–12, the first six related to working skills, values and attitudes. similarly out of the 20 learning goals of the finnish national curriculum for age groups 13–15, the first nine are not actual mathematics concepts. out of the resulting 19 learning goals (8 for age groups 9–12 and 11 for age groups 13–15), five duplicates were removed. the final list of 14 (4 for age groups 9–12 and 10 for age groups 13–15) mathematics learning goals and visualisations that provide affordances for learning them can be seen in figure 5. surprisingly, the nowadays little-used tracker visualisation had the most affordances in terms of quantity: basic calculus skills, rational numbers, real numbers and mathematical concepts and notations. moreover, pure code, that is, musicxml also covered several categories. however, these results do not take into account the effectiveness of each solution in education, as, for example, gazing at xml notation is probably not as motivating for students as seeing say, a colourful piano roll representation. discussion and future work the popularity of coordinate grid-based visualisations might be the result of the following factors: (1) they are the most intuitive and/or (2) the most practical tools for http://dx.doi.org/10.25304/rlt.v27.2259� s. laato et al. 14 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) composers to work with. a coordinate system-based visualisation is in fact present in more than 90% of the analysed software. its popularity could be partly due to how natural language is written and read, in fact, western music notation is read from top to bottom and from left to right, just as western written natural languages (smith and williams 1997). professional musicians are capable of accurately playing sheet music they have never heard before (truitt et al. 1997), but with the amount of training required it is not clear as to whether the same feat could be achieved with the same or even less training with another music visualisation method. the coordinate grid visualisation guides towards thinking music in a certain logical framework, thus enabling the learner to form an intuitive understanding of music theory based on empirical experiences. for primary and secondary school students getting familiar with the number line, composing music with the coordinate grid visualisation seems a promising activity to support the learning. based on the theory of constructivism and the current understanding of the human brain, learners construct new knowledge on top of their prior knowledge. the process can happen via empirical experiences, but changing existing knowledge structures requires conceptualisation and reflection (troelstra and van dalen 2014; von glasersfeld 2013). hence, visualising music through ways that students are already familiar with can assist in getting started with composing. furthermore, familiar mathematics concepts encountered while composing serve the same purpose ( troelstra and van dalen 2014). once students obtain a grasp of the underlying music theory, it is easier to guide them to connect it into mathematics and other related fields. many daws, for example, guitar pro, offer multiple visualisations of music including several instrument emulations as well as guitar tablature and traditional sheet music. being able to see the same piece of music through different visualisations can be a powerful pedagogical tool for assisting students to draw connections between music and various visual abstractions of it. different visualisations spark different thoughts in the brain, resulting in more creative thinking and to birth of new musical ideas. motivating children for creative thinking is important for several reasons, one of them being that creation itself is a process that sparks motivation to study further to be able to create a better outcome for the purpose of self-actualisation (runco et al. 1991). soundwave visualisation and other visualisations where the pitch and/or rhythm cannot be deciphered just by looking seem impractical from an educational standpoint. this is because they rely heavily on listening and often lack the required visual support. these presentations are therefore unable to support the learner’s construction of a logical theoretical structure of music. soundwave visualisation is special in figure 5. which mathematics concepts does each music visualisation support? http://dx.doi.org/10.25304/rlt.v27.2259� research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 15 (page number not for citation purpose) the way that it shows the music in a lossless analogue format, meaning that it represents real sound as opposed to many other visualisations of music that are digital and symbolical in nature. this further supports the point that the most intuitive and effective visualisations of music are abstractions instead of accurate depictions. interaction and feedback are essential in forming an understanding of the relationships between a phenomenon and an action (kolb 2014). previously, it was established that the note input methods can be divided into two categories: one where the user directly interacts with the visualisation and one where they do not. overall, it can be argued that input methods that are directly linked to the visual representation of the music have higher potential to be pedagogically effective for novice composers. this is due to the fact that they get immediate feedback when meddling with the visualisation about how it now sounds, and they see the effects of their actions without a delay, something that is not achieved with other input methods. professional composers that are already familiar with music theory might prefer external input methods, but for novice composers of the target age group in the current study, the note input methods that are directly linked to the visualisation are pedagogically most promising. limitations of the study the methodology for finding existing composing software could be improved upon to include applications on mobile platforms. for example, a promising mobile game with unique note input methods and visualisations, yatatoy’s bandimal (yatatoy 2019) and a mathematical composing software harmony hippo (laato et al. 2017) were excluded due to the platform limitations. in addition, the indicators based on which the eight visualisations and five note input methods were determined could be tweaked or changed. for example, editing interfaces could have been observed as a whole instead of a separate analysis of note input methods and visualisations. this change might have led to some insights, which did not emerge in the current study. the chosen theoretical frameworks, from which the software evaluation methods were derived from, influenced the outcome. empirical modelling (beynon 2012) and experiential learning (kolb 2014) are both very much focused on learning from empirical experiences, and emphasise conceptualisation and reflection less. yet, several recent studies on effective learning of mathematics highlight the importance of deliberate practice (ericsson, krampe, and tesch-römer 1993), which includes thinking, analysing, reflection, conceptualising and problem-solving (bonneville-roussy and bouffard 2015; lehtinen et al. 2017). however, experiential learning does not automatically mean the lack of deliberate practice, and meaningful learning activities, such as learning mathematics by composing music, can foster an intrinsic motivation for deliberate practice (bonneville-roussy and bouffard 2015; kolb 2014). finally, due to the magnitude of the study and the chosen research method, the understanding of the affordances existing composing softwares have for mathematics education remains theoretical. in addition, the current study focused mainly on music composing technologies, and pedagogical models and theories for mathematics education were only briefly touched. a similar study could be conducted the other way around, by looking at pedagogical models for mathematics education, and studying what affordances they have for combining mathematics education with music. nevertheless, the current study lays the groundwork for investigating the identified affordances in further detail in the future. http://dx.doi.org/10.25304/rlt.v27.2259� s. laato et al. 16 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) conclusions several similarities were observed between many of the 57 software analysed in this article. the genre classification system used in the music industry and in previous studies proved to be quite accurate in determining which types of music visualisations and note input methods were present in the software. daws had on average the most features, supporting multiple visualisations and note input methods ( marrington 2010). altogether, eight different visualisations and five types of note input methods were identified in the software. there were essentially two types of note input methods, which were classified as internal and external. based on theories of cognitive development (piaget 1970) and empirical modelling (beynon 2012), the internal methods were more promising for mathematics education and the construction of the learners’ mathematical music theory knowledge structures. piano roll, a coordinate grid-based visualisation, was the most prominent visualisation of all, with some versions of it being featured in almost all software except trackers. it was found to have the affordance for teaching at least two mathematics concepts in the primary and secondary schools: (1) understanding and interpreting tables and diagrams, and (2) functions and how they can be plotted into graphs. in terms of most affordances for mathematics education, the tracker visualisation had four (see figure 5): (1) basic calculus skills, (2) rational numbers, (3) real numbers and (4) mathematical concepts and notations. the analysis did not go in depth to consider the magnitude and pedagogical effect of the affordance. in addition, it is worth noting that even if a visualisation does not support directly learning some of the associated mathematics concepts, those skills can still be learnt because of the mental processes that a composer must undergo while creating original music. some visualisations also had the affordance to teach some concepts that were not present in the analysed national educational curricula, for example, working with number systems other than the 10-based number system. the current study laid the foundation for future research on technologies aimed to teach mathematics through music composing. it gives a framework (visualisations and note input methods) for understanding the existing software and gives ideas for developing future software. the results of this study, which music visualisations and note put methods have affordances to teach mathematics, are general and need to be studied in further detail in future studies to fully understand their pedagogical potential. acknowledgements this project was funded by the turku university foundation with a 1 year grant for doctoral studies to the first author. we would also like to thank and acknowledge dr nick pope for proofreading our work and giving important feedback and improvement suggestions. references airy, s. & parr, j. m. (2001) ‘midi, music and me: students’ perspectives on composing with midi’, music education research, vol. 3, no. 1, pp. 41–49. https://doi.org/10.1080/14613800020029941 an, s. a. & capraro, m. m. (2011) music-math integrated activities for elementary and middle school students, education for all, irvine, ca. http://dx.doi.org/10.25304/rlt.v27.2259� https://doi.org/10.1080/14613800020029941 research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 17 (page number not for citation purpose) an, s. a. & tillman, d. a. (2015) ‘music activities as a meaningful context for teaching elementary students mathematics: a quasi-experiment time series design with random assigned control group’, european journal of science and mathematics education, vol. 3, no. 1, pp. 45–60. atkinson, r. (2017) ‘the pedagogy of primary music teaching: talking about not talking’, music education research, pp. 1–10. https://doi.org/10.1080/14613808.2017.1327946 bamberger, j. (2003) ‘music as embodied mathematics: a study of a mutually informing affinity’, international journal of computers for mathematical learning, vol. 8, no. 2, pp. 123–160. https://doi.org/10.1023/b:ijco.0000003872.84260.96 bauer, w. i., reese, s. & mcallister, p. a. (2003) ‘transforming music teaching via technology: the role of professional development’, journal of research in music education, vol. 51, no. 4, pp. 289–301. https://doi.org/10.2307/3345656 beckstead, d. (2001) ‘will technology transform music education? although technological advances make composing easier, music educators tend to use these tools to make traditional methods more accessible rather than explore new possibilities in composing’, music educators journal, vol. 87, no. 6, pp. 44–49. beery, l. s. (1995) ‘teaching midi from an african american angle: african american students can be taught the music of their cultural heritage with the use of midi’, music educators journal, vol. 82, no. 3, pp. 35–38. berkley, r. (2004) ‘teaching composing as creative problem solving: conceptualising composing pedagogy’, british journal of music education, vol. 21, no. 3, pp. 239–263. https://doi. org/10.1017/s026505170400587x beynon, m. (2012) ‘modelling with experience: construal and construction for software’, in  automation, collaboration, & e-services, pp. 197–228. https://doi.org/10.1007/ 978-3-642-25209-9_9 bonneville-roussy, a. & bouffard, t. (2015) ‘when quantity is not enough: disentangling the roles of practice time, self-regulation and deliberate practice in musical achievement’, psychology of music, vol. 43, no. 5, pp. 686–704. https://doi.org/10.1177/0305735614534910 bunting, r. (1987) ‘composing music: case studies in the teaching and learning process’, british journal of music education, vol. 4, no. 1, pp. 25–52. byo, s. j. (1999) ‘classroom teachers’ and music specialists’ perceived ability to implement the national standards for music education’, journal of research in music education, vol. 47, no. 2, pp. 111–123. cambouropoulos, e. (2000, july) ‘from midi to traditional musical notation’, in proceedings of the aaai workshop on artificial intelligence and music: towards formal models for composition, performance and analysis, vol. 30. cope, d. (2015) ‘algorithmic music composition’, in patterns of intuition, springer, dordrecht, pp. 405–416. https://doi.org/10.1007/978-94-017-9561-6_19 edworthy, j. & waring, h. (2006) ‘the effects of music tempo and loudness level on treadmill exercise’, ergonomics, vol. 49, no. 15, pp. 1597–1610. https://doi.org/10.1080/00140130600899104 ericsson, k. a., krampe, r. t. & tesch-römer, c. (1993) ’the role of deliberate practice  in  the acquisition of expert performance’, psychological review, vol. 100, no. 3, p. 363. farbood, m. m., pasztor, e. & jennings, k. (2004) ‘hyperscore: a graphical sketchpad for novice composers’, ieee computer graphics and applications, vol. 24, no. 1, pp. 50–54. https:// doi.org/10.1109/mcg.2004.1255809 finney, j. & burnard, p. (eds.). (2010) music education with digital technology, bloomsbury publishing. new york, usa. finnish ministry of education. (2014) national education curriculum for elementary school 2014, pp. 142, 263–264, 422–425, [online] available at: http://www.oph.fi/download/163777_ perusopetuksen_opetussuunnitelman_perusteet_2014.pdf foote, j. (1999) ‘visualizing music and audio using self-similarity’, in proceedings of the seventh acm international conference on multimedia (part 1), acm, pp. 77–80. http://dx.doi.org/10.25304/rlt.v27.2259� https://doi.org/10.1080/14613808.2017.1327946 https://doi.org/10.1023/b:ijco.0000003872.84260.96 https://doi.org/10.2307/3345656 https://doi.org/10.1017/s026505170400587x https://doi.org/10.1017/s026505170400587x https://doi.org/10.1007/​978-3-642-25209-9_9 https://doi.org/10.1007/​978-3-642-25209-9_9 https://doi.org/10.1177/0305735614534910 https://doi.org/10.1007/978-94-017-9561-6_19 https://doi.org/10.1080/00140130600899104 https://doi.org/10.1109/mcg.2004.1255809 https://doi.org/10.1109/mcg.2004.1255809 http://www.oph.fi/download/163777_perusopetuksen_opetussuunnitelman_perusteet_2014.pdf http://www.oph.fi/download/163777_perusopetuksen_opetussuunnitelman_perusteet_2014.pdf s. laato et al. 18 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) fourney, d. w. & fels, d. i. (2009) ‘creating access to music through visualization’, in science and technology for humanity (tic-sth), 2009 ieee toronto international conference, ieee, pp. 939–944. https://doi.org/10.1109/tic-sth.2009.5444364 fox, t. (2015) undertale. self-published. played and tested on nintendo switch. gall, m. & breeze, n. (2005) ‘music composition lessons: the multimodal affordances of technology’, educational review, vol. 57, no. 4, pp. 415–433. https://doi.org/10.1080/00131910500278314 greeno, j. g. (1994) gibson’s affordances. psychological review, vol 101(2), apr 1994, 336–342. american psychological association. 750 first street ne, washington, dc. habibi, a. & damasio, a. (2014) ‘music, feelings, and the human brain’, psychomusicology: music, mind, and brain, vol. 24, no. 1, p. 92. https://doi.org/10.1037/pmu0000033 hendradjaya, b. (2018, november) ‘a framework for development of serious game for learning musical instrument’, in 2018 5th international conference on data and software engineering (icodse), ieee, pp. 1–6. https://doi.org/10.1109/icodse.2018.8705885 hiraga, r., watanabe, f. & fujishiro, i. (2002) ‘music learning through visualization’, in proceedings of the second international conference on web delivering of music, 2002. wedelmusic 2002, ieee, pp. 101–108. https://doi.org/10.1109/wdm.2002.1176199 holcombe, j. e. (2006) u.s. patent no. 6,987,220, u.s. patent and trademark office, washington, dc. holmes, t. b. & holmes, t. (2002) electronic and experimental music: pioneers in technology and composition, psychology press, new york, usa. hoos, h. h., et al., (1998) the guido notation format – a novel approach for adequately representing score-level music. [online] available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.122.1122&rep=rep1&type=pdf, checked 27.8.2019 hope, c. (2017) ‘electronic scores for music: the possibilities of animated notation’, computer music journal, vol. 41, no. 3, pp. 21–35. https://doi.org/10.1162/comj_a_00427 hosken, d. (2014) an introduction to music technology, routledge. new york. 2nd edition. https://doi.org/10.4324/9780203539149 hudak, p., et al., (1996) ‘haskore music notation – an algebra of music’, journal of functional programming, vol. 6, no. 3, pp. 465–484. https://doi.org/10.1017/s0956796800001805 indiana state. (2016) elementary and middle level subjects and descriptions, [online] available at: http://www.doe.in.gov/sites/default/files/ccr/elementary-and-middle-school subjects-2016-17-12-18-15.pdf james, w. (1976) essays in radical empiricism, vol. 3, harvard university press. cambridge, oxford and massachusetts. kolb, d. a. (2014) experiential learning: experience as the source of learning and development, pearson education canada ltd, usa. koutsomichalis, m. (2018) ‘ad-hoc aesthetics: context-dependent composition strategies in music and sound art’, organised sound, vol. 23, no. 1, pp. 12–19. https://doi.org/10.1017/ s1355771817000231 laato, s., et al., (2017, july) ’designing a game for learning math by composing: a finnish primary school case’, in 2017 ieee 17th international conference on advanced learning technologies (icalt), ieee, pp. 136–138. https://doi.org/10.1109/icalt.2017.45 leider, c. n. (2004) digital audio workstation, mcgraw-hill, inc, new york, usa lim, t., lee, s. & ke, f. (2017) ‘integrating music into math in a virtual reality game: learning fractions’, international journal of game-based learning (ijgbl), vol. 7, no. 1, pp. 57–73. lim, t., lee, s. & ke, f. (2018) ‘integrating music into math in a virtual reality game: learning fractions’, international journal of game-based learning (ijgbl), vol. 7, no.  1, pp. 57–73. https://doi.org/10.4018/978-1-5225-5469-1.ch053 lehtinen, e., et al., (2017) ’cultivating mathematical skills: from drill-and-practice to deliberate practice’, zdm, vol. 49, no. 4, pp. 625–636. https://doi.org/10.1007/s11858-017-0856-6 lonka, k., et al., (2018) phenomenal learning from finland, edita. helsinki, finland. macgill, a. (1988) ‘composing in the school: composing for the school’, british journal of music education, vol. 5, no. 1, pp. 35–43. http://dx.doi.org/10.25304/rlt.v27.2259� https://doi.org/10.1109/tic-sth.2009.5444364 https://doi.org/10.1080/00131910500278314 https://doi.org/10.1037/pmu0000033 https://doi.org/10.1109/icodse.2018.8705885 https://doi.org/10.1109/wdm.2002.1176199 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.122.1122&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.122.1122&rep=rep1&type=pdf https://doi.org/10.1162/comj_a_00427 https://doi.org/10.4324/9780203539149 https://doi.org/https://doi.org/10.1017/s0956796800001805 http://www.doe.in.gov/sites/default/files/ccr/elementary-and-middle-school-­subjects-2016-17-12-18-15.pdf http://www.doe.in.gov/sites/default/files/ccr/elementary-and-middle-school-­subjects-2016-17-12-18-15.pdf https://doi.org/10.1017/s1355771817000231 https://doi.org/10.1017/s1355771817000231 https://doi.org/10.1109/icalt.2017.45 https://doi.org/10.4018/978-1-5225-5469-1.ch053 https://doi.org/10.1007/s11858-017-0856-6 research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 19 (page number not for citation purpose) malmi, e., et al., (2016, august) ‘dopelearning: a computational approach to rap lyrics generation’, in proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, acm, pp. 195–204. https://doi.org/10.1145/ 2939672.2939679 manaris, b. & kohn, t. (2016, february) ‘making music with computers: creative programming in python’, in proceedings of the 47th acm technical symposium on computing science education, acm, p. 711. https://doi.org/10.1145/2839509.2844707 marquez, i. (2014) ‘playing new music with old games: the chiptune subculture’, g| a| m| e games as art, media, entertainment, vol. 1, no. 3, pp. 69–79. marrington, m. (2010) ‘experiencing musical composition in the daw: the software interface as mediator of the musical idea’, in proceedings of the 6th art of record production conference, p. 1. available at: http://www.arpjournal.com/asarpwp/experiencing-musical-composition-in-the-daw-the-software-interface-as-mediator-of-the-musical-idea-2/ martin, b. (2018) ‘soundscape composition: enhancing our understanding of changing soundscapes’, organised sound, vol. 23, no. 1, pp. 20–28. https://doi.org/10.1017/ s1355771817000243 mattila, p. & silander, p. (2015) how to create the school of the future – revolutionary thinking and design from finland, multiprint, oulu. mazzola, g., mannone, m. & pang, y. (2016) cool math for hot music, springer international publishing switzerland. https://doi.org/10.1007/978-3-319-42937-3 mccoid, s., et al., (2013) ‘earsketch: an integrated approach to teaching introductory computer music’, organised sound, vol. 18, no. 2, 146–160. https://doi.org/10.1017/ s135577181300006x mcgrenere, j. & ho, w. (2000, may) ‘affordances: clarifying and evolving a concept’, in graphics interface, vol. 2000, 179–186. montreal, canada. melodyne. (2019) melodyne, [online] available at: http://www.celemony.com/en/melodyne/ what-is-melodyne meyer, j. w., kamens, d. & benavot, a. (2017) school knowledge for the masses: world models and national primary curricular categories in the twentieth century, routledge, london. https://doi.org/10.4324/9781315225173 in press. miller, m., et al., (2019) analyzing visual mappings of traditional and alternative music notation. arxiv preprint arxiv:1810.10814. moher, d., et al., (2009) ‘preferred reporting items for systematic reviews and meta-analyses: the prisma statement’, annals of internal medicine, vol. 151, no. 4, pp. 264–269. https:// doi.org/10.7326/0003-4819-151-4-200908180-00135 nilsson, u. (2008) ‘the anxiety-and pain-reducing effects of music interventions: a systematic review’, aorn journal, vol. 87, no. 4, pp. 780–807. https://doi.org/10.1016/j. aorn.2007.09.013 norman, d. a. (1988) the psychology of everydaythings, basic books, new york. obarski, k. (1987) ultimate soundtracker. (software) released online in 1987. self published. paquette, k. r. & rieg, s. a. (2008) ’using music to support the literacy development of young english language learners’, early childhood education journal, vol. 36, no. 3, pp. 227–232. https://doi.org/10.1007/s10643-008-0277-9 pejrolo, a. (2012) creative sequencing techniques for music production: a practical guide to pro tools, logic, digital performer, and cubase, focal press. new york, usa. https://doi. org/10.4324/9780240522173 peters, g. d. (1992) ‘music software and emerging technology: g. david peters outlines the history of music software and hardware and explores the new developments and benefits of the emerging software for use in the classroom’, music educators journal, vol. 79, no. 3, pp. 22–63. piaget, j. (1970) piaget’s theory in ph mussen. london, wiley. pitts, a. & kwami, r. m. (2002) ‘raising students' performance in music composition through the use of information and communications technology (ict): a survey of secondary schools in england’, british journal of music education, vol. 19, no. 1, pp. 61–71. http://dx.doi.org/10.25304/rlt.v27.2259� https://doi.org/10.1145/​2939672.2939679 https://doi.org/10.1145/​2939672.2939679 https://doi.org/10.1017/s1355771817000243 https://doi.org/10.1017/s1355771817000243 https://doi.org/10.1007/978-3-319-42937-3 https://doi.org/10.1017/s135577181300006x https://doi.org/10.1017/s135577181300006x http://www.celemony.com/en/melodyne/what-is-melodyne http://www.celemony.com/en/melodyne/what-is-melodyne https://doi.org/10.4324/9781315225173 https://doi.org/10.7326/0003-4819-151-4-200908180-00135 https://doi.org/10.7326/0003-4819-151-4-200908180-00135 https://doi.org/10.1016/j.aorn.2007.09.013 https://doi.org/10.1016/j.aorn.2007.09.013 https://doi.org/10.1007/s10643-008-0277-9 https://doi.org/10.4324/9780240522173 https://doi.org/10.4324/9780240522173 s. laato et al. 20 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) pitts, s. (2000) ‘reasons to teach music: establishing a place in the contemporary curriculum’, british journal of music education, vol. 17, no. 1, pp. 32–42. pope, s. t., et al., (1995) ‘touched by machine?: composition and performance in the digital age’, computer music journal, vol. 19, no. 3, pp. 13–17. rabiner, l. r. & gold, b. (1975) theory and application of digital signal processing, prentice-hall, inc., englewood cliffs, nj, 777 p. rogers, g. l. (2016) ‘the music of the spheres: cross-curricular perspectives on music and science’, music educators journal, vol. 103, no. 1, pp. 41–48. https://doi. org/10.1177/0027432116654547 runco, m. a., ebersole, p. & mraz, w. (1991) ‘creativity and self-actualization’, journal of social behavior and personality, vol. 6, no. 5, pp. 147. ruokonen, i. & ruismäki, h. (2016) ‘e-learning in music: a case study of learning group composing in a blended learning environment’, procedia-social and behavioral sciences, vol. 217, pp. 109–115. https://doi.org/10.1016/j.sbspro.2016.02.039 sætre, j. h. (2011) ‘teaching and learning music composition in primary school settings’, music  education research, vol. 13, no. 1, pp. 29–50. https://doi.org/10.1080/14613808.20 11.553276 savage, j. (2005) ‘working towards a theory for music technologies in the classroom: how pupils engage with and organise sounds with new technologies’, british journal of music education, vol. 22, no. 2, pp. 167–180. https://doi.org/10.1017/s0265051705006133 selwyn, d. (1993) living history in the classroom: integrative arts activities for making social studies meaningful, zephyr press, tucson, az. shapiro, r. b., et al., (2017) ‘tangible distributed computer music for youth’, computer music journal. vol. 41, no. 2, pp. 52–68. https://doi.org/10.1162/comj_a_00420 smith, s. m. & williams, g. n. (1997, october) ‘a visualization of music’, in proceedings. visualization'97 (cat. no. 97cb36155), ieee, pp. 499–503. https://doi.org/10.1109/ visual.1997. 663931 song maker. (2018) song maker authored by chrome experiments music lab, [online] available at: https://musiclab.chromeexperiments.com/song-maker/ strand, k. (2006) ‘survey of indiana music teachers on using composition in the classroom’, journal of research in music education, vol. 54, no. 2, pp. 154–167. https://doi. org/10.1177/002242940605400206 sylwester, r. (1995) an educator's guide to the human brain, association for supervision and curriculum development, alexandria, va. (ascd stock no. 1-95085). symeonidis, v. & schwarz, j. f. (2016) ‘phenomenon-based teaching and learning through the pedagogical lenses of phenomenology: the recent curriculum reform in finland’, forum oświatowe, vol. 28, no. 2, pp. 31–47. taylor, t. d. (2014) strange sounds: music, technology and culture, routledge. new york. https://doi.org/10.4324/9780203700235 thompson, w. f. (2015) music, thought, and feeling: understanding the psychology of music, oxford university press, new york, usa. tossavainen, t. & juvonen, a. (2015) ‘finnish primary and secondary school students' interest in music and mathematics relating to enjoyment of the subject and perception of the importance and usefulness of the subject’, research studies in music education, vol. 37, no. 1, pp. 107–121. https://doi.org/10.1177/1321103x15589259 troelstra, a. s. & van dalen, d. (2014) constructivism in mathematics, vol. 2, elsevier. amsterdam, the netherlands. truitt, f. e., et al., (1997) ‘the perceptual span and the eye-hand span in sight reading music’, visual cognition, vol. 4, no. 2, pp. 143–161. uk government, (2013) britain primary school curriculum, [online] available at: https:// assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/ file/425601/primary_national_curriculum.pdf http://dx.doi.org/10.25304/rlt.v27.2259� https://doi.org/10.1177/0027432116654547 https://doi.org/10.1177/0027432116654547 https://doi.org/10.1016/j.sbspro.2016.02.039 https://doi.org/10.1080/14613808.2011.553276 https://doi.org/10.1080/14613808.2011.553276 https://doi.org/10.1017/s0265051705006133 https://doi.org/10.1162/comj_a_00420 https://doi.org/10.1109/visual.1997.​663931 https://doi.org/10.1109/visual.1997.​663931 https://musiclab.chromeexperiments.com/song-maker/ https://doi.org/10.1177/002242940605400206 https://doi.org/10.1177/002242940605400206 https://doi.org/10.4324/9780203700235 https://doi.org/10.1177/1321103x15589259 https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/425601/primary_national_curriculum.pdf https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/425601/primary_national_curriculum.pdf https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/425601/primary_national_curriculum.pdf research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 21 (page number not for citation purpose) von glasersfeld, e. (2013) radical constructivism, routledge, london, uk. https://doi. org/10.4324/9780203454220 walzer, d. a. (2016) ‘software-based scoring and sound design: an introductory guide for music technology instruction’, music educators journal, vol. 103, no. 1, pp. 19–26. wejam. (2018) wejam, [online] available at: http://wejam.com/ wikipedia. (2018) list of music software, [online] available at: https://en.wikipedia.org/wiki/ list_of_music_software winters, m. (2012) ‘the challenges of teaching composing’, british journal of music education, vol. 29, no. 1, pp. 19–24. https://doi.org/10.1017/s0265051711000489 yatatoy. (2019) bandimal, [online] available at: http://www.yatatoy.com/bandimal http://dx.doi.org/10.25304/rlt.v27.2259� https://doi.org/10.4324/9780203454220 https://doi.org/10.4324/9780203454220 http://wejam.com/ https://en.wikipedia.org/wiki/list_of_music_software https://en.wikipedia.org/wiki/list_of_music_software http://www.yatatoy.com/bandimal s. laato et al. 22 citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 (page number not for citation purpose) appendix a:list of analyzed software and their classification. name classification ableton live daw acid pro daw adobe audition daw anvil studio midi sequencer ardour daw aria maestosa midi sequencer audacity sound editing software audiotool daw b-step sequencer midi sequencer band-in-a-box midi sequencer cubase daw deluxe music construction set midi sequencer digital performer daw fairlight daw finale midi sequencer flstudio daw garageband daw guitar pro 7 daw hydrogen beat machine hyperscore 4.5 educational jfugue midi sequencer keykit midi sequencer logic pro x daw madtracker tracker mario paint composer game maschine daw milkytracker tracker mixbus daw mixcraft daw mozart midi sequencer mulab daw musescore midi sequencer noteflight notation software noteworthy composer midi sequencer nuendo daw numerology midi sequencer openmpt tracker podium midi sequencer protools daw psycle tracker qtractor daw reaper daw reason daw renoise daw rosegarden midi sequencer samplitude daw sequoia daw http://dx.doi.org/10.25304/rlt.v27.2259� research in learning technology citation: research in learning technology 2019, 27: 2259 http://dx.doi.org/10.25304/rlt.v27.2259 23 (page number not for citation purpose) appendix a: (continued) name classification sibelius notation software sonar x 1 daw song maker tracker soundtracker tracker studio one daw tracktion daw tuxguitar midi sequencer wejam game xequence midi sequencer z-maestro midi sequencer http://dx.doi.org/10.25304/rlt.v27.2259� (page number not for citation purpose) 1 *corresponding author. email: cbruguerat@uoc.edu research in learning technology 2019. © 2019 c. bruguera et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 research in learning technology vol. 27, 2019 review article social media and professional development: a systematic review carles bruguera*, montse guitert and teresa romeu faculty of psychology and education sciences, universitat oberta de catalunya (uoc), barcelona, spain (received: 17 june 2019; revised: 26 september 2019; accepted: 7 october 2019; published: 12 november 2019) the great popularisation of social media at the beginning of the 21st century has led to the production of many empirical studies in an attempt to explore the opportunities these platforms provide for different activities, such as learning and updating for professionals. this study aims to identify and summarise the main characteristics of research into social media and professional development published between 2013 and 2017. we analysed the years, journals, conceptual background, research methodologies, data collection tools, professional disciplines, educational contexts, types of social media and characteristics of social media that can generate learning opportunities. a total of 44 articles were selected and analysed from peer-reviewed journals. findings revealed that (1) an upward trend with respect to research on social media and professional development; (2) surveys were the main research method for collecting data about social media; (3) health and education sciences are the most studied fields of knowledge; (4) there is a special interest in the study of social media in informal learning contexts; (5) twitter is the most studied social media platform and (6) social media seems to be a sustainable support for professional development due to its open, social and flexible nature. implications of findings for future research are also discussed. keywords: lifelong learning; professional development; ict; social media; literature review introduction the intensive and widespread use of digital technologies by today’s society has modified and expanded the way in which people communicate and learn (donelan 2016). nowadays, citizens use these technologies for multiple purposes, both in their personal and professional spheres (hunter and hall 2018). for professionals, this digital transformation has also meant the need to continuously update their professional knowledge and skills to meet the challenges of the digital era, in what is framed within the idea of ‘lifelong learning’ or ‘professional development’ (collin, van der heijden, and lewis 2012). social media have emerged as a prominent element of digital technologies that can be defined as global digital communication platforms. these platforms, which connect large numbers of users producing and exchanging user-generated content, seem able to offer sustainable support for professionals’ lifelong learning, thanks to mailto:cbruguerat@uoc.edu http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2286 c. bruguera et al. 2 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) characteristics such as the ease with which users are able to create and share content, the breaking down of space and time barriers, and the flexibility and high customisation that its contents and services allow (ngai, tao, and moon 2015; pérez-latre, portilla, and blanco 2011; zimmer and proferes 2014). numerous literature reviews have been conducted since the emergence of social media at the beginning of the 21st century, with the purpose of exploring the opportunities offered by these media for learning and updating for professionals. recent years have seen a proliferation of reviews from health sciences (cheston, flickinger, and chisolm 2013; curran et al. 2017; lawson and cowling 2014; roberts et al. 2015; sterling et al. 2017) and education sciences (gao, luo, and zhang 2012; manca and ranieri 2015, 2016; tess 2013). the particular interest in these two fields of knowledge may be explained by diverse factors. in health sciences, as highlighted by some authors of previous reviews, participation of professionals in professional development activities is sometimes required, whether in courses, conferences, journal clubs or other activities. another main interest lies in the support that social media can offer to keep up-to-date with current clinical evidence and support the exchange of best professional practices. in education sciences, some authors highlight the teachers’ interest in incorporating social media platforms into their professional practice as education tools; interest in using social media to exchange professional practices, evidence and resources; or interest in using social media to enable sustainable networks among professionals. furthermore, in both fields the use of social media for professional development is also highly valued for helping to overcome geographical, temporal and economic barriers. the number of literature review-based studies and the results of these studies indicate a research field that is growing and beginning to have an abundant body of data, although reviews mentioned focused in a period of time prior to 2015. due to the constant developments of social media platforms and the increasing use made by society and its professionals, it is worth considering how research in social media and professional development has evolved in most recent years, from a multidisciplinary prism. building on the previous literature reviews and based on some of the most relevant types of information considered in those previous reviews, the aim of this article is to review and analyse the scientific literature on social media and professional development published between 2013 and 2017, to identify and summarise the main characteristics, draw conclusions and point to trends in this field of research. methodology a systematic review provided a summary of the research literature, quantitative and qualitative, using explicit and replicable methods to identify and select relevant studies and objective and replicable techniques to analyse and summarise these studies (arksey and o’malley 2005; cooper 2017). in line with these authors, a number of steps were followed to ensure a systematic literature review. we describe them below. formulation of the research objectives the first step of the systematic review was to formulate the research objectives. we aimed to identify and summarise the years, journals, conceptual background, research http://dx.doi.org/10.25304/rlt.v27.2286 research in learning technology citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 3 (page number not for citation purpose) methodologies, research methods, data collection tools, professional disciplines, educational contexts, types of social media and characteristics of social media capable of generating learning opportunities used in research articles in social media and professional development published between 2013 and 2017. the key research question we wished to answer was this: what were the main characteristics of the scientific literature on social media and professional development published between 2013 and 2017? parameters of the search for articles the second step of the systematic review was to establish the parameters of the search for articles. to ensure a comprehensive data collection and a multidisciplinary approach, we searched for articles that were indexed in two social sciences reference databases: ‘scopus’ and ‘isi web of science’. the terms used to search for articles consisted of, on the one hand, a combination of the keywords ‘social media’ and ‘social networks’, and a combination of the keywords ‘lifelong learning’, ‘professional development’ and ‘professional updating’, on the other. the use of these terms allowed us to align our search with the search strategies conducted in the previous literature reviews. the articles chosen had to satisfy the following specific criteria: 1. focus on the relationship between social media and learning; 2. learning approach including professional updating and the lifelong learning of professionals; 3. reporting of empirical data; 4. publication in a journal indexed in scopus or isi web of science between 2013 and 2017 or identification using the snowball sampling method; and 5. availability in english or spanish. in addition, the full text of the article had to be available through the researchers’ institutional library subscriptions. selection of the studies the third step of the systematic review consisted in selection and compilation of the studies. the first database consulted was isi web of science. a first search produced 328 results. the results were then filtered by article, reducing the results to 241. filtering by language to english and spanish reduced the number of articles to 228. in a first round of reading the abstracts, the filtered results were compared  with the established criteria, and the number of articles was reduced to a total of 56. in a second round of a complete reading of the articles, that total was reduced to 30. the second database consulted was scopus. a first search produced 426 results. the results were then filtered by article, reducing the number of articles to 262. filtering by language to english and spanish reduced the articles to 257 articles. in a first round of reading the abstracts, the filtered results were compared with the established criteria, and the number of articles was reduced to a total of 52. in a second round of a complete reading of the articles, that total was reduced to 34. http://dx.doi.org/10.25304/rlt.v27.2286 c. bruguera et al. 4 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) some articles were also selected using the snowball sampling method: the bibliographic references of the articles previously selected were consulted to obtain an additional four articles of interest. any duplicate articles (appearing in both databases) were then merged, leaving a final selection of 44 articles. the articles were compiled in mendeley, and the information were extracted and organised in an extensive spreadsheet. data extraction and research analysis the fourth step of the systematic review consisted of extracting and analysing the information from the selected studies. before applying the analysis, we decided on the type of information to be extracted from the articles. the name or label of the type of information ‘should be suggested by the context in which the event is located’ (strauss and corbin 2002, p. 116). firstly, the 44 articles were collected and organised with the bibliographic data, including article title, authors, journal, abstract, keywords and publication year. then, 10 additional categories related to the articles were coded, based on those considered in the previous literature reviews, to obtain relevant data useful to cover the basic characteristics of research on social media and professional development. finally, information from each publication was organised according to the following criteria: (1) year, (2) journal, (3) conceptual background, (4) research methodologies, (5)  research methods, (6) data collection tools, (7) professional disciplines, (8)  educational contexts, (9) type of social media and (10) characteristics of social media capable of generating learning opportunities. the different types of information were extracted from the articles in the form of categories. the objective of categorising the types of information was to ‘obtain key elements of the information obtained in the research articles’ (arksey and o’malley 2005, p. 26). a qualitative content analysis was performed. this enabled the information to be organised into categories and subcategories for further analysis (gibbs 2012). in this qualitative analysis, ‘data are broken down into discrete parts, closely examined, and compared for similarities and differences’ (strauss and corbin 2002, p. 111). presentation of results and interpretation the last step of the systematic review consisted of interpretation of the evidence and presentation of the results. due to the broad scope of this phase, the results are presented in the following section, with the different types of information analysed organised in separate sections, and using descriptive statistics for more effective description and visualisation. results in this section, we will present the results of the study, organised by the types of information indicated in the previous section. years the articles reviewed were published between 2013 and 2017 (figure 1). http://dx.doi.org/10.25304/rlt.v27.2286 research in learning technology citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 5 (page number not for citation purpose) journals the articles were published in 40 different journals. figure 2 shows the distribution of the articles in journals with two or more articles identified in our review. three of these journals are relevant to education sciences and two to health sciences. figure 1. distribution of number of publications by year. figure 2. distribution of journals with two or more articles by number of publications. conceptual background to understand how the different research is conceptually framed, another type of information extracted from the articles and subjected to a categorisation process was the set of conceptual backgrounds of the different articles (table 1). by ‘conceptual background’, we refer to the concepts that served as an umbrella to the different empirical studies, provided these were made explicit by the authors of the articles. concepts that were not clearly specified were categorised within the ‘not specified’ label. those closely related were grouped into broader categories. the list is not exhaustive. http://dx.doi.org/10.25304/rlt.v27.2286 c. bruguera et al. 6 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) research methodologies the studies reviewed used qualitative, quantitative or mixed methodology to carry out their research, as shown in figure 3. figure 3. distribution of research methodologies by publication. table 1. distribution of articles by conceptual background and definitions. conceptual background n % personal learning environment (ple)/personal learning network (pln) ‘set of tools, sources of information, connections and activities that each person uses regularly to learn’ (castañeda and adell 2011, p. 7). 8 20 community of practice (cop) ‘learning model based upon the principles of apprenticeship, and cooperative learning’ (davis 2015, p. 1552). 7 17 teachers’ professional development ‘means to improve schools, increase teaching quality, and improve students’ academic achievements’ (avidov-ungar 2016, p. 653). 3 7 affinity spaces digital spaces ‘where users can interact around topics of shared interest’ (carpenter 2015, p. 211). 3 7 lifelong learning approach broad term that encompasses learning throughout life, from early childhood through adulthood (weber and vincent 2014). 2 4 journal clubs regular meeting of a group of professionals to critically evaluate recent articles in academic literature (roberts et al. 2015). 1 2 not specified 15 34 research methods the research methods were also identified, as shown in figure 4. in this respect, a number of emergent methods related with social media research were identified alongside ‘traditional’ scientific research methods such as surveys http://dx.doi.org/10.25304/rlt.v27.2286 research in learning technology citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 7 (page number not for citation purpose) figure 4. distribution of research methods by publication. figure 5. distribution of data collection tools by publication. and interviews, in contrast to the methods found in previous literature reviews. one of these emergent methods is social network analysis (sna), a methodology for examining the interaction patterns that occur within a group of users, and which is ‘very suitable for the analysis of large quantities of data’ (rehm, littlejohn, and rienties 2018, p. 311). this is also the ‘digital methods’ label, which designates ‘a term that seeks to capture a recent development in internet-related research related to the internet, summarised as approaches to the web as dataset’ (rogers 2005, p. 1, cited in hutchinson 2016, p. 1). data collection tools in addition to the methodologies and methods, the data collection tools used in the studies were also identified. a distinction must be made between the kind of data obtained through, for example, twitter or linkedin web analytics, and data obtained from content generated and shared on social media through, for example, twitter posts, youtube comments or facebook chats (figure 5). http://dx.doi.org/10.25304/rlt.v27.2286 c. bruguera et al. 8 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) educational contexts the educational contexts addressed in the set of articles were also identified, as shown in table 2. to extract the information related to these contexts, the articles were categorised according to whether they focused on a formal, non-formal or informal learning context. gonzález-sanmamed et al. (2018) offer useful definitions of each context. formal education can be defined as ‘formal training that takes place in different organised and structured environments’ (p. 24). non-formal education, also known as organised informal education, can be defined as ‘activities organised by different institutions that take place outside of what is recognised as regulated training, and may or may not lead to the attainment of a certificate’ (p. 24). informal education can be defined as the ‘day-to-day learning that occurs in everyday situations (with family, friends, co-workers, etc.). this is not organised, and can occur by chance, in which case it is called incidental’ (p. 25). figure 6. distribution of professional disciplines by number of publications. table 2. distribution of educational contexts by publication. educational contexts number of articles % informal 32 73 formal 22 50 non-formal 19 43 types of social media the different social media platforms addressed in the set of studies were also identified (figure 7). professional disciplines professional disciplines that were the subject of study in the set of articles were also identified and categorised. this categorisation is shown in figure 6. http://dx.doi.org/10.25304/rlt.v27.2286 research in learning technology citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 9 (page number not for citation purpose) characteristics capable of generating learning opportunities the characteristics of social media that are able to generate the learning opportunities for professional development specified in the conclusions of the set of studies were also categorised. by ‘learning opportunity’, we refer to situations in which learning can be activated. to categorise these characteristics, the social media for learning (sm4l) framework was used. the sm4l framework aims to demonstrate how students and academics can use social media to promote learning (middleton and beckingham 2015); for this study, the original framework was adapted for a professional context. firstly, the characteristics identified were categorised using the original sm4l framework. then we removed, modified and added categories and subcategories in a new adaptation, following the qualitative content analysis previously described. table 3 shows the most-found characteristics of social media capable of generating learning opportunities as identified in the set of articles. each characteristic has figure 7. distribution of social media platform focuses of study by publication. table 3. distribution of main characteristics that generate learning opportunities found in the articles by publication. characteristics that generate learning opportunities number of articles % support access to new ideas 29 66 expand learners’ communication and facilitate interaction and sharing 28 64 enable connections through social, professional and learning networks 26 59 support and expand ‘traditional’ learning or professional development 22 50 promote self-direction, self-regulation, self-expression, enjoyment and confidence of students 22 50 support open access (geographically extended, democratic, inclusive and user-controlled; free, open and accessible content) 22 50 widely present and popular in academic and professional spheres 21 48 support and validate learning through communities of practice 20 45 accommodate common interests among learners and enable discussion of topics 19 43 facilitate asking for help or just-in-time information 18 41 http://dx.doi.org/10.25304/rlt.v27.2286 c. bruguera et al. 10 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) been counted independently, more than one characteristic having been identified per study. the list is not exhaustive. discussion this article highlights several points regarding research into social media and professional development. we describe them below. upward trend in research into social media and professional development firstly, we observed a mostly upward trend in the number of publications during the delimited period from 2013 to 2017, with a slight setback in 2016. the number of articles identified in recent years helps to overcome the 2015 barrier of previous literature reviews and serves to address the need for empirical studies on social media and learning research, which has been pointed out by some authors (tess 2013). the wide variety of journals in different fields that published the revised articles – 35 in total – is noteworthy. five journals contributed two or more articles to this review: three from educational sciences and two from health sciences. this increasing number of articles and the great variety of journals that have addressed this field of knowledge seem to indicate an interest in line with the popularisation of social media and the immense amount of space that these media have come to occupy in people’s lives (smith and anderson 2018). strong social component of conceptual backgrounds we analysed the conceptual approaches of the studies, as long as the authors made it explicit. that was useful to frame social media in the research field and better understand the author’s point of view. in the first place, it should be noted that, in almost a quarter of the articles reviewed, the conceptual approaches were not clearly stated beyond a general reference to a particular field of knowledge. among the articles that made their conceptual background explicit, personal learning environments (ples) and personal learning networks (plns) were the most-used conceptual frameworks when placing social media in the educational process, especially in a formal context. other concepts and terms with a marked social component such as ‘communities of practice’, ‘affinity spaces’ or ‘journal clubs’ have acquired a remarkable presence. surveys, the main research method for collecting social media data there is a certain balance among the reviewed articles regarding the use of quantitative, qualitative and mixed methodologies, though the quantitative methodologies are the most prevalent among the set of studies. of these quantitative methodologies, the survey method was by far the method most used, with questionnaires being the form of data collection most frequently used for obtaining data relevant to social media and learning: more than half of the reviewed articles used questionnaires to collect data, which aligns with results of previous literature reviews. however, a number of methods and techniques tailored to digital environments – encompassed in concepts such as digital methods, digital ethnography or sna – were observed to have emerged. health and education sciences remain the most studied fields of knowledge most of the scientific research indicated in previous literature reviews was produced in education and health sciences (cheston, flickinger, and chisolm 2013; curran et al. http://dx.doi.org/10.25304/rlt.v27.2286 research in learning technology citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 11 (page number not for citation purpose) 2017; gao, luo, and zhang 2012; lawson and cowling 2014; manca and ranieri 2015, 2016; roberts et al. 2015; sterling et al. 2017), probably for the particularities we described in the introduction section. although our literature review followed a multidisciplinary approach that did not exclude any field of knowledge or professional sector, we were able to observe that the education and health sectors remained the fields with the highest concentration of studies. we can state that most of the interest is still coming from these two fields. research in the health sector covered the greatest variety of professional disciplines, in a broad spectrum ranging from radiology to paediatrics. research in the educational sector was focused mainly on higher education, probably because of its greater accessibility to researchers. special interest in the informal learning context there was cross-disciplinary interest in studying the incidence of social media in different learning contexts, though the informal learning context was the most frequently studied. previous reviews showed a slightly greater interest in the study of social media in formal contexts. in our review, interest in the formal context seemed related to the way in which social media could be effective as an educational tool. in the non-formal context, interest seemed to be related to determining how these media could support and extend ‘traditional’ learning or professional development activities such as conferences, courses or journal clubs. in the informal context, interest was broader and more diverse. firstly, there was interest in studying the ‘spontaneous’ and ‘general’ use of social media made by users in their daily lives. then there was interest in the self-directed learning of individuals enabled and empowered by social media. and thirdly, there was also interest in the informal use of social media by citizens in their educational or professional contexts and activities. twitter, the most cited social media platform with respect to social media platforms, twitter was the most studied social media platform in our literature review. while twitter was not the social medium most used by internet users, it seems to have an active presence in the professional life of citizens, serving as a platform for professional global digital communication. this focus on twitter is aligned with the growing interest indicated in some of the previous reviews (zimmer and proferes 2014). facebook, the most widely used social media platform in general use today (smith and anderson 2018), was the most visited platform after twitter. twitter may have become the preferred platform over other platforms such as facebook ‘due to advantages such as short-form messaging, easily grouped themes with hashtags, and rapid spread of messages (going viral) with easy sharing through retweeting’ (goodyear, casey, and quennerstedt 2018). characteristics that generate learning opportunities are markedly open, social and accessible the many and varied characteristics of social media capable of generating learning opportunities were categorised using an adaptation of the sm4l framework (middleton and beckingham 2015). among the many characteristics to emerge from the set of studies, we would highlight opportunities derived from (1) the openness of social media, especially in terms of access to new ideas that allow people to keep up to date http://dx.doi.org/10.25304/rlt.v27.2286 c. bruguera et al. 12 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) with diverse aspects of their profession, overcoming spatial, temporal and economic barriers; (2) the multiple communication and sharing options that these media allow by enabling different forms of interaction between users and (3) by being authentically situated in the professional field and familiar to most users; (4) the diverse types of personal or professional networks facilitated by social media and (5) the user-centred focus, especially conducive to self-directed and informal learning. these characteristics seem defining when thinking about social media for professional development and its applications for professionals’ learning and updating. conclusions this systematic literature review analysed and synthesised 5 years of research in social media and professional development, covering years and disciplines not covered in the previous literature review. the review helps to point to the existing academic interest in the possibilities of social media for supporting professionals in their learning and professional development in different learning contexts. also, it shows the way research has been done, in terms of methodology, and the kind of evidence obtained. our review provides new information to that generated in the previous reviews, which did not consider the research carried out beyond 2015. in this sense, we would like to highlight some of the main characteristics found. firstly, although surveys prevail as the main method to collect data, emerging methods such as digital methods are being increasingly implemented, being indicated as especially useful when analysing the large amount of data generated in digital contexts. secondly, twitter has a very prominent situation in the professional context compared to other social media platforms, especially to get updated information useful for keeping up-todate professionally and connecting with other professionals. thirdly, the study of social media and professional development is focusing more on informal learning contexts. while in previous reviews there was a slightly greater focus on social media within formal learning contexts, many of the authors of our review focus on the informal use of social media and its implications for learning. fourthly, research in social media and professional development is still addressed primarily in the fields of education and health, although in our review we observed a growing interest in other professional fields, such as social work, librarianship, science, technology, engineering and mathematics (stem) disciplines, forestry and music. from the review we conducted, we also observed that numerous empirical studies coincide in indicating that social media can be a useful communication platform and serve as open and rich digital spaces for professionals to exchange information, opinions and ideas. overcoming geographical, temporal and economic barriers is one of the key points to use them for professional development purposes. in addition, the speed with which social media content is updated is pointed out as also very relevant, though we also found serious concerns about what are referred to as privacy conflicts. despite the huge popularisation of social media platforms such as facebook, twitter, instagram and youtube in the last decade, some authors pointed out that research into social media and professional development is still at an incipient stage (ngai, tao, and moon 2015) and, although progress is being made, a greater amount of empirical data still needs to be generated. as an overall conclusion, our findings http://dx.doi.org/10.25304/rlt.v27.2286 research in learning technology citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 13 (page number not for citation purpose) suggest that a considerable amount of data has been generated in recent years, although these data are a bit chaotic and loosely connected. several points of interest and possible future lines of study emerged from the results of this literature review. firstly, informal use of social media seems to require further study because much of social media use occurs outside formal and nonformal settings and for purposes other than learning. one question that arises in this respect is whether social media platforms can be incorporated into formal contexts for educational purposes, or whether the true virtue of these media is the informal, spontaneous and self-directed use made of them by their users. more research into social media and professional development outside the disciplines of education and health sciences is also needed to help meet the needs and peculiarities of the different groups of professionals in today’s society. research into social media needs to acknowledge that social media platforms are constantly evolving and adding new features to their service. facebook or twitter today has new features compared to the facebook and twitter of 10 years ago. in addition, other digital products, such as whatsapp or spotify, are incorporating more and more social network features into their functionalities, making it difficult to mark the boundary between what is social media and what is not. in any case, what seems clear is that the evolution of digital technologies, the social web and mobile devices has fostered a societal need for open, social and accessible platforms of communication that allow us to connect with other people and generate and share content in a simple way while offering us an attractive functionality and design. we believe, then, that research should focus not on specific platforms, but on what these platforms can enable in terms of learning and communication possibilities. as we have seen from the review conducted, conceptual approaches with a social and technological emphasis such as ple and pln, communities of practice and affinity spaces were used to locate social media in the process of learning and updating for professionals. in this respect, the learning ecologies framework – defined by barron (2004) as ‘the set of contexts composed of configurations of activities, resources and relationships that are generated in physical or virtual spaces and that offer learning opportunities’ (p. 6) – could be highly useful in determining how professionals currently learn. this ecological vision of how people learn could enable professionals to learn how to organise their learning ecology better, update themselves professionally, understand the role of social media in the framework of their learning ecology and be aware of the benefits these media can offer for their lifelong learning process. funding this work was supported by the government of catalonia as part of the ‘grants for the recruitment of early-stage research staff (fi-2018)’ initiative. references * abella, v. & delgado, v. (2015) ‘aprender a usar twitter y usar twitter para aprender’, profesorado, vol. 19, no. 1, pp. 364–378. available at: http://mendeley.csuc.cat/fitxers/ f1b181d0f1a1e56355d5ff6b9cc518fb *the references marked with an asterisk make up the 44 empirical studies reviewed. http://dx.doi.org/10.25304/rlt.v27.2286 http://mendeley.csuc.cat/fitxers/f1b181d0f1a1e56355d5ff6b9cc518fb http://mendeley.csuc.cat/fitxers/f1b181d0f1a1e56355d5ff6b9cc518fb c. bruguera et al. 14 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) * alshiekhly, u., et al., (2015) ‘facebook as a learning environment for teaching medical emergencies in dental practice’, education for health, vol. 28, no. 3, pp. 176–180. http://doi. org/10.4103/1357-6283.178609 * alsobayel, h. (2016) ‘use of social media for professional development by health care professionals: a cross-sectional web-based survey’, jmir medical education, vol. 2, no. 2, p. e15. http://doi.org/10.2196/mededu.6232 arksey, h. & o’malley, l. (2005) ‘scoping studies: towards a methodological framework’, international journal of social research methodology: theory and practice, vol. 8, no. 1, pp. 19–32. http://doi.org/10.1080/1364557032000119616 * baker, l. r. & hitchcock, l. i. (2017) ‘using pinterest in undergraduate social work education: assignment development and pilot survey results’, journal of social work education, vol. 53, no. 3, pp. 535–545. http://doi.org/10.1080/10437797.2016.1272515 barron, b. (2004) learning ecologies for technological fluency: gender and experience differences’, journal of educational computing research, vol. 31, no. 1, pp. 1–36. https:// doi.org/10.2190/1n20-vv12-4rb5-33va * bogdanou, t., et al., (2013) ‘use of the internet and social media in the forestry profession in the united kingdom’, international forestry review, vol. 15, no. 2, pp. 147–159. http://doi. org/10.1505/146554813806948521 * bolderston, a., et al., (2018) ‘twitter journal clubs and continuing professional development: an analysis of a #medradjclub tweet chat’, radiography, vol. 24, no. 1, pp. 3–8. http://doi. org/10.1016/j.radi.2017.09.005 * britt, v. g. & paulus, t. (2016) ‘“beyond the four walls of my building”: a case study of #edchat as a community of practice’, american journal of distance education, vol. 30, no. 1, pp. 48–59. http://doi.org/10.1080/08923647.2016.1119609 * carpenter, j. (2015) ‘preservice teachers’ microblogging: professional development via twitter’, contemporary issues in technology and teacher education, vol. 15, no. 2, pp. 1–21. available at: http://mendeley.csuc.cat/fitxers/dbb45a25ed2f92ba576751590c7ce764 * carpenter, j. p. & krutka, d. g. (2014) ‘how and why educators use twitter: a survey of the field’, journal of research on technology in education, vol. 46, no. 4, pp. 414–434. http://doi. org/10.1080/15391523.2014.925701 * carpenter, j. p. & krutka, d. g. (2015) ‘engagement through microblogging: educator professional development via twitter’, professional development in education, vol. 41, no. 4, pp. 707–728. http://doi.org/10.1080/19415257.2014.939294 * carpenter, j. p., tur, g. & marín, v. i. (2016) ‘what do u.s. and spanish pre-service teachers think about educational and professional use of twitter? a comparative study’, teaching and teacher education, vol. 60, pp. 131–143. http://doi.org/10.1016/j.tate.2016.08.011 * cartner, h. c. & hallas, j. l. (2017) ‘challenging teachers’ pedagogic practice and assumptions about social media’, online learning, vol. 21, no. 2. pp. 1–21. http://doi.org/10.24059/ olj.v21i2.1009 * cartwright, l. (2017) ‘supporting students to use social media and comply with professional standards’, social work education, vol. 36, no. 8, pp. 880–892. http://doi.org/10.1080/0261 5479.2017.1372409 castañeda, l. & adell, j. (2011) ‘el desarrollo profesional de los docentes en entornos personales de aprendizaje (ple)’, in la práctica educativa en la sociedad de la información: innovación a través de la investigación, eds r. roig vila & c. laneve, marfil, pp. 83–95. spain: marfil. isbn: 978-84-268-1563-7 * chaoyan, d., et  al., (2015) ‘using linkedin for continuing community of practice among hand surgeons worldwide’, journal of continuing education in the health professions, vol. 35, no. 3, pp. 185–191. http://doi.org/10.1002/chp.21300 cheston, c. c., flickinger, t. e. & chisolm, m. s. (2013, june) ‘social media use in medical education: a systematic review’, academic medicine, vol. 88, no. 6, pp. 893–901. lippincott williams and wilkins. http://doi.org/10.1097/acm.0b013e31828ffc23 http://dx.doi.org/10.25304/rlt.v27.2286 http://doi.org/10.4103/1357-6283.178609 http://doi.org/10.4103/1357-6283.178609 http://doi.org/10.2196/mededu.6232 http://doi.org/10.1080/1364557032000119616 http://doi.org/10.1080/10437797.2016.1272515 https://doi.org/10.2190/1n20-vv12-4rb5-33va https://doi.org/10.2190/1n20-vv12-4rb5-33va http://doi.org/10.1505/146554813806948521 http://doi.org/10.1505/146554813806948521 http://doi.org/10.1016/j.radi.2017.09.005 http://doi.org/10.1016/j.radi.2017.09.005 http://doi.org/10.1080/08923647.2016.1119609 http://mendeley.csuc.cat/fitxers/dbb45a25ed2f92ba576751590c7ce764 http://doi.org/10.1080/15391523.2014.925701 http://doi.org/10.1080/15391523.2014.925701 http://doi.org/10.1080/19415257.2014.939294 http://doi.org/10.1016/j.tate.2016.08.011 http://doi.org/10.24059/olj.v21i2.1009 http://doi.org/10.24059/olj.v21i2.1009 http://doi.org/10.1080/02615479.2017.1372409 http://doi.org/10.1080/02615479.2017.1372409 http://doi.org/10.1002/chp.21300 http://doi.org/10.1097/acm.0b013e31828ffc23 research in learning technology citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 15 (page number not for citation purpose) * chretien, k., et  al., (2015) ‘a digital ethnography of medical students who use twitter for professional development’, jgim: journal of general internal medicine, vol. 30, no. 11, pp. 1673–1680. http://doi.org/10.1007/s11606-015-3345-z collin, k., van der heijden, b. & lewis, p. (2012) ‘continuing professional development’, international journal of training and development, vol. 16, no. 3, pp. 155–163. https://doi. org/10.1111/j.1468-2419.2012.00410.x * cooner, t. s. (2014) ‘using facebook to explore boundary issues for social workers in a networked society: students’ perceptions of learning’, british journal of social work, vol. 44, no. 4, pp. 1063–1080. http://doi.org/10.1093/bjsw/bcs208 cooper, h. m. (2017) research synthesis and meta-analysis: a step-by-step approach. los angeles, ca: sage. isbn: 978-1-4833-4705-9 curran, v., et al., (2017) ‘a review of digital, social, and mobile technologies in health professional education’, journal of continuing education in the health professions, vol. 37, no. 3, pp. 195–206. http://doi.org/10.1097/ceh.0000000000000168 * currie, g., et  al., (2017) ‘twitter journal club in medical radiation science’, journal of medical imaging and radiation sciences, vol. 48, no. 1, pp. 83–89. http://doi.org/10.1016/j. jmir.2016.09.001 * davis, k. (2015) ‘teachers’ perceptions of twitter for professional development’, disability and rehabilitation, vol. 37, no. 17, pp. 1551–1558. http://doi.org/10.3109/09638288.2015. 1052576 * donelan, h. (2016) ‘social media for professional development and networking opportunities in academia’, journal of further and higher education, vol. 40, no. 5, pp. 706–729. http://doi.org/10.1080/0309877x.2015.1014321 * ford, c. & tolmie, d. (2016) ‘breaking the limits of time and space: how twitter is helping #medlibs collaborate and communicate. a descriptive study’, journal of hospital librarianship, vol. 16, no. 2, pp. 116–131. http://doi.org/10.1080/15323269.2016.1150737 * fuoco, m. & leveridge, m. j. (2015) ‘early adopters or laggards? attitudes toward and use of social media among urologists’, bju international, vol. 115, no. 3, pp. 491–497. http:// doi.org/10.1111/bju.12855 gao, f., luo, t. & zhang, k. (2012) ‘tweeting for learning: a critical analysis of research on microblogging in education published in 2008–2011’, british journal of educational technology, vol. 43, no. 5, pp. 783–801. http://doi.org/10.1111/j.1467-8535.2012.01357.x gibbs, g. r. (2012) el análisis de datos cualitativos en investigación cualitativa. madrid: ediciones morata, s. l. isbn: 978-84-7112-675-7 gonzález-sanmamed, m., et  al., (2018) ‘ecologías de aprendizaje en la era digital: desafíos para la educación superior’, publicaciones, vol. 48, no. 1, pp. 11–38. http://doi.org/10.30827/ publicaciones.v48i1.7329 * goodyear, v. a., casey, a. & quennerstedt, m. (2018) ‘social media as a tool for generating sustained and in-depth insights into sport and exercise practitioners’ ongoing practices’, qualitative research in sport, exercise and health, vol. 10, no. 1, pp. 1–16. http://doi.org/1 0.1080/2159676x.2017.1367715 * hamad, f., tbaishat, d. & al-fadel, m. (2017) ‘the role of social networks in enhancing the library profession and promoting academic library services: a comparative study of the university of jordan and al-balqaa’ applied university’, journal of librarianship and information science, vol. 49, no. 4, pp. 397–408. http://doi.org/10.1177/0961000616656043 * hart, m., et al., (2017a) ‘twitter and public health (part 1): how individual public health professionals use twitter for professional development’, jmir public health and surveillance, vol. 3, no. 3, p. e60. http://doi.org/10.2196/publichealth.6795 * hart, m., et al., (2017b) ‘twitter and public health (part 2): qualitative analysis of how individual health professionals outside organizations use microblogging to promote and disseminate health-related information’, jmir public health and surveillance, vol. 3, no. 4, p. e54. http://doi.org/10.2196/publichealth.6796 http://dx.doi.org/10.25304/rlt.v27.2286 http://doi.org/10.1007/s11606-015-3345-z https://doi.org/10.1111/j.1468-2419.2012.00410.x https://doi.org/10.1111/j.1468-2419.2012.00410.x http://doi.org/10.1093/bjsw/bcs208 http://doi.org/10.1097/ceh.0000000000000168 http://doi.org/10.1016/j.jmir.2016.09.001 http://doi.org/10.1016/j.jmir.2016.09.001 http://doi.org/10.3109/09638288.2015.​1052576 http://doi.org/10.3109/09638288.2015.​1052576 http://doi.org/10.1080/0309877x.2015.1014321 http://doi.org/10.1080/15323269.2016.1150737 http://doi.org/10.1111/bju.12855 http://doi.org/10.1111/bju.12855 http://doi.org/10.1111/j.1467-8535.2012.01357.x http://doi.org/10.30827/publicaciones.v48i1.7329 http://doi.org/10.30827/publicaciones.v48i1.7329 http://doi.org/10.1080/2159676x.2017.1367715 http://doi.org/10.1080/2159676x.2017.1367715 http://doi.org/10.1177/0961000616656043 http://doi.org/10.2196/publichealth.6795 http://doi.org/10.2196/publichealth.6796 c. bruguera et al. 16 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) hunter, l. j. & hall, c. m. (2018) ‘a survey of k-12 teachers’ utilization of social networks as a professional resource’, education and information technologies, vol. 23, no. 2, pp. 633–658. http://doi.org/10.1007/s10639-017-9627-9 hutchinson, j. (2016) ‘an introduction to digital media research methods: how to research and the implications of new media data’, communication research and practice, vol. 2, no. 1, pp. 1–6. http://doi.org/10.1080/22041451.2016.1155307 * jackson, h. t., et  al., (2017) ‘sages foregut surgery collaboration: a surgeon’s social media resource for collaboration, education and professional development’, surgical endoscopy and other interventional techniques, vol. 31, p. 43. http://doi.org/10.1007/s00464-017-5444-x * kelly, n. & antonio, a. (2016) ‘teacher peer support in social network sites’, teaching and teacher education, vol. 56, pp .138–149. http://doi.org/10.1016/j.tate.2016.02.007 * khan, a. & du, j. t. (2017) ‘professional development through social media applications: a study of female librarians in pakistan’, information and learning science, vol. 118, no. 7/8, pp. 342–353. http://doi.org/10.1108/ils-04-2017-0028 * klein, m., niebuhr, v. & d’alessandro, d. (2013) ‘innovative online faculty development utilizing the power of social media’, academic pediatrics, vol. 13, no. 6, pp. 564–569. http://doi.org/10.1016/j.acap.2013.07.005 * kleynhans, a. c., oosthuizen, a. h. & van hoving, d. j. (2017) ‘emergency medicine educational resource use in cape town: modern or traditional?’, postgraduate medical journal, vol. 93, pp. 250–255. http://doi.org/10.1136/postgradmedj-2016-134135 * krutka, d. g., carpenter, j. p. & trust, t. (2017) ‘enriching professional learning networks: a framework for identification, reflection, and intention’, techtrends, vol. 61, no. 3, pp. 246–252. http://doi.org/10.1007/s11528-016-0141-5 lawson, c. & cowling, c. (2014) ‘social media: the next frontier for professional development in radiography’, radiography, vol. 21, no. 2, pp. 74–80. http://doi.org/10.1016/j. radi.2014.11.006 * lee, c. s., et al., (2017) ‘making sense of comments on youtube educational videos: a self directed learning perspective’, online information review, vol. 41, no. 5, pp. 611–625. http://doi.org/10.1108/oir-09-2016-0274 * li, j. & greenhow, c. (2015) ‘scholars and social media: tweeting in the conference backchannel for professional learning’, educational media international, vol. 52, no. 1, pp. 1–14. http://doi.org/10.1080/09523987.2015.1005426 * li, x., et al., (2017) ‘understanding health professionals’ informal learning in online social networks: a cross-sectional survey’, studies in health technology and informatics, vol. 239, pp. 77–83. http://doi.org/10.3233/978-1-61499-783-2-77 * maloney, s., et al., (2017) ‘continuing professional development via social media or conference attendance: a cost analysis’, jmir medical education, vol. 3, no. 1, p. e5. http://doi. org/10.2196/mededu.6357 manca, s. & ranieri, m. (2015) ‘implications of social network sites for teaching and learning. where we are and where we want to go’, education and information technologies, vol. 22, no. 2, pp. 605–622. http://doi.org/10.1007/s10639-015-9429-x manca, s. & ranieri, m. (2016) ‘is facebook still a suitable technology-enhanced learning environment? an updated critical review of the literature from 2012 to 2015’, journal of computer assisted learning, vol. 32, no. 6, pp. 503–528. http://doi.org/10.1111/ jcal.12154 middleton, a. & beckingham, s. (2015) ‘social media for learning: a framework to inspire innovation’, in smart learning: teaching and learning with smartphones and tablets in post-compulsory education. melsig, sheffield hallam university, pp. 46–56. available at: http://shura.shu.ac.uk/9615/ ngai, e., tao, s. & moon, k. (2015) ‘social media research: theories, constructs, and conceptual frameworks’, international journal of information management, vol. 35, no. 1, pp. 33–44. http://doi.org/https://doi.org/10.1016/j.ijinfomgt.2014.09.004 http://dx.doi.org/10.25304/rlt.v27.2286 http://doi.org/10.1007/s10639-017-9627-9 http://doi.org/10.1080/22041451.2016.1155307 http://doi.org/10.1007/s00464-017-5444-x http://doi.org/10.1016/j.tate.2016.02.007 http://doi.org/10.1108/ils-04-2017-0028 http://doi.org/10.1016/j.acap.2013.07.005 http://doi.org/10.1136/postgradmedj-2016-134135 http://doi.org/10.1007/s11528-016-0141-5 http://doi.org/10.1016/j.radi.2014.11.006 http://doi.org/10.1016/j.radi.2014.11.006 http://doi.org/10.1108/oir-09-2016-0274 http://doi.org/10.1080/09523987.2015.1005426 http://doi.org/10.3233/978-1-61499-783-2-77 http://doi.org/10.2196/mededu.6357 http://doi.org/10.2196/mededu.6357 http://doi.org/10.1007/s10639-015-9429-x http://doi.org/10.1111/jcal.12154 http://doi.org/10.1111/jcal.12154 http://shura.shu.ac.uk/9615/ http://doi.org/https http://doi.org/10.1016/j.ijinfomgt.2014.09.004 research in learning technology citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 17 (page number not for citation purpose) avidov-ungar, o. (2016) ‘a model of professional development: teachers’ perceptions of their professional development’, teachers and teaching, vol. 22, no. 6, pp. 653–669. http://doi.org /10.1080/13540602.2016.1158955 pérez-latre, f. j., portilla, i. & blanco, c. s. (2011) ‘social networks, media and audiences: a literature review’, comunicación y sociedad, vol. 24, no. 1, pp. 63–74. isbn: 0214-0039 * rehm, m., littlejohn, a. & rienties, b. (2018) ‘does a formal wiki event contribute to the formation of a network of practice? a social capital perspective on the potential for informal learning’, interactive learning environments, vol. 26, no. 3, pp. 308–319. http://doi.org /10.1080/10494820.2017.1324495 * rehm, m. & notten, a. (2016) ‘twitter as an informal learning space for teachers!? the role of social capital in twitter conversations among teachers’, teaching and teacher education, vol. 60, pp. 215–223. http://doi.org/10.1016/j.tate.2016.08.015 * rickels, d. a. & brewer, w. d. (2017) ‘facebook band director’s group: member usage behaviors and perceived satisfaction for meeting professional development needs’, journal of music teacher education, vol. 26, no. 3, pp. 77–92. http://doi.org/10.1177/1057083717692380 roberts, m. j., et al., (2015) ‘globalization of continuing professional development by journal clubs via microblogging: a systematic review’, journal of medical internet research, vol. 17, no. 4, pp. 1–11. http://doi.org/10.2196/jmir.4194 * rodesiler, l. (2017) ‘local social media policies governing teachers’ professionally oriented participation online: a content analysis’, techtrends, vol. 61, no. 3, pp. 293–300. http://doi. org/10.1007/s11528-016-0139-z * romero-hall, e. (2017) ‘posting, sharing, networking, and connecting: use of social media content by graduate students’, techtrends, vol. 61, no. 6, pp. 580–588. http://doi. org/10.1007/s11528-017-0173-5 * schaper, e., et  al., (2013) ‘how do german veterinarians use social networks? a study, using the example of the “novice” veterinary medicine network’, gms zeitschrift fur medizinische ausbildung, vol. 30, no. 1, pp. 1–22. http://doi.org/10.3205/zma000855 schwenk, e. s., et al., (2017) ‘upgrading a social media strategy to increase twitter engagement during the spring annual meeting of the american society of regional anesthesia and pain medicine’, regional anesthesia and pain medicine, vol. 42, no. 3, pp. 283–288. issn: 10987339. http://doi.org/10.1097/aap.0000000000000586 smith, a. & anderson, m. (2018) social media use in 2018. washington, dc. available at: www.pewresearch.org sterling, m., et al., (2017) ‘the use of social media in graduate medical education: a systematic review’, academic medicine, vol. 92, no. 7, pp. 1043–1056. http://doi.org/10.1097/ acm.0000000000001617 strauss, a. l. & corbin, j. m. (2002) bases de la investigación cualitativa : técnicas y procedimientos para desarrollar la teoría fundada. medellín: editorial universidad de antioquía, facultad de enfermería de la universidad de antioquía. isbn: 9586556247 * sumuer, e., esfer, s. & yildirim, s. (2014) ‘teachers’ facebook use: their use habits, intensity, self-disclosure, privacy settings, and activities on facebook’, educational studies, vol. 40, no. 5, pp. 537–553. http://doi.org/10.1080/03055698.2014.952713 * thangasamy, i. a., et  al., (2014) ‘international urology journal club via twitter: 12-month experience’, european urology, vol. 66, no. 1, pp. 112–117. http://doi.org/10.1016/j. eururo.2014.01.034 tess, p. a. (2013) ‘the role of social media in higher education classes (real and virtual)-a literature review’, computers in human behavior, vol. 29, no. 5, pp. a60–a68. http://doi. org/10.1016/j.chb.2012.12.032 * trust, t. (2017) ‘motivation, empowerment, and innovation: teachers’ beliefs about how participating in the edmodo math subject community shapes teaching and learning’, journal of research on technology in education, vol. 49, no. 1–2, pp. 16–30. http://doi.org/ 10.1080/15391523.2017.1291317 http://dx.doi.org/10.25304/rlt.v27.2286 http://doi.org/10.1080/13540602.2016.1158955 http://doi.org/10.1080/13540602.2016.1158955 http://doi.org/10.1080/10494820.2017.1324495 http://doi.org/10.1080/10494820.2017.1324495 http://doi.org/10.1016/j.tate.2016.08.015 http://doi.org/10.1177/1057083717692380 http://doi.org/10.2196/jmir.4194 http://doi.org/10.1007/s11528-016-0139-z http://doi.org/10.1007/s11528-016-0139-z http://doi.org/10.1007/s11528-017-0173-5 http://doi.org/10.1007/s11528-017-0173-5 http://doi.org/10.3205/zma000855 http://doi.org/10.1097/aap.0000000000000586 http://www.pewresearch.org http://doi.org/10.1097/acm.0000000000001617 http://doi.org/10.1097/acm.0000000000001617 http://doi.org/10.1080/03055698.2014.952713 http://doi.org/10.1016/j.eururo.2014.01.034 http://doi.org/10.1016/j.eururo.2014.01.034 http://doi.org/10.1016/j.chb.2012.12.032 http://doi.org/10.1016/j.chb.2012.12.032 http://doi.org/10.1080/15391523.2017.1291317 http://doi.org/10.1080/15391523.2017.1291317 c. bruguera et al. 18 citation: research in learning technology 2019, 27: 2286 http://dx.doi.org/10.25304/rlt.v27.2286 (page number not for citation purpose) * tur, g. & marín, v. i. (2014) ‘enhancing learning with the social media: student teachers’ perceptions on twitter in a debate activity’, journal of new approaches in educational research, vol. 4, no. 1, pp. 46–59. http://doi.org/10.7821/naer.2015.1.102 * veletsianos, g. (2017) ‘three cases of hashtags used as learning and professional development environments’, techtrends, vol. 61, no. 3, pp. 284–292. http://doi.org/10.1007/ s11528-016-0143-3 * visser, r. d., evering, l. c. & barrett, d. e. (2014) ‘twitterforteachers: the implications of twitter as a self-directed professional development tool for k-12 teachers’, journal of research on technology in education, vol. 46, no. 4, pp. 396–413. http://doi.org/10.1080/15 391523.2014.925694 * wagner, j. p., et  al., (2017) ‘professional use of social media among surgeons: results of a multi-institutional study’, journal of surgical education, vol. 75, no. 3, pp. 804–810. http://doi.org/10.1016/j.jsurg.2017.09.008 * weber, z. a. & vincent, a. h. (2014) ‘facebook as a method to promote a mindset of continual learning in an ambulatory care pharmacy elective course’, currents in pharmacy teaching and learning, vol. 6, no. 4, pp. 478–482. http://doi.org/10.1016/j.cptl.2014.04.009 zimmer, m. & proferes, n. j. (2014) ‘a topology of twitter research: disciplines, methods, and ethics’, aslib journal of information management, vol. 66, no. 3, pp. 250–261. http://doi. org/10.1108/ajim-09-2013-0083 http://dx.doi.org/10.25304/rlt.v27.2286 http://doi.org/10.7821/naer.2015.1.102 http://doi.org/10.1007/s11528-016-0143-3 http://doi.org/10.1007/s11528-016-0143-3 http://doi.org/10.1080/15391523.2014.925694 http://doi.org/10.1080/15391523.2014.925694 http://doi.org/10.1016/j.jsurg.2017.09.008 http://doi.org/10.1016/j.cptl.2014.04.009 http://doi.org/10.1108/ajim-09-2013-0083 http://doi.org/10.1108/ajim-09-2013-0083 doi:10.3402/rlt.v19s1/7780 the development of a rich multimedia training environment for crisis management: using emotional affect to enhance learning liz bacon*, gill windall and lachlan mackinnon school of computing and mathematical sciences, university of greenwich, old royal naval college, park row, greenwich, london se10 9ls (received 2 march 2011; final version received 19 july 2011) pandora is an eu fp7-funded project developing a novel training and learning environment for gold commanders, individuals who carry executive responsibility for the services and facilities identified as strategically critical e.g. police, fire, in crisis management strategic planning situations. a key part of the work for this project is considering the emotional and behavioural state of the trainees, and the creation of more realistic, and thereby stressful, representations of multimedia information to impact on the decision-making of those trainees. existing training models are predominantly paper-based, table-top exercises, which require an exercise of imagination on the part of the trainees to consider not only the various aspects of a crisis situation but also the impacts of interventions, and remediating actions in the event of the failure of an intervention. however, existing computing models and tools are focused on supporting tactical and operational activities in crisis management, not strategic. therefore, the pandora system will provide a rich multimedia information environment, to provide trainees with the detailed information they require to develop strategic plans to deal with a crisis scenario, and will then provide information on the impacts of the implementation of those plans and provide the opportunity for the trainees to revise and remediate those plans. since this activity is invariably multi-agency, the training environment must support group-based strategic planning activities and trainees will occupy specific roles within the crisis scenario. the system will also provide a range of non-playing characters (npc) representing domain experts, high-level controllers (e.g. politicians, ministers), low-level controllers (tactical and operational commanders), and missing trainee roles, to ensure a fully populated scenario can be realised in each instantiation. within the environment, the emotional and behavioural state of the trainees will be monitored, and interventions, in the form of environmental information controls and mechanisms impacting on the stress levels and decisionmaking capabilities of the trainees, will be used to personalise the training environment. this approach enables a richer and more realistic representation of the crisis scenario to be enacted, leading to better strategic plans and providing trainees with structured feedback on their performance under stress. keywords: affective computing; augmented reality simulations; affective impact in elearning; timeline-based crisis scenarios; emotional markup 1. introduction it is often argued that learning under stress, for example in studying for exams or meeting deadlines for submission of coursework, focuses the mind and results in *corresponding author. email: e.bacon@gre.ac.uk alt-c 2011 conference proceedings 0151 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # l. bacon et al. this is an open access article distributed 67 under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7780 faster processing, storage and recall of information. while there is often debate about the retention of that information and the knowledge thereby gained, there can be no doubt that this model is frequently self-imposed by students. additionally, decisionmaking in stressful situations can be impacted by the affect of elements of the situation on the emotional and behavioural makeup of the decision-maker. this paper discusses the work of the eu fp7 project, pandora, and describes a system currently being developed which is designed to use emotional affect in order to impact decision-making and enhance learning. the application under development is designed to enhance and expand training exercises for gold commanders in crisis management. gold commanders are specifically engaged in the development of strategic plans to deal with a wide range of potential crisis situations that can arise in civil society. these crisis situations could be: . natural events, such as extreme weather, earthquake, landslides, etc. . transport events, such as plane, train or vehicle crashes. . service failures, such as electrical power plant failure, water supply failure, etc. . health crises, such as pandemics, epidemics, containment conditions. . technology failures, breakdown of automated control systems, central services. . policing and terrorism events. . some combination of some or all of the above. in order to develop strategic plans to deal with such situations, individuals who carry executive responsibility for the services and facilities identified as strategically critical within these situations e.g. police, fire, ambulance service, local authorities, health service, are expected to work together. these individuals are identified as gold commanders, and their role is explicitly strategic. they are in overall control of the emergency. however, they will not generally be at the site of the emergency, but typically co-located in a control room. they will set the direction and propose solutions for the tactical (silver) commanders to implement. silver commanders will also typically not be physically present at the site of the emergency but give direction to operational commanders (bronze) who are responsible for organising resources on the ground. in practice some gold commanders may also have tactical or operational responsibility. their objectives are to: save and protect life; relieve suffering; contain the emergency; provide the public with information; protect the health and safety of staff; safeguard the environment; protect property; maintain/restore critical services; maintain normal services appropriately; promote and facilitate self-help; facilitate the investigation/inquiry; facilitate community recovery and to evaluate and identify lessons learned. the training of gold commanders to prepare them to manage a crisis is very important and is currently typically undertaken in two ways: (1) through the use of table-top exercises: these are low cost, paper-based exercises, with some limited audio-visual input, undertaken by groups of gold commanders representing different emergency services etc. led by an expert trainer. these events take place in a dedicated training environment or in a standard meeting room at a gold commander venue, as required. the expert trainer provides guidance to the gold commanders on the case study l. bacon et al. 68 being used, tries to provide an intensive time constrained activity to simulate the pressure of a real crisis and provides feedback to the gold commanders after the event. this type of training exercise can be easily organised and is cheap to run but it lacks the authentic feel of a real crisis which would place the gold commanders under extreme pressure to make rapid and effective decisions. (2) real-world simulations: these train gold commanders in the field through the use of simulation exercises. these are very effective; however they are also extremely expensive, time consuming to set up and require specialist equipment etc. the purpose of these types of training events is to: . develop the collaborative skills of the trainees in formulating strategic responses across a number of organisations and events. . develop the strategic thinking of the trainees in considering the implications of their decisions and the effects on other services. . develop the responsive skills of trainees in formulating alternative strategies and remediating actions in the event of the failure of a strategic response. . determine the strategic planning ability, decision-making capability, flexibility and capability under pressure of the trainees. . develop skills to deal with the media, which are inevitably required in the event of a crisis. however, as outlined above, the typical table-top training model that is used has severe limitations in achieving these goals and is almost entirely dependent on the ability of the trainer to engage and motivate the trainees, and to assess their performance subjectively in the training event. when a crisis occurs, human behaviour and preparedness is critical to the delivery of an effective solution and therefore training needs to be as realistic as possible. it is important to be able to simulate the information overload and related stress, together with the pressure in making decisions. pandora therefore aims to bridge the gap between the low cost, table-top exercises and the expensive real-world simulations by providing an on-line e-learning environment in which the group and the trainer can participate in a realistic, dynamically changing, time sensitive, immersive crisis simulation exercise, that allows trainees to practice their decision making and negotiation skills within a realistic, stress-controlled environment (see figure 1). 2. background on affective and emotional computing one of the key features of the training system developed for pandora is that it should provide an environment that engages the trainees on an emotional level. ideally the trainees should experience emotions of a similar nature and intensity to those that they might experience when dealing with a real emergency. to this end their emotional state will be monitored and manipulated during the training in a variety of ways. this aspect of pandora is based on research into models of emotion, decision-making and learning. alt-c 2011 conference proceedings 69 research in neuroscience and psychology shows a strong connection between cognition and emotion. cognition plays an important role in creating emotions. emotions, in turn, cause a wide range of effects on attention, perception and cognitive processes involved in decision making, problem-solving and learning. often the word affect is used instead of emotion and indicates that a wider range of factors than those classically considered as emotions are involved. a special issue on ‘‘affective modelling and adaptation’’ of the user modelling and user-adapted interaction journal focuses on some issues that are relevant to pandora. in the introduction, affective computing is described as having four areas of interest. the first area is the analysis of affective states and the relationship between affection and cognition, such as learning. second is the automatic recognition of affective states e.g. through facial expression or physiological measurements. the third area is the adaptation of a system in response to the affective state of the person. the fourth aspect of affective computing concerns the design of avatars able to exhibit affective states. the second of these areas is considered the most difficult i.e. the ability to precisely and accurately recognise the affective state of a person. most of the work reported relates to tutoring systems and modelling affective states of learners (carberry and de rosis 2008). forbes-riley et al. (2008) demonstrate how frustration and uncertainty influence learning and show that adding affective state to learning models increases the level of accuracy of the model. d’mello et al. (2008) study the relationship between affect and features of interaction (such as the number of words in a student’s response, response time etc.) and show that affect can be recognised by these. they consider a wider range of states than forbes-riley et al. (2008). mcquiggan, mott, and lester (2008) try to identify the level of student self-efficacy and confidence. the initial results of this study show that physiological response may be a predictor of self-efficacy. the work of porayska-pomsta, mavrikis, and pain (2008) derives a set of rules to adapt interactions with learners to their affective state. yannakakis, hallam, and lund (2008) consider ways to estimate the level of engagement in games in order to adjust the virtual environment to the preferences of children. an important aspect of affective computing concerns the design of avatars able to exhibit affective states. the design and implementation of computational emotion figure 1. the virtual training environment. l. bacon et al. 70 models is needed in order to support this. marsella and gratch (2009) have carried out research aimed at building a stable computational model of emotions based on appraisal theory. figure 2 illustrates the relationship they see as existing between appraisal, emotion, coping and cognitive processes and illustrates the key sources of emotional dynamics. the ema model has been empirically assessed for three particular types of coping strategies: wishful thinking, resignation and distancing (marsella and gratch 2009). marsella, gratch and petta (2010) propose an analysis of the role and utility of computational models of emotions. psychological theories of emotions are typically formalised with a high level of abstraction through not very formalised natural language. this implies a high level of abstraction and a lack of detail and rigour. in contrast, computational models require a greater degree of precision since the theory must be implemented through a computational model. in this light computational models lead to the identification of underlying assumptions and complexity that are usually hidden and that need to be managed. computational models can then be seen as a way to substantiate theories as well as a framework for their construction. 3. description of pandora work to-date 3.1. modes of delivery the pandora crisis training room, which is where training is conducted, is designed to work in three different modes. these are: (1) single site training � this is where the training takes place in a physical room where the trainees and the trainer are co-located. the trainees sit around the table in the same way as they would have done for the paper-based table-top figure 2. theoretical assumptions of the relationship between appraisal, emotion, coping and cognition, and the sources of dynamics (marsella and gratch 2009). alt-c 2011 conference proceedings 71 exercise; however, with pandora, a range of consoles are used to provide multimedia information using sound, pictures, maps, animations, videos etc. for example, to simulate receiving information about the crisis such as a news broadcast. biometric sensors are also used to gather physiological information about the trainees to assist in an analysis of their stress levels etc. the trainer is able to configure the scenario to e.g. set up npc to role play an emergency service not represented within the group of trainees; subject matter experts; represent higher control (hicon) such as government ministers � these individuals would be above the level of gold commanders and have the authority to demand actions or constrain resources, and can impose their decisions on the crisis team and the scenario; lower control (locon) � these individuals represent the lower levels of command within the crisis team and can provide valuable feedback on the tactical level realisation of the strategy being developed by the gold commanders. (2) deployed training � this is essentially the same as for the single site training; however it is not delivered in a dedicated room, but elsewhere, for example at the site of one of the gold commanders taking part in the training. the pandora system, equipment and setup must therefore be portable to enable this delivery mode to be realised. (3) distributed training � in this mode, as shown in figure 3, the physical room is replaced by a virtual room and trainees participate through a web-based interface. the 3d virtual room contains npc. as with the other two modes these fulfil any key emergency service roles that are missing from the trainees. each trainee is represented by their own avatar. it provides the same multimedia channels as the physical room to provide the trainees with information on the unfolding crisis with which they have to deal. 3.2. pandora architecture the underpinning architecture of the system is the same for all three deployment modes and is made up of several key components which are described below: . the crisis module framework � this provides an event network to model a crisis scenario against a timeline, supporting the management of the training figure 3. in-world slide show, streaming video and map application. l. bacon et al. 72 process including the introduction of decision points for trainees incorporated into events within the crisis scenario. event network planning and mapping to timelines is managed through a knowledge-based approach, utilising rules stored in the crisis knowledge base. . the behavioural framework � this considers the behaviour of trainees, based on a pre-determined user model, and feedback from a variety of biometric sensors and the trainer during the training session. this component shows how a complete loop crisis-stimuli/trainee-reaction/pandora-behaviouranalysis can be implemented and shown to work in a training environment. . the trainer support framework � this allows the trainer to carry out three key functions: (1) the setup of a scenario for use with a particular group of trainees e.g. configuring an avatar to represent a missing trainee from one of the emergency services. (2) customise a training session and dynamically update a scenario whilst it is being executed e.g. by compressing the timeline in which events occur and/or to interject additional events, in order to increase the stress levels of one of more of the trainees. (3) record each run of the scenario so the trainer can review the training session after it has been completed with one or more trainees to reflect on the rationale for the decisions made and the alternative choices that could have been chosen during the simulation. . the emotion engine � this is a middleware component within the pandora system, providing facilities for the development, configuration and introduction of npc into the crisis scenario to interact with the trainees, and multimedia information assets, tagged for emotional affect. the npc framework also permits the trainer to take control of an npc to provide direct inputs, in specific events, to the trainees. the affective framework, which is a sub-component of the emotion engine, manages a repository of affectively tagged multimedia assets and uses inputs from the behavioural framework and local mashup rules to produce combinations of those assets to provide emotionally and behaviourally affective information to the trainees. the output of the emotion engine, generated through the environment framework builder, is a rendering specification describing the environmental conditions, multimedia information assets and npc to be generated in the training environment. . the emulated crisis room � in essence this is the trainee environment, since the rendering of the information generated from the other components is realised within this component. . integration of the above components is managed through a middleware model that has been developed for the project, and various test beds and test harnesses are also being constructed specifically to meet the needs of the pandora system. alt-c 2011 conference proceedings 73 3.3. design for emotional affect this section describes in more detail how the emotional affect is created by the pandora e-learning environment through both behavioural simulation and modelling, and the emotion engine. the focus of the behavioural simulation and modelling functionality is to: . select, model and monitor the relevant human factors or psychological variables that can influence decision making. . develop a model able to represent trainees’ actual behaviour/profile. . propose (plan) high-level personalised training goals and user interactions for the crisis planner. the trainee model takes into account both psycho�physiological parameters e.g. heartbeat rate, personality traits, self-efficacy and pedagogical parameters like training methods. user profiles are developed by asking the trainees to take neuropsychological and psychological tests in advance of the training in order to assess factors such as self-efficacy, self-estimate, affective style, anxiety etc. the pandora system will then determine a personalised training path for each trainee, customised with difficulty levels and challenges. pandora must also have the ability to relate a user’s emotional and psychological aspects within a computational user model that is able to recreate physical effects. the emotion engine has been designed with two components outlined as follows: (1) an npc framework, which can provide representations ranging from simple text information through to full avatar representation (dependent on the mode of deployment of the system) and can represent emotion within those characters within the limitations of the media format used and the current state of the art in emotion representation. (2) an affective state framework, which maintains a repository of multimedia assets related to the training scenario, tagged according to potential emotional affect and linked, where appropriate, to events in the scenario event network. this framework can combine assets using mashup rules to create emotionally affective multimedia artefacts, subject to the requirements generated from the behavioural framework. the emotional and behavioural condition of the trainees will be monitored in the behavioural framework, which will provide input to the affective state framework, primarily associated with determining the level of affective input to provide to the trainees, individually or as a group. this information can be pre-determined within the event network, dynamically created by the behavioural framework on an individual or group basis, or input directly by the trainer through that framework. within the affective state framework a local multimedia asset store will provide a repository for a wide variety of multimedia assets developed to support the scenarios. each of these assets will be meta-tagged with an xml emotion mark-up language specification, adapted specifically for pandora, which will provide standard information on the type and nature of the asset, the media channels for l. bacon et al. 74 which it is appropriate, the potential for combination with other multimedia assets, duration, etc. it will also indicate an affective level that the asset individually can be expected to engender in the trainees, based on an affective scale defined and categorised for the pandora project. these assets can also be specifically tagged for use in specified events in training scenarios, to support the rapid selection of assets. the key concept to consider in the development of multimedia information representations for the trainees is the affective impact that combinations of these assets may have. it is standard practice in film and tv design and production to utilise combinations of environmental outputs to generate affective conditions, often referred to as ambience. while in the pandora environment there will be no background music to create mood, it will be possible to use audio effects to reinforce outputs generated from a crisis situation. the use of video images combined with tv style voice-overs will permit differential levels of anxiety in the voice-over to be associated with the same images, permitting reuse to change the affective impact related to the same information, and a number of similar techniques may be applied to video and audio information. by using an xml-based tagging for multimedia assets, it becomes possible to use rule-based mechanisms to combine multimedia assets and create mashups to achieve a desired level of emotional affect within the environment. information management techniques, such as overloading the media channels with multiple inputs, noisy media channels, missing and incomplete information, and media channel failures, can all also be used to impact the stress level of the trainees. given that the system can identify the current emotional and behavioural level of the trainees, and for the purposes of the training event a target emotional level can be identified, then the affective state framework can be used to manage the information presentation to the trainees to impact on their emotional state on a trajectory towards the target level. the efficacy of affective impact of particular approaches will be determined from changes to the emotional and behavioural level of the trainees, determined by the behavioural framework. this information will be used to enable the system to learn and adapt the rules and techniques to provide more effective affective controls. 3.4. summary of key pandora features the pandora system addresses the shortcomings of the existing training model, enhances the range and scope of the training events and offers the potential for future development by: . offering a fully featured multimedia environment to provide information to the trainees, including audio, video, maps, texts, email, graphics and text. . developing a structured, timeline-based, sequence of events, crisis scenario model running in a computer-based simulation environment controlled by the trainer. . providing real-time operational inputs demonstrating strategic decision outcomes to trainees, asking them to dynamically revise strategic plans and decisions. . capturing trainee behaviour and emotional state, through the use of pre-event information capture, direct sensor inputs, self-reporting by trainees, and alt-c 2011 conference proceedings 75 trainer inputs, and using affective media effects and information presentation techniques to induce changes to those behavioural and emotional states. . providing a graphical virtual representation of the training environment to support on-line distributed training events. . providing virtual characters, in any form from textual through to full animation, to engage in the event, including replacements for missing trainees, to ensure that the full scenario enactment is supported in all training events. . providing the trainer with a full control system for the training event, including the ability to change events, add new events, expand and compress timelines, provide direct interventions into the scenario and increase or decrease the emotional stress applied to individual trainees. . maintaining a detailed log of the training event, to permit rerun of some or all events, modelling of individual trainee performance and capture of relevant and useful events as exemplars for future training. . maintaining configurable scenario models, knowledge, multimedia asset and databases to enable the system to build a wide range of crisis scenarios, to use as training events for those involved in crisis management at all levels. 4. contribution to the emotionml standard emotion markup language (emotionml) is an xml-based language designed to represent emotion in a machine-readable manner. it is going through the w3c standards process and is currently at the ‘working draft’ stage (w3c 2010a). emotionml is intended to be used as a plug-in language i.e. to be used in conjunction with other xml languages such as smil. the main areas in which emotionml is expected to be applied are: (1) annotation of data and media. (2) recognition of emotion expressed by people. (3) simulation of emotion by technological artefacts. all three of these areas are relevant to pandora, and so from the inception of the project, emotionml was considered as a likely technology for use in its implementation. detailed consideration of how emotionml could be used to implement aspects of pandora revealed several potential issues with the version 1.0 or the draft standard. three of these are outlined below: (1) emotionml considers scales (e.g. for representing intensity of emotion) to be continuous, linear values. this may not be adequate to capture important information in pandora. in some cases scales may need to be logarithmic or take into account the notion of a ‘tipping point’. in other cases it may be important to be able to represent discrete values. (2) emotionml allows for different vocabularies (e.g. for representing the category of emotion). the need for this is the lack of agreement amongst professionals, such as psychologists, about a single vocabulary. the problem of having a multiplicity of vocabularies, especially with none being specified as a default, is that of interoperability. as pandora moves beyond a pilot l. bacon et al. 76 project it is likely it will need to interact with external systems e.g. to use media originated and annotated externally and so interoperability will be important. (3) emotionml allows for emotions to be tagged with a timestamp. in pandora data received from monitoring the trainees’ affective state will be received in a stream throughout training and it will be useful to be able to represent the identification of emotion as starting at a time offset within the session. these issues and a description of planned use-cases for emotionml in implementing pandora were discussed at a w3c emotionml workshop in october 2010 and will be taken into account when revising the draft standard (mackinnon, windall, and bacon 2010; w3c 2010b). 5. conclusion and future work the pandora project will produce an advanced training system, targeted specifically at gold commanders in crisis management scenarios. since gold commanders represent the strategic level of crisis planning, improving the training and thereby the efficacy of their strategic thinking and the design of their remediation plans will have a significant beneficial effect in the handling of a variety of different crisis. better crisis management will have significant socio-economic impacts, in terms of reduced casualty rates, faster and more efficient remediation, reduced loss of working time, reduced loss of productivity and improved coordination of expensive resources. additionally, since the project will provide different deployment models for the training scenarios, it becomes possible to train larger numbers both at strategic and tactical levels, utilising distributed virtualised representations of information, and thereby advance the training scenarios into fully immersive digital environments. such an approach will enable the use of varied training scenarios that are too expensive for physical simulations to be realised in virtual form, thereby enabling training activities that are not currently practical. again this will have significant socio-economic benefits, in the ability of crisis managers to develop more wide-ranging, complex and detailed strategies and remediating actions to deal with the ever-growing range of crises that they might be called on to manage. the use of the pandora system in different partner countries within europe will also support the sharing of best practices in crisis management, scenario information and experiences, and will promote understanding of different response modes related to cultural, legal and social variations, which would be of particular importance when dealing with crises that cross national boundaries. whilst the pandora system is currently being developed for crisis management training, the e-learning architecture and component model is not specific to this particular situation and could be used for a variety of different training needs, since the key component is a scenario that can be modelled as a set of discrete events against a timeline. we can envisage a large number of different application areas ranging from business planning, through health and social care, to regional infrastructure planning, all of which could be modelled within the pandora system and then used for simulation and training purposes. in fact, if we consider that the pandora system offers a visualisation and simulation environment to alt-c 2011 conference proceedings 77 support event network-based scenarios, we can consider its use for almost any timeline-based process. the benefits of the training environment, which include the modelling of the behaviour of trainees, the potential for customisation of the immediate training session by the trainer and the introduction of affective elements to impact the emotions and behaviours of the trainees, provide a range of facilities that could be utilised in different ways. using the pandora approach to provide training for a variety of different sectors and scenarios would give europe a significant lead in the use of visualisation and simulation technologies to provide learning experiences that would otherwise be too expensive, too dangerous or simply impracticable for the general workforce. the socio-economic and social impacts of the widespread use of high-quality simulation and visualisation in distributed virtual environments to provide realistic learning experiences would be extremely significant. acknowledgements this paper is a product of research and development by consortium members of the pandora project funded by an eu grant under fp7-ict-sec-2007-1, grant number 225387. references carberry, s., and f. de rosis. 2008. introduction to special issue on ‘‘affective modelling and adaptation’’. user modelling and user-adapted interaction 18, no. 1�2 (february): 1�9. d’mello, s., s. craig, a. witherspoon, b. mcdaniel, a. graesser. 2008. automatic detection of learner’s affect from conversational cues. user modelling and user-adapted interaction 18, no. 1�2 (february): 45�80. forbes-riley, k., m., rotaru, and d.j. litman. 2008. the relative impact of student affect on performance models in a spoken dialogue tutoring system. user modelling and useradapted interaction 18, no. 1�2 (february): 11�43. mackinnon, l., g. windall, and l. bacon. 2010. tracking and influencing trainee emotions in a crisis-planning scenario, w3c. http://www.w3.org/2010/10/emotionml/papers/mackinnon.pdf. (accessed july 19, 2011). marsella, s., and j. gratch. 2009. ema: a process model of appraisal dynamics. cognitive systems research 10, no. 1: 70�90. marsella, s., j. gratch, and p. petta. 2010. computational models of emotion. in a blueprint for an affectively competent agent: cross-fertilization between emotion psychology, affective neuroscience, and affective computing, ed. k.r. scherer, t. bänziger, and e. roesch. oxford: oxford university press. mcquiggan, s., b.w. mott, and j. lester. 2008. modelling self-efficacy in intelligent tutoring systems: an inductive approach. user model. user modelling and user-adapted interaction 18, no. 1�2 (february): 81�123. porayska-pomsta, k., m. mavrikis, and h. pain. 2008. diagnosing and acting on student affect: the tutor’s perspective. user modelling and user-adapted interaction: the journal of personalization research 18: 125�73. w3c. 2010a. emotion markup language (emotionml) 1.0. http://www.w3.org/tr/ emotionml/. (accessed july 19, 2011). w3c. 2010b. w3c emotion markup language workshop � summary. http://www.w3.org/2010/ 10/emotionml/summary.html. (accessed july 19, 2011). yannakakis, g.n., j. hallam, and h.h. lund. 2008. entertainment capture through heart rate activity in physical interactive playgrounds. user modelling and user-adapted interaction, special issue: affective modelling and adaptation 18, no. 1�2 (february): 207�243. l. bacon et al. 78 http://www.w3.org/2010/10/emotionml/papers/mackinnon.pdf http://www.w3.org/2010/10/emotionml/papers/mackinnon.pdf http://www.w3.org/tr/emotionml/ http://www.w3.org/tr/emotionml/ http://www.w3.org/2010/10/emotionml/summary.html http://www.w3.org/2010/10/emotionml/summary.html (page number not for citation purpose) 1 *corresponding author. email: jennifer.vanallen@lehman.cuny.edu research in learning technology 2023. © 2023 j. van allen and s. katz. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 research in learning technology vol. 31, 2023 original research article viewing open education within the technological, pedagogical, content framework: illustrating educator knowledge, skills and dispositions jennifer van allena* and stacy katzb adepartment of counseling, leadership, literacy, and special education, lehman college, city university of new york, new york, ny, united states; bleonard lief library, lehman college, city university of new york, new york, ny, united states (received: 26 may 2022; revised: 11 november 2022; accepted: 7 january 2023; published: 29 march 2023) without interrogation, educators may reproduce hegemonic materials and learning opportunities that are simply easier to access in open educational practices (oep). thus, we argue that to effectively engage in oep, educators must not only possess knowledge, skills and dispositions related to their discipline, but also related to open education, cc licensing, open pedagogy, digital tools that facilitate oep, and intentionality and care in negotiating openness with students. while there are various frameworks for open education, none have been applied to explain what knowledge, skills and dispositions are needed to engage in oep. in this manuscript, we seek to conceptualise and provide examples of oep within the technological, pedagogical, content knowledge framework (tpack) through the intersections of content, technology, and pedagogy with equity, intentionality, and care at the forefront. keywords: open educational practices; open pedagogy; tpack introduction empowered by a desire for more equitable and inclusive access to knowledge, the open education movement is gaining momentum (william and flora hewlett foundation 2020). in 2015, the united nations (un) released sustainable development goals calling for a global partnership of all countries to achieve a better future for people worldwide by 2030 (un 2015). key among these goals is a commitment to ‘ensure inclusive and equitable quality education and promote lifelong learning for all’ (un 2015, p. 17). the open education movement is driven by the belief that learning materials and experiences should be available to everyone; however, this belief goes beyond just access. in constructing an open education ecosystem, skilled individuals must develop teaching and learning materials and educational opportunities that they are willing to proactively share beyond restrictions imposed by copyright and other barriers to access (hegarty 2015). sharing and participation undergird openness, as ‘open’ mailto:jennifer.vanallen@lehman.cuny.edu http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2829 j. van allen and s. katz 2 citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 (page number not for citation purpose) is an umbrella term applied to resources, instructional practices, institutional policies and educational technology applications. thus, openness is viewed as a community practice that values knowledge and resource sharing (wiley and green 2012). open educational practices (oep) embody teaching and learning approaches that draw on collaborative and flexible learning environments, social and participatory learning, knowledge creation and sharing, learner activity and agency and open sharing (cronin and maclaren 2018). open educational resources (oer), learning objects that are shared openly with a creative commons (cc) license, are often, but not always, central to oep (cronin and maclaren 2018). the permissions provided by cc licensing permit all users to use, adapt, and remix oer, and reshare the resources within a global community, which is prohibited under traditional copyright. while oer is content-centred, oep is process-centred, encouraging teachers to enhance equitable teaching and learning practices through openness (bali, cronin, and jhangiani 2020). accordingly, teaching and learning materials and educational opportunities are created that can continually grow and be updated for accessibility, cultural relevance, inclusivity and timeliness. although frameworks discussing the continuum of openness (hilton et al. 2010), attributes of open pedagogy (hegarty 2015), and social justice implications of oep (bali, cronin, and jhangiani 2020) exist, none have been applied to explain what knowledge, skills and dispositions are needed to engage in oep. widely used in educational technology, the technological, pedagogical, content knowledge framework (tpack) provides educators with a map for understanding how to successfully integrate technology into curriculum and instruction (koehler and mishra 2009). oep may be conceived through tpack since the creation, adaptation and remixing of resources require educators to purposefully integrate technology as a teaching tool. in this manuscript, we seek to conceptualise and provide examples of oep within the tpack framework through the intersections of content, technology, and pedagogy with equity, intentionality, and care at the forefront. we first frame our work in conversations of tpack, open education, social justice and critical digital pedagogy. then, we parse the intersections of open and tpack to show educators the knowledge, skills and dispositions needed to teach effectively, as well as the decisions educators make, as they teach openly. these connections are supported by examples and suggested tools and practices to encourage educators to teach with oep with considerations for an equitable learning environment. framing perspectives in order to engage in a discussion of oep through the tpack framework, we review the tenets of each guiding perspective. these next sections describe the elements of the tpack framework and frame the need for intentionality and criticality in current conversations of open education and digital pedagogy. tpack framework building on shulman’s (1986) notions of content and pedagogical knowledge, mishra and koehler (2006) proposed the tpack framework which added technological knowledge. arguing that simply adding technology to teaching is not enough, they examine specific knowledge needed to meaningfully integrate technology into http://dx.doi.org/10.25304/rlt.v31.2829 research in learning technology citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 3 (page number not for citation purpose) educational processes by first considering content and then merging effective instructional methods for teaching that content with technology. thus, three interrelated bases of knowledge are blended in complex, and often nuanced, ways. content knowledge is expertise in the subject matter one is teaching, such as facts, concepts, theories and procedures, how these ideas are connected, and evidence that supports this disciplinary knowledge. pedagogical knowledge refers to a deep understanding of how individuals learn, knowledge of various teaching methods and practices, and overall aims, purposes, and values of education. technological knowledge specifies knowledge of various tools, devices, software, applications, etc., along with the necessary skills to utilise these technologies and adapt to an ever-changing technological landscape. although many today narrowly view technological knowledge as 21st-century digital technologies and tools, mishra and koehler (2006) originally defined the term more broadly to include any tool that can be used in teaching and learning, such as books and whiteboards as well as computers and multimedia. figure 1. the technological pedagogical content knowledge (tpack) framework. source: reproduced by permission of the publisher, © 2012 by tpack.org. http://dx.doi.org/10.25304/rlt.v31.2829 http://tpack.org j. van allen and s. katz 4 citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 (page number not for citation purpose) the interactions between and among content, pedagogy and technology are depicted in the tpack framework (see figure 1; koehler and mishra 2009). these interactions are based on the belief that technology ‘integration efforts should be creatively designed or structured for particular subject matter ideas in specific classroom contexts’ (koehler and mishra 2009, p. 62). for example, an instructor who uses chemistry lab classes to lecture from powerpoints laden with bullet points of the content may demonstrate exemplary disciplinary knowledge but is not maximising their classroom time pedagogically for hands-on learning or making effective use of interactive visuals afforded by technology. from this perspective, educators must interweave their knowledge to design learning experiences that flexibly match their content areas and their contexts, which are inherently different in every classroom, with technology and effective instructional approaches. pedagogical content knowledge involves applying a variety of instructional methods to teach content in the most effective ways based on students’ prior knowledge, such as facilitating experiments to discover concepts rather than simply reading about them. technological pedagogical knowledge examines how technology can be utilised while teaching, such as showing video clips of a procedure versus modelling the procedure on the whiteboard. technological content knowledge represents how technology and content knowledge influence each other, such as tools utilised in different subject areas. for example, a middle school science instructor may demonstrate chemical reactions using a phet interactive simulation of chemical reactions which would be too dangerous to facilitate physically in the classroom. technological pedagogical content knowledge is the basis of effective teaching with technology. it is at the intersection of technological pedagogical content knowledge that educators must make key decisions for their students and classrooms based on knowing how and when to use technology in constructive ways to teach content. implications from this framework have impacted educational technology research (niess 2019), professional development of technology integration (pareto and willermark 2019), and teacher education (baran et al. 2020). educators interested in learning more about tpack can find valuable resources and good examples of implementation and planning in the literature, particularly in the special issue of the asia-pacific education researcher on tpack (koh, chai, and lee 2015). open education in discussions of open education, many tout the benefits for students of such initiatives. with the rising costs of textbooks, between 2013 and 2018, oer textbook use in introductory higher education courses was estimated to save students $1 billion worldwide (allen 2018). additionally, research shows that courses using oer provide students with easier access to course materials (hilton et al. 2019) and result in greater satisfaction with learning materials (ross, hendricks, and mowat 2018). these benefits are even greater when educators facilitate participatory, collaborative learning experiences encouraged in oep. open pedagogy deconstructs the traditional student–teacher relationship by providing space for and empowering students to actively engage in dialogue about concepts, curation of resources and co-creation of content (paskevicius 2017). derosa and jhangiani (2017) explain, http://dx.doi.org/10.25304/rlt.v31.2829 https://phet.colorado.edu/en/simulations/filter?subjects=chemistry&type=html,prototype https://phet.colorado.edu/en/simulations/filter?subjects=chemistry&type=html,prototype research in learning technology citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 5 (page number not for citation purpose) ‘open pedagogy, as we engage with it, is a site of praxis, a place where theories about learning, teaching, technology, and social justice enter into a conversation with each other . . . as a process of designing architectures and using tools for learning that enable students to shape the public knowledge commons of which they are a part.’ (para. 2) this pedagogical shift positions the instructor and students as co-constructors of knowledge motivating students to take more ownership of their learning and providing more equitable learning experiences that value diverse student voices (paskevicius 2017). although the utopian ideal of open education seems to meet the calls for inclusive and equitable education for all, without interrogation educators may reproduce hegemonic materials and learning opportunities that are simply easier to access (jhangiani 2019). therefore, we argue that to effectively engage in oep, educators must not only possess knowledge, skills and dispositions related to their discipline, but also related to open education, cc licensing, open pedagogy, digital tools that facilitate oep, and intentionality and care in negotiating openness with students. open education and social justice questions such as ‘open for whom?’ encourage critical examination of open education and oep placing principles of social justice and equity at the forefront by taking into account social, cultural and political contexts. lambert (2018) examined how social justice has been noticeably absent from open education literature. she posits that open education should be prioritised ‘by and for the benefit and empowerment of non-privileged learners who may be under-represented in education systems or marginalised in their global context’ (lambert 2018, p. 239). drawn from critical theory, cronin (2020) proposes the following questions about power and participation in openness: • who defines openness? • who is included and who is excluded when education is ‘opened’, and in what ways? • to what extent, by whom, in what contexts, and in what ways do specific open education initiatives achieve their stated aims of increasing access, fostering inclusivity, enhancing learning, developing capacity and agency, and empowering individuals, groups, and communities, if at all? • can open education initiatives, in practice, do the opposite of what they are intended to do? • what does emancipatory open education look like? (p. 5) these questions encourage educators to analyse and, subsequently, reconceptualise open education in ways that expand access, participation and social justice. for example, textbook affordability is a barrier for historically underserved students; jenkins et al. (2020) found that oer adoption fostered social justice through increased access and cost savings for students. focusing specifically on oep, bali, cronin, and jhangiani (2020) analysed how specific pedagogical practices impact economic, cultural and political social justice efforts in transformative, ameliorative, neutral or even negative ways. as an example, asking students to publicly blog about class content can be ameliorative by giving voice to groups who are often left out of public scholarship or transformative by allowing marginalised groups to challenge hegemonic ideals of http://dx.doi.org/10.25304/rlt.v31.2829 j. van allen and s. katz 6 citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 (page number not for citation purpose) academic knowledge. at the same time, public blogging can be negative for those who may be more vulnerable through a public online presence (e.g. witness protection, political surveillance) and those with minimal digital literacies or internet access (bali, cronin, and jhangiani 2020). in these critiques, we acknowledge that oep doesn’t automatically further social justice without intentional and thoughtful design. critical digital pedagogy according to collins and halverson (2018), inequities between those in different social classes continue to expand because technical advancements afford social and cultural advantages to the wealthy. therefore, educators must think critically about the nature and effects of technology and the tools they are using with students. stommel (2014) explains, ‘critical digital pedagogy demands that open and networked educational environments must not be merely repositories of content. they must be platforms for engaging students and teachers as full agents of their own learning’ (para. 18). from the perspective of critical digital pedagogy, it is clear that educators who are designing curriculum and instruction steeped in oep require a plethora of requisite knowledge, skills and dispositions. for example, educators seeking equitable learning experiences must take a critical stance in examining the affordances and constraints of digital tools (jhangiani 2019). educators must understand and develop their own digital literacy skills required to interact with these tools and know methods for developing and nurturing students’ digital literacy skills and digital identities. simultaneously, they should consider issues of access to digital tools and skills (see gilliard and culik’s 2016 work on digital redlining) and the priorities of their institutions and students as influenced by political, cultural and social contexts (cronin 2020). as previously discussed, these aspects should be critically considered alongside disciplinary applications as there will likely be variations based on the content to be taught. the intersections of tpack and oep similar to the argument proposed by mishra and koehler (2006) about integrating technology into instruction, we believe that merely integrating open practices into the curriculum is not enough to ensure social justice and equity. instead, there are layered complexities in considering how, and the extent to which, oer content (content knowledge), open pedagogy (pedagogical knowledge) and digital tools utilised in these practices (technological knowledge) increase access, foster inclusivity, enhance learning, develop capacity and agency and empower learners when designing curriculum and instruction. while others have designed oer using the tpack model to guide content (oer in mathematics professional development project, n.d.), designed oer about tpack (hofer and harris 2016), and recently proposed that oer be viewed through tpack (grotewold, kohler, and krimbill 2022), to our knowledge no one has fully analysed oep through tpack. yet, when oep is layered onto the tpack framework, educators become more aware of the content, pedagogy and technology knowledge needed specifically related to open practices and can consider these aspects to design more socially just learning environments. in the following sections, we explicate open knowledge connections within these areas with examples from a variety of disciplines. http://dx.doi.org/10.25304/rlt.v31.2829 research in learning technology citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 7 (page number not for citation purpose) content knowledge connections when engaging in oep, educators must possess knowledge of their disciplines as well as knowledge specific to open education. for many, basic understanding begins with the 5r’s which define the five rights of oer: retain, reuse, revise, remix and redistribute. given that oer can take various forms from books to full courses to audio and video files (and more!), the 5r’s provide a framework for thinking about the principles associated with open education. other essential content knowledge of open education includes identifying the various cc licensing symbols (see figure 2), understanding the permissions afforded by each, and examining the affordances, constraints, and impact of the licenses for the materials they use or choose to place on oer they create. for example, one must understand that selecting a resource with an nd (no derivatives) license constrains how the resource may be used. since no derivatives are allowed, educators may not remix chapters or sections of an nd textbook to create a new one contextualised for their local community, such as translations. selecting a license to apply to resources can become further complicated when one chooses to adapt or remix material from sources with multiple cc licenses (see fazzino and turley 2019). while cc represents the licenses most closely associated with written work, music and video, other open license designations may be associated with different disciplines, such as general public license (gnu) for open-source software. educators should also check their figure 2. creative commons terms and licensing symbols. http://dx.doi.org/10.25304/rlt.v31.2829 j. van allen and s. katz 8 citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 (page number not for citation purpose) employer’s policies to determine the copyright holder of the materials they create and if they can openly license these resources. utilising oer also involves knowledge of attribution practices; this differs from citation which can cause confusion for educators and students. an attribution statement is a legal requirement of all the cc licenses as all of the licenses require cc attribution (cc-by), which provides credit to the creator and marks the work as open content. additionally, although attribution statements may vary in formatting, the title, author, source and license should always be included. works that are a derivative of an original resource must include the information of original source along with the title, author and license of the derivative. when sharing work openly, educators should be aware of the decisions that are made by individuals in the process. cronin (2017) explains that there are four levels of balancing privacy and openness in open practices: macro (global level), meso (community or network level), micro (individual level) and nano (interaction level). at the macro level, one must determine whether or not they will participate in open sharing of their work or open networking with others. those who choose to share must then identify who they want to share with (meso level; e.g. the class, a specific community/ organisation or the wider public) and who they want to share as (micro level; e.g. their name, a pseudonym or anonymously). for example, educators may invite students to share work with the wider public on a general website as a class contribution or within a virtual learning environment as an individual contribution (see example in jeffrey et al. 2021). finally, individuals must negotiate what they will share and if they will interact with others at the nano level (cronin 2017). understanding the types of decisions that are made by individuals engaging in open practices is essential for future pedagogical practices. pedagogical knowledge connections while the content knowledge required by educators to engage in oep largely focuses on technical aspects of oer and openness, pedagogical knowledge includes awareness of and skill in implementing various instructional practices associated with open pedagogy. in the process of finding open materials, educators exhibit pedagogical knowledge by selecting lessons and resources that meet their class’ needs (jung, sasaki, and latchem 2016). for example, in selecting an open textbook for a course, educators may carefully consider the population of students they serve and evaluate textbooks to ensure their communities are represented and recognised therein. as discussed previously, open pedagogy prioritises students as contributors in the creation of knowledge shared publicly and openly. oep requires educators to commit to student-centred learning that fosters agency and encourages participation in knowledge creation for the public commons (derosa and jhangiani 2017). therefore, pedagogical knowledge is demonstrated by selecting and implementing activities and assessments that address learning outcomes and utilise open practices, such as those listed in table 1. pedagogical content knowledge connections becoming aware of various oep allows educators to flexibly apply different methods for actively engaging students in their discipline-specific content and demonstrate pedagogical content knowledge. for example, a history instructor whose learning http://dx.doi.org/10.25304/rlt.v31.2829 research in learning technology citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 9 (page number not for citation purpose) table 1. open pedagogy activities and assessments. open activities description edtech tools example(s) of use social annotation social annotation involves educators and learners commenting on a common text (written, video, image or audio) through a digital platform in a collaborative environment. students support one another in sharing their connections, thoughts and perspectives on the text through comments and replies to others. as a result, students read multiple viewpoints of their peers and collaboratively deepen their understanding. hypothes.is perusall nowcomment the marginal syllabus and open pedagogy renewable assignments renewable assignments are tasks in which students compile and openly publish their work so that the assignment outcome is inherently valuable to the community. in some cases, renewable assignments may be originally developed by the students, and in others, students may remix or adapt existing oer. varies based on task text processing software google docs websites evolving into the open wikipedia assignments wikipedia assignments provide an opportunity for students to examine wikipedia articles to identify gaps in knowledge, inaccuracies, biases, etc. in the information. then, using scholarly sources, they create a new wikipedia article, add to or edit an existing one, add citations, and/or add visuals. through wikipedia assignments, students develop information and digital literacy skills, writing and communication skills, reference skills and critical thinking. wikipedia wiki edu teaching wikipedia blogging educators can encourage students to show their understanding of course content through blogging. blogs may be used in classes in many ways such as entries that are responses to a prompt, responses to readings, reflective or exploratory entries, showcases of student work or open-ended entries related to the content being addressed. educators may set up class sites where students can create their blogs or encourage students to blog on a personal site that is linked to the class site. learning management systems often allow educators to set up class blogs within the system for students who do not wish to share their entries publicly. wordpress discussion strategies continued http://dx.doi.org/10.25304/rlt.v31.2829 https://web.hypothes.is/ https://perusall.com/ https://nowcomment.com/ http://openpedagogy.org/course-level/the-marginal-syllabus-and-open-pedagogy/ http://openpedagogy.org/course-level/the-marginal-syllabus-and-open-pedagogy/ http://openpedagogy.org/course-level/the-marginal-syllabus-and-open-pedagogy/ https://www.google.com/docs/about/ https://milnepublishing.geneseo.edu/openpedagogyapproaches/chapter/evolving-into-the-open-a-framework-for-collaborative-design-of-renewable-assignments/ https://milnepublishing.geneseo.edu/openpedagogyapproaches/chapter/evolving-into-the-open-a-framework-for-collaborative-design-of-renewable-assignments/ https://wikiedu.org/teach-with-wikipedia/ https://milnepublishing.geneseo.edu/openpedagogyapproaches/chapter/teaching-wikipedia-a-model-for-critical-engagement-with-open-information/ https://milnepublishing.geneseo.edu/openpedagogyapproaches/chapter/teaching-wikipedia-a-model-for-critical-engagement-with-open-information/ https://wordpress.com/ https://vp.commons.gc.cuny.edu/2017/03/10/discussion-strategies-sustained-groups-and-online-discussions/ https://vp.commons.gc.cuny.edu/2017/03/10/discussion-strategies-sustained-groups-and-online-discussions/ j. van allen and s. katz 10 citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 (page number not for citation purpose) objective is to compare and contrast two accounts of a historical event may have students engage in social annotation of the texts. social annotation encourages close reading and allows students to view the connections others made between the two texts (bakermans et al. 2022). whereas, a mathematics instructor with a learning objective of communicating mathematical ideas to others may have students blog about the steps they took to solve a mathematical equation. as students blog about their process, they clarify their own understandings and can also view others’ solutions to see the variety of methods that can be applied to arrive at one solution (dewaard and roberts 2021). when implementing oep, pedagogical content knowledge also reflects care when educators intentionally support students in making decisions about their level of openness while balancing privacy concerns. cronin (2017) notes that openness is ‘complex, personal, contextual, and continually negotiated’ (p. 28). highlighted table 1. continued open activities description edtech tools example(s) of use design oers by adapting, remixing or creating oer with students, educators help students contribute to knowledge. this can be as simple as creating problem sets, adding examples to existing oer or asking students to help reframe and re-present content in new and innovative ways to create a new oer. pressbooks manifold fulcrum google docs open textbook annotation project public chats with experts engage students in scholarly conversations with professionals and scholars in the field through public chats. using social media tools, students can engage in dialogue with others sharing their thoughts, asking questions, and receiving feedback on their work by tagging individuals or organisations on their posts. create a hashtag for your class to organise all of the students’ posts. twitter facebook tiktok class hashtags #edci336 #stratcomm2321 co-created syllabi build course policies, outcomes, assignments, rubrics and class schedules collaboratively with students. not only does this give them a voice but it also provides them more ownership over the content and work. the syllabus becomes a collaborative document that is shaped by and with students. google docs collaborative syllabus design co-created attendance policy curate course content educators can also empower students within their classes by allowing students to curate the content they engage with during the class. provide students with limited choices of different texts or allow them to curate texts for an upcoming unit/topic. pressbooks manifold against product based learning http://dx.doi.org/10.25304/rlt.v31.2829 https://pressbooks.com/ https://manifoldapp.org/ https://www.fulcrum.org/ https://www.google.com/docs/about/ https://rex.libraries.wsu.edu/esploro/outputs/other/open-textbook-annotation-project/99900501601701842 https://rex.libraries.wsu.edu/esploro/outputs/other/open-textbook-annotation-project/99900501601701842 https://rex.libraries.wsu.edu/esploro/outputs/other/open-textbook-annotation-project/99900501601701842 https://twitter.com/search?q=%23edci336&src=hashtag_click&f=live https://twitter.com/hashtag/stratcomm2321?src=hashtag_click https://www.google.com/docs/about/ http://openpedagogy.org/course-level/collaborative-syllabus-design-students-at-the-center/ http://openpedagogy.org/course-level/collaborative-syllabus-design-students-at-the-center/ https://twitter.com/profjasmine/status/1516485503994019843 https://twitter.com/profjasmine/status/1516485503994019843 https://pressbooks.com/ https://manifoldapp.org/ http://openpedagogy.org/course-level/against-product-based-learning-open-texts-are-never-finished/ http://openpedagogy.org/course-level/against-product-based-learning-open-texts-are-never-finished/ research in learning technology citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 11 (page number not for citation purpose) by bali, cronin, and jhangiani (2020) as social justice concerns, there are many reasons students may not want to share their work openly and publicly. likewise, hilton et al. (2019) urge instructors to consider the ethics of requiring or strongly encouraging students to share their work openly. thus, educators should intentionally apply their oep content knowledge of cronin’s levels of openness when designing open learning activities, assignments and projects. inviting students to share their work in open repositories and guiding students selection of cc licenses, and choices of who to share work as, while providing assignment alternatives, honours student agency. technological knowledge connections because oep almost always requires the use of technology to facilitate sharing and creation of resources, much technical knowledge is needed to engage in these practices. to make strategic decisions for their classrooms, educators need knowledge about the digital resources most appropriate for sharing and producing oer, such as those listed in table 1. in addition to raising awareness of various tools, platforms and applications associated with open education, educators should explore the resources to become familiar with them and simultaneously develop their own digital literacy skills in using them. although teaching with these resources does not typically require in-depth knowledge and skill, familiarity provides educators with a starting point for imagining pedagogical use with their specific student populations and content areas. for example, when involving students in the co-creation of open content, educators will be prepared to guide students to tools that can achieve the goals of the resource being co-created. then, as the project progresses, educators and students learn from each other and broaden their knowledge and digital literacy skills together. in any use of technology, the accessibility of a resource or making it usable by everyone, is important. it is particularly important to incorporate accessibility in open practices, where the goal is inclusivity and equity. as the bc campus open education accessibility toolkit asks, ‘if the materials are not accessible for each and every student, do they fulfil the mandate to deliver fully open textbooks?’ (coolidge, doner, and robertson 2015, para. 3). designing for accessibility requires an understanding of universal design, which is both a process and an outcome to create something that is usable by all, without needing specialised adaptation (see https://udlguidelines.cast.org/). open textbooks, such as the accessibility toolkit, include an accessibility statement detailing the accessibility features and the standards that it meets as well as other file formats available. they also include known accessibility issues and a form and email address for users to report accessibility issues so they can be addressed promptly. technological content knowledge while educators need to develop an awareness of general open repositories and tools, there are many discipline-specific resources available as well. the national science digital library provides links to high-quality open online educational resources in stem disciplines for educators to use with students at all levels. similarly, oer commons is an open repository designed for educators to search for, adapt and create open teaching and learning materials, with a particular emphasis on primary and secondary school subjects and lessons. educators in the humanities and arts, http://dx.doi.org/10.25304/rlt.v31.2829 https://udlguidelines.cast.org/ https://opentextbc.ca/accessibilitytoolkit/ https://nsdl.oercommons.org/ https://nsdl.oercommons.org/ https://nsdl.oercommons.org/ https://nsdl.oercommons.org/ j. van allen and s. katz 12 citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 (page number not for citation purpose) among others, may find smithsonian open access useful as it provides digital access to millions of open-access items from the smithsonian’s museum collections, as well as access to research data sets, all of which may be reused and remixed in the smithsonian learning lab. learning about repositories such as these provides educators with the knowledge needed to adopt oer tailored to their subjects and the resources needed to support oep with students. technological pedagogical knowledge just as educators must consider their own digital literacy skills, they should not automatically assume students possess the digital literacy skills needed to effectively engage in oep. when using specific open repositories and tools, educators should intentionally scaffold learning experiences to prepare students to use the digital tools. to illustrate, if an instructor wants students to engage in public social annotation throughout a course using hypothes.is, students will first need to know to use the google chrome browser, install the hypothes.is extension, and sign up for an account. educators will likely need to model how to create annotations and replies to others, including steps for creating annotations and expectations for producing culturally sensitive and respectful comments to others. additionally, sawyer et al. (2020) caution that educators must help students develop the dispositions of critical curators, consumers and creators. to do so, educators must intentionally plan oep activities and assignments that support these dispositions as related to their field. with a focus on equity, educators must also be mindful of issues with student access to technology and other challenges related to digital equity (resta et al. 2018). given the wide variety of tools available to access online content, educators need to be cognisant of the devices their students are using and the extent to which they will be able to navigate the materials. surveying students at the beginning of a class is one way to ensure equitable access to required class tools. if an educator finds that some students may not have access to digital oep tools, they should consider how to plan for participation and engagement offline as well as online. many oer can be downloaded and saved in different file formats but may not always translate well when printed. likewise, students without consistent access to the internet may not be able to engage in a public chat with an expert in the field. thus, these are technological pedagogical concerns and digital equity issues that must be intentionally considered and planned for when integrating oep. the open in tpack mishra and koehler (2006) originally argued that quality teaching ‘requires a thoughtful interweaving . . . [and] nuanced understanding of the complex relationships between technology, content, and pedagogy. . . to develop appropriate, context-specific strategies’ (p. 1029). it is impossible to view oep in isolation from content, pedagogy and technology since they exist in dynamic transactional relationships as the examples in the previous sections show. we argue that equitable teaching with oep requires an understanding of the complex ways open is embedded within content, pedagogy and technology and the intentional design of learning environments and instruction as shown in our open application of the tpack framework (see figure 3). for instance, educators integrating a renewable assignment for the first time are forced http://dx.doi.org/10.25304/rlt.v31.2829 https://www.si.edu/openaccess http://hypothes.is http://hypothes.is research in learning technology citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 13 (page number not for citation purpose) to consider what type of student work will demonstrate the learning objectives of the course and result in a good contribution to the public knowledge commons (katz and van allen 2020). among these factors, they must decide on specific tools they will ask students to use (ones accessible to students), and how they will scaffold support (such as including multiple rounds of peer review) so that students feel confident selecting a cc license and sharing their work publicly. the newness of these considerations can cause educators to interrogate their understandings of content, pedagogy and technology and rethink their teaching practices in new ways. since educators often find the technical aspects of open education to be nuanced and complex, it is essential they are aware of who to turn to for support. librarians and instructional designers often have expertise in open education and can alleviate some of the barriers associated with oep (van allen and katz 2020), particularly through collaborative relationships. for example, librarians can support educators in finding and selecting oer or understanding the permissions of an open license through professional development workshops or other training experiences (clifton and hoffman [eds] 2020). additionally, instructional designers can support educators in designing learning experiences that capitalise on oep and help problem-solve figure 3. the knowledge, skills and dispositions of open in tpack. http://dx.doi.org/10.25304/rlt.v31.2829 j. van allen and s. katz 14 citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 (page number not for citation purpose) challenges that arise during implementation (ren 2019). regardless of the method used, it is clear that training and support are essential to developing educators’ knowledge, skills, and dispositions of open education, cc licensing, open pedagogy, the digital tools that facilitate oep, and intentionality and care. conclusion our conceptualisations of the intersections between oep and the tpack framework are intended to provide educators adopting open practices, and those who support them, with an awareness of the kinds of knowledge, skills and dispositions needed. some characterise open as an ethos, a commitment to caring, democratic, participatory teaching and learning practices; yet oep requires educators to make decisions that are personal, contextual and nuanced (cronin 2017). some educators will prioritise oep to empower learners for more socially just classrooms, while others may adopt oer to reduce the costs of their textbooks and increase student access to learning. no matter their reasons for entering into oep, educators need competencies related to technological knowledge, pedagogical knowledge and content knowledge to provide successful learning experiences for students. however, making teaching and learning more widely available does not necessarily result in a more socially just education. to avoid replicating hegemonic materials and opportunities, careful application of oep when considered through the tpack framework can guide educators in integrating open practices in their content areas and classroom contexts for more inclusive and equitable education for all. acknowledgements we thank members of the faculty fellowship publication programme, including our mentors and colleagues, for the support and comments that greatly improved the manuscript. references allen, n. (2018) ‘$1 billion in savings through open education resources’, scholarly publishing and academic resources coalition (sparc), [online] available at: https://sparcopen.org/ news/2018/estimating-oer-student-savings/ bakermans, m., et al., (2022) ‘who writes and who responds? gender and race-based differences in open annotations’. journal for multicultural education, vol. 16, no. 5, pp. 508–521. doi: 10.1108/jme-12-2021-0232 bali, m., cronin, c. & jhangiani, r. (2020) ‘framing open educational practices from a social justice perspective’, journal of interactive media in education, vol. 1, no. 10, p. 10. doi: 10.5334/jime.565 baran, e., et al., (2020) ‘investigating the impact of teacher education strategies on preservice teachers’ tpack’, british journal of educational technology, vol. 50, no. 1, pp. 357–370. doi: 10.1111/bjet.12565 clifton, a. & hoffman, k. d. (eds) (2020) open pedagogy approaches: faculty, library and student collaborations, milne publishing, geneseo, ny. collins, a. & halverson, r. (2018) rethinking education in the age of technology: the digital revolution and schooling in america, teachers college press, new york, ny. coolidge, a., doner, s. & robertson, t. (2015) bccampus open education accessibility toolkit. bccampus, victoria, [online] available at: https://opentextbc.ca/accessibilitytoolkit/ http://dx.doi.org/10.25304/rlt.v31.2829 https://sparcopen.org/news/2018/estimating-oer-student-savings/ https://sparcopen.org/news/2018/estimating-oer-student-savings/ https://dx.doi.org/10.1108/jme-12-2021-0232 https://dx.doi.org/10.5334/jime.565 https://dx.doi.org/10.1111/bjet.12565 https://opentextbc.ca/accessibilitytoolkit/ research in learning technology citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 15 (page number not for citation purpose) cronin, c. & maclaren, i. (2018) ‘conceptualising oep: a review of theoretical and empirical literature in open educational practices’, open praxis, vol. 10, no. 2, pp. 127–143. doi: 10.5944/openpraxis.10.2.825 cronin, c. (2017) ‘openness and praxis: exploring the use of open educational practices in higher education’, the international review of research in open and distributed learning, vol. 18, no. 5, pp. 15–34. doi: 10.19173/irrodl.v18i5.3096 cronin, c. (2020) ‘open education: walking a critical path’, in open(ing) education: theory and practice, eds d. conrad & p. prinsloo, brill, leiden, pp. 2–13. derosa, r. & jhangiani, r. (2017) ‘open pedagogy’ in a guide to making open textbooks with students, eds e. mays, rebus, montreal, [online] available at: https://press.rebus. community/makingopentextbookswithstudents/chapter/open-pedagogy/ dewaard, h. & roberts, v. (2021) ‘revisioning the potential of freire’s principles of assessment: influences on the art of assessment in open and online learning through blogging’, distance education, vol. 42, no. 2, pp. 310–326. doi: 10.1080/01587919.2021.1910494 fazzino, l. & turley, j. (2019) ‘remixing an open educational resource: a case study of the uncommon “r”’, urban library journal, vol. 25, no. 1, p. 3. gilliard, c. & culik, h. (2016) ‘digital redlining, access, and privacy’, common sense education, [online] available at: https://www.commonsense.org/education/articles/ digital-redlining-access-and-privacy grotewold, k. s., kohler, k. l. & krimbill, e. m. (2022) ‘pay it forward: realizing the promise of oer for the next generation of learners’ in using open educational resources to promote social justice, eds c. j. ivory & a. pashia, association of college and research libraries, chicago, il, [online] available at: https://www.ala.org/acrl/sites/ala.org.acrl/files/ content/publications/booksanddigitalresources/digital/9780838936771_oa.pdf hegarty, b. (2015) ‘attributes of open pedagogy: a model for using open educational resources’, educational technology, vol. 55, no. 4, pp. 3–13, [online] available at: https://www.jstor.org/ stable/44430383 hilton iii, j., et al. (2010) ‘the four “r”s of openness and alms analysis: frameworks for open educational resources’, open learning, vol. 25, no. 1, pp. 37–44. doi: 10.1080/02680510903482132 hilton iii, j., et al. (2019) ‘student perceptions of open pedagogy: an exploratory study’, open praxis, vol. 11, no. 3, pp. 275–288. doi: 10.5944/openpraxis.11.3.973 hofer, m. & harris, j. (2016) ‘open educational resources (oer’s) for tpack development’, in proceedings of society for information technology & teacher education international conference 2016, eds m. searson & m. ochoa, aace, waynesville, nc, pp. 4865–4870. jeffrey, a. j., et al. (2021) ‘a flexible, open, and interactive digital platform to support online and blended experiential learning environments: thinglink and thin sections’, geoscience communication, vol. 4, no. 1, pp. 95–110. doi: 10.5194/gc-4-95-2021 jenkins, j. j., et al. (2020) ‘textbook broke: textbook affordability as a social justice issue’, journal of interactive media in education, vol. 1, no. 3, p. 1. doi: 10.5334/jime.549 jhangiani, r. s. (2019) ‘delivering on the promise of open educational resources: pitfalls and strategies’ in moocs and open education across emerging economies: challenges, successes, and opportunities, eds k. zhang, et al., taylor & francis, new york, ny, pp. 56–62. jung, i., sasaki, t. & latchem, c. (2016) ‘a framework for assessing fitness for purpose in open educational resources’, international journal of educational technology in higher education, vol. 13, no. 3, pp. 1–11. doi: 10.1186/s41239-016-0002-5 katz, s. & van allen, j. (2020) ‘evolving into the open: a framework for collaborative design of renewable assignments’ in open pedagogy approaches: faculty, library, and student collaborations, eds k. hoffman & a. clifton, milne publishing, geneseo, ny, [online] available at: https://milnepublishing.geneseo.edu/openpedagogyapproaches/chapter/ evolving-into-the-open-a-framework-for-collaborative-design-of-renewable-assignments/ koehler, m. j. & mishra, p. (2009) ‘what is technological pedagogical content knowledge?’ contemporary issues in technology and teacher education, vol. 9, no. 1, pp. 60–70. http://dx.doi.org/10.25304/rlt.v31.2829 https://dx.doi.org/10.5944/openpraxis.10.2.825 https://dx.doi.org/10.19173/irrodl.v18i5.3096 https://press.rebus.community/makingopentextbookswithstudents/chapter/open-pedagogy/ https://press.rebus.community/makingopentextbookswithstudents/chapter/open-pedagogy/ https://dx.doi.org/10.1080/01587919.2021.1910494 https://www.commonsense.org/education/articles/digital-redlining-access-and-privacy https://www.commonsense.org/education/articles/digital-redlining-access-and-privacy https://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/booksanddigitalresources/digital/9780838936771_oa.pdf https://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/booksanddigitalresources/digital/9780838936771_oa.pdf https://www.jstor.org/stable/44430383 https://www.jstor.org/stable/44430383 https://dx.doi.org/10.1080/02680510903482132 https://dx.doi.org/10.5944/openpraxis.11.3.973 https://dx.doi.org/10.5194/gc-4-95-2021 https://dx.doi.org/10.5334/jime.549 https://dx.doi.org/10.1186/s41239-016-0002-5 https://milnepublishing.geneseo.edu/openpedagogyapproaches/chapter/evolving-into-the-open-a-framework-for-collaborative-design-of-renewable-assignments/ https://milnepublishing.geneseo.edu/openpedagogyapproaches/chapter/evolving-into-the-open-a-framework-for-collaborative-design-of-renewable-assignments/ j. van allen and s. katz 16 citation: research in learning technology 2023, 31: 2829 http://dx.doi.org/10.25304/rlt.v31.2829 (page number not for citation purpose) koh, j. h. l., chai, c. s. & lee, m. h. (2015) ‘technological pedagogical content knowledge (tpack) for pedagogical improvement: editorial for special issue on tpack’, the asiapacific education researcher, vol. 24, no. 3, pp. 459–462. doi: 10.1007/s40299-015-0241-6 lambert, s. r. (2018) ‘changing our (dis)course: a distinctive social justice aligned definition of open education’, journal of learning for development, vol. 5, no. 3, pp. 225–244. doi: 10.56059/jl4d.v5i3.290 mishra, p. & koehler, m. j. (2006), ‘technological, pedagogical content knowledge: a framework for teacher knowledge’, teachers college record, vol. 8, no. 6, pp. 1017–1054. doi: 10.1111/j.1467-9620.2006.00684.x niess, m. l. (2019) blended online learning and instructional design for tpack: emerging research and opportunities, igi global, hershey. oer in mathematics professional development project (n.d.), connecting curriculum, assessment and instruction, [online] available at: http://courses.maine.edc.org/1/oer/math_ curricassessinstr.html pareto, l. & willermark, s. (2019) ‘tpack in situ: a design-based approach supporting professional development in practice’, journal of educational computing research, vol. 57, no. 5, pp. 1186–1226. doi: 10.1177/0735633118783180 paskevicius, m. (2017) ‘conceptualizing open educational practices through the lens of constructive alignment, open praxis, vol. 9, no. 2, pp. 125–140. doi: 10.5944/openpraxis.9.2.519 ren, x. (2019) ‘the undefined figure: instructional designers in the open educational resource (oer) movement in higher education’, education and information technologies, vol. 24, no. 2, pp. 3483–3500. doi: 10.1007/s10639-019-09940-0 resta, p. et al. (2018) ‘issues and challenges related to digital equity: an introduction’, in second handbook of information technology in primary and secondary education, eds j. voogt et al., springer international publishing, cham, switzerland, pp. 987–1004. ross, h. m., hendricks, c. & mowat, v. (2018) ‘open textbooks in an introductory sociology course in canada: student views and completion rates’, open praxis, vol. 10, no. 4, pp. 393–403. doi: 10.5944/openpraxis.10.4.892 sawyer, a. g. et al. (2020) ‘developing teachers as critical curators: investigating elementary preservice teachers’ inspirations for lesson planning’, journal of teacher education, vol. 71, no. 5, pp. 518–536. doi: 10.1177/00224871198798 shulman, l. s. (1986) ‘those who understand: knowledge growth in teaching’, educational researcher, vol. 15, no. 2, pp. 4–14. doi: 10.3102/0013189x015002004 stommel, j. (2014) critical digital pedagogy: a definition. hybrid pedagogy, [online] available at: https://hybridpedagogy.org/critical-digital-pedagogy-definition/ united nations. (2015) transforming our world: the 2030 agenda for sustainable development, [online] available at: https://sdgs.un.org/2030agenda van allen, j. & katz, s. (2020) ‘teaching with oer during pandemics and beyond’, journal for multicultural education, vol. 14, no. 3/4, pp. 209–218. doi: 10.1108/jme-04-2020-0027 wiley, d. & green, c. (2012). ‘why openness in education?’, in game changers: education and information technologies, ed d. g. oblinger, educause, [online] available at: https:// www.educause.edu/ir/library/pdf/pub7203.pdf william and flora hewlett foundation. (2020) open education strategy, [online] available at: https://hewlett.org/wp-content/uploads/2016/08/hewlett-opened-strategy-2020.pdf http://dx.doi.org/10.25304/rlt.v31.2829 https://dx.doi.org/10.1007/s40299-015-0241-6 https://dx.doi.org/10.56059/jl4d.v5i3.290 https://dx.doi.org/10.1111/j.1467-9620.2006.00684.x http://courses.maine.edc.org/1/oer/math_curricassessinstr.html http://courses.maine.edc.org/1/oer/math_curricassessinstr.html https://dx.doi.org/10.1177/0735633118783180 https://dx.doi.org/10.5944/openpraxis.9.2.519 https://dx.doi.org/10.1007/s10639-019-09940-0 https://dx.doi.org/10.5944/openpraxis.10.4.892 https://dx.doi.org/10.1177/00224871198798 https://dx.doi.org/10.3102/0013189x015002004 https://hybridpedagogy.org/critical-digital-pedagogy-definition/ https://sdgs.un.org/2030agenda https://dx.doi.org/10.1108/jme-04-2020-0027 https://www.educause.edu/ir/library/pdf/pub7203.pdf https://www.educause.edu/ir/library/pdf/pub7203.pdf https://hewlett.org/wp-content/uploads/2016/08/hewlett-opened-strategy-2020.pdf (page number not for citation purpose) 1 *corresponding author. email: d.turnbull@cqu.edu.au research in learning technology 2023. © 2023 d. turnbull et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 research in learning technology vol. 31, 2023 original research article learning management systems and social media: a case for their integration in higher education institutions darren turnbull* , ritesh chugh and jo luck school of engineering and technology, central queensland university, rockhampton, queensland, australia (received: 3 may 2022; revised: 24 march 2023; accepted: 11 april 2023; published: 30 may 2023) higher education institutions across the globe rely on learning management systems (lmss) to deliver course content, assess student learning, and maintain effective communication. however, contemporary learners may prefer to use popular social media platforms to share knowledge and collaborate with peers. higher education institutions can benefit by fusing the best features of social media and lmss into course delivery systems, particularly in online settings. this  study investigates the technological and pedagogical integration of social media and lmss in higher education institutions that incorporate these technologies into their course delivery infrastructure. from the 36 peer-reviewed papers examined, the identified benefits of successful social media-lmss integration were classified into six categories: access to learning materials, student recruitment, communication and peer support, improved results, a single access point to both online environments, and speed and reliability. three categories of disadvantages were also established: need for ongoing support, social media distractions, and technical and security issues. we propose that a close inter-relationship between social media platforms and lmss enhances course outcomes within a social constructivist framework and satisfies learner needs for social interaction. this study’s findings will benefit educational institutions seeking to enhance engagement with online learner communities. keywords: learning management system; social media; software integration; facebook; twitter; moodle; virtual learning environment introduction learning management systems (lmss) (or otherwise known as virtual learning environments, vles) are online software platforms that assist in the delivery of course materials and facilitate student learning (turnbull, chugh, and luck 2019). lmss can be used to enhance all three delivery modes: face-to-face, online, and blended learning. lmss host a variety of tools that can facilitate user communications, such as discussion boards, real-timse chat modules, and email clients. however, within lms environments, students often do not use the available tools for communication mailto:d.turnbull@cqu.edu.au http://dx.doi.org/10.25304/rlt.v31.2814 https://orcid.org/0000-0003-0509-8564 https://orcid.org/0000-0003-0061-7206 https://orcid.org/0000-0002-5478-7289 d. turnbull et al. 2 citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 (page number not for citation purpose) purposes, preferring instead to use dedicated social media (sm) platforms such as facebook (gosper et al. 2014; pilli 2014). a possible reason could be that institutionally-controlled lmss define institutional boundaries and roles (wise, skues, and williams 2011), and students could consider it too restrictive for social interactions. another reason could be that academics design lms course content primarily for desktop deployment, while students increasingly prefer the convenience of mobile devices to engage with their learning (baldwin and ching 2020). the authors contend that a sharper focus on mobile device use when designing an lms-based curriculum may enhance student satisfaction with online course delivery. in contrast to lmss that primarily focus on educational delivery, sm are webbased online communication technology platforms that facilitate the interchange of ideas and the development of socially engaged online communities (chugh and ruhi 2019). examples of popular sm platforms available in the english-speaking world include facebook, whatsapp, twitter, instagram, snapchat, youtube, pinterest, reddit, and linkedin (kellogg 2020; kircaburun et al. 2018). countries that restrict open internet access often sanction alternate home-grown sm platforms. for example, chinese citizens within mainland china are prevented from accessing foreign sm platforms but have access to local alternatives such as wechat, weibo, and qq (li, he, and zhang 2019). there is often little crossover between social interactions in established sm environments and communications contained within institutionally-controlled lmss (hrastinski and aghaee 2011). however, some evidence supports observations that many students enrolled in higher education institutions (heis) such as universities and colleges use sm applications to communicate with peers for course-related purposes (mozhaeva, feshchenko, and kulikov 2014; rahman, ramakrishnan, and ngamassi 2019). nevertheless, educators are often reluctant to embrace this communication medium as a vehicle to enhance student learning (chugh, grose, and macht 2021). possible reasons for this include faculty concerns about the privacy of personal information, the perceived inability to authenticate the originality of student work created via sm (martínez-alemán 2014), and the lack of control over the sm interfaces controlling the displayed content and user interaction (wise, skues, and williams 2011). a thorough investigation of the characteristics and features of sm platforms selected for possible inclusion into lms environments, including vigorous testing with representative samples of students, may help diminish resistance to the adoption of these technologies within existing course delivery structures. in this study, we explore this issue in the context of lms-sm integration. social media use in education is often guided by social constructivist principles. it  is important that educational technology research is founded on explicit theories to enable a deeper understanding of explored phenomena (hew et al. 2019). building from the relativist premise that knowledge is contextually dependent, constructivism focuses on how people develop meaning and understanding of a phenomenon of interest from interactions with its situated environment (schrader 2015). a variant of constructivism, social constructivism, developed by vygotsky in 1934, has as a central tenet a belief that knowledge is co-constructed in a social environment where people use language to develop meaning (churcher 2014). learning through sm supports social constructivism because of the technology’s focus on enabling collaboration (sarwar et al. 2018). however, this could be at odds with many behaviourist approaches towards lms use that emphasise the use of mechanical drills and trackable objects to deliver compliance training (pinner 2011). http://dx.doi.org/10.25304/rlt.v31.2814 research in learning technology citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 3 (page number not for citation purpose) it is also essential to look at the reasons why users engage with technologies. davis (1993)’s seminal work on the technology acceptance model (tam) highlights five influences that shape an individual’s intent to use information technology (it): system design features, perceived usefulness, perceived ease of use, attitude towards using technology, and actual system use. another important consideration is the influence of learner cultural characteristics on it adoption decisions. hofstede (2011) defines culture as consisting of six dimensions: power-distance, uncertainty avoidance, individualism versus collectivism, masculinity versus femininity, long-term versus shortterm orientation, and indulgence versus restraint. by unearthing the triggers that influence learner choices of technology, heis and their educators will be better positioned to argue for the use of a framework/lens to determine technology selection for maximising student engagement. higher education institutions and educators can benefit from leveraging sm, which students regularly use as part of their social communications. there are many advantages to communicating via sm applications such as facebook. for example, students may have more confidence in communicating via this medium when they know that the account is within their sphere of control (lampe et al. 2011). also, communication within sm tends to be more synchronous than in many traditional lmss such as moodle (awidi, paynter, and vujosevic 2019). however, integrating sm within an lms platform has significant challenges. many specialised functions within lmss cannot be replicated within an sm environment like facebook. the capacity of an lms to integrate with external systems is also a key consideration in selecting an appropriate platform to support student-centred learning in heis (kasim and khalid 2016). clear guidelines need to be made about sm’s purpose within a course delivery framework (chugh and ruhi 2017). technical and legal issues must be resolved to ensure that course functionality is not impeded by a third-party application outside an educator’s sphere of control (lockyer and patterson 2008). in a global communication environment increasingly influenced by rapid, synchronous information exchange, there is a need to identify, evaluate, and decide how best to accommodate contemporary learner preferences for communication via popular sm sites within traditional lms-dominated course delivery structures. this paper examines prior research on the integration of sm into the framework of institutionally-controlled lmss. integration, in the context of this study, refers to the seamless amalgamation of popular sm technologies (e.g. facebook, twitter) within heis’ lms infrastructure. however, integration could also be seamless both technologically and pedagogically. for example, seamless technological integration could involve a single-sign-on process, that is, student signs in to an lms, which authenticates access to an sm platform. in this case, students would be present on the sm platform with their institutional account (and associated email address). pedagogical integration could mean that in a classroom or session, teachers may ask students to do some tasks within the lms that involve using an sm platform such as facebook or linkedin. the main aim is to establish whether there is any evidence that  sm-lms integration leads to improved learning conditions in higher education. the next section outlines the research method, followed by the study’s main results. the findings are then discussed in the context of the main advantages and disadvantages of embedding sm applications within lms systems. finally, managerial implications are presented, followed by a conclusion outlining the study’s limitations and recommendations for further research. http://dx.doi.org/10.25304/rlt.v31.2814 d. turnbull et al. 4 citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 (page number not for citation purpose) method this study examined previous empirical research on course delivery using lmss and their relationship with sm. two research questions (rq) drove the study: (rq1) what are the advantages of sm-lms integration?, and (rq2) what are the disadvantages of sm-lms integration? this paper adopts a narrative approach to address the research aims and applies some of the rigour of systematic review methodologies to the selection and inclusion of the published articles reviewed in this paper. turnbull et  al. (2023) posit that this hybrid systematic-narrative methodology can provide a robust framework for exploring literature without overburdening researchers with systematic review procedures. a total of four separate providers were used in the search strategy: ebscohost (38 databases), gale (26 databases), informit (10 databases), and scopus. these databases were chosen because of their reputation as reliable repositories of peer-reviewed research in the field of education and technology. however, it is acknowledged that including additional databases could have augmented the number of papers identified in this review. the search query was defined as ‘the use of embedded sm in learning management systems’. a search was conducted in each database using the primary terms ‘social media’ and ‘learning management system’ along with the following commercial sm applications: wechat, twitter, youtube, whatsapp, facebook, instagram, snapchat and pinterest. the popular sm applications selected for inclusion in the search string were deemed relevant to the english language publications targeted in this study and are often used as synonyms for all sm platforms. while lms and vles are often used interchangeably (williams 2022), we have only searched for lms in this study because it is a commonly used synonym for vle (njoki 2021). the boolean search string applied to the search was: (social media or wechat or twitter or youtube or whatsapp or facebook or instagram or snapchat or pinterest) and (learning management system or lms or ‘online learning’ or elearning or e-learning). searches were conducted within the abstract field of each database for peer reviewed journal articles that were empirical studies. the search targeted english language articles published between november 2015 and january 2021 that contained retrievable pdfs. this date range restriction was applied to ensure each identified study had a contemporary focus. the results of each database search were added to a single repository within endnote. any duplicates were subsequently removed. there were a total of 61 unique articles in the initial list. three articles were removed because they were duplicate studies under different first author names. these articles were further examined to identify findings relevant to sm-lms integration. iterative thematic analysis using nvivo software was then used to identify and code important observations and analyses on sm-lms integration contained in the articles. a total of 36 papers were included in this coding scheme. these articles were also classified against two attributes: region of origin, and year of publication. the results are presented in the next section. results the 36 studies included in this review are marked with an asterisk (*) symbol in the reference list. each study was assigned a geographic location (africa, middle east, australasia, europe, sub-continent, north america, europe, south east asia, north http://dx.doi.org/10.25304/rlt.v31.2814 research in learning technology citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 5 (page number not for citation purpose) asia) indicating its origin. figure 1 provides a graphic representation of the distribution of included studies along with their year of publication. as shown in figure 1, most studies originated in north america, europe and australasia. however, this does not necessarily reflect the level of usage of sm and lms technologies per country, as studies conducted in languages other than english were not included in this review. articles were examined in further detail, and codes were created to identify comments attributed to sm and lms integration. all 36 studies had comments relevant to sm-lms integration. the relevant text was coded in an iterative process against emerging themes supporting or opposing sm-lms integration. this led to the establishment of six integration advantage categories and three integration disadvantage categories, as illustrated in figure 2. some studies had multiple comments highlighting both the advantages and disadvantages, which were coded in more than one category. discussion in many of the selected studies, sm and institutionally controlled lmss were regarded as distinct environments with separate and unrelated primary functions. analysis of the reviewed literature yielded more evidence in support of sm-lms integration than against it: there were a total of 37 instances of integration advantages compared to 16 instances of integration disadvantages. however, the data analysis did not include a weighted evaluation of the relative strength of each study’s advantage or disadvantage, so no definitive conclusions can be drawn on the overall ranking of each category. the six categories of advantages and three categories of disadvantages with respect to sm-lms integration are displayed in figure 2. the rest of this discussion is devoted to exploring the characteristics of each category. figure 1. sm-lms studies by region of origin. http://dx.doi.org/10.25304/rlt.v31.2814 d. turnbull et al. 6 citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 (page number not for citation purpose) sm-lms integration – advantages combining popular features of sm with structured lms functions facilitates the harvesting of synergies between these two information systems. this can result in an enhanced student learning environment and a more efficient course delivery system for educators and institutions. for example, students adept at using facebook to communicate with their peers on a social basis, may experience enhanced course engagement via facebook groups that focus on course-related issues (thai, sheeran, and cummings 2019), while faculty and institutions benefit from an additional channel to share courserelated resources (li, ganeshan, and xu 2012). the benefits of figure 2. breakdown of categories describing the advantages and disadvantages of integrating sm into lms platforms. http://dx.doi.org/10.25304/rlt.v31.2814 research in learning technology citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 7 (page number not for citation purpose) sm-lms integration are grouped into six categories, as displayed in figure 2: access to materials, student recruitment, communication and peer support, improved academic results, a single access point to both online environments, and speed and reliability. the following six sub-sections represent a synthesis of the research findings of the included articles that addressed lms-sm integration benefits in these categories. access to learning materials sm-lms integration can improve access to learning resources through information sharing (hamadi et al. 2020). access to learning materials contained within an lms may be subject to learners meeting certain access criteria. for example, learning materials may be made available to students only on a specific date, or restricted to a specific course or year of enrolment (judd and elliott 2017). facebook, on the other hand, permits the peer-to-peer exchange of learning resources outside the lms environment (ghounane 2020; kalelioğlu 2017; kent 2016; todorovic et al. 2021). youtube is also mentioned as a useful tool to provide supplemental materials for students in two papers (mansour 2021; turkyilmaz, hariri, and jahangiri 2019) and can be used for the informal exchange of ideas and resources. providing pathways for students to search and retrieve course-related materials via sm promotes ownership of the learning process and fosters inter-student cooperation (frankel 2020). student recruitment educational institutions need to attract students to participate in their courses. selecting social learning systems that deploy features that enhance enjoyment in addition to being functionally effective should be a primary concern of institutional decision-makers (khechine, raymond, and augier 2020). the integration of popular sm functions within an lms framework could attract potential new students who may be familiar with a particular sm platform (parusheva, aleksandrova, and hadzhikolev 2018). as a means of attracting new students, setting up a course with a familiar platform like facebook and then transitioning students to learning within an institutional lms is cited as an excellent way to introduce new learners to an institution’s learning environment (ingalls 2017). al-azawei (2019) points out that it is essential to provide potential students with details about how sm is used within a course to help ensure student retention after recruitment. communication and peer support communication and peer support was the most prominent category of advantages. thirteen studies indicated that communication, particularly peer-to-peer communication, is enhanced by integrating sm within lms frameworks. facebook featured prominently as an sm platform that could assist learners in communicating within a course (akcaoglu and lee 2018; aleksandrova and parusheva 2019; camus et al. 2016; charteris et al. 2018; dalsgaard 2016; kent 2016; moghavvemi et al. 2017). the main advantages of facebook when integrated within an lms that was cited in these studies include improved peer-to-peer communication between learners, improved access to other learners within a course, a more acceptable way to receive course-related announcements, student ownership of learning outcomes, and the http://dx.doi.org/10.25304/rlt.v31.2814 d. turnbull et al. 8 citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 (page number not for citation purpose) potential for facebook to function as a stand-alone entity if needed. two studies mentioned whatsapp as a useful sm platform to adopt, as it allows students to share better informal experiences (mpungose 2019; tagoe and cole 2020). other communications-related advantages cited in non-platform specific studies include: enhanced synchronous communication within sm (kalelioğlu 2017; trivedi, patra, and singh 2021), improved communications from shy or reserved learners (khan et al. 2017), increased interactivity when compared to stand-alone lms environments (oktavia, supangkat, and prabowo 2018), improved communication between teachers and students (hamadi et al. 2020; raspopovic et al. 2017), and enhanced frequency of communication between learners due to the familiarity of sm to the learner community (salmon et al. 2015). improved results the integration of sm applications within lms platforms was mentioned in two papers as an enabling factor for improved student results. the lack of focus of articles on improved results could be because it is difficult to measure and relate the performance of sm-lms to improved results. cao and saini (2021) posit that qq (a chinese sm platform) improves student performance through the provision of lecturing, conferences, discussions, group activities, and messaging functions, while chuang and liao (2021) point to the excellent connectivity provided by sm platforms as a factor contributing to enhanced learning outcomes. another study claimed that student use of whatsapp within a course led to improved academic results when used in conjunction with an lms (khan et al. 2017). this may be difficult for future studies to verify in practice because the privacy constraints on whatsapp accounts prevent the aggregation of user data by heis that could be used to establish a causal link between this platform’s use and improvements in academic performance. however, the study emphasised that improvements in academic performance were realised within a uniquely blended lms-whatsapp delivery that combined the administrative functions of the lms with whatsapp’s capacity to facilitate learner collaboration. single access point to both online environments the availability of multiple platforms for learners can sometimes create confusion and be difficult to navigate. two studies described the development and integration of course delivery environments that incorporate the features of sm and structured lmss within a single environment accessible through a single gateway. in the first study (sabin and olive 2018), a free web-based application called ‘slack’ was adapted to help deliver a second-year online course in environmental politics. slack became the defacto lms for course delivery and was the single access point for learners to engage with course content and each other. slack was transformed, in effect, into an lms-sm platform through the strategies adopted by educators for its deployment. microsoft teams, which is an sm platform with lms functionality, is now being used extensively across heis. both slack and teams are platforms with sm functionality (e.g. emoticons, chat, visual content, gifs, likes, peer-to-peer connections, voting, notifications, tagging, etc.). the second study (wautelet et al. 2016) took a different approach. they proposed the creation of a multi-agent system (mas) capable of http://dx.doi.org/10.25304/rlt.v31.2814 research in learning technology citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 9 (page number not for citation purpose) integrating disparate virtual environments such as lms and sm. as a result, learners would experience a unified learning environment using familiar features of sm platforms such as facebook and twitter. the main advantages of doing this include ease of access by only logging on to one integrated platform and improved access to course-related materials. speed and reliability in any online course environment, the time taken to authenticate access, exchange learning materials, and communicate with other users is critical to maintaining learner interest. akcaoglu and lee (2018)’s research into the use of facebook groups to support social presence in online learning, suggests that students find facebook facilitates faster user interactions and communications than would be the case within an lms. some of the reasons cited for this include a less cumbersome authentication system in facebook compared to an lms and the ability to easily deploy this sm environment on mobile devices. in addition, in some countries and regions with inadequate information and communication technology (ict) infrastructure, sm maintained by overseas companies may be an essential supplement to lmss to facilitate communication between learners in online environments (al-azawei 2019). the integration of sm communication tools within an lms platform, therefore, has the potential to enhance both the speed and reliability of online delivery, as supported in chaka, nkhobo, and lephalala (2021)’s exploration of the use of whatsapp to support lms-enabled e-learning in africa. however, this will depend on environmental factors such as available communications infrastructure and the ability of sm communication channels to support and not compromise lms functions such as assessments that are not generally available within sm platforms. sm-lms integration – disadvantages there are disadvantages to incorporating sm platforms within lmss highlighted in the selected studies, which can potentially reduce the effectiveness of course delivery. these disadvantages are grouped into three broad categories: the need for ongoing support, sm distractions, and technical and security issues (figure 2). the following is a discussion of the identified integration disadvantages. need for ongoing support if a course delivery structure incorporates sm, support to students is required from the educator delivering the course and the institution (camus et al. 2016). students often create their own learning space within sm environments, which are autonomous self-functioning spaces outside institutional and educator control. therefore, the capacity for students to support each other when using sm applications needs to be fully identified by teachers before developing a course delivery framework (cao and saini 2021; coman et al. 2021; tull, dabner, and ayebi-arthur 2017). one study also  pointed out that heis do not necessarily approve of sm environments such as facebook, so there may be some resistance to learners using these environments as learning tools (charteris et al. 2018). http://dx.doi.org/10.25304/rlt.v31.2814 d. turnbull et al. 10 citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 (page number not for citation purpose) social media distractions & privacy/welfare concerns the capacity for sm applications to distract learner attention was also cited as a significant disadvantage of incorporating their use within an lms framework. in one study, students indicated that specific distractions, such as advertisements, games, and communicating with friends, impeded progress in their studies (kalelioğlu 2017). in two other studies, sm was cited as a significant factor in distracting students from their studies (chuang and liao 2021; clavier et al. 2019). it is evident from these studies that sm has the potential to be a source of distraction; salmon et al. (2015) have proposed that the risk could be mitigated by reducing the opportunities for the use of sm as part of course delivery. there were also concerns about the capacity for sm to interfere with the personal lives of students if used for educational purposes. a study on the role of sm in supporting structured online courses found that students and faculty were concerned about possible negative impacts on students’ private lives if the platform were to be used for educational purposes (salmon et al. 2015). another study on using facebook as a developmental writing tool highlighted safety, security and privacy issues as significant drawbacks of sm compared to lmss such as blackboard (ingalls 2017). in their study on the use of facebook to support social presence in online learning, akcaoglu and lee (2018) go one step further on student safety and welfare concerns by recommending that all critical information, such as important dates and reminders, be disseminated only within an lms environment and not via facebook. another interesting perspective is that students could pay more attention to the information and system functionality within an sm environment rather than the lms itself (al-azawei 2019). this could lead to a belief that sm is a more reliable source of content and learning than the institutional lms. technical and security issues there are, of course, possible technical impediments to integrating sm into lms supported course delivery that educational institutions must consider. judd and elliott (2017) highlight that sm accounts are owned by the student and are beyond the control of course administrators and teachers. this makes it difficult to seamlessly authenticate sm accounts within a single institutional logon. two studies also highlighted that facebook was not designed as an lms and cannot substitute an institutional lms by itself (kalelioğlu 2017; niu 2017). for example, an lms such as moodle can be configured with time constraints for assignment submission, whereas no such functionality exists within facebook (kalelioğlu 2017). there is also the issue of security. for example, it is challenging to verify the identity of students via self-created profiles on their sm sites (parusheva, aleksandrova, and hadzhikolev 2018) and to maintain personal privacy in an educational environment that requires interconnectedness (hamadi et al. 2020). finally, poor internet connectivity can constrain the usefulness of sm and other online platforms, such as lmss, particularly for students enrolled in distance learning courses (paidamoyo nyambuya et al. 2021). managerial implications for higher education institutions as outlined in this paper, there are many factors to consider when attempting to integrate sm with institutional lmss. the potential benefits and drawbacks of integrating these disparate platforms must be carefully evaluated before implementing a solution. for example, the benefit of increasing access to learning materials hosted in an lms via http://dx.doi.org/10.25304/rlt.v31.2814 research in learning technology citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 11 (page number not for citation purpose) sm platforms such as facebook must be carefully weighed against the security risks of confidential student data ‘leaking’ into the public domain. in addition, the security and safety of student accounts and data are of concern, especially under european union’s general data protection regulation (eu-gdpr). for teaching staff to ask students to create accounts on sm which are not ‘controlled’ and safeguarded by institutional protocols appears to be a barrier to the use of sm for heis. pedagogical issues must also be a prominent component of sm-lms integration decisions. teaching and learning strategies based on social constructivism theory have been identified in the introduction to this paper as essential drivers of e-learning. other key categories of learning theories (such as social constructivism and behaviourism) that may impact sm-lms integration include behaviourism, based on stimulus and response; cognitivism, focusing on information transmission and processing; and constructivism, founded on the personal discovery of knowledge (hung 2001). learner characteristics are also an important input into sm-lms integration design. perceptual modality, or the means through which information is obtained, either visually, aurally, or kinaesthetically (wislock 1993), is one way of categorising learners; cultural differences, as defined by hofstede (2011), could be another. adoption intention or the likelihood of an individual to invest the time and energy to incorporate specific technology platforms into their learning efforts is another important consideration. davis (1993)’s tam is a reasonable starting point to assess this motivator. having established an understanding of learner characteristics and their motivation to engage in e-learning, the utilisation of course delivery infrastructure needs to be carefully crafted to maximise students’ propensity to engage with the learning process. this paper proposes that a tight inter-relationship between sm and lmss can benefit course delivery within a social constructivist framework that satisfies innate human needs for social interaction. we offer the following recommendations: • the potential positive and negative consequences of sm-lms integration need to be carefully evaluated for each proposed instance of course delivery in order to create effective synergies that will leverage the strengths of each platform while minimising any potential problems. for example, 13 papers in our study featured improved communication as a positive benefit of lms-sm integration which supports social constructivist teaching practices applied through the use of sm (sarwar et al. 2018). however, teachers must carefully weigh this potential benefit against the capacity for sm applications to direct learner attention away from curriculum goals (chuang and liao 2021; clavier et al. 2019). • for heis that offer programs in a purely online format, selecting and promoting an sm platform familiar to students is recommended to sustain effective learner engagement. for example, facebook or twitter may be ubiquitous in a usage context in north america, but whatsapp could be a better choice for learners in africa. for example, cellular data charges for whatsapp use are very low in many african countries (omanga 2018), making it a popular choice for mobile phone users in the continent. heis must also consider the available infrastructure to support learner interactions within a structured course environment. • there is little utility in deploying a sophisticated online lms integrated with sm platforms to students in countries with inadequate network infrastructure to support activities requiring high bandwidth services. it may be better in these circumstances to rely more on offline resources supplemented by low bandwidth communication modalities such as text-based messaging to deliver course content. http://dx.doi.org/10.25304/rlt.v31.2814 d. turnbull et al. 12 citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 (page number not for citation purpose) the key to successfully integrating sm and lmss is to remain vigilant to the natural need for learners to interact socially with each other and recognise that this need is situationally dependent. there is no ‘one-size-fits-all’ to a successful marriage between popular sm platforms and institutional lms. conclusion social media is a rapidly growing global phenomenon (chugh, grose, and macht 2021). in addition, heis worldwide are increasingly relying on the use of lms to deliver course content and interact with learners. moreover, learners place importance on the social aspect of pursuing education and the hei they select (capraro et al. 2004). the social interactions between people in a learning environment can be an important motivator to enrol in a specific study course. in an increasingly online learning environment, leveraging the power of sm platforms such as facebook and twitter in conjunction with lmss could enhance learner experiences and improve the reputations of institutions as quality providers of educational services. the studies investigated in this review indicate that the advantages of integrating popular sm applications within the framework of lmss are more prevalent than the potential problems. this has significant implications for decision-makers in institutions of higher learning who may be tasked with creating a regulatory environment for the use and deployment of sm within their institution’s course delivery framework. this study is limited by the inclusion of english language publications only, and a focus on sm platforms that are prevalent in countries with democratic prevalence. for example, other regions, such as china, may present a different picture of sm-lms integration, so it is hoped that future research may contribute to a more comprehensive snapshot of global sm-lms integration efforts. also, it is possible that not all keywords that capture the essence of lmss, such as vles, have been included in the search string. hence, future researchers should consider expanding their search terms. another potential limitation is the adoption of a narrative approach to addressing the rq, which did not quantify the relative strengths and weaknesses of each of the selected studies. heis, their educators, and technical support professionals should pay careful attention to evolving trends in sm that may impact the design and deployment of established lmss. this study makes a case for critically analysing sm-lms integration that will be of assistance to lms developers and educators interested in leveraging popular sm platforms to enhance the learning outcomes for students in a digital (or e-learning) context. disclosure statement the authors report that there are no competing interests to declare. references *akcaoglu, m. & lee, e. (2018) ‘using facebook groups to support social presence in online learning’, distance education, vol. 39, no. 3, pp. 334–352. doi: 10.1080/01587919.2018.1476842 *al-azawei, a. (2019) ‘what drives successful social media in education and e-learning? a comparative study on facebook and moodle’, journal of information technology education, vol. 18, pp. 1–22. doi: 10.28945/4360 http://dx.doi.org/10.25304/rlt.v31.2814 https://doi.org/10.1080/01587919.2018.1476842 https://doi.org/10.28945/4360 research in learning technology citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 13 (page number not for citation purpose) *aleksandrova, y. g. & parusheva, s. s. (2019) ‘social media usage patterns in higher education institutions – an empirical study’, international journal of emerging technologies in learning (ijet), vol. 14, no. 05, pp. 108–121. doi: 10.3991/ijet.v14i05.9720 awidi, i. t., paynter, m. & vujosevic, t. (2019) ‘facebook group in the learning design of a higher education course: an analysis of factors influencing positive learning experience for students’, computers & education, vol. 129, pp. 106–121. doi: 10.1016/j.compedu.2018.10.018 baldwin, s. j. & ching, y.-h. (2020) ‘guidelines for designing online courses for mobile devices’, techtrends, vol. 64, no. 3, pp. 413–422. doi: 10.1007/s11528-019-00463-6 *camus, m., hurt, n. e., larson, l. r. & prevost, l. (2016) ‘facebook as an online teaching tool: effects on student participation, learning, and overall course performance’, college teaching, vol. 64, no. 2, pp. 84–94. doi: 10.1080/87567555.2015.1099093 *cao, m. & saini, d. k. (2021) ‘design and implementation of multidimensional interaction in online english course under the assistance of omnimedia’, scientific programming, vol. 2021, pp. 1–10. doi: 10.1155/2021/3713161 capraro, a. j., patrick, m. l. & wilson, m. (2004) ‘attracting college candidates: the impact of perceived social life’, journal of marketing for higher education, vol. 14, no. 1, pp. 93–106. doi: 10.1300/j050v14n01_06 *chaka, c., nkhobo, t. & lephalala, m. (2021) ‘leveraging moyama, whatsapp and online discussion forum to support students at an open and distance e-learning university’, electronic journal of e-learning, vol. 18, no. 6. pp. 494–515. doi: 10.34190/jel.18.6.003 *charteris, j., parkes, m., gregory, s., fletcher, p. & reyes, v. (2018) ‘student-initiated facebook sites: nurturing personal learning environments or a place for the disenfranchised?’, technology, pedagogy and education, vol. 27, no. 4, pp. 459–472. doi: 10.1080/1475939x.2018.1507924 *chuang, h.-m. & liao, y.-d. (2021) ‘sustainability of the benefits of social media on socializing and learning: an empirical case of facebook’, sustainability, vol. 13, no. 12, p. 6731. doi: 10.3390/su13126731 chugh, r. & ruhi, u. (2017) ‘social media in higher education: a literature review of facebook’, education and information technologies, vol. 23, no. 2, pp. 605–616. doi: 10.1007/ s10639-017-9621-2 chugh, r. & ruhi, u. (2019) ‘social media for tertiary education’, in: encyclopedia of education and information technologies, ed a. tatnall, springer nature, cham, pp. 1–6. chugh, r., grose, r. & macht, s. a. (2021) ‘social media usage by higher education academics: a scoping review of the literature’, education and information technologies, vol. 26, no. 1, pp. 983–999. doi: 10.1007/s10639-020-10288-z churcher, k. (2014) “‘ friending” vygotsky: a social constructivist pedagogy of knowledge building through classroom social media use’, journal of effective teaching, vol. 14, no. 1, pp. 33–50. *clavier, t., et al., (2019) ‘use of the smartphone app whatsapp as an e-learning method for medical residents: multicenter controlled randomized trial’, jmir mhealth uhealth, vol. 7, no. 4, p. e12825. doi: 10.2196/12825 *coman, c., et al., (2021) ‘dear student, what should i write on my wall? a case study on academic uses of facebook and instagram during the pandemic’, plos one, vol. 16, no. 9, p. e0257729. doi: 10.1371/journal.pone.0257729 *dalsgaard, c. (2016) ‘students’ educational use of facebook groups’, educational media international, vol. 53, no. 4, pp. 261–273. doi: 10.1080/09523987.2016.1254879 davis, f. d. (1993) ‘user acceptance of information technology: system characteristics, user perceptions and behavioral impacts’, international journal of man-machine studies, vol. 38, no. 3, pp. 475–487. doi: 10.1006/imms.1993.1022 *frankel, c. e. (2020) a case study on a grassroots, student-led facebook community for online graduate students and alumni, final project, university of massachusetts, boston, pp. 1–55. *ghounane, n. (2020) ‘moodle or social networks: what alternative refuge is appropriate to algerian efl students to learn during covid-19 pandemic’, arab world english journal, vol. 11, no. 3, pp. 21–41. doi: 10.24093/awej/vol11no3.2 http://dx.doi.org/10.25304/rlt.v31.2814 https://doi.org/10.3991/ijet.v14i05.9720 https://doi.org/10.1016/j.compedu.2018.10.018 https://doi.org/10.1007/s11528-019-00463-6 https://doi.org/10.1080/87567555.2015.1099093 https://doi.org/10.1155/2021/3713161 https://doi.org/10.1300/j050v14n01_06 https://doi.org/10.34190/jel.18.6.003 https://doi.org/10.1080/1475939x.2018.1507924 https://doi.org/10.3390/su13126731 https://doi.org/10.1007/s10639-017-9621-2 https://doi.org/10.1007/s10639-017-9621-2 https://doi.org/10.1007/s10639-020-10288-z https://doi.org/10.2196/12825 https://doi.org/10.1371/journal.pone.0257729 https://doi.org/10.1080/09523987.2016.1254879 https://doi.org/10.1006/imms.1993.1022 https://doi.org/10.24093/awej/vol11no3.2 d. turnbull et al. 14 citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 (page number not for citation purpose) gosper, m., et al., (2014), ‘student use of technologies for learning – what has changed since 2010?’, proceedings of ascilite 2014-annual conference of the australian society for computers in tertiary education, dunedin, new zealand, pp. 1–12. *hamadi, m., et al., (2020) ‘a social media adoption framework as pedagogical instruments in higher education classrooms’, e-learning and digital media, vol. 18, no. 1, pp. 55–85. doi: 10.1177/2042753020950869 hew, k. f., et al., (2019) ‘where is the “theory” within the field of educational technology research?’, british journal of educational technology, vol. 50, no. 3, pp. 956–971. doi: 10.1111/bjet.12770 hofstede, g. (2011) ‘dimensionalizing cultures: the hofstede model in context’, online readings in psychology and culture, vol. 2, no. 1, pp. 1–26. doi: 10.9707/2307-0919.1014 hrastinski, s. & aghaee, n. m. (2011) ‘how are campus students using social media to support their studies? an explorative interview study’, education and information technologies, vol. 17, no. 4, pp. 451–464. doi: 10.1007/s10639-011-9169-5 hung, d. (2001) ‘theories of learning and computer-mediated instructional technologies’, educational media international, vol. 38, no. 4, pp. 281–287. doi: 10.1080/09523980110105114 *ingalls, a. l. (2017) ‘facebook as a learning-management system in developmental writing’, journal of developmental education, vol. 40, no. 2, pp. 26–28. *judd, t. & elliott, k. (2017) ‘methods and frequency of sharing of learning resources by medical students’, british journal of educational technology, vol. 48, no. 6, pp. 1345–1356. doi: 10.1111/bjet.12481 *kalelioğlu, f. (2017) ‘using facebook as a learning management system: experiences of pre-service teachers’, informatics in education, vol. 16, no. 1, pp. 83–101. doi: 10.15388/ infedu.2017.05 kasim, n. & khalid, f. (2016) ‘choosing the right learning management system (lms) for the higher education institution context: a systematic review’, international journal of emerging technologies in learning (ijet), vol. 11, no. 6, pp. 55–61. doi: 10.3991/ijet.v11i06.5644 kellogg, k. (2020) ‘the 7 biggest social media sites in 2020’, search engine journal, [online]. available at: https://www.searchenginejournal.com/social-media/biggest-social-media-sites/ *kent, m. (2016) ‘adding to the mix: students use of facebook groups and blackboard discussion forums in higher education’, knowledge management and e-learning, vol. 8, no. 3, pp. 444–463. doi: 10.34105/j.kmel.2016.08.028 *khan, a. a., et al., (2017) ‘impact of network aided platforms as educational tools on academic performance and attitude of pharmacology students’, pakistan journal of medical sciences, vol. 33, no. 6, pp. 1473–1478. doi: 10.12669/pjms.336.13290 *khechine, h., raymond, b. & augier, m. (2020) ‘the adoption of a social learning system: intrinsic value in the utaut model’, british journal of educational technology, vol. 51, no. 6, pp. 2306–2325. doi: 10.1111/bjet.12905 kircaburun, k., alhabash, s., tosuntaş, ş. b. & griffiths, m. d. (2018) ‘uses and gratifications of problematic social media use among university students: a simultaneous examination of the big five of personality traits, social media platforms, and social media use motives’, international journal of mental health and addiction, vol. 18, no. 3, pp. 525–547. doi: 10.1007/s11469-018-9940-6 lampe, c., et al., (2011) ‘student use of facebook for organizing collaborative classroom activities’, international journal of computer-supported collaborative learning, vol. 6, no. 3, pp. 329–347. doi: 10.1007/s11412-011-9115-y li, x. ganeshan, k. & xu, g. (2012) ‘the role of social networking sites in e-learning’, 2012 frontiers in education conference proceedings, seattle, wa, usa, pp. 1–6. li, x., he, x. & zhang, y. (2019) ‘the impact of social media on the business performance of small firms in china’, information technology for development, vol. 26, no. 2, pp. 346–368. doi: 10.1080/02681102.2019.1594661 http://dx.doi.org/10.25304/rlt.v31.2814 https://doi.org/10.1177/2042753020950869 https://doi.org/10.1111/bjet.12770 https://doi.org/10.9707/2307-0919.1014 https://doi.org/10.1007/s10639-011-9169-5 https://doi.org/10.1080/09523980110105114 https://doi.org/10.1111/bjet.12481 https://doi.org/10.15388/infedu.2017.05 https://doi.org/10.15388/infedu.2017.05 https://doi.org/10.3991/ijet.v11i06.5644 https://doi.org/10.34105/j.kmel.2016.08.028 https://doi.org/10.12669/pjms.336.13290 https://doi.org/10.1111/bjet.12905 https://doi.org/10.1007/s11469-018-9940-6 https://doi.org/10.1007/s11412-011-9115-y https://doi.org/10.1080/02681102.2019.1594661 research in learning technology citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 15 (page number not for citation purpose) lockyer, l. & patterson, j. (2008) ‘integrating social networking technologies in education: a case study of a formal learning environment’, 2008 eighth ieee international conference on advanced learning technologies, santander, spain, pp. 529–533. *mansour, e. (2021) ‘utilization of online learning platforms by lis arab faculty members during the coronavirus outbreak’, journal of library & information services in distance learning, vol. 15, no. 1, pp. 18–40. doi: 10.1080/1533290x.2021.1896619 martínez-alemán, a. m. (2014) ‘social media go to college’, change: the magazine of higher learning, vol. 46, no. 1, pp. 13–20. doi: 10.1080/00091383.2014.867203 *moghavvemi, s., et al., (2017) ‘student’s perceptions towards using e-learning via facebook’, behaviour & information technology, vol. 36, no. 10, pp. 1081–1100. doi: 10.1080/0144929x.2017.1347201 mozhaeva, g., feshchenko, a. & kulikov, i. (2014) ‘e-learning in the evaluation of students and teachers: lms or social networks?’, procedia – social and behavioral sciences, vol. 152, pp. 127–130. doi: 10.1016/j.sbspro.2014.09.168 *mpungose, c. b. (2019) ‘is moodle or whatsapp the preferred e-learning platform at a south african university? first-year students’ experiences’, education and information technologies, vol. 25, pp. 927–941. doi: 10.1007/s10639-019-10005-5 njoki, h. (2021) ‘influence of technology type on development of instructional materials for distance education’, journal of online and distance learning, vol. 1, no. 1, pp. 60–74. doi: 10.47941/jodl.638 *niu, l. (2017) ‘using facebook for academic purposes: current literature and directions for future research’, journal of educational computing research, vol. 56, no. 8, pp. 1384–1406. doi: 10.1177/0735633117745161 *oktavia, t., supangkat, s. h. & prabowo, h. (2018) ‘a comparison of learning experience: social learning systems and e-learning systems in higher education institution’, icic express letters. part b, applications: an international journal of research and surveys, vol. 9, no. 12, pp. 1201–1208. doi: 10.24507/icicelb.09.12.1201 omanga, d. (2018) “whatsapp as ‘digital publics’: the nakuru analysts and the evolution of participation in county governance in kenya”, journal of eastern african studies, vol. 13, no. 1, pp 175–191. doi: 10.1080/17531055.2018.1548211 *paidamoyo nyambuya, v., nyamaruze, p., dube, m. & shumba, k. (2021) ‘rethinking education in the age of “social distancing”: a qualitative inquiry on university of kwazulunatal students’ responses to online learning in the context of covid-19’, journal of african education, vol. 2, no. 1, pp. 107–126. doi: 10.31920/2633-2930/2021/v2n1a5 *parusheva, s., aleksandrova, y. & hadzhikolev, a. (2018) ‘use of social media in higher education institutions – an empirical study based on bulgarian learning experience’, tem journal, vol. 7, no. 1, p. 171. doi: 10.18421/tem71-21 pilli, o. (2014) ‘lms vs. sns: can social networking sites act as a learning management systems’, american international journal of contemporary research, vol. 4, no. 5, pp. 90–97. pinner, r. (2011) ‘what is the difference between a vle and an lms’, call (review), iatefl learning technologies newsletter, summer, pp. 1–8. rahman, s., ramakrishnan, t. & ngamassi, l. (2019) ‘impact of social media use on student satisfaction in higher education’, higher education quarterly, vol. 74, no. 3, pp. 304–319. doi: 10.1111/hequ.12228 *raspopovic, m., et al., (2017) ‘the effects of integrating social learning environment with online learning’, international review of research in open and distributed learning, vol. 18, no. 1, pp. 142–160. doi: 10.19173/irrodl.v18i1.2645 *sabin, j. & olive, a. (2018) ‘slack: adopting social-networking platforms for active learning’. ps: political science & politics, vol. 51, no. 1, pp. 183–189. doi: 10.1017/s1049096517001913 *salmon, g., et al., (2015) ‘the space for social media in structured online learning’, research in learning technology, vol. 23, no. 1, pp. 1–14. doi: 10.3402/rlt.v23.28507 http://dx.doi.org/10.25304/rlt.v31.2814 https://doi.org/10.1080/1533290x.2021.1896619 https://doi.org/10.1080/00091383.2014.867203 https://doi.org/10.1080/0144929x.2017.1347201 https://doi.org/10.1016/j.sbspro.2014.09.168 https://doi.org/10.1007/s10639-019-10005-5 https://doi.org/10.47941/jodl.638 https://doi.org/10.1177/0735633117745161 https://doi.org/10.24507/icicelb.09.12.1201 https://doi.org/10.1080/17531055.2018.1548211 https://doi.org/10.31920/2633-2930/2021/v2n1a5 https://doi.org/10.18421/tem71-21 https://doi.org/10.1111/hequ.12228 https://doi.org/10.19173/irrodl.v18i1.2645 https://doi.org/10.1017/s1049096517001913 https://doi.org/10.3402/rlt.v23.28507 d. turnbull et al. 16 citation: research in learning technology 2023, 31: 2814 http://dx.doi.org/10.25304/rlt.v31.2814 (page number not for citation purpose) sarwar, b., et al., (2018) ‘usage of social media tools for collaborative learning: the effect on learning success with the moderating role of cyberbullying’, journal of educational computing research, vol. 57, no. 1, pp. 246–279. doi: 10.1177/0735633117748415 schrader, d. e. (2015) ‘constructivism and learning in the age of social media: changing minds and learning communities’, new directions for teaching and learning, vol. 2015, no. 144, pp. 23–35. doi: 10.1002/tl.20160 *tagoe, m. a. & cole, y. (2020) ‘using the sakai learning management system to change the way distance education nursing students learn: are we getting it right?’, open learning, vol. 35, no. 3, pp. 201–221. doi: 10.1080/02680513.2019.1704232 thai, m., sheeran, n. & cummings, d. j. (2019) ‘we’re all in this together: the impact of facebook groups on social connectedness and other outcomes in higher education’, the internet and higher education, vol. 40, pp. 44–49. doi: 10.1016/j.iheduc.2018.10.001 *todorovic, m., et al., (2021) ‘twelve tips for using facebook as a learning platform’, medical teacher, vol. 43, no. 11, pp. 1261–1266. doi: 10.1080/0142159x.2020.1854708 *trivedi, s. k., patra, p. & singh, s. (2021) ‘a study on intention to use social media in higher education: the mediating effect of peer influence’, global knowledge, memory and communication, vol. 72, no. 1–2, pp. 52–69. doi: 10.1108/gkmc-11-2020-0169 *tull, s., dabner, n. & ayebi-arthur, k. (2017) ‘social media and e-learning in response to seismic events: resilient practices’, journal of open, flexible, and distance learning, vol. 21, no. 1, pp. 63–76. *turkyilmaz, i., hariri, n. h. & jahangiri, l. (2019) ‘student’s perception of the impact of e-learning on dental education’, the journal of contemporary dental practice, vol. 20, no. 5, pp. 616–621. doi: 10.5005/jp-journals-10024-2568 turnbull, d., chugh, r. & luck, j. (2019) ‘learning management systems: an overview’, in encyclopedia of education and information technologies, ed a. tatnall, springer nature, cham, pp. 1–7. doi: 10.1007/978-3-319-60013-0_248-1 turnbull, d., chugh, r. & luck, j. (2023) ‘systematic-narrative hybrid literature review: a strategy for integrating a concise methodology into a manuscript’, social sciences & humanities open, vol. 7, no. 1, pp. 1–4. doi: 10.1016/j.ssaho.2022.100381 *wautelet, et al., (2016) ‘designing an mooc as an agent-platform aggregating heterogeneous virtual learning environments’, behaviour & information technology, vol. 35, no. 11, pp. 980–997. doi: 10.1080/0144929x.2016.1212095 williams, r. (2022) ‘an academic review of virtual learning environments’, icrrd quality index research journal, vol. 3, no. 2, pp. 143–145. doi: 10.53272/icrrd wise, l., skues, j. & williams, b. (2011) ‘facebook in higher education promotes social but not academic engagement’, changing demands, changing directions. proceedings ascilite hobart, tasmania, australia, pp. 1332–1342. wislock, r. f. (1993) ‘what are perceptual modalities and how do they contribute to learning?’, new directions for adult and continuing education, vol. 59, pp. 5–13. doi: 10.1002/ ace.36719935903 http://dx.doi.org/10.25304/rlt.v31.2814 https://doi.org/10.1177/0735633117748415 https://doi.org/10.1002/tl.20160 https://doi.org/10.1080/02680513.2019.1704232 https://doi.org/10.1016/j.iheduc.2018.10.001 https://doi.org/10.1080/0142159x.2020.1854708 https://doi.org/10.1108/gkmc-11-2020-0169 https://doi.org/10.5005/jp-journals-10024-2568 https://doi.org/10.1007/978-3-319-60013-0_248-1 https://doi.org/10.1016/j.ssaho.2022.100381 https://doi.org/10.1080/0144929x.2016.1212095 https://doi.org/10.53272/icrrd https://doi.org/10.1002/ace.36719935903 https://doi.org/10.1002/ace.36719935903 doi:10.3402/rlt.v19s1/7801 studying the learning of programming using grounded theory to support activity theory maryam kheir abadi* and graham alsop information systems and mathematics, kingston university, kingston upon thames, surrey, uk (received 17 may 2011; accepted 20 june 2011) teaching programming to first year undergraduates in large numbers is challenging. currently, online supported learning is becoming more dominant, even on face-to-face courses, and this trend will increase in the future. this paper uses activity theory (at) to analyse the use of tools to support learning. data collection took place during 2008�2010 at kingston university and involves over one hundred responses. this has been analysed into activity systems offering a detailed analysis of the use of a number of tools being used (in at these include physical tools, such as technologies including books, and non-physical tools, such as conversation). when teaching programming to large numbers of students it is difficult to offer one-to-one attention and the reliance on such tools becomes more important. for example, in student responses a good integrated development environment (ide) is shown to make learning easier and more enjoyable, whereas a bad ide makes the learning experience poor. teaching materials, and access to these, were often mentioned positively. these included online communication, discussion boards and video lectures. using at offers sufficiently rich detail to identify key interventions and aids the redesign of the learning process. for example, the choice of an ide for a specific language can have a larger impact than is initially apparent. this paper will report on the data collected to show where simple improvements to the use of tools may have a large impact on students’ abilities to learn programming. keywords: learning programming; activity theory; grounded theory scope the communities that have been involved in this research are students, staff and the researchers. ‘‘staff ’’ is broadly defined as whoever teaches and helps students to learn. this includes technicians who support students with any difficulty they might have using software and associated development environments. however, this paper presents only the analysis of data from two groups of students. staff experiences will be reported subsequently. *corresponding author. email: maryam@kingston.ac.uk alt-c 2011 conference proceedings 0137 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # m.k. abadi and g. alsop. this is an open access article 57 distributed under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1.7801 methodology as alsop and tompsett (2002) suggest, the methods that researchers use to collect data seem unimportant to students and to stakeholders such as lecturers. however, to achieve accurate and reliable data, the choice of methodology is critical. activity theory (at) was selected because of the nature of the subject being examined. there are multiple communities involved in looking for the same outcome. the outcome could for instance be that of passing a specific assessment. at allows for a holistic consideration of the multiple perspectives involved. its ontology requires different research methods depending on the aspect being considered. in particular the choice of methodological approach to study the subject’s activity is key. however, at does not specify any particular research methodology to be used. we have chosen to combine it with grounded theory (gt). the case studies undertaken here have particular characteristics that need considering. these include: . the nature of the subject being examined (learning programming); . the number of students involved [in our case the numbers being relatively small and so qualitative research methods were chosen to ‘provide a rich description of the students’ behaviours’ (alsop and tompsett 2002) during the research]; . the changes in sample during the research life-cycle; and . the multiple communities involved in looking for the same outcome (in our case passing an assessment). we now present short introductions to at and gt to clarify why they were chosen and how they can work together. activity theory activity theory is a psychological framework used to understand human activities. at was introduced by vygotsky (1896, 1934) and developed by leont’ev (1981). thereafter, many researchers have used at in various subject areas. for example, kuutti (1995) and nardi (1996) used at as a potential framework for human computer interactions and for transforming work in information systems. scanlon and issroff (2002, 2005) specifically utilised at on the use of technology in higher education (he). engeström (1999, 2000, 2008) employed at to examine individual and social transformation. they also developed the concept of an activity system (as) to illustrate at. figure 1 illustrates the generic as. nardi (1996) explains that an activity is the unit of analysis in at and that the subject is the person or the group involved in the activity. the object stimulates the activity and provides goals and directions to the actions. tools are the artefacts that can be used in the process of an activity. other important factors in an at framework are rules, communities and the division of labour. the whole result of an activity is the outcome or objective of the activity. m.k. abadi and g. alsop 58 combining gt and at the choice of at was justified by reviewing the characteristics of the case being studied, learning programming using specific tools on undergraduate taught modules involving multiple communities. we required a rich collection of information to ensure that at ontology is described well. choosing a method to collect the key data about how and what is used by students in learning and using it to show whether they have ‘‘learnt programming’’ is a challenge. the decision to focus on a qualitative approach was driven by several factors: sample size, accommodating researcher bias, and a changing sample during the research cycle. grounded theory grounded theory (gt) was first introduced by glaser and strauss in social science almost 50 years ago. gt is an inductive qualitative research method that uses a systematic approach to constantly compare collected data and analysis. here inductive means that there are no initial hypotheses. accordingly, the researcher has to be as open minded as possible and design the research questions carefully. grounded theory interacts closely with data. any possible hypothesis or theory is driven from the data, as glaser and strauss suggest: . . .clearly, a grounded theory that is faithful to the everyday realities of the substantive area is one that has been carefully induced from the data. (1967, p.239) while glaser and strauss (1967) believe that new concepts and reality can be discovered from the collected data, corbin (2008) argues that there is no reality out there waiting to be discovered, rather there are concepts and ideas that can be invented. she continues that humans do not discover reality. for example, schwandt (1998) states that: . . .constructivist means that human beings do not find or discover knowledge so much as construct or make it. we invent concepts, models and schemes to make sense of figure 1. generic activity system. alt-c 2011 conference proceedings 59 experience and, further, we continually test and modify these constructions in light of new experiences. charmaz (2006) also believes that theory is constructed from the data: grounded theory involves taking comparisons from data and reaching up to construct abstractions and then down to tie these abstractions to data. it means learning about the specific and the general � and seeing what is new in them � then exploring their links to larger issues or creating larger unrecognized issues in entirety... grounded theory methods can provide a route to see beyond the obvious and a path to reach imaginative interpretations. (charmaz, 2006) gtm is categorized as an inductive method. induction can be defined as a type of reasoning that begins with study of a range of individuals’ cases and extrapolates from them to form a conceptual category. (charmaz, 2006) other methods and approaches were considered. these included action research and phenomenography. the choice of gt above other methods and approaches appears to have been sound in the light of the initial outcomes. a further paper on the methodological issues is in preparation. activity theory helped to break down complicated situations and made them easier to analyse. gt allowed us to have a flexible and open approach to data collection. it also allowed us to decrease the number of presumptions and hypotheses which would have limited the possibilities of findings. however, the decision to use at implied the need to conform to ontology. this led to data collection using some of the terms required by at. data collection two open-ended questions (adapted from alsop and tompsett 2002) were used in this research. students were asked to write about their best and worst educational experiences of learning programming and to specify the tools they used. they were also asked to summarise their stories in their own words. this was to ensure that the data received were framed in the language of the ‘students’ rather than the ‘researchers’. data analysis in using gt, the collected data were examined closely. in considering each response, questions were asked such as what has happened and why has it happened? an as was built for each response as well as associated notes that included the researcher’s analysis of the case. the early stages of gt analysis were then used. this included open coding (whilst keeping in mind at’s ontology which includes subject, object, tools, rules, communities and division of labour). in the first instance, the focus taken was on which tools the students have used and which communities have been involved in that event. thereafter, axial coding was undertaken in two sets; one for the worst experiences and one for the best experiences. for the former this represents, put in the language of at, ‘contradictions’ that in an as need to be overcome for the activity to be successful (the resulting redesign, again specifically in the language of at, is known as a ‘shift’ in an as.) m.k. abadi and g. alsop 60 building activity systems a first year student (subject), who attends the ‘‘programming essential module’’ (object) in the first semester, is taken as a starting point. she/he goes to the lecture and listens to the lecturer who aims to explain the basic concepts of programming in java. she/he picks up a handout and annotates it (this represents a tool in at terms). after the lecture, the student has the opportunity to go to the workshop to put the theory into practice. here, she/he practices java in a real environment using several other tools (textpad, the www, accessing the internet, notes and books.) she/he can also receive help from lab assistants (another community). she/he is interacting with a machine, reading her/his notes and books and interacting with students, lecturer and assistants in order to achieve her/his goal (outcome) of ‘‘learning programming’’. as engeström (1999) suggests in order to achieve a specific outcome there needs to be a subject, object, rules, tools, division of labour/effort and finally communities. figure 2 has been annotated from engeström to illustrate the example. in the process of learning programming there are very likely to be problems, clashes, breaks and difficulties. engeström (1999), nardi (1996) and roussou, oliver, and slater (2007) call these disturbances ‘‘contradictions’’. analysis of data in this research aims to identify and clarify these contradictions and help identify how they can be solved to make the process of learning programming better, more smooth and enjoyable for students. for example, if the student above begins by writing a simple program called ‘‘hello world’’ in java, compiles code and then faces syntax errors, she/he could either solve the errors with no help or call on help. this reflects the ability to self-correct or need an intervention from someone else in order to solve the difficulty. to move away from a general example to something more specific, we give a response citing a student’s worst experience: ‘‘while creating a game in c �� , using provided engine, i couldn’t get it to do what i needed. the program compiled fine, so it was down to my logic. the lecturer suggested using break points to find my mistake � but i could not understand what they were telling me or if i was using them correctly. this was the most frustrating experience of a figure 2. an illustration of the methodology. alt-c 2011 conference proceedings 61 few while using the visual studio suite � i didn’t like using such a complex program without understanding how to use it properly, or having a thorough knowledge of the language beforehand.’’ � ‘‘not knowing how to use the program’’. we build up an as. the coding is shown in table 1. figure 3 shows the as of the above event. in this example shifts (improvements) are needed in both the tool and communication process. the tool seems to be too complex for the student to use. why is this the case? it could be because the environment is new or perhaps because visual studio is not a good development tool for programming? we do not know the answer to the latter question. however, we can investigate the former question. the problem of a new environment can be solved by a ‘‘short lived goal directed’’ action (engeström 1999). a shift in the object of the activity of learning programming in c �� to ‘‘learn the ide’’, in this case visual studio, together with a short-term shift in the division of labour by having workshops to learn the integrated development environment (ide) instead of writing code/ programming could help. in figure 4, these shifts have been illustrated. in contrast to a failed as, an analysis of a good experience follows. this response was chosen from one of the set of best educational experiences from the same group as the previous student. this response, however, was not selected randomly. it was chosen because this student (subject) shares the same object with the randomly selected one above. however, the description is of a best rather than a worst educational experience. this shows that the same object can be the reason for both good and bad experiences. ‘‘the best educational experience when using a programming tool would be the time when i had to program a game using c��. normally i would find this challenging so i decided to do more independent work using the program such as reading books and practicing on simple programs. by the time i’d finished i had created what i thought exceeded my expectations and for which i received a good mark. this was very satisfying and now i spend longer on independent work.’’ � ‘‘spending time on a program is beneficial and is helpful for work’’. this leads to the following coding (table 2) and as (figure 5) developed in the same way as the previous example. figure 5 shows the as of the above event. table 1. an activity system coding. subject student object game design tool visual studio (vs � an ide from microsoft to develop programs in c �� ) community student and the lecturer division of labour (dol) student writes the codes and lecturer helps to find errors outcome confusion, frustration, student did not understand and not happy to use a complex program note: researcher’s interpretation: it seems that the student is not comfortable with the tool (vs), and finds it a to be a complex program. despite seeking help she/he still does not understand the problem. is it because the problem has not been fully explained or is it too hard for the student to digest? this student would prefer to understand the concepts before using it in the provided ide. she/he summarises the story: ‘‘not knowing how to use the program’’ m.k. abadi and g. alsop 62 subsequently, ass were built for each response and these were then categorised. in gt, corbin (2008) call this process ‘‘identify concepts from data’’. in other words, labelling data with specific words and terms, adding commentary about data analysis, stating comparisons and investigating ideas that appear in data. the result of this process for the failed ass is shown in figure 6. since the students were asked to summarise their stories, those summaries guided the researcher to label each response and then classify these into groups. knowledge, structure, tools and programming languages are the four main categories that were driven from the data. knowledge contradictions are mostly related to syntax and materials. some of the students found the syntax, taught for the specific programming language, hard to learn or too much for the duration of the semester. some other students pointed out that the materials available for the modules did not cover the harder assignments. figure 3. a specific activity system. figure 4. modified activity system. alt-c 2011 conference proceedings 63 tools are another contradiction in the process of learning programming for these students. ides specifically seem to be limited for the work they do. students also highlighted that some of the ides in use are not helpful in terms of solving errors. the same analysis is applied to the positive experiences in figure 7, which illustrates the best educational experiences. conclusions there are two main conclusions. firstly, using at with gt has led to the identification of contradictions that require shifts to lead to successful ass that ensure that students are better able to learn programming. these are that: 1. the choice of ide is important. simple ides do not provide the required feedback to ensure adequate problem solving. 2. it seems that the tools are not as important as the behaviour/motivation of the student toward learning. utilising all available tools (such as books, ides, online videos, search engines, blackboard, study space, etc.) can figure 5. associated activity system. table 2. a further activity system coding. subject student object game design tool books community student division of labour (dol) more independent work and reading books outcome good grade, satisfaction and encouragement note: researcher’s interpretation: from previous experience the student knew that she/he might have problems with designing the program, that is why she/he decided to do more independent work, study more, read related books to improve his/her ability to design the game. the student summary is clear: ‘‘spending time on a program is beneficial and is helpful for work’’. m.k. abadi and g. alsop 64 increase the motivation to learn. these tools make it easier to learn independently of location and time. 3. the communities involved in the activity of learning programming are another high priority in the responses. the better the level of communication, the more rewarding the activity. the interactions between communities figure 7. analysis of positive activity systems. figure 6. analysis of failed activity systems. alt-c 2011 conference proceedings 65 include: student�student, student�lecturer, student�helpers and student� technicians. the involvement of technicians is critical because most of the technical problems occur the first few times that students begin to use a new environment. 4. a large mixed group does not help to support learning. designing a suitable lecture/workshop for even the majority of a large diverse set of students is very difficult. examining all students’ programming knowledge before the first programming module is one possible solution to enable streaming or, at least, offering more targeted support/advice. secondly, in choosing at as the framework, there was an implied need to find an appropriate approach to collect and aid the analysis of data, which needed also to be compatible with at. gt was chosen to complement at. using gt offered sufficiently rich detail to identify key interventions and ways to redesign the learning process. at helps to clarify any contradictions in an as and provides a means to design changes/shifts to solve these contradictions. using at with a series of developing ass can show the history of contradictions, changes and shifts during the process of learning. this aids the development, knowledge, structure and design of a better learning environment for the future. as engeström (2000) argues, there is never a finished product in the learning process since there is always a moving target. references alsop, g., and c. tompsett. 2002. grounded theory as an approach to studying students’ uses of learning management systems. alt-j 10, no. 2: 63�76. corbin, j., and a.l. strauss. 2008. basic of qualitative research, techniques and procedure for developing grounded theory, 3rd ed. london: sage. charmaz, k. 2006. constructing grounded theory, a practical guide through qualitative analysis. london, thousand oaks: sage. engeström, y. 1999. activity theory and expansive design. http://projectsfinal.interaction ivrea.org/2004-2005/symposium%202005/communication%20material/activity%20 theory%20and%20expansive%20design_engestrom.pdf (accessed june 2009). engeström, y. 2000. activity theory as a framework for analyzing and redesigning work. ergonomics 43, no. 7: 960�74. engeström, y. 2008. enriching activity theory without shortcuts. interacting with computers 20, 2: 256�9. perseus digital library. www.elsevier.com/locate/intcom glaser, b.g., and a.l. strauss. 1967. the discovery of grounded theory, strategies for qualitative research. aldine pub. co: weildenfield and nicolson kuutti, k. 1995. activity theory as a potential framework for human computer interaction research, context and conciseness, 2 vols. 17�44. cambridge, ma: mit press. leont’ev, a.n. 1981. activity and consciousness in the development of mind, selected works of aleksei nikolaevich leontyev. trans. m. kopylova and ed. a. blunden. http://marxists.org/ archive/leontev/works/development-mind.pdf (accessed 5 july, 2010). nardi, b.a. 1996. studying context: a comparison of activity theory, situated action modules, and distributed cognition, 4 vols. 35�52. cambridge, ma: massachusetts institute of technology. roussou, m., m. oliver, and m. slater. 2008. exploring activity theory as a tool for evaluating interactivity and learning in virtual environments for children. congnition technology & work 10: 141�53. scanlon, e., and issroff, k. 2002. educational technology: the influence of theory. journal of interactive media in education 6, p. 1�13. [www-jime.open.ac.uk/2002/6] scanlon, e., and k. issroff. 2005. activity theory and higher education: evaluating learning technologies. journal of computers and learning 20, no. 6: 430�9. m.k. abadi and g. alsop 66 http://projectsfinal.interactionivrea.org/2004-2005/symposium%202005/communication%20material/activity%20theory%20and%20expansive%20design_engestrom.pdf http://projectsfinal.interactionivrea.org/2004-2005/symposium%202005/communication%20material/activity%20theory%20and%20expansive%20design_engestrom.pdf http://projectsfinal.interactionivrea.org/2004-2005/symposium%202005/communication%20material/activity%20theory%20and%20expansive%20design_engestrom.pdf http://www.elsevier.com/locate/intcom http://marxists.org/archive/leontev/works/development-mind.pdf http://marxists.org/archive/leontev/works/development-mind.pdf http://www-jime.open.ac.uk/2002/6 doi:10.3402/rlt.v19s1/7795 towards a framework for co-creating open scholarship fred garnett a, * and nigel ecclesfield b a london knowledge lab, london wc1n 3qs, uk; b nigel ecclesfield, technology, research and evaluation, learning & skills improvement service(lsis), coventry, cv1 2te, uk (received 13 june 2011; final version received 20 june 2011) a recent edition of alt-j made a call for papers that looked at ‘theoretical approaches in digitally mediated environments’. a key part of this call was to use the boyer model of scholarship as a frame of reference. the authors felt that there were limitations to this model which could be addressed in light of the recent moves to develop open scholarship. our concern with boyer is that he suggests a separation between researchers, who ‘build new knowledge through traditional research’ and teachers who ‘study teaching models and practices to achieve optimal learning’. boyer identifies four ‘types’ of scholarship, those of discovery, integration, application and teaching (diat), but places the responsibility for ‘creative work in established field’, with the traditional researcher role (discovery). furthermore this model implies a linear flow concerning how new knowledge becomes a part of teaching, implying that the teaching is mostly instructional, with a limited view of how new and emerging pedagogies might be utilised. the learner-generated contexts research group has been concerned to develop a co-creation approach to learning and find this separation curious. we argue that using the pedagogy, andragogy, heutagogy (pah) continuum enables more flexible approaches, through a mix of pah, allowing for a wide range of technology uses, which also changes the relationship to research. we look at how we might both apply a co-creation approach to boyer’s model, inspired by the open scholar movement, and also make diat more iterative and less discrete. consequently we have both extended boyer’s diat system to include co-creating as an additional type and changed some ‘measures of performance’ to enable an iterative process of scholarship to emerge which also involves learners. we also examine how network effects ‘enable generative network effects to occur’ on scholarship and how applying epistemic cognition to evolving subject frameworks might enable the co-creation of research agendas. the co-creation model of open scholarship is presented in a table designed to simulate debate on this subject. keywords: open scholar; scholarship; research; co-creation; contexts; teaching; pah continuum; epistemic cognition; open education resources; oer; networked learning introduction a recent edition of alt-j made a call for papers that looked at ‘theoretical approaches in digitally mediated environments’. a key part of this call was to use the boyer model of scholarship boyer (1997) as a frame of reference upon which to base *corresponding author. email: fred.garnett@gmail.com alt-c 2011 conference proceedings 0199 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # f. garnett and n. ecclesfield. this is an open access article 5 distributed under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproductioninanymedium,providedtheoriginalworkisproperlycited. doi:10.3402/rlt.v19s1/7795 such new theoretical approaches. the authors felt that there were limitations to this, perfectly valid, model which could be addressed in light of the recent moves to develop a model of open scholarship (anderson 2009), and other theories reflecting the ‘networked age’, such as haythornthwaite in new forms of doctorate (2009) and our own open context model of learning and the pedagogy, andragogy, heutagogy (pah) continuum (luckin et al. 2010). our concern with boyer’s model lies in the fact that it suggested a separation between researchers, who ‘build new knowledge through traditional research’ and teachers who ‘study teaching models and practices to achieve optimal learning’. boyer usefully identifies four ‘types of scholarship’, those of discovery, integration, application and teaching (diat), but arrogated the responsibility for ‘creative work in established fields’ solely to discovery scholarship (the ‘traditional researcher role’). furthermore this model also implies a linear flow concerning how new knowledge becomes a part of teaching, which suggests that the type of teaching that results is more instructional. in our opinion this reveals a perhaps limited view of how pedagogies, both existing and emerging, might be deployed by an experienced teacher. the learner-generated contexts research group has been concerned to develop a co-creation approach to learning and consequently find this separation curious. we would argue that using the pah continuum, in ways described by for example cochrane (2010a), enables more flexible approaches to learning and teaching by using a mix of pah (which also allows for a wide range of technology uses). this also changes the teacher’s relationship to ‘research’ through the development of ‘epistemic cognition’ in the learner (avramides and luckin 2007), or action research strategies (cochrane 2010b). so, in part inspired by the open scholar movement, we shall look at how we might both: (1) apply a co-creation of learning approach to boyer’s model, (2) make the four-stage process more iterative and less discrete. in so doing we will propose a framework for the ‘co-creation of open scholarship’ as a way of taking forward the strengths of each of the models under review as we perceive them in 2011. we will do this by examining each ‘type of scholarship’ in boyer’s diat model through reviewing the descriptors in detail before adding an additional type that we will propose calling ‘co-creating’. we hope therefore in this paper to re-examine the notion of scholarship in the age of social media, update our view of learning theory in light of the developments of learning technology and deepen our views of the notion of co-creation in learning and research in the emerging ‘networked society’. background marta nibert (2001) in her analysis of boyer’s modelling of the professional role of the academic within american ‘college faculty’, in their terms specifically the ‘professoriate’, explains that for both her and boyer the concern is with defining ‘scholarly pursuits’ with a ‘balanced focus on all forms of scholarship necessary to meet the demands of the information age’. the beauty of boyer’s model is indeed this f. garnett and n. ecclesfield 6 clarity; its limitations are that it perfectly describes a situation that had validity over a decade ago, since when we have had thorough-going changes, often in response to the aforementioned ‘demands of the information age’. these are mostly around notions relating to the various concepts of ‘open’ ideas that were not available to boyer and nibert. however boyer’s use of a clear structure of ‘types’ of scholarship, and the use of descriptors to define the related actions of professionals, enables the kind of discussion and review we are undertaking here. we are calling this the diat structure; discovery; the traditional researcher role, integration; focusing on making connections across disciplines, application; focusing on using research findings and innovations to remedy societal problems, teaching; which boyer considers a central element of scholarship. this provides a useful framework from which to review scholarship in the more ‘open’ era of 2011. the diat model offers clear descriptors within each type of scholarship and also defines what constitutes a scholarly career whilst attempting to create some balance of recognition across the phases of scholarship described. see table 1. boyer’s model of scholarship. type of scholarship purpose measures of performance discovery build new knowledge through traditional research. publishing in peer-reviewed forums producing and/or performing creative work within established field creating infrastructure for future studies integration interpret the use of knowledge across disciplines. preparing a comprehensive literature review writing a textbook for use in multiple disciplines collaborating with colleagues to design and deliver a core course application aid society and professions in addressing problems. serving industry or government as an external consultant assuming leadership roles in professional organisations advising student leaders, thereby fostering their professional growth teaching study teaching models and practices to achieve optimal learning. advancing learning theory through classroom research developing and testing instructional materials mentoring graduate students designing and implementing a programme-level assessment system alt-c 2011 conference proceedings 7 table 1 for a full description of each of these types of scholarship. open scholarship in a network society terry anderson’s discussion of open scholarship was given as a keynote talk at the alt-c conference (2009) as part of a broader discussion of trends in learning and technology practices in the twenty-first century. he talks of moving from communities of practice to networks of practice, arguing that ‘we are all in the change business’, capturing the sense of flux that we are now trying to analyse here. caroline haythornthwaite in new forms of doctorate (2009) also discusses the impact of network effects on learning and scholarship. building on the taxonomy of the many (dron and anderson 2008) anderson looks at how learning is moving from the group to the collective, challenging boyer’s institution-centric approach. anderson argues for a move to being an open scholar arguing that quality scholarship ‘is peer and public reviewed, accessible, persistent, syndicated, commented and transparent’ picking up on how the network effects of learning are being impacted upon by a range of social media, both generic and also dedicated to scholarly practice. anderson additionally sees a key function of open scholarship as being ‘empowering learners as future teachers’. haythornthwaite amplifies this by defining ‘learning is a relation that connects people’, emphasising the relational and networked qualities of learning. anderson is focusing on the affordances of learning in the emerging world of open learning and examining its possibilities, whereas boyer is looking at how professional scholarship can be embedded institutionally, whilst broadening its value by re-asserting the value of teaching, for example. haythornthwaite (2009) looks more deeply and precisely at the effects that a range of networks are having educationally and sees the future as being characterised by ubiquitous learning in society. so we have three approaches, respectively focusing on institutions and professionalism, open learning and social media and ubiquitous learning and network effects. boyer is concerned to clarify the current role of professional scholarship within institutions whilst anderson is arguing from a scholarly perspective for a move to a deeper view of networks as collectives, occurring simultaneously within and outside institutions. haythornthwaite takes the rise of networks as a given and discusses learning in the ‘networked age’. indeed she prefers to see learning as an epiphenomenon of networks, with technology as a critical enabler of this or, as she puts it, ‘technology is a mediator for network relations including the vital relation of learning’ in a networked society. she sees learning as a networked relation consisting of learning relations, production, outcomes and spaces in an emerging participatory culture (pace jenkins 2006). indeed, haythornthwaite sees ‘contributory, open and participatory practices’ as signifying trends in learning which signify the ‘emergent work’ that ‘teachers, learners, educators and researchers’ should currently be engaging in. she draws her work together more coherently, as a summative social vision of future learning in a networked society, than anderson. however anderson is more discursive in his observations on open scholarship flagging a range of emergent practices which an open scholar might respond to, into which he adds personal learning environments and social learning, amongst many others. he quotes gideon f. garnett and n. ecclesfield 8 burton ‘the open scholar is someone who makes their intellectual projects and processes digitally visible and who invites and encourages ongoing criticism of their work and secondary uses of any or all parts of it � at any stage of its development’. for anderson, being an open scholar represents a new type of education work which maximises: social learning, media richness, participatory and connectivist pedagogies, ubiquity and persistence, open data collection and research processes and creating connections. however for anderson the sine qua non of this process is the production of open education resources (oer), which is perhaps both a little reductive and limiting on how we might usefully characterise being an open scholar. as ‘change agents for the future’ open scholars are both ‘empowering learners as future teachers’, and also inducting their charges into being open students, which we read as the inter-generational work of developing co-creative practices in learning. so anderson’s work is concerned to identify a range of cutting-edge scholarly practices without fully detailing how they might be embedded within the institution, but perhaps with more of an emphasis on gideon burton’s notion of their ‘ethical value’. haythornthwaite, however, is concerned to identify the emerging affordances of a range of networks and how that might affect ubiquitous learning within society. boyer however is interested in the professional role of the researcher within an institutionalised ‘professoriat’. our interest is in how we might synthesise these approaches, starting with the pah continuum as a model of co-creation that might prove useful. pah continuum the pah continuum is part of the open context model of learning (luckin et al. 2010), and like anderson and haythornthwaite, it is cognisant of the affordances of new, networked, web 2.0 and later technologies for learning and is consequently designed to enable their emergence within the practices of teaching and learning. we have argued in the open context model of learning that the pah continuum allows for a teaching and learning process to be developed which delivers good subject-based learning, the prime concern of educational policymakers, whilst enabling collaborative learning strategies and creative forms of assessment to be deployed. cochrane has demonstrated how this might be done using mobile technologies on the product design degree at unitec, nz (cochrane 2010a) by incorporating it into the design of technology use, and into supporting the increasing self-management of learners. so we believe the pah continuum helps in incorporating open learning affordances and networked effects into institutional contexts, given appropriate institutional-readiness (cochrane 2010b). developing boyer’s types of scholarship so let us look at how we might review boyer’s four types of scholarship in light of the approaches mentioned earlier, inspired variously by social media, digital tools, open learning and network effects. alt-c 2011 conference proceedings 9 discovery for boyer this is the phase of scholarship where new knowledge is built through traditional research. whilst this is a reasonable description of subject-based research where new knowledge about say plant cells can be discretely studied and identified, it is less relevant to learning/interdisciplinary research. what it clearly identifies is how new knowledge that will be used in subject-based teaching will be determined. so for the moment we will leave the descriptors relating to discovery as one type of scholarship unchanged as that is not our immediate concern. however, we will review them at the end of the article as part of considering how we might develop scholarship as an ongoing iterative process, after examining the whole of boyer’s diat model (see figure 1). integration the integration phase of scholarship in boyer moves beyond the professional orientation of the traditional researcher, as described in the discovery phase, to look at a narrowly defined notion of an ‘interpretation of knowledge’, including descriptors of practice and also with a focus on the production of learning materials. these are identified very practically, as literature reviews, textbook creation and course design, but somewhat traditionally. this ignores developments coming from the learning technology community over the past 15 years as described by, for example, conole and alevizou (2010) and the newer affordances of social media and its network effects (haythornthwaite 2009). in our view, literature reviews themselves have also been supplemented by data mining techniques using a range of social media tools (kelly 2011) a number of groups are also examining digital research practice in the age of social media and are producing fresh taxonomies in this field from the librarian’s perspective (british library 2011) more importantly the process of learning content production is being transformed rapidly, most notably by the oer and open courseware (ocw) movements, so much so that anderson in particular sees this as a key descriptor of being an open scholar. additionally we are seeing a number of syllabus-free approaches to learning, such as those proposed by sugata mitra (2009) and ian cunningham (2005), who separate learning content from integration enable the use knowledge across disciplines. preparing comprehensive literature reviews undertaking data mining analysis producing open education resources (oer) & content creation tools enabling generative network effects to occur figure 2. integration ‘type’ of boyer’s model of scholarship (modified). discovery build new knowledge through traditional research. publishing in peer-reviewed forums producing and/or performing creative work within established field creating infrastructure for future studies figure 1. discovery ‘type’ of boyer’s model of scholarship. f. garnett and n. ecclesfield 10 learning process, something cochrane has also developed using the pah continuum in course design (cochrane 2010a). a more complex dimension is that of enabling ‘network relations’ (haythornthwaite) to ‘emerge’, which might mean allowing new social groupings to emerge around new contexts, as suggested in the emergent learning model (garnett 2010), or by enabling ‘flocking’ (dron and anderson 2008). this suggests that we need an approach reflecting the divergent design of resources for appropriation and use in multiple contexts, rather than a convergent design process concerned with educational instruction within an institution. an integration phase of scholarship might be better served by a process of enabling knowledge to be opened out by networked effects and used in a more inter-disciplinary way in a range of contexts. so we suggest the set of descriptors as highlighted in figure 2 (changes highlighted in red). application in the ‘application’ type of scholarship boyer’s looks for the external validation of the scholar through the application of their knowledge in other communities. whilst this is certainly a valuable social process, we would rather the research professional started with developing their professional communities of practice through a collaborative mentoring process, as described by cochrane (2010a) in his description of educational communities of practice as course teams. whilst becoming sufficiently expert as professionals to be able to advise industry and government is clearly of value to the scholarly academic, and also to their host institution, a broader notion of public engagement should also be considered as we move to a more networked society, with more of a peer-to-peer focus (shirky 2008) and away from the more traditional notion of institution to institution linkages to promote the career of one individual. this is closer to what dron and anderson call the ‘taxonomy of the many’ (2007) shifting the range and character of institutional linkages whilst adding in concerns with public engagement of he institutions as they evolve (nccpe 2009). the collaborative affordances of social media mean that possible new, networked effects (new partnerships, institutional models, new models of learning and teaching, new modes of innovation) need to be positively designed for institutionally, enabling what garnett and ecclesfield (2008) call ‘adaptive institutions working across collaborative networks’. so boyer’s institutional descriptors in ‘application’ need application aid society and professions in addressing problems through serving community and public needs and purposes mentoring colleagues collaboratively serving industry or government as an external consultant assuming leadership roles in professional organizations empowering learners through co-creation to become future scholars working with community groups and on public engagement strategies using network effect to transform practice figure 3. application ‘type’ of boyer’s model of scholarship (modified). alt-c 2011 conference proceedings 11 to be broadened beyond direct linkages just with industry and government, both of which are going through their own transformations anyway in the post web 2.0 world (enterprise 2.0 and gov 2.0). they need to be made adaptive, to be reflective of a broader range of stakeholder interests (as developed in the recent jisc curriculum development and design initiatives 2010) and also to incorporate community responsibilities and ethical approaches, like those defined by michael gurstein concerning community informatics (2007)� (see figure 3). teaching we feel that the existing descriptors in the type ‘teaching’ mostly reveal how little boyer’s model reflects the range of transformations in scholarly practice effected by learning technologies and social media in recent years. this might best be exemplified in the five-year-old self-organised teachmeet programme (2006). again, whilst this has the merit of clarity in how it describes teaching responsibilities, the descriptors have been overtaken by events. for a start it is now not unusual to link together the processes of learning and teaching, and not just in vygostky-based constructivist approaches, so it is impossible to discuss this type without incorporating a greater degree of issues concerning learning and the role of the student, thus capturing the more participative approaches to education that have been emerging in recent years (anderson 2009; conole and alevizou 2010; cochrane 2010a). in order to reflect this we have added the descriptor ‘teaching as a reflective and dialogic practice promoting learning’, which also mirrors the work we have done on developing the pah continuum in the ‘craft of teaching’ (ecclesfield and garnett 2010). this more dialogic approach to teaching and learning as practice means that the notion that a teacher would merely ‘study’ a pre-defined approach to teaching in the classroom has been replaced by the potential for more andragogic, or negotiated, approaches to the process of learning. as mitra (2009) has shown, resources can now be introduced from a range of contexts via the internet so teachers need to be capable of ‘brokering’ learning (jennings 2010) as resources can be introduced on the fly within the learning process by learners themselves. as anderson indicates, learners now have personal learning networks extending beyond their immediate learning environment, so teaching needs to be capable of negotiating a range of learning contexts (see figure 4). teaching promote teaching as a reflective and dialogic practice promoting learning advancing learning theory through contextual research and practice collaborating in the design and delivery of courses & learning programmes brokering new learning processes developing open students designing and implementing responsive assessment systems figure 4. teaching ‘type’ of boyer’s model of scholarship (modified). f. garnett and n. ecclesfield 12 co-creating finally we look at the proposed additional ‘type of scholarship’, that of cocreating. a key phrase in o’reilly’s description of web 2.0 (2005) is that it is in ‘permanent beta’ which might be highlighted as a factor in why some teachers resist new approaches to teaching, but which has transformed the way we now view a range of processes. we would argue that we are now in a world in which knowledge creation itself is in permanent beta, what weinberger describes as everything is miscellaneous (2008), or the ‘post-digital disorder’. consequently the notion of a linear process of knowledge creation with knowledge discovery as the role of researcher and knowledge transmission as the role of the teacher, as separate scholarly practices, has been replaced by a more fluid and dynamic process which we are only just beginning to understand. the emerging knowledge networks are no longer something about which we receive information from researchers, they are processes in which practitioners participate, and we need to design scholarship practices that reflect this. the dynamic outline of open scholarship that anderson has presented (2009) provides an insight into the ethical issues in developing this approach, whilst also indicating the ongoing range of initiatives in development that support an open scholarship approach, which will need to be adapted to as their mature and prove their scholarly value. haythornthwaite’s more synthetic vision of scholarly practice anticipates some of the cultural shifts that might change that practice in more participatory, networked societies. we see these as differing ways of addressing the positive aspects of the emerging ‘permanent beta’ world of knowledge resources and knowledge creation, but what we are trying to do here is to evolve the traditional notions of scholarship in light of these emerging theories of teaching and learning, post web 2.0, and integrate the worlds of scholarship, along with teaching and learning to reflect the changing qualities of knowledge in a networked world where the ubiquity of social media is a quality that also challenges our traditional notions of academic institutions. we think the essence of this lies in the notion of co-creating learning and so we have added this as an additional type of scholarship, namely ‘co-creating’. we see the dimensions of this new view of scholarship emerging from the process of engaging in collaborative peer-to-peer networks, which would also practice interdisciplinary approaches, which might also be disruptive of existing subject disciplines. this disruptive quality is what we describe as heutagogy and we have indicated how that can be deployed in the learning and teaching process in the pah continuum (luckin et al. 2010). the pah continuum is a framework of teaching and learning that allows for epistemic cognition to emerge by co-creating learning, and it is co-creating participating in the perpetual beta of knowledge creation through the cocreation of learning engaging and collaborating in peer networks engaging in activity to develop, disrupt or join up established fields enabling epistemic cognition to be a part of evolving subject frameworks creating infrastructure for future learning and research figure 5. co-creating ‘type’ scholarship (proposed). alt-c 2011 conference proceedings 13 through epistemic cognition that new knowledge can be forged (avramides and luckin 2007), see figure 5. reviewing discovery the discussion of the co-creation of open scholarship presented here, where we have also presented a deeper notion of the role of the co-creation of learning together with the learner, or the open student as anderson puts it, also enables us to incorporate epistemic cognition into the learning process. however the inclusion of epistemic cognition also changes the description of discovery as a type of scholarship because epistemic cognition, within the co-creation process described in the pah continuum, is capable of stimulating research agendas within the learning process. in which case we might wish to redefine discovery as the ‘co-creation of research agendas’. so that discovery as a type of scholarship might be better described as in figure 6. conclusion so through examining boyer’s traditional approach to scholarship and by contrasting it to a range of emerging practices, admittedly driven by new web and social technologies and the early responses of anderson in his reflections on open scholarship, and haythornthwaite in her reflections on networked societies, we believe that we can outline a framework in which a co-creation model of scholarship can be developed and recognised professionally. what is presented here is merely a proposed outline, which we hope will be discussed, torn apart and further developed. for now here is our proposition of what a co-creation model of open scholarship (table 2) might look like in light of the above discussion. caveat we have not discussed many new pedagogies, such as connectivism in this article, nor new approaches to scholarship, such as e-science or technology-enhanced research. this is not because we think they have nothing useful to say: obviously they do. however, our starting point was to find a bridge between boyer’s model of scholarship and open scholarship whilst taking account of relevant work, concerning the co-creation of learning. this then lead to a broadening out of the debate and the references used such that this might appear as an overview of networked learning theories, which it is not. we view this as perhaps the start of process of discussion and would obviously welcome the views of for instance siemens (2005) and downes (2005) from both their connectivist and e-learning 2.0 perspectives, amongst many others. discovery aggregate new forms of knowledge through the cocreation of research agendas identifying useful domains for research publishing collaboratively in peeredited fora performing creative work in education dynamically supporting new infrastructures for learning figure 6. discovery ‘type’ scholarship (proposed). f. garnett and n. ecclesfield 14 references anderson, t. 2009. in dreams begin responsibilities. keynote alt-c, http://www.slideshare. net/terrya/terry-anderson-alt-c-final (accessed june 13, 2011). avramides, k. and r. luckin. 2007. towards the design of a representational tool to scaffold students’ epistemic understanding of psychology in higher education. proceedings of the table 2. co-creation model of open scholarship. type of scholarship purpose measures of performance discovery aggregate new forms of knowledge through the co-creation of research agendas. performing creative work in education identifying useful domains for research publishing collaboratively in peer-edited fora dynamically supporting new infrastructures for learning integration enable the use knowledge across disciplines. preparing comprehensive literature reviews return undertaking data mining analysis producing open education resources (oer) & content creation tools enabling generative network effects to occur application aid society and professions in addressing problems through serving community and public needs and purposes mentoring colleagues collaboratively serving industry or government as an external consultant assuming leadership roles in professional organisations empowering learners through co-creation to become future scholars working with community groups and on public engagement strategies using network effects to transform practice teaching promote teaching as a reflective and dialogic practice promoting learning advancing learning theory through contextual research and practice collaborating in the design and delivery of courses & learning programmes brokering new learning processes developing open students designing and implementing responsive assessment systems co-creating participating in the perpetual beta of knowledge creation through the co-creation of learning engaging and collaborating in peer networks engaging in activity to develop, disrupt or join up established fields enabling epistemic cognition to be a part of evolving subject frameworks creating infrastructure for future learning and research alt-c 2011 conference proceedings 15 http://www.slideshare.net/terrya/terry-anderson-alt-c-final http://www.slideshare.net/terrya/terry-anderson-alt-c-final workshop on aied applications for ill-defined domains at the 13th international conference on artificial intelligence in education, in los angeles, ca. boyer, e.l. 1997. scholarship reconsidered: priorities of the professoriate. san francisco: jossey-bas. cochrane, t.d. 2010a. exploring mobile learning success factors. alt-j 18, no. 2: 133�48. cochrane, t.d. 2010b. beyond the yellow brick road: mobile web 2.0 informing a new institutional e-learning strategy. research in learning technology 18, no. 3: 221�31. conole, g., and p. alevizou. 2010. a literature review of the use of web 2.0 tools in higher education. york: hea academy. cunningham, i. 2005. what is self-managed learning? http://www.selfmanagedlearning.org/ about-sml/what-is-self-managed-learning/ (accessed june 13, 2011). downes, s. 2005. e-learning 2.0, http://www.downes.ca/post/31741 (accessed june 13, 2011). dron, j. and t. anderson. 2007. collectives, networks and groups in social software for elearning. in proceedings of world conference on e-learning in corporate, government, healthcare, and higher education 2007, ed. g. richards, 2460�2467. chesapeake, va: aace. dron, j., and t. anderson. 2008. how the crowd can teach. in s. handbook of research on social software and developing community ontologies, ed. s. hatzipanagos and s. warburton. pa: igi global. ecclesfield, n. and f. garnett. 2010. the open context model and the craft of teaching, iped 2010, http://www.slideshare.net/fredgarnett/nefg-opencontextmodelcraftteachingoutlinev4 (accessed june 13, 2010). garnett, f. 2010. emergent learning model, open university talk, http://www.slideshare.net/ fredgarnett/fg-ouemergenttable (accessed june 13, 2011). garnett, f., and n. ecclesfield. 2008. developing an organisational architecture of participation. british journal of educational technology 39, no. 33: 468�74. gurstein, m. 2007. what is community informatics (and why does it matter)? milan: polimetrica. haythornthwaite, c. 2009. new forms of doctorate, leverhume trust public lectures, institute of education, london, http://newdoctorates.blogspot.com/2009/10/leverhulme-trust-public-lectures.html (accessed june 13, 2011). jenkins, h. 2006. convergence culture: where old and new media collide. new york: new york university press. jennings, d. 2010. how to create new contexts for your learning, http://alchemi.co.uk/archives/ ele/fred_garnett_on.html (accessed june 13, 2011). jisc, curriculum design & delivery project. 2010. listening, blog post, http://jisccdd.jiscinvolve.org/wp/tag/stakeholders/ (accessed june 13, 2011). kelly, b. 2011. evidence, impact, value: metrics for understanding personal and institutional use of the social web, http://www.ukoln.ac.uk/web-focus/events/workshops/digital-im pacts-2011/ (accessed june 13, 2011) luckin, r., w. clark, f. garnett, a. whitworth, j. akass, j. cook, p. day, n. ecclesfield, t. hamilton and j. robertson. 2010. learner generated contexts: a framework to support the effective use of technology to support learning. in web 2.0-based e-learning: applying social informatics for tertiary teaching, ed. m.j.w. lee and c. mcloughlin. pa: igi global, 70� 84. mitra, s. 2009. the hole in the wall, self-organising systems in education, keynote, alt-c, http://repository.alt.ac.uk/855/1/altc_2010_keynote_sugata_mitra_transcript. pdf (accessed june 13, 2011). nccpe. 2009. national co-ordinating centre for public engagement, http://www.publicengagement. ac.uk/ (accessed june 13, 2011). nibert, m. 2001. boyer’s model of scholarship, http://www.pcrest.com/pc/fgb/test/ 2_5_1.htm (accessed june 13, 2011). f. garnett and n. ecclesfield 16 http://www.selfmanagedlearning.org/about-sml/what-is-self-managed-learning/ http://www.selfmanagedlearning.org/about-sml/what-is-self-managed-learning/ http://www.downes.ca/post/31741 http://www.slideshare.net/fredgarnett/nefg-opencontextmodelcraftteachingoutlinev4 http://www.slideshare.net/fredgarnett/fg-ouemergenttable http://www.slideshare.net/fredgarnett/fg-ouemergenttable http://newdoctorates.blogspot.com/2009/10/leverhulme-trust-public-lectures.html http://newdoctorates.blogspot.com/2009/10/leverhulme-trust-public-lectures.html http://alchemi.co.uk/archives/ele/fred_garnett_on.html http://alchemi.co.uk/archives/ele/fred_garnett_on.html http://jisccdd.jiscinvolve.org/wp/tag/stakeholders/ http://jisccdd.jiscinvolve.org/wp/tag/stakeholders/ http://www.ukoln.ac.uk/web-focus/events/workshops/digital-impacts-2011/ http://www.ukoln.ac.uk/web-focus/events/workshops/digital-impacts-2011/ http://repository.alt.ac.uk/855/1/altc_2010_keynote_sugata_mitra_transcript.pdf http://repository.alt.ac.uk/855/1/altc_2010_keynote_sugata_mitra_transcript.pdf http://www.publicengagement.ac.uk/ http://www.publicengagement.ac.uk/ http://www.pcrest.com/pc/fgb/test/2_5_1.htm http://www.pcrest.com/pc/fgb/test/2_5_1.htm o’reilly. 2005. what is web 2.0? http://oreilly.com/web2/archive/what-is-web-20.html (accessed june 13, 2011). shirky, c, 2008. here comes everybody: london, allen lane siemens, g. 2005. connectivism: a learning theory for the digital age. international journal of instructional technology and distance learning 2, no. 10: 3�10. teachmeet wiki, teachmeet. 2006. http://teachmeet.pbworks.com/ (accessed june 13, 2011). weinberger, d. 2008. everything is miscellaneous: the power of the new digital disorder. new york: holt. alt-c 2011 conference proceedings 17 http://oreilly.com/web2/archive/what-is-web-20.html http://teachmeet.pbworks.com/ doi:10.3402/rlt.v24.30717 original research article reaching the unreached: de-mystifying the role of ict in the process of doctoral research kwong nui sim a * and sarah stein b a centre for academic development, victoria university of wellington, wellington new zealand; b distance learning, university of otago, dunedin, new zealand (received 13 december 2012; final version received 13 august 2016) information and communication technology (ict) has become a necessary element of academic practice in higher education today. under normal circumstances, phd students from all disciplines have to use ict in some form throughout the process of their research, including the preparation, fieldwork, analysis and writing phases of their studies. nevertheless, there has been little research to date that explores phd students’ first-hand experiences of using various ict to support their research practices. this paper brings together the findings and the key points from a review of significant parts of the existing literature associated with the role played by ict in the processes phd students use in doctoral research. the review is based on 27 papers appearing in international peer-reviewed journals published from 2005 to 2014. the study seeks to address the under-researched area in the current literature of how ict plays a role in the processes of doctoral research. while there are many contributions taking the ‘institutional’ or ‘teaching’ perspectives, papers focusing on ‘student’ perspective, or the viewpoint of engaging ict in daily study routine, are relatively fewer. as far as research methodology is concerned, this review found that many of the papers that were examined were mostly based on perception data such as surveys or interviews, while actual practice data were rarely present. with their ready access to technologies, phd students are well positioned to take advantage of a range of technologies in order to carry out their research efficiently (in terms of means to an end) and effectively (in terms of reaching goals within a task). this review reveals that in the literature, this important area is under-represented. keywords: ict; graduate; postgraduate; research; information consumption; literacy; role of ict; graduate profile; actual practice; perception 1. introduction the use of information and communication technology (ict) has grown enormously in the last 10 years with computers and smart devices becoming indispensable to daily life. ict is seen as vital for those wishing to engage in higher education (e.g. aspden and thorpe 2009), and this includes the activities of graduate students. a considerable portion of the current literature on computer use in academic contexts suggests that student’s use of technology will result in students being efficient in their learning (smith, salaway, and caruso 2009), for example, to *corresponding author. email: kwongnui.sim@vuw.ac.nz responsible editor: peter reed, university of liverpool online, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 k.n. sim. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 facilitate online information searching. in addition, a number of studies claim that computer technology now plays a significant role in supporting undergraduate study (aspden and thorpe 2009; dahlstrom et al. 2011; guidry and brckalorenz 2010; smith and caruso 2010). it seems appropriate that ict should also help phd students to complete their research in doing background research for the thesis, in conducting the various research activities, in writing the thesis; in other words, in all phases of research and in the best possible ways (jackson 2005; onilude and apampa 2010). while this is a claim that might be difficult to refute, the importance of ict in a phd student’s research process may have been overlooked in the existing research literature. for example, a search of recent publications reveals that most empirical research on doctoral education has been focused on the notion of ‘doctorateness’ (e.g. wellington 2012), the candidature discourses (e.g. strengers 2014), the viva (e.g. chen 2014), supervision (e.g. mcalpine 2013) and the thesis examination (e.g. clarke 2013). in terms of phd students’ ict use (e.g. blignaut and els 2010), documented studies have been focused on graduate students’ computer literacy, communication (e.g. lawlor and donnelly 2010), entertainment use (e.g. mccarthy 2012) and the use of learning management systems (e.g. sultan 2010). of these studies, while focusing on the doctoral research process or the ict use in general, there are few that research these two aspects at the same time. the lack of studies considering ict use in doctoral research may suggest that institutions, that is, lecturers, supervisors and/or students, hold certain assumptions and expectations regarding phd students’ ict use. phd students’ ict use may have been taken for granted by academia or overlooked in general. the role of ict in supporting phd students’ research processes is thus unclear, especially the degree to which ict are being embedded into practice at different phases in the research process. thus, a study examining phd students’ ict use could be of benefit for different communities, including institutions, disciplines, lecturers, supervisors and students within the higher education context. this timeframed literature review, from 2005 to 2014, provides the foundational and theoretical underpinnings to a study that aims to explore the role that ict plays in supporting phd students’ research. this paper will begin by describing ict and its role among the students in the current generation as well as the notion of doctoral research. a description of the method used to gather relevant literature is then outlined. following this, findings and discussion from the reviewed literature on the basis of four themes are presented. lastly, the conclusion identifies future direction in this research area. 2. background/context 2.1. ict and current generation of students ict refers to information technology in the context of telecommunications, computers, software and the data systems that support, store and transmit unified communication technologies for users to access and manipulate information (murray 2011). for the purpose of this paper, ict is categorised into hardware, software and networks as described below: � hardware: this includes various types of computers such as smart devices, desktops, laptops and tablets. k.n. sim 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 � software: this is any set of machine-readable instruction that directs a computer’s processor to perform specific operations, such as microsoft office and endnote. � networks: these are systems of telecommunication that allow computers to exchange data, such as wireless and ethernet connections. students are well ‘e-equipped’ with increasing numbers of icts present in their daily lives. for example, a notebook and a pen may have formed the tools for study of prior generations but today’s students come to class with ‘e-equipment’ such as smart phones, laptops and mp3 players (economist intelligence unit 2008; its 2012). the existence of the above-mentioned hardware, software and networks offers students today the access to enormous information and knowledge beyond the classroom setting or the teachers’ control. such access enables students to capture, share, collaborate and publish in previously unavailable ways. it is undeniable that the internet provides easy access to vast quantities of information (williamson et al. 2007) and, as a result, it is claimed that students today ‘. . . think and process information fundamentally differently from their predecessors’ (prensky 2001, p. 1). this is evidenced by studies which describe how the present generation of students multi-task (even though the idea of multi-tasking has been critiqued in the existing literature) with digital technologies. students take notes on a laptop, send text messages on a smart phone, while they may simultaneously have social networking software such as facebook operating in the background on either their laptop or smart phone (economist intelligence unit 2008, p. 11; hembrooke and gay 2003; wood et al. 2011). according to lieutenant colonel greg conti, the director of west point’s information technology operations centre, ‘it is impossible to sit someone in front of the world wide web and expect them [students] not to use it’ in today’s world because ‘today’s students are used to getting what they need instantly’ (economist intelligence unit 2008, p. 12). it is believed, therefore, that, as time progresses and new technologies are introduced and embedded life, ict will become even more interwoven into academic life than it already has. as a consequence, ict will continue to have a significant impact on higher education (economist intelligence unit 2008; tltteam 2011). the new challenge for higher education institutions is how to ‘e-equip’ students with the skills and knowledge required to use ict effectively in the university as well as in preparing them for the workplace (economist intelligence unit 2008; güçlu 2010). this challenge can be especially significant for the postgraduate students. 2.2. what is doctoral research? as noted earlier, phd students’ progress through a number of phases in the process of doing their doctoral research. doctoral research is described as the process of developing the independent scholar, or a scholar who independently produces original research or new knowledge (council of graduate schools 2005). according to gardner (2008), overall, there are three stages in a phd research journey: (1) admission, including applying to prospective programmes and institutions and meeting and talking with faculty members, staff and graduate students in these prospective programmes; (2) integration, including social integration with peers and faculty; the eventual choice of an advisor/supervisor/committee; and (3) candidacy, which is the research in learning technology citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 time during which the student focuses primarily on the research. focusing on the candidacy stage as described by gardner (2008), this review assumes four phases: (1) preparation � when a phd student creates a research project proposal, reads relevant literature and constructs a research framework; (2) fieldwork � when the phd student collects data as planned according to his or her research framework; (3) analysis � when the phd student engages with the collected data, in alignment with the designed research framework and the existing literature; and (4) writing � when the phd student writes the dissertation or thesis as a fulfilment of the degree requirements. 3. starting point: methods a literature search was conducted in 2014 using the search engine of the university of otago library site (http://www.otago.ac.nz/library/) and the databases, google scholar, proquest central, science direct and ebsc host. search terms were generated based on the criteria listed in table 1. similar terms were used in the search engine and they were adapted according to the nomenclature of the individual databases. the search was conducted using the keywords and terms from table 1 in both the abstract and main body of papers. in august 2013, rich site summary/real simple syndication (rss) feeds of all the databases set up on netvibes (http://www.netvibes.com/en) using the same search terms. netvibes is a social media monitoring, analytics and alerts dashboard. this method facilitated the gathering and analysis of all the selected major journals in this research topic. in addition, further articles were identified from the reference list in the retrieved articles. the final collection of literature included original, peer-reviewed research articles with high relevance that met the aim and/or inclusion criteria for this planned investigation, written in english and published from year 2005 to 2014 in different parts of the world. the year 2005 was chosen, rather than an earlier date, because of the rapid changes in ict development since then. among the articles listed, the aspects of investigations and the methods used were examined to determine whether they were appropriate to be applied as reference data for the current review. the aspects of investigation included the focus on students’ computer literacy in communication, entertainment use, learning management systems use, library use and knowledge consumption process on internet. the studies identified highlighted a range of aspects including skills in ict use, the variety of ways different ict have been used for academic practices and students’ self-confidence in the use of ict. table 1. search terms based on question words as pointers. pointers search terms who graduates/postgraduates/phd students what ict/technologies where university/higher education why thesis/dissertation/research how use/utilise/adopt/integrate/engage when year 2005 to 2014 k.n. sim 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 http://www.otago.ac.nz/library/ http://www.netvibes.com/en http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 as for the methods used, the search emphasised the collection of students’ perceptions and/or behaviours related to ict use. this process is detailed below. 3.1. the aim this review aimed to summarise the relevant studies on the role of ict among graduate students since year 2005. specifically, it reviewed studies that have touched upon the ways graduate students engage ict in their research practice. 3.2. the inclusion criteria in this review, the literature relating to graduate students’ engagement and integration of ict is examined. more precisely, the viewpoints of different roles of ict held by graduate students is taken. thus, this includes literature dealing with the role of ict for both research-related and non-research-related practices. only those papers whose main focus was on the topic of this study were selected and therefore the articles that mentioned the topic only in their introductory remarks, or as collateral research themes, were excluded from the analysis. within this sub-set, a smaller cluster of articles that specifically focused on this research topic were further identified and considered for the review. further analysis resulted in the generation of themes, which consisted of 27 papers that were published from 2005 to 2014. these papers were examined in-depth. table 2 presents the details of these 27 papers. in summary, papers on this topic were published in various journals, presenting a wider research on computer use among graduate/postgraduate students in different countries (e.g. united kingdom, taiwan, india, turkey and greek). also included are articles that addressed the role of computers in research practice in general. 3.3. the review process the review process entailed a two-pronged approach. first, the papers were categorised on the basis of their content and the second categorisation was according to the research methods used. the included papers were organised according to the perspective on content that they took: (1) an institutional focus (n �10), which concentrated on ict facilities in institutions; (2) a teaching focus (n �3), which emphasised the ways ict was integrated into teaching activities (through advisors/supervisors/committees); and (3) a student focus (n �14), which reflected students’ perspectives and behaviours in relation to ict in learning. despite the three different perspectives, the theme of the reported studies appeared to cluster around four key areas (see table 1): (1) computer literacy; (2) ict integration in a research practice; (3) research productivity alongside information consumption; and (4) student graduate profile in relation to ict competency. papers were also examined on the basis of the research method(s) used. the review revealed that many of the examined articles were empirical studies (�89%), which used questionnaires, surveys, case studies or interviews as principle datagathering techniques. there were very few contributions based on students’ actual research practice with ict. such ‘actual research practice’ would include data gathered from, for example, observable behaviours while engaging with ict. only three papers (11%) out of the 27 articles included actual practice data. one paper used observation (ismail 2011), one paper used an online testing system research in learning technology citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 (wallace and clariana 2005) and another one used qualitative data capture through written comments (lawlor and donnelly 2010). 3.4. summary of the review contributions written from the institutional (viz. the role of university or library) and teaching (viz. the role of supervisors) perspectives were generally written in the mid2000s. papers focusing on student’s perspectives (i.e. their e-literacy) were relatively more recent and their numbers have progressively increased. additionally, in recent years, researchers have started to examine students’ perceptions of the role of ict and have included strategies to gather data on student behaviour and attitudes while using computers in research work. these two foci suggest the need for the collection of actual practice data alongside perception data. however, in this review no paper showing any specific relationship between the roles of ict in the process of doctoral research and how phd students engage ict in their research practice, as evidenced through practice data, was found. 4. findings and discussion: what is missing in the existing literature? 4.1. content based most studies in the extant literature from 2005 to 2014 have focused on students’ computer literacy (e.g. blignaut and els 2010), communication (e.g. lawlor and table 2. the selected 27 papers associated with their themes. authors (organised by year of publication) theme of the paper griffiths and brophy (2005) web searching behaviours wallace and clariana (2005) computer literacy skills aderibigbe and aramide (2006) use of internet george et al. (2006) information-seeking behaviours divaris, polychronopoulou, and mattheos (2007) computer literacy and attitudes williamson et al. (2007) information seeking and use darus (2008) word processing gardner (2008) independence of doing research li (2008) computer literacy alkhanak and mokhtar (2009) web services blignaut and els (2010) computer and information literacy skills dange (2010) computer and internet usage lawlor and donnelly (2010) podcast use meerah (2010) research skills ongoz (2010) information searching onilude and apampa (2010) use of icts winter et al. (2010) e-learning experiences gibbs, steel, and kuiper (2011) computing skills ismail and kareem (2011) information needs magliaro (2012) information literacy needs varshney (2012) information use wu and chen (2012) electronic resources nabeel, shahrir, and leng (2013) attitudes towards computer use odaci (2013) computer self-efficacy ahmad et al. (2013) ict competency lai and hong (2013) e-learning characteristics bowman et al. (2014) laptop ergonomic education k.n. sim 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 donnelly 2010), entertainment use (e.g. mccarthy 2012), use of learning management systems (e.g. sultan 2010), library use (e.g. sutton and jacoby 2008) and knowledge consumption (e.g. griffiths and brophy 2005). these studies have highlighted a range of aspects including skills in ict use, the variety of ways different ict have been used for academic practices and students’ self-confidence in the use of ict. where phd student’s use of ict to support their research processes is concerned, the place of ict with a focus on ways students use ict in their research practices and research activities is often discussed in a limited way in the literature. for example, graduate students are described as ‘binge’ users of e-journals and as having a preference for electronic resources during their thesis writing process (dange 2010; george et al. 2006; liew, foo, and chennupati 2000; rowlands et al. 2007; tenopir 2003). it is also acknowledged that all phd students will use ict for their doctoral research. depending on the academic discipline, some will use software applications such as spss and nvivo for data analysis, while some will use software designed specifically for work in their field of study. most, if not all, will use widespread applications that facilitate searching references and typing and archiving documents. what is important is that the nature of phd students’ use of ict for the integrated tasks involved in their study is unclear in the current literature. in other words, current studies do not offer a clear picture of how phd students integrate computer technologies into their daily research practices; only reporting what students use computer technologies for. in summary, a number of themes are under-represented or missing. the positive place of ict in graduate education has been extolled in the literature, with the focus being on information searching through web browsers. thus, the review undertaken here has identified four shortcomings, which are described below. 4.1.1. computer literacy among phd students first, there are no up-to-date comprehensive studies of the level of computer literacy among graduate or postgraduate students, particularly among phd students in their research practices. there is a common belief that students entering graduate study, especially phd study, have appropriate computing skills for study purposes and there is no longer a felt need for computer training programmes in tertiary education (dange 2010). the concern is that focus on phd students’ ability to integrate ict into their research practice is overshadowed or taken-for-granted as a consequence of their perceived or assumed readiness from undergraduate and/or master’s study experience. for example, there are studies showing that many postgraduates cannot cope with the demands of higher education, especially in terms of ict use (e.g. nair and pillay 2004). these findings align with research indicating that poor preparation for the demands of higher education includes students’ meager computer and information literacy skills, ‘technophobia’ and low computer literacy or competency (castles 2004). the results in some studies suggested that postgraduate students were not competent at using office-type applications such as microsoft excel, powerpoint and access (dange 2010). for example, there was one report that stated that students had ‘high levels of ownership of application types’ but these applications were not ‘frequently used’ (shaw 2000). do these findings and results indicate the limitations in graduate students’ ability to make the best use of ict in their research practice? it appears that the degrees to research in learning technology citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 which graduate students could be regarded as competent computer users in terms of using basic academic software in order to complete their dissertations are still unclear. the difference between being part of the ‘digital generation’ or digital migration (dobbins 2005; kennedy et al. 2009; prensky 2001) and being competent computer users could be deeper than much research to date assumes. ict is expected to be at the heart of all aspects of a student’s life, especially at graduate level. phd students are viewed as developing researchers, and therefore knowledge of the computer literacy of phd students and its association with ict integration into research processes is essential, if the best support and opportunity are to be provided to ensure their success as scholars. 4.1.2. the role of ict in the process of doing research a second shortcoming highlighted by this review is that limited attention has been paid in the literature to how ict integration happens in the process of research, especially at doctoral level. shaw’s (2000) study involving over 300 graduate students in a northeastern us university examined students’ academic computing attitudes, uses, needs and preferences. student perceptions were reflected in comments such as: ‘using a computer makes me more organised in my graduate work’; ‘using a computer makes me more motivated to do my graduate work’; ‘sharpening my computer skills in graduate school is essential in my professional work’; and (the negatively worded) ‘i prefer to do my academic work without much use of computers’ (shaw 2000). the students in the study claimed that the computer skills they used in their graduate and/or professional work included: ‘writing’ (91%) and ‘research’ (83%); nearly that proportion (79%) identified ‘doing regular course assignments’; and roughly two thirds indicated ‘corresponding with professors’ (68%) as well as ‘corresponding with classmates’ (61%) (shaw 2000). in short, the purposes identified by the students focused on typical activities that are part of the research preparation phase (searching information about the topic) and the write-up phase (writing and doing regular course assignments), as well as for communication tasks (corresponding with teachers and classmates). students in these studies did not seem to recognise the use of computers in the fieldwork phase of their research (e.g. survey tools, recording and note taking applications) or in the analysis phase (i.e. the use of data analysis software). in summary, one conclusion that could be drawn is that graduate or postgraduate students could be considered as active computer users, especially during their write-up phase, but could be considered less active users of ict in the process of doing research. given the growing access to internet-based digital devices, it seems that ict integration throughout the entire doctoral research process has not been extensively investigated. the majority of the reviewed studies discussing graduate or postgraduate students’ ict integration appeared to focus on the certain aspects of research practice only, that is, using e-resources and writing. 4.1.3. productivity during the process of doing a research a third shortcoming revealed in this review is that those studies on graduate students’ productivity, or how they make use of, or consume, information online, are rare. k.n. sim 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 this could be seen as particularly concerning, considering phd students are expected to be producers of ‘new’ knowledge. most reviewed studies on graduate students’ ict integration seem to focus on their literacy (blignaut and els 2010), communication (lawlor and donnelly 2010), recreation (mccarthy 2012) and their utilisation of learning management systems (sultan 2010). as for the role of ict in graduate students’ research practices, the existing studies are limited to their library use (sutton and jacoby 2008) or knowledge consumption (griffiths and brophy 2005). this might suggest that the role of ict in the process of doing research is limited to web browsing or data collection. it is also interesting to note that this view might match students’ view of ict too. one study (economist intelligence unit 2008) reported that about 75% of participants who were asked about ict use in their study said ‘the greatest potential benefit of technology is something far more straightforward*namely, the expanded access to educational and reference resources that it provides’ (p. 6). according to a substantial portion of literature, graduate students are ‘binge’ or excessive, users of e-journals and prefer to use electronic resources (aderibigbe and aramide 2006; dange 2010; george et al. 2006; liew, foo, and chennupati 2000; rowlands et al. 2007; tenopir 2003) during their dissertation writing journey. students seem to be treating computers as simple devices for accessing web-based information (dahlstrom et al. 2011), rather than as offering them production capabilities as well. there seems to be a need to expand research literature to inquire into the ways in which graduate students use, experience and integrate ict in their research practice, beyond web/journal/information searching. 4.1.4. graduate profile in relation to ict integration in research practice fourth and finally, there is little empirical evidence about the profiles of graduate students, specifically phd students, in relation to their engagement with ict in their daily life as researchers. according to one-third of the employers who responded to a questionnaire in a study about ict training for new graduates, ‘some on-the-job training will be necessary to acclimatise new employees’ and this is explained as ‘this generation is not content with passive involvement’ (economist intelligence unit 2008, p. 11). it appears that a student, especially a graduate or postgraduate student, is not aware of the intended graduate outcomes of their institutions. one study argues that graduate students ‘are benefiting from the pedagogical advantages of information technology and preparing for the professional world of work’, with ‘appropriately anticipating that skills in the use and management of information technology will be essential for advancement along their chosen career paths’ (shaw 2000, p. 34). some of the current literature on computer integration in academia stated that postgraduate students now require more computing skills, such as the basic concept of computer knowledge (e.g. computer applications) in order to advance their research practice (case, mackinnon, and dyer 2004; wallace and clariana 2005). these skills have been identified and they include: some basic ict skills, such as file management, word processing, spreadsheet manipulation and graphical presentation (blignaut and els 2010); familiarity with basic components, terms and conditions, such as basic choice and use of hardware and software; overcoming of ‘technophobia’; use of computer peripherals; basic formatting and editing functions; data entry; error management; use of operating research in learning technology citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 systems; backing up; and basic network interaction (blignaut and els 2010; meerah 2010; wallace and clariana 2005). studies on how the students apply these computer skills in their doctoral research are misty and how these skills support research practice is limited. further, it is also unclear how ict plays a role in assisting graduate or postgraduate research students to develop their graduate profiles or how students’ ict integration is shown in their research practice to support their graduate profiles. the beliefs and the findings in the existing studies thus far may have signalled a further investigation into students’ ict engagement. this is especially important when ict integration in the process of doing doctoral research is unavoidable, and ict proficiency is significantly related to a student’s graduate profile in today’s world. 4.2. data-gathering methods used in this research domain most studies on student’s use of ict in higher education rely on perception data, often gathered via surveys, interviews and questionnaires. perception data refer to students reporting on what they believe they do or what they have done through postevent recollection. in an above-mentioned example, graduate students self-reported as ‘binge’ users of e-journals or as having a preference for using electronic resources during their graduate study (aderibigbe and aramide 2006). however, the results in some studies suggested that graduate students are not competent to the same extent in using generic spreadsheet, presentation, word processing and database applications (e.g. dange 2010). as mentioned earlier, another report suggested that students had high levels of ownership of application types that they did not frequently use (shaw 2000). one of the reasons these studies raise different scenarios of graduate students’ use of ict could be that they rely on perception data, often gathered via surveys and questionnaires. thus, none of these studies can lay a claim to knowing what students actually do in practice. this situation prompted conole et al. (2008) to state that, ‘more in-depth research is needed to understand the nuances of how students are using technologies to support their learning’ (p. 512). in short, from the point of view of the research methods, the under-representation of certain research methodologies, such as actual practice data, suggests opportunities for future research. as suggested by dange (2010), there should be more ongoing studies to monitor the situation as technological and educational environments continue to change. the ‘first necessary step of this process is an accurate and realistic assessment of the actual computer skills of the student’ (divaris, polychronopoulou, and mattheos 2007). the studies should be based on students’ observable behaviours in relation to their engagement with ict. one reason is that such findings, while relevant to explorations of postgraduate students’ perceptions of ict use, also offer a convincing picture of student practice as experienced in their day-to-day practice. in sim and butson’s (2013) study on the use of personal computers by third-year undergraduate students, it was argued that ‘the difference between the students’ beliefs about their personal computer use and their computer use highlights that self-report data reliant on post-event recollections should not be relied on to represent actual practice’ (p. 338). furthermore, there is no, or little, practical significant correlation between performance and students’ expectations of additional technological support (blignaut and els 2010; sim and butson 2013; wallace and clariana 2005). in other words, data gathered through self-rating data collection techniques are not a reliable measure to assess ict use by graduate students. this assertion aligns with the k.n. sim 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 findings of van vliet, kletke, and chakraborty (1994) that the self-ratings are not accurate indicators of computer skills, as students often rate their skills lower or higher than their practical skills reflect (blignaut and els 2010; butson and sim 2013; sim and butson 2013, 2014). it is worth considering that actual practice datagathering methods be employed in order to reveal students’ daily technological academic practices, rather than through the more traditional approaches of questionnaires, surveys and interviews only. green et al. (2006, p. 1) suggested that students’ participation might be secured by allowing them to play a ‘researcher-like’ role in the study and experience the invested outcome of the study. conole et al. (2008) also agreed that research should focus more on the student voice, their ongoing use, experience and perception of technologies to elicit and explore their e-learning strategies in different contexts. although students might be more familiar with research methods such as survey and focus groups (dahlstrom 2011), there might be unexpected outcomes by engaging research participants as peers and colleagues when collecting the data. applying such methods recognises the participants’ power and uniqueness as sources of evidence, as well as engaging them in the inquiry. it enables the introduction of first-person observational perspectives to the collection and analysis of the data. 4.3. summary of findings and discussion generally, what has emerged in the reviewed literature is the lack of in-depth studies on ict use by phd students with regard to their practices. the research into phd students’ computer literacy is also limited. similarly, the existing empirical research has yet to consider phd students’ productivity with ict integration in their research practice, especially in relation to graduate capabilities. even though the literature indicates a rising interest in different perspectives on the role that ict plays in higher education, such growing interests from the research community have not yet expressed the awareness of students’ practices in this area. the increased inclusion of students’ perspectives in this research area in recent years seems to be mainly focused on information-searching activities or measurement of computer literacy. evidence of students’ actual practice alongside their perceptions is still limited. 5. conclusion: identified directions for future research this paper sought to identify what is lacking in the literature about the role that ict plays in phd students’ lives. in this paper, 27 research contributions published after year 2005 were examined. the papers were analysed using a two-pronged approach: the paper contents addressed and research methods adopted. the contribution of this review is twofold. first, this review provides a structured review and guide to earlier research on the role of ict in the process of doctoral research. second, this review identifies research issues for future investigation, mainly in the area of broadening the research methods being employed. in summary, research reporting on students’ ict use has increased in recent years but the focus is mainly limited to computer literacy measurement or information-searching activities. such growing interests from the research community have come with increased, though still limited awareness of students’ ict practices in this area. a focus on the importance of this for the development of students’ graduate profile in relation to ict is also still limited. research in learning technology citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 the difference found between perception and practice data in the studies thus far signals the need for a substantial shift in the way to understand and gather data in this emerging field. in terms of phd students’ ict use (e.g. blignaut and els 2010), as mentioned earlier, documented studies have been focused on graduate students’ computer literacy, communication (e.g. lawlor and donnelly 2010), entertainment use (e.g. mccarthy 2012) and the use of learning management systems (e.g. sultan 2010). these studies, while focusing on the doctoral research process or the ict use in general, there are few that research these two aspects at the same time. the lack of studies considering ict use in doctoral research may suggest that institutions, that is, lecturers, supervisors and/or students, hold certain assumptions and expectations regarding phd students’ ict use. phd students’ ict use may have been taken-for-granted by academia or overlooked in general. the role of ict in supporting phd students’ research processes is thus unclear, especially the degree to which ict are being embedded into practice at different phases in their research process. given the arguments concerning the role of ict in a process of doing research, what is the argument about the impact of ict on playing a role in a doctoral research? the ‘simplified’ assumptions about the ongoing relationship between ict and graduate or postgraduate students should be avoided (larsson 2002), and more emphasis should be placed on how technological support should be designed in order to assist phd students in their doctoral research in the best possible ways (jackson 2005; onilude and apampa 2010). future studies should start looking at the way that graduate students interact with ict and the way that ict supports their day-to-day practices, and then let this understanding inform the development of appropriate technological support that is firmly based on close investigations of how doctoral research is performed on a daily basis. again, the matter of interest is to study, understand and describe what phd students actually do, not to prescribe what they should do or what they think they are doing. references aderibigbe, n. a. & aramide, k. a. (2006) ‘institutional factors and perceived usefulness as predictors of internet use by postgraduate students at the university of ibadan, nigeria’, pnla quarterly, vol. 74, no. 4, p. 24. ahmad, m., karim, a. a., din, r. & albakri, i. s. m. a. (2013) ‘assessing ict competencies among postgraduate students based on the 21st century ict competency model’, asian social science, vol. 9, no. 16, pp. 32�39. alkhanak, e.n & mokhtar, s. (2009) ‘using services oriented architecture to improve efficient web-services for postgraduate students’, international journal of social and human sciences, vol. 3, pp. 1�4. aspden, e. j. & thorpe, l. (2009) ‘‘‘where do you learn?’’: tweeting to inform learning space development’, [online] available at: http://www.educause.edu/ero/article/where-do-youlearn-tweeting-inform-learning-space-development blignaut, a. s. & els, c. j. (2010) ‘comperacy assessment of postgraduate students’ readiness for higher education’, the internet and higher education, vol. 13, no. 3, pp. 101�107. doi: http://dx.doi.org/10.1016/j.iheduc.2010.02.007 bowman, p., et al., (2014) ‘benefits of laptop computer ergonomics education to graduate students’, open journal of therapy and rehabilitation, vol. 2, pp. 25�32. doi: http://dx.doi. org/10.4236/ojtr.2014.21006 butson, r. & sim, k. n. (2013) ‘the role of personal computers in undergraduate education’, international journal of digital literacy and digital competence, vol. 4, no. 3, pp. 1�9. doi: http://dx.doi.org/10.4018/ijdldc.2013070101 k.n. sim 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 http://www.educause.edu/ero/article/where-do-you-learn-tweeting-inform-learning-space-development http://www.educause.edu/ero/article/where-do-you-learn-tweeting-inform-learning-space-development http://dx.doi.org/10.1016/j.iheduc.2010.02.007 http://dx.doi.org/10.4236/ojtr.2014.21006 http://dx.doi.org/10.4236/ojtr.2014.21006 http://dx.doi.org/10.4018/ijdldc.2013070101 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 case, t., mackinnon, r. & dyer, j. (2004) ‘computer literacy and the introductory student: an analysis of perceived and actual knowledge of computers and computer applications’, sais 2004 proceedings, 27-28 february., 2004 at savannah, georgia castles, j. (2004) ‘persistence and the adult learner: factors affecting persistence in open university students’, active learning in higher education, vol. 5, no. 2, pp. 166�179. chen, s. (2014) ‘balancing knowing and not-knowing: an exploration of doctoral candidates’ performance of researcher selves in the dissertation defence’, assessment & evaluation in higher education, vol. 39, no. 3, pp. 364�379. clarke, g. (2013) ‘developments in doctoral assessment in the uk’, critical issues in higher education, eds m. kompf & p. m. denicole, sense publishers, rotterdam, the netherlands, pp. 23�36. conole, g., et al., (2008) ‘disruptive technologies’, ‘‘pedagogical innovation’’: what’s new? findings from an in-depth study of students’ use and perception of technology, computers & education, vol. 50, pp. 511�524. doi: http://dx.doi.org/10.1016/j.compedu.2007.09.009 council of graduate schools. (2005) the doctor of philosophy degree: a policy statement, council of graduate schools (cgs), washington, dc. dahlstrom, e. (2011) connecting student data from ecar and cds, educause center for applied research, educause, pp. 1�7. dahlstrom, e., et al., (2011) ecar national study of students and information technology in higher education, educause center for applied research: educause, boulder, co, pp. 1�54. dange, j. k. (2010) ‘post graduate students’ computing confidence, computer and internet usage at kuvempu university � an indian study’, international journal of instruction, vol. 3, no. 2, pp. 39�62. darus, s. (2008) ‘arab postgraduate students’ attitudes toward word processing: a case study’, european journal of social sciences, vol. 6, no. 3, pp. 306�314. divaris, k., polychronopoulou, a. & mattheos, n. (2007) ‘an investigation of computer literacy and attitudes amongst greek post-graduate dental students’, dental education, vol. 11, pp. 144�147. dobbins, k. w. (2005) getting ready for the net generation learner, educause center for applied research, educause, p. 9. economist intelligence unit. (2008) the future of higher education: how technology will shape learning, pp. 1�32. gardner, s. k. (2008) ‘‘‘what’s too much and what’s too little?’’: the process of becoming an independent researcher in doctoral education’, the journal of higher education, vol. 79, no. 3, pp. 326�350. doi: http://dx.doi.org/10.1353/jhe.0.0007 george, c. a., et al., (2006) ‘scholarly use of information: graduate students’ information seeking behaviour’, university libraries research, vol. 21, pp. 1�27. gibbs, s. s. g, & kuiper, a. (2011) ‘do new business graduates have the computing skills expected by employers? glob health action 2015; 8: 29359 paper presented at the 2nd international conference on society and information technologies: icsit, 27-30 march 2011, orlando, florida, usa. green, a. s., et al., (2006) ‘paper or plastic? data equivalence in paper and electronic diaries’, psychological methods, [online] vol. 11, no. 1, pp. 87�105. doi: http://dx.doi.org/10.1037/108 2-989x.11.1.87, available at: http://docsfiles.com/pdf_paper_or_plastic_data_equivalence_ in_paper_and_electronic_diaries.html griffiths, j. r. & brophy, p. (2005) ‘student searching behavior and the web: use of academic resources and google’, library trends, vol. 53, no. 4, pp. 539�554. güçlu, m. (2010) ‘university students’ computer skills: a comparative analysis’, tojet: the turkish online journal of educational technology, vol. 9, no. 2, pp. 264�269. guidry, k. & brckalorenz, a. (2010) ‘a comparison of student and faculty academic technology use across disciplines’, [online] available at: http://www.educause.edu/ero/ article/comparison-student-and-faculty-academic-technology-use-across-disciplines hembrooke, h. & gay, g. (2003) ‘the laptop and the lecture: the effects of multitasking in learning environments’, journal of computing in higher education, vol. 15, no. 1, pp. 46�64. ismail, m.a & kareem, s.a. (2001) ‘identifying how novice researchers search, locate, choose and use web resources at the early stage of research’, malaysian journal of library & information science, vol. 16, no. 3, pp. 67�85. research in learning technology citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 13 (page number not for citation purpose) http://dx.doi.org/10.1016/j.compedu.2007.09.009 http://dx.doi.org/10.1353/jhe.0.0007 http://dx.doi.org/10.1037/1082-989x.11.1.87 http://dx.doi.org/10.1037/1082-989x.11.1.87 http://docsfiles.com/pdf_paper_or_plastic_data_equivalence_in_paper_and_electronic_diaries.html http://docsfiles.com/pdf_paper_or_plastic_data_equivalence_in_paper_and_electronic_diaries.html http://www.educause.edu/ero/article/comparison-student-and-faculty-academic-technology-use-across-disciplines http://www.educause.edu/ero/article/comparison-student-and-faculty-academic-technology-use-across-disciplines http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 ismail, m. a. (2011) ‘identifying how novice researchers search, locate, choose and use web resources at the early stage of research’, malaysian journal of library & information science, vol. 16, no. 3, pp. 67�85. its. (2012) it training record (information technology services, trans.), university of otago, dunedin, new zealand. jackson, m. (2005) ‘the impact of ict on the development of information literacy by students in further education’, journal of eliteracy, vol. 2, pp. 15�26. kennedy, g. et al., (2009) educating the net generation: a handbook of findings for practice and policy [online], available at: http://www.netgen.unimelb.edu.au/downloads/handbook/ netgenhandbookall.pdf lai, k.-w & hong, k.-s. (2013) ‘technology use and learning characteristics of students in higher education: do generational differences exist? british journal of educational technology, vol. 46, no. 4, pp. 725�738. larsson, a. (2002) socio-technical aspects of distributed collaborative engineering, doctor of philosophy thesis, division of computer aided design, lulea university of technology, p. 36. lawlor, b. & donnelly, r. (2010) ‘using podcasts to support communication skills development: a case study for content format preferences among postgraduate research students’, computers & education, vol. 54, no. 4, pp. 962�971. li, (2008) ‘the relationship between computer literacy and online learning attitudes for students in the graduate school of education in taiwan’, dissertation abstracts international section a: humanities and social sciences, vol. 69, no. 6-a, p. 2236. liew, c. l., foo, s. & chennupati, k. r. (2000) ‘a study of graduate student end-users’ use and perception of electronic journals’, online information review, vol. 24, no. 4, pp. 302�315. magliaro, (2012) ‘comparing information literacy needs of graduate students in selected graduate programs through the technology acceptance model and affordance theory’, dissertation abstracts international section a: humanities and social sciences, vol. 73, no. 6-a, p. 2038. mcalpine, l. (2013) ‘doctoral supervision: not an individual but a collective institutional responsibility’, journal for the study of education and development, vol. 36, no. 3, pp. 259�280. mccarthy, j. (2012) ‘international design collaboration and mentoring for tertiary students through facebook’, australasian journal of educational technology, vol. 28, no. 5, pp. 755�775. meerah, t. s. m. (2010) ‘readiness of preparing postgraduate students in pursuit of their doctoral programme’, procedia � social and behavioural sciences, vol. 9, no. 5, pp. 184�188. doi: http://dx.doi.org/10.1016/j.sbspro.2010.12.133 murray, j. (2011) ‘cloud network architecture and ict’ [online], december 2011, available at: http://itknowledgeexchange.techtarget.com/modern-network-architecture/cloud-networkarchitecture-and-ict/ nabeel, a., shahrir, j. & leng, l.h. (2013) ‘measuring attitudes toward computer and internet usage among postgraduate students in malaysia’, tojet : the turkish online journal of educational technology, vol. 12, no. 2, pp. 200�217. nair, p. & pillay, j. (2004) ‘exploring the validity of the continuous assessment strategy in higher education institutions’, south african journal of higher education, vol. 18, no. 2, pp. 302�312. odaci, h. (2013) ‘risk-taking behavior and academic self-efficacy as variables accounting for problematic internet use in adolescent university students’, children and youth services review, vol. 35, no. 1, pp. 183�187. ongoz, (2010) ‘ e-book usage of graduate students studying educational sciences in turkiye’, the turkish online journal of distance education tojde, vol. 11, no. 1, pp. 198�210. onilude, o. o. & apampa, o. r. (2010) ‘effects of information and communication technology on research and development activities: the fiiro experience’ [online], available at: http://www.webpages.uidaho.edu/�mbolin/onilude-apampa.htm prensky, m. (2001) ‘digital natives, digital immigrants’, on the horizon, vol. 9, no. 5, pp. 1�6. rowlands, i., et al., (2007) ‘what do faculty and students really think about e-books?’ [online] available at: http://www.homepages.ucl.ac.uk/~uczciro/findings.pdf k.n. sim 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 http://www.netgen.unimelb.edu.au/downloads/handbook/netgenhandbookall.pdf http://www.netgen.unimelb.edu.au/downloads/handbook/netgenhandbookall.pdf http://dx.doi.org/10.1016/j.sbspro.2010.12.133 http://itknowledgeexchange.techtarget.com/modern-network-architecture/cloud-network-architecture-and-ict/ http://itknowledgeexchange.techtarget.com/modern-network-architecture/cloud-network-architecture-and-ict/ http://www.webpages.uidaho.edu/~mbolin/onilude-apampa.htm http://www.webpages.uidaho.edu/~mbolin/onilude-apampa.htm http://www.homepages.ucl.ac.uk/~uczciro/findings.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 shaw, f. s. (2000) ‘a survey of graduate students as end users of computer technology: new roles for faculty’, office systems research journal, vol. 18, no. 1, pp. 21�40. sim, k. n. & butson, r. (2013) ‘do undergraduates use their personal computers to support learning?’, procedia � social and behavioral sciences, vol. 103, pp. 330�339. sim, k. n. & butson, r. (2014) ‘to what degree are undergraduate students using their personal computers to support their daily study practices?’, iafor journal of education, vol. 2, no. 1, pp. 158�171. smith, s. d. & caruso, j. b. (2010) the ecar study of undergraduate students and information technology, 2010, educause, pp. 118. available at: http://net.educause.edu/ir/ library/pdf/ ers1006/rs/ers1006w.pdf smith, s. d., salaway, g. & caruso, j. b. (2009) the ecar study of undergraduate students and information technology, vol. 6, educause, p. 130, [online] available at: http://net.educause. edu/ir/library/pdf/ers0906/rs/ers0906w.pdf strengers, y. a.-a. (2014) ‘interdisciplinarity and industry collaboration in doctoral candidature: tensions within and between discourses’, studies in higher education, vol. 39, no. 4, pp. 546�559. sultan, n. (2010) ‘cloud computing for education: a new dawn? international journal of information management, vol. 30, no. 2, pp. 109�116. sutton, a. m. & jacoby, j. (2008) ‘a comparative study of book and journal use in four social science disciplines’, behavioral & social sciences librarian, vol. 27, no. 1, pp. 1�33. tenopir, c. (2003) use and users of electronic library resources: an overview and analysis of recent research studies, council on library and information resources, washington, dc, p. 172. tltteam. (2011) ‘equipped for online learning?’, november 2011, [online] available at: http:// www.timelesslearntech.com/blog/equipped-for-online-learning/ van vliet, p. j. a., kletke, m. g. & chakraborty, g. (1994) ‘the measurement of computer literacy: a comparison of self-appraisal and objective tests’, international journal of human-computer studies, vol. 40, pp. 835�857. varshney, l.r. (2012) ‘the google effect in doctoral theses’, scientometrics, vol. 92, no. 3, pp. 785�793. wallace, p. & clariana, r. b. (2005) ‘perception versus reality: determining business students’ computer literacy skills and need for instruction in information concepts and technology’, journal of information technology education, vol. 4, pp. 141�151. wellington, j. (2012) ‘searching for ‘‘doctorateness’’’, studies in higher education, vol. 38, no. 10, pp. 1490�1503. williamson, k., et al., (2007) ‘research students in the electronic age’, communications in information literacy, vol. 1, no. 2. [online] available at: http://www.comminfolit.org/index. php?journal=cil&page=article&op=viewarticle&path%5b%5d=fall2007ar1&path% 5b%5d=48. winter, j, cotton, d, gavin, j & yorke, j.d. (2010) ‘effective e-learning? multi-tasking, distractions and boundary management by graduate students in an online environment’, research in learning technology, vol. 18, no. 1, pp. 71�83. wood, e., et al., (2011) ‘examining the impact of off-task multi-tasking with technology on real-time classroom learning’, computers & education, vol. 58, pp. 365�374. doi: http://dx. doi.org/10.1016/j.compedu.2011.08.029 wu, m. d. & chen, s. c. (2012) ‘how graduate students perceive, use, and manage electronic resources’. aslib proceedings, vol. 64, no. 6, pp. 641�652. doi: http://dx.doi.org/10.1108/ 00012531211281779 research in learning technology citation: research in learning technology 2016, 24: 30717 http://dx.doi.org/10.3402/rlt.v24.30717 15 (page number not for citation purpose) http://net.educause.edu/ir/library/pdf/ ers1006/rs/ers1006w.pdf http://net.educause.edu/ir/library/pdf/ ers1006/rs/ers1006w.pdf http://net.educause.edu/ir/library/pdf/ers0906/rs/ers0906w.pdf http://net.educause.edu/ir/library/pdf/ers0906/rs/ers0906w.pdf http://www.timelesslearntech.com/blog/equipped-for-online-learning/ http://www.timelesslearntech.com/blog/equipped-for-online-learning/ http://www.comminfolit.org/index.php?journal=cil&page=article&'op=viewarticle&path%5b%5d=fall2007ar1&path%5b%5d=48 http://www.comminfolit.org/index.php?journal=cil&page=article&'op=viewarticle&path%5b%5d=fall2007ar1&path%5b%5d=48 http://www.comminfolit.org/index.php?journal=cil&page=article&'op=viewarticle&path%5b%5d=fall2007ar1&path%5b%5d=48 http://dx.doi.org/10.1016/j.compedu.2011.08.029 http://dx.doi.org/10.1016/j.compedu.2011.08.029 http://dx.doi.org/10.1108/00012531211281779 http://dx.doi.org/10.1108/00012531211281779 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30717 http://dx.doi.org/10.3402/rlt.v24.30717 (page number not for citation purpose) 1 *corresponding author. email: tgrigoryan@hct.ac.ae research in learning technology 2018. @ 2018 t. grigoryan. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology vol. 26, 2018 research article investigating digital native female learners’ attitudes towards paperless language learning tsoghik grigoryan* general studies program, higher colleges of technology, al ain, united arab emirates (received: 18 october 2016; final version received: 10 january 2018) this study is an investigation of paperless language learning in the context of the united arab emirates. the purpose of this study was to examine emirati level 1 english language learners’ attitudes towards the ipad use as a means of language learning. it was done through a cross-sectional survey questionnaire, wholly composed of fixed-choice questions, and through weekly reflective journals that were written by the teachers teaching the groups. the survey collected data through a questionnaire from 80 students who had been exposed to paperless language learning for a duration of 80 teaching periods. the data collected showed positive student attitudes towards ipad implementation as a language learning tool in terms of learner satisfaction, motivation, perceived tool usefulness and learning effectiveness. reflective journal analysis showed that the digital world presents the students with a direct link between the effort taken and the reward received, whereas the feedback or the reward given by the teachers in the traditional classroom was either too nebulous or too slow to motivate students to keep the pace of progressive learning. keywords: attitudes; english as a foreign language; ipad; language learning; mobile classroom; paperless introduction prensky (2012) has identified today’s generational change that leads to a really big discontinuity, which one can call ‘singularity’: an event that changes things so fundamentally that there is no way back. he has used the term ‘digital natives’ and argued that the new generation is different from the previous ones because of the technological changes. prensky’s (2001) ideas about digital natives and digital immigrants were published more than 10 years ago, according to which a new generation of digital natives are presently entering the schools and other educational institutions. the urge for educators to accept the needs of this generation differently has become even more crucial. a fundamental problem raised by prensky (2001) about today’s educational system in the eyes of the new generation regards the absolute boredom in schools, which he explains with the promptly growing divide between the information and knowledge learners can get out of school and the narrow confines of their lives within it. the traditional teaching methods, where teachers talk through textbooks and students listen, are regarded as an old practice not providing students with the skills they need for the modern world (jukes, mccain, and lee 2010). while teachers continue to teach many traditional skills, there will be a shift in emphasis of importance mailto:tgrigoryan@hct.ac.ae http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 2 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) of those skills (jukes, mccain, and lee 2010, p.63). there was a time when it was important to learn to write nicely and develop good handwriting. despite the cognitive benefits of practicing good handwriting, its emphasis as an important skill has changed significantly. learning has moved to a digital realm and writing is realised using digital software tools. this is only one example out of many possible ones that restates the importance of re-evaluating teaching and learning ways in light of the realities of the new digital world. teachers no longer have to be in the centre of attention, but to become facilitators who can guide the learners to the answers they search for. since there are no fixed right or wrong ways of integrating technology into the classroom, the solution is to come up with the most effective learning environment for learners. this is obvious that the description and understanding of the learning has changed significantly over time and is still in its transformational modern stage, which necessitates exploration and investigation of the pros and cons of innovative changes in the field of english as a foreign language paperless classroom. ‘this paperless system clears the way for communication beyond the space and time limitations inherent in any traditional course’ (dan 2002, p. 162). the move to paperless classroom is met mostly with fear and resistance to change by educators. on the contrary, it is mostly welcomed by today’s learners. changes and challenges are not easy, but they are inescapable and need time to research the advantages and disadvantages they could bring to today’s educational system. an example of this change was a tertiary level college in the united arab emirates (uae) that went paperless and implemented ipads for its foundations programme, eliminating paper and pen teaching-learning methods from the classroom. the innovation brought up enormous challenges in the technical and methodological fields, as well as in language learning ways, skills and practices. therefore, an investigation was needed to shed light on students’ attitudes towards various aspects of that innovative change. this study aimed at exploring the results of using mobile technology educationally by looking into learner attitudes towards using ipads in language learning. the research questions guiding this study were the following: 1. what are beginner level emirati students’ attitudes towards ipad implementation as a language learning tool in terms of learner satisfaction, motivation, perceived tool usefulness and learning effectiveness? 2. what are the emerging themes of the teachers’ reflective journals in the evaluation of their lessons and diagnosis of problems? the first research question was answered through a cross-sectional survey questionnaire, wholly composed of fixed-choice questions. the survey collected data through a questionnaire from four groups of 20 students who had been studying english paperless for 80 teaching-learning periods. the data analysis showed positive student attitudes towards ipad implementation as a language learning tool in terms of learner satisfaction, motivation, perceived tool usefulness and learning effectiveness. the second research question was answered through the weekly reflective journals written by four teachers throughout the two phases of the experiment. the analysis of the 24 reflective journals showed that ipad-based learning carries more benefits than textbook-based learning in terms of motivation and learning effectiveness. language achievement is more enjoyable when ipads are used as a means of learning. http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 3 (page number not for citation purpose) literature review theories and approaches related to innovation adoption include, but are not limited to, diffusion of innovation theory (rogers 2003), theory of reasoned action – tra (fishbein and ajzen 2010), enhanced technology acceptance model – tam (venkatesh and davis 2000), unified theory of acceptance and use of technology – utaut (venkatesh and davis 2000), social cognitive theory – sct (bandura 1999) and activity theory – at (engestrom 1999). many other approaches have also emerged and have been developed from the above-mentioned models and theories to conduct research into technology use. in turn, they have been practiced in various spheres of life as well as in education. though information and communication technology (ict) adoption is a well-researched area of study in information technology (it) research, its application to education is still under investigation (churchill and wang 2014; lai, sham, and tian 2014; meder and wegner 2015). some educational institutions today insist on integrating mobile devices without determining their benefits and possible negative effects on students’ education. funding, innovative ways of management, ecological problems, modern pedagogies and other factors are all mentioned as hurdles for ict adoption, yet the area lacks research and consensus (meder and wegner 2015). table 1 depicts six innovation adoption theories and discusses them accordingly. one of the approaches to the adoption of innovation that educational technologists will benefit greater understanding and awareness from is the diffusion of innovation theory. diffusion of innovation theory explains how an idea or a product gains an impetus over time and diffuses through a specific population (boczkowski 2010). it aims to study people’s technology adoption in terms of time, innovation, communication methods and the social system. this means that people adopt an innovative product or a new idea and perceive it as new or innovative. ghezzi, rangone, and balocco (2013) claim that diffusion theory should be revisited to identify external determinants that enable or hinder evaluation of the new technology prior to the technological activation phase in education. the model they propose addresses regulation, environment, strategy and technology (rest), which are the four determinants stimulating technology activation. another theory that postulates that the behavioural target is shaped by the individual’s attitudes and subjective norms is the tra. it was developed to describe the connection of individuals’ behaviours and attitudes within their actions. table 1. summary of innovation adoption theories. diffusion of innovation theory theory of reasoned action enhanced technology acceptance model unified theory of acceptance and use of technology social cognitive theory activity theory flexible attitudes •  beliefs •  attitudes •  perceived ease  of use •  perceived  usefulness •  performance  expectancy •  effort  expectancy self-efficacy perceived ease of use •  comparability •  observability subjective norm •  subjective •  result  demonstrability social influence facilitating conditions outcome expectations result demonstrability http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 4 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) the aim of the tra is to investigate the relationship between attitude and behaviour based on two major concepts: principles of compatibility and behavioural intention. with this characteristic, the tra is a predictive model and, therefore, is used in a variety of fields, such as banking, public, education, and industries to predict individuals’ actions based on certain criteria. (mishra, akman, and mishra 2014, p. 30) the attitude and subjective norm are the central factors of individuals’ objectives of implementing icts and were found to have a big impact on adopting ict (doane, pearson, and kelley 2014; mishra, akman, and mishra 2014). several studies found that subjective norm affects an individual’s behavioural intention (cooke and french 2008; doane, pearson, and kelley 2014), satisfaction, information sharing (tsai, chen, and chien 2012) and perceived usefulness (venkatesh and davis 2000). the enhanced tam suggests that perceived ease of use and perceived usefulness are direct channels of technology acceptance behaviours. according to gong, xu, and yu (2004), ‘perceived usefulness is defined as the prospective user’s subjective probability that using a specific application system would increase his or her job performance within an organizational context’ (p. 366). perceived ease of use, on the other hand, ‘refers to the degree to which the prospective user expects the target system to be free of effort’ (p. 366). several studies have used enhanced tam as their theoretical background for explaining technology use and adoption (gong, xu, and yu 2004; teo 2009) and found that perceived usefulness influences attitudes and satisfaction towards technology use (limayem, hirt, and cheung 2007; venkatesh and davis 2000). the utaut looks at how two factors – intention and behaviour – progress over time and are moderated by gender, age and experience (venkatesh and davis 2000). the determinants of intention are supposed to be the performance expectancy, effort expectancy and social influence, whereas the determinants of behaviour are supposed to be the intention and facilitating conditions (venkatesh and davis 2000). sct describes how the individual obtains and sustains a specific behaviour based on learning from others (bandura 1999). this theory suggests that the acquisition of knowledge is related to observing others within the context of social interactions. it also explains that a specific behaviour can be influenced by final expectations and self-efficacy, while final expectations and self-efficacy can be affected by prior behaviour. several studies have used sct and found significant relationships with other concepts in ict adoption and use, such as that self-efficacy can positively influence perceived ease of use and perceived usefulness (bandura 2002; chiu, hsu, and wang 2006; swearer et al. 2014). at, as propounded in soviet psychology, is the umbrella term for several eclectic social sciences theories. it is not a predictive theory, but more of a descriptive framework, which studies the whole activity system beyond one user. therefore, at was chosen as the framework of this study. ‘at is a cross-disciplinary framework for studying how humans purposefully transform natural and social reality, including themselves, as an ongoing culturally and historically situated, materially and socially mediated process’ (roth, radford, and lacroix 2012, p. 1). entrenched in the dialectal psychology, it transcends traditional dichotomies of macro and micro, thought and action, intervention and observation, qualitative and quantitative by integrating three perspectives: objective, ecological and sociocultural perspectives (engestrom 1999). another reason why at was used in this study was because in this theory the subject implements a tool to perform cognitive functions and cannot directly act on the object. the unit of analysis here is the concept of object-oriented, collective and culturally mediated human activity system. when ipad is used in language learning, it is not seen as the object of learning http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 5 (page number not for citation purpose) the language but as a device to realise the language acquisition process. thus, based on the technological outlook of at, individual functioning is considered to be distributed across and situated within the transaction of the contexts of the subject, available tools and community with the division of labour (uden 2007 in liaw and huang 2014). methodology site and subject selection the site of the study was the largest governmental higher learning institution in the uae that had 17 campuses across the country. the institution provided post-secondary education to 17–25-year-old emirati nationals. campuses were segregated for male and female students but the multinational staff of both sexes taught at either campuses. the institution started the ipad project in 2012 at the foundation studies programme and moved from traditional language learning and teaching to paperless and mobile methods, which meant teaching and learning was realised solely through ipad applications, online resources and electronic texts. the foundation studies programme supported students needing english language assistance in meeting the admission criteria of the bachelor’s degree programme prior to entering their chosen programme major. the programme offered four levels of english proficiency: beginner, pre-intermediate, intermediate and advanced. this experimental phase concentrated on the foundation studies programme and on the level of english proficiency called beginner level or level 1. this was the first of four english courses at the foundations level. students were expected to have common european framework of reference (cefr a1) competence at the beginning of the course with a target exit of a2. the beginner level and the foundations language programme were chosen for the experiment because this was the programme and the level that all students needed to complete first as they entered the institution and this was when they tried ipads for language learning purposes for the first time. research participants were 17–25-year-old emirati women who, after high school, were placed in higher institutions according to their common educational proficiency assessment (cepa) test results. cepa is a set of locally developed standardised tests used for admissions and placement by three federal institutions of higher education in the uae. students entered the institution at the cefr a1 level, having a cepa score of 150 and were expected to exit midway through cefr a2 level with a score of cepa 156. out of 250 newly admitted students, 80 were randomly chosen for the experiment. then, the 80 students were randomly placed in four different groups with 20 students each: two ipad groups and two textbook groups. the researcher and three other level 1 teachers were assigned to teach those groups, each teaching one group. the teachers were employed by the institution and were equally experienced and qualified to work with level 1 students. methods and instruments the groups used the following textbooks for the course as mentioned in the common course outline: q: skills for success intro: reading and writing and q: skills for success intro: listening and speaking. all four groups followed the work plan and covered the same material, although, with different methods of instruction. as illustrated in table 2, the ipad groups followed the conditions set by the institution and fully used ipads for their language learning, with no presence of traditional learning methods, such as paperback texts, stationary or pen-written notes. instead, http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 6 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) they used annotation apps to complete the exercises, presentation apps to create presentations, interactive texts to complete reading and writing tasks and individual audios to complete the listening exercises. on the contrary, the textbook groups did not use ipads for anything and followed the traditional methods of using pen and paper for writing and paperback texts for reading. all groups covered the same material and followed the same programme with the different ways of teaching: with paper and paperless. the experiment lasted for 80 teaching periods, which was followed by the questionnaire administration. each teaching period lasted for 50 min. students had four periods of english every day, from sunday to thursday. they did not take any other subjects except english throughout level 1. during both phases of the experiment, all four teachers kept weekly journals and reflected about the teaching and learning process. after the first phase of the experiment, the two textbook groups that were not exposed to ipads became ipad groups in phase 2 and commenced their studies by using ipads for another 80 teaching periods. in this second phase, the groups used annotation apps to complete the e-book exercises, presentation apps to create multimedia presentations, interactive texts to complete reading and writing tasks and individual table 2. the treatment plan. the treatment plan classes with ipad (ipad groups) classes with out ipad (text book groups) reading: reading texts interactive e-book paper book reading task completion (e.g. creating conceptual maps, story lines, time lines, meaning depiction etc.) annotation apps , screen chomp, educreations, skitch. puppet pals hd, docscan hd. popplet, etc. pen, pencil and highlighter vocabulary and spelling (e.g. working with oxford 3000 lists, practicing vocabulary and spelling) e-book interactive exercises, spelling city, socrative, my library, vocabulary puzzles, etc. textbook exercises, print out materials writing: writing scripts interactive e-dook and pages paper and pen writing task completion. (e.g. writing sentences and paragraphs, completing grammar and spelling exercises) academic writing in english, grammer checker, english spelling and punctuation iacadamic, etc. paper and pen brainstorming and outlining (e.g. planning ideas, couceptualizing the plot, outlining the steps) mindmeister, brainstorming canvas imindq, project planning, mindomo, etc, paper and pen listening: listening audios interactive e-book textbook attached cd listening task completion (e.g. listening to dialogues, talks watching videos and completing exercises, taking notes while listening) listening master, sound note, voice recorder. & soundc loud, wattpad, etc. cd and book exercises speaking: speaking tasks (e.g. discussing statements, giving suggestions, offering help, debating over a topic, presenting themes) oral and virtual discussions, audio blogs, voice messaging, keynote presentations, imovies, prezi presentations, etc. oral discussions, debates, poster presentations communication(e.g. sharing information with peers and teachers, asking and answers questions concerning the language) online and real time studentstudent and student-teacher interaction student-student and student-teacher interaction. http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 7 (page number not for citation purpose) audios to complete the listening exercises. another two units were covered from each textbook within 80 periods of english language class. like after phase 1, the questionnaire was administered after phase 2 as well. survey research this study presented the following 15 research hypotheses (see table 3). the cross-sectional self-completion fixed-design survey questionnaire was administered in two phases. the survey collected data through a questionnaire from predetermined population, that is to say, students from four groups under experiment: from two ipad groups in the first phase and two ipad groups in the second phase. though survey method lacks control over variables and threats to validity, it yields empirical results. this approach was important for this study to answer its 15 hypotheses, describe the trends in the data and measure current attitudes and practices of the target population. table 3. research hypothesis. this research reports an activity theory based investigation into the use of ipads for language learning in terms of learner motivation, perceived satisfaction, perceived tool usefulness and learning effectiveness. this research tries to understand learners’ attitudes toward ipads as language learning tools. h1: perceived self-efficacy has positive predictive value for perceived satisfaction toward ipads as language beaming tools. h2: perceived self-efficacy has positive predictive value for perceived usefulness toward ipads as language learning tools. h3: perceived self-regulation has positive predictive value for perceived satisfaction toward ipads as language learning tools. h4: perceived self-regulation has positive predictive value for perceived usefulness toward ipads as language learning tools. h5: interactive learning environment have positive predictive value for perceived satisfaction toward ipads as language learning tools. h6: interactive learning environments have positive predictive value for perceived usefulness toward ipads as language learning tools. h7: perceived ease of use has positive predictive value for perceived satisfaction toward ipads as language learning tools. h8: perceived ease of use has positive predictive value for perceived usefulness toward ipads as language learning tools. h9: ipad based tasks have positive predictive value for perceived satisfaction toward ipads as language learning tools. h10: ipad based tasks have positive predictive value for perceived usefulness toward ipads as language learning tools. h11 : perceived satisfaction has positive predictive value for learner motivation toward ipads as language learning tools. h12: perceived usefulness has positive predictive value for learner motivation toward ipads as language learning tools. h13 : perceived satisfaction has positive predictive value for the effectiveness of ipads as learning tools. h14: perceived usefulness has positive predictive value for the effectiveness of ipads as learning tools. h15 : learner motivation has positive predictive value for the effectiveness of ipads as learning tools. http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 8 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) firstly, the survey questionnaire was administered after the first phase of the experiment only to two ipad groups because that were exposed to ipads and had already formed attitudes towards the ipad use in language learning. secondly, it was administered after the second phase of the experiment to the two ipad groups that were exposed to ipads in the second phase of the experiment. each phase of the experiment lasted for 80 teaching periods. the time gap between the first-time and the secondtime survey administration was 80 teaching periods. the survey questionnaire followed robson’s (2005) checklists: to help avoid problems in question wording (p. 245) and factors in securing a good response rate to a postal questionnaire (p. 249). no open-ended questions were used in this survey since the responses were expected to be 80. robson suggests the following: ‘cut down openended questions to a minimum with this type of questionnaire unless you can afford to spend a lot of time on analysis or have only a small number of responses to deal with’ (robson 2005, p. 245). this survey addressed nine factors: 1. self-regulation 2. self-efficacy 3. interactive learning environments 4. ease of ipad use 5. ipad-based tasks 6. perceived satisfaction 7. perceived usefulness 8. motivation 9. learning effectiveness this nine-factor survey questionnaire used a seven-point likert scale rating from completely disagree = 1 to completely agree = 7. the questionnaire was revised based on liaw and huang’s (2014) study and was designed to address emirati students’ perceptions of ipad use as a means of language learning. initially, the survey underwent reliability checks and was piloted before it was administered. it was translated into arabic which was the students’ mother tongue. the purpose of arabic translation was to minimise misunderstanding and misinterpretation of data. the survey employed the translation, review, adjudication, pretesting and documentation (trapd) method as suggested by the european social survey guidelines (hoffmeyer-zlotnik, jürgen, and warner 2014). survey results after 80 responses were collected in two phases, factor analysis was run to identify traits from the administered question-level data (table 4). confirmatory factor analysis was done for each of the predetermined factors. factors were rotated by applying varimax method. each item was loaded with a score greater than 0.3. as shown in table 4, all items were retained except one loading 0.140 which was less than 0.3. the excluded item was in the factor called ‘perceived satisfaction’ and was numbered as satisf02. the item was excluded. the overall reliability of all 29 items of the survey is 0.808, which is greater than 0.7, the minimum qualifying score. this means that the instrument is reliable (appendix b). kaiser–meyer–olkin’s test and bartlett’s test of sampling adequacy showed http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 9 (page number not for citation purpose) t ab le 4 . f ac to r an al ys is . f ac to rs it em s c on te nt f ac to r lo ad in g s el fre gu la ti o n s l f r e g 0 1 s l f r e g 0 2 s l f r e g 0 3 ip ad s ar e ea sy t o c ar ry ip ad s ar e ac ti ve l an gu ag e l ea rn in g to o ls ip ad a p p s fo r la n gu ag e le ar n in g ar e n o t ea sy t o r eg u la te .3 76 .6 20 .7 33 s el fef fi ca cy s l f e f 0 1 s l f e f 0 2 s l f e f 0 3 i am c o n fi d en t u si n g m y ip ad i n c la as i am n o t co n fi d en t u si n g m y ip ad f o r th e te st i am c o n fi d en t u si n g m y ip ad f o r el ec tr o n ic r es o u rc es .4 09 .3 92 .4 16 in te ra ct iv e le ar n in g en vi ro n m en ts in t l r e n v 0 1 in t l r e n v 0 2 in t l r e n v 0 3 in t l r e n v 0 5 in t l k e n v 0 6 n t l r e n v 0 7 i b el ie ve t h e ip ad c an d ev el o p c o m m u n ic at io n b et w ee n t h e st u d en ts i b el ie ve t h e ip ad c an d ev el o p c o m m u n ic at io n b et w ee n t h e st u d en ts a n d t h e te ac h er i b el ie ve t h e ip ad c an b e a m ea n s o f in fo rm at io n g ai n in g i b el ie ve t h e ip ad c an b e a m ea n s o f in fo rm at io n s h ar in g i b el ie ve t h e ip ad c an b e a m ea n s o f la n gu ag e le ar n in g i b el ie ve l an gu ag e le ar n in e w it h i p ad s is i n te ra ct iv e .6 59 .8 93 .8 56 .8 75 .7 46 .8 75 e as e o f ip ad u se e a s e u s e 0 1 e a s e u s e 0 2 e a s e u s e 0 3 it i s d if fi cu lt t o r ea d o n t h e ip ad it i s ea sy t o w ri te o n t h e ip ad it i s ea sy t o l is te n o n t h e ip ad .5 01 .3 44 .5 76 ip ad b aa ed t as k s e p d t s k 0 1 e p d t s k 0 2 e p d t s k 0 3 ip ad b as ed t as k s p ro vi d e q u ic k l an gu ag e le ar n in g ip ad b as ed t as k s ar e in te re st in g to d o ip ad b as ed t as k s ar e d if fi cu lt t o s h ar e .4 36 .4 21 .4 33 p er ce iv ed s at is fa ct io n s a t is f 0 1 s a t is f 0 2 s a t is f 0 3 i am s at is fi ed w it h t h e av ai la b il it y o f ip ad l an gu ag e le ar n in g ap p li ca ti o n s i an a n o t sa ti sfi ed w it h t h e ip ad b as ed a ss es sm en t ap p li ca ti o n s .5 34 .1 40 .5 38 p er ce iv ed u se fu ln es s u s e f l n s 0 1 u s e f l x s 0 2 u s e f l x s 0 3 i b el ie ve l an gu ag e le ar n in g th ro u gh i p ad s is p ro d u ct iv e i b el ie ve a cq u ir in g la n gu ag e le ar n in g sk il ls t h ro u gh i p ad s is p ro d u ct iv e i b el ie ve a cq u ir in g li te ra cy a la li a th ro u gh i p ad s is n o t p ro d u ct iv e 59 5 59 5 .5 95 m o ti va ti o n m o t iv 0 1 m o t iv 0 2 m o t iv 0 3 i d o n 't en jo y u si n g ip ad f o r m y la n gu ag e cl as s i p la n t o c o n ti nu e le ar n in g e n gl is h t h ro u gh m y ip ad i en co u ra ge o th er s to s ta rt u si n g ip ad f o r la n gu ag e le ar n in g .3 54 .4 41 .4 38 l ea rn in g ef fe ct iv en es s l r n e f t c t 0 1 l r x e f f c t 0 2 l r x e f f c t 0 3 i fe el i p ad s co u ld e n h an ce l an gu ag e le ar n in g ef fe ct iv en es s i fe el i p ad s co u ld m o ti va te l ea rn er s in to l an gu ag e le ar n in g i fe el i p ad s co u ld n o t p ro vi d e in te ra ct iv e w ay s to d ev el o p l an gu ae e sk il ls .4 37 .3 85 .3 85 http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 10 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) table 5. comparison of attitudes between ipad groups in phase 1 and ipad groups in phase 2. factors groups in phase 1 and 2 number mean std. deviation sig, value selfregulation ipad groups 40 6.6083 .36893 .142 ipad groups 40 6.7167 .27786 self-efficacy ipad groups 40 6.4667 .48803 .768 ipad groups 40 6.5000 .51750 interactive learning environments ipad groups ipad groups 40 40 6.7333 6.6042 .29187 .40065 .103 ease of ipad use ipad groups 40 6.6083 .26026 .155 ipad groups 40 6.5000 .39943 ipad based tasks ipad groups 40 6.8000 .38895 .051 ipad groups 40 6.5750 .60500 perceived ipad groups 40 4.0125 .21145 .317 satisfaction ipad groups 40 4.0625 .23170 perceived ipad groups 40 6.7625 .46668 .792 usefulness ipad groups 40 6.7375 .37532 motivation ipad groups 40 6.9250 .17683 .189 ipad groups 40 6.8500 .31078 learning ipad groups 40 4.9917 .05270 .051 effectiveness ipad groups 40 4.9500 .12054 a statistical value of 0.805 with p = 0, which means the sample size was adequate for running the factor analysis. exploratory factor analysis was done using principal component analysis method. based on the eigenvalues greater than 1, five factors were extracted. these factors were rotated using varimax method. small coefficients with absolute value less than 0.3 were suppressed. after the factor analysis a comparison of attitudes towards using ipads for language learning was carried out between 40 ipad group students’ responses from phase one and 40 ipad group students’ responses from phase 2 (table 5). the aim of this statistical calculation was to see whether the responses were significantly different between the two phases and whether the students from phase 1 had different experience and attitudes towards ipad use compared with phase two students. as shown in the table, the sig. value for all factors is greater than 0.05, which means there is no difference between the ipad group answers of phases 1 and 2 for any of the factors and the attitudes towards using ipads for language learning are similar in all four groups within both phases. first, a comparison of attitudes towards using ipads for language learning was carried out between phases 1 and 2 with 40 ipad group students’ responses each. then, another comparison of attitudes was done, this time using both phases with 80 students’ responses together as one (table 6). the aim of this statistical calculation was to see the minimum and maximum grades of the factors and to identify the most and least favoured factors. as shown in table 6, for the seven-scale questionnaire, the average minimum score is 5, which is above the average 4. 1. completely disagree 2. mostly disagree http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 11 (page number not for citation purpose) table 6. comparison of attitudes in two phases with four groups. survey 9 factors self regalation self efficacy interactive learning environ. ease of ipad use ipad based tasks. perceived j3lisf, perceived usefulness. motivation leaning effective. n 30 so so 80 so 80 so so 80 mean 6.6625 6.4s33 6.6688 6.5542 6.6875 4.0375 6.7500 6.8875 4.9708 std. dev. .32906 .50007 .35429 .33937 .51788 .22183 .42097 .25405 .09478 minimu m 5.33 5.33 5.50 5.00 5.00 3.50 5.00 6.00 4.67 maximu m 7.00 7.00 7.00 7.00 7.00 4.50 7.00 7.00 5.00 3. slightly disagree 4. neither agree nor disagree 5. slightly agree 6. mostly agree 7. completely agree the highest minimum score among nine factors is 5 (slightly agree) and it is registered in the ‘motivation’ factor with the highest mean score of 6.8875, which also shows the highest maximum score of 7 (completely agree). the lowest minimum score among nine factors is 3.50 (between slightly disagree and neither agree nor disagree) and it is registered in the ‘perceived satisfaction’ factor, which also shows the lowest maximum score of all, 4.50 (between neither agree nor disagree and slightly agree). this means that most of the students who received the questionnaire thought that ipads motivated them into language learning. however, they stayed neutral when it came to their satisfaction about using ipads for their language learning. test of normality was done on self-efficacy, self-regulation, interactive learning environments, ease of use, ipad-based tasks, perceived usefulness, perceived satisfaction, motivation and learning effectiveness (table 7). table 7. test of normality. kolmogoro vsmirno va shapirowilk statistic df sig. statistic df sig. interactive learning .216 80 .000 .852 80 .000 self-efficacy .212 80 .000 .858 so .000 satisfaction .430 30 .000 .606 80 .000 usefulness .399 80 .000 .645 80 .000 ipad based tasks .377 so .000 .657 80 .000 motivation .471 so .000 .504 so .000 ease of use .305 so .000 .775 so .000 lrneffect .533 so .000 .317 so .000 lilliefors significance correction http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 12 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) as shown in table 7, the variables do not follow normal distribution. all sig. values are less than 0.05, which means variables are not normally distributed. because the variables were not normal, parametric regression analysis was not applicable. since the aim of the research was to establish a path model, structural equation modelling (sem) was done using the amos software (figure 1). then the regression weights were calculated and hypotheses were set (table 8). table 9 illustrates the tested and the accepted hypotheses. as shown in the table, self-regulation showed positive effect on students’ satisfaction and perceived tool usefulness. interactive learning environment and self-efficacy have positive effect on perceived usefulness. perceived usefulness has positive effect on motivation, which in turn has positive effect on learning effectiveness. these factors are interconnected and have positive effect on each other according to the survey questionnaire analysis. reflective journals the qualitative analysis was performed through weekly reflective journal logs used by the teachers to record the procedure of the teaching and learning process for all four groups (table 10). the two data sources that informed this part of the study were soft copies of four teachers’ weekly written reflections in phase 1, which lasted for 4 weeks, and soft copies of two teachers’ weekly written reflections in phase 2, which lasted for another 4 weeks. this means that four teachers teaching four groups for 4 weeks wrote 16 journals in phase 1, and two teachers teaching two groups for another 4 weeks wrote eight journals in phase 2. by the end of the experiment, the teachers produced 24 reflective journals. the descriptive journals were standardised through team meetings and contained information about 80 teaching periods with four groups and another 80 teaching periods with two groups. they were divided into three sections. section 1 contained information about the work plan, such as details about the unit, learning outcomes and skills to be taught. in section 2, teachers wrote about what they covered within the week. in section 3, they reflected on their teaching and students’ learning (appendix a). the journal summaries showed the advantages and disadvantages of the methods and ways the material was introduced by the teachers and reproduced by the figure 1. path model. http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 13 (page number not for citation purpose) table 8. regression weights. estimate s.e. c.r. p label satisf <— slfef −.177 .045 −3.933 *** pari satisf <— slfreg .163 .068 2.385 .017 par_2 usefln <— intlern .415 .089 4.670 *** par_3 satisf <— intlern .078 .063 1.229 .219 par_4 usefln <— easu −.067 .093 −.722 .470 par_5 usefln <— ipdtsk .299 .061 4.910 *** par 6 satisf <— easu .068 .066 1.032 .302 par_7 satisf <— ipdtsk −.083 .043 −1.913 .054 par_8 usefln <— slfef .152 .063 2.411 .016 par_14 usefln <— slfreg −.053 .096 −.550 .582 par_15 mottv <— usefln .321 .068 4.696 *** parl o mottv <— satisf .123 .107 1.156 .247 parl l lrneffect <— usefln −.012 .017 −.715 .474 par_9 lrneffect <— satisf −.013 .024 −.538 .590 par_12 lrneffect <— mottv .329 .025 13.020 *** par 13 table 9. the tested and accepted hypothesis. h1 : perceived self-efficacy has positive predictive value for perceived satisfaction toward ipads as language learning tools. h2: perceived self-efficacy has positive predictive value for perceived usefulness toward ipads as language learning tools. h3: perceived self-regulation has positive predictive value for perceived satisfaction toward ipads as language learning tools. h4: perceived self-regulation has positive predictive value for perceived usefulness toward ipads as language learning tools. h5: interactive learning environment have positive predictive value for perceived satisfaction toward ipads as language learning tools. h6: interactive learning environments have positive predictive value for perceived usefulness toward ipads as language learning tools. h7: perceived ease of use has positive predictive value for perceived satisfaction toward ipads as language learning tools. h8: perceived ease of use has positive predictive value for perceived usefulness toward ipads a3 language learning tools. h9: ipad based tasks have posith_ e predictive value for perceived satisfaction toward ipads as language learning tools. h10: ipad based tasks have positive predictive value for perceived usefulness toward ipads as language learning tools. h11: perceived satisfaction has positive predictive value for learner motivation toward ipads as language learning tools. h12: perceived usefulness has positive predictive value for learner motivation toward ipads as language learning tools. h13: perceived satisfaction has positive predictive value for the effectiveness of ipads as learning tools. h14: perceived usefulness has positive predictive value for the effectiveness of ipads as learning tools. h15: learner motivation has positive predictive value for the effectiveness of ipada as learning tools. http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 14 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) table 10. summary of reflections. summar y of positive an d negative teacher reflections from two experimental phases| positive reflections negative reflections ipad based classes •  learning became alive for the students while they were creating something that was new and meaningful to them. •  one of the tasks they enjoyed doing was the imovie project which they did in pairs. •  they liked working with the interactive textbook. •  to practice this grammar aspect we downloaded tense buster app and successfully practiced it through various interactive exercises. •  i was impressed with the mini projects that the pairs quickly and easily produced through the imovie. not only the unit vocabulary and grammar were mostly used accurately, the mood and motivation apparently took over the hard work. •  on this note i will say i had a strong feeling that students made friends with ipads and found it easy to work with them to achieve their goals. •  the interactive exercises allowed students to redo the difficult parts and get immediate auto feedback. •  comprehension checks were done through collaborative tasks; asking and answering questions and sharing students' voice recordings with the teacher. •  i felt my students enjoyed their learning this week and were on task at all times. •  every time i saw students had problems understanding a language aspect (ex. subject verb agreement), i created activities from online resources cm spot •  students had some issues with operating the programs and needed assistance. •  though we had couple of technical glitches with ipad apps and e-book codes, we successfully went through this first week. •  i had two students constantly asking for permission to write on a paper. they explained that they got irritated when writing on the screen and that the space for writing was too small. •  though they were not quite happy and willing to work on ipads, they tried to do their best. •  in the beginning, for some of us it [a new app] was difficult to operate and we wanted to give up. a student then insisted and suggested to explore the settings and options and to •  consider doing a dry run with the peers to iron out the kinks. •  it was a transition for this class from paper based to ipad based learning. therefore, some of the students took quite long to adjust. •  there were also negative reactions this week: a student was really upset when she accidentally deleted her work that she spend considerable time on. i tried to bring it back but was not successful. •  another case was with the paid apps. some students didn't have credit cards to purchase those apps we needed in class and had to use the free ones which didn't ha\-e all the necessary functions. http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 15 (page number not for citation purpose) table 10. (continued ) summar y of positive an d negative teacher reflections from two experimental phases| positive reflections negative reflections book based classes •  students effectively completed all reading activities in groups or pairs. •  the group successfully worked with the book exercises and completed all of them individually, in pairs and in groups. •  we conducted discussions about the school day length and time for homework and fun. students were quite active in expressing their opinions and volunteering to contribute to the discussion. •  they had fun and tried their best to express themselves to introduce their friends through the project they did in pairs. •  they used their notes and vocabulary logs to express opinion and bring reasons for their answers. •  they enjoyed working in pairs and planning a tour of their college. they drew a map and decided where the tour began and ended, and what places had to be on the tour. •  vocabulary building was discussed this time through the dictionary entries. they enjoyed looking through thick dictionaries and searching for words. •  before writing the paragraphs they learned how •  to write idea maps and use them for writing. they actually did it very well. •  in the end i asked my students to say what they thought about their friends' poster presentations and voted for the best one. it was a nice experience. •  jthe book activities were not enough to practice and understand present of be and i brought in extra materials every day. •  they easily got tired of the textbook but had fun with the new listening exercises and discussions. •  i felt classes were long and boring for the students and they complained about not having interesting exercises in the book. •  i tried to make the textbook more interesting for them by creating competitions and games, which took lots of effort and time to think and constantly come up with something. •  another drawback was the grammar issue with do and does. i had to create extra materials and adapt exercises, print, cu t and prepare them for group or pair work. i wouldn't go through this if students had interactive e-textbooks of course. •  when i asked them why they were so happy and enthusiastic about the project they simply reasoned that it was different from the boo k •  the units were long and the students hardly managed to complete all tasks and exercises from both books. •  grammar took little bit longer than expected because each and every student needed feedback on any written piece produced. •  .. . the biggest concern of the week for me was the lack of time. students. according to lamb (2013), ‘reflecting on different aspects of the research process when writing a research journal provides a forum to record concerns which may have otherwise been lost or simply not considered’ (p. 85). http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 16 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) reflective journal results the aim of the reflective journal data analysis was to identify the themes and patterns that were grounded in the data (corbin and strauss 2008) and gain insights from level 1 teachers’ reflections through weekly journal entries. the study used grounded theory and constant comparative method to analyse the data, as they aimed at generating a theory that was grounded in the data from the participant teachers who had experienced and reflected on the process. the first step was reading the reflections from beginning to end many times trying to understand the mood, feelings and experiences of the participants. with the second step, each section was examined in detail and memos were written. memos helped to form ideas and keep track of thinking about various matters. methodological notes were also inserted between the memos to explain, differentiate and understand how they fitted together. after the open coding, the concepts were related. next, the core concepts were identified from the coded data categories and subcategories through open and axial coding. finally, selective coding was done. when selecting and relating categories in this final step, two core concepts were revealed in the context of learning, language learning and motivation, which eventually became the selective codes. these two selective codes were then inserted into the at of using two methods in language teaching, ipad based and textbook based, and were analysed through multivariate angles of that framework. though tools exist whenever people are involved in specific activities, they are also created through those activities. therefore, a mediating action consists of a subject, an object and tools that are continuously transformed through the activity. this view of a mediated activity draws on a theory of learning that looks at language learners as subjects who actively construct meaning within the context. in this study, the facilitator is the method and subjects are language learners. the context of learning involves the two core concepts that emerged from the coded data categories: motivation and learning. they are to transform the object into an outcome. in this study, the at of using two methods for language learning – ipad-based and textbook-based language learning – starts out from the idea that students start using the methods in the context of their participation in language learning activities. hence, language learning is a mode of activity that can be characterised through a method that describes how the activity in general is accomplished. to describe this accomplishment, two conceptual models of at have been formed based on two methods: ipad-based and textbook-based language learning (figures 2 and 3). the models look at the activity as a purposeful, productive process carried out by subjects, who are 80 beginner-level language learners placed in four homogeneous groups, on an object, which is language achievement, via two mediating tools: ipads and textbooks (engestrom 1999). the subject, object and tool are observed within the context of a teaching-learning process, in which activity is embedded. in the at triangle, the control of learning is ipad-based versus textbook-based tasks, and the communication of learning is interactive versus traditional learning environment. in this section of reflective journal analysis of the study, these six elements comprise and govern the activity system. this practical view of the activity with emphasis on the mediating tools makes the at well suited for the analysis of process and activities involving significant components in higher education (khanova 2012). http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 17 (page number not for citation purpose) figure 2. activity theory conceptual model of ipad-based language learning. figure 3. activity theory conceptual model of textbook-based language learning. http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 18 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) the reflective journals, which were analysed through at loops, showed that the language achievement is more enjoyable when ipads are used as a means of learning. the level 1 language learners experienced the digital world for learning the target language that was completely out of sync with traditional approaches and assumptions about teaching, learning, giving feedback or assessing. despite the best of intentions to teach the language through traditional methods, the textbook-based teaching and learning did not really connect with modern language learners primarily because the traditional methods were targeted at learners from another age. as reflected in the journals, the students wanted their learning to be relevant, fast, applicable and instantly useful. what is more is that they wanted to know the possible relevance of learning specific language aspects for them and their world and why it could not be fun most of the time. while some experts may argue that the experiences the digital generation have are worthless and that play and games are simply preparation for work and life after school, for today’s digital generation, play is work, and work is increasingly seen in terms of games and game play. (jukes, mccain, and crockett 2010, p. 41) another interesting point raised in the journals was the constant complain of the textbook group students about not receiving immediate feedback to the completed work and teachers not being able to give enough of the class time for individual feedback to students after every language task. the clue here is that the digital world presents the students with a direct link between the effort taken and the reward received, whereas the feedback or the reward given by the teachers in the traditional classroom during the experiment was either too nebulous or too slow to motivate students to keep the pace of progressive learning. it must be mentioned that unlike the textbook groups, ipad groups had no issues connected with the pace of receiving feedback or waiting for their work to be checked. as jukes, mccain, and crockett (2010) mention, a direct connection between effort and reward, immediate or deferred, is why digital culture resonates so strongly with the digital generation. in terms of immediate rewards, digital culture provides them with exactly what they not only want, but what they need most _ positive feedback. (p. 40) discussion through the survey data analysis, the study recorded positive student attitudes towards ipad implementation as a language learning tool in terms of learner satisfaction, motivation, perceived tool usefulness and learning effectiveness. the perceived self-efficacy, which referred to students’ beliefs in their capacity to execute behaviours necessary to produce specific language performance, had negative effects on perceived satisfaction, which supposed the fulfilment of their expectations. on the other hand, the fulfilment of their expectations was encouraged by their self-regulation, that is to say, by their ability to monitor and control their own behaviour and learning habits through ipads. in other words, students were happy with the control they had over their learning through ipads as a means of learning. the perceived usefulness was the fundamental determinant of user acceptance. it was positively affected by the interactive learning environments, which was the pedagogical approach that incorporated virtual networking and communication by students. interestingly, according to the survey results, neither the interactive learning environment nor the ease of ipad use had any effects on students’ satisfaction http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 19 (page number not for citation purpose) of the device being implemented in their education. the ease of ipad use, which was the user-friendly operational system, had negative effect on the perceived usefulness. in other words, students did not see it as a useful learning tool because of its user-friendliness. on the contrary, the ipad-based tasks had a positive effect on the perceived usefulness. this means students saw the ipad as a useful learning tool because of the learning it was able to provide through online tasks. the self-regulation had positive effect on students’ satisfaction and perceived tool usefulness. in other words, working with ipads raised students’ satisfaction level as well as their level of language achievement. students felt positive studying paperless and did their best to achieve success. they were satisfied and positive about this method of learning because they could regulate their device to serve their individual learning needs and pace. because students were able to manipulate the electronic applications and multitask, they were happy with what they learned and, hence, accepted ipads as a means of learning. interactive learning environment had positive effect on perceived usefulness. moreover, self-efficacy had positive effect on perceived usefulness. this means that interactive learning environment, which is seen as communication between students, as well as between students and the teacher, made students positive about the usefulness of ipads as a means of language learning. ipads played a major role in how students efficiently approached their goals, overcame challenges, solved problems and completed various language tasks. therefore, self-efficacy, which is considered as students’ beliefs in their ability to succeed in language learning, helped them understand the importance and usefulness of the device. perceived usefulness had positive effect on motivation, which in turn had positive effect on learning effectiveness. this is like a chain reaction in a sense that students’ positive attitudes towards ipad use as a means of language learning motivated them into learning the target language, which resulted in higher language achievement and effectiveness. perceived usefulness motivated them into accomplishing language tasks and overcoming language difficulties by the help of the paperless learning. moreover, motivation and learning effectiveness resulted from the interaction of learning needs and positive outcomes that students had towards the accomplishment of language tasks. therefore, it must be stated that according to the survey analysis, ipads stimulate students’ desire and energy to keep them continuously interested in and committed to language learning tasks, assignments, projects and other language learning requirements and make efforts to attain the goals. not only the survey results but also the reflective journal analysis showed that ipad-based language learning is more productive than textbook-based language learning. the reflective journals that the four teachers involved in the experiment completed throughout the two phases revealed that the interactive learning environment in both phases showed quite a high level of student motivation and language achievement. the activity of learning through textbooks as compared to ipads, which was undertaken by the same subjects using two different tools in two different phases to achieve the same object, transformed it into an outcome in both phases and demonstrated positive results regardless of whether it was textbook related or technological challenges. however, as the 24 reflective journals showed, ipad-based learning carried more benefits than textbook-based learning in terms of motivation and learning effectiveness. therefore, the language achievement is fun when ipads are used as a http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 20 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) means of language learning. however, this is not to say that the skills and knowledge traditionally used or taught in the 20th century are obsolete today. much educational practice developed and used in the 20th century are effective nowadays and some of them are more effective than before. however, as the results of this study highlight, there has been a change in emphasis in what is essential for success, and a completely different skill set is required to live and learn in the modern digital world, such as information, solution, collaboration, creativity and media fluencies (jukes, mccain, and crockett 2010). unlike in the past, when students had to be patient and wait until they could get a chance to use whatever they had learnt in real life, today’s digital native students can easily and daily experience direct connections. they can participate in profound social revolutions like crowdsourcing and personally significant ones like online voting. above all, these students are in favour of digital devices because in the digital world they can easily cooperate, compete, share and learn with their peers around the world. this is what the results of this study highlighted. recommendations in this information and technology era, regardless of their preferred professions, graduates are facing growing demands to have high level of english communication skills, in addition to their majors, before entering the workforce. however, many students face difficulties meeting their needs within the limited class hours during their education. to overcome these challenges, many learners are implementing mobile technologies in their language learning as well as in other discipline areas. the reason of using mobile technologies is that they provide facility to access to authentic materials. though this study showed that it is enjoyable to use mobile technology and particularly ipads in language learning, it did not widen the scope of looking at different levels of english proficiency. therefore, a further study should concentrate on higher levels of english proficiency paperless classrooms and compare ipad-based learning with other traditional methods. this study concentrated on emirati female students only. hence, it would be interesting to investigate the ways that integration of mobile technology can change the learning outcomes and attitudes of other nationality second language learners’ english proficiency. in addition, it would be interesting to see whether gender plays a significant role in studying with mobile devices. moreover, it would be productive to experiment with ipad-related specific tasks and applications to see which exact language tasks, practices and skills can boost higher language achievement. limitations this study had a number of anticipated and unanticipated limitations; however, where possible, steps were taken to control or prevent them. though the students were randomly selected, by coincidence there were groups which had students with special needs. to control this limitation, simple main effect analysis was run to determine the mean difference between groups of students who scored at high, average and low levels on the pretest. another limitation of the study was out-of-class ipad use. since this study was based on in-class teaching and learning process, it did not consider out-of-class ipad use. however, this limitation could only be partially controlled by simply banning the language apps planned for in-class use from out-of-class use through the guided http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 21 (page number not for citation purpose) access code control. by setting those codes on students’ ipads, teachers stopped students from using their ipads for language learning unless the codes were changed. a further limitation that was anticipated was the communication and information exchange between the students during the breaks. since all four groups under experiment were in the same college and students shared the same eating and resting areas, they met and communicated. to partially control this limitation, students were informed about the experiment, asked to assist in conducting it and signed a consent form. conclusion today’s language classroom is undergoing an irreversible revolution and one of the most powerful drivers of this transformation is information and communication technologies. according to ouchi and segal (2003), ‘revolutionary change requires the perception that there is a crisis’ (p. 246). some teachers would appear to have doubts about reforms as they fear the chaos that innovations might bring. moreover, they are afraid that the innovative change in the curriculum could grow into their idea of hell. however, today’s students are ‘digital natives’ and today’s teachers need to listen to the kids they teach (prensky 2012, p. 105). this study provided evidence of the kind of learning that has a positive impact on language learners’ attitudes towards the paperless method. ipad-based language learning tasks and assignments were a useful way to spark learners’ interest, motivation and enthusiasm, and the interactive environment could make it easy for the students to settle down, concentrate and do their best in learning a foreign language. the students who are given freedom and choice to explore and create in whatever academic ways they want to as part of their learning coursework are far more willing and motivated than those who do not have this freedom. the pleasure of observing, grading, evaluating and giving feedback on these efforts must be extremely rewarding for both teachers and students. teachers are encouraged to give students as much latitude and support as possible for their creative efforts, continuously setting the bar higher and higher and ensuring all efforts get shared with the rest of the class. the medium of instruction in uae higher education degree programmes is english. the medium of instruction in the uae public primary and secondary programmes, on the other hand, is arabic. since english is taught as a foreign language, this makes it difficult and extremely challenging for the uae high school graduates to study degree programmes delivered through this medium. therefore, this study was designed and conducted as a response to the need to improve emirati language learners’ english proficiency and prepare them for undergraduate education delivered through english. according to gitsaki, robby, and bourini (2014), ‘often the problem is addressed by having high-school graduates attend long post-secondary academic bridging courses in preparation for higher education, but this reduces student motivation’. (p. 168). as jukes, mccain, and crockett (2010) mention, ‘children today are different’ (p. 20). based on the observations of these scholars, it must be mentioned that children are different in the ways they think, process or view the world. these differences are the influences of the digital world they live in today and what is more is that it holds insightful implications for teachers personally and professionally. the students’ experience away from school is highly visual. it is the world of online information which does not involve traditional reading, writing or even traditional ways of thinking. today’s language learners in the class are not the readers and writers the system is http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 22 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) designed for, nor are they the readers and writers most teachers have been trained to teach. above all, they are equipped with 21st century skills that enable them to process audio and visual information more effectively than traditional texts. what is more is that they are used to getting this information in an interactive environment where they are given ample opportunity to regulate it to serve their needs and experience. as said by jukes, mccain, and crockett (2010), ‘asking today’s students to sit while teachers talk or to do the traditional reading of long passages of uninterrupted text is like trying to fit a round peg in a square hole’ (p. 122). in order to motivate students for learning a foreign language, it is necessary to make them feel that the way they are acquiring it is relevant to the online visual world that awaits them when the language course is completed. according to the findings of this study, students’ motivation could be boosted by providing them with ipad-based language learning, which will assist in shorter term language progression and be more effective than the traditional paperand-pen methods. hence, long post-secondary academic bridging programmes could easily be replaced by short post-secondary paperless academic bridging programmes. references bandura, a. (1999) self-efficacy in changing societies, cambridge university press, new york. bandura, a. (2002) ‘social cognitive theory in cultural context’, applied psychology: an international review, vol. 51, no. 2, pp. 269–290. boczkowski, p. j. (2010) ‘the mutual shaping of technology and society in videotex newspapers: beyond the diffusion and social shaping perspectives’, the information society, vol. 20, no. 2, pp. 225–267. chiu, c., hsu, m. & wang, e. (2006) ‘understanding knowledge sharing in virtual communities: an integration of social capital and social cognitive theories’, science direct, vol. 42, no. 1, pp. 1872–1888. churchill, d. & wang, t. (2014) ‘teacher’s use of ipads in higher education’, educational media international, vol. 51, no. 3, pp. 214–225. cooke, r. & french, d. p. (2008) ‘how well do the theory of reasoned action and theory of planned behaviour predict intentions and attendance at screening programs? a meta-analysis’, psychology and health, [online] available at: http://www.tandfonline.com/loi/gpsh20 corbin, j. & strauss, a. (2008) ‘basics of qualitative research. techniques and procedures for developing grounded theory’. los angeles: sage dan, d. (2002) ‘the paperless classroom: e-filing and e-valuating students’ work in english composition’, teaching english in the two-year college, vol. 30, no. 2, pp. 162–178. doane, a. n., pearson, m. r. & kelley, m. l. (2014) ‘predictors of cyberbullying perpetration among college students: an application of the theory of reasoned action’, computers in human behaviour, vol. 36, no. 1, pp. 154–162. engestrom, y. (1999) perspectives on activity theory, google books, cambridge university press, cambridge enriquez, a. (2010) ‘enhancing student performance using tablet computers’, college teaching, vol. 58, no. 3, pp. 77–84. fishbein, m. & ajzen, i. (2010) predicting and changing behaviour, the reasoned action approach, taylor and francis group, new york. flick, u. (2006) the sage dictionary of social research methods, sage, london. ghezzi, a., rangone, a. & balocco, r. (2013) ‘technology diffusion theory revisited: a regulation, environment, strategy, technology model for technology activation analysis of mobile ict’, technology analysis & strategic management, vol. 25, no. 10, pp. 1223–1249. gitsaki, c. & robby, m. a. (2014) ‘post-secondary students using the ipad to learn english: an impact study’, journal of mobile and blended learning, vol. 6, no. 4, pp. 1–22. http://dx.doi.org/10.25304/rlt.v26.1937 http://www.tandfonline.com/loi/gpsh20 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 23 (page number not for citation purpose) gitsaki, c., et al., (2013) ‘a research agenda for the uae ipad initiative’, gulf perspectives, vol. 10, no. 2, [online] available at: http://the.zu.ac.ae gitsaki, c., robby, m. a. & bourini, a. (2014) ‘preparing emirati students to meet the english  language entry requirements for higher education: a pilot study’, journal of education, business and society, contemporary middle eastern issues, vol. 7, no. 2, pp. 167–184. gong, m., xu, y. & yu, y. (2004) ‘an enhanced technology acceptance model for web-based learning’, journal of information systems education, vol. 15, no. 4, pp. 365–374. hoffmeyer-zlotnik, j. h. p. & warner, u. (2014) harmonizing demographic and socioeconomic variables for cross-national comparative survey research, springer, new york. jukes, i., mccain, t. & crockett, l. (2010) understanding the digital generation, corwin, canada. khanova, j. (2012) moving courses online as a catalyst of pedagogical innovation: an activity theory based view, school of information and library science, university of north carolina, us lai, c., shum, m. & tian, y. (2014) ‘enhancing learners’ self-directed use of technology for language learning: the effectiveness of an online training platform’, computer assisted language learning, vol. 29, no. 1, pp. 40–60. lamb, d. (2013) ‘promoting the case for using a research journal to document and reflect on the research experience’, electronic journal of business research methods, vol. 11, no. 2, pp. 84–92. liaw, s. & huang, h. (2014) ‘investigating learner attitudes toward e-books as learning tools: based on the activity theory approach’, interactive learning environments, vol. 10, no. 9, pp. 10–80. limayem, m., hirt, s. g. & cheung, c. m. k. (2007) ‘how habit limits the predictive power of intention: the case of information systems continuance’, mis quarterly, vol. 31, no. 4, pp. 705–737. meder, a. m. & wegner, j. r. (2015) ‘ipads, mobile technologies, and communication applications: a survey of family wants, needs, and preferences’, augmentative and alternative communication, vol. 31, no. 1, pp. 27–36. mishra, d., akman, i. & mishra, a. (2014) ‘theory of reasoned action application for green information technology acceptance’, computers in human behaviour, vol. 36, no. 1, pp. 29–40. ouchi, w. g. & segal, l. g. (2003) making schools work, simon & schuster, new york. prensky, m. (2001) ‘digital natives, digital immigrants’, ncb university press, vol. 9, no. 5, pp. 1–6. prensky, m. (2012) teaching digital natives: partnering for real learning, corwin, thousand oaks, ca, california. robson, c. (2005) real world research, blackwell australia. rogers, e. m. (2003) diffusion of innovation, 5th edn, free press, new york. roth, w. m., radford, l. & lacroix, l. (2012) ‘working with cultural-historical activity theory’, qualitative social research, vol. 13, no. 2, pp. 1–15. swearer, s. m., et al., (2014) ‘reducing bullying: application of social cognitive theory’, theory into practice, vol. 53, no. 1, pp. 271–277. teo, t. (2009) ‘the impact of subjective norm and facilitating conditions on pre-service teachers’ attitude toward computer use: a structural equation modeling of an extended technology acceptance model’, educational computing research, vol. 40, no. 1, pp. 186–210. thomas, m. (2011) deconstructing digital natives, routledge, new york. tsai, m., chen, k. & chien, j. . . (2012) ‘the factors impact of knowledge sharing intentions: the theory of reasoned action perspective’, qualitative and quantitative, vol. 41, no.  1, pp. 1479–1491. venkatesh, v. & davis, f. d. (2000) ‘a theoretical extension of the technology acceptance model: four longitudinal field studies’, management science, vol. 46, no. 2, pp. 89–109. http://dx.doi.org/10.25304/rlt.v26.1937 http://the.zu.ac.ae t. grigoryan 24 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) appendix b. reliability statistics. appendix a. weekly journal. week _____________unit ___________________ learning outcomes _______________ reading and writing _____________ speaking and listening _____________ grammar and vocabulary _____________ ipad group 1 ipad group 2 textbook group 1 textbook group 2 day 1 day 2 day 3 day 4 day 5 day 1 day 2 day 3 day 4 day 5 day 1 day 2 day 3 day 4 day 5 day 1 day 2 day 3 day 4 day 5 the teacher’s reflections the teacher’s reflections the teacher’s reflections the teacher’s reflections reliability statistics cronbach's alph a n of items .808 29 kmo and bartlett’s test kaiser-meyer-olkin measure of sampling adequacy. .805 bardett's test of sphericity approx.. chi-square 1712.739 df 406 sig. .000 survey analysis component 1 2 3 4 5 6 7 slfreg01 −.309 slfreg02 .620 −.350 self-regulation 03 recoded .733 slfef01 .782 intlrenv05 .790 intlrenv03 .672 .353 intleenv01 .722 slfef02 −.733 slfef04 .615 .434 intleenv06 .486 .463 .402 easeus02 .351 .510 useflns03 .440 .418 .459 .385 ipdtsk02 .691 .320 satisf01 .814 http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 25 (page number not for citation purpose) confirmatory factor analysis: component 1 2 3 4 5 6 7 intlrenv02 .841 easeus03 .564 .586 intleenv07 .538 .496 .448 useflns02 -.441 -.485 -.337 ipdtsk01 .715 .447 .360 mottv02 .393 .683 .324 motm b .875 lrneffct01 .474 .559 lrkeffct02 .516 .901 ease of use recoded .406 .462 ipad task 3 recoded .430 .482 .393 satisfactionj recoded -.494 -.669 satisfaction 2 recoded .820 motivation 1 recoded .304 .747 learning effectiveness recoded .774 extraction method: principal component analysis. rotation method: varimax with kaiser normalisation. arotation converged in 13 iterations. survey analysis (continued) factor name: self-regulation component matrixa component 1 slfreg01 .378 slfreg02 .704 self-regulation 03 receded .753 extraction method: principal component analysis. a. 1 components extracted. factor name: self-efficacy component matrixa component 1 slfef01 .828 slfef04 .843 sifef02r .794 extraction method: principal component analysis. a. 1 components extracted. http://dx.doi.org/10.25304/rlt.v26.1937 t. grigoryan 26 citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) factor name: interactive learning component matrixa component 1 intlrenv05 .688 intlrenv03 .677 intlrenv01 .608 intlrenv06 .812 intlrenv02 .718 intlrenv07 .702 extraction method: principal component analysis. a. 1 components extracted. factor name: ease of use component matrix9 component 1 easeus02 .490 easeus03 .821 ease of use recoded .715 extraction method: principal component anarysis. a. 1 components extracted. factor name: ipad based tasks coinmunalities extraction satisf01 .809 satisf02 .055 satisf03 .821 extraction method: principal component analysis. factor name: perceived usefulness component matrixa component 1 ipdtsk02 .759 ipad task 3 recoded .780 ipdtsk01 .786 extraction method: principal component analysis. a. 1 components extracted. http://dx.doi.org/10.25304/rlt.v26.1937 research in learning technology citation: research in learning technology 2018, 26: 1937 http://dx.doi.org/10.25304/rlt.v26.1937 27 (page number not for citation purpose) factor name: perceived satisfaction component matrixa component 1 useflns01 .630 useflns02 .968 usefulness 3 recoded .968 extraction method: principal component analysis. a. 1 components extracted. factor name: motivation component matrix" component 1 motiv02 861 motw03 .857 motivation 1 recoded .692 extraction method: principal component analysis. a. 1 components extracted. factor name: learning effectiveness component matrix 1 component 1 lrneffct0 1 .891 lrneffct0 2 .790 learning effecitveness recoded .790 extraction method: principal component analysis. a. 1 components extracted. http://dx.doi.org/10.25304/rlt.v26.1937 (page number not for citation purpose) 1 *corresponding author. email: pauline.muljana@gmail.com research in learning technology 2023. © 2023 p.s. muljana and t. luo. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 research in learning technology vol. 31, 2023 original research article pursuing professional learning by using social media: how do instructional designers apply self-regulated learning? pauline salim muljanaa* and tian luob ajames r. watson & judy rodriguez watson college of education, california state university, san bernardino, ca, usa; bdarden college of education and professional studies, old dominion university, va, usa (received: 21 october 2022; revised: 31 march 2023; accepted: 8 may 2023; published: 19 june 2023) the instructional design and technology field is dynamic, requiring instructional designers to stay abreast through timely professional learning. social media offers characteristics to collapse the time, geographical, and financial limitations of informal professional learning, but challenges exist. continuous professional learning requires proactive actions, wherein self-regulated learning (srl) plays an important role. however, not all professionals know the effective strategies to promote srl skills. this study examines instructional designers’ (n = 17) experiences of professional learning on social media through an srl lens. data collected through semi-structured interviews were analyzed using thematic analysis. findings include srl strategies conducted by instructional designers and the challenges they faced. this study informs instructional-design professionals and educators of ways to encourage continuous professional learning using social media while fostering srl simultaneously. keywords: instructional designer; professional learning; self-regulated learning; social media introduction instructional design is ‘a system of procedures for developing education and training materials in a consistent and reliable fashion’ (branch, 2018, p. 23). instructional designers are typically responsible for designing and developing instructions and learning materials and utilizing technology to enhance learning, among others (jaramillo cherrez, 2021). they may work in higher education (e.g. colleges/universities), school, military, government, corporate, and non-profit settings (richey et al., 2011). the instructional design and technology field are dynamic, requiring instructional designers to stay abreast with the field (sharif & cho, 2015; wang et al., 2021) and conduct continuous professional learning. on the one hand, pursuing continuous professional learning requires proactive actions, strategic goal setting, and planning, wherein self-regulated learning (srl) plays an important role. unfortunately, not all professionals know the strategies to foster srl skills (siadaty et al., 2016a, 2016b). mailto:pauline.muljana@gmail.com http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2934 p.s. muljana and t. luo 2 citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 (page number not for citation purpose) on the other hand, instructional designers may encounter constraints when pursuing just-in-time learning (e.g., time, geographical, and budget limitations) (muljana et al., 2020, 2021). the omnipresent social-media technologies offer affordances to collapse such constraints, allowing them to reach out to remote colleagues and search for timely information and resources (muljana et al., 2021). however, social media may cause a distraction (dabbagh & kitsantas, 2013). additionally, there remains a question about translating information gained from social media into practice (eaton & pasquni, 2020). being on social media poses challenges as people may encounter abundant information, rude comments, among others (carpenter & harvey, 2019; dabbagh & kitsantas, 2013; dabbagh et al., 2015). this study was purported to address the aforementioned problems and explore instructional designers’ srl experiences conducting professional learning in social media and the challenges. two research questions guided this study: (1) how did instructional designers describe their srl experiences of conducting professional learning in social media environments? (2) what challenges did instructional designers experience when conducting professional learning using social media? literature review instructional designers and professional learning instructional designers need to pursue continuous professional learning to maintain job competencies (ritzhaupt & kumar, 2015). their work may extend beyond creating learning interventions, artifacts, and instructions (tracey & boling, 2014). they help enhance organizational performance by providing training opportunities to the employees; therefore, their work influences the knowledge and skill improvement of targeted learners (tracey & morrison, 2018). instructional designers often work with subject matter experts (smes) to convey information and select appropriate strategies to address learning goals (bratton-jeffery, 2018; litfield, 2018). when courses were converted into a remote mode to respond to the covid-19 pandemic, they assisted instructors and served as ‘first responders’ (abramenka-lachheb et al., 2021, p. 294). instructional designers sometimes encounter time pressure when the project requests increase, and the deadlines are tight. some projects may involve unfamiliar topics wherein they may have a knowledge gap. therefore, they need flexible learning opportunities to seek ready-to-apply information, enabled through an informal learning format, as it is not bounded by curriculum and timeline (carliner, 2018). self-regulating professional learning continuous professional learning requires proactive actions, wherein srl plays a crucial role. notably, its role in working professionals’ career growth has been perceived as imperative (littlejohn, 2017; littlejohn et al., 2012; siadaty et al., 2012, 2016a, 2016b). if they understand the value and relevance of acquiring up-to-date information and ensuring successful job performance, they are more likely to connect and interact with peers and collective knowledge (littlejohn, 2017; littlejohn & hood, 2016). there is an awareness among working professionals that learning the strategies from others who have been in similar experiences can be effective, and these strategies are likely to work in a relevant context (littlejohn, 2017). however, not all working professionals have the skills to initiate and regulate their professional learning (siadaty et al., 2016a, 2016b). most of the srl-related literature tends to focus on the http://dx.doi.org/10.25304/rlt.v31.2934 research in learning technology citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 3 (page number not for citation purpose) formal educational setting (dabbagh & kitsantas, 2012; järvelä et al., 2015; laru & järvelä, 2015; matzat & vrieling, 2016). research exploring the application and support of srl for working professionals, such as instructional designers, in the informal learning context is much needed to expand the literature (siadaty et al., 2016a, 2016b). zimmerman’s srl model is one of the most commonly cited srl frameworks, encompassing and organized into three distinct phases. while it is cyclical in nature, it allows us to look deeply into the three phases and examine the strategies within each phase in a granular manner. zimmerman’s srl model is employed as the theoretical framework for this study. srl is a self-directive learning process involving learners’ thoughts, behaviors, and affects to achieve their goals purposely and strategically in improving learning (zimmerman, 2002). learners applying srl initiate their proactive learning process by determining the sources of motivation, implementing goal-driven strategies and strategy monitoring, including help seeking, and adjusting these strategies to improve learning (bembenutty, 2011). zimmerman’s model consists of three recursive phases: (1) forethought phase: learners set a goal and identify the sources of motivation, (2) performance phase: learners determine, execute, and monitor their learning strategies; and (3) self-reflection phase: learners self-evaluate their performance and plan to adopt and/or adapt a better strategy for improving learning. in the professional learning context, the forethought phase entails thought processes wherein learners familiarize themselves by conducting task analysis and setting goals and strategic plans (siadaty et al., 2012, 2016a, 2016b). the performance phase is when learners conduct the task while executing suitable strategies, monitoring, and revising the strategies (siadaty et al., 2012, 2016a, 2016b). during the self-reflection phase, learners actively evaluate their progress while reflecting upon and sharing their own experiences (siadaty et al., 2012, 2016a, 2016b). social-media supported professional learning instructional designers may encounter constraints when pursuing professional learning opportunities, such as funding, time, and travel restrictions (muljana et al., 2020, 2021). social media can accommodate the urgency of pursuing just-intime learning without being limited by such barriers. when instructional designers need to learn new strategies (i.e., using the newly updated authoring software or creating a learning intervention for a discipline that is not in their expertise) or encounter design constraints, they reach out to colleagues in their networks for advice (muljana et al., 2021). in addition to seeking and processing knowledge, professionals share information and resources, contributing to the knowledge-building and practice-building community (gruzd et al., 2016; littlejohn, 2017; milligan et al., 2014). the definition of social media adopted for this study is ‘internet-based tools that allow users to build networks through which they may then communicate and share information with each other’ (dennen, 2018, p. 237). this definition indicates that social media tools are beyond social networking sites like facebook and twitter. luo et al. (2022) have provided various social media examples. for instance, blogs allow users to publish text-based content, including images and/or videos. youtube allows users to upload and share video, including educational content, subscribe to the producer’s channel, and watch videos. put simply, social media includes expansive genres such as media sharing sites, social networking sites, discussion forums, social bookmarking, blogging, and messaging (luo et al., 2022; ottenbreit-leftwich & brush, 2018). this study utilizes this definition and classification of social media, including http://dx.doi.org/10.25304/rlt.v31.2934 p.s. muljana and t. luo 4 citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 (page number not for citation purpose) various social technologies, that working professionals (e.g., instructional designers) use for professional purposes. despite the benefits of using social media for professional learning, there exist challenges. social media allows users to interact without meeting in person or knowing each other personally, providing an outlet for freedom to express insights and comments. shaming others and cyberbullying may occur in this environment, in addition to concerns about cyber safety and privacy issues (dabbagh et al., 2015). also, it can be challenging to balance the use of social media for personal and professional purposes (carpenter & harvey, 2019). because of the potential misand dis-information, professionals may not want to simply absorb the information on social media (greenhalgh et al., 2021). nevertheless, the practices of pursuing professional learning using social media deserve an in-depth investigation (luo et al., 2020). method we employed phenomenology to acquire insights based on the participants’ experiences (goulding, 2005) and gain an in-depth comprehension of the phenomenon under study (welman & kruger, 2001). we recruited participants from facebook and linkedin groups joined by instructional designers. the call for research participants included a study description, ethical considerations, consent form, eligibility questionnaire, and interview sign-up form. the questionnaire specifically asked whether they were working professionals holding instructional-design related responsibilities for at least one year and used social media for professional purposes. seventeen instructional designers from various sectors (e.g., corporate, government, self-employed, and higher education) participated in this study (table 1). they had various job titles (e.g., instructional designer, curriculum specialist, learning designer, and learning analyst) and were on different career levels. these diverse backgrounds served as a triangulation because we could contrast data with different meanings and validate those with similar meanings (arksey & knight, 1999; bloor, 1997; groenewald, 2004; holloway, 1997). we used semi-structured interviews to acquire experiences from the participants. we conducted the interviews conversationally to encourage detailed responses (smith et al., 2013). all interviews were recorded and transcribed. all participants member-checked the transcript and the interviewer’s notes. one question asked in the interview was related to typical tasks, and the participants reported similarly, such as: • analyzing performance gaps • identifying learning objectives • performing needs analysis • designing courses or learning solutions and performing design iterations • developing learning solutions by using authoring tools (some solutions might be in non-learning formats, such as job aid) • coaching others (e.g., teaching courses and providing consultation and training to instructors) • collaborating with smes and other design team members • managing projects http://dx.doi.org/10.25304/rlt.v31.2934 research in learning technology citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 5 (page number not for citation purpose) • assisting in the implementation • conducting evaluation. for data analysis, we employed thematic analysis (ta) (braun & clarke, 2006), consisting of six phases: (1) familiarizing with data; (2) generating initial codes – we employed deductive coding to address the first research question (rq1) using zimmerman’s (2002) three srl phases as coding scheme and inductive coding to address the second research question (rq2); (3) searching for themes; (4) reviewing themes; (5) defining themes; and (6) producing the report. when analyzing data, phenomenological researchers should consider the context under study as a whole (hycner, 1985). performing ta helped us follow hycner’s suggestion because ta provided guidelines for ‘analyzing meaning across the entire data set’ (braun & clarke, 2012, p. 58). results rq1: instructional designers describing their srl experiences forethought phase table 2 shows instructional designers’ activities during the forethought phase. sources of motivation. ning noted the value of acquiring multiple perspectives: ‘it’s really healthy to have different perspectives beyond your workplace co-workers’. table 1. participants’ information. pseudonym academic degree sector years of experience abigail master’s self-employed 11–15 years angela master’s, doctoral coursework higher education (he) 6–10 years ari master’s he 0–5 years emilio master’s he 6–10 years gwen master’s, doctoral coursework he 0–5 years jin master’s he 6–10 years jovina master’s he 6–10 years kelsey master’s, doctoral coursework corporate 6–10 years kiara bachelor he 0–5 years ning master’s corporate 0–5 years pam doctoral self-employed 6–10 years payton doctoral he 6–10 years prisca master’s corporate 6–10 years pru doctoral he, government, self-employed 16–20 years sari master’s, post-master certificate corporate 6–10 years tamara master’s he 6–10 years whitney master’s non-profit 6–10 years http://dx.doi.org/10.25304/rlt.v31.2934 p.s. muljana and t. luo 6 citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 (page number not for citation purpose) tamara believed it could increase her self-efficacy, which ‘helped me [become] more confident, then could play into my own performance and work’. the participants seemed to perform a metacognitive process to understand where they were and wanted to be regarding their performance quality. proximal goals: long-term and short-term goals. for a long-term goal, kelsey aspired to be a future learning and development (l&d) leader. therefore, she created podcast episodes to engage in relevant conversations about l&d. shortterm goals were related to what they needed to achieve at that moment. when abigail was learning to use articulate, she would reach out to colleagues in professional communities to ‘see if someone has had the same challenge, and how they’ve solved it’. task analysis. when assigned with a project, instructional designers analyzed  the  steps required to complete the project. when ari was new at work, he never  said, ‘i didn’t know’ how to complete a project. he accessed linkedin learning to learn the required tasks or steps or fill in any knowledge gaps. kiara  shared  an experience regarding creating learning materials for a business course; she ‘had to go to youtube and watch the videos […] for understanding [a new topic]’. strategic planning. participants’ strategic planning included continuous professional learning and searching for non-traditional learning resources using social media. tamara engaged with colleagues on linkedin to find mentors. angela actively searched for innovative strategies for facilitating online learning conducted by others. table 2. participants’ forethought phase experiences and number of mentions. pseudonym sources of motivation proximal goals: longterm goals proximal goals: shortterm goals task analysis strategic planning abigail • • • • angela • • • ari • • • emilio • gwen • • • jin • • • • • jovina • • • • kelsey • • • • kiara • • ning • • • • • pam • • payton • prisca • • • • • pru • • • sari • tamara • • • whitney • • • number of mentions 23 9 89 15 16 note: the symbol • shows that the respective participant’s experience supported the theme. http://dx.doi.org/10.25304/rlt.v31.2934 research in learning technology citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 7 (page number not for citation purpose) performance phase table 3 lists instructional designers’ activities conducted during the performance phase. help-seeking. instructional designers sought help using social media when they needed to find ideas for better design strategies. tamara said, ‘[…] it can feel sometimes isolating. so, having others out there is helpful’. sari saw people who ‘were stuck for ideas, and they were able to get help with ideas, such as how to reformat their resume, interview help, […] creative solutions to develop e-learning or ideas for instructor-led training […]’. applying new tips and adjusting current practices. learning about ‘what’s worked well’ for their peers could assist instructional designers in completing their tasks successfully. kiara applied the tips suggested by colleagues in social media communities and said, ‘just a few tips that people give, and […] you just put them in your instructional design, and […] it works’. she believed her work quality had increased; her design team did not ask her to clarify her storyboards anymore. learning and engaging with colleagues on social media influenced participants’ current practices. it inspired them to either tweak or change their practices. abigail noted, ‘[…] when i realized if there’s quick […] shortcuts […] in storyline […]; those are things that i end up applying’. in other words, abigail ‘tweaked’ her design strategy. self-reflection phase table 4 shows instructional designers’ activities performed during the self-reflection phase. table 3. participants’ performance phase experiences and number of mentions. pseudonym seeking help from colleagues trying and applying tips suggested by colleagues adjusting current practices abigail • • • angela • ari • emilio • • gwen • • • jin • • jovina • • • kelsey • • kiara • • ning • • pam • payton • • prisca • • • pru • • • sari • • • tamara • • whitney • • • number of mentions 20 12 25 http://dx.doi.org/10.25304/rlt.v31.2934 p.s. muljana and t. luo 8 citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 (page number not for citation purpose) open-mindedness to orient self-reflection. instructional designers noted that being open-minded could help them reflect on their performance. ning reported, ‘[…] the way that i’ve done things in the past is not always the most effective way[,]’ and therefore, she kept an open mindset for better design strategies. reflection on the information. participants reflected on the information gained from social media. they checked the information accuracy first. for example, pru used her ‘own diagnostic to see what makes the most sense’. next, they selected and combined the information based on their contexts and needs. jin synthesized information from an ‘infographic found on a linkedin post and the knowledge […] learned from linkedin learning’. there appears to be an application of critical thinking in this regard. sharing learning experiences. instructional designers reported that sharing their learning experiences and resources was an outcome of their self-reflection. for instance, abigail created youtube videos to share helpful tips regarding challenges she had faced in the past in hopes of helping other instructional designers to overcome similar challenges. this sharing activity appears to involve a reflection on which design strategies worked and which did not. rq2: challenges faced by instructional designers table 5 lists the challenges reported by instructional designers. table 4. participants’ self-reflection experiences and number of mentions. pseudonym open-mindedness reflecting on the information (consisting of three sub-themes) sharing learning experience 1. checking the information 2. selecting the information 3. reacting to the information abigail • • • • • angela • • • ari • • emilio • • • gwen • • • • jin • • • • jovina • • kelsey • • • kiara • ning • • • pam • • • payton • • • • prisca • • pru • • sari • • tamara • • • whitney • • • • number of mentions 14 20 22 11 9 http://dx.doi.org/10.25304/rlt.v31.2934 research in learning technology citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 9 (page number not for citation purpose) information overload most participants encountered irrelevant and inaccurate information on social media. gwen remembered receiving poor advice. ning noted, ‘there are also people who don’t know what they’re talking about’. ethical and privacy concerns instructional designers had concerns regarding ethics and privacy. ensuring that everyone had appropriate netiquette regarding using social media was difficult. kelsey had met ‘haters, […] disrespectful people […] who just want to argue’. additionally, they were worried about their activities being tracked on social media. emilio said, ‘our actions are tracked beyond for the purposes of selling us more things […]. some seem to be fairly harmless; some seem to be nefarious purposes […]’. additional challenges those located outside the us sometimes experienced unreliable internet. kiara lost the internet connection during the interview. a few instructional designers had a fear of judgment when posting questions. they worried colleagues might misinterpret their knowledge and skills. prisca stated, ‘i don’t want to appear stupid […] asking my question’. they had a fear of judgment or self-confident issues when deciding to post questions or comments, possibly because they had seen rude comments and snarky remarks in argumentative discussions. table 5. participants’ challenges and number of mentions. pseudonym information overload ethical and privacy concerns issues encountered individually abigail • • • angela • ari emilio • • • gwen • • jin • jovina • kelsey • • kiara • • ning • pam • payton • prisca • • • pru • • • sari • • tamara • whitney • • number of mentions 24 14 12 http://dx.doi.org/10.25304/rlt.v31.2934 p.s. muljana and t. luo 10 citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 (page number not for citation purpose) discussion and implications our findings suggest that instructional designers strategically utilize social media in srl phases with an emphasis on having motivation as a foundation of the forethought activities, modifying current strategies during the performance phase, and being open minded to prepare for the self-reflection phase. they also acknowledge the challenges, especially in seeing overwhelming information and being afraid of asking questions. forethought activities include task analysis, goal setting, and plan making that require motivation (zimmerman, 2002). our participants report similar activities, especially in the motivation aspect. when they need to improve skills or acquire ideas on design strategies, they are motivated to set goals and make plans. even when they do not perceive an urgent need, they proactively acquire new knowledge because they want to be prepared for potential future projects requiring innovative strategy. according to the literature, working professionals seek learning opportunities to keep up with job demands, stay abreast with the profession, and ensure successful job performance (littlejohn, 2017). therefore, it is expected that participants in this study have a short-term goal behind their social media usage (e.g., to help accomplish a current task). they follow up on their short-term goal(s) through strategic planning by participating in virtual professional development events that offer flexibility and by learning from colleagues’ strategies – all are mostly feasible through social media. during the performance phase, the instructional designers try and apply the tips suggested by colleagues and modify the current practice. they seem willing to adapt their current strategies for a smoother design process. such a strategy-adaptation process may begin with help-seeking, involving awareness of their knowledge and how colleagues’ knowledge serves as a learning source (milligan & littlejohn, 2016). this is coherent with prior literature, suggesting that self-regulatory learners tend to engage in just-in-time help-seeking activities such as searching previous conversations or posting a question to crowdsource ideas from peers. after learning from colleagues, our participants adjust the design methodology and adopt a more efficient use of authoring software. these experiences suggest that self-regulatory instructional designers take charge of the strategies to try, apply, monitor, and adjust, resonating with siadaty et al. (2016a) and zimmerman (2002). the self-reflection phase occurs when learners evaluate their performance, reflect upon their learning experiences, and plan to adopt or adapt a better strategy for improvement (siadaty et al., 2016a; zimmerman, 2000). this phase involves causal attribution – learners’ beliefs regarding the cause of a failure or success (zimmerman, 2002). learners with srl skills tend to attribute failure to an unsuitable strategy instead of a fixed ability and are likely to adjust the strategy (zimmerman, 2002). instructional designers under study believe that being open-minded enhances their self-awareness about their performance. without exercising open-mindedness, there may be barriers to the self-reflection phase, potentially leading to inappropriate causal attributions. instructional designers face challenges such as being overloaded with abundant information and resources. because they are goal driven and motivated, they focus their activities on learning goals and are likely to select relevant information and resources. also, self-efficacy plays an important role, especially for those afraid to ask. self-efficacy, a judgment of one’s abilities to perform a task, is associated with prior experience and task familiarity (zimmerman, 2000). new instructional designers can http://dx.doi.org/10.25304/rlt.v31.2934 research in learning technology citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 11 (page number not for citation purpose) reflect on their prior knowledge, skills, and abilities and focus on amplifying these transferable strengths and increasing self-efficacy. also, they may use social-media mediated communities (e.g., reddit and facebook groups) that offer anonymity to ask questions without identifying themselves. exercising critical thinking is an example of fostering srl leading to positive outcomes in online learning (broadbent & poon, 2015) and helpful for considering multiple perspectives in collaborative learning (cascolan, 2019). the characteristics of social media environments resonate with online and collaborative learning. when instructional designers encounter an influx of information on social media, including misand dis-information, they can exercise critical thinking to check the accuracy and credibility of the source and select relevant information. they may assess whether the information aligns with their goals, contexts, and design philosophy. knowledge of productivity software, soft skills (e.g., time management, persistence, collaboration, communication), and ability to work independently and collaboratively frequently appear in instructional-design related job announcements (north et al., 2021; wang et al., 2021). conducting lifelong learning, taking ownership of professional development, having self-motivation and self-directive learning initiative, and performing self-reflection are aligned with qualifications stated by professional organizations’ standards (atd, 2020; piña, 2017). these qualifications highlight essential keywords related to srl. educators guiding future instructional designers may want to foster such qualifications and infuse srl guidance into learning activities. learners can be encouraged to stay focused on the project goal, determine the strategic steps to accomplish the goal, increase team productivity, and explore productivity management techniques. it is worth adding individual reflective assignments, allowing them to self-evaluate their performance and contribution to the team project. future research and conclusion future researchers may consider involving multiple data sources beyond self-report instrumentations, such as digital traces (e.g., posts, comments, and reactions). employing digital traces allows researchers to explore behaviors on social media in a natural manner (lee et al., 2017). the digital traces can be analyzed to uncover how srl application occurs in social-media supported professional learning, current learning needs, and trending topics. the topic of open-mindedness appears in the results related to the self-reflection phase. future research may explore where and how open-mindedness may occur in an srl cycle and how it influences the self-reflection phase. social media potentially enables flexible professional learning, which can further enhance learning when srl strategies are applied. instructional designers share their experiences applying srl strategies and being motivated to stay abreast. they adapt and tweak current design practices, highlighting their open mindedness. although the challenges exist, they may overcome them through srl strategies. competing interests the authors declare that they have no conflict of interests. http://dx.doi.org/10.25304/rlt.v31.2934 p.s. muljana and t. luo 12 citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 (page number not for citation purpose) acknowledgments this work is based on a doctoral dissertation by pauline s. muljana. the guidance, feedback, and contributions of the dissertation committee chair (dr. tian luo), dissertation committee members (drs. john baaki and tony perez), and dissertation external reviewer and copyeditor (drs. hannah digges elliott and kim bullington) are gratefully acknowledged. also, the first author would like to thank the california state university chancellor’s doctoral incentive program for awarding mini grants, used as material support. finally, shout out to the instructional-design colleagues for staying active in the field. ethical considerations the old dominion university education human subjects review committee reviewed and approved this study according to federal regulations. references abramenka-lachheb, v. et al. (2021). instructional designers’ use of informal learning: how can we all support each other in times of crisis? the journal of applied instructional design, 10(3). doi: 10.51869/103/valaljlrsgs arksey, h. & knight, p. (1999). interviewing for social scientists. thousand oaks, ca, usa: sage. association for talent development (atd). (2020). talent development capability model: what talent development professionals should know and do to be successful. retrieved from https://d22bbllmj4tvv8.cloudfront.net/18/5b/1142b292431fb5393f2193211e1b/talent-developmentcapability-model-definitions.pdf bembenutty, h. (2011). meaningful and maladaptive homework practices: the role of self-efficacy and self-regulation. journal of advanced academics, 22(3), 448–473. doi: 10.1177/1932202x1102200304 bloor, m. (1997). techniques of validation in qualitative research: a critical commentary. in, millar, g. & dingwall, r. (eds.). context and method in qualitative research. thousand oaks, ca, usa: sage, 37–50. branch, r. m. (2018). instructional design opportunities in military education and training environments. in, reiser, r. a. & dempsey, j. v. (eds.). trends and issues in instructional design and technology. 4th edn. new york, ny, usa: pearson, 23–30. bratton-jeffery, m. f. (2018). instructional design opportunities in military education and training environments. in, reiser, r. a. & dempsey, j. v. (eds.). trends and issues in instructional design and technology. 4th edn. new york, ny, usa: pearson, 159–167. braun, v. & clarke, v. (2012). thematic analysis. in cooper, h., camic, p. m., long, d. l., panter, a. t., rindskopf, d. & sher, k. j. (eds.). apa handbooks in psychology®. apa handbook of research methods in psychology, vol. 2. research designs: quantitative, qualitative, neuropsychological, and biological. american psychological association, washington dc, 57–71. doi: 10.1037/13620-004 broadbent, j. & poon, w. l. (2015). self-regulated learning strategies & academic achievement in online higher education learning environments: a systematic review. internet and higher education, 27, 1–13. doi: 10.1016/j.iheduc.2015.04.007 carliner, s. (2018). informal learning. in, reiser, r. a. & dempsey, j. v. (eds.). trends and issues in instructional design and technology. 4th edn. new york, ny, usa: pearson, 142–151. carpenter, j. p. & harvey, s. (2019). ‘there’s no referee on social media’: challenges in educator professional social media use. teaching and teacher education, 86, 102904. doi: 10.1016/j. tate.2019.102904 http://dx.doi.org/10.25304/rlt.v31.2934 https://doi.org/10.51869/103/valaljlrsgs https://d22bbllmj4tvv8.cloudfront.net/18/5b/1142b292431fb5393f2193211e1b/talent-developmentcapability-model-definitions.pdf https://d22bbllmj4tvv8.cloudfront.net/18/5b/1142b292431fb5393f2193211e1b/talent-developmentcapability-model-definitions.pdf https://doi.org/10.1177/1932202x1102200304 https://doi.org/10.1037/13620-004 https://doi.org/10.1016/j.iheduc.2015.04.007 https://doi.org/10.1016/j.tate.2019.102904 https://doi.org/10.1016/j.tate.2019.102904 research in learning technology citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 13 (page number not for citation purpose) cascolan, h. m. s. (2019). students’ conceptual understanding, metacognitive awareness and self-regulated learning strategies towards chemistry using pogil approach. asean multidisciplinary research journal, 1(1). retrieved from https://paressu.org/online/index. php/aseanmrj/article/view/172 dabbagh, n. & kitsantas, a. (2012). personal learning environments, social media, and self-regulated learning: a natural formula for connecting formal and informal learning. internet and higher education, 15(1), 3–8. doi: 10.1016/j.iheduc.2011.06.002 dabbagh, n. & kitsantas, a. (2013). the role of social media in self-regulated learning. international journal of web based communities, 9(2), 256–273. doi: 10.1504/ ijwbc.2013.053248 dabbagh, n. et al. (2015). using social media to develop personal learning environments and self-regulated learning skills: a case study. international journal of social media and interactive learning environments, 3(3), 163–183. doi: 10.1504/ijsmile.2015.072300 dennen, v. (2018). social media and instructional design. in, reiser, r. a. & dempsey, j. v. (eds.). trends and issues in instructional design and technology. 4th edn. new york, ny, usa: pearson, 237–243. eaton, p. w. & pasquini, l. a. (2020). networked practices in higher education: a netnography of the #acadv chat community. the internet and higher education, 45, 160–188. doi: 10.1016/j.iheduc.2019.100723 goulding, c. (2005). grounded theory, ethnography and phenomenology: a comparative analysis of three qualitative strategies for marketing research. european journal of marketing, 39(3/4), 294–308. doi: 10.1108/03090560510581782 greenhalgh, s. p., krutka, d. & oltmann, s. (2021). gab, parler, and (mis)educational technologies: reconsidering informal learning on social media platforms. the journal of applied instructional design, 10(3). doi: 10.51869/103/sgdkso groenewald, t. (2004). a phenomenological research design illustrated. international journal of qualitative studies in education, 3(1), 1–26. retrieved from http://www.ualberta.ca/~iiqm/ backissues/3_1/html/groenewald.html gruzd, a., paulin, d. & haythornthwaite, c. (2016). analyzing social media and learning through content and social network analysis: a faceted methodological approach. journal of learning analytics, 3(3), 46–71. doi: 10.18608/jla.2016.33.4 holloway, i. (1997). basic concepts for qualitative research. malden, ma, usa: blackwell science. hycner, r. h. (1985). some guidelines for the phenomenological analysis of interview data. human studies, 8(3), 279–303. doi: 10.1007/bf00142995 jaramillo cherrez, n. (2021). instructional designers leading through research. in, stefaniak, j. e. et al. (eds.). a practitioner’s guide to instructional design in higher education. edtech books. retrieved from https://edtechbooks.org/id_highered/instructional_designv järvelä, s. et al. (2015). enhancing socially shared regulation in collaborative learning groups: designing for cscl regulation tools. educational technology research and development, 63(1), 125–142. doi: 10.1007/s11423-014-9358-1 laru, j. & järvelä, s. (2015). integrated use of multiple social software tools and face-to-face activities to support selfregulated learning: a case study in a higher education context. in, wong, l.-h., mildrad, m. & specht, m. (eds.). seamless learning in the age of mobile connectivity. singapore: springer, 471–484. lee, r. m., fielding, n. g. & blank, g. (2017). online research methods in the social sciences: an editorial introduction. in, fielding, n. g., lee, r. m. & blank, g. (eds.). the sage handbook of online research methods. 2nd edn. los angeles, usa: sage, 3–16. litfield, b. c. (2018). instructional design in higher education. in, reiser, r. a. & dempsey, j. v. (eds.). trends and issues in instructional design and technology. 4th edn. new york, ny, usa: pearson, 185–191. littlejohn, a. & hood, n. (2016). how educators build knowledge and expand their practice: the case of open education resources. british journal of educational technology, 48(2), 499–510. doi: 10.1111/bjet.12438 http://dx.doi.org/10.25304/rlt.v31.2934 https://paressu.org/online/index.php/aseanmrj/article/view/172 https://paressu.org/online/index.php/aseanmrj/article/view/172 https://doi.org/10.1016/j.iheduc.2011.06.002 https://doi.org/10.1504/ijwbc.2013.053248 https://doi.org/10.1504/ijwbc.2013.053248 https://doi.org/10.1504/ijsmile.2015.072300 https://doi.org/10.1016/j.iheduc.2019.100723 https://doi.org/10.1108/03090560510581782 https://doi.org/10.51869/103/sgdkso http://www.ualberta.ca/~iiqm/backissues/3_1/html/groenewald.html http://www.ualberta.ca/~iiqm/backissues/3_1/html/groenewald.html https://doi.org/10.18608/jla.2016.33.4 https://doi.org/10.1007/bf00142995 https://edtechbooks.org/id_highered/instructional_designv https://doi.org/10.1007/s11423-014-9358-1 https://doi.org/10.1111/bjet.12438 p.s. muljana and t. luo 14 citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 (page number not for citation purpose) littlejohn, a., milligan, c. & margaryan, a. (2012). charting collective knowledge: supporting self-regulated learning in the workplace. journal of workplace learning, 24(3), 226–238. doi: 10.1108/13665621211209285 luo, t., freeman, c. & stefaniak, j. (2020). ‘like, comment, and share’ – professional development through social media in higher education: a systematic review. educational technology research and development, 68(4), 1659–1683. doi: 10.1007/s11423-020-09790-5 luo, t. et al. (2022). an investigation of teachers’ perceptions and integration of web 2.0 tools into literacy instruction. international journal of social media and interactive learning environments, 6(4), 305–327. doi: 10.1504/ijsmile.2020.10031666 matzat, u. & vrieling, e. m. (2016). self-regulated learning and social media – a ‘natural alliance’? evidence on students’ self-regulation of learning, social media use, and student–teacher relationship. learning, media and technology, 41(1), 73–99. doi: 10.1080/17439884.2015.1064953 milligan, c. & littlejohn, a. (2016). how health professionals regulate their learning in massive open online courses. the internet and higher education, 31, 113–121. doi: 10.1016/j. iheduc.2016.07.005 milligan, c., littlejohn, a. & margaryan, a. (2014). workplace learning in informal networks. journal of interactive media in education, 1(6). doi: 10.5334/2014-06 muljana, p. s. et al. (2020). promoting instructional designers’ participation in free, asynchronous professional development: a formative evaluation. journal of formative design in learning, 4(2), 74–87. doi: 10.1007/s41686-020-00044-4 muljana, p. s. et al. (2021). free asynchronous professional development by, from, and for instructional designers: how informal learning opportunities shape our professional learning and design practices. the journal of applied instructional design, 10(3). doi: 10.51869/103/pmkjkagplp north, c. et al. (2021). how instructional design is operationalized in various industries for job-seeking learning designers: engaging the talent development capability model. techtrends, 65(5), 713–730. doi: 10.1007/s11528-021-00636-2 ottenbreit-leftwich, a. & brush, t. (2018). integrating technology into k-12 education. in, reiser, r. a. & dempsey, j. v. (eds.). trends and issues in instructional design and technology. 4th edn. new york, ny, usa: pearson, 176–184. piña, a. a. (ed.). (2017). instructional design standards for distance learning. association for educational communications and technology. retrieved from http://members.aect.org/ publications/designstandards/dl_design_standards.pdf richey, r. c., klein, j. d. & tracey, m. w. (2011). the instructional design knowledge base: theory, research and practice. new york, ny: routledge. ritzhaupt, a. d. & kumar, s. (2015). knowledge and skills needed by instructional designers in higher education. performance improvement quarterly, 28(3), 51–69. doi: 10.1002/ piq.21196 sharif, a. & cho, s. (2015). 21st-century instructional designers: bridging the perceptual gaps between identity, practice, impact and professional development. rusc university and knowledge society journal, 12(3), 72–85. doi: 10.7238/rusc.v12i3.2176 siadaty, m. et al. (2012). self-regulated workplace learning: a pedagogical framework and semantic web-based environment. educational technology & society, 15(4), 75–88. siadaty, m., gašević, d. & hatala, m. (2016a). associations between technological scaffolding and micro-level processes of self-regulated learning: a workplace study. computers in human behavior, 55, 1007–1019. doi: 10.1016/j.chb.2015.10.035 siadaty, m., gašević, d. & hatala, m. (2016b). measuring the impact of technological scaffolding interventions on micro-level processes of self-regulated workplace learning. computers in human behavior, 59, 469–482. doi: 10.1016/j.chb.2015.10.035 smith, j. a., flowers, p. & larkin, m. (2013). interpretative phenomenological analysis: theory, method and research. london: sage. tracey, m. w. & boling, e. (2014). preparing instructional designers and educational technologies: traditional and emerging perspectives. in, spector, m. et al. (eds.). handbook http://dx.doi.org/10.25304/rlt.v31.2934 https://doi.org/10.1108/13665621211209285 https://doi.org/10.1007/s11423-020-09790-5 https://doi.org/10.1504/ijsmile.2020.10031666 https://doi.org/10.1080/17439884.2015.1064953 https://doi.org/10.1016/j.iheduc.2016.07.005 https://doi.org/10.1016/j.iheduc.2016.07.005 https://doi.org/10.5334/2014-06 https://doi.org/10.1007/s41686-020-00044-4 https://doi.org/10.51869/103/pmkjkagplp https://doi.org/10.1007/s11528-021-00636-2 http://members.aect.org/publications/designstandards/dl_design_standards.pdf http://members.aect.org/publications/designstandards/dl_design_standards.pdf https://doi.org/10.1002/piq.21196 https://doi.org/10.1002/piq.21196 https://doi.org/10.7238/rusc.v12i3.2176 https://doi.org/10.1016/j.chb.2015.10.035 https://doi.org/10.1016/j.chb.2015.10.035 research in learning technology citation: research in learning technology 2023, 31: 2934 http://dx.doi.org/10.25304/rlt.v31.2934 15 (page number not for citation purpose) of research on educational communications and technology. 2nd edn. new york, usa: springer, 653–660. doi: 10.1007/978-1-4614-3185-5_52 tracey, m. w. & morrison, g. r. (2018). instructional design in business and industry. in, reiser, r. & dempsey, j. (eds.). trends and issues in instructional design and technology. 4th edn. new york, usa: pearson, 152–158. wang, x. et al. (2021). examining competencies for the instructional design professional: an exploratory job announcement analysis. international journal of training and development, 25(2), 95–123. doi: 10.1111/ijtd.12209 welman, j. c. & kruger, s. j. (2001). research methodology for the business and administrative sciences. south africa: oxford university press. zimmerman, b. j. (2000). a social cognitive perspective. in, boekaerts, m., pintrich, p. r. & zeidner, m. (eds.). handbook of self-regulation. new york, ny: academic press, 13–39. doi: 10.1016/b978-012109890-2/50031-7 zimmerman, b. j. (2002). becoming a self-regulated learner: an overview. theory into practice, 41(2), 64–70. doi: 10.1207/s15430421tip4102_2 http://dx.doi.org/10.25304/rlt.v31.2934 https://doi.org/10.1007/978-1-4614-3185-5_52 https://doi.org/10.1111/ijtd.12209 https://doi.org/10.1016/b978-012109890-2/50031-7 https://doi.org/10.1207/s15430421tip4102_2 doi:10.3402/rlt.v24.32630 original research article ‘i can use things, but i can’t make anything’: a qualitative exploration of team networks in the development and implementation of a new undergraduate e-compendium a. latif*, r. windle and h. wharrad school of health sciences, queen’s medical centre, university of nottingham, nottingham, united kingdom (received 17 june 2016; final version received 3 september 2016) in higher education, undergraduate teaching materials are increasingly becoming available online. there is a need to understand the complex processes that happen during their production and how social networks between different groups impact on their development. this paper draws on qualitative interviews and participant drawings of their social networks to understand the dynamics of creating a new e-compendium for a four-year online undergraduate nursing programme in norway. twenty staff interviews were undertaken to explore views of the e-compendium, the development process and the perceived networks that were formed during this course. interview data were thematically analysed along with networks drawings. the findings showed three main institutional stakeholder groups emerging: the ‘management team’, ‘design team’ and ‘lecturers’. analysis of social networks revealed variability of relations both within and between groups. the pedagogical designer, who was part of the design team, was central to communicating with and co-ordinating staff at all levels. the least well connected were the lecturers. to them, the e-compendium challenged and even threatened previously wellestablished notions of pedagogy. future development of e-compendiums should account for the perceived lack of time and existing workload of lecturers so they may be involved with the development process. keywords: e-compendium development; social networks; qualitative study introduction undergraduate online courses and e-compendiums are increasingly becoming commonplace and offer students’ convenience, flexibility and empowerment over their learning (lymn, bath-hextall, and wharrad 2008; wharrad et al. 2001; windle et al. 2011). undergraduate nursing students have reported satisfaction with such online learning materials (mancuso-murphy 2007; korhonen and lammintakanen 2005). academic staff also have broadly positive attitudes and value the pedagogical importance of technology in teaching and learning (blake 2009). however, more detailed investigation reveals that there are likely to be a range of views from those that are advocates to those who are sceptical, hesitant or lack confidence in e-learning teaching developments and delivery (blake 2009; dariel 2011). one reason for this is that lecturers themselves typically undertake the writing, format and presentation of *corresponding author. email: asam.latif@nottingham.ac.uk responsible editor: fiona smart, edinburgh napier university, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 a. latif et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 undergraduate course materials. however, when courses migrate online, many simply do not possess the skills to develop and manufacture e-learning materials. an example of the complexity can be seen in the screenshot from one such e-compendium shown in figure 1. as a result, a team-based stakeholder approach is used to develop the e-materials (alexander 2001; gwozdek et al. 2011). whereas beforehand lecturers worked solely to produce their learning materials, this new team may include flash developers, graphic design, text writers and audio commentators (eseryel and ganesan 2001). this model can be a challenge for lecturers as they make the transition from independence (i.e. autonomy over producing their teaching material) to a collaborative model, where they relinquish the format and presentation of the material so that this can be constructed online (meyen, tangen, and lian 1999). figure 1. example screenshot of an e-compendium. the topic of this particular student exercise is management of fever. the activity is of an interactive nature adding to the complexity of e-compendium. a. latif et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 to avoid potential tensions that can arise between lecturers and those who develop and produce online material, a team-based approach is needed (shephard 2004) as well as a detailed understanding of the social networks that are formed during this process. this study uses stakeholder interviews and illustrations of social networks to understand this process. drawing on a social network analysis (sna) approach (scott 2000) to extend the thematic analysis, the nature and consequences of ties between individuals or groups were investigated. sna has been described by wasserman and faust (1994, p. 3) as focusing on relationships among social entities and on the patterns and implications of these relationships. sna is based on the assumption that relationships have a large impact on influence and decision making and that a map of such relationships will help to identify where networks are resilient or vulnerable. this approach has been used in the social sciences in a variety of settings. for example, in healthcare, sna has been used to explore health care professional relationships and networks (keating et al. 2007; lewis, baeza, and alexander 2008). in education, the method has been used to evaluate programs that aim to improve schools by fostering greater collaboration between teachers (penuel et al. 2006) and understanding how learning relationships form in undergraduate classrooms (grunspan, wiggins, and goodreau 2014). there has been little research investigating the social networks that develop at the time when online course material is being manufactured and developed and the involvement of teaching staff during this process. this study draws upon a ‘real world’ example and aims to provide insight into the shape of these new social networks that are formed during the creation of online resources. methods this study took place in a university in norway as part of an evaluation of a new e-compendium for an online undergraduate nursing programme. the project was undertaken by independent evaluators from the university of nottingham (hw & rw). all university staff members who were involved in the project to develop the multimedia pdf-based online e-compendium were invited to face-to-face interviews. seventeen interviews were held in preikestolen during a staff development weekend in september 2009. three lecturers were interviewed at a university in norway in september 2010 giving a total of 20 interviews. each participant was given a code in order to maintain anonymity (table 1). at the start of their interview, participants were provided with a sheet of a0 paper and pens in order to illustrate their relationship with others in the project team and outside. examples of illustrations that were drawn are given in figure 2. all participants were then asked to discuss their involvement in the creation of the e-compendiums and social networks developed during this process. also explored were barriers and facilitators, any support and training they received, quality assurance issues and personal and perceived levels of control over the process. in addition, lecturers were asked about their existing teaching methods, their confidence and previous experience of e-learning (see appendix 1 for interview topic guides). interviews lasted between 10 and 30 minutes. with consent, interviews were audiorecorded and transcribed verbatim. research in learning technology citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 data analysis once the interview data had been transcribed, this was imported into the qualitative analysis package nvivo 9. each interview was read and the data were categorised using thematic analysis to produce anticipated and emergent themes. the social networks that were described during their interviews were further explored and analysis extended by using pictorial network maps to understand the relations between members of the team. the pictures that were drawn by the participants were figure 2. anonymised example of pictorial network map produced by lecturer (t6) & illustrator (a2). letters refer to other team members (m1 and m3 are project managers; t4 is a lecturer; d6 is a member of the development team and pd is the pedagogical designer). table 1. roles and assignment of staff members working on the project. participant code role within the team centrality (in-degree) centrality (out-degree) a1 design team: artist � computer illustrator 3 3 a2 design team: artist � traditional illustrator 2 2 d1 design team: developer (provided vocal commentary) 9 9 d2 design team: developer 6 6 d3 design team: developer 6 6 d4 design team: developer 8 8 d5 design team: developer 7 7 d6 design team: developer 7 11 el design team: european link (business administrator involved in funding e-learning projects) 0 0 pd design team: pedagogical designer 17 12 m1 management team: manager 6 5 m2 management team: manager 10 9 m3 management team: manager 11 12 t1 teaching staff: lecturer 7 7 t2 teaching staff: lecturer 5 5 t3 teaching staff: lecturer 4 5 t4 � m teaching staff: lecturer (but also included management responsibility) 9 9 t5 teaching staff: lecturer 0 1 t6 teaching staff: lecturer 4 4 t7 teaching staff: lecturer 1 1 a. latif et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 initially deconstructed to a binary figure (0 �no contact; 1 �contact) to establish what relations there were between stakeholders. two actors were said to be connected if they revealed this relationship in their interviews or on the networks they drew. the data on the reported ties between the actors were then inputted into the software ucinet (borgatti, everett, and freeman 2002) for analysis. a positional approach was taken whereby the actors were identified through their formally defined position or group memberships (scott 2000; wasserman and faust 1994). these were then represented visually using network maps (sociograms) to further the analysis alongside interview transcripts. findings project team roles and networks we categorised participants into three groups according to their institutional job role. a management team, design team and teaching staff (comprising of lecturers). the management team (m1, m2, m3 and t4�m) were responsible for initiation of the project, recruiting and allocating tasks to others, taking decisions, monitoring output and quality as well as co-ordinating work between different staff. the management team also had important roles in liaising at all levels including organisations outside the university to ensure success of the new online course. the teaching team (t1, t2, t3, t4, t5, t6 and t7) were the lecturers and content writers. the technical design team (d1, d2, d3, d4, d5, d6, el and pd) were based within the learning technology group, which was the university’s unit for web-based studies. the design team roles included a business administrator (el), a pedagogical designer (pd), graphic designers, programmers, flash developers and text editors. this team was responsible for developing the digital materials from the manuscripts developed by the lecturers and uploading them onto university systems for access by students. other contributors to the design team were freelance artists (a1, a2) based in the uk and australia, who worked remotely. decision to develop the online nursing course and compendiums when participants were asked about how the online course came about, they reported that this was due to growing institutional pressure to offer a nursing course off campus. this came at a time when there was a drive to develop e-learning digital media services related to broadband (europa 2010). another driver for initiating an online course was that the university had an experienced unit to produce web-based teaching materials. this led a perception from the lecturers that this was a ‘top down’ rather than ‘bottom up’ initiative. this created tensions between the teaching staff because the materials had to be developed on top of their existing workload: i find it frustrating because this is on top of all my tasks that i have at the institute (t2) i had to make [write] the compendium, it was on top of all my other jobs and i took personal time (t6) consequently, project managers initially found it challenging to make lecturers subscribe to the idea, but this became easier as the value of the project became more evident. research in learning technology citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 working relationships and networks when developing the e-compendiums analysis of contacts that existed within the three groups showed that the management team had the most well connected group network. figure 3 illustrates the internal connections within this group. each line represents an interaction that was reported at interview or illustrated on the network maps drawn by participants (the greater number of ties indicates a greater degree of connectively within the team). as can be seen each member reports ties to each other member. a density measurement, which describes the level of linkage in a network, can be taken of any given network and involves the calculation of the actual number of contacts observed in a network as a proportion of the potential number of contacts in that network. the management network has a density of 1 (where all participants in the network are directly linked). the design team had a similarly well connected network (density 0.73). however, the teaching team (lecturers) was found to be the least well connected (density 0.24). some (t5, t7) reported not being connected to others at all. two freelance artists were employed but worked remotely. work was often given to them when there were resource constraints within the learning technology group. however, they reported being unclear about how the team fitted together and the roles others had in the project: i knew very little about the project . . . i just assumed that it would be maybe someone putting their lecture notes online . . . i had no idea as to the scope of the project (a2) of course not knowing how the images you create will end up sort of leaves a bit of uncertainty while you are doing the illustrations so any long term project like this becomes a learning curve in itself as well (a1) the lack of ties to the freelance artists had some important implications for when work was given to these illustrators. detailed analysis of the networks revealed that the pd was the gatekeeper to the external illustrators (a1 & a2). the pd’s network was illustrative of a ‘star’ network where their positioning allowed them to network with all other actors (figure 4). this was supported by the data which showed the pd to have the highest score for in-degree and out-degree centrality (table 1). these measures determine the importance or how central an individual is in a network. a potentially powerful and an influential member of the network could be characterised as someone who both makes (out-degree) and receives (in-degree) numerous contacts. figure 3. network maps illustrating social networks. management team includes members m1�3, the pedagogical designer (pd), t4�m �teacher with management responsibility. the design team network includes d1�6 and the teaching network includes members t1�7. note: t5 and t7 are not connected to the teaching network as they reported no connections with other participants within the team, nor did others mention them during interviews or within their pictorial networks. a. latif et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 looking more closely at this network, if design team member d6 elects not to provide information to design team member d5, d5 still has a number of other contacts to receive this information. however, if the pd elects to not exchange with d5, then this information would not be conveyed to d5 at all. an example of this was described by one design team member (d5) when describing work that involved the freelance illustrators (a1 & a2). the lack of direct contact with the freelance illustrators constrained the workflow of participant d5: the only problems is that sometimes there is a waiting time for me because i got all the text ready but there’s an illustration missing (d5) likewise, networks between the design team and the lecturers (who were based elsewhere) were also less well established. the design team members shared an office in the learning technology group which allowed them to work closely together. however, the lack of direct contact with the lecturers meant their work was again constrained as they relied on the pd to communicate between them. this matter is illustrated in figure 5. this sociogram reveals how d2’s lack of network contact with the teaching staff made. as can be seen, d2 network with lecturers is limited making communication between them challenging. again the pd played a key connecting role. the management team worked closely with the design team and their day to day interactions meant problems could be resolved spontaneously: the learning and technology group team i actually see face-to-face every day and we solve problems on a minute to minute basis . . . (m2) however, in comparison, the management team had less developed networks with the teaching staff (figure 6). the lack of initial buy-in from lecturers and their poorly developed networks meant there was little engagement with lecturers at the beginning of the project: this project was decided by the management at university, it was not initiated by the staff itself . . . so they didn’t have any opinions about it, actually the other way around, they didn’t believe in it basically (m2) figure 4. sociogram illustrating ties between external illustrators (a1 & a2), design (d6, d5) and management team (m1�3) and the pedagogical designer (pd). research in learning technology citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 the position of the pd within the networks meant that she was the least dependent on any other specific actor. the pd was conscious of the workload of the teaching staff and only contacted them when necessary. nevertheless, there was an understanding that they could contact her when needed: i only contact them when i really need to but i’m always open for them to contact me anytime and it’s a kind of two way thing but i’m trying to be economical with only telling them what they really need, especially when they start teaching again now (pd) however, as will be looked at in the following section, the lack of engagement led to lecturers feeling disconnected with the project. ‘evolving pedagogy’: implications for lecturers the traditional methods of teaching described by lecturers included powerpoint presentations which were made available online either before or after the lecture. a discussion forum was also available to students to share experiences and for lecturers figure 6. sociograms comparing the networks between the management team and teaching/ design team. figure 5. sociograms illustrating d2 network map between lecturers. a. latif et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 to post multiple-choice questions. the lecturers reported being happy with these formats but were open to notions of newer teaching/learning methods. however, when describing the development process for the e-compendiums, the lecturers were not clear about this. the lecturers clearly saw their role as content writers using their existing lecture notes and powerpoints as a basis for the e-compendiums: i started off with my lectures i already had and used the lectures i had in the subject. so that was sort of a start off point. and then i combined that with text book (t5) there was little expectation from the lecturers themselves or other member of the project team that that they would need to be doing any technical development of the e-compendiums. no i’m not involved [with the technical development] and they shouldn’t involve me at all. i can use things, but i can’t make anything (t1) all participants acknowledged the importance of understanding online pedagogy. however, lecturers mentioned there was no formal staff development on this and wanted development opportunities to learn more: i haven’t had training yet, but i hope i can have it (t3) lecturers also felt they were working in the ‘dark’, not knowing exactly what was involved in developing the e-compendiums and importantly how long it was going to take. they were learning as they went along. despite concerns about workload and speculation about how an online course was appropriate for a caring and profession such as nursing, many expressed satisfaction with the e-compendiums once produced and appreciated being part of the process. yes i’m surprisingly content with everything, but i wasn’t negative in the start like a lot of other people here, i’ve never been negative to it, but i’m surprised how well it has turned out and i believe it very much that this is also a good way to learn(t5) discussion studies about online learning have focused on the quality of the e-learning material, user perceptions or staff engagement (bates 2011; chen and tseng 2012; newton 2003; ward, peters, and shelley 2010). others have examined the impact of workload on lecturers and the time involved in engaging with e-learning teaching (care and scanlon 2001; delgaty 2013; minnaar 2013). however, multi-stakeholder analyses of social networks and the implications these have on those teaching through this format are under-reported. this study aimed to fill this gap and provides insights into the social networks that are created between those who teach the material and those who manage and produce them. the poorly developed networks with the teaching staff had the potential to challenge or even threaten previously well-established notions of pedagogy. this situation was further exacerbated by the ‘top down’ approach and lack of engagement of all staff at the start of the project. these findings supported findings from connolly, jones, and jones (2007) who undertook a qualitative study exploring how a group of tutors involved in a major e-learning project reacted to developing and teaching in this new environment. in this study, it was found that all respondents research in learning technology citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 were keen to keep an element of face-to-face teaching in their modules and felt it was difficult to gauge the depth of learning of their students without this. the lack of communication about the potential value of the e-compendiums at the beginning of the project meant there was a lack of early buy-in from the lecturers. furthermore, the lack of early involvement meant that lecturers had poorly developed social networks, not only between themselves but also with other members of the project team. this led to a sense of overall disengagement with the process of production, because they did not perceive this to be part of their responsibility. the role of the pd was found to be crucial in facilitating communication within the team. the pd was the best connected of all of the participants, was central to the project workflow and enabled dissemination of information to all subgroups to ensure the success completion of the project. nevertheless, the reliance upon one individual to co-ordinate between the different groups, at times constrained effective workflow. several models have been proposed to facilitate engagement and ‘project manage’ online courses (bates 2011, p. 68). one project management model depends upon individual team members contributing appropriate skills and knowledge to the project. this model has been described as advantageous to teaching staff, because the project manager can assume most of the administrative and bureaucratic duties thus freeing faculty to function as content experts. by examining the roles and relationships of the various stakeholders involved, administrators and educators can build a better model for future interdisciplinary distance course development. likewise, another model known as ‘care of practice’ has been proposed by windle and wharrad (2010). this specifically encourages equity and collaboration within a multi-skilled workforce. however, this work showed that despite there being a wellconnected management and design structure, the lack of effective network connections to the lecturers suggest disempowerment, with the lecturer role restricted solely to the technical writing of teaching material. whereas, some may view this as advantageous because of the existing high workload, others may see this as a lost opportunity to actively utilise the pedagogical skills of lecturers during course development phase. this study has important implications for those commissioning and managing online projects. effective engagement with lecturers on decisions to develop online courses needs to occur at the onset to ensure all stakeholders are fully engaged and have the opportunity to input in the design and development process. major (2010) observed that faculty members believe teaching online changes the way they approach and think about teaching, course design, time, instruction and students. effective social networks need to exist if the experience of lecturers is to be fully utilised. for this to happen there needs to be early involvement and communication with lecturers. furthermore, there was a noticeable lack of involvement with potential users of the material. the quality of the e-compendiums may be strengthened if students are included in these networks. the implications for lecturers include the need to be proactive and become accustomed to the new networks that will be formed when developing online courses. resources need to be available to lecturers so that they can not only write the contents of the course material but also have an opportunity to input into the development process. blake (2009) suggests that raising staff awareness of the potential of e-learning tools and on-going support and mentoring may be beneficial to improve staff involvement. this study suggests that if teaching staff a. latif et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 voices are to be heard during the design and development of online course material, they will need to be a concerted effort to upskill lecturers to ensure their input into pedagogy is effective and they are engaged at all levels to ensure successful implementation of the project. conclusion recognising the importance of how social networks with differing roles and skills are formed and managed is key to effective management of these challenging projects. this study aimed to provide such insights into the social networks that are created during the development of online learning material. teaching staff need to be fully consulted on initiatives to migrate learning materials online to avoid challenging and even threatening previously well-established notions of pedagogy. future development of e-compendiums should account for the perceived lack of time and existing workload of lecturers so they may be involved with the development process. strengths and limitations of the study to our knowledge, this is the only study that has investigated social networks among different groups working on an e-compendium for an online nursing course. as such it provides novel insights into the social connections made during the production of e-compendiums. one limitation to this study is that interview accounts were used to collect data on the number of contacts within the network. the data are therefore open to limitations of recall bias of the respondents. future studies that seek to use social network approaches should find ways to triangulate connections so the robustness of networks can be validated. conflict of interest hw and rw are members of the school of health science’s education and technology for health research group. al declares no conflict of interest. ethical approval this study formed part of a wider evaluation project so ethical approval was not deemed necessary. nevertheless, all participants were fully informed about why the evaluation was taking place, were provided with a participant information sheet and written informed consent was taken before interviews. it was explained to all participants that they were free to withdraw at any time. anonymous codes have been used in this paper to protect the identity of participants. the evaluation was conducted according to the british educational research association (bera) ethical guidelines. acknowledgements the authors thank all the participants who took part in this study. research in learning technology citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 references alexander, s. (2001) ‘e-learning developments and experiences’, education�training, vol. 43, no. 4/5, pp. 240�248. bates, a. w. (2011) outlook for online learning and distance education, contact north, sudbury ontario, canada. blake, h. (2009) ‘staff perceptions of e-learning for teaching delivery in healthcare’, learning in health and social care, vol. 8, no. 3, pp. 223�234. borgatti, s. p., everett, m. g. & freeman, l. c. (2002) ucinet for windows: software for social network analysis, analytic technologies, harvard, ma. care, w. d. & scanlon, j. m. (2001) ‘planning and managing the development of courses for distance delivery: results from qualitative study’, online journal of distance learning administration, vol. 4, no. 2. [online] available at: http://www.westga.edu/~distance/ojdla/ summer42/care42.html chen, h. r. & tseng, h. f. (2012) ‘factors that influence acceptance of web-based e-learning systems for the in-service education of junior high school lecturers in taiwan’, evaluation and program planning, vol. 35, no. 3, pp. 398�406. connolly, m., jones, c. & jones, n. (2007) ‘new approaches, new vision: capturing teacher experiences in a brave new online world’, open learning, vol. 22, no. 1, pp. 43�56. dariel, o. p. (2011) exploring e-learning adoption in nurse education: a socio-cultural case study using q and bourdieu, doctoral dissertation, university of nottingham, [online] available at: http://eprints.nottingham.ac.uk/12256/ delgaty, l. (2013) ‘a critical examination of the time and workload involved in the design and delivery of an e-module in postgraduate clinical education’, medical teacher, vol. 35, no. 5, pp. e1173�e1180. eseryel, d. & ganesan, r. (2001) ‘distributed group design process: lessons learned’, in proceedings of ed-media 2001: world conference on educational multimedia, hypermedia, & telecommunications, eds c. montgomerie & j. vitelli, norfolk, va, usa, vol. 2001, no. 1, pp. 469�473. europa. (2010) ‘state aid: commission processes record number of broadband projects following new broadband guidelines’, [online] available at: http://europa.eu/rapid/press releasesaction.do?reference�memo/10/31 grunspan, d. z., wiggins, b. l. & goodreau, s. m. (2014) ‘understanding classrooms through social network analysis: a primer for social network analysis in education research’, cbe-life sciences education, vol. 13, no. 2, pp. 167�178. gwozdek, a. e., et al., (2011) ‘using online program development to foster curricular change and innovation’, journal of dental education, vol. 75, no. 3, pp. 339�350. keating, n., et al., (2007) ‘factors affecting influential discussions among physicians: a social network analysis of a primary care practice’, journal of general internal medicine, vol. 22, no. 6, pp. 794�798. korhonen, t. & lammintakanen, j. (2005) ‘web-based learning in professional development: experiences of finnish nurse managers’, journal of nursing management, vol. 13, no. 6, pp. 500�507. lewis, j. m., baeza, j. & alexander, d. (2008) ‘partnerships in primary care in australia: network structure, dynamics and sustainability’, social science & medicine, vol. 67, no. 2, pp. 280. lymn, j. s., bath-hextall, f. & wharrad, h. j. (2008) ‘pharmacology education for nurse prescribing students�a lesson in reusable learning objects’, biomedcentral nursing, vol. 7, no. 1, pp. 2. major, c. h. (2010) ‘do virtual professors dream of electric students? university faculty experiences with online distance education’, teachers college record, vol. 112, no. 8, pp. 2154�2208. mancuso-murphy, j. (2007) ‘distance education in nursing: an integrated review of online nursing students’ experiences with technology-delivered instruction’, journal of nursing education, vol. 46, no. 6, pp. 252�260. meyen, e. l., tangen, p. & lian, c. h. (1999) ‘developing online instruction: partnership between instructors and technical developers’, journal of special education technology, vol. 14, no. 1, pp. 18�31. a. latif et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 http://www.westga.edu/~distance/ojdla/summer42/care42.html http://www.westga.edu/~distance/ojdla/summer42/care42.html http://eprints.nottingham.ac.uk/12256/ http://europa.eu/rapid/pressreleasesaction.do?reference=memo/10/31 http://europa.eu/rapid/pressreleasesaction.do?reference=memo/10/31 http://europa.eu/rapid/pressreleasesaction.do?reference=memo/10/31 http://europa.eu/rapid/pressreleasesaction.do?reference=memo/10/31 http://europa.eu/rapid/pressreleasesaction.do?reference=memo/10/31 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 minnaar, a. (2013) ‘challenges for successful planning of open and distance learning (odl): a template analysis’, the international review of research in open and distributed learning, vol. 14, no. 3, pp. 81�108. newton, r. (2003) ‘staff attitudes to the development and delivery of e-learning’, new library world, vol. 104, no. 10, pp. 412�425. penuel, w. r., et al., (2006) ‘investigating the potential of using social network analysis in educational evaluation’, american journal of evaluation, vol. 27, no. 4, pp. 437�451. scott, j. (2000) social network analysis: a handbook. 2nd edn, sage, thousand oaks, ca. shephard, k. (2004) ‘the role of educational developers in the expansion of educational technology’, international journal for academic development, vol. 9, no. 1, pp. 67�83. ward, m. e., peters, g. & shelley, k. (2010) ‘student and faculty perceptions of the quality of online learning experiences’, the international review of research in open and distance learning, vol. 11, no. 3, pp. 57�77. wasserman, s. & faust, k. (1994) social network analysis: methods and applications, cambridge university press, new york. wharrad, h. j., et al., (2001) ‘a comparison of cal with a conventional method of delivery of cell biology to undergraduate nursing students using an experimental design’, nurse education today, vol. 21, no. 7, pp. 579�588. windle, r. & wharrad, h. j. (2010) ‘reusable learning objects in health care education’, in interprofessional e-learning and collaborative work: practices and technologies, eds a. bromage, l. clouder, f. gordon & j. thistlethewaite, igi global, hershey, pa. windle, r., et al., (2011) ‘the characteristics of reusable learning objects that enhance learning: a case-study in health-science education’, british journal of educational technology, vol. 42, pp. 811�823. research in learning technology citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 appendix 1: interview topic guides teaching staff interview topic guide: 1. describe the teaching methods that you currently use 2. describe your previous experiences of using e-learning 3. describe your computer confidence 4. using network drawings, describe the development process involved in the creation of the pdf documents. a. your role within it b. support and training c. how was quality assurance addressed? d. what level of control did you have over the process or who had the control over the process? e. reporting process 5. explore/further comments on online pedagogy design team interview topic guide 1. describe your previous experiences of developing e-learning 2. using network drawings, describe the development process involved in the creation of the pdf documents. a. your role within it b. support and training c. how was quality assurance addressed? d. what level of control did you have over the process or who had the control over the process? e. reporting process 3. explore/further comments on online pedagogy management team interview topic guide: 1. why and how was the decision made to develop an online learning course? explore: � university strategic � student learning � professional strategic 2. what were the drivers/barriers? 3. using network drawings, describe the development process involved in the creation of the pdf documents � your role within it � support and training � how was quality assurance addressed? � what level of control did you have over the process or who had the control over the process? � reporting process 4. explore/further comments on online pedagogy a. latif et al. 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32630 http://dx.doi.org/10.3402/rlt.v24.32630 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32630 http://dx.doi.org/10.3402/rlt.v24.32630 doi:10.3402/rlt.v19s1/7798 stepping beyond the paradigm wars: pluralist methods for research in learning technology chris jones* and gregor kennedy institute of educational technology, the open university, uk; director of elearning, the university of melbourne, australia (received 2 march 2011; final rersion received 12 june 2011) this paper outlines a problem we have found in our own practice when we have been developing new researchers at post-graduate level. when students begin research training and practice, they are often confused between different levels of thinking when they are faced with methods, methodologies and research paradigms. we argue that this confusion arises from the way research methods are taught, embedded and embodied in educational systems. we set out new ways of thinking about levels of research in the field of learning technology. we argue for a problem driven/pragmatic approach to research and consider the range of methods that can be applied as diverse lenses to particular research problems. the problem of developing a coherent approach to research and research methods is not confined to research in learning technology because it is arguably a problem for all educational research and one that also affects an even wider range of disciplinary and interdisciplinary subject areas. for the purposes of this paper we will discuss the problem in relation to research in learning technologies and make a distinction between developmental and basic research that we think is particularly relevant in this field. the paradigms of research adopted have real consequences for the ways research problems are conceived and articulated, and the ways in which research is conducted. this has become an even more pressing concern in the challenging funding climate that researchers now face. we argue that there is not a simple 1 to 1 relationship between levels and most particularly that there usually is not a direct association of particular methods with either a philosophical outlook or paradigm of research. we conclude by recommending a pluralist approach to thinking about research problems and we illustrate this with the suggestion that we should encourage researchers to think in terms of counterpositives. if the researcher suggests one way of doing research in an area, we suggest that they should then set out an opposing research approach from another perspective or paradigm. we link this conclusion to the provision of research training and the kinds of curricula that might be offered and we argue against the superficial and box ticking ‘coverage’ of different standard research perspectives e.g. ‘qualitative methods’ � ‘qualitative methods’ keywords: qualitative methods; quantitative methods; mixed methods; paradigms; methodology *corresponding author. email: c.r.jones@open.ac.uk alt-c 2011 conference proceedings 0156 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # c. jones and g. kennedy. this is an open access article 18 under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7798 the learning technology landscape learning technology is a developing field of study and an emerging area of work (for two recent reviews see czerniewicz 2008, 2010). the field is an emerging profession with its own community of workers and a pattern of employment (beetham, jones, and gornall 2001). it is also an area of academic interest, and the field has its own journals, conferences and related postgraduate qualifications. these two aspects are closely related to each other, for example through the accreditation of learning technologists (oliver et al. 2004). the emerging professional field around learning technology is also the audience for much of the research output from the academic world concerned with learning technology. this audience for research is also the source for much of the demand for qualifications at post-graduate level supporting the successful development of a variety of masters and doctoral level programmes. learning technology is a domain that has a boundary with other professional groups, including educational developers who have their own community and an overlapping area of interests (hudson 2009). diversity remains in the terms used to describe this still emerging field and there are also arguments about whether the field, for many refuse to call it a discipline, remains amorphous and disjointed or is now growing up and attaining a kind of intellectual unity (czerniewicz 2008). a unifying factor in the field is its location in relation to new technology. jones (2004a) has argued drawing on barley and orr (1997), that learning technologists, in a similar way to other technologists, have a distinct relationship to theoretical and scientific knowledge because they are largely consumers rather than producers of basic knowledge. the growth of scientific and technical knowledge has had an impact on education in two distinct ways. (1) the growth in demand for basic and applied scientific knowledge has led to the proliferation of new fields and disciplines, such as the learning technologist. in technical disciplines it is increasingly difficult for individuals to master the breadth of knowledge required and there is an increasing pressure to re-combine specialist technical functions created through a division of labour, that were once integrated in the person of the lecturer. (2) the second impact is in the contradictory process of re-skilling and deskilling in which routine duties are reallocated to less well trained staff alongside an increased demand for fully trained professional staff. in this context the demand for learning technologists comes in part from an increasing technical division of labour arising from the application of new technologies to teaching and learning. the application of new technologies in an educational context means that design has become a key term for research in learning technology. because design can be viewed as a social practice, which may be explicitly informed by scientific theory, it is a form of practical and ethically informed work. design involves both a systematic approach, which may involve rules and protocols derived from research, and an art applied in a set of local and context based practices. design, thought of in this way, is a skilful and creative activity which is open to improvement and development from the application of research and scholarship (jones and dirckinckholmfeld 2009). because of the applied nature of learning technology and the alt-c 2011 conference proceedings 19 multi-disciplinary nature of the intellectual resources for the field there are those that have drawn an analogy with design in other fields: understanding the character and limits of design is important in networked learning. i originally used analogies with ergonomics and especially with architecture to rethink educational design and i still find them useful sources of insight. architecture involves the crafting of affordances, rather than deterministic logics of human control. architecture has methods for managing complexity � not just complexities of construction but also complexities of representation and design. architecture draws on multiple sources of knowledge and combines ways of knowing. it understands people from � at least � the perspectives of biology, psychology and culture. it understands � at least � the physics, geometry, economics, aesthetics and history of buildings. its practices are imbued with epistemic fluency, to a degree that makes many educationalists look, unexpectedly, like members of the spanish inquisition. (goodyear 2009, viii) unlike goodyear, there are those who define the field (in this case described as ‘instructional technology’) more narrowly as a ‘design field’ (reeves, herrington, and oliver 2005, 7). the suggestion these authors make is that ‘design-based’ research is the primary solution for research deficiencies in the field. in our opinion this kind of restriction limits the responses of researchers in the field of learning technology. we suggest that researchers pursue a variety of research goals using high quality educational technology investigations. ross and morrison (1989) differentiate between ‘developmental’ research, which ‘‘is oriented toward improving technology as an instructional tool’’, and ‘basic’ research, which is ‘‘oriented towards furthering our understanding of how these applications affect learning and motivation’’ (20). more recently ross, morrison, and lowther concluded that: we encourage researchers to reduce efforts to prove the ‘‘effectiveness’’ of technology, while focusing on conducting rigorous and relevant mixed-methods studies to explicate which technology applications work to facilitate learning, in what ways, in which contexts, for whom, and why. (ross, morrison, and lowther 2010, 31) in essence what ross and morrison (1989) and ross, morrison, and lowther (2010) are arguing for is that different types of inquiry, with a range of approaches and foci, should be possible under the banner of ‘learning technology research’. this argument, and conflicts that have surrounded it, have to a certain extent been captured in the term ‘paradigm wars’ (gage 1989). paradigms in learning technology research the term normal science and the linked concept of paradigm are most commonly associated with thomas kuhn and his book the structure of scientific revolutions (1970). kuhn is remembered for providing an account of scientific progress that emphasised a form of punctuated equilibrium in which periods of normal science were occasionally disrupted and existing ways of thinking were replaced by new revolutionary changes. kuhn described paradigms as being closely related to the idea of normal science and exhibiting two characteristics: (1) a scientific achievement that was so unprecedented that it could attract an enduring group of adherents from other competing modes of scientific activity. (2) it was sufficiently open-ended as to leave many problems for the new group of adherents to resolve (adapted from kuhn 1970, 10) c. jones and g. kennedy 20 paradigms are social phenomena in which ‘‘accepted examples of actual scientific practice � examples which include law, theory, application, and instrumentation together � provide models from which spring particular coherent traditions of scientific research’’ (kuhn 1970, 10). the effect of paradigms for students is that because they join others who learned the basis of the field from the same concrete models, their subsequent practice will seldom evoke overt disagreement over fundamentals (kuhn 1970, 11). consensus is a pre-requisite for normal science and, by adopting a paradigm, students ‘‘are committed to the same rules and standards for scientific practice’’ (11). traditionally two separate paradigms of inquiry dominated research in education. the early years of educational research were dominated by psychology and a largely positivist understanding of scientific method. more recently a powerful counter current concentrated on the development of qualitative research using a largely interpretivist approach to analysis. these two research approaches have traditionally been seen in opposition which is well reflected in debates that took place many years ago in what have been described as the ‘paradigm wars’ (gage 1989). the paradigm wars saw researchers with particular philosophies and methods of inquiry arguing strongly that ‘their way’ was the most appropriate. in 1989, gage fittingly imagined the situation 30 years in the future; hence in our recent past in 2009. he argued that there were three possible outcomes available: � the positivist, establishment, mainstream, standard, objectivity-seeking and quantitative approach had died of the wounds inflicted by its critics. � peace had broken out in an earnest dialogue, lifting the discussion to a new level of insight, making progress toward workable solutions of and generating theory that fitted together. � nothing that was true in 1989 had really changed, and the wars were still going on.(adapted from gage 1989, 10) by 2009 peace had broken out, but not in the earnest and productive way that was envisaged, rather as kuhn might have anticipated, it had become peaceful with the restoration of a period of ‘normal science’ in which a single dominant paradigm settled the basis for major disputes through a division of spoils. so rather than being settled or resolved in favour of a clear winner, the paradigm of research in the social sciences embedded the distinction between quantitative and qualitative methods in a way that often implies that they are incommensurable approaches. jones (2004b) has argued previously that the division between quantitative and qualitative methods has become overdrawn and rooted in an excessively theoretical approach to social research. one result of the division between two distinct research methods has been that, increasingly, commentators on social science research, including that undertaken by educational technologist, advocate mixed-methods and pragmatic approaches to research (e.g. johnson and onwuegbuzie 2004; salomon 1991; shulman 1988). we argue that the research agenda embraced by learning technologists should indeed be pluralistic but perhaps more importantly that the field needs to step beyond the form of ‘normal science’ that has become institutionalised since the paradigm wars into the quantitative-qualitative divide in social science and hence learning technology research. alt-c 2011 conference proceedings 21 students’ exposure to research and use of opposing paradigms when students begin their research training they are often confused between the different levels of analysis when thinking about methods, methodologies and research paradigms. for example students regularly conflate quantitative methods with a positivist approach to research. these confusions arise from a number of sources, one of which is a desire or requirement to make their research plans consistent with what is often described as an overarching philosophical position in terms of ontology or epistemology. an example of this type of confusion is seen when students feel it is necessary, in their discussion of the methodology underpinning their research, to show that they have considered deep philosophical questions concerning the nature of phenomena and come to a definite conclusion. these students are frequently untrained in philosophy and are addressing profound and intractable problems, yet they feel obliged to make definitive statements. having engaged with and ‘covered’ the philosophy, students often take the argument forward by the selecting an appropriate paradigm for research prior to clarifying the research problem. when seen in this way, approaches to research become simple recipe-following, leading to a mechanical selection of a specific method. an example would be the choice of a qualitative approach to research and the adoption of one or other forms of grounded theory as the methodological outcome of the choice of paradigm. this kind of development in a research project is not simply the outcome of poor student choices; rather it often reflects implicit and explicit commitments within particular departments and research groups. it can be the influence of individual faculty members and the outcome of historical recruitment patterns of staff reflecting specific kinds of expertise in particular methods and research approaches. pathways in learning technology research training research training in the social sciences currently enforces the single dominant paradigm highlighted above in the agreed binary division of spoils into quantitative and qualitative research. the economic and social research council in the uk, the main funding body for social science research has until recently accredited research training in what are termed 3�1 phd programmes. the �1 element of the four year programme is a masters in research which provides a curriculum that generally includes modules called quantitative and qualitative research methods or variants on this distinction, for example qualitative research practice and introduction to statistical analysis. this conventional framework for research methods training is not confined to the uk. for example from the australian context, the 2010 charles sturt university handbook of subject offerings, available online, shows that postgraduate students can enrol in (our emphasis). qualitative research methods this subject introduces students to the field of qualitative research. the first half of the subject requires students to critically engage with some of the major theoretical debates, which both define the field and delineate between different kinds of qualitative research. the second half of the subject asks students to apply the ideas discussed in the first half by conducting a piece of qualitative research in an area of their own choosing. . . using a structured and sequential list of readings, stimulus questions and spaces for student reflection, the subject prosecutes a single objective; that all research method choices should derive from philosophical and theoretical principles c. jones and g. kennedy 22 which can be explained and defended, as opposed to simply conforming to taken-forgranted ideas about how research should be done. quantitative research methods this subject is designed to introduce students to research methodologies and statistical procedures that are commonly used in quantitative research. as the central aim of the subject is to enable students to become intelligent and critical readers of research literature, the emphasis is on the purposes and constraints of selected statistical procedures. this requires a basic understanding of fundamental constructs that underpin data collection procedures and data analysis in quantitative research. considerable emphasis is given to statistical procedures including univariate and bivariate analysis, as well as more sophisticated multivariate techniques. from this foundation, students are required to submit a proposal for quantitative research study, which asks students to identify a problem in the broad field of education, develop a research question or hypothesis, define the inherent constructs, select appropriate methods to investigate these constructs, and determine an analysis plan. a clear implication of our argument thus far is that graduate students undertaking research training in learning technology need to be exposed to a range of approaches to learning technology research. the standard approach to this area within postgraduate studies at university would be familiar to many: the unit, subject or course that provides students with discrete explanations of the popular historical and contemporary approaches to social science research. many of these courses will give students an opportunity to apply the research methods they have covered in the course to their own research project or problem. often this will result in students � perhaps after a period of reflection and consideration, perhaps after asking what their student colleagues ‘‘are using’’ for their research and even in consultation with their research advisors � adopting a research method that is consistent with their department, research group or advisor. in many cases this will result in research questions and aims being investigated using appropriately framed paradigms, methodological approaches and methods. but what is often missing from this approach is a genuine consideration of alternative framings and approaches to learning technology research. when asked to apply what they have learnt in ‘research methods’ courses to their own research problems, unsurprisingly students typically gravitate towards what they, their advisors or their departments, ‘know’, advocate and feel comfortable with. so while students are exposed to � or told about-different flavours and styles of learning technology research, they are often not, in our experience, encouraged to think deeply about the implications of these approaches when it comes to the actual conduct of learning technology research. given this, we offer an example of an approach to research training in learning technology that actively encourages students to consider alternative perspectives or pathways that can be taken in learning technology research. an approach kennedy has used in advising higher degree research students is the use of counter-positives. when students are describing and defining their investigations within a research project he will often ask them to clearly articulate their aims, goals or questions, their methodological approach to these questions and how they will actually go about collecting or generating data. while many students will need help in this, most will be able to come up with workable research approaches. in fact, some students are able to quickly articulate their methodology and method on the basis of their previous academic experiences. a common example of this from work in the health sciences is that students propose a clearly articulated experimental method as an approach to alt-c 2011 conference proceedings 23 investigating what are fundamentally exploratory research questions in the area of learning technology. regardless of what students propose in the first instance, we often find it useful for students to actively consider alternative approaches to their specific research investigation and problem. so the student who proposes an experimental method will be asked to consider how the same or similar question could be investigated using a contrasting paradigm, methodology and method. in doing this, students will see how the nature of their research aim or question may change, often quite subtly, in response to an alternative investigative approach. they will also see how an alternative methodological approach to a research question might generate data that would be neglected with the approach originally advocated; and the new type of data might seem more useful in responding to the question. this would often result in thinking more deeply about how the original question might be changed or how the methodology and approach to the research might be changed. by discussing and reflecting on counter positive research approaches, we hope students come to understand, whatever approach they ultimately choose, there is a need for pluralism in learning technology research. pressures on the current paradigm while we are in a period of normal science there are several pressures on the current paradigm, which embeds the division between qualitative and quantitative research in learning technology. firstly new technologies have opened up new kinds of research relevant to the field. some of these, such as virtual ethnography (hine 2000; wittel 2000) extend the range of possibilities for researchers, but pose no great challenge to the existing paradigm of normal research in learning technology. however there are other developments that threaten to undermine existing divisions into neat methodological categories. flyvbjerg (2004) argues for a proper and full place for case studies in social science research but notably in his conclusion he makes the point of arguing that this approach does not exclude whole population survey research, which he argues has a complimentary role to play. herring (2008) suggests integrating discourse analysis with social network analysis in an expanded form of content analysis and judd and kennedy (2010) used computer logs over a five year period to monitor students’ actual rather than reported technology use and the variation in that usage over time. commenting on the impact of internet technologies on qualitative research baym and markham (2009) note that: the internet brings into sharp relief previously assumed and invisible epistemologies and practices of inquiry. in fact, challenges of conducting internet research have prompted its researchers to confront head-on, numerous questions that lurk less visibly in traditional research contexts. consequently internet researchers have been compelled to reconsider basic principles and practices of qualitative inquiry, with important critiques of a priori methodological certainties (baym and markham 2009, viii) all these examples show how current research, especially that engaged with new technologies, questions the taught division between quantitative and qualitative research and these minor challenges to the joint quantitative-qualitative paradigm are amplified in a range of new types of research that rely on the naturally occurring data collected by computers and computer networks and access to new kinds of data. c. jones and g. kennedy 24 some of the emerging methods of research may pose a more fundamental challenge to the current paradigm. to illustrate these potential challenges we have chosen two emerging research areas. 1. network analysis (barabasi 2002), including sns (e.g. haythornthwaite 2005), learner analytics (retalis et al. 2006) and visualizations. large data sets can be mined for naturally occurring data that describe patterns of interaction that have stable features in aggregate even though individual interactions remain contingent. for example barabasi’s work points to the prevalence of scale free networks in a variety of phenomena including mobile phone links, internet and web connections. social network analysis has developed a language for research and a set of techniques as well as stable results, for example about the approximate size of personal networks. the techniques of sns can also be applied to generate powerful visualisations (dawson, bakharia, and heathcote 2010) 2. neurological studies e.g. studies on the brain in relation to the effects of immersion in new technologies (bavelier, green, and dye 2010; dalgarno, kennedy, and bennett 2009; meyler et al. 2008) neuroscience has an obvious connection to education but it has a specific relationship to ideas in learning technology through the claims made by authors such as prensky (2001) about the effects of technology immersion on the brains of young people. the suggestion to which our argument gives rise is that normal science, conducted within an overall paradigm of research allowing two different traditions to co-exist, is being challenged by a major shift in the research environment related to digital and networked technologies. there is a danger that the co-existence of two research approaches in one research area leads to a dialogue of the deaf with researchers only listening to research conducted within their own research domain and ignoring research using other approaches. it is the pressing issues and challenges that face learning technologists that will drive students and researchers to explore existing problems in new ways, using the new technologies as research instruments and platforms, and examining the novel problems that arise alongside the developing technological environment. the drive for change in research training is most likely to be driven by research students challenging existing training and research practices. the second potential source of challenges to the existing paradigm in learning technology research lies in the topics we address. both authors have engaged in recent years with issues concerning the relationship between new technologies and students’ attitudes and behaviour, often characterised using the terms net generation and digital natives. the kinds of claims made by prensky (2001) with regard to the brain, cannot be answered by the standard repertoire of educational research methods and require the use of additional techniques (bavelier, green, and dye 2010), such as magnetic resonance imaging (dalgarno, kennedy, and bennett 2009). this research topic requires complex approaches incorporating standard methods, including surveys, to describe what is happening and qualitative work to explore why students act in the ways that they do, but extending beyond this normal repertoire researchers have been engaged in exploring novel methodological approaches that stretch existing alt-c 2011 conference proceedings 25 boundaries (judd and kennedy 2010; dalgarno, kennedy, and bennett 2009; jones and healing 2010). conclusion we have argued that learning technology research is currently dominated by a paradigm that divides research into qualitative and quantitative types. we are by no means original in suggesting that the division is no longer useful and possibly false (layder 1993). we go on to argue that the division has become ‘normal science’ in learning technology and it has provided a consensus that has allowed researchers to avoid disagreements over fundamentals. it has also provided the framework for standard research training. we have argued that this standard framework is coming under pressure from developing research techniques which are particularly relevant to learning technologists. some of these, for example neuroscience methods, stand in a more or less traditional scientific paradigm. others such as the use of naturally recorded log data and data mining techniques applied to large corpuses of data sit less clearly within the standard framework. it is not yet clear if these new techniques will undermine the existing paradigm or simply be absorbed by it. in practical terms we have explored ways to focus more explicitly on the tension between research approaches through the use of the example of using counter positives in postgraduate students’ research training. this suggestion illustrates ways that we think it is necessary to develop research training that address the problems and confusions arising from adherence to a strong notion of the linkage between individual research methods and overall research philosophies. we argue for a pragmatic approach to method which pays greater attention to the research question being addressed rather than to any overall philosophical tradition. we conclude by highlighting that the current consensus about research methods in learning technology research may very well be under threat from the development of methods enabled by new technologies that do not fit within ‘normal science’ as practiced in learning technology research. references barabasi, a.-l. 2002. linked: the new science of networks. cambridge, ma: perseus publishing. barley, s.r., and j.e. orr. 1997. between craft and science: technical settings in u.s. settings. ithaca, ny, and london: ilr/cornell university press. bavelier, d., c.s. green, and m.w.g. dye. 2010. children, wired: for better and for worse. neuron 67, no. 5: 692�701. baym, n.k. and a.n. markham, eds 2009. internet inquiry: conversations about method. london: sage. beetham, h., s. jones, and l. gornall. 2001. career development of learning technology staff: scoping study final report for the jisc jcalt. bristol: university of plymouth, university of bristol. http://www.jiscinfonet.ac.uk/resources/externalresources/ltcdss/ltcdssfinal-report.doc (accessed july 22, 2011). czerniewicz, l. 2008. distinguishing the field of educational technology. the electronic journal of e-learning 6, no. 3: 171�178. available online at http://www.ejel.org (accessed july 22, 2011). czerniewicz, l. 2010. educational technology � mapping the terrain with bernstein as cartographer. journal of computer assisted learning 26, no. 6: 523�34. c. jones and g. kennedy 26 http://www.jiscinfonet.ac.uk/resources/externalresources/ltcdss/ltcdss-final-report.doc http://www.jiscinfonet.ac.uk/resources/externalresources/ltcdss/ltcdss-final-report.doc http://www.ejel.org dalgarno, b., g. kennedy, and s. bennett. 2009. using brain imaging to explore interactivity and cognition in multimedia learning environments. ozchi’09 (november 23�27), melbourne, australia. dawson, s., a. bakharia, and e. heathcote. 2010. snapp: realising the affordances of realtime sna within networked learning environments. in proceedings of the 7th international conference on networked learning, aalborg 3�4th may 2010, eds. l. dirckinck-holmfeld, v. hodgson, c. jones, d. mcconnell, and t. ryberg. http://www.lancs.ac.uk/fss/ organisations/netlc/past/nlc2010/abstracts/dawson.html (accessed july 22, 2011). flyvbjerg, b. 2004. five misunderstandings about case-study research. in qualitative research practice, ed. c. seale, g. gobo, j.f. gubrium, and d. silverman, 420�434. london and thousand oaks, ca: sage. gage, n.l. 1989. the paradigm wars and their aftermath: a ‘‘historical’’ sketch of research on teaching since 1989. educational researcher 18, no. 7: 4�10. goodyear, p. 2009. foreward. in analysing networked learning practices in higher education and continuing professional development, ed. l. dirckinck-holmfeld, c. jones, and b. lindström, vii�x. rotterdam: sense publishers. haythornthwaite, c. 2005. social networks and internet connectivity effects. information. communication & society 8, no. 2: 125�47. herring 2008. web content analysis: expanding the paradigm. in the international handbook of internet research, ed. j. hunsinger, m. allen, and l. klastrup, 233�45. berlin: springer verlag. hine, c. 2000. virtual ethnography. london: sage publications. hudson, a. 2009. new professionals and new technologies. phd diss., umea univ. www. diva-portal.org/smash/get/diva2:236168/fulltext01 (accessed july 22, 2011). johnson, r.b., and a.j. onwuegbuzie. 2004. mixed methods research: a research paradigm whose time has come. educational researcher 33, no. 7: 14�26. jones, c. 2004a. theory and the practices of learning technology. in networked learning 2004: proceedings of the fourth international conference on networked learning 2004, ed. s. banks, p. goodyear, v. hodgson, c. jones, v. lally, d. mcconnell, and c. steeples. lancaster: lancaster university and university of sheffield. jones, c. 2004b. quantitative and qualitative research: conflicting paradigms or perfect partners? in networked learning 2004: proceedings of the fourth international conference on networked learning 2004, ed. s. banks, p. goodyear, v. hodgson, c. jones, v. lally, d. mcconnell, and c. steeples, 106�12. lancaster: lancaster university and university of sheffield. jones, c., and l. dirckinck-holmfeld. 2009. analysing networked learning practices: an introduction. in analysing networked learning practices in higher education and continuing professional development, ed. l. dirckinck-holmfeld, c. jones, and b. lindström, 1�27. rotterdam: sense publishers. jones, c., and g. healing. 2010. networks and locations for student learning. learning media and technology 35, no. 4: 369�85. judd, t., and g. kennedy. 2010. a five-year study of on-campus internet use by undergraduate biomedical students. computers & education 55, no. 1: 564�71. kuhn, t.s. 1970. the structure of scientific revolutions, 2nd ed. chicago: university of chicago press. layder, d. 1993. new strategies in social research. cambridge, uk: polity press. meyler, a., t.a. keller, v.l. cherkassky, j.d.e. gabrieli, and m.a. just. 2008. modifying the brain activation of poor readers during sentence comprehension with extended remedial instruction: a longitudinal study of neuroplasticity. neuropsychologia 46: 2580�92. oliver, m., r. sharpe, j. duggleby, d. jennings, and d. kay. 2004. accrediting learning technologists: a review of the literature, schemes and programmes. final version 6. http:// www.ucl.ac.uk/calt/alt-accreditation/initial_review.doc (accessed july 22, 2011). prensky, m. 2001. digital natives, digital immigrants, part 2: do they really think differently? on the horizon 9, no. 6: 6. reeves, t.c., j. herrington, and r. oliver. 2005. design research: a socially responsible approach to instructional technology research in higher education. journal of computing in higher education 16, no. 2: 97�116. alt-c 2011 conference proceedings 27 http://www.lancs.ac.uk/fss/organisations/netlc/past/nlc2010/abstracts/dawson.html http://www.lancs.ac.uk/fss/organisations/netlc/past/nlc2010/abstracts/dawson.html http://www.diva-portal.org/smash/get/diva2:236168/fulltext01 http://www.diva-portal.org/smash/get/diva2:236168/fulltext01 http://www.ucl.ac.uk/calt/alt-accreditation/initial_review.doc http://www.ucl.ac.uk/calt/alt-accreditation/initial_review.doc retalis, s., a. papasalouros, y. psaromiligkos, s. siscos, and t. kargidis. 2006. towards networked learning analytics � a concept and a tool. in proceedings of the fifth international conference on networked learning 2006, eds. s. banks, v. hodgson, c. jones, b. kemp, d. mcconnell, and c. smith. lancaster: lancaster university. http:// www.lancs.ac.uk/fss/organisations/netlc/past/nlc2006/abstracts/retalis.htm (accessed july 22, 2011). ross, s.m., and g.r. morrison. 1989. in search of a happy medium in instructional technology research: issues concerning external validity, media replication, and learner control. educational technology research and development 37, no. 1: 1042�629. ross, s.m., g.r. morrison, and d.l. lowther. 2010. educational technology research past and present: balancing rigor and relevance to impact school learning. contemporary educational technology 1, no. 1: 17�35. salomon, g. 1991. transcending the qualitative-quantitative debate: the analytic and systemic approaches to educational research. educational researcher 20, no. 6: 10�8. shulman, l.s. 1988. disciplines of inquiry in education: an overview. in complementary methods for research in education, ed. r.m. jaeger, 3�17. washington, dc: aera. wittel, a. 2000. ethnography on the move: from field to net to internet [23 paragraphs]. forum qualitative sozialforschung/forum: qualitative research [on-line journal] 1, no. 1. http://qualitative-research.net/fqs (accessed july 22, 2011). c. jones and g. kennedy 28 http://www.lancs.ac.uk/fss/organisations/netlc/past/nlc2006/abstracts/retalis.htm http://www.lancs.ac.uk/fss/organisations/netlc/past/nlc2006/abstracts/retalis.htm http://qualitative-research.net/fqs (page number not for citation purpose) 1 *corresponding author. email: adam.cziboly@hvl.no research in learning technology 2023. © 2023 á. cziboly et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 research in learning technology vol. 31, 2023 original research article learning patterns and risks in distance learning during the covid-19 lockdown – the pupils’ perspective in drama pedagogy-based focus groups ádám czibolya*, ádám bethlenfalvyb, szilvia némethc and richárd rajnaic awestern norway university of applied sciences, bergen, norway; bkároli gáspár university of the reformed church in hungary, budapest, hungary; ct-tudok centre for knowledge management and educational research inc., budapest, hungary (received: 29 november 2022; revised: 18 march 2023; accepted: 19 april 2023; published: 05 june 2023) in this study, primary school pupils have been surveyed using the methodology of drama pedagogy, focusing on two research questions: what the risks of online activities are and how children cope with these, and what the experiences of children with distance learning were. this study investigated both areas jointly from the pupils’ perspective. three anonymous online focus groups were conducted with 16 hungarian pupils (4 boys and 12 girls; age range: 11–15 years) in july 2020, who joined to the research on a voluntary basis and have been recruited from three rural counties with different socioeconomic backgrounds. respondents unequivocally recounted that during the lockdown, they had spent a significant part of their time in front of a screen, mostly without adult supervision. whilst most only experienced different forms of teasing, some cyberbullying instances bordered on criminal cases and required the involvement of parents. high exposure to such risks was experienced during the months when online learning mainly consisted of receiving assignments to work on, and most teachers were almost unavailable. future research could compare the experiences of pupils and teachers, creating an online safe space for them where they could respond to each other’s perceptions, interpretations and opinions anonymously. keywords: digital education; covid-19; cyberbullying; drama pedagogy; drama in education; global kids online introduction: the effect of covid-19 on digital transformation in education already before the first wave of the covid-19 pandemic, several studies (e.g. livingstone and helsper 2008; livingstone, mascheroni, and staksrud 2018; ito et al., 2008) focused on uncovering to what extent the toolkit of the online world was capable of influencing and rearranging the usual routines of our everyday lives. in the first two decades of the millennium, time spent in the digital space increased amongst almost every social group (buckingham 2020). during the pandemic, the balance of time to access the supplementary material, please visit the article landing page mailto:adam.cziboly@hvl.no http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2968 http://dx.doi.org/10.25304/rlt.v31.2968 á. cziboly et al. 2 citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 (page number not for citation purpose) spent offline versus online underwent a dramatic shift towards the latter, not only in public education (e.g. daniel 2020) but in higher education as well (e.g. crawford et al. 2020). research into digital education during the pandemic has exponentially grown during the past 2 years (mustapha et al. 2021), even questioning whether the unforeseen prevalence of digital education will lead to ‘new normality’ (grek and landri 2021). at such a unique socialand media historical moment, a budapest-based research group headlined a project (németh et al. 2021; németh & rajnai, 2021) based on the freely accessible research methodology of the international organization global kids online (livingstone 2014; eu kids online 2014). originally, this study had been tailored to explore what the risks are of online activities and how children cope with these. however, in a world where almost all aspects of everyday life had to be reorganized to fit into the online space, it was obvious that we add another research question: what the experiences of children with distance learning were. although the reception of digital education has been excessively studied amongst higher education students both in hungary (e.g. bogdandy, tamas, and toth 2020) and internationally (e.g. händel et al. 2020), there are less studies about the experiences and attitudes of primary school pupils. as research on the functioning of schools and the behaviour of teachers during the first wave in 2020 has already pointed out, digital distance learning introduced during the lockdowns had a negative effect on children’s emotional development due to the limited number of online classes, the lack of interactions and the frequent absence of pupils (e.g. unicef 2020). digital education was especially challenging for children with special needs since related service hours decreased, and many of them could not attend the online courses without continuous support from parents (sonnenschein et al. 2022). as to whether schools managed to fulfil their academic function, contradicting results have been published. whereas a recent study in sweden has shown that there was no covid-19-related learning loss in reading in primary schools (hallin et al. 2022), williamson, macgilchrist, and potter (2021) would argue for an economically measurable learning loss. scholars, however, agree that the crucial function of schools as socializing environments could not have been fulfilled; to put bluntly, quarantine education did not contribute to emotional development (osváth & papp 2020). besides the state of their mental health, the quality and effectiveness of teachers’ work were also affected by two key factors: first, the level of their digital competence, and second, their level of methodological prowess, which, if higher, could have contributed to a more flexible adaptation to these new circumstances as well as to the individual treatment of pupils (n. kollár 2021). kang (2021) identified poor motivation, negative effects of device usage and the unequal division of resources as the key obstacles to digital transformation in education. research has shown that during the covid-19 pandemic, the prevalence of cyberbullying has significantly increased amongst us adults (barlett et al. 2021), korean adolescents (shin & choi 2021), chinese ‘netizens’ (yang 2021) or kazakh adolescents (utemissova, danna, and nikolaevna 2021), just to name a few. researching cyberbullying has received increasing attention in the 21st century (e.g. aboujaoude et al. 2015; notar, padgett, and roden 2013). according to campbell and bauman (2018), ‘cyberbullying has been defined as intentional harmful behaviour carried out by a group or individuals, repeated over time, using modern digital technology to aggress against a victim who is unable to defend him/herself ’ (p. 3). http://dx.doi.org/10.25304/rlt.v31.2968 research in learning technology citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 3 (page number not for citation purpose) the above-mentioned two phenomena (increased digital education and the increased prevalence of cyberbullying during the covid-19 pandemic) have been assessed parallelly. this study intends to investigate both phenomena and the connection between them, since pupils faced both at the same time. furthermore, this study focuses primarily on the pupils’ perspective, as the informants of most of the above quoted school-related research were either teachers or caretakers. methodology: drama pedagogy as research in the online space drama is an educational method which was created in the middle of the twentieth century in england and combines education with theatre. drama is an ‘as if ’-type of activity originating from children’s games (e.g. neelands 1984), during which the role and the situation offered retain their imaginative nature. the facilitator usually controls the game from the inside, often whilst in role. the participants can also reflect on the game in light of their real-life experiences (davis 2014). the educational goal of drama is to gain new experiences, examine multiple points of view regarding a central question or theme and deepen understanding. in some cases, it can also be aimed at bringing about change, for example, in attitudes, expectations, social behaviours and developing relationships (e.g. cziboly & bethlenfalvy 2013). its effectiveness in this regard has been confirmed by international large-scale assessment (ilsa) as well (cziboly & dice consortium 2010; eriksson et al., 2014; gjærum, cziboly, and eriksson 2022). when using drama for focus group research, the latter aim is abandoned; consequently, a research oriented drama lesson differs greatly from a drama lesson held for explicitly educational (e.g. preventative) purposes. dramatic methods such as role-play are used more and more frequently in research (e.g. bethlenfalvy 2020). this is also apparent role-play research appearing as an independent chapter in the eight edition of the research methods in education (cohen, manion, and morrison 2018). o’sullivan explains role-play as a research technique ‘useful in studying complex situations, interactions and evolving decisions, and in which issues capable of different interpretations, negotiations or potential conflicts are included’ (2018, p. 606). o’sullivan emphasizes the importance of the ‘no-penalty zone’ (p. 611) that this type of research technique creates for participants. she also lists a variety of ways in which role-play can become parts of various research frameworks; these span from the researcher observing the role-playing to active participation in the fictional situation. the role-play-based drama focus groups conducted within our research relied on the researcher taking on a role and becoming an active participant and facilitator of the fiction. besides engaging in role-play, participants were also given a so-called ‘commission’ (heathcote and bolton 1995; kipling and hickley-moody 2015) as experts on their own life experiences: no one has more knowledge and information than they do, and we rely on them. the game’s facilitator plays the role of ‘being in need of help’. thus, applying drama methodology for research purposes provides a narrative framework that makes the conversation interesting for participants and involves them in collective thinking. if facilitators play their roles well, participants will have a stake in the game: to present their lives as authentically as possible whilst solving the fictional problems of fictional characters, with fiction offering them protection (o’neill 1995). a similar methodology had been employed in two earlier studies, both of which involved the authors: one examined drug use amongst primary school pupils in http://dx.doi.org/10.25304/rlt.v31.2968 á. cziboly et al. 4 citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 (page number not for citation purpose) slums (szécsi et al. 2017) and the other concerned digital device use in early childhood education (cziboly et al. 2023). during the lockdown, there have been different attempts to use drama in the online space (e.g. davis & philips 2020; lorenza 2020, etc.). drama is a genre that requires co-creation in a shared space, so adopting it to an online modality where we had to work without spatiality or eye-contact was the key limitation of the research. experiences from online teaching during the lockdowns (e.g. cziboly & bethlenfalvy 2020) have been utilized. we conducted three focus group sessions with 16 pupils (4 boys and 12 girls) in july 2020. they joined to the research on a voluntary basis and have been recruited by the organization of the third author from three rural counties. the ages of participants ranged between 11 and 15 years with one exception (the 6-year-old sibling of one participant also joined on of the groups), and they came from different socioeconomic backgrounds. all focus groups were conducted online by the second author. focus groups were anonymous, and pupils joined the online meeting with avatar names and had the opportunity to turn off their cameras. the focus group sessions were conducted according to the scenario outlined in the appendix (available online). huser, dockett, and perry (2022) emphasize children participants’ rights to both informed assent and indication of dissent. a written consent was requested from the parents prior to focus groups, where parents were briefly informed about the aims and the methodology, that audio recordings will be deleted once analysed, and we do not collect any personal data, so children will not be identifiable. children were also offered the possibility to leave any time. coding the participants’ answers was carried out partly according to the guide offered by global kids online (2016), partly in an inductive way, considering the patterns emerging from the text. the summaries were illustrated with quotations in a few places; these are not precise quotations, rather general summaries of several answers. the names, places, ages and all personal data that could be used to identify the participants have been changed. results and discussion experiences with distance learning distance learning versus going to school about 50% of participants were happy with distance learning, and the other half would have preferred going to school. when it came to the advantages, participants from all three groups mentioned that they did not have to go to school, they did not have to wake up early and their daily routine was freer as they could learn whenever they wanted to. one participant also remarked: ‘i liked that i did not have to meet a lot of people’. the disadvantages varied significantly more. the pupils mentioned missing peers and friends, missing sports and the act of walking to school. most participants mentioned that it was difficult to understand the subject material, and also teachers were not really teaching classes in most cases and were often unavailable. online education often felt chaotic for pupils: ‘we had to download a lot of apps for no reason’. the extent to which a given subject could be learned from textbooks depended on the subject itself, too. whilst ‘in geography, the textbook explains everything, we http://dx.doi.org/10.25304/rlt.v31.2968 research in learning technology citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 5 (page number not for citation purpose) learn it and we’ll know it’, mathematics and foreign languages are skills that need to be practised. for most students, these two subjects posed the most difficulties. all pupils had phones but only those coming from better-off families had laptops. poorer students took part in education and online activities with their phones. this reflects the trends highlighted by iivari et al. (2020): the abruptly forced digital transformation has reinforced digital divides that already existed. the majority had no issue with adapting to the new situation. there are only two recalled difficulties: they both had problems with learning to use a certain app and had not received any help. the amount of assignments participants almost unequivocally agreed that teachers had assigned them much more tasks than before. whilst ‘disciplining takes up a lot of time in a normal class’, during the lockdown, teachers seemingly thought that pupils had much more time. most pupils felt that this increase in workload was burdensome, and all of them thought that the learning materials were much more difficult to understand. participants in all three groups remarked that there had been significant differences between the subjects, and some teachers assigned way more work than others. all participants said that they had spent significantly more time in front of a screen. they used several platforms: ‘we had to keep checking every app because tasks could have been assigned at any time of day’. many pupils said that before the lockdown, they could not use their smartphones during the day, only in the morning and in the evening. but after switching to online learning, ‘we spent way more time on our phones because we were at home, and nobody checked on us’. several kids used this ‘legal’ screen time for having fun: ‘i often pretended to be learning but i was playing brawl stars. others did this, too’. since besides playing together, keeping in touch with others also took place online, the screen time allocated for learning often overlapped with playing and chatting: ‘we video chatted more than ever’. only one of the 16 participants felt she had spent less time learning than before, and others complained that they had to spend way more time learning. one of them said that to him it felt he had been studying three times as much. the competence of teachers to adapt all participants said that ‘teaching’ almost exclusively consisted of assigning exercises to be solved and returned; no real education took place: ‘they didn’t really ask questions or explain anything, they only told us what assignments we must do. sometimes we got an instructional video. we didn’t understand the material. (...). i think most teachers did not even teach’. pupils from a village school said that they did not even receive assigned tasks daily, only once a week. in all three groups, several pupils mentioned situations when they had asked a teacher for help, and all they had received in response was ‘read the textbook again’! another pupil complained that he had only asked for the access code to the google classroom group and had not even gotten an answer to that. there were some exceptions in all three groups. there was a teacher who tried to explain the material on the telephone when he was asked to, and others sent voice files or tried to teach in the same way they had before on platforms like discord. http://dx.doi.org/10.25304/rlt.v31.2968 á. cziboly et al. 6 citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 (page number not for citation purpose) platforms pupils experienced two different kinds of pedagogical strategies. in some schools, all assignments were ‘uploaded to a website and you could download them from there’ on a daily or weekly basis. in most schools, however, every teacher assigned and collected exercises on different platforms. the 16 participants mentioned the following 9 platforms: kréta (the electronic administration platform used in hungarian public  education), google classroom, discord, redmenta, zoom, facebook, gmail, messenger and mozaweb. participants disagreed on which platforms were the best. participants thought using discord and zoom the platforms suitable for live teaching would have been the most important in foreign languages and math. assessments in all three groups, several pupils mentioned redmenta as the platform used for assessments. across groups, kids agreed that cheating was a realistic option: ‘we video chatted with our classmates or dad helped with the physics test, and so on...’ one pupil remarked: ‘if you didn’t cheat, something was wrong with you’. some teachers found peculiar ways to compensate for cheating: ‘the teacher gave worse marks to kids she knew weren’t so good’. several pupils said they got worse marks because of technical difficulties. one of them said that the teacher had assigned a 4-page long copying exercise with a deadline of approximately 20 days. she did it that very day and then got a failing mark as the teacher forgot that she had handed it in. in other cases, teachers did not check their messenger regularly and sent the answers to tests to the wrong addresses. also, some teachers were very strict with deadlines and deducted half of the points if someone was even half a minute late. experiences varied as to whether it was possible to retake tests for better marks. some pupils felt they had had more chances to improve their grades, and teachers had been more lenient with deadlines than before the lockdown. others experienced the opposite: ‘when i wanted to take an oral test to get a better grade, i couldn’t just do it as before. i could not talk to the teacher in person, i could only send an email’. grading had been erratic; sometimes home assignments had been the most important factor, and sometimes activity during class or tests had been given more emphasis: ‘you never knew what to put more effort into’. risks of online activities harassment by friends on social media according to most participants, it is more common to get harassed by people you know than by strangers. ‘there are some weak people who like to be aggressive online because they would not be brave enough to do anything in person. and this makes them feel powerful’. negative comments can range widely from remarks on someone’s appearance (‘your hands are ugly, you cannot dance’) to verbal assault and threats. simple misunderstandings can also escalate, especially when the receiving party misunderstands the message due to the lack of emojis. participants from a small village school mentioned a situation where the mother of a slightly overweight, lonely girl wanted to help her become more popular by http://dx.doi.org/10.25304/rlt.v31.2968 research in learning technology citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 7 (page number not for citation purpose) recording her as she danced and put it on youtube. opinions varied whether or not the girl was happy about this, some said: ‘she did not mind because she wanted attention’ or ‘she was just having a good time, and her mom recorded it’, whilst others noted that ‘the whole school watched, and everyone was mocking her and saying it was gross’. negative comments sometimes show a complete lack of empathy. ‘i have heard about a girl who went nuts after her mom died. and instead of offering help, others commented that they hope she would die too’. participants admitted to often laughing at content depicting someone being humiliated. at the same time, they are fully aware that this hurts the other person. several children mentioned instances where someone uploaded a photo or video of someone else and tagged them without asking their permission. for example, many kids post awkward pictures on their peers’ timelines on their birthdays to make them feel bad. also, in some cases, children took photos or videos of others (for example, the 4-year-old brother of one participant) without them being aware and shared them online, sometimes after editing them with faceapp. others say it is much more common that ‘they tag you when you are not even in the picture’. there was a situation that started as a joke and ended up involving the police. an eighth grader was using the facebook on a computer in the library. he went to the bathroom without logging out, and whilst he was away, another boy used his profile to write perverse messages to a girl. the girl cried and told her father who came into school and scolded the boy aggressively then proceeded to file charges. ‘a policeman came to the school and told the culprit that if he had the nerve to write such things on facebook then he should have the nerve to write them on the blackboard. the boy almost cried’. online interactions with strangers almost all participants said that they had been sent friend requests by strangers at some point. however, all three groups were quick to note that such friend requests should be deleted, and, thus, there is no problem. ‘once a lot of african men started to follow me. around 20 per minute. (...) i deleted all of them and then nothing happened’. a high school student told us that his father – after telling her to come to him if strangers texted her – created a new account in secret and sent her a heart emoji. ‘he wanted to test out whether i’d tell him. i didn’t tell him; i only blocked the profile. later he told me it was him. i was scared because i was only 11’. two groups mentioned dating advances on facebook. in one case, it later turned out that the boy had another girlfriend, which was a big disappointment for the girl involved. as participants said, it is easy to manipulate online, anyone can post anything, and ‘it’s easy to play with words online, to make others believe what you want them to believe’. this typically involves older boys who target 13–15-year-old girls. many participants warn that a lot of things should be discussed before meeting in person; one must get information on whether or not the other is a decent person. at the same time, it is not healthy to dig too much. the more conscious children feel it is safer and more controllable when communication with strangers starts on an anonymous platform. one participant noted that anonymous commenting on reddit can lead to a fruitful and friendly discussion without revealing one’s true identity. http://dx.doi.org/10.25304/rlt.v31.2968 á. cziboly et al. 8 citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 (page number not for citation purpose) online harassment by strangers however, anonymous communication is not always safe. with the app tellonym, someone with an instagram account (with a name and a picture) can ask for anonymous feedback (how do i look? etc.), and followers can write anonymous answers. most answers are typically rude and negative. anotalk is another anonymous app where two random strangers are connected via the application. in one group, more participants mentioned a girl ‘who was hungry for boys’ and who supposedly had gotten to know someone via this app. boys often use these platforms to write ‘perverse’ messages to girls. even the 6-year-old participant had such an experience: ‘we were watching videos, and a man wrote that he was coming for us. i got scared and ran away to mom’. two girls recounted that several strangers sent them friend requests due to their profiles, but they deleted them. ‘if you post pictures where you look attractive, you can even get a lot of nasty comments’. in some rare cases, anonymous comments can be positive. one participant texted girls who were her friends on facebook anonymously write that they were beautiful, just to see their reactions. one boy mentioned that he was once contacted by another boy on messenger. he answered, trying to uncover who he was but that other boy was being mysterious. when he wrote that he would send a picture, i blocked him to stop him from sending me a... nasty picture. (...) then he became my friend on facebook as he had sent me a friend request. and i saw he was like 40 or something. the boy deleted the man from his friends list and told his mother. the same participant recounted some other extreme experiences. there is a site called chat alternative, it’s like anotalk but with video. you cannot go online in the evening because there are 60-year-old perverts, and others, who kind of have a good time... should i say it? they masturbate in front of the webcam and show it to others. in the case of one of the girls, online harassment bordered on criminality: a boy she did not know sent her a friend request and then threatened to abduct her and murder her. ‘i got really scared, i threw away my phone and the screen broke’. based on his messages, he knew a lot about the girl and her family. after she told her father about it, he visited the boy at his home and filed a police charge. ‘dad was also really scared because he has three daughters’. parental involvement we have discussed four instances of parents getting involved in their children’s online life either voluntarily (the mother uploading the video of her dancing daughter, and the father testing his daughter via sending her a heart emoji) or out of necessity (the two fathers reacting to online harassment). in most cases, parental involvement is less drastic. in one group, multiple 13-year-old participants noted that a lot of their classmates were being monitored by their parents who wanted to know who they chatted with. they disagreed with this practice and opined this was only reasonable until the age http://dx.doi.org/10.25304/rlt.v31.2968 research in learning technology citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 9 (page number not for citation purpose) of 10. there was some discussion over whether or not a parent is entitled to monitor their children’s online activities. the participants concluded that parents could get involved if the kids agree to it, but this will likely end once these kids reach adolescence as they will be reluctant to share their passwords. several participants noted that they discussed problems of this nature with their friends instead of their parents. however, there is a consensus that parents should communicate with their children continuously. they should not question every step they take, but it is worth asking the child in an intimate situation what they usually do online or whether they have online friends. they also recommend parents make clearcut rules (what they can and cannot upload, for example) and be available when there is something wrong when the child does not know something or is being harassed. conclusions participants almost unequivocally said that online education mostly consisted of assigning exercises; that teachers did not communicate with one another; and mostly gave more assignments than they had the time to cover in the course of ‘normal’ education. whilst some teachers tried to help their pupils in this new situation, most teachers did not do so or barely consulted with them. in terms of assessment, it was rather uneven. it was easier to cheat on tests, but multiple participants recounted that in their final assessment, teachers often used earlier academic performance as a reference point. almost all pupils sorely missed communication and being involved in designing the learning process. these results partially confirm the findings of a recent meta-analysis in the field (betthäuser, bach-mortensen, and engzell 2022); however, results also highlight the pupils’ specific viewpoints in cardinal questions. during online learning, playing online and keeping in touch with peers all took place simultaneously. pupils had to register on several platforms, including some which, based on research, are known to pose risks such as zoom (e.g. kagan, alpert, and fire 2020) or facebook (e.g. dredge, gleeson, and de la piedad garcia 2014). pupils spent much more time online, typically on their own, without teacher’s supervision. in their free time, children mostly play, watch movies or videos and use social media platforms, but they might also use some potentially dangerous less-known apps such as anotalk, tellonym and chat alternative. all participants in all three focus groups were seemingly aware of the dangers and rules of internet use. also, all of them had encountered situations where this knowledge had been needed. several stories were recalled where pupils had been harassing each other online or where adult strangers had been texting the participants. stories ranged from friend requests from unknown persons to middle-aged men sending perverse texts. in terms of cyberbullying, most participants only knew different types of teasing such as taunting someone in a comment or tagging each other on awkward pictures. two more serious stories were mentioned; parental involvement was needed in both. knowing that during the lockdown, the adults normally responsible for the children during school hours (the teachers) practically left pupils to their own devices, and this situation seemed especially risky. members of all three focus groups missed and advocated the need for regular communication with adults, that is, the teachers and parents. as adults only met the children’s needs in part, they, in turn, also only followed the rules they otherwise knew well to a certain extent. it was this combination that might have led to the more serious cases. http://dx.doi.org/10.25304/rlt.v31.2968 á. cziboly et al. 10 citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 (page number not for citation purpose) future research could compare the experiences of pupils and teachers, as teachers also experienced severe difficulties according to a recent meta-analysis (ozamiz etxebarria et al. 2021), or even test drama-based methodologies where teachers and pupils could respond to each other’s perceptions, interpretations and opinions in an anonymous online space. research results could also contribute to the development of innovative drama-based interventions educating children to cope with online risks more effectively and assertively. declaration of interest statement the authors report there are no competing interests to declare. data availability statement the data that support the findings of this study are available on request from the corresponding author. the data are not publicly available due to their containing information that could compromise the privacy of research participants. references aboujaoude, e., et al., (2015) ‘cyberbullying: review of an old problem gone viral’, journal of adolescent health, vol. 57, no. 1, pp. 10–18. doi: 10.1016/j.jadohealth.2015.04.011 barlett, c. p., et al., (2021) comparing cyberbullying prevalence and process before and during the covid-19 pandemic. the journal of social psychology, vol. 161, no. 4, pp. 408–418. doi: 10.1080/00224545.2021.1918619 betthäuser, b., bach-mortensen, a. & engzell, p. (2022) ‘a systematic review and meta-analysis of the impact of the covid-19 pandemic on learning’, liepp working paper, vol. 134, p. ffhal-03681201 bethlenfalvy, a. (2020) living through extremes in process drama, kre & l’harmattan, budapest. doi: 10.56037/978-2-343-20662-2 bogdandy, b., tamas, j. & toth, z. (2020) ‘digital transformation in education during covid-19: a case study’, in 2020 11th ieee international conference on cognitive infocommunications (coginfocom), mariehamn, finland, pp. 000173–000178, ieee. doi: 10.1109/coginfocom50765.2020.9237840 buckingham, d. (2020) ‘epilogue: rethinking digital literacy: media education in the age of digital capitalism’, digital education review, vol. 37, pp. 230–239. doi: 10.1344/ der.2020.37.230-239 campbell, m. & bauman, s. (2018) ‘cyberbullying: definition, consequences, prevalence’, in reducing cyberbullying in schools, academic press, cambridge, pp. 3–16. cohen, l., manion, l. & morrison, k. (2018) research methods in education, 8th edn., routledge, london. crawford, j., et al., (2020) ‘covid-19: 20 countries’ higher education intra-period digital pedagogy responses’, journal of applied learning & teaching, vol. 3, no. 1, pp. 1–20. doi: 10.37074/jalt.2020.3.1.7 cziboly, á. & bethlenfalvy, á. (2013) színházi nevelési programok kézikönyve, l’harmattan [online] available at: https://www.szinhazineveles.hu/wp-content/uploads/2017/06/szinhazi_nevelesi_kezikonyv.pdf cziboly, a. & bethlenfalvy, a. (2020) ‘response to covid-19 zooming in on online process drama’, research in drama education: the journal of applied theatre and performance, vol. 25, no. 4, pp. 645–651. doi: 10.1080/13569783.2020.1816818 http://dx.doi.org/10.25304/rlt.v31.2968 https://doi.org/10.1016/j.jadohealth.2015.04.011 https://doi.org/10.1080/00224545.2021.1918619 https://doi.org/10.56037/978-2-343-20662-2 https://doi.org/10.1109/coginfocom50765.2020.9237840 https://doi.org/10.1344/der.2020.37.230-239 https://doi.org/10.1344/der.2020.37.230-239 https://doi.org/10.37074/jalt.2020.3.1.7 https://www.szinhazineveles.hu/wp-content/uploads/2017/06/szinhazi_nevelesi_kezikonyv.pdf https://www.szinhazineveles.hu/wp-content/uploads/2017/06/szinhazi_nevelesi_kezikonyv.pdf https://doi.org/10.1080/13569783.2020.1816818 research in learning technology citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 11 (page number not for citation purpose) cziboly, á. (ed.) & dice consortium. (2010) the dice has been cast. a dice resource  – research findings and recommendations on educational theatre and drama. dice consortium [online] available at: https://www.dramanetwork.eu/file/policy%20paper%20 long.pdf cziboly, á., et al., (2023) ‘a drama-based examination of hungarian kindergarteners’ activities, knowledge and attitudes relating to digital communication’. [manuscript submitted for publication]. daniel, s. j. (2020) ‘education and the covid-19 pandemic’, prospects, vol. 49, no. 1, pp. 91–96. davis, d. (2014) imagining the real. towards a new theory of drama in education, ioe press/ trentham books, stoke on trent. doi: 10.1007/s11125-020-09464-3 davis, s. & phillips, l. g. (2020) ‘teaching during covid 19 times – the experiences of drama and performing arts teachers and the human dimensions of learning’, nj, vol. 44, no. 2, pp. 66–87. doi: 10.1080/14452294.2021.1943838 dredge, r., gleeson, j. & de la piedad garcia, x. (2014) ‘presentation on facebook and the risk of cyberbullying victimisation’, computers in human behaviour, vol. 40, pp. 16–22. doi: 10.1016/j.chb.2014.07.035 eriksson, s. a., heggstad, k. m., heggstad, k. & cziboly, a. (2014) ‘rolling the dice’. introduction to the international research project drama improves lisbon key competences in education’, research in drama education: the journal of applied theatre and performance, vol. 19, no. 4, pp. 403–408. doi: 10.1080/13569783.2014.954814 eu kids online. (2014) the meaning of online problematic situations for children: results of qualitative cross-cultural investigation in nine european countries, eu kids online, london school of economics and political science [online] available at: http://eprints.lse. ac.uk/56972/1/eu_kids_online_report_online_problematic_situations_for_children_ june2014.pdf gjærum, r. g., cziboly, a. & eriksson, s. a. (2022) ‘ilsa in arts education: the effect of drama on competences’, in international handbook of comparative large-scale studies in education, eds t. nilsen, a. stancel-piątak & j.e. gustafsson, springer international handbooks of education. springer, cham, pp. 547–577. global kids online (2016) global kids online research toolkit [online] available at: http:// globalkidsonline.net/wp-content/uploads/2016/04/procedure-for-coding-and-analysisfinal-1.pdf grek, s. & landri, p. (2021) ‘education in europe and the covid-19 pandemic’, european educational research journal, vol. 20, no. 4, pp. 393–402. doi: 10.1177/1474904121 1024781 hallin, a. e., et al., (2022) ‘no learning loss in sweden during the pandemic evidence from primary school reading assessments’, international journal of educational research, vol. 114, p. 102011. doi: 10.1016/j.ijer.2022.102011 händel, m., et al., (2020) ‘digital readiness and its effects on higher education students’ socio-emotional perceptions in the context of the covid-19 pandemic’, journal of research on technology in education, vol. 54, no. 2, pp. 267–280. doi: 10.1080/15391523.2020.1846147 heathcote, d. & bolton, g. (1995) drama for learning: dorothy heathcote’s mantle of the expert approach to education, heinemann, london. huser, c., dockett, s. & perry, b. (2022) ‘young children’s assent and dissent in research: agency, privacy and relationships within ethical research spaces’, european early childhood education research journal, vol. 30, no. 1, pp. 48–62. ito, m., et al., (2008) living and learning with new media: summary of findings from the digital youth project, macarthur foundation, the mit press, cambridge/london. kagan, d., alpert, g. f. & fire, m. (2020) ‘zooming into video conferencing privacy and security threats’, arxiv:2007.01059. doi: 10.48550/arxiv.2007.01059 kang, b. (2021) ‘how the covid-19 pandemic is reshaping the education service’, the future of service post-covid-19 pandemic, vol. 1, pp. 15–36. doi: 10.1007/978-981-33 4126-5_2 http://dx.doi.org/10.25304/rlt.v31.2968 https://www.dramanetwork.eu/file/policy%20paper%20long.pdf https://www.dramanetwork.eu/file/policy%20paper%20long.pdf https://doi.org/10.1007/s11125-020-09464-3 https://doi.org/10.1080/14452294.2021.1943838 https://doi.org/10.1016/j.chb.2014.07.035 https://doi.org/10.1080/13569783.2014.954814 http://eprints.lse.ac.uk/56972/1/eu_kids_online_report_online_problematic_situations_for_children_june2014.pdf http://eprints.lse.ac.uk/56972/1/eu_kids_online_report_online_problematic_situations_for_children_june2014.pdf http://eprints.lse.ac.uk/56972/1/eu_kids_online_report_online_problematic_situations_for_children_june2014.pdf http://globalkidsonline.net/wp-content/uploads/2016/04/procedure-for-coding-and-analysis-final-1.pdf http://globalkidsonline.net/wp-content/uploads/2016/04/procedure-for-coding-and-analysis-final-1.pdf http://globalkidsonline.net/wp-content/uploads/2016/04/procedure-for-coding-and-analysis-final-1.pdf https://doi.org/10.1177/14749041211024781 https://doi.org/10.1016/j.ijer.2022.102011 https://doi.org/10.1080/15391523.2020.1846147 https://doi.org/10.48550/arxiv.2007.01059 https://doi.org/10.1007/978-981-33-4126-5_2 https://doi.org/10.1007/978-981-33-4126-5_2 á. cziboly et al. 12 citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 (page number not for citation purpose) kipling, a. & hickey-moody, a. c. (2015) ‘dorothy heathcote: practice as a pedagogy of resistance’, in arts, pedagogy and cultural resistance: new materialisms, eds a. hickeymoody & t. page, rowman and littlefield, london/new york, pp. 59–78. iivari, n., sharma, s. & ventä-olkkonen, l. (2020) ‘digital transformation of everyday life how covid-19 pandemic transformed the basic education of the young generation and why information management research should care?’, international journal of information management, vol. 55, no. 102183. doi: 10.1016/j.ijinfomgt.2020.102183 livingstone, s. (2014) ‘developing social media literacy: how children learn to interpret risky opportunities on social network sites’, communications: the european journal of communication research, vol. 39, no. 3. pp. 283–303. doi: 10.1515/commun-2014-0113 livingstone, s. & helsper, e. (2008) ‘parental mediation and children’s internet use’, journal of broad-casting & electronic media, vol. 52, no. 4, pp. 581–599. doi: 10.1080/08838150 802437396 livingstone, s., mascheroni, g. & staksrud, e. (2018) ‘european research on children’s internet use: assessing the past and anticipating the future’, new media and society, vol. 20, no. 3, pp. 1103–1122. doi: 10.1177/1461444816685930n lorenza, l. m. (2020) ‘collaborative performance during lockdown: combining devised drama with film concepts to present cyberdrama during the covid lockdown in australia’, nj, vol. 44, no. 2, pp. 88–105. doi: 10.1080/14452294.2021.1939506 mustapha, i., et al., (2021) ‘effectiveness of digital technology in education during covid-19 pandemic. a bibliometric analysis’, international journal of interactive mobile technologies, vol. 15, no. 8, pp. 136–154. doi: 10.3991/ijim.v15i08.20415 n. kollár, k. (2021) ‘az online oktatás tapasztalatai és gyakorlata a pedagógusok nézőpontjából’, iskolakultúra, vol. 31, no. 2, pp. 23–54. doi: 10.14232/isk-kult.2021.02.23 neelands, j. (1984) making sense of drama: a guide to classroom practice, heinemann, london. németh, sz. & rajnai, r. (2021) ‘hátrányos helyzetű tizenévesek médiahasználata a koronavírus-járvány első hulláma alatt és azt megelőzően a koronavírus-járvány első hullámát követő empirikus kutatás tapasztalatai’, régió, vol. 29, no. 4, pp. 75–90. doi: 10.17355/rkkpt. v29i4.75 németh, sz., et al., (2021) ‘a karanténoktatás tapasztalatai szegregátumban és azon kívül: 18 tanulói és szülői fókuszcsoportos beszélgetés alapján’, iskolakultúra, vol. 31, no. 6, pp. 17–34. doi: 10.14232/iskkult.2021.06.17 notar, c. e., padgett, s. & roden, j. (2013) ‘cyberbullying: a review of the literature’, universal journal of educational research, vol. 1, no. 1, pp. 1–9. doi: 10.13189/ujer.2013.010101 o’neill, c. (1995) drama worlds: a framework for process drama, heinemann, london. o’sullivan, c. (2018) ‘role-play and research’, in research methods in education, 8th edn, eds l. cohen, l. manion & k. morrison, routledge, london, pp. 606–627. osváth, a. & papp, z. a. (2020) ‘digitális fordulat az oktatásban? a digitális távoktatás tapasztalatai, lehetséges következményei’, szellem és tudomány, vol. 11, no. 2, pp. 179–205 [online] available at: https://atti.uni-miskolc.hu/docs/szellemestudomany2020_2.pdf ozamiz-etxebarria, n., et al., (2021) ‘prevalence of anxiety, depression, and stress among teachers during the covid-19 pandemic: a rapid systematic review with meta-analysis’, brain sciences, vol. 11, no. 9, p. 1172. doi: 10.3390/brainsci11091172 shin, s. y. & choi, y. j. (2021) ‘comparison of cyberbullying before and after the covid-19 pandemic in korea’, international journal of environmental research and public health, vol. 18, no. 19, p. 10085. doi: 10.3390/ijerph181910085 sonnenschein, s., et al., (2022) ‘“this will likely affect his entire life”: parents’ views of special education services during covid-19’, international journal of educational research, vol. 112, p. 101941. doi: 10.1016/j.ijer.2022.101941 szécsi, j., et al., (2017) ‘droghasználat a peremhelyzetben élő általános iskolás gyerekek körében’, in a magyar addiktológiai társaság xi. országos kongresszusa: előadáskivonatok, eds k. felvinczi & a. eisinger, magyar addiktológiai társaság, siófok, pp. 52–53. http://dx.doi.org/10.25304/rlt.v31.2968 https://doi.org/10.1016/j.ijinfomgt.2020.102183 https://doi.org/10.1515/commun-2014-0113 https://doi.org/10.1080/08838150802437396 https://doi.org/10.1080/08838150802437396 https://doi.org/10.1177/1461444816685930n https://doi.org/10.1080/14452294.2021.1939506 https://doi.org/10.3991/ijim.v15i08.20415 https://doi.org/10.14232/isk-kult.2021.02.23 https://doi.org/10.17355/rkkpt.v29i4.75 https://doi.org/10.17355/rkkpt.v29i4.75 https://doi.org/10.14232/iskkult.2021.06.17 https://doi.org/10.13189/ujer.2013.010101 https://atti.uni-miskolc.hu/docs/szellemestudomany2020_2.pdf https://doi.org/10.3390/brainsci11091172 https://doi.org/10.3390/ijerph181910085 https://doi.org/10.1016/j.ijer.2022.101941 research in learning technology citation: research in learning technology 2023, 31: 2968 http://dx.doi.org/10.25304/rlt.v31.2968 13 (page number not for citation purpose) unicef (2020) what have we learnt? overview of findings from a survey of ministries of education on national responses to covid-19 [online] available at: https://data.unicef. org/resources/national-education-responses-to-covid19/ utemissova, g. u., danna, s. & nikolaevna, v. n. (2021) ‘cyberbullying during the covid-19 pandemic’, global journal of guidance and counselling in schools: current perspectives, vol. 11, no. 2, pp. 77–87. doi: 10.18844/gjgc.v11i2.5471 williamson, b., macgilchrist, f. & potter, j. (2021) ‘covid-19 controversies and critical research in digital education’, learning, media and technology, vol. 46, no. 2, pp. 117–127. doi: 10.1080/17439884.2021.1922437 yang, f. (2021) ‘coping strategies, cyberbullying behaviours, and depression among chinese netizens during the covid-19 pandemic: a web-based nationwide survey’, journal of affective disorders, vol. 281, pp. 138–144. doi: 10.1016/j.jad.2020.12.023 http://dx.doi.org/10.25304/rlt.v31.2968 https://data.unicef.org/resources/national-education-responses-to-covid19/ https://data.unicef.org/resources/national-education-responses-to-covid19/ https://doi.org/10.18844/gjgc.v11i2.5471 https://doi.org/10.1080/17439884.2021.1922437 https://doi.org/10.1016/j.jad.2020.12.023 editorial standards one of the aims of alt is to promote good practice in the use of learning technology within higher education. few would not subscribe to this aim, but proclaiming it without a full awareness of the problems it entails is of the same order as proclaiming one's commitment to peace without further comment. except for the absolute pacifist, being against war does not mean being against it at any price, but rather being committed to ensuring circumstances in which war will not occur. so it is with good practice in educational technology, which can be achieved only if circumstances are propitious. such circumstances include sufficient funds, and a willingness in both teacher and learner to accept in whole or in part a technological route — good practice is unlikely to be achieved if technology has been incorporated into the curriculum merely in order to ensure that funds already spent do not appear to have been wasted. above all, in my view, good practice assumes that users of learning technology are able to concentrate on learning without (necessarily) thinking about the technology, since if the technology cannot be made to work transparently, actual practice will at best lag behind ideal practice, and at worst be abandoned altogether. impatience is a barrier to learning, and particularly if it is the result of struggling with the learning tools themselves. if the teacher or learner is constantly having to tweak the technology, or ending up with a half-baked implementation because the setting-up process has proven too difficult, the learning tool may well be left to gather dust. fax has rapidly become almost the staff of life in both commercial and educational institutions because it is on the whole reliable, and conforms to a single standard. generally speaking, however, leading-edge technology is unreliable and unfriendly. when television sets were first seen in homes, they required constant adjustment and professional attention, and the television repair business boomed. today, repairing television sets is a cottage industry, and not only because replacement may be less expensive than repair, but also because modern sets are technologically robust. what is more, in good measure television is now universally popular because the viewer knows that once a set has been bought, he or she merely has to plug in an aerial, tune in the stations, and all will work as expected. television sets would be less numerous than they are if at each replacement the viewer had to go through a complex routine to make the aerial 'handshake' with the new set, only to find that five times out of ten this task could not be accomplished without expert assistance, and that three times out of the remaining five it could not be accomplished at all. this is, unfortunately, much the case with interactive-multimedia technology. while word-processing is now at a stage where the user can be confident that the software and %ardware will work well together, interactive applications involving images and sound appear to-be recipes for frustration. the de facto standard in sound cards for ibm-compatibles, for example, is westpoint creative's soundblaster, which forms part of that de jure standard known as mpc (multimedia pc). an mpc system, according to those who defined the standard, is an ibm or ibm-compatible pc with a specified minimum central processor and graphics card, specified minimum amounts of internal memory and physical storage space, a cd-rom drive and a soundblaster or soundblaster-compatible sound card. given this agreed specification, a user might imagine that an application involving sound, developed for the mpc standard, would run perfectly on any mpc system. matters are not that simple. 2 alt-j volume 1 number i take a pc which conforms to the mpc standard, though not actually marked mpc (there are many such systems in use, while 'genuine' mpcs are rare). contact the producer or distributor of a commercial product intended for use on an mpc system and, before paying for it, ask for a copper-bottomed guarantee that it will work as it should, first time and without professional help with the installation. if the producer is honest, you are likely to be told that no such guarantees can be given, that it may well come down to some trial and error, and that when all the dust has cleared there is still a significant chance that the product may not work at all. such was the recent experience of a colleague of mine who was told by the distributors of a well-known american interactive-learning product that they could not guarantee first-time success, in particular with the audio element of the product, because their own technical staff had themselves spent several frustrating hours installing it on a genuinely marked mpc. first, they had had to change the sound card since it had turned out to be faulty only when this particular software product had been tried with it. then time had been spent setting the system parameters so that the pc's internal memory was being handled properly, and discovering the right interrupts. how could they be sure, they argued with commendable frankness, that the product would run properly on a pc system which had been built up over a period of time, whatever standard such a system now supposedly conformed to? indeed, when my colleague went ahead and bought the product on a sale-or-return basis, it would not run. and, not surprisingly, the installation manual was peppered with pass-the-buck advice such as: 'if you encounter problems, consult your sound card manual' enough to make a beginner give up before beginning. it took many telephone calls and two visits from the suppliers to adjust everything so that the controller software could be loaded and run. the product was then tested on some students. it crashed after less than 15 minutes, displaying that windows message which offers as little hope as dante's gates of hell: the one that informs you of an unrecoverable error and impending termination of the application (once you have clicked on ok microsoft at least has a sense of irony). my colleague's final comment on the episode was that she would wait until the technology had matured. the world of the apple macintosh suffers less from problems of standards than that of the pc, which is no doubt partly why some educational institutions have opted for it. but there is little chance that the macintosh or any other non-ibm system will oust the pc from its firm foothold in the majority of higher-educational establishments. many interactive-multimedia learning tools will thus have to struggle for a while to establish themselves on a platform which was not originally designed to handle them, and which has spawned innumerable deviations from its so-called standard in the area of graphics alone, there are more than 30 different pc image formats. yet despite all this indeed, because of it those of us who believe in the future of computer-based learning technology (and not merely because pressure of student numbers might make it inevitable) need not simply wait for a real standard to emerge out of commerce or entertainment in the way such standards have emerged in the past. on the contrary, academics can be a major force in determining an accepted standard if they are willing to keep an open mind (in other words, to eschew the not-invented-here and the even more childish my-system-is-better-than-yours syndromes), to co-operate in forming an upgradable standard which everyone can and will accept, and then to promote it by showing what it can achieve. that will be a good start for the good practice we in a l t wish to foster. gabriel jacobs 3 doi:10.3402/rlt.v24.30357 original research article to byod or not to byod: factors affecting academic acceptance of student mobile devices in the classroom christopher graham mckercher gillies* faculty of education, health and community, liverpool john moores university liverpool, uk (received 27 november 2015; final version received 13 september 2016) this article reports on factors affecting local academic acceptance of bring your own devices (byod). a review of the literature revealed a paucity of studies that have explored the complex factors that affect academic use and intention to use mobile devices in the classroom, with even less exploring truly ubiquitous and varied personal devices as opposed to supplied institutional or research study sets. a detailed qualitative investigation with 14 academics was undertaken, drawing upon and aiming to compliment mature acceptance research. firstly by employing a focus group to identify initial psychological factors and the relevance of acceptance theories to the local context. then, secondly by using in-depth semi-structured interviews, shaped by acceptance categories, to identify a breadth of psychological factors affecting faculty use and intention to use byod. this small-scale study found clear distinctions in local academic perceptions of byod compared with faculty devices and reported a range of factors that appeared to distinctly affect local academic acceptance of byod. keywords: higher education; bring your own devices; innovation diffusion theory; unified theory of acceptance and use of technology introduction much has been made of the educational opportunities ubiquitous mobile devices present to academics in higher education (he). yet, if personal mobile devices are to be effectively integrated into the autonomous classroom practices of academics, then academics must first accept these innovations. this small-scale exploratory study sets out to investigate factors affecting acceptance of bring your own devices (byod) in one uk he faculty of education. to help identify the complex multivariate factors shaping academic acceptance, two popular acceptance theories were checked against focus group findings and then drawn upon to shape semi-structured interview questions. these theories were the mature, innovation diffusion theory (idt) which due to its age and popularity has been well tested, and the contemporary, unified theory of acceptance and use of technology (utaut) which combines a range of acceptance constructs into one comprehensive model (rogers 2003; venkatesh, et al. 2003). *corresponding author. email: c.g.gillies@ljmu.ac.uk responsible editor: carlo perrotta, university of leeds, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 c.g.m. gillies. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 literature review mobile devices for learning ofcom (2015) describe the uk as a ‘smartphone society’ and report that 90% of 16�24 year-olds own a smartphone and that over half of uk households have access to a tablet computer. internationally in he, the educause and new media consortium (nmc) annual horizon reports have repeatedly identified mobile devices generally (johnson et al. 2013) and byod specifically (johnson et al. 2016), as likely to have a substantial impact on he institutions in the near term. but, the uk 2014 universities and colleges information systems association (ucisa) survey identified mobile technologies as placing the greatest demands on uk learning technology support (walker et al. 2014). given this context, it is unsurprising to note that mobile or ubiquitous learning is one of the fastest growing research areas in the field of information communication technology in education (hwang and tsai 2011; pegrum, oakley, and faulkner 2013). indeed, mobile learning researchers have identified a range of advantages including its potential for student-centred pedagogies (kukulska-hulme 2013) and improved student engagement and motivation (backer 2010; enriquez 2010; pegrum, oakley, and faulkner 2013; thomas, o’bannon, and bolton 2013). personal devices or supplied devices intriguingly, despite the apparent ubiquity of mobile devices, there is a tendency in the literature to draw on devices supplied and controlled either by institutions or by research teams (wright and parchoma 2011). this is surprising as it leaves a knowledge gap in relation to those particular resources that academics have the greatest access to, namely student devices. within the mobile device literature, whilst there is a paucity of peer-reviewed studies that explore byod in action (stavert 2013), those studies that exist tend to draw upon ubiquitous student mobile devices. because of this, benefits and drawbacks of using personal devices can be more readily discerned. for example, it has been claimed that byod can reduce costs for institutions (dykes and knight 2012; stavert 2013), and byod is a better fit with current student expectations and lifestyles (johnson et al. 2016). it has also been suggested that personal ownership and choice of mobile devices further improve student engagement and commitment (crown fibre holdings ltd 2012; naismith et al. 2004). in contrast, equity of access (pegrum, oakley, and faulkner 2013; stavert 2013), increased personal distractions (bayless, clipson, and wilson 2013; naismith et al. 2004; stavert 2013) and the risk of theft (stavert 2013) have been identified as problems with byod. whilst dahlstrom and difilipo (2013) argued that technical support and guidance on using byod for learning within institutions is lacking. a range of devices in the classroom (crown fibre holdings ltd 2012; parsons 2013), difficulty distributing and storing student work (parsons 2013) and a perceived loss of control have also been judged problematic (cristol and gimbert 2013). the large number of concerns identifiable across byod studies might indicate why there is an inclination for supplied and controlled devices within the mobile learning literature. yet, with ubiquity frequently identified as a key boon of mobile devices, more educational research needs to be undertaken that draws upon personal devices. c.g.m. gillies 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 technology acceptance theories another notable imbalance within the mobile device literature is the dominance of studies exploring student perspectives as opposed to academic perspectives (alrasheedi and capretz 2015a; hwang and tsai 2011). this is surprising considering the autonomy of academics in relation to their classroom activities (guest and clinton 2007; jacobsen 1998). clearly, a greater understanding of how academic decisions are made with regard to accepting mobile devices in their classroom practice could be valuable. fortunately, a substantial body of literature exists that explores the process of technology acceptance. acceptance theories attempt to explain either the expressed intention to use or the actual use itself of a system, idea or technology either at an individual or organisational level (venkatesh et al. 2003). there are a range of models and theories, which in turn report a range of aspects or determinants believed to effect individual acceptance; these are as diverse as individual attitudes, personal experience, social norms and contextual influences (rogers 2003; venkatesh et al. 2003). acceptance is a mature empirical area of study dominated by quantitative approaches (williams et al. 2009). a commonly suggested rationale for these models is that they are useful for ‘managers needing to assess the likelihood of success for new technology introductions and help them understand the drivers of acceptance in order to proactively design interventions’ (venkatesh et al. 2003, p. 426). such pragmatism certainly has value, but with claims that such models can explain between 17 and 70% of the variance in individual acceptance, across a wide range of innovations and diverse contexts, some caution is required (venkatesh et al. 2003). indeed, such generalisations have been contested within the acceptance literature itself (thomas, singh, and kemuel 2013), and the numerous models and determinants are perhaps indicative of the difficulties generalising the complexities, vagaries and occasionally irrational perspectives of individuals across a range of diverse contexts. with this in mind and the dominance of quantitative approaches, it is perhaps unsurprising that within the acceptance field there have been calls for more qualitative studies to be undertaken (hazen et al. 2012; williams et al. 2009). essentially, acceptance models present well-considered philosophies of change which might provide useful constructs to help interrogate the complexity of local academic acceptance (or not) of personal mobile devices. this study draws upon two specific theories, idt and utaut. the technology acceptance model (tam) (davis 1986) was also considered but because of the parsimonious nature of the model, offering two key determinants, which heavily overlap with determinants proferred by both idt and utuat, it was left out. what follows is a brief summary of these two models. idt is a mature theoretical framework developed since the 1960s and grounded in the reference discipline of sociology (hazen et al. 2012). it is both an exploratory theory designed to help researchers investigate the process of acceptance over time, as well as an instructional theory promising practitioners who employ it, insights into how to speed up or slow down the rate of innovation diffusion. rogers, one of the most influential researchers in the field (sahin and thompson 2006), argues that there are four main elements in the diffusion of new ideas: the perceived attributes of the innovation, the bounded social system, communication channels and time (rogers 2003). in relation to perceived attributes, rogers identifies five key categories: relative advantage, compatibility, complexity (now ease of use), trialability and observability research in learning technology citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 (rogers 2003). over time, these attributes have been expanded by other acceptance researchers to include: result demonstrability, image, voluntariness (tabata and johnsrud 2008), computer attitude and self-efficacy (lee, kozar, and larsen 2003). a useful theoretical breakdown of the original idt categories in relation to mobile learning can be found in mac callum (2010). in contrast, and because of the wide variety of models in the acceptance literature, venkatesh et al. (2003) set out to develop a contemporary unified theory that would capture the essential elements from all of them. after testing eight theoretical models, including idt and tam, venkatesh et al. (2003) recognised four key determinants moderated by age, gender, experience (with the technology) and setting (mandatory or voluntary), that they believed would be most significant for predicting behavioural intention. these determinants or categories were performance expectancy, effort expectancy, social influence and facilitating conditions. after empirically testing their model in three different organisations, venkatesh et al. (2003) claimed that across contexts utaut could explain 70% of the variance in user intention to use a technology. this sounds impressive, especially when compared to the 17�53% of variance in user intention which they suggest other acceptance models are able to achieve (venkatesh et al. 2003). yet interestingly and despite the acclaimed accuracy and efficacy of the utaut model, its application in complex educational contexts is rare (infenthaler and schweinbenz 2013). unlike idt with its micro and macro considerations, utaut focuses on the influencing effect of determinants on an individual user at one given time. considering the complex and highly autonomous practice of he tutors within the faculty, a model focused on personal psychological factors does seem a better fit. however, whilst utaut sounds more comprehensive in its construction and, its researchers claim, more effective in its predictive capacity, one wonders if its determinants will prove as relevant when used to frame a more open-ended qualitative approach in a complex uk he environment. in contrast, the maturity of idt is perhaps reflected in its size and complexity, yet its openness to other attributes means that, as a body of research, it surpasses utaut in the range of categories and questions that could be derived from it. this study attempts to complement both acceptance models by identifying initially which categories appear most relevant in this particular context and then drawing on those categories to shape the semi-structured interview questions used to explore factors affecting academic acceptance of byod. academic acceptance of byod the previous section explored acceptance models generally. unfortunately, studies drawing on acceptance theories that focused on mobile learning devices are limited (wang, wu, and wang 2009; williams et al. 2009) with those that exist focusing on student acceptance (çuhadar 2014; kevin thomas and o’bannon 2013; moran, hawkes, and el gayar 2010; park, nam, and cha 2012; wang, wu, and wang 2009). most recently, alrasheedi and capretz (2015b) confirmed this trend upon completion of a meta-review of factors perceived to affect m-learning success. they followed this up with a quantitative investigation in saudi arabia, drawing upon the few identified factors from their meta-analysis to formulate survey questions (alrasheedi and capretz 2015a). they found academics to be divided in what they thought were critical factors and had difficulty identifying any of these factors as statistically significant (alrasheedi and capretz 2015a). acceptance research has also c.g.m. gillies 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 been undertaken by mac callum within the new zealand tertiary sector, employing both utaut (2010) and an extended tam (mac callum, jeffrey, and kinshuk 2014) across separate quantitative studies. finding a range of factors such as time, cost, access, support, self-efficacy, anxiety and observability to be important (mac callum, jeffrey, and kinshuk 2014), yet along with the work of alrasheedi and capretz (2015b) no distinctions between byod and supplied devices appear to have been made. less recently, a qualitative acceptance investigation was carried out by infenthaler et al. (2013) that asked teachers across three german secondary schools to report their views on newly introduced tablet devices. drawing on utaut, the authors found that diversity was apparent in relation to performance expectancy, facilitating conditions and attitude (infenthaler et al. 2013). the authors were also surprised that few of the interviewees believed using tablets could improve learning. pollara (2011) conducted an extensive mixed methods study of both student and tutor acceptance of personal devices in an american university. drawing upon tam, he found that academic perceptions did not match those of the students and, in particular, that academics feared their students used mobile devices for socialising purposes when claiming they were performing class-related tasks. a lack of time, experience and training were also identified as factors affecting local academic acceptance. this study hopes to add to the limited research on academic acceptance of byod. following the advice of williams et al. (2009), it also avoids the popular yet parsimonious tam and draws upon idt and utaut to examine acceptance within the surprisingly underexplored area of a uk he faculty context. this study also answers calls for more detailed, qualitative acceptance investigations (hazen et al. 2012; williams et al. 2009), which allows for the identification of new factors and categories that could complement existing acceptance and byod research. methodology the aim of this small-scale case study was to draw out a range of factors that affect local faculty use or intention to use byod and faculty devices in the classroom. this section outlines the overall approach to the study, whilst this article specifically explores those factors related to byod. to support this qualitative study, a two-stage methodology was employed using an open-ended focus group, followed by semistructured interviews. focus group the first stage of this study aimed to check the contextual relevance of acceptance categories and distinctions between byod and faculty devices. a focus group was selected as it offered the potential for rich group discussion, which in turn can lead to the drawing out of both ‘depth of opinion’ (o’leary 2010) and insightful synthesis (krueger and casey 2009). critically, focus groups are also adept at examining how ideas develop and operate within a cultural context (robinson 1999). the author’s position, as learning technologist, within the faculty meant that a cross-section of participants could be selected based on known use of mobile devices. thus, a form of purposive heterogeneous sampling called maximum variation sampling (patton 1990) was possible. insider knowledge alongside a snowball approach allowed appropriate participants to be identified and contacted, face-to-face, to confirm research in learning technology citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 current usage and also explore interest in taking part in the study. six academics representing six separate programmes from across the faculty and with a range of byod and faculty device experiences were invited to attend. table 1 outlines the focus group sample. the design of the focus group was carefully considered. unlike the interviews that followed, no prompts with regard to distinctions between faculty and personal devices were offered nor potential factors suggested. instead, it was expected that these might surface naturally, if significant, during discussions. participants were asked individually to write barriers to their acceptance of mobile devices on separate pieces of paper. this individual task was selected to encourage a wide range of factors to be identified and also to tackle noted weaknesses with case study methods, specifically encouraging participants who lack confidence to share potentially contrary factors (basit 2010) whilst also discouraging the influence of dominant participants (o’leary 2010; sheppard, story, and jones 2013). then, as a group, participants discussed each barrier and agreed on its position on a continuum, in relation to how significant each was likely to be for staff across the faculty (see figure 1). this created opportunities for synthesis and a chance to add more factors participants believed could affect others in the faculty. it also provided a simple measure for showing the perceived strength of these factors in order to compare them to categories identified as important in the acceptance literature. mimicking the first part of the focus group, participants were next asked to consider positive factors affecting their acceptance of mobile devices and to write them down individually. then, once again as a group, they arranged these factors on a new continuum, adding any extra factors that their group discussion generated (see figure 2). upon completion of the focus group, audio and video recordings were reviewed. from this, distinct discussions about byod and faculty devices could be identified. acceptance factors on the completed continuums also linked overtly to byod or supplied devices at times. factors were also labelled whenever they overlapped with a category in the idt and utaut literature. from this, the performance expectancy category appeared to be important due to the number of factors (20) that would fit within it and also the position of these factors primarily at the top of both continuums. this seemed to support venkatesh et al.’s (2003) and rogers’ (rogers 2003) claims that performance expectancy/relative advantage tends to be the strongest predictors of acceptance. facilitating conditions (13) and compatibility/ attitude (8) were the second and third most relevant when applying the same criteria. indeed overall, all of the factors identified by the local academics were found to fit within the extended idt and utaut categories. with these results in mind, it was decided to accept idt and utaut as relevant theoretical constructs for this study, table 1. focus group sample. programme age range (years) gender mobile device class experience food, nutrition and health 40�49 f student devices education studies 40�49 f faculty devices tourism 30�39 f none events management 30�39 m faculty devices sport development 20�29 f both early childhood studies (prefer not to say) f both c.g.m. gillies 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 in this context, and use the categories from the above models to help with the design of the semi-structured interview questions. prompts would also be used to continue exploring differences in acceptance of byod and faculty devices. interviews for the second stage of this investigation, interviews were employed. this method allows researchers to engage with participants individually, providing an opportunity to immediately query key points (burton, brundrett, and jones 2008) and give prompts to gather richer data (basit 2010). kvale and brinkmann (2009) argue that qualitative interviews can help unfold the meaning of participant experiences and can help the interviewer understand the world from their perspective. this notion of interviews fitted well with an investigation attempting to draw out complex perceptions affecting academic acceptance of byod and faculty devices (bere and rambe 2013; moran, hawkes, and el gayar 2010). figure 1. barriers identified by the focus group. figure 2. positive factors identified by the focus group. research in learning technology citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 maximum variation sampling was again employed for the interviews as it was for the focus group; however, this time the emphasis was on mobile device use. table 2 shows an example of an interview sample. sampling followed a similar process to the focus group with insider knowledge and a snowball approach employed to identify a new cross-section of faculty that had used a range of devices. potential interviewees once identified were contacted face-toface and their previous mobile device use checked. following the findings of the focus group, the interview questions were drawn from categories within idt and utaut alongside a number of biographical questions with prompts used throughout to explore byod and faculty device distinctions. the interviews were conducted with eight academics between march 2014 and may 2014. most of the interviews were over 1 hour in length with the shortest being the pilot interview at 45 minutes and the longest interview lasting 1 hour and 35 minutes. themes from the interviews were coded inductively. notes were taken during the interviews with some general thoughts written down immediately afterwards. transcripts were then created by the author in nvivo 10 software with non-relevant indicators such as ‘oohs’ and ‘ahhs’, repetition and re-statement not included. a process of member checking transcripts was not engaged with due to the amount of data and time constraints; however, audio recordings were listened to multiple times to check accuracy. these transcripts were then repeatedly reviewed, labelling passages each time until themes emerged. these themes were then compared to focus group factors and factors found in the acceptance and mobile device literature. as an insider investigation, distinct ethical concerns needed to be considered. fortunately, the focus of the investigation meant that there were no identifiable conflicts of interest with any university policies and procedures or in particular with regard to sensitive or confidential information. despite this, anonymity, confidentiality and privacy were made a priority throughout the study (basit 2010), and it was made clear to participants that they could withdraw their data at any time (bera 2011; krueger and casey 2009). additionally, before any data collection methods were employed, participants were informed verbally and given a participant information sheet outlining the study and potential consequences (bera 2011; cohen, manion, and morrison 2011). as should be expected, all data collection and analysis procedures were conducted in accordance with university and british educational research association (bera 2011) guidelines. table 2. interview sample. mobile device type programme gender age range (years) student devices primary science m 40�49 faculty devices education studies f 50�59 both events management f 40�49 both tourism f 30�39 both primary/early years english f 40�49 none sport development f 40�49 none coaching m 40�49 none early childhood studies f 50�59 c.g.m. gillies 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 findings and discussion this small-scale explorative study identified a range of distinct factors affecting local academic acceptance of byod. whilst these findings are not generalisable to a larger population, some replication in similar contexts could be expected to occur, and the rich and informative data gathered offer valuable insights into an underexplored area. despite perceptions that byod and faculty ipads can complete similar tasks, distinctions were drawn by the interviewees when discussing their future classroom practice, with faculty ipads preferred to byod (table 3 and figure 3). interviewees with no experience suggested that they were more likely to employ faculty ipads, whilst a user of both faculty ipads and byod also reported that she would rather use faculty ipads. the remaining interviewees reported that they were likely to continue with their current practice. table 4 summarises the byod factors identified during the focus group discussions and interviews. as can be seen, more barriers than enablers were found, which might help clarify these reactions. the following discussion is split into three overarching categories: responsibility, attitude and device heterogeneity. although it is difficult to classify acceptance factors due to the complex multifaceted relationships between them (bingimlas 2009), these overarching categories incorporate the most popular factors reported by the participants. responsibility the focus group and some of the interviewees suggested that byod was conveniently accessible and scalable. echoing reports in the literature some interviewees also argued that students would find it easier to operate personal devices rather than supplied devices (rossing et al. 2012; williams 2012). interestingly, the byod-only user went further and stated that an important factor for him was that byod enabled a shift in technical accountability from academics to students which has been touched upon elsewhere (stavert 2013, p. 25). on the contrary, some of the interviewees expressed concern around the lack of local technical support for byod, particularly compared to the supplied faculty ipads: i can’t see [that] i could ring up . . . and go, ‘the students can’t use their mobile phones can you come over and help me’. i am booking the pads, i think they [ict support] are part of that package most of the interviewees, including all users of byod, outlined concerns about student reliability. these included: whether students would remember to charge their devices, install the correct apps, have enough space or even bring their device in when table 3. prior experience compared to preference. prior classroom experience prefer faculty ipads prefer byod comfortable with either faculty ipads only 1 byod only 1 both faculty ipads and byod 1 2 no experience 3 research in learning technology citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 required. this feared unreliability was in direct contrast to the faculty ipad service, bolstered by the perceived reliability of the technical team. additionally, and linked to both support and reliability, one surprising finding was a noticeable tendency to identify and use byod for short, replaceable or contingent tasks. perhaps, linking to the perceived convenience of byod and negative attitudes around mobile phones, both of which are touched upon in the quote below: table 4. factors affecting byod acceptance. enablers barriers students accountable for technical issues and academic device experience unlikely to be processes on their own devices relevant for whole class scalable unequal access existing student knowledge limited faculty technical support no need to book phone and class management tensions personal ownership may reduce silliness phone distractions academic not responsible for hardware phone attitude generally available any time variable functions students responsible for management limited technical and task control unknown quantity increased personal distractions possible student network costs flexible/multiple instructions required no organisational support/pressure academic issues with personal devices figure 3. mobile device preferences. c.g.m. gillies 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 i don’t think i will ever use mobile phones, i would be more prone to, rather than doing ad hoc or use things like twitter feeds or questionnaires. i would rather say here’s a set of ipads with something on, this is something you need to know about let’s go and explore this. i would be more prone to do that, it would be quite structured not just let’s explore something it would have to be more directed rather than open house. attitude as suggested in the previous quote, attitudes expressed towards mobile phones were often negative. although there were no direct phone questions, the focus group and most of the interviewees identified a range of concerns including the impact of phones on work life balance, people rudely using phones during conversation and inappropriate use of phones in class. these reservations are not unique and have been reported by academics previously (baker, lusk, and neuhauser 2012; bayless, clipson, and wilson 2013; henderson and chapman 2012; lauricella and kay 2013). furthermore, all the interviewees and the focus group openly struggled with the tension employing phones in class generates, with respective comments, ‘i would never use a phone, because a phone to me is something i will tell the students not to use, so it’s a conflict of interest’ and ‘you know when they have the phone out and are not doing work. but i feel like a dragon telling them to put them away. but then how can i ask them to get their phones out, it seems hypocritical!’ within the faculty, no consistent patterns could be discerned with regard to teaching preferences or practices and perceptions of phones or byod as being disruptive. links also could not be determined with regard to general mobile device use and a preference for student-centred practices either (kukulska-hulme 2013). this could be due to the small sample size and reliance on non-anonymous self-reported accounts. instead, perceptions on disruption appeared to link overtly to device type. indeed, despite the literature indicating that ipads can be equally disruptive (henderson, gibson, and gibb 2013; infenthaler and schweinbenz 2013; kinash, brand, and mathew 2012; rossing et al. 2012) only one interviewee reported such concerns about faculty ipads compared to an overwhelming majority expressing clear concerns about phones. this repeated negativity towards phones compared to other mobile devices compliments the findings of sad and göktaş (2013), who found an equally negative attitude when comparing phones to laptops. although byod is not reliant on phones, local perceptions that phones are a fundamental component of byod could be problematic. indeed, when asked about future byod practice, all non-users gave negative phone-related statements: i just, i don’t know [about] using phones, i don’t know, there is just something, this is me the dinosaur, but i just think again they are permanently switched on, it’s like an appendage. perhaps significantly previous education-based acceptance studies have identified attitude as an important determinant of acceptance (moran, hawkes, and el gayar 2010; park, nam, and cha 2012; thomas, singh, and kemuel 2013). device heterogeneity in the acceptance and diffusion literature, experience is identified as a key moderating variable that can affect many other psychological elements including research in learning technology citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 self-efficacy and acceptance directly (rogers 2003; venkatesh et al. 2003). this investigation does not contradict those studies, with a number of academics identifying the importance of congruence between faculty sets and the brand or operating system (os) of their own personal devices. one explanation for this is suggested by bingimlas (2009) who states that ‘it is important to remember that not only is access to resources used in the classroom for students’ learning important, but also access at home will help with self-training’. the distinctions made by academics, with regard to brand and os appear important here, as no similar boons were reported with regard to personal use and byod acceptance. this is perhaps unsurprising, as academics are unlikely to own enough devices to match the variety of brands and oss that students could conceivably bring to their classroom. unfortunately, the diversity of student devices and academic self-efficacy also seems to link closely to concerns about both failure in the classroom (balanskat, blamire, and kefala 2006; beggs 2000) and student satisfaction. for example, doubts were repeatedly expressed about byod’s ability to offer equity of access, a concern found in much of the literature (pegrum, oakley, and faulkner 2013; stavert 2013). furthermore, fears about negative academic and student experiences caused by variances in device functionality and performance echoed reports made by rossing et al. (2012) and lamaster and stager (2012), respectively. in contrast, academics identified faculty devices as ‘less of an unknown quantity’, ‘less complex’ and ‘more controllable’. conclusion this small-scale exploratory study compliments acceptance research by employing a qualitative design that both reviews and then draws upon acceptance constructs. furthermore, it extends research into mobile learning by sharing a range of factors reported by academics to inhibit or encourage acceptance of byod in a uk he faculty. the majority of factors identified can be grouped into three overarching categories: attitude, responsibility and device heterogeneity. with regard to the factors, this investigation found that despite some key enablers such as convenience, access and student accountability, local academics perceived more barriers than enablers with regard to using byod in the classroom. these barriers to acceptance include mobile phone attitude, scope of prior personal use, control, fears around equity of access, limited institutional support and student device management. many of the negative factors above touch upon a lack of knowledge or a fear of the unknown which may explain other findings within the small exploratory sample. firstly, an overall preference for faculty ipads as opposed to byod, particularly for academics with no prior experience of using mobile devices in the classroom. secondly, a noticeable preference by existing mobile device users for the approach that they already have experience with. thirdly, the tendency for byod to be employed as an informal contingent tool, with caution surrounding the use of byod for more substantial or critical activities. recommendations and limitations to help identify the psychological factors affecting tutor acceptance, this research employed a detailed qualitative methodology. yet, it is accepted that this study provides c.g.m. gillies 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 only an analysis of self-perceptions presented via non-anonymous descriptive accounts. thus, academics might not have been aware of their own emergent beliefs (levin and wadmany 2006) or they might not have been willing to share certain beliefs in such a context (cohen, manion, and morrison 2011; kvale and brinkmann 2009). a longitudinal investigation that analyses behaviour alongside perceptions could provide further insight. due to the limited research exploring academic acceptance of byod, some larger scale quantitative and qualitative studies would be invaluable, with the former perhaps drawing on the factors and categories identified in this study and the latter identifying parallels and new factors that can affect byod acceptance within other uk he institutions. references alrasheedi, m. & capretz, l. f. (2015a) ‘determination of critical success factors affecting mobile learning: a meta-analysis approach’, the turkish online journal of educational technology, vol. 14, no. 2, pp. 41�51. alrasheedi, m. & capretz, l. f. (2015b) ‘an empirical study of critical success factors of mobile learning platform from the perspective of instructors’, procedia � social and behavioral sciences, vol. 176, pp. 211�219. backer, e. (2010) ‘using smartphones and facebook in a major assessment: the student experience’, e-journal of business education and scholarship of teaching, vol. 4, no. 1, pp. 19�31. baker, w. m., lusk, e. j. & neuhauser, k. l. (2012) ‘on the use of cell phones and other electronic devices in the classroom: evidence from a survey of faculty and students’, journal of education for business, vol. 87, no. 5, pp. 275�289. balanskat, a., blamire, r. & kefala, s. (2006) the ict impact report, european schoolnet, brussels. basit, t. (2010) conducting research in educational contexts, continuum international publishing group, london. bayless, m. l., clipson, t. w. & wilson, s. a. (2013) ‘faculty perceptions and policies of students’ use of personal technology in the classroom’, faculty publications, vol. 32, pp. 119�137. beggs, t. (2000) ‘influences and barriers to the adoption of instructional technology’, mid-south instructional technology conference, middle tennessee state university, murfreesboro, tn, pp. 1�14. bera. (2011) ethical guidelines for educational research, british educational research association, london. bere, a. & rambe, p. (2013) ‘extending technology acceptance model in mobile learning adoption: south african university of technology students’ perspectives’, international conference on e-learning, pp. 52�61. bingimlas, k. (2009) ‘barriers to the successful integration of ict in teaching and learning environments: a review of the literature’, eurasia journal of mathematics’, science & technology education, vol. 5, no. 3, pp. 235�245. burton, n., brundrett, m. & jones, m. (2008) doing your education research project, sage, london. cohen, l., manion, l. & morrison, k. (2011) research methods in education, routledge, london. cristol, d. & gimbert, b. (2013) ‘academic achievement in byod classrooms’, qscience proceedings, vol. 4, no. 1, pp. 24�30. crown fibre holdings ltd. (2012) ‘getting excited about byod’, [online] available at: http:// www.crownfibre.govt.nz/2012/11/getting-excited-about-byod/ çuhadar, c. (2014) ‘information technologies pre-service teachers’ acceptance of tablet pcs as an innovative learning tool’, educational sciences: theory & practice, vol. 14, no. 2, pp. 741�754. research in learning technology citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 13 (page number not for citation purpose) http://www.crownfibre.govt.nz/2012/11/getting-excited-about-byod/ http://www.crownfibre.govt.nz/2012/11/getting-excited-about-byod/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 dahlstrom, e. & difilipo, s. (2013) ‘the consumerization of technology and the bring-yourown-everything (byoe) era of higher education’, educause center for applied research, [online] available at: http://net.educause.edu/ir/library/pdf/ers1301/ers1301.pdf davis, f. (1986) ‘perceived usefulness, perceived ease of use, and user acceptance of information technology’, mis quarterly, vol. 13, no. 3, pp. 319�340. dykes, g. & knight, h. (2012) mobile learning for teachers in europe � exploring the potential of mobile technologies to support teachers and improve practice, unesco working paper series on mobile learning, paris. enriquez, a. g. (2010) ‘enhancing student performance using tablet computers’, college teaching, vol. 58, no. 3, pp. 77�84. guest, d. e. & clinton, m. (2007) research and development series, human resource management and university performance, london. hazen, b. t., et al., (2012) ‘a proposed framework for educational innovation dissemination’, journal of educational technology systems, vol. 40, no. 3, pp. 301�321. henderson, r. g. & chapman, b. f. (2012) ‘business educators ’ perceptions concerning mobile learning’, delta pi epsilon journal, vol. liv, no. 1, pp. 16�27. henderson, k., gibson, c. & gibb, f. (2013) ‘the impact of tablet computers on students with disabilities in a higher education setting’, technology and disability, vol. 25, no. 2, pp. 61�76. hwang, g. j. & tsai, c. c. (2011) ‘research trends in mobile and ubiquitous learning: a review of publications in selected journals from 2001 to 2010’, british journal of educational technology, vol. 42, no. 4, pp. e65�70. infenthaler, d. & schweinbenz, v. (2013) ‘the acceptance of tablet�pcs in classroom instruction: the teachers’ perspectives’, computers in human behavior, vol. 29, no. 3, pp. 525�534. jacobsen, d. m. (1998) ‘adoption patterns of faculty who integrate computer technology for teaching and learning in higher education’, world conference on educational multimedia and hypermedia, association for the advancement of computing in education, freiburg, pp. 1�9. johnson, l., et al., (2016) nmc horizon report: 2016 higher education edition, austin, tx: the new media consortium, [online] available at: http://cdn.nmc.org/media/2016-nmchorizon-report-he-en.pdf johnson, l., et al., (2013) the nmc horizon report: 2013 higher education edition, austin, tx: the new media consortium, [online] available at: http://www.nmc.org/publications/ 2013-horizon-report-higher-ed kinash, s., brand, j. & mathew, t. (2012) ‘challenging mobile learning discourse through research: student perceptions of blackboard mobile learn and ipads’, australasian journal of educational technology, vol. 28, no. 4, pp. 639�655. krueger, r. a. & casey, m. a. (2009) focus groups: a practical guide for applied research. 4th edn, sage, london. kukulska-hulme, a. (2013) ‘perspectives: limelight on mobile learning: integrating education and innovation’, harvard international review, vol. 34, no. 4, pp. 12�16. kvale, s. & brinkmann, s. (2009) interviews: learning the craft of qualitative research interviewing. 2nd edn, sage, london. lamaster, j. & stager, g. (2012) ‘point/counterpoint’, learning and leading with technology, vol. 39, no. 5, pp. 6�7. lauricella, s. & kay, r. (2013) ‘exploring the use of text and instant messaging in higher education classrooms’, research in learning technology, vol. 21, no. 19061. lee, y., kozar, k. a. & larsen, k. r. t. (2003) ‘the technology acceptance model: past, present, and future’, communications of the association for information systems, vol. 12, no. 50, pp. 752�780. levin, t. & wadmany, r. (2006) ‘teachers’ beliefs and practices in technology-based classrooms: a developmental view’, journal of research on technology in education, vol. 39, no. 2, pp. 157�181. mac callum, k. (2010) ‘adoption theory and the integration of mobile technology in education’, in annual conference of computing and information technology education and research in new zealand, eds. s. mann & m. verhaart, dunedin, new zealand, pp. 139�150. c.g.m. gillies 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 http://net.educause.edu/ir/library/pdf/ers1301/ers1301.pdf http://cdn.nmc.org/media/2016-nmc-horizon-report-he-en.pdf http://cdn.nmc.org/media/2016-nmc-horizon-report-he-en.pdf http://www.nmc.org/publications/2013-horizon-report-higher-ed http://www.nmc.org/publications/2013-horizon-report-higher-ed http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 mac callum, k., jeffrey, l. & kinshuk. (2014) ‘factors impacting teachers’ adoption of mobile learning’, journal of information technology education, vol. 13, pp. 141� 162. moran, m., hawkes, m. & el gayar, o. (2010) ‘tablet personal computer integration in higher education: applying the unified theory of acceptance and use technology model to understand supporting factors’, journal of educational computing research, vol. 42, no. 1, pp. 79�101. naismith, l., et al., (2004) literature review in mobile technologies and learning, futurelab [online] available at: https://telearn.archives-ouvertes.fr/hal-00190143 ofcom. (2015) uk now a smartphone society, office for communication, [online] available at: http://consumers.ofcom.org.uk/news/uk-now-a-smartphone-society/ o’leary, z. (2010) the essential guide to doing your research project, sage, london. park, s. y., nam, m. w. & cha, s. b. (2012) ‘university students’ behavioral intention to use mobile learning: evaluating the technology acceptance model’, british journal of educational technology, vol. 43, no. 4, pp. 592�605. parsons, d. (2013) ‘jam today � embedding byod into classroom practice’, qscience proceedings, vol. 2013, [online] available at http://dx.doi.org/10.5339/qproc.2013.mlearn.25 patton, m. (1990) qualitative evaluation and research methods. 2nd edn, sage, california. pegrum, m., oakley, g. & faulkner, r. (2013) ‘schools going mobile?: a study of the adoption of mobile handheld technologies in western australian independent schools’, australasian journal of educational technology, vol. 29, no. 1, pp. 66�82. pollara, p. (2011) mobile learning in higher education: a glimpse and a comparison of student and faculty readiness, attitudes and perceptions, dissertation, louisiana state university, baton rouge. robinson, n. (1999) ‘the use of focus group methodology � with selected examples from sexual health research’, journal of advanced nursing, vol. 29, no. 4, pp. 905�13. rogers, e. (2003) diffusion of innovations. 5th edn, free press, london. rossing, j. p., et al., (2012) ‘ilearning: the future of higher education? student perceptions on learning with mobile tablets’, journal of the scholarship of teaching and learning, vol. 12, no. 2, pp. 1�26. şad, s. n. & göktaş, ö. (2013) ‘preservice teachers’ perceptions about using mobile phones and laptops in education as mobile learning tools’, british journal of educational technology, vol. 45, no. 4, pp. 606�618. sahin, i. & thompson, a. (2006) ‘using rogers theory to interpret instructional computer use by coe faculty’, journal of research on technology in education, vol. 39, no. 1, pp. 81�104. sheppard, s., story, b. & jones, h. (2013) ‘researching the ‘researched’ about research: a fresh perspective on the power of focus groups’, market and social research, vol. 21, no. 2, pp. 40�48. stavert, b. (2013) bring your own device (byod) in schools: 2013 literature review, nsw department of education and communities, p. 33. tabata, l. n. & johnsrud, l. k. (2008) ‘the impact of faculty attitudes toward technology, distance education, and innovation’, research in higher education, vol. 49, no. 7, pp. 625�646. thomas, k. m. & o’bannon, b. (2013) ‘cell phones in the classroom: preservice teachers’ perceptions’, journal of digital learning in teacher education, vol. 30, no. 1, pp. 11�20. thomas, k. m., o’bannon, b. w. & bolton, n. (2013) ‘cell phones in the classroom: teachers’ perspectives of inclusion, benefits, and barriers’, computers in the schools, vol. 30, no. 4, pp. 295�308. thomas, t., singh, l. & kemuel, g. (2013) ‘the utility of the utaut model in explaining mobile learning adoption in higher education in guyana’, international journal of education and development using information and communication technology, vol. 9, no. 3, pp. 71�85. venkatesh, v, et al., (2003) ‘user acceptance of information technology: toward a unified view’, mis quarterly, vol. 27, no. 3, pp. 425�478. walker, r, et al., (2014) 2014 survey of technology enhanced learning for higher education in the uk, universities and colleges information systems association, oxford. research in learning technology citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 15 (page number not for citation purpose) https://telearn.archives-ouvertes.fr/hal-00190143 http://consumers.ofcom.org.uk/news/uk-now-a-smartphone-society/ http://dx.doi.org/10.5339/qproc.2013.mlearn.25 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 wang, y. s., wu, m. c. & wang, h. y. (2009) ‘investigating the determinants and age and gender differences in the acceptance of mobile learning’, british journal of educational technology, vol. 40, no. 1, pp. 92�118. williams, c. (2012) ‘managing byod effectively’, district cio, vol. 48, no. 9, pp. 84�85. williams, m. d., et al., (2009) ‘contemporary trends and issues in it adoption and diffusion research’, journal of information technology, vol. 24, no. 1, pp. 1�10. wright, s. & parchoma, g. (2011) ‘technologies for learning? an actor�network theory critique of ‘affordances’ in research on mobile learning’, research in learning technology, vol. 19, no. 3, pp. 247�258. c.g.m. gillies 16 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30357 http://dx.doi.org/10.3402/rlt.v24.30357 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30357 http://dx.doi.org/10.3402/rlt.v24.30357 virtual reality: theoretical basis, practical applications philip barker interactive systems research group, school of computing and mathematics, university of teesside, middlesbrough, cleveland ts1 3ba abstract virtual reality (vr) is a powerful multimedia visualization technique offering a range of mechanisms by which many new experiences can be made available. this paper deals with the basic nature of vr, the technologies needed to create it, and its potential, especially for helping disabled people. it also offers an overview of some examples of existing vr systems. introduction an important property of computer systems is their ability to visualize information. a graphics program, for example, can turn a mass of numeric data into a digestible, pictorial representation of it. but faster processing speeds, and a good measure of human inventiveness, now allow us to take such visualizations much further, offering users many new types of experience. such visual experiences form the basis of the virtual-reality (vr) systems described in this paper, which is intended to fulfil a four-fold objective: to explain the principles of vr to provide a perspective of the basic technologies on which vr environments depend to describe some current vr systems to discuss some potential applications of vr with reference to the problems of disabled people. 15 philip barker virtual reality: theoretical basis, practical applications basics of virtual reality a vr system is a sophisticated multimedia environment in which users are exposed to, and can participate in, surrogate tacto-audio-visual experiences. these experiences are created by means of a computer system to which are attached special types of peripheral device, enabling users to interact with the real and artificial objects that exist within the interaction environment. degrees of realism as individuals we exist in a world that continually presents us with new experiences, and one way of learning to deal appropriately with them is to generate them artificially. many areas of education benefit from the use of pseudo-real situations which enable students to learn in a risk-free environment. examples of available techniques include role-playing, models and simulations, surrogations, synthetic actors, microworlds, artificial-reality kits, and vr systems. each of these techniques allows a different degree of realism to be achieved. part of the attraction of artificial and virtual realities is that they allow us to go beyond the limits of what is real (smith, 1986; kelly, 1989; fisher and tazelaar, 1990; magnenat-thalmann and thalmann, 1989). terminology and definitions books and other literature dealing with vr use a range of new terminology. four important ones for the purposes of this paper are artificial reality, virtual reality itself, cyberspace and cyberspace deck. artificial reality, as defined by krueger (1991), perceives a participant's action in terms of a body's relationship to a graphical world, and generates responses that maintain the illusion that his or her actions are taking place within that world, whereas (for krueger) virtual reality is another term for artificial reality that applies only to systems implemented with goggles and gloves, the special peripherals that enable the user of vr to perceive the virtual world and to interact with it. again according to krueger, cyberspace is a global artificial reality that can be visited simultaneously by millions of people. however, a more useful definition (helsel and paris roth, 1991) is: an interactive simulation which includes human beings as necessary components. walser (1991) suggests that cyberspace is a form of theatre which 'can be regarded as a computer-based simulation that enables groups of people to play roles of characters in cybernetic simulations of three-dimensional worlds: crucially, cyberspace gives the role players the ability to sense a virtual reality from the point of view of the characters they play.' a cyberspace deck, as defined by helsel and paris roth, is simply 'a gateway through which people are transported to cyberspace'. but according to walser, the term should be used to refer to 'the physical space containing the array of instruments which enable a player to act within, and feel part of, a virtual space.' 16 altj • volume 1 number 1 this last definition is important since it introduces the need for access to suitable equipment in order to experience vr worlds. however, all the above the definitions can be used to make up the description of a 'systems model' on which to base a discussion of vr. a systems model an appropriate systems model has been proposed by walser. the model is based on a cybernetic feedback loop between 'puppets' (which operate in virtual space) and 'patrons' (essentially, users who operate in physical space). puppets within the virtual space embody intellects which may be either artificial or human. they acquire 'knowledge' of events that take place in physical space by means of devices (called sensors). puppets also have a range of output devices (called effectors) with which they can influence physical space. a puppet's patron can influence virtual space by means of its sensors and can learn about events taking place in virtual space by means of its effectors. a cybernetic loop therefore arises because a puppet's sensors are a patron's effectors, and a puppet's effectors are a patron's sensors. the feedback loop implicit in this systems model is a fundamental part of all the vr systems described later in this paper. equipment n e e d s cyberspace decks for walser, a cyberspace deck is the generic name given to the equipment needed to reach, and travel within, a cyberspace system, with most decks having a fundamental architecture usually consisting of seven basic components: (i) a cyberspace 'engine* for generating a simulated world and for mediating the patron's interaction with it; (ii) a 'control space' (a section of physical space) in which the patron's movements are tracked; (iii) a set of sensors to monitor the patron's actions and body functions; (iv) a set of effectors which can be used to produce various physical effects and stimulate the patron's senses; (v) a set of 'props' used to give the patron solid analogues of virtual objects and vehicles; (vi) a 'network interface' which can be used to admit other patrons to the simulated world; and (vii) an 'enclosure' (consisting of some sort of physical framework) which can be used to house all the components listed above. obviously, the exact nature of a cyberspace deck will vary from application to application, and the particular components of any given deck will also strongly influence the nature of the virtual realities that can be created. some examples of cyberspace decks which have recently been mentioned in the vr literature are briefly outlined below. autodesk inc. in the usa have built a number of prototype cyberspace decks (such as hicycle described below) that are oriented towards athletical applications. in one of their decks they have linked together three computer systems: a compaq 386 pc (which runs the 17 philip barker • virtual reality: theoretical basis, practical applications simulation), a macintosh (used as a source of video images), and an amiga (for sound effects). the computers are linked together in various ways using serial communication ports and an ethernet local-area network. each computer connects to the patron by means of an umbilical link which provides the high-speed data highways that are necessary for the patron's helmet and data glove. the prop (a sort of conveyor belt upon which the patron walks) is connected to the simulation machine by standard parallel and serial data links. in the uk, two interesting vr systems have recently become commercially available: vision and virtuality. vision depends essentially on the use of transputers for its computing power, but it also uses intel i860 pipelined processors for image generation and rendering (pountain, 1991). vision is described as a vr server since it can be attached to a host computer such as an ibm pc, a sun sparcstation or a vax. virtuality, produced by w industries, comes in two forms a sit-down console, and a system based on a helmet and a glove which is designed to be used by a person standing or moving within a scanned area. of course, the availability of single-user decks marks only the beginning of what will eventually become much more complex multi-user systems the ability to connect cyberspace decks via local-area or wide-area networks will enable multi-person spaces to be created. furthermore, through networking it will also becomes possible for decks to have access to multiple cyberspaces, thereby enabling patrons to 'jump' from cyberspace to cyberspace. quality of experience the quality of a vr system can be measured in terms of its 'distance' from a real environment from the point of view of its patron. naturally, a patron's satisfaction or otherwise with a vr experience will depend on a large number of factors. however, two important considerations are deck ergonomics and psycho-semantics. the subject of deck ergonomics is concerned with issues such as how comfortable the vr equipment is to wear and the adverse physical and/or mental effects it might have on the human body. the subject of psycho-semantics is concerned with the quality and meaning of the artificial worlds generated, thus with issues such as the ways in which the brain handles illusion, and the problems of dissonance that may arise as a result of experiences in different cyberspace environments. however, because vr systems are so new, little work has yet been done in this area. a historical perspective five major technologies have influenced vr, and in all likelihood will continue to do so. they are: television; remote control; 3d graphics and animation; display technology; and 3d human-computer interfaces. television a major feature of television is that it enables us to observe remote events and so allows us to experience the phenomenon of 'tele-presence' in other words, to be 'present' in any location in which a tv camera can be placed. a conventional tv broadcast does not of course involve a viewer in the kind of direct interaction required by a vr system. 18 alt-j volume 1 number 1 remote control closed-circuit tv, on the other hand, can offer levels of interactivity unavailable in conventional broadcast tv, for example in situations where the viewer can control the orientation of the tv camera. in order to achieve this, it is necessary to provide some sort of remote-control facility such as a joystick, a steering wheel, or looking to vr systems a device worn on the head, allowing the viewer to control the displayed view. the importance of remote-control technology in this context lies in the fact that when it is coupled with tele-presence it allows the implementation of tele-operation and tele-robotics (iyengar and kashyap, 1989). in many cases, through the design of suitable effectors, these techniques can enable a patron to interact with the events being viewed, and are therefore of importance to designers of vr systems. remotely-controlled vehicles fitted with robot arms and tv cameras, which can be used in normally inaccessible or hazardous environments, are technically feasible with current technology. remote-control technology thus enables us to create artificial experiences in which a person can participate interactively. graphics and animation clearly, although it might be technically feasible, it would not be a practical proposition to provide everyone who wants to explore the world outside their living-room with a remotely-controlled go-where-you-please robotized vehicle and tv camera. thus if virtual worlds are to be generally available, some alternative mechanism has to be used. a number of approaches are possible. for example, video recordings of distant locations can be stored on videodisc or compressed and stored on cd-rom. indeed, this technique is quite widely used to create surrogate experiences, particularly for training purposes. alternatively, images can be generated in real time using computer-graphics techniques (magnenat-thalman and thalman, 1989; upstill, 1990; sharp, 1990), and most of the currently available vr systems depend very heavily on such computational graphics. unfortunately, however, extremely powerful computers are needed in order to create images which have the quality of those we continually perceive through our natural visual system. the high-speed parallel computers needed to produce images of this sort would be far too expensive for widespread use in vr systems. consequently, a compromise has to be achieved: image quality has to be sacrificed in order to meet realizable costs. within a vr system the major function of the graphics and animation software is to generate 3d graphical worlds within the memory space of the host computer. these worlds will contain two basic types of object: static and animated. the static objects provide the contextual environment. the animated objects move around the 3d world interacting with each other and with the static objects. but unlike the objects we perceive in our real world, the objects in artificial realities can be given any special properties the vr designer may wish them to have. display technology the commonest form of display is a simple flat screen of the everyday computer-monitor variety, but there are three major disadvantages associated with it. first, the range of colours and resolution that can be achieved is relatively low unless expensive equipment is used. secondly, it is difficult to project to the viewer a feeling of three-dimensional realism. 19 philip barker virtual reality: theoretical basis, practical applications thirdly, extraneous events and objects within the environment in which the display is located can exert a distractive influence on the viewer, thereby reducing the ambience of the vr. there are a number of ways of overcoming each of these problems through the use of special display technologies (tello, 1988; laurel, 1990; krueger, 1991). in particular, various types of helmet and goggles can exclude the viewer's local environment, thus reducing the chance of interference between reality and artificial reality. such head-mounted devices provide a stereoscopic display system that projects a wide-angle 3d panorama directly into the viewer's retina. the particular view observed can be controlled by head movements, voice commands and hand gestures. equipment of this sort allows exploration of a spherical field of view which can be derived from either a simulated or a remotely-sensed environment. human-computer interfaces in order, to interact with the real or the imaginary objects contained within a vr system, special types of 3d device are needed. the best known kinds of interaction peripheral for use in vr work are data gloves, body suits and various types of hand/foot control (eglowstein, 1990; laurel, 1990). the function of a data glove and its supporting hardware/software is to provide data that will enable hand position and orientation, in 3d space, to be calculated. the glove must also provide data that reflects the relative position and crooking of the fingers. if two data gloves are used simultaneously (one on each hand), further data can be made available which gives the position of one hand relative to that of the other. the data produced by data gloves is usually stored and processed in the form of movement-pattern vectors (morita, hashimoto and ohteru, 1991). because of the amount and complexity of the data produced, neural networks are often used to analyse and interpret the meaning of hand gestures. several types of glove are currently available, the most common being the dataglove produced by vpl research in california. this glove embeds a 3d location sensor, and uses optical-fibre cabling to detect finger movement. another well-known example is the dextrous hand master produced by exos, which uses an intricate exoskeletal arrangement of sensors that fit over the back of the hand. the sensors are held over each finger joint by light-weight pads and velcro straps, and each sensor houses both a hall-effect magnetic pick-up and a magnet. the type of glove used in any given vr situation will depend on the particular functions to be performed. for example, in some of our work we have been exploring the use of low-cost data gloves for developing simple gestural control languages based on finger-touching operations involving the simultaneous use of two gloves. examples certain attempts to generate experiences of artificial realities were made decades ago. this section of the paper describes some operational vr systems, both old and new, built either for research purposes or for public use. 20 alt-j volume 1 number 1 sensorama sensorama is a multi-sensory simulation environment developed by morton heilig as early as the mid-1950s (krueger, 1991; laurel, 1990), though it is still operational today. it was designed as an arcade game in which the participant sat on a seat, held a pair of motorcycle handlebars and pressed his or her eyes against a pair of binocular-like stereo-mounted lenses. with this equipment it was possible for the user to see and listen to a participative 3d movie taken at eye-level. while the movie was playing, the seat and handlebars vibrated and wind was blown into the face of the 'rider', creating the illusion of driving around a city. this illusory experience could be reinforced by the introduction of aromas (the smell of exhaust fumes and pizzas) at appropriate points during the presentation. metaplay metaplay, dating from the early 1970s, was designed by krueger (1991) as a multi-person experience within a 'responsive art' exhibit for use in a public gallery. as people flowed into the exhibition area, they were filmed using a closed-circuit video system. this enabled their moving images to be relayed back to them and displayed on a large screen by means of a back-projection system. while this was happening, an artist in another building could use a digital data tablet and stylus in order to sketch objects which could be super-imposed on the screen people were watching. by observing and monitoring the actions of particular people in the audience, the artist could give individuals the illusion that they could draw or create 'live graffiti' by moving their hands, head or body. videoplace videoplace is another example of an early artificial reality system designed by krueger (1991). fundamental to its design is the creation of a 'shared place' that two or more people can experience. videoplace is similar to metaplay in that participants' video images form part of a synthesized graphic world, but videoplace is based on the use of two-way video to create a shared visual environment in which participant's images can interact with each other and with objects contained within the shared world. since the visual world is computer-generated, the objects within it need not bear any relationship to objects encountered in the real world. the videoplace system was used to explore all manner of interaction, and to create many different types of illusory experience based on the body gestures made by participants. typically, these illusions could involve such activities as dancing, swimming, flying, and participating in various types of sporting activity. fluxbase fluxbase is another example of a virtual exhibit (helsel and paris roth, 1991). essentially, it is a virtual environment for generating contemporary dynamic and exploratory works of art. to the person using it, fluxbase appears as a 'box' which can be opened by clicking a mouse button, and which contains a wide assortment of virtual objects such as postcards, poems, sketches, film-strips, slides; musical scores, a pack of playing cards, games, puzzles, audio-tape recordings, bits of string indeed, anything that may be used to create a piece of contemporary art. objects can be 'pulled out' of the box and explored by means of mouse-based pointing operations. poems can be read, paintings can be examined in various ways, slides and film-strips can be watched, and audio recordings listened to. the artist's role 21 philip barker virtual reality: theoretical basis, practical applications in fluxbase is to decide upon and create the objects which will go in the box, while that of the patron (the user) is to experience and explore the virtual objects contained in it. hicycle autodesk's hicycle involves the use of a bicycle mounted within a rig connected to a control computer. the computer monitors the pseudo-motion of the bicycle and in response is able to generate various types of special effect. the cyclist wears a helmet similar to those described in section 3.5 above, which provides the field of view. if the cyclist pedals rapidly enough, the bicycle 'takes off and 'flies' over a graphic landscape. a number of similar vr systems based on the bicycle approach have been created. for example, one of the participative exhibits at panasonic's exhibition centre in tokyo enables the cyclist to explore various cities by 'pedal', though in this system the cyclist watches images on a screen retrieved from videodisc rather than wearing a helmet, and the bicycle is incapable of 'flight' view the aerospace human factors research division of nasa's ames research centre has developed view (virtual interactive environment workstation) which is used in a range of experiments related to space research, including investigations into tele-presence, teleoperation, tele-robotics, and virtual data-display facilities for spaceships and space stations (helsel and paris roth, 1991; laurel, 1990). fundamental to the vr environments created at the ames research centre is the use of a data glove and a head-mounted wide-angle stereoscopic display system controlled by its wearer's position, voice and gestures. using this basic equipment, view is able to provide a multi-sensory interactive display for exploring a 360-degree synthesized or remotely-sensed environment. space-research experiments in which this equipment has been used include a tele-robotic device which mimics its human controller's actions, and various simulations of different types of interaction environment for pilots of spaceships and lunar vehicles. the potential of vr the majority of vr applications described in the published literature rely heavily on a user's visual channels of perception. however, some years ago we became interested in exploring the possibility of generating virtual realities for blind people which rely on sound effects and tactile communication methods. as a result of our work, we introduced the concept of talking tactile books for blind people (barker, 1991a). these books were of a multimedia nature and depended on two basic technologies for their fabrication: braille and sound recorded on cd-rom. the 'pages' of braille could be mounted on various types of touch-sensitive surfaces to be used for information output and input. depending on the effects to be produced, two pages could simultaneously be 'read' and touched one with each hand. while tactile interaction was taking place, and in response to it, audio narrations and sound effects could be retrieved from the cd-rom. an important outcome of this work has been the design and testing of a prototype data glove to facilitate the computer interpretation of simple gesticulative sign languages using a neural network system. although there is yet much work 22 alt-j volume 1 number 1 to be done with this equipment, we feel that the progress made to date and its potential in supporting the generation of vr environments for the disabled is encouraging. vr technologies will undoubtedly have a significant impact over the next decade in terms of the potential benefits they can bring to disabled people in areas such as learning, new kinds of sensory and cognitive experience, new kinds of communication aids, training facilities, and various kinds of spin-off development. learning environments one of the attractive features of vr as a learning tool is its ability to display objects and situations not normally visible to humans, and to enable humans to interact with them for example, by reaching out to 'touch' the atoms of a complex molecule. just as easily, it could convert a wheel-chair into a vehicle capable of allowing its occupant to explore, for example, virtual safari parks or cities anywhere in the world. equally, because they are based on sophisticated simulations, vr systems also enable us to explore learning domains by means of what-if experiments, the results of which will stimulate the development of cognitive models and mental skills that could be applied in many situations. thus, future vr systems will undoubtedly offer exciting ways of learning, of course including experiential learning. generating new experiences normally, our ability to encounter new types of physical and cognitive experience depends on our ambulatory capability and the power of the perceptual tools we have available. but vr can provide mobility in situations where there may be immobility, and through the use of special peripherals and special effects, can generate sensation where previously there may have been none. in short, vr systems can produce a wide range of new experiences for people who would normally not be able to encounter them. of course, the events that create the experiences will be of an illusory nature, but the effects they produce can be as stimulating and as exciting as real events. new communication aids the development of new types of technology for sensing body movement and stimulating the senses of sight, sound and touch are making possible many new types of communication aid. the data glove and body suit have already been referred to. using this type of equipment it is possible to design an almost infinite variety of languages for communication based on almost any form of body movement, gesture, mime, touch or sonic utterance. indeed, the technology on which vr is based makes it possible for each individual to have access to a 'private' language that can then be mapped to one or more commonly accepted public language systems. our work with the low-cost tactile data glove illustrates how easily this can be done, and similar work has been carried out by kramer and leifer (1988) using a 'talking glove'. although these are not direct applications of vr, they are important research spin-offs. using vr itself as a total personal communication medium may be a future goal, though it has to be said that the concept of the vr phone is some way off. training facilities we are increasingly turning to the use of computer-based technologies to provide training facilities (barker, 1991b; barker, 1993), and given the nature of the simulated environments 23 philip barker virtual reality: theoretical basis, practical applications they are capable of producing, vr systems are becoming an important knowledge and skill transfer tool for training people to perform complex and difficult tasks such as driving a car, piloting an aeroplane, and performing delicate surgical operations (using simulated patients). vr environments can be used to develop and practise inter-personnel skills such as public speaking, interviewing, participation in group discussion, and speaking, understanding and reading foreign languages. vr systems are also being used to provide rehabilitation environments in which to re-develop basic cognitive and motor skills after a debilitating accident or traumatic experience. and there is significant potential for training systems based on vr within the area of medical diagnosis. it would clearly not be sensible to say that vr offers limitless opportunities for training, but over the next decade, it will certainly be used in an increasing number of applications in this area. spin-off areas a characteristic of technological research is that developments in one area often lead to spin-off developments in other areas, and such spin-off developments can sometimes be a way of broadening the application base for a product, thereby reducing the price at which it can be made available to end-users. the data glove that forms such an important part of many vr systems provides one example. the original dataglove developed by vpl research is a complex and expensive interaction peripheral (eglowstein, 1990). however, a similar peripheral (the power glove) developed by mattel (for use in home computers and video games) is far less expensive because of the much larger target market and the reductions in cost that can be accrued as a result of automated mass production. conclusion important, interesting and exciting research is currently being undertaken in the development and application of vr, much of which will lead to useful products. currently, the majority of vr systems are expensive and therefore beyond the reach of ordinary users, however, low-cost, less functional systems could be and must be developed, in particular for disabled people, systems which will enable them to participate in the new and unusual experiences which vr promises. acknowledgements i am indebted to dr dominique burger and mme jeanne suchard of the institute national de la sant6 et de la recherche mddicale (inserm) in paris for their help and useful comments on many aspects of this paper. i would also like to thank dr gabriel jacobs (university college of swansea) for his invaluable editorial assistance. 24 alt-j volume 1 number 1 references barker, p.g. (1991a), 'hypermedia interaction for the disabled', in burger, d. (ed), technologies hypermédias: implications pour l'enseignement aux jeunes déficients visuels, conference proceedings, paris, inserm. barker, p.g. (1991b), 'developing competence through c b t , in saunders, d and race, p, aspects of educational technology, volume xxv: developing and measuring competence, london, kogan page. barker, p.g. (1993), exploring hypermedia, london, kogan page (in press). eglowstein, h. (1990), 'reach out and touch your data', byte, 15, 7, 283-90. fisher, s.s. and tazelaar, j.m. (1990), 'living in a virtual world', byte, 15, 7, 215-21. helsel, s.k. and paris roth, j. (eds) (1991), virtual reality theory, practice and promise, westport, meckler. iyengar, s.s. and kashyap, r.l. (1989), 'autonomous intelligent machines', ieee computer, 22, 6, 14-15. kelly, k. (1989), 'virtual reality: an interview with jaron lanier, whole earth review, 64, 108-19. kramer, j. and leifer, l. (1988), 'the talking glove: a communication aid for deaf, deaf-blind and non-vocal individuals', in annual report of the rehabilitation and development centre, palo alto ca, veterans association medical centre. laurel, b. (ed) (1990), the art of human-computer interface design, reading ma, addison-wesley. krueger, m.w. (1991), virtual reality ii, reading ma, addison-wesley. magnenat-thalman, n. and thalman, d. (1989), 'synthetic actors: the simulation of human motion', the computer bulletin, 1, 112-14. morita, h., hashimoto, s. and ohteru, s. (1991), 'a computer music system that follows a human conductor', ieee computer, 24, 744-53. pountain, d. (1991), 'provision: the packaging of artificial reality', byte, 16, 10, 80is-53 to 80is-64. sharp, c. (1990), 'using animator', carmel ind, que corporation. smith, r.b. (1986), 'the alternative reality kit: an animated environment for creating interactive simulations', in proceedings if the 1986 ieee computer society workshop on visual languages, 25th-27th june, dallas, texas. tello, e.r. (1988), 'between man and machine', byte, 13, 9, 288-93. upstill, s. (1990), 'graphics go 3-d', byte, 15, 13, 253-8. walser, r. (1991), 'elements of a cyberspace playhouse', in helsel, s.k. and paris roth, j. (eds), virtual reality theory, practice and promise, westport, meckler. 25 cognitive style and computer-assisted learning: problems and a possible solution1 john a. clarke school of learning and development, kelvin grove campus, queensland university of technology, locked bag no 2, red hill, brisbane, australia 4059 abstract although the notion of cognitive style has been around for some time, only in relatively recent times has there been a research interest in examining its effect on the performance of computer-assisted learning (cal) users. there are a number of practical difficulties associated with catering for different cognitive styles of cal users. this paper identifies not only a style which influences cal-user performance and overcomes many of the difficulties, but also a possible suitable measure of that style. data on the reliability of this measure is reported, along with preliminary work on its use to cater for cal users with different cognitive styles. future work will focus on the development of the package and the predictive validity of the style measure. cognitive styles and computer-assisted learning (cal) cognitive style has been variously defined (e.g. kogan, 1971; messick, 1970, 1976, 1984) but essentially means the unique and preferred way in which individuals process information. although the notion can be traced back to the beginning of the century (coop and sigel, 1971), it gained recognition in the 1950s and 1960s as a significant personal characteristic which increased the understanding of how people learn (e.g. witkin et al, 1962). however, recognition of its importance in education generally (kogan, 1971; messick, 1970, 1976, 1984) and cal specifically (james and james, 1990; sharma, 1987; sims, 1988) has been relatively recent. since any computer-assisted information system consists of three major components hardware, software and the user it is important that the concept of cognitive style as a significant characteristic of the user be catered for in the production of a cal program. 47 john clarke cognitive style and cal: problems and a possible solution although there has been recognition of this for some time (barker, 1982), there are still consistent calls for the development of programs which allow users to choose how they wish to learn (chadwick and watson, 1986; duncan, 1985). a plethora of cognitive styles has been identified and reported (e.g. kogan, 1971; joughin, 1989; rowland and stuessy, 1987) but often, the styles differ more in name than in nature. the vast majority of cognitive styles can be classified into either a preference for a reasonable degree of structure and guidance to be provided in the learning situation, or a preference for considerable freedom to choose the direction that learning should take. essentially, this is preference for structured guided instruction versus unstructured discovery learning. as far as the delivery of cal is concerned, the 'structured' end of the continuum is catered for by the traditional tutorial-type programs while the 'unstructured' end is served best by simulation/ game-type programs. problems in catering for cognitive styles there are a number of practical difficulties associated with trying to cater for different cognitive styles: the vast number of styles as already noted, many styles have been identified, researched and reported but the vast majority can be coalesced into a continuum ranging from a heavy dependence on structure and guidance, to a strong preference for minimal structure and guidance. the measurement of styles many measures of cognitive style have to be individually administered such as the rod-and-frame test for field dependence-independence (witkin et al, 1962) and hence are not timeand cost-effective. most of these individual measures are complemented by group pencil-and-paper tests, the group embedded figures test for field dependence-independence, for example (witkin et al, 1971) but these still require considerable time to complete and analyse. incorporating the measure of style into the cal program research investigating the relationship between cognitive style and cal performance has invariably measured style independently of the user's involvement with the program and has used non-computer-based procedures (e.g. james and james, 1990; rowland and stuessy, 1987). this again is time-consuming, and feedback on the appropriate type of program for the user is delayed. ideally, any measure of style should occur as an immediate prerequisite to involvement in the program. a computer-based measurement procedure would also streamline the process. 48 alt-j volume 1 number 1 preparing appropriate versions of the same program to cater for different styles. most research focuses either on how cal users with different styles cope with a particular cal presentation (e.g. clements and gullo, 1984; post, 1985) or on learners' comparative performance in cal versus other non-computer-based modes of presenting material (e.g. dahl, 1985; wesley, 1984). there has been some research that has attempted to match style with an appropriate cal presentation (e.g. rowland and stuessy; 1987). a possible solution to overcome these problems, what is required is the identification of a style which is meaningful in the context of learners interacting with cal materials, which can be measured easily and reliably and which can be incorporated in the cal presentation. once this is achieved, the next step is to prepare an appropriate cal package which can accommodate the different learning styles which have been identified. the earlier discussion highlighted a style which not only subsumes many of the disparate styles in the literature but is also appropriate to cal presentation. the style ranges on a continuum from a preference for a reasonable degree of structure and guidance to be provided in the learning situation, to a preference for considerable freedom to choose the direction that learning should take. such a style has been identified by pask and his co-workers (pask, 1973, 1976; pask and scott, 1972). the learners at the two extremes of the continuum are referred to as 'serialists' and 'holists'. serialists prefer to learn just the essential facts which should be well defined, sequentially ordered and related by simple links. they like to use a simple step-by-step strategy and consequently regard embellishments such as anecdote, illustration and analogy as quite distracting. holists, on the other hand, prefer a more global approach to problem-solving. they use more complex non-linear 'globetrotting' strategies, and thrive on anecdote, illustration and analogy. by focusing on a 'global' approach, they concentrate on synthesizing broad overviews into which details may be fitted. by contrast, serialists use more of a 'local' approach, concentrating on specifics which may then lead to the emergence of an overall picture. individuals able to function happily either way are referred to as 'versatile'. entwistle (1978,1981) summarizes the research associated with this serialist/holist style. application of the serialist/holist style to cal identifying an appropriate style is only the first step. what is also required is a quick and reliable measure of that style which can be incorporated into a cal program. the original identification procedures used by pask (1973) are pencil-and-paper tests, and are quite lengthy and complex. building on pask's work, entwistle (1981) developed a more efficient 30-item short inventory of approaches to studying. in turn, ford (1985) has developed an 18-item questionnaire, study preference questionnaire (spq) but unfortunately provides no information on its reliability. a copy of the questionnaire is given in the appendix. the requirement here is a quick and reliable measure of style. ford found that using a small english sample of 25 postgraduate students, one of the items (question 16) in the spq significantly predicted serialist or holist tendencies at p<0.01 level. the item is: 49 john clarke cognitive style and cal: problems and a possible solution when i'm reading a book (or other information source) for my studies, i prefer to work * through it fairly logically from beginning to end. 2 3 4 when i'm reading a book (or other information source) for my stu* dies, i prefer to skip about and and 'dip in' quite a lot. (see the last page of the appendix for the interpretation of the response scale.) if a similar finding, derived from a reliable questionnaire can be produced using a sample for whom cal programs are to be produced australian in this case it would provide a simple method of indicating the cal user's cognitive style, and it could be very simply incorporated into any cal program. an australian replication of the ford study the ultimate aim of the project in which i am currently engaged (clarke, in process) is to produce cal materials for teacher-education students in educational psychology. hence, any replication of ford's work needs to involve such students. the spq was administered to 56 students involved in a one-year postgraduate pre-service teacher-education course at a large teacher-education institution in brisbane. all students had completed an undergraduate qualification at a different tertiary institution. the responses were analysed with the aims of producing a reliable questionnaire and of determining if responses to any individual question were significantly correlated with responses to the full questionnaire. the data was analysed using modules of the statistical package systat (wilkinson, 1988) and the results are reported below.2 results the reliability of the 18-item questionnaire was estimated by using the coefficient alpha procedure (cronbach, 1951). then, the correlation of each item with the alpha value, and simultaneously the effect on alpha of removing individual items from the calculations, were measured. if the value of alpha increases with the removal of an item, it is possible to conclude that the item is not measuring the same characteristics as the total test and in fact is interfering with obtaining a 'pure' measure of those characteristics. this process of removing items therefore increases the reliability of the test. it is continued until alpha is not increased by any further removal of items. five items were removed using this procedure. the results are summarized in table 1. the outcome of this analysis is the identification of a 13-item test which is much more reliable than the original 18-item measure. the alpha value increased from 0.594 to 0.670 which is significant at the 0.01 level. . 50 alt-j volume 1 number 1 coefficient alpha 0.594 0.635 0.657 0.668 0.669 0.670 item no. 2 12 9 11 8 no item with coefficient alpha if item exluded 0.635 0.657 0.688 0.669 0.670 alpha > 0.670 table 1: summary of the item elimination procedure table 2: individual item-total measure correlations item number 1 3 4 5 6 7 o s 10 13 14 15 16 17 18 item-alpha correlation 0.418 0.416 0.359 0.595 0.423 0.628 0.404 0.428 0.446 0.412 0.4 27 0.509 0.356 the ultimate aim here is to identify an individual item that can be used instead of the total (now reduced) questionnaire. the correlations between each of the remaining 13 items and the coefficient alpha are shown in table 2. question 7 has the highest correlation and hence could possibly be used to identify those students with serialist or holist tendencies. the item is as follows: 7. when i'm reading a book (or other information source) for my studies, i prefer to spend quite a long time skimming over .. and dipping into it to get a clear picture of what it's about and how it will be relevant. l o j a h when i'm reading a book (or other information source) for my studies, i prefer to get quite soon into a fairly detailed reading of it once i know that it's going to be useful, in the knowledge that its precise relevance and contribution will become clear from a detailed reading 51 john clarke cognitive style and cal: problems and a possible solution it is outside of the scope of this paper to speculate why question 16 did not emerge as the most important item as in the ford study.3 there are, however, a number of ways in which this data could be used to help cater for the different cognitive styles of cal users (clarke, 1990). one method is to incorporate question 7 into the cal program. incorporation of question 7 into a cal program since only one question is involved, it is a simple procedure to place it at the beginning of a cal package and use the response to direct users to the appropriate tutorial-type or welcome to the school of learning and development's computer based learning series of programs program 1 how students learn there are two different versions of this program available. the computer will allocate you to version a or version b according to your response to this question: when i'm reading a book w h e n vm r e a d i n 9 a b o o k (or other information (or other information source) source) for my studies, i f o r m y s t u d i e s ' p r e f e r t 0 prefer to spend quite a 9 e t ? u i t e s o o n i n t 0 a f a i r | y long time skimming over 1 2 3 4 5 detailed reading of it once i and dipping into it to get a k n o w it>s 9 o i n 9 to be useful, clear picture of what it's i n t h e knowledge that its about and how it will be precise relevance and conrelevant tribution will become clear from a detailed reading. 1 = i agree with the statement on the left 2 = i agree (with reservations) with the statement on the left 3 = no preference for either statement 4 = i agree (with reservations) with the statement on the right 5 = i agree with the statement on the right please place a response ( 1 , 2 , 3 , 4 or 5) in the space below: your response figure 1: procedure which could appear at the beginning of a cal lesson 52 altj volume 1 number 1 simulation-type version of the program. an appropriate procedure could be as shown in figure 1. this could appear as a frame at the beginning of a cal lesson. those who choose 1 or 2 will be allocated to version b; those who choose 4 or 5 will be allocated to version a; those who choose 3 will be told that it doesn't really matter which version they use, and will be given the option of choosing version a or or version b. the next steps there are at least two things that now need to be done. first, the preparation of a suite of cal programs where each program is available in two versions tutorial and simulation; and secondly a measure of the predictive validity of question 7. work is proceeding on the former (clarke, in process). two versions of a program designed to introduce pre-service teacher education students to 'how children learn' are being developed. once these are available, work can proceed on the latter. the second issue, that of predictive validity, can be addressed in several ways. one approach would be to gather data from users on the appropriateness or otherwise of the allocation made as a result of their response to question 7 and their satisfaction with the type of learning experience they subsequently have. another approach would be to investigate the relative reliability of an allocation based on one question, compared to an allocation based on a small group of questions or an allocation based on simple user choice premised on information about the two versions. notes 1. an earlier version of this paper was presented at the seventh annual conference of the australian society for computers in learning in tertiary education, bond university, gold coast, queensland, australia, 10-13 december, 1989. 2. the assistance of dr d. chant, statistical adviser to the social sciences, university of queensland, in this analysis is gratefully acknowledged. 3. one likely explanation is the potentially interfering effects of those items removed in the reliability analysis reported here. other explanations could relate to differences in the size and characteristics of the samples. 53 john clarke cognitive style and cal: problems and a possible solution references barker, p.g. (1982), 'some experiments in man-machine interaction relevant to computerassisted instruction', british journal of educational technology, 13, 1, 65-75. chadwick, s.s. and watson, j.a. (1986), 'computers and cognitive styles', academic therapy, 22, 2, 125-32. clarke, j.a. (1990), 'designing cal programs to cater for different learning styles', in mcdougall, a. and dowling, c. (eds), computers in education, amsterdam, elsevier science publishers, 351-8. clarke, j.a. (in process), 'the production of a computer-assisted learning (cal) package for tertiary students that is designed to cater for the preferred learning styles of students', report of a professional experience program (pep) being undertaken at the cal unit, university of queensland. clements, d.h. and gullo, d.f. (1984), 'effects of computer programming on young children's cognition', journal of educational psychology, 76, 6, 1051-8. coop, r.h. and sigel, i. (1971), 'cognitive style: implications for learning and instruction', psychology in the schools, 8, 152-61. cronbach, l.j. (1951), 'coefficient alpha and the internal structure of tests', psychometrica, 35, 509-11. dahl, r.d. (1985), 'interaction of field dependence-independence with computer assisted instruction structure in an orthographic projection lesson', unpublished doctoral dissertation, iowa state university, dissertation abstracts international, 45, 2012a. duncan, m.v. (1985), 'computer programs and learning styles', computers, reading and language arts, 2, 2, 8-12. entwistle, n. (1978), 'knowledge structures and styles of learning: a summary of pask's recent research', british journal of educational psychology, 48, 225-65. entwistle, n. (1981), styles of learning and teaching, new york, john wiley. ford, n. (1985), 'learning styles and strategies of postgraduate students', british journal of educational technology, 16, 1, 17-26. james, j.s. and james, l.l. (1990), 'learning styles and computer programming activities', in mcdougall, a. and dowling, c. (eds), computers in education, amsterdam, elsevier science publishers, 397-402. jougnin, g. (1989), 'adult learning principles and field-dependence/independence in training program design', paper presented at the australia and south pacific external studies association ninth biennial forum, gippsland institute of advanced education. kogan, n. (1971), 'educational implications of cognitive styles', in lesser, g.s. (ed), psychology and educational practice, illinois, scott foresman, 242-92. 54 alt-j volume 1 number 1 messick, s. (1970), 'the criterion problem in the evaluation of instruction: assessing possible, not just intended outcomes', in wittrock, m.c. and wiley, d.e. (eds), the evaluation of instruction: issues and problems, new york, holt rinehart and winston, 183-202. messick, s. (1976) 'personality consistency in cognition and creativity' in messick, s. (ed), individuality in learning, san francisco, jossey-bass, 4-22. messick, s. (1984), 'the nature of cognitive style: problems and promises in educational practice', educational psychologist, 19, 2, 59-74. pask, g. (1973) educational methods using information about individual styles and strategies of learning, final report of the ssrc project hr 1424/1 (2 volumes). pask, g. (1976), 'styles and strategies of learning', british journal of educational psychology, 46, 128-48. pask, g. and scott, b.c.e. (1972), 'learning strategies and individual, competence', international journal of man-machine studies, 4, 217-53. post, p.e. (1985), 'the effect of students' field independence/dependence on computerassisted instruction achievement', unpublished doctoral dissertation, purdue university, dissertation abstracts international, 45, 2013a-2014a. rowland, p. and stuessy, c.l. (1987), 'effects of modes of computer-assisted instruction on conceptual understanding and achievement of college students exhibiting individual differences in learning: a pilot study', paper presented at the annual meeting of the national association of research in science teaching, washington dc. sims, r. (1988), 'towards learner-centred computer-assisted learning' in fielden, k., hicks, f. and scott, n. (eds), computers in learning in tertiary education, canberra, ascilite. sharma, s. (1987), 'learners' cognitive styles and psychological types as intervening variables influencing performance in computer science courses', journal of educational technology systems, 15, 4, 391-9. wesley, b.e. (1984), 'the effect of computer-assisted instruction and locus of control upon preservice elementary teachers' acquisition of computer literacy and the integrated process skills', unpublished doctoral dissertation, purdue university, dissertation abstracts international, 44, 3652a. wilkinson, l. (1988), systat: the system for statistics, evanston, il, systat inc. witkin, h.a., dyk, r.b., faterson, h.f., goodenough, d.r. and karp, s a . (1962), psychological differentiation: studies of development, new york, john wiley. witkin, h.a., oltman, p.k., raskin, e. and karp, s.a. (1971), a manual for the embedded figures test, consulting psychologists press. 55 john clarke cognitive style and cal: problems and a possible solution appendix study preference questionnaire 1. when i'm reading a book (or other information source) for my studies, i generally tend to concentrate on certain parts, and skip over others quite markedly, going back later if necessary to fill in any 'gaps' or 'missing links'. when i'm reading a book (or other information source) for my studies, i tend to follow the author's presentation reasonably closely, rather than skipping about a lot. 2. when i'm studying for an essay, i try to gather as much 1 information as possible at the start. i prefer to have more of a 5 'steady flow' throughout my preparation for the essay. 3. when i'm studying i generally prefer to deal reasonably thoroughly with one book (or other information source) before moving on to another. i generally prefer to have a number of books (or other information sources) 'on the go' at the same time. 4. when i'm studying a new subject, i tend to want to keep the whole picture of the subject in my mind all the time and find it hard "to concentrate on very detailed aspects unless i can constantly relate them clearly to the broad overall picture. once i've analysed the subject into its component parts, i like to focus on each of these parts in detail, systematically building up the overall picture bit by bit. 5. when i'm in the library, generally speaking i tend to be looking for specific books, etc. rather than browsing. i tend to spend quite a lot of time browsing in the library. 56 1 2 3 4 5 2 3 4 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 alt-j volume 1 number 1 6. for students who learn in the way i do, the best form of training in library use would be to follow a well structured course, building up from the basics and going on to more complex skills that will be essential for them to know. for students who learn in the way i do, the best form of training in library use would be to learn skills (whether simple or complex) as they encounter the need for them during their studies (a sort of 'on demand' teaching). 7. when i'm reading a book (or other information source) for my studies, i prefer to spend quite a long time skimming over and dipping into it to get a clear picture of what it's about and how it will be relevant. when i'm reading a book (or other information source) for my studies, i prefer to get quite soon into a fairly detailed reading of it once i know that it's going to be useful, in the knowledge that its precise relevance and contribution will become clear from a detailed reading. 8. generally i prefer to concentrate on one (or a very few) aspect(s) of a subject at a time when i'm learning about it. generally i prefer to be learning about a number of different aspects of a subject at the same time. 9. once i've done a basic analysis of what the subject involves, i'm happy to put to the back of my mind temporarily the broad overall picture while i do some work on more detailed aspects. i tend to want to keep the overall picture uppermost in my mind the whole time, and only work on more detailed aspects if i can see precisely how they will fit into this picture. 10. summaries of a wide range of books (and other information sources) would not really be much use to me when i'm doing an essay i really need the full texts more or less straight away. summaries would be very useful in the initial stages of my work on an essay. 57 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 john clarke cognitive style and cal: problems and a possible solution 11. i like to approach a new subject in a broad way often looking at widely spaced aspects of the subject and seeing how they may all fit together, before going back to 'fill in' any strictly logical steps that i have skipped. i like the logical links between different aspects of a new subject to be very close, so that when i'm learning about a second aspect i can see very clearly how it relates to the first aspect that i have already learned about, and so on. 12. when i'm doing an essay, i think that i tend to use rather fewer information sources than most students on the course. i think that i tend to use 5 rather more information sources than most people on the course. 13. i find it too restrictive to wait until i have thoroughly 'mastered' one aspect of a new subject i am learning about before going on to study other aspects. 14. where a book chapter or journal article includes a separate summary of what it is about, i generally prefer to get straight into the main text since it contains all that is in the summary anyway. 15. when i'm studying for an essay, i like to start by 'soaking in' a wide range of information in order to get the 'feel' of the subject i l i k e to d e a l f a i r l y thoroughly with the particular aspect i'm working on before going on to study others. generally i prefer to read the summary before reading the full text even though it will all be found in greater detail in the main text. i prefer to analyse the topic fairly early on, and search for information which is more clearly focused on particular aspects of the topic. 16. when i'm reading a book (or other information source) for my studies, i prefer to work through it fairly logically from beginning to end. when i'm reading a a book (or other information source) for my studies, i prefer to skip about and 'dip in' quite a lot. 58 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 alt-j volume 1 number 1 17. where a book or journal has a separate summary of conclusions i prefer to get straight into the main text, where each conclusion is presented along with the evidence on which it is based. i prefer to spend quite a bit of time reading the summary of conclusions before going on to a detailed reading of the main text. 18. when i'm learning about a new subject, i like to keep coming back to particular aspects on different occasions, to get further details as i require them. i prefer to deal with particular aspects of a new subject as thoroughly as i can at one time, so that i need not keep returning to them on different occasions to get further details. response scale 1 = 1 agree with the statement on the left 2 = 1 agree (with reservations) with the statement on the left 3 = no preference for either statement 4 = 1 agree (with reservations) with the statement on the right 5 = 1 agree with the statement on the right 59 1 2 3 4 5 1 2 3 4 5 (page number not for citation purpose) 1 *corresponding author. email: simon.leonard@unisa.edu.au research in learning technology 2018. © 2018 n simon et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 research in learning technology vol. 26, 2018 original research article holographic learning: a mixed reality trial of microsoft hololens in an australian secondary school simon n. leonarda* and robert n. fitzgeraldb aschool of education, university of south australia, adelaide, australia; bcharles darwin university, darwin, australia (received: 24 september 2018; final version received: 28 october 2018) the technology supporting augmented and mixed reality educational environments is advancing with recent hardware including self-contained headsets that are able to simulate holographic additions to real spaces. these technical advances appear to offer greater capacity to actually realise the educational potential and promise of such technologies noted in the literature over the last decade. this article adds to this literature by reporting on the pilot phase of an educational design research project using the microsoft hololens device in a secondary school setting in australia. consistent with previous research in this area, this project found ongoing technical and managerial limitations in implementing augmented and mixed reality, including a continuing concern by many participating teachers of a lack of control of the mixed reality environment. notably, the pilot study also revealed different understandings of the potential for embodied learning between students, teachers and researchers that requires further research. keywords: mixed reality; augmented reality; embodied learning; expansive learning this paper is part of the special collection mobile mixed reality enhanced learning, edited by thom cochrane, fiona smart, helen farley and vickel narayan. more papers from this collection can be found here introduction microsoft hololens is a mixed reality (mr) device that allows the user to engage with digital content and interact with both real-world artefacts and virtual artefacts in the form of holographic-like images. there is some contestation and confusion regarding the definition of mr; however, following milgram and kishino (1994), it is perhaps useful to see it, along with augmented reality (ar), as lying somewhere on the continuum between the natural world and a completely virtual environment. a particular feature of the microsoft hololens device that relates to this special issue is its high level of mobility. the device places images onto a transparent near-eye screen so as to create the illusion of a holographic image in real space. notably, the device accounts for the user’s movement in space and adjusts the image accordingly so that it appears to act like a real object. it does this in an independent stand-alone headset that is not tethered. in addition, and unlike tablet or phone-based ar solutions, it leaves the natural world completely visible, allowing the wearer to move safely and freely mailto:simon.leonard@unisa.edu.au http://www.alt.ac.uk http://creativecommons.org/licenses/by http://dx.doi.org/10.25304/rlt.v26.2160 https://journal.alt.ac.uk/index.php/rlt/pages/view/collections s.n. leonard and r.n. fitzgerald 2 citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 (page number not for citation purpose) around a room or even larger area while engaging with virtual and natural objects. in providing such affordances, the microsoft hololens appears to offer a significant step towards realising the educational potential long seen in mr, virtual reality (vr) and ar technologies. the pilot phase of an educational design research project reported in this paper confirms this potential, while also pointing to significant technical and theoretical challenges. the pilot reported here was part of an educational design research project (mckenney and reeves 2012). the project was conducted in a well-resourced australian secondary school, with extensive support from the research team and a major academic publishing company. as is standard in educational design research, the pilot took a developmental stance. it was an opportunity for different actors in the design process – teachers, students, researchers and content designers – to work together to better understand the design and research problems. much of what the pilot discovered was not unexpected: the technology was broadly engaging and has a ‘wow factor’; some teachers saw great potential; and there were numerous implementation issues, both technical and pedagogical, even in the ideal conditions in which the pilot took place. one thing that was not anticipated, however, was an uncertainty from both teachers and students in thinking about the ‘enacted’ rather than visualisation affordances that the mobility the microsoft hololens device provided. that is, unlike the research team, the teachers and students involved in the pilot tended to use other visualisation technologies as their frame of reference for the use of this technology. in doing so, they tended to ‘bootstrap’ the technology to established classroom teaching and learning approaches and so restricted the capacity to think of the possibilities for the technology in its own right. the purpose of this article is to argue the need of establishing an ‘expansive’ design framework when embarking on projects that offer new ways of doing teaching and learning. the argument will be made through reflecting on this pilot while engaging with relevant literature. in doing so, the article reflects on the messiness of the design process. it moves from data to discussion and back again. in broad terms, however, it starts by seeking to understand what happened in the pilot and then moves to exploring where the project ought to go next. a mixed reality before we move on to describing the pilot, it is worthwhile pausing to consider the theoretical orientation that the research team brought to the technology used in this project. although both mr and ar provide visual simulations, they are different from completely virtual environments in ways that are educationally and theoretically important. mr, vr and ar technologies are all systems that have the ability to alter human perception (milgram and kishino 1994). the technologies developed have generally targeted visual and auditory perceptions, although haptic (touch) systems have also been developed (carijó, de almeida, and kastrup 2013). the technologies classified as vr have sought to create a completely artificial environment and provide the user with some capacity to interact with that environment. in doing so, they typically reduce or eliminate the capacity to interact with the real world. vr headsets, for example, allow an user to see only the virtual world created and make the user ‘blind’ to the real world around them. these technologies http://dx.doi.org/10.25304/rlt.v26.2160 research in learning technology citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 3 (page number not for citation purpose) have been well developed for the video games industry and become relatively cheap through the adaptation of mobile phone screens to serve as vr headsets, such as the samsung gear. the terms ‘augmented’ and ‘mixed’ reality, on the other hand, are often used interchangeably to describe technologies that alter perception, but still allow interaction with the real world. well-known examples of this are the abandoned google glass device and phoneor tablet-based solutions such as the game pokémon go. the differences between vr and ar/mr have important practical and theoretical implications for learning design and there is a strong argument available to set aside the technical similarities of the technologies and treat them separately (hugues, fuchs, and nannipieri 2011). in many respects, the affordances of vr have been well explored in the literature on the educational use of video games (gee 2006; waddington 2015), although the immersive nature of more advanced vr technologies appears to enhance these effects ( clark, tanner-smith, and killingsworth 2016). merchant et al. (2014) conducted a meta-analysis of desktop vr applications (i.e. games, simulation and virtual worlds) and concluded the following: the results of this meta-analysis are encouraging in that they provide evidence that virtual reality-based instruction is an effective means of enhance [sic] learning outcomes. educational institutions planning to invest time and financial resources are likely to see the learning benefits in their students. (p. 37) vr seems to have particular potential to improve spatial reasoning. in a recent study, for example, pan and niemeyer (2017) reported on how users’ experience of catching a real ball was impacted while they were immersed in vr. they showed that vr environment positively impacted the catching task and suggested that improved physical performance occurred because the physical task was able to be simplified through spatial vr affordances. such studies connect to a growing body of multi-disciplinary research that highlights the importance of spatial reasoning, and especially spatial skills associated with spatial visualisation, mental rotation and spatial orientation (newcombe 2010; stieff and uttal 2015) that show a positive correlation with mathematical competence and future success in science, technology, engineering and maths (stem) fields (lubinski 2010). ar/mr, however, seems to provide something other than visualisation. of particular relevance to this special issue is that they offer mobility, and therefore sensory– motor engagement in the real world. this was a standout affordance in the initial design thinking of our research team. the importance of such engagement to human cognition has become increasingly apparent within research in cognitive psychology over the past one or two decades, although it can be difficult to find because it goes under many names such as embodied, embedded, enacted, extended and situated cognition (kiverstein and clark 2009). ‘extended’ cognitive science views cognition as being connected in a mind–body– world system (clark 2011). this science tells us that the human brain is capable of incorporating tools into its motor and haptic systems; we don’t feel through the tool, rather the mind treats the tool as if it is an extension of the body. (carijó et al. 2013; froese et al. 2012). it shows that the mind can use the body in more abstract ways, such as the increasing evidence that gesture influences learning, both by influencing the learner’s environment and by influencing the learners themselves (goldin-meadow and wagner 2005). gesture appears to be particularly important when tasks are unfamiliar (johnson-glenberg and megowan-romanowicz 2017; logan, lowrie, and http://dx.doi.org/10.25304/rlt.v26.2160 s.n. leonard and r.n. fitzgerald 4 citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 (page number not for citation purpose) diezmann 2014), through providing the cognitive system with a stable external and visual presence that can provide the means with which to think (pouw et al. 2014). an important finding in our pilot study that we will report below, however, was that the teachers and students we were working with did not appear to have access to extended models of cognition when thinking about educational design. they almost entirely drew on the visualisation affordance of the device when thinking about how the device might be used and gave no consideration at all to its mobility and interactivity. although a tentative finding from a pilot study, this finding may have important implications for how researchers can work with teachers on the use of increasingly mobile technologies in education. it suggests that the teaching workforce may not have the extant conceptual models to take full advantage of the affordances of these technologies and that significant professional learning may be required. it supports questions such as those raised by hamilton, rosenberg and akcaoglu (2016) on the capacity of current practice-based models, such as the popular substitution, augmentation, modification, and redefinition (samr) model (puentedura 2006) to actually ‘redefine’ classroom practice. educational trial the project reported here took an educational design research (kelly, baek, and lesh 2008; mckenney and reeves 2012) stance. this approach, also known as designbased research, blends the systematic development of solutions to educational problems with scientific research. owing much to the approach of the design sciences such as engineering, it places a priority on developing useful solutions while also capturing theoretical understandings that can support the work of others. a common first step within educational design research is for researchers and practitioners to analyse the problem space together. later phases of educational design research involve the development and evaluation of solutions leading to the documentation of design principles. this article reports on the first phase of the project only, that is, it should be read as a collaborative exploration of the problem space, seeking to define more so than resolve the problems of practice and theory to be engaged. the project arose from a partnership between the publishing company, a large secondary school in australia and our research team. the project also received some support from the manufacturer, although the manufacturer did not participate directly in the trial. in this pilot phase, each of the partners was seeking to understand more about the educational possibilities of the microsoft hololens mr device in schools and an educational design research stance was taken to explore the researchable and designable aspects of the technology together. the publishing company was also interested in feedback on a number of apps they had developed for use with the device. the publishing company and the manufacturer had a clear commercial interest in promoting the use of this technology and the associated apps and entered the project seeking to develop high-quality products for the school sector. the school involved in the study was an independent preschool-12 school in a metropolitan location. the project involved only the 7–12 sections of the school. the school was well resourced with a solid it infrastructure and the financial capacity to invest in emerging technologies. the authors of this article and their university technical team worked with employees of the publishing company to facilitate http://dx.doi.org/10.25304/rlt.v26.2160 research in learning technology citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 5 (page number not for citation purpose) design workshops on how the apps that had been developed by the publisher might be integrated into teaching and learning at the school. teachers then trialled these initial designs with students from the school in subjects including chemistry, physics, music and health. coincidentally, the trial was conducted in the first year the school had become a co-educational school and there were still many more boys than girls enrolled in the school, although no significant gender difference was discernible in the trial data. seeking to better understand the problem space from the diverse perspectives available even in this small trial, evaluative data were collected in the forms of researcher field notes, a student survey and a number of semi-structured focus group interviews with the teachers involved. collected with the approval of the university’s ethics committee, these data were intended to support a process of reflection or developmental evaluation (leonard, fitzgerald, and riordan 2016). although the reflective process is the focus of this article, we will briefly provide some insight into the data that informed the process in the following section. some findings from the data the first data source was a brief survey of students comprising 17 questions. the first 14 questions were rated on an 11-point scale (0–10), while the last three questions called for a written response. seventy-three valid responses were received, representing a response rate of 50%. table 1 summarises the means and standard deviations for each of the rated items. acknowledging that these scales were developed with certain dimensions in the minds of the researchers, we sought to test the meaningfulness of the implicit dimensionality. to achieve this, a principal components analysis (pca) was conducted using spss software. pca is a variable reduction technique that is used to identify the most useful variables in a data set and provides a way of exploring the common themes or constructs that they represent. in terms of this evaluation, the aim was to explore whether there were patterns in the way students responded to the apps that went beyond simply enjoyment and engagement. a pca analysis with varimax rotation was performed in spss, which resulted in a three-component solution summarised in table 2. table 1. summary of student responses to survey: mean and sd. this app would make learning more interesting 8.99 1.643 this app was enjoyable to use 8.49 1.687 this app helped me see or visualise the main idea 8.46 1.913 i would like teachers to use this app in the classroom 8.39 2.135 i found it helpful to be able to walk around the object(s) 8.27 2.191 i found it helpful to be able to move/place the object(s) 7.93 2.434 this app helped me understand the main idea 7.89 2.032 this app would help me learn better than normal classroom activities 7.79 2.420 this app would help me learn/work with others 7.78 2.124 this app worked well 7.57 2.128 i found it helpful to be able to make the object(s) bigger/smaller 7.34 2.524 this app encouraged me to talk to others about the main idea 7.17 2.662 i found it helpful to be able to rotate the object(s) 6.64 3.420 this app was easy to use 6.40 2.448 http://dx.doi.org/10.25304/rlt.v26.2160 s.n. leonard and r.n. fitzgerald 6 citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 (page number not for citation purpose) table 2. pca analysis of student survey. component 1: engagement • this app was enjoyable to use • this app helped me see or visualise the main idea • this app helped me understand the main idea • this app worked well • this app was easy to use • this app encouraged me to talk to others about the main idea component 2: learning impact • i would like teachers to use this app in the classroom • this app would help me learn better than normal classroom activities • this app would help me learn/work with others • this app would make learning more interesting component 3: spatiality/perspective affordance • i found it helpful to be able to make the object(s) bigger/smaller • i found it helpful to be able to rotate the object(s) • i found it helpful to be able to walk around the object(s) • i found it helpful to be able to move/place the object(s) even though this was a small sample, the results are challenging and suggest that the students are thinking about their experience with this technology in quite complex ways. this analysis shows that while the students quite uniformly found the technology engaging even if it was difficult to use (component 1), they believed it to have a positive impact on learning (component 2), but they showed some ambivalence on its spatial and perspective affordances (component 3). notably, of all the factors measured, the ability to rotate objects was rated particularly poorly. another data source was a series of semi-structured focus group interviews carried out with the teachers involved in the trial. guided by a predetermined set of questions, the focus groups included representatives from all parts of the partnership. transcripts of the focus groups were analysed using a computer-assisted phenomenography (cap) technique (leonard and roberts 2014) supported by the leximancer software. leximancer uses a corpus linguistic approach to textual analysis and identifies concepts used within text, mapping those concepts and the relationships between them. the result is not dissimilar to the pca analysis in that it identifies concepts that ‘hang’ together. in practical terms, the analysis identifies themes or a cloud of concepts that are frequently discussed in relation to each other across the conversation. table 3 summaries this analysis and shows the major themes identified (column 1) and an example of relevant text from the transcriptions (column 2). evident even in the limited quotes included in table 3, the professional discourse here was striking. across several hours of combined conversation, the teachers consistently sought to identify known approaches from within their current learning design repertoire to connect their students and the new technology. in doing so, they consistently took an incremental approach and identified ways to redesign the known learning environment. at one level this reflects the early iterative stages suggested by transformation models such as samr (puentedura 2006). however, the teachers do not seem to show any signs of the reconceptualisation level of that model. as we can see in those quotes, the microsoft hololens was seen as an advance on video and powerpoint, allowing a student to engage in independent study while the teacher worked with her class, and offering an improved visualisation of content such as the human circulatory system. http://dx.doi.org/10.25304/rlt.v26.2160 research in learning technology citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 7 (page number not for citation purpose) there were exceptions to the overall trend of the discussion. for example, a music teacher saw the potential for new learning design through the use of an app on human anatomy: … [w]e were very interested in reminding ourselves of, of our [human body] structure – we’ve been talking a lot lately of posture and when we play music and it was very useful for the students to be able to see these models. it is hard to sort of visualise what is going on inside your body, you know. these exceptions, however, were rare and the overall shape of the professional discussion reflected the reports of the students via survey. that is, the way teachers and students in the trial engaged with the new technology was almost always with reference to current learning design and did not align well with the theoretical orientation of the research team. this finding is based on limited data. as we have outlined, this is the nature of the pilot phase of educational design research and our purpose is to widen the problem space rather than offer conclusive solutions at this stage. however, it seems to be a valid problem to explore. the teachers involved in this project were highly successful and well-respected, and the students were highly able too. these factors highlight that creative pedagogical design is extremely challenging, that the culture of schools conforms to well-understood modes of working and that it is not overly surprising that it has proven to be difficult to scale and sustain changes in enacted teaching and learning through educational design research projects (fishman et al. 2011). reflections, directions and practices the goal of this pilot phase of an educational design research project is to open up the  problem space through researchers and practitioners working together. table 3. cap analysis of focus groups discussion. theme exemplar quote things in and of the learning environment oh well absolutely, because there, certainly there’s loads of things, as you would be aware, the fact that they’re small and portable, is really, you know, you could take them outside, you don’t have to use them in the classroom space. you could, if they are designed for kids with learning needs or if there are those applications there, you could stick a kid in the side room with it and get them to carry on with their activity. students i mean, i’m not so sure if the technology so once again i’m so removed from what the technology has the capacity to do. but certainly you could get the students to… just like in a task you would ask a student to do a video or ask a student to do a powerpoint or something like that. hololens …as i said, i took the hololens home and had a bit of a play through some of the other apps so particularly things like the body systems, so they have a bit of a digestive system and a circulatory system and there was also the atom building kind of app that was there. http://dx.doi.org/10.25304/rlt.v26.2160 s.n. leonard and r.n. fitzgerald 8 citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 (page number not for citation purpose) the  previous section provided just a brief example of this and suggested areas for consideration on future design and research in this project. in this section, we will reflectively expand on these ideas through ‘conversation’ with the literature. to begin this conversation, we note that educational research on the use of mr technologies is really in its infancy. despite ar/mr technologies being around for some decades, the widespread and affordable availability of these technologies is still emerging (melatti and johnsen 2017). as commercial solutions have become available, a number of reviews have been undertaken (see, e.g., akçayır and akçayır 2017; bacca, baldiris, and fabregat 2014; cheng and tsai 2013; dunleavy, dede, and mitchell 2009; radu 2014; sung, yang, and lee 2017). to some extent these reviews highlight the immature nature of research in this area. in a review of 32 studies on ar published in the top five educational technology journals, bacca et al. (2014), for example, found that nearly all research reported was of case or pilot studies, with only one study adopting a causal research method. this review also found that nearly all of the learning designs reported were quite basic and focussed on the use of ar or mr to explain a topic or to provide additional information about a topic. despite the immature nature of the studies considered, bacca et al. (2014) found a consistent reporting of positive impacts of ar and mr on both learning and motivation . these positive findings were consistent with another review by radu (2014). considering 56 studies, this analysis found ar/mr to have a positive impact on the learning of spatial structures and language associations, as well as contributing to improvements in long-term memory retention, collaboration and motivation. both reviews, however, identified usability and classroom integration as continuing challenges across the field. the findings of our pilot phase are consistent with those in the literature, in that students and teachers reported a belief that the technology supports engagement and learning, but also that there are usability issues and areas for technical improvement. some of our partners also spontaneously suggested some of the benefits found in these studies, including motivation and collaboration. this spontaneous discernment, however, did not go beyond the enhancement of existing student practice or activity. our students and teachers did not ponder the possibilities of improved memory retention, language associations or – with the exception of the music example we quoted – spatial structures. our research is translational. we seek to understand how advances in basic research can be applied to innovation and change in real-world settings. in this context, we must be mindful of this apparent gap between professional teacher focus and the cutting edge of the research. at the same time, we must also be aware that, at the cutting edge, a range of opportunities are being conceived. in another review of 87 studies, for example, santos et al. (2014) found that ar/mr was effective in providing real-world annotation, contextual visualisation and vision-haptic visualisation. notably, this analysis found that majority of the studies reviewed called upon multimedia learning theory (mayer 2014) and experiential learning theory (kolb 1984) in understanding the educational benefits of the technology. this suggests that designers are seeing the potential for more complex and activity-oriented learning designs. the activity they suggest, however, is quite different from the existing modes of practice that our teachers and students were able to suggest. this points to a need in our project to design not only new activity but also ways to support a different understanding of the nature of student and teacher activity. http://dx.doi.org/10.25304/rlt.v26.2160 research in learning technology citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 9 (page number not for citation purpose) santos et al. (2014) analysis also shows a growing appreciation that the motivational benefits of ar are not driven by the novelty value of the ‘wow factor’ of technology alone. a number of studies in the review pointed also to the motivational effects of easing cognitive load and providing support for situated rather than decontextualised understandings. these findings are promising and have been confirmed in a more recent review by akçayır and akçayır (2017), who found evidence of ar/mr leading to heightened levels of motivation, engagement and interest; increased opportunities for interaction; decreases in cognitive load; and enabling the visualisation of abstract concepts. studies in this review also found evidence of ar/mr leading to stronger interaction among students (kamarainen et al. 2013), between students and learning materials (hsiao, chen, and huang 2012), and between students and teachers (zarraonandia et al. 2013). enyedy et al. (2017) highlight the social affordances of mr. in moving beyond a primary focus on technical implementation, and by working with young children in first and second grades, this study suggests that educational research in this area is beginning to mature. the study reports on a project that uses mr to support children in ‘doing science like scientists do’ through asking questions, modelling phenomena and arguing from the evidence. the study concluded that it was the ‘social aspects of the space, not the technical aspects alone’ that had the greatest impact on student learning. also notable within the learning design was that the students had ‘agency to pursue their emergent goals and to decide when they believed they had achieved these goals’ (p. 2104). from our perspective as translational researchers, the fact that our student and teacher partners in this project simply did not see opportunities such as enactment and agency in the technology is instructive. it points to a strong need for a research understanding of innovation in context. the design and research problem: achieving expansive learning a significant point of tension evident in current educational policy and practice can be summarised as being between a defined outcomes approach to learning and an expansive approach to learning. this is not a new debate. in the 4th century bce, the philosopher isocrates argued that ‘pedagogy’ should develop obedience, conformity and controllability, while plato argued for ‘philosophy’ or education which was about searching for truth, value and the meaning of life (jones 2013). the isocratic approach calls for certainty and known outcomes, while the platonic approach requires uncertainty. today the isocratic approach, pedagogy, dominates practice in formal education institutions in most parts of the world. in these contexts, learning design is typically driven by the design parameter of supporting the acquisition of clearly defined and highly specified learning outcomes. in the world beyond formal education, however, there is a growing call for this to change due to a recognition that the skills required to approach the social, environmental and economic challenges of the age require something more (see, e.g., commonwealth of australia – department of the prime minister and cabinet 2015). the academic response to this has been to develop a focus on transverse or 21st century skills (centre for educational research and innovation 2008; jang 2016; wan husin et al. 2016) including collaboration, communication and life-long learning. as we have seen in our pilot study, however, teachers and students are not turning to such ideas when they engage in the possibilities of a potentially http://dx.doi.org/10.25304/rlt.v26.2160 s.n. leonard and r.n. fitzgerald 10 citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 (page number not for citation purpose) transformative technology. this suggests a design and research problem for the project ahead – how do we support teachers and students to engage in the possibilities of the technology in an expansive rather than ‘pedagogic’ way? it is beyond the scope of this article to elaborate a design and research methodology that could answer this question, but we would like to point to a theoretical direction that others might explore in similar work.that direction is the use of various theories on the social nature of knowledge creation. cultural historical activity theory (chat, engeström 1987; roth and lee 2007) brings focus to the roles, rules and tools of the learning environment. similarly, the theory of practice architectures (kemmis et al. 2014; lowrie, leonard, and fitzgerald 2018) calls for consideration of ways of saying, doing and interacting as they relate to both the individual and the social world. this theoretical ensemble offers a refinement to the research and design question – does a design brief calling for teachers and students to consider how a new technology can change and improve the roles, rules, tools and ways of saying, doing and interacting lead to a different response than a design brief that asks for how a new technology can be used for teaching and learning? references akçayır, m. & akçayır, g. (2017) ‘advantages and challenges associated with augmented reality for education: a systematic review of the literature’, educational research review, vol. 20, pp. 1–11. doi: 10.1016/j.edurev.2016.11.002 bacca, j., baldiris, s. & fabregat, r. (2014) ‘augmented reality trends in education: a systematic review of research and applications’, journal of educational technology and society, vol. 17, no. 4, pp. 133–149. carijó, f. h., de almeida, m. c. & kastrup, v. (2013) ‘on haptic and motor incorporation of tools and other objects’, phenomenology and the cognitive sciences, vol. 12, no. 4, pp. 685–701. doi:10.1007/s11097-012-9269-8 centre for educational research and innovation. (2008) 21st century learning: research, innovation and policy directions from recent oecd analyses. available at: http://www.oecd. org/site/educeri21st/40554299.pdf cheng, k. h. & tsai, c. c. (2013) ‘affordances of augmented reality in science learning: suggestions for future research’, journal of science education and technology, vol. 22, no. 4, pp. 449–462. doi:10.1007/s10956-012-9405-9 clark, a. (2011) ‘finding the mind: book symposium on supersizing the mind: embodiment, action, and cognitive extension’, philosophical studies, vol. 152, no. 3, pp. 447–461. doi:10.1007/s11098-010-9598-9 clark, d. b., tanner-smith, e. e. & killingsworth, s. s. (2016) ‘digital games, design, and learning: a systematic review and meta-analysis’, review of educational research, vol. 86, no. 1, pp. 79–122. doi:10.3102/0034654315582065 commonwealth of australia department of the prime minister and cabinet. (2015) national innovation and science agenda. available at: https://www.innovation.gov.au/system/files/ case-study/national%20innovation%20and%20science%20agenda%20-%20report.pdf dunleavy, m., dede, c. & mitchell, r. (2009) ‘affordances and limitations of immersive participatory augmented reality simulations for teaching and learning’, journal of science education and technology, vol. 18, no. 1, pp. 7–22. doi:10.1007/s10956-008-9119-1 engeström, y. (1987) learning by expanding: an activity-theoretical approach to developmental research, orienta-konsultit, helsinki. enyedy, n., et al., (2017) ‘social affordances of mixed reality learning environments: a case from the science through technology enhanced play project (step)’, 50th hawaii international conference on system sciences, hawaii. http://dx.doi.org/10.25304/rlt.v26.2160 10.1016/j.edurev http://www.oecd.org/site/educeri21st/40554299.pdf http://www.oecd.org/site/educeri21st/40554299.pdf https://www.innovation.gov.au/system/files/case-study/national https://www.innovation.gov.au/system/files/case-study/national 20report.pdf research in learning technology citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 11 (page number not for citation purpose) fishman, b. j., et al., (2011) ‘what happens when the research ends? factors related to the sustainability of a technology-infused mathematics curriculum’, journal of computers in mathematics and science teaching, vol. 30, no. 4, pp. 329–353. froese, t., et al., (2012) ‘the enactive torch: a new tool for the science of perception’,. ieee transactions on haptics, vol. 5, no. 4, pp. 365–375. doi:10.1109/toh.2011.57 gee, j. p. (2006) ‘why are video games good for learning’, nordic journal for digital literacy, vol. 3, pp. 172–183. goldin-meadow, s. & wagner, s. m. (2005) ‘how our hands help us learn’, trends in cognitive sciences, vol. 9, no. 5, pp. 234–241. doi:10.1016/j.tics.2005.03.006 hamilton, e. r., rosenberg, j. m. & akcaoglu, m. (2016) ‘the substitution augmentation modification redefinition (samr) model: a critical review and suggestions for its use’, techtrends, vol. 60, no. 5, pp. 433–441. doi:10.1007/s11528-016-0091-y hsiao, k. -f., chen, n. -s. & huang, s. -y. (2012) ‘learning while exercising for science education in augmented reality among adolescents’, interactive learning environments, vol. 20, no. 4, pp. 331–349. doi:10.1080/10494820.2010.486682 hugues, o., fuchs, p. & nannipieri, o. (2011) ‘new augmented reality taxonomy: technologies and features of augmented environment’, in handbook of augmented reality, ed b. furht, pp. 47–63. doi: 10.1007/978-1-4614-0064-6 2 jang, h. (2016) ‘identifying 21st century stem competencies using workplace data’, journal of science education and technology, vol. 25, no. 2, pp. 284–301. doi:10.1007/ s10956-015-9593-1 johnson-glenberg, m. c. & megowan-romanowicz, c. (2017) ‘embodied science and mixed reality: how gesture and motion capture affect physics education’, cognitive research: principles and implications, vol. 2, no. 1, pp. 1–28. doi:10.1186/s41235-017-0060-9 jones, b. (2013) ‘education as commodity?: how creativity fell off the agenda and labour market factors took over’, available at: http://math.haifa.ac.il/yair/the-funneled-web-archive-(2001-2013)rip/pdf_documents/the%20hon%20dr%20barry%20jones%20 -%20education%20as%20commodity-final-tfw.pdf kamarainen, a. m., et al., (2013) ‘ecomobile: integrating augmented reality and probeware with environmental education field trips’, computers & education, vol. 68 no. c, pp. 545–556. doi:10.1016/j.compedu.2013.02.018 kelly, a. e., baek, j. y. & lesh, r. a. (2008) handbook of design research methods in education : innovations in science, technology, engineering, and mathematics, routledge, new york. kemmis, s., et al., (2014) changing practices, changing education, springer, singapore. kiverstein, j. & clark, a. (2009) ‘introduction: mind embodied, embedded, enacted: one church or many?’, topoi, vol. 28, no. 1, pp. 1–7. doi:10.1007/s11245-008-9041-4 kolb, d. a. (1984) experiential learning: experience as the source of learning and development (1st ed.). prentice hall, inc, upper saddle river, nj. leonard, s. n., fitzgerald, r. n. & riordan, g. (2016) ‘using developmental evaluation as a design thinking tool for curriculum innovation in professional higher education’, higher education research & development, vol. 35, no. 2, pp. 309–321. doi:10.1080/07294360.2015.1087386 leonard, s. n. & roberts, p. (2014) ‘performers and postulates: the role of evolving socio historical contexts in shaping new teacher professional identities’, critical studies in education, vol. 55, no. 3, pp. doi:10.1080/17508487.2014.904808 logan, t., lowrie, t. & diezmann, c. (2014) ‘co-thought gestures: supporting students to successfully navigate map tasks’, educational studies in mathematics, vol. 87, no. 1, pp. 87–102. doi: 10.1007/s10649-014-9546-2 lowrie, t., leonard, s. n. & fitzgerald, r. n. (2018) ‘stem practices: a translational framework for large-scale stem education design’, eder. educational design research, vol. 2, no. 1. doi:10.15460/eder.2.1.1243 lubinski, d. (2010) ‘spatial ability and stem: a sleeping giant for talent identification and development’, personality and individual differences, vol. 49, no. 4, pp. 344–351. doi:10.1016/j.paid.2010.03.022 http://dx.doi.org/10.25304/rlt.v26.2160 10.1109/toh 10.1016/j.tics http://math.haifa.ac.il/yair/the 20commodity-final-tfw.pdf 10.1016/j.compedu 10.15460/eder 10.1016/j.paid s.n. leonard and r.n. fitzgerald 12 citation: research in learning technology 2018, 26: 2160 http://dx.doi.org/10.25304/rlt.v26.2160 (page number not for citation purpose) mayer, r. e. (2014) ‘cognitive theory of multimedia learning’, the cambridge handbook of multimedia learning, 2 ed., ed r. e. mayer, cambridge university press, cambridge, pp. 43–71. mckenney, s. & reeves, t. c. (2012) conducting educational design research, routledge, new york. melatti, m. & johnsen, k. (2017) ‘virtual reality mediated instruction and learning’, 2017 ieee virtual reality workshop on k-12 embodied learning through virtual & augmented reality (kelvar), los angeles, ca. merchant, z., et al., (2014) ‘effectiveness of virtual reality-based instruction on students’ learning outcomes in k-12 and higher education: a meta-analysis’, computers & education, vol. 7, pp. 29–40. doi: 10.1016/j.compedu.2013.07.033 milgram, p. & kishino, f. (1994) ‘a taxonomy of mixed reality visual displays’, ieice transactions on information and systems, vol. e77-d, pp. 1321–1329. doi:10.1.1.102.4646 newcombe, n. s. (2010) ‘picture this: increasing math and science learning by improving spatial thinking’, american educator, vol. 34, no. 2, pp. 29–35. doi: 10.1016/j.cobeha.2016.04.010 pan, m. k. & niemeyer, g. (2017) ‘catching a real ball in virtual reality’, 2017 ieee virtual reality workshop on k-12 embodied learning through virtual & augmented reality (kelvar), pp. 269–270, los angeles, ca. pouw, w. t. j. l., et al., (2014) ‘toward a more embedded/extended perspective on the cognitive function of gestures’, frontiers in psychology, vol. 5, no. 359, pp. 1–14. doi:10.3389/ fpsyg.2014.00359 puentedura, r. (2006) transformation, technology, and education. available at: http://hippasus.com/resources/tte/. radu, i. (2014) ‘augmented reality in education: a meta-review and cross-media analysis’, personal and ubiquitous computing, vol. 18, no. 6, pp. 1–11. doi: 10.1007/s00779-013-0747-y roth, w. m. & lee, y. -j. (2007) ‘“vygotsky’s neglected legacy”: cultural-historical activity theory’, vol. 77, no. 2, pp. 186–232. doi:10.3102/0034654306298273 santos, m. e. c., et al., (2014) ‘augmented reality learning experiences: survey of prototype  design and evaluation’, ieee transactions on learning technologies, vol. 7, no. 1, pp. 38–56. doi:10.1109/tlt.2013.37 stieff, m. & uttal, d. (2015) ‘how much can spatial training improve stem achievement?’, educational psychology review, vol. 27, no. 4, pp. 607–615. doi:10.1007/s10648-015-9304-8 sung, y. t., yang, j. m. & lee, h. y. (2017) ‘the effects of mobile-computer-supported collaborative learning: meta-analysis and critical synthesis’, review of educational research, vol. 87, no. 4, pp. 768–805. doi: 10.3102/0034654317704307 waddington, d. i. (2015) ‘dewey and video games: from education through occupations to educations through simulations’, educational theory, vol. 65, no. 1, pp. 1–20. wan husin, w. n. f., et al., (2016) ‘fostering students' 21st century skills through project oriented problem based learning (popbl) in integrated stem education program’, asia-pacific forum on science learning & teaching, vol. 17, no. 1, pp. 60–77. zarraonandia, t., et al., (2013) ‘an augmented lecture feedback system to support learner and  teacher communication’, british journal of educational technology, vol. 44, no. 4, pp. 616–628. doi:10.1111/bjet.12047 http://dx.doi.org/10.25304/rlt.v26.2160 10.1016/j.compedu 10.1016/j.cobeha 10.3389/fpsyg 10.3389/fpsyg http://hippasus.com/resources/tte http://hippasus.com/resources/tte 10.1109/tlt 10.1111/bjet (page number not for citation purpose) 1 *corresponding author. email: kcameron@unitec.ac.nz research in learning technology 2019. © 2019 k.e. cameron and l.a. bizo. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 research in learning technology vol. 27, 2019 original research article use of the game-based learning platform kahoot! to facilitate learner engagement in animal science students kristie e. camerona* and lewis a. bizob aschool of environmental and animal sciences, unitec institute of technology, new zealand; bschool of psychology, university of new england, armidale, australia (received: 31 january 2019; revised: 12 april 2019; accepted: 29 april 2019; published: 15 may 2019) gamification of instructional activities is a useful approach that educators can use to promote more effective learning environments by increasing problemsolving, critical thinking and competence in the classroom. ‘kahoot!’ is an online multi-player real-time quiz game that allows students to measure learning in an engaging, immediate and entertaining manner. lecturers can measure how well students absorb information and tailor their teaching to the next step or re-teach a concept after poor uptake by students. seventy-two students participated in a 20-question survey about their experiences with ‘kahoot!’. engagement scores were correlated with assessment grades to measure if ‘kahoot!’ affected student learning and achievement. the survey was deemed statistically sound in reliability and validity testing, and a principal components analysis (pca) revealed that the attributes were strongly linked. there was no relationship between engagement score and assessment grade, indicating that ‘kahoot’!’ did not directly increase achievement. however, assessment of individual responses identified that students found it to be a positive social learning technology as it provided a fun, competitive and immersive end to a class. the benefits of fostering engagement, enjoyment and immersion within adult learning are especially important for maintaining a level of achievement within education to ensure that students are better equipped to deal with challenges and can turn a potential failure into an opportunity to improve their scholarship. the challenge provided by this study is to identify now how to measure the value of ‘fun’ activities in the tertiary classroom as a reinforcer for engagement, participation and learning. keywords: ‘kahoot!’; gamification; online-learning tool; engagement; tertiary; learning; enjoyment introduction educational gamification is a technology-based method for increasing student engagement and motivation for learning (simoes, redondo, and vilas 2003; whitton 2007, 2011). in an educational setting, gamification has primarily been used to promote reflective learning in formative assessment tasks. gamification has been shown to promote effective learning environments by increasing problem-solving (whitton 2011), as well as improving critical thinking and competence in the classroom (bicen and kocakoyun 2017; dellos 2015; karaaslan and budak 2012). in addition, it allows for reinvention and constant advances in educational technology (moreno-ger, burgos, and torrente 2009). the present study sought to assess student opinions and is the mailto:kcameron@unitec.ac.nz http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2225 k.e. cameron and l.a. bizo 2 citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 (page number not for citation purpose) first to attempt the identification of a direct relationship between the use of a quizbased game called ‘kahoot!’ and student’s final grades. educational gamification tasks can require students to compete with each other while completing tasks related to course content. ‘kahoot!’ is a popular online multiplayer real-time quiz package that allows students and lecturers to measure learning in an engaging, immediate, entertaining and anonymous manner (bicen and kocakoyun 2017; cutri et al. 2016; ‘kahoot!’, 2015). it has been claimed that the application allows for an interaction between intrinsic motivation and external motivation; the built-in, naturally satisfying motivation to play a game to obtain an external reward such as kudos or praise from others (özer and bicen 2017). in comparison with a paper quiz and a physical ‘clicker’ arrangement, the use of ‘kahoot!’ was rated as being more engaging, exciting and competitive, with a slightly higher overall mean score within a cohort of firstyear university information and technology (it) students (wang, zhu, and saetre 2016). students reported ‘satisfying’ engagement across all types of gamification. they also preferred to use the platform utilising their own devices. in a similar study of 131 students enrolled in a pre-teaching course, 39% identified ‘kahoot!’ as the most preferred online platform for learning compared to other online options such as ‘classcraft©’ (an adventure achievement-type game), ‘classdojo©’ (an animated communication and game application) and ‘socrative©’ (a general quiz platform). the built-in ‘kahoot!’ rating system allows students to ‘rate’ their experience of the quiz in a simple way. cutri et al. (2016) measured the responses of 191 undergraduate chemistry students as 4.59/5 for ‘fun’, with the vast majority having a positive feeling (77%) and reported learning something (87%) while using the application. also, 92% of the students would recommend the application. students across three science cohorts reported the immediacy of feedback, interaction, fun and enjoyable environment as benefits of using ‘kahoot!’ but identified that internet access is an issue for participation. students preferred the quizzes being at the beginning or middle of the class and the reasons they gave were because of fatigue and lack of time to process the information presented in the class. when a ‘kahoot!’ quiz is presented at the end of a lecture, students have the opportunity to gauge how well they have learnt new concepts and applied existing content and can assess and progress through their failures anonymously (icard 2014). in addition, to promote engagement, there is a measure of competition between students as they can compare their answers with others (cutri et al. 2016), and assess their progress against their previous performances (icard 2014). retention and consistent engagement of students across a 12-week short course increased with consistent use of ‘kahoot!’ as a formative assessment tool (fotaris et al. 2016). for a comprehensive guide on the use of ‘kahoot!’ see ren and wagner (2016). these gamified platforms offer some other useful features to educators. they allow lecturers to measure how well students (as a group) learn and apply information and can also be used to reinforce correct answers by the provision of points and verbal praise whereby they can move to the next learning step (cutri et al. 2016). the platforms also offer detailed information for a quiz about how quickly and accurately a question was answered. some quiz platforms offer detailed information about how quickly and accurately a question was answered by students. this information identifies material that the students have not mastered, and this affords lecturers with an opportunity to either correct at the time, reteach and direct students to look at the topics again during independent study or modify the delivery of information in future classes (cutri et al. 2016; dellos 2015). http://dx.doi.org/10.25304/rlt.v27.2225 research in learning technology citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 3 (page number not for citation purpose) recent studies have measured the use of ‘kahoot!’ in the classroom as a learning and engagement tool using a survey design centred on five aspects of engagement: challenge, control, immersion, interest and purpose of the game (ismail and mohammad 2017; whitton 2011). table 1 defines these factors (ismail and mohammad 2017; whitton 2007, 2011) that are expanded from the elements of flow theory (whitton 2011). researchers have discussed the importance of each of these elements (e.g. malone 1980), which has resulted in how students engage with an activity. it is not the event, but the subjective measure of the interaction by the individual with the activity. therefore, it is necessary to conduct a survey to measure engagement from an individual’s point of view such as with the likert-style 42-question survey iteratively developed and tested as statistically reliable and valid by whitton (2007). whitton’s survey was piloted by 33 adult participants playing online non-educational games and participating in a survey and face-to-face interviews about their experience for 12 participants. the following is a description of each element and how it ‘fits’ in the measurement of engagement using gamification. challenge elements of ‘challenge’ relate to the achievement of goals that have uncertain outcomes, such as the multiple-option endings of traditional gaming (malone table 1. learning engagement factor, with description (ismail and mohammad 2017; whitton 2011). factor description example of survey question see note below challenge the motivation to undertake the activity; clarity as to what it involves a perception that the task is achievable i wanted to complete the activity i wanted to explore all the options available to me the goal of the activity was not clear* i knew what i had to do to complete the activity i felt that i could achieve the goal of the activity i found the activity frustrating control the fairness of the activity, the level of choice over types of action available in the environment, and the speed and transparency of feedback the types of task were too limited* i could not tell what effect my actions had immersion the extent to which the individual is absorbed in the activity i felt excited during the activity i felt absorbed in the activity interest the intrinsic interest of the individual in the activity or subject matter i found the activity boring* the activity was aesthetically pleasing purpose the perceived value of the activity for learning, whether it is seen as being worthwhile in the context of study. the activity was pointless* i was given feedback at appropriate time the activity was worthwhile note: example questions from whitton (2007, p. 153) of which correlated in her pca analysis (note the negative phrasing marked with asterisks) covering each of the elements. http://dx.doi.org/10.25304/rlt.v27.2225 k.e. cameron and l.a. bizo 4 citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 (page number not for citation purpose) 1980). the challenge of ‘kahoot!’ is to answer questions with multiple opposing answers while also competing against other students to gain the top score (wang, zhu, and saetre 2016) by answering correctly and quickly (fotaris et al. 2016). to ensure the ‘challenge’ of the game is motivating the students, the task or requirement must be perceived as achievable, requiring careful administration from the lecturer. control students are encouraged to take ownership and control of their learning by choosing to answer questions in quizzes and play games as part of participation in the class (whitton 2007). students should be able to reflect on their learning progress as a factor of what they have learnt (and how they learned it) to further their independent learning (cochran and brown 2016; whitton 2007). ‘kahoot!’ quizzes incorporate ‘contingency’, ‘choice’ and ‘power’ (malone and lepper 1987) and provide a logical process where a question is presented, then related, relevant and intuitive options are shown which the student can select. a ‘positive’ affirmation is shown on their device’s screen if answered correctly, or a ‘commiserative’ screen if the answer was incorrect – but the latter is identified as a learning opportunity (dempsey et al. 2002). immersion whitton (2011) suggests that the level of concentration in the activity results in the gamer losing track of time and awareness. the fact that this could be possible using an educational game where students focus 100% on a ‘kahoot!’ quiz is a little unbelievable, but koster (2005) suggests that the act of solving a puzzle or playing a game is inherently fun and retains attention. it is also worth considering that students of today, especially recent school leavers, have been raised with computers in nearly every household (and pocket), and therefore the immersion in, and reliance on, technology has changed the way students learn and process information (byrne davis et al. 2015; prensky 2001). prensky (2001) stated that students of the gaming generation are used to fast feedback, respond to immediate gains and adapt faster to new forms of learning. interest according to whitton (2011; p. 604), interest is defined as the ‘intrinsic fascination or enthrallment’ with a task. malone (1980) also mentions ‘curiosity’, such as a sensory light or sound input or cognitive puzzle that retains our attention to learn and comprehend fully using feedback. in addition, to occur within andragogical learning, the student would find interest in the subject, and the playing of the game, as long as participation was relevant to the learning outcomes (whitton 2011). purpose adult learners reported that being able to see improvement during play and perceiving being good at playing the game were important in any online game (whitton 2011). they also reported that there was an initial barrier to participating in online games http://dx.doi.org/10.25304/rlt.v27.2225 research in learning technology citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 5 (page number not for citation purpose) due to difficulty in using the platform or not seeing value in the game goals, getting stuck in the game, not trusting the fairness of the game or a genuine lack of interest in games. combating these issues would be considerably easier in ‘kahoot!’ as a quiz platform because the purpose and learning outcomes in ‘practising’ recently taught content is quite clear compared to the less demanding activities used by whitton (2007, 2011). the perception of gamification in 113 entry-level medical students as a formative assessment reduced the whitton (2007) survey to 12 questions (ismail and mohammad 2017). the students responded positively to the questions relating to the use of ‘kahoot!’ as a challenge, immersive, interesting and providing purpose (table 1). the students, however, identified that elements of control were lacking where  ‘kahoot!’ failed to ‘simplify complex subjects’. the limitations of  this platform for teaching and reinforcing complicated course content were made apparent. one can see intuitively how this would be the case for many topics related to medicine for a platform with the capacity of a 95-character question and 60-character answers for each of four possible answers – only simple concepts would be adequately tested. also, ismail and mohammad (2017) did not conduct a reliability or validity test on their shortened survey and instead used a four-point likert scale, leading this author to speculate that the questions were perhaps too vague or oversimplified to encompass the degree of engagement they were trying to measure. this study aimed to assess student opinions of ‘kahoot!’ in the tertiary sector. it is the first study to attempt to identify a direct relationship between the use of ‘kahoot!’ and the student’s final grades. the cohort of students were animal science students undertaking an ‘introduction to animal husbandry’ class. animal science students were presented with regular ‘kahoot!’ quizzes at the end of each class as a method of solidifying the content and to provide feedback to the student and lecturer throughout semester 1, 2018. it was anticipated that students would react positively to the use of ‘kahoot!’ given the enthusiastic reports by others (e.g. cutri et al. 2016; fotaris et al. 2015, 2016; icard 2014; ismail and mohammad 2017; wang, zhu, and saetre 2016). the present research also sought to extend our understanding of the impact of gamified activities in a tertiary setting by examining the association between student engagement score and assessment and final grades within the class. method participants a total of 121 students enrolled in the level 5 principles of animal husbandry course at unitec institute of technology, new zealand, in semester 1, 2018 participated in this study. they belonged to one of four cohorts: level 4 certificate in companion animal management – companion animal stream, level 4 certificate in companion animal management canine behaviour and training stream, level 5 new zealand certificate in animal technologies (veterinary nursing), and level 5 bachelor of applied science students at the beginning of the semester. over the semester, there were 11 classes when a ‘kahoot!’ quiz was implemented. the remaining classes were practical sessions. on average, 62 students participated in each of the quizzes across the semester, gaining an average of 73.56% correct score. http://dx.doi.org/10.25304/rlt.v27.2225 k.e. cameron and l.a. bizo 6 citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 (page number not for citation purpose) the research was approved by our institution’s human ethics committee protocol 2018-1016. measures the participants were asked to complete a 20-question survey on their experience of using ‘kahoot!’ in the lectures according to a 7-point likert scale, where 1 = strongly disagree, 2 = disagree, 3 = slightly disagree, 4 = neutral, 5 = slightly agree, 6 = agree and 7 = strongly agree. the questions related to attributes of engagement: challenge, control, immersion, interest and purpose (table 1). the following is a list of questions that represent each engagement attribute (see appendix). eight items were written from a ‘negative’ perspective, requiring reverse coding for analysis. procedure the survey was conducted at the beginning of the final scheduled teaching period and required approximately 15 min to complete. a powerpoint slide detailing the purpose of the survey and how to answer the questions was presented to the students before data collection. the slide also indicated that if a student elected not to participate, that they fold their survey and pass to then end of the row with the rest of the surveys which were collected by the instructor. data analysis reliability and validity tests and a pca test using the ibm statistics package spss® were applied to the survey. an engagement score, reflecting how the student ‘felt’ about the task, was correlated with that of student’s assessment scores. these consisted of assessments including a written test referring to a practical health check on an animal worth 20%, two online tests worth 30% covering health, breeding and reproduction, the final exam worth 40% covering all topics and also the overall final grade of the paper covering 100% of the course. results a total of 78 students participated in the engagement survey, with 72 usable samples across the four cohorts (n l4 companion animal management = 9, n l4 canine behaviour and training = 14, n l5 animal technologies (veterinary nursing) = 38 and n l5 bachelor of applied science = 14). reliability and validity of survey the reliability and validity of the survey were assessed. the survey had good reliability, cronbach’s alpha was α = 0.794 with all questions of the survey considered, and this increased to α = 0.848 when items q2, q3, q8 and q9 were removed as per the pca analysis. http://dx.doi.org/10.25304/rlt.v27.2225 research in learning technology citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 7 (page number not for citation purpose) as a measure of construct validity, each item was correlated with engagement score to measure how well each item predicts the final score using the pearson product–moment correlation. the critical r-value for a sample of 72 is 0.231. of the survey items, 18 questions produced a correlation coefficient of greater than the critical value, indicating that they predict engagement score and had significant values of less  than 0.05. therefore, these items were valid. two items were not valid as they had  coefficient values less than the critical correlation value: q2 [r sp (71) = −0.054, p = 0.650] and q9 [r sp (71) = 0.057, p = 0.637]. factor analysis a pca test was conducted on the data to assess the adequacy of the survey to measure the perception of the effect of ‘kahoot!’ on learning in 72 tertiary students. the analysis revealed that 16 variables had at least one correlation coefficient greater than 0.3, indicating that these items were measuring a similar construct to another item; therefore, the items with correlations less than 0.03 were removed. the kaiser–meyer–olkin (kmo) score was 0.845, and individual kmo scores were all greater than 0.7, indicating ‘middling’ to ‘meritorious’ sampling adequacy according to kaiser (1974). bartlett’s test of sphericity was statistically significant [χ2 (120) = 424.31, p < 0.001], indicating that the data were likely factorisable into smaller components. the pca revealed four engagement attributes that had eigenvalues greater than 1, which explained 34.7%, 11.4%, 9.2% and 7.7% of the total variance, respectively. a fifth eigenvalue value was 0.822, accounting for another 5.7% of the variance and aligned with the visual inspection of the scree plot, indicating that the five components should be retained. the five-component solution using a varimax oblimin rotation (as the items were correlated) provided factor interpretability with single loadings for five items (q5, q10, q1, q14 and q16) and dual loadings for 11 items (q1, q4, q6, q7, q11, q12, q15, q17, q18, q19 and q20) when the criteria excluded coefficient values set to <0.4. a summary of the original loadings and pca-generated loadings of questions to the engagement factor is presented in table 2. considering the comparison of intended and pca loadings, the engagement factor being assessed by a particular table 2. the questions associated with the original question loadings, single pca loadings and dual pca loadings for each engagement factor are shown in separate columns. factor original question loadings single principal components analysis (pca) identified loadings dual pca loadings challenge q1, q4, q5, q6, q7 q5, q10, q13, q14, q16 q1, q4, q6, q7, q11, q12, q17, q19, q20 control q8, q9, q10 q18 immersion q11, q12, q13 q7 q11, q12, q15, q20 interest q14, q15, q16 q15, q17, q19 purpose q17, q18, q19, q20 q1, q4, q6 note: questions that loaded onto the intended attribute are shown in bold. http://dx.doi.org/10.25304/rlt.v27.2225 k.e. cameron and l.a. bizo 8 citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 (page number not for citation purpose) question could be improved; eight of the questions were loaded onto their intended engagement factor, even if these were dual loadings. there were no single loadings for components of ‘control’ or ‘interest’ or ‘purpose’, suggesting that these components have direct links with other components or in how each component is interpreted by students. survey results one-way analysis of variance confirmed the similarity across cohorts with no significant differences between mean scores across cohorts [f ’s (3, 68) = 0.10 ≤ χ ≤ 2.53, all p’s > 0.05, η2 = 0.05 ≤ χ ≤ 0.10]. the median selections for each question are shown in figure 1. generally, students selected either ‘agree’ (ordinal value 6) or ‘strongly agree’ (ordinal value 7) for the questions framed in a positive way (q1, q2, q4, q5, q7, q10, q11, q12, q14, q16, q17 and q20). students selected ‘slightly disagree’ (ordinal value 3), ‘disagree’ (ordinal value 2) and ‘strongly disagree’ (ordinal value 1) for questions that are framed in a negative way (q3, q6, q8, q13, q15, q18 and q10) and ‘neutral’ for a single question, q9. the interquartile ranges were wider for the negatively framed questions, extending up to two ordinal values above and below the median, whereas for the positively framed questions the interquartile range was typically one ordinal value below the median. the difference between positively and negatively framed questions is likely due to the large proportion of students selecting ‘strongly agree’ for positively framed questions q1, q7, q10 and q14. although, students appeared to perceive ‘kahoot!’ as a positive experience in the classroom, with most students selecting positive answers for individual items, there were no significant correlations between assessment grades and engagement score when q2, q3, q8 and q9 were removed as per the pca (r p ’s = 0.094 ≤ χ ≤ 0.134, all p’s > 0.05). the absence of a relationship between assessment grades and engagement score indicates that the survey as a whole did not provide a measure of engagement; however, the single question items do provide invaluable insights into positive student perception of the gamification tool. the comments given by ques�on and engagement a�ribute 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 m ed ia n o rd in al s co re 0 1 2 3 4 5 6 7 challenge control immersion interest purpose figure 1. median ordinal score (1–7) for each question related to each engagement attribute for the kahoot! survey across cohorts. the error bars are the interquartile ranges. http://dx.doi.org/10.25304/rlt.v27.2225 research in learning technology citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 9 (page number not for citation purpose) the students about the use of ‘kahoot!’ in their class confirm that assessment. out of the 29 responses, 26 were positive, and the quizzes were considered fun and a highlight of the class. students also indicated that it helped solidify learning, pay attention, and it functioned as a reward or to break up the class and compete with their friends. two comments were negative, with one suggesting that they found the format stressful, and one comment was neutral, suggesting questions should be available after the quiz (table 3). table 3. each comment provided by a student is given. comment all cool, fun, only bummer is don't have a smart phone better for facts than concepts; often too fast, speed of answering affects which is stressful do it more often great way to learn! i don’t like kahoot :( i enjoyed the opportunity to beat my friends in kahoot! and test my knowledge and understanding i find kahoot really fun, it allows me to acknowledge what i’ve learnt and focus on what i got wrong i love it :) i love the kahoot quizzes. it’s fun and engaging and i like to test what i have learnt i really enjoy it :) if i know there is kahoot i pay more attention in class as i want to win – it adds fun to learning – other classes should do this, anatomy would be great for kahoot :) it definitely made me pay more attention and i quite like the ones that were almost a trick question – made me pause and think harder! it is a fun way to get students engaged in the class it is nice to break up sometimes monotonous lectures with something fun like kahoot :) it is something that i look forward to in every class because it is a fun way to absorb what has been taught it was a fun, engaging way of learning even if i didn’t know the answer i still learnt the gaps in my knowledge it was fun and thanks for the crunchie bar :) it’s a great way to test knowledge and we then know what we need to improve on. it’s fun while it tests our knowledge – i love it :) it’s fun! it was used last year in biology as well and it helps to retain information it’s one of the highlights of the class. it feels like a small reward for sitting through a 2 h lecture kahoot is fun and helps left brain people with learning kahoot is life kahoot! is a great way to engage in learning and very useful for myself as it helps with my dyslexic brain. *leave answers in the app for later reference if applicable love it! love it! love kahoot love kahoot!!! loved it note: those comments classified as negative are presented in italics, and an asterisk accompanies those that are classified as neutral. http://dx.doi.org/10.25304/rlt.v27.2225 k.e. cameron and l.a. bizo 10 citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 (page number not for citation purpose) discussion the purpose of this study was to assess the engagement of students to the quizbased platform game ‘kahoot!’ as a method of formative self-assessment and to enhance the learning experience through gamification. in terms of the former, there was no correlation between the summative assessment grades of the animal science course and the engagement score generated by the 20-question survey; therefore; it cannot be concluded that the use of ‘kahoot!’ will result in higher grades for students. considering the former, however, it is clear from the responses to the individual questions in the survey and the comments given by the students that they, like many of their peers, found the use of ‘kahoot!’ to be fun, engaging and a motivating tool that helped maintain their attention during class (q12, figure 1) while also having a positive effect on their understanding of the material (q7 and q8, figure 1). to finish the class with a fun, entertaining and relevant activity which encompassed a bit of selfand class competition (q11 and q14, figure 1) ended the class on a high note that gave the students a goal for their independent work (q17 and q20, figure 1). future investigation into the use of ‘kahoot!’ in the tertiary classroom could look at the impact of game-play on course retention, use of other educational resources, or if learning is improved in a group of students using ‘kahoot!’ compared to those without access to gameplay. of particular interest is whether the timing of the quiz, at the beginning or middle (e.g. cutri et al. 2016), or the end of class as utilized in this study provides greater engagement or mastery of the course content. whitton (2011) made the point that adult learners need to be inherently interested in the topic and also should understand the value in the use of gamification in their learning. it seems that at least for the students in the current study, their engagement with the material via ‘kahoot!’ was enough to reinforce attention, provide an aspect of fun and healthy competition linked to the information and identify areas for further examination, if it did not necessarily make a measurable advance on their final grade. reinforcing ‘enjoyment’ in learning is not a factor considered in the literature specifically as it is expected to be inherent; however, considering the identification of learned helplessness (e.g. davis and mcpartland 2012; fotaris et al. 2015; mark 1983) in students, tools to maintain and increase positive feelings about learning are the missing pieces of adult education. the survey was considered to be reliable with a cronbach’s alpha value of 0.848 when items q2, q3, q8 and q9 were removed as suggested by the pca analysis. the pca revealed the adequacy of the modified survey to measure the perception of the effect of ‘kahoot!’ on learning. with both these measures and the pca, 16 questions would remain for future use to measure engagement using a gamification tool such as ‘kahoot!’. in the future, the test–retest method of confirming reliability and validity would be used to measure the engagement of ‘kahoot!’ over time and across different courses. gamification would likely prove to be useful for courses where attendance and retention are low. traditionally, surveys are designed so that certain questions load onto one specific dimension of the survey. for example, in the present study, q1–q7 were designed to measure the ‘challenge’ attribute, q8–q10 the ‘control’ attribute, q11–q13 the ‘immersion’ attribute, q14–q16 the ‘interest’ attribute and q17–q20 the ‘purpose’ attribute (tables 1 and 3). the results of the factor analysis, however, indicated that only questions of the ‘challenge’ attribute (q1, q5, q4, q6 and q7) and the http://dx.doi.org/10.25304/rlt.v27.2225 research in learning technology citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 11 (page number not for citation purpose) ‘immersion’ attribute (q7) loaded reliably onto this attribute and all but one loaded onto two attributes. it seems that the components of engagement associated with ‘kahoot!’ are strongly linked indicated by the dual loadings (table 3). it is evident that the students perceive the use of gameplay to be ‘challenging’ as the primary engagement tool as most of the questions loaded onto this attribute. the attributes of ‘immersion’, ‘interest’ and ‘purpose’ appear to be less important. the attribute of ‘control’ appears to be the least important to the students probably due to the overall purpose of using the tool from the lecturers’ perspective. implementation of gamified learning needs to consider a limitation raised elsewhere (ismail and muhammad 2017) that while these activities may promote engagement and reinforce learning of basic concepts, more thought needs to be given to their use if the intention is to encourage the application of that knowledge. in contrast, the implementation of gameplay in the context of this study was a short, formative activity provided at the end of the lecture to reinforce attention in class and end on a ‘high note’ rather than a formative assessment used in the middle of the class (e.g. cutri et al. 2016). therefore, it is not surprising that students do not consider the features of ‘control’ (table 1), based on fairness, level of choice or the speed and transparency of feedback within the gameplay, as important. the features of the more usual constructive-formative testing may have been overwhelmed by the ‘game-like’ social element of shared achievement or commiseration with classmates over correct and incorrect answers, or perhaps by the use of bright colours and music, that is ‘kahoot!’, compared to a summative test preparation activity completed independently of class time that is devoid of attention-grabbing visual effect. quiz platforms such as ‘kahoot!’ would not function as an engagement and learning tool without merging of some attributes, particularly that of ‘interest’ and ‘immersion’, and possibly that of ‘challenge’ and ‘control’ or ‘purpose’ and ‘interest’. one could argue that it would be difficult to be immersed in an activity without being interested in it – especially within a lecture environment that provides no other stimuli of which to attend (except ignoring the class or leaving) to a visually and auditorily stimulating game. the multi-attentional composite that is a game, even one that is quiz-based such as ‘kahoot!’, predicts multiple loadings on dimensions. while the relationship between assessment grade and engagement score was not conclusive, it was clear that students responded well to gameplay. in the future, the clarity of the questions in the current survey could be improved to ensure that dual loadings are not spurious and are in fact a real effect. therefore, in the next iteration of the quiz, students would be asked to attend to a single construct within each question rather than the compounds identified in the current quiz. for example, in q16 it was asked whether the students found kahoot! to be visually appealing and exciting. double-barrelled questions would be more effective if they were split into two questions to harness student perception of single outcomes. it was apparent that the animal science students in the present study were engaged with the game, content and identification of gaps in their learning. the engagement of the students with the game, however, did not automatically translate to an improvement in their final grade as evidenced by the poor correlations between assessment and engagement score. it might be that the 75-odd students who most likely attended and participated in most classes would have achieved the same grade regardless of gameplay. hakulinen and auvinen (2014) found that high-achieving students were not motivated by the reinforcement of ‘badges’, earned by achieving a particular goal http://dx.doi.org/10.25304/rlt.v27.2225 k.e. cameron and l.a. bizo 12 citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 (page number not for citation purpose) as reinforcer, whereas students who responded positively to earning a badge were also high achievers and intrinsically and extrinsically motivated by gameplay. in conclusion, to engage students in their learning by providing a fun, competitive and reinforcing end to a class using gameplay is a positive social learning strategy. implementation of the tool is simple, aesthetically pleasing and inclusive as well as being anonymous for selfand class assessment of student learning. to demonstrate that gamification of the classroom improves student learning outcomes would require more controlled research designs where separate cohorts of students are compared learning the same course content but with and without the gamified element and is a research objective for the future. while the present study has not provided evidence that this tool is associated with higher grades in assessments that should not, however, deter lecturers considering incorporating ‘kahoot!’ quizzes or other device-enabled games for tertiary students in their classes, the benefits of fostering engagement, enjoyment and immersion within adult learning are all essential for encouraging and reinforcing independent scholarship and are generally well received by students. references bicen, h. & kocakoyun, s. (2017) ‘educational technology: current issues’, world journal on educational technology, vol. 9, no. 1, pp. 18–23. retrieved from http://dergipark.org.tr/ itej/issue/39211/461500 byrne-davis, l., et al., (2015) ‘just-in-time research: a call to arms for research into mobile technologies in higher education’, research in learning technology, vol. 23, pp. 1–10. doi: 10.1136/bmj.326.7384.328 cochran, c. & brown, s. (2016) ‘andragogy and the adult learner’. in flores, k.a., kirstein, k. d., schieber, c. e., & olswang, s. g. (eds). supporting the success of adult and online students. washington, usa: createspace independent publishing platform, pp. 73–84 cutri, r. p. e., et al., (2016) ‘kahoot, a new and cheap way to get classroom-response instead of using clickers’. in asee’s 123rd annual conference & exposition, new orleans, la. #14649. june 26-29, 2016. retrieved from https://www.asee.org/public/conferences/64/ papers/14649/view davis, m. h. & mcpartland, j. m. (2012) ‘high school reform and student engagement’, in christenson, s. l., reschly, a. l., and wylie, c. (eds) handbook of research on student engagement. new york: springer, pp. 515–539. dellos, r. (2015) ‘‘kahoot!’ a digital game resource for learning’, international journal of instructional technology and distance learning, vol. 12 no. 4, pp. 49–52. retrieved from http://www.itdl.org/journal/apr_15/apr15.pdf dempsey, j. v., haynes, l. l., lucassen, b. a. and casey, m. s. (2002) ‘forty simple computer games and what they could mean to educators’, simulation & gaming: an interdisciplinary journal, vol. 33, pp. 157–168. retrieved from www.savie.ca/sage/articles/100013_013dempsey-2002.pdf fotaris, p., mastoras, t., leinfellner, r. and yasmine, r. (2015) from hiscore to high marks: empirical study of teaching programming through gamification. in proceedings of the 9th european conference on games based learning ecgbl 2015, steinkjer, norway, 8-9 october 2015. available from: http://eprints.uwe.ac.uk/28599 hakulinen, l. & auvinen, t. (2014) ‘the effect of gamification on students with different achievement goal orientations’. in the conference on learning and teaching in computing and engineering (latice) proceedings, kuching, malaysia in april 2014. doi:  10.1109/ latice.2014.10 http://dx.doi.org/10.25304/rlt.v27.2225 http://dergipark.org.tr/itej/issue/39211/461500� http://dergipark.org.tr/itej/issue/39211/461500� https://www.asee.org/public/conferences/64/papers/14649/view� https://www.asee.org/public/conferences/64/papers/14649/view� http://www.itdl.org/journal/apr_15/apr15.pdf� www.savie.ca/sage/articles/100013_013-dempsey-2002.pdf� www.savie.ca/sage/articles/100013_013-dempsey-2002.pdf� http://eprints.uwe.ac.uk/28599� research in learning technology citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 13 (page number not for citation purpose) icard, b. (2014) ‘educational technology best practices’, international journal of instructional technology & distance learning, vol. 11, no. 3, pp. 37–42. retrieved from http://www.itdl. org/journal/mar_14/mar14.pdf ismail, m. a.-a. & mohammad, j. a.-m. (2017) ‘kahoot: a promising tool for formative assessment in medical education’, education in medicine journal, vol. 9, no. 2, pp. 19–26. doi: 10.21315/eimj2017.9.2.2 kahoot!’ (2015) ‘kahoot!’ is one of the world’s fastest growing learning brands. accessed at https://support.kahoot.com/hc/en-us/categories/115000091647-about-kahoot kaiser, h. f. (1974). an index of factorial simplicity. psychometrika, 39, 31–36. doi: 10.1007/ bf02291575 karaaslan, i. a. & budak, l. (2012) ‘research on the use of mobile phone features by university students and its impact on their communication practices in everyday life.’ journal of yasar university, vol. 26, pp. 4548–4525. koster, r. (2005) a theory of fun for game design. scottsdale, az: paragylph press. malone, t.w. (1980) ‘what makes things fun to learn? a study of intrinsically motivating computer games’, technical report, xerox palo alto research center, palo alto, california. malone, t. & lepper (1987). making learning fun: a taxonomy of intrinsic motivations for learning. in snow, r. & farr, m. j. (eds), aptitude, learning, and instruction volume 3: conative and affective process analyses. hillsdale, nj. (pp 223-253). mark, s. f. (1983) ‘to succeed or not to succeed: a critical review of issues in learned helplessness.’, contemporary educational psychology, 8, pp. 1–19. doi: 16/0361-476x(83)90030-9 moreno-ger, p., burgos, d. & torrente, j. (2009) ‘digital games in elearning environments: current uses and emerging trends.’, simulation & gaming: an interdisciplinary journal, vol. 40, pp. 669–687. doi: 10.1177/1046878109340294 özer, h. h. & bicen, h. (2017) ‘the effect of gamified learning environment on student success’, international journal of scientific study, vol. 5, no. 8, pp. 108–113. doi: 10.1108/ ijilt-02-2017-0009 prensky, m. (2001) digital game-based learning. new york: mcgraw hill. ren, c. & wagner, j. (2016) ‘‘kahoot!’, the electronic journal for english as a second language august, vol. 20, no. 2, pp. 1–10. retrieved from http://www.tesl-ej.org/pdf/ej78/ m1.pdf simoes, j., redondo, r. d. & vilas, a. f. (2003) ‘a social gamification framework for a k-6 learning platform.’, computers in human behavior, vol. 2, pp 345–353. doi: 10.1016/j. chb.2012.06.007 wang, a. i., zhu, m. & saetre, r. (2016) ‘the effect of digitizing and gamifying quizzing in classrooms’, in proceedings of the 10th european conference on games based learning, paisley, uk, pp. 729–737. available at: https://brage.bibsys.no/xmlui/handle/11250/2426374 whitton, n. (2007) an investigation into the potential of collaborative computer game-based learning in higher education. napier university, edinburgh, scotland. whitton, n. (2011) ‘game engagement theory and adult learning’, simulation & gaming: an interdisciplinary journal, vol. 42, no. 5, pp. 596–609. doi: 10.1177/1046878110378587 http://dx.doi.org/10.25304/rlt.v27.2225 https://support.kahoot.com/hc/en-us/categories/115000091647-about-kahoot� http://www.tesl-ej.org/pdf/ej78/m1.pdf� http://www.tesl-ej.org/pdf/ej78/m1.pdf� https://brage.bibsys.no/xmlui/handle/11250/2426374� k.e. cameron and l.a. bizo 14 citation: research in learning technology 2019, 27: 2225 http://dx.doi.org/10.25304/rlt.v27.2225 (page number not for citation purpose) appendix kahoot survey 2018 challenge q1. i want to participate in the ‘kahoot!’ activities. *q2. i do not care if i win, only that i attempt the ‘kahoot!’ quiz. *q3. i find the technology of ‘kahoot!’ difficult to use. q4. i participate in the ‘kahoot!’ quizzes because i want to measure my own learning. q5. the ‘kahoot!’ quizzes challenge me in a positive way. *q6. i find the ‘kahoot!’ quizzes difficult. q7. i had all the information to complete the ‘kahoot!’ quizzes successfully. control *q8. i find the choices in the ‘kahoot!’ questions too limited to help my learning. q9. i complete the ‘kahoot!’ quizzes because they are anonymous q10. i wish ‘kahoot!’ was used in my other classes. immersion q11. i enjoy trying to beat my classmates in the ‘kahoot!’ on a regular basis. q12. knowing there will be a ‘kahoot!’ quiz helps me pay attention in the class. *q13. i do not enjoy the ‘kahoot!’ quizzes. interest q14. ‘kahoot!’ is fun and i enjoy trying to get the correct answer. *q15. if i answer a question incorrectly, i do not finish the ‘kahoot!’ quiz. q16. ‘kahoot!’ is visually appealing and exciting. purpose q17. i use ‘kahoot!’ to direct future learning and what i need to study. *q18. ‘kahoot!’ doesn’t help me measure my learning. *q19. i don’t think that ‘kahoot!’ helps me understand concepts in animal husbandry. q20. ‘kahoot!’ gives me context and the opportunity to apply new information. http://dx.doi.org/10.25304/rlt.v27.2225 reviews reviews by philip barker, university of teesside in the future, alt-j's policy will be to offer books and other products for review to members of alt or appropriate external experts. for this first issue of the journal, however, philip barker (associate editor in charge of reviews) has taken most of the burden on himself. a trilogy of books on cd-i compact disc interactive (cd-i) is a new electronic publishing medium for multimedia information. unlike conventional publishing media such as paper and film, cd-i provides an interactive method of accessing stored information and controlling its subsequent display on a tv screen. cd-i revolutionizes the publishing of all sorts of material such as music, text, images, computer graphics, film and video. it also adds many capabilities not possible with traditional publication media. until recently, however, the only widely available textbook on cd-i was preston's compact disc-interactive: a designer's overview, published in 1988 by kluwer technical books. now, with the recent release of cd-i in europe, three new books on the technology have become available. they form part of the cd-i series produced by philips interactive media systems (uk) and published by the addison-wesley. all three have 1992 imprints. introducing cd-i (isbn: 0-201-627-48-5) given its non-technical approach, this book is ideally suited to newcomers to cd-i technology. it is aimed at a broad audience from interactive producers and designers to all those who might be interested in the applications of cd-i as a communication and entertainment medium. the aim of the book is to provide an overview of cd-i and how this new standard for electronic publishing is likely to influence information provision and access in offices, schools, universities and home environments. the book is organized into four basic sections, plus a case study and a glossary. the first section deals primarily with some of the many potential uses of cd-i in a number of different contexts ranging from training, education and entertainment to corporate presentations and information provision for point-of-sale applications. this section also explains the basic nature of interactivity and how cd-i relates to interactive television. in the second section, the multimedia nature of cd-i is the focus: the medium allows text, sound, static pictures, animations and motion video to be incorporated and integrated into an interactive presentation. two of the most important features of cd-i are also emphasized here: the digital nature of each of the constituent media components, and the existence of a universal world standard (the green book) for cd-i publication and delivery platforms. the limitations of the finite capacity (650 megabytes) of a cd-i disc are discussed, and different ways of using the available storage are then outlined. the third section covers the basic steps involved in designing a cd-i product. each aspect of the design process, from product conception through to design specification, is introduced and described. producing a cd-i title usually requires a multi-disciplinary team with a range of different design and 72 altj volume 1 number 1 development skills, and this section concludes with a discussion of the importance of teamwork and project management. the fourth section describes how a design specification is taken through the various phases of development to final production. topics discussed in this section include copyright issues, authoring materials, production libraries, disc building, and development-team structure and responsibilities. the case study presented at the end of the book takes the reader through the steps involved in creating an actual cd-i publication {great british golf). it describes the roles of the different people involved in making the disc, its basic structure and content, and all the individual development and production processes (authoring, testing, mastering, and so on) needed to bring a commercial product to the market. the book provides a very readable introduction to cd-i and its applications for those who are still uninitiated into the technology. like the other two book in this series reviewed here, it is produced to a high standard (apart from one or two minor typographical errors) and is copiously illustrated. the cd-i design handbook (isbn: 0201-627-49-3) designing products for delivery on cd-i is in many ways different from designing for other publishing media. this design handbook therefore deals with a range of issues relating specifically to the cd-i design process from initial concept through to the beginning of production. it is aimed at designers who need to understand what they can do with cd-i technology rather than the detailed technical aspects of how cd-i works. there are ten chapters, an appendix, a reading list, a glossary and an index. the ten chapters are organized into three broad theme areas covering general introductory issues, technical perspectives, and detailed design considerations. none of the issues associated with asset or disc production is considered since these form the topic of a companion book (the cd-i production handbook see below). general introductory issues are dealt with in the first three chapters. the topics presented include an introduction to the features, facilities and potential of cd-i, a description of cd-i as an information storage and delivery medium, and a detailed exposition on how to design for interactivity. the technical perspectives (chapters 4 to 7) provide a more detailed description of the nature of cd-i's interactive interface, the use of video, image formats and image compression. this section also contains an outline of the various audio formats available, and gives examples of the different ways of using them. the section ends with more details about the overall cd-i delivery system, and how it works. the final section of the book (chapters 8 to 10) deals with various issues relating to the design stages involved in producing a cd-i title. the topics covered include a detailed description of the design process, design analysis, prototyping, copyright and legal issues, teamwork and management techniques such as developing the production plan, production scheduling, and budgeting. although the further reading section is rather sparse (containing only six references) both the appendix and the glossary are very useful. the appendix, for example, provides a valuable ready reckoner which enables designers to check that their design will fit within the storage and bandwidth requirements of cd-i. the cd-i production handbook (isbn: 0-201-627-50-7) this book is intended as a sequel to the design handbook reviewed above. it is 73 reviews aimed at those working on the production of cd-i titles (whatever their level of involvement) and those who are interested in commissioning cd-i material but are not sure about what to expect. there are eleven chapters, logically arranged into five theme areas: the pre-production phase of a project (chapters 1 to 3), cd-i facilities (4 and 5), gathering and producing cd-i resources (6 to 9), disc building (10), and follow-on projects (11). the reading list and glossary are essentially the same as those of the cd-i design handbook. the three chapters that make up the section dealing with the pre-production phase are of an introductory nature and are concerned with the infra-structure that must be created in order to undertake successful cd-i development. the first chapter covers issues such as the relationship between the producer and the client of a product, a description of the production process itself, and tools for cd-i production. the second chapter describes in detail the steps involved in setting up a cd-i studio and equipping it, and the nature of the hardware, software and human resources required. the final chapter in this section deals with project management issues such as setting up milestones, defining deliverables, scheduling, elapsed time, budgeting and copyright. the first chapter of the second section provides detailed descriptions of cd-i technology including the nature of the cd-roms on which interactive materials are published. there are descriptions of the parts of the cd-i player, the structure of discs (tracks and sectors), seek time, channels and data rate. synchronization, interleaving and real-time operation are also discussed. the second chapter in this section deals with cd-i effects. beginning with a discussion of picture planes and image types (such as fmv, rgb, dyuv, clut, etc.) it proceeds to a description of broadcasting conventions, resolution and aspect ratios. subscreens, partial updates and clut animation are then outlined, and there are descriptions of the various types of video plane effects such as cuts, wipes, fades, dissolves and mosaics. this section concludes with a short description of audio types and the use of soundmaps. the largest section of the book is the third, and it deals with the theme of gathering and producing cd-i resources. the first chapter of the section covers four important issues: the coding methods underlying the storage of materials in digital form on cd-i discs, how to generate the right sort of materials from scratch, how to convert existing materials from non-cd-i format, and how to manipulate and edit materials at different levels of authoring sophistication. the next two chapters deal with authoring and the creation of prototypes, authoring software such as mediamogul, and choosing an authoring system. the section concludes with a particularly informative description of programming, concentrating on philips's cd-i author (a programming environment for developing cd-i titles), the ims programmer's toolkit, the balboa runtime environment, cd-rtos and the os-9 real-time multitasking operating system. chapter 10 (which makes up the fourth section of the book) covers the disc-building to disc-pressing stages of cd-i production. consideration is given to a number of important issues such as the structure and creation of real-time files and the generation of cd-i disc images for testing purposes. there is then a discussion of the role of emulation (and how it works), followed by a description of worm discs and how they can be used for testing and validation. the final part of this section is concerned with the processes that take place at the cd-i production plant when the worm or tape 74 alt-j volume 1 number 1 streamer is used to create a master which is then used for replication purposes. the last section of the book consists of a single short chapter covering the various factors to be taken into account in anticipation of follow-up projects. it discusses issues such as creating sequels, house style and software efficiency through the design of re-usable code segments. this book really does contain lots of useful information for those wishing to find out more about cd-i production: of the three books in the series reviewed here, i found it the most interesting and instructive. at present there are four planned books in the series at the time of writing the cd-i programmer's handbook has not been released but is scheduled for publication later this year. undoubtedly, as cd-i takes off there will be scope for extending the series, perhaps with titles such as cd-i in the classroom or cd-i and higher education. the hutchinson electronic encyclopedia, first electronic version, oxford, random century and attica cybernetics, 1991. isbn: 1-873472-00-5. price £99. attaching a cd player to a personal computer is now a relatively simple process. so, as the cost of such players falls, the use of cd-rom for distributing large volumes of computer-based multimedia information becomes increasingly attractive, and electronic encyclopedias based on cd-rom are now becoming popular. compton's multimedia encyclopedia and the grolier electronic encyclopedia are two examples of this new type of information resource; the hutchinson electronic encyclopedia is another. the package consists of a standard iso 9660 cd-rom, a printed manual, and floppy disks (3.5 and 5.25 inch) containing the installation software. installation is relatively easy, taking only a few minutes. the system runs within microsoft windows. on start-up you are presented with a contents page designed to give easy access to the various categories of information available. the contents page is framed within a standard windows setting of a horizontal menu bar, scroll bars, dialogue boxes and reactive buttons. it is organized into three major areas: an eight-option iconic menu area, a window for the display of title and word indexes, and a horizontal array of 13 icons. the information within the encyclopedia is organized in the form of 'articles', each consisting of just one 'page* of information, so pages are of varying length. there are 25,147 articles listed in the general index, and they are sub-divided into seven basic subject categories: places (5,040), science and technology (3,378), history. (1,937), biographies (6,068), aspects of society (4,279), life sciences (3,396) and the arts (1,049). a particular information category is made the current one by clicking on the screen area associated with it, and once one of the categories has been selected it can be browsed through or searched. the icons in the icon array enable you to access the contents page from anywhere in the encyclopedia, display any page, examine the next and previous pages, perform word searches, examine the next and previous records retrieved by a search operation, create a backtrack list and actually backtrack, access a notepad facility, print out information in hard copy, and obtain online help. all these facilities are of course augmented by those provided in the standard windows environment. there are two basic ways of retrieving information. the first involves directly accessing articles using one of the two 75 reviews alphabetical indexes (title and word). the second method is based on full-text retrieval mechanisms using either a simple or complex keyword search facility. for a simple search strategy, you type in a string of characters that defines a descriptor representing your information requirement. all the articles indexed under this descriptor are then made available for display. if too many articles are retrieved, the search can be refined in order to reduce the number of hits. for example, a search for computer retrieves 220 articles; refining the search by also specifying digital reduces the number of hits to 20. complex searches are similar to simple searches except that up to nine keywords can be used simultaneously, each one linked to another by means of a logical operator (and, or, not). examples of complex searches are: comput* and babbage and turing (retrieves one article on the history of computing); and computer and digital and not optical (retrieves 18 articles about computers). unfortunately, many of the complex queries i tried resulted in that dreaded windows message: 'unrecoverable application error terminating current application'. that, however, is not my only complaint about this product. it performs too slowly, there are other bugs and errors, and the user-interface is of poor quality. i was also very disappointed by the quality of the printed manual. this is scanty with respect to both size and content. worse, it fails to reflect accurately what happens on-screen while the encyclopedia is being used. the slowness of the system is apparent in three important areas initialization, search time and retrieval time. in this context the grolier encyclopedia outperforms it by a long way. for example, once the hutchinson system has been activated by double clicking on the hutchinson icon, there is a wait of about 95 seconds before everything is ready to go. in comparison, grolier running on the same machine (a viglen 386) and using the same hitachi cd-rom drive takes less than 12 seconds to get going. grolier's search and retrieval times are also much faster: the hutchinson search takes over 27 seconds to locate a mere 452 references to the term 'paris'; to perform the same search, grolier takes less than four seconds to locate 1,425 references. equally irritating are the bugs and inconsistencies in the hutchinson package. examples include colour smudging on the opening page of the encyclopedia and in the photograph of john major (biographies section), angstrom being spelled 'a3b3ngstrom' in both the word index and in the article defining the term 'lumen' (in other places it is spelled correctly), picric acid being retrieved by the string 'c6h2' and not by its correct molecular formula c6h3n3o7 (which retrieves 'pyrogallol'), and entries in the title index appearing to be correctly capitalized whereas those in the word index are all in lower case something which makes chemical formulae and abbreviations appear unfamiliar. there are many other such little mistakes, but one of the most unforgivable limitations of the package is the lack of error checking in the underlying control software. for example, i started up hutchinson having inadvertently left my grolier disc in the cd-rom drive. the control software gave me no indication that i had an incorrect disc present and it took me a good while to determine why i could not get hutchinson to work. in comparison, grolier checks for the presence of the correct disc in the cd-rom drive and issues a warning if it is not there. in all, then, i hardly feel i can recommend hutchinson to anyone who requires fast and accurate access to encyclopedic information when at least one better alternative is available. 76 altj volume 1 number 1 rewritable optical storage technology, edited by j. paris roth, meckler, westport ct, usa, 1991. isbn: 0-88736534-5. because of its reliability, robustness and high storage capacity, read-only optical storage technology has become an important and well-established component of interactivemultimedia systems. although potentially as important, rewritable optical storage is not as well-established, but its use is growing. this book, produced by a group of authors, provides an introduction to the development and application of this relatively new approach to data and information storage and retrieval. there are seven chapters. the first provides an overview of rewritable optical storage technologies: magneto-optical (mo), dyepolymer, and phase-change. the merits and current de-merits of the most popular of these technologies mo are then described and a number of commercial products are compared. the need for inter-changeability and standards is discussed, and the chapter concludes with an annotated list of over 170 names and addresses of organisations currently undertaking active research into the development and/or application of rewritable optical storage technology. the second chapter describes how mo storage works and the advantages of this type of technology for storage-intensive applications such as cad (computer-aided design) and cae (computer-aided engineering). the benefits and uses of mo rewritable, worm and cdrom storage technologies are then compared, and the chapter ends with a description of a multifunction (hybrid) optical disc drive that could prove an attractive development for future computer systems. following the hardware descriptions given in the early part of the book, chapter 3 deals with some of the important software considerations relevant to rewritable and multifunctional optical drives. issues covered include the software requirements for worm and rewritable media, the need for a universal logical file format for rewritable media, and the small computer systems interface (scsi). chapter 4 describes the development of a new flexible rewritable storage technology suitable for use in tape and disc drives. the technology is based on the use of dyed polymer films for information storage, and a web-coating process for fabrication. the attractiveness of this latter process (compared with stamping processes) is that it can lead to substantial reductions in production costs. the last three chapters of the book deal with applications. chapter 5 covers the end-user requirements for networking an integrated rewritable optical disc workstation in order to support communication, storage and writing activities within a professional workgroup. similarly, the author of chapter 6 outlines the requirements of a workstation to support foreign-language instruction and how rewritable optical media could provide many of the required facilities. the final chapter deals with 'surrogate manipulations',, describing how a high-speed workstation fitted with a rewritable optical disc could be used to provide a multimedia workstation for the creation and storage of contemporary art material using images, text and sound. as well as a subject index, the book contains two appendices: a list of recommended readings relating to rewritable optical storage, and a glossary of terms and acronyms. the style and quality of the presentations varies from chapter to chapter. there are also a number of spelling mistakes and some important omissions. but the book is in general quite interesting, and will undoubtedly serve as a useful introduction for those wishing to find out more about the current 77 reviews state of the art in the area of rewritable optical storage and its applications. hypertext: state of the art, edited by r. mcaleese and c. green, oxford, intellect, 1990. isbn 1-871516-08-0. there has been growing interest over the last few years in hypertext and hypermedia, and a large number of commercial products have become available. these have included applications which deliver hypermedia information (for example, electronic books based on cd-rom), and hypermedia creation software (for example, authoring systems). this growing interest has developed from academic research which started back in the 1940s with vannevar bush and his memex concept. more recently, a great deal of research has been conducted into hypertext and hypermedia, and the book reviewed here aims to disseminate some of the findings of this research. it consists of 29 papers selected from those submitted to the 2nd uk human interface interactive learning systems sig conference on hypertext held in 1989 at the university of york. the principle keynote speaker was ted nelson, and although his speeches are not included in this book, a brief description of his contributions is provided in the preface. the papers are divided into four basic sections: navigation and browsing, hypertext and learning, prototypes, and designing hypertext and hypertext and design, each preceded by a short description of underlying issues. the first section, consisting of nine papers, addresses topics such as the use of metaphors, the effects that the underlying structure can have on navigation tools, and the relationship between navigational aids and control structures. section 2 contains seven papers covering the use of hypertext and hypermedia within learning applications. the reader is led to consider concepts such as learning styles, the degree of learner control, user models, and designing for precise learning outcomes. section 3 also has seven papers and deals with concepts associated with the creation of hypermedia applications. . a number of languages and environments are discussed (for example, hypercard and prolog) along with software maintenance and document processing. the last section has six papers, all dealing with design issues. two basic approaches emerge: a top-down approach, concentrating on the way knowledge or information should be represented, and a bottom-up approach concentrating on the system and user requirements. in addition to the theoretical treatment of the concepts and ideas relevant to hypertext and hypermedia, the papers presented in this book also generally provide the results of experimental work, allowing some quantification of the ideas discussed so that a clearer picture of the actual impact of a variety of hypermedia features can be assessed. this is not an introductory book. it is aimed at people who already have some knowledge of both hypertext and research techniques. it offers no overall conclusions, and it has the feel of conference proceedings rather than that of a textbook. nevertheless, the papers are of high quality, and the range of topics discussed makes this an extremely useful text for undergraduates, industry professionals, and anyone conducting research and/or development of hypermedia applications. a hypertext version of a previous book edited by ray mcaleese, hypertext: theory into practice (oxford, intellect, 1988) is provided free as a guide envelope application for the apple macintosh. stephen richards, university of teesside 78 doi:10.3402/rlt.v24.32036 research article different views on digital scholarship: separate worlds or cohesive research field? juliana e. raffaghelli, stefania cucchiara, flavio manganello and donatella persico* institute for educational technology, national research council of italy, genova, italy (received 23 april 2016; final version received 2 november 2016) this article presents a systematic review of the literature on digital scholarship, aimed at better understanding the collocation of this research area at the crossroad of several disciplines and strands of research. the authors analysed 45 articles in order to draw a picture of research in this area. in the first phase, the articles were classified, and relevant quantitative and qualitative data were analysed. results showed that three clear strands of research do exist: digital libraries, networked scholarship and digital humanities. moreover, researchers involved in this research area tackle the problems related to technological uptake in the scholar’s profession from different points of view, and define the field in different � often complementary � ways, thus generating the perception of a research area still in need of a unifying vision. in the second phase, authors searched for evidence of the disciplinary contributions and interdisciplinary cohesion of research carried out in this area through the use of bibliometric maps. results suggest that the area of digital scholarship, still in its infancy, is advancing in a rather fragmented way, shaping itself around the above-mentioned strands, each with its own research agenda. however, results from the cross-citation analysis suggest that the networked scholarship strand is more cohesive than the others in terms of cross-citations. keywords: digital scholarship; digital humanities; networked scholarship; digital libraries; systematic review 1. introduction the digital era is challenging all knowledge workers to develop new skills and literacies to work effectively within digital spaces (goodfellow 2014). the academic profession is no exception (borgman 2007; pearce et al. 2010; weller 2011), as digital technology offers unprecedented affordances to improve both research and teaching performance. the concept of digital scholarship emerged early in early in the 21st century (andersen and trinkle 2004; ayers 2004) and, according to wikipedia, refers to the use of information and communication technology to achieve scholarly and research goals. among the scholars’ activities that take advantage of technological affordances are: collecting evidence, carrying out investigations and research, publishing and disseminating results and preserving and making available outcomes. however, a fly-through the landscape of digital scholarship reveals that, although the term has become quite *corresponding author. email: persico@itd.cnr.it research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 j.e. raffaghelli et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 popular, it does not seem to have a widely agreed definition across the research disciplines that have contributed to its evolution. according to borgman (2007), for example, the concept of digital scholarship is tightly connected to the discourse about cyber infrastructures supporting new forms of doing research and science, namely eresearch and escience, which involves the progressive digitisation of institutional infrastructures and impacts on scholars’ practices in dealing with information and communication processes. borgman’s work, in fact, is deeply rooted in the field of information science whose primary aim is to improve the way libraries curate digital content and support scholarly work of all subject areas. this field of work also deals with the way scholars use the libraries’ digital facilities to increase their reputation (andersen 2004; holliman 2010; quigley et al. 2013; zhao 2014). at the same time, an important role in this field of research has been played by social science scholars who work at the crossroads between the humanities and digital technologies, thus identifying a new field of research, the digital humanities (terras, nyhan, and vanhoutte 2013), which is also strictly related to digital scholarship. as these authors point out, ‘digital humanities as a term (. . .) provides a big tent for all digital scholarship in the humanities’ (p. 140). these scholars have worked intensely to define the borders of this field of research (unsworth 2013), which embraces both the theory and the practices concerning the new forms of representation of cultural heritage, including history, arts and literature, through the digital medium (bentkowska-kafel 2013; gardiner and musto 2015; kaltenbrunner 2015). moreover, the term ‘digital humanities’ encompasses the area of debate about changing research methods and required professionalism in the humanities and the interdisciplinary dialogue with digital technologies (klein 2015). under the influence of the ideas of open science and open access (den besten, david, and schroeder 2010; suber 2009), the interest in the concept of digital scholarship has spread to social science researchers interested in investigating the complexities of the technological uptake by institutions and users as a cultural and social phenomenon. socio-technical studies played a highly important role in this case by expanding the focus of digital scholarship research in a direction different from those described above (borgman 2007, p. 43). this strand of research relates to academics’ professional learning and identity in the digital era and is tightly connected to educational technology research. its focus is on the ways scholars thrive to do (practices) and to be (identity) in the changing context of higher education, which pushes them � sometimes in rather conflictive and contradictory ways � to keep pace with innovations in digital, open and networked contexts (goodfellow 2014). the conundrum of opening up science and education is hereby faced through the exploration of professional learning by open, digital and networked scholars, that not only adopt technologies as a means but also reflect on the nature and ethics of research, through their deontological position, and create new scenarios of practice (costa 2014; scanlon 2014; veletsianos and kimmons 2012b). this approach aligns with socio-technical studies going beyond technological determinism (pearce et al. 2010). for this group of researchers, the research problems of digital scholarship are connected with the adoption of social media to do and share research, social scholarship (greenhow and gleason 2014; manca and ranieri 2016; veletsianos 2012), with emerging forms of reputation based on general and bespoke media tools (nicholas, herman, and jamali 2015; weller 2012); with fluid processes of collaborative research entailing interdisciplinary dialogue, teaching and dissemination (veletsianos and j.e. raffaghelli et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 kimmons 2012a); and with a vision of open science that engages public audiences in the making of science, by extending the forms of participation along with the research process (grand et al. 2012). the whole debate is connected to the need for improving scholars’ literacy to participate in digital, networked and open contexts of scholarship (goodfellow and lea 2013; veletsianos and kimmons 2012b). the work of this group of researchers is rooted in the model by boyer (1990) of the academic profession and suggests that boyer’s four dimensions (discovering, integration, application and teaching) are being enhanced and transformed by openness and networking, thus creating new professional ways of collaboration across geographical and institutional frontiers based on the affordances provided by web 2.0 (greenhow and gleason 2014; nicholas, herman, and jamali 2015; weller 2011). the above picture lets us appreciate that digital scholarship is a complex research area, guided by different research aims, rooted in several conceptual and methodological bases, and informed by diverse disciplinary traditions. moreover, it appears that the concept of digital scholarship is rather fuzzy, embracing different concerns and using a variety of research methods, professional practices and scholars’ identities. this blurred picture stimulated the authors of this article to analyse the literature on the topic in order to identify more clearly the different areas of research involved and better understand their relative importance, the reciprocal influences, the common concerns and the specificities, in terms of the problems tackled, the topics dealt with and, more generally, the interplay of the disciplines involved. to this end, a systematic review of the literature on digital scholarship has been carried out, complemented with bibliometric maps aimed to reveal and investigate the main views on digital scholarship, the keywords used and the extent to which they build upon each other’s results. the research aim is to explore whether, and to what extent, the emerging landscape depicts a unitary and cohesive research topic, or a fragmented disciplinary vision. as a result, our study should contribute to inform the evolution of this research topic, clarifying the areas where there is a need for better convergence of research problems and questions, and of connected constructs and methodological approaches. 2. methodological approach set out as a classic systematic literature review (petticrew and roberts 2006), this study encompassed an initial identification of a significant sample of publications concerning the field of digital scholarship, followed by the construction of a database where such publications are classified according to relevant categories. then, bibliometric maps have been used to identify the relationships between the papers and to spot existing agglomerates, corresponding to different strands of research. both the systematic review and the bibliometric maps were adopted to explore the relationships between the three strands of research identified, namely digital libraries, networked scholarship and digital humanities. we searched for juxtapositions in the classification of research areas, the research aims, the methodological approaches adopted, the citations between contributing authors and the concepts emerging as mostly used (keywords) to achieve a better picture of digital scholarship as a research topic. in the following subsections, we will describe the sample, the data collection process and the methods adopted for the data analysis. research in learning technology citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 2.1. sample selection the sample analysed comprised 45 papers of relevant scholarly literature published during the period january 2004�march 2015. the sample, derived from the initial exploration of six specialised databases, namely web of science (wos; 31% of all papers were found in this database), scopus (75% of papers), the directory of open access journals (doaj; 7%), educational resources information centre (eric; 20%), editlib digital library (11%) and google scholar (93%), was arrived at through a search for the term ‘digital scholarship’ in the title, the abstract and the keywords. the search yielded 154 papers, which were filtered by eliminating (1) duplicated papers, due to overlaps between databases; (2) papers with full text in languages other than english, (3) pieces of work other than research papers (reports, position papers, magazine articles, etc.) and (4) proceedings papers. technically, the authors searched for pieces of work representing consolidated research, thereby highlighting phenomena as well as conceptualisations that have passed a rigorous process of evaluation. this process led to the sample of 45 journal papers indexed by at least one of the above-mentioned databases. the complete information of every article is documented in annex 1 � references used for the review. 2.2. data collection process and analysis 2.2.1. first phase: classification of articles according to the systematic review approach, the next step consisted in defining the structure of a database destined to host the relevant information about the papers. the database records were structured as reported in table 1, according to a procedure previously used elsewhere by raffaghelli, cucchiara, and persico (2015). table 1 shows the dimensions that the researchers deemed relevant for the analysis of the field of digital scholarship. while the way the first three data fields of each record were filled in does not require further discussion, since researchers only had to report the data as found in the paper or in database sources, fields 4 and 5 require some additional explanations. field 4, corresponding to the dimension ‘view on digital scholarship’, refers to the three main perspectives on digital scholarship research described in the introduction of this paper. the first one was ‘networked scholarship’ and included all the papers that adopted social networks and other informal methods to disseminate research and teaching, as well as those that dealt with open science and open educational resources. the second one was ‘digital libraries’ and included papers analysing the digital infrastructures and their affordances, and the stakeholders’ policies with regard to them. the third category was ‘digital humanities’ and included papers on new research methods to capture or represent research objects within the humanities. although these three categories are consistent with the trends outlined in the analysis of the literature, in principle, some papers may simultaneously belong to two or even all three of the above-mentioned categories. for this reason, four hybrid categories were also created. however, there were no papers that were found to lie at the crossroad between the three categories. as for the ‘research approach’, the sub-field ‘research topic’ was an open field, and it was processed through a ‘thematic analysis’ procedure (guest, macqueen, and namey 2011), a widely used qualitative research method based on an inductive j.e. raffaghelli et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 approach. the full text of the articles under analysis was explored by two researchers according to the following procedure: (1) the research topics were extracted by one researcher who created ‘subcategories’ in a first round of classification of articles (free codification); (2) the results of phase (1) were shared between the two researchers (member-checking); (3) both researchers independently coded five papers table 1. database fields and values assigned. data field data sub-field assigned values type of data 1 � article article title title as published text/as found in identification source journal title paper author(s) author(s) name and surname publication date year key words keywords as published abstract full abstract as published 2 � scientific database wos https://apps. webofknowledge. com/ ‘the article is indexed in the database’ true/false (1/0) value/as found scopus http://www.scopus. com/ eric http://eric.ed.gov/ doaj https://doaj.org/ editlib http://www.editlib. org/ google scholar https://scholar. google.it/ 3 � research area on the scientific database research area on scientific database classification of research as extracted from the scientific database text/as found 4 � view on ds classification of research taking into consideration the research area as well as the theoretical approach: a. ns b. dh c. dl d. ns/dl e. ns/dh f. dl/dh g. dl/dh/ds text-label/ upon researcher’s interpretation 5 � research approach research topic the topic of research, if declared. text/upon researcher’s interpretation research aim the guiding research purpose text/upon researcher’s interpretation dh, digital humanities; dl, digital libraries; ds, digital scholarship; ns, networked scholarship. research in learning technology citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 5 (page number not for citation purpose) https://apps.webofknowledge.com/ https://apps.webofknowledge.com/ https://apps.webofknowledge.com/ http://www.scopus.com/ http://www.scopus.com/ http://eric.ed.gov/ https://doaj.org/ http://www.editlib.org/ http://www.editlib.org/ https://scholar.google.it/ https://scholar.google.it/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 using the agreed-upon sub-categories and the inter-rater analysis was carried out; and (4) both researchers proceeded with the classification by adopting the existing sub-categories as themes covering one or more free codes, which in this case represent the research topics. in order to deal with possible biases in the researchers’ judgement of database fields 4 and 5, the classification of the papers consisted three steps: the first step of joint ‘training’ was followed by the second step where both researchers classified independently the same five articles (12% of the whole sample) and the third step where the inter-rater agreement between the two raters was calculated. the inter-rater’s percentage of agreement was 82%. cohen’s kappa coefficient was also calculated, obtaining a value of 0.98, which can be considered a high level of agreement (hayes and krippendorff 2007). controversial cases were then discussed till a consensus was reached. 2.2.2. second phase: bibliometric maps production and analysis while the first phase of this study was meant to allow the authors to identify the main areas of investigation, the focus of the studies on digital scholarship and the type of research carried out, the second phase was based on bibliometric maps and aimed at investigating the relationships amongst the disciplinary perspectives. bibliometric maps are a form of representation of scientific networks (van eck and waltman 2014) used in scientometrics as a means to understand connections between researchers and their work. they are based on three main elements: statistical analysis of written publications (often including text and data mining); different methods of visualisation (distance-based, graph-based and timeline-based) and digital tools supporting analysis and visualisation. bibliometric maps are graphs consisting of nodes and edges; while the nodes may represent publications, journals, researchers or keywords, the edges represent relationships between the nodes. according to the type of nodes, the focus of analysis and the emerging map are different. the most frequent types of relationship studied through bibliometric maps are: citations among papers (to explore connections between publications), co-authorship relations (to explore connections inside a network of researchers) and keyword co-occurrences (providing information about the distribution of topics) (van eck et al. 2010). some forms of visualisation explore static relationships, highlighting groups (clusters) of nodes that are ‘closer’, while others explore their evolution in time. in this research, bibliometric maps were used to analyse the sample of 45 papers in order to: (1) study the keywords characterising the field and differentiating agglomerates of papers and their relationships (i.e. central/peripheral, related/not related). the operational hypothesis guiding this analysis was that the three main groups of keywords, respectively, connected to the three digital scholarship views would emerge as clusters within the semantic universe connected to the construct of digital scholarship, and (2) study the relationships between bibliographic items in terms of citations. the operational hypothesis here was that the distinction between the three ‘views’ on digital scholarship would be reflected by intense cross-citations within clusters of papers and few cross-citations between papers of different clusters. j.e. raffaghelli et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 after a careful analysis of existing tools for bibliometric maps analysis and visualisation, the authors selected two software tools to carry out this phase of the study: vosviewer, 1 for the analysis and visualisation of keywords, and citnetexplorer 2 for the analysis and visualisation of cross-citations. in the case of the cross-citations bibmap, the original sample consisting of the 45 papers studied in the first phase was integrated by three books (borgman 2007; boyer 1990; weller 2011), since they were highly cited by the 45 papers. furthermore, these books were perceived to be relevant to define the background and hence the relationships between the views. 3. results 3.1. first phase: ‘characterising disciplinary contributions to digital scholarship’ this section presents the results of the first phase of work, the systematic review. figure 1 shows that digital scholarship appears to be a fairly recent field of research, dating back to 2004 (although rooted in previous literature on scholarship), featuring a significant increase in papers on scientific productivity in the years 2013 and 2014 (the yearly number of papers almost doubled between 2012 and 2013 and doubled again in 2014); this highlights a fast emerging field of research. in addition, the papers are well distributed amongst several journals belonging to different subject areas, which confirm the relevance of the topic for different disciplines. figure 2 shows the distribution of research topics as they emerged from the ‘thematic analysis’ procedure described in the methodological approach section. we note the prevalence (46%) of the group of papers dealing with the issue of (academic) professional practices tightly connected to educational research; these papers deal with research in the field of higher education and focus on digital scholarship as a problem of professional learning and innovation. this is followed by a number of papers (16%) concerning the themes of ‘openness, democratisation of education’ and the ‘participatory culture’ of the web. less represented is the topic of ‘digital identity, interaction, social networks (sns) and social media’. besides, ‘e-publishing and 18 16 14 12 10 8 2 0 serie1 2004 2005 2006 2007 2008 articles on ds per year 2009 2011 2012 2013 2014 9 165412 22101 2010 6 4 figure 1. evolution of scientific production in the field (2015 is excluded because data were collected only for the first semester). research in learning technology citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 library services’ (16%), as well as ‘digital art and history’ (13%) deserve interest. the least represented topics are ‘e-science and information and communition technologies (ict)’ and ‘multimedia and innovation’ (3%). during this process of analysis, the researchers observed co-occurrences of keywords between papers dealing with the topics: ‘‘professional practices, educational practices’’; ‘‘openness and democratisation, participatory culture’’; and ‘‘digital identity, interaction, sns and social media’’. these three topics together, respresent 62% of the sample. besides, the topics of ‘‘e-publishing, library services’’ shared several keywords with ‘‘e-science and ict, multimedia, innovation’’, representing together 19% of the sample. lastly, the topic of digital art and history appeared to be a stand-alone category. the above situation relating to research areas as well as research topics revealed that the expected three main views were present in the sample: the view of digital scholarship as a networked process of collaboration on the open web connected to the 5;16% 2;6% 4;13% 5;16% 11;36% 1;3% 3;10% e-science and ict, multimedia, innovation identity, interaction, sns and social media professional practices, educational practices e-publishing, library services openess and democratization, participatory culture figure 2. topics of research. dl/ns (6;13% ) ns (17;38% ) dh (7; 16% ) h/ns (1; 2% ) dl (12; 27% ) dh/dl/ns (0) dh/dl (2;4%) figure 3. views on digital scholarship. j.e. raffaghelli et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 scholars’ endeavour to transform their own practices with a new deontology of scholarship; the view of digital infrastructures (libraries) leading researchers to adopt new affordances to do their work and hence requiring professional interventions to organise new, complex technological contexts; and the view of digital humanities as a strand of research focused on technological settings and objects supporting research in the humanities, but also transforming it. it was observed that most papers falling in the research area of ‘social sciences’ belonged to the first group; that in the case of the second group, the papers could be placed amongst the two research areas ‘information sciences’ and ‘computer sciences’; and in the third case, the papers were distributed between ‘computer sciences’ and ‘humanities’, showing the separations in the disciplines contributing to the ‘views on digital scholarship’. figure 2 illustrates the distribution of papers per ‘view’, including papers with overlapping or ‘mixed’ visions. according to figure 3, most papers in our sample are distributed between the dominant visions of networked scholarship (38%) and digital libraries (27%), with less presence of the digital humanities (16%). only20% of the examined papers are ‘hybrids’ and simultaneously belong to two visions. no paper belongs to the interception of the three visions. the set of papers belonging to the field of digital humanities, besides being smaller, is also more isolated than the other two (6% of overlapping with the other two). 3.2. second phase: ‘exploring disciplinary relationships within digital scholarship’ 3.2.1. the map of keywords co-occurrences the map of co-occurrences of keywords is a representation based on the number of occurrences of keywords within the ‘corpus’ of terms extracted from all the titles, keywords and abstracts of the articles within the sample. the software vosviewer extracts all the ‘noun-phrases’ from the corpus; therefore, the terms are organised by topics automatically generated by the software, namely the keywords. in this case, from the original corpus, the software extracted 1,198 relevant keywords from a sample of 6,898 terms. a total number of 73 nodes emerged; however, only 44 (60%) of these keywords are considered by the software for representational purposes. moreover, the authors removed irrelevant or ambiguous keywords from the representation such as too general terms (e.g. issue, author, purpose, role, publishing, scholar, academic, survey) or terms which conditioned the visualisation of a cluster, such as teaching, publication, implication, challenge and collaboration. the final representation, composed of 35 keywords/nodes, is shown in figure 4), where three bigger clusters and two smaller clusters are identifiable. table 2 introduces the details of keywords for each cluster, while in the second column, we have associated each cluster with the relevant perspective. cluster 1 (in red at the top of figure 5), contains eight nodes, with ‘library’ as its main node, and was connected semantically with the view of digital libraries research, focusing on the role of infrastructures allowing new ways of scientific production, the problem of preserving and using content, and the role of libraries and librarians in scientific information. on the right-hand side in figure 5 (in green), cluster 2, with the central word ‘network’, contains eight nodes. this cluster appears to be semantically connected with the perspective of ‘networked (and open) scholarship’ seen as scholars’ endeavour to embrace the social web with all its affordances to promote new practices (such as opening up education and research) in line with a new deontology of public engagement. cluster 5 (in purple), in between research in learning technology citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 clusters 1 and 2, features ‘openness’ as the main node, relates to the issue of open access to content and to open scholarship as professional practice. hence, this small cluster brings some evidence of the existence of contaminations between the digital libraries and the networked scholarship perspectives. the third biggest cluster, cluster 3 (in blue), at the bottom-right of the map, is composed by seven nodes, and its main nodes are ‘humanities’ and ‘collaboration’. we can assume that this cluster aligns with the perspective of digital humanities, dealing with how researchers interact with new digitised objects within the humanities as well as how the field evolves as an interdisciplinary field, between computer science and the humanities. cluster 4 (in yellow), a small cluster whose main node is ‘history’; is tightly connected with the digital humanities perspective. in figure 5, this cluster is rather isolated, and specifically there are no nodes that can be attributed to the view networked scholarship/digital humanities, which seems to be in line with the very small overlapping between these two perspectives already shown in figure 3. the clusters described above can be clearly put in relation with the ‘views’ on digital scholarship identified in the first phase of the study. one question that could be raised is whether the mere existence of clusters 1, 2 and 4 as separate clusters reflects little mutual awareness deriving from the respective disciplinary viewpoints; and whether the connections observed (cluster 3 and 5) can be regarded as a sort figure 4. bibliometric map of keywords (colours are attributed to nodes by vosviewer to highlight clusters). table 2. clusters of keywords and connected perspectives on digital scholarship. cluster keywords connected perspective 1-red access, digital age, librarian, library, literacy, open access, tool, web dl 2-green digital scholar, engagement, habitus, network, participatory web, scholarly practice, social medium, twitter ns 3-blue collaboration, digital humanity, humanities, humanity, infrastructural inversion, social science, visualisation dh/dl 4-yellow digital art history, digitalisation, discipline, history dh 5-purple education, open scholarship, openness ns/dl dh, digital humanities; dl, digital libraries; ds, digital scholarship; ns, networked scholarship. j.e. raffaghelli et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 of starting point for interdisciplinary analysis of the topic of digital scholarship. clearly, the keywords map is not informative enough to answer this question, while the cross-citation bibmap described in the next section can shed more light on it. 3.2.2. cross-citations bibliometric map across-citation bibliometric map was built to understand the relationships between cited and citing papers, that is, to understand whether the authors built upon the work of each other. more in general, this type of map allowed us to focus on the extent to which each research strand is aware of the work of the others. the software used for this purpose (citnetexplorer) visualises the relevant publications of our sample as well as their citational relationships across a time span. in our case, the time span (1990�2015) is the one covered by our sample, consisting of 45 papers plus three highly cited books (borgman 2007; boyer 1990; weller 2011). figure 5 shows the bi-dimensional representation of the citation network per year, organised in clusters of publications based on their citational relationships. in figure 5, a cluster is identified and its nodes highlighted. the parameter ‘minimum number of citation links’ was set at 3, which means that documents receiving less than three citations from other documents of the sample are not visualised in the map. this is a low value in typical bibliometric problems, but adequate for this small set of documents; in any case, the situation observed is typical of very specific research fields, as well as of the application of bibliometric indicators in the humanities and in educational research (hammarfelt 2014). in figure 5, it is possible to observe one main cluster of 25 publications, and some isolated nodes. the main cluster corresponds to papers belonging to the networked scholarship group, which are at the centre of the cluster (i.e. labelle, weller, veletsianos, costa, goodfellow), but it also includes publications belonging to ‘hybrid’ categories (i.e. wolski & richardson, pasquini, holliman from digital libraries/ 1990 1991 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1992 figure 5. bibliometric map of cross-citations. multiple papers with the same first author can be distinguished by the year, that is, the position along the y axes. no couples of papers/books with the same first author in the same year are present in the sample. research in learning technology citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 networked scholarship; najmi from digital humanities/networked scholarship ). in this cluster, the only publication which was classified as digital humanities is that by kaltenbrunner. however, it is crucial to highlight that all of these articles cite two core books in the cluster: boyer (1990) and borgman (2007). while the first author pioneered the debate on the need for revolutionising the academic profession in the very early nineties, the second has become a crucial landmark in the research about the changing cyber infrastructures supporting (and questioning) scholarship. boyer’s work is particularly considered as a model to understand academics’ professional practices. instead, borgman’s book is a pillar of the debate about the scholarly communication paradigms that the academics have to face. indeed, an analysis performed removing these two books shows a cluster of authors mainly belonging to the networked scholarship view (17), and the rest of publications completely scattered and isolated. another important book for the scientific community exploring the topic of digital scholarship is weller (2011) that can be seen at the centre of the cluster, with less cross-citations due to the fact that it is more recent than the other two. besides, there are very few cross-citations (lateral lines) between authors within the network. this emerges from the identification of the core publications (12 in total); these are publications that have at least a certain minimum number of citation relations with other core publications, taking into account that incoming and outgoing citation relations are treated identically. the 12 publications identified mainly coincide with the networked scholarship perspective identified in the prior phase. with regard to the isolated publications (20), the situation is mixed between digital libraries and digital humanities, which means that there are little citations between these perspectives, and the work considering the construct of digital scholarship in these two areas is not cohesive. to wrap up this part of the analysis, one could have expected a citation map clearly showing the three clusters networked scholarship, digital libraries and digital humanities consisting of publications reciprocally citing each other inside each view and with fewer citations across clusters. this does not seem to be the case. however, this analysis brings to light issues that are consistent with the prior analysis. the first is the low number of cross-citations, supporting the idea that the field of research is rather fragmented, which highlights that most contributions do not take into account the three disciplinary perspectives. the exception to the above consideration is provided by the publications belonging to the networked scholarship perspective, that is, those that explore academic professional practices and scholars. the existence of this cluster seems to confirm that scholars who study networked scholarship are actually more ‘networked’ than the others, and the identity of this field of research should and perhaps could be built on their shoulders. however, the weak connection with the other perspectives allows us to suppose that this group could be rather unaware of the contributions coming from the other two perspectives, their problems and their research agenda; as a result, we can conclude that interdisciplinary collaboration in this area is not strong enough. 4. discussion and conclusions this study was aimed at exploring and mapping a set of 45 selected papers on digital scholarship. most of these articles aimed to define the concept and to study related phenomena (‘in the wild’), that is, the academics’ practices and the supporting infrastructures in a digital, open and networked context of activity. at first sight, j.e. raffaghelli et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 the authors observed the coexistence of several interpretations of the term, each reflecting different disciplinary research perspectives on the construct. consequently, the whole study was set up to investigate digital scholarship by identifying the main research strands involved and their relationships, including common epistemological roots, reciprocal awareness, as well as key topics and concerns of the strands and their overlapping. the results of the study show not only the fragmentation of research efforts across three main disciplinary strands of research but also a relatively low degree of cohesion inside each strand, which might be due to the early stage of development of this research field (even if the first paper dates back to 2004, the field actually took off around 2010). the three main strands, networked scholarship, digital libraries and digital humanities, seem to differ as to the disciplinary background (respectively, social sciences, information sciences and humanities). in spite of the isolation observed, our exploration revealed some partial overlapping through the thematic analysis of keywords, as well as the bibliometric maps of keywords. this probably indicates that research problems and discourses are connected to some extent. in fact, networked scholarship is connected with some of the assumptions of digital libraries, while digital humanities seems loosely connected to digital libraries and networked scholarship. the authors could not classify any paper at the interception of the three. the cross-citation map shows a rather fragmented panorama, rooted in some previous seminal books, with more citations between publications of the ‘networked scholarship’ strand and a few cross-citations between publications of the other two. besides, there are a few citations between strands and very few citations between digital libraries and digital humanities. in this regard, the cross-citation map was not completely convergent with the researchers’ manual classification and the thematic analysis: the isolation observed was even higher than expected. the above considerations confirm that the construct of digital scholarship encompasses three strands of research with a rather clear focus and raises the question of whether there is a lack of reciprocal awareness, possibly preventing scholars to build on prior efforts, towards an interdisciplinary collaboration. the division between the disciplinary fields contributing to the topic of digital scholarship hereby presented is not new in the literature and has been pointed at by several authors (goodfellow 2014; quan-haase, suarez, and brown 2014; scanlon 2011). however, this analysis contributes to the discourse by highlighting both the forms of fragmentation assumed by the literature and the existing attempts to overcome this fragmentation. above all, the problem of coexistence of different digital scholarship definitions and the field conceptual fragmentation causes an entropic situation hindering further empirical research. for example, it makes it difficult to identify what is innovative and to put forward recommendations for practice (e.g. proposals for the training of scholars) and for policy-making (e.g. prioritising efforts of investment in scholars’ career development, in supporting infrastructures and in the evaluation systems based on scientific productivity). another important issue relates to the values attached to the research undertaken across the three ‘views’ of digital scholarship. while many studies, particularly within the networked scholarship perspective, focus on the positive ethical value of open scholarship, based on avant-garde practices and pioneering scholars, other studies bring to light the lack of participation of scholars to innovative practices, emphasising the limited concern about the need for changing the practices of research in learning technology citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 scholarship as well as the attritions between innovation and tradition in academic research and teaching (costa 2014). solving these issues requires an increase in the level of awareness among scholars, the adoption of convergent research methods and visions of the field and more interdisciplinary dialogue between researchers. notes 1. http://www.vosviewer.com/ 2. http://www.citnetexplorer.nl/ references andersen, d. l., ed. (2004) digital scholarship in the tenure, promotion, and review process, m.e. sharpe, london. andersen, d. l. & trinkle, d. (2004) ‘valuing digital scholarship in the tenure, promotion, and review process � a survey of academic historians’, in digital scholarship in the tenure, promotion and review process, d. andersen, m.e. sharpe, london, pp. 61�77. borgman, c. l. (2007) scholarship in the digital age, mit press, cambridge. boyer, e. l. (1990) scholarship reconsidered: priorities of the professoriate, vol. 1997, carnegie foundation for the advancement of teaching, wiley, new york. den besten, m., david, p. & schroeder, r. (2010) ‘research in e-science and open access to data information’, in international handbook of internet research, eds j. husinger, l. klastrup & j. allen, london, new york, springer, pp. 65�96. doi: 10.1007/978-1-40209789-8 gardiner, e. & musto, r. g. (2015) the digital humanities a primer for students and scholars, harvard university press, cambridge, ma. goodfellow, r. & lea, m. (2013) literacy in the digital university: critical perspectives on learning, scholarship, and technology, routledge, london. grand, a., et al., (2012) ‘open science: a new ‘‘trust technology’’?’, science communication, vol. 34, no. 5, pp. 679�689. doi: http://dx.doi.org/10.1177/1075547012443021 greenhow, c. & gleason, b. (2014) ‘social scholarship: reconsidering scholarly practices in the age of social media’’, british journal of educational technology, vol. 45, no. 3, pp. 392�402. doi: http://dx.doi.org/10.1111/bjet.12150 guest, g., macqueen, k. m. & namey, e. e. (2011) applied thematic analysis, vol. 9, sage, london. hammarfelt, b. (2014) ‘using altmetrics for assessing research impact in the humanities’’, scientometrics, vol. 101, no. 2, pp. 1419�1430. doi: http://dx.doi.org/10.1007/s11192-0141261�3 hayes, a. f. & krippendorff, k. (2007) ‘answering the call for a standard reliability measure for coding data’, communication methods and measures, vol. 1, no. 1, pp. 77�89. doi: http://dx.doi.org/10.1080/19312450709336664 klein, j. t. (2015) interdisciplining digital humanities: boundary work in an emerging field, university of michigan press, ann arbor, mi. [online] available at: http://quod.lib.umich. edu/cgi/t/text/text-idx?cc=dh;c=dh;idno=12869322.0001.001;rgn=full%20text;view=toc;xc=1; g=dculture manca, s. & ranieri, m. (2016) ‘yes for sharing, no for teaching!’’: social media in academic practices’, the internet and higher education, vol. 29, pp. 63�74. doi: http://dx.doi.org/10. 1016/j.iheduc.2015.12.004 nicholas, d., herman, e. & jamali, h. r. (2015) emerging reputation mechanisms for scholars, european commission jrc science and policy report, seville, spain. doi: http:// dx.doi.org/10.2791/891948 pearce, n., et al., (2010) ‘digital scholarship considered: how new technologies could transform academic work’, in education, [online] available at: http://ineducation.ca/ ineducation/article/view/44/508 j.e. raffaghelli et al. 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 http://www.vosviewer.com/ http://www.citnetexplorer.nl/ 10.1007/978-1-4020-9789-8 10.1007/978-1-4020-9789-8 http://dx.doi.org/10.1177/1075547012443021 http://dx.doi.org/10.1111/bjet.12150 http://dx.doi.org/10.1007/s11192-014-1261-3 http://dx.doi.org/10.1007/s11192-014-1261-3 http://dx.doi.org/10.1007/s11192-014-1261-3 http://dx.doi.org/10.1007/s11192-014-1261-3 http://dx.doi.org/10.1007/s11192-014-1261-3 http://dx.doi.org/10.1007/s11192-014-1261-3 http://dx.doi.org/10.1007/s11192-014-1261-3 http://dx.doi.org/10.1007/s11192-014-1261-3 http://dx.doi.org/10.1080/19312450709336664 http://quod.lib.umich.edu/cgi/t/text/text-idx?cc=dh;c=dh;idno=12869322.0001.001;rgn=full%20text;view=toc;xc=1;g=dculture http://quod.lib.umich.edu/cgi/t/text/text-idx?cc=dh;c=dh;idno=12869322.0001.001;rgn=full%20text;view=toc;xc=1;g=dculture http://quod.lib.umich.edu/cgi/t/text/text-idx?cc=dh;c=dh;idno=12869322.0001.001;rgn=full%20text;view=toc;xc=1;g=dculture http://dx.doi.org/10.1016/j.iheduc.2015.12.004 http://dx.doi.org/10.1016/j.iheduc.2015.12.004 http://dx.doi.org/10.2791/891948 http://dx.doi.org/10.2791/891948 http://ineducation.ca/ineducation/article/view/44/508 http://ineducation.ca/ineducation/article/view/44/508 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 petticrew, m. & roberts, h. (2006) systematic reviews in the social sciences, a practical guide, blackwell, oxford, uk. raffaghelli, j. e., cucchiara, s. & persico, d. (2015) ‘methodological approaches in mooc research: retracing the myth of proteus’, british journal of educational technology, vol. 46, no. 3, pp. 488�509. doi: http://dx.doi.org/10.1111/bjet.12279 suber, j. p. (2009) open-access timeline, [online] available at: http://legacy.earlham.edu/ ~peters/fos/timeline.htm terras, m., nyhan, j. & vanhoutte, e, eds. (2013) defining digital humanities: a reader, ashgate, london. unsworth, j. (2013) ‘what is humanities computing and what is not?’, in defining digital humanities: a reader, eds. m. terras, j. nyhan & e. vanhoutte, ashgate, london, pp. 51�65. van eck, n. j. & waltman, l. (2014) ‘visualizing bibliometric networks’, in measuring scholarly impact: methods and practice, eds. y. ding, r. rousseau & d. wolfram, springer, london, pp. 285�320. van eck, n. j., et al., (2010) ‘a comparison of two techniques for bibliometric mapping: multidimensional scaling and vos’, digital libraries; physics and society, [online] available at: http://arxiv.org/abs/1003.2551 weller, m. (2011) the digital scholar: how technology is transforming scholarly practice, bloomsbury academic, london. annex 1 � references used for the literature review antonijević, s. & cahoy, e. s. (2014) ‘personal library curation: an ethnographic study of scholars’ information practices’, portal-libraries and the academy, vol. 14, no. 2, pp. 279�298. ayers, e. (2004) ‘doing scholarship on the web: ten years of triumphs � and a disappointment’’, journal of scholarly publishing, vol. 35, no. 3, pp. 143�147. doi: http://dx.doi.org/ 10.3138/jsp.35.3.143 bennett, l. & folley, s. (2014) ‘a tale of two doctoral students: social media tools and hybridised identities’, research in learning technology, vol. 22, p. 23791. doi: http://dx.doi. org/10.3402/rlt.v22.23791 bentkowska-kafel, a. (2013) ‘i bought a piece of roman furniture on the internet. it’s quite good but low on polygons. � digital visualization of cultural heritage and its scholarly value in art history’, visual resources, vol. 29, no. 1�2, pp. 38�46. doi: http://dx.doi.org/10. 1080/01973762.2012.761117 burdick, a. & willis, h. (2011) ‘digital learning, digital scholarship and design thinking’, design studies, vol. 32, no. 6, pp. 546�556. christie, a. (2014) ‘interdisciplinary, interactive, and online: building open communication through multimodal scholarly articles and monographs’, scholarly and research communication, [online] available at: http://src-online.ca/index.php/src/article/view/190/367 costa, c. (2013) ‘the habitus of digital scholars’, research in learning technology, vol. 21, p. 21274. doi: http://dx.doi.org/10.3402/rlt.v21i0.21274 costa, c. (2014) ‘outcasts on the inside: academics reinventing themselves online’, international journal of lifelong education, vol. 34, no. 2, pp. 194�210. doi: http://dx. doi.org/10.1080/02601370.2014.985752 dority baker, m. l. (2013) ‘using buttons to better manage online presence: how one academic institution harnessed the power of flair’, journal of web librarianship, vol. 7, no. 3, pp. 322�332. doi: http://dx.doi.org/10.1080/19322909.2013.789333 goodfellow, r. (2014) ‘scholarly, digital, open: an impossible triangle? research in learning technology, vol. 21, p. 21366. doi: http://dx.doi.org/10.3402/rlt.v21.21366 hartelius, e. j. & mitchell, g. r. (2014) ‘big data and new metrics of scholarly expertise’, review of communication, vol. 14, no. 3�4, pp. 288�313. doi: http://dx.doi.org/10.1080/ 15358593.2014.979432 heap, t. & minocha, s. (2012) ‘an empirically grounded framework to guide blogging for digital scholarship’, research in learning technology, [online] available at: http://www. researchinlearningtechnology.net/index.php/rlt/article/view/19195/xml research in learning technology citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 15 (page number not for citation purpose) http://dx.doi.org/10.1111/bjet.12279 http://legacy.earlham.edu/~peters/fos/timeline.htm http://legacy.earlham.edu/~peters/fos/timeline.htm http://arxiv.org/abs/1003.2551 http://dx.doi.org/10.3138/jsp.35.3.143 http://dx.doi.org/10.3138/jsp.35.3.143 http://dx.doi.org/10.3402/rlt.v22.23791 http://dx.doi.org/10.3402/rlt.v22.23791 http://dx.doi.org/10.1080/01973762.2012.761117 http://dx.doi.org/10.1080/01973762.2012.761117 http://src-online.ca/index.php/src/article/view/190/367 http://dx.doi.org/10.3402/rlt.v21i0.21274 http://dx.doi.org/10.1080/02601370.2014.985752 http://dx.doi.org/10.1080/02601370.2014.985752 http://dx.doi.org/10.1080/19322909.2013.789333 http://dx.doi.org/10.3402/rlt.v21.21366 http://dx.doi.org/10.1080/15358593.2014.979432 http://dx.doi.org/10.1080/15358593.2014.979432 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/19195/xml http://www.researchinlearningtechnology.net/index.php/rlt/article/view/19195/xml http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 holliman, r. (2010) ‘from analogue to digital scholarship: implications for science communication researchers’, journal of science communication, vol. 9, no. 3, p. c05. hswe, p. (2007) ‘what you don’t know will hurt you: a slavic scholar’s perspective on the practicality, practicability, and practice of digital scholarship’, slavic & east european information resources, vol. 7, no. 4, pp. 3�15. doi: http://dx.doi.org/10.1300/j167v07n04_02 jakubowicz, a. (2007) ‘bridging the mire between e-research and e-publishing for multimedia digital scholarship in the humanities and social sciences: an australian case study’, webology, vol. 4, no. 1, [online] available at: http://www.webology.org/2007/v4n1/a38.html kaltenbrunner, w. (2014) ‘scholarly labour and digital collaboration in literary studies’, social epistemology, vol. 29, no. 2, pp. 207�233. doi: http://dx.doi.org/10.1080/02691728. 2014.907834 kaltenbrunner, w. (2015) ‘infrastructural inversion as a generative resource in digital scholarship’, science as culture, vol. 24, no. 1, pp. 1�23. doi: http://dx.doi.org/10.1080/ 09505431.2014.917621 kiefer, r. s. (2015) ‘digital preservation of scholarly content, focusing on the example of the clockss archive’, insights the uksg journal, vol. 28, no. 1, pp. 91�96. doi: http://dx. doi.org/10.1629/uksg.215 labelle, c., anderson-wilk, m. & emanuel, r. (2011) ‘leveraging new media in the scholarship of engagement: opportunities and incentives’, journal of extension, vol. 49, no. 6, [online] available at: http://www.joe.org/joe/2011december/a3.php llona, e. (2008) ‘what slavic scholars need to know about archiving digital materials in institutional repositories’, slavic & east european information resources, vol. 9, no. 1, pp. 30�42. doi: http://dx.doi.org/10.1080/15228880802104645 manoff, m. (2013) ‘unintended consequences: new materialist perspectives on library technologies and the digital record’, portal: libraries and the academy, vol. 13, no. 3, pp. 273�282. doi: http://dx.doi.org/10.1353/pla.2013.0018 martin, s. (2006) ‘collaboration in electronic scholarly communication: new possibilities for old books’, journal of the association for history and computing, vol. 9, no. 2. najmi, a. & keralis, s. d. c. (2013) ‘engaging the twitter backchannel as digital scholarship: methods for analyzing scholarly engagement in alternative media’, [online] available at: http://digital.library.unt.edu/ark:/67531/metadc159534/citation/ nowviskie, b. (2007) ‘a scholar’s guide to research, collaboration, and publication in nines’, romanticism and victorianism on the net, no. 47. doi: http://dx.doi.org/10.7202/016707ar pasquini, l. a., wakefield, j. s. & roman, t. (2014) ‘impact factor: early career research & digital scholarship’, techtrends, vol. 58, no. 6, pp. 12�13. doi: http://dx.doi.org/10.1007/ s11528-014-0797-7 pearce, n. (2010) ‘a study of technology adoption by researchers’, information, communication & society, vol. 13, no. 8, pp. 1191�1206. doi: http://dx.doi.org/10.1080/13691181003663601 pfannenschmidt, s. l. & clement, t. e. (2014) ‘evaluating digital scholarship: suggestions and strategies for the text encoding initiative’, journal of the text encoding initiative, no. 7. doi: http://dx.doi.org/10.4000/jtei.949 quan-haase, a., suarez, j. l. & brown, d. m. (2014) ‘collaborating, connecting, and clustering in the humanities: a case study of networked scholarship in an interdisciplinary, dispersed team’, american behavioral scientist, vol. 59, no. 5, pp. 565�581. doi: http://dx. doi.org/10.1177/0002764214556806 quigley, d. s., et al., (2013) ‘scholarship and digital publications: where research meets innovative technology’, visual resources, vol. 29, no. 1�2, pp. 97�106. doi: http://dx.doi. org/10.1080/01973762.2013.761122 romero-frı́as, e. & del-barrio-garcı́a, s. (2014) ‘una visión de las humanidades digitales a través de sus centros’, el profesional de la informacion, vol. 23, no. 5, pp. 485�492. doi: http://dx.doi.org/10.3145/epi.2014.sep.05 scanlon, e. (2011) ‘digital futures: changes in scholarship, open educational resources and the inevitability of interdisciplinarity’, arts and humanities in higher education, vol. 11, no. 1�2, pp. 177�184. doi: http://dx.doi.org/10.1177/1474022211429279 scanlon, e. (2014) ‘scholarship in the digital age: open educational resources, publication and public engagement’, british journal of educational technology, vol. 45, no. 1, pp. 12�23. doi: http://dx.doi.org/10.1111/bjet.12010 j.e. raffaghelli et al. 16 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 http://dx.doi.org/10.1300/j167v07n04_02 http://www.webology.org/2007/v4n1/a38.html http://dx.doi.org/10.1080/02691728.2014.907834 http://dx.doi.org/10.1080/02691728.2014.907834 http://dx.doi.org/10.1080/09505431.2014.917621 http://dx.doi.org/10.1080/09505431.2014.917621 http://dx.doi.org/10.1629/uksg.215 http://dx.doi.org/10.1629/uksg.215 http://www.joe.org/joe/2011december/a3.php http://dx.doi.org/10.1080/15228880802104645 http://dx.doi.org/10.1353/pla.2013.0018 http://digital.library.unt.edu/ark:/67531/metadc159534/citation/ http://dx.doi.org/10.7202/016707ar http://dx.doi.org/10.1007/s11528-014-0797-7 http://dx.doi.org/10.1007/s11528-014-0797-7 http://dx.doi.org/10.1080/13691181003663601 http://dx.doi.org/10.4000/jtei.949 http://dx.doi.org/10.1177/0002764214556806 http://dx.doi.org/10.1177/0002764214556806 http://dx.doi.org/10.1080/01973762.2013.761122 http://dx.doi.org/10.1080/01973762.2013.761122 http://dx.doi.org/10.3145/epi.2014.sep.05 http://dx.doi.org/10.1177/1474022211429279 http://dx.doi.org/10.1111/bjet.12010 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 stafne, e. t. (2013) ‘a view of digital scholarship in extension’, journal of extension, vol. 51, no. 5. stewart, b. e. (2015) ‘in abundance: networked participatory practices as scholarship’, the international review of research in open and distributed learning, vol. 6, no. 3, pp. 318�340. thomas, w. j. (2014) ‘the structure of scholarly communications within academic libraries’, serials review, vol. 39, no. 3, pp. 167�171. doi: http://dx.doi.org/10.1080/00987913.2013. 10766387 trehub, a. (2009) ‘slavic studies and slavic librarianship in the united states: a post-cold war perspective (excerpts)’, slavic & east european information resources, vol. 10, no. 2, pp. 158�169. doi: http://dx.doi.org/10.1080/15228880903044906 veletsianos, g. (2012) ‘higher education scholars’ participation and practices on twitter’, journal of computer assisted learning, vol. 28, no. 4, pp. 336�349. doi: http://dx.doi.org/ 10.1111/j.1365-2729.2011.00449.x veletsianos, g. & kimmons, r. (2012a) ‘networked participatory scholarship: emergent techno-cultural pressures toward open and digital scholarship in online networks’, computers & education, vol. 58, no. 2, pp. 766�774. doi: http://dx.doi.org/10.1016/j. compedu.2011.10.001 veletsianos, g. & kimmons, r. (2012b) ‘assumptions and challenges of open scholarship’, the international review of research in open and distance learning, vol. 13, no. 4, pp. 166�189. vilar, p., juznic, p. & bartol, t. (2015) ‘information-seeking behaviour of slovenian researchers: implications for information services’, international journal on grey literature, vol. 8, no. 1, pp. 43�53. vinopal, j. & mccormick, m. (2013) ‘supporting digital scholarship in research libraries: scalability and sustainability’, journal of library administration, vol. 53, no. 1, pp. 27�42. doi: http://dx.doi.org/10.1080/01930826.2013.756689 weller, m. (2012) ‘digital scholarship and the tenure process as an indicator of change in universities’, rusc’, revista de universidad y sociedad del conocimiento, vol. 9, no. 2, pp. 347�360. doi: http://dx.doi.org/10.7238/rusc.v9i2.1398 wolski, m. & richardson, j. (2014) ‘a model for institutional infrastructure to support digital scholarship’, publications, vol. 2, no. 4, pp. 83�99. doi: http://dx.doi.org/10.3390/ publications2040083 zhao, l. (2014) ‘riding the wave of open access: providing library research support for scholarly publishing literacy’, australian academic & research libraries, vol. 45, no. 1, pp. 3�18. doi: http://dx.doi.org/10.1080/00048623.2014.882873 zorich, d. m. (2013) ‘digital art history: a community assessment’, visual resources, vol. 29, no. 1�2, pp. 14�21. doi: http://dx.doi.org/10.1080/01973762.2013.761108 research in learning technology citation: research in learning technology 2016, 24: 32036 http://dx.doi.org/10.3402/rlt.v24.32036 17 (page number not for citation purpose) http://dx.doi.org/10.1080/00987913.2013.10766387 http://dx.doi.org/10.1080/00987913.2013.10766387 http://dx.doi.org/10.1080/15228880903044906 http://dx.doi.org/10.1111/j.1365-2729.2011.00449.x http://dx.doi.org/10.1111/j.1365-2729.2011.00449.x http://dx.doi.org/10.1016/j.compedu.2011.10.001 http://dx.doi.org/10.1016/j.compedu.2011.10.001 http://dx.doi.org/10.1080/01930826.2013.756689 http://dx.doi.org/10.7238/rusc.v9i2.1398 http://dx.doi.org/10.3390/publications2040083 http://dx.doi.org/10.3390/publications2040083 http://dx.doi.org/10.1080/00048623.2014.882873 http://dx.doi.org/10.1080/01973762.2013.761108 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32036 http://dx.doi.org/10.3402/rlt.v24.32036 doi:10.3402/rlt.v24.31818 original research article two-way text messaging: an interactive mobile learning environment in higher education h.k. salinda premadasa a * and r.g.n. meegama b a centre for computer studies, sabaragamuwa university of sri lanka, belihuloya, sri lanka; b department of computer science, faculty of applied sciences, university of sri jayewardenepura, nugegoda, sri lanka (received 2 april 2016; final version received 10 october 2016) short messaging service (sms) is perhaps the most popular mobile technology prevalent among students in higher education due to its ubiquitous nature and the capability of two-way communication. however, a major limitation in two-way text messaging is sending back a part of received data with the reply message. this limitation results in users of a mobile learning environment being unable to reply back to the correct destination. this article presents a two-way text messaging system that can be integrated into a learning management system (lms) to provide an interactive learning experience to the user community. initially, a database is integrated into the lms that holds message information such as recipient’s phone number, message body and user data header. a specific port associated with the sms is used to conceal and exchange data of a particular course unit. subsequently, software in the student’s mobile device captures this message and sends back the reply message to the appropriate course unit allowing both teachers and students to view messages sent and replies received pertaining to a particular course. results indicate the educational impact of the proposed system in improving the learning environment and benefits it offers to the community in a campus-wide implementation. keywords: sms; udh; pdu; lms; concealed data introduction as teachers are constantly engaged in various academic activities such as preparing learning materials, quizzes, assignments, group discussions, and news forums to organise a better learning environment, supporting them in such endeavours is an indirect form of sustaining a student’s learning ability (gaudioso, olmo, and montero 2012). a teacher can use a mobile-based authoring tool to create an interactive learning environment using a mobile phone to teach theories and obtain assessments. such authoring tools assist the teachers to evaluate students’ performance, record and monitor academic progress, provide advice and communicate with the students in a class (virvou and alepis 2014). moreover, the mobility and instant communication capabilities of mobile devices are the two most important factors affecting the learning process in accessing information, transferring data and interacting among the users in real time (shih, chu, and hwang 2011). *corresponding author. email: salinda@ccs.sab.ac.lk responsible editor: amanda jefferies, university of hertfordshire, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 h.k. salinda premadasa and r.g.n. meegama. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 as the motivation and the pressure are considered to be major factors that impact education and learning in higher educational institutions, the combination of mobile technology and internet-based services such as short messaging service (sms)� email or sms�online forums will certainly enhance the motivation of students’ learning without causing undue stress (lai et al. 2014). activity-based learning theories associated with mobile technology list out six broad areas in new learning and teaching practices, namely, learning and teaching support, behaviourist, informal and lifelong, constructivist, situated and collaborative (jonassen and land 2012). acquiring and applying knowledge continuously is a lifelong learning process that can occur at any time anywhere. students’ assessment and grading mechanism are vivid goals that should be embedded in the learning process for them to pursue knowledge in lifelong learning (arrigo et al. 2013). establishing a conversation between different knowledge systems can be referred to as conversational learning in which people practice learning via questioning, answering and sharing the knowledge among each other (sharples, taylor, and vavoula 2010). as the present mobile technology, with its enhanced technical capabilities, can make a profound impact on learning, the combination of mobile technology and learning theories sprouts ubiquitous learning environments. as a result, the concept of learning is shifting towards learners’ environment from the classroom and as such, seamless integration of mobile technology to learner-centred education has become a challenging task that requires a thorough understanding of both theoretical and technical aspects of mobile learning (jonassen and land 2012). unfortunately, the best practices to be followed when using mobile technology in a learning environment are still not defined at all. in this context, a systematic investigation of students’ and teachers’ experiences in using a mobile learning platform will definitely provide us an opportunity to overcome the challenges encountered in a classroom setup (lópez-pérez, pérez-lópez, and rodrı́guez-ariza 2011). sms is one of the best qualitative mediating tools adopted in the arena of mobile communication pedagogically, economically and technologically in the past decade (moura and carvalho 2010). recent reports support this claim in that more than 6.6 trillion text messages were sent out over carrier networks worldwide in the year 2010 (whitfield 2010). the use of mobile technologies for distance education has become a popular tool in higher education across the globe due to its ever-increasing demand for student-centric learning (brett 2011). in this scenario, an sms can be established for university students as a supportive tool to capture students’ retention in the existing learning management system (lms) that provides a blended learning environment (wanja 2014). sms-based tools can be implemented in an in-class or after-class environment to encourage students’ interactivity. such a tool will be of immense assistance to teachers to view messages and form interactive message loops verbally with the students throughout the classroom and also during online sessions done after class (lauricella and kay 2013). a majority of university students use mobile phones for routine voice and message communication. consequently, sms can be used as a trusted technology to form a better learning environment having a huge potential in higher education with active (sending a message) and passive (reading a message) interactions (moura and carvalho 2010). a mobile platform, referred to as university mobile learning environment (premadasa and meegama 2013), has the ability to offer its services to a larger segment of users than existing web-based learning environments via mobile devices. although there are number of web-based paid text message services available in the h.k. salinda premadasa and r.g.n. meegama 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 internet at present that facilitate sending secure messages through an lms, such paid services restrict the number of outgoing messages as well as the number of users (dewitt, alias, and siraj 2013; ismail and azizan 2012). teachers who work with this lms can send messages to a relevant course unit so that students who are currently logged in will receive this message on their mobile devices. the two-way text message interaction is complete if the student who receives a message relevant to a particular course is able to send back a reply to the same course unit from where the message was originated. a standard text message does not fully support two-way texting when there is a necessity to include a part of the message received in the reply body. this has hampered two-way communication within a mobile community when one entity uses a mobile gateway (aranyász 2012) to send out messages using a unique gsm number shared by all the users. this article presents a two-way text messaging system by concealing data in the message body using the user data header (udh) to include a part of the message received in the reply body while an encoding and a decoding technique in the mobile phone is used in the background to support such a texting mechanism. the proposed novel mechanism creates a campus-wide interactive mobile learning environment for teachers and students to obtain students’ assessments and grading process. the proposed system is a combination of java 2 micro edition (j2me) (belvin 2011; ericsson 2011) and personal home page (php) technologies. the reason for using such emerging technologies is that the php and j2me combination facilitates integrating the proposed framework to a widely used lms (such as moodle) (pearce 2011). related work khan and beg (khan and beg 2011a) have presented a methodology where stereo images can be sent through a number of consecutive short messages over a gsm network. a stereo image is converted into characters and these characters are set as the payload text of the short message. this payload text is sent as a concatenated short message on a specified port of the mobile device. a method concerning multiple-choice quizzing over concatenated messages is revealed by galih, and sodiq (2011) where students receive questions via short messages and after answering these questions, the results can be obtained in the client side. in this research, the authors have used steganography in a picture message to hide quiz answers. there are apparently two programs running on instructor-side and student-side where the instructor prepares and sends quizzes over messages and answers over steganography picture message. the student in turn takes the quiz, answers it and sends answers back to the instructor. chung, hung, and chow (2014) proposed two integrated sms frameworks in their blended learning environment to provide an interactive e-learning facilities among lectures and students. one framework is sms which is for homework, references and class events. the other sms framework is for creating pop-up quiz and short-answered examinations. the open university malaysia has implemented such a messaging system to prepare text-based quizzes for students (ismail and azizan 2012) where a client messaging server is connected via a gsm modem with related software to administer these text-based quizzes. khalid, sujan, and haque (2011) have also implemented a similar message-based system in bangladesh to attend text-based multiple-choice questions (mcq). in contrast, our proposed system is a secure, open source short research in learning technology citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 messaging system that can be easily integrated into the moodle lms for student assessment where the cost involved is only for sending messages. in recent literature, as presented in the previous section, applications were developed based on concatenation of messages and also complex data conversion techniques mainly to send voice, image and animation as well as to perform mcq tests only between two mobile devices where large amounts of messages are required, thereby increasing the cost of implementation. our proposed system is capable of achieving the same functionality of previous methods with a minimum number of messages exchanged between the communicating devices using a specific port number with associated software to exchange necessary information as concealed data. for this purpose, an open source sms gateway, a messaging daemon and specific port associated software in a mobile device are used. the system is fully functional at any time anywhere between both parties (a teacher and a student) in a secure, cost-effective and time-efficient manner, providing a mobile learning environment for assessing students’ work. theoretical framework it is traditionally known that short message service centre (smsc) performs a ‘store and forward’ mechanism for sending and receiving messages (ryan 2011). the mobile originated (mo)-sms is transported from a mobile station to the smsc where these mo-smss may be destined to other mobile stations or other services. in contrast, the mobile terminated (mt) sms is transported from an smsc to a mobile station. the mo-sms is transferred from a mobile phone to smsc by composing it with smssubmit protocol description unit (pdu) format and similarly, an mt-sms is received at a mobile phone in sms-deliver pdu format from the smsc (golde 2011). sms-submit pdu as depicted in figure 1, the sms-submit pdu can be expressed with values as with service centre address and the rest of transport protocol data unit (tpdu). the pdu format is a hexadecimal encoded binary format, which means that two hexadecimal digits represent a byte of data. a header with control information and user data make up a tpdu (butau 2010). user data header as illustrated in figure 2, the udh is divided into two parts, the first part, which is only one byte in length, is identified as a user data header length (udhl) of the udh. following this udhl, an information element (ie) instructs the sms-submit pdu 069149170000f3 41000a8170416998870000a71506 0504c350000054747a0e4ac f416110945805b5cbf379f85c06 transport protocol data unit (tpdu) service center address (sca) figure 1. sms-submit protocol description unit (pdu) with values. h.k. salinda premadasa and r.g.n. meegama 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 mobile phone to perform some action based on the information received. the common information element identifiers are port number addressing (representing byte 05) and message concatenation (representing byte 00). each port number is addressed to a specific port in the phone that provides clear instruction on putting together two or more messages to make a single message during concatenation of several messages. the second byte of the ie is the information element data length that tells the phone how many of the following bytes are part of the information identifier. the subsequent bytes of the ie are the actual information being conveyed and is referred to as the information element data (golde 2011; ryan 2011). port number addressing port number addressing can be used to deliver a particular message to an application running on a mobile phone where in most cases, these applications are developed using java technology (j2me). such an application with a port addressing technique allows us to specify a source port and a destination port number for a particular message in the udh resulting in port number information being transmitted through the udh of the message (ozeki 2000). table 1 illustrates the fields of the udh with port number addressing (golde 2011; ryan 2011). 140 bytes user data header (udh) udhl information element (ie) iei iedl port number addressing or message concatenation ied rest of message content figure 2. the architecture of a message with user data header (udh). table 1. fields of user data header (udh). fields description 06 user data header length � udhl (6 octets contained in this udh) port number addressing 05 iei: application port addressing scheme (16-bit address) 04 information element data length (iedl): indicates the number of fields in udh c350 mobile handset destination port (port 50000) 0000 originating port (port 0) research in learning technology citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 system function and architecture as illustrated in figure 3, the proposed system enables a teacher to post messages to students, to establish an assignment session via a mobile browser interface through an internet-enabled mobile device. once a teacher logs in to the system, a message relevant to a particular course (each course has a course code) is created with a maximum of 153 characters (this is a standard character length for a single text message with a udh). thereafter, this message is inserted into a database where an sms daemon checks the database periodically and sends out stored messages through an sms gateway to a gsm modem. the modem in turn sends out sms messages to all the recipients with access to that particular course unit through the smsc. the course code, student number, time allocated for the assignment and the character length allowed of the answer are wrapped automatically by the system. the information about the course code is extremely important for the students to identify from which course the message is originating from as several courses are available in the lms and, moreover, a student may receive multiple messages from each course unit. the student number is also important to the teacher for identification purposes. a student may receive this message-based assignment via specific port-associated software installed in the mobile phone. a student can accept this assignment and generate an answer depending on the time duration and character length defined by the teacher. while the answer is sent back to the system as concatenated messages (if the answer is more than 153 characters), the course code and the student number are sent as concealed data by wrapping them inside the message body itself. the theoretical concept behind this system functionality is built upon the convention of port number addressing and the udh of the sms pdu. in the standard format, in contrast to an email message, a text message cannot enclose the full or part of the original text message in the reply text. the mobile device only recognises the sender’s phone number, which is displayed in the reply message. as such, it is required to wrap information about the course code, student number, time allocation for the assignment and the character length of the answer within the teacher university wifi zone login authentication and inserting text message students (assignment session) internet login authentication, course information and message database e-learning server (moodle) with mysql database sms gateway and smsdaemon gsm modem connected via usb sms center text message text message figure 3. the architecture of the proposed system. h.k. salinda premadasa and r.g.n. meegama 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 message, and this piece of information should come back to the system with the reply message to be categorised into different course units as well as user identification. the port number addressing technique with a destination port number of the udh can be used to perform this concealment of additional data in the message. such a text message is not directly saved in the standard inbox of the mobile device. rather, the mobile application software installed in the recipient’s phone extracts this message having a specially defined destination port number. course unit details (say com321), student number (say ea1234), time allocated for the assignment (say 5 minutes) and the character length of the answer (say 459 characters �3 messages) are added automatically by the system to the body of the message to be identified at the recipient’s end. based on this information, for example, an assignment containing ‘question #1: what do you mean by subnetting in network?’, can be expressed as in figure 4 where the sms-submit pdu converts a message to a pdu format using the port number addressing (destination port 50,000 is converted to c350 in hex). this will be the pdu format of the message that is sent from the database by the daemon. details pertaining to the course unit are compared with the database to acquire a unique course code of the relevant course unit. while this function occurs in the background, the course code, student number, time allocated to the assignment and the character length of the answer are appended to the message body by the system. at the next stage, the complete sms-submit pdu is sent to the smsc to be forwarded to the student within a valid time period. the student then generates an answer as a reply message with the course unit details (i.e. com321) with the student number (i.e. ea1234) and sends it back to the database. this is done by the j2me application installed in the mobile device, which captures the incoming message and acquires the course unit information (i.e. com321) and the student number (i.e. ea1234) contained in the message body. while the student is ready to accept an assignment and create an answer as a reply message, the application software automatically conceals the information of course unit and student number securely in com321 assignment time: 5 type message question #1:what do you mean by subnetting in networking? ea1234 ea1235 ea1236 ea1237 answer with 3 insert insert sms-based assignment to the database by the teacher 069149170000f341000a 8170416998870000a74b 6773268b158331d98c56 a3d572d17a794e4fbf dda0514cd7025dd161 3a88fc06e5df7550bb1 c7683c479d0bc2e7697 e9f4b4fb0c4abb41ee 32fdfe96afd3eef30f message converted to the pdu format by using the port number addressing (destination port 50000 converted as c350 in hex) of udh and wrap up relevant information to the message body accept cancel message received by the student (displaying relevant information) com321 assignment you have only 5 min more for online assignment. do you accept this sms based assignment? ouestion #1:what do you mean by subnetting in networking? 060504c3500000c3 messages min figure 4. the theoretical concept behind the system functionality. research in learning technology citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 the background to the reply. the student, however, only sees the phone number of the message originator as well as the reply text and does not see course unit or student number information. in order to implement the proposed framework, it is essential to have mobile devices that support the udh concept. when the mobile application is installed, another message inbox, in addition to the standard inbox, appears in the menu. all the messages originating from the proposed system are delivered to this newly created inbox. figure 5 depicts the interfaces of the mobile phone giving different functionalities of the proposed system. results and discussion students’ feedback is a vital factor in evaluating the efficiency, effectiveness, educational impact and benefits of the proposed system in an academic environment. two groups of students, from sciences and non-sciences, familiar with traditional paper-based assessment method, were asked to fill out two questionnaires within a realistic timeframe according to preand post-usage of the proposed system. initially, students in a class answered a question posed by a teacher who then marked answer scripts and gave grades to each script. at first, a pre-usage questionnaire was given to them after completing a class-room-based learning process with the teacher. subsequently, the teacher was given an introduction to the proposed system who posed the same question to the same batch of students via a text message. students, in turn, replied back with the answers for assessment by the teacher. later, students received their grades immediately through text messages. a few common problems encountered during class room assessments in a university environment can be summarised as follows: (1) students may have to come from a faraway distance to the class room on time to attend assessments (2) student attendance (3) delay in receiving grade of the assignment the analysis was carried out using both samples as science (50 students) and nonscience (50 students) separately at first and, finally, as an overall sample (100 students). type reply subnetting is a method for getting the most out of the iimited 32-bit lpv4 addressing space and reducing the size of the routing tables in a large internetwork. type reply subnetting is a method for getting the most out of the iimited 32-bit lpv4 addressing space and reducing the size of the routing tables in a large internetwork. question #1:what do you mean by subnetting in networking? midlet 14:56 midlet mlearn@susl (299) 14:40 midlet 14:45 midlet 15:5515:40 com321 assignment (a) (b) (c) (d) (e) accept replycancel cancel cancelok ok ok you have 5 minutes more for online assignment. do you accecpt this sms based assignment> com321 assignment index number= ea1234>>> your grade= b+>>> mlearn@susl you have another 1 minute more! figure 5. the interface of the mobile phone with (a) message received, (b) question received, (c) answer typed, (d) alert and (e) grade received. h.k. salinda premadasa and r.g.n. meegama 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 a score value ( �10 �strongly disagreed, �5 �disagreed, 0 �neutral, 5 �agreed, 10 �strongly agreed) was assigned to each option in the list of answers at the preand post-usage survey. preand post-usage intervention surveys pre-usage survey results both groups of students were questioned to identify the usage of mobile technology for education prior to their admission to the university. at the high school level, a majority of them (84%) has used mobile phones mainly for voice calling and sending text messages, but hardly for educational activities. however, this mobile technology usage among the students has increased to 96% once they are enrolled at the university. thus, it reveals that the social preference of mobile phone usage among university students when compared with their pre-university education. student evaluation process is one of the most important tasks that can be used to motivate students in their learning endeavours. evaluation and subsequent grading of quizzes, assignments and examinations should be performed without any undue delays. in this scenario, teachers reported that they need approximately 2�5 days to complete the evaluation and grading process of a group having 100 students who submitted a paper-based assessment. initially, the teachers evaluate each answer script and assign a grade to each. then, the grades of each student is printed on a pre-prepared mark sheet and placed on a notice board for display. the pre-usage survey results indicate that only 13% of the students agreed with the paper-based evaluation and grading process and they had clearly indicated long delays in receiving grades. moreover, student attendance during compulsory assessments is always a critical issue with traditional paper-based method where students give various unacceptable excuses for being absent. during this survey, 83% of the students were present for the paper-based assessment while the remaining 17% (absentees) had given various reasons for their absence. the teachers who were engaged in this survey reported that some reasons given by the students for being absent are totally unacceptable. according to prevailing regulations in the university, teachers need to prepare another follow-up assessment (referred to as a ‘make up’ assessment) for the absentees. this places a further burden on the teachers who are presently loaded with heavy academic work and research. the secondary data collected during the past 3 years show that the attendance of such compulsory assessments is 81%, 85% and 83%, relatively somewhat equal in each year. post-usage survey results the paired sample t-test (parametric) is carried out to analyse the acceptance of the proposed mechanism by comparing data obtained during preand post-usage intervention surveys with a same sample set of students. the post-usage survey is mainly focused on observing the usage and the satisfaction level of the end users of the proposed mechanism. after defining a null hypothesis (h0), a paired sample t-test (parametric) is applied to compare the two final score values. h0 was accepted by considering the p-value of the test (i.e. if p b0.05, then reject h0 or accept otherwise). assume a null hypothesis h0 as: research in learning technology citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 h0 �students are not satisfied with the sms-based assessment mechanism. table 2 provides statistical results obtained after applying the paired sample t-test. questions focused on preand post-surveys are verified using a five-point likert scale as shown in table 2 in which for the overall sample as well as for each specific sample, the related p-value was b0.05. this results in h0 being rejected and h1 accepted. these results verify the students’ satisfaction in using the proposed mechanism for assessments is significantly higher than traditional paper-based method. the post-usage survey data are plotted in consistent with a normal distribution curve as shown in figure 6. the median value can be used to describe the data set by variation in the interval for the median between 0.0000 and 5.0000 with a 95% level of confidence. these post-usage survey results indicate the students the acceptance of the proposed sms-based assessment mechanism. however, it is observed a 96% attendance when the assessment is conducted using the proposed mechanism. the absentees (4%) had honestly reported that they had not received text messages containing the assignment because they are unaware that this text message is saved in a separate message box. while a significant percentage (96%) was present in the class during the mobile-based assessment, 3.125% (three students) did not complete the assignment due to delays in responding (for instance, if a student is unable to reply back an answer within a given time frame, the software in the mobile device automatically discards the reply message). this means the student had accepted the assignment but did not complete it. these drawbacks could be mitigated once the students get themselves familiarised in using the proposed system with time management skills. user-friendliness and cost-effectiveness of the proposed system the following crucial dimensions were used to evaluate the efficiency and effectiveness of proposed system. they are: (1) user-friendliness (2) cost-effectiveness table 3 illustrates the post-questionnaire, providing three questions on each of the above two dimensions. the students were asked whether it was convenient to use mobile devices with the proposed method. the responses were satisfactory with an overall mean score of 6.10 in agreement (user-friendliness � question 1 as in table 3). in this trial, table 2. results of paired sample t-test for proposed mechanism. sample variable n mean standard deviation p applied science pre 50 4.95000 4.17994 0.023 post 50 3.20000 4.81918 difference 50 1.75000 6.02397 social science pre 50 3.45000 5.07319 0.039 post 50 1.30000 6.60936 difference 50 2.15000 8.42146 overall pre 100 4.20000 4.68557 0.004 post 100 2.25000 5.83333 difference 100 1.95000 7.28722 h.k. salinda premadasa and r.g.n. meegama 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 participants from the non-science stream accounted for a mean of 5.50 while it was 6.70 for the science students. this can be attributed to the fact former group’s unfamiliarity with mobile device while using the proposed system. however, when taking into account the overall mean score, the participants had responded in the affirmative. a high mean score of 7.40 from non-science students indicates a satisfactory response with this proposed technique rather than the science students who scored a mean of 5.30 for the second question for user-friendliness as in table 3. the students did not wish to participate in conventional face-to-face interactions with the teacher, as the proposed technique would have given them a novel experience in their learning environment. the science students, however, had given several uncertain responses to this question which may have been due to the familiarity of such technological aspects in learning. this fact was later verified during follow-up discussions with the students. –8 –4 0 4 8 95% confidence intervals mean median 0 1 2 3 4 5 anderson-darling normality test a-squared p-value < mean stdev variance skewness kurtosis n 100 –0.205749 –0.675206 5.8333 2.2500 0.005 4.77 minimum –10.0000 0.0000 5.0000 5.0000 10.0000 1st quartile median 3rd quartile maximum 95% confidence interval for mean 95% confidence interval for median 95% confidence interval for stdev 1.0925 3.4075 0.0000 5.0000 5.1217 6.7764 34.0278 figure 6. distribution of post-usage survey data to evaluate the reliability of the proposed mechanism. table 3. questions relevant to each dimension. user-friendliness (1) it is easy to use mobile devices in this proposed method. (2) this text-message-based assessment experience is encouraging. (3) i would like to obtain assignments continuously with the proposed mechanism instead of the traditional method. (4) given instructions in the assignment process are formative and easy to understand. (5) i would recommend this technique as a method of assessment to other students as well as teachers. cost-effectiveness (1) proposed system is accessible with existing mobile devices. (2) cost per text message is affordable in replying back with answers. (3) other related costs can be minimised due to anytime anywhere accessibility. research in learning technology citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 in the third question in the scale of user-friendliness as given in table 3, the participants were asked whether they would like to continue their assignments through the proposed system instead of the traditional method. a high mean score of 7.1 from science students’ indicates a satisfactory response to continuing the proposed system in future for assignments instead of traditional method while non-science scored a mean 4.9. however, when considering an overall mean score of 6.0, the participants somewhat agreed to continue their assignments with the proposed system than the traditional methods. this result differentiate the impact of technology acceptance and interest among science and non-science students in their educational activities. when made an after discussion with two groups, most of non-science students responded that the proposed mechanism is an extra burden into the mind-set on their assessments because they need to pay attention for the technological aspects of the proposed system than paper-based method. however, most of the students in science group responded that the proposed mechanism is much more effective on their tough learning schedule. the high mean score 7.25 from the overall students indicates the satisfactory response to the formative and understandable instructions of the entire assignment process when using the mobile devices. especially non-science students also scored a mean value of 6.7, while science students scored a mean value of 7.8. in the later discussion made with two groups, students mentioned that the given instructions in the assignment process are clear, formative and easy to understand. the reason behind this high value of mean scores is the acceptability of given instructions in the proposed mechanism. in the fifth question associated with user-friendliness as given in table 3, the participants were asked whether they would recommend the proposed technique as a method of assessment to other students as well as teachers. fourteen (14%) of the participants in both groups were uncertain in responding to this question as this was the first time that many (78%) of them ever had an experience with a mobile device in a learning environment. also being not exposed to such an assessment method in a previous occasion may have influenced this result and, hence, a mean score of almost 5.00 is observed in both groups, individually as well as when taken as an overall sample. however, when considering an overall mean score of 5.55, the participants somewhat agreed in recommending the proposed system as an assessment method in a learning environment. this indicates that social attitudes among the students need to be changed in using such novel technologies and also further improvements are required in the proposed mechanism to ensure a wide acceptance in pursuing mobile-based assessments. the proposed system is accessible by a wide range of mobile devices into which specific software associated with port number addressing is installed. presently, the students possess mobile devices with sufficient technology to execute the proposed assessment mechanism and, therefore, they entirely agreed with the system’s accessibility using existing devices (the first question in cost-effectiveness as in table 3 with an overall mean score of 7.85). the length of a text message (answer) sent by a student during an assessment is a vital factor affecting the affordability of the mechanism. presently, the teacher limits the number of text messages that can be sent by a student for a particular assessment to three (this can be varied). as the cost in sending a text message is considerably low (around us$ 0.002), replying back the answers with a maximum of three text messages is affordable to the students. an overall mean score of 6.05 for the second h.k. salinda premadasa and r.g.n. meegama 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 question in cost-effectiveness as given in table 3 indicates the students’ agreement with the cost per text message to be paid to the service provider. with the proposed mechanism, the students and teachers do not have to personally come to a class room, which take up time and involves a cost as it requires travel through a mountainous region. as such, this mechanism is ideal to cut down expenses incurred by both students and teachers especially during travel. this fact generated an entire agreement with the proposed procedure as depicted by the third question in cost-effectiveness given in table 3 with an overall mean score of 7.70. the responses given for the three questions related to cost-effectiveness given in table 3 highlight the enormous potential of the proposed system in minimising the cost of an assessment. not only the cost of a text message, but also the university as a whole, is able to save costs associated with stationery, electricity, water, janitorial services, etc. that are required to conduct a class-room-based assessment. figure 7 depicts the mean scores on user-friendliness and the cost-effectiveness of the proposed system. educational impact of the proposed system the university in which the proposed system is implemented spans a wide geographical area containing mountainous regions with uneven weather conditions throughout the year. also, although all the students are given off-campus residential facilities by the university, the residence halls are located at faraway distances from the university. hence, the students need to come a long way to the class room either by bus, train, taxi or even walking to attend assessments. different schools in the university too are dispersed within a vast geographical space having valleys and ridges and students, especially students with physical disabilities, find it extremely difficult to personally meet instructors on a regular and daily basis. this is further aggravated as the university has still not developed sufficient infrastructure to accommodate students and staff with such physical disabilities. students who were engaged in the pre-usage survey reported this arduous task of being present on time for assignments and the resulting waste of transit time for travelling on a daily basis. only 24% of the students responded that they did not come across such difficulties before using 10.00 5.00 0.00 –5.00 –10.00 science non-science overall user friendliness q1 q2 q3 q1 q2 q3 cost effectiveness 6.70 5.50 6.10 5.30 7.40 6.35 5.30 5.80 5.55 8.10 7.60 7.85 5.90 6.20 6.05 7.80 7.60 7.70 m e a n s co re o n l ik e rt s ca le figure 7. the mean score on likert scale for the user-friendliness and cost-effectiveness. research in learning technology citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 the proposed system. the post-usage survey questionnaire revealed that 81% of the respondents agreed to use the proposed system as a convenient assessment mechanism. as reported by them, the earlier time wasted for travelling is now utilised for academic preparations, coursework, revision and research. there were two (2%) students having physical disabilities in the non-science streams, and the proposed mechanism has really benefitted their learning experience as they can now sit for classroom assessments from residences without having to toil along hilly roads within the university. the post-usage survey results indicate that the students’ attendance for continuous assessments has significantly increased after implementing the proposed mobile-based mechanism and most probably it may be possible to achieve a 100% attendance for these assessments in the near future. moreover, teachers now have to calculate grades for fewer numbers of make-up assessments conducted for the absentees thereby, saving lot of time and energy that could be utilised for other research work. the evaluation process of the proposed system allows teachers to log on, retrieve answers given by each student and insert relevant grades to each of them. then, the system automatically generates text messages destined for each student to notify assignment grades. according to the post-usage survey data, 90% of the students agreed with the proposed system as an effective, efficient and a time-saving mechanism. as such, these post-usage survey data clearly show that the students’ encouraging feedback on the outcome of the proposed mechanism in receiving grades of assessments without any delay. students have responded that this mechanism is quick and smart in sending assignment grades on time and that it had really benefitted their learning outcomes as they can now revise on assessments having low grades right away after reception of results. statistical analysis of survey results indicates the impact the proposed mechanism has made on the education and the learning process of a group of students. continuous assessment with an instant grading process is considered important for the students to enhance and revise their knowledge regularly. the proposed technique supports conducting such continuous assessments with instant grading at any time anywhere without making any undue influence on a particular encounter the students are faced with and thereby contributing to lifelong learning. further, the post-usage evaluation conversations generated between the teacher and the student enable them to fill the knowledge gap implying the ability of the proposed mechanism to initiate a conversational learning atmosphere between two different knowledge systems (a teacher and a student), enabling them to acquire knowledge via questioning, answering and sharing. however, the proposed mechanism does not provide a guarantee about the students’ response whether it is an own work or not. it would be a very difficult task to verify who is using the mobile device to complete the assignment in the proposed system environment. hence, the student is required to be honest with the present system. conversely, future research directions will certainly move towards to override this factor and also the smart messaging concept will fill the knowledge gap between teachers and students using such conversational learning. conclusion a novel text messaging technique, integrated into an lms, offering two-way texting is presented in this article. the proposed framework provides a more flexible and convenient way of passing messages between the users than a traditional web-based h.k. salinda premadasa and r.g.n. meegama 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 e-learning platform. the pilot survey carried out with 100 messages reveals performance of the system in sending messages with minimum delay. the system has made a profound impact on improving the learning environment of a campus located in a mountainous region. the system has benefitted both the students and staff, especially the members of this community having physical disabilities who find it difficult to negotiate sharp slopes and ridges encountered when visiting different schools to meet up with peers. the technology reported here would also be useful in any situation where many users must share a single phone number. since almost all students use mobile phones for texting already (escap 2011), the proposed two-way texting mechanism would provide a useful facility in the educational sector. references aranyász, j. (2012) ozeki ng � sms gateway, ozeki informatics, debrecen, hungary. arrigo, m., et al., (2013) ‘meta-analyses from a collaborative project in mobile lifelong learning’, british educational research journal, vol. 39, no. 2, pp. 222�247. belvin, g. (2011) a secure text messaging protocol, johns hopkins university, baltimore, md. brett, p. (2011) ‘students’ experiences and engagement with sms for learning in higher education’, innovations in education and teaching international, vol. 48, no. 2, pp. 137�147. butau, s. (2010) architectures for two-way sms services involving an sdi node, international institute for geo information science and earth observation, faculty of itc, university of twente, netherlands. chung, k. s., hung, v. c. s. & chow, t. p. (2014) ‘the design of economical blended mobile learning with sms’, 28th annual conference, asian association of open universities, the open university of hong kong, china, p. 716. dewitt, d., alias, n. & siraj, s. (2013) ‘online communication: the implementation of the collaborative mlearning science module in a malaysian secondary school’, life science journal, vol. 10, no. 1, pp. 1352�1363. ericsson, s. (2011) developers guidelines, sony ericsson mobile communications, london. escap. (2011) statistical yearbook for asia and pacific � 2011, united nations publication, bangkok, thailand. gaudioso, e., olmo, f. h. & montero, m. (2012) ‘supporting teachers in adaptive educational systems through predictive models: a proof of concept’, expert systems with applications, vol. 39, no. 1, pp. 621�625. golde, n. (2011) sms vulnerability analysis on feature phones, berlin institute of technology, berlin. ismail, i. & azizan, s. n. (2012) ‘distance learners’ needs on interactivity in sms-based learning system’, asian social science, vol. 8, no. 11, pp. 119�128. jonassen, d. & land, s. (2012) theoretical foundations of learning environments, routledge, ny. khalid, m. s., sujan, k. m. & haque, i. (2011) ‘an empirical study on the effect of different technology environments on students’ assessment’, 3rd international conference on machine learning and computing (icmlc 2011), singapore. p. 157�162. khan, m. f. & beg, s. (2011) ‘stereo image transference & retrieval over sms’, journal of computing, vol. 3, no. 7, pp. 1�3. lai, c. h., et al., (2014) ‘sms and emails as class reminders to assist subject learning’, journal of computers, vol. 24, no. 4, pp. 22�30. lauricella, s. & kay, r. (2013) ‘exploring the use of text and instant messaging in higher education classrooms’, research in learning technology, vol. 21, no. 1, pp. 1�17. lópez-pérez, m. v., pérez-lópez, m. c. & rodrı́guez-ariza, l. (2011) ‘blended learning in higher education: students’ perceptions and their relation to outcomes’, computers & education, vol. 56, no. 3, pp. 818�826. moura, a. & carvalho, a. a. (2010) ‘mobile learning: using sms in educational contexts’, in key competencies in the knowledge society, eds. nicholas reynolds & márta turcsányiszabó, springer berlin heidelberg, berlin, pp. 281�291. research in learning technology citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 15 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 ozeki. (2000) product guide [online] available at: http://www.ozekisms.com/index.php? owpn�221 pearce, j. (2011) professional mobile web development with wordpress, joomla and drupal, wiley, india. premadasa, s. & meegama, g. (2013) ‘mobile learning environment with short messaging service: application to a campus environment in a developing country’, campus wide information systems, vol. 30, no. 2, pp. 106�123. rachmaningrum, n., galih, s. & sodiq, m. (2011) ‘mobile quiz application for enhancing anytime anywhere learning activity’, proceedings of global learn 2011, melbourne, australia, pp. 216�222. ryan, m. (2011) system and method for electronic messaging with group sending, receiving, and replying capabilities, 20110165895, 1st edn, communication, enhancement products inc, las vegas, usa. sharples, m., taylor, j. & vavoula, g. (2010) ‘a theory of learning for the mobile age’, in medienbildung in neuen kulturräumen, ed. ben bachmair, springer berlin heidelberg, pp. 87�99. shih, j. l., chu, h. c. & hwang, g. j. (2011) ‘an investigation of attitudes of students and teachers about participating in a context-aware ubiquitous learning activity’, british journal of educational technology, vol. 42, no. 3, pp. 373�394. virvou, m. & alepis, e. (2014) ‘mobile authoring in educational software’, in object-oriented user interfaces for personalized mobile learning, ed. janusz kacprzyk & lakhmi c. jain, springer berlin heidelberg, berlin, pp. 31�46. wanja, s. n. (2014) utilization of mobile phone in enhancing learner support services for distance education programmes: a case of mount kenya university, university of nairobi, nairobi. whitfield, k. (2010) ‘mobile messaging market’, in mobile messaging futures 2010�2014, portio research limited, hertfordshire, uk. h.k. salinda premadasa and r.g.n. meegama 16 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31818 http://dx.doi.org/10.3402/rlt.v24.31818 http://www.ozekisms.com/index.php?owpn=221 http://www.ozekisms.com/index.php?owpn=221 http://www.ozekisms.com/index.php?owpn=221 http://www.ozekisms.com/index.php?owpn=221 http://www.ozekisms.com/index.php?owpn=221 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31818 http://dx.doi.org/10.3402/rlt.v24.31818 (page number not for citation purpose) 1 *corresponding author. email: kamakshi.rajagopal@kuleuven.be research in learning technology 2020. © 2020 k. rajagopal et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 research in learning technology vol. 28, 2020 original research article learner skills in open virtual mobility kamakshi rajagopala,b*, olga firssovac, ilse op de beeckb, elke van der stappenb, slavi stoyanovc, piet henderikxd, ilona bucheme aimec-itec, ku leuven, kortrijk, belgium; beducational development unit, ku leuven, leuven, belgium; cfaculty of educational sciences, open university of the netherlands, heerlen, netherlands; deadtu, netherlands, maastricht; edepartment of economics and social sciences, beuth university of applied sciences, berlin, germany (received: 3 april 2019; revised: 20 december 2019; accepted: 31 january 2020; published: 19 march 2020) internationalisation has been a key theme in higher education (he) for decades. multiple initiatives across the world have contributed to creating offerings of high-quality online education, with collaborations across national borders. two of the concepts that have proved to be influential are virtual mobility (vm) and open education (oe). virtual mobility refers to study activities that students enrolled in he in one country undertake online in other countries without physically moving. such activities are certified and mutually acknowledged by participating institutions. open education covers global initiatives increasing access to free online quality education, without or with alternative forms of certification. the research presented in this article identifies the learner skills and competences that are supported by open virtual mobility (openvm), a new trend in online education that builds upon these two concepts. a group concept mapping study based on the contributions of experts in both vm and oe resulted in defining seven learner skills and competence areas including: intercultural skills and attitudes; networked learning; active self-regulated learner skills; media and digital literacy; autonomy-driven learning, interactive and collaborative learning in an authentic international environment and open-mindedness. the study provided input for further conceptualising of openvm as a bridge between vm and oe. keywords: virtual mobility; open education; open virtual mobility; group concept mapping; learner skills; competence framework introduction internationalisation has been a key theme in higher education (he) for many centuries, from student mobility in the middle ages in europe, to the access to high-quality content from the most innovative american universities in the 21st century. what internationalisation is, and how it can be implemented has been the topic of much discussion in academia (knight 2003, 2004). knight (2003) offers the following working definition of internationalisation: ‘internationalization at the national/ sector/ institutional levels is defined as the process of integrating an international, mailto:kamakshi.rajagopal@kuleuven.be http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v28.2254 k. rajagopal et al. 2 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) intercultural or global dimension into the purpose, functions or delivery of post-secondary education’. (knight 2003, p. 2) internationalisation needs to be distinguished from globalisation (varghese 2018). the globalisation of he views higher educational institutions (heis) as an important group of organisations in the current global educational market. from this perspective, heis are key suppliers of high-quality human resources to a globalised knowledge economic market. as such, globalisation emphasises the economic role and commercial function of heis, whereas internationalisation looks at the socio-cultural aspects of education. internationalisation has taken many forms in implementation, from student exchanges, student and teacher mobility, to cross-border collaborative projects ( varghese 2008, 2018). however, these activities also emerge as an implementation of heis taking up their supplier role in the globalised educational market. often, offering these activities and courses to learners strengthens heis’ reputation as high-quality human resource and knowledge-developing institutions. internationalisation has been developing on the back of three disruptive drivers currently running through the global educational landscape: increased possibilities through digitalisation (flavin and quintero 2018), increased interest in global collaboration at the individual and institutional levels (blight, davis and olsen 1999; nascimbeni et al., 2018; ryan et al. 2017; van tryon, mcdonald & hirumi 2018) and the drive to openness to widen access to he (wiley 2010; wiley and hilton iii 2009). each of these drivers creates new opportunities for learners, requiring the development of new learner skills and competences and at the same time stimulating it. in this article, we will focus on the specific context of europe, with its transnational cooperation under the auspices of the european union. the european strategy on he is captured in the concept of the european higher education area (ehea) and the bologna process (bologna declaration 1999). through the bologna process, 48 european and partner countries have committed themselves to gradually align their different political, cultural and academic traditions around key values, and ‘agree to and adopt reforms on he on the basis of common key values – such as freedom of expression, autonomy for institutions, independent student unions, academic freedom, free movement of students and staff ’. the main goal of these countries is to ‘increase staff and students’ mobility and to facilitate employability’ (ehea, n.d.). this unique context has created a unique setting for various internationalisation activities that increase engagement between ehea partner countries. examples are the alignment of degrees in the bachelor/master system, the creation of the diploma supplement document, the erasmus student mobility programme and the marie sklodowska–curie staff exchanges and individual fellowships. we focus on two phenomena implementing internationalisation at he institutions within the ehea that build on the drivers of digitalisation, collaboration and openness: virtual mobility (vm) and open education (oe). the first phenomenon related to implementing internationalisation of higher educational curricula within the ehea, is the concept of vm, defined as ‘a set of ict supported activities, organized at institutional level, that realize or facilitate international, collaborative experiences in a context of teaching and/or learning’ (erasmus + programme guide 2018). this concept has been promoted with dedicated policies and funding. virtual mobility is a form of mobility between two or more (often european) he institutions, supported by a curricular, legal and institutional framework. through vm, learners enrolled as students in one higher educational institute have the opportunity http://dx.doi.org/10.25304/rlt.v28.2254 research in learning technology citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 3 (page number not for citation purpose) to follow a course at another higher educational institute in the online mode. as this is institutionally supported, vm participants enjoy the formal advantages of studying at that other institute, such as instructional support and assessment of their performance in the course. also, the gained credits for a successfully completed vm course are accepted by the students’ home institutions and accredited as part of the curriculum (bijnens, boussemaere, and rajagopal 2006). to support this form of student mobility, the student, the home institution and the ‘virtually’ visited institution can make use of a learning agreement as an instrument that stipulates the rights and duties of each party in the mobility (ubachs and henderikx 2018). virtual mobility therefore integrates the concepts of institutional collaboration and cooperation through digitalisation. it presumes some form of mutual value recognition between the institutions of each other’s educational and/or research offering. the overriding goal is to widen access to more students to engage in mobility activities, with all the benefits associated with those activities. the second phenomenon supporting internationalisation in the ehea is oe. in the last two decades, there has been an upsurge in open online learning developed and offered by a range of he institutions, including the world’s most prestigious universities. most notably, massive open online courses (moocs) and open educational resources (oer) have significantly influenced the global educational scene (daniel 2012; jansen and schuwer 2015; orr, weller and farrow 2018). during this period moocs in a multitude of instantiations of different scale have developed from a possibly ‘disruptive’ factor to a widely accepted form of learning in development existing next to, intermingled with and with varying links to formal curricula (rohs and ganz 2015). this upsurge in oe can clearly be seen as a consequence of the globalised educational market (varghese 2018). universities invest in online and blended learning in oe formats for diverse purposes, improving accessibility to high-quality education. they see the potential of open online education and oer in innovating he, developing and testing new designs and new educational formats. furthermore, universities use open online courses to present and promote their curricular offering (castano munos et  al. 2016; hollands and tirthali 2014; kiers 2016) and enrich or extend curricula or create an additional offering for graduates (pickering and swinnerton 2017). collaboration between oe providers can be organised on an individual level, for example in the form of reuse of oer and practices, and at an institutional level, for example through recognition of selected moocs offered by other institutions (cronin 2017; hew and cheung 2014; loeckx 2016). thus, oer and moocs have offered heis the means for establishing themselves as global organisations in a global educational market (varghese 2018). table 1 shows an overview of both concepts with respect to the three drivers. momentarily, vm as a form of cross-border online learning and oe, which is both online and borderless by default, notwithstanding their significant similarities, table 1. virtual mobility and open education against three disruptive drivers in higher education. virtual mobility open education digitalisation ++ict enabled ++ict enabled [institutional] collaboration ++ formalised cross-border collaboration between he institutions +primarily individual activities; sometimes networked collaboration between learners; collaboration between providers through european networks openness widening access to mobility ++ goal to widening access to higher education http://dx.doi.org/10.25304/rlt.v28.2254 k. rajagopal et al. 4 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) are two distinctly different strands, each with its own rich potential of contributing to the internationalisation of he (daniel 2012; ubachs and henderikx 2018). both these phenomena also fulfil economic arguments for the individual student and the institution to have access to or offer more quality learning opportunities, enabling individuals and institutions to operate in a globalised world. however, the communities around the two phenomena are also relatively distinct from each other. from a learner’s perspective, vm and oe offer different opportunities for developing learner skills and competences. from an institutional perspective, both facilitate different teaching models, but also have particular facilitation needs. however, current literature does not give clarity regarding which learner skills and competences are acquired and built by these activities, nor what the pre-requisite learner skills for successfully completing these activities are. few studies look at virtual collaboration within the international context. júnior and finardi (2018) look at a categorisation of the international collaborations based on minimal design features but do not consider learner skills or competences. van gaalen (2009) describes the mint tool used by nuffic in the netherlands for characterising internationalisation. although this tool does consider learner competence development, it uses self-evaluation and benchmarking and does not prescribe what learner competences are relevant in the context of internationalisation. green (2012) presents a quality framework that can support curriculum design and evaluation for internationalisation, but it does not integrate the broader competence development perspective. de kraker, cörvers, and lansu (2014) discuss the potential of virtual mobility to develop transboundary competence within the field of sustainable development. the specific contexts of oe create manifold opportunities for learners to develop skills and competences; however, these opportunities have not been investigated to the extent this theme deserves either. studying what learners actually learn in oe is challenged by the very variability of oe, from the personal perspective, to start with the drive to enrol, learn from oe (henderikx, kreijns and kalz 2017) to the institutional perspective, the drive to develop and deliver oe. a deeper understanding of how vm and oe relate to each other can inform a more profound appreciation of the complexities caused by digitalisation, institutional collaboration and openness and the effects on what individual learners and society in general learn and gain from them. in this highly dynamic context, re-conceptualising vm against the backdrop of oe (as open virtual mobility (openvm)) increases the sustainability of this form of learning. in the frame of the erasmus+ strategic partnership openvm 9 european partner organisations from he set the goals of developing a shared understanding of the concept of openvm and its core characteristics (buchem et al. 2018). this article describes an exploratory qualitative research conducted to establish the conceptual framework on openvm learner skills and competences. the research questions dealt with in this study are: which skills and competences do learners acquire and build when participating in openvm activities? what are the facilitation needs at an institutional level for openvm activities? method to answer the research questions. standpoints of experts on vm and oe were sought. knowledge and expertise on these phenomena is diffuse and scattered among individuals and institutions, university international offices, european networks and research http://dx.doi.org/10.25304/rlt.v28.2254 research in learning technology citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 5 (page number not for citation purpose) centres in different countries. therefore, a methodological approach was sought which supports not only online data collection but also joint knowledge construction and addresses validity checks to ensure the quality of outcomes. group concept mapping the applied methodology, the group concept mapping (gcm, kane and trochim 2007). supports knowledge construction through collecting and organising ideas of individuals so that a collective visual geography of a concept can be created to be further analysed, interpreted and used to feed understanding, design and/or decision or policy making. it is a mixed-methods approach in which advanced statistical analyses are applied to qualitative data. data generation and analysis in gcm is a structured multi-step approach in which (1) the target group is determined and participants are selected and invited; (2) participants generate ideas on the topic of the study supported by a prompt; (3) collected ideas are screened and cleaned up so that the resulting set contained unique unequivocal statements. (4) thereafter, participants group and rate the collected unique ideas on the relevant dimensions (i.e. importance and feasibility) and (5) the resulting data is analysed using multidimensional scaling (mds) and hierarchical cluster analysis (hca) to identify patterns in the data. the output of this analysis are maps representing individual standpoints on an issue in relation to each other and taken together, a collective standpoint of all participants. (6) such maps are then used to validate the shared understanding with study participants and (7) in order to formulate further actions or strategies (kane and rosas 2018; kane and trochim 2007). participants expertise on and affinity with the vm and oe at the conceptual level or in the educational practice formed the pre-requisite of the participation. such experts are scarce; therefore, for each phase in the study, additional recruitment was undertaken to complement the expertise within the openvm project. each project partner invited experts from their respective national networks for brainstorming, sorting and rating. oe global conference (oeglobal.org) was used as the venue for validation and the outcomes were finalised in a focus group with experts linked to the openvm project. table 2 gives an overview of the gcm study participants at different stages of the study. table 3 presents the occupational background of the respondents. table 2. participation in different phases of the gcm on open virtual mobility. participants brainstorm in gcm sorting in gcm rating in gcm validation activity (oe global conference) consolidation activity (finalising outcomes) openvm project members 11 13 12 6 5 non-members 8 15 12 3 0 total 19 28 24 9 5 http://dx.doi.org/10.25304/rlt.v28.2254 http://oeglobal.org k. rajagopal et al. 6 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) the question about experience in education is answered by half of the participants most of whom have more than 10 years of experience in university teaching. most participants indicated that they have experience related to both concepts of vm and oe. instruments the concept system® global max™ (2016) online environment (https://conceptsystemsglobal.com/) was used for data generation and analysis. printouts of the generated visual representations of the results (various maps) were used at the validation and interpretation workshop. procedure 1. recruiting participants. all openvm project members were invited to participate in the study and were requested to share the invitation in their respective networks including representatives of the research community, educators, internationalisation officers at higher education institutions, he boards representatives and policy makers. project members approached their contacts through e-mail with a reminder in case of non-response and invited to participate in the sorting and rating activities through the online tool. informed consent request was requested through the gcm tool. 2. idea generation. idea generation was steered by the focus prompt. please, complete the statement ‘in the context of open education, virtual mobility implies that students ….’. participants were invited to give as many answers as they wished in the form of short direct statements. in total, 101 statements were generated in the idea-generation phase of the study. table 3. background characteristics of gcm participants per phase. background characteristics responded to invitation completed sorting activity completed rating on at least one dimension completed status for all online activities together took part in the oe global validation workshop took part in the final consolidation university professors 19 14 14 15 6 2 researcher 7 5 5 5 2 1 international office staff 6 2 0 2 0 0 university board/ policy makers 2 1 2 2 0 0 educational support staff 4 3 2 3 1 2 ict support staff 4 3 0 3 0 0 other 4 0 4 4 0 0 total 46 28 27 34 9 5 http://dx.doi.org/10.25304/rlt.v28.2254 https://conceptsystemsglobal.com/ https://conceptsystemsglobal.com/ research in learning technology citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 7 (page number not for citation purpose) 3. data cleaning and removal duplicates. all duplicate statements were removed. statements were checked for repetition or ambiguity issues by two project members separately, results were compared and full agreement was reached on statements to be removed. the final set included 90 unique statements. 4. sorting and rating. participants were invited to group statements based on their similarity in meaning, provide the groups with meaningful labels and to evaluate each statement on dimensions of importance and feasibility on a scale from 1 (not important/feasible) to 5 (highly important/feasible). 5. analysis. mds and hca analyses were conducted by using the gcm tool, and visual representations of the data were generated for conceptual analysis, interpretation and validation with the stakeholders. preliminary data interpretation was conducted by the two lead authors in preparation for the validation and consolidation phase. 6. interpretation, validation and consolidation. two sessions were held to interpret, validate and consolidate the outcomes of the analysis. an oe global conference 2018, an action lab activity, was used as an interpretation and validation workshop. during the workshop visual representations of preliminary clusters, the respective statements and key statistics were presented to the workshop participants who discussed the presented data and provided feedback. contributions of the participants were written down and analysed by the lead authors. based on the outcomes of the validation workshop an overview of the clusters with tentative labels was carried out to support the final step – a consolidation workshop that was held with five selected project members with both theoretical and practical expertise on vm. the outcomes of the consolidation workshop are considered final results of the gcm study. results a total of 90 statements generated in gcm are presented in a detailed point map in quadrant 1 of figure 1. the point map visualises the position of individual statements in relation to each other that results from the mds analysis. statements that are situated closer to each other on this map are grouped together by individual participants more often. when clusters are defined, such statements have a greater chance to be put in the same cluster. the extent to which the data point map represents the way individual participants sorted the data is tested with the help of kruskal’s stress value statistic that indicates the goodness-of-fit of the data. a lower stress value is an indication of a better fit which should be in the range between 0.205 and 0.365 (kane and trochim 2007). in this study, the stress value constituted a goodness-of-fit of an acceptable level of 0.2531. quadrant 2 of figure 1 presents a visualisation of thematic groupings of ideas in 10 clusters. the 10-cluster solution results from the analysis of the individual sorting activity with hca supported by the bridging value statistic and the outcomes of the validation workshop. the bridging value index (table 3) indicates to what extent a cluster is a consistent and coherent entity, separate from other clusters. the lower the bridging value, the higher the level of cluster consistency is; the higher the bridging value, the more the cluster and its constituent elements (statements) are related to statements in other clusters. http://dx.doi.org/10.25304/rlt.v28.2254 k. rajagopal et al. 8 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) in the 10-cluster model of openvm competences, the average bridging values vary from 0.12 (a highly consistent cluster) to 0.54 (a moderately consistent cluster). as the clusters present thematic groupings, defining the name or the cluster label is relevant. the gcm tool suggests ‘a best match’ label to the clusters based on the individual sorting activity. this ‘best match’ option is used to steer discussion and reach agreement at the interpretation and validation workshops. the final labels resulting from the expert consolidation session are based on agreement of all five participating experts. table 3 shows the clusters with their labels, the respective bridging values and exemplary statements per cluster; figure 3 is a visualisation of the outcomes. the other two quadrants of figure 1 present respectively results of rating the statements on dimensions of importance (q3) and feasibility (q4). the more layers the cluster has, the higher the average rating index of the cluster which is an indication that a particular cluster combines items that are rated higher on importance of feasibility dimensions. as data presented in table 4 indicate, there is a high level of agreement between the participants in the gcm study on clusters 1, 2, 3, 4, 5 and 6. as labels of the clusters indicate, these clusters describe the skills that learners develop in openvm and the skills that they need to succeed in openvm. cluster 7 (open-mindedness) which has a higher though still moderate bridging value describes an attitude. figure 1. the outcomes of the group concept mapping study on openvm. quadrant 1, the point map; quadrant 2, the cluster map; quadrant 3, the rating map on the dimension of importance and quadrant 4, the rating map on the dimension of feasibility. http://dx.doi.org/10.25304/rlt.v28.2254 research in learning technology citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 9 (page number not for citation purpose) table 4. openvm clusters with exemplary statements per cluster with their respective bridging values. cluster with per cluster exemplary statements with their respective bridging values as an answer to the prompt in the context of open education, virtual mobility implies that students [develop]… m bridging values(sd) n statements per cluster 1. intercultural skills & attitudes • gain knowledge about the culture they ‘visit’ • have exposure to different working and cultural backgrounds, which could raise both new potentials and barriers at the same time • feel confident in interacting with people from other cultures after a vm experience 0.12 (0.13) 16 2. networked learning • learn to work and cooperate in an international setting with the use of ict and social platforms • learn about dealing with complex situations through the vm activity • learn about dealing with ambiguity through the vm activity 0.25 (0.05) 6 3. active self-regulated learner skills • should be able to plan and organise their own learning process • are able to self-reflect • aims of vm in student development -self-discipline in learning 0.16 (0.13) 10 4. media and digital literacy • are proficient in searching for good quality courses and resources • are digitally literate • are proficient in using digital platforms 0.22 (0.08) 12 5. autonomy-driven learning • develop persistence and creativity in organising their own study,. that is they might need to find suitable and feasible courses on their own and convince curriculum boards of the quality of learning in oe contexts • can enhance their lifelong learning skills • can learn in an open digital context 0.21 (0.07) 6 6. interactive and collaborative learning in an authentic international environment • exchange knowledge with peers from different disciplines • collaborate with peers from different disciplines • the open digital context facilitates collaboration about international students 0.31 (0.07) 9 7. open-mindedness • are open minded • are not afraid of interacting with peers and teaching staff at other institutions • are willing to improve their proficiency in foreign languages 0.53 (0.07) 5 http://dx.doi.org/10.25304/rlt.v28.2254 k. rajagopal et al. 10 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) the three remaining clusters also have higher, though still moderate bridging values. these clusters seem more related to external aspects of openvm such as programme characteristics or pre-requisites of vm. importance and feasibility ratings (see figure 1, quadrants 3 and 4) indicate that clusters that represent skills and attitudes (clusters 1–7) score higher both on importance and feasibility while the three clusters that represent external factors are considered relatively less important and at the time same less feasible. these results allow us to answer the two research questions posed above in a reasonably straightforward way. according to the conducted gcm study, openvm activities encourage the development of generic skills and competences, such as intercultural skills (1), networked learning (2), active self-regulated learning (3), media and digital literacy (4), autonomy-driven learning (5), interactive and collaborative learning in an authentic international environment (6) and open mindedness (7). figure 2 gives an overview of these skills and competences, with definitions formulated in the consolidation workshop. these skills seem to form a plausible point of departure in building a conceptual framework of openvm. the gcm study also points to three separate clusters that represent conditions and pre-requisites for development of openvm, namely: the added value of openvm (8); the way the study and learning process is organised in openvm (9) and openvm design characteristics (10). figure 3 is a graphical representation of the resulting concept map of the openvm. cluster with per cluster exemplary statements with their respective bridging values as an answer to the prompt in the context of open education, virtual mobility implies that students [develop]… m bridging values(sd) n statements per cluster 8. potentials of openvm • have access to high quality learning processes that otherwise would not be possible for them • do not have to pay any fees for attending a virtual mobility course of a foreign institution • receive academic recognition for participating in virtual mobility 0.39 (0.11) 11 9. openvm study characteristics • carry out the learning process under the characteristics of open education • students create their digital identity through the open context • using it tools in a transparent and efficient way to interact with other participants and the learning material 0.53 (0.11) 6 10. openvm programme design characteristics • are involved in predetermined learning activities, open and collaborative, through which they can acquired knowledge thanks to innovative learning methods • the construction of a well-defined learning path in which collaborative and international activities are pivotal • needs guidance and support to make the right choices and to stay motivated 0.54 (0.20) 9 table 4. (continued) http://dx.doi.org/10.25304/rlt.v28.2254 research in learning technology citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 11 (page number not for citation purpose) discussion from the point of view of conceptualising openvm as a new phenomenon related to but distinct from both vm and oe, the visual geography of gcm clusters and distance between them seems meaningful. as the visualisation in figure 4 demonstrates, the generic learner competences (1–6) occupy the central position forming the core of the openvm framework in the most direct sense of the word. the central position of clusters representing these competences is combined with high levels of agreement between contributing experts on these clusters, as low bridging values indicate. the seventh competence area (open mindedness) seems both statistically and geographically a borderline cluster. higher bridging values indicate that different experts relate the cluster and the underlying statements to different statements in different clusters. this cluster is less consistent, more diffuse, suggesting that there is less clarity on what it constitutes. semantically, though, a connection to oe is meaningful. figure 2. an overview of skills and competences relevant in open virtual mobility contexts as emerged from the gcm study. http://dx.doi.org/10.25304/rlt.v28.2254 k. rajagopal et al. 12 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) furthermore, open mindedness as a competence cluster occupies the furthest position from the clusters representing contextual and institutional determinants (conditions, prerequisites and study characteristics) of openvm. the framework points to a gradual trend from institution-related to individual (x-axis). the trend from institution to individual in figure 4 also suggests a trend from design-related to emerging (learner behaviour/learner attitudes). the statements in the three clusters representing contextual and institutional determinants discuss the context that supports learning and how an ideal context could be designed. the core competence clusters situated towards the left (clusters 9, 5 and 4) depend on instructional design of the vm activities to support them. clusters 8, 6, 10 and 7 however, deal much more with the development of individual learner attitudes and related behaviour and skills that are developed through both learning and life experiences. looking at the generic learner competences, we see an upward trend (y-axis) moving from skills related to collaboration and social competences (clusters 1, 2 and 6) more at the bottom to skills related to individual learner competences (clusters 5, 3 and 4) at the top. interestingly, this distinction between institution and an individual is also noticeable in the established competence and support frameworks within the ehea. the central competences emerging from the gcm study on openvm activities are related to skills and competences defined in existing individual competence frameworks, such as digital skills and competences covered in digcomp 2.1 (carretero, vuorikari and punie 2017) and the european reference framework on key competences for lifelong learning of esco (esco european skills/competences, qualifications and occupations 2018). the individual learner competences from our study – autonomy-driven learning, active self-regulated learning and media and digital literacy – take a prime position in these frameworks. figure 3. resulting map of the open virtual mobility concept. http://dx.doi.org/10.25304/rlt.v28.2254 research in learning technology citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 13 (page number not for citation purpose) the three clusters representing contextual and institutional determinants and their underlying statements are mirrored in the openedu support framework ( inamorato dos santos and punie 2016). this framework is aimed at the institutional level to describe the desired and aspired-to level of openness. it is interesting that openness does not yet feature in a competence framework at the individual learner’s level as a competence area or attitude – as it emerges as ‘openmindedness’ from this gcm study. the generic learner competences in our gcm study show overlap with six of the eight key competences for lifelong learning determined by the european union (european union 2019), including personal, social and learning to learn competence, cultural awareness and expression competence and citizenship competence. there are limitations to this study: the experts who participated in in the study as experts on the topics of oe and vm are primarily university teachers or researchers. other ‘adult’ stakeholders including internationalisation policy officials are marginally represented and students, who form the main target group of openvm activities, are not represented. an inclusion of students in the study might have identified different priorities and interests. such inclusion may be possible in the context of current vm activities and exchanges in new or existing consortia of universities. student perspective on vm competences deserves to become a new focus in future research of vm as a strand in he internationalisation. caution is needed in interpreting the outcomes on which there is less agreement between the participants, that is, open mindedness as an individual characteristic and the institutional characteristics. it is not only that more research is needed, the need is figure 4. analysis of the openvm concept map: visible trends. http://dx.doi.org/10.25304/rlt.v28.2254 k. rajagopal et al. 14 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) of different kind of research, based on data analytics, based on analysis of experiences of participants involved in openvm trajectories (foremost students but also those who organise and support vm)and analysis of effectiveness of varying design solutions. conclusions the outcomes of the study allow us to draw a number of conclusions about conceptualising openvm and establishing its focus and locus as part of the educational scene of the 21st century and formulate recommendations for further research. the conceptual crossing of vm and oe as new educational strands has brought to the fore a number of generic learner skills and competences as a distinct set of skills and competences that openvm supports. these competences represent generic competences coined as 21st-century learner skills (trilling and fadel 2009; voogt and pareja roblin 2010) and combine aspects of the three disruptive drivers in the current he landscape: digitalisation, collaboration and openness, to varying degrees. these are all complex skills that require complex and varied learning contexts, multiple application contexts and extended practice (van merrienboer and kirschner 2018). we do not say that openvm is the only way to develop these skills, nor that the insights that this study provided are exhaustive and conclusive. rather than that, the results of this study point to a viable direction for a discussion on the development of transversal skills and competences needed in a variety of 21st-century professions and forming a good fit to the generic learning goals of internationalised, globalised he. thus, openvm can be seen as a viable method in tackling the challenging tasks of supporting and enhancing development of these learner skills. further research can advance on our work in various ways. the first avenue for research concerns the challenge of validation of this study. one possible qualitative method to do this is to look at the extent to which the outcomes of the gcm study can be mapped to existing openvm activities. more quantitative methods require better measuring instruments – which we discuss further. a second option for further research concerns a categorisation of these types of activities. although the concepts of vm and openvm have a european foundation, and are embedded in european policies, we see many examples of these concepts in other parts of the world. notable examples are kiron open he (kiron 2019), an ngo working towards high-quality educational opportunities for refugees and underserved communities: by making individual learning programmes and paths using open education models, they support prospective learners in preparation for and enrolment in formalised learning programmes at established universities across the world. organisations such as soliya and unicollaboration, with their various formats to encourage intercultural dialogue across the globe, now partner in the erasmus+ virtual exchange, aimed at eu member states and countries of the south mediterranean region, to gain more traction for these forms of mobility (erasmus+ virtual exchange 2019; soliya 2019). these examples bring up the issue of definition and naming: we see various terms being used to name the same type of activities, with different nuances in design and formats (e.g. collaborative online international learning (coil) in rubin and guth 2016; telecollaboration in dooly 2017; globally networked teaching and learning in starke-meyerring et al. 2008; online intercultural exchange in o’dowd and lewis 2016).the results of our study can be a starting point to create a categorisation or typology of these different formats in more nuanced ways. http://dx.doi.org/10.25304/rlt.v28.2254 research in learning technology citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 15 (page number not for citation purpose) a final avenue for further research concerns measuring proficiency in these learner skills, and relatedly, effectiveness of design of these activities. this gcm study only identifies which learner skills and competences occur in these new educational formats, and does not say anything about how these skills and competences can be developed further. open virtual mobility learner activities give shape to a form of experiential learning, and the specific formats of the activities that students experience highly determine which skills are developed and to what extent they are acquired. therefore, we expect a potential openvm competence framework will need further operationalisation of these skills and corresponding proficiency levels. however, measuring learner competence development is complex. first, there is the issue of what you measure. as our gcm study shows, openvm activities afford the development of several complex learner competences. measuring these learner competences can happen on the plane of individual competences, or be focused on how the different competences interrelate in design. it is important to note that not all of the competences can currently be measured using quantitative methods (gosen and washbush 2004). secondly, the question arises of how you measure learning. gosen and washbush (2004) discuss the difficulties of measuring learning gains from experiential learning designs and indicate measures to overcome these difficulties, including clarity in design aims, highly interdisciplinary research approaches and data-driven work on large-scale test beds (gosen and washbush 2004). mcnamara (2013) points to the importance of multiple sources of evidence including learner self-reflection and supervisor/educators’ assessment. there are few studies on the impact of (open) virtual mobility activities on learner skills and competences (leh, grau and guiseppe 2015; o’dowd 2007). one relevant study is the 2018 impact report on the erasmus+ virtual exchange (helm and van der velden 2019), where a set of quantitative and qualitative instruments has been developed to assess the impact of participation in erasmus+ virtual exchange activities on learners’ soft skills (intercultural communicative competence, self-esteem, curiosity, affect towards other groups). the current study opens the scope for a more nuanced method to determine the learner skills involved in(open) virtual mobility activities, and to effectively assess emerging learner activity within a particular design (goodyear and carvalho 2013, 2014). acknowledgements this article is based on the joint work and research conducted by partner organisations in the erasmus+ project open virtual mobility, ka2 – cooperation for innovation and the exchange of good practices, ka203 – strategic partnerships for he, (partially) founded by the european union, project number 2017-1-de01-ka203-003494. disclaimer the creation of these resources has been (partially) funded by the erasmus+ grant program of the european union under grant no. 2017-1-de01-ka203-003494. neither the european commission nor the project’s national funding agency daad are responsible for the content or liable for any losses or damage resulting of the use of these resources. http://dx.doi.org/10.25304/rlt.v28.2254 k. rajagopal et al. 16 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) references bijnens, h., boussemaere, m. and & rajagopal, k. eds., (2006). european cooperation in education through virtual mobility: a best-practice manual., europace, leuven. buchem, i., et al., (2018) ‘designing a collaborative learning hub for virtual mobility skills-insights from the european project open virtual mobility’, in international conference on learning and collaboration technologies, springer, cham, pp. 350–375. blight, d., davis, d. & olsen, a. (1999) ‘the internationalisation of higher education’, in higher education through open and distance learning – world review of distance education and open learning, vol. 1, ed k. harry, routledge, london, pp. 15–31. bologna declaration. (1999) joint declaration of the european ministers of education, available at: http://ehea.info/upload/document/ministerial_declarations/1999_bologna_ declaration_english_553028.pdf carretero, s., vuorikari, r. & punie, y. (2017) digcomp 2.1: the digital competence framework for citizens with eight proficiency levels and examples of use, publications office of the european union, luxemburg, eur 28558 en. doi: 10.2760/38842 castaño muñoz, j., et al., (2016) how are higher education institutions dealing with openness? a survey of practices, beliefs and strategies in five european countries, institute for prospective technological studies, jrc science for policy report, publications office of the european union, luxemburg, eur 27750en. doi: 10.2791/709253 crabtree, r. d., et al., (2008) ‘realizing the university mission in partnership with nicaragua: internationalization, diversity, and social justice’, in designing globally networked learning environments: visionary partnerships, policies, and pedagogies, eds d starke-meyerring, sense publications, rotterdam, pp. 87–103. cronin, c. (2017) ‘openness and praxis: exploring the use of open educational practices in higher education’, the international review of research in open and distributed learning, vol. 18, no. 5. pp. 15–34. doi: 10.19173/irrodl.v18i5.3096 daniel, j. (2012) ‘making sense of moocs: musings in a maze of myth, paradox and possibility’, journal of interactive media in education, vol. 2012, no. 3. p. art. 18. doi: 10.5334/2012-18 de kraker, j., cörvers, r. j. m. & lansu, a. (2014) ‘e-learning for sustainable development: linking virtual mobility and transboundary competence’, in e-learning and sustainability, peter lang gmbh international academic publishers. dooly, m. (2017) ‘telecollaboration’, in the handbook of technology and second language teaching and learning, eds c. a. chapelle & s. shauro, wiley blackwell, oxford, pp. 169–183. ehea, n.d., european higher education area and bologna process, available at: http://www. ehea.info erasmus + programme guide. (2018) available at: https://ec.europa.eu/programmes/erasmus-plus/ sites/erasmusplus2/files/erasmus-plus-programme-guide2_en.pdf erasmus+ virtual exchange (2019). available at: https://europa.eu/youth/erasmusvirtual esco european skills/competences, qualifications. (2019), available at: https://ec.europa.eu/ esco/portal european union. (2019) key competences for lifelong learning, available at: https://op.europa.eu/en/publication-detail/-/publication/297a33c8-a1f3-11e9-9d01-01aa75ed71a1/ language-en flavin, m. & quintero, v. (2018) ‘uk higher education institutions’ technology-enhanced learning strategies from the perspective of disruptive innovation’, research in learning technology, vol. 26. doi: 10.25304/rlt.v26.1987 goodyear, p. & carvalho, l. (2014) the architecture of productive learning networks, routledge, new york. goodyear, p. & carvalho, l. (2019) ‘the analysis of complex learning environments’, in rethinking pedagogy for a digital age, 3rd edn., eds h. beetham & r. sharpe, routledge, new york, pp. 49–65. http://dx.doi.org/10.25304/rlt.v28.2254 http://ehea.info/upload/document/ministerial_declarations/1999_bologna_declaration_english_553028.pdf http://ehea.info/upload/document/ministerial_declarations/1999_bologna_declaration_english_553028.pdf http://dx.doi.org/10.2760/38842 http://dx.doi.org/10.2791/709253 http://dx.doi.org/10.19173/irrodl.v18i5.3096 http://dx.doi.org/10.5334/2012-18 http://www.ehea.info http://www.ehea.info https://ec.europa.eu/programmes/erasmus-plus/​sites/erasmusplus2/files/erasmus-plus-programme-guide2_en.pdf https://ec.europa.eu/programmes/erasmus-plus/​sites/erasmusplus2/files/erasmus-plus-programme-guide2_en.pdf https://europa.eu/youth/erasmusvirtual https://ec.europa.eu/esco/portal https://ec.europa.eu/esco/portal https://op.europa.eu/en/publication-detail/-/publication/297a33c8-a1f3-11e9-9d01-01aa75ed71a1/language-en https://op.europa.eu/en/publication-detail/-/publication/297a33c8-a1f3-11e9-9d01-01aa75ed71a1/language-en https://op.europa.eu/en/publication-detail/-/publication/297a33c8-a1f3-11e9-9d01-01aa75ed71a1/language-en http://dx.doi.org/10.25304/rlt.v26.1987 research in learning technology citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 17 (page number not for citation purpose) gosen, j. & washbush, j. (2004) ‘a review of scholarship on assessing experiential learning effectiveness’, simulation & gaming, vol. 35, no. 2, pp. 270–293. doi: 10.1177/1046878104263544 green, m. f. (2012) ‘measuring and assessing internationalization’, nafsa: association of international educators, vol. 1, pp.1–26. helm, f. & van der velden, b. (2019) erasmus+ virtual exchange: 2018 impact report, european union and eacea, publications office of the european union, luxembourg. doi: 10.2797/668291 henderikx, m. a., kreijns, k. &kalz, m. (2017) ‘refining success and dropout in massive open online courses based on the intention–behavior gap’, distance education, vol. 38, no. 3, pp. 353–368. doi: 10.1080/01587919.2017.1369006 hew, k. f. & cheung, w. s. (2014) ‘students’ and instructors’ use of massive open online courses (moocs): motivations and challenges’, educational research review, vol. 12, pp. 45–58. doi: 10.1016/j.edurev.2014.05.001 hollands, f. m. & tirthali, d. (2014) ‘why do institutions offer moocs?’, journal of asynchronous learning networks, vol. 18, no. 3, p. 3. available at: https://www.learntechlib.org/p/154185/ inamorato dos santos, a. & punie, y. (2016). opening up education: a support framework for higher education institutions (no. jrc101436), joint research centre (seville site). jansen, d. & schuwer, r. (2015) institutional mooc strategies in europe, status report based on a mapping survey conducted in october-december 2014, eadtu, available at: http://www. eadtu.eu/documents/publications/oeenm/institutional_mooc_strategies_in_europe.pdf júnior, c. a. h. & finardi, k. r. (2018) internationalization and virtual collaboration: insights from coil experiences’, ensinoem foco, vol. 1, no. 2, pp.19–33. kane, m. & rosas, s. (2017) conversations about group concept mapping: applications, examples, and enhancements, sage, los angeles, ca. kane, m. &trochim, w. m. (2007) concept mapping for planning and evaluation (vol. 50), sage, thousand oaks, ca. kiers, j. (2016) ‘moocs and their effect on the institution: experiences in course design, delivery and evaluation; research; faculty development; unbundling and credits for moocs’, foro de educación, vol. 14, no. 21. doi: 10.14516/fde.2016.014.021.007 kiron (2019). available at: https://kiron.ngo/en/about-us/ knight, j. (2003) ‘updated definition of internationalization’, international higher education, vol. 33. pp. 2–3. knight, j. (2004) ‘internationalization remodeled: definition, approaches, and rationales’, journal of studies in international education, vol. 8, no. 1, pp. 5–31. doi: 10.1177/102831 5303260832 leh, j. m., grau, m. & guiseppe, j. a. (2015) ‘navigating the development of pre-service teachers’ intercultural competence and understanding of diversity’, journal for multicultural education, vol. 9, no. 2, pp. 98–110. doi: 10.1108/jme-12-2014-0042 loeckx, j. (2016) ‘blurring boundaries in education: context and impact of moocs’, the international review of research in open and distributed learning, vol. 17, no. 3. pp. 92–121. doi: 10.19173/irrodl.v17i3.2395 mcnamara, j. (2013) ‘the challenge of assessing professional competence in work integrated learning’, assessment and evaluation in higher education, vol. 38, no. 2, pp.183–197. nascimbeni, f., et al., (2018) ‘exploring intercultural learning through a blended course about open education practices across the mediterranean’, in 2018 ieee global engineering education conference (educon), ed ieee, ieee, tenerife, pp. 285–288. o’dowd, r. (2007) ‘evaluating the outcomes of online intercultural exchange’, elt journal, vol. 61, no. 2, pp.144–152. doi: 10.1093/elt/ccm007 o'dowd, r. & lewis, t. (eds) (2016) online intercultural exchange: policy, pedagogy, practice, routledge. orr, d., weller, m. &farrow, r. (2018) ‘models for online, open, flexible and technology enhanced higher education across the globe–a comparative analysis’, international council http://dx.doi.org/10.25304/rlt.v28.2254 http://dx.doi.org/10.1177/1046878104263544 http://dx.doi.org/10.2797/668291 http://dx.doi.org/10.1080/01587919.2017.1369006 http://dx.doi.org/10.1016/j.edurev.2014.05.001 https://www.learntechlib.org/p/154185/ http://www.eadtu.eu/documents/publications/oeenm/institutional_mooc_strategies_in_europe.pdf http://www.eadtu.eu/documents/publications/oeenm/institutional_mooc_strategies_in_europe.pdf http://dx.doi.org/10.14516/fde.2016.014.021.007 https://kiron.ngo/en/about-us/ http://dx.doi.org/10.1177/102831​5303260832 http://dx.doi.org/10.1177/102831​5303260832 http://dx.doi.org/10.1108/jme-12-2014-0042 http://dx.doi.org/10.19173/irrodl.v17i3.2395 http://dx.doi.org/10.1093/elt/ccm007 k. rajagopal et al. 18 citation: research in learning technology 2020, 28: 2254 http://dx.doi.org/10.25304/rlt.v28.2254 (page number not for citation purpose) for open and distance education (icde),oslo, norway, available at: https://oofat.oerhub. net/oofat/.cc-by-sa pickering, j. d. & swinnerton, b. j. (2017) ‘an anatomy massive open online course as a continuing professional development tool for healthcare professionals’, medical science educator, vol. 27, no. 2, pp. 243–252. doi: 10.1007/s40670-017-0383-7 rohs, m. & ganz, m. (2015) ‘moocs and the claim of education for all: a disillusion by empirical data’, the international review of research in open and distributed learning, vol. 16, no. 6. pp. 1–19. doi: 10.19173/irrodl.v16i6.2033 ryan, c., et al., (2017) ‘managing the process of international collaboration in online course development: a case-example involving higher education institutions in ireland, switzerland, austria, and the united kingdom’, innovative higher education, vol. 42, no. 5–6, pp. 451–462. doi: 10.1007/s10755-017-9399-6 rubin, j. & guth, s. (2015) ‘collaborative online international learning: an emerging format for internationalizing curricula’, in globally networked teaching in the humanities, eds a. schultheis moore & s. simon, routledge, new york, pp. 27–39. soliya (2019). available at: https://www.soliya.net/ starke-meyerring, d., et al., (2008) designing globally networked learning environments: visionary partnerships, policies, and pedagogies, sense publications, rotterdam. the concept system® global max™ (build 2016.046.12) [web-based platform]. (2016), ithaca, ny, available at: http://www.conceptsystemsglobal.com trilling, b. & fadel, c. (2009) 21st century skills: learning for life in our times, john wiley & sons, san francisco, ca. ubachs, g., & henderikx, p. (2018) eadtu mobility matrix, eadtu, maastricht, nl, pp. 26, available at: https://tinyurl.com/eadtu-mobility-matrix van gaalen, a. (2009) ‘developing a tool for mapping internationalisation: a case study’, in measuring success in the internationalisation of higher education, ed h. de wit, european association for international education (eaie), amsterdam, p. 77. van merriënboer, j. j. g. & kirschner, p. a. (2018) ten steps to complex learning: a systematic approach to four-component instructional design, 3rd edn., routledge, new york. slagter van tryon, p. j. s., mcdonald, j. & hirumi, a. (2018) ‘preparing the next generation of instructional designers: a cross-institution faculty collaboration’, journal of computing in higher education, vol. 30, no. 1, pp. 125–153. doi: 10.1007/s12528-018-9167-3 varghese, n. v. (2008) globalization of higher education and cross-border student mobility, international institute for educational planning, unesco, paris. varghese, n. v. (2018) ‘globalization and cross-border mobility in higher education’, educational access and excellence, vol. 1, p. 9. voogt, j. & roblin, n. p. (2010) 21st century skills discussion paper, university of twente, kennisnet, enschede. vrients, m. & van petegem, w. (2016) eu-vip: make it work! integrating virtual mobility in international work placements, available at: https://www.eurashe.eu/library/ wg4-r-eu_vip-manual_online-pdf/ wiley, d. (2010) ‘openness as catalyst for an educational reformation’, educause review, vol. 45, no. 4, pp. 14–20 available at: http://hdl.lib.byu.edu/1877/2349 wiley, d. & hilton iii, j. (2009) ‘openness, dynamic specialization, and the disaggregated future of higher education’, the international review of research in open and distributed learning, vol. 10, no. 5. doi: 10.19173/irrodl.v10i5.768 http://dx.doi.org/10.25304/rlt.v28.2254 https://oofat.oerhub.net/oofat/.cc-by-sa https://oofat.oerhub.net/oofat/.cc-by-sa http://dx.doi.org/10.1007/s40670-017-0383-7 http://dx.doi.org/10.19173/irrodl.v16i6.2033 http://dx.doi.org/10.1007/s10755-017-9399-6 https://www.soliya.net/ http://www.conceptsystemsglobal.com https://tinyurl.com/eadtu-mobility-matrix http://dx.doi.org/10.1007/s12528-018-9167-3 https://www.eurashe.eu/library/wg4-r-eu_vip-manual_online-pdf/ https://www.eurashe.eu/library/wg4-r-eu_vip-manual_online-pdf/ http://hdl.lib.byu.edu/1877/2349 http://dx.doi.org/10.19173/irrodl.v10i5.768 doi:10.3402/rlt.v19s1/7777 development of a simulated internet for education andrew smith* department of communication and systems, the open university, milton keynes, uk (received 27 february 2011; final version received 22 june 2011) this paper describes the early stages of research and development of an educational environment designed to enable learners to participate in remote, group based large-scale activities based on local area network and wide area network technologies working on a range of systems and within different learning situations, such as in class group work, remote group work or independent learning. the environment covers specifically routing, switching and wireless principles in the domain of computer networking. this is accomplished using the ‘multiuser functionality’ feature found within the cisco academy programme, packet tracer application. the initial research explores how a ‘virtual internet’ can be implemented to enable learners to engage with the scale and complexity of the internet without interacting with active routing infrastructures thereby interfering with others. different communities of interest from cisco systems as well as their academy programme academic affiliates have contributed to the development of the resource as well as to research into how individuals participate in learning as a result of using this software. this paper tells the story of the iterative action research process with two initial learning situations of ‘remote many’ participation and ‘in class many’ participation in a large scale networking exercise. as research is still in the development process, this paper explores the experiences and observations gathered from engaging with the two learning scenarios, describing how each interaction exercise was perceived by participants and their educators. initial findings from both activities indicate that the concept of an ‘internet on the internet’ to deliver simulated practical learning has considerable potential and brings an alternative dimension to the practical learning experience. research is ongoing, with the work in this paper informing the continual iterative process. keywords: constructivism; situated-learning; simulation; packet-tracer; collaborativelearning introduction the open university in the uk commenced offering the cisco academy programme, the cisco certified networking associate (ccna) in 2005 as a blended distance-learning course as part of the foundation degree in information and communications technology (ict). since inception this course has reached in excess of 4000 students across the uk and some internationally, all taking the course in a blended distance-learning mode. a challenge is managing access to course specific router and switch technology, giving students an essential opportunity to engage in interactive practical activities. *corresponding author. email: a.smith@open.ac.uk alt-c 2011 conference proceedings 0081 isbn 978-91-977071 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # a. smith. this is an open access article distributed under 100 the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7777 this enables each student to acquire a view of the complexity involved in network environments, such as a corporate wide area network infrastructure or the internet itself. the course team responsible for the management of delivery explored a range of tools, including netlab� distributed by netdev group (http://www.netdev group.com) and packet tracer from cisco systems inc. (academy site at http:// www.cisco.com/web/learning/netacad/index.html) this included setting a range of assessment tasks using these tools and experimenting in group based delivery as researched by smith and moss (2008) and prieto-blázquez (2008), the research focused on the setting of course assessment items and the management of synchronous and asynchronous learning using both resources. with the introduction of multiuser functionality in packet tracer version 5.0 and the publication of the packet tracer multiuser protocol (ptmp) (wang 2008), the packet tracer application enables students in disparate locations to interact on a common simulated practical activity. an overview of packet tracer as a network simulation environment, packet tracer provides simulated router, switch, server, workstation and networking protocol resources for students and educators to create diverse and complex networking scenarios, extending the pedagogical and practical experience during participation in the cisco academy programme. the cisco certified networking associate (ccna) version 4.x exploration and discovery curriculum contains embedded laboratory exercises for the students to complete. these are in-class on live networking technology, remotely via the netlab� system or by launching the packet tracer application from within the curriculum content. packet tracer activities are goal based, giving students attainable milestones and feedback by indicating the completion percentage based on the given activity scenario. as a simulation tool, the ‘operating system’ deployed on simulated workstations and routers forms a critical subset of the actual technology, presenting similar behaviour, performance and idiosyncrasies within a contained experience. the inbuilt multiuser functionality allows students and academic centres to create environments that can interact, irrespective of locale, type of academic institution and supporting network infrastructure. this results in the ability for two students, in any location, being able to create a connection and complete a practical activity of their choice. figure 1 illustrates two independent instances of packet tracer using a peered network connection, with two simulated workstations exchanging simulated network traffic. a multiuser connection can be established on any transmission control protocol (tcp) port, with ports 38,000�38,999 selected by default. an academic centre may elect to use an alternative port according to local networking security policies. packet tracer is therefore able to handle multiple multiuser connections between many of users. it can support one:one, one:many and many:many options, with either remote or local collaboration scenarios available to students and academic centres alike. designed to be ‘easy to use’ anyone using the packet tracer application can quickly create a multiuser connection by using the default port and an internet protocol (ip) address or established domain name as illustrated in figure 2. alt-c 2011 conference proceedings 101 http://www.netdevgroup.com http://www.netdevgroup.com http://www.cisco.com/web/learning/netacad/index.html http://www.cisco.com/web/learning/netacad/index.html with the many:many multiuser interconnection available, multiple academic centres may create a mesh of connections, with students interconnecting to a collaborative environment internally or externally to complete a range of practicebased learning activities. creating a relay there is a clear pedagogical benefit in allowing students and academic centres to connect across a local area network (lan), a wide area network (wan) or the internet via packet tracer for their work. however, the application presents a serious networking trust issue in the explicit exchange of an ip address or domain information by any participant. any one:one connection between peers would carry an implied personal trust, assuming that both parties are aware of the other’s need to connect. with each party present at the workstation hosting the packet tracer application, they are able to permit or deny any initiated connection. packet tracer at this level does not allow anonymous connects unless the user specifically configures the application to do otherwise; this is not a default state. figure 1. a peered example of packet tracer multiuser communication. a. smith 102 with a one:many, in an internal scenario, the same trust principle as the one:one connection relationship is implied. the academic leading the practice-based session will be present to invoke the activity and therefore trust any incoming connections. at the stage where centre-to-centre communication takes place and communication has to interact with an academic organisation’s network security policies and firewalls, a question is raised regarding the authenticity of the remote connection and the trust surrounding from whom the connection may be coming. the complexity of the question increases with each new connection and potential participant. furthermore, when a remote learning scenario occurs, normally with students who study via blended distance-learning, the likelihood is that students have never met and therefore have not formed an albeit basic trust relationship. there is a need for social exchange to engender trust (xueming 2002), and this raises an immediate issue, insomuch as students are now expected to exchange their ip address, port or domain information to enable direct communication with someone with whom they have no personal contact or no potential conscience when it comes to unethical behaviour. with the lack of formal hierarchy, gurzick et al. (2011) identify that there need to be leaders and designated participants in an online collaborative environment. therefore in the establishment and creation of any one:one connection between distance based students (in respect of figure 2) who may have never met and therefore have a lower trust exchange there is the potential for abuse, via hacking, exploit, denial of service or cyber stalking. this would present many academic centres with the risk of multiple liabilities and thus encourage any reasonable figure 2. creating a multiuser connection from packet tracer. alt-c 2011 conference proceedings 103 academic or network manager to refuse to allow the multiuser feature to run on their system. resolution of this security threat is essential in maintaining trust between distance based students and remote centres. in exploring potential solutions, research focused on the many:many property of packet tracer, with the question: ‘‘what if a trusted intermediary was available?’’ with a many:many scenario, trusted secure devices could be created for academic centres as well as students to interconnect. now the trust is with the intermediary, as the domain and the port of the intermediary is known to everyone. no individual or academic entity would exchange any sensitive information regarding ip address, domain or tcp port. therefore creating a relay server, hosting a trusted instance of packet tracer, overcomes many of the immediate security and trust issues. with this in mind, all of the relays could be interconnected in a mesh (internet like) structure. having each relay connected to other relays would create a secure physical structure that would support a scalable community of practice, as new relays can easily be added as the community grows. overarching technical benefits would be: (1) secure communication using the ptmp (wang 2008). (2) the opportunity to implement a range of technical platforms for each relay implementation, assuring system agnosticism (no single preferred manufacturer or software vendor). (3) ensuring that the simulated network infrastructure is adaptable and diverse, presenting no restriction in the learning scenarios being developed by participants. (4) resilience, with multiple relays (some possibly acting as intermediaries), overcoming the indiscretions of technology. each has differing benefits, in learning delivery and technical deployment, and can be adapted to reflect of local academic and technical need. scenarios can be designed to encourage practice-based development and situated learning as described by lave and wenger (1991). students and academic centres have the freedom to create their own local networks on packet tracer and interact with the intermediary servers as they wish. in an experimental context, a default file has been provided to ensure everyone has a common entry point into the multiuser environment, giving a constructivist scaffold for a common level of entry, (rüschoff and ritter 2001). in addition, support has been given to the research by the cisco systems packet tracer development team in developing an add-in; the multiuser connection manager (mucm) tool manages redundant connections that have remained unused for a predetermined time period so that the system does not become overloaded with them. the pedagogy of creating an internet on the internet relying on one intermediary relay server limits the potential value in scale and reach of the collaborative opportunity available. the sole purpose of packet tracer is to engender understanding of complex network topologies and the interaction of protocols and devices in this environment. the multiuser feature, supported by the a. smith 104 formation of a collaborative interlinked mesh of intermediary relay servers, means that the system gains resilience and localisation, with the potential for a worldwide ‘ring’ of systems all interconnecting each accommodating geographic locale specific preferences and academic requirement. the practical outcome of such a mesh is the immediate advantage is to offer academic centres and individual learners a system able to provide them with the learning experience of building a complex internet work without the political and security complexities of using the ‘real’ internet. packet tracer provides access to otherwise inaccessible ipv4 and ipv6 address ranges, as well as an extensive range of networking technologies including: dhcp, nat, stp, vpn, qos, bgp, ospf, eigrp, rip, dot1q, vtp, in an environment allowing students to make mistakes and learn from their experiences without impacting others. in a collaborative distance based framework, the creation of an intermediary relay supported a mesh of relay servers. laurillard (2002) identified how academic centres as well as students can engage in distance based synchronous and asynchronous learning. the use of packet tracer in this context exemplifies these findings and allows local technological needs and conceptual needs of the students to be respected at the time of interaction. in addition, the flexible nature of packet tracer means that, with suitable core topology design, one group of learners can interact with the system at the same time independent of other learning collaborations. thus the distance based asynchronous and synchronous, collaboration can co-exist between classes of students in a specific geographic locale as well as internationally. assessment based learning may take place using the local packet tracer client in the activity mode, with students interacting in a ‘staged’ learning and problem solving scenario thereby providing discovery based exploratory learning. having a constructivist ethos, the concept of an environment open to personal interpretation of the student as well as the guiding academic is synonymous with many of the principles proposed by piaget (1978). the distributed learning methodology supported by packet tracer with the multiple sources of information, each giving form in a constructivist paradigm, gives credence to the emergent concept of connectivisim from landauer and dumais (1997). the first stage of research the research programme commenced in september 2009 with distance-learning students participating from the uk open university’s t216 (cisco networking/ ccna) and t824 (advanced networking/ccnp-bsci) courses. the test scenario was for each participant to connect to the relay server via two clients (both on the same local host), create an extended interior gateway routing protocol eigrp peering by: . adding a new local network into the autonomous system routing process for each client instance, . adding at least one workstation for each local network, . pinging the default gateway from each workstation and . pinging each other. alt-c 2011 conference proceedings 105 testing was confined to a one-hour window from 19:00 to 20:00 gmt. during this time over 40 participants joined with a peak of 15�20 different peers in the 19:20 to 19:40 window. to ensure a baseline network behaviour, one host ran a continuous simulated client-to-client ping and a continuous simulated ping to the intermediary server. the mucm, managed to rollback connections, ensuring deletion when disconnection notice expired after five minutes. at 19:40 (estimated) the packet tracer application crashed during a period of apparent peak activity. the research team subsequently revisited the test case by examining four scenarios in a controlled test environment: . emulating the same event, increasing load, to observe the factors leading to failure. . testing the system on linux without mucm interactivity. . testing the system with mucm in windows. . testing the system without mucm in windows. in addition, with all of the scenarios, upon failure the test was repeated without eigrp present. early findings have concluded that the 15 second hello timer for eigrp works within a lan based multiuser setting but as soon as internet communication is in play, the application, operating system and transmission latency all combine so that the next hello packet arrives too late for the simulator. this leads to a hold down state and increased application activity. an additional unanticipated challenge was the need to mentor/coach some of the participants during the session (via skype and email). anecdotally it would seem that for some, there was a difficulty in understanding the concept of how they were connecting to others in this remote environment. conclusions from the first stage of research the first test case proved the potential for disparate individuals from diverse locations to connect and engage in a semi-synchronous, primarily asynchronous practical activity. following the simple practical scenario, each participant completed the tasks set. therefore the development of group based participatory activities in the sphere of situated learning of lave and wenger (1991) in an online environment. a key finding, was the recognition that some of the participants needed to understand how to engage with the distance learning scenario. the challenge for the host was enabling the participating group to construct the mind-set to understand how they were each able to participate in this en masse exercise. to understand this, the researchers need to work with groups of students face-to-face to understand where the misunderstanding lay, through either experience or perception of the conceptual network structures. this is a constructivist model (piaget 1978) as students have to become active participants in the learning process, needing an initial environmental anchor to base their constructed ideas upon. the second stage of research a challenge in any activity requiring volunteers is the recruitment of those volunteers. since the initial research there have been three successful interactive a. smith 106 scenarios. the timing of these has been principally driven by availability. it was recognised that all participating students need to have a minimum of the cisco academy exploration or discovery first course behind them to comprehend the networking terminology and technology. to avoid other effects, the student age range was kept in a small window, with participating students being either second years on a high school equivalent technical vocational programme or first years on an undergraduate honours degree programme. this gave a range of 17�20 years of age with the majority in the 18/19 yearold age group. research sessions were in may, november and december 2010, reflecting academic calendars of each group and availability. the may and november sessions were with two groups of 18 and 11, 17�19 year olds at a college of further education. the december session was with a group of 30, 18-20 year-old year one undergraduate students at a london university. group selection was based on the class/ group available at the time suited to the demands of timetable, and availability of the willing volunteer teacher and the researcher. each of the sessions was scheduled for a three hour half day block. the first was a morning session 09:00 to 12:30 with break, the second session was 10:30 to 15:00 with intervening lunch break and the third session was a pm session from 14:00 to 17:00 with a short break. each session used the majority of the time, with an average of thirty minutes remaining to enable the students to complete an optional challenge activity. in each session the researcher acted as teacher/facilitator whilst the normal session teacher/ instructor acted as class-room support and secondary observer. each session was facilitated as an in-class teaching session, where each of the student participants was aware that they were helping to test the multiuser functionality of packet tracer and get in return additional networking skills (via practice-based learning). the use of a data projector connected to the teacher’s computer running the relay instance of packet tracer, provided the students with an essential conceptual cue for students supporting the work discussed by janitor and kniewald (2010). it enabled them to see how their own simulated lan and wan was behaving in relation to the greater relay based wan infrastructure. typical of many academic classrooms, each computer running during each session had the same hardware specification and operating system installation, including local policy constraints and user rights. this ensured that each student participating had the same technological advantages/disadvantages as all others during the lifecycle of each activity. the activity was managed in a systematic followthe-leader step-by-step format, keeping all students at same position in the process. the group-based activity is presented to students in two parts. the outcomes of the previous research showed that some did not understand the conceptual network and needed a scaffold to base their conceptual viewpoint upon. the student group would commence the activity with a formative warm-up exercise (the scaffold). in this, students were paired and given the task of creating a simple network of two hosts and being able to send a ‘virtual’ ping from one packet tracer instance to the other across the academic network as illustrated in figure 3. this short exercise sets the scene and ensures all participants are working from the same stand-point in their ability to use the software. all participants are already alt-c 2011 conference proceedings 107 low-level users of packet tracer, by virtue of their membership as students of the cisco academy programme. by introducing the participants to the ‘multiuser’ tool, their understanding of the additional tools available in packet tracer is increased. following the formative scaffold activity, the students participate in the largescale activity to build a simulated wan, with multiple individual simulated lan’s. the structure of the activity is a replication of the experimentation explored in the first stage of the research, with the relay no longer a remote server, but the teacher’s computer. this assists the learning process experienced by the students and observing instructor, discussed by laurillard (2002). in each session, the teacher’s computer is attached to a classroom data projector. each student is able to see their own multiuser connection locally as well as their remote connection on the teachers packet tracer instance thus reinforcing the assurance that they are correctly participating in the practical task and successfully building a remote (otherwise unseen) connection. the relay instance of packet tracer contains a simulated router with a series of simulated switches all connected to a core switch (figure 4). the simulated protocol selected is again eigrp and each student is presented with unique ip addresses to use during the exercise. routing protocols, as with many network technologies, can be configured in many different ways to achieve the same goal. to remove any confounding variance, all students are presented with an instruction sheet containing the commands they must use. the students own instance of packet tracer is a self-constructed system when assembled, and resembles the illustration in figure 5. the system is kept simple to reduce potential variance, by ensuring the students have specific devices and cable types to implement. conclusions from the second stage of research qualitative feedback was collected from each cohort, the intention was to understand their personal viewpoint of their experience in participating in the sessions as well as in the activities. at the end of each session, before departure, the students were asked to complete a short anonymous questionnaire, with questions listed in table 1. figure 3. peered example of packet tracer multiuser communication. a. smith 108 the questionnaire results are summarised in table 2. as the groups are small, and the questionnaire short, there are no missing responses; no additional personal information was requested. from the results in table 2, the dominant feedback implies that the students believed that using the simulated practical was a personal benefit, where the responses to questions one and three indicate a high percentage (table 3) of positive responses to the enquiry about their learning. it is notable that for the november cohort, the groups of students were in the early stages of their learning for the academic year, whereas the may and december cohorts were either at the end of their respective academic year or semester. questions two and four explored the student’s experience of packet tracer. apart from two outliers (reason unknown), question two indicated that the majority had not used the multiuser tool beforehand. with question four, the response indicates an interest held by the students to continue using the multiuser tool in packet tracer. this may have been stimulated by their feelings regarding the preceding session. in engaging with the practical activities, the students could be seen to personally link constructivist personal concepts as described by piaget (1978) and readily connect their own private concepts to a visual, simulated physical network environment. this is supported by parallel research described by lo (2010), who states that ‘‘appropriately used collaborative learning activities do promote student learning and student satisfaction’’. this is also supported by hare and graber (2007) figure 4. teacher relay-server structure. figure 5. student packet tracer instance. alt-c 2011 conference proceedings 109 in their research into how students engage in a constructivist-learning paradigm adopting and dismissing misconceptions when playing ‘invasion games’. the activities have demonstrated that once the student has been given an introduction to the multiuser tool, they are able to engage in a structured activity to build a complex simulated network environment, reflecting the model of situated learning discussed by lave and wenger (1991). formal questionnaire feedback as well as the in class anecdotal experience of the researcher, reflects an enthusiasm from the learners to continue studies using the packet tracer application in this mode. research discussed in this paper, allied with prior research suggests that the structured development of a system to create a simulated internet provides an alternate learning methodology for in-class as well as remote distance-based learners. future development the centres involved are willing to host future sessions, inviting the researcher back to continue the same exercise, as well as new different scenarios with their students. other centres are interested in participating in the research and are willing to engage in the activities described in this paper, as well as working towards more complex scenarios. the challenge for these centres, as for the original participants, is finding the right group at the right time in their year as well as in the study week. once the students and teacher/instructors become familiar with the technology and the constructivist learning experience, the plan is to move the activity to a remote ‘central’ relay server model, with more than one centre participating during the research and working on a collective multi-site learning activity. table 2. questionnaire result data. feedback may (18) november (11) december (30) question number y n y n y n 1 14 4 11 a 0 21 9 2 0 18 0 11 2 28 3 15 3 11 0 24 6 4 16 2 11 0 26 4 a this is earlier in the academic year for this cohort, where ip addressing was a new subject. table 1. questionnaire. question number question 1 has this exercise enhanced your practical understanding of ip addressing? (y/n) 2 have you used the packet tracer multiuser tool before this session? (y/n) 3 in your own view, has this given you some understanding of routing protocols? (y/n) 4 would you consider continuing to use packet tracer in the way demonstrated today? (y/n) a. smith 110 acknowledgements the author would like to thank dennis frezzo of cisco systems and his team for inspiration, insight and support as well as the enthusiastic support from cisco instructors. special thanks is due to paul wallin of kingston university (uk) and ken lamb, milton keynes college (uk) as well as the enthusiastic academy students alas too numerous to mention. references gurzick, d. et al. 2011. designing the future of collaborative workplace systems: lessons learned from a comparison with alternate reality games. proceedings of the 2011 iconference, 8�11 february 2011, seattle, washington state, usa. hare, m., and k. graber. 2007. investigating knowledge acquisition and developing misconceptions of high school students enrolled in an invasion games unit. high school journal 90, no. 4: 1�14. janitor, j., and k. kniewald. 2010. visual learning tools for teaching/learning computer networks. icns’10 proceedings of the 2010 6th international conference on networking and services, cancun, mexico. landauer, t.k., and s.t. dumais. 1997. a solution to plato’s problem: the latent semantic analysis theory of acquisition, induction and representation of knowledge, from http:// lsa.colorado.edu/papers/plato/plato.annote.html (accessed february 21, 2011). laurillard, d. 2002. rethinking university teaching: a framework for the effective use of educational technology, 2nd ed. london: routledge falmer, 154. lave, j., and wenger, e. 1991. situated learning: legitimate peripheral participation. cambridge: cambridge university press, 91. lo, c.c. 2010. student learning and student satisfaction in an interactive classroom. the journal of general education 59, no. 4: 238�63. piaget, j. 1978. success and understanding. london: routledge & kegan. prieto-blázquez, j. et al. 2008. an integrated structure for a virtual networking laboratory. ieee transactions on industrial electronics 55, no. 6: 2334�42. rüschoff, b., and m. ritter. 2001. technology-enhanced language learning: construction of knowledge and template-based learning in the foreign language classroom. computer assisted language learning 14: 219�32. smith, a., and n. moss. 2008. cisco networking: using skype and netlab� for distance practical learning. iadis e-learning 2008 conference, july 22�25, in amsterdam, netherlands. wang, m. 2008. packet tracer 5.0, packet tracer messaging protocol (ptmp), specifications document: cisco systems. xueming, l. 2002. trust production and privacy concerns on the internet, a framework based on relationship marketing and social exchange theory. industrial marketing management 31: 111�8. table 3. questionnaire result percentage. feedback as a percentage may (18) november (11) december (30) question number y n y n y n 1 77.7 22.3 100 0 70 30 2 0 100 0 100 6.7 93.3 3 83.3 16.7 100 0 80 20 4 88.8 11.2 100 0 86.6 13.4 alt-c 2011 conference proceedings 111 http://lsa.colorado.edu/papers/plato/plato.annote.html http://lsa.colorado.edu/papers/plato/plato.annote.html (page number not for citation purpose) 1 *corresponding author. email: brennandavid@outlook.com research in learning technology 2020. © 2020 d. brennan. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 research in learning technology vol. 28, 2020 original research article the effects of a moodle-based instructional unit on physical activity in schools on 15–20 years experienced permanent irish primary teachers physical activity knowledge, attitudes and behaviour david brennan* department of education, professional development service for teachers (pdst), galway, ireland (received 17 december 2018; final version revised 24 may 2019; accepted: 25 may 2019; published: 12 november 2020) childhood obesity is nearing epidemic proportions in ireland and abroad. childhood is a key period in the prevention of obesity and additional conditions that are associated with a sedentary lifestyle. the study aimed to discuss the effect of a short elearning course for irish primary teachers and its effect on their behaviours, attitudes and knowledge towards physical activity in schools. research on activity levels in children was also examined with schools being seen as a core change agent in the combat against childhood obesity. keywords: irish primary teachers; physical activity; knowledge; attitude; behaviour introduction a change in irish educational legislation and a national strategy to ensure a welldeveloped, educated workforce and increase competency in new areas of the curriculum is being implemented making teacher professional development mandatory (teachingcouncil.ie). this is quite a complex and challenging task to ensure training is relevant and flexible to cater for teacher experience and learning styles (education.ie). childhood obesity has emerged as a serious health challenge. globally 40 million children up to the age of 12 are overweight or obese (who 2011). obesity is defined as ‘a disease in which excess body fat has accumulated to an extent that health is adversely affected’ (irish heart 2012). the rise in the consumption of energy-dense and highly calorific foods with no relative increase in physical activity has resulted in weight gain. therefore, physical activity has a major role in the promotion of overall wellness and disease prevention in children (woods, moyna, and quinlan 2010). physical activity is defined ‘as any bodily movement produced by skeletal muscles that results in energy expenditure’ (chow 2008). evidence illustrates how behavioural patterns established at childhood are sustained through adulthood. unhealthy children become unhealthy adults and put financial pressure on the state. a positive teacher attitude towards physical activity promotion in schools is essential. attitude has been defined as a learned predisposition to respond positively or negatively to a specific object, situation, institution or person (aiken 2000). a change mailto:brennandavid@outlook.com http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v28.2203 http://teachingcouncil.ie http://education.ie d. brennan 2 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) in behaviour can be seen as a definition of learning. as attitude is a key factor in forming people’s behaviour, it is an essential aspect to teacher education. physical education at primary level school can be seen as a key agent in the promotion of physical activity as most children attend schools at some stage in their life with students spending half their waking hours at school (guinhouya et al. 2009). under irish regulations students must spend 183 days at school annually (department of education and skills 2012). the educational environment provides numerous opportunities for children to be physically fit and engage in physical activity through scheduled physical education (pe) classes and break times which allow students to partake in unstructured play. a recent study conducted of 55 child obesity prevention studies found that increasing physical activity sessions and developing physical activity skills during the school week were among the promising strategies for obesity prevention (waters 2011). the effective implementation of such would require a positive attitude and knowledgeable approach by teachers towards physical activity. at present, irish schools are allocated just over half of the european union average of 109 min of pe classes per week. primary teachers have identified that there is insufficient time in the day to cover all 11 subjects due to an overloaded curriculum (halbert 2010). in the irish context, 52% of school time is given to english, irish and mathematics. this imbalance leaves the remainder 48% of time for pe and other subjects. the european union education information network found that irish schools offered less pe time than any of their other european counterparts, where 45 h is the minimum. there has been an increase in pe participation in schools since 2004, with 31% of principals stating that pe was undertaken 2–3 times a week. by 2009 this had risen to 53%. schools participating in pe classes once a week have dropped from 62% in 2004 to 42% in 2009 (woods et al. 2010). forty-six minutes is reported as being the average amount of time given to pe. a decade ago the department of education acknowledged that the availability of amenities and pe resources was dependent on the ability of the school to offer a full and balanced pe curriculum (department of education and science). in 2010, woods et al. noted that 81% of principals stated that they had no access to onsite sports facilities for carrying out pe classes. cotter and the department of education (1977) also noted that less than 20% of schools had adequate equipment and pe apparatus. this has not improved since half of the schools in the kildare region do not have the recommended pe equipment for the revised curriculum (power 2009). a report carried out by the into highlights this lack of resources when schools were asked how adequate the equipment and resources in schools were to teach the following pe strands. strand good (%) adequate (%) poor (%) games 59 33 8 athletics 36 36 28 dance 26 39 35 outdoor and adventure 23 34 43 gymnastics 23 28 49 obtained: www.into.ie http://dx.doi.org/10.25304/rlt.v28.2203 http://www.into.ie research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 3 (page number not for citation purpose) recommendations for physical activity in schools children’s physical activity levels have decreased dramatically over the past decade. sedentary behaviours are continuing to replace outdoor activities that promote physical activity. children in the broader extent are being driven to school instead of walking and cycling and the participation in sporting exercises and physical leisure has decreased (rice and howell 2000). students should engage in 60 min of moderate-to-vigorous daily activity according to the health service executive (hse) ireland. the children’s sports participation and physical activity study found that 19% of 5–12-year-olds met the 60 min of moderate-to-vigorous physical activity recommendations with girls being less likely to meet the targets. over a quarter of the participants in the study were overweight, exhibited hypertension and were unfit (woods et al. 2010). the study is comparable to international research (cale and almond 1992; dale, corbin, and dale 2000) in which the following conclusions were drawn. 1. no clear urban/rural difference existed. 2. gender difference is evident in physical activity levels. 3. physical activity decreases with age. 4. low activity levels are evident among children. the growing up in ireland study highlighted that only 25% of children met the recommended 60 min of daily moderate-to-vigorous physical activity per day. twenty-nine per cent of boys were reported to meet the who (2009) recommendation compared to 21% of girls (murphy, 2010). similar patterns were seen during sports participation. a high proportion of children (97%) were recorded as taking part in sport at least once a week, with boys 56% more likely to take part in sport at least once a day with girls only 33%. as there is a proven link between physical activity and weight control, these findings indicate that the gender difference in weight control can be offset by increasing physical activity levels among girls which would be a key aspect of a physical activity teacher elearning course. incorporating ict into physical activity the increase usage of technology in all areas of society has prompted the integration of a module exploring how ict could be incorporated into supporting and enhancing physical activity promotion and pe departments (green 2002). cummings (2001) further suggests that the pervasion of ict in education is now impacting pe as much as on any other subject and has the capability to offset the challenge of access to resources for pe classes. learning is an active process and educationalists have realised the potential benefits interactive technology can contribute to encouraging active education (papert 2005). new online websites and ict tools have provided teachers with additional resources to contribute and enhance a physical activity lesson. however, little research has been conducted in the usage of ict in the development of students’ gross motor skill outside of a commercial which is especially true in pe lessons (thomas and stratton 2006). as students perform exercises and skills in their pe classes, pe teachers can use technological tools and systems to quantify processes and results to help them learn more about themselves according to kirkwood and manon, 2002 (papastergiou, 2010). http://dx.doi.org/10.25304/rlt.v28.2203 d. brennan 4 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) implementing ict into physical activity has produced mixed attitudes and opinions among teachers. computers can contribute to a sedentary leisure-time behaviours. it is estimated that children watch 3–4 h of tv at home, and a total of 74% of boys and 54% of girls spent some time each day playing video games, with 30% of boys and 12% of girls spending 1 h or more (growing up in ireland report 2009). they also send almost 100 text messages daily (pollak et al. 2010). this increase in children’s media consumption results in less time sleeping, reading, time for physical activity and social interaction which may lead to poor social skills in later life. in contrast, positive active technology usage has been used to promote physical activity and change exercise behaviours for a long period of time through pedometers, accelerometers and heart rate monitors which have been used as motivational tools. interactive online games and persuasive technology can be incorporated into teaching methodologies to promote physical activity and change exercise behaviour (marcus, ciccolo, and sciamanna 2009). evolution of approaches to physical activity to date pe within schools has been teacher led and focused on the traditional pedagogical foundations of the subject. little change has occurred in regard to methodologies, with teaching practices being replicated and set until recently. this traditional approach to pe has been described as ‘not fit for purpose’ by numerous academics (kirk 2010, lawson 2009). physical activity itself has been reflected on in terms of sports techniques on sports pitches and in gymnasiums in schools, which isolates the understanding and appreciation by students of the lifelong benefits of the subject. lawson’s (2009) argument that pe is not capable of educating in a post-industrial age reflects how ict could be incorporated into the subject in the future. it could aid in ensuring that the subject is: individualised: students must be helped at their own ability levels for optimal personal improvement with subject matter being differentiated, which is in contrast to the traditional approach providing the same instruction for all students at the same time. fair: students are assessed based on personal improvement and assisted in goal setting for improvement of health-related fitness. the traditional approach used standardised fitness test scores to determine grades and awards. educational: students must learn why physical activity is important and how it is beneficial. the traditional approach told students what to do, but not why. incorporating ict into the subject’s pedagogical framework can enhance learning and understanding, utilising gardner’s theory of multiple intelligences. ict resources and media can be used to present and individualise material and instruction in a way that children find it appealing to their intelligences productively and relevant to learning style (mokhtar, majid, and foo 2008). teacher professional development in physical activity teacher professional development in literary terms is broadly defined as in-service education, staff development, teacher development, career development and lifelong learning. it is ‘any activity which enhances their knowledge and skills and enables them to consider their attitudes and approaches to the education of children, with a view to improve the quality of teaching process’ (coulter 2012). it has become a priority in many countries which is highlighted in the organisation for economic http://dx.doi.org/10.25304/rlt.v28.2203 research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 5 (page number not for citation purpose) cooperation and development (oecd) report ‘teachers matter’ (o’gorman,2010). an interconnected approach to teacher education from initial teacher training through to career-long learning needs to be established (organisation for economic co-operation and development 2005). elearning courses can play a more significant role in when integrated into a supported professional development framework. when teachers are given the opportunity, via high-quality professional development, to learn new strategies for teaching to rigorous standards, they report changing their teaching in the classroom (alexander, heaviside, and farris 1998). school policies have an effect on the promotion of physical activity and cardiovascular fitness levels of children. through implicit and explicit policies, schools can endorse procedures during the course of the day to positively influence the promotion of physical activity levels among students in schools. elearning and changes to teacher’s attitudes and behaviours teacher attitudes are important. a sustainable change to teacher’s attitudes and behaviours towards physical activity must occur at three levels (sallis 1997). these include a change in resources, change in teaching approaches and a change in beliefs. this may be due to a lack of analysis by teachers of relevant research in the area of physical activity according to ward and o’sullivan (1998). huberman (1995) suggests the change process for teachers is cyclical in nature, with the change in beliefs leading to an adoption of practices that brings positive change to students’ learning and aca demic results (saunders 2013). figure: huberman (1995) this model corresponds to desimone’s model of teacher change which sees professional development as the key agitator in changing teacher attitudes and behaviours (desimone, smith, and philips 2011). teachers completing the elearning course came from a wide range of previous experiences, knowledge, self-direction, interests and competencies. an elearning course must accommodate this diversity and create a community of learners (specks 2001). the need for a positive teacher-led approach to physical activity has been identified as the key opportunity to ‘encourage’ school-age http://dx.doi.org/10.25304/rlt.v28.2203 d. brennan 6 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) children to be involved daily in 60 min or more of moderate-to-vigorous physical activity (chow, mckenzie, and louie 2008; strong et al. 2005). elearning content needs to be relevant to the education sector ensuring that teachers are motivated by the material and ensure that it adds to their own professional experiences (pope and o’sullivan 1998). it facilitates critical thinking needs through peer assessment and contributions which is ‘the capacity to question the actions, justifications and decisions, and to  imagine alternatives to current structures and moralities that are fairer and more  compassionate predictable preferences and capabilities in learning’ ( edmondson 2007a). determinants for a successful elearning course in physical activity the elearning course focused on physical activity content that is educationally relevant to irish primary teachers and aimed to target teachers’ learning needs to have positive effects on teacher knowledge, attitudes and instructional practices. in order for a successful elearning professional development course participants’ characteristics must be taken into account (palloff and pratt 2001). according to research effective learners seek further professional development because they have higher expectations, they enjoy learning, can work independently and recognise the benefits of peer interaction. participant’s reaction to elearning content is related to user’s attitude towards elearning, personal interest in the area of physical activity and satisfaction with technology during a learning experience (curran, fleet, and kirby 2010). research suggests successful elearning courses depend on individual learner variables, environmental variables, technology variables, contextual variables and pedagogic variables (attwell 2006). for a successful elearning course, each of these factors must be taken into consideration. according to phipps and merisotis (1999), three determinants illustrate the success of an elearning course: student learning outcome, student satisfaction and student’s attitude to elearning and the technologies used. elearning course content: an increase in teacher’s knowledge and skills in this area is linked to the degree the course focuses on content knowledge rather than generic professional development focusing on teaching strategies (berry and van dreil 2012). the existence of strong, relevant and educational focused content is essential for teacher professional development. the elearning course should have evidence of curricular strategies that have a high level of probability to affect students’ learning and adopt teacher’s attitudes (joyce and showers 2002). petrie (2009) determined that teachers’ confidence and motivation to teach pe was enhanced if they were provided with the learning opportunities regarding pe ped agogical knowledge, which must be taken into consideration when completing an elearning course. communal setting: a change in teachers’ attitude towards physical activity occurs in a communal setting (joyce and showers 2002). teachers must be given the opportunity to interact, discuss methodologies, proffer advice and share their own experiences (bandura 1995). a virtual community of practice, which, according to wenger and snyder (2000), is a group of people bound together through a shared passion for a joint enterprise. this social theory of learning is facilitated through forum postings on the elearning course. course participants can share resources, experiences, learning http://dx.doi.org/10.25304/rlt.v28.2203 research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 7 (page number not for citation purpose) tools and methodologies. this also facilitates peer persuasion which is a powerful method of changing attitudes (schunk, hanson, and cox 1987). an online community of learners develops and maintains a sense of efficacy regarding new physical activity teaching strategies (kennedy 1987). this group decision support system offers participants a way to catalogue, more comprehensively, the group’s total information and then share that information collectively. (blanco et  al. 2009). this aspect reflects the functional theory of group decision-making, which would be necessary to ensure that teachers decided to increase time allocated towards physical activity. participants through the forum postings are more likely to make use of group procedures to enhance the way they gather, analyse and weight information (wittenbaum, hollingshead, and botero 2004). literature review summary research from this review of literature has shown that initial teacher training accompanied by continual lifelong learning through elearning courses can be seen as an essential aspect in the promotion of physical activity in schools. research has appropriately identified schools as being a key setting in the promotion of physical activity which is an essential component of a healthy lifestyle and an important aspect in the education of children (biddle and mutrie 2007). children need 60 min of play with moderate-to-vigorous physical activity every day to grow up to a healthy weight. consequently, teachers must be up to date with all modern methodologies and strategies for promoting physical activity. a positive attitude towards physical activity is required by teachers who will commit to professional development and learning when the goals and objectives are considered realistic and important to them (sallins et al. 1997). ict solutions if they are known to teachers allows for then to be incorporated into a physical activity lesson. this provides a solution to the lack of resources in some schools. the hypothesis of the research is that through completing the elearning course teachers will increase their knowledge of the benefits of physical activity resulting in a more positive attitude towards the subject which will encourage them to incorporate it more into the school environment. method a combination of methods was chosen as it seemed most appropriate to address the research question. they are complementary in the essence of following the results from the questionnaires with a focus group interview to provide the opportunity to elaborate on findings from the initial research method. through using quantitative and qualitative approaches, the research incorporated the most valuable features of each method (bryman 2012). one short questionnaire was developed and used (see appendix 1). in the first instance, the questionnaire was used for establishing initial attitudes, practices and knowledge about physical activity in the educational environment. post-course, the questionnaire was administered to examine the change in knowledge and attitude towards physical activity after the short elearning course. a post-study semi-structured focus group was then undertaken which was led by the tutor. participants were asked about their experience of the elearning course. http://dx.doi.org/10.25304/rlt.v28.2203 d. brennan 8 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) the data collected from this small discussion were used to supplement the preand post-questionnaire data and build a more complete picture of the overall study. survey the preand post-course questionnaire was designed and administered through a combination of survey monkey and articulate storyline software. the design process was undertaken after the literature review and underwent modifications in response to feedback from individuals and pilot trials. the pre-course questionnaire was administered directly before the commencement of the elearning course with the post-course questionnaire directly after. a likert scale was administered (1 = strongly disagree, 5 = strongly agree) regarding teachers’ attitudes and practices towards physical activity. to maximise the response rate and due to the timing of the research over teacher holiday period, the questionnaire was confined to five questions. consequentially, the detail and depth of research which was initially envisaged was reduced, but was compensated with questions in the focus group being expanded to include elements that were withdrawn from the questionnaires. focus group a focus group was held with seven participants from the study group. the selection of participants for the teacher focus group involved purposeful sampling (merriam and tisdell 2015). this took place in galway for convenience due to transport and location. all participants were contacted by phone to arrange a time and date most suitable. on the day the researcher had an accompanying information sheet which outlined the study’s nature and purpose and provided space for notes on the main points discussed. a number of open set questions were posed with the researcher recording answers. this method, although it went exceptionally well, posed some considerations for the research. these included order effects where the participant’s performances may have been affected by either internal or external effects. social desirability could play a part in this process also where units behave differently to present themselves in what they believe to be the most socially acceptable light. (birley and moreland 2014) study design participants participants were selected by the researcher with n = 7. the teachers were selected from diverse settings by non-probability sampling from a selection of different primary school types (deis, rural and urban). this was chosen to avoid a misrepresentation of different sections and a sampling error. data analysis techniques employed the data from the questionnaire were collected and qualitatively correlated through tools available on survey monkey. http://dx.doi.org/10.25304/rlt.v28.2203 research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 9 (page number not for citation purpose) instructions to participants participants were emailed instructions for accessing the course and what the course entailed. the study group email included a link to the course, with each individual receiving the password and username for the website, which they were asked to keep confidential. results all participants completed every component of the trial – pre-questionnaire, post-questionnaire – and were available for the focus group discussion session. through the online evaluation questionnaire, participants rated their perceived attitude towards physical activity before and after the trial period along with their confidence and knowledge of incorporating physical activity in schools on a 5-point likert scale (1 = least difficult, 5 = most difficult). for this study, n = 7 participants. the results of the questionnaire are outlined below. pre-course questionnaire weighted average responses to pre-course questionnaire i am confident in teaching physical education question 1 i understand the benefits ict can bring to promoting and improving physical activity participation in schools question 2 i have a good range of ideas for increasing physical activity in the school and have competence in implementing them into everyday school life question 3 i understand the role i have as a teacher to promote it within a school question 4 i understand the importance of promoting physical activity and the overall benefits it brings to students question 5 http://dx.doi.org/10.25304/rlt.v28.2203 d. brennan 10 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) data table reflecting responses to pre-course questionnaire post-course questionnaire weighted average responses to post-course questionnaire i am confident in teaching physical education question 1 i understand the benefits ict can bring to promoting and improving physical activity participation in schools question 2 i have a good range of ideas for increasing physical activity in the school and have competence in implementing them into everyday school life question 3 i understand the role i have as a teacher to promote it within a school question 4 i understand the importance of promoting physical activity and the overall benefits it brings to students question 5 http://dx.doi.org/10.25304/rlt.v28.2203 research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 11 (page number not for citation purpose) data table reflecting responses to post-course questionnaire findings the first question in the survey asked about teacher’s confidence in teaching physical activity which would be representative of their initial knowledge and attitudes towards the area. initially, 28.57% participants felt they were confident in teaching physical activity; this increased to 71.43% in the post-course questionnaire, which highlighted the benefit that an elearning course could bring to increasing teachers’ confidence in incorporating physical activity throughout the school environment. there was a significant increase between the results of the second question pre and post-elearning course with a marked improvement on the questions average rating from 1.71 in the first questionnaire to 4.43 in the second. this reflected the need for any future teacher professional development course to include modern ict methods and resources in the course content that are suitable for physical activity. the third question highlighted that there was an increase in the study group’s knowledge of ideas and strategies for increasing physical activity in schools post-elearning course with a 28.57% increase in participants strongly agreeing with the statement. there was a significant increase in participants strongly agreeing with the fourth and fifth question, where participants stated that they had an increased appreciation of the teacher’s role in promoting physical activity and the benefit that this would bring to students and the overall wellness within the school environment. the post-course focus group evaluations of the e-learning course were very positive. participants in the study group found the following features of the module most helpful: interactivity, interface of the elearning course, the modern ideas and strategies highlighted and their relevancy to the education environment. in addition, participants commented that short questions about module concepts were useful at a modules completion. a selection of comments that represent the focus group session is outlined below. information was well presented, interesting and representative for the primary school environment. i enjoyed looking over material again through the course and the ability to look back on topics. http://dx.doi.org/10.25304/rlt.v28.2203 d. brennan 12 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) easy to follow, very accessible and would suit time wise to complete the course at my own pace. i have taken one online course previously and found this shorter succinct course on physical activity more appropriate and interesting. i enjoy elearning courses as they give me the flexibility to complete the material in my own time in comparison to face to face courses. it was very informative and interesting. i believe it would be suitable for both an introductory for novices in the area of physical activity and for those who have more experience and knowledge in the area. it was very relevant for the educational environment but also broadened my own interest in the field of promoting physical activity. in addition to the positive feedback, participants also suggested areas for improvement. a number suggested more usage of multimedia material, in particular instructional videos, on improving physical activity. although they enjoyed reading the material, a number expressed their view that there should be audio overview. a few in the study group had difficulty with internet speeds due to a rural location and found that material was slow to download. the research group consisted of teachers with between 15 and 20 years of teaching experience. they had no initial experience of elearning during their initial teacher education at college and had a basic knowledge of educational ict equipment and resources. they were cognisant of the new digital strategy in schools and adoption in educational policies being more digitally orientated. they have more recently accessed elearning content for cpd. conclusion this short study may have important implications for the design of future elearning material for irish primary teachers. catering content that is relevant to the irish educational sector and ensuring that it appeals to participants’ interests would optimise motivation for learning and participation. in particular, the findings suggest that a targeted elearning course, which combines examples of irish schools and general information, is more effective than generic courses and online learning material and would positively influence teachers’ attitudes towards physical activity. change is the only constant in the educational environment particularly with the introduction and development of new ict resources and methodologies. teacher professional development needs to be proactive and respond to these changes as initial teacher training needs to be supplemented. elearning courses, in particular, offer the flexibility and adaptability to cater for this need. teacher participation in online professional development can influence attitudes and behaviours which, in turn, lead to a progressive adoption of teaching practices, translating into improvements in targeted student outcomes. references aiken, l. (2002) attitudes and related psychosocial constructs: theories, assessment, and research, sage. aiken, r. l. (2000) psychological testing and assessment, 10th edn, allyn and bacon, boston, ma. doi: 10.1177/001316447103100436 http://dx.doi.org/10.25304/rlt.v28.2203 https://doi.org/10.1177/001316447103100436 research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 13 (page number not for citation purpose) alexander, d., heaviside, s. & farris, e. (1998) status of education reform in public elementary and secondary schools: teachers’ perspectives, u.s. department of education, national centre for education statistics, u.s. government printing office, washington, dc. doi: 10.3886/icpsr06943 alvunger, d., sundberg, d. & wahlström, n. (2017) teachers matter–but how? anmol, r. k. (2014) ‘potential of ict in improving performance in sports’, ijar, vol. 1, no. 1, pp. 49–52. armstrong, n. & welsman, j. r. (2006) ‘the physical activity patterns of european youth with reference to methods of assessment’, sports medicine, vol. 36, no. 12, pp. 1067–1086. attwell, g. (2006) ‘evaluating e-learning: a guide to the evaluation of e-learning’, evaluate europe handbook series, vol. 2, pp. 1610–0875. bailey, r. (2005) ‘evaluating the relationship between physical education, sport and social inclusion’, educational review, vol. 57, no. 1, pp. 71–90. bandura, a. (1995) self-efficacy in changing societies, cambridge university, new york, ny. doi: 1017/cbo9780511527692 bates, h. (2006) daily physical activity for children and youth: a review and synthesis of the literature. berry, a. & van driel, j. h. (2012) ‘teaching about teaching science: aims, strategies, and backgrounds of science teacher educators’, journal of teacher education, vol. 64, no. 2, pp. 117–128. doi: 10.1177/0022487112466266 biddle, s. j. & mutrie, n. (2007) psychology of physical activity: determinants, well-being and interventions, routledge. birley, g. & moreland, n. (2014) a practical guide to academic research, routledge, london. doi: 10.4324/9781315041964 blanco, h., et al., (2009) ‘hot, congested, crowded and diverse: emerging research agendas in planning’, progress in planning, vol. 71, no. 4, pp. 153–205. bryman, a. (2012) social research methods, 4th edn, oxford publisher. doi: 10.2307/j. ctt1t892hf campbell, j., gilmore, l. & cuskelly, m. (2003) ‘changing student teachers’ attitudes towards disability and inclusion’, journal of intellectual and developmental disability, vol. 28, no. 4, pp. 369–379. caspi, a., elder, g. h. & bem, d. j. (1988) ‘moving away from the world: life-course patterns of shy children’, developmental psychology, vol. 24, no. 6, p. 824. chi, t. (2009) ‘physical activity is defined as “any bodily movement produced by skeletal muscles that results in caloric expenditure (pender, 1996, p. 185). definitions of exercise are complex and vary according to scientific’, complementary & alternative therapies in nursing, vol. 349. childs, s., et al., (2005) ‘effective e-learning for health professionals and students – barriers and their solutions. a systematic review of the literature – findings from the hexl project’, health information & libraries journal, vol. 22, pp. 20–32. chow, b. c., mckenzie, t. l. & louie, l. (2008) ‘children’s physical activity and environmental influences during elementary school physical education’, journal of teaching in physical education, vol. 27, no. 1, pp. 38–50. doi: 10.1123/jtpe.27.1.38 cotter, m. (1977) unpublished thesis, trinity college dublin, cited in keating, n (1989) pe in the primary school, peai annual conference 1989. doi: 10.1111/j.1365-2451.1989. tb00675.x coulter, m. (2012) teaching outdoor and adventure activities: describing, analysing and understanding a primary school physical education professional development programme, doctoral dissertation, dublin city university. cummings, m. (2001) ‘internet, goal net or both: pe and the introduction of ict into the curriculum’, the british journal of teaching physical education, vol. 32, no. 3, pp. 19–22. doi: 10.4324/9781315113524-3 http://dx.doi.org/10.25304/rlt.v28.2203 https://doi.org/10.3886/icpsr06943 https://doi.org/10.3886/icpsr06943 https://doi.org/1017/cbo9780511527692 https://doi.org/10.1177/0022487112466266 https://doi.org/10.4324/9781315041964 https://doi.org/10.2307/j.ctt1t892hf https://doi.org/10.2307/j.ctt1t892hf https://doi.org/10.1123/jtpe.27.1.38 https://doi.org/10.1111/j.1365-2451.1989.tb00675.x https://doi.org/10.1111/j.1365-2451.1989.tb00675.x https://doi.org/10.4324/9781315113524-3 d. brennan 14 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) curran, v. r., fleet, l. & kirby, f. (2010) ‘a comparative evaluation of the effect of internet-based cme delivery format on satisfaction, knowledge and confidence’, medical education, vol. 10, no. 10, pp. 1–7. doi: 10.1186/1472-6920-10-10 dale, d., corbin, c. b. & dale, k. s. (2000) ‘restricting opportunities to be active during school time: do children compensate by increasing physical activity levels after school?’, research quarterly for exercise and sport, vol. 71, no. 3, pp. 240–248. doi: 10.10 80/02701367.2000.10608904 desimone, l., smith, t. & philips, j. r. (2011) teacher participation in content-focused professional development & the role of state policy. dobbins, m., et al., (2013) ‘school-based physical activity programs for promoting physical activity and fitness in children and adolescents aged 6 to 18’, cochrane database of systematic reviews, vol. 2. edmondson, b. (2007a) globalized e-learning cultural challenges, idea group inc (igi). edmondson, b. (2007b) ‘utilizing multiple intelligences in e-learning design’, international journal of learning, vol. 13, no. 12, pp. 35–41. doi: 10.18848/1447-9494/cgp/ v13i12/45133 enwald, h. p. k. & huotari, m. l. a. (2010) ‘preventing the obesity epidemic by second generation tailored health communication: an interdisciplinary review’, journal of medical internet research, vol. 12, no. 2, p. e24. freeman, s., wright, a. & lindqvist, s. (2010) ‘facilitator training for educators involved in interprofessional learning’, journal of interprofessional care, vol. 24, no. 4, pp. 375–385. futrell, j. (2009) a comparative study of virtual high school instruction versus traditional instruction of high school student outcomes and attitudes in physical education, doctoral dissertation, capella university. gagnon, m. p., et al., (2012) ‘systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals’, journal of medical systems, vol. 36, no. 1, pp. 241–277. green, n. (2002) ‘using ict within pe – its impact on a working department’, the british journal of teaching physical education, vol. 33, no. 2. doi: 10.1080/00971170.1977.10618814 guinhouya, b. c., et al., (2009) ‘how school time physical activity is the “big one” for daily activity among schoolchildren: a semi-experimental approach’, junior physical act health, vol. 6, no. 4, pp. 510–519. doi: 10.1123/jpah.6.4.510 güngör, n. k. (2014) overweight and obesity in children and adolescents. journal of clinical research in pediatric endocrinology, vol. 6, no. 3, p. 129. guskey, t. r. (2002) ‘professional development and teacher change’, teachers and teaching, vol. 8, no. 3, pp. 381–391. hardman, k. & marshall, j. (2005) ‘update on the state and status of physical education world-wide’, comunicación presentada en el 2nd world summit on physical education, magglingen, suiza. halbert, j., & macphail, a. (2010). curriculum dissemination and implementation in ireland: principal and teacher insight. irish educational studies, vol. 29, no. 1, pp. 25–40. hu, g. (2005) ‘professional development of secondary efl teachers: lessons from china’, teachers college record, vol. 107, no. 4, pp. 654–705. jones, l. (2012) an investigation into the impact of ict strategies in acheiveing high quality physical education, with the use of a good exemplar of practice, doctoral dissertation, university of wales institute cardiff. joyce, b. & showers, b. (2002) student achievement through staff development, association for supervision and curriculum development, alexandria, va. doi: 10.1016/ b978-012750053-9/50009-7 kennedy, m. m. (1987) ‘chapter 4: inexact sciences: professional education and the development of expertise’, review of research in education, vol. 14, no. 1, pp. 133–167. kirk, d. (2005) ‘physical education, youth sport and lifelong participation: the importance of early learning experiences’, european physical education review, vol. 11, no. 3, pp. 239–255. http://dx.doi.org/10.25304/rlt.v28.2203 https://doi.org/10.1186/1472-6920-10-10 https://doi.org/10.10 https://doi.org/10.18848/1447-9494/cgp/v13i12/45133 https://doi.org/10.18848/1447-9494/cgp/v13i12/45133 https://doi.org/10.1080/00971170.1977.10618814 https://doi.org/10.1123/jpah.6.4.510 https://doi.org/10.1016/ research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 15 (page number not for citation purpose) kirk, d. & vertinsky, p. (eds.). (2016) the female tradition in physical education: women first reconsidered, routledge. lawson, h. a. (2009) ‘paradigms, exemplars and social change’, sport, education & society, vol. 14, no. 1, pp. 97–119. doi: 10.1071/he12916 marcus, b. h., ciccolo, j. t., & sciamanna, c. n. (2009) using electronic/computer interventions to promote physical activity. british journal of sports medicine, vol. 43, no. 2, pp. 102–105. merriam, s. b. & tisdell, e. j. (2015) qualitative research: a guide to design and implementation, john wiley & sons. mokhtar, i., majid, s. & foo, s. (2008) ‘teaching information literacy through learning styles: the application of gardner’s multiple intelligences’, journal of librarianship and information science, vol. 40, p. 93. doi: 10.1177/0961000608089345 morgan, p. j. & hansen, v. (2008) ‘classroom teachers’ perceptions of the impact of barriers to teaching physical education on the quality of physical education programs’, research quarterly for exercise and sport, vol. 79, no. 4, pp. 506–516. murphy, f., & cosgrave, c. (2010) vision into action: pre-service teachers and physical education. fifth physical education, physical activity and youth sport (pe pays), 37. national association for sport and physical education. (2010) shape of the nation report. o’gorman, e., & drudy, s. (2010) addressing the professional development needs of teachers working in the area of special education/inclusion in mainstream schools in ireland. journal of research in special educational needs, vol. 10, pp. 157–167. o’brien, w., belton, s. & issartel, j. (2016) ‘the relationship between adolescents’ physical activity, fundamental movement skills and weight status’, journal of sports sciences, vol. 34, no. 12, pp. 1159–1167. palloff, r. & pratt, k. (2001) lessons from the cyberspace classroom, jossey bass, san francisco, ca. doi: 10.1145/566891.566895 papastergiou, m., (2010) enhancing physical education and sport science students’ self-efficacy and attitudes regarding information and communication technologies through a computer literacy course. computers & education, vol. 54, no. 1, pp. 298–308. papert, s. (2005) ‘a conversation with seymour papert, marvin minsky and alan kay’, communications of the acm, vol. 48, no. 1, pp. 35–38. philips, k. j., desimone, l. & smith, t. m. (2011) ‘teacher participation in content-focused professional development & the role of state policy’, teachers college record, vol. 113, no. 11, pp. 2586–2621. phipps, r. & merisotis, j. (1999) what’s the difference? a review of contemporary research on the effectiveness of distance learning in higher education, the institute for higher education policy, washington, dc. doi: 10.1080/ 00091389909602685 poekert, p. e. (2012) ‘teacher leadership and professional development: examining links between two concepts central to school improvement’, professional development in education, vol. 38, no. 2, pp. 169–188. pollak, j. g., et al., (2010) ‘it’s time to eat! using mobile games to promote healthy eating’, pervasive computing, ieee, vol. 9, no. 3, pp. 21–27. pollock, m., et al., (2010) ‘“but what can i do?”: three necessary tensions in teaching teachers about race’, journal of teacher education, vol. 61, no. 3, pp. 211–224. pope, c. c., & o’sullivan, m. (1998) culture, pedagogy and teacher change in an urban high school: how would you like tour eggs done? sport, education and society, vol. 3, no. 2, pp. 201–226. power, a., (2009) ‘it’s the system working for the system’: carers’ experiences of learning disability services in ireland. health & social care in the community, vol. 17, no. 1, pp. 92–98. rice, m. h. & howell, c. c. (2000) ‘measurement of physical activity, exercise and physical fitness in children: issue and concerns’, journal of pediatric nursing, vol. 15, no. 3, pp. 148–156. doi: 10.1053/jn.2000.6019 http://dx.doi.org/10.25304/rlt.v28.2203 https://doi.org/10.1071/he12916 https://doi.org/10.1177/0961000608089345 https://doi.org/10.1145/566891.566895 https://doi.org/10.1080/00091389909602685 https://doi.org/10.1053/jn.2000.6019 d. brennan 16 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) sallis, j. f., et al., (1997) ‘the effects of a 2-year physical education program (spark) on physical activity and fitness in elementary school students. sports, play and active recreation for kids’, american journal of public health, vol. 87, no. 8, pp. 1328–1334. schunk, d. h., hanson a. r. & cox, p. d. (1987) ‘peer model attributes and children’s achievement behaviors’, journal of educational psychology, vol. 79, pp. 54–61. saunders, r. (2013) the role of teacher emotions in change: experiences, patterns and implications for professional development. journal of educational change, vol. 14, no. 3, pp. 303–333. speck, b. w. (2001) why service-learning? new directions for higher education, vol. 2001, no. 114, pp. 3–13. spinola, a.m. & castro. (2014) ‘interventions for preventing obesity in children’, sao paulo medical journal [online], vol. 132, no. 2, pp. 128–129, available at: www.ncbi.nlm.nih. gov/pubmed/24714997 swinburn, b. a., et al., (2004) diet, nutrition and the prevention of excess weight gain and obesity. public health nutrition, vol. 7, no. 1a, pp. 123–146. telford, r. m., et al., (2013) ‘longitudinal patterns of physical activity in children aged 8 to 12 years: the look study’, international journal of behavioral nutrition and physical activity, vol. 10, no. 1, p. 81. thomas, a. & stratton, g. (2006) ‘what we are really doing with ict in physical education: a national audit of equipment, use, teacher attitudes, support, and training’, british journal of educational technology, vol. 37, no. 4, pp. 617–632. doi: 10.1111/j.1467-8535.2006.00520.x tudor-locke, c., et al., (2005) ‘pedometer-determined step count guidelines for classifying walking intensity in a young ostensibly healthy population’, canadian journal of applied physiology, vol. 30, no. 6, pp. 666–676. vale, s., et al., (2010) ‘compliance with physical activity guidelines in preschool children’, journal of sports sciences, vol. 28, no. 6, pp. 603–608. wallhead, t. & o’sullivan, m. (2005) ‘sport education: physical education for the new millennium?’, physical education and sport pedagogy, vol. 10, no. 2, pp. 181–210. waters, e., et al. (2014). interventions for preventing obesity in children. sao paulo medical journal, 132(2), 128–129. wenger, e. & snyder, w. (2000) ‘communities of practice: the organizational frontier’, harvard business review, vol. 78, no. 1, pp. 139–145. doi: 10.1016/ b978-0-7506-7293-1.50004-4 wittenbaum, g. m., hollingshead, a. b. & botero, i. c. (2004) ‘from cooperative to motivated information sharing in groups: the impact of members task relevant status’, small group research, vol. 29, pp. 57–84. doi: 10.1177/1046496498291003 woods, c. b., et al., (2007) the take part study, physical activity research for teenagers, a report for the midlands area, school of health and human performance, dublin city university. woods, c. b., et al., (2010) the children’s sport participation and physical activity study (csppa study), school of health and human performance, dublin city university and the irish sports council, dublin, ireland. doi: 10.18552/ijpblhsc.v6i1.422 available at: www.who.org world health organization available at: www.ncbi.nlm.nih.gov/pmc/articles/pmc1424733/ available at: www.irishsportscouncil.ie available at: www.growingup.ie available at: www.oecd.org http://dx.doi.org/10.25304/rlt.v28.2203 http://www.ncbi.nlm.nih.gov/pubmed/24714997 http://www.ncbi.nlm.nih.gov/pubmed/24714997 https://doi.org/10.1111/j.1467-8535.2006.00520.x https://doi.org/10.1111/j.1467-8535.2006.00520.x https://doi.org/10.1016/ https://doi.org/10.1177/1046496498291003 https://doi.org/10.18552/ijpblhsc.v6i1.422 http://www.who.org http://www.ncbi.nlm.nih.gov/pmc/articles/pmc1424733/ http://www.irishsportscouncil.ie http://www.growingup.ie http://www.oecd.org research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 17 (page number not for citation purpose) appendixes questionnaire dear sir/madam i am currently undertaking a masters in e-learning design and development at cork institute of technology. in fulfillment of my dissertation i am required to research a topic area. the topic i have chosen is: the effects of a moodle-based instructional unit on physical activity in schools on irish primary teachers physical activity knowledge, attitudes, and behavior i would be very grateful if you could, complete the questionnaire through the following link. needless to say all information provided will be treated with strict confidence and individual firms will not be identified. the questionnaire can be complete online, and should only take several minutes of your time. i would be very grateful if you could complete within one working week. yours faithfully dave brennan statement strongly disagree disagree neither disagree nor agree agree strongly agree i am confident in teaching physical activity i understand the benefits ict can bring to promoting and improving physical activity participation in schools i have a good range of ideas for increasing physical activity in the school and have competence in implementing them into everyday school life i understand the role i have as a teacher to promote it within a school  i understand the importance of promoting physical activity and the overall benefits it brings to students focus group material introduction by facilitator hello, my name is david brennan and i will be the facilitator for this focus group. the focus group is part of a masters in elearning project i am completing with cork institute of technology. thank you for taking the time to participate. this focus group was assembled to assess the effects of a moodle-based instructional unit on physical activity in schools on irish primary teacher’s physical activity knowledge, attitudes, and behavior. http://dx.doi.org/10.25304/rlt.v28.2203 d. brennan 18 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) during this focus group i will ask questions and facilitate a conversation about how you found the course and your ideas and thoughts on the benefit of the short learning course. please keep in mind that there are no “right” or “wrong” answers to any of the questions i will ask. the purpose is to stimulate conversation and hear the opinions of everyone in the room. i hope you will be comfortable speaking honestly and sharing your ideas with us. please note that i will be taking notes during the focus group to ensure we adequately capture your ideas during the conversation. however, the comments from the focus group will remain confidential and your name will not be attached to any comments you make. do you have any questions before we begin? focus group questions 1. let’s do a quick round of introductions. can each of you tell the group your name, where you are working, what class you have and briefly how you feel about teaching physical activity? 2. what do you understand by the term teacher professional development? 3. what do you think the role of e-learning is in teacher education? considering the potential benefits, disadvantages and role within an institution that pre-dominantly teaches through face-to-face sessions. 4. what is the role that physical activity can play in the school environment? 5. are there enough professional development courses available in the physical activity area and do they cover the most modern methods and strategies to increase physical activity in schools. 6. what positive impact has the course had on your outlook in regards to physical activity? 7. did you enjoy the course and was the information presented relevant to the irish primary education sector? 8. is there anything else anyone would like to add? pre course charts http://dx.doi.org/10.25304/rlt.v28.2203 research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 19 (page number not for citation purpose) i am confident in teaching physical education question 1 i understand the benefits ict can bring to promoting and improving physical activity participation in schools question 2 i have a good range of ideas for increasing physical activity in the school and have competence in implementing them into everyday school life question 3 i understand the role i have as a teacher to promote it within a school question 4 i understand the importance of promoting physical activity and the overall benefits it brings to students question 5 table pre course questionnaire distribution graph pre course questionnaire http://dx.doi.org/10.25304/rlt.v28.2203 d. brennan 20 citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) post course charts i am confident in teaching physical education question 1 i understand the benefits ict can bring to promoting and improving physical activity participation in schools question 2 i have a good range of ideas for increasing physical activity in the school and have competence in implementing them into everyday school life question 3 i understand the role i have as a teacher to promote it within a school question 4 i understand the importance of promoting physical activity and the overall benefits it brings to students question 5 table post course questionnaire http://dx.doi.org/10.25304/rlt.v28.2203 research in learning technology citation: research in learning technology 2020, 28: 2203 http://dx.doi.org/10.25304/rlt.v28.2203 21 (page number not for citation purpose) post course questionnaire distribution graph http://dx.doi.org/10.25304/rlt.v28.2203 (page number not for citation purpose) 1 *corresponding author. email: petr.lebedev@sydney.edu.au research in learning technology 2019. © 2019 p. lebedev and m.d. sharma. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 research in learning technology vol. 27, 2019 original research article riddles on youtube: investigating the potential to engage viewers in reflective thinking petr lebedev* and manjula devi sharma school of physics, the university of sydney, sydney, australia (received: 12 june 2019; revised: 15 october 2019; accepted: 15 october 2019; published: 27 november 2019) linear videos continue to be useful technologies for both formal and informal learning. the advent of youtube has seen the rise of educational channels engaging millions of viewers informally, in the spirit of ‘lifelong learning’. this article investigates the potential of a particular kind of video to engage viewers in reflective thinking. the premise is that viewers, followers who find the channel worthwhile or interesting, should reflect, affirming or changing their answers. the channel veritasium, with more than 5 million subscribers at the time of this study, posed four questions as ‘riddles’, and then a week later provided solutions. a survey appended to the solutions video was completed by 2220 respondents. about 16%–43% of respondents across all riddles indicated that their answer was correct or that they changed their answer, and 49%–73% reported to have changed their answers. the reasons for not changing answers were inductively coded for each riddle. the emergent categories were the same for the different riddles, from outward focused to inward focused, matching the types of reflective thinking in the extant literature. the responses illustrated how the riddles engaged the viewers in reflective thinking in this social media informal learning context. keywords: youtube; reflective thinking; educational multimedia; informal learning; social media introduction a variety of learning technologies are available and used for learning, ranging from interactive simulations, short linear videos and gaming to virtual reality. in science education, multimedia offerings with elaborate interactivity and feedback loops are used in formal settings to prompt reflective thinking with the intention of guiding learners’ understandings to those more congruent with current scientific understandings (see, e.g., coller and scott 2009; crook, sharma, and wilson 2015; stieff 2005; wieman, adams, and perkins 2008). contrary to the expectation that such interactive multimedia would dominate formal learning, linear videos in the form of lecture recordings (yeung, raju, and sharma 2016), video slices with classroom-based learning cycles (mayo, sharma, and muller 2009) and  social media sites sharing short videos are providing fierce competition and mailto:petr.lebedev@sydney.edu.au http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2280 p. lebedev and m.d. sharma 2 citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 (page number not for citation purpose) gaining popularity (rosenthal 2017; tan and pearce 2011). whilst youtube channels such as ‘crashcourse’ and ‘khan academy’ support formal learning through their links with curricula (gray, lindstrøm, and vestli 2017), it is not unusual to find teachers using carefully selected videos from other channels such as smartereveryday and veritasium in their classrooms (crookedscience 2019, pers. comm.). closer scrutiny shows such videos are often underpinned by sound pedagogical and andragogical principles, for example, aligning with dewey’s phases for prompting reflective thinking which guides learners in developing their understandings (dewey 1933; rodgers 2002). furthermore, specific design features could be used to engage learners. in this study, in one youtube video, a question was posed, viewers encouraged to find solutions, and a week later, solutions with explanations were provided in another video. the questions and solutions are called ‘riddles’ and align with dewey’s phases. whilst the riddles are suitable for formal learning and are of the format sought by teachers, followers of the youtube channel would be viewing these riddles as informal learning in the spirit of lifelong learning. do the viewers change their answers? what is the nature of their reflective thinking? reflective thinking amongst adults is multifaceted, captured by mezirow (1998) in his taxonomy for reflective thinking for adult learners. the process of reflective thinking is integral to how one responds depending on the nature of the response. rosenthal (2017) summarises the behaviour of viewers who engage informally with science-based learning technologies as reasoned and planned. viewers engage with youtube videos for several reasons: they perceive their engagement as worthwhile as it satisfies a need; their engagement is enjoyable – arising from self-interest; or they are engaging to gather knowledge. it motivates individuals to engage in the behaviour of seeking or following, potentially leading to reflecting and responding to messages. such behaviour is found in adult informal education on subjects ranging from immunisation, patient information and parent driven campaigns, to sustainable agriculture (see, e.g., bentley et al. 2019; carter et al. 2017; covolo et al. 2017; pithadia et al. 2019). given that youtube videos used in informal learning can lead to reflection and action, it is important to understand viewers’ responses. the broad aim of this study is to investigate how effective riddle videos are at stimulating critical reflection as defined by mezirow. the research questions of this study are specific for the individual riddles, from a set of four, presented on the youtube channel, veritasium: 1. research question 1: what percentages of respondents indicated that their answers were correct, they changed their answers or they did not change their answers? 2. research question 2: what are the reasons given by respondents for not changing their answers, thus elucidating the nature of their reflective thinking? the next section of this article presents pertinent background on videos, in particular those on youtube, as a learning technology, to improve reflective thinking from the perspectives of dewey’s phases and mezirow’s taxonomy for reflective thinking. the methodology describes the riddles on the youtube channel veritasium, data collection and analysis. this is followed by results and a discussion. http://dx.doi.org/10.25304/rlt.v27.2280 research in learning technology citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 3 (page number not for citation purpose) background linear videos: youtube as a learning technology each learning technology has its own affordance and challenges. whilst technologies with feedback loops and interactivity can place the learner in control, prompting reflective thinking, and are underpinned by substantive research (see, e.g., coller and scott 2009; stieff 2005; wieman, adams, and perkins 2008), videos, at first glance appear limiting in being linear. on the other hand, films and documentaries, which are unlike videos, have been used for both formal and informal learning for nearly a century. perhaps the forms of linear videos which are becoming standard in the adult education arena are lecture recordings (gosper, malfroy, and mckenzie 2013). whilst there are doubts about the effectiveness of lecture recordings in supporting learning, lecture recordings provide easy access, connectivity and community through shared experiences of the opportunity for visiting earlier lectures (yeung, raju, and sharma 2016). sinfield (2018), in discussing mobile mixed technologies, argues that ‘learning is a lifelong experience … pedagogy is applied to communities and not just institutions; thus, … part of this pedagogy becomes greater than the education system within tertiary institutions’. linear video in its various forms constitute part of the pedagogy of learning in the lifelong learning endeavour, beyond the institution (knox and bayne 2013). tan and pearce (2011) in investigating free educational resources in their course, including carefully selected youtube videos, found youtube videos practical and reported that learners found them worthwhile and interesting. the connectivity between students and the wider world is captured through ‘real world examples’, for example, liu and elms (2019) used short animated videos, whilst mayo, sharma, and muller (2009) used video slices. such examples capture dewey’s (1916) pragmatic conception of learning as a social activity. together with youtube videos bringing ‘real world examples’ into formal learning contexts, pedagogies used in formal learning are creeping into informal learning, connecting formal and informal pedagogies, where, in the spirit of lifelong learning, pedagogy becomes greater than the education system (knox and bayne 2013; sinfield 2018). in science education, examples are the youtube channels smartereveryday and veritasium. pedagogically and andragogically, most educational videos on youtube take the form of direct instruction (jones and cuthrell 2011), with little incorporation of feedback loops, registering audience responses and interactivity. this is partially due to the nature of the medium – youtube does not include inbuilt interactive capabilities such as multiple-choice answers (guo, kim, and rubin 2014). however, refutation, dialogue and socratic dialogue have proven to be allies in the absence of inbuilt interactive capabilities (muller et al. 2007). some creators have taken this one step further by asking a question and encouraging viewers to write down their answers in one video, and then providing the solutions in another. this interactivity provides, in particular, time for reflective thinking in a forum where one is aware of and connected with like-minded others. dewey (1916) argued that learning is a social activity. in the modern day context, youtube, being a social media platform, offers connectivity amongst like-minded individuals enabling learning as a social activity. channels can attract millions of subscribers and hundreds of millions of views. according to rosenthal (2017), the behaviour of viewers is reasoned and planned, their motivations falling into various categories; their engagement is worthwhile as it satisfies a need, is enjoyable due to http://dx.doi.org/10.25304/rlt.v27.2280 p. lebedev and m.d. sharma 4 citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 (page number not for citation purpose) the contextual relevance and self-interest or their engagement is for entertainment or gathering knowledge (see, e.g., bentley et al. 2019; carter et al. 2017; covolo et  al. 2017; pithadia et al. 2019). individuals seek youtube videos in line with their motivators, follow and subscribe, converging on and reinforcing their motivators. the behaviour of seeking or following whilst being aware of like-minded others who are socially connected via the channel can lead to reflecting and responding to messages. even though the use of pedagogically sound, short linear videos on youtube as a form of learning has increased in popularity over the last 10 years, research into their efficacy as educational tools is limited and under researched. because of the importance of reflection and the impact of linear multimedia content both in classroom and informal settings (dabbagh and kitsantas 2012), it is worth researching how to encourage reflective thinking amongst the viewers of linear multimedia content. reflective thinking and dewey’s phases in comparing reflection in a mirror with reflection in our thoughts, wade (1997) eloquently articulates: “as a mirror reflects a physical image, so does reflection as a thought process reveal to us aspects of our experience that might have remained hidden had we not taken the time to consider them.” taking time to discern what may not be obvious can lead to a change in behaviour, can bring an idea closer to application, and can commit one to action. (boud, keogh, and walker 2013) since reflection emerges in several different fields, as noted by rogers (2001), there are somewhat differing views on how it is defined. however, most scholars converge on and accept a few common features. these include that: reflection requires an experience that can leave the learner puzzled or perplexed, reflection is a process that requires active engagement with investment of time and allows for what may be hidden to become visible. these features imply that reflection is not linear, it is an organic, evolving and self-referential process. rogers (2001) notes that: the process of reflection does not always have a defined beginning and end. thus, it should be viewed as continuous, much like an ever-expanding spiral in which challenging situations lead to reflection and ultimately to new interpretations or understanding. these new understandings may then lead to new challenges and additional reflection. to study reflection as an evolving, complex and organic process through all these scenarios poses a challenge. dewey (1933) makes the study of reflection more accessible by splitting up reflection into phases which further elaborate the common features. • reflection requires an experience, and the spontaneous interpretation of that experience. • reflection is a process that requires active engagement with investment of time involving: ° the articulation of the problem or question that arises out of the experience ° the generation of possible explanations for the problem or question ° the explanations need to be examined and tested. dewey’s elaborations are not dissimilar to others in providing a mechanism for constructing environments to promote reflection (see, e.g., liu and elms 2019; mayo, http://dx.doi.org/10.25304/rlt.v27.2280 research in learning technology citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 5 (page number not for citation purpose) sharma, and muller 2009). these are non-linear, self-referential processes that can take on different forms, indicating that there are different types of reflection. another reason why reflection is important is because it underpins critical thinking – it is one of the pillars that support critical thinking. taxonomy of reflection when one writes a response after engaging in a formal or informal learning activity, what clues are there that they are reflecting? this study draws on reflective thinking in the field of adult education, in particular mezirow (1998), who distinguishes reflection, as articulated above, from critical reflection: reflection does not necessarily imply making an assessment of what is being reflected upon, a distinction that differentiates it from critical reflection. critical reflection may be implicit, as when we mindlessly choose between good and evil because of our assimilated values, or explicit, as when we bring the process of choice into awareness to examine and assess the reasons for making a choice. (mezirow 1998, p. 186) in making the assessment, mezirow (1997) asserts that the viewers involve their own assumptions underpinning the reflection, a feature which is particularly evident in adult learners. the term critical reflection on assumptions (cra) is used when the objects involved in the experience are being assessed based on the viewer’s assumptions regarding the presenter, the source, content and context – an outward focus. critical self-reflection on assumptions (csra), on the other hand, involves assessing one’s own concepts, beliefs, feelings or actions in view of the experience. this could mean assessing one’s own knowledge, emotions, or response – an inward focus. both of these types of critical reflection involve analysing warranted or unwarranted assumptions pertaining to the experience. cra focuses on the outward assumptions and csra focuses on the inner assumptions. this taxonomy provides a mechanism for ascertaining the nature of reflective thinking which can be applied in the social media based informal learning context. method the riddles the videos used in this study were from the youtube channel ‘veritasium’. video 1 contained four riddles, and video two the solutions to those riddles (videos to be found at https://www.youtube.com/watch?v=hgcxdnhvc1q and https://www.youtube. com/watch?v=72dcj3bztg4&t=243s). the videos align with the phases of reflection outlined by dewey (1933). video 1 captures dewey’s phases through an experience, posing a question, and the viewers generating possible explanations. video two examines explanations and provides the solution – see appendix for example. the survey instrument and data collection video 1 was published on youtube, followed a week later by video 2. the description box under video two contained a link to the survey hosted on surveymonkey®. the survey consisted of one question, containing two parts, duplicated for each of the riddles. for riddle one, the survey posed ‘after watching the solutions video, did your answer to question (riddle) 1 change?’ respondents could only select one of the http://dx.doi.org/10.25304/rlt.v27.2280 https://www.youtube.com/watch?v=hgcxdnhvc1q https://www.youtube.com/watch?v=72dcj3bztg4&t=243s https://www.youtube.com/watch?v=72dcj3bztg4&t=243s p. lebedev and m.d. sharma 6 citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 (page number not for citation purpose) following options, ‘yes’, ‘no’ and ‘my answer was correct’. this part of the survey quantitatively allows us to answer the first research question. to answer the second research question, we asked another question. ‘if you answered no — why did your answer not change?’ the survey included a text box where the respondents were asked to write their answers. figure 1 shows how the questions appeared to the viewer. data analysis the data from surveymonkey® were exported to an excel spreadsheet and examined to determine if responses were ‘legitimate’. in particular, with the written responses, if a respondent gave an ‘obvious silly’ answer they would be removed. there were no such cases, and no data were removed. data were analysed with a mixed methods approach (creswell and clark 2007). quantitative analysis – by counting the numbers of responses, converting them into percentages and thus generating graphs – was used for the first research question, whilst qualitative coding was used for the second research question. one researcher read the responses, immersing themselves in the data, identifying patterns and grouping responses in terms of similarities and differences. a discussion with a second researcher helped consolidate the patterns and articulate the descriptions of the initial categories. both researchers then took these initial category descriptions and independently coded a random set of 50 responses for each riddle. in the next discussion, the two researchers further refined and finalised these category descriptions. all the responses for each riddle were then independently coded by the same two researchers. the two researchers compared the sorting of responses and made minor changes. thus, the written responses were inductively coded through an iterative process of coding, comparing, consolidating and refining by two researchers (burla et al. 2008). as a further check, a third researcher was provided with 50 responses from each riddle to code. the inter-coder reliability was (82.5%) indicating that the categories are robust, and allocation of responses to the categories is reliable. sample the videos attracted more than one million views each, of which 2220 completed the survey within a week following the release of video two. the sample of 2220 was those who engage with the popular science youtube channel ‘veritasium’ for figure 1. the survey, administered after participants had watched video two containing the solutions, was duplicated for each of the four riddles. http://dx.doi.org/10.25304/rlt.v27.2280 research in learning technology citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 7 (page number not for citation purpose) informal learning. at the time of this study, the channel had approximately five million subscribers and 500 million views. the respondents would most probably be adults who are lifelong learners of science. the study thus uses convenience sampling. respondents are those who seek and follow channels; their behaviour is reasoned and planned giving us an interesting perspective on a range of questions, from what fraction say they are correct to why some ‘followers’ do not change their answers. results and discussion the first research question probes the percentages of respondents who indicated that their answers were correct, they did change their answers or did not change their answers. table 1 shows these percentages together with the numbers of respondents. it is worth noting that most respondents persisted in completing the survey for all four riddles. there is variation amongst the riddles; 73% of respondents to riddle one said that they did change their answer, whilst less than half did so for riddle four. only 16% of respondents said that their answer was correct for riddle one, whilst the other riddles had twice as much or more. whilst there are substantive shifts between the percentages of those who did change their answer and said that their answer is correct, the percentage of those who did not change their answer is within a narrow band of 7%–11%. the percentages of respondents who indicated that their answer was correct ranged from 16%–43%, affirming their reasoned and planned connection with the channel. noteworthy is the fact that a larger percentage, 49%–73% did actually indicate that they changed their answer – they responded and took action. speculating, there could be a combination of reasons, such as the riddles being pedagogically sound, aligning with the phases of reflection outlined by dewey (1933), prompting respondents to change their answers, and that the behaviour aligns with the seek and follow behaviour associated with youtube channels. if we add those who changed their answers with those who were correct, we obtain a more consistent value of around 89%–93% across all riddles, supporting rosenthal’s (2017) premise that engagement is perceived as worthwhile as it satisfies a need, is enjoyable and arises from self-interest or as a knowledge gathering exercise. in short, those who do not change their answer are a minority, 7%–11%. another prominent finding is that there is considerable variation from riddle to riddle. this variation exists even though all the riddles align with the phases of reflection outlined by dewey (1933), despite the anticipated seek and follow behaviour of viewers. we offer the suggestion that the specificity of the riddle matters; it could be the topic, the way it is posed or the actual props used. viewers who chose to complete table 1. the numbers and percentages of respondents who indicated that: their answer was correct, they did change their answer or did not change their answer to each of the four riddles. number and percentage of respondents for each riddle one two three four my answer is correct 361 (16%) 695 (31%) 843 (38%) 962 (43%) i did change my answer 1614 (73%) 1369 (62%) 1143 (52%) 1080 (49%) i did not change my answer 245 (11%) 154 (7%) 229 (10%) 176 (8%) total 2220 (100%) 2218 (100%) 2215 (100%) 2218 (100%) http://dx.doi.org/10.25304/rlt.v27.2280 p. lebedev and m.d. sharma 8 citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 (page number not for citation purpose) the survey connect with a community through shared experiences and respond in specific ways. they are sensitive to the experience posed in the riddle and its peculiarities; hence, their responses are nuanced. this is further borne out by the responses of those who chose not to change their answer. research question two focused on the respondents who had indicated that their answer was not correct, and yet, did not change their answer. whilst this is a small percentage of the respondents, understanding their reasons gives clues as to why people seek and follow, and yet, not change their answers. inductive coding of the written responses generated eight categories. the percentages of responses in each of the eight categories for each riddle are shown in table 2, those attracting more than 20% responses are shaded. two points are worth making. first, the riddles attracted different percentages of respondents in each category. second, the respondents did not respond in the same manner across the riddles, in other words, their responses fell into different categories for the different riddles. the descriptions and examples of typical responses are shown below. in the first two categories, the respondents did not need to change their answers because they had no answer in the first instance or their initial answer was correct. i had no answer was typified by the simple response, ‘i had no answer’. my answer was correct contained responses such as, ‘i had it right’, and ‘i have done this problem before and knew the physics behind it’. the responses were generally brief, but they diligently completed the survey. these two categories attracted the smallest percentage of responses, 8.5% for riddle one and, the largest, 48.2% for riddle four. the next three categories were premised on the assumption that the respondents have adequately analysed the science in the videos. they had an alternative solution or the science in the solution video was close enough to the science in their answer, hence the respondents did not need to change their answers. i have an alternative solution could contain creative alternatives, such as, ‘i still believe my answer (a cylinder with a fluid and a baffle with a few small holes) could exhibit the same behavior’. in other words, there is more than one solution to this problem. sometimes a simple statement was provided, such as, ‘because i have a different explanation to the same problem’. table 2. percentages of those who did not change their answer, coded into each category for each riddle (shaded in grey if higher than 20%). category % of responses for each riddle one two three four i had no answer 7.3 9.7 11.7 26.8 my answer was correct 1.2 15.3 18.8 21.4 i have an alternative solution 22.0 0.0 10.2 12.5 my answer was almost correct 24.5 5.6 1.5 4.5 my answer was almost correct, with explanation 28.2 20.2 4.5 5.4 i am discontented 12.2 24.2 26.9 8.0 i don’t fully understand 1.6 16.1 20.8 17.0 miscellaneous 3.0 8.9 5.6 4.4 total 100 100 100 100 http://dx.doi.org/10.25304/rlt.v27.2280 research in learning technology citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 9 (page number not for citation purpose) my answer was almost correct contained simple and direct statements, for example, ‘i was somewhat correct’. my answer was almost correct, with explanation contained explanations which often demonstrated a level of prior knowledge or that the respondent was drawing on some first-hand experience: i answered a viscous liquid. i don’t think the ping pong balls played a fundamental role, since the jar of honey behaved in a similar way. i did not predict the ping pong balls but was mostly correct. the ping pong balls seem to affect the cylinder mostly by keeping the air in two spheres which changes the weight distribution but it is still a viscous liquid with an air pocket. in these three categories are those who indicated that they had alternative solutions or their answers were close enough, often times offering an explanation. generating possible explanations is a common feature of reflective thinking, as is examining and testing explanations (dewey 1933; rogers 2001). the focus in these reflections was on the content of the experience, appropriateness of the science used to interpret the experience and rationale for the solution provided. these responses indicate that they are premised on the assumption that the viewer had adequately assessed the science in the videos, placing them in mezirow’s (1998) cra category. the respondents are critically reflecting on the experience, its credibility, plausibility and authenticity. the respondents are doing this by comparing their answers to the solution provided, reflecting on the validity of the solution and critically reflecting on the presenter, source, content and context. there is a comparison between their answers and the answer provided; the answers could be affirming, close enough, explanations generated, alternatives provided. the important aspect is that these reflections are outward focused, the object is external, the riddle. collectively these three categories attracted the smallest response, 16.2% for riddle three and, the largest, 74.7% for riddle one. in the next two categories, the respondent is more involved, emotions are invoked, questions are raised and beliefs are in play. these could be positive or negative. there could be disagreement or that they did not fully comprehend aspects of the riddle or know relevant detail, expressing what they had learned from the video or intended to learn. often they could identify flaws in their own reasoning or gaps in prior knowledge, or in the solutions. i am discontented often noted specific features of the video which respondents felt were incorrect or they disagreed with such as, ‘i disagree that the solution uses a reasonable interpretation of the question’. another example is ‘one component of a bike was forgotten, the ratchet on the back wheel. if your bike doesn’t have one, and you pull with less force than the traction … but the back wheel won’t spin backwards’. i don’t fully understand contained examples, such as, ‘i still don’t fully understand the mechanics of bicycles. need to re-watch the video again’. another example is, ‘i predicted the bike behaviour correctly, but my explanation was partially wrong … the net force points backwards, therefore the bike will move always backwards.’ the category ‘i am discontented’ is interesting as the reflection is again on the technicalities of the experience and its interpretation, but a discrepancy is articulated. there is a sense that something is missing or unfair; the respondent sometimes expresses feeling tricked or articulates that they had not expected this from this particular channel. there is discord or discontent, and emotions are involved. the category http://dx.doi.org/10.25304/rlt.v27.2280 p. lebedev and m.d. sharma 10 citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 (page number not for citation purpose) ‘i don’t fully understand’ contains responses which are not only reflections on the technicalities of the experience but also contain viewers own thinking. it is also of note that most respondents expressed an interest and intent to learn more about the topic, aligning with lifelong learning aspirations. both of these categories align with reflective thinking in that they are examining and testing explanations (dewey 1933; rogers 2001). in addition, the reflections pertain to the validity of one’s own concepts, beliefs, feelings or actions. there is an inward focus, falling into mezirow’s csra. collectively these two categories attracted the smallest percentage, 13.8% of responses for riddle one and, the largest, 47.7% for riddle three. finally, 3.0%–8.9% of the responses were coded as miscellaneous; these responses could not be meaningfully coded. miscellaneous contained responses that were ambiguous or unclear and were not coded. implications for theory and practice in this study, the structure of the videos aligns with dewey’s phases of reflection (see appendix), and affirms dewey’s framework in the social media, informal learning context. another important consideration for the practice of creating videos that encourage reflection is that riddle matters – there are variations in the types of reflection between the four riddles. focusing on the shades of grey, for riddle one, the second set of categories encompassing the outward focusing cra dominates, whilst for riddle three, the inward focusing csra dominates. this leads to interesting questions for further research. how does the nature of the riddle affect the type of reflection? what we found was that none of the respondents could be consistently placed in the same category for all four riddles; participants changed their responses based on the riddle. given the increasing popularity of youtube videos and other such resources to complement, supplement and enrich the lifelong learning experiences of those who engage informally, it is important to understand reflective thinking as the response can lead to action. being opportunistic in nature, this study needs to be repeated in different contexts with different riddles. our sample is specific to those who engage with the particular youtube channel, and also the subset who chose to complete the survey. nevertheless, the sample is sufficient to indicate that some reflection is occurring, and the responses align with mezirow’s taxonomy and behaviours of respondents align with what is expected for those engaging with youtube (rosenthal 2017). further research needs to be conducted on different samples, with different riddles, and different subject areas. conclusion the reach of social media such as popular youtube channels is captured by the sheer numbers of individuals who engage with them, more than one million within a fortnight for our study. the numbers who diligently engage, often in the spirit of lifelong learning, is particularly noteworthy. in our study, 2220 viewers completed the survey for all four riddles, providing unique answers for each riddle. the vast majority self reported that their answers were correct or they had changed their answers, affirming their answers or prompting them to change. the responses provided by those who did not change their answers fell into categories corresponding to mezirow’s (1988) http://dx.doi.org/10.25304/rlt.v27.2280 research in learning technology citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 11 (page number not for citation purpose) critical reflection categories for adult learners. in short, viewers who follow this particular social media youtube channel as part of a reasoned and planned behaviour appeared to be critically reflecting as a result of engaging with the riddles. investigating youtube as a learning technology for formal and informal learning is justified by the scope and reach as well as its potential impact. the unique feature of this research is that it investigates known pedagogical and andragogical perspectives on reflective thinking, and applies them in the domain of informal, social media facilitated learning. whilst more research needs to be carried out in this context, the initial results from this study are promising – dewey’s phases of reflection and mezirow’s taxonomy of critical reflection are useful tools for encouraging and analysing reflection, which happens in the informal learning context. acknowledgements we thank dr derek muller for providing access to the two youtube videos and for embedding the survey link. we also thank all the participants of this study, dr christine lindstrøm for the help with the coding, and the sydney university physics education research (super) group for constructive feedback. the study has approval from the human ethics committee of the university of sydney. references bentley, j., et al., (2019) ‘smallholders download and share videos from the internet to learn about sustainable agriculture’, international journal of agricultural sustainability, vol. 17, no. 1, pp. 92–107. doi: 10.1080/14735903.2019.1567246 boud, d., keogh, r. & walker, d. (1985) reflection, turning experience into learning, kogan page, london. burla, l., et al., (2008) ‘from text to codings’, nursing research, vol. 57, no. 2, pp. 113–117. doi: 10.1097/01.nnr.0000313482.33917.7d carter, b., et al., (2017) ‘parent-driven campaign videos: an analysis of the motivation and affect of videos created by parents of children with complex healthcare needs’, comprehensive child and adolescent nursing, vol. 41, no. 4, pp. 276–292. doi: 10.1080/24694193.2017. 1373160 coller, b. & scott, m. (2009) ‘effectiveness of using a video game to teach a course in mechanical engineering’, computers & education, vol. 53, no. 3, pp. 900–912. doi: 10.1016/j. compedu.2009.05.012 covolo, l., et al., (2017) ‘what arguments on vaccinations run through youtube videos in italy? a content analysis’, human vaccines & immunotherapeutics, vol. 13, no. 7, pp. 1693–1699. doi: 10.1080/21645515.2017.1306159 creswell, j. & plano clark, v. (2007) designing and conducting mixed methods research, 3rd edn, sage, thousand oaks, ca, pp. 801–804. doi: 10.1177/1094428108318066. crook, s. j., sharma, m. d. & wilson, r. (2015) ‘an evaluation of the impact of 1: 1 laptops on student attainment in senior high school sciences’, international journal of science education, vol. 37, no. 2, pp. 272–293. doi: 10.1080/09500693.2014.982229. crookedscience, 2019, personal communication. dabbagh, n. & kitsantas, a. (2012) ‘personal learning environments, social media, and self-regulated learning: a natural formula for connecting formal and informal learning’, the internet and higher education, vol. 15, no. 1, pp. 3–8. doi: 10.1016/j.iheduc.2011.06.002 dewey, j. (1916) democracy and education, southern illinois university press, carbondale and edwardsville. http://dx.doi.org/10.25304/rlt.v27.2280 https://doi.org/10.1080/14735903.2019.1567246 https://doi.org/10.1097/01.nnr.0000313482.33917.7d https://doi.org/10.1080/24694193.2017.1373160 https://doi.org/10.1080/24694193.2017.1373160 https://doi.org/10.1016/j.compedu.2009.05.012 https://doi.org/10.1016/j.compedu.2009.05.012 https://doi.org/10.1080/21645515.2017.1306159 https://doi.org/10.1177/1094428108318066 https://doi.org/10.1080/09500693.2014.982229 https://doi.org/10.1016/j.iheduc.2011.06.002 p. lebedev and m.d. sharma 12 citation: research in learning technology 2019, 27: 2280 http://dx.doi.org/10.25304/rlt.v27.2280 (page number not for citation purpose) dewey, j. (1933) how we think (1998 ed.). doi: 10.1037/10903-000. gosper, m., malfroy, j. & mckenzie, j. (2013) ‘students’ experiences and expectations of technologies: an australian study designed to inform planning and development decisions’, australasian journal of educational technology, vol. 29, no. 2. doi: 10.14742/ajet.127. gray, j., lindstrøm, c. & vestli, k. (2017) ‘khan academy as a resource for pre-service teachers: a controlled study’, in cerme 10, feb 2017, dublin, ireland. hal-01946341. doi: 10.1119/perc.2017.pr.057. guo, p. j., kim, j. & rubin, r. (2014) ‘how video production affects student engagement: an empirical study of mooc videos’, in proceedings of the first acm conference on learning@ scale conference, acm, pp. 41–50. doi: 10.1145/2556325.2566239 jones, t. & cuthrell, k. (2011) ‘youtube: educational potentials and pitfalls’, computers in the schools, vol. 28, no. 1, pp. 75–85. doi: 10.1080/07380569.2011.553149 knox, j. & bayne, s. (2013) ‘multimodal profusion in the literacies of the massive open online course’, research in learning technology, vol. 21. doi: 10.3402/rlt.v21.21422 liu, c. & elms, p. (2019) ‘animating student engagement: the impacts of cartoon instructional videos on learning experience’, research in learning technology, vol. 27. doi: 10.25304/rlt. v27.2124 mayo, a., sharma, m. d. & muller, d. a. (2009) ‘qualitative differences between learning environments using videos in small groups and whole class discussions: a preliminary study in physics’, research in science education, vol. 39, no. 4, pp. 477–493. doi: 10.1007/ s11165-008-9090-0. mezirow, j. (1997) ‘transformative learning: theory to practice’, new directions for adult and continuing education, vol. 1997, no. 74, pp. 5–12. doi: 10.1002/ace.7401. mezirow, j. (1998) ‘on critical reflection’, adult education quarterly, vol. 48, no. 3, pp. 185–198. doi: 10.1177/074171369804800305. muller, d. a., et al., (2007) ‘conceptual change through vicarious learning in an authentic physics setting’, instructional science, vol. 35, no. 6, pp. 519–533. doi: 10.1007/s11251-007-9017-6. pithadia, d. j., et al., (2019) ‘a cross-sectional study of youtube videos as a source of patient information for topical psoriasis therapies’, journal of dermatological treatment, (just-accepted), pp. 1–14. doi: 10.1080/09546634.2019.1597247. rodgers, c. (2002) ‘defining reflection: another look at john dewey and reflective thinking’, teachers college record, vol. 104, no. 4, pp. 842–866. doi: 10.1111/1467-9620.00181. rogers, r. r. (2001) ‘reflection in higher education: a concept analysis’, innovative higher education, vol. 26, no. 1, pp. 37–57. doi: 10.1023/a:1010986404527. rosenthal, s. (2017) ‘motivations to seek science videos on youtube: free-choice learning in a connected society’, international journal of science education, part b, vol. 8, no. 1, pp. 22–39. doi: 10.1080/21548455.2017.1371357. sinfield, d. (2018) ‘the boundaries of education: using mobile devices for connecting people to places’, altj-association for learning technology journal, vol. 26. doi: 10.25304/rlt. v26.2121. stieff, m. (2005) ‘connected chemistry—a novel modeling environment for the chemistry classroom’, journal of chemical education, vol. 82, no.3, p.489. doi: 10.1021/ed082p489. tan, e. & pearce, n. (2011) ‘open education videos in the classroom: exploring the opportunities and barriers to the use of youtube in teaching introductory sociology’, research in learning technology, vol. 19(sup1), p.7783. doi: 10.3402/rlt.v19s1/7783. wade, r. c. (ed) (1997) community service-learning: a guide to including service in the public school curriculum: challenges to effective practice, the social studies, 88(5), pp. 197–202. doi: 10.1080/00377999709603778. wieman, c. e., adams, w. k. & perkins, k. k. (2008) ‘phet: simulations that enhance learning’, science, vol. 322, no. 5902, pp. 682–683. doi: 10.1126/science.1161948. yeung, a., raju, s. & sharma, m. d. (2016) ‘online lecture recordings and lecture attendance: investigating student preferences in a large first year psychology course’, journal of learning design, vol. 9, no. 1, pp. 55–71. doi: 10.5204/jld.v9i1.243. http://dx.doi.org/10.25304/rlt.v27.2280 https://doi.org/10.1037/10903-000 https://doi.org/10.14742/ajet.127 https://doi.org/10.1119/perc.2017.pr.057 https://doi.org/10.1145/2556325.2566239 https://doi.org/10.1080/07380569.2011.553149 https://doi.org/10.3402/rlt.v21.21422 https://doi.org/10.25304/rlt.v27.2124 https://doi.org/10.25304/rlt.v27.2124 https://doi.org/10.1007/s11165-008-9090-0 https://doi.org/10.1007/s11165-008-9090-0 https://doi.org/10.1002/ace.7401 https://doi.org/10.1177/074171369804800305 https://doi.org/10.1007/s11251-007-9017-6 https://doi.org/10.1080/09546634.2019.1597247 https://doi.org/10.1111/1467-9620.00181 https://doi.org/10.1023/a:1010986404527 https://doi.org/10.1080/21548455.2017.1371357 https://doi.org/10.25304/rlt.v26.2121 https://doi.org/10.25304/rlt.v26.2121 https://doi.org/10.1021/ed082p489 https://doi.org/10.3402/rlt.v19s1/7783 https://doi.org/10.1080/00377999709603778 https://doi.org/10.1126/science.1161948 https://doi.org/10.5204/jld.v9i1.243 (page number not for citation purpose) 1 *corresponding author. email: zuh.khlaif@gmail.com research in learning technology 2019. © 2019 z.n khlaif et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 research in learning technology vol. 27, 2019 original research article exploring children experience with educational mobile technology zuheir n khlaifa*, jamil itmazib, shahid faridc, ali zuhdi shaqoura and bochra  kouraïchid ateaching methods department, an-najah national university, nablus, palestine; bcomputer science department, palestine ahliya university, bethlehem, palestine; cdepartment of computer science, bahauddin zakariya university, multan, pakistan; dministry of education, ibn rush school, kebili, tunisia (received: 3 march 2019; revised: 25 july 2019; accepted: 11 september 2019; published: 15 november 2019) the purpose of this study is to explore and analyse the user experience (ux) of arabic educational and entertainment applications designed for children aging 8–10 years. this paper reports the results of an experimental study conducted on a sample of 53 children in two elementary schools in qalqilya and nablus, palestine. the context of the current study is different from previous studies that were conducted in a western environment in terms of knowledge, awareness, demographic changes and background variables. qualitative methods captured the enjoyable and attractive parts of the software being used. the qualitative content analysis showed that participants’ enjoyment using the system was quite different. during sessions, children showed much excitement and interest while working with the math app more than the science app. however, girls were less interested to use the math application compared with their use of the device itself. following hassenzahl’s model, the descriptions of uxs were also categorised into two categories: pragmatic and hedonic; the functionality, technical and ease of use aspects of ux of both the tablet and the math application were categorised as pragmatic. enjoyable, exciting, confusing and upsetting experiences were categorised as hedonic. male participants were more familiar with the use of mobile devices than female participants. the results of the present study showed that gender, culture and religion are important factors that affect children’s experience to use new technological devices since three female participants were reluctant to use the tablet especially due to cultural and religious factors. keywords: user experience; mobile technology; hassenzahl’s model; ux; qualitative study; palestine introduction mobile devices are known as expensive devices amongst parents, but fortunately, the price has become cheaper with very fast technological development. various types of tablets emerged in the market, such as ipad, samsung tabs and others from different suppliers. moreover, the number of sales is growing each day. mobile devices are one of the leading innovative technological tools of communication which can be considered part of the lives of the young generation including students (shaqour 2014). the development of mobile technology has been attracting children’s attention to mailto:zuh.khlaif@gmail.com http://www.alt.ac.uk/� http://creativecommons.org/licenses/by/4.0/� http://dx.doi.org/10.25304/rlt.v27.2242� z.n khlaif et al. 2 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) explore the affordable of this technology in different aspects of their life (plowman and stevenson 2012). children’s interactions with the tablet are increasingly popular, and they have a place in their hearts (aziz 2013). palestinian children gain access to the world of the internet and spend a significant amount of time exploring websites and applications by using mobile technology. according to utair (2015), 80% of palestinian children use smartphones, 10% use tablets and the same percentage use ipad, while about 35% of these children use mobile devices from 3 to 4 h a day. more and more schools in palestine are now including one-to-one technology initiatives and the internet as part of their educational system (khlaif 2018a). one-to-one technology refers to one computer/mobile technology for every student (zucker and light 2009). however, there is a lack of mobile devices applications in arabic language (wea’am and alhussayen 2015). the main purpose of the arabic applications is to educate and entertain children, and these applications are still in their early stages (alhussayen, alrashed, and mansor 2015). moreover, the few arabic children’s applications and websites available are not appealing for children (wea’am and alhussayen 2015). the touch interface is the main feature of mobile technology devices. such facilities are the main input instruments which users use to do tasks such as browsing the web, playing games, watching videos, reading e-books and communicating synchronously and asynchronously through text, voice or video (shaqour 2014). existing literature has not enough research on children’s interaction methods for mobile technology (mcknight and cassidy 2012). many mobile applications are being designed for children and are becoming a growing field of research (deng et al. 2015). touch screens are prevalent in the domain of mobile computing. page (2013) mentioned that the growing inclusion of touch screens in mobile technology including smartphones and tablets has led to the argument that touch screens are soon becoming the standard of the development of technology. the development and design of learning environments including applications and websites for children to provide them with a joyful learning experience have taken place for many years (alhussayen, alrashed, and mansor 2015; wea’am and alhussayen 2015). since children are the key users of this technology, the design and development of applications as well as websites should meet children’s growing expectations to be successful and attractive (alhussayen, alrashed, and mansor 2015). the criteria for the success of applications are functionality as well as being desirable and emotionally appealing (chau 2014). the field of user experience (ux) has had an important impact on the development of different types of technological products (websites, applications, artefacts, etc.). experience is a story emerging from the dialogue of a person with the world through action (hassenzahl 2010). practitioners and researchers see ux as an umbrella term that covers all perceptions and responses, regardless of being subjectively or objectively measured, of a user (iso 2010). there is an agreement among researchers about the definition of ux in terms of how users interact with a new product. experiences are varying from person to person and it could be positive or negative (kim, kim, and moon 2013). for example, sharp, rogers, and preece (2007) consider ux as central to the interaction design and concern with ‘…how people feel about a product and their pleasure and satisfaction when using it, looking at it, opening it or closing it’ (p. 15). hassenzahl (2010) defines ux from a psychological perspective as ‘emerging from the integration of perception, action, motivation, and cognition into the inseparable, http://dx.doi.org/10.25304/rlt.v27.2242� research in learning technology citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 3 (page number not for citation purpose) meaningful whole’ (p. 6). furthermore, hassenzahl (2010) believes that emotions and fulfilment of physiological needs have a crucial role in ux. emotions provide an intrinsic evaluation, namely pleasure or pain, which yet again will provide the user with either a positive or negative ux. hassenzahl (2010) sees ux as a sub-category of experiences, where the focus is on interactive products. moreover, roto and kaasinen (2008) define ux as ‘a term that describes the user’s feelings towards a specific product, system or object during and after interacting with it. various aspects influence the feelings, such as user’s expectations, the conditions in which the interaction takes place and the system’s ability to serve user’s current needs’ (p. 572). researchers employed hassenzahl and tractinsky’s definition of ux because it is easy to follow and helps to achieve the purpose of this study. according to hassenzahl and tractinsky (2006), ux is ‘(a) consequence of a user’s internal state (predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the designed system (e.g. complexity, purpose, usability, functionality, etc.) and the context (or the environment) within which the interaction occurs (e.g. organizational/ social setting, the meaningfulness of the activity, the voluntaries of use, etc.)’ (p. 95). their definition concentrates on the inner emotions of users and their expectation of the designed system; it clearly describes the factors influencing ux. hassenzahl (2010) emphasises that experience is a complex construct, which emerges through interacting with the world. in this article, the researchers present a literature review and the details of the study in which children were observed while interacting with the devices and the focus group discussion sessions for a short time. the article concludes by reporting several issues and giving suggestions to improve the design of applications dedicated to children as well as recommendations for future research. statement of the problem the ministry of education (moe) in palestine provided schools with tablets to use in the educational system. the fact that students, mainly children, are using these devices for the first time, might affect their overall experience. chang (2008) studied the errors that occur while using new products and whether they have an impact on user‘s experience or not. in light of earlier research, this research aims to investigate the first-time use of new technology by children. there are important issues related to first-time users including their initial impressions, their pleasure and fun/frustration to assess their experience to accept and adopt new technology (rahman, ali, and mohd 2013). the use of technology is no longer limited to adult users (wea’am and alhussayen 2015). therefore, developers are trying to design and develop software that fits children’s abilities, interests and development needs (alhussayen, alrashed, and mansor 2015). the experience and knowledge of children are different from those of adults. indeed, children experience to perceive the world differently from adults, which motivated this study to explore the experience of children with new technology for the first time. it is equally important to explore and understand what children perceive and feel when using tablets and applications on it (roto, obrist, and väänänen-vainio-mattila 2009). previous studies mentioned that most children devoted more time and focus their attention on games applications designed for adults (alhussayen, alrashed, and mansor 2015; han, dieck, and jung 2017). based on the literature review, the present study seeks to study children’s first experience of new technology in a conflict zone http://dx.doi.org/10.25304/rlt.v27.2242� z.n khlaif et al. 4 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) with mobility restrictions. the motivation of this study is the lack of studies investigating the ux of using educational applications (math and science application) in palestine. in the current research, the study focuses on ux because of its ability to explore engagement and joyful emotions more effectively than traditional usability (kujala et al. 2011). moreover, users’ expectations about the system and their anticipations are important factors that affect their experience about the system itself (roto,  obrist, and väänänen-vainio-mattila 2009). this study focused on a shortterm evaluation and consequently on aspects relating to the initial use of a new product in the palestinian educational system. the contribution of the study mcknight and cassidy (2012) argued that most of the previous studies have investigated issues related to performance and task efficiency more than users’ experience and their attitudes towards it. in addition, the context of the current study is completely different from previous studies that were conducted in a western environment. to the best of our knowledge, the current study is considered as a contribution to the field of ux research because of the new context. the potential of the significance of the palestinian context is due to its unique case as a conflict zone with an unstable situation and mobility restriction which has a negative influence on the learning system outcomes. violence and mobility restriction have negative impact on the outcomes of using new technology (khlaif and farid 2018). moreover, religion and culture have some impact on children life because of the socio-cultural environment affected by the conflict in the country. to narrow the research gap in the palestinian context, the researchers present the results of this study on ux with math and science applications in the arabic language. this study tries to explore how children experience the educational apps by using the tablet for the first time, how their experiences influence their desire to use it in their free time and their impression while using the playful learning applications. the outcome of the study might be beneficial for decision-makers in the moe in palestine to adopt new learning software in children schools. theoretical framework models of ux have described the experience of using interactive products, the consequences of those experiences and the ways experiences were connected ( hornbæk and hertzum 2017). hassenzahl’s ux model was the framework of the study. this prominent ux model, according to hornbæk and hertzum (2017), posits that interface quality encompasses both hedonic and pragmatic aspects. according to the model, individual users assign some attributes to the system or the product when using it. hassenzahl assumes that users construct product attributes by combining the product’s features, for example, the presentation, content, functionality and interaction with personal expectations or standards. the attributes of any product can be grouped into four main categories including manipulation, identification, stimulation and evocation. these categories can be grouped on a higher level into hedonic and pragmatic which can describe a product’s characteristics. using a product with a particular characteristic in a given situation will lead to certain consequences, such http://dx.doi.org/10.25304/rlt.v27.2242� research in learning technology citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 5 (page number not for citation purpose) as experiencing emotions (e.g. satisfaction or pleasure), forming explicit evaluations (e.g. judgements of appeal, beauty and goodness) or the creation of overt behaviour (e.g. approach or avoidance). there is a general agreement among researchers that ux is shaped by a combination of pragmatic and hedonic qualities of a product (hassenzahl and tractinsky 2006; thüring and mahlke 2007). the pragmatic qualities refer to the perceived usefulness, efficiency and ease of use (so-called utility and usability aspects). in other words, it relates to the practical usage and functions of the product. on the other hand, the hedonic qualities take into account the ‘joy of use’ and emphasise stimulation, identification and evocation generated by the use of a system or a product. it relates to the user’s psychological well-being (bevan, 2009). good ux is the consequence of fulfilling the human needs for autonomy, competence and stimulation (self-oriented) through interacting with the product or service (knijnenburg et  al. 2012). moreover, walsh et al. (2014) mentioned that the combination of pragmatic and hedonic qualities leads to positive or negative emotions and consequently guides the acceptance of the product. hassenzahl’s ux model (figure 1) consists of four product features identifying its character including both pragmatic attributes and hedonic attributes leading to certain consequences, namely appeal, pleasure and satisfaction. manipulation manipulation described as the pragmatic attributes which are essentially related to the functionalities of the software and the ways to use these functions in the current research. hassenzahl (2004) states that satisfaction is a consequence of the pragmatic qualities. satisfaction appears if a user uses the software to achieve specific goals and the software helps to meet those goals. the attributes that are assigned to the software under investigation are useful, easy to use, supporting and controllable. manipulation is an important attribute that contributes to the user’s experience. identification with the development of social media websites, it is essential to include an additional important function, which is identification. this function allows users to express themselves and help them to communicate their identity with others. in the current research, the designer of the software used facebook and twitter as online services figure 1. the key elements in hassenzahl’s user experience model. adapted from hassenzahl 2006. http://dx.doi.org/10.25304/rlt.v27.2242� z.n khlaif et al. 6 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) to provide the users with more information about the company and users can connect their profiles with the facebook page of the company. the identification function is related to the hedonic attributes. stimulation rarely used functions can fill hedonic function. these functions can stimulate the user and satisfy the human urge for personal development and more skills. products should have unused functions and should not be dropped from any software. in this case, the software does not have unused functions. i will consider providing users with feedback and instruction to finish certain tasks as a stimulation function. evocation according to hassenzahl’s model, the fourth function a product has is evocation about recalling the previous memory/experience. most of the teaching applications/ software depend on recalling previous knowledge or skills to achieve specific goals. hassenzahl (2005) mentioned that pragmatic attributes should not be neglected as they help users to achieve hedonic goals, whereas the hedonic attributes are the main contributor to accept the new product. minge (2008) confirmed this conclusion in his experimental study. with the development of ux, hassenzahl (2010) gives experience of various attributes including being subjective, situated, holistic, dynamic and positive. subjectivity is defined as the creation of experience and it remains in the experimenter’s head. it is possible to shape experiences by knowing and using these rules. hassenzahl (2010) develops a form of categorisation based on needs which he calls experience patterns. hassenzahl (2010) pointed out that the needs of users are relatively independent of each other and positive experiences are often marked by a particular need. partala and saari (2015) identified two potential ux-related concepts, emotions, user needs and user values. all of these have been already applied successfully in practical ux studies based on well-established theories from psychological research, and experienced well-being impact of technology, which can be assumed to be especially important when studying technologies used daily or frequently such as different work systems and equipment. harbich and hassenzahl (2017) argued that while effective and efficient goal achievement (execute) is certainly the core of ux at the workplace, there are other goal achievement-related aspects to be considered such as the persistence at a task (engage). experiences change over time; it is like a story. it is packaged, interpreted and labelled and is a construction, but not an objective account of the experience. however, hassenzahl (2008) views the actual construction as only happening once and then being remembered unaltered. silvennoinen, vogel, and kujala (2014) mentioned that functionality is more important when people accomplish tasks, while pleasure aspects are important to have fun. additional factors influence user experience there are other factors besides product features in the formation of ux such as the characteristics of the user, the context of using the product and use overtime http://dx.doi.org/10.25304/rlt.v27.2242� research in learning technology citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 7 (page number not for citation purpose) (bargas-avila and hornbæk 2011). karapanos (2013) indicated that ux develops and changes over time. on the other hand, van der heijden (2004) observed that the factors that most affect the ux differ based on the purpose of the system and suggests that the perceived usefulness loses its power over time. according to karapanos, hassenzahl, and martens (2008), a positive ux is influenced by the time of using the product, the product itself, the context of use and the user. roto et al. (2011) indicate that there are three factors that affect ux including the context of use, the individual and the system. hassenzahl’s model in previous studies in human–computer interaction research, hassenzahl’s model has widely been used as a theoretical framework in various studies. furthermore, some researchers have extended hassenzahl’s model by merging factors such as overall quality and aesthetics. hassenzahl (2004) extended his model to include new factors to investigate the relationship between pragmatic, hedonic and goodness. hassenzahl’s model has been applied in different contexts, such as mobile learning, e-learning, systems, etc. for example, ozok, fan and norcio (2010) used hassenzahl’s model to study ux regarding the usage of a recommendation system. the findings of their study revealed that perceived interactivity and interface affected users’ attitudes and behaviour to use the system. schaik and ling (2008) explored the characteristics of web pages that affect ux and how the characteristics of the web pages influenced ux. other studies explored the impact of aesthetics and the website interface practically on ux (e.g. de angeli, sutcliffe, and hartmann 2006; tuch et al. 2012). the cognitive and physical capabilities of children are different from those of adults. therefore, children’s experience is measured in a different way. piaget (1970) divided children into different stages according to their cognitive and physical development. in the ages of 7 and 11, children are considered to be in the concrete operational stage (piaget 1970). in this stage, children are unable to formulate a hypothesis and have difficulty understanding abstract concepts. however, they can use a keyboard, use computer software, control the mouse and organise similar items together. also, children still admire and value software that incorporates playful activities (bruckman, bandlow, and forte 2002). children’s actions and behaviours reflect their thoughts because children have difficulty to express their feelings and thoughts with words (wea’am and alhussayen 2015). therefore, observers need to carefully read the behaviours and understand them within the context of concrete experiences. in addition, children respond truthfully about their opinion on the software they are interacting with (druin 2002). justification for using hassenzahl’s model in this study the researchers believe that hassenzahl’s model is an appropriate theoretical framework for addressing the research questions for two reasons. firstly, many studies in the field of human–computer interaction used the model (e.g. knijnenburg et al. 2012; ozok, fan, and norcio 2010). secondly, hassenzahl’s model could be used to study the factors that affect users experience in their interaction with the product, which fits the purpose of the study. http://dx.doi.org/10.25304/rlt.v27.2242� z.n khlaif et al. 8 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) research questions what is the first impression of children while using the tablet and educational applications for the first time? how do children interact with educational applications on tablets? how do children experience interaction with educational arabic applications on tablets? research design a qualitative approach was used for data collection such as three instruments observations, focus group discussion and sticky notes of the children to capture their first-time experience with math and science applications as well as using tablets. two focus group discussion sessions were held after two sessions of observing children. the two focus-group discussions were on thursday, september 31 and october 5, in the school for 1 h each session. focus-group discussion sessions enable students to talk about their own experience with both the device (tablet) and math and science applications. the researchers used open-ended questions to lead the focus group discussion and to be on track to achieve the purpose of the study and to obtain more details and insights on participants’ perceptions towards using the applications on tablets. in addition, we observed the kids while working on the application for around 3 h at different times. observational methods mean that the researchers stay in the room with users to observe their behaviour which reflects their feelings. druin (2002) stated that observation enables researchers to stay with the users throughout the evaluation period to explain the software and respond to questions. at the focus-group sessions, the researchers asked the students to talk about their experience with the device and the application. participants of the study the participants of the study were 53 children ageing from 8 to 10 years from different schools in different districts for 6 months. the study conducted in two schools in different districts. the main criteria to recruit the participants were: (1) school is within the electronic learning project (e-learning) and located in a marginalised area; (2) school had tablets to use for academic purposes; (3) math and science teachers are using different arabic applications in classrooms for academic purposes; (4) parents are willing to allow their children to participate in the study; (5) children are using the tablet and applications for the first time. description of the context palestine is a conflict zone and an unstable country with restrictions on the movement of people. many researchers mentioned some of this movement restrictions and its impact on the educational system in palestine (itmazi and tmeizeh 2008; khlaif 2018a; khlaif, gok, and kouraïchi 2019; shraim and khlaif 2010). the educational system influenced negatively by this conflict (shraim and khlaif 2010). the moe in palestine has adapted technological programmes with international funds to implement in  the palestinian schools including one-to-one technology (khlaif  2018a,  2018b). http://dx.doi.org/10.25304/rlt.v27.2242� research in learning technology citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 9 (page number not for citation purpose) in  addition, the moe is implementing the smart learning project and the national project of digitalization of education (khlaif and farid 2018) in many schools across the country. the moe distributes tablets in schools located in rural areas to improve the quality of education in four subjects such as math, science, english and arabic. after getting a permission from the directorate of education in both qalqilia and nablus (north of palestine), the researchers contacted the administration of a rural school to schedule a meeting with the principal and the math teacher to inform them about the purpose of the study. in the meeting between the researcher, the principal and the math teacher, the researchers chose third-grade students to be the subject of the study to achieve the purpose of the study. the criteria for choosing the students to participate in the study were that students have no previous experience with using the tablet and this is the first time they use the applications and the tablet. this criterion enabled the researchers to explore how students interact with the device for the first time. the researchers visited the classrooms after getting permission from the principal and the classroom teachers to inform them about the purpose of the study and invite them to participate in the study. the researchers sent a consent form to the parents of third grade to get their permission to involve their children in the study. fifty-three parents out of 60 agreed to allow their kids to participate in the study. therefore, the participants of the study were 53 students including 27 males and 26 females, aged 9 and 10 years old. they were all native arabic speakers and have basic english language skills. all students were from marginalised areas. some of them are relatives. most of the students came from limited-income families. the study took place in a room in each school where each child interacted with the tablet individually. the researchers chose the room to be comfortable for the students. the study was conducted in six sessions over 2 weeks. at the start of each session, the children were briefed on the purpose of the study and the tasks to be completed in the session. the children were given time to explore the application to complete the tasks. data collection several approaches were used to gather data regarding children’s thoughts and feelings. qualitative methods can capture the enjoyable and attractive parts of the software being used. observing children while interacting with the provided device and the applications (math and science) was the primary source of data. students’ comments on the sticky notes were another source of data together with the focus-group discussions of the participants’ experience. materials in the study, the researchers use tablets (galaxy tab3), third-grade math games application and science applications to achieve the purpose of the study procedures the study was conducted in a classroom for 25–30 min between 7:30 am and 8:00 am. at the beginning of the first session, the researchers informed the children purpose of the study and the tasks to be completed by using the apps. the teacher distributed the http://dx.doi.org/10.25304/rlt.v27.2242� z.n khlaif et al. 10 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) tablets to the students and assigned a number for each tablet for each student to use it in each session. in the first session of the meeting with the participants, the teacher introduced the tablets to the students and briefly instructed them about how to launch the device by pressing on the power button. then, she gave them brief instructions about the interface (figures 2 and 3) and the apps on it. the researchers talked briefly about the purpose of the study. after that, each student was invited to interact with the tablet individually to explore the device to be familiar with it. the teacher asked them to play with the device. the length of this activity was 25 min. at the end of each session, children were provided with sticky notes (figure 4) to report their experiences and perceptions about the tablet and the math app and express their opinion about the application in general. in the second part of the activity, the teacher gave the students an introduction about the math application and the software that they will use in their activities. figure 2. the desktop of the tablet and the educational applications on it. figure 3. the interface of the math application. http://dx.doi.org/10.25304/rlt.v27.2242� research in learning technology citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 11 (page number not for citation purpose) there was an agreement between the researcher, the school’s principal and the teacher to allow students to use tablets during their free time in school time to work on whatever they want. there are 45 min free time for children for eating and playing in the school (figure 5). usually, this time is in the middle of the school day. the researchers observed students while they were working on the tablets. the study was conducted in six sessions over 2 weeks. while students were working on the devices, three students asked about how to connect the devices to the internet. figure 4. students’ sticky notes. figure 5. children are playing with the device and the math application in and out of the classroom (a permission was taken from parents to get their kids pictures and share them). http://dx.doi.org/10.25304/rlt.v27.2242� z.n khlaif et al. 12 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) data analysis the researchers analysed the data by following the procedures suggested by marshall and rossman (2011). the researchers used the thematic analysis which is defined as a qualitative analytic method for categorising, analysing and reporting patterns (themes) within data (braun and clarke 2006). a grounded theory approach was used in our data analysis stage based on boschman et al.’s (2015) study which helped us to build the suband main themes. the six sessions were video-recorded and transcribed for data analysis. the researchers wrote down notes in the observation sessions. hassenzahl’s model was used in this study as a guideline for the data analysis process. the researchers coded the sticky notes into a positive and negative experience. furthermore, the researchers coded the focus group discussion based on hassenzah’s model. results the data collection methods help the researchers to answer the three research questions. for example, observations help to answer these research questions. in addition, students answered the questions when they describe their experience in the focus group discussions. the qualitative content analysis showed that participants’ enjoyment using the devices and the apps was quite different. during the sessions, children appeared excited and interested while interacting with math and science apps. however, girls were less excited to use the math application compared with their usage of the device itself. the girls were trying to explore other features of the device and to access the internet. following hassenzahl’s model, the descriptions of uxs were also categorised into two categories including pragmatic and hedonic. functionality, technicality and user-friendly aspects of ux of both the tablet and the applications were categorised as pragmatic. enjoyable, exciting, confusing and upsetting experiences were categorised as hedonic. males were more familiar to play with the devices than girls. some of the males mentioned that the tablet is like a smartphone (jawwal). two females did not explore the devices because of their fears to use it due to religious and cultural factors. these two girls mentioned that their families also talk about the disadvantages of using technology and how it tends to facilitate accessing bad websites on the internet. for that, these females hesitated to use the devices. as researchers, we consider this experience as a hedonic category. females expressed their excitement to use the devices because of the application on it. males opened the race competitions application, whereas females opened drawing applications. the researchers asked the students in the focus group sessions about their feelings while working on the device especially working on the applications and using the internet. students responded that they like the device, prefer to use the device in other subjects and want to take it home. students connected the devices to the internet and played online games in out-class activities. pragmatic features through analysing the observation notes and the sticky notes, it seems that students were happy while they were playing with the device. the researchers asked the children about the device and whether it was difficult to work on it or not. the majority http://dx.doi.org/10.25304/rlt.v27.2242� research in learning technology citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 13 (page number not for citation purpose) of students did not feel angry or upset while they were working on it. they were happy and laughing. ‘wow, it is easy to use’ (lina, female). a male told me his experience with it, he said ‘i thought it is difficult to use, … the interface is beautiful... i like its touch screen... i like to use it in my school…i wish i had one’ (ahmed, male) during the focus-group discussion, three participants (two females: rahaf and wesam, and one male: hamada) reported technical issues with the math application: ‘in the beginning, i couldn’t open the application, i tried to access to the internet, but i couldn’t… other students are working and i am not… i was frustrated’ (hamada, male). ‘it was difficult for me to do drag and drop to finish the task, i did not know why. i was confused…it is difficult to learn math by using a tablet’ (rahaf, female). from the observation notes, those students’ experience changed over time and they did not face the existed technical issues. apart from the technical issues, participants also stated that they perceived the tablet as well as the math app as user-friendly. ‘i can describe the tablet as well as the math app as “user-friendly” for me as a kid. this is the first time to use it, l learned how to use new technology’ (sama, female). ‘the interface was so friendly and it is easy to pick-up the application you want to use’ (seraj, male). however, two participants (one female and one male) noted that using the tablet and the math app were too difficult without the help of the teacher: ‘however, when i was left alone to use it in my free time, i understood that i cannot connect it to the internet and change the language of the interface from the english language into the arabic language. i asked my teacher to help me’ (manal, female). hedonic features students were excited and happy to use the device in their classroom as well as in their free time. others expressed their fears about using the device they were afraid to break it. ‘i do not like to play with it because i have to pay if i broke it…we do not have money’ (omer, male). on the other hand, two students (males) found that using math application to do some tasks becomes boring over time. ‘honestly, i didn’t enjoy solving math problems by using the app after some time. it just repeats using drag and drop over and over and gets a little boring then’ (aws, male). the majority of the participants were excited to use the math application and the tablet. learning math by using tablets and software made learning math easy and fun (as mentioned by nine participants). ‘coloring parts of the circle with my hand helps me to understand the fraction in math…it is fun to play with the application…i enjoy learning math by tablet…i hope to use it in other topics’ (sereen, female). some participants enjoyed playing the games: ‘overall i enjoyed the games on the tablet, i like to take it home to play with it with my sister’ (lina, female). two females were upset to use the math app because they did not finish the task. at the beginning of exploring the devices, the girls were hesitating to use the device, and they were uncomfortable to use it. the teacher intervened and told them it is like a toy; do not hesitate. analysing the notes on the sticky notes were categorised into positive and negative experience. most of the statements were positive, indicating positive exciting experience, happiness, enjoyment, wishing to use the tablet in other subjects, easy to finish http://dx.doi.org/10.25304/rlt.v27.2242� z.n khlaif et al. 14 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) math task by using math app, touch screen, and colouring by hand is easy. on the other hand, minor statements expressed negative feedback, indicating that it was upsetting, frustrating and boring because of using the math app to do tasks in the classroom. most of these negative experiences were related to technical issues and absence of technical support. children mentioned that using the tablet and the math application was helpful for them to understand math (mentioned by nine students: four females and five males). the researchers observed the students while they were working on the devices in their free time. gender differences from the observation notes and students’ sticky notes, it is clear that girls prefer to play with girls’ games such as colouring, weddings and dressing games. three students felt upset while using some features of the application such as the drag and drop feature. it was difficult to distinguish between the experience of children based on their gender due to the limited time as well as the sample size. it is a good idea to explore this factor (gender) on ux by including more males and females. however, gender differences were found with two girls that they were unsatisfied with the experience than their colleagues and other males. working on tablets in the free time the observation of children while working on the tablets in their free time revealed that students prefer to work on different applications rather than only on the math app. data from observing students while working on tablets in their free time revealed that students were happy and excited. they were working on whatever they wanted. the researchers observed the student who was complaining about the device and the application. ‘i got assistance from my teacher to access to the internet, i am happy to use it now…i enjoy playing with it because i need to explore it and i do not have a specific task to finish…now i do not feel bored’ (rahaf, female). two male students connect their happiness of using the tablet in their study to their financial situation. one said ‘…i am so happy to use the tablet and math app in my school… i cannot imagine that one day i will use it because it is difficult for my parents to buy one for me or for my brother because of our financial situation’ (sameer, male). children were happier when working on the math app in the classroom. children did not report any issues related to the functionality of the application they worked on or the device. they were comfortable while using the device. discussion based on data analysis of different data sources in this study, it could be concluded that the participants enjoyed using the tablet and the applications for the first time. the findings of the study do not show any important differences in the enjoyment of using the tablet and the application. however, the focus group discussion revealed that there seems to be a gender difference regarding the enjoyment of using the tablet and the applications that participants want to use in their free time. the interface of the tablet as well as the application were easy to use and had a positive impact on kids’ http://dx.doi.org/10.25304/rlt.v27.2242� research in learning technology citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 15 (page number not for citation purpose) experience. ux depends on different factors such as the environment (in-class or outof-class), whether using new technology is mandatory or voluntary. in this study, students were more excited and are happier when they worked on the tablets in their free time. technical support is important in ux especially if the users are kids and it could change their experience in a positive or a negative way. it depends on the context. according to the observation notes, the ux of using the tablet and the math application changed over a short time. however, some participants felt bored of using the math app for a long time. ux depends on the context of using the product, as well as whether it is optional or mandatory. these findings are congruent with the findings of previous studies (e.g. hassenzahl and tractinsky 2006; roto and kaasinen 2008). in this case, students were happier when they used the tablet without any restrictions compared with using the device with the math application. using tablets in their free time was not mandatory like implementing the tasks in the classroom and students could choose whatever they wanted to use. therefore, children chose the application which would help them to enjoy using the tablet. congruent with the findings of hassenzahl (2005) and minge (2008), children were eager to use the math application and the tablet because they were happy and they enjoyed using them, which has a positive impact on their experience. children knew the value of the device while they were working on it in their class to finish the assigned tasks. they considered it as a useful tool to understand math. many of them preferred to use it in other subjects and to take it home. the researchers did not explore the cultural and religious factors; and whether they influence the overall experience because it is out of the study scope. it could be examined in a different study in the future. based on the results of this study, children’s experience was influenced by many factors including ease of use of the application, technical issues, the context of using the tablet or the math application (whether mandatory or optional), the usefulness of the application and the device. conclusion and future research the purpose of the present research was to explore children’s first experience with tablets and the math app. most of the participants were excited and enjoyed learning with the tablet. however, some of them felt frustrated and bored using the tablets. ux is indeed affected by the environment where they are using it and whether it is mandatory or voluntary. technical issues have a negative impact on children experience. this study does not focus on the role of gender and the factor on the ux. so, a similar study can be conducted with another learning application that is generally used by a higher grade or the same grade but including more students and design the research to be a mixed-method research. references alhussayen, a., alrashed, w. & mansor, e. i. (2015) ‘evaluating the user experience of playful interactive learning interfaces with children’, procedia manufacturing, vol. 3, pp. 2318–2324. doi: 10.1016/j.promfg.2015.07.378 aziz, n. a. a. (2013) ‘children’s interaction with tablet applications: gestures and interface design’, children, vol. 2, no. 3, pp. 447–450. bargas-avila, j. a. & hornbæk, k. (2011) ‘old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience’, proceedings of the sigchi conference http://dx.doi.org/10.25304/rlt.v27.2242� https://doi.org/10.1016/j.promfg.2015.07.378� z.n khlaif et al. 16 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) on human factors in computing systems, acm, new york, usa, pp. 2689–2698. doi: 10.1145/1978942.1979336 bevan, n. (2009) ‘extending quality in use to provide a framework for usability measurement’, proceedings of hci international 2009, san diego, ca, usa. doi: 10.1007/978-3-642-02806-9_2 boschman, f., mckenney, s. & voogt, j. (2015) ‘exploring teachers’ use of tpack in design talk: the collaborative design of technology-rich early literacy activities’, computers & education, 82, pp. 250–262. doi: 10.1016/j.compedu.2014.11.010 braun, v. & clarke, v. (2006) ‘using thematic analysis in psychology’, qualitative research in psychology, 3(2), pp. 77–101. bruckman, a., bandlow, a. & forte, a. (2002) hci for kids. chang, c. k. (2008) ‘usability comparison of pen-based input for young children on mobile devices’, sensor networks, ubiquitous and trustworthy computing, 2008. sutc'08. ieee international conference, ieee, taiwan, pp. 531–536. doi: 10.1109/sutc.2008.8 chau, c. l. (2014) positive technological development for young children in the context of children's mobile apps, doctoral dissertation, tufts university. de angeli, a., sutcliffe, a. & hartmann, j. (2006) ‘interaction, usability and aesthetics: what influences users’ preferences?’, proceedings of the 6th conference on designing interactive systems, acm, new york, ny, pp. 271–280. doi: 10.1145/1142405.1142446 deng, s., et al., (2015) ‘understanding the factors influencing user experience of social question and answer services’, information research: an international electronic journal, vol.  20, no. 4, p. 4. druin, a. (2002) ‘the role of children in the design of new technology’, behaviour and information technology, vol. 21, no. 1, pp. 1–25. doi: 10.1080/01449290110108659 glanznig, m. (2012) ‘user experience research: modelling and describing the subjective’, interdisciplinary description of complex systems: indecs, vol. 10, no. 3, pp. 235–247. doi: 10.7906/indecs.10.3.3 han, d. i., tom dieck, m. c. & jung, t. (2017) ‘user experience model for augmented reality applications in urban heritage tourism’, journal of heritage tourism, vol. 13, pp. 1–16. doi: 10.1080/1743873x.2016.1251931 harbich, s. & hassenzahl, m. (2017) ‘user experience in the work domain: a longitudinal field study’, interacting with computers, vol. 29, no. 3, pp. 306–324. doi: 10.1093/iwc/iww022 hassenzahl, m. (2004) ‘the interplay of beauty, goodness, and usability in interactive products’, human-computer interaction, vol. 19, no. 4, pp. 319–349. doi: 10.1207/s15327051hci1904_2 hassenzahl, m. (2005) ‘the quality of interactive products: hedonic needs, emotions and experience’, in encyclopedia of human computer interaction. idea group. hassenzahl, m. (2008) ‘user experience (ux): towards an experiential perspective on product quality’, in proceedings of the 20th international conference of the association francophone d’interaction hommemachine on ihm '08, acm, new york, usa, p. 11. doi: 10.1145/1512714 hassenzahl, m. (2010) ‘experience design: technology for all the right reasons’, synthesis lectures on human-centered informatics, vol. 3, no. 1, pp. 1–95. doi: 10.2200/ s00261ed1v01y201003hci008 hassenzahl, m. & tractinsky, n. (2006) ‘user experience – a research agenda’, behavior & information technology, vol. 25, no. 2. pp. 91–97. doi: 10.1080/01449290500330331 hornbæk, k. & hertzum, m. (2017) ‘technology acceptance and user experience: a review of the experiential component in hci’, acm transactions on computer-human interaction (tochi), vol. 24, no. 5, p. 33. doi: 10.1145/3127358 iso dis 9241-210. (2010) ergonomics of human system interaction -part 210: human-centred design for interactive systems (formerly known as 13407), international standardization organization (iso), geneve, switzerland. http://dx.doi.org/10.25304/rlt.v27.2242� https://doi.org/10.1145/1978942.1979336� https://doi.org/10.1145/1978942.1979336� https://doi.org/10.1007/978-3-642-02806-9_2� https://doi.org/10.1007/978-3-642-02806-9_2� https://doi.org/10.1016/j.compedu.2014.11.010 https://doi.org/10.1109/sutc.2008.8� https://doi.org/10.1145/1142405.1142446� https://doi.org/10.1080/01449290110108659� https://doi.org/10.7906/indecs.10.3.3� https://doi.org/10.1080/1743873x.2016.1251931� https://doi.org/10.1080/1743873x.2016.1251931� https://doi.org/10.1093/iwc/iww022� https://doi.org/10.1207/s15327051hci1904_2� https://doi.org/10.1145/1512714� https://doi.org/10.2200/s00261ed1v01y201003hci008� https://doi.org/10.2200/s00261ed1v01y201003hci008� https://doi.org/10.1080/01449290500330331� https://doi.org/10.1145/3127358� research in learning technology citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 17 (page number not for citation purpose) itmazi, j. & tmeizeh, m. (2008) ‘blended e-learning approach for traditional palestinian universities’, ieee multidisciplinary engineering education magazine, vol. 3, no. 4, pp. 156–162. karapanos, e. (2013) ‘user experience over time’, modeling users' experiences with interactive systems, pp. 57–83, springer, berlin. doi: 10.1007/978-3-642-31000-3_4 karapanos, e., hassenzahl, m. & martens, j.b., 2008, user experience over time, chi 2008, april 5 – april 10, 2008, florence, italy, acm, 978-1-60558-012-8/08/04. doi: 10.1145/1358628.1358891 khlaif, z. (2018b) ‘teachers’ perceptions of factors affecting their adoption and acceptance of mobile technology in k-12 settings’, computers in the schools, vol. 35, no. 1, pp. 49–67. doi: 10.1080/07380569.2018.1428001 khlaif, z., gok, f. & kouraïchi, b. (2019) ‘how teachers in middle schools design technology integration activities’, teaching and teacher education, vol. 78, pp. 141–150. doi: 10.1016/j. tate.2018.11.014 khlaif, z. n. (2018a) ‘factors influencing teachers’ attitudes toward mobile technology integration in k-12’, technology, knowledge and learning, vol. 23, no. 1, pp. 161–175. doi: 10.1007/s10758-017-9311-6 khlaif, z. n. & farid, s. (2018) ‘transforming learning for the smart learning paradigm: lessons learned from the palestinian initiative’, smart learning environments, vol. 5, no. 1, p. 12. doi: 10.1186/s40561-018-0059-9 kim, j., kim, j. & moon, j. y. (2013) ‘does age matter in mobile user experience? impact of age on relative importance of antecedents of mobile user experience’, pacis, p. 189. knijnenburg, b. p., et  al., (2012) ‘explaining the user experience of recommender systems’, user modeling and user-adapted interaction, vol. 22, no. 4–5, pp. 441–504. doi: 10.1007/ s11257-011-9118-4 kujala, s., et  al., (2011) ‘ux curve: a method for evaluating long-term user experience’, interacting with computers, vol. 23, no. 5, pp. 473–483. marshall, c. & rossman, g. b. (2011) ‘managing, analyzing, and interpreting data’, c. marshall & gb rossman, designing qualitative research, vol. 5, pp. 205–227. mccarthy, j. & wright, p. c. (2004) technology as experience, mit press, boston, usa. doi: 10.7551/mitpress/6687.003.0011 mcknight, l. & cassidy, b. (2012) ‘children’s interaction with mobile touch-screen devices: experiences and guidelines for design’, in social and organizational impacts of emerging mobile devices: evaluating use, hershey, pa, usa, p. 72. doi: 10.4018/978-1-4666-0194-9. ch004 minge, m. (2008) ‘dynamics of user experience’, proceedings of the workshop on research goals and strategies for studying user experience and emotion, no rdichi, vol. 8. ozok, a. a., fan, q. & norcio, a. f. (2010) behaviour & information technology, vol. 29, no. 1, pp. 57–83. doi: 10.1080/01449290903004012 page, t. (2013) ‘use of mobile device apps in product design’, international journal of green computing (ijgc), vol. 4, no. 1, pp. 18–34. doi: 10.4018/jgc.2013010102 partala, t. & saari, t. (2015) ‘understanding the most influential user experiences in successful and unsuccessful technology adoptions’, computers in human behavior, vol. 53, pp. 381–395. doi: 10.1016/j.chb.2015.07.012 piaget, j. (1970) science of education and the psychology of the child. trans. d. coltman. orion press, new york. plowman, l. & stevenson, o. (2012) ‘using mobile phone diaries to explore children’s everyday lives’, childhood, vol. 19, no. 4, pp. 539–553. doi: 10.1177/0907568212440014 provost, g. & robert, j. m. (2013) ‘the dimensions of positive and negative user experiences with interactive products’, lecture notes in computer science, vol. 8012 lncs, no. part 1, pp. 399–408. doi: 10.1007/978-3-642-39229-0_43 http://dx.doi.org/10.25304/rlt.v27.2242� https://doi.org/10.1007/978-3-642-31000-3_4� https://doi.org/10.1145/1358628.1358891� https://doi.org/10.1145/1358628.1358891� https://doi.org/10.1080/07380569.2018.1428001� https://doi.org/10.1080/07380569.2018.1428001� https://doi.org/10.1016/j.tate.2018.11.014� https://doi.org/10.1016/j.tate.2018.11.014� https://doi.org/10.1007/s10758-017-9311-6� https://doi.org/10.1007/s10758-017-9311-6� https://doi.org/10.1186/s40561-018-0059-9� https://doi.org/10.1007/s11257-011-9118-4� https://doi.org/10.1007/s11257-011-9118-4� https://doi.org/10.7551/mitpress/6687.003.0011� https://doi.org/10.7551/mitpress/6687.003.0011� https://doi.org/10.4018/978-1-4666-0194-9.ch004� https://doi.org/10.4018/978-1-4666-0194-9.ch004� https://doi.org/10.1080/01449290903004012� https://doi.org/10.4018/jgc.2013010102� https://doi.org/10.1016/j.chb.2015.07.012� https://doi.org/10.1177/0907568212440014� https://doi.org/10.1007/978-3-642-39229-0_43� z.n khlaif et al. 18 citation: research in learning technology 2019, 27: 2242 http://dx.doi.org/10.25304/rlt.v27.2242 (page number not for citation purpose) rahman, m. s. a., ali, n. m. & mohd, m. (2013) ‘a study on the naturalness of gesture-based interaction for children’, in advances in visual informatics. ivic 2013, lecture notes in computer science, eds h.b. zaman, p. robinson, p. olivier, t.k. shih & s. velastin, springer, cham, vol. 8237. doi: 10.1007/978-3-319-02958-0_65 roto, v. & kaasinen, e. (2008) ‘the second international workshop on mobile internet user experience’, in proceedings of the 10th international conference on human computer interaction with mobile devices and services, acm, new york, pp. 571–573. doi: 10.1145/1409240.1409354 roto, v., obrist, m. & väänänen-vainio-mattila, k. (2009) ‘user experience evaluation methods in academic and industrial contexts’, in interact 2009 conference, user experience evaluation methods in product development (uxem'09), uppsala, sweden. doi: 10.1007/978-3-642-03658-3_141 roto, v., et al., (2011) ‘bringing clarity to the concept of user experience’, available at: www. allaboutux.org/files/ux-whitepaper.pdf schaik, p. v. & ling, j. (2008) ‘modelling user experience with web sites: usability, hedonic value, beauty and goodness’, interacting with computers, vol. 20, no. 3, pp. 419–432. doi: 10.1016/j.intcom.2008.03.001 shraim, k. & khlaif, z. (2010) ‘an e-learning approach to secondary education in palestine: opportunities and challenges’, information technology for development, vol. 16, no. 3, pp. 159–173. doi: 10.1080/02681102.2010.501782 shaqour, a. z. (2014) ‘students’ readiness towards m-learning: a case study of pre-service teachers in palestine’, journal of educational and social research, vol. 4, no. 6, p. 19. doi: 10.5901/jesr.2014.v4n6p19 sharp, h., rogers, y. & preece, j. (2007) interaction design: beyond human-computer interaction. wiley & son, usa. silvennoinen, j., vogel, m. & kujala, s. (2014) ‘experiencing visual usability and aesthetics in two mobile application contexts’, journal of usability studies, vol. 10, no. 1, pp. 46–62. thüring, m. & mahlke, s. (2007) ‘usability, aesthetics, and emotions in human-technology interaction’, international journal of psychology, vol. 42, pp. 253–264. doi: 10.1080/0020 7590701396674 tuch, a. n, et al., (2012) ‘is beautiful really usable? toward understanding the relation between usability, aesthetics, and affect in hci’, computers in human behavior, vol. 28, no. 5, pp. 1596–1607. doi: 10.1016/j.chb.2012.03.024 utair, n. (2015) the impact of smart devices on raising children, al quds open university, ramallah. van der heijden, h. (2004) ‘user acceptance of hedonic information systems’, mis quarterly, pp. 695–704. doi: 10.2307/25148660 walsh, t., et al., (2014) ‘axe ux: exploring long-term user experience with iscale and attrakdiff ’, in proceedings of the 18th international academic mindtrek conference: media business, management, content & services, acm, pp. 32–39. doi: 10.1145/2676467.2676480 wea’am, a, a. & alhussayen, a. a. (2015) ‘examining the user experience (ux) of children’s interaction with arabic interfaces in educational learning contexts’, in international  conference on human-computer interaction, springer, cham, pp. 349–354. doi: 10.1007/978-3-319-21380-4_59 zucker, a. a. & light, d. (2009) ‘laptop programs for students’, science, vol. 323, no. 5910, pp. 82–85. doi: 10.1126/science.1167705 http://dx.doi.org/10.25304/rlt.v27.2242� https://doi.org/10.1007/978-3-319-02958-0_65� https://doi.org/10.1007/978-3-642-03658-3_141� https://doi.org/10.1007/978-3-642-03658-3_141� www.allaboutux.org/files/ux-whitepaper.pdf� www.allaboutux.org/files/ux-whitepaper.pdf� https://doi.org/10.1016/j.intcom.2008.03.001� https://doi.org/10.1016/j.intcom.2008.03.001� https://doi.org/10.1080/02681102.2010.501782� https://doi.org/10.5901/jesr.2014.v4n6p19� https://doi.org/10.5901/jesr.2014.v4n6p19� https://doi.org/10.1080/00207590701396674� https://doi.org/10.1080/00207590701396674� https://doi.org/10.1016/j.chb.2012.03.024� https://doi.org/10.2307/25148660 https://doi.org/10.1145/2676467.2676480� https://doi.org/10.1007/978-3-319-21380-4_59� https://doi.org/10.1126/science.1167705� (page number not for citation purpose) 1 *corresponding author. email: nadine.schroeder@fh-bielefeld.de research in learning technology 2023. © 2023 n. schroeder and s. donat. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 research in learning technology vol. 31, 2023 original research article ‘give and take’ – higher education teachers using open educational resources nadine schroeder* and sophia donat department of educational sciences, learning lab, university of duisburg-essen, essen, germany (received: 05 october 2022; revised: 12 january 2023; accepted: 26 february 2023; published: 17 april 2023) open educational resource (oer) as free teaching and learning materials can contribute to the collaborative design and development of teaching. to support higher education teachers in their work with teaching in general and oer in particular and to encourage their use of oer, it is necessary to pay attention to their needs and requirements. this paper presents the results of a research project, identifying the usage behaviour of german-speaking higher education teachers. in an interview study, they were asked about their experience with oer to get detailed insights into their practices concerning their ‘use’ and ‘revise’ of materials. from this, four user types were derived according to different oer activities, such as creating, reusing, editing, and publishing oer, and their scope. finally, these user types are transferred to considerations when designing oer infrastructures and establishing support options. these are aligned with the specifics of each user type, making the research findings a complementary contribution for application in higher education. keywords: open educational resources; higher education; usage; practices; reusing introduction open educational resources (oer) provide access to and allow the exchange of digital teaching and learning materials. in addition, oer also enables access to open material and gives educators the opportunity to network within a community. they can create learning materials with feedback and input from all over the world and thereby improve their teaching professionalism. therefore, oer have great potential to make teaching and learning more efficient and more collaborative in open and distributed environments. one challenge here is to establish a culture of give and take among educators so that communities can emerge to create and edit oer. therefore, detailed insights into teachers’ practices when working with oer are necessary to further develop support structures and technical infrastructures. this paper presents the results of a research project that explores oer usage behaviour by higher education teachers as well as their needs and requirements concerning oer. for this purpose, an interview study was conducted with oer mailto:nadine.schroeder@fh-bielefeld.de http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2916 n. schroeder and s. donat 2 citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 (page number not for citation purpose) experienced higher education teachers about their practical usage with oer. however, barriers that are common among non-users of oer, such as licensing, or technology acceptance, are not considered in this study. after presenting theoretical concepts and usage studies that resulted in the methodological approach, basic results are described. from this, four user types according to different oer activities and their scope were derived. finally, these findings are discussed in the current discourse and the user types are transferred to considerations when developing and establishing support. background concepts of oer usage the idea of oer can be presented in the context of open education (zawackirichter et al. 2020) and can be seen as part of the concept of open educational practices (oep), which describes the creation and use of oer related to the application of open teaching methods and collaborative practices (cronin 2017; ehlers 2011). in addition, learners can benefit from open practices by developing open materials and  shaping their learning process. according to wiley and hilton (2018), this oer-enabled pedagogy is made possible by the 5rs. the concept of 5rs according to wiley (2014) illustrates the possibilities of open licenses when dealing with educational materials. the access to materials and the permission to own materials as a personal copy (retain) are the prerequisites for the further rights to use, edit and share oer. thus, materials with open licenses can be reused by using and integrating them into one’s own materials (reuse), edited by making changes and adaptations (revise), mixed by combining different contents (remix), and shared publicly (redistribute). these usage rights can be reflected in different oer lifecycles (beaven 2018; fulantelli et al. 2008; pawlowski 2012) to illustrate how practitioners create learning and teaching materials. these lifecycles have in common that they include the elements search, design, use, edit and share which reflect the phases of working with oer. these approaches systemize the usage possibilities and behaviour regarding oer so that these are summarized for the theoretical classification of the research interest of this study. based on the elements of oer lifecycles and the 5rs, four activities as central part of the oer concept (gurell and wiley 2008) can be mapped in the practical usage of oer in the context of creating, reusing, editing and publishing open materials (see table 1). in the production of oer, external open content can be used (reuse) and, if necessary, inserted into own materials with modifications and adaptations (edit). in distinction to the integration of external table 1. oer activities related to oer cycle and 5rs. oer activity oer cycle 5r create design reuse use reuse edit edit revise, remix publish share redistribute http://dx.doi.org/10.25304/rlt.v31.2916 research in learning technology citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 3 (page number not for citation purpose) oer, own materials can also serve as a basis for the development of oer (create). finally, compiled oer are publicly shared and disseminated under an open license (publish). usage studies of oer the application of each element of the 5rs and the oer cycle has been explored in various studies in higher education contexts presented below. educators seem to engage in various oer activities in their practice by searching, merging, modifying and using oer (beaven 2018). integrating external oer into one’s own materials occurs with and without modifications (rodés et al. 2019). external materials are adapted for individual contexts, usually by adapting or changing their content ( cardoso, morgado, and teixeira 2019). different practices could be observed for different types of materials. while videos and images are reused without modifications, instructors often make adaptations for course units. presentations and assignments are material types that are predominantly self-created and not reused by others (baas and schuwer 2020). by reusing oer, instructors aim to improve the quality of their teaching, but they want to avoid too much effort for adaptations and remixing (baas, admiraal, and van den berg 2019). it is an open question how and to what extent remix activities are applied (wiley, bliss, and mcewen 2014). it could be found that teachers rarely publish oer in repositories, but instead share their materials with students and colleagues in informal settings (beaven 2018). this provided evidence of dark reuse (wiley 2009), as use and dissemination take place outside of repositories (baas, admiraal, and van den berg 2019; beaven 2018). moreover, materials are often shared without open licenses (baas, admiraal, and van den berg 2019; schuwer and janssen 2018). five usage types were derived from different usage patterns of educators based on survey data from the oer research hub (admiraal 2022; open university 2015). the characteristics of the usage types are based on activities of instructors related to the creation, use, adaptation, publication, and commenting on resources in distinction of material types as well as the purpose of use. almost half of the participants could be assigned to type 1 (49%), which is characterized by a high value of reuse and editing of oer for various purposes, while own materials are not shared. the other types are distributed in comparable proportions. type 2 (12%) has the highest value in commenting on resources in a repository, in addition to adapting oer. while type 3 (12%) creates, edits and publishes oer, type 4 (11%) participants have a high value in all usage characteristics (creating, editing, publishing and commenting). type 5 (16%) excels in consuming oer by scoring low on all characteristics. thus, adapting oer was found to be the largest proportion, while creating and publishing their own oer were lowest. quantitative usage studies provide basic information about low usage of oer (baas, admiraal, and van den berg 2019; schuwer and janssen 2018) by teachers and observations determined use and sharing as non-visible (beaven 2018). however, detailed insights into practices and behaviours of teachers in using and revising oer and their needs are lacking (heck et al. 2020). these insights from qualitative studies are necessary to support instructors in using oer in their teaching practices. therefore, this paper presents the results of a qualitative research project that identifies http://dx.doi.org/10.25304/rlt.v31.2916 n. schroeder and s. donat 4 citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 (page number not for citation purpose) usage patterns of oer by higher education teachers in order to facilitate the design of support measures specifically for different user types on this basis. method since previous studies were mostly able to determine rather general information quantitatively (heck et al. 2020), in light of the research needs and the depth of oer activities presented above, this study aims to identify actual practices of higher education teachers in dealing with oer. the qualitative approach was chosen in order to gain these deeper insights into the actual work with oer, which is often not possible with quantitative research approaches due to operationalization processes (kelle 2006). the data were collected through semi-structured interviews as those allow to gather both specific pieces of information in a comparable format and leave room for details on individual user experience (galletta 2013). based on the overall research question ‘how do higher education teachers use oer?’, the following sub-research questions were formulated: • what types of oer-material do teachers create, use, edit and publish? • how do teachers create, use, edit and share oer in their educational practice? • what needs and requirements do teachers have when dealing with oer? • can certain types of use be identified? the interviews (n = 18) were conducted with german-speaking university lecturers (8 female; 10 male) from germany and austria in video interviews (30–60 min) between july and september 2020. since the focus of this study is on the practices of higher education teachers with oer as well as their explicit and implicit knowledge about the use of oer, the interviews were conducted with teachers with oer experience only. participants were identified from existing networks of oer research projects and internet searches for active individuals. based on this, personal recommendations from colleagues were included. this had the advantage that further participants could be selected who, due to their background, could contribute to the broadest possible overall picture. 22 potential participants were contacted directly by e-mail, resulting in 18 interview confirmations. prerequisites for the final participation in the interviews were: • teaching at a german-speaking university • creating open materials for teaching • at least one further oer activity (reuse, edit, publish oer) accordingly, the interviewees were all active in university teaching (eight doctoral students, six postdocs, four professors) and have practical knowledge regarding oer. to adequately represent the diversity of active oer users some of the interviewees first encountered with creating oer within the framework of projects, while other teachers have been continuously practicing oer for several years. however, differences in the extent of experience were not recorded, so no conclusions can be drawn in this regard in the analysis. in terms of subject, six teachers can be assigned to the natural sciences and 11 participants to the humanities and social sciences. to structure the interviews according to the research questions, a semi-structured interview guide with four main content areas was developed (roulston and choi 2018). as an introductory narrative stimulus, the teachers were asked to present one http://dx.doi.org/10.25304/rlt.v31.2916 research in learning technology citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 5 (page number not for citation purpose) of their own oer. the other topics – design, development and implementation/evaluation – were based on the presented theoretical background on oer and supplemented with more specific questions. the collected data were partially transcribed using interview-accompanying documentation, that is, only segments related to the specific research topic were transcribed and considered for the analysis. the corresponding interview segments were coded and categorized using the maxqda analysis software according to the content structuring qualitative method of kuckartz (2018), which in contrast to other qualitative approaches, such as the grounded theory methodology (e.g. corbin and strauss 2015), allows a rather structured analysis of the collected interview data. for the category formation, main categories oriented to the underlying research questions were first formed deductively for content framing and focusing. these are, based on the framework of the interviewees, different types of material, their practices, and needs and requirements. the first sub-category was formed on the basis of the topics from the interview guide (see table 2). to specify the characteristics, various further subcategories were formulated inductively (kennedy and thornberg 2018). instead of following up with a type-forming qualitative content analysis, approaches to type formation were made based on the identified content-structured categories that refer specifically to the usage behaviour of individuals, using the document map in maxqda. the typing was done exclusively with first and second order subcategories that explicitly refer to experiences and practices working with oer. this approach allows a focused look at the different types of users without including too many environmental factors. the individual subcategories were then assigned to the characterising oer activities (see table 3). table 2. coding scheme. main category first order subcategory general conditions oer experience oer motivation example materials origin of materials material types creating reusing editing publishing creating and reusing collaboration scope of reusing remix changes editing after use in teaching feedback types of changes after use further materials sharing internal public needs, requirements difficulties suggestions version management http://dx.doi.org/10.25304/rlt.v31.2916 n. schroeder and s. donat 6 citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 (page number not for citation purpose) results the evaluation of the qualitative study has resulted in findings about the use of oer in university teaching. after presenting the main results with accompanying quotes from the participants, derived user types of higher education teachers dealing with oer are presented. the teachers’ sample oer provided the framework for the details and background on their open activities and practices. for seven participants, large-scale materials (e.g. courses, videos) were supported by projects aimed at publication as oer. four projects on providing oer were self-initiated and occurred on a one-time basis. however, all materials have been used in their university’s own teaching. the conception of these oer was either directly openly accessible or corresponding oer was published based on closed courses. material types the first research sub-question aimed to find out what types of materials teachers create, use, edit and publish. the evaluation was then evaluated based on these oer activities (see figure 1). presentations, tasks, h5p elements and didactic concepts are mostly created as open materials. in contrast to this, these material types are not published individualy, but integrated into entire courses. similar observations can be made for other large-scale resources with high production effort, such as videos, textbooks, or an interactive app. even though published oer usually have a larger scope, teachers tend to use independent, smaller types of material for their teaching, for example, images. due to easy search options and simple possibilities for integration, images and graphics are the most frequently reused type of material. images are usually adapted by minor changes, for example, cut or color adjusted, before embedding in own materials. graphics without open licenses are often reworked or repurposed to avoid legal consequences. the reuse of videos is mostly done by linking to external platforms. edits of videos take place, for example, by integrating h5p elements with interactive exercises. worksheets comprise all oer activities by creating own working materials, reusing external ones, individually adapting, and combining them by remix and finally making them publicly available. table 3. relevant subcategories for creating user types linked to oer activities. oer activity main category first + second order subcategory create material types creating general conditions origin of materials: own materials reuse material types reusing general conditions origin of materials: external oer edit material types editing creating and reusing remix: oer publish material types publishing sharing public http://dx.doi.org/10.25304/rlt.v31.2916 research in learning technology citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 7 (page number not for citation purpose) practices the second research sub-question ‘how do teachers create, use, edit and share oer in their educational practice?’ was intended to provide a more detailed look into the practices of the teachers. here it was found that as a material base for creating oer, almost half of the participants (8) rely on their own materials and only occasionally (8) or not at all (2) on external oer. poor findability and availability in the subject areas, lack of open licenses or high adaptation effort are mentioned as hurdles for the use of oer (‘if material doesn’t fit 100% and it’s too much that i would have to change, […] then i rather do it myself ’. participant 13; translation by the authors). however, available resources are an important source for impulses because ‘it’s also living from getting inspired’ (participant 14). eight participants integrate external oer from different types of materials as well as individual elements of a resource into their own content. using materials without any changes happens rather rarely, because usually adaptations to contexts and to the respective target group are necessary. to integrate external content into one’s own concept, not only the content but also the scope, structure, arrangement, or design need to be customized. combining different materials into a new resource is a typical scenario for the creation and reuse of external content, to ‘put things together like a construction kit’ (participant 01), because ‘it is also about mixing things up, for example, putting things into a different context again and comparing them […]’ (participant 14). one challenge when remixing oer is finding materials that are available with open licenses because only those can be reused. five participants stated that they actively use oer remixing in their daily practice. for these users, ‘remix [is] a central element of oer. swarm intelligence can be used in the process’ (participant 06). instead of using entire courses (‘it would be great coincidence if i found exactly one course that fits’. participant 02) some (5) prefer to adopt individual course elements with the addition of their own content and target group-specific adaptation. in some cases, videos and courses are applied as complete materials by linking to them as self-study materials. figure 1. material types by oer activities. http://dx.doi.org/10.25304/rlt.v31.2916 n. schroeder and s. donat 8 citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 (page number not for citation purpose) the interviewees usually share materials both in a closed space with students (18) and partly with colleagues (11) within their own institution or the professional community as well as publicly. although materials on a smaller scale are preferably reused, they are rarely published. with regard to reusing and editing materials, it emerged during the interviews that transparency of changes and improvements as well as feedback and exchange within a community are of particular importance. the resources presented by the interviewees were created either alone (6), with colleagues from their own institution (6) or across institutions in project teams (6). in some cases, correction loops were integrated for collaborative revision of the content. according to the interviewees, intradepartmental exchange among colleagues and inter-institutional exchange are only common in a few cases. it can be stated that ‘this community idea […] has not yet been established’ (participant 02). feedback and suggestions for possible changes are considered valuable by all participants. ‘every material […] is somehow in need of improvement’ (participant 01) and ‘it […] [gains] quality when you have the view from the outside again’ (participant 03). an oer-experienced and -active participant who has already received feedback via a personal blog and twitter emphasizes, ‘what i really appreciate about oer is that you get peer feedback and can really improve’ (participant 16). peer feedback is thus appreciated, however, some participants ‘have not yet found an active community to contribute’ (participant 07). often there is not enough time for regular necessary updates, especially with project-based materials. this is where an active community can provide useful support (‘at best, others develop it further so that it doesn’t necessarily have to be done by the person who created the material’. participant 18). needs and requirements expectations and needs of the users could be identified, especially regarding the technical infrastructures. low-threshold access to standardized and user-friendly oer tools appears to be central. according to the interviewees, platforms for searching oer and publishing different types of material could be expanded. the subdivision of extensive materials into individual units, for example, images from a presentation, could promote reuse. functions for creating of materials collaboratively and interactive exchange options are also desirable. to make further developments of content visible, teachers expect a possibility to provide new versions. for half of the participants, it is relevant that they can access older versions and track changes between version. (‘i also wanted to show what didn’t work so well and where there is still potential for improvement’. participant 13). however, the freedom of the individual authors must be considered. one participant summarizes: ‘i would like to be able to decide, this can stay, this is still relevant […] or i can say i don’t want this to be found anymore either, it’s so outdated now’ (participant 04). the desire for feedback and exchange as well as an interest in learning about external reuses of their own materials can be identified. the interviewees themselves provided feedback on oer, which they reused and edited, only in a few cases. accordingly, there is a lack of an easy way to contact creators and functions for commenting to realise voluntary feedback in the oer-sense of ‘give and take’ ( participant 04). http://dx.doi.org/10.25304/rlt.v31.2916 research in learning technology citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 9 (page number not for citation purpose) user types the final research focus addressed the question of whether different types of users can be identified among teachers. based on the results presented, the individual interviewees can be assigned to identified oer activities: create, reuse, edit and publish (see table 1). for that, coded elements identified in the course of the qualitative content analysis (see chapter method) that can be considered relevant to oer activities were selected. this involves the origin of the used content, differentiated by own materials as a characteristic for ‘create’ and external open materials as a characteristic for ‘reuse’. editing of different types of materials and the use of remix in oer serve as a reference point for the activity ‘edit’. ‘publish’ refers to public provision on the one hand, and to the types of materials shared on the other (see table 3). based on these enabled codes, a document map was created (see figure 2), with interviewees clustered according to the raediker and kuckartz (2019) similarity measure, which weights the presence of codes double and the absence of codes single. after plotting the participants in a distance matrix, four groups matched in terms of oer activities. whereupon four types of use emerged in the practical handling and use of oer (see  table  4). the identification of the types of oer users here primarily serves as a simple illustration of typical patterns of use (kuckartz 2018), so that clues can be drawn to promote the application of oer in higher education. since these are exclusively participants with oer experience, the user types do not include users who do not work with oer. instead, all user types create oer so that this activity is fulfilled for all of them. the user types differ in the extent and frequency of use of the other activities. this could be determined on the basis of the individual coded elements (see table 3). type a (see table 4) is the largest group, with ten interviewees, who based their developed materials mainly on their own materials. the creation of the materials was figure 2. document map from maxqda showing the similarities of the participants. http://dx.doi.org/10.25304/rlt.v31.2916 n. schroeder and s. donat 10 citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 (page number not for citation purpose) either done openly from the start or was later adapted or transformed to allow public sharing. legal difficulties were either avoided in advance by making little reuse of external resources or by refraining from using materials without open licenses. one characteristic of this type is that the publication of materials as oer was predominantly once-only and sometimes supported as part of projects. type b develops its content both through own materials and through the reuse and integration of external oer, although this is predominantly done without special editing. the finished resources are also mostly published. three participants could be assigned to this type. type c users create materials with open licenses and, in addition to their own materials, also reuse and partially edit a high number of external oer, but do not make them publicly available. two interviewees practice these oer activities. type d includes all four oer activities and is practiced by three oer users. in addition to the creation, reuse and publication, external materials are also frequently edited and redesigned. discussion the results of the interview study supplement the current discussion about the use of oer among higher education teachers with oer experience. usage studies from current literature that include participants without an oer background have found that use and adaptation of oer among teachers are low (baas, admiraal, and van den berg 2019; schuwer and janssen 2018), and sharing does not take place publicly, so that reuse is not visible (beaven 2018). the present study shows that the participating higher education teachers prefer to build their materials on their own materials instead of reusing external oer. results on dealing with different types of materials are also comparable. it was found that presentations and assignments are predominantly self-created and videos and images are often reused without modifications. beyond that, however, adaptations are made for use in individual contexts. editing in the sense of redesigning and remixing is rarely investigated in the literature. this study has determined that this occurs only among a few instructors who have fully aligned their teaching with open materials and practices and thus use all oer activities. teachers who created and published oer as part of one-time projects were also observed to share materials with colleagues in regular teaching contexts. teachers also reported that students elaborated content independently using external oer and created their own oer. thus, with the help of oer, open teaching methods were supported in terms of oep. compared to admiraal (2022), the identified user types and their oer activities were examined in more detail due to the qualitative approach. for this, a high table 4. user types based on oer activities. user type number participants typical oer activity a 10 01, 03, 05, 07, 10, 11, 12, 13, 15, 17 create, publish b 3 08, 09, 18 create, reuse, publish c 2 04, 14 create, reuse, edit d 3 02, 06, 16 create, reuse, edit, publish http://dx.doi.org/10.25304/rlt.v31.2916 research in learning technology citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 11 (page number not for citation purpose) number of participants could not be considered to support the distribution of the types whereas admiraal (2022) built on a large data basis. in addition, differences in the underlying activities are present in both approaches. in contrast to the types according to admiraal (2022), commenting on oer was not considered in the type formation of this study, so type 2 and type 4 are not comparable. similarly, type 5, which exclusively consumes oer, does not find a counterpart in the presented user types, since the creation of oer was defined as a prerequisite for participation. type 1, the largest group with 50%, records the highest values for reuse and adaptation. this is not reflected in the presented oer user types, especially since here, instead of minor individual adaptations, edits in the form of redesign and remixing were included in the evaluation. type 3 is to be compared with type d, as all oer activities are represented by a similar number of people (type 1: 12%, type d: 16.67%). the distribution among the identified user types of this study is primarily due to the selection of interviewees. to investigate open practices of teachers, the prerequisite was the involvement in the creation of an open resource. while recruiting participants, a first clue was to identify creators of published oer which often had been created and promoted as part of projects. this is also reflected in the practices of type a (55.55%). this user type, on the one hand, mainly uses its own materials served as a basis in the creation and, on the other hand, often uses non-open materials in their regular teaching. in contrast, there was a difficulty in identifying participants who reuse external oer in their teaching but do not make their own materials publicly available. therefore, it can be assumed that type c in fact applies to a larger number of instructors who use oer with external openly licensed materials, but do not make them public. type b and type d, which differ in whether external oer are processed, can be classified as experienced and active oer users. the low percentage in type d, which practices all oer activities, shows that editing of external oer is limited. overall, the identified oer user types contribute to an overview of different oer activities that should be considered differently in technical infrastructures and support offerings. in each case, the distribution of the user types must be determined individually in institutions to develop coordinated offers. the active involvement of different types can ensure the creation of sustainable and demand-oriented offerings. type a, for example, requires the availability of external materials with open licenses and professional quality. since teachers prefer to reuse materials with a smaller scope and integrate individual elements into their content, it should be possible to divide extensive materials into individual thematic units or formats. the reuse of materials can be further increased by building communities with opportunities for exchange and feedback options. type b can also benefit from this function to find cooperation partners for the processing of materials. type c needs support in publishing resources so that advice on legal issues, licensing, and publishing venues are beneficial for them. since type a usually only publishes individual resources on an irregular basis, this support can also be valuable for this user group to achieve a continuation of oer in teaching. in addition, guidance on open file formats and tools could help increase the editing of external materials. type d corresponds to the ideal idea of oer users. therefore, the aim is to increase this proportion among teachers by promoting their visibility and active participation in oer communities so that they can serve as an orientation for other teachers. for example, type d teachers could act in a multiplier function to promote the culture of ‘give and take’ among type a, b and c individuals. http://dx.doi.org/10.25304/rlt.v31.2916 n. schroeder and s. donat 12 citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 (page number not for citation purpose) conclusion focus of this study was on the usage behaviour and application of open practices by higher education teachers who have already had experiences with oer in their teaching. the identified expectations, concerns and requirements of the teaching staff, must be taken into account by the responsible actors when designing support measures. the derived user types therefore serve as an orientation for supporting oer in higher education. initial options have been identified that need to be individually expanded and shaped. it should be noted that due to the selection of interviewees, only a limited impression of open practices and usage behaviours could be considered. apart from the interviewed oer users, there are of course other users with further expectations and requirements. furthermore, the limitation to german-speaking higher education teachers must be taken into account. copyright regulations in german-speaking countries may have an influence on these specific results, for example with regard to the use of external materials. nevertheless, the presented interview study as well as the main results can be used as a starting point for research in other countries or educational areas. therefore, the results should be taken into consideration when designing infrastructures and fostering support structures for oer. it was found that some participants, in addition to oer activities in the context of projects, do not use oer practices in their regular teaching. nevertheless, the findings cannot be projected on teachers who do not use open practices at all. for further research, it is therefore necessary to also interview higher education teachers with no oer experience, to evaluate their reasons for barriers, for example, possible fears. as an addition to the user types presented here, this could create the possibility of creating a more general and in-depth concept for the implementation of a sustainable oer strategy. overall, the overarching goal of increasing and promoting the use of oer in higher education as well as making it more widespread is still to be pursued to achieve a cultural change. this can be seen as a prerequisite for developing subject-specific and interdisciplinary communities that exchange information on materials. thus, teachers can benefit from the quality improvement of their materials and possible applications for their teaching. through the exchange of didactic concepts, more extensive open teaching practices can be promoted beyond individual materials. in addition, learners can be more involved in the design and development of materials and be motivated to create, share and use them independently. the appropriate design and features of technical infrastructures can also support this and should be taken into account by developers of repositories and platforms. references admiraal, w. (2022) ‘a typology of educators using open educational resources for teaching’, international journal on studies in education, vol. 4, no. 1, pp. 1–23. doi: 10.46328/ijonse.60 baas, m., admiraal, w. & van den berg, e. (2019) ‘teachers’ adoption of open educational resources in higher education’, journal of interactive media in education, vol. 2019, no. 1, art. 9. doi: 10.5334/jime.510 baas, m. & schuwer, r. (2020) ‘what about reuse? a study on the use of open educational resources in dutch higher education’, open praxis, vol. 12, no. 4, pp. 527–540. doi: 10.5944/ openpraxis.12.4.1139 beaven, t. (2018) ‘“dark reuse”: an empirical study of teachers’ oer engagement’, open praxis, vol. 10, no. 4, pp. 377–391. doi: 10.5944/openpraxis.10.4.889 http://dx.doi.org/10.25304/rlt.v31.2916 https://dx.doi.org/10.46328/ijonse.60 https://dx.doi.org/10.5334/jime.510 https://dx.doi.org/10.5944/openpraxis.12.4.1139 https://dx.doi.org/10.5944/openpraxis.12.4.1139 https://dx.doi.org/10.5944/openpraxis.10.4.889 research in learning technology citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 13 (page number not for citation purpose) cardoso, p., morgado, l. & teixeira, a. (2019) ‘open practices in public higher education in portugal: faculty perspectives’, open praxis, vol. 11, no. 1, pp. 55–70. doi: 10.5944/ openpraxis.11.1.823 corbin, j. & strauss, a. (2015) basics of qualitative research. techniques and procedures for developing grounded theory, 4th edn, sage, thousand oaks, ca. cronin, c. (2017) ‘openness and praxis: exploring the use of open educational practices in higher education’, the international review of research in open and distributed learning, vol. 18, no. 5, pp. 1–21. doi: 10.19173/irrodl.v18i5.3096 ehlers, u.-d. (2011) ‘from open educational resources to open educational practices’, elearning papers, no. 23, [online] available at: https://www.oerknowledgecloud.org/ archive/media25161.pdf fulantelli, g. et al., (2008) ‘the open learning object model to promote open educational resources’, journal of interactive media in education, vol. 2019, no. 1, art. 9. doi: 10.5334/2008-9 galletta, a. (2013) mastering the semi-structured interview and beyond: from research design to analysis and publication. new york university press, new york. doi: 10.18574/ nyu/9780814732939.001.0001 gurell, s., & wiley, d. (2008) oer handbook for educators, [online] available at: http://wikieducator.org/oer_handbook/educator_version_one heck, t., kullmann, s., hiebl, j., schroeder, n., otto, d. & sander, p. (2020)‚ designing open informational ecosystems on the concept of open educational resources’, open education studies, vol. 2, no. 1, pp. 252–264. doi: 10.1515/edu-2020-0130 kelle, u. (2006) ‘combining qualitative and quantitative methods in research practice: purposes and advantages’, qualitative research in psychology, vol. 3, pp. 293–311. kennedy, b. l. & thornberg, r. (2018) ‘deduction, induction, and abduction’, in the sage handbook of qualitative data collection, ed u. flick, sage, london, pp. 49–64. doi: 10.4135/9781526416070.n4 kuckartz, u. (2018) qualitative inhaltsanalyse, methoden, praxis, computerunterstützung, 5th edn, beltz juventa, weinheim. open university (2015) ‘oerrh survey data 2013–2014’, figshare. doi: 10.6084/ m9.figshare.1317313.v1 pawlowski, j. m. (2012) ‘emotional ownership as the key to oer adoption: from sharing products and resources to sharing ideas and commitment across borders’, efquel innovation forum 2012, [online] available at: https://pressbooks.bccampus.ca/facultyoertoolkit/ wp-content/uploads/sites/114/2017/02/oer_emotional_ownership_pawlowski20120529citation.pdf raediker, s. & kuckartz, u. (2019) analyse qualitativer daten mit maxqda, springer, wiesbaden. doi: 10.1007/978-3-658-22095-2 rodés, v., gewerc-barujel, a., & llamas-nistal, m. (2019). `university teachers and open educational resources: case studies from latin america’, the international review of research in open and distributed learning, vol. 20, no. 1, pp. 165–183. doi: 10.19173/ irrodl.v20i1.3853 roulston, k. & choi, m. (2018) ‘qualitative interviews’, in the sage handbook of qualitative data collection, ed u. flick, sage, london pp. 233–249. doi: 10.4135/9781526416070.n15 schuwer, r. & janssen, b. (2018) ‘adoption of sharing and reuse of open resources by educators in higher education institutions in the netherlands: a qualitative research of practices, motives, and conditions’, the international review of research in open and distributed learning, vol. 19, no. 3, pp. 151–170. doi: 10.19173/irrodl.v19i3.3390 wiley, d. (2009, june 10) ‘dark matter, dark reuse, and the irrational zeal of a believer’, iterating toward openness, [online] available at: http://opencontent.org/blog/archives/905 wiley, d. (2014, march 5) ‘the access compromise and the 5th r’, in iterating toward openness, [online] available at: http://opencontent.org/blog/archives/3221 http://dx.doi.org/10.25304/rlt.v31.2916 https://dx.doi.org/10.5944/openpraxis.11.1.823 https://dx.doi.org/10.5944/openpraxis.11.1.823 https://dx.doi.org/10.19173/irrodl.v18i5.3096 https://www.oerknowledgecloud.org/archive/media25161.pdf https://www.oerknowledgecloud.org/archive/media25161.pdf https://dx.doi.org/10.5334/2008-9 https://dx.doi.org/10.18574/nyu/9780814732939.001.0001 https://dx.doi.org/10.18574/nyu/9780814732939.001.0001 http://wikieducator.org/oer_handbook/educator_version_one http://wikieducator.org/oer_handbook/educator_version_one https://dx.doi.org/10.1515/edu-2020-0130 https://dx.doi.org/10.4135/9781526416070.n4 https://dx.doi.org/10.6084/m9.figshare.1317313.v1 https://dx.doi.org/10.6084/m9.figshare.1317313.v1 https://pressbooks.bccampus.ca/facultyoertoolkit/wp-content/uploads/sites/114/2017/02/oer_emotional_ownership_pawlowski20120529citation.pdf https://pressbooks.bccampus.ca/facultyoertoolkit/wp-content/uploads/sites/114/2017/02/oer_emotional_ownership_pawlowski20120529citation.pdf https://pressbooks.bccampus.ca/facultyoertoolkit/wp-content/uploads/sites/114/2017/02/oer_emotional_ownership_pawlowski20120529citation.pdf https://dx.doi.org/10.1007/978-3-658-22095-2 https://dx.doi.org/10.19173/irrodl.v20i1.3853 https://dx.doi.org/10.19173/irrodl.v20i1.3853 https://dx.doi.org/10.4135/9781526416070.n15 https://dx.doi.org/10.19173/irrodl.v19i3.3390 http://opencontent.org/blog/archives/905 http://opencontent.org/blog/archives/3221 n. schroeder and s. donat 14 citation: research in learning technology 2023, 31: 2916 http://dx.doi.org/10.25304/rlt.v31.2916 (page number not for citation purpose) wiley, d., bliss, t. j. & mcewen, m. (2014) ‘open educational resources: a review of the literature’, in handbook of research on educational communications and technology, eds j. m. spector et al., springer, new york, pp. 781–789. doi: 10.1007/978-1-4614-3185-5_63 wiley, d., & hilton iii, j. l. (2018) ‘defining oer-enabled pedagogy’, the international review of research in open and distributed learning, vol. 19, no. 4, pp. 133–147. doi: 10.19173/irrodl.v19i4.3601 zawacki-richter, o. et al., (2020) ‘elements of open education: an invitation to future research’, the international review of research in open and distributed learning, vol. 21, no. 3, pp. 319–334. doi: 10.19173/irrodl.v21i3.4659 http://dx.doi.org/10.25304/rlt.v31.2916 https://dx.doi.org/10.1007/978-1-4614-3185-5_63 https://dx.doi.org/10.19173/irrodl.v19i4.3601 https://dx.doi.org/10.19173/irrodl.v21i3.4659 (page number not for citation purpose) 1 *corresponding author. email: nai.li@imperial.ac.uk research in learning technology 2020. © 2020 nai li and david lefevre. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 research in learning technology vol. 28, 2020 original research article holographic teaching presence: participant experiences of interactive synchronous seminars delivered via holographic videoconferencing nai li* and david lefevre edtech lab, imperial college business school, imperial college london, london, uk (received: 2 may 2019; revised: 11 february 2020; accepted: 17 february 2020; published: 4 may 2020) this study seeks to identify potential advantages of using holographic videoconferencing to deliver seminars within higher education as compared to the use of alternative non-holographic videoconferencing. holographic videoconferencing offers opportunities to enhance attendees’ experience of remotely delivered seminars but has not been widely researched. data were collected from 127 attendees attending one of three seminars, each of which featured a combination of physically present presenters and remote presenters participating via holographic videoconferencing. in this study, the holographic representations were three-dimensional and life-size. monitors and holographic images were calibrated in a manner such that the remote presenters were able to point to and achieve eye-contact with members of the audience. results indicate that the use of holographic videoconferencing can enhance the teaching presence of remote presenters, the engagement between participants and attendees’ enjoyment of a seminar. almost all participants reported this to be their first experience of a holographic event and the positive results are partly explained by a sense of novelty. this suggests that the benefits of holographic videoconferencing may reduce over time. however, we argue that some benefit, resulting from an enhanced degree of teaching presence, will be sustained. the relative impact on learning gain is not explored in the current study. we believe that this would likely require a more controlled experiment in future research. keywords: holographic teaching presence; telepresence; online learning environments; holography in higher education; community of inquiry framework introduction digital technology continues to evolve at a rapid rate and present opportunities to enhance the delivery of education. one of the benefits of digital technology is that it is able to remove geographical barriers to learning. teachers and students can interact via the virtual learning environment of an educational institution regardless of their proximity to its physical location. one of the technologies that facilitates such interaction is videoconferencing that enables parties in different physical locations to communicate simultaneously through the transmission of video and audio signals. mailto:nai.li@imperial.ac.uk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v28.2265 nai li and david lefevre 2 citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 (page number not for citation purpose) non-holographic videoconferencing, in which the video image is two dimensional, enables instructors and students to participate in a live class from different physical locations and has become commonplace within higher education. for example, this technology enables instructors to invite ‘real-world’ practitioners from industry into the classroom and has enabled institutions to simultaneously host classes across multiple physical and virtual locations. the present study reports on the use of holographic videoconferencing (hvc) technology. hvc differs from non-holographic videoconferencing in that the instructor appears as a three-dimensional hologram rather than a two-dimensional video image. literature review the use of holography in education the term ‘hologram’ is a combination of two greek terms holos, meaning the whole view, and gram, meaning to be written down (ghuloum 2010). holography, a technique for creating and displaying holograms, was invented by the physicist dennis gabor in 1947 (elmorshidy 2010). however, the holographic technique was only practically utilised from 1962 onwards when scientists from the united states and the soviet union created a 3d holographic technology. since the 1980s, the technology enabling the production of 3d holograms has advanced notably, largely due to the availability of low-cost solid-state lasers developed for common consumer goods such as dvd (ghuloum 2010; sudeep 2009). educators have started to adopt holographic technology for use in teaching and learning; however, this practice is still at an exploratory stage and most frequently occurs within the field of medical education, specifically in the teaching of anatomy (freeman 2010; hackett 2013; jamali et al. 2015). hackett (2013) performed a study with medical students aiming to identify possible benefits for the use of holograms in cardiac anatomy training and found that the use of holograms increased students’ learning outcomes. jamali et al. (2015) conducted a study investigating the use of mobile-augmented reality to render 3d holograms for use in studying the human skeletal structure. the results from this pilot test with 30 undergraduate students as subjects indicated that students were satisfied with the 3d holographic technology as a learning tool, with its usability and features and that this may have a positive impact on students’ learning process. freeman (2010) demonstrated that it was possible to use hologram technology in a pharmacology class to demonstrate the manner in which pharmaceuticals moved through the internal organs of the human body. orcos and magreñán (2018) focused on the use of recorded holograms to represent objects within a class on cellular biology. they found that the use of a hologram to portray cellular division enhanced students’ learning of the process, that the students had a greater sense of the recorded object being ‘present’ and that they were more predisposed towards learning as a result. research to date has focused on the use of pre-recorded holograms in an experimental lab or in other science learning contexts (barkhaya and halim 2016). the use of hvc, that is, the use of holograms as a synchronous communication tool, within education has received less attention. luevano, delara and castro (2015) report on the development of a number of technologies, including an hvc system, aiming to improve the students’ sensation of telepresence of a remote instructor. http://dx.doi.org/10.25304/rlt.v28.2265 research in learning technology citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 3 (page number not for citation purpose) in  an experiment involving 43 students attending classes on the topic ‘accounting and costs’, 93% of the students stated they ‘felt’ the presence of the instructor when hvc was used. notions of ‘presence’ one aim of the present research is to investigate whether use of hvc engenders a greater sense of the ‘presence’ of the presenter as compared to the use of non holographic alternative tools. various conceptualisations of presence have been defined. witmer, jerome and singer (2005) define presence to be ‘the subjective experience of being in one place or environment, even when one is physically situated in another’. this notion has also been termed ‘telepresence’ which minsky (1980) used to describe two different phenomenon: (1) a technology that enables the projection of a person to or from a remote environment; and (2) a personal experience of either being present in the remote environment or being present in a mediated environment (bailenson and beall 2006; minsky 1980). lombard and ditton (1997) describe telepresence as ‘the perceptual illusion of non-mediation’, and is determined by the characteristics of the media and of the user. in practice, the terms ‘presence’ and ‘telepresence’ often refer to the same concept. sheridan (1992a, 1992b, 1996) built on minsky’s original definition of telepresence and used the shorter term ‘presence’ to refer to the effect felt when controlling realworld objects remotely and also the effect people feel when they interact with, and immerse themselves in, virtual reality or virtual environments. a more specific term ‘spatial presence’ can be defined as ‘the subjective experience of being in one place or an environment, even when one is physically situated in another’ (witmer, jerome and singer 2005; witmer and singer 1998). the relationship between a sense of special presence and learning within a virtual environment was explored by tichon (2007) who found a relationship between participants’ sense of spatial presence within a virtual environment and the effectiveness of a programme aiming to train drivers in the operation of complex heavy machinery. ‘teaching presence’ and the community of inquiry (coi) framework in the context of educational research, the notion of presence has been widely adopted and used in frameworks for online or blended learning (baker 2010; dunlap, verma, and johnson 2016; garrison, anderson, and archer 2000; kozan and caskurlu 2018; nagel and kotzé 2010; shea, li, and pickett 2006). a prominent example can be seen in garrison, anderson and archer’s (2000) application of the community of inquiry (coi) (figure 1) framework to online learning, originally within the context of textbased communication. this has been widely adopted as a theoretical model to explain and explore concepts in both online and blended learning experience (chandler et al. 2013; chen, zydney, and patton 2017; díaz et al. 2010; evans, ward, and reeves 2017; nagel and kotzé 2010, katz and halpern 2015). in this context, ‘community’ refers to ‘a community of learners’ which is claimed to be ‘at the heart of a meaningful educational experience’ in higher education (garrison and arbaugh 2007). garrison, anderson and archer identified three distinct notions of presence: social presence, cognitive presence and teaching presence. http://dx.doi.org/10.25304/rlt.v28.2265 nai li and david lefevre 4 citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 (page number not for citation purpose) ‘teaching presence’ as referred to by garrison, anderson, and archer (2000) refers to two distinct activities in online education. first is the design of the educational experience. second is the facilitation of the learning experience. in subsequent work, usage of the term has evolved to incorporate a broader range of teaching roles (daspit and d’souza 2012; scagnoli, choo, and tian 2019; scagnoli, mckinney, and moorereynen 2015). the critical role played by teaching presence within students’ learning experiences and level of satisfaction is supported in a number of studies (chen and wu 2015; evans and cordova 2015; garrison 2017; holland 2014; scagnoli, choo, and tian 2019; scagnoli, mckinney, and moore-reynen 2015; schrader and bastiaens 2012). as garrison (2017) argues ‘there should be no doubt of the essential role teaching presence plays in integrating the various elements of an educational experience’ (2017, p. 71). in such research, the notion of ‘teaching presence’ focuses on the actions taken by the teacher rather than the student perception of the instructor being physically present. however, this latter notion is explored by scagnoli, choo, and tian (2019) who investigated the impact of placing video recording of an instructor within a course on students’ sense of that instructor being ‘present’ in the course. this and related research found that students believe that their interaction with videos increases their motivation and engagement due to perceived personal connection with the instructor (evans and cordova 2015; giannakos, jaccheri, and krogstie 2016; hughes 2009; scagnoli, choo, and tian 2019; tichon 2007). in the present research, the term ‘teaching presence’ refers to attendees’ sense of a presenter being physically present in a seminar. research questions this study sought to identify any potential advantages in the use of hvc compared to non-holographic videoconferencing. specifically, the research questions were the following: figure 1. the community of inquiry framework. note: garrison, anderson, and archer 2000. http://dx.doi.org/10.25304/rlt.v28.2265 research in learning technology citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 5 (page number not for citation purpose) compared to non-holographic videoconferencing, does the use of hvc enhance: presenters’ degree of teaching presence? attendees’ engagement with a seminar? attendees’ enjoyment of a seminar? do presenters consider hvc to be a valuable and practical technology? research methods and design the seminars in the present study, data were collected from attendees of one of three live seminars, each featuring a presenter participating via hvc. each seminar featured an audience physically located in london and was facilitated using the hvc technology described above. each seminar also included a host who was physically in the same place as the audience. the first seminar on the topic ‘women in technology’, comprised three parts: a synchronous presentation from a presenter physically located in los angeles, a pre-recorded holographic presentation and a live panel debate comprising a combination of two physically present panellists and two panellists participating via hvc from new york. the second seminar comprised a student-led seminar on the topic of technological disruption and featured a keynote speaker participating from a different location in london via the hvc technology. the third seminar titled ‘international perspectives on sustainable finance’ contained two presenters, physically located in canada, participating via the hvc technology. the holographic technology the hvc technology enabled presenters to appear as 3d, life-size entities and to interact with the audience in real time. monitors and holographic images were calibrated so that presenters were able to point to and achieve eye-contact with members of the audience. as such, they were able to respond to audience reactions and take questions in a manner similar to being physically present in the lecture theatre. the equipment comprised a ‘capture studio’ and a ‘display studio’. the capture studio (figure 2) comprises a video production studio containing a high-definition camera, lighting, audio equipment and set dressing. in addition, the set contains replicas of any furniture that appears in the display studio, a black curtain as a backdrop and one or more reference monitors for the presenter showing a live feed of the audience. the video footage captured in the ‘capture studio’ is transmitted to the ‘display studio’ via an internet connection using proprietary software supplied by a commercial vendor. a projector projects this footage onto a transparent mesh screen. a black curtain backdrop together with lighting effects create the illusion of a three-dimensional holographic image (figure 3). data collection and analysis data collection comprised a questionnaire delivered to attendees and semi-structured interviews with presenters. the same data-collection instruments were used for each of the three seminars. the questionnaire, comprising closed and open questions, was http://dx.doi.org/10.25304/rlt.v28.2265 nai li and david lefevre 6 citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 (page number not for citation purpose) sent to all attendees following their attendance at a seminar and focused on research question 1. questions asked attendees to compare their experience of the hvc conferencing technology with non-holographic videoconferencing. as shown in table 1, of the 221 attendees across the three seminars, 127 (57%) completed and returned the questionnaire. a total of 54% of these respondents were female and 46% male. figure 3. an example of a display studio. figure 2. an example of a capture studio. http://dx.doi.org/10.25304/rlt.v28.2265 research in learning technology citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 7 (page number not for citation purpose) semi-structured interviews were conducted involving four of the six presenters, who participated via the hvc technology, and focused on research question 2. quantitative data analysis comprised univariate descriptive statistics. the qualitative data were explored using thematic content analysis. results 1 a) did the use of holographic videoconferencing enhance presenters’ degree of  teaching presence? as shown in table 2, attendees strongly agreed with the premise that the use of hvc enhanced their sense of the remote presenter being present. a total of 84% of respondents agreed with this premise, 2% disagreed. the following indicative responses to the open questions within the questionnaire suggest the reasoning behind this result: respondents frequently referred to a sense of the presenter being physically present in the seminar ‘the wow factor derives from the presence of the speaker, who looks almost on stage’. (attendee s) ‘the speaker beamed in from new york who looks “being there” physically on stage, the experience was truly amazing, holograms were like out of star wars!’ (attendee q) ‘it’s kind of surreal, i feel the presence of the speaker as she is in the same lecture theatre here when she listens to, looks at and talks to me, but she is in canada!’. (attendee a) respondents also stated the event to have a greater sense of ‘realism’ compared to experiences of non-holographic videoconferencing. ‘the event was good and very interesting with the hologram, which offer realistic face-to-face presence breakthrough that could not be provided by a skype call… because the images on a 2d flat screen with skype calls is less realistic’. (attendee h) 1 b) did the use of holographic videoconferencing enhance attendees’ engagement with a seminar? as shown in table 3, the respondents in this study strongly agreed that the use of hvc enhanced their sense of engagement in the seminar. the majority of respondents (78%) agreed that the use of hvc is more likely to facilitate attendee engagement and 73% of respondents stated a belief that holograms enhance the effectiveness of interaction between attendees and remote  presenters. respondents commented that the enhanced sense of the presenter being physically present resulted in more sustained attention during the seminar. table 1. participants and response rate. seminar 1 seminar 2 seminar 3 attendees 100 46 73 attendees responding 68 (68%) 30 (65%) 29 (40%) http://dx.doi.org/10.25304/rlt.v28.2265 nai li and david lefevre 8 citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 (page number not for citation purpose) ‘i find it’s very easy to get distracted with webinar or video-lectures but with hologram lecturers, they are in front of you as in the same room, it’s very different experience’. (attendee j) ‘the experience is interesting, i’d pay more attention when the instructor appears as holograms than if he is only present in video-lectures, the realistic presence-inperson is more powerful i guess…’. (attendee t) a total of 22% of respondents stated a neutral view on whether holograms enhanced attendees’ engagement. however, the majority stated this was due to technical issues experienced during the set-up of the seminar. ‘there were some sound issues at one point of the lecture with some of speakers that took away the feel of the holograms, i’d be more engaged with speakers if the sound is better from back’. (attendee b) ‘the event was good and very interesting with the holograms, however (perhaps because of the backlight) the 3d feeling was not perfectly captured. i did not feel like it was a lot different from live streaming’. (attendee m) ‘when two hologram speakers presented, at the one point, one of the holograms kept looking in the wrong direction – the opposite to the speaker, although she realised later on’. (attendee c) 1 c) did the use of holographic videoconferencing enhance attendees’ enjoyment of the seminar? in this study, attendees strongly supported the statement that the use of hvc enhanced their enjoyment of the seminars (table 4). table 3. engagement. minimum maximum mean std. deviation variance holograms enhance effectiveness of interaction between students and speakers (both locally and remotely). 7 10 9.16 0.855 0.731 holograms offer much more efficient communication among participants. 7 10 8.89 0.739 0.546 holograms enable speakers to address participants in different locations simultaneously. 7 10 8.66 0.872 0.760 note: perceived comparison with non-holographic videoconferencing, 1 = strongly disagree, 10 = strongly agree. table 2. teaching presence. minimum maximum mean std. deviation variance holograms offer more realistic in-person presence of speakers who are based remotely 7 10 9.22 0.750 0.562 note: perceived comparison with non-holographic videoconferencing, 1 = strongly disagree, 10 = strongly agree. http://dx.doi.org/10.25304/rlt.v28.2265 research in learning technology citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 9 (page number not for citation purpose) of relevance here is the fact that only 12 of the 127 respondents reported that they had previously attended an hvc-facilitated event. the majority of participants were therefore experiencing hvc for the first time and comments suggested that a sense of novelty was a factor promoting the reported sense of enjoyment. ‘the lecture was very well put together. it was enjoyable and a delight to attend for the first time’. (attendee a) ‘the whole experience was truly magical! i felt extremely proud as an msc ihm student to have the opportunity to be part of this pioneering lecture’. student j) 2. do presenters consider holographic videoconferencing to be a valuable and practical technology? four of the six presenters who participated in the seminars via hvc were interviewed. all stated their experience to be positive, supporting the premise that hvc can be a practical technology to facilitate and enhance the delivery of seminars. the primary advantage offered by the presenters was that the hvc technology provided an experience closer to that of a face-to-face presentation in that they were able to interact with the audience in a more realistic manner. ‘i was part of an amazing discussion with impressive women that allowed for twoway audience interaction from thousands of miles away. it was a bit surreal to see myself projected on a stage between three live humans, but i felt much more engaged and part of the discussion than if it were a video chat. the conversation flowed freely, and i think the audience at times forget we were being beamed in from new york’. (presenter physically present in new york) discussion the respondents in this study were not ambiguous in their enthusiasm for the hvc format adopted. the results suggest that the use of hvc, when compared to alternative non-holographic conferencing formats, may enhance the teaching presence of remote presenters, enhance engagement between participants and increase attendees’ enjoyment of a seminar. attendees suggested that these concepts are linked in that it was the more realistic in-person presenter presence afforded by the hvc that fostered engagement which made the seminar more engaging and enjoyable. of particular interest is the fact that attendee responses also suggest that use of the hvc technology may help to sustain attendee attention to a presenter. table 4. enjoyment. minimum maximum mean std. deviation variance holograms make the lecture much more fun. 7 10 9.22 0.910 0.829 holograms make the lecture much more enjoyable. 7 10 9.11 0.895 0.800 note: perceived comparison with non-holographic videoconferencing, 1 = strongly disagree, 10 = strongly agree. http://dx.doi.org/10.25304/rlt.v28.2265 nai li and david lefevre 10 citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 (page number not for citation purpose) almost all attendees reported that this was the first time they ever attended a holographic event and a sense of novelty was present. this suggests that the benefits of hvc may reduce over time. however, we argue that some of the benefits relating to the enhanced degree of teaching presence will be sustained. in particular, the use of hvc increased attendees’ sense of ‘teaching presence’, a component of the coi framework shown to aid motivation and engagement (chen, zydney, and patton 2017; dunlap, verma, and johnson 2016; garrison, anderson, and archer 2000; garrison and arbaugh 2007; kozan and caskurlu 2018; nagel and kotzé 2010). this research also found that the effectiveness of the hvc depends on appropriate technical set-up, participants reported that better sound and lighting would have significantly improved their experience. in addition, it is necessary for presenters to be appropriately trained in the nature of delivering presentations through hvc due to the novel nature of presenting involved. conclusion the adoption of hvc within higher education is at an early stage; however, the  results of this study offer encouragement that this technology may offer value  with regard to the student learning experience. this study does not make claims regarding any positive effects on learning gain or knowledge retention; however, the impacts on attendees’ sense of teaching presence, engagement with the seminars and level of enjoyment in themselves seems valuable and worthy of further exploration. limitations of the study this study does have limitations. there is potential bias arising from the selection of the participants in that the attendees for each of the three seminars were self-selected. the topics of seminars 1 and 2 both had a technology focus and, in addition, the topic of seminar 1 had a gender focus. therefore, female students were slightly overrepresented and technology-inclined individuals were also likely to be overrepresented. in addition, this relies heavily on attendees self-reporting on their experience following an event. studies adopting more observational methods would provide more robust support for the results. a further limitation is that a range of hvc and non-holographic technologies exists, with varying functionality. this study investigated the use of one hvc with particular characteristics. further studies comparing alternative technologies would be valuable. for example, this study did not directly compare the hvc technology to non-holographic videoconferencing systems that render the participants in life size. such a study would provide further insight on effects directly attributable to the holographic representation. acknowledgements the authors acknowledge the work of the edtech lab team behind the hologram project, especially andrew parry and phil tulip for their technical support. http://dx.doi.org/10.25304/rlt.v28.2265 research in learning technology citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 11 (page number not for citation purpose) references bailenson, j. & beall, a. (2006) ‘transformed social interaction: exploring the digital plasticity of avatars’, in avatars at work and play: collaboration and interaction in shared virtual environments, eds r. schroeder & a. s. axelsson, springer, london. baker, c. (2010) ‘the impact of instructor immediacy and presence for online student affective learning, cognition, and motivation’, journal of educators online, vol. 7, pp. 1–30. doi: 10.9743/jeo.2010.1.2 barkhaya, n. m. m. & halim, n. d. a. (2016) ‘a review of application of 3d hologram in education: a metaanalysis’, 2016 ieee 8th international conference on engineering education (iceed), december 2016, kuala lumpur, pp. 257–260. chandler, t., et al., (2013) ‘the incorporation of hands-on tasks in an online course: an analysis of a blended learning environment’, interactive learning environments, vol. 21, pp. 456–468. doi: 10.1080/10494820.2011.593524 chen, b., zydney, j. & patton, k. (2017) ‘creating a community of inquiry in large-enrollment online courses: an exploratory study on the effect of protocols within online discussion’, online learning, vol. 21, pp. 165–188. doi: 10.24059/olj.v21i1.816 chen, c. m. & wu, c. h. (2015) ‘effects of different video lecture types on sustained attention, emotion, cognitive load, and learning performance’. computers & education, vol. 80, pp. 108–121. doi: 10.1016/j.compedu.2014.08.015 daspit, j. j. & d’souza, d. e. (2012) ‘using the community of inquiry framework to introduce wiki environments in blended-learning pedagogies: evidence from a business capstone course’, academy of management learning & education, vol. 11, pp. 666–683. doi: 10.5465/ amle.2010.0154 díaz, s. r., et al., (2010) ‘student ratings of the importance of survey items, multiplicative factor analysis, and the validity of the community of inquiry survey’. internet and higher education, vol. 13, pp. 22–30. doi: 10.1016/j.iheduc.2009.11.004 dunlap, j. c., verma, g. & johnson, h. l. (2016) ‘presence+experience: a framework for the purposeful design of presence in online courses’, techtrends, vol. 60, pp. 145–151. doi: 10.1007/s11528-016-0029-4 elmorshidy, a. (2010) ‘holographic projection technology: the world is changing’, journal of telecommunications, vol. 2, pp. 104–112. evans, h. k. & cordova, v. (2015) ‘lecture videos in online courses: a follow-up’. journal of political science education and information technologies, vol. 11, pp. 472–482. doi: 10.1080/15512169.2015.1069198 evans, s. m., ward, c. & reeves, s. (2017) ‘an exploration of teaching presence in online interprofessional education facilitation’. medical teacher, vol. 39, pp. 773–779. doi: 10.1080/0142159x.2017.1297531 freeman, d. m. (2010) ‘holographic mri and ct scans of the human body’, available at: https:// eyedave4.wordpress.com/2010/04/01/holographic-mri-and-ct-scans-of-the-human-body/ garrison, d. r., anderson, t. & archer, w. (2000) ‘critical inquiry in a text-based environment: computer conferencing in higher education’, the internet and higher education, vol. 2, pp. 87–105. doi: 10.1016/s1096-7516(00)00016-6 garrison, d. r. & arbaugh, j. b. (2007) ‘researching the community of inquiry framework: review, issues, and future directions’, internet and higher education, vol. 10, pp. 157–172. doi: 10.1016/j.iheduc.2007.04.001 garrison, d. r. (2017) e-learning in the 21st century: a community of inquiry framework for research and practice, routledge, new york, ny. giannakos, m. n., jaccheri, l. & krogstie, j. (2016) ‘exploring the relationship between video lecture usage patterns and students’ attitudes’, british journal of educational technology, vol. 47, pp. 1259–1275. doi: 10.1111/bjet.12313 ghuloum, h. (2010) ‘3d hologram technology in learning environment’, proceedings of informing science & it education conference (insite) 2010, pp. 693–704, cassino, italy. http://dx.doi.org/10.25304/rlt.v28.2265 http://dx.doi.org/10.9743/jeo.2010.1.2 http://dx.doi.org/10.1080/10494820.2011.593524 http://dx.doi.org/10.24059/olj.v21i1.816 http://dx.doi.org/10.1016/j.compedu.2014.08.015 http://dx.doi.org/10.5465/amle.2010.0154 http://dx.doi.org/10.5465/amle.2010.0154 https://dx.doi.org/10.1016/j.iheduc.2009.11.004 http://dx.doi.org/10.1007/s11528-016-0029-4 http://dx.doi.org/10.1080/15512169.2015.1069198 https://dx.doi.org/10.1080/0142159x.2017.1297531 https://eyedave4.wordpress.com/2010/04/01/holographic-mri-and-ct-scans-of-the-human-body/ https://eyedave4.wordpress.com/2010/04/01/holographic-mri-and-ct-scans-of-the-human-body/ https://dx.doi.org/10.1016/s1096-7516(00)00016-6 https://dx.doi.org/10.1016/j.iheduc.2007.04.001 https://dx.doi.org/10.1111/bjet.12313 nai li and david lefevre 12 citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 (page number not for citation purpose) hackett, m. (2013) ‘medical holography for basic anatomy training’, interservice/industry training, simulation, and education conference (i/itsec) 2013, paper no. 13207, pp. 1–10, orlando, fl. holland, j. (2014) ‘video use and the student learning experience in politics and international relations’, politics, vol. 34, pp. 263–274. doi: 10.1111/1467-9256.12022 hughes, g. d. (2009) ‘using videos to bring lecture to the online classroom’, college quarterly, vol. 12, pp. 7–12. [online] available at: https://eric.ed.gov/?id=ej864472 jamali, s. s., et al., (2015) ‘utilising mobile-augmented reality for learning human anatomy’, 7th world conference on educational sciences, (wces-2015), novotel, athens convention center, athens, greece, procedia – social and behavioral sciences, pp. 659–668. katz, j. e. & halpern, d. (2015) ‘can virtual museums motivate students? toward a constructivist learning approach’, journal science educational technology, vol. 24, pp. 776–788. [online] available at: https://www.learntechlib.org/p/175258/ kozan, k. k. & caskurlu, s. (2018) ‘on the nth presence for the community of inquiry framework’, computers & education, vol. 122, pp. 104–118. doi: 10.1016/j. compedu.2018.03.010 lombard, m. & ditton, t. b. (1997) ‘at the heart of it all: the concept of presence’, journal of  computer-mediated communication, vol. 3, pp. 72–81. doi: 10.1016/j.procs.2015. 12.256 luévano, e., delara, e. l. & castro, j. e. (2015) ‘use of telepresence and holographic projection mobile device for college degree level’, procedia computer science, vol. 75, pp. 339–347. minsky, m. (1980) ‘telepresence’, omni, vol. 2, pp. 45–51. nagel, l. & kotzé, t. g. (2010) ‘supersizing e-learning: what a coi survey reveals about teaching presence in a large online class’, internet and higher education, vol. 13, pp. 45–51. doi: 10.1016/j.iheduc.2009.12.001 orcos, l. & magreñán, á. a. (2018) ‘the hologram as a teaching medium for the acquisition of stem contents’, international journal learning technology, vol. 13, pp. 163–177. doi: 10.1504/ijlt.2018.092097 scagnoli, n. i., choo, j. & tian, j. (2019) ‘students’ insights on the use of video lectures in online classes’, british journal of educational technology, vol. 50, pp. 399–414. doi: 10.1111/ bjet.12572 scagnoli, n. i., mckinney, a. & moore-reynen, j. (2015) ‘video lectures in elearning’, in  handbook of research on innovative technology integration in higher education, eds f. nafukho & b. irby, information science reference, hershey, pa, pp. 115–134. schrader, c. & bastiaens, t. j. (2012) ‘the influence of virtual presence: effects on experienced cognitive load and learning outcomes in educational computer games’, computers in human behavior, vol. 28, pp. 648–658. doi: 10.1016/j.chb.2011.11.011 shea, p., li, c. a. & pickett, a. (2006) ‘a study of teaching presence and student sense of learning community in fully online and web-enhanced college courses’, the internet and higher education, vol. 9, pp. 175–190. doi: 10.1016/j.iheduc.2006.06.005 sheridan, t. b. (1992a). ‘musings on telepresence and virtual presence’, presence: teleoperators and virtual environments, vol. 1, pp. 120–126. doi: 10.1162/pres.1992.1.1.120 sheridan, t. b. 1992b. ‘defining our terms’, presence: teleoperators and virtual environments, vol. 2, pp. 272–274. sheridan, t. b. (1996) ‘further musings on the psychophysics of presence’, presence: teleoperators and virtual environments, vol. 5, pp. 241–246. sudeep, u. (2009) ‘use of 3d hologram technology in engineering education’, second international conference on emerging trends in engineering & technology (icetet-09), nagpur, india, osr journal of mechanical and civil engineering (iosr-jmce), pp. 62–67. tichon, j. g. (2007) ‘using presence to improve a virtual training environment’, cyberpsychology & behavior, vol. 10, pp. 781–787. doi: 10.1089/cpb.2007.0005 http://dx.doi.org/10.25304/rlt.v28.2265 https://dx.doi.org/10.1111/1467-9256.12022 https://eric.ed.gov/?id=ej864472 https://www.learntechlib.org/p/175258/ https://dx.doi.org/10.1016/j.compedu.2018.03.010 https://dx.doi.org/10.1016/j.compedu.2018.03.010 https://dx.doi.org/10.1016/j.procs.2015.12.256 https://dx.doi.org/10.1016/j.procs.2015.12.256 https://dx.doi.org/10.1016/j.iheduc.2009.12.001 https://dx.doi.org/10.1504/ijlt.2018.092097 https://dx.doi.org/10.1111/bjet.12572 https://dx.doi.org/10.1111/bjet.12572 https://dx.doi.org/10.1016/j.chb.2011.11.011 https://dx.doi.org/10.1016/j.iheduc.2006.06.005 https://dx.doi.org/10.1162/pres.1992.1.1.120 https://dx.doi.org/10.1089/cpb.2007.0005 research in learning technology citation: research in learning technology 2020, 28: 2265 http://dx.doi.org/10.25304/rlt.v28.2265 13 (page number not for citation purpose) witmer, b. g. & singer, m. j. (1998) ‘measuring presence in virtual environments: a presence questionnaire’, presence: teleoperators and virtual environments, vol. 7, pp. 225–240. doi: 10.1162/105474698565686 witmer, b. g., jerome, c. j. & singer, m. j. (2005) ‘the factor structure of the presence questionnaire’, presence: teleoperators and virtual environments, vol. 14, pp. 298–312. doi: 10.1162/105474605323384654 http://dx.doi.org/10.25304/rlt.v28.2265 https://dx.doi.org/10.1162/105474698565686 https://dx.doi.org/10.1162/105474605323384654 (page number not for citation purpose) 1 *corresponding author. email: h.walmsley-smith@staffs.ac.uk research in learning technology 2019. © 2019 h. walmsley-smith et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 research in learning technology vol. 27, 2019 original research article the e-design assessment tool: an evidence-informed approach towards a consistent terminology for quantifying online distance learning activities helen walmsley-smitha*, lynn machinb and geoff waltonc aacademic development unit, staffordshire university, stoke-on-trent, staffordshire, uk; bschool of life sciences and education, staffordshire university, stoke-on-trent, staffordshire, uk cdepartment of languages, information & communications, mmu, manchester uk (received: 10 july 2018; revised: 31 october 2018; accepted: 18 december 2018; published: 8 february 2019) online distance learning (odl) continues to expand rapidly, despite persistent concerns that student experience is poorer and retention lower than for face-toface courses. various factors affect odl quality, but the impact of recommended learning activities, such as student interaction activities and those involving feedback, have proven difficult to assess because of challenges in definition and measurement. although learning design frameworks and learning analytics have been used to evaluate learning designs, their use is hampered by this lack of an agreed terminology. this study addresses these challenges by initially identifying key odl activities that are associated with higher quality learning designs. the learning activity terminology was tested using independent raters, who categorised the learning activities in four odl courses as ‘interaction’, ‘feedback’ or ‘other’, with inter-rater reliability near or above recommended levels. whilst challenges remain for consistent categorisation, the analysis suggests that increased clarity in the learning activity will aid categorisation. as a result of this analysis, the e-design assessment tool (edat) has been developed to incorporate this key terminology and enable improved quantification of learning designs. this can be used with learning analytics, particularly retention and attainment data, thus providing an effective feedback loop on the learning design. keywords: learning activity; technology enhanced learning; terminology; online learning; learning design introduction higher education students are increasingly combining face-to-face learning with online distance and blended courses. in the united states 6 million students took at least one online course as part of their degree, which represented 30% of students in 2015 (allen and seaman 2017). in the uk, 10% of students in 2012–2013 were distance learners (garrett 2015). the university of edinburgh intends to include at least one fully online course in every undergraduate programme by 2025 (haywood 2016), a trend that is likely to continue because of demand for more flexible learning. online distance learning has its critics. high retention rates are often used as a measure of overall course quality (lenert and janes 2017), but retention is of concern, mailto:h.walmsley-smith@staffs.ac.uk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2106 h. walmsley-smith et al. 2 citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 (page number not for citation purpose) often being much lower than the equivalent face-to face version (simpson 2013). for example, the uk open university retention rate was 22% in 2010 despite its specialism in distance learning (simpson 2010). a range of possible factors affecting retention have been examined, ranging from learner-specific factors including age, gender, prior educational experience, levels of motivation and self-efficacy to institutional and course-specific factors including support available, course structure and the development of a learning community (e.g. bawa 2016). a study of distance learning course designs identified that some courses did not contain quality course features, for example, synchronous activities or projects (lenert and janes 2017). furthermore, ‘[w]hat is missing is the trajectory that would complete the feedback loop: the built-in evaluation of designs to see whether they achieved the expected outcomes’ (mor, ferguson, and wasson 2015, p. 224). a feedback loop would enable exploration of the specific impact online learning designs have on students’ learning and make possible recommendations for effective learning activities to enhance learning and retention. evaluation of learning designs is hampered by a lack of shared vocabularies for pedagogic practice (currier et al. 2006, section 2.2, no pagination). to achieve effective evaluation through a feedback loop requires ‘a more widely used language or framework for sharing learning designs’ (dalziel et al. 2016, p. 260). for laurillard (2012) it is an educational imperative to describe and represent online learning designs so that they can provide feedback to tutors about their effectiveness. research objectives a variety of common educational terminology is used by tutors to describe learning activities, but the extent to which they agree with the meaning and application is not known. this study therefore aimed to provide a reliable quantitative framework for categorising online activities by means of the following: objective 1: identifying types of effective online learning activities that support retention objective 2: testing terminology used to describe learning activities to identify the extent to which different users agree objective 3: developing the e-design assessment tool (edat) utilising this terminology to describe and quantify learning activities literature review: effective online learning activities levels of feedback and interaction in the course are two course design features often cited as having a significant impact on retention and each are discussed in the following sections. interaction support for interaction in learning comes from social constructivist learning theory (vygotsky and cole 1978). moreover, croxton’s (2014) meta-analysis indicates that both level and quality of interaction influence online retention. the literature includes different ways to define and measure interaction (wanstreet  2006). ‘transactional distance’ (moore and kearsley 2011) suggests physical and psychological distance between tutor and student is the main difficulty http://dx.doi.org/10.25304/rlt.v27.2106 research in learning technology citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 3 (page number not for citation purpose) of distance learning. moore (1989) identified three types of interaction: student– student, student–tutor and student–content. a fourth type of student-interface interaction has been proposed (hillman, willis, and gunawardena 1994). despite the wide use of moore’s interaction types, there is no clear agreement on how to measure them (ekwunife-orakwue and teng 2014). the following examples demonstrate how different surveys and data have been used to explore the impact of interaction on student retention. the community of inquiry model (garrison 2011) for online learning emphasises interaction between students and tutors, referred to as ‘social presence’. liu, gomez, and yen (2009) used the social presence and privacy questionnaire to measure social presence and identified it as a significant predictor of course retention and final grade. ‘resonance’ was used as a way to increase social presence by the use of video lectures, and analysis of the video access data suggested that this increased retention (geri 2012). an analysis suggested that the number of communication activities designed into a course was the primary predictor for retention (rienties and toetenel 2016). they examined 151 odl courses and calculated the time students were expected to spend on ‘communication’ using conole’s learning activity taxonomy (fill and conole 2005). a combination of data mining of forum posts and the use of their own student survey showed a positive correlation between student satisfaction and interaction rates (fasse, humbert, and rappold 2009). however, the challenge of isolating individual features of online courses to assess the impact of retention was highlighted by godwin, thorpe, and richardson (2008). they found no significant difference between courses with a variety of interaction patterns when comparing retention and attainment. ekwunife-orakwue and teng (2014) found a positive correlation between tutor–student interaction and retention by using student satisfaction and computer self-efficacy surveys. hawkins et al. (2013), using their own survey, found that feedback, procedural interaction and social interaction positively impacted on course completion. a web-based peer-tutoring system called online peer-assisted learning, which enhanced interaction by supporting students tutoring each other, also resulted in improved retention (evans and moore 2013). the study used social network analysis and the student assessment of learning gains survey. the use of webconferencing and structured group tasks achieved high retention as measured by course data and a course experience survey (thorpe 2008). interaction in collaborative group assignments using synchronous and asynchronous discussion as well as social media activities increased retention, according to data in the virtual learning environment (vle) student activity log (fisher and baird 2005). furthermore, frequency, rather than degree, of student interaction was identified as a positive marker for retention when vle data was analysed (shelton, hung, and lowenthal 2017). few studies have explored the impact of student–content interaction in online learning, making this is an area for possible further development (xiao 2017). the use of the edat as discussed in the following will enable further research in this area. feedback assessment and feedback activities are common in online learning. there are a variety of types, including formative individual and group tasks, online quizzes and tests, simulations, provision of model answers and summative assignments. hattie’s (2003) http://dx.doi.org/10.25304/rlt.v27.2106 h. walmsley-smith et al. 4 citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 (page number not for citation purpose) meta-analysis of teacher effectiveness found that giving students feedback was identified as a highly effective intervention. the impact of regular feedback to student postings was highlighted by stott’s (2016) case study, suggesting that low levels of student engagement and satisfaction may be the result of a lack of tutor feedback. a series of analytical writing assignments with feedback increased retention on a phd programme by 39% (sutton 2014). a cross-unit diagnostic that gave feedback to online learners from different learning units also had a positive effect on retention (lin et al. 2014). bonk and khoo (2014) highlighted the negative impact on online retention when prompt and individual feedback was not given. choi et al.’s (2013) survey identified that a lack of feedback from tutors was a key reason for students not re-enrolling. a systematic review of the impact of peer-assessment in online learning indicated that this ‘improves performance of students in learning environments in over 60% of the evaluated articles’ (tenório et al. 2016, p. 103). a course redesigned to include regular tests with automatic feedback increased attainment and reduced withdrawal (sancho-vinuesa, escudero-viladoms, and masià 2013). interaction and feedback are inherently linked: a tutor giving feedback to students is a form of interaction, and interactions with students provide feedback to tutors on how students are progressing (hatzipanagos and warburton 2009). representing learning designs the impact of course design features on retention can be investigated using the learning design conceptual framework (dalziel et al. 2013). dalziel argues that learning design can be used in fine-grained comparisons in educational research and that there is a need ‘to keep trying to develop a broadly accepted representational framework(s)’ (dalziel et al. 2016, p. 256). laurillard agrees: perhaps the attempt is doomed. but without it there is no basis for the comparative analysis of the range of conventional and digital teaching methods that will tell us how they may best be used to support student learning. that is an imperative for our education systems now, so we have to try. (laurillard 2012, chapter 5, no pagination) learning design representations are ways to represent or ‘codify’ learning designs to help online tutors and learning designers analyse and innovate, facilitate software developers to instantiate lessons in software or share designs with others (conole 2013). representations can include practice-based, conceptual, abstract or technical learning designs and those based on a specific theoretical approach. they can represent individual lessons or whole courses and provide different lenses to explore specific features including the nature of the task, the tools, resources or pedagogic principles. the most common type of representation is textual; other examples include content and course maps, pedagogy profiles, task swim lanes (visualisations) and learning outcome maps (conole 2013). however, each representation uses different terminology and formats, some embedding pedagogic guidance and others not. the learning design representations in table 1 illustrate the variety of terminology used to describe learning activities by different tools. this variety of learning activity terminology is challenging for learning designers when evaluating the effectiveness of learning designs. for example, the open http://dx.doi.org/10.25304/rlt.v27.2106 research in learning technology citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 5 (page number not for citation purpose) university mapping project used conole’s taxonomy (cross et al. 2012) to create a learning activity map over many courses. however, the authors commented on the difficulty of applying these terms, saying the process was ‘subjective’ and that they held ‘regular meetings to improve consistency’ (rienties, toetenel, and bryan 2015, p.  316). swan edited and applied six of reeves’ (1996) 14 pedagogical dimensions to her work describing mooc pedagogies; she also commented that raters needed a number of discussions to agree on their application (swan et al. 2015). similarly, laurillard observed that although tutors were able to map their own activities to a taxonomy, they were unable to agree when asked to map another tutor’s task (charlton, magoulas, and laurillard 2012). analysis of a number of us online courses used a rubric for raters to score each of four key elements on a three-point scale; it experienced similar difficulties (jaggars and xu 2016). even very simple terms seem to cause difficulties; for example, some users thought there was ambiguity between ‘resource’ and ‘support’ in the autc representation (agostinho 2011). a group of learning designers conducted an interesting study to apply different learning design tools to a single lesson plan to ‘represent’ the design. their challenges and varied results highlight the lack of consistency in learning design tools (persico et al. 2013). this variety of disparate terms makes consistent analysis of learning activities difficult. the edat, as described in the following, utilises the two commonly used terms (‘interaction’ and ‘feedback’) that are associated with higher retention in odl. the consistent use of these terms, as suggested by the following analysis, could enable a more accurate and effective way for tutors and learning designers to describe learning activities. when learning activities can be accurately described, they can be quantified and used with learning analytics to provide evidence for effective learning designs that increase retention (bakharia et al. 2016). table 1. learning activity taxonomies. name of learning design framework or tool terminology for learning activities updated bloom’s taxonomy (anderson and krathwohl 2001) type of activity: remember, understand, apply, analyse, evaluate, create australian universities teacher committee (autc) learning design (agostinho et al. 2002) elements of online learning design: resources, tasks, supports ulster hybrid (university of ulster 2008) online learning events: receives, debates, experiments, creates, explores, practises, imitates, meta-learns open university learning design initiative (ouldi) project (cross et al. 2012) online learning activity types: assimilative, finding and handling information, communication, productive, experiential, interactive or adaptive, assessment 7cs framework (conole 2014) online learning activities: capture, communicate, collaborate, consider learning designer online design tool (london knowledge lab 2016) based on the conversational framework (laurillard 2002, 2012) online learning activities: read, watch, listen (acquisition); collaborate; discuss; investigate; practice; produce e-design template (walmsley 2017), based on stephenson and coomey (2001) online learning activity types: student managed, tutor-managed, open activity, closed task http://dx.doi.org/10.25304/rlt.v27.2106 h. walmsley-smith et al. 6 citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 (page number not for citation purpose) methods the literature discussed suggests that retention is increased when odl includes interaction and feedback activities. however, these terms may not be used by tutors in the same way. these terms were tested using content analysis methodology to identify the extent to which tutors were using them consistently. content analysis content analysis is a method of quantifying text to enable statistical analysis of the text by a process of ‘coding’ or categorising. it is a ‘research technique for making replicable and valid inferences from texts (or other meaningful matter) to the context of their use’ (krippendorff 2013, p. 24). it has been used in a variety of educational settings, for example to analyse the impact of tutors’ roles in online discussions (dubuclet, lou, and macgregor 2015). to carry out a valid and reliable content analysis for this study, the following steps were taken: 1. specifying the units of analysis 2. identifying learning activity vocabulary to test 3. recruiting raters 4. calculating inter-rater reliability (irr) (adapted from neuendorf 2002, p. 50) specifying the units of analysis for this study the specific learning activities or task descriptions written by tutors and presented in the vle for students were analysed. a convenience sample of four distance learning modules from one higher education (he) institution were chosen to represent a variety of courses. they were varied and from different subject areas (law, politics, games and sport), aimed at different levels (undergraduate and postgraduate) and included a total of 215 learning activities of different types and lengths. identification of units of analysis is critical but also challenging (gorsky and blau 2009). if the unit of analysis is too general, it may be easy to categorise but hard to analyse; if too small, it may be difficult to categorise reliably. for this study the units of analysis were prepared by splitting activities into multiple parts based on the learning activity ‘verbs’. for example, a typical student activity was as follows: 1. read xx, answer the following [structured] question and then post your response to the forum. this was divided into the following for analysis: (i) read xx, (ii) answer the following [structured] question and then (iii) post your response to the forum. some courses included ‘optional activities’, for example, extended reading or open forums. these were also included as units of analysis because the impact of voluntary participation may be significant (so 2009). http://dx.doi.org/10.25304/rlt.v27.2106 research in learning technology citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 7 (page number not for citation purpose) identifying learning activity terminology to test based on the literature mentioned, analysis was conducted on the learning activity terms ‘interaction’, ‘feedback’ and ‘other’. activity types and examples were provided to assist the rater when categorising each activity, as in table 2. table 2. terminology tested. activity terminology activity type example interaction with … a. the tutor online webinar/lecture, 1–1 tutorial, coaching session, email, phone b. other students forum discussion (may include tutor), group work, peer assessment, adding comments to peer wikis or blogs c. (interactive) content computer simulation, multimedia interactions (excludes interaction with text or video) feedback from … 1. the tutor formative or summative feedback or grades 2. peers structured peer-assessment exercise, grading activity 3. self-feedback using model answers, self-reflection, trial and error exercises 4. computer (automatic) from computer simulation, computer marked test other activities reading or watching, research, creating recruiting raters in many studies, only two raters are used when a larger number would produce greater validity. independent raters may be unbiased, but in many studies raters are either researchers or the researchers’ assistants (e.g. rienties and toetenel 2016). raters require familiarity with the language and context for analysis but not overfamiliarity with specialised vocabulary, which may reduce the universality of their analysis (krippendorff 2013). here, all four raters were academic colleagues, familiar with educational terminology and who completed the content analysis task independently following training. calculating inter-rater reliability when raters all agree, this increases confidence that the analysis is consistent and objective and that other raters would be likely to obtain the same result. however, even high reliability scores do not guarantee validity. for example, raters may all display the same prejudice or use the same concepts as others in a specialised community. high reliability may also indicate a loss of validity; for example, the categories may be oversimplified or superficial (krippendorff 2013). in addition, high agreement between raters may simply mean that a particular item is missing from the content being analysed or that there is a high degree of similarity between the items being rated. http://dx.doi.org/10.25304/rlt.v27.2106 h. walmsley-smith et al. 8 citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 (page number not for citation purpose) inter-rater reliability is often measured using cohen’s kappa, but this has been criticised as it encourages the use of just two raters when more raters would provide more robust findings (krippendorff 2013). krippendorff ’s alpha (α) is a more effective measure of irr as it can be applied to any number of observers, any number of categories, any metric or level of measurement, as well as to incomplete data and large and small sample sizes (krippendorff 2011), and has been used in this study to calculate irr. there is no statistical rationale presented in the literature for acceptable levels of  irr. krippendorff (2004) suggests that where the analysis is critical, a level of α ≥ 0.800 should be considered necessary, and in situations where conclusions may be more tentative an irr of α ≥ 0.667 may be acceptable. all 215 learning activities from four courses were categorised by four raters. each course was rated independently, and each activity was categorised as ‘interaction’ and/ or ‘feedback’ or ‘other’. results the raters’ overall categorisations of ‘interaction’ or ‘feedback’ for each activity were compared, and irr was calculated with krippendorff ’s alpha. there was some disagreement among raters and although the ‘interaction’ category had an acceptable level of agreement, the ‘feedback’ categorisations were near to but did not reach an acceptable level of irr as in table 3. table 3. inter-rater reliability results. ‘interaction’ ‘feedback’ inter-rater reliability: 4 raters and 215 activities 0.815 0.612 discussion the irr figures show the difficulties in categorising learning activities even when using the commonly used terms ‘interaction’ and ‘feedback’. in total, of the 308 possible discussion-type categorisations, 285 were categorised as peer interaction and 197 as peer feedback. a significant issue was the way discussion forum activities were written; for example, discussion-type activities included five different terms: ‘discuss’, ‘post’, ‘comment’, ‘post & comment’ and ‘post & discuss’. raters categorised both ‘discuss’ and ‘post’ activities as including feedback when this was not indicated in the task. sixteen ‘discussions’ were rated as peer feedback. in addition, discussion activities were sometimes categorised as ‘other’, perhaps because raters thought that posting on a forum did not comprise interaction. within this variety of categorisations there was also noted a lack of consistency within raters. the highest level of agreement was for activities that specified both ‘post/comment’ and ‘post/discuss’, suggesting a greater clarity in the task. there were noticeable differences between raters when categorising feedback activities. for example, one rater categorised the activity ‘students access blackboard http://dx.doi.org/10.25304/rlt.v27.2106 research in learning technology citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 9 (page number not for citation purpose) for topic lecture notes, videos, etc. try to apply these techniques to your own work’ as feedback when no other rater had categorised it as such. another rater categorised the activity ‘please post … on the discussion board’ as feedback 22 times when the other raters did not. assessment activities were not consistently categorised as feedback, presumably because this was not specified in the activity. some learning activities that were inconsistently categorised did not conform to  good practice recommendations for interaction activities (e.g. akin and neal 2007;  salmon 2004) or recommendations for feedback (e.g. nicol and macfarlanedick 2006). however, a good practice example – ‘students post questions/comments in bulletin board for peer and tutor discussion’ – was categorised the same way by all raters. the selection of courses for this study included a variety of subject disciplines, and the raters were from different disciplines. this may have impacted on the ways the learning activities were written and also on the individual ways that the raters interpreted both the learning activity and the terms in the edat when completing the content analysis task. further research in this area is needed. conclusion feedback on the effectiveness of learning designs is needed to improve odl, but this is difficult to obtain without a consistent way to describe learning activities. two types of activities are highlighted in the literature as having the potential to improve retention and quality of online learning: interaction with tutor and peers and feedback on learning. however, despite these terms being commonly used, they were difficult to apply consistently to the learning activities in this study. the edat utilises this terminology to help improve categorisation and quantification. the difficulties in using common terms to categorise learning activities was surprising. the irr for interaction was acceptable, but the irr for feedback did not reach an acceptable level, suggesting that this is a complex term, difficult to use consistently. these terms, as used by the online course designers and by the raters, have different implicit meanings and reflect different teaching perspectives (trigwell, prosser, and ginns 2005). however, the example given of an activity categorised consistently suggests that increased clarity about opportunities for interaction and feedback in a task will improve consistent use of these terms. the edat has been developed to attempt to address these issues. it builds on the other learning design representation tools mentioned but focusses on two key online learning activities that are associated with higher retention. the edat enables tutors and designers to carry out the analysis described, that is, to categorise their learning activities using the terms ‘interaction’ and ‘feedback’ and to quantify them. interaction activities can be categorised with some confidence, but feedback activities may be less easy to identify and require review and editing for clarity. further analysis of the effectiveness of the tool is being conducted and will be reported separately. using the edat to categorise learning activities helps to provides quantitative data about the learning design. it also highlights to tutors the need to specify clearly to students when and how they will be interacting with others and when they can expect to receive feedback on each of their activities, thus potentially improving the learning design. http://dx.doi.org/10.25304/rlt.v27.2106 h. walmsley-smith et al. 10 citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 (page number not for citation purpose) appendix 1: the e-design assessment tool the e-design assessment tool (walmsley 2017) employs the tested terminology in both a word template and excel for use by tutors and designers, together with examples and a  guide  to quantifying learning activities. a sample follows, and both are freely available for download from the edat site: http://blogs.staffs.ac.uk/ bestpracticemodels/edat/. e-design assessment tool tutor instructions: add your activities below and indicate where you have specifically included interaction and/or feedback activities. calculate the % of each activity type to help you reflect on your learning design. use retention and attainment rates to evaluate the quality of the learning design. no specific learning activities/tasks (you may need to split activities that include separate parts) interaction with… a. tutor b. peers c. (interactive) content feedback from… 1. tutor 2. peers 3. self 4. computer (automatic) other content or activities ¸ activity text here … [add interaction type here if present in activity] [add feedback type here if present in activity] [insert additional rows as required] total activities: __ _% with interaction _% with feedback _% other references agostinho, s. (2011) ‘the use of a visual learning design representation to support the design process of teaching in higher education’, australasian journal of educational technology, vol. 27, no. 6, pp. 961–978. http://doi.org/10.14742/ajet.923 agostinho, s., et al., (2002) ‘a tool to evaluate the potential for an ict-based learning design to foster “high-quality learning”’, winds of change in the sea of learning. proceedings of the 19th annual conference of the australasian society for computers in learning in tertiary education, auckland, pp. 29–38, [online] available at: http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1128&context=edupapers akin, l. & neal, d. (2007) ‘crest+ model: writing effective online discussion questions’, journal of online learning and teaching, vol. 3, no. 2, pp. 191–202, [online] available at: http://jolt.merlot.org/vol3no2/akin.htm allen, i. e. & seaman, j. (2017) distance learning compass: distance education enrollment report, [online] available at: https://onlinelearningsurvey.com/reports/digtiallearning compassenrollment2017.pdf anderson, l. w. & krathwohl, d. r. (2001) a taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives, longman: new york. bakharia, a., et al., (2016) ‘a conceptual framework linking learning design with learning analytics’, in proceedings of the 6th international conference on learning analytics & knowledge, edinburgh, pp. 329–338. bawa, p. (2016) ‘retention in online courses: exploring issues and solutions-a literature review’, sage open, vol. 6, no. 1, pp. 1–11. http://doi.org/10.1177/2158244015621777 http://dx.doi.org/10.25304/rlt.v27.2106 http://blogs.staffs.ac.uk/bestpracticemodels/edat/ http://blogs.staffs.ac.uk/bestpracticemodels/edat/ http://doi.org/10.14742/ajet.923 http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1128&context=edupapers http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1128&context=edupapers http://jolt.merlot.org/vol3no2/akin.htm https://onlinelearningsurvey.com/reports/digtiallearningcompassenrollment2017.pdf https://onlinelearningsurvey.com/reports/digtiallearningcompassenrollment2017.pdf http://doi.org/10.1177/2158244015621777 research in learning technology citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 11 (page number not for citation purpose) bonk, c. j. & khoo, e. (2014) adding some tec-variety: 100+ activities for motivating and retaining learners online, open world books, bloomington, in. charlton, p., magoulas, g. & laurillard, d. (2012) ‘enabling creative learning design through semantic technologies’, technology, pedagogy and education, vol. 21, no. 2, pp. 231–253. http://doi.org/10.1080/1475939x.2012.698165 choi, h., et al., (2013) ‘the extent of and reasons for non re-enrollment: a case of korea national open university’, international review of research in open & distance learning, vol. 14, no. 4, pp. 19–35. http://dx.doi.org/10.19173/irrodl.v14i4.1314 conole, g. (2013) designing for learning in an open world, springer science & business media, milton keynes. conole, g. (2014) ‘the 7cs of learning design – a new approach to rethinking design practice’, in proceedings of the 9th international conference on networked learning, pp. 502–509, [online] available at: http://www.lancaster.ac.uk/fss/organisations/netlc/past/ nlc2014/abstracts/pdf/conole.pdf cross, s., et al., (2012) challenge and change in curriculum design process, communities, visualisation and practice, [online] available at: http://www.open.ac.uk/blogs/ouldi/wpcontent/uploads/2010/11/ouldi_final_report_final.pdf croxton, r. a. (2014) ‘the role of interactivity in student satisfaction and persistence in online learning’, journal of online learning and teaching, vol. 10, no. 2, pp. 314–325, [online] available from: http://jolt.merlot.org/vol10no2/croxton_0614.pdf currier, s., et al., (2006) ‘vocabularies for describing pedagogical approach in e-learning: a scoping study’, in proceedings of the international conference on dublin core and metadata applications: metadata for knowledge and learning, colima, oct 2006. dalziel, j. r., et al., (2013) the larnaca declaration on learning design, [online] available at: https://larnacadeclaration.wordpress.com/ dalziel, j. r., et al., (2016) ‘learning design: where do we go from here?’, in learning design: conceptualizing a framework for teaching and learning online, ed j. r. dalziel, routledge, abingdon. dubuclet, k. s., lou, y. & macgregor, k. (2015) ‘design and cognitive level of student dialogue in secondary school online courses’, american journal of distance education, vol. 29, no. 4, pp. 283–296. http://doi.org/10.1080/08923647.2015.1085722 ekwunife-orakwue, k. c. v. & teng, t. l. (2014) ‘the impact of transactional distance dialogic interactions on student learning outcomes in online and blended environments’, computers and education, vol. 78, pp. 414–427. http://doi.org/10.1016/j.compedu.2014.06.011 evans, m. j. & moore, j. s. (2013) ‘peer tutoring with the aid of the internet’, british journal of educational technology, vol. 44, no. 1, pp. 144–155. http://doi.org/10.1111/j. 14678535. 2011.01280.x fasse, r., humbert, j. & rappold, r. (2009) ‘rochester institute of technology: analyzing student success’, journal of asynchronous learning networks, vol. 13, no. 3, pp. 37–48. available from: https://files.eric.ed.gov/fulltext/ej862354.pdf fill, k. & conole, g. (2005) ‘a learning design toolkit to create pedagogically effective learning activities’, journal of interactive media in education, vol. 1, no. 9, pp. 1–16. http://doi. org/10.5334/2005-8 fisher, m. & baird, d. e. (2005) ‘online learning design that fosters student support, self regulation, and retention’, campus-wide information systems, vol. 22, no. 2, pp. 88–107. http://doi.org/10.1108/10650740510587100 garrett, r. (2015) up, down, flat: distance learning data collection and enrollment patterns in australia, uk, and usa, [online] available at: http://www.obhe.ac.uk/documents/download?id=1023 garrison, d. r. (2011) community of inquiry model, [online] available at: https://coi.athabascau. ca/coi-model/an-interactive-coi-model/ geri, n. (2012) ‘the resonance factor: probing the impact of video on student retention in distance learning’, interdisciplinary journal of e-learning and learning objects, vol. 8, [online] available at: https://www.learntechlib.org/p/44757/ http://dx.doi.org/10.25304/rlt.v27.2106 http://doi.org/10.1080/1475939x.2012.698165 http://dx.doi.org/10.19173/irrodl.v14i4.1314 http://www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2014/abstracts/pdf/conole.pdf http://www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2014/abstracts/pdf/conole.pdf http://www.open.ac.uk/blogs/ouldi/wp-content/uploads/2010/11/ouldi_final_report_final.pdf http://www.open.ac.uk/blogs/ouldi/wp-content/uploads/2010/11/ouldi_final_report_final.pdf http://jolt.merlot.org/vol10no2/croxton_0614.pdf https://larnacadeclaration.wordpress.com/ http://doi.org/10.1080/08923647.2015.1085722 http://doi.org/10.1016/j.compedu.2014.06.011 http://doi.org/10.1111/j.1467-8535.2011.01280.x http://doi.org/10.1111/j.1467-8535.2011.01280.x https://files.eric.ed.gov/fulltext/ej862354.pdf http://doi.org/10.5334/2005-8 http://doi.org/10.5334/2005-8 http://doi.org/10.1108/10650740510587100 http://www.obhe.ac.uk/documents/download?id=1023 https://coi.athabascau.ca/coi-model/an-interactive-coi-model/ https://coi.athabascau.ca/coi-model/an-interactive-coi-model/ https://www.learntechlib.org/p/44757/ h. walmsley-smith et al. 12 citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 (page number not for citation purpose) godwin, s. j., thorpe, m. & richardson, j. t. e. (2008) ‘the impact of computer-mediated interaction on distance learning’, british journal of educational technology, vol. 39, no. 1, pp. 52–70. http://doi.org/10.1111/j.1467-8535.2007.00727.x gorsky, p. & blau, i. (2009) ‘online teaching effectiveness: a tale of two instructors’, international review of research in open & distance learning, vol. 10, no. 3, pp. 1–28. http://dx.doi.org/10.19173/irrodl.v10i3.712 hattie, j. (2003) ‘teachers make a difference: what is the research evidence?’, in proceedings of the australian council for educational research annual conference: building teacher quality, melbourne, oct 2003. hatzipanagos, s. & warburton, s. (2009) ‘feedback as dialogue: exploring the links between formative assessment and social software in distance learning’, learning, media and technology, vol. 34, no. 1, pp. 45–59. http://doi.org/10.1080/17439880902759919 hawkins, a., et al., (2013) ‘academic performance, course completion rates, and student perception of the quality and frequency of interaction in a virtual high school’, distance education, vol. 34, no. 1, pp. 64–83. http://doi.org/10.1080/01587919.2013.770430 haywood, j. (2016) ‘learning from moocs: lessons for the future’, in from books to moocs? emerging models of learning and teaching in higher education, eds e. de corte, l. engwall & u. teichler, portland press, london, pp. 69–80. hillman, d., willis, d. & gunawardena, c. (1994) ‘learner-interface interaction in distance education: an extension of contemporary models and strategies for practitioners’, american journal of distance education, vol. 8, no. 2, pp. 30–42. http://doi.org/10.1080/ 08923649409526853 jaggars, s. s. & xu, d. (2016) ‘how do online course design features influence student performance?’, computers & education, vol. 95, pp. 270–284. http://doi.org/10.1016/j.compedu. 2016.01.014 krippendorff, k. (2004) ‘reliability in content analysis: some common misconceptions and recommendations’, human communication research, vol. 30, no. 3, pp. 411–433. http://doi. org/10.1111/j.1468-2958.2004.tb00738.x krippendorff, k. (2011) computing krippendorff’s alpha-reliability, annenberg school for communication departmental papers, philadelphia. krippendorff, k. (2013) content analysis: an introduction to its methodology, 3rd edn., sage, london. laurillard, d. (2002) rethinking university teaching: a conversational framework for the effective use of educational technology, 2nd edn., routledge, london. laurillard, d. (2012) teaching as a design science: building pedagogical patterns for learning and technology [vitalsource e-book], routledge, abingdon. lenert, k. a. & janes, d. p. (2017) ‘the incorporation of quality attributes into online course design in higher education’, international journal of e-learning & distance education, vol. 33, no. 1, pp. 1–14, [online] available at: http://www.ijede.ca/index.php/jde/article/view/987/1658 lin, j.-w., et al., (2014) ‘development and evaluation of across-unit diagnostic feedback mechanism for online learning’, journal of educational technology & society, vol. 17, no. 3, pp. 138–153, [online] available at: https://www.jstor.org/stable/jeductechsoci.17.3.138 liu, s., gomez, j. & yen, c.-j. (2009) ‘community college online course retention and final grade: predictability of social presence’, journal of interactive online learning, vol. 8, no. 2, pp. 165–182. available from https://pdfs.semanticscholar.org/3bea/7b0a25381625b933f0d91f6e3a5286ff9ac2.pdf london knowledge lab. (2016) learning designer, ucl institute of education, [online] available at: https://www.ucl.ac.uk/learning-designer/index.php moore, m. g. (1989) ‘three types of interaction’, american journal of distance education, vol. 3, no. 2, pp. 1–7. http://doi.org/10.1080/08923648909526659 moore, m. g. & kearsley, g. (2011) distance education: a systems view of online learning, 3rd edn., wadsworth, belmont. mor, y., ferguson, r. & wasson, b. (2015) ‘editorial: learning design, teacher inquiry into student learning and learning analytics: a call for action’, british journal of educational technology, vol. 46, no. 2, pp. 221–229. http://doi.org/10.1111/bjet.12273 http://dx.doi.org/10.25304/rlt.v27.2106 http://doi.org/10.1111/j.1467-8535.2007.00727.x http://dx.doi.org/10.19173/irrodl.v10i3.712 http://doi.org/10.1080/17439880902759919 http://doi.org/10.1080/01587919.2013.770430 http://doi.org/10.1080/08923649409526853 http://doi.org/10.1016/j.compedu.2016.01.014 http://doi.org/10.1111/j.1468-2958.2004.tb00738.x http://doi.org/10.1111/j.1468-2958.2004.tb00738.x http://www.ijede.ca/index.php/jde/article/view/987/1658 https://www.jstor.org/stable/jeductechsoci.17.3.138 https://pdfs.semanticscholar.org/3bea/7b0a25381625b933f0d91f6e3a5286ff9ac2.pdf https://pdfs.semanticscholar.org/3bea/7b0a25381625b933f0d91f6e3a5286ff9ac2.pdf https://www.ucl.ac.uk/learning-designer/index.php http://doi.org/10.1080/08923648909526659 http://doi.org/10.1111/bjet.12273 research in learning technology citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 13 (page number not for citation purpose) neuendorf, k. a. (2002) the content analysis guidebook, sage, london. nicol, d. j. & macfarlane-dick, d. (2006) ‘formative assessment and self-regulated learning: a model and seven principles of good feedback practice’, studies in higher education, vol. 31, no. 2, pp. 199–218. http://doi.org/10.1080/03075070600572090 persico, d., et al., (2013) ‘learning design rashomon i – supporting the design of one lesson through different approaches’, research in learning technology, vol. 21. http://doi. org/10.3402/rlt.v21i0.20224 reeves, t. (1996) evaluating what really matters in computer-based education, [online] available at: http://eduworks.com/documents/workshops/edmedia1998/docs/reeves.html rienties, b. & toetenel, l. (2016) ‘the impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules’, computers in human behavior, vol. 60, pp. 333–341. http://doi.org/10.1016/j.chb.2016.02.074 rienties, b., toetenel, l. & bryan, a. (2015) ‘“scaling up” learning design: impact of learning design activities on lms behavior and performance’, in proceedings of the 5th international conference on learning analytics and knowledge, poughkeepsie, march 2015. salmon, g. (2004) e-moderating: the key to online teaching and learning, routledge, london. sancho-vinuesa, t., escudero-viladoms, n. & masià, r. (2013) ‘continuous activity with immediate feedback: a good strategy to guarantee student engagement with the course’, open learning: the journal of open, distance and e-learning, vol. 28, no. 1, pp. 51–66. http://doi.org/10.1080/02680513.2013.776479 shelton, b. e., hung, j.-l. & lowenthal, p. r. (2017) ‘predicting student success by modeling student interaction in asynchronous online courses’, distance education, vol. 38, no. 1, pp. 59–69. http://doi.org/10.1080/01587919.2017.1299562 simpson, o. (2010) ‘22% – can we do better?’ – the cwp retention literature review, centre for widening participation, the open university, milton keynes. simpson, o. (2013) ‘student retention in distance education: are we failing our students?’, open learning: the journal of open, distance and e-learning, vol. 28, no. 2, pp. 105–119. http:// doi.org/10.1080/02680513.2013.847363 so, h.-j. (2009) ‘when groups decide to use asynchronous online discussions: collaborative learning and social presence under a voluntary participation structure’, journal of computer assisted learning, vol. 25, no. 2, pp. 143–160. http://doi.org/10.1111/j.1365-2729.2008.00293.x stephenson, j. & coomey, m. (2001) ‘online learning: it is all about dialogue, involvement, support and control – according to the research’, in teaching and learning online: new pedagogies for new technologies (creating success), ed j. stephenson, kogan page, london. pp. 37–52. stott, p. (2016) ‘the perils of a lack of student engagement: reflections of a “lonely, brave, and rather exposed” online instructor’, british journal of educational technology, vol. 47, no. 1, pp. 51–64. http://doi.org/10.1111/bjet.12215 sutton, r. (2014) ‘unlearning the past: new foundations for online student retention’, journal of educators online, vol. 11, no. 3, pp. 1–30. available from https://www.thejeo.com/ archive/archive/2014_113/suttonpdf swan, k., et al., (2015) ‘metaphors for learning and the pedagogies of moocs’, in meeting of the american educational research association, chicago, il, april 2015. tenório, t., et al., (2016) ‘does peer assessment in on-line learning environments work? a systematic review of the literature’, computers in human behavior, vol. 64, pp. 94–107. http:// doi.org/10.1016/j.chb.2016.06.020 thorpe, m. (2008) ‘effective online interaction: mapping course design to bridge from research to practice’, australasian journal of educational technology, vol. 24, no. 1, pp. 57–72. http://doi.org/10.14742/ajet.1230 trigwell, k., prosser, m. & ginns, p. (2005) ‘phenomenographic pedagogy and a revised “approaches to teaching inventory”’, higher education research & development, vol. 24, no. 4, pp. 349–360. http://doi.org/10.1080/07294360500284730 http://dx.doi.org/10.25304/rlt.v27.2106 http://doi.org/10.1080/03075070600572090 http://doi.org/10.3402/rlt.v21i0.20224 http://doi.org/10.3402/rlt.v21i0.20224 http://eduworks.com/documents/workshops/edmedia1998/docs/reeves.html http://doi.org/10.1016/j.chb.2016.02.074 http://doi.org/10.1080/02680513.2013.776479 http://doi.org/10.1080/01587919.2017.1299562 http://doi.org/10.1080/02680513.2013.847363 http://doi.org/10.1080/02680513.2013.847363 http://doi.org/10.1111/j.1365-2729.2008.00293.x http://doi.org/10.1111/bjet.12215 https://www.thejeo.com/archive/archive/2014_113/suttonpdf https://www.thejeo.com/archive/archive/2014_113/suttonpdf http://doi.org/10.1016/j.chb.2016.06.020 http://doi.org/10.1016/j.chb.2016.06.020 http://doi.org/10.14742/ajet.1230 http://doi.org/10.1080/07294360500284730 h. walmsley-smith et al. 14 citation: research in learning technology 2019, 27: 2106 http://dx.doi.org/10.25304/rlt.v27.2106 (page number not for citation purpose) university of ulster. (2008) hybrid learning model, [online] available at: http://addl.ulster. ac.uk/odl/hybridlearningmodel vygotsky, l. s. & cole, m. (1978) mind in society: the development of higher psychological processes, harvard university press. cambridge, ma. walmsley, h. (2017) best practice models for e-design, [online] stoke-on-trent available at: http://blogs.staffs.ac.uk/bestpracticemodels/ wanstreet, c. e. (2006) ‘interaction in online learning environments: a review of the literature’, the quarterly review of distance education, vol. 7, no. 4, pp. 399–411. xiao, j. (2017) ‘learner-content interaction in distance education: the weakest link in interaction research’, distance education, vol. 38, no. 1, pp. 123–135. http://doi.org/10.1080/0158 7919.2017.1298982 http://dx.doi.org/10.25304/rlt.v27.2106 http://addl.ulster.ac.uk/odl/hybridlearningmodel http://addl.ulster.ac.uk/odl/hybridlearningmodel http://blogs.staffs.ac.uk/bestpracticemodels/ http://doi.org/10.1080/01587919.2017.1298982 http://doi.org/10.1080/01587919.2017.1298982 editorial technology and plagiarism in many disciplines within higher education, there has been a steady move over the last decade or so away from traditional examinations at the end of courses. such examinations are seen as inherently unfair, partly because only in rare circumstances can a single set of timed tests genuinely reflect the content of an entire course, and partly because factors extraneous to normal intellectual capabilities, such as a headache, may unexpectedly depress a student's mark. modularization may go some way to easing educationists' anxieties on this score, but will not in itself completely dispel the perceived problems. other than dispensing with testing altogether (there are advocates of such an approach), there are only two ways of overcoming, or at least cushioning, the potentially unrepresentative effects of a final examination on which all or a significant part depends. hie first is to test in the traditional manner but at intervals throughout a course, with the consequent periodic examination results making up the final assessment, or counting towards it. the second way which has recently gained considerable ground is to introduce continuous assessment of work done outside the examination room (essays, dissertations, projects, assignments, group work and so forth) either as the sole set of criteria for the final mark or, again, as forming part of it. the overriding arguments in favour of this second form of continuous assessment in undergraduate courses, and therefore the principal reasons for its increasing usage, are twofold. first, it keeps a moderate amount of steady pressure on students as opposed to forcing them into occasional bursts of frantic revision (or frantic attempts to start more or less from scratch as an examination approaches). secondly, it leads students into the basics of scholarship and research methodology. but it is not without its disadvantages. for example, unless marks are kept secret until after the end of a course (which means limited feedback for the student), poor early results may demotivate. furthermore, a choice of topics may encourage an adherence to areas for which at least a moderately good mark is likely, thus limiting that exploration which, many would maintain, should be at the very heart of education. continuous assessment by project also has, however, a potential downside beyond any strict pedagogical disadvantages, namely the temptation to plagiarize. there is of course a grey area between blatant plagiarism on the one hand, and the inclusion of a certain amount of unacknowledged source material on the other. i am sure that many students who write dissertations but who are not yet fully familiar with scholarly procedures fail to provide source references not out of an intention to cheat but simply because they have not yet learned how to handle the referencing process, a process which involves not only using appropriate mechanisms but also judgement about when to include and when to omit a reference (indeed, the reverse of missing references over-referencing to the point of preventing a natural flow of the text is almost as common in student assignments within certain disciplines). nevertheless, as one moves out of the grey towards the black, there eventually comes an undeniable point at which an unacknowledged source represents sheer plagiarism. 2 alt-j volume 2 number 2 identifying such plagiarism is necessarily a hit-and-miss affair. an examiner, however expert in a subject, cannot be expected to recognize every unacknowledged source. in the majority of subject-areas, a student has such an enormous range of sources available (large numbers of published articles in numerous journals, little-known specialist reports, electronic information culled from obscure databases) that no single examiner, or even a set of examiners, could possibly know the details of them all. including proper references to such material may be to the student's advantage, since finding things out for oneself and showing how one has done so is something most examiners are keen to reward, but the temptation to omit references and thus to passideas off as one's own is probably even more common among undergraduates than among researchers. after all, having an original idea is seen by most students indeed most people as even better than showing how good one is at winkling out the ideas of others, whatever the high moral ground taken in the rubrics and guidelines for assessed assignments. then there is plagiarism by copying the work of fellow students (difficult in the examination room but all too easy outside it), so-called 'syndicating' (sharing the workload on an assignment without the knowledge of the examiner), and even buying the work of professional scribes. while such professionals are no doubt few and far between, it is an open secret in places of higher education that written work on a wide variety of topics can be bought, off the shelf or even tailor-made. there is a small but worrying black market in essays, dissertations, and numerical analyses of all kinds. one can apparently order a first, an upper second or if the disparity between one's former third-class work and the work one is about to submit would perhaps appear too great a lower second. the chances are that if the required level is judiciously chosen and the work carried out to the right specifications, most examiners tackling a daunting pile of assignments, in many cases from mostly anonymous students, will be oblivious to the dishonesty lurking behind perhaps one or two of them. now, it goes almost without saying that student plagiarism in one form or another has always existed, but the advent of information in digital form has given the activity a new dimension. digital information can not only be copied very rapidly, but also does not degrade from one generation of copying to the next: revealing little errors do not creep in during the copying process as they can and often do when re-typing or copying by hand. indeed, there is no foolproof way of detecting digital plagiarism. the plagiarism detection tool, briefly described by benford et al in this issue of alt-j, sounds fascinating and clearly has some interesting potential for certain applications within certain subject-areas, but obviously can provide only an approximate pointer towards possible cheating. other methods of detecting plagiarism, such as those sometimes employed in courts of law to prove authorship or otherwise, are of little or no use when it comes to numerical analyses, and in any event are far too cumbersome (and too contentious) to be used on a regular basis in places of higher education. nor is there any practical way of discouraging plagiarism other than by exhorting academic honesty or by giving an accompanying written or oral test under examination conditions either to all students or to a random sample. there is much to recommend the use of such tests, since copying without understanding may lead to being subsequently caught out, and though actually proving plagiarism may be beyond examiners, the threat of a test will tend to deter all but the most desperate or foolhardy of students. but accompanying tests add to the marking load, to a certain extent detract from the very object of project work, and in any case may 3 gabriel jacobs technology and plagiarism well do nothing to distinguish between original work on the one hand and, on the other, material which has been understood but which happens to have been plagiarized as well. leaving aside the problem of catching out students who copy assignments (or parts of them) written by other students or who buy assignment work from professionals, does it matter that material plagiarized from respectable sources is regurgitated as if it were original? clearly, at present it does matter, since the perception of examinees that examiners are favourably disposed towards originality is no doubt accurate (and indeed inadvertently reinforced by teachers who warn students against doing no more than throwing back lecture notes at them). how, then, is one to deal with what the increasing use of computer technology in education will inevitably bring with it: ever-easier access to an abundance of source material in digital form? without an accompanying test or viva voce examination, how does one judge an assignment which, one suspects, may be a compilation of chunks of unacknowledged material lifted from encyclopedias held on cd-rom and data sets held in electronic libraries? it may be that having given students the tools in the form of learning technology with which to access and manipulate digital information, we need to re-think our examining criteria when it comes to project work, at least at the undergraduate level. it may be that in terms of marks we should no longer attempt to distinguish, as most of us have tended to do in the past, between an assignment which has involved original thought and one which is simply a montage of unaltered source material. original thought, if one can be sure of recognizing it, must of course be rewarded, but should a student necessarily be penalized for having sourced material, then having presented it unmodified and even unacknowledged? by how much must it be altered and/or reorganized to demonstrate at least originality of form? this latter question is far from new, but the use of learning technology makes it all the more poignant, since many computer-based learning tools can also be used to lift source material and deposit it in pristine condition into assignments. some will argue that students have always been over-concerned with assessment by examiners, and that the kind of self-testing facilities often incorporated into learningtechnology applications are what they should really be concentrating on. this may be a valid point from the standpoint of a teacher, but try selling it to a student whose future career or academic progress depends on gaining a required pass mark. human nature being what it is, from the moment one grades assignments, and/or passes or fails them, the undoubted pedagogical value of self-assessment, however well appreciated, is bound to take something of a back seat in the mind of the average student. one way or another, then, those of us involved in examining project work and it looks as if we shall be increasing in number despite those who are currently forecasting a move back to tests and examinations have to come to terms with the new problems of assessment highlighted by the general availability of computer technology. perhaps the way forward is to turn a necessity into a virtue by thinking of term-time assessed assignments as primarily exercises in information-gathering and therefore less as projects intended to demonstrate understanding and/or learning. but that is a perilous stance indeed ... gabriel jacobs 4 finely integrated media for language learning henry hamburger department of computer science, george mason university, fairfax, virginia 22030-4444, usa; and naval research laboratory abstract fluent, an immersive foreign-language learning environment, was developed without recourse to hypermedia techniques. nevertheless, if one accepts the premisses, proposed in this paper, on which the idea of hypermedia has been constructed, fluent shows a strong relationship with it. the paper discusses this relationship after attempting to distil the essence of educational hypermedia, and after presenting a taxonomy of media for language learning. overview the hypermedia technologies seem to promise an explosion of opportunity for computerco-ordinated educational environments. some emerging possibilities were in evidence at the fourth biennial exeter conference on computer-assisted language learning (september, 1991), featuring the use of hypermedia as central theme. in this exciting but sometimes confusing environment, it will help to impose some sort of conceptual framework. to that end, i will first attempt to distil the essence of the field of educational hypermedia, and will then present a simple taxonomy of individual media for language learning. these two analytical lenses permit a clear view, in the latter half of the paper, of fluent, an immersive foreign-language learning environment developed by my research group. fluent grew up outside the hypermedia, but the two bear a strong kinship. after describing our system, i will examine the media challenges it faces and show how it embodies the spirit of hypermedia. hypermedia conceding in advance that opinions vary as to the nature and best use of hypermedia, here is an attempt to synthesize in one sentence some widely held views on the matter: 60 alt-j volume 1 number 1 the essence of hypermedia for learning is the integrated use of multiple media, with associated activities in the various media that promote implicit learning and student initiative. close reading reveals four assertions in the foregoing, which i now take up briefly. first, meaningfully integrated use should at least be possible, for otherwise the only benefit of making the media all available at once would be a slight improvement in convenience. for the whole system to exceed the sum of its media, there must be some kind of potential synergy, even if the responsibility of finding and exercising it falls to the student. next, technology-based activities for foreign-language learning range from computerization of traditional syntactic exercises to the literarily sophisticated, interactive videodisc story worlds of the athena language learning project (murray, 1987). a middle ground is the hypertext translation environment of the tiger project (deepwell, 1991). third, implicit or indirect learning means getting away from explicit statements about the way things are and letting students pick up an understanding of the target subject matter in the context of a meaningful activity. the importance of this kind of 'situated' learning has been argued for education in general (brown et al, 1989). in foreign language circles, the 'communicative' approach stresses the role of a meaningful situation, enabling the learner to acquire the regularities that grammars codify, without explicit mention of grammatical rules. the last feature is student initiative. allowing the student to get involved in deciding what to do may have the cognitive benefits of both active learning and the development of study skills, not to mention the personal benefit of self-expression. a cautionary note is in order, though: while the efficacy of active learning is well-established, it will not do for designers to abandon their responsibility to provide adequate guidance, leaving the student afflicted with 'cognitive agoraphobia' (twidale, 1991) or bogged down in a corner of the curriculum. media for language learning turning from hypermedia to media, we present a simple taxonomy, summarized in figure 1 opposite. note that there is no reference to particular pieces of equipment, nor to modalities like music, for which there is at most secondary use in the foreign-language context. our approach to foreign language immersion consists of conversations that take place in a supporting visual context that helps the student to understand remarks that would otherwise be slightly too difficult. in addition, the student can use interactive graphics for replies, while gaining exposure to new words and grammatical patterns. briefly put, this means two-modality, two-directional communication. the figure shows the two modes and the two directions, and indicates further subdivisions pertinent to each. one modality is language, of course, and the other is the spatial world that most of us (sighted people) receive by vision and which a computer may present in two dimensions by means of graphics algorithms. the two directions of language performance are comprehension and production, analogous to the familiar computerese terms input and output. the usefulness of this taxonomy will become clear in the final section of this paper, where it becomes a means to express design choices for our fluent immersion system, to which i now turn. 61 henry hamburger finely integrated media for language learning language y textual production / — recorded modality speech direction • — generated still spatial ^^^^^ moving comprehension ^ ^ precise rough figure 1: summary of a simple media taxonomy the fluent immersion project the idea of immersion is to help the student understand and use the new language directly and automatically, not mentally to translate or compute grammatical formulae. to this end, we get the computer to provide some of the immersion activities that good language tutors often use, employing the student's own language only to set the stage, and leaving the explicit teaching of grammar to others. (on the reasonableness of this approach, see richards and rodgers, 1986. also, note that it can be used with other systems and teachers and books that complement it.) immersion must be gradual, with exposure and comprehension preceding generation, and with new words and patterns introduced one at a time in situations where they can be figured out from the visual activity and the continuity of the conversation. let us see how these ideas play out in one of the microworlds that we have implemented (hamburger and maney, 1989; hamburger and hashim, 1991). upon entering washroom world, the student sees on the screen a drawing of a bathroom, suitably equipped. there is also a person with a movable hand that the student can manipulate with a mouse to pick up 62 alt-j volume 1 number 1 the soap or toothbrush, turn on the water, and so on. in one activity, the student's job is to make the figure go through the moves of washing its own face, while the tutor makes relevant comments and suggestions, all in spanish. this activity promotes recognition of language, but other exercises check the student's language production ability, and leave it to the tutor (system) to make screen moves. in either case, the tutor is directly responsive to whatever the student has just done or said. the responses can vary somewhat, according to different levels of general achievement the student may have attained. the most distinctive characteristic of fluent, is the very tight integration between the two modes (language and spatial) and between student and system. another important aspect is that the screen actions influence and reflect the workings of an internal model, representing the current situation and the properties of objects. having such a model enables the tutor to describe an action or its resulting state or even to critique an action in terms of a common-sense goal structure. thus if the student tells the figure to wipe his face, the response may be: "but i don't have the towel." the current implementation does these things with limited flexibility. the next version, now under construction, is making more extensive use of the techniques of artificial intelligence, including reasoning and especially natural-language processing. it will also draw upon techniques of intelligent tutoring systems to guide the pedagogical decisions. bringing it together i can now fulfil the promise to relate fluent to individual media and hypermedia, perhaps shedding some light on them all. first, referring to the media taxonomy of figure 1, i will mention a few design issues for fluent, touching on each modality and direction. speech would be ideal for fluent, but on the input side, recognizers are expensive and cannot readily deal with students' poor pronunciation. on the output side, inexpensive generators have a robotic sound that one hardly wants a student to mimic, and canned speech prohibits generativity. we plan to continue with text but will experiment with a compromise of cutting and pasting recorded speech, using multiple prosodic patterns for each recorded word or phrase. for spatial output, we have opted for generated images rather than recorded (photographed) ones, sacrificing visual fidelity in favour of close interaction between the student and the electronic tutor. showing movement involves another compromise, one between still and moving graphics, a technique we call partial animation. this approach is exemplified by the screen figure whose hand is manipulable but who is otherwise stationary, and a scene where some objects are movable. reaction from test users, even including a five-year-old, suggests that this approach can be fully engaging. as for language structure, however, compromise is far less in order. since the system's very purpose is to convey, albeit implicitly, the structure and vocabulary of a foreign language, it would be hobbled from the start if it did not handle language accurately and broadly. this entails using detailed, formalized language knowledge in a generative fashion. therefore, although the current system gets by with simple pattern-based analysis of language input, and filling sentence templates with simple generated noun phrases on the output side, the new version is importing a very strong natural-language processing component. 63 henry hamburger finely integrated media for language learning in sum, although we do not bill fluent as a hypermedia system, it turns out to have all four characteristics proposed above as the essence of educational hypermedia. certainly it thoroughly integrates the linguistic and the spatial media in a way that is essential to its goal. the second feature, the use of multiple and integrated activities, is also present. fluent provides activities for initial exposure to a microworld and its language aspects, as well as a variety of activities designed for comprehension, production, and assessment. third, student initiative too is clearly in evidence. for example, in some activities the student controls the direction of conversation by making things happen with the mouse, while in others the system carries out student wishes expressed in language. finally, the whole point of immersion is implicit learning. abstaining from explicit statement of grammatical patterns, fluent instead provides a visual and linguistic environment in which those patterns should come to seem familiar and natural. references brown, j.s., collins, a. and duguid p. (1989), 'situated cognition and the culture of learning', education researcher, 18, 1, 32-42. deepwell, f.h. (1991), 'the tiger project, coventry polytechnic', presented at the conference on call and the hypermedia, exeter university, september 18-20, 1991. hamburger, h. and hashim, r. (1991), 'a foreign language tutoring and learning environment' in swartz, m. and yazdani, m. (eds), the bridge to international communication: intelligent tutoring systems for second language learning, berlin, springer-verlag. hamburger, h. and maney t. (1991), 'twofold continuity in immersive language learning', computer-assisted language learning, 4, 2, 81-92. murray, j. (1987), 'humanists in an institute of technology: how foreign languages are reshaping workstation computing at mit', academic computing, september 1987. richards, j.c. and rodgers, t.s. (1986), approaches and methods in language teaching: a description and analysis, cambridge, cambridge language teaching library. twidale, m. (1991), 'cognitive agoraphobia and dilettantism: issues for reactive learning environments' in birnbaum, l. (ed), proceedings of the 1991 international conference on the learning sciences, charlottesville, va, association for the advancement of computing in education. 64 doi:10.3402/rlt.v19s1/7782 the making assessment count (mac) consortium � maximising assessment and feedback design by working together mark j.p. kerrigan a *, simon walker a , mark gamble b , sian lindsay c , kate reader d , maria-christiana papaefthimiou e , loretta newman-ford f , mark clements g and gunter saunders h a educational development unit, university of greenwich, london, uk; b centre for learning excellence, university of bedfordshire, uk; c learning development centre, city university, london, uk; d schools of arts and social sciences, city university, london, uk; e centre for the development of teaching and learning, university of reading, reading, uk; f learning and teaching development unit, uwic, cardiff, uk; g school of life sciences, university of westminster, london, uk; h westminster exchange, university of westminster, london, uk (received 12 june 2011; final version received 30 june 2011) the making assessment count (mac) project started at the university of westminster in 2008. it sought to align staff and student expectations of feedback and support greater use of feed-forward approaches. a baseline analysis of staff views in the school of life sciences suggested that students did not make strategic use of the feedback they received. a similar analysis of the student position revealed that as a group they felt that the feedback provided to them was often insufficiently helpful. to address this dichotomy, a mac process was developed in the school of life sciences and trialled with a cohort of about 350 first year undergraduate students. the process was based on a student-centred, three-stage model of feedback: subject specific, operational, and strategic (sos model). the student uses the subject tutor’s feedback on an assignment to complete an online self-review questionnaire delivered by a simple tool. the student answers are processed by a web application called e-reflect to generate a further feedback report. contained within this report are personalised graphical representations of performance, time management, satisfaction and other operational feedback designed to help the student reflect on their approach to preparation and completion of future work. the student then writes in an online learning journal, which is shared with their personal tutor to support the personal tutorial process and the student’s own development plan (pdp). since the initial development and implementation of the mac process within life sciences at westminster, a consortium of universities has worked together to maximise the benefits of the project outcomes and collaboratively explore how the sos model and e-reflect can be exploited in different institutional and subject contexts. this paper presents and discusses an evaluation of the use of the mac process within life sciences at westminster from both staff and student perspective. in addition, the paper will show how the consortium is working to develop a number of scenarios for utilisation of the process as a whole as well as the key individual process components, the sos model and e-reflect. keywords: assessment; consortium; coursework; efficiency; e-reflect; exam; feedback; feedforward; jisc; mac; online; pdp; peer; reflection; sos model; strategy; tutor; vle *corresponding author. email: m.j.p.kerrigan@greenwich.ac.uk alt-c 2011 conference proceedings 0161 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # m.j.p. kerrigan et al. this is an open access article distributed 43 under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1.7782 introduction in recent years, the university sector has focussed attention and resource on feedback, not directly because of its centrality in the learning process, but because it has been consistently flagged as a problem area by the national student survey (nss). students complain about ineffective feedback via the nss but staff also have reason to be disgruntled. many spend considerable time marking work and providing valuable feedback only to see piles of uncollected assignments suggesting disinterest and disillusionment amongst the students (winter and dye 2005). many staff strongly believe that students only pay attention to the mark they receive and make little attempt to engage with their feedback (wotjas 1998; mutch 2003). some research has suggested that withholding the mark can lead to greater engagement by students with their feedback (carless 2006) and an increase in the value added by the feedback (nichol 2007). assessment feedback has the potential to enhance achievement but only if the right balance can be struck between measuring performance and shaping and developing the individual (gibbs and simpson 2004). it is accepted that the balance between assessment of learning and for learning has leaned too much towards the former, driven by the need to measure student performance. such emphasis, driven to some extent by greater numbers of students and modularisation, leads to bunching of assessments (price and o’donovan 2008) and less opportunity for students to derive benefit from ‘practice’ application of knowledge and ideas with feedback often coming too late for it to make much difference to their performance (higgins, hartley, and skelton 2002). given the problems with formative assessments, it is imperative that feedback provided on marked work is sufficiently well crafted to help the student move forward. the feedback, irrespective of delivery, also needs to be linked to the processes that provide an opportunity for the student to analyse and make use of the feedback in what is now termed a ‘feed-forward approach’ (hounsell, xu, and tai 2007). one missed avenue for feed-forward on assessed coursework is the personal tutor as they are often out of the loop on their tutee’s performance until it is too late for them to make a difference in terms of ensuring that feedback is acted upon and leads to a definitive plan for improvement. this apparent decline in effectiveness can be linked to higher numbers of students and modularisation. many universities have responded to the need to fundamentally revisit the feedback process; from delivery through to action, by making more use of technology; for example, online marking and coursework return to increase speed. this should enhance the likelihood that a student utilise the feedback received in their next similar assignment (denton et al. 2008). in addition, technology allows for feedback to be returned in different ways, and there is emerging evidence that suggests alternatives to written feedback can lead to better formative feedback experiences (macgregor, spiers, and taylor 2011). at sheffield hallam university a system integral to the institutional virtual learning environment links the electronic release of marks to action on the part of the student. here students do not see their grade until they have looked at the electronic feedback that has been provided and have had the opportunity to write a reflective entry into the system (hepplestone 2010). however, there is the potential, given the nss driver, that the use of technology m.j.p. kerrigan et al. 44 could fuel superficial tick box approaches that may satisfy the customer but do little better to guarantee genuine personal development and improvement. technology may make a quality difference to feedback where it is part of a process that has at its heart the facilitation of meaningful interaction between students and their tutors, either online, face to face or both. in this paper, a process [the making assessment count (mac) process] designed to facilitate a dialogue that connects the feedback the student receives on their work, through their reflection, to the support and guidance of their tutor will be described and evaluated. the consequences of the process on students and staff will be discussed. in addition the paper will show how it is becoming possible for other universities and different subject areas to adapt the process to suit their own needs and priorities by working together as part of a strategic consortium. methods data from our baseline activities demonstrated that the feedback staff provided was deemed useful, yet students wanted more as they felt that would enhance their learning (kerrigan et al. 2009). conversely, there was a significant misalignment between student actions and their perceptions by staff in relation to how students use feedback and the value they place on the written comments. to address these core issues, a process, termed mac, was developed to enhance the amount of feedback students receive, demonstrate action on feedback to staff and enhance communication between students and their tutors. the summation of these ideas developed into a new model of student feedback into what can be defined as the sos model of a tripartite feedback; subject, operational and strategic (figure 1). figure 1. the sos model of student feedback. alt-c 2011 conference proceedings 45 subject the feedback on the students’ returned work mostly focusses on material aligned to academic performance and the development of subject matter and skills, indicating a level of performance, suggesting improvements and highlighting good achievement. operational after collecting their assignment and reading their subject feedback, students complete an online self-review questionnaire. this requires close engagement with the feedback they have been given and focusses on both the process and outcomes of learning � for example, students have to indicate how long they spent on the assignment, whether the guidance they had been given was appropriate and how well they understood the feedback they had received. their responses are then processed, server side, and personalised reports sent to students instantaneously via email. importantly, the report also contains a graphical representation of the student’s performance on all assignment completed. the online system used to complete the online questionnaires and produce the reports is known as e-reflect. strategic in the final step of the sos model, students use the operational report and their subject feedback as a prompt to write short reflective entries, focussing on actions they believe they need to take to improve, in an online learning journal that is shared with their personal tutor. personal tutors can comment on and extend the reflections in the learning journals and suggest further strategic action. once this stage is completed, both tutors and students can take better advantage of their faceto-face contact time: students enter the tutorial better able to articulate their difficulties, while tutors, who can refer to their tutees’ learning journals both before and during the tutorial, are better prepared to give appropriate support and guidance. the e-reflect tool the e-reflect tool is an integral component of the mac process (step 2), helping to encourage the student to think about their feedback and approaches to study, as well as providing additional operational feedback. importantly, e-reflect also serves as a ‘bridge’ between the student’s assignment feedback and their personal tutor. during the initial pilot phase, e-reflect 1.0 was built using excel macros and then for the larger-scale rollout, e-reflect 2.0 was developed with central computing services that linked to the student records system (via rss), enabling more effective processing and report generation. as more institutions become interested in adopting e-reflect, version reflect 3.0 was developed as a freestanding open source tool (figure 2). in e-reflect 3.2 (current version), questionnaire authoring by staff, questionnaire completion by students, report generation and storage and updating/sharing of the learning journal is all completed within a single system. there is also a system of alerts via email, m.j.p. kerrigan et al. 46 which communicate to staff when a tutee has completed a reflection and to students when their tutor has commented on their journal. evaluation the mac process was trialled on a large scale involving 380 undergraduates and 35 staff in the school of life sciences and subsequently evaluated by questionnaire and face-to-face interviews. this school was chosen as two members of the project team were active lecturers in it and had access to large numbers of students. it is important to note that the mac process is not discipline specific and is easily figure 2. screenshots of the mac tool. (1) the customisable online questionnaire students are asked to complete and (2) the area where they enter their reflective log. alt-c 2011 conference proceedings 47 transferable to any subject. both staff and students experiences were explored and analysed. the student view prior to the face-to-face interviews, a questionnaire was offered to the entire cohort and completed by 65 students. this disappointing response was attributed to the timing of the questionnaire: it coincided with the end of term. to increase our understanding of the student experience a second evaluation is planned at the beginning of the new academic year aimed at the same initial cohort. analysis largely reveals a picture of positive student views of the mac process. a majority of respondents indicated that they had used e-reflect either because it gave them extra feedback, helping to realise mistakes and prepare for other assignments, or because they thought that it was a way to improve and keep track of their progress. about 30% of respondents thought that using e-reflect had helped them to build a better relationship/communicate more with their tutor. conversely, about 10% said the process was time consuming and like an extra assignment. this response was surprising: the online questionnaire only took a few minutes to complete and the online learning journal only required a paragraph of text. a review of the student responses indicated that in some cases they were writing large reflections and thus spending too long on the process. to address this, students were supported on how to complete the learning journal. furthermore, in one instance the completion of ereflect was linked to a grade and so viewed by some as an assignment � following a review it was decided that this was not the best embedding of the process and a more student-centred, participatory approach was adopted instead. a majority of the respondents answered yes to the question ‘did using e-reflect help you to do any more of the things you feel you should do when you get a piece of marked coursework returned with feedback’? it should be noted that only a small proportion of students who did not engage with mac at all completed the questionnaire. it has to date proved impossible to gather together significant such students to elicit more on the basis of their lack of engagement. there is anecdotal evidence from academic staff that it was not just the high achieving students that engaged, although equally many staff felt that it was likely that the majority of the ‘non-engagers’ would be those students needing help and support most. this is borne out by the fact that of the set of students in the undergraduate cohort who did not progress, hardly any of them had engaged at all with e-reflect. whilst the questionnaire data provided a good empirical base, the data from the face-to-face interviews highlighted a variety of straightforward benefits. excerpts are shown in table 1. the students who took part in the face-to-face interviews were drawn from those who completed the initial questionnaire. not everything that students have said was positive. however, consistent amongst students was the view that the mac process was only going to be ‘really good’ or ‘make a difference in the long run’ if the feedback received on work was understandable and if personal tutors regularly commented on students’ learning journals and spoke to students about their feedback. this notion that students are now able to give feedback to their tutor about the quality and ‘fitness-for-purpose’ of their feedback has raised some interesting questions and prompted useful actions. m.j.p. kerrigan et al. 48 comments around these two themes of the quality of feedback and tutor engagement with the mac process are shown in table 2. all of the students who were interviewed and recorded had volunteered so to do in response to an email request. it is therefore possible that our sample of interviews is biassed towards students who are perhaps the ‘high achievers’, the more ‘aspirational’ or the more proactive and engaging. whilst this cannot be ruled out, it was the case that none of the students interviewed were shy about providing negative feedback about the process. across the 11 interviewed, 8 achieved ]60% marks on average and 3 achieved ]40% marks to b60%. when comparing the overall view of e-reflect and its real/potential value there was no difference between the groups thus suggested that academic performance was not the sole determinant in their perception. the staff view generally academic staff were not as positive about the mac process. a minority (1 out of 11 interviewed) thought that mac was of no use whatsoever whilst three table 2. student comments on the mac process in relation to quality and feedback. ‘‘i think e-reflect it’s really great but its going to work only if we get proper feedback. written, with good handwriting and with good points’’. ‘‘i just need some hints about where i went wrong. some kind of directions not something like so and a question mark and that’s it’’. ‘‘. . .sometimes the feedback on your essay might just be, ‘a good job’. now that doesn’t really give me enough information to work with’’. ‘‘after submitting 3 or 4 e-reflect questionnaires you remember what feedback is likely to come back. it’s nice but alone it can’t work, some tutors need to be continuously attached to it’’. ‘‘e-reflect without tutor comments is useless. if i want to reflect alone i can use word. the whole point of e-reflect is to get tutor and tutee to engage more around the work i have done’’. table 1. student comments on the mac process. ‘‘it has helped me especially on a couple of assignments where i’ve actually taken up the opportunity to see a module leader. one of my marks was quite low and this is not normally to my standard. the e-reflect feedback suggested that i speak to the module leader and so i took this advice and made an appointment. i wouldn’t normally have thought to do this and it helped. he gave me ways i could improve and suggested a different way of approaching some of the subject matter. it has helped i think because since then my marks have shot up’’. ‘‘first of all when i fill in the questionnaire and i put a new entry in my learning journal i identify my problem which i was thinking of. and i see it more clearly when i put it as a blog. and basically when i get feedback that i can understand i can share my thoughts with my personal tutor in my blog and sometimes he will come back with something i didn’t think of ’’. ‘‘for me personally it’s the way that i can see every piece of coursework that i’ve done. with the graphs it gives i can see where i’m falling down, where my strengths are and my weaknesses are. and also it was very helpful to get advice on how much time i should have spent on an assessment’’. ‘‘well for me e-reflect has helped me to workout exactly where i am going. i come from a previous degree which already set me high standards and with e-reflect, the three that i’ve done so far have allowed me to see that actually i was slowly dropping but i can actually identify where i’m dropping in’’. ‘‘it’s made me think more about the assignments. before if i got a good mark i just thought yeah that’s a good mark and left it at that. but now i’m actually going back and thinking about what i could have done better’’. alt-c 2011 conference proceedings 49 others agreed that ‘it clearly affected some students in a positive way’. the rest of the staff interviewed held a view somewhere in between, feeling that any initiative to highlight to students the need to pay attention to their feedback was good. although almost all staff could see the potential of the mac process and thought that it was something worth developing further, a number of significant issues and problems were raised. these ranged from not having the time to engage fully, through to significant doubts that the process would help the ‘weaker’ student. a questionnaire sent to staff also highlighted the degree to which staff see potential for the mac process with 10 out of 12 respondents agreeing that it is ‘very good in principle’. however 33% highlighted a lack of student engagement as a problem whilst 25% of respondents highlighted the value of the mac process for monitoring student progress. encouragingly, over half of the staff respondents thought that the mac process had improved active dialogue between students and staff over student’s work and development and around 40% thought that the mac process had impacted on the way in which they tutored and/or provided feedback to students (table 3). of note, two staff also stated that implementation of the mac process had impacted on the way that they approached provision of feedback. another member of staff changed the way he instructed his teaching team to deliver coursework feedback influencing 12 members of academic staff, some of whom were not directly engaged with mac. collectively this course team agreed to focus more on providing students with ‘action points’ that they should consider in order to improve. combined view on feedback sources as part of the larger evaluation and support for future design, staff were asked how they deliver support on feedback and students were asked who they talk to about their coursework. this resulted in some interesting data linked to the use of the mac process and highlighting a potential miss-alignment in activity. students appear to be willing to discuss their feedback with their personal tutor as well as the person who marked their work and indeed with a ‘third party’ member of staff who they like (figure 3). this suggests that the model of a tutor supporting action on feedback is valid and highlights that a professional relationship is important in this process. interestingly, students seek more than one source of support; this may not always include the marker or their personal tutor. whilst it is well known that peer feedback is important, the ability for students to comment and suggest actions on their peers’ feedback is an interesting extension. indeed, one could build this into the mac process and permit students to select with whom their feedback is shared. furthermore, these data show that feedback on a script is often shared with more than one audience: this should be considered when feedback is constructed. table 3. staff comments on the mac process. ‘‘i have tried to give clear, concise and justifiable feedback on all work and embed corrections and tips for future work’’. ‘‘e-reflect has enabled me to observe potential problems earlier and to give generic advice on how to help the students improve’’. ‘‘e-reflect has enabled me to get to know my students better. it has made me think about the way i give feedback to ensure that it is clear to the students exactly what it is they need to improve by providing additional action points at the end of the work’’. m.j.p. kerrigan et al. 50 as these data suggest that students are engaging with staff about their feedback, we then asked staff how they are being contacted (figure 4). those who responded with either agree or strongly agree were grouped as positive, neither agree nor disagree as neutral and disagree or strongly disagree as negative. interestingly, there did not appear to be a dominant method by which the students contacted staff for figure 3. the students’ sharing of feedback. figure 4. how staff are contacted by student to provide feedback support. alt-c 2011 conference proceedings 51 support with their feedback. there was a slight preference towards support in class, which would align with figure 3 but the number of staff who gave a neutral response prevent this from being conclusive. these data do suggest that students are willing to accept additional feedback support by email and that some staff provide this. finally, we then asked staff how many students contacted them about the last piece of marked work they had returned (figure 5). whilst there were a few instances wherein all students contacted the member of staff, in the main most staff indicated that 10% or under of students had contacted them. whilst initially this could appear to be of concern, the data from figure 3 suggest that 43% of the feedback support students engage with is from peers and 56% from academics of which 10% may not be related to the piece of work. it could be argued therefore that under 1% of respondents took no action on their feedback. the mac consortium following the successes of the original mac project, a number of other universities [specifically bedfordshire, city university london, greenwich, reading and university of wales institute cardiff (uwic)] have been exploring how best to make use of the project outputs. this has included consideration of the mac process as a whole, the sos model and how best to utilise the e-reflect tool. the notion of working with ‘competitors’ is complex as each institution within the consortium draws from the same student pools but importantly, we have realised that the learning and support form working together is significant. with each institution developing a different ‘flavour’ of mac, building on their own interests/expertise, and running a pilot with associated evaluation, everyone will benefit from the future findings (figure 6). figure 5. analysis of students who contacted staff for support. m.j.p. kerrigan et al. 52 at uwic the mac process was piloted on an essay assignment with 50 first year undergraduate students on the bsc sports science programme. contextual differences in the self-review questionnaire including whether students had made a plan for their essay before starting, whether they thought the mark received was as high as they could have achieved and what they plan to do next with the assignment gave a novel approach for mac. a focus group of students at uwic were generally extremely positive about the experience. they all said that they had benefited from the experience of using the mac model, including the e-reflect questionnaire and journal. when asked how they had benefited, most responded that it had made them think deeply about their own strengths and weaknesses, the ways in which they approached their assessment task, areas for future development and the usefulness of tutor feedback. at city university london there is strong emphasis on engaging students with the feedback that they receive from their tutors as evidenced in the international politics department that has a well-developed face-to-face personal tutorial scheme for students. as a consequence, the international politics department plans to use an adapted form of the mac process (academic year 2011�2012) to help link the students’ work with the face-to-face tutorial meetings. as in the original mac model, students will complete an online self-review questionnaire, however, staff will not be commenting online on students’ reflections but instead students will book a face-to face-tutorial using an online moodle scheduler. to achieve this, the educational support team has built the mac process into moodle (table 4). the student’s reflection on their work will inform the tutorial. at the university of reading, the e-reflect tool will be used at various stages of a year-long research project to assist undergraduate students. this novel approach of using mac to track and support a substantive piece of coursework is an exciting development. students will be asked to reflect on the preparation of the literature review for their research project immediately after the submission deadline and again after the return of the assessed review with appropriate feedback generated soon after their reflection. also, students will reflect on their performance at the end of the laboratory work or data collection period and again feedback will be generated. at the university of westminster, the school of life sciences has adapted the original mac process so that it can be used to facilitate feedback on written figure 6. enhancement of the mac process by working as a consortium. alt-c 2011 conference proceedings 53 examinations; this is a two-stage process. this use of the mac process addresses the continuous concern that students do not receive feedback on exams. the first stage requires students to complete a questionnaire on an exam they have just taken that prompts the student to predict the grade they expect for the exam. it asks several questions about the way the student prepared for and answered the exam paper. the automated report derived from questionnaire completion provides tips on how to improve their future exam performance based on the responses they gave. students then go on to write a reflection in their learning journal. the second stage of the process is initiated one month later (once all exam papers have been marked and approved by the exam board) at which point students are provided with an opportunity to see one of their exam papers annotated with written feedback. students then complete a second e-reflect questionnaire about their performance comparing the grade they predicted with their actual grade. they are sent a second automated report with suggestions of areas for further reflection after which they complete a second entry in their learning journal. this is shared with their personal tutor who is able to comment. finally, at the university of greenwich the mac process is currently being adapted to support staff development by integrating it with a postgraduate teaching and learning course. by substituting the coursework element with objectives and skills this will permit a reflective approach whilst promoting a student-centric strategy for enhancement around teaching and learning. at the university of bedfordshire there are plans to use the e-reflect tool to help international students to adapt to studying in the uk. reflective questionnaires will be used iteratively in an attempt to support students more in the critical early stages of their taught programme. conclusions there is strong evidence that the mac process can help some students engage with, and make more of their feedback. it seems that a straightforward technology (e-reflect) can be used to encourage students to think more about their feedback. importantly, the introduction of the technology can potentially change the nature of a face-to-face tutorial system to focus the tutee/tutor relationship more on academic performance as well as influence how academic staff approach delivering feedback. these three connected transformations are of clear significance and, provided that groups of staff are convinced of the payback, and that mac or some ‘flavour’ can be table 4. delivery of a mac process using moodle. moodle tool process moodle gradebook online submission, dissemination of grades, subject specific and operational feedback and reflections to tutors and students. moodle quiz tool delivery and collection of operational feedback questionnaire through multiple-choice questions and collection of student reflections using the essay question. moodle scheduler booking and management of tutorials. moodle html block aggregation and display of the activities and guidance to ensure correct path through the mac model. m.j.p. kerrigan et al. 54 readily integrated readily into support mechanisms, there is every possibility that students will benefit. the consortium strategy has enhanced the development and realisation of the mac process by a trial in a second virtual learning environment (vle) as well as developing moodle blocks, links to ug research and exam feedback � all significant enhancements that would not have been achievable by a single institution. however, variations in delivery of the approach are emerging, as are uses of a mac process linked explicitly to personal development planning and employability. the mac process has strong potential to support students in their understanding and acting on feedback, as well as being a catalyst for enhancing student�tutor academic relationships. furthermore, with increasing pressures on staff and student time, a tool that can enhance the effectiveness of face-to-face tutorials as well as support learning through reflection, could be a welcome addition to an institution’s technology enhanced learning strategy. by working as part of an effective consortium, the development of the mac process has been significantly enhanced, maximising the benefits of a project for those institutions involved thereby increasing sustainability and dissemination within the sector. information on the mac project can be found at: http://www.makingassessment count.ac.uk acknowledgements the mac project and the subsequent benefits realisation project were funded by joint information systems committee (jisc) as part of the curriculum delivery programme. http:// www.jisc.ac.uk/whatwedo/programmes/elearning/curriculumdelivery.aspx references carless, d. 2006. differing perceptions in the feedback process. studies in higher education 21, no. 2: 219�33. denton, p., j. madden, m. roberts, and p. rowe. 2008. students’ response to traditional and computer-assisted formative feedback: a comparative case study. british journal of educational technology 39, no. 3: 486�500. gibbs, g., and c. simpson. 2004. conditions under which assessment supports students’ learning. learning and teaching in higher education 1: 3�31. hepplestone, s. 2010. technology, feedback action! a short report. https://files.pbworks.com/ download/ma1253u15k/evidencenet/23202481/guide%20for%20senior%20managers%20 final.pdf (accessed june 10, 2011) higgins, r., p. hartley, and a. skelton. 2002. the conscientious consumer; reconsidering the role of assessment feedback in student learning. studies in higher education 27, no. 1: 53�64. hounsell, d., r. xu, and c.m. tai. 2007. monitoring students’ experiences of assessment. (scottish enhancement themes: guides to integrative assessment, no.1). gloucester: quality assurance agency for higher education. http://www.enhancementthemes.ac.uk/ publications/ (accessed june 10, 2011) kerrigan, m.j.p., m. clements, a. bond, and g. saunders. 2009. e-reflect-making assessment count. proceedings of the fourth international blended learning conference, june 2009 in university of hertfordshire. isbn: 978-1-905313-66-2, 219�32. macgregor, g., a. spiers, and c. taylor. 2011. exploratory evaluation of audio email technology in formative assessment feedback. research in learning technology 19, no. 1: 39�59. mutch, a. 2003. exploring the practice of feedback to students. active learning in higher education 4, no. 24: 24�38. alt-c 2011 conference proceedings 55 http://www.makingassessmentcount.ac.uk http://www.makingassessmentcount.ac.uk http://www.jisc.ac.uk/whatwedo/programmes/elearning/curriculumdelivery.aspx http://www.jisc.ac.uk/whatwedo/programmes/elearning/curriculumdelivery.aspx https://files.pbworks.com/download/ma1253u15k/evidencenet/23202481/guide%20for%20senior%20managers%20final.pdf https://files.pbworks.com/download/ma1253u15k/evidencenet/23202481/guide%20for%20senior%20managers%20final.pdf https://files.pbworks.com/download/ma1253u15k/evidencenet/23202481/guide%20for%20senior%20managers%20final.pdf http://www.enhancementthemes.ac.uk/publications/ http://www.enhancementthemes.ac.uk/publications/ nichol, d. 2007. improving assessment after the task. re-engineering assessment practices in scottish education (reap) online resources. http://www.reap.ac.uk/nss/nssafter02.html (accessed june 8, 2011) price, m., and b. o’donovan. 2008. feedback � all that effort, but what is the effect? paper presented at earli/northumbria assessment conference, august 27�29, seminaris seehotel, potsdam, germany. winter, c., and v.l. dye. 2005. an investigation into the reasons why students do not collect marked assignments and the accompanying feedback. http://wlv.openrepository.com/wlv/ bitstream/2436/3780/1/an%20investigation%20pgs%20133-141.pdf (accessed june 8, 2011) wojtas, o. 1998. feedback? no, just give us the answers. times higher education supplement september 25: 7. m.j.p. kerrigan et al. 56 http://www.reap.ac.uk/nss/nssafter02.html http://wlv.openrepository.com/wlv/bitstream/2436/3780/1/an%20investigation%20pgs%20133-141.pdf http://wlv.openrepository.com/wlv/bitstream/2436/3780/1/an%20investigation%20pgs%20133-141.pdf doi:10.3402/rlt.v19s1/7796 defrosting professional development: reconceptualising teaching using social learning technologies thomas cochrane* and vickel narayan te puna ako, unitec, carrington road, mt albert, auckland, new zealand (received 01 march 2011; final version received 27 may 2011) in this paper we discuss the impact of redesigning a lecturer professional development course with the aim of embedding a community of practice (cop) model supported by the use of mobile web 2.0 technologies. this approach was based upon a model developed to support 30 mlearning projects between 2006 and 2010, which also informed the institutions’ new elearning strategy developed in 2009. participating lecturers were brought into the course as participants in an intentional cop investigating the pedagogical application of social learning theories and frameworks, facilitated by the course lecturers who took on the role of technology stewards guiding the cop in the appropriation of mobile web 2.0. three examples of participants’ journeys of discovery throughout the course are highlighted to illustrate the impact of this approach to professional development. reflections on the first 2010 iteration of the course are then used to inform the following iterations in 2011. keywords: professional development; communities of practice; social learning theories; pedagogy-andragogy-heutagogy continuum introduction in the 2010 movie ‘‘kick-ass’’ (vaughn 2010) nicholas cage plays a fanatic vigilante (damon macready) fighting crime and training his daughter to do likewise through experiential learning. cage fires a round of a pistol at his character’s daughter (mindy) wearing a bullet-proof vest: (mindy) daddy i’m scared (damon) come on mindy, honey, be a big girl now, there’s nothing to be afraid of. (mindy) is it gonna hurt bad? (damon) only for a second sugar. a handgun bullet travels at more than? (mindy) 700 miles an hour. (damon) so at close range the force is going to take you off your feet for sure, but it’s really no more painful than a punch in the chest. (mindy) i hate getting punched in the chest. (damon) you’re going to be fine baby doll. shot (damon) how was that? not so bad, kinda fun huh? now you know how it feels, you won’t be scared when some chunky asshole pulls a glock. (vaughn 2010) in a similar way to cage’s kick-ass character, the researchers developed the social learning technologies (slt) course as an experiential learning environment for the *corresponding author. email: tcochrane@unitec.ac.nz alt-c 2011 conference proceedings 0094 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # t. cochrane and v. narayan. this is an open access article 158 distributed under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7796 participants, while informed by a graduate-level critique and reflection upon emergent learning theory. the goal was to provide participants with a model and experience of both a community of practice (cop) and enabling mobile web 2.0 tools that they could then continue to develop within their own teaching and learning contexts after the completion of the course. this was underpinned by a rigorous investigation of social learning theories and frameworks throughout the course, and scaffolding the experiential learning via the establishment of the course as a supportive cop. development of the social learning technologies course the graduate diploma of higher education (gdhe) is one of the institution’s primary methods of lecturer professional development. however the learning technologies paper of the gdhe had become dated and antiquated. the authors were tasked with redeveloping this paper and bringing it into alignment with the institution’s new elearning strategy. the context unitec is new zealand’s largest polytechnic and is in the process of differentiating itself from new zealand’s eight universities by the roll-out of a distinctive pedagogical approach termed the living curriculum and exemplified in the institution’s new elearning strategy. the cop model for professional development a cop model was developed (cochrane 2007; cochrane and kligyte 2007) to support the implementation of over 30 mlearning projects managed and implemented in partnership with a variety of lecturers by the authors between 2006 and 2010, and has become a core element of the institution’s new elearning strategy (cochrane 2010). the 2006�2010 research was interested in bringing about sustainable and transferable pedagogical change that would benefit lecturers and students, transforming pedagogy from a face-to-face classroom based instructivist paradigm to a context bridging social constructivist paradigm. mobile web 2.0 tools were used as a catalyst for this pedagogical change. to achieve this goal, the second problem was creating an implementation approach that did not rely upon (or never go beyond) already techno-savvy (‘geek’) lecturers, but was capable of supporting and scaffolding the average lecturer to become confident integrating innovative technologies into their curriculum. rather than relying upon a series of workshops, the sustained engagement of a cop was found to achieve significant ontological shifts for both lecturers’ conceptions of teaching, and students’ conceptions of what it means to be a learner. research methodology a participatory action research methodology was used for evaluating the impact of the redesigned slt course, which was embedded within the roll-out of the institution’s new elearning strategy, developed with strategic input from the authors of this paper. all slt students signed ethics consent forms and an acceptable use policy relating to the use of the mobile web 2.0 tools throughout the course. the 2010 slt class began with alt-c 2011 conference proceedings 159 nine enrolling students with two students withdrawing in the first week of the course due to time constraints, leaving a small but committed class of seven students, and two facilitating lecturers. the course participants were expected to have a wifi capable laptop computer for use during the course. the one student who did not have access to a laptop was supplied with a netbook for use throughout the course. additionally, all of the course students were supplied with an iphone 4 for use during the course, allowing them to experience the affordances of mobile web 2.0. the introductory session of the course established the core collaboration tools used to enable the cop to operate beyond the face-to-face sessions, including: twitter (including a course hashtag), personal blogs, a group wiki page (http://ctliwiki.unitec.ac.nz/index.php/social learningtechnologies), gmail and associated google apps, and a course moodle hub where students added their web 2.0 contact details to their moodle profiles. the moodle lms (learning management system) was therefore used as a scaffold while students established their own ple (personal learning environment) consisting of a mashup of web 2.0 tools. data collection and triangulation data collection consisted of: (1) beginning of course surveys of lecturers and students, to establish current practice, expertise and experience. (2) post-course surveys and focus group, to measure the impact of the mobile web 2.0 environment, and identify emergent themes. (3) lecturer and student reflections via their own blogs and eportfolios throughout the course, collated via rss feeds. the research used the technologies that were an integral part of the redesigned course assessment, such as participant blog posts, peer blog comments, and vodcast reflections to capture data on the progression and impact of mobile web 2.0 on the participants’ learning experience. communities of practice ‘communities of practice’ (cop) is a social learning theory. the concepts were proposed by lave and wenger (1991), while studying the apprenticeship model of learning. wenger (1998) later further developed the concepts, and then simplified the concepts for wider contexts: ‘‘communities of practice are formed by people who engage in a process of collective learning in a shared domain of human endeavour’’ (wenger 2005, 1). though not originally intended as a pedagogical strategy or teaching technique, rather an analytical viewpoint on learning (lave and wenger 1991), the concepts of cop have found popularity within educational contexts. the main differences between traditional teacher-directed (didactic) educational environments and cop are: an emphasis on inventiveness with a continual evolution of ideas and direction of the community (brown 2006), a lack of hierarchy (head and dakers 2005; langelier 2005) and teachers take on the role of expert mentor (herrington et al. 2006) rather than delivery of content. the slt course was designed as an intentional cop. wenger’s (2005) definition of cop ‘‘allows for, but does not assume, intentionality’’ (1). while cop often form organically and spontaneously, they can also be created intentionally and cultivated t. cochrane and v. narayan 160 http://ctliwiki.unitec.ac.nz/index.php/sociallearningtechnologies http://ctliwiki.unitec.ac.nz/index.php/sociallearningtechnologies for specific purposes. intentional cop share the same characteristics as organic cop, but have at their core a plan. one of the key concepts developed out of cop has been the importance of ‘technology stewards’ (wenger, white, and smith 2009; wenger et al. 2005) within cops to guide the use of technologies supporting the cop. within the context of the slt course, the course lecturers took on the role of technology stewards, attempting to model the pedagogical use of mobile web 2.0 as part of a collaborative partnership with the course students. social learning theory and frameworks the slt course was explicitly founded upon a social constructivist pedagogy (vygotsky 1978) and focused upon students investigating related pedagogical theory and frameworks and the appropriation of web 2.0 tools to implement these theories and frameworks within their pedagogical practice. these included both established and emerging theories and frameworks such as: cop (lave and wenger 1991), the conversational framework (laurillard 2001), learner-generated content and learnergenerated contexts (luckin et al. 2008, 2010), authentic learning (herrington and herrington 2007; herrington and oliver 2000), connectivism (siemens 2004) and activity theory (engestrom 1987). links were provided to educational research organisations that publish regular reports and rss feeds to new resources, thus keeping the course ‘readings’ up to date rather than reliant upon rapidly aging set texts. these included: . educause, 7 things you should know about series [http://www.educause. edu/7things] . jisc reports [http://www.jisc.ac.uk/publications.aspx] . new consortium reports [http://www.nmc.org/publications] . educause resources [http://www.educause.edu/resources] . becta [http://research.becta.org.uk/] redesigning the gdhe slt paper the redesign of the gdhe learning technologies paper into the new slt paper was a collaborative process by the two authors during 2009. the final course was approved late 2009 and ran for the first time in semester two of 2010 with the two authors as the course lecturers. course outline: 2009 vs 2010 the original learning technologies paper centred round the course participants creating a resource for their students to use, i.e. teacher-generated content. the redesigned slt course focused upon modelling the use of mobile web 2.0 tools as a catalyst for pedagogical transformation, leading to the participants’ developing their own theory and experience-informed teaching and learning framework. this framework was to establish links between new and emerging learning technologies and social learning theories, and then became the basis from which they developed student-centred learning activities for their context, i.e. enabling student-generated alt-c 2011 conference proceedings 161 http://www.educause.edu/7things http://www.educause.edu/7things http://www.jisc.ac.uk/publications.aspx http://www.nmc.org/publications http://www.educause.edu/resources http://research.becta.org.uk/ content and student-generated learning contexts. table 1 outlines the key differences in the redesign of the slt paper. the slt course ran over the period of a semester, with six 3-hour long face-toface sessions. figure 1 illustrates the structure of the course, within the framework of an intentional cop. results this section discusses the findings of the research into the impact on the professional development of the participants resulting from the design of the slt course around an experiential cop. 2010 participant profile the bulk of the participants in the course were from the vocational training departments at unitec, including: boat building, automotive, carpentry and electrical trades. the students were skilled tradesmen, but not necessarily skilled teachers, and most had limited experience of integrating technology into their teaching practice, but were keen to explore the potential beneficial impact for their students. the participants’ ages ranged from 29 to 59, with an initial enrolling cohort of seven male and two female participants. table 1. key differences in the redesign of the social learning technologies course old lt course new slt course design prescribed course resources (book and printed journal articles provided to learners in class) open � students determine appropriateness of the content according to discipline, their own contexts and learning technologies chosen only theory applied theory exploring potential use of learning technologies exploring potential use of technology and applied within the learner’s own context facilitation focus on individuals in class (learning alone) focus on the community and the role the individuals play in the community (learning together � collaboration, co-creation, peer-feedback and communication) emphasis on strategies for delivery of content (passive learning strategies) emphasis on active learning, learner-generated content and authentic learning learning context control by the teacher learning context determined by the needs on the community and individuals assessment two separate assessments assessments embedded within the learning process, each building on the other department involved in teaching the course lecturers from the education department academic advisors from te puna ako (learning and teaching development unit) t. cochrane and v. narayan 162 student surveys the beginning of course student survey provided data on students’ previous experience. figure 2 indicates that while the slt participants all had computer and internet access, and the majority owned a cellphone, most of their web experience had previously been as consumers of information and media rather than producers. there was minimal use of interactive web 2.0 technologies prior to the course, with those that were already engaging in web 2.0 having previously worked with the authors on projects. the students’ responses to the end of course survey were overwhelmingly positive about their experience of mobile web 2.0 during the course. transformational journeys the key goal of the course was for the lecturers to model the pedagogical use of mobile web 2.0 tools embedded within an intentional cop comprised of the course lecturers and the course students. the course students were then guided to apply their experience to create a personal framework for authentic experiential learning within their own teaching contexts. this represented a significant process of reconceptionalising the participants’ notions of identity and agency within teaching, i.e. an ontological shift. for many lecturers this will require an ‘ontological shift’ in their understanding of what it means to teach, and can represent a fundamental challenge to the lecturer’s understanding of self within the context of the nature of teaching and learning. an figure 1. outline of the slt course. alt-c 2011 conference proceedings 163 ‘ontological shift’ is ‘‘the re-assignment or re-categorising of an instance from one ontological category to another’’ (chi and hausmann 2003, 432), or simply put, a reconceptualisation. this shift involves a reconceptualisation of lecturers’ understanding of teaching and learning from their prior experience to understandings built upon the foundation of learning theory such as social constructivism. this ontological shift can take significant time as lecturers reconceptualise and develop new and appropriate forms of assessment, collaboration and communication strategies. for several of the course students the course facilitated an ontological shift from tradesman to teacher. examples of the impact of the slt course on participating students are discussed in the following sections. boat building lecturer 1 this participant became a key peer mentor and driver for the group. he helped to establish a real sense of community, encouraged the group to try and contextualise their learning, and he modelled collaborative discussion and critique using a range of technologies. for example, he initially experimented with creating personal reflective vodcasts and then extended the concept to establish skype video call discussions between the slt students, screen captured these, and shared them on youtube as examples of critical reflection upon the theoretical pedagogical frameworks (http:// www.youtube.com/watch?v�bplyqirsvhu). the social collaboration built into the slt course was very important for the participant’s transformational journey, as he expressed in a blog post, contextualised using boating terminology: the fog is still at sea level. but i’m hearing others sounding off, so there is hope out there. some are still at a distance but i can feel that others are close by. i think at last i’m starting to get my mind around what links might look like. the links i’m starting to see figure 2. slt students’ previous technology experience. t. cochrane and v. narayan 164 are those that are between emerging learning technologies (web 2.0 and stuff) and social learning theories. (slt student blog post 2010) the experience of the slt course impacted this lecturer’s own teaching practice by enabling him to form a theoretical foundation for his approach to teaching based upon social constructivism that he has explicitly implemented with his students in 2011 http://www.youtube.com/watch?v�qojeggkvygw. carpentry lecturer 1 this participant synthesised his experience as a student on the slt course and his own teaching practice to create innovative ideas for use with his own students. his goal in participating in the slt course was to explore how to more closely link the theory and practical components of his carpentry course by getting his students involved in capturing, sharing and critiquing their practical on-site work via short videos recorded on their camera phones and uploaded to their blogs. he enjoyed the experience of the course: this has been a very interesting course and i have gained a lot from my peers, vickel, thom and the readings. also getting the chance to use the iphone has been a real learning curve and an eye opener to what we could possibly achieve with our students and some interesting thoughts about empowering student ownership and responsibility. i have enjoyed experimenting with different web 2.0 tools and having the opportunity to participate as a student and also facilitation possibilities from a teacher’s perspective. (slt student blog post 2010) by the end of the course this participant also demonstrated a new level of critical pedagogical reflection: key new knowledge gained for me is vygotsky’s zone of proximal development and the fundamentals that almost feel specific to our learners, although i realize it is for a wider community. this is an important aspect to our frame work, understanding where we are, where we need to be, and what we can build on to eventual empowering students negotiation and enquiry. (slt student blog post 2010) the experience of the slt course impacted this lecturer’s own teaching practice in 2011 by enabling him to conceptualise ways of integrating mobile web 2.0 tools into the context of bridging the theory and practice of building onto the building site with his students. this led to the design and building of a portable ‘eshed’ for theory lessons on site http://www.youtube.com/watch?v�-tedxhcv-4w. boat building lecturer 2 this participant began the slt course with the least previous experience of computing and web 2.0 of all of the 2010 participants. initially he was dubious of the benefit or applicability of mobile web 2.0 to his teaching context. however, during the process of investigative reading around theoretical frameworks for educational technology, he experienced a ‘eureka’ moment: a dawning of how the combination of reading social constructivist theory, his slt experience, and his previous teaching alt-c 2011 conference proceedings 165 http://www.youtube.com/watch?v=qojeggkvygw http://www.youtube.com/watch?v=qojeggkvygw http://www.youtube.com/watch?v=qojeggkvygw http://www.youtube.com/watch?v=qojeggkvygw http://www.youtube.com/watch?v=qojeggkvygw http://www.youtube.com/watch?v=-tedxhcv-4w http://www.youtube.com/watch?v=-tedxhcv-4w http://www.youtube.com/watch?v=-tedxhcv-4w http://www.youtube.com/watch?v=-tedxhcv-4w http://www.youtube.com/watch?v=-tedxhcv-4w experience aligned to create a deeper understanding of teaching and learning. the participant reflected upon what brought about this eureka moment in a blog post: where did the learning finally happen? was it in a societal environmental? you bet it was, the daily collegiate banter between colleagues in the slt group and staff that just get into it, with lunchtime discussions, items of interest being distributed freely, online bog posts from a variety of educationalists and tutors, suggested readings that then promoted surfing wider topics and views, all had a hand in it. has web 2.0 tools played a role? of course. (slt student blog post 2010) following this experience this participant became an educational technology evangelist, to the point of buying his own ipad and iphone, and presenting his transformational journey using his brand new ipad at a subsequent minisymposium organised by the researchers (http://www.youtube.com/watch?v�zgequkzzmyu &feature�feedf). the experience of the slt course impacted this lecturer’s own teaching practice by providing him with a foundation to conceptualise how his own students could utilise ipod touches to record and document their learning via blogbased eportfolios in 2011. discussion while the number of participants in the 2010 slt course was small with a 2010 cohort of six students (although average for the gdhe courses in general), the results are indicative of those observed by the researchers’ throughout over 30 mlearning projects using the developed intentional cop support model between 2006 and 2010. the slt course serves as an example of the impact of mobile web 2.0 integration supported by cops involving over 50 lecturers, from 13 different departments at unitec. the authors redesigned the course around a social constructivist pedagogy that leveraged several emergent learning frameworks. creating the foundation and circumstances for pedagogical transformation was the goal. this transformation is aptly described by the learner-generated contexts group and the concept of bridging the pedagogy-andragogy-heutagogy (pah) continuum. luckin et al. (2010) argue that heutagogy (student-directed learning) need not be the domain of post-graduate research students only, and propose the concept of learner-generated contexts as a framework to help achieve this. garnett (2010) describes the process of this transformation of lecturer’s reconception of pedagogy in three steps following the pah continuum: moving from pedagogy (teacher-directed) to andragogy (studentcentred, student-generated content), and towards heutagogy (student-directed or negotiated learning). (1) the ability to understand how to use their subject for teaching, that is an effective pedagogy? (2) to understand how to manage the learning environment they are working in and treat each learner as an individual, that is the andragogy of learning relationships (3) then having learnt how to manage the learning process related to their subject they then turned their control over to their learners, enabling the heutagogy of creativity to kick in (garnett 2010) achieving this reconception takes significant time, involving sustained engagement. t. cochrane and v. narayan 166 sustained engagement leading to ontological shifts the case study illustrates that creating sustained engagement around the integration of mobile web 2.0 tools supported by cop can facilitate ontological shifts among the participants. two key issues around reconceptualising teaching and learning representing ontological shifts in the participants’ understanding were identified: (1) shifting lecturers from pedagogy to heutagogy, reconceptualising teaching as proposed by luckin et al. (2008, 2010) and mcloughlin and lee (2008). (2) shifting students beyond their previous experience, reconceptualising learning, and using the mobile web 2.0 tools to engage students via a focus upon student-generated content and student-generated contexts. there were certain elements of the slt course that the participants found harder than others. for example: the participants took a while to get used to using correct referencing and bibliographic tools, particularly within the context of blogging. this was important to underpin the course experience with graduate level critical thinking. some students took a while to get into the swing of using twitter for communicating, with several ‘lurking’ until a momentum developed, and then they became quite engaged by using twitter once a community had been established around its use in the course, effectively moving from legitimate peripheral participation to full participation in the core of the cop. the ‘intentionality’ of the slt community of practice was embedded in the course design and assessment activities, with the authors purposely building the course as a learning experience. in contrast to an organic cop active participation in the course cop was mandated as an assessed activity. however, this intentional cop kick-started the participants’ experience of cop formation, and has led to the organic development of a continued cop of the course graduates. as the majority of 2010 slt students were located within the same faculty, these slt graduates have continued to build their own cop after the end of the slt course, inviting their peers to join this cop. the 2010 graduates have also taken a keen interest in the 2011 iteration of the course: joining in twitter conversations with the 2011 participants, and offering links to resources and even technology support for the 2011 cohort, effectively becoming brokers of their own transformational journeys. participant feedback informing 2011 implementation feedback was gathered from a variety of sources from the 2010 participants, including: analysis of participants’ blog posts, a face-to-face debrief between each participant and the course lecturers at the end of the course, final student surveys and feedback elicited by an independent course reviewer after the course had finished via email and personal phone call interviews with participants. feedback indicated that some participants initially felt a bit thrown in the deep end with the new learning experience represented by the slt course and the embedded use of mobile web 2.0 tools. however, by the end of the course, feedback from the students indicated that they were ‘‘no longer fearful’’ of trying new technologies. some participants suggested adding extra scaffolding of the mobile web 2.0 tools via extra drop-in tutorials (these were offered during the course, but no one took up the offer). bridging the other gdhe courses into the slt course was also alt-c 2011 conference proceedings 167 suggested. the integration of elements of the slt course throughout the rest of the gdhe is one of the goals of the authors. limitations as an assessed course, the researchers attempted to model an intentional cop as far as possible without the assessment becoming the core driver for participation. the slt course was designed to provide students with an experience of social constructivist learning, underpinned by reflection upon sound pedagogical theory, and enabled by mobile web 2.0 technologies. as such we (as the ‘teachers’) of the course attempted to model this approach in our facilitation of the course, for example: we used alternative web 2.0 tools for in class presentations including prezi (http://www.prezi.com), we used web 2.0 communication tools such as twitter for remote and in-class brainstorming, and we modelled the pedagogical use of blogs and moblogging in our own practice. these helped the students conceptualise how to use these tools in their own practice. however this generally required significant time and reflection by the students, for whom the ‘lights came on’ near the end of the course. we also allowed a certain amount of negotiation with the students around the course goals and assessment activities (as far as the redesigned course descriptor would allow) � allowing the cop that developed to be unique to the participants, which students tended to find a new experience. conclusions the slt course demonstrates the transformative impact of a cop model of lecturer professional development. the 2010 course graduates have now become technology stewards within their own departments, effectively drawing in their peers from the periphery of the slt community of practice and forming spin-off cops within their own departments. scaffolding the integration of mobile and social technologies within the slt cop involved a range of approaches, including modelling by technology stewards, peer mentoring and the utilisation of flexible technologies beyond the face-to-face contact. as nicholas cage stated ‘‘now you know how it feels, you won’t be scared’’ (vaughn 2010). references brown, j.s. 2006. new learning environments for the 21st century: exploring the edge. change, (september/october): 18�24. chi, m., and r. hausmann. 2003. do radical discoveries require ontological shifts? in international handbook on innovation, ed. l. shavinina and r. sternberg, vol. 3, 430�444. new york: elsevier science ltd. cochrane, t. 2007. moving mobile mainstream: using communities of practice to develop educational technology literacy in tertiary academics. paper presented at the mlearn 2007 � making the connections 6th international conference on mobile learning, october 16�19, in melbourne exhibition centre, melbourne. cochrane, t. 2010. beyond the yellow brick road: mobile web 2.0 informing a new institutional elearning strategy. special issue ‘the transformational impact of learning technology’ of alt-j. research in learning technology 18, no. 3: 221�31. cochrane, t., and g. kligyte. 2007. dummies2delight: using communities of practice to develop educational technology literacy in tertiary academics. paper presented at the jisc online conference: innovating elearning, jisc online conference, june 11�14. t. cochrane and v. narayan 168 engestrom, y. 1987. learning by expanding: an activity-theoretical approach to developmental research. helsinki: orienta-konsultit. garnett, f. 2010. heutagogy and the craft of teaching. the heutagogic archives (november 18). http://heutagogicarchive.wordpress.com/2010/11/18/heutagogy-the-craft-of-teaching/ #more-340 (accessed november 19, 2010) head, g., and j. dakers. 2005. verillon’s trio and wenger’s community: learning in technology education. international journal of technology and design education 15: 33�46. herrington, a., and j. herrington. 2007. authentic mobile learning in higher education. paper presented at the aare 2007 international educational research conference, fremantle, australia. herrington, a., j. herrington, l. kervin, and b. ferry. 2006. the design of an online community of practice for beginning teachers. contemporary issues in technology and teacher education 6, no. 1. http://www.citejournal.org/vol6/iss1/general/article1.cfm (accessed july 15, 2010) herrington, j., and r. oliver. 2000. an instructional design framework for authentic learning environments. educational technology research and development 48, no. 3: 23�48. langelier, l. 2005. working, learning and collaborating in a network: guide to the implementation and leadership of intentional communities of practice. quebec city: cefiro (recherche et études de cas collection). laurillard, d. 2001. rethinking university teaching: a framework for the effective use of educational technology, 2nd ed. london: routledge. lave, j., and e. wenger. 1991. situated learning: legitimate peripheral participation. cambridge: cambridge university press. luckin, r., w. clark, f. garnett, a. whitworth, j. akass, j. cook, p. day, n. ecclesfield, t. hamilton, and j. robertson. 2008. learner generated contexts: a framework to support the effective use of technology to support learning. http://api.ning.com/files/ij6j7ucs b9vgb11pkphu6lkmgqqkr-ydvnxrui9tbgf1q-esyudv-mil6uwqx4f1jya1puk zrxvbxhnxuhusyl1lrxvrbkno/lgcopencontextmodelning.doc (accessed november 5, 2008) luckin, r., w. clark, f. garnett, a. whitworth, j. akass, j. cook, p. day, n. ecclesfield, t. hamilton, and j. robertson. 2010. learner-generated contexts: a framework to support the effective use of technology for learning. web 2.0-based e-learning: applying social informatics for tertiary teaching, ed. m. lee and c. mcloughlin, 70�84. hershey, pa: igi global. mcloughlin, c., and m. lee. 2008. mapping the digital terrain: new media and social software as catalysts for pedagogical change. paper presented at the ascilite melbourne 2008. siemens, g. 2004. connectivism: a learning theory for the digital age. elearnspace (12 december). http://www.elearnspace.org/articles/connectivism.htm (accessed december 31, 2009) vaughn, m. (writer). 2010. kick-ass [film]. in a. bohling (producer). london, england: universal. vygotsky, l. 1978. mind in society. cambridge, ma: harvard university press. wenger, e. 1998. communities of practice: learning, meaning, and identity. cambridge: cambridge university press. wenger, e. 2005. communities of practice: a brief introduction. (17 july). http://www. ewenger.com/theory/index.htm (accessed october 14, 2006) wenger, e., n. white, and j. smith. 2009. digital habitats: stewarding technology for communities. portland, oregon: cpsquare. wenger, e., n. white, j. smith, and k. rowe. 2005. technology for communities. in working, learning and collaborating in a network: guide to the implementation and leadership of intentional communities of practice, ed. l. langelier, pp. 71�94. quebec city: cefiro. alt-c 2011 conference proceedings 169 http://heutagogicarchive.wordpress.com/2010/11/18/heutagogy-the-craft-of-teaching/#more-340 http://heutagogicarchive.wordpress.com/2010/11/18/heutagogy-the-craft-of-teaching/#more-340 http://www.citejournal.org/vol6/iss1/general/article1.cfm http://api.ning.com/files/ij6j7ucsb9vgb11pkphu6lkmgqqkr-ydvnxrui9tbgf1q-esyudv-mil6uwqx4f1jya1pukzrxvbxhnxuhusyl1lrxvrbkno/lgcopencontextmodelning.doc http://api.ning.com/files/ij6j7ucsb9vgb11pkphu6lkmgqqkr-ydvnxrui9tbgf1q-esyudv-mil6uwqx4f1jya1pukzrxvbxhnxuhusyl1lrxvrbkno/lgcopencontextmodelning.doc http://api.ning.com/files/ij6j7ucsb9vgb11pkphu6lkmgqqkr-ydvnxrui9tbgf1q-esyudv-mil6uwqx4f1jya1pukzrxvbxhnxuhusyl1lrxvrbkno/lgcopencontextmodelning.doc http://www.elearnspace.org/articles/connectivism.htm http://www.ewenger.com/theory/index.htm http://www.ewenger.com/theory/index.htm doi:10.3402/rlt.v19s1/7802 implementation and sustainability of a global ict company’s programme to help teachers integrate technology into learning and teaching in germany, france and the uk danny arati a *, albena todorova b , and ruth merrett a a corporate affairs, intel corporation, swindon, uk; b psychology of excellence in business and education, ludwig-maximilians-university, munich, germany (received 2 june 2011; final version received 20 june 2011) this paper will discuss the implementation of the professional development programme ‘‘intel† teach’’ in germany, the uk and france, as a public�private partnership. the programme is designed to help school teachers to effectively integrate technology into learning and teaching and to help students develop key ‘‘twenty-first century skills’’. the implementation of the programme, which has so far involved over 400,000 teachers spread across the three countries since 2004, followed different models in the three countries, as a result of differing national education policies, systems and needs. data from the external evaluation of the programme in germany are used to examine the factors on a systemic level, which affected the implementation, effectiveness and sustainability of the programme. these factors are grouped into three categories: (1) concept transfer, (2) experience transfer and (3) establishing standards. exploring these factors provides a framework for analysing how the changing conditions in the three countries and the global trends in education will influence the further development of the programme. recent developments in the programme will be discussed, including: . open-source solutions . integration of new features (including e-portfolio); . improved collaboration and sharing. the paper will also address the introduction of new content and approaches that target specific current issues in teacher professional development, for example: . project-based approaches; . collaboration in the digital classroom; . technology-based approaches to assessment; . educational leadership. the experience from the implementation of the programme through public� private partnerships in different countries shows how such collaborations can shape the educational landscape in a way that makes educational provision more effective and efficient, and of greater relevance and value to students. keywords: teacher professional development; technology integration; sustainability; public�private partnership *corresponding author. email: danny.arati@intel.com alt-c 2011 conference proceedings 0160 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # d. arati et al. this is an open access article distributed under 170 the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http:// creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7802 introduction recent policy developments towards e-readiness and e-skills have confronted national education authorities with the question of how to effectively foster teachers’ skills in integrating information and communication technologies (ict) in teaching and learning processes. education authorities attempted measures such as implementation of ict-related policies, introduction of ict certification for teachers and provision of pre-service and in-service training in the technical and pedagogical use of ict. despite these measures, recent evidence in international perspective showed that ict teaching skills is one of the areas with the greatest need for professional development (oecd 2009). this finding reveals a necessity to gain a better understanding of the complex set of factors, which determine the long-term effects of programmes and initiatives aiming to enhance teachers’ ict competencies. professional development literature categorises the factors which mediate the impact of programmes and initiatives, according to different criteria. ottoson (1997) distinguishes between educational, innovating, predisposing, enabling and reinforcing factors affecting teachers’ adoption and application of what they have learned during their professional development courses. a broader categorisation of the factors influencing the impact of professional development programmes places them into two main groups � individual (teacher) and school, programme and system factors (smith and gillespie 2007). system-level policies and practices are considered to influence indirectly the effectiveness of teacher professional development to integrate technology (darlinghammond and mclaughlin 1995). such policies and practices can promote and legitimatise particular professional development programmes and their goals, as well as enhance or inhibit the ability of schools to support them (smylie et al. 2001). education authorities can positively affect the implementation and effectiveness of professional development by promoting professional learning, by ensuring consistency between system-level policies and school-level actions (elmore and burney 1999), by providing human, financial and material resources and by establishing supportive policies (darling-hammond and mclaughlin 1995; youngs 2001). national ict strategies have already prioritised the use of technology in education in many countries (adamali, coffey, and safdar 2006). however, the successful implementation of such strategies depends to a large extent on large-scale governmental efforts which are sustained over time (voogt and knezek 2009). in addition to policies, education authorities can influence technology professional development programmes in various ways, which are specific in every case. this is particularly true when such professional development programmes are implemented in public�private partnerships (ppp) � a model becoming more accepted and frequent as the technology industry takes a more proactive role in ict integration in education. exploring the factors at educational system level which influence the successful implementation and sustainability of a particular professional development programme sheds light on the existing barriers and facilitators. this paper will discuss findings from the evaluation of the large-scale professional development programme aimed at integrating technology in the classroom ‘‘intel† teach � advanced online’’, and particularly the identified system-level factors for its implementation and sustainability. furthermore, the paper will review the current developments in the implementation of the programme in relation to the identified factors and educational contexts in germany, france and the uk. alt-c 2011 conference proceedings 171 intel teach � advanced online the teacher professional development programme ‘‘intel teach � advanced online’’ is one of the projects designed and implemented within the intel† education initiative of intel corporation for the advancement of education through technology. along with environment and community development, education is one of the focus areas of the company’s social responsibility actions and engagement with social issues. as a technology company, intel’s success rests on the availability of skilled workers, a healthy technology ecosystem and knowledgeable customers. therefore, intel strategically invests in improving education globally, partnering with educators, governments, and other companies to develop a range of transformative programmes and technology solutions. intel has over 200 programmes in 70 countries that provide professional development for teachers, support student achievement in science, technology, engineering and math, and enable access to relevant digitised content. one of the main components of the initiative is the intel† teach 1 programme � a programme for professional development aimed at training classroom teachers to effectively integrate technology in instruction to enhance student learning. the programme was developed in collaboration with ministries of education and educational institutions. since 1999 it is provided to elementary and secondary school teachers around the world and encompasses a portfolio of courses targeting different aspects of integrating technology in classroom teaching. ‘‘intel teach � advanced online’’ is one of the offerings within the intel† teach programme, developed in germany in 2004 in co-operation with the academy for teacher training and school management in dillingen � a teacher training centre run by the ministry of education in bavaria (ganz and reinmann 2007). the professional development programme was developed following the successful implementation of a basic course for technology use for teaching and learning offered within the intel teach programme in germany. the implementation of the advanced programme was organised through individual arrangements with the ministries of education in every federal state in germany. this lead to different models and intensity of teacher participation in the programme. however, the main structure and the content were delivered in the same format across states. the programme was subsequently localised, and a new version of the programme supported by a moodle-based online platform was implemented in england, france, ireland, israel, italy, jordan, spain and sweden. intel teach � advanced online is based on a blended learning format of face-toface meetings and individual and collaborative learning. the programme is organised through an online platform, designed to support and drive all steps in the process and to enable self-paced on-the-job professional development. the delivery is organised through ‘‘train-the-trainer’’ approach, in which senior trainers are trained in advance and subsequently guide and support regional mentors (master teachers), who train and assist the participating teachers in the programme. supported by the mentor, the participants form teams and choose a pedagogical approach or technology tool to learn about. subsequently, the participating teachers work collaboratively to develop a unit plan, implement it in their classroom practice, evaluate and enhance it for further use. d. arati et al. 172 evaluation in the period from 2005 to 2008, the programme ‘‘intel teach � advanced online’’ in germany was externally evaluated by the institute for media and educational technology in the university of augsburg. the first phase of the evaluation aimed to determine the direct effects of the training. the evaluation findings are reported in detail elsewhere (ganz and reinmann 2007). overall they show that the programme had a positive impact on teachers’ skills to integrate technology in the classroom, on their attitudes towards technology-enhanced and student-centred learning and on their practices of technology use. according to the self-assessment of the participating teachers, the programme improved their technical and methodological competencies for using digital technology in instruction. as a result of their participation in the programme, teachers reported that they had a lot of new ideas to use digital media in teaching and that they increased their confidence to use new media in the classroom, and their appreciation for self-evaluation and collaboration. regarding the effect of the programme on students, teachers reported improvements in students’ skills for using digital media in terms of handling applications and using technology to reach the class objectives, and increase students’ use of digital media for individual learning at home. it was indicated that using technology in class had a positive effect on students’ motivation and collaboration, and on students’ active, self-regulated learning in terms of generating more own ideas to reach the objectives of the lesson and raising questions. during the second phase of the evaluation, the institute for media and educational technology in the university of augsburg conducted research to determine the external factors influencing the success of the programme and its sustainability. the sustainability of the programme was examined through case studies of 16 schools in four federal states in germany (häuptle, florian, and reinmann 2008). target population were teachers in the schools who participated in the programme (n�40), teachers who did not participate in the programme (n�24) and headteachers (n�15). the chosen federal states represented different policies of federal education authorities regarding the use of mentors for the implementation of the programme. thus, states supporting (bavaria, thueringen) and not supporting (rheinland-pfalz and hamburg) regional and region-wide mentorship concepts were included. in order to explore how the education policy of a federal state in germany and the implementation approach of the senior trainers influence the sustainability of the programme, the evaluators conducted group discussions with 12 senior trainers. further findings from questionnaires filled in by the senior trainers during the evaluative period 2006 were also included. the results and implications from the qualitative analysis of the collected data were validated in discussion with nine senior trainers within a workshop in october 2007 and in written commentaries by five senior trainers. the report from the evaluation, including a detailed description of the method and all findings regarding school-level and system-level factors, is available online in the german language (häuptle, florian, and reinmann 2008). the current paper focuses on the results concerning the system-level factors and how they relate to the implementation of the programme in the context of other education systems. alt-c 2011 conference proceedings 173 system-level factors influencing the implementation and sustainability of the programme the findings from the external evaluation provide evidence for the system-level facilitators and barriers to the sustainable implementation of the programme intel teach � advanced online. factors at the level of the school system were grouped into three categories: (1) concept transfer, or dissemination and transfer of the professional development; (2) experience transfer, or learning from experience and lessons learned; and (3) establishment of standards. factors supporting the dissemination and transfer of professional development according to häuptle, florian, and reinmann (2008), system-level sustainability of a professional development programme is demonstrated by the extent to which the professional development concept is accepted and implemented by different schools in an administrative region. the implementation of the programme is facilitated when the authorities provide support for it, integrate it in the structure of professional development and are directly engaged in its promotion. according to the senior trainers, educational policy developments in the federal states can both benefit and hinder the dissemination of professional development. factors for the sustainable implementation of the programme in this group are related to educational policy developments, the mechanisms for reaching teachers and for motivating them to participate in the professional development offering, as well as communicating the objectives and nature of the programme effectively (table 1). as an example, curricula provide for the integration of digital media in subject teaching, and increase the openness and readiness of teachers to participate in the programme. in some federal states, the policies allowed greater autonomy for schools, which was used by senior trainers to align the professional development programme to increase personal skills and promote teaching standards and school development. external influence in the form of evaluation and standards positively affected the dissemination and transfer of the professional development programme. in the federal states where the quality of teaching and of the school was assessed through external evaluation, the consequent recommendations for improvement were incorporated into the content offered by the advanced programme. at the same time, the programme included content relevant to some national educational standards, and the implementation of the education standards presented an incentive for taking part in the programme. one of the impeding factors for the sustainability of the programme was the low value associates with the classroom use of digital media in school policies. in such cases, after the end of a media project the related activities and innovations are discontinued. in some cases authorities preferred customised professional development offerings, whereas the programme ‘‘intel teach � advanced online’’ was considered as supplemental. further factors on the level of the school system were related to the mechanisms for reaching schools and teachers. informing teachers about the offering of ‘‘intel teach � advanced online’’ and stimulating their interest in participating in the programme were improved when senior trainers provided clear information on how the programme could fulfil a task prescribed by education policies. such positive influence was also observed when the programme was offered in flexible formats, d. arati et al. 174 table 1. factors with positive and negative effect on the dissemination and transfer of the professional development programme intel teach � advanced online in germany. factors influencing the dissemination and transfer of professional development positive negative factors related to educational policy developments . digital media is a component of the curriculum and syllabus . policies place low value on the use of digital media in teaching . greater school autonomy . high workload due to restructuring of the work . external evaluation for quality development . preference for customised offerings . establishment of educational standards factors related to reaching the target population . demand due to obligation for professional development alignment of the programme with the objectives for professional development of the education authorities . objectives of the programme do not represent current school policy of the education authorities . flexibility in implementation . mentors integrate different functions . informing teachers through various channels . appealing to teachers not interested in technology . activating positive experiences factors related to the programme presentation . stimulate interest with presentation of specific content: teaching scenarios; connecting content to the current interest of teachers; introducing the online platform . the frequency and place of information meetings is regulated by demand . sufficient preparation and support of the school mentors for their role . insufficient staff to organise programme presentations . absence of mentors a l t -c 2 0 1 1 c o n fe re n c e p ro c e e d in g s 1 7 5 allowing for different arrangements with educational authorities and for structuring of the introductory events and communication with schools. using different channels to inform teachers about the programme was also found to be advantageous, especially when the offering was presented through official communications of the education authorities and professional development institutes, such as direct mail, newsletters and portals, through public relation releases and information events. additional positive impact was attributed to effective ways to approach teachers who were not interested in using technology in class. reaching this group of teachers could be accomplished through indirect pressure through teaching quality control, such as school evaluation practices. another method was for senior trainers to clearly communicate the goals of the programme and the advantages of participation, removing the negative attitudes towards technology. in some cases, senior trainers integrated different functions, such as working on education standards, being in charge of school ict equipment or media adviser, or being school principals or teachers, which facilitated establishing contact with teachers and increasing their receptiveness and acceptance of the programme. according to senior trainers, willingness to participate in the advanced course was affected for some teachers by prior positive experiences with ict basic courses, and by recommendation from other teachers. another set of factors influencing the dissemination and transfer of the professional development programme was related to the structuring of the information events, during which the programme was introduced to school mentors and teachers. such information meetings were effective when the benefits of participating in the programme were exemplified with specific teaching scenarios and demonstration of the online platform, and when the content was related to currently interesting topics for teachers, e.g. how the programme supports teachers in the implementation of school policies regarding education standards, media competencies, self-evaluation and others. an important feature within the successful meetings was the preparation of school mentors to present the programme. a lack of support by senior trainers and a lack of school mentors limited the potential of information meetings to motivate teachers to participate in professional development courses. factors supporting learning from experience and lessons learned in the context of experience transfer within intel teach � advanced online, sharing of practices between schools, however, was not supported in the federal states due to the school-based implementation of the programme. an annual symposium in dillingen and further meetings of the senior trainers at education fairs (didacta systems) facilitated the exchange of experiences and ideas, and made it possible to discuss how to take things forward. factors supporting the establishment of standards the last set of factors on system level is related to the establishment of standards, which refers to the systematic implementation of professional development requirements and technology integration policies within and across federal states. due to the differences d. arati et al. 176 in federal policies and conditions in schools, the establishment of standards could not be an influence to improve the sustainability of the programme. the outlined factors on the level of the school system bear the specific characteristics of the federal state structure of the education system in germany. thus, the influence of different policies and mentor support on the implementation of the programme could be observed. in comparison, the delivery of the programme in france and the uk demonstrates different approaches to offering the same concept for professional development to teachers. implementation of the programme in different contexts: the role of system-level factors ‘‘intel teach � advanced online’’ was developed originally in germany with the aim of being implemented in the country. the consequent introduction of the programme to other national states through public�private partnerships required more than simply localising the model and content for the different national contexts, but also tailoring the delivery mode according to the specifics of the education systems and the cooperation partners involved. france in france, the training of teachers is highly decentralised in regional academies but the organisational and administrative framework is set nationally. the project ‘‘pairform@nce’’, within which the concept of the professional development programme ‘‘intel teach � advanced online’’ was implemented in france is driven nationally by the department for teaching curriculum, teacher training and digital development in education. the project combines different partners and currently includes all 30 academies � regional structures of the ministry of education in charge of implementing national directives and policies. the objectives of the programme are in reference to the it certificate c2i level 2 (c2i2e) for teachers, which aims to validate the professional skills required by all teachers to perform the pedagogical, educational and societal aspects of their job. for the implementation of the programme, intel provided the royalty-free programme, and has facilitated the linking with the different national and european partners. the implementation of the programme in france is fully integrated in the national and regional education policies and structures, through the authority of the regional teacher training academies. the dissemination and transfer of the professional development concept and the transfer of experiences is facilitated by the support provided by the central education authorities. thus, the training is implemented on a remote and dedicated national virtual environment for learning featuring a wide range of training courses, educational resources, information and discussion tools. the content of the resources is developed in the ‘‘factory’’ by volunteer teachers under the education authorities, based on needs identified by inspectors and school principals. the reference of the programme to the national it standards for teachers and the support for the delivery of the programme are major factors for the current successful implementation of ‘‘pairform@nce’’ on a large scale in france (soury-lavergne et al. 2010). the plan is to expand pairform@nce usage to other disciplines where it is not more than a tool. alt-c 2011 conference proceedings 177 united kingdom in the uk, at the time that intel teach was implemented, the government’s e-strategy aimed to transform the management and interaction capacities of educational institutions for the benefit of learners and parents (becta 2007). schools are controlled for use of the technology, trainee teachers are required to pass a test in ict literacy, whereas practicing teachers are expected to improve their own ict skills to the same level. the intel teach � advanced online programme is implemented in the uk in partnership with the specialist schools and academies trust (ssat) � an independent, charitable trust with a network of over 5600 schools and organisations. the trust works with head teachers, teachers and students for developing and sharing new and effective teaching and learning practice, with the object to raise standards and levels of achievement of schools. the programme was offered under the name ‘‘icpd online’’ through ssat, which determines the way of dissemination, transfer of experiences and reference to standards. the total number of teachers trained under icpd online is just over 36,000. the programme has now developed in the uk and the ssat have decided to implement the icpd tool into their developing leaders training. this model is quite different and allows teachers to be more autonomous in their usage of the tool but still maintains the benefits of peer reviewing and submitting content online. this has been implemented in the latest cohort of developing leaders from september 2010 and at the end of the 2010/11 the ssat and intel can evaluate the model and its outcome. the uk has benefited from this open approach to the implementation of intel teach advanced online and has been able to make it integral to other programmes which could be a more sustainable model. the different models of public�private partnership in the three countries, the level of integration of the programme in the education policies and structures and the amount and type of support by education authorities, but also the current conditions and trends, determine to a large extent the new developments and changes in the programme’s design and delivery. current developments the programme is currently being expanded and transformed into an offering with increased flexibility and is taking into account the preferences of education authorities, as well as factors at the level of the education system, which affect the implementation and sustainability. some of the new features of the programme enable better collaboration with education authorities, organisations and other companies. a major change is the development towards open-source solutions and flexibility of the new platform of intel teach � advanced online, which is a customised application built on the moodle 1.9 platform. the codebase is written in php and supports the mysql or postgresql databases, using moodle’s standard database abstraction layer. the new platform also enables the integration of external software. thus, local education authorities can provide teachers with all training opportunities available in the region through a single platform or integrate the programme with other trainings to meet specific needs. this is particularly useful in the context of the different policies and requirements in every federal state in germany and for the implementation of the programme in france through regional education authorities. d. arati et al. 178 another development includes expanding the open education resources available to teachers, to meet their specific needs for professional development. a new offering of free, just-in-time courses � intel† teach elements, target specific learning concepts, such as project-based learning, assessment of twenty-first century skills, or collaboration with web 2.0 tools. additional resources are provided to teachers for self-directed learning and classroom use, such as ms office courses, different tools and the learning resources on skoool tm interactive learning and teaching technology programme. the new programme also provides an improved environment for collaboration between teachers through the collaboration features of moodle, the integration with external platforms for communication and collaboration, such as live@edu in germany, and through the added functionality of e-portfolio. a key new feature is the concept of e-portfolio and, for this purpose, the new platform can be bundled with the mahara e-portfolio application, which adds resource sharing and additional social networking elements to the platform. mahara is an open source system comprising electronic portfolio, weblog, resume builder and social networking system, which connects users and creates online communities. a different eportfolio system is integrated with the programme in the uk, within the virtual leadership & innovation academy on the ssat online platform. this ‘‘active portfolio’’ will be driven by a dynamic profile, smart use of meta-data and automatically harvested evidence of achievement. after a revision of the significance of collaboration between teachers and the influence of tutors, a different concept is implemented in the new version of the programme � peer coaching. it is expected that this additional support will enhance teachers’ acquisition of competencies and skills, according to their individual needs. this can also be addressed by the introduction of e-portfolios as part of the professional development, as a mechanism to identify gaps, track development and find peers with relevant knowledge and skills for coaching and collaboration. further possibilities for interaction between teachers and for forming a virtual community of practice or community of professional learning will contribute to the impact of the programme on teaching practices and on the sustainability of the programme. conclusion it has been shown that there are various factors on the level of the education system to be considered, which influence the effectiveness and sustainability of a professional development programme for technology integration. the different models of implementation of intel teach � advanced online in germany, england and france are linked to different expectations for the reach and scaling of the programme, according to the national policies and standards, and the support by education authorities. some of the challenges for the current and future implementation of the programme are how to align the programme to changing policies and standards, and target hot issues in national education systems with flexible content and modes of implementation. these challenges are partly addressed in the new version of the programme, organised through a moodle-based online platform, which offers more flexibility and improved capacity for online visibility and for reaching and engaging teachers. the development towards open-source solutions in this case reflects the preference of education authorities for cheaper, more flexible and easy to customise training system. additionally, linking the programme to other initiatives, school and alt-c 2011 conference proceedings 179 teacher networks can increase the alternatives for communicating the content and objectives of the programme and for dissemination. identifying the factors which contribute to or impede the effectiveness and sustainability of a professional development programme allows for planning to strengthen the beneficial conditions and to avoid some negative conditions. understanding such factors can contribute to improving the programme and achieving a greater impact both on the level of the individual schools and on the of the educational system level. it has been identified that the countries involved would like to see the platform develop as a general collaboration tool for their professional development provision. particularly in france, but also in germany and the uk, the tool was seen as a necessary way to communicate with their peers. the challenge is to see how other activities can be introduced into the platform as teachers do not want to have multiple platforms for such collaboration, but a central place where they can collaborate and share best practice. intel hopes to develop the platform despite systemic challenges, to promote the platform as a social media tool and a place for the development of twenty-first century skills among teachers and the consequent positive impact on the teaching and learning experience. note 1. the intel teach† advanced online programme discussed in this paper forms part of intel’s intel teach programme which is a worldwide teacher training initiative. the objectives behind it are improving teacher effectiveness through professional development, helping teachers integrate technology into their lessons and promoting students’ problemsolving, critical thinking, and collaboration skills. worldwide, the intel teach programme has trained more than 9 million teachers in over 60 countries. intel teach is part of intel’s corporate responsibility strategy which is run by their corporate affairs department which works solely on non-profit initiatives. references adamali, a., j.o. coffey, and z. safdar. 2006. trends in national e-strategies: a review of 40 countries. in information and communications for development: global trends and policies, ed., the international bank for reconstruction and development, 87�124. washington, dc: the world bank. becta. 2007. harnessing technology review 2007: progress and impact of technology in education. summary report, http://publications.becta.org.uk/display.cfm?resid�33979. darling-hammond, l., and m.w. mclaughlin. 1995. policies that support professional development in an era of reform. phi delta kappa 76/8: 597�604. elmore, r., and d. burney. 1999. investing in teacher learning: staff development and instructional improvement. in teaching as the learning profession: handbook of policy and practice, ed. l. darling-hammond and g. sykes, 263�91. san francisco: jossey-bass. ganz, a., and g. reinmann. 2007. blended learning in der lehrerfortbildungevaluation einer fortbildungsinitiative zum einsatz digitaler medien im fachunterricht. unterrichtswissenschaft 35/2: 169�91. häuptle, e., a. florian, and g. reinmann. 2008. nachhaltigkeit von medienprojekten in der lehrerfortbildung. abschlussbericht zur evaluation des blended learninglehrerfortbildungsprogrammems, intel† lehren � aufbaukurs online [sustainability of media projects in teacher professional development. report on the evaluation of the blended learning programme for professional development of teachers, intel teach � advanced online]. arbeitsbericht nr. 20, augsburg: universitaet augsburg. oecd. 2009. creating effective teaching and learning environments: first results from talis, oecd. http://www.oecd.org/edu/talis/firstresults d. arati et al. 180 http://publications.becta.org.uk/display.cfm?resid=33979 http://publications.becta.org.uk/display.cfm?resid=33979 http://publications.becta.org.uk/display.cfm?resid=33979 http://publications.becta.org.uk/display.cfm?resid=33979 http://publications.becta.org.uk/display.cfm?resid=33979 http://www.oecd.org/edu/talis/firstresults ottoson, j.m. 1997. after the applause: exploring multiple influences on application following an adult education programme. adult education quarterly 47: 92�107. soury-lavergne, s., g. gueudet, c. loisy, and l. trouche, eds. 2010. parcours de formation de formateurs et de stagiaires: suivi et analyse [learning path of trainers and trainees: monitoring and analysis]. inrp report 2010. http://eductice.inrp.fr/eductice/projets/ pairformance/rapport-prf-2010.pdf smith, c., and m. gillespie. 2007. research on professional development and teacher change: implications for adult basic education. review of adult learning and literacy 7: 205�44. smylie, m.a., e. allensworth, r.c. greenberg, r. harris, and s. luppescu. 2001. teacher professional development in chicago: supporting effective practice. chicago, il: consortium on chicago school research. voogt, j., and g. knezek, eds. 2009. international handbook of information technology in primary and secondary education. 20 vols. new york: springer. series: springer international handbooks of education. youngs, p. 2001. district and state policy influences on professional development and school capacity. educational policy 15/2: 278�301. alt-c 2011 conference proceedings 181 http://eductice.inrp.fr/eductice/projets/pairformance/rapport-prf-2010.pdf http://eductice.inrp.fr/eductice/projets/pairformance/rapport-prf-2010.pdf doi:10.3402/rlt.v24.30089 original research article crashed � a live immersive, learning experience embedding stem subjects in a realistic, interactive crime scene marie l. bassford a *, annette crisp b , angela o’sullivan c , joanne bacon d and mark fowler e a school of engineering and sustainable development, faculty of technology, de montfort university, leicester, uk; b school of applied social sciences, faculty of health and life sciences, de montfort university, leicester, uk; c school of allied health sciences, faculty of health and life sciences, de montfort university, leicester, uk; d school of computer science and informatics, faculty of technology, de montfort university, leicester, uk; e leicester school of pharmacy, faculty of health and life sciences, de montfort university, leicester, uk (received 20 october 2015; final version received 25 june 2016) interactive experiences are rapidly becoming popular via the surge of ‘escape rooms’; part game and part theatre, the ‘escape’ experience is exploding globally, having gone from zero offered at the outset of 2010 to at least 2800 different experiences available worldwide today. crashed is an interactive learning experience that parallels many of the attractions of an escape room � it incorporates a staged, realistic ‘crime scene’ and invites participants to work together to gather forensic evidence and question a witness in order to solve a crime, all whilst competing against a ticking clock. an animation can enhance reality and engage with cognitive processes to help learning; in crashed, it is the last piece of the jigsaw that consolidates the students’ incremental acquisition of knowledge to tie together the pieces of evidence, identify a suspect and ultimately solve the crime. this article presents the background to crashed and an overview of how a timely placed animation at the end of an educational experience can enhance learning. the lessons learned, from delivering bespoke versions of the experience to different demographic groups, are discussed. the article will consider the successes and challenges raised by the collaborative project, future developments and potential wider implications of the development of crashed. keywords: animations; avatars; connectivism; higher education introduction unfortunately, a crime scene might be found anywhere. this includes the home, the workplace and anywhere within our towns, cities or on our transport systems. according to figures issued by the office for national statistics (ons), there were approximately 7.8 million crimes committed against households and adults (aged 16 and over), in the year ending june 2014 (ons 2015). whilst this figure, according to the ons, represented a decrease in crime overall at the time, records for individuals killed or seriously injured on our roads revealed a 4% rise for the same period (department of transport 2014). *corresponding author. email: mbassford@dmu.ac.uk responsible editor: peter reed, university of liverpool online, united kingdom research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 m. bassford et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http:// www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http:// creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 fuelled by media reports of the tragic events caused by the impact of crime and their secondary representation within tv drama and documentary, society now has a largely voyeuristic relationship with crime and its investigation. according to chermak (1994), greer (2009) and hall (1973), amongst others, contemporary media content has seen an expansion from the lexical to a more visual interpretation of criminal behaviours and the investigative procedures employed by enforcement agencies to reduce them. it is therefore perhaps unsurprising that, like the generation before them, young people have a fascination with crime. as a catalyst for learning, texts, codes and conventions representing the content of popular crime drama, have for a number of years been increasingly employed in studies of english, drama and media at gcse and a level. there is even a suggestion that as a result of the number of tv programmes such as csi (crime scene investigation), there is the potential for influencing legal procedures. whilst the ‘csi effect’ is as yet unproven, research does however indicate that there is an association between the levels of audience viewing and increased expectations of evidence types and an appreciation of scientific evidence. the evidence provided by maeder and corbett (2015) amongst others suggests that the interest in crime reflected in the media reinforces the potential for it to be used as a tool for education. this is where the idea of using a ‘real’ crime scene as a hook to engage young people in science, technology, engineering and maths (stem) subjects was initially conceived. in collaboration with police from the east midlands serious collision investigation unit (emsciu), de montfort university (dmu) has applied a range of specialist knowledge to create an exciting project that has exposed hundreds of young people to stem subjects. raising aspirations in stem careers is a major priority in uk education, with a focus on young people from disadvantaged backgrounds. to capture the imagination and develop an interest, academic staff from forensic science, maths, physics and criminology developed an interactive learning experience, showcasing stem subjects in a hands-on car crash scenario. another step towards the improvement of student engagement is the provision of clear, immediate feedback. our crime scene scenario concludes with an animated film that illustrates the cause of the accident � texting while driving, which immediately confirms the students’ success or otherwise in solving the crime. in 2015, the police provided a crashed car, along with film and data of the crash, to enhance the realism. crashed has since become part of dmu’s popular open day programme and has been delivered to hundreds of year 8�12 pupils as part of the university outreach initiative, both on and off campus. the ‘experience’, whilst similar in basic concept every time it is delivered, has to be tailored to suit the particular age, background and demographic of each student group involved. there have also been a number of additional variables including duration and order of each sub-activity, number of pieces of evidence to collect and analyse, level of understanding of mathematics required to interpret data and size of group. many lessons have been learned as a result of delivering crashed in subtle but significant variations. this article first presents the background to the development of the collaborative crashed project and describes the core crime scene scenario that has been developed so far. the successes and challenges raised by repeatedly adapting the same core experience for a variety of different audiences will be discussed, followed by a summary of lessons learned. the glue that holds the crashed experience together is the timely presentation of a tailored animation after pupils/students have analysed m. bassford et al. 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 evidence and formulated their own theories and attempted to solve the crime. the article therefore secondly explores the use of such animations, in particular as part of an immersive, interactive educational experience. development of an interactive educational experience crashed was first delivered by dmu in april 2014 at an event organised by teentech, a uk company that engages teenagers with stem careers. police officers from emsciu consulted on the forensic details, and support from leicester college artistic make-up and special effects students further enhanced the realism of the scene. the university supplied a partly dismantled car that had been used in another project, to enhance the realism of the crash scene. arriving at the site of the crash, students were asked to apply basic investigation skills, using tools from science, maths and physics to solve a crime. their challenges included questioning a witness, measuring skid marks to calculate speed and collating additional forensic evidence to identify the cause of the crash. in addition to showing how stem subjects can be used in exciting, practical applications and in potential future careers, crashed also incorporated a public safety message. crashed was delivered to 350 students in leicestershire schools in 2014. this included 60 year 8 pupils from black and minority ethnic backgrounds who had been identified as potentially disengaged � 85% of whom said they were interested in attending university in post-event feedback. feedback from pupils and teachers has been unanimously positive. in 2015, crashed was delivered to over 200 students in schools across the east midlands and 100 and 66 students in staffordshire. this included 36 year 8 gifted and talented mathematics pupils from 15 state schools in leicester as well as underrepresented and potentially disengaged groups such as ‘looked after children’, travellers and young carers. the first scenario that was developed as part of the project concept centred around the networks, contacts and resources to hand. the use of a partly dismantled car and contacts with local crash investigators led to the development of a car crash scenario, but has led the way to the proposal and early development of many other scenarios, as described below. car crash scenario the basic premise of the ‘live experience’ centres about the discovery of skid marks, which appear to be associated with a crashed car and a possible victim of crime. our victim of the ‘crash’ appears to be very badly injured and is represented by a realistic manikin. in order to investigate the scene and engage with the exercise, student participants are initially asked to prepare by dressing in forensic overalls, shoe covers and gloves. they are told that they will be presented with a crime scene and it is their task to preserve the scene, gather the evidence and formulate a theory. a number of elements are then introduced to the students. at the scene, the students have to apply stem-based skills and knowledge in order to measure the skid marks to calculate the speed of the vehicle prior to the collision. observation and logic is subsequently applied to additional evidence found at the scene to enable them to piece together a theory about what actually happened research in learning technology citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 and why. this is supported by the analysis of additional forensic evidence that is to be found at the scene. students are supported in their investigations by academic specialists who facilitate their collection of important evidence, such as fibres, hairs and pollen from within the car, and who demonstrate new skills that can then be applied to further evidential materials such as a bag of suspected counterfeit money from the car boot. at each stage, students consolidate their learning by completing worksheets. the students also have an opportunity to question a witness. the duration of interview time and amount of information the witness reveals can also be tailored according to the age of the student group. at the end of the exercise, students are debriefed and asked to discuss their findings before being shown an animated account of events leading up to the crash which reveals the cause of the accident � texting while driving. development of other crime scene scenarios the crashed team secured internal university teaching innovation project funding to further enhance the resource by purchasing specialist equipment for blood pattern analysis, forensic photography and image analysis, access to forensic databases for identification of counterfeit money and video animation projects. most recently, the team acquired a realistic manikin that can incorporate special-effects blood, wounds and fingerprints. this will enable the development of further crime scene scenarios, utilizing the university crime house. furthermore, emsciu and the university are employing 360-degree filming using state-of-the-art equipment to develop online resources, making the resource available permanently, both as an academic resource and for community outreach. this is especially important as since the demise of aimhigher it is essential that sustainable outreach resources are developed. using an animation to enhance learning interacting as a network in the car crash scenario described above, an animated film finally shows the cause of the accident � texting while driving. an animation can enhance reality and engage with cognitive processes to help learning, and in our project, this is the last piece of the jigsaw that embeds the students’ incremental acquisition of knowledge to tie together the pieces of evidence, identify a suspect and ultimately solve the crime. ainsworth’s six different levels of explanation that can and should be evoked to understand learning with animation (ainsworth 2008) are explored as part of this article, namely (1) expressive, (2) cognitive, motor and perceptual, (3) affective and motivational, (4) strategic, (5) metacognitive and (6) rhetorical. it has proven especially effective as a way to enthuse and motivate pupils identified as disengaged from education and demonstrates the importance of connective knowledge requiring an interaction. downes (2007) posed ‘is the knowledge being produced the product of an interaction between the members, or is it a (mere) aggregation of the members’ perspectives?’. we have observed that a different type of knowledge is indeed produced when student learners interact as a network to solve a crime and concurs with one of siemen’s principle of connectivism (siemens 2005), m. bassford et al. 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 namely that the ‘ability to see connections between fields, ideas, and concepts is a core skill’. this engagement may also provide these disadvantaged learners with a further benefit if we consider bourdieu’s theory of social reproduction (bourdieu 1977, 1984, 1986, 2004, 2005; bourdieu and passeron 1979; bourdieu and wacquant 1992) in that the ‘capital’ or ‘social advantage’ obtained by engagement with this process provides them with access to ‘science capital’ which, according to a research by archer, dewitt, and willis (2014), is currently unevenly spread and affected by lack of equality and opportunity in education (archer, dewitt, and willis 2014; archer and francis 2007). scenario-based learning gordon (2014) identified the need to develop alternative strategies and flexible pedagogies as part of an audit undertaken by the higher education academy in 2014. technology provides opportunities to enhance learning and to educate as well as entertain. research by deslauriers, schelew, and wieman (2011) supports the evidence that students will learn more in classes where there are interactive tools as opposed to the more traditional lecture structures. the active ‘hands-on’ real-life simulation learning processes, which are core to the principles of crashed, are based loosely on the concepts of situated learning theory (lave and wenger 1991). these state that situated or scenario-based learning (sbl) should take place in the environment in which it would normally be applied. mariappan, shih, and schrader (2004) note the importance of an authentic context; whilst sbl generally reflects the improvement of performance in learning environments such as nursing, engineering or policing, when associated with scenarios such as those developed in crashed, it facilitates the engagement and authentic learning of non-practitioners (evans and taylor 2005). shaffer (2004a, 2004b, 2005) suggests that this type of learning may also be transposed to the virtual world where such simulations aligned to the interests of learners can promote collaborative practice and problem solving. learners may use ‘real tools’ to address problems in what shaffer notes is an epistemic frame within a pedagogical praxis. the crashed simulations provide students with an opportunity to solve a complex problem in a realistic setting thereby encouraging an appreciation of a professional domain comprised of knowledge, skills and values associated with several crime-related professional or practical ‘epistemologies’. whilst shaffer’s work has focused principally on the development of opportunities to virtually develop professional epistemic frames, thus enhancing the skills and understanding of the practitioner, the success of crashed suggests that this pedagogical approach may be transferable and, indeed, cross a number of learning boundaries. carroll (2000) noted the benefits of the technological enhancement of scenarios to augment the potential for multiple levels of learner reflection and cognitive interaction. this is reinforced by research undertaken by meyer and moreno (2002) who found that graphical images produce visual and verbal mental representations which link and integrate prior knowledge in order to construct new knowledge. research in learning technology citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 ‘what if ?’ � creating virtual scenarios that reinforce the learning experience throughout the development and generation of the virtual context for the crashed scenarios, one of the co-authors of this article (the developer of the technological component) was mindful of work by fisher, higgins, and loveless (2006), gee (2004), plowman and stephen (2005) and shaffer (2005) who identified the potential for clusters of ‘purposeful activity’ when associated with digital technology, particularly if associated with creativity and the development of responses to ‘what if ?’ questions as part of a problem-solving process. the virtual scenarios thus needed to logically reflect the choice of possible actions that had subsequently resulted in and were at the heart of the crime scene. responses to the, ‘what if ?’ questions posed as part of the practical student engagement activities thus needed to have a resolution within the ‘back story’ which would be revealed to the students at the conclusion of the exercise. this information was manufactured and created as a film that not only provided the context to the incident under investigation but, in revealing the answers to the questions posed in the student crime scene investigation, reinforced the learning experience. development of the animation the film, based on the back story, needed to reflect the possible motivation and other actions of the offender or offenders/victims in the planning and commission of the crime but, more importantly, to challenge the perceptions of those involved in the exercise as to who typically might or might not be capable of committing crime. as a result, the context revealed that those individuals who might stereotypically be perceived as offenders might not be responsible for the crime under investigation (figure 1). in addition to the development of a storyboard, which would logically plot the course of the crime, the actions and activities of the offenders were ‘captured’ by designing and developing a realistic community landscape in iclone software (figure 2). the avatar or virtual actor who represented the main protagonist in the film was designed to be physically similar in appearance to a member of the crashed team (figure 3). by looking like a real actor in the simulation, associations could thus be made directly by students between the real actor (a lecturer in criminology who figure 1. challenging stereotype perceptions of offenders. m. bassford et al. 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 played the role of an unlikely offender), with the avatar whom they would later see on screen. the car, which was the hub of the forensic and mathematical investigation in the exercise, and a collie dog, which provided some of the forensic evidence in the case, were also virtually recreated and reflected on screen in the consolidation movie and in reality, with both the real damaged motor vehicle and a physical representation of the dog (figure 4). it would have been impractical to use a real dog for the ‘hands-on’ element of the simulation because of health and safety issues, and so a realistic cuddly toy was purchased as the key forensic link between the offender and the crime scene. even the police specialist who supported the exercise was represented as an avatar on screen. in addition to the creation of the avatar movement and activity, ‘live action’ video film of the car crashing and other activities, which might have led the offenders towards the eventual crime scene, was edited and blended with the avatar-based materials (figure 5). the voices of the characters talking and planning the offence and subsequently arguing as the crime later began to go sour, and were scripted, developed and recorded in audacity (version 2.1.1 free online) software. this not only enabled changes to be made to the sound levels but also alterations to be made to the pitch and tone of the voice recording which could then be used to create both a male and female voice reflecting the gender of the films’ two main actors. the film was cut and edited in movie maker (a free microsoft product), before the vocal element was re-introduced to ensure clarity of sound and enable the figure 2. design and development of a realistic community landscape in iclone software. figure 3. the design and development of avatars (or virtual actors). research in learning technology citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 introduction of background noises, such as a dog barking or glass breaking. finally, a subtle musical soundtrack was threaded through the film to highlight events and tensions associated with actions on screen. discussion benefits research by becker and park (2011) revealed that there is evidence that integrative approaches to teaching stem subjects had a positive effect on students learning experiences. their research noted that, in particular, the effect was most profound at the elementary school stage. integrated stem approaches, defined by sanders (2009) as ‘those which explore teaching and learning between/amongst any two or more of the stem subject areas, and/or between a stem subject and one or more other school subjects’, are additionally acknowledged by sanders and others as being intrinsically linked to social sciences and humanities subjects which suggests that both academic and collective benefits are possible if the learning experience is appropriately managed before ‘students make their exit from the ‘stem pipeline’ (sanders 2009). as formerly highlighted, there are numerous nodes and connections within the project, each one vital for ensuring the project is a live and immersive educational experience for pupils and students alike. it is perhaps easiest to first outline the obvious benefits to those experiencing the car crash scenario. benefits to pupils/ students include the following: figure 4. virtual recreation of a real damaged motor vehicle and a physical representation of the dog. figure 5. editing and blending of ‘live action’ video film with avatar-based materials. m. bassford et al. 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 � an interactive learning experience to engage and encourage student exploration � fun, interesting, inquisitive, relevant task � promotes teamwork � shows students how subject disciplines complement each other � illustrates how mathematical calculations and scientific principles are applied in ‘real’ jobs � introduces important ethical messages to the education providers, the time invested has been considerable at times, but with such positive feedback its benefits to the university are noted: � crashed has been recognised as an excellent widening participation/outreach tool and an innovative approach to education, and it has been nominated for an award. � the project has been used on university open days and is currently used at faculty open days for the bsc mathematics and bsc physics programmes. � publicity � crashed empowers all members of our scholarly community and illustrates the contribution universities can make to inspire the next generation to future stem careers. � sustainability � online crashed will be a cost-effective outreach tool. lastly, the benefits to society include the provision of novel opportunities/ experiences to schools that they themselves cannot offer. successes, challenges and lessons learned one of the key successes of the collaboration is the clear engagement of the pupils/ students with learning. altogether, this process supported a more psychological engagement with the process, as described by de rosis et al. (2005), matsumoto and tokosumi (2005) and morrison and zemke (2005) as avatar empathy. this is apparent from verbal feedback from children, teachers, as well as testaments from the police, teentech and other agency organisers: this has not only inspired the students but has opened their eyes into a new world of science. this hands-on learning environment has enhanced the learning cycle and empowered the students. this multi-agency approach to learning has opened up a vast learning potential. (pc tim o’donnell, forensic collision investigator/motorcycle instructor, east midlands serious collision investigation unit, leicestershire police) it makes such a difference when high-quality, well-thought-through activities are run which are not only exciting and fun but also give students real insight into the relevance of work they are doing in the classroom and the possibilities open to them if they develop the right skills. the activity also underlined the way teamwork, attention to detail, tenacity, curiosity, critical thinking and leadership are just as important as pure academic knowledge. the students were buzzing with enthusiasm � essentially they’d spent 30 minutes in a highly creative maths lesson. (maggie philbin, ceo, teentech) one of the most popular experience days available on the ‘yes you can’ programme is the visit to x university, the practical, hands-on activity surrounding the car crash is one that readily appeals to the pupils. the activity brings together science, maths and research in learning technology citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 criminology in an exciting package. the pupils discussion of and interest in careers around science and maths has increased, as has their interest in wanting to go to university � over 85 per cent of our pupils expressed a desire to go to university following the visits. (gregory smith, kainé management) whilst the current focus of this exercise has been to engage students with mathematics and the physical sciences, this form of active and virtual engagement additionally provides opportunities for the social sciences (crisp 2012). a recent, more flexible, version of the scenario, which was tailored to the specific needs of a local college, also provided opportunities to engage with students about the moral dynamics of crime law and criminal justice and connect with elements of psychology and criminology. this blend of physical and virtual engagement thus endeavoured to operate in a similar way to games for change (games for change). challenges have been varied but centre on the following issues: � timing issues: running events to tight schedules, sometimes having only a few minutes to set the scene for a group of pupils, before they are immersed in the experience and set to task, was perhaps the biggest frustration experienced by the educators. if too little time is assigned, the pupils/students do not get to experience the whole activity and therefore are unable to gather adequate evidence and formulate a theory to solve the crime. the length of time required depends on a number of factors including age, group size, background of students and any other factors that might affect their attention/interest. � communication issues: owing to the large number of stakeholders in the project, there is an obvious need for clear communication between all parties, at all times, to ensure logistically that plans can come into fruition. � fit for purpose: this issue expands upon the timing problems of tailoring a scenario to suit a particular group. without clear communication and timing, in addition to planning, if the scenario is not adapted to suit a particular age or demographic, the experience may not be a success. in particular, the level of mathematics required to complete a task has to be pitched appropriately in order to maintain student engagement with the activity. � funding issues: complications with cross-faculty budgets. � staff availability: these events have to be blended with teaching availability and commitments to the ‘day job’. � currently, the focus is on stem subjects alone. whilst this is currently fit for purpose, recent opportunities for presenting the crashed concept in a roadshow format enabled opportunities for the social sciences and highlighted ‘morality tales’ and citizenship issues. this aspect is an important and valuable element of the student engagement. in summary, the lessons learned from our experiences are vast (bassford et al. 2015a, bassford et al. 2015b) and can be categorised as follows: � the crime scene is best divided into different activities � small groups can simultaneously gather evidence and then rotate around the scene similar to a ‘circus event’. � there is an optimum group size for the activity, based upon the number of different activities within a crime scene. depending on their age, seven students m. bassford et al. 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 appears to be an ideal number to fully engage in a ‘hands-on’ experience. at a recent event for example, each group comprised approximately 11 students (or 44 in total) and this was not compatible with either the physical size of the scene or the activities involved. � there is an optimum activity length, based upon predominantly the age of the participant. for disengaged and under-represented groups (travellers, young carers and ‘looked after children’), activity length was 15 minutes per activity. it was noted that the group was keen to ask questions rather than to analyse. pupils from a ‘gifted and talented’ group however engaged with lengthy and mathematically challenging activities up to 25 minutes in length. � pupils/students require very little information prior to commencing the activity. it is sufficient, for a crime scene scenario, to simply inform participants that they are about to enter a crime scene and that they are required to follow activity sheets. � clear activity sheets need to be prepared. the level, depth and quantity of mathematics included are crucial for facilitating the participants’ gathering of evidence. the future of crashed in 2015, the academic team acquired university funding to adapt and enhance the scenario to develop a new, level 5, cross-faculty module on forensic investigation aimed at physics, mathematics and forensics undergraduates. more than 50% of current mathematics and physics first-year students have indicated they will choose to study this module. the new module involves the study of key elements of modern forensic investigation, including crime-scene processing, analytical techniques, laboratory skills, the principles of experimental design and the scientific analysis of data. with a focus on the use of mathematics and physics, students will also have the opportunity to gain hands-on experience of simulated crime scenes, learning about the role of the expert witness. the translation of a single, focused activity lasting between 20 and 30 minutes into a 12-week module involves significant new material development; however, the ethos of the module will continue to focus on our innovative ‘hands-on’ crime-solving scenarios. this is because, with striking consistency, studies show that innovative, active, collaborative and constructivist instructional approaches shape learning more powerfully, in some forms by substantial margins, than do conventional lecture discussion and text-based approaches (pascarella and terenzini 2005). lastly, our project has enabled us to gather extensive verbal feedback from staff and students involved in the activities. however, given the typically short length that each student spends on an activity, it has not proven possible to gather significant quantitative data so far. this is notoriously difficult to do with an outreach activity and is a future aspect of our research for the crashed team to seize opportunities in future activities that will facilitate the collection of more comprehensive data to evaluate specific aspects of each activity. conclusions crashed is an original approach to practical learning, as it provides a practical example of how to stimulate young minds to explore stem subjects. it demonstrates research in learning technology citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 the contribution universities can make by transferring existing skills and knowledge in an engaging format. through the collaboration with teentech, this good practice is now being made available throughout the country. our project has enabled us to develop bespoke scenarios, dependent upon pupil age and background, and many lessons have been learnt. crashed has proven especially effective as a way to enthuse and motivate pupils identified as disengaged from education, yet also has the scientific and moral depth and scope to engage talented stem students and undergraduate students alike. acknowledgements the team would like to thank support from de montfort university (the dmu square mile project and the teaching innovation project fund) and pc tim o’donnell, forensic collision investigator/motorcycle instructor, east midlands serious collision investigation unit, leicestershire police. references ainsworth, s. (2008) ‘how do animations influence learning?’, in current perspectives on cognition, learning, and instruction: recent innovations in educational technology that facilitate student learning, eds d. robinson & g. schraw, information age publishing, pp. 37�67. archer, l., dewitt, j. & willis, b. (2014) ‘adolescent boys’ science aspirations: masculinity, capital and power’, journal of research in science teachingno, vol. 51, no. 1, pp. 1�30. archer, l. & francis, b. (2007) understanding minority ethnic achievement, routledge, london. bassford, m., o’sullivan, a., crisp, a., bacon, j. & fowler, m. (2015a) ‘crime scenes: cultivating colleagueship and enabling connectivist learning’, proceedings of the 22nd association for learning technology conference 2015, 8�10 sep., 2015, university of manchester. bassford, m., o’sullivan, a., bacon, j., crisp, a. & fowler, m. (2015b) ‘the evidence: translating a crime scene outreach activity for school pupils into a higher education module whilst maintaining student engagement’, proceedings of the raising, advancing and inspiring student engagement conference (raise) 2015, 10�11 sep., 2015, university of nottingham. becker, k. & park, k. (2011) ‘effects of integrative approaches among science, technology, engineering, and mathematics (stem) subjects on students’ learning: a preliminary meta-analysis’, journal of stem education: innovations and research, vol. 12, no. 5, p. 23. bourdieu, p. (1977) outline of a theory of practice (translated by r. nice), cambridge university press, cambridge. bourdieu, p. (1984) distinction: a social critique of the judgement of taste (translated by r. nice), routledge and kegan paul, ltd., london. bourdieu, p. (1986) ‘forms of capital’, in handbook of theory and research for the sociology of education, ed j. richardson, greenwood, new york, pp. 241�258. bourdieu, p. (2004) science of science and reflexivity (translated by r. nice), university of chicago press and polity press, cambridge. bourdieu, p. (2005) the social structures of the economy, polity press, cambridge. bourdieu, p. & passeron, j.-c. (1979) the inheritors: french students and their relation to culture (translated by r. nice), the university of chicago press, chicago, il. bourdieu, p. & wacquant, l. (1992) an invitation to reflexive sociology, university of chicago press, chicago, il. carroll, j. m. (2000) ‘five reasons for scenario based design’, interacting with computers, vol. 13, no. 1, pp. 43�60. chermak, s. (1994) ‘crime in the news media; a refined understanding of how crimes become news’, in media process and the social construction of crime, ed g. barak, garland, new york, pp. 95�129. m. bassford et al. 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 crisp, a. (2012) ‘the use of avatar based learning as a medium for criminal justice education’, british journal of community justice, vol. 10, no. 1, p. 15. department of transport. (2014) transport statistics great britain 2014, [online] available at: https://www.gov.uk/government/statistics/transport-statistics-great-britain-2014 de rosis, f., cavalluzi, a., mazzotta, i. & novelli, n. (2005) ‘can embodied conversational agents induce empathy in users?’, in asib’05 joint symposium on empathetic interaction with synthetic characters. proceedings of the joint symposium on virtual social agents: artificial intelligence and the simulation of behaviour: social intelligence and interaction in animals, robots and agents. asib, uk, hatfield, pp. 65�73. deslauriers, l., schelew, e. & wieman, c. (2011) ‘improved learning in a large-enrollment physics class’, science, vol. 332, [online] available at: https://info.maths.ed.ac.uk/assets/ files/landt/deslauriers_science_may2011.pdf downes, s. (2007) ‘an introduction to connective knowledge in hug, theo (ed.): media, knowledge & education � exploring new spaces, relations and dynamics’, in digital media ecologies. proceedings of the international conference, 25�26 june. 2007. type: b publications in refereed conference proceedings available at: http://www.downes.ca/post/33034 evans, d. & taylor, j. (2005) ‘pulling together: keeping track of pedagogy, design and evaluation thought the development of scenarios � a case study’, learning, media and technology, vol. 30, no. 2, pp. 131�145. fisher, t., higgins, c. & loveless, a. (2006) teachers learning with digital technologies: a review of research and projects, vol. 14, futurelab, bristol. games for change, [online] available at: http://www.gamesforchange.org/ gee, j. p. (2004) what video games have to teach us about learning and literacy, palgrave macmillan. gordon, n. (2014) flexible pedagogies: technology-enhanced learning higher education academy, [online] available at: http://www.heacademy.ac.uk/assets/documents/flexiblelearning/flexiblepedagogies/tech_enhanced_learning/tel_report.pdf greer, c. (2009) crime and media: a reader, routledge, london. hall, s. (1973) encoding and decoding in the television discoursem, centre for contemporary cultural studies, birmingham. lave, j. & wenger, e. (1991) situated learning, legitimate peripheral participation, university of cambridge press, cambridge. maeder, e. m. & corbett, r. (2015) ‘beyond frequency: perceived realism and the csi effect’, canadian journal of criminology and criminal justice, vol. 57, no. 1, pp. 83�114. mariappan, j., shih, a. & schrader, p. g. (2004) ‘use of scenario-based learning approach in teaching statics’, proceedings of the 2004 american society for engineering education annual conference and exposition, american society for engineering education. available at: https://peer.asee.org/scenario-based-learning-approach-in-teaching-statics.pdf. [accessed 9 november 2016]. matsumoto, n. & tokosumi, a. (2005) ‘the sense of well-being derived from empathy and attachment for a cohabitant character’, in asib’05 joint symposium on artificial intelligence and the simulation of behaviour, proceedings of the joint symposium on virtual social agents: artificial intelligence and the simulation of behaviour: social intelligence and interaction in animals, robots and agents, hatfield, pp. 80�87. meyer, r. e. & moreno, r. (2002) ‘aids to computer based multimedia’, learning and instruction, vol. 12, no. 1, pp. 107�119. morrison, i. & ziemke, t. (2005) ‘empathy with computer game characters: a cognitive neuroscience perspective’, in asib’05 proceedings of the joint symposium on virtual social agents: artificial intelligence and the simulation of behaviour: social intelligence and interaction in animals, robots and agents asib, uk, hatfield, pp. 73�79. office for national statistics. (2016) uk crime statistics on line. available at: https://www.ons. gov.uk/ pascarella, e. t. & terenzini, p. t. (2005) how college affects students, vol. 2, a third decade of research, jossey-bass, san francisco, ca, p. 646. plowman, l. & stephen, c. (2005) ‘children, play, and computers in preschool education’, british journal of educational technology, vol. 36, no. 2, pp. 145�157. sanders, m. (2009) ‘stem, stem education, stemmania’, the technology teacher, vol. 68, no. 4, pp. 20�26. research in learning technology citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 13 (page number not for citation purpose) https://www.gov.uk/government/statistics/transport-statistics-great-britain-2014 https://info.maths.ed.ac.uk/assets/files/landt/deslauriers_science_may2011.pdf https://info.maths.ed.ac.uk/assets/files/landt/deslauriers_science_may2011.pdf http://www.downes.ca/post/33034 http://www.gamesforchange.org/ http://www.heacademy.ac.uk/assets/documents/flexiblelearning/flexiblepedagogies/tech_enhanced_learning/tel_report.pdf http://www.heacademy.ac.uk/assets/documents/flexiblelearning/flexiblepedagogies/tech_enhanced_learning/tel_report.pdf https://peer.asee.org/scenario-based-learning-approach-in-teaching-statics.pdf https://www.ons.gov.uk/ https://www.ons.gov.uk/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 shaffer, d. w. (2004a) ‘epistemic frames and islands of expertise: learning from infusion experiences’ international conference of the learning sciences (icls), santa monica, ca, pp. 473�480. shaffer, d. w. (2004b) ‘pedagogical praxis: the professions as a model for post industrial education’, teachers college record, vol. 106, no. 7, pp. 1401�1421. shaffer, d. w. (2005) ‘epistemic games’, innovate: journal of online education, vol. 1, no. 6. available at: https://nsuworks.nova.edu/innovate/vol1/iss6/2 siemens, g. (2005) ‘connectivism: a learning theory for the digital age’, international journal of instructional technology and distance learning, vol. 2, no. 1. issn 1550-6908 on line available at: https://www.itdl.org/journal/jan_05/jan_05.pdf m. bassford et al. 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 30089 http://dx.doi.org/10.3402/rlt.v24.30089 https://nsuworks.nova.edu/innovate/vol1/iss6/2 https://www.itdl.org/journal/jan_05/jan_05.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30089 http://dx.doi.org/10.3402/rlt.v24.30089 (page number not for citation purpose) 1 *corresponding author. email: s.douglas1@rgu.ac.uk research in learning technology 2023. © 2023 s. douglas. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 research in learning technology vol. 31, 2023 original research article achieving online dialogic learning using breakout rooms shonagh douglas* aberdeen business school, robert gordon university, aberdeen, uk (received: 26 august 2022; revised: 20 december 2022; accepted: 21 december 2022; published: 24 february 2023) the use of breakout rooms is an increasingly used tool in online teaching. this study uses laurillard’s (2013) conversational framework to evaluate the effectiveness of breakout rooms in achieving learning through peer-to-peer dialogue in large-scale teaching. data were collected through online surveys, comprising likert ratings and open questions, to undergraduate students (n = 115) and tutors (n  =  9) at aberdeen business school (robert gordon university) reflecting on year 1 studies in the 2020–2021 academic year. key findings indicate that breakout rooms can be successful in achieving effective learning through peer-to-peer dialogue. however, this is highly dependent on the participation by students, which was variable. in order to facilitate effective breakout rooms, tutors need to ensure they set a clear task, with evidence suggesting a perception gap between tutors and students on how effectively this was done, and regularly visit breakout rooms to encourage participation and provide support. keywords: breakout rooms; dialogic learning; conversational framework; online; learning introduction dialogue has long been identified as an important component of learning (jung and brady 2020; littleton and howe 2010; simpson 2016). this dialogue can take many forms, but ‘talk’ within a conversation is a common way in which this is achieved within educational settings, with a number of empirical studies identifying pedagogical benefit (alexander 2018; howe et al. 2019; jones and hammond 2016). in the traditional classroom, tutors regularly make use of small group discussions to effectively scaffold conversations between learners. to maintain this peer-to-peer dialogue, following the large-scale movement to online teaching during the coronavirus 2019 (covid-19) pandemic, tutors turned to the technology that could enable this and the use of breakout rooms drastically increased (sharmin and zhang 2022). in practice, particularly when used for large scale teaching, the success of breakout rooms has been variable. while post-pandemic studies of this are limited, mcgrath and wolstencroft (2021) wrote an opinion piece indicating ‘students dislike or at worst dread it’ (p. 1) with problems cited including anxiety, awkwardness, little interaction, and cameras firmly turned off. mailto:s.douglas1@rgu.ac.uk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2882 s. douglas 2 citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 (page number not for citation purpose) this research aims to investigate the effectiveness of breakout rooms to achieve peer-to-peer dialogic learning. in doing this, the following objectives are identified: 1. to evaluate, from the perspective of students and tutors, the effectiveness of breakout rooms in achieving learning through peer-to-peer dialogue. 2. to disseminate the knowledge gained on factors which can facilitate effective use of breakout rooms for learning. this will add to the existing literature in two ways. firstly, by evaluating, with the use of a theoretical framework, the use of breakout rooms as a pedagogical tool for achieving dialogue. while this novel use of the framework will be demonstrated in the context of breakout rooms, it may be transferrable to other vehicles of peer-to-peer learning. and secondly, in the identification and evaluation of factors, which facilitate breakout rooms in large-scale teaching. theoretical framing the importance of dialogue in education is not a new concept. while differing in epistemological stances, and without evidence of engagement, similarities can be drawn in early work in this area by freire (1921–1997) and bakhtin (1895–1975) in the importance of dialogue in education (rule 2011; skidmore and murakami 2016). vygotsky (in vygotsky and cole 1978), another well cited scholar, introduced the concept of learning as a social process with the importance of speech, which transforms into inner speech, and in turn sets out the thinking that results in learning. this has led to the now widely accepted view that speech goes beyond expressing thought to enabling and shaping it. vygotsky’s (vygotsky and cole 1978) work has been classed as dialectic rather than dialogic (wegerif 2008), with learning being directional from tutor to student with differences in knowledge being something to overcome, as opposed to a dialogic approach where meaning arises in the context of differences. however, the thinking of vygotsky (vygotsky and cole 1978) that speech goes beyond expressing thought to enabling and shaping it is continuing to contribute the understanding of learning in this area. the role of conversation was formalised by pask (1976) in his ‘conversation theory’ and the benefit of conversation in learning has become widely accepted with many studies identifying the positive impact of conversation in learning (bjuland and helgevold 2018; chappell and craft 2011). in the learning environment, this conversation can occur between the student and tutor or between students. in line with the first objective of evaluating the effectiveness of breakout rooms in achieving peer-topeer dialogue, this study will initially focus on evaluating the effectiveness of conversation between students. however, the study will also go beyond this to recognise for this to be effective it must be carefully planned and supported by the tutor, which may also include dialogue between the tutor and students. for conversation between students to result in successful learning, laurillard (2012) summarises the results of several studies to identify the steps that need to take place. firstly, the student takes a position and then defends this position through discussion with other students where they respond to critique of this position and consider alternatives offered. following this, the student reflects on their initial position and adapts or changes position. to be effective, laurillard (2012) identifies that this is ‘highly dependent’ (p. 143) on the type of dialogue between learners, with success http://dx.doi.org/10.25304/rlt.v31.2882 research in learning technology citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 3 (page number not for citation purpose) requiring learners to challenge, critique and offer alternatives to ideas generated and explained by the learner. the peer-to-peer learning cycle element of the conversational framework is replicated in figure 1. this element of the conversational framework will be used as the framework for analysing the effectiveness of breakout rooms in achieving peer-to-peer learning. this framework was chosen as it is specifically designed for technologically enhanced learning, and is widely cited in educational technology research (domakin 2017; holmberg 2016; tomczak and bel 2021; von der heyden 2019, inter alia). in addition, it specifically separates out the elements within a conversation that must take place in order for learning to occur and therefore provided an analysis tool, which could be applied to evaluate the success of breakout rooms in achieving this in line with the study objectives. a criticism of an earlier version of the framework but still applicable to the latest version was identified by brewster (2009) in the failure of the framework to include ‘disruptions’, at a technical, conceptual or contextual level, or ‘unintended consequences’. however, brewster (2009) notes this is a particular problem when using the framework for planning purposes, rather than the evaluative focus of this study. breakout rooms as a conversation vehicle: effective facilitation while not framed using the peer-to-peer learning cycle, many studies support the use of breakout rooms as a mechanism for enabling conversation between students. tonsmann (2014, p. 59) identified breakout rooms as an ‘invaluable technique for student understanding and assimilation of concepts’ and saltz and heckman (2020) identified that breakout rooms increase motivation, productivity and help students to connect with other students. cadieux et al. (2020) also noticed that ‘rich’ discussion occurred during breakout rooms. in addition, benefits beyond the peer-to-peer learning cycle may also be found with breakout rooms being identified as giving students the confidence to ask tutors questions and therefore also contributing to student to tutor conversation (chandler 2016). student conceptions student actions (laurillarrd, 2013, p.87) adaptation of actions in the light of theory, goal and feedback reflection on concept in the light of experience figure 1. peer-to-peer learning cycle, the conversational framework. http://dx.doi.org/10.25304/rlt.v31.2882 s. douglas 4 citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 (page number not for citation purpose) however, much of these studies are situated in classes or groups with small numbers and while providing excellent examples of effective use of breakout rooms, such as exposing medical students to telemedical encounters (rucker et al. 2020) or undertaking a structured pair activity for computer programming (saltz and heckman 2020), it cannot be assumed that these findings are transferrable to large-scale teaching. indeed, few studies have been undertaken, with sharmin and zhang (2022, p. 329) identifying literature in this area is ‘severely limited’. two very recent studies have aimed to start closing this gap. sharmin and zhang (2022) looked at weekly and end of term feedback from a class of 330 students and found a more mixed response from students, with many expressing positive feelings towards breakout room usage, but an average of 47% of students each week expressing negative feelings. in tsihouridis et al.’s (2021) study of 327 students, a more positive overall response was identified particularly in giving the opportunity to communicate and discuss. however, it also identified those in their first year of study (freshmen) were more content with the use of breakout rooms due to the social benefits they brought than those in later years of study. both of these studies also identified a participation challenge, with the identification of student frustration with non-participative group members (sharmin and zhang 2022; tsihouridis et al. 2021). macdonald and campbell (2012) identified that for online synchronous teaching to be effective, tutors need to learn how to effectively facilitate online sessions and how to design effective learning activities. while many previous studies are based in small-scale teaching environments, it is still useful to reflect on previous studies breakout room facilitation findings and evaluate the relevance to large-scale teaching. in terms of facilitation, saltz and heckman (2020) identified a particular challenge for breakout rooms in that all students cannot be monitored at the same time compared with a high-level scan of group discussions in a traditional classroom. while technology is emerging that allows tutors to view output, such as a written chat or a whiteboard simultaneously, there still remains a challenge in how to prioritise support, such as in encouraging participation by all participants. for some, this has been overcome by having a tutor in each room (rucker et al. 2020). practically, this may not be possible for large-scale teaching when staff-student ratios may be much higher. pedagogically, this may be more suitable when the focus is on tutor to student learning rather than student to student learning, with research from classroom-based group discussions indicating constant tutor presence may hinder discussion (powell 1974). a compromise to this may be found in an approach of the tutor ‘visiting’ each breakout room, whether randomly or when called on for help, as used by sharmin and zhang (2022). a further complication for large-scale teaching in visiting breakout rooms is setting the desired small number of participants in each group. both tsihouridis et al. (2021) and sharmin and zhang (2022) identified 4–5 students as the optimal group size. however, smaller group sizes come with a trade-off in number of groups a tutor has to visit and the time they can spend with each, particularly in the light of findings by venton and pompano (2021) who estimate an additional 10%–50% of time is required to visit individual breakout rooms. the composition of breakout room groups also results in challenges. sharmin and zhang (2022) found 52% of students preferred working in the same group, however 16% observed that they like the chance to meet and work with other people. http://dx.doi.org/10.25304/rlt.v31.2882 research in learning technology citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 5 (page number not for citation purpose) in terms of designing effective learning activities for online breakout rooms, many studies where success has been achieved are grounded in examples of very specific use or for a very specific task (cadieux et al. 2020; rucker et al. 2020; saltz and heckman 2020). while the specific task may not be transferrable across disciplines, these do promote the success of breakout rooms when being used for a clearly defined and structured activity. however, designing effective tasks does require an investment of time with macdonald and campbell (2012) identifying these can be labour intensive to create and requires knowledge beyond the technology functionality in effective online facilitation and design of online activities. finally, for breakout rooms to be successful both tutors and students need to be confident in the use of technology. as experience grows, this appears to be a decreasing barrier, with many recent studies not commenting on any issues with this. chandler (2016) specifically comments that technology did not cause frequent problems during breakout rooms, although tutors occasionally perceive they may be a problem in advance of teaching, which needs to be overcome. methodology participants data were collected via a survey to all continuing year 2 undergraduate students at aberdeen business school (robert gordon university) (n = 208). the survey was completed during academic year 2021/22 with year 2 students reflecting on their year 1 experience during the prior 2020/21 academic year, which due to the covid-19 pandemic, had been completed primarily online. a further survey was distributed to all module co-coordinators who taught these students in year 1 (n = 9). a survey was selected to reach the large number of potential participants to ensure as many views were taken into consideration as possible. it was felt that students reluctant to speak in breakout room discussions in class may also be reluctant to participate in alternatives, such as interviews or focus groups, and the ease and anonymity of an online survey may encourage participation from all. in addition, this was a consistent approach with previous large-scale studies in this area (sharmin and zhang 2022; tsihouridis et al. 2021). the student sample comprised students on five courses: accounting and finance, management, management with human resource management, management with marketing, and international business management. during the year, each student studied seven specified modules, determined by course being undertaken. four of these modules were undertaken by all year 1 students and other 3 being bespoke to either the accounting and finance students or those on the other management degrees, giving a total delivery of 10 modules. for each module studied, students received a pre-recorded lecture and a live tutorial most weeks (occasionally in person in semester 1 only). each module co-ordinator designed the content and approach of the live tutorials and were free to use, or not use, breakout rooms during these. no breakout room discussions were assessed or graded. all of the tutors, except one, used zoom as the platform for running the online breakout rooms with the other tutor using blackboard collaborate. of the 10 modules delivered, these were run by 9 module coordinators due to one module coordinator running two of these modules. while the module coordinator http://dx.doi.org/10.25304/rlt.v31.2882 s. douglas 6 citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 (page number not for citation purpose) may have been assisted by further tutors in delivery, they were responsible for setting activities, supporting the tutors in delivery and discussing any delivery issues. as such they have overview of the full modules and were determined as the best person to complete the survey. the tutors involved have not received formal training on the use of breakout rooms. however, informal training, such as discussions with colleagues, and reflecting on previous experience helped to shape their approach. data collection and analysis online surveys was used to administrate the survey. this is general data protection regulations (gdpr) compliant and certified to iso27001 standard. questions were generated by using the steps identified by laurillard (2012) for effective peer-to-peer learning and which feed into the peer-to-peer learning cycle (see table 1). further questions were generated through an extensive literature review to identify factors which have previously contributed to the success of breakout rooms. questions comprised 5-point likert scale questions to show levels of agreement or occurrence of given statements and open narrative questions to allow student and tutors to expand and give further information. once complete, the survey data were entered into spss for statistical analysis. various descriptive analyses were undertaken to understand the data gathered. in addition, spearman’s rho (r s ) was applied to identify any significant relationships between identified elements of laurillard’s peer-to-peer learning cycle. this non-parametric statistic treats the likert rating as ordinal data and is consistent table 1. student rating of peer-to-peer learning elements. survey question always (%) often (%) sometimes (%) seldom (%) never (%) students take a position (conception) and defend this through discussion (action) with other students i shared my ideas and explanations in breakout room discussions. 20 47 26 4 2 students receive feedback from peers i received useful feedback from other students on my ideas or explanations during breakout rooms. 4 27 30 25 12 my ideas or explanations were challenged by other students in breakout rooms. 0 26 37 24 12 i responded to or challenged other students’ ideas or explanations in breakout rooms. 5 24 36 21 12 students reflect (action) in the light of discussion and adapt or change (action) position (conceptions) in the light of discussion i adapted or changed my understanding as a result of breakout room discussions. 7 37 41 11 3 i got confirmation that my ideas or explanations were correct as a result of breakout room discussions. 12 37 36 12 1 even when i wasn’t speaking, listening to other students’ discussion in breakout rooms helped me construct and validate knowledge. 16 43 27 9 4 http://dx.doi.org/10.25304/rlt.v31.2882 research in learning technology citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 7 (page number not for citation purpose) with previous education studies looking at the relationship between likert ratings (lim and richardson 2016; mullen and tallent-runnels 2006). ethical considerations ethical approval was received through the robert gordon university ethical approval process in advance of data collection. in addition, it was explained to participants that participation was voluntary with the option to withdraw at any time through leaving the questionnaire. assurance was also given that responses would be anonymous and could not be traced back to individuals. each participant gave consent to participate in the research through selecting an option to this effect before commencing the survey. results a total of 115 students completed the student survey giving a completion rate of 55%. of these, 42% were from the accounting and finance programme and 58% were from the management programme. this shows a slightly higher completion by accounting and finance students with the full population split between management and accounting and finance. descriptive statistics identified an almost equal split between male and female respondents with 46% of respondents identified as female and 51% as male (with three respondents choosing not to disclose this). all of the tutors completed the tutor survey giving a 100% response rate. breakout room effectiveness: tutor and student perspective the tutor survey responses identified that breakout rooms were a popular pedagogical tool with tutors. all but one of the tutors used breakout rooms with one tutor using them in every tutorial, three tutors using in most tutorials and four using occasionally. in order to understand why these were used, tutors were asked this as an open question and responses given resonated, directly or indirectly, with encouraging dialogue between students. enabling group work was the most common reason given (n = 4) with one tutor specifically identifying breakout rooms were used so that students ‘can learn from each other’ and another ‘to encourage them to speak to each other’. two module coordinators identified breakout rooms were used to respond to the challenge of encouraging ‘talk’ when online, with one tutor noting ‘this may be easier when in a small group’. a variety of other reason were given: to foster a community, to encourage active participation, to get a feel for understanding or difficulties, to give individual feedback and to give variety in approach. in rating the effectiveness of breakout rooms in achieving dialogue between students and collaborative learning, the majority of tutors identified these as somewhat effective, with detail shown in chart 1 and chart 2. the ‘somewhat effective’ ratings were backed up with narrative responses from tutors. when asked of their assessment of the learning that was being undertaken, a key theme emerged in responses that when students participated there was evidence of learning, but student participation did not always happen, which limited learning. this adds evidence of emerging themes from the previous literature in the use of breakout rooms in large-scale teaching (sharmin and zhang 2022; tsihouridis et al. 2021). http://dx.doi.org/10.25304/rlt.v31.2882 s. douglas 8 citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 (page number not for citation purpose) however, as tutors were not present in breakout rooms for most of the time, it is also useful to triangulate this data by also looking at the students’ perspective of learning. an original approach was taken to this where the elements required for peer-to-peer learning were identified from laurillard’s (2012, 2013) peer-topeer learning cycle and students were asked to rate their agreement on a 5-point likert scale of how well the identified elements were achieved. this is summarised in table 1. these initial descriptive statistics give an indication of the frequency of participation in the different identified elements of laurillard’s peer-to-peer learning cycle. to further understand if student participation in the elements of the peer-topeer learning cycle impacted the cycle’s desired output, identified as an adapted or changed conception or confirmation that initial conception was correct, spearman’s rho (r s ) test statistics were undertaken with the results shown in table 2. these initial results indicate the potential for breakout rooms to achieve peer-to-peer learning, with individual elements of the peer-to-peer learning cycle highly correlated to overall identification of learning being achieved. however, the key word here is ‘potential’ as this success is significantly linked to student participation – for those who shared their ideas and explanations and were then able to receive feedback, the somewhat effective somewhat ineffective chart 1: how effective, based on your experience, are breakout rooms in achieiving collaborative learning somewhat effective somewhat ineffective chart 2: how effective, based on your experience, are breakout rooms in achieiving dialogue between students http://dx.doi.org/10.25304/rlt.v31.2882 research in learning technology citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 9 (page number not for citation purpose) results indicate a significant relationship to learning. however, for those who were not willing to participate and even share ideas or explanations, the peer-to-peer learning cycle cannot really begin. this was reflected in the overall question to students where 47% agreed or strongly agreed that breakout rooms helped to improve understanding of a topic, while 25% neither agreed or disagreed, 19% disagreed and 9% strongly disagreed with this. in addition, while this study focused on peer-to-peer learning, evidence from both tutors and students indicated a contribution to learning beyond the peer-to-peer learning cycle but still within the conversational framework. this was in breakout rooms aiding discussion between students and tutors and thus contributing to the tutor-to-peer learning cycle (laurillard 2013). this benefit was identified in a small-scale project by chandler (2016) and the findings from this study provide evidence that this is also true for large-scale teaching, with 59% of students either agreeing (44%) or strongly agreeing (15%) that discussions in breakout rooms gave them confidence to raise concerns or ask the tutor for further guidance once they knew other students had the same concerns or questions. in addition, all but one tutors who used breakout rooms agreed that breakout rooms were very effective (n = 3) or somewhat effective (n = 5) in giving students confidence to ask the tutor questions once they had discussed it with their peers. facilitating effective breakout rooms while this analysis indicated the potential for peer-to-peer learning, there were still several students whose responses indicated that they did not experience all or some of the components required to achieve peer-to-peer learning. this was explored further. macdonald and campbell (2012) identified that for online synchronous teaching to be effective, tutors need to learn how to effectively facilitate online sessions and how to design effective learning activities. so, an open question on what tutors could do to aid successful breakout rooms, along with likert ratings, developed on findings from table 2. bivariate correlation coefficient [spearman’s rho (r s )]. evidence of learning evidence of participation in peer-to-peer learning cycle i adapted or changed my understanding as a result of breakout room discussions. i got confirmation that my ideas or explanations were correct as a result of breakout room discussions. i shared my ideas and explanations in breakout room discussions. 0.462** 0.496** i received useful feedback from other students on my ideas or explanations during breakout rooms. 0.508** 0.528** my ideas or explanations were challenged by other students in breakout rooms. 0.483** 0.479** i responded to or challenged other students’ ideas or explanations in breakout rooms. 0.555** 0.461** **correlation significant at the 0.01 level (2-tailed). http://dx.doi.org/10.25304/rlt.v31.2882 s. douglas 10 citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 (page number not for citation purpose) previous literature and which aimed to identify what tutors can do to improve facilitation, were asked to students. the most popular response (n = 34/86) to this open question on what tutors can do, was the setting of a clear task. this importance of a clear task was also evidenced by a spearman’s rho (r s ) showing a significant correlation (0.320; p ≤ 0.001) between ratings on how ‘breakout rooms helped my understanding of a topic’ and ‘i was always clear on what we were meant to be doing in breakout rooms’. regarding setting the task, a number of students referred to the need for ‘clear instructions’, a ‘clear explanation’ or a ‘clear aim’. indeed, 21 students specifically used the word ‘clear’. linked to this, further students identified a preference for a specific task rather than general discussion as summarised by the following student: ‘a good idea would be to give a specific task to be done instead of splitting the class into rooms just for them to discuss something’. this was also corroborated with the likert rating where there was strong agreement with the students that breakout rooms worked better when there was a specific output to produce (61% strongly agreeing, 24% agreeing). tutors agreed with this approach as indicated by 75% strongly agreeing and 25% agreeing that students had to produce a clear output from each breakout room. however, an interesting discrepancy was identified between tutors and students in terms of perception of how this was enacted. all tutors who used breakout rooms strongly agreed (n = 7) or agreed (n = 1) that students were given clear instructions on what they were expected to do. however, when students were asked if they were clear on what they were meant to be doing in breakout rooms, the responses were much more mixed as highlighted in chart 3. the second most common answer by students on what a tutor can do to aid successful breakout rooms regarded tutor visits. of those who elaborated on why this aided an effective breakout room, the most common explanation (n = 13) was to help get the discussion going and encourage participation. examples included, ‘go around and make sure the conversation is flowing’ and: ‘go through the breakout rooms and ask for cameras to be turned on/ensure conversation is happening. a lot of teachers did go through the groups, but they just ask if any help is needed etc then only one student replies to say yes or no. they could do 0% 10% 20% 30% 40% strongly agree agree neither agree or disagree disagree strongly disagree chart 3: i was always clear on what we were meant to be doing in breakout rooms. http://dx.doi.org/10.25304/rlt.v31.2882 research in learning technology citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 11 (page number not for citation purpose) with listening in on the group for a couple minutes to ensure all students are speaking to each other and if not maybe contact those students who haven’t been putting in any input to the breakout rooms’. a lack of participation has previous been identified as an issue, particularly for largescale teaching (sharmin and zhang 2022; tsihouridis et al. 2021) and was also an issue for these students. student ratings of their own participation and others’ participation are summarised in chart 4. this student perception can also be corroborated by the tutors’ perspective. when tutors were asked if they encountered any problems when using breakout rooms, five tutors mentioned problems with student participation. this included one tutor identifying rooms where students ‘refused to engage’ and another noting ‘the entire rooms remains silent until the tutor drops in’. to further understand what specifically could be done to improve participation, cross tabulation, between those who rated their own participation as ‘sometime’ (n = 29) or ‘seldom’ (n = 2) and their narrative response to what tutors could do to aid effective breakout rooms, was undertaken. five students simply identified that they did not want tutors to use them but gave no further explanation. outside of this, two recurring themes were identified that resonated with previous analysis: setting a clear task and regular tutor visits to encourage participation. alongside setting a clear task, this highlights from the students’ perspective the value of tutor visits. this view was also shared by the tutors who were able to identify that some rooms ‘remains quiet until the tutor drops in’. all tutors visited breakout rooms, though the approach to this varied from ‘tried to visit once’ to multiple visits. however, almost all tutors directly mentioned or alluded to time pressures. for example, one tutor noticed that ‘time was a real issue – difficult without some finishing tasks and wanting to move on’ and another noting ‘it took some time to get round all the groups’. this challenge is exacerbated by 84% of students also rating the time given for breakout rooms to be too long (‘always’, ‘often’ or ‘sometimes’). in the absence of additional teaching resource, this results in a challenge for tutors in terms of visiting teaching rooms at the right time. a further contentious area was if students should be encouraged to put on cameras. while fully acknowledging the varying views (castelli and sarvary 2021; 0% 5% 10% 15% 20% 25% 30% 35% strongly agree agree neither agree or disagree disagree strongly disagree the majority of breakout room participants contributed to discussions. chart 4: the majority of breakout room participants contributed to discussions. http://dx.doi.org/10.25304/rlt.v31.2882 s. douglas 12 citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 (page number not for citation purpose) jayasundara et al. 2022; pavlov, smirnova, and nuzhnia 2021) and how this may not be appropriate for all students, it is important to reflect on the views of students on this. while discussing how to encourage students to participate, some referred to tutors encouraging students to switch on microphones and speak, while others specifically mentioned switching on cameras. charts 5 and 6 summarise students’ opinion on cameras being used and the impact this has had on discussion. to understand further, students were asked ‘if you didn’t feel comfortable putting your camera on, can you explain why?’ forty-nine students gave an explanation for this. interestingly, by far the most common explanation could be summarised as because nobody else was, as one responder describes as ‘no one else did it so didn’t want to be the weirdo with my camera on’. this is in keeping with earlier responses that identified many students actually wanted tutors to encourage them to put cameras on. a smaller number of students had other more personal reasons for not putting on their cameras. possibly fixable with some advance planning, six students commented that tutorials were often early and they were not camera ready, while a further three commented they were either in bed or their pyjamas. promoting the use of virtual backgrounds could respond to the five further students’ concerns over sharing their home environment. however, five students gave personal reasons for not wanting to 0% 5% 10% 15% 20% 25% 30% 35% always often sometimes seldon never the majority of students had their camera on during break out room discussions chart 5: the majority of students had their camera on during breakout room discussions. 0% 5% 10% 15% 20% 25% 30% 35% 40% strongly agree agree neither agree or disagree disagree strongly disagree having cameras on in breakout rooms aided discussion chart 6: having cameras on in breakout rooms aided discussion http://dx.doi.org/10.25304/rlt.v31.2882 research in learning technology citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 13 (page number not for citation purpose) switch these on, such as ‘i am not comfortable showing my face in public’ or ‘feel weird with people looking at me’ and it is important these minority opinions are also considered. tutors may consider encouragement rather than requirement or if this is a requirement, ensure that support is available for these students. limitations and further research limitations for this research are identified as follows. firstly, the sample represents a unique group of students and while from a range of courses these were all in business/ accounting areas at the same level of study. the findings may not be directly transferrable to other levels and areas of study. in addition, the study is limited to the technology available at the time of the study, namely zoom and blackboard collaborate, and as new technology emerges this may change the way in which breakout rooms are facilitated. for example, features in adobe connect can allow tutors to view chat or whiteboards of multiple groups simultaneously and further research could establish if this could improve facilitation of breakout rooms through responding to challenges identified in this research. finally, further research could also extend the knowledge in this area by identifying other more specific factors that impact the success of breakout rooms, such as the specific subject being taught and the nature of the subject (e.g. difference between more factual/numeric topics and more discursive topics), as well as analysing the relationship between breakout room activity and assessment performance. conclusion this research aimed to investigate the effectiveness of breakout rooms to achieve peerto-peer dialogic learning. a unique application of laurillard’s peer-to-peer learning cycle, extracted from the conversational framework (2013), was used as a framework to evaluate if, from the students’ perspective, effective learning was achieved. the key finding from this analysis was that for students who participated, and as measured by ‘actions’ in laurillard’s (2013) peer-to-peer learning cycle, evidence of effective learning was identified. this finding was corroborated by tutor perception that breakout rooms had the potential to achieve effective peer-to-peer learning, but this is strongly influenced by participation. participation was therefore identified as a key element in success but achieving this was identified as challenging with several students not participating in discussion and therefore limiting both their own and other students learning. strategies to improve participation were therefore explored further and two key themes emerged: setting a clear task with a tangible output and making effective tutor visits. while setting a clear task seems perhaps obvious, the results revealed an interesting discrepancy between tutors and students regarding whether a clear task was being set. tutors should regularly reflect on this – good evidence of success, or otherwise, can be gained from visiting rooms and identifying if the task was clear and if not, it is important to reflect on what was unclear and to use this knowledge going forward. in addition, tutors should also ensure that a mechanism is in place if students are in a breakout room and the task is unclear. this could include training students in the ‘call for help’ or ‘message the tutor’ type tools available in most breakout rooms, which can direct tutor attention quickly to the room most in need. having the task http://dx.doi.org/10.25304/rlt.v31.2882 s. douglas 14 citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 (page number not for citation purpose) written down and either sharing this or encouraging the students to take a photo or screenshot if written on screen can also help. finally, students preferred to have a specific identified output rather than a general ‘discuss…’ this can take many forms such as completing a collaborative document, making a powerpoint slide or producing a short text answer which can then be shared after with as a whole group via a chat message. as well as clarifying the task, tutor visits can also play an important role in encouraging participation. however, this can be inhibited by time, particularly with students indicating a preference for shorter breakout rooms. a potential solution may come in the form of collaborative tools where group responses are updated on a separate or integrated platform. this would allow the tutor to order visits in a more productive manner – targeting those not getting started for initial help, followed by identification of misunderstanding as discussions progress and visiting those ‘finished’ with feedback or ways to extend the discussion. in addition, this could include an ‘are you clear with the task’ as the first required response to ensure the task is clear. acknowledgements with thanks to dr denise sweeney, university of nottingham, for her guidance and feedback as my advisor on the elesig scholar scheme. references alexander, r. (2018) ‘developing dialogic teaching: genesis, process, trial’, research papers in education, vol. 33, no. 5, pp. 561–598. doi: 10.1080/02671522.2018.1481140 bjuland, r. & helgevold, n. (2018) ‘dialogic processes that enable student teachers’ learning about pupil learning in mentoring conversations in a lesson study field practice’, teaching and teacher education, vol. 70, pp. 246–254. doi: 10.1016/j.tate.2017.11.026 brewster, d. m. (2009) interrupted conversations: enhancing laurillard’s conversational framework to take account of ‘disruption’, doctoral dissertation, university of sussex. cadieux, m., et al., (2020) ‘journal club using virtual breakout rooms: interactive continuing education with no learner preparation during covid-19’, journal of continuing education in the health professions, vol. 40, no. 4, pp. 217–219. doi: 10.1097/ceh.0000000000000321 castelli, f. r. & sarvary, m. a. (2021) ‘why students do not turn on their video cameras during online classes and an equitable and inclusive plan to encourage them to do so’, ecology and evolution, vol. 11, no. 8, pp. 3565–3576. doi: 10.1002/ece3.7123 chandler, k. (2016) ‘using breakout rooms in synchronous online tutorials’, journal of perspectives in applied academic practice, vol. 4, no. 3, pp. 16–23. doi: 10.14297/jpaap. v4i3.216 chappell, k. & craft, a. (2011) ‘creative learning conversations: producing living dialogic spaces’, educational research, vol. 53, no. 3, pp. 363–385. doi: 10.1080/00131881.2011.598663 holmberg, j. (2016) ‘applying a conceptual design framework to study teachers’ use of educational technology’, education and information technologies, vol. 22, no. 5, pp. 2333–2349. doi: 10.1007/s10639-016-9536-3 domakin, a. (2017) ‘practice makes perfect: designing integrated learning experiences in social work education using laurillard’s ‘conversational framework’ for learning’ the journal of practice teaching & learning, vol. 15, no. 2, pp. 56–75. doi: 10.1921/jpts.v15i2.1077 howe, c., et al., (2019) ‘teacher–student dialogue during classroom teaching: does it really impact on student outcomes?’, journal of the learning sciences, vol. 28, no. 4–5, pp. 462–512. doi: 10.1080/10508406.2019.1573730 http://dx.doi.org/10.25304/rlt.v31.2882 https://dx.doi.org/10.1080/02671522.2018.1481140 https://dx.doi.org/10.1016/j.tate.2017.11.026 https://dx.doi.org/10.1097/ceh.0000000000000321 https://dx.doi.org/10.1002/ece3.7123 https://dx.doi.org/10.14297/jpaap.v4i3.216 https://dx.doi.org/10.14297/jpaap.v4i3.216 https://dx.doi.org/10.1080/00131881.2011.598663 https://dx.doi.org/10.1007/s10639-016-9536-3 https://dx.doi.org/10.1921/jpts.v15i2.1077 https://dx.doi.org/10.1080/10508406.2019.1573730 research in learning technology citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 15 (page number not for citation purpose) jayasundara, j. m. p. v. k., et al., (2022) ‘how uk he stem students were motivated to switch their cameras on: a study of the development of compassionate communications in task-focused online group meetings’, education sciences, vol. 12, no. 5, p. 317. doi: 10.3390/educsci12050317 jones, p. & hammond, j. (2016) ‘talking to learn: dialogic teaching in conversation with educational linguistics’, research papers in education, vol. 31, no. 1, pp. 1–4. doi: 10.1080/026 71522.2016.1106691 jung, h. & brady, c. (2020) ‘maintaining rich dialogic interactions in the transition to synchronous online learning’, information and learning sciences, vol. 121, no. 5/6, pp. 391–400. doi: 10.1108/ils-04-2020-0096 laurillard, d. (2012) teaching as a design science building pedagogical patterns for learning and technology, routledge, new york, ny. laurillard, d. (2013) rethinking university teaching a conversational framework for the effective use of learning technologies, 2nd edn., taylor and francis, hoboken. lim, j. and richardson, j.c. (2016) ‘exploring the effects of students’ social networking experience on social presence and perceptions of using snss for educational purposes’ the internet and higher education, vol. 29, pp. 31–39. doi: 10.1016/j.iheduc.2015.12.001 littleton, k. & howe, c. (2010) educational dialogues: understanding and promoting productive interaction, routledge, london. macdonald, j. & campbell, a. (2012) ‘demonstrating online teaching in the disciplines. a systematic approach to activity design for online synchronous tuition’, british journal of educational technology, vol. 43, no. 6, pp. 883–891. doi: 10.1111/j.1467-8535.2011.01238.x mcgrath, l. & wolstencroft, p. (2021) ‘i want to break free’ how online breakout rooms have become a modern educational curse,’  chartered association of business schools blog, [online] available at https://charteredabs.org/i-want-to-break-free-how-online-breakoutrooms-have-become-a-modern-educational-curse/ mullen, g. e. & tallent-runnels, m. k. (2006) ‘student outcomes and perceptions of instructors’ demands and support in online and traditional classrooms’, the internet and higher education, vol. 9, no. 4, pp. 257–266. doi: 10.1016/j.iheduc.2006.08.005 pask, g. (1976) conversation theory: applications in education and epistemology, elsevier scientific publishing company, amsterdam. pavlov, v., smirnova, n. v. & nuzhnaia, e. (2021) ‘beyond the avatar: using video cameras to achieve effective collaboration in an online second language classroom’, journal of university teaching & learning practice, vol. 18, no. 7, pp. 228–243. doi: 10.53761/1.18.7.14 powell, j. p. (1974) ‘small group teaching methods in higher education’, educational research, vol. 16, no. 3, pp. 163–171. doi: 10.1080/0013188740160301 rucker, j., et al., (2020) ‘utilizing zoom breakout rooms to expose preclerkship medical students to telemedicine encounters’, medical science educator, vol. 30, no. 4, pp. 1359–1360. doi: 10.1007/s40670-020-01113-w rule, p. (2011) ‘bakhtin and freire: dialogue, dialectic and boundary learning’, educational philosophy and theory, vol. 43, no. 9, pp. 924–942. doi: 10.1111/j.1469-5812.2009.00606.x saltz, j. & heckman, r. (2020) ‘using structured pair activities in a distributed online breakout room’, online learning, vol. 24, no. 1, pp. 227–244. doi: 10.24059/olj.v24i1.1632 sharmin, s. & zhang, l. y. (2022) ‘experience report on the use of breakout rooms in a large online course’, proceedings of the 53rd acm technical symposium on computer science education v. 1, pp. 328–334. doi: 10.1145/3478431.3499328 simpson, a. (2016) ‘designing pedagogic strategies for dialogic learning in higher education’, technology, pedagogy and education, vol. 25, no. 2, pp. 135–151. doi: 10.1080/1475939x. 2015.1038580 skidmore, d. & murakami, k. (eds) (2016) dialogic pedagogy: the importance of dialogue in teaching and learning, multilingual matters, bristol. tomczak, j. & bel, e. (2021) ‘a conversational framework for learning design (in adverse times)’, journal of learning development in higher education, vol. 22, no. 22, pp 1–7. doi: 10.47408/jldhe.vi22.705 http://dx.doi.org/10.25304/rlt.v31.2882 https://dx.doi.org/10.3390/educsci12050317 https://dx.doi.org/10.1080/02671522.2016.1106691 https://dx.doi.org/10.1080/02671522.2016.1106691 https://dx.doi.org/10.1108/ils-04-2020-0096 https://dx.doi.org/10.1016/j.iheduc.2015.12.001 https://dx.doi.org/10.1111/j.1467-8535.2011.01238.x https://charteredabs.org/i-want-to-break-free-how-online-breakout-rooms-have-become-a-modern-educational-curse/ https://charteredabs.org/i-want-to-break-free-how-online-breakout-rooms-have-become-a-modern-educational-curse/ https://dx.doi.org/10.1016/j.iheduc.2006.08.005 https://dx.doi.org/10.53761/1.18.7.14 https://dx.doi.org/10.1080/0013188740160301 https://dx.doi.org/10.1007/s40670-020-01113-w https://dx.doi.org/10.1111/j.1469-5812.2009.00606.x https://dx.doi.org/10.24059/olj.v24i1.1632 https://dx.doi.org/10.1145/3478431.3499328 https://dx.doi.org/10.1080/1475939x.2015.1038580 https://dx.doi.org/10.1080/1475939x.2015.1038580 https://dx.doi.org/10.47408/jldhe.vi22.705 s. douglas 16 citation: research in learning technology 2023, 31: 2882 http://dx.doi.org/10.25304/rlt.v31.2882 (page number not for citation purpose) tonsmann, g. (2014) ‘a study of the effectiveness of blackboard collaborate for conducting synchronous courses at multiple locations’, insight: a journal of scholarly teaching, vol. 9, pp. 54–63. doi: 10.46504/09201404to tsihouridis, c., et al., (2021). ‘learning to be together again!–using virtual breakout rooms to fill the communication and cognitive gap in online classrooms’, international conference on interactive collaborative learning, vol. 389, pp. 370–381. doi:10.1007/978-3-030-93904-5_38 venton, b. j. & pompano, r. r. (2021) strategies for enhancing remote student engagement through active learning. venton, b.j. and pompano, r. r. (2021). ‘strategies for enhancing remote student engagement through active learning’, anal bioanal chem, vol. 413, pp. 1507–1512. doi: 10.1007/ s00216-021-03159-0 von der heyden, b. p. (2019) ‘interviews with professional geologists enhance learning about the applied aspects of economic geology for final-year university students’, journal of geoscience education, vol. 67, no. 1, pp. 20–33. doi: 10.1080/10899995.2018.1511959 vygotsky, l. s. & cole, m. (1978) mind in society: development of higher psychological processes, harvard university press, cambridge. wegerif, r. (2008) ‘dialogic or dialectic? the significance of ontological assumptions in research on educational dialogue’, british educational research journal, vol. 34, no. 3, pp. 347–361. doi: 10.1080/01411920701532228 http://dx.doi.org/10.25304/rlt.v31.2882 https://dx.doi.org/10.46504/09201404to https://dx.doi.org/10.1007/978-3-030-93904-5_38 https://dx.doi.org/10.1007/s00216-021-03159-0 https://dx.doi.org/10.1007/s00216-021-03159-0 https://dx.doi.org/10.1080/10899995.2018.1511959 https://dx.doi.org/10.1080/01411920701532228 academics' use of courseware materials: a survey diana laurillard*, betty swift* and jonathan darby** * institute of educational technology, open university, milton keynes mk7 6aj ** ctiss, university of oxford, 13 banbury road, oxford ox2 6nn abstract learning technology has yet to enter the mainstream of higher education. the ufc-funded teaching and learning technology (tlt) programme is attempting to change this by sponsoring projects concerned with courseware production and delivery. these efforts could be thwarted if the not invented here syndrome prevents the use of technology-based teaching and learning materials outside the originating departments. to gain a clearer understanding of why academics have been rejecting much existing courseware, and to establish the extent of the not invented here syndrome, we carried out a survey of 800 academics in eight uk universities. the survey proved to be exceptionally revealing. introduction the increasing interest in the use of educational technology methods for teaching, fuelled now by the teaching and learning technologies (tlt) programme funding, establishes the creation of such courseware as a widespread academic exercise. this is encouraging for those of us who believe that educational media can enhance student learning and understanding, but it begs the question of who is going to use all the material currently under development. one of the key arguments against course materials as an alternative to lectures has always been the not invented here syndrome: academics are just about prepared to recommend a colleague's textbook, but would not take on video or computer-based teaching material developed elsewhere, as the teaching philosophy would be unlikely to match their own. this is an important argument because the development of course materials, especially computer programs, is economically feasible as a teaching method only if large numbers of student users are likely. technology-based materials have to travel beyond the originating department, but how can they if the not invented here syndrome applies? this was the question that exercised the isc courseware development working group earlier this year in its deliberations on future mechanisms for universities moving into the field 4 alt-j volume 1 number 1 (ufc, 1992). we wanted to establish whether the plan to develop materials for use in a range of universities was doomed by academics' preference for in-house teaching. accordingly, the working group designed a questionnaire to investigate the issue among a randomly-selected group of academics. this paper reports on the results. little attention has been paid in the past to the attitudes of academic staff in higher education towards the use of computer-based learning (cbl) materials. researchers trying to identify the factors inhibiting the adoption of courseware have mostly focused their attention elsewhere. kemeny (1990) sees the problem primarily in terms of the poor quality of available software resulting from the individual, non-collaborative nature of its production. glick (1990) cites higher education's lack of flexibility and inertia to change as the foremost barrier. darby (1992) highlights the blockages caused by the secondary importance attached to teaching by the majority of academics (research invariably comes first), the lack of recognition afforded to courseware designers, the limited suitability of much available courseware, and the support problems associated with courseware delivery. attitude surveys in higher education have tended to focus on developers (gardner, 1990), computer centre directors (mcdonough, 1991), or university registrars (csup, 1992), and have mainly concentrated on technical and resource issues. the office for humanities communication, in its comprehensive survey of computer use in the humanities, surveyed academic departments as well as libraries and computer centres. referring to responses from academic departments mullings reports: just over half the sample reported problems associated with their use of computers in teaching. these were generally organizational and financial rather than technical. problems mentioned most frequently were cost, staffing and hardware. (mullings, 1992) several centres in the computers in teaching initiative (cti) have surveyed academic attitudes in the discipline communities they serve. hammond et al (1992) report on two such surveys: land use and psychology. land use lecturers identified 'lack of time' as the main factor inhibiting their use of computers in teaching. psychology lecturers were generally positive about computer use but had a narrow perception of potential teaching applications. indeed hammond et al note: 'it is likely that some of the respondents were hypothesizing from a state of blissful ignorance.' the survey the working group's survey was designed to find o u t : # the extent to which academics already use teaching and learning technologies in their teaching • where the materials being used had originated # their awareness and opinion of cbl materials in their field, and • their attitudes to using cbl materials developed elsewhere. the questionnaire was kept to two sides of straightforward questions, using mainly multiple-choice questions but also generating some open-ended comments (see appendix). 5 diana laurillard et al academics' use of courseware materials: a survey the questionnaire was sent to 800 academics selected at random from the universities handbook. to minimize costs of mailing, only eight universities were involved in the sample, chosen to represent different types and locations: aston, bristol, cambridge, glasgow, hull, university college london, cardiff, and york. addressed questionnaires, with covering letters, were sent to a contact at each university for internal circulation to the selected academics. addressed reminder cards were sent after four weeks, and further copies of the questionnaire with new covering letters were sent after a further four weeks. this elicited a 50% response rate. several replies had to be discarded as recipients had changed jobs or status, or were not involved in teaching, giving a usable response rate of 47%. quantitative data was coded for computer analysis. qualitative data, in the form of written open-ended comments, were typed into a computer file, giving some 80 pages of comments. one third were analysed by inspection to produce a range of categories which were then used to categorize the remaining comments. these are described below. existing use of teaching and learning technologies the great majority of academics recommend textbooks by other authors in their teaching, only 14% recommending none (appendix, question 2). 43% recommend four or more textbooks as required reading for the three or four courses they are teaching during the year. other kinds of educational materials require the use of technology of some kind, either in the classroom or by individual students in a resource centre. slides were the most popular alternative form, 46% having used them at some time; videocassettes were next with 35%, then cbl (defined as teaching programs, not just application tools such as word processing or spreadsheets) with 27%. audio cassettes attracted a much lower response of 11%, perhaps because it is essentially a private study medium, whereas the others can all be class-based and used within lectures (question 3). there is a large minority of academics who already make use of teaching and learning technology, therefore, though the proportion using cbl drops to near a quarter. that is partly due to lack of awareness: only 46% have ever seen cbl programs in their field (question 4), and only 19% have had any contact with their cti centre. almost 40% of the academics using cbl had used only programs originating in their own department or institution (question 3ii). that is a disturbingly high figure. individual institutions cannot afford to develop such a high proportion of the cbl programs they need if such programs are to be of good quality, and the medium exploited to its full potential. at present many academics find the programs they need are not available, and develop their own for that reason, which suggests it is a short-term corrective that will change as more courseware becomes available. or is it a manifestation of the not invented here syndrome? certainly, there is much criticism of the efforts of others in the field, as the next section below demonstrates. attitudes to cbl materials of those in a position to know (i.e. who have seen cbl programs in their field), only 26% think the majority of them are reasonably good; half find them very mixed in quality 6 alt-j volume 1 number 1 (question 4ii). in the written comments on their 'reservations' about computer-based teaching (question 5), the criticisms voiced tended to focus on the pedagogy rather than on any problems with the accuracy of what was presented: much more effort needs to be spent on the user interface. it almost needs to be the focus of development rather than an after-thought. the present software is designed with idiots in mind. we don't have enough idiots here to make it viable ... [there should be] more-emphasis on software being intellectually stimulating. however, reservations concerning quality of software were very infrequent; there is a sizeable minority who express fundamental concerns about the nature of computer-based teaching in general: students can be distanced further from staff members than they are already. this may result in a fundamental misunderstanding of a topic without staff realizing it. i am strongly against the use of the computer screen as an alternative to hands-on experience with laboratory hardware in such fields as mechanical engineering. computers seem inappropriate for material that needs to be pondered over. i would be afraid that they would trivialize the subject, reducing it to a series of responses to problems on the calculation side. i fear that it would be used as a crutch by students and as an alternative to the work that is required for true understanding. all these criticisms could be answered with the assurance that computers will not take over, that they will be used only for those parts of the teaching process for which they are suitable, and that they will be properly integrated with the more traditional forms of teacher involvement. these perceptions highlight the importance of clarifying the role computers should take, and confirm its limited nature. some clarification of the perceived role can be found in written replies on the 'advantages' of computer-based teaching. the key educational benefits are seen as: • dynamic visual presentation • provision for rehearsal of basic skills • access to data and information sources • improvement of learning/understanding. only the last of these is at all contentious. the category includes achievable expectations such as: allows students to progress at their own pace students can test themselves as they go along 7 diana laurillard et at academics' use of courseware materials: a survey it allows students to learn actively without increasing contact time would allow more flexibility and freedom for students to examine process alternatives with the first being mentioned most frequently, but the category also includes a few optimistic generalities concerning better understanding and help with exploring the subject. no medium can offer these advantages by its very nature; they all need inspired creative input to do that. in general, however, the respondents had a realistic and positive approach to the pedagogic capabilities of computer-based teaching. the not invented here syndrome if at present only a quarter of the university teaching population uses cbl, and only 60% of the cbl programs an academic uses are developed elsewhere, does this mean that academics are reluctant to take on other people's teaching programs? to gauge the feasibility of a national programme for courseware development, such as tlt, we asked respondents if they would be interested in such collaboration (question 8a). interest was high. 52% of all respondents, not just those who already used cbl, were interested in collaborating with other academics either in their own department or in other institutions in order to develop programs. the inter-university consortium approach to courseware development, advocated by the isc working group (ufc, 1992), and manifested in the tlt call for proposals, was clearly feasible, therefore, as was borne out by the unexpectedly large number of consortium proposals received by the funding council. that so many academics are now interested in the development of computer-based courseware in their subject is vital for adequate provision of a suitable library of courseware. but the real test of the feasibility of cbl resides also in the preparedness of academics to use courseware originating elsewhere. again, the responses in the survey were extremely encouraging. almost 70% of all respondents were interested in using cbl programs developed by a national programme. moreover, 60% of all respondents were interested in evaluating such programs, itself an essential activity if quality of design is to improve. the survey elicited a fuller picture of attitudes to introducing cbl by asking respondents to react to statements describing different aspects of the issue (question 6). around 60% disagreed with statements that 'a computer-based teaching program would be intrusive' and 'others' ideas would not fit', though 16% agreed with them, coincident with the proportion who use no textbooks. it is an inevitable consequence of the progress of academic knowledge that at any one time some sizeable proportion of the teaching in universities could and should be promulgated independently of standard texts. that is a figure we cannot expect to change very much. at the same time, many teachers welcome the intrusion of others' ideas, and almost 90% agree that they would not mind their students being exposed to 'alternative perspectives' on their subject, though this is not the same as alternative 'teaching', which cbl may well embody. lack of use of cbl does not reflect its innate unsuitability. nearly 73% of respondents agreed that 'some aspects of the subject would benefit from a well-designed computer program'. 8 alt-j volume 1 number 1 opinion is exactly balanced when considering the relative merits of cbl. as many agree as disagree that 'limited resources are better spent on other forms of teaching than on computer-based teaching programs', with about 40% neutral. the survey shows that while usage of cbl is still low, at around 25% of academics, interest is much higher. most importantly, it counters with impressive evidence the traditional excuse for lack of take-up. the not invented here syndrome is no longer a potent force, if it ever was. the majority of academics are interested in using or testing programs originating elsewhere. there are still barriers to take-up, as the next section describes, but they are logistical and financial, not cultural. academics, as represented by the respondents to this survey, are open to alternative perspectives in their teaching, aware of the potential benefits of cbl, willing to contribute to its development as a teaching method, but also balanced in their opinion of its relative value. academics' concerns the radical reconstruction of higher education in the uk is being experienced by academics both as a catalyst for change and as a barrier. recognition of the former comes from several written comments (question 8b). barriers are seen as 'insufficient time', as pressures in their department make it difficult to take advantage of new teaching methods: it is very difficult to take a new initiative when one is totally occupied coping with the present demands of the university. i wish we had more time to develop our teaching, but the demands of academic life make time a virtual impossibility for the foreseeable future. (strong requirement here is to get on with research). given the expansion in student numbers, i would much rather see limited resources available used to increase staff so we can keep class sizes down to maintain teaching quality. we lost an excellent specialist in this field because of short-sighted administration. now we have neither the time nor the facilities to use the programs he prepared for us. staff time is one problem, 'inadequate resources' is another: availability of hardware in sufficient quantity for all students to have enough access. we simply do not have the resources to utilize computer-based teaching because of the' costs of providing hardware to numbers of students who would be involved. at present we lack the hardware to make use of programs a realistic and regular part of our teaching to 200+ students per year. 9 diana laurillard et al academics' use of courseware materials: a survey a related aspect is a concern about the lack of administrative and technical support for the use of cbl: the main problem is ... maintenance (especially if printers are involved) and general care, up-dating when they become outmoded. the use of computer programs assumes total efficiency of equipment ... including technicians. this is not necessarily forthcoming. use of computers is likely to require substantial demonstrator/teacher support. apart from these essentially financial barriers to take-up, many academics raised the more technical/logistical problems of the immutability of software and the lack of standardization on hardware and software platforms. in particular, as subject knowledge develops, how will the updating of courseware be maintained? [courseware] developed elsewhere could mean difficulties in adapting to new material or data when it's available. how would it remain contemporary, i.e. this year's science? very fast-moving topic. every year i update material to reflect new perspectives. would this be practical with computer-based teaching? i am teaching in an area that is changing rapidly. computer-based teaching would only be worthwhile for a few of the basics. it is in discussing the related problem of customizing software that academics exhibit their reservations about material originating elsewhere: it is essential that materials developed leave the user free to make additions, deletions and changes, and allow the lecturer to give his/her own start to the material. alternative perspectives may be ok, but what is taught must be firmly in the hands of the course organiser ... 'authorware' support is what is needed most. the ideal computer-assisted teaching program would be customizable by staff as well as interactive with the students, allowing for local preferences. the solution anticipated in these comments is adaptable courseware, rather than opposition to any form of cbl. courseware has to adapt not only to teaching, however, but also to hardware and software environments, as many respondents recognized: the portability issue is also critical you need portability across (1) machines, (2) operating systems, (3) versions of operating systems, (4) window systems, (5) versions of window systems, (6) other conventions of the environment in which students work. these issues have always seemed prohibitive to me. it is quite easy to waste time and money developing packages that are not really much use because they are too specific to certain hardware. 10 alt-j volume 1 number 1 all the points raised in this section are fair comment. uk university teaching is undergoing rapid change, and academics are very aware of the difficulties inherent in making cbl work. all these issues need to be addressed at all levels of the educational system, and on a national basis, if academics are to be persuaded that cbl can go beyond the occasional specialist area into the mainstream of university teaching. conclusions the survey was able to show that nearly half the university teaching population already use teaching and learning technologies, but only a quarter use cbl. awareness of cbl is much higher, perhaps due to those who have had contact with their cti centre, but opinion of what they have seen is mixed. there is a general sense that cbl has some value and could be a lot better. perhaps because of this, a high proportion of cbl programs originate in the academic's own institution. this is not inevitable, however, as a large majority are interested in using and evaluating materials developed elsewhere. there is a healthy criticism of existing materials, therefore, but this does not stifle a willingness to collaborate on the development of cbl. clearly the tlt programme is timely. the danger is that scepticism about what has gone before will lead academics to act as though nothing worthwhile precedes them, and the reinvention of wheels or worse, the invention of square wheels will abound, as has happened so often before in this field. the existence of the cti centres facilitates the grounding of new work in the lessons of the old. the great advantage this bestows on the new tlt programme should not be lost. the most important message from the survey, however, is the positive one that the not invented here syndrome is not a major influence on british academics considering the use of cbl. acknowledgements our thanks are due to the following individuals who assisted in the distribution of the questionnaire in their universities: sue furber, dr robert harding, phil perkins, june thompson, professor brian butterworth, annie trapp. references csup (committee of scottish university principals) (1992), teaching and learning in an expanding higher education system, edinburgh, scfc. darby, j. (1992), 'the future of computers in teaching and learning', computers in education, 19, 1/2, 193-197. 11 diana laurillard et al academics'use of courseware materials: a survey gardner, n. and darby, j. (1990), 'using computers in university teaching: a perspective on key issues', computers in education, 15, 1-3, 27-32. glick, m.d. (1990), 'integrating computing into higher education', educom review, 25, 2, 35-38. hammond, n., gardner, n., heath, s., kibby, m., mayes, t., mcaleese, r., mullings, c. and trapp, a. (1992), 'blocks to the effective use of information technology in higher education', computers in education, 18, 1-3, 155-162. kemeny, j.g. (1990), 'computers in education: progress at a snail's pace', educom review 25, 3, 44-47. mcdonough, r.w. (1991), report of the working party on the provision of computing facilities for teaching, durham, inter-university committee on computing. mullings, c. (1992), computers and communication in the humanities: a survey of use, oxford, ohc. ufc (universities funding council) (1992), beyond lectures: the report of the information systems committee courseware development working party, oxford, ctiss publications. (appendix follows...) 12 alt-j volume 1 number 1 appendix the use of courseware materials • p l | | (1-6) 1 how many courses are you teaching this year? ring one none 0 (7) one 1 two 2 three 3 four or more 4 2 how many textbooks, written by other academics, none 1 (8) are you recommending as required reading this year? one 2 two to three 3 four to six 4 more than six 5 3 have you ever used other kinds of materials slides 1 (9) developed by other academics in any of your videocassette 1 (10) courses'! please ring any you have used audiocassette 1 (11) computer-based teaching programme (see note 1 below) 1 (12) note 1: not just word-processing packages, empty spreadsheets/databases, calculation tools, analysis tools, programming languages etc. if you tinged 'computer-based program', where was it in your department 1 (13) (were they) developed? please ring as appropriate in your institution 1 (14) in the uk 1 (15) elsewhere 1 (16) 4 have you ever seen any computer-based teaching yes 1 (17) programs in your field? no 2 if yes: which of these applies? the majority are reasonably good 1 (18) the majority are rather poor 2 they are very mixed in quality 3 5 what are your views on the notion of using computer-based teaching program developed elsewhere in your teaching? please write in relation to one of the courses you teach discipline: course topic: | | (19,20) advantages: your reservations: serious disadvantages/problems: 13 diana laurillard et al academics' use of courseware materials: a survey 6a still thinking about teaching that particular course and using computer-based teaching programs developed elsewhere, to what extent do you agree with the following? ring one answer for each item i have my own way of teaching this course, and: a computer-based teaching program would be intrusive others ideas would not fit some aspects of this subject would benefit from a well-designed computer-based teaching program i would not mind my students being exposed to alternative perspectives on this subject limited resources are better spent on other forms of teaching than on computer-based teaching programs 6b have you ever had any contact with the computers in teaching initiative (cti) centre in your subject area? if yes: please specify the nature of the contact 7a if a properly funded national consortium was set up to develop computer-based teaching programs in your subject, would you be interested hi any of the following? ring any that apply collaborating with academics in your department in developing a program collaborating with academics in other institutions in developing a program evaluating programs developed by the scheme using programs developed under this scheme 7b have you any further comments on this issue? disagree srrongly 1 2 1 2 1 2 1 2 1 2 i in teaching 3 3 3 3 3 agree strongly 4 4 4 4 4 yes no 5 5 5 5 5 1 2 (21) (22) (23) (24) (25) (26) 1 1 1 1 (27) (28) (29) (30) 7c are there particular things you would like to do or see done? t h a n k you very m u c h for your help. please return your form to the survey office, institute of educational technology, the open university, p o box 175, milton keynes, mk7 6br. a prepaid label is enclosed. 14 (page number not for citation purpose) 1 *corresponding author. email: thomas.cochrane@aut.ac.nz research in learning technology 2018. © 2018 dr thomas cochrane et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2018, 26: 2195 http://dx.doi.org/10.25304/rlt.v26.2195 research in learning technology vol. 26, 2018 editorial special issue on mobile mixed reality dr thomas cochranea*, dr fiona smartb and dr vickel narayana acentre for learning and teaching, auckland university of technology; bdepartment of learning and teaching enhancement, edinburgh napier university. this special collection of research in learning technology explores the state of the art of mobile mixed reality (mmr) in education. the special collection includes eight articles that cover; a systematic review of mmr in healthcare higher education, using mobile devices for connecting people to places, and a variety of case studies of implementing mmr in educational contexts. the range of papers illustrates the emergence of mmr as a platform for designing authentic learning environments in both formal and informal learning situations. the papers also highlight a general lack of engagement with new learning theories and models in the use of mmr to design transformative learning experiences. keywords: design-based research; scholarship of technology enhanced learning; augmented reality; virtual reality; mixed reality; mobile learning introduction mobile mixed reality is part of what has become popularly termed immersive reality (peña-rios et al., 2016), encompassing a continuum from augmented reality to virtual reality. mmr is part of long established education strategies that include simulation as an established educational methodology utilised particularly in healthcare higher education (kaufman, 2010, laschinger et al., 2008). the three main limitations of simulation – authenticity, access, and cost can be mitigated by utilising the affordances of mmr enabled by the development of mobile devices into powerful multi-sensory ubiquitously connected computing devices. in particular, mmr provides a potential practical and cost-effective solution to design innovative and authentic training environments (plahouras et al., 2018, barr and foster, 2017, birt et al., 2017). augmented and virtual reality (mixed reality) is an emerging technology that bridges the gap between computer generated and real-world environments (uluyol & sahin, 2016). mobile mixed reality enables the design of authentic learning environments that explore the impact of socio-culture influences, and lead to deeper student engagement with the real world via digitally enhanced gamified environments. this is illustrated by the phenomenon of pokemon go, and emergent mobile mixed reality projects with the likes of the microsoft hololens and google cardboard. the use of ar and vr to support teaching and learning has shown to have many advantages which include enhancement of learning achievement in terms of enhanced learning outcomes, motivation and engagement (bacca, baldiris, fabregat, graf,  &  kinshuk,  2014). mailto:thomas.cochrane@aut.ac.nz http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v26.2195 dr thomas cochrane et al. 2 citation: research in learning technology 2018, 26: 2195 http://dx.doi.org/10.25304/rlt.v26.2195 (page number not for citation purpose) interest  in  the application of vr and ar have increased dramatically over the last few years (akçayır & akçayır, 2017). however due to the relatively emerging nature of these technologies the implications of these tools in education are still largely still being explored. the rlt special collection on mmr builds upon the publication (november 2017) of a special issue of ajet guest edited by dr thomas cochrane and dr helen farley, with the theme: “exploring mobile augmented and virtual reality in higher education” (cochrane and farley, 2017). in this special collection we explore the design of educational experiences that integrate these mmr technologies into both formal and informal learning environments. mobile mixed reality in education ar and vr are currently two of the hottest topics in emerging mobile learning technologies. coupled with the ubiquity of mobile device ownership, mobile ar and vr provide the potential to enhance educational technology research and practice on a level previously impractical. by focusing upon design-based research the special issue focuses on new emancipative pedagogies beyond adding another technology fad for content delivery and teacher-directed pedagogies. design based research facilitates the design of authentic innovation in teaching and learning rather than research that is technology driven or merely substitutionary (rushby, 2012, traxler, 2016, bannan et al., 2015). mobile devices have been considered as especially well-suited to engaging with virtual and augmented environments due to their accessibility, portability and their ability to immerse students into their own personal environment but still engage with others (furió, gonzález-gancedo, juan, seguí, & costa, 2013). due to the pervasive adoption of mobile technology the focus on mobile devices and a byod (bring your own device) student-owned model ensures inclusivity (traxler, 2010, 2016a). many of the emergent mobile ar and vr projects such as google expeditions include the potential for international collaborations and partnerships. for example, mobile mixed reality enables students to experience authentic scenarios, field trips, and simulations from across the globe. this special collection explores some of the emergent examples of the design of mmr learning environments. in this issue here we explore the emergent themes drawn from the eight articles included in this special collection. triangulating the impact of mmr learning through biometric feedback in their paper “embodied reports in paramedicine mixed reality learning” aguayo et al., (2018) describe a pilot project that is part of a design based research project exploring the impact of mobile vr simulation learning environments in paramedicine education utilizing both self-reported and biometric feedback data. they highlight the shift in thinking behind the project through the dbr iterations from a focus upon vr to a focus upon mixed reality (mr) through the addition of biometric feedback that anchors the project in both the virtual and real spaces. they argue for the http://dx.doi.org/10.25304/rlt.v26.2195 research in learning technology citation: research in learning technology 2018, 26: 2195 http://dx.doi.org/10.25304/rlt.v26.2195 3 (page number not for citation purpose) value of incorporating biometric feedback to enhance the design of authentic learning experiences through mobile mixed reality. evaluating learning in virtual reality in their paper “learning in virtual reality: effects on performance, emotion, and engagement”, allcoat and von muhlenen (2018) compared the impact of three different modes of learning for students, including: traditional textbook-style, vr, and video. they found that overall, participants found vr learning an improved learning experience compared to traditional textbook and video learning methods, resulting in longer-term benefits for learners. however, they caution that the results may be partially due to the novelty factor of vr, and longitudinal studies are required to explore this further. assessing mobile mixed reality affordances in their paper “assessing mobile mixed reality affordances as a comparative visualization pedagogy for design communication”, birt and cowling (2018) use a design based research methodology to explore mixed reality approaches to enhancing design student spatial recognition skills. they evaluate the impact and use of a range of mobile comparative visualization technologies within a first-year architectural design classroom. their findings highlight the benefit of mr visualisation for exposing layers of complex architectural designs, and how these visualisations push the limits of current mobile technology in comparison to dedicated non-mobile devices. future research plans are to explore the potential of designing collaboration within these mr learning environments. designing authentic interdisciplinary health education scenarios in their paper “authentic interdisciplinary health education scenarios using mobile vr”, cochrane et al., (2018) explore the potential of mobile mixed reality environments for designing authentic interdisciplinary team interaction that mirror professional practice but are difficult to achieve in the typically dislocated university department structure where each health discipline is separated. they present a design based research methodology that leverages mobile social media to produce an agile, cost-effective and scalable model to design mmr learning environments for transdisciplinary team collaboration. trialing the hololens in their paper “holographic learning: a mixed reality trial of microsoft hololens in an australian secondary school”, leonard and fitzgerald (2018) evaluate the potential of the hololens in education through a pilot phase of a design based research project. they found the main limitation of ar and vr technologies in secondary education is the dominant default use of the technology to design for teacherdirected pedagogy or isocratic learning. the authors argue that their observations of the pilot project indicate a general lack of engagement of teachers with diverse http://dx.doi.org/10.25304/rlt.v26.2195 dr thomas cochrane et al. 4 citation: research in learning technology 2018, 26: 2195 http://dx.doi.org/10.25304/rlt.v26.2195 (page number not for citation purpose) models for designing learning experiences and environments that allow for student customisation and uncertainty in the learning experience. in response the authors appeal for teachers to utilise new technologies to design transformative learning experiences, moving beyond substitutionary use of new technologies for existing pedagogical practice. using mobile devices for connecting people to places in their paper “the boundaries of education: using mobile devices for connecting people to places”, sinfield (2018) explores the use of mmr technologies to mediate learning outside of formal classroom environments through a sociocultural lens that links a community to its history. the paper outlines a potentially transferable framework for using mmr to design authentic informal learning environments. in this project mmr is used to capture and bring to life significant aspects of a communities past, linking the past to the present in the design of a highly interactive learning environment. the author argues that mmr technologies can be used by sociocultural researchers in a variety of contexts and informal learning environments. a systematic review of mmr in health education in their paper “exploring mobile mixed reality in healthcare higher education: a  systematic review”, stretton et al., (2018) describe the results of a systematic review of the literature surrounding mobile learning and mixed reality in healthcare higher education. from an initial 1484 studies 18 were identified that met the inclusion criteria focused on “healthcare”, “higher education”, “mobile learning” and “mixed reality”. their critique of these studies indicates that mmr is predominantly utilised in healthcare education for clinical simulation exercises, with a focus upon skill development rather than higher level diagnosis or higher order clinical reasoning. consequently, the authors identify that there is little evidence of engagement with learning theory in the design of mmr for hhe, and a subsequent reliance upon behavioural learning design. immersive mobile displays in their paper “using immersive mobile displays for observational learning of motor skills. a design research approach enhancing memory recall and usability”, watson and livingston (2018) explore the effectiveness of mmr platforms to facilitate observational learning of action sequences. the paper evaluates the development of the recovery position application that displays two virtual avatars performing the steps of the recovery position as an example of mmr technologies used for virtual demonstrations. based upon their research, the authors develop guidelines for virtual demonstration to supplement instructor demonstrations beyond simple recorded videos of demonstrations. the authors argue that 3d virtual environments can provide effective learning environments for specific training experiences. http://dx.doi.org/10.25304/rlt.v26.2195 research in learning technology citation: research in learning technology 2018, 26: 2195 http://dx.doi.org/10.25304/rlt.v26.2195 5 (page number not for citation purpose) acknowledgements the special issue guest editors would like to express our thanks to the rlt editorial team for allowing us to explore a special issue on mmr. special thanks also go to our team of reviewers and authors for getting the special issue through in such a short time-frame. references aguayo, c., dañobeitia, c., cochrane, t., aiello, s., cook, s. & cuevas, a. 2018. embodied reports in paramedicine mixed reality learning. research in learning technology, 26. allcoat, d. & mühlenen, a. v. 2018. learning in virtual reality: effects on performance, emotion, and engagement. research in learning technology, 26. bannan, b., cook, j. & pachler, n. 2015. reconceptualizing design research in the age of mobile learning. interactive learning environments, 24, 1-16. barr, n. & foster, j.-a. 2017. using consensus group methods to improve simulation using immersive media in paramedicine. journal of paramedic practice, 9, 121-125. birt, j. & cowling, m. 2018. assessing mobile mixed reality affordances as a comparative visualization pedagogy for design communication. research in learning technology, 26. birt, j., moore, e. & cowling, m. 2017. improving paramedic distance education through mobile mixed reality simulation. australasian journal of educational technology (ajet), 33, 69-83. cochrane, t. & farley, h. 2017. editorial: volume 33 special issue on mobile ar & vr integrating sotel in learning design. australasian journal of educational technology (ajet), 33. cochrane, t., stretton, t., aiello, s., britnell, s., cook, s. & narayan, v. 2018. authentic interprofessional health education scenarios using mobile vr. research in learning technology, 26. kaufman, d. 2010. simulation in health professional education. in: kaufman, d. & sauvé, l. (eds.) educational gameplay and simulation environments: case studies and lessons learned. hershey, pa, usa: igi global. laschinger, s., medves, j., pulling, c., mcgraw, d. r., waytuck, b., harrison, m. b. & gambeta, k. 2008. effectiveness of simulation on health profession students' knowledge, skills, confidence and satisfaction. international journal of evidence-based healthcare, 6, 278-302. leonard, s. n. & fitzgerald, r. n. 2018. holographic learning: a mixed reality trial of microsoft hololens in an australian secondary school. research in learning technology, 26. peña-rios, a., pirker, j., gütl, c. & gardner, m. 2016. editorial: preface to the special issue on immersive environments: challenges, research and new developments. eai endorsed transactions on future intelligent educational environments, 2, 1-2. plahouras, j., khan, r., johnston, b. c., scaffidi, m. a., grover, s. c. & walsh, c. m. 2018. pth-140 virtual reality simulation training in gastrointestinal endoscopy: a cochrane systematic review. gut, 67, a275. rushby, n. 2012. editorial: an agenda for mobile learning. british journal of educational technology, 43, 355-356. sinfield, d. 2018. the boundaries of education: using mobile devices for connecting people to places. research in learning technology, 26. stretton, t., cochrane, t. & narayan, v. 2018. exploring mobile mixed reality in healthcare higher education: a systematic review. research in learning technology, 26. traxler, j. 2016. what killed the mobile learning dream? [online]. available: https://www.jisc. ac.uk/inform-feature/what-killed-the-mobile-learning-dream-26-feb-2016? watson, p. & livingstone, d. 2018. using mixed reality displays for observational learning of motor skills: a design research approach enhancing memory recall and usability. research in learning technology, 26. http://dx.doi.org/10.25304/rlt.v26.2195 https://www.jisc.ac.uk/inform-feature/what-killed-the-mobile-learning-dream-26-feb-2016? https://www.jisc.ac.uk/inform-feature/what-killed-the-mobile-learning-dream-26-feb-2016? (page number not for citation purpose) 1 *corresponding author. email: msahal@yildiz.edu.tr research in learning technology 2020. © 2020 m. sahal and a. s. ozdemir. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 research in learning technology vol. 28, 2020 original research article pre-service primary teachers’ views and use of technology in mathematics lessons muhammet sahala* and ahmet sukru ozdemirb adepartment of mathematics and science education, yildiz technical university, istanbul, turkey; bdepartment of mathematics and science education, marmara university, istanbul, turkey received: 16 july 2019; revised: 18 march 2020; accepted: 20 may 2020; published: 7 august 2020 pre-service teachers who are future practitioners of the curriculum cannot be considered independent of their views on education and technology. the goal of this study is to determine the use of technology and the opinions of pre-service primary teachers (ppts) regarding the use of technology in classroom activities in mathematics lessons. the research was conducted with 62 ppts studying in a state university. the study is based on a case study. the ppts designed and implemented activities with respect to the objective(s) in the primary school mathematics-teaching programme. these activities were observed and recorded in video. at the end of the semester, the opinion form prepared by the researchers was applied to the ppts. descriptive statistics, descriptive analysis and content analysis methods were used in the analysis of the data. according to the findings of the research, almost all of the ppts expressed opinions about the positive and negative aspects of technology usage related to education. furthermore, while 83.86% of the ppts indicated that they wanted to use technology effectively in their professional lives in the future, only 19.35% of the observed activities benefited from the technology. ppts advocated two main reasons for not using technology in classroom activities. the first was that concrete material is more effective where physical conditions are inadequate and the difficulty in accessing materials, especially at schools in rural areas. the second main reason concerned time constraints while following the curriculum. keywords: primary mathematics education; technology; teacher education; classroom activity; technology integration introduction the continuous change in the field of technology has influenced applications in different areas. nowadays, technological tools are used in many areas such as security, health, industry, economics and in almost all institutions. it is an indispensable fact that technology has become a prerequisite for the quality of a job (ertmer and ottenbreit-leftwich 2010). there is almost no day for an individual where she/he is not dealing with technology. mathematics education is also influenced by these developments mailto:msahal@yildiz.edu.tr http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v28.2302 m. sahal and a. s. ozdemir 2 citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 (page number not for citation purpose) because of the relationships between mathematics and mathematics education, with technological developments (aydın 2005). integration of the technology into an educational context is becoming more important every day (kim et al. 2013; smith, kim, and mcintyre 2016; taimalu and luik 2019). most educators, teachers and researchers have an opportunity to develop new perspectives on students’ learning concerning mathematics with advances in technology (drijvers et al. 2016). one of the ways to achieve high quality in teaching is to enrich educational environments with technology (taimalu and luik 2019). many countries have made large investments for the use of technology in education (the organization for economic co-operation and development [oecd] 2013), so many schools have access to different technological tools used for mathematics courses or general purposes (hoyles 2018). the national council for the teachers of mathematics [nctm] (2014) emphasised the idea that technology will support effective teaching and facilitate meaningful learning and that technology could be used appropriately to help students understand mathematics, carrying out reasoning and improving communication skills. studies have shown that the use of technology in mathematics classrooms is beneficial for students’ conceptual learning, positive attitudes and the development of metacognitive skills (drijvers et al. 2016; harju, koskinen, and pehkonen 2019; olkun, altun, and smith 2005; sinclair 2004). in this regard, it was stressed that technology should be used in primary mathematics education (higgins, huscroft-d’angelo, and crawford 2019; kersaint et al. 2003). sarama et al. (2012) stated that if activities are designed and implemented well, computers provide many benefits for young children both psychologically and pedagogically. teachers as practitioners have a great responsibility in order to integrate technology into education. today, teachers need to be informed about how to use technological tools in educational environments, how it affects students’ conceptual knowledge and skills, and the potential problems that may arise. in other words, today’s teachers should have technological skills in their repertoire to use in educational settings (kersaint et al. 2003) because the potential of technology only emerges with the effective use of teachers (galbraith 2006). however, despite the positive views towards technology, it cannot be considered that it could be fully integrated into learning environments (angeli and valanides 2009; cheok et al. 2016; ertmer and ottenbreit-leftwich 2010; kim et al. 2013; martin 2018; taimalu and luik 2019). this is mainly due to the traditional and teacher-centred understanding of education, as well as too much focus on technical skills (angeli and valanides 2009; atasoy, uzun, and aygun 2015; avci and coskuntuncel 2018; aydin 2005; ertmer and ottenbreit-leftwich 2010). before these barriers, it was emphasised that teachers should have positive beliefs and opinions about the use of technology (lin 2008; sedoyeka 2012). the examination of beliefs is thought to be determinant in the decision-making process of teachers in order for technology to successfully operate in teaching environments (ertmer and ottenbreit-leftwich 2010). ertmer and ottenbreit-leftwich (2010) pointed out that four key components, including teacher knowledge, self-efficacy, school-subject culture and pedagogical beliefs, are essential for the integration of technology. among these components is first of all, it is necessary that teachers believe that the activities with educational technology carried out in teaching environments are beneficial (ertmer and ottenbreit-leftwich 2010). taimalu and luik (2019) and ertmer (2005) considered the belief factor as the ‘root concept’ and described it as the basis for all other factors. beliefs, according to rokeach (1972), are defined as phrases beginning with ‘i believe that...’ (as cited in ertmer and ottenbreit-leftwich 2010). it is emphasised that beliefs which http://dx.doi.org/10.25304/rlt.v28.2302 research in learning technology citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 3 (page number not for citation purpose) deeply affect individuals’ knowledge, thinking processes and behaviours, are resistant to change (pajares 1992). in this respect, the beliefs carried by teachers are one of the major obstacles for the integration of technology into learning environments (mama and hennesy 2013), so it has been seen as one of the first problems to overcome in order to integrate technology into educational learning environments (taimalu and luik 2019; wachira and keengwe 2011). ertmer (1999) also classified the barriers for the integration of technology as external barriers, which are related to technical problems, and internal barriers such as teachers’ opinions, thoughts and beliefs. ertmer (1999) stressed that internal barriers are more important because they are more difficult to overcome. the teachers’ undergraduate education periods have been considered important in the development of teachers’ technological skills and beliefs about the use of technology (kersaint et al. 2003). because it was seen that pre-service teachers who did not have enough experience in teaching with technology in their undergraduate years did not feel comfortable using technology in an educational context in their future careers (lin 2008). it is not likely that these pre-service teachers use technology in educational situations in their profession effectively. technological tools such as computers, smart phones, tablets and different applications have become more and more widespread recently. however, pre-service primary teachers (ppts) need to be competent in how these technologies should be transformed in the context of specific educational objectives because it may not be possible to create a specific learning environment suitable for every class, every purpose, every student or every school (mishra and koehler 2006). being good at the use of technological tools may not guarantee technology integration. different approaches have been suggested in order to offer insights into technological integration in an educational context. hooper and rieber’s (1995) theoretical framework, which is one of the initial suggestions, describes the basic steps needed to be considered for integrating technology into instruction. this theoretical framework attempts to provide simpler explanations about technology integration when compared to others (jang 2019). for this reason, hooper and rieber’s framework was thought to be appropriate for the current study. in this respect, hooper and rieber (1995) distinguished between educational technology and technology in education. they emphasised that the quality of technological tools or how they are used (educational technology) in the classroom is important, and not the quantity (technology in education). similarly, drijvers (2013) stated that teachers have a key role for the effective use of technologies in educational contexts . hooper and rieber (1995) proposed a five-stage theoretical framework for the use of technology in educational environments. in this framework, there are five different phases of technological use from simple to complex. these phases are familiarisation, utilisation, integration, reorientation and evolution, and are described as follows: the familiarization phase involves basic knowledge to learn how to use simple programs for technology. in the utilization phase, the teacher attempts to use technology in the classroom. this type of usage is simple and limited. during the integration phase, the teacher uses technology consciously in the teaching phase. however, when encountering some negativity unexpectedly, it cannot continue to be taught in a planned manner. this stage can be perceived as the equivalent of the blackboard in the electronic environment. in the reorientation phase, the teacher can integrate technology with the teaching environment consciously and purposefully. if the teacher is in a redirection phase, students are able to be in http://dx.doi.org/10.25304/rlt.v28.2302 m. sahal and a. s. ozdemir 4 citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 (page number not for citation purpose) the guide position when creating their own information. in other words, students in the classroom can be called the driver’s seat during the learning phase. for a teacher in the final stage of development, change is not a frightening feature. by considering the conditions, the learning environment is integrated with technology in a student-centered manner. (hooper and riebers 1995, pp. 156–158). as mentioned earlier, it has been noted that the experiences of pre-service teachers in their undergraduate years will affect their future educational activities (unlu and sarpkaya aktas 2017). in this sense, teachers and their professional development play a key role in the effective use of technology in educational learning environments (hoyles and lagrange 2009; pamuk et al. 2013). most of the studies conducted with teachers and pre-service teachers have been aimed at determining attitudes, opinions or beliefs towards technology (aktas et al. 2014; ertmer 2005; pamuk et al. 2013; smith, kim, and mcintyre 2016; usta and korkmaz 2010). it was emphasised that pre-service teachers should be given opportunities to carry out their own technology-supported instructions (willis 2001). shin et al. (2009) stated that there are many studies on technology in a teacher’s education, and the emphasis on the studies in classroom observations and interviews can be useful in this area because more study is needed to understand how pre-service teachers use technology in their teaching processes (liu 2016). similarly, in this study, the aim is to reveal the opinions of the ppts in the use of technology in their classroom activities. additionally, the other goal of this research is to determine how ppts use technology in instructional settings. besides, if the ppts are not using technology in classroom activities, then the probable reasons underlying this are the focus of the current study. this aspect of the study is thought to contribute to the literature. for this purpose, the following research questions were determined: 1. what is the current status of ppts’ technology usage in classroom activities? 2. what are the opinions and beliefs of the ppts on the use of technology in classroom activities for mathematics lessons? 3. what are the reasons of ppts concerning the non-use of technology in classroom activities during mathematics lessons? method in this study, the case study model was adopted. the case study is a research methodology that investigates a contemporary phenomenon in depth and within its own context and provides researchers the opportunity to make an in-depth and holistic examination of a phenomenon (yin 2015). participants this research was conducted on 62 ppts enrolled in the mathematics teaching ii course, who were from a state university in turkey. the ppts received education in turkey for 4 years at the university. according to the faculty programme, the ppts take basic mathematics 1–2 and teaching mathematics 1–2 courses related to mathematics education. the participants were in third grade at the time of the study. they took the teaching mathematics 1 course in the fall semester of that academic year. the ppts are expected to put into practice theoretical knowledge, which they have taken from teaching mathematics 1, into the teaching mathematics 2 course. http://dx.doi.org/10.25304/rlt.v28.2302 research in learning technology citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 5 (page number not for citation purpose) the teaching mathematics 1 course consists of strategies used in mathematics teaching, basic skills in mathematics education, problem solving and the development of number sense; and the teaching objectives of the curriculum such as natural numbers, fractions, decimals and the operations with them. moreover, there are computer 1 and computer 2 courses, which focus on basic and simple technological knowledge such as the introduction to information technologies, the history of computers and the basic components of computers in the pre-service teacher training programme. the participants of the study were determined through the convenience sampling method. according to this sampling method, the researcher prefers easy-to-access and inexpensive to study (patton 2001). a total of 80.64% of the ppts participating in the study were female and 19.35% were male. instruments and data collection for the first research question, the observation notes were taken by the first author, and the video recordings of the activities were performed in respect to their voluntariness. for the second and third research questions, a structured interview form with six open-ended questions was developed by the researchers. the interview form was examined by a linguist and two faculty members. after the necessary arrangements were completed, a pilot application was carried out with three ppts from different classes. the form was administrated to the ppts at the end of the semester. it was assumed that the ppts responded sincerely to these questions. the implementation continued for 14 weeks in total. the ppts were informed at the beginning of the semester. they were asked to design activities for any objectives in the primary school mathematics curriculum and to practice this activity in the classroom for 15–20 min. every week, —four to five ppts implemented activities that were designed by themselves in the classroom. during the video recordings, the first author was involved as a participant observer and field notes were taken. no intervention was carried out during the activities. throughout the implementation of the activities, the ppts performed their activities as in a real primary school class, with the rest of the class including the researcher assuming the role of the student. the distribution of the objectives in the curriculum determined by the ppts according to their wishes is presented in table 1, according to the class levels and learning domains. table 1. objectives of the activities by classes and content areas. class level f % 1st grade 13 20.96 2nd grade 20 32.25 3rd grade 15 24.19 4th grade 14 22.58 total 62 100 learning domain numbers and operations 38 61.29 geometry 7 11.29 measurement 17 27.41 data processing 0 0 total 62 100 http://dx.doi.org/10.25304/rlt.v28.2302 m. sahal and a. s. ozdemir 6 citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 (page number not for citation purpose) data analysis descriptive analysis and content analysis were used for the analysis of the data obtained through the observation notes, video recordings and interview forms. while the findings related to the first research question were analysed by using the descriptive analysis method, the analysis of the findings of the second and third research questions were conducting using the content analysis method. the answers of the ppts’ were encoded and collected under certain themes. for example, the opinions of the advantages and disadvantages of technology were categorised as ‘visualisation’, ‘saving’, ‘easy accessibility’, ‘concretisation’, ‘reinforcement’, ‘ technical difficulties’, ‘course content’, ‘lecture process’ and ‘attitude’. in determining codes and themes, the studies of aktas et al. (2014) and ocal and simsek (2017) were utilised. the analysis of the observations and videos was done based on the theoretical framework for the use of technology in educational environments developed by hooper and rieber (1995). after the ppts completed their activities, a structured interview with six open-ended questions was administrated. video recordings, observation notes and responses to the interview were examined by the authors separately. the observation notes of the first author were checked by the second author. after this procedure, comparisons were made with the findings and the analyses were clarified (figure 1). to ensure the validity and reliability of qualitative research, it is necessary to take different precautions from quantitative research (creswell 2009). the following precautions were taken to ensure internal validity and external validity in the study: 1. to increase the validity and reliability in qualitative research, it is important to spend considerable time gathering information (creswell 2009). for the current study, the duration of the study lasted 14 weeks, so it is assumed that the time was sufficient to gather knowledge. 2. triangulation (observation notes, video recordings and interview form) was done to confirm the findings obtained from the different sources. 3. the answers to the interview form of the ppts were transferred without any changes. figure 1. data collection and analysis process. http://dx.doi.org/10.25304/rlt.v28.2302 research in learning technology citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 7 (page number not for citation purpose) 4. the findings were evaluated by different researchers and consensus between the researchers was found to be 87.56% for the interview. there was a consensus between the authors about the levels of ppts according to the theoretical framework of hooper and rieber (1995). 5. the technology dimension of the research was mentioned in order for the ppts not to feel pressure to use technology during the activity planning and implementation process. results ppts’ usage of technology in classroom activities for mathematics lessons during the implementation of the activities, it was observed that 19.35% of the ppts used technology, but 80.64% did not use technology. seven of the participants (11.29%) used a powerpoint presentation, four (6.45%) used music playback and one (1.61%) used a video demonstration. the ppts stated that they used the music playback or video demonstration at the beginning of the activity in order to raise students’ interests and motivate them in the activities. it can be said that the participants intended to use these technologies consciously when designing the event. the ppts, who used the powerpoint presentation in their activities, were observed to use technology like a blackboard. in figures 2 and 3, images from the participants using the powerpoint application in their activities are presented. figure 2. activity footage from ppt 8. table 2. ppts’ use of technology in classroom activities. was technology used during the activity? f % yes powerpoint presentation 7 11.29 music playback 4 6.45 video presentation 1 1.61 no 50 80.64 total 62 100 http://dx.doi.org/10.25304/rlt.v28.2302 m. sahal and a. s. ozdemir 8 citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 (page number not for citation purpose) the answers to the question ‘do you plan to use technology effectively in your profession?’ from the interview form are presented in table 3. table 3 shows that 70.96% of the participants stated that they would benefit from technology effectively in their professional lives, 12.90% will use it in necessary situations, 7% have no idea and 4.83% will not use technology in mathematics lessons. ppts’ opinions on the use of technology in classroom activities in mathematics lessons the data obtained from the participants are displayed in two separate categories: ‘advantages’ and ‘disadvantages’. according to table 4, the participants mostly indicated the following advantages of technology; time savings (50.00%), attention-catching (41.93%), permanent learning table 3. ppts’ responses concerning the use of technology in their professional lives. category f % examples of ppts’ answers yes 44 70.96 yes. i think it would be useful for the students. yes. i definitely think so. yes. the positive side of technology is quite high. i’m thinking it will definitely yield results. for students’ permanent and effective learning, yes. it’s the technology age so, for that reason, i think so. when required, yes. 8 12.90 in some situations, yes. i’m thinking about using it as required. depending on the topic, yes. yes, if necessary. but i would not use technology in every lesson. no 3 4.83 i don’t think it would be effective in mathematics. yes, but not in math. no opinion 7 11.29 i have no idea. total 62 100 figure 3. activity footage from ppt 58. http://dx.doi.org/10.25304/rlt.v28.2302 research in learning technology citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 9 (page number not for citation purpose) (33.87%), rapid and easy access to information (32.25%), visualisation (27.41%) and a simulation opportunity (25.80%). examining table 5, the lack of technological knowledge concerning the disadvantages of technology (27.41%) was the most cited reason, followed by the cost of technology (22.58%); and the inability to provide sufficient opportunities in all environments (19.35%), difficulties in classroom management (19.35%) and non-purpose usage (19.35%) were cited. reasons for not using technology in classroom activities in mathematics lessons if the data in tables 2 and 3 are handled together while most of the ppts (83.86%) would like to use technology in their teaching professions, only 12 ppts (19.35%) used technology in the activities. these findings show that there is an inconsistency between the opinions and practices of the ppts. the findings of the third research question are presented in this section to indicate the underlying reasons for the activities, which do not involve technology, performed by the ppts. according to table 6, the prominent idea by the ppts is that ‘concrete material is more effective for teaching’ (54.83%) as the reason for not using the technology in the activities. also, it was found that the ppts reported that they used technology during the preparation phase (14.51%) and they think that they will work in schools in rural areas (9.67%) in their professional lives. table 4. opinions of ppts on the advantages of technology. categories codes f % opinions about course content provides visualisation. 17 27.41 facilitates understanding/learning. 10 16.12 situations that are not possible in the class environment can be shown. 16 25.80 provides concretisation. 10 16.12 provides persistent learning. 21 33.87 it makes reinforcement easier. 8 12.90 opinions about the instruction process saves time. 31 50.00 it appeals to a wider audience. 7 11.29 it allows you to provide various activities and examples at the same time. 10 16.12 it appeals to multiple senses. 8 12.90 complements the teacher’s shortcomings. 2 3.22 enables active participation. 4 6.45 provides opportunities for interactive environments. 7 11.29 it facilitates individual learning. 5 8.06 opinions about attitudes attention/attraction. 26 41.93 it is motivating. 7 11.29 it makes the lesson fun. 6 9.67 other opinions it provides easy and quick access to information. 20 32.25 it contributes to creativity. 4 6.45 contributes to media literacy. 1 1.61 it allows socialisation. 2 3.22 http://dx.doi.org/10.25304/rlt.v28.2302 m. sahal and a. s. ozdemir 10 citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 (page number not for citation purpose) some of the answers from the ppts with respect to the third research question are as follows: ppt 28: i did not use technology. i don’t think it could be very useful in a math class. i believe that children will learn better from more tangible, hand-held, and real life examples. ppt 52: i did not prefer to use technology because i’m going to have trouble accessing a computer and the internet in the village. for this reason, i made materials from equipment that i could easily access. ppt 22: i did not want to use technology because my subject was the number patterns listed within a certain difference. i don’t think i can just use slides in the classroom to keep the attention alive. ppt 39: i didn’t use technology in my presentation. however, when i was preparing for the presentation, i took advantage of technology. i did some research. table 5. opinions of ppts on the disadvantages of technology. categories codes f % views on the technical aspect it can lead to a loss of time. 9 14.51 electronic devices may break down 7 11.29 expensiveness 14 22.58 there may be no technology in every environment. 12 19.35 opinions about teacher class control can be difficult. 12 19.35 the teacher may not be fully able to use the technology. 17 27.41 passivates the teacher/the teacher may cut corners 12 19.35 opinions about student could passivate the student 5 8.06 can be distracting 12 19.35 may cause health problems 6 9.67 may cause non-purpose use 12 19.35 it can lead to technology addiction. 9 14.51 could cause laziness in the student 7 11.29 it may not be appropriate for every subject and in every situation. 2 3.22 other there may be issues related to the security of the information source. 7 11.29 it can adversely affect socialisation. 4 6.45 table 6. reasons of ppts for not using technology. category f % the view that using concrete material is more effective 34 54.83 utilising technology in lesson preparation 9 14.51 to serve in village schools 6 9.67 limited time 3 4.83 classroom management 3 4.83 technical problems 3 4.83 not to cut-corners 2 3.22 not feeling competent about technology 1 1.61 no views 2 3.22 http://dx.doi.org/10.25304/rlt.v28.2302 research in learning technology citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 11 (page number not for citation purpose) ppt 35: i did not use it because i thought there was no need for technology in the subject, and i can get away from the classroom when i’m connected to technology. ppt 55: i didn’t use it. my materials were adequate. i thought the materials i  prepared would be more effective. ppt 7: i took advantage of technology in the research phase. i made my presentation in the teaching environment with the materials i prepared. the reason for this is that i think that the more concrete things we convey, the more permanent they will be. discussion based on the findings, for the first sub-problem of the study, it can be said that the use of technology in activities was low (19.35%). batane and ngwako (2017) revealed that even though pre-service teachers had different technological competencies and positive attitudes towards using technology, the majority of them did not use it in their lesson practices. the other studies in the literature reached the same conclusion, which is technological integration into learning environments, which cannot be fully achieved (angeli and valanides 2009; cheok et al. 2016; ertmer and ottenbreit-leftwich 2010; higgins, huscroft-d’angelo, and crawford 2019; kim et al. 2013; martin 2018; taimalu and luik 2019). the teachers who benefited from technology were seen as being in the ‘utilisation’ and ‘integration’ stages of the phases that were developed by hooper and rieber (1995). the ppts stated that they intended to save time by reflecting the problems and examples in the classroom environment. for this reason, it can be said that the ppts who benefited from the powerpoint presentation were in the transition from the utilisation phase to the integration phase. because the participants used technology by projecting printed resources such as images and problems, as hooper and riebers (1995) pointed out, while also consciously planning to use technology. hooper and rieber (1995) stated that most educators use technology in the ‘integration’ phase. stobaugh and tassell (2011) expressed this situation in another way and concluded that teachers in their study had the ability to use technology in educational settings, but they had difficulties in using technology in contextual situations. aslan and zhu (2016) found that pre-service teachers use technology for demonstrative purposes at basic level. ppts used technology like blackboard as hooper and riebers (1995) stated. the results of the current study show consistency with ones done by aslan and zhu (2016), hooper and rieber (1995) and stobaugh and tassell (2011). it was obtained that while a large majority of the ppts want to use technology in their professional lives, a small group of the ppts used it in their instructions. the difference between the opinions of the ppts and their practices is noticeable. this result is consistent with the results of the studies done by smith, kim and mcintyre (2016) and liljedahl (2009). smith, kim and mcintyre (2016) revealed that teachers believed that technology is useful for students, but they needed help about how to use it. therefore, teachers are not able to reflect what they believe in their practices. liljedahl (2009) pointed out the tension between ideal and real about use of technology in education in his study also. unfortunately, the ppts are not given training at the undergraduate level to use technology effectively in their professional lives (ciftci  et al. 2013; tondeur et al. 2013). the difference between what participants think and what they put in practice may be related to the learning experiences of participants, especially with the education they receive at the undergraduate level, there are limited courses and opportunities for technology integration in teacher training http://dx.doi.org/10.25304/rlt.v28.2302 m. sahal and a. s. ozdemir 12 citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 (page number not for citation purpose) institutions. the ppts were asked to express their opinions about technology, and the answers were examined under the heading of advantages and disadvantages. concerning the advantages of technology use in an educational context, in the study of atasoy, uzun, and aygun (2015), pre-service teachers’ thoughts were revealed as permanent learning, improving students’ thinking skills, making the lesson enjoyable and simplifying. in addition, ciftci and tatar (2014) found that pre-service teachers had opinions that technology is interesting and easily accessible. the ppts have views that technology enables visualisation, concretisation, making connections between mathematics and real world, and thus, saving time (ocal and simsek 2017). smith, kim and mcintyre (2016) reached parallel findings in relation to what pre-service teachers have opinions on technology use also. considering the disadvantages of technology use in educational context, similar to the current study, it was seen that the ppts presented opinions related to technical issues, classroom management, misuse and teachers’ lack of technological knowledge in the other studies (atasoy, uzun, and aygun 2015; ciftci and tatar 2014; ocal and simsek 2017). in light of these results, it can be said that the results obtained from the current study are in line with those obtained from other studies in the literature. besides, the studies conducted with in-service teachers reached similar findings (aktaş et al. 2014; avci and coşkuntuncel 2018; birisci and calık uzun 2014; ciftci et al. 2013; delice and karaaslan 2015; drijvers et al. 2016). it is noteworthy that the opinions of the ppts are in accordance with the opinions of the in-service teachers. in the third sub-problem of the study, the aim was to reveal the reasons why the ppts did not use technology in their activities. the beliefs of individuals for mathematics, mathematics teaching and the role of technology influence the decisions they make in educational learning environments (smith, kim, and mcintyre 2016). in mathematics classes, the abstract structure of the concepts cannot be embodied in every situation or any environment (birisci and calık uzun 2014). therefore, technology is one of the tools used for the concretisation of concepts or situations that cannot be shown in the classroom environment (aktaş et al. 2014; ocal and simsek 2017). while participants indicated that technology is advantageous for visualisation and concretisation, the idea that concrete materials are more effective than technology appears to be more dominant. this situation may be related to the learning experiences they exposed, which have an impact on the decisions they make in their teaching environment (stickles 2011; van dooren, verschaffel, and onghena 2002). the ppts enrolled in courses mainly taught by using concrete materials. using technology in learning environments was a relatively new situation for them. that is why it is possible that they are still eager to use hands-on activities during their teaching activities. unfortunately, teacher training institutes are not adequate to prepare pre-service teachers for the future in order to use technology (aslan and zhu 2016). even if the ppts have positive opinions about technology, due to not having enough experience with technology, it can be said that they do not have in-depth knowledge of how to concretise or visualise any objective by using technology. thus, ppts used technology in their teaching at utilisation and integration levels as benefiting from power point, music playback and video demonstration. secondly, it may be related to the fact that ppts will teach in elementary classrooms. students in elementary classrooms are aged 6–10 and they are at the concrete operational stage according to piaget’s framework. for this reason, the ppts may have the opinion that concrete materials are more effective than technology. besides, maschietto and trouche (2010) stated that despite developments in digital tools, the tendency to use physical tools in http://dx.doi.org/10.25304/rlt.v28.2302 research in learning technology citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 13 (page number not for citation purpose) mathematics education still exists. ppts also reported that because they have benefited from technology during the preparatory stage, they did not need to utilise technology in classroom activities. this result is similar to the results of aslan and zhu’s (2016) study in which teachers attended to use search engines to seek and evaluate information. in addition, it was seen that the ppts were concerned that they would serve in rural schools, and therefore, they thought that the lack of physical conditions would constitute an obstacle to the use of technology. similarly, batane and ngwako (2017) concluded that lack of technological tools and inadequate physical conditions are two major reasons for not using technology in classes. in this study, in parallel with the findings of wachira and keengwe (2001), other reasons, which were categorised as external or internal barriers, reported by the ppts as follows: limited time, classroom management issues and technical problems. conclusion in this study, the aim was to determine the use of technology and the opinions of ppts regarding the use of technology in classroom activities in mathematics lessons. in addition, there was an attempt to reveal the reasons underlying the ppts’ not using technology in mathematics teaching. based on the findings, it was seen that the ppts have positive views about using technology and they are willing to use technology in their future classes. nevertheless, the issue of technology integration in elementary mathematics education still remains current as a problem. the major reasons for not using technology were seen as the notion that concrete materials are more effective than technology, and using technology in the preparation phase, and technical-physical inadequacies. suggestions it is noted that new educational institutions should be intertwined with the ever changing technological world (martin 2018). for example, the increasing opportunities and improving technology movement (fatih) project in turkey was launched in 2010, and interactive boards were provided for 54 000 schools, and approximately 1 million teachers and 18 million students were affected by this project (yegitek 2017). most of the schools in turkey were equipped with technological tools via the project. that is why most of the ppts have an opportunity to use technology in their future careers. in this respect, the coordination of faculties of education at universities and public schools is important in terms of educating pre-service teachers as individuals who are able to integrate technology into learning environments. by offering practical opportunities to pre-service teachers about technological integration, their thoughts, attitudes and beliefs should be encouraged positively. limitations the study is thought to have some limitations. one of them relates to the working group. the study group had ppts from only one state university. it would be better to involve ppts from various universities. in addition, the research was limited to the primary school mathematics curriculum. therefore, research can be proposed for pre-school, middle and high school teaching programmes with different objectives. http://dx.doi.org/10.25304/rlt.v28.2302 m. sahal and a. s. ozdemir 14 citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 (page number not for citation purpose) acknowledgements this study was partly presented at the asos 5th international symposium on educational sciences in istanbul, 25–27 october 2018. references aktas, i., et al., (2014) ‘teachers’ opinions about fatih project: awareness, foresight and expectations’, necatibey faculty of education electronic journal of science and mathematics education, vol. 8, no. 1, pp. 28–46. doi: 10.12973/nefmed.2014.8.1.a11 angeli, c. & valanides, n. (2009) ‘epistemological and methodological issues for the conceptualization, development, and assessment of ict–tpck: advances in technological pedagogical content knowledge (tpck)’, computers & education, vol. 52, no. 1, pp. 154–168. doi: 10.1016/j.compedu.2008.07.006 aslan, a. & zhu, c. (2016) ‘influencing factors and integration of ict into teaching practices of pre-service and starting teachers’, international journal of research in education and science, vol. 2, no. 2, pp. 359–370. atasoy, e., uzun, n. & aygun, b. (2015) ‘investigating pre-service teachers’ technological pedagogical content knowledge in learning environment supported by dynamic mathematics software’, bartın university journal of faculty of education, vol. 4, no. 2, pp. 611–633. doi: 10.14686/buefad.v4i2.5000143622 avci, e. & coskuntuncel, o. (2018) ‘middle school teachers’ opinions about using vustat and tinkerplots in the data processing in middle school mathematics’, pegem journal of education and instruction, vol. 9, no. 1, pp. 01–36. doi: 10.14527/pegegog.2019.001. aydin, e. (2005) ‘the use of computers in mathematics education a paradigm shift from “computer assisted instruction” towards “students’ programming”’, the turkish online journal of educational technology, vol. 4, no. 2, pp. 27–34. batane, t. & ngwako, a. (2017) ‘technology use by pre-service teachers during teaching practice: are new teachers embracing technology right away in their first teaching experience?’, australasian journal of educational technology, vol. 33, no. 1, pp. 48–61. doi: 10.14742/ ajet.2299 birisci, s. & calik uzun, s. (2014) ‘mathematics teachers’ views on interactive whiteboard use in their courses: a sample of artvin province’, elementary education online, vol. 13, no. 4, pp. 1278–1295. doi: 10.17051/io.2014.19504. cheok, m. l., et al., (2016) ‘understanding teacher educators’ beliefs and use of information and communication technologies in teacher training institute’, in envisioning the future of online learning, eds. j. luaran, j. sardi, a. aziz, & n. alias, singapore, springer, pp. 11–21. ciftci, o., taskaya, s. m. & alemdar, m. (2013) ‘the opinions of classroom teachers about fatih project’, elementary education online, vol. 12, no. 1, pp. 227–240. ciftci, o. & tatar, e. (2014) ‘the comparison of the effectiveness of the using compass straightedge and a dynamic software on achievement’, journal of computer and educational research, vol. 4, no. 2, pp. 111–133. creswell, j. w. (2009) research design: qualitative, quantitative, and mixed methods approaches, 3rd ed., sage, thousand oaks, ca. delice, a. & karaaslan, g. (2015) ‘the reflection of the activities prepared on the polygons and dynamic geometry softwares to the perceptions of the teachers and the students’ performances’ karaelmas journal of educational sciences, vol. 3, pp. 133–148. drijvers, p. (2013) ‘digital technology in mathematics education: why it works (or doesn’t)’, pna, vol. 8, no. 1, pp. 1–20. doi: 10.1007/978-3-319-17187-6_8 drijvers, p., et al., (2016) uses of technology in lower secondary mathematics education; a concise topical survey, springer, new york, ny. http://dx.doi.org/10.25304/rlt.v28.2302 https://dx.doi.org/10.12973/nefmed.2014.8.1.a11 https://dx.doi.org/10.1016/j.compedu.2008.07.006 https://dx.doi.org/10.14686/buefad.v4i2.5000143622 https://dx.doi.org/10.14527/pegegog.2019.001 https://doi.org/10.14742/ajet.2299 https://doi.org/10.14742/ajet.2299 https://dx.doi.org/10.17051/io.2014.19504 https://dx.doi.org/10.1007/978-3-319-17187-6_8 research in learning technology citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 15 (page number not for citation purpose) ertmer, p. a. (1999) ‘addressing firstand second-order barriers to change: strategies for technology integration’, educational technology research and development, vol. 47, no. 4, pp. 47–61. doi: 10.1007/bf02299597 ertmer, p. a. (2005) ‘teacher pedagogical beliefs: the final frontier in our quest for technology integration?’, educational technology research and development, vol. 53, no. 4, pp. 25–39. doi: 10.1007/bf02504683. ertmer, p. & ottenbreit-leftwich, a. t. (2010) ‘teacher technology change: how knowledge, confidence, beliefs, and culture intersect’, journal of research on technology in education, vol. 42, no. 3, pp. 255–284. doi: 10.1080/15391523.2010.10782551. galbraith, p. (2006) ‘students, mathematics, and technology: assessing the present – challenging the future’, international journal of mathematical education in science & technology, vol. 37, no. 3, pp. 277–290. harju, v., koskinen, a. & pehkonen, l. (2019) ‘an exploration of longitudinal studies of digital learning’, educational research, vol. 61, no. 4, pp. 388–407. doi: 10.1080/00131881.2019.1660586 higgins, k., huscroft-d’angelo, j & crawford, l. (2019) ‘effects of technology in mathematics on achievement, motivation, and attitude: a meta-analysis’, journal of educational computing research, vol. 57, no. 2, pp. 283–319. doi: 10.1177/0735633117748416 hooper, s. & rieber, l. p. (1995) ‘teaching with technology’, in ed. a. c. ornstein, teaching: theory into practice, needham heights, ma, allyn and bacon, pp. 154–170. hoyles, c. (2018) ‘transforming the mathematical practices of learners and teachers through digital technology’, research in mathematics education, vol. 20, no. 3, pp. 209–228. doi: 10.1080/14794802.2018.1484799 hoyles, c. & lagrange, j. b. (eds.) (2009) mathematics education and technology – rethinking the terrain, the 17th icmi study, springer, new york, ny. jang, j. (2019) reımagınıng technol gınıng technology prep ogy preparatıon for pre-ser tıon for pre-servıce teachers: explorıng how the use of a vıdeo selfanalysıs ınstruc sıs ınstructıonal componen al component, based on the , based on the evıdentıal reasonıng and decısıon support model, ımpacts pre-ser s pre-servıce tea vıce teachers’ technol chers’ technologıcal pedagogıcal content knowledge, doctoral dissertation, syracuse university. kersaint, g., et al., (2003) ‘technology beliefs and practices of mathematics education faculty’, journal of technology and teacher education, vol. 11. no. 4, pp. 549–577. kim, c., et al., (2013) ‘teacher beliefs and technology integration’, teaching and teacher education, vol. 29, pp. 76–85. doi: 10.1016/j.tate.2012.08.005 liljedahl, p. (2009) ‘teachers’ insights into the relationship between beliefs and practice’, in beliefs and attitudes in mathematics education: new research results, eds. j. maaβ & w. schlöglmann, sense publishers, rotterdam, the netherlands, pp. 33–43. lin, c. (2008) ‘preservice teachers’ beliefs about using technology in the mathematics classroom’, journal of computers in mathematics and science teaching, vol. 27, no. 3, pp. 341–360. liu, p. (2016) ‘technology integration in elementary classrooms: teaching practices of student teachers’, australian journal of teacher education, vol. 41, no. 3, pp. 86–104. mama, m. & hennessy, s. (2013) ‘developing a typology of teacher beliefs and practices concerning classroom use of ict’, computers & education, vol. 68, pp. 380–387. doi: 10.1016/j. compedu.2013.05.022 martin, b. (2018) ‘faculty technology beliefs and practices in teacher preparation through a tpack lens’, education and information technologies, vol. 23, pp. 1775–1788. doi: 10.1007/ s10639-017-9680-4 maschietto, m. & trouche, l. (2010) ‘mathematics learning and tools from theoretical, historical  and practical points of view: the productive notion of mathematics laboratories’,  zdm-the international journal on mathematics education, vol. 42, no. 1, pp. 33–47. http://dx.doi.org/10.25304/rlt.v28.2302 https://dx.doi.org/10.1007/bf02299597 https://dx.doi.org/10.1007/bf02504683 https://doi.org/10.1080/15391523.2010.10782551 https://dx.doi.org/10.1177/0735633117748416 https://dx.doi.org/10.1080/14794802.2018.1484799 https://dx.doi.org/10.1016/j.tate.2012.08.005 https://dx.doi.org/10.1016/j.compedu.2013.05.022 https://dx.doi.org/10.1016/j.compedu.2013.05.022 https://dx.doi.org/10.1007/s10639-017-9680-4 https://dx.doi.org/10.1007/s10639-017-9680-4 m. sahal and a. s. ozdemir 16 citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 (page number not for citation purpose) mishra, p. & koehler, m. j. (2006) ‘technological pedagogical content knowledge: a framework for teacher knowledge’ teachers college record, vol. 108, no. 6, pp. 1017–1054. doi: 10.1111/j.1467-9620.2006.00684.x national council of teachers of mathematics. (2014) principles to actions: ensuring mathematical success for all, national council of teachers of mathematics, reston, va. ocal, m. f. & simsek, m. (2017) ‘pre-service mathematics teachers’ opinions about fati̇h project and technology use in mathematics education’, turkish online journal of qualitative inquiry, vol. 8, no. 1, pp. 91–121. doi: 10.17569/tojqi.288857 olkun, s., altun, a. & smith, g. (2005) ‘computers and 2d geometric learning of turkish fourth and fifth graders’, british journal of educational technology, vol. 36, no. 2, pp. 317–326. organisation for economic co-operation and development. (2013) lessons from pisa 2012 for the united states, strong performers and successful reformers in education. doi: 10.1787/9789264207585-en pajares, f. (1992) ‘teachers’ beliefs and educational research: cleaning up a messy construct’, review of educational research, vol. 62, pp. 307–332. doi: 10.3102/00346543062003307. pamuk, s., et al., (2013) ‘the use of tablet pc and interactive board from the perspectives of teachers and students: evaluation of the fati̇h project’, educational sciences: theory & practice, vol. 13, no. 3, pp. 1799–1822. doi: 10.12738/estp.2013.3.1734 patton, m. q. (2001) qualitative research & evaluation methods, 3rd edn, sage publications, thousand oaks, ca. sarama, j., et al., (2012) ‘the impacts of an early mathematics curriculum on emerging literacy and language’, early child research quaterly, vol. 27, pp. 489–502. doi: 10.1080/1350293 x.2010.500070 sedoyeka, e. (2012) ‘obstacles in bridging the digital divide in tanzania’, international journal of computing and ict research, vol. 6, no. 1, pp. 60–72. shin, t., et al., (2009) ‘changing technological pedagogical content knowledge (tpack) through course experiences’, in proceedings of society for information technology & teacher education international conference 2009, eds. i. gibson et al., aace, chesapeake, va, pp. 4152–4159. sinclair, m. (2004) ‘working with accurate representations: the case of preconstructed dynamic geometry sketches’, journal of computers in mathematics and science teaching, vol. 23, no. 2, pp. 191–208. smith, r. c., kim, s. & mcintyre, l. (2016) ‘relationships between prospective middle grades mathematics teachers’ beliefs and tpack’, canadian journal of science, mathematics and technology education, vol. 16, no. 4, pp. 359–373. doi: 10.1080/14926156.2016.1189624 stickles, p. r. (2011) ‘an analysis of secondary and middle school teachers’ mathematical problem posing, investigations in mathematics learning, vol. 3, no. 2, pp. 1–34. doi: 10.1080/24 727466.2011.11790301 stobaugh, r. r. & tassell, j. l. (2011) ‘analyzing the degree of technology use occurring in pre-service teacher education’, educational assessment, evaluation and accountability, vol. 23, pp. 143–157. doi: 10.1007/s11092-011-9118-2 taimalu, m. & luik, p. (2019) ‘the impact of beliefs and knowledge on the integration of technology among teacher educators: a path analysis’, teacher and teacher education, vol. 79, pp. 101–110. doi: 10.1016/j.tate.2018.12.012 tondeur, j., et al., (2013) ‘technological pedagogical content knowledge in teacher education: in search of a new curriculum’, educational studies, vol. 39, pp. 239–243. doi: 10.1080/030 55698.2012.713548. unlu, m. & sarpkaya aktas, g. (2017) ‘examination of pre-service elementary mathematics teachers’ problems posed about algebraic expressions and equations’, turkish journal of computer and mathematics education, vol. 8, no. 1, pp. 161–187. doi: 10.16949/ turkbilmat.303966. http://dx.doi.org/10.25304/rlt.v28.2302 https://dx.doi.org/10.1111/j.1467-9620.2006.00684.x https://dx.doi.org/10.17569/tojqi.288857 https://doi.org/10.1787/9789264207585-en https://dx.doi.org/10.3102/00346543062003307 https://doi.org/10.12738/estp.2013.3.1734 https://dx.doi.org/10.1080/1350293x.2010.500070 https://dx.doi.org/10.1080/1350293x.2010.500070 https://doi.org/10.1080/14926156.2016.1189624 https://dx.doi.org/10.1080/24727466.2011.11790301 https://dx.doi.org/10.1080/24727466.2011.11790301 https://dx.doi.org/10.1007/s11092-011-9118-2 https://dx.doi.org/10.1016/j.tate.2018.12.012 https://dx.doi.org/10.1080/03055698.2012.713548 https://dx.doi.org/10.1080/03055698.2012.713548 https://dx.doi.org/10.16949/turkbilmat.303966 https://dx.doi.org/10.16949/turkbilmat.303966 research in learning technology citation: research in learning technology 2020, 28: 2302 http://dx.doi.org/10.25304/rlt.v28.2302 17 (page number not for citation purpose) usta, e. & korkmaz, o. (2010) ‘pre-service teachers’ computer competencies, perception of technology use and attitudes toward teaching career’, international journal of human sciences, vol. 7, no. 1, pp. 1335–1349. van dooren, w., verschaffel, l. & onghena, p. (2002) ‘the impact of preservice teachers’ content knowledge on their evaluation of students’ strategies for solving arithmetic and algebra word problems’, journal for research in mathematics education, vol. 33, no. 5, pp. 319–351. doi: 10.2307/4149957 wachira, p. & keengwe, j. (2011) ‘technology integration barriers: urban school mathematics teachers perspectives’, journal of science education and technology, vol. 20, no. 1, pp. 17–25. doi: 10.1007/s10956-010-9230-y willis, j. (2001) ‘foundational assumptions for information technology and teacher education’, contemporary issues in technology and teacher education, vol. 1, no. 3, pp. 305–320. yegitek (directorate general of innovation and educational technologies). (2017) fati̇h project. available at: http://fatihprojesi.meb.gov.tr/en/ yin, r. k. (2015) case study research design and methods, 5th edn, sage, london. http://dx.doi.org/10.25304/rlt.v28.2302 https://dx.doi.org/10.2307/4149957 https://dx.doi.org/10.1007/s10956-010-9230-y http://fatihprojesi.meb.gov.tr/en/ (page number not for citation purpose) 1 *corresponding author. email: carl boel – carl.boel@thomasmore.be research in learning technology 2023. © 2023 c. boel et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 original research article are teachers ready to immerse? acceptance of mobile immersive virtual reality in secondary education teachers carl boela,b*, tijs rotsaertb, martin valckeb, yves rosseelb, dieter struyfa and tammy schellensb athomas more university of applied sciences, department of digital media experiences; bghent university, department of educational sciences (received: 14 july 2022; revised: 26 november 2022; accepted: 18 december 2022; published: 13 february 2023) the increasing mainstream adoption of immersive virtual reality (ivr) in education has triggered research about key variables explaining acceptance of ivr by teachers. in this study we adopted the utaut2 acceptance model as a theoretical framework enriched with the variable personal innovativeness. 379 flemish secondary education teachers watched a video about ivr learning experiences, after which an online survey concerning their perceptions was administered. general linear modeling was performed to test the hypotheses. results indicate performance expectancy, social influence, facilitating conditions, hedonic motivation and personal innovativeness to be significantly associated with behavioural intention to use. no moderating effect of age, gender or experience was observed. the results account for 54% of the variance in behavioural intention to use. the findings help to understand which factors are key in the acceptance of mobile ivr by secondary education teachers and might help defining successful ivr implementation strategies. keywords: mobile immersive virtual reality; secondary education; acceptance; utaut2; personal innovativeness; perceptions introduction immersive virtual reality (ivr) has become popular, with millions of virtual reality headsets (head-mounted displays, hmds) sold and over 16 million users (alsop 2022). this ivr rise is often attributed to improved usability and affordability (bower, de witt, and lai 2020). within ivr two types of hmds can be discerned: mobile and tethered devices. mobile ivr headsets are so-called standalone devices that generate the ivr content via a built-in smartphone. tethered devices on the other hand need a pc or laptop to push the content to the ivr headset, using a cable or wireless streaming. following teachers’ concerns about mobility of ivr technology (boel et al. 2021b; fransson, holmberg, and westelius 2020) this study will focus on teachers’ perceptions of mobile ivr solely. research in learning technology vol. 31, 2023 mailto:carl.boel@thomasmore.be http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2855 c. boel et al. 2 citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 (page number not for citation purpose) background literature on virtual reality in education following the technological advancements of ivr headsets, ivr has also caught the attention of the educational sector, which is also reflected in the european horizon programme on extended reality learning (european commission 2021). half a decade of educational research about ivr points at affordances to k–12 education fostering spatial knowledge representation, experiential learning, transfer of knowledge and students’ intrinsic motivation and engagement (di natale et al. 2020) but also for the development of critical thinking and problem-solving skills, communication skills and collaborative learning (maas and hughes 2020). a recent meta-analysis study of ivr research points at the positive impact on cognitive learning outcomes, especially for k–12 learners in science education and for rich learning experiences resulting from simulations or virtual world representations (wu, yu, and gu 2020). however, several challenges are reported in terms of ethical concerns, lack of integration in existing curricula and educational organisation constraints (fransson, holmberg, and westelius 2020; southgate et al. 2019) and the lack of instructional design principles for ivr educational applications (boel et al. 2021a). recent technological advancements push the adoption of mobile ivr systems such as meta quest, pico neo and others. these systems challenge available research evidence due to emerging new affordances, such as being standalone and mobile, potentially offering additional educational gains. research on acceptance of ivr technology as stated by venkatesh, thong, and xu (2012, p. 157) research on ‘individual acceptance and use of information technology is one of the most mature streams of information systems research’. to understand how teachers could integrate ivr technology in their classroom, it is essential to examine which factors influence their acceptance (alfalah 2018). hussin, jaafar, and downe (2011) investigated the acceptance of desktop virtual reality by 41 college teachers. effort expectancy proved to be the single significant factor predicting behavioural intention to use. chen, shih, and yu (2012) found that playfulness is more significant than perceived usefulness or perceived ease of use. the study by alfalah (2018) on the acceptance of vr by 30 higher education faculty members, pointed at a need for pedagogical training, to increase awareness within the faculty, to reduce administrative support and to support collaboration between faculty members. the study by jang et al. (2021) investigated teachers’ acceptance of augmented and virtual reality by 292 teachers adopting the tam-model extended with t-pack variables. all factors in their model proved to be of significance and made clear professional development of and support for teachers is essential in the acceptance. the need for professional development initiatives was confirmed in the recent study by mystakidis and christopoulos (2022) looking at 41 k–12 teachers’ perceptions of ivr escape rooms for stem education. a large-scale survey study of 20 876 russian teachers (khukalenko et al. 2022) showed educators consider ivr a valuable educational tool, provided they are supported by it personnel, when it does not take considerable extra effort to learn how to operate ivr devices or to plan ivr learning activities. the above studies adopted different acceptance models, ranging from tam (davis 1989) over unified theory of acceptance and use of technology (utaut) (e.g. hussin, jaafar, and downe 2011) to self-generated models (e.g. alfalah 2018; http://dx.doi.org/10.25304/rlt.v31.2855 research in learning technology citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 3 (page number not for citation purpose) khukalenko et al. 2022). however, nearly all identified factors predicting ivr acceptance by teachers can be synthetised into the factors of the utaut acceptance model (venkatesh et al. 2003) related to behavioural intention to use, namely performance expectancy, effort expectancy, social influence, facilitating conditions adding hedonic motivation from utaut2 (venkatesh, thong, and xu 2012). this is also in line with our prior qualitative, exploratory study using utaut2 to investigate which factors contribute to the acceptance of mobile ivr by secondary education teachers (boel et al. 2021b). we will now discuss the factors of utaut2 in greater detail. for a comprehensive review of utaut2 refer to venkatesh, thong, and xu (2016). performance expectancy refers to the extent to which a person believes the technology improves working conditions. this factor seems significantly associated with behavioural intention to use ivr (boel et al. 2021b; bower, de witt, and lai 2020; khukalenko et al. 2022; sagnier et al. 2020; shen et al. 2019). effort expectancy is the extent to which a person thinks efforts are needed to use a technology. teachers must learn to operate ivr, to integrate this into their curriculum and so on. previous exploratory research (boel et al. 2021b; pletz 2021) proved this to be of major concern in teachers and instructors. social influence refers to the extent someone feels influenced by others. ‘others’ can be colleagues or persons valued by teachers, such as teacher-experts, trainers, it-staff and principals. available ivr research points to the significant association with behavioural intention to use (bower, de witt, and lai 2020; jang et al. 2021; shen et al. 2019). facilitating conditions comprise a person’s feeling of being supported in his or her technology use. it refers to organisational, instrumental and infrastructural support. facilitating conditions has proven to be a key factor predicting behavioural intention to use of teachers in general (pynoo et al. 2011) and for ivr more specifically (boel et al. 2021b; bower, de witt, and lai 2020; bracq et al. 2019; khukalenko et al. 2022; pletz 2021; shen et al. 2019). resulting from these findings, we hypothesised: h1. performance expectancy is significantly associated with behavioural intention to use. h2. effort expectancy is significantly associated with behavioural intention to use. h3. social influence is significantly associated with behavioural intention to use. h4. facilitating conditions is significantly associated with behavioural intention to use. whereas utaut focuses on technology acceptance and use from the perspective of an organisation, utaut2 rather aims at individual level variables (see e.g. tamilmani, rana, and dwivedi 2021). this fits the present study because ivr is yet not adopted as a general educational tool in schools. therefore, we considered the three factors of habit, price value and hedonic motivation. habit reflects prior experiences and refers to the extent to which teachers already adopt technology in their courses (venkatesh, thong, and xu 2012). as we expected teachers do not yet integrate ivr technology in their courses at a level which would fit the construct of habit, we chose not to add this factor to our research model. although price value is another significant factor in predicting behavioural intention to use (venkatesh, thong, and xu 2012), our prior research (boel et al. 2021b) proved price value not to come into play as in general teachers are less concerned with expenses, compared to principals and it-staff. therefore, price value was not included in this study. hedonic motivation is defined as the enjoyment of the information system by the user (van der heijden 2004). the pleasure http://dx.doi.org/10.25304/rlt.v31.2855 c. boel et al. 4 citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 (page number not for citation purpose) arising from an ivr experience is one of the main attraction elements to ivr (bracq et al. 2019; bower, de witt, and lai 2020; chen, shih, and yu 2012; makransky, terkildsen, and mayer 2019; yang and han 2020). these findings led to this hypothesis: h5. hedonic motivation is significantly associated with behavioural intention to use. our prior exploratory study (boel et al. 2021b) proved the utaut2 framework to be useful, but also pointed at shortcomings. interview data from nearly all interviewees revealed the need to consider personal innovativeness in the domain of information technology (personal innovativeness [pi]). this was underpinned by other qualitative research on ivr in professional training settings by pletz (2021). agarwal and prasad (1998) defined personal innovativeness as ‘the willingness of an individual to try out new information technology’ (p. 206). personal innovativeness has proven to have a significant effect on intention to use (amid and din 2021; blut et al. 2021; cao et al. 2019; fagan, kilmon, and pandey 2012; sagnier et al. 2020; zhao et al. 2021). based on these findings, we therefore enrich the model with the factor of personal innovativeness, adding this hypothesis: h6. personal innovativeness is significantly associated with behavioural intention to use. the original utaut model put forward four moderators: age, gender, experience, and voluntariness of use. in utaut2, due to its focus on individual consumers, voluntariness of use is redundant. this is also the case in the present study since it is at this moment up to the individual teacher to adopt this technology and not yet a school based or policy choice. this led to these hypotheses: h7a. age and gender moderate the association between facilitating conditions and behavioural intention to use. h7b. age, gender and experience moderate the association between hedonic motivation and behavioural intention to use. h7c. age, gender and experience moderate the association between personal innovativeness and behavioural intention to use. as stated before, we expected mobile ivr technology is not yet integrated in most teachers’ educational practice. therefore, we limited the construct of acceptance to the factor of behavioural intention to use leaving out use as a dependent variable in this study. pulling together available ivr research resulted in a further development of the model in view of the present study as depicted in figure 1. methodology participants and data collection the present study was set up considering covid-19 restrictions. this challenged both the sampling and the presentation of an ivr experience. consequently, the study was moved completely online. participants were recruited via social media, a specialist educational e-zine in flanders, local educational technology organisations and http://dx.doi.org/10.25304/rlt.v31.2855 research in learning technology citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 5 (page number not for citation purpose) networks of the researchers. in total, 505 teachers from 26 schools responded to the call for participation. after data cleaning, 379 complete responses remained, with input to be dropped from 126 teachers due to non-completion of the full research procedure. at the start of the procedure, active informed consent was obtained and only data from consenting participants were included in subsequent analyses. table 1 summarises information about participant profiles. measures the entire study was moved online, using qualtrics. the survey started with the collection of demographic data of age, gender and school. prior experience with ivr was measured using a 7-point likert scale (‘no experience’ to ‘extensive experience’). before moving to survey items addressing their mobile ivr perceptions, table 1. demographic overview of participants. variable value frequency % gender male 144 38.0 female 235 62.0 age under 25 9 2.4 25–34 73 19.3 35–44 130 34.3 45–54 116 30.6 over 54 51 13.5 experience with ivr no experience 168 44.3 very little experience 102 26.9 little experience 57 15.0 some experience 25 6.6 moderate experience 22 5.8 a lot of experience 2 0.5 extensive experience 3 0.8 figure 1. model diagram of study on behavioural intention to use of ivr by secondary education teachers in flanders. http://dx.doi.org/10.25304/rlt.v31.2855 c. boel et al. 6 citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 (page number not for citation purpose) participants first viewed a tailor-made youtube-video. research stresses that expectations about technology cannot be assessed properly when respondents have no or little experience (hussin, jaafar, and downe 2011). covid-regulations however did not allow for teachers to experience ivr at first-hand. these restrictions were managed by presenting teachers with a video addressing the technology tools and the educational affordances of ivr. this mirrors approaches from earlier research (see e.g. shen et al. 2019). the video showed an ivr user in an empty play area of 3 by 3 meter, putting on a mobile ivr headset, taking the controllers and starting the ivr experience. this helped teachers in understanding the set-up required for mobile ivr in a classroom. next, 20 examples of educational ivr  experiences were shown, with a voice-over of one of the researchers explaining the aims and design of the experiences. the video fragments aimed to familiarise the teachers with the possible affordances of ivr for educational purposes. the examples covered 10 different course subjects, such as foreign language learning, chemistry, mathematics, assembly training, artistry and so on (figure 2). duration of the video was 4 min 39 s. research participants could not skip the video before tackling the next part of the survey: their perceptions about mobile ivr as a teaching aid in their classes. to tap into these perceptions, all survey items from the original utaut were used (venkatesh et al. 2003), adding three items (n = 3) for hedonic motivation (venkatesh, thong, and xu 2012) and four items (n = 4) for personal innovativeness (agarwal and karahanna 2000). these 27 items had to be rated on a 7-point likert scale, ranging from 1 (‘completely disagree’) to 7 (‘completely agree’). all items were randomly presented to the respondents. full overview of items can be found in the supplementary materials. figure 2. screenshots of the video used to show the participants the affordances of ivr for educational purposes. http://dx.doi.org/10.25304/rlt.v31.2855 research in learning technology citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 7 (page number not for citation purpose) results descriptive analysis analysis of the data was performed using spss28 (ibm). first, unidimensionality of the measurement instrument was tested using exploratory factor analysis. for performance expectancy one item (‘using ivr will strengthen my position’) was below the threshold of 0.50 and was deleted from the instrument. as ivr is not yet introduced at the organisational level of schools, teachers probably did not distinguish between their personal perception and the perspective at the school level. all scale items related to effort expectancy reflect satisfactory factor loadings. analysis of the items measuring social influence resulted in distinguishing between two distinct factors: social influence at a personal level (‘people who influence my behavior’, ‘people who are important to me’ and ‘people whose opinions i value’) and social influence at an organisational level (‘the school supports use of ivr’ and ‘in general, the school stimulates the use of ivr’). nonetheless, we returned to the original theoretical construct as put forward by venkatesh, thong, and xu (2012). concerning the remaining factor analyses, all items reflected high specific factor loadings for the constructs of hedonic motivation, personal innovativeness and behavioural intention to use (>0.50; standardised factor loading). one item was removed due to a too low loading: ‘ivr is not compatible with other technologies which i use’ in the construct of facilitating conditions. results of the individual item loadings can be found in the supplementary materials. next, a reliability analysis was carried out by calculating cronbach’s alpha. table 2 summarises the results, including the means (m) and standard deviation (sd). all mirror good to very good reliability, except for facilitating conditions reflecting moderate reliability. last, to account for possible contextual effects, such as school characteristics, we calculated the intra-class correlation coefficient (icc) for average measures for behavioural intention to use. based on an absolute-agreement, 2-way mixed-effects model, icc was 0.220. to address this heteroscedasticity, we used robust standard errors for the regression models. hypothesis testing in view of hypothesis testing, general linear modeling was applied. first, main effects between predictors and dependent variables were analyzed. the next stage involved the interaction effects of gender, age and experience with ivr. results are summarised in table 3. table 2. results of reliability analysis. constructs cronbach’s α means sd performance expectancy 0.90 4.79 1.36 effort expectancy 0.86 4.71 1.27 social influence 0.86 3.44 1.55 facilitating conditions 0.63 3.04 1.68 hedonic motivation 0.81 4.96 1.18 personal innovativeness 0.85 4.59 1.49 behavioural intention to use 0.96 3.39 1.70 http://dx.doi.org/10.25304/rlt.v31.2855 c. boel et al. 8 citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 (page number not for citation purpose) looking at the main effects, significant associations are observed between behavioural intention to use and the factors performance expectancy, social influence, facilitating conditions, hedonic motivation and personal innovativeness. association with effort expectancy proved not being significant. looking at the interaction effects of gender (female as comparison base), age and experience with ivr, none were significant. these results imply that performance expectancy is significantly associated with behavioural intention to use, so h1 was confirmed, as was h3 concerning social influence, h4 for facilitating conditions, h5 for hedonic motivation and h6 for personal innovativeness. effort expectancy proved not to be significant, rejecting h2. the proportion of explained variance in behavioural intention to use by the direct and indirect effects in the research model was respectively 53.5% and 54.1%. model comparison between main effects only and adding interaction effects proved not to be significant: f (8, 361) = 0.6154; p = 0.765. discussion the aim of our study was to identify which factors contribute to the behavioural intention to use mobile ivr by secondary education teachers in flanders, belgium. we adopted the utaut2-model (venkatesh, thong, and xu 2012) as a theoretical framework, and added personal innovativeness in the domain of information technology (agarwal and prasad 1998), based on prior exploratory research (boel et al. table 3. results of the cluster-robust regression models with behavioural intention to use as the dependent variable. direct effects only direct effects and interactions parameter estimates parameter estimates performance expectancy (pe) 0.161* 0.162* effort expectancy (ee) -0.055 social influence (si) 0.389** 0.390** facilitating conditions (fc) 0.288** 0.244** hedonic motivation (hm) 0.159* 0.159 personal innovativeness (pi) 0.350** 0.324** age 0.002 gender male: -0.018a experience 0.071 r2 0.535 fc × age -0.003 fc × gender male: 0.095 hm × age 0.000 hm × gender male: 0.018 hm × experience 0.002 pi × age 0.001 pi × gender male: 0.075a pi × experience 0.047 r2 0.541 a to test interaction with gender, female is used as a comparison base with value 0. *p < 0.05; ** p < 0.001. http://dx.doi.org/10.25304/rlt.v31.2855 research in learning technology citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 9 (page number not for citation purpose) 2021b; pletz 2021). results of general linear model tests were successful in explaining 54% of the variance in behavioural intention to use. in this paragraph, we discuss the results by looking for explanations and future research directions. results show that performance expectancy is significantly associated with behavioural intention to use. teachers seem to expect educational benefits of mobile ivr for their own teaching. this is in line with previous research indicating the significance of performance expectancy (e.g. fagan, kilmon, and pandey 2012; jang et al. 2021; mystakidis and christopoulos 2022; sagnier et al. 2020; shen et al. 2019). even though we were not able to immerse the participants in an actual ivr learning experience, due to covid-19 regulations, teachers nonetheless expect ivr could enhance their teaching practice. fit between educational affordances or technologies and the personal preferences of teachers seem critical to induce acceptance in teachers (e.g. mumtaz 2000). future research could investigate which of these affordances especially promoted teacher acceptance and how this is related to the currently available technological pedagogical content knowledge (abbitt 2011). tamilmani, rana, and dwivedi (2021) found that effort expectancy was only a minor significant factor in the model. therefore, it was not a surprise to see that effort expectancy is not significantly associated with behavioural intention to use and is negatively correlated with it. teachers apparently are not significantly concerned with how many efforts it would take to use ivr in their classes. this confirms earlier findings (bracq et al. 2019; sagnier et al. 2020). however, effort expectancy or the related construct of ease of use (tam) did prove to be of significance in predicting behavioural intention to use in other vr acceptance studies (bower, de witt, and lai 2020; fagan, kilmon, and pandey 2012; shen et al. 2019) and in the original utaut2 framework (venkatesh, thong, and xu 2012). probably, and as expressed by pletz (2021), the lack of exposure to an actual ivr learning experience and the lack of prior experience with it can explain for these contradictory results. when having no actual experience with ivr it is difficult to estimate the actual efforts needed. this was also explained by our own prior exploratory research (boel et al. 2021b): when teachers are engaged with an actual ivr learning application, they experience ivr is not as difficult as they had expected. social influence seems to be the most important factor in predicting intention to use. this is in line with other research (bower, de witt, and lai 2020; jang et al. 2021; shen et al. 2019). this result can also be linked to the findings of prior qualitative, exploratory research (boel et al. 2021b; pletz 2021), in that ivr use is at this moment not organised at a school level, leaving it to the personal initiative of the teacher. however, most interviewees in those studies expressed that teachers would integrate ivr in their courses when they experienced influence mainly from the principal, but also from teacher-colleagues. acceptance of educational technology seemingly cannot be restricted to the personal choice of a teacher given the conditions to implement and integrate technologies at the school level scale. this was already stressed in relation to the adoption of mobile technologies in schools (e.g. leem and sung 2019) and reflected in school readiness and teacher readiness studies for educational technology integration (e.g. petko, prasse, and cantieni 2018). the school level factors go beyond access, support and technological support. this level also introduces the importance of shared beliefs and expertise (chien, wu, and wu 2018; howard, chan, and caputi 2015). teachers expect mobile ivr can be beneficial for their teaching, but they clearly indicate the need for support. this adds to other research on ivr learning acceptance http://dx.doi.org/10.25304/rlt.v31.2855 c. boel et al. 10 citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 (page number not for citation purpose) using utaut (bower, de witt, and lai 2020; bracq et al. 2019; shen et al. 2019). when adopting a new technology such as mobile ivr, users need to feel themselves supported in implementing this technology. specifically for ivr, teachers ask for both technical, practical and pedagogical support (boel et al. 2021b). this finding reintroduces the importance to consider in future research the school level when studying acceptance because educational technology is mostly supported at the school level (niederhauser et al. 2018). although the participants were not involved in an actual ivr learning experience, they indicate the pleasure of it to be significantly associated with their intention to use. this builds on the earlier findings by bower, de witt, and lai (2020), bracq et al. (2019) and mystakidis and christopoulos (2022). we added personal innovativeness as a new predicting factor in our model. this proved to be a valid assumption, as it is significantly associated with behavioural intention to use. prior exploratory research (boel et al. 2021b; pletz 2021) found personal innovativeness to be of importance in the adoption and use of ivr technology by teachers. this is now being confirmed in this study. the results also confirm the findings from previous quantitative research (fagan, kilmon, and pandey 2012). but care must be taken with this conclusion. research by george, schwuchow, and hussmann (2019) suggests that innovativeness plays a role but should be linked to perceived usefulness. none of the moderators seemed to exert a significant effect. looking at gender, this fits the recurrent finding that gender is rather inconsistently associated with acceptance in studies. gender as a moderator can also be criticised and seen as a confounding variable: is it gender as such that plays a role or is it the gender role or gendered content of the educational technology that plays a role (e.g. iachini et al. 2016). the non-significant role of age can be explained by the overall lack of experience with ivr in all age groups, although research suggests that older respondents might be more willing due to a longer lasting fascination with the potential of new technologies (e.g. george, schwuchow, and hussmann 2019). experience proved not to play a significantly moderating role. probably the very skewed results towards no or little experience can explain this. it is however in contrast with utaut2 (venkatesh, thong, and xu 2012) and the findings of bracq et al. (2019). we believe therefore real life and first-hand experiences are critical to develop an experiential base to guide potential acceptance and adoption of technologies. this also calls for ecological valid research designs to study the current research question. even though we were successful in explaining 54% of the variance for behavioural intention to use, there are some limitations in our study. first, due to covid-19 sanitary regulations, participants could not be immersed into an actual mobile ivr learning experience. this is endorsed by shen et al. (2019) stating that ‘future studies should evaluate real vr hmd experiences to yield comprehensive findings’. second, the utaut2 remains somewhat limited as it makes abstraction of the unique features of ivr technology, such as immersion, presence, motion sickness and so on (southgate et al. 2019). the multi-level framework of technology acceptance and use (mlf, venkatesh, thong, and xu 2016) could be adopted in future ivr studies in education (mütterlein and hess 2017). third, the utaut model in origin is designed to test acceptance over a longer period of time. this was not feasible since adoption is still very low, and only few teachers have access to ivr headsets in their schools to be used over a longer time span in different courses. future research should try to incorporate this longer-term perspective as to identify which factors in the acceptance http://dx.doi.org/10.25304/rlt.v31.2855 research in learning technology citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 11 (page number not for citation purpose) alter over time and what causes this evolution. a follow-up study with the same participants could address this gap. a key limitation was uncovered when discussing the results about the impact of social influence. the utaut2 focuses strongly on the individual, whereas teachers function in and are dependent on the social context of their school. this already led to the suggestion to adopt richer models that respect the multilevel structure of factors that influence technology acceptance, such as mlf (venkatesh, thong, and xu 2016). lastly, this study focused on mobile ivr acceptance by secondary education teachers in flanders. findings should be extended to other teachers with caution, as technology acceptance studies are often limited to the specific research context (mütterlein and hess 2017). conclusion this study tried to identify which factors contribute to the acceptance of mobile ivr by secondary education teachers in flanders. analysis of our results accounted for 54% of the variance for behavioural intention to use. no moderators seemed to play a significant role. we used the utaut2 model as a theoretical framework enriched with the factor personal innovativeness. this updated model proved valuable to explain associations between factors and intention to use. future research on acceptance of ivr should therefore incorporate the construct of personal innovativeness. on a more practical level, this study can contribute to the research-informed design and development of implementation strategies for ivr in secondary education. our findings suggest the need for a pedagogical view from the school board on how ivr fits in the paradigms of the school, to facilitate teachers as much as possible, showing them the potential benefits, and led by the innovative educators. references abbitt, j. t. (2011) ‘measuring technological pedagogical content knowledge in preservice teacher education: a review of current methods and instruments’, journal of research on technology in education, vol. 43, no. 4, pp. 281–300. doi: 10.1080/15391523.2011.10782573 agarwal, r. & karahanna, e. (2000) ‘time flies when you’re having fun: cognitive absorption and beliefs about information technology usage’, mis quarterly: management information systems, vol. 24, no. 4, pp. 665–694. available at: https://www.jstor.org/stable/3250951 agarwal, r. & prasad, j. (1998) ‘a conceptual and operational definition of personal innovativeness in the domain of information technology’, information systems research, vol. 9, no. 2, pp. 204–215. doi: 10.1287/isre.9.2.204 alfalah, s. f. m. (2018) ‘perceptions toward adopting virtual reality as a teaching aid in information technology’, education and information technologies, vol. 23, no. 6, pp. 2633–2653. doi: 10.1007/s10639-018-9734-2 alsop, t. (2022) vr headset unit sales worldwide 2019–2024. available at: https://www. statista.com/statistics/677096/vr-headsets-worldwide/ amid, a. & din, r. (2021) ‘acceptance and use of massive open online courses: extending utaut2 with personal innovativeness’, journal of personalized learning, vol. 4, no. 1, pp. 57–66. available at: https://spaj.ukm.my/jplearning/index.php/jplearning/article/ view/165 blut, m., et al., (2021) ‘meta-analysis of the unified theory of acceptance and use of technology (utaut): challenging its validity and charting a research agenda in the red ocean’, jais-journal of the association for information systems [preprint]. available at: https:// papers.ssrn.com/sol3/papers.cfm?abstract_id=3834872 http://dx.doi.org/10.25304/rlt.v31.2855 https://dx.doi.org/10.1080/15391523.2011.10782573 https://www.jstor.org/stable/3250951 https://dx.doi.org/10.1287/isre.9.2.204 https://dx.doi.org/10.1007/s10639-018-9734-2 https://www.statista.com/statistics/677096/vr-headsets-worldwide/ https://www.statista.com/statistics/677096/vr-headsets-worldwide/ https://spaj.ukm.my/jplearning/index.php/jplearning/article/view/165 https://spaj.ukm.my/jplearning/index.php/jplearning/article/view/165 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3834872 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3834872 c. boel et al. 12 citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 (page number not for citation purpose) boel, c., et al., (2021a) ‘acceptance of immersive virtual reality in secondary education teachers. an explorative study of perceptions from teachers, it-staff, principals and teachers’ trainers’, inted2021 proceedings, pp. 589–596. doi: 10.21125/inted.2021.0149 boel, c., et al., (2021b) ‘six years after google cardboard: what has happened in the classroom? a scoping review of empirical research on the use of immersive virtual reality in secondary education’, 13th international conference on education and new learning technologies, pp. 7504–7513. doi: 10.21125/edulearn.2021.1524 bower, m., dewitt, d. & lai, j. w. m. (2020) ‘reasons associated with preservice teachers’ intention to use immersive virtual reality in education’, british journal of educational technology, vol. 51, no. 6, pp. 2214–2232. doi: 10.1111/bjet.13009 bracq, m. s., et al., (2019) ‘learning procedural skills with a virtual reality simulator: an acceptability study’, nurse education today, vol. 79, pp. 153–160. doi: 10.1016/j.nedt.2019.05.026 cao, j., et al., (2019) ‘the impact of personal innovativeness on the intention to use cloud classroom: an empirical study in china’, communications in computer and information science, vol. 1048, pp. 179–188. doi: 10.1007/978-981-13-9895-7_16 chen, c. y., shih, b. y. & yu, s. h. (2012) ‘disaster prevention and reduction for exploring teachers’ technology acceptance using a virtual reality system and partial least squares techniques’, natural hazards, vol. 62, no. 3, pp. 1217–1231. doi: 10.1007/s11069-012-0146-0 chien, s. p., wu, h. k. & wu, p. h. (2018) ‘teachers’ beliefs about, attitudes toward, and intention to use technology-based assessments: a structural equation modeling approach’, eurasia journal of mathematics, science and technology education, vol. 14, no. 10, pp. 1–17. doi: 10.29333/ejmste/93379 davis, f. d. (1989) ‘perceived usefulness, perceived ease of use, and user acceptance of information technology’, mis quarterly: management information systems, vol. 13, no. 3, pp. 319–340. doi: 10.2307/249008 di natale, a. f., et al., (2020) ‘immersive virtual reality in k-12 and higher education: a 10-year systematic review of empirical research’, british journal of educational technology, vol. 51, no. 6, pp. 2006–2033. doi: 10.1111/bjet.13030 european commission (2021) extended reality learning – engage and interact (ia), available at: https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/opportunities/ topic-details/horizon-cl4-2022-human-01-19 fagan, m., kilmon, c. & pandey, v. (2012) ‘exploring the adoption of a virtual reality simulation: the role of perceived ease of use, perceived usefulness and personal innovativeness’, campus-wide information systems, vol. 29, no. 2, pp. 117–127. doi: 10.1108/10650741211212368 fransson, g., holmberg, j. & westelius, c. (2020) ‘the challenges of using head mounted virtual reality in k-12 schools from a teacher perspective’, education and information technologies [preprint]. vol. 25, pp. 3383–3404. doi: 10.1007/s10639-020-10119-1 george, c., schwuchow, j. & hussmann, h. (2019) ‘fearing disengagement from the realworld: exploring non-experts’ mental model and expectations of mobile virtual reality’, proceedings of the acm symposium on virtual reality software and technology, vrst. association for computing machinery, parramatta nsw australia. doi: 10.1145/3359996.3364273 howard, s. k., chan, a. & caputi, p. (2015) ‘more than beliefs: subject areas and teachers’ integration of laptops in secondary teaching’, british journal of educational technology, vol. 46, no. 2, pp. 360–369. doi: 10.1111/bjet.12139 hussin, n. h., jaafar, j. & downe, a. g. (2011) ‘assessing educators’ acceptance of virtual reality (vr) in the classroom using the unified theory of acceptance and use of technology (utaut)’, lecture notes in computer science, vol. 7066, pp. 216–225. doi: 10.1007/978-3-642-25191-7_21 iachini, t., et al., (2016) ‘peripersonal and interpersonal space in virtual and real environments: effects of gender and age’, journal of environmental psychology, vol. 45, pp. 154–164. doi: 10.1016/j.jenvp.2016.01.004 http://dx.doi.org/10.25304/rlt.v31.2855 https://dx.doi.org/10.21125/inted.2021.0149 https://dx.doi.org/10.21125/edulearn.2021.1524 https://dx.doi.org/10.1111/bjet.13009 https://dx.doi.org/10.1016/j.nedt.2019.05.026 https://dx.doi.org/10.1007/978-981-13-9895-7_16 https://dx.doi.org/10.1007/s11069-012-0146-0 https://dx.doi.org/10.29333/ejmste/93379 https://dx.doi.org/10.2307/249008 https://dx.doi.org/10.1111/bjet.13030 https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/opportunities/topic-details/horizon-cl4-2022-human-01-19 https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/opportunities/topic-details/horizon-cl4-2022-human-01-19 https://dx.doi.org/10.1108/10650741211212368 https://dx.doi.org/10.1007/s10639-020-10119-1 https://dx.doi.org/10.1145/3359996.3364273 https://dx.doi.org/10.1111/bjet.12139 https://dx.doi.org/10.1007/978-3-642-25191-7_21 https://dx.doi.org/10.1016/j.jenvp.2016.01.004 research in learning technology citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 13 (page number not for citation purpose) jang, j., et al., (2021) ‘augmented reality and virtual reality for learning: an examination using an extended technology acceptance model’, ieee access, vol. 9, pp. 6798–6809. doi: 10.1109/access.2020.3048708 khukalenko, i.s., et al., (2022) ‘teachers’ perceptions of using virtual reality technology in classrooms: a large-scale survey’, education and information technologies [preprint], vol. 27, pp. 11.591–11.613. doi: 10.1007/s10639-022-11061-0 leem, j. & sung, e. (2019) ‘teachers’ beliefs and technology acceptance concerning smart mobile devices for smart education in south korea’, british journal of educational technology, vol. 50, no. 2, pp. 601–613. doi: 10.1111/bjet.12612 maas, m. j. & hughes, j. m. (2020) ‘virtual, augmented and mixed reality in k–12 education: a review of the literature’, technology, pedagogy and education, vol. 29, no. 2, pp. 231–249. doi: 10.1080/1475939x.2020.1737210 makransky, g., terkildsen, t. s. & mayer, r. e. (2019) ‘adding immersive virtual reality to a science lab simulation causes more presence but less learning’, learning and instruction, vol. 60, pp. 225–236. doi: 10.1016/j.learninstruc.2017.12.007 mumtaz, s. (2000) ‘factors affecting teachers’ use of information and communications technology: a review of the literature’, journal of information technology for teacher education, vol. 9, no. 3, pp. 319–342. doi: 10.1080/14759390000200096 mütterlein, j. & hess, t. (2017) ‘immersion, presence, interactivity: towards a joint understanding of factors influencing virtual reality acceptance and use’, amcis 2017 – america’s conference on information systems, 2017 august, pp. 1–10. available at: https://core.ac.uk/ download/pdf/301371675.pdf mystakidis, s. & christopoulos, a. (2022) ‘teacher perceptions on virtual reality escape rooms for stem education’, information (switzerland), vol. 13, no. 3. doi: 10.3390/ info13030136 niederhauser, d. s., et al., (2018) ‘sustainability and scalability in educational technology initiatives: research-informed practice’, technology, knowledge and learning, vol. 23, no. 3, pp. 507–523. doi: 10.1007/s10758-018-9382-z petko, d., prasse, d. & cantieni, a. (2018) ‘the interplay of school readiness and teacher readiness for educational technology integration: a structural equation model’, computers in the schools, vol. 35, no. 1, pp. 1–18. doi: 10.1080/07380569.2018.1428007 pletz, c. (2021) ‘which factors promote and inhibit the technology acceptance of immersive virtual reality technology in teaching-learning contexts? results of an expert survey’, international journal of emerging technologies in learning, vol. 16, no. 13, pp. 248–272. doi: 10.3991/ijet.v16i13.20521 pynoo, b., et al., (2011) ‘predicting secondary school teachers’ acceptance and use of a digital learning environment: a cross-sectional study’, computers in human behavior, vol. 27, no. 1, pp. 568–575. doi: 10.1016/j.chb.2010.10.005 sagnier, c., et al., (2020) ‘user acceptance of virtual reality: an extended technology acceptance model’, international journal of human-computer interaction, vol. 36, pp. 1–15. doi: 10.1080/10447318.2019.1708612 shen, c., et al., (2019) ‘behavioural intentions of using virtual reality in learning: perspectives of acceptance of information technology and learning style’, virtual reality, vol. 23, no. 3, pp. 313–324. doi: 10.1007/s10055-018-0348-1 southgate, e., et al., (2019) ‘embedding immersive virtual reality in classrooms: ethical, organisational and educational lessons in bridging research and practice’, international journal of child-computer interaction, vol. 19, pp. 19–29. doi: 10.1016/j.ijcci.2018.10.002 tamilmani, k., rana, n. p. & dwivedi, y. k. (2021) ‘consumer acceptance and use of information technology: a meta-analytic evaluation of utaut2’, information systems frontiers, vol. 23, no. 4, pp. 987–1005. doi: 10.1007/s10796-020-10007-6 van der heijden, h. (2004) ‘user acceptance of hedonic information systems’, source: mis quarterly, vol. 28, no. 4, pp. 695–704. available at: https://www.jstor.org/stable/25148660 http://dx.doi.org/10.25304/rlt.v31.2855 https://dx.doi.org/10.1109/access.2020.3048708 https://dx.doi.org/10.1007/s10639-022-11061-0 https://dx.doi.org/10.1111/bjet.12612 https://dx.doi.org/10.1080/1475939x.2020.1737210 https://dx.doi.org/10.1016/j.learninstruc.2017.12.007 https://dx.doi.org/10.1080/14759390000200096 https://core.ac.uk/download/pdf/301371675.pdf https://core.ac.uk/download/pdf/301371675.pdf https://dx.doi.org/10.3390/info13030136 https://dx.doi.org/10.3390/info13030136 https://dx.doi.org/10.1007/s10758-018-9382-z https://dx.doi.org/10.1080/07380569.2018.1428007 https://dx.doi.org/10.3991/ijet.v16i13.20521 https://dx.doi.org/10.1016/j.chb.2010.10.005 https://dx.doi.org/10.1080/10447318.2019.1708612 https://dx.doi.org/10.1007/s10055-018-0348-1 https://dx.doi.org/10.1016/j.ijcci.2018.10.002 https://dx.doi.org/10.1007/s10796-020-10007-6 https://www.jstor.org/stable/25148660 c. boel et al. 14 citation: research in learning technology 2023, 31: 2855 http://dx.doi.org/10.25304/rlt.v31.2855 (page number not for citation purpose) venkatesh, v., et al., (2003) ‘user acceptance of information technology: toward a unfied view’, mis quarterly: management information systems, vol. 27, no. 3, pp. 425–478. doi: 10.2307/30036540 venkatesh, v., thong, j. y. l. & xu, x. (2012) ‘consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology’, mis quarterly: management information systems, vol. 36, no. 1, pp. 157–178. doi: 10.2307/41410412 venkatesh, v., thong, j. y. l. & xu, x. (2016) ‘unified theory of acceptance and use of technology: a synthesis and the road ahead’, journal of the association for information systems, vol. 17, no. 5, pp. 328–376. doi: 10.17705/1jais.00428 wu, b., yu, x. & gu, x. (2020) ‘effectiveness of immersive virtual reality using head-mounted displays on learning performance: a meta-analysis’, british journal of educational technology, vol. 51, no. 6, pp. 1991–2005. doi: 10.1111/bjet.13023 yang, h. & han, s. -y. (2021) ‘understanding virtual reality continuance: an extended perspective of perceived value’, online information review, vol. 45, no. 2, pp. 422–439. doi: 10.1108/oir-02-2020-0058 zhao, y., et al., (2021) ‘do cultural differences affect users’ e-learning adoption? a meta analysis’, british journal of educational technology, vol. 52, no. 1, pp. 20–41. doi: 10.1111/ bjet.13002 http://dx.doi.org/10.25304/rlt.v31.2855 https://dx.doi.org/10.2307/30036540 https://dx.doi.org/10.2307/41410412 https://dx.doi.org/10.17705/1jais.00428 https://dx.doi.org/10.1111/bjet.13023 https://dx.doi.org/10.1108/oir-02-2020-0058 https://dx.doi.org/10.1111/bjet.13002 https://dx.doi.org/10.1111/bjet.13002 (page number not for citation purpose) 1 *corresponding author. email: damar@ppi.ac.id research in learning technology 2023. © 2023 k. j. anggoro and d. i. pratiwi. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 research in learning technology vol. 31, 2023 original research article university students’ perceptions of interactive response system in an english language course: a case of pear deck kiki juli anggoroa and damar isti pratiwib* aschool of languages and general education, walailak university, nakhon si thammarat, thailand; brailway mechanical technology study program, politeknik perkeretaapian indonesia madiun, madiun, indonesia (received: 3 november 2022; revised: 4 january 2023; accepted: 2 february 2023; published: 24 april 2023) pear deck is one interactive response system that has gained popularity in recent years. this study addressed the gap in the literature and considered students’ experience of the platform in a thai university context. this was a mixed-method study in which 320 students completed a survey including closed and open-ended components. quantitative data measuring students’ perceptions using likert-scale surveys were collected, while qualitative data were used to get a deeper understanding of students’ experience in learning using pear deck in the classroom. the data were analysed based on gender differences and students’ proficiency levels. according to the findings of this study, students’ perceptions were not significantly different based on gender. however, despite the finding that both basic and independent users had good attitudes towards the platforms, the latter group gave a substantially higher score. furthermore, the study revealed that the students had a favourable impression of pear deck. they believed that the platform was engaging, easy to use, and had the potential to aid learning. keywords: english language courses; interactive response system; learning technology; pear deck; students’ perceptions. introduction technology integration has become commonplace in english language teaching around the world. it has been demonstrated that technology-mediated training promotes active learning. active learning is the polar opposite of traditional learning in that it refers to a series of learning activities that result in active engagement and discoveries by students (javed and odhabi 2018; jones and palmer 2017). active learning in english teaching can take place individually, such as via brainstorming, reading, and writing, or in groups, such as through debates and roleplaying (jones and palmer 2017; kariadi and pratiwi 2022). positive learning outcomes have been linked to active learning (el shaban 2017; kariadi and pratiwi 2022; pratiwi and waluyo 2022). active learning encourages student-centred education, which helps pupils build motivation, positive perspective, and autonomy in language acquisition (lei and lin 2022; leong et al. 2018). mailto:damar@ppi.ac.id http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2944 k. j. anggoro and d. i. pratiwi 2 citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 (page number not for citation purpose) at present, active learning has been improved by using interactive response systems (irss) such as pear deck, kahoot, and quizizz, which provide several advantages. irss have been recognised for encouraging student participation in class and assisting students in directing their attention, offering real-time feedback, and assessing students’ understanding (collins 2007; paul et al. 2020). waluyo (2020) found that students’ success across all skills was enhanced by incorporating active learning and smart classroom tools such as socrative, google forms, and facebook into basic english courses. since its debut in 2014, pear deck has grown in popularity among a wide range of users. this platform enables students to give real-time responses in various formats, including selecting from multiple choices, writing short and long passages and drawing. recent studies reported that pear deck could enhance students’ engagement in class (dong et al. 2018). most importantly, pear deck can improve students’ achievement (liu, sands-meyer, and audran 2019). however, little research on the platform in english as a foreign language (efl) contexts in asia, including thailand, has been conducted. it is critical to comprehend the experiences of its users to develop future uses of the platform (anggoro 2021). as a result, the focus of this study was on thai university students’ perceptions of pear deck in english language courses. this study also compared students’ perceptions based on gender and english proficiency levels. pear deck as an irs an irs, as defined by liu (2003), is a technology-enabled learning environment that improves student engagement. an irs, as defined by awedh et al. (2014), is an online student response system that allows teachers to test student comprehension and track their progress by assigning educational assignments. an irs enables educators to improve student participation in the classroom, stimulate deeper conversation, foster cooperation, and provide quick feedback (turner 2015). an irs is typically comprised of hardware, such as a set of simple personal handheld signal transmitters and a response signal receiver connected to a classroom computer to collect responses from classroom participants, and software configured on a classroom computer to process the collected responses and present the results on a screen (liu 2003). nonetheless, as technology advances, the irs increasingly employs mobile technologies, which are practical for students. recent irs research included the usage of students’ mobile phones (balta and hamza 2017; sun and hsieh 2018). pear deck, a platform that combines slides with interactive features, is one irs that has gained much popularity. mache et al. (2017) mentioned that pear deck acts as an online presenting platform that is combined with an interactive classroom response system. this platform has a real-time response system that enables teachers to observe students’ answers synchronously; thus, giving immediate feedback becomes possible (anggoro 2021). concerning accessibility, liu, sands-meyer and audran (2019) stated that pear deck is more convenient, simple to use, and inexpensive than other student response systems because any digital device with internet connectivity can access the platform. anggoro (2021) also states that people familiar with google slides and microsoft powerpoint can quickly learn how to operate pear deck. on the platform, students can promptly respond to the various types of questions provided in the interactive slide, and teachers can check the students’ answers http://dx.doi.org/10.25304/rlt.v31.2944 research in learning technology citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 3 (page number not for citation purpose) simultaneously through the teacher dashboard (sengsri and anggoro 2021). teachers can display the answers anonymously on the projector screen while giving verbal feedback (dong et al. 2018). pear deck also enables review sessions. teachers can access all students’ answers any time after the lesson (anggoro and khasanah 2022). at the same time, students receive takeaways which are their answers, and the slides are sent back to them through email (anggoro 2021). several studies reported on the benefits of pear deck to learning. ni et al. (2020) studied the use of the site along with a game to facilitate vocabulary instruction, and their investigation found that students’ motivation and achievement were improved. mache et al. (2017) reported that the tool could improve students’ engagement. this study is congruent with javed and odhabi (2018), who found that the platform can improve engagement and facilitate students’ active learning. in efl, pear deck was also utilised in several studies. liu, sands-meyer and audran (2019) utilised pear deck to facilitate english grammar learning and reported improved learning motivation and self-efficacy in learning english grammar. in addition, to use it in a classroom, pear deck can also enhance online instruction in terms of solving problems with students’ engagement and participation (anggoro 2021). since previous research (archambault, pagani, and fitzpatrick 2013; javed and odhabi 2018) reported that engagement relates to achievement, pear deck has the potential to improve online learning. students’ perceptions of technology tools based on gender and english proficiency levels to investigate students’ perceptions of pear deck, further, this study aims to analyse students’ comments based on their gender. even though technology has been widely employed, gender differences should be considered. according to several researchers, female students are more comfortable utilising technology tools than male pupils (pratiwi and ubaedillah 2021; shin et al. 2018). their studies revealed that women were more likely to believe that technology tools were useful. these findings contradicted several other studies (cuadrado-garcía, ruiz-molina, and montoro-pons 2010; tucker 2014). tucker (2014) found that men were more involved in discussions utilising computers as new media, as a result they gained more helpful knowledge than women who perceived computers as more of a social media tool. furthermore, cuadrado-garcía, ruiz-molina and montoro-pons (2010) discovered that young men are more familiar with virtual language than women, influencing their online performance. another finding showed no substantial differences between genders (gunamala and sneha 2013). they stated that gender disparities had little effect on their ability to use technology tools and both sexes had similar attitudes towards it. another discussion is on the connection between students’ perceptions of pear deck and their english proficiency levels. previous studies have disclosed the benefits of pear deck for english learners. the platform can be advantageous for improving students’ engagement (mache et al. 2017) and achievement (anggoro and khasanah 2022). also, it is user-friendly (liu, sands-meyer, and audran 2019). nonetheless, there is a lack of studies discussing how groups of learners with different english proficiency levels react to pear deck. by knowing the reactions of learners from different english proficiency levels, we might figure out which group might benefit more from its utilisation. http://dx.doi.org/10.25304/rlt.v31.2944 k. j. anggoro and d. i. pratiwi 4 citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 (page number not for citation purpose) methodology research objectives and design this research aimed to investigate students’ perceptions of using pear deck in english language classes. furthermore, it compared students’ perceptions of pear deck in relation to their genders and english proficiency levels. the mixed-method research design was utilised to achieve the research objectives. it was used to strengthen the triangulation of data (creamer 2017). triangulation of data helps to achieve trustworthiness (korstjens and moser 2018). hence, it is expected that the results obtained in this study are trustworthy. setting and participants the study was conducted at a university in the south of thailand. a total of 320 consented students participated in the survey. the students, aged from 19 to 21 years old, were enrolled in a course named english communication skills. the students came from eight different sections. each section approximately consisted of 35–40 students. they were diverse in relation to their english proficiency levels. to determine their english proficiency levels, their common european framework of reference for languages (cefr) levels were investigated from the results of their university english proficiency tests. cefr has been widely used worldwide and has impacted language standards, curricula, and revision reform (north 2014). the most prominent part of the framework is its vertical dimension, which includes proficiency levels ranging from basic user to proficient user (huhta 2012). most importantly, the participants were taught using pear deck slides. five classes were onsite and seven were online because of the re-rising case of covid-19 (covid19). pear deck was often utilised from the beginning to the end of the class for lesson review and language practice purposes. two lecturers designed pear deck slides for 2-h classes. the slides normally included several activities, including a warm-up activity, vocabulary building, reading or listening activity, grammar points, and speaking or writing activity. the following figure 1 shows the types of slides used in each activity: data collection an online questionnaire consisting of closed-ended and open-ended sections was utilised to collect data. a five-point likert scale was used in the closed section. the questions were about the use of irs in online english classes. for instance, students were given a statement, ‘i understand the lesson more because of pear deck’., and asked to figure 1. flow of lesson and types of pear deck slides used. http://dx.doi.org/10.25304/rlt.v31.2944 research in learning technology citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 5 (page number not for citation purpose) select an option that came closest to their opinion. the options were strongly agree (sa), agree (a), undecided (u), disagree (d), or strongly disagree (sd). more sample statements are available in the following table 1. in the open-ended section, students had the opportunity to write comments and opinions on their experiences with pear deck as well as suggestions for future use. data analysis the gathered data were held in strict confidence and only accessible to the researchers. throughout the study’s duration, participants were informed of and assured of their confidentiality. the collected quantitative data were analysed by using descriptive statistics and inferential statistics. content analysis was applied to analyse the qualitative data. the collected qualitative data were carefully and repetitively examined to discover themes, which were then classified and organised according to the research questions given for this study. results students’ perception of pear deck in an english language class table 2 shows that students’ overall perception of pear deck in an english language class is high (m = 4.32, sd = 0.70). table 3 further explains students’ perceptions based on three categories, engagement, accessibility, and achievement. the mean in each category is over 4.0, showing the students’ positive perceptions. in the open-ended section of the survey, 85% of the students left a comment, while 15% did not write anything. from 85% of the comments, 81% showed a positive reaction to the use of pear deck in class. a total of 4% students conveyed about some problems they had with the platform. students’ positive comments were categorised into three parts: engagement, convenience, and achievement. the following table 4 contains some students’ comments based on the three categories. table 1. sample items from the survey. perceptions item samples achievement 1. i understand the lesson more because of pear deck. 2. activities on pear deck make me practice and understand more. engagement 1. i feel engaged when pear deck is used. 2. pear deck makes my class more fun. convenience 1. pear deck is convenient to use. 2. i am happy to use pear deck in my classes. table 2. descriptive statistics of students’ overall perception. descriptive statistics n minimum maximum mean std. deviation perception 320 1.00 5.00 4.3275 0.70701 valid n (listwise) 320 http://dx.doi.org/10.25304/rlt.v31.2944 k. j. anggoro and d. i. pratiwi 6 citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 (page number not for citation purpose) as previously mentioned, several students mentioned a major issue they had while using pear deck in an online class. the issue is a bad internet signal. the following are students’ comments. yes, pear deck is difficult to use because the internet is lost, and the images in the webex are not exactly in the pear deck. yes, but i have a problem with the internet. i think it is good. but stuck with the problem that the internet contract is unstable. students’ perceptions based on gender to gain more in-depth results, students’ perceptions were compared based on their gender. table 5 points out that both female and male students showed a positive attitude towards the use of pear deck. additionally, the independent samples t-test, as shown in table 6, discloses that there was no significant difference between the female (m = 4.36, sd = 0.68) and male (m = 4.20, sd = 0.76) students’ perceptions, t(318) = 1.65, p = 0.41. however, the mean of female students’ perception was slightly higher than that of the male. students’ perceptions in the open-ended section were also categorised based on their gender. the majority of both female and male students showed positive perceptions of pear deck. the comments from both genders can also be categorised table 3. descriptive statistics of students’ perception by categories. no. categories mean sd 1 engagement 4.36 0.72 2 convenience 4.32 0.72 3 achievement 4.29 0.75 table 4. students’ positive comments. no. categories comments 1 engagement 1.1. can make us excited and fun in learning. 1.2. using pear deck is a lot of fun and made the lesson more understandable. 1.3. it is good for using the app this year, study online because it can create more enthusiasm for students to study online. 2 convenience 2.1. easy to get into, easy to use. 2.2. nice because the teacher can see the student’s answer so i have no need to say it again and again due to my poor internet connection. 2.3. it is convenient, fast and easy to access. 3 achievement 3.1. activities on the pear deck made me practice and understand more, make learning more efficient 3.2. the teacher taught me vocabulary, grammar through this programme and made me understand. 3.3. it is a good app for learning english skills, including writing, listening, and drawing. http://dx.doi.org/10.25304/rlt.v31.2944 research in learning technology citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 7 (page number not for citation purpose) into engagement, convenience, and achievement. the minority (4%) who wrote about internet issues were all female students. students’ perceptions based on english proficiency levels students’ proficiency levels were categorised into two parts: the basic user or cefr a1 and a2 and the independent user or cefr b1 and b2. table 7 discloses that the mean of the independent users’ perception (m = 4.44) is higher than that of the basic user (m = 4.20). an independent samples t-test, as shown in table 8, further elaborates that there was a significant difference in the perceptions between the independent users (m = 4.44, sd = 0.53) and basic users (m = 4.20, sd = 0.84), t(318) = 2.98, p = 0.00. students’ perceptions in the open-ended section were also categorised based on their english proficiency levels: basic user and independent user. the majority of both levels showed positive perceptions of pear deck. the comments can also be categorised into engagement, convenience, and achievement. the minority (4%) who wrote about internet issues came from two different levels: 2.5% were from the basic user group and 1.5% were independent users. discussion the use of irss has become more common at present. pear deck is one platform that can be utilised in english classes. this study reports that the platform was enjoyed by the vast majority of survey respondents. most respondents agreed or strongly agreed with positive statements about pear deck in the survey. the open-ended section of the survey also mainly consisted of favourable comments on the platform. one prominent category in the findings is engagement. the findings show that pear deck has the potential to engage english learners during class. this is in line with the finding of archambault, pagani and fitzpatrick (2013) and mache et al. (2017). students also conveyed that they became more active in class when pear deck was used. this is congruent with the finding of javed and odhabi (2018) that pear deck can facilitate active learning. another category is convenience or accessibility. this category is important for irss because they are expected to make the class easier, not more complicated. a large number of the survey respondents agreed that using pear deck was convenient. it was user friendly and did not require intensive training. this finding supports that of anggoro (2021) about the easy accessibility of pear deck. it is also congruent with the finding of liu, sands-meyer and audran (2019). however, for students with a bad internet signal, the experience might be different. the minority of the respondents informed us that due to unreliable internet connection, they were not able to table 5. descriptive statistics of students’ perception based on gender. group statistics gender n mean std. deviation std. error mean perception female 250 4.3620 0.68685 0.04344 male 70 4.2043 0.76735 0.09172 http://dx.doi.org/10.25304/rlt.v31.2944 k. j. anggoro and d. i. pratiwi 8 citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 (page number not for citation purpose) t ab le 6 . in d ep en d en t sa m p le s tte st f o r st u d en ts ’ p er ce p ti o n b as ed o n g en d er . in d ep en d en t sa m p le s te st l ev en e’ s te st f o r eq u al it y o f va ri an ce s tte st f o r eq u al it y o f m ea n s f s ig . t d f s ig . (2 -t ai le d ) m ea n d if fe re n ce s td . e rr o r d if fe re n ce 95 % c o n fi d en ce i n te rv al o f th e d if fe re n ce l o w er u p p er p er ce p ti o n e q u al va ri an ce s as su m ed 0. 67 1 0. 41 3 1. 65 4 31 8 0. 09 9 0. 15 77 1 0. 09 53 5 –0 .0 29 88 0. 34 53 0 e q u al va ri an ce s n o t  as su m ed 1. 55 4 10 2. 00 8 0. 12 3 0. 15 77 1 0. 10 14 8 –0 .0 43 58 0. 35 90 1 http://dx.doi.org/10.25304/rlt.v31.2944 research in learning technology citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 9 (page number not for citation purpose) access the platform. a similar finding was also reported by anggoro and khasanah (2022). hence, a stable internet signal is a prerequisite to utilising the platform. the last category is achievement. the respondents positively reacted to statements on learning achievement. they mentioned that the platform made them understand the lesson more. several comments made by students talked about the use of pear deck to learn specific english skills, including vocabulary, listening, writing, reading, and grammar. hence, the platform has the potential to help improve students’ english skills. this is in line with the finding of archambault, pagani and fitzpatrick (2013) and javed and odhabi (2018). several students also pointed out that due to the covid-19 restrictions, they were forced to study online. they further explained that using pear deck made them able to comprehend the online class more, and they were able to practice several english skills during class. this finding supports that of anggoro and khasanah (2022) about the use of pear deck in an online class. apart from the three categories, this study discussed students’ perceptions based on gender and english proficiency levels. based on gender, even though the female participants did score a higher mean than the male, there was no significant difference in the respondents’ perceptions of pear deck. hence, the finding that female students are more comfortable using technological tools than male might not be in line with that of huo (2013) and shin et al. (2018). this finding is more congruent to that of gunamala and sneha (2013), that there are no significant differences between the female and male students’ performances in using technology tools for learning. in relation to the english proficiency level, this study found that both the basic user and independent user groups showed positive reactions to pear deck. both groups conveyed their satisfaction with the use of the platform and wished that it would be used in other courses as well. the independent user groups, moreover, scored a significantly higher mean than that of the basic user. this implies that pear deck might be a beneficial platform to select to teach basic user students and it might be more advantageous for independent user students. the basic user students can enjoy the lesson more, while the independent user students can increase their achievement. conclusion as an irs, pear deck has been used in classrooms, especially in english class of higher education context. this study reports that the platform might have the potential to improve english classes by engaging students and assisting their learning. the platform is also easy to use, so that students will get used to it quickly. pear deck is a versatile tool since it can be used for both onsite and online classes as long as the internet connection is strong. students with bad internet signals might struggle to participate in activities on the platform. this study also discussed students’ perceptions based on gender and english proficiency levels and found that the platform was table 7. descriptive statistics of students’ perception based on english proficiency levels. group statistics english proficiency (ep) n mean std. deviation std. error mean perception basic user 155 4.2071 0.84000 0.06747 independent user 165 4.4406 0.53213 0.04143 http://dx.doi.org/10.25304/rlt.v31.2944 k. j. anggoro and d. i. pratiwi 10 citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 (page number not for citation purpose) t ab le 8 . in d ep en d en t sa m p le s tte st f o r st u d en ts ’ p er ce p ti o n s b as ed o n e n gl is h p ro fi ci en cy l ev el s. in d ep en d en t sa m p le s te st l ev en e’ s te st f o r eq u al it y o f va ri an ce s tte st f o r eq u al it y o f m ea n s f s ig . t d f s ig . (2 -t ai le d ) m ea n d if fe re n ce s td . er ro r d if fe re n ce 95 % c o n fi d en ce i n te rv al o f th e d if fe re n ce l o w er u p p er p er ce p ti o n e q u al v ar ia n ce s as su m ed 12 .8 35 0. 00 0 -2 .9 89 31 8 0. 00 3 -0 .2 33 51 0. 07 81 2 -0 .3 87 21 -0 .0 79 81 e q u al v ar ia n ce s n o t as su m ed -2 .9 49 25 7. 61 7 0. 00 3 -0 .2 33 51 0. 07 91 7 -0 .3 89 42 -0 .0 77 60 http://dx.doi.org/10.25304/rlt.v31.2944 research in learning technology citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 11 (page number not for citation purpose) enjoyed by both female and male students, as well as the two different english proficiency groups found in the study: basic users and independent users. although both users showed a positive attitude towards the use of pear deck, the independent user showed higher satisfaction than the basic user. there are several limitations to this study. firstly, the imbalance of the number of students based on gender. the female and male students were 250 and 70 people, respectively. secondly, it was the english proficiency levels. there were only two out of three levels found. no advanced user participated in the study. the last was the different environments where pear deck was utilised. several students had experiences using it in the classroom, while others were online. a future study comparing students’ perceptions of the platform in online, onsite, and hybrid environments might enrich the results of this study. references anggoro, k. j. (2021) pear deck. relc journal, vol. 52, no. 3, pp. 645–647. doi: 10.1177/0033688220936735 anggoro, k. j. & khasanah, u. (2022) efl students’ independent learning with pear deck interactive slides. studies in self-access learning journal, vol. 13, no. 1, pp. 172–176. doi: 10.37237/130111 archambault, i., pagani, l. s. & fitzpatrick, c. (2013) transactional associations between classroom engagement and relations with teachers from first through fourth grade. learning and instruction, vol. 23, pp. 1–9. doi: 10.1016/j.learninstruc.2012.09.003 awedh, m., et al., (2014) using socrative and smartphones for the support of collaborative learning. international journal on integrating technology in education, vol. 3, no. 4, pp. 17–24. doi: 10.5121/ijite.2014.3402 balta, n. & hamza, m. (2017) the effect of student collaboration in solving physics problems using an online interactive response system. european journal of educational research, vol. 6, no. 3, pp. 385–394. doi: 10.12973/eu-jer.6.3.385 collins, l. j. (2007) livening up the classroom. medical reference services quarterly, vol. 26, no. 1, pp. 81–88. doi: 10.1300/j115v26n01\_08 creamer, e. g. (2017) an introduction to fully integrated mixed methods research, 1st edn, los angeles: sage. cuadrado-garcía, m., ruiz-molina, m.-e. & montoro-pons, j. d. (2010) are there gender differences in e-learning use and assessment? evidence from an interuniversity online project in europe. procedia – social and behavioral sciences, vol. 2, no. 2, pp. 367–371. doi: 10.1016/j. sbspro.2010.03.027 dong, y., et al., (2018) interactive presentation tools using mobile devices. society for information technology & teacher education international conference, vol. 2018, no. 1, pp. 743–748. available at: http://www.learntechlib.org/p/182605/ el shaban, a. (2017) the use of socrative in esl classrooms. teaching english with technology, vol. 17, no. 4, pp. 64–77. available at: http://www.tewtjournal.org gunamala, s. & sneha, s. (2013) application of a gamified interactive response system to enhance the intrinsic and extrinsic motivation, student engagement, and attention of english learners. international journal of engineering research, vol. 2, no. 2. available at: https://www. indianjournals.com/ijor.aspx?target=ijor:ijer&volume=2&issue=2&article=022 huhta, a. (2012) common european framework of reference. in the encyclopedia of applied linguistics. doi: 10.1002/9781405198431.wbeal0157 wiley-blackwell: new jersey. huo, q. (2013) gender difference in perception and adoption of technology to enhance second language learning out-of-school, hku theses online (hkuto), the university of hong kong. http://dx.doi.org/10.25304/rlt.v31.2944 https://doi.org/10.1177/0033688220936735 https://doi.org/10.37237/130111 https://doi.org/10.1016/j.learninstruc.2012.09.003 https://doi.org/10.5121/ijite.2014.3402 https://doi.org/10.12973/eu-jer.6.3.385 https://doi.org/10.1300/j115v26n01 https://doi.org/10.1016/j.sbspro.2010.03.027 https://doi.org/10.1016/j.sbspro.2010.03.027 http://www.learntechlib.org/p/182605/ http://www.tewtjournal.org https://www.indianjournals.com/ijor.aspx?target=ijor https://www.indianjournals.com/ijor.aspx?target=ijor https://doi.org/10.1002/9781405198431.wbeal0157 k. j. anggoro and d. i. pratiwi 12 citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 (page number not for citation purpose) javed, y. & odhabi, h. (2018) active learning in classrooms using online tools: evaluating peardeck for students’ engagement. 2018 fifth hct information technology trends (itt), pp. 126–131. doi: 10.1109/ctit.2018.8649515 jones, b. a. & palmer, r. (2017) active learning in japanese university efl classes: clarifying the construct. hirao school of management review, vol. 7, pp. 107–125. doi: 10.14990/00002306 kariadi, m. t. & pratiwi, d. i. (2022) revealing university students ’ attitudes toward english language. the journal of asia tefl, vol. 19, no. 3, pp. 1053–1062. doi: 10.18823/ asiatefl.2022.19.3.20.1053 korstjens, i. & moser, a. (2018) series: practical guidance to qualitative research. part 4: trustworthiness and publishing. european journal of general practice, vol. 24, no. 1, pp. 120– 124. doi: 10.1080/13814788.2017.1375092 lei, j. & lin, t. (2022) emergency online learning: the effects of interactional, motivational, self-regulatory, and situational factors on learning outcomes and continuation intentions. international review of research in open and distributed learning, vol. 23, no. 3, pp. 43–60. doi: 10.19173/irrodl.v23i3.6078 leong, l. c., et al., (2018) mobile x-space design, teaching strategies and undergraduate students’ collaborative learning behaviour: a case study in taylor’s university, malaysia. malaysian journal of learning and instruction, vol. 15, no. 2, pp. 175–205. doi: 10.32890/mjli2018.15.2.7 liu, c., sands-meyer, s. & audran, j. (2019) the effectiveness of the student response system (srs) in english grammar learning in a flipped english as a foreign language (efl) class. interactive learning environments, vol. 27, no. 8, pp. 1178–1191. doi: 10.1080/10494820.2018.1528283 liu, t. c. (2003) the features and potential of interactive response system. proceedings of international conference on computers in education (icce), pp. 315–322 . 2–5 december. hong kong. mache, j., et al., (2017) pear deck: an interactive classroom response system to encourage student engagement. journal of computing sciences in colleges, vol. 33, no. 1, pp. 156–158. ni, c. k., et al., (2020) enhancing malaysian primary pupils’ vocabulary skills using pocable game and pear deck. international journal of learning, teaching and educational research, vol. 19, no. 6, pp. 145–160. doi: 10.26803/ijlter.19.6.9 north, b. (2014) the cefr in practice. cambridge university press, cambridge. paul, m. w., et al., (2020) engaging the control-value theory: a new era of student response systems and formative assessment to improve student achievement. research in learning technology, vol. 28, no. 1063519, pp. 1–16. doi: 10.25304/rlt.v28.2454 pratiwi, d. i. & ubaedillah, u. (2021) digital vocabulary class in english for railway mechanical technology. teaching english with technology, vol. 21, no. 3, pp. 67–88. available at: https://tewtjournal.org/download/digital-vocabulary-class-in-english-for-railway-mechanical-technology-by-damar-isti-pratiwi-and-ubaedillah-ubaedillah/ pratiwi, d. i. & waluyo, b. (2022) integrating task and game-based learning into an online toefl preparatory course during the covid-19 outbreak at two indonesian higher education institutions. malaysian journal of learning & instruction, vol. 19, no. 2, pp. 37–67. doi: 10.32890/mjli2022.19.2.2 sengsri, s. & anggoro, k. j. (2021) students’ perceptions of content and language integrated pedagogy via electronic communication (e-clip). world journal on educational technology: current issues, vol. 13, no. 1, pp. 54–67. doi: 10.18844/wjet.v13i1.5363 shin, i.-s., et al., (2018) evaluations of interactive learning tools among engineering students: effects of grit and gender. asee il-in section conference. available at: http://www.ncbi.nlm.nih.gov/ pubmed/15636205%0ahttps://docs.lib.purdue.edu/aseeil-insectionconference/2018/assess/3/ sun, j. c.-y. & hsieh, p.-h. (2018) application of a gamified interactive response system to enhance the intrinsic and extrinsic motivation, student engagement, and attention of english learners. journal of educational technology & society, vol. 21, no. 3, pp. 104–116. http://www.jstor.org/stable/26458511 http://dx.doi.org/10.25304/rlt.v31.2944 https://doi.org/10.1109/ctit.2018.8649515 https://doi.org/10.14990/00002306 https://doi.org/10.18823/asiatefl.2022.19.3.20.1053 https://doi.org/10.18823/asiatefl.2022.19.3.20.1053 https://doi.org/10.1080/13814788.2017.1375092 https://doi.org/10.19173/irrodl.v23i3.6078 https://doi.org/10.32890/mjli2018.15.2.7 https://doi.org/10.1080/10494820.2018.1528283 https://doi.org/10.26803/ijlter.19.6.9 https://doi.org/10.25304/rlt.v28.2454 https://tewtjournal.org/download/digital-vocabulary-class-in-english-for-railway-mechanical-technology-by-damar-isti-pratiwi-and-ubaedillah-ubaedillah/ https://tewtjournal.org/download/digital-vocabulary-class-in-english-for-railway-mechanical-technology-by-damar-isti-pratiwi-and-ubaedillah-ubaedillah/ https://doi.org/10.32890/mjli2022.19.2.2 https://doi.org/10.18844/wjet.v13i1.5363 http://www.ncbi.nlm.nih.gov/pubmed/15636205%0ahttps http://www.ncbi.nlm.nih.gov/pubmed/15636205%0ahttps http://docs.lib.purdue.edu/aseeil-insectionconference/2018/assess/3/ http://www.jstor.org/stable/26458511 research in learning technology citation: research in learning technology 2023, 31: 2944 http://dx.doi.org/10.25304/rlt.v31.2944 13 (page number not for citation purpose) tucker, r. (2014) sex does not matter: gender bias and gender differences in peer assessments of contributions to group work. assessment & evaluation in higher education, vol. 39, no. 3, pp. 293–309. doi: 10.1080/02602938.2013.830282 turner, d. (2015) psychosocial aspects of engagement with social media and digital technology – personal thoughts from the frontier. in smart learning: teaching and learning with smartphones and tablets in post-compulsory education. media-enhanced learning special interest group and sheffield hallam university. available at: http://melsig.shu.ac.uk waluyo, b. (2020) learning outcomes of a general english course implementing multiple e-learning technologies and active learning concepts. the journal of asia tefl, vol. 17, no. 1, pp. 160–181. doi: 10.18823/asiatefl.2020.17.1.10.160 http://dx.doi.org/10.25304/rlt.v31.2944 https://doi.org/10.1080/02602938.2013.830282 http://melsig.shu.ac.uk https://doi.org/10.18823/asiatefl.2020.17.1.10.160 talking with pictures: exploring the possibilities of iconic communication colin beardon*, claire dormann*, stuart mealing** and masoud yazdani** * faculty of art, design and humanities, university of brighton, brighton bn2 2yj **department of computer science, university of exeter, exeter ex4 qh abstract as multimedia computing becomes the order of the day, so there is a greater need to understand and to come to terms with the problems of visual presentation. this paper deals with iconic languages as a means of communicating ideas and concepts without words. two example systems, developed respectively at the universities of exeter and brighton, are described. both embody basic principles of the iconic communication which,, though not unique to learning technology, is forming an increasingly important part of user-interfaces, including those in the area computer-assisted learning. introduction koji kobayashi looks forward to a time when the telephone system will automatically translate between the languages of users in real time (kabayashi, 1986). it will then be possible to speak in your native tongue to a frenchman, a japanese and a russian, and for each of them to hear your words spoken in their own language as you speak. in the hitch-hiker's guide to the galaxy (adams, 1979) it becomes possible to understand communication in any language by plugging a babel fish into your ear a fish with the natural ability to receive speech in any language and translate it into the language of the person wearing it. while these visions are certainly futuristic, even fantastic, the need for people knowing different languages to communicate quickly and easily is rapidly increasing at the same time as the technology for transmitting and receiving information is spreading and becoming more powerful. attempts to create international languages have not been very successful, partly because of the need for a significant number of people to know them, and partly because they have to be learned like any other new language. understanding is, however, possible across language 26 altj volume 1number 1 barriers in at least two ways. first, there are a number of internationally recognized signs, symbols and icons which one can find on the roadside and in airports, railway stations and similar places. they are not only used directly to denote the place where one might find trains or change money, but may also contain a number of 'meta-icons' such as the red diagonal and the arrow. secondly, it is possible to make oneself understood, at least at some basic level, by means of gestures and mime. people who find themselves in a country where they know little or nothing of the local language can often communicate simple ideas by pointing or indicating their intentions through more performative signs, which may border on acting. iconic communication is the attempt to build cross-language communication systems1 that completely avoid the use of words and rely solely on pictorial symbols. the system which manipulates these symbols should be as simple to use as possible, though it is clear that it has to be learned in some way. there are various forms of learning which can apply: users can receive some form of instruction (either by a tutor, or by some non-linguistic tutorial system); the elements of the system can contain an explanation of their meaning ('self-explaining icons'); the system can adopt a powerful metaphor; or the user can experiment and learn by trial and error. icons it is important to say first of all what we mean by an icon. figure 1 contains various ways of referring to 'men', and they vary from the arbitrary (figure la) to the pictorially descriptive (figure id). icons tend to exist between these extremes (figures lb and lc). figure 1: different ways of referring to 'men' words are essentially arbitrary in the way they refer, so there is no alternative but to learn their meaning. of course words may be morphologically complex (spaceship, for example), but the components themselves are ultimately atomic and have arbitrary referents. with icons, the relationship to the referent is not arbitrary, but neither is it as direct as in the case of pictures. it would be quite wrong to see pictures as an ideal form of icon because there are a number of things they are not good at expressing. for example, it is difficult to draw a 27 colin beardon et al. talking with pictures: iconic communication naturalistic picture of love or of general classes of things, such as the class of all mammals. for the purpose of pictorial language, pictures reveal too much information: the man in figure l(d) might stand for businessmen, middle-aged men, or any of a number of other interpretations. quite apart from this, the amount of detail means that differentiation from other symbols is not so easy, and therefore recognition can be considerably slower than with the more stylized forms. while an icon suggests its referent, its form is often insufficient to describe it precisely. what advantage, then, does it give us over words? there are two reasons for which an icon can be advantageous: it can be easier to learn and easier to remember. it can be easier to learn because its appearance at least suggests a set of possible referents and because it is often part of a consistent system of representation which itself will provide a context. for example, the symbol in figure 2 is given its reference by being placed on a white background in a red triangle on a black and white striped poll beside a road. this tells us immediately that it is a sign intended for drivers and concerns some potential hazard. it can be easier to figure 2: a hump-backed bridge r e m e m b e r b e c a u s e i l i s a well-known aid to memory to associate the thing to be remembered with a simple object to which it has a defined relationship. for example, to remember the french word for pig (cochon) i may think that it sounds like the english word cushion. i therefore associate the concept of the french for pig with an image of a cushion, and this serves as an aid to memory.2 this illustrates well the fact that an icon does not necessarily have to be directly representational but can indicate its reference by means of a convention, as in figure l(b). such conventions are of course also applicable to user-interfaces, including those in cal programs. iconic languages it has been suggested that most modern written languages are derived from pictorial languages, but iconic languages seem to be a more recent invention. some of the more interesting examples are semantography (blis, 1965), isotype (international system of typographic picture education) (neurath, 1978) and worldsign (jones and cregan, 1986), a language created for mentally handicapped children which allows dynamic representation. in semantography (or bliss symbols), one symbol corresponds to one word in natural language. it was proposed as an auxiliary writing tool for communication between different nations, and as a device to specify relative and vague meanings. the aim of the language is to communicate through simple pictorial symbols, with those representing physical things using outline and those representing non-physical things using geometric symbols. the first 25 symbols are already internationally accepted; they include the digits 0 to 9 and symbols such as a question mark, a full stop and a plus sign. bliss based his grammar on the assumption that all languages are used to describe the phenomena of our physical world, that the main manifestations of our world can be classified into matter, energy and the mental, and that everything happens in space and time. there is a 28 alt-j volume 1 number 1 specific logic behind symbols constructed to represent words. a word like telephone takes its symbolization from the symbols for electric and language (a telephone is an electrical apparatus in which you can talk) and the language symbol is derived from the ear and mouth symbols (both used in conversation). the isotype picture language does not have a simple correspondence between signs and words. an example given by neurath is that there is no sign for the word foot that is common to expressions such as the foot of a man, the foot of mountain, and the foot of a table. these expressions are composed of simple signs of a very different sort. furthermore, the final translation of the 'language picture' is a structured group of statements, and the system of connection between signs is far richer than in linear text. several rows of connected signs are interpreted simultaneously, whereas one-dimensional text requires readers to bear in mind what they have read and to make connections between dispersed elements for themselves. neurath's writing suggests two central rules for generating the vocabulary of an international picture language: reduction, for determining the style of individual signs, and consistency, for giving a group of signs the appearance of a coherent system. then there needs to be a set of conventions to allow the user to know how the information is structured. neurath's work was directed towards making statistical charts. he introduced two basic rules: the first of these related to the presentation of statistics by means of icons, and held that a icon represents a certain quantity or amount of things and that more signs represent a greater quantity or amount. the second was a general rule that perspective should not be used. his work includes a series of posters for an anti-tuberculosis campaign (non-statistical) and the publication of many books and charts. the bliss approach is similar to text-based language and susceptible to some of the same pitfalls, but it presents some interesting insights for an international human-human computer system. isotype in its present form is best suited for statistics and is too restricted for a computer environment but it could be seen as a precursor of an iconic computer language. computer-based icons the computer provides a significant new environment for the use of icons. whereas previous systems of icons have been essentially textual, their messages are one-dimensional sequences of icons in which everything is explicit. each icon has its reference, and icons are placed in sequences (occasionally there are suband super-scripts) with blank spaces to indicate grouping. the typical windows environment on a modern computer provides a much richer environment. first, there are fully two dimensions to exploit so that any grammar which exploits relative positioning of icons is not restricted in the way that text-based languages have been (though from a practical point of view there are limits on how much one can present at one time). there is also the possibility of using colour (or greyscale) and animation. most importantly, an icon in a computer environment needs to be defined both representationally and operationally. in addition to asking what an icon represents (pictorially), one can also ask what happens when the user clicks on it, or double-clicks on it, or clicks-and-drags it to some other part of the screen. here we are opening up a world in which the sentence 'don't look for the meaning, look for the use' has a new potency. the 29 colin beardon et al. talking with pictures: iconic communication meaning of an icon is no longer simply what it resembles, but also what happens when you do something with it. this realization gives a dramatic new lease of life to what may appear to have been a marginalized form of communication. the remainder of this paper describes some of the work we have undertaken so far in this area. it is centred around two example systems: a hotel booking system (developed at the university of exeter) and cd-icon (developed at brighton polytechnic). while the hotel booking example does not relate specifically to learning technology, the principles on which it is based are relevant to any user-interface design, and cd-icon has obvious similar relevance. example 1 hotel booking system hotel booking is a typical activity that requires communication across language barriers. it offers us the opportunity to apply iconic language in a simple dialogue between a potential guest and a hotel or city-wide hotel booking facility. in a final system we can envisage a touch-sensitive screen with plenty of interaction, but at this stage of development we are concerned with the initial formulation of a request by the customer that will be transmitted to a hotel for the compilation of a reply. a demonstration system has been built using hypercard. the compilation of the booking message is accomplished in stages, and at each stage the current context is cued by a picture resident in the background. in sequence these are: a 'typical' hotel front (figure 3), a 'typical' hotel reception area (figure 4), and a 'typical' hotel bedroom (figure 5). each new screen holds the background picture before the other information is faded in over it. the initial screen shows a hotel overlaid by an appropriate caption, and clicking anywhere on the image starts the booking sequence. the first screen invites the user to indicate the intended destination (the name of a town or a hotel) and grade of hotel, by selecting from cyclable 'star' ratings (figure 3). movement to the next screen is initiated by clicking on the 'tick' icon, a convention used throughout the package. the second screen (figure 4) shows a hotel reception area and invites selection of the dates and times of arrival and departure. the number of nights that are implied by these dates and times is indicated by black bars which appear (and disappear) as each night is added (or removed). once again, the 'tick' icon moves the user to the next screen. the third screen (figure 5) shows a room overlaid with icons permitting the selection of rooms and their required facilities. a room is shown as a white rectangle. one room is shown initially and the number of rooms can be altered by clicking on the ' + ' and ' ' icons. four icons at the top right of the screen each unlock further related icons to enable the selection, for each room, of: (a) the number and type of occupants, (b) the number and type of beds, (c) the type of bathroom facilities, and (d) the range of other facilities required. the various features are selected by clicking on an icon which causes a clone to be produced beside it, which is then dragged into the relevant room. the 'tick' icon moves the user to screen 4 (figure 6) which displays the complete booking requirement. if this is satisfactory a further 'tick' sends the message to the hotel. 30 altj volume 1 number 1 figure 3: screen inviting input of destination figure 4: screen inviting selection of dates and times of arrival and departure 31 colin beardon et al. talking with pictures: iconic communication figure 5: screen allowing selection of room types and facilities figure 6: screen displaying complete booking requirements 32 alt-j volume 1 number 1 figure 7: screen showing availability of 'star' ratings the message is revealed to the hotel in stages. confirmation of the acceptability of each part of the message (tick) moves on to the next part of the message, whilst unavailability (cross) brings up a range of possible alternatives. figure 7 shows that the required dates of stay are acceptable but the requested hotel grade is unavailable. the hotel is therefore presented with four options to propose in reply. once the entire message has been processed by the hotel in this way, the final message is sent back to the customer who will be able to accept or reject the alternatives offered, continue the dialogue, and confirm a booking. the application, as it stands, does not pretend to be either comprehensive or the most practical solution in real terms, but is an initial attempt to create a simple, interactive, iconic dialogue using hotel booking as a convenient theme. it does, however, offer much that could be used in a real system, and serves its purpose in starting to explore the possibility of communicating with icons. example 2 the cd-icon language the cd-icon system is an attempt to build an iconic communication system based on the principles that underlie natural-language processing systems. the standard way of specifying semantics in such systems is to assume some other system (a meaning representation language, or mrl) for which the semantics are already known (woods, 1978). the task then becomes that of expressing rules by means of which statements in natural language are transformed into statements into the mrl, and vice versa. schank's conceptual dependency representation (schank, 1973) is an mrl in this sense, and its own semantics are either taken to be intuitive, or are established by their success in various practical projects, for example margie (schank, 1975) and pam (wilensky, 1981). cd-icon is a means of testing the validity of conceptual dependency directly by making it the basis of a communication system that uses only icons and no words. a message is composed by selecting options from a series of interconnected screens (in the spirit of 33 colin beardon et al. talking with pictures: iconic communication systemic grammar). the message is then transmitted, also as a set of interconnected screens, but not showing options that have not been selected. we will illustrate the system by composing the message equivalent to 'the big man went home'. figure 8 shows the message expressed in schank's cd formalism. ptrans * ^ man figure 8 schank's cd representation of 'the big man went home' in cd-icon a message is composed in four stages. the first stage is concerned with what schank calls 'conceptual relations' and 'conceptual tenses'. a screen is presented (figure 9) which enables the user to select an assertion, a question or an imperative, between simple and compound messages, and to decide upon negation. if the message is compound, the nature of the relationship between the two component conceptualizations is chosen (logical and, logical or, implication, temporal or spatial). clicking on an icon for a conceptualization transfers control to stage 2 which is typically concerned with an event. events, according to schank, are based around a primitive 'act' so the first selection is between icons representing various primitive 'acts' (figure 10). to assist the user, a help facility presents a short animated explanation, in the manner of mealing and yazdani (1990). having selected the appropriate 'act', the corresponding screen is presented. in this example it is the ptrans screen which is shown in figure 11. it contains a basic background for ptrans with grey icons representing the object, origin, destination and instrument cases, as well as tense. (there is a divergence from schank here in that we do not use the agent case.) grey icons denote options, whereas black-and-white icons represent selections. a grey house, for example, represents the class of places, the grey question mark represents the class of objects, the grey clock represents the class of times, and the grey spade represents the class of instrument cases. clicking on any one of these icons (except the clock or spade) will result in transfer to stage 3 which is concerned with the production of what schank refers to as a 'picture'. the 'picture' 34 altj volume 1 number 1 figure 9: screen for selecting a simple assertive sentence figure 10: selecting a primitive 'act' 35 colin beardon et al talking with pictures: iconic communication figure 11: the screen for ptrans figure 12: picture screen for 'big man' 36 altj volume 1 number 1 figure 13: a screen from the lexicon figure 14: the four screens representing the complete message 37 colin beardon et al talking with pictures: iconic communication screen initially contains only the head icon (the icon that was clicked on at stage 2). this will eventually be defined along with any modifiers to produce the screen in figure 12. clicking on the 'head' icon will result in transfer to stage 4 which is the lexicon, or rather that part of the lexicon which contains objects of the type specified by the grey icon (see figure 13). the user selects an appropriate icon, in this case the one for 'man', and control is returned to the 'picture' screen (stage 3). there are two changes however: the selected icon replaces the grey icon, and those classes of object which normally modify the 'head' icon are shown in grey. the user can click on any of these modifying icons and be taken to the appropriate part of the lexicon to select the precise colour, size, location, etc. in our example, there is one modifier, 'big', and the outcome of stage 3 is shown in figure 12. when the user is satisfied, the tick box is clicked and the grey icons are deleted. control is passed back to the ptrans screen (stage 2) with an icon composed of the head icon from the picture plus an asterisk in the top right corner. if this new icon is selected in help mode, the full iconic representation will be displayed. the process is repeated for all icons which the user decides to specify. temporal reference is established with respect to an imported clock icon representing 'now' (set at 6 o'clock). a clock internal to the 'act' can be set at 'past', 'present' or 'future' (3, 6, or 9 o'clock). in our example, the past tense is used. the instrument case is handled differently. in schank's system the instrument case never points to an object, but always to a conceptualization. in cd-icon, control will be passed recursively to stage 2 to devise a new conceptualization, which will be represented by a new icon (a black spade with an asterisk) by means of which the instrument case can be made explicit. when the user is satisfied, clicking on the tick box will return control to the message-level screen (stage 1) with an icon for ptrans plus an asterisk in the top right corner. the final, message will be represented by the four interconnected screens shown in figure 14. at present the system is being used to explore the validity of mrls and the possibility of unrestricted communication by icons. it is hoped to soon have a system that will allow users to try to compose and understand simple messages, at which point empirical testing will take place. future directions these two projects have already raised some important issues. we need to distinguish different communicative tasks, for example to distinguish between an iconic system that serves as a front-end to existing computer software and an iconic system for person-to-person communication. this distinction seems to mirror the distinction between systems that have a known mrl and those that do not. the systems also raise the possibility of a complete escape from linguistic forms. at present there is a tendency to explain them with reference to linguistic examples — that is to say, one tries to compose a message that corresponds to a sentence which has already been formulated in english. the intention is, however, to escape from this and view the systems as communication channels in their own right. the output should not be verbalized, except 38 alt-j volume 1 number 1 perhaps in some indirect way in order to test the degree to which communication has taken place. this raises the question of the form of the communication itself. in the hotel booking system it is a single screen, whereas in cd-icon it is a set of connected screens. the question is being considered, particularly when dealing with larger texts, of whether an animated screen may not be more appropriate. in the case of our second example, the 'man' icon could appear moving to a house. this would be no ordinary animation, for the actors would be iconic and the animation may be stopped at any point and icons selected to reveal more information about themselves. notes 1. cross-language communication means communication irrespective of the language spoken by the participants. it is distinguished from cross-cultural communication which raises a number of specific problems, and while we can see how cross-cultural issues might be addressed, to date little research has been carried out in the area. 2. this system of language learning has been commercially exploited (gruneberg 1987-1992, gruneberg and jacobs, 1991). references adams, d. (1979), the hitch-hiker's guide to the galaxy, london, pan books. bliss, c.k. (1965), semantography, australia, semantography publications. gruneberg, m. (1987-1992); gruneberg, m. and jacobs, g. (1992), linkword language system (various languages), london, corgi books. gruneberg, m. and jacobs, g. (1991), 'in defence of linkword', language learning journal, 3, 25-29. kobayashi, k. (1986), computers and communications, cambridge (mass), mit press. mealing, s. (1992), 'talking pictures', intelligent tutoring media, 2, 2, 63-69. mealing, s. and yazdani, m. (1992), 'a computer-based iconic language', intelligent tutoring media, 1, 3, 133-36. neurath, o. (1978), international picture language, university of reading, department of typography and graphic communication. schank. r.c. (1973), 'identification of conceptualisations underlying natural language' in schank, r.c. and colby, m.c. (eds), computer models of thought and language, san francisco, freeman, 187-247. schank, r.c. (1975), conceptual information processing, new york, north-holland. wilensky, r. (1981), 'pam' in schank. r. and reisbeck, c. (eds), inside computer understanding, new jersey, lawrence erlbaum, 136-179. woods, w. (1978), 'semantics and quantification in natural language question answering' in yovits, m. (ed), advances in computers (17, 2-64), new york, academic press. 39 doi:10.3402/rlt.v19s1/7799 using hypermedia annotations to teach vocabulary on the web bahman gorjian* islamic azad university, abadan branch, abadan, khuzestan, iran (received 11 december 2010; final version received 26 july 2011) this project measured the effect of using hypermedia annotations on short and long-term vocabulary retention in teaching vocabulary through web-based language learning activities. a total of 62 university students were randomly assigned into two homogeneous groups; and then both groups were given a pretest. both groups covered 12 expository passages selected by the researchers from the bbc website. the subjects had to sit for an immediate quiz to measure the short-term effect of the treatment and finally, at the end of the course and a twoweek interval, subjects sat for their post-test. findings revealed that there was a significant effect of the hypermedia annotations on the retention of vocabulary in the short term ( pb0.05). however, the post-test results indicated that the effect of the treatment in the long term faded away, and the significance of the means was not sufficiently high to reject the null hypothesis. keywords: hypermedia annotations; plain text; vocabulary retention; wbll introduction teaching vocabulary through web-based language learning (wbll) activities has been popularly used in english as a foreign/second language learning (efl/esl) context (son 2008). hypermedia as a multidimensional computer tool has been practiced by language teachers to facilitate learning and teaching processes (cummins 2008a). it provides an integrative network tool utilised in classrooms around the world. knowledge of vocabulary is the backbone of learners’ competency which facilitates learning of any language tasks. decarrico (2001, as cited in celce-murcia 2001) claims that ‘‘vocabulary learning is central to first and second language acquisition and specialists now emphasise the need for a systematic and principled approach to vocabulary by both teachers and learners’’ (285). therefore, learning vocabulary is often perceived to be ‘‘of critical importance to the typical language learner’’ (zimmerman 2001, 5). teachers and learners can utilise website resources for various pedagogical purposes to scaffold teacher�student interaction in and outside the classroom (cummins 2008b). the present study evaluated the effect of using hypermedia annotations, as opposed to plain or printed texts, on learners’ vocabulary retention. hypermedia is defined as an audio and external presentation of the passage in addition to the picture presentation of the passage provided by the authors of the passages. slatin (1991) defined hypermedia (or hyper-document) as an assemblage of texts, images and sounds-nodes-connected by electronic links so as to form a system *corresponding author. email: bahgorji@yahoo.com alt-c 2011 conference proceedings 0001 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # b. gorjian. this is an open access article distributed under 112 the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http:// creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7799 whose existence is contingent upon the computer. the passage may accompany extra-textual annotations or computer software (i.e. encyclopedias or online dictionaries). in contrast, plain or printed texts are paper print of the material to provide the learners with vocabulary learning tasks. for instance, monolingual or bilingual dictionaries (e.g. monolingual longman dictionary of contemporary english) which are used by learners to pick up the right definition at the time they recognise a need. statement of the problem baker and westrup (2003) refer to the stages of vocabulary teaching as: ‘‘first the teacher conveys the pronunciation and meaning of the new vocabulary item (presentation). second, the teacher checks that the student has understood properly (practice). third, the teacher consolidates and tries to get the students to relate the word to their personal experience, and use it in context (production)’’ (37). this is the teachers’ point of view: at the same time that the teachers are trying to teach vocabularies based on presentation, practice, production (ppp) approach, most learners feel embarrassed, trying to look each and every word up in their dictionaries, making them easily give up. when it comes to vocabulary learning in foreign languages, nguyen and khuat (2003) assert that ‘‘vocabulary learning is considered as boring as they [foreign language learners] have to memorise unfamiliar words and spelling’’. in both cases (teaching and learning vocabulary), eventually we see that most of the class time is spent on activities other than the intended one (hulstijn, in press). therefore, vocabulary teaching and learning is a time-consuming effort in traditional approaches. first, learners will always show a need for more and teachers will always see and attempt to satisfy it. second, because of the time and energy involved in teaching and learning vocabulary, the prime focus is on the meaning of the intended vocabulary: other features of the vocabulary (multidimensionality of vocabulary knowledge) would be overshadowed for the sake of meaning. according to hulstijn and laufer (2001), ‘‘if learners pay careful attention to the words’ pronunciation, orthography, grammatical category, meaning and semantic relations to other words, they are more likely to retain the word than if they pay attention to only one or two of the above word properties’’ (541). nation (2005) claims that successful comprehension requires automatic recognition and decoding of 95�99% of the words in a text. it has also been claimed that reading is one of the main ways language learners acquire new vocabulary knowledge (bogaards 2001). for these reasons, in this study a reading approach will be used to examine vocabulary retention. on the other hand, the researchers attempted to eliminate parameters which hindered the acquisition of vocabulary and introduced the medium through wbll activities which language learners may focus more on their learning, rather than being distracted by the process of finding a vocabulary meaning in their dictionaries. using computer-assisted language learning (call) approaches to teaching vocabulary, teachers are also freed from the long and boring process of teaching vocabulary and are allowed to focus more on other needs of language learners (gorjian 2008). in this study it is believed that using hypermedia may enhance the quality of the input which ultimately encourages meaningful language learning; provision of such detailed information, often called ‘‘rich instruction’’ or ‘‘rich scripting’’ (mcwilliam 1998) which aims to provide a deeper understanding of a word, and alt-c 2011 conference proceedings 113 make it an ‘‘accessible vocabulary item’’ (nation 2001, 95). accordingly, the main research questions to be pursued in this study are: (1) could enriched texts (hypermedia) help iranian efl learners acquire and retain new vocabularies? and (2) regarding multidimensionality of vocabulary knowledge, could hypermedia be used as a solution to teach a comprehensive knowledge of the intended vocabularies? the study seeks to address the following null hypothesis: enriched texts (hypermedia annotations) have no effect on iranian efl learners in learning and retaining new vocabulary. background the ideas mentioned above are all well dealt with the advent of hypermedia. the use of computer technology in teaching languages has been dramatically increasing worldwide over the past decade (e.g. o’dowd 2003; chen, belkada, and okamoto 2004; hayati 2005; hubbard and levy 2006; son 2008). using this technology not only facilitates learning processes (gorjian 2008), but also holds other great potentials for language learning. one of these potentialities is the ability to present information in different formats using graphics, sound, text and video with links to other chunks of information through using wbll activities (robb 2006; son 2007; cummins 2008a, 2008b). hypermedia annotations have several advantages; researches carried out by boers, eyckmans, and stengers (2004) and abraham (2008) have provided evidence of an overall beneficial role for computer-mediated text glosses providing lexical support on comprehending authentic readings and learning vocabulary. researchers were inspired by the premise that a variety of glosses for words in various modalities, such as printed text, graphics, dynamic video and sound, might have differing capacities to facilitate vocabulary acquisition and retention (de ridder 2002; boers and lindstromberg 2005). presenting information in this way enables readers to access information in the order most appropriate to their purposes. using appropriate presentation methods enables learners to obtain a deeper impression of and richer information about the target words to make them enter the long-term memory more easily (zhang 2008). in addition, in traditional approaches any unexpected subject matter (vocabulary, in our case) cannot be dealt with in advance, no matter how necessary. in our case, if a word out of the blue poses a problem, in traditional approaches teachers could not think of ways of dealing with that word without shifting the focus in class and diverting time (mcdonald 2008). for these reasons, comprehensive vocabulary knowledge seems indispensable but almost impractical to achieve in traditional classes. the idea of hypermedia learning/teaching is also suggested by cognitive psychology. cognitive psychologists and language acquisition scholars working within the framework of cognitive psychology believe that retention of information is determined by the way in which this information is processed (hulstijn and laufer 2001). they suggest the involvement load hypothesis that ‘‘the retention of unfamiliar words is, generally, conditional upon the degree of involvement in processing these words’’ (545). son (2008) proposed that ‘‘hyperlinked multimedia documents and computer mediated communication (cmc) tools; the web can support language teachers to integrate web resources into the language classroom’’ (34). smith and stacy (2003) emphasised that cmc ‘‘has changed the nature of distance from an individual b. gorjian 114 experience that is largely remote and isolated from other students, to one in which the technology can enable more ongoing interaction with fellow students’’ (165). the potential for manipulating online technology within a collaborative learning environment is one of the greatest strengths of cmc. hubbard and levy (2006) argue that the influence of technology on language teaching and learning has developed along with the parallel growth in the development of course work to prepare language teachers with appropriate methods of using the technology in the classrooms. they focused on classroom teachers who should use call ‘‘to promote, manage, or assess their students’ learning. note that ‘classroom’ is used in its broadest sense to subsume language teaching in a traditional physical space, a computer lab, a mix of physical classroom or lab and online, or entirely online’’ (13). robb (2006) believes in maximising the opportunities for the efl/esl teachers to experience ‘‘with technology, both new and old, to interact with their colleagues and to access other sources of information on technology’’ (346). he also emphasises the effect of fostering positive attitudes towards computer technology in the classroom and educational settings ‘‘by providing multiple examples of good practice, as well as the printed, digital and human resources that are required to attain this goal’’ (346). methodology subjects this study was conducted with sample of 62 iranian efl students based on nonrandom convenient sampling. they had entered university for their undergraduate studies. subjects were selected based on a given toefl test, barron’s 2003 edition, and the results of the test indicated that in terms of educational background, level of english mastery and vocabulary knowledge, the subjects were homogeneous and could be considered as intermediate in their proficiency stamina. then they were divided into two groups based on systematic random sampling. the subjects were 62 (15 males and 47 females) students with the age ranging from 21 to 39. they were measured under two conditions: plain text group (i.e. control group who dealt with the plain texts) and hypermedia group (i.e. experimental group who dealt with hypermedia annotations). instrumentation this study used the following instruments: 1. pilot test: to investigate the suitability of the level of the text and to examine how much time it would take the subjects to complete the task, a pilot test of hypermedia version of the text was conducted with five students who did not take part in the actual research. the reliability of each and every test used in this research was calculated by kuder-richardson formula (kr-21). the reliability coefficients for the pre-test and the post-test were 0.75 and 0.74 respectively. 2. pre-test: a pre-test containing the actual test items was administered to the subjects before treatment in order to determine how well the subjects knew the contents before treatment. the subjects were asked to answer 40 multiple-choice alt-c 2011 conference proceedings 115 vocabulary questions, selected from the course passages, in 30 minutes. to ensure that students did not give more attention than they should to the words appearing in the pre-test, no mention was made of the subsequent learning lessons and the immediate quizzes or post-test. 3. immediate quizzes: after each session where students had covered the two passages given to them to be read for the sake of comprehension, there was a 2�3 minutes rest and right after that there was a short multiple question quiz asking the meaning of the new vocabularies learnt in that session. 4. post-test: two weeks later after the end of the course, the instructor administered the post-test without notice. the sudden presence of the instructor in the class was to test the retention of words in a longer period to see the real effect of the treatment. the only difference of this test to the pre-test was that the order of questions was changed to wipe out the probable recall of pre-test answers. 5. multidimensionality (mi) test: it was claimed earlier that hypermedia can boost multidimensionality knowledge of vocabulary in language learners. therefore a pronunciation test was designed at two levels, in one level, which was comprised of a written part of the test, components of vocabulary were put into questions. this part consisted of 10 items and each item was testing phonetic transcription, part of speech, past form of the verb, superlative form of adjective and plural form of noun for a chosen vocabulary from the passages covered. procedure since this comparative study consists of two distinct approaches to vocabulary learning, the materials used were the same for both groups except for the medium of presentation, for this reason two kinds of presentation were used, namely, hypermedia presentation and the plain text presentation. expository passages from the bbc website (www.bbclearningenglish.com) were selected for this study. subjects viewed 12 passages over six sessions where each session lasted for about one hour. subjects in both groups were not informed in advance that they would be tested because it was assumed that if they knew, they would consciously try to learn the new words. it was hoped that attempting to prevent the subjects from making such a conscious effort would create a more natural environment. the subjects in the hypermedia group (n�31) were introduced to a hypermedialearning programme, designed by the researchers for the vocabulary retention. the programme provides users reading an expository english text with a variety of glosses or annotations for words in the form of text, graphics, video and sound, all of which are intended to aid in the understanding and learning of unknown words. the plain text group (n�31) were put into the control group with the same material except for the medium of presentation (i.e. paper). a pre-test containing the actual test items was administered to the subjects before treatment in order to determine how well the subjects knew the contents before treatment. both groups completed an identical pre-test; subjects were asked to answer 40 multiple-choice vocabulary questions in 30 minutes. these 40 questions were selected from words picked out from the course passages. b. gorjian 116 http://www.bbclearningenglish.com based on the interactive theory of reading, two types of annotations were identified as facilitators of top-down and bottom-up processes: textual annotations provide information about the text, such as definitions of words (text annotations), their pronunciation (audio annotations) and illustration (graphics annotations). extra textual annotations provide extra background information about the topic in the form of text, audio, illustration and video. textual annotations were linked directly to the text which gave students the same amount of information about each word, while extra textual annotations, on the other hand, were not directly linked to the text and were presented in the form of encyclopedia in this study. this information was different for each student and students selected different annotations based on their preferences in the form of media the information was available (i.e. text, graphics, sound or video). to make sure everyone had at least studied the passages once and in order to expedite the process, the instructor used a cd player available in the laboratory to play the audio track for the passages, so that students heard the words pronounced by a native speaker. after playing the audio track for each passage the students were asked a few comprehension questions. subjects on the plain text group were taught according to the convention of teaching in normal class in iranian university context. that is to say, a printed form of the material designed for this study was prepared and distributed to the students. students were told to bring their dictionaries (mono and bilingual dictionaries) into the class to compensate for textual annotation in the hypermedia group; the extra textual annotations used in the hypermedia group were not available for these students unless they asked the instructor (as is the case in traditional classrooms). students were told to read the passage in groups (five groups of five, and one group of six students). the reason behind this division was to make the condition as close as possible to the hypermedia group. the criterion for the division of students in conventional group was the mi test (as was for the hypermedia group). the time allocated to complete the task was also the same one hour as was for the hypermedia group. subjects in both groups read a text that contains words that the researchers have targeted for learning, but the subjects did not know this. they read the text in the normal way, that is, they read to comprehend its informational content. two weeks later after the end of the course and again without warning the post-test was conducted. statistical analyses were performed using spss software version 15. results descriptive statistics of the pre-test were computed for both groups. the results showed that both groups were almost at the same level in terms of vocabulary knowledge of the mean scores (before the treatment) which also could be counted as another indication of homogeneity of both groups as it is presented in table 1. table 1. descriptive statistics for pre-test. groups n mean sd hypermedia 31 8.8065 2.01112 plain text 31 8.5323 2.81643 total 62 alt-c 2011 conference proceedings 117 immediate quizzes were administered to the subjects of both groups to check their short-term memory retention. first, mean and standard deviation of each immediate quiz was determined for both groups. then the total mean and standard deviation of these six quizzes were calculated. as can be seen in table 2, hypermedia group outperformed the subjects of the plain text group in each immediate quiz. since descriptive statistics could not offer the researchers valid information to reject or sustain the null hypothesis, a two-tailed independent t-test was run to see whether the observed difference between the groups was significant or not. table 3 presents descriptive statistics of immediate quizzes. the t observed value for immediate quizzes was 3.049, while the critical value is 2.042 at 0.05 level of significance. so the results of the immediate quizzes indicate that the difference between mean scores of both groups was significant enough to reject the null hypothesis. for the post-tests, long-term memory of both groups was necessary. table 4 shows the results of post-test. the results showed that observed t (�0.119) was less than the critical t which indicated that the treatment did not work for the long-term retention. the results of immediate quizzes were in favour of hypermedia group but in the long run both plain table 2. descriptive statistics for immediate quizzes. immediate quizzes groups mean sd min max 1 hypermedia 12.9286 3.28778 2 18 plain text 9.2500 2.81687 4 18 2 hypermedia 7.7857 4.66156 0 18 plain text 7.0833 4.60545 0 16 3 hypermedia 13.3333 3.33563 8 20 plain text 9.7143 3.59894 2 16 4 hypermedia 12.4667 3.43143 2 18 plain text 10.1429 4.10703 6 18 5 hypermedia 12.2222 4.05096 4 20 plain text 12.0000 4.17029 4 20 6 hypermedia 9.1111 3.05505 2 16 plain text 9.0833 3.86643 2 14 total hypermedia 11.3529 4.17716 plain text 9.5658 4.09113 table 3. immediate quizzes results. test groups mean sd tobs immediate quizzes hypermedia 11.3529 4.17716 3.049* plain text 9.5658 4.09113 *significant at pb0.05. table 4. results for post-test. test groups mean sd tobs post-test hypermedia 10.7800 3.28215 �0.119 plain text 10.6731 2.98644 b. gorjian 118 text group and hypermedia group had close results. therefore, the plain text group had a growth in its mean (from 9.5658 to 10.6731); on the other hand, hypermedia group had a regression (from 11.3529 to 10.7800). the hypermedia groups’ retention, although it has regressed, still is higher than that of the plain text group. table 5 shows the results of immediate quizzes and post-test were put into calculation. in other words, hypermedia materials could benefit learners better in short-term effect. however, considering the data in tables 4 and 5 in the long run, there would not be such a big difference between the two groups (plain text group’s mean� 10.6731 and the hypermedia group’s mean�10.7800) and also there was not a significant difference between the groups (progress of the plain text group�0.889 and regression of the hypermedia group��0.475) that could be counted as superiority of one over the other. the results are depicted in figure 1. after the post-test was administered, subjects were asked to take part in a conversation test. to have a fair judgement about their performance another teacher was asked to subjects in scoring their pronunciation. in order to divert their attention from the main objective of this test, the subjects were told they were to be scored based on the degree to which they could remember the passages they were going to be asked to recite. although the subjects were struggling to remember different parts of the passages the instructor was naming, they had no clue that it was their pronunciation that was being scored rather than the memory of the passage they were reciting. after the data were gathered from their conversation test, the scores each subject had achieved from two scorers was averaged, then the obtained score was averaged table 5. matched pairs: immediate quizzes and post-test. test groups quizzes mean sd tobs matched pair hypermedia immediate quiz 11.3529 4.17716 �0.475 post-test 10.7800 3.28215 plain text immediate quiz 9.5658 4.09113 0.889 post-test 10.6731 2.98644 figure 1. learners’ vocabulary retention in hypermedia and plain text groups. alt-c 2011 conference proceedings 119 with the score each learner had gained in the written part of the test. then the scores of both groups were given to statistical t-test analysis and the results are presented in table 6. discussion in light of the results obtained from tables 1 to 6 and also as shown in figure 1, the effect of the treatment on learning and retention of the vocabulary in the long run was not significant. besides, the results of the immediate quizzes indicated that the retention of the vocabularies was better in hypermedia group, and the hypermedia group has outperformed their counterparts in short term in the mean scores of the plain text group (3.049�2.042). the results as indicated in previous sections showed a better retention of vocabularies in the hypermedia group at this stage (hypermedia group’s mean was 11.3529, while the mean for plain text group was 9.5658 at the immediate quiz level). the short-term results of this study were in compliance with the results obtained by davis (1989) and roby (1999). it could be implied from this study and other studies in the field of language learning that hypermedia could help language learners in achieving the desired results but care should be taken. as mentioned above, one of the benefits of hypermedia was providing fast, easy and accessible information. this advantage can turn into disadvantage if it becomes an end to itself rather than a means to an end. concerned with the first research question, the short-term retention of vocabulary was high enough in this study to reject the null hypothesis ( pb0.05); this was also supposed to be the result of the long-term retention of the study which was not obtained. it could be implied from this comparison that the plain text group not only kept the retention of vocabularies at the whole stage of the course, but also showed progress compared to previous tests (pre-test and immediate quizzes test) in comparison to post-test (8.5323b9.5658b10.6731). the hypermedia group showed progress in immediate quiz level but failed to progress in the post-test (8.8065b 11.3529�10.7800). hypermedia was supposed to give an enhanced, comprehensive and in depth knowledge to the subjects to support the routes of the retention of the knowledge. however, the obtained results from this study have indicated that it falls short in proving the expected results of the treatment in the long run. the researchers believe the reason behind these results may lie in the advantage of the treatment over other media. that is to say, the fast, easy and accessible information available at the whole time demolishes the sense of need in subjects, which is in opposition with involvement load hypothesis (laufer and hulstijn 2001). subjects of this study felt no longer the need (thirst) to learn from what was supposed to be a learning experience, simply because they had already found the whole knowledge available table 6. results of multidimensionality vocabulary knowledge. groups n mean sd tobs hypermedia 31 15.2258 2.15576 0.424 plain text 31 15.1333 2.23966 total 62 b. gorjian 120 right in front of their eyes. it seems that providing all the information which a student needs will only work as long as the capacity of his/her short-term memory allows. the subjects of hypermedia group, when given all they need and when they felt that there was no more pressure on them, started to establish what the researchers called ‘‘go-ahead-i-know character’’. in establishing this character, students no longer feel the necessity of the presence of their instructor because they know they can find the answers to their question without asking him/her. in addition, their sense of autonomy rose to a high level which kept them from appreciating the transient nature of hypermedia. it seems that providing all that students need builds a mirage that learners know everything. finding the answers to every question, learners may answer every related question to the passage by the help of textual and extra textual information, especially those which are related to vocabulary, seems to be enough to satisfy the immediate needs of the learners while not providing enough bonds for long-term retention. in other words, subjects, when finding the information available to them, forget their role as learners and just focus on a specific task. it seems that the results of this study support the mental effort hypothesis (rott, williams, and cameron 2002), that since the learners’ effort reduced greatly in learning a vocabulary, the retention of vocabulary has been reduced greatly in the long-term memory. generally, it seems that the transient effect of hypermedia on learning could be to blame but the educational system in the iranian context also contributed. the researchers believe that a factor is the long standing plain text-based nature of language learning and teaching in the iranian educational system: learners were not accustomed to the roles and experiences this study was imposing on them. it seems that studies like this need more time before their real results will be judged. until then serious thinking is needed to prepare the grounds to shift from the present situation to a more cutting edge one. in respect to the second question proposed earlier, the results to this question have indicated that both groups performed equally. it should be added that considering the capacity of hypermedia it is obvious that it can support the multidimensional knowledge of vocabulary in different modalities. however, it became clear that the prime concern of the language learners in reading situations is to grasp the meaning of the unknown vocabulary. other features of the vocabulary take second place for the language learner. the results obtained from this study confirm mayer (2003) who noted that the methods used in an instructional programme, not the delivery media by itself, impacts learning. although every aspect of hypermedia was in favour of the hypermedia group, even a well-equipped study lends its success to practicality of the methods used in that study. in other words, no matter how equipped or multimodal a study could be, as long as the method used in that study does not generate the sense of need in learners it would not grant high-quality results. but what could be said about the success of hypermedia group in the short-term retention of vocabulary could be due to the results of the immediacy of feedback and the effect of short-term memory on retaining the new vocabularies which fade in the longer term. in addition, this study suggests that when language learners are given the liberty to decide on what is right and what is needed for their future success, they are making wrong decisions. iranian efl learners take only what comes from the authoritative power of the class, that is, teachers, as the information to be learned and do not take material as seriously as when it is presented by their teacher. thus as long alt-c 2011 conference proceedings 121 as teachers take the full responsibility of teaching, and learners see themselves as the sole recipient of the presented knowledge to them, independent, autonomous approaches like the one in this study will falter. conclusion in the traditional classes the responsibility is two-folded; on one fold there is teacher, on the other students, whereas in hypermedia classes this responsibility turns out to be three folded where part of the teacher’s knowledge disguises itself in the form of hypermedia. the considerable amount of information either needed by students or imposed by the material poses a great force on teachers, but using computers to take some of these pressures will liberate some time for teachers to think of other important issues. as for extra textual annotations, the assumption was to provide learners with the background knowledge needed to understand the materials easily. it is quite clear that background knowledge plays a crucial role in understanding the materials, but heterogeneous classes in addition to the nature of traditional approaches make presenting background knowledge in the classroom impracticable while in hypermedia activities, learners found the information tailored to their needs. if they lacked any information needed to comprehend the passage, they would refer to extra textual information. the results showed that more than of the subjects used the use of encyclopedia, if available. hypermedia as one of the multidimensional tools of call approach plays a significant facilitative role in developing short-term vocabulary retention and recall. in long-term vocabulary retention this is not necessarily the case but we need to examine the hypermedia efficiency cautiously due to the parameters of the study as well as the effect of the iranian setting as an efl context. the study has provided preliminary work on using hypermedia concerned with wbll tasks in teaching vocabulary. thus there is a need to conduct further experimental research to discover the role of hypermedia annotations in teaching vocabulary and its effects on learners’ vocabulary retention in the long run. references abraham, l.b. 2008. computer-mediated glosses in second language reading comprehension and vocabulary learning: a meta-analysis. computer assisted language learning 21, no. 3: 199�226. baker, j., and h. westrup. 2003. essential speaking skills: a handbook for english language teaching. london: continuum. boers, f., j. eyckmans, and h. stengers. 2004. researching mnemonic techniques through call: the case of multiword expressions. proceedings of the eleventh international call conference, 43�8. antwerp: university of antwerp. boers, f., and s. lindstromberg. 2005. finding ways to make phrase-learning feasible: the mnemonic effect of alliteration. system 33: 225�38. bogaards, p. 2001. lexical units and the learning of foreign vocabulary. studies in second language acquisition 23, no. 3: 321�43. celce-murcia, m., ed. 2001. teaching english as a foreign language. 3rd ed. london: heinle & heinle. chen, j., s. belkada, and t. okamoto. 2004. how a web-based course facilitates acquisition of english for academic purposes. language learning & technology 8, no. 2: 33�49. http:// llt.msu.edu/vol8num2/chen/default.html (accessed january 1, 2009). b. gorjian 122 http://llt.msu.edu/vol8num2/chen/default.html http://llt.msu.edu/vol8num2/chen/default.html cummins, j. 2008a. computer assisted text scaffolding for curriculum access and language learning/acquisition. www.iteachilearn.com/cummins/comptextsum.html (accessed march 24, 2009). cummins, j. 2008b. e-lective language learning: design of a computer-assisted text-based esl/ efl learning system. www.iteachilearn.com/cummins/elective.html (accessed april 18, 2009). davis, n. 1989. facilitating effects of marginal glosses on foreign language reading. the modern language journal 73, no. 1: 41�8. de ridder, i. 2002. visible or invisible links: does the highlighting of hyperlinks affect incidental vocabulary learning, text comprehension, and the reading process? language learning & technology 6, no. 1: 123�46. gorjian, b. 2008. developing linguistic and cultural perspectives of english as a foreign language through email discussion. the jalt call journal 4, no. 3: 3�14. hayati, a. 2005. the computer and language teaching. asian efl journal: professional teaching articles collection 4, 75�81. http://www.asian-efl-journal.com/pta_may_05_hm.php (accessed april 15, 2007). hubbard, p., and levy, m., eds. 2006. the scope of call education. in teacher education on call, ed. p. hubbard and m. levy, 3�20. amsterdam: john benjamins publishing company. hulstijn, j.h. in press. intentional and incidental second language vocabulary learning: a reappraisal of elaboration, rehearsal and automaticity. in cognition and second language instruction, ed. p. robinson, cambridge university press. hulstijn, j.h., and b. laufer. 2001. some empirical evidence for the involvement load hypothesis in vocabulary acquisition. language learning 51, no. 3: 539�58. laufer, b., and j. hulstijn. 2001. incidental vocabulary acquisition in a second language: the construct of task-induced involvement. applied linguistics 22: 1�26. mayer, r.e. 2003. learning and instruction. upper saddle river, nj: prentice hall. mcdonald, c. 2008. unplanned vocabulary instruction in the adult efl classroom. asian efl journal. professional teaching articles, 29�44. http://www.asian-efl-journal.com/ pta_may_08.swf (accessed january 2, 2009). mcwilliam, n. 1998. what’s in a word? vocabulary development in multilingual classrooms. stoke o trent: trentham books. nation, p. 2001. learning vocabulary in another language. cambridge: cambridge university press. nation, p. 2005. teaching vocabulary. asian efl journal 7, no. 3. http://www.asian-efljournal.com/sept_05_pn.pdf (accessed january 26, 2009). nguyen, t.t.h., and t.t.n. khuat. 2003. the effectiveness of learning vocabulary through games. asian efl journal quarterly 5, no. 4. http://www.asian-efl-journal.com/dec_03_vn.pdf (accessed january 11, 2009). o’dowd, r. 2003. understanding the ‘‘other side’’: intercultural learning in a spanish-english e-mail exchange. language learning & technology 7, no. 2: 118�44. http://llt.msu.edu/ vol7num2/odoed/default.html (accessed january 1, 2009). robb, t.n. 2006. helping teachers to help themselves. in teacher education on call, ed. p. hubbard and m. levy, 335�47. amsterdam: john benjamins publishing company. roby, w.b. 1999. what’s in a gloss? language learning and technology 2, no. 2: 94�101. rott, s., j. williams, and r. cameron. 2002. the effect of multiple-choice l1 glosses and input-output cycles on lexical acquisition and retention. language teaching research 6: 183�222. slatin, j.m. 1991. composing hypertext: a discussion for writing teachers. in hypertext/ hypermedia handbook, ed. e. berk and j. devlin, 55�64. new york: intertext. smith, p., and e. stacy. 2003. socialization through cmc in differently structured environments. in learning & teaching with technology: principles and practices, ed. s. naidu, 165�76. london: routledge falmers. son, j.-b. 2007. learner experiences in web-based language learning. computer assisted language learning 20, no. 1: 21�36. alt-c 2011 conference proceedings 123 http://www.iteachilearn.com/cummins/comptextsum.html http://www.iteachilearn.com/cummins/elective.html http://www.asian-efl-journal.com/pta_may_05_hm.php http://www.asian-efl-journal.com/pta_may_08.swf http://www.asian-efl-journal.com/pta_may_08.swf http://www.asian-efl-journal.com/sept_05_pn.pdf http://www.asian-efl-journal.com/sept_05_pn.pdf http://www.asian-efl-journal.com/dec_03_vn.pdf http://llt.msu.edu/vol7num2/odoed/default.html http://llt.msu.edu/vol7num2/odoed/default.html son, j.-b. 2008. using web-based language learning activities. international journal of pedagogies and learning 4, no. 4: 34�43. zhang, b. 2008. presentation modes and vocabulary learning and retention effects. http://bibliotecavirtualut.suagm.edu/glossa2/journal/jun2008/presentation_modes_and_ vocabulary_learning.pdf (accessed march 14, 2009). zimmerman, c.b. 2001. historical trends in second language vocabulary instruction. in second language vocabulary acquisition, ed. j. coady and t. huckin, 5�19. shanghai, china: shanghai foreign wbll tasks language education press. b. gorjian 124 http://bibliotecavirtualut.suagm.edu/glossa2/journal/jun2008/presentation_modes_and_vocabulary_learning.pdf http://bibliotecavirtualut.suagm.edu/glossa2/journal/jun2008/presentation_modes_and_vocabulary_learning.pdf (page number not for citation purpose) 1 *corresponding author. email: jayamalide@uom.lk research in learning technology 2023. © 2023 r.k.j. de silva and a. peramunugamage. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 research in learning technology vol. 30, 2022 original research article emergency remote cad teaching using licensed software in apparel during the covid-19 pandemic: a collaborative learning approach ranapanada kelum jayamali de silvaa* and anuradha peramunugamageb adepartment of textile and apparel engineering, university of moratuwa, moratuwa, sri lanka; bdepartment of electrical and computer engineering, the open university of sri lanka, nugegoda, sri lanka (received: 9 june 2022; revised: 12 november 2022; accepted: 18 november 2022; published: 9 february 2023) computer-aided design (cad) training has become essential in apparel education as it is widely applied in design and development activities in the industry. this study presents how physical cad teaching converted to remote delivery during the emergency covid-19 pandemic using online technologies. this study evaluated five distinct methods adopted in this period: online zoom sessions, pre-recorded practical demonstrations, guided hand-outs, online collaborative learning methods and forum discussions using moodle. teamviewer application was utilised for real-time remote access and support during teaching. this study instrumented two online questionnaires intended to assess the effectiveness of online hands-on sessions and collaborative learning in a remote online environment. this study was conducted with 58 participants at a recognised sri lankan state university. more importantly, the results confirmed the feasibility of collaborative engagement within the online learning environment. this study discovered students’ preferences for synchronous teaching and learning approaches. also, it revealed the limitations of remote cad teaching using online technologies. finally, this study underlined the success of the collaborative learning approach and students’ perspectives on flipped classroom model for apparel cad training. keywords: apparel industry; cad teaching; covid-19; emergency remote teaching; online collaborative learning introduction the covid-19 pandemic hampered most educational programs, especially practical courses (hoofman and secord 2021). educators and learners swiftly adapted their work to online educational platforms in this emergency. although the apparel industry uses computer-aided design (cad) technologies in product development activities, the industry had to entirely depend on the virtual sampling process to receive customer feedback during the covid-19 pandemic. in addition, virtual sampling can effectively employ in emerging mass-personalised apparel businesses mailto:jayamalide@uom.lk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2821 r.k.j. de silva and a. peramunugamage 2 citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 (page number not for citation purpose) (uduwela, de silva, and rupasinghe 2020). therefore, skilled cad professionals are highly demanded in the apparel business. researchers in the field of education tested various instructional strategies for teaching cad (covill, katz, and morris 2008; dosen et  al. 2012; kanetaki et  al. 2020, 2021; lee 2021; vargas et  al. 2019). for example, project-based learning was tested by sola-guirado, guerrero-vacas and rodríguez-alabanda (2022) in a distance mode for engineering students. students in mechanical engineering were taught cad using both ms teams and moodle, whilst pieta (2009) used blended instruction in cad for secondary education. lee (2021) stated the feasibility of creating an efficient e-learning environment for fashion cad education, where students could learn concepts and achieve academic competence without face-to-face instructions. this paper compares different teaching, learning & assessment (tla) methods in emergency remote teaching and suggests the most appropriate delivery of practical-based courses, where hands-on experience is needed. typically, students do not have opportunities to attend collaborative learning activities in online classes (scott et al. 2018). therefore, this paper highlights the application of collaborative learning in the online distance learning environment. overall, this study evaluates the effectiveness of remote and online tla for a licensed apparel cad and shows the limitations of these methods in developing countries. following techniques were started to instruct the students during the pandemic across the world: learning management systems (lms), video conferencing software, broadcast technologies (tv and radio programmes), youtube videos, virtual classrooms/labs and simulators (bates 2015; hayashi et al. 2020; kanetaki et al. 2020, 2021; keengwe 2008; khalil et al. 2020). all educational institutes in sri lanka were closed due to the covid-19 pandemic in march 2020. even though academic institutions switched to online delivery, it was not easy to conduct practical sessions (bhutta et al. 2020; hayashi et al. 2020; khalil et al. 2020). practical sessions are essential in apparel engineering education; therefore, alternative methods to conduct practical sessions were demanded. the lack of digital infrastructure in the country has made the transition to online remote learning mode difficult during the pandemic. however, moodle lms and zoom were the tools our institution used; therefore, this study also used the same tools. the research objectives of this study were as follows: (1) to evaluate the effectiveness of moodle lms, (2) to analyse the most effective learning approach of apparel cad perceived by the students, (3) to measure the effectiveness of the online assessment strategy, (4) to identify limitations of online cad learning and (5) to review online collaborative learning (ocl) using moodle wiki tool, in emergency remote cad teaching during covid-19 pandemic. distance learning and teaching distance learning is also known as e-learning or online learning. there is a physical separation between teachers and students during instruction, and various technologies facilitate teacher–student communication (bates 2015). synchronous and asynchronous are the two modes used in distance learning design. different technologies such as teleconferencing, interactive webinars, chat-based online discussions and phone calls are used in the synchronous approach. asynchronous refers to online courses with weekly deadlines (bates 2015). students can access course content outside scheduled meetings or classes and participate in online discussions, quizzes or http://dx.doi.org/10.25304/rlt.v31.2821 research in learning technology citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 3 (page number not for citation purpose) video comments. online forums, wikis, audio and video recordings are common tools used in asynchronous distance learning (littlefield 2020). including both asynchronous and synchronous online teaching provides more equitable opportunities for those who cannot access face-to-face instruction (peimani and kamalipour 2021). in asynchronous mode, the student is self-directed and can access the materials when needed. lms provides complete flexibility for asynchronous communication and self-directed learning opportunities (deepak 2017; ferreira de farias, spanhol, and vieira de souza 2016; garcia-robles et al. 2009; shylesh 2017). however, the main drawback of e-learning is that it provides only theoretical and restricts the application of practical skills (maatuk et al. 2021). currently, academic institutions are blending and adapting their teaching methods (affairs 2020; world bank 2020), such as intelligent tutoring systems, artificial intelligence, pre-determined learning outcomes, problem-based learning and instructional design approaches to achieve learning objectives irrespective of environmental conditions. the ability of students to connect socially with one another and their choice of communication methods is critical to the effectiveness of online course delivery. therefore, the learning environment must foster a feeling of belongingness; consequently, students may develop the necessary skills for constructive interaction with each other (bender and vredevoogd 2006). institutions will increasingly rely on dynamic video rather than talking heads for demonstrations, simulations and animations, amongst other purposes. simon fraser university has defined experiential learning as the strategic and active engagement of students to learn by doing and reflecting on those activities, which empowers them to apply their theoretical knowledge to practical endeavours in a multitude of settings inside and outside the classroom (beairsto, neufeld, and chinnery 2017). the collaborative learning environment and moodle lms ocl is an educational paradigm that promotes and facilitates student collaboration. lmss have been widely utilised at all levels of education for decades, and moodle is one of the most used open-source platforms internationally (kerimbayev et  al. 2017). moodle’s group activities and forums have facilitated collaboration and communication between students and instructors in an online or hybrid learning environment. moodle wikis enables students to explore topics and create content collaboratively, and it is an excellent collaborative writing and learning tool, which facilitates creating, editing and commenting on assigned tasks (bold 2006; greenhow, robelia, and hughes 2009; lund and smrdal 2006). likewise, moodle features allow course designers to embed and promote interactions between instructors and students. according to costa, alvelos and teixeira (2012), moodle is primarily used to store materials and information, therefore no generalised evidence of a change in pedagogical practice. it also discovered that moodle’s collaborative tools were rarely used due to a lack of awareness. especially, forum or wiki activity enables students to contribute according to their convenience and their own pace. during a live session, students can use moodle’s chat feature. furthermore, the students can use moodle’s workshop activity to collect, review and peer-assessment of learning tasks. students grade their peers’ positions using a multi-criteria form (jabbar and hasmy 2020; kasimatis et al. 2010). therefore, this study includes the application of ocl in this emergency remote learning environment. http://dx.doi.org/10.25304/rlt.v31.2821 r.k.j. de silva and a. peramunugamage 4 citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 (page number not for citation purpose) background of this study this study used 58 third-year students in a textile and apparel degree program at a sri lankan university. this student group learned manual pattern making during their first year at the university. the students joined from all nine provinces of the country, and the average attendance for the online classes was 98%. all the sessions were conducted remotely and online during this covid-19 pandemic. teamviewer software was utilised for realtime remote access to the licensed cad software, and the zoom platform was used to host the online sessions where all the students connect with the teacher. also, moodle lms created the learning environment for entire communication. four practical assignments were completed individually during the semester, and the duration of each assignment was 1 hour. practical assignments of this apparel cad course were mainly designed to (1) create patterns to the given garment specification using the licensed cad software, (2) validate the garment measurements using the pattern measurements, (3) modify the pattern drafts and develop patterns to achieve required styling, (4) perform marker making and marker planning to calculate the fabric consumption and (5) to generate the required number of markers to process the production order. course activities have been redesigned during the pandemic to achieve the above learning objectives. a collaborative group assignment was conducted using moodle wiki to explore current trends and future developments in apparel cad applications. since it was the students’ first online collaborative effort, the course coordinator trained and motivated students. groups of five students were formed, and they contributed to the wiki assignment over 6 weeks. table 1 shows how this apparel cad course was delivered remotely before the covid-19 pandemic and during this emergency. table 1. the apparel cad delivery before covid-19 and during the covid-19. tla the delivery methods before covid-19 the delivery methods during covid-19 teaching, assessment and feedback face-to-face instructions using powerpoint presentations, software demonstrations using a multimedia projector and seminars conducted by solution providers in sri lanka. online delivery using zoom, access to licensed cad software using teamviewer and forum discussion published in moodle. learn and practice face-to-face using the lecturer’s instructions. peer-assisted learning during open access practical sessions. forum discussions in moodle, pre-recorded video demonstrations of the cad software, video files/ url published in moodle and remote access to cad software for open access. peer-assisted learning groups using zoom breakout rooms. practical assessments in-class practical test. printed practical sheets are provided. cad files are submitted to a network location. in-class test online using licensed cad software. practical sheets are uploaded to moodle, and assignments are submitted to moodle. group assessment in-person group discussions and face-to-face slide presentations collaborative group assignment using moodle wiki. http://dx.doi.org/10.25304/rlt.v31.2821 research in learning technology citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 5 (page number not for citation purpose) methodology there were two questionnaire instruments utilised to address all objectives of the study. the first questionnaire instrument consisted of quantitative and qualitative elements, posted as a google form at the end of the semester. quantitative questions used a 5-point likert scale. this course was delivered for 14 weeks, from september to december 2021. the open university of sri lanka granted ethics approval for the study and obtained informed consent from the participants. the first questionnaire contains 22 questions covering all research objectives. questions 1–5 (q1-q5) were designed to obtain preliminary information from the participants, including gender, province, device availability and internet connectivity. q6 had nine sub-questions that examined the effectiveness of moodle lms in emergency remote teaching. q7-q11 analysed the most effective learning approach perceived by students. table 2 lists the five tested methods. q12-q18 measured the effectiveness of the online assessment strategy. q19-q22 discovered the limitations and overall feedback on this emergency remote cad teaching. the second questionnaire was instrumented to review online collaborative teaching and learning based on the given wiki assignment. four questions were asked to determine five themes. these were coded as qw1 – wiki motivation, qw2 – wiki engagement, qw3 – wiki interest, qw4 – wiki discouragement and qw5 – wiki peer intervention. table 2. five key approaches used during the distance learning period. approach (sync/ async) description of the approach objectives of each approach teaching aids 1 synchronous real-time teacher and student communication using zoom. to deliver lessons using real-time student connectivity. zoom central live session, powerpoint presentations and youtube videos. 2 synchronous and asynchronous pre-recorded video demonstrations for practical sessions. to deliver instructions and practical demonstrations for scheduled practicals. moodle learning management system and video-recorded practical demonstrations. gerber accumark software and teamviewer. 3 asynchronous printed handouts to guide the practical session. to available only a written guide to delivering instructions to conduct the practical. moodle, written instructions to self-study the software. gerber accumark software and teamviewer. 4 synchronous group activity – implement student-teacher to available mutual support learning support system. to implement peer assisted learning to support average and slow learners. moodle and zoom breakout rooms. gerber accumark software, teamviewer and practical assignment sheet. 5 asynchronous use forum discussions in moodle. to obtain students’ feedback on how assessments supported learning of the practical component of this module. moodle – forum. http://dx.doi.org/10.25304/rlt.v31.2821 r.k.j. de silva and a. peramunugamage 6 citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 (page number not for citation purpose) data analysis and key findings the responses were imported to a spreadsheet and analysed according to the research objectives. out of all 58 participants, 24 were male and 34 were female. a significant proportion of students (57%) have been connected to this course from western and southern provinces; comparatively, those provinces consist of an urban population with a proper internet connection. the balance of 43% of students is connected from seven provinces in the country; however, each province represents less than 10% of students. all 58 students used laptops to connect to the remote access cad session using teamviewer. students connected to a zoom meeting during the remote cad session using the same laptop or smartphone. out of all 58 students, 14 connected to zoom using a separate smartphone, and the rest used the same laptops. forty-seven households had wi-fi connections, and 33 had mobile 4g connections. q5 measured the overall course content and organisation in general. 93% of students were transparent with the course objectives. 96% of the students mentioned that the learning materials, including lesson plans and course notes, were relevant and valuable. 76% of the students believed the course was well organised and had timely access to materials and notification of changes. objective 1 – evaluate the effectiveness of moodle in emergency remote teaching according to the mean values in table 3, moodle was recognised as an effective lms with enormous benefits in sharing study materials, communicating with the students, submitting assignments and connecting with peers. however, some limitations were experienced in using moodle, which is discussed under objective 5. objective 2 – analyse the most effective learning approach perceived by the students q7 gathered responses related to different learning approaches, as discussed in table 2 and illustrated in figure 1. most students preferred approaches 1 and 2, emphasising synchronous teaching. in the second approach, students tend to watch pre-recorded table 3. feedback on moodle lms. feedback on lms-moodle mean sdev variance all online study materials were helpful 4.23 1.03 1.06 i can access learning activities at any time convenient to me 3.92 0.89 0.79 i faced difficulties working with moodle 3.17 1.28 1.63 online submissions were better than printed submissions 4.07 0.95 0.91 i found the various functions in moodle were well-integrated 4.07 0.95 0.91 moodle enabled me to get to know my peers 3.78 0.99 0.99 moodle enabled me to participate in course discussions comfortably 3.95 0.98 0.96 i could explore new knowledge in this course due to moodle activities 4.20 0.92 0.84 forum discussion helped to collaborate with peers and share knowledge through moodle 4.13 1.00 1.00 http://dx.doi.org/10.25304/rlt.v31.2821 research in learning technology citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 7 (page number not for citation purpose) video sessions before the scheduled class; then at the scheduled class time, students tend to practice what they have seen in the video in class with support from peers and lecturers. study results confirmed that the students were interested in such flipped classroom model. as per q8, 90% of the students agreed that the online teaching materials adequately support practical sessions. with regard to q9, half of the students decided that the online teacher interactions were sufficient to learn and practice cad, and only 2% disagreed. according to q10, only 5% of students found it easy to follow this cad practical work; however, 16% found it extremely difficult to follow it using remote access online learning platforms. moreover, 22% of students found it easy to follow this cad practical work, whilst 38% found it challenging. the study’s ultimate purpose is to examine the causes of such challenges. as per the results of q11, the students confirmed the benefits of the pre-recorded video tutorials uploaded to moodle or shared as a youtube link. furthermore, two students suggested including pattern-making lessons in the same video content. as students highlighted, their lack of pattern-making knowledge hindered them from acquiring cad skills. moreover, few students requested printed material as a learning resource. objective 3 – measure the effectiveness of the online assessment strategy this study could use the practical assessment sheets that were used before the pandemic as the same workload could cover even within this emergency setting. however, the physical group project was replaced with the group wiki activity. according to q12, 98% of students affirmed that continuous assessments (ca) aligned well with the intended learning outcomes (ilos). according to q13, 89% of students said they had been informed about the assessments at the beginning of the semester. out of the 89% of students, 55% strongly agreed with their understanding of the assessment plan. according to the results of q14, 42% of students were not satisfied with the allocated time for each in-class online practical assignment. only 10% of students figure 1. responses for different learning approaches in online remote learning. http://dx.doi.org/10.25304/rlt.v31.2821 r.k.j. de silva and a. peramunugamage 8 citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 (page number not for citation purpose) strongly agreed with the given time for each assignment. students received feedback at the end of each practical assignment, and prevalent mistakes were explained during the online demonstrations. as such, q15 affirmed that 86% of the students mentioned that such assessment feedback was helpful. according to q16, 87.7% of students stated that in-class practical assignments helped improve their apparel cad knowledge and skills in this learning environment. q17 confirmed that 19.3% of students strongly agreed with how they have been assessed on practical cad skills; furthermore, 56.1% of students somewhat decided on the same. moreover, no one strongly disagreed with the evaluation methods. under q18, qualitative feedback was obtained to analyse how online continuous practical assessments supported completing set ilos of the course during this emergency remote teaching. sub-themes motivated to compare practical online delivery to physical practical sessions and suggest additional resources should be in-built on par with this online practice. five of 58 students reasoned that their lack of pattern-making knowledge led to the slow adaptation to this computerised product development system. a statement quoted as ‘it is great if lecturer can provide additional pattern making tutorials within the current course’. underlining one of the main themes, students demanded more ‘open access’ lab hours to perform well in their practical assignments when learning new software. 23 of 58 students explained how additional practice led to higher scores for ca. they showed the limitations of using licensed software in this course as it restricts their practice time. this response will explore alternative solutions to offer a similar learning opportunity. students’ comments inferred that the students’ duration for continuous practical assignments was insufficient. therefore, students’ submissions were further examined as most of the previous practical submissions were not completed; however, this timing issue improved towards the end of the semester. therefore, four students commented on disregarding scores relevant to early assignments and provided some model tests before in-class practical assignments. conversely, five students stated that current continuous assignments are more outcome-based; it acts as a positive drive to learn the subject and acquire practical skills efficiently. on par with the collaborative learning approach, 7 of 58 students declare their enthusiasm for group projects. they highlighted the effectiveness of the peer-assisted learning session using zoom breakout sessions. two sessions were organised for group activities, systematically implementing ‘peer-assisted learning’ and ‘student-teacher’ immediately before the final practical assignment. a student quoted as ‘properly structured, group projects can reinforce various skills while learning the subject’. there was an indication from the core themes to establish practical course delivery in a hybrid model in the future. one student expressed the benefits of online practical assignments: ‘everyone can work independently without distractions. we could refer to our study materials and do the practical and the assignments without a hassle’. through further analysis to evaluate the effectiveness of feedback, it is evident that continuous constructive feedback sessions predominately supported students’ learning, which they should conduct before attempting the next assignment. overall, assessments successfully guided students’ cad skill development and improved their confidence. objective 4 – identify limitations of online cad learning during the pandemic the results of q19-q22 analysed obstacles encountered during the emergency remote cad teaching using online technologies (see figure 2). http://dx.doi.org/10.25304/rlt.v31.2821 research in learning technology citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 9 (page number not for citation purpose) as depicted in figure 2, the most glaring challenges were the poor quality of internet services and the student’s inability to complete the practical session on time power failures. low performance of some of their devices, lack of pattern-making knowledge and absence of open access sessions for additional practice were also identified as obstacles. in addition, they encountered numerous disturbances when connecting from their respective locations. most students desired open lab access to practice this licensed cad software. importantly, they could obtain peer assistance effectively compared to this emergence of remote training. objective 5 – review online collaborative learning using moodle as mentioned earlier, the second questionnaire was instrumented to address this objective. the percentage was computed using the weighted average of the mean and the 5-point likert scale. 48 students completed the second survey. as shown in table 4, learners strongly agreed, and the mean response exceeded three. figure 2. obstacles faced during an in-class online session. table 4. encouragement to work using wiki. qw 1 – wiki participation % mean sdev variance wiki motivated me to participate actively and interact with colleagues 85 4.25 0.636 0.396 i could access content and resources through the wiki 85 4.25 0.636 0.396 peer interactions are promoted through the wiki 82 4.08 0.895 0.785 wiki creates opportunities to generate, present and disseminate knowledge 83 4.17 0.724 0.514 http://dx.doi.org/10.25304/rlt.v31.2821 r.k.j. de silva and a. peramunugamage 10 citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 (page number not for citation purpose) as shown in table 4 and figure 2, more than 82% of the students in the online survey agreed that the online environment and wiki encouraged them to participate actively and communicate with colleagues; get access to material and resources; foster peer relationships; and to offer chances for knowledge generation, presentation and dissemination. as illustrated in the figure 3 box plot, the mean pleasure score for collaborative wiki projects is more than 4, and the mean rating for all demotivating questions is less than 4. students share their knowledge in real time with peers during the task because they may assume that receiving feedback on their understanding and engagement would enhance their knowledge and participation. the data analysis reveals that all mean values are more significant than 3.00, and standard deviations range from 0.78 to 1.145. students found online collaboration more efficient than face-to-face collaboration; they learned more from their peers through online activities than through face-to-face collaboration. they were confident in providing and receiving feedback from/to other students via moodle wikis. additionally, students were asked to evaluate their frequency of access to wiki activity on a scale of 1 to 5. students engage in collaborative activities five to 10 times each week, as depicted in table 6. as a result, the mean is 2.73. additionally, students posted on their sites more than five times each week and commented on peer pages. furthermore, 53% of students check their peers’ work more than five times per week, which indicates that teachers should monitor students’ involvement with learning materials and activities that assist them in meeting course goals. discussion the key findings reflect students’ interest in a synchronous learning environment similar to face-to-face teaching because students are familiar with conventional learning aspects from their primary and secondary education. this is similar to the key table 5. interest and discouragement in working with the wiki. qw 2 – motivation and enjoyment % mean sdev variance i find each task in the online learning activities is easy to complete 78 3.90 0.778 0.593 i have received helpful feedback/comments on my contribution to the online learning activities from tutors 82 4.08 0.679 0.451 i have received helpful feedback/comments on my contribution to the online learning activities from other students 82 4.08 0.539 0.285 i enjoy using the wiki learning activity 79 3.96 0.849 0.707 qw3 – demotivation       i only use online learning activities because it is compulsory for the module 62 3.08 1.164 1.326 i feel reluctant to use the wiki activity 60 3.02 1.194 1.395 i do not like the fact that other people can see my work before it is finished 54 2.69 1.151 1.298 i still have concerns over possible vandalism and plagiarism of my work on the wiki 65 3.25 1.101 1.188 http://dx.doi.org/10.25304/rlt.v31.2821 research in learning technology citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 11 (page number not for citation purpose) findings of the study by kanetaki et  al. (2021), which is a mixed approach of ms teams (synchronous learning) and moodle delightful for students. however, students seek modern, effective educational methods to fulfil their course requirements in an online environment. besides live zoom instructions, students strongly encouraged video-recorded practical demonstrations to be available before the practical sessions. additionally, the authors’ previous study discovered that video-recorded lectures encouraged participants to access the moodle environment and catch up on missed lessons (peramunugamage, usoof, and dias 2019; peramunugamage et al. 2020). as per the results, students highlighted that lack of previous pattern-making knowledge was an obstacle to succeeding in cad. students expected different materials to refresh their pattern-making knowledge, which should have been included in the new learning system. as such, this study strongly recommends flipped classroom model to include in designing online teaching in the future. moodle immensely supported students to succeed in this emergency remote teaching with its elements such as assignment submissions, class links, announcements, chat, h5p plugin for interactive content, forum, wiki, url, video and lesson notes, as reported in other studies (jabbar and hasmy 2020; kanetaki et al. 2021; khalil et al. 2020; liyanagunawardena and williams 2021). in this study, we found that students desired the collaborative learning techniques they had encountered in a traditional learning environment. students are motivated to complete course assessments by their figure 3. factors affected by working with the wiki. http://dx.doi.org/10.25304/rlt.v31.2821 r.k.j. de silva and a. peramunugamage 12 citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 (page number not for citation purpose) teachers and peers. therefore, the wiki activity provided an opportunity to explore the subject’s contemporary developments engagingly than other moodle elements. the teacher created adequate content for students using moodle to make working in the remote context easier. the results revealed that the online course materials were sufficient to support students in achieving the course ilos. the results revealed the necessity of open access and self-directed studies to build confidence in using the system. moreover, as shown in figure 3, formative assessments with constructive feedback helped improve their scores for the final practical assignment. finally, the most critical issue was a lack of equal access to digital infrastructure across the country, which hampered online education in developing countries, as emphasised in unesco and the adb reports during the pandemic (hayashi et al. 2020; world bank 2020). many students said internet connectivity and internet speed issues; similar reflections can be seen in the study conducted by hayashi et al. (2020) for the broader sri lankan audience. the internet issues interrupted students during the cad activities in our study; our students were given extra time or another appointment to complete and submit their work. this highlights the need for educators to be flexible and has procedures to deal with digital infrastructure issues in their courses. conclusion and future recommendations in times of global crisis, social priorities are dynamically reordered worldwide to meet varying demands. during this covid-19 emergency, remote teaching with novel educational styles has been implemented globally temporarily and persistently ( benabdallah and bourgault 2021; khalil et  al. 2020; liyanagunawardena and williams 2021; snoussi et al. 2020). therefore, this study was an excellent example of teaching and learning practical courses online using moodle lms. since this study showed various moodle applications to enhance remote learning, lack of knowledge in moodle applications could not provide the expected results. out of all the applied approaches, students’ highest preference was synchronous teaching methods and flipped classroom model. this study highlights the importance of embedding flipped classroom concepts in our courses, where students can view the pre-recorded demonstration before synchronous practical sessions. moreover, this study showed how ‘peer-assisted learning’ and ‘student-teacher’ could apply in the online learning environment and consequently how they helped to improve students’ grades. all practical assessments were carried out online using remote access to the licensed software, and students received formative and summative feedback where necessary. also, wiki exercise and collaborative feedback mechanisms proved the table 6. wiki participation during a week. qw2 – wiki engagement % mean sdev variance how often do you access the wiki activity during the week? 55 2.73 0.765 0.572 how often do you write on your pages during a week? 48 2.42 0.846 0.701 how often do you write comments on colleagues’ pages during a week? 43 2.19 0.724 0.514 on average, how frequently do you check the work of colleagues on moodle 53 2.65 0.838 0.687 http://dx.doi.org/10.25304/rlt.v31.2821 research in learning technology citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 13 (page number not for citation purpose) success of ocl. through examination of sub-themes, it was found that the insufficient digital landscape hinders online delivery in developing countries. this study demonstrated the success of ocl, an educational paradigm that promotes and facilitates student collaboration. this effort can assist course developers in developing a comprehensive learning solution for preparing apparel cad or similar software that integrates with other subject-specific inputs. most importantly, this flipped classroom and collaborative practice approach can be utilised to train a wider audience with proper learning materials, especially pre-recorded videos. effective e-learning platforms can overcome constraints. as cabero-almenara, arancibia and del prete (2019) suggested, if the teacher is motivated and committed to designing and restructuring the course effectively even when the environment changes abruptly, such good designs can overcome constraints. as a whole, this whole project develops students’ independent learning skills, which will aid them in their future studies. as a result, this implementation was an excellent example of a well-organised venture that resulted in real change through the effective integration of online tools. references affairs, s. (2020) unesco covid-19 education response education sector issue notes school reopening, developed by the section of education policy and unesco international institute for educational planning with support from unesco offices in abuja, bangkok and santiago, pp. 1–8. bates, t. (2015) ‘teaching in a digital age’, teaching in a digital age by anthony william (tony) bates is licensed under a creative commons attribution-noncommercial 4.0 international license, vol. 16, no. 4, p. 99. beairsto, b., neufeld, p. & chinnery, a. (2017) task force on teacher education for the 21st century phase 3 report : thought-provoking models of teacher education. benabdallah, g. & bourgault, s. (2021) ‘remote learners, home makers: how digital fabricationwas taught online during a pandemic’, conference on human factors in computing systems – proceedings, yokohama, japan. doi: 10.1145/3411764.3445450 bender, d. m. & vredevoogd, j. d. (2006) ‘using online education technologies to support studio instruction’, educational technology and society, vol. 9, no. 4, pp. 114–122. bhutta, z. a. et al., (2020) ‘covid-19 risks and response in south asia’, the bmj, vol. 368, no. march, pp. 1–2. doi: 10.1136/bmj.m1190 bold, m. (2006) ‘journal of interactive learning research’, association for the advancement of computing in education (aace), waynesville, nc, vol. 17, no 1, (january), issn 1093-023x cabero-almenara, j., arancibia, m. l. & del prete, a. (2019) ‘technical and didactic knowledge of the moodle lms in higher education. beyond functional use’, journal of new approaches in educational research. vol. 8. no. 1. pp. 27–35. doi: 10.7821/naer.2019.1.327 costa, c., alvelos, h. & teixeira, l. (2012) ‘the use of moodle e-learning platform: a study in a portuguese university’, procedia technology, vol. 5, pp. 334–343. doi: 10.1016/j. protcy.2012.09.037 covill, d., katz, t. & morris, r. (2008) teaching and assessing cad using online demonstrations, ds 46: proceedings of e and pde 2008, the 10th international conference on engineering and product design education. universitat polytecnica de catalunya, barcelona, spain, 4–5 (september). deepak, k. c. (2017) ‘evaluation of moodle features at kajaani university of applied sciences-case study’, procedia computer science, vol. 116, pp. 121–128. doi: 10.1016/j. procs.2017.10.021 http://dx.doi.org/10.25304/rlt.v31.2821 http://dx.doi.org/10.1145/3411764.3445450 http://dx.doi.org/10.1136/bmj.m1190 http://dx.doi.org/10.7821/naer.2019.1.327 http://dx.doi.org/10.1016/j.protcy.2012.09.037 http://dx.doi.org/10.1016/j.protcy.2012.09.037 http://dx.doi.org/10.1016/j.procs.2017.10.021 http://dx.doi.org/10.1016/j.procs.2017.10.021 r.k.j. de silva and a. peramunugamage 14 citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 (page number not for citation purpose) dosen, a. s. et al., (2012) ‘teaching cad: the challenges of online delivery to distance learning students’, proceedings of the australasian universities building educators association (aubea): 37th annual international conference, (july), pp. 44–55. available at: https:// ogma.newcastle.edu.au/vital/access/manager/repository/uon:11758 ferreira de farias, g., spanhol, f. j. & vieira de souza, m. (2016) ‘the use of lms to support pbl practices: a systematic review’, iosr journal of research & method in education (iosr-jrme), vol. 6, no. 5, pp. 51–59. doi: 10.9790/7388-0605035159 garcia-robles, r. et  al., (2009) ‘an elearning standard approach for supporting pbl in computer engineering’, ieee transactions on education, vol. 52, no. 3, pp. 328–339. doi: 10.1109/te.2008.928220 greenhow, c., robelia, b. & hughes, j. (2009) ‘learning, teaching and scholarship in a digital age’, educational researcher, vol. 38, no. 4, pp. 246–259. doi: 10.3102/0013189x09336671 hayashi, r. et al., (2020) online-learning-sri-lanka-during-covid-19, vol. 5, no. 151, asian development bank, mandaluyong, philippines. doi: 10.22617/brf200260-2 hoofman, j. & secord, e. (2021) ‘the effect of covid-19 on education’, in pediatric clinics of north america. pediatr clin north am. oct; vol. 68 no. 5, pp. 1071–1079. pmc8445757 doi: 10.1016/j.pcl.2021.05.009 jabbar, a. & hasmy, m. (2020) effective use of collaboration tools in moodle lms by lecturers and students at south eastern university of sri lanka, journal of information systems & information technology (jisit). kanetaki, z. et  al., (2020) ‘machine learning and statistical analysis applied on mechanical engineering cad course: a case study during erte pahse in the context of higher education’, 4th international symposium on multidisciplinary studies and innovative technologies, ismsit 2020 – proceedings, istanbul, turkey. doi: 10.1109/ismsit50672.2020.9254924 kanetaki, z. et al., (2021) ‘the impact of different learning approaches based on ms teams and moodle on students’ performance in an on-line mechanical cad module’, global journal of engineering education, vol. 23, no. 3, pp. 185–190. kasimatis, a. et  al., (2010) ‘a new learning analytics tool in moodle for assessingstudents’, performance bulletin of the ieee technical committee on learning technology, vol. 16, no. 1, january 2014, pp. 1–7. keengwe, j. (2008) ‘the use of computer tools to support meaningful learning’, aace journal, vol. 16, pp. 77–92. kerimbayev, n., kultan, j. abdykarimova, s., & akramova, a. (2017) ‘lms moodle: distance international education in cooperation of higher education institutions of different countries. educ inf technol, vol. 22, pp. 2125–2139. https://doi.org/10.1007/s10639-016-9534-5 khalil, r. et  al., (2020) ‘the sudden transition to synchronized online learning during the covid-19 pandemic in saudi arabia: a qualitative study exploring medical students’ perspectives’, bmc medical education, vol. 20, no. 1, pp. 1–10. doi: 10.1186/ s12909-020-02208-z lee, y. k. (2021) ‘fashion cad education during the covid-19 pandemic in south korea: comparison of online and offline learning achievements’, international journal of fashion design, technology and education, vol. 15, no. 2, pp. 245–255. doi: 10.1080/ 17543266.2021.2017005 littlefield, j. (2020) the difference between synchronous and asynchronous distance learning, thoughtco, available at: http://thoughtco.com/synchronous-distance-learning asynchronous-distance-learning-1097959 liyanagunawardena, t. r. & williams, s. a. (2021) ‘emergency remote education: experience from sri lanka during covid-19’, asian journal of distance education, vol. 16, no. 1, p. 2021. available at: http://www.asianjde.com/ lund, a. & smørdal, o. (2006) ‘is there a space for the teacher in a wiki?’, proceedings of wikisym‘06 – 2006 international symposium on wikis, odense, denmark, pp. 37–46. http://dx.doi.org/10.25304/rlt.v31.2821 https://ogma.newcastle.edu.au/vital/access/manager/repository/uon https://ogma.newcastle.edu.au/vital/access/manager/repository/uon http://dx.doi.org/10.9790/7388-0605035159 http://dx.doi.org/10.1109/te.2008.928220 http://dx.doi.org/10.3102/0013189x09336671 http://dx.doi.org/10.22617/brf200260-2 http://dx.doi.org/10.1016/j.pcl.2021.05.009 http://dx.doi.org/10.1109/ismsit50672.2020.9254924 https://doi.org/10.1007/s10639-016-9534-5 http://dx.doi.org/10.1186/s12909-020-02208-z http://dx.doi.org/10.1186/s12909-020-02208-z http://dx.doi.org/10.1080/17543266.2021.2017005 http://dx.doi.org/10.1080/17543266.2021.2017005 http://thoughtco.com/synchronous-distance-learning-asynchronous-distance-learning-1097959 http://thoughtco.com/synchronous-distance-learning-asynchronous-distance-learning-1097959 http://www.asianjde.com/ research in learning technology citation: research in learning technology 2023, 31: 2821 http://dx.doi.org/10.25304/rlt.v31.2821 15 (page number not for citation purpose) maatuk, a. m. et al., (2021) ‘the covid-19 pandemic and e-learning: challenges and opportunities from the perspective of students and instructors’, journal of computing in higher education, vol. 34, no. 1, pp. 21–38. doi: 10.1007/s12528-021-09274-2 peimani, n. & kamalipour, h. (2021) ‘online education and the covid-19 outbreak: a case study of online teaching during lockdown’, education sciences, vol. 11, no. 2, p. 72. doi: 10.3390/ educsci11020072 peramunugamage, a. et al., (2020) ‘problem-based learning (pbl) in engineering education in sri lanka: a moodle based approach’, advances in intelligent systems and computing, vol. 1134, pp. 770–780. doi: 10.1007/978-3-030-40274-7_74 peramunugamage, a., usoof, h. & dias, p. (2019) ‘can technology make a difference to the level of engagement within large classes in engineering education? a case study in sri lanka’, in icl2018 – 21th international conference on interactive collaborative learning, kos island, greece, pp. 826–833. doi: 10.1007/978-3-030-11935-5_78 pieta, d. (2009) ‘study on blended instruction in automotive technology for secondary education students’, tcc 2009: educational technology master’s paper, university of hawai‘i at mānoa honolulu, hawaii, u.s.a (december), pp. 17–18. scott, c. e. et al., (2018) ‘the what, when, and how of preservice teachers and literacy across the disciplines: a systematic literature review of nearly 50 years of research’, teaching and teacher education, vol. 73, pp. 1–13. doi: 10.1016/j.tate.2018.03.010 shylesh, s. (2017) ‘teaching with moodle in higher education’, ssrn, vol. 47, pp. 1320–1324. doi: 10.2139/ssrn.2967882 snoussi, t. et  al., (2020) ‘social media for learning: perceptions and behaviors’, periodicals of engineering and natural sciences, vol. 8, no. 4, pp. 2195–2207. doi: 10.21533/pen. v8i4.1709.g700 sola-guirado, r. r., guerrero-vacas, g. & rodríguez-alabanda, ó. (2022) ‘teaching cad/ cam/cae tools with project-based learning in virtual distance education’, education and information technologies, vol. 27, pp. 5051–5073. doi: 10.1007/s10639-021-10826-3 uduwela, w. c. et al., (2020) ‘digital transformations in the apparel value chain for mass personalization’, in ieee international conference on industrial engineering and engineering management, singapore, vol. 2020 december. doi: 10.1109/ieem45057.2020.9309852 vargas, j. et al., (2019) ‘computer systems and cad/cae in teaching engineering’, moj civil engineering, vol. 5, no. 1, pp. 27–29. doi: 10.15406/mojce.2019.05.00145 world bank. (2020) the covid-19 crisis response : supporting tertiary education for continuity, aaptation, and innovation. available at: http://pubdocs.worldbank.org/ en/621991586463915490/wb-tertiary-ed-and-covid-19-crisis-for-public-use-april-9.pdf http://dx.doi.org/10.25304/rlt.v31.2821 http://dx.doi.org/10.1007/s12528-021-09274-2 http://dx.doi.org/10.3390/educsci11020072 http://dx.doi.org/10.3390/educsci11020072 http://dx.doi.org/10.1007/978-3-030-40274-7_74 http://dx.doi.org/10.1007/978-3-030-11935-5_78 http://dx.doi.org/10.1016/j.tate.2018.03.010 http://dx.doi.org/10.2139/ssrn.2967882 http://dx.doi.org/10.21533/pen.v8i4.1709.g700 http://dx.doi.org/10.21533/pen.v8i4.1709.g700 http://dx.doi.org/10.1007/s10639-021-10826-3 http://dx.doi.org/10.1109/ieem45057.2020.9309852 http://dx.doi.org/10.15406/mojce.2019.05.00145 http://pubdocs.worldbank.org/en/621991586463915490/wb-tertiary-ed-and-covid-19-crisis-for-public-use-april-9.pdf http://pubdocs.worldbank.org/en/621991586463915490/wb-tertiary-ed-and-covid-19-crisis-for-public-use-april-9.pdf student reaction to parallel hypertext and menu-based interfaces c.k. ramaiah, mubarak sulaiman and a.j. meadows department of information and library studies, loughborough university, loughborough le11 3tu abstract it is often necessary to consider the question of what sort of interface is most useful for retrieving information from a particular land of database. a small database of text-based but multi-faceted items is used here in order to compare ease and speed of retrieval from two commonly used combinations of interface — hypercard on an apple macintosh and dbase iii+ on a pc. for the restricted range of tasks employed here, the latter combination appears to be more acceptable to students with limited computer experience. however, in more general terms, the acceptability of an interface for information retrieval depends on what particular aspect of information retrieval is being emphasized, and what conceptual frameworks students bring to their tasks. introduction many types of interface to electronic databases now exist. however, they can readily be grouped into a small number of categories according to their style of approach. thus shneiderman (1991) has recently suggested a taxonomy which distinguishes five styles of interaction. these consist of: menu selection, form fill-in, command language, natural language, and direct manipulation. they can be used as a convenient basis for considering and comparing the properties of different types of interface. interfaces aimed at beginners in electronic information retrieval typically rely heavily on computer-initiated interaction with users. two of the commonest such interfaces currently are (1) menu-based database management systems (dbms) and (2) hypertext systems. the differences between these interfaces correspond to two differing interaction styles in shneiderman's classification. the former come within the 'menu selection' group, while the latter are classified under 'direct manipulation'. the styles, and correspondingly the interfaces, 17 c.k. ramaiah et al style menu selection direct manipulation advantages little training required. easy structure for decision-making. easy to remember, once learnt. high subjective satisfaction rating. student reaction to hypertext and menus disadvantages access via a sequence of menus may be irritating. lack of flexibility. may be cumbersome to initiate some actions. navigation may be difficult. table 1: advantages/disadvantages of different interaction styles differ in their advantages and disadvantages, as indicated below in table 1 (where only those relevant to the present study have been included). these particular two interaction styles are especially interesting from the viewpoint of teaching requirements, because they relate to a fundamental distinction (which is, perhaps, understressed in shneiderman's formulation). this concerns the difference between ease of learning (the ability of a novice user to become proficient in using a system with minimal training) and ease of use (the ability of knowledgeable users to perform tasks with minimum impediment). our main purpose here is to use shneiderman's differentiation of styles, with this slight change of emphasis, to compare the introductory presentation of two popular database interfaces to university students. the present study is concerned with evaluating the comparative usability of a menu-based interface (dbase m + and pc) and a hypertext interface (hypercard and macintosh). for this study of 'usability', the specific set of users comprised student volunteers from the department of information and library studies at loughborough university. they were observed while using the two computer combinations in an isolated laboratory as a means of retrieving a series of items from a common database. the study concentrated on examining two parameters: performance in answering a set of test questions, and user ratings. the two commercial packages investigated here dbase m + and hypercard have different overall purposes. the dbase iii+ package is a relational database management software program. hypercard is a programming system which uses a graphical user-interface to relate on-screen objects to an underlying database. however, both obviously provide informationretrieval facilities which can be studied in parallel. it is hoped that the results will cast some light on the wider question of the differences between menu-based and hypertext-based information retrieval. both packages are widely used for educational purposes. there have been several general studies of ways in which hypercard can be employed in teaching (eckols and rossett, 1989; muir, 1992) and training (flynn, 1988). there has also been a more limited number of 18 altj volume 1 number 2 subject-specific studies. the most interesting of these for the present work have been in psychology (giesler-brenstein and brenstein, 1989) and in teaching information systems (fritz, 1991). there have been considerably fewer studies of the use of dbase hi+ in teaching. some attention has been paid to it in terms of evaluating different dbms for teaching (miller, 1986), but there has been little in generally available sources on its use for teaching information and library studies (burrough, 1987). method the database used in this study was a catalogue of teaching software prepared by the computers in training initiative centre for library and information studies, which is situated in the department at loughborough. this database has the advantage of being of direct interest to library and information students. it is a small catalogue (containing 40 entries), but each entry may need to be retrieved under a range of headings. altogether there are 11 aspects of each software entry in terms of which retrieval might be necessary (i.e. who uses the software for teaching, what sort of teaching is involved, how much does the software cost, etc.). all retrieval headings are available simultaneously on the screen. they comprise: software designation, card number, type of software, price, author, author address, known users, hardware requirements, availability, supplier, teaching applications. the hypercard and dbase ih+ interfaces to the database were prepared in parallel by the authors of this paper. the intention in each case was to provide an interface which was acceptable to relative beginners in the art of information retrieval, but which also exploited the particular advantages of each type of interface. both versions were pre-tested with students and staff from loughborough and elsewhere. modifications were made, where necessary, in the light of this pre-testing. it should be emphasized that although two interfaces were used, the catalogue content remained the same for both. volunteers were obtained from among undergraduate and postgraduate students in the department of information and library studies. altogether 40 students participated, 9 of whom were postgraduate research students, 22 postgraduate master's course students and 9 ' undergraduate students. they were asked to answer a series of questions concerning the contents of the database using successively the hypercard/mac and dbase/pc combinations. to allow for transfer effects (i.e. experience from the first trial affecting the results of the second), the participants were divided into two groups which will be referred to from here on as group a and group b each consisting of 20 members. members of group a used the hypercard/mac combination first, while members of group b used the dbase/pc combination first. group a consisted of 14 females and 6 males, whereas group b contained 6 females and 14 males. the spread of ages of the two groups was similar, with averages of 31.3 years for group a and 33.0 years for group b (with standard deviations of +/7.7 and +/5.9, respectively). in each group, 18 members had had previous training on computers (mainly microcomputers), while two had not. previous training on computers had mainly been on pcs, with word-processing as the most popular activity. of group a members, 13 had previous experience with the macintosh as compared with 9 members of group b. previous acquaintance with hypercard and dbase iii+ was similar between the two groups, as indicated in table 2. 19 c.k. ramaidh et al student reaction to hypertext and menus previous experience with hypercard previous experience with dbase m + group a yes no 3 15 8 10 group b yes no 5 13 11 7 table 2: participants' prior acquaintance with hypercard and dbase 7zt+ the participants consisted of a mix of 20 overseas students and 20 uk-based students, evenly divided (10 overseas and 10 home-based students) in each group. this division formed an important part of the experiment, since the overseas students differed systematically in terms of background from home-based students. in general, overseas students were more experienced in terms of the use of catalogues (i.e. their acquaintance with the domain under test was better), but they had had appreciably less experience with computers (i.e. their system knowledge was less extensive). the interesting question, therefore, is whether overseas students have differing training needs from other students. the evaluation exercises were carried out with one member of group a and one of group b participating simultaneously. as far as possible, each pair of participants was matched (for example if the group a participant was an older overseas student, so would be the group b participant). this was intended to compensate for any changes in the environment (time of day, external noise, etc.) as the evaluation exercises proceeded. students were set a series of 12 questions concerning the contents of the catalogue. finding the answers entailed accessing different parts of the database, starting each time from the initial screen. the questions were graded so that the routes by which the answers could be retrieved varied from the obvious (in terms of the initial menu) and quick (in terms of the number of steps required) to the less obvious and more roundabout. in terms of this grading, the questions were asked in random sequence (to allow for any learning experience that might accrue during the experiment itself). the time taken to find an answer to each question was recorded by an observer, together with the number of steps required to answer the question. (a 'step' here is defined as any input by the user to the system, for example a command to move from one screenful of information to another.) the most efficient routes for answering the questions had been worked out beforehand by the authors. the number of steps taken by a participant was compared with this predetermined figure: any excess was recorded under the heading 'extra steps'. the intention here was to examine whether differences in domain knowledge and systems knowledge were reflected in the number of steps required to answer a question. results from the performance tests table 3 shows the times taken (in seconds) averaged over all the participants' answers, along with the number of extra steps similarly averaged. there is no significant difference (at the 1% level) in times and extra steps taken between groups a and b. (the implication is that 20 alt-j volume 1 number 2 group a group b number in group 20 20 time taken for hypercard 136 126 time taken for dbase m + 104 110 extra steps for hypercard 6.4 5.9 extra steps for dbase m + 1.8 2.0 table 3: a comparison of retrieval (group a v. group b) any transfer effect works to the same extent in either direction.) however, there are differences in both time and steps taken between hypercard and dbase m+, with the former offering a less rapid and less obvious passage for the user. (only the difference in the number of steps is significant at the 1% level.) an alternative breakdown of the data provides additional illumination. table 4 shows the time and extra steps taken separately for overseas and home-based students. the significant difference between hypercard and dbase iii+ times remains for both groups, but the overseas overseas home-based number in group 20 20 time taken for hypercard 165 98 time taken for dbase m + 137 77 extra steps for hypercard 6.7 5.7 •extra steps for dbase m + 1.5 1.5 table 4: a comparison for retrieval (overseas students v. home-based students) students clearly have greater problems in rapid retrieval than home-based students (t = 4.075 for hypercard and 3.760 for dbase hi+, as compared with a two-tailed probability of 2.704 at the 1% level.) there was, however, no significant difference in the number of extra steps taken. the value of prior acquaintance with the system for rapid retrieval can be examined by looking separately at the results for students who had used a mac previously. (since most of the students had a prior acquaintance with pcs, this factor could not be examined separately.) as table 5 shows, prior experience of a mac appreciably affected speed of extracting 21 c.k. ramaiah et al student reaction to hypertext and menus used mac previously not used mac previously number in group 22 18 time taken for hypercard 105 164 extra steps for hypercard 5.4 7.1 table 5: a comparison of retrieval (mac users v. non-users) information via hypercard (t = 3.20 at the 1% level), but not the number of extra steps taken. it will be noted that the average time recorded for experienced mac users retrieving information via hypercard is similar to the average time recorded in table 3 for retrieval by all users via dbase eh+. this may be compared with the effect of prior knowledge of the hypercard software. as table 6 shows, there is no significant difference in retrieval times between those who had previous experience of hypercard no previous experience number in group 8 32 time taken for hypercard 124 133 extra steps for hypercard 6.1 6.3 table 6: a comparison of retrieval (hypercard users v. non-users) previous experience of dbase m + no previous experience number in group 19 21 time taken for hypercard 97 116 extra steps for hypercard 1.6 2.1 table 7: a comparison of retrieval (dbase iii+ users v. non-users) 22 alt-j ' volume 1 number 2 used hypercard before, and those who had not (though the number of students in the former category was small). almost half of the students had used dbase di+ before. table 7 suggests some difference in retrieval speed by this group, but it is not significant at the 1% level. results from the user ratings at the end of the exercises on mac/hypercard and pc/dbase m + , participants were asked to complete a questionnaire recording their subjective assessment of the two interfaces. in the first place, they were asked with which interface they found it easier to retrieve information rapidly. the responses (contained in table 8) indicate a strong preference among both overseas and home-based students for dbase iii+. this accords with the picture derived from the timings in table 3 a point worth noting, since some evidence exists for supposing that users are not always able to recognize which is the fastest interface (hayes, 1989). there was much less agreement concerning which was the more flexible interface for seeking information. table 9 suggests a fairly even division of opinion, but that conceals an appreciable difference between overseas and home-based students. the latter were appreciably more in favour of hypercard, and the former in favour of dbase iii+. the participants were next asked about the appearance of the information on the screen and how well they could absorb it. as table 10 indicates, both interfaces proved to be generally hypercard dbase m + equal total 8 28 4 overseas 5 13 2 home-based 3 15 2 table 8: which systems permitted more rapid retrieval? hypercard dbase ffl+ total 22 18 • overseas 9 11 home-based 13 7 table 9: which system was more flexible? 23 c.k. ramaiah et al student reaction to hypertext and menus user grading poor acceptable good hypercard 2 15 23 dbase m + 0 11 29 table 10: relative assessment of presentation user grading hard fairly hard fairly easy easy hypercard 3 10 14 13 dbase m+ 0 2 16 22 table 11: relative assessment of movement between screens user grading poor acceptable good hypercard 8 19 13 dbase m+ 0 11 29 table 12: relative assessment of organization for rapid retrieval acceptable in this respect. the main difference was that some users found the hypercard screen too full of different types of information for a beginner to disentangle easily. a typical comment was that dbase iii+ was easier to read, but not so attractive as hypercard. (overseas and home-based students agreed in these comments.) 24 alt-j volume 1 number 2 participants were next questioned concerning ease of movement from one screenful of information to another. their responses (table 11) • show that greater difficulty was encountered with hypercard. this point was taken up in a further question, which asked participants to assess how well-organized the information was to allow rapid retrieval via the two interfaces. their responses (table 12) clearly favour dbase m + , in correspondence both with observation of their retrieval activities and of the measured times. conclusion along with the hands-on tests and the questionnaires, the student participants were asked to write short essays on their opinions of the two systems they had used. data from all three sources has been used here in assessing the results: agreement between them seems to be good. it is clear, in the first place, that dbase 111+ and hypercard provide good examples of shneiderman's 'menu selection' and 'direct manipulation' styles, respectively. as he suggests, menu selection is easier; navigation, for the latter, is more difficult; and so on. in practice, the relative importance of the advantages and disadvantages he lists depends on the form of the interface as much as on the style. in this case, the initial problems of handling a mac compounded the problems of handling a hypertext interface for the first time. a typical comment was: 'hypercard looks more user-friendly, but the use of a mouse is not as familiar to me as using a keyboard; so i prefer dbase m + ' . however, ultimate judgement of an interface must depend on its suitability for carrying out the kind of task that immediately concerns the user. for library and information students, the typical requirement for this kind of database is rapid retrieval. as the results show, experienced mac users were retrieving information via the hypercard interface at only about the same rate as average users of dbase iii+. this disadvantage outweighed the advantages of hypercard for many students. as one of them commented: 'i thought that pc/dbase m + was clearly laid out, uncluttered and could get the required information without resorting to "gimmicks". the mac/hypercard combination is more fun to use, but if information finding is the main prerequisite, then pc/dbase iii+ wins.' overseas students were even more strongly in favour of dbase hi+ than home-based students. for example, home-based students agreed with the implication of shneiderman's classification that hypercard was a more flexible interface; this was less accepted by overseas students. here, it seems from the responses that their greater domain knowledge over other students is at work. the dbase iii+ interface resembles more closely the cataloguing approach with which most overseas students are well acquainted, and they therefore find it easier to accept. the implications for teaching are twofold. first, new forms of interface (hi this case, hypertext) may best be presented initially in terms of their special capabilities, rather than as an alternative for standard activities which may fit better from a student's viewpoint into another framework (and so a different interface). secondly, the greater the knowledge students have of a particular domain, the more likely they are to prefer an interface that fits their conceptual framework. '25 c.k. ramaiah et al student reaction to hypertext and menus references burrough, h.c. (1987), exploring dbase ih+: the development of a database for use in the teaching of library and information studies (m.sc. dissertation, loughborough university). eckols, s.l. and rossett, a. (1989), 'hypercard for the design, development and delivery of instruction', performance improvement quarterly, 2, 2-20. flynn, l. (1988), 'more firms using hypercard for training as a front end', infoworld, 10, 6. fritz, j.m. (1991), 'hypercard application for teaching information systems', sigcse bulletin, 23, 55-61. geisler-brenstein, e. and brenstein, r. (1989) 'the potential of hypercard for psychological research and instruction', behavior research methods, instruments and computers, 2 1 , 307-311. hayes, b. (1989), designing and implementing user interfaces for microcomputer programs, esrc programme on taxation, incentives and the distribution of income, lse, london. miller, r.p. (1986), 'using geocoded databases in teaching urban historical geography', journal of geography, 85, 148-153. muir, m. (1992), 'how do you run hypercard projects?', computing teacher, 20, 10-12. shneiderman, b. (1991), 'a taxonomy and rule base for the selection of interaction styles' in shackel b. and richardson, s.j. (eds), human factors for informatics usability, cambridge, cup, 325-342. 26 (page number not for citation purpose) 1 *corresponding author. email: mariaraouna@gmail.com research in learning technology 2020. © 2020 m. loizou and k. lee. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 research in learning technology vol. 28, 2020 original research article a flipped classroom model for inquiry-based learning in primary education context maria loizoua* and kyungmee leeb aprimary education, cyprus ministry of education and culture, episkopi, limassol, cyprus; beducational research, lancaster university, lancaster, uk received: 17 june 2019; revised: 29 march 2020; accepted: 5 may 2020; published: 9 july 2020 a multi-case study will be presented in this publication which aimed to address an important gap in the current literature concerning the effective implementation of a flipped classroom (fc) model in a particular educational setting. there has been limited research focusing on utilising a fc model within the primary education context despite its potential benefits for young students, such as facilitating student-centred inquiry-based learning (ibl) and developing their higher order cognitive skills. this multi-case study has been drawn from authors’ collaborative action research project with other teacher participants, during which the authors explored the effective ways in which a fc model can be utilised to promote students’ ibl in primary school settings. the authors first develop an inquiry-based flipped classroom (ib-fc) model and applied the model into five primary schools in cyprus for a school year (2017–2018). a total number of five teachers, 77 students and 48 of their parents were invited to participate in the project. a large volume of qualitative data was collected mainly through classroom observations and interviews. data analysis of teachers’, students’ and parents’ experiences and perceptions led to the development of seven universal design principles. these principles can be used to support primary school teachers’ attempts to design effective instructions using the ib-fc model. keywords: flipped classroom; inquiry-based learning; primary education; universal design principles introduction and literature review over the past decades, education standards have stressed the value of student-centred learning in which students are responsible of their own learning and they are actively involved in higher order cognitive tasks (hannafin and land 1997; shea et al. 2012). a flipped classroom (fc) model has attempted to achieve these standards by allocating more class time for student-centred learning activities and by leveraging accessibility and use of advanced technologies to support a blended learning approach. the fc model is ‘a pedagogical model in which direct instruction moves from the group learning space to the individual learning space, and the resulting group space is transformed into a dynamic, interactive, learning environment’ (flipped learning network 2014). that is, the fc model offers students access to online video lectures mailto:mariaraouna@gmail.com http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v28.2287 m. loizou and k. lee 2 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) and tutorials (i.e. flips) prior to in-class sessions, so that they are prepared to take part in more interactive, collaborative and higher order cognitive activities such as research, debates, problem-solving and discussions (i.e. inquiry-based learning [ibl]) (bergmann, overmyer, and wilie 2012; davies, dean, and ball 2013; fulton 2012; hughes 2012; lage, platt, and treglia 2000; talbert 2012; zappe et al. 2009). students benefit from the pre-class exposure to content and outside classroom events because they can effectively distribute their time and independently pace their learning through online options given to meet their individual learning needs, cognitive abilities and prior knowledge. during in-class sessions, students have a range of opportunities to engage with ibl activities, mainly through actively participating in group work, rather than passively listening to the teacher (i.e. lecturing). teachers, in turn, spend most of their in-class time for facilitating student-centred learning and monitoring student performance and progress and providing adaptive and instant feedback to individual students or to student groups accordingly (fulton 2012; herreid and schiller 2013; hughes 2012). flipped classroom there has been growing research attention to utilising the fc model, grounded on its pedagogical potential to enrich teaching and learning activities, promote better pedagogical outcomes (giannakos, krogstie, and chrisochoides 2014) and improve overall learners’ experiences and competencies (bergmann and sams 2012). although some researchers (e.g. rutherfoord and rutherfoord 2000; tenneson and mcglasson 2006) have argued that the fc model is neither a new nor an innovative pedagogical approach – given that teachers have always used pre-class learning materials such as readings or pre-quizzes to better prepare their students for in-class lessons – the fc model has gained a fast-growing attention in current technology-enhanced learning (tel) contexts (strayer 2012). tel research concerning the implementation of the fc model has exclusively focused on higher education contexts with minor exceptions (see hultén and larsson 2016; kim and chin 2011). when it comes to primary education contexts, there have been general concerns about the feasibility of the fc practices due to a lack of self-regulation skills among primary school students at a relatively young age. moreover, most fc studies have focused on comparing the fc model with more traditional pedagogical approaches with an educator’s perspective (herreid and schiller 2013; teo et  al. 2014) and only a few have more clearly examined students’ perceptions of the usefulness and attractiveness of the model. students’ experiences during the three phases of an fc model (i.e. pre-class, in-class and after-class phases), which could effectively inform the effective design, implementation and evaluation of fc practices, are rarely available, thus necessary in primary education context in particular (kim and chin 2011). therefore, there has been a limited understanding of the effectiveness of the fc implementation, particularly in primary education settings. inquiry-based learning ibl can be defined as ‘the process of posing questions, problems or issues, gathering information, thinking creatively about possibilities, becoming proficient in providing http://dx.doi.org/10.25304/rlt.v28.2287 research in learning technology citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 3 (page number not for citation purpose) evidence, making decisions, justifying conclusions, and learning the ways of challenging, building upon and improving knowledge of the topic or field of study’ (friesen 2013). ibl, therefore, encourages students to explore, discover, collaborate and communicate with their peers (laursen and kogan 2014; stephenson 2012) by operating multiple perspectives (short and harste 1996). however, it has been reported as a rather challenging task to implement effective ibl activities in primary education contexts (capaldi 2015), given that the necessary higher order skills as missing. similar to the concerns about the feasibility of the effective implementation of the fc model in primary education settings, young students are often perceived being incapable or less capable to complete meaningful ibl activities without direct, ongoing teacher interventions (kim and chin 2011). this can be proved very challenging especially for those low achievers who do not possess a required level of prior knowledge (flick and lederman 2004). previous researchers also observed the negative attitudes towards and worries about their potential failures in their ibl learning processes (e.g. wrong results of experiments; unexpected difficulties) among many primary school students (magee and flessner 2012). despite those suggested limitations and obstacles, however, there are a good number of pedagogical benefits of successful ibl, including the development of in-depth understandings, autonomous learning abilities and critical thinking skills (çakıroğlu and öztürk 2017; mazur, brown, and jacobsen 2015), promoted if in-class time is saved through an fc approach. inquiry-based flipped classroom model this article suggests that such a positive pedagogical transformation, from traditional lecture-based instruction to student-centred ibl, can be made possible in primary education settings by utilising the fc model in the ibl design and implementation, that is, more practically speaking, by freeing up classroom time for more guided ibl activities (çakıroğlu and öztürk 2017; chen and chang 2017; huang and lin 2017; love et al. 2015). such an attempt to integrate the fc model and ibl has been lacking in the primary education contexts despite the potential to address the perceived limitations and challenges reported by the previous studies as discussed above. this study, therefore, aims to address those pedagogical concerns that arise from both fcand ibl-oriented primary education contexts by exploring the pedagogical possibilities for utilising the fc model to enhance the quality of ibl design. that is, this research addresses the identified gaps in the current literature: (1) a lack of understandings about teachers’, students’ and parents’ lived experiences and perceptions of the implementation of the fc model in k-12 education and (2) an absent of a systematic pedagogical and research attempt to combine the two models of ibl and fc which may effectively assist teachers in developing effective instruction in k-12 education and further enhance students’ learning experiences (giannakos, krogstie, and chrisochoides 2014; rahman et al. 2014). thus, the authors first developed a theory-informed instructional model, inquiry-based flipped classroom (ib-fc) model (figure 1), based on their literature review results and a series of small-scale pilot studies (loizou-raouna and lee 2018a) in the first author’s own teaching contexts. this model has been then used, in the present multi-case study, to design and develop a number of ib-fc cases, situated across five http://dx.doi.org/10.25304/rlt.v28.2287 m. loizou and k. lee 4 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) primary schools in cyprus. the model distinguishes between the individual and group learning space, assisted through the use of multimedia and screencasting technologies. teachers’ lectures in the form of the flips given (e.g. video-tutorials, readings, and so on) and learner’s inquiries which arise through the entrance ticket completion serve as the link between learning at home and in the classroom. at the group learning space, ibl activities are grouped into four sectors: (1) how to begin/facilitate the lesson; (2) what students do during the lessons; (3) what do students do after the lessons and (4) what students as a group do during the lessons. this multi-case study, drawn from authors’ collaborative action research project with five primary school teachers in cyprus, focuses on supporting primary school teachers’ attempt to implement the ib-fc model in their classroom contexts by developing and providing universal design principles that can be used to guide the teachers’ practice. this investigation has been guided by the following research questions: rq1: what are the experiences of teachers, students and parents in different ib-fcs in cyprus primary school context? rq2: what are the overall perceptions of teachers, students and parents on implementing the ib-fc model in different subject matters in cyprus primary school context (i.e. benefits, challenges and limitations of the model implementation)? rq3: what are the universal design principles for effective implementation of the ib-fc model in cyprus primary school context across different subject matters? research methodology and implementation the research was carried out in five different classes, in five different public primary schools in cyprus, that is, working with students aged between 8 and 11 years old. figure 2 illustrates the basic information about the five different classes (cs1–cs5), including the drop-out case. figure 1. ib-fc model. http://dx.doi.org/10.25304/rlt.v28.2287 research in learning technology citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 5 (page number not for citation purpose) ‘six sources of evidence’ (yin 2009, p. 101) for qualitative studies were used for data collection. the collected dataset includes a mixture of primary data and secondary data: (1) pedagogical documents: ib-fc design reports and lesson plans; (2) archival records: for example, educational policy (moec 2019); (3) teacher interviews and student focus groups; (4) parents’ survey; (5) classroom observations and (6) researchers’ fieldnotes including reflections and insights. nvivo, as a qualitative research software, assisted the researchers in managing all the different data collected, walking through the themes, nodes, patterns, flowcharts and key issues arising, presented in the results section. after all stages of analysis, the themes regarding teacher, student and parent experiences (research question 1) were grouped as pre-class, in-class and after-class, whereas teacher, student and parent perceptions (research question 2) were grouped into benefits and challenges with limitations. universal principles (research question 3) which arise from the theme analysis are discussed right after. according to the research framework, different ib-fc tools were developed which aimed to guide the teachers towards the design of the ib-fc lessons, uploaded on the moodle platform developed for this research (which can be found here: http://www.protyposxoleio.com). these include suggestions for the various stages of implementation, as well as suggestions on the orchestration routines and digital tools which could be used. a simple lesson template was developed (figure 3) with all the steps of the ib-fc design (introductory in-class, pre-class, in-class, after-class), important notes for the teachers and the research framework. the aim of the template was to assist the teachers in their initial development of the ib-fc lessons, making sure they follow the correct structure which they could gradually evolve themselves. further examples of flips were also given to teachers for preparing their own. ethical approval for undertaking this research was granted initially from lancaster university, allowing for the high-risk research because the students who took part were aged 8–12 years. given that the research was carried out in public primary schools in cyprus, ethical approval was also granted from cyprus ministry of education and culture and the cyprus centre of educational research and evaluation. figure 2. participants’ profile (school, student and teacher). http://dx.doi.org/10.25304/rlt.v28.2287 http://www.protyposxoleio.com http://www.protyposxoleio.com m. loizou and k. lee 6 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) research results rosemary, tesa, ben, james and mary are the teacher pseudonyms which will be used in the presentation of results, whereas s1.#–s5.# (s1.#: students in case study 1, etc.) and p# symbols will refer to student and parent participants, respectively. overview normal practice in cs1–cs4 settings had been traditional, with minor use of technology; hence, ib-fc implementation was at a lower rate, implemented mainly in the main subjects of greek language and mathematics. in cs5, a ‘bring your own figure 3. initial ib-fc lesson template. http://dx.doi.org/10.25304/rlt.v28.2287 research in learning technology citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 7 (page number not for citation purpose) device’ initiative was already followed for 2 years and therefore it had been easier for the teacher and the students to switch to an ib-fc model and exploit its potentials and limitations in all school subjects. implementation was also easier and more common in the upper primary classes, cs3–cs5. teacher, student and parent experiences pre-class experiences flips teachers recognised that it is of great significance to have a flip which would substitute ‘what i would otherwise lecture the students about anyway’ (mary), and that ‘the flips should have a “value-added” to the learning process’ (ben). all student participants were excited with the video-tutorials as they all recalled that the best time they had was when they watched the videos at home, especially the ones with completely unknown content, for example, ‘i liked videos that were showing things i didn’t know before’ (s1.5). teachers preferred to use ready-made videos as flips embedded on moodle for various reasons. one of the teachers, ben, specifically said that it is very easy for him to find ready-made videos and spend less time preparing for the lesson. parents also found it ‘very interesting’, with only a few (n = 9 out of 48) pointing out the fact that their children are sometimes busy enough with other homework so they felt tired and pressured to watch the flips. most teachers agreed that there were times that they have chosen to give out flips in other formats, other than video (e.g. presentations, online content). entrance ticket the term ‘entrance ticket’ refers to the activities the students had to complete after watching/studying the flip/flips given at home. these activities are assigned together with the flips to make sure the students actually watch/study them and are well prepared. teachers admitted that the flips gave them a chance to assign activities as homework/entrance ticket which they would otherwise not give out ‘…because for being able to do it they must have had access to what i’m telling them in class which is what the flip does now for me…’ (james). concerning students, most of them in tesa’s and ben’s classroom mentioned that it was a bit demanding for them to complete the entrance ticket. ‘we were writing answers and a lot of things’ (s1.5), especially when they should answer questions (s4.20, s4.23), or create something on their own, without the presence of the teacher. teachers claimed that their choice in preparing particular entrance tickets always had to do with the age of the students, the lesson goals and the available moodle activities/tools (e.g. chats, wikis, quizzes, forums). time the length of the video or flips in other formats was also critical. teachers acknowledged the value of short videos/flips, for example: ‘i was trying to find short ones [videos], 5–10 minutes, but even the longer ones when i couldn’t edit them and make them shorter, i suggested to the students to watch let’s say from the 5th till the 12th minute’. (ben) http://dx.doi.org/10.25304/rlt.v28.2287 m. loizou and k. lee 8 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) teachers also preferred to use short activities as entrance tickets, ranging from multiple-choice questions created with the equivalent moodle tool to using a moodle forum for posting short answers, ‘…taking no more than 20 minutes’ (tesa). students also never liked complicated assignments, neither the parents some of whom claimed that: ‘we do not have all the time this requires’ (p15). supports parents: teachers would guide the parents in how to help their children at home through parent–teacher meetings at school. it was also done periodically in person later during the model’s implementation, responding to particular parents’ requests. what actually happened at home was that some of the students, as they reported, asked their parents (both or one of them) for help (n = 12 out of 77), for example, ‘my  mum knows what i am doing, my dad doesn’t know’ (s2.14). the fact that some parents cannot help their children due to their incompetence in the use of information and communication technologies (ict) was also evident in the students’ interviews. however, most of the parents’ responses and experiences (n = 35 out of 48) have been positive indeed, despite the challenges. parents (n = 39 out of 48) generally spoke of how they liked the fact that they have participated in the research, especially at the later stages of the model’s implementation. teachers: almost all students (n = 71 out of 77) said that they got some kind of support they might have needed from their teacher while completing their entrance ticket. james’, ben’s and mary’s students even had the opportunity to personally contact them during afternoon hours, either through moodle, viber, sms or through a phone call. teachers would check on the entrance tickets before the in-class session and provide feedback, either during the same afternoon, giving time for correction of errors, or during the night before coming to the class where errors would be corrected by students the day after. in-class experiences less engaged and management when students do not watch the videos at home and do not complete their entrance ticket, this is seen as a ‘crisis’ situation in the fc model because the teacher cannot proceed to the in-class activities. indeed, students’ lack of responsibility and ‘feeling lazy’ (ben) was the main implementation problem teachers had to deal with. teachers talked of how impossible it was to move on to the ibl activities in class whenever they would realise that only a few students have watched the flips and completed the entrance tickets (or completed it in a haste), occurring mostly in rosemary’s and ben’s classrooms. how would teachers deal with the less-engaged? the teachers would sometimes play the video again in class for everyone to watch it again; at other times they would ask the students to watch it on their own in class (using their headphones). rosemary also suggested the creation of an in-flip station in class, whereas teachers also agreed that if the students use the same device, both at school and at home, they are more likely to work more efficiently during pre-class. http://dx.doi.org/10.25304/rlt.v28.2287 research in learning technology citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 9 (page number not for citation purpose) ibl activities the activities in the class were inquiry-based ‘engaging and very interesting to the students’ (mary) just like what the ib-fc model encompasses. these activities were the ‘extension’ of the flip/flips the students had to watch at home. ‘going beyond the book’ (james) had been the purpose of every ibl activity as teachers noted and those should have been fun, interesting and engaging to the students. figures 4–6 show the kind of ibl activities completed during class time in the greek language lesson, mathematics and in all other social studies and sciences, as these have been drawn from the lesson plan analysis (note: cs = case study). teachers’ experiences in creating ibl activities mostly focused in how moodle tools have been used because ‘they (moodle forums) had been very easy to use’ (rosemary), especially in the greek language lesson, given that some activities were not clearly ibl, for example, drill and practice in maths. teachers specifically mentioned figure 4. types of ibl activities used in the greek lesson (cs1–cs5). 0 2 4 6 8 10 12 14 16 18 online games book ac�vi�es online tutorials online quiz moodle forum discussion/problem wri�ng online research interac�ve/represena�on so�ware game crea�on drill and prac�ce cs1 cs2 cs3 cs4 cs5 figure 5. types of ibl activities used in the math lesson (cs1–cs5). http://dx.doi.org/10.25304/rlt.v28.2287 m. loizou and k. lee 10 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) how forums were useful for asking their students to answer questions, post comments on flips, give feedback to their peers or post the outcomes of individual or group research any time during the learning process. students witnessed using moodle chats ‘…for discussing a topic at home or in class’ (s5.6) usually in groups so as to avoid too many lines. however, the most effective activities for the upper primary students (cs3–cs5), as their teachers claimed regarding their own experience, had been the online research following a flip. most students preferred working in groups because they could combine their personal input from the flips they have watched at home and complete in-class activities more easily. after-class (ib-fcs evaluation) most of the students reported that it is easier for them now (i.e. through the ib-fc model) to complete their after-class work/extra activities because they have the preclass videos (flips) available, for example, ‘because our teacher will not upload or ask something we don’t know’ (s5.1). portfolio development had been the after-class challenge for the students in mary’s classroom, using mahara platform, because none of them liked the process because it was very demanding, for example, ‘i didn’t like the portfolios. it was a lot of work for us’ (s5.10). nevertheless, the important thing is that most of them enjoyed its development (n = 15 out of 17 cs5). computer or online assessment methods, either through moodle or through other online tools would work for the students. observation notes strongly emphasise how students got more excited with online assessment versus a hard copy, achieving high scores, for example, ‘we liked it better. and let’s say it was more pleasant/joyful because we were feeling different from the other classrooms’ (s4.19). teachers, student and parent perceptions benefits the flips and work at home overall, students preferred this new way of learning over their traditional class as they value the videos more than the books (n = 72 out 77), for example, ‘it was better 0 0.5 1 1.5 2 2.5 3 3.5 online turorials game crea on revision ac vi es argumenta ve wri ng interac ve presenta on online research mindmapping note taking cs1 cs2 cs3 cs4 cs5 figure 6. types of ibl activities used in the social studies lessons and science (cs1–cs5). http://dx.doi.org/10.25304/rlt.v28.2287 research in learning technology citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 11 (page number not for citation purpose) than having a book. the video is better’. (s4.18); students also mentioned that they could watch them as many times at it was necessary to be able to complete their work (entrance ticket). some of the teachers stressed the fact that students could create their own digital work (mary’s and ben’s classroom) or e-portfolios (mary’s classroom) with the information they would gather, related to the flip. student interest, engagement and participation in class james’ and mary’s students (n = 25) reported how easy everything was as they had teachers’ support in class. teachers also thought of this new methodology as an approach to attract students’ attention and elevate motivation and participation because they are asked to function just like in their everyday life: use of visual aspects and access to electronic devices. many students (n = 28) even said that they would like to work in this way in all subjects. parents also agreed that students’ engagement was highly encouraged. among the benefits parents wrote was the chance of developing students’ critical thinking through interesting and innovative activities. personalised learning and differentiation ‘as teachers, we have long strived to maintain what is called “personalised learning” and “differentiation”. it’s a huge challenge, somehow i feel made more possible with this methodology’ (mary). savings in class time also offered teachers the opportunity to work more closely with the students who needed further guidance and assistance with the new and more complicated concepts introduced. challenges and limitations age and skills of students the young age of the students and their limited ict skills posed challenges and set limitations during implementation, especially at the early stages. overall planned ibl activities did not always work as expected, especially with the lower primary students (tesa’s and rosemary’s classrooms), either because they would not all watch the flips or understand them well or because they both lacked ict and ibl skills required. shared class accounts were created to overcome some of these challenges. unsuccessful activities long flips and pre-class activities (entrance tickets) were disliked by all students who would prefer shorter and less-complicated work. the same applied for parents who sometimes neither had the skills nor had time to offer guidance to their children. this sometimes led to unfinished work, misconceptions and students’ disinterest. technical challenges although most of the students confirmed that they enjoyed their work at home and denied having device or connectivity problems, internet safety had been an issue for a few parents (n = 9 out of 48). teachers also agreed that there was no particular device http://dx.doi.org/10.25304/rlt.v28.2287 m. loizou and k. lee 12 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) problem in class. however, students who brought in their laptops (ben and mary) worked better than the ones working on tablets because they avoided compatibility problems with some of the software. this is one of the reasons that mary used a ‘bring your own device’ programme. however, periodic problems with internet connectivity in the class had been the main challenge reported. parents’ concerns many parents (n = 25 out of 48) underlined that the weakest part of the programme was their children spending too many hours on the computer/device, increasing their total screen-time. they associated this with the possible development of eye problems because they believe that ‘…the students are trapped with the computer’ (p10). discussion: universal design principles the study and analysis of the teachers’, students’ and parents’ experiences and perceptions have given rise to particular practical suggestions for teachers, discussed here. these were further coded and defined as ib-fc universal design principles, while some of them were validated through past theories and research on fc design principles in higher education (brame 2013; chen and chang 2017; kim 2016; kim et  al. 2014). below the seven universal design principles (udp1–udp7) this study proposes are presented. udp1: structure and flexibility structure during the preparation of the ib-fc lesson designs, educators should initially follow particular design steps, such as the ones given in the ib-fc lesson template in table 1, which would ensure a clearly defined structure both for them and their young students. the use of virtual learning environments (vle) tools indeed assisted towards well-structured ib-fc lessons, whereas teacher support through a variety of synchronous and asynchronous tools is important. moreover, the structure can be achieved when orchestration routines are adopted. this means that educators should provide all the necessary instructions for the students to be able to find their way out on their own and achieve higher orders of cognitive work, as lankford 2013, and zainuddin and halili (2016) have also proposed. the main examples arising from rq1 and rq2 analysis are as follows: (1) creation of common classroom accounts; (2) monitoring of activity timings; (3) provision of devices for all; (4) sequencing of activities and provision of ‘extension’ activities; (5) organisation of in-flip stations; (6) structuring through a vle and (7) classroom management, for example, rewarding/marking system. flexibility the model should be used selectively during design and flexibly during implementation. parents noted that there should be a combination with traditional teaching (e.g. p28). if the model is universally implemented, it would also translate into a huge http://dx.doi.org/10.25304/rlt.v28.2287 research in learning technology citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 13 (page number not for citation purpose) workload for the teacher in preparing all the flips (lage, platt, and treglia 2000) because primary school teachers teach almost all subjects to their classroom students and do not specialise in one (as opposed to secondary and higher education settings). udp2: simplicity and accessibility simple and universally accessible technologies should be used (internet connection, devices and software). students should be able to use them and the teachers should feel confident to solve technical issues. the first step towards deciding on the adoption of an ib-fc model implementation altogether, or on the lesson designs, is to have good internet connectivity, both in class and at home. indeed, access to networked and school technology by students is one of the challenges of the fc approach (ullman 2013). during the research, teachers preferred using mobile devices in class and not in the computer lab because ‘that provided flexibility’ (james). maintaining a 1:1 student–device ratio in class was easy but not always necessary. a 1:2 ratio works best with the young students, whereas the adoption of a ‘bring-your-own-device’ initiative seemed to solve many device problems for ben and mary. udp3: interconnectivity and community interconnectivity ibl activities which clearly connect the individual learning space with the group learning space should be chosen. students should be able to make the connection (hamdan et  al. 2013). depending on the content and length of the video, students should watch it more than once and as many times it is needed. the flips should also contain the right amount of new knowledge for the students and have a value-added, similarly emphasised by kong and song (2015). overall, students in the research proposed a simple note-taking strategy to deal with the new information. community ib-fc implementation in primary education needs the building of a young learners’ community, developed through multiple means of communication between all, with clearly defined and well-structured guidance and support. parents and students highly emphasised that they should also be able to personally contact the teacher in a synchronous or asynchronous way (communication tools) at any time during the learning process, especially during their work at home. teachers should circulate around the classroom and support students, providing facilitation for building a learning community. this clearly refers to kim’s et al. (2014) social presence principle. udp4: differentiation and personalisation an opportunity should be given for all students to gain basic understanding and exposure to content prior to class. the very nature of the fc approach provides flips at the individual learning space so that students can watch them as many times they need to understand the content, promoting differentiated instruction and personalised http://dx.doi.org/10.25304/rlt.v28.2287 m. loizou and k. lee 14 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) high-order in-class learning (yarbro et al. 2014). students have recognised that they do not all share the same skills, abilities or prior knowledge, for example, ‘…we are all different’ (s4.1). udp5: development and progression activities which gradually promote ibl and transversal skills should be offered, that is, teachers should spend time to help lower primary students gradually attain ibl skills. ‘learning how to learn’ is a difficult task for young students to master (flick  and lederman 2004). as many planned ibl activities did not work best in tesa’s and rosemary’s classrooms, it is important that teachers spend time during the first months of implementation into building ibl skills, assigning easier and simpler tasks and guiding the students through it (kim and chin 2011). for the upper primary students, it had been easier to promote ibl skills and teachers have really worked with their students on different ibl activities. udp6: motivation and engagement one of the key principles of ib-fc implementation is how teachers offer motivation and an incentive to students to prepare for class so as to avoid disengagement and in-class revision problems (cole, martin, and dennis 2004). this had been again one  of the four design principles suggested by brame (2013). participant teachers dealt with disengaged students by creating in-flip stations, playing the videotutorial in class and explaining content, either on a personal or on a whole-class level. this should help develop both intrinsic and extrinsic motivation (abeysekeraa and dawson 2015). udp7: assessment and evaluation a mechanism to assess student understanding and address misconceptions at every stage of the learning process should be created (prompt and adaptive feedback). educators need to make sure that they give opportunities/time for review and correction of errors after peer/teacher/self-review. students understand what is needed to be done because the instructor evaluates and gives feedback throughout the learning process (pre-class, in-class, after-class), overcoming deficiencies in learning (kim et al. 2014). conclusion this research aimed to study actual ib-fc implementation in primary school settings. it was found that teachers, students and parents had mostly positive experiences and perceptions. overall students were satisfied with the fc activities, all enjoyed the flips and many agreed that the class time interaction through the ibl activities was key to their understanding. students and teachers perceived that the fc activities were more interesting, motivating and engaging, especially with the use of technology, than a traditional class. teachers have enjoyed the design and implementation process and they were willing to offer all the support and guidance the students needed throughout the learning process. teachers collaborated well with the parents who appreciated  the effort and were also willing, happy and supportive  towards their children. http://dx.doi.org/10.25304/rlt.v28.2287 research in learning technology citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 15 (page number not for citation purpose) these findings are in line with many other previous studies, as this has been emphasised in the discussion of the universal design principles arising from the research. contrary though to many previously published studies, the present study is distinctive in the following two ways. firstly, it tested the feasibility of implementing a fc in a primary school context in cyprus, through an action research methodology within multiple case studies. a majority of previous studies did not explicate any specific conceptual framework to help instructors design their fc s at any level of education (bishop and verleger 2013; giannakos, krogstie, and chrisochoides 2014), not even the few studies which exist in primary education (aidinopoulou and sampson 2017; gough et al. 2017; hultén and larsson 2016; kostaris et al. 2017). moreover, it is the first time the fc model has been implemented at any level within the educational context of cyprus (loizou-raouna and lee 2018b). the current research proposed and tested the implementation of a combination of fc with ibl to address the limitations and challenges of both methodologies and help educators, students and parents enjoy the benefits of both. all in all, the present study proposed universal design principles based on the practical implementation suggestions arising from students’, teacher’s and parents’ experiences and perceptions as well as relevant literature. further research could concentrate on investigating the implementation of a new ib-fc model based on these principles, in k-12 education, testing their effectiveness and further benefits and challenges arising. references abeysekera, l. & dawson, p. (2015) ‘motivation and cognitive load in the flipped classroom: definition, rationale and a call for research’, higher education research & development, vol. 34, no. 1, pp. 1–14. doi: 10.1023/a:1002997414652 aidinopoulou, v. & sampson, d. g. (2017) ‘an action research study from implementing the flipped classroom model in primary school history teaching and learning’, educational technology & society, vol. 20, no. 1, pp. 237–247. doi: 10.1080/07294360.2014.934336 bergmann, j., overmyer, j. & wilie, b. (2012) ‘the flipped class: myths versus reality’, the daily riff. available at: http://www.thedailyriff.com/articles/the-flipped-class-conversation-689.php bergmann, j. & sams, a. (2012) flip your classroom: reach every student in every class every day. eugene, or: international society for technology in education. doi:10.1177/073989131401100120 bishop, j. l. & verleger, m. a. (2013) ‘the flipped classroom: a survey of the research’, in proceedings of the 120th asee national conference, 30, asee, atlanta, ga, pp. 1–18. brame, c. j. (2013) flipping the classroom. the vanderbilt university center for teaching. available at: http://cft.vanderbilt.edu/teaching-guides/teaching-activities/ flipping-the-classroom/ çakıroğlu, ü. & öztürk, m.(2017) ‘flipped classroom with problem based activities: exploring self-regulated learning in a programming language course’, educational technology & society, vol. 20, no. 1, pp. 337–349. doi:10.1016/j. compedu.2015.06.003 capaldi, m. (2015) ‘including inquiry-based learning in a flipped class’, primus, vol. 25, no. 8, pp. 736–744. doi: 10.1080/10511970.2015.1031303 chen, h. l. & chang, c. y. (2017) ‘integrating the sop2 model into the flipped classroom to foster cognitive presence and learning achievements’, educational technology & society, vol. 20, no. 1, pp. 274–291. cole, p., martiin, s. & dennis, t. (2004) ‘emotion regulation as a scientific construct: methodological challenges and directions for child development research’, child development, vol. 75, pp. 317–333. doi: 0009-3920/2004/ 7502–0002 http://dx.doi.org/10.25304/rlt.v28.2287 https://dx.doi.org/10.1023/a:1002997414652 https://dx.doi.org/10.1080/07294360.2014.934336 http://www.thedailyriff.com/articles/the-flipped-class-conversation-689.php http://www.thedailyriff.com/articles/the-flipped-class-conversation-689.php https://dx.doi.org/10.1177/073989131401100120 http://cft.vanderbilt.edu/teaching-guides/teaching-activities/flipping-the-classroom/ http://cft.vanderbilt.edu/teaching-guides/teaching-activities/flipping-the-classroom/ https://dx.doi.org/10.1016/j. compedu.2015.06.003 https://dx.doi.org/10.1080/10511970.2015.1031303 https://dx.doi.org/0009-3920/2004/ 7502–0002 m. loizou and k. lee 16 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) davies, r. s., dean, d. l. & ball, n. (2013) ‘flipping the classroom and instructional technology integration in a college-level information systems spreadsheet course’, educational technology research and development, vol. 61, no. 4, pp. 563–580. doi:  10.1007/ s11423-013-9305-6 flick, l. b. & lederman, n. g. (2004) scientific inquiry and nature of science: implications teaching, learning, and teacher education. dordrecht, the netherlands: kluwer academic publishers. flipped learning network. (2014) what is flipped learning: the four pillars of f–l–i–p™. available at: http://flippedlearning.org/cms/lib07/va01923112/centricity/domain/46/ flip_handout_fnl_web.pdf friesen, s. (2013) inquiry based learning. in encyclopedia of terminology for educational and communications and technology, ed r.c. richey, springer, new york, ny, pp. 153–155. fulton, k. (2012) ‘upside down and inside out: flip your classroom to improve student learning’, learning & leading with technology, vol. 39, no. 8, pp. 12–17. giannakos, m. n., krogstie, j. & chrisochoides, n. (2014) ‘reviewing the flipped classroom research: reflections for computer science education’, in proceedings of the computer science education research conference, acm, new york, ny, pp. 23–29. doi: 10.1145/2691352.2691354 gough, e., et  al., (2017) ‘k-12 teacher perceptions regarding the flipped classroom model for  teaching and learning’, journal of educational technology systems, vol. 45, no. 3, pp. 390–423. doi: 10.1177/0047239516658444 hamdan, n., et  al., (2013) research, reports & studies / lit review. flippedlearning.org. available at: http://www.flippedlearning.org/review hannafin, m. j. & land, s. (1997) ‘the foundations and assumptions of technology-enhanced, student-centered learning environments’, instructional science, vol. 25, pp. 167–202. doi: 10.1023/a:1002997414652 herreid, c. f. & schiller, n. a. (2013) ‘case studies and the flipped classroom’, journal of science teaching, vol. 42, no. 5, pp. 62–67. huang, c. k. & lin, c. y. (2017) ‘flipping business education: transformative use of teambased learning in human resource management classrooms’, educational technology & society, vol. 20, no. 1, pp. 323–336. hughes, h. (2012) ‘introduction to flipping the college classroom’, in proceedings from world conference on educational multimedia, hypermedia and telecommunications, 2012, eds t. amiel & b. wilson, aace, chesapeake, pp. 2434–2438. hultén, m. & larsson, b. (2016) ‘the flipped classroom: primary and secondary teachers’ views on an educational movement in schools in sweden today’, scandinavian journal of educational research, vol. 2016, pp. 1–11. doi: 10.1080/00313831.2016.1258662 kong, s. c. & song, y. j. (2015) ‘an experience of personalized learning hub initiative embedding byod for reflective engagement in higher education’, computers & education, vol. 88, pp. 227–240. doi:10.1016/j. compedu.2015.06.003 kostaris, c., et al., (2017) ‘investigating the potential of the flipped classroom model in k-12 ict teaching and learning: an action research study’, journal of educational technology & society, vol. 20, no. 1, pp. 261–273. kim, d. (2016) ‘flipped interpreting classroom: flipping approaches, student perceptions and design considerations’, the interpreter and translator trainer, vol. 11, pp. 38–55. doi: 10.1080/1750399x. 2016.1198180 kim, d. (2017) the interpreter and translator trainer flipped interpreting classroom : flipping approaches, student perceptions and design considerations perceptions and design considerations, the interpreter and translator trainer, routledge, pp. 1–18. doi: 10.1080/ 1750399x.2016.1198180 kim, m. & chin, c. (2011) ‘pre-service teachers’ views on practical work with inquiry orientation in textbookoriented science classrooms’, international journal of environmental & science education, vol. 6, no. 1, pp. 23–37. http://dx.doi.org/10.25304/rlt.v28.2287 https://dx.doi.org/10.1007/s11423-013-9305-6 https://dx.doi.org/10.1007/s11423-013-9305-6 http://flippedlearning.org/cms/lib07/va01923112/centricity/domain/46/flip_handout_fnl_web.pdf http://flippedlearning.org/cms/lib07/va01923112/centricity/domain/46/flip_handout_fnl_web.pdf https://dx.doi.org/10.1145/2691352.2691354 https://dx.doi.org/10.1145/2691352.2691354 https://dx.doi.org/10.1177/0047239516658444 http://flippedlearning.org http://www.flippedlearning.org/review https://dx.doi.org/10.1023/a:1002997414652 https://dx.doi.org/10.1023/a:1002997414652 https://dx.doi.org/10.1080/00313831.2016.1258662 https://dx.doi.org/10.1016/j. compedu.2015.06.003 https://dx.doi.org/10.1080/1750399x. 2016.1198180 https://dx.doi.org/10.1080/1750399x.2016.1198180 https://dx.doi.org/10.1080/1750399x.2016.1198180 research in learning technology citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 17 (page number not for citation purpose) kim, m. k., et al., (2014) ‘the experience of three flipped classrooms in an urban university: an exploration of design principles’, the internet and higher education, vol. 22, pp. 37–50. doi: 10.1016/j.iheduc.2014.04.003 lage, m. j., platt, g. j. & treglia, m. (2000) ‘inverting the classroom: a gateway to creating an inclusive learning environment’, the journal of economic education, vol. 31, no. 1, pp. 30–43. doi: 10.2307/1183338 lankford, l. (2013) isn’t the flipped classroom just blended learning? [blog post] available at: https://ileighanne.wordpress.com/2013/01/24/isnt-the-flipped-classroom-justblended learning/ laursen, s. & kogan, m. (2014) ‘assessing long-term effects of inquirybased learning: a case study from college mathematics’, innovative higher education, vol. 39, no. 3, pp. 183–199. doi: 10.1007/s10755-013-9269-9 loizou-raouna, m. & lee, k. (2018a) ‘a flipped classroom model for inquiry-based learning in cyprus primary education context – a multiple case study’, in 12th international technology, education and development conference, inted2018, valencia, spain, 5–7 march 2018, pp. 3189–3196. doi: 10.21125/inted.2018.0610 loizou-raouna, m. & lee, k. (2018b) ‘a flipped classroom model for inquiry-based learning in cyprus primary education context’, in proceedings of the 11th international conference on networked learning 2018, eds m. bajić, n. b. dohn, m. de laat, p. jandrić, & t. ryberg, zagreb, croatia, 14–16 may 2018, pp. 210–217. love, b., et  al., (2015) ‘inquiry-based learning and the flipped classroom model’, primus, vol. 25, no. 8, pp. 745–762. doi: 10.1080/10511970.2015.1046005 magee, p. a. & flessner. r. (2012) ‘collaborating to improve inquiry-based teaching in elementary science and mathematics methods courses’, science education international, vol.  23, no. 4, pp. 353–365. available at: http://www.icaseonline.net/sei/december2012/ p4.pdf mazur, a. d., brown, b. & jacobsen, m. (2015) ‘learning designs using flipped classroom instruction’, canadian journal of learning and technology, vol. 41, no. 2, pp. 1–16. moec. (2019) cyprus national curriculum. available at: http://www.moec.gov.cy/analytika_ programmata/ rahman, a. a., et  al., (2014) ‘the influences of flipped classroom: a meta-analysis’, in proceedings of the 6th ieee conference on engineering education, ieee, kuala lumpur, malaysia, pp. 24–28. rutherfoord, r. h. & rutherfoord, j. k. (2000) ‘flipping the classroom – is it for you?’, sigite, vol. 13, pp. 19–22. doi: 10.1145/2512276.2512299 shea, p., et  al., (2012) ‘learning presence: additional research on a new conceptual element within the community of inquiry (coi) framework’, the internet and higher education, vol. 15, no. 2, pp. 89–95. short, k. g. & harste, j. c. (1996) creating classrooms for authors and inquirers. portsmouth, nh: heinemann. stephenson, n. (2012) introduction to inquiry based learning. available at: teach. inquiry: http://www.teachinquiry.com/index/introduction.html strayer, j. f. (2012) ‘how learning in an inverted classroom influences cooperation, innovation and task orientation’, learning environments research, vol. 15, no. 2, pp. 171–193. doi: 10.1007/s10984-012-9108-4 talbert, r. (2012) ‘inverted classroom’, colleagues, vol. 9, no. 1, article 7. available at: https:// scholarworks.gvsu.edu/colleagues/vol9/iss1/7 tenneson, m. & mcglasson, b. (2006, april) the classroom flip. presentation at fontbonne university, missouri teaching and learning mentor program. available at: www.fontbonne. edu/upload/theclassroomflip.ppt teo, t. w., et al., (2014) ‘how flip teaching supports undergraduate chemistry laboratory learning’, chemistry education research and practice, vol. 15, no. 4, pp. 550–567. available at: https://pubs.rsc.org/en/content/getauthorversionpdf/c4rp00003j http://dx.doi.org/10.25304/rlt.v28.2287 https://dx.doi.org/10.1016/j.iheduc.2014.04.003 https://dx.doi.org/doi: 10.2307/1183338 https://ileighanne.wordpress.com/2013/01/24/isnt-the-flipped-classroom-just-blended-learning/ https://ileighanne.wordpress.com/2013/01/24/isnt-the-flipped-classroom-just-blended-learning/ https://dx.doi.org/10.1007/s10755-013-9269-9 https://dx.doi.org/10.21125/inted.2018.0610 https://dx.doi.org/10.1080/10511970.2015.1046005 http://www.icaseonline.net/sei/december2012/p4.pdf http://www.icaseonline.net/sei/december2012/p4.pdf http://www.moec.gov.cy/analytika_programmata/ http://www.moec.gov.cy/analytika_programmata/ http://dx.doi.org/10.1145/2512276.2512299 http://www.teachinquiry.com/index/introduction.html https://dx.doi.org/10.1007/s10984-012-9108-4 https://dx.doi.org/10.1007/s10984-012-9108-4 https://scholarworks.gvsu.edu/colleagues/vol9/iss1/7 https://scholarworks.gvsu.edu/colleagues/vol9/iss1/7 http://www.fontbonne.edu/upload/theclassroomflip.ppt http://www.fontbonne.edu/upload/theclassroomflip.ppt https://pubs.rsc.org/en/content/getauthorversionpdf/c4rp00003j m. loizou and k. lee 18 citation: research in learning technology 2020, 28: 2287 http://dx.doi.org/10.25304/rlt.v28.2287 (page number not for citation purpose) ullman, e. (2013) ‘tips to help flip your classroom: teachers offer their strategies for making the most out of the flipped classroom model’, ascd education update, vol. 55, no. 2, pp. 1–5. yarbro, j., et  al., (2014) extension of a review of flipped learning. available at: http:// flippedlearning.org/cms/lib07/va01923112/centricity/domain/41/extension%20of%20 flipped%20learning%20lit%20review%20june%202014.pdf yin, r. k. (2009) case study research: design and methods, 4th edn, sage, thousand oaks, ca. doi: 10.33524/cjar.v14i1.73 zainuddin, z. & halili, s. h. (2016) ‘flipped classroom research and trends from different fields of study’, international review of research in open and distributed learning, vol. 17, no. 3, pp. 313–340. doi: 10.19173/irrodl.v17i3.2274 zappe, s., et al., (2009) flipping the classroom to explore active learning in a large undergraduate course. washington, dc: american society for engineering education. available at: http://search.asee.org/search/fetch?url=file%3a%2f%2flocalhost%2fe%3a%2fsearch%2fconference%2f19%2fac%25202009full92.pdf&index=conference_papers&sp ace=129746797203605791716676178&type=application%2fpdf&charset= http://dx.doi.org/10.25304/rlt.v28.2287 http://flippedlearning.org/cms/lib07/va01923112/centricity/domain/41/extension%20of%20flipped%20learning%20lit%20review%20june%202014.pdf http://flippedlearning.org/cms/lib07/va01923112/centricity/domain/41/extension%20of%20flipped%20learning%20lit%20review%20june%202014.pdf http://flippedlearning.org/cms/lib07/va01923112/centricity/domain/41/extension%20of%20flipped%20learning%20lit%20review%20june%202014.pdf https://dx.doi.org/10.33524/cjar.v14i1.73 https://dx.doi.org/10.19173/irrodl.v17i3.2274 http://search.asee.org/search/fetch?url=file%3a%2f%2flocalhost%2fe%3a%2fsearch%2fconference%2f19%2fac%25202009full92.pdf&index=conference_papers&space=129746797203605791716676178&type=application%2fpdf&charset= http://search.asee.org/search/fetch?url=file%3a%2f%2flocalhost%2fe%3a%2fsearch%2fconference%2f19%2fac%25202009full92.pdf&index=conference_papers&space=129746797203605791716676178&type=application%2fpdf&charset= http://search.asee.org/search/fetch?url=file%3a%2f%2flocalhost%2fe%3a%2fsearch%2fconference%2f19%2fac%25202009full92.pdf&index=conference_papers&space=129746797203605791716676178&type=application%2fpdf&charset= (page number not for citation purpose) 1 *corresponding author. email: horvath.laszlo@ppk.elte.hu research in learning technology 2023. © 2023 l. horváth. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 original research article adaptive pedagogical strategies responding to emergency remote teaching – immediate responses of hungarian primary school teachers lászló horváth* institute of education, elte eötvös loránd university, budapest, hungary (received: 15 december 2022; revised: 8 march 2023; accepted: 15 march 2023; published: 11 april 2023) digital disruption is not a new phenomenon in education; however, it has become more prominent during the covid-19 pandemic due to school closures and the related emergency remote teaching (ert) period. our study aims to explore the different pedagogical strategies that primary school teachers adopted during this period and determine how successful these strategies were in involving and engaging students. altogether, 4028 teachers from 343 primary schools answered our online survey from all the regions of hungary. the sample adequately represents the hungarian primary school teacher population in terms of gender and age. we used cluster analysis and identified four clusters of pedagogical strategies; then, we used analysis of variance to explore how teachers’ digital competence and their ability to involve students in online learning varied across different clusters. our analysis grasps the complexity of the issue, as it shows that two rather distinct strategies were both successful in involving students, and thus, there is no single solution best suited to digital learning. overall, digitally competent teachers loosened the originally strict structure of education and provided more feedback, which proved to be an important element in successfully involving students in digital learning during ert. the framework validated in our research can be used by policymakers and school administrators in different national and educational contexts, enabling them to understand the complexity of online teaching and learning. furthermore, our results can offer some practical pointers for school teachers on how to combine different pedagogical strategies. keywords: covid-19; ict integration in education; teachers’ digital competence; cluster analysis. research in learning technology vol. 31, 2023 introduction the education sector globally has experienced increased exposure to digital disruption in recent years, particularly during the emergency remote teaching (ert) period that occurred as a response to covid-19 and the related school closures. the central question of this paper is how the education sector can adapt and react to the changes and challenges of digital disruption. more specifically, we would like to examine how teachers adapt their teaching strategies by considering technology integration. this paper chose to analyse one country, examining empirical data from hungary, focusing on the ert period starting in march 2020. mailto:horvath.laszlo@ppk.elte.hu http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2978 https://orcid.org/0000-0003-0003-1045 l. horváth 2 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) digital disruption is not a new phenomenon in education; however, it has increased during the ert period and related school closures. digital disruption can be defined as ‘the rapidly unfolding processes through which digital innovation comes to fundamentally alter historically sustainable logics for value creation and capture by unbundling and recombining linkages among resources or generating new ones’ (skog, wimelius, and sandberg 2018, p. 432). it is especially interesting to examine the effects of digital disruption on the field of education because digital disruption has a two-fold impact. it not only affects the process of teaching and learning and the operation of schools, but it also affects the wider socio-economic contexts of students newly entering education and the world of work, as education must equip students with the necessary working skills. in our article, we first describe the context of our study: the ert period (in general) and the hungarian education system (in particular, related to issues of digital disruption). the theoretical background of our article builds on the intersection of technology integration and innovation theory; therefore, we then introduce this related context. after establishing the theoretical background of our article, we present the research methodology. we employed a quantitative research strategy to explore how teachers adapt their teaching strategies to the challenges of digital disruption. we conducted an online survey among hungarian primary school teachers (n = 4028) to gather their perceptions regarding changes in their pedagogical practice and digital competence. we first present the descriptive data; then, we try to establish typical pedagogical strategies via cluster analysis using the introduced variables. finally, we present the details of the established clusters in relation to teachers’ digital competence (tdc) and their ability to involve students in the teaching and learning processes. to conclude, we discuss the relevance of our research, practical implications, possibilities to generalise the research, and limitations and further research opportunities. the context of this study: ert there is empirical evidence from all around the world about how teachers have reacted to ert. a narrative study by nilsberth et al. (2021) emphasised how assessment in teaching shifted from a summative to a formative approach. another study by beattie, wilson, and hendry (2022) referred to the challenges of providing personalised learning experiences during ert, while others focused on the realisation of professional development needs and the lack of teacher competencies (castañeda-trujillo et al. 2021; yong et al. 2021). the term ert refers to a sudden change in the teaching context whereby stakeholders do not have enough time to plan for this transition or the necessary infrastructure and competencies (barbour et al. 2020). ert provides a unique context to explore how teachers integrate technology into their teaching and learning practices. it is especially important to explore how teachers reacted to ert in the case of hungary. the 2016 hungarian digital education strategy concluded that teachers rarely used digital tools: less than 20% of teachers used digital solutions in more than a quarter of their classes (digital success programme 2016). in 2019, the european commission (2019) found that hungary is one of the lowest-ranking eu member states in terms of the percentage of classes supported by ict tools. suddenly, in march 2020, the situation changed from 1 week to the next, as schoolteachers were required to ensure the continuity of teaching via remote tools. this global phenomenon allows http://dx.doi.org/10.25304/rlt.v31.2978 research in learning technology citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 3 (page number not for citation purpose) researchers to gauge teachers’ initial considerations and strategies related to technology integration. the circumstances for technology integration were not ideal; thus, these responses should be considered sudden responses in an emergency and not wellplanned strategic approaches. nevertheless, we believe that our data can provide an in-depth understanding of technology integration processes in education. first, we will review existing empirical research on technology integration. after discussing the relevant dimensions of the phenomenon, we will focus on tdc as an important factor determining successful technology integration. technology integration through the lens of innovation theory according to the national center for educational statistics (2005), technology integration can be defined as ‘the incorporation of technology and technology-based practices into the daily routines, work, and management of an organization’. our study focuses on daily routines and work rather than management. daily routines and work are the core business of schools: the processes and organisation of teaching and learning. to encompass both the top-down and bottom-up approaches, we interpret technology integration through the lens of educational innovation theory. using innovation theory also aligns with the broader theme of digital disruption, as it can be understood from the perspectives of the schumpeterian notion of creative destruction, putting the dynamics of disruptive innovation (christensen 1997) into focus. the oecd (2019) defines innovation using the oslo manual’s definition (oecd 2018): ‘a new or improved product or process (or a combination thereof) that differs significantly from the unit’s previous products or processes and that has been made available to potential users (product) or brought into use by the unit (process)’. the broad definition of innovation encompasses various types of innovation. furthermore, the innovation theory perspective distinguishes between the output, product, process, and nature of the phenomenon, where researchers’ focus could shift between the specifics of the given innovation and the characteristics of the agent responsible for creating/adapting the given innovation. the oslo manual (oecd 2018) defines the former as an ‘object’ and the latter as a ‘subject’ method for measuring innovation. our analysis adopts a subject method as we do not focus on the specific digital innovations, technologies, and solutions adopted but on the agent (teachers) who are adopting the innovations and changing their practices and processes. innovation theory (which is predominant in public sector innovation studies; arundel, bloch, and ferguson 2019) could shed light on the usually invisible (fuglsang 2010) and hidden (nesta 2007) everyday innovation processes (lippke and wegener 2014) driven by employees –in this case, teachers (darsø and høyrup 2012). innovation processes can also be described using the notion of bricolage or tinkering; that is, those with lived experience of the process can create new structures through intrinsic actions that allow new ways of working (fuglsang 2010). this latter perspective is especially relevant for analysing emergent strategies in ert as teachers had little to no time to plan and prepare for this situation; therefore, they had to rely on their existing experiences to try to solve the challenge. in the next section, we will review recent research results regarding technology integration in education to provide a theoretical base for our analysis of emergent http://dx.doi.org/10.25304/rlt.v31.2978 l. horváth 4 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) strategies. we will examine technology integration through the lens of innovation theory, which was presented in this section. technology integration in education the previously cited definition of technology integration does not encompass the complex nature of the phenomenon. amiel, kubota, and wives (2016) also draw attention to the importance of context in the emergence, diffusion, and adoption of (digital) innovations in an educational setting: technology integration does not happen in isolation; it must be understood from a systemic perspective. following an ecosystem approach, amiel, kubota, and wives (2016) emphasise (after nardi and o’day 1999) that the focus should not be on the technology itself but on the activities executed with the help of the technology and the context of implementation. it is important to investigate issues related to technology integration from a critical perspective as this integration does not always have positive results and can even lead to the reproduction of traditional methods (chand, deshmukh, and shukla 2020). drumm (2019) emphasises that teachers often rationalise their digital teaching with folk pedagogies (personalised mental models of teaching) and pseudo-theories of learning, which drumm partly considers a hindrance but also sees as a starting point for teachers. considering that teachers’ beliefs play an important role in successful technology integration, a failed or unsuccessful attempt at integration could reinforce traditional beliefs about the importance of ‘knowing content’ and about inequalities in learning (chand, deshmukh, and shukla 2020). technology integration without the necessary pedagogical considerations could easily facilitate behaviourist approaches to learning through automation and interactivity (drumm 2019). based on talis data, gil-flores, rodríguez-santero, and torres-gordillo (2017) highlight that high ict use is not only linked with constructivist beliefs about learning but also a higher level of collaboration among teachers. besides individual factors (digital competence, beliefs), group and organisational level factors also play an important role in explaining technology integration. school leaders play an important role in supporting teachers in technology integration by improving the ict infrastructure and creating a positive learning culture (vermeulen et al. 2017). there are many models of technology integration, which change depending on the different elements they highlight. among the most prominent ones are the technology acceptance model (tam) and its updated version, the unified theory of acceptance and use of technology (utaut; davis 1989; davis, bagozzi, and warshaw 1989; venkatesh et al. 2003). these models are based on different theories, including the theory of reasoned action, the theory of planned behaviour, the innovation diffusion theory, and the social cognitive theory. the models predict technology use by considering factors such as performance and effort expectancy, social factors, facilitating conditions, and moderating variables such as gender, age, and experience. there are also other models that are more pedagogically focused, such as the substitution – augmentation – modification – redefinition (samr) model (puentedura 2003) or the passive-interactive-creative – replace-amplify-transform (pic-rat) model (kimmons, graham, and west 2020). however, these models do not have widespread, validated tools that would enable researchers to use them as valid and reliable instruments in explaining technology integration. http://dx.doi.org/10.25304/rlt.v31.2978 research in learning technology citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 5 (page number not for citation purpose) although these are prominent models widely used in the literature and educational practice, they do not address an important factor explaining technology integration in education: tdc. teachers’ digital competence falloon’s (2020) article summarises the different frameworks for understanding tdc: the technological, pedagogical, and content knowledge (tpack) framework, the distributed thinking and knowing, engagement, communication and community, knowledge building (deck) framework, the critical digital literacy framework (cdl), the teacher education information literacy (teil) framework, the international society for technology in education (iste) standards, the digital competence for educators (digcompedu) framework and the tdc framework. the majority of studies on tdc use one or a combination of these frameworks (see, for example, akturk and ozturk 2019; cerratto pargman, nouri, and milrad 2018; chou et al. 2020). this long list of different frameworks shows that there is ambiguity surrounding tdc in the scientific discourse. for this study, we selected a recently developed framework, the digcompedu, which has been, so far, validated in a few countries (benali, kaddouri, and azzimani 2018; cabero-almenara et al. 2020; ghomi and redecker 2019). the framework describes six areas of tdc (using 22 indicators), considering both teachers’ professional and pedagogical competencies and learners’ competencies. the six areas are (1) professional engagement, (2) digital resources, (3) teaching and learning, (4) assessment, (5) empowering learners, and (6) facilitating learners’ digital competence (redecker 2017). research methodology this study uses the results of an online survey of hungarian primary school teachers to explore the response of the school system to the digitally disruptive phenomenon caused by covid-19 in the form of ert. the aim of our study is to explore the different pedagogical strategies that primary school teachers used during this period. by analysing what strategies digitally competent teachers used and how successful these strategies were in involving and engaging students, we can discuss the effectiveness of these pedagogical strategies. as our study is explorative, we used a quantitative research strategy to answer our research questions. we created a database using our online survey and analysed the data with jamovi in alignment with our research questions. the following table (table 1) describes the main aim of this study with its sub-questions, along with the data analysis method used to answer the research questions. the next subsections detail the sample and the instrument used in this study to answer our research questions, and then the study presents the results. sample the survey was sent out to 450 primary schools in hungary that participated in the efop-3.1.7 project (focusing on supporting primary schools’ capacity to combat early school leaving) in may 2020 (just a few months after the sudden shift to ert). the final sample consists of 4028 teachers from 343 primary schools from all the regions http://dx.doi.org/10.25304/rlt.v31.2978 l. horváth 6 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) in hungary. in each responding school, nearly half (47%) of the teachers completed our survey, which is overall a high response rate (76% of schools participated). the sample has the following demographic characteristics. the majority of the respondents are female (85.2%), which aligns with the proportion of women in the teacher population as a whole (85%; hajdu et al. 2022). hungarian public education is characterised by an aging teacher workforce: 47.7% of teachers are above 50 years old (hajdu et al. 2022). in our sample, 43.7% of teachers are above 51 years old; therefore, we consider our sample as representative in terms of age. respondents were also asked about their position and disciplinary focus (table 2). overall, our sample adequately characterises the hungarian primary school teacher population in terms of gender and age. instrument to assess the changes in the pedagogical practices and the organisation of education, we created a framework of items where respondents could indicate their chosen approach in their previous routine. respondents could then indicate the direction of the change on a scale of 0–100. a point ranging from 0 to 49 meant a shift towards the element indicated on the left side of the scale, while a point between 51 and 100 indicated a shift towards the element on the right side of the scale. the following items were used: • structure of learning (tight – loose): whether respondents opted for keeping disciplinary subjects separate in 45-minute classes or moved towards a looser structure utilising cross-curricular teaching. • role of teacher (support – control): whether respondents opted to have a more supportive learning environment as opposed to a more controlled and structured one. • quantity of feedback (less – more): whether respondents gave less or more feedback than before. table 1. research questions and data analysis methods. aim: explore how teachers adapt their teaching strategies considering technology integration. rq1: what are the different pedagogical strategies that primary school teachers adopted during the early days of ert? cluster analysis and a description of clusters along the main variables. rq2: what pedagogical strategies did digitally competent teachers adopt in their response to the sudden shift to online learning? analysis of variance: comparing the previously identified clusters (pedagogical strategies) with teachers’ digital competence. rq3: how successful was each pedagogical strategy in terms of student involvement in online learning? analysis of variance: comparing the previously identified clusters (pedagogical strategies) with the reported percentage of involved students. ert, emergency remote teaching. http://dx.doi.org/10.25304/rlt.v31.2978 research in learning technology citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 7 (page number not for citation purpose) • organisation of learning (asynchronous – synchronous): whether classes were scheduled asynchronously or respondents tried to meet regularly with their students online. • type of assessment (formative – summative): whether respondents’ assessment practices shifted more towards a formative or summative approach. several pedagogical strategies exist depending on the combination of these items. therefore, this study aimed to explore the different pedagogical strategies that primary school teachers in hungary followed during the early days of ert. additionally, it is important to understand how teachers’ digital competencies influenced their choice of pedagogical strategy during the ert period. another aim of our study was to explore what pedagogical strategies digitally competent teachers followed in response to the sudden shift to online learning. to assess tdc, we used digcompedu. the instrument measures the level of tdc by summing a set of items. we used the instrument by offering a 5-point likert scale where respondents could assess each item and clarify how often they use it in their everyday practices. the instrument was translated into hungarian by horváth et al. (2020) for a teacher-educator sample; their translation was reworded in this study for a primary education sample. table 2. background information on the sample. background variables n % of valid responses gender female 3364 85.2 male 584 14.8 age 25 years old or younger 46 1.2 25–30 years old 161 4.1 31–40 years old 739 18.8 41–50 years old 1268 32.3 51–60 years old 1467 37.3 61–65 years old 235 6 65 years old or older 15 0.4 position teacher with managerial roles (e.g. principal, vice-principal, head of committee) 1271 31.7 teachers without managerial roles 2551 63.6 other (e.g. school psychologist, pedagogical assistant) 190 4.7 disciplinary area based on the national core curriculum (multiple choice) physical education and sports 1092 27.9 lifestyle and practice 915 23.3 informatics 319 8.1 arts 1217 31 our earth and our environment 740 18.9 human and nature 819 20.9 human and society 513 13.1 mathematics 1484 37.9 ethnic language and literature 71 1.8 foreign languages 453 11.6 hungarian language and literature 1626 41.5 http://dx.doi.org/10.25304/rlt.v31.2978 l. horváth 8 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) finally, student involvement in online learning during ert was measured by teachers’ approximation of the percentage of students they were able to actively involve in the teaching and learning process. thus, we wanted to explore how successful each pedagogical strategy was in terms of student involvement in online learning. this was measured separately for grades 1 to 4 (isced 1) and grades 5 to 8 (isced 2) as we felt that younger students could be considered a different target group as they are in the process of learning how to read, which could prove an important factor in using online learning tools and connecting to online lessons. results descriptive statistics to provide an overall picture of hungarian primary schools’ responses to ert, we present the main descriptive statistics, the level of tdc, and the percentage of students involved in digital learning as reported by teachers. first, the results regarding the changes in pedagogical practice in terms of the examined dimensions show (figure 1) that the main strategy was to have a looser structure of classes with teachers providing more control than in face-to-face settings. teachers provided feedback more frequently and reported using a summative type of assessment while shifting towards asynchronous solutions in the organisation of learning. to measure tdc, the 22 items of the digcompedu (assessed on a 5-point likert scale) were summed. the reliability statistic according to mcdonald’s ω is high (0.930); therefore, there is internal consistency. we also checked the validity of the one-factor model with confirmatory factor analysis (cfa) using a diagonally weighted least squares (dwls) method and robust standard errors (considering the ordered nature of the items). the cfa yielded acceptable model fit measures (χ2 (203) 59.0 55.7 62.4 48.7 53.2 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 structure of learning (tight loose) role of teacher (support control) quantity of feedback (less more) organization of learning (asynchronous synchronous) type of assessment (formative summative) figure 1. the mean and standard deviation of the scales (0–100) showing the shift in pedagogical strategies of hungarian primary education teachers during the emergency remote teaching (ert) period. points lower than 50 mean a shift towards the left side of the scale, and points higher than 50 mean a shift towards the right side of the scale. http://dx.doi.org/10.25304/rlt.v31.2978 research in learning technology citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 9 (page number not for citation purpose) = 6367; p < 0.001; cfi = 0.978; tli = 0.975; rmsea = 0.092; 95% ci [0.090; 0.094]; srmr = 0.079) according to the thresholds proposed (cli and tli > 0.95; rmsea < 0.10, and srmr < 0.08) by boateng et al. (2018). the summed-up tdc variable ranges from 22 to 110, with a mean of 88 and a standard deviation of 14.8. by calculating the levels of tdc according to the original instrument (similar to the levels of the common european framework of reference for languages), we can see that primary school teachers reported a high level of digital competency (45% [n = 1699] c1 – leader; 24.6% [n = 929] b2 – expert). finally, the teachers believed they were able to involve around 75%–80% of their students in digital learning in the ert period. they reported higher success with involving younger children (isced 1; m = 79.5%; sd = 23%) as compared to isced2 students (m = 74.6%; sd = 20.6%). identifying typical pedagogical strategies responding to ert (rq1) the main aim of our study is to explore pedagogical strategies that primary school teachers used immediately after shifting to ert. in accordance with our first research question (rq1: what are the different pedagogical strategies that primary school teachers followed during the early days of ert?), we conducted a cluster analysis on the variables used to describe changes in pedagogical practices and the organisation of education. the following variables were included in the cluster analysis: • structure of learning (tight – loose) • role of the teacher (support – control) • quantity of feedback (less – more) • organisation of learning (asynchronous – synchronous) • type of assessment (formative – summative) a k-means cluster analysis method was used with the hartigan-wong algorithm using jamovi. by looking at the gap statistic (k) using the method developed by kassambara and mundt (2020), we established that the optimal number of clusters was four. the cluster membership variable was saved to the database, providing opportunities to analyse and compare the different groups. the following diagram (figure 2) shows the descriptive statistics of the cluster variables for each of the four clusters with the number of respondents for each cluster. teachers in cluster 1 loosened up the structure of learning, deviating from the usual schedule of 45-minute classes and opting for a more flexible arrangement. respondents in this group reported that they acted in more supportive ways and provided more formative feedback than before. they also opted for an asynchronous type of teaching. teachers in cluster 2 adopted the opposite strategy of cluster 1 in some regards. they tightened the structure, strictly sticking to the previous schedule, and were more controlling (as in providing more structure). while they did not change the organisation of learning nor the quantity of feedback provided, they still noted a shift towards summative assessments. teachers in cluster 3 were similar to those in cluster 1 in that they opted for a loosened structure, but they accompanied this with stricter control in learning. they opted for a synchronous approach with a summative type of feedback and a stronger controlling role. finally, cluster 4 teachers provided http://dx.doi.org/10.25304/rlt.v31.2978 l. horváth 10 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) a tighter structure of learning and emphasised asynchronous learning opportunities; they also noted they took on a more supportive teacher role with less, although formative, feedback. table 3 describes and highlights the key characteristics of each cluster to provide a comprehensive overview of each pedagogical strategy that we identified through our cluster analysis. once we identify the characteristics of the pedagogical strategies of hungarian primary school teachers, it is worth it to explore the demographic composition of these clusters. table 4 indicates the gender, age, and disciplinary composition of the clusters. although the contingency table analysis shows there are no significant associations between variables (except for gender, which has a weak association), looking at the descriptive percentages provides insights regarding the composition of the clusters. according to the results, most male teachers are in cluster 4, while cluster 3 has the most senior teachers. regarding disciplinary composition, cluster 2 has the most pe teachers, while cluster 1 has humanistic-oriented teachers, and cluster 4 mostly hosts natural science-oriented teachers. our analysis successfully identified and described the pedagogical strategies that hungarian primary school teachers followed in the early days of ert. this enables us to answer our first research question (rq1: what are the different pedagogical strategies that primary school teachers followed during the early days of ert?), identifying and describing four pedagogical strategies based on different characteristics. in the next section, these clusters represent groups of teachers following distinctive pedagogical strategies. according to our research aims, we explored what characterises digitally competent (based on the digcompedu) teachers’ behaviour and how effective these strategies were in involving and engaging students. 67.4 28.5 76.9 36.9 31.9 48.0 66.4 63.6 54.5 62.0 76.5 80.9 79.3 78.0 80.9 46.4 41.1 30.3 32.4 39.1 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 structure of learning (tight loose) role of teacher (support control) quantity of feedback (less more) organization of learning (asynchronoussynchronous) type of assessment (formativesummative) cluster1 (n=377) cluster2 (n=546) cluster3 (n=486) cluster4 (n=453) figure 2. means and standard deviation of the scales (0–100) for each cluster describing the shift in hungarian primary education teachers’ pedagogical strategies during the emergency remote teaching (ert) period. points lower than 50 mean a shift towards the left side of the scale, and points higher than 50 mean a shift towards the right side of the scale. http://dx.doi.org/10.25304/rlt.v31.2978 research in learning technology citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 11 (page number not for citation purpose) table 3. basic characteristics of pedagogical strategies employed by hungarian primary school teachers during ert. cluster no. cluster 1 (n = 377) cluster 2 (n = 546) cluster 3 (n = 486) cluster 4 (n = 453) structure of learning (tight – loose) tight tight tight tight no change no change no change no change loose loose loose loose role of teacher (support – control) support support support support no change no change no change no change control control control control quantity of feedback (less – more) less less less less no change no change no change no change more more more more organisation of learning (asynchronous – synchronous) asynchronous asynchronous asynchronous asynchronous no change no change no change no change synchronous synchronous synchronous synchronous type of assessment (formative – summative) formative formative formative formative no change no change no change no change summative summative summative summative ert, emergency remote teaching. table 4. demographic composition of clusters. descriptive indicators cluster1 cluster2 cluster3 cluster4 contingency table analysis gender (% of male teachers within clusters) 13.3 17.5 14.0 20.1 χ2 (3) = 9.46; p = 0.024 cramer’s v = 0.0717 age (% of teachers above 50 within clusters) 43.6 43.5 49.2 43.8 χ2 (18) = 14.6; p = 0.690 cramer’s v = 0.0515 d is ci p li n ar y fi el d ( % o f te ac h er s te ac h in g [… ] w it h in c lu st er s) physical education and sports 24.20 29.80 25.10 28.20 χ2 (3) = 4.90; p = 0.179 cramer’s v = 0.0517 lifestyle and practice 23.60 21.10 20.50 21.20 χ2 (3) = 1.38; p = 0.711 cramer’s v = 0.0274 informatics 10.30 8 10.20 8.60 χ2 (3) = 2.36; p = 0.500 cramer’s v = 0.0359 arts 34.80 25.70 30.70 26.60 χ2 (3) = 10.6; p = 0.014 cramer’s v = 0.0761 our earth and our environment 16.80 18 18.60 19.90 χ2 (3) = 1.30; p = 0.728 cramer’s v = 0.0267 human and nature 23.60 17.60 21.70 19.40 χ2 (3) = 5–78; p = 0.123 cramer’s v = 0.0562 human and society 15.50 13.10 11.30 12.40 χ2 (3) = 3.43; p = 0.330 cramer’s v = 0.0433 mathematics 35.60 37 34.40 39.10 χ2 (3) = 2.31; p = 0.511 cramer’s v = 0.0355 ethnic language and literature 1.60 1.50 1.50 2.50 χ2 (3) = 1.88; p = 0.599 cramer’s v = 0.0320 foreign languages 14.10 11.50 12.50 12.40 χ2 (3) = 1.41; p = 0.704 cramer’s v = 0.0277 hungarian language and literature 42.10 39.60 40.30 38.40 χ2 (3) = 1.22; p = 0.747 cramer’s v = 0.0259 note: largest percentages within clusters are highlighted in bold. http://dx.doi.org/10.25304/rlt.v31.2978 l. horváth 12 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) assessing tdc and teachers’ ability to involve students in ert (rq2 and rq3) to answer our second research question (rq2: what pedagogical strategies did digitally competent teachers adopt in their response to the sudden shift to online learning?), we investigated how digitally competent teachers acted during ert and what kind of pedagogical strategies they followed. as discussed, we summed up the indicators of the digcompedu scale to get an overall indicator of tdc. the sub-dimensions of the digcompedu framework show that hungarian primary school teachers are advanced in selecting, creating, and managing digital resources (m = 4.35; sd = 0.68) and in professional engagement (m = 4.21; sd = 0.66), focusing on organisational communication, professional collaboration, reflective practice, and digital continuous professional development. there are areas that need development, such as teaching and learning (m = 3.89; sd = 0.79), where teachers reported lower levels of competence in teaching, guidance, collaborative learning, and self-regulated learning with digital solutions. another area that needs improvement is facilitating learners’ digital competence (m = 3.77, sd = 0.90), meaning developing learners’ information and media literacy, communication, content creation, responsible use, and problem-solving. the summed-up variable of digcompedu measuring tdc failed to demonstrate normality (shapiro-wilk test: w = 0.969; p < 0.001). therefore, we used a non parametric one-way analysis of variance (kruskal-wallis) to explore differences between the clusters of pedagogical strategies. we used the dwass-steel-critchlowfligner method to compare the different groups pairwise. figure 3 shows the results (descriptive data and statistically significant comparisons). according to the results of the kruskall-wallis test (χ2 (3) = 107.2; p < 0.001), there are significant and intermediate (ε2 = 0.058) differences between the clusters. furthermore, all pairwise comparisons are significant. teachers in cluster 3 (m = 93.5; sd = 12.2) and cluster 1 (m = 91.1; 91.1 88.1 93.5 84.2 12.0 32.0 52.0 72.0 92.0 112.0 132.0 152.0 ** * *** *** *** *** cluster 1 cluster 2 cluster 3 cluster 4 figure 3. means and standard deviations of teachers’ digital competence (tdc) (measured by the sum of digcompedu scores) for each cluster, indicating statistically significant differences between groups according to the results of the dwass-steel-critchlowfligner pairwise comparisons after the non-parametric anova (kruskal-wallis). *p < 0.05; **p < 0.01; ***p < 0.001. http://dx.doi.org/10.25304/rlt.v31.2978 research in learning technology citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 13 (page number not for citation purpose) sd = 12.8) had higher levels of digital competence compared to those in cluster 2 (m = 88.1; sd = 13.5) and cluster 4 (m = 84.2; sd = 15.4). thus, in answer to our second research question (rq2: what pedagogical strategies did digitally competent teachers adopt in their response to the sudden shift to online learning?), the results show that teachers who were more digitally competent reacted to ert by loosening the structure but providing stricter control over the teaching and learning process and using summative assessments and synchronous learning opportunities. as all the digcompedu means for the clusters are between 82.5 and 100 points (c1 – leader domain), we must treat this conclusion with caution as it can mean, in reality teachers are basically on the same level of expertise. it is important to examine the effectiveness of teachers in each cluster in terms of their ability to involve and engage students during ert periods, which is our third research question (rq3: how successful can each pedagogical strategy be considered in terms of student involvement in online learning?). as stated, primary school teachers reported that they could involve, on average, 79.5% (sd = 23%) of students on the isced1 level and, on average, 74.6% (sd = 20.6%) of students on the isced2 level. however, as before, both variables failed the prerequisite of normality (shapiro-wilk test: isced1: w = 0.835; p < 0.001; isced2: w = 0.917; p < 0.001); therefore, the non-parametric anova was used. figure 4 shows the descriptive statistics and the results of pairwise comparisons. the results and comparisons show that the pedagogical strategies used by teachers in cluster 3 (isced1 m = 83.3%; sd = 20.3%; isced 2 m = 80.4%; sd = 18.9%) and cluster 1 (isced 1 m = 85.2%; sd = 19.8%; isced2 m = 78.8%; sd = 18.5%) are successful. there are no significant differences between the percentage of involved students for cluster 3 and cluster 1; however, both percentages are significantly different 85.2 78.877.0 73.0 83.3 80.4 73.7 68.7 0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 % of students involved (isced 1) % of students involved (isced 2) *** *** *** *** *** *** *** *** cluster1 cluster2 cluster3 cluster4 figure 4. means and standard deviations of the percentage of involved students (isced 1 and 2) in emergency remote teaching (ert) (based on self-declaration of teachers) for each cluster, indicating statistically significant differences between groups according to the results of the dwass-steel-critchlow-fligner pairwise comparisons after the non-parametric anova (kruskal-wallis). *p < 0.05; **p < 0.01; ***p < 0.001. http://dx.doi.org/10.25304/rlt.v31.2978 l. horváth 14 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) from cluster 2 (isced1 m = 77%; sd = 24%; isced2 m = 73%; sd = 20.1%) and cluster 4 (isced1 m = 73.7%; sd = 25.6%; isced2 m = 68.7%; sd = 22.5%). the same pattern can be seen for both isced1 and isced2 students. however, on the descriptive level, it seems that the strategies applied by cluster 1 teachers are slightly more successful for isced1 students, while strategies used by cluster 3 teachers are slightly more successful for isced2 students. thus, in answer to our third research question (rq3: how successful can each pedagogical strategy be considered in terms of student involvement in online learning?), it seems that the most successful approach for isced1 students is a loosened structure with more supportive teacher behaviour, more formative-type feedback, and more asynchronous learning opportunities. instead, more control, summative feedback, and synchronous learning opportunities worked better for isced2 students. discussion and conclusion the main aim of our study was to explore how the education sector adapted and reacted to the changes and challenges of digital disruption. we used the example of the covid-19-induced ert period to gauge teachers’ emergency responses. to identify the different pedagogical strategies that primary school teachers followed during the early days of ert (rq1), we clustered respondents based on their answers regarding the change in practice in the structure of education, the role of the teacher, quantity of feedback, organisation of learning and type of assessment. at the end of our analysis, four distinctive groups emerged. to better understand the different strategies primary school teachers used, we examined what strategies digitally competent teachers followed (rq2) and which strategy led to the highest percentage of students being involved in digital learning (rq3). according to our results, the successful strategies involved teachers loosening the strict frameworks, teaching structure, and learning processes and providing more feedback than usual (common elements of cluster 3 and 1). digitally competent teachers (rq2) exercised more control over the teaching and learning processes than usual and relied on synchronous teaching solutions and more summative assessments during digital learning (cluster 3). the same strategies (more control, synchronous teaching solutions, and summative assessments) were also best at successfully involving students in the learning process (rq3; cluster 3 and 1). the study found that a pedagogical strategy that is more supportive and provides more formative assessment with dominantly asynchronous teaching methods can also be successful. it seems this latter strategy was more successful for isced1 students, while the former strategy (cluster 3) was more successful for isced2 students. our results show that both synchronous and asynchronous strategies can be successful. in cluster 1, teachers paired asynchronous learning with more support, while in cluster 3, teachers paired the synchronous approach with more control. our results emphasise that online teaching and learning is a complex issue that cannot be described using one dimension (e.g. asynchronous or synchronous) but requires different characteristics to be considered together. this finding is in line with fernandez, ramesh, and manivannan (2022), who emphasised that synchronous learning was more stressful for students, but asynchronous learning put a greater burden on them. therefore, our findings indicate that the complex pedagogical strategy linking teachers’ role (support/control) to their learning organisation (synchronous/asynchronous) http://dx.doi.org/10.25304/rlt.v31.2978 research in learning technology citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 15 (page number not for citation purpose) can have important practical implications: teachers’ control behaviours can help manage and structure workload, while their supportive behaviours can help manage stress. our cluster analysis approach can be generalised and should be tested in other educational and national contexts as it seems to grasp the complexity of the issue. adding to this complexity, the results’ implications are that two distinct strategies can both be successful, emphasising the importance of personalisation (reiterating the claims of beattie, wilson, and hendry 2022), and that there is no one-size-fits-all approach to digital learning. the literature suggests that technology integration can lead to the reproduction of traditional methods (such as in the case of cluster  3), as confirmed by chand, deshmukh, and shukla (2020), but it can also link to constructivist beliefs (such as in the case of cluster 1) as concluded by gil-flores, rodríguez-santero, and torres-gordillo (2017). overall, loosening the originally strict structure of education and providing more feedback (either summative or formative – nilsberth et al. 2021, supported the effectiveness of the latter form) was important for digitally competent teachers. it also proved to be an important element in successfully involving students in digital learning during ert. finally, this study has some limitations. although our sample is representative of hungarian primary school teachers, it is unclear how it can be generalised to other national and educational contexts. although the international literature discussed in this paper provides some support for the applicability of our claims to other contexts, the results should be further explored. additionally, as the study employed a quantitative approach using a self-reporting questionnaire, further data could ensure the validity of our results by conducting qualitative studies to understand the different pedagogical strategies in practice. overall, our research sheds light on the complexity of online teaching and learning during emergencies (in this case, the covid-19 pandemic). although teachers’ strategies were immediate responses to an external phenomenon, it seems there were deliberate pedagogical considerations behind these choices. our research adds to the discourse on online teaching and learning by introducing a framework to identify different pedagogical strategies for online teaching and learning. policymakers and school administrators can use the framework and underlying results to better understand their approach to online teaching and learning and introduce supporting policies that consider the complexity of the model. finally, the results can be useful for school teachers as they can help them see the complex interrelations and different elements that a pedagogical strategy can contain, highlighting possible pitfalls and opportunities. funding project no. 134206 has been implemented with the support provided from the national research, development and innovation fund of hungary, financed under the otka postdoctoral excellence funding scheme. references akturk, a. o. & ozturk, h. s. (2019) ‘teachers’ tpack levels and students’ self-efficacy as predictors of students’ academic achievement’, international journal of research in education and science, vol. 5, no. 1, pp. 283–294. http://dx.doi.org/10.25304/rlt.v31.2978 l. horváth 16 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) amiel, t., kubota, l. c. & wives, w. w. (2016) ‘a systemic model for differentiating school technology integration’, research in learning technology, vol. 24, p. 31856. doi: 10.3402/ rlt.v24.31856 arundel, a., bloch, c. & ferguson, b. (2019) ‘advancing innovation in the public sector: aligning innovation measurement with policy goals’, research policy, vol. 48, no. 3, pp. 789–798. doi: 10.1016/j.respol.2018.12.001 barbour, m., et al., (2020) understanding pandemic pedagogy: differences between emergency remote, remote, and online teaching, canadian elearning network, halfmoon bay. doi: 10.13140/rg.2.2.31848.70401 beattie, m., wilson, c. & hendry, g. (2022) ‘learning from lockdown: examining scottish primary teachers’ experiences of emergency remote teaching’, british journal of educational studies, vol. 70, no. 2, pp. 217–234. doi: 10.1080/00071005.2021.1915958 benali, m., kaddouri, m. & azzimani, t. (2018) ‘digital competence of moroccan teachers of english’, international journal of education and development using information and communication technology, vol. 14, no. 2, pp. 99–120. boateng, g. o., et al., (2018) ‘a novel household water insecurity scale: procedures and psychometric analysis among postpartum women in western kenya’, plos one, vol. 13, no. 6, p. e0198591. doi: 10.1371/journal.pone.0198591 cabero-almenara, j., et al., (2020) ‘development of the teacher digital competence validation of digcompedu check-in questionnaire in the university context of andalusia (spain)’, sustainability, vol. 12, no. 5, p. 6094. doi: 10.3390/su12156094 castañeda-trujillo, j. e., et al., (2021) ‘pedagogical strategies used by english teacher educators to overcome the challenges posed by emergency remote teaching during the covid-19 pandemic’, íkala, revista de lenguaje y cultura, vol. 26, no. 3, pp. 697–713. doi: 10.17533/ udea.ikala/v26n3a12 cerratto pargman, t., nouri, j. & milrad, m. (2018) ‘taking an instrumental genesis lens: new insights into collaborative mobile learning’, british journal of educational technology, vol. 49, no. 2, pp. 219–234. doi: 10.1111/bjet.12585 chand, v. s., deshmukh, k. s. & shukla, a. (2020) ‘why does technology integration fail? teacher beliefs and content developer assumptions in an indian initiative’, educational technology research and development, vol. 68, no. 5, pp. 2753–2774. doi: 10.1007/s11423-020-09760-x chou, c., et al., (2020) ‘developing and validating a scale for measuring teachers’ readiness for flipped classrooms in junior high schools’, british journal of educational technology, vol. 51, no. 4, pp. 1420–1435. doi: 10.1111/bjet.12895 christensen, c. m. (1997) innovator’s dilemma, harvard business review press, boston, ma. darsø, l. & høyrup, s. (2012) ‘developing a framework for innovation and learning in the workplace’, in practice-based innovation: insights, applications and policy implications, eds h. melkas & v. harmaakorpi, springer-verlag, berlin, pp. 135–154. davis, f. d. (1989) ‘perceived usefulness, perceived ease of use, and user acceptance of information technology’, mis quarterly, vol. 13, no. 3, pp. 319–340. doi: 10.2307/249008 davis, f. d., bagozzi, r. p. & warshaw, p. r. (1989) ‘user acceptance of computer technology: a comparison of two theoretical models’, management science, vol. 35, no. 8, pp. 982–1003. doi: 10.1287/mnsc.35.8.982 digital success programme (2016) digital education strategy of hungary, digital success programme, [online] available at: https://digitalisjoletprogram.hu/files/d4/6b/d46bf17fdef3c9b5c1d38bd6db64c2a7.pdf drumm, l. (2019) ‘folk pedagogies and pseudo-theories: how lecturers rationalise their digital teaching’, research in learning technology, vol. 27. doi: 10.25304/rlt.v27.2094 european commission (2019) 2nd survey of schools: ict in education: hungary country report, european commission, luxembourg. doi: 10.2759/76901 falloon, g. (2020) ‘from digital literacy to digital competence: the teacher digital competency (tdc) framework’, educational technology research and development, vol. 68, no. 5, pp. 2449–2472. doi: 10.1007/s11423-020-09767-4 http://dx.doi.org/10.25304/rlt.v31.2978 https://doi.org/10.3402/rlt.v24.31856 https://doi.org/10.3402/rlt.v24.31856 https://doi.org/10.1016/j.respol.2018.12.001 https://doi.org/10.13140/rg.2.2.31848.70401 https://doi.org/10.1080/00071005.2021.1915958 https://doi.org/10.1371/journal.pone.0198591 https://doi.org/10.3390/su12156094 https://doi.org/10.17533/udea.ikala/v26n3a12 https://doi.org/10.17533/udea.ikala/v26n3a12 https://doi.org/10.1111/bjet.12585 https://doi.org/10.1007/s11423-020-09760-x https://doi.org/10.1111/bjet.12895 https://doi.org/10.2307/249008 https://doi.org/10.1287/mnsc.35.8.982 https://digitalisjoletprogram.hu/files/d4/6b/d46bf17fdef3c9b5c1d38bd6db64c2a7.pdf https://digitalisjoletprogram.hu/files/d4/6b/d46bf17fdef3c9b5c1d38bd6db64c2a7.pdf https://doi.org/10.25304/rlt.v27.2094 https://doi.org/10.2759/76901 https://doi.org/10.1007/s11423-020-09767-4 research in learning technology citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 17 (page number not for citation purpose) fernandez, c. j., ramesh, r. & manivannan, a. s. r. (2022) ‘synchronous learning and asynchronous learning during covid-19 pandemic: a case study in india’, asian association of open universities journal, vol. 17, no. 1, pp. 1–14. doi: 10.1108/aaouj-02-2021-0027 fuglsang, l. (2010) ‘bricolage and invisible innovation in public service innovation’, journal of innovation economics & management, vol. 5, no. 1, pp. 67–87. doi: 10.3917/jie.005.0067 ghomi, m. & redecker, c. (2019) ‘digital competence of educators (digcompedu): development and evaluation of a self-assessment instrument for teachers’ digital competence’, proceedings of the 11th international conference on computer supported education, scitepress – science and technology publications, heraklion, pp. 541–548. doi: 10.5220/0007679005410548 gil-flores, j., rodríguez-santero, j. & torres-gordillo, j.-j. (2017) ‘factors that explain the use of ict in secondary-education classrooms: the role of teacher characteristics and school infrastructure’, computers in human behavior, vol. 68, pp. 441–449. doi: 10.1016/j.chb.2016.11.057 hajdu, t., et al., (2022) a közoktatás indikátorrendszere 2021 [indicator system of the hungarian public education 2021], eötvös loránd kutatási hálózat – közgazdaságés regionális tudományi kutatóközpont, közgazdaság-tudományi intézet, budapest. horváth, l., et al., (2020) ‘tanárképzők digitális kompetenciájának mérése – a digcompedu adaptálása a hazai felsőoktatási környezetre [measuring the digital competence of teacher educators – adapting the digcompedu in hungarian higher education]’, neveléstudomány, vol. 2020, no. 2, pp. 5–25. doi: 10.21549/ntny.29.2020.2.1 kassambara, a. & mundt, f. (2020) factoextra: extract and visualize the results of multivariate data analyses, [online] available at: https://cran.r-project.org/package=factoextra kimmons, r., graham, c. r. & west, r. e. (2020) ‘the picrat model for technology integration in teacher preparation’, contemporary issues in technology and teacher education, vol. 20, no. 1, [online] available at: https://citejournal.org/volume-20/issue-1-20/general/ the-picrat-model-for-technology-integration-in-teacher-preparation lippke, l. & wegener, c. (2014) ‘everyday innovation – pushing boundaries while maintaining stability’, journal of workplace learning, vol. 26, no. 6/7, pp. 376–391. doi: 10.1108/ jwl-10-2013-0086 nardi, b. a. & o’day, v. (1999) information ecologies: using technology with heart, mit press, cambridge, ma. national center for educational statistics (2005) forum unified education technology suite, national center for educational statistics, [online] available at: https://nces.ed.gov/ pubs2005/tech_suite/index.asp nesta (2007) hidden innovation. how innovation happens in six ‘low innovation’ sectors, london, [online] available at: https://media.nesta.org.uk/documents/hidden_innovation.pdf nilsberth, m., et al., (2021) ‘digital teaching as the new normal? swedish upper secondary teachers’ experiences of emergency remote teaching during the covid-19 crisis’, european educational research journal, vol. 20, no. 4, pp. 442–462. doi: 10.1177/14749041211022480 oecd (2018) the measurement of scientific, technological and innovation activities, oslo manual 2018, 4th edn, oecd, paris. doi: 10.1787/24132764 oecd (2019) measuring innovation in education 2019 what has changed in the classroom? oecd, paris. doi: 10.1787/9789264311671-en puentedura, r. r. (2003) ‘a matrix model for designing and assessing network-enhanced courses’, hippasus [preprint], [online] available at: http://www.hippasus.com/resources/matrixmodel/ redecker, c. (2017) digital competence of educators, ed y. punie, publications office of the european union, luxembourg. skog, d. a., wimelius, h. & sandberg, j. (2018) ‘digital disruption’, business & information systems engineering, vol. 60, no. 5, pp. 431–437. doi: 10.1007/s12599-018-0550-4 venkatesh, v., et al., (2003) ‘user acceptance of information technology: toward a unified view’, mis quarterly, vol. 27, no. 3, p. 425. doi: 10.2307/30036540 vermeulen, m., et al., (2017) ‘the role of transformative leadership, ict-infrastructure and learning climate in teachers’ use of digital learning materials during their classes: teachers’ http://dx.doi.org/10.25304/rlt.v31.2978 https://doi.org/10.1108/aaouj-02-2021-0027 https://doi.org/10.3917/jie.005.0067 https://doi.org/10.5220/0007679005410548 https://doi.org/10.1016/j.chb.2016.11.057 https://doi.org/10.21549/ntny.29.2020.2.1 https://cran.r-project.org/package=factoextra https://citejournal.org/volume-20/issue-1-20/general/the-picrat-model-for-technology-integration-in-teacher-preparation https://citejournal.org/volume-20/issue-1-20/general/the-picrat-model-for-technology-integration-in-teacher-preparation https://doi.org/10.1108/jwl-10-2013-0086 https://doi.org/10.1108/jwl-10-2013-0086 https://nces.ed.gov/pubs2005/tech_suite/index.asp https://nces.ed.gov/pubs2005/tech_suite/index.asp https://media.nesta.org.uk/documents/hidden_innovation.pdf https://doi.org/10.1177/14749041211022480 https://doi.org/10.1787/24132764 https://doi.org/10.1787/9789264311671-en http://www.hippasus.com/resources/matrixmodel/ https://doi.org/10.1007/s12599-018-0550-4 https://doi.org/10.2307/30036540 l. horváth 18 citation: research in learning technology 2023, 31: 2978 http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) use of digital learning materials’, british journal of educational technology, vol. 48, no. 6, pp. 1427–1440. doi: 10.1111/bjet.12478 yong, k. l., et al., (2021) ‘challenges in emergency remote teaching among malaysian public elementary school teachers’, international journal of emerging technologies in learning (ijet), vol. 16, no. 24, pp. 74–90. doi: 10.3991/ijet.v16i24.27453 http://dx.doi.org/10.25304/rlt.v31.2978 https://doi.org/10.1111/bjet.12478 https://doi.org/10.3991/ijet.v16i24.27453 rlt1915 1..18 research in learning technology vol. 25, 2017 original research article how to merge courses via skype™†? lessons from an international blended learning project‡ yasemin akbabaa and filiz başkanb adepartment of political science, gettysburg college, gettysburg, pa, usa; bdepartment of political science and international relations, i̇zmir university of economics, i̇zmir, turkey (received 25 august 2016; final version received 28 february 2017) this study reports on an international project in which students taking the course contemporary issues in turkish politics in spring 2011 and fall 2011 at two institutions of higher education, ‘gettysburg college’ in the united states and ‘izmir university of economics’ in turkey, worked together in virtual learning environments to complete various tasks as part of their course work. the project employed a blend of traditional and technology-based teaching methods in order to introduce a technology like skype in a bi-national learning environment in turkey. students collaborated and interacted with their international counterparts in two different virtual contexts. first, classrooms in the two countries were merged via skype three times to conduct classroom-to-classroom discussion sessions on turkish politics. second, students were paired across locations to work on several assignments. in this paper, our goal is to present how skype is used in a bi-national context as a blended teaching tool in an upper-level college course for instructors pursuing a similar exercise. in addition to outlining the process with a focus on skype discussions and one-on-one student projects, we provide actual assignments and discussion questions. students’ views elicited through surveys administered throughout the semester are presented alongside anecdotal evidence to reflect how the project was received. keywords: cross-cultural collaboration; e-learning environment; institutional cooperation; technology-based teaching; international blended learning introduction technology-based teaching practices associated with virtual learning tools, such as distance learning, social media, emails, blogging, wikis and social bookmarking, and communication applications, such as skype, have been praised for performing better in increasing student engagement in comparison to traditional techniques, such as lectures (taylor 2011). it has also been suggested that traditional education practices do not provide the ideal learning environment for the so-called ‘digital †skype is a communication tool that offers voice and video connections to any computer running the skype software application. ‡this project is supported by the mellon foundation’s embedding research in the gettysburg curriculum grant. *corresponding author. email: yakbaba@gettsyburg.edu 1research in learning technology 2017. @ 2017 y. akbaba and f. başkan. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v25.1915 natives’ (coates 2007; hersch and merrow 2005; tapscott 2009). however, there are studies that highlight benefits of non-digital learning tools such as longhand note taking in comparison to laptop note taking ‘even when laptops are used solely to take notes’ (mueller and oppenheimer 2016, p. 1159). so, to what extent should we abandon traditional learning tools and methods? as we utilise technology-based teaching practices, how can we make sure we are not throwing out the baby with the bathwater? blended learning, that is, ‘a course that uses a combination of face-to-face and online learning’,1 provides a middle ground to explore. yet, blended models come in different forms, shapes and sizes. teaching scholars, developing a blended learning course, may find it challenging to pick and choose from various options available. also, like any learning environment, blended learning has its own share of potential pitfalls. the goal of this paper is to present how skype is used as a teaching tool to virtually connect students in turkey with students in the united states in an upper-level college course for instructors pursuing a similar exercise. in other words, our focus is explaining nuts and bolts of how skype as a technology-based teaching and learning tool can be utilized in various contexts (classroom-to-classroom discussion sessions and one-on-one team assignments). our project aimed to introduce a technology like skype in a binational learning environment in turkey. students taking the course, contemporary issues in turkish politics in the spring of 2011 and fall of 2011 at two institutions of higher education, ‘gettysburg college’ in the united states and ‘izmir university of economics’ in turkey, worked together in virtual learning environments to complete various tasks as part of their course work. students collaborated and interacted with their international counterparts in two different virtual contexts. first, classrooms in the two countries were merged via skype three times to conduct classroom-to-classroom discussion sessions on turkish politics. second, students were paired across locations to work on several assignments. instructors also utilised face-to-face methods such as lectures to enhance learning experience. to the best of knowledge of the authors, this is the first blended learning experience that connects students internationally both as a group and as a team of two throughout the semester with a common set of assignments and reading/visual material. surveys given to students in both institutions suggest that students found the virtual learning environment to be rewarding. the overwhelming majority of students in both institutions indicated that the project provided valuable learning opportunities for them, and that classroom-to-classroom discussion sessions and one-on-one team assignments were engaging. the remainder of this study is organised as follows. the next section reviews the existing literature on cross-cultural teaching and technology-based learning, followed by a description of the implementation of the project. in addition to outlining the process, we provide actual assignments, and discussion questions with notes on lessons learned. we also present a summary of students’ views elicited through anonymous surveys administered throughout the semester alongside anecdotal evidence and reflection of instructors. the final section offers concluding remarks. cross-cultural teaching and learning with technology in the last decade, there has been an increased focus on teaching with technology (andone et al. 2007; bryant 2006; elia 2006; palfrey and gasser 2008; frydenberg and andone 2010; liaw 2006; liaw and johnson 2001; gasson and agusto 2008; y. akbaba and f. başkan 2 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 http://dx.doi.org/10.25304/rlt.v25.1915 frydenberg and davi 2006; chan, frydenberg and lee 2007; lukosch 2007; schmidt 2005; degagne and wolk 2007). in the same vein, many educators have used such technology to develop virtual learning environments for students. earlier work on specific techniques for improving student learning enhanced our understanding of deploying information technology for educational purposes and contributed to a body of knowledge that enables a better understanding of technology-based learning. the previous literature with a focus on developing an understanding of technology-based approaches to teaching, addressed the distance-learning dimension of teaching with technology, analysed the use of technology in the classroom, examined ways of improving specific skills of students such as the language proficiency through communication technology and explored methods of assessing and improving digital learning environments. whether it is comparison of ‘massive open online courses (moocs)’2 and ‘small private online courses (spocs)’ (garlock 2015), incorporating blogs and desktop video into a course (beise 2006) or examining how learning was facilitated in a virtual classroom community using online discussion forums (zhu 2010), the main threads of this line of research is about revealing highlights from digital teaching and learning experiences. blended (or hybrid) learning environments,3 where traditional and digital teaching techniques join forces, became an intriguing area to explore for teaching scholars. blended course is ‘a course that uses a combination of face-to-face and online learning’.4 blended courses could potentially create high-impact learning environment by tailoring traditional course structure with the help of online features. for instance, use of social software, such as blogs and wikis, as teaching tools could expand collaboration possibilities for students (bryant 2006). these online features could help instructors to achieve learning outcomes that would not be possible otherwise. they could also provide an individualised learning space that could complement traditional learning setting. a sub-group of technology-based learning studies reported the use of skype technology as a teaching tool. a pilot study connected home-bound students to their schools using skype (samsonov and harris 2010). blau and caspi (2008) compared audio conferencing and traditional face-to-face learning. macharaschwili and coggin (2013) reported their ‘skype-buddy model’ project that pairs students in traditional classroom with distance learners in order to increase interaction opportunities for online learners. their work (macharaschwili and coggin 2013, p. 168) asks ‘how does using skype in a blended learning environment shape participants’ (distance student and proxy student) engagement in the (online and traditional) classroom?’ and ‘what are the satisfactions, benefits, challenges and surprises for other students and instructors in a traditional classroom when skype is used to include a distance student in full class and small group work in the classroom?’ kirschner (2015) combined skype, blogging and class discussions to reduce stereotypes in tandem language learning environment. similarly, abdelmalak (2015) reports how various virtual learning tools such as twitter, google docs, skype, blogs and wikis were used in an online course. findings of the study suggest google docs, wikis, blogs and twitter to perform better in providing students ‘a sense of learning community while using skype did not’ (abdelmalak 2015, p. 1). one of the enriching dimensions of digital learning experiences is creating multicultural and/or international learning environments. communication across cultures and nations provides benefits such as acquiring much needed diversity of research in learning technology citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 3 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1915 skills and knowledge. for instance, international learning communities such as icollab connect learners across the globe without typical course formats (cronin, cochrane and gordon 2016). ‘icollab students work across cohorts, levels, institutions, countries, academic terms and time zones, using social media and other mobile, web-based tools for multimedia production, collaboration, cooperation and reflective practice as well as exploring the creation and negotiation of digital identities and personal learning networks’ (cronin et al. 2016, p. 3). smaller scope international collaborations are reported as well. students studying comparable it courses in the united states and australia formed virtual teams and engaged in collaborative content creation via the internet (chan et al. 2007). similarly, college students in the united states and romania worked together for a month to create a multimedia presentation (frydenberg and andone 2010). liaw and johnson (2001) analysed email correspondence between university efl students in taiwan and pre-service bilingual/esl teachers in the united states to yield insight into the cross-cultural communication process. this project situates itself within works on blended learning. in attempting to emphasise the development of cross-cultural understanding among students and enhance learning experience, we integrated technology-based tools into a typical college course and connected students and classrooms in the united states and turkey. although many studies report the use of internet-based technologies in the united states, there are not many studies that report how far these virtual learning tools are used in turkey. however, there are many studies on perceptions of students and professionals in turkey on technology-based teaching and learning methods. for example, tekinarslan’s (2008) quantitative analysis of attitudes of students towards internet-based learning at sakarya university in turkey suggests attitude levels change based on demographic identifiers. he adds that his findings are consistent with the larger literature on the subject matter, that is, change in attitudes of students in turkey is comparable to students in other countries such as china and the united states. similarly, akkoyunlu and soylu (2008, p. 183) studied perceptions of students at hacettepe university in turkey on blended learning and found that ‘students’ views on blended learning process, such as ease of use of the web environment, evaluation, face-to-face environment etc., differ according to their learning styles’. simsek’s (2005) work, on the contrary, focused on the perception of professionals in six countries: the united states, canada, turkey, indonesia, nigeria and poland. we couldn’t identify any studies on the use of communication applications such as skype in turkey. although skype is reported to be used elsewhere, to the best of our knowledge, however, none of the previously reported projects have connected two traditional classrooms internationally both as a group and as a team of two throughout the semester with a common set of assignments and course material. the next section provides an overview of the project. project overview contemporary issues in turkish politics is an upper-level college course that seeks to familiarise students with the main issues around which politics revolves in turkey during the post-1980 period. using scholarly work conducted on different dimensions of politics in turkey, special attention is paid to the discussion of the issues of the consolidation of democracy, civil society, secularism, the rise of islam, nationalism, y. akbaba and f. başkan 4 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 http://dx.doi.org/10.25304/rlt.v25.1915 identity politics, socio-economic changes, political parties and modernity in order to capture the essence of the changing nature of turkish politics. the course was offered by two institutions of higher education, ‘gettysburg college’ in the united states and ‘izmir university of economics’ in turkey in the spring of 2011 and fall of 2011.5 students worked together in virtual learning environments to complete various tasks as part of their course work. students collaborated with their international counterparts in two different contexts. first, the two classrooms were merged via skype three times to conduct classroom-to-classroom discussion sessions on turkish politics. second, students were paired up across the locations to work on several assignments.6 each assignment required them to have a one-on-one discussion session with their international counterparts. the goal was to enhance learning experience by creating international peer-learning opportunities for the students in both large and small groups. students were also encouraged to locally share what they have learned from their one-on-one sessions with their classmates. we recommended students to use video chat systems, such as skype, for their one-on-one virtual meetings. classroom-to-classroom skype discussion sessions the dates of all three classroom-to-classroom discussion sessions were posted on the syllabi. since class meeting times were coordinated such that 75 min of class time intersected across the two locations, all of the discussion sessions were scheduled during class meeting times. prior to each session, students received discussion questions accompanied by guidelines. the instructors communicated their expectations of students prior to each session. students were asked to come to class with a copy of the questions provided to them.7 instructors then coordinated and facilitated the discussion sessions and summarised students’ inputs and asked follow-up questions after each question. the sessions covered a wide variety of topics, including but not limited to turkish secularism, the role of the military in turkish politics, the role of various political parties in turkish politics, political islam and the kurdish question. the following are sample questions from discussion sessions. • do you think the secularism formula of turkey was successful? • is it harder for muslim majority states to separate religion and state? • turkey has a history of military interventions followed by transitions to democracy. do you think the transition to democracy justifies the intervention? is it acceptable to have military interventions when there is transition to democracy in the end? • which of the following statements do you agree with the most? explain why you agree with a particular statement. (1) in the early years of the republic of turkey, it was necessary to restrict the cultural rights of the kurdish minority to maintain the territorial integrity of the country. however, over time, these restrictions should have been eased. (2) kurdish nationalism is a security challenge for turkey. (3) it is possible to maintain territorial integrity of turkey and advocate kurdish cultural and political rights. • this course provided opportunities for cross-cultural exchange. did you refute any stereotypes about the ‘other’ culture? the discussion sessions provided a high-impact learning environment for students. while some decided on their research project topics because of what they had heard research in learning technology citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 5 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1915 during these sessions, some integrated discussion content to other assignments such as essays. instructors received various questions from their students on the discussion content during their lectures. remarks made by international counterparts worked as a point of departure for some of the lectures. most importantly, students remained engaged and committed to the discussion sessions and there were no shortage of comments from both ends. team assignments and one-on-one discussion sessions in addition to classroom-to-classroom skype sessions, students were paired up across locations to complete several assignments. each assignment required students to talk with their counterparts on various topics. while some of the questions aimed to produce specific discussion content, some encouraged students to explore the position of their international counterpart on various topics (see appendix for assignments). for instance, one assignment concerned a film8 that depicted honour killings in turkey. students were asked to compare gender-based violence data in turkey with the united states. the goal of this particular assignment was to show students that, although it is a major concern in turkey, the united states is not immune from gender-based violence. another assignment asked students to compare the politics and culture of the united states and turkey on a subject from a list provided to the students. unlike the previous one, this assignment did not aim to produce a specific discussion framework. instead, the goal was to help students explore various dimensions of the other culture through the eyes of their counterparts. therefore, each team had a slightly different discussion session.9 once each assignment was turned in, students were asked if there was anything that stood out to them during their one-on-one session. many were eager to share various parts of their conversation. post-assignment discussions helped students to understand diversity of responses. the project aimed to provide students with the opportunity to exchange ideas and opinions with international peers in small and large group discussion settings, using information technology. to achieve this, traditional learning methods were blended with technology-based approaches. while a large percentage of the reading materials overlapped, only the classroom-to-classroom skype sessions and team assignments were identical for both groups. each instructor used a different set of traditional teaching and evaluation techniques, such as lectures, exams, and research papers to complement and enhance the technology-based efforts. instructors made sure students were well-prepared for each discussion taking into consideration gaps of knowledge. for instance, prior to the discussion on secularism in turkey, ‘gettysburg college’ students had a lecture on secularism since they were not as familiar with the subject matter as their turkish counterparts. in the next section, we will present the survey results. participants in the survey administered at the beginning of spring 2011 semester, out of 25 ‘gettysburg college’ students, 13 identified as female and 12 as male; 14 students had studied abroad, but only 3 had been to turkey before. out of 19 ‘izmir university of economics’ students, 11 identified as female and 8 as male; 5 had studied abroad, and 3 of them had visited the united states. according to the survey y. akbaba and f. başkan 6 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 http://dx.doi.org/10.25304/rlt.v25.1915 administered at the beginning of fall 2011 semester, out of 20 ‘gettysburg college’ students, 7 identified as female and 13 were male; 8 students had studied abroad, but only 1 had been to turkey before. out of 19 ‘izmir university of economics’ students, 6 identified as female and 13 were male; 3 had studied abroad, and 2 of them had visited the united states. assessment in this section, we present some highlights from the survey and anecdotal data. a set of surveys were given throughout the semester10 in ‘gettysburg college’ and ‘izmir university of economics’. the survey responses were completely anonymous. students were informed that their participation was voluntary and their decision to participate and/or their responses would not have an impact on their course grade. surveys were given in ‘exam’ conditions, that is, students were not supposed to discuss their answers. in the last survey of spring 2011 course, students were asked three specific questions regarding the virtual learning experience. tables 1 and 2 display the distribution of students’ responses in both schools. although, in general, students in both institutions indicated that they benefited from the experience, ‘gettysburg college’ students’ responses were more positive. for instance, out of 24 ‘gettysburg college’ students, 22 ‘strongly’ or ‘somewhat’ agreed with the statement, ‘interaction with teammates in turkey helped me to learn a lot about turkish politics and culture’. among ‘izmir university of economics’ respondents only 9 out of 18 ‘strongly’ or ‘somewhat’ agreed with the same statement. interestingly, although ‘izmir university of economics’ students predominantly indicated that the project was worth their time, their responses to the more specific questions were not as enthusiastic as those of ‘gettysburg college’ students. in contrast, in the last survey of fall 2011, both groups rated the experience similarly. for example, out of 20 ‘gettysburg college’ respondents, 17 ‘strongly’ or ‘somewhat’ agreed with the statement, ‘interaction with teammates in turkey helped me to learn a lot about turkish politics and culture’. out of 14 ‘izmir university of economics’ students, 13 ‘strongly’ or ‘somewhat’ agreed with the same statement. (see table 3 and 4 for further details.) the surveys also included open-ended questions, some of which were repeated in the first and last survey to identify changes in students’ perceptions of the politics and culture of the other country over time.11 students were given the option of writing ‘i don’t know’ in the space provided. in the first survey of spring 2011, out of table 1. final survey, ‘gettysburg college’ spring 2011: views of the virtual learning environment. strongly agree somewhat agree neutral somewhat disagree strongly disagree interaction with teammates in turkey helped me to learn a lot about turkish politics and culture. 15 7 1 1 0 classroom-to-classroom skype sessions were productive and engaging. 11 10 1 2 0 i learned a lot from classroom-to-classroom skype sessions. 15 4 5 0 0 research in learning technology citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 7 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1915 25 ‘gettysburg college’ students, 6 to 11 students (depending on the question) responded with ‘i don’t know’. in contrast, in the last survey, out of 24 ‘gettysburg college’ students, only 0 to 2 responded with ‘i don’t know’. in the first survey of fall 2011, out of 20 ‘gettysburg college’ students, 3 to 7 students (depending on the question) responded with ‘i don’t know’. in contrast, in the last survey, out of 20 ‘gettysburg college’ students, only 4 responded with ‘i don’t know’ to only one of the seven open-ended questions. this suggests that more ‘gettysburg college’ students could articulate opinions on turkey by the end of the semester. what is more, in the final survey many of the ‘gettysburg college’ students wrote statements that provided insight from their interaction with students in turkey. for instance, the statement ‘i think people from turkey are ….’ was completed with responses like the following: ‘exceptional in their moderate perspectives and advance to secularism, particularly compared to their neighbors’; ‘interesting, unique, and that we share more similarities than we think i also think they are very smart and futuristic and hopeful, meaning they table 2. final survey, izmir university of economics spring 2011: views of the virtual learning environment. strongly agree somewhat agree neutral somewhat disagree strongly disagree missing interaction with teammates in the united states helped me to learn a lot about politics in the united states, american culture and us– turkey relations. 4 5 4 3 2 0 classroom-toclassroom skype sessions were productive and engaging. 7 4 5 2 0 i learned a lot from classroom-toclassroom skype sessions. 5 6 3 1 2 1 table 3. final survey, ‘gettysburg college’ fall 2011: views of the virtual learning environment. strongly agree somewhat agree neutral somewhat disagree strongly disagree interaction with teammates in turkey helped me to learn a lot about turkish politics and culture. 8 9 1 1 1 classroom-to-classroom skype sessions were productive and engaging. 11 8 1 0 0 i learned a lot from classroom-to-classroom skype sessions. 11 8 1 0 0 y. akbaba and f. başkan 8 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 http://dx.doi.org/10.25304/rlt.v25.1915 want the best for their country’; ‘people that we should be learning more about. indeed as global citizens people from every country should attempt to learn more about other cultures/peoples, dispelling myths and inaccuracies and broadening cultural horizons’; ‘you can’t generalize about people in any country’. this was a shift from the first survey conducted in the beginning of the semester. in the first survey, when the same statement was filled in with an opinion, these tended to be more generic, less informed phrases, such as ‘muslim’; ‘interesting’; ‘usually good cooks’. students were also asked to complete ‘i think turkish politics is…’ the following are examples of ‘gettysburg college’ students’ responses from the final survey: ‘a complicated example of a different form of democracy at work in a region where democracy normally fails’; ‘extremely interesting to observe. the interplay between secularists and conservatives will capture my attention for some time’; ‘very interesting! very unique – turkey is at a crossroads and i will be interested to keep watching’; ‘definitely intertwined with islam no matter how much they try to keep it out’; ‘drastically different between urban and rural areas’; ‘much more similar to life in the west than i previously thought’. again, this is a major change from the first survey conducted at the beginning of the semester where the same statement opening elicited completions like ‘confusing’; ‘likely an interesting subject’; ‘corrupt at times. religion has influence’. similar changes were observed across all seven open ended questions.12 turning now to the ‘izmir university of economics’ spring 2011 responses, in the first survey, 6 to 9 students (depending on the question) out of 19 wrote ‘i don’t know’. in the last survey, out of 18 students, 3 to 8 students (depending on the question) filled in the blanks with ‘i don’t know’. in the fall of 2011, responses suggest a stronger shift by the end of semester. in the first survey, 3 to 8 students (depending on the question) out of 19 wrote ‘i don’t know’. in the last survey, out of 18 students, 3 to 8 students (depending on the question) filled in the blanks with ‘i don’t know’. in fall of 2011, right after each classroom-to-classroom skype discussion session, instructors administered a brief survey to understand students’ take on each session.13 in each survey, students were asked the following: (1) what were the most intriguing/ interesting/eye-opening parts of the skype discussion today?, (2) (open-ended question.) i think this discussion changed the way i think about… one of the most important highlights is response rate. almost all of the students who were present in class that day completed the survey. students were not required to complete it, and surveys table 4. final survey, ‘izmir university of economics’ fall 2011: views of the virtual learning environment. strongly agree somewhat agree neutral somewhat disagree strongly disagree interaction with teammates in the united states helped me to learn a lot about politics in the united states, american culture and us– turkey relations. 4 9 1 0 0 classroom-to-classroom skype sessions were productive and engaging. 7 6 1 0 0 i learned a lot from classroom-to-classroom skype sessions. 7 6 1 0 0 research in learning technology citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 9 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1915 were administered at the tail end of each class when students were ready to leave. what is more, most responses included rich content. although we cannot know for sure whether students were eager to provide their input, we could speculate that skype sessions made an impression on students and that they had something to share regarding their experience. second aspect that stood out to instructors was parallels across ‘gettysburg college’ and ‘izmir university of economics’ students’ responses. they made reference to similar topics/point of views. for instance, in the first skype session, out of 19 ‘izmir university of economics’ students, 17 wrote about the discussion on whether turkey is a european or middle eastern country. out of 20 ‘gettysburg college’ students, 16 wrote about the discussion on whether turkey is a european or middle eastern country. although there were many other discussion topics covered that day, discussion on geopolitical identity of turkey stood out for students across the atlantic. in the third survey, most mentioned topics were kurdish problem and crosscultural component of the course for both groups. eleven students of ‘gettysburg college’ and 7 of ‘izmir university of economics’ made reference to the discussion on the kurdish question. second most discussed was how counterparts changed their opinion on turkish politics over the semester. 6 ‘gettysburg college’ and 8 ‘izmir university of economics’ students wrote about the discussion on ‘changing opinions’. in addition to collecting data through surveys, students shared their opinions and concerns regarding the project in different settings. informal conversations with students were consistent with survey results. some ‘izmir university of economics’ students claimed to have improved confidence in relation to their english language abilities. in different settings, students ranked the international peer-learning opportunity as a high-impact learning experience. one student even defined this experience to be ‘revolutionary’, in the sense that it was different from any other learning environment since they had shared the same course with students living in another continent. unlike survey data, personal accounts of instructors have limited explanatory power. these are not hard-facts, and it is difficult to reach to conclusions regarding students’ overall perception of the course. we do not present these to complement survey findings. however, we would like to unveil aspects of learning climate through anecdotal evidence. it had been a productive learning environment even when there were technical difficulties such as setting up virtual meeting times due to time difference. throughout the semester, most students had a positive outlook regarding the project. as instructors, we found this experience rewarding and uplifting due to enthusiastic learning climate created and embraced by all participants. in this project, virtual learning platforms were created to complement rather than replace traditional learning tools. our experience suggests that hybrid/blended models create a productive and engaging learning environment that enables students to use the various tools provided to them. anonymous surveys, anecdotal evidence and informal conversations with students support this claim. students mentioned that the combination of skype discussions with the traditional components of the course enhanced the learning experience. bridging the atlantic: challenges and opportunities of a bi-national teaching project for instructors like any partnership, international partnership to team teach a course requires committed project leaders who are willing to compromise and plan ahead. ironing out y. akbaba and f. başkan 10 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 http://dx.doi.org/10.25304/rlt.v25.1915 details of a collaborative project takes time. it also requires rapid exchange of ideas to find solutions to unexpected problems. prior to the initiation of the project, instructors had many face-to-face and virtual meetings over the course of 15 months in order to coordinate class meeting times, develop the syllabi, create assignments/ discussion questions and discuss pedagogical priorities/principles. maintaining contact prior to the project was vital for the success of the implementation stage. although the project had been demanding on the time of instructors, it provided valuable learning opportunities for all parties involved. higher education instructors tend to work on their syllabi and make decisions alone, unless they are team teaching. working on some aspects of a course with a colleague teaching at another institution in another country opens up new opportunities for pedagogical improvement. in a way, through the use of technology, both instructors and students get the benefit of bi-national team teaching without the financial cost to institutions. for students for students too, an international project like this one brings about challenges and opportunities. the most significant challenge for the students was the 7-h time difference. for classroom-to-classroom sessions, this was not an issue since the course schedules were arranged by the instructors accordingly beforehand. however, some students found it difficult to plan real-time meetings with their international partners due to the time difference. another challenge was the different academic calendars of the two institutions. ‘gettysburg college’ started and finished the semester approximately 3 weeks before ‘izmir university of economics’. moreover, breaks and/or holidays did not overlap. although these differences created logistic difficulties, they also provided learning opportunities for students about the geography and traditions of the other country, and a chance for students to learn to collaborate with someone who is living in another country. informal conversations with students suggested that students found communication across cultures to be a valuable experience given the global nature of contemporary work environments. they approached each component of the project with the understanding that it should benefit all students. for instance, while students in turkey were willing to spend time to explain political history of turkey, students in the united states encouraged their counterparts to express themselves in their second language. actually, although students in both countries were concerned about the language barrier prior to the project, by the end, this was not a major concern. the ‘izmir university of economics’ students’ english language proficiency was more than sufficient to accomplish the tasks given to them. there were minor issues related with technology. in particular, internet connections were not always sufficiently reliable. occasionally, there were dropped calls, and sometimes there were voice and video quality issues. however, these were negligible problems in that we never had to end a classroom-to-classroom session earlier than planned due to a technology-related issue. working in an increasingly interconnected world requires cross-cultural knowledge and skills. independent from the course topic, courses that require students to work across cultures provide the distinct experience and opportunity to develop or advance cross-cultural skills and knowledge. in addition to the obvious benefits of developing a peer-to-peer collaborative environment and being able to have discussions with students from another country on course material, this experience also research in learning technology citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 11 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1915 provided new ways for students to work together. for instance, some of the ‘gettysburg college’ students run their research topic ideas by their international counterparts. some teams had extensive conversations on turkish politics that went above and beyond the assignments required. students were excited to learn from each other and share what they learned from their international teammates with their classmates. we offered the same course, with minor changes one more time in fall of 2012. in other words, contemporary issues in turkish politics course, with its blended teaching component, was offered three times. we look forward to offering the course together again. also in fall of 2015, we used similar tools and assignment structure for another upper-level course. as explained above, as teachers, we found this experience to be rewarding and engaging for students. the quality and type of internet-based tools increased since the first time we offered the course in spring of 2011. these improvements provide a diverse set of virtual learning environments that can be utilised for different assignments. next time we offer the course, we plan to revise one-on-one assignments in a way that students can produce specific products, such as short documentaries, together. in addition to classroom-to-classroom discussion sessions, we plan to merge classes in other ways. for instance, we plan to develop small group assignments where students with similar interests can work together. we think an application like slack could provide the digital platform for such assignments. we also would like students to develop and work on a grand project together by using applications such as google drive. for such a project, students will be asked to come up with an overarching theme, for example, refugee crisis, and identify components of the project, for example, identifying local communities in need. smaller groups will be charged with specific tasks and by the end of the semester, students will evaluate the project together. conclusion currently, there is a variety of educational and communication tools available on the web. these tools provide great opportunities to expand, not only our classrooms, but students’ personal learning spaces as well. in an attempt to contribute to the discussion on teaching with technology, this paper has summarised the implementation of an international collaborative blended learning project using such tools, the challenges faced and the students’ responses to the course. as instructors, our intention with the project was to help students understand subject matter in an engaging way and expand their inter-cultural competencies and introduce a technology like skype in a bi-national learning environment in turkey. students in both institutions found the virtual learning environment to be rewarding. the overwhelming majority of students in both institutions suggested that the project provided high-impact learning opportunities for them. they also explicitly indicated that virtual learning environments enhanced their overall learning experience. although the technology-based components were welcomed by students in both institutions, it is important to note that instructors did not just rely on technology to teach the course. face-to-face instruction had been the primary teaching tool of the course that shaped the online component. in fact, anecdotal evidence from this study indicates that students benefit from a blend of digital and traditional learning y. akbaba and f. başkan 12 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 http://dx.doi.org/10.25304/rlt.v25.1915 environments. this is consistent with previous research that suggested blended learning could potentially bring in the best of two teaching methods together by personalizing learning experience (friedman 2015). however, it is crucial to take into account learning goals of individual courses and tailoring the right ‘blend’ for students, keeping an open mind to update the ratio as you go along. in this project, communication among instructors, and instructors and students guided the tailoring process. front-end planning process of this project required instructors to discuss pedagogical priorities, logistic challenges, nature and content of assignments, common course material and learning goals. like many educators, we ask ourselves: how do we engage ‘digital natives’ (prensky 2001) to courses we develop? what makes a course an intriguing, productive and rewarding experience for this generation? we think real learning requires commitment of all parties involved. our experience suggests that our students observed the work we put in to develop, implement and sustain digital platform of this course. the online component of the course relied on face-to-face conversations of instructors prior to the project. students valued the project’s goal that is developing bi-national connections. students responded well by putting in extra hours of work, and focusing on positive aspects of the experience. this created an overall engaging learning climate for students. we hope that our positive experience will be helpful to other educators who are planning to undertake similar projects. as mentioned before, we did not have a control group. this limited the assessment of the project with regard to which course learning outcomes are fulfilled through technology-based learning environment. the next step should be a comparative study to investigate whether the learning outcomes are fulfilled better with or without the technology support. more specifically, it would be enlightening to capture what type of learning outcomes are achieved through communication technologies like skype and what kinds of supportive material enhanced the learning experience. notes 1. faculty focus special report. blended and flipped: exploring new models for effective teaching and learning. july 2014. p. 2. 2. for more on moocs, see laurillard (2016). 3. for more on blended learning, see garrison and kanuka (2004); reasons, valdares, and slavkin (2005); zhao and breslow (2013); smith and brame 2016; mirriahi, alonzo and fox (2015); draffan and rainger (2006); simpson (2010); scott (2014); wilson and randall (2012). for more on shortcomings of hybrid models, see reasons et al. (2005). 4. faculty focus special report. blended and flipped: exploring new models for effective teaching and learning. july 2014. p. 2. 5. the course is offered by the department of political science in ‘gettysburg college’. at ‘izmir university of economics’, the department of political science and international relations offers the course. ‘gettysburg college’ and ‘izmir university of economics’ are both private higher education institutions. in both institutions, the course is offered as an undergraduate elective. although course materials were not identical, part of reading and visual materials overlapped. we modified some of the assignments in fall of 2011 in line with the feed back we received from students. 6. the focus of the project has not been about enhancing the it skills of students. rather, we used their existing skills to expand academic discussion beyond classroom and provide opportunities for international, peer-to-peer collaboration. 7. discussion sessions incorporated a common set of reading and/or visual material. 8. zülfü livaneli’s novel, bliss is adopted into a film. the film, also called bliss, is directed by abdullah oğuz. research in learning technology citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 13 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1915 9. although students received identical assignments, each instructor graded assignments separately using their own rubric. 10. while five surveys were administered in fall of 2011, two surveys were administered in spring of 2011. after reviewing students’ input, we decided to add a post-skype session survey in fall of 2011 which increased number of surveys given to students. 11. please note that since the surveys were anonymous, we could not match changes in responses at an individual level from t1 to t2. here, we only report aggregate changes. 12. it is important to note that there might be various reasons for this particular change. since we did not have a control group, we cannot suggest this was solely due to the virtual component of the course. in addition, this is not a rigorous content analysis but a simpler qualitative interpretation of the students’ responses. more importantly, the goal of this paper is not about revealing which learning tool helps us to achieve what specific objectives, but explaining how we used a particular technology-based teaching tool and how the project was received. 13. due to a problem, we could not administer the second post-skype session survey in turkey. references abdelmalak, m. m. m. (2015) ‘web 2.0 technologies and building online learning communities: students’ perspectives’, journal of asynchronous learning networks, vol. 19, no. 2, [online] avaialable at: http://www.editlib.org/p/157859 akkoyunlu, b. & soylu, m. y. (2008) ‘a study of student’s perceptions in a blended learning environment based on different learning styles’, educational technology and society, vol. 11, no. 1, pp. 183–193. andone, d., et al., (2007) ‘e-learning for digitally-minded students’, journal of interactive research, vol. 18, no. 1, pp. 41–53. beise, c. m. (2006) ‘global media: incorporating videocams and blogs in a global is management class’, information systems education journal, vol. 4, no. 74, [online] available at: http://isedj.org/4/74/ blau, i. & caspi, a. (2008) ‘do media richness and visual anonymity influence learning? a comparative study using skype™’, proceedings of the chais conference on instructional technologies research 2008: learning in the technological era, eds y. eshet-alkalai, a. caspi & n. geri, the open university of israel, raanana. bryant, t. (2006) ‘social software in academia’, educause quarterly, vol. 2, pp. 61–64. chan, a., frydenberg, m. & lee, m. j. w. (2007) ‘facilitating cross-cultural learning through collaborative skypecasting’, sigite’07 proceedings of the 8th acm sigite conference on information technology education, pp. 59–66. new york. coates, j. (2007) generational learning styles, lern books, river falls, wi. cronin, c., cochrane, t. & gordon, a. (2016) ‘nurturing global collaboration and networked learning in higher education’, research in learning technology, vol. 24. pp. 1–14. doi: 10. 3402/rlt.v24.26497 degagne, m. & wolk, r.m. (2007) ‘unwired: student use of technology in the ubiquitous computing world’, information systems education journal, vol. 5, no. 13, [online] available at: http://isedj.org/5/13/ draffan, e. a. & rainger, p. (2006) ‘a model for the identification of challenges to blended learning’, research in learning technology, vol. 14, no. 1, pp. 55–67. elia, a. (2006) ‘language learning in tandem via skype’, the reading matrix, vol. 6, no. 3, pp. 269–280. friedman, j. (2015) ‘decide between online, blended courses’, 4 mar., [online] available at: http://www.usnews.com/education/online-education/articles/2015/03/04/decide-betweenonline-blended-courses frydenberg, m. & andone, d. (2010) ‘two screens and an ocean: collaborating across continents and cultures with web-based tools’, information systems education journal, vol. 8, no. 55, pp. 1–12. frydenberg, m. & davi, a. (2006) ‘integrating liberal learning principles into the information technology classroom’, merlot journal of online learning and teaching, vol. 2, no. 3, pp. 195–204. y. akbaba and f. başkan 14 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 http://www.editlib.org/p/157859 http://isedj.org/4/74/ http://isedj.org/5/13/ http://www.usnews.com/education/online-education/articles/2015/03/04/decide-between-online-blended-courses http://www.usnews.com/education/online-education/articles/2015/03/04/decide-between-online-blended-courses http://dx.doi.org/10.25304/rlt.v25.1915 garlock, s. (2015) ‘is small beautiful? online education looks beyond the mooc’, harvard magazine, july–august. [online] available at: http://harvardmagazine.com/2015/07/issmall-beautiful garrison, d. r. & kanuka, h. (2004) ‘blended learning: uncovering its transformational potential in higher education’, the internet and higher education, vol. 7, no. 2, pp. 95–105. gasson, s. & agusto, d. e. (2008) ‘millennial students and technology use: implications for undergraduate education’, education in hci; hci in education, the hcic 2008 winter workshop, fraser, co. hersch, r. h. & merrow, j. (2005) declining by degrees: higher education at risk, palgrave macmillan press, new york. kirschner, l. l. (2015) ‘combining skype with blogging: a chance to stop reinforcement of stereotypes in intercultural exchanges?’, the eurocall review, vol. 23, no. 1, pp. 24–30. laurillard, d. (2016) ‘the educational problem that moocs could solve: professional development for teachers of disadvantaged students’, research in learning technology, vol. 24. pp. 1–18. doi:10.3402/rlt.v24.29369 liaw, m.-l. (2006) ‘e-learning and the development of intercultural competence’, language learning and technology, vol. 10, no. 3, pp. 49–64. liaw, m.-l. & johnson, r. j. (2001) ‘e-mail writing as a cross-cultural learning experience’, system, vol. 29, pp. 235–251. lukosch, s. (2007) ‘facilitating shared knowledge construction in colloborative learning’, informatica, vol. 31, pp. 167–174. macharaschwili, c. e. & coggin, l. s. (2013) ‘a skype-buddy model for blended learning’, journal of interactive learning research, vol. 24, no. 2, pp. 167–190. mirriahi, n., alonzo, d. & fox, b. (2015) ‘a blended learning framework for curriculum design and professional development’, research in learning technology, vol. 23. pp. 1–14. doi: 10. 3402/rlt.v23.28451 mueller, p. a. & oppenheimer, d. m. (2016) ‘the pen is mightier than the keyboard: advantages of longhand over laptop note taking’, psychological science, vol. 25, no. 6, pp. 1159–1168. palfrey, j. & gasser, u. (2008) born digital: understanding the first generation of digital natives, basic books, new york. prensky, m. (2001) ‘digital natives, digital immigrants’, on the horizon, vol. 9, no. 5, pp. 1–6. reasons, s.g.,valadares, k.& slavkin, m.(2005) ‘questioningthehybrid model: student outcomes in different course formats’, journal of asynchronous learning networks, vol. 9, no. 1, pp. 85–98. samsonov, p. & harris, s. (2010) ‘leading a horse to water: connecting home-bound students to their classrooms using skype technology’, in proceedings of edmedia: world conference on educational media and technology 2010, eds j. herrington & c. montgomerie, association for the advancement of computing in education (aace), chesapeake, va. pp. 3474–3480. scott, k. m. (2014) ‘taking over someone else’s e-learning design: challenges trigger change in e-learning beliefs and practices’, research in learning technology, vol. 22. pp. 1–15. doi: 10. 3402/rlt.v22.23362 simpson, a. (2010) ‘integrating technology with literacy: using teacher-guided collaborative online learning to encourage critical thinking’, research in learning technology, vol. 18, no. 2, pp. 119–131. simsek, n. (2005) ‘perceptions and opinions of educational technologists related to educational technology’, educational technology and society, vol. 8, no. 4, pp. 178–190. schmidt, s. (2005) ‘teaching at logan international airport’, journal of political science education, vol. 1, pp. 143–149. smith, b. & brame, c. (2016) blended learning and online learning, center for teaching, vanderbilt university, [online] available at: https://cft.vanderbilt.edu/guides-sub-pages/ blended-and-online-learning/#research tapscott, d. (2009) grown up digital, mcgraw-hill, new york. taylor, m. (2011) teaching generation next: methods and techniques for today’s learners, a collection of papers on self-study and institutional improvement, [online] available at: http://www.taylorprograms.com/images/techniques_article_2011.pdf tekinarslan, e. (2008) ‘attitudes of turkish distance learners toward internet-based learning: an investigation depending on demographical characteristics’, turkish online journal of distance education, vol. 9, no. 1, pp. 67–84. research in learning technology citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 15 (page number not for citation purpose) http://harvardmagazine.com/2015/07/is-small-beautiful http://harvardmagazine.com/2015/07/is-small-beautiful https://cft.vanderbilt.edu/guides-sub-pages/blended-and-online-learning/#research https://cft.vanderbilt.edu/guides-sub-pages/blended-and-online-learning/#research http://www.taylorprograms.com/images/techniques_article_2011.pdf http://dx.doi.org/10.25304/rlt.v25.1915 wilson, g. & randall, m. (2012) ‘the implementation and evaluations of a new learning space: a pilot study’, research in learning technology, vol. 20. pp. 1–17. doi: 10.3402/rlt.v20i0. 14431 zhao, y. & breslow, l. (2013) literature review on hybrid/blended learning, teaching and learning laboratory, [online] available at: https://tll.mit.edu/sites/default/files/library/ blended_learning_lit_reveiw.pdf zhu, j. (2010) ‘business students’ learning with online discussion forums: the case of a virtual classroom community’, information systems education journal, vol. 8, no. 22, [online] available at: http://isedj.org/8/22/ y. akbaba and f. başkan 16 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 https://tll.mit.edu/sites/default/files/library/blended_learning_lit_reveiw.pdf https://tll.mit.edu/sites/default/files/library/blended_learning_lit_reveiw.pdf http://isedj.org/8/22/ http://dx.doi.org/10.25304/rlt.v25.1915 appendix a sample of team assignments team assignment: bliss directions: please answer the following questions based on the film bliss. your answers should reflect a clear discussion with your teammate. your answers should also be informative, articulate and coherent. prewriting your answers, you need to do the following: • watch the film and jot down your ideas. • have a discussion on the film with your teammate. (1) explain one part that you could not possibly relate to in the film. this has to be something that you had a very hard time understanding. discuss with your teammate. also, ask your teammate whether there was part s/he could not relate to. (2) read blogs and film reviews to understand how the film bliss is received in the united states. what are the major themes highlighted? ask your teammate to help you understand how the film was received in turkey. please briefly compare reactions to the film in both countries. please provide examples (with web links). (3) have a discussion on gender-based violence in the united states and turkey with your teammate. please share statistics from reliable sources. discuss. team assignment: crossing the bridge: the sound of istanbul directions: please answer the following questions based on the film crossing the bridge: the sound of istanbul. your answers should reflect a clear discussion with your teammate. your answers should also be informative, articulate and coherent. prewriting your answers, you need to do the following: • watch the film and jot down your ideas. • have a discussion on the film with your teammate. (1) among the featured artists in the film, who is your teammate’s favourite? who is your favourite? please summarise your discussion as a team. (2) if you were making a film on ‘the sound of new york/los angeles/(or any other city in us)’ who would be the featured artists in your film? please share those names with your teammate. is s/he familiar with the artists? please summarise your discussion as a team. (3) the film depicts both the historic and recent expressions of musical creativity. ask your teammate if s/he listens either or both of them. also, discuss unique expressions of music in the united states. talk about for instance bluegrass, jazz, blues and country music. feel free to share music videos. please write a brief essay that reflects your discussion. research in learning technology citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 17 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1915 team assignment: research paper goal of this assignment is to give an opportunity to teammates to talk about their papers. your answers should reflect a clear discussion on potential venues of collaboration. (1) as a team, please exchange your ideas about your paper topic. topic of your paper might eventually change, but as of now, what would you like to write your paper on? does it intersect with your teammate’s topic? please summarise your discussion on this. (2) what are the possibilities of corroboration? please summarise your discussion. team assignment: aspects of culture directions: pick one of the following topics and discuss with your teammate. please write a summary of your discussion. please be articulate, informative and coherent. • being a college (university) student in the united states and turkey. (please discuss challenges and opportunities) • family (expectations, family traditions, mutual responsibilities, how much time do family members spend together, when do they get together, rules of the house, etc.) y. akbaba and f. başkan 18 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1915 http://dx.doi.org/10.25304/rlt.v25.1915 http://dx.doi.org/10.25304/rlt.v25.1915 (page number not for citation purpose) 1 *corresponding author. email: satu-maarit.frangou@ulapland.fi research in learning technology 2019. © 2019 s.-m. frangou et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 research in learning technology vol. 27, 2019 original research article the effect of writing modality on recollection in children and adolescents satu-maarit frangoua*, jan wikgrenb, sara sintonenc, leila kairaluomad and pekka vasarie acentre for media pedagogy, faculty of education, university of lapland, rovaniemi, finland; bcentre for interdisciplinary brain research, faculty of education and psychology, university of jyväskylä, jyväskylä, finland; cfaculty of education, university of helsinki, helsinki, finland; dfaculty of education, university of oulu, oulu, finland; efaculty of social sciences, university of lapland, rovaniemi, finland (received: 21 february 2019; revised: 27 september 2019; accepted: 7 october 2019; published: 28 10 2019) we set out to assess the extent to which writing modality affects recollection in children and adolescents. we examined 10to 11-year-old children’s (n = 63) and 16-year-old adolescents’ (n = 43) handwriting, keyboarding with a laptop computer and keyboarding with a touchscreen tablet computer or mobile phone in a within-subjects experimental design. participants were instructed to write down stories dictated to them in the three writing modalities. recollection of the stories was assessed using free recall of details in the stories. the results indicate that the writing modality affects recollection, handwriting leading to better recollection. however, currently, digital writing tools are inundating classrooms and workplaces around the globe, making their competent use a necessity in today’s world. for example, in finland, students are obligated to use a laptop in upper secondary education and in the national final examination. in light of the results, we highlight the importance of balancing the instruction and practice of different writing modalities. given the limitations of this study, we suggest conducting a larger-scale study and further research on the educational and cognitive implications of using and learning to write using multiple writing modalities. keywords: handwriting; keyboarding; writing instruction; recollection introduction writing is communication through letters, words and sentences (peterson and grimes 2018) produced with writing mediums or tools (haas 1996). writing is an essential competence for social communication in the 21st century; however, the demands of contemporary society continually raise the bar for digital competence requirements, which also extends to the different modalities of writing. keyboarding (i.e. typing) with different digital devices has taken precedence over writing with pen and paper (mangen and velay 2014), and literacy now encompasses much more than reading and producing linear text (kallionpää 2017). furthermore, digital tools are presently used in reading and writing instruction, and in finland, information and communication mailto:satu-maarit.frangou@ulapland.fi http://www.alt.ac.uk/� http://creativecommons.org/licenses/by/4.0/� http://dx.doi.org/10.25304/rlt.v27.2239� s.-m. frangou et al. 2 citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 (page number not for citation purpose) technologies (icts) are widely becoming part of the school infrastructure and a key component of literacy. the finnish national core curriculum for basic education was revised in 2014 and was implemented in 2016 in the teaching of all subjects (vahtivuori-hänninen et al. 2014). the new curriculum introduced multiliteracy skills to equip students with the necessary proficiencies to communicate and function in the digitised world, meaning also the competence to use different writing modalities. furthermore, multiliteracy refers not only to reading and writing, but also to the ability to create and interpret different types of information with and through different modes of communication and media (buckley-walker et al. 2017). in upper secondary education in finland and in the matriculation examination (the national examination on the upper secondary education syllabus), laptops are the main writing medium. this examination is currently undergoing the process of digitalisation (kupiainen, marjanen, and ouakrim-soivio 2018), and hence, students taking this examination can benefit only from a solid keyboarding competence. competence in writing with any modality could again have a connection to recollection. during the development of handwriting and keyboarding skills, improving basic transcription skills is essential, that is, to express oneself in writing, one must be proficient in the foundations of language, spelling and orthographic skills (berninger and swanson 1994). handwriting proficiency is fully developed before 15 years of age (accardo, genna, and borean 2013), but initially children are able to associate specific letters (characters) with their corresponding sound. they gradually become aware of the writing conventions of their own language while simultaneously developing finger dexterity (dinehart 2015). similarly, as keyboard proficiency develops, so does spelling accuracy and keyboarding speed with less focus on finding the correct keys (rønneberg and torrance 2019). this is also consistent with the keyboarding research conducted by christensen (2004), in which the texts produced by eighth and ninth graders via keyboarding improved considerably after keyboarding practice. moreover, research has shown that when children learn keyboarding, they benefit from a new kind of motivation to express themselves, as their ability to edit improves and they no longer keyboard with an improvised, frustrating and time-consuming hunt-and-peck system (rogers et al. 2003). as berninger and swanson (1994) and klein (1999) put it, a competent writer by hand can concentrate on the writing topic at a higher level of planning while reviewing the produced text and does not need to concentrate on the graphomotor execution process of writing, which could also be the case in proficient keyboarding. more cognitive resources can be used to process the content, planning and monitoring of writing (berninger and swanson 1994; rønneberg and torrance 2019; yeganeh doost et al. 2017). hence, in this study, we posited that the writing modality can be considered an encoding medium that can leave a significant memory trace. we asked: do different writing modalities have differing influences on children’s and adolescents’ recollection? handwriting and keyboarding handwriting can be conceptualised as a skill that unites perceptual and motor skills. the component of perception is associated with the perceived letter shape, while the motor component deals with letter production (vinter and chartrel 2010). http://dx.doi.org/10.25304/rlt.v27.2239� research in learning technology citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 3 (page number not for citation purpose) hence, the development and coordination of fine motor skills, neuromotor processes and multiple cognitive processes are essential in handwriting (dinehart 2015). at the same time, erthal (1998) suggested that a good age to learn keyboarding is approximately eight or nine, as children at this age have the necessary competencies, including the ability to read, and the fine motor skills and hand–eye coordination needed to write in the visual space of the screen and in the motor space of the keyboard (erthal 1998; mangen and velay 2010). from theoretical and empirical research, several differences between keyboarding and handwriting are apparent, although both skills require practice to progress from novice to proficient levels. yamaguchi and logan (2014) studied the keyboarding process and identified three major forms of linguistic-level associations during competent automated keyboarding – the association between words and specific letters, the association between letters and specific keys and the association between specific finger and the correct keys to strike. logan and crump’s (2011) cognitive model of typing dealt more with lower level cognitive functions, such as the location retrieval and graphomotor execution of keystrokes, than with higher level planning or reviewing of text, which is common in handwriting studies (berninger and swanson 1994; flower and hayes 1981). cognitive process theory, introduced by flower and hayes (1981), clarifies the cognitive process model introduced by the authors a year earlier (hayes and flower 1980). the theory proposes that handwriting is a recursive action, which has three components. firstly, in the writing process, writers are continually and simultaneously planning, translating and revising as they compose a text. secondly, the writer’s long-term memory stores various types of knowledge that can affect the outcome of the text, such as previous knowledge about the topic or knowledge about the reading audience. however, the ideas created in long-term memory are often disorganised and therefore require cognitive revision and structure before being written down. thirdly, the task environment refers to relevant and related text produced thus far and how such text affects the writer in terms of framing the topic, goals and audience (flower and hayes 1981). empirical studies concerning keyboards have found keyboarding to be more productive compared to handwriting not only in terms of speed and legibility (rogers and case-smith 2002) but also due to its features for organised notetaking, which enhance recall (bui, myerson, and hale 2013). keyboarding has also yielded benefits for poor (in speed or legibility) or disabled handwriters, which improves their written output (rogers and case-smith 2002; weigelt marom and weintraub 2015). keyboarding on touchscreen devices follows the same principles as those for a conventional or laptop keyboard; however, the keys cannot be separated by touch, which can cause keyboarding accuracy to decline and slow down the writing process (kim et al. 2014) by shifting the focus from the text to the keyboard. however, tablet computers with touchscreens were found to be effective instructional tools for children with specific learning disabilities, improving their handwriting, spelling and composing competences (berninger et al. 2015). in empirical research, handwriting has been reported to have multiple benefits, from improving orthography and structuring to enhancing composing and creative writing skills (berninger and swanson 1994; christensen 2005). however, studies regarding the connection of keyboarding and handwriting to recollection are scarce and mainly concentrate on the accuracy of the letters. research on children’s letter, or character, recognition, including short-term and long-term retention, was conducted http://dx.doi.org/10.25304/rlt.v27.2239� s.-m. frangou et al. 4 citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 (page number not for citation purpose) by longcamp, zerbato-poudou, and velay (2005) and longcamp et al. (2006). similar research was also conducted on adults by longcamp et al. (2008). these studies suggest that handwriting may enhance the recollection of letters, and in the case of children, this effect would become more pronounced with age. concerning adults’ word recollection, research by mangen et al. (2015) with female university students reported that handwriting produced better recollection scores compared to keyboarding on computers and touchscreen devices; these findings were corroborated by frangou et al. (2018) for handwritten text recall. furthermore, a study by mueller and oppenheimer (2014) compared university students’ recollection of handwritten and typed lecture notes, and found that handwritten lecture notes composed in one’s own words (not verbatim) promoted a deeper understanding of the subject and its later recollection. the current study context in the current study, we compared three writing modalities – handwriting, keyboarding on a laptop and keyboarding on the touchscreen keyboard of an ipad (10to 11-year-olds) and a mobile phone (16-year-olds) – and the recollection of short logical  stories. generally, recollection is understood as the retrieval of stored information from memory system (tulving 1983). however, recollection in the current context refers to episodic memory because the written stories were logically composed and ordered (tulving 1983). a story is recalled through story schemas, as put forward by schema theory (mandler 2014). this means that a story generally has a structure, with setting, theme, plot (events) and resolution, which can be recalled in logical order (mandler 2014). this study tested long-term retention, which is the broad aim of formal teaching, and exclusively measured the content knowledge of the written verbatim content of texts that needed no prior knowledge. we conducted two experiments, one with 10to 11-year-old finnish children (n = 63) and another with 16-year-old finnish adolescents (n = 43), to examine their recollection after handwriting, laptop keyboarding and touchscreen keyboarding (ipads for children and mobile phones for adolescents). participants aged 10 or 11 were chosen because, at this age, children’s writing competence is transitioning from learning to write to writing to learn (sedita 2013). the 16-year-old participants were chosen because they were ninth graders, that is, at the very end of their basic education in finland when students are assumed to have all the competences needed to communicate with digital devices and to function in contemporary society. methodology participant recruitment and data collection were conducted in accordance with the standards and guidelines of the university of lapland research ethics committee and the guidelines of the finnish national board on research integrity (tenk). the municipality of rovaniemi school board gave permission to carry out the research in the schools, and prior to task performance demographic information of the participants was collected and consent forms sent to be signed by the participants and the participants’ legal guardians. the first author was responsible for the data collection in the schools’ premises that were familiar to the participants. http://dx.doi.org/10.25304/rlt.v27.2239� research in learning technology citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 5 (page number not for citation purpose) pilot experiment with 10and 11-year-old children participants a pilot experiment was conducted with 10–11-year-old children. from the participants (n = 29), 19 were 10-year-old children (11f, 8m) and 10 were 11-year-old children (4f, 6m). four participants were left-handed and 25 were right-handed, and all were proficient finnish speakers. the participants began keyboarding at the age of 5.62 (sd = 1.45) and used 5.00 (sd = 3.21) fingers for keyboarding. touchscreen keyboarding began a bit later, at the age of 6.34 (sd = 1.32). similar to frangou et al. (2018), the pilot experiment was conducted using three stories. upon reflection of the day the participants wrote the texts, it became evident that the texts were too demanding in terms of content and length. given the limitations of the pilot experiment in terms of scope, inconclusive results and a demanding assessment tool, a second study was designed. therefore, experiment 1 replicated the pilot study, but with adjusted parameters. experiment 1 with 10and 11-year-old children participants participants (n = 63) included 31 10-year-old children (17f, 14m) and 32 11-year-old children (20f, 12m) from one school who were all proficient finnish speakers. four participants were left-handed and 59 were right-handed. the age at which keyboarding began was 6.68 (sd = 1.50), while the corresponding age for keyboarding on a touchscreen device was 6.56 (sd = 1.43). the children used 4.89 fingers for keyboarding (sd = 2.68). material the pilot experiment material had been too demanding with two stories from the wechsler memory scale (wms-r) logical memory subtest (wechsler 1987) and a third story that had been created to be similar to the other two (story c, appendix a). in experiment 1, the stories were revised and shortened to suit this age group. all three stories were 40 words long with a logical structure that required no prior knowledge. for score-recording purposes, a 20-item list of details, persons and occurrences was created from each (story a, b & c, appendix b). for the handwritten test, a pen and paper were provided. for laptop keyboarding, asus chromebooks were used; for touchscreen keyboarding, ipads (model md510ks/a) were used. the quantitative data arising from the memory tests were analysed using descriptive statistics (statistical package for the social sciences [spss] 24 software). research design and procedure an experimental within-subjects research design was used; therefore, all participants used handwriting, a laptop keyboard and an ipad touchscreen keyboard to write three different stories – one for each writing medium. at the beginning of the session, the participants were instructed to write down everything that was dictated to them and told that they would be asked to recall information from their written stories 1 week http://dx.doi.org/10.25304/rlt.v27.2239� s.-m. frangou et al. 6 citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 (page number not for citation purpose) later. as the three stories were dictated in random order, the children wrote them verbatim with a pen, laptop keyboard and touchscreen keyboard one after the other in small groups. reading pace was adjusted according to the children’s ability to write. after each writing task, the story was read aloud once again so that the children could determine whether their written text was correct. the writing modality sequence was also randomised to prevent the same method from being used first or last. one week later, the participants were individually asked to orally recall everything they remembered about their written stories, in a free-recall condition with as much time as needed. for score-recording, a 20-item list of details was used in which one point was earned for each recalled detail (a maximum of 20 points). one cue word concerning the story’s theme (e.g., bear) from the item list was designed to be used if a child could not recall anything at all. experiment 1 results the mean value for recollection scores after handwriting was 9.83 (sd = 3.97); the mean value for recollection scores after keyboarding was 8.83 (sd = 4.00); and for touchscreen keyboarding, this value was 8.98 (sd = 3.96). interestingly, particularly in handwriting, a difference was found between the age groups; the 11-year-old participants’ recollection score value in handwriting was 11.56 (sd = 3.35), while for the 10-year-olds it was 8.03 (sd = 3.79). firstly, we assessed the effect of the writing modality on time spent writing ( figure  1a). anova with repeated measures for each modality as a within-subjects factor and age (10 and 11) and gender as between-subjects factors revealed a significant writing mode – age interaction [f(2,118) = 63.79, p < 0.001], the main effect of writing modality on time spent writing [f(2,118) = 18.36, p < 0.001] and the main effect of age [f(1,59) = 399.33, p < 0.001]. further paired samples t-tests on both age groups separately indicated that in 10-year-old children, handwriting is more time-consuming than tablet writing [t(30) = 23.87, p < 0.001] or keyboard writing [t(30) = 3.92, p < 0.001]. it was also found that the 10-year-olds spent less time a b figure 1. results from experiment 1 with 10–11-year-old participants. http://dx.doi.org/10.25304/rlt.v27.2239� research in learning technology citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 7 (page number not for citation purpose) writing with a tablet than with a keyboard [t(30) = 4.87, p < 0.001]. the pattern for 11-year-old participants was somewhat different. for them, writing with a tablet was the most time-consuming method, as indicated by significant differences from both handwriting [t(31) = 14.86, p < 0.001] and keyboard writing [t(31) = 4.76, p < 0.001]. the findings indicate that younger pupils spend significantly more time writing, especially handwriting. a similar analysis run on the recollection scores (figure 1b) yielded a significant main effect of age [f(1,59) = 9.24, p < 0.01]. thus, the recollection scores were analysed for both age groups separately. a significant main effect of writing mode was found for the 11-year-old children [f(2, 62) = 4.28, p < 0.05)]; however, no such effect was found for the 10-year-olds [f(2, 60) = .001, p = 0.99]. to further analyse the differences among the writing modalities, paired samples t-tests were run between the writing modalities. the recollection scores for the 11-year-old children’s handwriting were better than either of the other modalities – keyboard writing [t(31) = 3.32, p < 0.01] and touch screen keyboarding [t(31) = 2.15, p < 0.05]. experiment 2 with 16-year-old adolescents participants the participants of experiment 2 included 43 16-year-old adolescents (21m, 22f), representing all the students of this age group at one school. six of the participants were left-handed and 37 right-handed, and all were proficient finnish speakers. the participants started using keyboards at the age of 7.98 (sd = 2.02) and used 7.91 (sd = 1.45) fingers for keyboarding; they began using touchscreen keyboards at the age of 10.60 (sd = 1.14). material the material for experiment 2 was identical to that for the pilot experiment with two stories from the wms-r logical memory subtest (wechsler 1987), which is designed to test episodic memory, and a similar one that was created for this study (story c, appendix a). the three stories were each around 60 words long. for score-recording purposes, there was a 25-item list of details, persons and occurrences from each story (two from wms-r and one created for this study). for each recalled detail, one point was earned, meaning that one could score a maximum of 25 points. data analysis methods were identical to those used in experiment 1. for the handwritten test, a pen and paper were used, whereas the participants’ own laptops were used for keyboarding and their own mobile phones were used for touchscreen keyboarding. on the mobile phones, only thumbs were used for writing. research design and procedure the research design and procedure were identical to those used in experiment 1, with the exception that the 25-item lists were used for score recording and no cue words were given. http://dx.doi.org/10.25304/rlt.v27.2239� s.-m. frangou et al. 8 citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 (page number not for citation purpose) experiment 2 results the participants’ mean score for recollecting handwritten texts after 1 week was 6.46 (sd = 3.82), for laptop-keyboarded texts 4.74 (sd = 2.43) and for touchscreen keyboarded texts 5.44 (sd = 4.13). anova with repeated measures revealed a significant main effect of writing mode on time spent writing [f(2,82) = 77.39, p < 0.001]. time spent writing is depicted in figure 2a. paired samples t-tests indicated that handwriting was slower compared to both the mobile phone touchscreen [t(42) = 18.52, p < 0.001] and keyboard [t(42) = 9.17, p < 0.001]. curiously, adolescents wrote faster with a phone, using only their thumbs, than with a keyboard [t(42) = 3.68, p < 0.01]. a similar kind of repeated measures anova was run on recollection scores ( figure 2b). we found a significant main effect of writing mode [f(2,82) = 4.19, p < 0.05]. the anova with repeated measures tests of within-subjects effects revealed the significant main effect of writing modality on recollection [f(2,84) = 4.24, p = 0.018]. the subsequent pairwise comparisons revealed that handwriting was significantly better recalled than laptop-keyboarded texts (p = 0.011) with bonferroni adjustment. recall scores were subsequently analysed with pairwise comparisons to determine differences between writing methods. the results indicated that handwritten material is better recalled than texts written on a laptop keyboard [t(42) = 3.09, p < 0.01]. a  comparison between handwriting and mobile phone writing yielded a borderline significant difference in favour of handwriting [t(42) = 1.76, p = 0.085]. discussion the interrelation of learning and memory (carroll 1993) emerged in this study through the degree of recollection by individuals after stories were written down using different writing modalities. stories are generally not a subject that children are tasked with memorising at school; yet, stories represent a good source for writing without the need of prior knowledge and, in this way, equalise the starting point among children. our research provides valuable insight into writing modalities and their influence on recollection. a key finding is that handwriting resulted in better recollection than laptop keyboarding and touch screen keyboarding among 11-year-old children and 16-year-old adolescents. however, children aged 10 recalled almost equally well after a b figure 2. results from experiment 2 with 16-year-old participants. http://dx.doi.org/10.25304/rlt.v27.2239� research in learning technology citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 9 (page number not for citation purpose) writing with all three writing modalities. this leads to the question of the keyboarding proficiency of the 11and 16-year-old participants, which would decrease the amount of attention needed to locate and strike the keys (rønneberg and torrance 2019) and would thereby enable the participants to concentrate more on the topic (berninger and swanson 1994; klein 1999) and recall more of the text. furthermore, for 11-year-olds, a difference was found between the recollection scores of handwriting and touchscreen keyboarding; however, for 16-year-olds the difference between handwriting and touchscreen was not significant. these results suggest that handwriting is likely to be the most proficient modality for this age group, with touchscreen keyboarding not far behind, the fast speed of keyboarding on mobile phones suggesting this as well. furthermore, the 10-year-olds spent less time writing with a tablet than with a keyboard. this could demonstrate how familiar these age groups are with touchscreen devices, a finding consistent with those of mangen et al.’s (2015) study, in which a positive correlation was found between recollecting words that had been written with a touchscreen device and the number of years spent using touchscreen devices. the study design used in this research, that is, comparing three different writing modalities with similar texts for all three, seemed to work well. the texts were suitable for adolescents; however, the parameters for 10to 11-year-old children were adjusted after the pilot experiment. the adjusted parameters worked well for this age group. study limitations this research has some limitations that should be acknowledged. firstly, to our knowledge, a test designed to measure recollection after writing with different modalities does not exist, and therefore, there were no empirical studies with which to compare this study. furthermore, large-scale experimental research on different age groups would produce more generalisable results. therefore, the confidence interval for means was calculated from the results of experiment 2 because the number of participants was 43, and the distribution of gender was close to equal. the confidence interval for means was calculated to determine an adequate number of participants for test standardisation. we are 95% confident that the number of participants for each age group needed to standardise the test is 262. were it possible to conduct this study on a larger scale, a control group for comparing the outcomes would have certainly yielded clearer results and would have helped make the appropriate adjustments to the texts. lastly, the test used for this research generates only quantitative data and does not consider any possible individual differences, such as mnemonic abilities, experiences with different modalities or personal preferences; neither does it consider the writing participants’ backgrounds or contexts. conclusion this article addressed the question: do different writing modalities have differing influences on children’s and adolescents’ recollection? the question is highly relevant in the era of widening digital, multimodal text worlds. using experimental within-subjects research design, the current study investigated 106 finnish children’s and adolescents’ recollection after handwriting and keyboarding on laptops and touchscreen keyboards and found that different writing modalities influence recollection; http://dx.doi.org/10.25304/rlt.v27.2239� s.-m. frangou et al. 10 citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 (page number not for citation purpose) handwriting was the best writing modality among 11-year-old children and 16-yearold adolescents. the increase and variety of new writing modalities present challenges for the education system. to improve the writing skills of students, systematic keyboarding instruction, supplementary instruction in areas in need of more support and standardised and uncomplicated tests for monitoring the development of writing skills should be developed. as multiliteracy is viewed as a transversal competence within the framework of the new curriculum in finland, good foundations, both in handwriting and keyboarding, are invaluable for future. it is evident that further empirical research on the educational and cognitive implications is needed in writing research to understand the effects of technology together with modality on recollection in different age groups. the study design and pilot test developed during this research (appendix b) may be a useful starting point for others who are considering developing new methods to research writing modalities; however, for other age groups the parameters need to be adjusted again. at the same time, more research is needed on the multiliteracy competences and adaptivity to new technologies that are expected in contemporary society. furthermore, new writing research instruments are needed to evaluate students’ diversifying literacy skills and their influence on recollection to provide educators with valuable information about the students’ abilities and potential need for targeted interventions. references accardo, a. p., genna, m. & borean, m. (2013) ‘development, maturation and learning influence on handwriting kinematics’, human movement science, vol. 32, pp. 136–146. doi: 10.1016/j.humov.2012.10.004 berninger, v. w., et al., (1994) ‘developmental skills related to writing and reading acquisition in the intermediate grades’, reading and writing, vol. 6, no. 2, pp. 161–196. doi: 10.1007/ bf01026911 berninger, v. w., et al., (2015) ‘computer instruction in handwriting, spelling, and composing for students with specific learning disabilities in grades 4–9’, computers & education, vol. 81, pp. 154–168. doi: 10.1016/j.compedu.2014.10.005 berninger, v. w. & swanson, h. l. (1994) ‘modifying hayes and flower’s model of skilled writing to explain beginning and developing writing’, in children’s writing: toward a process theory of the development of skilled writing, eds e. c. butterfield & j. carlson, jai press, london, pp. 57–81. buckley-walker, k., et al., (2017) ‘evaluating the validity of the online multiliteracy assessment tool’, australian journal of education, vol. 61, no. 3, pp. 305–327. doi: 10.1177/0004944117729056 bui, d. c., myerson, j. & hale, s. (2013) ‘note-taking with computers: exploring alternative strategies for improved recall’, journal of educational psychology, vol. 105, no. 2, pp. 299– 309. doi: 10.1037/a0030367 carroll, j. b. (1993) human cognitive abilities: a survey of factor-analytical studies, cambridge university press, new york. christensen, c. a. (2004) ‘relationship between orthographic-motor integration and computer use for the production of creative and well-structured written text’, british journal of educational psychology, vol. 74, pp. 551–564. doi: 10.1348/0007099042376373 christensen, c. a. (2005) ‘the role of orthographic–motor integration in the production of creative and well-structured written text for students in secondary school’, educational psychology, vol. 25, no. 5, pp. 441–453. doi: 10.1080/01443410500042076 http://dx.doi.org/10.25304/rlt.v27.2239� research in learning technology citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 11 (page number not for citation purpose) dinehart, l. (2015) ‘handwriting in early childhood education: current research and future implications’, journal of early childhood literacy, vol. 15, no. 1, pp. 97–118. doi: 10.1177/1468798414522825 erthal, m. (1998) ‘who should teach keyboarding and when should it be taught?’, business education forum, vol. 53, no. 1, pp. 36–37. flower, l. & hayes, j. (1981) ‘a cognitive process theory of writing’, college composition and communication, vol. 32, no. 4, pp. 365–387. doi:10.2307/356600 frangou, s-m., et al., (2018) ‘can you put your finger on it? the effects of writing modality on finnish students’ recollection’, journal of writing systems research, vol. 10, no. 2, 82–94. doi: 10.1080/17586801.2018.1536015 haas, c. (1996) writing technology studies on the materiality of literacy. hillsdale, nj: lawrence erlbaum associates. hayes, j. r. & flower, l. s. (1980) ‘identifying the organization of writing processes’, in cognitive processes in writing: an interdisciplinary approach, eds l. gregg & e. steinberg, erlbaum, hillsdale, nj, pp. 3–30. kallionpää, o. (2017) ‘uuden kirjoittamisen opetus: osallistavaa luovuutta verkossa [teaching new writing: inclusive creativity online]’, scriptum: creative writing research journal, vol. 4, no. 1, pp. 1–232. kim, j. h., et al., (2014) ‘differences in typing forces, muscle activity, comfort, and typing performance among virtual, notebook, and desktop keyboards’, applied ergonomics, vol. 45, no. 6, pp. 1406–1413. doi: 10.1016/j.apergo.2014.04.001 klein, p. d. (1999) ‘reopening inquiry into cognitive processes in writing-to-learn’, educational psychology review, vol. 11, pp. 203–270. doi:10.1023/a:1021913217147 kupiainen, s., marjanen, j. & ouakrim-soivio, n. (2018) ‘ylioppilas valintojen pyörteissä. lukio-opinnot, ylioppilastutkinto ja korkeakoulujen opiskelijavalinta [student in the whirlwind choices. high school studies, matriculation examination and college student admission]’, suomen ainedidaktisen tutkimusseuran julkaisuja. ainedidaktisia tutkimuksia 14, suomen ainedidaktinen tutkimusseura, turenki. logan, g. d. & crump, m. j. (2011) ‘hierarchical control of cognitive processes: the case for skilled typewriting’, psychology of learning and motivation, vol. 54, pp. 1–27. doi: 10.1016/ b978-0-12-385527-5.00001-2 longcamp, m., zerbato-poudou, m.-t. & velay, j.-l. (2005) ‘the influence of writing practice on letter recognition in preschool children: a comparison between handwriting and typing’, acta psychologica, vol. 119, pp. 67–79. doi:10.1016/j.actpsy.2004.10.019 longcamp, m., et al., (2008) ‘learning through handor typewriting influences visual recognition of new graphic shapes: behavioral and functional imaging evidence’, journal of cognitive neuroscience, vol. 20, no. 5, pp. 802–815. doi:10.1162/jocn.2008.20504 longcamp, m., et al., (2006) ‘remembering the orientation of newly learned characters depends on the associated writing knowledge: a comparison between handwriting and typing’, human movement science, vol. 25, no. 4–5, pp. 646–656. doi:10.1016/j.humov.2006.07.007 mandler, j. m. (2014) stories, scripts, and scenes: aspects of schema theory, psychology press, new york. mangen, a., et al., (2015) ‘handwriting versus keyboard writing: effect on word recall’, journal of writing research, vol. 7, no. 2. doi: 10.17239/jowr-2015.07.02.01 mangen, a. & velay, j.-l. (2010) ’digitizing literacy: reflections on the haptics of writing’, in advances in haptics, ed m. h. zadeh, intechopen, rijeka, croatia, pp. 385–402. doi: 10.5772/8710. mangen, a. & velay, j.-l. (2014) cognitive implications of new media’, in m.-l. ryan, l. emerson & b.j. robertson (eds.) the johns hopkins guide to digital media. baltimore: johns hopkins university press, pp. 72–77. mueller, p. a. & oppenheimer, d. m. (2014) ‘the pen is mightier than the keyboard: advantages of longhand over laptop note taking’, psychological science, vol. 25, no. 6, pp. 1159–1168. doi:10.1177/0956797614524581 http://dx.doi.org/10.25304/rlt.v27.2239� s.-m. frangou et al. 12 citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 (page number not for citation purpose) peterson, s. s. & grimes, a. (2018) ‘authentic writing in children’s lives outside school’, early childhood education, vol. 45, no. 1, pp. 20–25. retrieved from https://www.ecec-ata.com/ journal-early-childhood-education.html rogers, j. & case-smith, j. (2002) ‘relationships between handwriting and keyboarding performance of sixth-grade students’, american journal of occupational therapy, vol. 56, no. 1, pp. 34–39. doi: 10.5014/ajot.56.1.34 rogers, h., et al., (2003) the status of elementary keyboarding: a longitudinal study, university of wisconsin, whitewater, wi. rønneberg, v. & torrance, m. (2019) ‘cognitive predictors of shallow-orthography spelling speed and accuracy in 6th grade children’, reading and writing, vol. 32, no. 1, pp. 197–216. doi: 10.1007/s11145-017-9751-3 sedita, j. (2013) ‘learning to write and writing to learn’ in fundamentals of literacy instruction and assessment, ed m. c. hougen, paul h. brookes, baltimore, md, pp. 6–12. tulving, e. (1983) elements of episodic memory, clarendon press, oxford. vahtivuori-hänninen, s., et al., (2014) ‘a new finnish national core curriculum for basic education (2014) and technology as an integrated tool for learning’ in finnish innovations and technologies in schools, eds h. niemi, j. multisilta, l. lipponen & m. vivitsou, sense publishers, rotterdam, pp. 21–32. vinter, a. & chartrel, e. (2010) ‘effects of different types of learning on handwriting movements in young children’, learning and instruction, vol. 20, pp. 476–486. doi:10.1016/j. learninstruc.2009.07.001 wechsler, d. (1987) wms-r. käsikirja [handbook]. psykologien kustannus oy, helsinki. weigelt marom, h. & weintraub, n. (2015) ‘the effect of a touch-typing program on keyboarding skills of higher education students with and without learning disabilities’, research in development disabilities, vol. 47, pp. 208–217. doi: 10.1016/j.ridd.2015.09.014 yamaguchi, m. & logan, g. d. (2014) ‘pushing typists back on the learning curve: revealing chunking in skilled typewriting’, journal of experimental psychology: human perception and performance, vol. 40, no. 2, p. 592–612. doi: 10.1037/a0033809 yeganeh doost, m., et al., (2017) ‘two processes in early bimanual motor skill learning’, frontiers in human neuroscience, vol. 11, no. 618, pp. 1–13. doi: 10.3389/fnhum.2017.00618 http://dx.doi.org/10.25304/rlt.v27.2239� https://www.ecec-ata.com/journal-early-childhood-education.html https://www.ecec-ata.com/journal-early-childhood-education.html research in learning technology citation: research in learning technology 2019, 27: 2239 http://dx.doi.org/10.25304/rlt.v27.2239 13 (page number not for citation purpose) appendix a story c, pilot experiment and experiment 2 (translation) large beasts of prey are familiar to the border guards of the eastern division. border guard jacob smith has seen a bear three times during his career. the previous case happened 2 years ago when he was patrolling with his dog one july morning. they passed a valley when they heard a sound. a bear had touched the barbed wire and came across the border. the man decided to whistle, and the bear turned right back. the border guards of the lake district, for example, meet bears about five times a year. each of these items earned one point if recalled: large beasts of prey, familiar, border guards, eastern division, jacob, smith, bear, three times, career, previous case, 2 years ago, patrolling, dog, july morning, valley, heard a sound, touched, barbed wire, came across, border, whistle, turned right back, lake district, five times, a year. appendix b story a, experiment 1 (translation) a student named leevi from ranua found a small and strange animal yesterday on the way to school. he drove on a moped to take it to a zoo where the vet recognised the animal to be a rare reptile. the beautiful reptile must certainly have escaped from a pet shop because it would not have been able to survive in northern nature. here, it needs a terrarium and a heat lamp. each of these items earned one point if recalled: student, leevi, ranua, small, animal, yesterday, on the way to school, moped, zoo, vet, recognised, rare, reptile, beautiful, escaped, pet shop, northern nature, terrarium, heat lamp. story b, experiment 1 (translation) during the weekend, four men went to norway on a fishing trip. they did not stay in a cottage, but in a caravan. on sunday, the sun was shining and the fish were biting well. then the boat’s cap disappeared and the boat was filled with water. some nearby japanese tourists came to help and the men started their trip home, wet. each of these items earned one point if recalled: weekend, four, men, norway, fishing trip, cottage, caravan, sunday, sun, fish, biting well, boat’s, cap/cork, disappeared, filled, japanese, tourists, help, trip home, wet. story c, experiment 1 (translation) bears are familiar to old emma who lives in the woods. last summer emma saw a bear three times. last time it happened when she went to pick berries with the dog. they walked on a forest path when she heard a sound. the bear was stealing honey. the bear was startled by the barking of the dog and slowly walked away. each of these items earned one point if recalled: bears, old, emma, lives, in the woods, last (1 year ago), summer, three, times, last time, pick berries, dog, forest, path, sound, stealing honey, startled, barking, slowly walked, away. http://dx.doi.org/10.25304/rlt.v27.2239� does interactivity require multimedia? the case of saki michael horwood eurotech training, oakfield road, east wittering, chichester, west sussex po20 8rp abstract saki is a self-adaptive touch-typing tutor with a pedigree dating back to the mid-1950s. even in its most recent form it eschews the temptation to present itself with the trimmings now commonly associated with microcomputer products. this paper argues that while the absence of such features may be a limiting factor in the commercial success of the program, saki is nevertheless a prime example of the way in which a computer can successfully react to and interact with a user, and indeed one which would actually lose educational value if it were to undergo an interface-lift. it should be noted that eurotech is the official distributor of saki introduction landa (1976) envisages five fundamental categories of instructional systems: 1. linear programmes, such as those of programmed learning or page-turning with question-and-answer sessions for verifying understanding before allowing the learner to continue. 2. intrinsic programmes, where learners are branched to other parts of the programme either as a remedial activity or in order to skip a part where they have already demonstrated understanding. 65 michael horwood interactivity and multimedia: the case ofsaki 3. extrinsic programmes, which react to a learner's response pattern over a period of time rather than to a single response. 4. adaptive programmes, which learn from a learner's response pattern and automatically adapt the programme of instruction. 5. structure-diagnostic programmes, which not only react to the responses made by the student, irrespective of whether they are right or wrong, but also make a diagnosis of the underlying reasons for any incorrect responses. saki (self-adaptive keyboard instructor) meets the criteria for the adaptive category 4. it is an intelligent computer-based training program which uses a fully adaptive conversational tutorial mode of instruction, to: • teach 10-finger touch-typing to complete beginners as an automatic reflex-skill • enable competent typists to build on existing skills and improve accuracy, rhythm and speed • convert two-finger 'peckers' to competent touch-typists • teach touch-typing to certain categories of handicapped persons. using a complex algorithm, it measures the variations in response rates as well as reacting to typing errors (by simple pattern matching) and speed of typing (by using the computer's internal timing device). it has been developed as a sophisticated ms-dos version of gordon pask's adaptive teaching machine (pask, 1958a, 1958b, 1960, 1965a, 1982). its pedigree dates back to the design and testing of such concepts in the late 1950s, and uses pask's conversational tutorial mode of instruction (pask, 1980) to learn about individual students. it automatically adapts the instructional process to the student's personal learning style by using error patterns and response rates as controls. in contrast to other typing tutors which use mechanical repetition, saki has no pre-set lessons, but uses the controls to concentrate on the areas where the learner is experiencing difficulty. the history of saki saki was first designed in 1955 at system research (uk) by gordon pask, elizabeth pask and robin mckinnon-wood (pask, 1982). one of the first commercial versions was a keyboard trainer for the hollerith 12 keyboard, constructed in 1957 for solartron electronic group uk. the machine was used for keypunch training, using one-depression keystrokes for numerical entry and double-depression for alphabetic characters, punctuation, etc. solartron is said to have sold over 200 machines, each one having been hand-built. they took the form of electromechanical devices, using small film sheets with imprinted rows of letters. each film was inserted in front of a panel of lights where it remained stationary, cueing the typist by a row of lights illuminated in sequence. the 'exercise block' that was literally screwed into the 66 alt-j volume 1 number 1 machine was a mate to the film, such that the system knew which letters were being illuminated. each block contained approximately 128 bytes of memory. there were about 40 pairs of exercises and films covering the instructional process. the first fully software-based version of saki was developed by system research in the late 1970s. this version was sold as a dedicated machine based on the research machines 280z microcomputer with the alphanumeric characters on the qwerty keyboard blanked off. the machine was purchased by a handful of major organizations concerned with training large numbers of typists. among the british organizations which used this version were honeywell control systems, scheme management, and scottish gas. in the late 1970s the single training station sold for over £2,500. in 1981, a new design was developed which achieved greater flexibility and adaptability for the individual student, and easier tailoring to different applications and national languages. in 1986, yet another new version was written in the c programming language, extra features added, and the software tailored to run under ms-dos. this version still the current one has extensive facilities for record keeping, for adjusting the responsiveness of the program, and for customizing the exercise material to individual vocabularies and national languages. an adaptive approach to learning sensory-motor skills in order effectively to examine the public reaction to saki, it is first necessary to define the difference between touch-typing and keyboard skills. keyboard skills are little more than keyboard knowledge: in effect an understanding of the keyboard layout, and purpose and functions of the keys. touch-typing, on the other hand, is an automatic reflex skill: the user types at a keyboard without mentally processing the individual characters which make up words, and in some cases complete sentences. research on typing skills carried out in the course of the development of the original saki has been subsequently confirmed (gentner, 1984). the research has shown that that the skill of an expert typist is due to more than finger-speed alone. overall typing speed can be measured as a combination of finger speed and lag time between keystrokes, and the primary determinant of the expert's overall typing speed is a consistently short lag between the typing of individual keys. the more expert the typist, the faster the transition from key to key, and as the speed of key transitions increases, typing becomes less sequential. instead of recognizing and typing one character at a time, the expert can mentally process more than one character simultaneously, without being aware of so doing. romiszowski (1986) classifies the various viewpoints on teaching methodology as being either 'prescriptive' or 'democratic'. the prescriptive approach supports individualization on the basis of a comparison between the individual student's approach and some ideal model. such a comparative diagnosis leads to an individual prescription of learning activities for the student. the democratic (student-centred) approach supports individualization for the student's own sake, the course therefore being adapted to his or her needs. 67 . michael horwood interactivity and multimedia: the case of saki there is a certain amount of partisan warfare between these two points of view, on philosophical grounds. for some time, however, an alternative to both of them has been under construction by cyberneticians. work such as that of pask (1965b, 1973) suggests that machine-based systems can developed which can learn from the learner, can adapt to the learner's strategy for learning, and can redesign the presentations (on a conversational-tutorial pattern) in ways superior to those achieved by human tutors. romiszowski concludes that this has certainly been the case with machines such as the saki keyboard trainer for high-speed sensory-motor skills. in the case of saki, a complex algorithm measures variations in response rates, and uses those control measures to redesign each line of the typing exercises for the learner around his or her personal learning curve. where the learner achieves a predetermined level of accuracy and smoothness (this being the consistency of the lag between key depressions), saki will remove characters) from the current exercise and replace them with characters corresponding to the next level of difficulty, accompanied by an appropriate message, such as "jkl; is smooth". any characters which fall below the acceptable limits of accuracy and smoothness appear more frequently, together with a message such as "difficulty with asdf'. where the learner's response rates and error patterns fall between the two limits, saki repeats the exercise in a slightly different combination without changing the level of difficulty. referring to the earlier version of saki, with its relative slowness and low level of sophistication, romiszowski (p. 23) nevertheless writes: randomly selected letters appear on screen 1 and the position of each approximate key is 'cued* by the lights that appear on the simulated keyboard on screen 2. [the latest ms-dos version of saki has a screen which is horizontally split into two segments in order to avoid the need for two separate screens.] as practice progresses, the general speed of presentation of data to be typed increases, and the intensity of lights on the simulated keyboard decreases. however all this happens in function of the error pattern of the trainee. he/she gets longer to respond to letters which give difficulty, gets more intense clues and over a longer period of time for these letters and, furthermore, these letters begin to occur at a frequency proportional to their level of difficulty. all this occurs in response to a trainee's individual pattern of errors. he/she gets more practice, more time to respond and more help, adjusted on a continuous, 'on-line' basis by the control algorithm built into-the trainer. this highly adaptive learning environment is extremely efficient, leading typical learners to achieve first-rate typing skills in only a few hours of drill-and-practice. romiszowski goes on to discuss how similar adaptive computer-based training devices are now fairly common in military training for developing other psychomotor and perceptual skills such as radar tracking, missile firing and aircraft gunnery. military budgets, at least during the time of the cold war, permitted the release of finance to fund research into such sensory-motor training devices. . 68 altj volume 1 number 1 bell (1990) takes the following line on saki: the programme is interesting in that it structures an individual's learning, paying due regard to their initial skills, subsequent errors made and speed of progress. saki monitors the learners' performance and adjusts exercises to provide a constant challenge but not one so great as to become demotivating. in doing so it clearly draws upon research into learning (particularly that of gordon pask), and into the teaching of typewriting skills. rushby (1988) takes a similar positive attitude: ironically, one of the oldest and most widely used intelligent cbt packages hides its intelligence so successfully that users are deceived by its apparent simplicity [...] the skill of classical touch-typing is well-defined, and so it is possible for saki to contain an accurate model of the topic that is to be learned. the model of the learner is built and continually adapted by saki as the learner works through successive exercises. the program then applies its 'intelligence' to devise new exercises which build on the learner's strengths and tackle the weaknesses. the result is, at first sight, unexceptional, but after about 20 minutes' use the learner's concentration is intense and the rate at which touch-typing skills are acquired is remarkable. rushby's article draws attention to the primary dilemma facing a cbt package as sophisticated as saki, and which brings me to the principal point of this paper, impinging as it does on the design of training materials in general and interactive multimedia in particular. multimedia for its own sake? the question is: given that the presentation of a product is vital to its success, should a designer lower its pedagogical value by changing the the design features in favour of a greater acceptability? for example, does one decide on a multi-colour screen presentation, which may well appeal to an audience unskilled or untrained in the fundamental basis of good instructional design, when the use of colour can actually reduce the effectiveness of the training programme? the size of the evaluation problem is illustrated by an interesting case. a major international organization specializing in cbt design and assessment (i am unfortunately obliged to withhold the name) carried out an evaluation of saki supposedly under controlled conditions. the evaluation team failed to erase the records saki was keeping of every move made, thus leaving precise evidence of the assessment procedures which had been followed. the result of the evaluation was that the package was considered unsuitable as a training device for 69 michael horwood interactivity and multimedia: the case ofsaki delivering touch-typing skills. saki's records were later examined. they showed that of the two individuals who carried out the evaluation, one had stayed with the package for less than five minutes, the other for less than two minutes. neither took a test, nor examined the testing facilities, the customization features or the extensive record-keeping features. i bring this case to light in order to illustrate the practical problems associated with marketing a high-quality training package where all the effort in the design has been concentrated into the instructional process rather than into a striking or trendy user-interface. the bulk of available touch-typing programs are decorated with multi-coloured screens, flashing "well done, fred" messages, and other such distractions from the fundamental job of delivering touch-typing as a psychomotor skill. these are the programs, however, which attract and hold the attention of professional reviewers and evaluators. one thing is certain. many touch-typing tutors have been designed over the years, some of the simpler ones apparently having been constructed very rapidly and with little real consideration of genuine pedagogical value. the longest surviving is still saki, after some 35 years. it does not typify an ultra-modern user interface, but nevertheless provides precisely the kind of automatic adaptivity and interactivity which current computer technology aims to achieve. in short, multimedia may be about attractive pictures, colourful screen displays and digitized sound, but if in a training package these features detract from the educational value of a program, they can do more harm than good to the reputation and therefore future prospects of interactive applications. interactivity does not and should not depend on fancy wrapping. references bell, c., etti, 26, 4. gentner, d. (1984), expertise in typewriting, centre for human interface processing, university of california, san diego, april 1984. landa, l.n. (1976), instructional regulation and control: cybernetics, algorithization and heuristics in education, englewood cliffs nj, educational technology publications. pask, g. (1958a), 'electronic keyboard teaching machines', journal of the national association for education and commerce, july. reprinted in glazer, r. and lumsdaine, a. (eds), teaching machines and programmed learning, washington dc, national education association, 1, 336-49. pask, g. (1958b), 'research on the design of adaptive teaching systems with a capability for selecting and altering their criteria for adaptation', reports af61 (052)-402, aerospace medical research laboratories. 70 altj volume 1 number 1 pask, g. (1960), 'the teaching machine as a control mechanism', transactions of the society for instrument technology, 12, 2 (june). pask, g. and lewis, b.n. (1965a), 'the theory and practice of adaptive teaching systems' in glazer, r. (ed), teaching machines and programmed learning, washington dc, national education association, vol. ii, data and directions, pp. 213-66. pask, g. (1965b), the cybernetics of human learning and performance, london, hutchinson. pask, g. (1980), 'developments in conversation theory: part i', international journal of man-machine studies, 13, 357-411. pask, g. (1982), 'saki: twenty-five years of adaptive training into the microprocessor era', international journal of man-machine studies, 17, 69-74. pask, g. and scott, b.c.e. (1973), 'caste: a system for exhibiting learning strategies and regulating uncertainty', international journal of man-machine studies, 5, 17-52. romiszowski, a.j. (1986), developing auto-instructional materials, london, kogan page. rushby, n.j. (1988), 'do we really need intelligent teaching systems?', personnel management (march). 71 (page number not for citation purpose) 1 *corresponding author. email: claudio.aguayo@aut.ac.nz research in learning technology 2018. © 2018 c. aguayo et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 research in learning technology vol. 26, 2018 original research article embodied reports in paramedicine mixed reality learning claudio aguayoa*, cristóbal dañobeitiab, thomas cochranea, stephen aielloc, stuart cookc and alberto cuevasb acentre for learning and teaching, auckland university of technology, auckland, new zealand bembodied group, santiago, chile cparamedicine department, auckland university of technology, auckland, new zealand (received: 5 september 2018; final version received: 7 november 2018) this paper is based on the second stage of a design-based research (dbr) project encompassing the initial prototyping of virtual reality (vr) simulation in paramedicine education using self-reported and biometric feedback data. in this discussion paper we present the range of reflections and theoretical possibilities that arose from the piloting experience and their implications in redesigning practice in paramedicine education. we focus on the foundational literature and epistemological understandings coming from neurophenomenological cognitive science applied in technology-enhanced learning, using mixed reality (mr) in paramedicine simulation learning as a case. we do so following the logic of a dbr methodological framework, in part demonstrating the usefulness of dbr when reflecting on applied practice to inform newer theoretical developments, leading to further integrated solutions in future practice. in addition, we also put attention on a conceptual shift from a focus on vr, to a focus on mr with emphasis on the associated benefits offered by mr learning situations within paramedicine education. finally, we discuss the benefits of incorporating self-reported and biometric feedback data in paramedicine education in particular, and in technology-enhanced learning in general, for the design of meaningful learning experiences informed by emotional and physiological responses of learners. keywords: mixed reality; biometric feedback; self-report; clinical simulation this paper is part of the special collection mobile mixed reality enhanced learning, edited by thom cochrane, fiona smart, helen farley and vickel narayan. more papers from this collection can be found here. introduction in this section we present the theoretical design process of a mixed reality (mr) learning experience in paramedicine education, informed by participant self-reported and biometric feedback data, which we define as ‘embodied report’ data. this mr learning design is part of the multiple environment simulation hub (#mesh360) project, which originally was set up during 2016 to explore the use of virtual reality (vr) to enhance paramedicine education, focusing on authentic mobile vr simulation scenarios and exercises (cochrane et al. 2016). since its original conceptualisation, the mesh360 project has evolved to now include the exploration of the use of mr learning scenarios informed by embodied reports data. such a process is fundamented on a neurophenomenological approach to cognition and emotional mailto:claudio.aguayo@aut.ac.nz http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v26.2150 https://journal.alt.ac.uk/index.php/rlt/pages/view/collections c. aguayo et al. 2 citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 (page number not for citation purpose) experience (dañobeitia 2017; varela 1996) and follows a design-based research (dbr) methodology to design solutions to practical problems. dbr consists of constant feedback loops between theory, reflection, testing and applied practice, while accounting for the uptake of newer theoretical knowledge, technologies, methodologies and contextual challenges (amiel and reeves 2008). following the recent attention to the use of immersive technologies in education and professional training simulation, such as vr and augmented reality (ar) (aguayo, cochrane, and narayan 2017), focus has been put on the combination of ar and vr with real environment (re) situations (i.e. non-digital learning elements) to produce mr learning affordances. the rationale behind this is to maximise the learning affordances offered by the integration of different digital elements of a mr learning scenario, so learners can benefit from a range of learning possibilities available within the digital continuum of mr learning experiences (jowsey and aguayo 2017; milgram and kishino 1994). such an approach seeks not only to benefit from the individual learning opportunities offered by the different learning affordances of each independent element present in a mr learning intervention but also to address the unique social, cultural and cognitive particularities of individual learners to produce enhanced learning outcomes across broad audiences (aguayo 2017). the research is also informed by the exploration of subjective experiences correlated to empirical biophysiological data coming from recent developments in enactive cognitive science (froese and di paolo 2011; varela 1996). this permits a better understanding of how users experience a given learning situation based on self-reported subjective data and empirical biometric feedback data (brouwer et al. 2015). basically, by assessing self-reported and biometric information of users during a learning experience, it is possible to extrapolate their emotional and physiological responses, that is, their ‘embodied reports’, to specific emotional stimulus (dañobeitia 2017). this has led us to explore how the use of embodied reports data can inform and improve the redesign of learning scenarios in paramedicine education using mr learning scenarios. for such a data-driven process, dbr is a useful methodology, as constant feedback loops between initial theoretical solutions tested in practice permit the generation of newer practical solutions informed by evolving theory, methodologies and technologies (cochrane et al. 2017). literature review the following sections explore the literature surrounding the foundational theoretical frameworks and contexts that underpin the project. background: the mesh360 project and dbr in mr learning clinical simulation is a widely accepted and utilised element of healthcare education (cook et al. 2013; pike and o’donnell 2010). traditional forms of health education simulation involve the use of high-fidelity mannequins for students to practise clinical techniques within a safe environment for themselves in a manner that does not endanger live patients (cook et al. 2012). the mesh360 project was born out of the need to find more flexible and cost-effective simulation experiences (zendejas et al. 2013) for paramedicine students who are predominantly working professionals scattered around the country. the goal is to enhance traditional simulation experiences through http://dx.doi.org/10.25304/rlt.v26.2150 research in learning technology citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 3 (page number not for citation purpose) the use of mobile mr, while also redefining the boundaries of traditional simulation pedagogies that are predominantly competency focused and consequently informed by behaviourist learning theories (hamstra et al. 2014). we are interested in redefining healthcare education to enable the development of student critical thinking, diagnosis, problem-solving and creative approaches to critical care, and therefore we leverage pedagogies that are informed by learning theories such as constructivism, connectivism, experiential learning and authentic learning, for example heutagogy (hase and kenyon 2001, 2007). this led to the development of an implementation strategy that also aims to keep the project costs low and enables rapid and agile development. we chose a dbr methodology to guide the development and iterative redesign of the mesh360 project. dbr provides a structured iterative four-stage approach to curriculum redesign and critically informs the integration of new technologies into the curriculum through a thorough investigation of the literature as a first stage. through our initial literature review we identified five key design principles to guide the development of the mesh360 project (cochrane et al. 2016, 2017). • design principle (dp) 1: basing the project within a dbr methodology • dp2: supporting the project through the establishment of a community of practice • dp3: using heutagogy (student-determined learning) as a guiding pedagogical framework • dp4: designing around the authentic use of mobile devices and vr • dp5: integrating collaboration and teamwork into the project activities this was followed by our rapid prototyping and redesign stages of an mr enhanced simulation (cochrane et al. 2018). we used mckenney and reeves’ (2012) model of dbr mapped to our identified design principles, as shown in figure 1. figure 1. design-based research mapped to the mesh360 design principles (adapted from mckenney and reeves 2012). eor, ecology of resources; ar,  augmented reality; vr, virtual reality; dp, design principle. http://dx.doi.org/10.25304/rlt.v26.2150 c. aguayo et al. 4 citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 (page number not for citation purpose) in the following sections we explore the literature informing the integration of embodied reports into the mesh360 project. emotions and cognitive science ‘cognitive science’ refers to the scientific study of the mind, and it brings together different disciplines such as neuroscience, psychology, artificial intelligence, linguistics, anthropology, philosophy and cybernetics. cognitive science allows us to understand human and animal behaviour from a multidisciplinary perspective considering biological, cultural, social and philosophical aspects. from a biological perspective, each type of organism, from the simplest virus to the largest tree or the complicated human being, has its own functional physiological characteristics. this refers to the set of underlying physical and chemical mechanisms responsible for the origin, development and progression of life and living organisms. the science of human physiology, in particular, tries to explain the characteristics and specific mechanisms of the human body that make it a living being (hall 2017). human physiological responses, as well as those in many invertebrates, are commanded by the nervous system. the nervous system is anatomically divided between the central and peripheral nervous systems (cns and pns). the cns is made up of the human brain and the spinal cord, and the pns comprises all nerves that are placed outside the brain and the spinal cord. the snp comprises somatic, autonomic and enteric nervous systems (hall 2017). in this regard, it has been recorded that the behaviour of the nervous system during its homeostasis, that is, in its self-regulating function to maintain a stable internal condition against a stimulus, is linked to different emotions and cognitive functions (guyton and hall 2003). of particular interest for physiological biometric feedback in cognitive science are the two subsystems of the autonomic system, which is divided into the sympathetic and parasympathetic systems. the sympathetic system is supporting the stress response (‘fight or flight’) to threatening events; it is a central component of emotional experience and, by extension, manifests itself at the cognitive and behavioural level (christopoulos, uy, and yap 2016). the sympathetic system, in turn, supports the mobilisation of the body towards appetitive or gratifying stimuli. the typical measurable manifestation of the sympathetic system is the control, mostly involuntary, of physiological responses, such as heart rate (hr), sweating, breathing and eye blinking (christopoulos, uy, and yap 2016). theories of emotion there are several theories about what is meant by an emotion, so that an exact definition of it remains unfinished. from descartes to darwin, the assumption has been that the perception of danger produces a condition, fear, which in turn is expressed in the body, for example as vasomotor discharge and behaviour (perception → emotion → physiology). in this perspective, emotions are expressed bodily; the corporal expression is a manifestation of something 'internal', of a non-corporal experience (haye and carballo 2017). tomkins (1962), cited in ekman (1992), states that emotions are sets of organised responses that ‘are capable, when activated, of simultaneously capturing organs as widely distributed as the face, heart and endocrine, and imposing a specific pattern of correlated responses’ (p. 2). http://dx.doi.org/10.25304/rlt.v26.2150 research in learning technology citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 5 (page number not for citation purpose) james’s theory, on the other hand, holds that emotion, far from being an expression of an internal state, is a bodily modification (physiological in a broad sense) that is felt or perceived ‘inside’, during the process of this bodily change (cannon 1927; haye an carbayo 2017). the causal scheme perception → physiology → emotion corresponds properly to the theory of lange (schioldann 2001), whose familiarity with james has come to be called ‘james-lange hypothesis’ from cannon (1927) (haye and carbayo 2017). the james-lange theory was subject to much criticism, however, until the 1980s: ‘neuroscience research on the role of peptides and receptor cells established that (both) james-lange and his critics were right in that the emotions originate as much in the corporal changes as in the mental perceptions’ (coleman and snarey 2011). this means that both theories are considered correct. today in the literature we can find numerous studies where it has been possible to relate the autonomic nervous system (ans) with emotional states. there is evidence of distinctive patterns of activity of the ans with emotions (ekman, levenson, and friesen 1983; kreibig 2010), such as anger, fear, disgust, sadness (ekman 1992; levenson, carstensen, friesen and ekman 1991). these correspond to negative emotions, as well as joy, which corresponds to a positive emotion (ekman, levenson, and friesen 1983), and surprise, which does not have a clear valence (kreibig 2010). these six emotions have been called ‘basic emotions’, and also ‘primary’ or ‘fundamental emotions’, because these emotions are manifested with the same patterns in all cultures, as well as in some animals (ortony and turner 1990). measuring emotions through the autonomic nervous system the different emotional states are a reflection of the activity of the nervous system, which can be assessed through different methods. some methods measure changes in neural activity, such as electroencephalogram (ciorciari 2012), magnetoencephalography and topography of steady state (harris, ciorciari, and gountas 2018). other measure changes in metabolic activity, such as functional magnetic resonance imaging, positron emission tomography and functional transcranial doppler ultrasound (ahlert, kenning, and plassmann 2006; harris, ciorciari, and gountas 2018). on the other hand, biometric measurement methods measure the response of the sympathetic system, which is associated with the control of physiological responses, most of them involuntary. all these different methods, that is, biometric measurement, neural activity measurement and metabolic activity measurement of the neural system, are methods used to study cognitive and affective processes (harris, ciorciari, and gountas 2018). relevant sympathetic physiological responses, for the purpose of this article, are heart frequency rate and sweating, as these can be correlated to emotional responses during particular cognitive and affective conditions, such as for example during a learning process. these two physiological responses can be assessed through the measurement of hr and galvanic skin response (gsr), respectively. heart rate hr represents one of the most promising markers. it is believed that hr can be a great indicator of stress; in fact, it has been reported that salivary cortisol levels, which are  an indicator of stress, increase when there is a high hr (michels et al.  2013). http://dx.doi.org/10.25304/rlt.v26.2150 c. aguayo et al. 6 citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 (page number not for citation purpose) although there is a wide range of factors that affect hr, the ans is the most prominent one. the heart is doubly innervated by the ans so that relative increases in sympathetic activity are associated with increases in hr and relative increases in parasympathetic activity are associated with decreases in hr. therefore, the relative sympathetic increases cause the time between heartbeats (the interval between beats) to shorten and increase the relative parasympathetic (thayer et al. 2012). the most frequent method used to measure the hr is the electrocardiogram, which measures the hr and the electrical wave potential detected by two electrodes, one on each side of the heart (andreassi 2010). another way to measure hr is through pulse photoplethysmography, introduced by hertzman (1938), which is a simple and useful method to measure the relative changes of blood volume in the microvascular bed of peripheral tissues and evaluate the peripheral circulation. this signal is obtained through non-invasive pulse oximetry systems and is based on the absorption of light by the bloodstream (gil et al. 2010). within the study of emotions, it has been observed that there is an increase in the acceleration of the hr against pleasant patterns, and an increase in the deceleration of the hr against unpleasant patterns (balconi, vanutelli, and finocchiaro 2014). it has also been seen that there are variations according to the intensity of the stimulus. stimuli with low intensity are related to a deceleration of the hr, while intense stimuli have been seen to activate defence responses associated with the acceleration of the hr (balconi, vanutelli, and finocchiaro 2014). galvanic skin response sweating is an indicator of stress or arousal. an arousal has been conceptually defined as a ‘driving state or an energizer of non-specific behavior, something that describes the intensity of an experience, but not its quality’ (dillon et al. 2000, p. 2); that is, arousal is an indicator of intensity (duffy 1962). gsr is a frequent method used to measure the activation of the sympathetic nervous system through sweating. this gives direct information about the electrical conductance of the skin, which is related to the levels of sweating in sweat glands (gil et al. 2010; labarbera and tucciarone 1995). the neural control of the sweat glands in humans is entirely under the sympathetic control. an increase in sympathetic nervous system activation is used as a large indicator of arousal. several studies using the image view paradigm have shown that the gsr is highly correlated with the emotional arousal self-report. gsr can be used as an indicator of different processes such as activation, attention and significance of tasks or affective intensities of a stimulus as experienced by a subject (labarbera and tucciarone 1995; montagu and coles 1966; ohme et al. 2009). however, it is not a good indicator of valence; its great limitation is that the valence of the stimulus cannot be measured – that is, it cannot be said whether it is a positive or negative stimulus (labarbera and tucciarone 1995; ohme et al. 2009). for many theorists, an important characteristic of emotions is their coherence of response, variously labelled as coherence of the response system (ekman 1992), concordance (nesse et al. 1985; wilhelm and roth 2001) or organisation of trends of answers (lazarus and lazarus 1991; levenson 1994). although there are reports showing a correlation between emotional experience and physiology (kreibig 2010; shu et al. 2018; siedlecka and denson 2018), there is a small niche studying this, which calls for further explorative work. http://dx.doi.org/10.25304/rlt.v26.2150 research in learning technology citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 7 (page number not for citation purpose) neurophenomenology and self-report in recent decades much importance has been given to the use of first-person methods in cognitive sciences, in particular to the science of consciousness, where it has become a need to ask participants whether they perceived the stimulus presented or not (kriegel 2013). the chilean neurobiologist francisco varela proposed a study model to study consciousness called ‘neurophenomenology’ (varela 1996). neurophenomenology emphasises the importance of collecting data in the first person of phenomenologically trained subjects as a strategy to describe and quantify physiological processes relevant to consciousness. the general approach, at the methodological level, is to obtain richer data in the first person through disciplined phenomenological explorations of experience and to use this original data in the first person to discover new data in the third person through physiological processes crucial for consciousness. therefore, a central objective of neurophenomenology is to generate new data by incorporating refined and rigorous phenomenological explorations of experience in the experimental protocols of neuroscientific cognitive research on consciousness (lutz and thompson 2003). self-reporting is one of the most common procedures for collecting data in psychology; it is also an important tool for the social sciences in general. it is possibly the most traditional way to measure emotional experiences in these areas. strictly speaking, self-report refers to the act of a person giving a detailed account or statement describing their actions or any personal event (hersen et al. 2004). yet one of the main problems in accessing the experience of an individual lies precisely in that current methodologies explain the behaviour of subjects through traditional instruments, such as surveys and interviews, which are unsatisfactory in terms of not taking measurements at the same time that they are living the experience of a visual and/or sound content. this produces a bias because the interviewees must remember past experiences and bring them to the present. in other words, they only collect personal reports and past memories, or projections of possible or hypothetical actions (babbie 2013). even more, not all of these methods are capable of generating data because the report is vulnerable to variables that can distort the description of those who are actually experiencing it. within these variables, it is found that (1) the subject is not willing to do the introspective exercise required to accurately report their experience or (2) the subject feels pressured to respond according to expectations that may differ from their real experience (dañobeitia 2017). the embodied reports methodology from the preceding exploration of the literature, we see an opportunity for methodologies that measure the factors that determine the subjective experience of individuals by accounting for their enjoyment, emotional involvement and retention of content, in addition to assessing their behaviour and physiological responses to a given stimuli. from a science of experience perspective, it becomes necessary to know, identify and characterise people according to their needs and tastes when confronted with given content. in this sense, emotional involvement of individuals with content is related to the degree of retention of the same. the more intense the emotional involvement of the participants with content, the more they remember it (baraybar-fernández et al. 2017). http://dx.doi.org/10.25304/rlt.v26.2150 c. aguayo et al. 8 citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 (page number not for citation purpose) inspired by the enactive neurophenomenological approaches to cognition proposed by varela (1996), and the epistemological underpinnings derived from the santiago school of cognition (hallowell 2009; luisi 2016), the embodied group from chile (www.embodiedgroup.com) developed a multimodal data approach methodology (lahat, adali, and jutten 2015), coined ‘embodied reports’, to assess emotional experiences via physiological and self-reported data (dañobeitia 2017). the embodied reports approach constitutes a methodology to elucidate the level of emotional engagement, based on the subjective experience and differential biological responses of individuals faced with a particular content presentation and/or emotional experience. the embodied reports methodology offers an innovative and technological solution to assessing the emotional response of individuals by providing a non-invasive way of measuring a subjective experience through a physiological and emotional analysis of the participants, pointing to a greater possibility of contrasts compared with other methodologies (dañobeitia 2017). at the core of the methodology is the integration of both the self-reported subjective experience of an individual (‘first-person data’) with physiological correlates that reflect autonomic activity that accompanies a subjective state under study (‘third-person data’) (dañobeitia 2017). for this, the embodied reports methodology uses an expression pedal for obtaining self-reported first person data, and modern wearable physiological sensor devices that are able to measure physiological responses of the body as third-person data, such as gsr and hr. the expression pedal works like a gas pedal in a motor vehicle, where the subject presses it, obtaining numerical answers from 1 to 100, with 1 meaning a low value and 100 a high value. the use of the pedal allows for a real-time report of the preferences of the subjects, which can be contrasted with the physiological correlates obtained by the wearable sensor devices, thus permitting the researcher to know at each moment what emotional state a particular subject was in. this, complemented with rational reflection and sample statements (subjective elements), allows for the measuring of participants’ experiences rather than rational behaviour from them, thus providing indicative information about the emotional states of individual participants. therefore, the embodied reports methodology integrates conscious information with the unconscious data of people participating in the measurement process. the research team identified the synergy between the embodied reports methodology and the mesh360 project, and thus we integrated the embodied reports methodology within the mesh360 dbr project. this resulted in the development of a transdisciplinary dbr team (table 1). as shown in table 1 the research project evolved to incorporate the collaboration of  four teams, each with specific expertise required to design and develop the theoretical and practical elements of the project. these included three teams based at the table 1. mesh360 design-based research team. dbr collaborators mesh360 project design team paramedic lecturers academic advisors mmr development team embodied biometric research team key responsibility discipline context experts educational technology foundations technology implementation advice embodied enactive biometric data collection experts http://dx.doi.org/10.25304/rlt.v26.2150 www.embodiedgroup.com research in learning technology citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 9 (page number not for citation purpose) university: discipline experts (paramedic lecturers); academic advisors as educational technology experts; an mobile mixed reality (mmr) development team providing the mmr platform choice and integration advice; and a team of enactive biometric researchers based in chile (embodied group), who visited the main project team for 2 weeks in 2017, providing prototype embodied biometric data collection instruments and methods. redesigning practice in paramedicine mr learning our initial research questions for the 2017 mesh360 project included: (1) how can we enhance clinical simulation learning environments using mmr to more authentically reflect real-world scenarios? (2) can first-person (enactive) and biometric feedback be used to triangulate subjective student feedback and thus provide evidence of the impact upon student learning using mmr scenarios? in this paper we focus upon the 2018 redevelopment of the project to better address the second research question. participants for the study included volunteers from first year to third year bachelor’s of health science undergraduate degree students, as well as invited professional (employed) paramedics, with a total of 32 participants in the first iteration in 2017 and a further 21 participants in 2018. ethics consent was gained through the university ethics committee, and all participants were supplied with an information sheet and consent forms for the research element of the study, with all participant data anonymised. embodied reports piloting in paramedicine vr simulation during august 2017 we piloted the embodied reports methodology using the mesh360 project in paramedicine education as a case study. the aim of the pilot was to triangulate self-reported subjective student feedback and objective hr and gsr biometric data with student learning, while engaged in a virtual paramedicine scenario. for this purpose, students were presented a critical care 360-degree scenario with hidden points of interests (using a google cardboard-compatible headmounted display) and were asked to explore and identify risks within the virtual environment. the virtual scene represented a building site with multiple potential hazards, including chemicals, exposed electrical and water pipes and people hiding (cochrane et al. 2018). the chosen 360-degree online visualisation platform, seekbeak, allowed for the background recording of the viewing of the hidden point of interests in the form of a ‘hotspot heat map’ (figure 2a). to record a hotspot in the heat map, students had to stare at a given potential hazard for more than 2 s. students were also connected to an embodied reports unit recording first-person data through a self-reporting pedal, and third-person data through hr and gsr sensors (figure 2b). students were presented the 360-degree scenario for a minute and were asked to increasingly press the pedal when they felt emotionally engaged with the scene. note that all participants from the pilot study (n = 10) reported not being distracted by the task of pressing the pedal while visualising the 360-degree scenario. however we acknowledge such task may alter the biofeedback response, which calls for future methodological improvements http://dx.doi.org/10.25304/rlt.v26.2150 c. aguayo et al. 10 citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 (page number not for citation purpose) to eliminate and/or isolate this potential issue. finally, the embodied reports data was processed and plotted against time for a global representation of firstand third person data sets in relation to the duration of the experience (figure 2c; for details see cochrane et al. 2018). the piloting of the embodied reports methodology with the mesh360 project was a data-driven process. only by measuring and interpreting firstand third-person data across participants with different degrees of contextual awareness knowledge in paramedicine could we make sense of the data in relation to the pedagogical aims of the project. we did expect to be able to inform and redesign consequent practice based on embodied reports findings; yet analytical reflection and discussion sessions between the researchers and practitioners were required first. once again, the dbr methodology proved to be adequate in accommodating new data-informed findings with prior theoretical guidelines tested in practice during the piloting, which led to reflection to produce a new set of theoretical solutions to be tested in contextual practice (amiel and reeves 2008; cochrane et al. 2017). in the case of the mesh360 project, a whole range of new theoretical avenues emerged from the piloting of embodied reports data in paramedicine education. these are presented next. redesigning practice using mr learning informed by embodied reports data the mesh360 piloting during 2017 based on the embodied reports methodology provided a range of emerging methodological considerations for the research team. these principally related to methodological enhancements, hardware and software figure 2. experimental piloting set-up using an embodied reports unit to record first and third-person data during a paramedicine virtual reality scenario: (a) hotspot heat map representing the immersive experience of a participant; (b) experimental set-up including the self-reporting pedal (green) and hr and gsr sensors (blue and red, respectively); and (c) analytical representation of embodied reports data sets against time of the virtual reality experience. http://dx.doi.org/10.25304/rlt.v26.2150 research in learning technology citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 11 (page number not for citation purpose) improvements, characterisation of participants in relation to their stress levels prior to, during and post-event, design considerations in relation to vr and mr, and new research pathways. table 2 presents a summary of such emerging key points regarding the use of embodied reports data in paramedicine education. table 2. summary of key points that emerged from the mesh360 2017 piloting using the embodied reports methodology. domain emerging key points methodology enhancement • explore use of mixed reality (mr) over virtual reality (vr). • define baseline levels of stress prior presentation of vr/mr event (using hr, gsr and questionnaires). • complement paramedicine experience with sound stimulus (beeping machine, crying person, etc.). • ask participants if they recall external stimuli (multisensorial recall training). • measure how the job simulation emotionally affects the student (post-event). • understanding of vr/mr clinical environments influencing working diagnosis. • need to replace gsr and hr units with a haptic glove (or similar), to measure biometric data during work with mannequin. participant characterisation • determine level of qualification, expertise or clinical background of participants (including control participants). • determine previous states potentially affecting biometric measurement, e.g. recent consumption of coffee, medicine or other similar products pre-simulation. • implement cortisol measurement through saliva samples (good stress indicator – 30 min adjustments), to complement biometric measurements. • determine the level of stress/anxiety before the vr/mr simulation. this is very important in entering a simulation process. vr/mr experience design • increase duration of vr/mr experience to homogenise the experience across participants (needed for comparison purposes). ideal: sequence of two to three vr scenarios, e.g. ambulance + scene 1 + scene 2. • make experience as immersive as possible (technology dependent). type of vr/mr technology influences the experience. • adding multisensorial elements, e.g. sounds (ambulance, machinery), smell, etc. emerging research pathways • research pre-simulation levels of stress (biometric baseline). • research correlations between pedal hit patterns across demographic groups. • explore if wearing current version of embodied unit during simulation with mannequin is possible or viable • investigation of the cognitive influence of teamwork, e.g. measurement of crew mate during mannequin work. this leads to practitioner–patient relations as well • amount of ‘appropriate’ stress level for paramedicine students. every student responds differently to stress (e.g. some act better under stress): ° look into how different lecturers deal with highly stressed students. ° look into how students respond after 2–3 days of high stress scenarios. • longitudinal study to follow the student journey over a 5-yr period determining evolution of stress levels across the study journey. • emotional triggers: how students connect with particular medical events that are familiar to them, e.g. family members with similar experiences. hr, heart rate; gsr, galvanic skin response. http://dx.doi.org/10.25304/rlt.v26.2150 c. aguayo et al. 12 citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 (page number not for citation purpose) the set of key points and considerations that arose from the mesh360 piloting event during august 2017 allowed the research team to reconsider the methodological procedures employed during the piloting. following the dbr methodology ( figure 1), the research team carried out different reflection and brainstorming sessions to consider how the data-driven experience from 2017 could inform a new methodology and redesign of the original mesh360 2017 intervention, in preparation for the following data collection event to occur during september 2018. redesigned methodology findings from the mesh360 2017 piloting event, in combination with recent mr literature and experiences (e.g. akçayır and akçayır 2017; jowsey and aguayo 2017) and vr/mr hardware and associated development tools (e.g. use of oculusgo standalone vr headset), directed the design of a new mesh360 clinical simulation mr experience in paramedicine education. from a methodological point of view, more emphasis was put on determining previous states of stress of participating students through both a prequestionnaire and ‘baseline’ (i.e. pre-vr simulation) biometric feedback recording. this particular aspect emerged from an anecdotal finding from the mesh360 piloting event, where one student presented a relatively high hr (120 bpm) when entering the simulation, yet seemed calm to the lecturers. this led the research team to reflect on the type of stress levels students bring into clinical simulations and how that can affect their learning, and potentially their future clinical practice. this prompted the research team to explore and give attention to prior levels of stress before entering clinical simulation events; and beyond that, to consider longitudinal studies in relation to students’ stress level management throughout their academic journey and the implications that pedagogical practices can have on students. from a user experience (ux) point of view, and in consideration of theoretical insights underpinning the embodied reports methodology, it became clear that the vr simulation needed to be longer than the original ‘1 min experience’ from the piloting event. this led the research team to implement longer exposures to vr simulation experiences to allow for longer embodied reports data collection. related to this point, and in consideration of the principle of making experiences as immersive as possible, a shift from a static 360-degree scenario to a dynamic 360-degree scenario (i.e. use of 360-degree video) was adopted. the combination of these ux related factors, that is, longer exposure to vr content and associated recording of baseline stress levels, led to the implementation of a series of vr scenes composing the vr simulation. the starting vr scenes presented ‘calming’ scenarios (e.g. a forest) aiming to record baseline data, and subsequent scenes presented ‘stressing’ content (e.g. an ambulance ride), permitting recording of individual responses to specific and increasingly stressing stimuli. in addition, a shift from an isolated vr experience to an mr experience was implemented by combining the vr experience with a ‘re’ component, as described by milgram and kishino (1994) in reference to the digital continuum of mr experiences. the re component involved clinical simulation work with a high-fidelity simulation mannequin (about 10–15 min long), which presented the symptoms described to students over a ‘radio call’ during the vr ambulance ride. this was also intended to present students with a more authentic environment and was complemented with environmental sounds such as paramedic monitoring equipment sounds and ‘ crying people’. this ux continuum between the vr experience and the re experience is http://dx.doi.org/10.25304/rlt.v26.2150 research in learning technology citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 13 (page number not for citation purpose) what permitted the implementation of a mr learning simulation. in addition, a mid-mr experience assessment was also implemented by asking students to provide a ‘working diagnosis’ and ‘action plan’ between the vr experience and the re simulation room work with the mannequin. finally, from a research point of view, the mesh360 2017 piloting provided a range of novel research considerations and pathways for the research team, including the investigation of potential correlations between embodied reports data patterns with demographic data of participants; the feasibility of using embodied reports recording equipment during the re mannequin simulation work (currently limited by the physical nature of the recording equipment); the investigation of cognitive and embodied synchronisation between team crew members and between paramedics and patients (konvalinka et al. 2011); and research into the ‘appropriate’ levels of stress exposure during clinical simulations, as well as emotional triggers and stress management practices in paramedicine education. discussion here we present and discuss the design process of the mr learning experience in paramedicine education, informed by embodied reports data. findings indicate that the theoretical underpinnings coming from neurophenomenological approaches derived from the santiago school of cognition (varela 1996), informing the embodied reports methodology, and the methodology itself, are adequate to inform the research process related to emotional engagement of students during vr/mr clinical simulation in paramedicine education. the insights provided by the embodied reports methodology during the mesh360 2017 piloting event, in relation to the emotional experiences of paramedicine students during simulation tasks accessed via physiological and selfreported data, were useful in directing the redesign of a new methodology for upcoming events using mr learning scenarios. in addition, the different research pathways that emerged were only accessible through the embodied reports methodology, not only highlighting its usefulness in paramedicine education but also its value in advancing our knowledge in the clinical simulation field using vr and mr. overall, we believe the embodied reports methodology can advance and improve students’ learning outcomes and professional practice in paramedicine education. during this process we also valued the usefulness and adequacy of the reflective feedback loops of dbr as a research methodology to investigate innovation in practice, therefore confirming previous conclusions in designing mobile vr learning environments (cochrane et al. 2017), here tested in relation to mobile mr environments (cochrane et al. 2018). in particular, the reflection process advocated by dbr, in consideration of findings from the first mesh360 piloting stage, allowed for the redesign of newly theory-informed solutions (amiel and reeves 2008; mckenney and reeves 2012), yet to be tested in practice during the upcoming mesh360 events. this set of findings permitted the research team to address the mesh360 research questions exploring ways of enhancing clinical simulation learning environments using mobile mr and therefore more authentic and real-world scenarios through technology-enhanced learning. in particular, findings allowed assessment of whether first-person (enactive) and biometric feedback data, that is, embodied reports data, could be used to triangulate and evidence students’ learning using mobile mr scenarios. in this regard, the embodied reports methodology largely exceeded the initial expectations of the mesh360 research team. in this sense we http://dx.doi.org/10.25304/rlt.v26.2150 c. aguayo et al. 14 citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 (page number not for citation purpose) see a real potential for the embodied reports methodology across educational contexts and sectors to inform learning outcome enhancements. however, we also recognise that methodological improvements to the embodied reports equipment and associated tools are still required when working within vr, mr and, by extension, other emerging learning technology affordances such as ar. future iterations of the research will explore participant feedback and the prototyping of the embodied reports methodology within other clinical simulation contexts beyond paramedicine education. conclusion based on the initial prototyping use of vr simulation in paramedicine education using embodied reports data equipment (cochrane et al. 2018), in this discussion paper we have presented the range of theoretical and practical implications that emerged from the piloting experience and their influence in redesigning practice in paramedicine education. the embodied reports methodology (dañobeitia 2017) is grounded on appropriate epistemological understandings coming from neurophenomenological cognitive science (lutz and thompson 2003; varela 1996), which proved to be applicable within paramedicine clinical simulation based on vr and mr learning experiences. by extension, we also see potential for such theoretical underpinnings to be applied within other technology-enhanced learning situations and contexts (aguayo 2018), yet this remains to be explored. findings also demonstrated the usefulness of a dbr framework for designing mobile vr learning environments (cochrane et al. 2017), particularly in relation to reflecting on applied practice to inform newer theoretical developments and practical solutions. in our case, such solutions involved a conceptual shift from a focus on vr to a focus on mr, with emphasis on the associated benefits offered by mr learning situations within paramedicine education. finally, we see clear benefits in incorporating self-reported and biometric feedback data in paramedicine education and potentially across other technology-enhanced learning situations involving the design of meaningful and authentic learning experiences. references aguayo, c. (2017) ‘ar/vr/mr & rr in learning and teaching digital projects’, theta 2017 conference: connecting minds. creating the future, auckland, new zealand. (online proceedings with no page number) aguayo, c. (2018) ‘exploring autopoietic principles in technology-enhanced learning’, scholarship of technology-enhanced learning (sotel) 2018 inaugural symposium, auckland, new zealand. aguayo, c., cochrane, t. & narayan, v. (2017) ‘key themes in mobile learning: prospects for  learner-generated learning through ar and vr’, australasian journal of educational technology, vol.33, no. 6, pp. 27–40. doi: 10.14742/ajet.3671 ahlert, d., kenning, p. & plassmann, h. (2006) ‘a window to the consumer’s mind: application of functional brain imaging techniques to advertising research’, in international advertising and communication, eds s. diehl & r. terlutter, springer, pp. 163–178. akçayır, m. & akçayır, g. (2017) ‘advantages and challenges associated with augmented reality for education: a systematic review of the literature’, educational research review, vol. 20, pp. 1–11. doi: 10.1016/j.edurev.2016.11.002 http://dx.doi.org/10.25304/rlt.v26.2150 research in learning technology citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 15 (page number not for citation purpose) amiel, t. & reeves, t. c. (2008) ‘design-based research and educational technology: rethinking technology and the research agenda’, journal of educational technology & society, vol 11, no. 4, p. 29. andreassi, j. l. (2010) psychophysiology: human behavior and physiological response, psychology press, new york. babbie, e. r. (2013) the basics of social research, cengage learning, belmont. balconi, m., vanutelli, m. e. & finocchiaro, r. (2014) ‘multilevel analysis of facial expressions of emotion and script: self-report (arousal and valence) and psychophysiological correlates’, behavioral and brain functions, vol. 10, no. 1, p. 32. doi: 10.1186/1744-9081-10-32 baraybar-fernández, a., baños-gonzález, m., barquero-pérez, ó., goya-esteban, r. & de-lamorena-gómez, a. (2017) ‘evaluation of emotional responses to television advertising through neuromarketing’, comunicar: media education research journal, vol. 25, no. 52, pp. 19–28. brouwer, a. m., hogervorst, m., reuderink, b., van der werf, y. & van erp, j. (2015) ‘physiological signals distinguish between reading emotional and non-emotional sections in a novel’, brain-computer interfaces, vol. 2, no. 2–3, pp. 76–89. doi: 10.1080/2326263x.2015. 1100037 cannon, w. b. (1927) ‘the james-lange theory of emotions: a critical examination and an alternative theory’, the american journal of psychology, vol. 39, no. 1/4, pp. 106–124. christopoulos, g. i., uy, m. a. & yap, w. j. (2016) ‘the body and the brain: measuring skin conductance responses to understand the emotional experience’, organizational research methods, vol. 27, no. 1, pp.1–27. doi: 10.1177/1094428116681073 ciorciari, j. (2012) bioelectrical signals: the electroencephalogram, physiology, biophysics, and biomedical engineering, p. 539, boca raton. cochrane, t., et al., (2016) ‘designing virtual reality environments for paramedic education: mesh360’, ascilite 2016 adelaide, pp. 125–135. cochrane, t., et al., (2017) ‘a dbr framework for designing mobile virtual reality learning environments’, australasian journal of educational technology, vol. 33, no. 6, pp. 27–40. doi: 10.14742/ajet.3613 cochrane, t., et al., (2018) ‘designing immersive mobile learning mixed reality for paramedic education’, ieee-tale 2018: education and technology conference, in press. coleman, a. e. & snarey, j. (2011) ‘james-lange theory of emotion’, in encyclopedia of child behavior and development, eds s. goldstein & j.a. naglieri, springer, boston, ma, pp. 844–846. cook, d., et al., (2012) ‘comparative effectiveness of technology-enhanced simulation versus other instructional methods: a systematic review and meta-analysis’, simulation in healthcare, vol. 7, no. 5, pp. 308–320. doi: 10.1097/sih.0b013e3182614f95 cook, d. a., et al., (2013) ‘comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis’, medical teacher, vol. 35, no. 1, pp. 867–898. doi: 10.3109/0142159x.2012.714886 dañobeitia, c. (2017) ‘embodied reports: developing first and third-person methods for a science of experience’, [online] available at: https://www.researchgate.net/publication/327390873_embodied_reports_developing_first_and_third-person_methods_ for_a_science_of_experience dillon, c., et al., (2000) ‘aroused and immersed: the psychophysiology of presence’, proceedings of 3rd international workshop on presence, delft university of technology, delft, the netherlands, pp. 27–28. duffy, e. (1962) activation and behavior, wiley, oxford. ekman, p. (1992) ‘an argument for basic emotions’, cognition & emotion, vol. 6, no. 3–4, pp. 169–200. doi: 10.1080/02699939208411068 ekman, p. & friesen, w. v. (1975) unmasking the face: a guide to recognizing emotions from facial cues, prentice hall, englewood cliffs. http://dx.doi.org/10.25304/rlt.v26.2150 https://www.researchgate.net/publication/327390873_embodied_reports_developing_first_and_third-person_methods_for_a_science_of_experience https://www.researchgate.net/publication/327390873_embodied_reports_developing_first_and_third-person_methods_for_a_science_of_experience https://www.researchgate.net/publication/327390873_embodied_reports_developing_first_and_third-person_methods_for_a_science_of_experience c. aguayo et al. 16 citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 (page number not for citation purpose) ekman, p., levenson, r. w. & friesen, w. v. (1983) ‘autonomic nervous system activity distinguishes among emotions’, science, vol. 221, no. 4616, pp. 1208–1210. doi: 10.1126/ science.6612338 froese, t. & di paolo, e. a. (2011) ‘the enactive approach: theoretical sketches from cell to society’, pragmatics & cognition, vol. 19, no. 1, pp. 1–36. doi: 10.1075/pc.19.1.01fro gil, e., et al., (2010) ‘photoplethysmography pulse rate variability as a surrogate measurement of heart rate variability during non-stationary conditions’, physiological measurement, vol. 31, no. 9, p. 1271. doi: 10.1088/0967-3334/31/9/015 gilissen, r., et al., (2008) ‘parent–child relationship, temperament, and physiological reactions to fear-inducing film clips: further evidence for differential susceptibility’, journal of experimental child psychology, vol. 99, no. 3, pp. 182–195. doi: 10.1016/j.jecp.2007.06.004 guyton, a. c. & john, e. (2003) hall, medical physiology, 10th edn, elsevier health sciences, philadelphia. hall, j. e. (2017) guyton e hall tratado de fisiologia médica, elsevier, brasil. hallowell, r. (2009) ‘humberto maturana and francisco varela’s contribution to media ecology: autopoiesis, the santiago school of cognition, and enactive cognitive science’, proceedings of the media ecology association, media ecology association, saint louis, mo, pp. 143–158. hamstra, s. j., et al., (2014) ‘reconsidering fidelity in simulation-based training’, academic medicine, vol. 89, no. 3, pp. 387–392. doi: 10.1097/acm.0000000000000130 harris, j. m., ciorciari, j. & gountas, j. (2018) ‘consumer neuroscience for marketing researchers’, journal of consumer behaviour, vol. 17, no. 3, pp. 239–252. doi: 10.1002/cb.1710 hase, s. & kenyon, c. (2001) ‘from andragogy to heutagogy’, ultibase articles, vol. 5, no. 3, pp. 1–10. hase, s. & kenyon, c. (2007) ‘heutagogy: a child of complexity theory’, complicity: an inter national journal of complexity and education, vol. 4, no. 1, pp. 111–118. haye, a. & carballo, m. (2017) ‘james’s theory of emotion and affective turn in psychology/ la teoría de las emociones de james y el giro afectivo en psicología’, estudios de psicología, vol. 38, no. 1, pp. 115–139. doi: 10.1080/02109395.2016.1268390 hersen, m., et al., (eds) (2004) comprehensive handbook of psychological assessment, volume 3: behavioral assessment, john wiley & sons, new jersey. hertzman, a. b. (1938) ‘the blood supply of various skin areas as estimated by the photoelectric plethysmograph’, american journal of physiology-legacy content, vol. 124, no. 2, pp. 328–340. jowsey, s. & aguayo, c. (2017) ‘o-tū-kapua (‘what clouds see’): a mixed reality experience bridging art, science, technology in meaningful ways’, teachers and curriculum, vol. 17, no. 2, pp. 95–102. doi: 10.15663/tandc.v17i2.166 kreibig, s. d. (2010) ‘autonomic nervous system activity in emotion: a review’, biological psychology, vol. 84, no. 3, pp. 394–421. doi: 10.1016/j.biopsycho.2010.03.010 kriegel, u. (2013) ‘a hesitant defense of introspection’, philosophical studies, vol. 165, no. 3, pp. 1165–1176. doi: 10.1007/s11098-013-0148-0 konvalinka, i., et al., (2011) ‘synchronized arousal between performers and related spectators in a fire-walking ritual’, proceedings of the national academy of sciences, 108(20), pp. 8514–8519. doi: 10.1073/pnas.1016955108 labarbera, p. a. & tucciarone, j. d. (1995) ‘gsr reconsidered: a behavior-based approach to evaluating and improving the sales potency of advertising’, journal of advertising research, vol. 35, no. 5, pp. 33–54. lahat, d., adali, t. & jutten, c. (2015) ‘multimodal data fusion: an overview of methods, challenges, and prospects’, proceedings of the ieee, vol. 103, no. 9, pp. 1449–1477. lazarus, r. s. & lazarus, r. s. (1991) emotion and adaptation, oxford university press on demand, oxford. levenson, r. w. (1992) ‘autonomic nervous system differences among emotions’, psychological science, vol. 3, no. 1, pp. 23–27. doi: 10.1111/j.1467-9280.1992.tb00251.x. http://dx.doi.org/10.25304/rlt.v26.2150 research in learning technology citation: research in learning technology 2018, 26: 2150 http://dx.doi.org/10.25304/rlt.v26.2150 17 (page number not for citation purpose) levenson, r. w. (1994) ‘human emotion: a functional view’, the nature of emotion: fundamental questions, vol. 1, pp. 123–126. levenson, r. w., carstensen, l. l., friesen, w. v. & ekman, p. (1991) ‘emotion, physiology, and expression in old age’, psychology and aging, vol. 6, no. 1, pp. 28–35. doi: 10.1037/ 0882-7974.6.1.28 luisi, p. l. (2016) the santiago school: autopoiesis and the biologics of life. [online] available at: https://wsimag.com/science-and-technology/19657-the-santiago-school lutz, a. & thompson, e. (2003) ‘neurophenomenology integrating subjective experience and brain dynamics in the neuroscience of consciousness’, journal of consciousness studies, vol. 10, no. 9–10, pp. 31–52. mckenney, s. & reeves, t. c. (2012) conducting educational design research. routledge, london. michels, n., et al., (2013) ‘children’s heart rate variability as stress indicator: association with reported stress and cortisol’, biological psychology, vol. 94, no. 2, pp. 433–440. doi: 10.1016/j.biopsycho.2013.08.005 milgram, p. & kishino, f. (1994) ‘a taxonomy of mixed reality visual displays’, ieice transactions on information and systems, vol. 77, no. 12, pp. 1321–1329. montagu, j. d. & coles, e. m. (1966) ‘mechanism and measurement of the galvanic skin response’, psychological bulletin, vol. 65, no. 5, p. 261. doi: 10.1037/h0023204 nesse, r. m., et al., (1985) ‘endocrine and cardiovascular responses during phobic anxiety’, psychosomatic medicine, vol. 47, no. 4, pp. 320–332. ohme, r., et al., (2009) ‘analysis of neurophysiological reactions to advertising stimuli by means of eeg and galvanic skin response measures’, journal of neuroscience, psychology, and economics, vol. 2, no. 1, p. 21. doi: 10.1037/a0015462 ortony, a. & turner, t. j. (1990) ‘what's basic about basic emotions?’, psychological review, vol. 97, no. 3, p. 315. pike, t. & o’donnell, v. (2010) ‘the impact of clinical simulation on learner self-efficacy in pre-registration nursing education’, nurse education today, vol. 30, no. 5, pp. 405–410. doi: 10.1016/j.nedt.2009.09.013 schioldann, j. (2001) in commemoration of the centenary of the death of carl lange: the lange theory of ‘periodical depressions’-a landmark in the history of lithium therapy, adelaide academic press, burnside. shu, l., et al., (2018) ‘a review of emotion recognition using physiological signals’, sensors, vol. 18, no. 7, p. 2074. doi: 10.3390/s18072074 siedlecka, e. & denson, t. f. (2018) ‘experimental methods for inducing basic emotions: a qualitative review’, emotion review, vol. 11, no. 1, p. 1754073917749016. doi: 10.1177/ 1754073917749016 thayer, j. f., et al., (2012) ‘a meta-analysis of heart rate variability and neuroimaging studies: implications for heart rate variability as a marker of stress and health’, neuroscience & biobehavioral reviews, vol. 36, no. 2, pp. 747–756. doi: 10.1016/j.neubiorev.2011.11.009 tomkins, s. (1962) affect imagery consciousness: volume i: the positive affects, springer publishing company, new york. varela, f. j. (1996) ‘neurophenomenology: a methodological remedy for the hard problem’, journal of consciousness studies, vol. 3, no. 4, pp. 330–349. wilhelm, f. h. & roth, w. t. (2001) ‘the somatic symptom paradox in dsm-iv anxiety disorders: suggestions for a clinical focus in psychophysiology’, biological psychology, vol. 57, no. 1–3, pp. 105–140. doi: 10.1016/s0301-0511(01)00091-6 zendejas, b., wang, a. t., brydges, r., hamstra, s. j. & cook, d. a. (2013) ‘cost: the missing outcome in simulation-based medical education research: a systematic review’, surgery, vol. 153, no. 2, pp. 160–176. http://dx.doi.org/10.25304/rlt.v26.2150 https://wsimag.com/science-and-technology/19657-the-santiago-school doi:10.3402/rlt.v24.31372 original research article inclusion in an age of mobility john traxler* institute of education, university of wolverhampton, wolverhampton, united kingdom (received 21 february 2016; final version received 17 june 2016) learning with mobiles in uk universities is not new and is not novel. it is, in fact, at least 10 years old, well-documented and comparable to activity in universities elsewhere in western europe, america and asia pacific. continued and dramatic changes in the ownership, access and expectations of mobiles amongst university students and equally across uk society have suddenly propelled learning with mobiles to centre-stage as a feasible proposition but, it is now argued, only if students can bring-your-own-device. this has already catalysed discussion about authority, agency and control within university settings but the equally significant and profound implications for the inclusion agenda have not been articulated. this paper begins that process. a theoretical framework for social inclusion in this context is considered, identified and discussed. the paper reviews the progress and problems of the substantial and unique programme of mobile learning across uk higher education since 2000 in relation to its stance on inclusion, where this is apparent. these are all well-documented in academic and official sources; the paper does however also draw on the author’s involvement in many of the events and initiatives. the paper raises however significant questions about this programme’s meaning and direction in a world where now there is more, better, cheaper, faster, newer but different digital technology in the hands of students, potential students and everyone else than there is routinely in the educational institutions themselves. this digital technology, mobile technology, now allows learners to create, own, transform, discuss, discard, share, store and broadcast ideas, opinions, images and information, and to create and transform identities and communities. the paper argues that this epistemological revolution may mean that universities and colleges are no longer credible and authoritative gatekeepers to knowledge and its technologies and so the meaning and relevance of inclusion are much less clear. the paper proposes a new stance on inclusion. keywords: mobility; inclusion; mobile learning; participation; bring-your-own-device introduction in order to understand the case to be articulated, we first need to put it into the recent historical context of the uk higher education policy agenda. through the 1990s, this was characterised by trends, pressures and policies that included inclusion, widening participation and increased opportunity as part of the educational, economic and social policies of centre-left administrations in the uk, and elsewhere in europe (documented, contextualised and critiqued in blanden and machin 2004; macdonald and stratta 2001; osborne 2003). we are discussing inclusion in uk higher education *corresponding author. email: john.traxler@wlv.ac.uk responsible editor: lesley diack, robert gordon university, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 j. traxler. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 in the broadest sense, meaning the incorporation within the mainstream of higher education of minority, marginal, disadvantaged or under-represented groups. we should recognise that inclusion is now contested and needs redefining and is now generally regarded to be more than simply removing barriers or equivalent to widening participation. different phrases have been used for these various different constituencies but together they were part of a larger fundamental change: ‘the change from an elite to a mass system of higher education occurred in the last decade of the twentieth century in virtually all developed societies’ (schuetze and slowey 2002: 309). these initiatives had only partial success, with non-traditional students talking of, ‘a range of experiences of fitting in and standing out in higher education’. for some this involves combining a sense of belonging in both middle-class higher education and working-class homes, while others only partially absorb a sense of themselves as students’ (reay, crozier, and clayton 2010: 107). this discussion of identity runs parallel to any discussions of digital identity, for example the characterisation of students along a spectrum from digital residents to digital visitors (wright et al. 2014; white and le cornu 2011) and a comparative analysis of these moving targets would be valuable and relevant. the moves for greater inclusion took place alongside and perhaps inseparable from this move to mass higher education (described in mayhew, deer, and dua 2004), sometimes called the massification of higher education (contextualised in brennan and naidoo 2008; guri-rosenblit, šebková, and teichler 2007; rossi 2010). throughout this process the assumption, largely unquestioned, seems to have been that these various social groups must be incorporated, assimilated and acculturated into the academy as it was then constituted (discussed from different perspectives in crossan et al. 2003; greenbank 2007; o’donnell and tobbell 2007; reay, crozier, and clayton 2010). the notion that greater inclusion would lead to greater social equality and economic opportunity has however been challenged. an alternative view is that ‘education expansion in a structure defined by unequal access to resources could deepen inequalities’ and that ‘by expanding higher education without paying attention to the mechanisms by which we expanded, we created a two-tier system where the most disadvantaged students paid the most for the least quality of education’ (tressie mcmillan cottom discussing her analysis of usa, quoted in world university news (wun 2015: 4) and in the uk, education in many countries is a ‘race between competing social groups with unequal resources’, ‘that actually reproduces the competitive foundation of inequality’ (professor susan robertson, quoted in a report tellingly entitled, ‘higher education is ‘‘part of the problem of social inequality, not the solution’’’ (the 2015: 8)). meaning earlier longitudinal studies show that . . . he expansion has not been equally distributed across people from richer and poorer backgrounds. rather, it has disproportionately benefited children from relatively rich families. despite the fact that many more children from higher income backgrounds participated in he before the recent expansion of the system, the expansion acted to widen participation gaps between rich and poor children. (blanden and machin 2004: 230) this was at odds with earlier optimism, or perhaps just reflects confusing data, analysis, outcomes and criteria, so j. traxler 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 . . . while these students face numerous and varied barriers to their learning, they are motivated by the impact their studying will have on themselves and their families. of notable significance is how higher education is perceived to reverberate within the home, promoting a culture of learning among, and encouraging the educational aspirations of, children. (wainwright and marandet 2010: 449) information and communications technology (icti) had a role in this process of massification (selwyn and gorard 2003). ‘[t]he acceptance of e-learning throughout the education system was inevitable. the space left for practitioners in higher education is either to embrace the new media enthusiastically or to stand aside and watch its inevitable unfolding’ (clegg, hudson, and steel 2003: 39). there were however only a few attempts to understand the relationships between ict, as it was then understood, and the non-traditional students coming into academia (o’driscoll et al. 2010; jones and lau 2010) but any analysis and recommendations were framed in terms of helping these students to better adjust to the institutional systems being provided. coincidentally, jones and lau link ict and widening participation as two major challenges, widening participation has increased in emphasis in higher education over recent years. while there has been some progress in making higher education more accessible to students who have traditionally been excluded, widening participation is still being described as one of the major challenges facing higher education institutions. likewise, technological development has led to an increased use of online learning in higher education and this has also challenged educators and their practice. (2010: 405) one analysis argued that the massification of higher education without the disproportionately necessary increased resources to support non-traditional students was only possible with the industrialisation of the educational processes that could be delivered by ict (traxler 2010a) � this does however problematise the role of mobile learning, given its roots in ict. recently, an uncritical focus on lecture capture and the flipped classroom (the 2016) may be having the accidental side-effect of making the establishment of the identity student amongst non-traditional entrants more difficult and may ignore some of emotional, social and cultural reasons that brings students onto a campus. the growing use of open and distance learning formats also formed part of the same set of initiatives. reaching into the community using open and distance learning formats did, on the one hand, facilitate the inclusion of disadvantaged groups within the existing bounds of the academy, but it did, on the other hand, have only mixed success in terms of reaching and including non-traditional communities (tonks and farr 2003). our purpose here is to explore the impact of widespread popular ownership of personal mobile digital technologies in the following decade on these formulations of inclusion. we must however now look at the evolution of ict across the same decade. this is outlined in various more-or-less contemporary sources (conole 2002; harasim 2006; jenkins, walker, and voce 2014; nicholson 2007; somekh 2000). it was an era still characterised, because of history, cost and expertise, of ict being an institutional, mostly centralised, phenomenon, not a dispersed personal or social one. this was true of the networked desktop computers and of the software that ran on them, mostly virtual learning environments of varying capacity and sophistication (benfield, roberts, and francis 2006; conole and oliver 2006; mayes and de freitas 2004; research in learning technology citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 3 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 o’neill, singh, and o’donoghue 2004). one judgement was that, globally, ‘conventional institutions of learning have changed far more slowly than the modes of inventive, collaborative, participatory learning offered by the internet and an array of contemporary mobile technologies’ (davidson, goldberg, and jones 2010: 3) and certainly non-pc/post-pc technologies were a challenge to how universities procured, delivered and supported it services at a time of shrinking resources (jenkins et al. 2011). at some point, technology enhanced learning emerged as a more ostensibly scientific and research-focussed term (kirkwood and price 2014; walker, voce, and jenkins 2013) and its implications for inclusion were considered (bell, neary, and stevenson 2009) but at this point e-learning as an institutional phenomenon across the sector had crystallised and solidified as virtual learning environment (vles) functioned largely as repositories and bulletin boards. there is an argument that these repositories house the institutional ip, accessible only through a pay-wall, an integral part of the institutional business model and the antithesis of inclusion based around open learning, the digital analogue of the secure campus. looking backward � mobile learning in higher education meanwhile, popular mobile technologies were changing the nature of communication and content delivery in many industries and in many countries, often dramatically. this dated back to the turn of the century. in education it led to the idea of mobile learning, back to the eu’s flagship mobilearn and m-learning projects starting in 2002, followed over the succeeding decade by hundreds of pilots and projects in schools, colleges, universities and communities in many different parts of the world (kukulska-hulme et al. 2011; traxler 2013). this idea and indeed much of the research in using mobile devices for learning, mobile learning or m-learning as it became known briefly, is about 15 years old. the first mlearn conference, held in 2002 was a landmark. in the intervening decade, mobile learning research projects, driven by exploration of educational theory and exploitation of each new technology, have shown how mobile technologies can motivate engage and enthuse learners, often the most disengaged; can challenge existing educational thinking; can extend the reach of learning to individuals and communities that were previously too difficult or expensive to reach and can enrich and enhance the experience of learning (traxler 2013). through this time the definition of mobile learning was the subject of considerable debate. the early and obvious definitions focussed on mobile technologies as the defining characteristic but later and more thoughtful definitions focussed on the mobility of the learner and of the learning, specifically on its capacity to cross contexts, from for example home to school, formal to informal and field-trip to lecture theatre (el-hussein and cronje 2010; laouris and eteokleous 2005; park 2011). throughout this first decade, the projects revolved however around mobile technologies, around implementations and deployment that were relatively expensive, fragile, formal, small-scale, short-term, institutional and subsidised, taking place in a benign albeit deceptive and deteriorating, global economic climate (eaton et al. 2011; imbs 2010). this debate, expressing how mobile learning was conceptualised, is important for our discussion of inclusion. these mobile technologies were increasingly advanced and engaging but never became fully and formally integrated into higher education, despite the massive potential. generally, and unlike other educational icts, for example pcs, data j. traxler 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 projectors or tvs, they are personal, cheap and found at the bottom-of-the-pyramid in socio-economic terms. these are two reasons for asserting that mobile technologies have a massive educational potential (winters 2013). these projects grew out of the ideas and lexicon of innovation, for example, early adopters, critical mass and change agents, that were popular at the time; they were often funded as developmental projects, intended to become established within institutions by a process of downward and outward diffusion and intended to become embedded and mainstreamed (rogers 2010). this may or may not have happened but generally, the finances and culture were against them (coverdale-jones 2012; romiszowski 2004; russell 2009; salmon 2005; sharpe, benfield, and francis 2006; white 2007). the finances were against them because the innovation looked like an extra direct cost for an un-quantified benefit and the culture was against them because innovators were driven by very different ideals and objectives compared to mainstream lecturers, their managers and their quality assurance regimes (sharpe et al. 2005). the evidence was not always convincing or apparent but the extra costs were (traxler and kukulska-hulme 2006) and subsequently policy changes reduced the earlier access to funds for innovation (tonkiss and dommett 2013). therefore, most mobile learning projects from this era never got beyond a pilot phase and researchers moved on from one innovation to the next innovation, in a parallel universe of research funding (ng and cumming 2015). it was probably the case in the uk that the agencies funding and supporting mobile learning innovation, deployment, evaluation and dissemination, mainly becta (originally the british educational communications and technology agency, then just becta, closed in 2011), jisc (originally the joint information services committee, www.jisc.ac.uk, now just jisc) and perhaps the higher education academy (hea) subject centres never had the resources, priorities or confidence to create enough projects across their respective sectors and disciplines to achieve a critical mass and sustaining community. it was also the case that to some extent the funding agencies moved on from the mind-set of innovation, preferring to addressing change at a more systemic and institutional level, leaving mobile learning in a limbo. these projects were usually funded across a year or two, with a handful of staff, the enthusiasts and visionaries, alongside rather than inside the core assessed curriculum, and all these features militated against an embedded sustainable future. hardware, that is mobiles or earlier personal digital assistant (pdas), was usually built into the budget. it guaranteed a uniform and consistent platform and experience, removed a confounding variable and reduced technical problems. it also limited the size of any sample and produced no exit strategy. it had no sustainability in terms of finance or culture. latterly, the mobile-specific initiatives merged with educational exploration of other popular digital technologies such as podcasts, micro-blogging and social networks but by this time the national funding environment both for innovation and for subsequent embedding had become distinctly bleak (traxler 2016). looking back, one brief exception to this account might be the molenet programme of the learning and skills network (lsn, closed in 2011), putting mobile learning hardware and infrastructure into the further education sector from 2007 to 2010. this grew out of the eu project, m-learning � one that specifically addressed mobile technology and inclusion (attewell and savill-smith 2004) � and various smaller successor projects that built capacity and credibility within lsda and its funders, and meant that lsn (the successor in 2006 to the learning and research in learning technology citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 5 (page number not for citation purpose) www.jisc.ac.uk http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 skills development agency, the lsda) could grasp the opportunity of learning and skills council (lsc) funding when it became available. this was not however a direct consequence of evidence and evaluation, more of a certain local climate and specific relationships. by the same token, the programme, funded on a year-by-year basis for 3 years to the eventual tune of £14 m, was not explicitly tasked with producing further evidence nor did it prioritise external evaluation. (in fact the dispersed and heterogeneous nature of the projects compounded any difficulties with evaluation as did the programmes rather general objectives.) this perhaps tells us that pilot projects and their outputs and evidence have not always played the kind of primary role that researchers would imagine, and that change takes place in other ways. the restrictions imposed on the funding by the lsc meant it was not only short-term and un-predictable but also intended only for capital hardware, neither for devices that were essentially disposable nor for their connectivity. this may seem a minor point but illustrates how the procedures of institutional procurement and support have trouble as their focus moves from large stable desktop devices to many small personal ones and is one more hurdle to embedding mobile learning within the established practices of e-learning (attewell et al. 2010; parker 2010). the early mobile learning projects also grew out of the e-learning of the time, out of the e-learning community, its aspirations and objectives, and seemed to offer learning anywhere, anytime. indeed, many early projects attempted to port e-learning systems, for example vles, from desktop computers onto mobile phones, whilst others more adventurously tried to incorporate the affordances of mobile phones, for example image-capture or location-awareness, into an e-learning ethos. they also borrowed extensively from the e-learning pedagogy of the time, specifically in the uk the social constructivism expressed in laurillard’s conversational framework (laurillard 2007) stretching what was a conceptualisation of formal learning further into the rapidly evolving, technically-mediated informal. all of these projects did however work within the confines of the existing institutions and their established curricula, either enriching or enhancing them or extending their reach to people and communities otherwise too difficult or costly to reach, whatever the rhetoric of transformation and paradigm shift (rajasingham 2011). subsequent notions of students as agents of change (healey 2012) and learnergenerated content (narayan 2011) may be only technical fixes that also ignore the more systemic challenges outlined in the current paper. time will tell. the end result did however seem be to a pedagogic research movement that somehow had difficulty becoming embedded inside the institutions and thereby achieving its full potential, a set of interesting projects and results but not ones that spoke to the higher education sector as a whole. the reason for this historical account is to identify what differentiates the past from the future (traxler 2012). looking forward � the social trends of mobile technology in the coming decade the technology of mobiles will continue the trend of becoming more popular, personal, robust, cheap and social. the technology has already become democratic, or rather has become more demotic, in nature and society itself has become mobile and connected. it has become increasingly difficult to imagine everyday life before or without mobile technology as its functionality and capability have increased, as a generation of young people have matured and as network take-up, j. traxler 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 competition and coverage have increased to near saturation. this is leading to a new world, with new communities, expectations and behaviour. there is a rich and growing literature describing the profound and transformative impact of the ownership of mobile technologies on most aspects of our societies and cultures. significant contributions include agar (2003), fortunati (2002), geser (2004), goggin (2006), katz and aakhus (2002), kirby (2009), ling (2004), ling and donner (2009), nyı́ri (2007), plant (2002) and urry (2007), ranging from the empirical and quotidian to the speculative and philosophical. they describe from different perspectives how these technologies transform social, economic, cultural and personal behaviour and change ideas about � identity, community, relationships � ethics, behaviour & expectations � jobs, work, employment, economy � creativity, self-expression, artistic production � learning, understanding, knowing the most obvious and the tangible of these is the transformation, at national, organisational and individual levels, of the artefacts, resources, commodities and assets that constitute economic life and the ways that we, as individuals, organisations and nations, produce, consume and exchange them; these technologies also transform the nature of much work itself by facilitating remote and extended working, out of hours, off the premises, and by supervising and monitoring and potentially deskilling peripatetic and dispersed workers (dutta 2009). other obvious transformations are in forms of artistic expression, creating or mutating genres for art, creating new artists and new markets for their work, from blogs and flash mobs to ringtones and downloads (for example, epstein 2004; lillie 2012; senft and baym 2015), in our spiritual life (cho and campbell 2015; williams and gray 2010) and in our political life (hermanns 2008; miard 2012) as the old sedentary institutions and organisations lose touch with the values and concerns of people growing up in a different world; and to crime and wrong-doing from blackberry-enabled rioters to trolling, sexting, happyslapping, blue-jacking and cyber-sex (traxler 2009). this short account clearly abbreviates and exaggerates something far more complex and subtle but the points are nevertheless relevant to our discussion. the implications for education are manifold. even if the role of education is only to service the economy, the nature of that economy is changing rapidly, in ways in which the static institutions of education might be ill suited to monitor. another role for education is to address exclusion and disadvantage and we shall see how these too might become transformed in a mobile and connected society. not only are the facts of a mobile and connected society important for education but so are its fictions, the moral panics that cite mobile technologies as the cause of students who stay in bed, the decline in literacy, the deterioration in manners and the rise of the couch potato (coleman 2010). educators sadly must view and review their mobile learning innovations in this light. there are also many accounts of how mobility and connection change how we think of ourselves, our identities, our affiliations, our relationships; nowadays many people have multiple on-line identities, sometimes even within the same cyber-space domain and sometimes different genders. these are not merely their real identities and personalities enacted on-line � any more than their avatar in secondlife is merely research in learning technology citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 a collection of pixels � nor are they somehow less real than the real ones, not a sort of online fancy-dress or impersonation but are actually and obviously authentic, locations where emotions and values are established and transformed (see for example, stald 2008). the implications for education are the transience, fragmentation and complication of the identities and communities being served, potentially changing the ideas of out-reach and recruitment as people shift the places and spaces that they inhabit (childs 2010). a different facet is the extent to which mobiles are embodied or prosthetic, part of us, inseparable, an umbilical cord to much that we now value, the last thing at night, the first thing in the morning; organisations expecting to separate people from their mobiles threaten these attachments (pertierra 2005; rettie 2005; vincent 2006), though they also represent yet another technology of surveillance (kietzmann and angell 2010). these changes drive further changes in expectations about behaviour, about what is good, acceptable, appropriate and okay in our interactions, our relationships, our conversations; our ideas about what is correct, ethical. this happens as the mobile phone intrudes more and more into everyday life and as the mobile phone is increasingly the portal to online content, activities and communities. what defines and characterises a community is a shared consensus about ethics and expectation. what is acceptable as a gesture, an interaction or a topic in one community is not necessarily so in another, and online communities as opposed to physical communities are much more volatile and tacit � offense is easily given in the wrong place and educators find themselves venturing into the equivalent of foreign countries with strange customs and traditions when they take or seek students in cyberspace (farrow 2011; traxler 2011). the mobile phone is also becoming an essential component or presence of every face-to-face social and inter-personal interaction, in the café, the conference, the classroom and the concert, the street corner, the bus and the pub, and these interactions change and the rules evolve, quickly or slowly, willingly or unwillingly, consensually or conflictedly. educators must work with and within a society where the private and social online spaces of music, community and interaction intrude to the physical and educational spaces and where for example phone calls interrupt conversations, classes and concerts (katz and aakhus 2002). joining or creating communities, online or otherwise, means acceding to a set of expectations about how to behave and how to interact; in the online world, these communities are fragmentary, transient and complex, accessible more to those digital native learners than many of their digital immigrant teachers, if we may use such a flimsy generalisation. connected universal mobile devices, the portal onto web2.0 services, also change the nature of learning and knowing. everyone with a smartphone and a network can generate, store, share, discuss and consume images, ideas, information and opinions, can access the cloud, and the services it provides, and can access each other; they can pursue, sustain or invent interests specific to them, their location, their community and their history. we could appropriate the phrase knowledge economy and use it to stand for the processes and transactions by which knowledge is produced, owned, distributed, stored, reworked, shared and consumed; we are talking now about a move away from a knowledge economy based around specialised knowledge-factories producing durable standardised knowledge to a more local, dispersed, grass-roots cottage-industry producing and distributing more varied knowledge for highly segmented markets. j. traxler 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 this is the meaning behind the phrase epistemological revolution (des bordes and ferdi 2008). this sounds like education by another name, but an education without the gatekeepers, barriers and constraints of most schools, colleges and universities, and without the support, standards, structure, stability and incentives of these established institutions. the challenge to education systems is of course the shift or discrepancy in control, authority and agency represented not by the technologies themselves but by the social changes around them. this is perhaps at the heart of notions that mobile learning is disruptive, not just a nuisance, but profoundly disruptive (sharples 2002). we should perhaps point out that we are not making the kind of generational case exemplified by distinctions such as digital native/digital immigrant (prensky 2010), widely used to understand change amongst students within the university system (jones 2012). looking backwards and forward � the tipping point this paper is describing the pressure or momentum for sustainable educational innovation with mobiles shifting from top-down, which usually did not happen for financial reasons, to outside-in, where the expectations and attitudes acquired by learners outside the institution, out in real life, were brought into the institution, sometimes uncomfortably and certainly not with any consistency, coherence or homogeneity. these two brief accounts, looking backwards and looking forwards, allow us to describe the present as a tipping point, one with considerable consequences for the nature of education, especially for the uk and those other countries with widespread and sophisticated take-up of mobiles amongst students. this has a systemic significance, beyond just the educational technology community. traditionally the institutions of education, especially universities, have acted as the custodians, enablers, gatekeepers and arbiters of higher learning and latterly also the custodians and gatekeepers to the technologies of learning. these were usually for reasons of cost and complexity beyond the reach of private citizens. if these private citizens want access to learning then they would do so on the terms and technologies specified by the institutions that procured, provided and controlled the technologies of their choice. they did this in order to provide equality and equity of access, experience and opportunity, to guarantee reliable access to resources and transactions of a safe, consistent and acceptable technical and educational standard. the way forward � bring-your-own-device, . . . and services into this rapidly evolving situation came the idea that many of the resource issues associated with sustainable mobile learning might be solved if educational institutions encouraged learners to bring their own devices. the bring-your-own-device (byod) debate, in the form of discussions about learner devices (cosn 2012; traxler 2010b) sponsored by becta, actually goes back several years before its emergence in its present form in the corporate and us domains. it is a debate in several different levels, the obvious and easiest being the financial one. ucisa reports its members, the it service department of uk universities, feel themselves (ucisa 2008) unable to sustain the current level of it provision of even the existing it portfolio � this is problematic in terms of students’ expectations even before we research in learning technology citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 add mobile devices into the equation. many students will have experience of newer and better computers than those available in their university. in the usa, where the phrase byod has gained most visibility before making its way back to the uk, the acronyms byot and byos, referring to technologies and services, have also surfaced, drawing our attention to the fact that we cannot separate users’ devices from the services such as social networking, micro-blogging, instant messaging, cloud computing, gaming and whatever else they choose to access. the device is the trojan horse for the services. there are several local or tactical problems with byod. firstly, the technical and infrastructural challenges, secondly the operational aspects and thirdly, the ethical and political issues. none of these are insurmountable but they may account for the current timidity in the uk � as we implied earlier mobiles are notorious for their capacity to generate moral panics in the media (goggin 2012). the first challenges do not intrinsically relate to the ownership of devices, more to their sheer numbers, diversity and unpredictability and to concerns about the capacity of existing staff and infrastructure to cope safely and reliably. these are often resource issues in the sense that they are issues that could be solved by money. we should remember that byod might deliver substantial savings by reducing or eliminating the annual hardware replacement requirement and turning it suites into general teaching rooms. resources would be transferred to learner advice and support. it services must shift from managing a limited number of technologies procured by the institution to supporting a potent unlimited number of technologies accessed by learners, many of whom may not be adequately digitally literate, in the sense of understanding the educational potential or the principles behind the technologies they daily use. the second set of aspects, the operational, is essentially about educational quality and the measures needed to ensure, again irrespective of ownership, that students have an adequate educational experience, equivalent perhaps to that before mobiles were introduced into the blend. these are regulatory issues and cover content, delivery and training. mobile device management was much in evidence. services such as airwatch 1 offer institutional managers a practical way to ensure their networks can support a wide variety of approved devices and authenticate their users, enabling tracking of what is being done on the networks � when, what and by whom and in detail. these services give the assurance that learners are operating within the agreed norms and that revised codes of acceptable use can be enforced. the third set of issues, the ethical and political ones, is centred on reconciling (consumer) choice with equality of access to educational experiences and opportunities, and to inclusion and opportunity. with an impersonal institutional technology such as the desktop pc, the expectation was historically that the institution procured, provided and controlled these and the buildings that housed them. furthermore, provision must recognise how personal and private, indeed intimate, this technology is and the strength of personal preference and attachment. any policy must recognise that mobile devices are unlike other icts in terms of equity. equity in this context implies equity of hardware and on-going equity of connectivity. furthermore, even the early days of mobile learning, research (traxler and riordan 2003) showed that providing � giving or lending � students with a university device was providing them with a second and secondary device, one they had not chosen, and it was soon forgotten or discarded. addressing the equity issue j. traxler 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 by the means-tested provision of devices compounds this problem by potentially stigmatising the recipients, the digital equivalent of free school lunches. if these arguments were in principle plausible, then the next steps would be to look at quick-easy-wins, those changes and initiatives that would save money, increase institutional confidence and pose no risk specific to each institution. jisc have documented some in their mobile learning infokit (jisc 2012); one example is increased use of sms messaging, another is deploying web-based apps for student support, a third might be discussion about changed codes of acceptable use. so, if we believe that learning with mobiles based on byod is potentially sustainable but potentially disruptive, we need to address the likely byod scenarios, and the implications in terms of it support and education support. we must also explore the challenges to leaders, teachers and institutions, the changed nature of acceptable use, and the implications for privacy, etiquette and trust. as we have said, however, this will not merely be a discussion around the devices but also the services that they afford; they are inseparable, learners always bring both. this discussion must however be local to the culture and ethos of each institution, to its history, mission and resources. access to university education � the inclusion agenda we now take the discussion in a different direction and start to explore the ramifications of our analysis for the inclusion agenda described earlier. this in the uk has been a decade-long political project to increase the representation in the national student body of various different categories of people outside the traditional demographics of able-bodied, middle-class, white male school-leavers, a project to represent non-traditional students, that is, to represent women, older people, ethnic minorities and those with disabilities. increasing inclusion, and participation and access, involved various programmes and initiatives, all predicated on the understanding that learning and the technologies that supported, enhanced and delivered it were the attributes or characteristics of institutions, of the existing institutions, and so inclusion was the process of getting under-represented communities into those institutions and supporting them once they were inside. access and inclusion � whom to what looking back, what we see could be interpreted as an increasing challenge to the relevance and authority of our educational institutions as they struggle, or do not struggle, to adapt to a world very different from that for which they were designed, even for the economy they were supposed to serve. this brings us to the critical and novel part of the discussion. we have talked about inclusion and participation from the perspectives of institutions that historically acted as gatekeepers to education and gatekeepers to the technologies of education, essentially facilitating people from outside the institution coming in and partaking of the institution, and undergoing the acculturation and identity transformation implicit in education, alongside the explicit curriculum. they acquired the identity student as the way to subsequently acquire the identity professional person. meaning that the process of inclusion was the process of coming on to privileged territory and being assimilated into the culture of its existing inhabitants. there is a risk, in these research in learning technology citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 discussions, of talking of the disadvantaged and perpetuating the language of othering. we could however view inclusion (and assimilation into university norms and culture) as being predicated on just such othering, and our current critique of inclusion as an antidote to this. some of the literature of participation described it in just these terms, of nontraditional, that is working class, students undergoing a kind of identity transformation � to a greater or lesser extent depending on the prestige and traditions of the institution � as they respond to an implied curriculum expressing the cultural, socio-economic and cognitive norms implicitly defined by institutions to which they seek admission (hinton-smith 2012; measor, wilcox, and frame 2012; reay 2002). we argue that the apparent solidity of these norms is no longer as authoritative or as appropriate at it used to be, and many of the reasons are implicit and intrinsic in our essentially mobile and connected society (traxler 2008). our argument has been that the static and sedentary nature of institutions, and the procedures, their professions and their estates, are becoming increasingly irrelevant and so we must revitalise and re-orient the idea of inclusion and think in future of education being included again in the community rather than hoping to preserve the idea of including the community in education. this is now the challenge, to ask what does inclusion means after the tipping point. the changed and challenged inclusion agenda policies for participation, inclusion and access are drivers for institutional resources and priorities. our concern is to draw attention to the changed environment in which they operate, though without ignoring or under-stating those aspects of exclusion untouched by the digital and by the novel. bad housing, unemployment and poor health will always need recognition in resources, policies and priorities. this paper has a specific agenda amongst the more general discussion of the changing nature of learning with mobiles in uk higher education and that is to argue for a paradigm shift in the inclusion agenda; quite literally a paradigm shift in the terms that kuhn (2012) would have recognised, requiring the paradigm’s centre to move from the institutional environment to the learners’ social and cultural environment, to think of inclusion no longer as the inclusion of learners into the institutions but instead to think of inclusion as the inclusion of institutions amongst learners and their communities, recapturing or reinventing institutional relevance, authority and legitimacy. this resonates with schon’s (1973) work on the decline of the stable state, examining the idea of knowledge generation activity at the edge rather than direction from the centre. this is particularly relevant when mobility and social connectedness allow new ways of generating knowledge. he notes that, ‘the movement of learning is as much from periphery to periphery, or from periphery to centre, as from centre to periphery’ (1973: 165). conclusions to summarise, the capacity of mobile technologies to generate, share, store, access and consume ideas, opinions, information and images, specific to people, locations, communities and their contexts means that they are a quintessential web2.0 technology, challenging the stability and authority of the established educational forms and stakeholders. previous generations of technology enhanced learning j. traxler 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 have exploited expensive, complex institutional systems, such as vles, running on institutional hardware, networks and infrastructure, and resonated with ideas of inclusion that involved bringing the excluded, the marginal and the disenfranchised into the institution, to exploit its experts, its expertise and its resources, to learn from them. by comparison mobile devices, social networks and other popular digital technologies facilitate the generation, valorisation, transformation, sharing, dissemination and production of the ideas, images and information and the opinions that constitute much of education and its acquisition but taking place outside the institution, its education professionals and outside their control and jurisdiction, challenging the established practices and standards of education and challenging the older notions of inclusion. there are massive implications for the practice and philosophy of inclusion, participation and access. education could be transformed in a world where academic institutions and their technologies are no longer the gatekeepers or the arbiters to learning. the sustainable use of popular digital technologies especially mobile devices, depends on byod strategies but this significantly alters the dynamics of agency, control and direction within the institutions, within the curricular and within the educational professions. the implications for policies of inclusion are that these must be re-framed; the institutions, curricular and professions of education must themselves seek inclusion within the changed mobile and connected worlds of the communities, of whatever sort, that they seek to serve. the alternative is local educational versions of an arab spring that sweeps away stable and sedentary but sadly irrelevant institutions, organisations, professions and procedures (traxler 2015). we argue for a shift to a theoretical framework or paradigm for inclusion that puts students and their digital habitus at the centre of inclusion. this is clearly a big challenge since we have characterised this emergent culture and society as fluid, fragmented, partial and subjective and the wider world as teetering on economic, ecological and political crises where the point and purpose of education is no longer clear (traxler and lally 2016). acknowledgements the author is grateful for the friendly critical feedback of professors peter lavender and alan tuckett. note 1. see http://www.air-watch.com/solutions/bring-your-own-device-byod/ references agar, j. (2003) constant touch, icon, cambridge. attewell, j. & savill-smith, c. (2004) ‘mobile learning and social inclusion: focusing on learners and learning’, learning with mobile devices, research and development conference (mlearn), rome, pp. 3�11. attewell, j., et al., (2010) modernising education and training: mobilising technology for learning, learning and skills network, london. bell, l., neary, m. & stevenson, h. (2009) the future of higher education: policy, pedagogy and the student experience, a&c black, london, uk. benfield, g., roberts, g. & francis, r. (2006) the undergraduate experience of blended e-learning: a review of uk literature and practice, higher education academy, york. research in learning technology citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 13 (page number not for citation purpose) http://www.air-watch.com/solutions/bring-your-own-device-byod/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 blanden, j. & machin, s. (2004) ‘educational inequality and the expansion of uk higher education’, scottish journal of political economy, vol. 51, no. 2, pp. 230�249. brennan, j. & naidoo, r. (2008) ‘higher education and the achievement (and/or prevention) of equity and social justice’, higher education, vol. 56, no. 3, pp. 287�302. childs, m. (2010) learners’ experience of presence in virtual worlds, doctoral dissertation, institute of education, university of warwick. cho, k. j. & campbell, h. a. (2015) ‘religious use of mobile phones’, in encyclopedia of mobile phone behavior, ed z. yan, hershey, pa: igi. clegg, s., hudson, a. & steel, j. (2003) ‘the emperor’s new clothes: globalisation and e-learning in higher education’, british journal of sociology of education, vol. 24, no. 1, pp. 39�53. coleman, e. g. (2010) ‘ethnographic approaches to digital media’, annual review of anthropology, vol. 39, pp. 487�505. conole, g. (2002) ‘the evolving landscape of learning technology’, research in learning technology, vol. 10, no. 3, pp. 4�18. conole, g. & oliver, m. (eds) (2006) contemporary perspectives in e-learning research: themes, methods and impact on practice, routledge, abingdon, uk. cosn. (2012) making progress: rethinking state and school district policies concerning mobile technologies and social media, consortium for school networking (cosn), washington, dc. coverdale-jones, t. (2012) ‘success factors in the implementation of e-learning in a uk higher education institution’, in post-secondary education and technology: a global perspective on opportunities and obstacles to development, palgrave macmillan, eds r. clothey, s. austin-li & j. weidman, london, uk. pp. 111. crossan, b., et al., (2003) ‘understanding participation in learning for non-traditional adult learners: learning careers and the construction of learning identities’, british journal of sociology of education, vol. 24, no. 1, pp. 55�67. davidson, n., goldberg, d. t. & jones, m. j. (2010) the future of thinking: learning institutions in a digital age, the mit press, cambridge, ma. des bordes, a. & ferdi, s. (2008) ‘do knowledge and new technologies need a new epistemology?’ proceedings of bobcatsss 2008 providing access to information for everyone, zadar, croatia. dutta, s. (ed) (2009) global information technology report 2008�2009, world economic forum, geneva, switzerland. eaton, j., et al., (2011) trade and the global recession (no. w16666), national bureau of economic research, cambridge, ma. el-hussein, m. o. m. & cronje, j. c. (2010) ‘defining mobile learning in the higher education landscape’, journal of educational technology & society, vol. 13, no. 3, pp. 12�21. epstein, m. (2004) moving fiction: novelists, technology designers, and the art of the exchange, doctoral dissertation, massachusetts institute of technology. farrow, r. (2011) mobile learning: a meta-ethical taxonomy, available at: http://oro.open. ac.uk/29149/2/5f9deac7.pdf fortunati, l. (2002) ‘the mobile phone: towards new categories and social relations’, information communication & society, vol. 5, no. 4, pp. 513�528. geser, h. (2004) towards a sociological theory of the mobile phone, university of zurich, zurich. goggin, g. (2006) cell phone culture, routledge, abingdon, uk. goggin, g. (2012) cell phone culture, routledge, abingdon, uk. greenbank, p. (2007) ‘introducing widening participation policies in higher education: the influence of institutional culture’, research in post-compulsory education, vol. 12, no. 2, pp. 209�224. guri-rosenblit, s., šebková, h. & teichler, u. (2007) ‘massification and diversity of higher education systems: interplay of complex dimensions’, higher education policy, vol. 20, no. 4, pp. 373�389. harasim, l. (2006) ‘a history of e-learning: shift happened’, in the international handbook of virtual learning environments, eds j. weiss, j. hunsinger, j. nolan & p. trifonas, springer, dordrecht, the netherlands, pp. 59�94. healey, m. (2012) ‘students as change agents’, international society for the scholarship of teaching and learning conference. ‘‘research on teaching and learning: integrating practices’’ in hamilton, canada (october 24�27). j. traxler 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://oro.open.ac.uk/29149/2/5f9deac7.pdf http://oro.open.ac.uk/29149/2/5f9deac7.pdf http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 hermanns, h. (2008) ‘mobile democracy: mobile phones as democratic tools’, politics, vol. 28, no. 2, pp. 74�82. hinton-smith, t. (2012) widening participation in higher education: casting the net wide? palgrave macmillan, london. imbs, j. (2010) ‘the first global recession in decades’, imf economic review, vol. 58, no. 2, pp. 327�354. jenkins, m., et al., (2011) ‘the development of technology enhanced learning: findings from a 2008 survey of uk higher education institutions’, interactive learning environments, vol. 19, no. 5, pp. 447�465. jenkins, m., walker, r. & voce, j. (2014) ‘achieving flexibility? the rhetoric and reality of the role of learning technologies in uk higher education’, in rhetoric and reality: critical perspectives on educational technology. proceedings ascilite dunedin, eds b. hegarty, j. mcdonald & s.-k. loke, ascilite, tugun, queensland, australia, pp. 544�548. jisc. (2012) ‘mobile learning infokit’, available at: https://jisc.ac.uk/guides/mobile-learning jones, c. (2012) ‘networked learning, stepping beyond the net generation and digital natives’, in exploring the theory, pedagogy and practice of networked learning, eds l. dirckinckholmfeld, v. hodgson & d. mcconnell, springer, new york, pp. 27�41. jones, n. & lau, a. m. s. (2010) ‘blending learning: widening participation in higher education’, innovations in education and teaching international, vol. 47, no. 4, pp. 405�416. katz, j. e. & aakhus, m. (eds) (2002) perpetual contact � mobile communications, private talk, public performance, cambridge university press, cambridge, uk. kietzmann, j. & angell, i. (2010) ‘panopticon revisited’, communications of the acm, vol. 53, no. 6, pp. 135�138. kirby, a. (2009) digimodernism: how new technologies dismantle the postmodern and reconfigure our culture, bloomsbury publishing, new york, ny, usa. kirkwood, a. & price, l. (2014) ‘technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? a critical literature review’, learning media and technology, vol. 39, no. 1, pp. 6�36. kuhn, t. s. (2012) the structure of scientific revolutions, university of chicago press, chicago, il. kukulska-hulme, a., sharples, m., milrad, m., arnedillo-sánchez, i., & vavoula, g. (2011) ‘the genesis and development of mobile learning in europe’, in combining e-learning and m-learning: newapplications of blended resources, ed d. parsons, igi global, hershey, pa, pp. 151�177. laouris, y. & eteokleous, n. (2005) ‘we need an educationally relevant definition of mobile learning’, proceedings of the 4th world conference on mobile learning, cape town, pp. 290�294. laurillard, d. (2007) ‘pedagogical forms of mobile learning: framing research questions’, in mobile learning: towards a research agenda, ed n. pachler, wle centre, ioe, london, available at: http://eprints.ioe.ac.uk/627/1/mobile_c6_laurillard.pdf lillie, j. (2012) ‘nokia’s mms: a cultural analysis of mobile picture messaging’, new media & society, vol. 14, no. 1, pp. 80�97. ling, r. (2004) the mobile connection � the cell phone’s impact on society, morgan kaufmann publishers, san francisco, ca. ling, r. & donner, j. (2009) mobile communication, polity, cambridge. macdonald, c. & stratta, e. (2001) ‘from access to widening participation: responses to the changing population in higher education in the uk’, journal of further and higher education, vol. 25, no. 2, pp. 249�258. mayes, t. & de freitas, s. (2004) review of e-learning theories, frameworks and models, jisc e-learning models desk study (1), london, the joint information systems committee. available at: http://www.jisc.ac.uk/elp_outcomes.html mayhew, k., deer, c. & dua, m. (2004) ‘the move to mass higher education in the uk: many questions and some answers’, oxford review of education, vol. 30, no. 1, pp. 65�82. measor, l., wilcox, p. & frame, p. (2012) ‘transformation or trauma: the transition to higher education of non-traditional students’, in widening participation in higher education: casting the net wide? palgrave macmillan, london, uk. 146 p. miard, f. (2012) ‘call for power? mobile phones as facilitators of political activism’, in cyberspaces and global affairs eds s. s. costigan & j. perry cyberspace & global affairs, routledge, abingdon, uk. pp. 119�144. research in learning technology citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 15 (page number not for citation purpose) https://jisc.ac.uk/guides/mobile-learning http://eprints.ioe.ac.uk/627/1/mobile_c6_laurillard.pdf http://www.jisc.ac.uk/elp_outcomes.html http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 narayan, v. (2011) learner-generated content as a pedagogical change agent, ascilite, hobart, tasmania, australia. ng, w. & cumming, t. m. (eds) (2015) sustaining mobile learning: theory, research and practice, routledge, london. nicholson, p. (2007) a history of e-learning. computers and education, springer, dordrecht, the netherlands. pp. 1�11. nyı́ri, k. (2007) mobile studies: paradigms and perspectives, passagen verlag, austria hungarian academy of sciences, vienna. o’donnell, v. l. & tobbell, j. (2007) ‘the transition of adult students to higher education: legitimate peripheral participation in a community of practice?’, adult education quarterly, vol. 57, no. 4, pp. 312�328. o’driscoll, m., et al., (2010) how can the university use e-learning to improve the learning experience of non-traditional students? university of surrey, guildford, uk. o’neill, k., singh, g. & o’donoghue, j. (2004) ‘implementing e-learning programmes for higher education: a review of the literature’, journal of information technology education: research, vol. 3, no. 1, pp. 313�323. osborne, m. (2003) ‘increasing or widening participation in higher education? � a european overview’, european journal of education, vol. 38, no. 1, pp. 5�24. park, y. (2011) ‘a pedagogical framework for mobile learning: categorizing educational applications of mobile technologies into four types’, the international review of research in open and distributed learning, vol. 12, no. 2, pp. 78�102. parker, g. (2010) molenet mobile learning conference 2009: research papers, learning and skills network, (nj3), london. pertierra, r. (2005) ‘mobile phones, identity and discursive intimacy’, human technology: an interdisciplinary journal on humans in ict environments, vol. 1, no. 1, pp. 23�44. plant, s. (2002) on the mobile � the effects of mobile telephones on individual and social life, motorola. available at: http://www.momentarium.org/experiments/7a10me/sadie_plant. pdf prensky, m. (2010) teaching digital natives: partnering for real learning, sage, london. rajasingham, l. (2011) ‘will mobile learning bring a paradigm shift in higher education?’ education research international. available at: http://www.hindawi.com/journals/edri/ 2011/528495/ reay, d. (2002) ‘class, authenticity and the transition to higher education for mature students’, the sociological review, vol. 50, no. 3, pp. 398�418. reay, d., crozier, g. & clayton, j. (2010) ‘‘‘fitting in’’ or ‘‘standing out’’: working-class students in uk higher education’, british educational research journal, vol. 36, no. 1, pp. 107�124. rettie, r. (2005) ‘presence and embodiment in mobile phone communication’, psychology journal, vol. 3, no. 1, pp. 16�34. rogers, e. m. (2010) diffusion of innovations, simon and schuster, london, uk. romiszowski, a. j. (2004) ‘how’s the e-learning baby? factors leading to success or failure of an educational technology innovation’, educational technology, vol. 44, no. 1, pp. 5�27. rossi, f. (2010) ‘massification, competition and organizational diversity in higher education: evidence from italy’, studies in higher education, vol. 35, no. 3, pp. 277�300. russell, c. (2009) ‘a systemic framework for managing e-learning adoption in campus universities: individual strategies in context’, association for learning technology journal, vol. 17, no. 1, pp. 3�19. salmon, g. (2005) ‘flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions’, research in learning technology, vol. 13, no. 3, pp. 201�218. schon, d. a. (1973) beyond the stable state: public and private learning in a changing society, penguin, harmondsworth, uk. schuetze, h. g. & slowey, m. (2002) ‘participation and exclusion: a comparative analysis of nontraditional students and lifelong learners in higher education’, higher education, vol. 44, no. 3�4, pp. 309�327. selwyn, n. & gorard, s. (2003) ‘reality bytes: examining the rhetoric of widening educational participation via ict’, british journal of educational technology, vol. 34, no. 2, pp. 169�181. senft, t. m. & baym, n. k. (2015) ‘selfies introduction � what does the selfie say? investigating a global phenomenon’, international journal of communication, vol. 9, 19 p. j. traxler 16 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://www.momentarium.org/experiments/7a10me/sadie_plant.pdf http://www.momentarium.org/experiments/7a10me/sadie_plant.pdf http://www.hindawi.com/journals/edri/2011/528495/ http://www.hindawi.com/journals/edri/2011/528495/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 sharpe, r., benfield, g., lessner, e., & decicco, e., et al., (2005) ‘scoping study for the pedagogy strand of the jisc e-learning programme’, final report, jisc, bristol, uk. sharpe, r., benfield, g. & francis, r. (2006) ‘implementing a university e-learning strategy: levers for change within academic schools’, research in learning technology, vol. 14, no. 2, pp. 135�151. sharples, m. (2002) ‘disruptive devices: mobile technology for conversational learning’, international journal of continuing engineering education and life long learning, vol. 12, no. 5�6, pp. 504�520. somekh, b. (2000) ‘new technology and learning: policy and practice in the uk, 1980�2010’, education and information technologies, vol. 5, no. 1, pp. 19�37. stald, g. (2008) ‘mobile identity: youth, identity, and mobile communication media’, in youth, identity, and digital media, ed d. buckingham, the mit press, cambridge, ma, pp. 143�164. the. (2015) ‘higher education is ‘‘part of the problem of social inequality, not the solution’’’, times higher education, 10 dec 2015. the. (2016) ‘bett 2016: ‘‘students may not always be experts in technology’’’, times higher education, 26 jan 2016. tonkiss, k. & dommett, k. (2013) the bonfires of quangos has thus far only smouldered, british politics and policy at lse, london school of economics, london, uk. tonks, d. & farr, m. (2003) ‘widening access and participation in uk higher education’, international journal of educational management, vol. 17, no. 1, pp. 26�36. traxler, j. (2008) ‘modernity, mobility and the digital divides’, in research proceedings of altc2008, ed m. macpherson, association for learning technology, oxford, uk. traxler, j. (2010a) ‘e-learning � the next wave’, in proceedings of alt-c, ed a. jeffries, association for learning technology, oxford. available at: http://repository.alt.ac.uk/798/ 2/abstracts_handbook_web.pdf traxler, j. (2010b) ‘students and mobile devices’, research in learning technology, vol. 18, no. 2, pp. 149�160. traxler, j. (2011) ‘ethics in mobile learning � a moral movement?’ in proceedings of iadis mobile learning, eds i. a. sánchez & p. isaı́as, avila, spain. traxler, j. (2012) ‘sustaining mobile learning and its institutions’, in refining current practices in mobile and blended learning: new applications, ed d. parsons, igi global, hershey, pa. pp. 1�9. traxler, j. (2013) ‘mobile learning � shaping the frontiers of learning technologies in global contexts’, in reshaping learning � the frontiers of learning technologies in a global context, eds r. h. kinshuk & m. spector, springer, new york, ny. traxler, j. (2015) ‘digital life, mobile learning and the arab spring of education’, glokalde, vol. 1, no. 3, pp. 12�20. traxler, j. (2016) ‘mobile learning and social networking’, in wiley handbook of learning technology, eds n. rushby & d. surry, wiley, malden, ma, pp. 390�405. traxler, j. & kukulska-hulme, a. (2006) ‘the evaluation of next generation learning technologies: the case of mobile learning’, research proceedings of alt-c2006, alt, oxford, uk. traxler, j. & lally, v. (2016) ‘the crisis and the response: after the dust had settled’, interactive learning environments, pp. 1�9. traxler, j. & riordan, b. (2003) ‘evaluating the effectiveness of retention strategies using sms, wap and www student support’, proceedings of annual conference 2003, ics-ltsn, galway, ireland. traxler, j. & traxler, j. (2009) ‘mobile subcultures’, in connected minds, emerging cultures: cybercultures in online learning, ed s. wheeler, information age publishing, charlotte, nc. pp. 17�28. ucisa. (2008) top concerns, ucisa, cheltenham. available at: https://www.ucisa.ac.uk/ bestpractice/surveys/tc/tc2008-9.aspx urry, j. (2007) mobilities, polity, cambridge. vincent, j. (2006) ‘emotional attachment and mobile phones’, knowledge, technology & policy, vol. 19, no. 1, pp. 39�44. wainwright, e. & marandet, e. (2010) ‘parents in higher education: impacts of university learning on the self and the family’, educational review, vol. 62, no. 4, pp. 449�465. research in learning technology citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 17 (page number not for citation purpose) http://repository.alt.ac.uk/798/2/abstracts_handbook_web.pdf http://repository.alt.ac.uk/798/2/abstracts_handbook_web.pdf https://www.ucisa.ac.uk/bestpractice/surveys/tc/tc2008-9.aspx https://www.ucisa.ac.uk/bestpractice/surveys/tc/tc2008-9.aspx http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 walker, r., voce, j. & jenkins, m. (2013) charting the development of technology-enhanced learning developments across the uk higher education sector a longitudinal perspective (2001�2012). interactive learning environments, routledge, london. white, d. s. & le cornu, a. (2011) ‘visitors and residents: a new typology for online engagement’, first monday, vol. 16, no. 9, available at: http://firstmonday.org/article/view/ 3171/3049 white, s. (2007) ‘critical success factors for e-learning and institutional change � some organisational perspectives on campus-wide e-learning’, british journal of educational technology, vol. 38, no. 5, pp. 840�850. williams, k. & gray, l. (2010) ‘the little phone that could: mobile-empowered ministry’, international journal of frontier missiology, vol. 27, no. 3, pp. 139�145. winters, n. (2013) ‘mobile learning in the majority world: a critique of the gsma position’, in the sage handbook of digital technology research, sage publications ltd., london, uk. pp. 402�411. wright, f., et al., (2014) ‘visitors and residents: mapping student attitudes to acad semic use of social networks’, learning, media and technology, vol. 39, no. 1, pp. 126�141. wun. (2015) ‘a special report on the 26th icde world conference’, world university news. available at: www.universityworldnews.com j. traxler 18 (page number not for citation purpose) citation: research in learning technology 2016, 24: 31372 http://dx.doi.org/10.3402/rlt.v24.31372 http://firstmonday.org/article/view/3171/3049 http://firstmonday.org/article/view/3171/3049 http://www.universityworldnews.com http://www.researchinlearningtechnology.net/index.php/rlt/article/view/31372 http://dx.doi.org/10.3402/rlt.v24.31372 doi:10.3402/rlt.v24.32166 original research article exploring the transformative potential of bluetooth beacons in higher education kieran mcdonald* and ian glover learning enhancement and academic development, sheffield hallam university, sheffield, uk (received 6 may 2016; final version received 10 october 2016) the growing ubiquity of smartphones and tablet devices integrated into personal, social and professional life, facilitated by expansive communication networks globally, has the potential to disrupt higher education. academics and students are considering the future possibilities of exploiting these tools and utilising networks to consolidate and expand knowledge, enhancing learning gain. bluetooth beacon technology has been developed by both apple and google as a way to situate digital information within physical spaces, and this paper reflects on a beacon intervention in a contemporary art school in higher education conducted by the authors intended to develop a situated community of practice in art & design. the paper describes the project, including relevant theoretical foundations and background to the beacon technology, with regards to the potential of using these devices to create a connected learning community by enhancing learning and facilitating knowledge creation in a borderless learning space. keywords: learning space; technology; community of practice; internet of things; design education introduction prior to the industrial revolution and the subsequent development of mass education, learning was a relatively unstructured process that relied heavily on interaction with others, such as in the ancient greek model of socratic questioning, or through observation, practice and critique by an expert tutor, such as in the apprentice model used in renaissance artists’ studios. however, with the need to create a workforce with at least some rudimentary education as a result of the new demands of the industrial world, these models were replaced by increasing use of standardised curricula using standardised learning materials � the so-called ‘factory model’ of education based on behaviourist principles. however, as economies move away from manual labour and manufacturing, their societies require people capable and experienced in a different type of learning, with less emphasis on the production of standardised workers to fill standardised jobs, and more on the developing learners who are readily able to acquire assess and apply new knowledge and skills throughout their lifetime, in jobs which may have no other exact match in the world and that are constantly changing. *corresponding author. email: k.mcdonald@shu.ac.uk responsible editor: amanda jefferies, university of hertfordshire, united kingdom. research in learning technology vol. 24, 2016 research in learning technology 2016. # 2016 k. mcdonald and i. glover. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 (page number not for citation purpose) http://creativecommons.org/licenses/by/4.0/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 this need for a different type of learner is transforming education in ways that look more like large-scale versions of the pre-industrial models than the stereotypical idea of a classroom of students being completely directed in their learning by a teacher at the front of the room. a major factor in both the societal and economic changes driving this change and the methods by which the change is addressed within education is that of the growing use of and reliance on technology within these societies. likewise, the value of interaction with other learners and experts is increasingly being recognised and facilitated through expanding use of technology to support investigation and communication among groups of learners. in recent years, the connectivism learning theory conceived by siemens (2005) has encouraged consideration of how technology mediates learning by supporting the social construction of new knowledge through the interaction with people, objects and environments. in addition, these changes to the vision of what education needs to be and the methods used to achieve them have also led to increased consideration of the spaces in which learning takes place, whether physical, virtual or a hybrid of the two. bluetooth beacons are an example of a simple technology that can be used to create connections and interactions between people, objects and spaces, and so offer significant potential for enhancing the ascendant social methods of learning by transforming learning spaces. this paper provides an overview of bluetooth beacon technology and outlines some of the relevant theoretical underpinnings for their use in supporting learning. it also describes a project undertaken by the authors to use beacons as a method of fostering the development of a learning community among graphic design students at sheffield hallam university and concludes with a discussion of the potential benefits of bluetooth beacons for enhancing learning, in addition to outlining other use cases for their introduction. beacons in more detail currently, there are two competing beacon technologies, apple’s ibeacon (introduced in 2013) and google’s eddystone (introduced in 2015); however, while they are incompatible with each other, for the most part they work in a similar way. both technologies send out a small piece of data, a unique identification number for ibeacon and either a similar identification number or a short url for eddystone, over bluetooth to any devices within range. this data is then received by these other devices, processed by associated software and an appropriate action taken according to the data received, shown in figure 1. this means that, in the most basic usage, beacons can be used to trigger an associated action on any devices with the appropriate software within range of their signal. however, an additional layer of information can also be created by analysing beacon identifier app triggered by beacon identifier requested data triggered request for data internet figure 1. typical beacon interaction process. k. mcdonald and i. glover 2 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 the strength of the signal to determine the distance from it. in this way, a single beacon could trigger different actions when a device is in its immediate proximity and when it is at the limit of its range. in common with the lighthouse metaphor used by these technologies, the beacons transmit only their presence and do not perform any active monitoring of devices within their range � meaning that they are not in themselves capable of tracking devices and, by extension, people. the complexity of action that the beacons can perform depends partially on the type of signal being used and partially on the logic within the app that is being triggered by that signal. eddystone’s url signals, the underpinning technology of google’s ‘physical web’ concept (https://google.github.io/physical-web/), offer a very simple way for objects and locations to transmit a web link to all devices within a defined proximity, whereas ibeacon and the equivalent mode in eddystone can enable highly specific actions to be performed. this is possible through combining data available to the relevant app, such that a user in a specific location at a specific time might have a particular action triggered while the same user in the same place at a different time would have a different one triggered. for example, at the start of a lecture, the signal from the beacon triggers the app to download and open the slides and other materials that are going to be used in the session, while a few minutes later the same signal triggers the opening of a voting app. likewise, the same information could be used to trigger different actions for people in the same place at the same time, such as by automatically directing students into different virtual collaboration spaces based on the group to which they are assigned. benefits discrete the version of bluetooth used by beacons, known as bluetooth low energy or bluetooth smart, dramatically reduces power usage compared to older bluetooth versions found in devices such as hands-free headsets and wireless speakers. this means that beacons can be small and run for several years on a compact battery, which in turn means that they can be discretely placed within an environment and left for an extended period of time with no need for maintenance. in addition, they still work when hidden from view, meaning that, unlike visual tags such as qr codes, they can be used without affecting the aesthetic of a space. passive unlike other similar technologies, beacons do not necessitate an individual making a conscious decision to access content, such as by deliberately scanning a qr code. this enables the beacons to be used to push information at individuals as they go about their normal activities rather than relying on the individual to actively seek out the information. multipurpose the beacons perform a very simple task, namely alerting other devices of their presence, and leave it to the developers of software for those devices to decide how the information will be used. this means that, once a beacon infrastructure is in research in learning technology citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 3 (page number not for citation purpose) https://google.github.io/physical-web/ http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 place, it becomes a resource for use by multiple parties and for a variety of uses. therefore, an infrastructure put in place to support way-finding can also be used by students to develop software that suits their own requirements. the use of the beacons for any particular purpose by one individual has no effect on others using them for a different purpose. potential issues the simplicity of the technology means that there are several potential limitations that can affect the usefulness of the beacons in some situations. the most significant of these are: finite battery life commercial beacons are designed to be discrete and usable in a variety of environments, including both indoors and outdoors. as a result, the small size and need to be robust limit the effective size of the battery that can be used and also mean that it may not be possible to replace once it has lost charge. complex actions need bespoke software while the ability of eddystone to transmit urls offers a simple method of sharing online resources, for more complex actions it is necessary to create custom software, such as mobile apps. this requires particular skills and expertise, meaning users need to install specific software and that the app needs to be updated whenever beacons are added, removed or repurposed. lack of remote management in order to preserve battery life, commercial beacons typically do not feature any ability to connect to the internet. this means that there is no ability to remotely manage the information being transmitted by the beacons and any updates require being within range of the bluetooth signal. while this is not typically an issue when the beacons are only transmitting a unique id number, it may present problems when using eddystone urls instead; however, this can be mitigated by using a url shortening service that allows short urls that can be edited to refer to different online resources. ability to spoof beacons as the beacons simply announce their presence and do not communicate directly with nearby devices, there is no way to prevent malicious spoofing of a beacon to trigger an action in the wrong place. however, while this means that there are some potential uses for beacons that are more difficult to achieve, such as recording attendance at formal sessions, it would be possible to mitigate this by requiring being in proximity to several beacons rather than a single one � though this would make managing both the beacon infrastructure and the app actions more difficult. k. mcdonald and i. glover 4 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 theoretical foundations the influence of people on an individual’s learning has been well established in the literature, as evidenced through a range of learning theories from the zone of proximal development (vygotsky 1978) to connectivism (siemens 2005). while these theories are explicit on the impact of people on the learning process, whether direct peers or more expert individuals, an important other aspect is often only implied � space. yet, without spaces that support and encourage the coming together of individuals for learning, the potential benefits to learning are reduced. the recent growth into research on learning spaces (ellis and goodyear 2016; kolb and kolb 2005; long and ehrmann 2005) is evidence that the influence of physical space on learning is becoming increasingly recognised, while the discourse around virtual learning environments is evidence of this same recognition in the digital sphere (glover and oliver 2008; dabbagh and kitsantas 2012; williams and jacobs 2004). the built pedagogies being developed as a result of these collisions between technology and space are leading to the development of ‘borderless’ learning spaces, where the physical and digital are integrated to enhance rather than dazzle (oblinger and lippincott 2006). the communities of practice theory (wenger 1998) describes a (frequently informal) approach to learning that is built around subject context and social interaction among peers within a shared space, while experts provide guidance and assist by facilitating the exchange of knowledge by members of the community. the use of online social networks in learning, whether as part of formal studies or an organic community of self-motivated lifelong-learners, utilises a ‘network effect’ (fenwick and edwards 2010) provided by these communities by enabling the incorporation of a wider range of people, including peers, experts and practitioners, within a common online space. this interrelationship between learning and social interaction is fundamental to social constructivism theory (berger and luckmann 1967), which posits that knowledge is constructed through the shared understanding of and interaction between individuals. the association between authentic spaces, practice and learning is also a central tenet of lave and wenger’s (1991) situated learning theory, which expands on vygotsky’s (1978) work around the influence of a ‘more knowledgeable other’ on increasing a learner’s understanding by also recognising the impact of the environment in which the learning takes place. the pipe is more important than the content within the pipe. our ability to learn what we need for tomorrow is more important than what we know today. connectivism provides insight into learning skills and tasks needed for learners to flourish in a digital era. (siemens 2005) siemens’ connectivism theory (2005) places the emphasis for learning in the future not on the acquisition of specific knowledge but on developing the skills to learn using whatever mechanisms are already in place and may come in the future. this is an explicitly digitally enabled learning theory, with its focus on the ability to retrieve information from sources ranging from other people through to online resources and non-human (i.e. machine) agents has parallels with the hollan, hutchins, and kirsh (2000) concept of distributed cognition. it also recalls the work of latour (1999), who suggests that technology can simplify the process of making connections between student, teacher and information by using these machine agents and other devices to research in learning technology citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 5 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 enable access and provide the ability to process information, filter, disseminate, remix and share. this thinking about learning mediated through non-human agents leads to consideration of a range of technical developments that are beginning to have an influence on the idea of what it means to be a learner in the early-to-mid 21st century. the growth of mobile learning initiatives, along with the subsequent research into related affordances and impact (hamm, drysdale, and moore 2014; woodill 2014), and the associated links to the concept of ubiquitous/pervasive computing (poslad 2009) positions technology mediated learning as breaking out from the confines of a dedicated physical learning space, such as a formal classroom, and instead making use of entire towns and cities as environments to support, facilitate and encourage learning (buchem and perez-sanagustin 2013). in particular, by using technologies commonly found on smartphones and tablets, such as wi-fi, gps and bluetooth, highly situated, serendipitous learning experiences can be created through the learner’s movements through different spaces and through the random interaction with other learners in their proximity. project information background in january 2016, sheffield hallam university’s arts and design department, the sheffield institute of art moved into new accommodation that had been designed from the outset to support the studio-based learning approach traditional to these creative disciplines. now, instead of students being timetabled in different rooms around the campus depending on the type of learning activity being undertaken, the students have all of their classes in large, multi-purpose, discipline-specific studios that provide an academic home for both students and the academic staff. in addition to the benefits of having a designated learning space for all formal sessions, the studios are also available for students to use outside of their timetabled learning regardless of whether another formal session is using part of the room. this provides new opportunities for the students to mix with and view the work of students from other years of the course, encouraging a wider potential community of practice and enabling a greater range of potential inspiration than was previously the case. this module was also designed to give students the opportunity to connect with professionals in the graphic design industry beyond the institution. this move, and the opportunities for exploring innovative teaching methods and technologies that the new space provides, was used by the authors as a catalyst for exploring the potential of technology to enhance the student learning experience with a group of 100 second-year undergraduate graphic design students. the studio into which these students were to move was formed from three-quarters of a large, roughly square, open area and was divided into three notional, loosely bounded zones based on the locations of equipment, resources and furniture � presentation, production and collaboration (figure 2). this subdivision of the space into zones, based primarily on the activity, intended to take place within assisted in targeting resources to students based on the type of learning that they are involved in at any particular time and enabled students to informally use the space even when another group was taking part in a formal teaching session. k. mcdonald and i. glover 6 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 hypothesis the division of the studio into zones allows digital resources to be targeted into the spaces where they are most appropriate. therefore, the authors hypothesised that bridging the gap between physical and online spaces and communities would help create a more productive learning community than the previous method of working with each in relative isolation. methodology in order to ensure that the digital spaces and tools being ‘brought into’ the physical studio were appropriate and useful to the students, an initial set of focus groups and an online survey were conducted with the students. the aims of these were to identify online tools and collaboration spaces that the students were already using and to elicit ways in which their learning, and experience of learning, could be enhanced. based on the findings of these data gathering exercises, an intervention was developed that used beacons to address the requests from the students by connecting the physical spaces with the tools that the majority were already using. this intervention was active during the module, and subsequent, semi-structured interviews with representatives of the cohort and with the module leader were held to evaluate its impact. the interview participants were identified as part of the initial online survey, where respondents were able to register their interest in taking part in the post-intervention study cohort. overall approval for the project and research instruments was obtained from the university’s research ethics review panel. findings of initial investigations the initial investigations were designed to elicit information in three areas, namely how and where the students prefer to work, the digital tools and resources that they find most beneficial to their practice and any ideas for improving their learning and their experience as a graphic design student and aspiring professional. of the 100 students enrolled on the module, 34 completed the survey. graphic design studio beacon zones we discover we construct we apply presentation zone production zone collaboration zone figure 2. simplified diagram of the learning space and the notional zones. research in learning technology citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 7 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 ways of working the respondents were all in agreement that confidence in using digital technology was beneficial to their current studies and likely to be in their on-going development as a professional designer, with 26 (76%) stating that it offered considerable benefit and the remaining 8 (24%) recognising at least some benefit. a similar result was found for the importance of being confident in the use of digital tools as part of professional practice, with 24 (71%) stating there was considerable benefit, 8 (24%) claiming there was some benefit and the remaining 2 (6%) stating there was minimal benefit, with no students reporting that there was likely to be no benefit at all. the question of where the students preferred to learn and undertake their design work was split between working alone at home (9; 26%), working in the studio in informal groups with peers (n �12; 35%) and working alone in the studio (4; 12%), with the remaining students (9; 26%) having no real preference. this split between working alone and communally in the studio was also reflected in the amount of time students spend in that space outside of formal sessions, with 21 (61%) spending less than 5 hours, 5 (15%) spending between 5 and 10 hours and the remaining 8 (24%) using the studio for over 10 hours outside their timetabled sessions. tools the digital tools that were suggested by the students as being particularly useful for their learning and on-going development fell into two broad categories. the first were those that are directly used as part of the course and in which they will need to become proficient in order to work as professional graphic designers, principally the creative software suite from adobe such as indesign and photoshop. the second were those tools that they use to discover, curate and share the works of others that they find inspirational or useful in some way or those that they can use to raise their own profile when developing an online presence with which to promote themselves to potential employers or for freelancing work, in particular the use of social media tools with a strong emphasis on imagery such as instagram, pinterest and tumblr. suggestions for improvements when asked to select the three most valuable aspect of the studio as a learning space, there was particularly strong representation for the following options: academic staff presence and availability (20; 59%); the ability to use their own devices (18; 53%); opportunity to mix and collaborate with peers (16; 47%); access to technical support and resources (9; 26%); and physical and virtual showcasing space (5; 14%). additional support for these being the most valuable aspect of the space come from the students’ responses regarding specific improvements that could be made where 13 (41%) wanted improved abilities to use their own devices; 13 (41%) wanting more opportunities to showcase and critique work; 12 (37%) asking for improved methods of receiving key information for the course; and 9 (28%) wanting more collaborative spaces in which to develop a community of practice. k. mcdonald and i. glover 8 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 intervention outline based on the information received from the students in the initial investigations, paired with an understanding of the capabilities of beacons and the resources available to develop a pilot intervention, the authors made the decision to make use of the zones that had been set up within the studio (figure 2). these were used to target online resources that were appropriate to the activities intended in those areas, that were based around tools that the students were already using, and that addressed the improvement areas highlighted by the students. as the students had requested more timely information about the technical aspects of their graphic design practice, a beacon was placed in the ‘production’ zone to direct them to online training resources related to the core tools and techniques of the discipline, as well as some more inspirational materials to explore as necessary and information on how to use the specialist equipment in this part of the studio. the ‘presentation’ zone was viewed as a slightly more formal space occupied by the academic staff and the place where their presence in the studio was most obvious, therefore a beacon was placed in this area that led to a tumblr blog curated by the tutors to collate resources that they found useful and stimulating for their own work as practicing graphic designers. in contrast, the ‘collaboration’ zone was viewed as a less formal breakout space for students to share and comment on their peers’ ideas and work-in-progress, therefore the beacon placed here pointed to a pinterest board that collated instagram posts with a module-specific tag by using the ifttt (‘if this, then that’) online service to automate the collation process. as all of the zones were overlaid with resources that were publicly available on the web, the primary purpose of the beacons was to prompt the students to access and interact with the resources and to build upon the existing demarcation within the overall studio space by reinforcing the intended activities for each zone. wherever possible, the choice of platform or tool used in each zone was based upon whatever was the best fit for the intended purpose selected from those that the students were already using. due to this focus on the student’s existing digital practices, there was a need to ensure that the teaching staff were able to make use of the same tools and to articulate their use in this particular project to the students. where possible, many of the complex workflows were automated using ifttt, allowing the focus to be purely on the content being posted to these shared spaces and the building of the community around them. however, a small amount of informal training was provided to the academic lead to ensure that he or she understood the processes and could answer questions from the students without the delay of having to refer back to the authors. results at the conclusion of the intervention, a semi-structured interview was held with student representatives of the cohort to explore their impressions from the pilot, and an interview was later held with the module leader to both explore the students’ views from his perspective and to gain his own insight into the project. during the course of the interview, several major themes emerged and these are detailed here along with relevant commentary from the module leader. research in learning technology citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 9 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 closeness while an overarching objective of the project was to use the technology and studio space to aid in the development of a community of practice among the students, the interview revealed that the integration of technology as a means for tutors to share works that they find personally and professionally interesting or inspiring resulted in the students developing a stronger informal connection to, and understanding of, the academic. the students specifically identified this connection as sense of feeling ‘closer’ to their tutors as a result of being more connected to them through the online networks and by both parties being increasingly visible in the same space. student: definitely i think everyone has felt, and i don’t know if that because of the beacons or whatever, you can tell everyone is closer to the tutors this module than any other module that we have done. [. . .] everyone has been just felt more confident going up and speaking to [the tutors] and has just built up a better relationship with the tutors than just tutor-student. this feeling of closeness and the connections that both created it and then were encouraged by it have expanded the pathways for insight, particularly in sharing the iterative design process students have undertaken while producing their final pieces for submission. in design disciplines and projects, the process is just as important as the outcome and this informal peer network, mediated in part by the beacons, facilitated points for interjection and reflection by staff and students. the ability of academic staff to maintain this connection outside formal sessions was a shift in dynamic highlighted by staff and students. the concept of ‘social presence’, or in this case the student and tutor occupation of a digital space, has suggested changes in practice that would benefit both parties. student: yes it would be useful when you don’t have sessions but he is still finding and posting stuff you can still get to it whenever you’re in the studio. however, the feedback suggests that this closeness is likely to present significant challenges, particularly around managing student expectations and the blurring of traditional boundaries between the professional and personal online domains. tutor: i’m a little unsure as it connects me to [the students] in ways i’m a little uncertain about as most of them use [instagram] for their private life and suddenly seeing someone was [on holiday] when they should have been in and students off for legitimate reasons but they are posting things that seem to contradict [these reasons]. showcasing the potential to showcase and create efficiencies using digital is a potentially powerful facet of contemporary curriculum design. this is particularly evident from the students, who viewed digital creation, critiquing and submission as a way to speed up the process by removing limiting factors associated with the physical objects. student: in structured peer reviews, [. . .] i know we had done one a few weeks back where we had to print off a3 sheets and get it all prepped whereas if it was all on a digital platform, for instance connected by a tablet or something like that, it’s easier than having to fight over the printers and print everything off. k. mcdonald and i. glover 10 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 the students were also able to articulate some of the affordances of working in a more digital, online way. in particular, they were conscious of the ability to showcase their work with external designers and agencies when looking for employment. student: ‘kind of feels like it could be done like a digital ‘‘new designers’’ [event] which [would help students get] their work out there to get employed. that would be really good’. integration of digital and analogue practice as shown above, the students were particularly keen to make greater use of digital tools and practice, seeing these as a significant aspect of the future of the graphic design industry. however, the module leader was more cautious about moving more toward a fully digital module experience, pointing out that while digital working can be more efficient, there is a strong analogue and printed aspect to the discipline that the students need to be constantly aware of and understand. tutor: there is something about the reification � to reify is to lock solid � digital spaces are often not as reified because they can be constantly changed, updated, etc. there is something about the level of finality about the printed act that i’m not sure is necessarily present in digital. [. . .] i’m not disputing there might be potential in [working more digitally] but i wonder what the student’s motivation is � one of convenience? therefore, it is clear that embedding digital in the curriculum is a challenge and finding correct points of entry for fit alongside the core concepts of the module had proven difficult. even though the students had previously used the digital platforms that were shared through the beacons, one of the challenges for them was around permission and confidence to take hold of this fully. in reflection, the academic team was also breaking new territory in considering the best way to successfully embed the tools and technologies in order to encourage enough engagement to achieve the crucial tipping point for student adoption. tutor: it’s almost as if instagram is [. . .] not a space to get into discussions. i think that been quite important for me to understand that. it has its role but it’s kind of suspended in a strange way � a space without criticism? value of space to the development of community the ability to asynchronously scaffold students’ learning and provide challenge outside the formal sessions is an important aspect of a studio-based learning culture. the students predominantly enjoy studying in the studio and ensuring accessible content at the time of need in the physical space was important, while also being able to engage with the resources and activities from outside the studio offering an added layer of convenience and flexibility. student: this studio is fantastic! i’ve never seen so many people actually coming to uni to do their work and that’s created a little community. therefore, the studio has been a central anchor for this intervention and a welcome development for the students. it has stimulated further engagement, collaboration and attracted students to learn inside and outside of formal teaching sessions. research in learning technology citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 11 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 as a result, through the integration of ‘home’ spaces, both physical and digital, a greater sense of a peer-led learning community has developed within the cohort. student: everyone has their own little areas and intermingle. it helps your design practice in itself as you can go and see what other people are doing and gain inspiration from them or help people with what they are doing. it’s good that everyone is communicating with each other now and that’s largely because of the studio student: i think [the benefit of] the pinterest thing is the curation of everyone’s work, so on the same token that when everybody is the studio you can walk round and have a look at everyone’s work, [. . .] if you’re not in the studio [the fact that] that [the pinterest board collates work] from the hashtag off instagram helps ease of access to other people’s work and see what other people are doing and building the community that way. tutor: yes, they built a studio culture running through that module that was largely conditioned by the opportunities of the space. barriers the main barriers identified were those related to familiarity with the technologies being used and access to appropriate equipment. in particular, as the online tools selected for the intervention were informed by the students’ existing practices, technical familiarity presented a larger barrier to the tutors than the students. from a learning and engagement perspective, the tutor’s comments suggest that a significant barrier was in getting the students to appreciate the potential value of developing a digital community prior to point at which it would have had the most value for them. tutor: i think there was close to 90 out of 100 of them following [the instagram account, but] it that might have taken 5 or 6 weeks to build from early adopters, and the frustrating thing was it was the first thing we wanted to use it for when we all went to [visit design studios in] london. this is the place we really want to see what you do. awareness of, and experience with, all of the various platforms being used by both staff and students as part of the learning community was an issue for the students. while the ‘official’ channels that were developed and promoted as part of the intervention were known to the students and used by them, the informal nature of the learning community and the studio work ethic meant that they may have missed some of the other online spaces that were being used by the cohort. student: you have got the instagram, the pinterest, pretty sure there is a twitter feed as well, i found a facebook one this morning didn’t know that existed. suggested future development efficiency on campus came through as a real priority for students, particularly them having access to information at the exact place and time that it is required or appropriate. one suggestion for example was the beacons could be deployed in specific locations to deploy key information that was interchangeable. student: like this morning, for instance, with the [assessment] hand-in. we had on the whiteboard on the corner what needed to be done, whereas the tutors could post [through the beacon] exactly what need to be handed in when so it could be used for that advantage. k. mcdonald and i. glover 12 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 a strong theme of personalisation was raised by the students as something that beacons could facilitate. in particular, the use of beacons to trigger actions within an app that relate directly to the individual, rather than transmitting a generic web address to any device in range, was thought to be a powerful mechanism for encouraging engagement and ensuring awareness of critical information. being able to target messages at groups or individuals as they move through the campus, based on knowledge of where they are, who they are and their past activity, was seen as a potentially powerful use for the beacons. student: i think the push notifications would encourage people to use it more, you could get annoyed but people would use it a lot more if every time they walked in there was a notification. if it’s a constant reminder it could be a good way of getting [key information] across. the intervention provided lots of points for the tutors to think about regarding how, when and why to incorporate digital and online practices into their teaching, both on this module and the rest of the course. in particular, it has encouraged reflection on how information has been curated and presented to the students prior to the intervention and how that process could be modified in the future as a result of the technology available to the staff and students. tutor: what’s interesting with [this pilot] is, in sessions we would usually photograph and post something. i was pinging a lot of work on, partly because i was thinking i must fill this tumblr, so it was just bang, bang. but i suppose i haven’t really reflected on this and the amount of posts that congregate there and the things we think are interesting compared to what we would talk about on a daily basis. that’s interesting for me to reflect on that and that’s actually very quick to do! discussion while the project described in this paper was an exploratory study to identify whether there are opportunities to use beacons as a mechanism to support better use of learning spaces, in particular, the specialist studio space that was the focus of the intervention, it is apparent that even this extremely simple use, when combined with an appropriate physical space, can have an impact on students’ learning and interaction. it is interesting to note that the students developed a feeling of ‘closeness’ to their tutors partly as a result of a continued digital presence from the course academics through the tumblr blog accessed through the beacon. this suggests that, even though the same resource could be accessed from outside the studio, using the beacons to strongly situate this presence within the studio had a positive effect on the students. from the interviews with both staff and students, it is clear that, once a certain level of familiarity with the technology was obtained, the beacons effectively faded into the background and became a convenient method of being prompted to check for updates to the online spaces. instead, unless their attention was explicitly drawn to the beacons, the participants’ discussion centred around the underlying platforms and the effect on interactions within the student cohort. by selecting the online platforms based on those that the majority of the students were already using, the barrier for the students to participate was reduced � though, in this particular case, the burden of learning and adapting to a new platform and way of working was transferred to the tutors. research in learning technology citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 13 (page number not for citation purpose) http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 overall, it would appear that the original hypothesis for the project was, in this small pilot, found to be an accurate prediction of the effects of the intervention. while the small scale and focus of this project limit the ability to make direct inferences regarding the potential for beacons in higher education more widely, the increasing demand from students for personalised, just-in-time information through their mobile devices suggests that use of beacons will continue to grow and increase in the complexity of interaction they enable. this is likely to result in greater benefits to the students and enable alternative and novel method of peer and tutor interaction within the learning space. increasingly, we see student expectation in universities demanding ‘smart’ responses regarding resource, spaces, communication, curriculum design and employability opportunities and bluetooth beacons, if implemented appropriately, have the potential to address these expectations. conclusion in spite of their simplicity, beacons offer many ways to support a wide variety of learning, and more general student experience, initiatives. in particular, beacons offer a supportive lever to build broad and dynamic learning communities that can exist in the physical and digital space both synchronously and asynchronously. for learners, this offers connection, timely access to context-specific information and the ability to share and collaborate in learning on campus in a range of spaces, across the web and potentially around the world with peers, academics and professionals. for academics, the technology can enable more flexible delivery, broaden pedagogical approaches, open up the learning space to internal and external expertise, and encourage an exploratory and connectivist approach to learning that makes us of a multitude of different spaces. the inclusion of non-human agents in the learning process offers the possibility to shape, support and disrupt learning. considering learning mediated through these non-human agents, such as an infrastructure built from mobile apps that leverage beacons, institutional data sources and external services to target learners with highly personalised, contextual information, leads to consideration of technical developments that are beginning to influence what it means to be a learner in the early-tomid 21st century. while there are, and will continue to be, challenges around digital confidence and fluency from both students and staff, there is a growing pressure from both of these groups to integrate technology into learning and teaching practice that is unobtrusive and supports them in ways in which they want to work. the line between formal, timetabled, face-to-face sessions and informal, non-timetabled, selfdirected learning becomes blurred in these spaces linking the physical and the digital. coupled with the growth in use of mobile devices by students and academics, beacons have significant potential to dramatically transform learning by offering new ways to students to interact with their peers and tutors. at the same, time they also provide an infrastructure that can be utilised for a myriad of other purposes, both planned and organic, to enhance the overall experience of learning and teaching, and of other aspects of higher education organisations such as outreach and cultural enrichment. the potential benefits of beacons suggest that they are a powerful, yet simple, unobtrusive and flexible technology that can be applied to a wide variety of situations within higher education. k. mcdonald and i. glover 14 (page number not for citation purpose) citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 acknowledgements this research has been supported through an internal teaching enhancement fund, and through a google internet of things research award. references berger, p. l. & luckmann, t. (1967) the social construction of reality: a treatise in the sociology of knowledge, allen lane, london. buchem, i. & perez-sanagustin, m. (2013) ‘personal learning environments in smart cities: current approaches and future scenarios’, open education europa elearning papers, vol. 35. dabbagh, n. & kitsantas, a. (2012) ‘personal learning environments, social media, and selfregulated learning: a natural formula for connecting formal and informal learning’, the internet and higher education, vol. 15, no. 1, pp. 3�8. doi: http://dx.doi.org/10.1016/j. iheduc.2011.06.002 ellis, r. a. & goodyear, p. (2016) ‘models of learning space: integrating research on space, place and learning in higher education’, review of education, vol. 4, no. 2, pp. 149�191. doi: http://dx.doi.org/10.1002/rev3.3056 fenwick, t. & edwards, r. (2010) actor-network theory in education, routledge, abingdon, uk. glover, i. & oliver, a. (2008) ‘hybridisation of social networking and learning environments’, in proceedings of world conference on educational multimedia, eds j. luca & e. weippl, hypermedia and telecommunications, pp. 4951�4958. hamm, s. e., drysdale, j. & moore, d. (2014) ‘towards mobile learning pedagogy’, in mobile pedagogy and perspectives on teaching and learning, eds d. mcconatha, c. penny, j. schugar & d. bolton, igi global, hershey, pa, usa, pp. 1�20. hollan, j., edwin hutchins, e. & kirsh, d. (2000) ‘distributed cognition: toward a new foundation for human-computer interaction research’, acm transactions on computerhuman interaction, vol. 7, no. 2, pp. 174�196. doi: http://dx.doi.org/10.1145/353485.353487 kolb, a. y. & kolb, d. a. (2005) ‘learning styles and learning spaces: enhancing experiential learning in higher education’, academy of management learning & education, vol. 4, pp. 193�212. doi: http://dx.doi.org/10.5465/amle.2005.17268566 latour, b. (1999) ‘on recalling ant’, the sociological review, vol. 47, no. 1, pp. 15�25. doi: http://dx.doi.org/10.1111/j.1467-954x.1999.tb03480.x lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation, cambridge university press, cambridge, uk. long, p. d. & ehrmann, s. c. (2005) ‘the future of the learning space: breaking out of the box’, educause review, vol. 40, no. 4, pp. 42�58. oblinger, d. & lippincott, j. k. (2006) ‘learning spaces’, brockport bookshelf, book 78, [online] available at: http://digitalcommons.brockport.edu/bookshelf/78 poslad, s. (2009) ubiquitous computing: smart devices, environments and interactions, john wiley & sons, chichester, uk. siemens, g. (2005) ‘connectivism: learning as network-creation’, e-learning space.org, [online] available at: http://www.elearnspace.org/articles/networks.htm vygotsky, l. s. (1978) mind in society: the development of higher psychological processes, harvard university press, cambridge, usa. wenger, e. (1998) communities of practice: learning, meaning and identity, cambridge university press, cambridge, uk. williams, j. b. & jacobs, j. (2004) ‘exploring the use of blogs as learning spaces in the higher education sector’, australasian journal of educational technology, vol. 20, no. 2, pp. 232�247. woodill, g. (2014) ‘unique affordances of mobile learning’, in mastering mobile learning, eds c. udell & g. woodill, john wiley & sons, hoboken, nj, usa, pp. 111�126. doi: http:// dx.doi.org/10.1002/9781119036883.ch15 research in learning technology citation: research in learning technology 2016, 24: 32166 http://dx.doi.org/10.3402/rlt.v24.32166 15 (page number not for citation purpose) http://dx.doi.org/10.1016/j.iheduc.2011.06.002 http://dx.doi.org/10.1016/j.iheduc.2011.06.002 http://dx.doi.org/10.1002/rev3.3056 http://dx.doi.org/10.1145/353485.353487 http://dx.doi.org/10.5465/amle.2005.17268566 http://dx.doi.org/10.1111/j.1467-954x.1999.tb03480.x http://digitalcommons.brockport.edu/bookshelf/78 http://www.elearnspace.org/articles/networks.htm http://dx.doi.org/10.1002/9781119036883.ch15 http://dx.doi.org/10.1002/9781119036883.ch15 http://www.researchinlearningtechnology.net/index.php/rlt/article/view/32166 http://dx.doi.org/10.3402/rlt.v24.32166 doi:10.3402/rlt.v19s1/7783 open education videos in the classroom: exploring the opportunities and barriers to the use of youtube in teaching introductory sociology elaine tan* and nick pearce graduate school/learning technologies team, durham university, durham, uk (received 20 march 2011; final version received 21 june 2011) the use of open education resources has become more commonplace in classroom teaching and this has been an observable and growing trend. the accessibility of the same materials further reinforces the change in roles of the teacher, from gatekeeper of knowledge to learning facilitator. our research question is that if a student has free and easy access to the same materials that are being used to teach them in class, how does this affect their perceptions when they are presented with this material in the classroom environment? what are their perceptions regarding the perceived value for money, efficacy and authority of the material? this research specifically investigated the use of open education videos in the classroom environment and their incorporation into an associated space in the virtual learning environment. the research questions of this investigation surrounded the practical, technical and pedagogical issues that arise from the incorporation of these resources within class and online course materials as well as exploring student perceptions about the use of this material in the class and online. keywords: youtube; online video; open educational resources introduction this project is a case study of the use of youtube videos in learning and teaching in a 10 week introductory sociology course at the foundation centre at durham university, which prepares mature and international students for their undergraduate degree. this course was taught across two campuses to three classes of students a week, with 75 students in total. the foundation year at durham teaches a range of subjects to its students, with specific courses relating to the degree programme for which they are registered. the nature of the centre means that there is a diverse student body in terms of age, nationality and subject specialism. this has an impact on the design of the course, which is intended to be a stimulating introduction to a wide variety of sociological topics such as class, gender, crime and media. during this programme, online education videos were integrated into the classroom practice of the lecturer. in addition to this a youtube playlist (pearce 2011) was established and made available to students in the associated online environment. the research questions are focussed on evaluating student perceptions of this teaching approach and investigating independent use of the created playlist. *corresponding author. email: elaine.tan@durham.ac.uk alt-c 2011 conference proceedings 0174 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # e. tan and n. pearce. this is an open access article distributed 125 under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7783 videos were used to illustrate topics and followed by further explanation and discussion, both class-wide and smaller group. literature review since 2005 youtube has emerged as a major host of online video content and is now the third most popular website behind google and facebook (alexa 2011). the site hosts an enormous range of material and is popular for sports clips and music videos but has also been used within higher education as a way to communicate with current and potential students and disseminate research and teaching-based material (wilkes, pearce, and barker 2011). as of september 2009 there were 102 university youtube channels in the uk, and at the time of writing this report over 400 university channels worldwide (azyan 2011). there is a creative tension within youtube as a platform for mainstream broadcasters (maybe even including universities) and as a community of individual content creators who see the site as a social network (moran et al 2011). this has resulted in a wide range of content and uses/users which has been widely studied in the social sciences (burgess and green 2009; lange 2007; snickars and vonderau 2009; wesch 2008). the site offers a wealth of multimedia content that could be used for sociology teaching. this will include material specifically developed with sociology in mind such as interviews with leading theorists and teacher-created content, as well as more general content that may be useful in illustrating key concepts and theories. talking about technology more generally, weigel argues that it has the potential to improve both the quality and access (‘richness’ and ‘reach’) of teaching, that is the level of engagement with learners and the numbers of learners engaged. in practice institutions tend to focus on the reach potential of the internet (weigel 2002; wilkes, pearce, and barker 2011). in the context of youtube this is shown by the number of institutions hosting content on their youtube channels for promotional purposes. however weigel argues that new technologies can enhance the richness of the learning environment through combining ‘bricks and clicks’: where online video resources can ‘‘enrich and extend the students’ exploration of new territory’’ (weigel 2002). the potential impact of youtube on teaching has begun to be explored in the academic literature. a recently published literature review examined 188 peer reviewed journal articles and conference papers with ‘youtube’ in the title (snelson 2011). the recent arrival of youtube and the length of time typically involved in peer review, as well as the review’s narrow focus, would suggest that 188 peer reviewed scholarly artefacts represent an under-estimate of the academic interest in the use of online video. whilst the review considered a wide range of articles, of interest to this project was a subset of 13 articles that included instructional strategies and general tips for incorporating youtube videos in the classroom. many of these were from fields with limited application to sociology (e.g. medical education, where videos could be used to demonstrate complex procedures). of particular relevance is an article about the incorporation of multimedia content in a sociology course in the usa (miller 2009). here the author incorporates multimedia content (audio as well as video) into their introductory sociology course and states: e. tan and n. pearce 126 [the] most critical function in terms of cognitive learning appears to lie in their capacity to serve as representational applications for key ideas. whether in the form of a news story, movie clip, interview or documentary, information and illustrations afforded by media are particularly valuable in helping students acquire the initial mental imagery essential for conceptual understanding. (miller 2009) this quite clearly relates to weigel’s idea of the internet enhancing the richness of the educational experience. in addition to this use of multimedia a variety of other uses are suggested, including as an icebreaker for initiating classes. this is similar to another us sociologist who uses topically relevant songs to start his sessions (palmer 2011). the miller article provides a good list of some of the potential issues and problems that may arise from employing online media, and these include student resistance (possibly as a result of technological limitations) and technical issues such as broken links, poor image quality and classroom technical problems, although he concludes that ‘‘multimedia integration is not a daunting task’’ (miller 2009). john seely brown discusses a case study where video materials were viewed by groups of students who were unable to access more traditional lectures. they viewed the videos as a group in a social setting. viewed in this way students collaboratively constructed their own meaning of the material, and went on to outperform the students who had only attended lectures (brown 2000). this is an important point: this project does not just propose incorporating videos as a replacement for lecture material, or as a way of ‘flipping the classroom’ where information transmission takes place outside the class allowing for other classroom activities (which might usually be set as homework), but uses videos as a means of supporting and enhancing learning within the traditional classroom environment. using videos in the classroom can be the starting point for class discussions where students use the multimedia potential of youtube to engage with new and diverse topics and apply their knowledge and understanding of new topics within and beyond the classroom. this specifically social consumption of online video in class has yet to be explored in the literature that has been surveyed, and is the basis of this study which examines the role of online video in students’ learning both within the classroom and outside of it. methodology the first activity carried out as part of this project was the collation of videos from youtube and the creation of an online playlist which at the time of writing contains 32 videos. this playlist covers a range of topic matter from a wide variety of sources. as an indication it includes a feature length documentary about pierre bourdieu (in seven parts), a 10 minute animation video produced by an further education (fe) lecturer about weber’s protestant work ethic and the spirit of capitalism, and a comedy sketch from john cleese, ronnie barker and ronnie corbett illustrating social class in britain. some of these have been produced specifically for a sociological audience and others have not, but they have all been selected by the lecturer as a useful resource for students taking an introductory sociology class. the initial intention was to publicise the playlist via the c-sap community and encourage contributions from other sociology teachers. this achieved a certain degree of success although not the impact hoped for. the playlist was embedded within the virtual learning environment (vle) and promoted to students. the playlist was viewed 290 times (as of 20th june 2011) but there are no further data on alt-c 2011 conference proceedings 127 unique viewers or their location. this figure may seem relatively disappointing but only relates to users who accessed the playlist, rather than aggregating users who viewed the individual videos within it, which were linked to independently within the course materials. the next stage of the project was incorporating the videos into the course. not all of the videos were used in class (e.g. the bourdieu film) but many were incorporated into the class sessions, which were three hours long. there was a range of videos used, and they were used for a variety of reasons. some were used to introduce key sociologists to the group, others illustrated key points or data in an engaging way and yet others provided light relief whilst still reinforcing key concepts. these videos were included as links or embedded within the powerpoint slides which were available in the vle alongside the separately embedded playlist. questions were displayed whilst the videos were being viewed by the students to stimulate discussion, encourage critical analysis and promote deeper learning on the part of the students. in the ninth week of the course a series of three focus groups (24 students in total) was held across both campuses with representation from international, mature and domestic students. the focus groups were promoted in class and through an e-mail list. they were conducted by an experienced co-ordinator who was not linked with the course in any way; this was to ensure that the students could be open about their views of the use of video in class, and confident in being critical if need be. food and beverages were provided. these sessions were recorded, and the notes were anonymised before being analysed. results in-class use of videos in the first place the students were asked about watching the videos in class and whether they believed that this was a valid and effective way of supporting and enhancing their learning. the results suggested broad support for this practice and the students also raised certain elements that they believed added to their understanding of the subjective nature of sociology with comments including: i think it’s desperately important to get the opinion of others than the lecturer. that’s where ideas come from, you get discussions going and you bounce back ideas and this leads to something new doesn’t it? if you only just had one opinion you wouldn’t learn anything. acceptable use the students presented their opinions as to when it was both appropriate and acceptable for the use of videos in class. one theme that frequently came up was the inclusion of further explanation and proper integration of materials. many of the students commented on specific issues with ensuring that the videos were relevant and integrated into the class. the tutor kind of explains the video afterwards as well, which is key. this also relates to a discussion about the suitable length of video to be used in class. there was no real consensus on this, but it was felt that short ‘taster’ videos were e. tan and n. pearce 128 preferable and there was evidence that some students would follow these up in their own time. facilitated value one of the key benefits of using videos in class was this ability to initiate class discussion based on the video that had been watched in the whole class environment. in one of the focus groups a student mentioned that they felt that they could easily watch the videos at home, and so did not need to watch them in class. this was disputed by the rest of the group, with comments such as this: we’ve got to understand that not everyone has that time at home to watch these things. people are not just students, they have jobs they have kids. in class they actually get the time to watch the video if you watch things at home, i’ve watched things and just thought ‘ah that’s pretty’ but to watch it in a lecture situation, you analyse it and the lecturer’s saying this is because of this [. . .] and you read so much more into it because you’re watching it with someone who knows what they’re talking about they’re explaining it to you. this suggests that there is quite a lot of benefit in watching the videos in class as a group, as discussed by seely brown. bringing in other sociological voices, by video, is an option that may be more applicable to the social sciences with multiple competing paradigms. recommended ‘viewing’ students were also asked about how they assessed the usefulness of the online videos that they watched. there was a range of responses to this with a common theme being the importance of the lecturer as a gatekeeper or trusted guide. i hold this as better than anything i could find myself. i could google sociology and youtube it, but you just don’t know what you are going to get, but if you use something that’s been recommended it seems more relevant to me. it’s the modern version of just being given a reading list isn’t it? it’s just the same as being given something to read in class. it’s quicker, you can learn more and make more connections. there was some concern about the user generated content as in the comment below: i do worry though. we’re told not to go onto wikipedia, and with youtube it’s exactly the same, it’s just people uploading things as well, i mean is everything vetted? i mean are things updated between him showing us the link and us going and seeing the material? this is a valid point, and highlights the overall way in which education engages with social media in (e.g. wikipedia, facebook etc.). the user community of youtube was also mentioned in relation to assessing the quality of the videos: you have to look at the comment on it, and how many stars there are on it. clearly there is an issue with using the user community ratings to assess the educational value of content. the user community will be rating the videos on their own terms for things such as entertainment value, which may not overlap with alt-c 2011 conference proceedings 129 educational quality. another student took a more measured view on assessing video quality: i usually rate the quality of a video by comparison, once you’ve watched 4 or 5 you sort of know the first one was really good, the second two pale in comparison. discussion role of the teacher: retention of imparter of knowledge the results of this research highlight a number of factors surrounding the use of videos in education. firstly the role of discussion is highly prized by the students, and the video’s role in stimulating this was frequently mentioned. secondly the students valued the teachers’ input into these discussions and appreciated the additional commentary provided whilst watching the videos in class. the results indicate that the students felt that the combination of being able to ask questions and offer opinions as well as the benefit from the additional expertise of the lecturer, meant that the video’s quality was somehow ‘added to’. it would seem that, even though students had access to and were given exactly the same resources, they still felt that there was added value when these were viewed collectively and the role of the teacher in this process was key. establishing a benchmark the students in this group, when asked about how they evaluate the quality of video online stated that they referred to the videos provided by the lecturer as a way of judging the value of a video that they had found themselves. the students appreciated the presentation of differing opinions within the classroom, although they seemed to require an evaluative framework when presented with the choice of selecting their own additional learning material. it was mentioned by several of the students that they saw this material provided by lecturer input as having been ‘validated’. this is interesting as it contradicts the assumptions that students are happy to find their own material online and calls into question the extent to which they use this as a mechanism to support their own learning, preferring more traditional approaches of lecture handouts and textbooks to sourcing their own material. it seems that the students on this course were overwhelmed by the variety of learning resources available and as such welcomed a seemingly ‘validated’ resource as a way of creating a comparative framework and a means of charting a course through other self located material. diversity and democracy one interesting aspect to come out of the research is surrounding the use of videos in the sociology course to highlight the subjective nature of social science. the students recalled incidents where the lecturer had used videos that opposed information he had just outlined to them as a way of representing other views and described these incidents positively, stating that this had ‘helped [them] make up [their] own mind(s)’. being able to analyse different arguments and weigh the merit of these is a process involved in deep learning, where a learner has to actively engage with material and make value judgements based on their own opinion. in this use of video, there is e. tan and n. pearce 130 evidence to suggest that by providing these other opinions and opposing arguments, the students were critically analysing information with which they were presented and synthesising their own conclusions, a fundamental feature of deep learning. this indicates that the facilitated use of video can be escalated on bloom’s taxonomy to reach higher order thinking skills and not simply the lower order of understanding and remembering. the students spoke about ‘humility’ on the part of the lecturer in allowing other oppositional arguments be shown and as such felt ‘freer to express [their] own opinions in class discussions’. social and sharing the students used the videos almost as a social currency between members of the class and as a way of fostering bonds in social networking sites. they spoke of how they would often post videos, (not always education related) onto their friend’s walls, and use this as a way of starting conversation that would sometimes lead to conversations about work. the students also had positive feelings and were actively seeking to share materials with other members of their class. in the focus group, the students asked about the possibility of sharing videos that they found online with a wider audience than just friends on facebook. when asked about what format they thought that this could take and what could be done with it, they suggested it could be given to the next year’s cohort as a way of finding some of the resources that they had found useful during the time on their course. conclusions the focus groups discovered a wide range of complex issues surrounding the use of online videos in learning and teaching. these included the extent to which video was already incorporated into some of their learning, the willingness to collaborate and contribute to the communal playlist and the strategies that students used to assess the quality of videos that they discovered and are actively establishing their own mechanisms for quality assurance and benchmarking. some of these strategies will be effective, but there is a danger of being exposed to misleading or incorrect material, in particular about potentially contentious issues such as feminism. the lecturer was sent a link to a video by a student which was a satirical description of feminists by a right wing american group, confusingly presented as if created by a feminist group. to a student unfamiliar with these kinds of debates it could have been taken as an entertaining look at a complex issue, despite the educational content being negligible. this might suggest that in future some form of video literacy could be included within key skills provision, to encourage the kinds of critical thinking that students are already being encouraged to develop with text-based resources. the results of this research indicate that the students interviewed felt that the use of videos was an effective way of supporting their learning. they offer a number of explanations for this, providing alternative views and opinions on subjects, providing variety in delivery mechanisms, and using every day examples to illustrate points. the students overall did not feel that the use of videos represented poor value for money and felt that the facilitated use of these teaching materials surpassed any autonomous use of these as they valued the additional explanations and discussion that accompanied them. alt-c 2011 conference proceedings 131 what can be determined by the results of this research is that the students interviewed had a traditional assumption of the role and authority of the teacher and that these views are not easily displaced by the introduction of video resources. the aspect which students valued most was the discussions surrounding the resources. this is an interesting outcome as it supports a constructivist approach to teaching and learning that whilst including content is an important element, the focus should be on the discussions that surround this. the results of this research confirm that the use of video in education can be an effective way of engaging students and supporting their understanding. video production can be a costly and time consuming activity, in both staff time and if done to a high quality, equipment. the results show that the use of open educational resources is not viewed by students as a poor alternative and that, as long as properly facilitated and integrated into the lesson, the perceptions of students of this material do not diminish the perceived effectiveness of this method. in a ‘colder climate’ this has implications since the use of open access content can allow staff to focus the ways in which to facilitate the delivery of these open access educational resources instead of being concerned with generation of new content thus encouraging deeper learning and, in the case of this research, potentially improving the student learning experience by supporting communication, increasing interaction and giving a wider view of their chosen subject. the use of freely available online materials in class can enhance students’ learning if it is used to stimulate class discussion and not as a substitute for it. references alexaweb information company 2011. top sites in united kingdom. http://www.alexa.com/ topsites/countries/gb (accessed april 7, 2011). azyan, l. 2011. list of uk universities and their departments on youtube. http:// www.lizazyan.com/uk-universities-on-youtube/ (accessed april 10, 2011). brown, j.s. 2000. growing up digital: how the web changes work, education, and the ways people learn. change, march/april, 10�20. burgess, j., and j. green. 2009. youtube: online video and participatory culture. polity. lange, p. 2007. publicly private and privately public: social networking on youtube. journal of computer-mediated communication 13, no. 1: 361�80. miller, m. 2009. integrating online multimedia into college course and classroom: with application to the social sciences. merlot journal of online learning and teaching 5, no. 2: 395�423. moran, m., j. seaman, and h. tinti-kane. 2011. teaching, learning, and sharing: how today’s higher education faculty use social media for work and for play.http://www.pearson custom.com/educators/social_media_report.pdf (accessed april 9, 2011). palmer, n. 2011. iteach with itunes how about you? sociology source: resources and ideas for teaching sociology [blog]. http://www.sociologysource.com/home/2011/4/4/iteach-withitunes-how-about-you.html (accessed april 10, 2011). pearce, n. 2011. playlist of videos that i use when teaching introductory sociology at the durham university foundation centre. http://www.youtube.com/view_play_list?p� 93ff46f5bc6a27cf (accessed april 10, 2011). snelson, c. 2011. youtube across the disciplines: a review of the literature. merlot journal of online learning and teaching 7, no. 1: 159�69. snickars, p., and p. vonderau, eds. 2009. the youtube reader. harrow: wallflower press. weigel, v.b. 2002. deep learning for a digital age: technology’s untapped potential to enrich higher education. san francisco: jossey-bass. e. tan and n. pearce 132 http://www.alexa.com/topsites/countries/gb http://www.alexa.com/topsites/countries/gb http://www.lizazyan.com/uk-universities-on-youtube/ http://www.lizazyan.com/uk-universities-on-youtube/ http://www.pearsoncustom.com/educators/social_media_report.pdf http://www.pearsoncustom.com/educators/social_media_report.pdf http://www.sociologysource.com/home/2011/4/4/iteach-with-itunes-how-about-you.html http://www.sociologysource.com/home/2011/4/4/iteach-with-itunes-how-about-you.html http://www.youtube.com/view_play_list?p=93ff46f5bc6a27cf http://www.youtube.com/view_play_list?p=93ff46f5bc6a27cf http://www.youtube.com/view_play_list?p=93ff46f5bc6a27cf http://www.youtube.com/view_play_list?p=93ff46f5bc6a27cf http://www.youtube.com/view_play_list?p=93ff46f5bc6a27cf wesch, m. 2008. an anthropological introduction to youtube. recording of a lecture titled ‘‘the anthropology of youtube,’’ given by michael wesch on june 23, 2008 at the library of congress, washington, dc. http://hdl.handle.net/2097/6520 (accessed april 10, 2011). wilkes, l., n. pearce, and e. barker. 2011. fostering an ecology of openness: the role of social media in public engagement at the open university. in teaching arts and science with the new social media, ed. wankel. emerald publishing group. alt-c 2011 conference proceedings 133 http://hdl.handle.net/2097/6520 doi:10.3402/rlt.v19s1/7779 learning through online discussion: a framework evidenced in learners’ interactions yvonne c. bain* school of education, university of aberdeen, aberdeen, uk (received 14 june 2011; final version received 22 june 2011) online learning, often supported through online discussion, is not only a popular means of supporting off-campus learners, but increasingly has a place within campus-based learning courses. laurillard and others suggest that there are assumptions being made about learning through online discussion that have yet to be fully tested, and therefore there is a need to examine this area further. tutors and learners may benefit from having a greater insight and understanding of how engaging in asynchronous online discussion presents opportunities for learning on an individual and a collective basis. this research study focused on learners’ engagement with online discussion and their perceptions of how engaging in online discussion impacts on learning. this paper revisits learning through online discussion and proposes a framework, which emerges from the analysis of learners’ experiences. a grounded theory approach was used in the collection and analysis of six learner case studies within a higher education setting, exploring learners’ interactions in online discussion, and their perceptions of learning through online discussion. insights into the learners’ interactions were provided by the learners themselves through semi-structured interviews. the grounded approach to the analysis of the interviews enabled the learners’ voices to be heard in terms of what they thought about learning through online discussion. the insight enabled through the depth of description from the learners and the examination of the online interactions led to the development of a framework for learning through online discussion. the framework raises the importance of articulation as a key process in learning whilst highlighting the opportunities for collaborative informed thinking by engaging with the ideas of others. the focus given to the learning process through the framework will be of interest to tutors and learners who use online asynchronous discussion environments for learning. keywords: computer mediated communication; learners’ experiences; asynchronous discussion; grounded theory. introduction the potential of computer mediated communication (cmc) as a means of enabling interactions and sharing of ideas between learners, wherever and whenever the learner is situated, has been extolled for more than two decades now. ‘‘cmc has the potential to provide a means for the weaving together of ideas and information from many people’s minds, regardless of when and from where they contribute.’’ kaye (1989, 3) *corresponding author. email: y.c.bain@abdn.ac.uk alt-c 2011 conference proceedings 0136 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # y.c. bain. this is an open access article distributed under the 29 terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http:// creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7779 however, these benefits are not always realised by learners and there is much still to know about learning within an online discussion environment, as has been identified in other studies such as downing et al. (2007) who recognise the need to know more about effectively supporting and sustaining learner engagement in online environments. peters and hewitt (2010) note that there is a need to know more about the online behaviours of students in asynchronous discussion and a need to focus more on learning outcomes. the need to know more about learning, and the experiences of learning, in online discussion is highlighted within this paper. the paper reports on a qualitative, phenomenographic study which focuses on six learners’ experiences in a higher education setting in scotland, within two different subject disciplines. a grounded approach to analysing the learners’ interactions and their perceptions of their experiences, offers an insight of learning in these discipline settings. the findings from the study led to the development of a framework for learning through online discussion. the proposed framework draws together the experiences of the learners with insight available within research literature about learning through the processes of writing, as may be applicable to posting messages in an online discussion environment. implications for tutors and learners in asynchronous discussion are discussed. background mayes and de freitas (2007), and more recently laurillard (2009), expressed the view that there is nothing new to know about learning, with the implication that what is known about traditional modes of learning will transfer to learning in online environments. despite the increasing availability of guidelines relating to tutoring online (e.g. salmon 2000; laurillard 2002; garrison and anderson 2003), the guidance offered does not seem to transfer into consistent approaches to learning online. mcconnell (2006) highlights that not all learners are enthusiastic about engaging in online learning. other studies raise other issues in relation to the use of learning through online discussion, such as some learners not having the skills required to work in collaborative social constructivist environments (murphy et al. 2005). sharpe et al. (2006) and ellis et al. (2007) caution that some students do not know how to engage effectively in face-to-face discussion and that this may impact on them being able to benefit from the online discussion. kim and bateman (2010) likewise consider that there is a need for students to be helped in developing skills to engage with discussion. therefore, tutors need to know more about supporting online discussion, and there is a need to understand more about learners’ engagement with online discussion, as suggested by ravenscroft (2005) and goodyear and ellis (2008). engaging in asynchronous discussion potentially benefits learners by enabling them to take time to reflect on messages previously posted before making their own contribution. browne (2003) and macdonald (2006) consider that the time delay affords thinking time. others such as mcconnell (2000) and andrews and haythornthwaite (2007) highlight that the permanent nature of the messages posted provides opportunities to reflect on the messages as and when required by the learners. however, the extent to which message posts are actually used for reflection is questioned by ellis et al. (2007). the potential benefits of learning through online discussion are well documented but not always evidenced in practice, and current research suggests that there is a need to know more about how learners actually y.c. bain 30 spend their time online. gilbert et al. (2007) and so (2009) call for further research about student interaction in online environments. however, further consideration needs to be given to the nature of online discussion in that it is neither like oral discussion nor individual writing. andrews and haythornthwaite (2007) draw attention to the differences in modes of communicating in an asynchronous discussion environment, citing erickson (1999) who considered online discussion to be sometimes like formal published text and at other times to be like informal chat. the text-based medium of online discussion creates opportunities for enhancing learning by engaging in writing as a process of learning. bereiter and scardamalia (1987) consider that the act of composing helps to shape and develop understanding whilst writing, if the writing is consciously reviewed whilst composing. ritchhart and perkins (2008) further consider that the process of articulation or externalising thinking either by writing or by talking helps deepen thinking. in asynchronous discussion, there is an opportunity to use writing to articulate thoughts and therefore externalise thinking for others, and to respond to the thinking that is articulated by others. mercer (2000) suggests that group interactions, which are as a result of paying attention to others’ contributions, create opportunities for collective, shared thinking in which the individual thinking of a learner may be shaped by engaging with the thinking of others. therefore, there is recognition within the literature on the process of writing to shape thinking, and for individual thinking to be informed by collective thinking. however, the possible lack of skills to engage in discussion or the lack of attention to the messages posted by others means that the opportunities are not fully understood or recognised by learners. the learners’ contexts the study reported in this paper aimed to explore how learners engage with online discussion. in particular, the study sought to examine the approaches used by the learners and their perceptions of learning with others within an asynchronous discussion environment. the learners were studying at an undergraduate level within a higher education institution in scotland and from two different subject disciplines. a total of six learner case studies informed the research. three learners were studying a scottish degree level 3 (third year) history of art campus-based blended learning course; two learners were studying a scottish degree level 2 (second year) online theology course and one learner was studying a first year online theology course. all six learners were female with a range of prior educational experience. the history of art courses were taught on campus but had an online discussion component in the form of a virtual seminar which contributed 20% of the total course assessment. during the online component of the course, the usual class contact time was suspended for a period of two weeks. learners engaged in discussion about a given virtual seminar topic in groups of four learners. each learner in the group had a different topic for which they had to write a 2000-word essay prior to the start of the virtual seminar. the essay had to be uploaded in the form of a web page making it publicly available to the others in the group. learners were explicitly told to engage in the online discussion as an interview format, by asking a minimum of three questions about their peers’ essays and responding to questions asked of them within 72 hours. within these guidelines, they could engage in the discussion wherever and whenever was suitable to them. the learners undertook this course as alt-c 2011 conference proceedings 31 part of an honours degree in history of art. as this was the third year of study in this discipline setting, they were used to engaging in oral discussion within campus-based seminars, and writing about history of art, but this was the first time that the learning experience required engagement in asynchronous online discussion. the theology courses were totally online, and had no campus-based learning component. the course learning materials were available within the online environment. there were six different topics in which the learners were expected to engage in discussion for a period of two weeks per topic. in the first week the asynchronous discussion of the topic was within a small group. in the second week the groups then contributed to a whole class asynchronous discussion. each learner was at a geographical distance from the campus and was studying the course as part of a certificate, diploma or degree in theology. each of theology case studies was a mature learner and had used online learning and asynchronous discussion in a previous course, with the same tutor and with the same organisation of the learning environment and the learning activities. engagement in the online discussion for the theology courses was not compulsory. however, participation in the online discussion was clearly expected. the course study guide highlighted that the learners were to engage with each other and to contribute to the online discussion, and noted that each learner would take on the task of summarising the group discussion at the end of a week. the case studies case studies were selected from learners who had agreed to have their interactions tracked and had been interviewed about their perceptions of learning through online discussion. preference was given to learners who were in the same online discussion group. the three history of art case studies were drawn from a class of 24 learners. in the theology settings, the three case studies were self-selecting as they were the only volunteers from two classes (of a total of 15 learners). the researcher was an observer of the online interactions and had no tutoring role in either of the discipline settings. the setting for the study was naturalistic (gubrium and holstein 1997) as the data were gathered from the online discussion areas, which were part of the intended course learning processes for the learners. there has not been any attempt to try to impose a particular structure or approach to the online discussions to satisfy a research requirement. methodology the research study aimed to examine the learners’ perspective of their experience of online discussion and how they engage with online discussion. this required a qualitative research methodology which, in this study, takes a phenomenological and interpretative approach to the analysis of semi-structured interviews with the learners. in order to do so, the study adopted a grounded approach to deriving a theory from the data (glaser and strauss 1967). a case study approach (yin 2003; stake 2006) was used to focus on the individual learners, to gather in-depth data about the learner and her experiences of engaging in online discussion. data were gathered from one institution, two different discipline contexts each with a different mode of delivery of learning, and from three different learners in each of the discipline contexts. participants were invited to take part in the research before y.c. bain 32 the online discussion activity took place. for each case study, tracking tools available within the webct online environment were used to gather data about the learner’s online interactions; discussion posts were captured to explore the nature of the contributions made; and a semi-structured interview was used to gain insight to the learner’s perceptions of her engagement in, and with, asynchronous discussion. the interviews took place after the discussion activity was completed and made use of the tracking detail to prompt recall of activity and discussion of the online interactions. visual timelines of interactions were manually created (see figures 1 and 2). gibbs et al. (2006) have created a tool for online discussion which will generate visual timelines, but this was not used in this study. the grounded approach to analysing the interviews highlighted themes within students’ comments in relation to the need to articulate clearly for others; engaging with the ideas of others; temporal aspects of using the time delay to research and prepare responses; and the impact of assessment for example. an iterative process was involved in creating the framework by examining the interview statements, exploring the interactions that took place, making connections between students’ perceptions and their online behaviours, and reconsidering what could be understood about the learning processes from the research literature. learners’ interactions the tracking data revealed differences in how learners engaged with each other in the online discussion environment. a visual timeline was created to show the pattern of interactions. for example, in the history of art contexts, two of the case studies, camille and rosalba (pseudonyms) regularly logged on to the learning environment, but had different time-scales for engaging with others. camille was shown to respond regularly within 24 hours of questions being asked of her, as shown in figure 1, which shows the date and time of posting of a particular message (number), and the length of response given (words) for two of the discussion threads. rosalba chose to delay her responses as shown in figure 2. this resulted in a more restricted engagement with her peers as there was very little time left for peers to ask anything further about the responses given by rosalba. the pattern of interaction in rosalba’s discussion forum shows a pattern of question�answer only with no follow-up, whereas camille’s interactions show a more extended question� answer engagement with the interactions in the third thread (figure 1, thread 3: social viewpoint). rosalba’s experience supports the views expressed by jeong and frazier (2008) and dringus and ellis (2010) that late posts are less likely to receive a response, as can be seen in figure 1 when the late post by rosalba does not receive a response from camille. further insight was gained from the semi-structured interviews with the learners, which revealed for instance that rosalba was unaware of restricting her opportunities for engaging with others. her perception was that the time delay allowed for further research on a topic and that ‘‘you’re more likely to sort of interact and have a longer sort of discussion about something’’. she also made reference to the convenience of being able to engage in discussion as anytime-anywhere learning (hiltz and goldman 2005), commenting that: alt-c 2011 conference proceedings 33 thread 3: social viewpoint 14th november 15th november 16th november 17th november 18th november 19th november 20th november 491 rose posts question 14/11 17:47 493 camille replies 152 words 15/11 10:21 494 tutor posts further question 16/11 12:24 496 camille responds 166 words 16/11 22:10 497 lilla asks question 17/11/08 12:04 498 camille responds 243 words 17/11 14:54 499 rose asks further question 17/11 15:45 503 camille responds 174 words 17/11 18:00 508 rosalba asks question on different aspect 20/11 23:05 figure 1. visual timeline of camille’s interactions. y .c . b a in 3 4 they are more likely to get involved because you don’t have to turn up anywhere really i mean you could just sit in bed and do it. . . . if you’ve got something else to do you can go and do it and it’s learning in your time. [rosalba] rosalba’s perceptions of engaging with others did not necessarily match up with the reality of her engagement with others. the history of art online discussions formed part of the course assessment and that may have impacted on how the learners interacted with each other. rosalba and other learners commented on engaging with others because it was assessed and they wished to get a good mark. marguerite for example noted that: people want to get the best marks they can and i think that probably shapes how people do things. [marguerite] however, the assessment was not necessarily enough of a motivator for all participants, as in one discussion forum it was observed that three learners did not engage with the fourth member of the group. in the theology course settings, there was no compulsion to participate and no assessment of the online contributions. in the theology settings, there were very low levels of interaction between learners. the lack of engagement with others meant that a sense of frustration developed as learners realised that this could have impacted on their learning. for example, in the level 2 course, martha noted her frustration at the lack of interaction by others: i found it very frustrating.. the lack of input from people, so whenever there was a contribution, i would have replied to it. [later in the interview] i think i could have got more out of it if, other people had contributed a bit more... i was a bit frustrated that folk didn’t take part. [martha, level 2 theology] ruth for instance, relied on interacting with the tutor mainly. in her interview she too commented on the lack of presence of other learners: i felt sometimes, you were putting your answers and you were almost, well, forgetting about the others that were there. [ruth, level 1 theology] this was clearly not the level of interactive discussion that the tutor had planned for the course. the lack of response, or delayed response, was experienced in both disciplines and impacted on learners in different ways. in the theology course, martha commented valued art form thread initiated by camille corot and classicism thread initiated by rose 13th november 14th november 20th november 21st november 455 camille asks question 13/11 13:49 456 rosalba replies 674 words 21/11 10:46 457 rose asks question 14/11 14:48 458 rosalba replies 251 words 20/11 20:30 figure 2. visual timeline of rosalba’s threads 2 and 3. alt-c 2011 conference proceedings 35 on realising that the lack of engagement may have impacted on what she could have learned. in the history of art course, camille commented on losing her thought processes when peers did not respond in a timely manner: there was one person in my group who i answered i asked two questions and it was about four of five days before they bothered to reply which was a bit annoying because by the time when i went back to read their thing i’d forgotten what my first thought was. [camille, history of art] her comment is a reference to the interactions shown in figure 2, in which rosalba takes several days to respond. the interactions of learners were not necessarily matched with the learners’ perceptions of how they engaged with other. the negative impact that the lack of engagement can have suggests that for small group discussion there is a need to ensure that all are actively involved. studies, such as romiszowski and mason (2004) and beaudoin (2005), highlight that lurkers (those who read messages but do not post) are engaged in learning through reading the contributions of others. however, the potential impact of their lack of overt engagement should be noted. insights to learning in the interviews the learners commented on broadening and deepening their understanding of a topic within their subject discipline. for instance, camille commented that her engagement with others in the discussion area and with reading other people’s essays helped her broaden her thinking about her own essay. marguerite spoke of developing a more critical approach to examining someone’s work as a result of reading her peers’ contributions. marguerite considered that her engagement in the discussion, and the questioning of her peers, and articulating her thoughts for others helped clarify her understanding about a topic. marguerite (in history of art) and martha (in theology) each raised the aspect of writing for an audience, being aware that their responses would have to be constructed carefully to try and get an argument across without misinterpretation by others which helped articulate their thinking. whilst there are contradictions in the literature about whether the participants of asynchronous discussion actually use the time delay for reflection, there was evidence of reflection in these case studies. reflection was implied by camille, who made reference to having time to collect her thoughts; whilst ruth (in theology) spoke of using time to reflect on reading. the learners benefited in some way by engaging in reflection, by taking time to shape and develop their messages and in articulating their thoughts for others to understand. contribution from the learners’ experiences, it seems that there is a need for learners to understand more about the processes involved in learning through online discussion so that they can maximise the opportunities that are presented. there are opportunities to develop learning through online discussion, but these are not fully recognised or acted upon by learners. drawing from the learners’ experiences and the literature, a framework for learning through online discussion emerges. y.c. bain 36 the learners provided evidence of making use of the asynchronous nature to reflect and to think some more about the topic of discussion in order to further shape their understanding of the topic. these activities are covert activities, not evident to others unless some tangible output of this is shared with others. figure 3 represents the hidden (covert) processes involved. as identified by the learners, there are benefits to be gained from articulating their thinking for others to read. the act of writing can help develop understanding of the subject itself (bereiter and scardamalia 1987; mercer 2000). figure 4 highlights the role of articulture in learning. when a learner chooses to post a message, she/he may choose to make overt connections with the thinking of others by making a response which has connections with the previous messages posted and adding to the thread of discussion (e.g. as happened in camille’s interactions in figure 1). alternatively, a message may be posted that has no connections with other messages, and is posted as an isolated, unconnected message figure 5 represents the overt articulation which occurs when posting a message. figure 6 highlights the potential benefits to an individual learner from being informed by others through reading and reflecting on the messages posted by others, from composing a written response and articulating thinking to make thinking ‘visible’ to others (ritchhart and perkins 2008). this is not a one-way process, however, and just as an individual can be informed by the contributions of the group, the individual can inform the thinking of others in the group, potentially building up collective thinking. the individual’s learning may be prompted by, and contribute to, the thinking of others if she/he chooses to read and take account of the messages posted by others. thus, there are reciprocal learning opportunities for an individual and for the others in the group. figure 7 summarises this reciprocity of learning opportunities created when learners engage in learning through online discussion. the top half of the figure represents the opportunities for an individual learner whilst the bottom half represents the opportunities of others that arise as a result of their articulation of thinking and their reflections of the contributions made by other individuals. thus all learners have the opportunities to benefit from the thinking of others, and from articulating their own thinking. articulation is at the heart of making thinking known to others. articulation of thoughts may be connected with the thinking of others (overt collective informed reflecting on the thinking of others reflecting on own perspective / understanding (1) covert individual thinking individual learner responds to some stimulus such as a set learning activity. this may involve researching, rehearsing or reshaping thoughts. requires thinking time. thinking is individual and covert. (2) covert collective informed thinking individual reads messages of others. connections are made with others' thinking which can reaffirm, negate or reshape thinking. thinking is individual and covert but is influenced or informed by others. figure 3. covert processes. alt-c 2011 conference proceedings 37 thinking) when the message posted draws from, or is openly influenced by, the messages of others. articulation may be unconnected to others (overt individual thinking) exemplifying individual thinking shown as isolated posts. to move from individual thinking to collective, shared thinking requires the individual learners to read, reflect and make connections which are then articulated in messages posted. the overt articulation of thinking creates possibilities for individual thinking to develop into new, collective and shared thinking. mercer (2000, 129) cautions that: ‘‘cmc will only be as good for collective thinking as its users make it’’. if learners are not fully aware of the opportunities for developing learning through cmc then the result may be a lack of overt engagement which impacts on the individual and other learners in the group. reflecting on the thinking of others reflecting on own perspective / understanding composing reflecting whilst writing (1) covert individual thinking individual learner responds to some stimulus such as a set learning activity. this may involve researching, rehearsing or reshaping thoughts. requires thinking time. thinking is individual and covert. (2) covert collective informed thinking individual reads messages of others. connections are made with others' thinking which can reaffirm, negate or reshape thinking. thinking is individual and covert but is influenced or informed by others. (3) articulation individual learner articulates thinking by composing a message moving from the internal process of thinking to externalising thinking through writing. this may be edited and reviewed any number of times before being finalised. figure 4. articulation as a process of learning. unconnected post connected post (3) articulation individual learner articulates thinking by composing a message moving from the internal process of thinking to externalising thinking through writing. this may be edited and reviewed any number of times before being finalised. oi (4) overt individual thinking individual learner posts a message, making thinking visible but unconnected to other posts. covert thinking becomes overt and potentially contributes to collective thinking. messages posted are not part of, or relevant to, a threaded discussion oc (4) overt collective thinking individual learner posts a message, which builds on or connects with other posts making thinking visible and contributes to shared understanding. individual covert thinking becomes overt and collective. thread of discussion build up. figure 5. overt articulation. y.c. bain 38 this study is limited in that none of the case studies focused on a male participant and none of the case studies focused on a non-contributor to online discussion. further, it may be that the learners’ interactions have been influenced by the non-participating presence of the researcher, although there was no evidence to suggest that this was so. the lack of inclusion of learners who chose not to contribute to the online discussion does not invalidate the analysis of the case studies, but rather highlights that this is an aspect that would be worth following up in further studies. it may be particularly helpful to explore the relevance of the framework for learning through discussion for non-contributors and to gain insight into their reasons for not contributing to online discussion. conclusion the case studies provide an insight into learners’ approaches to online discussion, the strategies that they use, their reactions to engaging with others and their perceptions of learning through discussion that is not readily available in other research studies. the framework for learning through online discussion which emerged from the reconsideration of the processes, as identified by the learners and related to in the literature, highlights the importance of articulation as part of the learning process and the importance of building on the contributions of others. in the two discipline settings within this study, the tutor had designed activities specifically intended to engage learners in online discussion, but in both settings there were limitations to how learners chose to engage in this. tutors need to be explicit about the rationale for including online discussion as part of the learning experiences, not just in terms of noting expectations of frequency of posting or desired response times, but in terms of reflecting on the thinking of others reflecting on own perspective unconnected post connected post composing reflecting (1) covert individual thinking individual learner responds to some stimulus. (2) covert collective informed thinking individual reads others' messages. (3) articulation of own thoughts oi (4) overt individual thinking individual posts an unconnected message. oc (4) overt collective thinking individual posts a connected, threaded message. figure 6. individual learning opportunities created through online discussion. alt-c 2011 conference proceedings 39 what is meant by discussion and how discussion is expected to develop as the learning activity progresses. tutors should explain to learners why online discussion in the form of written contributions may be helpful in terms of developing thinking and moving from an individual perspective to a collective informed perspective or shared perspective. the proposed framework for learning through online discussion provides an alternative perspective to the conversational framework of laurillard (2002), by focusing on the learner engagement rather than the tutoring role, and by raising the essential role of articulation in learning through online discussion, without which thinking cannot be shared and feedback (which is a key aspect of laurillard’s framework) cannot be provided. further development of the framework might provide a means of using the framework for learners’ self-analysis or tutor-analysis of engagement with the learning opportunities. it is possible for the framework to be used as the basis of evaluative questions which encourage learner and/or tutor reflection on the covert processes of engaging with discussion, the overt processes of engaging in discussion, and the manner of articulation. it may be possible for further research to share the unconnected post connected post reflecting on the thinking of others reflecting on own perspective composing reflecting reflecting on the thinking of others reflecting on own perspective unconnected connected composing reflecting (4b) other's message adds to discussion thread (3a) articulation other (learner or tutor) (4a) overt individual thinking of other unconnected, isolated message (1a) other covert thinking another individual considers response. (2a) other informed by collective thinking 'other' reads groups' messages. (1) covert individual thinking individual learner responds to some stimulus. (2) covert collective informed thinking individual reads others' messages. (3) articulation oi (4) overt individual thinking individual posts an unconnected message. oc (4) overt collective thinking individual posts a connected, threaded message.individual others individual others figure 7. a framework for learning through online discussion. y.c. bain 40 framework with the learners and tutors, and to then use that as a basis for examining the learning experiences. whether having an understanding of learning through online discussion expressed in this way is helpful for tutors and/or learners in terms of supporting and developing their use of online discussion remains for further research. references andrews, r., and c. haythornthwaite. 2007. introduction to e-learning research. in the sage handbook of e-learning research, ed. r. andrews and c. haythornthwaite, 1�52. london: sage publications ltd. beaudoin, m. 2005. reflections on seeking the ‘invisible’ online learner. aect 2005. http:// www.learndev.org/dl/ibstpi-aect2005-beaudoin.pdf (accessed september 17, 2010). bereiter, c., and m. scardamalia. 1987. the psychology of written composition. hillsdale, nj: lawrence erlbaum associates, inc. browne, e. 2003. conversations in cyberspace: a study of online learning. open learning 18, no. 3: 245�59. downing, k.j., t.-f. lam, t. kwong, w.-k. downing, and s.-w. chan. 2007. creating interaction in online learning: a case study. alt-j 15, no. 3: 201�15. dringus, l.p., and t.j. ellis. 2010. temporal transitions in participation flow in an asynchronous discussion forum. computers & education 54: 340�9. ellis, a.r., p. goodyear, a. o’hara, and m. prosser. 2007. the university student experience of face-to-face and online discussions: coherence, reflection and meaning. alt-j-association for learning technology journal 15, no. 1: 83�97. erickson, t. 1999. persistent conversation: an introduction. journal of computer mediated communication 4, no. 4 jcmc http://jcmc.indiana.edu/vol4/issue4/ericksonintro.html (accessed september 17, 2010). garrison, d.r., and t. anderson. 2003. e-learning in the 21st century a framework for research and practice. oxon: routledgefalmer. gibbs, w.j., v. olexa, and r.s. bernas. 2006. a visualization tool for managing and studying online communications. educational technology & society 9, no. 3: 232�43. gilbert, j., s. morton, and j. rowley. 2007. e-learning: the student experience. british journal of educational technology 38, no. 4: 560�73. glaser, b., and a. strauss. 1967. the discovery of grounded theory strategies for qualitative research (observations). london: weidenfeld and nicolson. goodyear, p., and r.a. ellis. 2008. university students’ approaches to learning: rethinking the place of technology. distance education 29, no. 2: 141�52. gubrium, j., and j. holstein. 1997. the new language of qualitative method. ny: oxford university press. hiltz, s.r., and r. goldman. 2005. what are asynchronous learning networks? in learning together online research on asynchronous learning networks, ed. s.r. hiltz and r. goldman, 3�18. london: lawrence erlbaum associates. jeong, a., and s. frazier. 2008. how day of posting affects level of critical discourse in asynchronous discussions and computer-supported collaborative argumentation. british journal of educational technology 39, no. 5: 875�87. kaye, a. 1989. computer-mediated communication and distance education. in mindweave: communication, computers and distance education, ed. r. mason and a. kaye, 1�21. oxford: pergamon press. kim, h.k., and b. bateman. 2010. student participation patterns in online discussion. incorporating constructivist discussion into online courses. international journal on e-learning 9, no. 1: 79�98. laurillard, d. 2002. rethinking university teaching a conversational framework for the effective use of learning technologies, 2nd ed. abingdon, oxon: routledgefalmer. laurillard, d. 2009. the pedagogical challenges to collaborative technologies. computer supportive collaborative learning 4: 5�20. alt-c 2011 conference proceedings 41 http://www.learndev.org/dl/ibstpi-aect2005-beaudoin.pdf http://www.learndev.org/dl/ibstpi-aect2005-beaudoin.pdf http://jcmc.indiana.edu/vol4/issue4/ericksonintro.html macdonald, j. 2006. blended learning and online tutoring. aldershot: gower publishing limited. mayes, t., and s. de freitas. 2007. learning and e-learning: the role of theory. in rethinking pedagogy for a digital age designing and delivering e-learning, ed. h. beetham and r. sharpe, 13�25. oxon: routledge. mcconnell, d. 2000. implementing computer supported cooperative learning, 2nd ed. london: kogan page. mcconnell, d. 2006. e-learning groups and communities. maidenhead: the society for research in to higher education & open university press. mercer, n. 2000. words and minds how we use language to think together. london: routledge. murphy, k., s. mahoney, c.-y. chen, n. mendoza-diaz, and x. yang. 2005. a constructivist model of mentoring, coaching, and facilitating online discussion. distance education 26, no. 3: 341�66. peters, v.l., and j. hewitt. 2010. an investigation of student practices in asynchronous computer conferencing courses. computers & education 54: 951�61. ravenscroft, a. 2005. towards highly communicative e-learning communities. in education in cyberspace, ed. r. land and s. bayne, 130�45. abingdon, oxon: routledgefalmer. ritchhart, r. and d. perkins. 2008. making thinking visible. http://www.pz.harvard. edu/vt/visiblethinking_html_files/06_additionalresources/makingthinkingvisibleel.pdf (accessed july 27, 2010). romiszowski, a., and r. mason. 2004. computer-mediated communication. in handbook of research on educational communications and technology. 2nd ed., ed. d. jonassen, 397�431. association for educational communications and technology: http://www.aect.org/ edtech/15.pdf (accessed august 9, 2010). salmon, g. 2000. e-moderating the key to teaching and learning online. london: kogan page limited. sharpe, r., g. benfield, g. roberts, and r. francis. 2006. the undergraduate experience of blended e-learning: a review of uk literature and practice. the higher education academy. http://www.heacademy.ac.uk/resources/detail/ourwork/teachingandresearch/under graduate_experience (accessed july 09, 2010). so, h. 2009. when groups decide to use asynchronous online discussions: collaborative learning and social presence under a voluntary participation structure. journal of computer assisted learning 25, no. 2: 143�60. stake, r.e. 2006. multiple case study analysis. new york: the guilford press. yin, r.k. 2003. case study research design and methods vol. 5, 3rd ed. applied social research methods’’. thousand oaks, ca: sage publications ltd. y.c. bain 42 http://www.pz.harvard.edu/vt/visiblethinking_html_files/06_additionalresources/makingthinkingvisibleel.pdf http://www.pz.harvard.edu/vt/visiblethinking_html_files/06_additionalresources/makingthinkingvisibleel.pdf http://www.aect.org/edtech/15.pdf http://www.aect.org/edtech/15.pdf http://www.heacademy.ac.uk/resources/detail/ourwork/teachingandresearch/undergraduate_experience http://www.heacademy.ac.uk/resources/detail/ourwork/teachingandresearch/undergraduate_experience recognition of animated icons by elementaryaged children tom jones department of educational psychology, university of calgary, 2500 university drive n.w., calgary, alberta, canada abstract this paper describes a study to investigate the recognizability of and preference for animated icons by elementary-aged aged children. fourteen typical computer-related tasks (e.g., copy, move) were viewed by 60 school-children in two iconic formats: animated and static. the content of the icons and the computer process or action they mimicked were drawn from a previous study in which a similar group of children was asked to depict gesturally their interpretation of the 14 tasks. results indicated that the animated version of the icons was more recognizable and that the children greatly preferred the animated icons over the static icons. implications for the design of enhanced user-interfaces for children are noted. introduction two advancements in the microcomputer industry direct-manipulation user interface (dmui) and multimedia have opened the door for the provision of enhanced computer interaction. the former has received a great deal of attention and has resulted in a proliferation of such interfaces running on many different platforms. the latter is also causing a stir in the industry and promises to allow for the integration and manipulation of text, graphics and sounds (voice and music). from the perspective of a direct-manipulation interface, however, children in educational settings have benefited little from these advancements. presently, they have access to microcomputers which have been designed for general-purpose and adult use (ibm, apple ii series, apple macintosh, amiga). some work has been carried out by the designers of the ontario government-supported microcomputer (icon), a system targeted solely for educational use (mclean, 1983) and by researchers at the children's television workshop (strommen, 1990), but there still exists a wide gap between the features of existing interfaces 40 altj volume 1 number 1 and features which reflect a sensitivity to developmental change. the expectation is that children will be able to interpret and to make productive and efficient use of an interface that was designed for another class of users, i.e. adults. in the classroom, children are provided with 'out-of-package' software (macwrite, pcpaint or the like) and the assumption is made that either that the children will intuit how many of the functions of these programs are accessed and executed or that they will become productive users after some minimal instruction. given that there are great differences both qualitative and quantitative between children and adults, it follows that the nature of child-computer interaction will be quite different from adult-computer interaction and that a starting point for a closer look at child-computer interaction might be the features inherent in the dmui. to investigate one of these features, i carried out a study1 which focused on the recognizability of and preference for animated icons by young children. building on previous work which attempted to determine the pictorial content of the icons (jones, 1990), elementary-aged school children were asked to describe what computer-related action or message was being depicted by a moving image (animated icon) or by a static image, and to indicate which of the two formats they preferred. results are reported here, together with a discussion of future directions. animated icons given that graphical icons (as part of a more general metaphor) comprise one half of a direct-manipulation interface (gittins, 1986), one might ask: what should constitute the features of these icons with which children will interact in a 'seeing and pointing' i.e. direct-manipulation interface? townsend (1986) recommends that icons which are designed for use by children should "clearly depict, indicate and distinguish a program's commands and operations" and "should suggest and indicate a command intention rather than just duplicate or represent a particular pictorial form." the goal of educational software developers, then, is to incorporate icons which are realistic and meaningful for children and which leave little room for misinterpretation. for the most part, the iconic component in existing interfaces for children's use merely mirrors what is available in the commercial marketplace and does not respond to the developmental world of the child. one possibility is to consider the utility of animated icons as a feature in a dmui. a similar approach has been successfully used in the area of pre-reading bridge reading program (dewsbury, 1983) in which the instructor employs sounds, lpgogiaphs (drawings) and physical gestures to teach young children how to read. in this program, the physical gestures by both instructors and students are viewed as a temporary event in the instructional process and are meant, at the outset, to provide additional cues to the beginning reader. eventually the gestures are dropped as the child moves toward a word-recognition capability. the gestural component, which is heavily used at the outset of the instruction, is considered pivotal in that the instructional process is, at that beginning stage, drawing on the child's knowledge of commonplace movements. thus, the children are exposed to not only a static series of letters in a book or on a card, but also are provided with additional information by means of the accompanying gesture. in a previous study (jones, 1990), an attempt was made to determine how children would indicate to a second party by gesture their notion of computer tasks or operations (e.g. cut, 41 tom jones animated icons and elementary-aged children move, copy). the 14 operations selected involved actions and were provided to the subjects during the experiment as verbs (table 1). the goal was to solicit from the population of interest their ideas on how best to indicate overtly (but without words) these actions, and to use this information in the design of the content of the computer icons. during a 40-minute testing session, 40 children (6, 8 and 10 year-olds) were asked to put themselves in a situation in which they could see a friend but could not verbally communicate with that friend. the subjects were asked to look at a drawing of a child who was standing behind, a window and to move their hands or body in such a way as to communicate to the friend behind the window what it was that they wanted the friend to do. for example, in order to indicate to the friend that the subject wanted something moved, the majority of the subjects moved their two hands in tandem from left to right or right to left in an arc-like fashion. task 1. put away 2. get 3. send 4. open 5. close 6. write 7. draw 8. point 9. move 10. erase 11. drag 12. copy 13. cut 14. paste table 1: operators presented to three age groups although there was disparity across age levels for some of the 14 tasks, the content of the responses and the percentage of subjects who exhibited similar responses resulted in obvious trends. to produce animated icons for testing, each operator in table 1 for each of the three age levels was scripted in a storyboard fashion; a model posed for each step in the scripts and these steps were then digitized using a video camera and a screen-capture program (macvision). each image was subsequently cleaned up or 'outlined' and the images or steps were then cut and pasted to an animation program (videoworks ii) as cast-members, and assembled into an animated icon. one example of these animated icons is presented in figure 1. in order to determine if the icon content generated during the previous study (jones, 1990) had the same connotation for other children of the same age, the icons were presented to a similar sample of elementary-aged students in two formats: animated and static. the goals were two-fold: • to see if these children attributed the same meaning to i.e. recognized the actions generated by the previous subjects # to solicit from them their preference as to an animated or static version of the icon for the computer task. method subjects sixty children from an elementary school in calgary (20 from each of three age groups 6, 8 and 10) were tested. there were an equal number of males and females. the primary criterion for subject selection was that the child had not previously used a macintosh computer or a, computer with a similar interface. 42 alt-j volume 1 number 1 procedure each subject was tested in two phases. for the first phase, the subjects were randomly assigned to the animated or static icon condition and were asked to describe the' action that they saw depicted on the monitor for all 14 computer-like tasks. for the second phase, they were told what each of the 14 icons (animated or static) stood for, and were then asked which of the two formats they preferred. for both phases, the 14 items were randomly presented to the subjects. analysis frequency counts were carried out for both experimental phases and chi-square tests were run. tttk figure 1: example of an animated sequence for 'draw' results for the first phase of the testing ("what does this picture [animated or static] mean?"), the results are contained in table 2. for 13 of the 14 computer tasks listed in table 2, the percentage of subjects who were asked to describe the computer task depicted by the animated icons was higher than those who were exposed to the static icon condition. however, in many instances, the spread between the two percentage scores was not high. for four of the animated tasks and six of the static tasks, less than 50% of the subjects correctly indicated the action involved. this is not surprising given that the subjects in this study were simply asked to state in one or two words what the moving or static drawing on the computer screen represented; that is, they were not provided a context in which to respond. analysis of the age and gender factors did not result in marked differences. the sex-by-animatedicon analysis showed no differences, while the age-by-animated-icon results indicated that there were differences for the 'draw' and 'copy' tasks. for the former, the 8 and 10 year-olds correctly 43 tom jones animated icons and elementary-aged children operation 1. put away 2. get 3. send 4. open 5. close 6. write 7. draw 8. point 9. move 10. erase 11. drag 12. copy 13. cut 14. paste animated 16.7 3.3 66.7 43.3 20.0 96.7 83.3 80.0 70.0 40.0 60.0 83.7 80.0 63.7 static 6.7 6.7 60.0 30.0 16.7 90.0 60.0 76.7 20.0 20.0 56.7 76.7 66.7 63.3 operation 1. put away 2. get 3. send 4. open 5. close 6. write 7. draw 8. point 9. move 10. erase 11. drag 12. copy 13. cut 14. paste animated 75.0 65.0 81.7 51.7 51.7 78.3 78.3 51.7 85.0 91.7 76.7 70.0 86.7 78.3 static 25.0 35.0 18.3 48.3 48.3 21.7 21.7 48.3 15.0 8.3 23.3 30.0 13.3 21.7 table 2: correct response for animated versus static condition (%) table 3: preferences for icon format (%) identified the animated 'draw' icon more often than the 6 year-olds (x (2,n=60) = 6.24, p <.o5) and the same result held for the animated 'copy' icon (x^ (2,n=60) = 12.0, p <.o5). with respect to the static icons, the age-by-static-icon analysis showed no differences, while the sex-by-staticicon comparison showed the male subjects incorrectly identified the 'open' icon more often than the female subjects (y~ (l,n=60) = 14.36, p <.o5). the results for which format the subjects preferred were more clear-cut. in each instance, more than 50% of the subjects selected the animated icon as the one which best captured the designated action. table 3 lists the results across all subjects. what should be noted here is that, for this treatment, the subjects were aware of what each icon stood for in that the stimulus consisted of the animated icon, the static icon, and the operator word which described the action. the subjects were asked to pick the icon format (animated or static) which they thought best described the operation. conclusion it appears that the use of animated icons as a feature in a child-computer interface has merit. in addition to the fact that children are better able to decipher the meaning of animated icons over static icons, they also show a marked preference for the moving icons. given that one of the tenets of a dmui is that the user should spend less time processing (and eventually 44 alt-j volume 1 number 1 deciding) what a system's options are, if the system itself is to be transparent to the user (shneiderman, 1987), then the incorporation of animated icons which provide children with meaningful information should be seriously considered. this would fit with the goal of designing an interface for children which puts as little demand as possible on their short-term or working memory. not surprisingly, there were no age or sex differences for either of the two icon formats or for the preferences. what warrants further study is the content (animated or otherwise) of the icon itself. with respect to the results reported here, a number of the icons presented a problem to all three age groups in that the icons either did not contain enough information or contained ambiguous information and, as a result, were not correctly identified by the majority of the subjects. the reasons for this might range from a simple mis-reading by the subjects, to the gap between the pictorial content of the icon and the lack of an overall context or, in dmui terms, the metaphor in which the recognition task was presented. the more-encompassing matter of a metaphor for classroom use is deserving of more detailed analysis. a possible next step is to generate a series of prototypical interfaces for children and to test them for effectiveness and for children's reactions to them. a number of questions should be addressed: • what other features should be built into an interface designed for use by children? • if animated icons are an effective way to provide information to children about the functionalities of a system, how might the child proceed once the animated icon is selected? would the child select the object upon which he or she wishes to carry out some operation first, and then select the animated icon which represents the desired operation, or vice-versa? • how many levels of options should be made available to the child? should the system or application program contain options that can be crossed with each and every other option, or should there be some measure of a hierarchical structure? all of these questions (and many others) call for a comprehensive analysis of generalizable interface for child-computer interaction. it seems only fair that if children in schools are going to be using these technologies for learning, and if large expenditures are going to be required to provide these tools, a rigorous and child-centred approach must be taken to the design and implementation of this interface. note 1. the author wishes to thank the apple canada education foundation and the alberta teachers' association for their support. 45 tom jones animated icons and elementary-aged children references dewsbury, a. (1983), bridge reading handbook, toronto, o.i.s.e. press. gittins, d. (1986), 'icon-based human-computer interaction', international journal of man-machine studies, 24, 519-43. hutchins, e.l., holland, j.d. and norman, d.a. (1986), 'direct manipulation interfaces', in norman, d . a and draper, s.w. (eds), user centered system design: new perspectives on human-computer interaction, new jersey, lawrence erlbaum associates. jones, t. (1990), 'children and animated computer icons', journal of research on computing in education, 22, 3, 300-9. mclean, r. s. (1983), 'ontario ministry of education specifies its microcomputer', in proceedings of the fourth canadian symposium on instructional technology, winnipeg, national research council of canada. shneiderman, b. (1987), designing the user interface: strategies for effective humancomputer interaction, reading, ma, addison-wesley. strommen e. (1990), 'research in interactive technologies at the children's television workshop', educational technology research and development, 38, 4, 65-80. townsend, b. m. (1986), iconic commands and command systems, unpublished manuscript, sheridan college, toronto, software evaluation unit. 46 doi:10.3402/rlt.v19s1/7815 towards a personalised learning mesh: the implementation of a low overhead, multipath learning tool iain stewart*, william mckee and kevin porteous school of engineering and computing, glasgow caledonian university, glasgow, uk (received 2 march 2011; final version received 1 june 2011) many studies have shown that students today live in an environment of multiple, simultaneous, short-lived stimuli which they access from wherever they may be. however institutional teaching is still based on traditional, long, sequential, attended presentations. in order to bridge that gap, there have been a number of moves over the past few years to develop and integrate lecture capture into the learning environment. often these systems are large and require a major commitment from the institution in terms of licences and infrastructure. given the constrained financial environment for many academic institutions, these systems are not a viable option for many. the authors have extended their normal lecture capture activity in their teaching to form an integrated learning resource. the captured media is mounted into a content management system which allows the media to be repurposed along with other content to provide an integrated support tool for student enquiry and self study which better matches their unstructured social experience. this paper describes the development of the pilot system based on a minimal hardware requirement and limited post processing. the evolution of the system pilot is described and the development of the specification which then led to the live prototype is discussed. issues that impact on the effectiveness of the prototype are covered and the strategy (based on classroom feedback) for developing the prototype into a full system for deployment across a range of desktop and mobile platforms is introduced. keywords: web 2.0; personalised learning; student centred; lecture capture; repurposing; integration introduction and goals it has been generally recognised that lectures are not a particularly good mechanism for engaging students with course content. they do have several advantages however, when done well. these include: . allowing everyone to be presented with a common baseline of material. . enabling the lecturer to recognise areas where the concepts are not coming over well and to make digressions and expansions on the content as appropriate. . permitting students to raise queries and for the entire class to hear the response. . providing cost effective access to subject experts. . ensuring that information is presented in a structured and coherent manner. *corresponding author. email: i.f.stewart@gcu.ac.uk alt-c 2011 conference proceedings 0218 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # i. stewart et al. this is an open access article distributed under 146 the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http:// creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7815 the goals of the project were to make better and more efficient use of lectures as one of a range of learning resources and to support distance and asynchronous learning. in particular the aim was to move beyond normal lecture capture into the production of an integrated learning resource, where the lecture content was just part of the overall package. while a number of commercial products have been developed (echo 360 etc), in general these products can be classified as being enterprise level systems and require a major investment on the part of the institution. they are designed to be easy to use but normally require a significant infrastructure commitment and are not optimised for portable use (they are designed for fully equipped lecture rooms). other products (e.g. camtasia) will capture everything that is entered on the screen of the presenter’s system. this can be a very powerful resource, capturing much of the content of the lecture and making it available in an easily downloaded format. they do however, miss out on the interactions of the lecturer as they miss the gestures and other forms of body language which are often used to reinforce meaning and content. in addition they are still focused on capturing the presentation. modern students and web 2.0 resources the digital environment within which modern students work has been characterised as being ‘‘continual partial attention’’, a concept developed by stone (1998). in this type of study environment the student is switching between a range of resources and information feeds. this model does not fit well with the focused approach that is required to get the best results from lectures. many students who use existing resources developed by the authors and discussed in the following sections, describe how they use the content as a ‘‘background activity’’ while they engage in other online content, switching to full attention when particularly challenging elements of the content are reached. the students are also used to short, conversational interactions with content. this is exemplified not only in systems such as twitter and facebook but also in the feedback sections of sites such as youtube. students are used to asynchronous interaction with content and browsing through online media to find relevant content to support them in their studies. the challenge for the product was how to develop it in such a way to provide the level of interactivity that students now expect in their general online interactions, while at the same time retaining the benefits of the existing system as a learning resource. underpinning activities � pilot activity 1 the authors have been working for a number of years in the area of technology supported learning. initially their work was targeted at providing support for deaf and hearing impaired students by the use of voice recognition systems to provide real time transcription of lectures. this led to a number of publications and the development of several prototype products. one of the interesting responses to the work was from the hearing students who also wished to access the transcript of the lecture as it provided a range of benefits to them. alt-c 2011 conference proceedings 147 . they found that, unlike captured audio or video, it was easy to scan through the content to identify the area that they wanted to review. in some cases the students used search tools to make this process more efficient. . the compact nature of the file meant that it could be stored and displayed on devices with relatively little memory such as smart phones. . the ability to print out the content and annotate the captured text was found to be extremely useful as a study tool. the work moved on to the development of a low cost system for capturing and presenting the lecture content in a number of formats. these were integrated into an application which allowed the student to browse through the lectures in their own time. the interface of this product is illustrated in figure 1 and is discussed in detail in mckee et al (2008). significant benefits were noted from this activity. a class of 50 students were used for the pilot project. they were surveyed by questionnaire after a series of 18 lectures, with 25 completed responses which were followed up by a series of random follow-up interviews. the key outcomes were: . the deaf students were highly appreciative of the subtitles, both in the presentation and in the recording. . although the subtitles were intended for the deaf, several international students said they helped them to understand the local accent. . no-one was concerned that the ‘‘production values’’ of the recording were not of ‘‘broadcast quality’’ � the content was clear. . more than half of the class filled in a questionnaire, but even those who had not filled it in said they had viewed and used the recording. . virtually every student wanted the mechanism employed by other presenters. figure 1. initial lecture capture playback tool. i. stewart et al. 148 underpinning activities � pilot activity 2 in parallel with the first activity, another set of lecture capture activities were undertaken which did not utilise voice recognition but were more concerned with making a wide variety of integrated learning resources available to support the learning styles of the students. the particular cohort that undertook the activity was working at honours level on a module with significant technical content, a large number of graphical examples and screen based demonstrations as well as a significant amount of supporting reading materials. the content was integrated into the blackboard vle for easy access and download. the students in this second pilot, who were utilising the captured lecture content to support their studies, were asked a range of questions in a questionnaire and also by an external facilitator. seventeen students from the class of 31 provided detailed answers and engaged in discussions with the facilitator. the questions are listed in table 1. the key points that were extracted from the responses were: . approximately 50% of the respondents felt that the capture process impacted on the lecture presentation but everyone wanted the activity to be continued. . the preference for video vs. audio synchronised to the slides was similarly evenly split across the respondents. in general those who expressed a preference for the video tended to use it to catch up on a missed lecture or to see the gestures, visual cues and other interactions which were not necessarily available from the audio content. the students who preferred the audio synchronised slides preferred it as a resource to review a particular element of the lecture. . when questioned about their learning style, many of the students did not recognise that they had a particular learning style. however, when they were asked to describe how they would study a new topic, the majority described a process where they would take a section of content and review and summarise the content until they were comfortable with the concept. this suggested an alignment with the reflective category as defined by honey and mumford (1992). further discussion showed that their approach was also strongly biased towards a logical sequential development of ideas and in addition in certain circumstances they would study in groups or apply other active learning activities. overall it appeared that rather than having a particular table 1. questions used to evaluate the second pilot. did you find the use of the technology impacted on the delivery of the lecture? did you use the captured lecture material? if so, did you have a preference for the captured video or the captured powerpoint presentation and why? how did you use the captured media? (e.g. for general revision, to catch a lecture i missed, to deal with a specific topic in a tutorial or to revist a topic i was struggling with). do you consider the capture activity to be worthwhile/useful? how could it be improved? (e.g. provide the media in more formats, split it into smaller sections to tie more directly into particular parts of the lecture, incorporate other media more tightly into the presentation) are you aware of your preferred learning style? how could the captured media be modified to better suit how you study? alt-c 2011 conference proceedings 149 learning style, many of the students were adapting their learning strategy to the content that was available and the nature of the subject being studied. this is consistent with the detailed discussion of the fragmented nature of much learning styles research in the work of coffield et al. (2004). this pragmatic approach to learning was likely to be in part due to the nature of the class itself which was in the final year of a set of technical programmes with a high level of applied content. few of the students had experience of (or interest in) psychology or educational theory. the students were generally very comfortable with using a wide range of digital devices to access the information in a variety of formats and when appropriate, to edit the content or change the format to allow access on devices other than those anticipated by staff when making them available. this level of comfort in working with digital media across a range of platforms tied in well with the concept of the digital native (prensky 2001). in addition, the preferred approach for many students was to take parts of the content and to access these pieces when convenient rather than setting aside large blocks of time to concentrate on a specific area of study. in the discussions, some students described how they would run the captured presentation in the background while surfing the internet, and then when a particular topic or point was reached that was of importance or that they were struggling with, they would focus their attention on the presentation. this type of example tends to support the relevance of stone’s continuous partial attention concept which was outlined earlier. overall conclusions from the underpinning activities the most important result from the analysis was that there was a general desire to be able to access the media in a greater variety of formats (e.g. more suited to mobile devices) and that the content should be broken down into smaller parts which would form part of a larger structure to allow more focused use of the resources and to allow them to be downloaded more quickly. in terms of monitoring the effectiveness of the tool at that stage, it is worth noting that the module feedback for the modules where it had been deployed had been very positive, with a large number of comments stating how much the students appreciated the resource. this had also been fed back in comments from both of the external examiners who dealt with the modules and programmes where the work was being deployed. they noted in their reports: recording of teaching material: students appreciate the availability of teaching and learning materials on blackboard and in particular the video recording of lectures that were very useful for revision on modules such as multimedia technology 2. honours students were particularly appreciative of the effort made by one of the staff to make available video recordings of the main sections of his lectures, in addition to a wide range of other learning support materials. students stated that their learning process was greatly enhanced by being able to revisit the lecture material. similarly the usage statistics from blackboard show that almost every student had accessed and downloaded the content. i. stewart et al. 150 an additional benefit from this has also been that students who have had to miss the modules due to serious illness have been able to study the content over the summer and successfully pass the resit at a first attempt, which would not normally be possible as they had not been able to attend the classes. development of the specification of the integrated learning resource while the previous activities and resources were found to be useful to themselves, it was always intended that the development process would be continued to try and produce a more effective learning tool. the issue with the product as it stood was that it was still (in the main) a passive resource where the student accessed the content of the lectures in a variety of formats and could review and revise from it. it was not possible to interact with the content. any form of question and answer activity, linking to other resources or requests for clarification of the content after the lecture had been captured, could not be done within the product but had to be done through other mechanisms. in our case this was mainly via the blackboard vle. despite these limitations the product was well received by the students and a significant amount of feedback was gathered to inform the development of the next generation of the product. one of the key points that emerged from the feedback was that the ‘‘one size fits all’’ approach to education is not suited to the range of learning styles that the students use in their learning (dimitrova et al. 2003). the next stage was to take the wide range of content and fit it into a more flexible and adaptable framework while still ensuring that the structure and organisation of the information is logical. at the heart of this development was the creation of small learning elements or chunks. these included sections of a captured lecture, individual tutorial questions, links to online resources or simulations, faqs or any other relevant educational item. key goals for this content were that it be integrated into the system in such a manner as to allow easy modification, adaptation and linking between the elements. these elements are to be mounted within a structure or ‘‘mesh’’, showing their relationships and allowing optimal learning sequences to be described. this allows the learner to choose their route through the content and the elements of the larger themes on which they wish to concentrate. the key features of the specification are: . the system should be easy for staff to use. . the system should not impact significantly on the ability of the lecturer to present the material in a manner that suits their lecture style. . the presented content should be available in a variety of formats suitable for a range of delivery platforms. . the content should be personalisable by the student . the content should be able to be added to either by staff or students. the ultimate aim is to allow the individual staff member or student to modify and adapt the content to suit their own learning style and to increase the student’s engagement with the content. this proposal ties in with the desire of students to work with web 2.0 technologies (andone et al. 2007) which are an increasing element of their online environment. it is important to recognise that by providing greater flexibility and user control over the structure of the learning materials and hence potentially over the depth of the alt-c 2011 conference proceedings 151 learning experience, there is potential that some students may choose learning strategies which are not effective in achieving the learning goals. in recognition of this, the product will always allow a ‘‘non-interactive route’’ which basically follows the standard linear structure of the normal lecture/tutorial activity and provides a standard baseline experience of the content. the next stage of the process was to design and develop a software platform to support the presentation and distribution of the content. technical design of the integrated learning resource the prototype required multiple learning elements of different data types to be interrelated, linking content topic, presentation slides, captured video, tutorial questions, practice exam questions and faqs. this would allow students the freedom of being introduced to a topic via different forms of multimedia. to allow these relationships to be constructed, an xml-based approach was chosen. this allowed a schema to be quickly defined that identified all related materials, how they should be displayed to the student and at what point in time these assets are required. synchronisation data between the captured video and presentation slides are also stored in this document, with each slide having start/ end timestamps. using an xml approach as a data source for the backend of the application allows the system to be hosted on low performance infrastructure, reducing operating costs as there is no intense processing of large data volumes which may be experienced when working with larger databases with complex relationships. most data processing and user interaction handling occurs client-side with the implementation of javascript. this allows the xml documents stored on the server to be converted into json (javascript object notation) strings and then parsed efficiently on the client’s machine. javascript is also responsible for synchronising the video and presentation slides, utilising the timestamp information stored in xml documents to keep content presented to the student up to date. the prototype in use a prototype learning framework has been developed. the intent of the framework is to allow the lecture content and its related resources to be accessed in a variety of ways. the software development methodology being applied during the development stage is that of prototyping. this allows early versions with reduced functionality to be deployed and the feedback from the users incorporated into the subsequent iterations of the design. the current version of the software consists of three elements. . a user view that the students can access. . an edit view that allows the lecturer to organise the core content. . a content management system to allow the uploading of the materials. the core of the prototype is still the captured lecture content as is shown in figure 2. this allows the students to navigate through the lecture content via the slides or the video which are synchronised together. where this product differs from previous developments is in the areas shown underneath the lecture presentation. firstly, they provide access to all of the related content in a single space. secondly, they provide an alternative mechanism to navigate the content. i. stewart et al. 152 the full navigation structure of the prototype is based around themes rather than around individual lectures. for example one theme is the generation of 3d graphics. this unit consists of a number of captured videos and slides (one set per lecture) merged into a single ‘‘presentation’’. as with traditional lecture capture, this content can be navigated linearly through the lectures. the other elements of the theme provide for other forms of navigation and interaction. . tutorial questions allow navigation of the content by relating sections of the presentation to the answers. figure 3 shows how selecting the question jumps the lecture to the appropriate point. . exam questions provide access to relevant past paper questions and solutions as well as links to the appropriate presentation elements. . student faqs allow regular questions from the class (either online or from tutorials) to be posted along with the response. . additional materials provides a range of resources to extend the interaction (e.g. for the 3d theme this consisted of links to downloadable demonstrations and interactive learning resources relevant to the topic). . further reading linked to a range of relevant electronic articles and websites. in almost all of these sections there is the ability for the students to add comments and feedback on the content and to discuss the particular topic on which they are currently working. figure 2. screenshot of the prototype showing the general interface. alt-c 2011 conference proceedings 153 the key area where this prototype can be considered to be innovative when compared to traditional lecture capture is that the lecture is just viewed as one of the learning resources available to the student in the environment. the environment provides a simple interface to the content of a particular theme of the module. within this the student should find all of the related content of the topic in a clear, linked and navigable structure and be able to use the resources in a way that suits their particular needs and learning style at that time. the initial feedback from users has been gathered in two ways. informal discussions with the users were undertaken in class time and by email to pick up feedback on the prototype and to allow the rapid prototyping methodology to be applied to the development. the final evaluation of the current version is still under way at the time of writing. it is being done using a fully anonymous web based questionnaire which has been sent to each of the 44 students who took the module. after 2 days of running the survey, 11 responses have been received. the survey uses a token based system to ensure that only one response per student is recorded. in general the respondents recognised that there were benefits to having all of the resources integrated into a single interface and they find the navigation of the content easier than in the previous systems. there were issues with needing to scroll up and down the screen to see the video when navigating via tutorials. the ability to approach the content from different directions (e.g. to be able to go from a difficult question to the relevant parts of the lecture and supporting content) made the figure 3. screenshot of the prototype showing the linking of questions to content and the addition of comments. i. stewart et al. 154 material much more useful to them. the discussion tool as it stands is seen as of some value depending on how it is used, but the students were not clear about how it was meant to be used. where it has proved most effective to date is when students have provided links to other resources that they have found useful and made these resources available to their colleagues. at present the prototype has been tested across a range of digital devices. the original content was designed to be accessed by students using networked pcs/macs and netbooks. the interface was designed to work best on those screen sizes. the choice of flash as a video format with the other content based on xml meant that the content was accessible from any standards compliant browser on these platforms. testing showed that pcs and apple systems running a range of browsers had no problems presenting the content. while handheld devices were not the focus of the original design, the interface was designed to support this form of access and the prototype can run successfully on many high end smartphones. figure 4 shows the application running on a samsung nexus s android reference smartphone. however it is recognised that the decision to use flash as a video playback standard has excluded ios devices from being able to access the full content. part of the future development work will be a range of ‘‘apps’’ optimised for mobile platforms, which will support offline access. discussion and further development the overall goals of the project were to: . make better and more efficient use of lectures as one of a range of learning resources; . support distance and asynchronous learning; . move beyond normal lecture capture into the production of an integrated learning resource, where the lecture content was just part of the overall package. figure 4. screenshot of the prototype running within a mobile browser (android platform). alt-c 2011 conference proceedings 155 the feedback from the students who used the resource is very positive with many comments supporting the work. the resources have been used by a number of students who missed classes through illness to keep up with the content. in particular, the integration of the resources into a single interface was appreciated as a means of allowing the students to approach the content in their own way rather than being constrained by the linear flow of traditional lecture capture. the framework is currently being extended to better support distance access provision by incorporating an interactive element which allows questions and issues to be raised after the lecture and then linked back into the lecture content. in particular it is intended to encourage greater student involvement with the content of the course. by allowing the students to take ‘‘ownership’’ of the content and to manipulate it in a manner that is appropriate to their learning style and mode of access, it is hoped that the engagement of the students with the content will be increased. those familiar with artificial intelligence studies will recognise that there is a valuable way of looking at such a complex environment and mapping routes to satisfactory outcomes for approaches with differing input parameters � that of the ‘‘weighted mesh’’. in this model the individual elements (or nodes) are produced and then linked to preceding/succeeding nodes by a set of pointers given appropriate ‘‘weights’’ according to the likelihood of producing the desired solution. for example, a user having utilised a node can be directed to one of the following nodes depending on their expressed desire for sequence, or depth, or by the results of a test, or by the ‘‘learning style’’ they have chosen (overview/short topics/detail/etc.) the current prototype is a first step in the way to such a system in that it allows the production of ‘‘nodes’’, in the form of learning objects, and a mechanism of implementing the links between them. at present this needs to be extended to provide greater flexibility and interactivity. once this is done the prototype can be taken further to begin to incorporate a mechanism to weight the paths and to develop more personalisable elements. while there is still work to be done to improve and extend the prototype, there is already evidence that it is encouraging the students to interact more with the material and to take a more active and engaged role in their studies. this has been recognised within the authors’ university where the work has been demonstrated at workshops, nominated by students for a teaching award and one of the authors has been awarded funding by the institution’s learning research centre to support the further development of the prototype to produce a tool which can be deployed more widely. particular interest has been shown by colleagues who are involved with distance and part time learning programmes which form an ever increasing part of the educational delivery in higher education. acknowledgements the authors would like to thank the past and present students who have supported the project and provided the feedback that has been essential to the development of the project. references andone, d., j. dron, l. pemberton, and c. boyne. 2007. the desires of digital students. 14th association for learning technology international conference, september 4�6, in nottingham, uk. isbn 978-0-9545870-6-2 i. stewart et al. 156 coffield, f., d. moseley, e. hall, and k. ecclestone. 2004. should we be using learning styles? what research has to say to practice. london: learning and skills research centre. isbn 1 85338 914 5 dimitrova, m., c. sadler, s. hatzipanagos, and a. murphy. 2003. addressing learner diversity by promoting flexibility in e-learning environments. 14th international workshop on database and expert systems applications (dexa 2003), september 1�5, in prague, czech republic. isbn: 0-7695-1993-8 honey, p., and a. mumford. 1992. manual of learning styles. 3rd ed. maidenhead: p honey. isbn-10: 0950844403 mckee, w.a, d.k. harrison, m. allan, 2008. evaluation of methods of volume-production of vodcasts of presentations. international journal of educational technology 3, no. 4: 85�89. issn 1863-0383 prensky, m. 2001. digital natives, digital immigrants. http://www.marcprensky.com/writing/ prensky%20-%20digital%20natives,%20digital%20immigrants%20-%20part1.pdf (accessed october 12, 2010). stone, l. 1998. thoughts on attention and specifically, continuous partial attention. http:// www.lindastone.net/ (accessed october 9, 2010). alt-c 2011 conference proceedings 157 http://www.marcprensky.com/writing/prensky%20-%20digital%20natives,%20digital%20immigrants%20-%20part1.pdf http://www.marcprensky.com/writing/prensky%20-%20digital%20natives,%20digital%20immigrants%20-%20part1.pdf http://www.lindastone.net/ http://www.lindastone.net/ http://www.lindastone.net/ http://www.lindastone.net/ teaching models and local-area networks tom jones* and brant parker** * department of educational psychology, university of calgary, canada ** rocky view school district, canada abstract the thesis of this paper is that new advances in both microtechnology and lan technology can now provide teachers with flexible and exciting instructional tools which allow for a powerful integration of teaching model, curriculum content and technology. the first section describes some of the current applications of school-based lans. the second section discusses various teaching models, and describes an in-depth example of how a teacher may go about providing instruction by combining a lan and these models. the third section addresses the feasibility of such an instructional approach. introduction the introduction and integration of school-based local-area network (lan) technology is increasing annually. teachers and administrators who support lans cite economics, more efficient and effective use of computer resources, easier management of software and overall ease-of-use as some of the major advantages (reinhold 1989). however (and unfortunately), the use of networked microcomputers in schools continues to reflect a 'one-station, one-user' approach, especially when viewed from the perspective of the integration of teaching models and technology. that is, a lan is for the most part considered to be a system of connected, single-cpus that have access to shared resources rather than a powerful platform that can attend to individual learning styles and abilities and preferred teaching models and accompanying strategies. recent advances in operating system functionality have opened the door to a potentially innovative and exciting melding of teaching models and network technology. current applications of school-based lans presently, most school-based lans facilitate what would normally be done on a stand-alone computer (vernot 1989; reinhold 1989). for example, after working through the procedure to gain access to the network (logging on), students proceed to carry out some task such as using an application program (word processor, database program), programming, using electronic mail or account housekeeping. 4 alt-j volume 2 number 1 although these lan-related activities in schools are primarily single-user in nature, some very innovative instruction has resulted. thrush and hardisty (1988) describe how lans can facilitate the writing process by providing the writer with access to centralized databases, common files and other users. with respect to the teaching of writing, thrush and hardisty provide many practical suggestions for teachers and students. these include having a specific file which contains ideas for writing topics. this file can be browsed or added to by any student. they also propose a system of allowing copies of student composition files* to be posted and to be read and electronically commented on by others. another suggestion involves the writing of 'shared' stories by having students write collaboratively in small groups on the same topic or in the same style (narrative, expository). electronic forums have also been successfully used to enhance user communication on a lan. these forums are topical message areas available to any user where students can read and respond to the comments of others. the advantage of this type of interaction is that each participant has the same access and limitations that other users have. this equality is not as easily attainable in other situations. the electronic environment removes any biases that might influence involvement in oral discussions and provides even the most reticent student with the same opportunity for participation. bump (1990) found that students who were able to participate in what he termed computer-assisted class discussion preferred this to face-toface interaction. furthermore, bump found that synchronous communication (users communicating online simultaneously) was preferred by regular lan users to asynchronous communication (for example, electronic mail). however, his research also indicates that there may be a tendency for students to write messages on the computer that are more direct and pointed than would be the norm in ordinary conversation, and that this can often result in a confrontational style. lans have also been used to provide for more specific and guided user activities. for example, one school in southern california uses a lan, authoring software and a library database to facilitate the writing of research reports and papers (dezell 1991). via a hypercard stack and an online library database, students are guided through the steps of formulating research questions, making an outline, locating library resources that relate to the topic, and determining whether or not these materials are currently available from the library. skor (1988) discusses a scenario in which a lan is used as an integral part of a grade 11 physics programme. information about class topics and daily assignments are posted fpr students. answering of preliminary lab questions is completed on the lan using a word processor which may take several days but which can be done from a variety of locations within the school. a spreadsheet program is used to collect the data and perform necessary calculations, and the word processor is used again to compose the lab report. additional information is available to the students through the use of an extensive physics database. the end product is submitted electronically to the instructor and the results are returned to the students in the same manner. having recognized the proliferation of lans in schools, network products are being marketed specifically for use in school settings in order to make the maintenance and operation of a • network less troublesome. one such product, ibm's iclas (ibm classroom lan administration system), provides a network interface which allows for five levels of users: generic users, students, teachers, a network administrator and a system operator. with iclas, a teacher can quickly and easily provide a student or a group of students access rights to certain applications or files and allow them to have read/write privileges to selected directories on the 5 tom jones and brant parker teaching models and local-area networks file server. thus a teacher can place students into groups or classes and create user menus which list applications that are available to each group. the teacher can optionally create a student directory on the server which will become the default location where files will be saved, or assign the workstation's local drive as the default. in addition, iclas will also permit teachers to control whether or not students are able to send messages and interact in real time with other students and/or teachers who are using the network, or to any other user via electronic mail. products like iclas go a long way to making a network an attractive teaching resource. in the near future, local-area and wide-area networking technology (lan, wan) will give students access to information that might be located in the same room (for example, common datafiles), in the same building (for example, network cd-rom) or even in other countries. because the communication infrastructure provided by networking technology rapidly handles the transfer of data, there exists great potential for interactivity between the classroom and the information source. with emerging standards in layered networking protocols (for example, the integrated services digital network (isdn)), teachers will be able toprovide students with access to information in a variety of formats: text, graphics, digitized and compressed video, and digitized sound (madron 1990). the information source may be a centralized database (for example, cd rom), shared datafiles, synchronous communication between users (electronic conferencing), asynchronous communication (electronic mail), or remote access to online information systems. indeed, future wide-area networks will allow for students to collect information on a topic by accessing simultaneously various information sources both local and global on the network. this could be made possible by means of a software 'gopher' which would use student-defined keywords, graphical cues, or other information discriminators to search systematically each source for the specified information. in this way, students could retrieve information from a number of external sources. however, even with these software and hardware advances, the power and functionality of lans have been primarily used to provide supplemental support for teachers and students. that is, the use of lans in educational settings has resembled more a business model than a model that reflects a consideration of teaching models and the concomitant strategies. although the advances in centralized and distributed computing have been remarkable, they have yet to be exploited in a manner that will allow teachers to teach some curricular content in a particular manner for a particular group of students that is, match a teaching model with the individual characteristics of the learner. traditional educational software (cai, drill and practice programs, simulations, educational games) have not been designed to accommodate different teaching styles for the most part, but rather to be used by the individual learner or by a teacher who can creatively work the software into the overall instructional program. this can be effective but it is rather like putting the cart before the horse. on a broader scale, many educators have concluded that computers do not necessarily provide the enhancements to learning that were promised in the early 1980s (skor 1988), although it should be pointed out that there has been some success (roblyer 1988). rather, a decade of experience has shown that the educational benefits of computers have resulted from the manner in which the technology has been applied by teachers and students (knapp 1989; collins 1991). educators have concluded that the more closely the application of the technology complements the teaching and learning styles, the greater the impact on learning (migliorelli 1987; tesler 1991). 6 alt-j volume 2 number 1 figure 1: integration of teaching models and technology teachinq models information processing model concept attainment -inductive thinking advance organizers -developmental personal models •synecbcs social models group investigation role playing sodal science inquiry behavioral models -mastery learning simulations -behavior modification curriculum content <technology uanj what is needed, then, is a different approach to the use of technology in schools, one which attends first to the teaching model that the teacher wishes to employ for a certain instructional unit or event, and one which also can be adapted to the different abilities and interests of the learner. interestingly, the latest advances in lan technology can now support the demands that would be placed on a school-based lan if the approach delineated above and fleshed-out below were to be implemented. teaching and learning that is supported by technology not adapted to technology is now possible. teaching models and lans if the previous attempts to integrate technology into the curriculum have been only partially successful, what approach should be taken and what role do local-area networks play? one answer is to start from the pedagogical viewpoint (that is, teaching and learning), and then to determine how a lan might provide a powerful resource in which teachers and students will be able to carry out activities which are not possible in a lan-less classroom. in their seminal book, joyce and weil (1986) discuss 20 approaches to teaching and learning and the underlying theories. these models describe approaches that range from didactic to constructivistic, from structured to non-structured, and from teacher-directed to student-directed. joyce and weil group these approaches into four broad categories or 'families of models': information processing models, personal models, social models, and behavioural systems models. a close look at each model within the four families reveals that not all readily lend themselves to a match with instructional technology (see joyce and weil for details). the models that do, however, are presented in figure 1, and a short description of each model's main thesis is given in appendix a. the point is that this innovative approach will address the bringing together of three factors: the teaching model, the curricular content and a local-area network. what teacher would not like to have the option of planning a unit on a topic (for example, trial balance) and have at his or her fingertips a means by which the instruction could be made to reflect an appropriate teaching model listed in figure 1? for instance, one teacher might prefer to have the students first achieve fluency (mastery learning) in the terms which describe the" report (assets, liabilities, equity, etc.), then move on to a presentation (advance organizers) of sample trial balances, and finally to run a simulation of a small business in which the goal is to come up with a trial balance (simulations). 7 tom jones and brant parker teaching models and local-area networks this could well be followed by some activities on the part of the students, and followed up with a brainstorming session on how to prepare a trial balance for easy comprehension (synectics). underlying these activities might be a desire on the part of the teacher to ensure that no one student is left out and that some remediation can be provided to a certain number of students who are not able to carry the necessary operations at a rate commensurate with the other students (behaviour modification). another teacher might wish to provide instruction on the trial balance by presenting the students with the results of a series of journal entries, and then leading them toward the concept (and its concrete counterpart) of a summary statement (inductive thinking and concept attainment). numerous examples could be generated along these lines, but the question would remain about the feasibility of being able to teach some content using one or a number of these models. are teachers able to do this now? the answer is perhaps: a little, but not as much as they would like, and it would definitely take a great deal of time and effort even if it were possible. one solution is to structure the curricular content and its subsequent delivery in a way that it lends itself (at least to a large extent) to being run on a classroom-based lan. inherent in this is the necessity to re-fashion the subject matter so that it can be moved to an electronic medium. this is no small concern, but it should also be noted that use of a lan as an integral if not central component in a course of studies will also open up the classroom to other sources of information (see above), and these other sources can provide unique opportunities for the teacher and students. at the core of an implementation along these lines is a recognition that the traditional manner in which a curriculum is developed must change. in particular, after decisions have been made about goals and learning objectives, a great deal of attention will have to be paid to how various topics will be treated vis-a-vis the teaching models. for example, if it is decided that the overall model will provide for a group investigation, then the options for dividing the first settlers on the prairies planners/problem-solvers artists/designers figure 2: group learning example 8 aur-j volume 2 number 1 students into groups must be pre-determined. also the nature of the tools that will be available to each group will also have to be commensurate with that group's contribution. in any event, the driver for these types of decisions when developing the curriculum will be the content itself. the ideal, of course, would be a curricular structure which would allow for a maximal crossing of the teaching models and the course topics. example to provide a classroom-based example of how these ideas might be implemented, imagine a typical (north american) grade 7 class (teacher and students) who are beginning a unit on the settling of the prairies. without going into detail about how this event took place, let us take a look at how a teacher who is teaching this unit and who has decided on a 'group investigation' model might take advantage of a lan. set-up the lan would initially be configured by assigning students to groups (figure 2). as part of the unit set-up, the teacher could allocate the stations in such a manner as to match the groups' tasks and tools with the strengths of each of the students. in this example, there might be provision for a maximum of six groups (planners/problem solvers, archivists/linguists, mathematicians, scouts, cooks and designers/engineers). the teacher may decide that all six groups should be used, and that the 30 or so students in his or her classroom can be appropriately placed. this calls for an in-depth knowledge of a . number of factors which impact on learning past achievement, level of mastery in required areas, group dynamics, motivation, preferences and a detailed profile of how the students fare on these factors. in other words, the membership of the groups (and note that not all six have to be assigned) is controlled directly by the teacher and should result in the most effective matching of required skill and student expertise. group tools once the make-up of the groups has been determined, each group would be given access to that group's tools that is, the programs they will be using (figure 3). the planners/problemsolvers would be supplied with information about the ultimate goal of the unit (for example, settling in some western locale), an overall inventory of the group's resources, a decisionmaking program and an updated database. the archivists/linguists would be responsible for maintaining an accurate record of the pioneers' day-to-day efforts, for interpreting the records of previous explorers and for the translation and interpretation of communications between the settlers and the natives. this group would have access to a text editor, a separate database or compendium of earlier explorers' records, and a visual database of native ideographic writings. the mathematicians, as might be expected, would carry out the necessary calculations required by any of the other groups. their tools would be a calculator and spreadsheet-like software which would provide for what-if? questions. the scouts would have at their disposal maps of varying degrees of accuracy and completeness. the nutritionists would be charged with the acquisition, preparation and preservation of food. the designers/engineers would be required to design (and to modify, if necessary) the means of transportation, and to design and build temporary structures (huts, bridges, rafts). 9 tom jones and brant parker teaching models and local-area networks inter-group communication a unique and very powerful feature of establishing groups as described above is that although each group will have its unique tools, responsibilities and activities, all the groups will have to interact in order to acquire pertinent information for their own activities and decisionmaking. for example, the planners/problem solvers will have to maintain close contact with the scouts when the settlers are about to break camp and enter unknown territory. the information they will need may be visual (a first-time map of a region), textual (written descriptions of up-coming terrain or obstacles) and perhaps auditory (sound of waterfalls/rapids). they will also require updates from each of the other groups and will either request them or acknowledge them when they are available. the mathematicians in this instance might be giving assistance to a number of the other groups. based on what information the scouts have provided to the planners/problem solvers, this group could be asked by the designers to work out the number of days it will take for the settlers to reach a particular destination. the cooks, after having decided on what food will be available, might request the mathematicians to determine how much fuel will be required in order to prepare the food. the archivists/ linguists might query the scouts about the natives who were encountered in order to determine which native language will be used for communication. note that the results of each group's decisions, calculations or designs would have an immediate effect on the central database (and updated on each group's screen), and would be accessible by each group as input for further group activities. note also that the information shared among groups may be textual, visual (still frame or full-motion video) and auditory. teaching models during all this activity, the teacher would have a great deal of control over what transpires. in particular, the teacher would be able to alter instructional events, based on the teaching models described above, with each of the groups so that the members might acquire the learning objectives of the curriculum while contributing to the unit goal that is, moving the settlers westward. thus, the teacher might be aware that the scouts are lacking the concept of a map legend and the skill of interpreting various markings. this group would then be presented with information on legends (concept attainment), given opportunities to demonstrate mastery of these concepts (mastery learning), and perhaps individual members of the group would be required to work a while longer until mastery is achieved (behaviour modification). for another group, the mathematicians, it may arise that they need certain formulae (for example, rate x distance) in order to compute distances to be travelled. the teacher in this instance might apply an inductive thinking model by presenting to the group a number of examples which call for predicting a final answer. the examples could include data which has to be generated by the group. what is being argued for here is the ability for a teacher to control how content is delivered (in other words, which model) to each group based on that group's learning characteristics. realistically, not all models will be covered for each instructional event; the onus will be on the curriculum designers to decide which sub-topics of the content lend themselves to which model, and which of these sub-topics are amenable to being developed for a lan. it should also be emphasized that if the type of system being described were to be implemented, it does not preclude the introduction of offline activities. in fact, designing a curriculum with no provision for such activities would restrict a teacher's options and would not be recommended. 10 alt-j volume 2 number 1 figure 3: group tools first settlers on the prairies | query/update inventory map | query/update | artists/designers planners/problem-solvers query/update editor \ archivists/linguists | query/update 8151 mathematicians master console (teacher) common data-base query/update | inventory recipes nutritionists query/update map scouts implementation is it possible to implement these ideas given the state of today's technology? it appears so. what is required is a peer-to-peer network which includes a single file-, printand communications-server (preferably with a fast cpu and, more importantly, fast disk access) and a number of clients (emphasis on the mastery learning model would call for more stations than emphasis on the group investigation model). the server, over and above providing for access to shared resources printers, cd-rom, modems, video sources , will also turn full duty as a database server given that all clients (workstations) will be reading and writing to a common database (figures 2 and 3) which will provide the teacher and all students with a 'latest' version of their progress. because individual clients will possibly need to multitask, a good choice for network software would be either novell's netware or 3com's lan manager running over os/2. none of these are suggestions that reflect low-end computing (and the matter of client software has not yet been addressed), but it will be obvious that what has been outlined in this paper calls for a major revamping of how instruction and technology are integrated. summary the object of this paper has been to consider an approach for the integration of (a) various commonly-held approaches to teaching (models) with (b) a sophisticated, computer-based resource as the vehicle for delivery. the rationale for such a plan has been to provide teachers and learners with an effective, powerful and flexible tool for both instruction and learning. can it be implemented? we believe so, given that lan technology has matured to the point where such a resource could be developed. does it put the horse (pedagogy) before the cart (technology)? we think so. 11 tom jones and brant parker teaching models and local-area networks appendix a: teaching models information-processing models the information processing family of models encompasses those which are primarily concerned with the acquisition of new knowledge, testing of this knowledge, collecting and organizing data, and/or developing an awareness and understanding of problems and arriving at solutions. the central tenets of these models are as follows: • concept attainment: an effective method for organizing information and for learning concepts; • inductive thinking: how to find and organize information and creating and testing hypotheses; • advance organizers: provides a cognitive structure for comprehending material; • developmental model: the goal is to match instruction with the learner's level of cognitive ability. personal models of teaching models grouped into the personal family are those which revolve around the individual and his or her experiences. the emphasis in these models is on the perspective and knowledge a person acquires through his or her own actions and interactions with others. unlike their information processing counterparts, the main goal of these models is not to improve individuals' cognitive skill which assists them in acquiring new information. rather, personal approaches aim at improving the individual learner's self-concept, confidence, effectiveness, while helping the individual construct his or her own understanding and knowledge. they challenge individuals to assume responsibility for their own learning and to extend the limits of their own understanding. synectics: teaches collaborative problem-solving to result in new, creative perspectives. social models of teaching the social family of models consists of those which are primarily based on group work and social interaction. that is, the models operate on the social processes found in group and collaborative learning environments. these co-operative settings, and the group dynamics that are prevalent within them, provide a means by which social growth is fostered and new knowledge, skills, and shared or negotiated understandings are acquired. group investigation: application of the principles of the democratic process to encourage group members to construct new knowledge; role playing: situations are presented to students who act out a problematic situation and then analyse it; social science inquiry: defining a problem, testing hypotheses, gathering data and making generalizations. • behavioural systems models of teaching the behavioural systems models are built on the premise that learning can occur through feedback received in response to one's actions. the approaches are largely founded on the work of classical conditioning theorists, and are based on a stimulus-response-reinforcement 12 alt-j volume 2 number 1 model. in these approaches, large tasks are broken down into smaller behaviours. the feedback provided to the learner is the primary means by which observable behaviours are either repeated or modified. mastery learning: individualized learning of highly stratified skills and concepts; simulations: principles of cybernetics are applied to learning environments; behaviour modification: learning occurs as a result of the manipulation of the environment. references bump, j. (1990), 'radical changes in class discussion using networked computers', computers and the humanities, 24, 1-2, 49-65. collins, a. (1991), 'the role of the computer technology in restructuring schools', phi delta kappan, 34, 9, 28-36. dezell, j. (1991), 'the impact of networking on education', curriculum product news, february, 2-3. joyce, b. and weil, m. (1986), models of teaching, london, prentice-hall. knapp, l. (1989), 'classroom networks: today and tomorrow', apple classroom of tomorrow, report 5, cupertino ca, apple computers. madron, t. (1990), local area networks: the next generation, new york, john wiley. migliorelli, f.a. (1987), 'microcomputers and teaching styles: creative applications and integration of commercial courseware', paper presented at the adult education and technology conference, pittsburgh pa. reinhold, f. (1989), 'educators explore the lay of the lan', electronic learning, 8, 5, 34-37. roblyer (1988), 'the effectiveness of microcomputers in education: a review of the research from 1980-1987', t.h.e. journal, september, 85-89. skor, r. (1988), 'the educational importance of local area networks in secondary schools', computers in education, 13, 2, 129-134. tesler, l. (1991), 'networked computing in the 1990s', scientific american, 265, 3, 86-93. thrush, e. and hardisty, d. (1989), 'computer networks for language learning: the creation of meaning through interaction', eric document 325668. vernot, d. (1989), 'get the whole story before you plug into a computer network', the executive educator, 11, 3, 21-23. 13 designing an interactive multimedia instructional environment: the civil war interactive ward m. cates, lynn a. fontana and charles s. white department of leadership, instruction and technology, lehigh university, 111 research drive, mountaintop campus, bethlehem pa, usa abstract this article describes the rationales behind the design decisions made in creating the civil war interactive, an interactive multimedia instructional product based on ken burns''s film series the civil war. introduction ken burns's film series the civil war was first aired on the public broadcasting system in the united states in the fall of 1990. its impact was immediate. from the first night, it was the subject of nationwide discussion and its audience grew progressively larger until, by the final evening, the total viewing audience was estimated to be 50 million. since then, the series has been rebroadcast several times, reaching still more viewers. in the century and a quarter since the end of the civil war, the united states has changed dramatically, moving through the industrial revolution, the american civil rights movement, the jet age, the space age, and now into the information age. despite these changes and the intervening years, the american civil war appears still deeply embedded in the american psyche. it was at once both a unifying and divisive event. in the words of shelby foote: 'before the civil war, the united states were, after the war, the united states was.' the country therefore gained a sense of unity and indivisibility as a result of the war, 'one nation, under god, indivisible, with liberty and justice for all' (pledge of allegiance to the flag of the united states of america). at the same time, a battle between two large regions of the country had been settled by force of arms, producing a 'winner' and a 'loser'. there are few who would question that the civil war had a powerful effect on both northern and southern philosophy, and on american politics in general. the american civil war is an event of great real and symbolic importance in american history, an event that students need to appreciate fully in order to understand many of the social, political, and economic issues at work in the united states today. 5 wardm. cates et al designing a multimedia instructional environment unfortunately, students display seriously limited knowledge about the civil war. in a national survey of history knowledge conducted in 1986 (ravitch and finn, 1987), 68% of american 17-year-old students could not place the war in the correct half century, and only one in four could place lincoln's presidency in the correct 20-year period. as ravitch and finn wrote: not even the strongest history students have a secure purchase on the chronological setting of the civil war ... yet we would contend that it is impossible to understand american history at all if one lacks any idea of when the civil war occurred. it is not only the single most traumatic and decisive domestic event since the thirteen colonies won their independence from britain; it is also the anchoring event of the nineteenth century, the climatic conflict to which other major events led and from which many others resulted, (pp. 49-50) the results of the 1988 assessment were no more reassuring (naep, 1990c). that assessment showed little change from the 1986 findings. further, it showed that almost half of the students surveyed lacked an understanding of key 'historical terms, texts, and relationships' (p.7). students demonstrated a tenuous grasp of the inter-relationships among events prior to and subsequent to the civil war. in short, students did not appear to understand the significance of the war, nor to comprehend the diversity of both its causes and results. the report of the national commission on social studies in the schools (1989) suggested principles for teaching aimed at producing greater understanding in american students. among their recommendations were the following: content knowledge from the social studies should not be treated merely as received knowledge to be accepted and memorized, but as the means through which open and vital questions may be explored and confronted, (p.3) students must be made aware that just as contemporary events have been shaped by actions taken by people in the past, they themselves have the capacity to shape the future, (p.3) learning materials must incorporate a rich mix of written matter, including original sources, literature and expository writing; a variety of audiovisual materials including films, television, and interactive media; a collection of items of material culture including artefacts, photographs, census records and historical maps; and computer programs for writing and analyzing social, economic, and geographic data. social studies coursework should teach students to evaluate the reliability of all such sources of information and to be aware of the ways in which various media select, shape, and constrain information, (p.4) working with funds from the corporation for public broadcasting, the civil war interactive project worked with ken burns to design an interactive multimedia instructional product based on burns's film series. after consulting with our advisory board of noted historians and educators, we established four goals for the project. our principal goal was to help learners recognize that the civil war, like all events in history, was the product of numerous social, political, and economic forces that inexorably influenced the lives of individuals. we hoped to help students learn to recognize the human factor in the equation of history, to understand the importance of inter-relationship among events, 6 alt-j volume 1 number 2 individuals, groups, ideas, and locations, and to see how a single pivotal event can influence the flow of history from that date onward. the second goal was to help students comprehend why there is so much disagreement about historical events in general, and about the civil war in particular, even among extremely well-informed people. in addressing this goal, we hoped to teach students to identify how point-of-view shapes perception and interpretation, and to develop in them the ability to recognize examples of bias, limited competence to judge, fallacious reasoning, and similar critical issues in evaluating primary and secondary sources. we felt that students needed to understand the important role that regional and personal interest play in interpretation, and to learn to spot such interests in materials to which they are exposed. we also hoped to help students come to appreciate how complex the process of historical interpretation is, and to learn to use the tools of historians and other logical thinkers to assist in structuring their own arguments. our third goal was to help students and teachers learn to work in hypermedia/multimedia environments, thereby providing them with information-handling skills that they may need in future as they are exposed to more and more sources and types of information. in this way, students and teachers could learn to cope with the demands of new technology and to make the most of the power of that technology. we wanted to equip them with strategies for managing information, to help them learn to distinguish relevant from irrelevant information, and to use that criterion to select appropriate supporting materials from among those available. the fourth goal for the project was to use the emotional power of film and video to draw students into the inquiry process. we hoped that this would, in turn, lead students to explore primary source materials and develop their inquiry skills. in keeping with this goal, we included in the product additional primary source materials that do not appear in the film series. we also worked carefully to incorporate both implicit and explicit instruction in the application of the inquiry process to historical investigation. early in the project we decided that our initial designs and early prototype would utilize level 3 multimedia. this meant storing the video segments from the series on a two-sided 12-inch videodisc, along with selected additional graphics, and using a computer program as the control for access to this disc. in addition, the computer program would supply extra text and graphics, and would provide both instructional organization and support for learners. materials stored on the videodisc would be displayed on a separate large classroom monitor connected to the videodisc player while the computer control program, additional text materials, graphs, and animations would be displayed on the computer's monitor. as we examined the products currently on the market and considered exactly how we wanted our product to operate, we became aware that some educational multimedia products appeared to confuse a visual database with an instructional environment. products whose principal use is as a visual database or reference work need, primarily, to have a well-organized and logical database structure and a powerful user interface that allows users to manipulate the database in as many ways as might prove useful. the usual assumption is that 'more powerful' users will manipulate the database in more powerful and incisive ways and that 'less powerful' users will utilize the materials in the database in less powerful and more superficial ways. novice users, almost by definition, will likely be less powerful users. implicit in the design of most reference works is a belief that users bring understanding to their searches and that this 7 wardm. cates et al designing a multimedia instructional environment understanding then enables them to take advantage of powerful features of the program to extract the desired materials. in contrast, in an instructional environment, we expect users to be novices. learners traditionally are expected to develop understandings of the material under study as they study it, not to bring such understandings to the setting as prerequisites. an instructional environment is expected to be supportive, to help novice learners make transitions from non-understanding or naivet6 to increasingly sophisticated understandings and conclusions. classroom teachers often help learners accomplish these transitions through careful structuring of the materials to be presented, through socratic questioning, and through carefully guided exploration and discovery followed by debriefing sessions in which learners are given a chance to draw conclusions about what they have learned. activity selection screen c^j select an activity by clicking: , ) ferrc11 inquiry bureau of invest!gotion laun the steps of inquiry f n j ymvf x n u l h d i productions. four introductory toun oflhtgvflwir. access to topics* indmognphiec. figure 1: main activity selection screen from our discussions with teachers and our own observations, we were aware that students and teachers appeared to use present multimedia products in three main ways: (1) as source materials for student reports; (2) as packaged presentations of content, not unlike regular films or videotapes; and (3) as source materials for individual student or teacher presentations. present uses did not appear to emphasize use of multimedia materials as a primary form of instruction, however. this fact may be attributable in large part to the earlier discussed failure of many products to provide any actual instruction, viewing themselves, instead, as reference resources. the civil war interactive^ instructional environment was designed to address the three most frequent uses through its three main strands: inquiry bureau of investigation, guided tours, and custom expeditions. a major distinction, however, was the product's explicit emphasis on providing direct instruction and instructional support in all three strands. 8 alt-j volume 1 number 2 inquiry bureau of investigation the inquiry bureau of investigation (or ibi) consists of two parts: the bureau and the production console. the bureau is actually a bureau of drawers, each drawer representing one of the five steps in the inquiry process. inside each drawer, users find tools and guidance appropriate to that step. users wishing to proceed systematically step by step through the inquiry process, or learners who are new to the inquiry process, can use the progression through the drawers to help them follow the inquiry process. users can go directly from an inquiry-process-step drawer to the production console to explore the multimedia database or to locate a particular sight they wish to find and can then go back to the bureau drawer they left. in this way, it is possible for a learner to be supported in the inquiry process while still locating the materials he or she needs to create a multimedia presentation or report. presentations by noted historians and experienced historical investigators can supply learners with role models to help them learn how to approach an investigation, and presentations addressing the steps in the inquiry process are also available in the bureau's drawers. inquiry bureau of investigation dick on dr. know for advice on using the bureau of investigation or on the user's guide to learn how to use the production console. | « — qu h uideo eoitda o o fiudio editor . . i 1 i i noden figure 2: inquiry bureau of investigation more experienced users can go directly to the production console where they may explore the multimedia database freely. they can locate any material in the database using a series of powerful search strategies. once the desired material is located, they may examine it and decide if it is what they wish to use. if they decide to use it, they can then include it in the presentation or report they are producing. at any point in this production process, users can jump over to the bureau to get some guidance on how to use the inquiry process to help in their efforts. the production console is designed to enable users to edit video and audio segments, manipulate the database, and import and export digitized versions of materials for use in their multimedia presentations. these latter capabilities should enable users to bring into the system 9 wardm. cates et al designing a multimedia instructional environment such things as files of text, digitized audio, and digitized video, as well as allowing them to get printed copies of documents or graphics in the database or to export their own files for use in an external word processor or graphics program. in addition, the production console is designed to allow users to connect with appropriate online information services as such become available. users may also use the production console to manipulate the materials they collect in their scrapbooks (see below) as they complete guided tours. these materials may then be converted to more formal presentations or reports using the powerful editing capabilities of the production console. guided tours guided tours focus on presenting learners with carefully structured presentations of a limited scope of the materials available in the multimedia database. each tour focuses on a specific theme and helps to illustrate that theme through presenting the learner with information drawn from the database. using the tourist-through-time metaphor, we created tours made up of stops, with each stop containing a limited number of sights. while we intentionally used the s-i-g-h-t spelling, emphasizing the sightseeing aspect of exploration, we also hoped that learners, as they viewed primary documents and saw images of various locales of import during the civil war, might unconsciously think also of sight's homophonic cousins, cite and site. in order to help students gain an understanding of the inquiry process, we organized each tour around a thoughtful question. thoughtful questions help learners to identify why they are about to see what they are. each thoughtful question is further illuminated by two enabling questions. enabling questions help to focus the learner's attention on questions whose answers may enable learners to find the answer to a tour's thoughtful question. each tour is also designed to help students learn the inquiry process and requires the learner to choose between two available hypotheses. once a learner selects a hypothesis, all later advice and guidance relates to the testing of that specific hypothesis. guided tours model the inquiry process, thereby teaching the process implicitly. we were particularly sensitive to the need to support learners. we were aware that they appeared as a group to lack content knowledge and understanding (naep, 1990a,b,c), thinking skills (beyer, 1988, 1991; costa, 1985), and an understanding of the persuasive nature of film (cates, 1990). we felt that the civil war interactive needed to address all three of these inadequacies in its instructional environment. we therefore included an advisement feature, dr. know. dr. know is available from all tour screens and offers three types of advice, which we labelled as es, ts, and qs. es consist of advice on how to explore the sights available at a stop. they focus on addressing varied learner intentions, and take a more global view of the content and purpose of sights at the stop. ts consist of guidance and advice related to thinking skills. they focus on helping learners understand how individual sights at a stop, or a stop as a whole, relate to a tour's questions and to the hypothesis that the learner selected. 10 alt-j volume 1 number 2 thoughtful question: how do people cope when their lives are interrupted by war? war is a form of national crisis which transforms institutions and the lives of people. it changes the social, political, economic, and cultural life of a nation. this tour explores the civil war as a national crisis and investigates how people coped with the changes that it produced. help •wet figure 3: sample guided tour thoughtful question i enabling .1 how do individuals in different walks of life i""15™"1'! cope with the effects of war? in any nation, the course of an individual's life is lied to his or her station i n life — rich or poor, vorker or employer, slave or slave-ovner, man or voman, leader or rollover. to understand h o v people coped v i t h the civil war, v e must explore h o v individuals v i t h different roles in society and in different valks of life coped v i t h the upheaval caused by the var. enabling/ question what aspects of daily life are altered by war? h o v people coped v i t h the civil war depended on how their lives v e r e interrupted. we v i l l get a clearer understanding of the interruption and disruption of lives by exploring h o v people's daily lives v e r e altered by the civil war. n figure 4: sample guided tour enabling questions qs are questions learners might consider while examining the sights at a stop. these questions attempt to do what a talented teacher guiding a learner in discovery might do: focus student attention on critical aspects of the material under study, point out inconsistencies or dissonant contentions, or pose questions that stimulate deeper thought about key issues. to help learners recognize the role viewpoint plays in historical interpretation, dr. know often plays up differences in viewpoint and opinion. to encourage students to think about and interact with the sights they see, they are encouraged to make entries in an electronic journal 11 ward m. cates et al designing a multimedia instructional environment 1 >g rrewarlife 1 in the south agriculture & the plantation system t dr. know i s available t o help you. please click on the type of advice you vould like. i tj i i vould like advice on hov to explore the i *-• i lights at this stop. pjm i vould like advice on the thinking process. i q i i vould like to see some questions to consider. click here to leave dr. know.... smou^*q;s;i exit t; figure 5: dr know's e?t-q advice selection screen exploration advice: each of the sights provides information that helps you understand the conditions mat led the nation to civil war. many of these conditions are linked to economic circumstances, a fact which explains much of what you will see in agricuhun i «w plantation sytten, industrialization, a n d shipping * commtrc*. political circumstances and competing political beliefs helped bring the nation into comflict and those issues can be explored in gonnanant. the plantation system in the south and its growing dependency on slavery was another source of disagreement mat inflamed passions on both sides of the evential conflict you may wish to start with asricidtura * flu plantation syttm since mis may help to give you a feel for slavery's place in southern society. ^r click here to return to the dr. know e-t-q screen. figure 6: sample exploration advice as they go along. in fact, dr. know often recommends that they make notations in their journals at particular points in a tour. hie electronic journal is part of a larger scrapbook function. in keeping with the tourist and sightseeing metaphor, students may collect souvenirs at stops. souvenirs may include video or audio clips, copies of documents, graphic illustrations or animation, or any sight the learner feels is worth saving. all go in the learner's scrapbook. students can collect materials to take to the inquiry bureau of investigation where 12 alt-j volume 1 number 2 qr.k pi e b i 1 thinhnp process advice" you'll meet a wide variety of people as you view the tights here. since your hypothesis focuses on differences among individuals, you should find a good deal of useful information about people who lived under starkly different social, political, and economic circumstances. understanding the kinds of lives led by individuals prior to the war will help you to recognize die effects of war when (and if) war comes in your lifetime. use journal to record descriptions of everyday life associated with southern agriculture and plantations. be sure to note differences between what you expect and what you find. " g ^ click hers to return to the dr. know e-t-0 screen. figure 7: sample thinking process advice questions to consider * why did slavery flourish in the south? * who owned slaves? * how were slaves treated? * how did someone become a plantation owner? * why were cotton and tobacco grown in the south? * how did life on a plantation differ from life on a small farm? * what was the "way of life" that influential southerners wanted to preserve? what other questions would you ask here? record those in your journal along with your observations as you view sights. f click here to return to the d r . know e-t-q screen. figure 8: examples of questions offered by dr know they can use the production console to convert any or all of the contents of their scrapbook into formal presentations or reports. after students have gone to all stops on a tour, they are asked to draw a conclusion about the hypothesis selected earlier. dr. know is not available to provide advice at this point, except in terms of explaining what a conclusion is and how one draws a conclusion. once the student enters a conclusion, dr. know becomes available to provide information on what sorts of conclusions might be drawn from the sights seen. 13 ward m. cates et al designing a multimedia instructional environment custom expeditions this portion of the civil war interactive allows users to create their own tours to share with others. as was the case in the inquiry bureau of investigation, learners also have access to the search capabilities of the production console. in custom expeditions, however, users are able to impose formal tour structure on the materials they select by identifying and grouping sights into stops and then grouping stops into tours. users can also add thoughtful and enabling questions and other properties of guided tours to the custom expeditions they create. in this way, students or teachers may utilize the materials in the database to create their own highly structured and supported instructional presentations. we felt that the ability to build such customized tours using the structure of guided tours (including implicit modelling of the inquiry process) could be of great value in helping teachers make the most of the product, particularly in terms of matching it to local school curricula. overall design concerns many interactive multimedia products appear almost too powerful and complex for their intended users. in this case, our intended audience of users consists primarily of 8th and 11th grade us students taking american history courses. a well-documented failing of many multimedia programs is cognitive overload (oren, 1990; phye and andre, 1988). cognitive overload occurs when the user is overwhelmed by the number or complexity of the options available, and loses a sense of direction and purpose. often designers refer to this as getting lost in hyperspace. the civil war interactive attempts to prevent cognitive overload in a number of ways. it uses no more than eight icons in consistent locations along a control strip at the bottom of the screen, thereby honouring miller's (1956) guideline on working memory limitations. each icon is coupled with a descriptive oneor two-word label to help make its function clear. screens always contain instructions to the user (such as 'click on any of the items below to ..." ) and online help is always available from any screen. to prevent users from losing a sense of where they are in the program, the left-hand portion of guided tour screens is dedicated to a navigation column, which displays in descending sequence the progression of steps that led the user to where he or she is in the program. pictures in the navigation column illustrate the screens that led to the present screen. recognizing how easily users can become disoriented in navigating, the product never goes more than four levels down. clicking on any of the four screen-pictures that may appear in the navigation column returns users to that screen immediately. another design concern was the level and nature of learner engagement. we believed that user choices should be meaningful and that users should have some sense of control. to encourage meaningful interaction and a sense of personal control, we included choices at all levels of the program and emphasized the personal nature of inquiry, particularly in terms of selecting tours, choosing a hypothesis, and collecting souvenirs and writing notes in the joumal/scrapbook. recognizing that learner control also means the ability to quit and restart without penalty, we created the map pin feature. the map pin enables users who exit before completing an activity to rejoin the activity at precisely that point when they next restart the program. to encourage learners to assume a sense of responsibility, we employed the language of self-attribution: 'my choice of hypothesis is ...' and 'i would like ...' (keller and suzuki, 1988). 14 alt-j volume 1 number 2 n prewar life | 1 in the south click on any photograph to learn more: agriculture & the plantation i system figure 9: screen illustrating the navigation column the civil war interactive also emphasizes a sense of control by providing users with devices to help them estimate the scope and length of time required to complete activities. users can examine brief descriptions of tours (tourbooks) before selecting them; stops often utilize descriptive markers that summarize what the stop covers and time clocks that indicate how long a stop should require to complete if all sights are viewed; and users can skip sights or entire stops as they choose. we observed that many computer-based educational materials were used by single students in isolation. this often appeared to be the case when they worked with educational multimedia products, particularly those that appear to be designed to be used primarily as references. we hope to encourage a more collaborative and cooperative approach to learning using the civil war interactive, an approach in which learners and teachers work together to find the answers to meaningful questions and to see how history, in general, and the civil war in particular, relate to the present and the future. accompanying print materials are designed to illustrate a wide range of ways in which to use the product, including one student working alone, small groups of students working independently, one teacher working with a small group, one teacher working with an entire class, and several subject-matter teachers working together with their students. the future the civil war interactive project is an on-going one. in december 1990 we completed a 200-page design document specifying how the product was to function. in the first half of 1992 we created a small working prototype of the product to test out its operation and refine its design. over the next year we demonstrated the prototype, observing reactions and gathering suggestions for changes. the third stage of the project will be the actual development of the commercial version of the product, complete with field tests in a variety of instructional settings and configurations. 15 ward m. cates et al designing a multimedia instructional environment references beyer, b. (1988), developing a thinking skills program, boston, allyn & bacon. beyer, b. (1991), 'improving student thinking while learning about the civil war' in the civil war hypermedia project design document, fairfax va, center for interactive educational technology, george mason university, the corporation for public broadcasting, and florentine films, 145-154. cates, w. m. (1990). 'helping students learn to think critically: detecting and analyzing bias in films', the social studies, 81, 15-18. costa, a. (ed) (1985), developing minds: a resource book for teaching thinking, washington dc, association for supervision and curriculum development. keller, j. and suzuki, k. (1988), 'use of the arcs motivation model in courseware design' in jonassen, d. (ed), instructional designs for microcomputer courseware, hillsdale nj, lawrence erlbaum, 401-434. miller, g. (1956), 'the magical number seven, plus or minus two: some limits on our capacity for processing information', the psychological review, 63, 81-97. national assessment of educational progress (naep) (1990a), the civics report, princeton nj, educational testing service. national assessment of educational progress (naep) (1990b), the geography learning of high school seniors, princeton nj, educational testing service. national assessment of educational progress (naep) (1990c), the history report card, princeton nj, educational testing service. national commission on social studies in the schools (1989), charting a course: social studies for the 21st century, washington dc. oren, t. (1990), 'cognitive load in hypermedia: designing for the exploratory learner', in ambron, s. and hooper, k. (eds), learning with interactive multimedia: developing and using multimedia tools in education, redmond wa, microsoft, 126-136. phye, g. and andre, t. (1986), 'cognition, learning, and education' in andre, t. and phye, g. (eds), cognition and classroom learning, san diego ca, academic press, 1-19. ravitch, d. and finn, c. e. jr. (1987), what do our 17-year-olds know? a report on the first national assessment of history and literature, new york, harper and row. 16 rlt1786 1..24 research in learning technology vol. 25, 2017 original research article supporting law students’ skills development online – a strategy to improve skills and reduce student stress? anne hewitt and matthew stubbs*† adelaide law school, the university of adelaide, adelaide, australia (received 23 november 2015; final version received 28 february 2017) law students internationally suffer from a high level of psychological distress compared with the general and student populations, and anecdotal evidence suggests that students developing skills without adequate support experience significant stress and anxiety. this article considers an initiative at one australian law school to develop a degree-wide structured online skills development programme as a means to both improve student skills acquisition and reduce student stress. the project implements, through the use of learning technology, the principles proposed by mckinney for making small changes to law school teaching, informed by self-efficacy theory, which can have powerful results. keywords: legal skills; online learning; student stress; wellbeing; skills; blended learning introduction law schools cater to a variety of stakeholders, some of which may have conflicting interests. in 2003, the johnstone report identified ‘competing demands … from the university, from students, from employers and law societies, from admission boards, not all of whom share the same vision for legal education’ (johnstone and vignaendra 2003, p. 468). law students rightly demand a high-quality education with good job prospects. admitting authorities and the legal profession require legal graduates to have a variety of substantive knowledge and skills (handsley, davis, and israel 2005, pp. 112–113; kift, israel, and field 2010). the community requires that graduates be ethical and business competent; universities require that students demonstrate certain ‘graduate attributes’. perhaps this myriad of pressures contributes to the fact that law students australia-wide suffer from a high level of psychological distress compared with the general and student populations (tani and vines 2009). this article considers a curriculum-wide project at one australian law school that attempted to balance some of the stakeholder interests in producing skilled law graduates with the school’s pastoral obligations towards students through a supported skills development framework. we examined the ability of learning technology to enable us to fulfil our obligation to provide an environment that will maximise learning while also providing the support necessary to minimise student distress. we then created a series of online skills development modules which were deployed throughout †the authors wish to thank vanessa white for research assistance and shane rogers for outstanding assistance with the survey and statistics. the project which informs this article was supported by a 2013 university of adelaide e-learning grant. *corresponding author. email: matthew.stubbs@adelaide.edu.au. 1research in learning technology 2017. @ 2017 a. hewitt and m. stubbs. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 (page number not for citation purpose) mailto:matthew.stubbs@adelaide.edu.au http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v25.1786 the law curriculum, and evaluated them to determine both their efficacy in assisting students to acquire skills, and also their contribution to establishing a supportive learning environment in which student stress was minimised. in part 1, this article will more fully explain the requirements for law schools to actively engage with developing students’ skills and the efficacy of online skills development pedagogies. in part 2, the issue of law students’ mental health will be briefly explored. part 3 will consider incidental issues associated with online and blended learning environments for students. part 4 will introduce the project and part 5 will summarise the evaluation methodology and results. law schools, skills development and the efficiency of online delivery it is now accepted that teaching some practical skills to students is necessary in order to equip them for professional life, in whatever field that may be (australian law reform commission 1999, p. 139, 151; edwards 1992; lord chancellor’s advisory committee on legal education and conduct 1996, p. 14; sullivan et al. 2007, p. 13). expecting students to acquire skills automatically is an ineffective strategy (woods et al. 2000), and it can lead to disengagement (deci and ryan 1985). for many students, it is also likely to negatively affect their perception of their own competence and control, which can contribute to emotional stress and have negative effects on self-esteem (mckinney 2002). however, developing practical skills often requires a very different method of teaching than is effective for substantive legal knowledge (hewitt and toole 2013). teaching skills can be time and resource intensive, which is problematic in a resource-poor environment (national alternative dispute resolution advisory council 2012, p. 30, 33). thus, developing cost-efficient and pedagogically effective methods for assisting students to acquire skills is important. in this context, online learning tools may be useful. while such resources are often time or cost-intensive to create, online learning tools can often be used multiple times with limited additional commitment of resources (especially in areas which remain relatively static, as some skills do), which may offer long-term institutional cost savings (butler 2010; kendrick 2010, p. 1392). online learning is not only potentially efficient; it can also be effective. a metaanalysis of research literature on online learning from 1996 to 2008 concluded that ‘students in online learning conditions performed modestly better than those receiving face-to-face instruction’ (u.s. department of education, office of planning, evaluation, and policy development 2010, p. 9). there is also some evidence that online learning environments can be used to facilitate the development of specific legal skills (hewitt 2015; martin 2003; yule, mcnamara, and thomas 2010). law students’ distress the well-being of law students in australia and elsewhere is a matter of concern. a 2009 australian study of students from 13 tertiary institutions found that law students suffer disproportionately high levels of psychological distress (kelk et al. 2009). similar results have also been found in institution-based studies (leahy et al. 2010; tani and vines 2009). indeed, studies around the world have demonstrated that law students suffer high levels of psychological distress (dammeyer and nunez 1999; a. hewitt and m. stubbs 2 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 helmers et al. 1997). this is also consistent with evidence of high levels of psychological distress and risk of depression in the australian legal profession (kelk et al. 2009). these results remain disturbing, even though they are now well known. a variety of reasons have been posited for the high rates of psychological distress suffered by law students. in united states law schools, blame has been attributed to factors as varied as the emphasis on achievement within law schools, and student competition for grades (nordmo and samara 2009); the vague process of ‘learning to think like a lawyer’ which includes a focus on linear thinking at the expense of student creativity and personal values (culp 1994, p. 69; peterson and peterson 2009) and legal formalism (peterson and peterson 2009; stefancic and delgado 2005, p. 35). the workload associated with the study of law, limited staff–student interaction and inadequate development of students’ interpersonal skills have also been identified as causative factors (benjamin et al. 1986). while the australian context differs from the united states (where law is a purely postgraduate discipline), many of these possible causes ring true here. in australia, suggested factors resulting in poor mental health amongst law students include heavy workloads, the highly competitive nature of law and law students, inadequate feedback and feelings of a lack of social connectedness, competence and autonomy (huggins 2012). all these potential causes support the suggestion that law school is a ‘breeding ground for depression, anxiety, and other stress related illness’ (mckinney 2002, p. 229). however, identifying solutions to the problem is more difficult. against that background, this project is predicated on the basis that the legal academy has a responsibility to create positive learning experiences for our students, and that a project evaluating the potential for harnessing learning technology to help address this serious issue is both timely and important. in this context, we have been influenced by watson and field (2011, p. 396), who argue for the adoption of a ‘range of innovative curricular and co-curricular strategies to support the use of legal education as a tool for the promotion of resilience and wellbeing’ and promote student engagement. in developing our project, we have also adopted the principles proposed by mckinney (2002), for making small changes to law school teaching informed by self-efficacy theory, which we hope will have powerful results. our online learning modules addressing legal skills acquisition respond to this literature on law student distress, as explained in part 4. online learning environments: opportunities and risks in addition to their potential efficiency, online learning environments can be pedagogically effective to facilitate substantive student learning. however, there are a number of issues associated with the use of technology in learning that should be considered when students’ well-being is also a concern. for example, the benefits of online learning resources such as flexibility and ‘just-in-time’ access must be weighed against issues of equity and accessibility. this part identifies the opportunities and risks of online learning environments which will be analysed in the evaluation of our online learning modules in part 5. flexibility online teaching and learning offer the potential for levels of flexibility and accessibility that cannot be matched by traditional face-to-face learning. it has been observed that research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 3 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 ‘students like to use e-learning if it … allows them to learn any time anywhere in their own way’ (selim 2007, p. 399). online teaching allows students a greater level of control: for example, compared with the standard face-to-face lecture, online teaching allows students to start, stop, go back or advance, as they wish (ireland 2008, p. 154) rather than being progressed through the course at the same pace, regardless of their understanding (glance, forsey and riley 2003). thus, online teaching assists both highand low-performing students by allowing them to work at their own pace and helping to alleviate the risk of students either becoming bored or left behind in a classroom geared towards ‘average’ students. for example, a 2008 australian study of online teaching in an intellectual property law course found that podcasts were particularly popular with students, and noted that ‘inclusion of podcasts … clearly introduced a whole new level of flexibility’ (ireland 2008, p. 154). student-driven learning online learning can also provide students with a safe, private learning environment in which they can experiment with new skills (martin 2003, p. 92), and work through the material at their own pace (mcnamara 2000, p. 167). because online learning is flexible, it provides an opportunity for students to engage in ‘mastery learning’ (guskey 1997) by breaking material into small stages, with assessment, before moving from one stage to the next, to ensure that students master each concept before progressing. traditionally, mastery learning is a teacher-paced and classroom-based system, with the primary source of instruction being the teacher. however, another form of mastery learning developed by keller (1968), called personal system of instruction, is a student-paced system, where the primary source of instruction is learning materials, so students can take course units at any time that suits them (grant and spencer 2003; keller 1968, p. 79). online modules can usefully contribute to this pedagogy. on-demand revision of previously learned concepts online content has the obvious advantage that it is available for on-demand revision. some research suggests that repeated studying after learning produces no significant effect on ability to recall material, although repeated testing results in a large positive effect (karpick and roedinger 2008). however, student feedback to the authors suggests that on-demand revision is of significant benefit to learning outcomes, and an australian study confirms high usage rates of content recorded for later revision (mascher and skead 2011, p. 421). there is also some evidence that on-demand revision is beneficial to learning outcomes (ireland 2008, p. 154; mascher and skead 2011, pp. 429–430; mckinney, dyck and luber 2009). equity equity issues exist in all teaching modes, although the online environment may tend to negate some issues, while amplifying or introducing others (coldwell et al. 2007, p. 10). four significant equity issues in online teaching will be considered below. a. hewitt and m. stubbs 4 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 digital literacy and access digital literacy is critical to a student’s ability to learn effectively in a digital environment (cagiltay 2006; hill and hannafin 2001; selim 2007; schrum 2002). the importance of examining the interaction of digital literacy with online education increases now that the popular notion of tertiary students as universally comprising ‘digital natives’ (howe and strausse 2000; tapscott 1998) has been debunked (buckingham 2007; hartman, moskal, and dziuban 2005), not only for mature-age students (oliver and goerke 2007), but also for students in general (lorenzo, oblinger, and dziuban 2007). disability online teaching may have complex impacts on students with a disability. it may reduce barriers faced by physically disabled students, including transport difficulties and mobility restrictions, which can make attending classroom-based teaching difficult. for students with a learning disability, e-learning may allow them to work at their own pace, to adapt materials to suit their needs, and may facilitate communication with peers and teachers (fichten et al. 2009, p. 250). online learning may also allow students with a disability to choose whether to reveal their disability at their discretion, while promoting equity and reducing discrimination (fichten et al. 2009, p. 242; mcfarland and fuller 2001). however, online learning may also exclude students with a disability if it is not thoughtfully designed to be mindful of their accessibility needs (fichten et al. 2000, 2001; mcfarland and fuller 2001). for example, use of online learning can pose problems for students with learning, visual and neuromuscular disabilities if it impairs their ability to use adaptive software capable of screen magnification, screen reading and dictation (burgstahler, corrigan, and mccarter 2004, p. 234; seale and cooper 2010, p. 1108). gender evidence collected a decade ago suggested that women were less enthusiastic about using information and communication technologies than men (katz 2006). according to kirkup (2002, p. 13), the online environment is ‘a gendered world operating in similar ways to the material world’. studies reported that women displayed higher levels of computer anxiety and lower levels of self-efficacy towards computers and the internet than men (comber et al. 1997; durndell and hagg 2002; okebukola and woda 1993; whitely 1997). however, more recent research suggests that the online learning environment is no longer intimidating to female students. for example, a 2007 australian study found no significant differences between female and male students with respect to being able to use an online learning environment confidently and effectively (coldwell et al. 2007, p. 15). indeed, greater value was placed by female students on using the online environment for communicating and collaborating with students from diverse backgrounds, and it has been suggested that the anonymity afforded by online environments could be empowering for women (coldwell et al. 2007, pp. 19–20). international students there is surprisingly little consideration of international students’ perceptions of online learning. existing studies have reached conflicting conclusions that cultural research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 5 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 difference can have a negative effect on students’ participation in online learning (shattuck 2005) or that cultural differences are mitigated in online education (walker-fernandez 1999). limited language proficiency can clearly affect learning outcomes (zhang and kenny 2010). ireland (2008, p. 154) noted that the on-demand revision facilitated by online learning was particularly beneficial ‘for international students and those for whom english is a second language.’ however, learners in an online environment may feel isolated and uncertain because of the absence of nonverbal cues and social presence (muilenburg and berge 2005), which may be particularly challenging for those international students raised in a cultural environment in which social context is emphasised (ky and lohr 2003). the project introduction and aims this project was designed to address the significant issues with law student distress identified in part 2, as well as to respond to areas for improvement within legal skills teaching identified both as a result of a curriculum mapping exercise and also on the basis of a 2012 survey of the whole bachelor of laws (llb) student cohort (and associated focus groups) which identified flexibility, accessibility and consistency of skills teaching as areas for improvement.1 the aims of our project were, therefore, to reduce student stress and maximise students’ satisfaction with their learning experience by providing scaffolded support in legal skills development across the llb curriculum. the project design was also informed by self-efficacy theory. self-efficacy theory the term ‘self-efficacy’ was developed to explain how individuals’ perceptions of their own competence and control develop, and affect their ability to cope with actual challenges (bandura 1977). the theory has been used by mckinney (2002) to inform the development of curriculum to minimise depression and anxiety amongst law students, and maximise their satisfaction and academic achievement. in essence, self-efficacy theory indicates that promoting a personal belief that you can control an outcome and achieve a desired result is likely to also increase the effort expended to achieve the result and, as a consequence, the result realised, and thus to improve self-esteem in addition to improving the achievement of the learning outcome (maddux 1995). there is evidence linking a lack of self-efficacy to depression (maddux 1995; maddux and meier, 1995) and anxiety (lloyd williams 1995). in a law school, the theory suggests that increasing the self-efficacy of students will not only increase the chances they will succeed, but will also reduce anxiety and promote wellness through improving self-esteem (mckinney 2002, pp. 245–255). a person’s degree of self-efficacy is affected by four factors (mckinney 2002, p. 237): (1) personal and imagined experience; (2) vicarious experience; (3) feedback and (4) physiological and or emotional reactions to an event. a. hewitt and m. stubbs 6 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 our project focussed on the aspects of personal experience and feedback. we sought to develop tools that would allow students to master particular skills through activity, creating an increased belief in their personal capacity. this was supplemented by the provision, throughout the modules, of feedback developed to be positive, and to focus on those parts of the module already completed, which has been demonstrated to increase self-efficacy (bandura 1997, p. 103, 106). the modules were also designed to be implemented with a focus on courses early in the degree, at the point when students were mastering the particular skills each module addressed. this was both a practical pedagogic tool, and one that would increase the impact of the modules on improving self-efficacy; positive messages about ability are most effective in increasing self-efficacy when provided early on in the development of a new skill (bandura 1997, p. 102). methodology this project commenced with a review of the literature to identify best practice in flexible skills development, focussing on online teaching methodologies. this review demonstrated that it was feasible to develop online resources to enhance students’ legal skills, and identified some of the equity and access issues which could be managed through planning. for example, it was determined that issues of digital literacy and access could be minimised by the choice of an accessible online tool to deliver the modules developed, which was compatible with the other learning management software used within the university, and was supported by our information technology team. after considerable research, and after waiting for the results of an internal audit of all learning technologies, it was decided to use articulate storyline for the project. the anticipated benefits of this particular technology were that it would: • be sufficiently intuitive and user friendly as to minimise access issues for all students, regardless of age or level of digital literacy; and • facilitate accessibility for all students by allowing for easy creation of pdf versions of the modules which can be printed as a learning tool and accessed by students requiring the use of assisted reading technologies, and permitting the use of both audio and text instruction where desired. the second stage of the project consisted of consultation with our colleagues to determine which skills should be addressed, and how the support tools developed could best be integrated within the llb curriculum. this research was the foundation on which we planned the development of 10 online learning modules which would focus on foundational skills, be embedded in compulsory courses, and be concentrated in the early years of the llb curriculum. it was also decided that the resources should be centrally available to all students, to facilitate their use as revision and ‘just-in-time’ learning tools. the skills addressed by each module, and the course in which the module sits, are set out in the table at the top of the following page.2 in designing the modules using articulate storyline software, we focussed on: • adding to the range of learning resources available to students, so as to cater to the diversity of learning styles; research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 7 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 • giving students greater autonomy in their learning, by creating resources which students can access at the time and place of their choosing; • supporting students in the development of foundational skills which we have anecdotal evidence are often found challenging; • utilising delivery strategies that are engaging, empowering, motivating and active; and • increasing self-efficacy by helping students to master new skills, and providing positive feedback at an early stage of the skills acquisition process. after refining the module design to take into account feedback from conference presentations (hewitt and stubbs 2013; stubbs and hewitt 2013), consultations with the coordinators of the courses to which the modules related, and feedback from two external expert referees,3 the modules were finalised. in february 2014, they were deployed to students via course-specific blackboard sites as well as a specifically created ‘professional legal skills’ blackboard site into which all law students were enrolled. each of the 10 modules integrated information delivery with a variety of engagement and active learning strategies, including • embedded activities; • self-evaluation opportunities; • provision of positive and constructive feedback; and • links between resource content and students’ day to day experiences. following deployment, investigation of the modules’ efficacy was conducted by way of an online survey, and student focus groups.4 evaluation sample a total of 277 students studying at the university of adelaide law school in semester 1, 2014 responded to this survey. the vast majority of respondents (95%) were llb students, with postgraduate students amounting to only 5%. the mean age was 23.83 years (sd = 7.78). the gender distribution was 66% female and 34% male, which closely matches the distribution of the law student population. ten per cent of participants were international students, which again is similar to the population skill course and year level 1 preparing for and engaging in seminars for effective learning foundations of law [year 1 (y1), semester 1 (s1)] 2 locating, using and understanding legislation foundations of law (y1, s1) 3 locating, using and understanding cases foundations of law (y1, s1) 4 locating and using secondary sources principles of public law (y1, s2) 5 citing legal resources correctly principles of public law (y1, s2) 6 writing a legal essay international law (y1, s2) 7 advanced case analysis equity (y2, s1) 8 professional legal ethics in corporate law settings corporate law (y2, s2) 9 writing a letter of advice to a client evidence and proof (y3, s2) 10 academic honesty a. hewitt and m. stubbs 8 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 of law students. respondents were also relatively evenly distributed across year levels, as figure 1 depicts. the number of participants that had completed each module (and a subset that had not completed any modules) is presented in figure 2. the number of students participating in the academic honesty module reflects the fact that, for many students, this module was compulsory (whereas all other modules were optional). efficacy in assisting students to acquire skills as demonstrated by figures 3 and 4, survey respondents largely agreed that the modules were useful and, overall, students had a positive opinion of the ability of the modules to complement traditional learning. as figure 3 demonstrates, there is a noticeable difference in perception of usefulness from students at different year levels. although students at all levels found the modules useful, first-year students reported the greatest satisfaction. this likely reflects that students in later years will 25% 17% 20% 14% 16% 5% 3% year level of respondents 1st 2nd 3rd 4th 5th 6th 7th+ figure 1. distribution of sample across different year levels. 0 50 100 150 200 250 academic honesty seminar classes case analysis and research interpre�ng legisla�on legal cita�on secondary source cita�on no module number of par�cipants figure 2. number of participants completing each of the assessed modules (or no modules). research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 9 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 have had other opportunities to acquire the skills that these modules aim to teach. this also reflects the project design, as the modules were primarily aimed at fundamental skills acquisition and predominantly embedded in lower-level courses. nonetheless, later-year students were still positive about the modules. indeed, one participant in the focus group noted approvingly that ‘it took me 3 years to learn the things you could get out of these modules’. another commented: ‘coming into law i didn’t know anything. i wish i had used these modules 4 years ago!’ another student emailed us to say: i’ve just transferred from [another group of 8 university law school] and i wasn’t sure if i needed to do them, but … they were great, especially the research based modules. i wish i had done something like this back when i started my law degree … i probably would have saved a lot of time and effort. strongly disagree moderately disagree slightly disagree slightly agree moderately agree strongly agree academic honesty seminar classes case analysis and research interpre�ng legisla�on legal cita�on secondary source research 'usefulness' of modules 1 2 3 4+ figure 3. ‘usefulness’ of each module across four different year group levels. strongly disagree moderately disagree slightly disagree slightly agree moderately agree strongly agree easy to access useful revision (current course) revision (different course) length appropriate overall assessment of the modules academic honesty seminar classes case analysis and research interpre�ng legisla�on legal cita�on principles of public law figure 4. mean ratings for the different evaluation questions for each module. a. hewitt and m. stubbs 10 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 survey participants were also asked to comment on individual modules. a common source of dissatisfaction was the requirement for all students to complete the academic honesty module. this demonstrates a resistance by students to skills modules being compulsory. however, student text responses indicate that this is a feature of the modules being introduced at a time when many students had already been studying for a number of years. the most common comments on the compulsory academic honesty module were that it was not appropriate for final year students but would be excellent for first year students. one student described the modules as ‘relevan[t] and important for first years’ but ‘unnecessary for final year students’. the overall assessment of modules by participants for the different questions asked is presented in figure 4. participants reported that the modules were easy to access, and useful, with no large difference between modules. their length was also typically perceived to be appropriate. on the whole, participants did not respond as positively regarding the likelihood of them revisiting the modules for revision purposes. this is particularly the case for the academic honesty and seminar classes modules. presumably, once students were aware of the information contained within these modules, they felt they had ‘got it’, with not much subsequent need for revisiting the information. nonetheless, as one focus group participant noted, ‘having the modules accessible for revision would be beneficial for many students’; another student emailed us to ask ‘will the modules remain up on [blackboard]? i’d like to be able to go back and access them again once i’ve got a research assignment’. overall the response regarding likely revision was positive for all except the academic honesty module, albeit that the student response regarding revision was not as strong as elsewhere. features of the modules most appreciated by students participants who completed one or more online modules were asked to provide their level of agreement (on a 1–6 likert scale) with a number of statements regarding the capacity of online modules to complement more traditional forms of teaching. the responses to these statements are presented in figure 5. overall, participants appear to have a positive appraisal of the ability of online modules to complement their learning. respondents identified as the most favourable features of the online learning modules their ability to clarify expectations and provide flexibility for students to control their own learning timetables. students were also positive about the modules’ capacity to facilitate review of the learning materials on demand (notwithstanding that student intention to use the modules for revision, reported in figure 4, was not especially strong), and the fact the modules offered a greater variety of materials to support their learning. students viewed the modules as complementary to their traditional skills learning, but were not as strongly in agreement as to how much new knowledge and skills was gained as opposed to opportunities to further develop skills already understood to some extent. this might suggest that students felt that online learning was most useful when reinforcing material introduced at least at a basic level via traditional face-toface teaching. student responses to qualitative questions identified the use of practical examples as a significant strength of the modules. one student observed ‘[i] liked how research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 11 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 it was guided, step by step with examples of each concept’, another praised the inclusion of ‘lots of examples which helped’, and another told us they ‘particularly liked reading the real-life situations’. similarly, a focus group participant commented that ‘[t]he really practical examples were quite helpful’. students also commented positively on the extensive provision of formative feedback through practical problems and quiz questions deployed throughout the modules. one student identified as the best feature of the modules that the ‘[p]lacement of relevant test questions immediately after each section was helpful in clarifying learning’; another said ‘i liked the problem scenarios’; another told us ‘i particularly liked the test questions’. that students especially commented on the use of real-life examples and provision of timely formative feedback as part of these online modules reminds us that pedagogy remains an essential foundation on which to build when implementing teaching innovations through technological means. student ranking of the benefits of the modules participants were asked to rank some advantages that online teaching modules can provide in order of importance (1 being the most important). their responses appear in figure 6 below. overall, law students perceived that obtaining extra opportunities for revision to prepare for assessments was the most important advantage online teaching methods can provide (compared with the comparison options). this was followed by having additional information presented to help them improve upon their skills. the critical themes to note are the assessment-centric focus by students, and the modules being appreciated as a supplement and extension of material introduced in traditional face-to-face classes, as well as being valued as a means of teaching additional skills not included in the traditional curriculum. 0% 50% 100% greater variety of materials for my learning greater flexibility for me to learn at a �me of my choosing clear informa�on regarding what is expected of me as a law student knowledge/skills that i have not developed via tradi�onal methods alone the ability to review learning materials whenever i wish to 'refresh my knowledge' propor�on of par�cipants benefits of the modules strongly disagree moderately disagree slightly disagree slightly agree moderately agree strongly agree figure 5. participant agreement with a number of statements about ways that online modules can complement traditional teaching methods. a. hewitt and m. stubbs 12 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 interestingly, flexibility was not the most highly ranked attribute of the modules in terms of student perceptions of importance, although students did agree that the modules had been successful at providing increased flexibility in their study. as set out in figures 5 and 6, respondents generally agreed that not only is increased flexibility important for their learning but also online learning provides that flexibility. this both reflects the importance ascribed to flexible learning and recognises the success of the project in this area. however, although students agreed the modules provided clarity around learning objectives, and introduced variety (one student commented that the modules added ‘an appreciated variety to traditional learning methods, feels not like typical studying and more like a quiz or game’), overall these benefits were valued less than the substantive skills-learning outcomes. impact of the modules on reducing student stress and anxiety a majority of participants agreed, as set out below in figure 7, that developing legal skills is stressful. one student explained in the focus group how the modules could reduce student stress associated with skills acquisition: there is nothing worse than sitting on lexis nexis and you know you are typing in the wrong search terms – and you know the information is out there – but you are doing the wrong thing and figuring out what to do. being able to know what you need to know would be invaluable. importantly, students also agreed that the online modules provided useful support to assist in the development of skills, and that knowing they could redo modules at any 0% 20% 40% 60% 80% 100% extra variety for my learning extra opportuni�es for learning objec�ves to be made clear to students extra flexibility for my learning addi�onal informa�on and examples addressing materials and skills that are not covered by tradi�onal teaching formats addi�onal informa�on and examples addressing materials and skills that are covered by tradi�onal teaching formats extra opportuni�es for revision material to be made available to help students prepare for assessments propor�on of par�cipants ranking of benefits of online teaching methods rank 6 rank 5 rank 4 rank 3 rank 2 rank 1 figure 6. a number of benefits of online teaching methods rank ordered from 1 (most important) to 6 (least important). this question was completed by participants who had experience with one or more of the online modules assessed by this survey. research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 13 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 time reduced stress. furthermore, the majority of participants agreed that the presence of online modules made them feel the law school has a support system to help them develop skills. similarly, as depicted in figure 6, students praised the fact that the online modules facilitated the ability to review learning materials whenever they wished and the extra opportunities that such modules provided for revision material to be made available. not surprisingly after the consistently high positive response evident from the previous questions asked in this survey, the majority of law students felt there needed to be more online teaching methods in law school. the critical messages from these responses are that (1) the majority of students find skills acquisition a stressful process, (2) these online modules were positively regarded by students as evidence of support by the school and (3) the availability of the modules, and their ability to support review as desired, was perceived by students to reduce their stress. these results confirm the expectation that skills acquisition contributes to law student stress, and the hypothesis underlying this project that online modules can contribute to mitigating the stress that students experience when they are learning new skills. evolutionary but not revolutionary benefits an examination of how students’ experiences with the modules impacted on their perceptions raises a cautionary note regarding the strength of the impact of the modules. for a number of skills pertaining to each of the modules, survey participants were asked whether they felt confident of what was expected of them regarding those skills; 0% 50% 100% online modules provide useful support to assist development of my skills online modules make me feel that law school has support system to help me develop skills knowing i can re-do modules at any �me reduces my stress i find developing law skills stressful i feel there needs to be more on-line teaching methods in law propor�on of par�cipants percep�ons of online learning methods strongly disagree moderately disagree slightly disagree slightly agree moderately agree strongly agree figure 7. participant agreement with a number of general statements regarding online learning methods. for the statements ‘i find developing law skills stressful’ and ‘i feel there needs to be more on-line teaching methods in law’, there was no significant difference found between participants with or without any experience of the online modules assessed in this survey. the remainder of statements presented in this figure were only asked of participants that had experience with one or more online modules. a. hewitt and m. stubbs 14 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 whether they believed they had the necessary knowledge and skills to perform at a high level; and their perception of the quality of the law school’s teaching in the relevant area. the responses, as set out in figures 8–13, demonstrate that in most cases students who had completed the relevant module were only slightly more confident or satisfied with the quality of education than those who had not completed the relevant module but who had experience with one or more other online modules, and this second group itself was only slightly more confident or satisfied than the group of students who had not completed any modules. thus, the impact of participating in an online module on these key outcomes is demonstrable but not revolutionary. however, considering that these modules are only one method whereby students learn the skills in question, and that the modules themselves are a one-off short experience, even a moderate impact can be taken as a success. overall, these results support the positive impact of the modules, but caution against overstatement of their beneficial effects. further, that students were almost as positive if they had taken any module, as opposed to the specific module addressing the skill being interrogated, suggests that part of the benefit of the modules is the knowledge that they exist and the general feeling of comfort and pastoral care it generates. this supports the findings reported in figure 7 that the modules both reduce stress and give students the feeling that a support system is in place to assist them. minimising the downsides of online learning the effective use of online learning to establish a supported learning environment is predicated on the minimisation of the potential disadvantages, which include the equity and accessibility issues discussed above. amongst these, the issue most commonly identified by respondents in their comments was digital literacy and access. while the majority of respondents agreed that the modules were easy to access, appealing and user-friendly, several students, particularly mature-age students, noted difficulties associated with using the technology. further, several students noted other accessibility issues including difficulties associated with translating words due to the inability to copy the text within the modules, the lack of a text-to-speech tool and the incompatibility of the modules with certain operating systems. some students also commented that they would be more likely to use the modules on an ongoing basis if they allowed the user to download, print or copy material from them, not at all slightly somewhat moderately very extremely expected of me knowledge/skills teaching of skill completed par�cular module not completed (par�cular module) not completed (any modules) figure 8. ratings for skill ‘being an active member in seminar classes’. research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 15 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 not at all slightly somewhat moderately very extremely expected of me knowledge/skills teaching of skill completed par�cular module not completed (par�cular module) not completed (any modules) figure 9. ratings for skill ‘finding and using cases’. not at all slightly somewhat moderately very extremely expected of me knowledge/skills teaching of skill completed par�cular module not completed (par�cular module) not completed (any modules) figure 10. ratings for skill ‘interpreting legislation’. not at all slightly somewhat moderately very extremely expected of me knowledge/skills teaching of skill completed par�cular module not completed (par�cular module) not completed (any modules) figure 11. ratings for skill ‘legal citation’. not at all slightly somewhat moderately very extremely expected of me knowledge/skills teaching of skill completed par�cular module not completed (par�cular module) not completed (any modules) figure 12. ratings for skill ‘finding secondary sources’. a. hewitt and m. stubbs 16 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 rather than having to revisit and resubmit the entire module. this may suggest an accessibility issue. in fact students did have the ability to download a pdf file of all instructional slides from each module, but for a small number of students this was evidently not obvious, so greater efforts were made in a later revision of the modules to draw all potentially helpful features to the attention of students. further, while a benefit often attributed to online learning is that it allows students a greater level of control over how the information is delivered to them, a number of the respondents did not feel that this was necessarily the case with these online learning modules. despite the fact that the length of the modules was generally perceived to be appropriate, several respondents were of the view that the modules would be more effective if broken down into smaller component parts. this would more effectively provide the flexibility for students to move through topics in a non-linear way, which some students felt was inhibited by the need to complete the entire module in a step-by-step fashion, including when revisiting the module. cohort analyses the literature examined earlier raised the possibility that student experiences might differ on the bases of gender, age, domestic or international student status, disability and stage of advancement through the degree. examination of the data reveals that only two of these factors are accompanied by a statistically significant difference in survey responses. disability the survey did not ask students to self-report disability, so no quantitative indication can be given of the impact of disability on the student experience of the modules. there were also no free text responses addressing disability as an issue. some free text comments were positive about accessibility more generally – students noted appreciatively that the modules were ‘easy to follow’, ‘simple to use’, and ‘[s] not at all slightly somewhat moderately very extremely plagiarism chea�ng collusion academic honesty impact on marks impact on admission to legal profession completed par�cular module not completed (par�cular module) not completed (any modules) figure 13. ratings for confidence in understanding for a number of concepts related to academic honesty. research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 17 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 traight forward with an appealing design’. thus, accessibility of the online modules generally was regarded positively, and we were not made aware of any access issues for students with a disability. gender our data indicates that gender was not a factor affecting the student experience of our online modules. responses to questions regarding the benefits of the modules did not vary significantly across genders, as figure 14 (below) shows. the magnitude of these results is fairly comparable, and all show a more favourable response from female students. however, none of these differences in mean was statistically significant in a series of independent samples t-tests. thus, our findings support that branch of the literature (including coldwell et al. 2007) that suggests there is not a gender difference in engagement with online learning. age and progression through the degree some differences in student responses were observed according to both age and progression through the degree. the only statistically significant difference according to age was not to do with the modules at all, but concerned the question ‘i find developing skills relevant to the study of law to be a stressful aspect of my learning’. students under 21 years of age agreed with that statement (4.11, n = 125) compared with students aged 21 years or over who were ambivalent (3.5, n = 101), the difference being statistically significant [independent samples t-tests, t(224) = 3.13, p < 0.05]. the lack of different experience of the modules according to age supports that part of the literature analysed above which suggests that age is not very relevant in understanding students’ abilities to interact with online learning (including lorenzo, oblinger, and dziuban 2007), although a small number of qualitative comments by mature-age students suggest that some members of that cohort experience challenges. figure 3 revealed that students were considerably more positive about the online learning modules if they were earlier in their degree, whereas more advanced students were less positive (presumably because they had already developed the skills being taught). this difference according to advancement in the degree was statistically significant in four categories, as figure 15 shows: i feel that the on-line modules i completed have complemented the more tradi�onal learning methods (i.e. lectures and seminar classes) by providing: mean response for gender: greater variety of materials for my learning greater flexibility for me to learn at a �me of my choosing clear informa�on regarding what is expected of me as a law student knowledge/skills that i have not developed via tradi�onal methods alone y to the abilit review learning materials whenever i wish to 'refresh' my knowledge female (n = 152) 4.20 4.72 4.74 3.92 4.49 male (n = 74) 4.01 4.59 4.49 3.74 4.30 figure 14. gender differences in mean student responses a. hewitt and m. stubbs 18 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 more advanced students still found the modules advantageous, but did not feel as strongly as earlier-year students. interestingly, later-year students also reported less anxiety over skills acquisition, which may account for the lower perceived usefulness of the online modules amongst that cohort. age accordingly did not statistically impact on perceptions of the online learning modules, although progression through the degree did appear to reduce student perceptions of the benefits arising from the modules. international students our results revealed some interesting differences in the perceptions of the online learning modules by international students compared with domestic students. international student responses were not statistically significantly different from domestic students across the measures of the benefits offered by the online learning modules, nor did their experience differ in finding skills development ‘a stressful aspect of learning’ which was ameliorated by the modules which ‘provided useful support’ in skills development. international students were, however, significantly more positive in their responses to three questions about the modules, as reported in figure 16 (at the top of the following page). as figure 16 shows, international students reported a significantly greater reduction in stress due to being able to revise the modules on demand, confirming ireland’s (2008, p. 154) finding that particular benefits arise from online learning for international students in terms of revision. further, international students overall, on the basis of the existence of these online learning modules, felt much more positively about the support systems the school provided. this is particularly interesting given a prior study which suggests international students have comparable levels of psychological distress to domestic students in australian law schools (larcombe et al. 2013, p. 423). international students, therefore, derived particularly strong support from the online learning modules compared with domestic students, with the ability to revise on demand being a key area where international students perceived a stronger advantage than domestic students. mean response for students according to degree progression: knowing i can come back and recomplete on-line modules at any �me reduces my stress i find developing skills relevant to the study of law to be a stressful aspect of my learning the on-line modules i completed provided useful support in assis�ng me to develop my skills the fact that these on-line modules exist makes me feel that the law school has a useful support system to help me develop the skills i require first year (n = 64) 4.63 4.81 4.64 4.25 second and third year (n = 83) 4.17 4.12 4.23 3.99 fourth and subsequent years (n = 79) 3.91 3.91 3.73 3.35 significance (one-way independent anova) f(2,225) = 5.75, p < .05 f(2,225) = 9.38, p < .05 f(2,225) = 6.48, p < .05 f(2,225) = 7.58, p < .05 figure 15. degree progression differences in student mean responses research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 19 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 overall, our cohort analyses indicate that the modules were differently experienced for students depending on their progression through the degree, and by international students compared with domestic students. we found no evidence of a difference arising from age or gender. conclusion our empirical analysis of student survey and focus group data generally supports a conclusion that the project was successful in achieving its key aims of improving student skills learning, and in so doing minimising one source of the anxiety and stress that impacts so severely on law students. partly as a result of the design of our modules, student responses indicated the most positive experience was for less experienced students, with a positive but weaker response from students who were further advanced in their degrees. international students also found the modules to be even more beneficial (particularly in terms of the ability to revise the modules on demand) compared with domestic students. our results did not support the existence of gender or age differences in the experience offered by the modules. overall, students found the modules useful. they valued the flexibility to control their learning process offered to them by the modules, and the ability to review the modules subsequently. however, the most highly valued attribute for students was the perceived ability of the modules to assist in completing assessment tasks. that students found the modules helpful in this way is a demonstration of their relevance – however, it also demonstrates an assessment-driven approach to learning that may be problematic in relation to student well-being. interestingly, students placed little value on the simple fact the modules introduced a variety of learning modes, being discerning enough to focus on their view of the effectiveness of the modules as opposed to their novelty. our data also confirm our suspicion that skills acquisition is a source of stress for australian law students. moreover, our results demonstrate that online skills development can be effective in reducing the level of stress reported by students and in generating a sense of well-being arising from the feeling that appropriate supports are in place to assist student learning. these results are consistent with the expectation based on this project’s use of self-efficacy theory. mean response for interna�onal/domes�c students: the fact that these online modules exist makes me feel that the law school has a useful support system to help me develop the skills i require knowing i can come back and re-complete on-line modules at any �me reduces my stress i feel that there needs to be more on-line teaching methods used by the law school (i.e. on-line modules, and on-line exercises) domes�c (n = 204) 4.18 4.10 3.65 interna�onal (n = 22) 4.82 4.82 4.36 significance (independent samples t-tests) t(224) = 2.16, p < .05 t(224) = 2.08, p < .05 t(224) = 2.19, p < .05 figure 16. differences in response from international and domestic students. a. hewitt and m. stubbs 20 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://dx.doi.org/10.25304/rlt.v25.1786 our research, therefore, demonstrates the potential for online skills development to complement and supplement existing face-to-face teaching of legal skills in law schools, leading to enhanced skills learning and the potential to reduce the level of stress and anxiety experienced by law students. it is anticipated that these benefits would be generalisable to law students in jurisdictions other than australia who are known to suffer similar issues with well-being and anxiety. future research could extend this study by employing an instrument to expressly measure student psychological well-being before and after an intervention using online professional skills development – such psychometric analysis was beyond the scope of this project. notes 1. university of adelaide human research ethics committee approval number hp2012-069. 2. year levels indicated are based on a full-time graduate student pathway. undergraduate students, those undertaking an llb combined with another degree, and part-time students, would progress more slowly. 3. dr edward palmer, senior lecturer, school of education, university of adelaide and professor vicki waye, professor of law and dean of teaching and learning, business school, the university of south australia. 4. university of adelaide human research ethics committee approval number h-2014-014. references australian law reform commission. (1999) managing justice: a review of the federal civil justice system, alrc, sydney. bandura, a. (1977) ‘self-efficacy theory: toward a unifying theory of behavioural change’, psychological review, vol. 84, pp. 191–215. bandura, a. (1997) self-efficacy: the exercise of control, freeman, new york. benjamin, g. a., et al., (1986) ‘the role of legal education in producing psychological distress among law students and lawyers’, american bar foundation research journal, vol. 2, pp. 225–252. buckingham, d. (2007) ‘digital media literacies: rethinking media education in the age of the internet’, research in comparative and international education, vol. 21, no. 1, pp. 43–55. burgstahler, s., corrigan, b. & mccarter, j. (2004) ‘making distance learning courses accessible to students and instructors with disabilities: a case study’, internet and higher education, vol. 7, pp. 233–246. butler, d. (2010) ‘entry into valhalla: contextualising the learning of legal ethics through the use of second life machima’, legal education review, vol. 20, pp. 87–110. cagiltay, k. (2006) ‘scaffolding strategies in electronic performance support systems: types and challenges’, innovations in education and teaching international, vol. 43, no. 1, 93–103. coldwell, j., et al., (2007) ‘gender and equity in e-learning’, australasian journal of information systems, vol. 15, no. 1, pp. 9–24. comber, c., et al., (1997) ‘the effects of age, gender and computer experience upon computer attitudes’, educational research, vol. 39, no. 2, pp. 123–133. culp, d. (1994) ‘law school: a mortuary for poets and moral reason’, campbell law review, vol. 16, pp. 61–101. dammeyer, m. & nunez, n. (1999) ‘anxiety and depression among law students: current knowledge and future directions’, law and human behavior, vol. 23, no. 1, pp. 55–73. deci, e. & ryan, r. (1985) intrinsic motivation and self-determination in human behavior, plenum press, new york. durndell, a. & hagg, z. (2002) ‘computer self efficacy, computer anxiety, attitudes towards the internet and reported experience with the internet, by gender, in an east european sample’, computers in human behavior, vol. 18, pp. 521–535. edwards, h. t. (1992) ‘the growing disjunction between legal education and the legal profession’, michigan law review, vol. 91, pp. 34–70. research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 21 (page number not for citation purpose) http://dx.doi.org/10.25304/rlt.v25.1786 fichten, c., et al., (2000) ‘access to educational and instructional computer technologies for post-secondary students with disabilities: lessons from three empirical studies’, journal of educational media, vol. 25, no. 3, pp. 179–201. fichten, c., et al., (2001) ‘technology integration for students with disabilities: empirically based recommendations for faculty’, educational research and evaluation, vol. 7, no. 2, pp. 185–221. fichten, c., et al., (2009) ‘disabilities and e-learning problems and solutions: an exploratory study’, journal of educational technology & society, vol. 12, no. 4, pp. 241–256. glance, d., forsey, m. & riley, m. (2003) ‘the pedagogical foundations of massive open online courses’, first monday, vol. 18, no. 5, [online] available at: http://firstmonday.org/ ojs/index.php/fm/article/view/4350/3673 grant, l. & spencer, r. (2003) ‘the personalized system of instruction: review and applications to distance education’, international review of research in open and distance learning, vol. 4, no. 2, pp. 1–17. guskey, t. (1997) implementing mastery learning, 2nd edn, wadsworth publishing company, belmont, ca. handsley, e., davis, g. & israel, m. (2005) ‘law school lemonade: or can you turn external pressures into educational advantages?’, griffith law review, vol. 14, pp. 108–133. hartman, j., moskal, p. & dziuban, c. (2005) ‘preparing the academy of today for the learner of tomorrow’, in educating the net generation, eds d. g. oblinger & j. l. oblinger, educause, washington, dc, [online] available at: http://www.educause.edu/research-andpublications/books/educating-net-generation helmers, k., et al., (1997) ‘stress and depressed mood in medical students, law students, and graduate students at mcgill university’, academic medicine, vol. 72, no. 8, pp. 708–714. hewitt, a. (2015) ‘can you learn to lawyer online? a blended learning environment case study’, the law teacher: the international journal of legal education, vol. 1, pp. 92–121. hewitt, a. & stubbs, m. (2013) ‘using e-learning tools to open the gate to professional skills acquisition’, paper presented to australian law teachers association conference, australian national university, canberra, 30 september 2013. hewitt, a. & toole, k. (2013) ‘the practical knowledge conundrum: what practical knowledge should be included in a law school curriculum and how can it be taught?’, new zealand universities law review, vol. 25, pp. 980–1022. hill, j. & hannafin, m. (2001) ‘teaching and learning in digital environments: the resurgence of resource-based learning’, educational technology research and development, vol. 49, no. 3, pp. 37–52. howe, n. & strausse, w. (2000) millennials rising: the next great generation, vintage, new york. huggins, a. (2012) ‘autonomy supportive curriculum design: a salient factor is promoting law students’ wellbeing’, university of new south wales law journal, vol. 35, pp. 683–716. ireland, j. (2008) ‘blended learning in intellectual property: the best of both worlds’, legal education review, vol. 18, pp. 139–162. johnstone, r. & vignaendra, s. (2003) learning outcomes and curriculum development in law, australian universities teaching committee, canberra. karpick, j. & roedinger, h. (2008) ‘the critical importance of retrieval for learning’, science, vol. 319, pp. 966–968. katz, r. (2006) the ecar study of undergraduate students and information technology, educause centre for applied research, boulder, co. kelk, n., et al., (2009) courting the blues: attitudes towards depression in australian law students and lawyers, brain & mind research institute, university of sydney, camperdown. keller, f. (1968) ‘goodbye teacher’, journal of applied behavior analysis, vol. 1, pp. 79–89. kendrick, d. (2010) ‘cost effectiveness in course redesign: the transformation towards e-learning’, in web-based education: concepts, methodologies, tools and applications, ed information resources management association, information science reference, hershey, pa, vol. 1, pp. 1392–1399. a. hewitt and m. stubbs 22 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://firstmonday.org/ojs/index.php/fm/article/view/4350/3673 http://firstmonday.org/ojs/index.php/fm/article/view/4350/3673 http://www.educause.edu/research-and-publications/books/educating-net-generation http://www.educause.edu/research-and-publications/books/educating-net-generation http://dx.doi.org/10.25304/rlt.v25.1786 kift, s., israel, m. & field, r. (2010) bachelor of laws learning and teaching academic standards statement, [online] available at: http://www.cald.asn.au/assets/lists/alsscresources/ kiftetalltasstandardsstatement2010.pdf kirkup, g. (2002) ict as a tool for enhancing women’s education opportunities; and new educational and professional opportunities for women in new technologies, united nations division for the advancement of women, [online] available at: http://oro.open. ac.uk/id/eprint/681 ky, h. y. & lohr, l. l. (2003) ‘a case study of chinese students’ attitudes toward their first online learning experience’, educational technology research and development, vol. 51, no. 3, pp. 94–102. larcombe, w., et al., (2013) ‘does an improved experience of law school protect students against depression, anxiety and stress? an empirical study of wellbeing and the law school experience of llb and jd students’, sydney law review, vol. 35, pp. 407–432. leahy, c., et al., (2010) ‘distress levels and self-reported treatment rates for medicine, law, psychology and mechanical engineering tertiary students: cross-sectional study’ australian and new zealand journal of psychiatry, vol. 44, pp. 608–615. lloyd williams, s. (1995) ‘self-efficacy and anxiety and phobic disorders’, in self-efficacy, adaption and adjustment: theory, research and application, ed. j. maddux, plenum press, new york, pp. 69–107. lord chancellor’s advisory committee on legal education and conduct. (1996) first report on legal education and training, aclec, london. lorenzo, g., oblinger, d. & dziuban, c. (2007) ‘how choice, co-creation, and culture are changing what it means to be net savvy’, educause quarterly, vol. 1, pp. 6–12. maddux, j. (ed) (1995) self-efficacy, adaption and adjustment: theory, research and application, plenum press, new york. maddux, j. & meier, l. (1995) ‘self-efficacy and depression’, in self-efficacy, adaption and adjustment: theory, research and application, ed. j. maddux, plenum press, new york, pp. 143–169. martin, f. (2003) ‘teaching legal problem solving: a problem-based learning approach combined with a computerised generic problem’, legal education review, vol. 14, no. 1, pp. 77–92. mascher, s. & skead, n. (2011) ‘on the record: the trials and tribulations of lecture recording in law’, university of western australia law review, vol. 35, pp. 407–432. mcfarland, p. & fuller, a. (2001) ‘equity issues in e-education’, journal of law and information science, vol. 12, pp. 87–96. mckinney, d., dyck, j. & luber, e. (2009) ‘itunes university and the classroom: can podcasts replace professors?’, computers & education, vol. 52, pp. 617–623. mckinney, r. a. (2002) ‘depression and anxiety in law students: are we part of the problem and can we be part of the solution?’ legal writing: the journal of the legal writing institute, vol. 8, pp. 229–255. mcnamara, l. (2000) ‘lecturing (and not lecturing) using the web: part 1’, legal education review, vol. 11, pp. 149–174. muilenburg, l. y. & berge, z. l. (2005) ‘student barriers to online learning: a factor analytic study’, distance education, vol. 26, no. 1, pp. 29–48. national alternative dispute resolution advisory council. (2012) teaching alternative dispute resolution in australian law schools, nadrac, canberra. nordmo, i. & samara, a. (2009) ‘the study experiences of the high achievers in a competitive academic environment: a cost of success?’, issues in educational research, vol. 19, no. 3, pp. 255–270. okebukola, p. & woda, a. (1993) ‘the gender factor in computer anxiety and interest among some australian high school students’, educational research, vol. 35, no. 2, pp. 181–189. oliver, b. & goerke, v. (2007) ‘australian undergraduates’ use and ownership of emerging technologies: implications and opportunities for creating engaging learning experiences for the net generation’, australasian journal of educational technology, vol. 23, no. 2, pp. 171–186. peterson, t. & peterson, e. w. (2009) ‘stemming the tide of law student depression: what law schools need to learn from the science of positive psychology’, yale journal of health, policy, law & ethics, vol. 9, no. 2, pp. 357–434. schrum, l. (2002) ‘dimensions and strategies for online success: voices from experienced educators’, journal of asynchronous learning networks, vol. 6, no. 1, pp. 57–67. research in learning technology citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 23 (page number not for citation purpose) http://www.cald.asn.au/assets/lists/alsscresources/kiftetalltasstandardsstatement2010.pdf http://www.cald.asn.au/assets/lists/alsscresources/kiftetalltasstandardsstatement2010.pdf http://oro.open.ac.uk/id/eprint/681 http://oro.open.ac.uk/id/eprint/681 http://dx.doi.org/10.25304/rlt.v25.1786 seale, j. & cooper, m. (2010) ‘e-learning and accessibility: an exploration of the potential role of generic pedagogical tools’, computers & education, vol. 54, no. 4, pp. 1107–1116. selim, h. (2007) ‘critical success factors for e-learning acceptance: confirmatory factor models’, computers & education, vol. 49, pp. 396–413. shattuck, k. (2005) glimpse of the global coral gardens: insights of international adult learners on the interactions of cultures in online distance education, unpublished doctoral thesis, pennsylvania state university. stefancic, j. & delgado, r. (2005) how lawyers lose their way: a profession fails its creative minds, duke university press, durham, nc. stubbs, m. & hewitt, a. (2013) ‘can we teach professional skills in the cyber world? creating e-learning legal skills development tools’, paper presented to ‘bricks to bytes?’: higher education research group of adelaide conference, adelaide, 27 september 2013. sullivan, w. m., et al., (2007) educating lawyers: preparation for the profession of law, josseybass, san francisco, ca. tani, m. & vines, p. (2009) ‘law students’ attitudes to education: pointers to depression in the legal academy and the profession?’, legal education review, vol. 19, pp. 3–40. tapscott, c. (1998) growing up digital: the rise of the net generation, mcgraw-hill, new york. u.s. department of education, office of planning, evaluation, and policy development. (2010) evaluation of evidence-based practices in online learning: a meta-analysis and review of online learning studies, [online] available at: http://www.ed.gov/about/offices/ list/opepd/ppss/reports.html walker-fernandez, s. e. (1999) toward understanding the study experience of culturally sensitive graduate students in american distance education programs, unpublished doctoral dissertation, florida international university, [online] available at: http:// digitalcommons.fiu.edu/dissertations/aai9952373/ watson, p. & field, r. (2011) ‘promoting student well-being and resilience at law school’ in excellence and innovation in legal education, ed s. kift et al., lexisnexis, chatswood, nsw, pp. 389–420. whitely, b. (1997) ‘gender differences in computer related attitudes and behaviour: a meta analysis’, computers in human behavior, vol. 13, no. 1, pp. 1–22. woods, d., et al., (2000) ‘the future of engineering education iii: developing critical skills’, chemical engineering education, vol. 34, no. 2, pp. 108–117. yule, j., ncnamara, j. & thomas, m. (2010) ‘mooting and technology: to what extent does using technology improve the mooting experience for students?’, legal education review, vol. 20, pp. 137–156. zhang, z. & kenny, r. f. (2010) ‘learning in an online distance education course: experiences of three international students’, the international review of research in open and distributed learning, vol. 11, no. 1, [online] available at: http://www.irrodl.org/index. php/irrodl/article/view/775/1481 a. hewitt and m. stubbs 24 (page number not for citation purpose) citation: research in learning technology 2017, 25: 1786 http://dx.doi.org/10.25304/rlt.v25.1786 http://www.ed.gov/about/offices/list/opepd/ppss/reports.html http://www.ed.gov/about/offices/list/opepd/ppss/reports.html http://digitalcommons.fiu.edu/dissertations/aai9952373/ http://digitalcommons.fiu.edu/dissertations/aai9952373/ http://www.irrodl.org/index.php/irrodl/article/view/775/1481 http://www.irrodl.org/index.php/irrodl/article/view/775/1481 http://dx.doi.org/10.25304/rlt.v25.1786 doi:10.3402/rlt.v19s1/7797 school trip photomarathons: engaging primary school visitors using a topic focused photo competition trevor collins a, *, richard joiner b , alice white b and lindsey braidley c a knowledge media institute, the open university, walton hall, milton keynes mk7 6aa, uk; b department of psychology, university of bath, bath ba2 7ay, uk; c museum, roman baths, abbey church yard, bath ba1 1lz, uk (received 2 march 2011; final version revised 17 may 2011) the aim of this study was to explore the potential of photomarathons as a fun and engaging way to support students making connections between what they learn during a museum visit and what they learn in school or other contexts. sixty primary school pupils aged between six and eleven took part in a photomarathon activity during their trip to the roman baths. the children were split into three groups. during their visit each group undertook three one-hour activities, namely: a photomarathon, a hands-on artefact exploration activity with a museum education officer, and a school-group handheld audio tour. for the photomarathon activity the children worked in subgroups of three and, for 15�20 minutes, took photos on three themes around the museum. at the end of the available time the children submitted a set of photos, one photo for each theme. photo galleries for each theme were then generated and made available on a website for the pupils. the students voted for the best photo in each theme gallery, and a small prize was awarded to the members of each team that took the winning photo. a week after the visit the children were asked a number of questions concerning their visit. the photomarathon was spontaneously mentioned by 41% (23/56) of the children as the best activity in their visit to the roman baths, which was more than any other activity they engaged in during the visit. overall, of the three activities the children liked the photomarathon the best. there were no age differences in how engaging the children found the photomarathon activity and all children regardless of age were able to take photographs. keywords: museum learning; school visits; photomarathons; empirical studies introduction photomarathons are a timed competition in which a group of photographers are given a set of themes to take photos on. at the end of the available time (typically around 12 hours) each entrant submits a set of photos, one photo for each theme in the given order. the judges then choose the best photos and award prizes. as a school trip activity, a photomarathon can be completed within an hour (including the judging and prize giving). the visiting group of pupils are first introduced to the rules of the competition and the judging process; they are then given cameras and three or four curriculum-related topics to take photos on. after a *corresponding author. email: t.d.collins@open.ac.uk alt-c 2011 conference proceedings 0135 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # t. collins et al. this is an open access article distributed 79 under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7797 set time limit (typically 15�20 minutes) the pupils return to upload their photos and vote for their favourite within each topic category. by using topics provided prior to the visit by the teacher and/or the pupils, the photos taken during the visit form a personal collection of resources that can be used after the trip as a basis for further class discussion or student research activities. we were interested to see if pupils enjoy the photomarathon activity and to what extent it can help motivate visitors to engage with museum exhibits. we argue that aligning the competition topics with the curriculum encourages pupils to think about those topics during their trip, and helps them to relate what they learn during the trip to the work they do afterwards in class. this paper introduces the research that motivated the development of the school trip photomarathon activity and the outcomes of a formative evaluation trial, including the design of the activity, the development of the technology used, the feedback received and the subsequent re-development of our photomarathon toolset. related work class visits to museums are a long established and popular activity for schools. for example, the roman baths in bath had 50,140 visitors in self guided educational groups between april 2009 and march 2010, and a further 9889 visitors in educational groups that took part in a teaching session organised by the roman baths. school museum visits such as these are important because they: . expose students to subject matter that cannot be covered in the classroom, . introduce them to resources in their community, . provide a varied social experience, . are memorable experiences for the students which can be drawn upon after the visit by the teachers in appropriate learning situations, and . can offer cross curriculum learning opportunities. research on museums has reported mixed findings in terms of knowledge and cognitive gains after visits to museums (donald 1991; griffin 2004). however, knowledge gains are not the only outcome of museum visits. students can also gain in terms of positive attitudes and motivation towards the museum and its subject matter (falk et al. 2004; jarvis and pell 2005; rennie and mcclafferty 1996). these gains are in some ways more important because they not only inspire further study and visits to the museum, but also motivate and engage the students in further school work based on their visit. considerable research has found that high levels of motivation and engagement in schools are related to higher academic achievement (for a recent review see ryan and deci 2009). further research has shown that post-visit visit activities in the school are important because they help students assimilate new concepts and resolve possible misconceptions (anderson, lucas, and ginns 2003; anderson et al. 2000). falk and dieking (2000) argue that it is only after a visit that the experience becomes relevant and useful. there has been considerable research on the use of mobile devices to support school visits (see for example cabrera et al. 2005; galani et al. 2003; hsi and fait 2005; mulholland, collins, and zdrahal 2005; o’hara et al. 2007; papadimitriou et al. 2006). this work has moved beyond the audio guide found in most museums, t. collins et al. 80 and developed systems that facilitate exploration, information searching, communication and documentation (hsi, 2002). there have, however, been few studies that have tried to bridge pre-visit, visit and post visit learning in the museum and the school. one notable exception is myartspace, which was a service developed to run on mobile phones to support inquiry learning by (vavoula et al. 2009). the process begins prior to the visit with goal-setting at the school. during the visit the students use myartspace to gather information either through taking photographs or field notes during their school field trip. this information is automatically sent to a website where the students can view and present their work after the visit. the authors report that the service was effective in enabling students to gather information in a museum, and provided resources for effective construction and reflection in the classroom. another recent example that supports inquiry learning across contexts is the zydeco system (cahill et al. 2010; kuhn et al. 2010). the system includes an online web component allowing students to define goals, questions and information for their science inquiries. this information is uploaded to a handheld device so that students can photograph, tag and annotate information in a museum. students can then access their museum work back in the classroom to complete their investigations. both of these systems are designed to support inquiry learning, an approach intended to encourage students to explore topics of interest that relates resources and activities to a focused problem or topic under investigation. the aim of this paper is to build on this work by exploring the potential of photomarathons as a fun and engaging way to support students making connections between what they learn in their museum visit and what they learn in school or other contexts. formative evaluation trial participants the sixty primary school visitors were divided into three more manageable groups. the first two groups were made up of pupils from two classes with an age range of nine to eleven year olds; the third group consisted of younger pupils, aged seven to eight. table 1 presents the distribution of pupils, teachers and assistants between the three groups. during the day each group undertook three one-hour activities, namely: the photomarathon, a hands-on artefact exploration session with one of the museum education officers, and a school-group audio tour using individual handheld audio guides. the session plan for the one-hour photomarathon was as follows: . introductions (10 minutes) k introduce the researchers and explain the competition rules and regulations, give each subgroup a camera, and explain how to use the camera (reminding the pupils about their three topics, which are also printed along with the rules on a label attached to each camera). . pupils exploration and photo taking (20 minutes) k pupils (supervised by their teacher and the assistants) go around the museum taking photos on each topic. . photo uploading (5 minutes) alt-c 2011 conference proceedings 81 k upload the photos to a computer using three topic folders. each subgroup selects a photo for each topic, which is placed in the corresponding topic folder. . reviewing and judging (15 minutes) k show each topic folder as a photo gallery and ask the pupils which ones they prefer and why (keeping the discussion focused on the specified topic). before moving on to the next topic, ask the pupils to vote for their favourite (having discussed them for a couple of minutes). the goal is to identify the photograph that best represents the topic (i.e. not the clearest or best composed photo). . prizes (5 minutes) k award a small prize (age appropriate) for the best photo in each topic. . reset (5 minutes) k the group moves on to the next activity while the researchers reset the cameras and computer for the next session. resources a large office was made available in the back rooms of the museum for the photomarathon sessions (see figure 1). a data projector and screen were used to figure 1. the room setting and equipment used for the photomarathon activity. table 1. the distribution of pupils, teachers and parental assistants across the three groups undertaking the photo marathon activity during a one day school visit in october 2010. group start time (duration 1 hour) age range number of pupils per group number of pupils per subgroup number of teachers number of assistants group 1 10:30 9�11 24 pupils in seven subgroups 4 groups of three pupils and 3 groups of four pupils 1 2 group 2 11:30 9�11 21 in six subgroups 3 groups of three pupils and 3 groups of four pupils 1 2 group 3 14:00 7�8 15 in five subgroups 5 groups of three pupils 1 4 t. collins et al. 82 show the photo galleries and some introductory powerpoint slides. canon powershot a460 (point-and-click) cameras were used by the pupils. as noted above, laminated labels reminding the pupils of their photomarathon themes and rules were attached to the pupils’ cameras. a printed copy of the powerpoint slides containing information on the activity and details of the website were given to each teacher (prior to the pupils taking photos), along with a set of small paper leaflets with the web address and password details to enable each of the pupils to access their gallery online after the visit (these were given to each of the teachers at the end of the photomarathon session). three computers (two netbooks and a laptop) were used by the authors to upload the photos. the photos were then copied from the two netbooks onto the laptop, which was running a local web server xampp. this is an easy to install apache distribution containing mysql, php and perl (http://www.apachefriends.org/en/ xampp.html). it could generate image galleries for each topic folder. the resulting gallery folders were copied to a publicly available web server at the end of the day. formative evaluation the children completed a pre-test one week before their trip to the roman baths, which was a general assessment of their knowledge of the romans. a week after their visit to the roman baths, the children were given a post-test. it consisted of a general assessment of their knowledge of the romans, which was the same as the pre-test, and an evaluation questionnaire of their experience at the roman baths. the questionnaire consisted of an open ended question asking ‘‘what was the best thing about the visit to the roman baths?’’ the next four questions asked the children to rate their visit to the roman baths overall, the photomarathon, the audio tour, and the teaching session. the children used a five point ‘smiley face’ scale which ranged from ‘awful’ to ‘brilliant’. the smiley face scale was developed and validated by read and macfarlane (2006) as a means of surveying children in the age group taking part in the study. the children were asked which activity they liked the most and which activity they liked the least. finally, they were asked whether they found taking pictures easy and whether they had visited the photomarathon website. findings the 18 subgroups took 154 photos in total. on average each group took 8.6 photos (with only one group taking less than three). in the evaluation questionnaire, the children were first asked what they thought was the best thing about the roman baths. we noted the number of children who mentioned either: the photomarathon, the audio tour, the teaching session, a general comment about the roman baths, a specific comment about the roman baths, or lunch. figure 2 shows that 41% (23/56) of the children commented that the photomarathon was the best thing about their trip to the roman baths, compared to 18% (10/56) who mentioned the teaching session, 11% (6/56) who mentioned the audio tour, 14% (8/56) who mentioned the roman baths in general; 12% (7/56) who mentioned something specific about the roman baths and 4% (2/56) who mentioned lunch. on this measure the photomarathon was the best aspect of their visit. alt-c 2011 conference proceedings 83 the next set of questions asked children to rate their visit to the roman baths using a five point smiley face scale. figure 3 shows that 64% (36/56) of students rated their trip to the roman baths as brilliant, 20% (11/56) rated it as really good and only one child rated it as not very good. there was an age difference, with 83% (10/12) of younger children (i.e. class 2) rating it brilliant compared to 35% (7/20) of the older children (x2�16.8, df�6, pb 0.05). fifty percent of the children (28/56) rated the photomarathon as brilliant, 29% (16/ 56) rated it really good and only one child rated it not very good (see figure 4). there was no age difference (x2�3.0, df�6, p�0.05). fifty seven percent of children (32/56) rated the audio tour as brilliant, 16% (9/56) as really good, 7% (4/56) rated it not very good and one child rated it as awful. there was no age difference (x2�11.6, df�8, p� 0.05). fifty five percent of the children (30/55) rated the teaching session as brilliant, 16% (9/55) rated it as really good, one child rated it not very good and one child rated it awful. there was no age difference (x2�9.9, df�8, p�0.05). comparing the three activities, the most highly rated was the audio tour followed by the photomarathon and then the teaching session. the differences are very small and not significant. the children next had to select which of the activities they liked the best (figure 5 left). overall, the most popular activities were the audio tour and the photomarathon, both selected by 40% of children (21/53). the teaching activity was selected by 20% of the students (11/53). they were next asked which activity they liked the least (see figure 5 right). forty nine percent of the students (24/49) said they least liked the teaching activity, followed by 31% (15/49) choosing the audio tour and 20% (10/49) least liking the photomarathon. a third of the children (19/57) said they had looked figure 2. the students’ reported best thing about the visit, for each class and total. t. collins et al. 84 at the photomarathon website and 88% of the children (50/57) said they had found the cameras easy to use. comparing the children’s knowledge of roman britain as assessed by the pre and post questionnaire, there were no significant differences in the children’s knowledge after visiting the roman baths compared to before. discussion the aim of this evaluation was to investigate the use of photomarathons as a way of supporting students making connections between what they learn during their museum visit and what they learn in school or other contexts. the photomarathon was spontaneously mentioned by 41% of the children as the best activity in their visit to the roman baths, which was more than any other activity they engaged in during the visit. fifty percent of the children thought the photomarathon was brilliant, and only one child did not think the photomarathon was good. there were no age differences in how engaging the children found the photomarathon activity and all children regardless of age were able to take photographs. thus, as an activity the students found it a fun and engaging experience. unfortunately, we found no significant difference in learning outcome after the visit, which is not unusual (griffin 2004) and possibly can be explained in terms of the short duration of the visit and the unfamiliar and unusual location (donald 1991). there were a number of difficulties that emerged with the photomarathon activity. the timings of each photomarathon session were challenging to manage. the pupils quickly grasped the goal of the competition, they understood the topics, figure 3. the students’ ratings of their visit to the roman baths, for each class and total. alt-c 2011 conference proceedings 85 and had no difficulty operating the cameras. the photo uploading process took longer than expected. this was partially due to the logistics of asking each group to identify their selected photo for each topic. the peer judging of photos generally worked well. each student could vote for one photo in each topic category, but could not vote for their own photos. in cases where the vote was split between entries one of the researchers took on the role of a (television talent show) judge to choose a winner. the teacher for the pupils in the youngest age group suggested after the session that it may also be appropriate to include a small prize for taking part in the competition. some problems also occurred regarding the attribution of photos to each group. as identical cameras were used, the automatically generated image filenames were similar for every group. for some of the topics multiple groups had photographed the same or figure 4. the students’ ratings of the photomarathon activity, for each class and total. figure 5. the students’ most liked (left) and least liked (right) activity for each class. t. collins et al. 86 very similar objects, resulting in a couple of cases were the attribution of the winning photo was difficult. activity revision and toolset re-development in general, the photomarathon activity went well and the pupils enjoyed it. however, due to the timing and photo attribution challenges discussed above, an alternate means was sought for uploading and labelling photos. a solution has been developed that uses android mobile phones; a web server, such as the temporary laptop server used above; and a wireless local area network. the photos taken with the camera on each phone can be automatically synchronised onto the local web server over the wireless network (see figure 6). this approach uses commonly available relatively cheap technology, along with an existing set of free phone applications. to automatically synchronise the images from each phone, a scaled down web server application (such as kws) running on each phone will make the photos on each phone’s sd card available as a website. a web synchronising program (such as wget or rsync) running on the local laptop web server is then used to download the images from each phone to the relevant group folder on the web server. as the students return from taking their photos the phones automatically connect to the wifi network (if only available locally) and the photos are uploaded to the website. each group can then select their preferred image for each topic and submit it (i.e. drag it) to the corresponding topic gallery. the attribution of photos is handled automatically by appending a group name or identifier to each photo file as it is uploaded to the website. conclusions critical for the success of photomarathons is the engagement of the teachers. the teachers in our study were very engaged, and were active in selecting the appropriate themes prior to the visit to the roman baths and the follow on activities using the photo galleries in the school. it was also important to discuss the themes with the educational staff at the roman baths to ensure they were suitable for the site. figure 6. an example photomarathon toolset for collecting photos from a set of android mobile phones, comprising of a wireless router, a netbook computer and some phones. alt-c 2011 conference proceedings 87 the revised design of the proposed system means that the photomarathons could be an activity ran by the museum or the school. it only requires android mobile phones with a digital camera and wifi connectivity, and a network enabled netbook (or laptop) computer. in our study a projector and screen were set up in a room at the roman baths, but it would be possible to run the presentation and judging process at the school after the children return from their visit. it is unclear whether it would be more beneficial educationally to run the judging process during the visit or after the visit. after the visit, it would be another way to connect the visit to the museum with the learning taking place at the school. the advantage of supporting the activity in the museum would be that it would have dedicated resources and staff for supporting and running photomarathons, which would require minimal preparation on the part of the teachers to set up. the flexibility of the system also means that it could be run in non-institutional contexts (e.g. field trips) and visits to locations where it may be difficult to run the judging process in situ (e.g. visits to cities). another issue which may be worth exploring is whether photomarathons are more beneficial as a group activity or an individual activity. currently, they are a group based activity and research has consistently shown the benefits of group based activities over individual activities (lou, abrami, and d’apollonia 2001). informal observation of the groups provided evidence for and against this view. some groups would discuss the themes and have discussion about which images would best illustrate those themes, however other groups would distribute the activity between themselves with each child taking one photograph with little or no discussion. in future research, we are planning to investigate the discussion children have concerning the photographs and whether this is related to their benefits of participating in photomarathons. this study has shown that photomarathons are a fun and engaging activity. further research is being undertaken to investigate the potential of photomarathons, using the revised design as a means of successfully linking visits to museums with learning in schools. acknowledgements we would like to acknowledge the support of the staff at the roman baths, without whose support and enthusiasm we would have been unable to conduct the study. we would also like to thank eileen scanlon and the personal inquiry project for the loan of the cameras and clifton school for allowing us to observe their visit to the roman baths, which gave us the initial idea for the project. above all, we would like to thank anne hewitt, jenny crossthwaite, darren roberts, ruth roberts and the children of farmborough school for participating in the activity during their visit, and for offering activity design suggestions and competition themes prior to visit, as well as their feedback and comments afterwards. references anderson, d., k.b. lucas, and i.s. ginns. 2003. theoretical perspectives on learning in an informal setting. journal of research in science teaching 40, no. 2: 177�99. anderson, d., k.b. lucas, i.s. ginns, and l.d. dierking. 2000. development of knowledge about electricity and magnetism during a visit to a science museum and related postvisitactivities. science education 84: 658�79. cabrera, j.s., h.m. frutos, a.g. stoica, n. avouris, g. fiotakis, and k.d. liveri. 2005. mystery in the museum: collaborative learning activities using handheld devices. in proceedings of mobile hci 2005. salzburg, austria. cahill, c., a. kuhn, s. schmoll, a. pompe, and c. quintana. 2010. zydeco: using mobile and web technologies to support seamless inquiry between museum and school contexts. in t. collins et al. 88 proceedings of the 9th international conference on interaction design and children (idc’10), 174�177. new york: acm. donald, j.g. 1991. the measurement of learning in the museum. canadian journal of education 16, no. 3: 371�82. falk, j.h., and l.d. dierking. 2000. learning from museums: visitors’ experiences and their making of meaning. walnut creek, ca: altamira press. falk, j.h., c. scott, l.d. dierking, l. rennie, and m. cohen-jones.. 2004. interactives and visitor learning. curator 47: 171�98. galani, a., m. chalmers, b. brown, i. maccoll, c. randell, and a. steed. 2003. developing a mixed reality co-visiting experience for local and remote museum companions. in proceedings of hci international 2003 (hcii 2003). crete, greece: lawrence erlbaum associates, 1143�1147. griffin, j. 2004. research on students and museums: looking more closely at the students in school groups. science education 88, suppl. i: s59�s70. hsi, s. 2002. the electronic guidebook: a study of user experiences using mobile web conent in a museum. in proceedings of ieee international workshop on wireless and mobile technologies in education, ed. m. milrad, u. hoppe, and kinshuk. sweden: vaxjo. hsi, s., and h. fait. 2005. rfid enhances visitors’ museum experience at the exploratorium. communications of the acm 48, no. 9: 60�65. jarvis, t., and a. pell. 2005. factors influencing elementary school children’s attitudes toward science before, during, and after a visit to the uk national space centre. journal of research in science teaching 42, no. 1: 53�83. kuhn, a., c. cahill, c. quintana, and e. soloway. 2010. scaffolding science inquiry in museums with zydeco. proceedings of the 28th of the international conference extended abstracts on human factors in computing systems (chi ea’10), april 10�15, in atlanta, georgia, usa. acm, new york, ny, 3373�3378. lou, y., p.c. abrami, and s. d’apollonia. 2001. small group and individual learning with technology: a meta-analysis. review of educational research 71: 449�521. mulholland, p., t. collins, and z. zdrahal. 2005. bletchley park text: using mobile and semantic web technologies to support the post-visit use of online museum resources. special issue: portable learning: experiences with mobile devices. journal of interactive media in education, no. 24: 1�21. o’hara, k., t. kindberg, m. glancy, l. baptista, b. sukumaran, g. kahana, j. rowbotham. 2007. collecting and sharing location-based content on mobile phones in a zoo visitor experience. computer supported cooperative work 16, no. 1, 11�44. papadimitriou, i., v. komis, n. tselios, and n. avouris. 2006. designing pda mediated educational activities for a museum visit. in proceedings of cognition and exploratory learning in the digital age (celda 2006). barcelona, spain. read, j.c., and s. macfarlane. 2006. using the fun toolkit and other survey methods to gather opinions in child computer interaction. proceedings of the 2006 conference on interaction design and children (idc’06), 81�88. 7�9 june 2006, tampere, finland. new york: acm. rennie, l.j., and t.p. mcclafferty. 1995. using visits to interactive science and technology centers, museums, aquaria, and zoos to promote learning in science. journal of science teacher educational evaluation and policy analysis 6: 175�85. ryan, r.m., and e.l. deci. 2009. promoting self-determined school engagement: motivation, learning, and well-being. in handbook on motivation at school, ed. k.r. wentzel and a. wigfield, 171�196. new york: routledge. vavoula, g., m. sharples, p. rudman, j. meek, and p. lonsdale. 2009. myartspace: design and evaluation of support for learning with multimedia phones between classrooms and museums. computers & education 53, no. 2: 286�99. alt-c 2011 conference proceedings 89 a theoretical view on concept mapping ray mcaleese heriot-watt university, uk abstract auto-monitoring is the pivotal concept in understanding the operation of concept maps, which have been used to help learners make sense of their study and plan learning activities. central to auto-monitoring is the idea of a 'learning arena' where individuals can manipulate concept representations and engage in the processes of checking, resolving and confirming understandings. the learner is assisted by familiar metaphors (for example, networks) and the possibility of thinking 'on action' while 'in action'. this paper discusses these concepts, and concludes by arguing that maps are part of the process of learning rather than a manifestation of learning itself. auto-monitoring is suggested as an appropriate term to describe the process of engaging in the learning arena. 'it is in these shimmering and incessant embraces that the infinite patterns, the infinite maps of the mind, are created, nurtured and grown. radiant thinking reflects your internal structure and processes. the mind map is your external mirror of your own radiant thinking and allows you to access this vast thinking powerhouse.' tony buzan (1993). definition auto-monitoring is a technique of personal or group knowledge-presentation and re-presentation. such techniques are often supported by computer applications that make graphical representations of knowledge structures. this process is at the core of what is called concept mapping. context and background learning technology comprises a collection of pragmatic and theoretical positions on the way learners come to know and teachers provide the appropriate experiences. at its core is a developing design philosophy that uses instructional design theory or educational technology. within learning technology is an interest in what some might call content free courseware or mindtools (kommers 1992). the idea of using concept maps (novak and gowin 1984) or mind maps (buzan 1993) as part of an instructional strategy is not new (bogden 1977; 38 alt-j volume 2 number 1 reader and hammond 1993), but it is timely to consider the contribution that such techniques have made to learning technology. this paper sets out a theoretical base for thinking about concept maps. earlier attempts have suggested some elements in the process (deikhoff 1982; gold 1984; mcaleese 1985; mcaleese 1987; mcaleese 1990; mcaleese 1992). the principal idea behind such tools is that of auto-monitoring the ability of learners to develop thinking skills. the term auto-monitoring (a/m) will be used here to encompass components of mindtools, mind maps, concept mapping, thinking skills, and so on. before directly addressing the theory behind a/m, however, it will be useful to take as a springboard the current debate in higher education. the main strand to this background is the need for students to acquire study skills. the need for better learning and study skills in a significant sector of the student population is identified in the howie report (1992), the macfarlane report (1993), the edinburgh university crli report on the transition from school to higher education in scotland (wall et al 1991) and the report of the national commission on education {learning to succeed 1993). all these sources identify the need for students in higher education to be able to think and to develop critical thinking skills. cognitive tools a realistic starting point is what has been called situated cognition (brown et al 1989) the most useful contribution is the claim that concepts are both situated and progressively developed through activity. knowledge is claimed to be like tools, and thinking occurs when the "tools are used for some task. brown et al (1989) and carroll (1989), along with many others, provide a stable raft of research supporting the idea of conceptual tools. a/m is the executive control and operation of such tools (for example, clancey 1993). not everyone is convinced that there are practical applications for situated cognition, but it does provide a scaffolding for the argument (hoppe 1993). a/m is the craft that the apprentice is acquiring in formal and informal education. that is, the learner is acquiring the skills needed to operate with and on concepts. some would argue that this is also the function of language, namely a means by which learners can operate on concepts. learning, seen in this way, is the outcome of learner activity. it is something learners cause by their.actions: learning is a process of re-constructing and re-presenting what is known (arzi and west 1986). a/m is therefore based on a set of assumptions and procedures that allow learners to be and to become responsible for their own learning. learning is seen as being something that occurs as a result of student/learner activities. an a/m model for concept mapping figures 1 and 2 indicate the main components and suggest their interdependencies. the learner is engaged in a cognitively active process in an environment that supports the manipulation of knowledge artefacts: ideas, concepts etc. the general process can be called engagement. the learning arena is the virtual space where operations on entities occur. in this space, the learner is able to engage with representations of information, ideas and concepts. this is a type of rhetorical space (scardamelia and bereiter 1987). it is not constrained except with the application of a cognitive template (see streitz and hammeman 1990). the engagement occurs at two levels. first, the operational level where the computer interface allows nodes to be moved, nodes created, etc. secondly, the cognitive level where thinking occurs. in general thinking precedes operations; however, operations will precipitate thinking, and there 39 ray mcaleese a theoretical view on concept mapping • node creation • node deletion • linking ^ /tools) t^^nnatioi,,-concept • ~ ~ learning arena q7mplate<r^) tf • network ' • procedure • heirarchy . engagement v /stages a v ina/m ) si. i s learner figure 2: figure 1: the learning arena auto-monitoring (aim) is therefore a cybernetic system in operation. in addition, the learners can assess previous understandings. this accession is dynamic that is, it occurs while operations are on-going in the arena. ideas are created on the fly, concepts are recalled when appropriate, and so forth. the learner has tools to use and operates through a series of stages (figure 2), though the stages, while explicit and sequential in terms of the operations, need not necessarily be followed. acknowledgement is the state of initial awareness: the learner thinks about some concept. representation is the formal activity allowed by the tools (for example, concept label creation). viewing is the core activity of being perceptually aware of the entities in the learning arena. there is a loop between representation and viewing. checking is a process whereby concept labels are compared, link values are scrutinized, and the learner attempts to make sense of the nodes, links, etc. resolution is required when conflicts occur in labelling or linking. the processes of viewing, checking and resolving operate a loop. confirming inacknowledgement confirming • re figure 2: the stages of aim 40 alt-j volume 2 number 1 volves operations such as agreeing that the representation is what was intended and has no inconsistencies. confirming is a form of commitment (see below). in addition, the learner has available a number of templates that constrain the operations performed. in general, the principal template is the net, consisting of nodes and links. conceptually the net is n-dimensional. the tools are the computer-related operations using a wimp (graphical) interface. tools will change with the instantiation of different templates. in general, tools will allow representation of elements of templates to be manipulated. for example, nodes can be moved, created, deleted, and the like. although the net is the most popular template, the outline or constrained net is also useful. the outline is a hierarchy, and the freedom to manipulate concepts is pre-determined by an [is-a] relation: (b is-a a -> b has attributes of a) processes are cognitive operations that operate on their own or in conceit with engagement. processes are dependent on the template selected. for example, in a hierarchy template there is an inheritance influence on the operations. dependencies are created and attribute effects are transmitted through the underlying structure. an operational definition of a/m in concept maps learners represent concepts, and in so doing make propositions. kathleen fisher calls these propositions instances in semnet semantic net (fisher et al 1990). here ambiguities arise, propositions suggest other propositions, and elements in the propositions need disambiguation that is, differentiating and integrating. the learner will experience dissonance and will have to make commitments. this state leads to a heightened awareness of what is known and what is not (mcaleese 1985). tacit or ill-remembered ideas are exteriorized through engagement with represented propositions (pask 1984). the acquisition of knowledge (that is, the outcome of activities) is a negotiated state. meaning is negotiated by resolving and making explicit concepts, their attributes, etc. (moyse 1991). figure 3: activities, processes and visualisation prior knowledge proces accomodation differentiation integration contextualising exemplifying dissonance learner 1. represents concepts 2. makes claims/ propositions 3. resolves ambiguities in propositions 4. is aware of the decisons being made 5. "exteriorises" tacit knowldege activity task orientation learning arena knowledge re-presentation to 41 ray mcaleese . a theoretical view on concept mapping this activity is operationalized in the wimp interface. mouse movement indicates, but does not determine, point of view (fairchild 1993); node creation (involving several sub-actions depending on the a/m application) indicates the exteriorization of a prepositional element; linking nodes indicates the explication of the proposition. nodes are represented by an implicit semantic distance (holley and dansereau 1988). the processes at work are general cognitive processes of accommodation, differentiation, etc. the computer's role is to manage the activities and to provide a virtual learning arena where actions and intentions can be seen, recorded and if necessary replayed. the computer, while not essential to a/m, is thus a powerful adjunct in the supportive learning environment. a key aspect of a/m is visualization. while a graphical representation is not necessary for a/m to occur, it does seem that the ability to see the results of thinking has a strong effect on the process. work by piagetian thinkers suggests that the concrete portrayal of abstract, illconceived ideas is an important precursor to problem solving. there is compelling evidence from research in information management that graphical metaphors facilitate the specification of intent and the presentation of choices. indeed, work on degrees of interest, for example fish-eye viewers, suggests that data-presentation techniques are not only important in handling large numbers of nodes and relationships, but also allow learners to make use of different perspectives on the same data set concurrently (fairchild 1993). figure 3 provides further explanation. visualization and the problem of scale the above model suggests an over-arching process of visualization to encompass the process by which learners make sense of the web of interconnected concept labels. as long as the learning arena is a bounded 2-dimensional area, the problem of too many concepts does not arise. however, sooner or later learners create many nodes and wish to make many associations. the problem of visualizing large semantic spaces has been tackled by mcc (microelectronics and computer technology corporation) with their work on semnet (the name is the same as that used by kathleen fisher for her semnet, based on novak et al, referred to previously). semnet research at mcc suggests that in visualization, three states for learners are important: • they must be able to identify (see) individual concept labels; • the relative positions of concept labels must be obvious in the semantic space (the learning arena); • learners must be able easily to detect relationships between concept labels. in addition, learners must be able to make sense of the dimensions of the space. a learning arena portrayed on a computer screen is confined by the 2-dimensional window. judicious use of overviews and fish-eye views can make the space more extensive. semnet by mcc used the idea of point-of-view very effectively to reduce complexity and to show, at the same time, the relevant nodes. in general, the 2-dimensional space is concerned only with semantic closeness. learners can place concept labels as closely or as remotely as they choose. the meaning of the location is known only to the individual who creates the network. i have suggested that two views are of particular value:.terrain views and street views (mcaleese 1983). in the former, the learner is able to see the extent and context of knowledge propositions. in the latter, the learner sees the detail and relative interdependencies. 42 alt-j volume 2 number 1 backing and warrant the main problem is that a/m is such a universal concept that it draws on a wide range of developed areas of understanding. therefore, the number of related ideas that contribute to the thinking on a/m is open to debate and question. the influences on a/m range from constructivist positions explaining the way learners make sense, to normative positions that argue for active reflective learners who debate and evaluate data with a degree of rigour. the tension between a description of 'how it is ' and a description of 'how it should be' is not unique to a/m. the key result influences are concept mapping, meta-cognition, and reflective practice, though these areas are not mutually exclusive. concept mapping key points the technique is usually associated in this context with joseph novak (novak and gowin 1984). concept maps represent meaningful relationships between entities (concepts) in the form of propositions. a proposition is two or more concept labels linked by a labelled relationship in a semantic unit (see also fisher et al 1990 for her work on semnet). a concept map, according to novak and other 'mappers' is a technique for externalizing concept labels and propositions created by concept labels and relationships. the key points are as follows. • meaningful relationships the representations make sense to the learner. they may or may not be true it is possible to have a raft of untrue meaningful relationships. indeed, many learners do have such entities, especially in their understanding of basic science. • concept labels the representation is a label or a sign for something that is signified. therefore, the sign { m o l e c u l e } is a sign for a chemical entity that consists of atoms held together and acting in a specified way. • propositions the meaningful units make sense according to the learner. therefore, '{isomers} have {n different forms}' is a proposition created by two concept labels and one relationship. in this case, the relationship is of special form it indicates an attribute of the first concept (see mcaleese 1990 for a more detailed view of the syntax of such propositions). the mind map and flowscape many other authors have contributed to the understanding of concept maps, most notably tony buzan (mind maps), and recently edward de bono (flowscapes). the mind map (buzan 1993) is a formalized procedure for using associations to create a pictorial representation of one or more ideas. the technique relies on brain-storming. the precise mapping between cortical function and the neuro-physiology of brain cells to bitmaps and 'clickable objects' on a computer screen remains uncertain. it is very tempting to say the mind map or other concept map can be isomorphic with 'understandings' as they are or must be, in the brain. not even buzan or de bono would go that far, yet the metaphor is very apt, and buzan's lucid writing makes many learners perform more effectively by providing a plausible study technique. learners do seem to improve their overall performance in independent tests when they have 'sorted out their thinking' using a mind map. de bono, using the flowscape (de bono 1993), takes a very similar approach to buzan. he has deeper roots in piaget, but the technique of brain-storming creates the network or flowscape. 43 ray mcaleese '• a theoretical view on concept mapping de bono takes the argument further by insisting that it is the dynamic nature of the interconnections that creates what he likes to call water logic. that is, a dynamic form of associations which is more concerned with what 'will happen' or 'becomes', than what 'is'. an important feature of the flowscape is the identification of clusters of concepts that are or become interdependent. they form loops, and de bono uses such ideas in a diagnostic fashion to help learners identify key areas of thinking. good or bad maps? many workers have tried to score or evaluate maps. in particular, novak has suggested, along with gowin, two methods of scoring or formalizing thinking. novak's technique is based on a detailed application of ausubel (novak and gowin 1984). the degree of subsumption and differentiation, along with levels of differentiation, can be used to score a map once created. this technique does not address an ongoing evaluative function that may act as control over learners. learners can only use a post-production facility. the place of commitment buzan and others suggest that there is an affective aspect to mapping. there has to be a commitment to the consequences of what is emerging. it is as if the creation of maps leads to a contract that the learner adopts. an idea that associates with this is cognitive dissonance (festinger 1957). cognitive dissonance is the state of mind that occurs when there is a tension between what should be and what is. ideas conflict, and there is dissonance between concepts, positions and relationships that requires resolution. often this happens when differentiation or integration occur. two concepts are so similar that one label can be used to denote them. the association with festinger is that cognitive dissonance involves not only cognitive attributes but affective attributes. in one sense, this is the tension between what 'i think i know' and 'i can re-present what i know'. commitment to a re-presentation may be dangerous as it fixes the players and their positions. in essence, there is always going to be cognitive dissonance in a/m, as dissonance is a creative tension (see dole and niederhauser 1990). it drives the process by 'looking for' a state of rest or resolution. dissonance is the cybernetic oil that makes the process work. consequently, commitment to maps created during a/m is not necessary. meta-cognition meta-cognition generally refers to awareness of knowledge and an ability to comprehend, control and manipulate cognitive processes. the rather crude term helicoptering has been applied to the state in which learners can see their own conceptual terrain. they can rise above their understandings and explore their cognitive space as it is. this description seems very close to what is implied by auto-monitoring. the principal flaw here lies in the state of the conceptual terrain: the metaphor breaks down as the conceptual landscape does not necessarily exist prior to 'levitation'. a/m creates the terrain as well as providing a vehicle for viewing its topography. this distinction can be implied by that made by buzan between 'note taking' and 'note making'. in the former, the terrain is almost complete; in the latter, the terrain is still evolving. note making is rather akin to helicoptering above a volcanic eruption where the new terrain is still being formed. in a recent review, osman and hannifin (1992) suggest that the principal components for meta-cognition are: 44 ait-j volume 2 number 1 • meta-memory knowledge and awareness of strategic behaviours and memory systems; • meta-comprehension the conscious process of knowing about comprehending and knowing how to comprehend; • self-regulation the continuous fine tuning by learners in response to and in the absence of knowledge of results concerning cognitive errors; • schema training the development of cognitive structures providing a framework for comprehension (schemas help learners generate perspectives that can be tested, reconciled and reconstructed with new engagements); • transfer the application of one set of procedures or skills to a different situation (transfer can be either vertical or lateral). from the above conditions, it is apparent that meta-cognition is associated with reflective practice, to which i now turn. reflective practice reflective practice, according to schon (1983, 1988), is the integration of thought and practice. it involves thinking critically and analysing actions, goals and activity. the reflective practitioner assumes the perspective of an external observer and speculates on how practice is effected (kottkamp 1990; osterman 1990; peters 1991). according to peters (1992), '[reflective practice] is a special kind of practice [...] that involves a systematic inquiry into the practice itself.' educators have become familiar with the concept of reflective practice through schsn's (1983, 1988) writings about reflective practitioners. schon's work has a historical foundation in a tradition of learning supported by dewey, lewin and piaget, each of whom advocated that learning is dependent on the integration of experience with reflection and of theory with practice. although each argued that experience is the basis for learning, each also maintained that learning cannot take place without reflection. in reflective practice, reflection is the essential part of the learning process because it results in niaking sense of or extracting meaning from the experience (osterman 1990). for schon (1988), the stage is set for reflection when 'knowing-in-action' the sort of knowledge that professionals come to depend on to perform their work spontaneously produces an unexpected outcome or surprise. this surprise can lead to one of two kinds of reflection: reflection on action, which occurs either following or by interrupting the activity, or reflection in action, which occurs during (without interrupting) the activity by thinking about how to re-shape the activity while it is under way (eric 1992). kottkamp (1990) uses the terms offline and online to distinguish between reflection-on-action and reflection-in-action. reflection-on-action takes place after the activity (that is, offline), when full attention can be given to analysis without the necessity for immediate action and when there are opportunities for the professional to receive assistance from others. reflectionin-action, which occurs during the event, may be more effective in improving practice. conclusion this paper suggests a theoretical framework for auto-monitoring. without some form of warrant to concept mapping, the development of applications will be driven by available 45 ray mcaleese a theoretical view on concept mapping technology, or what can be done rather than what is useful. the central argument is that a process entitled auto-monitoring exists which is best seen as a learning arena. in this arena, learners can be aware of and manipulate learning activities as if they were a form of lego or meccano. the outcome of such operations is called a concept map. maps are not the purpose, but rather an artefact of a process by which learning occurs. acknowledgements the thinking reported in this paper was supported by the courseware for learning study skills (class) project as part of the tltp (i) initiative. partners in this work are the university of strathclyde, glasgow caledonian university, and glasgow university with heriot-watt university. i am particularly indebted to cathy gunn in icbl (institute for computer based learning, heriot-watt university) and to my other colleagues on the project, geir granum and mike kibby, for helping me think about auto-monitoring and concept maps. the class project has produced a pc-windows concept mapping tool. the ideas in this paper are not necessarily those of the project team. references arzi, h.j. and west, l.h.t. (1986), 'ausubel revisited: more insights from a seemingly exhausted source', paper presented at the aera meeting, san francisco. bogden, c.a.b. (1977), the use of concept mapping for instructional design, msc. thesis, cornell university. brookfield, s.d. (1988), developing critical thinkers, san francisco, jossey-bass. brown, j.s., collins, a. and duguid, p. (1989), 'situated cognition and the culture of learning', educational researcher, 18, 1, 32-42. buzan, t. (1993), the mind map book: radiant thinking the major evolution in human thought, london, bbc publications. carroll, j.b. (1989), 'the carroll model a 25 year retrospective and prospective view', educational researcher, 18, 1, 26-31. cervero, r.m. (1988), effective continuing education for professionals, san francisco, jossey-bass. cervero, r.m. (1989), 'becoming more effective in everyday practice' in quigley, b.a. (ed), fulfilling the promise of adult and continuing education new directions for continuing education, no. 44, san francisco, jossey-bass. clancey, w.j. (1993), 'guidon-manage revisited: a socio-technical systems approach', journal of artificial intelligence in education, 4, 1, 5-34. de bono, e. (1993), water logic: the alternative to i am right, you are wrong, london, viking press. deikhoff, g.m. (1982), 'cognitive maps as a way of presenting the dimensions of comparison within the history of psychology', teaching of psychology, 9, 115-16. dole, j.a. and niederhauser, d.s. (1990), 'students' level of commitment to their naive conceptions and their conceptual change learning from texts' in zutell, j. and mccormick, s. 46 avr-j volume 2 number 1 (eds), literacy theory and research: analysis from multiple paradigms, chicago, national reading conference inc., 303-10. eric (1992), 'reflective practice in adult education', eric digest 122 (imel, s.), ed346319 92. fairchild, k.m. (1993), 'information management using virtual reality-based visualisations', in wexelblat, a. (ed), virtual reality applications and explorations, london, academic press. festinger, l. (1957), a theory of cognitive dissonance, stanford university press. fisher, k.m., faletti, j. and quinn, c.n. (1990), exploring cognitive learning with semantic networks, crmse, san diego state university. gold, p.c. (1984), 'cognitive mapping', academic therapy, 19, 277-84. holley, c.d. and dansereau, d.f. (eds) (1988), spatial learning strategies, new york, academic press. hoppe, h.u. (1993), 'cognitive apprenticeship the emperor's new method: a problematical reaction to the debate on situated cognition and cognitive apprenticeship', journal of artificial intelligence in education, 4, 1, 49-54. howie report (1992), upper secondary education in scotland, soed/hmso. imel, s. (1989), 'teaching adults: is it different?', eric digest 82, columbus (ohio), eric clearinghouse on adult, career, and vocational education, 305-495. kommers. p. (1992), 'graph computation as an orientation device in extended and cyclic hypertext networks' in jonassen, d.j. and mandal, h. (eds), designing hypermedia for learning, nato series f: 67, berlin, springer verlag. kottkamp, r.b. (1990), 'means for facilitating reflection', education and urban society 22, 2, 182-203. lasley, t. (1989), editorial, journal of teacher education, 40, 2. learning to succeed (1993), report of the national commission on education, london, heinemann. macfarlane, a.g.j. (1993), teaching and learning in an expanding higher education system, the committee of scottish university principals. mcaleese, r. (1985), 'some problems of knowledge representation in an authoring environment: exteriorisation, anomalous state meta-cognition and self-confrontation, programmed learning and education technology, 22, 4, 299-306. mcaleese, r. (1987), 'design and authoring: a model of cognitive processes' in mathias, h. (ed), aspects of educational technology xxi, london, kogan page. mcaleese, r. (1990), 'concepts as hypertext nodes: the ability to learn while navigating through hypertext nets' in jonassen, d.j. and mandal, h. (eds), designing hypermedia for learning, nato series f: 67, berlin, springer verlag. mcaleese, r. (1992), 'the acquisition and representation of domain specific knowledge using notecards' in olivero, a., hypermedia courseware: structures of communication and intelligent help, nato asi series f:92, berlin, springer verlag. 47 ray mcaleese a theoretical view on concept mapping mezirow, j. (1990) (ed), fostering critical reflection in adulthood, san francisco, josseybass. moyse, r. (1991), 'multiple viewpoints imply knowledge negotiation', interactive learning international 7, 1, 21-38. novak, j.d. and godwin, d.b. (1984), learning how to learn, cambridge, cup. osman, m.e. and hannifin, m.j. (1992), 'meta-cognition research and theory: analysis and implications for instructional design', et r & d, 40, 2, 83-99. osterman, k.f. (1990), 'reflective practice: a new agenda for education', education and urban society, 22, 2, 133-52. pask. g. (1984), 'review of conversation theory and a protologic (or protolanguage), lp', educational communications and technology journal, 32, 1, 3-40. peters, j. (1992), 'strategies for reflective practice. professional development for educators of adults' in brockett, r. (ed), new directions for adult and continuing education, no. 51, san francisco, jossey-bass. reader w. and hammond n. (1993), computer based tools to support learning from hypertext; concept mapping tools and beyond, department of psychology, university of york (mineo). rose, a. (1992), 'framing our experience: research notes on reflective practice', adult learning, 3, 4, 5. roth, r.a. (1989), 'preparing the reflective practitioner: transforming the apprentice through the dialectic', journal of teacher education, 40, 2, 31-35. scardamalia, m. and bereiter, c. (1987), 'knowledge telling and knowledge transforming in written composition' in rosenberg, s. (ed), advances in applied linguistics, 2, 142-75. schon, d. (1983), the reflective practitioner, new york, basic books. schon, d. (1988), educating the reflective practitioner, san francisco, jossey-bass. streitz, n. and hanneman, j. (1990), 'elaborating arguments: writing, learning and reasoning in a hypertext based environment for authoring' in jonassen, d.j. and mandal, h. (eds), designing hypermedia for learning, nato series f: 67, berlin, springer verlag. wall, d., macauley, c , entwistle, d. and entwistle, n. (1991), the transition from school to higher education in scotland, centre for research on learning and instruction, university of edinburgh. editorial note ray mcaleese's paper was sent to two referees: nick hammond and david jonassen. their comments (those of david jonassen were finally written jointly with rose marra) proved sufficiently stimulating to warrant publication in their own right as replies to the ideas presented in the paper. they are therefore included here. 48 (page number not for citation purpose) 1 *corresponding author. email: andrej.flogie@z-ams.si research in learning technology 2019. © 2019 andrej flogie et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 research in learning technology vol. 27, 2019 original research article the impact of innovative learning environments on social competences of youth andrej flogie*, boris aberšek and igor pesek faculty of natural sciences and mathematics, university of maribor, maribor, slovenia (received: 14 december 2018; revised: 11 july 2019; accepted: 11 july 2019; published:14 august 2019) the exponential development of learning environments supported by information and communication technology (ict), coupled with new insights from the fields of cognitive and neuroscience and artificial intelligence (ai), is a huge challenge for both the educational system and youths and their social competences. this article defines the impact of an effective use of contemporary education technologies on young people’s engagement and their interest in collaborating with their peers and teachers on the level of interpersonal relationships. it investigates the teachers’ and parents’ perspective on youth and their cooperation when using ict in the pedagogical process. the study confirms that innovative didactic approaches to teaching, supported by ict, exert a positive influence on the collaboration between students, as well as between students and their teachers, and that youths, teachers and parents are all equally aware of that. parents and teachers have a unified position in support of the idea that an effective use of ict makes a positive contribution to collaboration between students, and thereby directly affects an increase in their social competences. ict is becoming one of the important variables that affect the development of young people’s social competences. the 2-year study was conducted in the framework of a national project. keywords: education system; ict; psychosocial influence; social competences introduction one of the problems of contemporary society is that the educational system must be able to train youths for life, equipping them with not only knowledge and different skills, but in particular teaching them how to confront everyday challenges and problems, and, in turn, how to resolve them. young people have to develop not only their cognitive competences, but also cooperativeness and social competences because these are one of the basic conditions for life-long learning and improved employability. to achieve this, flexible forms of learning have to be implemented (aberšek 2018). the characteristics of today’s generations, defined from different perspectives (sociological, technological, psychological, philosophical), and their expectations, are a new challenge for modern-day schools. the educational process must be more closely related to the individual’s needs, their personal development and the cultural environment in which they live. the complexity of all the things affecting the youth of today (the environment, technology, a large amount of immediately accessible information, mailto:andrej.flogie@z-ams.si http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2214 a. flogie et al. 2 citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 (page number not for citation purpose) the possibility of direct communication with the entire world, newest insights from the fields of cognitive and neuroscience, artificial intelligence [ai], etc.) requires a well thought-out and quicker response on behalf of the creators of school policies than it did in the past (oecd 2016), mainly because the social environment in which we live (society, technology, etc.) is changing very rapidly, and because the school of today must prepare students for occupations and social environments, which at this moment don’t even exist yet. all these changes in the social environment, in turn, require different, innovative ways of learning and teaching, to which the entire school system must be able to adapt on a paradigmatic level. we have to realise that we will not be able to achieve such great changes in the time we have by taking small steps. in the proposed learning and teaching strategy, special attention should be given to increasing the level of interest and motivation in youth. research shows that students, who are exposed to innovative didactic information and communication technology (ict)-supported teaching methods, express a less depreciative attitude towards school (carrió-pastor and skorczynska 2015; flogie and aberšek 2017; oecd 2011). social interaction is integral to the healthy psychosocial development of adolescents. ‘because healthy development during early adolescence is complex and fraught with both challenges and opportunities, both parents and schools may need to conscientiously educate early adolescents about the potential risks the problematic or maladaptive engagement with specific ict tools or applications can bring about in their lives’ (shaljan and myint 2017). the use of innovative ict by youths in the context of social competence should be given special attention because it plays an important role in the process of socialisation (li and kirkup 2007). such a model of didactical framework of incorporating social competences in innovative learning environment is schematically shown in the figure 1. main research questions and hypothesis main research questions are as follows: how can we increase the students’ social skills, their social awareness, their emotional intelligence and their social competences when we introduce individualisation, personalisation and differentiation in teaching/ learning process? and also, can we increase social competences by using ict? our main hypothesis is that with the use of innovative learning environment and appropriate teaching/learning strategies we can increase also students’ social skills. review of the phenomenon under research many authors point out the positive influence of innovative learning environments on students’ cognitive achievements. howard-jones, pickering, and diack (2007), aberšek (2013), aberšek and dolenc (2015) and mcnaughton et al. (2018) have proven the positive impact of using learning environments supported by icts and intelligent tutoring systems (see figure 2) on the development of creative thinking, critical thinking and problem-solving. according to bloom (1956) and anderson and krathwohl (2001), these are called high cognitive levels and are also in accordance with the bloom’s digital taxonomy (churches 2008). according to the pedagogical doctrine, this is a consequence of the individualisation, personalisation and differentiation offered by intelligent learning environments. the positive influence of innovative learning environments on students’ cognitive achievement is not an unexpected result. http://dx.doi.org/10.25304/rlt.v27.2214 research in learning technology citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 3 (page number not for citation purpose) such results on the side of students’ cognitive improvement are remarkable. but at the same time a new and a very relevant question arises: what happens with the students’ social skills, their social awareness, their emotional intelligence and their social competences? and also, can we increase social competences by using ict? human beings are definitely social beings, which means that their social competences are equally, if not more, important than their cognitive skills. when speaking about social competences, emotional intelligence (ei) and emotional competences, it figure 1. model of didactical framework to enhance social competences in its. figure 2. achievement distribution for students under different learning conditions. http://dx.doi.org/10.25304/rlt.v27.2214 a. flogie et al. 4 citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 (page number not for citation purpose) can easily be observed that a lot of research has been carried out in this area; however, such studies were mostly focused on people with special needs. studies in the field of autism are extremely important because this disorder is manifested by deficits in social–emotional reciprocity. this deficit greatly hinders children with autism spectrum disorder (asd) from responding appropriately and adapting themselves in various social situations. although evidence has shown that virtual reality environment is a promising tool for emotional and social adaptation skills training on asd population, there is a lack of large-scale trials with intensive evaluations to support such findings (ip et al. 2018). research in the field of schizophrenia points out impairments in self-assessment, which are common in people with schizophrenia. impairments in self-assessment of cognitive ability have been found to predict impaired functional outcome. one of the studies examined self-assessment of social cognitive ability and related it to assessments of social cognition provided by informants, to performance on tests of social cognition and to everyday outcomes. the difference between self-reported social cognition and informant ratings was used to predict everyday functioning (hajdúk et al. 2018; silberstein et al. 2018). social competences are also emphasised in various non-traditional learning approaches, such as, for example, the montessori pedagogy (miranda, marzano, and lytras 2017). many studies limit their focus only on particular problems related to social competences and emotional competences. lin and kao (2018) studied the use of eeg data in e-learning contexts; khalfallah and ben hadjslama (2015) analysed facial expression recognition; schroeder et al. (2018) focused on the influence of learners’ perceptions of virtual humans on learning transfer and bohlouli et al. (2017) were concerned with competence assessment as an expert system for human resource management, using a mathematical approach. mcnaughton et al. (2018) developed a conditional augmentation hypothesis, which reviews theoretical rationales for and, where available, research evidence about the impact of digital tools and technologies on the development of these skills under two conditions. one is where there are direct effects of access to and use of digital technologies relatively independently from other influences of the teacher. an entirely different situation can be observed in education that takes place in mainstream schools, where in the implementation of innovative learning environments studies are generally not focused on the social and emotional competences, but rather emphasise the cognitive component of the learning process. u.s. department of education, office of educational research and improvement (1996) many years ago made massive assessment of social competence, adaptive behaviors, and approaches to learning with young children. they report that children’s affect, particularly anxiety, regarding school, has been associated with attention to task, intrinsic motivation and achievement, and also affects teacher reactions and support of the student. they conclude that children’s social competence is strongly influenced by their family, which is their first socialisation experience. kühn (2017) investigates the enhancement of social competence for disadvantaged young people based on the example of the `werkschule bremen’ educational course. the results show that a basic didactical concept is feasible, while the teaching and learning environment has a huge impact on the comparability. as a core result, this study delivers a didactic model that is based on target-controlled experience-oriented learning environments on the practical side and the social information processing approach on the theoretical side. also, hing keung ma (2013) point out that social competence has a positive youth development construct. honeychurch and patrick (2018) describe a participatory online culture – clmooc – and ask how its http://dx.doi.org/10.25304/rlt.v27.2214 research in learning technology citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 5 (page number not for citation purpose) ethos of reciprocity and creative playfulness occurs and suggest that gee’s concept of an affinity space is an appropriate model for massive open online courses, and ask how this might be replicated in a higher education setting. from our own research (aberšek 2018; aberšek, borstner, and bregant 2014; aberšek and dolenc 2015) as well as from the research conducted by many other researchers (bano et al. 2018; brevik, gunnulfsen, and renzulli 2018; kaneko, kishita, and umeda 2018; kim and baylor 2006; van lehn 2011; wong, vuong, and liu 2017), it is obvious that with intensive individualisation and differentiation of the teaching/learning process (one-on-one tutoring) and introducing intelligent (smart) learning environment, a drastic decrease in social skills (social competence) of the whole student population has been observed, which means the level of social skills and awareness (in comparison to traditional schools) is moved towards lower social competences. in the analysis of self-awareness and social awareness (part of social competences), this research also took into account the guidelines of the trait emotional intelligence questionnaire teiq-sf (petrides 2009), that is, the guidelines for its short form (katz and mcclellan 1997; petrides et al. 2010). this research focuses mainly on self-awareness and social awareness as part of ei (see figure 3). social competence and/or social skills are used here as general terms defining the complex social interactions in society. social competences the quality and added value of an individuals’ competences, skills and knowledge are the foundation for creating a competitive advantage in the globalised world and, in turn, the well-being of individual societies. on a global level, increasing demographic changes are reflected in the decrease of social and economic resources and consequently in the weakening of family ties. the term competence refers to much more than just knowledge and skills. it involves the ability to meet complex demands, by drawing on and mobilising psychosocial resources (including skills and attitudes) in a particular context. in key competences for lifelong learning, the european reference framework sets out eight key competences, among them digital literacy and social and civic competences. the key competences are all considered equally important because each of them can contribute to a successful life in a knowledge society (ceu 2007, 2018). figure 3. emotional intelligence (ei) and social competences. http://dx.doi.org/10.25304/rlt.v27.2214 a. flogie et al. 6 citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 (page number not for citation purpose) social competence refers to a series of interpersonal abilities and characteristics that have an integral effect on the individual. we are hereby referring mainly to a sense of self-worth, that is, self-confidence, self-discipline and responsibility. adler defines social competences as a form of self-management skills – skills that enable us to live in harmony with ourselves and the environment. primarily this refers to maintaining a positive relationship with oneself, and consequently to interactions with other people. social competence in an individual, thus, refers to a healthy dose of self-worth and self-confidence, a sense of personal responsibility and self-discipline. in interpersonal relationships, this is expressed as attention and empathy, that is, the capacity to place oneself in another’s position, the ability to make compromises but also enter into conflicts, the idea of getting to know people, the ability to criticise, respect and tolerate, and the ability to express all of this verbally (i.e. linguistic competence) (adler 2012). social competences are thus interpreted as interactions between an individual and other people. the concept of social competence is directly linked to the idea of ei, which is shown schematically in figure 1 (aberšek 2018; goleman 2006). social competences and ei are those skills and methods that the individual uses to successfully fulfil his or her own needs within a social environment, depending on different life situations, with the purpose of achieving personal growth and progress, and of being able to empathise and share one’s emotions with others. these are skills that enable a person to live a quality life, in harmony with oneself and the surroundings. they refer to the way an individual responds and cooperates in interpersonal relationships, which occurs at three levels (rozman 2006; van der stouwe et al. 2018): • at a personal level (building one’s self-image, solving one’s own problems, expressing ideas); • at the level of relationships (negotiation, cooperation, making compromises, networking); and • at the level of society (macrosystem) as a whole (sensibility towards others, contributing to the general well-being of society). social competence and the school environment if we consider social competences from the perspective of education and school, we can say that – just like learning anything else – the ability for a successful management of social skills requires for the basic physical and emotional needs of young people to be fulfilled. social interactions are an integral part of a young person’s healthy psychosocial development (areepattamannil and myinte 2017). as these needs cannot be met fully in the home environment, that is, the family, part of this responsibility is being increasingly shifted to the school environment. one of the tasks of school, therefore, is to fill in this social gap, without which a young person cannot be as successful as he or she could otherwise be (koplow 2002). during the years of primary and secondary education, young people focus more on school and less and less on family; friends are becoming more and more important for their socialisation. emotional regulation and sociability are another two key elements that are important during a young person’s growing up. we can say that these are two important dynamic variables of every individual, which significantly affect their ability to maintain friendships, while social skills and the ability of realising and searching for common activities are important for maintaining friendships (semrud-clikeman 2007). in this kind of socialisation process, the role of modern ict, e-services and e-contents http://dx.doi.org/10.25304/rlt.v27.2214 research in learning technology citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 7 (page number not for citation purpose) is a challenging one (mainly from the point of view of modern social networks and the networks of friends within these networks). but is the e-social network really as important as the real one? it is an interesting finding that even as early as in the preschool period, the use of contemporary technology does not have a negative impact, as it was initially expected. indeed, there are no statistically significant correlations between preschool children’s engagement in ict usage and their social competence; however, what can be perceived is a link between the evaluation of the child’s social competence (by parents and educators) and their understanding of negative social situations (proekt et al. 2017). innovative didactic teaching approaches, supported by modern ict, change traditional methods and the students’ learning environment, and contribute greatly to achieving better taxonomic and cognitive competence levels than the use of traditional teaching methods (flogie, barle lakota, and aberšek 2018). however, the impact of these modern-day technologies (or the learning environments supported by them) on the social competence and ei of youth has not been researched extensively. this is why the purpose of our study was to explore the impact of an effective use of ict on young people’s engagement and their interest in cooperation and interpersonal relationships, and at the same time to analyse differences in the perception of this issue between youths, their parents and teachers. social competence and today’s youth generation social networking and instant messaging are something quite ordinary to the young people of today; applications supporting these and similar kinds of processes are constantly running in the background on their cell phones. they spend an enormous amount of time online, shaping different online identities. according to the older generation, they are careless in posting their online content, and unaware of the consequences of their actions (tapscott 2009). from a sociological point of view, we can say that this generation wants to be actively involved in everything that touches their lives in any kind of way (šverc, flogie, and razbornik 2011). their social networks are virtual, physical and even hybrid. they establish new social connections online with ease. social interactions are the foundation of a young person’s psychosocial development (areepattamannil and myinte 2017), both in the private and in the school environment. when young people feel a sense of belonging at school, their engagement is often enhanced; when they do not, behavioural problems often follow (juvonen, espinoza, and knifsend 2012). undesirable behaviour, disinterest and non-cooperation in school and a negative attitude towards school are, of course, related not only to lower learning achievements but also to lower levels of emotional well-being, dropping out of school, misbehaviour and delinquency and, last but not least, substance abuse (axon et al. 2018; kupferberg, bicks, and hasler 2016; oecd 2013). ‘moreover, experts tend to alarm that digital technologies influence negatively on cognitive development and self-control of modern children. an extreme point of view has been reflected in an elaboration of the “digital dementia” concept that combines cognition states of young ict users and similar states of brain with cognitive disabilities’ (proekt et al. 2017). however, neuroscientists’ views on this matter are divided; pirtošek, for example, claims that digital technologies do not have a negative impact on the cognitive development of young persons’ brains, nor do they influence dementia (pirtošek 2017). http://dx.doi.org/10.25304/rlt.v27.2214 a. flogie et al. 8 citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 (page number not for citation purpose) innovative pedagogy the innovative pedagogy project has developed innovative implementation curricula based on modern pedagogical paradigms and methods that support individualisation and personalisation of teaching, using ict. in the framework of the developed implementation curricula, innovative strategies, models and approaches to teaching and learning in the classroom have been tested. the methodology (the 21-step model) of implementing pedagogy 1:1 was tested in real-life classrooms, with monitoring and evaluation of all the phases of the process. within the project, special focus was given on monitoring the direct connection between contemporary teaching and learning methods, and their effect on students’ achievements and their social and psychophysical characteristics, such as motivation for learning, engagement and approaches to learning, attendance at school, confidence in their learning abilities and own efficiency, and so on. the methodology used is known as the triangulation technique (collecting information from students, parents, teachers and principals) (aberšek et al. 2017). research methodology the study was conducted in 2015–2016 as part of the pilot project innovative pedagogy 1:1 (innovative pedagogy, 2018) and is currently in the second phase with the name innovative learning environments supported with ict (2017–2022). the project is now in the implementation part. the final results of the second phase will be available at the end of 2021. initially, a pilot study was used to train the teachers, who then taught according to this new methodology (the 21-step sequence to implement innovative approaches to ict-supported teaching and learning) for 2 years (queensland government, department of education, training and arts 2018). throughout this period, the parents were able to monitor the pedagogic process indirectly, through their children’s responses, and directly, through the information received from the school authorities because innovative pedagogy assumes, among other things, an active involvement of parents (balaskant 2013) to successfully set up an effective innovative learning environment. for the evaluation of the project, a specialised instrument was developed, consisting of questionnaires for youths and their parents and teachers. some of the students’ responses have already been published in a parallel study, which showed that the proposed approach has a positive psychosocial impact on youths (their motivation for going to school was higher, and they achieved higher taxonomic levels of knowledge) (flogie 2017). previous studies (fiksl, flogie, and aberšek 2017) have confirmed a positive attitude of youths towards innovative teaching methods. the aim of the present study was to define the impact of innovative ict-supported teaching approaches on young people’s engagement and their interest in collaborating and interpersonal relationships, from the perspective of teachers and parents. these aspects are an important segment of an individual’s social competence and ei. the survey questions referring to the level of interest in collaboration were thus addressed to parents and teachers, who have kept track of the students’ progress and attitudes throughout the entire survey period. we believe that the research questions were thus answered from a different and broader perspective. http://dx.doi.org/10.25304/rlt.v27.2214 research in learning technology citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 9 (page number not for citation purpose) research sample a sample of 127 parents and 69 sixth-grade primary school teachers from five regions participated in this study. from each of the schools, an entire classroom was selected, and no further selection of students from those classes was made. the survey therefore involved these students’ parents and the teachers teaching in these classes. in most cases, the viewpoints from both parents of an individual student were unified, meaning that a single answer was provided to each of the survey questions; some parents, however, answered separately, that is, the mother and the father provided answers individually. a five-point likert scale was applied in the survey, with 1 representing ‘completely disagree’ and 5 representing ‘completely agree’. research results the key research question was whether the use of ict in classes has an impact on the level of engagement and interest in collaboration, and whether it affects the relationships between the students, as well as between students and teachers. the ibm spps program was applied for statistical analysis. the reliability of the questionnaires was tested using cronbach’s alpha. in our case, the latter is α = 0.943, which indicates a high degree of reliability. descriptive statistics for individual groups (parents and teachers) are shown in table 1. a levene’s test assessing the equality of variances showed that the variance was not homogeneous, which is why the welch test, which is more robust, was applied (field 2005). the test was used to establish whether there are statistically significant differences between the parents’ and the teachers’ viewpoints on the research question (table 2). table 1. descriptive statistics. group statistics parent/ teacher n mean std. deviation std. error mean the use of ict in classes has a positive impact on the engagement and motivation of students in school, and on learning in general. parent 127 3.72 0.983 0.087 teacher 69 3.83 0.727 0.087 the use of ict in classes has a positive impact on collaboration between students and teacher. parent 127 3.77 1.009 0.090 teacher 69 3.88 0.758 0.091 the use of ict in classes has a positive impact on collaboration between students. parent 127 3.79 1.044 0.093 teacher 69 3.78 0.802 0.097 the use of ict in classes has a positive impact on communication between students, teachers and parents. parent 127 3.69 1.158 0.103 teacher 69 3.68 0.795 0.096 the use of ict in classes has a positive impact on building a student’s self-image; students consider classes interesting and like going to school. parent 127 4.05 0.765 0.068 teacher 69 3.19 0.896 0.108 http://dx.doi.org/10.25304/rlt.v27.2214 a. flogie et al. 10 citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 (page number not for citation purpose) discussion the results of descriptive statistics presented in table 1 show that the viewpoints of parents and teachers of youths in innovative classes are fairly uniform and with a high average response value of around 4, which is equivalent to the answer ‘i agree’ in the questionnaire. parents and teachers consider that the use of ict in classes has a positive impact on the engagement and motivation of youths in school and on learning in general. they also believe that a meaningful and effective use of ict in classes has a positive impact on collaboration and communication between students themselves, as well as between students and teachers. the only question that indicates statistically significant differences between the viewpoints of parents and teachers is ‘the use of table 2. welch test. independent samples test levene’s test for equality of variances t-test for equality of means f sig. t df sig. (2 tailed) mean difference std. error difference 95% confidence interval of the difference lower upper the use of ict in classes has a positive impact on engagement and motivation of children in school, and on learning in general. −0.887 176.362 0.376 −0.110 0.124 −0.353 0.134 the use of ict in classes has a positive impact on collaboration between students and teacher. −0.879 174.652 0.381 −0.112 0.128 −0.365 0.140 the use of ict in classes has a positive impact on collaboration between students. 0.036 172.083 0.971 0.005 0.134 −0.259 0.269 the use of ict in classes has a positive impact on communication between students, teachers and parents. 0.084 183.529 0.933 0.012 0.140 −0.265 0.289 6.741 122.209 0.000 0.859 0.127 0.607 1.111 http://dx.doi.org/10.25304/rlt.v27.2214 research in learning technology citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 11 (page number not for citation purpose) ict in classes has a positive impact on building a student’s self-image; students consider classes interesting and like going to school’. this statement is on the border between a psychosocial and a social description of youths, and the statistically significant difference between views was also confirmed by the welch test. in comparison to parents, teachers consider that the positive impact of using ict in classes is lesser. if we compare this question to the national timss 2015 survey and examine the answers provided by students from innovative classes (flogie 2017), we can observe that the parents’ outlook on this question is a much more optimistic one than that of students and teachers. we can conclude that a meaningful and effective use of modern ict is of great importance to young people; it can be perceived as a new kind of an educational communication channel, which is something their teachers and parents are very well aware of. through such an approach, the engagement of youths in school is also greater, which also contributes to better relationships and an increased interest in cooperation. we can confirm that young people’s parents and teachers have a positive attitude towards a meaningful use of modern technology and innovative didactic teaching approaches that are supported by contemporary information and communication technologies. this kind of approach generates a positive climate and creates an environment that helps to increase an individual’s social competence and their ei. after 2 years of the second phase of the new project, first preliminary results are available. they are in the line with the results of this pilot study. especially important is that preliminary results again show that effective use of ict in classes has a positive impact on collaboration and communication between students themselves, as well as between students and teachers. we will be able to prepare final longitudinal study, when the final evaluation will be finished (presumably in 2022). conclusions this study confirmed that innovative didactic teaching approaches supported by modern ict have a positive impact on the engagement of youths in school and their interest in collaborating with their peers and teachers, which is the foundation for developing their social competence and ei, and a fact that is recognised by both youths themselves and their teachers and especially parents, who are the most optimistic in this whole process because they consider, from a critical distance, the contribution of a meaningful use of ict in the field of social competences and ei of young people to be even greater (in comparison to the teachers’ views). with a comprehensive innovative pedagogical approach that directly involves parents, we can set up a learning environment where technology represents an important educational communication channel, that is, an environment that makes a positive contribution to the development of social competences in youths on the level of interpersonal relationships. in any case, it should be emphasised that the great success of the first project was the reason that the ministry of education of the republic of slovenia began with the second phase of the project called innovative learning environments supported by ict acknowledging that the question of the impact of innovative education and ict on changes in the pedagogical paradigm is more topical than ever. acknowledgements the research leading to these results received funding from the following project: innovative pedagogy 1:1 in the light of the 21st century under grant agreement number http://dx.doi.org/10.25304/rlt.v27.2214 a. flogie et al. 12 citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 (page number not for citation purpose) c 3330-13-319003. the authors would like to thank the slovenian ministry of education, science and sports, and the european social fund. references aberšek, b. (2013) ‘cogito ergo sum homomachine’, journal of baltic science education, vol. 12, no. 3, pp. 268–270. available at: http://www.scientiasocialis.lt/jbse/view/biblio/ year/2013/volume/12/issue/3 aberšek, b. (2018) problem-based learning and proprioception, cambridge scholars publishing, newcastle upon tyne. aberšek, b., borstner, b. & bregant, j. (2014) virtual teacher: cognitive approach to e-learning material, cambridge scholars publishing, newcastle upon tyne. aberšek, b. & dolenc, k. (2015) ‘tech8 intelligent and adaptive e-learning system’, computers & education, vol. 82, pp. 354–365. doi: 10.1016/j.compedu.2014.12.010. aberšek, b., et al., (2017) cognitive science in education and alternative teaching strategies, cambridge scholars, newcastle upon tyne. adler, e. (2012) schlüsselfaktorsozialkompetenz: was unsallenfehlt und wirnochlernenkönnen, econ, berlin. anderson, l. w. & krathwohl, d. r. (eds) (2001) a taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives, allyn and bacon, boston, ma. areepattamannil, s. & myinte, s. (2017) ‘early adolescents’ use of information and communication technologies (icts) for social communication in 20 countries: examining the roles of ict-related behavioral and motivational characteristics’, computers in human behavior, vol. 73, pp. 263–272. doi: 10.1016/j.chb.2017.03.058. axon, s., et al., (2018) ‘the human factor: classification of european community-based behaviour change initiatives’, journal of cleaner production, vol. 182, pp. 567–586. doi: 10.1016/j.jclepro.2018.01.232. balaskant, a. (2013) introducing tablets in schools, european schoolnet, brussels. bano, m., et al., (2018) ‘mobile learning for science and mathematics school education: a systematic review of empirical evidence’, computers & education, vol. 121, pp. 30–58. doi: 10.1016/j.compedu.2018.02.006. bloom, b. s. (1956) taxonomy of educational objectives, allyn and bacon, boston, ma. bohlouli, m., et al., (2017) ‘competence assessment as an expert system for human resource management: a mathematical approach’, journal of expert systems with applications, vol. 70, pp. 83–102. doi: 10.1016/j.eswa.2016.10.046. brevik, l. m., gunnulfsen, a. e. & renzulli, j. s. (2018) ‘student teachers’ practice and experience with differentiated instruction for students with higher learning potential’, teaching and teacher education, vol. 71, pp. 34–45. doi: 10.1016/j.tate.2017.12.003. carrió-pastor, m. l. & skorczynska, h. (2015) ‘collaborative learning and communication technologies in teaching business english’, procedia – social and behavioral sciences, vol. 178, pp. 32–37. doi: 10.1016/j.sbspro.2015.03.142. ceu. (2007) key competences for lifelong learning, council eu, brussels. ceu. (2018) council recommendation on key competences for lifelong learning, council eu, brussels. churches, a. (2008) bloom’s digital taxonomy, [online] available at: http://linksandthings. pbworks.com/f/churches_2008_digitalbloomstaxonomyguide.pdf field, a. (2005) discovering statistic using spss, sage, london. fiksl, m., flogie, a. &aberšek, b. (2017) ‘innovative teaching/learning methods to improve science, technology and engineering classroom climate and interest’, journal of baltic science education, vol. 16, pp. 1009–1019. available at: http://www.scientiasocialis. lt/jbse/?q= node/628 http://dx.doi.org/10.25304/rlt.v27.2214 http://www.scientiasocialis.lt/jbse/view/biblio/year/2013/volume/12/issue/3 http://www.scientiasocialis.lt/jbse/view/biblio/year/2013/volume/12/issue/3 http://linksandthings.pbworks.com/f/churches_2008_digitalbloomstaxonomyguide.pdf http://linksandthings.pbworks.com/f/churches_2008_digitalbloomstaxonomyguide.pdf http://www.scientiasocialis.lt/jbse/?q=​node/628 http://www.scientiasocialis.lt/jbse/?q=​node/628 research in learning technology citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 13 (page number not for citation purpose) flogie, a. (2017) vplivinovativnegaizobraževanja in informacijsko-komunikacijsketehnologijenaspremembepedagoškeparadigme [influence of innovative education and ict to changinfodf pedagogical paradigm], phd. thesis, univerza v mariboru, maribor. flogie, a. & aberšek, b. (2017) ‘transdisciplinary approach of science, technology, engineering and mathematics education’, journal of baltic science education, vol. 14, pp. 779–790. flogie, a., barle lakota, a. & aberšek, b. (2018) ‘the psychosocial and cognitive influence of ict on competences od stem students’, journal of baltic science education, vol. 17, pp. 267–276. goleman, d. (2006) emotional intelligence, bantam dell, new york. hajdúk, m., et al., (2018) ‘social cognitive impairments in individuals with schizophrenia vary in severity’, journal of psychiatric research, vol. 104, pp. 65–71. doi: 10.1016/j. jpsychires.2018.06.017. hing keung ma. (2013) ‘social competence as a positive youth development construct: a conceptual review’, the scientific world journal, vol. 2012, no. 3. doi: 10.1100/2012/287472. honeychurch, s. & patrick, f. (2018) ‘massive open online courses as affinity spaces for connected learning: exploring effective learning interactions in one massive online community’, research in comparative and international education, vol. 13, no. 1, pp. 117–134. doi: 10.1177/1745499918768112. howard-jones, p., pickering, s. & diack, a. (2007) perception of the role of neuroscience in education, university of bristol, bristol. ip, h. s. p., et al., (2018) ‘enhance emotional and social adaptation skills for children with autism spectrum disorder: a virtual reality enabled approach’, computers & education, vol. 117, pp. 1–15. doi: 10.1016/j.compedu.2017.09.010. juvonen, j., espinoza, g. & knifsend, c. (2012) the role of peer relationships in student academic and extracurricular engagement, handbook of student engagement, department of psychology, university of california, los angeles, ca. kaneko, k., kishita, y. & umeda, y. (2018) ‘toward developing a design method of personalization: proposal of a personalization procedure’, procedia cirp, vol. 69, pp. 740–745. doi: 10.1016/j.procir.2017.11.134. katz, l. g. & mcclellan, d. e. (1997) fostering children’s social competence: the teacher’s role, national association for the education of young children, washington, dc. khalfallah, j. & ben hadjslama, j. (2015) ‘facial expression recognition for intelligent tutoring systems in remote laboratories platform’, procedia computer science, vol. 73, pp. 274–281. doi: 10.1016/j.procs.2015.12.030. kim, y. & baylor, a. l. (2006) ‘a social-cognitive framework for pedagogical agents as learning companions’, educational technology research and development, vol. 54, no. 6, pp. 569–596. doi: 10.1007/s11423-006-0637-3. koplow, l. (2002) creating schools that heal, teachers college press, columbia university, new york. kühn, i. k. (2017) ‘enhancing social competence for disadvantaged youth in pre-vocational education: model development through design-based research’, international journal for research in vocational education and training, vol. 4, no. 4, pp. 346–368, doi: 10.13152/ ijrvet.4.4.3. kupferberg, a., bicks, l. & hasler, g. (2016) ‘social functioning in major depressive disorder’, neuroscience & biobehavioral reviews, vol. 69, pp. 313–332. doi: 10.1016/j. neubiorev.2016.07.002. li, n. & kirkup, g. (2007) ‘gender and cultural differences in internet use: a study of china and the uk’, computers & education, vol. 48, pp. 301–317. doi: 10.1016/j.compedu. 2005.01.007. lin, f. r. & kao, c. m. (2018) ‘mental effort detection using eeg data in e-learning contexts’, computers & education, vol. 122, pp. 63–79. doi: 10.1016/j.compedu.2018.03.020. mcnaughton, s., et al., (2018) ‘how digital environments in schools might be used to boost social skills: developing a conditional augmentation hypothesis’, computers & education, vol. 126, pp. 311–323. doi: 10.1016/j.compedu.2018.07.018. http://dx.doi.org/10.25304/rlt.v27.2214 a. flogie et al. 14 citation: research in learning technology 2019, 27: 2214 http://dx.doi.org/10.25304/rlt.v27.2214 (page number not for citation purpose) meisels, s. j., atkins-burnet, s. & nicholson, j. (1996) assessment of social competence, adaptive behaviors, and approaches to learning with young children, u. s. department of education, office of educational research and improvement, washington, dc, [online] available at: https://nces.ed.gov/pubs96/9618.pdf miranda, s., marzano, a. & lytras, m. d. (2017) ‘a research initiative on the construction of innovative environments for teaching and learning. montessori and munari based psycho-pedagogical insights in computers and human behavior for the new school’, computer in human behavior, vol. 66, pp. 282–290. doi: 10.1016/j.chb.2016.09.056. oecd. (2011) oecd guide to measuring the information society 2011, oecd publishing, paris. oecd. (2013) education at a glance 2013: oecd indicators, oecd publishing, paris. doi: 10.1787/eag-2013-en. oecd. (2016) pisa 2015 results (volume ii): policies and practices for successful schools, pisa, oecd publishing, paris. doi: 10.1787/9789264267510-en. petrides, k. v. (2009) technical manual for the trait emotional intelligence questionnaires (teique), london psychometric laboratory, london. petrides, k. v., et al., (2010) ‘relationships between trait emotional intelligence and the big five in the netherlands’, personality and individual differences, vol. 48, pp. 906–910. doi: 10.1016/j.paid.2010.02.019. pirtošek, z. (2017) ‘možgani, nevrotehnologija in etika [brain, neuro technology and ethics]’, analiza; časopis za kritičnomisel, vol. 21, pp. 91–109. proekt, y., et al., (2017) ‘developing social competence of preschoolers in digital era: gender dimension’, in communications in computer and information science, eds v. d. alexandrov et al., vol. 745, pp. 87–101, springer international publishing, cham. queensland government, department of education, training and arts. (2018) 21 steps to 21st century 1-to-1 success, queensland government, brisbane. rozman, u. (2006) trening socialnih veščin [training of the social skills], educa, nova gorica. schroeder, n. l., et al., (2018) ‘the influence of learners’ perceptions of virtual humans on learning transfer’, computers and education, vol. 126, pp. 170–182. doi: 10.1016/j. compedu.2018.07.005. semrud-clikeman, m. (2007) social competence in children, springer science+business media, new york. shaljan, a. & myint, s. (2017) ‘early adolescents’ use of information and communication technologies (icts) for social communication in 20 countries: examining the roles of ictrelated behavioral and motivational characteristics’, computers in human behavior, vol. 73, pp. 263–272. doi: 10.1016/j.chb.2017.03.058. silberstein, j. m., et al., (2018) ‘self-assessment of social cognitive ability in schizophrenia: association with social cognitive test performance, informant assessments of social cognitive ability, and everyday outcomes’, schizophrenia research, vol. 199, pp. 75–82. doi: 10.1016/j.schres.2018.04.015. šverc, a., flogie, a. & razbornik, i. (2011) net generation calls for different approaches towards the education, youth in focus of world changes, inštitut za razvoj družbene odgovornosti, maribor. tapscott, d. (2009) growing up digital, mcgraw-hill, new york. van der stouwe, t., et al., (2018) ‘social skills training (sst) effects on social information processing skills in justice-involved adolescents: affective empathy as predictor or moderator’, children and youth services review, vol. 90, pp. 1–7. doi: 10.1016/j.childyouth.2018.05.006. van lehn, k. (2011) ‘the relative effectiveness of human tutoring, intelligent tutoring systems,  and other tutoring systems’, educational psychologist, vol. 46, no. 4, pp. 197–221. doi: 10.1080/00461520.2011.611369. wong, p. c. m., vuong, l. c. & liu, k. (2017) ‘personalized learning: from neurogenetics of behaviors to designing optimal language training’, neuropsychologia, vol. 98, pp. 192–200. doi: 10.1016/j.neuropsychologia.2016.10.002. http://dx.doi.org/10.25304/rlt.v27.2214 https://nces.ed.gov/pubs96/9618.pdf (page number not for citation purpose) 1 *corresponding author. email: daniel.ebbert@uni-muenster.de research in learning technology 2020. © 2020 d. ebbert and s. dutke. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 research in learning technology vol. 28, 2020 original research article patterns in students’ usage of lecture recordings: a cluster analysis of self-report data daniel ebbert* and stephan dutke institute for psychology in education, university of münster, münster, germany (received: 12 april 2019; revised: 15 november 2019; accepted: 18 november 2019; published: 9 january 2020) students’ usage of lecture recordings can be characterised by usage frequency, repetitiveness and selectivity in watching, lecture attendance, and social context and location in which students watch the lecture recordings. at the university of münster (germany), the lecture recording service was evaluated over three semesters. the data were combined and used for a cluster analysis with the aim of being able to describe the students’ distinct usage patterns. the cluster analysis was performed using partitioning around medoids with gower distance. five clusters of students were identified, which differed mainly on the amount of lecture recordings watched, whether the lecture recordings were watched completely or partially, whether the recordings were watched once or multiple times, and the number of lectures the students missed. the five clusters are interpreted as representing different ways of utilising lecture recordings. the clustering provides a basis for investigating the usage of lecture recordings in the context of different approaches to learning and learning strategies. keywords: lecture recording; lecture capture; approaches to learning; educational technology evaluation; self-regulated learning introduction lecture recording at universities has become more common (edwards and clinton 2018). however, whether lectures should be recorded or not is a controversial topic. many students want lectures to be recorded (dolch and zawacki-richter 2018) and report that having access to lecture recordings is useful for them (bacro, gebregziabher, and fitzharris 2010). however, lecturers have many concerns about lecture recordings (newland 2017), for example, that lecture attendance would decrease with the availability of lecture recordings (draper, gibbon, and thomas 2018). triggered by these controversial positions, many studies have been conducted about the effects of lecture recordings on student attendance and academic performance (see o’callaghan et al. 2017 for a review of the recent literature). these studies yielded mixed results, both with regard to the effects on lecture attendance and academic performance. for example, walls et al. (2010) reported no mailto:daniel.ebbert@uni-muenster.de http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v28.2258 d. ebbert and s. dutke 2 citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 (page number not for citation purpose) correlation between using lecture podcasts and attendance, whereas edwards and clinton (2018) found a decrease in attendance when lecture recordings were available. dommeyer (2017) reported that students who had access to lecture recordings had lower absentee rates compared to students who did not have access to such recordings. when the outcome is decrease in attendance, the question remains to what extent other learning resources contributed to the absences. traphagan, kucsera, and kishi (2010), for example, found that the availability of the powerpoint presentations of a lecture series had a greater negative impact on attendance than the availability of lecture recordings. thus, whether and under which conditions the use of lecture recordings reduces lecture attendance remained an open question. similarly, studies investigating the relationship between the use of lecture recordings and academic performance have also yielded mixed results. some studies demonstrated that students’ self-reported (e.g., gosper et al. 2008) as well as their objectively measured academic performance increased when lecture recordings were available (e.g., whitley-grassi 2017). in contrast, williams et al. (2016) found no learning gain even for students who attended the lectures and used the lecture recordings. actually, the effects of using lecture recordings on academic performance and attendance might be even more heterogeneous, as some studies demonstrated interaction effects. kalnikaitė and whittaker (2010), for example, found that students who watched the lecture recordings instead of attending the lecture performed worse than those who attended the live lectures. traphagan, kucsera, and kishi (2010), however, stated that watching the recording of a lecture might nullify the negative effects occurring when students are unable to attend the lecture in the lecture hall. however, nordmann et al. (2018) found that only high-performing students would be able to compensate for the negative effects of missing a lecture by watching its recording. weaker students only benefit from the supplementary use of lecture recordings (nordmann et al. 2018). summarised, evidence on how the availability of lecture recordings affects students’ study behaviour and academic performance is largely unclear. we pursue the hypothesis that these diverse and partially inconsistent findings might be due to individual differences in how the lecture recordings are used and how they are embedded in the students’ learning behaviour. imagine, for example, a sample of students of which 80% simply view the existence of lecture recordings as a justification for not visiting the lecture, whereas 20% of them use the lecture recordings for deepening their understanding of the contents or for preparing exams. the effects on attendance and performance would be probably different if a sample is studied of which 80% see the recordings as a valuable resource for self-regulated learning and only 20% use the recording as a surrogate of the face-to-face contact in the lecture. heterogeneity of the studied samples might explain diverging results. this assumption is corroborated by studies showing substantial variations in how students use lecture recordings. there is, for example, substantial variance in the number of lecture recordings watched (elliott and neal 2016). some students watched only a few of the available recordings, while others watched most, if not all (zupancic and horz 2002). furthermore, some students watched the same lecture recording multiple times (de boer and tolboom 2008), and a considerable variance has been found in the length of the lecture watching session (mark and vrijmoed 2017) and in terms of whether the recording was watched completely or in parts (gosper et al. 2008). introducing lecture recordings increases the flexibility of students’ study routines because the time, location and social setting in which a lecture recording is watched can be freely chosen http://dx.doi.org/10.25304/rlt.v28.2258 research in learning technology citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 3 (page number not for citation purpose) (woo et al. 2008). thus, investigating the extent that students utilise these additional degrees of freedom can help us to understand under which conditions students might benefit from the availability of lecture recordings. therefore, the aim of this study is to identify clusters of students who use lecture recordings in different ways. our understanding of diverging and contradicting empirical findings would benefit, if we were able to distinguish different strategies in working with lecture recordings, as these different strategies might yield different effects on attendance and performance. based on the studies cited previously, use of lecture recordings can be characterised by assessing the variability of usage among students in the following six study behaviour variables: • frequency of usage: how many of the available lecture recordings of a lecture series are watched during the semester? • repetitive watching: are lecture recordings or parts of the recordings watched multiple times? • selectivity in watching: are the lecture recordings watched completely or in parts? • lecture attendance: how many of the lectures are attended in the lecture hall and how many are missed because lecture recordings are available? • social context: are the lecture recordings watched alone or together with other students? • location: where are lecture recordings watched? in this study, these variables served as input variables in a cluster analysis to identify different strategies in using the lecture recordings. methods participants in this survey, n = 1079 students enrolled at the university of münster (germany) answered questions on the lecture recording service (mean age: 22.32 years, sd = 6.38 years; 54.6% female, 42.9% male, 2.4% no gender was indicated). each student attended at least one of n = 47 lecture series which were recorded by the university’s central lecture recording service. the investigated lecture series referred to nine academic domains: economics (24 lecture series), law (9), theology (5), psychology (3), biology (1), history (1), sport sciences (1), pedagogy (1), political sciences (1), and philosophy (1). table 1 shows the distribution of participants and lecture series across the three semesters of the evaluation period. in the following analyses, we included only those students who reported that they used the lecture recordings (right column in table 1). the data re-analysis was consistent with the ethical guidelines of the american psychological association (american psychological association 2017) and the ethical meta code of the european federation of psychologists’ associations ( european federation of psychologists’ associations 2005). procedure the lecture capture service was evaluated by the local service operator and his staff, including the first author. the data used in this study were gathered during http://dx.doi.org/10.25304/rlt.v28.2258 d. ebbert and s. dutke 4 citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 (page number not for citation purpose) this initial evaluation phase. the evaluations were conducted in the winter semester 2016/2017, summer semester 2017 and winter semester 2017/2018. at the end of each semester, all students who had access to the lecture recordings received a questionnaire via the learning platform with the request to participate in the evaluation. students participated voluntarily. the online questionnaire was completed anonymously. therefore, it was impossible to reconstruct who did or did not participate. materials the online questionnaire (26 items in total) encompassed three types of items. the first type of items referred to how the participants used the lecture recordings. these variables except items with open response format were used as input variables for the cluster analysis (the corresponding items and response formats are described in table 2). the second set of items, which were used to gather additional information about the participants (age and gender), their experience with lecture recordings, their purposes of using lecture recordings and how they evaluated the lecture capture service, served as dependent variables. the identified clusters were analysed for differences on these variables. this analysis was expected to provide initial indications to which other variables usage patterns of lecture recordings may be related. the third set of items referred to technical aspects of using lecture recordings (how to access them, adequacy of the video format, quality of sound and picture). these variables were of specific interest for improving the service. however, the data were not included in the following analyses. data analyses to identify the patterns of using lecture recordings, a cluster analysis was conducted using the variables listed in table 2 as input variables. the responses to these items provided information about how students utilised the recordings in their learning behaviour. in a cluster analysis, a set of data points, in this study the students, are grouped by similarity in the input variables with the aim of identifying groups that are internally as homogeneous as possible but differ from each other as much as possible. the clustering algorithm we used in this study (partitioning around medoids; pam; kaufman and rousseeuw 1990) aims at grouping the data points by similarity to other data points chosen as the centres of their clusters. similarity is determined by a distance measure (gower distance; gower 1971), which is suitable to determine the distance between data points that contain categorical as well as numerical values. pam requires the number of clusters k to be manually determined. we determined k based on the average silhouette table 1. number of lecture series and number of students by semester. semester lecture series students who completed the questionnaire students who completed the questionnaire and used the lecture recordings winter term 2016/17 6 198 188 summer term 2017 13 370 344 winter term 2017/18 28 511 491 sum 47 1079 1023 http://dx.doi.org/10.25304/rlt.v28.2258 research in learning technology citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 5 (page number not for citation purpose) coefficient (rousseeuw 1987), which is a measure of consistency of clustering that ranges from −1 to +1 with higher values indicating a higher quality of clustering (more internally consistent clusters). we calculated the average silhouette coefficients between k = 2 and k = 10. the average silhouette coefficient was highest at k = 3 and k = 5, with an average silhouette coefficient of 0.48 for both solutions. the five-cluster solution was more differentiated because one of the clusters represented students who missed most of the live lectures – a phenomenon previously reported in the literature (e.g., o’callaghan et al. 2017). therefore, five clusters were extracted. following the identification of the clusters, we tested whether the differences between the clusters were significant on the input variables as well as on the dependent variables. the kruskal–wallis rank sum test (kruskal and wallis 1952) was performed for continuous variables, and for significant test results, the conover–iman test of multiple comparisons using rank sums (conover and iman 1979) was performed as a post-hoc test to determine which pairwise comparisons were significant. for the categorical variables, pearson’s chi-square test for count data with monte carlo simulation (hope 1968) was employed (8,000 iterations as recommended by mundform et al. 2011). post-hoc analyses for significant results were conducted based on the standardised residuals of the pearson’s chi-square test (beasley and schumacker 1995). for post-hoc analyses, bonferroni corrections (dunn 1961) were applied to the p values. table 2. items describing how students used the lecture recordings (input variables for the cluster analysis). no. question possible answers coding 1 how many of the lecture recordings, of the lecture you attended, did you watch? no answer a few about half most almost all missing 1 2 3 4 2 did you watch some lecture recordings multiple times? no answer yes no 1 2 3 3 how did you use the lecture recordings? no answer i usually watched the lecture recordings completely i usually only watch parts of a lecture recording 1 2 3 4 how many lectures did you not attend because you could expect to be able to watch the lecture recording? no answer none a fewabout half most almost all missing 1 2 3 4 5 5 how did you use the lecture recordings? no answer always alone mostly alone mostly together with other people always together with other people missing 1 2 3 4 6 where did you usually watch the lecture recordings? no answer on the go at home in the university 1 2 3 4 http://dx.doi.org/10.25304/rlt.v28.2258 d. ebbert and s. dutke 6 citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 (page number not for citation purpose) results in the following, the extracted clusters are described in terms of the differences revealed on the input variables listed in table 2. significant differences were found on the input variables 1 to 5 but not on input variable 6. thus, all clusters have in common that lecture recordings were mainly watched at home. table 3 shows the statistics of the input variables for each cluster. cluster 1 – frequent repetition (n = 291): cluster 1 students watched most of the lecture recordings, usually completely and some of them even multiple times. although the students knew that the lecture recordings would be available, they missed only a few live lectures, but they missed more lectures than the students in the clusters 2, 3 and 4. summarised, we interpret this pattern as an attempt to repeat the content of nearly all lectures of a series. cluster 2 – selective repetition (n = 296): the students of this cluster also seem to repeat complete lectures by watching the lecture recordings, but less consistently than cluster 1. cluster 2 students watched only about half of the lecture recordings, and they watched them only once and not repeatedly. although these students knew that the lecture recordings would be available, they missed only a few live lectures. cluster 3 – frequent consultation (n = 107): students in cluster 3 watched most of the lecture recordings and some even multiple times. however, they usually watched only parts of a lecture recording. thus, we interpret this pattern as an attempt to immerse oneself into specific topics. this interpretation is consistent with the observation that students of this cluster consulted only specific passages of a lecture recording and refrained from watching them completely compared to clusters 1 and 2. in line with this idea, cluster 3 students missed only a few lectures in the lecture hall. cluster 4 – selective consultation (n = 158): cluster 4 students also watched lecture recordings only partially but less frequently than cluster 3 and usually only once and not multiple times. although they knew the lecture recordings would be available, they missed only a few live lectures. cluster 5 – increased absenteeism (n = 171): in contrast to all other clusters, these students missed most of the lectures in the lecture hall and instead watched almost all lecture recordings, usually completely but only once. we interpret this pattern as the attempt to substitute attending the live lectures by watching the recordings. table 4 shows the extent to which the cluster size varies across the semesters of the evaluation period. although the number of participants varied across the three semesters of the evaluation period, the proportions of the clusters within each semester did not, χ² (8, n = 1023) = 9.25, p = .32. in each semester, the clusters of (frequent and selective) repeaters (clusters 1 and 2) were larger than clusters 3 and 4 representing students who mainly used the lecture recordings for the deeper consideration of specific topics. across semesters, the relative sizes of the clusters varied within narrow boundaries. the largest cluster (cluster 2), for example, ranged from 25.58% (of the total semester n) to 30.85%; the smallest cluster (cluster 3) ranged from 9.01% to 11.41%. the stability of the clusters in their relative size is remarkable because in each semester, different samples of students and lecture series were studied. we further explored whether the clusters differed on additional variables describing the participants and specific aspects of their study behaviour. for means, standard deviations and frequencies of the clusters on these variables, see tables 5 and 6. these additional variables have the status of dependent variables because they were not included in identifying the clusters. http://dx.doi.org/10.25304/rlt.v28.2258 research in learning technology citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 7 (page number not for citation purpose) t ab le 3 . d if fe re n ce s b et w ee n c lu st er s o n t h e in p u t va ri ab le s. in p u t va ri ab le s c lu st er s f re q u en t re p et it io n s el ec ti ve r ep et it io n f re q u en t co n su lt at io n s el ec ti ve c o n su lt at io n in cr ea se d a b se n te ei sm p (1 ). h o w m an y o f th e le ct u re re co rd in gs , o f th e le ct u re yo u a tt en d ed , d id y o u w at ch ? m os t m = 3 .4 4 s d = 0 .9 2 a bo ut h al f m = 2 .1 1 s d = 1 .1 m os t m = 2 .6 8 s d = 1 .0 3 a bo ut h al f m = 1 .7 3 s d = 0 .9 1 a lm os t al l m = 3 .6 3 s d = 0 .6 7 < 0. 00 1¹ (2 ). d id y o u w at ch s o m e re co rd in gs m u lt ip le t im es ? y es 97 .9 % n o 95 .9 % y es 98 .1 % n o 97 .5 % n o 10 0% < 0. 00 1² (3 ). h o w d id y o u u se t h e le ct u re r ec o rd in gs ? c om pl et el y 99 .7 % c om pl et el y 10 0% p ar ti al ly 96 .3 % p ar ti al ly 98 .1 % c om pl et el y 99 .4 % < 0. 00 1³ (4 ). h ow m an y le ct u re s d id y o u no t a tt en d be ca us e yo u  co ul d ex p ec t to b e ab le t o w at ch th e le ct u re r ec o rd in g? a f ew m = 2 .2 8 s d = 1 .5 1 a f ew m = 1 .5 1 s d = 0 .6 6 a f ew m = 1 .8 7 s d = 1 .3 7 a f ew m = 1 .5 6 s d = 1 .0 5 m os t m = 4 .0 2 s d = 0 .9 6 < 0. 00 14 (5 ). h o w d id y o u u se t h e le ct u re  r ec o rd in gs ? a lw ay s al on e m = 1 .1 6 s d = 0 .3 8 a lw ay s al on e m = 1 .0 4 s d = 0 .2 1 a lw ay s al on e m = 1 .1 4 s d = 0 .3 8 a lw ay s al on e m = 1 .1 1 s d = 0 .3 4 a lw ay s al on e m = 1 .0 9 s d = 0 .2 8 < 0. 00 15 (6 ). w h er e d id y o u u su al ly w at ch t h e le ct u re re co rd in gs ? a t ho m e 93 .5 % a t ho m e 90 .9 % a t ho m e 91 .6 % a t ho m e 91 .8 % a t ho m e 90 .1 % 0. 43 86 n o te : t h e la b el i n e ac h c lu st er c o lu m n ( in i ta li cs ) re fe rs e it h er t o t h e m o st f re q u en tl y ch o se n r es p o n se o r th e ca te go ry i n w h ic h t h e m ea n w as l o ca te d . ¹ k ru sk al –w al li s ra nk s um t es t: h ( 4, n = 1 01 2) = 3 86 .2 6; ² p ea rs on ’s c hi -s qu ar ed t es t fo r co un t da ta : χ² ( 8, n = 1 02 3) = 1 00 8. 5; 3 p ea rs on ’s c hi -s qu ar ed t es t fo r co un t da ta : χ² ( 8, n = 1 02 3) = 1 01 6. 4; 4 k ru sk al –w al li s ra nk s um t es t: h ( 4, n = 9 74 ) = 32 9. 55 ; 5 k ru sk al –w al li s ra nk s um t es t: h ( 4, n = 1 02 0) = 2 4. 39 8; 6 p ea rs on ’s c hi -s qu ar ed t es t fo r co un t da ta : χ² ( 12 , n = 1 02 3) = 1 2. 09 9. http://dx.doi.org/10.25304/rlt.v28.2258 d. ebbert and s. dutke 8 citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 (page number not for citation purpose) no significant gender differences were found among the five clusters. all clusters contained slightly more women than men. the self-reported experience in using lecture recordings also did not differ between the clusters. between 66.9% and 75.5% of the students per cluster reported to have no prior experience with lecture recordings. although the mean age of the clusters varied only between 21.64 and 22.97 years, cluster 1 was significantly older than cluster 2 (p = 0.006) and cluster 4 (p = 0.002), and the latter was also significantly different to cluster 5 (p = 0.027). a significant difference between the clusters was also found on the item that addressed whether the lecture recordings were used as the only resource or in addition to attending the lectures. in clusters 1 and 5, more students than expected stated to have used the lecture recordings solely (cluster 1: p < 0.001; cluster 5: p < 0.001), whereas more students than expected stated to have used the lecture recordings parallel to the lectures in clusters 2 (p < 0.001) and cluster 4 (p < 0.001). this result is consistent with the interpretation that clusters 2 and 4 used the lecture recordings selectively for repeating or revisiting lecture contents, which makes sense when students attended the lecture and were familiar with its contents. the item that assessed the extent that students were satisfied with the lecture capture service at their university was answered positively and the cluster means ranged between satisfied and very satisfied. nevertheless, cluster 4 was significantly less satisfied than clusters 1 (p < 0.001), 2 (p = 0.046) and 5 (p < 0.001), which might be related to the observation that cluster 4 used the recording less intensively than all other clusters. table 6 presents three responses given by students from the five clusters to the item that assessed the purpose for which the students used the lecture recordings. the distribution of answers varied substantially across the clusters on each variable. the lecture recordings were used to review the lectures more often than expected by the students in cluster 1 (p = 0.023) and less often than expected in cluster 4 (p = 0.031). as a reference the lecture recordings were used more frequently than expected in clusters 1, 3 and 5 (p = 0.004, p < 0.001, p = 0.047) and less frequently than expected in cluster 2 (p < 0.001). using the lecture recordings to prepare for the exam was reported to have occurred more often than expected by the students in clusters 1 (p < 0.001) and 3 (p = 0.020) and less often than expected in clusters 2 (p < 0.001) and cluster 4 (p = 0.048). discussion the most important result is that we found a clear and meaningful cluster structure. the participants could be clearly grouped according to their usage of the lecture table 4. proportion of the clusters by semester. clusters sum frequent repetition selective repetition frequent consultation selective consultation increased absenteeism winter term 2016/2017, n (%) 54 (28.72) 58 (30.85) 20 (10.64) 21 (11.17) 35 (18.62) 188 summer term 2017, n (%) 108 (31.40) 88 (25.58) 31 (9.01) 62 (18.02) 55 (15.99) 344 winter term 2017/2018, n (%) 129 (26.27) 150 (30.55) 56 (11.41) 75 (15.27) 81 (16.50) 491 sum, n 291 296 107 158 171 1023 http://dx.doi.org/10.25304/rlt.v28.2258 research in learning technology citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 9 (page number not for citation purpose) t ab le 5 . d if fe re n ce s b et w ee n c lu st er s o n v ar ia b le s ch ar ac te ri si n g th e st u d en ts a n d t h ei r st u d y b eh av io u r. v ar ia b le s c lu st er s f re q u en t re p et it io n s el ec ti ve r ep et it io n f re q u en t co n su lt at io n s el ec ti ve c o n su lt at io n in cr ea se d a b se n te ei sm p w h at i s yo u r ge n d er ? f em al e 54 .0 % m al e 43 .3 % f em al e 54 .6 % m al e 43 .6 % f em al e 48 .6 % m al e 46 .7 % f em al e 58 .2 % m al e 41 .1 % f em al e 56 .3 % m al e 41 .9 % 0. 50 2¹ h o w o ld a re y o u ? m = 2 2. 97 s d = 7 .5 9 m = 2 1. 89 s d = 5 .8 7 m = 2 2. 15 s d = 5 .0 9 m = 2 1. 64 s d = 5 .8 9 m = 2 1. 99 s d = 3 .0 0 < 0. 00 1² d id y o u u se t h e le ct u re r ec o rd in gs p ro vi d ed f o r th e le ct u re t h at y o u at te n d ed ? p ar al le l to t he l ec tu re s 57 .7 % s ol el y us ed t he l ec tu re re co rd in gs 42 .3 % p ar al le l to t he l ec tu re s 87 .2 % s ol el y us ed t he l ec tu re re co rd in gs 12 .8 % p ar al le l to t he l ec tu re s 79 .4 % s ol el y us ed t he l ec tu re re co rd in gs 20 .6 % p ar al le l to t he l ec tu re s 84 .8 % s ol el y us ed t he l ec tu re re co rd in gs 15 .2 % p ar al le l to t he l ec tu re s 26 .9 % s ol el y us ed t he l ec tu re re co rd in gs 73 .1 % < 0. 00 1³ d id y o u h av e an y ex p er ie n ce u si n g le ct u re r ec o rd in gs b ef o re t h is s em es te r? y es 32 .1 % n o 66 .9 % y es 24 .1 % n o 75 .5 % y es 28 .0 % n o 72 .0 % y es 26 .1 % n o 73 .2 % y es 29 .8 % n o 70 .2 % 0. 38 24 h o w s at is fi ed a re y o u , al l in a ll , w it h t h e le ct u re r ec o rd in gs ? m = 4 .5 3 s d = 0 .7 2 m = 4 .3 9 s d = 0 .7 8 m = 4 .3 5 s d = 0 .7 8 m = 4 .1 9 s d = 0 .8 6 m = 4 .5 3 s d = 0 .6 9 < 0. 00 15 n o te : t h e la b el i n e ac h c el l (i n i ta li cs ) re fe rs t o e it h er t h e m o st f re q u en tl y ch o se n r es p o n se o r th e ca te go ry i n w h ic h t h e m ea n w as l o ca te d . ¹ p ea rs on ’s c hi -s qu ar ed t es t fo r co un t da ta : χ² ( 8, n = 1 01 2) = 7 .3 90 6; ² k ru sk al –w al li s ra nk s um t es t: h ( 4, n = 1 00 2) = 2 1. 60 7; 3 p ea rs on ’s c hi -s qu ar ed t es t fo r co un t da ta : χ² ( 4, n = 1 02 3) = 2 22 ; 4 p ea rs on ’s c hi -s qu ar ed t es t fo r co un t da ta : χ² ( 8, n = 1 01 9) = 8. 61 38 ; 5 k ru sk al –w al li s ra nk s um t es t: h ( 4, n = 1 01 5) = 2 7. 83 8. http://dx.doi.org/10.25304/rlt.v28.2258 d. ebbert and s. dutke 10 citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 (page number not for citation purpose) recordings, and the proportions of the clusters remained stable across the three semesters, although this comparison involved different samples and different lecture series. thus, we hypothesise that this cluster structure represents strategies in using lecture recordings that have some potential for generalisation. the five clusters we identified can be interpreted as a hierarchical structure: some students used the recordings in lieu of attending the lectures (cluster 5), while others used them to supplement and enrich the lecture (clusters 1–4). the students in the latter clusters either aimed at repeating the content presented in the recorded lecture (clusters 1 and 2) or used the recordings for deepening their knowledge concerning specific topics (clusters 3 and 4). the repetition clusters have in common that they usually watched complete lecture recordings although the frequent repeaters (cluster 1) watched lecture recordings multiple times, whereas the selective repeaters did not (cluster 2). the consultation clusters have in common that they usually watched only parts of the lecture recordings. the main difference between these clusters is that the frequent consulters (cluster 3) watched lecture recordings multiple times, whereas selective consulters (cluster 4) did not. furthermore, there is a striking similarity between the frequent (clusters 1 and 3) and the selective variants (clusters 2 and 4). students in the frequent cluster variants watched most of the lecture recordings (many of them multiple times), whereas students in the selective variants watched only about half of the lecture recordings and usually only once. whereas the clusters did not differ with regard to gender and experience in using lecture recordings, they pursued different goals. across all clusters, reviewing lectures was the most frequently mentioned goal, especially in clusters 1 (frequent repetition) and 3 (frequent consultation). the same clusters also often reported that preparing for an exam was the goal of viewing the lecture recordings. using the lecture recordings as a reference, for example, during exercises, was most often reported by cluster 3 (frequent consultation) but least frequently in cluster 5 (increased absenteeism). these results demonstrate that the students used lecture recordings in different ways pursuing different goals. whatever strategy they employed, the students were highly satisfied with the lecture capture service, which increased their degrees of freedom in organising their studies – including the opportunity not to visit the lecture. whereas lecturers may be concerned about this option, students may benefit from the increased flexibility in their study schedule, especially in challenging situations such as time conflicts between different lectures or between lectures and job or child care. in our sample, only about 17% of the students chose the cluster 5 option and were table 6. differences between clusters on the question ‘which role did the lecture recordings have in your learning process?’. variables clusters frequent repetition selective repetition frequent consultation selective consultation increased absenteeism p to review the lecture 89.0 82.4 90.7 75.3 78.4 <0.001¹ as a reference, e.g. during exercises 39.5 16.2 52.3 40.5 22.2 <0.001² to prepare for the exam 88.7 54.7 84.1 62.0 71.3 <0.001³ note: values are percent of students per cluster who chose the respective response. multiple responses were possible. ¹ pearson’s chi-squared test for count data: χ² (4, n = 1023) = 21.44; ² pearson’s chi-squared test for count data: χ² (4, n = 1023) = 75.157; 3 pearson’s chi-squared test for count data: χ² (4, n = 1023) = 97.917. http://dx.doi.org/10.25304/rlt.v28.2258 research in learning technology citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 11 (page number not for citation purpose) consistently absent (with low variance across semesters). in many types of lectures, this percentage of absent students can be tolerated – particularly taking into account that at least a part of them would not participate anyway, for example, when time conflicts are the reason why they do not visit the lecture. our results also help to reconcile and interpret seemingly contradicting results from studies investigating the effects of lecture recording availability on lecture attendance (dommeyer 2017; edwards and clinton 2018; walls et al. 2010). a positive correlation between the usage of lecture recordings and absenteeism may emerge from fundamentally different situations. if, for example, all students of a lecture series sometimes watch the video and sometimes skip a lecture, a moderate positive correlation could be the result. a similar correlation could result when some students consistently watch the videos and do not visit the lecture, whereas many other students sporadically use the lecture recordings and sometimes miss a live lecture. our results corroborate the second interpretation. in our sample, a smaller number of students substituted nearly every lecture by watching the recording (around 17%), whereas a larger number of students nearly always used the recording as a supplement to attending the live lecture. thus, mean absenteeism rates may be misleading because of the large heterogeneity in user behaviour. however, this interpretation requires differentiating between different strategies of using lecture recordings as proposed in this study. similarly, considering different subgroups of students who consistently show different usage behaviour may also help to understand that the availability of lecture recordings may or may not help to increase academic performance. the consultation clusters, for example, used the recordings to deepen their understanding, which could enhance performance (wiese and newton 2013). a comparable effect for students who primarily replace attendance in the live lecture by viewing the recording is less probable, although this hypothesis still needs to be tested. limitations an important limitation of this study was that the data were gathered for evaluating the lecture capture service. consequently, not all data were related to the students’ usage behaviour but they were also related to the technical and organisational aspects of the lecture capture service. interactions between these different types of questions cannot be excluded. furthermore, the available data were all self-reported. methods using log or tracking data can be more reliable, especially in the context of study behaviour (e.g., gyllen et al. 2019). similarly, testing the potential relations between strategically using lecture recordings and academic performance would require reliable measures of performance, which were unavailable in the context of this study. another limitation is related to an organisational problem of the evaluation procedure. in the summer semester 2017 and the winter semester 2017/2018, some students could have attended other recorded lecture series and thus might have evaluated more than one recorded lecture. future research perspectives our results demonstrate that students use lecture recordings in diverse ways, which we interpret to be related to the students’ learning strategies. this view is corroborated by two studies exploring different approaches to learning (marton and säljö http://dx.doi.org/10.25304/rlt.v28.2258 d. ebbert and s. dutke 12 citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 (page number not for citation purpose) 1976). for example, vajoczki et al. (2011) found that students following a deep learning approach used lecture recordings more often to review a lecture and to prepare for exams than surface learning students, which could apply to our clusters 1 and 3. students following a more surface-oriented approach were more likely to use the lecture recordings as a substitute for the lectures, which is consistent with the behaviour of our cluster 5 students. another example for relations between the way students use lecture recordings and approaches to learning was reported by wiese and newton (2013). they found that deep learners compared to surface learners watched more lecture recordings and were more likely to watch them completely, which is consistent with our cluster 1 but also with cluster 5 students who demonstrated absenteeism. these examples show that for future research, a more differentiated analysis of students’ usage of (digital) learning resources is needed to explore how the availability of lecture recordings affects strategic learning behaviour. exploring these relationships may help to understand how learning situations can incorporate new technical options, such as lecture recordings or interactive videos, to augment self-directed study behaviour. data and materials the data that the results are based on are available in the open science framework at https://doi.org/10.17605/osf.io/dnwk5. the r scripts used to analyse the data are also available in the open science framework at https://doi.org/10.17605/ osf.io/e65hj. references american psychological association (2017) ethical principles of psychologists and code of conduct, american psychological association [online] available at: https://www.apa.org/ ethics/code/ bacro, t. r. h., gebregziabher, m. & fitzharris, t. p. (2010) ‘evaluation of a lecture recording system in a medical curriculum’, anatomical sciences education, vol. 3, no. 6, pp. 300–308. doi: 10.1002/ase.183 beasley, t. m. & schumacker, r. e. (1995) ‘multiple regression approach to analyzing contingency tables: post hoc and planned comparison procedures’, the journal of experimental education, vol. 64, no. 1, pp. 79–93. doi: 10.1080/00220973.1995.9943797 conover, w. j. & iman, r. l. (1979) multiple-comparisons procedures. informal report, los alamos scientific lab, los alamos, [online] available at: https://www.osti.gov/ biblio/6057803 de boer, j. & tolboom, j. (2008) ‘how to interpret viewing scenarios in log files from streaming media servers’, international journal of continuing engineering education and life-long learning, vol. 18, no. 4, pp. 432–445. doi: 10.1504/ijceell.2008.019643 dolch, c. & zawacki-richter, o. (2018) ‘are students getting used to learning technology? changing media usage patterns of traditional and non-traditional students in higher education’, research in learning technology, vol. 26. doi: 10.25304/rlt.v26.2038 dommeyer, c. j. (2017) ‘lecture capturing: its effects on students’ absenteeism, performance, and impressions in a traditional marketing research course’, journal of education for business, vol. 92, no. 8, pp. 388–395. doi: 10.1080/08832323.2017.1398129 draper, m. j., gibbon, s. & thomas, j. (2018) ‘lecture recording: a new norm’, the law teacher, vol. 52, no. 3, pp. 316–334. doi: 10.1080/03069400.2018.1450598 http://dx.doi.org/10.25304/rlt.v28.2258 https://doi.org/10.17605/osf.io/dnwk5 https://doi.org/10.17605/osf.io/e65hj https://doi.org/10.17605/osf.io/e65hj https://www.apa.org/ethics/code/ https://www.apa.org/ethics/code/ http://doi.org/10.1002/ase.183 http://doi.org/10.1080/00220973.1995.9943797 https://www.osti.gov/biblio/6057803 https://www.osti.gov/biblio/6057803 http://doi.org/10.1504/ijceell.2008.019643 http://doi.org/10.25304/rlt.v26.2038 http://doi.org/10.1080/08832323.2017.1398129 http://doi.org/10.1080/03069400.2018.1450598 research in learning technology citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 13 (page number not for citation purpose) dunn, o. j. (1961) ‘multiple comparisons among means’, journal of the american statistical association, vol. 56, no. 293, pp. 52–64. doi: 10.1080/01621459.1961.10482090 edwards, m. r. & clinton, m. e. (2018) ‘a study exploring the impact of lecture capture availability and lecture capture usage on student attendance and attainment’, higher education, vol. 77, no. 3, pp. 403–421. doi: 10.1007/s10734-018-0275-9 elliott, c. & neal, d. (2016) ‘evaluating the use of lecture capture using a revealed preference approach’, active learning in higher education, vol. 17, no. 2, pp. 153–167. doi: 10.1177/1469787416637463 european federation of psychologists’ associations (2005) meta-code of ethics, [online] european federation of psychologists’ associations. available at: http://ethics.efpa.eu/ metaand-model-code/meta-code/ gosper, m., et al., (2008) the impact of web-based lecture technologies on current and future practices in learning and teaching. [online] strawberry hills, nsw: australian learning and teaching council, pp. 1–7. available at: https://www.mq.edu.au/lih/altc/wblt/docs/ report/ce6-22_final2.pdf gower, j. c. (1971) ‘a general coefficient of similarity and some of its properties’, biometrics, vol. 27, no. 4, pp. 857–871. doi: 10.2307/2528823 gyllen, j. g., et al., (2019) ‘accuracy in judgments of study time predicts academic success in an engineering course’, metacognition learning, vol. 14, no. 2, pp. 215–228. doi: 10.1007/ s11409-019-09207-6 hope, a. c. a. (1968) ‘a simplified monte carlo significance test procedure’, journal of the royal statistical society. series b (methodological), vol. 30, no. 3, pp. 582–598. doi: 10.1111/j.2517-6161.1968.tb00759.x kalnikaitė, v. & whittaker, s. (2010) ‘beyond being there? evaluating augmented digital records’, international journal of human-computer studies, vol. 68, no. 10, pp. 627–640. doi: 10.1016/j.ijhcs.2010.05.003 kaufman, l. & rousseeuw, p. j. (1990) ‘partitioning around medoids (program pam)’, in finding groups in data: an introduction to cluster analysis, eds l. kaufman & p. j.  rousseeuw, john wiley & sons, hoboken, nj, pp. 68–125. doi: 10.1002/9780470316801.ch2 kruskal, w. h. & wallis, w. a. (1952) ‘use of ranks in one-criterion variance analysis’, journal of the american statistical association, vol. 47, no. 260, pp. 583–621. doi: 10.1080/01621459.1952.10483441 marton, f. & säljö, r. (1976) ‘on qualitative differences in learning i – outcome and process’, british journal of educational psychology, vol. 46, no. 1, pp. 4–11. doi: 10.1111/j.20448279.1976.tb02980.x mark, k. p. & vrijmoed, l. l. p. (2017) ‘does lecture capturing improve learning? a data driven exploratory study on the effectiveness of lecture capture on learning in a foundation it course’, proceedings of 2016 ieee international conference on teaching, assessment and learning for engineering, tale 2016, bangkok, pp. 338–344. doi: 10.1109/ tale.2016.7851818 mundform, d. j., et al., (2011) ‘number of replications required in monte carlo simulation studies: a synthesis of four studies’, journal of modern applied statistical methods, vol. 10, no. 1, pp. 19–28. doi: 10.1109/tale.2016.7851818 newland, d. b. (2017) lecture capture in uk he 2017. a helf survey report, heads of elearning forum, brighton [online] available at: https://drive.google.com/ open?id=0bx0bp7czgltpruppz2naaepkb28 nordmann, e., et al., (2018) ‘turn up, tune in, don’t drop out: the relationship between lecture attendance, use of lecture recordings, and achievement at different levels of study’, higher education, vol. 77, no. 6, pp. 1065–1084. doi: 10.1007/s10734-018-0320-8 o’callaghan, f. v., et al., (2017) ‘the use of lecture recordings in higher education: a review of institutional, student, and lecturer issues’, education and information technologies, vol. 22, no. 1, pp. 399–415. doi: 10.1007/s10639-015-9451-z http://dx.doi.org/10.25304/rlt.v28.2258 http://doi.org/10.1080/01621459.1961.10482090 http://doi.org/10.1007/s10734-018-0275-9 http://doi.org/10.1177/1469787416637463 http://doi.org/10.1177/1469787416637463 http://ethics.efpa.eu/metaand-model-code/meta-code/ http://ethics.efpa.eu/metaand-model-code/meta-code/ https://www.mq.edu.au/lih/altc/wblt/docs/report/ce6-22_final2.pdf https://www.mq.edu.au/lih/altc/wblt/docs/report/ce6-22_final2.pdf http://doi.org/10.2307/2528823 http://doi.org/10.1007/s11409-019-09207-6 http://doi.org/10.1007/s11409-019-09207-6 http://doi.org/10.1111/j.2517-6161.1968.tb00759.x http://doi.org/10.1111/j.2517-6161.1968.tb00759.x http://doi.org/10.1016/j.ijhcs.2010.05.003 http://doi.org/10.1002/9780470316801.ch2 http://doi.org/10.1080/01621459.1952.10483441 http://doi.org/10.1080/01621459.1952.10483441 http://doi.org/10.1111/j.2044-8279.1976.tb02980.x http://doi.org/10.1111/j.2044-8279.1976.tb02980.x http://doi.org/10.1109/tale.2016.7851818 http://doi.org/10.1109/tale.2016.7851818 http://doi.org/10.1109/tale.2016.7851818 https://drive.google.com/open?id=0bx0bp7czgltpruppz2naaepkb28 https://drive.google.com/open?id=0bx0bp7czgltpruppz2naaepkb28 http://doi.org/10.1007/s10734-018-0320-8 http://doi.org/10.1007/s10639-015-9451-z d. ebbert and s. dutke 14 citation: research in learning technology 2020, 28: 2258 http://dx.doi.org/10.25304/rlt.v28.2258 (page number not for citation purpose) rousseeuw, p. j. (1987) ‘silhouettes: a graphical aid to the interpretation and validation of cluster analysis’, journal of computational and applied mathematics, vol. 20, pp. 53–65. doi: 10.1016/0377-0427(87)90125-7 traphagan, t., kucsera, j. v. & kishi, k. (2010) ‘impact of class lecture webcasting on attendance and learning’, educational technology research and development, vol. 58, no. 1, pp. 19–37. doi: 10.1007/s11423-009-9128-7 vajoczki, s., et al., (2011) ‘students approach to learning and their use of lecture capture’, journal of educational multimedia and hypermedia, vol. 20, no. 2, pp. 195–214. https:// www.learntechlib.org/primary/p/36105/ walls, s. m., et al., (2010) ‘podcasting in education: are students as ready and eager as we think they are?’, computers & education, vol. 54, no. 2, pp. 371–378. doi: 10.1016/j. compedu.2009.08.018 whitley-grassi, n. e. (2017) evaluating student use patterns of streaming video lecture capture in a large undergraduate classroom, dissertation thesis, walden university. wiese, c. & newton, g. (2013) ‘use of lecture capture in undergraduate biological science education’, canadian journal for the scholarship of teaching and learning, vol. 4, no. 2, pp. 1–24. doi: 10.5206/cjsotl-rcacea.2013.2.4 williams, a. e., aguilar-roca, n. m. & o’dowd, d. k. (2016) ‘lecture capture podcasts: differential student use and performance in a large introductory course’, educational technology research and development, vol. 64, no. 1, pp. 1–12. doi: 10.1007/s11423-015-9406-5 woo, k., et al., (2008) ‘web-based lecture technologies: blurring the boundaries between faceto-face and distance learning’, research in learning technology, vol. 16, no. 2, pp. 81–93. doi: 10.3402/rlt.v16i2.10887 zupancic, b. & horz, h. (2002) ‘lecture recording and its use in a traditional university course’, proceedings of the 7th annual conference on innovation and technology in computer science education iticse’02, arhus, pp. 24–24. doi: 10.1145/637610.544424 http://dx.doi.org/10.25304/rlt.v28.2258 http://doi.org/10.1007/s11423-009-9128-7 https://www.learntechlib.org/primary/p/36105/ https://www.learntechlib.org/primary/p/36105/ http://doi.org/10.1016/j.compedu.2009.08.018 http://doi.org/10.1016/j.compedu.2009.08.018 http://doi.org/10.5206/cjsotl-rcacea.2013.2.4 http://doi.org/10.1007/s11423-015-9406-5 http://doi.org/10.3402/rlt.v16i2.10887 http://doi.org/10.1145/637610.544424 (page number not for citation purpose) 1 *corresponding author. email: vicki.dale@glasgow.ac.uk research in learning technology 2019. © 2019 v.h.m. dale and j. singer. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 research in learning technology vol. 27, 2019 original research article learner experiences of a blended course incorporating a mooc on haskell functional programming vicki h.m. dalea* and jeremy singerb alearning enhancement and academic development service, student and academic services, southpark house, university of glasgow, glasgow, scotland; bschool of computing science, sir alwyn williams building, university of glasgow, glasgow, scotland (received: 14 march 2019; revised: 3 june 2019: accepted: 3 june 2019; published: 10 july 2019) there is an increasing move in higher education to blend university courses to include a massive open online course (mooc). this article reports on the learner experiences of such a course, which incorporated a purposely designed mooc as part of the blend, to teach haskell functional programming. a survey revealed that students most valued the programming exercises, quizzes and instructional videos, while the follow-up focus group highlighted the flexibility of the mooc, usefulness of the videos, drop-in sessions and programming exercises. the overall mix of activities was regarded as particularly useful. while discussions were not rated as highly in the survey, students in the focus group commented on their value, particularly for getting support from external learners. the perceived lack of face-to-face contact was the biggest issue; however, this reflected a lack of awareness of lab sessions which could have been better signposted. there was perceived to be a gap between the mooc and the rest of the course in terms of level of difficulty and authenticity of learning tasks. these issues were positively addressed in subsequent runs of the course. the outcomes of this study are relevant to educators seeking to incorporate moocs into blended courses. keywords: mooc; learner experience research; blended learning; haskell programming introduction the changing digital landscape is impacting on learning and teaching within higher education (gardiner 2015). the sector is seeing increasing adoption of blended learning, in response to technological advancements, and the opportunity to support a more flexible learning experience that also enables students to take ownership of their learning (gordon 2014). blended learning may be defined as the optimal combination of face-to-face learning with online learning opportunities, with true blended learning affording the potential to reduce face-to-face contact time (adekola, dale, and gardiner 2017), particularly when that time has traditionally been used for information transmission. alongside the increasing adoption of blended learning more generally, massive open online courses (moocs) are seen to be having a disruptive effect on higher education (conole 2013). v.h.m. dale and j. singer 2 citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 (page number not for citation purpose) there are increasing instances where educators at tertiary level have formally adopted moocs as part of their blended provision. koller (2012) first commented on this possibility, recognising developments at antioch and vanderbilt universities. early examples of moocs being integrated into curricula use the ‘content licensing’ method of integration (sandeen 2013); that is, adopting moocs produced from institutions outside the institution offering a hybrid or blended course for their students. caulfield et al. (2013) call this the ‘distributed flip’ approach, in terms of a mooc serving as the online preparation for face-to-face teaching, although other authors refer to ‘wrapping’ a campus-based course around a mooc (bruff et al. 2013). it is important not to confuse either of these approaches with blended moocs (bmoocs), such as sending online learners physical ‘kit’ to use in their home environment (atiq et al. 2016). several studies document learner experiences of blended learning involving a mooc. caulfield et al. (2013) examined the experiences of learners engaged on a stanford mooc as part of their distributed flip curriculum at the university of puerto rico. students engaged with the video and interactive assessments, but not discussion forums, as they were able to converse face-to-face with their peers. a blended course that involved wrapping local teaching around a stanford mooc was delivered at vanderbilt university by bruff et al. (2013). students appreciated the flexibility and accessibility of the videos, but preferred to interact with their local learning community, rather than other mooc learners. they also expressed dissatisfaction with the ‘subject coupling’ (complementarity of content) of the vanderbilt course material running alongside the stanford-produced mooc. holotescu et al. (2014) described how students studying web programming at university politehnica timisoara, romania, were asked to spend 10% of their study time engaging with a mooc of their choice, enabling a flipped classroom approach. while students recognised the value of discussions and feedback in mooc forums, they felt a lack of tutor engagement on the mooc. in a study of postgraduate students in south africa who voluntarily took a blended course including a mooc to improve their study skills and graduate attributes, participants developed a sense of belonging with their peers as a result of the face-to-face contact, but the study did not examine their interaction with the wider mooc community (jaffer, govender, and brown 2017). some studies have investigated student performance in blended courses including a mooc. firmin et al. (2014) described the outcomes of a study comparing the performance of two groups in three mathematics and statistics courses; a student group from san jose state university (sjsu) who needed the courses in order to progress, and a group of adult/community and student learners. the study concluded that student effort – in terms of engagement with videos and submission of problem sets – was the single biggest predictor of performance, which was shown to be higher in the student group. a study of preand postperformance in relation to an introductory physics mooc at massachusetts institute of technology (mit) revealed that all mooc cohorts demonstrated learning gain; however, there was no evidence that on-campus learners who took the mooc as part of a blended course performed any better (colvin et al. 2014). griffiths et al. (2014) from ithaca s+r and the university system of maryland did several side-by-side comparisons of traditional courses versus blended ones that incorporated an external mooc. students on the blended courses performed as well as or slightly better than those on traditional courses, but were more dissatisfied in terms of how much they enjoyed the course, how much research in learning technology citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 3 (page number not for citation purpose) they thought they learned and how much the course had increased their interest in the subject. there were also issues of ‘content fit’ in relation to local curricula. the issue of content mismatch was also observed in a study of engineering students at sjsu, despite their teachers working with mit to adapt their mooc for sjsu students (ghadiri et al. 2013). in that study, 91% of students passed the blended course, compared with 59% of the previous cohort who undertook the traditional course. sjsu mooc learners valued the flexibility to watch lectures in their own time, and quizzes, though some learners requested fewer quizzes and more problem-solving demonstrations. in the netherlands, conijn, van den beemt, and cuijpers (2018) showed that the frequency and order of activity access in a mooc undertaken as part of a blended course were correlated with the final on-campus course exam grade. at the university of zagreb, students studying mathematics as part of an engineering qualification were given the option of undertaking an external mooc of their own choice instead of project work (bralić and divjak 2018). mooc students outperformed project students, and learning diaries revealed that students valued the flexibility (the self-paced nature of the mooc, the ability to pause videos) and more frequent opportunities for self-testing. students also perceived linkage between the mooc and their college course, though one student was unmotivated by the relative lack of face-to-face contact. at the time this study was undertaken, there were few published studies of learner experience research on a blended course that incorporated a mooc as core material, authored within the same institution, rather than as a supplement. a  study of the integration of a mooc produced by the university of leeds with  their first-year medical anatomy course (as an optional addition rather than core resource) had mixed results – while students appreciated videos and quizzes, they were less  engaged with collaborative tasks such as discussion boards, and  they actively  rejected the idea of moocs replacing campus-based teaching ( swinnerton et al. 2016). more recently, cornelius, calder, and mtika (2019) described the blending of  an undergraduate course on sustainable development in  africa at the university of aberdeen, incorporating a purposely designed mooc  to replace lectures, encourage critical thinking and facilitate learners to engage with a global community. a student engagement survey revealed that students who took the blended course scored higher on learning with others, and reflective skills, than non-mooc cohorts. a focus group highlighted that the mooc students appreciated a variety of media and benefited from discussion with the extended mooc community. another study described a proposed design and associated evaluation of a purposely designed mooc as part of a blended provision for health care postgraduates and practitioners, but is still a work in progress (meinert et al. 2018). this article contributes to this growing field of study by investigating the learner experiences of students enrolled on an innovative blended course that included a purposely designed mooc which was concurrently available to external learners. the study sought to understand the benefits and challenges associated with this new flexible mode of studying, and to assess the impact of integrating university learners with a wider learning community, and to identify support needs to help the institution move forward in its provision for blended learning incorporating moocs. the study was undertaken as part of a wider investigation into institutional, student and staff transitions to blended learning (adekola, dale, and gardiner 2017; adekola et al. 2017). v.h.m. dale and j. singer 4 citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 (page number not for citation purpose) methodology context this study examines learner experiences of a new blended fourth-year honours undergraduate course ‘haskell functional programming’ at the university of glasgow (uofg), traditionally taught in the classroom. the university’s strategic investment in blended and online learning development (the bold project) provided an opportunity for academics to apply for funding to develop blended or online courses or programmes to enhance the student learning experience and quality of learning outcomes. the 10-week course was redesigned to include a 6-week mooc, followed by 4 weeks of blended learning. the first 6 weeks constituted the futurelearn ‘functional programming in haskell’ mooc.1 the last 4 weeks were blended in the sense that materials were largely online in the institution’s virtual learning environment (vle), moodle, and optional face-to-face drop-in sessions were timetabled throughout the 10 weeks to allow participants to interact directly with their tutors and each other. the first run of the blended course, incorporating the mooc, was offered in the 2016–17 session. cognisant of the need to provide personalised tuition and immediate feedback on students’ programming skills (vihavainen, luukkainen, and kurhila 2012), the mooc was designed to include an interactive programming interface (‘tryhaskell’). using an adaptive learning approach, the mooc included a number of programming exercises that became increasingly complex as students progressed through the mooc (singer and archibald 2018). the mooc also included readings, instructional videos, videos of interviews with experts, and quizzes. we considered that interacting with a global community of learners was a required skill of programming graduates, who would – in practice – be interacting professionally on online networks to help source solutions to complex problems. in addition, we  believed that because the mooc would likely also include professional learners, they would be introduced to interesting examples of haskell programming from industry. therefore, the research questions underpinning this study were: 1. what were learners’ experiences of a mooc as part of a blended learning design? a. what did they value most? b. what did they find most challenging? 2. how did students experience being part of a massive community of learners? 3. how did the mooc prepare learners for the second part of the course? 4. how should the blended course be modified for a future iteration? methods a sequential explanatory mixed methods approach was undertaken (jones, torres, and arminio 2013), combining a survey with a follow-up focus group. all fourth-year students who were enrolled on the course were invited to take part in the survey, and a call among this group for volunteers was made to recruit focus group participants. the focus group was facilitated by author dale with assistance from a graduate teaching assistant tutor on the mooc, who was able to clarify any subject-related  details. 1 https://www.futurelearn.com/courses/functional-programming-haskell research in learning technology citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 5 (page number not for citation purpose) both facilitators were therefore independent of the course lecturers, to avoid biasing students’ responses. the focus group was audio-recorded and professionally transcribed by an external company. the survey was hosted on the bristol online survey platform. the survey included nominal items (demographics, prior experience and motivations), likert-scale data (on a scale of 1 to 5 where 1 = most negative and 5 = most positive) and open text questions. quantitative data were exported into excel and spss for further data representation, summarisation and analysis. qualitative data from the survey and focus group were subject to thematic analysis (braun and clarke 2006), by author dale. the open text survey responses and focus group transcript were first printed and read through, then annotated, to identify relevant codes. codes were grouped into categories. ideally, emerging ‘themes’ would then have been identified, although there were insufficient qualitative data to make the case for this. to establish reliability, the original anonymised data were shared with author singer, and a face-to-face meeting was held to review and agree the coding structure. we have focused our discussion on what we feel are the most salient and strongly expressed points. to enhance the credibility aspect of trustworthiness (o’brien et al. 2014), quotes from the survey and focus group have been included. ethical approval was granted by the college of science and engineering ethics committee (#300160028). potential participants were contacted via email, and a plain language statement and consent form (the latter for the focus group only) were circulated in advance of participation. participants were assured that their responses would be kept confidential, although the aggregated anonymised results would be shared with the course leads. raw data were stored in accordance with the 1998 data protection act, and participants were assured that they could withdraw from the study, and that their decision to participate or not would not influence their relationship with their teachers or the co-researcher. no incentive was made available for the survey; however, a catered lunch was used to incentivise focus group participants, who were also given a £10 amazon voucher each as a reward for participating. results thirty-six students participated in a survey (a response rate of 29%) and six students participated in the focus group. the question prompts for the focus group are listed in appendix 1. learners’ participation and motivation thirty-four participants were undergraduate computing students and two were postgraduate students who had undertaken the course as an elective. the majority of learners (78%) had previously participated in a mooc. all study participants engaged with the mooc, with 87% completing it. approximately half the students (49%) followed the 6-week recommended schedule, while nine (24%) studied at the own pace in advance of the schedule, eight (22%) studied at their own pace behind schedule and two (6%) answered ‘other’, being ahead for some parts of the mooc and behind for others. approximately half of the survey participants had expected the course to include a mooc (49%); the others were unaware that it would include a mooc. only one v.h.m. dale and j. singer 6 citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 (page number not for citation purpose) participant (3%) signed up for the course on the basis that it included a mooc; another 11 were partly motivated by the inclusion of a mooc (30%) but the majority’s (67%) decision to sign up for the course was not predicated on this fact. once they knew the course would include a mooc, the majority (59%) were either motivated or extremely motivated to undertake part of their course on a mooc. thirty-two per cent were somewhat motivated. only two students (5%) were not very motivated, and only one was not motivated at all by the inclusion of the mooc. usefulness of different learning activities and sources of mooc support figure 1 shows that the most valued types of activity in terms of their perceived usefulness (all median 4) were programming exercises (86% useful/extremely useful), quizzes (81%), instructional videos (73%), the programming interface (65%), the faceto-face drop in sessions (61%), discussions (56%) and expert interviews (51%). the least valued type of activity (median 3) was the readings (49% useful/extremely useful). figure 2 shows that learners felt most supported by their lecturers (70% well/ extremely well supported, median 4). they felt less supported by other sources of 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% readings expert interviews discussions face-to-face drop-in sessions programming interface instruc onal videos quizzes programming exercises useful not very usefulnot at all useful somewhat useful extremely useful figure 1. comparative usefulness of different learning activities. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% other futurelearn learners glasgow peers tutors (gtas) tryhaskell lecturers not very usefulnot at all useful somewhat useful extremely usefulwell supported figure 2. sources of support on the blended course. research in learning technology citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 7 (page number not for citation purpose) help (all median 3): the tryhaskell programming interface (46% well/extremely well supported), graduate teaching assistants (45%), their university of glasgow peers (43%) and other learners on the futurelearn mooc (42%). what worked well in terms of the survey comments relating to what went well (n = 29), students appreciated most the flexibility afforded by the mooc, allowing them to undertake learning at their own pace (n = 10): “for people with no functional programming experience it is useful to have online training (you can re-read, google stuff), while if done in classroom might not be able to go at the pace of the lecture. this is particularly good at the beginning of a new study area, when you have no base and foundation for it already.” the videos (n = 5), drop-in sessions/workshops (n = 5), programming exercises (n = 4) and the tryhaskell compiler (n = 3) were also valued: “videos were informative and well-paced.” “workshops to discuss mooc content.” “exercises enhanced the work we were learning on.” “the online exercises worked very well.” in particular, it was the overall design of the mooc, and the combination of activities, which strengthened the learning experience (n = 5): “the quizzes, videos, and exercises formed a very strong combination. the reading provided greater detail on topics given in the lighter materials e.g. videos. the discussions that sat alongside the materials were extremely useful.” “the course was great overall.” other miscellaneous survey comments on what worked well (n = 1 or 2) related to the quizzes, the social element of online discussions, the amount of support offered, the practical elements being better than a lecture for learning coding and the readings. in the focus group, the main benefits to learning were identified as the flexibility associated with the mooc: “yes, so basically it was nice that you could just go at your own pace. if you had a busy week or whatever you could do it beforehand or after. you don’t have to be like specifically in person there. you could just basically do it whenever you wanted to.” related benefits in the focus group included control over video speed, and the ability to revisit lectures. the quizzes were also considered useful but theoretical: “the quizzes were pretty good but there could be more of them.” “i mean the quizzes were all very technical as in you were going through the lecture and the quizzes were more like asking you some specific part of the lecture.” v.h.m. dale and j. singer 8 citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 (page number not for citation purpose) what did not work so well from the survey comments (n = 26), the biggest frustration proved to be the lack of face-to-face contact with teachers and peers (n = 8), particularly the lack of labs (n = 5): “would have been useful to have a lab session, let’s say around week 5 or 6 with some exercises or challenges and active feedback.” the lack of lectures affected a small minority of students (n = 2) and one student felt that it was easier to have concepts explained face-to-face: “as mentioned before, non-mandatory face-to-face sessions made me feel less inclined to attend (and more likely to miss out on useful learning/understanding).” “explaining a difficult concept using a video or text is not as good as explaining it in class, where you can ask questions right away when it’s still fresh in mind and face-to-face explanations are clearer as well.” other miscellaneous survey comments on what did not work so well (n = 1 or 2) related to the expert interviews, reading that was too long or in-depth, the ‘big jump’ between weeks, lack of long form programming practice, lacking confidence in knowledge or a poor pairing of the mooc and the glasgow course (running sequentially rather than in tandem). in the focus group, the biggest frustrations related to not feeling prepared for the assignment, due to the exercises being theoretical or disaggregated rather than allowing students to prepare for long-form programming: “well, it’s very difficult for the assessed exercise in the sense that…at least for me i was faced with new stuff there because otherwise you’d made them really simple examples, like we have some optional exercises and that’s all easy. when you go into assessed exercise you’re actually faced with i/o and everything. so basically you, kind of, need some sort of experience with that sort of stuff beforehand and i don’t think mooc really prepared us in any way for that or not really properly at least.” despite being well received, the tryhaskell interactive programming tool was sometimes buggy and students felt restricted by the line-by-line coding approach, with students requesting longer working programme examples and more complex exercises, which would have helped them prepare for their higher level assignment: “definitely some more just bigger kind of things, for example, the exercises. the ones we had were like we were given a list of numbers, find me the biggest one or give me the sum or something which is obviously pretty basic. i would say give some more real life examples or something where you can actually have more i/o interactions and stuff like that or whatever you need further on … or something, you know, i mean, if you have an assessed exercise at such a level you want to have some sort of progressive exercises to that level, don’t just throw me there.” research in learning technology citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 9 (page number not for citation purpose) not having labs was also raised in the focus group as an issue: “no labs. i feel like there should be like all upper case, like no labs … there definitely should be labs.” being part of a massive community of learners student attitudes to studying within a massive cohort were diverse; 37% considered it not at all or not very useful, while 27% considered it somewhat useful, and 35% considered it useful or extremely useful. in terms of what was gained studying as part of a massive cohort (n = 24 comments), the functionality in the mooc for comments and discussion was identified as the biggest advantage. this included having more people to interact with and more support (n = 4) as well as having broader discussions (n = 4), interacting with people with the same problem (n = 3), receiving interesting comments on students’ posts (n = 3) and seeing how others approached tasks (n = 2): “most of my questions on mooc were answered by those who are not glasgow students.” “there was more discussion on things that aren’t usually brought up in class. also there was a wide range of backgrounds to pull from.” “when i got stuck at some of the exercises i had a look through the comments, and it turned out a few other people had similar problems.” in terms of what was lost studying as part of a massive cohort (out of 20 comments), eight students considered ‘nothing’ or ‘not much’, while others commented on the lack of face-to-face contact with teachers or peers in timetabled classes (n = 6), a reduced focus on glasgow students (n = 2), reduced motivation (n = 2) or a lack of individualised teaching and feedback (n = 2): “personal interaction and a motivation to learn [was lost]. experience of learning became less ‘real’.” “difficult to find answers that may be specific to assignments of glasgow students.” “having the mooc meant (to some extent) that i didn’t feel obliged to go to face-to-face sessions outside of the final lectures. even though this clashes with one of my demonstration slots, i still feel having some impetus to go would have deepened my understanding more than the mooc.” preparedness for the rest of the course forty-two per cent of respondents felt either prepared or extremely well prepared for the rest of the course (the remaining 4 weeks of the semester, taught on campus), while 57% felt not very prepared or somewhat prepared. no participants considered themselves not at all prepared. v.h.m. dale and j. singer 10 citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 (page number not for citation purpose) attitudes towards this type of blended learning the majority of students’ expectation of the course were either partly met (44%) or fully met (47%) with two participants (6%) feeling that their expectations had been exceeded. only one participant felt that their expectations had not been met at all (3%). when asked in the focus group why some participants’ expectations were not met, responses indicated that some students might have had unrealistic expectations, having never done a mooc before, while others commented on the big ‘jump’ at week 4 into a more complex topic. there was also a sense that the mooc component was very ‘hands off ’. over half (51%) of participants considered that more courses should be taught in this blended way incorporating a mooc, while just under half (46%) were not sure and 3% (one student) stated that more courses should not be taught this way. discussion the major benefit to learners in this study was the flexibility offered by the blended format, in terms of the first part of the course being delivered online. the benefits of such a flexible approach have been advocated in relation to technology-enhanced learning generally (gordon 2014) and in blended courses that incorporate a mooc (bralić and divjak 2018; bruff et al. 2013). however, this assumes that students have the study skills and self-directed learning readiness to take responsibility for their learning in this way (malik 2008). related to this, one disadvantage was the perceived lack of labs, which would have allowed students to interact face-to-face with their teachers and peers. face-to-face interactions in blended and online courses are associated with a sense of belonging, essential to positive student learning outcomes (tayebinik and puteh 2012; thomas, herbert, and teras 2014). while this perception was due to unclear communication and signposting about the availability of face-toface interaction, it does highlight the importance of teacher presence including the need to provide clear signposting for students not used to managing their own learning in a blended curriculum (greener 2008). a face-to-face induction is considered critically important by learners engaging with a blended curriculum that includes a mooc (cornelius, calder, and mtika 2019). perhaps, this would suggest that the face-to-face component of a blended curriculum should take precedence over the mooc in terms of signposting activities for learners (wrapping a course around a mooc, rather than a distributed flip approach that requires the learners to independently engage with the mooc up front). the mooc was designed to cater for a massive audience, and therefore used a combination of media and activities in line with futurelearn’s (2018) recommendations. students in this study particularly valued the mix of different types of learning activity. however, there was a paradox in terms of the perceived value of the discussions; these were among the least valued in terms of rating, and fellow mooc learners were considered the source of least support overall. as identified by israel (2015), this is consistent with other studies (bruff et al. 2013; caulfield et al. 2013), where students favoured videos and quizzes, and interacting with local students in preference to a massive cohort. reluctance to engage in discussions was also observed in conijn, van den beemt, and cuijpers’s (2018) study. however, it is clear that a subset of our learners did particularly value the engagement with a massive cohort in order to find solutions to problems, consistent with holotescu et al. (2014). this suggests that some learners are more comfortable engaging in discussion on moocs than research in learning technology citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 11 (page number not for citation purpose) others, which gives weight to the argument that there are different types of learners in moocs, reflecting their preference for different types of activity (poellhuber, roy, and bouchoucha 2019). one of the issues of incorporating a mooc into a higher education course is the need to balance the needs of local learners with those of a massive cohort with varying experience of the subject. most moocs are designed at ‘entry level’ into higher education. however, the second part of the course was developed independently for delivery on the vle, specifically for honours level and masters students. this created a ‘gap’ between the two halves, leading to a disjointed experience in terms of the level of thinking, but also in terms of constructively aligning the entry-level programming exercises with the higher level end-of-course assessment. this was despite the fact that the programming exercises were considered the most useful aspect of the mooc, perhaps unsurprising since this was most aligned with the expected outcomes of the course. limitations of this study include the fact that only 29% of learners responded; there may have been a response bias favouring those students with extreme views (phillips, reddy, and durning 2016). in addition, this study sought only to explore learner experiences, rather than learning gains. future research is needed to consider whether learners’ performance is enhanced by this blended course approach. the study was also of one cohort undertaking a single course within one institution. the findings are therefore not intended to be generalisable but offer insights for other institutions seeking to design and deliver a blended course that includes a mooc. two further iterations of the course have taken place since this initial case study. the course has been adapted in three distinct ways, in response to student feedback. 1. an additional mooc learning activity was incorporated, involving peer review of a multi-line source code exercise facilitated using adaptive comparative judgement (acj) (singer et al. 2019). this was designed to provide learners with experience in developing and comprehending larger haskell programmes. 2. the tryhaskell interactive tool was improved in terms of stability and usability. glitches in this initial coding environment appear to have compromised some learners’ experience. the presentation of some concepts was refined, based on analysis of common programming errors observed in tryhaskell sessions (singer and archibald 2018). 3. the face-to-face drop-in sessions were emphasised to a greater degree. the sessions were advertised in the initial year group briefing session, for which attendance is compulsory. at the start of each week’s drop-in, the appropriate mooc progress for that week was highlighted. conversation themes initiated in the mooc forum were also introduced to the face-to-face discussion. ideally, a longitudinal study would be carried out to assess the benefits of these changes;  however, ethical approval was only sought and granted to survey uofg learners for the first run of the course, to assess learners’ first perceptions of what we believed at the time to be a brand new form of blended course design. however, ethical permission was subsequently granted to the authors to review the outcome of a separate evaluation of haskell students’ use of acj (#400150118). only four students completed the evaluation in 2018; however, their experiences were overwhelmingly positive in terms of the perceived helpfulness of seeing peers’ coding submissions (all students rated this 5/5), and comparing others’ submissions for their own learning v.h.m. dale and j. singer 12 citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 (page number not for citation purpose) (all students rated this 5/5). following our initial ethics application (#300160028), permission was subsequently granted to examine routinely gathered end-of-course evaluation survey data in the subsequent two academic sessions. in relation to improvement 2, no students in the subsequent two cohorts commented negatively on the stability of the tryhaskell interface; however, a minority of students (n = 2 in each year) commented that key concepts were not well explained, which reflects the complexity of some topics in the course. in relation to improvement 3, only one student reported a perceived lack of sufficient lab time in the 2017–18 survey, and no students reported this as an issue in the 2018–19 survey. ongoing evaluation data will be used to continually refine and enhance the course design. conclusions this paper describes the outcomes of a learner experience research study on students undertaking a mooc designed intentionally as part of a blended course. this type of blended learning is still relatively rare and understudied; therefore, the results are intended to guide other educators wishing to develop a blended course that includes a purposely designed mooc. in particular, the study highlighted the importance of effective learning design, incorporating a range of different learning types, as well as the need for clear signposting about the face-to-face element of a blended course, associated with a strong teacher presence, to encourage learners to take responsibility for their learning. acknowledgements we would like to thank qaa scotland and the university of glasgow for co-funding this study, and all study participants for their time and input. we also thank cristian urlea for co-facilitating the focus group, as well as dr wim vanderbauwhede (co-course lead) and niall barr (systems developer), without whom this new blended  course would not have been delivered. we would also like to thank sarah honeychurch for sharing the outcomes of her acj evaluation study. references adekola, j., dale, v. h. m. & gardiner, k. (2017) ‘development of an institutional framework to guide transitions into enhanced blended learning in higher education’, research in learning technology, vol. 25, pp. 1–16. doi: 10.25304/rlt.v25.1973 adekola, j., et al., (2017) ‘student transitions to blended learning; an institutional case study’, journal of perspectives in applied academic practice, vol. 5, no. 2, pp. 58–65. doi: 10.14297/ jpaap.v5i2.273 atiq, m. s. z., et al., (2016) ‘understanding student experiences in a blended-learning mooc: a phenomenographic study’. paper presented at the asee’s 123rd annual conference & exposition, new orleans, la. doi: 10.18260/p.27102 bralić, a. & divjak, b. (2018) ‘integrating moocs in traditionally taught courses: achieving learning outcomes with blended learning’, international journal of educational  technology in higher education, vol. 15, no. 1, p. 2. doi: 10.1186/s41239 017-0085-7 braun, v. & clarke, v. (2006) ‘using thematic analysis in psychology’, qualitative research in psychology, vol. 3, no. 2, pp. 77–101. doi: 10.1191/1478088706qp063oa research in learning technology citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 13 (page number not for citation purpose) bruff, d. o., et al., (2013) ‘wrapping a mooc: student perceptions of an experiment in blended learning’, journal of online learning and teaching, vol. 9, no. 2, p. 187. caulfield, m., collier, a. & halawa, s. (2013) ‘rethinking online community in moocs used for blended learning’, [online] available at: http://er.educause.edu/articles/2013/10/ rethinking-online-community-in-moocs-used-for-blended-learning colvin, k. f., et al., (2014) ‘learning in an introductory physics mooc: all cohorts learn equally, including an on-campus class’, the international review of research in open and distributed learning, vol. 15, no. 4. pp. 263–283. doi: 10.19173/irrodl.v15i4.1902 conijn, r., van den beemt, a. & cuijpers, p. (2018) ‘predicting student performance in a blended mooc’, journal of computer assisted learning. doi: 10.1111/jcal.12270 conole, g. (2013) ‘moocs as disruptive technologies: strategies for enhancing the learner experience and quality of moocs’, [online] available at: http://eprints.rclis.org/19388/ cornelius, s., calder, c. & mtika, p. (2019) ‘understanding learner engagement on a blended course including a mooc’, research in learning technology, vol. 27. doi: 10.25304/rlt. v27.2097 firmin, r., et al., (2014) ‘case study: using moocs for conventional college coursework’, distance education, vol. 35, no. 2, pp. 178–201. doi: 10.1080/01587919.2014.917707 futurelearn. (2018) ‘pedagogy and learning design’, [online] available at: https://partners. futurelearn.com/course-creation/pedagogy-and-learning-design/ gardiner, k. (2015) ‘reasons to be open – embracing the digital landscape’, paper presented at the association for learning technology conference (alt-c), university of manchester, manchester. ghadiri, k., et al., (2013) ‘the transformative potential of blended learning using mit edx’s 6.002 x online mooc content combined with student team-based learning in class’, environment, vol. 8, p. 14. gordon, n. (2014) flexible pedagogies: technology enhanced learning, [online] available at: https://www.heacademy.ac.uk/flexible-pedagogies-technology-enhanced-learning greener, s. l. (2008) ‘self-aware and self-directed: student conceptions of blended learning’, merlot journal of online learning and teaching, vol. 4, no. 2, pp. 243–253. griffiths, r., et al., (2014) ‘interactive online learning on campus: testing moocs and other platforms in hybrid formats in the university system of maryland’, [online] available at: http://www.sr.ithaka.org/wp-content/mig/reports/s-r_interactive_online_learning_ campus_20140710.pdf holotescu, c., et al., (2014) ‘integrating moocs in blended courses’, paper presented at the the international scientific conference elearning and software for education. israel, m. j. (2015) ‘effectiveness of integrating moocs in traditional classrooms for undergraduate students’, international review of research in open and distance learning, vol. 16, no. 5, pp. 102–118. doi: 10.19173/irrodl.v16i5.2222 jaffer, t., govender, s. & brown, c. (2017) ‘“the best part was the contact!”: understanding postgraduate students’ experiences of wrapped moocs’, open praxis, vol. 9, no 2, pp. 15. doi: 10.5944/openpraxis.9.2.565 jones, s. r., torres, v. & arminio, j. (2006) ‘choices and consequences of mixing methods in qualitative research’, in negotiating the complexities of qualitative research in higher education: fundamental elements and issues, routledge, london, pp. 135–151. koller, d. (2012) ‘how online courses can form a basis for on-campus teaching [blog post]’, [online] available at: https://www.forbes.com/sites/coursera/2012/11/07/how-onlinecoursescan-form-a-basis-for-on-campus-teaching/#6bc5ac2f2887 malik, s. (2008) ‘on campus, but out of class: an investigation into students’ experiences of learning technologies in their self-directed study au – deepwell, frances’, alt-j, vol. 16, no 1, pp. 5–14. doi: 10.1080/09687760701850166 meinert, e., et al., (2018) ‘real-world evidence for postgraduate students and professionals in healthcare: protocol for the design of a blended massive open online course’, bmj open, vol. 8, no. 9. doi: 10.1136/bmjopen-2018-025196 v.h.m. dale and j. singer 14 citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 (page number not for citation purpose) o’brien, b. c., et al., (2014) ‘standards for reporting qualitative research: a synthesis of recommendations’, academic medicine, vol. 89, no. 9, pp. 1245–1251. doi: 10.1097/ acm.0000000000000388 phillips, a. w., reddy, s. & durning, s. j. (2016) ‘improving response rates and evaluating nonresponse bias in surveys: amee guide no. 102’, medical teacher, vol. 38, no. 3, pp. 217–228. doi: 10.3109/0142159x.2015.1105945 poellhuber, b., roy, n. & bouchoucha, i. (2019) ‘understanding participant’s behaviour in massively open online courses’, international review of research in open and distance learning, vol. 20, no. 1. pp. 222–242. sandeen, c. (2013) ‘integrating moocs into traditional higher education: the emerging “mooc 3.0” era’, change: the magazine of higher learning, vol. 45, no. 6, pp. 34–39. doi: 10.1080/00091383.2013.842103 singer, j. & archibald, b. (2018) ‘functional baby talk: analysis of code fragments from novice haskell programmers’, electronic proceedings in theoretical computer science, vol. 270, pp. 37–51. doi: 10.4204/eptcs.270.3 singer, j., et al., (2019) ‘from a thousand learners to a thousand markers: scaling peer feedback with adaptive comparative judgement’, paper presented at the 12th annual university of glasgow learning and teaching conference, glasgow, [online] available at: http://eprints. gla.ac.uk/183564/1/183564.pdf swinnerton, b. j., et al., (2016) ‘the integration of an anatomy massive open online course (mooc) into a medical anatomy curriculum’, anatomical sciences education, vol. 10, no. 1, pp. 53–67. doi: 10.1002/ase.1625 tayebinik, m. & puteh, m. (2012) ‘sense of community: how important is this quality in blended courses’, paper presented at the the international conference on education and management innovation, singapore, [online] available from: https://arxiv.org/ftp/arxiv/ papers/1504/1504.00249.pdf thomas, l., herbert, j. & teras, m. (2014) ‘a sense of belonging to enhance participation, success and retention in online programs’. the international journal of the first year in higher education, vol. 5, no. 2, pp. 69–80. doi: 10.5204/intjfyhe.v5i2.233 vihavainen, a., luukkainen, m. & kurhila, j. (2012) ‘multi-faceted support for mooc in programming’, paper presented at the proceedings of the 13th annual conference on information technology education. research in learning technology citation: research in learning technology 2019, 27: 2248 http://dx.doi.org/10.25304/rlt.v27.2248 15 (page number not for citation purpose) appendix 1: focus group prompts 1. has anyone here had previous experience of learning on a mooc? how did this one compare? what are your thoughts on the futurelearn platform? 2. what, in your experience, are the benefits of learning in this blended way? 3. what are the challenges or difficulties around learning in this blended way? 4. for some participants, their expectations were only partly met – why do you think this was? 5. what was it like, learning within a bigger community? 6. not everyone felt prepared for the rest of the course after the mooc – thinking about your own experience, why do you think this was? 7. how could we better support future cohorts of students doing this course? 8. do you feel the course was ‘blended’ appropriately? if not, what ‘blend’ would you say would be optimal? 9. the respondents to the survey were split 50/50 about whether more courses should be taught this way – based on your own experiences, why do you think this is? 10. finally, are there any other comments you’d like to add? (page number not for citation purpose) 1 *corresponding author. email: isisibika@mzumbe.ac.tz research in learning technology 2023. © 2023 i.s. isibika et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 research in learning technology vol. 31, 2023 original research article the influence of user-perceived benefits on the acceptance of microlearning for librarians’ training irene shubi isibikaa*, chang zhua, egbert de smetb and albogast k. musabilac adepartment of education sciences, vrije universiteit brussels, brussels, belgium; bdepartment of information and library science, antwerpen university, antwerp, belgium; cschool of business, mzumbe university, morogoro, tanzania (received: 18 october 2022; revised: 25 february 2023; accepted: 8 march 2023; published: 20 april 2023) microlearning has shifted professional training and development and its acceptance depends on perceived user benefits. this study examines the influence of userperceived benefits on librarians’ acceptance of the microlearning approach in selected universities in tanzania. using a questionnaire informed by the variables of the technology acceptance model (tam) model to collect data from a sample of 64 librarians, the results indicated that the perceived ease of use was found to play a more significant role in the acceptance of microlearning than perceived usefulness. however, the lack of in-person/physical interaction with peers during the training deterred the use of microlearning by the librarians. the positive influence of perceived usefulness and ease of use on the acceptance of microlearning strengthens the relevance of adopting microlearning as a didactic technology for librarians’ training. the findings imply that proper implementation of microlearning as a training avenue is key due to the plenteous benefits it affords over the challenges. keywords: microlearning; learning technologies; training; librarians; didactic; pedagogical introduction technology has transformed workplace training and professional development. many organisations are embracing the use of didactic technologies to conduct professional development. one such didactic technology is microlearning. the microlearning concept has quickly gained much traction in the training industry, especially in workplace training. research indicates the rise of and usefulness of microlearning in employee training, increasing its use by companies (taylor and hung 2022). researchers have tried to define microlearning as a form of e-learning delivered in small chunks, focused on delivering skill-based and just-in-time knowledge (zhang and west 2019). it may refer to short-term learning activities based on small pieces of information, brief video segments, short podcasts, etc. (reinhardt and elwood 2019). generally, the microlearning concept has adopted the mailto:isisibika@mzumbe.ac.tz http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2930 i.s. isibika et al. 2 citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 (page number not for citation purpose) following characteristics to fit current global dimensions in its design as pointed out by (redondo et al. 2021) • time: a limited effort leads to short-time requirements. • content: short units with well-delimited subject matters and relatively simple problems. • curriculum: parts of modules or parts of curricular content, brief didactic elements, etc. • format: diversity of formats, such as fragments, lab assignments, etc. • process: activities that are either independent or integrated into a broader context, iterative processes, etc. • media: on-demand learning or distance learning is based on different multimedia content. • learning models: repetitive, reflexive, pragmatic, constructivist, concept-based, connectivist, etc. microlearning enables learners to reduce information overload, enrich learning processes, improve training effectiveness, and increase learners’ level of engagement in online environments (nikou and economides 2018; shamir-inbal and blau 2020). this is because it aids better retention thus allowing learners to develop selfregulation skills and lifelong learning capacities (aldosemani 2019; reinhardt and elwood 2019). microlearning has an asynchronistic aspect, allowing the learner to control the place, method, and time of access to information (de gagne et al. 2019). continued employee training and development keeps companies competitive, increases employee retention, and augments the quality of a company’s services thus training and development establishments are increasingly looking for ways to reach more employees (brown et al. 2018; elearning industry 2022). fostering lifelong learning and continuing education in working adults has been an essential custom in organisations and technology has enhanced change in workplace learning. the change has been influenced by today’s fast-paced and digitally savvy working environment as stated by puah et al. (2022) leading to a notable increase in demand for just-in-time instruction and training. moreover, learners have a greater volition in deciding what technologies to use and when and how learning should occur. the librarianship profession has experienced a shift in operations. complex library users, an influx of readily available information, and the introduction of automated libraries have required a change in the skillset of librarians. the modern libraries are managed by a intergrated library management system that incorporates various modules of library functions (zainab et al. 2018). this has necessitated librarians to acquire new skill sets. the decreasing training budgets have led universities to use less costly means for employee training (isibika et al. 2021, 2022). moocs and webinars have been used to equip library staff with new knowledge and skills. however, time allocated for training, extensive coverage of topics in a short time, and the inability to quickly revisit training content have incapacitated librarians from fully utilising and implementing the knowledge. microlearning is now paving a better training avenue since it overcomes such barriers (chai-arayalert and puttinaovarat 2020; dolasinski and reynolds 2020). the current study was constituted following two earlier studies; 1 on librarians needs analysis and 2 on the design and development of microlearning training content (videos) to train librarians on the use of the abcd library management http://dx.doi.org/10.25304/rlt.v31.2930 research in learning technology citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 3 (page number not for citation purpose) system. the training content were deployed in a dedicated youtube channel https:// www.youtube.com/@ireneshubiisibika1609/videos and also uploaded to the available library computers. librarians were then free to upload them to their mobile devices for use whenever needed. this encouraged self-regulated learning. the user-perceived benefits of microlearning for this study perceived benefit embodies the perceived usefulness and ease of use adopted from the tam model (davis 1989). microlearning has been widely used for workplace training and research has shown a significant improvement and outperformance of employees when learning was performed using microlearning (bannister, neve, and kolanko 2020; cheng, liu, and wang 2017; gross et al. 2019; ichiuji et al. 2022; pascual et al. 2018; sawarynski and baxa 2019; wakam et al. 2022). shamirinbal and blau (2020) employed microlearning in designing teacher professional development (tpd) for ict teacher leaders and realized that microlearning is appropriate for pedagogical innovations. reinhardt and elwood (2019) researched the possibility of using microlearning and personal learning environments to promote a growth mindset in learners. they analysed how university faculty responsible for the preparation of teachers face challenges in integrating technology into their duties. they discovered that microlearning allows the professional development of faculty and students to be situated in a flexible way that can be integrated into everyday activities supporting individual learning aims and goals. microlearning enables learners to practice learning on relatively small, focused learning units and activities usually completed in < 10 min and accessible on multiple devices (shail 2019). it is an effective strategy because of its learner-centred, easy-toaccess, interactive, and well-designed features. the successful use of microlearning depends on various aspects and measurements including learning content, duration, curriculum level, form, process, mediality, and learning type (chai-arayalert and puttinaovarat 2020; redondo et al. 2021; yin et al. 2020) and conceals the boundary between formal and informal learning. it allows collaboration between each learner to access the skill set that will push their development further, while developing self-regulated, lifelong learning capacities (buchem and hamelmann 2010). microlearning user acceptance the success of microlearning is mainly dependent on the motivation for self learning. the learner’s lack of motivation may lead to poor reception of the microlearning lessons (ogange and mishra 2021). without acceptance, discretionary users will seek alternatives. acceptance has been conceived as an outcome variable in a psychological process that users go through in making decisions about the use or unuse of a particular technology. it is anticipated that librarians may quickly adopt and use microlearning for its many benefits, such as self-controlled learning. given the increasing interest and demand for microlearning in workplace training, studying the acceptance and usage of microlearning is paramount. several studies on microlearning have focused on its characteristics, traits, strengths, and limitations (wen and zhang 2015; zhang and west 2019) with few focusing on the practical implementation and attitude/acceptance of microlearning among users. according to puah et al. (2022), one of the most challenging impediments to overcome is employee resistance toward technology. http://dx.doi.org/10.25304/rlt.v31.2930 https://www.youtube.com/@ireneshubiisibika1609/videos https://www.youtube.com/@ireneshubiisibika1609/videos i.s. isibika et al. 4 citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 (page number not for citation purpose) challenging aspects of microlearning microlearning has generally realized a positive shift in employee training. however, challenges alongside its implementation have brought about discussions (aldosemani 2019; isibika et al. 2022; wen and zhang 2015; zhang and west 2019) in training and development on matters such as suitability, that is, if every subject can be trained using microlearning and accessibility for all. for instance, not all trainees are digitally literate. the use of microlearning requires one to be digitally literate to access digital learning tools and participate in e-activities. the librarians for this study were all digital-savvy and hence suitable for this microlearning training. microlearning is also neither merely the act of uploading learning materials online nor creating a complete learning experience into small bites running in a few seconds/minutes. its  foremost goal is to close knowledge and performance gaps identified at the comfort of trainees. thus, challenges such as personalising training content to fit every individual need, updating content in relevance and structure, and meeting trainees’ needs timely are still concealing the full implementation and utilisation of microlearning at a large scale. it is thus vital to determine the challenges experienced during training to develop possible solutions since some challenges can only be known after the implementation phase. this study aims to ascertain the challenges encountered by  librarians in using microlearning for training. conceptual model of this study and research questions this research aims to determine the influence of perceived benefits and challenges on the acceptance of microlearning for training librarians. the librarians’ acceptance of microlearning is dependent on the perceived usefulness (pu) and perceived ease of use (peou). figure 1 shows the conceptual model. specific research questions are addressed as follows: (1) what are the perceived benefits (pu and peou) of microlearning and the developed content for the librarians? (2) what are the challenges that librarians face in using microlearning and the developed content that influence the perceived benefits? (3) what are the differences between the pu and peou of previous traditional training on abcd and the microlearning training on abcd? figure 1. conceptual framework derived from tam model (davis 1989). perceived usefulness microlearning developed content perceived ease of use microlearning developed content acceptance of microlearning challenges of microlearning and developed content http://dx.doi.org/10.25304/rlt.v31.2930 research in learning technology citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 5 (page number not for citation purpose) (4) what is the relationship between the benefits (pu and peou) of microlearning and the acceptance of using microlearning for training? (5) to what extent do the librarians differ on their perceived benefits of microlearning according to when their previous (traditional) training took place? methodology this study examined the user-perceived benefits of using microlearning and the developed training content to train librarians and whether these benefits influence the acceptance of microlearning. perceived usefulness and perceived ease of use have been highlighted in prior literature to be prominently influential in enhancing the intention to adopt and accept different information technologies. four determinants were identified: perceived usefulness, perceived ease of use, challenges, and acceptance of microlearning. instrument design the study employed a quantitative methodology. to answer the research questions, a self-administered five-point likert-scale questionnaire labeled as ‘strongly disagree’, ‘disagree’, ‘neutral’, ‘agree’, and ‘strongly agree’, and ranged from 1 to 5, respectively, was used. the questionnaire format was predominantly closed with a small number of open questions. the respondents were informed that anonymity was guaranteed and that the data would be used for research only. research participants the respondents of this study were all librarians who worked in the selected university libraries in tanzania and confirmed having accessed and used the developed training content. those who had not yet used the content for training were automatically disqualified from participating in this survey. the developed content was given to the librarians 3 months before conducting this study. this ensured enough time for the librarians to interact with the content. participants’ demographic data are highlighted in table 1. data collection the research team identified and recruited ‘microlearning head librarians’ who acted as champions for the microlearning training, encouraging peers to complete and use developed content for training purposes. the ‘microlearning head librarians’ were also responsible for the closer follow-up of the data collection exercise and sometimes clarified questions amongst their peers. data analysis the collected data were analysed using spss v.22. for the first three research questions, we employed descriptive analysis. descriptive analysis interprets information patterns that might emerge from data and summarises the findings http://dx.doi.org/10.25304/rlt.v31.2930 i.s. isibika et al. 6 citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 (page number not for citation purpose) meaningfully. spearman correlation analysis was conducted to explore the relationship between pu  and peou regarding the acceptance of microlearning as a training format. the instrument is checked for reliability based on cronbach’s alpha value. the cronbach alpha (α) is commonly used to verify the internal consistency reliability of constructs. table 2 displays the cronbach’s alpha (α) values for each of this study’s constructs and all computed values are above the 0.7 standards recommended by (hair, ringle, and sarstedt 2011), which suggests adequate reliability and high internal consistency. results perceived benefits (pu and peou) of microlearning training format (rq1) in the self-administered questionnaire, librarians were asked how they perceived microlearning usefulness and ease of use. in general, librarians’ perception of the perceived benefit of microlearning is positively strong pu (m = 3.96, standard deviation [sd] = 0.55), peou (m = 3.81, sd = 0.57). the microlearning peculiarities most useful to the librarians included the increasing availability of training resources and higher retention of learned skills. for the perceived ease of use, flexibility of microlearning enabled them to learn more efficiently while the convenience of learning and training on the job was advantageous to them. these results imply that using microlearning for training librarians will conveniently provide access to training materials whenever needed and thus increase their performance. the results prove that pu and peou are important aspects of technological acceptance even more so for new pedagogical technologies such as microlearning. table 1. demographic profile of respondents. demographics no of respondents percentage (%) which university do you belong to? mzumbe university sua juco cucom mweka 26 16 7 9 6 40.6 25.0 10.9 14.1 9.4 years of interaction with the abcd system < 1 year 1–5 years 5–10 > 10 years 9 28 17 10 14.1 43.8 26.6 15.6 if you have ever received training on the abcd circulation module yes no 57 7 89.1 10.9 frequency of training never < 1 year ago 1–3 years > 3 years 10 36 15 3 15.6 56.3 23.4 4.7 http://dx.doi.org/10.25304/rlt.v31.2930 research in learning technology citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 7 (page number not for citation purpose) perceived benefits (pu and peou) of the microlearning developed content (rq1) the librarians were also asked to indicate their perception of the developed training content. the participants strongly perceived the developed training content to be useful and easy to use pu (m = 3.78, sd = 0.59) and peou (m = 3.81, sd = 0.55). the results show that using the developed videos to train the librarians on the use of the abcd system was poignant. it can be concluded that using microlearning will enhance librarians’ job performance and effectiveness while increasing their knowledge and skills. challenges faced by librarians in using the microlearning training format and developed content (rq2) results from table 3 show the challenges experienced by librarians while using microlearning. the mean score (m = 3.20) is elaborated by challenges such as a lack of in-person/physical interaction with peers. librarians expressed this as a challenge table 2. constructs, sample items, reliability coefficients indices. construct no. of items sample items cronbach’s alpha perceived usefulness of microlearning 7 microlearning improves the retention of learned skills and knowledge 0.827 perceived ease of use of microlearning 6 the control over training allows me to learn whenever needed 0.813 perceived usefulness of content 6 developed content reduces the time and effort required to learn about the abcd circulation module 0.845 perceived ease of use of content 6 it is easy for me to remember how to perform tasks from the developed content 0.870 challenges of microlearning training format 7 lack of available infrastructure (e.g. computers, smartphones) to access microlearning training 0.764 challenges of the developed content 4 language barrier in the developed content 0.902 accessibility 4 the microlearning training format promotes re-accessibility 0.789 flexibility 3 the microlearning training format encourages self-paced learning 0.754 effectiveness 3 the microlearning training format enhances knowledge retention and upgradation 0.765 interaction 3 the microlearning training format supports peer interactions 0.826 acceptance 4 i am satisfied with microlearning as a training technique 0.707 http://dx.doi.org/10.25304/rlt.v31.2930 i.s. isibika et al. 8 citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 (page number not for citation purpose) because they are used to face-to-face classroom-based training. in addition, the lack of available infrastructure in the libraries to access microlearning training was also seen as a challenge. however, the librarians opted to train using their mobile devices to benefit from the training. as for the developed content (m = 2.89), few librarians experienced a lack of clarity of the developed content. to gain a deeper understanding of this result, a crosstabulation was run between the university they belong to, years of interaction with the system, and the challenges encountered. results indicate that for the universities that have used the abcd system for more than 10 years, more than half of their librarians, who had less than 5 years of interaction with the system reported a lack of clarity of the developed content and the content being too advanced for them. these results show the unfamiliarity of the abcd system by librarians and hence a need for more frequent training (microlearning) on using the abcd. the differences between the previous traditional training on abcd and the microlearning training on abcd (rq3) the study sought to investigate whether librarians perceived a difference (positive or negative) between previous training on the usage of the abcd system and the current microlearning training. previous traditional training included face-to-face training, reading the software manual for instructions, workshops, seminars, etc. four main scales were used to determine the differences: accessibility, flexibility, effectiveness, and interaction. table 4 shows the results. the results reveal that the accessibility, flexibility, and effectiveness of microlearning brought about a difference making the microlearning training technique a better preference for training. even though librarians expressed a lack of physical human interaction during the training, they still highly ranked the whole microlearning experience. accessibility, flexibility, and effectiveness of training have been emphasised as important factors in determining the impact that training sessions could have on employees and so these results reemphasise the importance and therefore accentuate why microlearning is apt. it can conclusively be said that microlearning should be implemented as a new training technique for librarians. table 4. difference between microlearning and traditional training format. differences mean sd accessibility 3.97 0.59 flexibility 3.91 0.53 effectiveness 3.90 0.59 interaction 3.19 0.85 table 3. challenges of microlearning format and developed content. challenges mean sd challenges of the microlearning format 3.20 0.72 challenges of the developed content 2.89 0.94 http://dx.doi.org/10.25304/rlt.v31.2930 research in learning technology citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 9 (page number not for citation purpose) the relationship between the benefits of microlearning and the acceptance of using microlearning for training (rq4) this study sought the relationship between the perceived benefits (pu and peou) of microlearning as a training format and its acceptance. the acceptance scale included four statements on acceptance: i will use microlearning to learn other skills, i am satisfied with microlearning as a training technique, i will recommend the use of microlearning to other people, and i do not like the microlearning technique. a multiple regression analysis was conducted to determine the relationship between the perceived benefits and the acceptance of microlearning. this statistical method was used to study the strength of the relationship between perceived benefit (pu and peou) and acceptance of the microlearning format. results reveal a significantly strong positive correlation between pu (r = 0.501, p  < 0.001) and peou (r = 0.520, p < 0.001) demonstrating that both perceived usefulness and ease of use positively influence the acceptance of microlearning as a training technique for the librarians. the regression results indicate that the model explained 30.0% of the variance and that the model was a significant predictor of acceptance of microlearning, f(2,61) = 13.05, p ≤ 0.001. both perceived benefits positively influence the acceptance of microlearning with perceived ease of use contributing more significantly (β = 0.330, p ≤ 0.043) than perceived usefulness (β = 0.256, p = 0.114). if perceived ease of use increases, the acceptance of microlearning increases by 0.430 times. the extent to which the librarians differ in their perceived benefits of microlearning when compared with when their previous (traditional) training took place (rq5) the study investigated whether there is a significant difference between the frequency of training and the perceived benefit of using microlearning for training. the frequency of training was grouped into four categories as seen in table 1. a one-way analysis of variance (anova) test was performed between pu and peou categories. results show that there is no statistically significant difference between training frequency and pu (f (3,54) = 0.89, p = 0.45) and peou (f (3,54) = 0.64, p = 0.60). these findings indicate no difference between the perceived usefulness and ease of use of the microlearning training format amongst librarians regardless of when they received their last training. the results indicate that frequency of training has no moderating role to play in determining the perceived benefit of microlearning meaning that whether they just received training or not, microlearning is still very beneficial and thus validates the benefits of microlearning to librarians’ training regardless of their training frequency. discussion the main aim of this study is to determine the influence of user-perceived benefits (pu and peou) and challenges on the acceptance of microlearning for training. the tam model (davis 1986) was adopted to achieve this aim. perceived usefulness and ease of use of microlearning overall, librarians perceived the microlearning technique useful since microlearning enhanced their learning, improved their retention of learned skills, and saved their time. http://dx.doi.org/10.25304/rlt.v31.2930 i.s. isibika et al. 10 citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 (page number not for citation purpose) our results corroborate findings from other researchers (aldosemani 2019; redondo et al. 2021; reinhardt and elwood 2019) who discovered that such characteristics of microlearning outweigh other training techniques. the study’s findings suggest that librarians also perceived microlearning as easy to use. this is because of the flexibility of microlearning that allows them to learn more efficiently, the control over learning that allows them to train whenever needed, and the convenience to learn and train on the job. the perceived benefits of the librarians on the microlearning training format as revealed by this study show that using microlearning for training will positively impact skills generation and knowledge gain for the librarians. these findings are also in line with study by jomah et al. 2016; kovachev et al. 2011; shail 2019. the results of this research suggest that librarians will benefit from using microlearning for training. this implies that there will be a significant increase in the use of microlearning for skill and knowledge requisition eventually leading to a revamp of their competencies. perceived usefulness and ease of use of the developed content the study’s aim to determine the perceived usefulness and ease of use of the developed content enriched and provided an in-depth understanding of the developed microlearning content. the librarians perceived the developed content as useful because it increased their professional abilities and productivity while enhancing their skills and understanding of the abcd library management system. the developed content also boosted the librarians’ effectiveness in using the abcd system. the librarians signified the ease of use as more influential than its usefulness. they denoted that the developed content was easy to use since it became easy for them to remember how to perform tasks on the system enhancing their capabilities to use the system. these findings align with those of lee-fiedler (2021) who discovered that nurse educators were more satisfied with the ease of use of microlearning compared with its usefulness. the use of a library management system augments service delivery of library operations and hence librarians must be appropriately skill-equipped to use the available systems. the results have shown that using the microlearning approach to train librarians will increase the rate at which librarians acquire these skills. challenges using microlearning and the developed content the implementation of new technology brings about several challenges. librarians used the microlearning approach despite the challenges they faced. the biggest challenge for librarians was the lack of in-person interaction with peers during the training sessions. the underlying reason for this is that librarians are habituated to the classroom learning environment. this is also indicated by ikolo and nongo (2022) who discovered that librarians’ training sessions are usually organised in the form of seminars/workshops. however, this challenge will be overcome by the continual use of microlearning for their training needs, which will bring about the many benefits and advantages of microlearning. another challenge encountered by librarians is the lack of library computers to access microlearning training. despite this, librarians still used their mobile devices to train using microlearning. it is thus advisable to install enough computers in these libraries to fully grasp the benefits of microlearning. the use of electronic devices facilitates mobility, dynamism, and flexibility, all of which are inherent to microlearning (redondo et al. 2021). http://dx.doi.org/10.25304/rlt.v31.2930 research in learning technology citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 11 (page number not for citation purpose) librarians expressed concern about the lack of clarity regarding the developed training content. the results indicated that for universities that used the abcd system for more than 10 years, the librarians who had less than 5 years of interaction with the system reported a lack of clarity in the developed content. research shows that the more users use a system, the less severity of challenges and usability problems (mclellan, muddimer, and peres 2012). for these librarians, the lack of clarity may have stemmed from their inexperience in using the system and, hence, having less time to interact with it. this means they do not fully understand its functionality and therefore fail to gain clarity on what the training content is about. self-learning may have also limited the developed content’s understanding since many librarians are not used to this sort of learning. this study, therefore, suggests that more frequent use of microlearning for training the librarians on the use of the abcd system, will equip them with more abcd skills and thus increase their understandability of the system use. traditional training versus the microlearning training librarians have been exposed to face-to-face training sessions to equip them with proper knowledge and skills, but technology use, pedagogical and didactic learning technologies, different training avenues, and the ever-decreasing budget have made it difficult to meet the ever-changing training needs of librarians (isibika et al. 2021). the acceptance of microlearning by the librarians is concretised with how they perceive the differences in the training methods and to what extent this difference is significant. this study suggests that if librarians notice a difference in training formats, they are likely to accept and adopt the use of microlearning for training. as they adapt, they will perceive its benefit and no longer perceive the lack of human interaction as a challenge. perceived benefits versus acceptance of microlearning the findings show a positive correlation between pu and peou and acceptance of microlearning suggesting that the perceived benefit of microlearning is key to its successful use. evidently from the findings, when peou increases, the acceptance of microlearning increases. this implies that workplaces, organisations, and training providers may leverage the peou of microlearning to encourage its more use in employee training. the results are opposite to those from other studies (ahmed and ward 2016; dos santos, miguel, and okazaki 2015), which mainly refer to pu being a stronger predictor of acceptance of technology when compared with peou. however, these results may be influenced by the fact that the librarians liked using videos to learn skills when compared with their normal training avenues. we, therefore, suggest that it is imperative to foster librarians’ motivation to use the microlearning training format for training, as it will quickly and cost-effectively impart new knowledge and skills, increasing productivity in their daily activities. training frequency against perceived benefits of microlearning the study investigated whether there is a significant difference between the training frequency and the perceived benefit of using microlearning for training. the http://dx.doi.org/10.25304/rlt.v31.2930 i.s. isibika et al. 12 citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 (page number not for citation purpose) study believes that the lesser the training frequency, the more the perceived benefit of microlearning. if the librarians lack frequent training on using the abcd system, they will find microlearning training beneficial because of its advantage of anywhere-anytime training sessions. even though librarians mentioned a lack of in-person interaction with peers during training, what is interesting to discover is that there is no significant statistical difference between the perceived benefit of microlearning training over the frequency of training. the frequency of training has no moderating role to play in determining the perceived benefit of microlearning. the results validate the benefits of microlearning to librarians’ training regardless of their training frequency. implications this study demonstrates the influence of perceived usefulness and ease of use on the acceptance of microlearning as a training format for librarians. the positive relationship between the variables (pu and peou) and acceptance strengthens the relevance of adoption of new educational technologies for employee training. despite challenges, librarians are still keen on using microlearning for training and universities should ensure the availability of enough computers to enable easy access to microlearning content. this will bring forth a much-desired difference in training practices and overcome training barriers. conclusion this study found that microlearning is perceived as useful, flexible, and effective in offering workplace training to librarians. these results are of great importance, as they allow for the consolidation of technological solutions in the area of microlearning that allow to train librarians vis-a-vis employees on specific topics in particular and adapt to the activities and changes that occur in today’s workplaces. the results provide a strong foundation for further studies to assess the outcomes of using microlearning in other professional disciplines and practices. disclosure statement no potential conflict of interest was reported by the author. funding this work was supported by the vlaamse interuniversitaire raad. references ahmed, e. & ward, r. (2016) ‘a comparison of competing technology acceptance models to explore personal, academic and professional portfolio acceptance behaviour’, journal of computers in education, vol. 3, pp. 169–191. doi: 10.1007/s40692-016-0058-1 aldosemani, t. i. (2019) ‘microlearning for macro-outcomes: students’ perceptions of tele gram as a microlearning tool’, in digital turn in schools—research, policy, practice. lecture http://dx.doi.org/10.25304/rlt.v31.2930 https://doi.org/10.1007/s40692-016-0058-1 research in learning technology citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 13 (page number not for citation purpose) notes in educational technology, eds t. väljataga & m. laanpere, springer, singapore. https://doi.org/10.1007/978-981-13-7361-9_13 bannister, j., neve, m. & kolanko, c. (2020) ‘increased educational reach through a microlearning approach: can higher participation translate to improved outcomes?’, journal of european cme, vol. 9, no. 1. doi: 10.1080/21614083.2020.1834761 brown, j., et al., (2018) workforce of the future: the competing forces shaping 2030. available at: www.pwc.com/people buchem, i., & hamelmann, h. (2010) ‘microlearning: a strategy for ongoing professional development’, elearning papers, vol. 21, no. 7, pp. 1–15. chai-arayalert, s. & puttinaovarat, s. (2020) ‘designing mangrove ecology self-learning application based on a micro-learning approach’, international journal of emerging technologies in learning, vol. 15, no. 11, pp. 29–41. doi: 10.3991/ijet.v15i11.12585 cheng, y. t., liu, d. r. & wang, v. j. (2017) ‘teaching splinting techniques using a just-in-time training instructional video’, pediatric emergency care, vol. 33, no. 3, pp. 166–170. doi: 10.1097/pec.0000000000000390 davis, f. d. (1986) a technology acceptance model for empirically testing new end-user information systems: theory and results [massachusetts institute of technology]. https://dspace. mit.edu/handle/1721.1/15192 davis, f. d. (1989) ‘perceived usefulness, perceived ease of use, and user acceptance of information technology’, mis quarterly: management information systems, vol. 13, no. 3, pp. 319–339. doi: 10.2307/249008 dolasinski, m. j. & reynolds, j. (2020) ‘microlearning: a new learning model’, journal of hospitality & tourism research, vol. 44, no. 3, pp. 551–561. doi: 10.1177/1096348020901579 de gagne, j. c., et al., (2019) ‘microlearning in health professions education: scoping review’, jmir medical education, vol. 5, no. 2, p. e13997. doi: 10.2196/13997 dos santos, r., miguel, l. & okazaki, s. (2015) ‘planned e-learning adoption and occupational socialisation in brazilian higher education’, vol. 41, no. 11, pp. 1974–1994. doi: 10.1080/03075079.2015.1007940 elearning industry. (2022) the benefits of employee training. available at: https://elearningindustry.com/benefits-of-employee-training gross, b., et al., (2019) ‘microlearning for patient safety: crew resource management training in 15-minutes’, plos one, vol. 14, no. 3, p. e0213178. doi: 10.1371/journal. pone.0213178 hair, j. f., ringle, c. m. & sarstedt, m. (2011) ‘pls-sem: indeed a silver bullet’, journal of marketing theory and practice, vol. 19, no. 2, pp. 139–152. doi: 10.2753/ mtp1069-6679190202 ichiuji, b. a., et al., (2022) ‘the effect of a microlearning module on knowledge acquisition in surgery clerkship students’, journal of surgical education, vol. 79, no. 2, pp. 409–416. doi: 10.1016/j.jsurg.2021.11.001 ikolo, v. e. & nongo, c. j. (2022) ‘training needs of medical librarians in nigeria: a survey’, health information & libraries journal, vol. 39, no. 2, pp. 155–165. doi: 10.1111/hir.12383 isibika, i. s., et al., (2021) ‘perceived training needs assessment of librarians in tanzanian academic libraries aimed at introducing microlearning intervention to training’, international journal of training research, vol. 19, no. 2, pp. 107–124. doi: 10.1080/14480220.2021.1896565 isibika, i. s., et al., (2022) ‘microlearning training intervention for librarians: design and development’, inted2022 proceedings, vol. 1, pp. 7432–7440. doi: 10.21125/inted.2022.1878 jomah, o., et al., (2016) ‘micro learning: a modernized education system’, brain. broad research in artificial intelligence and neuroscience, vol. 7, no. 1, pp. 103–110. available at: https://www.edusoft.ro/brain/index.php/brain/article/view/582/627. kovachev, d., cao, y., klamma, r. & jarke, m. (2011) ‘learn-as-you-go: new ways of cloud-based micro-learning for the mobile web’, in advances in web-based learning icwl 2011. icwl 2011. lecture notes in computer science, vol. 7048, eds h. leung, http://dx.doi.org/10.25304/rlt.v31.2930 https://doi.org/10.1007/978-981-13-7361-9_13 https://doi.org/10.1080/21614083.2020.1834761 http://www.pwc.com/people https://doi.org/10.3991/ijet.v15i11.12585 https://doi.org/10.1097/pec.0000000000000390 https://dspace.mit.edu/handle/1721.1/15192 https://dspace.mit.edu/handle/1721.1/15192 https://doi.org/10.2307/249008 https://doi.org/10.1177/1096348020901579 https://doi.org/10.2196/13997 https://doi.org/10.1080/03075079.2015.1007940 https://elearningindustry.com/benefits-of-employee-training https://elearningindustry.com/benefits-of-employee-training https://doi.org/10.1371/journal.pone.0213178 https://doi.org/10.1371/journal.pone.0213178 https://doi.org/10.2753/mtp1069-6679190202 https://doi.org/10.2753/mtp1069-6679190202 https://doi.org/10.1016/j.jsurg.2021.11.001 https://doi.org/10.1111/hir.12383 https://doi.org/10.1080/14480220.2021.1896565 https://doi.org/10.21125/inted.2022.1878 https://www.edusoft.ro/brain/index.php/brain/article/view/582/627 i.s. isibika et al. 14 citation: research in learning technology 2023, 31: 2930 http://dx.doi.org/10.25304/rlt.v31.2930 (page number not for citation purpose) e. popescu, y. cao, r.w.h. lau & w. nejdl, springer, berlin, heidelberg. https://doi. org/10.1007/978-3-642-25813-8_6 lee-fiedler, j. d. (2021) nurse educator’s experience using microlearning strategies: a basic qualitative study. phd thesis. capella university. mclellan, s., muddimer, a. & peres, s. c. (2012) ‘the effect of experience on system usability scale ratings’, journal of usability studies, vol. 7, no. 2, pp. 56–67. nikou, s. a. & economides, a. a. (2018) ‘mobile-based micro-learning and assessment: impact on learning performance and motivation of high school students’, journal of computer assisted learning, vol. 34, no. 3, pp. 269–278. doi: 10.1111/jcal.12240 ogange, b. & mishra, s. (2021) ‘introduction to microlearning 2’, commonwealth of learning. available at: http://oasis.col.org/bitstream/handle/11599/3877/2021_allela_introduction_ to_microlearning_course.pdf ?sequence=8&isallowed=y#:~:text=microlearning%20 supports%20flexible%20and%20self,the%20end%20of%20each%20lesson pascual, k. j., et al., (2018) ‘evaluating the impact of personalized stroke management tool kits on patient experience and stroke recovery’, journal of patient experience, vol. 5, no. 4, pp. 244–249. doi: 10.1177/2374373517750416 puah, s., et al., (2022) ‘investigating working adults’ intentions to participate in microlearning using the decomposed theory of planned behaviour’, british journal of educational technology, vol. 53, no. 2, pp. 367–390. doi: 10.1111/bjet.13170 redondo, r. p. d., et al., (2021) ‘integrating micro-learning content in traditional e-learning platforms’, multimedia tools and applications, vol. 80, no. 2, pp. 3121–3151. doi: 10.1007/ s11042-020-09523-z/figures/16 reinhardt, k. s. & elwood, s. (2019) promising practices in online training and support: microlearning and personal learning environments to promote a growth mindset in learners, igi global. available at: https://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-5225-6322-8.ch013 sawarynski, k. e. & baxa, d. m. (2019) ‘utilization of an online module bank for a research training curriculum: development, implementation, evolution, evaluation, and lessons learned’, medical education online, vol. 24, no. 1, p. 1611297. doi: 10.1080/10872981.2019.1611297 shail, m. s. (2019) ‘using micro-learning on mobile applications to increase knowledge retention and work performance: a review of literature’, cureus, vol. 11, no. 8, p. e5307. doi: 10.7759/cureus.5307 shamir-inbal, t. & blau, i. (2020) ‘micro-learning in designing professional development for ict teacher leaders: the role of self-regulation and perceived learning’, professional development in education, vol. 48, no. 5, pp. 734–750. doi: 10.1080/19415257.2020.1763434 taylor, a. d. & hung, w. (2022) ‘the effects of microlearning: a scoping review’, educational technology research and development, vol. 70, no. 2, pp. 363–395. doi: https://doi. org/10.1007/s11423-022-10084-1 wakam, g. k., et al., (2022) ‘adapting to the times: combining microlearning videos and twitter to teach surgical technique’, journal of surgical education, vol. 79, no. 4, pp. 850–854. doi: 10.1016/j.jsurg.2022.02.001 wen, c. & zhang, j. (2015) ‘design of a microlecture mobile learning system based on smartphone and web platforms’, ieee transactions on education, vol. 58, no. 3, pp. 1–5. doi: 10.1109/te.2014.2363627 yin, j., et al., (2020) ‘conversation technology with micro-learning: the impact of chatbot-based learning on students’ learning motivation and performance’, journal of educational computing research, vol. 59, no. 1, pp. 154–177. doi: 10.1177/0735633120952067 zainab, a. m., et al., (2018) ‘utaut’s performance consistency: empirical evidence from a library management system’, malaysian journal of library and information science, vol. 23, no. 1, pp. 17–32. doi: 10.22452/mjlis.vol23no1.2 zhang, j. & west, r. e. (2019) ‘designing microlearning instruction for professional development through a competency based approach’, techtrends, vol. 64, no. 2, pp. 310–318. doi: 10.1007/s11528-019-00449-4 http://dx.doi.org/10.25304/rlt.v31.2930 https://doi.org/10.1007/978-3-642-25813-8_6 https://doi.org/10.1007/978-3-642-25813-8_6 https://doi.org/10.1111/jcal.12240 http://oasis.col.org/bitstream/handle/11599/3877/2021_allela_introduction_to_microlearning_course.pdf?sequence=8&isallowed=y#:~:text=microlearning%20supports%20flexible%20and%20self,the%20end%20of%20each%20lesson http://oasis.col.org/bitstream/handle/11599/3877/2021_allela_introduction_to_microlearning_course.pdf?sequence=8&isallowed=y#:~:text=microlearning%20supports%20flexible%20and%20self,the%20end%20of%20each%20lesson http://oasis.col.org/bitstream/handle/11599/3877/2021_allela_introduction_to_microlearning_course.pdf?sequence=8&isallowed=y#:~:text=microlearning%20supports%20flexible%20and%20self,the%20end%20of%20each%20lesson https://doi.org/10.1177/2374373517750416 https://doi.org/10.1111/bjet.13170 https://doi.org/10.1007/s11042-020-09523-z/figures/16 https://doi.org/10.1007/s11042-020-09523-z/figures/16 https://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-5225-6322-8.ch013 https://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-5225-6322-8.ch013 https://doi.org/10.1080/10872981.2019.1611297 https://doi.org/10.7759/cureus.5307 https://doi.org/10.1080/19415257.2020.1763434 https://doi.org/10.1007/s11423-022-10084-1 https://doi.org/10.1007/s11423-022-10084-1 https://doi.org/10.1016/j.jsurg.2022.02.001 https://doi.org/10.1109/te.2014.2363627 https://doi.org/10.1177/0735633120952067 https://doi.org/10.22452/mjlis.vol23no1.2 https://doi.org/10.1007/s11528-019-00449-4 editorial enabling active learning the first conference of the association for learning technology (alt-c 94), sub-titled enabling active learning, took place at the university of hull last september. it provided a unique opportunity for lecturers, courseware developers, library staff, computing support staff, educational development staff and institutional managers to meet and discuss the effective implementation of educational technology. space has allowed only a small selection of the papers presented to be included in this special issue of alt-j devoted mainly to the conference, but these papers address the major issues discussed. enabling active learning through the use of educational technology requires students to have adequate access to computers and training in using them, lecturers to have the confidence and skills necessary for using technology in their courses, and crucially senior managers in institutions of higher education to take a proactive role in supporting innovation in teaching and learning, both financially and by adopting appropriate staffsupport policies. real learning occurs when students are motivated, when they create, practise and synthesize, and are given feedback on their progress. such activities require inputs, processing and outputs hence active learning. these issues were addressed at the conference in an attempt to tease out the unique benefits of using educational technology in teaching and learning, and to identify winning strategies. however, it is not sufficient only to focus on the opportunities for successful learning. it is also necessary to consider cost: cost in terms of software development, staff training, and the investment required in hardware. unfortunately, too few papers at the conference measured the benefits of educational innovations and the associated costs. this is a serious shortcoming when it is necessary to persuade senior management in institutions of higher education to allocate scarce resources probably away from research to support innovations in teaching and learning. this is a far from easy task in light of the current funding drivers in uk universities and the widely-held view that strong research leads to high-quality teaching. the latter view is supported by the recent thes analysis which suggested a strong 3 editorial correlation between a research-assessment rating of 5 and an excellent hefce teachingquality assessment (whatever the underlying reasons for the correlation). we therefore need to be very clear about the real benefits of using technology in teaching and learning. computer-aided assessment, for example, can be used not only to ensure objectivity in course-end examinations, but also to provide feedback to students on their progress. the latter is becoming increasingly difficult with the large classes now commonly found in the first year of many degree programmes. computer-mediated communication can provide better contact between staff and students. improved communications also provide opportunities for sharing information and teaching materials on a world-wide basis. simulations can expose students to situations which would otherwise be difficult if not impossible to set up on account of cost, inaccessibility, scale, danger, time or ethics, and offer the potential to reduce costs through replacement of real but expensive laboratory exercises. some of the papers in these proceedings discuss these benefits. clearly emerging from the conference was the enormous enthusiasm of the majority of delegates for the undoubted potential of educational technology. however, enthusiasm for it, and a deep belief in its benefits, are not sufficient. senior management in institutions of higher education need to be persuaded with pragmatic arguments if they are to agree to go down a technological route and adequately reward staff for their efforts. one of the problems confronting the innovator in justifying technological innovation is the difficulty of estimating costs in advance of implementation, but this should not stop developers monitoring the costs, nor lecturers measuring the benefits retrospectively. such evidence is crucial for ensuring long-term investment in the application of educational technology in teaching and learning, and for developing a mature partnership between senior management and educational technologists. gabriel jacobs simon heath (conference chair and guest joint editor), university of aberdeen 4 (page number not for citation purpose) 1 *corresponding author. email: d.b.allcoat@warwick.ac.uk research in learning technology 2018. © 2018 d. allcoat and a. von mühlenen. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 research in learning technology vol. 26, 2018 original research article learning in virtual reality: effects on performance, emotion and engagement devon allcoat* and adrian von mühlenen department of psychology, university of warwick, coventry, uk (received 12 june 2018; final version received 23 october 2018) recent advances in virtual reality (vr) technology allow for potential learning and education applications. for this study, 99 participants were assigned to one of three learning conditions: traditional (textbook style), vr and video (a passive control). the learning materials used the same text and 3d model for all conditions. each participant was given a knowledge test before and after learning. participants in the traditional and vr conditions had improved overall performance (i.e. learning, including knowledge acquisition and understanding) compared to those in the video condition. participants in the vr condition also showed better performance for ‘remembering’ than those in the traditional and the video conditions. emotion self-ratings before and after the learning phase showed an increase in positive emotions and a decrease in negative emotions for the vr condition. conversely there was a decrease in positive emotions in both the traditional and video conditions. the web-based learning tools evaluation scale also found that participants in the vr condition reported higher engagement than those in the other conditions. overall, vr displayed an improved learning experience when compared to traditional and video learning methods. keywords: vr; education; experience; mood this paper is part of the special collection mobile mixed reality enhanced learning, edited by thom cochrane, fiona smart, helen farley and vickel narayan. more papers from this collection can be found here introduction interactive technology is progressing at an incredibly fast rate, and advances in virtual reality (vr) technology have led to many potential new applications. commercial vr headsets are widely used for entertainment purposes, with many individuals’ experiences of vr being from video games and other widely distributed media, as these media are widely advertised and well known, leading to higher popularity. however, vr has broader application possibilities, thanks to significant advances in the technology, including the technology now available in a mobile format. vr technologies allow the user to see and interact with virtual environments and objects. modern vr is delivered through a headset, which allows the user to see – and in some cases, hear – the 3d environment. in this way the user is totally immersed in the virtual environment, as it replaces the physical environment around them. immersion and engagement can be considered intrinsically linked in virtual environments mailto:d.b.allcoat@warwick.ac.uk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v26.2140 https://journal.alt.ac.uk/index.php/rlt/pages/view/collections d. allcoat and a. von mühlenen 2 citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 (page number not for citation purpose) (mcmahan 2003). mount et al. (2009) discussed the relationship between immersion, presence and engagement. they explored what it means for a learner to be immersed and considered immersion and engagement in 3d virtual environments, to outline how 3d virtual environments can be used to enhance learner engagement. vr boasts a number of features that could be useful for education: it presents environments in 3d, it is interactive and it is able to give audio, visual and even haptic feedback. presenting learning materials in 3d can be especially beneficial for teaching subjects where it is important to visualise the learning materials (e.g. in chemistry or in engineering). though visualising is one of the most obvious benefits of vr, this could also be accomplished with simple video. however, videos are passive learning objects, whereas vr allows for a direct interaction with the environment. interactivity and feedback can be valuable for all subjects, as there are specific benefits of interactive learning because it promotes active learning instead of passive learning. the usefulness of vr in education might also depend on the type of learning. learning styles theories suggest that there are various ways to learn, and some individuals learn better with some methods than others, as they have different approaches to information processing. the well-known visual–auditory–kinaesthetic learning styles model (barbe, swassing, and milone 1979) suggests there are three types of learning styles: visual, auditory and kinaesthetic. vr allows all three of these learning styles to be targeted in one application, as vr headsets allow for complex visual renderings, audio and movement tracking. though there has been much contention over learning styles theories, as discussed in the following, having one learning environment that can encompass multiple learning styles could be very beneficial as it would be suitable for a much wider range of individuals. other learning styles models include the importance of learning through different perceptual modalities, many of which are able to be targeted in vr (for an overview of various learning styles theories, see cassidy 2004). it has been suggested that having a variety of learning methods is valuable; gaytan and mcewen (2007) concluded that it is beneficial to use a variety of instructional methods to appeal to students’ learning preferences. vr activities could be designed to include multiple learning methods, so learners can choose to engage with the learning materials in the manner that interests them the most, as students have a preference for multiple modes of information presentation (lujan and dicarlo 2006). scholars now are more critical of learning styles theories (e.g. pashler et al. 2008; riener and willingham 2010), stating that, though there are many theories, there is little empirical evidence for learning styles. however, others still consider it important to be aware of varying sensory modalities and learning approaches because of students’ differing learning habits and preferences (e.g. hawk and shah 2007; kharb et al. 2013). the impact of learning styles on e-learning is also debated (truong 2016), including how best to design adaptive virtual learning environments whilst considering learning styles (kanninen 2008). there are potential benefits of targeting multiple methods of learning within vr, to allow for different information processing. this could be not only a result of learning methods and individuals’ preferences but also an illustration of how different types of information may be better presented in some formats than others (e.g. language may be best learnt with audio, whereas engineering may be better suited to visualisation). vr is not necessarily equally suitable for all subject areas; benefits of visualising are more significant in some subjects than others. as such, vr applications may be more suited to some areas of education than others. the revised bloom’s http://dx.doi.org/10.25304/rlt.v26.2140 research in learning technology citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 3 (page number not for citation purpose) taxonomy (bloom et al. 1956; see also anderson et al. 2001) suggests that there is not simply one way in which information is processed and learnt; instead it presents learning as a hierarchy of learning, consisting of six stages that involve cognitive processes from simplest to most complex (from remember, understand, apply, analyse and evaluate to create). it is suggested that these different types of learning can be processed differently; some methods of study that are used in education are only applicable to some subjects. debates, for example, are often good at engaging students with material that requires critical thinking (camp and schnader 2010; scott 2008) but are less suited to learning more concrete information, such as for sciences like physics or chemistry. vr, for example, may be less beneficial for learning to play a musical instrument that requires tactile feedback, such as a guitar, but may be particularly useful for topics where spatial arrangement is important or there are dynamic changes. though not many empirical studies have yet been conducted, vr has been compared to traditional learning in some areas. in one study a group of military students were taught with either the lecture-based teaching methods that are traditionally used for the subject material (corrosion prevention and control) or with an immersive vr-based teaching method (webster 2015). they found that whereas the traditional learning group had an improvement of 11%, the vr group had a higher improvement of 26%. bellamy and warren (2011) conducted a case study using simple online interactive simulations which mimicked real experiments. eighty-three per cent of their students reported that they found these online simulations helpful or very helpful, and their demonstrators stated that the students seemed much better prepared and more willing to answer questions when they had done the online simulations. these and other examples promote for learning the usefulness of simulated environments as alternatives to real-life scenarios. creating educational applications for vr could be a laborious and costly endeavour, so it is important to investigate whether these applications are useful for learning or not. therefore, explorative research can help answer whether the development of educational applications for this type of hardware is worth pursuing. as vr technology has only recently become more accessible and affordable, research in the past using vr in educational and pedagogic settings has typically used smaller sample sizes with less rigorous methodologies. this study looks to address that, considering not only test performance (used as a measure of learning) but also other outcomes of using vr for learning, such as effects on emotion and engagement. method participants all participants were first-year psychology students at the university of warwick (uk), who completed the study for course credit. a total of 99 participants (84 females, 15 males) who were 19 years of age on average were assigned randomly to one of three learning conditions: traditional (textbook style), vr and video. all participants reported normal or corrected-to-normal vision. the study was approved by the university’s humanities and social sciences research ethics committee, and all participants gave informed written consent and were aware of their right to withdraw at any time. http://dx.doi.org/10.25304/rlt.v26.2140 d. allcoat and a. von mühlenen 4 citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 (page number not for citation purpose) apparatus the questionnaires and learning materials were presented on a 19" lcd computer screen (1920 × 1080 pixels, 60 hz) using microsoft word and qualtrics. responses were collected through mouse and keyboard. a htc vive (xindian, new taipei, taiwan) (figure 1) was used for the vr condition. the headset weighs 550 g and displays a 3d environment via two oled displays (1080 × 1200 pixels per eye, 90 hz) with a field of view of 100 × 110 degrees. participants controlled the vr environment with the standard handheld htc vive controller. learning materials the learning materials used the same text and 3d model of a plant cell for all three conditions. the vr condition presented the model from the application ‘lifeliqe museum’ on the htc vive headset, allowing the participants to see and interact with a 3d model, with accompanying descriptive text (figure 1). the 3d plant cell model was fully interactive, allowing participants to highlight individual cell parts, change the size of the cell and rotate it. they could also teleport around the virtual room, with the plant cell appearing as a floating object in the room with them, which they could navigate around. a menu was available, virtually attached to one of the controllers, showing names of each part of the plant cell. participants could select one of these parts from the menu (e.g. the golgi apparatus) and it would highlight the part on the 3d model. this could also be done the opposite way, by selecting the part on the model, which would highlight the name on the menu. a written explanation of the purpose of each part of the plant cell was also available on this menu. the option of a narrator was disabled for this study, in order to remove audio learning as a confounding variable. figure 1. the htc vive headset and examples of the 3d model used as learning material for all conditions from the lifeliqe museum virtual reality environment. http://dx.doi.org/10.25304/rlt.v26.2140 research in learning technology citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 5 (page number not for citation purpose) the video condition used a 2d recording from the htc vive, matched from participants in the vr condition and presented on a computer screen. participants were informed that they could navigate this video at will (play or pause, fast forward, rewind), as they would in a distance learning scenario. this was a control to the vr condition as it presented the same visual information, with the same graphics, but did not have other vr features, such as interactivity and immersive 3d display. as such, this condition acted as a carefully matched control condition. the textbook condition used screenshots of the 3d model with the same accompanying text and presented them on a computer screen as a pdf file (figure 2). this ensured that all three groups had the same information and visuals to learn with, with the only difference being the format in which these materials were presented. rating scales an adapted version of the differential emotions scale (des, izard et al. 1974), with nine emotion categories (interest, amusement, sadness, anger, fear, anxiety, contempt, surprise and elatedness), was used to measure participants’ mood before and after the learning phase. participants were asked to rate to which extent the emotional adjectives, each represented with three words (e.g. surprised, amazed, astonished), applied to them on a scale from 1 (not at all) to 5 (very strongly). five of the categories related to negative emotions, and four related to positive emotions. the web-based learning tools (wblt) evaluation scale questionnaire (kay 2011) was used to measure engagement. the wblt evaluation scale asks participants to rate what they thought about the learning tools across 13 questions on a scale of 1 (strongly disagree) to 5 (strongly agree). the questions included items such as ‘the learning object helped teach me a new concept’ and ‘i would like to use the learning object again’. the questions can be grouped into the three categories ‘learning’, ‘design’ and ‘engagement’. procedure the procedure was the same for each participant, starting with a pretest and the des, followed by the learning phase. for the learning phase participants were instructed to learn as much as they could from the learning materials, and all conditions were given the same amount of time (7 min). after the learning phase, participants completed a post-test consisting of the same questions as the pretest, the des, the wblt and one figure 2. example of the textbook conditions, using the same text and screenshots from the lifeliqe museum virtual reality environment. http://dx.doi.org/10.25304/rlt.v26.2140 d. allcoat and a. von mühlenen 6 citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 (page number not for citation purpose) question that allowed for qualitative feedback. the improvement from pretest to posttest was used as the main measure of learning performance. this method was used in order to account for any participants with prior knowledge of the subject (plant cells). questions used for the test were either sourced directly from a british aqa biology a-level exam or were in the same style as these questions. results learning the 17 biology knowledge questions were marked as correct or incorrect and used in the calculation of an overall percentage correct, separately for each participant. the top half of table 1 shows the average knowledge scores in the pretest and in the posttest, together with the difference scores, as an indicator for learning. here the overall difference between pretest and post-test is referred to as ‘performance’ to differentiate it from the ‘learning’ scores of the wblt evaluation scale. the corresponding average confidence ratings are given in the bottom half of the table. the knowledge scores were analysed with a mixed-design anova with the betweensubject factor condition (textbook, video, virtual) and the within-subject factor test (pre-, post). the anova revealed a significant main effect for test, f(1,96) = 273.25, p < 0.001, η p 2 = 0.740, indicating that knowledge improved overall by 23.2% from pretest to post-test, and a significant test × condition interaction, f(2,96) = 6.80, p = 0.002, η p 2 = 0.124. the significant interaction was further analysed with two split-up anovas, separately for pretest and for post-test. the anova on the post-test data revealed a significant condition effect, f(2,96) = 3.51, p = 0.034, η p 2 = 0.068. post-hoc least significant difference (lsd) showed that participants in the vr condition scored significantly higher than participants in the video condition (56.5% vs. 43.9%, respectively; p = 0.009). the pretest anova showed no significant effect (p = 0.793). the confidence ratings showed a similar pattern of results as the knowledge data (see bottom half of table 1). the equivalent mixed-design anova revealed a significant effect for test, f(1,96) = 266.96, p < 0.001, η p 2 = 0.736, as a result of participants being more confident in the post-test than in the pretest (3.24 vs. 2.24, respectively), as well as a significant test × condition interaction, f(2,96) = 5.80, p = 0.004, η p 2 = 0.108, because of less confidence gain in the video than in the vr or textbook condition (0.71 vs. 1.12 and 1.18, respectively). table 1. number of participants (n), knowledge scores (percentage correct) and confidence ratings (1–5) in the pretest and post-test separately for the three conditions. condition n pretest post-test difference knowledge scores virtual 34 28.1% 56.5% 28.5% video 34 27.9% 43.9% 16.1% textbook 31 25.3% 50.2% 24.9% confidence ratings virtual 34 2.24 3.35 1.12 video 34 2.33 3.04 0.71 textbook 31 2.14 3.32 1.18 http://dx.doi.org/10.25304/rlt.v26.2140 research in learning technology citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 7 (page number not for citation purpose) the knowledge questionnaire data was further analysed by splitting the questions into two categories on the basis of bloom’s taxonomy (bloom et al. 1956). the first group (12 questions) related to the remembering of information, whereas the second group (5 questions) was more concerned with the understanding of information. the overall percentage correct in each category is shown in figure 3. a 3 × 2-way anova on the remembering scores showed a significant test × condition interaction, f(2,96) = 6.28, p = 0.003, η p 2 = 0.116. further split-up anovas and lsd tests revealed that in the post-test participants scored significantly higher in the vr than in the video and the textbook condition (53.1% vs. 40.6% and 43.6; p = 0.008 and p = 0.041, respectively). the corresponding analysis of the understanding scores also revealed a significant interaction, f(2,96) = 3.15, p = 0.047, η p 2 = 0.062; however, further tests showed no difference between vr and textbook, but scores in the video condition were lower than scores in the vr and textbook conditions (50.2% vs. 60.2% and 62.3%; p = 0.071 and p = 0.79, respectively). in summary, participants in the vr group showed better remembering than participants in the textbook group, but there was no difference between the two groups in terms of understanding. emotional response des ratings were split into the two categories: positive emotions (interest, amusement, surprise and elatedness) and negative emotions (sadness, anger, fear, anxiety and disgust), and average ratings are shown in figure 4. a 3 × 2-way anova with the factors condition and test on the positive emotions revealed a significant main effect of condition, f(2,96) = 13.24, p < 0.001, η p 2 = 0.216, and a significant interaction effect, f(2,96) = 31.40, p < 0.001, η p 2 = 0.395. the significant interaction was further analysed with three split-up t-tests, to see whether ratings changed from preto post-test. positive emotion significantly increased from 3.2 to 3.8 in the vr condition, t(30) = 4.73, p < 0.001, and significantly decreased in the video condition, t(33) = 4.92, p < 0.001, and in the textbook condition, t(30) = 4.37, p < 0.001. the corresponding anova on the negative emotions also revealed a significant interaction effect, f(2,96) = 4.37, p = 0.015, η p 2 = 0.084, which was a result of a significant decrease in negative emotion 0 20 40 60 80 100 vr video textbook te st s co re ( % ) learning condi�on remembering pre-test post-test pre-test post-test 0 20 40 60 80 100 vr video textbook te st s co re ( % ) learning condi�on understanding figure 3. percentage test scores and standard error mean (sem) (error bars) for the remembering questions (left) and for the understanding questions (right). vr, virtual reality. http://dx.doi.org/10.25304/rlt.v26.2140 d. allcoat and a. von mühlenen 8 citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 (page number not for citation purpose) (from 1.7 to 1.3) in the vr condition, t(30) = 4.20, p < 0.001, and no change in the video or textbook condition (both p’s > 0.50). learning experience average wblt ratings were grouped into the three categories ‘learning’, ‘design’ and ‘engagement’ and calculated separately for each category (see figure 5). three separate one-way anovas revealed a significant effect of condition for each of the three subscales (all p < 0.001). post-hoc lsd tests showed that both learning and engagement ratings were significantly higher in the vr than in the textbook condition (p = 0.005 and p < 0.001, respectively), and they were significantly higher in 1 2 3 4 5 vr video textbook r a� n g sc al e learning condi�on posi�ve emo�ons 0 1 2 3 4 5 vr video textbook r a� n g sc al e learning condi�on nega�ve emo�ons pre-test post-test pre-test post-test figure 4. mean rating and sem (error bars) for positive emotions (left) and for negative emotions (right). 1 2 3 4 5 vr video textbook learning condi�on wblt evalua�on scale learning design engagement r a� n g sc al e figure 5. mean wblt ratings and sem (error bars) for learning, design and engagement. wblt, web-based learning tools. http://dx.doi.org/10.25304/rlt.v26.2140 research in learning technology citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 9 (page number not for citation purpose) the textbook than in the video condition (p < 0.001 and p = 0.016, respectively). for design, ratings were significantly higher in the vr and textbook conditions than in the video condition (both p < 0.001), but there was no difference between the vr and the textbook condition. qualitative feedback qualitative data was also gathered; participants were asked as part of their online questionnaire: ‘what did you think of the format of the learning materials/the equipment used?’ the question was optional, and about half of the participants (n = 52) gave some written feedback. each participant who responded with qualitative feedback was grouped into positive, negative and mixed feedback, and the overall counts for each category and condition are given in table 2. multiple participants reported that the video learning material was ‘confusing’, with one participant stating that it was ‘engaging but confusing’ and another saying it was ‘difficult to navigate’. participants described the textbook-style learning materials as ‘basic’, ‘boring’ and ‘bland’. there were discrepancies in reports, with some participants stating the materials were ‘clear’ and ‘easy to learn from’ but others expressing that the materials were ‘unclear’ and the diagrams ‘weren’t very helpful’. on the other hand, participants found that the vr was ‘difficult’ to use, often clarifying ‘at first’, but found it more ‘engaging’, with one participant stating that it ‘made learning more exciting’ and another stating that it was ‘very useful and immersive’. the qualitative feedback suggests that because of the difficulty using the equipment ‘at first’, future studies may benefit from giving vr participants a trial period with the equipment first, to familiarise themselves with controls. similarly, video recording of vr would not be suitable as a primary learning condition (as opposed to a control condition as in this study) because of the jarring and confusing nature of the movement and interaction when they are not controlling it, with one participant stating that the ‘video felt all over the place’. discussion the aim of this study was to consider the effects of using vr headsets for learning. overall, participants in both the vr and the textbook-style conditions showed better learning than participants in the video condition. further breakdown of the learning data showed that participants in the vr condition were better at ‘remembering’ than those in the video and traditional conditions, and participants in both vr and traditional conditions were better at ‘understanding’ than those in the video condition. table 2. number of participants who responded with qualitative feedback in grouped types: positive, negative and mixed feedback. condition positive negative mixed virtual 5 3 5 video 2 13 2 textbook 1 15 6 http://dx.doi.org/10.25304/rlt.v26.2140 d. allcoat and a. von mühlenen 10 citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 (page number not for citation purpose) that the vr condition showed better test results compared to the video condition suggests that the learning in the vr condition is not a result of the graphics or visuals of the equipment, as these were the same in both conditions. instead, the learning appears to be attributable to either the 3d immersion or the interactivity of the vr environment. a further study may benefit from comparing vr to other active learning methods. this study compares interactive vr, an active learning method, to passive video watching and traditional textbook-based methods. the distinction between active learning and passive learning plays an important role in many existing educational theories. there is evidence that active learning is beneficial to students (e.g. pereira-santos, prudêncio and carvalho 2017), which could suggest that the benefits found for vr are simply the benefits of active learning. however, active learning is not always found to be better than passive learning (e.g. haidet et al. 2004); therefore, the benefits shown in vr may also be a result of other factors. the current results show a difference in learning stages as defined by bloom’s taxonomy; further research into the other stages would be of interest. this study looked at the lower ends of the learning hierarchy, remembering and understanding. vr may compare differently to traditional methods for applying, analysing, evaluating and creating. in particular, the 3d aspects of vr, along with the interactivity it affords, may be beneficial for ‘creating’ in many subjects. alternatively, participants’ unfamiliarity with the equipment, which they hadn’t used before, may mean that improvements of the vr condition were diminished, as individuals need time to adapt to new technology systems (e.g. cook and woods 1996). this could explain why participants in the vr condition were not significantly better at ‘understanding’ compared to participants in the traditional condition. vr was also found to have a very positive impact on mood, with participants having an overall increase in positive emotions and an overall decrease in negative emotions. conversely the other conditions showed a decrease in positive emotions. enjoyment has been previously linked as an important part of student performance (e.g. goetz et al. 2006; valiente, swanson, and eisenberg 2012). this suggests that using vr headsets can have a positive impact on the learning experience. the wblt evaluation scale also shows that engagement can be increased through the use of vr. the importance of student engagement has been recognised previously (e.g. kuh 2009; strydom, mentz and kuh 2010; wolf-wendel, ward and kinzie 2009). participants also rated the vr environment higher for learning, demonstrating that they felt that they had learnt better from the vr. student self-rating of learning has been shown to be a valid measure of student performance (benton, duchon, and pallett 2013), with participants here reporting higher learning in the vr condition, which was found for ‘remembering’. the positive effects on emotion and engagement in vr are important benefits for both within and outside classroom learning (e.g. distance learning, self-teaching). these aspects of learning are sometimes overlooked, with the focus being on other outcomes, such as test scores. however, it has been demonstrated that individuals’ emotions, engagement and motivation are highly linked with each other and they are all important aspects of learning (pintrich 2003). this research has demonstrated how vr can replicate or complement traditional learning methods. it is important to consider how vr technology allows for learning beyond the classroom. the technology, though suitable for classroom use, is also particularly suitable for distance learning, self-teaching and other learning environments. http://dx.doi.org/10.25304/rlt.v26.2140 research in learning technology citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 11 (page number not for citation purpose) this can be achieved, as the equipment can allow for rich, detailed learning environments that can be programmed to any scenario. such vr environments can allow for learning that could not be replicated in reality (e.g. dangerous environments or experiments) or would be too costly to be accessible (e.g. expensive equipment or materials). future studies, for example, may want to consider the possible advantages of the auditory options available with equipment such as vr, which were not utilised for this project as they may be a confounding variable. as discussed, this could be of interest in relation to learning styles, which are prevalent in a number of learning theories, though the concept of learning styles has received some criticism (e.g. pashler et al. 2008). regardless of learning styles, there may be some benefit to including audio to increase immersion and engagement (e.g. paterson and conway 2014; wharton and collins 2011). many vr headsets also share the benefits of mobile learning, most obviously the vr headsets that run through mobile phones. though not as powerful and capable of detailed environments as pc-based vr headsets like the htc vive and oculus rift, these mobile headsets share many of the same benefits. though the headset used in the study, the htc vive, is currently only mobile with the use of a portable backpack pc, there is a new mobile, portable version of the htc vive headset called the ‘vive focus’. the vive focus is currently available for developers and is expected to be released later this year, which means that applications such as the one used in this study will be fully mobile, allowing for more flexible learning. overall, vr does seem to be a potential alternative to traditional textbook-style learning, with similar performance levels and improved mood and engagement. these benefits may have a longer-term impact on learning, such as improvements resulting from the learning experience. however, the results may be partially because of the novelty of the vr equipment, so the improvements may not be sustained over longitudinal studies. conversely, these improvements could increase over time, as individuals become more familiar with the equipment and more able to navigate it easily. therefore, further longitudinal studies are needed to address these questions. vr does show great potential, not only as an option to supplement or replace traditional learning methods, but to develop novel learning experiences that have not been used before. references anderson, l.w., et al., (2002) a taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives, longman, new york, vol. 41, p. 302. barbe, w., milone, m. & swassing, r. (1988) teaching through modality strengths, zanerbloser, columbus, oh. bellamy, m., & warren, a. (2011) using online practicals to support lab sessions, unpublished, edshare.soton.ac.uk. [online] available at: http://edshare.soton.ac.uk/id/document/243301. benton, s., duchon, d. & pallett, w. (2013) ‘validity of student self-reported ratings of learning’, assessment & evaluation in higher education, vol. 38, no. 4, pp. 377–388. doi: 10.1080/ 02602938.2011.636799. bloom, b. s., et al., (1956) taxonomy of educational objectives: the classification of educational goals: handbook i cognitive domain, longmans, green and co ltd, london. doi: 10.1177/001316445601600310. camp, j. & schnader, a. (2010) ‘using debate to enhance critical thinking in the accounting classroom: the sarbanes-oxley act and u.s. tax policy’, issues in accounting education, vol. 25, no. 4, pp. 655–675. doi: 10.2308/iace.2010.25.4.655. http://dx.doi.org/10.25304/rlt.v26.2140 edshare.soton.ac.uk http://edshare.soton.ac.uk/id/document/243301 d. allcoat and a. von mühlenen 12 citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 (page number not for citation purpose) cassidy, s. (2004) ‘learning styles: an overview of theories, models, and measures’, educational psychology, vol. 24, no. 4, pp. 419–444. doi: 10.1080/0144341042000228834. cook, r. & woods, d. (1996) ‘special section: adapting to new technology in the operating room’, human factors: the journal of the human factors and ergonomics society, vol. 38, no. 4, pp. 593–613. doi: 10.1518/001872096778827224. gaytan, j. & mcewen, b.c. (2007) ‘effective online instructional and assessment strategies’, american journal of distance education, vol. 21, pp. 117–132. goetz, t., et al., (2006) ‘a hierarchical conceptualization of enjoyment in students’, learning and instruction, vol. 16, no. 4, pp. 323–338. doi: 10.1016/j.learninstruc. 2006.07.004. haidet, p., et al., (2004) ‘a controlled trial of active versus passive learning strategies in a large group setting’, advances in health sciences education, vol. 9, no. 1, pp. 15–27. doi:10.1023/b:ahse.0000012213.62043.45. hawk, t. & shah, a. (2007) ‘using learning style instruments to enhance student learning’, decision sciences journal of innovative education, vol. 5, no. 1, pp. 1–19. doi:  10.1111/ j.1540-4609.2007.00125.x. izard, c. e., et al., (1974) the differential emotions scale: a method of measuring the meaning of subjective experience of discrete emotions, vanderbilt university, department of psychology, nashville, tn. kanninen, e. (2008) learning styles and e-learning, master of science thesis, tampere university of technology, vol. 12, pp. 1–76. doi: 10.4018/978-1-4666-4313-0.ch020. kay, r. (2011) ‘evaluating learning, design, and engagement in web-based learning tools (wblts): the wblt evaluation scale’, computers in human behavior, vol. 27, no. 5, pp. 1849–1856. doi: 10.1016/j.chb.2011.04.007. kharb, p. (2013) ‘the learning styles and the preferred teaching–learning strategies of first year medical students’, journal of clinical and diagnostic research, vol. 7, no. 6, pp. 1089–1092. doi: 10.7860/jcdr/2013/5809.3090. kuh, g. (2009) ‘what student affairs professionals need to know about student engagement’, journal of college student development, vol. 50, no. 6, pp. 683–706. doi: 10.1353/csd.0.0099. lujan, h.l. & dicarlo, s.e. (2006) ‘first-year medical students prefer multiple learning styles’, advances in physiology education, vol. 30, pp. 13–16. mcmahan, a. (2003) ‘immersion, engagement, and presence: a method for analyzing 3-d video games’, in the video game theory reader, routledge, london, pp. 67–86. mount, n., et al., (2009) ‘learner immersion engagement in the 3d virtual world: principles emerging from the delve project’, innovation in teaching and learning in information and computer sciences, vol. 8, no. 3, pp. 40–55. doi: 10.11120/ital.2009.08030040. pashler, h., et al., (2008) ‘learning styles’, psychological science in the public interest, vol. 9, no. 3, pp. 105–119. doi: 10.1111/j.1539-6053.2009.01038.x. paterson, n., & conway, f. (2014) ‘engagement, immersion and presence: the role of audio interactivity in location-aware sound design’, in oxford handbook of interactive audio, eds k.  collins, b. kapralos & h. tessler, 1st edn., oxford university press, oxford, pp. 263–280. doi: 10.1093/oxfordhb/9780199797226.013.016. pereira-santos, d., prudêncio, r. & de carvalho, a. (2017) ‘empirical investigation of active learning strategies’, neurocomputing, [online] available at: https://doi.org/10.1016/j.neucom. 2017.05.105. pintrich, p. (2003) ‘a motivational science perspective on the role of student motivation in learning and teaching contexts’, journal of educational psychology, vol. 95, no. 4, pp. 667–686. doi: 10.1037/0022-0663.95.4.667. riener, c. & willingham, d. (2010) ‘the myth of learning styles’, change: the magazine of higher learning, vol. 42, no. 5, pp. 32–35. doi: 10.1080/00091383.2010.503139. scott, s., (2009) ‘perceptions of students’ learning critical thinking through debate in a technology classroom: a case study’, the journal of technology studies, vol. 34, pp. 39–44. doi: 10.21061/jots.v34i1.a.5. http://dx.doi.org/10.25304/rlt.v26.2140 https://doi.org/10.1016/j.neucom.2017.05.105 https://doi.org/10.1016/j.neucom.2017.05.105 research in learning technology citation: research in learning technology 2018, 26: 2140 http://dx.doi.org/10.25304/rlt.v26.2140 13 (page number not for citation purpose) strydom, j. f., mentz, m. & kuh, g. d. (2010) ‘enhancing success in higher education by measuring student engagement in south africa’, acta academica, vol. 42, pp. 1–13. truong, h. (2016) ‘integrating learning styles and adaptive e-learning system: current developments, problems and opportunities’, computers in human behavior, vol. 55, pp. 1185–1193. doi: 10.1016/j.chb.2015.02.014. valiente, c., swanson, j. and eisenberg, n. (2011) ‘linking students’ emotions and academic achievement: when and why emotions matter’, child development perspectives, vol. 6, no. 2, pp. 129–135. doi: 10.1111/j.1750-8606.2011.00192.x. webster, r. (2015) ‘declarative knowledge acquisition in immersive virtual learning environments’, interactive learning environments, vol. 24, no. 6, pp. 1319–1333. doi:  10.1080/ 10494820.2014.994533. wharton, a., and collins, k. (2011) ‘subjective measures of the influence of music customization on the video game play experience: a pilot study’, game studies: the international journal of computer game research, vol. 11, no. 2. [online] available from: http://gamestudies.org/1102/ articles/wharton_collins [accessed: 16th november 2018]. doi: 10.4172/2161-0487.1000191. wolf-wendel, l., ward, k. and kinzie, j. (2009) ‘a tangled web of terms: the overlap and unique contribution of involvement, engagement, and integration to understanding college student success’, journal of college student development, vol. 50, no. 4, pp. 407–428. doi: 10.1353/csd.0.0077. http://dx.doi.org/10.25304/rlt.v26.2140 (page number not for citation purpose) 1 *corresponding author. email: victoria.marin@uni-oldenburg.de research in learning technology 2019. © 2019 v.i. marín. et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 research in learning technology vol. 27, 2019 original research article open educational resources for research training: quality assurance through a collaborative evaluation victoria i. marína*, martha lucía orellanab and nancy peréc acenter for open education research (coer), faculty of education and social sciences, carl von ossietzky universität oldenburg, oldenburg, germany; bunab creative, universidad autónoma de bucaramanga (unab), bucaramanga, colombia; cacademic unit, universidad de la república, montevideo, uruguay (received: 18 may 2019; revised: 4 october 2019; accepted: 21 october 2019; published: 18 november 2019) although it is considered that open educational resources offer vast pedagogical opportunities for any educational context, there have been only few studies so far that have linked their use or application in the field of research training, and even less works that have addressed their quality assurance for that context. as part of an inter-institutional project, the main aim of this article is the collaborative selection and evaluation of appropriate educational resources for research training. the mixed method approach of the article includes needs’ analysis of researchers in training through questionnaires and interviews. this was the starting point for the collaborative evaluation of educational resources using the agreed common criteria derived from the learning object review instrument (lori) evaluation instrument. this article suggests recommendations regarding the collaborative evaluation of educational resources and the use of lori, and suggestions for creators of educational resources for research training to facilitate the quality assurance of their materials. a website is being developed to bind together the resources that have met the quality criteria established in collaborative evaluation. keywords: evaluation criteria; higher education; distance education introduction students who initiate research tasks, either with a master’s or doctoral thesis, often find deficiencies in their training that make the research process more difficult (wang and li 2011). research supervisors also expose this situation. while master’s or doctoral students receive the support of their research supervisors in the process of research training, a balance between the supervisor’s accompaniment and the development of autonomy as a researcher is crucial. in addition, common issues regarding the thesis arise among researchers during training. as a result, researchers explore support possibilities available on the internet. examples of this type of support are online communities, such as the #phdchat generated on twitter (ford, veletsianos, and resta 2014), or specific communities, for instance covif (a learning mailto:victoria.marin@uni-oldenburg.de http://www.alt.ac.uk/� http://creativecommons.org/licenses/by/4.0/� http://dx.doi.org/10.25304/rlt.v27.2271� v.i. marín et al. 2 citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 (page number not for citation purpose) community for researchers in training in the field of educational technology; moreno and salinas 2011). the use of open educational resources (oer) as a possible source of content has not yet been exploited to address the shortcomings or the need to deepen knowledge in certain aspects of research training. the open and mobile educational resources for educational researchers training project, carried out by several universities in mexico, focused on the creation and use of an oer repository of mobile learning for educational researchers during training, it resulted in the production of 37 mobile oer and the development of a digital repository to lodge them (ramírez and burgos 2012). no study has focussed on identifying the existing oer, and making them available to researchers during training in any field. such a study would need to consider that oer are not always easy to locate or recover (atenas, havemann, and priego 2014), and that opportunities discussed around oer, especially those related to universal access to education have not been realised (caswell et al. 2008). a study locating oer also needs to consider the discussions around the quality of these resources and how to evaluate that quality (camilleri, ehlers, and pawlowski 2014). this research aims to select and collaboratively evaluate digital educational resources, especially open but not exclusively, and make them available to researchers during training. for this purpose, both the needs expressed by researchers during training and the existing criteria and instruments in the field of the evaluation of educational resources are considered. this effort is a part of an ibero-american inter-institutional project (investiga+) led by the autonomous university of bucaramanga (unab, colombia), which is oriented towards the strengthening of postgraduate research (orellana et al. 2016). based on the fact that there is a growing number of master’s and doctoral students worldwide (association of universities and colleges of canada (aucc) 2011; ministerio de educación nacional (men) 2013; snyder and dillow 2013), investiga+ seeks to contribute to supervisors’ training and practice. in this article we focus on researchers in training, although research supervisors could also benefit from the results of this work. framework training to be a researcher according to wisker (2012), research as a form of learning that values the creation and questioning of knowledge has become a central component of curricula throughout the world; the number of postgraduate and undergraduate students has increased, as has the number of research supervisors who are expected to encourage, support and train students to develop competencies, values and practices essential for research work. following are the responsibilities that taylor, kiley, and humphrey (2017) attribute to effective supervisors in relation to the expected achievements of researchers in training: to allow the student to be able to start and plan a research project; to acquire research competencies necessary to accomplish research and have adequate access to resources; and to develop creative, critical and analytical skills. however, supervisors do not have to assume these responsibilities by themselves, but should provide guidance and academic support, and the institution is expected to be a facilitator of structure, policies, conditions and resources that enable supervision (orellana 2016). http://dx.doi.org/10.25304/rlt.v27.2271� research in learning technology citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 3 (page number not for citation purpose) remote supervision, either completely or partially using online communication, adds to the challenges of supervision. despite these challenges of remote supervision, here are also advantages and opportunities. according to sussex (2008), the same fact of using different forms of communication and information exchange increases the quality of supervision by making it richer and more flexible than without the combination of media. these advantages and opportunities can also be harnessed by face-to-face supervision supported by information and communication technology (ict). ict also supports one of the main purposes of research training, that is, to achieve student autonomy as a researcher. according to adham et al. (2018), the training of students as researchers is enhanced by their interactions within professional and social networks, self-study and self-reflection. in the same way, caplan and graham (2008) suggest viewing the web as more than a tool, and also as an aid in the creation of learning environments that promote active student-centred learning and support the development of critical and high-level thinking skills. within the frame of the investiga+ project, the current work promotes the creation of a website where digital educational resources for research training, especially open resources, are linked. this website facilitates autonomous learning, communication and interaction in cross-disciplinary and multicultural environments, especially in online education, although the resources could also be used in the context of face-toface and blended learning. digital educational resources: learning objects (lo) and oers according to fernández-pampillón, dominguez, and de armas (2013), digital educational content or resources are digital resources used in the teaching-learning process of the courses taught by teachers or the collection of resources that a teacher or a student uses to pursue a course: a lesson plan, a calendar, the teaching guide, a proposal of activities, tutorials, ... [...] when the digital educational resource is created with the objective to be scalable, reusable, interoperable and accessible is considered to be a learning object. (fernández-pampillón, dominguez, & de armas 2013, p. 14) learning objects, as educational resources, are minimum units of information in multiple formats with an interactive nature, self-contained in the context of learning and oriented to a single objective through micro-learning units that include content, resources, activities and evaluation (del moral and cernea 2005). they also present features of accessibility (identifiable through metadata), interoperability (technical compatibility of the resource among different platforms and devices), reusability in other contexts, adaptability to situations and needs, and durability in the face of technological changes (del moral and cernea 2005). according to camilleri, ehlers and pawlowski (2014, p. 7), los are ‘materials used to support learning (that) can be broken down into (or constructed from), a number of elements which can be combined differently and reused in various scenarios’. when los, as minimum educational resources, are licensed with an open licence, they can be considered oer. the term oer was coined by the unesco (2002) and defined as ‘teaching, learning, and research resources that reside in the public domain http://dx.doi.org/10.25304/rlt.v27.2271� v.i. marín et al. 4 citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 (page number not for citation purpose) or have been released under an intellectual property license that allows their free use or re-purposing by others’ (atkins, brown, and hammond 2007, p. 4). oer as well as digital educational resources in general may include complete courses, course materials, modules or units, syllabus, books, research articles, videos, evaluation tools, interactive materials, databases, software, applications and any other useful educational material (unesco and commonwealth of learning (col) 2011). it is widely agreed that oer can offer great educational possibilities, especially for the developing countries (kawachi 2014), but their potential in distance education and in research training, through the reuse and sharing of digital educational resources, can be also considered (ramírez and burgos 2012). oers are openly available, often accessible through educational or institutional repositories but not always easy to find or retrieve (atenas, havemann, and priego 2014). likewise, just as the use of oer is open to any person, so is the creation of new oer, and therefore the pedagogical, technical and content quality can be variable, and educators might have problems judging the oer quality and relevance (hylén 2006). in comparison to commercially published resources, which follow traditional peer review, licensing and publication (wiley 2013), there are still doubts about the oer quality, which can be managed through different (centralised/decentralised; open/closed) processes (hylén 2006). as a main part of our study, it is a key to review what the literature proposes in relation to the quality of digital educational resources, primarily oer. ensuring the quality of digital educational resources quality is one of the most discussed topics in the field of digital educational resources (especially oer), and also one of the considered reasons why these resources are scarcely used (camilleri, ehlers, and pawlowski 2014). the same authors also emphasise that the potential of oer to transform the educational practice has not yet been exploited; and that innovative ways are needed for the creation and evaluation of oer, as well as an evolving empirical database on oer effectiveness. in order to assess and evaluate oer, and other educational resources, camilleri, ehlers, and pawlowski (2014) propose a social qualification where the evaluators are the users, and the results of the evaluation are available to other users of resources. according to mcgill (2011), the quality of educational resources is usually determined by their accuracy, the reputation of their author/originated institution, the technical production standards, the accessibility and the suitability for specific purpose. on the other hand, there are different approaches to the concept of quality, which include generic perspectives of quality related to quality management or quality assurance without taking into account the context of use, specific quality perspectives for learning, education and training (e.g. standards and guidelines to ensure quality in the european higher education area or unique criteria for excellence in technology-enhanced learning), and specific instruments for quality assessment (e.g. recommendation systems or peer reviews; camilleri, ehlers, and pawlowski 2014). these instruments could be ratings, rubrics or frameworks and facilitate the evaluation and selection of digital educational resources (merlot, the learning object evaluation instrument and the rubric to evaluate learner generated content, among others). in studies by yuan and recker (2015) and zawacki-richter and mayrberger (2017), detailed analyses of these instruments were carried out, including a comparison and evaluation of their elements. http://dx.doi.org/10.25304/rlt.v27.2271� research in learning technology citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 5 (page number not for citation purpose) both studies point towards the learning object review instrument (lori) (nesbit, belfer, and leacock 2007) as one of the most easily accessible evaluation instruments. lori has also been reviewed for several times and there are examples of empirical results of its implementation (yuan and recker 2015; zawacki-richter and mayrberger 2017). resting on the evidence from these studies, lori appears to be one of the most valid instruments for evaluating los and also the most inclusive in terms of the quality criteria or dimensions described in zawacki-richter and mayrberger (2017) -technical, pedagogical and intellectual property rights – and the spanish standard une 71632 (quality criteria for digital educational materials; fernández-pampillón cesteros et al. 2017) -pedagogical effectiveness, technological effectiveness and effectiveness regarding accessibility. lori is designed for individual or collaborative evaluation of multimedia los, although it can also be used in a generic way for los with other formats. lori comprises nine items, which can be specified when evaluating the educational resource, according to the mentioned quality criteria or dimensions (table 1). for each item, a rating of 1 to 5 stars is established, meaning 1 is the lowest score and 5 is the highest. examples of what is meant by assigning 1–5 stars to each of the items are specified in lori. in an effort to create an integrated platform for lo evaluation, lori and other evaluation instruments (e.g. une 71362 or the learning object evaluation metric) have been included in the learning evaluation object platform (loep). this platform facilitates collaboration in the evaluation of educational resources, and generates automatic scores derived from that joint evaluation (gordillo, barra, and quemada 2015). loep is an important tool for the current study, since it provides an organised system for the allocation of assessment of educational resources, and produces summary representations of their quality according to different instruments. these features are the reasons why loep and the integrated lori were used in this study for the collaborative evaluation of educational resources, ensuring quality criteria in the process and outcome of the process. table 1. quality items of learning object review instrument according to the quality criteria and dimensions. quality criteria (zawacki richter and mayrberger 2017) quality dimensions (fernández pampillón cesteros et al. 2017) lori items (nesbit, belfer, and leacock 2007) technical (usability, accessibility and reusability) technological effectiveness presentation design reusability compliance with standards effectiveness regarding accessibility accessibility interaction usability pedagogical (content, learning design and evaluation) pedagogical effectiveness content quality alignment of learning objectives motivation feedback and adaptation intellectual property rights (license) http://dx.doi.org/10.25304/rlt.v27.2271� v.i. marín et al. 6 citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 (page number not for citation purpose) methodology research objectives as a major purpose of the investiga+ project, a range of digital educational resources (existing and newly created) are expected to put at disposal of researchers during training. these resources include content related to the design and development of a research project. the objectives of this study contribute to the above-mentioned major purpose of the investiga+ project and are specified as follows: 1. to identify the relevant elements of structures and topics of digital educational resources for research training derived from the needs of researchers in training and the related literature. 2. to collaboratively evaluate and select existing digital educational resources according to the established quality parameters suitable for research training. therefore, following would be the research questions: 1. which elements of structures and topics of educational resources are relevant for researchers during training? 2. how appropriate, in terms of their quality, are the existing educational resources to be used in research training? phases of the study the study has been divided into different consecutive phases (see figure 1), starting with the literature review and data collection regarding elements of structures and topics of digital educational resources carried out through questionnaires and interviews with researchers in training. after this first identification of educational figure 1. phases of the study. http://dx.doi.org/10.25304/rlt.v27.2271� research in learning technology citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 7 (page number not for citation purpose) resources’ needs for researchers in training, a pilot phase to test the collaborative evaluation was completed through the use of the lori instrument. this allowed us to define elements of quality for an educational resource in the context of research training and, consequently, to generate some consensus around the use of lori for collaborative evaluation, which took place thereafter. these quality criteria are also the basis for the definition of an appropriate techno-pedagogical structure for the educational resources that are created in a later stage of the project (future work). the study was approved by the university’s institutional committee of ethics in research (code 062). instruments as part of a larger study, a mixed methodology was used in the initial analysis: online questionnaires were administered to research students in master’s and doctoral programmes in ibero-american universities, and online interviews were conducted with research students who voluntarily offered to be contacted in the online questionnaire. the questionnaire was validated by experts through the international research panel in educational technology (http://www.edutec.es/panel). the questionnaire included many items, but only the ones that concerned specific objectives of this study are presented. in the case of the questionnaire, the selected items were as follows: • classify in the order of priority from 1 to 5 different types of educational resources (e.g. self-contained course) to be considered in the design of support resources for research training. • select five topics of educational resources required for research training (e.g. methodological aspects). in both items, the research students were given the opportunity to include supplementary options. in the interview, the researchers during training delved into the characteristics of digital resources considered to support research. in addition, they explained their reasons for choosing specific topics of digital resources for research training. the next phase in the study included the use of loep by project researchers to evaluate an educational resource and the subsequent discussion on the suitability of each item from the context of research training and the type of lo found. based on the agreements for the use of lori, the project researchers proceeded to the collaborative and quantitative evaluation of educational resources. these resources were previously selected from 34 educational and institutional repositories, catalogues and massive online open courses (moocs), mostly in the spanish language, according to the target group. results and discussion as indicated above, several instruments have been used in this study that produced sets of results in each of the study phases. in this section we present these results by dividing them into: (1) a summary of the analysis of the needs of researchers during training, and (2) the work developed by project researchers regarding the revision of lori and the evaluation of educational resources through loep. http://dx.doi.org/10.25304/rlt.v27.2271� http://www.edutec.es/panel� v.i. marín et al. 8 citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 (page number not for citation purpose) analysis of researchers in training’s needs a total of 142 postgraduate students from 15 ibero-american countries participated in the questionnaire aimed at discovering how to strengthen the master’s and doctoral programmes at ibero-american universities. the majority (60.57%) of participants came from colombian universities. when researchers during training were asked about their preference regarding the type of educational resources, they classified them as follows: first of all, virtual courses arranged in a platform allowing for student–content and student–student interaction; in the second place, self-contained courses with self-evaluation options and activities, and with the choice of a student–content interaction; and finally in third place, interactive resources in multimedia format. the interviews were conducted with a small sample of research students (n = 6), who described the following statements as important characteristics of digital resources for research training: 1. openness. digital resources must be fully open in digital format and accessible to the entire community. 2. up-to-date. obsolete tools should be avoided. 3. availability in a single platform. it is important to be able to access the resources whenever wanted or needed, and not only through specific courses. 4. quality material. with relevant contents. 5. disposal of a clear and broad structure. this covers the important aspects of contents. 6. accessibility through different devices. universal design should be also considered. 7. format of lo and training capsules. 8. offer of dynamic assistance and interaction support. once the project team reviewed and discussed the results, the outcomes were refined taking into account the literature and the specificities of the target group. in conclusion, digital resources to be selected (and eventually created) to support the research training must encompass the following characteristics: • interactive and audio-visual/multimedia format. • format of training pills/capsules or los, e.g. interactive videos. • adaptive (accessible from any device). • quality based on the evaluation of the material before its development by designers/researchers, and after its development by research students and supervisors. • promotion of interaction with other research students and supervisors in a chat, forum or video conference format (dynamic assistance); for instance, through activities. regarding the topics of digital resources, the analysis of the answers of questionnaires and interviews involved the consideration of seven topics aimed at researchers in training. codes needed for the subsequent labelling of resources were assigned to them: (1) research methodology and structure of research projects (met), (2) tools for information management, reference managers and citation styles (gest), (3)  ethical aspects (eti), (4) scientific writing and scientific publication process (esc), http://dx.doi.org/10.25304/rlt.v27.2271� research in learning technology citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 9 (page number not for citation purpose) (5) preparation for research process (prep), (6) examples of research practices in disciplinary fields (eje) and (7) use of knowledge management tools (con). collaborative evaluation of educational resources as mentioned above, the researchers also identified quality elements for the evaluation of educational resources suitable for research training, which also provided clues for the techno-pedagogical structure to be proposed for the future creation of new resources. with that purpose in mind, several project researchers made use of lori to collaboratively evaluate an educational resource that had been previously selected from a list of repositories, catalogues and moocs. this initial evaluation resulted in a series of agreements regarding the use of lori in the next evaluations of educational resources, which would contribute to a homogeneous evaluation among the project researchers involved in this process. agreements for the use of lori in a pilot phase to test the evaluation of selected educational resources, the evaluated resource was a video belonging to an mooc about the aspects of research methodology. the result of the collaborative evaluation work conducted by four project researchers is shown in figure 2. considering that each item could get a score between 1 and 5, the evaluation was generally positive, with its maximum score in terms of standard compliance, accessibility, interaction usability, and presentation design (at the left side of figure 2, from top to bottom). the content quality and reusability items (at top and left side of the figure) also obtained a high score, although not the maximum score (between 4 and 5). at the right side of the figure and from top to bottom, learning goal alignment, feedback and adaptation, and motivation are the items that obtained an average score. in addition to rating, each project researcher included notes in the open text field, which were discussed later in an online meeting. it was agreed that most items would be maintained. however, different considerations were made for the use of lori in subsequent evaluations, partly due to the type of resources that were mostly found (videos from moocs): • learning goal alignment: we agreed that we would focus on the definition of a general learning objective regarding the project proposal (to strengthen research training in aspects related to the development of a research project) instead of concrete aspects (e.g. a specific qualitative methodology, or a special tool to analyse data). the aim with this agreement was to avoid narrowing the selection and creation of digital educational materials. • feedback and adaptation. in the case of educational resources in video format, and as long as they are not interactive, this item would get a score of 1, meaning that this quality aspect is not present. • motivation. this quality aspect should be evaluated considering how the expert presenting in the resource generates interest in the learner (e.g. mental challenge, changes in tone, mood and presentation format). http://dx.doi.org/10.25304/rlt.v27.2271� v.i. marín et al. 10 citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 (page number not for citation purpose) • interaction usability. we would refer to usability of the educational resource itself, and not of the platform where the resource is published (e.g. the video, instead of the youtube platform). • accessibility. the item would be evaluated with a high score if the resource has subtitles/transcription (it could be differentiated between automatic and human), and if a version of the resource for mobile devices is available. • reusability. the possibility to reuse the resource in diverse disciplines would be considered in its evaluation for this aspect (e.g. a video about quantitative methodology procedures suitable for educational sciences and health sciences). regarding the item ‘standard compliance’, it was difficult to identify whether standards were followed in most of the selected resources, and therefore this item was not used in this evaluation. it was agreed that this item should be considered in subsequent creation of new resources. this item would include the following information on three aspects: (1) technical, with information about requirements and technical characteristics of the content; (2) educational, considering descriptions of pedagogical and educational characteristics of the content; and (3) rights, which refer to information about intellectual property rights and terms and conditions of use of the content. figure 2. graph derived from the collaborative evaluation of educational resource using lori in loep. http://dx.doi.org/10.25304/rlt.v27.2271� research in learning technology citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 11 (page number not for citation purpose) results of the evaluation of educational resources the search for educational resources in 34 repositories, moocs and catalogues guided by the ratings (discussed above) resulted in a final selection of one to five resources in each of the previously identified seven important topics for research training, although in some exceptional cases these limits were exceeded. thirty-two educational resources -mostly but not exclusively, videos, were labelled with a code and a number according to the topic, and included in loep. always two project researchers evaluated each resource that was not selected by them so that a double evaluation was ensured for each resource. the evaluation team agreed on an average cut-off score of 7.5 in collaborative evaluation. this cut-off score was used to determine the inclusion of educational resources based on their quality. the automatic calculation of this arithmetic average of lori by loep was done according to a mathematical equation that leads to the items being averaged taking into account all the evaluations made and then transformed into a score on a scale of 0-10 (gordillo 2017). this resulted in the positive assessment of 17 educational resources. a summary of the lori averages of the selected resources’ evaluations calculated by loep is presented in table 2. figures 3 and 4 show the representation of the evaluation of a resource with a low and high score respectively. conclusions and the future work the quality assurance process for the evaluation of digital educational resources presented in this study has resulted in selecting good-quality materials for research training, which can, eventually, be also an additional support for research supervisors. as wiley (2013) suggested and we would also like to remark again: ‘quality is not necessarily a function of copyright status [...]. local experts must vet the quality of whatever resources they choose to adopt [...]’. furthermore, the current work shows the importance of sharing criteria for the collaborative evaluation of educational resources, especially in the case of international research groups working remotely together in a project. we recognise that platforms, such as loep, greatly facilitate collaborative evaluation processes and minimise the effort of evaluation. we also recognise that lori is an appropriate evaluation approach in determining which resources are of suitable pedagogical and technical quality. in addition, other dimensions such as support in the choice of relevant criteria for each type of resource and more guides for evaluation could be of great value: in the first place, for education professionals, to decide on the resources to be used in their courses and, eventually, for other professionals that may not be involved in the education area but need resources in their fields of knowledge for their own professional development. sharing the results of evaluation with a working group (reviewers and administrator) could also be an opportunity for open discussion within the team. we emphasise from our research the importance that the authors of educational resources label them explicitly as oers wherever appropriate and describe their metadata, especially the type of licence. it is also relevant that the repositories and catalogues of educational resources take greater care in providing a clear resource classification and updated links. some of these difficulties were also mentioned by  atenas, havemann, and priego (2014), and therefore atenas and havemann http://dx.doi.org/10.25304/rlt.v27.2271� v.i. marín et al. 12 citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 (page number not for citation purpose) (2013, 2014) established quality assurance indicators for repositories, such as the inclusion of keywords in incorporated resources or the availability of user evaluation tools. the latter is a very useful aspect, especially when that evaluation may be available to others. the contribution of this study to the existing literature lies especially in the collaborative evaluation of educational resources for research training, and based on multiple perspectives, namely the analysis of the needs of researchers during training, the joint analysis of a broadly recognised evaluation instrument (lori), and the use of its items to ensure the quality of those resources. in addition, this work represents a contribution to the field of oer for the target group of researchers in training, especially in the spanish-speaking world. we recognise some limitations of this study in its different phases. firstly, the samples in the analysis of needs, which were of a voluntary nature, coincidentally came mostly from colombia, which may imply that needs in terms of topics and table 2. average lori rating of educational resources evaluated in loep according to each topic (automatically calculated by loep). educational resource according to the topic arithmetic average of evaluation lori met1 met2 met3 = gest1 met4 met5 8.66 8.61 8.19 5.28 4.03 gest2 gest3 gest4 5 7.92 6.76 esc1 esc2 esc3 esc4 esc5 esc6 esc7 esc8 9.17 8.89 8.61 5.56 9.31 7.22 7.64 7.36 eje1 4 prep1 prep2 prep3 prep4 prep5 5 8.61 5.42 7.5 9.17 con1 con2 con3 con4 7.64 4.44 6.11 8.47 eti1 eti2 eti3 eti4 eti5 8.33 7.22 6.94 8.06 8.33 http://dx.doi.org/10.25304/rlt.v27.2271� research in learning technology citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 13 (page number not for citation purpose) figure 3. score graph of esc1 (positive). figure 4. score graph of met5 (negative). http://dx.doi.org/10.25304/rlt.v27.2271� v.i. marín et al. 14 citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 (page number not for citation purpose) types of educational resources for research training cannot be generalised. perhaps a more detailed analysis of the reasons for this geographical bias would be necessary. secondly, we must also mention incompleteness in the creation of the list of repositories, catalogues and moocs used during resource selection. it was not always possible to locate them all, for example, because they were no longer available. finally, it is worth mentioning that in the collaborative resource evaluation there was a limitation in time and personal resources related to the number of resources to be evaluated and the number of researchers involved in the evaluation of each of them. although a community of appropriate peer reviewers is desirable, perhaps solutions that combine this community-based model with other kind of measures could make the ( collaborative) evaluation of oers more feasible and sustainable. as proposed by orr, rimini and van damme (2015) for developing oers, some examples could be as follows: generating revenue from additional services related to the evaluation of oers ( revenue-based model) such as an evaluator certification; further training as evaluator of oers or counselling services on further user/reuse of those oers; or involving philanthropic organisations to support the quality of oer by offering donations and funding the evaluation of oers (philanthropy-based model). the next phases of this study include a second round of identification and evaluation of educational resources, creation of new resources related to topics where no oer exist, and the development and evaluation of an open website with high quality resources evaluated and oriented towards researchers in training and research supervisors. the website is currently under development and will involve the evaluation of the actual use of the selected and created resources by researchers in training. this evaluation of the use of educational resources by the target group in the website should be especially considered, as presented by canto, guillermo, and tejada (2012). this would allow triangulating evaluations from the point of view of project researchers and from actual users. references adham, k. a., et al., (2018) ‘learning to complete the phd thesis’, issues in educational research, vol. 28, no. 4, pp. 811–829, [online] available at:http://www.iier.org.au/iier28/ adham.pdf association of universities and colleges of canada (aucc) (2011) trends in higher education: volume 1 – enrolment, association of universities and colleges of canada, ottawa, on. atenas, j. & havemann, l. (2013) ‘quality assurance in the open: an evaluation of oer repositories’, the international journal for innovation and quality in learning, vol. 2, pp. 22–34, [online] available at: http://eprints.soas.ac.uk/17347/1/30-288-1-pb.pdf atenas, j. & havemann, l. (2014) ‘questions of quality in repositories of open educational resources: a literature review’, research in learning technology, vol. 22, p. 20889. doi: 10.3402/rlt.v22.20889 atenas, j., havemann, l. & priego, e. (2014) ‘opening teaching landscapes: the importance of quality assurance in the delivery of open educational resources’, open praxis, vol. 6, no. 1, pp. 29–43. doi: 10.5944/openpraxis.6.1.81 atkins, d. e., brown, j. s. & hammond, a. l. (2007) ‘a review of the open educational resources (oer) movement: achievements, challenges, and new opportunities’, report to the william and flora hewlett foundation, [online] available at: http://www.hewlett.org/ uploads/files/reviewoftheoermovement.pdf http://dx.doi.org/10.25304/rlt.v27.2271� http://www.iier.org.au/iier28/adham.pdf� http://www.iier.org.au/iier28/adham.pdf� http://eprints.soas.ac.uk/17347/1/30-288-1-pb.pdf� https://doi.org/10.3402/rlt.v22.20889 https://doi.org/10.5944/openpraxis.6.1.81 http://www.hewlett.org/uploads/files/reviewoftheoermovement.pdf� http://www.hewlett.org/uploads/files/reviewoftheoermovement.pdf� research in learning technology citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 15 (page number not for citation purpose) camilleri, a. f., ehlers, u. d. & pawlowski, j. (2014) state of the art review of quality issues related to open educational resources (oer) (no. jrc88304), joint research centre (seville site), [online] available at: http://is.jrc.ec.europa.eu/pages/eap/documents/ 201405jrc88304.pdf canto, p. j., guillermo, m. c. & tejada, m. a. (2012) ‘recursos educativos abiertos para la formación de investigadores educativos: opinión de usuarios’, in recursos educativos abiertos y móviles para la formación de investigadores: investigaciones y experiencias prácticas, coord. m. s. ramírez & j. v. burgos, lulú editorial digital, méxico, pp. 38–48. caplan, d. & graham, r. (2008) ‘the development of online courses’, in the theory and practice of online learning, ed t. anderson, athabasca university press, edmonton, pp. 247–265. caswell, t., et al., (2008) ‘open content and open educational resources: enabling universal education’, the international review of research in open and distributed learning, vol. 9, no. 1, pp. 1–11, doi: 10.19173/irrodl.v9i1.469 del moral, m. e. & cernea, d. a. (2005) ‘diseñando objetos de aprendizaje como facilitadores de la construcción del conocimiento’, ii simposio pluridisciplinar sobre diseño, evaluación y descripción de contenidos educativos reutilizables (spdece05), barcelona, españa. fernández-pampillón, a. m., dominguez, e. & de armas, i. (2013) ‘análisis de la evolución de los repositorios institucionales de material educativo digital de las universidades españolas’, relatec: revista latinoamericana de tecnología educativa, vol. 12, no. 2, pp. 11–25, [online] available at: http://relatec.unex.es/article/view/1165 fernández-pampillón cesteros, a. m., et al., (2017) ‘herramienta de evaluación de la calidad de los materiales educativos digitales: perfiles de aplicación del profesor y del alumno’, in une 71362 calidad de los materiales educativos digitales. 35.240.90/aplicaciones de las tecnologías de la información en educación, aenor, madrid, pp. 114–131, [online] available at: https://eprints.ucm.es/45338/ ford, k. c., veletsianos, g. & resta, p. (2014) ‘the structure and characteristics of #phdchat, an emergent online social network’, journal of interactive media in education, vol. 2014, no. 1. doi: 10.5334/2014-08 gordillo, a. (2017) contribution to the authoring, distribution, evaluation and integration of learning objects, dissertation e.t.s.i. telecommunications, universidad politécnica de madrid, [online] available at: https://core.ac.uk/display/148690656?recsetid= gordillo, a., barra, e. & quemada, j. (2015) ‘a flexible open source web platform to facilitate learning object evaluation’, 2014 ieee frontiers in education conference (fie) proceedings, madrid, spain. doi: 10.1109/fie.2014.7044498 hylén, j. (2006) open educational resources: opportunities and challenges, oecd, paris, [online] available at: http://www.oecd.org/education/ceri/37351085.pdf kawachi, p. (2014) the tips framework version-2.0: quality assurance guidelines for teachers as creators of open educational resources, cemca, new delhi, [online] available at: http://cemca.org.in/ckfinder/userfiles/files/tips framework_version 2_0_low.pdf mcgill, l. (2011) ‘quality considerations’, [online] available at: https://www.jisc.ac.uk/full-guide/ open-educational-resources ministerio de educación nacional (men) (2013) ‘ministerio de educación nacional, sistema nacional de información de educación superior snies. resumen de indicadores de educación superior’, [online] available at: http://www.mineducacion.gov.co/sistemasdeinformacion moreno, j. & salinas, j. (2011) ‘resultados del proceso de diseño, desarrollo e implementación de un prototipo de entorno virtual para una comunidad de investigadores en formación’, congreso internacional edutec 2011, pachuca, méxico, [online] available at: https:// www.academia.edu/3010238/resultados_del_proceso_de_dise%c3%b1o_desarrollo_e_ implementaci%c3%b3n_de_un_prototipo_de_entorno_virtual_para_una_comunidad_ de_investigadores_en_formaci%c3%b3n nesbit, j., belfer, k. & leacock, t. (2007) ‘learning object review instrument (lori)’, [online] available at: http://transplantedgoose.net/gradstudies/educ892/lori1.5.pdf http://dx.doi.org/10.25304/rlt.v27.2271� http://is.jrc.ec.europa.eu/pages/eap/documents/201405jrc88304.pdf� http://is.jrc.ec.europa.eu/pages/eap/documents/201405jrc88304.pdf� https://doi.org/10.19173/irrodl.v9i1.469 http://relatec.unex.es/article/view/1165� https://eprints.ucm.es/45338/� https://doi.org/10.5334/2014-08 https://core.ac.uk/display/148690656?recsetid=� https://doi.org/10.1109/fie.2014.7044498 http://www.oecd.org/education/ceri/37351085.pdf� http://cemca.org.in/ckfinder/userfiles/files/tips framework_version 2_0_low.pdf https://www.jisc.ac.uk/full-guide/open-educational-resources https://www.jisc.ac.uk/full-guide/open-educational-resources http://www.mineducacion.gov.co/sistemasdeinformacion� https://www.academia.edu/3010238/resultados_del_proceso_de_dise%c3%b1o_desarrollo_e_implementaci%c3%b3n_de_un_prototipo_de_entorno_virtual_para_una_comunidad_de_investigadores_en_formaci%c3%b3n� https://www.academia.edu/3010238/resultados_del_proceso_de_dise%c3%b1o_desarrollo_e_implementaci%c3%b3n_de_un_prototipo_de_entorno_virtual_para_una_comunidad_de_investigadores_en_formaci%c3%b3n� https://www.academia.edu/3010238/resultados_del_proceso_de_dise%c3%b1o_desarrollo_e_implementaci%c3%b3n_de_un_prototipo_de_entorno_virtual_para_una_comunidad_de_investigadores_en_formaci%c3%b3n� https://www.academia.edu/3010238/resultados_del_proceso_de_dise%c3%b1o_desarrollo_e_implementaci%c3%b3n_de_un_prototipo_de_entorno_virtual_para_una_comunidad_de_investigadores_en_formaci%c3%b3n� http://transplantedgoose.net/gradstudies/educ892/lori1.5.pdf� v.i. marín et al. 16 citation: research in learning technology 2019, 27: 2271 http://dx.doi.org/10.25304/rlt.v27.2271 (page number not for citation purpose) orellana, m. l. (2014) supervisión en línea de proyectos de investigación en maestrías y doctorados. estrategia de apoyo para la interacción, diseño y desarrollo de la investigación, haciendo uso de mapas conceptuales. tesis doctoral. universidad de las islas baleares, departamento de pedagogía aplicada y psicología de la educación. orellana, m. l., et al., (2016) programa para el fortalecimiento de la calidad de la investigación en posgrados. tutoría de la investigación y formación de investigadores en posgrados: modelos, recursos y entorno virtual, [online] available at: https://www.researchgate.net/ publication/330726923_programa_para_el_fortalecimiento_de_la_calidad_de_la_ investigacion_en_posgrados_tutoria_de_la_investigacion_y_formacion_de_investigadores_en_posgrados_modelos_recursos_y_entorno_virtual orr, d., rimini, m. & van damme, d. (2015) open educational resources: a catalyst for  innovation, educational research and innovation, oecd publishing, paris. doi: 10.1787/9789264247543-en ramírez, m. s. & burgos, j. v. (coords) (2012) recursos educativos abiertos y móviles para la formación de investigadores: investigaciones y experiencias prácticas, lulú editorial digital, monterrey, méxico. snyder, t. & dillow, s. (2013) digest of education statistics 2012 (nces 2014-015), national center for education statistics, institute of education sciences, u.s., department of education. washington, dc. sussex, r. (2008) ‘technological options in supervising remote research students’, higher education, vol. 55, no. 1, pp. 121–137. doi: 10.1007/s10734-006-9038-0 taylor, s., kiley, m. & humphrey, r. (2017) a handbook for doctoral supervisors, routledge, london. unesco (2002) ‘forum on the impact of open courseware for higher education in developing countries’, [online] available at: http://unesdoc.unesco.org/images/0012/001285/128515e. pdf unesco & commonwealth of learning (col) (2011) ‘guidelines for open educational resources (oer) in higher education’, paris, [online] available at: http://unesdoc.unesco. org/images/0021/002136/213605e.pdf wang, t. & li, l. y. (2011) ‘“tell me what to do” vs. “guide me through it”: feedback experiences of international doctoral students’, active learning in higher education, vol. 12, no. 2, pp. 101–112. doi: 10.1177/1469787411402438 wiley, d. a. (2013) ‘on quality and oer’, in iterating toward openness, [online] available at: https://opencontent.org/blog/archives/2947 wisker, g. (2012) the good supervisor: supervising postgraduate and undergraduate research for doctoral theses and dissertations, macmillan international higher education, basingstoke, hampshire, united kingdom. yuan, m. & recker, m. (2015) ‘not all rubrics are equal: a review of rubrics for evaluating the quality of open educational resources’, international review of research in open and distance learning, vol. 16, no. 5, pp. 16–38. doi: 10.19173/irrodl.v16i5.2389 zawacki-richter, o. & mayrberger, k. (2017) ‘qualität von oer: international bestandsaufnahme von instrumentenzurqualitätssicherung von open educational resources (oer) – schrittezueinemdeutschen modell am beispiel der hamburg open online university’, synergie (sonderband), vol. january, [online] available at: https://www. synergie.uni-hamburg.de/media/sonderbaende/qualitaet-von-oer-2017.pdf http://dx.doi.org/10.25304/rlt.v27.2271� https://www.researchgate.net/publication/330726923_programa_para_el_fortalecimiento_de_la_calidad_de_la_investigacion_en_posgrados_tutoria_de_la_investigacion_y_formacion_de_investigadores_en_posgrados_modelos_recursos_y_entorno_virtual https://www.researchgate.net/publication/330726923_programa_para_el_fortalecimiento_de_la_calidad_de_la_investigacion_en_posgrados_tutoria_de_la_investigacion_y_formacion_de_investigadores_en_posgrados_modelos_recursos_y_entorno_virtual https://www.researchgate.net/publication/330726923_programa_para_el_fortalecimiento_de_la_calidad_de_la_investigacion_en_posgrados_tutoria_de_la_investigacion_y_formacion_de_investigadores_en_posgrados_modelos_recursos_y_entorno_virtual https://www.researchgate.net/publication/330726923_programa_para_el_fortalecimiento_de_la_calidad_de_la_investigacion_en_posgrados_tutoria_de_la_investigacion_y_formacion_de_investigadores_en_posgrados_modelos_recursos_y_entorno_virtual https://doi.org/10.1787/9789264247543-en https://doi.org/10.1007/s10734-006-9038-0 http://unesdoc.unesco.org/images/0012/001285/128515e.pdf� http://unesdoc.unesco.org/images/0012/001285/128515e.pdf� http://unesdoc.unesco.org/images/0021/002136/213605e.pdf� http://unesdoc.unesco.org/images/0021/002136/213605e.pdf� https://doi.org/10.1177/1469787411402438 https://opencontent.org/blog/archives/2947� https://doi.org/10.19173/irrodl.v16i5.2389 https://www.synergie.uni-hamburg.de/media/sonderbaende/qualitaet-von-oer-2017.pdf� https://www.synergie.uni-hamburg.de/media/sonderbaende/qualitaet-von-oer-2017.pdf� a mathematica-based cal matrix-theory tutor for scientists and engineers m.a. kelmanson, s.b. maunders and s.y. cheng department of applied mathematical studies, university of leeds, leeds ls2 9jt abstract under the tltp initiative, the mathematics departments at imperial college and leeds university are jointly developing a cal method directed at supplementing the level of mathematics of students entering science and engineering courses from diverse a-level (or equivalent) backgrounds. the aim of the joint project is to maintain — even increase the number of students enrolling on such first-year courses without lowering the courses' existing mathematical standards. a cal tutor for matrix theory is presented in this paper, in the form of mathematica notebooks. this constitutes one of a list of specific a-level mathematics core options required by science and engineering departments. the module has been written so as to recognize students' errors and advise accordingly. questions are generated randomly, at run time, in order to preclude copying between users. the module incorporates automated performance indicators so as to impinge minimally on existing staff resources. as an aid to other cal authors considering the use of mathematica notebooks, idiosyncratic difficulties encountered within mathematica notebooks are catalogued and discussed in detail introduction the problem students entering science and engineering courses are frequently faced with the problem of weakness in their mathematical preparation for further education; this is due to a number of factors, not least of which is the vast number of possible combinations of a-level mathematics syllabi. many university science and engineering departments continue to offer courses requiring a 'reasonable' level of mathematical comprehension, so that students deficient in specific a-level mathematics topics are often inadequately prepared for such courses. 27 ma. kelmanson etal a mathematica-based cal tutor for scientists the problem, as described, affects a number of first-year courses at universities throughout great britain; it has, in recent years, been further exacerbated by the introduction of the gcse, since the subsequent jump to a-level mathematics is often considered to be too large. moreover, many engineering departments accept students with a very wide range of entry qualifications besides a-levels, btec certificates and diplomas being the most notable examples. unfortunately, the mathematical content of such syllabi is much lower than that of a-level. the impending reorganization of the many a-level mathematics syllabi will lead to an inevitable dilution in many areas especially calculus and will further aggravate the situation currently experienced by many university science and engineering departments. in later academic years, deficiencies in mathematical grounding affect almost all the theoretical courses taught in science and engineering departments, which are noticeably reluctant to defer their inherent course material in favour of the necessary extra mathematics material. this recalcitrance towards course postponement is wholly understandable in the light of the inevitable knock-on effect which would both require an increase in the overall course duration, and affect the course validation with professional institutions. the worst possible scenario would be that a serious hiatus in degree standards between british and european universities might occur. in the light of the above information, there is clearly a need to establish a convenient and flexible mechanism for teaching this transitional mathematics material in order to bring first-year students up to the required standard. (transitional is used here in preference to remedial since the latter is definitively associated with slow-learning: in the context of this paper, this is clearly not the case.) finally, any such mechanism should ideally impinge only minimally on existing staff resources. the suggested remedy in late 1992, imperial college (ic) and leeds university (lu) were jointly successful in procuring a bid, under the tltp initiative of the ufc (now the hefce), which described a mathematica-based system for presenting mathematics tutorials to scientists and engineers. the suggested remedy for the problems described above was the production of a suite of cal interactive tutorial modules based on maihematica and its inherent front-end, notebooks, which allows a certain degree of control over the navigation through the tutorial material. maihematica (wolfram, 1991) was suggested since, as a sophisticated and versatile integrated package incorporating programming, numeric and symbolic computation, text processing and graphics, it would allow flexible production of an extremely wide range of detailed mathematical material. moreover, maihematica has in recent years enjoyed a high profile in numerous teaching projects; see, for example, brown, 1991; gray, 1991; burbulla, 1992; the calculus reader, 1993. further to the above, maihematica could be launched from a variety of platforms and operating systems, making it portable both within and between sites, primarily for development, ultimately for nation-wide distribution. finally, through the interactive medium of maihematica notebooks, it was envisaged that a certain degree of genuine interaction between student and cal module might be achieved, allowing students to use the same module on different levels solving different problems. more detailed discussions concerning maihematica are deferred until later. 28 alt-j volume 1 number 2 since the project began, on 1st november 1992, all necessary module development staff three at each site have been appointed, and all required hardware and software is in situ. in the (ongoing) initial phase of the project, ic are responsible for modules in complex numbers, trigonometry and vectors, while lu are producing modules in differential calculus, integral calculus and first-order differential equations; both sites are producing separate modules on matrices, for comparison. it is lu's matrix module which will be presented and discussed, from both pedagogical and computational points of view, in subsequent sections of this paper. module design interaction philosophy a criticism which might be made of existing cal modules is that much of their interaction is 'hard wired' into them, so that after a few attempts at the same module, a student might perhaps achieve correct responses by memorizing problems that he/she has already tackled, rather than by actually understanding the principles underlying the module's material. a natural extension of this is to question wherein lies the difference between such a module and a textbook. in the light of that question, the ensuing criticism would be that a textbook would, in fact, rank more useful than such a cal module, since the former is truly portable. therefore, an essential feature of our approach was to ensure that our modules were somehow more 'human' in their interaction with the student: this constitutes a considerable challenge from both programming and pedagogical points of view. with regard to this aspect of a quasi-human helpful approach, the module should further be able not only to point out errors, but also to recognize, trap, correct and advise accordingly in the light of student difficulties. if errors are merely indicated as either right/wrong, our experience dictates that user-frustration will ensue, at which point the module ceases to be effective. interaction implementation one of the authors of this paper had already written a cal fortran 77 course given to first-year mathematics students at bradford university. in that course, students were initially shielded from the predictably confusing side-issues of compilers and loading, since these are, in effect, system-dependent. by the same token, the lu team decided that module users should be entirely shielded from the mathematica 'maths engine' in the present project. we felt that the students should certainly not be expected to assimilate the language of mathematica, since it might detract from their concentrating on the issue at hand, to wit: the transitional mathematical material. this was not as easy to achieve as one might think, since the only (partially suitable) interactive medium available to us (on which the initial project proposal was based) was notebooks, mathematica's inherent front end. moreover, it was envisaged that the modules were to be developed for a range of platforms, but since the majority of uk universities and sixth-forms are more likely to be ibm-pc based, all further discussion will be thus directed. we are also developing mathematica software for unix machines, but these presently do not support notebooks. mathematica notebooks for the purposes of subsequent discussion, it is necessary briefly to outline the module-authoring medium, mathematica notebooks. this is a windows-based menu-driven front end through which the user may interact with the mathematica kernel to produce mathematical and graphical output. notebooks is presently limited to the pc, next and 29 ma. kelmanson et al a mathematica-based cal tutor for scientists macintosh platforms, the last of which has been used extensively for project calc (the calculus reader, 1993), which has been used (in the usa) to teach first-year university calculus in a laboratory environment mathematica notebooks constitute an integrated environment in which the user can be prompted to perform calculations and/or exercises. information appears in notebooks in cells, which are of differing types: input, output, text, graphics, etc. hence, from a pedagogical point of view, notebooks enjoy the property that they are able to unite demonstrative text with supporting graphics. consecutive cells pertaining to a particular area or problem may be grouped, in which case only the first cell in the group is subsequently visible, the rest being closed; the closed cells may be opened again by double-clicking on designated group-boxes which appear on the screen. similarly, groups of cells may themselves be grouped and closed; in this way, a potentially large document, although in the notebooks in a linear form, may be 'condensed' so that it can be viewed rather like the contents page of a book. the user may then choose a particular topic title from within the notebooks titles by double-clicking on its group-box, which will then open and display either subtitles or text. hence it is easy to see that notes, in the form of text, constitute the most-nested level of information, with section banners residing at the outermost level of cell nesting. figures 1, 2 and 3 illustrate some of the points raised so far. mathematics for windows |ma(j 'm eile edit £ell graph fiction style options ivindow help is. i input m interactive cbl module — matrices n mark kelmanson simon maunder and sui cheng, hefce tl.tp mathematica project. january-march 1993. • completely revised (or mathematics 2.2 beta release. april-june 1993. b pkeominar.y ins-trtjctions: "double-click" the left-hand xnonse button on the little box below and right -—> ematrices figure 1: the matrix notebook as it first appears 30 alt-j volume 1 number 2 ."1 eile edit £ell graph action style options svlndow help m matrices ; b part 1: basic properties of matrices j b part 2: matrix ax.gebra and transformations j l 1 . . . . . . . . . . . . . c jy. r r. . . ..--•.. ... -. —. b figure 2: the matrix notebook after double-clicking on the cell marked 'a' in figure 1 mattiernatica iot windows (mat] w.1 ble edit cell graph fiction style options ¥flndow help as. twe • p a r t 2: matpjdc a1.qebra aot) t r a n s f o r m a t i o n s • prerequisites • matrix algebra i • transpormatiok"s in the plane • q-tvekse transformations • summary ol part 2 m end of matrix module. j -i % rfrjjjr?/;.^gv'yyy,g'jr.•:':.<!'•• y:;••>da 5£4^i^s:±3^ii^^^r2^^ ;. figure 3: the matrix notebook after double-clicking on the cell marked 'b' in figure 2 31 ma. kelmanson et al a mathematica-based cal tutor for scientists difficulties within notebooks in producing the matrix module, we encountered the occasional difficulty within the notebooks environment. for example, once opened, groups remain thus even though the material contained therein may no longer be required. thus the notebooks become more linear in appearance as progress is made through the module unless the user explicitly closes the 'finished' cells. similarly, when the user is interacting with mathematica via its dialogue box, the autoscroll in the notebooks main window is disabled, thus forcing users explicitly to search for the responses to their input in version 2.2 of mathematica, there is no control over either the size or location of the dialogue box; this, coupled with the aforementioned loss of autoscroll, constitutes a non-trivial problem in terms of ease of navigation by uninitiated users. finally, the absence of mathematical text and the inability to change fonts within a cell proved somewhat restrictive; much effort is required to circumvent this rather artificial restraint. mathematica's absence of hotwords or 'intuition-driven' buttons etc. enforce a certain degree of linearity on the style of the module. we hope that future versions of mathematica will accommodate the necessary features to support a more tree-structured module design: if they do not, then toolbook may yet be employed upon arrival of mathlink to front-end the interaction, which is a further reason supporting the shielding of students from mathematical programming language. it is important to remark that the lu team has enjoyed an excellent rapport with wolfram research and their in-house technical support group who, in the light of our findings, were quick to remedy anomalies in the 2.2 p* version of mathematica. we are confident that our aforementioned comments will also be considered seriously, and that, through interaction with cal groups such as ours, wolfram research will undertake relevant developments in future versions of mathematica. the matrix module content in order to facilitate navigation through the module, it was 'naturally' split into two sections (see figure 2): basic properties of matrices, for complete beginners, and matrix algebra and transformations for users possessing a basic knowledge of principles and notation. each such section was further subdivided into more detailed sections (see figure 3). note that the (horizontal) size of any group-box in figures 1 to 9 is indicative of the amount of material contained within that group. each general title contained a combination of tutorial notes, supplementary exercises and, where relevant, animated graphics. for illustrative purposes, we shall concentrate further on the section entitled transformations in the plane; a typical section of text is depicted in figure 4. interaction and questions in the absence of hotwords and buttons in the notebooks environment, interaction had to be 'invited', as shown in figure 4. this is far from ideal, as it essentially means that users must be aware of how to make mathematica recognize their input; while such information can be placed in a supposedly prominent location on the screen, there is no guarantee that the user will assimilate it. we would prefer to see a more intuition-based front-end for this purpose. 32 alt-j volume 1 number 2 mathematics lor windows -jmaf] file edit cell graph action style options window help \mm\msmml [ e ] — > [ c ] [ 1 ] [ « ] [ -i ] " > [ a j [ « ] [ d ] then t [ a c ] [ b d ] . o t t i e idgntity transformation o rotation f o r an anticlockwise rotation of angle a about the origin. t [ cos fi s i n fl ] [ s i n a cos 0 ] • animation 1 [v; s o t j l e c u c k t h * m c i 3 « oaths r.fstlitdei.heflw) b a i r t o t s * bottats n g b t e f cii) click ui* mcrj!« or; u>« fitl* ' f j n * syitibol wsuch is the ssvette icon from to stop ik? <£i£u4uorl cii;fc 3n*p.<rfi*r$ m '&* fifuawuon 4re4. rotation ot 60 dttjeees about (0,01 figure 4: textual content and 'invitation' into interaction mathematics for windows |mat] «g pie edit cell eraph fiction style options fflindow help title » exercise 4 now activate the following exercises on matrix transformations. quecfmat2d (* fill here, then piess shift * return » n combinntion of transformations we saw above that a point (x,y) could be tr ' j r i kernel input let die transformation matrix which does this may itself be transformed into a new point (x second transformation be called q (which, \ from f ). then, m terms of matrices and vec t xt y p [ x ] c y i [ x" [ 9" i f w e substitute the first of these equations inti q pt x" ] [ y ] q [ x] c ii • ] cnter the inattu (oi a fcflctiion (n the line y d i i'm rlirlf thf rnnuse in thi-. winrirtw lypp in mur i ok figure 5: the dialogue box with associated question text 33 ma. kelmanson et at a mathematica-based cal tutor for scientists on initiating a question, a dialogue box appears as in figure 5. unfortunately, the tutor's dialogue is restricted to only two or three lines of text; couple this with the absence of mathematical fonts and the wish to shield users from mathematica's language, and it soon becomes apparent that a great deal of programming effort is required to obtain seemingly straightforward information. as a means of discouraging the memorizing of previously-attempted examples, random numbers are used in all the interactive exercises. students are encouraged to perform calculations using pen and paper and, in the later sections of the module, open-ended examples are proffered (just as they might be by a lecturer!). in producing this (and other) modules, we sought to achieve an approach whereby the user was not merely told that their responses were 'right' or 'wrong'. in the former case, this is not too much of a problem, but in the latter, somewhat more is required if the cal user is to gain a true understanding of the principles involved. in this respect, mathematica's inherent high-level functions proved invaluable in assisting us to locate various types of user-errors. error trapping when prompting users for input, error-trapping code had to be written in order to circumvent the restrictions imposed by the pedagogically artificial mathematica format. we stress that this effort is very much justified since it is common knowledge that once an 'automated' package fails to perform correctly, user-disenchantment and frustration quickly follow. in order to allow the users to enter solutions in an intuitive manner i.e. not tied to mathematica input-format considerable effort was expended in ensuring that the packages could recognize a large array of possible input formats. for example, our module accepts any of the following possible inputs for ««(-40): sin[40], sin(40), sin -40, sin-40, -sin40, sin 40, sin[40], sin(40), sin -40, sin 40, sin-40, -sin40. to interpret all of these (perfectly feasible) inputs as one and the same quantity, some sophisticated aspects of mathematica's programming language were employed. for the reasons described above, we do not consider that if an expression is mathematically correct but syntactically wrong with respect to mathematica, users should be penalized (and therefore disenchanted) for their input style. fortunately, mathematica supports an extremely powerful programming language which facilitates the implementation of interactive questions that are robust, syntactically flexible and corrective in nature. in the experience of the authors of this paper, mathematica's inherent language is more flexible than that of similar packages (see, for example, char, 1991). in general, the user is provided with two chances to enter the correct response, which is then parsed using a custom-written mathematica package, to determine whether or not it is essentially correct. minor errors are highlighted but not penalized. if possible, the specific nature of more serious errors is identified, and appropriate corrective action is then suggested; otherwise, more general information on the required approach is supplied. as an example, figure 6 shows such a message in response to an incorrect user input. this parsing and subsequent interpretation of user input is, in our view, a crucial element in the philosophy of cal. many cal users including the authors of this paper are frequently impressed by (static or animated) examples involving sophisticated threedimensional multi-coloured graphics (see, for example, maeder, 1991)); to most users, these are the flagship of algebraic manipulators. however, from a pedagogical point of view such a 34 altj volume 1 number 2 mathcmatica for windows [mat] %-i ehe edh cell graph action style options mffndow help rz\> i sv 1 ] • animation a o example 1: shear whh x-oxis invariant o exsmpls 2:shssr<«thy-a3ria invariant • exercise 4 now activate the following exercises on p fill here, th incorrect: you entered [ 0 1 ] [ 1 1 ] n combination of transformations "we s«w above that a point qc.y) coidd be let the transformation csstriz which dies tl second transfcfgnaticn be called q fwlac* enter mii tnsirbc lor » refaction in ttie tine y you (i«w made'« mistake tn ttie fabwiiifi: raw* / ' • . first dfick the mnuao in ths window . type in raw i , . .' j , figure 6: the dialogue box with error-correction prompt product, visually impressive though it is, may prove far easier to achieve than the auto-simulation of truly sympathetic, discerning, corrective interaction. we believe that our approach goes some way towards achieving these aspects. animated graphics an important feature of the notebooks is the ability to incorporate animated graphics within the same environment as mathematical output and tutorial notes; it is this feature of mathematica notebooks which proved so attractive for the purposes of the present project. in notebooks, an animation is obtained by grouping together a sequence of cells each of which contains a postscript image produced by mathematica graphics functions. if the group of cells is then closed in the manner described above in the section headed mathematica notebooks, only the first image in the sequence remains visible. an animation icon may then be used to initiate the animation of the cells so grouped. as an example, figures 7 and 8 show animations illustrating the application of rotation matrix to a prescribed polygon: in this paper, the two images are necessarily shown 'in series' whereas in the module, the polygon flips cyclically from one position to the other with the axes remaining fixed. the matrix module contains a substantial number of animations illustrating various types of transformation. geometric interpretations, such as the general non-commutativity of matrix multiplication, are ideally illustrated via this medium. from an interactive point of view, the user may provide the parameters for a given transformation, thus permitting a degree of investigative experimentation; on this note, notebooks may still be used to initiate open-ended investigations, as they constitute a powerful tool for the required analysis. 35 ma. kelmanson et al a mathematica-based cal tutor for scientists \ mic ldit cell graph action style options window jjelp to rap tie ttniiastoc, cs_* rjkî ierc ialhe octuuce trsc roce.-ion o~ so aogrses ecocuc ic.-o) 3 i 3 2 1 1 2 3 1 rote.:nm o^ 6o \ir ic en ' figure 7: animation cell showing pre-transformed polygon mattiematica for wirtdovs • |matl f'\ eile edit cell graph fictipn ^tyle qptiong window help -4 pre-tcanafonnat ion rotocion of 60 drtjrees about (0,0) • 5 3 2 1 -1 -2 -3 1 2 3 st-trsnsfomat ion i ' . l i ax ' . . / . . / . . i*!., s.. st. / / /t , s, , figure 8: animation cell showing post-transformed polygon 36 alt-j volume 1 number 2 automated performance indicators an initial requirement of the project was that the modules produced therein should impinge minimally on existing staff resources. in order to ensure that student progress could be monitored, an automatic performance-recording procedure was implemented. results of the user's attempts at the (or, indeed, any) module are automatically written to a file to which there is only staff access. for each user, only a single entry is maintained: this is updated in a cumulative sense insofar as it monitors attempts at a particular question from a number of different sessions. the user-record indicates both the percentage ratings on a given question and the number of attempts at a given question (this is to distinguish those users who, for example, score 1/1, from those who score 19/20). in order to implement this automated logging of performance, an alphabetically-sorted list of entries is maintained. any interactive question ends with an update of the appropriate entry. the update for a question may take one of two forms: amendment of existing information for that particular question, or insertion of performance information for that question into the relevant part of the user's entry (if that part has not previously been created). it should be noted that each entry is an alphabetically sorted list comprised of sub-entries for each attempted question. since there exists only a single line in the file for each user, and furthermore that line consists of information for the attempted questions, the file is particularly concise. this file is encrypted and its contents accessed by only the (human) course tutor, using a custom-written, simple-to-use mathe/natica package which (see figure 9) displays graphically the required information. mauiematica (or windows • inewnt>-11 •a) file edit cell graph action style options window help hodule 1 : no. of attempts against correct e n t r i e s dall degas honet picasso renoir f, '•?; r figure 9: tutor-viewable performance graphics 37 ma. kelmanson etal a mathematica-based cal tutor for scientists evaluation this paper would be incomplete without some indication of the performance of the module 'in the field'. at the end of may 1993, the module described above was tested by a group of sixth-formers from the further maths group at harrogate grammar school. none of the testers had used mathematica previously, and about half were familiar with windows. nearly all had used some form of pc before. feedback on the module was extremely encouraging: 90-100% (of the testers) were positive about the content and presentation; 60-70% liked the style and mode of feedback; 60% found notebooks clear and navigable. on this last point, the navigability is a factor which, in essence, lies in the hands of wolfram research. overall, 20% of the users still preferred a text-book approach to that of c a l many of the criticisms raised related to the inherent properties of the mathematica front-end (briefly discussed above in the section headed difficulties within notebooks): position of dialogue box, loss of autoscrolling during interaction, absence of standard 'mathematical' notation, etc. these had already been predicted as they had essentially been the subject of correspondence between wolfram research and the authors of this paper. it is hoped that future versions of mathematica will accommodate features which will alleviate or eradicate these particular problems. future developments further modules the module described in this paper has essentially constituted a pilot investigation, the results of which can now be implemented in the other modules which will complete the first inter-site suite. as mentioned in our introduction, these are: vectors, trigonometry, complex numbers, differential calculus, integral calculus and first-order differential equations. in the light of current exterior interest in the joint project, both sites are at present clarifying copyright arrangements with a view to distributing freely the modules so far produced (any interested parties may contact any of the authors of this paper). alternative front ends an essential difficulty of the notebooks style is the inherent 'linearity'. navigation through notebooks is possible via user-search rather than by 'hotwords' or buttons; in our view, this is somewhat limiting. a separate, button-driven front-end, such as toolbook, would seem more suitable for driving the mathematica maths engine, but toolbook also suffers from limitations in mathematical text presentation. one can circumvent this problem using equation editors and graphics packages, but that is far from ideal. in any case, not until mathlink becomes available from wolfram research can mathematical kernel be easily linked to an alternative front end. 38 alt-j volume 1 number 2 acknowledgements the authors of this paper wish to express their gratitude to the following people involved in the project: professors m.i.g, bloor and j. brindley at leeds university, the tltp team at imperial college, and mr p. heyes of harrogate grammar school. references brown d., porta, h. and uhl, j. (1991), calculus and mathematica: part i, reading ma, addison wesley. burbulla, d.c.m. and dodson, c.t.j (1992), self tutor for computer calculus using mathematica, london, prentice hall. the calculus reader (1993), project calc manual, college mathematics department, d.c. health co., lexington, usa, autumn 1993. char, b.w. et al. (1991), maple v library reference manual, berlin, springer verlag. gray, t.w. and glynn, j. (1991), exploring mathematics with mathematica, reading ma, addison wesley. maeder, r. (1991), programming in mathematica, reading ma, addison wesley. wolfram, s. (1991) mathematica: a system for doing mathematics by computer, 2nd edn., reading ma, addison wesley. 39 early experiences of computer-aided assessment and administration when teaching computer programming steve benford, edmund burke, eric foxley, neil gutteridge and abdullah mohd zin department of computer science, university of nottingham, university park, nottingham ng7 2rd abstract this paper describes early experiences with the ceilidh system currently being piloted at over 30 institutions of higher education. ceilidh is a course-management system for teaching computer programming whose core is an auto-assessment facility. this facility automatically marks students programs from a range of perspectives, and may be used in an iterative manner, enabling students to work towards a target level of attainment. ceilidh also includes extensive courseadministration and progress-monitoring facilities, as well as support for other forms of assessment including short-answer marking and the collation of essays for later hand-marking. the paper discusses the motivation for developing ceilidh, outlines its major facilities, then summarizes experiences of developing and actually using it at the coal-face over three years of teaching. introduction assessment and feedback are key aspects of the educational process. this is particularly true when learning practical skills where hands-on experience is of critical importance. computer programming is one such skill. indeed, like riding a bicycle or playing the piano, it is not possible to learn to program a computer without extensive practical experience. however, computer programs are also notoriously difficult to assess by traditional methods such as hand-marking printed output. firstly, it is virtually impossible to tell by visual inspection of a program source whether it will run at all, let alone whether it will run as required. detailed assessment using print-outs is simply out of the question. secondly, program development is an iterative process, often involving many cycles of development and testing. hand-marking is too slow to help in this process. these problems are exacerbated by increasing student 55 steve benford et ah computer-aided assessment and administration numbers, particularly on first-year programming courses where class sizes in excess of a hundred are common. a dedicated lecturer teaching such a course would be hard pressed to manage one iteration of one exercise a week, even if he or she had little else to do. one possible solution to this problem may be for a group of people (for example, postgraduate assistants) to mark the solutions. however, this often leads to inconsistencies in the level of marks awarded, and drastically increases the administrative burden. of course, such problems are not confined to the realm of computer programming, and there are a number of more general problems with hand-marking paper scripts. scripts can easily get lost, detached or damaged. marking on paper is inappropriate for most computer-based learning (not just for programming). administration of assessment is a time-consuming process, made more difficult by having to deal with large quantities of paper. finally, there is increasing pressure to save paper. our solution to these problems has been to explore the role of the computer in directly assessing students' programs. to this end, we have constructed a system called ceilidh based on an automatic assessment mechanism which tests programs from several perspectives, including dynamic correctness (i.e. do they work?) and programming style and complexity (i.e. are they elegant solutions?). ceilidh also supports course administration by providing facilities for setting exercises, handing in all types of work online, handing out solutions, dealing with questions, and tracking and summarizing progress. the idea of automatically assessing computer programs has direct parallels in industry where automated test-harnesses and software quality-control environments are becoming increasingly common. thus, ceilidh provides students with valuable experience of working in a quality controlled environment. ceilidh also aims to support the administration and assessment of a range of courses beyond programming. to this end, we have begun experimenting with other complementary assessment techniques, including multiple-choice and short-answer marking. the first version of ceilidh was developed in 1988 and used to support a c programming course. the second phase of development was funded through nottingham university's enterprise in higher education (ehe) initiative, and resulted in ceilidh being used to support a c++ course for a class of 160 in the 1991/92 session. at present, ceilidh is at the centre of a ufc teaching and learning technology programme (tltp) project courseware for the automatic assessment of programming which aims to pilot the system at over 30 departments of computer science throughout the uk in the next three years. ceilidh has also been installed at sites in north america, australia, belgium, india, malaysia, new zealand, portugal, russia and spain. beyond this, ceilidh should be of interest within the many other disciplines involved in teaching computer programming (particularly in engineering and science) and also computer centres which are often heavily involved in service teaching. the administrative side of ceilidh might also interest a much broader range of people. overview of ceilidh let us begin with an overview of the functionality of the ceilidh system, starting with a high-level summary of the functions available, and moving on to a more detailed examination of the steps involved in setting up, completing and marking a programming exercise. the logical structure of the ceilidh system is summarized in figure 1. ceilidh can support a number of on-going courses. each course is divided into units representing different chapters or topics. each unit may include a number of exercises which may be programming exercises, short-answer questions or essay questions. of these, 56 alt-j volume 1 number 2 ceilidh course course course unit unit unit staff, student tutor, lists course summary weights exercise exercise exercise notes solutions questions model solutions tests 1 1—•—1 answer marks figure 1: the ceilidh information structure programming and short-answer exercises are supported by automated assessment facilities. essay-style exercises include facilities for online collection of work for later paper hand-marking. the overall functionality of ceilidh is categorized according to four different classes of user: students, tutors, teachers and course developers. students use the system to obtain, complete and mark work, and to access course resources such as notes, work and lecture schedules and even the teachers. tutors represent teaching assistants, and are provided with additional 57 steve benford et ah computer-aided assessment and administration facilities to inspect work and to summarize the progress of individuals or groups of students. teachers are provided with facilities to administer entire courses. course developers can amend the course information including notes, exercises and test data. we now consider each class of user in turn. student facilities ceilidh provides the following key facilities to students: (i) it allows students to view general temporal course information such as hand-in times for coursework and more permanent information such as lecture notes. this information can be viewed on-line or to be printed. (ii) it provides access to the questions set in exercises. (iii) it offers outline program source (skeleton programs to be completed by students), and associated modules and header files for exercises where appropriate, to assist in the solution of programming exercises. for an essay exercise, an outline of the main sections/heading expected in the essay may be given. (iv) it allows students to edit, compile and test-run their computer programs. the extent to which compilation details are hidden from the student is determined by the teacher. (v) the students can submit their work to the system. if the exercise is programming, the system will mark their program. a summary of the marks is made available to the students to help them assess their program quality (and a copy of the program and of the marks awarded is retained centrally for later reference). marking can take place many times providing an iterative process of development and assessment. if the exercise is an essay, the system simply stores a copy of the solution. (vi) it allows them to view a model solution, to run this solution, to view test data and to run both their own solution and the model solution against the test data. the model solution can be seen only after the deadline for submission has passed. (vii) it allows them to comment on a specific exercise or on the system as a whole. comments are stored for later browsing by teachers. (viii) it offers help facilities, including an overview of the marking metrics employed by the system and a good programming style guide. tutor facilities ceilidh provides tutors with the following additional facilities: (i) list details of the work submitted by all of the tutor's students, or by any named student. for each item of coursework, the listing gives a summary of the marks awarded • and the tune at which it was submitted, including whether it was early or late. details of the work, such as the program source code and a more detailed breakdown of the marks, can be inspected if requested. (ii) list the names of students who have not submitted work, or who have submitted late. (iii) list the marks awarded to students for a particular exercise. (iv) summarize the average marks across all the exercises on a given course. 58 alt-j volume 1 number 2 teacher facilities ceilidh provides teachers with the following additional facilities: (i) declare certain exercises to be open. these are the exercises on which the student will be expected to work. (ii) declare certain previously opened exercises to be late. students submitting after this date will be warned that their work is late. for each exercise made late, the teacher will be invited to request plagiarism tests, and overall class software metrics. the latter is particularly important to enable the teacher to keep in close touch with the class progress, and its strengths and weaknesses. (iii) declare certain exercises to be dosed. students will no longer be able to submit work for these exercises. (iv) teachers can browse and respond to the students' comments. (v) set weighting and scaling factors so as to calculate final assessments from marked exercises in any desired way. course developer facilities ceilidh provides course developers with the following additional facilities: (i) create new coursework, or amend existing coursework. the system prompts the user to ensure that all the necessary data items have been input (see below). select «ode of operation: for fill student* for all tutees for haaed student jedlt course hotd ) |vien registerc:introduction t o programing i n c {eric fox cua:co*>utar use and applications {steve benford) lcp:introduction t o c++ programing (dr pr eduards): the nuts and bolts •c;lntroductlon t o programing i n c < e r l c foxley, haths pe*:pe»c*l programing {by royal hollouay college) prl;introductlon t o c++ programing ( e r i c fox l e y ) : the nuts and bolts prz:c+* programing part 2 (steve benford); programing i n the large lc testing [edit course sumanj] [edit register : introduction to unix (eric fox ley): t figure 2: ceilidh x-windows interface 59 steve benford et al computer-aided assessment and administration (ii) set up new courses and manage registers of tutors and students. the various different roles within the ceilidh system are illustrated in the screen dump of the x-windows interface in figure 2. completing exercises let us consider in more detail the steps involved when a student completes a specific exercise. we will consider programming exercises as these currently form the core of ceilidh's work. the student selects the exercise to be tackled (a default is usually set by the teacher) and reads the specification (the programming term for the question). specifications form a reasonably complete description of the problem to be tackled, and may vary greatly in complexity and difficulty. figure 3 shows a student viewing an early c++ exercise, and the skeleton file provided. later in the course, specifications may run to several pages, including descriptions of c++ class definitions to be implemented or used, and details of exception states. the goal at this stage is to develop the students' ability to work to specification. more advanced tasks might typically involve implementing a c++ class to work with a given interface program, writing a program to use a given class, or writing a program which combines several given classes. for example, the following specification might be given later on in our c++ course: a telephone book consists of an arbitrarily long collection of entries where each entry contains a name and a number. users can add new entries, delete entries, look up a number belonging to a specific name, and produce an alphabetical listing of the whole telephone book. a telephone book can be represented by a 'phonebook' abstract data type which could be implemented in c++ as a 'phonebook' class. the following is an outline class definition for a simplified 'phonebook' class. outline c++ class definition here the following details should be carefully noted: * a number of status codes are defined to be returned by phonebook methods. * each person is limited to having a maximum of one entry (meaning that a person's name can only appear in one entry). * a name cannot be the empty string. * a phone number must consist of digits in the range 0-9 inclusive, and must be exactly number_length digits long. * the overloaded < operator produces a list of all entries currently in the phonebook. this list should be alphabetically ordered by name. the status codes have the following meanings: ok operation completed successfully. bad_name the supplied name has the wrong format. bad_number the supplied number has the wrong format. already_exists an entry for this person already exists. does_not_exist there is no entry for this person. 60 alt-j volume 1 number 2 from this example, one can deduce that the current emphasis of ceilidh is on encouraging students to produce correct programs from existing specifications. ceilidh may offer students varying levels of support for developing their programs including skeleton implementation files which provide common definitions and outline code, invocation of editors, automatic compilation and linking and even automatic running of their program against test data. the level of support provided can be tailored by the teacher throughout the course. thus, at one extreme, ceilidh can provide a fully integrated implementation environment which completely shields the student from the details of the operating system, and at the other the student can be exposed to all the details of compiling and linking (in our case at the unix operating-system level). a half-way house is also possible where ceilidh invokes the necessary operatingsystem commands but also echoes them back to the student. at any stage throughout development, students can request the system to mark their program. a variety of standard software metrics are used, combining dynamic and static analysis techniques these are more fully described in zin (1991) and benford (1993). proowhing cxercise/exflhple j {view solution] |te»t i k sobitt) [copa solution! 1 opt it |help| jqutt course : p r l |s«l«ct c [solact unit i [unit |select exarcl«a 1 |vi«w question select e»<rclw of * circle 2:seconds to hoc*-*, plnutes «nd «econd« 3;use of "slzeof" calslue 6:cantimtrm to feet end inches 7;ftdd times together / / convert feet and inches to centinetres ' written by ef 1988 const int insperft « 12; const float o1sperin • 2.54; float cms, ins; int ft; cout « "type feet and inches: " ; cln » f t » ins; lulllllllllllllllllllllllllllllllllllllll / / put ycx»* progran instructions here '/llllllllllllllllllll/lli/llllllllllllllll cout « ft « " feet " « ins « " irchbs is " « ctis « } / / end rain i\n"; write « proeran to read a nurber of feet (inteser) and inches (float) and print the equivalent nurtaer of centimetres (float). assune 2.54 eentlratrss per inch. if you want to be clever, add the fol lowing features: negative quantities of inchbs are not permitted, inches greater than or equal to twelve are not permitted, print a nessage containing the word "error" if this rule is broken. negative numbers of feet are assuned to negate the whole result. print your output in thb fom 12 feet 9.5 inches is 3s9.89 ens figure 3: student viewing a question and skeleton program a summary of the marks is made available to the students to help them assess their program's quality, and a common question concerns how the system helps students to identify the problems with their program given the relatively terse summary of marks. the answer is that ceilidh does not try to identify what a problem is in detail it only reports that there is one and roughly what it is. the student is then expected to go back to his or her program and figure out what is going on, usually by more extensive testing on his or her own. this seems quite satisfactory, since the goal of the system is to aid the learning process, not to provide an instant solution. however, we recognize that determining an appropriate level of feedback is a 61 steve benford et al computer-aided assessment and administration difficult issue, particularly in relation to the tightness of the specification and the generality of the test data. ceilidh leaves all these factors configurable to some degree. a key feature of ceilidh is that students can re-mark their work as many times as they like, although a teacher may optionally impose a minimum time-delay between markings. the system maintains a record of the full mark history for each piece of work, including the date and time of submission. a copy of the current program is also held for later analysis. on submission, the system notes whether the coursework is late and, if it is, asks the student if he or she still wishes to go ahead with submission. it also asks if he or she wishes to comment on the mark. short-answer exercises are different in that the student is presented with an on-line test in the form of a set of short questions. ceilidh then reads simple answers in the form of words, phrases or maybe whole sentences, and matches them against keywords supplied by the teacher. marks are recorded by the system. essay exercises involve the student reading the question and perhaps being given a skeleton essay to complete which can be submitted back to the system when ready. the teacher may then hand-mark the essays and enter the marks back into ceilidh for subsequent processing. setting up exercises in order to set up an exercise, the course developer needs to provide a number of files: (i) a question/specification. (ii) a working model solution or model answer. (iii) a file of test data for each of the dynamic tests to be carried out in programming exercises. test data may either take the form of raw data to be input to the program, or may take the form of a unix shell script which can be used to drive the program in a more flexible way. (iv) a file of 'keywords' for each test. these are matched against the program output for programming exercises and against the students' responses for short-answer exercises. the keywords are actually unix regular expressions to be matched against the program's output. the use of regular expressions offers a high degree of flexibility in making this comparison. for example, the expressions: ft|feet ins|inches [ee]rror might appear in the keywords for a distance conversion problem. once again we can see that the teacher needs to strike a careful balance between tightness of specification and flexibility of testing. finally, the teacher needs to provide a mark-weighting file which assigns a name and relative mark-weighting to each test involved. such a file for a simple distance problem might be: 15 simple test 15 simple test 20 check "feet" " i n s " 62 alt-j volume! number 2 25 inches > 12 25 negative inches the teacher may also configure the relative weighting of style and complexity tests if required, and may even alter the relative weightings of the various software metrics used. monitoring progress both teachers and tutors have the ability to track the progress of individuals or groups of students. this is supported by a statistical summary package within ceilidh. three kinds of summary can be produced. (i) a student summary shows the progress of an individual student over a given course. this includes displaying his or her mark for each exercise along with the class average. it also includes the number of submission attempts per exercise, thus providing some indication of how hard the student found the exercise. (ii) an exercise summary displays the distribution of student marks for a given exercise, including the number of submission attempts per student. (iii) a course summary displays the average marks across the whole course, as well as the average number of submissions across the whole course. these facilities are available to tutors and teachers. students are encouraged to obtain a stripped-down personal summary of their own progress across a course. summaries may be presented as tables or as graphs which may appear on-screen or be printed. facilities are also h gnuplot percentage 100 80 60 40 20 i student zvc harks and group •-* "]"j •-1 — \ ._, l.i 1.2 2.5 2.6 2.7 3.2 3.6 3.7 4.3 4.4 4.5 5.1 5.2 unit.exercise m averages •-1 -i 5.5 6.1 6.4 6.6 7.1 7.5 7.6 8.2 8.3 figure 4: student summary mark (solid) against class average (dashed) for all exercises 63 steve benford et al computer-aided assessment and administration provided to batch-print summaries for each student on the course. for example, at nottingham, we have established the practice of mailing summaries to all tutors on a fortnightly basis. figure 4 shows a typical student summary. the plagiarism detection tool provides a quite different kind of summary. this facility takes all the solutions for a given exercise and compares them, looking for similarities. it reports both near-misses and what it believes to be exact matches. the facility is quite sophisticated in that it first removes all personalized aspects of each program (for example, comments and identifier names) in order to arrive at a core structure before making the comparison. final assessment the marking described so far is used to provide instantaneous feedback to students and staff, but several additional issues have to be considered if the system is to be used for formal assessment. first comes that of weighting. when computing a final mark for a course, it may not be sensible to give all exercises the same value (for example, early exercises are often worth less than later ones). secondly, the use of ceilidh tends to result in high marks, often above 80%. it is therefore necessary to scale marks to arrive at a final assessment in order to comply with institutional marking conventions (for example, a 60% average in ceilidh may represent a basic pass on the course). finally, there is the issue of deadlines for exercises and the problems of dealing with late work. ceilidh deals with these issues by letting the course teacher set up two files, the weights and scale files. the weights file lists each exercise along with an indication of its relative weighting and also its current status. the status may be open so that work can be submitted; late so that work can be submitted but is noted as late; or closed so that nothing can be submitted. the weights are used to automatically calculate a weighted average course mark for each student. teachers are provided with menu options to set the weights and alter the status of exercises. a separate scale file allows a teacher to define a scale mapping between raw marks obtained and marks on some final scale of assessment. implementation in this section we very briefly touch on some implementation issues. the current version of ceilidh runs, under unix and several of its derivatives. it provides three styles of user interface: (i) a line interface for dumb terminals, (ii) an x-windows interface. (iii) a command interface, where each ceilidh facility is available as a unix command. this allows people to build their own interfaces on top of ceilidh, and to integrate it with other applications. work is also currently underway to port ceilidh to a networked pc environment and to provide a pc microsoft windows interface. experiences with ceilidh ceilidh was initially conceived and constructed not as a research project but out of the necessity of having to teach programming to large first-year classes. consequently, ceilidh has always been used to support real programming courses, and has evolved to meet the needs 64 altj volume 1 number 2 of these courses. the fact that ceilidh was constructed while in actual use gave rise to many difficulties, particularly in the area of managing rapid updates to a live system. however, the great benefit of this approach is that instead of being an experimental prototype briefly demonstrated in the research laboratory, ceilidh is a fully working system that meets a real need and that has evolved to meet the needs of teachers and students. sources of feedback here we summarize our experiences of building and using ceilidh. this summary does not represent the results of a formal statistical/experimental evaluation of the effects of the system on the educational process. instead, it encapsulates several years of feedback, observation and opinion from a wide range of sources. these include: (i) our own subjective view as teachers and implementors. (ii) analysis of user-questionnaires. ceilidh contains questionnaires about the system and the courses supported, and these are made available online at the end of each course. (iii) archives of the many student comments and questions submitted through the online comment facility. (iv) the marks obtained across three years of use. (v) experiences of discussing ceilidh in examiners' meetings. (vi) feedback from other members of staff passing on student comments from tutorials. (vii) feedback from other organizations involved in piloting the system worldwide. (viii) student essays discussing their opinions of the system. more specifically over 100 students recently answered the following essay question: does the ceilidh system for teaching c++ produce good programmers who can write high-quality, thoroughly tested and stylistically correct programs, or does it merely produce students who have learned how to 'play the system' and who effectively rely on it to solve their problems for them? impact on students we begin this section with observations on the ways in which ceilidh has affected different kinds of student: (i) the system has acted as a confidence builder for novice programmers who benefit greatly from the kind of positive early feedback that arises from the early simple exercises. one particular aspect of this has been building the confidence of female students who may often initially feel intimidated by the 'macho' image associated with programming. (ii) the system has enabled us to spot the really weak students early on, enabling us to focus effort on helping them. (iii) nearly all the critical feedback has come from experienced students, particularly over the marks assigned. the notion of good programming style was a particularly contentious issue, and we had several discussions on the issues of style and 65 steve benford et al computer-aided assessment and administration standardization of layout. this turned out to be beneficial as it is the self-taught 'experts' who often need the most help in this area (even if they think they do not). this last observation is of key importance. one of the most surprising and pleasing aspects of ceilidh was its role in consciousness-raising. the provision of immediate feedback by a machine produced much more discussion of programming correctness and style than did previous hand-marking. ceilidh's online comment facility played a key role in this discussion, and one of the early extensions to the system was to offer the students the opportunity to comment on each mark when given. consequently, our experience of automatic assessment is that far from reducing contact with students, the quantity and quality of discussion is increased. looking back, we suspect that this effect stems from a number of factors. firstly, immediate feedback means that marks are received while the problem is uppermost in a student's mind. secondly, students may be happier to argue with a machine than with a teacher. thirdly, the marking process is generally more open to inspection than with hand-marking (for example, style rules are published and are applied consistently). as a further comment, hand-marking of any form of coursework can lead to a student being treated less fairly than others. for instance, coursework marked by more than one person will lead to inconsistencies in marks awarded due to differing ideas of what the correct answer should be. this coupled with other problems such as racism, sexism and favouritism can lead to certain students achieving poorer or better marks than they deserve. we believe that such explicit discrimination is reduced, if not eliminated, by the use of the ceilidh system, since it marks each solution consistently. however, we recognize that implicit discrimination through inappropriately chosen questions must still be guarded against. of course, the introduction of ceilidh was not without its problems and we did observe a number of more negative reactions to the system. one problem-group came to be known as the perfectionists. these students seemed unable to stop working on an exercise even when a satisfactory mark had been obtained. a second problem-group was that of gamblers: students who iterated around the marking cycle many times, tweaking their programs in an attempt to pick up extra marks without necessarily thinking through the problem at hand. perfectionists and gamblers resulted in our making two further extensions to the system. the first was to introduce a minimum time-delay between markings during which time a student could not re-mark his or her program. this delay is tailorable by the teacher, and may range from a few seconds to several days. a more general advantage of this extension is that it allows ceilidh marking to be used in a once-only fashion if the teacher wishes. the second extension was to stretch the progress-monitoring facilities to spot the problem-students. this involved three new facilities: (i) producing graphs of number of marking attempts per student across an exercise or course. (ii) producing graphs of the 'development profile' for each student completing an exercise. the development profile charts the mark obtained against each marking attempt and so visually shows progress across an exercise. (iii) calculating and producing graphs of a 'development ratio' for each student and exercise, a numerical measure that indicates whether the student has been making progress. 66 alt-j volume 1 number 2 the development ratio is calculated as the ratio of the number of times a student's mark increases minus the number of times it decreases divided by the total number of marking attempts. this results in a scaled numerical measure in the range +1 to -1 which shows general progress across an exercise. at the extremes, +1 indicates the student's mark always increased on each successive re-marking, and -1 that it always decreased. a development ratio of 0 would indicate that no progress was made, and one close to 0 that little progress was made. ceilidh automatically calculates these ratios for all students and exercises, and can display them graphically. as an example, figure 5 shows the development profile (i.e. mark awarded plotted against the submission attempt) for an individual doing a single exercise. in this case, we see a steady improvement in the mark awarded with each attempt until nearly 100% is reached. mark profile for student kin. unit 8 exe 3 figure 5: example graph of progress on an exercise we emphasize that the development profile and ratio provide only a rough general guide to progress, not a certain measure. furthermore, graphs have been used by staff to spot potential problems and have not counted towards final marks. graphs provide teachers with a rough guide to potential problem-students (for example, perfectionists or gamblers) who might need additional teaching support. teachers might then encourage these students to change their working method, either by learning better to manage their work according to sensible quality targets, or to think more about the problem at hand before changing their program. in more general terms, the cases of perfectionists and gamblers both highlight ceilidh's role in 67 steve benford et al computer-aided assessment and administration encouraging students to manage their own work in an effective way. we believe that this is possible because ceilidh devolves greater responsibility to students for determining final marks, and therefore for managing their work load. before passing on, we should briefly mention one hidden problem. ceilidh requires that students hand in their work in electronic format. indeed, at nottingham we currently insist that nearly 75% of all core first-year work is handed in online (not just programs). this makes it difficult for students to work with pen and paper in their own residences. of course, many students own their own computers and are able to transfer their work across to our system. however, we fear that those who do not may be disadvantaged. we believe that the solution to this problem quite simply (in technical if not financial terms) lies in the provision of a better computing infrastructure. in particular, we would like to see all our students armed with personal or notebook computers within the next few years. impact on teachers and tutors so far, we have considered ceilidh's effect on the students. we also need to consider its effect on the staff. the clear and simple benefit to teachers has been a massive reduction in time spent marking students' programs. ceilidh has also helped with more efficient administration of courses. collection and collation of work has been trivial, deadlines are published in advance and are stuck to, and marks are returned to examiners on time. ceilidh also helps with the trivial, but often annoying, aspects of coursework such as making sure that work is clearly labelled and legible. this was particularly noticeable where the collection and collation of essays was concerned. clearly some of these issues extend to courses other than programming. the general progress monitoring facilities within ceilidh have been of benefit to both tutors and teachers and have allowed us to keep track of our students. as an example, some tutors actually hand out progress charts to students during tutorials to confirm progress on the course. interestingly, far from viewing this as a draconian regime, as some people feared, many students seem to appreciate regular confirmation of satisfactory progress. use in formal assessment ceilidh is used both to give feedback to students and for formal course assessment. the current first semester programming course at nottingham is entirely coursework-assessed and the second is 50% coursework-assessed with the other 50% being a formal written examination. this dual use of ceilidh has given rise to several interesting issues. not only does the iterative use of auto-assessment tend to result in very high raw marks (with marks around 80% and 90% being common) but also the marks tend to be far more tightly grouped (often the case in continuous assessment). even with the scaling facilities mentioned previously, it may be difficult to categorize the students according to an 'expected' normal distribution. this raises the issue of the relation between feedback and assessment. indeed, it is obvious that ceilidh, by providing interactive feedback, is increasing the likelihood of obtaining high marks. one solution is to re-mark the students' work at the end of the course using different criteria (for example, using different and harder tests and more rigorous style checking). however, we then encounter the problem that students feel cheated because the system has been reporting excellent progress which may not match their final mark. in general, this issue relates to whether we are seeking normative or criterion assessment of students. so far we have used ceilidh on first-year foundation courses where criterion 68 alt-j volume 1 number 2 assessment is a sensible approach (i.e. if you can demonstrate the necessary skills, you can pass the course). the role of ceilidh, and indeed of any iterative auto-assessment technique, may be limited where normative assessment is required (i.e. on courses in later years). the issue of students feeling cheated is also an interesting one. there is a danger that ceilidh might encourage students in the belief that the ability to pass a few tests makes their programs certifiably correct. of course, this is not true, and it is recognized that testing, no matter how thorough, does not prove program correctness. it is therefore important that teachers carefully introduce the notion of program correctness and its relation to testing, particularly with respect to other approaches such as the use of formal methods in program verification. use at other sites so far, ceilidh has been installed at 25 sites outside nottingham and has been used to teach courses in at least five of these. styles of use have been quite diverse. more than one site has informed us that they installed the system and ran our course directly as given, even using our slides for lecturing. other sites found it more difficult to accept the given course structure but were happy to use the exercises to form their own course. at the other extreme, one site used the ceilidh system as an envelope to write an entirely new course for a different programming language (pascal). this also necessitated slotting their own auto-assessment module into the ceilidh framework. this course has now been re-installed at nottingham and will soon be distributed to other pilot sites. one key point to emerge from early piloting has been the importance of providing an open and extensible framework for ceilidh. not only should staff be able to build their own exercises, units and courses, but should also be able to bolt in their own marking modules for other programming languages or even for other kinds of assessment (we know of at least one other site where automated essay marking is being researched). in particular, there should be a separation between ceilidh's general course administration framework and different marking techniques, with the latter ranging from completely automated marking though semiautomation to hand-marking. although our current experience would broadly agree with laurillard et al (1993) in that 'the not invented here syndrome is not a major influence on academics considering the use of cbl', we would also stress that courseware should aim to provide as modular a framework as possible so as to allow teachers to mix and match resources, blending them into their own courses with individually tailored structures. multi-site use also requires support for the ability to transfer resources across networks for installation in local systems. in turn, this requires the development of adequate standards for course and resource structure (for example, is our current course-unit-exercise hierarchy general enough?). it also requires integration with more general information dissemination technologies including email, file transfer, directory services and network information retrieval systems (for example, gopher, wais, archie and world wide web). we are hoping that our current pilot project will help develop the necessary standards and undertake the necessary integration to achieve this. future work future developments of ceilidh fall into two categories. those that are scheduled to take place during the tltp pilot project, and more general developments. 69 steve benford et at computer-aided assessment and administration during the tltp pilot project, the following developments are scheduled: (i) porting ceilidh to a networked pc environment. (ii) supporting additional programming languages. the working set should at least include c++, c, pascal, modula-2, ada and ml. (iii) providing an extended statistics-gathering and progress-monitoring facility. (iv) integrating the current crude facility for reading course notes with common authoring and hypertext systems (key candidates include authorware and guide) (v) developing full x-windows and microsoft windows interfaces. (vi) extending the current 'comment' facility towards a fully-fledged help desk with support for rostering queries between a team of advisers working across a computer network. (vii) extending facilities for exchanging course resources (e.g. notes and exercises) between sites, perhaps via integration with emerging network information retrieval systems. a further goal for the near future is to conduct a more formal evaluation of the ceilidh system. some limited funding has recently been obtained towards this end, and we hope to compare student performance under ceilidh with other indicators such as examination results and a-level entry grades. we also aim to cany out some marking experiments comparing automatic and human marking of the same scripts. we plan a number of longer-term ceilidh developments. firstly, we will try to integrate the system with other local administration facilities, including nottingham university's examinations and timetabling databases. we also aim to provide greater support for co-operative working. we plan experimental integration of conference facilities (both audio and text) to support discussion between students and staff, and also inclusion of bulletin board facilities. our target here is a kind of networked programming laboratory distributed across several sites. a final word: we would like to stress that ceilidh is a fully working system and available from the authors. references benford, s., burke, e. and foxley, e. (1993), 'learning to construct quality software with the ceilidh system', the software quality journal. laurillard, d., swift, b. and darby, j. (1993), 'academics' use of courseware materials: a survey', association for learning technology journal, 1 , 1 . zin, a.m. and foxley, f. (1991), 'automatic program quality assessment system', proceedings of the ifip conference on software quality, s p university, vidyanagar, india (march). 70 doi:10.3402/rlt.v19s1/7800 effectiveness of technology to support work based learning: the stakeholders’ perspective rebecca strachan*, lalith liyanage, biddy casselden and roger penlington school of computing, engineering and information sciences, northumbria university, newcastle upon tyne, uk (received 2 march 2011; final version received 4 june 2011) higher education provision typically requires learners to physically attend sessions on campus. the economic climate has changed significantly over the past few years in the uk and globally. inevitably changes to student funding and the increased competitive nature of the job market have impacted on university teaching. the use of work based learning (wbl) is an alternative flexible form of learning that attempts to tackle these issues. it enables students to learn whilst they work, addressing the funding issues, and enhancing their employability through the acquisition of higher professional qualifications. often such wbl programmes are designed, delivered and supported from the view of the student and academic staff with little consideration of other stakeholders such as employers, workplace mentors and professional bodies and the input they can bring to enrich the learning and teaching provision. this paper presents the findings from a survey conducted among stakeholders from all four pillars of wbl, namely the learner, the academic environment, the workplace and the external context. online questionnaires and interviews were carried out with students, tutors, program leaders, employers and professional bodies from four postgraduate programmes at the university. the results show that while there is a reluctance to embrace technology among some academic staff, students are generally positive about using the technology. the survey also demonstrates that there is a lack of creativity and imagination in the use of technology, where often platforms such as virtual learning environments are used simply as repositories for presentation slides, handouts, etc. the results of the study conclude or rather remind all involving parties to pay more emphasis on quality of online programme delivery by embracing technology and use it in novel and imaginative ways to provide a learning and teaching provision fit for the twenty-first century. keywords: work based learning; professional body; e-learning; distance learning; online learning introduction work based learning (wbl) is the term used to describe a class of university programmes that brings together universities and work organisations to create new learning opportunities in workplaces (boud and solomon 2001). such programmes meet the continuing professional development (cpd) needs of learners, contribute to the longer-term development of the organisation and are formally accredited as university courses. *corresponding author. email: rebecca.strachan@northumbria.ac.uk alt-c 2011 conference proceedings 0126 isbn 978-91-977071-4-5 (print), 978-91-977071-5-2 (online) 2011 association for learning technology. # r. strachan et al. this is an open access article distributed 134 under the terms of the creative commons attribution-noncommercial-noderivs 2.0 uk: england & wales licence (http://creativecommons.org/licenses/by-nc-nd/2.0/uk/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. doi: 10.3402/rlt.v19s1/7800 in this context, online wbl has been viewed as a way to increase access to higher and continuing education that attempts to engage seriously with the economic, social and educational demands of our time. interestingly, it provides a fundamental challenge to existing practices. this study looks at the technology aspects of wbl from a number of perspectives including the external professional and workplace environments, the academic environment and the student experience. the paper provides an overview of wbl concepts, the context in the uk and in particular that at northumbria university. the case study research methodology is explained, results are presented as an evaluation of technology and the main findings and conclusions are given in terms of the various stakeholders. background work based learning has increasingly become an area of interest for the higher education (he) sector and can support the personal and professional development of students who are already in work. the focus of learning and development tends to be on the student’s workplace activities rather than a set curriculum (brennan and little 2006; durrant, rhodes, and young 2009). deploying technology is one solution used to overcome the issue of increasing access to ‘opportunity lost’ or ‘demand driven’ students. how to effectively conduct distance education (de) has been a key topic for researchers for many years. the primary difference between face-to-face and de systems is that the former is mainly ‘‘teacher-centred’’ while the latter is ‘‘learner-centred’’ (liyanage 2010), though this distinction is becoming blurred. taylor (2001) describes the evolution of technological innovation in de (see table 1). context of wbl in uk evans (2001) explains that wbl for academic credit was developed in the uk in the 1980s to respond to the rapid change in the social and economic and hence educational life of the country and the perceived inadequate skills and knowledge levels of the workforce in general. it challenged the myth that learning at he level cannot happen in the workplace. greater effort was put into expanding he while urging companies and he to be more active through collaborations to widen access and challenge previous boundaries. wbl introduced many mutual benefits for both institutions and employers with the main focus on ‘learning from experience’ and a shift away from the traditional curriculum and institutional structures. flexible access into wbl was provided through the introduction of accreditation of prior learning (apl) and accreditation of prior and experiential learning (apel) (boud and solomon 2001). wbl at northumbria university northumbria university, a pioneering and leading institution for wbl, recognises it as a vital mode of learning for increasing participation and supporting professional development among employers and their staff. several important endeavours have taken place in the university to support wbl. the work related learning services alt-c 2011 conference proceedings 135 table 1. generations of de. characteristics of delivery technologies flexibility models of de and associated delivery technologies time place pace highly refined materials advanced interactive delivery institutional variable costs approaching zero 1st generation: correspondence print yes yes yes yes no no 2nd generation: multimedia print, audio tape and videotape yes yes yes yes no no computer-based learning (e.g. cml/cal/imm), and interactive video (disk and tape) yes yes yes yes yes no 3rd generation: telelearning audio-teleconferencing, and video-conferencing no no no no yes no audiographic communication, broadcast tv/radio and audio-teleconferencing no no no yes yes no 4th generation: flexible learning interactive multimedia (imm), online yes yes yes yes yes yes internet-based access to www resources, computer-mediated communication yes yes yes yes yes no 5th generation: intelligent flexible learning as 4th generation plus computer-mediated communication using automated response systems, campus portal access to institutional processes and resources yes yes yes yes yes yes source: adapted from taylor (2001). r . s tra ch a n e t a l. 1 3 6 (wrls) established in 1999, developed a portfolio of innovative and relevant workrelated learning products across the institution. the service explores current thinking to identify and advise on strategy, direction and new opportunities and develops and tests curricula, learning products and infrastructure responding to the demands of employers, students, the university and other agencies (bennett 2010). its role in wbl has been acknowledged by the higher education academy (nixon et al. 2006). in 2005, northumbria university developed a work based learning framework (wblf) allowing organisations to offer their workforce highly relevant professional development programmes designed to fit their specific needs. the wblf offers awards that can be customised to the learners’ requirements and is designed to be flexible and accessible (university of northumbria 2010). in addition a central university team of learning technologists (ltech) provides a service to academic staff and students on how ‘‘to enable the best use of new and existing technologies to enhance the student learning experience’’ (ltech 2011). these initiatives have enabled northumbria university to offer alternative modes of study effectively (liyanage et al. 2010) and about a third of northumbria’s 30,000� students study in part-time rather than full-time mode (hesa 2011). aim and background of the study wbl endeavours have helped employees and their organisations access he in a more flexible way. however, one area that needs further attention is the support provided during the learning experience itself, and evaluating to what extent it caters for the needs of all those involved in the wbl programme. liyanage, pasqual, and wright (2010) illustrate that the expectations of various stakeholders in an online learning environment are very different from each other yet are rarely addressed. for example chong, martinsons, and wong (2004) in their study of the factors that influence the learners’ perception and adoption of work-based e-training only pay attention to the learner. the current study builds upon a model of wbl with four pillars: the learner, the academic environment, the workplace and the external context. the key aim is to investigate the perceptions of the various stakeholders on the effectiveness of wbl programmes and their use of technology. it draws on four contrasting programmes within the university of northumbria, three being closely linked to their professional body (pb). these programmes are the ma/msc in information and library management, the msc in records managefigure 1. wbl and the stakeholder contexts. alt-c 2011 conference proceedings 137 ment, both two years distance learning delivery with supporting study schools, and the msc professional engineering, a three year wbl programme. the fourth programme, msc computing and information technology (it) is three years by distance learning and is not linked to a pb. it was initially set up for adult working ‘women returners’, although it now caters for anyone looking for a postgraduate it qualification via distance learning. method this research adopts the case study method, appropriate when the purpose of the research requires holistic, in-depth investigation of a phenomenon or a situation from the perspective of all stakeholders involved. case studies are not intended to produce generalisations, they allow for transferability of findings based on contextual applicability (pickard 2007, 93). yin (2002) defines case study research as ‘‘empirical enquiry that investigates a contemporary phenomenon within its real-life context; when the boundaries between phenomenon and context are not clearly evident’’. a case study can be qualitative in nature, quantitative or a mix (stake 2003). this study takes the latter approach with quantitative and qualitative data obtained via student questionnaires to maximise the numbers of students and qualitative data acquired from the rest of the stakeholders using interviews and documentation. triangulation is achieved within the case study by using multiple data collection techniques ‘to pick triangulation sources that have different biases, different strengths, so they can ‘‘compliment’’ each other’ (miles and huberman 1994). the design of the case study research is an iterative process which gives flexibility for discovery and exploration in the field as it goes. sample size in the case is representative of the human population that are involved in wbl at postgraduate masters level in the school of computing, engineering and information sciences of northumbria university. all current students were asked to complete the online questionnaire while interviews were conducted with the four programme leaders, a representative sample of the module tutors and workplace mentors and relevant officials of the professional bodies. contribution the significant difference between this study and others is the addition of both the workplace and the external context, to give a model with four pillars (see figure 1). this four-way dialogue does not suit standard online learning platforms as the mentor and pb do not have the same contract with the university that exists between the academic staff and the learner. the results from the survey support this. when members of the pb were questioned about the type of communications that happen with the universities/employers’ associations with regard to wbl, one answered ‘‘as head of the accreditation team, i will visit the programmes every five years, for the accreditation visit. all programme directors are free to contact the team at any point between accreditation visits, although no formal meetings are arranged’’. yet it is also clear they want better communication as demonstrated by the following comment: ‘‘partnership, rotation, and proximity (or at least lack of barriers generally) between ‘‘academia’’ and ‘‘work’’ is much to be desired’’. employers saw remoteness, lack of feedback and lack of student contact as a major disadvantage of wbl for their employees while students and academic staff r. strachan et al. 138 both indicated they would like to be able to communicate with each other and employers in an effective and easily accessible manner. the significant contribution from this study stems from the efforts taken to understand and evaluate the link between the profession and workplace and the traditional learner-tutor academic environment. in the short-term this should aid understanding of these relationships, the support they require and determine to what extent technology can be an enabling factor. in the longer term this study should help improve the quality and effectiveness of wbl by catering for all the stakeholders involved and drawing on technology in more creative and valuable ways, leading ultimately to a more appropriately educated and developed workforce. results and evaluation the collected data were analysed using narrative and statistical analysis using nvivo (qsr 2011) and spss (ibm 2011) software tools respectively. one hundred and fifty-five students were asked to take part in the online questionnaire and 60 responses were received giving an overall response rate of 38.7%. fourteen interviews were conducted with programme leaders, tutors and professional bodies. the following presents the main findings in terms of five main areas of technology: (1) elearning portal (elp) (2) communication (3) assessment (4) content (5) technological support to students elearning portal the elp is the main mechanism for supporting the delivery of learning and teaching and is used to replace the physical classroom environment for these learners. this has been customised from the ‘blackboard’ virtual learning environment (vle). students and tutors had contradicting views on the user friendliness of the elp. among students 61.7% of students were happy about the user-friendliness of the elp while a further 26.7% were neutral. only 11.7% of students disagreed and found the elp not user friendly. this contrasts with the results of the interviews held with academic staff (both module tutors and programme leaders) that generally held quite negative views on using the elp. typical academic staff comments included: ‘‘i wouldn’t say it’s perfect it’s clunky and too many functionalities, which is frustrating which takes a lot of time. you have no other option you’ve got to live with it’’ and ‘‘it’s tedious to upload content especially attachments because you cannot upload more than one at a time. . .. formatting is a big problem in the elp having to re-do documents/copy-paste content. formatting is very poor and tedious’’. one reason for this is that students primarily access the elp as users to contribute to online activities and study content. academic staff accesses the elp to set up modules and populate and manipulate them to provide online content and activities for the students. therefore their views reflect the difficulties in using the elp from a control and management viewpoint rather than as a learner. other than the elp, alt-c 2011 conference proceedings 139 some academics and students prefer to use the pebblepad e-portfolio for their teaching and learning activities. communication the main communication channel among learners, tutors and programme leaders is email while telephone, elp discussion areas and occasionally skype/video are also used. the main issue raised with regard to using skype was timing and issues with access at the university as noted in the following comments: ‘‘i want to do vc from my pc but the problem is due to the fact that as my students are working in other roles � they would not be able to sit at their desks and skype about something not to do with work’’. with regard to emails, tutors raised concerns about the response time and the overall time taken for each and every student’s email queries: ‘‘my standard response time for student queries is 48 hours although i normally respond within 2 hours. but i do not access office mail after 5 pm on weekdays and entire weekends because do not want to become a slave to emails. i do appreciate that wbl/dl students’ work style is different (after work hours and weekends) but i work full time during the week!’’ tutors also recognise that asynchronous chats and discussion boards (db) are useful although they are not always used. ‘‘i have only discussions and asynchronous chats because we cannot synchronise with everybody’s time schedules’’ and ‘‘my students don’t make use of the discussion board even though it is available on most modules. they prefer to engage on an individual email discussion with the relevant tutor’’. only 23% of students indicate that they would prefer a physical community environment for learning. however 48% of students indicate they would prefer blended learning where distance online learning is supported by some physical classroom sessions compared to ‘pure’ distance learning with a further 27% being neutral on this issue. as one student comments ‘‘although db are helpful they cannot replace the classroom atmosphere with its spontaneous interaction’’. communication between the university, professional bodies and employers mainly happens via phone or email or in the occasional face to face meeting. this communication focuses on strategy and high level elements such as accreditation rather than operational issues or direct support for students. for example, one pb comments ‘‘we held a meeting of employers. . . to inform them how wbl might be incorporated in professional development’’ and another commented ‘‘universities have a good relationship with the pb through the work of the accreditation team and we are in regular contact’’. the latest trend for communication is the use of social networking media like twitter, facebook, and blogs and also for collaboration tools like youtube, bookmarking and wikis. the university has a system that links the student information system to a texting system which enables the university to text students on their mobile phones. currently used primarily to inform students of late changes to their timetables, etc. the system has been welcomed by staff and students alike although care has to be taken not to ‘overload’ students with too many texts, so use is restricted to a small subset of staff to control the overall number being sent. r. strachan et al. 140 assessment tutors can choose the form of assignment submission from a physical hard copy to electronic submission via the elp, email or a mix of these. even within the elp there are different methods of submission available. this causes confusion among the students. with last minute stress as the assignment deadline looms, students submit assignments using the method they find most easy or can remember and this sometimes leads to assignments being misplaced/not received by tutors for marking. some tutors still prefer hard copy submission for two reasons: it avoids any technical issues (both with tutor set-up and student submission) and tutors prefer marking physical copies rather than online versions. providing marks and feedback is another area where technology could help but tutors have different views on this: i don’t use the assessment facility � i do post up percentage grades � but not all module tutors do this, and my style of marking means i do not use the other facilities offered in grade center � it doesn’t suit my marking approach � and would take me longer. i need to mark as efficiently as i can in a way that suits me. for assessments, i do not use any online facilities because i mark on the go in the train, at home, in the evenings etc so i mark on the paper by pen ‘‘digital plagiarism is a problem for educators all over the world’’ (butakov and vladislav 2009) and (rowe 2004). online assessment submission raises serious security issues as methods of cheating are facilitated, some quite new, and it is inevitable that plagiarism will increasingly be automated and distributed as software packages. while there are countermeasures, online assessment in distance-learning programs should be done with caution, make use of the software tools available to uncover digital plagiarism and be continually reviewed. tutors were aware of the dangers and commented ‘‘i have found one incident where collusion was established between home and distant students. i use video conferencing in assessments of projects where students have to demonstrate the project kind of a viva’’ and ‘‘we put suspicious papers through turnitin software. rather than creating them opportunities to cheat, if we can design assessments tactfully it would be better’’. the main criticism against online learning is that students are more inclined to plagiarise than in face-to-face situation due to the fact that distance makes it hard for tutors to distinguish between genuine and plagiarised work, but the following tutor quote also held by other tutors, provides an alternative view: ‘‘dl students seem less inclined to plagiarise than face-to-face students’’. content unlike face to face, where tutor-student contact happens through lectures and seminars, online wbl students mainly rely on online content. questioned about their four most recent modules, the survey showed that for their most recent module, 78% (plus 18% neutral) of students agreed that the online learning material was of a high quality (and for their second module there was a 75% (plus 10% neutral) agreement rate. when questioned about the format of the content 67% of students prefer multimedia elements to aid learning/understanding, with a further 13% being neutral on this issue. one student comments ‘‘the learning materials could have been more varied (e.g. video casts or lectures, live chats)’’. surprisingly, 18% of students either do not like the inclusion of them or can see no difference in having them. alt-c 2011 conference proceedings 141 quality is subjective and for students, this could be their first online learning experience and thus they may have little to compare their experience against. the professional bodies are also satisfied with the quality of the online learning materials stating ‘‘. . . the accreditation teams are happy with the materials in terms of relevance, interactivity and currency’’. the tutors’ view on the quality and interactivity of the learning materials is valuable: ‘‘a lot of dl students like to have materials with interactions embedded into it through self-assessment activities and dbs but not necessarily have to be online to do them. especially, they don’t like to have online activities with deadlines which could become hectic with their other commitments’’ and importantly ‘‘i believe in ‘‘technology should not drive pedagogy but pedagogy should drive technology’’’’. interestingly the tutors recognise the university support provided in this area via the central learning technology support team, ltech but as two tutors comment: ‘‘university provides loads of training but i don’t have time. would like to use podcasts and video clips in my materials but the time is the constraint again’’ and the content is ‘‘. . . essentially word documents � and not that innovative electronically � but at least they can be printed out in full, and contain exercises for checking understanding etc. they are updated � but the task is a mammoth one � and there never seems enough time to fully update materials’’. some of the tools being used for online content development are flash, wimbacreate, podcasts, turningpoint and smartboard. technological support to students universities must recognise the importance of this mode of education and provide due recognition and technical support wherever possible. there are two main ways that students can access university resources: firstly is via the university website and elp, and secondly via a virtual tunnel and a thin client application called ‘desktop anywhere’. this acts as a remote access facility to allow students to access specialised software and the shared drive similar to logging onto one of the campus pcs. students find ‘desktop anywhere’ cumbersome to use due to technical incompatibilities. in the online survey, almost 50% of the students failed to access the questionnaire which was hosted on one of the servers through ‘desktop anywhere’. subsequently, a microsoft office version of the questionnaire had to be sent to students. students’ comments on this included: ‘‘‘desktop anywhere’ should be clearly explained as it allows non uk users to access the library in a timely manner’’ and ‘‘my computer doesn’t like ‘desktop anywhere’ at allhave had real problems trying to use itso it wasn’t just your questionnaire. online learning mainly depends on technical support provided by the delivery institution therefore the it services and online library fall into the category of ‘vital’ in this sense. students have assessed them as follows: (1) it � 70% satisfied (2) library � 75% satisfied r. strachan et al. 142 but academics were less favourable in their comments ‘‘i would prefer to have direct contact with elp rather than going through it helpline first. this would save time and effort’’. the other concern regarding online learning is the challenges faced by students and tutors when coping with technology. when asked about the ease of adapting to online learning, 33% of students agree it was easy with 53% disagreeing. this response reflects the distribution of students across the disciplines and their individual backgrounds in terms of it literacy and previous online learning experience. academic tutors commented: ‘‘students require appropriate equipment to access courses (pc�internet) whereas f2f students can access or learn from university facilities’’ and it is ‘‘. . . costlier for students in some countries where communication infrastructure is less developed (3rd world countries)’’. they also commented on the challenges of keeping up with the technologies and the incompatibilities between different equipment and systems that students may have access to. interestingly ‘keeping up with technological developments’ is seen by both employees and employers as one of the core benefits of wbl (glass, higgins and mcgregor 2002). currently there is no access to university it systems for employers or professional bodies. any information they need is communicated via email or in hard copy. limitations of using technology observed in the survey are . elp does not work on some mobiles due to embedded flash content in learning materials or special software is needed. . students’ and staff digital literacy plays a major role when implementing new media tools. . use of synchronous video conferencing is often not practical with distance learning students due to work commitments and time differences across different geographical locations. . sustainability over time is another problem with the rapid development of technology. . compatibility among different software/hardware systems and networks. as one tutor commented ‘‘cannot update content with the rapid development of technology and evolution of web 2.0 technologies which young 18� under graduate students like to explore. technical incompatibilities with different systems like mac/windows/apple etc with different specifications’’. the management of the university has taken several steps to address the issues identified in the research as follows: (1) appreciate and allocate wbl/dl time in the staff time table (2) enhance ltech support by allocating individual representatives/coordinators to each school (3) improve ease and speed of online access through dta (4) provide better awareness about university facilities for wbl/dl students (5) create more friendly and efficient it, library, finance and student services for dl/wbl students alt-c 2011 conference proceedings 143 conclusion there is previous research on technology-enabled wbl, but so far there has been limited consideration of all the various stakeholders. this study looks at four pillars of wbl: the learner, the academic environment, the workplace and the external context through a questionnaire survey of students and interviews with other stakeholders. the results show that a number of factors facilitate and/or obstruct the effective implementation of technology to support wbl and there are still a number of barriers to using technology in novel and imaginative ways to provide learning and teaching provision fit for the twenty-first century. references bennett, s. 2010. interview on wrls. (accessed june 16, 2010). boud, d., and n. solomon. 2001. work-based learning: a new higher education? 1st ed. buckingham: open university press. brennan, j., and b. little. 2006. towards a strategy for workplace learning: report of a study to assist hefce in the development of a strategy for workplace learning. london: centre for higher education research & information. http://oro.open.ac.uk/6437/1/towards astrategyrd09_06.pdf. (accessed june 8, 2010). butakov, s. and s. vladislav. 2009. the toolbox for local and global plagiarism detection. computers and education, 52, no. 4: 781�788 [online]. http://www.sicencedirect.com (accessed july 25, 2011). chong, b., m.g. martinsons, and m. wong. 2004. adoption of e-learning for work-based training: an exploratory study of the hong kong apparel industry. international journal innovation and learning, 1, no. 3: 312�26 [online]. durrant, a., g. rhodes, and d. young. 2009. university-level work based learning. middlesex, uk: middlesex university press. evans, n. 2001. from once upon a time to happily ever after: the work-based learning in the uk higher education sector. in work-based learning: a new higher education, ed. d. boud and n. solomon. 61�73. buckingham, uk: open university press. glass, a., k. higgins, and a. mcgregor. 2002. delivering work based learning. http:// www.scotland.gov.uk/publications/2002/06/14558/3244. (accessed july 25, 2011). hesa. 2011. students and qualifiers data tables, statistics online, higher education statistics agency. http://www.hesa.ac.uk/index.php/content/view/1973/239/. (accessed july 25, 2011). ibm. 2011. spss statistical analysis software package, ibm. http://www-01.ibm.com/software/ analytics/spss. (accessed july 25, 2011). liyanage, l. 2010. from traditional to technology enabled tertiary education in sri lanka. germany: lambert academic publishing. liyanage, l., a. pasqual, and c. wright. 2010. lessons learned in managing ict systems for online-learning. 5th conference of learning international networks consortium (linc) university leadership: bringing technology-enabled education to learners of all ages from may 23�26. the campus of the massachusetts institute of technology in cambridge, massachusetts, usa. http://linc.mit.edu/linc2010/proceedings/session9liyanage.pdf (accessed april 30, 2011). liyanage, l., r. strachan, s. keogh, and s. hanlon. 2010. evaluation of using distance learning to support work-based learning programmes: a case study from northumbria university, uk. e-learn 2010, orlando, florida (accessed october 19, 2010). ltech. 2011. learning technologies of northumbria university. http://www.northumbria.ac.uk/ sd/central/ar/academy/ltechhomepage/?view�standard[accesssed date] miles, m.b., and a.m. huberman. 1994. qualitative data analysis: a sourcebook of new methods, 2nd ed. london: sage. r. strachan et al. 144 http://oro.open.ac.uk/6437/1/towardsastrategyrd09_06.pdf. http://oro.open.ac.uk/6437/1/towardsastrategyrd09_06.pdf. http://www.sicencedirect.com http://www.scotland.gov.uk/publications/2002/06/14558/3244 http://www.scotland.gov.uk/publications/2002/06/14558/3244 http://www.hesa.ac.uk/index.php/content/view/1973/239/ http://www-01.ibm.com/software/analytics/spss http://www-01.ibm.com/software/analytics/spss http://linc.mit.edu/linc2010/proceedings/session9liyanage.pdf http://www.northumbria.ac.uk/sd/central/ar/academy/ltechhomepage/?view=standard http://www.northumbria.ac.uk/sd/central/ar/academy/ltechhomepage/?view=standard http://www.northumbria.ac.uk/sd/central/ar/academy/ltechhomepage/?view=standard http://www.northumbria.ac.uk/sd/central/ar/academy/ltechhomepage/?view=standard http://www.northumbria.ac.uk/sd/central/ar/academy/ltechhomepage/?view=standard http://www.northumbria.ac.uk/sd/central/ar/academy/ltechhomepage/?view=standard nixon, l., k. smith, r. stafford, and s. camm. 2006. work-based learning: llluminating the higher education landscape [online]. york: the higher education academy. http:// www.asetonline.org/documents/wbl_illuminating.pdf (accessed may 30, 2010). pickard, a.j. 2007. research methods in information. london: facet publishing. qsr. 2011. nvivo software, qsr international available at: http://www.qsrinternational.com/ products_nvivo.aspx. rowe, n.c. 2004. cheating in online student assessment: beyond plagiarism. online journal of distance learning administration, vii, [online]. http://www.westga.edu/�distance/ojdla/ summer72/rowe72.html (accessed february 18, 2011). stake, r.e. 2003. case studies. in strategies of qualitative inquiry, ed. n.k. denzin and y.s. lincoln. chapter 16. london: sage. taylor, j. 2001. fifth generation distance education. e-journal of instructional science & technology, 4, no. 1: 1�14 [online]. http://eprints.usq.edu.au/136/. (accessed july 25, 2011). university of northumbria. 2010. work-based learning framework. www.northumbria.ac.uk/ sd/central/ar/lll/lla/. (accessed july 25, 2011). yin, r.k. 2002. case study research: design and methods, 3rd ed. newbury park, london: sage. alt-c 2011 conference proceedings 145 http://www.asetonline.org/documents/wbl_illuminating.pdf http://www.asetonline.org/documents/wbl_illuminating.pdf http://www.qsrinternational.com/products_nvivo.aspx http://www.qsrinternational.com/products_nvivo.aspx http://www.westga.edu/~distance/ojdla/summer72/rowe72.html http://www.westga.edu/~distance/ojdla/summer72/rowe72.html http://www.westga.edu/~distance/ojdla/summer72/rowe72.html http://www.westga.edu/~distance/ojdla/summer72/rowe72.html http://www.westga.edu/~distance/ojdla/summer72/rowe72.html http://www.westga.edu/~distance/ojdla/summer72/rowe72.html http://eprints.usq.edu.au/136/ http://eprints.usq.edu.au/136/ http://eprints.usq.edu.au/136/ http://www.northumbria.ac.uk/sd/central/ar/lll/lla/ http://www.northumbria.ac.uk/sd/central/ar/lll/lla/ http://www.northumbria.ac.uk/sd/central/ar/lll/lla/ http://www.northumbria.ac.uk/sd/central/ar/lll/lla/ david jonassen and rose marra mindtoolsfor representing knowledge mapping supports a diverse range of learning and comprehension tasks. the hypothesis is therefore a challenging one, though not yet proven. but perhaps i have gone further than mcaleese is claiming. i am certainly happy with his view of auto-monitoring as a 'learning arena' in which the learner can be aware of and manipulate learning activities. references hammond, n.v. (1993), 'learning with hypertext: problems, principles and prospects', in mcknight, c , dillon, a. and richardson, j. (eds), hypertext: a psychological perspective, chichester, ellis horwood. •mckendree, j. and reader, w. (1994). 'the "homeopathic fallacy" in hypertext: misconceptions of psychology in design of computer courseware', alt news (in press). trapp, a.l., reader, w. and hammond, n.v. (1992), 'tools for knowledge mapping: a framework for understanding' in brusilovsky, p. and stefanuk, v. (eds), east-west conference on emerging technologies in education, moscow, april. concept mapping and other formalisms as mindtools for representing knowledge david h. jonassen* and rose m. marra** * pennsylvania state university and **at&t, usa introduction we seek to provide an alternative theoretical perspective on concept mapping (a formalism for representing structural knowledge) to that provided by ray mcaleese in this issue of alt-j (auto-monitoring). we begin with an overview of concept maps as a means of describing a learner's knowledge constructs, and then discuss a broader class of tools, mindtools, of which concept maps are a member. we proceed by defining mindtools as formalisms for representing knowledge, and further elaborate on concept maps as a formalism for representing a particular kind of knowledge: structural knowledge. we then address mcaleese's use of the term auto-monitoring and some of the steps in his model of concept maps. finally, we describe some limitations of concept mapping as a formalism and as a cognitive learning strategy. concept maps, both in how they are used and theoretically based, touch some of the central elements of instructional technology and learning. concept maps address the critical issue of how we organize our knowledge structures within and between domains. they, in some sense, model how we represent and apply knowledge. assuming that instructional technology, as a field, has a goal of being able systematically to make instruction more effective, then tools 50 au-j volume 2 number 1 such as concept maps that allow us to see and understand what the learner is starting with seem critical. concept maps may fall into a larger category of tools, mindtools, that enhance our understanding of how learners organize and use knowledge. mindtools mindtools are computer-based tools and learning environments that have been adapted or developed to support critical thinking in the form of knowledge representation (jonassen, in press). these tools include (but are not necessarily limited to) databases, microworlds and learning environments, spreadsheets, semantic networks and concept maps, expert systems, multimedia construction, computer-mediated communication, and programming languages. mindtools, otherwise known as cognitive tools (kommers et al 1992), or 'content-free courseware' as mcaleese refers to them, are generalizable computer tools that are intended to engage and facilitate cognitive and meta-cognitive processing hence cognitive tools. derry (1990) has defined cognitive tools as both mental and computational devices that support, guide, and extend the thinking processes of their users. they are knowledge-construction and facilitation tools that can be applied to a variety of subject-matter domains. we believe that students cannot use these tools without thinking deeply about the content that they are learning. cognitive tools are an example of learning with information processing technologies as opposed to learning from them (salomon et al 1991). learning with technologies amplifies the learner's cognitive processes while using those technologies. computer-based cognitive tools and environments are, in effect, cognitive amplification tools. environments that employ cognitive tools distribute cognition to the person plus their 'surround' the physical and social resources available to them (perkins 1993). this is perhaps the most powerful rationale for mindtools. if we consider the learner and the computer as parts of a total learning system, our goal should be to distribute the cognitive responsibility appropriately to the different parts of the system. rather than requiring learners to store and retrieve information, which is something that computers do much better, relegate those responsibilities to the computer and require the learners to identify patterns, develop organizational schemes, and synthesize information, which are things that humans do much better than computers. mindtools more appropriately distribute the cognitive responsibilities to the appropriate parts of the learning system. mindtools are not intended necessarily to reduce information processing, that is, make learning easier, as has been the goal of instructional design and most instructional technologies, but rather to engage learners in more effective processes of learning. nor are they 'fingertip' tools (perkins 1993) that learners use naturally, effortlessly, and effectively. rather, mindtools provide an environment and vehicle that often require learners to think harder about the subject-matter domain being studied while generating thoughts that would be impossible without the tool. they are cognitive reflection and amplification tools that help learners to construct their own realities using the constructs and processes in the environment on a new content domain. mcaleese's description of a learner engaged in a 'cognitively active process in an environment that supports the manipulation of knowledge artefacts' could in fact be described as a learner who is engaged with some sort of a mindtool in this case, one that supports concept mapping. the tool allows the learner to represent and explicitly manipulate knowledge formalisms and thus 'amplifies' or makes concrete what would otherwise be abstract and difficult to grasp. 51 david jonassen and rose marra mindtools for representing knowledge mindtools as representational formalisms mindtools consist of different formalisms for representing knowledge; that is, they use different schemes that can be used to represent content or personal knowledge. expository text is the most common formalism that we use to capture and represent what we know or have learned. however, expository prose constrains the learner's representation. writing is a complex activity whose syntax and multiple text-structures entail a great deal of cognitive overhead. mindtools provide different, more constrained formalisms for representing personal knowledge. mindtools, like text, require that learners represent domain knowledge in a knowledge base with formal, constrained statements of relationships between the ideas in the knowledge base. however, the relationships and fundamental structures that are generated by each formalism vary. there are many classes of mindtools. each class engages different critical thinking skills (jonassen, in press) and involves different syntax and representational formalisms. since this issue focuses on concept mapping, we shall next treat it in greater depth. mapping structural knowledge via concept maps concept mapping is a formalism for representing structural knowledge (jonassen et al 1993). structural knowledge is the knowledge of how concepts within a domain are interrelated (diekhoff 1983). if schemas are defined by their interrelationships to other schemas, then explicit awareness of those interrelationships and the ability to explicate those relationships are essential for higher-order procedural knowledge. declarative knowledge is not a sufficient prerequisite for procedural knowledge. in order to acquire procedural knowledge, it is necessary to understand the propositional relationships between the entities involved in both the procedural and declarative knowledge structures. structural knowledge provides that link between procedural and declarative knowledge. structural knowledge, the understanding of these relationships, enables learners to form the connections they need in order to use scripts or complex schemas: that is, the ability actually to apply what they know. structural knowledge has been conceived and described by several different authors (preece 1976; tennyson and cocciarella 1986), but the underlying assumption of all descriptions of structural knowledge is that meaning for any concept or construct is implicit in the pattern of relationships to other concept or constructs. concept mapping, whether accomplished with paper and pencil or using sophisticated semantic networking tools such as semnet, learning tool, textvision, or cmap, are means for representing cognitive structures (jonassen 1987), ideas and dieir complex interrelationships (schemata) in memory that provide meaning to the language used to transmit those ideas. schema theory and semantic networking theory are the theoretical bases for concept mapping. these theories are based on schemas, which are arranged in networks (schemata) of interrelated concepts. these networks are known as semantic networks. perhaps the best-known conceptualization of a semantic network is active structural networks (quillian 1968), structures that are composed of nodes (the equivalent of schemas) and ordered, typed relationships or links connecting them. the nodes are token instances of concepts or propositions, and the links described the propositional relationship between them. as such, concept maps and semantic networks describe a person's knowledge of the interrelationships between ideas 52 acr-j volume 2 number 1 (jonassen et al 1993). in addition to the tools mentioned above (semnet, etc.) that allow learners to create their own concept maps, there are other tools designed to create concept maps from similarity ratings between concept pairs in a particular domain. these raw similarity ratings from the learners are algorithmically transformed (via an algorithm such as pathfinder net schvaneveldt et al 1985) into a concept-map form of knowledge representation. this representation can be evaluated via some criteria, such as whether it can accurately classify members of dissimilar classes (for example, expert and novice). pathfinder nets is another concept-mapping tool used to represent structural knowledge. response to mcaleese: auto-monitoring and internal negotiations so far, our commentary has centred on our view of concept'maps and how they fit with the concept of mindtools and structural knowledge. this section discusses a couple of points from mcaleese's concept-mapping rationale. we shall start by looking at the idea of auto-monitoring and how it relates to some previously addressed ideas, and then address mcaleese's stages of concept modelling from a more constructivistic perspective. structural knowledge, meta-cognition and auto-monitoring perhaps the difference between these terms is just a matter of semantics, but even if that is the case, it is worth discussing, since meaning (semantics) is the essence of the'topic we are discussing. we consider mcaleese's use of the term auto-monitoring. we propose that there is very little difference between mcaleese's term and other ideas namely, what we have previously described as structural knowledge and what is universally known as meta-cognition. the rationale for our concern is a growing need for parsimony in our communications. mcaleese defines auto-monitoring as a technique 'of personal or group knowledge presentation and re-presentation', and further indicates that these techniques are often supported by various computer applications that graphically represent knowledge structures. he further acknowledges that the 'main problem' with auto-monitoring is its universality; that is, mcaleese himself admits the influence of the concepts of meta-cognition and reflective practice on auto-monitoring. we agree with these influences, and would prefer to say that the distinction between these concepts and auto-monitoring is perhaps only in the application of the term auto-monitoring tq the meta-cognitive process that occurs as a learner self-negotiates while creating a concept map. mcaleese argues that auto-monitoring is distinct from meta-cognition because auto-monitoring not only is aware of the cognitive processes, but also is in charge of creating them. we reply that without the awareness of the creation process and the knowledge structures (declarative and procedural) on which the creation is based, the creation itself would be impossible. in other words, the meta-cognitive controls that allow learners to look inward and understand their own knowledge structures is a necessary step for the subsequent instantiation of these structures into a cognitive map. this is supported by the common distinction in the metacognition literature between executive control processes and comprehension monitoring. further, we would argue that what learners must actually be aware of is not only their declarative and procedural knowledge structures, but the structural knowledge structures that link and provide the meaning necessary to apply the previous two. 53 david jonassen and rose marra mindtools for representing knowledge commitment, checking and confirming: a constructivistic perspective mcaleese describes a series of stages in which learners engage when creating a concept map. once concepts and nodes have been created, the learner engages in a process of linking nodes and describing the relationship between nodes, and then a re-scrutinization of those nodes and links (checking) to make sure they agree with their representations and resolving any conflicts discovered in labelling links (resolution). from a constructivist perspective, this process of learners attempting to resolve their own conflicts about how to represent and label knowledge is referred to as internal negotiation (jonassen et al, in press). just as meaning is derived and socially negotiated between individuals from a constructivist perspective, so individuals must do the same with themselves. we engage in this internal negotiation activity almost constantly, and often without explicitly realizing it as we process new stimuli and link them to existing concepts and knowledge structures. perhaps the greatest strength of concept-mapping tools is that they make learners deal explicitly and consciously with what is normally an implicit activity. and further, explicitly instantiating this normally implicit/internal process may be why it is difficult for some learners to commit to a particular concept map. (tools such as pathfinder nets that create a concept map from similarity ratings avoid some of the difficulty of the learner having to sort out all of the relationships for themselves). mcaleese describes the confirming stage as where the learner 'agrees' that the representation is what was intended and further elaborates on the significance of the 'commitment' process. we wish to agree with mcaleese's position that total commitment to a learner's representation may not occur due to cognitive dissonance and the tension between describing a concept relationship in one of several similar fashions. further, we emphasize that it seems very likely that a map a learner creates in one sitting (and thought they were 'finished' with), may look very incomplete or even incorrect when revisited. this seems perfectly reasonable, given the fluid nature of an individual's knowledge structures. in different contexts or at different times, an individual's representation will be very different. once again, from a constructivistic perspective, existing knowledge structures are continually being modified as we sense and process and meld new ideas to existing schemas and schemata. if this is the case, then a learner's conception of relationships between various concepts will undoubtedly change, thus making total commitment to a concept map over a long period of time seem rather unlikely. like mcaleese we would agree this is a good thing in that if learners (and instructors) are aware that their views of the world may legitimately change and grow, then more may be gained by revisiting previous concept maps. limitations of concept maps and semantic networks it is easy to become enamoured of representational tools like concept mapping and semantic networking. they are powerful tools, but they are easily attributed too much power. we now discuss a few cautionary notes. reification of representations semantic network theory is an example of a symbolic processing model of cognition. this traditional view of learning assumes that human memory consists of stored facts and procedures. clancy (1992) challenges the 'identity hypothesis' of these symbolic representation models, claiming that semantic-network theorists see nets as isomorphic representations of what is stored in memory. that is, memory structures (and by inference, structural knowledge) 54 alt-j volume 2 number 1 literally exist as representations in the mind that they represent not only behaviour but also internal representational structures and mental processes. concept maps and semantic networks can be too readily thought to reify the structures of the mind. this implies that the semantic store of information in humans can be cognitively mapped and literally searched just as a computer searches its memory stores. these assumptions impute far too much reality to concept maps. they are not truly maps of the mind, but rather representations of what we think is in the mind. in truth, they are not even really good at that for the following reasons. ephemerality, dynamics, and context dependence of semantic networks connectionist models, the alternatives to symbolic representation models, believe that memory is distributed in different memory locations. they see structural knowledge as neural networks that encode memories in multiple locations that memory is not static as symbolic representations assume. memory changes according to the context in which the events occurred or the context in which ideas are being recalled. structural knowledge is dynamic. what is recalled depends on the context, the need, and a number of other variables. structural knowledge is therefore ephemeral. it is not the same over time. yet many people take concepts maps as literal, static, and immutable representations of what is known. concept mapping tools would need to enable minute-by-minute, context-by-context changes in the concepts, relationships and structures that are represented in it in order more accurately to represent knowledge. complexity of internal semantic networks and simplicity of their representations the prepositional networks in the mind, in whatever form they really exist, are far more complex than anything that can be represented in a concept map. conceptually, all the constructs that we know are interrelated and multiply, encoded in rich, very redundant networks of ideas. concepts are not just linked to one another in one way. two concepts may be linked in multiple ways. these networks are multidimensional, yet n-dimensionality is a concept that is extremely difficult to grasp and even harder to represent in two or threedimensional space. at best, concept maps are distillations of what we know. conclusion in this short contribution, we have argued that: • the main strength of concept maps is making explicit what is normally implicit; • we agree with much of what mcaleese proposes but do not see auto-monitoring as different from rheta-cognition and structural knowledge; • from a constructivist perspective, the concept map creation process supports the role of internal knowledge negotiation; ' • • concept maps should be interpreted cautiously and used only as an ephemeral knowledge-representation tool for internally and socially negotiating the knowledge construction process. 55 david jonassen and rose marra mindtools for representing knowledge references clancy, w.j. (1992). 'representations of knowing: in defense of cognitive apprenticeship.' journal of artifical intelligence in education, 3 139-168. derry, s.j. (1990), 'flexible cognitive tools for problem solving instruction', paper presented at the annual meeting of the american educational research association, boston ma, april 16-20. diekhoff, g.m. (1983), 'relationship judgments in the evaluation of structural understanding', journal of educational psychology, 75, 227-233. jonassen, d.h., beissner, k. and yacci, m.a. (1993), structural knowledge: techniques for conveying, assessing, and acquiring structural knowledge, hillsdale nj, lawrence erlbaum. jonassen, d.h., davidson, m.e., campbell, j.r, haag, b.b. and hettinger, g.a. (in press), 'evaluating learning from constructivist learning environments', training research journal. jonassen, d.h. (1987), 'assessing cognitive structure: verifying a method using pattern notes', journal of research and development in education, 20 (3), 1-14. jonassen, d.h. (in press), mindtools for schools, new york, macmillan. kommers, p.a.m., jonassen, d.h., & mayes, t. (1992), cognitive tools for learning, berlin, springer verlag. perkins, d.n. (1993), 'person plus: a distributed view of thinking and learning' in salomon, g. (ed), distributed cognitions. cambridge university press. preece, p.f.w. (1976), 'mapping cognitive structure: a comparison of methods', journal of educational psychology, 68, 1-8. quillian, m.r. (1968), 'semantic memory' in minsky, m. (ed), semantic information processing, cambridge ma, mit press. salomon, g., perkins, d.n. and globerson, t. (1991), 'partners in cognition: extending human intelligence with intelligent technologies', educational researcher, 20, 3, 2-9. schvaneveldt, r.w., durso, f.t., goldsmith, t.e., breen, t.j., cooke, n.m., tucker, r.g. and demaio, j.c. (1985), 'measuring the structure of expertise, international journal of manmachine studies, 23, 699-728. tennyson, r.d. and cocciarella, m.j. (1986), 'an empirically based instructional design theory for teaching concepts', review of educational research, 56, 40-71. 56 (page number not for citation purpose) 1 *corresponding author. email: rhechanova@ateneo.edu research in learning technology 2023. © 2023 m.r.m. hechanova et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 research in learning technology vol. 31, 2023 original research article developing resilience online: evaluation of synchronous and asynchronous resilience interventions for filipino college students maria regina m. hechanovaa* , rosanne m. jocsonb , arsenio s. alianana, junix jerald i. delos santosa,c , jason o. manaoisa,d, gilda a. gomeze and gina r. lamzonf adepartment of psychology, ateneo de manila university, quezon city, philippines; bnational institute of education, nanyang technological university, singapore; cschool of teacher education and liberal arts, university of baguio, baguio city, philippines; ddepartment of psychology, xavier university, cagayan de oro city, philippines; edepartment of psychology, ateneo de naga university, naga city, philippines; fdepartment of psychology, ateneo de davao university, davao city, philippines (received: 11 september 2022; accepted: 12 november 2022; published: 24 february 2023) this study evaluated two forms of a resilience intervention amongst college students during the covid-19 pandemic. utilising a randomised controlled trial design, it examined the impact of a synchronous and asynchronous resilience interventions versus a control group that did a journaling intervention. outcomes measured included coping behaviour, non-reactivity, wellbeing, stress, depression and anxiety. participants consisted of filipino college students randomly assigned to three groups: synchronous online resilience group (n = 135), asynchronous resilience group (n = 121) and control group (n = 127). results revealed that students who went through the online synchronous resilience reported a significant reduction in depression at post-intervention compared to those who went through an asynchronous intervention. post-intervention scores for nonreactivity were also higher in the synchronous group compared to both asynchronous and journaling groups. effect sizes were small to moderate. this study suggests that online resilience interventions are viable means to address the mental health needs of students, especially in countries with limited mental health resources. keywords: resilience; mental health; mhpss; pandemic; well-being; depression; philippines the covid-19 pandemic has brought about a plethora of mental health outcomes. issues of physical safety coupled with the lack of social support due to physical distancing regulations gave rise to emotional distress, fear, helplessness, anxiety, grief and sorrow (soklaridis et al. 2020). beyond social distancing and isolation, fear of infection and death led to anger, confusion and post-traumatic distress (mukhtar 2020). this research was supported by a grant from the university research council of the ateneo de manila university. mailto:rhechanova@ateneo.edu http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2890 https://orcid.org/0000-0002-3922-1340 https://orcid.org/0000-0003-4220-0695 https://orcid.org/0000-0003-0452-3987 m.r.m. hechanova et al. 2 citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 (page number not for citation purpose) although many groups have been affected, the youth appear to be particularly at risk. prolonged school closure, isolation from peers and economic difficulties appear to have led to disturbances in sleep and exercise routines and have led to excessive use of technology. a meta-analysis of 23 studies amongst the youth revealed that 29% reported depressive symptoms, 26% reported anxiety symptoms, 44% had sleep disorders and 48% reported post-traumatic stress symptoms (ma et al. 2021). similarly, a study on the impact of the covid-19 pandemic amongst filipino youth reported greater symptoms of depression, anxiety and stress and lower well-being compared to older respondents (hechanova, valentin, and alampay 2022). the authors suggested that this can be attributed to a still-developing brain and the youth’s limited ability to communicate their feelings, lifestyle and inadequate coping strategies. imran, zeshan and pervaiz (2020) suggest the need to improve the ability of the young to express their fears and concerns and need to encourage helpful routines and physical activities. the past decade has seen a rise in tele-mental health interventions. there is evidence that tele-mental health services improve access and provide flexibility in interactions between clients and their service providers. tele-mental health also reduces the cost of services and, as such, is advantageous for countries with limited access to mental health services. unfortunately, much of the evidence on tele-mental health services have come from developed economies, and there is a dearth of research from non-western and developing economies such as the philippines. in addition, although there have been studies comparing different modalities of tele-mental health interventions, they have mostly utilised adult samples. this study seeks to fill these gaps by examining the impact of online resilience interventions for filipino young adults. resilience and tele-mental health interventions luthar and cicchetti (2000) define resilience has been defined as ‘the ability of individuals to display positive adaptation despite the experiences of significant adversity or trauma’. rather than merely being a personal trait, they describe resilience as a social competence that can be learned. this perspective has given rise to various resilience interventions. a meta-analysis of resilience interventions suggests that they generally consist of cognitive behavioural therapy, mindfulness and a combination thereof (joyce et al. 2018). although a vast majority of resilience interventions are delivered via face-to-face, innovative approaches such as bibliotherapy, online webinars or phone coaching are also being used (joyce et al. 2018). a meta-analysis of 64 trials examining internet-delivered cognitive behavioural therapy modules or lessons reported positive benefits for those with major depression, panic disorder, social anxiety disorder or generalised anxiety disorder (andrews et al. 2010). the covid-19 pandemic increased the provision of tele-mental health services by paraprofessionals. in china, nurses conducted 10-min wechat consultations twice daily to covid-19 patients with elevated anxiety and depression scores. results revealed improvements in anxiety and depression of patients (zhou et  al. 2020). another study found improvements in sleep and anxiety of covid-19 patients who were provided progressive muscle relaxation intervention audio (liu et  al. 2020). a similar study provided audio-recordings on relaxation, mindfulness and bilateral stimulation through patients’ mobile phones every day for 2 weeks. those who went through the intervention reported reduced anxiety symptoms compared to those who did not (wei et al. 2020). tele-mental health interventions were also delivered to health http://dx.doi.org/10.25304/rlt.v31.2890 research in learning technology citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 3 (page number not for citation purpose) workers in self-isolation in italy (giordano et al. 2020). a music therapy intervention consisting of a playlist that focused on breathing, energy and serenity was delivered using mobile phones. results reported reduced levels of tiredness, sadness, fear and worry amongst healthcare workers. beyond patients and health workers, the increased mental health concerns of adolescents also gave rise to the delivery of mental health support using educational platforms. for example, a brief acceptance commitment therapy psychoeducation was delivered via videoconferencing (zoom) to undergraduates in the united states. the four-session intervention was delivered as part of a psychology seminar course. preand posttests showed that stress and anxiety decreased at follow-up. students reported the intervention as helpful and accessible and reported moderate engagement with skills outside of class (browning et al. 2022). there have been a few studies comparing synchronous and asynchronous interventions with mixed findings. aller et al. (2022) compared the effectiveness of mental health literacy delivered as an online asynchronous course and as in-person workshop compared to a control group. this quasi-experimental study included 1049 undergraduate students who self-selected enrolment in a mental health awareness and advocacy (mhaa) course (treatment; n = 474) or a general lifespan development course (comparison; n = 575). students in the treatment group significantly increased their mental health literacy knowledge and self-efficacy compared to the control group. treatment effects did not differ across modalities, suggesting that students’ mental health literacy improved regardless of whether it was developed synchronously or asynchronously. on the other hand, a study compared synchronous online community mental health groups with asynchronous forum-based online mental health groups. the study reported that synchronous groups were more cohesive than asynchronous groups (li, kraut, and zhu 2021). the majority of the studies on tele-mental health have emanated from developed economies, and there is a dearth of studies from low resource countries (constantine 2002). as such, this study seeks to contribute to the literature by evaluating a synchronous and asynchronous delivery of an online resilience intervention created for filipinos. katatagan resilience intervention this study examines a resilience intervention named katatagan,1 which was designed to develop resilience skills using principles and strategies from positive psychology, cognitive behaviour therapy and mindfulness. it consists of six modules with each module developing a resilience skill. module 1 aims to encourage participants’ self-efficacy and address their sense of helplessness by asking them to reflect on their internal and external sources of strength. module 2 seeks to help participants manage their somatic reactions using techniques such as progressive muscle relaxation, bioenergetic exercises and emotional freedom technique. module 3 aims to help participants deal with their negative thoughts and emotions using cognitive reframing and thought substitution strategies. module 4 focuses on routines and activities that can improve mood. module 5 seeks to build problem-solving skills and encourage participants to seek social support in solving their problems. module 6 focuses on encouraging meaning making and invites participants to set goals for the future. mindfulness was developed throughout the program by introducing it as a centring exercise at the 1filipino term meaning resilience. http://dx.doi.org/10.25304/rlt.v31.2890 m.r.m. hechanova et al. 4 citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 (page number not for citation purpose) beginning of each module. given the collective nature of filipinos, the intervention was implemented in small groups (hechanova et al. 2015). the katatagan program was first implemented amongst college student survivors of supertyphoon haiyan in 2014. the pilot-test found students who participated in the program reported less anxiety and depressive symptoms and improved coping behaviours (hechanova et  al. 2015). another longitudinal study reported increased self-efficacy on resilience skills 6 months after the program (hechanova, waelde, and ramos 2016). the program was also implemented amongst displaced survivors, and participants reported decreased anxiety and increased resilience scores compared to non-participants (hechanova et al. 2018). during the covid-19 pandemic, katatagan was adapted for online delivery in two modalities: synchronous resilience classes and asynchronous self-paced classes. this study sought to evaluate the outcomes of these two versions of the resilience program compared to a control group. specifically, we asked: (1) to what extent would the synchronous and asynchronous resilience intervention improve psychological well-being and adaptive coping skills compared to a control group? (2) to what extent would the synchronous and asynchronous intervention decrease stress, anxiety and depression symptoms of participants compared to a control group? (3) would there be differences in the impact of synchronous and asynchronous resilience interventions? methods this study utilised a randomised controlled trial design comparing three groups – a synchronous resilience group, an asynchronous resilience group and a control group that did journal writing. sample participants were recruited through convenience sampling via social media using the following inclusion criteria: students currently enrolled in any college or university in the philippines; aged 18 to 25 years old; do not have any diagnosed or underlying mental health condition; have not attended any previous katatagan online program; and have adequate internet connection. figure 1 summarises the participant flow from enrolment to data analysis. a total of 482 participants were assessed for eligibility to participate in the study; however, 99 participants were excluded before randomisation as they declined to participate (n = 95) and decided to withdraw from the study (n = 4). of the 383 participants, 135 were allocated to the synchronous resilience classes (i.e. condition 1), 121 were assigned to the asynchronous self-paced classes (i.e. condition 2) and 127 participants were allocated to the journal writing activity (i.e. condition 3). participants were between the ages of 18 and 25 years old (m = 20.85, standard deviation [sd] = 1.27), with the majority of them being female (78.6%). other baseline characteristics of participants across three conditions can be found in table 1. http://dx.doi.org/10.25304/rlt.v31.2890 research in learning technology citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 5 (page number not for citation purpose) figure 1. consort 2010 flow diagram for ko rct. intervention the synchronous resilience intervention was run in small groups (of no more than  12  participants). the sessions were conducted with two facilitators and ran for 1.5 to 2 h depending on the size of the group. the intervention ran for 6 weeks with one module per week. facilitators consisted of trained paraprofessionals with behavioural science backgrounds and previous experience in facilitating group interventions. facilitators were also trained on the use of zoom and ethics in the delivery of tele-mental health services. in the asynchronous version of the program, participants enrolled in an online course on canvas. the online course consisted of the six modules. each module featured videos, reflection questions and a facilitated discussion board. participants were given 6 weeks to finish the online course. the control group was given journaling exercise consisting of six weekly prompts sent via email. the self-help prompts instructed them to write about the highlights of their week and their reactions to these. measures psychological well-being was assessed through who-5 well-being index (world health organization 1998), a 5-item survey. a 6-point likert scale was used (0 = at no http://dx.doi.org/10.25304/rlt.v31.2890 m.r.m. hechanova et al. 6 citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 (page number not for citation purpose) time, 5 = all of the time). sample items are ‘i have felt cheerful and in good spirits’ and ‘i have felt calm and relaxed’. reliability of the scale was 0.92. stress, anxiety and depression were measured using the dass-21 scale (antony et al. 1998). the 21-item scales utilised a 4-point likert scale (0 = did not apply to me at all; 3 = applied to me very much, or most of the time) and includes items measuring depression (‘i felt i had nothing to look forward to’), anxiety (‘i experienced difficulty breathing’) and stress (‘i found it difficult to relax’). reliabilities of the subscales were 0.88 (depression), 0.82 (anxiety) and 0.86 (stress). non-reactivity was measured using the five factor mindfulness questionnaire – nonreactivity subscale (baer et al. 2006). the 7-item survey uses a 5-point likert scale rated from 1 (never or very rarely true) to 5 (very often or always true). the internal reliability of the scale was 0.86. adaptive coping was assessed using an 18-item abridged version of carver’s (1997) brief cope scale. it included two items each from the following domains: active coping (α = 0.67), use of emotional support (α = 0.77), use of instrumental support (α  =  0.81), planning (α = 0.68), positive reframing (α = 0.72), religious coping (α = 0.84) and behavioural disengagement (α = 0.77). the measure utilised a 4-point likert scale (1 = i haven’t been doing this at all, 4 = i’ve been doing this a lot). sample items are ‘i have been getting emotional support from others’ and ‘i’ve been concentrating my efforts on doing something about the situation i’m in’. table 1. baseline characteristics of participants in the three conditions. demographic variable synchronous (n = 135) asynchronous (n = 121) control/journaling (n = 127) total (n = 383) female, n (%) 107 (79.3%) 95 (78.5%) 99 (78.0%) 301 (78.6%) age, m (sd) 20.95 (1.34) 20.73 (1.19) 20.85 (1.25) 20.85 (1.27) region, n (%) ncr 21 (15.6%) 17 (14.0%) 24 (18.9%) 62 (16.2%) car 8 (5.9%) 2 (1.7%) 6 (4.7%) 16 (4.2%) region i 14 (10.4%) 12 (9.9%) 14 (11.0%) 40 (10.4%) region ii 0 (0.0%) 0 (0.0%) 3 (2.4%) 3 (0.8%) region iii 28 (20.7%) 18 (14.9%) 12 (9.4%) 58 (15.1%) region iva 12 (8.9%) 10 (8.3%) 11 (8.7%) 33 (8.6%) region v 3 (2.2%) 5 (4.1%) 3 (2.4%) 11 (2.9%) region vi 2 (1.5%) 2 (1.7%) 3 (2.4%) 7 (1.8%) region vii 1 (0.7%) 6 (5.0%) 0 (0.0%) 7 (1.8%) region viii 2 (1.5%) 0 (0.0%) 2 (1.6%) 4 (1.0%) region ix 2 (1.5%) 3 (2.5%) 5 (3.9%) 10 (2.6%) region x 40 (29.6%) 44 (36.4%) 39 (30.7%) 123 (32.1%) region xi 1 (0.7%) 0 (0.0%) 2 (1.6%) 3 (0.8%) region xii 0 (0.0%) 1 (0.8%) 0 (0.0%) 1 (0.3%) region xiii 1 (0.7%) 0 (0.0%) 2 (1.6%) 3 (0.8%) barmm 0 (0.0%) 1 (0.8%) 1 (0.8%) 2 (0.5%) note. sd = standard deviation; ncr = national capital region; car = cordillera administrative region; barmm = bangsamoro autonomous region in muslim mindanao. http://dx.doi.org/10.25304/rlt.v31.2890 research in learning technology citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 7 (page number not for citation purpose) procedures and ethics ethics approval was obtained from the ateneo de manila university institutional research ethics board. publicity material to invite potential participants to join the research was published through the official social media account of katatagan online. research participants were recruited from the pool of individuals who voluntarily enrolled in the katatagan online program from september 1 to october 12, 2022. the registration link to the study was provided in the recruitment poster, where potential participants read the online informed consent form and expressed their consent digitally by clicking a button on google forms. those who did not wish to participate also clicked on a button, signifying that they do not wish to participate in the study. consenting participants were directed to answer the pretest online questionnaire. an assigned officer from the katatagan online program team who is external to the research team randomly assigned participants to one of three conditions: (1) synchronous resilience classes, (2) asynchronous self-paced classes and (3) journal writing activity. randomisation was done using a simple randomisation technique that followed a 2-1-3 sequence to the list of participants who signed up for the study. after enrolling in the program, participants were provided the informed consent form, and those who consented to be part of the study were led to the online pretest survey. all the three groups were given 6 weeks after which participants were invited to answer a posttest survey. confidentiality and privacy were ensured through the removal of identifying information and the use of password protected computer files. data analysis at posttest, 37% of the initial respondents had complete data, with a large proportion not responding after being randomised to one of the three conditions (figure 1). attrition analyses showed that participants who had complete data were slightly older, had lower baseline depressive symptoms and had higher baseline nonreactivity and active coping scores compared to those who did not complete the posttest. to reduce bias due to the high attrition rate, the analyses followed an intention-to-treat approach in which all participants who were randomised were analysed in their allocated group regardless of program attendance or completion. missing data were handled using full information maximum likelihood in mplus 8 (muthén and muthén 1998–2017). data analysis was conducted by a statistician blind to participants’ allocation to the three conditions. first, intraclass correlation coefficients (iccs) and design effect estimates were computed to determine the need to conduct a nested analysis that would account for participant groupings (facilitators and school). linear regression analyses were conducted to examine intervention effects on posttest scores controlling for baseline scores using maximum likelihood estimation with robust standard errors (mlr) to account for skewed variables. two sets of analyses were conducted: one comparing post outcomes across conditions with the journaling condition as the reference group, and one comparing outcomes with the synchronous condition as the reference group. analyses used type=complex and cluster to correct for non-independence of observations within schools. effect sizes were estimated to compare posttest scores across groups. finally, sensitivity analyses were conducted using complete cases to determine the robustness of the results to the missing data handling strategy. http://dx.doi.org/10.25304/rlt.v31.2890 m.r.m. hechanova et al. 8 citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 (page number not for citation purpose) results because participants came from different schools and were assigned to different facilitators, design effect estimates were computed to determine if a nested analysis is necessary. researchers note that design effect estimates greater than 2 indicate the need for multilevel models (muthen and satorra 1995; peugh 2010). at the facilitator level, none of the design effects was greater than 2. at the school level, the design effect was table 2. preand postscores on study outcomes by condition. outcome synchronous (n = 135) asynchronous (n = 121) control/journaling (n = 127) m sd m sd m sd psychological well-being pre 13.09 5.44 12.55 5.97 13.55 5.75 post 16.79 5.92 14.50 6.63 15.23 5.77 depression pre 8.14 4.99 7.88 4.84 6.93 4.74 post 5.05 4.15 6.53 4.77 6.07 4.64 anxiety pre 7.76 4.64 7.30 4.44 6.72 4.52 post 6.34 5.05 6.91 4.39 6.96 5.05 stress pre 9.19 4.45 8.61 4.91 7.90 4.58 post 6.64 4.55 7.66 4.12 7.98 4.68 nonreactivity pre 20.82 5.75 20.57 6.05 21.30 5.79 post 24.73 6.19 22.50 5.43 22.72 4.76 instrumental support seeking pre 1.15 0.91 1.14 0.89 1.06 0.94 post 1.53 0.88 1.36 0.90 1.41 0.96 emotional support seeking pre 1.81 0.82 1.75 0.81 1.79 0.86 post 2.19 0.77 2.12 0.74 2.07 0.80 active coping pre 2.10 0.73 2.00 0.75 2.01 0.72 post 2.35 0.56 2.42 0.52 2.32 0.68 planning pre 2.15 0.70 2.14 0.70 2.15 0.70 post 2.37 0.66 2.39 0.59 2.24 0.62 behavioural disengagement pre 0.94 0.87 0.71 0.75 0.68 0.78 post 0.63 0.72 0.66 0.63 0.76 0.89 positive reframing pre 1.94 0.76 2.06 0.79 1.95 0.74 post 2.31 0.63 2.37 0.57 2.11 0.75 religious coping pre 1.52 1.00 1.72 1.00 1.62 0.96 post 1.70 0.93 2.08 0.93 1.74 0.93 sd = standard deviation. http://dx.doi.org/10.25304/rlt.v31.2890 research in learning technology citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 9 (page number not for citation purpose) t ab le 3 . l in ea r re gr es si o n a n d e ff ec t si ze e st im at es c o m p ar in g p o st te st s co re s ac ro ss c o n d it io n s. o u tc o m e s yn ch ro n o u s vs . jo u rn al in g a sy n ch ro n o u s vs . jo u rn al in g s yn ch ro n o u s vs . as yn ch ro n o u s b s e β d [9 5% c i] b s e β d [9 5% c i] b s e β d [9 5% c i] p sy ch o lo gi ca l w el lb ei n g 1. 55 0. 68 0. 12 0. 24 [ 0. 01 , 0. 49 ] − 0. 62 1. 34 − 0. 05 − 0. 10 [ − 0. 35 , 0. 15 ] 2. 17 1. 24 0. 17 0. 34 [ 0. 10 , 0. 59 ] d ep re ss io n − 1. 28 0. 71 − 0. 14 − 0. 28 [ − 0. 53 , − 0. 04 ] 0. 68 0. 78 0. 07 0. 14 [ − 0. 11 , 0. 39 ] − 1. 96 ** 0. 75 − 0. 20 − 0. 41 [ − 0. 66 , − 0. 16 ] a n xi et y − 0. 79 0. 77 − 0. 08 − 0. 16 [ − 0. 40 , 0. 08 ] 0. 13 0. 90 0. 01 0. 02 [ − 0. 23 , 0. 27 ] − 0. 92 0. 89 − 0. 09 − 0. 18 [ − 0. 43 , 0. 06 ] s tr es s − 1. 44 0. 81 − 0. 15 − 0. 30 [ − 0. 55 , − 0. 06 ] − 0. 04 0. 90 − 0. 00 − 0. 01 [ − 0. 26 , 0. 24 ] − 1. 40 0. 85 − 0. 14 0. 28 [ − 0. 53 , − 0. 04 ] n o n re ac ti vi ty 1. 88 * 0. 96 0. 16 0. 33 [ 0. 08 , 0. 57 ] − 0. 71 0. 99 − 0. 06 − 0. 12 [ − 0. 37 , 0. 13 ] 2. 59 * 1. 07 0. 22 0. 45 [ 0. 29 , 0. 70 ] in st ru m en ta l su p p o rt u se 0. 09 0. 11 0. 07 0. 14 [ − 0. 10 , 0. 38 ] − 0. 11 0. 17 − 0. 06 − 0. 12 [ − 0. 37 , 0. 13 ] − 0. 01 0. 11 − 0. 01 − 0. 02 [ − 0. 27 , 0. 23 ] e m o ti o n al s u p p o rt u se 0. 10 0. 14 0. 06 0. 12 [ − 0. 12 , 0. 36 ] 0. 05 0. 16 0. 03 0. 06 [− 0. 19 , 0. 31 ] 0. 05 0. 15 0. 03 0. 06 [ − 0. 19 , 0. 31 ] a ct iv e co p in g − 0. 06 0. 11 − 0. 04 − 0. 08 [ − 0. 32 , 0. 16 ] 0. 07 0. 11 0. 05 0. 10 [ − 0. 15 , 0. 35 ] − 0. 12 0. 11 − 0. 09 − 0. 18 [ − 0. 43 , 0. 06 ] p la n n in g 0. 11 0. 11 0. 08 0. 16 [ − 0. 08 , 0. 40 ] 0. 12 0. 12 0. 09 0. 18 [ − 0. 07 , 0. 43 ] − 0. 01 0. 12 − 0. 01 − 0. 02 [ − 0. 26 , 0. 23 ] b eh av io u ra l d is en ga ge m en t1 − 0. 26 0. 14 − 0. 16 0. 33 [ 0. 08 , 0. 57 ] − 0. 12 0. 13 − 0. 07 − 0. 14 [ − 0. 39 , 0. 11 ] − 0. 14 0. 12 − 0. 08 − 0. 16 [ − 0. 41 , 0. 08 ] p o si ti ve r ef ra m in g 0. 21 0. 12 0. 15 0. 30 [ 0. 06 , 0. 55 ] 0. 14 0. 12 0. 10 0. 20 [ − 0. 05 ,0 .4 5] 0. 07 0. 11 0. 05 0. 10 [ − 0. 15 , 0. 35 ] r el ig io u s co p in g − 0. 03 0. 12 − 0. 02 − 0. 04 [ − 0. 28 , 0. 20 ] 0. 13 0. 14 0. 06 0. 12 [ − 0. 13 , 0. 37 ] − 0. 16 0. 15 − 0. 08 0. 16 [ − 0. 41 , 0. 08 ] n ot e. a ll a n al ys es c o n tr o ll ed f o r p re te st s co re s. s d = s ta n d ar d d ev ia ti o n ; s e = s ta n d ar d e rr o r; c i = c o n fi d en ce i n te rv al . 1 a n al ys es u se d t y p e = c o m p l e x a n d c l u s t e r c o m m an d s to c o rr ec t fo r n o n in d ep en d en ce o f o b se rv at io n s w it h in s ch o o ls . *p < 0 .0 5, * * p < 0 .0 1. http://dx.doi.org/10.25304/rlt.v31.2890 m.r.m. hechanova et al. 10 citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 (page number not for citation purpose) 2.05 for behavioural disengagement. we, thus, accounted for school clustering for the analysis of this variable. table 2 presents a summary of pre and post scores on the primary outcomes across the synchronous, asynchronous and journaling conditions. table 3 presents the results of linear regression analyses comparing posttest scores across the three conditions, controlling for pretest scores. results showed a significant difference between the synchronous and asynchronous groups on depression symptoms (table 3). there was also a significant difference between the synchronous and both asynchronous and journaling groups on non-reactivity. specifically, posttest depression scores were significantly lower amongst synchronous participants compared to asynchronous participants, and non-reactivity scores were significantly higher amongst synchronous participants compared to both asynchronous and journaling participants (table 2). in contrast, there were no significant differences between the asynchronous and journaling conditions on any of the outcomes. the effect sizes of the comparisons were small to moderate (table 3). sensitivity analyses using data with complete cases showed similar findings, suggesting that results are robust to the missing data handling strategy. discussion amongst the three groups included in this study, those who participated in the synchronous katatagan online program evidenced better outcomes on depression and non-reactivity than those who participated in the asynchronous and the journaling conditions. this appears to run contrary to aller et  al.’s (2022) study that suggests that both synchronous and asynchronous interventions have the same effects. similarly, an earlier study that used a less robust pretest–posttest design also found small to medium effect sizes on wellbeing and resiliency amongst participants who underwent the asynchronous katatagan online program (hechanova et al. 2022). however, the results are consistent with a meta-analysis of randomized control trial (rcts) involving online resiliency programs which showed similar small effect sizes, and a relatively larger effect size amongst synchronous programs compared to asynchronous ones (ang et  al. 2022). one explanation for why the synchronous programs could have yielded better outcomes is because they are more comparable to face-to-face treatment (andrews et al. 2018) that provides real time conversations (blau, weiser, and eshet-alkalai 2017). there is evidence that social support enables psychological resilience (sassaki et  al. 2019) and synchronous groups provide that. these synchronous groups may have provided the social contact that the youth participants craved, given that many schools were still closed to on-site schooling at the time of this study. the results may also reflect the collectivist nature of filipinos. haque (2010) suggests that those in collectivist cultures derive their resilience from their support groups. this was validated by a study that filipino disaster survivors tend to cite external sources of strength such as family and friends and god (hechanova et  al. 2015). a pilot study on katatagan online reports that talking to people who shared similar experiences, thoughts and emotions was validating for them because they felt that they were not alone (hechanova et al. 2022). in contrast to synchronous groups, those in the asynchronous condition may have felt less connected to their cohorts. this is suggested by li, kraut and zhu (2021) who reported that synchronous online community mental health groups are more cohesive than asynchronous forum-based online mental health groups. the researchers http://dx.doi.org/10.25304/rlt.v31.2890 research in learning technology citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 11 (page number not for citation purpose) suggested that technical features of different online community platforms affect community cohesion, and that speed and evenness of communication are as important as the frequency of communication. examination of descriptive statistics shows an improvement from pretest to posttest across most, if not all, coping sub-sales. however, the changes were not statistically significant. this is contrary to an earlier study on the asynchronous katatagan online program that found a significant increase in seeking emotional and instrumental support and active coping strategies (hechanova et al. 2022). as the synchronous online intervention had a significant impact on the participants’ wellbeing, this finding may suggest that the meetings themselves were helpful in bringing about greater wellbeing and lower depression even with slight changes in coping behaviours. similar to the findings of the asynchronous katatagan online study (hechanova et al. 2022), although the scores on stress and anxiety were slightly lower post-program, the change was not significant. scores in wellbeing were likewise non-significant albeit preand posttest scores increasing numerically across groups. nevertheless, these non-significant results across the different groups may point to the continuing stressors and uncertainty brought about by the pandemic. at the time of this study, many participants were still studying from home as colleges and universities were still closed to on-site classes. limitations and implications for future studies this study utilised a randomised controlled treatment design. however, a limitation was unequal sample sizes. as such, more studies with a similar design using larger samples will provide more robust information on the efficacy and effectiveness of this intervention. results showed significant improvements in non-reactivity and a decrease in depression symptoms. future studies may uncover the specific mechanisms of the intervention that led to these particular outcomes. both the medium of delivery of the program and program contents may need to be further investigated. there is a need for more understanding as to which active ingredients in the program have the most impact. this information would be useful in determining whether abridged versions of the intervention may be delivered without sacrificing desired outcomes. the non-significant findings in coping may be due to a number of factors such as a relatively small sample size. future studies utilising larger samples may enable more robust conclusions. in addition, moderating variables such as preexisting disabilities may be examined. additionally, future studies may consider measuring the role of the facilitators on the overall experience of the participants. the quality of the relationship and the kind of connectedness that facilitators fostered during the intervention might have played a role in the outcomes of this study as well. implications for practice the use of telepsychology practices has become invaluable from a public mental health perspective. this study presents initial evidence of online synchronous intervention in improving wellbeing, depression and non-reactivity of participants. this bodes well particularly for resource-poor economies that have limited budgets and personnel for mental health services. scalable interventions such as these may enable access to those who may need mental health psychosocial support. http://dx.doi.org/10.25304/rlt.v31.2890 m.r.m. hechanova et al. 12 citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 (page number not for citation purpose) references aller, t. b., et al., (2022) ‘a non-randomized, quasi-experimental comparison of effects between an in-person and online delivery of a college mental health literacy curriculum’, prevention science, 23(7):1208–1215. doi: 10.1007/s11121-022-01350-y andrews, g., et al., (2010) ‘computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis’, plos one, vol. 5, no. 10, p. e13196. doi: 10.1371/journal.pone.0013196 andrews, g., et al., (2018) ‘computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis’, journal of anxiety disorders, 55, pp. 70–78, doi: 10.1016/j.janxdis.2018.01.001 ang, w. h. d., et al., (2022) ‘effectiveness of resilience interventions for higher education students: a meta-analysis and meta-regression’, journal of educational psychology, 114(7), 1670–1694. https://doi.org/10.1037/edu0000719 antony, m. m., et  al., (1998) ‘psychometric properties of the 42-item and 21-item versions of the depression anxiety stress scales in clinical groups and a community sample’, psychological assessment, vol. 10, no. 2, p. 176. doi: 10.1037/1040-3590.10.2.176 baer, r., et al., (2006) ‘using self-report assessment methods to explore facets of mindfulness’, assessment, vol. 13. no. 1, pp. 27–45. doi: 10.1177/1073191105283504 blau, i., weiser, o. & eshet-alkalai, y. (2017) ‘how do medium naturalness and personal ity traits shape academic achievement and perceived learning? an experimental study of face-to-face and synchronous e-learning’, research in learning technology, 25,1–23. doi: 10.25304/rlt.v25.1974 browning, m. e., et al., (2022) ‘brief act for undergraduates: a mixed-methods pilot investigation of acceptance and commitment therapy delivered over zoom’, journal of college student psychotherapy. doi: 10.1080/87568225.2022.2029659 carver, c. s. (1997) ‘you want to measure coping but your protocol’s too long: consider the brief cope’, international journal of behavioral medicine, vol. 4, no. 1, pp. 92–100. doi: 10.1207/s15327558ijbm0401_6 constantine, m. g. (2002) ‘predictors of satisfaction with counseling: racial and ethnic minority clients’ attitudes toward counseling and ratings of their counselors’ general and multicultural counseling competence’, journal of counseling psychology, vol. 49, no. 2, pp. 255. doi: 10.1037/0022-0167.49.2.255 giordano, f., et  al., (2020) ‘receptive music therapy to reduce stress and improve wellbeing in italian clinical staff involved in covid-19 pandemic: a preliminary study’, the arts in psychotherapy, vol. 70, p. 101688. doi: 10.1016/j.aip.2020.101688 haque, a. (2010) ‘mental health concepts in southeast asia: diagnostic considerations and treatment implications’, psychology, health and medicine, vol. 15, pp. 127–134. doi: 10.1080/13548501003615266 hechanova, m. r. m., et al., (2015) ‘the development and initial evaluation of katatagan: a resilience intervention for filipino disaster survivors’, philippine journal of psychology, vol. 48, no. 2, pp. 105–131. hechanova, m. r., et al., (2018) ‘evaluation of a resilience intervention for filipino displaced survivors of super typhoon haiyan’, disaster prevention and management, vol. 27, no. 3, pp. 346–359. doi: 10.1108/dpm-01-2018-0001 hechanova, m. r., et al., (2022) ‘online resilience support groups during the covid-19 pandemic: the philippine experience’, mental health and social inclusion, vol. 26, no. 1, pp. 56–65. doi: 10.1108/mhsi-06-2021-0038 hechanova, m. r., valentin, p. & alampay, k. (2022) ‘age, covid-19 and mental health in the philippines: a multi-dimensional perspective’, asia pacific social science review, vol. 22, no. 1, pp. 142–157. hechanova, m. r. m., waelde, l. c. & ramos, p. a. p. (2016) ‘evaluation of a group-based resilience intervention for typhoon haiyan survivors’, journal of pacific rim psychology, vol. 10, no. 12, pp. 1–10. doi: 10.1017/prp.2016.9 http://dx.doi.org/10.25304/rlt.v31.2890 http://dx.doi.org/10.1007/s11121-022-01350-y http://dx.doi.org/10.1371/journal.pone.0013196 http://dx.doi.org/10.1016/j.janxdis.2018.01.001 https://doi.org/10.1037/edu0000719 http://dx.doi.org/10.1037/1040-3590.10.2.176 http://dx.doi.org/10.1177/1073191105283504 http://dx.doi.org/10.25304/rlt.v25.1974 http://dx.doi.org/10.1080/87568225.2022.2029659 http://dx.doi.org/10.1207/s15327558ijbm0401_6 http://dx.doi.org/10.1037/0022-0167.49.2.255 http://dx.doi.org/10.1016/j.aip.2020.101688 http://dx.doi.org/10.1080/13548501003615266 http://dx.doi.org/10.1108/dpm-01-2018-0001 http://dx.doi.org/10.1108/mhsi-06-2021-0038 http://dx.doi.org/10.1017/prp.2016.9 research in learning technology citation: research in learning technology 2023, 31: 2890 http://dx.doi.org/10.25304/rlt.v31.2890 13 (page number not for citation purpose) imran, n., zeshan, m. & pervaiz, z. (2020) ‘mental health considerations for children & adolescents in covid-19 pandemic’, pakistan journal of medical sciences, vol. 36, no. covid19-s4, pp. s67–s72. doi: 10.12669/pjms.36.covid19-s4.2759 joyce, s., et  al., (2018) ‘road to resilience: a systematic review and meta-analysis of resilience training programmes and interventions’, bmj open, vol. 8, pp. 1–9. doi: 10.1136/ bmjopen-2017-017858 li, h., kraut, r. e. & zhu, h. (2021) ‘technical features of asynchronous and synchronous community platforms and their effects on community cohesion: a comparative study of forum-based and chat-based online mental health communities’, journal of computermediated communication, vol. 26, no. 6, pp. 403–421. doi: 10.1093/jcmc/zmab016 liu, k., et al., (2020) ‘effects of progressive muscle relaxation on anxiety and sleep quality in patients with covid-19’, complementary therapies in clinical practice, vol. 39, p. 101132. doi: 10.1016/j.ctcp.2020.101132 luthar, s. s. & cicchetti, d. (2000) ‘the construct of resilience: implications for interventions and social policies’, development and psychopathology, vol. 12, no. 4, pp. 857–885. doi: 10.1017/s0954579400004156 ma, l., et  al., (2021) ‘prevalence of mental health problems among children and adolescents during the covid-19 pandemic: a systematic review and meta-analysis’, journal of affective disorders, vol. 293, pp. 78–89. doi: 10.1016/j.jad.2021.06.021 mukhtar, s. (2020) ‘pakistanis’ mental health during the covid-19’, asian journal of psychiatry, vol. 51, p. 102127. doi: 10.1016/j.ajp.2020.102127 muthén, l. k. & muthén, b. o. (1998–2017) mplus user’s guide, 8th edn, los angeles, ca: muthén & muthén muthen, b. o. & satorra, a. (1995) ‘complex sample data in structural equation modeling’, sociological methodology, vol. 25, pp. 267–316. doi: 10.2307/271070 peugh, j. l. (2010) ‘a practical guide to multilevel modeling’, journal of school psychology, vol. 48, no. 1, pp. 85–112. doi: 10.1016/j.jsp.2009.09.002 sassaki, y., et  al., (2019) ‘pre-disaster social support is protective for onset of post-disaster depression: prospective study from the great east japan earthquake and tsunami’, nature, vol. 9, p. 19427. doi: 10.1038/s41598-019-55953-7 soklaridis, s., et al., (2020) ‘mental health interventions and supports during covid-19 and other medical pandemics: a rapid systematic review of the evidence’, general hospital psychiatry, vol. 66, pp. 133–146. doi: 10.1016/j.genhosppsych.2020.08.007 wei, n., et  al., (2020) ‘efficacy of internet-based integrated intervention on depression and anxiety symptoms in patients with covid-19’, journal of zhejiang university-science b, vol. 21, no. 5, pp. 400–404. doi: 10.1631/jzus.b2010013 world health organization (who). (1998) who-5 well-being index. available at: https:// www.psykiatri-regionh.dk/who-5/pages/default.aspx zhou, l., et  al., (2020) ‘feasibility and preliminary results of effectiveness of social mediabased intervention on the psychological well-being of suspected covid-19 cases during quarantine’, the canadian journal of psychiatry, vol. 65, no. 10, pp. 736–738. doi: 10.1177/0706743720932041 http://dx.doi.org/10.25304/rlt.v31.2890 http://dx.doi.org/10.12669/pjms.36.covid19-s4.2759 http://dx.doi.org/10.1136/bmjopen-2017-017858 http://dx.doi.org/10.1136/bmjopen-2017-017858 http://dx.doi.org/10.1093/jcmc/zmab016 http://dx.doi.org/10.1016/j.ctcp.2020.101132 http://dx.doi.org/10.1017/s0954579400004156 http://dx.doi.org/10.1016/j.jad.2021.06.021 http://dx.doi.org/10.1016/j.ajp.2020.102127 http://dx.doi.org/10.2307/271070 http://dx.doi.org/10.1016/j.jsp.2009.09.002 http://dx.doi.org/10.1038/s41598-019-55953-7 http://dx.doi.org/10.1016/j.genhosppsych.2020.08.007 http://dx.doi.org/10.1631/jzus.b2010013 https://www.psykiatri-regionh.dk/who-5/pages/default.aspx https://www.psykiatri-regionh.dk/who-5/pages/default.aspx http://dx.doi.org/10.1177/0706743720932041 (page number not for citation purpose) 1 *corresponding author. email: elmustapha.baytar@enscasa.ma research in learning technology 2023. © 2023 e.m. baytar et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 research in learning technology vol. 31, 2023 original research article teachers’ sense of competence in terms of ict use: the case of secondary school teachers el mustapha baytar* , hayat elyacoubi , nadia saqri and lynda ouchaouka multidisciplinary laboratory in educational sciences and training engineering, higher normal school, hassan ii university of casablanca, 50069, casablanca, morocco (received: 11 august 2022; revised: 7 january 2023; accepted: 16 april 2023; published: 26 may 2023) in the current digital age, the integration of information and communication technologies (ict) into teaching practices has become a determining factor in learning quality. the teachers’ digital competence issue has come back to the forefront because of the schools’ closure due to the covid-19 pandemic. our study aims to assess the sense of competence in terms of ict use of a sample of 260 secondary school teachers in the directorate of education in rhamna, morocco, by adopting a quantitative methodology. our findings suggested that only 26.1% of the individuals in our sample reported they feel effectively competent. the pandemic made a significant change in teachers’ perceptions of the importance of ict integration and training in the field. moreover, a cross-analysis highlighted significant relationships between the sense of competence in terms of ict use and six independent variables: continuous training, learning readiness, gender, age, teaching experience, and school subject. our findings would be helpful for policymakers to guide educational policies by focusing on ict continuous training to enhance teachers’ digital competence. keywords: ict; educational technology; pedagogical uses; lemke and coughlin framework introduction information and communication technologies (ict) are increasingly used in all areas of human activity and are having a growing influence on the world of education. thus, schools cannot afford to remain disengaged. technology has invaded teachers’ daily lives (şahal and ozdemir 2020), but it has not yet yielded the desired impact on their practice in the classroom. therefore, training on digital competence represents one of the main challenges teachers face as they strive to integrate ict into their teaching practices (garzón-artacho et al. 2021). hence, teachers are required to have the digital attitudes and skills that enable them to design active and innovative teaching strategies. a well-trained teacher is more likely to integrate ict in a relevant way (hsu 2010). the covid-19 lockdown has revealed the vulnerability of many educational systems around the world (lawrence and fakuade 2021). schools’ reactions ranged from conservative to completely innovative (stracke et al. 2022). mailto:elmustapha.baytar@enscasa.ma http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2874 https://orcid.org/0009-0008-9322-5554 https://orcid.org/0000-0002-4036-7769 https://orcid.org/0000-0003-0437-4627 e.m. baytar et al. 2 citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 (page number not for citation purpose) the concept of digital competence has been used extensively in the literature related to ict use to assess the extent to which teachers are able to integrate ict into their teaching practice. this competence is described in the literature by several designations, such as digital literacy, information literacy, digital skills, technology skills, ict skills, teachers’ confidence in using ict, etc. (perifanou et al. 2019). in the same context, in order to assess the digital competence of the individuals in our sample, we measured their ‘sense of competence’ in terms of ict use. moreover, while conducting our literature review, we have noted the use of a range of terms that often refer to teachers’ ict continuous training, such as continuing education (e.g. artacho et al. 2020), continuing/continuous professional development (e.g. holmes 2013), lifelong learning (e.g. garzón-artacho et al. 2021), and pre-service/in-service training (e.g. panagiotis et al. 2011). our understanding of continuous training in this paper is embodied in bjursell’s definition of lifelong learning: ‘[…] it comprises “cradle-to-grave” learning throughout a person’s life and includes formal, non-formal, and informal learning’ (bjursell 2020, p. 675). likewise, the neologism of ‘apprenance’ which reflects the same understanding, has been widely addressed in french literature by p. carré (frétigné 2021). current literature has thus tried to build theoretical models describing the process of pedagogical integration of ict. the common denominator between these models is that they attempt to categorise teachers’ skills along a continuum of progressive development regarding ict integration. this is carried out by describing each stage of the integration process, often ranging from non-use to innovative use. hence, applying these models could provide insight into teachers’ digital competence level, even if each model has its own conception of this competence. the following are three examples of theoretical frameworks for integrating ict into teaching practices: the digital competence of educators framework (digcompedu) (2017) the digicompedu framework describes 22 specific digital skills for educators, constituted of six domains (redecker 2017, p. 9). the framework’s core is described in domains 2–5, where technologies are incorporated in a pedagogically meaningful way (ghomi and redecker 2019). according to ghomi and redecker (2019), it describes six different levels of an educator’s digital competence: newcomers (a1) and explorers (a2) who have basic digital skills, integrators (b1) and experts (b2) who already use digital technologies in their practice in multiple ways, and finally, leaders (c1) and pioneers (c2) who are able to criticise and develop existing practices. concern-based adoption model (1974) this framework was elaborated by hall in 1974 (hall 1974). it displays three main techniques to describe the process of implementing an innovation. the stages of concerns (soc) questionnaire is one of these three tools. it describes teachers’ concerns, that is, feelings regarding integrating an innovation (george, hall, and stiegelbauer 2013, p. ix). it suggests a continuum of six stages (george, hall, and stiegelbauer 2013, p. 8): (0) awareness stage, (1) informational stage, (2) personal stage, (3) management stage, (4) consequence stage, (5) collaboration stage, (6) refocusing stage. generally, a teacher often begins with higher intensity of concern in the lower stages (1 and 2) until developing a higher level of concern in the higher stages (5 and 6) http://dx.doi.org/10.25304/rlt.v31.2874 research in learning technology citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 3 (page number not for citation purpose) through continuous progress (dele-ajayi, fasae, and okoli 2021). although it is an old theoretical model, the cbam is still used in recent research which considers the integration of technology in education as an innovation (e.g. alnujaidi 2021; ashrafzadeh and sayadian 2015; dele-ajayi, fasae, and okoli 2021; hao and lee 2015; st-laurent and poellhuber 2018). the theoretical model of lemke and coughlin (1998) it is a competence model suggested by lemke and coughlin (1998) that describes the necessary competencies for a teacher to use ict effectively in his or her teaching practice (bérubé and poellhuber 2005, p. 33). these competencies are described along a continuum of three stages (coughlin and lemke 1999, p. 11): entry teachers at this stage are aware of the potential of ict to support their practice. however, the reality of their teaching does not reflect this belief. teachers’ access to ict is low as they still need to gain the necessary skills to integrate them effectively and thus make their teaching more meaningful. adaptation ict are increasingly present in teachers’ pedagogical practices at this stage. however, despite succeeding in developing their digital competence, they mainly try to ‘automate, accelerate and improve’ the already existing teaching practices. transformation ict bring radical changes to teaching practices at this stage. they change the roles of teachers and learners as well as their relationships. in addition, teachers are implementing creative practices that were not possible before. in the same context, our study used a validated questionnaire based on lemke and coughlin’s theoretical model (1998) to measure teachers’ sense of competence according to the three levels mentioned earlier. likewise, in our literature review, we found several studies attempting to assess teachers’ digital competence using different theoretical models. it is worth noting that few studies have addressed the digital competence issue during the covid-19 crisis. moreover, we have noticed a lack of studies addressing this issue in the moroccan context, either before or during the pandemic. the same remark has been raised by benali, kaddouri, and azzimani (2018). based on the digcompedu framework, a study was conducted in the moroccan context on a sample of 160 english language teachers (benali, kaddouri, and azzimani 2018). high mean scores were registered in digital resource selection, teaching, and reflective practice. differences in digital competence were significant regarding the digital teaching confidence level and teaching experience. moreover, a survey of 251 greek primary and secondary school teachers conducted just before the closure of schools due to covid-19 (fernández-sánchez, dominguez, and sosa-díaz 2021) found weaknesses in both the digital competence of teachers and continuous training in ict, which explains, according to the study, the difficulties encountered by teachers http://dx.doi.org/10.25304/rlt.v31.2874 e.m. baytar et al. 4 citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 (page number not for citation purpose) during the lockdown period. another study attempted to explore the ict uses of 806 secondary and primary school teachers amid the covid-19 crisis (march to may 2020) (perifanou, economides, and tzafilkou 2021). according to its results, teachers found it difficult to use ict for specific uses, such as final assessment of students, reviewing pedagogical resources, and feedback. furthermore, another study has adopted the concerns-based adoption model (cbam) to assess the digital competence of 340 nigerian teachers by identifying statistical frequencies along a continuum of six concern stages (dele-ajayi, fasae, and okoli 2021). the overall profile suggests that teachers do not use ict in their teaching practices. in general, the non-users’ concerns were highest at the first three stages (awareness, informational, and personal) and lowest at the last three stages (consequence, collaboration, and refocusing). in addition, statistically significant differences were found, in the latter study, between the soc on the one hand and teaching experience and age on the other hand. within this framework, we investigate in this study the level of teachers’ sense of competence in terms of ict use and whether or not continuous training contributes to improving this sense. moreover, our research was conducted in an unprecedented context marked by the constraints of the covid-19 pandemic. thus, we investigate how this crisis would impact teachers’ perceptions and attitudes towards ict use and training. bearing in mind all these questions, the objectives of this study are: objective 1. highlighting the impact of the covid-19 pandemic on secondary school teachers’ perceptions of the importance of integrating ict and training in this area. objective 2. measuring secondary school teachers’ sense of competence in terms of ict use and identifying its relationship with the main variables of our survey. method participants our target population consists of secondary school teachers of the directorate of education in rhamna, which belongs to the regional academy of education – marrakech-safi, one of the 12 regional academies in the kingdom of morocco. the final sample comprised 260 secondary school teachers. their consent was obtained when filling out the questionnaire. it was clear to them what the objectives of the research were. we respected their anonymity and confidentiality. the following table (table 1) illustrates details of teachers’ demographics. instrument to measure teachers’ sense of competence, we drew on the questionnaire developed by lebrun, lison, and batier (2016) based on the theoretical model of lemke and coughlin (1998) of ict integration. the questionnaire of the sense of competence in terms of ict use is made up of 22 items distributed over three levels (entry – adaptation – transformation). the 22 items are distinguished as follows: items from 1 to 11 for entry level (11 items), items from 12 to 17 for adaptation level (six items), and items from 18 to 22 for transformation level (five items). a likert scale was adopted for answers with three options: (1) no = i do not feel competent at all, (2) yes, i think so = i think i am competent, and (3) yes, i do = i feel competent and i perform this action without any problem. moreover, closed-ended choice questions http://dx.doi.org/10.25304/rlt.v31.2874 research in learning technology citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 5 (page number not for citation purpose) were added to gather data about the characteristics of the individuals in our sample, explore teachers’ continuous training, and highlight the impact of the covid-19 crisis on teachers’ perception regarding ict integration. concerning the validation process, we followed a psychometric approach to validate the questionnaire content by seeking the expertise of three referees, who are ict specialists and who have bachelor’s degrees in psychology. they checked that each item belongs to the level it represents, that the items are linguistically correct and understandable, and their effectiveness in measuring what they are supposed to measure. on the other hand, we calculated cronbach’s alpha coefficient to assess the instrument’s reliability. thus, cronbach’s alpha coefficient for the whole questionnaire was 0.960, and those for each of the three dimensions, entry, adaptation, and transformation, were respectively 0.919, 0.898, and 0.924. therefore, these optimal cronbach’s alpha coefficient values ensured our survey’s proper conduct. furthermore, to ensure the impact of an item on the reliability of the questionnaire, we have calculated cronbach’s alpha coefficient for the questionnaire in case of deletion of this item (table 2). since all the values obtained, in the case of deletion of each item, did not exceed the global coefficient (0.96), no item undermines the reliability of the questionnaire. therefore, we decided to keep all items. data collection and analysis two modes of administration were adopted to collect data via the questionnaire: direct administration, which consisted of distributing paper questionnaires, and online administration, which consisted of sharing links to the questionnaire developed in google forms. the questionnaire was written in arabic and french for both modes of administration. in sum, we judged 106 paper questionnaires and 154 digital response forms to be valid and serious. thus, the final size of our sample was n = 260. it is worth noting that the questionnaire was administered just after the first easing of pandemic-related restrictions (april 2021). the analysis of the collected data was carried out to meet the objectives we mentioned above. thus, we proceeded to two modes of analysis using spss statistical table 1. participants’ demographics. variables measurement groups n % variables measurement groups n % gender male 190 73.1 the last diploma baccalaureate 4 1.5 female 70 26.9 baccalaureate + 2 12 4.6 age (years) 20–30 60 23.1 bachelor 158 60.8 31–40 112 43.1 master 74 28.5 41–50 56 21.5 doctorate 12 4.6 51–60 31 11.9 teaching experience (years) 1–5 67 25.8 61 and over 1 0.4 6–15 118 45.4 school subject literary 150 57.7 16–25 61 23.5 scientific 84 32.3 26 and over 14 5.4 activity 26 10 teaching cycle middle school 139 53.5 workplace urban 168 64.6 rural 92 35.4 high school 121 46.5 http://dx.doi.org/10.25304/rlt.v31.2874 e.m. baytar et al. 6 citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 (page number not for citation purpose) software version 22. we carried out a flat sorting analysis to make a descriptive statistics of the collected data and a cross-analysis to highlight possible relationships between our survey variables. results continuous training in ict to the question ‘have you received any type of continuous training in using ict?’ roughly 52% of the participants answered yes. as for the source of this training, about two-thirds of the participants said that it was provided by the ministry of education, and about half of them have engaged in a self-training process. moreover, ‘courses’ and ‘workshops’ were the most common forms of formal continuous training, according to more than half of the respondents. furthermore, more than half of the individuals in the sample, who are at least somewhat satisfied with their level of competence in terms of using ict in their practice, declared that this competence is mainly linked to non-formal training that corresponds to their self-training. in addition, among people who declared to be engaged in a self-training process in terms of ict use, 42% of them consider themselves ‘avid learners’. table 2. cronbach’s alpha coefficient in case of deletion of each item. items ca* items ca* 1. using a computer. 0.959 12. integrating videos into my teaching. 0.958 2. using one of these peripherals: video projector, camera, scanner, ... 0.959 13. using the slide show to structure my teaching sessions. 0.958 3. playing a video on my computer. 0.960 14. participating actively in virtual meetings. 0.958 4. modifying a video on my computer. 0.958 15. organizing my teaching using the ‘teams platform’. 0.957 5. creating a slide show (ppt ...). 0.958 16. integrating web resources into my teaching. 0.958 6. using virtual meeting tools (skype, zoom, google meet …). 0.958 17. using exercisers in my teaching. 0.958 7. using the ‘teams’ learning platform**. 0.957 18. accompanying students in the creation of multimedia resources (videos ...). 0.958 8. searching for information on the web. 0.960 19. training my students to use the technologies involved in my teaching. 0.958 9. managing emails (spam, filters …). 0.960 20. communicating about my techno-pedagogical practices. 0.958 10. handling an exercise editor. 0.957 21. supervising research on techno-pedagogical practices for learning. 0.959 11. handling a web editor. 0.957 22. justifying the interest in my technological and pedagogical approach. 0.958 *cronbach’s alpha coefficient. **a platform created by the ministry of education in the wake of the pandemic. http://dx.doi.org/10.25304/rlt.v31.2874 research in learning technology citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 7 (page number not for citation purpose) perceptions’ change regarding ict integration at the time of the pandemic before the pandemic, about 15% of teachers (figure 1) considered that it was not important for a teacher to be able to use ict in the classroom for her/his teaching activities, and 32% of them considered that it was rather very important. while during the pandemic, the first portion decreased to 6%, and the second increased to 78%. as for learners, 18% of the participants considered, before the pandemic, that it was not important for the learners to use ict in their classroom learning activities, and 27% of them thought that it was rather very important. while during the pandemic, the first portion decreased to 6%, and the second increased to 69%. in the same sense, among the 180 individuals, that is, 69% in the sample, who responded to the question, ‘if you did not feel fully competent to use ict during the pandemic, do you intend to fill your gaps by engaging in a self-training process?’, 92% answered affirmatively. the sense of competence regarding the use of ict as a first step, we present the distribution of the 260 participants according to their  global scores obtained for all the sense of competence questionnaire items (figure 2). the results suggest that 31.5% of the individuals do not feel competent at all in terms of ict use, while 26.1% feel effectively competent. however, these results gave us an overarching idea about the teachers’ digital competence level. thus, to situate the individuals in the sample on the continuum of competence suggested by lemke and coughlin’s theoretical model (1998) (entry, adaptation, transformation), another work of grouping the participants according to the three levels of the questionnaire was necessary (figure 3). therefore, the results of this work indicated that the highest portion of the individuals who reported they feel effectively competent in terms of ict use was observed in the first level (entry), that is, 40%. then, a portion of 30.8% was observed in the second level (adaptation). finally, the lowest portion was in the last level of transformation (17.3%), which is theoretically a level of innovative practices. figure 1. teacher’s perception of the importance of information and communication technologies (ict) integration, both for her/himself and for her/his learners before and during the pandemic. http://dx.doi.org/10.25304/rlt.v31.2874 e.m. baytar et al. 8 citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 (page number not for citation purpose) the sense of competence and the main independent variables using the independent samples t-test, we tested the relationship between the sense of competence on one side and all the independent variables with two measurement groups on the other side. the differences were statistically significant for three variables (p < 0.05): continuous ict training, learning readiness, and gender (table 3). hence, it can be concluded that: • people who have benefited from ict training feel significantly more competent than those who have not; • people who consider themselves avid learners feel significantly more competent than those who do not; • males significantly more than females declared themselves competent in using ict. figure 2. the sense of competence according to the global score of the questionnaire. figure 3. the sense of competence according to the three levels of the questionnaire. http://dx.doi.org/10.25304/rlt.v31.2874 research in learning technology citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 9 (page number not for citation purpose) in the same way, as earlier, we also tested, using the anova test (analysis of variance), the relationship between teachers’ sense of competence in terms of ict use and all other independent variables with more than two measurement groups. differences were statistically significant for three variables: teaching experience, age, and school subject. furthermore, to specify which pairs of groups have a significant difference, we carried out a post hoc test (the least significant difference test). below is an example of an lsd test regarding teaching experience (table 4). the lsd test results suggested that recent hires (1–5 years) feel significantly more competent than more experienced hires (16–25; 26 years over). likewise, as for the age and the school subject variables, and following the same procedure, results have suggested that: • individuals in the youngest age ranges, namely 20–30, 31–40, and 41–50, feel  significantly more competent than individuals in the 51–60 age range (p = 0.02 < 0.05). • teachers in science and activity school subjects feel significantly more competent in terms of ict use than those in literary school subjects (p = 0.04 < 0.05). discussion our study was an attempt to measure teachers’ sense of competence in terms of ict use and to ascertain a potential relationship between this dependent variable table 3. the sense of competence and the two measurement groups variables. n = 260 *ct (%) **lr (%) gender (%) yes no yes no m f 1. i do not feel competent at all 37.8 62.2 75.6 24.4 62.2 37.8 2. i think i am competent 58.2 41.8 61.8 38.2 76.4 23.6 3. i feel i am competent 57.4 42.6 45.6 54.4 80.9 19.1 * t-test ict continuous training: m (no) = 1.83 m (yes) = 2.06 p = 0.013 ** t-test learning readiness: m (no) = 1.81 m (yes) = 2.17 p = 0.000 t-test gender: m (m) = 2.02 m (f) = 1.74 p = 0.009 ict, information and communication technologies. table 4. lsd test (multiple comparison). (i) (j) md (i-j) sig. (i) (j) md (i-j) sig. 1–5 6–15 0.162 0.161 16–25 1–5 –0.316* 0.018 16–25 0.316* 0.018 6–15 –0.154 0.194 26 and over 0.477* 0.032 26 and over 0.160 0.472 6–15 1–5 –0.162 0.161 26 and over 1–5 –0.477* 0.032 16–25 0.154 0.194 6–15 –0.315 0.140 26 and over 0.315 0.140 16–25 –0.160 0.472 * the mean difference (md) is significant at the 0.05 level. lsd, the least significant difference test. http://dx.doi.org/10.25304/rlt.v31.2874 e.m. baytar et al. 10 citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 (page number not for citation purpose) and other independent variables (o2). moreover, the conduct of this study during the covid-19 crisis was an opportunity to examine a potential change in these teachers’ perceptions and attitudes towards the use of ict and training in this field (o1). one of our study’s particularities is that it was conducted amid the covi-19 crisis, so it was an opportunity to identify its impact on teachers’ perceptions and attitudes towards the importance of ict use by asking about their perception before and during the pandemic. many studies have found connections and affinities between teachers’ attitudes and their digital competence (yusuf and balogun 2011). people with positive attitudes are more likely to integrate ict into their teaching practice (teo 2008). our results have explicitly indicated the pandemic impact in favour of the importance of ict as perceived by teachers, whether for their teaching or for their learners’ activities. these findings would be relevant to the ministry of education. it should endeavour to foster a supportive learning environment for teachers in both formal and non-formal settings since our results showed that formal training is mostly provided by this ministry, and the majority of teachers are engaged in a self-training process for their professional development. moreover, it would be helpful if the ‘unified system for validating individuals’ cognitive and professional learning’, recommended by the 2015–2030 strategic vision (csefrs maroc 2015, p. 71), is implemented as soon as possible to promote teachers’ lifelong learning. furthermore, our results of the sense of competence questionnaire indicated that about a quarter of the individuals in our sample feel effectively competent (26.1%). in addition, according to the three questionnaire levels, the highest portion of these individuals was in the first level of entry (40%) which is theoretically a basic level. these findings are consistent with perifanou et al. results (2019), which stressed a lack of teachers’ digital competencies that include not only ict use but also the capability to integrate technology through innovative methods and creative ideas. in the same context, the results of benali, kaddouri, and azzimani study (2018), which was conducted in the moroccan context and adopted the digicompedu framework, indicated that the majority of participants (english language teachers) scored in the intermediate levels of digital competence (b1 and b2). in addition, our findings are reflected, with similar trends, in other studies in different contexts, especially when it comes to the dimensions of digital competence that require creative and innovative abilities, despite using various measurement tools and labels of digital competence (e.g. artacho et al. 2020; dele-ajayi, fasae, and okoli 2021; garzón-artacho et al. 2021). furthermore, our study suggested significant relationships between teachers’ sense of competence in terms of ict use and six other independent variables. firstly, people who have benefited from ict training feel significantly more competent than those who have not. other studies have suggested the same finding. (e.g. artacho et al. 2020; hsu 2010; perifanou et al. 2019). secondly, differences in the sense of competence based on teaching experience were significant. recent hires (1–5 years) feel significantly more competent than more experienced hires (16–25; 26 years over). this conclusion is backed up by other studies’ findings (e.g. benali, kaddouri, and azzimani 2018; dele-ajayi, fasae, and okoli 2021). thirdly, significant differences were found regarding teachers’ learning readiness. individual characteristics and attitudes towards implementing ict truly influence digital competence (benali, kaddouri, and azzimani 2018). fourthly, significant differences http://dx.doi.org/10.25304/rlt.v31.2874 research in learning technology citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 11 (page number not for citation purpose) were indicated in connection with individuals’ age. thus, those in the youngest age ranges (20–30, 31–40, and 41–50) feel more competent than individuals in the 51–60 age range. ‘this is arguably a generational issue, not simply a pedagogical one’ (tan and mcwilliam 2009, p. 9). technology is fully integrated into young people’s lives compared to earlier when individuals had difficulties dealing with the electronic world (tapscott 2009, p. 7). however, perifanou et al. (2019) found no correlation between teachers’ age and their confidence in using ict. fifthly, significant differences in terms of gender were concluded, where more male than female participants declared themselves to be more competent in using ict. a surprising remark is that none of the studies we consulted indicated significant differences in digital competence regarding gender (e.g. benali, kaddouri, and azzimani 2018; dele-ajayi, fasae, and okoli 2021; garzón-artacho et al. 2021; sánchez prieto et al. 2020). lastly, it was found that teachers in science and activity school subjects feel significantly more competent in terms of ict use than those in literary school subjects. none of the studies we reviewed have raised this point or adopted our classification for school subjects. conclusion teacher digital competence has been the subject of several studies using different labels and measurement tools in different contexts. our study is part of this scientific dynamic. its relevance is due to its conduct amid the covid-19 crisis. in order to meet its two research objectives, our study has adopted a quantitative approach. thus, the present research suggested that pandemic-related constraints, especially school closures during the lockdown, have made a significant change in teachers’ attitudes towards the importance of the integration of ict and of training in that field. moreover, only 26.1% of the individuals in our sample reported that they feel effectively competent. their distribution according to the three levels of the sense of competence questionnaire showed that the highest portion (40%) performs basic uses of ict (entry), and only 17.3% performs innovative uses (transformation). in addition, a cross-analysis highlighted the impact of continuous training and readiness learning on teachers’ sense of competence. likewise, significant relationships were found between the sense of competence in terms of ict use and four other participants’ attributes: gender, age, teaching experience, and school subject. finally, it should be noted that this paper’s results do not claim to be exhaustive. we have tried to present some answers to our research questions that would benefit from being more in-depth. we are aware of our research limitations related mainly to the reduced sample size, the limited geographical area of study, and the use of a self-perception questionnaire where the risk of inducing subjective responses is considerable, which could affect the quality of the results. thus, it would be difficult to generalise these results to the entire kingdom territory. however, such research can still be helpful for policymakers to guide their educational policies. hence, to overcome the limitations mentioned earlier, it would be interesting to extend the reflection of the present paper to further research by adopting a qualitative methodology or by conducting action research. likewise, it might be interesting to understand what role other stakeholders, such as the learner and school leaders, would play in fostering teachers’ digital competence. http://dx.doi.org/10.25304/rlt.v31.2874 e.m. baytar et al. 12 citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 (page number not for citation purpose) references alnujaidi, s. (2021) ‘adoption of mobile assisted language learning (mall) in saudi arabian efl classrooms’, journal of language teaching and research, vol. 12, no. 2, pp. 312–323. doi: 10.17507/jltr.1202.13 artacho, e. g., et al., (2020) ‘teacher training in lifelong learning-the importance of digital competence in the encouragement of teaching innovation’, sustainability (switzerland), vol. 12, no. 7, p. 2852. doi: 10.3390/su12072852 ashrafzadeh, a. & sayadian, s. (2015) ‘university instructors’ concerns and perceptions of technology integration’, computers in human behavior, vol. 49, pp. 62–73. doi: 10.1016/j. chb.2015.01.071 benali, m., kaddouri, m. & azzimani, t. (2018) ‘digital competence of moroccan teachers of english’, international journal of education and development using ict, vol. 14, no. 2, pp. 99–120. bérubé, b. & poellhuber, b. (2005) ‘un référentiel de compétences technopédagogiques destiné au personnel enseignant du réseau collégial’, regroupement des collèges performa, [online] available at: http://cmap.ccdmd.qc.ca/rid=1lvlxncls-1zk34t7-4wt/referentiel.pdf bjursell, c. (2020) ‘the covid-19 pandemic as disjuncture: lifelong learning in a context of fear’, international review of education, vol. 66, nos. 5–6, pp. 673–689. doi: 10.1007/ s11159-020-09863-w coughlin, e. & lemke, c. (1999) professional competency continuum: professional skills for the digital age classroom, milken exchange on education technology, [online] available at: https://www.mff.org/assets/uploads/newsroom_archive/publications/me159.pdf csefrs maroc. (2015) ‘la vision stratégique 2015–2030’, conseil supérieur de l’éducation, de la formation et de la recherche scientifique, [online] available at: https://www.csefrs.ma/ wp-content/uploads/2017/09/vision_vf_fr.pdf dele-ajayi, o., fasae, o. d. & okoli, a. (2021) ‘teachers’ concerns about integrating information and communication technologies in the classrooms’, plos one, vol. 16, no. 5, p. e0249703. doi: 10.1371/journal.pone.0249703 fernández-sánchez, m. r., dominguez, f. i. r. & sosa-díaz, m. j. (2021) ‘the educational integration of digital technologies precovid-19: lessons for teacher education’, plos one, vol. 16, no. 8, p. e0256283. doi: 10.1371/journal.pone.0256283 frétigné, c. (2021) ‘carré, p. (2020). pourquoi et comment les adultes apprennent. de la formation à l’apprenance’, recherche et formation, [online] available at: https://journals.openedition.org/rechercheformation/6615 garzón-artacho, e., et al., (2021) ‘teachers’ perceptions of digital competence at the lifelong learning stage’, heliyon, vol. 7, no. 7, p. e07513. doi: 10.1016/j.heliyon.2021.e07513 george, a. a., hall, g. e. & stiegelbauer, s. (eds) (2013) measuring implementation in schools: the stages of concern questionnaire, 3rd printing, sedl, austin, tx. ghomi, m. & redecker, c. (2019) ‘digital competence of educators (digcompedu): development and evaluation of a self-assessment instrument for teachers’ digital competence’, proceedings of the 11th international conference on computer supported education, scitepress, heraklion, pp. 541–548. doi: 10.5220/0007679005410548 hall, g. e. (1974) the concerns-based adoption model: a developmental conceptualization of the adoption process within educational institutions, the univerity of texasat, [online] available at: https://eric.ed.gov/?id=ed111791 hao, y. & lee, k. s. (2015) ‘teachers’ concern about integrating web 2.0 technologies and its relationship with teacher characteristics’, computers in human behavior, vol. 48, pp. 1–8. doi: 10.1016/j.chb.2015.01.028 holmes, b. (2013) ‘school teachers’ continuous professional development in an online learning community: lessons from a case study of an etwinning learning event’, european journal of education, vol. 48, no. 1, pp. 97–112. doi: 10.1111/ejed.12015 http://dx.doi.org/10.25304/rlt.v31.2874 https://doi.org/10.17507/jltr.1202.13 https://doi.org/10.3390/su12072852 https://doi.org/10.1016/j.chb.2015.01.071 https://doi.org/10.1016/j.chb.2015.01.071 http://cmap.ccdmd.qc.ca/rid=1lvlxncls-1zk34t7-4wt/referentiel.pdf http://cmap.ccdmd.qc.ca/rid=1lvlxncls-1zk34t7-4wt/referentiel.pdf https://doi.org/10.1007/s11159-020-09863-w https://doi.org/10.1007/s11159-020-09863-w https://www.mff.org/assets/uploads/newsroom_archive/publications/me159.pdf https://www.csefrs.ma/wp-content/uploads/2017/09/vision_vf_fr.pdf https://www.csefrs.ma/wp-content/uploads/2017/09/vision_vf_fr.pdf https://doi.org/10.1371/journal.pone.0249703 https://doi.org/10.1371/journal.pone.0256283 https://journals.openedition.org/rechercheformation/6615 https://journals.openedition.org/rechercheformation/6615 https://doi.org/10.1016/j.heliyon.2021.e07513 https://doi.org/10.5220/0007679005410548 https://eric.ed.gov/?id=ed111791 https://doi.org/10.1016/j.chb.2015.01.028 https://doi.org/10.1111/ejed.12015 research in learning technology citation: research in learning technology 2023, 31: 2874 http://dx.doi.org/10.25304/rlt.v31.2874 13 (page number not for citation purpose) hsu, s. (2010) ‘the relationship between teacher’s technology integration ability and usage’, journal of educational computing research, vol. 43, no. 3, pp. 309–325. doi: 10.2190/ ec.43.3.c lawrence, k. c. & fakuade, o. v. (2021) ‘parental involvement, learning participation and online learning commitment of adolescent learners during the covid-19 lockdown’, research in learning technology, vol. 29. doi: 10.25304/rlt.v29.2544 lebrun, m., lison, c. & batier, c. (2016) ‘les effets de l’accompagnement technopédagogique des enseignants sur leurs options pédagogiques, leurs pratiques et leur développement professionnel’, revue internationale de pédagogie de l’enseignement supérieur, vol. 32, no. 1. doi: 10.4000/ripes.1028 lemke, c. & coughlin, e. (1998) technology in american schools: seven dimensions for gauging progress. a policymaker’s guide, milken exchange on education technology, [online] available at: https://eric.ed.gov/?id=ed460677 panagiotis, g., et al., (2011) ‘informatics and communication technologies (ict) and in-service teachers’ training’, review of european studies, vol. 3, no. 1, pp. 2–12. doi: 10.5539/res. v3n1p2 perifanou, m., et al., (2019) ‘investigating teachers’ confidence and training needs on digital literacy across four european countries’, inted2019 proceedings, valencia, pp. 6808–6817. perifanou, m., economides, a. a. & tzafilkou, k. (2021) ‘teachers’ digital skills readiness during covid-19 pandemic’, international journal of emerging technologies in learning (ijet), vol. 16, no. 8, pp. 238–251. doi: 10.3991/ijet.v16i08.21011 redecker, c. (2017) ‘european framework for the digital competence of educators: digcompedu’, joint research centre (seville site), [online] available at: https://ideas. repec.org/p/ipt/iptwpa/jrc107466.html şahal, m. & ozdemir, a. ş. (2020) ‘pre-service primary teachers’ views and use of technology in mathematics lessons’, research in learning technology, vol. 28. doi: 10.25304/rlt.v28.2302 sánchez prieto, j., et al., (2020) ‘gender and digital teaching competence in dual vocational education and training’, education sciences, vol. 10, no. 3, p. 84. doi: 10.3390/educsci10030084 st-laurent, s. f. & poellhuber, b. (2018) ‘change process of two postsecondary teachers in the early adoption of an active learning classroom’, frontiers in ict, vol. 5, p. 12. doi: 10.3389/ fict.2018.00012 stracke, c. m., et al., (2022) ‘responding to the initial challenge of the covid-19 pandemic: analysis of international responses and impact in school and higher education’, sustainability (switzerland), vol. 14, no. 3, p. 1876. doi: 10.3390/su14031876 tan, j. p.-l. & mcwilliam, e. (2009) ‘from literacy to multiliteracies: diverse learners and pedagogical practice’, pedagogies: an international journal, vol. 4, no. 3, pp. 213–225. doi: 10.1080/15544800903076119 tapscott, d. (ed) (2009) grown up digital how the net generation is changing your world, mcgraw hill professional. teo, t. (2008) ‘pre-service teachers’ attitudes towards computer use: a singapore survey’, australasian journal of educational technology, vol. 24, no. 4, pp. 413–424. doi: 10.14742/ ajet.1201 yusuf, m. o. & balogun, m. r. (2011) ‘student-teachers’ competence and attitude towards information and communication technology: a case study in a nigerian university’, contemporary educational technology, vol. 2, no. 1, pp. 18–36. doi: 10.30935/cedtech/6041 http://dx.doi.org/10.25304/rlt.v31.2874 https://doi.org/10.2190/ec.43.3.c https://doi.org/10.2190/ec.43.3.c https://doi.org/10.25304/rlt.v29.2544 https://doi.org/10.4000/ripes.1028 https://eric.ed.gov/?id=ed460677 https://doi.org/10.5539/res.v3n1p2 https://doi.org/10.5539/res.v3n1p2 https://doi.org/10.3991/ijet.v16i08.21011 https://ideas.repec.org/p/ipt/iptwpa/jrc107466.html https://ideas.repec.org/p/ipt/iptwpa/jrc107466.html https://doi.org/10.25304/rlt.v28.2302 https://doi.org/10.3390/educsci10030084 https://doi.org/10.3389/fict.2018.00012 https://doi.org/10.3389/fict.2018.00012 https://doi.org/10.3390/su14031876 https://doi.org/10.1080/15544800903076119 https://doi.org/10.14742/ajet.1201 https://doi.org/10.14742/ajet.1201 https://doi.org/10.30935/cedtech/6041 (page number not for citation purpose) 1 *corresponding author. email: s.berry@swansea.ac.uk research in learning technology 2019. © 2019 s. berry research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology vol. 27, 2019 original research article non-institutional learning technologies, risks and responsibilities: a critical discourse analysis of university artefacts sam berry* swansea university medical school, swansea university, swansea, uk (received: 14 june 2019; revised: 2 october 2019; accepted: 15 october 2019; published: 21 november 2019) non-institutional technologies include external or third-party technologies that are not officially sanctioned or supported by higher education institutions (heis) but may be used by staff for educational purposes. these include free, open-source and open-access technologies such as social media sites, apps and online services. the literature identifies a number of risks and ethical considerations when using digital technologies, such as security, safety, privacy and legal compliance (common sense n.d.). this study analyses institutional artefacts, including policy and guidance documents, to explore how institutions are addressing the risks of educational technologies identified throughout the literature. critical discourse analysis was conducted on nine artefacts, obtained from seven uk heis. the study found that institutional policies and guidance documents do not sufficiently address some of the key risks identified in the literature (e.g. security risks), nor consider the ethical issues emerging from the use of profit-making educational products. users of these technologies (including teaching staff) are assigned a broad range of complex and potentially time-consuming responsibilities concerning the evaluation, selection and operation of these technologies. for example, to ensure compliance with data protection legislation, however, no artefact stated how this should be achieved. the study therefore identifies significant inadequacies in institutional policies and guidelines, and questions whether appropriate quality assurance processes and safeguards are in place when non-institutional technologies are used for higher education. keywords: third party; security; privacy; data protection; policy introduction there has been increasing development and availability of open-access, open-source and free online services that can be used by learners and educators, i.e. a decentralised technology model (weller 2010). there are a number of benefits of these technologies such as scalability, familiarity and cross-platform compatibility (conole and alevizou 2010). there are also risks and considerations. for example, what mechanisms are in place to protect learners’ data when stored in the cloud? is it appropriate to use an educational app if the app collects personal data from students’ devices or sells that data? to what extent are teaching staff responsible for answering these questions, when the technology is not institutionally provided or supported? mailto:s.berry@swansea.ac.uk http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v27.2284 s. berry 2 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) this study evaluates institutional artefacts which relate to the use of non-institutional technologies by teaching staff for educational purposes. the emphasis on educational purposes intends to distinguish this from other potential uses within academia, such as to conduct or disseminate research. although this study does not focus on how these artefacts are implemented in practice, this study intends to make a contribution to the field by exploring the extent to which higher education institutions (heis) consider risks and ethical issues, and the standards and procedures set out to address these issues when non-institutional technologies are adopted by staff for educational purposes. document analysis can also contextualise empirical findings, and suggest areas for further investigation (bowen 2009). research questions rq1: how do institutional texts distinguish between institutional and non-institutional technologies? rq2: according to these texts, what responsibilities are assigned to teaching staff when non-institutional technologies are utilised for educational purposes? rq3: according to these texts, to what extent do teaching staff have autonomy to use non-institutional technologies? literature review technology risk in higher education there are many examples in the literature of the various ways that digital technologies are being used to support learning, teaching and assessment within higher education. for example, this might include students using a mobile device to participate in a live poll (shon and smith 2011), using whatsapp to facilitate learning whilst on placement (raiman, antbring, and mahmood 2017), or maintaining an eportfolio for assessment (garrett, macphee, and jackson 2013). security and privacy issues have been identified and widely reported concerning educational technologies (e.g. alim et al. 2017; kelly et al. 2018; kelly, graham, and fitzgerald 2018; lorenz, kalde, and kikkas 2012). there are many threats to educational systems beyond deliberate and malicious attacks, such as accidental data loss, data corruption and loss of service (due to upgrade interruption or through discontinuation) (akande and van belle 2016). online and cloud-based services also pose a number of considerations, such as whether it is appropriate for third-party service providers to be able to access confidential and sensitive data that may exist if these services are being used for educational purposes. this could include the student’s personal details (such as name and email address), demographic information, learning and assessment data (such as grades, feedback, failures and attempts) and information concerning special circumstances or specific learning difficulties (regan and jesse 2018). in addition, polonetsky and tene (2014) report that online educational services surreptitiously collect data using passive methods such as cookies. passive data collection occurs through the use of a device or application, and can be used to continuously capture data such as location and movement tracking, activity data (such as search history) and emotional data (herold 2018), which may occur without the user’s knowledge or informed consent, and even when a device or application is not in use (brandtzaeg, pultier, and moen 2018). http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 3 (page number not for citation purpose) some uk heis have developed ‘bring your own device’ (byod) policies (walker et al. 2014), which may require staff and students to access educational services and applications from personal devices (chatzigavriil et al. 2014). security, privacy and data protection issues are therefore of particular concern given the wealth of personal data that may exist on or be generated from personal devices (miller, voas, and hurlburt 2012). there are also a number of ethical issues to consider concerning the use of profit-making online tools (purvis, rodger, and beckingham 2016), which can collect and disseminate personal data for advertising firms to exploit (lindh and nolin 2016), or who may share or sell users data or content (kelly et al. 2018b). previous studies have found that students are concerned about their online privacy when using digital technologies in higher education (aymerich-franch and fedele 2014). students’ perceptions of security and privacy can influence the educational use of cloud-based technologies (arpaci, kilicer, and bardakci 2015) and their participation in online learning (lorenz, sousa, and tomberg 2013). furthermore, a previous study found that students assumed and expected their tutors and/or the institution to have conducted checks (such as for malware) prior to recommending or requesting the use of third-party educational technologies (author 2019). hardré (2016) has asserted that when individuals believe that someone or something else is observing or monitoring a situation, they become subconsciously less watchful of it themselves, i.e. there is diminished vigilance. there is therefore the risk that if students expect someone else to have performed checks and implemented safeguards, that they assume these products are trustworthy and safe, and do not take precautions themselves e.g. reading the terms and conditions. furthermore, it has been argued that when risks are imposed without consent, they may be considered less acceptable than when undertaken voluntarily (farahmand, yadav, and spafford 2013). however, conducting such checks would require technical and legal knowledge. privacy statements have been criticised for being verbose and containing technical or vague terms that are difficult to understand (reidenberg et al. 2015). furthermore, there are limitations to the checks that can be conducted by end users as it is unlikely that access to information or systems would be available to conduct comprehensive evaluations. for example, how might an individual check for the presence of malware prior to downloading a product? higher education policy on technology use in teaching and learning conole (2013) asserts that there needs to be a balance between institutional coordination and individual experimentation with digital technologies. this requires the need for clear policies and guidelines (conole 2013). within this study, a policy is defined as ‘…administrative regulation …used to define the obligations of the institution, expectations of employees and/or students, and consequences if the expectations are violated’ (lenartz 2012, p. 346). a literature search was conducted on 10th april 2017, using scopus, to identify the literature on uk hei policies, regulations or guidelines, and the use of non-institutional technologies. the following search terms were used: (higher education) and (uk) and (external and services) or (third and party) or (cloud) or (web 2.0) or (outsource*) or (non-institutional) http://dx.doi.org/10.25304/rlt.v27.2284 s. berry 4 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) and (polic*) or (regulation*) or (guidance) or (guidelines) or (strateg*) or (best and practice) the search generated 26 results, none of which were found to be relevant to this study. similar studies were found through conducting further searches on scopus and google scholar. these included an evaluation of social media policies (mcneill 2012; pomerantz, hank, and sugimoto 2015), copyright of elearning and teaching materials (gadd and weedon 2017), esubmission (newland, martin, and ramsden 2011) and the electronic management of assessment (voce 2015). in a similar study, pomerantz, hank, and sugimoto (2015) reported that as of 2015, there is only one study that has assessed the content of social media policies. this demonstrates that there is a lack of empirical and theoretical work in this area. furthermore, studies that focus on policies may be limited, as there are many types of institutional documentation, such as contracts, information guides and staff training materials that can also establish and promote institutional expectations, quality standards and employee responsibilities. research design methodology within this study, a social constructivist view is taken, in which the language of institutional texts ‘…does not describe social processes and structures, but creates and supports them’ (saarinen 2008, p. 719). one limitation of this study is that it will not investigate the perspectives or experiences of individuals. however, these texts are seen as ways in which institutions represent and account for themselves (coffey 2014), promote and constrain particular practices and provide an insight into the conditions that can have an impact upon the phenomena under investigation (bowen 2009). discourse is considered to play a significant role in the creation and shaping of social processes (saarinen 2008). critical discourse analysis (cda) is both an analytical tool and a theoretical approach, which emphasises making the hidden visible (saarinen 2008); therefore, cda can be used to interrogate texts to explore how social relations are reproduced or contested, assess whose interests are served or negated, how the text is positioned or positioning and explore the consequences of this positioning (janks 1997). method document analysis is a systematic procedure for reviewing or evaluating documents (bowen 2009). within cda, there is no set procedure to generate a sample (or corpus) for evaluation, as ‘…people approach it in different ways according to the specific nature of the project, as well as their own views of discourse’ (fairclough 1993, p. 225). the lack of literature in this field means that the type of artefacts to be collected and the prevalence of those artefacts are unknown. therefore, 30 uk heis (18%) were randomly chosen from the list of 167 uk heis, produced by hesa. the purpose of the sampling exercise was not to generate a statistically representative sample, but an analytically diverse sample. ethical approval for this study was granted on 9th march 2017, by a higher education research ethics committee. http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 5 (page number not for citation purpose) a scoping exercise was conducted using google search and hei websites to explore artefacts that may be relevant to this investigation and generate search criteria. this informed the development of a process outlined in figure 1. the prisma 2009 flow diagram (moher et al. 2009) inspired the development of this process, namely in terms of providing a systematic framework in which to organise the data collection phases. stage 1 was conducted for each institution, and upon completion, all artefacts were reviewed for eligibility (stage 2) based on three key factors: 1. the intended audience must include teaching staff (whether explicitly stated or implied). 2. the artefact must include non-institutional technologies. 3. the artefact must include educational purposes. it is recognised that this process is not exhaustive and that alternative steps could have been taken by contacting the institutions or organisations such as the heads of elearning forum (as has been conducted in previous he policy studies such as newland, martin, and ramsden 2011). however, the benefit of conducting the study this way is that it intends to not limit the artefacts to those produced by particular communities within the institution, or to particular genres of discourse. cda was conducted on a subset of the eligible artefacts retrieved (as explained in the following section). a framework was produced to guide the analysis process based on the work of fairclough (2001), boag-munroe (2004), o’connell (2015) and voce (2015). the framework was organised into structural and linguistic factors for analysis, such as interdiscursivity, accessibility, speech functions, grammatical mood and modality. these areas of analysis were prioritised based on their relevance to the research questions. speech functions can signal the authors commitment to truth (epistemic modality) and the authors commitment to obligation or necessity (deontic modality) (fairclough 2003). modality can be used to determine the authors ‘degrees of affinity’ figure 1. a summary of the research process. http://dx.doi.org/10.25304/rlt.v27.2284 s. berry 6 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) to the proposition (fairclough 1993), and can therefore signify the institutions commitment to propositions, and the level of authority present within the texts (voce 2015). data analysis data collection was conducted on april 2017. figure 2 summarises the number of artefacts found and reviewed at each stage.1 artefacts were obtained for 25 (83%) of the institutions as a result of stage 1. for the remaining institutions, it is possible that relevant artefacts may not be publicly available. assessing the eligibility of the artefacts was very difficult, given the range of texts and limitations of interpretation as an outsider. for example, a number of artefacts were found relating to the use of external or cloud-based services but did not explicitly reference educational uses of these technologies or were unclear on whether the text applied to teaching staff. each of the documents reviewed in stage two (n = 100) were assigned a colour (shades of blue), which signified the degree to which it aligned with the eligibility criteria. the stronger the colour, the greater the relevance to this study (summarised in table 1). a total of 29 artefacts were excluded upon completion of stage 2, either because they did not meet the eligibility criteria (n = 27, marked in grey in table 1) or because it was not possible to determine whether they applied to teaching staff (n = 2, marked in orange). a total of 71 artefacts (from 22 institutions) were retrieved which related (to some degree) to the use of non-institutional technologies within heis. however, to facilitate an in-depth cda of these texts, the nine most relevant artefacts (from seven institutions) were selected, eight documents and one website. table 2 provides an overview 1 note that for copyright reasons and to preserve anonymity, quotations could not be reproduced from these artefacts. instead, the examples presented throughout are fabricated, but based on actual statements. figure 2. a summary of the research process. http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 7 (page number not for citation purpose) of the heis included in the study, and table 3 summarises the production details for each artefact. there was great variation in the tone of the artefacts. artefacts 3, 5 and 8 (policy documents) were more formal in tone, signified by institutional logos, publication details, structure (e.g. numbered headings and sub-headings), lexical choice and declarations. for example, a5 declared that it was a mandatory policy. in contrast, artefacts 1, 2 and 9 were less formal, and a1 was more collegial in tone. artefacts 4–7 table 1. summary of the eligibility score for each institutional artefact reviewed. institution no. eligibility score institution no. eligibility score institution no. eligibility score institution no. eligibility score i01 i07 i13 i21 i23 i09 i15 i24 i25 i02 i16 i27 i10 i17 i28 i04 i11 i18 i12 i29 i19 i30 i20 i05 i06 table 2. hei mission group affiliation and student numbers (hesa n.d.). institution no. mission group no. students0 i04 university alliance 20000 -< 25000 i11 unaffiliated 5000 -< 10000 i16 university alliance 20000 -< 25000 i23 million+ 10000 -< 15000 i25 unaffiliated 5000 -< 10000 i27 guild he 5000 -< 10000 i30 unaffiliated 10000 -< 15000 note: please note that the precise number of students is not provided to preserve anonymity. http://dx.doi.org/10.25304/rlt.v27.2284 s. berry 8 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) t ab le 3 . p ro d u ct io n d et ai ls f o r ea ch a rt ef ac t. a rt ef ac t n o. a rt ef ac t ty p e p u b li ca ti o n d at e r ev ie w d at e v er si o n nu m b er a u th o r/ s p ro d u ct io n i n fo rm at io n l en gt h (w o rd s) a 1 g u id el in es – u se o f ex te rn al s er vi ce s o r so ft w ar e fo r le ar n in g n o n o n o u n k n o w n l ea rn in g an d t ea ch in g c o m m it te e 79 2 a 2 s ta ff i n d u ct io n d o cu m en t – le ar n in g te ch n o lo gi es n o n o n o u n k n o w n 11 0 a 3 a cc ep ta b le u se p o li cy 20 14 20 16 2. 0 in d iv id u al ( in fo rm at io n s er vi ce s an d t ec h n o lo gy d ep t. ) 39 78 a 4 s o ci al m ed ia p o li cy n o n o n o e d u ca ti o n al d ev el o p m en t u n it 15 63 a 5 s o ci al m ed ia p o li cy 20 15 n /a 1 u n k n o w n l eg al , d ig it al s er vi ce s, i n fo rm at io n s ec u ri ty & p o li ci es c o m m it te e 38 83 a 6 s o ci al m ed ia p o li cy n o n o n o u n k n o w n in d iv id u al ( m ar ke ti n g d ep ar tm en t) 44 36 a 7 s o ci al m ed ia g u id el in es n o n o n o u n k n o w n 23 81 a 8 s o ci al m ed ia p o li cy 20 12 n /a n o u n k n o w n m ar ke ti n g 14 42 a 9 s o ci al m ed ia : st af f u se r gu id e (w eb si te ) n o n o n o u n k n o w n m ar ke ti n g an d s tu d en t r ec ru it m en t 36 76 http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 9 (page number not for citation purpose) were inconsistent in tone. for example, a6 used both formal and informal language in different parts of the document. each sentence (including bullet points) was analysed and thematically coded through an iterative deductive process. if a sentence related to multiple themes, the sentence was separated into thematically distinct phrases. table 4 is an overview of the themes and frequency of occurrence, i.e. the number of phrases or sentences. rules (including guidelines) took the form of commands (i.e. assigned actions and/ or responsibilities), restrictions (i.e. prohibited activities), boundaries (i.e. defined concepts and/or responsibilities), monitoring activities and the consequences for non-compliance. commands were the most predominant form of rule in each of the artefacts, and were expressed with the speech function of demand, a form of deontic modality which signifies the ‘…“author’s” commitment to obligation [or] necessity’ (fairclough 2003, p. 168). commands were articulated in one of the following ways (based on fairclough 2003). • prescriptions, usually positive imperative clauses, e.g. ‘complete a risk assessment’. • proscriptions, usually negative imperative clauses, e.g. ‘do not post anonymously or use pseudonyms’. • modalised demands, which lie between prescriptions and proscriptions, and may contain markers such as modal verbs, e.g. ‘staff must adhere to the data protection act’. • interrogative demands (i.e. question-requests), e.g. ‘does the learning tool comply with accessibility standards?’ • statements with deontic modality, e.g. ‘all university social media activities are subject to uk legislation’. figure 3 summarises the format and number of commands for each of the artefacts. two types of modal markers were analysed: modal verbs and participial adjectives (namely ‘required’, ‘supposed’ and ‘allowed’). however, participial adjectives were seldom used. using the halliday and matthiessen’s (2013) modal classification model, the level and frequency of modal verbs were analysed, as summarised in table 5. this was to analyse the level of authority present within the texts which can indicate the level of compliance required. the colours in the table highlight the level of modulation most frequently used within each of the artefacts, with a7 and a8 (from the same institution) containing the greatest use of high-level modal verbs. all phrases/sentences relating to commands, restrictions and boundaries (crb) were analysed and thematically coded. a very broad range of themes were prevalent across the artefacts, as evidenced in table 6. the total frequencies present a skewed perspective as each of the artefacts varies greatly in length. therefore, the most frequent crb category is highlighted for each artefact, demonstrating two important findings. firstly, the artefacts vary in terms of the breadth of crb themes included within the texts, with some artefacts focussing more on some areas than others. for example, artefact a3 (acceptable use policy) contained more statements relating to security, malicious acts (i.e. purposeful malicious behaviours such as distributing viruses) and password management than a1 (guidelines on the use of external software/services for learning) which contained more statements relating to data protection and the selection of technologies. secondly, the table reveals themes that are http://dx.doi.org/10.25304/rlt.v27.2284 s. berry 10 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) t ab le 4 . c o n te n t th em es a n d f re q u en cy . t he m e a 1 a 2 a 3 a 4 a 5 a 6 a 7 a 8 a 9 t ot al p u rp o se ( o f th e ar te fa ct ) 3 0 4 2 3 4 2 1 1 20 e n co u ra ge m en t (t o u se t ec h n o lo gi es ) 2 0 2 2 1 1 0 0 0 8 b en efi ts ( o f te ch n o lo gi es ) 5 0 2 6 2 2 6 1 3 27 r is k s (o f te ch n o lo gi es ) 1 0 8 2 2 1 8 0 1 23 r u le s c o m m an d s 20 1 63 41 80 77 57 27 47 41 3 r es tr ic ti o n s 1 2 9 1 5 7 3 2 10 40 b o u n d ar ie s 0 0 1 5 15 3 7 4 1 36 m o n it o ri n g 0 0 5 1 3 8 3 0 0 20 c o n se q u en ce s 0 0 5 2 4 4 0 1 0 16 t ec h n ic al r es tr ic ti o n s 0 0 2 0 0 0 0 0 0 2 in te re st s 3 1 9 5 18 5 6 2 2 51 s o u rc es o f s u p p o rt /g u id an ce 5 1 2 1 9 5 2 6 15 46 t ot al 40 5 11 2 68 14 2 11 7 94 44 80 70 2 http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 11 (page number not for citation purpose) absent within the artefacts. for example, only artefacts a1, a4 and a8 refer to accessibility requirements. discussion how do institutional texts distinguish between institutional and non-institutional technologies? no artefact explicitly defined nor stated the distinguishing characteristics between institutional and non-institutional technologies. definitions were provided for the term ‘social media’ in artefacts a4, a5, a6 and a8; however, no distinction was made between institutional and non-institutional social media tools (such as discussion forums and blogs on institutional virtual learning environments (vles)). artefact a1 was a dedicated guidance document on the use of external services or software for learning, but did not define ‘external’. the artefact included a link to a list of ‘provided’ software and services, implying institutional technologies, although the page was no longer available. verbs were analysed for any related phrases, revealing that the terms maintained, provided, owned, offered, approved and administered were used in reference to internal, central or university technologies, i.e. institutional technologies. the term ‘ external’ was used in relation to technologies in five artefacts (a1, a2, a3, a6 and a7), which implies non-institutional technologies, and the terms ‘hosted’ and ‘not supported’ were used in association with these technologies. table 5. frequency of modal verbs. a1 a2 a3 a4 a5 a6 a7 a8 a9 totals high (must,ought to, need to,) 1 0 18 5 29 11 31 90 2 106 median (will, would, should, shall) 8 0 35 29 42 45 20 8 15 202 low (could, can/not, may, might) 7 1 22 8 24 31 25 6 15 139 total 16 1 75 42 95 87 76 23 32 447 a1 0 prescribe f re q u e n cy proscribe modalised interrogative statement 10 20 30 40 50 60 70 80 a2 a3 a4 a5 a6 artefact number a7 a8 a9 figure 3. linguistic style of commands. http://dx.doi.org/10.25304/rlt.v27.2284 s. berry 12 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) the texts therefore suggest that what distinguishes institutional from non-institutional technologies, is not whether the technology is used or promoted by members of the institution, but whether the technology is owned, provided, approved or managed in some way, possibly by a central department. this is consistent with weller’s (2010) concept of ‘centralised’ and ‘decentralised’ technologies. what responsibilities are assigned to teaching staff when non-institutional technologies are utilised for educational purposes? two artefacts explicitly stated that the policy/guidance applied to academic staff, and two explicitly stated that they applied to all staff. the remaining five artefacts did table 6. frequency of commands, restrictions and boundaries (crb), categorised by theme. artefact no. theme a1 a2 a3 a4 a5 a6 a7 a8 a9 total in/appropriate use/conduct/content 1 8 7 27 26 10 10 16 105 account/content mgmt 8 9 13 3 9 42 moderating and responding 2 6 11 5 1 8 33 data management (incl.backup) 3 1 6 2 1 3 16 data protection 4 3 4 5 2 12 3 33 privacy 1 1 2 9 3 6 7 29 security (general) 16 1 1 4 1 1 24 password management 9 2 11 confidentiality 1 3 1 2 1 8 malware 7 7 spam 1 1 2 comply with policies 8 8 7 4 3 4 1 35 copyright and lpr 1 2 13 1 3 3 23 comply with legislation 1 2 2 2 3 2 12 freedom of information 1 1 2 aims/purpose 2 2 2 3 4 13 internal systems 2 2 3 1 5 13 assigning responsibility 1 2 6 2 1 1 13 branding 2 2 2 2 1 3 12 terms and conditions 1 4 2 7 selection (of technologies) 4 1 1 6 identity 1 1 1 2 5 registering/access 1 1 1 1 4 accessibility 1 1 1 3 support/maintenance 1 1 other 4 4 6 16 14 4 3 20 71 malicious acts 13 1 5 3 3 2 27 total 26 3 80 54 121 90 75 40 68 557 http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 13 (page number not for citation purpose) not explicitly state the intended audience, but indicated that they applied to all staff through use of the term ‘staff ’ in the artefact title or throughout the text. over 500 rules were identified in the artefacts analysed, many of which related directly and explicitly to issues of safety, security, privacy and data protection (see table 6). rules and responsibilities concerning these risks were diverse and in some cases complex. for example: • to conduct risk assessments (to include assessment of copyright and data protection risks, compliance with institutional policies and potential impact on institutional reputation), • to identify and assess privacy implications, • to ensure that online accounts are adequately secured (e.g. through appropriate password selection and maintenance), • to determine whether users will receive spam mail, • to set up and manage accounts and data, • to monitor and intervene in circumstances of inappropriate behaviour or activity and • to inform students of the risks and seek consent. this raises a number of questions concerning the role and duties of teaching staff, and the digital literacies that would be necessary to fulfil these requirements. however, it is important to note that a number of artefacts make reference to institutional sources of support, such as it and learning technology departments, legal services and library teams. despite the quantity and breadth of the rules identified, there were some notable omissions. three artefacts did not contain any rule or reference to security risks, and two artefacts only made one statement relating to security (specifically concerning individual responsibilities to ensure accounts are not compromised, and to change passwords when members leave the institution). encrypting data is widely recognised as an essential requirement for a minimal level of security (alim et al. 2017). nevertheless, reviews of educational technologies have found insufficient encryption, which can put devices and data at significant risk (kelly et al. 2018a). despite these reports, no artefact contained any rule or reference to data encryption. a previous study which explored how students perceive requests to use personal devices for classroom learning (author 2019) found that some participants assumed or expected that learning technologies would have been checked for malware. however, only one artefact (a3, produced by an it department) made any reference to malware detection and prevention. furthermore, ethical considerations discussed in the literature (such as the use of student data or content by third parties for commercial gain) were absent within the artefacts. the degrees of obligation required were suggested by the lexical choices and use of modulation within the artefacts. the term ‘degrees of obligation’ refers to the ‘…degrees of obligation to act’ (martin 1992, p. 369), and can indicate the level of authority present within the text. for example, high-level modal verbs, such as ‘must’, signify a necessity to act and establish a high level of authority, whereas low-level modal verbs, such as ‘could’, approve or recommend certain behaviours or actions. most artefacts expressed a median level of authority, but two (a7 and a8, produced by the same institution) opted for a high level of authority, suggesting that a greater degree of compliance is required. http://dx.doi.org/10.25304/rlt.v27.2284 s. berry 14 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) artefacts a1, a4, a6, a7 and a9 contained question-requests (see figure 3), that is, demands which are interrogative in their grammatical mood (fairclough 2003), such as: • is the tool suitable for the intended audience, or does it broaden the audience beyond what is appropriate? • how will the service provider capture, retain and process users’ data? • could the institutional vle or other university provided services fulfil the requirements? • does the tool comply with accessibility standards? this approach may be used to persuade rather than dictate responsibilities, and may therefore be considered more collegial in nature (as found in an analysis of social media policies conducted by mcneill 2012). however, according to ervin-tripp (1976, as cited in flöck 2016), question-requests enable the possibility of non-compliance. furthermore, there is no indication of what would be appropriate answers to these questions. therefore, if statements are advisory rather than mandatory, could non-institutional technologies be used without the appropriate quality assurance processes or safeguards being in place? in addition, no artefact clarified the procedures to fulfil the requirements concerning security, privacy and data protection. for example, what steps would you take to determine if the service/app complies with the data protection act? what information would be required and how would this be sourced? what if the information was incomplete or unclear? as outlined within the literature review, there are surreptitious ways that apps and services can collect data, thus demonstrating that this would be very challenging to accomplish, particularly by staff with limited technical and legal knowledge in this field. to what extent do teaching staff have autonomy to use non-institutional technologies? forty restrictions were identified which limit how and when non-institutional technologies can be used. these restrictions include prohibiting the use of non-institutional technologies where equivalent features and functions are provided by institutional technologies, prohibiting use for summative assessment and prohibiting staff from requiring students to use non-institutional technologies. however, the artefact did not clearly define a ‘requirement’ and thus could be open to interpretation. for example, if a lecturer asked students to watch youtube videos to prepare for a class, would students interpret this to be a requirement, and if so would this be classified as a prohibited activity (assuming that youtube would be classified as a non-institutional technology)? artefact a3 also expressly stated that technical procedures were in place to prevent staff from installing software. a number of commands were identified (n = 33) which required that staff obtain approval or consent from different individuals or groups including marketing and it services, line managers and students. this included permission to use (a3, a4, a7, a8), create accounts or profiles (a5, a6, a7, a8) or post-information or content (a4, a5, a6, a7, a8). this is consistent with the findings reported in similar studies (mcneill 2012). despite the number of commands, restrictions, boundaries and statements establishing monitoring procedures and consequences for non-compliance, the use of external technologies and social media was actively encouraged in five artefacts, http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 15 (page number not for citation purpose) and some artefacts stated the benefits of these technologies more so than the risks (see table 4). in conclusion, some institutional artefacts support greater levels of staff autonomy than others. artefacts that were more advisory in tone may support greater autonomy by encouraging rather than enforcing specific actions. those that presented rules in the format of question-requests could be seen to promote autonomy by enabling the reader to take an active role in determining appropriate technologies and appropriate practices with those technologies. however, question-requests could be perceived by staff as a barrier rather than an enabler of autonomous practice, if they feel that they do not have sufficient knowledge, capabilities or support to ‘answer’ the questions presented. some artefacts could serve to constrain autonomy by requiring, limiting or prohibiting particular uses or behaviours (as discussed above). in addition, the breadth and complexity of the rules contained within these artefacts could suppress autonomy, where staff may feel that they do not have the time, capabilities or resources to comply with the rules. in relation to this, there were 16 statements referring to the consequences for non-compliance and 20 statements related to institutional monitoring, which either occur at a technical or human level, to ensure staff adhere to the rules. these statements may serve to reinforce control through fear of repercussions or monitoring, and thus constrain autonomy. however, institutions may consider this control to be necessary in order to protect the institution, its members and its assets, particularly given the challenges and risks posed by non-institutional technologies. conclusion to explore how uk heis are addressing the risks of educational technologies identified throughout the literature, the study assessed 100 artefacts produced by 30 uk heis. nine artefacts (obtained from seven heis) were then selected for in-depth analysis. only one artefact was a dedicated guidance document relating to the use of non-institutional technologies for learning. six artefacts were focussed on social media technologies. however, there are many third-party learning technologies, such as subject-specific apps and software, content production tools and quizzing/polling services which may not be classified as ‘social media’. this raises the question as to whether there are sufficient policies and guidelines in place concerning the diverse range of non-institutional technologies that can, and are being used for teaching, learning and assessment in higher education. over 500 rules were analysed, and the study found that a broad range of responsibilities were assigned to staff (including teaching staff) concerning the selection and use of technologies. whilst the limitations of data analysis are acknowledged (i.e. conducted by a single researcher, external to the institutions), it can be argued that these artefacts are contributing to defining the role of teaching staff in contemporary heis. mishra and koehler’s tpack (technological pedagogical content knowledge) framework (2006) establishes that technological knowledge is a key component of teachers’ knowledge required for effective technology integration in education. this study has also identified institutional expectations concerning teachers’ technological knowledge. many of these responsibilities and associated duties require sufficient time, specialist technical and/or legal knowledge and digital capabilities. there is some evidence that existing institutional policies and guidelines are attempting to address some of the risks of educational technologies as reported within the literature. however, statements or standards relating to security risks and http://dx.doi.org/10.25304/rlt.v27.2284 s. berry 16 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) protective measures were absent or inadequate within the artefacts assessed. in addition, the procedures to fulfil these requirements were not sufficiently described. for example, staff were assigned responsibilities to ensure compliance with data protection legislation; however, no artefact stated how this should be achieved. furthermore, many of these requirements could be difficult to fulfil given the surreptitious ways that data can be collected by apps and services, the difficulties in obtaining information about these technologies due to ambiguous and verbose privacy policies (reidenberg et al. 2015) and the technical and legal knowledge required to ascertain compliance. this may suggest that these artefacts are not establishing achievable standards, but may instead serve to transfer the accountability onto individual users (including teaching staff) and limit the liability of the institution. a number of contradictions were noted within and across artefacts, including those produced by the same institution. according to tan (2009), differences and contradictions in policies may result in a misalignment between the intended meaning and implementation in practice. however, this study did not explore how these texts are perceived and implemented in practice or whether existing practices relating to the evaluation and selection of learning technologies are sufficient to identify and minimise significant risks. furthermore, the study did not explore whether these texts are aligned with student expectations concerning risks and safeguards, or whether they sufficiently address issues concerning students’ digital rights. these are areas recommended for further research. in conclusion, the main aim of this study was to explore how institutions are addressing the risks and ethical considerations of non-institutional technologies, when appropriated for educational purposes. conole (2013) argues that there must be clear policies and guidelines to support effective institutional coordination and individual experimentation with digital technologies. this study highlights significant inadequacies in institutional policies and guidelines, which could result in ineffective practice in the selection and use of these technologies. a potential consequence could be that technologies are being used without appropriate quality assurance processes and safeguards in place (such as data encryption). additionally, educational use of these technologies could be discouraged or limited due to the complexities of the responsibilities assigned to users, and insufficient clarity within institutional documentation regarding the quality standards, procedures and terminology used. however, policies, guidelines and other institutional documents are only one way to support the effective and appropriate use of non-institutional technologies. hall has asserted that academics must ‘…have a critical or ethical lens through which to critique the nature of the technologies that they use and re-purpose inside the university’ (hall 2013, p. 52). thus, this problem also needs to be addressed in the development of digital literacies to empower staff and students to make informed decisions about their devices, data and digital profiles. competing interests the author certifies that there are no relationships or involvement with any organisations which could be considered as a conflict of interest. acknowledgements this research was undertaken as part of the phd in e-research and technology enhanced learning in the department of educational research at lancaster university. http://dx.doi.org/10.25304/rlt.v27.2284 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 17 (page number not for citation purpose) the author would like to acknowledge the contribution of tutors and peers in supporting the development of this study and its report as an assignment paper. the author would also like to thank the two anonymous reviewers for their valuable comments. references akande, a. o. & van belle, j. (2016) ‘the use of software as a service by students in higher education institutions: a systematic literature review’, acm press, pp. 1–6. doi: 10.1145/2971603.2971604. alim, f., et al., (2017) ‘spying on students: school-issued devices and student privacy’, electronic frontier foundation (eff), [online] available at: https://www.eff.org/wp/ school-issued-devices-and-student-privacy arpaci, i., kilicer, k. & bardakci, s. (2015) ‘effects of security and privacy concerns on educational use of cloud services’, computers in human behavior, vol. 45, pp. 93–98. doi: 10.1016/j.chb.2014.11.075. author (2019) ‘an investigation of medical students’ perception and willingness to use personal digital devices (pdds) for classroom learning’, manuscript in preparation. aymerich-franch, l. & fedele, m. (2014) ‘students’ privacy concerns on the use of social media in higher education’, in cutting-edge technologies and social media use in higher education, eds v. benson & s. morgan, igi global, hershey, pa, pp. 54–75. boag-munroe, g. (2004) ‘wrestling with words and meanings: finding a tool for analysing language in activity theory’, educational review, vol. 56, no. 2, pp. 165–182. doi: 10.1080/0031910410001693254 bowen, g. a. (2009) ‘document analysis as a qualitative research method’, qualitative research journal, vol. 9, no. 2, pp. 27–40. doi: 10.3316/qrj0902027 brandtzaeg, p. b., pultier, a. & moen, g. m. (2018) ‘losing control to data-hungry apps: a mixed-methods approach to mobile app privacy’, social science computer review. doi: 10.1177/0894439318777706 chatzigavriil, a. et al., (2014) 2014 survey of technology enhanced learning: case studies, [online] available at: http://www.ucisa.ac.uk/groups/dsdg/asg/~/media/7bcb3f2ff0e141a79a66bc87ddb34a14.ashx coffey, a. (2014) ‘analysing documents’, in the sage handbook of qualitative data analysis, sage, london, united kingdom, pp. 367–379, [online] available at: http://methods.sagepub.com/book/the-sage-handbook-of-qualitative-data-analysis/n25.xml common sense (n.d.) standard privacy report questions, [online] available at: https://privacy. commonsense.org/resource/standard-privacy-report-questions conole, g. (2013) designing for learning in an open world, springer, new york. conole, g. & alevizou, p. (2010) a literature review of the use of web 2.0 tools in higher education, [online] available at: https://www.heacademy.ac.uk/system/files/conole_alevizou_2010.pdf fairclough, n. (1993) discourse and social change, polity press, cambridge. fairclough, n. (2001) ‘critical discourse analysis as a method in social scientific research’, in methods of critical discourse analysis, eds r. wodak & m. meyer, sage, london, pp. 121–138. fairclough, n. (2003) analysing discourse: textual analysis for social research, routledge, london. farahmand, f., yadav, a. & spafford, e. h. (2013) ‘risks and uncertainties in virtual worlds: an educators’ perspective’, journal of computing in higher education, vol. 25, no. 2, pp. 49–67. doi: 10.1007/s12528-013-9067-5 flöck, i. (2016). requests in american and british english: a contrastive multi-method analysis, john benjamins publishing company, amsterdam. gadd, e. & weedon, r. (2017) ‘copyright ownership of e-learning and teaching materials: policy approaches taken by uk universities’, education and information technologies. doi: 10.1007/s10639-017-9583-4 http://dx.doi.org/10.25304/rlt.v27.2284 http://doi.org/10.1145/2971603.2971604 https://www.eff.org/wp/school-issued-devices-and-student-privacy https://www.eff.org/wp/school-issued-devices-and-student-privacy http://doi.org/10.1016/j.chb.2014.11.075 http://doi.org/10.1016/j.chb.2014.11.075 http://doi.org/10.1080/0031910410001693254 http://doi.org/10.1080/0031910410001693254 http://doi.org/10.3316/qrj0902027 http://doi.org/10.1177/0894439318777706 http://doi.org/10.1177/0894439318777706 http://www.ucisa.ac.uk/groups/dsdg/asg/~/media/7bcb3f2ff0e141a79a66bc87ddb34a14.ashx http://www.ucisa.ac.uk/groups/dsdg/asg/~/media/7bcb3f2ff0e141a79a66bc87ddb34a14.ashx http://methods.sagepub.com/book/the-sage-handbook-of-qualitative-data-analysis/n25.xml http://methods.sagepub.com/book/the-sage-handbook-of-qualitative-data-analysis/n25.xml https://privacy.commonsense.org/resource/standard-privacy-report-questions https://privacy.commonsense.org/resource/standard-privacy-report-questions https://www.heacademy.ac.uk/system/files/conole_alevizou_2010.pdf https://www.heacademy.ac.uk/system/files/conole_alevizou_2010.pdf http://doi.org/10.1007/s12528-013-9067-5 http://doi.org/10.1007/s10639-017-9583-4 http://doi.org/10.1007/s10639-017-9583-4 s. berry 18 citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 (page number not for citation purpose) garrett, b. m., macphee, m. & jackson, c. (2013) ‘evaluation of an eportfolio for the assessment of clinical competence in a baccalaureate nursing program’, nurse education today, vol. 33, no. 10, pp. 1207–1213. doi: 10.1016/j.nedt.2012.06.015 hall, r. (2013) ‘educational technology and the enclosure of academic labour inside public higher education’, journal for critical education policy studies, vol. 11, no. 3, pp. 52–82. [online] available at: http://www.jceps.com/archives/437 halliday, m. a. k. & matthiessen, c. m. i. m. (2013) halliday’s introduction to functional grammar, [online] available at: https://www.routledge.com/hallidays-introduction-tofunctional-grammar-4th-edition/halliday-matthiessen/p/book/ 9780203431269 hardré, p. l. (2016) ‘when, how, and why do we trust technology too much?’, in emotions, technology, and behaviors, eds s. y. tettegah & d. l. espelage, elsevier, pp. 85–106, [online] available at: http://linkinghub.elsevier.com/retrieve/pii/b9780128018736000054 herold, b. (2018) ‘how (and why) ed-tech companies are tracking students’ feelings’, education week, vol. 37, no. 36, pp. 14–15. hesa (n.d.) student numbers by he provider and subject of study, [online] available at: https://www.hesa.ac.uk/data-and-analysis/students/whos-in-he janks, h. (1997) ‘critical discourse analysis as a research tool’, discourse: studies in the cultural politics of education, vol. 18, no. 3, pp. 329–342. doi: 10.1080/0159630970180302 kelly, g., et al., (2018a) 2018 state of edtech security survey, common sense, san francisco, ca, [online] available at: https://www.commonsense.org/education/articles/ 2018-state-of-edtech-security-survey kelly, g., graham, j. & fitzgerald, b. (2018b) 2018 state of edtech privacy report, common sense, san francisco, ca, [online] available at: https://www.commonsense.org/education/ sites/default/files/tlr-blog/cs-state-of-edtech-privacy-report.pdf lenartz, a. j. (2012). ‘establishing guidelines for the use of social media in higher education’, in misbehavior online in higher education, eds l. a. wankel & c. wankel, emerald, bingley, uk, pp. 333–354. lindh, m. & nolin, j. (2016). ‘information we collect: surveillance and privacy in the implementation of google apps for education’, european educational research journal, vol. 15, no. 6, pp. 644–663. doi: 10.1177/1474904116654917 lorenz, b., kalde, k. & kikkas, k. (2012) ‘trust and security issues in cloud-based learning and management’, in advances in web-based learning – icwl 2012, eds e. popescu et al., springer berlin heidelberg, berlin, heidelberg, pp. 99–108. lorenz, b., sousa, s. & tomberg, v. (2013) ‘privacy awareness of students and its impact on online learning participation – a case study’, in open and social technologies for networked learning, eds t. ley et al., springer berlin heidelberg, berlin, heidelberg, pp. 189–192. martin, j. r. (1992) ‘macro proposals: meaning by degree’, in discourse description: diverse linguistic analyses of a fund-raising text, eds w. c. mann & s. thompson, benjamins, amsterdam, pp. 359–396. mcneill, t. (2012) ‘“don’t affect the share price”: social media policy in higher education as reputation management’, research in learning technology, vol. 20, no. suppl. 1, p. 19194. doi: 10.3402/rlt.v20i0.19194 miller, k. w., voas, j. & hurlburt, g. f. (2012) ‘byod: security and privacy considerations’, it professional, vol. 14, no. 5, pp. 53–55. doi: 10.1109/mitp.2012.93 mishra, p. & koehler, m. j. (2006) ‘technological pedagogical content knowledge: a framework for teacher knowledge’, teachers college record, vol. 108, no. 6, pp. 1017–1054. [online] available at: https://www.learntechlib.org/p/99246/?nl=1 moher, d., et al., (2009) ‘preferred reporting items for systematic reviews and meta-analyses: the prisma statement’, plos medicine, vol. 6, no. 7, p. e1000097. doi: 10.1371/journal. pmed.1000097 newland, b., martin, l. & ramsden, a. (2011). ‘esubmission – uk policies, practice and support’, proceedings of the european conference on e-learning, 2011, brighton, uk. http://dx.doi.org/10.25304/rlt.v27.2284 http://doi.org/10.1016/j.nedt.2012.06.015 http://www.jceps.com/archives/437 https://www.routledge.com/hallidays-introduction-to-functional-grammar-4th-edition/halliday-matthiessen/p/book/​9780203431269 https://www.routledge.com/hallidays-introduction-to-functional-grammar-4th-edition/halliday-matthiessen/p/book/​9780203431269 http://linkinghub.elsevier.com/retrieve/pii/b9780128018736000054 https://www.hesa.ac.uk/data-and-analysis/students/whos-in-he http://doi.org/10.1080/0159630970180302 https://www.commonsense.org/education/articles/​2018-state-of-edtech-security-survey https://www.commonsense.org/education/articles/​2018-state-of-edtech-security-survey https://www.commonsense.org/education/sites/default/files/tlr-blog/cs-state-of-edtech-privacy-report.pdf https://www.commonsense.org/education/sites/default/files/tlr-blog/cs-state-of-edtech-privacy-report.pdf http://doi.org/10.1177/1474904116654917 http://doi.org/10.3402/rlt.v20i0.19194 http://doi.org/10.1109/mitp.2012.93 https://www.learntechlib.org/p/99246/?nl=1 http://doi.org/10.1371/journal.pmed.1000097 http://doi.org/10.1371/journal.pmed.1000097 research in learning technology citation: research in learning technology 2019, 27: 2284 http://dx.doi.org/10.25304/rlt.v27.2284 19 (page number not for citation purpose) o’connell, c. (2015) ‘close-up examination of discourses associated with global university rankings: counter-narratives in uk policy context: discourses associated with gurs’, higher education quarterly, vol. 69, no. 3, pp. 279–294. doi: 10.1111/hequ.12072 polonetsky, j. & tene, o. (2014) ‘who is reading whom now: privacy in education from books to moocs’, vanderbilt journal of entertainment & technology law, vol. 17, no. 4, pp. 927–990. [online] available at: https://heinonline.org/hol/landingpage?handle=hein. journals/vanep17&div=31&id=&page= pomerantz, j., hank, c. & sugimoto, c. r. (2015) ‘the state of social media policies in higher education’, plos one, vol. 10, no. 5, p. e0127485. doi: 10.1371/journal.pone.0127485. purvis, a., rodger, h. & beckingham, s. (2016) ‘engagement or distraction: the use of social media for learning in higher education’, student engagement and experience journal, vol. 5, no. 1. doi: 10.7190/seej.v5.i1.104 raiman, l., antbring, r. & mahmood, a. (2017) ‘whatsapp messenger as a tool to supplement medical education for medical students on clinical attachment’, bmc medical education, vol. 17, no. 1. doi: 10.1186/s12909-017-0855-x regan, p. m. & jesse, j. (2018) ‘ethical challenges of edtech, big data and personalized learning: twenty-first century student sorting and tracking’, ethics and information technology. pp. 167–179. doi: 10.1007/s10676-018-9492-2 reidenberg, j. r., et al., (2015) ‘disagreeable privacy policies: mismatches between meaning and users’ understanding’, berkeley technology law journal, vol. 30, p. 39. [online] available at: https://www. jstor.org/journal/berktechlawj?refreqid=excelsior%3a58e6bcb3a0072224316cfbddc06200f1 saarinen, t. (2008) ‘position of text and discourse analysis in higher education policy research’, studies in higher education, vol. 33, no. 6, pp. 719–728. doi: 10.1080/03075070802457090 shon, h. & smith, l. (2011) ‘a review of poll everywhere audience response system’, journal of technology in human services, vol. 29, no. 3, pp. 236–245. doi: 10.1080/15228835.2011.616475 tan, k. (2009) ‘variation theory and the different ways of experiencing educational policy’, educational research for policy and practice, vol. 8, no. 2, pp. 95–109. doi: 10.1007/ s10671-008-9060-3 voce, j. (2015) ‘reviewing institutional policies for electronic management of assessment’, higher education, vol. 69, no. 6, pp. 915–929. doi: 10.1007/s10734-014-9813-2 walker, r., et al., (2014) 2014 survey of technology enhanced learning for higher education in the uk, [online] available at: http://www.ucisa.ac.uk/~/media/groups/dsdg/tel%20 2014%20final%2018%20august.ashx weller, m. (2010) ‘the centralisation dilemma in educational it’, international journal of virtual and personal learning environments, vol. 1, no. 1, pp. 1–9. doi: 10.4018/jvple.2010091701 http://dx.doi.org/10.25304/rlt.v27.2284 http://doi.org/10.1111/hequ.12072 https://heinonline.org/hol/landingpage?handle=hein.journals/vanep17&div=31&id=&page= https://heinonline.org/hol/landingpage?handle=hein.journals/vanep17&div=31&id=&page= http://doi.org/10.1371/journal.pone.0127485 http://doi.org/10.7190/seej.v5.i1.104 http://doi.org/10.1186/s12909-017-0855-x http://doi.org/10.1007/s10676-018-9492-2 https://www.jstor.org/journal/berktechlawj?refreqid=excelsior%3a58e6bcb3a0072224316cfbddc06200f1 https://www.jstor.org/journal/berktechlawj?refreqid=excelsior%3a58e6bcb3a0072224316cfbddc06200f1 http://doi.org/10.1080/03075070802457090 http://doi.org/10.1080/15228835.2011.616475 http://doi.org/10.1007/s10671-008-9060-3 http://doi.org/10.1007/s10671-008-9060-3 http://doi.org/10.1007/s10734-014-9813-2 http://www.ucisa.ac.uk/~/media/groups/dsdg/tel%202014%20final%2018%20august.ashx http://www.ucisa.ac.uk/~/media/groups/dsdg/tel%202014%20final%2018%20august.ashx http://doi.org/10.4018/jvple.2010091701 computer-based learning in psychology using interactive laboratories stephen richards department of psychology, university of newcastle-upon-tyne abstract traditional approaches to computer-based learning often focus on the delivery of information. such applications usually provide large stores of information which can be accessed in a wide variety of ways. typical access facilities provided within such applications include boolean search engines and hypermedia (non-linear) browsing. these types of approach often centre on providing human-computer dialogues which are relatively low on interaction. the interactive-laboratory approach, however, aims to limit the quantity of information presented, and instead to provide a highly interactive learning environment. in the field of psychology, users are able interactively to design and deliver a broad range of psychological experiments. this paper details the approach, and describes how it can be used to teach psychology within a university environment. the way in which its effectiveness as a learning tool can be evaluated is also discussed introduction educational technology is currently receiving a great deal of attention and resource, particularly in higher education. the reason for this is that higher-education establishments are being required to deliver ever-increasing numbers of graduates on evertightening budgets. a number of approaches can be taken to counter the problems associated with this situation. some institutions are building ever-bigger lecture theatres to allow greater numbers of students to access the expertise of the teacher. but this approach tends to produce passive learning experiences when compared to the interactive learning which often takes place when only small numbers of students are addressed, and although such lectures are useful, within many subject domains the transmission of appropriate procedural, experimental and practical skills is essential to useful learning experiences. another approach, as readers of this journal will know, centres on the use of computer technology for the dissemination of both skills and knowledge, since computer-based learning applications can provide an interactive medium for learning experiences. many traditional paper-based learning-support materials are essentially static and unreactive: alt j volume 2 number 2 information is usually presented in a predetermined way with a fixed set of pedagogical objectives. although users are often able to override such objectives, interaction with traditional learning materials is limited (see richards, 1993), while interactive computerbased learning environments are able to respond to the individual requirements of users from information elicited through the interaction process. in this way, a learner-centred approach can be adopted. although this does not preclude teacher-centred approaches, it nevertheless allows individual users to tailor their learning experiences. related work a number of programs have been written which make use of an experimental laboratory approach to the teaching of psychology (see, for example, chute, 1989; tversky, 1990; see also cox et al, 1992). several programs of this kind are reviewed in detail in psychology software news, for instance laboratory in cognition and perception (smith, 1990). this typical application allows users to run experiments in a number of areas within the general field of cognition and perception, and when the program is operating in advanced mode, users are able to set a number of parameters prior to running an experiment. results are stored in files which can then be analysed fully at a later date. but this application was designed for relatively primitive computing technology by today's standards, as shown by the fact that although a colour monitor is recommended, all the experiments are designed so that they can be delivered in monochrome. today's technology allows far more ambitious uses of a laboratory approach. in particular, multimedia provides a much broader scope for both the quality and the range of experimental work that can be supported. in any case, the use of multimedia aside, such applications differ from the genuine interactive laboratory in the following ways: • they do not employ multiple pedagogic strategies; • student remediation strategies are usually not embedded; • neither existing laboratory materials nor experiment generators are integrated. objectives the principle objective in this area of my research, therefore, is to create highly interactive learning materials which allow students to become actively engaged in the setting up and delivery of psychological experiments. these experiments are to be controlled through the use of graphical user-interfaces which will allow relatively naive computer users rapidly to develop, deliver and analyse data for a wide range of psychological theories and concepts, the whole being enhanced through the use of multimedia technologies. in addition, appropriate existing software can be integrated into the system. this requires copyright clearance from authors and, of course, some changes may need to be made to such applications to allow their integration into the interactive laboratory, which means obtaining permission to make alterations to the source code. the resulting learning materials are being developed and integrated in close consultation with academic staff engaged in teaching psychology at newcastle university. of course, a major factor in the success of such software is the degree to which students use, enjoy and learn from the approach that has been adopted. simply providing stimulating and interactive learning environments does not guarantee success. methods 19 stephen richards computer-based learning in psychology are required to monitor student take-up, continued use, and attitudes towards the materials and strategies used for its delivery. for these reasons, it is also the objective of my current research to develop extensions to the system for monitoring who uses it, when they use it and for how long. development environment a wide range of environments can be adopted for the development of interactive laboratories. there are, however, a number of features which are potentially extremely important, and which i now outline, as well as the way in which the adopted environment addresses the problems of delivering the necessary facilities. hardware the use of interactive computer-based learning materials in an educational environment represents a delicate balance between cost and power. fortunately, the rapid reduction in the cost/power differentials of desktop computers has meant that appropriate technologies are now widely available. in the psychology department at the university of newcastle, there has been a move towards the use of macintosh networks. the principal student machine, therefore, will be based on new macintosh lc475s which have already been installed in one of the laboratories. these are networked (along with the other macintosh machines in the department) using ethernet and appletalk. the development machine (which will later act as a file server) is a macintosh centris 660av. this machine has a number of advantages for the development of the interactive laboratory which stem from its in-built audio-visual capability, including integral audio/ video digital signal processing hardware and a multi-session double-spin cd-rom drive. apple macintosh machines have long been equipped with audio digital signal processing hardware as standard, something which has given apple an advantage over ibmcompatible hardware in the move towards multimedia. the audio circuitry is capable of producing 16-bit stereo audio with sampling rates of 22.05, 24, 44.1 and 48 khz. the centris 660av, however, has the additional capacity to take in analogue video signals and produce digital video files for delivery from hard disk. north american, uk, and european video standards (ntsc, pal and secam) are all supported. the video source (which may be a camcorder, video cassette recorder, or videodisk) can be linked through either the composite video or the supervhs port. film clips can then be recorded to the hard disk in quicktime digital video format. if the audio channel is also connected, the audio stream is interleaved to form a digital audio/video file. such files can then be delivered in software alone on machines which do not possess the specialized video hardware. as for cd-rom, the multi-session capability of the drive in the centris 660av means that cd-rom discs can be written to more than once. this is a problem for some drives as they are only able to read discs which have been written in a single session, but such multi-session compatibility is essential for handling kodak photocd discs. it is possible to create a photocd disc, then have additional images added at a later date (hence the term m«/f/-sessionj. authoring most of my development work so far for the interactive laboratory has been based 20 altj volume 2 number 2 around hypercard 2.19, though this has recently been upgraded to 2.2. this new release provides significant increases in both power and flexibility, and in particular offers support for colour and multimedia. colour was previously available to developers of hypercard stacks only through the use of third-party executable commands and functions. such facilities have now been included with hypercard in the form of the colortools extensions which allow pict format graphics files to be imported and resized. in addition, colortools allows the colouring of hypercard objects such as buttons, fields, cards, and backgrounds. multimedia support is provided through the integration of quicktime and addmotion tools. addmotion provides 24-bit colour paint, animation and sound extensions. the quicktime extensions allow interleaved digital video/audio files to be controlled from within hypercard. a wide variety of window types and control options can now be simply embedded within an application using the toolkit provided. this means that video, animation and sound can now be controlled (with relative ease) from within hypercard. experiment generators it is envisaged that the use of two experiment generators (mindlab and superlab) will be integrated into the interactive laboratories. the modules that have been developed to date, however, have made use only of superlab {superlab, 1991) which allows psychological experiments to be set up and delivered on a computer. both visual and auditory stimuli can be presented, and reaction times can be recorded accurately through a wide selection of input devices. one of the features of superlab that makes it particularly useful in this type of work is the file formats it produces. two basic types of file are used: script and output. the script file provides all the information superlab needs in order for it to run an experiment, while the output file stores a record of the stimuli presented, user-responses and reaction times. both these files are tab-delimited text files. this means that a superlab-compatible script file can be generated within other environments, then superlab invoked with the generated script file. in addition, although the output file is a microsoft excel file, it simply consists of tab-delimited text and can therefore be imported into a wide range of applications. design, monitoring and pedagogic strategy although an interactive-laboratory approach can be used to teach a variety of scientific subjects, my current research is limited to teaching psychology. the following section therefore describes the general design and implementation of an interactive laboratory for psychology. interface design one of the key problems with developing materials for teaching non-computing disciplines can be the relative lack of computing skills exhibited by some of the potential users. this is a particular problem with first-year students who may have little or no computing experience. naturally, as part of any modern degree course in psychology, the use of computer resources has become a necessity, something which can result in a substantial load for non-computer-literate students. i have therefore adopted a highly 21 stephen richards computer-based learning in psychology graphical approach to user-interface design. the two principle interaction methods within any graphical user-interface are menus and icons. although the interactive laboratory makes use of both these strategies simultaneously, all essential navigation controls are provided iconically at the base of the screen. this allows the completely naive user to be able rapidly to access the essential set of sub-functions without a significant learning curve. a greater range and subtlety of interaction is available through the menus for users who are more experienced. user monitoring the two most common methods used to address the problem of additional assistance for students experiencing learning problems are testing and interaction monitoring. testing can be effective when students obtain immediate feedback on their performance on computer-based tests embedded in the learning environment. the object of this strategy is that it allows students to effect their own remedial action. but associated problems include de-motivation through poor attainment, lack of constructive personal feedback, and, in some cases, an inability to take appropriate remedial action. for these reasons, i have opted for interaction monitoring. this can take many forms, ranging from recording every interaction that the user makes within an application, to keeping a simple record of the length of time spent using a particular part of the application. where all interactions are recorded, a very large amount of information is generated which in itself can obscure the identification of those students who may require remedial action. thus, in the interactive laboratory, a time-monitoring strategy using student log files has been adopted, implemented in such a way that when a teacher accesses student-monitoring jnteractiue lab , tttademt l e g arc1 i • " ( * ' £c atkinson adeil 8 dewson anthony jones andrew jones brian lawson george nott ken peterson steve ragworth julie richards stephen sheldon peter smith helen tomllnson janet xanadu steve use the scrollbar on the "student names" list to bring your name into view, select your name by clicking on it in the l i s t using the mouse. after a moment your password w i l l be requested. type your password and then press the return key. if your password becomes insecure please change it using the "change password" facility. • * ; exft . >' figure 1: the student logging protocol used for interaction monitoring information, he or she is able to identify the modules that have been accessed, by whom, when, and for how long. this means that students who either fail to access a particular module, or who take a disproportionately long time to complete a particular module, can be easily identified. (such information is naturally held in protected files to prevent unauthorized viewing.) 22 altj volume 2 number 2 pedagogic strategies three basic pedagogic strategies are implemented within the interactive laboratory. these involve the delivery of theoretical background information, the provision of instruction for the various software tools, and interactive control of practical sessions. the delivery of background information is based on a simple presentation. this involves several screens of information outlining the key features of research conducted in the area, and any details of how current methodology differs from the original work. however, in order to avoid long periods of reading from screen, information is kept to a minimum and relies on references to further reading. at present, the training approach to the use of software tools supports only a brief introduction to superlab. users are provided with active learning facilities which allow them to toggle between superlab and the instructional materials, thus providing realtime training. in this way, they are led through the process of running experiments interactively within a superlab session. through the use of this technique, even students who have no knowledge of superlab, and who are also low in computing experience, are able successfully to run experiments from within the superlab environment. the educational approach to the laboratory sessions is based on an active learning strategy. naturally, the specific design of each laboratory session is specific to an individual module, since the particular features of each experiment have a determining effect on the facilities and methods employed in the user-interface. even so, the basic strategy is to allow users to set up a wide range of experimental parameters. in this way, students can design their own experiment which they are then able to deliver through the computer. results data is automatically collected into a file of their choice for later statistical analysis. 20 rotation lobs mania. f leoge bight 30 f itlrrer right 30 f tilrror left 60 r leoge right 60 f itlrror 180 f l u g * left 120 ffllrror right so f fllrror right so f l»ooe bight 90 i left t blgh' ssl. f leage right iso f jilrrsr right iso f leoge 180 fr30 fbfoo fbuo fb60 fbibo ft, 120 fbbco fmq0 froo -ossl fr1s0 fmtso f180 dialogue direction type letter gjo_fe left •»• ' t w f • ok instructions: tae aethods for setting up your experiment or* available: dialogue; and edit. h w i oil trials haw* b**n specified using either one of these techniques select the 'load labbutton to start «our experimental session. the dialogue and edit input i{,.aa. \ . . i n**w,j,.i figure 2: the graphical user-interface used in setting up experiments case studies two techniques are used in the creation of laboratory sessions: experiment generators and direct coding in hypercard. the experiment-generator approach involves the creation of 23 stephen richards computer-based learning in psychology interfaces which allow the necessary parameters to be written to a superlab script. superlab is then invoked to deliver the stimuli and collect the data. the direct coding approach can involve the modification and use of existing software and/or the complete coding of new modules in hypercard. in the following outline case studies, both of these approaches are illustrated. mental rotation a laboratory session has been produced which addresses the issues of mental rotation following the work of shepard and metzler (1971). this experiment measures the time taken (in milliseconds) to rotate an image or a mirror image mentally to its original orientation. for the purposes of this experiment, superlab was integrated to deliver the experimental stimuli and collect the resulting data. this was achieved by launching the run-time version of superlab with a generated superlab script file from within a hypercard stack. within the interactive laboratory, superlab script files are created from the parameters set by students. this is achieved through selecting the parameters required for each event within a particular experiment using a graphical user-interface. once the parameters for all events have been set, they are then passed to superlab via the script file. at present, the interactive laboratory supports up to 200 events within a single experimental administration. '4 file edit ehpertment tub i 1 : blocks trials stlmull/euents block! o instructions o1fr30 «2fbr30 o 3fdl60 o4fr60 »5fbl80 o6f1120 o7fbr60 o instructions e 1 «2 »3 o4 o5 06 o7 f nirror right 60 f dln-cr right 90 f leoge right 00 f leoge left 60 f laoge right iso f ttlrrer right iso f laoge iso fbfteo fbr90 fb00 fl60 fr130 fw1s0 f180 oiologus; and edit. u w i a l l t r i a l * hav* t»«n specified using • ithar one of these techniques select the "load lab" button to start your «xp«rlmnta! session. th« 0lalogu« and edit input pr»v1«jf ttitorv, i hov? j figure 3: users can toggle between superlab and the interactive laboratory in order to make modifications to the experiment or access instructional materials short-term memory a different approach from that used for the mental rotation module has been used for the module dealing with short-term memory. this involved making use of an existing set of hypercard stacks which were developed for the delivery of words and nonsense words using digital audio techniques. a full description of the original work can be found in cox, hulme and brown (1992). in order to integrate this set of stacks, it was necessary to supplement the original code so that students could be provided with a brief introduction 24 alt j volume 2 number 2 to the background theory, and online instruction on setting up and running experiments using the module. it was also necessary to embed the code for logging student interaction. future directions although i have discussed the monitoring of student interaction in the context of remediation, such monitoring can of course also be used in the wider context of software evaluation. this provides one of the key issues to be addressed in the near future as the modules go online in the teaching of psychology. in addition, although the use of text, graphics and digital audio has been mentioned above, the full potential of multimedia has not been addressed. multimedia the strict definition of multimedia is the use of a number of presentation modalities within a computing environment, but it is sometimes indeed often taken to mean the use of digital motion video. until very recently the use of digital video was almost prohibitively complex. more recently, however, improvements in hardware and software tools have meant that it can now be usefully employed in a wide range of applications (richards and barker, 1994). within the interactive laboratory, a number of uses for digital video are being investigated. areas currently under investigation include eye-witness testimony and animal behaviour experiments. in the eye-witness testimony work, quicktime movies will be used to deliver digital-video scenarios to subjects from a scenario library held on cd-rom. students will be able to select a variety of approaches to gathering eye-witness testimonies via the computer which may include multiple-choice, free-flow response, structured response, graphical representations of evidence, and simulated identity parades. it is envisaged that animal and human-behaviour experiments will be presented to students through the creation of an integrated behaviour-recording environment. effectively, field studies will be simulated by presenting digital video clips from a resource library, and students will be able to specify the behaviours to be recorded along with the associated codes and key responses. behaviour and timing information can then be recorded by the student while viewing the selected digital video clip. evaluation formal assessment of the impact of the approach described in this paper to teaching psychology will be conducted over the coming year. in addition to measuring the overall impact of the software, attempts will be made to identify the types of student who can benefit from such an approach, and perhaps more importantly, the types of student for whom such an approach may be inappropriate. a number of tools are under development to support this evaluation process. as already mentioned, the logging of student interaction is already in place and it is expected that this will yield useful data. in addition, questionnaires are being formulated which will address the use of such methods from both educationist and student perspectives. 25 stephen richards computer-based learning in psychology conclusion the work described in this paper illustrates the way in which a number of pedagogic approaches can be implemented using computer-based learning materials. these include both relatively passive information presentation and highly active learning strategies in the form of interactive experiments and real-time training. in addition, the work illustrates ways in which computers can be used to augment traditional approaches to teaching psychology. in many instances, traditional teaching methods can no longer support the extremely large numbers of students now faced by the modern lecturer undertaking practical work. the interactive-laboratory approach can facilitate access to active learning experiences. finally, although the focus of the work described here has been within the field of psychology, it is clearly possible to apply the same techniques within a variety of disciplines. multimedia could have a particularly large impact in this regard, allowing the creation of life-like simulations which it otherwise may be difficult or impossible for students to access. acknowledgements i acknowledge the university of newcastle for funding through the teaching initiatives fund; charles hulme for permission to integrate the short-term memory experimenter which has been used in one of the modules; and both academic and technical staff in the department of psychology at newcastle who have provided information, support, suggestions and criticism. references chute, d. (1989), maclaboratory psychology version 1.0 (computer program), devon, pa, usa. cox, r., hulme, c. and brown, g.d.a. (1992), 'stm experimenter: using hypercard and macrecorder in short-term memory experiments', behaviour research methods, instruments, and computers, 24, 4, 575-9. richards, s.m. (1993), end-user interfaces to electronic books, ph.d. thesis, university of teesside. richards, s.m. and barker, p.g. (1994), 'a multimedia trip to mit', proceedings of the 11th international conference on technology and education, london, in press. shepard, r.n. and metzler, j. (1971), 'the mental rotation of three-dimensional objects', science, 171, 701-3. smith, p.t. (1990), review of laboratory in cognition and perception in hammond, n. and trapp, a. (eds), psychology software news, 1, 3, university of york, cti centre for psychology, 1990, 52-4. superlab (1991) (computer program), version 1.4, silver spring, md, usa, cedrus corporation. tversky, b. (1990), experiments in cognitive psychology (computer program), stanford university software distribution center, stanford, ca, usa. 26 interactive learning in biology with photocd and associated software linda baggott and bruce wright school of education, university of exeter this paper describes the pioneering development of flexible-learning courseware in the teaching and learning of cell biology using a novel optical disk-based technology — photocd — and its associated computer software. the use of this format in an educational context for the manipulation of, and interaction with, photographic images in electronic form is presented for the first time. examples of courseware which use photocd, image-enhancement and imagemanagement software in biology education are given, and an evaluation by undergraduates in biology education (student teachers) of these teaching and learning strategies is presented photocd: what is it? photocd is a novel electronic format which uses digital technology to scan colour or monochrome photographs, store them on compact discs, and then display them as highquality images on a tv or on a monitor screen. a technical description of this technology can be found in beisner (1993), bowers (1994) and chen (1993). supporting software is available to access and manipulate the images for use in interactive learning. photocd equipment and materials hardware a-photocd player (kodak pcd 5865) displays images directly from photocd to tv or monitor screen. the player can: • automatically display the images in sequence; • allow two, six or eight seconds' duration of display during an automatic slide-show sequence; • eliminate undesired images from the sequence (skip); • allow random access to images; • pan left and right across the screen image; 62 alt-j volume 3 number i • rotate the image in 90° increments; • display a 2x enlargement of a part of any image; this is achieved by positioning a rectangular cursor on the required part of the image using four direction buttons on the remote control unit; • remember the format of edited images; • restore edited images to their original format; • display a reduced version of the complete image; • play audio cds at high-quality sound. a portable photocd player (kodak pcd 885) enables the system to be used in any teaching room, off-site for outside lectures, presentations etc., or to be taken home where it can be plugged in to a domestic tv set. a colour monitor (philips 8833 mkll 34cm cga) gives higher-quality resolution than is possible with a standard tv for use by students in the classroom for individual or smallgroup learning. a microcomputer is used for dtp and image manipulation. software appropriate software, in addition to desktop publishing packages, is available to be used in conjunction with a computer and a suitable cd-rom drive. photocd access (kodak) allows images to be called onto the computer screen. it can be used to rotate, flip, crop, and cut-and-paste images into other applications. it allows multiple versions of the same image to be opened. shoebox (kodak) is an image-management program which creates thumbnail representations of images. these are then used to create a flexible index using descriptive information for storage and retrieval. this graphical database application allows the user to search, count, mark, and display the digitized images stored on photocd. it can also be used to zoom, crop, copy and paste images into other applications, view a slideshow of images, and store information from other formats. it is possible to print out any edited contact sheet or individual image. photoedge (kodak) is an image-enhancement program which allows detail to be added or deleted from the image: exposure adjustment, sharpening and softening, colour correction, cropping, zooming and panning, and addition of arrows and script. it can be used to access photocd images, display contact sheets, adjust brightness and contrast, and resize and rescale images. images the photocd we used contained micrographs of ultrathin (>1 m) sections of animal cells and tissues taken at very high magnification (xl200 <x20k) through an electron microscope (such instruments are used mainly for research, and these images were taken — linda baggott and bruce wright interactive learning in biology with photocd from the authors' own work). transfer to the photocd format from 35mm monochrome slides was by commercial process via a high-street photographic retailer. why is photocd important for biology education? obviously, photocd is not uniquely suitable for biology education, but visual images are fundamentally important in the study of biology. students' understanding of biological processes requires observation of suboptical structures and their related functions, and first-hand observation and experience traditionally is gained by using a microscope. an increase in our understanding of biological function at the molecular level has occurred with the development and refinement of microscope technology. with few exceptions, however, such instruments are funded by large research grants, and are rarely available for the use of undergraduates. in circumstances where students are not specifically being trained in microscopy, where, as is the case in many general or introductory courses, acquisition of overview or basic level of understanding is required by the course, photocd is an affordable and flexible alternative. practical applications of photocd there are many practical applications of photocd, some of which we list below: • illustration of teaching points during lectures, tutorials or seminars. portable photocd players, handset operated, can be used to display, highlight, enlarge or rotate selected images to illustrate teaching points. • class interaction students use the photocd player in response to questioning, and in discussion. • practical activities students work at practical problems directly on the images. programming sequences of images, to demonstrate processes, measuring and comparing structures, and other aspects of biometry relating structure to function are examples of the use of photocd in the study of microscopic preparations. • presentations students work together or individually to produce presentations of their work. original photographs or photomicrographs, or sequences of these, can be incorporated into text images to communicate students' understanding. • information retrieval image management software is used to set up a visual database for students to question, or to add to with further information about the images. • supported self-study individually, or in groups, students work at courseware which reinforces the subject matter of courses. photocd and associated software as a learning medium on trial it has been demonstrated that students' understanding of biological processes is achieved more quickly when they use interactive learning technologies in conjunction with more traditional teaching methods (huang & aloi, 1991; baggott & wright, 1994). the strength of photocd as a tool in teaching and learning lies in its application for freeing 64 alt-j volume 3 number i the user from technical constraints, while allowing grasp of fundamental concepts. highquality images can readily and repeatedly be studied at a relatively low cost. the 26 students involved in our photocd pilot study were biology education undergraduates at the university of exeter, in the third year of a four-year course. these students are able to provide high-quality feedback on new learning formats as a result of their professional education combined with the variety of backgrounds from which they come to the course: there is a wider age range than is usual in undergraduate courses, and many come to teaching from other professions and occupations. in view of the fact that they are obliged to acquire a breadth and depth of scientific knowledge as well as professional expertise in practice, methods of gaining an accurate overview while encouraging a spirit of enquiry are especially relevant for their education. examples of courseware the photocd images were displayed on a monitor or computer-monitor screen, and the students worked at specific tasks relating to them. various approaches were used in framing the tasks, but all required the students to make very close observations and to exercise their subject knowledge in discussion. the following are representative tasks using the photocd images and software: examples of tasks involving use of a photocd player what is this structure? (image no. 4) with reference to your lecture notes and course handbook, annotate the image with information not only about the visible structures, but also the physiological processes leading to and resulting from, its development. image no. 4 browse through images 20, 23, 43, 44, 45, 50, 56, to find a good example of a golgi 65 lindo ßoggott and bruce wri$it interactive learning in biology with photocd apparatus. use the zoom facility to magnify the organelle, and then make a labelled diagram of it, giving some indication of scale, and stating from which image it came. examples of tasks involving the use of image-enhancement software (photoshop) identify an image which shows a micrograph ofoviductal epithelial tissue. scan and print out an image with clear contrast and good definition. label the features of particular functional significance for this tissue. scan and print out good copies of images 34 (oviduct) and 45 (epididymis). compare and label the common features of these tissues. annotate the copies to indicate the differences in function between the two tissues. examples of tasks involving the use of image-management software (shoebox) set up and print out a contact sheet containing images showing examples of a variety of tissues to be found in the mammalian female reproductive tract. label each image, and provide suitable explanatory captions. search the catalogue for an image showing a clear example ofoviductal epithelium. print out the image, and annotate it to explain the function of the cell types contained in this tissue. student evaluation there is a substantial academic component to the science education course at the university of exeter, part of which is a sixty-hour course on developmental biology. the photocd materials were used during this course to reinforce the students' understanding of the microanatomy of the mammalian reproductive tract. clearly, there is insufficient time for such students to work directly from an electron microscope, and the study of electron micrographs gives limited information. students were given a demonstration of use of the photocd equipment, and were left with written instructions for both hardware and software. they then worked through the courseware in their own time in groups of two to four. none had had any prior experience of working with an electron microscope: their study of biological specimens at the ultrastructural level had been via text-book micrographs and diagrams. they were asked to give their reactions to the photocd materials in terms of perceived advantages and disadvantages, and how their motivation for learning had been affected. all responded positively about the user-friendliness of the hardware, and the documentation for use. they were impressed with the quality of resolution, and the ease with they could move from image to image, pan, zoom and magnify. some reported that work with a light microscope was usually a solitary activity, and they appreciated the collaboration and discussion which occurred when they worked together around a screen. the most valuable outcome for many students, however, emerged from the tasks with the image-enhancement software. there was a considerable sense of empowerment in taking home an accurate record of a microscopic image which they had studied together, and which could be further labelled, annotated and filed for future reference. this was 66 alt-j volume 3 number i particularly apparent when hard copies resulted from tasks which involved questioning the database. the majority of students (88%) felt that their interest in and enthusiasm for the subject had been enhanced by using the photocd materials. some (7%) said that their motivation had been unaffected, and 5% felt that as it was not 'the real thing', it was a pointless exercise. discussion one skill which is traditionally practised in biology is drawing. observation occupies a large proportion of the time spent in studying organisms, their structure, function and inter-relationships, and a key component of any scientific investigation is accurate recording. although drawing has almost entirely been superseded in biological research by photography, it is still highly valued by many biology educators as a means of developing the skill of observation. however, it is a time-consuming skill to acquire, and relies to some extent on a combination of prior experience and natural talent. the traditional technique for drawing specimens from a microscope requires the observer to keep the paper on the side of the drawing hand, use the opposite eye for the microscope eyepiece, and keep both eyes open while looking from the specimen to the drawing (jones et al, 1994). this can be extremely tiring, and unless they can produce reasonably accurate drawings fairly quickly, it is probably an inappropriate use of time for undergraduate students. photocd is a means of liberating them from this difficulty. a previous intervention study, in which the photocd materials were used by half the students on a first-year cell biology course following a pre-test, demonstrated that the post-test scores of those students who used the photocd materials showed a highly significant improvement over the scores of those who had not used the materials (baggott & wright, 1994b). the sample size in both case-studies was small, and other factors such as the background knowledge of the students, or the fact of having done extra study because of their involvement in the trial, may have affected the outcome of their work. nevertheless, in both intervention studies, the photocd materials do appear to have positively affected either the learning of the students, or their motivation for learning. however, further detailed work is needed to make any assertive statements about the value of the materials for learning, and to elucidate the reasons for the improvement. acknowledgements we acknowledge with gratitude the enthusiasm and insight of our technical research assistant, oenone thomas. our work was supported by a grant from the university of exeter teaching development fund. references baggott, l.m. & wright, b. (1994a), the use of interactive video in teaching about cell division', journal of biology (in press). 67 linda boggott and bruce wright interactive learning in biology with photocd baggott, l.m. and wright, b. (1994b), 'photocd in biology education', american biology teacher (submitted). beisner, k. (1993), 'imaging technology in libraries: photocd offers new possibilities', online, 17, 6, 16-18. bennett, h. (1993), 'photocd: a macintosh primer', cd-rom-professional, 6, 4, 93-8, 100-101. bowers, r.a. (1994), 'welcome to the second computer revolution: a beginner's guide to cd-rom', cd-rom-professional, 7, 1, 20-4, 26-30, 32. chen, c.c. (1993), 'photocd and other digital imaging technologies: what's out there and what's it for?', microcomputers for information management, 10, 1, 29-42. huang, s.d. & aloi, j. (1991), 'the impact of using interactive video in teaching general biology', american biology teacher, 53, 5, 281-4. jones, a., reed, r., & weyers, j. (1994), practical skills in biology, london, longman. 68 issues in making courseware exploitable and issues in making exploitable courseware douglas sivrter* and phil siviter** *school of computing information systems and mathematics, south bank university **department of computing, university of brighton part 1 of the paper, 'issues in making courseware exploitable', is about dealing with the legacy of large volumes of incompatible non-integrated courseware which are currently being generated within initiatives such as the teaching and learning technology programme (tltp). we suggest strategies for allowing end-users to apply courseware management techniques belatedly to current courseware developments, thereby offering ways of making the emerging courseware more exploitable than it otherwise would be. part 2 of the paper, 'issues in making exploitable courseware', takes a forward-looking approach which recognizes that future courseware development efforts must pre-empt these problems of incompatibility and non-integration. courseware development must mature to the stage where it makes use of courseware design standards, embraces a host of essential lessons from conventional software development, and recognizes the importance of courseware management issues. introduction there are currently substantial investments being made in courseware development in the uk, in particular in the hèfcs' teaching and learning technology programme (tltp), and the hope which underpins these development efforts is that higher education institutions will eagerly adopt and exploit the freely available courseware which emerges from them. unfortunately, there are many reasons why this is probably going to be an over-optimistic assumption (laurillard et al, 1993). the bottlenecks which can impair the successful exploitation of courseware range from cultural to technical, and are quite diverse. in this paper we discuss some of these bottleneck issues and suggest approaches to dealing with them. we deliberately restrict our discussion to technical bottlenecks, and therefore discuss issues of courseware design, courseware development, and courseware management we first suggest strategies for dealing with a current courseware management challenge, 22 alt-] volume 3 number i i.e. is' it possible to take the current legacy of large volumes of incompatible nonintegrated courseware and somehow make it feel more integrated and compatible than it actually is, thereby enhancing its exploitability? sorting out courseware management 'after the event' is of course the wrong way to proceed (but inevitable if the currently emerging courseware is to be exploited at all). we therefore take a forward-looking approach and suggest that it is essential and inevitable for the courseware-development community to move forwards to a technical culture which embraces courseware design standards (jacobs, 1993) and embraces a host of essential lessons from the mainstream software development world. in such a culture, courseware-management facilities will not be a totally neglected aspect of development as they are today, but will instead be regarded as an essential aspect of the courseware lifecycle. courseware management will also benefit from dealing with courseware objects which are substantially more co-operative than today's 'dumb' courseware. part i issues in making courseware exploitable (tidying up the current mess) large-scale vs. small-scale, courseware vs. lessonware within the uk there are now many courseware-development efforts which have progressed well beyond the small cottage-industry approach, and there is a growing trend towards large-scale developments via consortia of institutions. collaborative developments are increasingly being shown to be more viable (both educationally and economically) than small-scale isolated developments. it is not just the scale of the development activities which has substantially increased; the courseware itself is larger and more sophisticated than it has ever been (a fact which is mostly attributable to advances in computer software and hardware). most previous developments of computerassisted learning material would be more appropriately labelled as lessonware; they usually tackle only a small fraction of what anyone in an academic institution would call a course. by contrast with previous small-scale lessonware products, today's consortiumbased developments are trying to produce large-scale courseware which might embody hundreds of pieces of lessonware. as the size and sophistication of the courseware grows, and as more complex patterns of use emerge, many new problems are encountered and new courseware management requirements emerge. limitations of current authoring tools it is, of course, tempting simply to extrapolate upwards and assume that large-scale courseware can be built out of lots of pieces of small-scale lessonware, an assumption which is fine in principle but complicated in practice. though there are now many wonderful software packages which support the development of small-scale lessonware, there is much less software support for managing the development and delivery at the larger courseware level. authors can (and do) argue endlessly over their favourite development tools, but they are usually just comparing the built-in facilities for making clever things happen on screen. none of these lessonware tools provide ready-made 23 douglas siw'ter and phil siv/ter courseware exploitation sophisticated support for developing and endlessly reconfiguring very large collections of cal modules; nor do they offer sophisticated support for integrating cal resources into computer supported collaborative learning frameworks; nor do they offer sophisticated support for integrating resources developed using a diverse range of software tools and applications. these excellent lessonware tools are simply not geared up to address largescale courseware management issues. problems of diversity most of the consortium-based developments in the uk at this time have realized that offthe-shelf solutions to courseware management requirements are not readily available, and hence they have either resorted to building their own solutions or have sadly neglected the problems altogether. there are now far too many instances of projects reinventing each other's wheels, i.e. they individually discover the same sets of problems, then partly solve them with in-house developments while remaining totally oblivious to anyone else solving the same problems. usually, the in-house solutions have failed to consider the extra complexities of ensuring that their products remain open, and almost no effort has gone into pursuing compatibility with solutions produced by different projects. far from removing technical bottlenecks, we now have a culture of generating them. as more sophisticated courseware becomes more widely available, academic staff will try to experiment with new ways of exploiting it (we hope). we should recognize that if willing and eager university staff (those not suffering too badly from the not-inventedhere syndrome) try to exploit the currently emerging courseware, they will face a range of technical barriers which will probably deter their enthusiasm and make them give up. the current courseware development efforts are producing diverse courseware. diversity in an educational sense (in content and in pedagogic style) is an asset, but accidental diversity in courseware management techniques and in basic software management techniques just translates to bewildering incompatibility for the typical end-user, and is therefore a liability. partial solutions to current courseware management problems the courseware development scene is still very young and has not yet matured to the stage where it collectively embraces useful, non-restrictive standards for design and development. as a result of this, end-users now face large volumes of incompatible nonintegrated courseware from a diverse range of courseware development initiatives. we present a framework called hypercourseware which immediately offers partial solutions to these problems. in the latter part of this paper, we also assert that the hypercourseware framework can be expanded to form the basis for genuinely open courseware which can achieve substantially higher levels of courseware compatibility. the hypercourseware framework hypercourseware is not a new idea; our first attempts at providing a hypercourseware management system date back to a project which started in 1989 (siviter & brown, 1992). a minimal description of hypercourseware structuring principles is as follows: 24 alt-] volume 3 number i • a course is a hierarchical network of topics. • a topic is a collection of educational activities. • an educational activity i s . . . (virtually anything that an author decides): — described educationally e.g. presentations, assignments, assessments, simulations; — technically implemented using anything, e.g. authoring tools, programming languages, software applications (spreadsheets, databases, etc.), multimedia audio-visual applications, away-from-machine activities. the above framework is sufficiently general to allow virtually any courseware structure to be generated, but sufficiently precise to allow software tools to support the manipulation of such structures. a hypercourseware management system is a set of software tools which enables the creation and endless adaptation of courseware structures and which partially or totally automates the creation of interactive views of the courseware structures. a hypercourseware management system therefore enables non-programmers to manipulate large collections of 'lessonware objects' which may have been developed using a diverse range of authoring tools. dealing with the legacy of incompatible courseware a classic scenario within universities is for lecturers to pick and mix from a variety of texts in order to deliver the particular perspective on a subject which they feel is most appropriate. it is a rare occurrence for any lecturer to run a unit totally based on a single set textbook. for courseware to be acceptable to academic staff, it must offer the same kind of pick and mix adaptability that text sources currently provide. academic staff must have realistic options locally to customize courseware or even embark on major reconfigurations of the material the courseware must be capable of evolving. as the simplest of examples, a lecturer might like to edit the aims and objectives section of a piece of courseware the day before she uses it with a new group of students. a more involved example might be a lecturer creating a totally new courseware structure but populating it with many pieces of existing lessonware, i.e. no new development of resources takes place but a new course is created out of existing resources. is this courseware development or courseware management? there is now a national requirement for efforts to be made to bring together the many strands of courseware development. consumer universities (i.e. all of us) need to have much higher level courseware-management facilities which shield them from technical incompatibilities within the courseware. in effect, there needs to be an after-the-event fudge towards integration which is now a necessary but a poor substitute for integrated design. hypercourseware can make some contribution to this belated integration. it offers conceptual support by providing a formalized but non-restrictive framework for courseware structures. it offers practical support in the form of hypercourseware management systems: software tools and templates which directly support end-users in courseware manipulation. 25 douglas s/wter and phil smter courseware exploitation part 2 issues in making exploitable courseware (avoiding a future mess) the previous section discussed after-the-event fudges which are now required in order to allow existing diverse lessonware to be used and re-used in larger courseware structures and hence be more exploitable. this after-the-event fix can at best be described as a shallow integration which relies on a fairly crude management of multiple software applications and the generation of multiple 'educationally friendly' views of available resources. how can this shallow integration be improved upon in future? what can we learn from many years of producing incompatible courseware? the need to learn from the software development world within this discussion, it is useful to regard courseware as just a specialized form of software. obviously, educationists can point out an infinite number of ways in which courseware differs from other forms of software, but there is no escaping the fact that courseware is still software, and there are many lessons which courseware development could and should learn from other areas of software development. the world of commercial software has been forced to address issues of software management, software development, software evolution and maintenance, interoperability of software, etc. the courseware-development community in comparison seems to be barely aware of the problems looming over the horizon. this is tragic, given that an awareness of these problems might have enabled the courseware-development projects to exploit some of the solutions which have emerged within other software development scenarios. courseware standards vs. software standards there will always be a requirement for unrestrained, exploratory development, especially in areas as sophisticated as education, but there are also immense benefits to be derived from standardization in areas which are frankly boring, routine software-development issues. nobody in their right mind would look at a pc in 1994 and say: 'because i don't like windows i would prefer to build my own graphical user interface'. yet similar nonsensical decisions have been made in relative ignorance about a range of software development issues. it is salutary to note that the art of reinventing boring old-fashioned wheels is alive and well within the world of courseware development (and, as many observers have pointed out, because the courseware development world is relatively new and inexperienced, the reinvented wheels frequently turn out to be square). the time is right (indeed, overdue) for a much greater effort, preferably on a national scale, to be directed towards establishing courseware development standards. before, the instructional design experts scream about how prematurely suffocating these standards might be, we would emphasize that the areas best suited for standardization are not concerned directly with instructional design but with the underlying software development issues. the standardization efforts we are advocating have a clear aim of enabling exploratory instructional design rather than constraining it. it must be appropriate for the courseware development community to recognize that it is wasting opportunities to benefit from current experiences from mainstream software 26 volume 3 number / development. furthermore, given that mainstream software development is advancing more quickly than courseware development, it would be preferable specifically to try to close the experience gap by looking forward to emerging software trends. there are many examples, e.g. there are massive increases in software productivity to be gained from exploiting reusable object-based software components. microsoft's object linking and embedding, apple's opendoc, and ibm's distributed system object model are examples of a software-development idea which will have a profound impact on how all software (and therefore also courseware) will be developed. yet it is probably true that only a small minority of courseware developers have any idea of what these ideas represent, and an even smaller proportion are gearing up ready to exploit these inevitable ideas. there are two overlapping areas of concern. one is as mentioned above: what can (or must) courseware development learn from software development? the other is the identification of generic issues purely within courseware development (generic issues which may have only tentative parallels in software development) and the attempt to engender standards for dealing with these generic courseware development issues. moving hypercourseware towards a framework for open courseware one on-going development within our hypercourseware projects has been to map out the courseware development terrain in a way which can draw attention to areas which all courseware development projects have in common, i.e. to identify what are genuinely generic problems associated with courseware management, courseware design, courseware development, and so on. the on-going development of hypercourseware has also been concerned with providing techniques and tools for addressing these generic issues. the hypercourseware framework strives to be totally flexible so that in principle any approach to courseware development can be mapped onto the framework, somehow. as a result of this work we now have: • a conceptual framework for courseware and a vocabulary which can be used a reference model, i.e. any courseware design can be compared with the reference model, if only to verify that the design has, in some way, suitably addressed a known collection of essential issues; • demonstrator products which practically illustrate many courseware design issues; • a collection of tools and templates which genuinely enable courseware developers to avoid being distracted by the mundane reinvention of software-development wheels, and instead to raise their sights to the genuinely difficult aims of good instructional design. current developments our hypercourseware management system is object-based. it has well-defined internal protocols which allow us endlessly to evolve the tools and which allow end-users to evolve courseware. even so, it is fair to say that hypercourseware is not yet a truly open system. it currently provides an elegant way of linking diverse applications (as do a few other systems such as microcosm from southampton university), but these approaches are still 27 douglas smter and phil smter courseware exploitation well short of true openness. much of our current research effort is going towards substantially upgrading the hypercourseware reference model so that it can seriously contribute to a framework for open courseware. in effect, we are now developing distributed hypercourseware. the software-development world pursues goals like 'exceptional ease of use', 'application integration', and 'evolutionary development'. these are all goals which the courseware development world also needs to pursue. some of the most dramatic gains in software development are emerging from strategies which apply object-orientation techniques to client/server systems. by adopting similar strategies for courseware development it should be possible to establish a framework for open courseware based on the idea of cooperative courseware objects. co-operative courseware objects are of course communicating objects which require standard protocols. we believe that the internal protocols for our current hypercourseware system can form the basis for such protocols, hence we are currently engaged in a redesign and redevelopment effort in order to demonstrate this. we also believe that a framework for open courseware is an eventual necessity for the courseware development community. we are tentatively trying to explore what it might look like. references jacobs, g. (1993), 'standards', association for learning technology journal, 1, 1, 2-3. laurillard, d. swift, b. & darby, j. (1993), 'academics' use of courseware materials: a survey', association for learning technology journal, 1, 1, 4-14. siviter, d. & brown, k. (1992), 'hypercourseware', computers and education, 18, 1-3, january, 163-70. 28 (page number not for citation purpose) 1 *corresponding author. email: jonathan.adams@toiohomai.ac.nz research in learning technology 2022. © 2022 j. adams et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 research in learning technology vol. 30, 2022 original research article mixed reality results in vocational education: a case study with hololens 2 jonathan adamsa*, fallyn flavellb and ramari rauretic awhanake ake | academic development, toi ohomai institute of technology, te pūkenga, new zealand; bwhanake ake | academic development, toi ohomai institute of technology, te pūkenga, new zealand; ctiriti and māori relationships, toi-ohomai, nzist | te pūkenga, new zealand (received: 12 may 2022; revised: 24 october 2022; published: 22 december 2022) pedagogy in vocational education is challenged by the increasing availability of mixed reality (mr) technologies. wearable technologies such as the microsoft hololens are being embedded in nursing education programmes in aotearoa new zealand and australia and explored in other subject areas such as construction, architecture, and engineering. empirical research undertaken with staff and students in a nursing programme reinforces and expands on previous research findings about enhanced student motivation, learning and organisational change implications, highlighted in the 2020 special issue of research in learning technology that provided a ‘state of the art’ review of mobile mixed reality (mmr) in education as at 2019. this new research also identifies a number of technical or design constraints that need to be overcome to optimise mr for vocational and higher education use. importantly, it highlights the need for significant improvement in integration of mr platforms into the learner journey so that use of the technology is not confined to professional users and early adopter student cohorts. finally, the authors note the portability that mr headsets offer with the potential to deliver training across multiple campus and work-based learning environments to a consistent standard. keywords: mixed reality; vocational education; tikanga māori; education technology; hololens introduction in aotearoa new zealand’s institutes of technology and polytechnics (itp) sector, blended learning had been increasingly implemented, even prior to the onset of the covid-19 pandemic in early 2020 with some fully online qualifications. in both face to face and blended learning, course content and theory delivery are typically mediated through web-based learning management systems (lmss), with formats that reflect to varying degrees, a conventional classroom delivery. all aotearoa new zealand universities and itps use lms’s (matear 2021; nichols, cited in nanayakkara and mailto:jonathan.adams@toiohomai.ac.nz http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v30.2803 j. adams et al. 2 citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 (page number not for citation purpose) whiddett 2005, p. 181) with the training aspects of vocational education performed either in classroom, or the skills-related environment such as a workshop, clinic or worksite in person by the teacher. in nursing education there is a recognised ‘theory–practice gap’ (scully 2011) that occurs between the necessary course theory and practical clinical experience that also contrasts with learning in the context of real-life situations and alignment of motivation (knowles, holton, and swanson 2015). to explore the potential of mixed reality (mr) providing a stronger connectivity between theoretical and applied content, to ‘prioritise applying theory in context specific and workable ways’ (greenway, butt, and walthall 2019, p. 4), research was undertaken at toi ohomai institute of technology / te pūkenga (toi ohomai / te pūkenga) using science-based content in nursing degrees. this leads on from the findings on measuring student progress in 2019 by nichols, cited in collins and ditzel (2021, p. 181), and student success in australia undertaken by the university of canberra (frost, delaney, and fitzgerald 2020), and desk-based research on the use of mr in nursing education by kim, choi and kim (2021). the research was to serve as a reference point for the student experience and be available for comparisons with similar hololens implementations in three other aotearoa new zealand itp nursing institutions. in addition, the research sought to explore the wider potential for using commercial mr technology in training beyond nursing and health sciences. this new empirical data was particularly timely given that the itp sector is being merged into one new national entity with a significant scope and scale not previously found in the sector. the mr in the reality-virtuality continuum when discussing mr it is important to understand the differences in the visual and immersive experiences between mr, where it sits between augmented reality  (ar)  and virtual reality (vr) and their place on a reality-virtuality continuum described by milgram and kishino (1994) that has different levels of immersion as perceived by the user. the reality-virtuality continuum was revised by (skarbez, smith, and whitton 2021) who proposed a three dimensional taxonomy comprising (1) extent of world knowledge (ewk); (2) immersion and (3) coherence. augmented reality, adds information to augment or feedback into the real-world environment, common examples use head up displays and audio or haptic feedback (touch) in motor vehicles. these augmented feedback systems have knowledge about the real world but a low level of immersion (skarbez, smith, and whitton 2021). in mixed reality, information and objects are within the physical environment of a user, but the virtual objects also possess knowledge about the physical world, the (ewk) dimension (skarbez, smith, and whitton 2021). the virtual objects can react to, give physical cues or be controlled by the user. this gives students ‘the potential to reduce cognitive overload by providing students with perfectly situated scaffolding’ (bower et al. 2014, p. 1). a virtual reality experience completely immerses the student in the artificial environment. these experiences can be used in education to simulate hard to access, or high risk real-world environments such as an operating theatre fire (wunder  et al.  2020). virtual reality (vr) can also create experiences unrelated to real-world environments, typically seen in entertainment and gaming. http://dx.doi.org/10.25304/rlt.v30.2803 research in learning technology citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 3 (page number not for citation purpose) benefits of mr in training the top-ranked benefits of mr identified by (forrester consulting 2021) were training, task instructions and visualisation, remote collaboration and design, and decision making. these benefits are directly relevant to the itp and training sector where delivery of learning is through both applied hands-on experiences that require students to utilise spatial knowledge, and classroom delivery of theory components. key competencies that can be developed with mr include spatial reasoning and situational judgement (papakostas et al. 2021). research on the use of mr in the architectural design industry (osorto carrasco and chen 2021) found a mr-based design can communicate over 20% more information to the client compared with two-dimensional (2d) media. the mr experience has also been found to contribute to nursing student’s clinical reasoning and judgement skills in 2019 research with the hololens (collins and ditzel 2021). it also helped the educators to assess a student’s development so that they were able to customise the student’s learning to further develop those skills. these benefits were supported by (frost, delaney, and fitzgerald 2020) who were also using the same hololens and gigxr content for second year students while in a 3-year bachelor of nursing programme in australia. all the students in their study stated that the experience with virtual standardised patients assisted them in their learning. a scoping review of mr simulation in nursing education (kim, choi, and kim 2021) found the hololens introduced by microsoft in 2016 was the most commonly used device and had a number of advantages for nursing education which requires students to develop both clinical skills and professional judgement. the mr was also able to avoid some of the negative aspects of vr such as lack of spatial awareness that can lead to nausea, or headaches or blurred vision while fully immersed in vr. the review concluded: nursing education simulation requires programs that integrate both nursing skills and situational judgment, which are essential traits for nurses. mr can properly utilize and reproduce virtuality and reality. thus, mr may be effective in nursing education simulations. (kim, choi, and kim 2021, p. 8) method toi ohomai / te pūkenga delivers vocational education to approximately 10 000 students in the bay of plenty and southern waikato regions of aotearoa new zealand. in 2021 toi ohomai / te pūkenga offered around 150 programmes from certificate to master’s degrees, across five campuses. toi ohomai / te pūkenga is one of 17 subsidiaries including all aotearoa new zealand’s itps, which from the beginning of 2023, will be merged into the new national entity te pūkenga, the new zealand institute of skills and technology. the content from a range of vocational courses in architecture, engineering, construction (aec) and nursing was tested in partnership with students and staff to assess its use with the hololens. from initial exploration, research was developed around nursing education, as staff and students became familiar with the technology and its potential. http://dx.doi.org/10.25304/rlt.v30.2803 j. adams et al. 4 citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 (page number not for citation purpose) an example of mr in itp construction courses is converting and presenting what were flat two-dimensional (2d) building plans for a student into three-dimensional (3d) form or a digital twin on a building site, where they can walk through the design at 1:1 scale in the building on its site and foundations (see figure 1). this introduces the student to commercial applications for the hololens 2 that are already used for larger construction and civil works (bim holoview 2022). a review by huang, shakya and odeleye (2019) observed that information (such as the building plans used by toi ohomai / te pūkenga) can be used to detect any conflicts in design in relation to the location or construction. the spatial presentation or presence, in context with the physical world presence provides an extension of the learning environment or place and is a key benefit in using mr in aec subjects, in contrast to an immersive vr experience without the real-world context. hololens 2 headsets are also used in the nursing programme with students undertaking anatomy lessons and viewing standardised patients for clinic reasoning cycle training (see figure 2), using applications licensed from gigxr (putland 2021). the study in this report involved quantitative data collection from 120 first-year science students at toi ohomai / te pūkenga where students were enrolled as part of the bachelor of nursing degree. hololens headsets were used in on-campus tutorials within the nursing degree courses in semester 1 (february–june) and semester 2 (july– november) 2021. student feedback on the impact on their learning was obtained through an online survey instrument where responses were collected after each tutorial. responses were voluntary and anonymously recorded. participants were asked to rate what their thoughts were about the hololens experience on a likert scale of 1 (no change) through to 5 (strongly agree). the question scale displayed for the students is shown in figure a1. an open-ended question at the end allowed capture of some qualitative data. the study provided baseline data about the first-year student experience with plans to survey the students through their 3-year course. field notes figure 1. student view of construction plans with the bimholoview app on hololens 2 at toi ohomai / te pūkenga. source: reproduced with permission from bim holoview. http://dx.doi.org/10.25304/rlt.v30.2803 research in learning technology citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 5 (page number not for citation purpose) (observations) were taken by the lead author who was the educational technologist who supported the nursing tutors with the mr technology. anatomy models (gigxr 2021) were accessed on the hololens 2 within scheduled science tutorial classes, with one tutorial in a laboratory and a second tutorial on the same topic in a classroom setting. virtual anatomy models were viewed by the student that showed the position and structure of organs and systems. the headset displays a 3d model in the user’s real-world environment with a 100-degree field of view (microsoft corporation 2022a). the application models from gigxr could be hidden, scaled and moved or manipulated by the student with a user menu, viewed at 1:1 scale and larger. a total of six headsets were used in each tutorial, typically five headsets were being used simultaneously. the student-controlled menu was virtually attached, showing anatomy sections and available body systems (see figure 3). in figure 3, the student user’s menu is on the right, and another student facing the viewer is also viewing the same anatomy model. the teachers lead the sessions with an appropriate andragogical model (knowles 2015) by setting the guidelines for students who then take on responsibility for their learning. the students’ tutorial exercises required them to identify and locate a range of body systems and anatomy that were listed for them on a whiteboard. students were able to see the room, the virtual models and other participants, and could discuss their experience and ask questions throughout each session with their colleagues and tutor as they proceeded in their self-directed learning objectives. when using hololens 2 headsets the learner engages and manipulates the content with an illusion of presence for the student without being highly immersed (slater 2018). students can control and get feedback with audio and can still interact and collaborate with others in the class or remotely. each tutorial also had at least one additional support person from the toi ohomai / te pūkenga academic support team for teachers to assist students to login and reset sessions to expedite the number of sessions that could run within the scheduled time. this close collaboration and support figure 2. students viewing a standardised holographic patient in clinic reasoning cycle training at toi ohomai / te pūkenga. http://dx.doi.org/10.25304/rlt.v30.2803 j. adams et al. 6 citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 (page number not for citation purpose) between the educational technology teams and nursing teachers was fundamental to the successful incorporation of mr into the nursing programme. embedding cultural competencies with learning technologies aotearoa new zealand’s founding document, te tiriti o waitangi, requires public sector tertiary education organisations to respect and value tikanga māori (the customs and practices of māori). meetings were held with māori cultural advisors, nursing teachers and student support staff prior to the start of the academic year to discuss the implications of using the technology and appropriate tikanga when using the hololens with students. this was seen as particularly important given the nature of the subject of nursing with its close contact with the human body. in addition, nursing education in aotearoa new zealand has included cultural safety as an important focus of the curriculum and pedagogy since the early 1990s (hickey and wilson 2017; papps and ramsden 1996; ramsden 1993; richardson and carryer 2005). this includes respect for mātauranga māori (māori knowledge) and cultural beliefs and values, concepts such as ‘tapu’ (loosely interpreted as ‘sacred’) and noa (not tapu), and the tapu nature of the human body especially the head. respect for the tūpāpaku or bodies of the deceased is also central to māori cultural practices (haami 2019; mcclintock and mcclintock 2018). consideration was given to the impact and use of highly realistic models within a classroom setting, where previously human anatomy was accessed through plastic models, illustrations and 2d animations. a second consideration was around the hololens hardware and implications of touching another person’s head, placing headgear on tables that are used for food (not tapu) or seats used for sitting (not tapu) and care in handling the mr equipment in a classroom setting. lastly, there are four domains of competence for the aotearoa new zealand registered nurse scope of practice. the first domain, professional responsibility, includes competencies with legal, ethical and cultural safety responsibilities, reinforced with new zealand nurses organisation code of ethics (new zealand nurses organisation 2019, p. 10) figure 3. student view of the gigxr holohuman app on hololens 2 at toi ohomai / te pūkenga. source: reproduced with permission from gigxr. http://dx.doi.org/10.25304/rlt.v30.2803 research in learning technology citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 7 (page number not for citation purpose) with these responsibilities in mind, each headset included a qr code to launch the session, and a printed karakia (incantation) so students are reminded to approach the learning experience with respect. teachers include mātauranga māori in face-to-face and online lessons with the intention that the tikanga students practice in the classroom carries with them into their professional careers. although students are interacting with a hologram, it simulates a real body, so we saw this as an opportunity to: • provide both cultural safety and acknowledgement • instill the same level of respect expected of the nursing students in their work environments • demonstrate the expression of the nursing profession’s ethical values including the concept of manaakitanga (care). finally, before the first session began, a short video was played for the class showing what the student would be seeing, including menus, models and opportunities for questions. this was to help students scaffold from the technology setup and into the new learning experience. question categories the design of the online survey was guided by the learning object evaluation scale for students (loes-s) categories of ‘learning’, ‘quality or design’ and ‘engagement’ (kay and knaack 2009). table 1. online survey questions aligned with broad loes-s categories. category online survey question wording with 5-point likert scale response learning did the hololens session improve your understanding of the subject? (response scale from ‘no change’ to ‘strongly agree’) do you think hololens sessions are useful in preparing you for your future work life? (response scale from ‘not really useful’ to ‘very helpful’) design how comfortable did you feel using the equipment and viewing the anatomy content at the beginning of the session? (response scale from ‘uncomfortable /didn’t feel well supported’ to ‘very comfortable’) engagement did the hololens session increase your interest or motivation in learning more about the subject? (response scale from ‘no change’ to ‘yes a lot’) loes-s, learning object evaluation scale for students. results during the sessions using mr it was observed that most students experienced a ‘lightbulb’ moment as they viewed what were once abstract or 2d models or plans presented spatially or as an illusion of presence (slater 2018) when they wore a hololens headset (see figure 4). http://dx.doi.org/10.25304/rlt.v30.2803 j. adams et al. 8 citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 (page number not for citation purpose) there were some observable differences based on broad age groupings of the students. mature age students had a relatively longer learning curve to become comfortable with using the headset, but then tended to express stronger feelings or results in using the technology, such as impact on motivation or interest. younger students (those who typically enrolled directly from high school) were quicker to be comfortable using the headsets and then move on to explore the content and menu options. one inference is to examine the existing digital literacy of the participant as a dimension when measuring student receptivity or engagement with mr experiences. measuring the student experience of the 120 students who participated in the tutorials with the hololens, 78 unique online survey forms were completed. increased understanding just over half, or 51% (40 students) strongly agreed that the hololens session improved their understanding of the subject (anatomy), while 14% (11 students) reported no or slight change (see figure 5). increase in motivation or interest students were then asked if the hololens session improved their interest or motivation in learning more about the subject, 71% (55 students) or participants strongly agreed, while 10% (8 students) reported no change or slight change in their motivation. relevance all students reported some relevance and 67% (52 students) strongly agreed that the hololens session was useful in preparing them for their future work life, while the actual context for that use in a nursing career was not given or described. figure 4. using the gigxr anatomy app on hololens 2 at toi ohomai / te pūkenga in tutorials. http://dx.doi.org/10.25304/rlt.v30.2803 research in learning technology citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 9 (page number not for citation purpose) comfort or support in use after a karakia and watching an introduction or scaffolding video prior to use only 73% (57 students) felt comfortable or very comfortable, while 9% (7 students) still felt uncomfortable or slightly uncomfortable or not well supported using the equipment, with remainder neutral. students were also asked to report on anything they experienced during the hololens session, or on anything that could be improved. feedback was grouped into positive, negative and neutral categories (see appendix 1). discussion impact on students the aim of this study was to consider the impact of using mr in nursing education and in particular anatomy studies. as observed in the results section in figure 5, participants in the mr condition reported increased understanding and motivation for the subject after they used the technology, alongside their existing course materials and practical experience tutorials. however, the impacts of mr go beyond the student experience with opportunities for innovation in format and location of learning, and identifying improvements in the evolution of mr product design. toi ohomai / te pūkenga has run several hundred hololens sessions with students and teachers since 2020. the research with the 2021 first-year nursing students provided an opportunity for systematic empirical data collection to supplement observations and implementation experience. that experience highlights some constraints with the hololens in the integration to campus networks, figure 5. student survey results from a scale of 1 on the left (no change) through to 5 on the right (strongly agree). http://dx.doi.org/10.25304/rlt.v30.2803 j. adams et al. 10 citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 (page number not for citation purpose) and on user id access. the ecosystem of hardware and software (as of august 2022) needs to work together more seamlessly if the use of hololens in vocational or formal education settings is to become more widespread. impact on organisations technology provides the opportunity to change how an organisation operates, with the degree of change or impact depending on how it is used. technology can be used to just bring efficiencies to existing operations with existing processes or it can be used to innovate and ultimately disrupt the status quo. these same strategic objectives and ambitions to disrupt and innovate are set out in the toi ohomai teaching and learning strategy 2020–23: interventionist and disruptive change and innovation is needed to our teaching, learning and delivery if we are to be a viable and sustainable organisation able to meet future changes, growth and the adaptations that will certainly come with the advent of [the national review of vocational education]. (toi ohomai institute of technology 2020) impact on learning design introducing hololens into itp programmes provides the opportunity to contribute to these strategic objectives by creating significant change in how students learn in place and form, and what they can learn, shifting from consuming knowledge, to experiencing and creating knowledge. the microsoft ecosystem provides clients with the capability of authoring mr training packages that can be delivered on campus or work-site environments. this de-couples training from a specific place or campus and opens the potential for less reliance on text-based publications as a student resource. however, there remains work to integrate these standalone learning experiences so accessing the technology is not a distraction from the learning experience itself. impact on collaboration there are opportunities for wider collaboration in programme delivery at a consistent standard with the inclusion of mr in vocational education. three of the core objectives for aotearoa new zealand’s largest tertiary education provider are to provide equitable and nationally consistent outcomes for all regions, and become a ‘provider driven by innovation, collaboration and teaching excellence’. the national itp also aims to ‘partner with employers to deliver relevant work-integrated education and training that meets skills needs’ (te pūkenga n.d.). implementation journey when considering adopting the hololens, the authors assessed it using the new zealand ministry of education’s samr model (ministry of education, n.d.). we considered its alignment with the institute’s strategic objectives to create innovation in education delivery and contribute to student success. table 2 presents the outcome of this decision-making process. http://dx.doi.org/10.25304/rlt.v30.2803 research in learning technology citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 11 (page number not for citation purpose) for the implementation of hololens the authors used the framework developed by tyler-smith et al. (cited in marshall and shepherd 2016, p. 10) with capability building for teachers and support staff, and then evaluation phases in 2021, including the results of this study. at the end of 2021 the evaluation stage of implementation was concluded, building a sustainable delivery model. subsequently, the authors are undertaking testing of hololens mr authoring software, using ms guides application (microsoft corporation 2022c) for training packages on campus. the next step on the implementation journey would be external collaboration for consistency where mr applications are developed alongside other itps and employers for student learning outside of campus classrooms in workplace settings. constraints despite the clear evidence of the positive impact on student engagement, some technical limitations or constraints need to be overcome if mr is to be more broadly adopted as a new mode of learning. a seamless student journey from picking up the hardware to accessing the content needs to be designed so that adoption can move easily beyond professional users and early adopter student cohorts. being part of the microsoft product suite, hololens headsets easily connected to toi ohomai / te pūkenga azure ad and business intelligence applications, providing a well-supported technology platform to build employment-relevant projects within and beyond the campus. the technology was used on campus, and based on the authors experience of the configuration and student support required, we do not see this as suitable for distance learning environments with the hololens to date focused on enterprise and business applications. open interfaces and security independent software vendors and content developers are building learning content for the full capabilities of the hololens’s range of sensors and interactivity with peerto-peer video and audio communication and remote-control capabilities. however, these capabilities and attributes of the hololens with its portability, flexibility and independent wearable computing can be at odds with enterprise it security policies and controls. for education and corporate environments, the challenge is taking the innovation of a mr platform that can create content such as hololens, and table 2. application of the substitution augmentation modification redefinition (samr) model. effort to change redefinition create new approaches or methods with this platform. yes modification create significant or substantial change with this platform. yes augmentation create or implement functional improvements to elearning delivery with this platform. yes substitution consider hololens as a direct substitute with no functional change. no http://dx.doi.org/10.25304/rlt.v30.2803 j. adams et al. 12 citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 (page number not for citation purpose) integrating that capability and personalisation seamlessly within an organisation’s network. user authentication our aim was to provide a personalised experience for students by utilising existing unique student ids to differentiate their experience and measure results. the current designs in the hololens headset use a keyboard to enter a user id and login, or you can choose a pin and iris biometric after registering the individual. at the start of a student’s session, tapping away at a virtual keyboard is a poor user experience (ux) and a reminder of the links to enterprise desk-bound computing. the virtual keyboard metaphor to login contrasts with the remainder of the heads-up navigation conducted with hand, voice and eye movements. the keyboard login also contrasts with the iris login experience for hololens, which is immediate and handsfree. it is this seamless personalised experience (with or without biometrics) that should be the goal in delivering the learning content for every user. while teachers and support staff can use iris profiles, there is not enough capacity in the headsets for campus-wide student use, as it is limited to 64 iris profiles per headset, (as of august 2022) despite being connected to the organisation’s azure (profile) directory network, and only 10 iris identities per headset are recommended (microsoft corporation 2022b). beyond the hardware logins are the typical subscription content logins needed to get into licensed applications at the learning content or app stage. we found pins are the best compromise between quick login and scalable use in the current format. the challenge is for both hardware vendors and content developers to improve the ux, balancing their commercial needs for controlling access without adding to a student’s cognitive load navigating to content (buchner, buntins, and kerres 2022). an anonymous or ‘kiosk’ mode configured in hololens profiles can allow access for anyone, but this reduces the value of personalised learning that the platform offers. an anonymous mode in the hololens situates the learner between passive consumption and the higher order learning afforded by individualised access. unique logins and access give us the ability to customise learning for each student, report and measure progress, and leverage open network communications for collaboration. the mixed reality (mr) design evolution currently users have to pre-select the type of experience they want to utilise, because vr is experienced in different hardware from mr or ar environments. users have to first select a hardware vendor such as the oculus brand, now meta platforms, inc. for vr, or microsoft with the hololens headset for mr, that then pre-determines the ux. these artificially imposed experience boundaries should be removed with newer hardware displays that vary their immersion with opacity, occlusion and content to suit the scenario or lesson, be it full vr, or mr across all of the reality-virtuality continuum (milgram and kishino 1994). at this stage of the mr design evolution the major technology companies such as apple, microsoft and meta still largely control and mediate access to these separate worlds. from a user or student-centric view, the ideal experience would be a single display to deliver the appropriate content depending on the task or context, across the continuum that blends real and fully immersive environments. this would enable a seamless technology-enhanced learning http://dx.doi.org/10.25304/rlt.v30.2803 research in learning technology citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 13 (page number not for citation purpose) continuum mirroring the technology integration for diverse learners, learner needs and digital literacies proposed by aguayo, eames and cochrane (2020). early examples of this blended vr-ar mode are the pure mr vendors such as magic leap, which have pivoted from consumer to enterprise markets (magic leap n.d.), new ‘open’ platform entrants such as the lynx-r1 (lynx mixed reality n.d.), and potentially headsets from consumer technology companies such as apple (gurman 2022). in tandem with this hardware evolution are contributions to open-source software with toolkits (microsoft corporation 2022d) and application programming interfaces (apis) for interoperability (khronos group n.d.) to help build a consistent ux for navigating the immersive environment across platforms. conclusion the mr technology continues to evolve and even the definitions of what is augmented and mr need reviewing (skarbez, smith, and whitton 2021) with opportunities for integrating mr into a learning continuum for diverse learning and learners (aguayo, eames, and cochrane 2020). there is a significant increase in student motivation and engagement, with an improved student experience discussed in this review. the mr experiences also create significant change in how students learn, in place and format and what they can learn, shifting from consuming knowledge to higher order synthesis and evaluation (bloom 1956). given not only the positive impact on students, but also the commitment required by organisations to design, integrate and implement, then the priorities in research should be to determine to what extent mr contributes to students’ academic success. the results of this research can help course designers decide when to integrate mr experiences and support decision-making for education providers considering new modes of education and training for on-campus and work-based learning. acknowledgements thanks to the nursing students at toi ohomai / te pūkenga who participated in this study with the hololens 2. thanks to current and former kaimahi including kate akers, josh burrell, angela cuff, tina dellabarca, tepora emery, josh hahunga, jesse henwood, kieran hewitson, jacinda hills, philip lopez, marguerite marsh, tracy narbey, roseanne sadd, linda shaw, george stroud. thanks especially to our participating investigator christine m. cheyne, research office, toi ohomai institute of technology for assistance with the manuscript and preparation. references aguayo, c., eames, c. & cochrane, t. (2020) ‘a framework for mixed reality free-choice, self-determined learning’, research in learning technology, vol. 28, pp. 1–19. doi: 10.25304/ rlt.v28.2347 bim holoview. (2022) extended reality for construction, [online] available at: https://www. bimholoview.com/ bloom, b. s. (1956) taxonomy of educational objectives, handbook i. the cognitive domain, david mckay co inc, new york, ny. bower, m., et al., (2014) ‘augmented reality in education – cases, places and potentials’, educational media international, vol. 51, no. 1, pp. 1–15. doi: 10.1080/09523987.2014. 889400 http://dx.doi.org/10.25304/rlt.v30.2803 https://dx.doi.org/10.25304/rlt.v28.2347 https://dx.doi.org/10.25304/rlt.v28.2347 https://www.bimholoview.com/ https://www.bimholoview.com/ https://dx.doi.org/10.1080/09523987.2014.889400 j. adams et al. 14 citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 (page number not for citation purpose) buchner, j., buntins, k. & kerres, m. (2022) ‘the impact of augmented reality on cognitive load and performance: a systematic review’, journal of computer assisted learning, vol. 38, no. 1, pp. 285–303. doi: 10.1111/jcal.12617 collins, e. & ditzel, l. (2021) ‘standardised holographic patients: an evaluation of their role in developing clinical reasoning skills’, studies in health technology and informatics, vol. 284, pp. 148–152. doi: 10.3233/shti210687 forrester consulting. (2021) the total economic impact™ of mixed reality using microsoft hololens 2. business benefits and cost savings enabled by mixed reality solutions running on hololens 2 devices, [online] available at: https://query.prod.cms.rt.microsoft.com/cms/ api/am/binary/rwqppn frost, j., delaney, l. & fitzgerald, r. (2020) ‘exploring the application of mixed reality in nurse education’, bmj simulation and technology enhanced learning, vol. 6, no. 4, pp. 214–219. doi: 10.1136/bmjstel-2019-000464 gigxr. (2021) extending reality, [online] available at: https://www.gigxr.com/ greenway, k., butt, g. & walthall, h. (2019) ‘what is a theory-practice gap? an exploration of the concept’, nurse education in practice, vol. 34, pp. 1–6. doi: 10.1016/j.nepr.2018.10.005 gurman, m. (2022) trademark filings suggest apple may be securing ‘reality’ names for ar/vr headset, bloomberg, [online] available at: https://www.bloomberg.com/news/articles/2022-08-28/what-will-apple-call-its-ar-vr-headsets-reality-one-reality-pro-names-emerge haami, b. (2019) bringing culture into care. a biography of amohaere tangitu, huia publishers, wellington. hickey, h. & wilson, d. (2017) ‘whānau hauā: reframing disability from an indigenous perspective’, mai journal. a new zealand journal of indigenous scholarship, vol. 6, no. 1, pp. 82–94. doi: 10.20507/maijournal.2017.6.1.7 huang, y., shakya, s. & odeleye, t. (2019) ‘comparing the functionality between virtual reality and mixed reality for architecture and construction uses’, journal of civil engineering and architecture, vol. 13, no. 1, pp. 409–414. doi: 10.17265/1934-7359/2019.07.001 kay, r. h. & knaack, l. (2009) ‘assessing learning, quality and engagement in learning objects:  the learning object evaluation scale for students (loes-s)’, educational technology research and development, vol. 57, no. 2, pp. 147–168. doi: 10.1007/ s11423-008-9094-5 khronos group (n.d.) openxr, [online] available at: https://www.khronos.org/openxr/ kim, k.-j., choi, m.-j. & kim, k.-j. (2021) ‘effects of nursing simulation using mixed reality: a scoping review’, healthcare (basel, switzerland), vol. 9, no. 8, p. 947. doi: 10.3390/ healthcare9080947 knowles, m. s., holton, e. f. & swanson, r. a. (2015) the adult learner: the definitive classic in adult education and human resource development, routledge, [online] available at: https://www.taylorfrancis.com/books/mono/10.4324/9781315816951/ adult-learner-malcolm-knowles-elwood-holton-iii-richard-swanson lynx mixed reality (n.d.) lynx r-1 headset, [online] available at: https://www.lynx-r.com/ magic leap. (n.d.) magic leap 2: the most immersive ar headset for enterprise, [online] available at: https://www.magicleap.com/solutions marshall, s. & shepherd, d. (2016) e-learning in tertiary education: highlights from ako aotearoa supported research, [online] available at: https://ako.ac.nz/assets/reports/ synthesis-reports/synthesis-report-e-learning-in-tertiary-education-highlightsfrom-ako-aotearoa-supported-research.pdf matear, s. (2021) good practice assessment of online teaching in universities in aotearoa new zealand during the covid-19 pandemic and lessons for the future, academic quality agency, [online] available at: https://www.aqa.ac.nz/sites/all/files/gpa%20report%20 final.pdf mcclintock, k. & mcclintock, r. (2018) tuku iho, tuku iho, culture in māori health service provision, te rau matatini, [online] available at: https://terauora.com/wp-content/ uploads/2019/05/tuku-iho-tuku-iho.pdf http://dx.doi.org/10.25304/rlt.v30.2803 https://dx.doi.org/10.1111/jcal.12617 https://dx.doi.org/10.3233/shti210687 https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/rwqppn https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/rwqppn https://dx.doi.org/10.1136/bmjstel-2019-000464 https://www.gigxr.com/ https://dx.doi.org/10.1016/j.nepr.2018.10.005 https://www.bloomberg.com/news/articles/2022-08-28/what-will-apple-call-its-ar-vr-headsets-reality-one-reality-pro-names-emerge https://www.bloomberg.com/news/articles/2022-08-28/what-will-apple-call-its-ar-vr-headsets-reality-one-reality-pro-names-emerge https://dx.doi.org/10.20507/maijournal.2017.6.1.7 https://dx.doi.org/10.17265/1934-7359/2019.07.001 https://dx.doi.org/10.1007/s11423-008-9094-5 https://dx.doi.org/10.1007/s11423-008-9094-5 https://www.khronos.org/openxr/ https://dx.doi.org/10.3390/healthcare9080947 https://dx.doi.org/10.3390/healthcare9080947 https://www.taylorfrancis.com/books/mono/10.4324/9781315816951/adult-learner-malcolm-knowles-elwood-holton-iii-richard-swanson https://www.taylorfrancis.com/books/mono/10.4324/9781315816951/adult-learner-malcolm-knowles-elwood-holton-iii-richard-swanson https://www.lynx-r.com/ https://www.magicleap.com/solutions https://ako.ac.nz/assets/reports/synthesis-reports/synthesis-report-e-learning-in-tertiary-education-highlights-from-ako-aotearoa-supported-research.pdf https://ako.ac.nz/assets/reports/synthesis-reports/synthesis-report-e-learning-in-tertiary-education-highlights-from-ako-aotearoa-supported-research.pdf https://ako.ac.nz/assets/reports/synthesis-reports/synthesis-report-e-learning-in-tertiary-education-highlights-from-ako-aotearoa-supported-research.pdf https://www.aqa.ac.nz/sites/all/files/gpa%20report%20final.pdf https://www.aqa.ac.nz/sites/all/files/gpa%20report%20final.pdf https://terauora.com/wp-content/uploads/2019/05/tuku-iho-tuku-iho.pdf https://terauora.com/wp-content/uploads/2019/05/tuku-iho-tuku-iho.pdf research in learning technology citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 15 (page number not for citation purpose) microsoft corporation. (2022a) about hololens 2, [online] available at: https://docs.microsoft. com/en-us/hololens/hololens2-hardware microsoft corporation. (2022b) manage user identity and login for hololens, [online] available at: https://docs.microsoft.com/en-us/hololens/hololens-identity microsoft corporation. (2022c) microsoft mixed reality / ar guides | microsoft dynamics 365, [online] available at: https://dynamics.microsoft.com/en-nz/mixed-reality/guides/ microsoft corporation. (2022d) what is the mixed reality toolkit 2? [online] available at: https://docs.microsoft.com/en-gb/windows/mixed-reality/mrtk-unity/mrtk2/?view= mrtkunity-2022-05 milgram, p. & kishino, f. (1994) ‘a taxonomy of mixed reality visual displays’, ieice transactions on information and systems, vol. 77, pp. 1321–1329. ministry of education. (n.d.) samr model from te kete ipurangi, by the ministry of education, [online] available at: https://elearning.tki.org.nz/professional-learning/teacher-inquiry/ samr-model nanayakkara, c. & whiddett, r. (2005) a model of user acceptance of e-learning technologies: a case study of a polytechnic in new zealand, international conference on information systems technology and its application (ista’2005), palmerston north. new zealand nurses organisation. (2019) guideline – code of ethics 2019, [online] available at: https://www.nzno.org.nz/portals/0/publications/guideline%20-%20code%20of%20 ethics%202019.pdf osorto carrasco, m. d. & chen, p.-h. (2021) ‘application of mixed reality for improving architectural design comprehension effectiveness’, automation in construction, vol. 126, article 103677. doi: 10.1016/j.autcon.2021.103677 papakostas, c., troussas, c., krouska, a. & sgouropoulou, c. (2021) ‘exploration of augmented reality in spatial abilities training: a systematic literature review for the last decade’, informatics in education, vol. 20, no. 1, pp. 107–130. doi: 10.15388/infedu.2021.06 papps, e. & ramsden, i. (1996) ‘cultural safety in nursing: the new zealand experience’, international journal for quality in health care, vol. 8, no. 5, pp. 491–497. doi: 10.1093/ intqhc/8.5.491 putland, g. (2021) gigxr helps nursing programs teach clinical reasoning, meet clinical simulation requirements, healthy simulation, [online] available at: https://www.healthysimulation.com/31948/gigxr-mixed-reality/ ramsden, i. (1993) ‘kawa whakaruruhau. cultural safety in nursing education in aotearoa (new zealand)’, nursing praxis in new zealand, vol. 8, no. 3, pp. 4–10. doi: 10.36951/ ngpxnz.1993.009 richardson, f. & carryer, j. (2005) ‘teaching cultural safety in a new zealand nursing education program’, journal of nursing education, vol. 44, no. 5, pp. 201–208. doi: 10.3928/ 01484834-20050501-02 scully, n. j. (2011) ‘the theory-practice gap and skill acquisition: an issue for nursing education’, collegian, vol. 18, no. 2, pp. 93–98. doi: 10.1016/j.colegn.2010.04.002 skarbez, r., smith, m. & whitton, m. c. (2021) ‘revisiting milgram and kishino’s reality-virtuality continuum’, frontiers in virtual reality, vol. 2, article 64997. doi: 10.3389/ frvir.2021.647997 slater, m. (2018) ‘immersion and the illusion of presence in virtual reality’, british journal of psychology, vol. 109, pp. 431–433. doi: 10.1111/bjop.12305 te pūkenga. (n.d.) about te pūkenga – new zealand institute of skills and technology, [online] available at: https://xn--tepkenga-szb.ac.nz/about-us/ toi ohomai institute of technology. (2020) toi ohomai 2020 – 2023 teaching and learning strategy, toi ohomai institute of technology, tauranga. wunder, l., et al., (2020) ‘fire in the operating room: use of mixed reality simulation with nurse anesthesia students’, informatics, vol. 7, no. 4, article 7040040. doi: 10.3390/ informatics7040040 http://dx.doi.org/10.25304/rlt.v30.2803 https://docs.microsoft.com/en-us/hololens/hololens2-hardware https://docs.microsoft.com/en-us/hololens/hololens2-hardware https://docs.microsoft.com/en-us/hololens/hololens-identity https://dynamics.microsoft.com/en-nz/mixed-reality/guides/ https://docs.microsoft.com/en-gb/windows/mixed-reality/mrtk-unity/mrtk2/?view=mrtkunity-2022-05 https://docs.microsoft.com/en-gb/windows/mixed-reality/mrtk-unity/mrtk2/?view=mrtkunity-2022-05 https://elearning.tki.org.nz/professional-learning/teacher-inquiry/samr-model https://elearning.tki.org.nz/professional-learning/teacher-inquiry/samr-model https://www.nzno.org.nz/portals/0/publications/guideline%20-%20code%20of%20ethics%202019.pdf https://www.nzno.org.nz/portals/0/publications/guideline%20-%20code%20of%20ethics%202019.pdf https://dx.doi.org/10.1016/j.autcon.2021.103677 https://dx.doi.org/10.15388/infedu.2021.06 https://dx.doi.org/10.1093/intqhc/8.5.491 https://dx.doi.org/10.1093/intqhc/8.5.491 https://www.healthysimulation.com/31948/gigxr-mixed-reality/ https://www.healthysimulation.com/31948/gigxr-mixed-reality/ https://dx.doi.org/10.36951/ngpxnz.1993.009 https://dx.doi.org/10.36951/ngpxnz.1993.009 https://dx.doi.org/10.3928/01484834-20050501-02 https://dx.doi.org/10.3928/01484834-20050501-02 https://dx.doi.org/10.1016/j.colegn.2010.04.002 https://dx.doi.org/10.3389/frvir.2021.647997 https://dx.doi.org/10.3389/frvir.2021.647997 https://dx.doi.org/10.1111/bjop.12305 https://xn--tepkenga-szb.ac.nz/about-us/ https://dx.doi.org/10.3390/informatics7040040 https://dx.doi.org/10.3390/informatics7040040 j. adams et al. 16 citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 (page number not for citation purpose) appendix 1 open-ended question responses june 2021 question: open feedback on anything you experienced during your session, good or bad, or is there anything we can improve? feedback in survey classification 1. buttons were hard to press negative 2. hurts eyes negative 3. the selection (baby shark) part was frustrating personally negative 4. aiding students individually through it rather than giving us something new that’s hard to use with no help. it was very claustrophobic and trapping with no help made something really cool be really awful and scary negative 5. better explain action of how to use neutral 6. i think the equipment would be better if (the room) was blacked out apart from the virtual things. neutral 7. i found it a little slow and confusing but i am sure with more use, i will work it out neutral 8. we weren’t given much information or instructions as to how to use it and fully utilise the technology. otherwise, it was a fun experience. positive 9. awesome positive 10. such a respectful tutorial by the presenters. i really enjoyed the experience of viewing human anatomy as if it was right in from of me. positive 11. it was cool, seeing it like real. positive 12. reproductive system would be cool positive 13. interesting positive 14. it was so cool seeing everything like that and easy to interact with positive 15. once i got the hang of it it was very easy to use. accurate instructions from john also. positive 16. i would like to see a female model, so i can view the female reproductive system. otherwise really fun positive 17. it was great to see all the different aspects of the anatomy in this way. i felt truly amazed by the model. positive 18. i think this would be an incredibly beneficial resource positive 19. was a very exciting experience for me as it was the first time i have ever used a virtual reality. positive 20. i thought it was really well done, easy to use and had a lot of help to further understand. positive 21. amazing experience positive http://dx.doi.org/10.25304/rlt.v30.2803 research in learning technology citation: research in learning technology 2022, 30: 2803 http://dx.doi.org/10.25304/rlt.v30.2803 17 (page number not for citation purpose) survey questions your feedback on the hololens session q1. did the hololens session improve your understanding of the subject? scale: from 1. no change in my understanding of the subject 2 3 4 5. yes strongly agree: the session helped me a lot. figure a1. student view, layout of question and scale. q2. did the hololens session increase your interest or motivation in learning more about the subject? scale: from 1. no change in my motivation, to 5. yes a lot: it helped increase my interest or motivation in the subject. q3. how comfortable did you feel using the equipment, and viewing the anatomy content at the beginning of the session? scale: from 1. i was uncomfortable / i didn’t feel well supported, to 5. i was very comfortable / i felt supported at the start. transferring your learning to real life q4. do you think hololens sessions are useful in preparing you for your future work life? scale: from 1. not really useful for preparing me for future work, to 5. yes very helpful in preparing me. http://dx.doi.org/10.25304/rlt.v30.2803 enhancing laboratory activity with computer-based tutorials gordon ritchie* and paul garner** * department of electronic systems engineering, university of essex **department of electronics, university of york students should be encouraged to take full advantage of expensive laboratory resources by being adequately prepared computer-based laboratory tutorials have proved very successful in generating competence and confidence in the laboratory as well as reinforcing lecture material. this paper presents an outline of the tutorial process with its on-line referencing of the recommended textbook, the favourable reaction of students, and discussion of logistical and authoring problems. introduction in a degree course in electronic engineering, great importance is attached to laboratory work, in which students have the opportunity to develop their creative skills in a practical environment. for example, in the first year of the course they are expected to design and test some basic circuits using data available on the characteristics of the semiconductor devices to be used. many of the students cannot be prepared sufficiently for this activity by attendance at lectures, in which basic principles are expounded to large classes. firstyear students have widely differing knowledge, experience and ability in circuit design. therefore, without individual tuition many of them are insufficiently prepared for their laboratory work. weaker students often neglect to study the laboratory documentation thoroughly in advance and they make poor progress in the laboratory. we are developing computer-based tutorials specifically designed to prepare students for laboratory work by enabling them to grasp the relevant theory and carry out preliminary calculations in a self-paced interactive environment made interesting by the use of graphics and animation. in this way, students are enticed to prepare in advance for the laboratory. while they would like simply to be given the answers, they are required to earn them in a structure where lecture material is reinforced and they are exposed to 49 gordon ritchie and paul gamer enhancing laboratory activity methodical procedures. with highly-motivated and able students the printed word can suffice but computer-based learning (cbl) is ideal for interactive study enhanced by use of colour and animation; in many spheres it is still a novel medium. the computer-based laboratory tutorial written using authorware professional, our computer-based laboratory tutorials guide electronics students through the preliminary work they are expected to carry out before coming into the teaching laboratory. they are taken through successive procedures and detail of the experiments; their prerequisite knowledge and understanding of material are tested as the tutorial progresses. they extract from data (a skill on its own) typical device parameter values and apply them in preliminary design and calculation of circuit performance. access to relevant circuit diagrams is provided, as is the ability to return to earlier material. throughout the tutorials, on-line reference is provided to the recommended text which has been adapted to disk as required. also, at specific points in the tutorial, the reference text can be accessed directly at the relevant section. the essential theory inevitably requires the use of text and equations but such material is interspersed with questions and answers which provide interactive work for the student. graphical material is used wherever possible and is animated by, for example, moving components and connections around in circuit diagrams. colour is used to distinguish different variables such as voltages at different points in a circuit. students are guided through the application of theory in the specific context of the practical laboratory experiment. this requires at least a basic knowledge of the subject, as no attempt is made to develop the subject from fundamentals as would be done in a lecture course. however, the reference text is at hand. potentially the computer-based tutorial is very compact, with the pc/terminal capable of handling extra items such as the experiment script itself, reference texts and data material, calculator, notepad, etc. in additional windows. it may be suggested that this preparation is rather passive. on the contrary, students are involved in significant activity while they interact within the tutorial. and, although they work in the tutorial with typical values extracted from data sheets, in the actual laboratory they have to recalculate using real measured values of parameters. this repetition serves further to reinforce the material. on-line referencing a disk version of the recommended textbook (ritchie, 1993) was available. unfortunately this was intended for typesetting purposes rather than end-user consumption. it contained many commands for controlling fonts, sizes and spacing which had to be deleted. figures had to be scanned from the printed text and cleaned up before inserting. also, equations had to be reconstructed. these processes were very time-consuming. access to the relevant chapter of the reference text (423k) from the tutorial uses the 50 alt-} volume 3 number i authorware jumpfilereturn function which, by passing the page number of the reference, allows access to the specified reference point. using packaged files, access is rapid (seven seconds using a 25 mhz 486 pc). the facility to browse either side of the reference point is provided. quitting the reference returns (in five seconds) to the exit point from the tutorial. it is interesting to note that jumpfilereturn functions may be nested, thus enabling referencing within references. consumer reaction the first of these tutorials was written for the first-year hardware laboratory in the electronics department at the university of york, supporting an experiment on fieldeffect transistors (fets). running this tutorial met with unqualified enthusiasm from students and staff. the value is immense, introducing a new structured phase into the active learning process which engenders confidence and competence. in comparison with previous cohorts, staff were impressed by the extremely searching questions posed by the most able students and by the improved level of understanding shown by their weaker colleagues. all students, strong and weak, enjoyed the tutorial and found it very useful. it complemented the lecture course on fets as well as reinforcing it, and was valuable for revision. being able to work at their own speed was appreciated. there were many requests that the computer-based laboratory support be extended to cover lecture material as well. some students felt that answers to questions in the tutorial were provided too readily, but an authoring decision had been taken specifically to direct the tutorial towards the laboratory experiment and not to a study of the general field. in doing so, students are guided through a logical approach to the work which in itself, as with case studies and design examples, forms an informative and stimulating experience. yet more interaction, animation and graphics have been requested. as with printed material, content and presentation are under continuous review and future versions will enhance the offering. problems unfortunately, a cbl framework has its drawbacks, especially the overhead of expensive authoring time. even with experience, it is unlikely that one hour of quality screen tutorial can be produced in less than forty hours. a laboratory experiment (like a lecture course) tends to be very parochial, having been developed by an individual member of staff over several years. invariably, investment has been made in specialized equipment and circuit boards. so it is unlikely that there exists cbl material which is immediately applicable to an experiment one has to start from scratch. however, fewer staff are required to demonstrate in the laboratory; this saving can be diverted to generating new material. 51 gordon ritchie and paul gamer enhancing laboratory activity a significant problem is access to the pc laboratory; an hour session for each of 100 students makes significant inroads into bookable and open-access time. scheduled classes for software training should not be reduced to accommodate this activity. the problem multiplies if other groups have similar activity at the same time. in time, however, as more and more students possess their own powerful pcs, access to tutorial software via network or by disk distribution will enable them to participate while in their own personal environment, independent of the central facility. at present, the tutorial system is not compact. students work from a printed laboratory script' and still need pencil and paper, not only for rough working but to make notes for later reference in the laboratory and for retention. hand-held calculators are much in evidence. browsing is rather more difficult than with printed text as the computer-based tutorial is a focused sequential activity. a few rather more sophisticated word-processing and drawing facilities would be welcomed in authorware professional, in particular an ability to draw arcs and sinewaves plus pixel-erase and pixel-edit (fatbits); subscripting with auto size reduction; and greek symbols such as omega and pi as options rather than in a different font. conclusion computer-based tutorials have proved very successful in enticing students to prepare themselves for laboratory work. written to guide students through preliminary calculations for an experiment, the tutorial reinforces understanding of the material and provides on-line access to the recommended text. it is a self-paced exercise in which a student takes not more than one hour to become well-prepared for a six-hour laboratory session, thus making far more productive use of that session. this activity is seen as an enhancement of paper-based study, not as an extra imposition. in some establishments it is felt necessary to pass a test before entering the laboratory; with the enthusiasm generated by the computer-based tutorial this will not be necessary. acknowledgement the authors wish to thank professor greville bloodworth of the university of york for his enthusiastic and constructive contribution to this work. reference ritchie, g.j. (1993), transistor circuit design discrete and integrated (third edition), london, chapman & hall. 52 effective cal: theory and practice hassan khan hypertext support unit, university of kent, canterbury this paper looks at the creation of cal material in theory and practice through the perspective of a cal development unit, the hypertext support unit at the university of kent. with similar units now in place in other institutions, the production of cal is taking place at a rapid pace, often with the evaluation of the applications disproportionately lagging behind. this inevitably raises questions of whether what we are producing is effective cal, and what approaches should be taken to use the potential of multimedia in a learning environment. introduction modern-day authoring systems have made the production of cal applications so easy that people with little computer literacy are able quite quickly to create elaborate multimedia applications. the point, however, is that while so many of us have become authors in the past few years, the objective of our creations has been somewhat missed. it is all to easy to see visually impressive multimedia cal, and to convince ourselves that they represent good cal material. an application may be quickly sanctioned, produced and implemented at universities, then attention is focused on to the next project. while evaluation is normally costed into a project, various constraints, such as shifting personnel or additional demands on funds, limit the evaluation of the application to ascertain whether the investment of producing it was worthwhile. the hypertext support unit (hsu) at the university of kent was set up in 1992 to promote the pervasive use of hypertext across the campus. in its role as a support unit, it facilitates the development of cal material in all disciplines in close collaboration with the content specialists, i.e. the lecturers. the hsu, along with many other units or departments, produce many cal application paying attention to the aesthetics of interface design, but largely glossing over the learning instructions so vital to good cal applications in harnessing the potential of multimedia in an educational environment. too many people put a linear book on-line, 98 volume 3 number i give it some bookmarks, and call it hypertext; worse yet, they add a few scanned-in photographs and a soundtrack and call it multimedia (fisher, 1994). effective cal the view that multimedia would add a new dimension to teaching and learning and enhance the learning process, has not yet been realized. this may be in part untrue, due to the lack of documentation and research into the evaluation of current cal systems and their impact on learning, but developers have certainly still not tapped into the true potential that multimedia has to offer in the scope of knowledge construction. nelson and palumbo (1992) state that 'simply allowing learners to quickly access a body of information, even in the learner-controlled, non-linear manner supported by hypermedia systems, does not ensure learning any more than a library does'. there is a case for arguing that even a very flat, predominantly text-based publication, when put on-line over a network, can greatly increase access to the information. it is not to be confused, however, with an application that is to be used to augment learning. such systems need to use the cognitive processes of the brain by incorporating learning instruction. these systems cannot merely concern themselves with the elaborate presentation of information making use of sound, graphics, video or animation, but rather must facilitate the assimilation of the information into a knowledge base of content from which inferences as to the principles or concepts that are to be learned, may be articulated. nelson and palumbo (1992) go on to say that 'merely developing knowledge bases for presentation to learners in a hypermedia environment will not be particularly effective unless instructional activities are supported by the hypermedia system'. learning theory in cal most popular authoring systems support at least four inherent cognitive features. these can be classified as: browsing, searching, integrating chunks of information, and angling, as mentioned by duchastel (1990), a process that describes the navigation to the same chunk of information via a number of different pathways. angling can be used to good effect in cal applications to allow a topic to be perused in different ways, each one bringing a new perspective to the learner. all of these processes allow the incorporation of some sort of learning instruction in cal to aid knowledge representation, construction and assimilation. lindsay & norman (in dufresne et ai, 1990) describe the definition of concepts as being central to comprehension; through the use of careful organization of information into defined categories with further subdivisions of related information, authors can build up a linked knowledge base that can be explored by learners and assist in the articulation of concepts. the four processes described above are central to authoring principles, and their use ultimately determines the accessibility and flexibility to be found in the cal application (khan et al, 1994). authoring systems are coming in for much criticism for making the incorporation of learning instruction difficult. however, it is probably more a case of a lack of understanding about the learning theory and the under-use of the true hypertext features _ hassan khan effective cal theory and practice of authoring systems. learning tools that may be constructed in cal applications are advanced organizers (ausubel, 1960). these can serve to revise information and concepts that should have been learnt, and also introduce the new information that is to follow. advance organizers may be extended to offer suggested pathways (hammond, 1989) that guide learners through the information. at the outset of the application, a concept map or navigational web can be used to present the learner with an outline of the information available, and to highlight alternate navigational pathways which may be based on different levels of comprehension. concept maps provide learners with a top-level look at the mesh of hypertext information. halasz et al. (1987) have gone as far as saying that no system can be described as a true hypertext system unless it has something akin to a concept map. for the purpose of development, however, where current systems do not support this feature it is normally possible to exploit the interactivity of the authoring system with other applications. these learning tools provide mechanisms for theme highlighting, restatement, introduction, summary pages and the use of analogies, all of which promote the understanding of information (kintsch & van dijk, 1978). another useful learning skill is knowledge construction, the building of new nodes and links between information that is pertinent to a particular individual: open hypermedia systems such as microcosm (hill & hall, 1994) allow the construction of a personal library linkbase of information in a distributed environment. even world wide web, through the manipulation of the hotlist feature, allows users to create their own information base. the future of cal development authors of cal must possess a variety of skills, some of which include computing, art and graphic design, audio and visual manipulation skills, and some knowledge of learning theory. it is in this capacity that cal development differs from traditional software development. it is important to grasp the fact that authors of multimedia cal applications cannot ignore the use of these skills during development. an author possessing even some of these skills is a vital commodity, yet realistically it is not always fair to expect lecturers who have become developers to turn their ideas into effective cal applications. development of cal involves an investment of time, money and resources, and another factor that is so easily forgotten: the subsequent maintenance of the applications. poorly designed applications are so often required to be completely redeveloped when the content has become dated. as i have mentioned, development of cal is increasingly being shifted from isolated departments to centralized development units, which serve as a focal point and repository of development, support and maintenance activities. yet while it is probable that such entities will possess more of the skills required for effective cal development, additional specific expertise will still be required. there is no one solution to this problem, but there are several options. institutions can deliberately try to recruit multi-skilled personnel for development units, though it is improbable that any one person will possess the right mix of skills. another possible approach is to train recruited personnel by allowing them to spend time in 100 alt-] volume 3 number i various departments whose expertise would be useful to cal development. the most idealistic approach is the collaboration of development units with relevant departments, though this may be more or less difficult from university to university depending on the individual culture of the institution. conclusion learning is a highly individual process, and cannot be standardized. multimedia allows information to be constructed so that it may be adapted to various cognitive styles. current systems need to be evaluated to assess the benefits of investments involved in their production and maintenance. central to the process of development of effective cal is the need for a variety of skills. these skills are often available within a single institution and should be tapped. the university of kent has a campus where there exists a climate of receptiveness to learning technology, and the hypertext support unit has been able to use the campus-wide co-operation (khan et al, 1994) in striving to develop effective cal material. but each institution has a particular culture, and where collaboration is not practicable, other ways of accessing the skills required for cal development must be sought. references ausubel, d.p. (1960), 'the use of the advance organiser in the learning and retention of meaningful verbal material', journal of psychology, 51, 67-72. duchastel, p.c. (1990) 'examining cognitive processing in hypermedia usage', hypermedia, 2, 3. dufresne, a., jolin, n., & sentini, a. (1990), 'hypertext documents for the learning of procedures', in mcaleese, r. and green, c. (eds), hypertext state of the art, oxford, intellect. fisher, s. (1994), multimedia authoring, london, ap professional. halasz, f., moran, t. & trigg, r. (1987) , 'notecards in a nutshell', in carrol, j. and tanner, p. (eds.), in human factors in computing systems and graphic interface (chi + gi 1987 conference proceedings), acm, 45-52. hammond, n. (1989), 'hypermedia and learning: who guides whom?', in maurer, h. (ed.), computer-assisted learning, (proceedings of the second international conference, iccal 1989), berlin, springer verlag. hill, g. & hall, w. (1994), 'extending the microcosm model to a distributed environment', in proceedings of the echt94 conference, acm, 32-40. khan, h., slater, d., simpson, v.b. & strang, w. (1994), 'campus-wide collaboration in the delivery of distributed learning technology', paper presented at the 29th annual international conference of the association for educational and training technology, edinburgh, april 1994. 101 hassan khan effective c a l theory and practice kintsch, w. & van dijk, t. (1978), 'toward a model of text comprehension and production.', psychological review, 85, 5, 363-94. nelson, w.a. & palumbo, d.b. (1992), 'learning instruction and hypermedia', journal of educational multimedia and hypermedia, 1, 3. 102 calt586809 87..88 editorial the role of people and organisations in learning technology practice this issue of research in learning technology reflects the importance of people and organisations in the effective application of learning technology. our journal aims (amongst other goals) to spread good practice in the use of learning technology, and the papers in this issue can help to achieve this aim. creating an issue is as much about happenstance as planning but it is always interesting to see what the assembled papers have in common. in this collection of papers, there is a strong practice theme emerging. in the previous issue (vol. 19, no. 1), my co-editor rhona sharpe wrote about the nature of evaluative evidence that we need and can produce. in the special issue on the transformational impact of learning technology (vol. 18, no. 3), the editors considered issues of strategy, implementation and integration of learning technologies in organisations. the present issue (vol. 19, no. 2) builds on these by providing further evidence of attitudes, perceptions and behaviours of staff and students that can inform the implementation of learning technologies by institutions and at the more local levels of subjects and courses. in the first paper in this issue, williams, boyle, molloy, brightwell, munro, service, and brown explore student attitudes to e-learning in undergraduate paramedic programmes from five universities, revealing student reluctance to engage in elearning that was not clearly related to their needs. this valuable finding can inform innovation in related programmes. chen, pedersen, and murphy’s thorough investigation of students’ perceptions of information overload is based on quantity of information, quality of information and the medium interface through which the information is presented. the authors offer detailed recommendations for course design, and some of their findings are relevant to staff and students. hepplestone, holden, irwin, parkin and thorpe review the literature on the use of technology in increasing student engagement with feedback, with recommendations on the use of online assessment and feedback tools and the need to uncover existing good practice. smailes and gannon-leary consider virtual learning environments, social networking sites and virtual worlds as potential sites for a peer mentoring scheme, based on students’ expressed desire for online peer mentoring and their current use of these three environments. these four papers have a strong emphasis on student practice, and can inform institutional innovation and what staff and students do, as long as the findings are communicated to them. benson, anderson and ooms’ case study of staff attitudes to blended learning will be useful to organisations developing and implementing blended learning research in learning technology vol. 19, no. 2, july 2011, 87–88 issn 2156-7069 print/issn 2156-7077 online � 2011 association for learning technology doi: 10.1080/21567069.2011.586809 http://www.informaworld.com strategies. it is not enough to provide learning technologies: they have to be successfully embedded in staff and student practice. the implementation theme is developed further in the final paper where buchan develops an adaptive cycle framework in the context of learning technology introduction in an australian university. our challenge in the research in learning technology journal is to collect and disseminate this real-world knowledge to those who need it. we hope that reading these articles will encourage you to reflect on your practice and to have conversations with colleagues and students about how learning technology can be used most effectively in your own institution. frances bell salford business school, university of salford, uk 88 editorial (page number not for citation purpose) 1 *corresponding author: email: chinara.jumabaeva@manas.edu.kg research in learning technology 2023. © 2023 k. baryktabasov et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 research in learning technology vol. 31, 2023 original research article using information and communication technologies for the assessment of a large number of students kasym baryktabasov, chinara jumabaeva* and ulan brimkulov computer engineering department, kyrgyz-turkish manas university, bishkek, kyrgyz republic (received: 3 november 2022; revised: 16 may 2023; accepted: 25 may 2023; published: 20 july 2023) many examinations with thousands of participating students are organized worldwide every year. usually, this large number of students sit the exams simultaneously and answer almost the same set of questions. this method of learning assessment requires tremendous effort and resources to prepare the venues, print question books and organize the whole process. additional restrictions and obstacles may appear in conditions similar to those during the covid-19 pandemic. one way to obviate the necessity of having all the students take an exam during the same period of time is to use a computer-assisted assessment with random item selection, so that every student receives an individual set of questions. the objective of this study is to investigate students’ perceptions of using random item selection from item banks in order to apply this method in large-scale assessments. an analysis of the responses of more than 1000 surveyed students revealed that most of them agree or completely agree with using the proposed method of assessment. the students from natural science departments showed more tolerance of this method of assessment compared with students from other groups. based on the findings of this study, the authors concluded that higher-education institutions could benefit from implementing the abovementioned assessment method. keywords: computer-assisted assessment; computer-based assessment; e-assessment; learning assessment; education; information and communication technologies introduction ‘assessment is an essential component of learning and teaching’ (ferrari et al., 2009). assessment is critical to student learning and certification (bennett et al., 2017). it is always present in higher education and influences all stakeholders, including educational institutions, teachers, and students (stödberg, 2012). in higher education, it shapes the experiences of students and influences their behavior more than the teaching they receive (bloxham & boyd, 2007). assessment can be summative or formative. formative assessment usually takes place during the learning process and aims to provide feedback in order to support students’ learning. summative assessment aims to summarize students’ accomplishments and usually takes place at the end of the mailto:chinara.jumabaeva@manas.edu.kg http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2945 k. baryktabasov et al. 2 citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 (page number not for citation purpose) course or education process. the most important part of summative assessment is grading and making judgments (stödberg, 2012). in the modern world, the implementation of assessment practices through information and communication technologies (ict) is very important (boud & soler, 2016). the use of ict for learning assessment appears in the literature under such names as computer-assisted assessment (caa) (bull & mckenna, 2004), computer-based assessment (cba) (thelwall, 2000), technology-based assessment (tba) (csapó & molnár, 2019), and finally e-assessment (adesemowo et al., 2016).1 e-assessment works very well for formative assessment, summative assessment (stödberg, 2012), and self-assessment. e-assessment has many advantages compared with paper-based testing (alruwais et al., 2018). it allows more complex item types, such as the use of audiovisual materials, and more complex interactions between the learner and the computer (conole & warburton, 2005). caa systems provide richer data about students’ performance and rapid feedback (brimkulov et al., 2017). different types of caa tools have already been developed (contreras-higuera et al., 2016; christie et al., 2015). many caa systems and tools allow questions to be generated randomly (kruger et al., 2015). piaw argued that replacing paper-based testing with computer-based testing does not cause significant differences in test scores (piaw, 2012). e-assessment provides an opportunity to decrease lecturers’ administrative burden, giving them more time for research and professional self-development. it acquires special significance when it comes to implementing large-scale assessments (adesemowo et al., 2017). today, it is clear that systematic large-scale assessments cannot be conducted with traditional instruments (csapó & molnár, 2019). many large-scale assessments take place every year. in some cases, thousands of students take an exam simultaneously. for example, this occurs in university admission examinations. in 2011, 9.33 million participants took the national higher education entrance examination (also called gaokao) in the people’s republic of china (haifeng, 2012). in china, all students throughout the country sit the above-mentioned exam at the same time (davey et al., 2007). in south korea, some 600,000 college applicants took the college scholastic ability test (suneung in korean) in 2016 (kim et al., 2017). in vietnam, the ministry of education and training administers a national university entrance examination, which is undertaken annually by over one million final-year secondary school students (hayden & thiep, 2010). in the russian federation, 800,000 candidates took the unified state exam (a country-wide standardized examination that combines in a single procedure the examination at the end of secondary school with entrance exams for tertiary education) in 2012 (piattoeva, 2015).the number of participants in the university entrance examination in turkey has exceeded one million in recent years (içbay, 2005). a number of other countries also organize these kinds of examinations. the authors of this study also faced the necessity of conducting assessments for large classes. they were teaching ‘introduction to information technologies’, which is a mandatory course for all first-year students at kyrgyz-turkish manas university (ktmu). the number of students enrolled in the course each semester was about 300–500. during the semester, students had at least two exams: a midterm (formative assessment) and a final (summative assessment). the assessment was conducted in the form of a paper-based test using multiple-choice questions. organizing this 1hereinafter the terms ‘e-assessment’, ‘technology-based assessment’, ‘computer-assisted assessment’, and ‘computer-based assessment’ will be used interchangeably. http://dx.doi.org/10.25304/rlt.v31.2945 research in learning technology citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 3 (page number not for citation purpose) assessment was quite challenging, because it was necessary to bring in all the enrolled students at the same time, prepare venues, print questions on paper, ask additional staff for help (in order to prevent students from cheating), etc. the students were from different departments with different schedules, which made it impossible to find a convenient time on a weekday. it was also challenging to find enough free venues for examinations. this is why the exams were conducted on weekends (usually on saturdays). in order to avoid all of these inconveniences, we decided to use the benefits of e-assessment. we developed a caa system, which is described in brimkulov et al. (2017). using this system, the students sit exams in small groups at the scheduled lesson time in the computer-equipped classroom where the normal lessons are usually taught. there was a risk of the dissemination of the exam questions by students who had taken an exam earlier to other students. thus, we decided to use random question selection from an item bank, so that every student received a unique set of questions. the system attracted the attention of other colleagues who also taught courses with large numbers of enrolled students, as they faced the same issues. after several years of the successful usage of this e-assessment system, we started considering the possibility of using it for the university admission examination. every year, ktmu organizes a university admission exam in the form of a paper-based test using multiple-choice questions. however, the scale of that examination is much larger. the number of enrollees participating in the exam is about 6000–8000. when we suggested that the university admission examination organizing committee apply the same method that we used in the ‘introduction to information technologies’ course, the committee members expressed concerns about the fairness of the examination. the committee members were afraid that some students might complain that they could get higher scores if they were given another set of questions. it was unknown to what extent such complaints would be received. this fact motivated us to do an empirical study of this issue. as we planned to involve a large number of students (about 1000), performing a survey through a questionnaire was found to be the most appropriate method of collecting data. the objective of this study is to investigate students’ perceptions of using random item selection from item banks in order to apply this method in large-scale assessments. the following research questions are studied: 1. to what extent do students agree with the use of random item selection from item banks in formative assessments? 2. to what extent do students agree with the use of random item selection from item banks in summative assessments? 3. to what extent do students agree with the use of random item selection from item banks in university admission examinations? 4. to what extent do students agree with the use of random item selection from item banks in self-assessments? 5. is there any difference in the perceptions of student groups from different domains of learning? there are a number of studies on students’ attitudes towards using ict for learning in general and caa in particular. silin and kwok (2017) examined the factors that support or hinder students’ attitudes towards using ict in problem-based learning among polytechnic students. karl et al. (2011) stated that students found caa to http://dx.doi.org/10.25304/rlt.v31.2945 k. baryktabasov et al. 4 citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 (page number not for citation purpose) be equivalent to written multiple-choice tests. dammas (2016) surveyed chemistry department students and revealed that the majority of respondents had a positive attitude towards computer-based testing. there are also studies that suggest effective algorithms for selecting questions randomly from a database (liu & feng, 2009). binnahedh (2022) examined the attitudes of students and teachers toward the washback effect of electronic tests. the results of the study show that students’ perceptions of electronic testing are more positive than teachers’ perceptions of such tests (binnahedh, 2022). st-onge et al. (2021) examined how educators have coped with the challenge of adapting their assessment methods during the pandemic to determine what is required to support and facilitate the future development of quality assessments and the introduction of e-assessment in higher education. the covid-19 pandemic has provided an unprecedented opportunity to critically evaluate and change assessment methods. some studies also suggest paying attention to the ‘sustainable assessment’ concept. boud and soler (2016) consider this term. they believe that sustainable assessment will allow students to regulate and evaluate their own learning and continue their studies outside the period of a course. the development of knowledge assessment is of particular interest (boud & soler, 2016). stödberg (2012) studied the major areas of interest in e-assessment. based on a literature review, he revealed that most of the research was devoted to the study of the learning environment. he also found that the number of research papers on e-assessment was increasing at a high speed (stödberg, 2012). clarke-midura and dede (2010) pointed to the inadequacy of automated versions of item-based paper-and-pencil tests in 21st-century education. in order to use the full power of ict to innovate via providing richer observations of student learning, the authors explored a virtual assessment of learning achievements (clarke-midura & dede, 2010). other authors consider technologically rich environments (tres) and the pedagogical opportunities they offer to learners and teachers (shute et al., 2016). adesemowo et al. (2016) emphasized the importance of considering security concerns when implementing an e-assessment platform, especially for the assessment of large classes. they also studied the opinions of students concerning the introduction of e-assessment and its impact on the workload of teachers. based on the results of the study, the use of e-assessment is viable, scalable, and safe, reducing the administrative burden and increasing student productivity. adesemowo et al. (2017) revealed that learning management system platforms could provide a tre for designing innovative text-based assessments for relatively large classes. a number of studies have examined the features, advantages, and disadvantages of e-assessment. jordan (2013) proposed that the online environment makes computer assessments of knowledge more accessible. jordan (2013) stated the following: ‘questions for computer-marked assessment need to be delivered at low cost and quickly and there is a danger that this will lead to poor quality assessment’. to avoid low-quality assessments of students’ knowledge, it is necessary to provide the system with high-quality questions of various types; however, the questions should assess the same learning outcome and they should have the same level of difficulty. jordan (2013) found that in assessing large numbers of students, the possibility of different students receiving http://dx.doi.org/10.25304/rlt.v31.2945 research in learning technology citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 5 (page number not for citation purpose) different sets of questions is very important. in this case, it is necessary to create question banks (jordan, 2013). the authors of this study could find only one research paper that discussed students’ perceptions regarding e-assessments in which random questions are selected from item banks (dermo, 2009). thus, the authors of this study had the impression that there is a lack of sufficient research on students’ attitudes towards random item selection in caa. in order to fill this gap, this study surveyed more than 1000 students, asking them to express their perceptions of the abovementioned assessment method. the rest of the paper is organized as follows. the next section describes the survey and analysis methodology, and it is followed by a section that describes the collected data and analysis results. then, there is a section with discussions of this study’s findings. the final section concludes the paper. methodology in order to conduct this study, questionnaires in three languages (kyrgyz, russian, and turkish) were developed from scratch by the authors using the google forms tool. these questionnaires consisted of two parts. the first part contained general questions about the university, department, and year of the student. the questions in the second part are given in table 1. the survey was conducted among university students in the city of bishkek, kyrgyz republic. there is an ‘introduction to information and communication technologies’ course that is mandatory for all first-year students of ktmu. during one of the lessons of that course, after the midterm examination in the form of computer-based testing with random item selection, the survey was introduced to the students by one of the authors of the study. then, the students were asked to fill out the questionnaire right there in the computer classroom. some thirdand fourth-year ktmu students were surveyed in exactly the same manner but during other courses. the students from other universities were provided with a link and asked to fill out the form on their own. the survey was conducted from 2018 to 2019. the responses to statements #3, #4, and #5, placed in the second part of the questionnaire, are of particular interest, as they are directly related to the aim of the study. the collected data were represented using a likert scale (likert, 1932). there has been considerable discussion regarding the statistical methods that are appropriate for the analysis of likert-scale data. some authors argue that parametric methods cannot be used to analyze ordinal data like likert-scale data (tabachnick & fiddell, 2007). others argue that ‘parametric statistics are robust with respect to violations of these assumptions’ (norman, 2010). there are many papers with recommendations for the analysis of likert-scale data (harpe, 2015; joshi et al., 2015). there was also a computer simulation study in which five-point likertscale data were analyzed using the t-test and mann-whitney-wilcoxon test. the results of the study ‘showed that the two tests had equivalent power for most of the pairs’ (de winter & dodou, 2010). however, the authors of this study decided to use both parametric (t-test, analysis of variance [anova]) and nonparametric (mann-whitney-wilcoxon test, kruskal-wallis h-test) methods and then compare the results in order to ensure the correctness and robustness of the analysis results. for the analysis, the spss software was used. http://dx.doi.org/10.25304/rlt.v31.2945 k. baryktabasov et al. 6 citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 (page number not for citation purpose) collected data and analysis results the total number of students who participated in the study is 1034. most of them (91.7%) are students at ktmu. the rest of the students represent five other higher-education institutions (kyrgyz-russian slavonic university, kyrgyz state medical academy, kyrgyz national university, international university of central asia, and kyrgyz state technical university) in different proportions. as mentioned above, most of the respondents were surveyed during the lesson of one of the mandatory courses for first-year students at ktmu. this is the reason that most of the students are first-year students (87.6%). the surveyed students represent 42 different departments that have been divided into three groups: natural sciences, economics, and humanities (the number of students in each group can be found in table 2). departments such as computer engineering, software engineering, chemical engineering, environmental engineering, food engineering, information technologies, building construction, etc., have been included in the natural sciences group. departments such as economics, finance and credit, accounting, management, etc., have been included in the economics group. the rest of the departments have been included in the humanities group (philosophy, history, turcology, linguistics and translation, etc.). table 1. the questions in the second part of questionnaire. question answer options 1 have you ever taken an examination in the form of computer testing? • yes • no 2 what is your level of confidence in the computer testing results? 1 i don’t trust them at all. 2 3 4 5 i completely trust them. example: there are 500 questions in the question bank. the difficulty level is the same across all questions. the computer will randomly select 50 questions from the question bank for the exam. thus, every student will have a unique set of questions. please provide your level of agreement with the following statements. 3 it would be good to use the assessment method described in the example above in midterm examinations. 1 i completely disagree. 2 3 4 5 i completely agree. 4 it would be good to use the assessment method described in the example above in final examinations. 1 i completely disagree. 2 3 4 5 i completely agree. 5 it would be good to use the assessment method described in the example above in university admission examinations. 1 i completely disagree. 2 3 4 5 i completely agree. 6 it would be good to use the assessment method described in the example above for self-assessment purposes. 1 i completely disagree. 2 3 4 5 i completely agree. 7 using the assessment method described in the example above will lead to unfair results. 1 i completely disagree. 2 3 4 5 i completely agree. 8 it is not necessary to ask the same questions in order to compare students’ competence. 1 i completely disagree. 2 3 4 5 i completely agree. http://dx.doi.org/10.25304/rlt.v31.2945 research in learning technology citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 7 (page number not for citation purpose) when the survey was administered, most of the students already had experience with computer testing (88.1%). seventy percent of the respondents trust or completely trust computer testing results. the percentage of students who do not trust computer testing results was 7%, and 23% were not sure. the total percentage of students who agree or completely agree with using a random item selection method in midterm examinations was 58.22%. on the other hand, the total percentage of students who disagree or completely disagree with using the proposed assessment method in midterm examinations was 16.05% (see table 3). the total percentage of students who agree or completely agree with using a random item selection method in final examinations was 57.7%. on the other hand, the total percentage of students who disagree or completely disagree with using the proposed assessment method in final examinations was 23.0% (see table 3). the total percentage of students who agree or completely agree with using a random item selection method in university admission examinations was 56.38%. on the other hand, the total percentage of students who disagree or completely disagree with using the proposed assessment method in university admission examinations was 23.12% (see table 3). the total percentage of students who agree or completely agree with using a random item selection method for self-assessment purposes was 77.2%. on the other hand, the total percentage of students who disagree or completely disagree with using the proposed assessment method for self-assessment purposes was 10.7% (see table 3). most of the surveyed students said that using the method of random item selection will not lead to unfair results (see table 3). after the survey, it was revealed that the wording of the statement ‘it is not necessary to ask the same questions in order to compare students’ competence’ was unclear to students. this is the reason that, in the opinion of the authors of this study, the responses to that statement were uncertain (see table 3). the responses to statements #3, #4, and #5 (in the second part of the questionnaire) are of particular interest. together, they form the core of the study. the sum of the number representations of a student’s responses to these statements provides information about the student’s general attitude (negative, neutral, or positive). if a student’s response consists of only neutral values (the number representation of the response to each of the given statements is equal to 3), then the sum value will be equal to nine. thus, a sum value that is equal to 10 indicates a positive attitude, because in this case, a student must have given at least one positive response (3+3+4 or 3+2+5). the sum values of the number representations of the responses to statements #3, #4, and #5 calculated for every respondent represent the aggregated data, which is also table 2. descriptive statistics for the groups of students. n mean sd sd 95% confidence interval for mean min. max. lower bound upper bound natural sc. 243 11.3086 3.14777 0.20193 10.9109 11.7064 3 15 economics 132 10.4394 3.22741 0.28091 9.8837 10.9951 3 15 humanities 659 10.4841 3.19207 0.12435 10.2399 10.7282 3 15 total 1034 10.6721 3.20271 0.09960 10.4767 10.8676 3 15 http://dx.doi.org/10.25304/rlt.v31.2945 k. baryktabasov et al. 8 citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 (page number not for citation purpose) referred to as ‘likert-scale data’. the descriptive statistics of the obtained aggregated data are presented in table 4; the frequencies are presented in figure 1. these aggregated data represent the general attitude of the respondents toward the assessment method with random item selection. the values equal to or greater than 10 indicate a positive attitude. the value for cronbach’s alpha for these three statements was α = 0.754. according to field (2013), cronbach’s alpha scores above 0.7 are considered ‘acceptable’ in most social sciences. it was interesting to determine whether the attitudes of the students differ among the groups of departments (natural sciences, economics, and humanities) or not. since the data are not normal (see table 5) and ordinal, the authors of this study decided to run both a parametric one-way anova and a non-parametric kruskal-wallis h-test, and then compare the results. descriptive statistics for the groups of students are given in table 6. table 3. the distribution of the responses to the statements from the second part of the questionnaire. statement responses total 1. i completely disagree 2 3 4 5. i completely agree 3. ‘it would be good to use the assessment method described in the example above in midterm examinations’. 81 85 266 301 301 1034 7.83% 8.22% 25.73% 29.11% 29.11% 100% 4. ‘it would be good to use the assessment method described in the example above in final examinations’. 125 113 199 322 275 1034 12.10% 10.90% 19.30% 31.10% 26.60% 100% 5. ‘it would be good to use the assessment method described in the example above in university admission examinations’. 135 104 212 228 355 1034 13.06% 10.06% 20.50% 22.05% 34.33% 100% 6. ‘it would be good to use the assessment method described in the example above for self-assessment purposes’. 56 55 125 227 571 1034 5.40% 5.30% 12.10% 22.00% 55.20% 100% 7. ‘using the assessment method described in the example above will lead to unfair results’. 311 218 201 169 135 1034 30.10% 21.10% 19.40% 16.30% 13.10% 100% 8. ‘it is not necessary to ask the same questions in order to compare students’ competence’. 180 142 252 218 242 1034 17.40% 13.70% 24.40% 21.10% 23.40% 100% http://dx.doi.org/10.25304/rlt.v31.2945 research in learning technology citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 9 (page number not for citation purpose) there was a statistically significant difference between the groups according to the one-way anova (f(2,1031) = 6.349, p = 0.002). levene’s test showed that the variances for students’ attitudes were equal (f(2,1031) = 0.383, p = 0.682). the results of the post hoc test are given in table 6. table 4. descriptive statistics of students’ attitudes according to the aggregated data. statistic names statistic se n 1034 mean 10.6721 0.0996 95% confidence interval for mean lower bound upper bound 10.4767 10.8676 5% trimmed mean 10.8446 median 11.0000 sd 3.20271 variance 10.257 minimum 3 maximum 15 range 12 interquartile range 4.00 skewness −0.547 0.076 kurtosis −0.357 0.152 table 5. results of the normality tests. statistic name kolmogorov-smirnova shapiro-wilk statistic df sig. statistic df sig. students’ attitude 0.110 1034 0.000 0.943 1034 0.000 alilliefors significance correction. figure 1. graphical representation of the frequencies of the aggregated data. http://dx.doi.org/10.25304/rlt.v31.2945 k. baryktabasov et al. 10 citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 (page number not for citation purpose) based on the post hoc test results, it can be stated that there is a statistically significant difference between the attitudes of the students from the natural science group and the students from the other groups. a kruskal-wallis h-test showed that there was a statistically significant difference in students’ attitudes between the different groups (h(2) = 15.127, p = 0.001), with a mean rank attitude of 582.24 for the natural science group, 493.36 for the economics group, and 498.46 for the humanities group. then, post hoc tests were conducted to test pairwise comparisons. a statistically significant difference was found between the natural science group and the economics group (p = 0.006). there was also a statistically significant difference between the natural science group and the humanities group (p = 0.000). the difference between the economics group and humanities group was not statistically significant (p = 0.857). it was also interesting to determine whether there is a difference in attitude between ktmu students (n = 948) and students from other universities (n = 86). the t-test showed that there was not a statistically significant difference in the attitudes of ktmu students (m = 10.6445, standard deviation [sd] = 3.20) and students from other universities (m = 10.9767, sd = 3.23) (t(1032) = −0.921, p = 0.357). a mann-whitney u-test also indicated that there was not a statistically significant difference between the attitudes of students from other universities (mean rank = 547.61) and ktmu students (mean rank = 514.77) (z = −0.982, p = 0.326). as we can see, the results of the parametric and nonparametric tests lead to the same conclusions. discussion the results of the current study show that the majority of the respondents agree with using random item selection in caa. however, the more important the examination, the smaller the number of students that support this method. the results mean that the proposed method could be used in different types of examinations. this method has been in use since 2014 at ktmu on the midterm and final examinations of the ‘introduction to information and communication technologies’ mandatory course. thousands of students were tested using this method up to the year 2020. no complaints from students have been received. however, many requests have been received from lecturers to use this method in the other mandatory courses. taking into consideration the fact that the students from the natural science group expressed a more positive attitude towards using the method of random item selection table 6. tukey hsd post hoc test results. (i) group (j) group mean difference (i-j) sd sig. 95% confidence interval lower bound upper bound natural sc. economics humanities 0.86925 0.82458 0.34451 0.23913 0.032 0.002 0.0606 0.2633 1.6779 1.3858 economics natural sc. humanities −0.86925 −0.04467 0.34451 0.30384 0.032 0.988 −1.6779 −0.7578 −0.0606 0.6685 humanities natural sc. economics −0.82458 0.04467 0.23913 0.30384 0.002 0.988 −1.3858 −0.6685 −0.2633 0.7578 http://dx.doi.org/10.25304/rlt.v31.2945 research in learning technology citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 11 (page number not for citation purpose) in caa, this method could be applied even more broadly in the departments and universities that focus on this field of study. this finding correlates with the results obtained by abdullah et al. (2015). they revealed a ‘difference between arts and science students in terms of their attitude towards it in favor of science students’ (abdullah et al., 2015). the authors of this study also agree with jordan (2013), who argued that the selection of questions from a question bank or the use of multiple variants of each question can discourage plagiarism. it is worthwhile to invest in the use of e-assessment for the assessment of a large number of students (jordan, 2013). it should be noted that dermo (2009) analyzed students’ perceptions of e-assessments and found that the use of random question selection from item banks was seen by the students as unfair. this result is the complete opposite of the results of our study. this might be due to several factors, one of which is the time that passed between these two studies. the students’ attitudes might have changed. the more e-assessment is used in education, the more students become accustomed to this kind of approach. however, the authors of this study agree with dermo (2009), who stated that it is necessary to ‘take steps to ensure the quality of these item banks, for example, using item analysis to check the difficulty level of the items in the bank’. the learning outcomes assessed by the items should also be the same (jordan, 2013). in other words, the focus of the lecturers should change from organizing the examinations, reading students’ responses, and grading to preparing quality item banks, validating and ensuring that items have equal difficulty levels, and monitoring the assessed learning outcomes. in the opinion of the authors of this study, higher-education institutions should organize an assessment and certification center equipped with the necessary hardware and software. the center would be responsible for performing learning assessments using modern ict and could be available 24/7 for the convenience of the students. the teachers, lecturers, and other academic staff could be exempt from the duty of supervising exams. they would be responsible for preparing question banks containing high-quality questions. this kind of assessment center would make it possible to obviate the necessity of having all the students take an exam at the same time and place (which is difficult to organize when there is a large number of participating students or in conditions similar to those during the covid-19 pandemic). it would reduce paper use, reduce the time that academic staff must spend on evaluation and grading, provide the students with an opportunity to choose convenient times for and a convenient order of exams, and finally allow the use of more complex question types in order to assess competencies such as problem-solving, reflection, creativity, critical thinking, etc. the same assessment and certification center could be used for university admission examinations. in countries where the national authorities are responsible for holding university admission examinations, the computer-equipped classrooms of schools and universities with available internet access could be used for caa. conclusion ict plays an important role in education today. many different kinds of systems and tools have been developed to support the educational process, including software for learning assessment. however, there is still much room for improvement in this area. http://dx.doi.org/10.25304/rlt.v31.2945 k. baryktabasov et al. 12 citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 (page number not for citation purpose) many examinations organized worldwide, with a large number of participating students, are still paper-based. switching to cbas can potentially provide many advantages. the objective of this study was to explore the attitudes of students towards the method of random item selection that is often used in cba and that makes it possible to obviate the necessity of having all the students take an exam at the same time and place. the key findings of this study are the following: • most of the surveyed students agree with the use of the method of random item selection in cba (58.22% for midterm examinations, 57.7% for final examinations, and 56.38% for university admission examinations; the average value is 57.43%). • the percentage of students who do not agree with the use of this method of assessment was 16.05% for midterm examinations, 23% for final examinations, and 23.12% for university admission examinations (with an average of 20.7%). • the students from natural science departments showed more tolerance of this method of assessment compared with students from economics and humanities fields. • there was no difference in the attitudes of students from different universities towards using the abovementioned method of learning assessment. these findings mean that there should not be many complaints from students regarding the fairness of the examinations when the proposed method is used. based on their own experience of using the proposed method of assessment and the analysis of the survey data, the authors of this study assume that the method of random item selection in caa could be used more broadly in different types of examinations with a large number of participating students at higher-education institutions. the findings of this study may be useful for academic staff, the decision-makers of higher-education institutions, and policy-makers. as one of the directions for future research, a study could be conducted in order to understand the attitudes of high school students toward the proposed assessment method and whether there is any difference between high school students and university students. another study could explore whether there is any difference between the attitudes of students from different countries. the authors of this study would also be interested in doing research to understand why some students do not agree with using the proposed method by interviewing them in the future. declarations availability of data and materials the datasets used and/or analyzed in the current study are available from the corresponding author on reasonable request. funding no funds, grants, or other support was received. http://dx.doi.org/10.25304/rlt.v31.2945 research in learning technology citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 13 (page number not for citation purpose) acknowledgements not applicable. references abdullah, z. d. et al. (2015). students’ attitudes towards information technology and the relationship with their academic achievement. contemporary educational technology, 6(4), 338–354. doi: 10.30935/cedtech/6158 adesemowo, a. k. et al. (2016). the experience of introducing secure e-assessment in a south african university first-year foundational ict networking course. africa education review, 13(1), 67–86. doi: 10.1080/18146627.2016.1186922 adesemowo, a. k., oyedele, y. & oyedele, o. (2017). text-based sustainable assessment: a case of first-year information and communication technology networking students. studies in educational evaluation, 55, 1–8. doi: 10.1016/j.stueduc.2017.04.005 alruwais, n., wills, g. & wald, m. (2018). advantages and challenges of using e-assessment. international journal of information and education technology, 8(1), 34–37. doi: 10.18178/ ijiet.2018.8.1.1008 bennett, s. et al. (2017). how technology shapes assessment design: findings from a study of university teachers. british journal of educational technology, 48(2), 672–682. doi: 10.1111/ bjet.12439 binnahedh, i. a. (2022). e-assessment: wash-back effects and challenges (examining students’ and teachers’ attitudes towards e-tests). theory and practice in language studies, 12(1), 203–211. doi: 10.17507/tpls.1201.25 bloxham, s. & boyd, p. (2007). developing effective assessment in higher education: a practical guide: a practical guide. united kingdom: mcgraw-hill education. boud, d. & soler, r. (2016). sustainable assessment revisited. assessment & evaluation in higher education, 41(3), 400–413. doi: 10.1080/02602938.2015.1018133 brimkulov, u., baryktabasov, k. & jumabaeva, c. (2017). information technologies in education: the learning assessment tools. manas journal of engineering, 5(2), 27–33. bull, j. & mckenna, c. (2004). blueprint for computer-assisted assessment. london: routledge falmer. christie, m. f. et al. (2015). improving the quality of assessment grading tools in master of education courses: a comparative case study in the scholarship of teaching and learning. journal of the scholarship of teaching and learning, 15(5), 22–35. doi: 10.14434/josotl. v15i5.13783 clarke-midura, j. & dede, c. (2010). assessment, technology, and change. journal of research on technology in education, 42(3), 309–328. doi: 10.1080/15391523.2010.10782553 conole, g. & warburton, b. (2005). a review of computer-assisted assessment. alt-j, 13(1), 17–31. doi: 10.3402/rlt.v13i1.10970 contreras-higuera, w. e. et al. (2016). university students’ perceptions of e-portfolios and rubrics as combined assessment tools in education courses. journal of educational computing research, 54(1), 85–107. doi: 10.1177/0735633115612784 csapó, b. & molnár, g. (2019). online diagnostic assessment in support of personalized teaching and learning: the edia system. frontiers in psychology, 10, 1522. doi: 10.3389/ fpsyg.2019.01522 dammas, a. h. (2016). investigate students’ attitudes toward computer based test (cbt) at chemistry course. archives of business research, 4(6), 58–71. doi: 10.14738/abr.46.2325 davey, g., de lian, c. & higgins, l. (2007). the university entrance examination system in china. journal of further and higher education, 31(4), 385–396. doi: 10.1080/03098770701625761 dermo, j. (2009). e-assessment and the student learning experience: a survey of student perceptions of e-assessment. british journal of educational technology, 40(2), 203–214. doi: 10.1111/j.1467-8535.2008.00915.x http://dx.doi.org/10.25304/rlt.v31.2945 https://doi.org/10.30935/cedtech/6158 https://doi.org/10.1080/18146627.2016.1186922 https://doi.org/10.1016/j.stueduc.2017.04.005 https://doi.org/10.18178/ijiet.2018.8.1.1008 https://doi.org/10.18178/ijiet.2018.8.1.1008 https://doi.org/10.1111/bjet.12439 https://doi.org/10.1111/bjet.12439 https://doi.org/10.17507/tpls.1201.25 https://doi.org/10.1080/02602938.2015.1018133 https://doi.org/10.14434/josotl.v15i5.13783 https://doi.org/10.14434/josotl.v15i5.13783 https://doi.org/10.1080/15391523.2010.10782553 https://doi.org/10.3402/rlt.v13i1.10970 https://doi.org/10.1177/0735633115612784 https://doi.org/10.3389/fpsyg.2019.01522 https://doi.org/10.3389/fpsyg.2019.01522 https://doi.org/10.14738/abr.46.2325 https://doi.org/10.1080/03098770701625761 https://doi.org/10.1111/j.1467-8535.2008.00915.x k. baryktabasov et al. 14 citation: research in learning technology 2023, 31: 2945 http://dx.doi.org/10.25304/rlt.v31.2945 (page number not for citation purpose) de winter, j. c. f. & dodou, d. (2010). five-point likert items: t test versus mann-whitneywilcoxon. practical assessment, research, and evaluation, 15, article 11. doi: 10.7275/bj1p-ts64 ferrari, a., cachia, r. & punie, y. (2009). innovation and creativity in education and training in the eu member states: fostering creative learning and supporting innovative teaching [jrc technical note 52374], european commission. joint research centre. field, a. (2013). discovering statistics using ibm spss statistics. 4th ed. london: sage publications limited. haifeng, l. (2012). the college entrance examination in china. international higher education, 68, 23–25. doi: 10.6017/ihe.2012.68.8617 harpe, s. e. (2015). how to fp analyze likert and other rating scale data. currents in pharmacy teaching and learning, 7(6), 836–850. hayden, m. & thiep, l. q. (2010). vietnam’s higher education system. in: reforming higher education in vietnam. higher education dynamics, 29, 15–30. doi: 10.1007/978-90-481-3694-0_2 içbay, m. a. (2005). a swot analysis on the university entrance examination in turkey: a case study. mersin university journal of the faculty of education, 1(1), 126–140. doi: 10.17860/ efd.08133 jordan, s. (2013). e-assessment: past, present and future. new directions, 9(1), 87–106. doi: 10.29311/ndtps.v0i9.504 joshi, a. et al. (2015). likert scale: explored and explained. british journal of applied science and technology, 7(4), 396–403. doi: 10.9734/bjast/2015/14975 karl, m. et al. (2011). student attitudes towards computer-aided testing. european journal of dental education, 15(2), 69–72. doi: 10.1111/j.1600-0579.2010.00637.x kim, y., kang, t.-s. & rhie, j. (2017). development and application of a real-time warning system based on a mems seismic network and response procedure for the day of the national college entrance examination in south korea. seismological research letters, 88(5), 1322–1326. doi: 10.1785/0220160208 kruger, d. et al. (2015). improving teacher effectiveness: designing better assessment tools in learning management systems. future internet, 7(4), 484–499. doi: 10.3390/fi7040484 likert, r. (1932). a technique for the measurements of attitudes. archives of psychology, 140(22), 5–55. liu, q.-j. & feng, y.-r. (2009). research and implementation of random question selection based on genetic and tabu algorithm. journal of linyi normal university, 31(6), 136–139. norman, g. (2010). likert scales, levels of measurement and the ‘laws’ of statistics. advances in health sciences education, 15(5), 625–632. doi: 10.1007/s10459-010-9222-y piattoeva, n. (2015). elastic numbers: national examinations data as a technology of government. journal of education policy, 30(3), 316–334. doi: 10.1080/02680939.2014.937830 piaw, c. y. (2012). replacing paper-based testing with computer-based testing in assessment: are we doing wrong? procedia – social and behavioral sciences, 64, 655–664. doi: 10.1016/j. sbspro.2012.11.077 shute, v. j. et al. (2016). advances in the science of assessment. educational assessment, 21(1), 34–59. doi: 10.1080/10627197.2015.1127752 silin, y. & kwok, d. (2017). a study of students’ attitudes towards using ict in a social constructivist environment. australasian journal of educational technology, 33(5), 50–62. doi: 10.14742/ajet.2890 stödberg, u. (2012). a research review of e-assessment. assessment & evaluation in higher education, 37(5), 591–604. doi: 10.1080/02602938.2011.557496 st-onge, c. et al. (2021). covid-19 as the tipping point for integrating e-assessment in higher education practices. british journal of educational technology, 53(2), 349–366. doi: 10.1111/bjet.13169 tabachnick, b. g. & fidell, l. s. (2007). experimental designs using anova (vol. 724). belmont, ca: thomson/brooks/cole. thelwall, m. (2000). computer-based assessment: a versatile educational tool. computers & education, 34, 37–49. doi: 10.1016/s0360-1315(99)00037-8 http://dx.doi.org/10.25304/rlt.v31.2945 https://doi.org/10.7275/bj1p-ts64 https://doi.org/10.6017/ihe.2012.68.8617 https://doi.org/10.1007/978-90-481-3694-0_2 https://doi.org/10.17860/efd.08133 https://doi.org/10.17860/efd.08133 https://doi.org/10.29311/ndtps.v0i9.504 https://doi.org/10.9734/bjast/2015/14975 https://doi.org/10.1111/j.1600-0579.2010.00637.x https://doi.org/10.1785/0220160208 https://doi.org/10.3390/fi7040484 https://doi.org/10.1007/s10459-010-9222-y https://doi.org/10.1080/02680939.2014.937830 https://doi.org/10.1016/j.sbspro.2012.11.077 https://doi.org/10.1016/j.sbspro.2012.11.077 https://doi.org/10.1080/10627197.2015.1127752 https://doi.org/10.14742/ajet.2890 https://doi.org/10.1080/02602938.2011.557496 https://doi.org/10.1111/bjet.13169 https://doi.org/10.1016/s0360-1315(99)00037-8 confidence assessment in the teaching of basic science a. r. gardner-medwin department of physiology, university college, london a scheme is described for including information about confidence in the computer-based assessment of students. after each answer, students declare a confidence level of 1, 2, or 3. if the answer is correct, then this is the mark awarded. if not, marks of 0, -2, or -6 are awarded students do well on this scheme if they can discriminate between when they are sure of correct answers and when they are partly guessing. in self-assessment, students are trained to reflect on their reasoning, and to develop the skills of correct confidence judgement. the task of writing tests is simplified, since it becomes less important to ask complex questions. simple direct questions discriminate better between students than they do with ordinary marking. good students answer correctly with high confidence, while weak students moderate their confidence level if they know they are uncertain, or else lose heavily when they make mistakes. preliminary data are presented from self-assessment trials amongst medical students. introduction automated assessment suffers from two problems that are considered here. firstly, it seldom makes use of information about how confident a student is in the answer given, which is part of what we take into account in assessing students person-to-person. secondly, it often involves the construction of complex questions to ensure that students cannot get good marks by a combination of partial knowledge and guesswork. such questions can be ambiguous and open to different levels of interpretation, so the creation of satisfactory tests is time-consuming. this paper is primarily about confidence assessment. however, the use of confidence assessment also simplifies the construction of tests, since it means that knowledge can be tested more thoroughly with simple direct questions. with confidence assessment, a student cannot do well by employing guesswork. the marking scheme is such that he/she must either declare the low confidence that is appropriate if guesswork is employed, or else lose heavily whenever a mistake is made. a student who declares confidence in wrong answers will expect to do particularly badly. 80 aa-} volume 3 number i the importance of confidence judgements in the study of basics it is important that students should have confident knowledge of the basics in their subject. if they are uncertain about things, for example the meanings of words, they are handicapped in future study. they need, of course, to be able to produce correct answers to a good fraction of the relevant questions one might ask, as tested in conventional marking schemes. however, it is equally important, perhaps even more important, that they should be able to identify when they are likely to be getting the answers right and when not. confident belief in answers that are wrong is far worse than recognition that one simply does not know the answer. confidence judgements are well recognized as relevant to knowledge assessment (see, for example, de finetti, 1965; good, 1979; and literature reviewed in hutchinson, 1991). confidence is a form of probability forecast (dawid, 1986): an estimate of the probability that an answer will turn out to be correct. of course one can have high or low confidence about things that are matters of fact, not of probability: for example, you might judge your confidence that canberra is the capital of australia to be 0.3, 0.7 or 0.95, etc. what is estimated here is the probability that a proposition will turn out to be true when, in your own brain, it is subject to whatever degree of uncertainty of recall and confusion or unreliability of reasoning applies to the present issue. if your confidence judgement is good, you will be correct on about 0.7 of those occasions when you judge your confidence to be 0.7. there is a substantial psychological literature on the extent to which people make errors of such confidence judgements (see, for example, erev et al., 1994). the main point for the present is simply that students benefit from good confidence judgement. it is an essential skill for efficient study and work practice; for example, it is obviously necessary for selective and efficient use of a dictionary, or for deciding when it is appropriate to take time to check a calculation or the expression of a formula. we expect good students to show good confidence judgement, yet we seldom teach it explicitly or give them explicit practice. a scheme for combined assessment of accuracy and confidence students are asked to state with each answer their level of confidence (c = 1,2 or 3) in the correctness of their decision. if the answer is correct, then this is the score awarded. a wrong answer leads to a score of zero for level 1, and -2 or -6 for levels 2 and 3 respectively. level 2 gives equally weighted negative marking for wrong answers, as in many standard assessment schemes including that used at university college london for medical examinations. students are told to choose level 2 unless they are very confident (>80% chance of being right: odds 4:1), when they should choose level 3, or rather hesitant (<67% chance of being correct: odds 2:1), when level 1 is appropriate. this strategy is optimal to maximize their expected scores. the confidence assessment scheme is a proper marking scheme (dawid, 1986), in the sense that a student can never expect to gain systematically by either overestimating or underestimating confidence. high marks are gained firstly, of course, by getting the 81 a r. gardner-medwin confidence assessment in the teaching of basic science • log (probability assigned to the correct answer) figure i. the relation between score and the probability assigned by a student to the right answer, in a binary test the score is approximately linearly related to the logarithm of the probability, plotted on the abscissa. note that an assigned probability of 20% means that the student has assigned 80% probability to the wrong answer. answers correct. however, students will lose out unless they develop the introspective skills that lead to correct confidence judgement, or discrimination between when they are likely to be getting answers right or wrong. since reflecting on confidence may lead to students changing their minds about the preferred answer, they are given the opportunity (by entering -1 as the confidence) to go back and change the answer. the new answer is then what counts, and confidence is requested over again. once a valid confidence level has been entered, the student is immediately told (for formative assessment) whether the answer was correct and what score was awarded. with binary (true/false) questions, the score on this simple scheme is approximately linearly related, over its six discrete values from 3 to -6, to the logarithm of the confidence expressed for the correct answer (figure 1). such a logarithmic function is one of the more commonly proposed proper scoring functions for eliciting honest confidence declarations (good, 1979), and in fact has important theoretical merit as a measure of knowledge or ignorance when it comes to combining scores for knowledge about different aspects of a topic. note that with binary questions, 80% confidence expressed for an answer that is wrong implies 20% confidence for the correct answer. with questions having free-format answers, this would simply be an upper bound for the confidence for a correct answer. the delivery system a new authoring and delivery system (lapttop: london agreed protocol for teaching and t-esting of physiology) has been developed to incorporate confidence assessment. this is supported by the physiology departments in most of the london medical schools, and is also being used by several other departments in university college. although designed with the needs of physiologists in mind, it is equally applicable to many other disciplines. it runs exercises defined in ascii text files, capable of being prepared and edited on any wordprocessor, together with diagrams prepared as .pcx or .bmp files. 82 ait-] volume 3 number i the main novel feature is the incorporation of confidence assessment. important related features, however, are the ability to store and collate student comments entered in the context of individual questions (e.g. indicating that questions seem to be ambiguous or marked incorrectly), and to total how often each question is answered correctly and incorrectly at each confidence level. questions that are answered incorrectly with high confidence can be particularly revealing to teachers on the relevant course. confidence judgements are handled in the same way for binary and free-format answers. the lapttop system can verify answers consisting of words, phrases, numbers within specified ranges, or quantities (numbers + units). it is used at present solely for formative assessment, i.e. self-assessment and tutorial exercises. for these, the flexibility to vary the question format in any sequence is particularly valuable. the system runs on the university college ethernet, collating usage information in one place from any number of simultaneous users. students can also download the files and use them for study at home, since they run on any ibm-compatible pc without stringent specification requirements. confidence data for binary questions in physiology statistics are presented here from use of a bank of 376 binary questions in basic physiology over a period of seven weeks prior to exams. students were encouraged not required to use the system, and data was collected anonymously. a total of 121 sessions were recorded, in 14 of which the students chose not to use confidence assessment. the remaining 107 sessions were analysed (77 male students, 30 female). 6,950 questions were presented on the screen, of which 7% were skipped without answering, 26% answered at confidence c = 1, 20% at c = 2 and 48% at c = 3. the percentages of the answers given at each confidence level that were in fact correct were 56% at c = 1, 71% at c = 2 and 83% at c = 3. these are within the limits of confidence for which it is optimal for the students to choose each of the three levels (50%-67% for c = 1, 67%-80% for c = 2, 80%-100% for c = 3). this suggests that on average these students were well calibrated in their confidence judgements, getting about the expected percentages correct at each level. four sessions identified individuals who were significantly overestimating their confidence when using c = 3: percentages correct (out of 16-45 questions answered at this confidence level) were in each case more than three standard deviations below expectation for an 80% chance of being correct (p<0.002). there was no evidence of significant underestimation of confidence in any sessions. the male students in this self-selected sample had better knowledge than the females (71% correct overall for the males, 64% for the females). the males answered a substantially higher fraction of the questions with high confidence (58% at c = 3, compared with 31% for the females). the percentages correct at each confidence level were not, however, much different for the sexes (56%, 71% and 82% for the males and 57%, 72% and 85% for the females). thus the greater fraction of males choosing c = 3 was due largely to the difference in ability rather than to the known tendency of females to declare lower confidence than males for the same level of performance (walkerdine, 1990). such problems as were revealed in the calibration of confidence judgements seem to have been 83 a r. gardner-medwin confidence assessment in the teaching of basic science largely among the males. fourteen out of fifteen sessions in which the scores at c=3 or c=2 were more than two standard deviations below the appropriate expectation were male (including all four extreme cases mentioned above). the benefits of confidence assessment the students' informal reactions indicate that they appreciate confidence assessment. they see it as testing something important in relation to their knowledge, and they like the option of avoiding negative marking (with c = 1) when they are hesitant. in only 12% of sessions did they opt not to use confidence assessment, or not to vary their declared confidence level, despite the fact that their multiple-choice-question examinations do not involve confidence assessment. with some students, the principles involved have seemed complicated when first explained, but present no problem once they gain experience. in many situations, university teachers are faced with a wide range of student backgrounds and abilities. it is important to identify and assist students early on, where knowledge is weak or absent. because of the diversity of student problems, this is an area where automated techniques can help a great deal. one of the merits of confidence assessment is that it is possible to give students relatively easy basic tests without seeming condescending to good students who get them mostly correct. these students get a kick out of getting the answers right at maximum confidence, while the weak students can answer questions initially with low confidence, then increase confidence as they repeat exercises in an area. we use the system in this way, for example, to help with the basic mathematical skills necessary for a bsc degree. there is little evidence from our data so far to suggest that many medical students are radically wrong in their confidence judgements overall. where students do have such a problem, use of a confidence-assessment scheme seems a potentially valuable tool for helping with the problem. even for students who are well calibrated in this respect, it is instructive for them to think, on the occasions when they make mistakes with high confidence, why they failed to see the risks they were taking. such experiences should help students to develop careful habits of thought and to identify when their knowledge is tentative, or subject to serious misconceptions. acknowledgements the physiology questions were written by staff at the charing cross and westminster medical school. the lapttop system is available to university departments agreeing to pool material with other registered users. references de finetti, b. (1965), 'methods for discriminating levels of partial knowledge concerning a test item', british journal of mathematical and statistical psychology, 18, 87-123. dawid a.p. (1986), 'probability forecasting' in kotz, s., johnson, n.l. & reid, c.b. (eds), encyclopedia of statistical sciences, 7, 210-18. 84 alt-} volume 3 number i erev, i., wallstein, t.s. & budescu d.v. (1994), 'simultaneous overconfidence and underconfidence the role of error in judgement processes', psychological review, 101, 519-27. good, i.j. (1979), ' "proper fees" in multiple choice examinations', journal of statistical and computational simulation, 9, 164-5. hutchinson, t.p. (1991), ability, partial information, guessing, adelaide, rumsby. walkerdine, v. (1990), schoolgirl fictions, london, virago. 85 dissertations and databases: the historian as software engineer gervase phillips manchester metropolitan university this article argues that historians have always been closer to programmers than has perhaps been recognized, and that historical software projects undertaken within the framework of the traditional third-year dissertation are useful training not just for the potential historian, but also for the potential software engineer. the historian of tomorrow will be a programmer or he will be nothing. (emmanuel le roy, le territoire de l'historien (paris, 1973)) the final weeks of the summer term see the normal frantic rush of second-year students looking for a suitable topic for a dissertation. traditionally, the aim is to produce a piece of work from ten to fifteen thousand words, comprising a significant amount of original research, drawn from primary material, and demonstrating the ability to formulate and sustain a cogent argument, evaluate evidence and communicate original ideas. all this should be within the conventional, scholarly framework, logically structured and scrupulously referenced. with increased student numbers and strained traditional resources, one possible solution to the problem of finding a topic is a software project. this choice utilizes it skills in a manner which the student can immediately appreciate, and integrates it into the mainstream of the history student's experience. there is no real reason why a software project cannot be considered by the same criteria as a normal dissertation, and open to similar judgements: essentially historical judgements. but the tight methodological rules which have governed the writing of academic history can be of more than passing relevance to the software engineering industry itself. as the use of it becomes more widespread in the teaching of history, and, as le roy predicted, more historians become programmers, it is important to avoid some of the pitfalls into which the burgeoning software industry fell. interestingly, those rigorous methodological rules insisted upon in the production of history dissertations are not only good practice for potential historians, but also for potential software engineers. 48 alt-] volume 3 number 2 the production of software was, and to some extent still is, bedevilled by the idiosyncrasies of individual programmers. since there was very rarely a single right way of writing a program, a single correct algorithm to solve a given problem, much code has been produced with features reflecting only the individual proclivities of the programmer. there is a parallel here with the 'production' of history. starting with the same problem and the same primary sources, different historians have produced widely differing interpretations of the same events. thus american historians, influenced by political beliefs, sectional loyalty, or the social mores of the times in which they wrote, have given a plethora of differing accounts of the outbreak of the american civil war. the difference, vis-a-vis the production of software, however, is that there is an established methodological framework within which history is written. the conventions of historical writing, such as footnoting and referencing, facilitate comprehension. the origins of arguments, interpretations, and particular lines of thought can be traced back to their sources. new interpretations, revisions, and previously unknown original sources can all be incorporated into the existing corpus of historical knowledge within the conventional methodological framework. software engineers, on the other hand, had no such framework or conventions. if a program needed to be revised, up-dated or debugged, the programmer was faced with a mass of code, possibly in a low-level language, where it could be very difficult to identify what any particular line was doing at any stage, making the amendment of the code immensely problematic. given a lapse of time, even the original authors of code could be unable to unravel their own work', and even greater problems ensued when different programmers had to struggle with other peoples' unannotated and undocumented code. knowledge of a historian's sources makes it easier to 'debug' his or her work. what is required is precisely the kind of conscientious referencing that is (or should be) second nature to the historian. code should be elucidated by textual comment in the manner of good history. object orientation will make this more plausible and more obvious: objects have a history as well as formal parentage. all this has interesting implications in teaching at undergraduate level. history students are likely to have a better grounding in documenting their work than their more technically gifted peers in the sciences. it is important that this aspect is not neglected in teaching the use of it, and if the use of computers is taught as an intrinsic part of the historian's method of production, there is no reason why it should be. if it is taught as a bolt-on unit, with little reference to the subject a student is reading, it may not be recognized that some of the skills the student is taught as a historian remain relevant, even essential, when using computers. it may prove, ironically, that students who produce substantial and well-referenced text for history assessments come to regard the use of computers as so distinct from their normal work, that they, like many software engineers, fail to produce adequate documentation in that part of their studies. however, if the need to produce sound documentation is stressed, then the training that a degree in history gives students should potentially make them promising software engineers. consider, for example, a database project. databases are one of the most exciting and useful it applications for the historian, a mainstay of undergraduate history and computing courses. the nature of historical sources makes the creation of historical databases a challenging exercise for historian and programmer alike. all the standard 49 gervase phillips dissertations and databases: the historian as software engineer < historical problems arise and the standard historical criteria apply: the material included in a database must be subject to the same care and attention as any primary source material. an obvious example is that the historian must ensure that any statistical evidence he or she is working with is reliable, comparable, representative and fruitfully categorized. the programmer/historian must address the motives of those who compiled the figures and constructed the categories and must be aware that the definitions of categorizations and classifications are themselves matters of history. furthermore, the programmer/historian must think carefully about the design of the database. historical databases, resources for analysis rather than functionally analytical, may seem both objective and empirical, yet the data stored in them is selective. the design of the database, the names and data definition of fields, and the structure of tables in a relational database, are essentially historical choices, judgements open to the criticism of other historians. attempts were made to quantify the morale of the british army in 1917 by counting the occurrence of words indicative of good or bad morale in their letters home during censoring (cab. 24/26 g.t.2052 and cab. 24/36 g.t.3044). were the views expressed in letters that men knew would be censored, to relatives they did not want to distress, really likely to be representative of their thoughts? to follow this judgement in database fields of 'good' and 'bad' would be to fail in historical analysis, just as would be the suppression of the use of this contrast. the programmer/historian must justify a design (argue a case), select and interpret data (evaluate evidence), and present the data clearly in a format in which others may use it (communicate original ideas). in practice, this means that the construction of a working database imposes further problems on the trainee historian, providing at least as great an intellectual challenge as a more conventional piece of research. all this must be done within a rigorous framework including annotated code, referenced sources and textual and technical explanations of design strategy. this is an exercise which addresses fundamental methodological questions concerning the academic discipline of history. the application of computers to the study of history that has accompanied the shift in emphasis from the great men of the past to the masses, is well illustrated in a project based around census returns. however, further issues are raised, particularly by the use of statistics. some american historians, the 'cliometricians' (most notably robert w. fogel and stanley l. engerman), have argued that there are areas of human behaviour best understood as a system in which both variables and their relationships can be quantified (fogel and engerman, 1974; tosh, 1984, pp. 198-202). what could be a better basis for a discussion of their ideas than students using computers to formulate their own historical models and theories? and in the realms of more conventional history, a database project gives awareness of methodological debates. the selection and interpretation of historical data raises questions about metahistory itself; if the building up of a source of empirical evidence is demonstrably subjective, what of the wider implications for the concept of historical objectivity and the existence of the historical fact? the value of this kind of exercise in history is readily apparent. the added boon is that such a project equips the students with transferable skills of a valuable nature, both in programming and in writing documentation. for such purposes, the documentation produced should be of comparable length to a standard dissertation. as an example of what i have in mind, the following notes are a brief suggested outline 50 atrj volume 3 number 2 for a third-year dissertation project. databases are probably the most suitable exercise, but there are alternatives. more experienced programmers might like to attempt simulation programs to test some of the great what-ifs? of history, or interactive educational programs that quiz users (perhaps secondary-school pupils) on their historical knowledge and mark them appropriately. the possibilities are very wide. an example outline for a third-year software project based around the design and implementation of a historical database programming implemented paradox 4 database with data normalized to third normal form. documentation see sommerville (1989). requirements specifications functional system requirements these are the system services which are expected by the user of the system. since the user (in this case a historian) is uninterested in the technical detail, technical implementation concepts should be avoided in this document. in short, this document should be a straight, plain-english statement of what the system is required to do. non-functional requirements these set out the constraints under which the system must operate, for example all input being expressible using the ascii character set. these documents should be complete and consistent: that is, they should contain all the requirements, and no contradictions. formal specification this is a technical analysis of requirements, expressing which paradox functions will be utilized to meet the above specifications. this document may be finalized after a process of prototyping and requirements validation (see below). system design this is a detailed explanation of the design of the database, the tables implemented, relationships and key fields. the design strategy adopted for a database may be prototyping, where the designer experiments with different implementations to find the most suitable technical approach, and the approach which best fulfils the specified requirements. put simply, this allows the programmer to implement a database, discuss any shortcomings with potential users, then adapt or re-design the database until it is satisfactory. this is, perhaps, the most appropriate methodology for fledgeling programmers. after all, historical databases are not safety-critical. if at first students do not succeed, they can try again, with no harm done. so long as the process is adequately documented, improvisations, experiments, and thinking on the feet is to be encouraged. testing, validation and verification this is a full account of the strategies adopted in the testing of the finished database and the results of that testing. particular attention should be paid to verification (does this system meet the requirements of the user?) and validation (is the technical implementation of the system correct?). 51 gervuse philips dissertations and databases: the historian as software engineer operation and maintenance manual this documentation should instruct the user, in how to operate the system and, if necessary, make alterations and updates. again, since the system is to be used by historians rather than computer scientists, it should be non-technical, as far as possible. report the final document should be a full evaluative report on the implementation of the database. the student might like to take the following areas into consideration: • how successful was the implementation? • are there alternative design/implementation strategies which might produce a better system for the historian? • how useful are such databases to the historian? • what are the methodological problems associated with implementing databases of historical data? references fogel r.w. and engerman s.l. (1974), time on the cross: the economics of negro slavery, london, wildwood house. cab. 24/26 g.t.2052, 'notes on the morale of british troops in france as disclosed by the censorship, 13th september, 1917'. cab. 24/36 g.t.3044, 'the british armies in france as gathered from censorship, 12th december, 1917'. phillips, g. (1992), a quantitative analysis package for historians, m.sc. thesis, university college of wales, aberystwyth. sommerville, i. (1989), software engineering, london, addison-wesley. tosh, j. (1984), the pursuit of history, london, longman. 52 (page number not for citation purpose) 1 *corresponding author. email: sarah.wakes@otago.ac.nz research in learning technology 2023. © 2023 s. j. wakes and l. a. dunn. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 research in learning technology vol. 31, 2023 original research article i do, i understand: engaging distance and campus students in sustainability through active learning sarah j. wakes* and linda. a. dunn department of mathematics and statistics, university of otago, dunedin, otago, new zealand (received: 18 may 2022; revised: 24 october 2022; accepted: 12 november 2022; published: 12 january 2023) distance online learning connects students to education opportunities without having to be present at the institution offering the learning module. this case study involved development of a dual on-campus and distance course into a fully online course. it required a student-focused approach and an innovative application of learning technologies, additional resources and learning frameworks to encourage student engagement, independent learning and growth of critical-thinking skills. changes to the design of the teaching approach and the use of technology improved the quality and quantity of interaction and communication between staff–students and students–students and facilitated a hands-on learning experience for online students interacting asynchronously. student engagement with the course material, other students and teachers increased compared to previous years. additional resource videos, learning and assessment guides (written and video), and online field trips helped develop critical-thinking skills and connectedness of learning to real-world applications. recommendations are provided of learning approaches that could be used by other educators in different subject courses. keywords: learning technologies; online; asynchronous learning; learning resources; distance teaching; sustainability introduction asynchronous distance learning has become more popular, especially in a world wrestling with a pandemic (covid-19), but it has been the chosen method of teaching for many years by some educators (garrison 2003; laurillard 2012; tyler and zurick 2014). for students of online learning, the benefits include the ability to continue working on other commitments, not having to move to the location of the learning institute, and the associated financial burden and time out of their regular lives, which can accompany face-to-face learning systems (nieuwoudt 2020; stone 2017; stone and springer 2019). engagement of students is an issue when a course is taught online, both asynchronously and synchronously. in order to reduce attrition associated with on-line learning, it is vital to engage students as early as possible and keep them engaged (angelino, williams, and natvig 2007). however, it is important to mailto:sarah.wakes@otago.ac.nz http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2823 s. j. wakes and l. a. dunn 2 citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 (page number not for citation purpose) note that higher engagement cannot be gained using just any form of learning activity (dixson 2010). good communication with students and the relationship between technology and pedagogy are important (dixson 2010; hege 2011). often online courses are delivered over a shorter time period (i.e. are more intensive) than face-to-face learning but allow flexibility as to when students access the learning and engage in the activities and assessment work (devlin and mckay 2016; mimirinis and bhattacharya 2007; nieuwoudt 2020; vlachopoulos, jan, and lockyer 2019). however, despite the advantages of online learning, problems have been noted, such as feeling isolated due to reduced or lack of connectedness with instructors and fellow students (stone and springer 2019; tanis 2020), motivation and time management due to work and personal commitments (greenland and moore 2014; kebritchi, lipschuetz, and santiague 2017), and technology and course design-related difficulties and anxiety (diep et al. 2019; mimirinis and bhattacharya 2007). for online courses to be effective for students and encourage participation and learning, design of the course needs to move from a teacher-centred to a student-centred format with students having more responsibility for their learning and success (annansingh 2019; mimirinis and bhattacharya 2007). a student-centred course format helps students build understanding from the course information and through their own experiences and learning (annansingh 2019). on-line lectures and assistance videos allow students to revisit the teaching material multiple times to reinforce what they are learning (nicol 2021). this requires students to take responsibility for engagement (baran and correia 2009) along with facilitation by the educator to encourage student engagement with the content and context, educator and other students in order to achieve the learning outcomes (vlachopoulos, jan, and lockyer 2019). online learning technology alters the role of the educator (devlin and mckay 2016, p. 101), and an online course is not simply a face-to-face course adjusted to an online platform but must be developed with the technology, and student engagement and participation in mind using a range of resources and multimedia (devlin and mckay 2016; kebritchi, lipschuetz, and santiague 2017). short lecture videos; un-assessed quizzes; and short supporting videos, readings and activities provide a more useable learning environment and aid student understanding (delotell, millam, and reinhardt 2010; stone and springer 2019), as do discussion forums, case studies and real-world examples and experiences (annansingh 2019; martin and bolliger 2018). a range of resources, assessments and approaches to presenting information and encouraging student engagement enables the content to address student’s differing learning goals, experiences, cognitive processes, interests and motivation through acquisition, inquiry, practice, production, discussion and collaboration types of learning (laurillard 2012, p. 96). choosing a learning platform and other multimedia that are familiar to students, and easy to navigate, will assist in reducing technology barriers to learning (devlin and mckay 2016). this requires a course that has been developed specifically for online learning rather than developed similar to or out of a face-to-face course (stone and springer 2019; vlachopoulos, jan, and lockyer 2019). the learning environment needs to deliver the flexibility students require and a mode of operating that aids students in engaging with the course content and people (devlin and mckay 2016). the online learning platform needs to facilitate quality interactions and learning (nieuwoudt 2020). students may not be familiar with all technologies used by the educator, http://dx.doi.org/10.25304/rlt.v31.2823 research in learning technology citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 3 (page number not for citation purpose) and this can be a barrier to learning and engagement, so inclusion of technical support is required (diep et al. 2019). students will require clear course objectives, content, assessment requirements and expected outcomes (dai 2019; kebritchi, lipschuetz, and santiague 2017; kuiper, solomonides, and hardy 2015; stone 2017; tanis 2020) along with appropriate technology support and guidance, in order to be well informed and have confidence and competence (diep et al. 2019; kebritchi, lipschuetz, and santiague 2017; nieuwoudt 2020) as they engage in online self-directed learning. this content must be well structured, and assessments and communication need to develop enquiry, exploration and meaningful connection to help develop students’ deep learning (annansingh 2019; delotell, millam, and reinhardt 2010; stone 2017). for students to be assured of the presence and support of the teacher, educators can facilitate student learning through weekly reminders and encouragement (written, audio and/or video) (kuiper, solomonides, and hardy 2015; martin and bolliger 2018; stone 2017; tanis 2020). this requires staff to be well organised and disciplined in initiating regular contact with students (tanis 2020). in addition, educators need to be available during times that suit students best (e.g. evenings), rather than just during normal work hours, to answer enquires or engage in discussion (stone and springer 2019). some element of synchronous learning (e.g. real-time online discussion forums or zoom meetings) can aid in this sense of presence (tyler and zurick 2014). all these forms of engagement create an online presence, personalise the experience, help students to feel known, supported and therefore more likely to engage in communication that will facilitate their learning (devlin and mckay 2016; nieuwoudt 2020; stone 2017; stone and springer 2019). this must be balanced with the increase in student email messages that can be overwhelming for the educator, and requests for a response at times not acceptable to the educator. therefore, student’s expectations have to be managed without impacting on their engagement and learning (annansingh 2019; nicol 2021). there are many strategies for engaging students in active online learning; however, how to implement them is less obvious. the rules identified by nordmann et al. (2020) encompass many of the strategies utilised in this study, although we were not aware of them at the time. this study seeks, through use of technology, and changes into the design of the learning unit, to trial a number of approaches to increase engagement of students with course material, staff and each other, and to gauge the success of these strategies. a further aim was to identify learning approaches that could be used by other educators in different subject areas. method: the case study at a university in new zealand, a course in the sustainability of materials is offered as a compressed summer school distance option. it is a stand-alone learning unit, not associated with any particular discipline or course of study, available to all students at the university. sustainability of materials is a fast-changing subject area that requires a system thinking approach to understand the complex interactions between the various system components. peer learning and support is a valuable part of the course. it was initially available on campus (~third of students) and distance (~two-thirds of students) from 2018 to 2020, but distance only from 2021. with student numbers doubling from 2018 to 2019, it became apparent that greater engagement of online students with staff and each other was needed. in addition, http://dx.doi.org/10.25304/rlt.v31.2823 s. j. wakes and l. a. dunn 4 citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 (page number not for citation purpose) the teaching approach in 2019 was becoming more time consuming. the crux of the problem was that students required a learning experience where student–student and student–staff interactions built a shared learning platform, where all felt enabled to contribute. students also needed to behave as one cohort and not one split by geographical boundaries. the subject requires students to engage in the practice, not just theory of sustainability, so there was a need to take the classroom outside the classroom. this, however, posed difficulties for online learning. interventions were trialled to improve these interactions in 2020 and 2021, with changes for 2021 informed by data and feedback from 2020. for the first 3 years (2018, 2019 and 2020), the material was presented linearly, following the weekly schedule of on campus seminars, albeit delivered and recorded as lecture fragments. prior to the 2020 covid-19 pandemic, the decision was made for the course to only be offered only online in 2021 due to low on-campus student numbers. additionally, using data obtained from 2020 on engagement of the students with each other and the course material, the means of delivering the course was altered to be asynchronous with course content fitted into a cyclic lifecycle model. this move away from a linear, progressive structure allowed students to ‘dip in and out’ of the course material as they undertook their own learning and worked on the various assignments and marked discussion boards. students could access the course material when and in whatever sequence they chose. all course material was made available online from the start, along with a detailed course book outlining the course material, an overview road map and suggested timeline recommending when course material should be read/watched and assignment work started to meet due dates. weekly update videos were recorded by teaching staff and uploaded, along with a weekly ‘check in’ email (agins 2000) to provide encouragement and guidance to student through the course. staff were also available via email and zoom. in addition, a weekly 2 hour evening open zoom session was held for students to ‘drop in’ and ask questions, and a variety of meaningful interaction activities were developed to help create social presence (dixson 2010). other changes made were as follows. resource videos resource videos for the course were produced to provide students with additional information on support, technology and assignments. clear course, and assignment requirements and expectations are highly important to students (sheridan and kelly 2010), and the videos (along with written information) helped provide a visual explanation, not too dissimilar to a face-to-face classroom setting and often involved a demonstration of a process or where information could be found. each video was kept short (generally 2–5 min) and to a specific topic, rather than one large video covering multiple things, to aid the students in finding them quickly and only needing to access the information they required. information about zoom technology was in the form of five short videos, each dealing with one specific aspect of zoom (table 1). teaching staff produced short videos that provided information on the course, components of blackboard used in the course and assignments (table 1). the videos produced by teaching staff were intentionally not made professionally and had a more spontaneous ‘live’ feel. it was hoped the students would understand from this that there was not an expectation of high-quality recorded assignments for their recorded assessments. the ‘at desk’ videos showed the educator in their office; a usual environment for in-person student-staff meetings. http://dx.doi.org/10.25304/rlt.v31.2823 research in learning technology citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 5 (page number not for citation purpose) introduction and weekly videos teaching staff recorded short (2–3 min) videos introducing themselves and their sustainability challenge, so that students got to ‘meet them’ and understand that the teachers were on a sustainability journey with the students. it was hoped that these videos would build connections with staff early on in order to help keep them engaged and reduce attrition (angelino, williams, and natvig 2007). each week a video was uploaded (along with accompanying video) reminding students of what they should be doing based on the learning schedule, advising of any new resources and providing encouragement. sustainability challenge – marked discussion board assignment students made short, guided videos highlighting an issue based on their sustainability challenge and shared them in a marked discussion board assignment as a way of engaging in the practice of sustainability and connecting to the theory being taught. students were expected to contribute comments and suggestions on other videos to develop peer learning and support. students were able to use current technology and applications (e.g. smartphones and zoom) to record, so it was easy to implement and in their comfort zone. changes made for 2021 involved the addition of a final review/ feedback summary of the student’s individual challenge, as well as the discussion table 1. 2020 resource videos (length min:sec) produced to assist student skill development and understanding. econferencing general mats204 information discussion boards assignments introduction to zoom (2:05) course outline (3:01) [coursebook] discussion board 1: sustainability challenge (4:51) submitting assignments on blackboard (1:29) download and install zoom (1:06) seminar schedule (1:48) [2021 suggested timeline] discussion board 2: future materials and products (1:42) [2021 guest lectures] assignment 1: presentation and marking (4:39) zoom on mobile devices (1:54) study readiness (3:51) discussion board 3: field trip (2:09) assignment 1: presentation workshop (48:29) how to join a zoom meeting (1:54) mats204 blackboard introduction (3:59) discussion board 4: waste audit (2:54) [not in 2021] assignment 1: how to record a presentation on zoom (5:12) how to share a screen on zoom (1:23) accessing e-reserve (1:05)   assignment 1: peer marking of presentations (3:06)   general discussion boards (2:48)   assignment 2: material analysis and life cycle mapping (7:06)   accessing echo360 (1:59)   assignment 3: report (3:05)   staff introduction videos (various; 1–3 min)     note: changes made in 2021 are noted in brackets [ ]. http://dx.doi.org/10.25304/rlt.v31.2823 s. j. wakes and l. a. dunn 6 citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 (page number not for citation purpose) board having a grade, formative feedback, which was given by staff who participated in the discussions. field trips – marked discussion board assignment in 2020, staff live streamed and recorded two field trips to local businesses. although group interaction was not possible, each student had a field trip experience whether on campus or distance, and the ability to respond to this via the discussion board was designed to help them feel that their experience and voice were important (zilka, cohen, and rahimi 2018). consequently, in 2021, staff pre-recorded the field trips with the number of field trips increasing to add variety. students then had to comment on issues relating to social and/or environmental sustainability practices of one of these organisations. guest lecturers – marked discussion board assignment experts in areas of materials sustainability contributed pre-recorded seminars. in 2021, additional guest lectures were added to widen the breath of topic areas covered by guests. a marked discussion board asked students to consider two of the guest lectures and comment on how they related to issues being addressed in the course. online marking of presentations an essential aspect built into the recorded presentation assignment was a requirement to mark five other students’ presentations. peer marking and feedback assist students in their learning because they need to understand the assignment requirements and marking criteria, and provide them with a greater array of comments that encourage and help develop motivation and self-esteem (nicol 2008). the qualtrics survey platform for students to do peer marking, with a rubric and space for constructive and supporting comments, was introduced in 2020. this involved using a rubric for the marking and giving constructive, supportive feedback. a rubric is useful in helping students understand the marking parameters and provides useful feedback (tanis 2020). a time period was allowed for the marking process, so that students could work at a time more suited to them. the final mark for the presentation was then a combination mark from staff and students moderated by staff with comments collated and returned to the students. in 2018 (23 students) and 2019 (43 students), the template used was a word document, which students completed for each presentation. a time-consuming collation process then had to be carried out. in 2020 (47 students) and 2021 (104 students), a qualtrics survey was used as the tool to record the marks and comments, and an r analysis tool averaged the marks and collate the comments. the reduction in workload was considerable and freed staff from a tedious task. evaluation of changes data were gathered relating to student engagement with course seminar and video content on echo360 and blackboard. additionally, a survey (with appropriate ethical approval in line with the host university human ethics committee category b http://dx.doi.org/10.25304/rlt.v31.2823 research in learning technology citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 7 (page number not for citation purpose) approval, which is at departmental level) was emailed to all students following the release of final marks in 2020 and following the completion of the course in 2021. the anonymous survey consisted of multiple-choice responses and written responses. written comments were grouped based on dominant themes and then grouped into categories (james 2006). the purpose of the survey was to gauge how students perceived the effectiveness of aspects of the course: that is, assignments and discussion boards; the means in which the course material was presented; engagement with the course topic; contact with staff; development of their learning; engagement with each other. in 2020, the post-course survey was made available for 5 weeks, with three reminders sent out and 16 out of 44 students completed it. in 2021 only, 20 out of the 105 students actually completed the survey. due to time constraints to meet the funding report deadline, the survey was only made available for 2 weeks, and only one reminder was sent. results and discussion engagement with the learning documents course content in 2020, and earlier years, was made available to all students in a linear progression, following the pattern of on-campus delivery of seminars. there was a noticeable drop in engagement with course content in 2020 over the 6 weeks of the course (table 2). the implication was that the course content in the latter weeks was not seen as important, especially as the course had no final exam. in 2021, as a distance only course, all content was provided at the start of the course. the cyclic nature of sustainability and the modular course design lended itself to being engaged with multiple entry points. in addition, a suggested timeline on how to engage with the content was provided to give structure to those requiring it. survey results indicated that student engagement with the resources, staff, guest, field trip and lecture content videos was higher and did not drop so quickly as the course progressed (table  2) possibly because students were accessing content in a different order from previous years because it provided a student-centred format for learning that encouraged them to take control of their own learning process (annansingh 2019). students had more table 2. student accessing/viewing of mats204 all course videos via echo360 each week of summer school (includes lecture content, resource, field trip, staff and guest videos). ending of summer school week… total access/views to videos and slides on echo360 2020 – 46 students total access/views to videos and slides on echo360 2021–105 students % of total access/ views to videos and slides on echo360 (2020) % of total access/ views to videos and slides on echo360 (2021) 1 1488 3553 42.3 30.2 2 477 2566 13.6 21.8 3 651 1809 18.5 15.4 4 269 1789 7.7 15.2 5 430 843 12.2 7.2 6 201 1200 5.7 10.2 total 3516 11 760 100% 100% http://dx.doi.org/10.25304/rlt.v31.2823 s. j. wakes and l. a. dunn 8 citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 (page number not for citation purpose) control over what they viewed and when, according to their learning patterns and available times for learning (devlin and mckay 2016; stone and springer 2019). student access of blackboard was assessed by weekday over the 6 week period; the average time viewed per number of students who accessed blackboard that weekday, and the longest access time of any one student are given in table 3 for 2020 and 2021. in 2021 (cyclic course design), student engagement with blackboard content was for a longer duration, compared with 2020 (linear course design) for every day over the whole 6-week period. in addition, the lowest–highest view time for a specific blackboard access event increased in 2021 (9.4–18.6 min) compared to 2020 (5.8–13.9 min). in 2021, actual access of the lecture content videos compared with the suggested timeline indicated that access of the content on echo360 did align with it (table 4). students also accessed course content before and after the suggested time. hence, 2021 students were engaging with the course content more and were motivated to take table 3. student access of blackboard 2020 and 2021.     blackboard access over 6 weeks 2020 (46 students)     blackboard access over 6 weeks 2021 (105 students) total hours per day number of viewers on day hours per viewer on day highest view time (min) on 1 day total hours per day number of viewers on day hours per viewer on day highest view time (min) on 1 day monday 82.47 45 1.8 5.8 229.71 95 2.4 9.7 tuesday 86.701 45 1.9 7.8 196.78 92 2.1 9.4 wednesday 88.79 40 2.2 10 203.47 99 2.1 10.6 thursday 65.26 44 1.5 12.8 220.02 91 2.4 15.8 friday 88.4 44 2 13.9 234.79 97 2.4 18.6 saturday 46.7 39 1.1 6.4 138.36 74 1.9 14.3 sunday 73.64 46 1.6 6.3   233.37 92 2.5 17 table 4. number of views of lecture content videos 2021 on echo360 (excludes resource, staff, guest and field trip videos) compared with the suggested order (guide) of viewing lecture content videos.     overview extraction processing manufacture use disposal guide actual guide actual guide actual guide actual guide actual guide actual week 1   2350   664   29 1   1 1 week 2   707   930   242 28   17 1 week 3   89 366   492   298   52 5 week 4   37 14   10   486   145 20 week 5   56 20   40 13   303   36 week 6   66 26   2 98   42   97 total   3046   1925   1302   712   468   333 http://dx.doi.org/10.25304/rlt.v31.2823 research in learning technology citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 9 (page number not for citation purpose) responsibility for learning rather than simply doing things in the way indicated by teaching staff (annansingh 2019; devlin and mckay 2016; stone and springer 2019). table 5 presents the timeline of 2021 students viewing the resources, guest lectures and field trip videos. it was not possible from the data collected to know if all students watched each video; however, some students began watching content well ahead of the suggested timeline and when it was needed for assignments, although a large number also left watching some videos to the final week. the eight guest lecture videos were viewed on average 110.9 views per video, implying that most students watch all of the videos, even though only watching two of the eight was required. this pattern was repeated with the field trip videos (average 52.3 views per video) when comment was required on only one. there was an average of 110.7 views per video over the 10 assessment help videos. assessment guidance provided the minimum engagement with the course resources many students chose to view more than this, indicating that the course design encouraged deeper engagement (annansingh 2019; diep et al. 2019). over the covid-19 lockdown in 2020, university teaching in new zealand went online, and it is probable that during this time, many students had to access classes/ tutorials via zoom and so picked up these skills during that period. this is reflected in the questionnaire results in 2021 when fewer students accessed the resource videos relating to use of zoom compared to 2020 (~50% compared to 75%). students bring skills and experience to the learning environment, and these need to be taken into account, whilst being mindful that some students will have lower literacy related to the digital platforms utilised (devlin and mckay 2016; laurillard 2012; stone and springer 2019). hence, care must be taken to not provide resources that are not required as they can clutter the course content to and add to a sense of being overwhelmed whilst at the same time being mindful that some students will have limited literacy related to the digital platform utilised (annansingh 2019; mimirinis and bhattacharya 2007). it would be ideal to monitor the value of still providing the zoom resource videos. ease of access and use of teaching resources the peer marking of the recorded presentation assignment not only helped students learn about five other topics from their peers but also helped them connect with other students through their encouraging comments. additionally, from an educator table 5. timeline of views of resource, field trip, staff and guest videos 2021 (on echo360). zoom assessment help guest lecturer field trip staff introductions week 1 81 229 98 0 81 week 2 39 253 142 30 21 week 3 91 335 63 37 0 week 4 1 131 95 66 0 week 5 0 81 120 162 0 week 6 0 145 377 333 0 total 212 1174 895 628 102 http://dx.doi.org/10.25304/rlt.v31.2823 s. j. wakes and l. a. dunn 10 citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 (page number not for citation purpose) perspective, the qualtrics survey and associated rubric reduced the time teaching staff spent collating marks and comments, and feedback was given to students more quickly. comments from respondents to the post-course survey in 2020 and 2021 included: (1) the process – easy to use, clear, very effective and fast, straightforward, simple and handy; (2) the experience – found this interesting, enjoyed doing this, watched the videos and read the emails before doing this. the introduction of unfamiliar forms of assessment requires clear instruction to facilitate understanding and use (diep et al. 2019; kuiper, solomonides, and hardy 2015). the feedback from students indicates that the qualtrics program, resource videos and written assessment guide, collectively resulted in students engaging positively with the peer-making experience. usefulness of supporting resources educators often produce all manner of supporting resources that they consider useful to assist students in their learning. however, it is important to determine just how useful these resources are, otherwise they become unnecessary clutter and may result in preventing students from actually accessing them and possibly other more important resources. over 90% of students who responded in the 2021 survey said that the guidance provided from supporting resources was useful, and all of them (100%) were able to navigate the course material, despite it all being made available at the start of the course. in 2020, the course book and seminar schedule were considered helpful by 87.5% of students who responded to the survey. in 2021, with the move of the course to fully distance, the course book and suggested timeline were both considered helpful by 98.7% and 92.3% of the students, respectively. written comments about the suggested timeline included: helpful, and easy to follow and provided good structure to follow. in addition, assignment and marked discussion board resource videos (table 1) were considered helpful to engage with the topic by the majority of students. a resource video on study readiness was noted as being helpful in 2021 (81%) by a greater percentage of students compared with students in 2020 (60%), and fewer students said they did not access the video in 2021 (19%) compared to 2020 (33%). assignment and marked discussion board resource videos were considered helpful by a large percentage of students (over 85%) in both years. the presentation workshop resource video was available to assist students with presentation skills and preparing a quality presentation format. in 2020, 94% of student respondents said the video was helpful, and in 2021, by 69%. the road map provided in 2021 was helpful for 76.9%, which indicates there is scope for developing the road map to increase usefulness to students. it would be ideal to monitor the value of these resource videos to determine student engagement with them and assess the value in continuing to make them available. in addition, thought needs to be given to alternative or additional resources that will support students’ deep learning (devlin and mckay 2016; knightley 2007) as student access of those currently made available changes. http://dx.doi.org/10.25304/rlt.v31.2823 research in learning technology citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 11 (page number not for citation purpose) examples of student/student engagement for the sustainability challenge, students made videos talking about their challenge assignment, and these videos were noted by over 85% of respondents as being helpful in getting to know others doing the course. commenting on other students’ challenge videos also aided engagement with others and helped over 75% of respondents feel part of a learning community. the challenge assignment assisted in connecting students and increased their sense of being part of a group (annansingh 2019; stone 2017; tanis 2020), despite all being distance learners. examples of student/educator engagement staff communicated with students via weekly emails and videos to guide learning and encourage students, as well as being available via email, phone and zoom when students chose to make contact (martin and bolliger 2018; stone and springer 2019; tanis 2020). examples of student comments about the interaction/communication between staff and students indicate that students did feel able to engage with staff: easy to connect with them; regular email communication was great; staff were very approachable; replied swiftly to emails; i felt confident reaching out. weekly open zoom sessions were also available in 2021 as suggested by students in 2020. these drop-in zoom sessions were available one night each week during summer school, and teaching staff were available for a designated 2 hour period. very few students made use of the sessions despite them being notified of them through the course book, video communication and emails. those who attend said they found it helpful. interestingly, in the 2021 post-course survey, zoom sessions were suggested again. hence, more work is required to increase the availability and awareness of these drop-in zoom sessions, so that a real-time presence is experienced by those requiring this form of contact (tyler and zurick 2014). examples of critical thinking and connection to ‘real world’ and own experiences the inclusion of guest speakers and online field trips provided real-life experiences. real-life experiences facilitate student deep learning, motivate students to engage proactively with the course content and encourage them to explore learning beyond provided course content and the ‘classroom’ (annansingh 2019; diep et al. 2019; martin and bolliger 2018; stone and springer 2019; tanis 2020). the individual student challenge assignment required students to engage in the challenge for the duration of the 6-week intensive course. student summaries (part of the assessment) indicated that the majority intended to continue with the changes made during the challenge. most respondents (92%) said that guest lecturers helped them engage with the topic and broaden their understanding because of the expertise they provided. examples of student comments about what guest lecturers brought were as follows: specific/ relevant knowledge or experience; unique, outside perspectives and specialties; different ways of explaining/teaching things; their area of expertise/interest; helped fill knowledge gaps; provided different insights; real-life examples. although field trips were pre-recorded and online, the experience still helped the students to get an insight into practices in local businesses and have a greater understanding of practices within a business/organisation. examples of student comments http://dx.doi.org/10.25304/rlt.v31.2823 s. j. wakes and l. a. dunn 12 citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 (page number not for citation purpose) about these online field trips were as follows: was unaware of these great businesses; good to see things happening in real-life; often in papers you learn about the big scale but don’t narrow it down to the local scale; good to see how different aspects integrate together. relevant feedback that indicates what ‘worked’ and what could be improved all information and course material were made available from the start of summer school, which was a bit overwhelming for many, which potentially could hinder student learning (annansingh 2019; mimirinis and bhattacharya 2007). nearly 60% said that initially, the amount of course work seemed overwhelming, but as they progressed through the course, this dropped to 40%. students did indicate that the sense of being overwhelmed had reduced part-way thorough the course for many of them, but around 40% were still feeling that way. the suggested timeline of what to read/ watch/do each week was considered helpful by all. although these findings indicate that more needs to be done to reduce this sense of being overwhelmed, post-course survey responses indicate that a student dictated learning schedule, guided by a suggested timeline, worked well for the majority of students. the online learning processes and resources documented in this case study provide insight into the activities and technologies used for distance online learning. implementing a successful online distance programme involves continual development and refinement. the authors’ recommendations to increase student–student and staff–student engagement, student engagement with learning, and student engagement with practice and critical thinking, are listed in table 6, along with examples of how these have been applied. conclusion the decision to move a campus/distance taught course to being fully distance (online) involved developing aspects of the learning modules to encourage students to take more responsibility and control of their learning, so as to develop the necessary critical-thinking skills required to take learned knowledge and experiences into future learning and action. all course content was made available to students from the start of the course to give students autonomy as to when they accessed the learning material and engaged with the content that connected students to each other and teaching staff. the course content was developed to assist students in engaging with each other, teaching and guest staff, and with the depth of information available through the internet. student engagement with the learning resources increased in hours of engagement and was more consistent over each week of the course. student feedback indicated that the course content helped them to get to know other students, feel part of a learning community, engage proactively with the course content and connect their learning to their previous and course directed experiences. it is hoped the recommendations resulting from this case study will assist other educators in developing new ways to engage with students to enhance student motivation, learning outcomes, and their application of acquired knowledge and critical thinking to lifetime learning. the examples were ways of implementing the recommendations given, are adaptable to many types of courses and are not limited to only those that are online or taught at a distance. a key message is that getting to a level of http://dx.doi.org/10.25304/rlt.v31.2823 research in learning technology citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 13 (page number not for citation purpose) communication with students where they feel supported and encouraged rather than lost or overwhelmed by information can take time but can reap multiple benefits for students and staff. suggested further research • develop online ‘field trips’ to help students engage with real-world situations and gauge student level of connection through a suitable question as part of the assignment. • consider ways students could work in groups via technology to engage in a field trip created/videoed by the group. currently, this is difficult from many perspectives: finding technology readily available that allows group collaboration of video; designing the assignment in a way that ensures equitable availability and table 6. recommendations. recommendation examples communicate regularly with students to increase approachability, sense of presence/connection and availability (staff–student engagement) • staff introduction videos; • staff engaged in the sustainability challenge assignment; • assignment ‘how to’ videos; • weekly guidance and encouragement videos; • weekly open/drop-in zoom sessions facilitate opportunities for students to interact online and build connection as a learning community (student–student engagement) • sustainability challenge videos and comments; • peer marking and feedback on presentations; • marked discussion boards. ensure learning resources assist engagement and learning • provided all information and learning resources at start – students could engage with it as and when preferred; • roadmap giving visual overview; • suggested timeline for engagement with learning resources and timing of assignments; • resource videos on relevant technology; • resource videos on accessing teaching material; • assignment ‘how to’ resource videos; • weekly guidance and encouragement videos; • use of qualtrics for peer marking and feedback; • guests lecturers; • sustainability challenge enabled students to engage with the learning through their own practice/realworld application. opportunities to engage with real world examples and facilitate critical thinking and application of learning • pre-recorded online field trips; • guest lectures from specialists/practitioners; • assessment that required consideration of connections between theory and practice, for example, sustainability challenge, field trip and guest lecturer marked discussion boards. http://dx.doi.org/10.25304/rlt.v31.2823 s. j. wakes and l. a. dunn 14 citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 (page number not for citation purpose) student involvement; ensuring the work has value to student learning and is not just a teaching/assignment exercise. acknowledgement funding was provided through a university of otago calt teaching development grant. references agins, t. (2000) the end of fashion: how marketing changed the clothing business forever, harper collins publishers inc., new york. angelino, l. m., williams, f. k. & natvig, d. (2007) ‘strategies to engage students and reduce attrituin rates’, the journal of educators online, vol. 4, no. 2, pp. 1–14. doi: 10.1007/ s10639-019-09949-5 annansingh, f. (2019) ‘mind the gap: cognitive active learning in virtual learning environment perception of instructors and students’, education and information technologies, vol. 24, no. 6, pp. 3669–3688. doi: 10.1007/s10639-019-09949-5 baran, e. & correia, a. p. (2009) ‘student-led facilitation strategies in online discussions’, distance education, vol. 30, no. 3, pp. 339–361. doi: 10.1080/01587910903236510 dai, y. (2019) ‘situating videoconferencing in a connected class toward intercultural knowledge development: a comparative reflection approach’, the internet and higher education, vol. 41, pp. 1–10. doi: 10.1016/j.iheduc.2018.11.001 delotell, p. j., millam, l. a. & reinhardt, m. m. (2010) ‘the use of deep learning strategies in online business courses to impact student retention’, american journal of business education, vol. 3, no. 12, pp. 49–55. doi: 10.19030/ajbe.v3i12.964 devlin, m. & mckay, j. (2016) ‘teaching students using technology: facilitating success for students from low socioeconomic status backgrounds in australian universites’, australasian journal of educational technology, vol. 32, no. 1, pp. 92–106. doi: 10.14742/ ajet.2053 diep, a. n., et al., (2019) ‘adult learners’ needs in online and blended learning’, australasian journal of adult learning, vol. 59, no. 2, pp. 223–253. doi: 10.3316/ielapa.592385127057281 dixson, m. d. (2010) ‘creating effective student engagement in online courses: what do students find engaging?’, journal of the scholarship of teaching and learning, vol. 10, no. 2, pp. 1–13. garrison, d. r. (2003) ‘cognitive presence for effective asynchronous online learning: the role of reflective inquiry, self-direction and metacognition’, elements of quality online education: ptractie and direction, vol. 4, no. pp. 47–58. greenland, s. j. & moore, c. (2014) ‘patterns of student enrolment and attrition in australian open access online education: a preliminary case study’, open praxis, vol. 6, no. 1, pp. 45–54. doi: 10.3316/informit.935906696497074 hege, b. a. (2011) ‘the online theology classroom: strategies for engaging a community of distance learners in a hybrid model of online education’, teaching theology and religion, vol. 14, no. 1, pp. 13–20. doi: 10.1111/j.1467-9647.2010.00668.x james, a. (2006) ‘critical moments in the production of ‘rigorous’ and ‘relevant’ cultural economic geographies’, progress in human geography, vol. 30, no. 1, pp. 289–308. doi: 10.1191/0309132506ph610oa kebritchi, m., lipschuetz, a. & santiague, l. (2017) ‘issues and challenges of teaching successful online courses in higher education: a literature review’, journal of educational technology systems, vol. 46, no. 1, pp. 4–29. doi: 10.1177/0047239516661713 knightley, w. m. (2007) ‘adult learners online: students’ experiences of learning online’, australian journal of adult learning, vol. 47, no. 2, pp. 264–288. http://dx.doi.org/10.25304/rlt.v31.2823 http://dx.doi.org/10.1007/s10639-019-09949-5 http://dx.doi.org/10.1007/s10639-019-09949-5 http://dx.doi.org/10.1007/s10639-019-09949-5 http://dx.doi.org/10.1080/01587910903236510 http://dx.doi.org/10.1016/j.iheduc.2018.11.001 http://dx.doi.org/10.19030/ajbe.v3i12.964 http://dx.doi.org/10.14742/ajet.2053 http://dx.doi.org/10.14742/ajet.2053 http://dx.doi.org/10.3316/ielapa.592385127057281 http://dx.doi.org/10.3316/informit.935906696497074 http://dx.doi.org/10.1111/j.1467-9647.2010.00668.x http://dx.doi.org/10.1191/0309132506ph610oa http://dx.doi.org/10.1177/0047239516661713 research in learning technology citation: research in learning technology 2023, 31: 2823 http://dx.doi.org/10.25304/rlt.v31.2823 15 (page number not for citation purpose) kuiper, a., solomonides, i. & hardy, l. (2015) ‘time on task in intensive modes of delivery’, distance education, vol. 36, no. 2, pp. 231–245. doi: 10.1080/01587919.2015.1055058 laurillard, d. (2012) teaching as a design science: building pedagogical patterns for learning and technology, routledge, new york. martin, f. & bolliger, d. u. (2018) ‘engagement matters: student perceptions on the importance of engagement strategies in the online learning environment’, online learning journal, vol. 22, no. 1, pp. 205–222. doi: 10.24059/olj.v22il.1092 mimirinis, m. & bhattacharya, m. (2007) ‘design of virtual learning environments for deep learning’, journal of interactive learning research, vol. 18, no. 1, pp. 55–64. nicol, d. (2008) technology supported assessment [online]. available at: https://www.reap. ac.uk/portals/101/documents/reap/technology_supported_assessment.pdf nicol, d. (2021) ‘promoting dialogue in technology-enhanced learning’, in early careers in education: perspectives for studentsa and nqt, eds a. gillespie, emerald publishing limited, bingley. pp. 133–142. nieuwoudt, j. e. (2020) ‘investigating synchronous and asynchronous class attendance as predictors of academic success in online education’, australasian journal of education technology, vol. 36, no. 3, pp. 15–26. doi: 10.14742/ajet.5137 nordmann, e., et al., (2020) ‘ten simple rules for supporting a temporary online pivot in higher education’, plos computing biology, vol. 16, no. 10, pp. 1–18. doi: 10.1371/journal.pcbi.1008242 sheridan, k. & kelly, m. a. (2010) ‘the indicators of instructor presence that are important to students in online courses’, merlot journal of online learning and teaching, vol. 6, no. 4, pp. 767–779. stone, c. (2017) ‘opportunity through online learning: improving student access, participation and success in higher education’, national centre for student equity in higher education [online], available at: https://apo.org.au/node/94591 stone, c. & springer, m. (2019) ‘interactivity, connectedness and “teacher-presence”: engaging and retaining students online’, australian journal of adult learning, vol. 59, no. 2, pp. 146–169. tanis, c. j. (2020) ‘the seven principles of online learning: feedback from faculty and alumni on its importance for teaching and learning’, research in learning technology, vol. 28, https://journal.alt.ac.uk/index.php/rlt/article/view/2319 tyler, j. & zurick, a. (2014) ‘synchronous versus asynchronous learning: is there a measurable difference?’, proceedings of the 2014 institute for behavioural and applied management conference, orlando, fl. pp. 52–54. vlachopoulos, p., jan, s. k. & lockyer, l. (2019) ‘a comparative study on the traditional and intensive delivery of an online course: design and facilitation recommendations’, research in learning technology, vol. 27, no., pp. doi: 10.25304/rlt.v27.2196 zilka, g. c., cohen, r. & rahimi, i. d. (2018) ‘teacher presence and social presence in virtual and blended courses’, journal of information technology teaching: research, vol. 17, pp. 103–126. doi: 10.28945/4061 http://dx.doi.org/10.25304/rlt.v31.2823 http://dx.doi.org/10.1080/01587919.2015.1055058 http://dx.doi.org/10.24059/olj.v22il.1092 https://www.reap.ac.uk/portals/101/documents/reap/technology_supported_assessment.pdf https://www.reap.ac.uk/portals/101/documents/reap/technology_supported_assessment.pdf http://dx.doi.org/10.14742/ajet.5137 http://dx.doi.org/10.1371/journal.pcbi.1008242 http://dx.doi.org/10.1371/journal.pcbi.1008242 https://apo.org.au/node/94591 https://journal.alt.ac.uk/index.php/rlt/article/view/2319 http://dx.doi.org/10.25304/rlt.v27.2196 http://dx.doi.org/10.28945/4061 comparison of the cost-effectiveness of a computer-assisted learning program with a tutored demonstration to teach intestinal motility to medical students h. l. leathard* and d. g. dewhurst** * department of health studies and human science, st. martin's college, lancaster ** faculty of health and social care, leeds metropolitan university computer-based simulations of undergraduate experiments in pharmacology and physiology may offer a cost-effective alternative to the traditional live laboratory for some students, for whom laboratory skills are less important. here we describe a study which compares two approaches to teaching preclinical medical students the pharmacology of colonic motility. half of one cohort received a tutored live demonstration of an isolated tissue laboratory, while the other half used a computer simulation program covering the same subject. the study demonstrated that student learning was comparable for both groups, that many students found the computer simulation an acceptable alternative and that the latter required significantly less resource. introduction the computer-assisted learning program intestinal motility was produced to help preclinical medical students to understand the variety of ways in which drugs or endogenous mediators can alter intestinal muscle movements and thereby influence propulsion of luminal contents. it simulates experiments actually carried out on a rat colon preparation, but the principles apply equally well to the small intestine and to the stomach and oesophagus of rat or other species. it can be used alone as a learning aid or can be used to support practical class teaching. the program consists of four discrete sections: introduction, method, demonstration and experiments, and the accompanying work booklet includes a set of student assignment questions together with model answers for tutors. introduction contains a description of the objectives of the program, followed by an illustrated guide (a combination of text and colour graphics) to the general structure and innervation of the intestines. methods uses a similar combination of text and graphics to describe the preparation itself, the experimental set-up and the apparatus used. demonstration uses 118 alt-} volume 3 number i animated graphics to show a physiological peristaltic response elicited by a brief, suprathreshold increase in intraluminal pressure and responses to intraluminal injections of test agents such as acetylcholine and adrenaline. this section is intended to give a visual impression of how the colon is seen to behave when the experiment is performed live. the main part of the program, experiments, allows investigation of the direct effects of, and alterations in, the reflex response caused by a range of substances when applied either alone or in combination e.g. saline (control), neostigmine, atropine, acetylcholine, carbachol, phenolphthalein, adrenaline. the program has been demonstrated to the british pharmacological society (dewhurst et al, 1992) here we present data from a study designed to determine the usefulness of the computer program (cal) as a stand-alone method evaluated by comparison with the conventional teaching method a tutored demonstration (dem)) using students for whom the program was originally designed, i.e. second-year preclinical students at the charing cross and westminster medical school in london. a brief account of part of the evaluation has also been presented to the british pharmacological society, but published only as an abstract (leathard et al, 1994). method the 1992-93 practical class 'circus' cohort of 156 students were randomized to one often groups, five dem and five cal groups (n = 14-16), for their intestinal motility practical. each group received a common introduction in which the purpose and procedures of the evaluation were explained. students were requested not to confer with members of the other group before handing in their work (on the fourth day after each class was held). they were assured that although their answers would be scored for the purpose of the evaluation, the scores would not contribute to their course-work mark (which would be determined separately), and that tutorial support, if needed, would be available prior to the practical course assessment. the completed assignments which were handed in by the appropriate deadline were photocopied. the originals were corrected and returned to the students. the unmarked copies were retained and scored for the evaluation exercise. five of the twenty questions answered by the students were scored (table 1). these were selected to test the following: students' extant knowledge; their understanding of the tissue preparation used; their data-interpretation ability; and the extent of their progress through a common sequence of tests. both the dem and cal groups each had a tutor present throughout the 90-minute class, and tutors were randomized to take dem and cal classes on equal numbers of different occasions so that their individual characteristics should not influence the outcome of the evaluation. students in both groups were advised to work through the various tests in the sequence given in the practical schedules that were provided, but the cal group were not constrained to use this sequence. this study compares: • the resources used for the two teaching methods; 119 h. l leathard and d. g. dewhurst comparison of the cost-effectiveness of a computer assisted learning program table i: comparison of the learning scores of students from the two groups ¡n the study. the dem group (n = 52) received a tutored demonstration of a live laboratory practical and the cal group (n = 53) received a computer simulation which allowed them to access typical results from a number of experiments performed on this preparation. student learning was assessed using an assignment comprising twenty questions which were selected to test the following: students' extant knowledge; their understanding of the tissue preparation used; their data interpretation ability; and the extent of their progress through a common sequence of tests. for this study five of the questions answered by the students were scored. learning scores are expressed as the percentage of students giving satisfactory answers to the questions. question qi explain the response elicited by the control injection of saline. q7 the cone, of acetylcholine injected into the lumen to stimulate longitudinal muscle in this preparation is 10,000 times the concentration required in a conventional organ bath. what are the possible explanations? q11 explain the mechanism of action of neostigmine and suggest whether cholinesterases in this preparation will remain inhibited for (a) minutes, (b) hours, or (c) days. q14 explain the recorded flow pattern (in the presence of phenolphthalein) in terms of circular muscle activity. q20 suggest ways in which the atropine-resistant component of the stimulant effect of phenolphthalein could be mediated. dem 95 70 86 63 34 cal 98 50 72 51 19 mean 97 61 78 56 26 • the students' learning; • the attitudes of students from both groups to pertinent aspects of computer-assisted learning. the attitude assessment was based on a slightly modified version of the questionnaire used previously by dewhurst et al. (1994). the first three questions sought information about the students, access to and familiarity with computers. twenty-five statements about pertinent aspects of computer-assisted learning were then given, and students were asked to indicate their level of agreement/disagreement with each statement using a fivepoint likert scale (strongly agree; agree; undecided; disagree; strongly disagree). a final open question invited any further comments the student might wish to make. results i student learning one hundred and twelve students submitted their answers by the stringent deadlines set for the evaluation. of these, fifty-two were from the cal group and fifty-three from the dem group, with seven failing to indicate the group they had belonged to. their learning 120 alt-j volume 3 number i scores are shown in table 1. while virtually all students gave satisfactory answers to the first question, subsequent questions were progressively less well answered, and the dem students performed slightly better than the cal students in each case. there was no statistically significant difference in the mean scores of the two groups. 2 resources as detailed in table 2, the human resources used for the dem students totalled fifty-six hours of technician's time and twenty-four hours of tutor's time. equivalent figures for the cal students were four hours and fifteen hours respectively. conservatively, these hours could be costed at £10 per hour, in which case the staff costs for the dem groups would be £800, and for the cal groups £190. the costs of materials were £130 for the cal groups and £60 for the dem groups. it should be noted that the cal costs are a table 2: comparison of the resources (staff and materials/consumables) required for the two groups in the study. the dem group were taught using a tutored demonstration of the live experiment in the laboratory, while the cal group used a computer simulation of the experiment staff resources are measured in hours of staff time while materials are costed in £ sterling. the equipment required for both approaches has not been considered since it was available from stock resources 1 staff resources (i) technical staff support advance preparation for class preparation on days of class (3 h on each of 5 days) technical support during class clearing up after class (ii) demonstrators (post-graduate students) advance preparation class supervision total 2 consumables/materials drugs, solutions, gases rats computer program intestinal motility (reusable) diskettes (reusable) total dem 16h 15h 15h 10h 4h 20h 80h £10 ' £50 0 0 £60 cal 2h 0 0 2h 0 15h 19h 0 0 £120 £10 £130 121 h. l leathard and d. g. dewburst comparison of the cost-effectiveness of a computer assisted learning program one-off expenditure since the program and disks may be used again. apparatus for both teaching methods was available from stock and has not been considered. 3 student attitudes one hundred students returned the attitudes questionnaire, of whom 40 belonged to the cal group, 48 to the dem group; 12 did not indicate to which group they had belonged. of these, 26 owned a computer, 82 stated that they had access to a computer, and 69 had some (albeit limited) previous experience of using a computer program as an alternative to performing a practical using live tissue (questions 1-3 on the questionnaire). in general, the views of students from each group were similar, so the main part of this analysis is based on the responses of all the 100 students, points where there was some divergence being addressed afterwards. in the following summary, where responses are grouped, the number of students giving a particular response is indicated in parentheses after the response. very small numbers of students responded 'strongly' to any of the statements. more than two-thirds of the respondents agreed or strongly agreed that 'computer simulations enable the student to control the pace of learning' (93), 'computer simulation programs allow acquisition of accurate results in the allocated time' (85) and 'students who need laboratory skills for their future careers can only obtain these skills by performing practical experiments on live tissues' (75); and similar proportions disagreed or strongly disagreed that 'there is no place for computer simulations of laboratory practicáis in undergraduate bioscience degree courses' (91), 'computers are difficult to use, and should be avoided when possible' (86), 'ah undergraduate experiments using live tissues should stop' (82), 'computer simulations of practical experiments are of very limited value in the teaching of practical physiology and pharmacology' (69) and 'practical laboratory work on live tissues is the only successful method of reinforcing the theory of physiological principles' (68). between half and two-thirds of the students agreed or strongly agreed that 'an important bonus in laboratory practicáis is that teacher-student and student-student contact is encouraged and learning is thereby enhanced' (66), 'undergraduate laboratory-based practicáis are more expensive to carry out than utilizing computer simulation programs' (54), 'replacing laboratory-based experiments with computer simulations provides the lecturer with more time for lesson planning, marking work, etc' (52) and 'during laboratory practical experiments, the time spent learning how to use equipment often leaves insufficient time to complete the experiment' (52). similar proportions of students disagreed or strongly disagreed that 'students need to receive special training to use a computer simulation' (62), 'students always get poor results from their laboratory practical experiments' (59), 'computer simulations of laboratory experiments can lead to the acquisition of laboratory skills' (56) 'the use of live tissue to demonstrate physiological/pharmacological principles poses no moral dilemma' (53) and 'computer simulations of such experiments are difficult to use' (51). between one-third and a half of the students agreed or strongly agreed that 'the use of computer simulations of practical experiments enables the lecturer to spend more time 122 volume 3 number i with the students' (48), 'hands-on experience with live tissues is essential for all students of medicine' (45), 'there is a place for computer simulations only if they are used as a supplement to demonstrations so that the real experiment can be seen first' (45) and 'computer simulations of experiments on live tissues can replace laboratory practical experiments as they meet the majority of course objectives (45). similar proportions of students disagreed or strongly disagreed that 'computer simulations of practical experiments are less instructive than laboratory-based practicáis' (49), 'computer simulation programs cannot adequately replace laboratory-based practical experiments in the teaching of physiological and pharmacological principals' (46), 'computer-based teaching is boring, thus limiting student learning' (45), 'computer simulations convey little sense of the "real" experiment, therefore all appreciation is lost' (40), 'hands-on experience with live tissues is essential for all students of medicine' (34) and 'during practical experiments the time spent learning how to use equipment often leaves insufficient time to complete the experiment' (34). the highest proportions of'undecided' responses were elicited by the statements that 'undergraduate laboratory-based practicáis are more expensive to carry out than utilizing computer simulation programs' (40), 'replacing laboratory-based experiments with computer simulations provides the lecturer with more time for lesson planning, marking work, etc' (37) and 'computer simulations of such experiments are difficult to use' (34). there were some substantial differences in the proportions of responses to a few of the statements by students from the cal and dem groups. in the following summary the proportions are given as percentages of the forty cal students and forty-eight dem students for ease of comparison. whereas 65% of the dem group felt that use of computer simulations freed lecturers' time, only 43% of the cal group expressed this view. forty percent of the cal group but 17% of the dem group concurred with the statement about computer simulations being less instructive than the real thing, whereas 15% and 38% respectively were undecided. also, 45% and 65% respectively of cal and dem groups agreed that laboratory experiments are more expensive than computer simulations, but 50% and 29% respectively were undecided. discussion many of the results of this evaluation are unequivocal and need no further comment. the computer simulation class required significantly less staff resource (hours of technicians' and tutors' time) than did the live demonstration, and can therefore be considered to be less expensive since other costs were comparable. the overall cost-effectiveness depends not only on actual costs but also on the adequacy of student learning. in this respect the cal group performed less well than the dem group although this varied from question to question. question 1 was based on a fundamental concept that was explained carefully, and related to the first of the common sequence of tests. it is entirely appropriate that almost all the students gave a satisfactory answer. in contrast, question 20 was based on the last section of the sequence of tests and involved concepts that were relatively novel to the students. consequently, the numbers 123 h. l leathard and d. g. dewhurst comparison of the cost-effectiveness of a computer assisted learning program giving satisfactory explanations were low. most of those in the cal group reached this stage within the time allocated, whereas the demonstration of the live practical did not regularly progress this far. to compensate for the incomplete experiment, however, the tutor explained the final parts with the aid of a sample tracing from a previous experiment (all students were provided with these). thus, it seems that the tutors' explanations and the integral discussion enhanced the students' understanding of the challenging parts. from students' comments and tutors' observations, however, it is evident that cal students needed an opportunity for discussion with a tutor after completing the program, not while they were concentrating on the screen; so tutors' time was not used optimally for this teaching method during the evaluation. the students' attitudes were generally favourable to computer simulations of experiments but, from the responses to the open question and the views summarized above, it seems that the majority would like to have some experience of the live practical with computer simulations to support this. clearly in a financial climate where the unit of resource available for teaching and learning is being reduced, teachers must consider carefully the costs of certain teaching styles relative to the learning objectives. in this study it might be argued that, for medical students, a live demonstration is unnecessary since it is the knowledge and understanding of the pharmacology of colonie motility which is the important learning objective rather than the laboratory skills associated with the investigation. in this study student learning, as measured by answers to key questions, is comparable for both groups, while the cost of the live laboratory is significantly higher. the findings of this study are similar to those of similar studies where computer-based learning methods have been compared to traditional live laboratories in undergraduate physiology and anatomy (dewhurst et al, 1994; coleman et al, 1994; guy & frisby, 1992; fawver et al, 1990). acknowledgement we wish to thank professor julia buckingham and her staff in the department of pharmacology at the charing cross and westminster medical school in london for their support of and contributions to this evaluation. references coleman, i., dewhurst, d.g., meehan, a.s. & williams, a.d. (1994) 'a computer simulation for learning about the physiological response to exercise', american journal of physiology, (advances in physiology education), ii, s2-s9. dewhurst, d.g., leathard, h.l., higman, m. and ullyot, r. (1992), 'a computer simulation to teach the action of drugs on intestinal (colonic) motility', british journal of pharmacology, 106, 143. fawver, a.l., branch, c.e., trentham, l., robertson, b.t., and beckett, s.d. (1990), 'a comparison of interactive video instruction with live animal laboratories', american journal of physiology, 259 (advanced physiological education, 4), s11-s14. 124 volume 3 number i guy, j.f. & frisby, a.j. (1992), 'using interactive videodiscs to teach gross anatomy to undergraduates at the ohio state university', acad med, 67, 132-133. leathard, h.l, cover, p.o., dewhurst, d.g., kumari, m. and rantle, c. (1994), 'evaluation of a computer-assisted learning program on intestinal motility', british journal of pharmacology, 112, 179. 125 conversion of a mathematics course t o cal: a case study of a large-scale rapid change of resources and organization maggie pollock,* erica mcateer,** gordon doughty* and ian turner* *robert clark centre for technological education, university of glasgow **department of psychology, university of glasgow during 1994-95, first-year maths for the bteched degree at the university of glasgow was studentcentred, teacher-supported a modular online maths course replaced a traditional, lecture-based course. students worked at their own pace, with timetabled and open access computer classes and/or paper handbooks. the course was evaluated by open-ended measures, and study of examination outcomes, providing us with some pedagogical questions and some recommendations for change. with some adaptation, and with important questions still open, the new course will continue to run. background glasgow university's teaching with independent learning technologies (tilt) project is an institutional initiative funded under phase 1 of the higher education funding councils' teaching and learning technology programme (tltp), we have developed or adapted and integrated computer-assisted learning programs into a variety of courses in subjects ranging from accountancy to zoology. this paper reports a complete change of learning resources and teaching strategies in a faculty of engineering course. overcoming 'not invented here' resistance, we used an externally produced program to deliver all maths teaching over a first-year class, and evaluated effectiveness for both teachers and students. the course is for first-year students in the faculty of engineering taking the bachelor of technological education (bteched) concurrent initial teacher education degree, taught by the robert clark centre for technological education in collaboration with st. andrew's college, glasgow. it produces school teachers of technological studies, craft and design and graphic communication. 28 alt-} volume 4 number i problems the students vary widely in age and past experience, and in maths background. the 1994-95 cohort entered with any of scottish higher 'a', ' b ' or ' c \ a-level (pass), hnd, hnc, onc, scottish wider access programme, or scotvec modules. most qualifications were recent, but some were years old. for these students, the purpose of their maths learning is to acquire manipulative skills rather than high theoretical understanding. the level aimed for approximates that of first-year engineers. before 1992, the class was taught separately, with close co-operation between maths teachers and technology teachers. then partnership changes and timetabling problems put the students in with 200 first-year engineers and spread the (lecture-based) course over two years. the bteched students struggled, neglecting other subjects, and 29% (6 out of 21) failed the course and dropped out. our teaching dilemma, ubiquitous in maths, was how to simultaneously motivate the students by applying maths learning to real life problems of concern to them, and help them to quickly gain the basic skills to do the necessary maths manipulations almost automatically. in practice, students cycle between the two. the duration of a cycle may range from the few minutes required to tackle a real-life maths problem to several months' delay before an application to the maths skills learning emerges in another course. although the joint class of 200 did cycle between theory, example and practice within the lecture period, our target students were not able to keep up with the manipulative skills. the degree course organizers would have preferred to change to teaching all maths inside the technology courses, as and when needed, but could not get agreement from all course teachers. an efficient and manageable alternative was needed urgently. solution? we designed a change to learning with long cycle times, providing the first year of a two-year maths course on basic skills, and treating their application in other courses. integration of the cycles would be provided by two of the authors, who also teach or organize several other courses. we bought in a computer-based maths course: calmat, developed by jean cook (cook, 1994, cook and hornby, 1995) and colleagues at glasgow caledonian university. this provides 50 modules of drill and practice tutorial exercises, with self-assessment tests and a built-in diagnostic examination. a handbook with paper exercises is provided as a supplement (or even alternative) to each module. although dos-based (a windows version is under development), it has a clear and easy-to-use interface, has evolved over many years use in higher-education institutions with students much like our own (tabor, 1993), and is supported by a maths teaching team. we felt that its pedagogy suited our objectives: to provide practical maths knowledge and skills, applicable in engineering and technology project work and beyond, when students are themselves teaching in secondary-school classrooms. for practical as well as pedagogical reasons, we also needed to provide student-centred learning with independent study resources while being able to access diagnostic information to promote appropriate teacher support. 29 moggie pollock et 0/ rapid conversion of a mathematics course t o cal course procedure at the beginning of the course, all students sat a diagnostic test of 20 multiple-choice questions. this allowed individual programmes of work to be scheduled for the first term. a weekly three-hour (non-compulsory) class was timetabled in the computer laboratory for the year's teaching period, which had at least one tutor present. students were free to use the computer lab at other times during term or vacation, with teaching assistance available often on demand, but always by appointment copies of the software could be purchased for home use. no formal lectures or tutorials were run throughout the year, except to go over classexamination outcomes. two text books were recommended to complement coursework. a repeat of the diagnostic test was taken by students at the end of term 1. during term 2, all students followed the same work programme (rather than individualized programmes), but at their own pace. 'poor achievers', who reported themselves as falling behind schedule, were given extra tutor support. the course was assessed by a degree examination half-way through the third term. exemption could be gained by students achieving an average of at least 60% by combining the results of the two class examinations, held at the end of the first term and the beginning of the third term. a summer re-sit examination was also available. information from students computer experience and attitudes maths learning experience and attitudes confidence about specific maths tasks course attitudes and experience skills gain information from teachers advantages, disadvantages, questions questionnaire — focus group — checklist questionnaire — focus group diagnostic test and exam marks interviews debriefing session table i: evaluation targets and methods evaluation with less than three evaluation staff, working across 20 sites, we could provide no more than a minimal evaluation of the course. table 1 lists the information sought and the methods used. students' attitudes to computers, and their knowledge and experience of basic computing, were surveyed. feelings about learning maths and preferred teaching methods were addressed by informal interviews and a focus-group meeting. students' statements of confidence about specific mathematical principles and practices, before and after working through relevant modules, were gathered throughout the course. a 'task experience' questionnaire was completed by students during term 1, to get an idea of how the course was progressing from their point of view. at the end of the course, shortly before the examination, a second focus meeting was held. the year's assessment grades were studied. 30 alt-j volume 4 number i teaching staff, who had collaborated in both the planning and the implementation of the evaluation exercise, as well as in the interpretation of interim findings, were interviewed. evaluation outcomes student perspective * table 2 summarizes findings from questionnaires and interviews. most of the students came into the course computer-literate, in that 14 had had some training in computer use, either at school or at college, with the software they had used ranging from macwrite to c++. most used a computer at least once a week. modestly they stated themselves as 'novice' more often than 'competent' almost all said they had at least some confidence about using a computer. computing experience attitudes, practical knowledge confidence with computers maths experience experiences with maths learning confidence about practical maths experience of bteched course positive neutral negative high middling to good generally negative increasing with calmat independent learning own pace immediate feedback peer support learning with computer three-hour sessions — occasional incorrect feedback, 'one method' teaching need for tutorial support when 'losing it' table 2: student information summary students' attitudes to and basic knowledge of computers were surveyed (with permission) by questions from a questionnaire developed by queen's university, belfast. almost all attitudes were positive, and almost all knowledge questions were answered correctly. their experience with maths learning was varied and, generally, negative. the main impression was that it worried them. lectures were definitely not favoured: 'once you get behind, you stay there', 'two hours and aching fingers', were two students' views about listening and note-taking, a third adding: 'you're concentrating on getting everything down, not on taking it in'. questioned at the end of term 1, students' experience with calmat was positive, with some 31 maggie pollock et al rapid conversion of a mathematics course t o cal criticisms. the human-computer interaction presented no problems except where they felt it gave them 'wrong' when it should not (an issue of decimal places), and when they could not retry a question without restarting a complete module. for the majority of students, three hours (with coffee break) at the computer screen was not a problem. however, for a few it was these students would spend some time at the machine, then work through paper exercises in the relevant handbook: they very much liked the own-pace element, although there were still some fears of 'getting lost' or falling behind. students were encouraged (rather than required) to complete their recommended exercises to a schedule by three-quarters of the way through the term, many were behind on this. in general, they rated themselves more confident about maths principles and practice as they went through the course, and most students actively liked the working environment, which they felt included a lot of peer support. teacher perspective discussions with teachers during the course and afterwards showed them to be even more positive than the students. table 3 summarizes the information. the feedback from the evaluation exercises fitted in with their own perceptions. • importing established and supported software, in this case, has been very cost effective. • n o time needed to prepare and deliver lectures. • less stressful, allows support structures to be developed where needed rather than globally. • possible improvement in students' practical application of maths in other courses. • intend t o provide weekly tutorial sessions and support further 'peer' tutorials at need. • intend t o closely monitor next year's project w o r k for evidence of practical maths knowledge. table 3: teacher information summary information from diagnostic test and class-examination results the first class-examination posed 40 multiple-choice questions and 10 standard problems. among the multiple-choice questions were the original 20 questions from the diagnostic test, to measure any improvement. the second class-examination set 10 problem-type questions. the degree examination had the same format as the second class-examination. table 4 gives an overview of the results. the diagnostic post-test at the end of term 1 indicates some improvement in their skills the range is wide both before and after, and the central tendency is high. ten students scored 70% or over at pre-test, so had little scope or need for improvement. polynomials, equations of straight lines, and trigonometry were all topics which showed a significant overall improvement. 32 alt-j volume 4 number i diagnostic test significant overall improvement in scores from high start 65% (range 30% to 85%) at end of term i 76% (range 40% to 100%) better on some topics than others (e.g. improvement in algebra, co-ordinate geometry and trigonometry) class exams december multiple choice 69% december open questions 36% (some students did not attempt these questions) may open questions 38% . (five students obtained < 8%) exemptions i student with > 80% 6 students with > 60% first year degree exam assessment grades ' i student with £ 70% i students with > 60% 3 students with > 50% 4 students with > 45% 5 students with < 35% resit exams i student with £ 45% 4 students with > 35% table 4: diagnostic test, classand degree-examination outcomes we are concerned about the outcomes for the 'open' questions standard mathematical problems which depend on knowledge of different topics covered in the course. we do not yet know whether the material is not teaching, the learners are not learning, or the questions are inappropriate probes of their skills. it should be noted that the average in the may classexamination included five students whose marks were below 8%. without these marks, the average would have been 48%. the first-year degree examination outcomes were acceptable in that they were on a par with the previous year. it is encouraging that, unlike findings elsewhere (noss, 1995), entry qualifications were not direct predictors of grades. the top six had entered with a mixture of highers, hnc and onc, the lowest six with highers, hnc, hnd and access. although the assessment procedures, of necessity, differed from those for previous years, the same external examiner's criteria had been satisfied. it is not, however, possible to make any direct comparisons between the examination outcomes of 1994-95 and the previous year, apart from being encouraged that the number of failures after re-sits fell from six (after normative 33 moggie pollock et al rapid conversion of a mathematics course t o cal scaling was applied to the raw marks) to four (out of 21) with no normative scaling. a good measure of their learning will be their competence and progress during next year's courses — electronics, energy, design, etc. where much of their work will depend upon maths 'fluency'. the best predictor of examination performance was the post-test diagnostic scores those who came out best on this at the end of term 1 got the best degree grades, those at the bottom end stayed there. conclusions changing the first-year maths course to student-centred, self-paced learning using a cal package, plus supplementary written material and textbooks, has been shown to be no worse than if the course were delivered using traditional lectures, and in fact was preferred by both the students and lecturers. the students responded well to self-paced learning and peer self• help groups developed spontaneously. staff found it less stressful to supervise the computer labs than to prepare and deliver lectures. they could devote more time to those students who were slow and struggling with the work. every effort will be made to identify earlier those students who are having difficulty. we intend to improve the learning of these students with the second year of the new course. we are planning fortnightly tutorial classes, led by 'demand issues' and supported by some peer tutoring. we also intend a closer monitoring of students' progress through the modules, though interfering as little as possible with the student-centred, own-pace independent-learning model, which we feel is right for these students. references cook, j. (1994), 'bridge the gap with calmat', proceedings of the international conference on technology in collegiate mathematics, november 1994. cook, j. and hornby, j. (1995), 'calmat mathematics courseware for access to higher education', proceedings of the smc conference, stirling, may 1995 (in press). noss, r. (1995), reading the sines, final report of the mid-term evaluation of the transitional mathematics/transmath project, institute of education, university of london. tabor, j.h. (1993), 'using calmat in "levelling up" teaching', cti quarterly newsletter, 4 (1). 34 reviews computers and typography edited by rosemary sassoon, oxford, intellect, 1993. isbn: 1-871516-23-4. the term typography refers to the style, arrangement, appearance, and design of typefaces and typeset material. in this book, sassoon rightly assumes that typography now extends way beyond words printed on paper, and in particular to the world of computers, where alphanumeric information displayed on a screen is often as important as printed output. both forms of typography are dealt with here. the book consists of eleven chapters, written by a variety of different authors, and grouped together into five main sections, each one covering a different aspect of typography. section 1 (spacing and layout) contains three contributions. the first of these provides an introduction to 'text massage', basically summarizing half a dozen guidelines for improving textual presentation by means of various massaging techniques. the chapter is simple and brief, though unfortunately its brevity also means that the guidelines described are not supported by any references to relevant research in the field. the second contribution in this section addresses the concept of layout. this is achieved by illustrating two versions of a questionnaire, each making use of a slightly different spatial arrangement. the importance of white space in the design of textual information is discussed in the context of information highlighting and the provision of structural cues. the final chapter in the section looks primarily at rules of composition. it discusses the effects of inter-character and inter-sentence spacing. rules based on the use of various character shapes and punctuation marks are also presented there for determining optimal sizes for the spacing used in presenting text. the second section (typographic choices latin and other alphabets) contains two papers (as do all the remaining sections). the first paper describes some of the problems of using alphabets other than latin, with major consideration given over to handling hebrew on a computer. the problems involved in creating multilingual documents are also considered. the second paper discusses the flexibility provided by computers with respect to typography in general, the variety provided by font types and the control of inter-letter spacing. there is also a discussion of the costs involved in fine-tuning computer-based text. the third section is entitled technical issues in type design. the first paper here describes various aspects of the effects of technology on type design, one of the principal effects being found in screen or printing resolutions. low resolutions can result in a syndrome known as 'the jaggies': low-quality text with jagged edges. the chapter concludes with a description of the development of type design and font production tools. the second paper deals with character description techniques in type manufacture. four techniques are described, two used in the production of metal type, and two relating to bitmap font production and mathematically specified contours (both used in the manufacture of digital type). the two papers making up the fourth section of the book (lessons to be learned from the history of typography) each relate to the complications of computerisation. the first deals with education in the context of writing. problems relating to the use of computers in the development of writing skills are discussed, and arguments for the importance of handwriting are presented. the second paper discusses the impact of computerization (particularly, in the form of 70 au-j volume 2 number 1 desktop publishing) on traditional typesetting, and the implications of the 'democratisation' of typesetting, through the wide availability of desktop-publishing software, are debated. the two papers in the final section of the book come under the heading research and the perception of type. the first deals with research into typography and reading, with particular reference to the lack of research into the effects of the typographic features of text on children. the chapter includes both a survey of teacher attitudes towards type suitable for children and a description of research into the effects various type forms can have on children's ability to read and write. the final paper of the book is devoted to the visual analysis of text. obviously, since the human visual system is fixed, it is essential that the features of textual material are designed to match its inherent limitations. the paper presents a method for the computational analysis of textual layouts which might be useful in analysing the potential of various changes to typesetting parameters. this book is aimed not only at those interested in typography for its own sake but also at anyone who wants to produce professional-looking documents with a computer. in its latter aim, in my view it is only partially successful. much of the work attempts to convince the reader that typography is essential to creating quality computergenerated documents, but although guidelines for good typographic practice are offered, these are not generally brought out clearly enough. furthermore, as is often the case in multiauthor works, the different writing styles of the various authors result in some chapters being easy to read, and others less so. the work would also have benefited from some form of conclusion drawing together the major findings of the papers. stephen richards, university of teesside active library on corrosion by w.f. bogaerts and k.s. agema, a cd-rom published by elsevier science publishers (amsterdam) and the national association of corrosion engineers (houston, texas), 1992. isbn: 0-44489607-4. for a variety of reasons, electronic books published on cd-rom are valuable learning resources, and publishers are increasingly realizing the many attractive advantages they offer. the active library on corrosion (alc) is just one of many publications elsevier intends to produce as part of its new active library series. the alc runs under windows on an ibm pc or compatible fitted with an internal or external cd-rom drive accessible via the microsoft cd extensions (mscdex). the software is fairly easy to install from the cd-rom it is supplied on, and requires about 3.5 mb of hard-disk .space. the active library concept is essentially a generic hypermedia shell into which can be plugged a range of different specialized subject-orientated materials. the shell is written in guide, the hypermedia authoring tool marketed by office workstations limited (owl) and widely used in many british universities for developing interactive teaching material. two of the reasons why guide was chosen for this project are (a) the ease with which hypertext links can be created, and (b) its ability to handle sgml (standard generalized markup language). • sgml was used to mark up many of the electronic documents prior to importing them into guide for interlinking using a special tool called guide writer. a good range of electronic tools is available within the system. functions include specific searches (searching and consulting for information relating to a specific query), highlighting and annotating documents or parts of documents, browsing (associations based on links, with back-tracking fully supported, and current awareness (providing 71 reviews up-to-date information on a specific field of interest through a user-profile). two other important facilities (based on path functionality) are hitlists and trails. hitlists are system-generated collections of related documents, and trails are user-defined sequences of interesting documents held within the knowledge corpus. the alc contains 6,000 documents covering a broad range of topics in the subject-area of corrosion engineering. it is made up of eight previously published reference works and 1,200 newly created documents written by leading corrosion experts. it also contains over 2,000 images (mostly full-colour), 600 formatted tables, and numerous mathematical and chemical formulae. the system, then, has a very large amount of information in it, and in order to give readers a sense of where they are in the publication,, the information is organized around the use of a library metaphor designed to provide a mental reference map. after the alc is started by clicking on its windows icon, one of the initial top-level screens is used to show readers the library map. this indicates all the available library sections that can be accessed by clicking on appropriate buttons. within the individual sections themselves, other maps are provided to aid navigation and to prevent readers from getting lost. the main library map provides eight basic access mechanisms into the alc system. these are the reference cube, case histories, corrosion control, quick reference, books, databases, dictionary and help. the reference cube is an innovative threedimensional table of contents in which the reader can specify a combination of corrosion type and/or material and/or environment. using the specifications entered by a particular reader, the reference cube can consult its underlying database and provide that reader with relevant information, which may be a single document or a collection of documents. the case histories section contains a database of 400 cases of corrosion damage taken from during's corrosion atlas. the corrosion control section is a collection of documents which give an overview of general corrosion control measures and typical measures against different corrosion types. it also presents discussions on individual techniques of corrosion prevention. the quick reference section is based on the national association of corrosion engineers' (nace) reference book. it embeds an extensive set of look-up tables containing corrosion and materials background information, and it includes a glossary for readers who are not familiar with corrosion engineering. the books section contains the four books now regarded as standard works in the field of corrosion engineering and corrosion prevention. each has been converted to hypertext form, and so are integrated into the main body of documents stored in the alc. the database section contains two standard nace databases (cor-sur and corsur2) which together document the performance of 25 metals and alloys (in more than 1,000 chemicals and corrosive environments) and 36 non-metals (in more than 850 chemicals and corrosive environments). for non-expert users, the dictionary section provides alphabetical access to a comprehensive selection of corrosion terminology. similarly, for new users of the alc the help section contains online help information on each of the other library sections, along with a series of tips on the basic use of the system. alc is aimed primarily at corrosion engineers, and is intended to serve as a reference work for day-to-day use as well as a source of information on less common corrosion problems, but it will be an extremely useful asset in many an academic library or laboratory. it is produced to a very high standard both in terms of its content and its usability. 72 alt-j volume 2 number 1 indeed, it sets a standard for electronic publication that many other publishers will find difficult to exceed. philip barker, university of teesside a handbook of computer-based training by christopher dean and quentin whitlock, 3rd edition, 1993, london, kogan page. isbn: 0-88415088-7. the title a handbook of computer-based training is somewhat misleading. two thirds of the book deal directly with the subject, but one third is a general description of different aspects of computer systems covering, for example, the difference between microprocessors, minicomputers, and mainframes; what functions operating systems perform; descriptions of a number of input devices, and so on. the authors might rightly argue, however, that this information is useful for part of the intended readership. a training department, for example, might use computers for word-processing and spreadsheets and not really appreciate the computing details of those systems. if they were thinking of adopting computer-based training (cbt) for much of their central work, this basic computing information would be valuable. and indeed, for anyone who secretly wants to be initiated into the world of computers, this book presents a balanced overview to a reasonable depth. i have a few niggles. for instance, the lower-case m used in megabytes (a very common error in newspapers, but not one that should appear in a technical book), or the description of a light emanating from a lightpen which leads the reader to assume that the light plays a part in the detection mechanism, when in fact it only provides feedback to determine where the pen is pointing. but these are the niggles of a pedant. so what about the major part of the book that covers cbt? one of its pleasing aspects is the emphasis it gives to professional production: it dispels any thought that the job should be given solely to a junior programmer in the computing applications group as a spare-time pursuit the production of cbt requires the same detailed planning as does any software project, but the personnel involved must have knowledge of the topic being conveyed, the authoring software, the techniques for teaching, the artistic skills to design material that has the necessary impact on the trainee, and an awareness of the skills and experience of the typical trainee. thus cbt material is often prepared by a team rather than one individual, and no team member is likely to be particularly skilled in writing conventional software (although in some cases such a team member may be necessary). other key points raised in the book are that cbt is rarely used alone, but is usually combined with other media such as videotape and paper booklets. the authors emphasize the need to pre-test and post-test to determine how much students have really learnt using the cbt system, and that the body of the lesson should not be conveyed merely by 'page turning'. instead, trainees should be regularly active, with their responses determining their path through the material. a number of situations are described, illustrating how cbt has been employed. the authors point out that not all situations demand a cbt' solution (for example a video might be preferable). the book is oriented towards the front-end of cbt development. is cbt really needed in this case? what is needed? have the prospective trainees been consulted as to what they require? what are the arguments for the various strategies for cbt provision? what' are the financial implications for the strategy? what are the computer resource implications for the strategy? and so on. all well answered for the prospective cbt organizer to use in discussion with higher management. 73 reviews the book describes how the outline plan is broken down into constituent elements and how this is organized, planned, managed, and executed. it includes descriptions of the tools that may be available for specifying the paths through the material and the instruments that might be used to document the design. it also describes the tools that manage the training sessions, including trainee records and progress, and methods for directing students through the course. however, how teaching skills are exploited in the user-interface, and how the perceptual workings of the human are exploited are not covered except at a very superficial level. again, the authors might argue that what they do cover is all that is necessary for the cbt material currently produced. however, with computer systems increasingly being more graphically interactive, and cbt designers intent on exploiting that fact, further guidance is needed to ensure that the user-interface does not confuse the trainee. the authors do describe a system which fails to pick up the fact that the supporting material the trainee is meant to have while using the computer is absent, despite the trainee identifying the problem to the computer. this was a failure in the interface, but little guidance is given as to how to avoid the problem. tips from experienced designers would have been valuable. this is nevertheless a useful book for the training manager's bookshelf. it is particularly aimed at the manager who is to introduce cbt, and that person will be made aware 6f the computing resources needed, the skills needed of those who will be involved in its production and maintenance, and the stages involved in producing cbt material. it briefly describes the functionality that can be expected in software such as course-authoring tools, computer-managed learning tools, and flowchart tools. it does not (nor should it) go into how these tools achieve their results. perhaps the fourth edition might include more detail of screen design, particularly those aspects that must be foremost in the designer's mind while deciding the best way of conveying specific material. i.d. benest, university of york learning conversations by s. harriaugstein and l. thomas, london, routledge, 1991. isbn: 0-415-02867-1. in reviewing the development of courseware, it is rare to read of the educational or psychological reasons driving that development. all too often, computational, technical or economic issues dominate. perhaps in the avoidance of this 'why' question, it is inevitable that a book should appear that challenges almost all the conventional wisdoms at once, which fires harmonious cannons at practical experience and offers a new model on which all resources for learners should be built. in their book, sheila harri-augstein and laurie thomas review 40 years of action research to draw out a unifying theory of behavioural and experiential psychology, and eastern and western philosophy. the fundamental premiss of the thesis is that conversation is priori, and learning is a skill which itself can be learnt. the book is divided into three main parts. the first looks at learning and how cultural and historical precedents have acquired a language to describe the process of learning which might be generalized as modelling meaning. five types of meaning are identified, ranging from 'rote' to 'creative' an increasing awareness of the learning process being seen as the key to moving from one to the other. the second part looks at the role of the 'learning conversation' in helping this transition occur. the 'life', 'task' and 'learning to learn' phases are identified, which can be thought to map learners' purposes, experiences and interpretation of records in computerized systems. the 74 alt-j volume 2 number 1 'monitor, analyse, record, reflect, reconstruct' heuristic is described and exemplified by tools used to structure learning conversations and make the learning process explicit. traditional measures of ability are shown to reflect a learner's ability to internalize learning conversations and become more self-organized. the third part looks at how self-organization can be encouraged. in passing, many anecdotal findings of others are explained: for example, why, when the effectiveness of a computer-based resource is compared with its paper-based equivalent, no significant 'size effect' can be measured. the learner's ability to converse is shown to encapsulate his/her potential to learn. a radical challenge is made to the current educational establishment to improve the quality of the conversations they manage and remove what are revealed to be barriers to learning competence. the expansive appendices and postscript illustrate the authors' ideas with remarkable clarity and give plenty of scope for further reading. one of particular interest focuses on the requirement for computer-based resources that support self-organized learning. they give a reasoned explanation for the common finding that as objectives become more experimental, computer-based reflexive tools are more effective than rubrics. indeed, gaining a language to talk about learner's experiences and environments is one of the main benefits to be gained from this book. the omission of a bibliography does not detract from the work as a whole; references can be obtained from the authors. in the light of previous reviewers' remarks, any further comment seems superfluous: 'i commend [...] learning conversations as an important work to be widely read'. professor gordon pask (founder of cybernetic theory). 'i believe learning conversations is a book no psychologist or teacher should be without'. dr david fontana (editor of the psychology for... series). 'the s-o-l system has enabled [...] progress towards higher productivity'. sir bryan nicholson (cbi educational training affairs committee). martin reynolds, university of manchester collins electronic english dictionary and thesaurus, san francisco ca, reference software international, 1992. the collins electronic dictionary is just one example of what is becoming an increasingly popular type of reference work for use with a personal computer system. this particular publication was originally produced by reference software international but is now distributed and maintained by the wordperfect corporation. the ibm pc version of the product is supplied as a set of seven 3.5 inch disks that have to be installed in a separate directory on the pc's hard-disk drive. the system takes about 15 to 20 minutes to install, and requires about 9 mb of hard-disk space for full installation. the software itself is made up of four basic modules: the dictionary (which requires 5.7 mb), the thesaurus (1.02 mb), the pronunciation guide (0.54 mb), and the etymology module (1.1 mb). the only module that must be installed is the dictionary. • the dictionary is based on the 3rd edition (1991) of the paper-based version of the collins english dictionary and the collins thesaurus (1984). together, these make available 190,000 definitions (comprising 3.5 million words of text) and over 275,000 alternatives for 16,000 main-entry words. entries in the dictionary contain pronunciation details as well as etymological information. a relatively simple manual is provided, giving details of how to install the product and how to use it. it is quite easy to read yet 75 reviews fairly comprehensive in its treatment. unfortunately, however, in some places it is inaccurate for example, the telephone number for hotline support is incorrect, as are some of the instructions for running the system. once installed, the dictionary can be invoked in either of two ways: as a free-standing conventional program, or as a terminate and stay resident (tsr) system. each of these modes of operation is available within dos and windows. the tsr version requires the use of hotkeys, and has to be unloaded when it is no longer needed. once the program has been invoked, the system provides the same basic user-interface and facilities no matter which mode is being used. the user-interface is in three parts: a horizontal menu bar running across the top of the program's display window, a central display area for presenting word definitions (and other information), and a second horizontal menu bar running across the bottom of the display window. there are seven entries in the topmost menu bar: dictionary, thesaurus, search, history, options, quit and help. quit simply terminates the dictionary and passes control back to the program that called it either dos (or windows) or an application (such as a word-processing package or a text editor). help provides access to a pop-down menu that offers a number of different choices such as pronunciation, glossary and so on. the range of help facilities available depends on whether the windows or the dos version is being used. the options entry allows users to customize the dictionary to individual needs by changing the colour scheme for display windows, case sensitivity (in searching), matching criteria, and so on.the history option can be used to obtain a list of the last 16 items retrieved from the dictionary or thesaurus. the dictionary and thesaurus options create dialogue boxes allowing the entry of words to be looked up in the dictionary and thesaurus respectively. the search menu option provides three different word-search options within the 'dictionary: definition search, anagram and wildcard. the most powerful of the three is the definition search. this searches every definition in the dictionary in order to find entries that contain selected keywords, according to criteria that have been established using a special search command. such commands contain keywords connected by the logical operators and, or and not. the results retrieved in any particular search will depend on whether or not case sensitivity has been selected, and also upon the exactness level specified. wildcard searches use the two symbols ? and * within a search string to represent, respectively, a single unspecified letter and multiple unspecified letters. the anagram search can be used to find all words that can be made from a given string of letters. the horizontal menu bar at the bottom of the dictionary's display window provides access to a range of tools which can be accessed by mouse selection or by function keys. the facilities these tools provide include browsing, switching between dictionary and thesaurus, and copying a marked block of text from the dictionary and pasting it into an application. there is also a replace facility for use in conjunction with the thesaurus. although i found this program useful, i was disappointed to find a number of bugs in it. the three major bugs i found were: (a) the tsr version of the program would not unload properly when i wanted to remove it from memory (i had to re-boot my machine); (b) the copy and paste facilities would not work correctly; and (c) using the tsr version with windows led 76 au-j volume 2 number 1 to an accumulation of undeleted work files within the directory holding the software (i had to delete these manually). however, despite the bugs and the inaccuracies in the documentation, i still think that at £64-63 (including vat, packing and delivery), this fascinating educational product is good value for money. philip barker, university of teesside key terms and issues in open and distance learning by barbara hodgson, london, kogan page, 1993. isbn: 0-7494-07107. this book began for me promisingly. first, the title itself, coming from an author at an institute highly active in both traditional and innovative design and delivery of distance education, suggested an up-to-date book that would reflect issues central to the field and also the considerable work going on in distance and open learning in europe overall. then, the author's introduction, giving an overview of different perspectives on open and distance learning, their shared as well as their distinct characteristics, was well written and conceptually helpful. i hoped the book could become a useful resource. however, two warning signs soon arose. the first was the lack of an index or any kind of overview of the content, no indication of the descriptors used for entries, and no sort of conceptual grouping of the entries. only alphabetical order (and bold-faced terms within entries, for cross-referencing) were there to guide me. the first term was access devices followed by a page and a half of discussion with cross-indexed terms: content lists, summary, overview, objectives, index, glossary, headings, cross-references, advance organizers, pre-tests, and concept structure diagram. once i read the entry, i could see the logic of calling these access devices, but i never would have thought to go to an entry entitled access devices if i had wished to find out more about key issues with, say, concept structure diagrams being used in open learning. the second warning sign came when i then tried to look up, via alphabetical order, various key issues that appear frequently in discussions about distance and open learning, issues such as (among many) how to facilitate co-operative work among learners at a distance or how to design just-in-time open learning and the implications of this kind of flexibility for traditional distance education. i could find nothing that seemed to relate to these and a number of other key issues. perhaps it was the lack of an index or overview that was preventing me from finding what was in the book. i decided to read it from cover to cover, and coded the content. i found four major orientations that together left me disappointed and frustrated. my general impressions, after my analysis, were that (a) this book was most focused on the design of highly structured print materials for distance delivery where the assumption was that learners would generally work in isolation; (b> the little of what i see and hear as key issues in open and distance learning in europe circa 1993 are not represented in this book; (c) most of the entries for communication and information technologies as tools and channels of distance and open learning are unsatisfying; and (d) the book is unbalanced in its choice of topics. the first point is the most far-reaching. the book appears to be written primarily from the perspective of the designer of structured print materials for distance delivery, where it is assumed that the designer, not the learner, is in essential control that is, 'activities [...] are things you ask the learner to do ...' (p. 19); 'an overview thus tells the learners where they have been, where they are now, and not only where they are going but why they are going there' (p. 88); and so on. although this approach is certainly relevant 77 reviews for some learning situations, it reflects nothing of the major discussion now occurring about constructivist approaches to learning or learner control. also, the book seems frequently to assume a model of the learner working in isolation, generally only with the print materials ('since learners usually have no one to turn to for clarification', p. 22). the awareness of the importance of human interaction within the distance and open-learning paradigm, and the many strategies being used to include interaction as a key part of the distancelearning situation, does not seem to be a key issue relative to this book, while i think it is one in practice and research. the coverage of key terms and issues with respect to the use of and impact of communications and information technologies in distance and open learning is particularly unsatisfying. sometimes what is included is rather odd in its expression compared to general practice. for example, teleconferencing is described in a two-line entry as 'a system for linking individuals to telephones. very useful for telephone tutorials' (p. 118). audio conferencing, for which the above definition would be more accurate although still strange, and audiographic conferencing, have no entries at all, although they are important channels for adding interaction to distance learning in countries throughout the world. instead of audio conferencing or audiographic conferencing there are two strange entries, audio letter and aitdiovision, that describe techniques i have not seen in current practice, nor certainly would call key issues. similarly, video conferencing does not merit an entry although something called radiovision does. in addition to somewhat odd definitions, others relating to communication and information technologies seem not only out of date but uninformed. computer drill and practice is described as 'seldom giving feedback to learners who make mistakes' (p. 26). computer-marked assignments are introduced by saying: 'all that is needed is a template that can be placed over a standardized answer form to reveal whether the learner's coded responses are in the right place' (p. 28). icons are described with an explanation that makes little sense: 'they [...] can signal, at a glance, the necessity for having to hand your audio-cassette player or that you will be expected to write something before the end of this particular section' (p. 59). television is dismissed in 10 lines as an 'expensive medium' (p. 119), despite its strategic and extensive use in distance education in many countries, for example china and turkey. and it is not only media-related terminology that sometimes surprised me. how many people would look up the terms reflective action guide, student stoppers or transformers? agro-botany evaluation is a highlighted term who is going to look for that as a key issue, especially when genuinely key issues such as criteria for cost-effectiveness evaluation are not mentioned? without an index, how will the reader know that people with disabilities has been chosen as a heading? the impression of imbalance i obtained related not only to content but also to implicit frames of reference. while some key issues are represented minimally or not at all, others are over-represented. for example, entries relating to learner evaluation predominate, with major headings that include: assessment, assignments, competence, computer-marked testing, evaluation, examinations, formative evaluation, in-text questions, marking, norm-referenced assessment, objective tests, post-tests, pre-tests, self-assessment, and summative evaluation. most of these are written in a introductory way, such as one would expect in a reader for general pre-service teacher training. with regard to frames of reference, those of many actors in the overall distance and openlearning setting are not represented at all. for instance, key issues facing administrators 78 alt-j volume 2 number 1 in distance teaching organizations are not represented, neither are issues such as those relating to broadening access while maintaining quality or transferability of experience and accreditation, nor issues relating to the training of instructors to support the addition of distance and open-learning options to their teaching. some specifically british references occur (for example, national vocational qualifications), but no hint is given of the extensive experiences relating to open and flexible learning that are happening at the european or other international level, such as the delta project, nor the issues being discussed relative to those experiences. on reflection, if the book had been titled differently, clearly indicating its orientation, and if it had provided an index or overview, my reaction would have been less negative. the book could more accurately be described as 'an introduction to the design and validation of (predominately) print materials for structured lesson sequences offered via distance education institutions' than by its actual title. with this forewarning, i would have looked for different things and at a different level than the current title required of me. then i could have appreciated the detail about the construction of such print materials. the three pages, for example, under the heading in-text questions include very useful guidelines if one is not waiting for key issues to emerge somewhere. the discussions about learner characteristics and learner profiles are good, if one is not expecting also to find something about key issues relating to group interaction at a distance. in summary, and in my opinion, the writing in this book about the instructional design aspects of (predominately) print materials for structured lessons delivered at a distance is clear and good and helpful. in contrast, again in my opinion, the overall coverage of key terms and issues in open and distance learning is weak and unbalanced in selection. i would suggest an editing to take out the generally poor entries relating to media and communications and information technologies, a revision of the title, the addition of an index, and a re-presentation of the book as a reference text for print-materials design for structured lessons delivered at a distance. this latter topic is important, and the many entries related to it in this book are relevant and helpful. betty collis, university of twente, the netherlands rethinking university teaching a framework for the effective use of educational technology by diana laurillard, london and new york, routledge, 1993. isbn: 0-41509288-4 (hardback), 0-41509289-2 (paperback). diana laurillard begins her book with an account of her attendance at her very first lecture as a student, one which will ring familiar bells for many of us: 'with 199 other students i counted myself lucky that i was in the main lecture theatre and not in the overspill room receiving closed circuit television. the lecturer was talking formulae as he came in, and for fifty minutes he scribbled them on the board as he talked and we all scribbled more in a desperate attempt to keep up with his dictation.' she then goes on to recount what happened in the first lecture she herself gave as a teacher: more or less the same thing. but this time, a student asked a question at the end of the lecture which showed that nothing of it had really been understood, either by that student or, it soon emerged, by any of the others listening and scribbling furiously. laurillard says that she learned a great deal more from that lecture than did the students. thus she came to believe that university teachers must take the main responsibility for what and how students learn. now, given what i say in partial defence of the tradi79 reviews tional lecture in my editorial in this issue of alt-j, one might assume that i would not be in entire agreement with the further conclusions laurillard comes to as a result of her early experiences, in particular that the lecture form as generally practised in universities is a kind of insanity. 'it is truly a miracle,' she insists, 'and a tribute to human ingenuity that any student ever leams anything worthwhile in such a system.' but, throughout this book, laurillard talks a great deal of sense about the failings of current academic practice, and she and i are not as far apart as it may at first appear even when it comes to the traditional lecture, which she too sees as a usable medium if well handled. laurillard's main thesis is that the academic system must be made to change, and not by new guidelines being issued on how to teach but by universities reorganizing their teaching infrastructure. thus her book is essentially a discussion of how to think about teaching, and the final chapter, towards which the book steadily builds up, is an attempt to define the kind of infrastructure in which university teachers can be as efficient in their teaching as they try to be in their research. so well executed is this discussion, and so clear, consistent and convincing the direction throughout, that i have absolutely no hesitation in recommending the book. space does not allow me to give the details of all the strands of argument which laurillard weaves together with the utmost clarity and intelligence, and in plain but literate english. briefly, however, part i consists of a lucid treatment of what the aim of teaching should be and of the complexity of the learning process, a study of student attitudes, and a clear-headed discussion of ways in which an effective teaching strategy can be created. part ii is an evaluation of the educational tools available to teachers, from print to interactive media. it is a straightforward but penetrating evaluation not surprisingly, given laurillard's own respected work on evaluation based on key research studies about how students react to such tools and how they use them. this is of very great interest to those of us (which, of course, should be all of us) who wish to be as sure as we can be about which tools are best for which jobs. part hi is concerned with design methodology in educational technology. there is a wealth of literature some good, some mediocre, some awful on design issues in this area. laurillard talks (her usual) eminent sense about general design considerations, about the ways in which design relates to content, about sensible approaches to combining different media, and about what it means in practice to use educational technology as an integral part of academic life. this, then, is a work the result of many years of professional experience and active campaigning for better teaching, and in particular the effective use of educational technology which puts into the clearest possible perspective the issues involved, and cogently argues for a rational approach to them. few readers of the book could fail to be convinced that educational technology, if used wisely, is the right way forward for academic institutions. it is therefore a book which all those involved in the use of educational technology, or those considering its use, should read. in fact, a copy of it should be sent to the vicechancellor of every university with an exhortation to take some time to read it carefully. its ideas might then filter down to those working at the coalface who are hardly aware of the technological tools available for teaching, or who are as yet unconvinced of their value. gabriel jacobs, university college of swansea 80 the works metallurgist: an evaluation of a cal package on phase diagrams canan tosunoglu,* philip butcher;** eileen scanlon* and ann jones* institute of educational technology, the open university **academic computing service, the open university the works metallurgist is used in the open university course: materials — engineering and science. it is being evaluated as part of a larger study of cal teaching in science and technology at the open university. with 540 students, the course provides a sufficiently large sample for a range of evaluation methods to be employed in a variety of settings (e.g. home, residential school). the methods used include questionnaires, attitude scales and pre/post achievement tests, observations and interviews. the focus of the evaluation exercise is on the effectiveness of cal in contributing to student learning. the issue of the educational quality of the program and its instructional features are considered in this paper, in association with the goals of the program. introduction a number of factors have converged today to force the development of more efficient and costeffective educational strategies. these include rapidly changing technology, the increasing number of students in higher education, and resource restrictions imposed on all sectors of education. educational systems are therefore finding it increasingly difficult to respond to learners' needs, highlighting the need for alternative educational strategies. an increase in the educational use of computers and new information technologies is often one of the suggested alternatives, baised on the premise that computers can support student learning in an individualized way and at the learner's own pace. in this context, evaluation in general and cal evaluation in particular can contribute to the process of defining and refining educational strategies. the open university (ou) has included the use of computers to support learning in its courses since 1971, and since 1979 attempts to evaluate the ou's cal provision have been reported in a number of publications. despite a large volume of research into cal evaluation, we believe that there is a need to develop a more comprehensive and systematic approach which can contribute more directly to the development of improved cal provision. 51 canon tosunoglu et al the works metallurgist an evaluation of a cal package on phase diagrams aims of the evaluation study one purpose of this evaluation was to contribute to decision-making on the amount of cal materials in courses. more specifically, our aim was to determine whether a particular cal program was effective in contributing to students' learning about phase diagrams. through a series of studies, we aim to build up a more general knowledge of what features contribute to effective cal in order to develop and improve other educational software. our study aimed to answer the following questions: • how do students use the program? • does the software contribute to student learning? • what do students learn from the program? • how can the program be improved? our main concern is the learning benefits to students, and the relatively small size of the program helped us to investigate a number of aspects of the program in relation to learning, and the extent to which the program's objectives were attained. fortunately, the objectives of this program were already well defined, as it was developed in response to students' difficulties in interpreting phase diagrams. phase diagrams a phase diagram maps the ranges of composition and temperature over which particular phases are stable in a given system of materials. they enable the student to anticipate the alloys that are likely to have desirable properties, the treatments that can be used to develop best properties, and the treatments that should be avoided (newey and weaver, 1990). an understanding of phase diagrams is an important part of chemistry and related curricula (williams and collins, 1994), although the coverage of the topic in many textbooks has been criticized (kildahl, 1994). the phase diagram used in the works metallurgist is a binary system consisting of tin and lead, and it has commercial importance as soft solders. software: the works metallurgist the works metallurgist is an interactive tutorial designed to teach interpretations of phase diagrams and the lever rule (a method of calculating percentage of solid and liquid in an alloy at a given temperature from a phase diagram) to open university students who are studying at a distance. it was developed for materials engineering and science, a second-level course (equivalent to second-year full-time study) which runs from february until october and includes a residential summer school. course evaluations had shown that students had difficulties in interpreting and applying phase diagrams, and the program was specifically developed to help students in this area. it is in a game format, and students are given job titles ranging from applicant for apprentice metallurgist to works metallurgist, according to their performance. the software is available for students to buy, and is one of five programs produced for the 52 volume 4 number i course. in order to run the programs, students need access to a computer with the ou's specification (basically, a pc) and either windows or gem. the package is also available at the course's residential school which provides another opportunity to use it. use of the works metallurgist at residential summer school students used the cal programs during their laboratory work, both while waiting for parts of the practicals to reach the next stage, and when they finished the experiments early. although the program was designed for individual use, students mainly worked in the laboratories in pairs, and usually chose to use the program in pairs too, and to discuss their answers with each other before typing them in. three computers were also provided in the student hall for use at any time. sample and design of study the population consisted of 540 students who were studying materials: engineering and science in 1995. the researcher attended two weeks of residential school (out of seven) and conducted observations and interviews; the students had reasonable familiarity with computers, and were given a questionnaire, attitude scale, and knowledge pre-test, along with the evaluation disk (a special version of the program that recorded some usage information). methods used we used a number of different approaches, and gathered different types of data. our rationale for using these different methods is reported in jones et al (in press). observation forty-four sessions were observed, with session times ranging from 3 to 53 minutes. observations were carried out both in the laboratories and in the student hall. students' inputs were recorded, and where appropriate the students were asked to supply reasons for their answers. although the observer tried to minimize the disturbance to students' natural progress with the program, she was occasionally asked for help with the tasks, and this was used as an opportunity to ask probing questions. questionnaires and evaluation disk a two-stage questionnaire was given to 60 students who were randomly selected from those with access to a computer. stage 1 of the questionnaire concerned previous ou courses, study of phase diagrams, and the students' computing background. students filled in this part of the questionnaire before they used the software in order to complete an attitude measure and a knowledge pre-test. stage 2 concerned the efficiency of the program, the quality and nature of students' interactions with it, their difficulties, their opinions of the program, and general comments. these 60 students were given a special version of the program which recorded usage information and were asked to return the disk after about five weeks. this enabled us to see how much time students spent on the program, which sections took most time, and how much improvement on their understanding they made in that limited time. attitude measure this was used to obtain a measure of students attitudes towards computers and learning with computers both before and after using the program, although the post-use questionnaire also 53 canon tosunog/u et ol the w o r t s metallurgist an evaluation of a cal package on phase diagrams contained items related to the program, and was to be completed by students along with the second part of the questionnaire at the end of the five-week period. achievement measure students completed a knowledge pre-test along with the first part of the questionnaire, and they were tested again (using the same questions) after they had finished working with the program. four more short questions covered the important concept of phase. interviews formal and informal interviews were conducted in the laboratories and in the computer suite in the student hall. tutorials related to phase diagrams were also attended by the researcher, and students' attendance and activities during the tutorial were observed. how do students use the program? observations revealed that the works metallurgist was the program used most during the two summer-school weeks. comparatively very little time was spent on other programs, and even students who had the program at home used it in the summer school. most students who saw it for the first time decided to buy it, if they had access to an appropriate computer. these observations about its use and popularity are supported by the sales figures. students reported that the program was very useful, and most could only suggest making minor changes to it. none of the students said the program was difficult to use, and they were easily able to use specific features of it such as crosshairs, and the draw tie line and show labels facilities.1 the use of the calculator was found to be somewhat tedious in windows, but less so in the gem version. evaluation disks returned by students showed that they spent a great deal of time using the program, ranging from 12 to 276 minutes with a mean of 128 minutes. most of the students spread their use of the program over a number of sessions. the average number of sessions was 3.6. sections 1 to 4 require students to interact with the computer. there are also two further sections where students carry out the activity away from the computer. the overall results for sections 1 to 4 are given in table 1. (the total is given for all students across the whole program. mean, maximum and minimum values are per student for one section.) time spent using the program (minutes) errors made sections 1-4 total 2440 mean 9.25 maximum 36 minimum i mean z75 maximum 13 minimum 0 table i: analysis of returned evaluation disks (n = 25) attitude scale and questionnaire fifty percent of the questionnaires were finally returned and findings from the returned questionnaires are consistent with this positive view as can be seen in table 2. 54 alt-j volume 4 number i strongly agree agree neutral disagree strongly disagree i i did not learn anything new from 19 9the program 2 the program is capable of teaching about phase diagrams 3 the program makes you try to get a higher job title 4 using the works metallurgist assisted in my understanding of phase diagrams 5 the program is helpful in preparing for the exam 6 1 still would have preferred to learn about phase diagrams from a tutor 7 4 ii 10 21 ii 17 15 7 1 1 8 1 4 16 table 2: items from the attitude scale related to the works metallurgist (n = 29) do students learn from the program? students views related to the ease of use, value, learning, difficulty and motivational aspects of the program were positive, and the aim of the software was clear to every student. they also believed they had benefited: 'i think i have learnt more from this than reading the text book, it is very good, very good.' 'i need it.' 'quite educational.' the one student who was negative 'i don't like anything related to computers' initially chose not to use the program, but when he did use it later, he said to the observer: 'it's not like using a computer, it's like a game'. pre-test post test mean 3.82 3.91 std dev 1.06 1.08 n 28 23table 3: comparison ofpreand post-test scores observations supported their beliefs: students were observed to interpret phase diagrams and apply the lever rule correctly, although they were unsure about this procedure or unable to apply the rule correctly before using the program. however, the analyses of knowledge questions did not show any differences between the preand post-test scores (see table 3). as can be seen from the sample size in table 3, there was some reluctance to answer the posttest, and because the students were working at home, the timing and circumstances of the post-test were beyond our control. nevertheless, the lack of difference between preand posttest findings is surprising in this study as observations as well as the students' own perceptions did suggest some benefit. however, there are a number of difficulties with the use of preand post-tests in cal evaluation which are well documented (jones et al, in press). such 55 canon tosunoglu et ol the works metallurgist an evaluation of a cal package on phase diagrams instruments are not sensitive to the complexity of the learning situation, and may not reveal anything about the learning process itself. indeed, data from the evaluation disks clearly shows that the time spent and errors made in each section decreased with repeated use of the program. how can the program be improved? when asked about possible changes to the program, students suggested that it could be more flexible about spelling mistakes and routing of questions, and also include a don't know button. students mentioned the following points when asked about difficulties: using the calculator (27%), finishing the tasks in one session (20%), understanding questions (20%), understanding explanations to wrong answers (20%), calculating the answers (13%), and using the mouse and crosshairs (6%). these findings again are consistent with observations. fiftythree percent of students reported that when they had difficulties using the program, they repeated the program. the program can be further improved by small changes in question order, and wording to help students understand the questions better, and by providing help for students in deciding which side of the tie line represents what phase (i.e. solid or liquid), simplifying the operation of the calculator, clarifying the distinction between certain questions about solid or liquid composition, and allowing a little more flexibility with spelling mistakes. conclusion our overall findings showed that this program was perceived as very useful and necessary by students. they also stated that the game-like nature of the program, or getting it wrong, made them try again and again. this feature is very important because it shows that the program engages students, and as a result their chances of learning increases. the clear and simple interface of the works metallurgist is among the positive factors mentioned by students. we wanted to find out the extent to which students learn from the program. our findings did not show any direct relationship between its use and student performance, but our indirect findings indicate that it can contribute to student learning. in summary, the works metallurgist has certain critical properties that help its success: • students think they learn with it; • it is enjoyable to use; • it does not require too much computer familiarity; • it is highly motivational (humour, game-like format). note 1. 'crosshairs' gives the temperature and composition of any point on the diagram when the cursor arrow is placed on it. 'draw tie line' draws the tie line and gives the solid and liquid 56 alt-j volume 4 number i composition of a given alloy. 'show labels' gives the names of the different regions and lines of the phase diagram. references jones, a., scanlon, e., tosunoglu, c., ross, s., butcher, p., murphy, p. and greenberg, j., 'evaluating cal at the open university: 15 years on', computers and education, (in press). kildahl, n.k. (1994), 'journey around a phase diagram', journal of chemical education, 71 (12), 1052-5. newey, c. and weaver, g. (1990), materials: principles and practice, london, butterworth (in association with the open university). williams, k.r. and collins, s.e. (1994), 'the solid-liquid phase diagram experiment: updated for the physical chemistry laboratory', journal of chemical education, 71 (7), 617-20. 57 social psychology: new directions in computer-based learning lesley j. allinson school of social sciences and professional studies, university of humberside the provision of cbl within social psychology has been, to date, largely restricted to providing introductory teaching material and event-logging tools. experiment generators, as exist in the more formal domains of cognitive psychology, are required to actively engage students in acquiring practical professional skills. here we introduce the concept of multimedia experiment packages (meps), which lie somewhere between conventional cbl and free-form experiment generator packages. two case studies are described which offer integrated mep environments to promote sound research skills. introduction perhaps surprisingly, psychology has been a discipline eager to capitalize on the application of computers for teaching. traditionally, this has been for statistical calculations, and the presentation of experimental stimuli and the automatic collection of timed events (e.g., reaction times, choice-decision times). here, the traditional capabilities of computers are being exploited namely, their accurate temporal sequencing, graphical performance, and, above all, their number crunching. as such, they have been powerful and essential tools for those involved in the more psychophysical or cognitive areas of psychology. computer-based learning (cbl) remains very much a preserve of these more formal domains. the arrival of hypermedia has opened the way for cbl to be exploited within the less formal domains of psychology; but the level of interactivity is usually very restricted, and the constrained presentational styles means that even this technological progression fails to meet the contextual richness needed in the teaching of much of the behavioural sciences. the advent of multimedia has for the first time provided the potential to explore, within the normal undergraduate learning environment, real behaviour using the observational techniques that form the basic methodology of the practising social psychologist. this paper outlines the needs for multimedia experiment packages (meps) in social psychology tuition in terms of both their pedagogical and technological requirements. these 7 les/eyj. amnson social psychology: new directions in computer-based learning are illustrated by two case studies of prototype meps one for initial student exposure to observational methods, and the other for the detailed exploration of a complex category system for the microanalysis of conversational dialogue. the current development stage of these packages is more akin to traditional open-ended laboratory practicals in their functionality. however, the aim is to extend the current authoring aspects (now chiefly seen by the tutor) so that the students can incorporate their own material (e.g., edited video and transcripts), develop their own behavioural classification schemes, and effectively integrate the teaching of research methods with original research of their own. multimedia experiment packages for social psychology psychology and the behavioural sciences are — by their very nature — empirical. an essential facet of any undergraduate course is the exposure of the students to the design, implementation, and analysis of experiments. such activities have been traditionally very resource-intensive in terms of staff time and expertise, and equipment needs. computer-based experiment generators are readily available in the formal domains of psychology. here, the experiment is active in the sense that the experimenter designs and executes all aspects of the experiment. in social psychology, the typical approach is to observe human behaviour in a naturalistic setting. the experiment, or more correctly the non-participant observational study, can be said to be passive, as the experimenter merely observes what is occurring in some aspect of the everyday world. the scientific observation of complex social behaviour demands adeptness in detailed classification techniques whereby the precise nature of the individual differences in behaviour can be identified and recorded. these passive studies require a very active engagement in the data-collection process. the intricate and perceptual nature of the behavioural classification schemes demands that they are best taught by a guided exposure to numerous examples. it is necessary to learn by example rather than through a more rigorous approach. it is difficult to organize, for the teaching of these practical skills, a naturalistic exposure to the appropriate social behaviour. the behaviour of interest may occur only in specific non-public circumstances or in remote locations. so the apparatus that we have been forced to employ, in the main, is the video-recorder with all the associated ordeals of slow access and poor usercontrol. even accepting the limitations of carefully edited videos, the acquisition of skills in observing, recording and analysing social situations, interactions and conversations is tutorintensive if these skills are to be mastered at more than a superficial level. individual, or even small-group, tuition of observational skills is often no longer economically feasible within the rapidly changing realm of higher education. the seamless inclusion of video and sound in an integrated learning environment has made the computer-mediated approach not only viable but also capable of creating an enhanced educational experience tailored to the individual. the potential of cbl to promote deep active learning has been extensively reported and discussed elsewhere (laurillard, 1993), as have the requirement to recognize the differing learning styles and needs of the individual student (allinson, 1992). case studies the following two case studies illustrate different aspects of prototype meps. the first is aimed at a wide audience of students, and not only demonstrates the logistical benefits of 8 alt-] volume 4 number i computer delivery over traditional means but indicates how experience of experimental methodology, data collection and analysis, and assignment generation may be combined within a single package. the second is a more advanced, and specialized, topic where the underlying educational objective is to provide tools for the deep appreciation of a complex classification system. in this short paper, the emphasis will be functionality of the packages rather than detailed evaluation findings (which will form the basis of a later paper). case study introduction to observational methods this mep was designed to introduce a large number of first-year undergraduates, taking a range of social science courses, to the elements of observational methodology. the concept of observing social behaviour is not new to students; it is an everyday activity for all of us. mostly, these observations are unfocused and unintentional. observations that are conducted as part of social science research need to be deliberate, systematic, accurate and record interesting aspects of behaviour. it is these differences that need to be emphasized to students, and solid foundations laid for further courses both in appreciating the work of others, and in designing and conducting their own research. the example behaviour chosen for this first exposure to observational methodology is based on the collision avoidance behaviour of pedestrians. when individuals approach one another on a collision pathway, a collision will occur unless one or both of the pedestrians take some form of avoidance action. collett and marsh (1981) found that while men oriented themselves towards the person they were passing, women tended to turn away. turning in towards a person has been termed an open pass, while turning away has been termed a closed pass. the example is chosen as it a relatively simple explicit instance of non-verbal behaviour, and can form the basis of further discussion (either in tutorial classes or through assignments), and the students can be encouraged to observe this activity in their everyday environment. this final point is important as it reinforces the relevance of social science studies in attempting to explain previously unnoticed activities. the existing mechanisms for conducting this exercise were the provision of paper documentation and an edited videotape giving examples of open and closed passes, followed by examples which need to be classified. this edited tape was, in fact, produced from over three and half hours of recorded video. the practical difficulties were the logistical ones of booking out one of a small number of tapes, finding a videoplayer, and then struggling with its poor operational interface. a computer-based solution overcomes all these constraints on the student's learning opportunities. the standard integrated quicktime video controller offers far greater control than that of a videoplayer. the students are more acquainted with manipulating a computer interface through extensive exposure to other cbl packages than operating a unfamiliar videoplayer. computer use is familiar to students as it skills are compulsory elements for all first-year students at the university of humberside. the original paper documentation could not take into account differences between videoplayers; but the mep presents a uniform interface together with the relevant online assistance. the learning package has been deliberately kept simple in its presentational form (see figure 1). however, the package provides much more than the previous delivery method not only is background information, data-entry and statistical testing integrated, but the relatively small number of events analysed by the student can be merged with a larger data set, and the results (both raw data and results of the statistical tests) can be transferred directly to their assignment reports. the original videotape exercise has been presented as both a group activity and as an individual self-learning activity. when conducted as a group exercise, there was poor 9 lesleyj. allinson social psychology: new directions in computer-based learning :. collision open pass example observe man in white shirt walking towards camera cbserve woman in pink towards camera [ i introdactios i 1 crfssicm.avoidai«ab sequences figure /: training examples of open and dosed passes from collision avoidance: an introduction to observational methods agreement between the students in classifying the events even with carefully edited and wellspaced videoclips. this suggests that improved user control is required to ensure the required level of concentration on the task. the ease of access, selection and manipulation of the video provided by the computer interface, have overcome this problem. the computer-based approach, though intended as an individual self-learning activity, does allow students to work in ad hoc groups if they wish. as these introductory practicals are integrated into the it skill acquisition programme, for all social science students, this enriched utilization of computers is most appropriate at this time. case study microanalysis of political interviews the analysis of social interaction has played a significant role in social psychological research; and various techniques have been developed in order to further our understanding of this aspect of social behaviour. the example used in this mep concerns the detailed structure of how politicians and interviewers relate within the bounds of televised interviews (allinson and bull, 1995). the topic has an inherent general attraction to students; and notwithstanding its importance as an example of social-psychology research, it provides an excellent basis for presenting the complex classification schemes that can result from detailed conversation analysis. the analysis of equivocation in political interviews by bull and mayer (1993), which 10 alt-j volume 4 number i political flnnlysls r: us; f.mwr ^ sgj^d dd: what about your attitude to trade unions you've said you're going to give a massive of power to trade unions if labour comes back isn't that something again that people are fearful of that is going to lose you votes? nk: yes 1 haven't said by the way that we're going to give massive return of power i've never used such a phrase in my life, simply because the politician responds with a yes or no, does not always mean they are giving a reply. thus, "yes" maybe used simply to acknowledge the question rather than reply to it, as in this example from the 1987 general election. in sayingyes, neil kinnocx is not replying to j the question but simply acknowledging it; in ' fact, he goes on to attack the question bv « t e < .* >' . v~* ".f ' > "•',, "-" ' -"' 1 1 rupo replies/delayed replies |> intermediate replies ignores q acknowledge no r questions the q modifies q rttacks q rttacks interviewer declines to answer • political point self interruption non-specific answer negative answer repeats preuious r rlreadg answered rpology literalism ns ekternal attack presenting policy justifying policy reassurance rppeal to nationalism offerinq analysis self-justification talking own side talks up self figure 2: self-exploration access mode from political interview analysis. the user can select a complete or partial question/response pair from the hierarchical menus (shown in the lower figure) and then play the recovered video clips. note the provision of video (with' sound), transcript and associated notes i i lesleyj. ajiinson social psychology, new directions in computer-based learning forms the background of this package, specifies a set of criteria for identifying questions, replies and non-replies to questions in political interviews, and a scheme whereby different types of non-replies could be categorized to date, 35 different types of non-replies have been identified! this on-going work forms part of a second-year social psychology module at the university of york. existing practical classes were very expensive in terms of expert time. with the expert tutor able to demonstrate only a few categories of question/response pairs and provide edited video examples, the practical class was forced to resort to the analysis of audio cassettes (aided by transcripts and guidance notes). it was impossible to provide a classroom of videoplayers. with the provision of computer delivery, the students are now able to engage in more in-depth analysis, one outcome being a much enhanced standard of assessment. previously, it had not proved possible to assess this activity in a meaningful manner. the structure of this hierarchical classification system differs from the intrinsic knowledge base of more conventional hypermedia systems in that the knowledge base is not a generalized network of information screens but a well-ordered hierarchy. there is still a need to allow the user to transverse the structure of the classification system in a variety of ways; namely: • guided tour author-prescribed route aimed at presenting an ordered view of the classification scheme • self-exploration user selects partial/complete question/response types to retrieve a specified subset of examples (see figure 2) • self-assessment user presented with example(s) and required to guess the correct question/response type(s) • categories user selects question/response type or a subset via a diagrammatic representation of the classification scheme. these differing navigational/learning modes not only endorse such cognitive principles as tulving's encoding specificity i.e., material is easier to recall if it contains distinctive retrieval cues that can be regenerated at the time of recall and encoding variability i.e., multiple exposure to the same material in differing contexts will aid recall (tulving and thomson, 1973), but there are other educational strengths. for example, access via the categories mode enables the hierarchical nature of the classification scheme to be directly discernible through its spatial arrangement. the interface is necessarily more complex than in the first case study, but across all the modes it presents as far as possible a uniform functionality and appearance. the interface can be briefly described in terms of the selfexploration mode. the user selects a complete or partial question/response pair from the hierarchical menus. the selection appears in the question and response frame, and the number of available examples is also indicated. the details of the first example (i.e., participants, transcript, notes and first frame of video) are displayed. the video can be played (any number of types). the user can move through the various examples using the previous clip and next clip buttons. details are displayed of the user's current position within the example sequence. all data files (e.g., video clips, transcripts, associated notes and classification codings) are maintained separately to the mep application. this allows the easy replacement or addition of 12 volume 4 number / material. a number of support tools are available assist in the authoring processes. at present, these are used only by the tutor, but it is intended that more comprehensive tools will be made available to the students so that they can construct their own classification systems for related discourse activities. this particular application lends itself well to multimedia presentation. firstly, the microanalysis of discourse is necessarily an intense activity. the analysis of the discourse from written transcripts alone is tedious, and it is therefore difficult to motivate students. the inclusion of the video jointly with the textual transcript helps to add interest and increase motivation as well as emphasizing the detailed context of the message. the non-verbal aspects of voice intonation, phrasing, gesture and facial expression are of immense additional value. the complete package provides direct access to full annotated background papers and introductory material. there are over 60 videoclips, which together with associated transcripts and notes, comprise over 130 megabytes of data. the investment, in terms of the development effort, has relieved the tutor from much of the menial work associated with this teaching module, and has provided tools for easily modifying the content as well as improving the educational experience and assessment. future directions these two case studies illustrate the capacity of current cbl tools for the implementation of social psychology meps. the first study presents a simple system for widespread use among novice social-science students. the aim is to offer a seamless extension from the limited supplied examples of collision events to where students can incorporate their own video recordings. there should be a natural progression from introductory/training material, through open-ended practicals, to open research activity. the second study though exemplifying the strengths of multimedia to provide cbl packages for the study of complex human interactions, has revealed that there is much within the authoring aspects that is still very time-consuming the transcripts have to be produced manually. the current standing of automatic speech-to-text transcription is still far removed from coping with the subtleties of finely structured conversations. the same can be said about the automatic recognition of bodily and facial gestures. advances in these areas will come, and will have their initial impact on research but they will feed through into teaching. the boundaries between these currently distinct activities will become less perceptible as computational power and availability increases. the concept of the mep introduced in this paper goes some way in providing the generic experiment generator shell for social psychology. increasing productivity and rising student numbers are often quoted as major factors influencing the introduction of cbl; but the application of comprehensive multimedia implementations for the teaching of practical social psychology skills can be viewed as an augmentation of traditional methods and not merely an imposed substitute. there should be a merging of teaching and research tools, so enabling active learning and furthering the eventual goal of the virtual real-world social psychology laboratory. references allinson, l.j. (1992), 'learning styles and computer-based learning environments', in tomek, i. (ed) computer assisted learning, berlin, springer-verlag, 61-73. 13 lesley j. allinson social psychology, new directions in computer-based learning allinson, l.j. and bull, p. (1995), 'teaching the practical skills of the social psychologist: the analysis of political interviews', psychology teaching review, 4, 90-102. bull, p.e. and mayer, k. (1993), 'how not to answer questions in political interviews', political psychology, 14, 651-66. collett, p.r. and marsh, p.e. (1981), 'patterns of public behaviour: collision avoidance on a pedestrian crossing' in kendon, a. (ed), non-verbal communication. interaction and gesture, the hague, mouton press, 199-217. laurillard, d. (1993), rethinking university teaching, london, routledge. tulving, e. and thomson, d.m. (1973), 'encoding specificity and retrieval processes in episodic memory', psychological review, 80, 353-73. 14 (page number not for citation purpose) 1 *corresponding author. email: ninabe@dsv.su.se research in learning technology 2020. © 2020 n. bergdahl and j. nouri. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 research in learning technology vol. 28, 2020 original research article student engagement and disengagement in tel – the role of gaming, gender and non-native students nina bergdahl* and jalal nouri department of computer and systems sciences (dsv), stockholm university, stockholm, sweden received: 28 june 2019; revised: 6 december 2019; accepted: 15 may 2020; published: 19 august 2020 student engagement is critical for learning. however, little is known about engagement and disengagement and particular social groups. recent research has alerted that engagement in technology-enhanced learning (tel) settings may manifest differently than engagement in analogue learning settings. this study explores how different social groups of upper secondary school students (n = 410) engage and disengage when learning with digital technologies. we used an instrument to approach dimensions of engagement and disengagement in tel. using thematic analysis, we identified cognitive, emotional, behavioural and social aspects of engagement and disengagement in eight-student interviews which together with theory, informed a questionnaire. using statistical methods, we explored the relationship between engagement, disengagement and the social categories: gamers, gender and non native speakers. we found significant differences between the groups. for example: that high-frequency gaming students were not as easily distracted as students reporting low-frequency gaming, that female students engaged in tel in different ways than male students, and that non-native speakers displayed significantly fewer tendencies to engage in unauthorised uses of digital technologies than native speakers. identifying indicators reflecting engagement and disengagement in tel in social groups can inform successful practices that stimulate student engagement and can be used to avoid, or redeem, group-specific challenges that trigger disengagement. keywords: student engagement; student disengagement; technology-enhanced learning; gender; gaming; non-native speakers introduction engagement is often seen as a key concept in education with its strong relations to school retention, grades, academic outcome and student well-being (e.g. alrashidi, phan, and ngu 2016; fredricks, blumenfeld, and paris 2004). while educational psychologists have explored engagement for some 30 years in traditional classrooms, researchers in other fields have recently forwarded that student engagement in technology-enhanced learning (tel) settings may be different from engagement in analogue learning environments (bergdahl et al. 2019a; halverson 2016; ma, cheng, and han 2018). they propose that engagement in tel is rather unexplored (henrie, mailto:ninabe@dsv.su.se http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v28.2293 n. bergdahl and j. nouri 2 citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 (page number not for citation purpose) halverson, and graham 2015; ma, cheng, and han 2018) and ‘ill-conceived’ (halverson 2016). engagement is manifested in the interaction between the individual and the subject, in which engagement is shaped by context (wang and hofkens 2019). thus, new conditions for engagement may arise when digital technologies reshape the student–content, student–machine (hardware and software), student–teacher and student–student interactions. while some research gather system log data and conduct data-driven analytics to measure engagement (e.g. counting clicks, words or time spent in learning management systems) (henrie, halverson, and graham 2015), this type of system log data has been criticised as it, at best, can reflect a one-dimensional aspect of in-the-moment online behaviour and thus will not capture the emotional, behavioural, cognitive and social aspects of engagement (and disengagement) (henrie et al. 2018). several studies approaching digital technologies in education suggest that the vast majority of students are comfortable with digital technologies; that the students of today find the digital environment and internet nested in their everyday lives. unsurprisingly, students generally believe that laptops support them in their school work (howard, ma, and yang 2016; rashid and asghar 2016). however, other researchers have pointed out that some digital technologies may impacts the students profoundly. for example; while students admit that their mobile phones cause distraction, they still invest heavily in their devices, rely on them to manage their education and even allow the mobile phone to merge with their identity (gikas and grant 2013). students do not view devices or the internet as something they would turn on or off; instead, they expect to be able to use these technologies for all purposes, including learning. for example, students reported that technologies were time-saving, that they used them for deep learning, to support basic tasks, augment teacher materials, managing education and allow for flexibility of location (henderson, selwyn, and aston 2017). furthermore, students often get frustrated with teachers’ low skills, tolerance or inclusion of technologies (gikas and grant 2013). while researchers have started to approach both engagement and disengagement (balwant 2018; bergdahl et al. 2019b; fredricks, blumenfeld, and paris 2014), there has been a call for more nuanced studies that approach differences of engagement between groups of students (fredricks et al. 2018; hietajärvi et al. 2019). as it is well known that disengagement may escalate into school dropout (finn 1989; wang and fredricks 2014), schools could identify early signs of disengagement and use preventive measures to curb future absenteeism. while it has been suggested that absenteeism is the strongest predictor for disengagement (pellerin 2005), school attendance is not enough for students to engage (finn 1989). yet, little research has explored how social groups, are related to engagement in learning, which leaves a gap for a more nuanced understanding of how different social groups of students engage and disengage in tel. learning more of underperformance and productive engagement in tel for specific groups of students, approaching social groups is critical. research has highlighted that students who belong to specific social groups (such as gender, ethnicity or even gamers) may experience being part of a disadvantaged group, which, in turn, may impact their performance (chatman et al. 2008; wiggins et al. 2017). focusing on social groups, such as gender and non-native speakers, when exploring engagement, rather than adopting an ‘across population’ approach, is thus needed to understand particular sub-groups (scherer, rohatgi, and hatlevik 2017). we included gaming students, as time spent gaming has been seen to be a problem related to decreasing grades http://dx.doi.org/10.25304/rlt.v28.2293 research in learning technology citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 3 (page number not for citation purpose) (bergdahl et al. 2019a; hietajärvi et al. 2015). we included the perspectives of gender and of non-native speakers too, as research has suggested that certain social groups (i.e. females and non-native speakers) may have lower digital skills or less access to digital technologies compared to male students, and therefore may face unequal possibilities in tel (siddiq and scherer 2019; skryabin et al. 2015). understanding female and male students and native and non-native speakers, learning design can build in personalisation and prevention before disengagement escalates into a problem. all in all, little is known about engagement and disengagement in tel in general, less the role of gaming, gender and non-native speakers, in particular. there are several studies that explore gaming, gender, non-native speakers in relation to school results. while highly relevant, these studies have not explored the behavioural, cognitive, emotional and social aspects of engagement and disengagement in tel and how these are related to the specific social groups. to contribute to knowledge in this area, we raised the following three hypotheses: h1: we hypothesise that students’ academic engagement and disengagement, when learning with technologies, differ between students who report highand low-frequency gaming. h2: we hypothesise that students’ academic engagement and disengagement, when learning with technologies, differ between female and male students. h3: we hypothesise that students’ academic engagement and disengagement, when learning with technologies, differ between native and non-native speakers. theoretical background student engagement and disengagement are related but separate constructs (skinner, kindermann, and furrer 2009; wang et al. 2017). thus, research which only focus on engagement would potentially overlook facets of disengagement. student engagement can be described as the visible and measurable outcome of motivation (boekaerts 2016; fredricks and mccolskey 2012), and has been related to overall school success for all students (alrashidi, phan, and ngu 2016; fredricks, blumenfeld, and paris 2004) and defined as students’ initiation of effort and their persistence in learning activities (alrashidi et al. 2016). research exploring cognitive, behavioural, emotional and social aspects of engagement in tel, in particular, has shown that high-performing students own the proactive ways of using digital technologies to succeed in education (bergdahl et al. 2019a), while low-performers significantly more often were seen to switch between different digital technologies for other purposes than learning (ibid.). eventually, student disengagement may lead to withdrawal from peers and school, decreasing grades, absenteeism and school dropout (balwant 2018; wang et al. 2017), cementing states that are unproductive for learning (chipchase et al. 2017; mann 2001). fredricks et al. (2004) brought together common operationalisations and definitions of engagement and summarised that engagement consists of interrelated dimensions consisting of, at least, a behavioural, an emotional and a cognitive aspect. the behavioural aspect refers to effort, persistence, attention, participation, positive conduct and lack of disruptive behaviour. the emotional aspect included the students’ reaction towards teacher instruction or peers. the cognitive aspect refers to students’ http://dx.doi.org/10.25304/rlt.v28.2293 n. bergdahl and j. nouri 4 citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 (page number not for citation purpose) self-regulated learning and the focus and effort directed to comprehend complex ideas and master demanding skills. today, several researchers include a social dimension when conceptualising engagement (skinner, kindermann, and furrer 2009; wang et al. 2017) and suggest that the social dimension reflects collaboration and interaction with peers. the questionnaire in this study was informed by students’ experiences of their own engagement and disengagement when learning with digital technologies, and inspired by the four-dimensional conceptualisation of engagement and disengagement that include emotional, cognitive, behavioural and social dimensions, proposed by wang et al. (2017). however, as this study reflects engagement and disengagement in tel, in particular, we used graham’s definition of tel, (graham 2006). graham proposed that blended learning is a combination of face-to-face learning and learning with digital technologies, which can be viewed in a continuum reflecting how digital technologies are used and their inherent modalities. depending on the extent to which the digital technologies change the conditions for learning, the lower end is exemplified with technologies that enable learning and the higher end is exemplified as learning technologies that transform learning, leaving technologies that enhance learning in the middle. the lowto high-end continua reflects how the use of digital technologies range from not impacting pedagogical approaches, to overthrowing the traditional teaching practices in terms of for example pace, place and time (ibid.). exploring the cognitive and emotional aspects of engagement in blended learning, halverson’s concluded that student engagement in “traditional” (or analogue) learning settings and student engagement in tel must be treated as separate constructs, as digital technologies significantly affects engagement (halverson 2016). recent research has suggested that students who display high levels of engagement, in traditional (analogue) learning settings, are more likely to display high levels of engagement in tel (bergdahl et al. 2019a). however, with the uptake of digital technologies, tel disengagement has been seen to increase. for example, students have been seen to increasingly work in isolation and to use the digital technologies to disengage from learning (tallvid et al. 2015). tallvid observed that the unauthorised use of digital technologies increased in parallel to the overall use of digital technologies (ibid.). other research has highlighted that teachers’ social presence in blended learning is related to student engagement and grades, and while equally important in online as in traditional (analogue) learning settings, it may be harder to realise in online (bergdahl et al. 2019a, b). while many approaches to engagement research have aimed to cover students’ overall engagement and disengagement, students are not a homogeneous group. in this study, we investigated engagement and disengagement in tel in relation to specific social groups of students, namely high-frequency gamers, gender and non-native speakers. gaming, gender and non-native students research on gamers’ engagement in learning has been inconsistent. unsurprisingly, research has shown that gaming in class is related to lower grades (bergdahl et al. 2019b). while bergdahl (ibid.). could not find any negative correlations between playing games outside class and student grades, research findings are inconsistent. for example, strittmatter and colleagues concluded that too much gaming can have negative effects when the gamer tries to engage with people face-to-face as they are increasingly deprived of face-to-face contact (strittmatter et al. 2015). as such, http://dx.doi.org/10.25304/rlt.v28.2293 research in learning technology citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 5 (page number not for citation purpose) high-frequency gamers may feel less competent in social relationships, which, in turn, can have negative effects on their social engagement when learning at school. however, other studies suggest that gaming can help develop pro-learning skills, like being familiar with multiple platforms, or developing their ability to focus in technology-mediated activities (yurov et al. 2014) and forward that even violent games have little influence on anti-social behaviour, decreased academic performance or attention deficits (ferguson 2015). approaching gaming preferences specifically hietajärvi and others highlighted that gamers who preferred action and sports games, displayed lower levels of engagement, while using digital technologies to gain and share knowledge was associated with higher levels of engagement (hietajärvi et al. 2019). it is, however, worth noting that a longitudinal study concluded that student engagement tended to decline more for gamers than other students (wylies and hodgen 2012). studies have also suggested that while different engagement dispositions were related to different preferences (action games, sports games, etc.), male students were significantly more likely to engage in all kinds of games, whether they displayed indifference for learning or goal orientation (hietajärvi et al. 2019). the differences in engagement disposition with female and male students indicate that a gender perspective is essential to explore in relation to engagement and disengagement in tel. previous research approaching the gender perspective has suggested that female students are more likely than male students to express that their teacher relationship is critical for their engagement. female students are less likely to participate when feeling afraid or worried (fredricks et al. 2018). in contrast, male students were seen to display persistence and deep learning strategies, reflecting higher levels of self-confidence (fredricks et al. 2018). skinner, kindermann, and furrer (2009) identified that female students displayed higher levels of behavioural and emotional engagement (than male students), as well as lower levels of behavioural and emotional disengagement. applying a gender perspective in tel, research has indicated that females and males take on different tasks when using and learning with technologies (ilomäki 2011; volman et al. 2005) which implies that female and male students may differ when engaging in and disengaging from tel. however, studies show disparate results. for instance, while ilomäki (2011) has highlighted that male students are using digital technologies in proactive ways and outperform female students, other researchers have suggested that male students more frequently used it for recreational purposes ( gebhardt et al. 2014). hietajärvi et al. (2019) concluded that disengagement with digital technologies differ between the genders: female students would use social media more frequently and male students would prefer gaming. the access to, and use of, digital technologies also influence the possibilities to develop the skills needed. it is suggested that the past decade’s male dominance in digital skills might be in the process of levelling out and that owning high digital skills is becoming increasingly equal between men and women (gebhardt et al. 2014; siddiq and scherer 2019). however, the digital divide is not only about exploring differences in access and it skills between the genders. while these two aspects (access and skills) have been forwarded as a first and second wave of the ‘digital divide’, a third wave explores a situation in which the individual has access to digital technologies as well as skills, but may not obtain the same returns on their use digital technologies (van deursen and helsper 2015). such situation may be the reality for non-native speakers. in a recent spanish study, the authors suggested that teachers use fewer digital technologies in class when they teach immigrant students and if they perceive http://dx.doi.org/10.25304/rlt.v28.2293 n. bergdahl and j. nouri 6 citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 (page number not for citation purpose) that it leads to disruptive behaviour (gómez-fernández and mediavilla 2019). this may have its explanation in understandings that non-native speakers may find it hard to interpret social signals and/or perceive their cultural practices to be devalued in the school context which might lead to a lack of engagement in school (bingham and okagaki 2012), or, that immigrant students may have had limited access to digital technologies (skryabin et al. 2015), and thus limited opportunities to develop digital skills to be able to engage effectively in tel. opposing findings were presented by scherer, rohatgi, and hatlevik (2017) who when approaching sub-groups of students (gender and non-native students) and found that female and non-native students were significantly more likely to display proactive learning behaviours in tel. they also found that both female and non-native students used digital technologies, across varied contexts, for a variety of reasons, almost to the same extent as male and native students, (scherer, rohatgi, and hatlevik 2017). we found little research exploring tel conditions and social groups (i.e.  gaming/non-gaming students, male/female students and native/non-native students). yet, the social groups are subjected to the same classrooms, teacher and tools, and thus, not separated in reality. exploring how these groups engage and disengage, and the specific components of engagement and disengagement when learning with digital technologies is thus essential, as different social groups are likely to respond differently to similar situations. method research design we used a mixed-methods exploratory sequential design which allows an initial qualitative phase of data collection and analysis, and a second phase of quantitative data collection (berman 2017). the exploratory sequential design was used to develop a multidimensional measurement to assess student engagement and disengagement in tel, and thereby gain a nuanced understanding of students’ needs and preferences when learning with technologies. in the initial phase, we conducted eight interviews to inductively operationalise students’ reports of engagement and disengagement in tel. these interviews were analysed using thematic analysis (clarke and braun 2016) which, together with engagement theory, informed the instrument (following bryman 2016). in the second phase, we distributed, analysed and validated the questionnaire using statistical tests (field 2018). context and participants to approach engagement in tel in upper secondary schools, we first selected all schools in stockholm that had participated in a digitalisation project with the development of a school platform (ensuring one laptop per student). we then limited the selection to cover the largest of the national programmes that qualify students for higher education: economics, social and natural sciences and the humanities programme. demographic variables such as gender and non-native speaker were assessed by student self-reports. of the 410 students answering the questionnaire, 33% considered themselves high-frequency gamers, and 67% low-frequency gamers; 37% were male, 63% were female, 87% of the students reported they were native speakers, and 13% that they were non-native speakers (see table 2). http://dx.doi.org/10.25304/rlt.v28.2293 research in learning technology citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 7 (page number not for citation purpose) the instrument we approached two teachers in two separate upper secondary schools, with which our department had established connections, and asked for their interest to participate in a study. eight students in their second or third year (four from each school; six boys, two girls) agreed to be interviewed. we asked the participants for their experiences of (dis)engagement in tel settings (e.g. ‘can you describe your engagement when learning with digital technologies?’ and ‘what was a learning situation with digital technologies that you felt maximised your engagement?’). a thematic analysis was conducted to analyse the interviews (clarke and braun 2016). coding was performed in nvivo version 11.4.43 to identify instances in which students reported feeling engaged or disengaged when learning with technologies, considering behavioural, cognitive, emotional and social dimensions (wang et al. 2017). to capture student-reported engagement in tel 49 questions (with an even distribution across behavioural, emotional, cognitive and social dimensions of engagement or disengagement) formed the basis of the developed questionnaire. we adopted a six-item likert scale, ranging from ‘not at all like me’ to ‘very much like me’ (if nothing else is stated). we then pre-tested the questionnaire on two separate occasions. to ensure the questionnaire would be understood and relevant to the age group, the pre-test was first conducted with five lower secondary school students, and the second time with five students who had just graduated from upper secondary school. the pre-test resulted in further clarification and merging of some questions. ethical considerations our ethical considerations adhered to good research practice (hermerén 2011). the respondents received both verbal and written information about the study, prior to asking for their consent. the students were informed of their right to withdraw without questions asked. all data were treated using pseudonyms at all times. data collection two types of data were collected. in phase 1, eight students were interviewed, and in phase 2, we collected questionnaire results. we distributed the questionnaire to students in 11 upper secondary schools. out of 472 responses, 62 responses were excluded, due to not being filling the requirements for the selection. data screening was done to check for standard normal distribution (snd) of the participants across the studied variables using spss version 25. missing data were below 1%. following field (2018), variables were examined for skewness and kurtosis, and outliers were identified using mean substitution, reflecting report within snd. we used descriptive statistics to explore the data, and independent sample t-test to test our hypotheses. validation in addition to pre-test the instrument for ecological validity (using interviews and theory as described above), the researchers critically examined items that required clarification, were overlapping or were not sufficiently represented to reach face validity. table 1 presents the instrument with two separate four-dimensional constructs (engagement and disengagement) with cognitive (cog/dcog), emotional (emo/demo), behavioural http://dx.doi.org/10.25304/rlt.v28.2293 n. bergdahl and j. nouri 8 citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 (page number not for citation purpose) table 1. indicators of engagement and disengagement. engagement disengagement beh l uses digital technologies as a support for learnirg dbeh l turns in assignments late due to unauthorised use of technologies beh 2 uses the internet to research what others have written and find facts dbeh 2 delegates group work using technologies to a peer, as no more than one is required to complete the task beh 3 uses a synchronous media to rehearse and master content dbeh 3 is prevented from working due to undeveloped technologies and system breakdowns beh 4 uses technologies to work on school assignments dcog l chooses to use digital technologies in unauthorised ways cog l concentrates well when using digital technologies dcog 2 notifications (i.e. from a mobile phone) easily cause a distraction cog 2 takes own initiative and decides on what digital technologies to use dcog 3 is overwhelmed by information overflow cog 3 needs digital technologies to maximise learning demo l is emotionally drawn to an application or digital technology cog 4 uses it as a cognitive enhancement demo 2 uses digital technologies to escape feelings of boredom emo l wants to use more digital technologies for learning than what is used today demo 3 feels frustration over poor communication over the learning platform emo 2 perceives using digital technologies for learning as engaging demo4 believes that teachers lack the it skills needed to support individual learning effectively emo 3 relies on technologies to manage education demo 5 resists using the laptop for all reading emo 4 finds creating with technologies satisfying demo 6 resists using the laptop for all writing emo 5 desires spatiotemporal solutions and personalisation that technologies may offer dsoc l is left to manage digital technologies for learning themselves soc l is satisfied with using digital technologies that mediate teachers’ insight into student’s learning process dsoc 2 coming to school is perceived as meaningless when there are no interpersonal activities soc 2 is satisfied that teachers use digital technologies to provide feedback dsoc 3 is unhappy with repeatedly being directed to learn by looking things up online soc 3 feels that digital technologies are used in ways that enable participation, inclusion and belonging dsoc 4 feels upset/dispirited that group work does not involve all students soc 4 experiences teachers’ social presence with, as well as, inside applications dsoc 5 experiences decreasing engagement due to feeling isolated while using technology (beh/dbeh) and social (soc/dsoc) indicators of engagement and disengagement when students learn with digital technologies. each question was mapped to the relevant indicator. the instrument was then further tested and validated using principal component analysis and confirmatory factor analysis (bergdahl et al. 2019a). http://dx.doi.org/10.25304/rlt.v28.2293 research in learning technology citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 9 (page number not for citation purpose) results to explore how different groups of students engage and disengage when learning with technologies, we divided the students into groups reflecting gaming frequency, gender and native/non-native speakers, based on the questionnaire answers (see table 2). the calculated mean was used to divide students into highand low-frequency gaming students. we then continued with testing hypotheses (h1–h3) using independent sample t tests. hypothesis 1 (h1) gaming, engagement and disengagement an independent sample t-test was performed to test the hypothesis: that students’ academic engagement and disengagement, when learning with technologies, differ between highand low-frequency gaming students. table 3 shows the significant differences between highand low-frequency gaming student’s engagement and disengagement. while the most variables reflecting engagement and disengagement were similar for highand low-frequency gaming students, certain variables reflecting cognitive engagement and disengagement when learning with technologies showed significant differences ( p < 0.01). interestingly, we found that high-frequency gaming students did not use the internet to conduct research (explore what others have written on a topic), nor did they perceive that it was a cognitive enhancement as much as students reporting low-frequency gaming. on the other hand, low-frequency gaming students reported that they were more easily distracted by notifications (i.e. from mobile phones), as well as overwhelmed by information, to a higher extent than high-frequency gaming students. reflecting emotional table 2. background variables. student background data n % gender men 152 37 women 258 63 non-native speakers native speakers 356 87 non-native speaker 54 13 highand low-frequency gaming student high-frequency gamer 136 33 low-frequency gamer 274 67 table 3. differences between highand low-frequency gaming student’s engagement and disengagement in tel. variables low-frequency gaming high-frequency gaming n m sd n m sd t p beh2 274 5.09 0.90 136 4.70 1.22 3.72 <0.01 cog4 4.40 1.29 3.91 1.54 1.17 <0.01 dcog2 3.21 1.45 2.82 1.40 2.57 0.01 dcog3 3.07 1.36 2.71 1.34 2.50 0.01 demo6 3.01 1.60 2.68 1.59 1.99 0.04 http://dx.doi.org/10.25304/rlt.v28.2293 n. bergdahl and j. nouri 10 citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 (page number not for citation purpose) disengagement, low-frequency gaming students (more than high-frequency gaming students) resisted using a keyboard for all in-class writing. having tested our hypothesis: that engagement and disengagement when learning with technologies differ between highand low-frequency gaming students, we conclude that the hypothesis is partly supported. more specifically, while results revealed similarities between the behavioural, emotional and social dimensions, there were significant differences between highand low-frequency gaming students in tel, within the cognitive dimension of both engagement and disengagement. hypothesis 2 (h2) gender, engagement and disengagement we hypothesised that students’ academic engagement and disengagement, when learning with technologies, differ between female and male students. to test the hypothesis, we performed an independent sample t-test with gender as a grouping variable. table 3 reveals that the statistically significant differences between female and male engagement and disengagement in tel span the behavioural, cognitive, emotional and social dimensions. in variables measuring behavioural engagement, there were differences significant at the level of p < 0.01 between male and female students, revealing that female students to a significantly higher extent than male students use digital technologies to support learning, by using digital technologies to work on school assignments, by using the internet to conduct research (explore what others have written on a topic) and to find facts, and by using asynchronous media to rehearse and master content. female students displayed a significantly higher degree of cognitive engagement (p < 0.01–0.04) for half of the variables (see table 4): reporting that they use it as cognitive enhancement and need the digital technologies to maximise their learning. addressing emotional and social engagement, female and male students displayed many similarities. in sum, the differences identified, significant at the level of ( p  <  0.01),  showed that female students rely on technologies to manage education. table 4. differences between male/female students’ engagement and disengagement in tel. variables male female n m sd n m sd t p beh1 152 4.86 1.02 258 5.23 0.81 -4.07 <0.01 beh2 4.67 1.13 5.13 0.92 -4.42 <0.01 beh3 4.06 1.41 4.57 1.20 -3.93 <0.01 beh4 4.45 1.30 4.98 1.01 -4.61 <0.01 cog3 4.00 1.55 4.31 1.33 -2.07 0.04 cog4 3.93 1.38 4.42 1.38 -3.40 0.01 emo3 4.69 1.26 5.23 0.90 -4.98 <0.01 soc4 3.50 1.35 3.03 1.03 -3.48 0.01 dcog2 2.76 1.44 3.27 1.43 -3.44 0.01 demo1 3.04 1.40 3.48 1.50 -2.91 0.04 demo6 2.69 1.61 3.01 1.59 -1.98 <0.05 dsoc1 3.83 1.36 4.10 1.25 -2.03 0.04 dsoc4 3.99 1.35 4.59 1.31 -4.48 <0.01 dsoc5 2.89 1.46 2.50 1.30 2.78 <0.05 http://dx.doi.org/10.25304/rlt.v28.2293 research in learning technology citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 11 (page number not for citation purpose) this may indicate that female students find it more important to plan, schedule educational tasks, event and deadlines, coordinate, stay informed and take overall responsibility for their education. conversely, there was a significant difference between female and male students regarding the teacher’s social presence. male students would, to a higher degree than female students (significant at the level of p < 0.01), experience that teachers were socially present inside online applications or through the use of technologies. noteworthily, while there were no significant differences between the female and male students’ behavioural disengagement there were a few (significant at the level of p  < 0.01) differences between female and male students cognitive and emotional disengagement. for example, female students reported being easily distracted by notifications (i.e. from a mobile phone), experienced that they were emotionally drawn to their digital technologies, and resisted using a keyboard for all in-class writing. interestingly, the social dimension of disengagement revealed the most differences, with three out of five variables displaying significant differences ( p < 0.01–0.05) between female and male students. male students reported, to a greater extent than female students, that their engagement decreased when they were left to work alone. female students, on the other hand, reported being disengaged when they were left to manage the digital technologies for learning by themselves and also felt upset and dispirited when group work did not succeed in including all students in the group. conclusively, in, not all, but many of the pro-learning engagement variables, female students, displayed significantly higher levels than male students. towards this background we conclude, that female students do not seem to be impaired by the increasing digitalisation in education, and that while cognitive, social and emotional disengagement differed between female and male students, efforts directed to avoid or redeem student behavioural disengagement when learning with technologies must be directed to all students regardless of gender. we also conclude that the hypothesis was verified: that engagement and disengagement differ between female and male students when learning with technologies. hypothesis 3 (h3) non-native speakers, engagement and disengagement we hypothesise that students’ academic engagement and disengagement, when learning with technologies, differ between native and non-native speakers, and performed an additional independent sample t-test to test the hypothesis. table 5 reflects statistically significant differences between native and non-native speakers in terms of their engagement and disengagement when learning with technologies. interestingly, we found no significant differences with regard to native and non-native speakers’ engagement in tel. instead, we have identified results revealing that non-native speakers to a lesser extent than native speakers reported feeling emotionally attracted to their digital technologies and using digital technologies to escape feelings of boredom in class ( p < 0.01–0.07). we have seen that some students engage in unauthorised use of digital technologies to a certain extent. some of these students continued this behaviour even when it impacted their learning negatively (becoming late with school assignments). approaching native and non-native speakers, we found that non-native speakers were significantly under-represented in these aspects of disengagement ( p < 0.01–0.03). we conclude that the hypothesis is partly supported as there were no significant differences in terms of native and non-native speakers’ engagement http://dx.doi.org/10.25304/rlt.v28.2293 n. bergdahl and j. nouri 12 citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 (page number not for citation purpose) in tel, but non-native speakers reported significantly fewer (behavioural, emotional and cognitive) disengagement inclinations than native speakers. discussion as digital technologies change the conditions for engagement (and disengagement), we approached three social groups of students and explored their engagement and disengagement in tel. we identified variables that reflect emotional, cognitive, behavioural and social engagement and disengagement in tel specific for highand low-frequency gaming students, female and male students and native and non-native speakers. approaching highand low-frequency gaming students, we identified certain differences reflecting engagement and disengagement in tel. in line with the findings of (yurov et al. 2014) which suggest that high-frequency gaming students develop some abilities that may be useful for learning, our findings reveal that gamers did not disengage due to information overflow or notifications to the same extent as low-frequency gaming students, and did not resist using digital technologies for educational purposes. on the other hand, high-frequency gaming students did not display essential pro-learning behaviours. more studies are needed to discern longitudinal effects between gaming, engagement and disengagement in tel. however, our results reveal that within the groups, the largest differences were found between female and male students. in line with skinner, kindermann and furrer (2009), we found that female students display higher levels of behavioural and emotional engagement than male students, but also higher levels of cognitive and social engagement. we found that male students’ engagement in tel decreased when working alone; that they did not rely on technologies to manage their education. together, these findings (not utilising the digital resources, isolating and decreased engagement) might indicate a laissez faire attitude towards learning, and thus, directed instruction could be helpful for students displaying similar responses when displayed conjunction with decreasing performance. the results also revealed that female students display a significantly higher degree of behavioural, cognitive, emotional and social engagement in tel than male students (but there were no differences in male and female students behavioural and cognitive disengagement). exploring the cognitive, emotional and social disengagement dimensions, there were differences between female and male students – but not enough to conclude that female or male students disengage more than the other, rather, examining specific components of engagement and disengagement, revealed male and female students disengage differently: for example, our findings reveal that male students experienced that the teachers’ were socially present when female students did not, table 5. differences between native and non-native speakers’ engagement and disengagement in tel. variables non-native speaker native speaker n m sd n m sd t p dbeh1 54 2.26 1.35 356 2.85 1.39 -2.94 0.03 dcog1 2.31 1.33 2.96 1.26 -3.35 0.01 demo1 2.81 1.43 3.39 1.47 -2.70 0.07 demo2 2.69 1.72 3.64 1.55 -4.14 <0.01 http://dx.doi.org/10.25304/rlt.v28.2293 research in learning technology citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 13 (page number not for citation purpose) which expands findings of bergdahl et al. (2018) and bergdahl et al. (2019) who alerted that teachers social presence was as important online as in an analogues setting and also would predict student performance (bergdahl et al. 2019a). it also expands fredricks et al.’s (2018) recognition that female students are more likely than male students to find teacher–student relationships critical for their engagement. this finding suggests that the female students need to interact more with the teachers to feel satisfied, which can be due to having a higher need and preference for relational support, or not receiving their teachers’ attention to the same extent as male students. interestingly, we did not identify any difference between non-native and native students’ engagement in tel. instead, we noted that native speakers displayed more disengaging tendencies than non-native speakers, in terms of being emotionally attracted to technologies, using them to escape feelings of boredom in class, even when there were negative consequences on learning. implications for learning designs and research approaches surveying students’ engagement and disengagement from different angles allow for nuanced findings relevant to specific social groups. these findings can be used to shape more effective designs of learning in order to meet the needs and preferences of different students and help identify early signals of student disengagement. for example, we found that non-native speakers and female students, who might be suspected to disengage due to lack of digital skills or interest, did not display less engagement. surprisingly, our results indicate that high-frequency gaming students might develop specific pro-learning behaviours, which suggest that gaming in the out-of-school context may contribute to some pro-learning reactions and preferences. at the same time, however, our results show that high-frequency gaming students were less prone to elicit certain pro-learning behaviours, for example, to search the internet for facts. thus, we argue that it is important that both engagement and disengagement are studied together if we are to understand students’ engagement and disengagement when learning with digital technologies. limitations with asking students if they regard themselves as native or non-native speakers, we aimed to capture potential language barriers and inherent cultural orientation that students may have. the study is limited as we cannot say anything about students’ individual cultural identification, socio-economic status or parents’ educational levels. since socio-economic status can majorly impact technological access, it is possible that the study did not accurately parallel the first wave of the digital divide. yet, we believe that the findings reflect (dis)engagement in tel that future research might want to explore further. conclusion we investigated different social groups’ engagement and disengagement when learning with digital technologies. we found that female students often display higher levels of engagement, but that there were no differences between female and male students’ http://dx.doi.org/10.25304/rlt.v28.2293 n. bergdahl and j. nouri 14 citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 (page number not for citation purpose) behavioural disengagement. non-native speakers refrained from unauthorised use of digital technologies to a significantly greater extent than native speakers, and while high-frequency gaming students did not use the internet to research what others have written, as much as students reporting low-frequency gaming, we also noted that high-frequency gaming students did not report being as easily distracted by notifications nor by information overflow as much as the low frequency of gaming students. future work qualitative approaches can complement and expand our findings, to contribute further to the understanding of underlying factors that drive engagement and disengagement in tel. it would also be of interest to approach the differences of engagement and disengagement between programmes that qualify students for higher education and vocational programmes. references alrashidi, o., phan, h. p. & ngu, b. h. (2016) ‘academic engagement: an overview of its definitions, dimensions, and major conceptualisations’, international education studies, vol. 9, no. 12, p. 41. doi: 10.5539/ies.v9n12p41 balwant, p. t. (2018) ‘the meaning of student engagement and disengagement in the classroom context: lessons from organisational behaviour’, journal of further and higher education, vol. 42, no. 3, pp. 389–401. doi: 10.1080/0309877x.2017.1281887 bergdahl, n., et al., (2018) ‘the use of learning technologies and student engagement in learning activities’, nordic journal of digital literacy, vol. 13, no. 2, pp. 113–130. doi: 10.18261/ issn.18919-943x-2018-02-04 bergdahl, n., et al., (2019a) ‘engagement, disengagement and performance when learning with technologies in upper secondary school’, computers & education, vol. 149, p. 103783. doi: 10.1016/j.compedu.2019.103783 bergdahl, n., et al., (2019b) ‘disengagement, engagement and digital skills in technology enhanced learning’, education and information technologies, vol. 25, no. 2, pp. 957–983. doi: 10.1007/s10639-019-09998-w berman, e. a. (2017) ‘an exploratory sequential mixed methods approach to understanding researchers’ data management practices at uvm: integrated findings to develop research data services’, journal of escience librarianship, vol. 6, no. 1, p. 1104. doi: 10.7191/jeslib.2017.1104 bingham, g. & okagaki, l. (2012) ‘ethnicity and student engagement’, in handbook of student engagement research on student engagement, eds s. l. christenson, a. l. reschly & c. wylie, springer, boston, ma. boekaerts, m. (2016) ‘engagement as an inherent aspect of the learning process’, learning and instruction, vol. 43, pp. 76–83. doi: 10.1016/j.learninstruc.2016.02.001 bryman, a. (2016) social research methods, 5th edn, oxford university press, oxford. doi: 10.1017/cbo9781107415324.004 chatman, j. a., et al., (2008) being distinctive versus being conspicuous: the effects of numeric status and sex-stereotyped tasks on individual performance in groups. organizational behavior and human decision processes, vol. 107, no. 2, pp. 141–160. doi: 10.1016/j. obhdp.2008.02.006 chipchase, l., et al., (2017) ‘conceptualising and measuring student disengagement in higher education: a synthesis of the literature’, international journal of higher education, vol. 6, no. 2, p. 31. doi: 10.5430/ijhe.v6n2p31 clarke, v. & braun, v. (2016) ‘thematic analysis’, in analysing qualitative data in psychology, eds e. lyons & a. coyle, 2nd edn, sage, pp. 84–103. http://dx.doi.org/10.25304/rlt.v28.2293 https://dx.doi.org/10.5539/ies.v9n12p41 https://dx.doi.org/10.1080/0309877x.2017.1281887 https://doi.org/10.18261/issn.18919-943x-2018-02-04 https://doi.org/10.18261/issn.18919-943x-2018-02-04 https://dx.doi.org/10.1016/j.compedu.2019.103783 https://dx.doi.org/10.1007/s10639-019-09998-w https://dx.doi.org/10.7191/jeslib.2017.1104 https://dx.doi.org/10.1016/j.learninstruc.2016.02.001 https://dx.doi.org/10.1017/cbo9781107415324.004 https://dx.doi.org/10.1016/j.obhdp.2008.02.006 https://dx.doi.org/10.1016/j.obhdp.2008.02.006 https://dx.doi.org/10.5430/ijhe.v6n2p31 research in learning technology citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 15 (page number not for citation purpose) ferguson, c. j. (2015) ‘do angry birds make for angry children? a meta-analysis of video game influences on children’s and adolescents’ aggression, mental health, prosocial behavior, and academic performance’, perspectives on psychological science, vol. 10, no. 5, pp. 646–666. doi: 10.1177/1745691615592234 field, a. (2018) discovering statistics using ibm spss statistics, 5th edn, sage, london. doi: 10.1016/b978-012691360-6/50012-4 finn, j. d. (1989) withdrawing from school. review of educational research, vol. 59, no. 2, pp. 117–142. doi: 10.3102/00346543059002117 fredricks, j., blumenfeld, p. & paris, a. (2004) ‘school engagement: potential of the concept, state of the evidence’, review of educational research spring, vol. 74, no. 1, pp. 59–109. doi: 10.3102/ 00346543074001059 fredricks, j. & mccolskey, w. (2012) ‘the measurement of student engagement: a comparative analysis of various methods and student self-report instruments’, in handbook of research on student engagement, eds s. l. christenson, a. l. reschly & c. wylie, springer us, boston, ma, pp. 763–782. doi: 10.1007/978-1-4614-2018-7_37 fredricks, j. a., et al., (2018) ‘supporting girls’ and boys’ engagement in math and science learning: a mixed methods study’, journal of research in science teaching, vol. 55, no. 2, pp. 271–298. doi: 10.1002/tea.21419 gebhardt, e., et al., (2014) preparing for life in a digital age. the iea international computer and information literacy study international report, springer nature, cham. doi: 10.1007/978-3-319-14222-7 gikas, j. & grant, m. m. (2013) ‘mobile computing devices in higher education: student perspectives on learning with cellphones, smartphones &amp; social media’, the internet and higher education, vol. 19, pp. 18–26. doi: 10.1016/j.iheduc.2013.06.002 gómez-fernández, n. & mediavilla, m. (2019) what are the factors that influence the use of ict in the classroom by teachers? evidence from a census survey in madrid public policies (no. 08), the barcelona institute of economics, barcelona, spain. graham, c. r. (2006). blended learning systems: definition, current trends, and future directions. in b. miller (ed.), the handbook of blended learning, pp. 3–21. pfeiffer. halverson, l. r. (2016) conceptualizing blended learning engagement, doctoral dissertation, brigham young university, usa. henderson, m., selwyn, n. & aston, r. (2017) ‘what works and why? student perceptions of “useful” digital technology in university teaching and learning’, studies in higher education, vol. 42, no. 8, pp. 1567–1579. doi: 10.1080/03075079.2015.1007946 henrie, c. r., et al., (2018) ‘exploring the potential of lms log data as a proxy measure of student engagement’, journal of computing in higher education, vol. 30, no. 2, pp. 344–362. doi: 10.1007/s12528-017-9161-1 henrie, c. r., halverson, l. r. & graham, c. r. (2015) ‘measuring student engagement in technology-mediated learning: a review’, computers & education, vol. 90, pp. 36–53. doi: 10.1016/j.compedu.2015.09.005 hermerén, g. (2011) good research practice, swedish research council, stockholm, sweden. hietajärvi, l., et al., (2019) ‘beyond screen time: multidimensionality of socio-digital participation and relations to academic well-being in three educational phases’, computers in human behavior, vol. 93, pp. 13–24. doi: 10.1016/j.chb.2018.11.049 hietajärvi, l., et al., (2015) ‘is student motivation related to socio-digital participation? a person-oriented approach’, procedia – social and behavioral sciences, vol. 171, pp. 1156–1167. doi: 10.1016/j.sbspro.2015.01.226 howard, s. k., ma, j. & yang, j. (2016) ‘student rules: exploring patterns of students’ computer-efficacy and engagement with digital technologies in learning’, computers and education, 101, 29–42. doi: 10.1016/j.compedu.2016.05.008 ilomäki, l. (2011) does gender have a role in ict among finnish teachers and students? scandinavian journal of educational research, vol. 55, no. 3, pp. 325–340. doi: 10.1080/00313831.2011.576910 http://dx.doi.org/10.25304/rlt.v28.2293 https://dx.doi.org/10.1177/1745691615592234 https://dx.doi.org/10.1016/b978-012691360-6/50012-4 https://dx.doi.org/10.3102/00346543059002117 https://dx.doi.org/10.3102/ 00346543074001059 https://dx.doi.org/10.1007/978-1-4614-2018-7_37 https://dx.doi.org/10.1002/tea.21419 https://dx.doi.org/10.1007/978-3-319-14222-7 https://dx.doi.org/10.1016/j.iheduc.2013.06.002 https://dx.doi.org/10.1080/03075079.2015.1007946 https://dx.doi.org/10.1007/s12528-017-9161-1 https://dx.doi.org/10.1016/j.compedu.2015.09.005 https://dx.doi.org/10.1016/j.chb.2018.11.049 https://dx.doi.org/10.1016/j.sbspro.2015.01.226 https://dx.doi.org/10.1016/j.compedu.2016.05.008 https://dx.doi.org/10.1080/00313831.2011.576910 n. bergdahl and j. nouri 16 citation: research in learning technology 2020, 28: 2293 http://dx.doi.org/10.25304/rlt.v28.2293 (page number not for citation purpose) ma, j., cheng, j. & han, x. (2018) ‘initial development process of a student engagement scale in blended learning environment’, proceedings – 6th international conference of educational innovation through technology, eitt 2017, 2018-march, pp. 234–237. doi: 10.1109/eitt.2017.63 mann, s. j. (2001) ‘alternative perspectives on the student experience: alienation and engagement’, studies in higher education. vol. 26, no. 1, pp. 7–19. doi: 10.1080/03075070020030689 pellerin, l. a. (2005). ‘student disengagement and the socialization styles of high schools’, social forces, 84, 1161–1179. rashid, t. & asghar, h. m. (2016) ‘technology use, self-directed learning, student engagement and academic performance: examining the interrelations’, computers in human behavior, vol. 63, pp. 604–612. doi: 10.1016/j.chb.2016.05.084 scherer, r., rohatgi, a. & hatlevik, o. e. (2017) ‘students’ profiles of ict use: identification, determinants, and relations to achievement in a computer and information literacy test’, computers in human behavior, vol. 70, pp. 486–499. doi: 10.1016/j.chb.2017.01.034 siddiq, f. & scherer, r. (2019) ‘is there a gender gap? a meta-analysis of the gender differences in students’ ict literacy’, educational research review, vol. 27, pp. 205–217. doi: 10.1016/j. edurev.2019.03.007 skinner, e. a., kindermann, t. a. & furrer, c. j. (2009) ‘a motivational perspective on engagement and disaffection’, educational and psychological measurement, vol. 69, no. 3, pp. 493–525. doi: 10.1177/0013164408323233 skryabin, m., et al., (2015) ‘how the ict development level and usage influence student achievement in reading, mathematics, and science’, computers & education, vol. 85, pp. 49–58. doi: 10.1016/j.compedu.2015.02.004 strittmatter, e., et al., (2015) ‘pathological internet use among adolescents: comparing gamers and non-gamers’, psychiatry research, vol. 228, no. 1, pp. 128–135. doi: 10.1016/j. psychres.2015.04.029 tallvid, m., et al., (2015) ‘exploring the relationship between sanctioned and unsanctioned laptop use in a 1:1 classroom’, educational technology and society, vol. 18, no. 1, pp. 237–249. van deursen, a. j. a. m. & helsper, e. j. (2015) ‘the third-level digital divide: who benefits most from being online?’ in communication and information technologies annual, eds r. laura, s. r. cotten, j. schulz, t. m. hale & a. williams, emerald group publishing limited, bingley, uk. vol 10. pp. 29–52. volman, m., et al., (2005) ’new technologies, new differences. gender and ethnic differences in pupils’ use of ict in primary and secondary education’, computers and education, vol. 45, no. 1, pp. 35–55. doi: 10.1016/j.compedu.2004.03.001 wang, m.-t. & hofkens, t. l. (2019) ‘beyond classroom academics: a school-wide and multi-contextual perspective on student engagement in school’, adolescent research review, vol. 0, no. 0, p. 0. doi: 10.1007/s40894-019-00115-z wang, m.-t. et al., (2017) ‘conceptualization and assessment of adolescents’ engagement and disengagement in school’,   european journal of psychological assessment, vol. 35, no. 4, pp. 592–606. doi: 10.1027/1015-5759/a000431 wang, m. t. & fredricks, j. a. (2014) ‘the reciprocal links between school engagement, youth problem behaviors, and school dropout during adolescence’, child development, vol. 85, no. 2, pp. 722–737. doi: 10.1111/cdev.12138 wiggins, b. l., et al., (2017) ‘aspect: a survey to assess student perspective of engagement in an active-learning classroom’, cbe life sciences education, vol. 16, no. 2, pp. 1–13. doi: 10.1187/cbe.16-08-0244 wylies, c. & hodgen, e. (2012) ‘trajectories and patterns of student engagement: evidence from a longitudinal study’, in handbook of research on student engagement, eds s. l. christenson, a. l. reschly & c. wylie, springer, boston, ma, pp. 585–599. yurov, k. m., et al., (2014) ‘the effect of psychological and environmental factors on academic performance of video gamers’, issues in information systems, vol. 15, no. 2, pp. 393–398. http://dx.doi.org/10.25304/rlt.v28.2293 https://dx.doi.org/10.1109/eitt.2017.63 https://dx.doi.org/10.1080/03075070020030689 https://dx.doi.org/10.1016/j.chb.2016.05.084 https://dx.doi.org/10.1016/j.chb.2017.01.034 https://dx.doi.org/10.1016/j.edurev.2019.03.007 https://dx.doi.org/10.1016/j.edurev.2019.03.007 https://dx.doi.org/10.1177/0013164408323233 https://dx.doi.org/10.1016/j.compedu.2015.02.004 https://dx.doi.org/10.1016/j.psychres.2015.04.029 https://dx.doi.org/10.1016/j.psychres.2015.04.029 https://dx.doi.org/10.1016/j.compedu.2004.03.001 https://dx.doi.org/10.1007/s40894-019-00115-z https://dx.doi.org/10.1027/1015-5759/a000431 https://dx.doi.org/10.1111/cdev.12138 https://dx.doi.org/10.1187/cbe.16-08-0244 altj volume 3 number i catering for different learning styles manfred kwok and clifford jones european bus/ness management school, university of wales, swansea this paper describes a study to investigate the possibility of applying a quick and simple front-end to multimedia presentations. the front-end will determine the user's learning style and suggest a suitable navigation method the paper recognizes not only different learning styles which influence learners' performance, but also the practical use of valid learning styles' measurements. introduction the use of computers for teaching and training has long been a controversial issue. up to the 1980s, many educators who were keen to use computers to pass on knowledge developed various computer-based teaching and training materials, but most of these materials were capable only of delivering information in text with perhaps some computer graphics, and in many cases it proved difficult to raise learners' interest in the subject matters (patrick & fitzgibbon, 1988). in the 1990s, there has been cheerful news for procomputer educators; the latest multimedia technology could be the solution to the problem. using a multimedia personal computer we can retrieve information in diverse formats: text, high-quality audio, still and animated graphics, and still and moving pictures. however, to understand how to teach people effectively with computers, it is first necessary to find out what people want and what they expect from computer teaching and training materials. it would be foolhardy to produce a computer teaching/training program and expect all users to learn things equally well. in fact, much past research has shown that people have different learning styles, and one's learning style is built up by many different past experiences (entwistle, 1987). computer experience, and how the user feels before and during looking at computerized material, may also affect learning style, and hence interactions with the computer. ideally, every computerized teaching/training program should have many versions in the s manfred kwok and cliffordpnes catering for different learning styles presentation of information, covering all users' learning styles, computer experience, mood, etc. for example, one version may be a straightforward presentation, another version may be fully interactive. the problem of catering for different learning styles learning styles if we all accept the level of interactivity as being a key factor which affects how well and how much learners learn from programmed materials, the immediate problems are to identify what factors may affect learners' study styles, and to find out how to provide optimal computer interaction. the notion of cognitive styles gained recognition in the 1950s and 1960s as a significant personal characteristic which raised the understanding of how people learn. cognitive style has been variously defined (kogan, 1971; messick, 1976). however the authors here would define the term as the unique and preferred way in which an individual processes, stores and retrieves information. study strategies are simply cognitive styles applied when individuals go about learning. messick defined as many as nine distinctive dimensions (messick, 1970). witkin's work on two different cognitive styles, called field-dependent and field-independent, showed the differences in the way an individual structures and analyses information (witkin, 1976). pask and scott (1972) and pask (1976) distinguished two distinctive styles of learning, identified as holist and serialist strategies. marton & säljö (1976) identified two study approaches: deep approach and surface approach. kolb (1984) defined four learning styles: accommodation, assimilation, divergence, and convergence. all this produces a confusing picture of different learning styles, strategies and approaches. clarke (1993) (together with the authors of this paper) suggests bravely that these styles 'differ more in name than nature' and that they 'can be classified into either a preference for a reasonable degree of structure and guidance [serial] ... or a preference for considerable freedom to explore [holistic]'. learning styles measurement one problem is to find out users' learning styles. some learning-style test procedures, inventories and questionnaires have been suggested for determining learning styles effectively and sufficiently, but these tests and questionnaires are mostly very timeconsuming to complete and/or they may require instructors to assess their outcomes (pask, 1973; kolb, 1984; weinstein et al, 1983). it would be highly impractical, perhaps somewhat cruel, to ask users first to spend considerable time completing those tests or questionnaires before using any computerized programs. what we want is a quick and easy, reliable and valid measure of style which occurs as an immediate prerequisite to involvement in the program, such that users can determine their own learning strategy and preferred navigation method within the system. a well-designed multimedia system, together with opportunities for multiple navigational strategies provided by a range of navigational methods, will encourage users to adopt a 'best method' to suit their individual study strategy. we will call such an approach to programs and presentations: front-end applied programs orfront-end applied presentations. 6 alt-j volume 3 number i experimental study with front-end applied multimedia presentations computerized front-end the front-end of multimedia presentations contains 13 items suggested by clarke (1993) taken from ford's original 18-item study preference questionnaire (spq) (ford, 1985). this gave us an opportunity to test the effectiveness of clarke's refinement made to the spq. subjects rated each item on a likert scale from 1 (strongly agree with the holistic statement) to 5 (strongly agree with the serial statement). figure 1 shows the first screen of the front-end. the program then sums the scores for all questions. the maximum possible front-end score is 65; this would be obtained only if someone strongly agreed with all thirteen serial statements. initially, the boundary between the two learning styles was set at 39.5, the mid-point between highest and lowest possible scores. the front-end also asks for some more details of the users (name, age and sex) as well as their past computer experience and how they feel prior to operating certain presentations. surriaitië: ebms user details • front end application first initial: how do you feel at this moment: nervous< o o g o o >ve»yrelaxed :;çpmpaterexpenence;;::;|;;;::;.;..::;;:̂ ::;..::v sptèfcse'anwerthofouowiflgj^ the appropriate answer according to your own preference.: p'ress'the ; ;• ' 'next question' button to see the next question. ( 1 3 questions in total ) 1 . when i'm reading a book (of olhei. inloimation souice) loi my studies,} generally tend to ; ; ; : concentiate on certain pails, and skip over otheis quite maikedly. going back later if necessaiy la lili in any 'gaps m missing link»' '.< strongly agree > when i'm leading a book (or othsi infomation souicc) tot tny ; «tudies,. i tend to follow the ;;;;;;;;, a u t h o r ' s p t e « e n t a t i o n :••'•.:'••••••. reatonablyr clotefy, rathei than skipping about a kitnext question figure i: the front-end sample • fifty-eight first-year business studies undergraduates were involved in the study. within the overall sample, thirty-four were female and twenty-four were male, reflecting the proportion of sexes in the business school. all were volunteers and had all received at least two hours of training on operating personal computers and microsoft's windows. 7 manfred kwok and cliffordpnes catering for different teaming styles materials the materials used in the study were taken from the multimedia information presentations called the european business management school visual prospectus (ebmsvp) which was developed by one of the authors (mk) as a by-product of this research. the system includes a wide range of information about the european business management school and general information about swansea and the university. it has three operation modes: self-running, menu driven, and fully interactive. as its name suggests, the selfrunning mode will run without interaction from users, but they may interrupt the system by pressing the fast forward paging button. the menu-driven mode allows users to choose what they want to see from the menus and then to page through the topic forward or backward by using appropriate buttons on the screen. the fully interactive mode increases the freedom of travelling around the system, and allows users to jump from menu to menu, and from one topic to another. they can also make use of the index facility quickly to look up any particular topic within the system. such an index facility does not exist in the other two modes. however, considering the testing time and the number of students who could be asked to take part in the project, the content of the testing materials has been made general. in the actual study, only two versions, menu driven and fully interactive, were used. results of the 58 students, 30 students (51.7%) were classified as having a holistic learning style, and 28 students (48.3%) as having a serial learning style. they were assigned randomly to either the fully interactive version or the menu-driven version; 31 students (53.4%) used the version which matched their learning styles, and 27 students (46.6%) did not. we applied a kolmogorov-smirnov goodness of fit test to test the normality of both sets of data. the significant figures were p=0.7717 and p=0.3201, hence we had no evidence to say that both sets of data were not normally distributed. the mean performance scores for matched and mismatched groups were 11.58 (sd=3.69) and 10.22 (sd=3.62) respectively. our initial assumption was that if we were using the improved version of the original 18-item spq and we also used the correct boundary (i.e. 39.5) to divide students into ywo groups based on learning styles, we expected to see the mean score of the matched cases significantly greater than the mismatched cases. t-test for two independent samples was applied to compare their mean performances. unfortunately, the test result was insignificant (at the 5% level), the one-tailed significance figure was p=0.082. in other words, our 13-item front-end failed to detect and provide a suitable presentation for users who have different learning styles. perhaps 39.5 was not the correct boundary. we reclassified all 58 cases with the mean front-end score 37.31, 29 of each were in the matching and mismatching categories. by comparing the mean scores of the two categories, we found a significant answer (p=0.047). we also attempted to apply a single item item 7 of the original spq to define subjects' learning strategies as originally suggested by clarke (1993). this gave 19 matched cases and 34 mismatched cases, and their mean performance scores were 10.63 (sd=4.06) and 10.79 (sd=3.46) respectively. (note: the mismatched group has a slightly higher mean score). five cases 8 alt-j volume 3 number i were ignored, since those subjects expressed no clear preference for either statement of the item 7. a follow-up questionnaire was used to collect users' opinions on the front-end applied multimedia presentation, and answers were included in an exploratory factor analysis to assess the dimensionality of the learning style determination. this resulted in five factors, and crossloading situations also existed. after a reliability analysis, six items were removed. the remaining seven items proved to have a better degree of internal consistency, with reliability coefficient alpha value reaching 0.7086 from the initial alpha value 0.5799 (it has been suggested that a minimum alpha value of 0.90 should be achieved to have a reliable measure nunnally, 1967). we then carried out the analysis by assessing our subjects only with the seven items of the front-end. disappointing results were again found. although the average performance of the matching group (11.61) was higher than the average performance of the mismatching group (10.08), there was no evidence that this difference was significant statistically (p=0,0595, p>0.05). there was one possible explanation for this failure to find significant differences on the students' study performances despite using the more reliable seven items of the front-end only. we might have a substantial number of students who were capable of performing equally well with different levels of interactivity of computer presentation. these were described by pask (1976) as versatile learners, and their front-end scores would have dominated the central area of the distribution. therefore, we excluded students from around the mean score of the distribution with the hope that these seven items of the front-end could efficiently detect extreme cases. nine different factors were used to assess signifiant differences between matched and mismatched cases (see table 1). table 1: meanings variable name perform anxiety1 easybut easyop lost teacher textsys overall time ofvariables meaning study performance anxiety during the presentation how easy to understand those on screen buttons how easy to operate the presentation sense of position of the presentation how do they like multimedia presentations compared with conventional lectures how do they like multimedia presentations compared with text-based computer presentations overall opinion on multimedia presentations time spent on the presentation at this stage, close to half of our overall sample, 48.28%, were excluded from the centre of the front-end distribution. a significant difference between two groups' mean performance was found, and there were other significant results (see table 2). 9 manfred kwok and clifford fines catering for different learning styles table 2: comparison of different variables with different cut-off points variable name perform anxiety1 easybut easyop lost overall time teacher textsys 20.793 0.0595 0.2105 0.1500 0.0050 0.0145 0.1160 0.0285 0.1350 0.3040 cut off point also exclude 20,21 0.1240 0.2100 0.2645 0.0085 0.0605 0.1330 0.0550 0.2045 0.4595 also exclude 19,22 0.0760 0.2440 0.4555 0.0135 0.0500 0.0570 0.0685 0.1400 0.1685 one-tailed significance: parametric (non-parametric) significant figures are italicized also exclude 18,23 0.0465 (0.0371) 0.3700 (0.4267) 0.4760 (0.5000) 0.0420 (0,0389) 0.0200 (0.0329) 0.0110 (0.0196) 0.1700 (0.1897) 0.1565 (0.1285) 0.0685 (0.0369) conclusions although we have found significant study performance differences between learning-style matching and mismatching groups using the seven items of the study preference questionnaire, the result was strictly based on the 13-item study preference questionnaire that clarke suggested (clarke, 1993). there were disagreements between both studies. work should therefore be done once again on ford's original 18-item study preference questionnaire covering more variety of subject matters. there were two possible factors which could affect our test results: • the correct use of cut-off point of the front-end scores; • the validity of the 13-item front-end. however, the results suggest that the learners who really need guidance in selecting an appropriate navigation method in order to raise their interest in the material, i.e. those at the extremes of learning styles, can be helped. references clarke, j. (1993), 'cognitive style and computer-assisted learning: problems and a possible solution', association for learning technology journal, 1, 1, 47-59. entwistle, n . (1987), understanding classroom learning, london, hodder and stoughton. ford, n . (1985), 'learning styles and strategies of postgraduate students', british journal of educational technology, 16, 1, 65-77. 10 alt-j volume 3 number i kogan, n. (1971), 'educational implications of cognitive styles', in lesser, g.s. (ed), psychology and educational practice, illinois, scott foresman, 242-92. kolb, d.a. (1984), experiential learning: experience as the source of learning and development, englewood cliffs nj, prentice-hall. marton, f. & säljö, r. (1976), 'on qualitative differences in learning: 1. outcome and process', british journal of educational psychology, 46, 4-11. messick, s. (1970), "the criterion problems in the evaluation of instruction: assessing possible, not just intended outcomes', in wittrock, m.c. & wiley, d (eds), the evaluation of instruction: issues and problems, new york, holt, rinehart & winston. messick, s. (1976), personality consistency in cognition and creativity, individuality in learning, san francisco, jossey-bass. nunnally, j.c. (1967), psychometric theory, new york, mcgraw-hill. pask, g. & scott, b.c.e. (1972), 'learning strategies and individual competence', international journal of man-machine studies, 4, 217-53. pask, g. (1973), educational methods using information about individual styles and strategies of learning, final report of the ssrc project hr 1424/1 (2 volumes). pask, g. (1976), 'style and strategies of learning', british journal of educational psychology, 46, 128-48. patrick, j. & fitzgibbon, l. (1988), 'structural displays as learning aids', international journal of man-machine studies, 28, 625-35. weinstein, c.e., schulte, a.c. & cascallar (1983), the learning and strategies inventory (lassi), initial design and development, technical report, us army research institute for the social and behavioral science, alexandria, virginia. witkin, h.a. (1976), 'cognitive style in academic performance and in teacher-student relations', in messick, s. et al. (eds), individuality in learning. san francisco, jossey-bass. n the influence of audio communications technology on computer-supported collaborative learning denise whitelock, eileen scanlon and josie taylor institute of educational technology, the open university our study set out to investigate how collaborative learning between pairs of adult subjects solving physics problems with a computer is effected when the subjects are not co-present but have an audio and computer link only. it also examined how the groups' gender composition could influence both group productivity and individual learning. this paper reports that all pairs performed significantly better in the audio condition with respect to on-task problem-solving measures, and pre-to post-test change scores. introduction in recent years there has been an appreciation of the benefits that can be obtained by students working in groups or teams using computers (eraut and hoyles, 1988). but there is a difference in opinion as to how a partner enhances learning, and in particular how well adult learners do in the collaborative setting. in order to capitalize on the opportunities offered by new technologies, we need to understand more fully how the process of collaboration is effected by different communication technologies, and how the technologies themselves might be used to best advantage for the benefit of distance learners. however, another factor, apart from the technology itself, which is thought to influence computer-supported collaborative learning is in the gender distribution of the group. a number of classroom studies have shown gender differences when children work together with computers, and these have been reported from a number of subject-domains including science (scanlon et al, 1993; littleton et al, 1992). since our own expertise is in the area of science learning, we selected a non-trivial physics task for the subjects to work with, and that is in the area of elastic collisions. previous studies (villani and pacca, 1990; whitelock et al, 1993) have shown that both adults and school-children have difficulty in predicting the subsequent motion of balls or ice pucks after they collide. villani's study stresses that even postgraduate students tend to revert to their informal commonsense notions unless they are cued to use formal representations of these types of problem. these studies have demonstrated that the topic of elastic collisions is a complex yet fruitful one in which to engage students in group work. 77 denise wh/te/ock et 0/ the influence of audio communications technology on computer-supported learning the piece of research reported in this paper set out to investigate two important questions with respect to communication technologies and adult learners. these were: 1. do pairs of students perform better when they work side by side, with a physics simulation, than when their only contact with one another is via a telephone and their respective computers? 2. does group composition with respect to gender affect task performance in the above two conditions? with these questions in mind, our objectives included monitoring both group productivity and individual learning in both the audio and side-by-side conditions , nd investigating whether the influence of gender had any identifiable consequences on boththese outcomes. method the computer simulation the puckland simulation version 2.6 was written in hypercard for use with the apple macintosh. it uses a direct manipulation approach which allows students to investigate a series of collisions between two ice pucks. it consists of a pair of pinball-style flippers on either side of the screen with which subjects can flick pucks. the amount of force with which the flippers hit the pucks can be varied, as can the mass of the pucks. the pucks, ranging from 1 to 100 units of mass, can be dragged into position ready to be struck by the flippers. the initial velocity of the collision is controlled by directly manipulating the angle of the flipper from 90 to 180 degrees in the vertical plane. before the go button could be activated, subjects have to qualitatively predict the speed and direction of the pucks after collision. when the go button is pressed, the pucks move towards each other on the screen, and are animated with a speed proportional to that set by the initial angle of impact executed by the flipper. the program also highlights the icons with the correct answers after the experiment is run. after the pucks collide, they move away from each other with a speed which is calculated from the correct physics formalisms. this means that the principles of conservation of momentum and kinetic energy are obeyed, and again the apparent screen velocities of the pucks is proportional to their calculated values. at the bottom of the screen is a grid which provides numerical information about the amount of energy and momentum that the system had initially, and then, subsequent to being run, it shows what the effect of the collision is on these two factors. every experiment attempted by users of the simulation is automatically logged by the computer. the networked version of puckland has a communication protocol written in hypertalk 2 which allows two to eight machines to be simultaneously linked together. with this system you cannot continuously see your partner's screen or take control of it, as with other communication protocols such as timbuktu. although screens can be exchanged between the partners, this was not an automatic process, since all subjects could choose to accept or reject data from their partner. this latter feature was incorporated into the system to prevent one partner, who might have more computer or physics experience than the other, dominating the proceedings. 78 volume 4 number i using the simulation all subjects had access to their pre-test predictions about the motion of the ice pucks in a number of given scenarios, during the experimental period. they were explicitly requested to work together as a team while using the simulation, in order to aid their joint understanding of elastic collisions. they were given time at the beginning of the session to explore the simulation and to try out any scenarios that interested them. once the subjects felt they understood the domain and had exploited the simulation as an information resource, they were asked to solve three different problems. these problems required the subjects not to predict the result of a collision but to state the original conditions for a given outcome. the problems were posed in order of difficulty and were as follows: 1. what initial conditions are needed to send the pucks travelling away from each other at the same speed? 2. what initial conditions are needed to stop one puck after impact? 3. what initial conditions are needed to make pucks of unequal masses move away, after impact, at the same speed? all subjects were video-recorded throughout the experimental sessions, and the subjects' dialogue in the audio condition was tape-recorded on a telephone answering machine. the experiments which were run with the simulation were logged, and the networked version of puckland also recorded subjects requests to exchange screens. the program finally logged the screens which were accepted or rejected by each partner. subjects all subjects in this study were adults. they held a first degree or equivalent professional qualification, and 89% of them had experienced some form of physics education. the mean age of the group was 39.9 years (sd = 11.8). the subjects were paired with people they knew and were happy to collaborate with. they formed both single-sex and mixed-gender dyads. they were assigned to the side-by-side or audio condition depending on which of two test sites they could more easily reach. the numbers in each group are shown in table 1. however, some of the pairs had worked together before while others had not. the main factor in determining this relationship was whether subjects were available to participate in the experiment on the same day. we aimed for a minimum of 12 subjects in each condition, which was achieved comfortably, but the final number was determined by the availability of subjects during the experimental period. procedure the subjects were pre-tested with a version of a questionnaire developed by whitelock et al (1993). this included a prediction task where subjects were asked to predict the subsequent motion of two ice pucks after collision for the following five conditions: condition 1: puck a is small, light and moving; puck b is larger, heavier and stationary. condition 2: puck a is large, heavy and moving; puck b is small, light and stationary. condition 3: puck a and puck b are identical in size but only puck a is moving. 79 denise whitetock et 0/ the influence of audio communications technology on computer-supported learning side by side side by side side by side audio audio audio subject group male female. mixed male female mixed n 14 16 18 12 12 16 table i: numbers of subjects who worked in the audio-only or side-by-side condition when using the physics simulation condition 4: puck a is small, light and moving; puck b is larger, heavier and moving. condition 5: puck a and puck b are identical in size and both are moving. the subjects were not only asked about their understanding of kinetic energy and momentum, but also about their expectations of the computer task. in the post-test, all the questions were repeated and some extra information was obtained about the subjects' perceptions of the whole experience. results the key finding from this experiment is that all the pairs did significantly better in the audio rather than in the side-by-side condition (see figure 1). a two-way anova performed on the pre-post test change scores revealed a significant difference for the communication condition (f = 11, p<0.001), but there was no overall difference according to gender. as well as simply looking at improvement on preto post-test scores, it was possible to investigate problem-solving success while subjects were using the simulation (see figure 2). the three problems were given in order of difficulty and all the pairs were able to solve problem 1. problem 2 proved to be difficult only for the mixed-gender group in the side-byside condition, while a solution to problem 3 was more frequently obtained by all groups working in the audio condition. a two-way anova performed on the problem-solving scores revealed a significant difference for the communication condition (f = 15, p<0.001), but there was no overall difference according to gender. there was no significant difference in the amount of time spent working with the simulation for all the groups tested. therefore further analysis of the number of experiments performed was undertaken to find reasons for these group differences. as for the number of experiments performed in each communication condition for each gender group studied (see figure 3), the male pairs conducted more experiments in the side-by-side than in the audio condition. the reverse result is true for the female pairs, while subjects of mixed gender carried out exactly the same number of experiments regardless of the communication channel. a two-way anova performed on the number of experiments performed revealed no overall difference according to communication or gender condition. with respect to subjects' management and exchange of experimental information in the audio 80 alt-j volume 4 number i m.m mixed key: mm = male; ff= female; mixed = mixed gender figure i: graph to illustrate change scores versus mode of communication and gender mix m.m f.f gender group mixed key: m.m = male; f.f= female; mixed = mixed gender figure 2: graph to illustrate problem-solving scores versus mode of communication and gender mix condition, only information on 25% of the experiments attempted was transmitted to partners during the session (see figure 4). from this small number, only four screen-updates were rejected. this suggests that the exchange-screen facility was used only when subjects assumed it would be mutually beneficial. the videos were consulted to ascertain what sorts of scenario prompted partners to send screen-data to one another. two types of events prompted this 81 den/se whitelock et a\ "me influence of audio communications technology on computer-supported learning key: m.m = male; f.f= female; mixed = mixed gender figure 3: graph to illustrate mean number of experiments versus mode of communication and gender mix 350 t performed sent accepted rejected experimental interactions between the subjects key: mm = male; f.f= female; mixed = mixed gender figure 4: graph to illustrate number of experiments exchanged when subjects were working in the audio condition only 82 alt-j volume 4 number i course of action, and these occurred when a partner wanted to help a floundering colleague or when one of the subjects had a particularly interesting finding to communicate. the main difference between subjects' behaviour during the audio session on the one hand, and in the side by side condition on the other is that the 'audio' pairs did not choose to continually work through everything together. they branched off on their own at times, and explored their individual hunches or hypotheses about what they thought was going on. they then came back together again to discuss their findings. in fact, this was a strategy employed by the most successful problem-solvers. an extract from the transcript of one of the male/male pairs illustrates this particular modus operandi: 1. one way of using the resources, as it were, would be if we both played around independently without communicating back too much about it. 2. and then come back. 1. come back and compare notes. all right? 2. all right. 1. so we're still trying to create a situation where one of the pucks is stationary after impact; all right, are you happy to do that? 2. yes i am. 1. right, let's pursue a little hunt here and then maintain radio silence. this strategy certainly contributed to subjects in the audio condition solving more problems while working together online than subjects in the face-to-face condition (see figure 2). however, some of the problems were solved not only by students testing hypotheses, but by also adopting a trial-and-error strategy. this latter course of action would also increase the number of experiments performed with the simulation. conclusions the results indicate that with this particular computer-based investigation subjects working in the audio condition were more likely to improve their post-test scores and to succeed in the problem-solving exercises than subjects working side by side. we have found that it is not just a difference in the number of experiments but in the type of experiments attempted that contributed to joint problem-solving success. these results are also a reflection of how the subjects chose to work together in the different communication conditions. for example, in the audio condition, because each subject had his or her own computer, it was possible for individuals to explore their own different hypotheses when their joint deliberations proved unsuccessful. we are in the process of analysing the discourse of the pairs of subjects throughout the experimental sessions, and a preliminary pass at the data suggests that cooperative construction of shared meaning (barbieri and light, 1992) rather than conflict is more important for successful collaboration. this study did not reveal any significant difference in group problem-solving results on individual preto post-test change scores with respect to gender, but there was more improvement in the mixed-gender pairs change scores in the audio than in the side-by-side 83 den/se wh/tetock et 0/ the influence of audio communications technology on computer-supported learning condition. one possible explanation for this is that there was more male dominance when couples were working side by side. however, in the audio condition, both males and females were testing out their own predictions. our findings suggest gender does not influence learning with this particular computer task but that subjects in certain communication settings benefit more than others. and rather than being an inferior substitute to working side by side, it seems that the audio condition had some advantages. this study forms part of an ongoing research programme, and we will investigate students solving the same physics problems, working at a distance, but this time linked by a video conferencing system. references barbieri, s. and light p. (1992), 'interaction, gender and performance on a computer-based problem-solving task', learning and instruction, 12, 199-213. eraut, m. and hoyles, c. (1988), 'group work with computers', journal of computer assisted learning, 5, 12-24. littleton, k., light, p., joiner r., messer, d. and barnes p. (1992), 'pairing and gender effects in computer-based learning', european journal of psychology of education, 7 (4), 1-14. scanlon, e., o'shea, t., hennessy, s., taylor, j. and whitelock, d. (1993), 'gender, computers and group work in science', 7th conference on gender, science and technology, gasat, august 1993. whitelock, d., taylor, j., o'shea, t., scanlon, e., clark, p. and o'malley, c. (1993), 'challenging models of elastic collisions with a computer simulation', journal of computers in education, 20 (1), 1-9. villani, a. and pacca, j.l.a. (1990), 'spontaneous reasoning of graduate students', international journal of science education, 12, 589-600. 84 replacement of traditional lectures with computer-based tutorials: a case study derek lavelle department of the built environment university of northumbria at newcastle a set of computer-based tutorial files were produced in-house and used to replace two traditional lectures. students submitted feedback on their use of the cal material. the ease of production and delivery, and the positive responses from the students, have provided justification for further development of in-house cal material. further insights are given into the integration of cal into a curriculum. introduction this paper reports on a pilot project with a group of 60 second-year undergraduates studying the use of standard forms of contract in the construction industry. the project entailed the replacement of two of a series of nine scheduled lectures with a computer-based tutorial. the two main aims of the project were to test the viability of converting existing lecture material into computer-based material on an in-house production basis, and to obtain feedback from the student cohort on their behavioural response to the change in media. the effect on student performance was not measured at this stage of development. description of the courseware windows help author was used to create a set of tutorial files on the subject of payment under standard forms of contract in the construction industry. the files were created in hypertext style to allow the students to navigate through the material as they wished, although a suggested route was provided which mirrored the approach taken in a normal lecture. the computer-based material deliberately replicated that which would have been available via the two normal lectures in order not to distort the comparison between the two modes of delivery, although some of the group interactions in normal lectures were postponed until a subsequent seminar. a total of 22 windows were produced, an example of which is shown in figure 1. blue or red hot words were used to guide students to further windows or pop-ups respectively, taking account of published screen design principles (clarke, 1992). advantages to the lecturer in using help author included the negligible expense of the 99 derek love/je replacement of traditional lectures with computer-based tutorials: a case study interim payments ml 3 interim valuation eile £dit bookmark help fipntenbt p the usual monthly cycle of valuation / certificate /payment is described in clause 42 and is illustrated below. (click on any of the coloured topics or phrases for further information) interim valuation 7 days interim certificate i"days payment this cycle is repeated until practica completion occurs it is normally the qs -who ascertains the amount of payment the total amount of a valuation will comprise: a: amounts subject to retention (i) the total value of the work properly executed by the contractor (clause 4.2.1 (a)), and (ii) materials on site (clause 4.2.1 (b)), and possibly (iii) materials off site (clause 42.1 (c)), the value of any off-site goods and materials may be included at the discretion of the architect/ca. (see supplementary notes) heading. clickheretoretumto figure j: example of the courseware screen display package, its ability to dovetail with microsoft word where most of the subject matter already resided, and its ease of use for authoring once the initial techniques had been mastered. a further advantage in the delivery of the material was that the resultant set of files required only windows 3.1 to be available, thus enabling study to be conducted on any departmental computer, networked or otherwise. sixty students undertook the computer-based study and 3& (63%) voluntarily completed a feedback questionnaire on the following points of interest. location and timing of study the departmental computer laboratory was booked for the timetable periods corresponding to the cancelled lectures. the computer-based material was made available to the students to copy to their own disks such that they could choose when and where to carry out the study, whether during the booked laboratory periods or at another time elsewhere. a major advantage to the students was that the resultant files were quite small in terms of the disk space required (80 kbytes) and in only requiring windows 3.1 to be available, they could be used on a wide variety of machines both inside and outside the university. the students were asked whether they used the computer tutorial at the university in the booked laboratory period, at the university at another time, or at home/elsewhere. eight students chose more than one place of use and the responses are shown in table 1. 100 volume 4 number i place of use university in booked time university at other times home /elsewhere number of students 17 11 18 table i: location and timing of study of the computer tutorial a significant number of students took advantage of the opportunity to study either 'out of hours' and/or outside the university. it was apparent that the more computer-literate students who liked to work from home were very enthusiastic about the exercise. however, a significant number of the student group took advantage of the booked laboratory time, and they commented that the success of such an exercise depended on their gaining access to a university computer and that, in their experience, casual access was often not available. the implications of a more widespread use of this method of delivery on the demand for university computers would need careful consideration. ease of use of the computer-based material the responses to a question regarding the ease of use of the tutorial are shown in table 2. the majority of students experienced little or no difficulty in using the tutorial which was gratifying for the author of this paper in his first attempt at using the authorware. of those few who experienced difficulty, it was clear that a small number of students were 'anti-computers' regardless of this exercise. gilleard (1995), in the field of building services engineering, found a similarly positive response to cal tutorial work, especially where cal design and presentation promoted ease of understanding. ease of use very easy easy difficult very difficult number of students 12 22 4 0 table 2: ease of use of the computer tutorial i noted several distractions during the laboratory periods with regard to noise from other users (vacant machines were available for other students), demands of other work (including the temptation to use the computer time to word-process outstanding assignments) and technical problems (usually with regard to the students' own disks). all these factors were disruptive, and mitigated against effective study. some form of controlled environment, as on 'silence' floors in a library, may be necessary if this form of delivery were to become more prevalent. lecture vs. computer-based study the students were asked whether they preferred study of this subject by traditional lecture, computer tutorial or a mixture of the two. their responses are shown in table 3. 101 derek lavelle replacement of traditional lectures with computer-based tutorials: a case study preference of studslecture only computer tutorial only mixture of lecture /cbt no preference number of students 7 4 24 3 table 3: preference for form of study the responses to this question are particularly interesting. despite a very positive overall reaction to the use of the computer to assist their study, a notable majority of the student group still favoured retention of the traditional lecture in some form. a mixture of the two types of study was the strongly preferred option. this overall response is matched by the experiences of others. cordell (1991), studying learning styles and cal in the field of health care, recommends that educators should continue to use a variety of instructional methods including cal to facilitate learning. edward (1995) concludes that an effective computer-based laboratory simulation will enhance but not replace traditional techniques. the experiences of howard et al (1995) in the field of construction technology, suggest that while being positive about the adoption of new teaching technologies, students still value human interaction with lecturers and that an effective integration of technology into existing teaching and learning strategies is required. however, alternative opinions have also been offered. garrud et al (1993) in the field of medicine, conclude, with reservations, that cal can effectively substitute for traditional didactic teaching. stokes (1994) describes a project whereby lectures and seminars were replaced with cal material. the use of commercial cal material proved problematic, but the subsequent development of in-house cal material is proving to be a success. the issue of replacing versus enhancing traditional teaching with computer-based instruction is a delicate one. it is evident that truly effective cal does not bolt on to existing lectures and seminars, but rather causes the focus of them to alter. contact time may be more suitably used to offer support and feedback rather than the traditional emphasis on provision of information. there is considerable value in a human expert setting the scene, establishing expectations and being available for consultation as a course of study progresses. experience to date would suggest that, once good quality cal is available, a judicious balance of teaching modes is required to allow for effective integration of cal into a curriculum. flexibility of study the computer tutorial was designed to offer flexibility in timing and pace of study. the responses of the students as to whether this was a good or bad idea are shown in table 4. the computer tutorial was designed to offer the opportunity to review material as required. the responses of the students as to whether or not the material worked in this respect are shown in table 5. not unexpectedly, there was a very positive response to the opportunity to break free from the constraints of fixed timing and pace of delivery associated with traditional lectures. the responses may seem to be obvious, but this was a novice group with regard to cal, so their 102 alt-] volume 4 number i table 4: ability to control the timing and pace of study flexibility of timing and pace of study considered to be a: goodidea bad idea not concerned number of students 29 5 4 ability to review the material considered to be: beneficial not beneficial not concerned number of students 30 3 5 table 5: ability to review the computer-based material initial reactions are particularly noteworthy. it was noticeable that some students, having been directed into the material, took the opportunity to engage in more intensive study than would normally be expected in following up a traditional lecture. striking a more negative tone, some students did not enter into such 'deep' learning and felt the need for the clear and strict direction obtainable in a lecture, despite supplementary notes being available to them. isaacs (1990) concludes that tutorial cal has a definite role in assisting students to learn freely, that is to learn what they will and when they will. however, he also highlights the range of study skills which students require before gaining maximum benefit from tutorial cal. further computer-based study students were asked if they would like to see more of this type of study on their course. their responses are shown in table 6. table 6: preference for further computer-based study would y o u like further study of this type on your course ? yes no not concerned number of students 20 6 12 in considering this exercise as a pilot study, the overall positive response acts as a suitable encouragement for further development of the computer-based material. a sizeable number of students replied that they did not mind whether there was more computer-based study or not, implying that they are indifferent to the medium adopted and are content to leave such choices to the academics concerned. 103 derek love/fe replacement of traditional lectures with computer-based tutorials: a case study use of staff time it is difficult to be totally unequivocal about the efficiency of staff time when engaged in such work. there is obviously a front-end investment of time required in converting and updating materials to be suitable for use in electronic mode. however, if carried out wisely, this should result in many hours of student learning time being provided, hopefully with only minimum effort required in future updating of the material, this clearly being dependent on the subject under consideration. garrud et al (1993) were very clear that good quality cal requires substantial resources and high-calibre staff to develop and maintain it. stokes (1994) reported a lower than average requirement of development hours for in-house production of cal, indicating that efficiency gains can result from good management and adequate resourcing. the courseware supporting the project described in this paper was developed totally in-house with the help of suitable authorware. built environment does not have access as yet to much subject-specific material in the public domain, and we are now actively considering a strategy for in-house cal development. learning outcomes all of the results reported above are with regard to the behavioural response of the students to the change in media. the effect on academic performance is usually considered as the main criterion for evaluation in this type of study. however, it has not been possible in this pilot study to measure discrete effects on performance. the assessment of the subject was integrated in such a way that it was difficult to identify the results in the topics delivered by computer separately from those delivered traditionally. the study did not extend to altering the basis of assesment at this stage. in further development of this material, it may be possible to use selfassessment to provide more definite feedback on learning outcomes as well as to improve the learning process. academic performance may not be totally independent of behavioural response. my own earlier work (lavelle, 1992) demonstrated how a systematic use of appropriate it facilities helped to promote the development of cognitive skills and the adoption of new techniques. it is possible that the generally positive response of the students to their use of the computer-based material may demonstrate sufficient motivation towards a higher level of performance. however, the range of marks in the subject revealed nothing unusual when compared with other subjects, nor with previous student cohorts. on this basis, one could infer that there was no significant effect on performance either way, but the evidence is not conclusive because of the number of interdependent variables affecting performance at this level. conclusion with reference to the aims of this pilot study, the conversion of existing lecture material into computer-based material on an in-house production basis is proving viable, and shows promise for further development. the positive feedback from the student cohort on their use of the pilot cal material also gives encouragement for further development. 104 alt-j volume 4 number i references clarke, a. (1992), the principles of screen design for computer-based learning materials, learning methods project report ol121, learning methods branch, employment department group (2nd edition). cordell, b.j. (1991), 'a study of learning styles and computer-assisted instruction', computers and education, 16 (2), 175-83. edward, n.s. (1995), 'evaluations of student perceptions of a computer-based laboratory simulation', paper presented at alt-c conference, open university, milton keynes, 11-13 september 1995. garrud, p., chapman, i.r., gordon, s.a. and herbert, m. (1993), 'non-verbal communication: evaluation of a computer-assisted learning package', medical education, 27 (6), 474-8. gilleard, j. (1995), 'cal: an application in building services engineering', proceedings of the cib w89 conference: construction and building education and research beyond 2000, orlando, florida, 381-7. howard, c., pearson, a. and riley, m. (1995), 'learning from everything multimedia in construction technology education', proceedings of the cib w89 conference: construction and building education and research beyond 2000, orlando, florida, 765-73. isaacs, g. (1990), 'course and tutorial cal lesson design: helping students take control of their learning', educational and training technology international, 27 (1), 85-91. lavelle, d. (1992), 'promoting student learning and motivation via a systematic introduction to the use of it' in sloan, b. and schofield d. (eds), proceedings of the conference: computer based training in property and construction, university of salford, september 1992, 7-15. stokes, r.j.s. (1994), computer assisted learning in engineering at the university of humberside, open learning case studies, open learning foundation. 105 (page number not for citation purpose) 1 *corresponding author. email: vicki.dale@glasgow.ac.uk research in learning technology 2017. @ 2017 j. adekola et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 research in learning technology vol. 25, 2017 original research article development of an institutional framework to guide transitions into enhanced blended learning in higher education josephine adekola, vicki h.m. dale* and kerr gardiner learning enhancement and academic development service, university of glasgow, glasgow, uk (received 16 december 2016; final version received 5 may 2017) the rapidly changing digital landscape is having a significant influence on learning and teaching. our study assesses the response of one higher education institution (hei) to the changing digital landscape and its transition into enhanced blended learning, which seeks to go beyond the early implementation stage to make the most effective use of online learning technologies to enhance the student experience and student learning outcomes. evidence from a qualitative study comprising 20 semi-structured interviews, informed by a literature review, has resulted in the development of a holistic framework to guide heis transitioning into enhanced blended learning. the proposed framework addresses questions relating to the why (change agents), what (institutional considerations), how (organisational preparedness) and who (stakeholders) of transitions into enhanced blended learning. the involvement of all stakeholder groups is essential to a successful institutional transition into enhanced blended learning. keywords: digital education; technology-enhanced learning; e-learning; higher education; change introduction the changing digital landscape the changing digital landscape (gardiner 2015) has generated a new set of expectations in learning and teaching in higher education. new technologies offer more affordances for flexible learning (gordon 2014), and learners’ expectations are increasingly focused on a digital experience, regardless of debates surrounding variation in student and teacher digital literacies (kennedy et al. 2008; kirschner and van merriënboer 2013; margaryan, littlejohn and vojt 2011). today’s students expect an engaging educational experience where they can learn through exploration, interaction and collaboration in a way that is relevant, active, immediately useful and fun rather than the traditional model of teaching based largely on didactic face-to-face lectures (jukes, mccain, and crockett 2010). this realisation has led many universities around the world to explore blended learning (concannon, flynn, and campbell 2005; popovich and neel 2005; stacey and gerbic 2006). blended learning this study subscribes to garrison and kanuka’s (2004) definition of blended learning as an optimally designed combination of online and face-to-face learning, as well as mailto:vicki.dale@glasgow.ac.uk http://www.alt.ac.uk http://creativecommons.org/licenses/by http://dx.doi.org/10.25304/rlt.v25.1973 j. adekola et al. 2 citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 (page number not for citation purpose) graham, woodfield and harrison’s (2013) assertion that true blended learning is not an optional add-on, but actually results in a reduction of face-to-face contact time. blended learning may be differentiated from e-learning in that the latter is concerned only with computer-based or online learning and does not include the on-campus experience of face-to-face learning. a blended approach can support deep and meaningful learning (garrison and kanuka 2004). however, many institutions are at the initial stages of blended learning adoption according to graham, woodfield, and harrison (2013): awareness/exploration (stage 1), or adoption/early implementation (stage 2). the third stage; mature implementation/growth, is what we would refer to as enhanced blended learning. while there is increasing awareness of blended learning, only a few studies provide direction for heis looking to transition into enhanced blended learning (graham, woodfield, and harrison 2013; halverson et al. 2012; marshall 2011). marshall (2011) argues that there is too little evidence of critical self-reflection needed to bring about the required organisational change for an enhanced learning environment. according to him, overreliance on early adopters is one of the reasons heis fail to produce the enabling environment for a wider technology uptake at an institutional level. an institutional level vision, direction and energy is required for a successful strategic transition (taylor and newton 2013). existing e-learning frameworks khan’s (2005) octagonal framework introduced eight dimensions in relation to the design, delivery, implementation and evaluation of e-learning: pedagogical, technological, interface design, evaluation, management, resource support, ethical and institutional. in his framework, pedagogy relates to teaching and the learning needs of e-learners, especially the relationship between the content of the course and appropriate methods of delivery to enable learners to achieve the learning objectives. technology concerns technological infrastructure, hardware and software; the learning environment and the tools used for delivery. interface design includes elements such as page and site design, content design, navigation and accessibility; however, this aspect may be considered too specific within the broader context of institutional transitions to enhanced blended learning (although accessibility is an ethical requirement). evaluation is the north american term for assessment of learners, which we would place under pedagogy. management relates to people, process and product, that is, the management team, management of e-learning content delivery and the e-learning environment. resource support considers different types of resources (offline and online) that are available for learners. ethics encompasses issues such as social and cultural diversity, the digital divide and [n]etiquette. institution relates to the preparedness of the organisation in terms of administrative and academic affairs, for example, organisation and change, policy, faculty and staff support, and student services. the e-learning planning framework (elpf) is an alternative online tool that enables schools and teachers to evaluate their e-learning capability and consider community engagement with digital learning (te kete ipurangi n.d.). the framework identifies five dimensions that must work hand-in-hand for an institution to sustain its e-capability development over time. leadership and strategic direction includes the school’s vision, leadership of e-learning and strategic planning. learning and teaching http://dx.doi.org/10.25304/rlt.v25.1973 research in learning technology citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 3 (page number not for citation purpose) includes curriculum, learning areas, digital citizenship, pedagogy and assessment. professional learning encompasses learning communities and e-learning inquiry. technologies and infrastructure includes tools and technologies, technical support and procurement. the fifth dimension, beyond the classroom, looks at how schools can engage with the local community in promoting digital learning. using a case studies approach to consider blended learning in the united states, graham, woodfield, and harrison (2013) developed a blended learning adoption framework. the framework offered a tripartite model for the institutional adoption and implementation of blended learning in higher education; looking first at strategy (purpose, advocacy, implementation, definition and policy) then structure (governance, models, scheduling and evaluation) and finally support (technical, pedagogical and incentives). growing clarity, collaboration and investment in all dimensions were associated with effective transitions towards mature implementation and growth, or enhanced blended learning dimensions were associated with effective transitions. these overlapping frameworks provide useful guidance for institutions moving to enhance their e-learning or blended learning provision; however, khan’s framework fails to address the infrastructure needs of a distributed campus and, like te kete ipurangi’s framework, concentrates exclusively on e-learning. the blended learning adoption framework was based on interviews with senior administrators but did not incorporate the views of teachers or student representatives. it has been argued elsewhere (marshall 2011) that for educational strategy to be effective and efficient it ought to include all stakeholders. aim of this study given the limitations of existing frameworks, the study provided an opportunity to develop a holistic framework to guide institutional transitions into enhanced blended learning, in the context of uk higher education, informed by the views and experiences of a range of stakeholders. methodology research context the university of glasgow’s (uofg) (2015) institutional strategy outlines the centrality of new technologies in enhancing learning and teaching. substantial investment has been made in the institution’s provision for blended learning, through its digital learning spaces and accompanying physical infrastructure including technology-enabled active learning (teal) spaces. strategic, institutionally funded projects are expanding the provision of online distance and blended offerings, supported by the targeted recruitment of learning technologists and staff development provision. within graham, woodfield, and harrison’s (2013) blended learning adoption framework, the university can be said to be at the early implementation level. in 2014, the authors received funding by the quality assurance agency (qaa) scotland, to explore the current enhancement theme of ‘student transitions’. this provided the opportunity to use uofg as a case study to investigate student, staff and institutional transitions into enhanced blended learning. this article focuses on the institution’s transition. student experiences of transitions to blended learning are explored in a separate article (adekola et al. 2017). http://dx.doi.org/10.25304/rlt.v25.1973 j. adekola et al. 4 citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 (page number not for citation purpose) research questions in its broadest sense, this study was underpinned by research questions designed to investigate institutional motivations, benefits, challenges and barriers to enhanced blended learning, support required to overcome the challenges and barriers, and the perceived role of stakeholders in blended learning. the detailed motivations, benefits, challenges/barriers and enablers will be reported in a separate article. however, as will be explained in the findings, what emerged from the study, in terms of overall themes, was a deeper understanding of key considerations for heis looking to transition into enhanced blended learning. research methods semi-structured interviews were conducted and recorded by researcher ja with 20 participants from across the institution. a list of representative questions are included as appendix 1. participants were identified through purposive sampling based on their engagement with, and experience of, blended learning. members of senior management and heads of services, teachers, learning technologists and a member of the student representatives council were directly invited to take part. the student representative was the formal representative for learning and teaching across the institution and attended all relevant education committees; therefore, they were considered well placed to comment on student needs with regard to blended learning. appendix 2 serves as a key to the numbers accompanying each quote in the results section, according to role. participants were asked to reflect on their individual views and experiences of transitions to enhanced blended learning. semi-structured interviews were chosen as they enable a degree of conformity with regard to data collection from all participants while allowing for specific issues to be raised by individual participants (cohen and crabtree 2006). the interview recordings were sent to a professional transcription company and the resulting transcripts reviewed by researcher ja for accuracy. the study was reviewed and approved by the university’s college of social sciences research ethics committee. data analysis researchers ja and vhd used thematic analysis as defined by braun and clarke (2006), in such as way as to make no epistemological or ontological assumptions, and mainly in an inductive manner. transcripts were read through, coded, codes grouped into categories and emerging ‘themes’ identified. however, our analysis was also informed through engagement with appropriate literature (prior to and in parallel with interview data collection), including peer-reviewed journal articles, books, conference presentations and non-peer-reviewed reports. this allowed us to situate our findings in relation to the existing evidence base. two of the researchers (ja and vhd) independently coded the interview transcripts (ja manually on paper, and vhd first manually on paper and then importing the data into nvivo) and negotiated a consensus, collaboratively verifying the codes and categories against the original interview data, our interpretation being informed by the prior and ongoing literature review. the results presented here represent key themes, around drivers for enhanced blended learning, institutional considerations, organisational readiness and stakeholders’ roles. http://dx.doi.org/10.25304/rlt.v25.1973 research in learning technology citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 5 (page number not for citation purpose) results while the original study sought to address specific research questions as indicated above, out of this work emerged an understanding of key drivers or change agents, institutional considerations or support needs, processes to facilitate institutional alignment of stakeholders and an understanding of stakeholder roles in this new digital landscape of blended learning. these overall themes form the basis of the resulting framework. these are represented in figure 1. key drivers sector-level key drivers were identified and categorised as the changing digital landscape, stakeholders’ expectations of he, internationalisation, and quality assurance and enhancement (table 1). one of the key messages that emerged from the study was the need for heis to meet the demands of this evolving digital landscape. technology is increasingly seen as part of everyday life and its enhanced use in education is expected. the internet and rise of social media have widened access to education (sometimes at no cost to the learner, e.g., through massive open online courses [moocs]) without needing to be based physically at an institution. for an institution to stay competitive in this globalised world, its presence in the online space is essential. there was also attention to the issue of internationalisation as institutions have become aware of the increasing importance for education to have global relevance management and organisa�on learning technology support commitment c o m p et en ce c o m m u n ic a� o n changing digital landscape ins�tu�onal considera�ons organisa�onal preparedness interna�onalisa�on st ak eh o ld er s’ e xp ec ta � o n s change agents collabora�on the ins�tu�on staff student com n ceec ta � o n s stakeholders’ roles p ll b � the ins�tu�on staff students d ins�tu�onal alignment u k h ig h er e d u ca � o n (q u al it y a ss u ra n ce & e n h an ce m en t) figure 1. holistic framework to support effective institutional transitions into enhanced blended learning. http://dx.doi.org/10.25304/rlt.v25.1973 j. adekola et al. 6 citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 (page number not for citation purpose) while reaching out to an international audience. furthermore, uk he policies and standards, such as those devised by the qaa, were seen to regulate the academic community and the standards of higher education, ensuring an optimal student learning experience. support needs and recommendation of good practice the findings from the interviews, shown in table 2, map to six dimensions of institutional considerations. these findings allow realignment of khan’s (2005) octagonal e-learning framework to the context of our present study to include a campus-based, uk he setting. the six dimensions that our study evidenced include physical infrastructure, learning technology support, pedagogy, management and organisation, institutional culture, and ethical/legal. these are detailed in figure 2 with representative quotes in table 2. table 1. narratives around key drivers for change. theme illustrative quotes changing digital landscape • ‘we live in this technology-rich environment. it’s no longer good enough just to say we’ll put in technology if it enhances. you’ve got to think about the landscape you inhabit and make sure that you’re making the appropriate tools for that landscape and you’re not using last century’s tools’. (#2). • ‘technology in general, is embedded into our lives, and therefore, it’s now become part of our way that we learn, as well’. (#14) stakeholders expectations • ‘i think there’s a general perception that students are becoming more and more used to doing everything online, and used to being able to do all sorts of things on their laptop or their phone or their tablet or whatever, on the move as well, and that it would be good for them or good for us, rather, if we could offer that kind of service’. (#20). • ‘the students themselves, because they’re obviously very used to technology now’. (#5) internationalisation • ‘we will have the opportunity in the future to look at connecting cohorts of students who are studying for the same degrees but in different parts of the world … we can have that cultural engagement in that learning space’. (#18) • ‘the need to expand into these emerging markets, places like china and india, to deliver the education that these people want but possibly physically can’t access’. (#10) quality assurance and enhancement • ‘i think key considerations for me are the quality of the student experience and the viability of the approach… [the] quality of the student experience is paramount’. (#18) • ‘for me the whole process that the university of glasgow is going through at the moment around blended and online learning, it’s a learning experience for us as an institution. and we need to be very careful about how we measure that and what lessons we take from that as well’. (#13) http://dx.doi.org/10.25304/rlt.v25.1973 research in learning technology citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 7 (page number not for citation purpose) table 2. narratives around support requirements and institutional considerations. themes illustrative quotes physical infrastructure • ‘we’ll be looking at the way we repurpose learning spaces, so that these learning spaces will be suitable for… the kind of learning that is now becoming pervasive through technology-enhanced learning and teaching, particularly collaborative learning spaces, we’ll be building more of those. we will want students to be able to go into those spaces and have…a flipped classroom experience or a technology-enhanced learning and teaching experience in that space’. (#18) • ‘well, what’s working well in terms of the library building as a whole is that we have enough space to be able to zone it, so that on the lower floors, and we’re about to refurbish level two and it will become even more a flexible learning space … so on the lower floors of the library we have it coded green, which means that students can … there’s no silence policy at all on these floors. so the students are allowed to talk, to work together in groups’. (#17) learning technology support • ‘i think it’s got to continue to invest in a level of centred support but also to encourage schools and colleges to take a joined up approach and have local support which is specialised, and not just think that it’s it support’. (#2) • ‘making sure we have the resources, by which i mean the it infrastructure, the software, the learning technology staff who support academic staff …and staff training’. (#17) pedagogy • ‘the research on learning and pedagogy comes from western contexts... that’s very valuable research, but it doesn’t seem to work as well with my chinese students … perhaps western students would embrace that uncertainty, because in the real world, you need to show skills that you can handle uncertainty … however, i’m wrong with chinese students, because their real world in china and the chinese workplace, they are being quite prescriptive. so what am i equipping them for?’. (#12) • ‘i have seen many examples of the use of technology for the sake of technology, where there’s been no real benefit either to the educator or to the student, and for me any evolution of technology enhanced learning has to actually be led by the pedagogy and the learning requirement rather than the technology’. (#18) management and organisation • ‘the leadership …. i think that’s a key driver’. (#1) • ‘i think that [the institution] already rewards and recognises people who are innovative within the promotions criteria and the reward and recognition sort of criteria, but probably more could be done to make it easy for people to evidence their innovation. i think probably allowing staff some protected time for scholarship, including scholarship within technology and innovation. i think that needs to be built into the workload model … it’s doing a lot in that area, but it needs to do more’. (#19) institutional culture • ‘senior management have to have a willingness to try and also because it is quite new, we are going into new territory with a lot of technology, there has to be a willingness to fail. there has to be a willingness to say “we’ll give it a go and if it doesn’t work we’ll learn from that and move on.” there cannot be a fear of change because if there’s a fear of failure then nothing will ever change’. (#5) • ‘mindset, yes, it’s culture, mindset, it’s the world views that people have about teaching and learning, and that is a fundamental challenge’. (#3) http://dx.doi.org/10.25304/rlt.v25.1973 j. adekola et al. 8 citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 (page number not for citation purpose) physical infrastructure a key finding from the interviews was the fact that learning environments include physical aspects (flexible learning spaces and connectivity) and virtual spaces (virtual learning environment, digital libraries) and that there is a need to repurpose learning spaces to support a blended environment, with the library taking a lead in piloting flexible learning spaces. learning technology support additional learning technology support was seen as essential in order to cater for varying levels of digital literacy among staff within the changing digital landscape. table 2. (continued) themes illustrative quotes ethics/legal • ‘bring your own device is quite a big hot topic at the moment and it’s probably to an extent how universities are tending to be going. if i have 100 students in a lecture … and i tell them to bring their own device, there might be one student there who doesn’t have their own device because they don’t want to use technology or they don’t have the money to buy the technology’. (#13) • ‘i’m sure there are legal ones [challenges], particularly around copyright, … if you’re starting to create stuff that’s online, taking images from the internet is a breach of copyright, so we are really into some very thorny legal issues’. (#3) management & organisation learning technology support • providing leadership • providing support & resources • rewarding staff engaged in bl • addressing student learning needs & expectations • digital literacies • new approaches to l&t • flexible, active learning spaces • robust it infrastructure • enabling innovation, being mindful of risks • greater tolerance towards failure • distributed learning technology specialist support • telt communities of practice • peer mentoring by early adopters • equity of access to technology • learner support for engaging in bl • copyright compliance, training & support • internationalisation of curriculum institutional considerations for blended learning figure 2. institutional considerations for enhanced blended learning. http://dx.doi.org/10.25304/rlt.v25.1973 research in learning technology citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 9 (page number not for citation purpose) central and distributed learning technology support, peer mentoring by early adopters, support through technology-enhanced learning and teaching (telt) communities of practice (at school, college, institution and sector levels) and access to local equipment were all viewed as important. pedagogy participants viewed pedagogical considerations as essential, especially around the need to conceptualise new models of learning and teaching, addressing student learning needs and expectations including those of diverse cohorts, and constructive alignment of assessment with intended learning outcomes and learning activities. however, the data suggest that the institution needs to be cautious in applying ‘western’ pedagogies to international student cohorts with different cultural expectations around learning and teaching and different workplace cultures; student transitions need to be supported. providing evidence for the increased use of blended learning was also considered important; there was a sense that technology is often viewed as the solution to an undefined ‘problem’ and so concentrating on student learning remains paramount. management and organisation participants emphasised the role of senior management in articulating vision to provide leadership by informing strategy, providing support and resources, allowing staff autonomy while encouraging buy-in, and recognising and rewarding staff for engaging in blended learning. the study revealed the need for clear linkage between high-level strategy and operational delivery at a local level, requiring communication across the institution. a tension between research and teaching in relation to promotion criteria and staff workload was seen to require a reassessment of how staff effort is recognised and rewarded, as well as creating time in the staff workload for teaching innovation. institutional culture despite the clear support from senior management for innovation in learning and teaching, it was suggested by some participants that the institutional culture needs to evolve to embrace a greater tolerance to failure as part of this transition to greater blended learning, although this approach needs to be carefully managed. developments in relation to all dimensions of blended learning were seen to be enabled within a responsive and progressive institutional culture regarding the adoption of new technologies for learning and teaching. ethical/legal data relating to the ethical aspects suggested the need to consider the accessibility of learning technology for students with disabilities, equity of access to devices among learners, internationalisation of the curriculum, and ensuring adequate learner support. from a legal perspective, copyright compliance, training and support was regarded as a requirement to overcome risks around breaching copyright, and to set a good example to students with regards to attributing ownership. http://dx.doi.org/10.25304/rlt.v25.1973 j. adekola et al. 10 citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 (page number not for citation purpose) organisational preparedness for change this aspect of our study is largely derived from the literature review, in particular reasons (1993), cited by fischbacher-smith (2016), who notes that in order to ensure effective organisational change management, there must organisational commitment, competence and awareness. based on the outcomes of our study, we divided ‘awareness’ into communication and collaboration, to reflect the need for ongoing dialogue and partnership working between stakeholders (table 3). stakeholder roles table 4 details participant narratives around stakeholder roles, derived from the interview data. teachers involved in blended learning were largely regarded as facilitators of student learning, encouraging critical thinking and problem-solving. students were considered as co-producers of knowledge, expected to take responsibility for their own learning. senior management were regarded to have an enabling role in terms of providing resources and opportunities for blended learning and staff development. table 3. narratives around effective institutional change management process. themes illustrative quotes competence • ‘it’s all about competence, yes, competence and confidence … those two should go hand in hand … i know for myself i’m very confident using, for example, video software, but there’s also the competence about understanding copyright, so for example, i’ve made up videos that were put onto itunes u, but i have to be aware of the copyright issues and make sure i have permission for all the different photographs and so on, and sometimes that gets lost in the circle’. (#5) • ‘[there is] lack of technical competence of staff so we would need to be trained’. (#3) commitment • ‘senior management have to have a commitment to encouraging change but also having robust policies in place to say, okay, well, for us to do that what are the issues that may come up and how can we solve them?’. (#5) • ‘how can they show commitment? well, just by enabling the people that teach to do the best that they can towards blended learning, to provide the right support, training, the right tools for people that are providing this material, so that students can engage with it. and also to take on board what students think because, ultimately, they’re the consumer’. (#4) communication • ‘my experience at glasgow is that there is quite strong dialogue, evidenced by the learning and teaching strategic consultation, the e-learning strategic consultation, the e-learning and teaching hub consultation … there’s a lot of consultation going on’. (#2) • ‘we need to set that direction, but we do it in consultation with our colleagues, and we make sure that again developing the vision, is a vision that our colleagues can buy into’. (#18) collaboration • ‘i think we might also need to be much more collaborative in terms of teaching and learning’. (#3) • ‘being open to working collaboratively, generally, as an institution, and also encouraging staff in the institution to work across disciplines, within disciplines with one another, sharing information, going down the open resources route, possibly, is going to be beneficial to the institution’. (#4) http://dx.doi.org/10.25304/rlt.v25.1973 research in learning technology citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 11 (page number not for citation purpose) discussion: an institutional framework for transitioning into enhanced blended learning the literature review revealed the absence of a uk-focused framework to guide a large, long established and complex hei with a distributed physical campus in making an effective transition to enhanced blended learning. this study has led to the development of a framework comprising four layers: change agents, institutional considerations, organisational preparedness for change and stakeholders’ roles (figure 1). change agents these key drivers for enhanced blended learning include the changing digital landscape (jukes, mccain, and crockett 2010), internationalisation (barber, donnelly, and rizvi 2013), quality assurance and enhancement (varlamis and apostolakis 2010), and stakeholder expectations including those of increasingly digitally fluent students and staff (frand 2000; garrison and kanuka 2004). understanding these is essential in strategically positioning an institution considering the transition into enhanced table 4. narratives around stakeholder roles. themes illustrative quotes teacher as facilitators of learning • ‘the teacher’s role in blended learning is the same as in fully online or fully face-to-face learning; it’s about scaffolding and building that environment where the students will actually learn … what i do see overall is an increasing co-creation type activity’. (#3) • ‘in my particular experience, i don’t necessarily see myself as a teacher. i’m a facilitator. it’s the students who are doing the learning. i’m just facilitating that’. (#5) student as co-creators of knowledge • ‘the same role that they play if they’re in the classroom, which is to some extent a co-producer of knowledge, because the dialogue, whether it’s electronically mediated or face to face, it’s still about how we understand, how we shape knowledge, how we interpret it, how we apply it’. (#7) • ‘students have a really important role to play, and i’m not sure they appreciate what that role is … it’s our responsibility to make this role very explicit to them. so we’re working on the assumption of knowledge co-creation’. (#12) senior management as enablers • ‘i think it’s to build the institutional capability, is the key one. so it’s to lead and build it, and to make sure that the key enablers and the key decision makers are aware of what’s happening in the sector, what’s happening in this whole area, and to foster a general feeling within the university that this is something which is worth doing and that we want to do, and that resources are pushed towards it’. (#2) • ‘the main reason why blended learning is taking off in the institution is because senior management … are starting to take notice of the benefits of e-learning, and technology enhanced learning, but also giving us resource to do so … it does take time to create good e-learning … just making the place an environment that’s more conducive to developing that type of learning is going to be useful’. (#4) http://dx.doi.org/10.25304/rlt.v25.1973 j. adekola et al. 12 citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 (page number not for citation purpose) blended learning. these key drivers also secure the commitment of stakeholders, especially senior management, in leading the desired change. institutional considerations the institutional considerations are derived from the findings on support needs and recommendations of good practice, and reuse elements of khan’s (2005) octagonal e-learning framework, reframing it for a uk he context and focusing on campus-based blended learning. these dimensions overlap with considerations identified in the frameworks proposed by te kete ipurangi (n.d.) and graham, woodfield, and harrison (2013). figure 2 summarises individual considerations within the six dimensions. a supportive institutional culture is recognised as essential for successful educational change (garrison and kanuka 2004; kezar and eckel 2002). management and organisation is seen as pivotal in providing institutional commitment and leadership (garrison and vaughan 2013) and strategic seed funding (garrison and kanuka 2004, porter et al. 2014), incentivising staff through recognition and reward (moskal, dziuban and hartman 2013; porter et al. 2014), providing time for blended learning in the workload model (garrison and vaughan 2013; porter et al. 2014), and continuing to evaluate the benefits of blended learning (garrison and kanuka 2004; moskal, dziuban and hartman 2013; porter et al. 2014). the need for technological and pedagogical support is highlighted by garrison and kanuka (2004), graham, woodfield, and harrison (2013), moskal, dziuban, and hartman (2013) and porter et al. (2014). effective pedagogy underpinning the use of technology to enhance student learning outcomes is essential (porter et al. 2014). according to taylor and newton (2013), this requires upskilling staff and students in relation to digital literacies. the importance of having clear communication with students to manage expectations around blended learning is also emphasised, including clarity around the roles of learners and teachers (garrison and vaughan 2013; porter et al. 2014; quinn et al. 2012). in relation to learning technology support, moskal, dziuban, and hartman (2013) highlighted the need for mid-level organisational capacity enabled by the recruitment of staff with expertise in learning technology and academic staff development, including instructional designers. taylor and newton (2013) advise clarity around the complementary roles of support staff to avoid confusion. our study additionally identifies the need for distributed support within a coordinated hub-and-spoke model. physical infrastructure conveys an institutional philosophy or statement about its approaches to teaching and learning (skill and young 2002). the importance of a robust and reliable technical infrastructure is recognised to be critical in transitioning to enhanced blended learning (garrison and kanuka 2004; moskal, dziuba, and hartman 2013; quinn et al. 2012). our study also highlighted the importance of teal spaces, seen to enhance student experience, engagement, collaboration and flexibility (neill and etheridge 2008). finally, the ethical/legal dimension in our study verifies the findings of others. ethical issues around learner diversity, geographic diversity and the digital divide were identified by khan (2005), as well as legal or regulatory aspects such as privacy, plagiarism and copyright. the need for staff development about copyright and accessibility is also highlighted by moskal, dziuban, and hartman (2013). all these developments require careful change management, which requires examining the institution’s preparedness for blended learning. http://dx.doi.org/10.25304/rlt.v25.1973 research in learning technology citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 13 (page number not for citation purpose) organisational preparedness for change organisational preparedness requires the institution to reflect on its capability and readiness for change. informed by the literature review and the interview findings, four key elements were identified: competence, commitment, communication and collaboration. staff and students require competence to undertake the relevant tasks; digital literacies are critical. stakeholder commitment is essential; commitment of staff may be secured by senior management effectively recognising and rewarding staff for engaging in educational innovation. communication (to raise awareness of the possibilities of blended learning) and collaboration (e.g. peer learning and sharing of good practice between teachers, and staff–student partnership working) are also essential, and help secure student and staff commitment. the organisational readiness aspect of the framework overlaps to some extent with the change management framework by quinn et al. (2012), which incorporates and cites kotter’s (2007) directional change framework. stakeholder roles at the heart of the framework are key stakeholders. institutional alignment and regular dialogue between these stakeholders is critical in effecting institutional change towards enhanced blended learning (moskal, dziuban, and hartman 2013). conclusions and implications our study assesses one hei’s response to the changing digital landscape and its transition into enhanced blended learning, and argues that a holistic approach is required to enable a successful institutional transition into enhanced blended learning. a framework has been developed to guide heis, comprising four overall themes: an overview of change agents driving the move towards enhanced blended learning (the why), institutional considerations to address support needs (the what), processes that facilitate enhanced blended learning (the how) and stakeholder groups that should be involved in the transition (the who). our findings echo and validate those of other studies in blended learning. in terms of implementing this framework, discussions with the institutional team (which overlap with but do not exclusively represent the participants of this study) has led to the identification of ‘anchor points’ to help move the university forward to the enhanced blended learning stage. proposed interventions, based on the framework, include more explicit guidelines for staff on blended learning design, a networking event for stakeholders to share experiences and expertise, and staff–student partnership working around blended course development. acknowledgements the authors are indebted to the quality assurance agency in scotland for funding this research under the enhancement themes initiative (2014–17). we are also incredibly grateful to our study participants for their time and valuable insights regarding transitions to enhanced blended learning. we would also like to thank professors denis and moira fischbacher-smith and dr angela jaap for their constructive comments on a previous draft of this article. http://dx.doi.org/10.25304/rlt.v25.1973 j. adekola et al. 14 citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 (page number not for citation purpose) references adekola, j., dale, v. h. m., gardiner, k. & fischbacher-smith, m. (2017) ‘student transitions to blended learning; an institutional case study’, journal of perspectives in applied academic practice. 5(2), 58–65. barber, m., donnelly, k. & rizvi, s. (2013) an avanlanche is coming: higher education and the revolution ahead, [online] available at: http://med.stanford.edu/smili/support/final%20 avalanche%20paper%20110313%20(2).pdf braun, v. & clarke, v. (2006) ‘using thematic analysis in psychology’, qualitative research in psychology, vol. 3, pp. 77–101. cohen, d. & crabtree, b. (2006) semi-structured interviews, [online] available at: http://www. qualres.org/homesemi-3629.html. concannon, f., flynn, a. & campbell, m. (2005) ‘what campus-based students think about the quality and benefits of e-learning’, british journal of educational technology, vol. 36, no. 3, pp. 501–512. fischbacher-smith, d. (2016) ‘breaking bad? in search of a (softer) systems view of security ergonomics’, security journal, vol. 29, no. 1, pp. 5–22. frand, j. l. (2000) ‘the information-age mindset changes in students and implications for higher education’, educause review, vol. 35, no. 5, pp. 14–24. gardiner, k. (2015) ‘reasons to be open – embracing the digital landscape’, paper presented at the association for learning technology conference (alt-c), 8–10 sep., 2015. garrison, d. r. & kanuka, h. (2004) ‘blended learning: uncovering its transformative potential in higher education’, the internet and higher education, vol. 7, no. 2, pp. 95–105. garrison, d. r. & vaughan, n. d. (2013) ‘institutional change and leadership associated with blended learning innovation: two case studies’, the internet and higher education, vol. 18, pp. 24–28. gordon, n. (2014) flexible pedagogies: technology enhanced learning, [online] available at: https://www.heacademy.ac.uk/flexible-pedagogies-technology-enhanced-learning graham, c. r., woodfield, w. & harrison, j. b. (2013) ‘a framework for institutional adoption and implementation of blended learning in higher education’, the internet and higher education, vol. 18, pp. 4–14. halverson, l. r., graham, c. r., spring, k. j. & drysdale, j. s. (2012) ‘an analysis of high impact scholarship and publication trends in blended learning’, distance education, vol. 33, no. 3, pp. 381–413. jukes, i., mccain, t. & crockett, l. (2010) understanding the digital generation: teaching and learning in the new digital landscape, corwin press. kennedy, g., judd, t., churchward, a., gray, k. & krause, k.-l. (2008) ‘first year students’ experiences with technology: are they really digital natives?’, australasian journal of educational technology, vol. 24, no. 1, pp. 108–122. kezar, a. j. & eckel, p. d. (2002) ‘the effect of institutional culture on change strategies in higher education: universal principles or culturally responsive concepts?’, the journal of higher education, vol. 73, no. 4, pp. 435–460. khan, b. h. (2005) managing e-learning: design, delivery, implementation, and evaluation, igi global. kirschner, p. a. & van merriënboer, j. j. g. (2013) ‘do learners really know best? urban legends in education’, educational psychologist, vol. 48, no. 3, pp. 169–183. kotter, j. (2007). leading change: why transformation efforts fail. harvard business review, 1 january, 96–103. http://hbr.org/product/leading-change-why-transformation-effortsfailhar/an/r0701j-pdf-eng; http://www.stratafrica.com/media/2a8f73ccf40110beffff80be7f000101.pdf margaryan, a., littlejohn, a. & vojt, g. (2011) ‘are digital natives a myth or reality? university students’ use of digital technologies’, computers & education, vol. 56, vol. 2, pp. 429–440. marshall, s. (2011) ‘change, technology and higher education: are universities capable of organisational change?’, journal of asynchronous learning networks, vol. 15, no. 4, pp. 22–34. http://dx.doi.org/10.25304/rlt.v25.1973 http://med.stanford.edu/smili/support/final%20avalanche%20paper%20110313%20(2).pdf http://med.stanford.edu/smili/support/final%20avalanche%20paper%20110313%20(2).pdf http://www.qualres.org/homesemi-3629.html http://www.qualres.org/homesemi-3629.html https://www.heacademy.ac.uk/flexible-pedagogies-technology-enhanced-learning http://hbr.org/product/leading-change-why-transformation-efforts-failhar/an/r0701j-pdf-eng http://hbr.org/product/leading-change-why-transformation-efforts-failhar/an/r0701j-pdf-eng http://www.stratafrica.com/media/2a8f73ccf40110beffff80be7f000101.pdf http://www.stratafrica.com/media/2a8f73ccf40110beffff80be7f000101.pdf research in learning technology citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 15 (page number not for citation purpose) moskal, p., dziuban, c. & hartman, j. (2013) ‘blended learning: a dangerous idea?’, the internet and higher education, vol. 18, no.7, pp. 15–23. neill, s. & etheridge, r. (2008) ‘flexible learning spaces: the integration of pedagogy, physical design, and instructional technology’, marketing education review, vol. 18, no. 1, pp. 47–53. popovich, c. j. & neel, r. e. (2005) ‘characteristics of distance education programs at accredited business schools’, american journal of distance education, vol. 19, no. 4, pp. 229–240. porter, w. w., graham, c. r., spring, k. a. & welch, k. r. (2014) ‘blended learning in higher education: institutional adoption and implementation’, computers & education vol. 75, pp. 185–195. quinn, d., amer, y., lonie, a., blackmore, k., thompson, l. & pettigrove, m. (2012) ‘leading change: applying change management approaches to engage students in blended learning’, australasian journal of educational technology, vol. 28, no. 1, pp. 16–29. skill, t. d. & young, b. a. (2002) ‘embracing the hybrid model: working at the intersections of virtual and physical learning spaces’, new directions for teaching and learning, vol. 92, pp. 23–32. stacey, e. & gerbic, p. (2006) ‘teaching for blended learning’, in education for the 21st century – impact of ict and digital resources: ifip 19th world computer congress, tc-3, education, eds d. kumar & j. turner, august 21–24, 2006, santiago, chile, springer us, boston, ma, pp. 225–234. taylor, j. a. & newton, d. (2013) ‘beyond blended learning: a case study of institutional change at an australian regional university’, the internet and higher education, vol. 18, pp. 54–60. te kete ipurangi (n.d.) e-learning planning framework, [online] available at: http://elearning. tki.org.nz/professional-learning/e-learning-planning-framework university of glasgow (2015) university strategy 2015–2020, [online] available at: http://www. gla.ac.uk/about/strategy/ varlamis, i. & apostolakis, i. (2010) ‘a framework for the quality assurance of blended e-learning communities’, in knowledge-based and intelligent information and engineering systems, vol. 6278, eds r. setchi, i. jordanov, r. howlett & l. jain, springer, berlin, pp. 23–32. http://dx.doi.org/10.25304/rlt.v25.1973 https://www.heacademy.ac.uk/flexible-pedagogies-technology-enhanced-learning https://www.heacademy.ac.uk/flexible-pedagogies-technology-enhanced-learning http://www.gla.ac.uk/about/strategy http://www.gla.ac.uk/about/strategy j. adekola et al. 16 citation: research in learning technology 2017, 25: 1973 http://dx.doi.org/10.25304/rlt.v25.1973 (page number not for citation purpose) appendix 1. representative interview questions • tell us about your background/experience in relation to blended learning • what do you understand by the term ‘blended learning’? • what are the factors driving blended learning in the sector and the institution? • what are the benefits of blended learning? what has worked well at uofg in relation to blended learning? • what are the challenges and/or barriers encountered in relation to blended learning? • how do you perceive the available support for blended learning? what additional support is needed? • what are the roles of students/staff/management in blended learning? • what key considerations must be put in place as we move towards more investment in bl? appendix 2. participant roles role participant number educators (university teachers, lecturers, professors) 3, 5, 8, 12, 13, 14, 15, 16, 20 senior management including deans of learning & teaching 7, 11, 15, 18, 19 learning technologists 1, 4, 10 heads of services with remit to support learning and teaching 2, 9, 17 formal student representative for learning and teaching across the institution 6 http://dx.doi.org/10.25304/rlt.v25.1973 aix-j volume 2 number 1 auto-monitoring: theoretical touchstone or circular catch-all? nick hammond cti centre for psychology, university of york, uk until i read ray mcaleese's paper, i perhaps had a rather simplistic psychologist's view of concept mapping. it was, i felt, a technique one of several that helped learners to articulate their burgeoning understanding of some topic, providing a canvas on which to record, expand and manipulate their knowledge. the activities involved in concept mapping could be linked to well-known psychological principles of understanding, memorization and learning: effort, elaboration and depth of processing; generation and enactment effects; encoding specificity and encoding variability; the distinctions between explicit and implicit representations; metacognitive strategies and reflection, and so forth (hammond 1993). these psychological underpinnings, while not in any sense providing an integrated 'theory' of concept mapping, give a view of when and why the use of concept mapping might be effective in some situations and not in others, and how different concept mapping tools differ in the claims they are making about their educational use (trapp, reader and hammond 1992). the concept of auto-monitoring provides a related but higher-level perspective. thus, as mcaleese argues, it links the areas of reflection, meta-cognition and concept mapping. it also provides a process view of the stages involved in concept mapping (mcaleese, figure 2). there is perhaps some theoretical leverage here. (mcaleese wisely steers clear of the tempting but fallacious argument that maps are good for learning because they reflect structures in the head in some way this fallacy is analysed by mckendree and reader 1994.) however, mcaleese is claiming more than this: it is not just that concept mapping can be a rather effective way of learning because it requires explication and reflection; the claim (as i understand it) is that concept mapping helps students develop auto-monitoring techniques and so enhance their critical thinking skills. while acknowledging that this might be the case, i have yet to see either any evidence or a persuasive theoretical argument that supports it. there have, after all, been many contenders for this educational equivalent of the philosopher's stone, from latin grammar to logo programming. whether or not concept mapping does assist in the development of thinking skills (over and above providing a specific technique for helping lay out one's knowledge) is still an open empirical question. it is here that i have a slightly uncomfortable feeling of circularity. to simplify the argument: critical thinking skills depend substantially on reflection, self-monitoring and gaining an overview, in short on auto-monitoring; concept mapping is a set of activities which both requires and develops auto-monitoring; therefore concept mapping develops critical thinking skills. but is the concept of auto-monitoring as yet sufficiently well-defined or robust to serve as the bridge? the flaw is that none of these three concepts (thinking skills, auto-monitoring or concept mapping) is unitary, and the links between them are strictly hypothetical. thinking involves a complex amalgam of different skills, which have to be appropriately mobilized (implicitly or explicitly), depending on the situation. as mcaleese points out, auto-monitoring as an 'executive control' is likely to involve a range of mental procedures. and concept 49 david jonassen and rose marra mindtoolsfor representing knowledge mapping supports a diverse range of learning and comprehension tasks. the hypothesis is therefore a challenging one, though not yet proven. but perhaps i have gone further than mcaleese is claiming. i am certainly happy with his view of auto-monitoring as a 'learning arena' in which the learner can be aware of and manipulate learning activities. references hammond, n.v. (1993), 'learning with hypertext: problems, principles and prospects', in mcknight, c., dillon, a. and richardson, j. (eds), hypertext: a psychological perspective, chichester, ellis horwood. mckendree, j. and reader, w. (1994). 'the "homeopathic fallacy" in hypertext: misconceptions of psychology in design of computer courseware', alt news (in press). trapp, a.l., reader, w. and hammond, n.v. (1992), 'tools for knowledge mapping: a framework for understanding' in brusilovsky, p. and stefanuk, v. (eds), east-west conference on emerging technologies in education, moscow, april. concept mapping and other formalisms as mindtools for representing knowledge david h. jonassen* and rose m. marra** * pennsylvania state university and **at&t, usa introduction we seek to provide an alternative theoretical perspective on concept mapping (a formalism for representing structural knowledge) to that provided by ray mcaleese in this issue of alt-j (auto-monitoring). we begin with an overview of concept maps as a means of describing a learner's knowledge constructs, and then discuss a broader class of tools, mindtools, of which concept maps are a member. we proceed by defining mindtools as formalisms for representing knowledge, and further elaborate on concept maps as a formalism for representing a particular kind of knowledge: structural knowledge. we then address mcaleese's use of the term auto-monitoring and some of the steps in his model of concept maps. finally, we describe some limitations of concept mapping as a formalism and as a cognitive learning strategy. concept maps, both in how they are used and theoretically based, touch some of the central elements of instructional technology and learning. concept maps address the critical issue of how we organize our knowledge structures within and between domains. they, in some sense, model how we represent and apply knowledge. assuming that instructional technology, as a field, has a goal of being able systematically to make instruction more effective, then tools 50 alt-j volume 2 number 1 knowledgepro windows: the order of merit? martin reynolds department of computer science, the victoria university of manchester, uk abstract the producers of knowledgepro look set with their latest release of kpwin (knowledgepro windows) to fulfil richard hale-shaw's prophecy that it will become 'one of the most powerful visual development environments' (hale-shaw 1992). comparisons are drawn in this paper between the kpwin family of products and other authoring tools. the conclusion is that kpwin is worthy of being included in any courseware developer's tool set. reasons for preferring a tool from the knowledgepro family of products for courseware development over three main competitors authorware, toolbook and visual basic are explained, and the merits of kpwin and kpwin++ (a version that generates c++ code) are examined. introduction courseware development is invariably a difficult and costly task. the quality of the endproduct often depends critically on the types of tool used to develop it. over the last decade, a wide variety of courseware and authoring tools has been produced (barker 1989), and the range continues to grow. methods and reasons for selecting one particular tool in preference to another are therefore of considerable importance. the introduction of courseware can affect changes which produce organizational, computational, or motivational benefits. the effectiveness of courseware in an educational context is solely dependent on its use (kulik and kulik 1991). and the more experimental the educational objectives are, the more important it is for courseware to act as a reflexive tool rather than as a rubric. for courseware to be accepted, its quality must justify its use in a course as a whole, in supporting particular tasks and in helping learners how to learn. when substantial projects or complex products are envisaged, the cost-to-quality ratio is smallest when the author's role is divided between a specialist subject expert, programmer, educationist, and graphic artist/photographer. storing the work of each professional separately allows the most to be made of each team member. it is as unfortunate for the programmer as for other team members that the information available about software tools is distorted by the hidden third-party code of surreal demonstra57 martin reynolds knowledgepro windows: the order of merit? tions, the hidden agenda of marketing hype and the hidden naivety of the eager but inexperienced author. however, using the nature of costs, benefits and effects as a guide, the relative merits of each product can be determined, and the effect that visual programming, object orientation, integrated development environments, multimedia and hypermedia can have on the quality of learners' environments can also be revealed. programmers have always used tools to help generate program code. this has been particularly true for code that represents the human-computer interface and its screen elements such as windows, buttons and boxes. in order to improve the quality of communication between the programmer and the other team members involved in the development of a product, this visual dimension to programming has been extended in some products to encompass the paradigm of interaction as well as visual appearance. visual programming tools provide the other team members with a structure with which to order the resources content. the programmer shares this structure, and in building an interface for the user is restricted to the tools paradigm of conversing with the content. when the other team members' intentions for the interaction match those imposed, the programmer will find that realizing the team members' wishes in code terms is straightforward. conversely, when there is a mismatch in intention, the programmer will find implementation difficult if not impossible. the more visual the tool, the more matched the team members' intentions and the tools' capabilities for interaction will need to be, and the fewer means the programmer has to specify the implementation. the implicit assumptions that visual programming tools contain about team members' intentions tend to reflect the structure of the programming language underlying the tool. knowledgepro and other products knowledgepro began as a dos product (kpdos) in 1986. after establishing itself in an artificial intelligence market, a windows version (kpwin) was released in 1990 when it won pc magazine's best product of 1990. in 1992, a capacity to generate c code was added (kpwin++). its three main competitors are now authorware professional, toolbook and visual basic. authorware professional (£800 educational) provides iconic flowcharting as the structure for authoring. it constrains interaction to a single instance, a single window of fixed size, and a linear pathway, and it represents its intentions by just 11 icons. it provides many variables that can be set to alter the way the program operates but, paradoxically, this warns of its limited ability to be programmed if your intentions do not match one of the 11 icons, beware! unlike the other products mentioned in this paper, it does not support all the standard windows screen elements. it requires write access to the directory in which it resides when running, and does not provide file-pointers or database functions, but rather encourages the hard coding of data in an application. iterative procedures are noticeably slow, and in not being able to act as dde (dynamic data exchange) server, halt while other applications run or parse more than 13 characters to other applications, authorware exhibits a limited ability to converse with other programs. a 3% royalty is payable on applications that are re-sold. this reflects the product's suitability for presentations or other passive applications whose production involves no team-work and no programmer, whose delivery involves no multitasking or networking, and whose future involves no maintenance (daeche and reynolds 1993). toolbook (£250 educational) adopts a form-filling approach to authoring which is especially suited to the production of electronic books. toolbook thus uses a book metaphor for its 58 alt-j volume 2 number 1 forms, though it would be misleading to suggest that this limits its use to linear presentations of passive text. the links between 'pages' can be displayed in iconic form and manipulated using the mouse one abstraction removed from authorware's iconization of screen elements. each toolbook page has a foreground and a background. in the foreground, screen elements are placed and sized by selecting them from mouse-driven toolboxes. in the background, toolbook's event-driven openscript language is used to define the reaction of screen elements. although background scripts can be shared between pages or between books, this represents only a limited ability to inherit or export variables and code. openscript has been optimized for the description of a user-interface rather than for the description of the consequences of a user's action. as if to underline this, toolbook provides a routine to exchange windows messages and openscript messaging so that it can easily front code for processes stored as dll (dynamic link libraries) or running as hidden or secondary programs, typically written with tools such as turbo pascal. openscript is syntactically simpler than basic but similar in structure it is reliant on event-driven equivalents of gotos and gosubs rather than function descriptions. visual basic (from £90 educational) provides a visual environment not unlike that of toolbook, but for a version of basic. for toolbox read custom control, although unlike toolbook there is no graphical way of connecting forms. double-clicking on a screen element placed on a visual basic form allows basic code to be entered which describes the form's reaction to the user-interface. as with toolbook and authorware, much of the code that will eventually make up the application the code that represents the custom-controlled screen elements remains unseen and uneditable. since it belongs to the basic family, visual basic is a typed language, and there are limitations on how code or data can be shared between forms. the assumption, as with toolbook, is that different forms will represent different content or jobs. it becomes awkward if not contrived to code anything other than simple datadriven applications. although toolboxes invite the amateur programmer, toolbook and visual basic are properly targeted at the professional who wants to get a quick prototype up and running and who wishes to take the advice of non-programming colleagues when it comes to screen design. knowledgepro (from £100 educational) uses a topic paradigm which, unlike the form or icon, is a definitive rather than a visual representation of the underlying knowledgepro language. there are tools provided to generate topics for screen elements, and definitives in the language aimed at fast prototyping. in function these are similar to authorware's icons or toolbook's toolboxes, but they have the advantage of generating readable and editable code. this makes knowledgepro accessible to the amateur programmer, yet by not imposing a structure, allows authors the freedom to define their own, typically by developing a shell for their courseware. the programmer can mix and match techniques to optimize the implementation's performance. it is implicitly assumed that the programmer may wish to use objectorientated programming, and artificial intelligences like backward chaining, neural networks and matched lists, yet is free not to do so. the limits of c on inheritance, persistence and encapsulation do not exist in kpwin, yet one can write c in-line with knowledgepro ++ code. however, knowledgpro is typeless, and syntactically closer to lisp than to c. ood (object orientated design) has been found to aid the process of communication between other team members and programmer more than other formal methods (graham 1991). it is particularly well suited to the evolutionary rather than revolutionary or revelationary development cycles. work in the educational field is almost exclusively co-operative and . 59 martin reynolds knowledgepro windows: the order of merit? evolutionary in nature, continual user-involvement refinement and maintenance being important factors. kpwin clearly supports ood, and in so doing maximizes the re-use of code and simplifies the maintenance of code that represents many similar but different situations. its strength in evolutionary development is underlined by the kpwin++ version's ability to generate c code integrally from the kpwin 4gl, and use standard c compilers to make and compile it. visual basic, toolbook and authorware are object-based rather than objectorientated and better suited to revolutionary and revelationary developments, in which substantial re-coding is necessary when an author's requirements change or when the speed of compiled rather than interpreted code is required (winblad et al 1990). as an application's complexity grows so does the demand for effective communication between programmer and other team members. concomitantly, the programmer will demand an increasing ability to optimize the code run. at one time, programmers would first have to write a program's code using one tool, then make it with another, link and run it with a third, and (if they were fortunate) debug it with a fourth. ides (integrated development environments) bring all these tools together in one interface and can vastly improve a programmer's productivity. as a corollary, costs can be reduced while the quality of code and the ease with which code can be maintained or changed can be increased. kpwin supplies an ide familiar to most c-level programmers one can trace or step through code as it is run, access context-sensitive help, build libraries of functions defined in knowledgepro to be re-used, and overload functions defined in kpwin with personalized implementations. technical support is available in the uk via knowledge garden (europe), and there is a uk user group. there are a number of code libraries available to extend the core kpwin language including dbase, clipper, q&e sql and archive. kpwin source code is provided for all the tools of the ide as well as the shareware multimedia tool textpro which exemplifies the replaying of digital video and animation. the toolbook and visual basic ides are extensible, but by the addition of custom controls or toolboxes rather than code libraries per se. toolboxes for standard screen elements, rudimentary graphics and text come as standard. ones for audio and video elements are available. visual basic (professional edition) comes with custom controls for deriving charts, graphs and tables from microsoft access databases or sql servers. the breadth of functionality represented in their third-party extensions is perhaps the greatest strength of visual basic and toolbook. both products sport string-handling functions, online help and debugging tools. but even with microsoft's support, visual basic cannot overcome the constraint of its underlying basic-like language. toolbook and visual basic seek to reconcile the difficulties in re-using code defined in their own terms by making the inclusion of functions written in more primitive languages relatively straightforward. authorware can be extended only with functions built into lower-level languages like c or pascal. consequently there are no comparable third-party libraries available. it has no debugger and no online help, nor any uk technical support from its producers macromedia. authorware does supply rudimentary tools to create simple graphics or animations, and icons to represent erase effects and video replay, though these encourage an amateurish approach to multimedia. successful multimedia avoids the pitfalls of inappropriate use by combining the experience of film makers, photographers, graphic artists and animators with that of a behaviourist, and the quality sought by these professionals is more likely to be realized through the custom controls available in toolbook and visual basic or the functions in kpwin which 60 alt-j volume 2 number 1 create elements and display those created by third-party applications (though it is worth mentioning that many multimedia applications remain undelivered because the cost of delivery is unsustainable). as for hypermedia, that subclass of multimedia which uses windows, icons and the mouse to link one phrase in a text or region in an art form with another in order to bring out an author's view of their relationship, some form of conversation with the learner needs to be facilitated, and some form of mutual adaptation needs to take place, in order to transform the 'hyperspace' into a useful learning environment (harri-augstein and thomas, 1991). although toolbook does have a hypertext toolbox, kpwin has a unique ability to create hyperlinks on the fly as well as from files and from program code. and it has a specialized set of commands for dealing with the list, tree and network structures. these aid the implementation of artificially intelligent or expert elements as adaptable models of learners' or teachers' meaning. kpwin's support for ood can be a great advantage to programmers asked to develop groupware, a means for groups to model meaning (baecker, 1993). ood complements the record locking and data compression of kpwin's dbase and archive extensions by aiding the implementation of programs which can run concurrently and across a network. in programming terms, kpwin is an ideal vehicle for the illustration of various techniques without the encumbrance of learning the syntax and paradigms of languages as divergent as clisp and c. conclusion the only certainty about the use of newer technology is that it is more expensive than older technology. educational effectiveness is dependent on use rather than its technological implementation. the total total amount of time spent using any tool represents only about 20% of the total time spent on a project, whatever the tool and however easy or complex the project, so that the choice of tool does not have much bearing on cost. but tools differ in what they allow one to do each has its own specialities and limitations. the choice of tool can therefore determine what one is able to produce, and so what benefit might be gained (pedlar and reynolds 1993). it is unlikely that any tool can replace the assistance of a specialist programmer unless the author's intention matches that of the tool exactly. the chance to buy additional libraries or toolkits may make a match more likely than in the past, but if there is a mismatch between the tools and the author's intention, or if multifarious, large, complex or distributed projects are contemplated, it is certain that knowledgepro's flexibility and maturity as a language put it ahead of toolbook, 'visual basic and authorware, in an order of merit which values cost savings and benefits without compromise on performance or maintenance. references baecker, r.m. (1993) (ed), readings in groupware and computer supported co-operative work, san mateo ca, morgan kaufmann. barker, p.g. (1989), 'knowledgepro: a review and assessment', engineering applications of artificial intelligence, 2, 4, 325-338. daeche, n. and reynolds, m. (1993), report on authoring tools for the electronic design education consortium, unpublished report, august 1993. graham, i. (1991), object-oriented methods, reading ma, addison-wesley. 61 martin reynolds knowledgepro windows: the order of merit? hale-shaw, r. (1992), 'the visual development environment more than just a pretty face?', pc magazine, 1, 6, 195-237. harri-augstein, s. and thomas, l. (1991), learning conversations, london, routledge. kulik, c. and kulik, j. (1991), 'effectiveness of computer-based instruction an updated analysis', computers in human behaviour, 7, 74-94. pedlar, j. and reynolds, m. (1993), the national oral surgery computer-assisted learning programme final report, unpublished report, nuffield foundation, february 1993. reynolds, m. and pedlar, j. (1993), 'choosing an authoring tool for a national cal programme', in proceedings of cal '93, london, pergamon (in press). winblad, a.l., edwards, s.d. and king, d.r. (1990), object-orientated software, reading ma, addison-wesley. addresses knowledge garden inc., 12-8 technology drive, setauket, ny 11733, usa. knowledge garden (europe) ltd., mount batton house, fairacres, windsor, berks, sl4 4le. ursula hayes knowledgepro user group peveit department of computer science the 'victoria university of manchester oxford road manchester m13 9pl 62 visualization of database structures for information retrieval annelise mark pejtersen, grete lisbjerg jensen, henrik jensen and steen weber ris0 national laboratory, roskilde, denmark abstract this paper describes the book house system, which is designed to support children's information retrieval in libraries as part of their education. it is a shareware program available on cd-rom or floppy disks, and comprises functionality for database searching as well as for classifying and storing book information in the database. the system concept is based on an understanding of children's domain structures and their capabilities for categorization of information needs in connection with their activities in schools, in school libraries or in public libraries. these structures are visualized in the interface by using metaphors and multimedia technology. through the use of text, images and animation, the book house encourages children even at a very early age to learn by doing in an enjoyable way, which plays on their previous experiences with computer games. both words and pictures can be used for searching; this makes the system suitable for all age groups. even children who have not yet learned to read properly can, by selecting pictures, search for and find those books they would like to have read aloud. thus, at the very beginning of their school life, they can learn to search for books on their own. for the library community, such a system will provide an extended service which will increase the number of children's own searches and also improve the relevance, quality and utilization of the book collections in the libraries. a market research report on the need for an annual indexing service for books in the book house format is in preparation by the danish library centre a/s. 1. introduction it is generally agreed that current information retrieval systems in both public and school libraries present major problems for child users. these difficulties are particularly perplexing today when information-retrieval skills play a major role both in education and, increasingly, in society as a whole. one of the barriers against providing children with information retrieval skills is the lack of programs for casual and novice users; other barriers exist between public libraries, school libraries and education in schools. the purpose of this paper is to describe how a new library system the book house based 50 alt j volume 2 number 2 on a perspective of end users' needs, can provide a major contribution to improving education as well as the utilization of libraries in schools. in this area, many institutional borders and professional barriers have given rise to artificial distinctions among what in reality are related disciplines public libraries, school libraries and education in schools, though from an organizational viewpoint, these distinctions are already disappearing. in many countries, no physical distinction can be found between public libraries, school libraries and schools. public libraries can be placed in schools and school libraries can be located in public libraries as well as in schools (amy, 1987; sandvik, 1988; valentine and nelson, 1988). currently, in most european countries, widespread co-operation exists between school libraries and public-library work, and in services for the preparation, distribution and circulation of library materials (kinnell, 1992). yet from the viewpoint of computerized library systems, a distinction is usually made between system concepts for (a) public libraries, the so-called online public access catalogues; (b) school library systems, which are usually quite close to public-library system concepts; (c) computer programs used for training students in computer literacy; and (d) teaching material for education in the use of libraries. from the users' point of view, no such distinctions between institutions and systems can be justified, inasmuch as their information needs are motivated from their tasks which, to be solved properly, often require information retrieval across institutional barriers, rigid subject domains and school disciplines. interviews with children in libraries have shown that their sources and motivations for information retrieval are a mixture of needs and ideas coming from schools, leisure activities, school libraries, public libraries and mass media; the expected outcome from the use of material from a search is connected to an overlapping mixture of tasks, such as solving school problems, gaining new knowledge and satisfying needs for various reading experiences (pejtersen, 1986 a and b). hence, the book house system described in this paper is conceived from an understanding of the users' task-oriented perspective, their information needs and queries, and the levels of skill needed in information retrieval and computer use. there turns out to be a great commonalty in children's information-retrieval behaviour and skills across school and library institutions, which has been taken into account in the design. as a consequence, the book house system has proven to be efficient as a tool for both education and support of information-retrieval tasks in all the above mentioned contexts of public libraries, school libraries and classroom work. the system encourages an integrated and interrelated approach. 2. school library use in education due to the many developments and innovations in teaching methods, the tasks to be performed by the school library, and expectations about its support of pedagogical activities, have gained considerable attention. project-related teaching, and work in groups around shifting specific themes and subjects, have forced students to change their way of working, and the school library has had to do likewise in order to support this change. projects and theme work are often cross-disciplinary in their subject matter, and involve several school subjects. in the higher grades and levels, themes are sometimes 51 annelise mark pejtersen et al visualization of database structures chosen by the pupils themselves. thus information retrieval has become a necessary work activity for students as part of this expanded project and theme work. they must learn to use the school library in the classroom, and gain easy and efficient access to up-to-date information. in many countries, school libraries have become the central place for pupils' information-seeking tasks. in keeping with the times, the school library must be able to provide students their own easy access to electronic information-retrieval (ir) systems with comprehensive database coverage. current emphasis on key phrases, such as selfaction, teach and use and learn by doing, reflect the approach under development in all modern pedagogical activities having to do with school libraries. yet despite the widespread introduction of computers in modern education programmes, school libraries have until now lacked a tool which could be used by the students themselves. 3. teaching computer use and skills recently, it has become mandatory in many european schools to integrate education in computing into the curriculum of the different disciplines taught. a shift in emphasis has taken place from teaching computing as a single isolated domain towards teaching how computers can be used to support problem-solving in different kinds of tasks and settings. several implications arise from this shift: (a) integration of teaching the use of computers into traditional subjects, such as reading and learning about the mother tongue, social and natural sciences, etc., leads to their use as a natural tool for problem-solving activities related to learning the subject content of a domain. (b) teaching computing to provide knowledge about the application of computers in specific task-functions and in actual work-domains implies exceeding borderlines between the subjects taught in the classroom context and actual work places outside the school. programs are therefore needed which can serve both as the student's own learning tool for solving problems, writing papers and exercises, as well as a professional support vehicle in a work-domain familiar to students or closely related to children's' work in the classroom. 4. children's problems with catalogues several problems are encountered by children in connection with their comprehension and full exploitation of a computer system for retrieving information. such systems were originally developed to meet librarians' needs for keeping records of their book-stock by providing computerized access to automated card catalogues. today, they are used as a collection of access tools supporting the complete range of disciplines relevant to a library, as well as self-service information-seeking support tools for untrained and inexperienced casual library users. despite continuing re-designs, including the incorporation of online user-assistance in the form of menus, help displays, informative error messages and browsing facilities, etc., it is generally agreed that there still exist major problems for users who are not experts in a field or experienced searchers, or are not generally familiar with how libraries work (hancock-beaulieu, 1989; hildreth, 1989; markey, 1990, pejtersen, 1992 a and b). 52 alt j volume 2 number 2 these problems are mainly related to the large number of mapping processes required of the user during the search process in order to convert and translate his/her knowledge structures and language in accordance with the constraints imposed by the search language and classification schemes used both to classify and to retrieve information in ir systems. the only stable structure typically available in ir systems to guide users is the hierarchical structure of a thesaurus and/or classification scheme, which will give most youngsters problems. search languages require advanced verbal skills, and classification schemes require advanced categorizing skills and/or domain knowledge obtainable only through exhaustive training. the use of ir systems requires capabilities for making appropriate deductions and inferences about the hierarchical category memberships of the user's own need, as well as formulating and modifying systematic combinatorial searches in generic terms. although some work indicates that, at the age of four, children's language capability includes some categorical structures (borgman et ah, 1989), and that, at an early age, they do have some ability to process hierarchically structured information (keil, 1979), they do not possess skills for making hypotheses and predictions in order to plan a search successfully in conventional information systems. piaget's work (piaget, 1958) suggests that children's capabilities for structural thinking and abstract reasoning can be divided into several developmental stages. three of these have been of relevance in the design context of the book house for young users from age six to sixteen: (a) the intuitive perceptually based stage at which children from four to seven think in terms of analogues, and rely on sometimes systematic trial-and-error approaches; (b) the concrete operational stage at which children from seven to eleven possess an understanding of logical relations and can experiment in a systematic way with some of the operational structures needed for classification; and (c) the operational stage at which children from eleven to fifteen can think in terms of abstract relations and classes, and make exhaustive hypotheses and deductions completely independently of perceptually based intuitions. independent of discussions of specific details of piaget's work (boden, 1980; carey, 1985), it is obvious that the capability for making appropriate deductive inferences based on category membership, the understanding of formal logic structures and the use of welldefined categories will not be available skills in young children to the extent needed in order to operate within a comprehensive, formal library-classification scheme with its associated artificial and standardized retrieval language. even for older children who have well-developed abilities for making deductive inferences from hierarchical library schemes, operational use of these schemes will require domain knowledge beyond their mental capacity. in short, online catalogues require different types of knowledge and skills that children do not yet possess, in spite of their growing experience with personal computers, and familiarity with other modern media all of which have contributed to their motivation for using a compatible form for computerized ir. as a result, usable ir systems must not only be competitive with existing, familiar computer programs and systems accepted by children, but must also minimize the need for excessively demanding mental juggling among incompatible languages and categories. 53 annelise mark pejtersen et al visualization of database structures 5. mapping database structures to children's categorization skills given these user-characteristics, many conversion and mapping problems can be avoided by using a classification scheme and a database structure which are compatible with children's ways of categorizing information. as opposed to hierarchical library classification models, which are too cognitively demanding to be learned and understood well enough by children, such a system will make it possible to make database structures and representations of book content correspond directly to children's intuitive ways of categorizing their needs. an analysis of 200 protocols reflecting children's search behaviour and ways of categorizing the domain of books which were recorded in ten different types of libraries revealed structures and other aspects different from traditional library classification schemes (pejtersen, 1986 a and b, 1992 a and b). these were used as the basis for the classification scheme and database structure, mentioned in paragraph 7.6 below, in the book house system. since a great variety of cognitive skills in the categorization of information is likely to be found among children, the book house system was designed to: (a) match children's different developmental stages and problem-solving skills by supporting both young children's trial-and-error and intuitive perception-based searches, and older children's more or less systematic use of abstract relations and classes, in searching in a stable database structure compatible with their ways of categorizing information (this also guides children's learning of domain knowledge); (b) extend their categorization skills by several different interface representations of the classification structure used in the database each placing different requirements on users' intellectual skills and domain knowledge. each representation needs an interface content of classification categories and book description that is understandable by children at varying levels of perceptual and cognitive skills and training and, in addition, an interface form that is familiar enough to children to speed up their learning to search in abstract classification categories across the abovementioned developmental stages. in section 7.3 to 7.6 below, four different interface representations of the same database content are shown, ranging from a graphical display of the complete classification structure to displays of a structured textual description and unstructured displays of icons. 6. visualization of database with icons and metaphors the degree of difficulty involved in problem-solving activities will not only depend on sufficiently well-developed innate structures needed for operational thinking in piaget's terms, but also on the characteristics of the task, its requirements to precision, degree of detail and, of course, on the user's familiarity with the task situation. it is well known that everyday thinking is more likely to rely on analogies with past examples and rules of thumb than on logical abstract reasoning within clear and well-formulated categories (rasmussen, 1986). this implies that interface displays should take advantage of these abilities by using icons or pictures with reference to familiar, preferably, concrete objects, which tap familiar situations and prompt common-sense reasoning. first, icons used as metaphors can enable the user to transfer intuition and skill from a familiar situation in another context to the functionality of the ir system. in order fully 54 altj volume 2 number 2 to support users' ability to make analogies to familiar objects and events, icons should represent both the database system, the system's retrieval functionality including options for user actions, and the database content. usually, the interface menus of ir systems do not reveal much of the basic semantic structure of the database content even if they do, users will often still be in trouble since the structure used for organizing the semantics of the database content will most likely not match their own way of structuring the concepts used to reflect their reading need. genuine support is best given by providing a visualization of the structure imposed on the database content through several types of icon reflecting the categories and principles actually used intuitively by children. the book house interface is tailored to children's abilities to categorize and understand concepts, as determined from their associative responses to pictures and words in a controlled, multiple-choice association test (pejtersen, 1990). secondly, it is well known that pictures are easier and faster to learn than text. they can also be perceived independently of the users' language capabilities. they are faster to 'read' than text since one icon can communicate very complex messages that would take much language to explain a picture can say much more than a thousand words. pictures can thus efficiently express subject information and conceptual relationships. they are easier to remember and to recognize once learned, which is advantageous since recognition of information is easier than recall of information. visually, we can process a far larger amount of information than in any other form. we can, for instance, recognize a well-known face in a crowd, while we might have problems in recalling and verbally describing this face. in the same way, it is easier to use a map of a town than process a verbal instruction. 7. book house search: database retrieval there are two entries into the book house system: book house search for searching information from the database, and book house write for indexing and storing new information in the database. book house search supports several repetitive decision tasks: formulation of a user's problem and need, choice of search strategy, execution of search, evaluation of relevance of retrieved books compared with user-need, revision of search, etc. the book house assists children's accomplishment of retrieval and indexing tasks without assistance from adults. several types of help are available, as well as facilities for experimenting and generally playing around with the system in a trial-and-error mode. 7.1 book house metaphor as they learn, children are likely to try to build a mental model of their problem-space and its representation in the database system in order to understand and predict the content and functionality of the system. spavold (spavold, 1990) found that children would spontaneously make images of databases during their use in the classroom. our evaluation studies offer ample evidence that the choice of a book house metaphor enables children to make easy analogies between their everyday life experiences and corresponding objects and tools in a house, and their functionality (goodstein and pejtersen, 1989). even very young children have been able to use the system and achieve successful results. when the user explores the screen with the mouse, a text line at the bottom follows the 55 annelise mark pejtersen et ah visualization of database structures figure 1 mouse movements and gives two kinds of messages. if the mouse is not in an area with selectable objects, the text 'move the mouse and see what you can do' appears. when the mouse points at a selectable object, a square is drawn around the icon to evoke the user's attention and to indicate the mouse-sensitive limits of the object. a message appears telling the user to press the mouse button, with information on what will happen next. these help texts support interactive, self-activated 'learning by doing' during the user's exploration of information on the screen. they also support first-time users in understandmg the meaning of each icon: prototype testing with users has shown that not all children will be able to interpret the meaning of icons and predict a unique retrieval action associated with a particular icon. 7.2 a room for database choice figure 2 56 altj volume 2 number 2 the book house system contains a database with children's fiction, a database with children's non-fiction, and a database that combines fiction and non-fiction. 7.3 a room for choice of search strategy figure 3 the four strategies are (figure 3, from the left): (a) the associate search by analogy based on prototypes for books similar to the book the user has in mind; (b) the intuitive strategy of browsing through icons for recognizing a perhaps unidentified need by means of pictures on a 'flip-over' projector; (c) the analytical subject search strategy represented as objects on a work table; and (d) browsing shelves, which give access to a browsing through the book stock strategy as a means for getting inspiration on what to read. 7.4 find books similar to a known book when a search by analogy is chosen from the left side of the strategy room, a model book figure 4 57 annelise mark pejtersen et al. visualization of database structures is displayed with two options on the front cover: author and title. based on the user's specification of his/her model book, the system uses the corresponding model-book description on the screen as the basis for finding other similar books in the database by means of probabilistic techniques. 7.5 browse album with picture association thesaurus figures children often use pictures on the front cover in their searches, as well as in their assessment of the relevance and value of content of books. in a significant extension of this, users can browse and search for books in the picture association thesaurus by selecting among icons symbolizing the subject content of books. when searching via an icon, book meanings normally mediated by many keywords become available. such a pictorial representation produces only associatively related concepts displayed at levels corresponding to a broad range of more or less well-defined categorization skills. no verbal skills are needed to search by selection of an icon with a mouse. 7.6 a room with work table for classification subject search figure 6 58 altj volume 2 number 2 requiring knowledge about the names of fields in database records when formulating a precise search query can be avoided by making the structure of the database visible using appropriate icons (figure 6). the poster on the wall gives access to subject-matter keywords: books about family life, quarrels, anorexia nervosa, happiness, etc. the world globe on the table and the clock on the wall give access to keywords on time and geographical places such as books about teenagers' life in los angeles in the nineties. the view out of the window gives access to keywords about the environment, social and professional settings, for example books about sport among jewish children living in busy families in big cities. the masks on the wall give access to keywords related to emotional experiences, such as curiosity or excitement, evoked in the reader during the reading process. the icon showing the author sitting with his typewriter gives access to keywords concerning the author's intention to put forward ideas and/or opinions. the glasses on the table give access to keywords on the accessibility of books, their readability, language level, the age for being read aloud or read by the child him/herself, the size of the typography, etc. the card-catalogue drawers give access to keywords on traditional genres such as war stories or suspense stories. 7.7 browse through book descriptions find subject; seebookdescri place setting time information experience genre readability edition ingemarm, b.s. erik menveds the murder of king erik the nobleman being murdered by childhood 231 pages. erik dipping a finderup lade. pet a hcsselaad his allegiance to the king, who is rebellious noblemen lead by marskstig. upper cliss. kings 1288-**. a description of exiting. novel biography. difficult to read. 1982. seriil court. a! egiance and patriotism. historical novel. normal. ad it b o o t noblemen figure 7 when the icon browsing through book shelves is chosen in the right side of the strategy room, the system pops up with randomly chosen book descriptions and information on book content. each time this strategy is activated, a new set of book descriptions is used (see book description immediately below in 7.8). 7.8 open book with book description book descriptions are structures according to the classification system to enable the user to see the correspondence between his/her selected category of terms and the contents of each book. the user can then browse and turn pages one after the other in both directions by clicking on the lower red corners of the book showing one arrow. the user can modify 59 annelise mark pejtersen et at visualization of database structures the current need by adding search terms through a selection of red keywords from the book description, or removing terms with the eraser icon. when selected with the mouse, they are automatically combined with boolean operators. the book case shows the number of retrieved books. 7.9 search example a realistic search query culled from our user studies illustrates a typical use of the system. a child of twelve needs to find books for her natural science class; she wants to choose a topic for group work on 'pollution'. she wants to find 'an exciting book with a blue cover and with a moon and whales on the front cover*. she does not remember the title or the author's name, since the source of information about the book is a friend from another class who showed it to her. her motivation for looking for this particular item is the recommendation by her friend, who had been told by her teacher to read this book on pollution for a mother-tongue lesson. she selects the door action icon and enters the book house. she selects the room in the middle to search in the database with fiction and non-fiction for children. having a fairly good idea about the characteristics of the desired book, she chooses the work table with classification access. first, she selects the masks on the wall to search for exciting books, then the book on the table to search for colours and pictures on the front cover. she looks at the result of the search shown as book descriptions. having checked the emotional experience 'exciting' to be gamed by reading the books and the cover designs 'blue, moon and whales', she realizes that the subject is on 'whales at the point of extinction due to pollution of the sea'. she decides to take a print-out and uses the printer icon. she continues to browse retrieved books and uses the book in hand icon to put aside interesting candidates to be printed at the end of the search. 8. book house write: classification and editing the database new books can be added to the database and existing books can be edited. a typical task of storing information about a book will involve indexing, i.e. the analysis and representation of the book to be searched for and displayed in the book house search system. this activity includes: check whether the book is already in the database; if not, select a card to fill in a book description; skim the book, possible reviews and other material; formulate a new description using standard categories; save the book description in the database, which is then updated. 8.1 blank card for a new book when adding new descriptions to the database, the blue new book card catalogue is chosen, with blank cards and labels for empty categories ready to be filled in. when editing existing book descriptions, the yellow titles catalogue or the green authors catalogue gives access to a known item. the red all catalogue gives access to browsing through all the books in the database for examples of book descriptions. this is a useful option relevant in unfamiliar situations since it provides inspiration on how to solve a classification problem. the basic principle is that a book description can be made with very little effort; and 60 altj volume 2 number 2 ttboo iktctolbtc l*t>*"*^~-< * * ' ^ * y ^ s ^ « * figure 8 practically no knowledge about classification or indexing is needed. the intention has been to make children progress gradually in their learning about the storage of information in the database. as in book house search, help is provided by a text line at the bottom of the screen informing the user about what to do by showing an example of a book description typical for the category touched with the mouse. the life belt icon activates help by giving an example of a fully indexed book and, if desired, detailed information about the purpose and principles that need to be applied when indexing within varied book categories. finally, a written folder is available on indexing options. 8.2 card with book description from the database figure 9 changing descriptions of books already stored in the database will require the user to fetch the book from the database by selecting the action icons of author or title card catalogues at the top of the screen. a yellow card filled in with book descriptions already 61 annelise mark pejtersen et al. visualization of database structures saved in the database is then displayed, and the user can now work with this card in exactly the same way as with the white card. 8.3 classification example a school class in the 6th grade (children of about 11 years old) at the peder lykke school in copenhagen had a one-month project in which the mother-tongue teacher co-operated with the school librarian. the overall purpose of the project was to teach children to review books and to prepare their reviews by means of a computer for subsequent database retrieval. children chose books from the school library according to their own interests, and their task was to make a book review of their own books for the benefit of the other children in the class. they prepared their review on a paper form corresponding to the blank card for new books, and later used the card in the book house to store their reviews. this took about is minutes. teaching in advance of and during the project involved an introduction to literary text analysis by explaining the meaning of the categories in the form. they were also taught about how to prepare data for information retrieval through the selection of the keywords necessary to communicate the most important aspects of a book. the class created a database with 45 books, and enjoyed retrieving each others' books as well as accessing another database, which they found, created by a 9th grade class (children of about 14 years old) (jensen and acker, 1992 a, b andc). 9. integrating ir and computer use in libraries and schools recently, many efforts have been made to integrate the traditional concept of a library as a place for information retrieval of books with the role of the school in the training of young people to be skilled and knowledgeable in the use of computers. one approach is the extension of the library concept into a media centre in public and school libraries through the development of systems comprising all types of media: multimedia technologies, computer programs, video and tape recordings, etc., and building a collection of programs for loan in the media library. several experiments have been carried out to try to integrate school-teaching of traditional subjects with teaching about work in a library, i.e. its book collection and its use. for instance, this has been done by training young people in book-acquisition tasks, and having them overtake the actual task of buying books for the library for a couple of months during mother-tongue classes. through the review process, they learned to generate literary-quality assessments, and to formulate their reading experiences. children actually learned the trade-off involved in carrying out real-life tasks, and they enjoyed for once not being encumbered by the usual 'let's pretend' type of project. activating co-operation between school children, teachers and librarians in serious, reallife tasks had positive effects on children's reading habits and their use of the library (weinrich 1991; hussmann, 1992). such initiatives can continue throughout life-long educational activities, and can carry with them an active contact with the local community and its organizations involved in communicating knowledge (kinnell, 1992). working with the book house in integrated activities in both a library and a school setting has given many valuable experiences about the effects of the system on children. apart from the obvious purpose of teaching computer skills, there is evidence that the system has other advantageous effects: 62 alt j volume 2 number 2 • it encourages children to do their own searches in public and school libraries, and to do them more effectively, in direct response to their daily information needs for the preparation of class-work and leisure activities. the system has much in common with computer games and is perceived by children to be as exciting a challenge. it is easy to use for the first-time user, but is not tedious or too simple for expert users. with the current database of children's books, it is suitable for many different types of subject searches. the browse-through-icons option is particularly useful for children who do not master the reading of lists of keywords. that this is feasible has been demonstrated by the book house evaluation process, where children proved to be more than enthusiastic about doing their own searches on computers in libraries (goodstein and pejtersen, 1989). • it increases the number of books read, and widens reading tastes. evaluation of the book house system in libraries, and results from the experiments mentioned here, clearly indicate that use of the book house in school libraries and in class work will increase the number of books read and the number of visits to the library, change children's reading patterns, and widen their taste as they become aware of hitherto neglected literature. special care was devoted to the development of a database for fiction subject-retrieval in order to increase the use of the fiction stock in public and school libraries. indexing of specific subjects in fiction is usually not a service offered to libraries, mainly due to the many problems that have been encountered in earlier attempts to develop classification schemes for fiction. the retrieval and promotion of fiction have become even more difficult because of the high number of new published books each year, since (ideally speaking) these have to be read and remembered by the librarians who are the traditional mediators of borrowed fiction. not all books in the library stock can be read, nor remembered when read, during actual dialogues with users. • it teaches how to formulate and solve an information-retrieval problem, improving the education and training of children in storing and retrieving information via a uniform platform, i.e., a system common for both the teaching situation in the classroom and the retrieval situation in the library. book house write can be an important facility for unking language from text analysis and evaluation to language used for database development and information retrieval. dealing with terminology problems between school class-work and database language has been reported by borgman (1991) as an important problem. children from the 4th grade (nine years old) can make their own database, and search on their own book descriptions with automatically converted red keywords, without having to go through all the categories and fields of a blank card. only author and title are mandatory. children's subjective experiences and principles adopted for the classification of a book can be communicated to other children in a special field for personal comments. older children can try to develop a controlled vocabulary and begin to understand the differences between free, natural-language text and a controlled vocabulary, as well as the important roles of precision and consistency in handling language in connection with database construction. this experience will not only increase knowledge about the library; it can be generalized to the use of computers for ir and indexing in many other fields than the library. experiments by spavold (1990) involving the use of a database in classroom teaching confirmed that children who used it both for data input and for output/retrieval understood the system's functionality and search 63 annelise mark pejtersen et at visualization of database structures procedures better, and could formulate more diverse queries, than those children who used the database only for searching. it helped in the learning of interrogation and formulation of good queries. most children enjoyed both coding and data entry, as well as searching for information. there were no preferences for more problem-solving activity (involved in retrieval) as contrasted with the more repetitive, routine and yet highprecision task involved in the preparation and coding of data. this has been confirmed in experiments with the book house system. • it improves skills for text analysis and evaluation. two traditional goals which teachers and school librarians have in common can control the use of the book house: to motivate children to read books of a quality not typical for their age group, and to teach children literary analysis as a basis for generating book reviews. work then takes on a practical goal, and has the effect that is physically visible and useful for the whole class. in a book house search and write project for the 9th grade (children of about 14 years old), the teacher and school librarian chose 20 normally not very popular books, a mixture of adult books that might be read by teenagers and books of high quality for teenagers. they asked the students to classify these books for the book house database. the purpose was not only to draw their attention to these books, but also to create a database for retrieval in the school library that would draw the attention of the children of the same age for these particular types of book. thus the database serves as a communication tool which widens children's reading perspectives, and broadens their view from purely personal reading experiences to more of a critical-analysis approach. as a tool for the analysis of books from about the 4th to 10th grade (children from 9 to about 16), the classification categories can be used to prompt a number of questions about a book. • it supports cross-disciplinary teaching, and the existing custom in education of integrating fiction and non-fiction in those subject fields where such an integration is appropriate for instance, history, social science, natural science by allowing similar and specific subject accesses to both fiction and non-fiction. 10. present and future the book house is now available as two shareware programs. one is for the retrieval and indexing of fiction for children and adults in public libraries, with a database content of 3,500 novels from different countries which have been translated into danish. the second is for school libraries, and covers the retrieval and indexing of both fiction and nonfiction with a database containing both children's fiction and non-fiction. it was released in the summer of. 1993 by apple on a cd-rom, and by ris0 on floppy disks, and will be distributed to scandinavian school libraries in the near future. the danish library centre a/s, which provides danish libraries with bibliographical data, will conduct market research this autumn, 1994, on the number of libraries that will subscribe to a centralized indexing service which supplies books with new data in a format that matches the classification of books in the book house database. if the response from libraries is sufficient in number, the delivery of data to the book house system will become a service similar to other data deliverables to danish libraries. the project is currently being continued by extending the use of multimedia, and by the development of an adaptive, multimedia interface including interactive user-modelling. in addition, the book house 64 altj volume 2 number 2 project is being expanded to a nordic book house system with participation from the other nordic countries. acknowledgements a large number of people from the library community, research institutes and industrial companies in denmark are gratefully acknowledged for their important and influential contributions to the book house system design, software development in c++, indexing, graphics and evaluation: steen agger, len goodstein, jutta austin, inge cramer, finn nielsen, peter speck, lena lundgren, ringa sandelin, steffan villumsen, henrik gren, and bjarne acker. institutions and funding: ris0 national laboratory, the royal academy of arts, peder lykke school, the royal school of librarianship, hjortespring public library, apple computer a/s (tom jacobsgard), the educational university (kurt hartvig, gunnar jacobsen), the national foundation for public and school libraries, and the council of nordic ministers. a special thanks to l.p. goodstein for making the english more readable. references amy, l. j. (ed.) (1987), combining libraries: the canadian and australian experience, dalhousie university, canada, school of library and information studies. boden, m.a. (1980), jean piaget, london, penguin (modern masters series). borgman, c.l. (1991), 'designing an information retrieval interface based on children's categorization of knowledge', proceedings of the 11th international online information conference, london, 1992. borgman, c.l., chignel, m. and valdez, f. (1989) (eds), proceedings of the 52nd annual meeting of the american society for information science (washington, dc), 26, 81-95. carey (1985), conceptual change in childhood, cambridge, ma, usa, mit press. goodstein l.p. and pejtersen a.m. (1989), the book house. system functionality and evaluation, ris0 national laboratory, denmark (riso-m-2793). hancock-beaulieu, m. (1989), 'evaluating the impact of an on-line public library catalogue on subject searching behavior at the catalog and at the shelves', journal of documentation, 46, 4. hildreth, c.r. (ed.) (1989), the on-line catalogue: development and directions, london, library associations. hildreth, c.r. (1991), intelligent interfaces and retrieval methods for subject searching in bibliographic retrieval systems, washington, dc, usa, k.g. saur. hussmann, h. (1992), children's selection of books, børnebiblioteket sundbyvester, denmark. jensen, g.l. and acker, b. (1992a), søg og du skalfinde (search and find. a program for teaching fiction in schools), bom og beger, no. 7, pp. 21-4. jensen, g.l. and acker (1992b), the book house in the school library, nyt om data i skolan, no. 2, pp. 10-13. 65 annelise mark pejtersen et al. visualization of database structures jensen, g.l. and acker, b. (1992c), the book house in the school library, skolebiblioteket, no. 7, pp. 16-20. keil (1979), semantic and conceptual development: an ontological perspective, cambridge ma, usa, harvard university press. kinnel, m. (1992), 'change management for school library services' library management, 13 3, 10-19. lundgren, l. (1993), how to find books in the library, stockholms stadsbibliotek. markey, k. (1990), 'experiences with on-line catalogues in the usa using a classification system as a subject searching tool', tools for knowledge organization and the human interface. pejtersen, a.m. (1980), 'design of a classification scheme for fiction based on an analysis of actual user-librarian communication, and use of the scheme for control of librarians' search strategies', harbo, o. and kajberg, l., theory and application of information research, london, mansell. pejtersen, a.m. (1984), 'design of a computer-aided user-system dialog based on an analysis of users' search behavior', social science information studies, 4, 167-83. pejtersen, a.m. (1986a), 'design and test of a database for fiction based on an analysis of children's search behaviour', in ingwersen, p., kajberg, l., and pejtersen, a.m. (eds), information technology and information use. towards a unified view of information and information technology, london, taylor graham, 125-47. pejtersen, a.m. (1986b), 'implications of users' value perception for the design of a bibliographic retrieval system', in agrawal, j.c. and zunde, p. (eds), empirical foundation of information and software science, new york and london, plenum press. pejtersen, a.m. (1988), 'search strategies and data base design' in goodstein, l.p., andersen, h.b. and olsen, s.e. (eds), tasks, errors and models, london, taylor and francis. pejtersen, a.m. (1989), 'a library system for information retrieval based on a cognitive task analysis and supported by an icon based interface' acm, sigir conference proceedings, boston, usa. pejtersen, a.m. (1990), interfaces based on associative semantics for browsing in information retrieval, ris0 national laboratory, denmark, ris0 m-2794. pejtersen, a.m. (1992a), 'new model for multimedia interfaces to online public access catalogues', the electronic library, 10, 6. pejtersen, a.m. (1992b), 'the book house. an icon based database system for fiction retrieval in public libraries' in cronin, b. (ed.), the marketing of library information services, 2, london, aslib, 572-91. pejtersen, a.m., and austin, j. (1984), 'fiction retrieval: experimental design and evaluation of a search system based on users' value criteria' (parts 1 and 2), journal of documentation, 39, 4, 230-246 and 40, 1, 25-35. pejtersen, a.m., olsen, sv. e. and zunde, p. (1987), 'a retrieval aid for browsing strategies in bibliographic databases based on users' associative semantics: a term 66 altj volume 2 number 2 association thesaurus', in wormell, i. (ed.), knowledge engineering: expert systems and information retrieval. london, taylor graham. pejtersen, a.m., villumsen, steffan, weber, steen (1992), the book house in the school library, riso national laboratory, denmark, riso r-632 (in danish). piaget, j. (1958), the growth of logical thinking: from childhood to adolescence, new york, basic books. rasmussen (1986), cognitive engineering and information processing, amsterdam, north holland. sandvik, a. (1988), learning in information society, report from the nordinfo conference on the design and use of information-retrieval systems in primary schools and libraries. spavold, j. (1990), 'the child as naive user: a study of database use with young children', international journal of man-machine studies, 32, 603-25. valentine, p. and nelson, b. (1988), sneaky teaching. the role of the school librarian, london, british library (lirr 63). walker, p. and hancockbeaulieu, m. (1991), okapi at city: an evaluation facility for interactive ir, london, british library research report 6056. weinrich, t. (1991), children buy books, denmark, høby biblioteksbutik. 67 educational software design: applying models of learning stephen richards university of teesside numerous psychological models exist which aim to explain how people learn, but the knowledge from educational theory is often missing from the design of computer-based learning applications. this may stem from the abstract and complex nature of many learning models. in order to address this problem, there is a need for simplified models of learning which include guidelines indicating how such models can be implemented in courseware design. this paper identifies such a model, and presents a series of guidelines intended to enable courseware designers to apply educational theory to the practical design of quality computer-based learning materials. the 'spreading ripples' model of learning the model of learning adopted within this paper is the 'spreading ripples' (sr) model proposed by race (1994). this model was chosen for two important reasons. first, it makes use of accessible ideas and language, .and is therefore simple. second, .race suggests that the model can be used in the design, of educational and training programmes (and can thereby be applied to the design of computer-based learning materials). there are four key terms central to the sr model: wanting, doing, feedback and digesting. wanting. race uses the term wanting in place of the term motivation which is more, often used by psychologists. he argues that wanting implies much more than motivation since it lies at the centre of 'human urges'. when individuals attempt to learn, and are also highly motivated, excellent results are often possible. however, when wanting is lacking it is unlikely that efficient learning will take place. doing. when individuals are asked how they learn, very few refer to being taught or trained. most refer insteadto the process of 'doing' in some form or other. race sums up these responses into four statements of active learning: practice, doing, trial and error and learning from mistakes. here the emphasis is firmly on the activity of the learner. feedback. an important issue in all human behaviour, including learning, relates to 17 stephen richards educational software design: applying models of learning feelings (mueller, 1975). when individuals have positive feelings, they are likely to be capable of more effective learningan issue often overlooked by learning theorists (race, 1994). it is therefore important to understand how people most frequently obtain positive feelings about their behaviour so that attempts can be made to produce the same positive feelings when they are involved in learning behaviours. when individuals are asked about where they obtain positive feelings from, a typical pattern of responses emerges. few claim that the feelings come from within; most obtain their positive feelings from the comments made about their behaviour by others. in other words, people need feedback about their behaviour in order to feel good about it. feedback is therefore an important mechanism by which positive feelings towards learning processes can be engendered. digesting. when discussing the processes that learners undergo in order to make sense of new information, educational psychologists often use terms such as reflection and reflective observation (kolb, 1984). the term digesting is used by race as an everyday alternative to these terms. for effective learning to take place, the learner must not only take in new information, but must also sort out what the important points were and extract or formulate any underlying principles that may exist. it is only by actively processing information in the context of previous knowledge that the new information can be effectively assimilated. guidelines for designers from all this, eight guidelines can be produced, two for each of the learning processes identified by the model. wanting guidelines / 1. wanting must be consciously programmed into courseware. an assumption often made in higher education is that students already possess the wanting, and that this is evident simply by virtue of the fact that they are undertaking advanced education as a matter of choice. in reality, wanting is often lacking for some students. if motivational issues are not addressed directly, some will fail to make adequate use of the learning materials provided. 2. two basic ways exist for embedding wanting: increased value and increased enjoyment. within all areas of learning, there are aspects of the subject matter that students will find uninteresting, and others that they will find enthralling. unfortunately, although trends can be identified, the areas in which students are intrinsically motivated vary from one student to another. for this reason, it is important to provide embedded motivation. increased value can provide that motivation by showing students the purpose of studying each aspect of a course, and also a schema of how the various aspects of the course link together. in this way, topics are no longer seen as isolated items, and the value placed on each topic by students can be increased. motivation can also be increased by trying to present the learning in a more enjoyable format. this can be achieved in a number of ways, including the use of multimedia and the use of game-playing strategies. extreme care must be taken, however, not unnecessarily to embed expensive multimedia resources onto trivialize the learning experience by an inappropriate use of the game approach. 18 alt-j volume 4 number 3 doing guidelines 1. avoid passive interaction. a problem found in many computer-based learning applications is a reliance on the relatively passive interaction involved in page turning. here the user is not interacting with the information in any meaningful sense. instead, the user's primary task becomes the monotonous and repeated selection of a 'next-page' icon. interestingly, it seems to be a particular problem in those applications which use digital video as an informationpresentation medium. it seems that the inclusion of high-cost video resources, which improve the aesthetic appearance of learning products, has led to a de-emphasis of the learning design. a return to more meaningful interactions that stimulate active thinking processes is required (for example, investigating interrelationships between concepts, and controlling simulations). l.^support practice and/or trial-and-error learning. when people are asked how they learned to do something which they are good at, their most common responses centre on the terms practice and trial and error.people seem to learn most effectively by doing something over and over again. each time they make a mistake, the error is recorded and attempts are made to avoid making the same error again. in order to produce effective computer-based learning products, this capacity to provide students with the opportunities to practice new skills or knowledge, to make mistakes and correct their errors, is crucial. feedback guidelines 1. provide students with a sense of progression. in order for students to improve their skills or knowledge, they must have some sense of their progression. they must be able to see the improvements that are being achieved and compare those achievements with their own progression up a particular learning step. in terms of computer-based education, this means that the scope of the learning target should always be readily available. a problem encountered in many hypermedia learning jjroduclsis^hat the scope of the learning is never clearly available to students. this means that-they-areoften-unable to see how they are progressing with regard to the learning goals, and may rapidly lose interest because there is no way of assessing their direction or achievement. this can be remedied by providing students with a clear idea of the scope of the learning objectives. students should also be provided with the means to assess their own progress through appropriate feedback. 2. encourage use within a social context. the impact of both teacher and peer feedback is often much more powerful than that automatically generated by the computer. as a result, it is important that learning software is designed so that advantage can be taken of use within a social context (for example, with friends, in a classroom, or even in a remote or virtual class). in such social contexts, students are able to obtain personal feedback with respect to their progress. such feedback als\j has the advantage that it can be targeted specifically at individuals, taking into consideration individual personalities, abilities and expectations. 19 stephen richards educational software design: applying models of learning digesting guidelines 1. embed computer-based student evaluation. in order to assimilate new information, students must undergo a process of making sense of new material in relation to their existing knowledge. this is a process that can be encouraged by providing an environment in which students are required to think about the materials they are learning. perhaps the simplest way of achieving this in computerbased learning applications is through the embedding of appropriate evaluation techniques. such evaluations must be designed to cause students to analyse the concepts or information delivered, and then link the material together in new ways. this process causes the student actively to digest what has been learned, rather than simply trying to reproduce the delivered information without considering meaning. 2. encourage reflection away from the computer environment. as already mentioned, learning tends to take place most effectively within a social (rather than a purely computer-based) environment. just as feedback can be obtained through mechanisms outside the computer environment, so too can digesting take place away from the machine. there are several ways in which this can be achieved. the most obvious way is to encourage students to undertake tasks away from the computer which involve them in trying to apply the concepts or information they have been exposed to in the computerbased environment. such tasks can also encourage the use of team work or discussion groups which provide the social context in which digesting can take place most effectively. conclusion the guidelines outlined in this paper are intended to be used by designers in the practical development of courseware. such work has already been undertaken in one project which is targeted at teaching undergraduate psychology (richards, 1996). preliminary findings have shown that the guidelines are useful (from the developer's perspective), and that they lead to the production of effective learning materials (from the student's perspective). the wider practical application of race's model should lead courseware engineers to reconsider the importance of educational theory in the design of effective computer-based learning packages. references kolb, d. a. (1984), experiential learning: experience as the source of learning and development, new jersey: prentice-hall. mueller, r. j. (1975), principles of classroom learning and perception, london: unwin education books. race, p. (1994), 'how real people learn not what educational psychologists think!', in hoey, r. (ed.), designing for learning: effectiveness and efficiency, london: kogan page. richards, s. m. (1996), 'designing educational software: a case study', proceedings of 13th international confefence on technology and education, new orleans, pp. 40-2. 20 a stile project case study: the evaluation of a computer-based visual key for fossil identification jean underwood,* arne dahlberg,** simon fitzpatrick* and malcolm greenwood*** *school of education, leicester university, **university ofilmeå, sweden ***department of geography, loughborough university of technology this paper presents an investigation of the effectiveness, for specimen identification, of a visual representation of a biological key. the example used in the investigation, the beetles database, is one of a number of resources developed under the stile project (students' and teachers' integrated learning environment). this project uses hypermedia to provide greater opportunities for independent and flexible modes of learning both in a campus situation and for distance learning. the beetles database was constructed to aid final-year project students' in their identification of palaeo-ecological field specimens. the development of this database was a response to a perceived need to reduce time spent on the time-consuming skill of identification, and to focus students' efforts on the significance of their field data. four third-year student undergraduates, two experienced and two inexperienced users of paper keys, were presented with a range of field specimens to identify using either the paper key or the stile visual database. our results show that the visual database was both the preferred way of operating and more effective than paper keys for all students. introduction the stile project (students' and teachers' integrated learning environment) is one of 76 projects set up under the uk government's teaching and learning technology programme (tltp) initiative sponsored by the british higher education funding councils (hefcs). the stile project uses hypermedia to provide greater opportunities for independent and flexible modes of learning both in a campus situation and for distance learning. the approach is resource-based. stile provides a mechanism for both tutors and learners to discover and access relevant resources when they need them, together with facilities that enable users readily to use and re-use existing materials, to integrate them together, and to add further materials of their own in a way that seems natural to them (see ruggles et al, 1995). the result is not a closed and finished product, 40 alt-j volume 4 number 2 but a set of tools and services and a continually developing resource base. the effect is to ease the load on academic staff in maintaining and supporting student access to resources, and to enrich the set of resources available to both staff and students. one such resource is an expanding database of beetles (insects of the coleopterous order). this paper presents an investigation of the effectiveness of a specific use of that database, as a visual representation of a biological key for the identification palaeo-ecological fragments. we would emphasize, however, that this type of resource could be developed, and is being developed, across a range of subject domains. the problem the final year of undergraduate courses in the uk is generally one of increasing specialization leading to an individual long study or dissertation. the reconstruction of past environments using palaeo-ecological field data is typical of such specialist studies. these studies have high academic credibility but are also resource-hungry in both tutor and student time, and as such are increasingly threatened by the need to reduce unit costs. this need for efficiency gains was the initial stimulus for the development of the beetles database. specifically, our task was to ease the load on academic staff by providing easy access to resources for these and other students. as part of our brief, we completed a task analysis from the collection of the field data to the final reconstruction of the past environment. from this we noted that while some 40% of the students' project time was spent on the core activities such as the collecting and preparation of field samples, analysis and report writing, 60% was spent on identification of field specimens. we felt that a disproportionately large amount of time was being consumed by this conceptually low-level task. it became apparent that there was scope not only for efficiency gains in staff time, and therefore costs, but that it might also prove possible to reduce the workload of students. this in turn would lead to benefits by releasing scarce laboratory time. the resources the task of identification is a subset of the overall skill of categorization. categorization skills are the key to both the organization and retrieval of data. they allow the reduction of environmental data into a hierarchy of classes by discrimination, abstraction, generalization and organization of common elements or crucial aspects of stimuli. it has been argued that the organization of knowledge is the key to problem-solving and thus to the individual's successful adaptation to the world. this research emphasis on knowledge structures is underpinned by an extensive literature that has supported two main models of semantic memory: network models, based on the relationship between elements; and feature models, in which memory is organized by the identification and grouping of features (see underwood, 1976; solso, 1988; anderson, 1995). the two identification tools evaluated here, the paper-based key and the computer-based visual key, have different organizational structures as well as different modes of presentation. the former is strictly hierarchical, but the latter, although having a hierarchical overstructure, focuses on feature identification. 41 jean underwood et a\. a stile project case study: the evaluation of a computer-based visual key for fossil identification the paper-based key there is only one current guide to british beetles (joy, 1976), and this is the core text for specimen identification using a traditional paper-based key as shown in figure 1. it has an overall hierarchical structure, in that within the hierarchy information is differentiated, and the properties at one level apply to all related units at the next lower level. for example, the description of the family donacia supersedes and applies to the specific case of donacia cinerea. membership of a category is not a matter of degree but is all or nothing. donaoa strongly metallic, several very variable in colour: l£j is generally more narrowed behind thanlzj; on ike leaves and stems of water plants, chiefly catex; from may to august. -/1 (2). dull, with fine p ubescence. [grey or copp ery; apex of el almost rounded; hind femurwithout a tooth; l. 7-10 mm; eng.,irelu.;bi| cm erea lib st. figure i: a typical key entry the use of such keys is difficult for the inexperienced. the terminology consists of a mixture of english and latin names, and abbreviations are frequently employed. in addition, there is no use of visual information. students at the start of their entomological studies may be excused for finding this key difficult to use. the computer-based visual key the beetles database has a topic net structure (ruggles et al, 1995) which allows great flexibility in its use and focuses on the relationships between objects. for the specific purpose of specimen identification, a hierarchical structure was laid over the net with a focus on specific features or objects. the key starts at the level of families, and is further subdivided into orders. within each order, each insect has a file consisting of one or more whole-specimen images, images of key body fragments, and information about the insect's food preferences and habitat (see fitzpatrick and greenwood, 1996). the trial the development of the beetles database was a response to the perceived need to reduce time spent on the low-level skill of specimen identification, and to re-focus students' efforts on the significance of their field data. it was hypothesized that for moderately experienced users, use of the visual key would lead to faster identification of specimens. in 42 alt-] volume 4 number 2 1) fragment bytra thorax c7 • h«ad donada oassipes hrages of.. o \j decidewhetpartofthe beetle is pictured c ompare the fragments with the range of similar parts end choose when patterns match asim pie click leads to information and more images of i he chosen subject figure 2: identifying the order of the found fragment the user goes through three steps. addition, it was hypothesized that the use of the visual key, in removing the need to interpret paper keys, would provide a lower task entry-level which could be exploited by novice or inexperienced user. subjects four third-year undergraduates, two moderately experienced and two inexperienced workers in this field, were presented with a range of field specimens to identify, using either the paper key or the stile visual database. the two experienced subjects had a working knowledge of paper keys and of computers. the inexperienced users had no knowledge of paper keys and very little experience of using this technology. materials all the target specimens were elytra or wingcases the most commonly found field fragment from members of the chrysomelidae family. subjects inspected the specimens using a microscope with a video camera, linked to a screen. identification was completed either using the visual key, presented on a macintosh lciii computer through the netscape world wide web browser, or using a traditional paper-based key (joy, 1976). procedure before the trial began, each subject was introduced to the identification tools. they were reassured that the trial was an assessment of the tools (that is, the computer database and paper key), and that they should at no time feel that this was a test of their own performance. following this familiarization, the subjects were asked to identify four specimens. they were allowed up to 15 minutes with each specimen. to further aid our 43 jean underwood et al. a stile project case study: the evaluation of a computer-based visual key for fossil identification evaluation of the visual key, they.were asked to provide a concurrent dialogue as they worked through the task. again, reassurance of the focus of the research was provided. a within-subject design with counterbalancing of factors was used. each subject used the visual and the paper key for the identification of two of the specimens. the order of use of keys was counterbalanced, using an abba design, to reduce the effects of learning during the trial itself. performance measures were task time and success rates. all subjects' comments were recorded. post-trial interviews were conducted. results subjects attempted to complete the specimen identification in only 14 of the 16 trials. two subjects, one experienced user (d) and one novice user (a), each identified one of the specimens correctly, and the second experienced user (c) produced a close match judged by our expert to be an acceptable answer; that is, user c had successfully negotiated the higher levels of the identification hierarchy, but had failed to distinguish between closely related samples. in all three of these cases the subjects were using the visual computerbased key. the second inexperienced user (b) failed to identify any of the specimens. the use of the visual key was not without problems, however. students seemed very disturbed by what they saw as disparities in the colour match between specimens and the images in the key. secondly, two of the four students had navigational problems in recovering from an abortive trail. no acceptable answers were recorded for subjects using the paper-based key. this was surprising, as subject c was an experienced user of this tool although he had not worked with the danacia family. one subject refused to complete the identification process when using the paper-based key. the following statements are typical of users' comments of the paper-based key: i reckon it's virtually impossible to tell from the key really. (subject d) • / think i'm going to find this impossible. (subject a) yeah, i'm going to have to give up again. i couldn't do it. (subject b) there was no discernible difference in the overall mean time on task between trials using the computer key (mean of 390.6 seconds) and the paper key (mean of 382.5 seconds).the shortest time was recorded for a non-computer trial when the subject had actively curtailed the trial (table 1). there was an interaction between type of tool used and level of experience, however. figure 3 shows that the more experienced users spent longer with the paper-based key than the inexperienced users but that the position was reversed for the visual key. this small pilot study suggests that whereas the visual-based key enables the student to take part in the identification process, the paper-based key acts as a barrier. this barrier can be so great that the student effectively gives up. more experienced subjects were able to come to a conclusion more rapidly using the visual key with a success rate of 50% (two out of the four specimens). they were more willing to persevere with the paper-based key than the inexperienced users, and struggled to complete the task, hence the increase in 44 volume 4 number 2 visual key papakey inexperienced subject a 586* 500 265 490 subject b 496 284 257 • 327* experienced subject c@ 540 254** 554 505 subject d 2 4 3 * 222 459 203 table i: time taken to complete an identification using the two keys by the four subjects. key * correct identification ** close identification • curtailed task @ this subject has mild colour-blindness, but the problem was not identified until the end of the project. 600 4s0 400 n "g 390 e a> 300 sreconds 5 250 o> e 200 150 100 so 0 _••___ lne>perienced u«en« • exper'enced u«er» • visual key d paper key —1 subjects figure 3: mean time taken to complete an identification using the two keys by level of experience. 45 jean underwood et al. a stile project case study: the evaluation of a computer-based visual key for fossil identification identification time compared to the visual key. this perseverance was not met by success, however. even the small shift from one family of beetles to another made the use of the paper key extremely difficult for subject c. post-trial interviews confirmed that the visual database was both the preferred and the more effective way of operating for all students. subject a, inexperienced but with one successful identification using the visual key, admitted that she struggled with the paperbased key but found the visual key easy to use. she was most impressed by the low level of computer skills required to operate the software effectively. subject b, also inexperienced, had found the paper-based key impossible but appreciated the facility to quickly browse through the visual key. subject c, experienced, felt the visual key was potentially very useful and very easy to use but he had some problems because of his mild colour-blindness. subject d, experienced, was convinced that the visual key would save him time. conclusion while the small sample size for this pilot study necessarily means that the results should be viewed with caution, they are nevertheless encouraging. our initial hypotheses were that moderately experienced users would identify the specimens more quickly using the visual key but at first sight the data does not support our hypothesis. mean times for the use of the two keys are indistinguishable. this result can be explained, however, by the tendency of one inexperienced student to curtail sessions when working with the paper key and by other students' willingness to continue using the computer key after a preliminary identification had been made (subjects a and c) to confirm a decision. long search times with the paper-based key did not lead to success, and the shorter times were associated with a refusal to continue the trial. the efficiency gains that were apparent derive from the differential success rate for students when using the visual key compared to the paper key in addition, it was hypothesized that the use of the visual key would reduce the level of skills required by the novice or inexperienced researcher when first encountering the identification task. this was certainly confirmed. this initial simple measure of performance correct identification showed one novice user performing at a similar level to more experienced counterparts, and the second novice at least completing the task. references anderson, j.r. (1995), cognitive psychology and its implications, new york, w.h. freeman. fitzpatrick, s. and greenwood, m. (1995), antenna (in press). joy, n.h. (1976) a practical handbook of british beetles, london, classey . ruggles, c , underwood, j. and walker, d. (1995), 'stile: a hypermedia approach to the provision of a flexible, interdisciplinary resource base for teaching and learning', innovations in education and training international, 32, 209-19. 46 alt-j volume 4 number 2 solso, r.l. (1988), cognitive psychology, boston ma, allyn and bacon. underwood, g. (1976), attention and memory, oxford, pergamon press. 47 a virtual laboratory system for physiology teaching geoff brindle*, william mitchell*, len seal** and karl bohme** manchester metropolitan university, *department of computing; **department of biological sciences the problem of understanding physiological processes can be aided by visualization tools. traditionally this has been achieved by the use of schematic paper diagrams. however, many physiological processes are quite complex, and in many instances students encounter difficulties in understanding the dynamics. this paper describes the rationale behind an alternative approach using interactive three-dimensional computer-graphics simulation to aid comprehension of scientific concepts. introduction there exist a number of areas in the teaching of physiology which potentially lend themselves to a computer-based learning approach. one such area which has been explored at manchester metropolitan university (mmu) and elsewhere (dewhurst, 1993; kwan, 1993) is the use of multimedia tools to simulate aspects of experiments traditionally performed on animals. the use of real animal specimens (for example, frogs or rats) for dissection and experimentation is both costly and contrary to the ethics of some students. a related but different area which we are currently investigating is that of scientific visualization in relation to animal physiology that is, the visualization of processes which often are difficult or impossible to see except either through the measurements generated on instruments such as chart recorders and oscilloscopes, or the diagrams provided by textbooks. this information is generally of an abstract nature, and students find some of it difficult to grasp using two-dimensional static diagrams of the kind provided in textbooks or handouts. the nature of the textbook medium prohibits some key dynamic ideas coming across. at mmu a collaborative study between the departments of computing and biological sciences is investigating the possibility of using three-dimensional interactive simulations, or virtual laboratory environments, to 53 geoff bändle et o/. a virtual laboratory system for physiology teaching provide concept visualization tools for biologists. the present study is focused on one particular set of physiological processes those which occur at the neuromuscular junction. that process, it is hoped, will provide a vehicle for exploring the requirements of related applications and lead to the development of software tools which can be used to generate solutions more generally. the neuromuscular junction the junction between nerve strands and muscle fibres is typical of a number of physiological processes where understanding is difficult using standard diagrams. the sequence of events which occurs at the neuromuscular junction during the initiation of muscle contraction is complex. it includes the arrival of action potentials, calcium flux changes, merging of synaptic vesicles with junction membrane, release of neurotransmitter from vesicles into the junction cleft, interaction with muscle membrane receptors, pore changes and ion fluxes. an appreciation of the dynamic aspects of the sequence is essential to an understanding of the varying levels of activity seen in muscle ranging from a single twitch to a sustained contraction. a clear understanding of the basic process is also essential before consideration of variation in activity at these junctions associated with the effects of drugs, toxins and disease processes. the virtual reality approach there are many variants of virtual reality (vr) systems. a good overview is provided by barker in an earlier issue of alt-j (barker, 1993). vr systems are less of a new technology than an integration of several existing technologies, the chief of which are 3d computer graphics display and animation, computer modelling and simulation, real-time systems and novel input and output device technologies. when these technologies are combined fully, it is possible to produce systems where the user can feel immersed within an authentic, albeit synthetic world and where he or she can manipulate that world in a natural and familiar way. in such a system, the user interface becomes transparent, and the user feels as though he/she is inside the model data space. rheingold (1991) provides a very readable and detailed historical perspective of vr technologies. advantages of a vr approach the underlying assumption of this project is that the construction and use of systems can be guided by the vr ideal even though they may fall somewhat short of a fully immersive system. some of the reasons for such a stance are: • the use of a 3d model and 3d viewing facilities allow more complex objects, behaviours and relationships to be displayed and manipulated. this is important where the relationships in the world being modelled are complex. the value of 3d representations has been repeatedly shown in the field of computer aided design (cad). 54 alt-j volume 3 number i • vr systems go beyond traditional cad modelling in the degree of realism provided. texture is added to the surface of object geometries to provide detail, and the use of perspective and illumination/reflection modelling provides depth cues. this gives a sense of authenticity so that the user can use intuitive notions of depth and space derived from everyday life to understand the structures and relationships in the conceptual world. • vr provides simulation based on a stored model of the process under study so that the parameters of this model can be changed. understanding can be actively acquired by experimentation, a method in keeping with current educational thinking. • a key feature of virtual worlds is interactivity. it should be possible to interact with synthetic worlds in the same way as we interact with real worlds. as a result, interaction tasks are more natural, and the amount of training needed is reduced (balaguer & gobbetti, 1993). • scientists are trained in the use of instruments and measuring devices as well as in the specifics of the particular process or object under consideration. typically, students need to learn how to calibrate instruments, how to position measurement probes, and how to use the options and switches on instrument panels. if learning how to use the computer interface can be made identical to learning to use traditional instrument interfaces, then existing knowledge of such interfaces which the student brings to the interaction can be profitably used and reinforced. a prototype virtual laboratory a prototype virtual laboratory is currently at an early stage of development at mmu. for development purposes, a vr development system world tool kit for windows provides a set of rendering tools and interaction controls which are called from c programming language interface. the software runs on a pc but requires good floatingpoint capabilities (dx2 or pentium) processors. since the intention is to use the system with undergraduate students, it is important that it can be implemented on a desktop system using relatively cheap interaction equipment. some areas of difficulty the vr laboratory is clearly an ambitious project offering many potential problems and challenges. while initial work has been promising, we are aware of a number of key problem areas: animation desktop vr development environments are proving to be usable in a number of contexts. however, animation is usually in the form of a walkthrough or flyby where the world being navigated stays relatively static. the provision of worlds containing multiple interacting objects is more problematic. convincing animation requires attention to many details concerning the way that objects move, and it is traditionally a time-consuming programming activity. ¡5 geoff brindle et al. a virtual laboratory system for physiology teaching interaction the design and development of virtual tools is an area requiring substantial design and development effort. researchers are only just beginning to explore the possibilities (balaguer & gobetti, 1993). development lead times it is clear that the provision of authoring tools is ultimately necessary if technologies like vr are to have any impact on learning. the amount of work involved in producing learning material is otherwise excessive and the need for specialist subject knowledge is a limitation. it should be possible for lecturers in a particular specialism to produce interactive simulations themselves easily, and within a reasonable timescale (van dam, 1992). conclusions the virtual-world ideal, while perhaps not perfectly realizable given today's technology and the need for desktop solutions, can nevertheless be used as a model to guide the design of cal tools. a range of vr development toolkits and authoring systems are beginning to emerge, and these will be added to as more experience is gained. already these tools are being used in entertainment and in the design of products in industry. at mmu we are developing tools aimed at the application of vr in specific educational contexts, with the aim of providing software tools that can be used generally in the area of conceptual visualization. references balaguer, j.f. & gobbetti, e. (1993), 'virtuality builder ii: on the topic of 3d interaction' in thalmann, n.m. & thalmann, d. (eds), virtual worlds and multimedia, chichester, wiley. barker, p. (1993), 'virtual reality: theoretical basis, practical applications', association for learning technology journal, 1, 1, 15-25. dewhurst, d.g. & williams, a. (1993), 'frog skin: a computer simulation of experiments performed on frog skin in vitro to investigate the epithelial transport of ions', atla, 21, 3, 350-8. kwan, t.k. (1993), a multimedia learning environment for dissection skills in biology, m.sc. dissertation, sheffield university. rheingold, h. (1991), virtual reality, london, secker & warburg. van dam, a. (1992), 'electronic books and interactive illustrations' in cunningham, s. & hubbold, r.j. (eds), interactive learning through visualisation, berlin, springer-verlag. 56 elf: the electronic learning facilitator r. kirby, r mcandrew, a. kilgour, h. taylor and j. t. mayes icbl, heriot-watt university, edinburgh as the world-wide computer network becomes ubiquitous, new tools have been developed, such as the world wide web (www), for the delivery of multimedia hypertext-based documents. similarly, there has been an explosion in the amount of email, bulletin boards, and usenet news available. this has led to a major problem of information overload: we are slowly but surely being overwhelmed by the amount of information available to us. to address this problem, the new technology initiative (nti) of the joint information systems committee (jisc) is funding the elf project over a period of two years to develop a tool, the electronic learning facilitator. this tool is aimed at assisting learners to make effective use of the information available. the elf project started in december 1993, with the goal of producing a tool for installation in uk academic sites. this paper first discusses the new networked information sources that are appearing around the globe, and then analyses the requirements of learners for accessing this information. from the requirements, the paper shows the desirable features of an elf, such as monitoring the user at work to gain hints as to what he or she is doing, being guided by the user directly, and using information gathered from other elfs, or from monitoring the various information sources for announcements of new services, updates of information, etc. the main body of the paper illustrates the current progress on the project, outlining the implementation of the elfelf is written using various high-level languages such as the tool command language (tcl) and perl, as well as the more traditional c++. it consists of various agents, each agent having a specific task to perform, such as monitoring a www site for any new documents, or monitoring a usenet newsgroup for articles that might be of particular interest to the owner of the agent. finally, the paper reports on our plans to test users on a prototype. introduction recent years have seen an unprecedented increase in the use of the internet and other 69 r kjrby et al. elf: the electronic learning facilitator global networks of computers. we have seen estimates of between 100,000 and 1,000,000 new users each month joining the internet. predominantly these are information seekers, but the figures do also represent a significant increase in the number of information providers. for learners, we now have a significant set of information sources, albeit obscured by a lot of noise. there are many different types of information sources now available, all with varying attributes. for many years, the internet has provided email. for learners, this has allowed direct contact with the authors of research papers at different (and sometimes the same!) institutions. the usenet news system has enabled learners to reach a far wider audience, with less intrusion (since it is up to each user to decide to read the news, whereas email is generally always read). again, this provides a good resource for the learner to find out more by asking questions and receiving a wide variety of responses, although often this requires wading through a variety of 'me-too' responses, and flames (inflammatory responses which tend to solicit ad hominem attacks). electronic mailing lists are the email equivalent of a newsgroup, although they are harder to find out about, but tend to have less noise. ftp sites are another, now familiar, information source. here electronic copies of research papers can be stored at an ftp (file transfer protocol) site and others can then download (copy) them to their local machine for printing and reading. this is a rich source of information in the sense that there is rarely any noise stored at an ftp site, but it is hard to know where to look and what to look for,.except from some other information source (such as a news article). recently, new types of information sources, such as the world wide web (www), gopher, and the wide area information servers (wais) databases have proliferated (berners-lee et ai, 1992). gopher can be viewed as a distributed menu system, where users select items on a menu which may lead to another menu, or to some actual information (such as a picture or a research paper). the www is a distributed hypertext system where links to objects are embedded within other objects (where the objects can be text, pictures, video, and other multimedia). both gopher and the www support the browsing model; the user has actively to follow links to find information, in contrast to email or news where the user passively receives information. wais databases are essentially free-text search engines which allow for keyword searching of collections of documents. the increase in the desire to use these networks, and the increase in the amount of information being made available over them, is causing problems for both experienced and novice users. this paper will introduce the problems, outline a possible solution, and end with an indication of current progress. problems information sources such as mailing lists and usenet tend to overwhelm users with the sheer volume of information, often with a low ratio of signal to noise. both can be thought of as unsolicited information, where the user must invest large amounts of time in reading to find the nuggets of useful information. by contrast, users of the www 70 an-j volume 3 number i often have a different problem, that of navigation, by which we mean knowing where the desired information is to be found. with wais databases, the problem is similar: knowing which database to search. there is a directory-of-servers (a database consisting of short descriptions about a large set of registered databases) to help with this, but as with any centralized solution, it can be tied up with the multitude of requests from users around the world. it is also not complete nor up to date. our current work is predicated on the belief that there is useful information in all these information sources, but the trick is to know where to look. for example, one often finds out about new www sites from the www newsgroups. another major source of pointers to useful sites is from colleagues. there are also index sites which build a database of pointers to other sites using some keyword mechanism to act as the key into the database. these include the wais directory-of-servers, and there are several others on the www (such as the jumpstation, and the www worm). a solution elf elf is an attempt to address these problems. it is essentially a collection of agents which co-operate together to aid users in finding information of relevance to their interests or tasks, in a timely and efficient manner. one of the main aims of this project is for elf to work without explicitly requiring direct control by the user. to do this, elf monitors the user at work to determine (from files recently edited, for example) the areas of current interest. elf attempts to categorize the current work by a few keywords which are then used to seed a search conducted by various agents. the results of this search are presented to the user at some appropriate point (to avoid unnecessary interruptions in the user's current task). elf can also be controlled directly by the user. this means that the user controls what information sources are searched, and decides on the search query terms. there are two major sides to elf. the first handles the user, and attempts to gain knowledge about his or her tasks. the second is a back-end which consists of agents that 'know' how to query and search the various information sources. we will discuss each of these parts of elf in more detail below. elf allows for varying interaction with the user. the minimal level of interaction involves a discrete icon on the screen from which various control panels can be accessed. the only other elf tool that needs to be seen is called prism (personal information space manager). this is a hierarchical structuring tool which can be used as a simple file/directory browser, but can allow for grouping together related tasks. prism allows access to the keywords that elf calculates for a directory or file, and provides for the user to edit these. users can also use prism to determine and control which agents are currently working on their behalf. while it is indeed true that the hierarchical structuring model leaves a lot to be desired, it is easy to understand, and matches naturally to the common computer file system. elf provides several different mechanisms for notifying search results. in the case where a search finishes and the user is currently logged on, elf can notify by forcing mosaic 71 r. kirby et al. elf: the electronic learning facilitator (or certain other www browsers) to pop up on the screen with a page containing pointers to the results found, using the fact that mosaic can access most types of information sources, thus providing a simple click-to-access interface to the results. another option is to indicate within prism the presence of results at the relevant part of the hierarchical structure. a third option is to provide an xbiff type program (xbiff is a small icon that notifies the arrival of new mail by changing colour) to flag the arrival of results. these clearly vary in intrusiveness to the user, from the popping-up mosaic to the small discrete icon in the corner of the screen. if the user is currently logged off when the results arrive, elf can either send a mail message containing the pointers to the results, or allow for prism to indicate the presence of results the next time the user logs on. more experienced users can control elf directly, using search parameters, and requesting immediate searches (cf. traditional database searching where the user makes a query and waits for the result). elf also collects statistics about which keywords seem to work best, and which information sources produce the best results (as selected by the user). these can then be used either by elf or the user to tailor the searching to fit in more with the working practices of the user. the back-end of elf consists of the agents that perform the actual searching of the various information sources. one agent (called the news agent) is responsible for querying a database of news articles. one of its responsibilities is to find articles that announce new www sites. this is done in a simple way by looking for announce (or some derivative), followed by a universal resource locator (url) indicating where the www site is. another agent interfaces to the jumpstation search engine on the www. it mimics a user-request to the jumpstation and reformats the returned result (which is a www page of links to the results) before presenting it to the user. similarly, agents can be provided for other www search engines, such as the www worm and aliweb. other agents can include wais searchers, gopher searchers, and mailing-list handlers. agents can also be designed to access interactive services such as bids (one of the library catalogue systems). one of the drawbacks of providing agents that use the same interfaces as the user, rather than requiring database implementers to provide a special application programming interface (api), is that if the interfaces change, the agents will no longer work. this could be as simple a change as a rearrangement of some menus. we will address this issue somewhat in the next section. since different information sources provide different search strategies (such as boolean exact matches, set operations, or partial matching), some agents have to do more work than just sending a query on and retrieving the results. for example, an agent that accesses a search engine that only accepts a single keyword, might need to submit several searches for a multiple keyword query, and then combine the results, so that, as far as the user or the rest of elf is concerned, it answered the query in one fell swoop. 72 i volume 3 number i the departmental elf as outlined above, since elf is a computer program, it can generate large amounts of queries in a short space of time for long periods without rest (after all, this is why we replace people with computers). it was quickly realized that if everyone used an elf, then the networks would be flooded, and the search engines accessed by the elf agents would overload. to address this, we propose a departmental level elf. this elf will be responsible for actually doing the work. the agents of a user's elf will submit their queries to the departmental level elf> which then has the opportunity to multiplex queries across users (in other words, perform duplicate queries only once). results can be cached and used to answer later queries (recognizing that the results may be out of date). the departmental elf could also help mimic the word of mouth phenomenon by allowing for transferring of information between user elfs. another benefit of the departmental elf is to provide centralized management of agents, which addresses the changing interface problem alluded to in the previous section, by. requiring just one place to be changed. current state and future plans a prototype for a single elf is near completion. this elf includes a database of news articles (dynamically changing due to new articles coming in and old ones expiring), a news agent that operates over this database looking for announcements or new www sites, or just general keyword searching. other agents include a jumpstation agent and an archie agent (archie is an index of most ftp sites in the world). the prototype also includes prism, and a user-monitoring agent which periodically searches the user's directory space for new or changed files, and attempts to categorize them with a few keywords, using a combination of full-text analysis, typed-text analysis, and clustering techniques. this elf runs under the unix environment running x, and is implemented in a mixture of c++, tcl (tool command language) and perl. the development of elf will involve iterative design, with user testing at each stage. by employing usability metrics, we are aiming to test groups of students, lecturers and researchers using the initial prototype. the development of elf as a tool to support the process of learning in a highereducation context is being conducted within an overall framework provided by the concept of the intensely supportive learning environment (sykes et al, 1993). this adopts a constructivist approach to learning (kommers et al, 1992) in which the learner has available a range of tools for the performance of learning tasks. the process of learning through active seeking and structuring knowledge is emphasized at the expense of the traditional view of courseware as packaged and pre-digested exposition. an important issue raised by this is the identification of the point at which the work done by elf agents might become counter-productive for an individual learner. final aims of the project are to build a departmental elf, to port the user interface to ms windows, and to release the system to the uk academic community. 73 r. kirby et al. elf: the electronic learning facilitator references berners-lee, t., cailliau, r., groff, j. & pollermann, b. (1992), 'world-wide web: the information universe' in electronic networking: research, applications and policy, 1, 2 (spring), westport ct, meckler. kommers, p., jonassen, d. & mayes, j.t. (1992) (eds), cognitive tools for learning, berlin, springer-verlag. sykes, p., schaper, j., mayes, j.t., palmin, h. & marcipont, v., 'isle: a collaborative project to build an intensely supportive learning environment' in maurer, h. (ed), educational multimedia and hypermedia annual: proceedings of ed-media 93, charlottesville va, aace. 74 reviews edited by philip barker technology-based learning environments: psychological and educational foundations edited by s. vosniadou, e. de corte and h. mandl, volume 137 in nato asi series f (computer and systems sciences), berlin, springer-verlag, isbn: 0-387-58253-3, 1994. this publication is one of a collection of texts that has been produced within the nato asi f series books published as a result of the activities of the special programme on advanced educational technology. currently, the list of books in this series totals over 35. the present volume contains 37 papers that were contributed to the advanced study institute (asi) on educational technology held in crete in the summer of 1992. the purpose of the institute was to bring together lecturers and graduate students in order to investigate and discuss the psychological and educational foundations of technology-based learning environments, and to debate the implications of recent research findings in the area of cognitive science for the development of educational technology. the papers that make up these proceedings are organized into four basic sections, each devoted to a particular theme. part 1 contains 10 contributions dealing with design principles for learning environments. a range of relevant issues is discussed, some of the more important of which are: theoretically-based visions of educational environments; active and constructive learning; exploratory learning environments; simulations; socio-cultural issues; the use of intelligent co-operative systems; role play; experimentation; and assessing the effects of a learning environment. the ten papers that make up the second part of the book deal with the use of educational technology to promote conceptual change in the physical sciences. some of the interesting topics discussed in this section include multimedia environments for enhancing science instruction; generative problem-solving in school science; use of hypermedia; model-based reasoning; promoting the study of conceptual change; diagnosis and remediation; software tools integration, environmental education; and the development of subject-oriented learning environments for teaching basic electricity, the human nervous system and physics. the third part contains eight papers. these explore, describe and discuss various aspects of technology-based learning environments devoted to knowledge acquisition in mathematics and natural language. the first five papers deal with topics related to the study of mathematics, namely the use of multimedia environments for enhancing student learning in mathematics; task analysis and didactic tools to support mathematics tutoring; 'unintelligent' tutoring in algebra based on psychological theories and experimentation; the use of intermediate models for solving word problems involving the development of arithmetic skills; and the use of computer programming as a mechanism to support the development of open-ended project work. the remaining three papers deal with specific issues in language learning, covering the use of computer-based methods to develop reading and writing skills; computer facilitation of skilled narrative writing processes; and the development of a 'flowdriven' english (as a foreign language) course. 76 alt-j volume 3 number i the final part of the book containing the remaining nine papers is devoted to the theme: 'taking into consideration the needs of the learner'. the important issues covered here include the use of knowledge profiles in a flexible learning environment; the use of student models; semantic networks; representational systems in mathematics and science; the use of graphics in the computer-aided learning of chemistry; an investigation into motivation and co-operation in computer-assisted learning; and various issues relating to the use of hypertext. the three papers on hypertext discuss the effects of visible-link types on learning, knowledge construction and acquisition in customized hypermedia environments, and the relationship between hypertext learning environments and epistemic beliefs. i found the contents of this book extremely interesting from both the point of view of topics and that of treatment. obviously, the wide range of writing styles encountered in a multiauthor book of this type means that continuity and content often fluctuate from one chapter to another. however, the editors have done a masterly job in selecting and grouping papers, and in ensuring that a reasonably consistent style has been maintained. the book is expensive (£47.00 sterling), so i have to recommend it for a departmental library rather than for individual purchase, but i do so recommend it. philip barker, university of teesside aspects of educational and training technology, volume xxviii: computer-assisted and open access education edited by f. percival, r. land and d. edgar-nevill, london, kogan page, isbn: 0-7494-1414-6, 1995. this is a selection of the proceedings of the 1994 international conference of the association for educational and training technology (on computer-assisted and open-access education) hosted by napier university in edinburgh. around 85 papers were presented at the conference; 44 have been reproduced in full in this volume, with the inclusion of abstracts from the rest. reflecting the organization of the conference, the papers have been divided into four sections: perspectives on teaching and learning; computer-assisted learning (cal) and computer-based training (cbt); multimedia and hypermedia; and open and distance learning. discussions of political issues, underlying concepts, and project reports appear in each section. clearly, then, the conference attempted to cover a lot of ground, and quite successfully on the evidence of these proceedings. as a result, the book works effectively on two levels: as a useful update for experts, and as an informative introduction for newcomers to any of the subject areas described above. however, as the editors readily acknowledge, there is considerable overlap between these areas, leading to the somewhat arbitrary placement of some papers. the use of computer-based technology in the classroom still continues to excite healthy debate, as evidenced by some of the arguments advanced in the first section. there are some who predict a radical shift towards technologybased solutions in the near future; others take a more cautious approach. separating those contributors actually using the technology in teaching/learning situations from the politicians might assist the reader in evaluating the arguments. it was also heartening to see the work of price, hobbs and cheah in which the use of standard software-development control and quality assurance procedures are applied to courseware development. all too often where courseware development is concerned, software design is confused with interface design, often leading to predictably unfortunate consequences. sections 2 and 3 will be particularly useful to those involved directly in the production of cal/cbt packages. many of the papers here describe applications of some of the betterknown authoring tools, such as toolbook and guide, and the inclusion of a large number of screenshots gives the reader a real feel for what is taking place. jacquetta megarry's paper on design challenges is particularly refreshing, questioning the indiscriminate use of resourcehungry video in desktop packages. section 4 is again largely project-based. the final section, containing the abstracts, may certainly prove useful, as readers may obtain full papers from the authors, though (given this) the inclusion of full addresses here would obviously have been helpful. the selection criteria used to determine whether a paper was included in full or in abstract form are not included. the volume has been produced to a very high standard. the hardback format combined with good quality print and paper give substance to a 77 reviews book which would be a very useful purchase on the basis of content alone. lorraine warren, university of humberside groupwork in education and training: ideas in practice by michael reynolds, london, kogan page, isbn: 0-7494-1027-2, 1994. the intended audience for this book is broad. it is aimed at practitioners involved in education and training who use, or want to use, groupwork methods, and anyone interested in the social, psychological or educational processes associated with group work. the author states the overall purpose of the book as being 'to make sense of the complex array of ideas, methods and issues' associated with designing and using group activities, and 'making sense of the consequences of using them". my overall verdict is positive. i enjoyed reading the book, and on several occasions it challenged my assumptions about the way i work with groups. the one downside lies in the fact that the subject area is vast, and to do justice to all aspects of it would need substantially more than the 138 pages of this book. to be fair to the author, however, he acknowledges that he has gone for breadth rather than depth, and he gives excellent references for following up particular points of interest. as an overview of the subject, then, it offers a well constmcted, easy-to-read account of many of the key issues. chapter 1 addresses the question of why someone might want to use group methods in the educational process. it offers a description of the different types of group methodologies (for example, simulations, discussions, and action learning) and how they differ. the author concludes that the key reason why group work is so valuable is that it enables learning to be 'grounded in the personal and social experience' of the learner, and that it places the responsibility and (to different degrees) control of the learning in his or her hands. chapter 2 focuses on the design of group work and looks at what factors actually contribute to the learning outcomes of a group session. the author's key point is that while the design is important, the outcomes can be influenced by a great many things, such as participant expectations, the group's history or the social context good comfort for those of us who agonize over 'failed' designs! chapter 3 gives an excellent overview of some of the theories that underpin group work. the author emphasizes how important it is for trainers and teachers to be aware of the assumptions, values and beliefs that inform their approach (theories are seldom value-free), and adds that no single school of thought is the 'unarguable truth'. he summarizes a few of the models in current use which can act as a good starting point for observing the behaviour of groups, and sets out some complex ideas simply, though i found that his description of how to apply these models in practice was less clear. chapters 4, 5 and 6 offer examples from practice which are intended to illustrate some of the issues associated with learning groups which tend to be neglected. these chapters make a number of key points which made me think. the first is that whatever the intended focus of an activity, the actual experience of it will be broader, as participants relate it to their own individual experience/circumstance. the second point is that it is easy to assume that there are neat boundaries that can be drawn around particular group activities. however, that so many such activities have unintended consequences points to the fact that they are often not as 'bounded' as one might imagine. the final point, is that the design of a group session inevitably transmits to participants certain social values (for example, about who is in control, what is/is not open to influence). i found these chapters very thought-provoking. chapter 7 looks at some of the newest approaches to group work, and takes the form of a series of 'interviews' between the author and practitioners who are using these methods. i found the descriptions a little too cursory to be very useful, but they act well as an introduction to the subject areas. this very well-written book is, therefore, a good introduction to the area of group work, and although a little superficial in places, it raises some important issues. marion gillie partner, sheppard moscow ltd management consultants (tel: 0181 468 7975) teaching and learning with audio in open and distance learning by derek rowntree, london, kogan page (open and distance learning series), isbn: 0-7494-1154-6, 1994. this publication is an audio-print package intended for teachers and trainers. it consists of 78 volume 3 number i a book of 124 pages together with an audio cassette, presented together in a durable booksize container. the audio cassette contains examples of audio teaching which are referred to in the book. side a contains fifteen short clips as illustrative material; side b contains two more extensive examples. as might be expected from an author with derek rowntree's background and experience, both the book and the audio tape are introduced in a way which gives excellent, friendly study guidance to users of the package. the tape begins with a personal introduction and advice from the author on how best to study the material, and the book opens, as all good training packages should, with a statement of aims and objectives. this is followed immediately by a concise summary of the overall organization of the package and a restatement of the study advice given on the audio tape. the package is divided into five parts: (1) the case for using audio; (2) examples of the diversity of audio teaching; (3) examples of extended audio teaching and their analysis; (4) making your own audio package; and (5) evaluating audio packages. the book also includes a list of 17 specialist references which are relevant to the use of audio packages in teaching and training. in each case, the author has indicated how that reference might be useful. at an early stage, rowntree emphasizes that good audio tuition will probably make use of both the sense of hearing and sight, and this is reflected in the exemplary material he has chosen. in part 1, he reviews the potential objectives of using audio in teaching, either as an aural source or combined with visual materials and activities. this section is typical of the book as a whole, in that the layout is pleasing and well-organized, with excellent use of different fonts, borders and headings to distinguish author contributions from examples, and something like 20 per cent of the pages in the book feature diagrams and drawings referred to in the taped examples. i stress the appearance of the book because it is presented and structured as a self-study package rather than as a textbook with accompanying audio tape. excellent use is made of boxed instructions and reminders within the formal textual materials so that the behaviour and thoughts of users can be appropriately organized. this means that the package is free standing and ready for immediate use by interested tutors or by students following courses in educational technology or flexible and open learning, and i shall have no hesitation in offering this package to my own students. the taped extracts in part 2 illustrate the use of audio for listening, for listening and looking, and for listening and doing. in each case, the text poses questions for readers to consider as the tape is listened to, and then comments on the extract afterwards. one very useful feature is that the text then invites readers to consider how they might use the strategy they have just listened to in their own teaching. it is worth emphasizing that the illustrative examples are all examples of good practice there is no attempt to create a somewhat artificial exercise of comparing good and bad practice. however, part 3 of the package does compare and contrast two extended examples of good practice using different techniques. both illustrate techniques unique to audio, and could not easily be provided by alternative means. the final two sections give clear advice on how to prepare and evaluate audio packages, represent an excellent compilation of good practice in the audio field, and are sufficiently detailed to allow an instructor to use audio for the first time. it will by now be clear that i was very impressed with the overall quality of this package: its excellent study guidance (both in the text and on the audio tape), and the integration between the text and the audio tape. my single quibble is with the price (£19.95 sterling) which is presumably related to a potentially limited sales volume. equivalent text plus audio packages in modern languages are much more modestly priced. dennis moss, university of wales, cardiff using communications media in open and flexible learning by robin mason, london, kogan page (open and distance learning series), isbn: 0-7494-1149-x, 1994. as the range of available educational technologies and their respective proponents multiply, so does the need for some rational and enlightened view on what these technologies offer those who are engaged in the process of education. mason's book, as part of the open and distance learning series, provides this 79 reviews evaluation for the subset of educational technologies she calls communication media. the book is intended as an introduction to the subject. it has approximately 120 pages of content, and is quick and easy to read. it contains eight chapters, an annotated bibliography and an index. the structure is both logical and straightforward. the first chapter discusses the uses of telecommunications media in education, and introduces the three main approaches to using them (computer conferencing, audiographics and videoconferencing). the following two chapters are devoted to the important issues of interactivity and the implications of using these technologies. mason then considers each of the technologies in subsequent chapters using similar section headings for consistency. the penultimate chapter describes nine live projects that are currently using these techniques. the concluding chapter explains why the different technologies do not compete with each other, and gives some advice on best practice. in chapter 2, mason gives her personal view on what attributes of interactivity are desirable in an educational context, and evaluates these for computer conferencing, audiographics and videoconferencing. the view taken is the constructivist one that interactivity can enhance learning. the author proceeds to explain how these technologies can facilitate collaborative working and help cultivate a social presence. chapter 4 examines the implications for those using and developing communications-media applications in education, and the overriding message is that using these technologies can be very rewarding for students and staff alike, but that it requires commitment and hard work. the message to organizations is that these technologies do not necessarily mean cost savings, and that they must provide appropriate support and investment in infrastructure. chapter 5 explores computer conferencing by outlining the current state of the technology, its educational uses and its advantages and disadvantages. the impression given is that computer conferencing is the most accessible of the three technologies and is also the most educationally effective. chapter 6 examines audiographics following the themes of the analysis introduced in the previous chapter. audio and graphics seem to work well together, the graphics being the focal point for audio interaction analogous to the ohp in a conventional lecture. currently, the main use for audiographics seems to be for connecting smaller groups to larger centres for tutorial purposes, but trends suggest that this fascinating technology has quite a promising future, especially when combined with multimedia functionality. chapter 7 discusses the educational uses of video-conferencing; unfortunately, this is limited (perhaps reflecting the cost of the technology?). the section on advantages concludes that one-way video conferencing facilitates didactic lecture-delivery over a wide area with the potential of reaching a large audience, while two-way videoconferencing is best reserved for small tutorial groups. mason suggests that visual communication courses are probably best suited for exploiting videoconferencing techniques. a chapter concerned with live project case studies is very useful and puts the material covered in previous chapters into context. the case studies come from all over the world and, taken together, are inspiring. each project includes an evaluation of its educational effectiveness with pointers for success, though it would have been interesting to have had more information on the curriculum design aspects of the projects. in the concluding chapter, mason is emphatic about the fact that the three technologies discussed in her book do not compete with each other. each technology supports different learning styles, and the context in which they are used is different. good pragmatic and pedagogic reasons are given for adopting these technologies. communication media are important elements in the emergence of a new type of learning environment ('close distance education') that will capture the best of our conventional learning environments and our newer open and distance learning systems. the book finishes with some best-practice advice. mason's clear intention with this book is to provide an easily accessible introduction to communications media for education, and this she does very successfully. specialists and nonspecialists alike will find the book easy to understand, and it may even inspire them to try out the technologies for themselves. indeed, it will most likely leave many readers with the question: 'how do i redesign my course to include one of these technologies?' geoff elliott, cardiff institute of higher education 80 alt-j volume 3 number i computer-based learning in science proceedings of the international conference cblis '95 edited by graham m. chapman, open education and sciences, opava, czech republic, isbn: 80-901974-0-x, 1995. cblis '95 was held in opava (czech republic) in july 1995, and provided a forum for researchers and practitioners interested in technology-based science education. the proceedings consist of 60 contributions selected from a total of 110 papers originally submitted for review. the papers are grouped into 11 sections which vary greatly in size from a single paper (forming the section on intelligent software) to 12 papers (in the section on teaching packages). this reflects quite well the focus of research in computer-based education generally, where solutions are required relatively quickly to address current teaching problems. systems utilizing aspects of artificial intelligence, on the other hand, can have significant time, cost and complexity overheads due to the additional dimension of embedded 'intelligence'. as a result, far fewer projects aim at using potentially useful techniques drawn from the field of artificial intelligence. the first section (invited lectures) consists of four invited contributions which provide a broad overview of some important issues arising from the development of computer-based education in science. in this review, i shall present a short summary of each of the invited papers before providing a brief guide to the remaining sections of the book. in the opening paper by philip barker, the main issues covered concern cognitive engineering and interface design. some of the key cognitive engineering concepts introduced here are mental models, learning styles and the use of metaphors. interface design issues explored by barker include interactivity, graphical userinterfaces, hypermedia, and intelligent interfaces. each of these issues is discussed in the context of designing human-computer interfaces to support effective learning. the second of the invited papers, by graham chapman, provides a brief review of some recent trends in computer-based learning and hardware development. chapman places considerable emphasis on a few key issues and identifies the use of expert computer programmers as perhaps his most important message. chapman argues that expert programmers must be used, and that the production of quality courseware necessarily involves a close cooperation between academics who are subject experts and computer programmers who provide production expertise. milos lansky takes a somewhat philosophical view of computers in education in the third of the invited contributions. lansky makes parallels between the work of comenius (a sixteenth-century czech scholar) and computerbased learning. this analysis results in nine statements (or guidelines) which indicate that historically important pedagogical concerns remain relevant in computer-based learning. the final invited paper, by peter wright, addresses the problems surrounding educating the educators. here the main concern is the preparation of technologically literate teachers who can act as both advocates and role models for the use of computers in education. both inservice and pre-service teacher training are considered as means by which teachers in general can become catalysts in the widening educational use of the new technologies. it is impractical to present a detailed discussion of the wide range of topics addressed in the remaining sections of the proceedings. since some effort has been made to organize the proceedings into sections, however, a list of the areas covered can provide an indication of content. topics addressed are: (a) review lectures (b) teaching and learning: educational strategy (c) teaching and learning: cbl structures (d) monitoring of performance (e) software and hardware developments (f) intelligent software (g) teaching packages (h) teaching with simulation (i) interactive programmes (j) networking and distance learning. the book also has a preface, contents list and author index.-the preface is light, to say the least, and gives little more than the location of the conference and the names of the organizers, papers committee and sponsors. the inclusion of an author index will be of use, though (in line with many proceedings) the absence of a traditional index prevents the work being used efficiently as a reference source (perhaps the main use to which academic books of this kind are put). even better than the inclusion of an index, however, would be availability on cd-rom along with an appropriate search engine. this 81 reviews would allow the primary target audience of the book, academics, to use the information effectively in their work. the falling costs of cd-rom publication could make this a viable option for many conference proceedings and would greatly increase their utility. copies of these proceedings can be obtained from: open education & sciences, po box 84, 746 20 opava, czech republic. stephen richards, university of teesside isdn applications in education and training edited by robin mason and paul bacsich, london, the institution of electrical engineers, isbn: 0-85296-860-4, 1994. this is an edited collection of papers describing the exploitation of isdn for distance education. it is unfortunate that the publisher chose to print the acronym isdn in huge letters on both the spine and the front of the book since it misleads the bookshop browser. it is not a book primarily about isdn, though it does describe the benefits and limitations resulting from its exploitation in education and training. a summary on the back of the book explains isdn: integrated services digital network (isdn) is the provision of a digital-based coupling between two sites, using the ordinary telephone network and the existing twisted-pair cables into customers' premises, for the purpose of conveying data at a relatively high speed as a cost-effective alternative to high-speed leased communication lines. as a result it provides a motion-video communication link operating at several frames a second (using special compression/decompression hardware) to organisations that would otherwise be unable to justify a leased line. while the importance of this infrastructure provision must not be understated, the excessive enthusiasm over isdn in this book is unwarranted, and the subject is often needlessly incorporated. however, this comment should not put off the potential buyer looking for a source of information that explains the ramifications of exploiting isdn as a transmission medium for both passive and interactive distance learning. the book consists of 13 chapters divided into three sections, plus a concluding chapter, an index and an extensive glossary. the first section introduces the nomenclature of digital communication standards. it identifies the requirements for fast transmission of text, images and speech, together with the possibility of exploiting data compression techniques. the problem of re-synchronizing speech (which is difficult to compress) with its corresponding moving image (which is not so difficult to compress) is briefly mentioned. since basic-rate isdn is not capable of transmitting televisionquality video in real time, various compromises are described (e.g. using monochrome only, or accepting a jerkily changing image) with particular reference to their use in education and training. this first section also describes what is meant by audiographics, videoconferencing and computer conferencing, and relates them to their exploitation in education and training. some interesting views on the future methods of teaching are expressed, including the fundamental need for interaction in order for learning and monitoring to take place.-the section ends with a world view as to which countries have adopted isdn, its continued and future use as a means of communicating information not specifically targeted at education and training, and isdn's strategic relationship with leased lines, local-area networks, and so on. the issue as to whether basic-rate isdn is fast enough is revived at the very end of the section with views as to how speed improvements will become available at a cost that is affordable for educational establishments and for students located at home. videoconferencing provides for two-way audio and video communication, and section 2 consists of six chapters, each with a different perspective on this facility. the first describes a university's provision for staff members at two major companies to participate in the university's courses. the organizational issues, the perceived benefits, and three common lecturing approaches are discussed. there are problems (e.g. reliance on last-minute preparation, and ignoring off-campus students), but we are told that these are largely overcome by coaching the lecturers into handling this new environment. the next three chapters describe the exploitation of videoconferencing in the british isles, australia and norway, the main rationale for this exploitation being to provide important courses 'to a dispersed student population with a thinly distributed demand'. the descriptions are mainly about the equipment and technical formats involved, but 82 alt-j vo/ume 3 number i there is also some focus on the way in which this method of teaching changes work patterns. for example, it 'encouraged students to become more responsible for their own learning patterns' and 'promoted group cohesiveness and group independence from the tutor*. there is also some emphasis on the lecturers' concerns about videoconferencing and their perceived benefits of the medium, and comments about the technology degrading the prepared visual material so that special care has to be taken over its preparation. for most of these systems, the lecturer was physically present in a lecture theatre teaching a group of students, with the other students located at centres with two-way audio-visual links, but a french system had the lecturer shut in a booth remote from any student. the natural consequence of this isolation was that the apprehension of being in front of a live audience was diluted. it is a very 'presidential' style, but does 'allow you to concentrate on (giving) the lecture', though students did demand a greater degree of interaction with the lecturer. section 2 also contains a cost-benefit analysis of videoconferencing for both companies and universities. it is based on a range of equipment and a range of features available. the analysis includes illustrations of various jobs from factory-floor training through engineering to sales and management, and should offer helpful arguments for adopting videoconferencing in both industry and academe. this second section covers distributed lecturing with the opportunity for students at remote sites to ask questions. section 3, in contrast, describes systems in which students are more participative. these systems involve the technology of computer-supported co-operative work based on personal computers. teacher and students share the same information displayed on their own computers, and of course there is an audio link. one of the systems described enables a shared window to display the output of a program so that, for example, a number of people can experience the hands-on learning of a drawing program; the locus of control passes from teacher to student and back again as the learning progresses. various scenarios, are described to illustrate the usefulness of such a system. unfortunately, it seems, each window took many seconds to update, and the teacher had to' fill in while each person's computer synchronized, making the whole session much longer than if it had been face to face. a view of participants' faces was not felt to be too important although the absence of some turntaking cues caused minor problems. section 3 generally reviews teleconferencing, audiographics and computer conferencing, and concludes that 'conversation represents the strongest underlying paradigm for any computer-mediated form of collaborative learning'. nevertheless, the lack of visual cues means that tutors will adopt a more interrogative style in order to monitor the 'presence' of students than they would need to do in face-to-face tutorials. two systems designed to support collaborative learning are described. one was an attempt to integrate, in one system, audio teleconferencing, audiographics and computer conferencing, where each student sees on his or her personal computer a resource centre consisting of lectures whose material (slides and drawings) has been previously down-line loaded and the live lecture then given via a two-way audio link, a 'meeting room' within which participants can orally discuss and present documents, a 'study room' in which the tutor can observe the state of each learner's computer screen.a conventional computer-conferencing tool, and a multimedia resource library. the view is expressed that social interaction is an essential component of the learning process and that informal peer learning is as important as formal teaching. is the book worth buying? it certainly should be bought by libraries and retained on their shelves for decades to come, since it will become an important source of historical information recording the pioneering days of distance education using digital communication links. for current educators, it does provide an introduction to the transmissive technology at present available, identifies whether it is useful to them, and describes how it has been exploited. ian benest, university of york 83 using information technology t o help business students learn about contract law john s. edwards and robert i. akroyd aston business school, aston university this paper describes continuing work in using information technology (it) to help business students learn about contract law. the approach adopted uses a model of the contracting process as being one of negotiation, where the decisions made by the parties involve the acceptance or rejection of certain risks. normal discussion tutorials are therefore replaced by a role-play exercise in which students learn by taking part in simulated negotiations, each interested party being represented by a team of students. it is being introduced into the learning process, both to provide decision-support for the student teams, and to improve the mechanics of the exercise. introduction almost all business students study law. however, business students have a different perspective on law from that of law students. a common problem, therefore, in legal courses for business students is how to provide a sufficiently wide-ranging, practically relevant programme without the sacrifice of intellectual depth. at aston business school, one approach adopted has been to supplement lectures with role-play exercises, rather than conventional tutorials. this paper concentrates on contract law, which also forms a key part of the commercial law module. the principal exercise, wheeler-dealer, involves the negotiation of contracts, legal relationships and disputes. it has been in use at aston for several years, originally in paper-based form. students are divided into two or more teams. each team is given a particular (real or imaginary) scenario, but is prevented from knowing the full scenario of the opposing team(s). initially, the teams negotiate a contract or other legal relationship and then, when eventually the deal goes wrong, handle the resulting dispute. the role of the tutor is no longer that of teacher, as it would be in a conventional tutorial. instead it is to act as referee of the role-play, to advise on the law, and to play the role(s) of any public officials, such as the judge in court proceedings. a more detailed description may be found in akroyd and edwards (1994). 103 john s. edwards and robert i. akmyd using information technology t o help business students this paper describes the introduction of information technology (it) into this learning process. other technology has been incorporated from the beginning, parts of the exercise being video-recorded to enhance the student participants' learning of legal skills. it support for role-play exercises in negotiation there are two ways in which it can help students learn from wheeler-dealer and similar role-play exercises: supporting the student groups in their decision-making (decision support), and supporting the administration of the exercise and the process of negotiation (mechanical support). our work so far has concentrated on the former. decision support the field of decision-support systems is only some twenty years old, and many of its problems have not yet been satisfactorily resolved (see, for example, silver, 1991). providing decision support which enhances learning is a harder problem still. fortunately, this task is made feasible because wheeler-dealer is based on a model of the contracting process, developed by akroyd (1987). this characterizes the process in terms of negotiation and risk, and thereby provides an appropriate theoretical framework for activities revolving around the practical application of legal principles. a typical wheeler-dealer roleplay takes place in four phases, two for the relationship stage and two for the dispute stage, as shown in table 1. they are described further in akroyd and edwards (1995a). table 1: the stages and phases in the negotiating process stage preliminary 1. relationship 2. dispute phase introduction la lb 2a 2b title introduction to the exercise negotiation fulfilment litigation enforcement activity creation of legal interests and risks (non-)fulfilment interests (non-)avoidance of risks resolution of disputes enforcement of awards or judgements initial scenarios are given to student teams at the start of each phase, supplemented as necessary in accordance with a developing situation. student teams respond by carrying out five functions: • allocate individual roles; • research legal and factual information; • develop a plan to maximize legal interests with a minimum of legal risk; 104 alt-] volume 3 number i • implement the plan in role-play; • review the outcome of the plan strategically. as well as giving and receiving evidence of transactions in accordance with the applicable law, play includes the obtaining of any registration, consent, sanction or adjudication from the relevant public authorities (all of which will be role-played by a tutor) in the correct manner. this necessitates accurate prior identification of such bodies and careful appraisal of the process involved. one system (sales contractor) to provide decision support for the players has been developed, and another (contract negotiator) is in the process of construction. sales contractor is written in the expert system shell xi plus, and advises on drawing up a contract for the sale of goods (see akroyd and edwards, 1995b). its function is that of a decision-support system rather than that of the classic expert system, since the final choice is always left with the user. this is particularly satisfactory for learning purposes. in wheeler-dealer, it is most useful in helping to draft documentary evidence of sales contract terms and conditions of trade. the detailed wording of clauses or advice was based on akroyd (1984). one screen from sales contractor, regarding provisions for checking the goods, is shown in figure 1. application: sales contractor u. 0.3 knowledaebase: core knowledge «xt«5hi*tiwe clwrce* r«9«fâlins provision for <?3wckii»s the fifi*«*: ciboos« the ottft irfhitk £6tt f e e l is propriât* thft are claused upon receipt of the goods, the buyer will inspect then and uill lodge it is expressif; aijreed. that the buyer will be entitled to inspect and test ;îie «oods dupincr manufacture. brocessin«. or storage, at any tine prior to c13us-0g3 the buyer shall carry out testing operations on the contract goods within :he week innediately following deliuery, in the presence of an agent of the you* chalee c l a u s e d » clausec2 esc caneelf¡cti»l+rtn endí|f3 uhy||rtn sélectif figure / : a saks contractor screen regarding provisions for checking the goods contract negotiator is intended to provide more comprehensive support for the entire 105 john s. edwards and robert /. akroyd using information technology t o help business students negotiation, and is at present under development using the hypermedia package guide. it consists of a series of 36 questions, each relating to a set of legal issues, running through phases la, lb, 2a and 2b (see table 1). this main structure is supported by: • a research menu, allowing access to a digest of legal advice, the online database lexis, and databases of contract and procedural forms; • a risk advisor, in the event of the user wishing to proceed without sufficient proof of relevant legal principle or fact; • an action pad, allowing access to sales contractor, a word-processing package, electronic mail, and in principle also telephone or fax. contract negotiator is at present in prototype form. an example of a question from phase la is shown in figure 2. ^ ^ 2; >an example of a question from contract negotiator for each question, the only possible answers are don't know, no or yes. figure 3 shows the relationship between these answers and the parts of contract negotiator, and how these relate to the functions in the negotiating process. the current (third) prototype of contract negotiator is the first networked version, adding access to lexis and electronic mail to the capabilities of the second prototype. 106 volume 3 number i figure 4: the structure of contract negotiator and its relationship to the negotiating process can you discharge your party's burden of proof for issue k? risk advisor options (n order of increasing risk, ending in advice to stop user may still choose to go on action pad • word processor • electronic mail • sales contractor • goto issue (k+1) research menl digest lexis legal forms • return to main menu (still on issue k) there is no significant software problem in including links to further applications packages, or to communications media such as telephone and fax in the action pad only the hardware problem of ensuring that suitable connections, such as fax modems, exist. mechanical support the use of networked systems to enable file transfer and electronic mail between participants has been technologically feasible for some years, and has now reached the stage where the use of such facilities is sufficiently familiar as to impose no extra burden on business students. however, two related issues remain. the first is that replacing physical paper delivery of documents and face-to-face contact by electronic means may reduce the opportunity for feedback between tutor and student. we believe that an application such as fujitsu's dtc (desktop conferencing) software, enabling tutor and student to work on the same display while in different locations, may avoid this. we hope 107 john s. edwards and robert i. akruyd using information technology to help business students student to work on the same display while in different locations, may avoid this. we hope to experiment with this in the coming year. the second problem is less easily dealt with. under english law, the legal status of a document stored only as a file on a computer is not the same as that of a printed one. it would thus be both unwise and unrealistic to replace paper with electronic 'documents' in wheeler-dealer to any greater extent than is already done in the real world (for example, the use of a word processor to produce standard contracts). evaluation trials of sales contractor have been carried out with commercial law students, who used the system to produce contracts as part of their coursework. this meant that a fully controlled experiment was not possible; it would not have been fair, in academic terms, to have a control group. comparisons are therefore at a broad level, and on an indirect basis, against students undertaking the module in previous years. sales contractor has proved easy to use and robust. specific advantages are: • the production of a contract is very much quicker, taking about five minutes for a user moderately familiar with the choices offered. this compares with a typical time period previously of between one and three hours, often spread over more than one day. • the contracts produced are, in the tutors' judgement, inherently better than those produced previously, even with access to standard practitioners' works such as the encyclopaedia of forms and precedents (1985). this is because the system facilitates the choice of suitable, well drafted provisions, and militates against total omission of provisions, previously a common fault. • the students find the on-screen legal advice more digestible than the textbook format, for two reasons. the first is that it comes in relatively small and manageable chunks, and the second is the use of colour to distinguish references to legal authority and cases from the text of the advice. • the system can be used to produce alternative contract versions for comparison. a number of limitations and weaknesses have also become apparent: • the present system has no facility for the user to modify the provisions. such a facility would have to allow change while preventing the introduction of errors, which is difficult to achieve without compromising the normative effect of the system, since the users, by definition, have little legal expertise. • it is still possible for a user to produce a wilfully or recklessly inappropriate contract. • as a learning aid, the full range of potential clauses on each point has been curtailed. • the suggested clauses are often too long to fit into the fields available. this is because we have been using xi plus running under dos. an implementation based on windows or another graphical user-interface which allowed more flexible scrolling and 108 alt-j volume 3 number i resizing would minimize this problem. we intend to try the version of xi plus for windows which has recently become available. conclusion sales contractor is designed for a specific task, namely producing contracts for the sale of goods, and has already proved useful to business students studying law. contract negotiator will provide more general support for the negotiation of contracts, legal relationships and disputes. more than just a stand-alone system, it is also a structure linking several other applications, including sales contractor (it is due to be tested with students during the academic year 1994/95). the suitability of these two systems for student learning, although they are decision-support systems, arises because they are structured around the relevant legal principles. references akroyd, r.i. (1984), a guide to contracting for the sale of goods, london, routledge and kegan paul. akroyd, r.i. (1987), a guide to contracting: negotiation and the law, london, routledge and kegan paul. akroyd, r.i. & edwards, j.s. (1994), 'wheeler-dealer: a role-play concept for learning about legal negotiation', international journal of independent studies (submitted). akroyd, r.i. & edwards, j.s. (1995a), 'computer-supported contract negotiation: contract negotiator an experimental prototype', international journal of applied expert systems (submitted. akroyd, r.i. & edwards, j.s. (1995b) 'computer-supported contract drafting: sales contractor an experimental prototype', international journal of applied expert systems, 3 (in press). silver, m.s. (1991), systems that support decision makers: description and analysis, chichester, wiley. walton, r. (ed.) (1985 onwards), the encyclopaedia of forms and precedents (5th edition), in 44 volumes, london, butterworths. 109 the implementation of a multimedia learning environment for graduate civil engineers chris smith and peter jagodzinski university of plymouth this paper examines the underpinning theory, design and implementation of a computer-based multimedia learning environment (mle) for graduate civil engineers. the mle brings together multimedia technology and intelligent tutoring systems techniques for the purpose of developing in the graduate engineer the skills and understanding needed to produce initial design proposals for real-world dam spillway design problems. introduction underlying our approach to the development of a multimedia learning environment (mle) is the proposition that conventional learning needs analysis, for computer-based learning does not address the richness of delivery possible with mles. we propose that knowledge-engineering techniques (from the field of artificial intelligence) can provide a richer understanding of the domain, thus enabling a better mle to be produced. the knowledge-engineering approach we have used, outlined in this paper, involved acquiring knowledge from a range of sources, then analysing this knowledge so that individual knowledge items could be categorized according to identified underlying knowledge-representation paradigms which each imply their own, most suitable, presentation style. this knowledge engineering analysis was then used to select an instructional model capable of conveying all the acquired types of knowledge in their paradigmatic styles, and detailed and adaptable enough to be implemented on a computer. the chosen model, the cognitive apprenticeship model (cam), was explicitly used as the basis for the architecture of the mle. this paper is intended to demonstrate the validity of our approach. it concentrates on 29 chris smith and peterjogodzinski the implementation of a multimedia learning environment how the acquired knowledge and its representation styles map onto the cam, the development of an architecture for the mle, and how the core elements of the cam can be implemented within the mle. knowledge acquisition the developers, who are not civil engineers, were able to compare objectively the relative merits of the three distinct views of the domain knowledge and learning needs proposed by different categories of expert: (a) the journeyman. the expert for this part of the process was a post-doctoral research fellow with several years' of recent experience in hydropower design. the first part of the knowledge acquisition process involved this 'journeyman' explaining an example design problem involving the design of an energy-dissipation device for a dam spillway that was taken from the design of small dams (us bureau of reclamation, 1987,400-402). (b) the master. secondly, we referred to a 'master', a professor of civil engineering, for confirmation and higher levels of abstraction of knowledge. (c) the industry. thirdly, we were influenced by our discussions with industry and ice training managers, who identified a number of practical learning needs relating to job performance in the context of the real world. in addition, a number of research studies and models of the design process were examined (jeffries et al, 1981, ullman et al, 1988, and ball, 1990) to provide confirmation for and a more detailed and comprehensive characterization of the design strategies that seemed to be employed by the journeyman as he solved the example design problem. knowledge representation paradigms underlying the acquired knowledge analysis of the acquired knowledge determined that it could be categorized under three broad classes of representation paradigm: 1. symbolic (elicited from the journeyman and confirmed by the master) (a) the conceptual and factual knowledge of natural phenomena and engineered structures which can be effectively expressed as equations, graphs or rules to show the relationships between the values involved. (b) the procedural knowledge of, for example, the steps needed to solve an equation or the steps involved in deriving an equation. 2. experiential (a) knowledge and intuitive understanding of complex real-world physical phenomena which can most easily be understood through some physical sensory process, for example the erodability of a stream bed or the buildability of a concrete form. 30 alt-] volume 3 number i (b) intuitive understanding of values, relationships and abstract theoretical constructs (which may not be observable in the real world) which can be expressed symbolically but which are hard to grasp unless they can be sensorily experienced, for example the force of 1,000 cubic meters of water flowing down a spillway, the kinetic energy of water before and after a hydraulic jump, or the triangle of forces. type (a) was disclosed by the master, and type (b) was stressed by both the master and the journeyman. 3. metacognitive, i.e. the framework within which the symbolic and experiential knowledge is applied in order to accomplish a task. (a) high-level project structuring metacognitive knowledge, i.e. knowledge that enables the engineer to identify and sequence project activities to provide an activity network for, say, a two-year project. the importance of this type of knowledge was emphasized by industry practitioners. (b) lower-level (applicable over a shorter time frame, say one day) knowledge of strategies and heuristics which are needed to be able to apply the symbolic and experiential knowledge so that a small task within the project life-cycle can be accomplished. these strategies were elicited from the journeyman during his solution of the example design problem, and were confirmed from examination of models of the engineering design process. representation styles required by the three categories of knowledge each category of knowledge seems most suited to particular methods of presentation: symbolic knowledge (for example mathematical relationships between flow rates and spillway dimensions) needs to be expressed in symbolic terms, i.e. equations or rules. however, much symbolic knowledge can often also be expressed using experiential techniques such as those involving simulation, animation, video and still images, so that it becomes understandable at an intuitive level through sensory experience. metacognitive knowledge is often learnt through experience, for example the patterns of procedures and approaches which best suit particular real-world problems. although it would be possible to teach such strategies and heuristics as rules (i.e. symbolically), in reality a thorough understanding of the subtleties and variations of strategy is acquired only through experience. such experience can be offered on computer by simulating realworld problem-solving. experiential knowledge can thus be seen not as a separate category but as an important alternative perspective on symbolic and metacognitive knowledge which enables these to be understood intuitively as well as formally. a more detailed account of knowledge elicitation and analysis is given in jagodzinski et al. (1994). 31 chris smith and peter jagodzinsti the implementation of a multimedia learning environment a simplified outline of some of the major elements of the cam in order to provide an effective, integrated learning experience, a suitable psychologically recognized, coherent and comprehensive framework was required. after reviewing several approaches, such a framework was found in the cognitive apprenticeship model (cam) developed by collins et al. (1989) and collins (1991). the cam is an adaptation of the traditional approach of craft apprenticeship for skill-learning into methods for teaching and learning the reasoning and problem-solving cognitive skills of a domain. its main elements are listed in table 1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. content 1.1 domain knowledge 1.2 heuristic strategies 1.3 control strategies 1.4 learning strategies modelling and explaining 2.1 modelling of real-world processes 2.2 modelling of expert performance situated learning coaching scaffolding fading articulation reflection exploration 10.1 increasing complexity 10.2 increasing diversity 10.3 global before local skills table i: elements of the cognitive apprenticeship model (cam) the knowledge content of the cam approach is defined in the first two of its elements, as shown in figure 1 which illustrates way in which the acquired knowledge and its representation styles map onto the cam. domain knowledge (1.1) corresponds with conventional declarative text-book knowledge, and 1.2 to 2.2, various aspects of procedural knowledge including problem solving strategies and modelling real-world performance. together, these two aspects of the cam are able to comprehend the type of knowledge representation required by the domain of dam spillway design. the instructional process of the cam (elements 3 to 10 in table 1) conveys metacognitive knowledge and improves understanding of the symbolic and its related experiential knowledge by placing all teaching within the context of real-world problem solving. this is accomplished by having the tutor first model the whole solution to a problem (element 2.2 in table 1). then work begins on a similar problem with the tutor carrying out some parts of the problem and the learner attempting the other parts of it. work continues on similar 32 volume 3 number i domain of a dam spillway design experiential (a) dynamic, complex real-world physical phenomenon (b) values, relationships and abstract theoretical constructs metacognitive ") *^^_ __^-— cam 1. content 1.1. domain knowledge 1.2. heuristic strategies 1.3. control strategies 2. modelling and explaining 2.1. modelling of real world processes 2.2. modelling of expert performance figure i: how the knowledge representation paradigms of the domain of dam spillway design map onto the types of knowledge recognized by the cam new problems with the learner taking on as much of the new problem as possible, as soon as possible, until he or she can comfortably solve a whole problem on his or her own. having the tutor carry out parts of the problem is termed 'scaffolding' (element 5 in table 1) within the cam, and having learners take on more and more of the problem by themselves is known as 'fading' the scaffolding (element 6 in table 1). the cam offers guidance on the provision and removal of scaffolding. initially, support should be offered to learners to relieve them of the lower-level skills needed to perform a complex task so that they first examine, practise and learn the higher-level reasoning skills necessary to carry out the task. they thus develop a conceptual map of the overall activity (which is also gained from modelling) before they go on to develop specific skills. this is known as teaching 'global before local' skills (element 10.3 in table 1). the cam also suggests that coaching (element 4 in table 1) should be provided. this involves monitoring learners as they work, and offering hints, scaffolding, feedback, modelling, reminders, and new tasks, all of which are intended to move their performance closer to the performance of an expert. 33 chris smith and peter jagpdzinsti the implementation of a multimedia learning environment the way in which these core cam elements of modelling, coaching, scaffolding and fading can be provided by the mle will be discussed below. a detailed examination of the other important aspects of the cam is given in brown and newman (1989) and collins (1991). an architecture for the mle figure 2 provides a schematic overview of the components of the mle. the 'coach' first models the expert solution of a problem. the learner is shown the solution sequentially, a page at a time. pages already seen can be re-examined by being 'shelves' d coaching books & coached examples visuarcard index* reference 'books* '•s declarative knowledge -a author 'card index* project documentation 1 per problem attempt subject 'card index* example problems various versions vit h differing levels of scaffolding active passive figure 2: mle architecture the text 34 alt-] volume 3 number i scrolled through. after being shown how to solve a problem, the learner can then sequentially work his or her way through a similar type of problem but one which uses different values. the system will support the learner by alternately demonstrating the solution to part of the problem and then asking him or her to attempt part of the problem, and so on. this is done according to the 'global before local' sequencing of skills. as a case study problem is worked through, values are filled in on examples of project documentation held within the 'project folder'. this provides students with a correct model of the appropriate ways of recording their findings. 'books' of reference material are referred to in the 'coach' to provide background understanding for learners and to help them complete their parts of the task. the 'books' are structured and contain organized multimedia explanations and demonstrations of symbolic and related experiential knowledge which can be used to solve many different types of problem (see figure 3 for examples). 'books' are referenced within 'card indexes' and the coached examples themselves are also initially taken from an 'index'. all 'books', 'coached examples' and 'indexes' are available on a number of'shelves'. after completing a section, learners ask for it to be evaluated. if they have done badly, they can be helped. for example, the system can demonstrate the solution to that section, and levels of demonstration with varying detail can also be shown. if learners reach an impasse, they can, again, be given similar help. how well a learner does overall on a runthrough determines how much of the scaffolding is faded for the next coached run-through of a similar problem with different values. the coach the following is a simplified outline of the underlying architecture and mechanisms needed to implement the cam's core elements of modelling, coaching, scaffolding and fading. modelling each of the tasks in figure 4, which shows the decomposition of a typical problem into individual steps, would be represented as a node within the system. each node has a set of 'system demonstration' pages and a set of 'student work-through' pages associated with it. the system demonstration pages illustrate how an expert would solve the task, and the student work-through pages invite students to solve the task for themselves. initially, the expert's solution to the problem, using a particular set of values, is modelled. this is accomplished by using a stack to deliver the system demonstration pages for each node, in turn, i.e. a, b, c, d, e and f. scaffolding and fading each leaf node in figure 4 (b, d, e and f) also has an evaluation procedure, a student score and a difficulty rating and sequence ranking (see figure 5). a task's difficulty rating is a measure of how complex and time-consuming a task is to complete. the task sequence rankings are used in the process of allocating the next task to be attempted by the student. 35 chris smith and peter jagpdzinski the implementation of a multimedia learning environment look and feel of a real book. user can turn book over, flip pages, turn a page and drag bookmarks. 1 m • 1% h h h ĥ y» 1 * ! *** | m —&ç 'jl v u m • some items, e.g. pictures, video and simulations that are too big to be placed on the page full-size, appear shrunken on the book and can be zoomed in and out. 'books' not full screen size. the 'coach' and a number of books' can appear on screen at once (overlapping). figure 3: examples of screens from a book 36 alt-j volume 3 number i a — b r<n c l — f e figure 4: a generic design problem decomposed into a number of tasks on a 'scaffolded run-through', the same design problem is used but with different values, so different choices may have to be made. for example, in one of our civil engineering problems, different types of spillway stilling basin may need to be considered. when 'scaffolding', the system will demonstrate some parts of the problem, and the learner will attempt other parts of the problem. the basic method for allocating tasks to the learner uses the students scores, difficulty ratings and sequence rankings: • leaf nodes are ordered according to their sequence rankings and difficulty ratings: f (sr = 1, dr = 40), e (sr = 2, dr = 10), d (sr = 2, dr = 20), b (sr = 3, dr = 30). • say 50% of the given problem (in terms of difficulty) is to be allocated to the learner on dr = 30 dr = 20 sr = 2 f dr = 40 s r = 1 dr difficulty rating, sr sequence ranking a 'real' problem would include many more task nodes. figure 5: design problem nodes with example difficulty ratings and sequence rankings 37 chris smith and peter jagpdzinski the implementation of a multimedia learning environment the next run-through (the higher a learner's overall score on a run-through, the more of the problem he or she will have to attempt on the next run-through). • 50% of the total difficulty of the whole problem is calculated, i.e. 50 (in this case). so, a number of tasks with a difficulty totalling 50 (or as close to it as possible without exceeding it) will have their node settings set to show student work-through pages. thus, during the run-through, at these points, the student will be asked to attempt those tasks on his or her own. • the first tasks whose difficulty ratings total 50, or as near as possible to 50, are allocated to the student according to his or her sequence rankings. so, on the next runthrough, nodes f (sr = 1, dr = 40) and e (sr = 2, dr = 10) will be allocated to the student to attempt. • the sequence rankings are intended to deliver the cam's 'global before local' sequencing because the centrally important guiding tasks and decisions will be given higher sequence rankings in order to ensure that students will practise them earlier. coaching each section attempted by a student is evaluated. if the student does badly on a section, he or she can be shown the system demonstration pages for the node concerned, its child nodes and so on (varying demonstration/help pages can be given depending on the errors made). if a student is unable to continue on a section, again he or she can be shown the system demonstration pages for the node concerned and its child nodes (and, again, varying demonstration/hint pages can be given depending on the experience of the student, for example how much of the problem he or she has been allocated). conclusion our use of a knowledge-engineering approach to the development of multimedia learning environments has brought four benefits: 1. recognition of the need for a range of domain expertise. 2. a richer perception of the learning domain three categories of knowledge have been recognized, and the importance of experiential knowledge in particular has been an unexpected revelation. 3. a set of paradigms for the representation of knowledge in the architecture of the mle. 4. a basis for selecting an instructional model for the architecture of the mle which is capable of integrating and delivering the different types of knowledge in their most appropriate representation styles. the cam model has been of enormous value in providing us with a clear and detailed rationale for the way in which material is presented to learners, for the modelling of their performance and for the interaction between these two elements. the premise of this research was that careful attention to design can unlock the potential 38 alt-j. volume 3 number i of multimedia as a learning tool. we believe that we have shown that the approaches we have used are capable of facilitating a penetrating understanding of a learning domain, and consequently a rich design which situates the learning environment in the context of realistic problem-solving. references ball, l.j. (1990), cognitive processes in engineering design, ph.d. thesis, university of plymouth. collins, a., brown, j.s. and newman, s.e. (1989), 'cognitive apprenticeship: teaching the crafts of reading, writing and mathematics' in resnick, b. (ed), essays in honour of robert glaser, hillsdale nj, lawrence erlbaum. collins, a., (1991), 'cognitive apprenticeship and instructional technology' in idol, l. and jones, b.f. (eds), educational values and cognitive instruction: implications for reform, hillsdale nj, lawrence erlbaum. jagodzinski, p., parmee, i. and smith, c. (1994), 'the use of multiple paradigms for the representation of knowledge in a multimedia learning environment for engineering design', paper submitted to the internationaljournal of human-computer studies. jeffries, r., turner, a.a., poison, p.g. and atwood, m.e. (1981), the processes involved in designing software' in anderson, j.r. (ed), cognitive skills and their acquisition, hillsdale nj, lawrence erlbaum. ullman, d.g., dietterich, t.g. and stauffer, l.a. (1988), 'a model of the mechanical design process based on empirical data', ai edam 2, 1, 33-52. united states department of the interior, bureau of reclamation (1987), the design of small dams (third edition), denver co, united states government printing office. 39 the use of computers as substitute tutors for marketing students miriam catterall and patrick ibbotson school of management, university of ulster the use of computers as substitute tutors is associated primarily, though not exclusively, with multiple-choice question formats (ellington, percival and race, 1993). we report the findings of a project that involved the design, testing and evaluation of a set of computer-based tutorials employing multiple-choice questions with 700 students on postgraduate and undergraduate introductory marketing modules. the computer-based tutorials were designed to meet two main objectives, namely to help students in their formative assessment and to help staff monitor any difficulties students were experiencing with module content. however, students incorporated the tutorials into their learning in ways that had not been anticipated specifically; they used the tutorials for a number of related but different purposes, and their usage patterns varied considerably. introduction the importance of feedback in the learning process is self-evident and well documented (rowntree, 1987; brown & knight, 1994). students tend to place high value on the feedback they obtain from teaching staff through the assessment process. assessment can consume as many staff hours as teaching (gibbs & jenkins, 1992). with increasing class sizes, teaching staff are encouraged to find ways to reduce the demands the assessment process places on their time. peer and computer-marked assignments can help in this respect. our aim was to develop a computer-based tutorial (cbt) that would provide students with early and frequent opportunities to assess their own progress. the cbt contains eleven separate tutorials corresponding to the main topic areas covered on the introduction to marketing module. when students select a tutorial, they are presented, with twenty-five multiple-choice questions (mcqs) selected at random from a bank of over fifty. students have two chances to answer each question, and if they still answer incorrectly they are told the correct answer and directed to the appropriate page in the text (see figure 1). on completion of the tutorial, students are given their score and taken 86 a i t j volume 3 number i if uonfiir, a hairdryer manufacturer« is focusing on custoner satisfaction, it uill find that what corauners really want i figure /: typical mcq from the test bank back to the main menu where another choice can be made. students are able to access the cbt on campus machines or, if they have a machine at work or at home, a take-away version is available on floppy disk. a pre-pilot study with fifty-seven students during the 1993 summer semester revealed that students were using the cbt more frequently and for longer periods of time than was anticipated. their recommendations for improving it suggested they were using it for reasons other than self-assessment. the main pilot was undertaken during the winter and spring semesters of 1993-94, during which time over 700 students had been given access to the cbt. log files were used to capture data on the frequency and patterns of cbt usage on both onand off-campus machines. feedback from a sample of student users was obtained by structured questionnaire, unstructured interviews with individual students, projective techniques and direct observation. the main pilot confirmed earlier findings, and these are reported in detail elsewhere (catterall & ibbotson, 1994). in the remainder of this paper, we discuss the findings of both the preand main pilot studies, and present our understanding of how and why the cbt is incorporated into the student learning process. main findings part-time students are the heaviest users part-time students were more likely to use the cbt, use it more frequently and for longer periods of time. for example, single sessions of up to one hour were not uncommon among part-time students. 87 miriam caaerall and patrick ibbotson the use of computers as substitute tutors one of the reasons for this difference was the availability of the cbt on a take-away disk for use with off-campus computers, at home or at work. while under 10% of full-time students obtained take-away disks, over 60% of part-time students did so. however, parttime students without access to off-campus machines were more likely to use the cbt than their full-time counterparts, even though they are on campus for considerably fewer hours per week. use decreases over time use of the cbt decreased over time among all groups of students including part-timers and those with access to off-campus machines. to illustrate: 74% of a group of full-time undergraduates used the cbt regularly over the first half of the semester; during the second half, only 32% were still regular users. there are a number of possible explanations for this finding: mcq assessment. half of the groups or classes who participated in the pilot study were given an mcq test (pen and paper) as one of their coursework assignments. this mcq test was administered at the halfway point in the semester. it is reasonable to assume that a number of cbt users were practising for this test, and on its completion would no longer feel the need to use the cbt. however, since cbt usage decreased also in groups that were not assessed by mcq test, this is not a complete explanation. novelty. very few students who participated in the pilot had any previous experience of this type of computer-based learning. it could be hypothesized that a new and different learning resource will hold some novelty value for students that will wear thin after a few use-experiences. evidence to support or reject this is not immediately apparent. we did find some evidence that newer learners were more likely to perceive the cbt as a useful learning resource than more experienced learners. full-time postgraduate groups (experienced learners) were the least likely to use the cbt. a number of students in these groups reported that 'real' study involves texts and journal articles, and that in their perception cbts are of marginal value. by contrast, students new to the university environment (first-year undergraduates) and those returning after a break from formal education (part-time postgraduates) were more likely to use the cbt and look upon it as part of the total package of learning resources. one group of parttime undergraduates who studied marketing as their first module assumed that other modules would also be offering cbts as part of the learning resources, and were surprised and disappointed to find this would not be the case. the need for support reduces? on the basis of the evidence we have on how students use the cbt and the reasons why they use it, it appears reasonable to conclude that the need for the type of learning support provided by the cbt will decrease over time. students used the cbt in a variety of ways and for,/different reasons, including: preparation for learning. this involves working through a tutorial as quickly as possible (any answer will do) to identify the subject matter of the questions and the concepts or alt-] volume 3 number i learning points that are questioned most frequently. this gives the student clues as to which concepts or issues are most important and where effort needs to be concentrated. this is likely to be more important for students in the early weeks of a new module when every bit of information seems vital. as the weeks progress, they may be better able to differentiate between key learning points and interesting information during lectures. confidence building. the student has not read the lecture notes or the text, and works through the cbt trying to answer the questions correctly. good luck or common sense will ensure that they get some answers correct. students report that this gives them a sense of achievement and confidence. in other words, they already know something about this subject even before they begin to study it. they report that this makes the study of text and notes a less daunting prospect. this is also likely to be more important to the student during the first weeks of a new module. as they learn more about the subject and become familiar with the text, they may need this type of cbt support less frequently, if at all. studying. the student has not studied the lecture notes or the text, though he/she may have read them through once or twice. the student works through the cbt with the notes and text to hand, answering the questions and checking the appropriate reference in the textbook. as the weeks progress, and the student becomes more familiar with the material, the need for this type of support should diminish. this type of studying appeared to be very time-consuming though many students continued to use it throughout the semester. when asked why, the most frequent responses were that they preferred to learn this way and it helped them concentrate. self assessment. after having studied the material, students use the cbt to assess their learning. when asked about the benefits of this approach, they reported that it gave them a sense of achievement and confidence. part-time students reported that it was important to feel they had achieved something early on in the module. in other words, in the first few weeks they were uncertain as to whether they had made the right decision in returning to study: would they be able to study, were they studying enough or too little? the early feedback from the cbt provided reassurance in these respects. again, it could be argued that as students' confidence increases, the need for this kind of cbt support reduces. drawbacks with heavy cbt usage the design of the cbt was based on the following assumptions: • a typical cbt session would involve the student using a single tutorial; and • a single tutorial would typically take up to fifteen minutes to complete. the data reveals that there is no typical session or typical session-length. some students would use five or more different tutorials or the same tutorial five or more times in a session. very few users would undertake only one tutorial in a session. tutorial times varied in length between four minutes (presumably for preparation for learning) to forty minutes or more (probably using with a textbook). 89 miriam catlerall and patrick ibbotson the use of computers as substitute tutors given the long duration of many individual sessions, it is not surprising that a number of students reported suffering from eye strain. it emerged also that question design can contribute to this; specifically, questions where the answer options were long and wordy did not work well. ideally, when designing mcqs for the screen, the answer options need to be short and sharp. a number of students reported that the cbt was easy and fun in the beginning but became monotonous and boring in later weeks. this is understandable, given the very heavy use made of the cbt by these students in the first half of the semester. we have addressed this issue in two ways: more variety in question design. the mcq format is flexible and offers many advantages over other question types (ebel & frisbie, 1986). the addition of graphics to illustrate questions should be an improvement on the current all-text format. the number of answer options is now more varied, ranging from two to five, when previously all questions had five options. the design of open questions will reduce the monotony of simply using the mouse to point to and select the chosen response. progression. one of our original design specifications was that progression should be built into the cbt. specifically, later tutorials should contain more difficult questions than earlier tutorials, and later tutorials should build on knowledge and comprehension of earlier tutorial material. earlier tutorials do contain a higher proportion of easy questions, and this was identified by students. the proportion of difficult questions in later tutorials has been increased. designing mcqs is acknowledged to be a time-consuming and difficult task (osterlind, 1989). designing mcqs sufficiently short and sharp for the screen that also build on earlier tutorial material is proving difficult to realize. conclusions the most disturbing finding from the pilot study was that the number of users of the cbt and their frequency of use decreased over time. further, some of the disadvantages identified by students give cause for concern; notably that it becomes boring and causes eye strain. however, it would be inappropriate to take these findings at face value and conclude that the cbt has failed to engage students sufficiently. the cbt was created with the overall aim of providing students with early and frequent opportunities to assess their progress on the marketing module. in the event, selfassessment was only one of the reasons why students used the cbt. students were as likely to use the cbt to help prepare them for learning and to build confidence as to selfassess what and how much learning had been achieved. in this respect, the cbt proved especially useful for part-time students who needed the reassurance in the first few weeks of the module that, under ideal circumstances, a human tutor would be able to provide on a one-to-one basis. as students gain confidence and familiarity with the subject matter and their study of it, the need for the kind of support the cbt provided will reduce. 90 alt-j volume 3 number i the disadvantages identified by students need to be addressed. for example, if some students are experiencing eye strain from sitting for long periods in front of the screen, instructing them not to do so is unlikely to solve the problem. similarly, if students get bored because they use the cbt far more frequently than was ever intended, telling them to use it less frequently is not a helpful response. to this end, we have attempted to introduce more variety into the question format, reduced the amount of text that appears on screen and made tutorials progressively more challenging. the pilot study revealed the likely demand among students for computer-based learning resources that can be used off the campus. the majority of part-time students have access to off-campus machines. there are a number of advantages in encouraging students to use cbt off-campus material, not least of which is that it reduces the demand for oncampus machines to the benefit of students without off-campus access. marketers employ research to identify how consumers perceive and relate to products, since consumers will identify new uses and benefits not envisaged by manufacturers. our pilot study illustrates this applies equally to cbt development. references brown, s. and knight, p. (1994), assessing learners in higher education, london, kogan page. catterall, d.m. and ibbotson, p.g. (1994), 'the development of a low technology marketing cbt', account, 6, 1 (spring). ebel, r.l. and frisbie, d.a. (1986), essentials of education measurement, london, prentice hall. ellington, h., percival, f. & race, p. (1993), handbook of educational technology, london, kogan page. gibbs, g. and jenkins, a. (1992), teaching large classes in higher education, london, kogan page. osterlind, s.j. (1989), constructing test items, boston ma, usa, kluwer academic publishers. rowntree, d. (1987), assessing students: how should we know them?, london, kogan page. 91 editorial mind synthesis the president of the usa and i have something in common: we play the saxophone. and having briefly heard clinton play, i would say we are at roughly the same acceptable, though unmistakably amateur, standard. nevertheless, we are no doubt both so attuned to the sound of the saxophone that it would not be easy to fool either of us into believing that a real instrument is being played if in reality the sound of the instrument had been electronically synthesized. certain instruments are easier to synthesize than others. for instance, it is hard even for sensitive ears to distinguish between a genuine church organ and a church-organ sound coming out of a good synthesizer. this is partly because the sound quality of the organ is not dependent on the organist: a two-year-old child who presses an organ key achieves no better or worse sound quality with that key than the most gifted organist. on the other hand, a synthesized solo saxophone (like most synthesized instruments) tends to have a certain unnatural, mechanical quality which is difficult to pin-point but which to the saxophonist immediately rings false. it was therefore with astonishment that i recently listened to two solo saxophone pieces, one played on a real tenor saxophone by a top-flight jazz saxophonist, the other the result of some keyboard input and clever programming by computer scientists who may well have had only a rudimentary knowledge of music and perhaps no playing skills whatsoever. i was quite unable to say which of the two was synthesized even after repeated listenings. the fact is that the perceived perfection achieved by a professional saxophonist is not perfection at all but an amalgam of skill, flexible reactions and numerous little human errors, and provided that this skill, these reactions and these errors can be identified and quantified clearly, they can they can all be coded so that the result is perceived perfection. in short, perceived perfection in instrumental synthesis seems to be merely a matter of how long a programmer is willing to spend on fine-tuning the program code. can the same sort of thing be applied to the sole use of educational technology as opposed to using human teachers? the problem for educational technologists writing cal programs is one of producing a level of flexible reaction and interaction equal to that which an experienced human teacher can offer. is it thus only a matter of the amount of care and effort put into the programming? can a brilliant, patient programmer, drawing on the expertise of brilliant human teachers, produce the software equivalent of a brilliant human teacher? if not, might this be possible in the foreseeable future? the answer to all these questions is i would have thought obviously no, except within certain categories of training software involving repetitive tasks and/or motor skills (and even then . . . ) . of course, there is hardly an educationist who will not add when theorizing about the use of technology that a human teacher will always be necessary, but i often get the feeling when reading academic papers concerned with the theory of learning technology that remarks about the need for human intervention are there either as comforters or safety nets to catch the crystal ball if it eventually has to be discarded as junk. indeed, i detect among many academics writing about software for use in education a deeply held belief that technology will soon reach in certain cases, has already reached a stage at which the human teacher can be dispensed with. yet the number of potential variables in most teacher-learner situations is vastly greater than the number of variables involved in accurately reproducing the sound of a musical instrument so great, in fact, that even imagining the technology of 10, 20 or more years from now, it is hard if not impossible to envisage a teaching machine so advanced that it could entirely replace a human being for all but those learning situations where understanding concepts is of secondary importance. medical diagnosis entirely by computer, pilotless 2 alt j volume 2 number 2 passenger jets, robots that do the housework . . . all are conceivable within the lifetime of today's young people. but despite the uncanny accuracy of some of arthur c. clarke's predictions of thirty or so years ago in 2001: a space odyssey, the chances of our producing by the beginning of the 21st century (or even well beyond) a computer like hal, able to react more or less like a human being, are inflnitesimally small; and it would take a hal to make human teachers completely redundant. it is always dangerous to predict the pace of technological progress, but the teacher-student relationship is potentially so complex that encompassing all possible subtleties of it in a teaching machine seems as out of reach as a manned mission to the nearest star or accurately forecasting next week's weather. two things flow from this. the first has become almost a cliche: that human teachers who fear redundancy because of educational technology are needlessly concerned. in the 1960s, the language laboratory appeared to many language teachers as a threat to jobs, when in fact there was no more possibility of their being replaced by a tape recorder than there is now by a computer. the second is that we must always approach with a degree of healthy circumspection the value of educational software in the comparatively inflexible form it has now and will certainly have for a long time to come. in the 1950s, the behaviourist psychologist b.f. skinner believed that the teaching machine had such obvious advantages infinite patience, consistent reaction to student input, the offer of self-paced learning, the fascination of youth with technology and so on that any disadvantages were far outweighed, and this at a time when teaching machines were rudimentary and cumbersome affairs by modern standards. skinner's ideas about programmed learning were later toppled from first place in the educational popularity league by bruner and other advocates of discovery-based learning; but again, with the advent of hypertext and hypermedia encouraging browsing and guesswork, the by now more manageable teaching machine still fitted nicely into the scheme of things. today, if anything, the pendulum is swinging back to the apparent security of the step-by-step approach, though neural networks look set to prolong the popularity of hypermedia as a discovery-based learning tool. the teaching machine will thus doubtless continue to be promoted, if only to relieve pressure on teachers faced with ever larger numbers of students. it is right to promote it, but we should also continually re-assess the extent to which it can successfully be used in the absence of a human teacher. in my view, it can very successfully be used, provided that two basic conditions are met. the first is that human teachers are fully aware of the dangers of non-intervention in the teaching environment. the second is that both teachers and students are fully aware of the limitations of cal software, and therefore prepared to question computer-generated responses and assessments. a computer program is the result of decisions initially made by human beings, and it is human not only to err, but also to be so unimaginably complex that no machine yet designed can come anywhere near an acceptable synthesis of the way our minds work. those of us involved in educational technology should therefore be blowing our own trumpets but not before having pushed some mutes into their bells. what i say here is not new. but i believe it is certainly worth repeating. gabriel jacobs 3 reviews successful instructional diagrams by ric lowe, london, kogan page, 1993. isbn: 0-7494-0711-5. this publication is part of the educational and training technology series edited by chris bell of the university of plymouth. it is aimed directly at teachers, trainers and instructional designers who wish to incorporate diagrams into their instructional materials. the first section of the book deals with diagrams in relation to learning processes and the acquisition of visual, as opposed to textual, information. although a series of important points are clearly indicated, there is no reference to published work in the field, and this weakens the value of this section of the book. furthermore, there is a bibliography but it contains just fourteen entries. this is not therefore a book to use in researching visual literacy. the central section of the work is concerned with the design and production of instructional diagrams, and here there is a wealth of straightforward information and ideas presented in a clear and helpful manner which would benefit even experienced instructional designers. one feature of this section in particular, but also of the book in general, is the use of a common thematic topic (a pepper mill) for exemplary illustrations. this technique allows the writer to focus on specific features of the illustrations without introducing new, distracting information. having said that, i did feel that the book might have benefited from including, at least in one section, a wider variety of illustrative examples. the final section is concerned with the role of diagrams in an instructional context and the potential problems which a learner might have in using diagrams. again the approach is clear and useful. the book is directed at less experienced instructional designers or those who feel they should be exploiting diagrams more completely. it contains a range of useful procedural checklists, and draws together a number of selected issues ranging from design considerations to evaluation procedures. it is a useful addition to the practitioner's bookshelf. dennis moss, university of wales at cardiff hypertext in context by c. mcknight, a. dillon and j. richardson, cambridge university press, 1991. isbn 0-52137488-x. the term hypertext was coined by theodore nelson in 1967. he used it to describe interactive systems which combine natural language text with a computer's capacity for interactive branching. since it is now possible to combine text with a variety of other forms of presentation (such as graphics, audio and video), the term hypermedia is now often adopted for such systems, hypertext being reserved for applications based on just non-linear text. in this book, however, mcknight et al refer to applications which embed document or information linking as hypertext, regardless of the modes of presentation they employ. in other words, the term hypertext is used in a generic way to refer to electronic documents which utilize any number of presentation modalities. 71 reviews the book is divided into eight chapters which together chart the development of hypertext from aristotle (and the earliest notions of inter-connected information) through the history of writing and hypertext applications to future potential. chapter 1 provides a discussion of the meaning (or definition) of hypertext and its potential impact on the way people organize information. it is suggested that the 'real' potential of hypertext goes beyond the provision of new ways of reading and writing to fundamental changes in the way people think. this is followed by a discussion of some of the historical highlights in the development of the hypertext paradigm, including mention of some of the key figures in the evolution of the concept. in chapter 2, the case for hypertext providing new ways of thinking is further endorsed. however, it is argued that many conceptions about the linearity of traditional textual systems are inaccurate, and the basic character of both oral and written information is questioned. this is used to focus on the idea (popularly attributed to mcluhan) that the medium determines the message. the evolution of written language is then discussed in great detail, with emphasis being given to changes in the style and purpose of written material. within this context, various techniques for the production of textual resources are described (such as printing and the organization of printed material into books). there is then a discussion of the impact these developments have had on the cognitive skills of readers. chapter 3 concentrates on a reader-centred analysis of hypertext. the fact that paperbased text and hypertext constitute different media is noted, and this is followed by a psychological and physiological discussion of reader characteristics. this includes a description of the physical characteristics of the human visual system and the findings of research into the cognitive ergonomics of screen-based text. then a classification for information types is developed. the classification of textual documents is based on three dimensions which relate to: how they are read, why they are read, and what information they contain. this brings to the fore a clear concern with looking at hypertext in the context of actual tasks, since a simple comparison of the effectiveness of paperbased text as opposed to hypertext is unrealistic without an account of the types of task users wish to perform with the particular information resource in question. chapter 4 deals with the idea that departure from linear information structures can cause users to become lost within an information space. the psychology of navigation is discussed both within the context of paperbased and electronic documents, taking into account the types of schemata and models used for navigation within textual environments. the way in which users can construct cognitive maps of the information space is dealt with, along with the ways in which such maps can provide navigational knowledge. there is also an outline of the ways in which metaphors can be used to address navigational problems. conversion from text to hypertext is the subject of chapter 5, where techniques for converting paper-based resources are contrasted with approaches involving the development of original hypertext materials. the underlying structures are also treated, and consideration given to implementations which retain an underlying linear structure while facilitating web-like semantic links throughout the knowledge corpus. semantic networks are described along with the deficiencies that can result from embedding information within a network structure. 72 alt-j volume 1 number 2 chapter 6 deals with the potential effectiveness of hypertext within applications aimed at learning and education. a general view of education is briefly covered, followed by a look at the question of whether hypertext can play an effective role in the fulfilment of educational goals viewed in terms of both developmental and performance criteria. individual learner characteristics are also briefly discussed along with the concept of incidental learning which may occur while browsing. chapter 7 presents an outline of a practical implementation of a hypertext system. the hypertext database is an application intended to support research workers with the objective of discovering whether a hypertext database can be of significant use to people who would ordinarily use a paper-based equivalent. the system is described in detail along with the findings of experimental work into its effectiveness. the final chapter, not surprisingly, looks at the future potential of hypertext, and identifies some key areas for future research. areas such as computer supported co-operative work and intelligent hypertext are briefly treated, and an attempt is made to clarify the overall position of the authors with regard to hypertext. there is a glossary, a reference list, an author index and a subject index facilities essential in a book like this which makes significant and useful reference to related work. the glossary is atypical, however, since it consists entirely of short descriptions of hypertext systems, not definitions of keywords. i cannot fully endorse all the ideas presented in this book, but it is readable, and it will no doubt prove useful. stephen richards, university of teesside hypertext: a psychological perspective edited by c. mcknight, a. dillon and j. richardson, chichester, ellis horwood, 1993. isbn: 0-13-441643-0. this is an edited volume of seven chapters, each discussing the design of electronic text and how that design might be informed by psychological theory. good coverage is given to most of the key controversial issues. this could be a useful book for students as well as a provocative one for researchers active in this area. the editors adopt a commonplace view that there are really three relevant traditions of psychological theorizing to contemplate. first, there is behaviourism. hardly anyone claims behaviourism as an inspiration any more, but custom requires that it is identified and vigorously dismissed. certainly, no chapter here represents this position. secondly, there is cognitive psychology/science. nick hammond contributes a very readable chapter exploring the dominant themes from this tradition. he briefly reviews key empirical research, and derives a sort of learning cube: a figure comprising three dimensions of computer-based learning (and finding accommodation for hypertext in a rather murky corner of this shape.) behaviourism and cognitivism differ in their preferred conceptual vocabulary for analysing knowing and learning. in particular, they disagree on the need to theorize in terms of symbols and symbol-relations within private representational systems. this seems a fairly stark alternative: deal with behaviour or with information processing. it is hard to imagine how there could be an in-between perspective. but a third theoretical tradition identified here is claimed to occupy the middle ground. this third tradition is constructivism. donald cunningham and colleagues articulate such a position and illustrate, step 73 reviews by step, how electronic media can be made to harmonize with it. of the other chapters in the book, three are focused on more particular features of psychoogical theorizing and how they inform the development of hypertext. so, patricia wright considers what can be learned from empirical research on reading. then the editors themselves have written a chapter locating 'navigation' as a general topic within psychology and, of course, relating this to thinking about hypertext development. finally, david jonassen reports some studies exploring the effects of making the semantic network of expert knowledge explicitly available in the linking structures of a hypertext application, thus relating hypertext design to models of semantic memory. there are two more chapters. peter whalley starts the book with a critical essay on the (liberation) rhetoric of hypertext, and thomas landauer and colleagues describe a series of elegant experiments that make up the formative development of their highspecification electronic book. landauer's chapter gives the strongest sense of progress arising from research (it is also the longest piece occupying a third of the book). but the chapter reminds us of an important distinction: electronic text for information retrieval versus for learning. these resear-. chers are clearly working with the former: their 'superbook' is a technical manual for telephone engineers. the challenge of designing this kind of document seems more amenable to cognitive psychological research methods (and less dependent on highly articulated theory). most of the other authors in this volume are oriented towards creating tools for learning, and here progress seems more halting. in the end, i fear the truth may be that psychology is not delivering the goods for this enterprise. for example, wright feels obliged to conclude: 'despite a century of work on the psychology of reading there is no theory of the reader that designers can employ to develop a usable hypertext document from first principles.' for me, this tends to expose psychology as a rather narrow discipline. here is a hint of it taken from within this book: studies described in one chapter are said to be 'concerned with the psychology of information access by humans.' 'humans'? well, this is a discipline over-anxious to affiliate with biology (and, of course, computer science). a discipline where people participating in research are often termed s's. in short, psychology is curiously isolated from the culture of other social sciences and not very interested in their progress. for problems defined in the present context, this probably matters. so, greater sensitivity to developments in literary theory, discourse analysis, ethnography or semiotics might enrich empirical research on some of the problems defined in this book (for example, the topical social science interest in 'narrative' rather passes psychology by). psychology is also a discipline fixated on one particular conceptual vocabulary for analysing cognition: dwelling on component processes manifest by de-contextualized actors usually acting in the abstract settings of laboratories. so, for example, it is very rare in psychological literature on 'learning' to find reference to (or a presence in) the events of classrooms. this is cognitivism. could the so-called middle ground of constructivism then be an alternative way forward? i find the characterization of cunningham et al here too sweeping; it blurs too many important distinctions (for instance, they say their position is 'variously labelled constructivism, experientialism, semiotics'). however, by any account, constructivism remains a cognitivist tradition, albeit one that has a welcome emphasis on studying activities 74 alt-j volume 1 number 2 during the circumstances of learning. i believe there is an alternative theoretical perspective. hammond flirts with it briefly in a section (also rather broad brush) on situated action and affordances. alas, the editors chose to defuse this with the label 'new behaviorism'. cunningham et al embrace it as socio-cultural theory, but it fails to surface on their landscape as a distinctive alternative. a more socially grounded theory of mind might help in many ways. one would be in forcing us to take seriously the significance of context. if this volume captures representative 'psychological' research on hypertext (and i think it does), what is striking is a lack of concern for the broader settings in which these computer-based applications get used. our understanding of this new medium will be impoverished until we take seriously both the interpretative practices that particular constituencies of users bring to it, and the broader framework of teaching and learning into which it gets variously located. i take whalley to be raising these issues in a helpful way, and his is a chapter that i wish was longer. yet, this book is a balanced, accessible overview of recent research. i found it helpful in provoking new lines of thought on the subject in my own mind, and i am sure it will do so for many other readers, although not necessarily the same lines of thought! charles crook, university of durham the art and science of computer animation by stuart mealing, oxford, intellect, 1992. isbn: 1-871516-16-1. computer animation is the creation of the illusion of movement from discrete static images on a computer screen. it is an emerging discipline which embraces a wide range of subject areas, spanning both the arts and the sciences. there are many books available which deal with the mathematics, geometry and programming of computer graphics, while others deal with the traditional animation techniques used by the film industry, but there are very few which deal with the integrated subject of computer animation at any length. this book aims to fill that gap. it is divided into two main parts. part 1 covers the basic background material necessary to gain a full understanding of computer animation. the first chapter introduces traditional animation techniques in relation to the role of the computer, and the following chapters describe various applications of computer animation (from tv graphics to medical imaging), the basic principles of computer graphics and movement control, and various hardware and software considerations. part 2 builds on part 1, describing in greater detail the latest developments in the area, including an attempt to predict the future applications of emergent technologies. the book concludes with detailed technical appendices, a glossary of terms and a bibliography. it is produced to a very high standard, attractively laid out with numerous effective illustrations. in fact, this is certainly one of the most readable computing texts i have encountered. it well conveys the author's enthusiasm for his subject: clearly, computer animation should be fun, yet this does not make the book lightweight. what is most impressive is that all the necessary subject areas are dealt with comprehensively enough to make it an excellent starting point for those readers seriously considering undertaking a computer-animation project. little prior knowledge of either mathematics or computing is assumed, making it possible for the more general reader to develop a broad understand75 reviews ing of the field. and a comprehensive bibliography, combined with recommended texts, enables interested readers to make further inroads. most importantly, the author has concentrated on concepts and principles, rather than on supplying pages of mathematical formulae, or the ephemeral minutiae of proprietary hardware and software. this was a wise choice which will maximize the shelf-life of the book an important consideration, with the rapidly increasing rate of technological advance. however, to provide balance, a case study, the making of pencil test, from a paper by galyn susman of apple computer, is supplied in an appendix. this paper describes in detail the complete process of making a computer animation, and is particularly useful as it highlights the realistic day-to-day problems which must be dealt with during production. while the book provides a great deal of useful information throughout, it is the material contained in part 2 which makes it an essential read, since much of the material in part 1 is available in a range of other texts (though this is not a criticism). in part 2, i would single out chapters 9 to 12 as being of particular merit. these are effectively stateof-the-art reviews of the areas of simulation, soft modelling, behavioural animation and the simulation of human movement: information which is not readily available elsewhere. in all, i can confidently recommend this book to students, industry professionals, and any enthusiasts who wish to learn about computer animation. lorraine warren, university of hull instant multimedia for windows 3.1 by kris jamsa, new york, john wiley and sons, 1993. isbn: 0-471-58972-1. the use of computer systems for the generation and delivery of multimedia resources (text, sound, static pictures, animation and motion video) is growing rapidly. the ease with which these activities can be undertaken is also rapidly increasing. this is due mainly to the advent of the mpc (multimedia personal computer) and the general availability and popularity of graphical user interfaces. this book is timely in that it provides a technical (and very readable) introduction to multimedia generation and applications using windows version 3.1. it also offers hands-on use of a pc, since it comes with two floppy disks containing a range of multimedia material linked to various chapters. there are 23 chapters. some of them are very short but all are to the point. together, they cover most of the important aspects of multimedia deliverable on a pc, although certain topics (such as digital full-motion video) are not treated in any great depth. there are also two appendices and a detailed subject index, but unfortunately there is no bibliography; nor does the author makes any reference whatosever to other sources of information, not even a windows 3.1 manual! the book is organized into four broad sections. the first (chapters 1 to 10) deals primarily with the creation and use of sound. the second (chapters 11 to 14) treats cd-rom and its applications, the mpc standard, and more advanced uses of sound. the third (chapters 15 to 20) deals with fonts, scanners, video and animation. the final section (chapters 21 to 23) covers a range of topics including future possibilities and applications. 76 alt-j volume 1 number 2 the first two chapters serve to introduce simple sound effects (based on the mpc's internal speaker) using the driver and digital sound files provided on the companion disks. chapter 3 covers multimedia screen savers, while chapter 4 (being slightly more 'academic' in that it deals with the physics of sound) brings out the relationship between sound quality, storage requirements and digitization parameters. chapter 5 introduces a program called excuses which uses the digital sound effects provided on one of the companion disks in a variety of interesting ways, and chapter 6 gives the reader the task of experimenting with the speaker driver by exploring various customization parameters and the ways in which they effect sound quality. chapter 7 introduces another program called presidents which offers multimedia information about some of the presidents of the united states. it is possible to explore a textual biography, listen to a quotation and view a photograph of any of the seven presidents included in the demonstration. chapter 8 introduces the mpc standard by describing the basic configuration required to support it and the facilities it should offer (such as a sound board and a cd-rom drive). the 'ultimedia' (ibm's mpc rendering based on a ps/2) is also mentioned. sound boards are treated in greater depth, however, in chapter 9 (here, the material on irqs and dma transfers is quite useful), while chapter 10 deals with how to use a board to record and mix sound using the windows sound recorder accessory. further material on sound effects is presented in the two subsequent chapters, the first of which describes the windows media player, mci (media control interface) and the control of external devices such as cd-rom drives, videodiscs, and so on, while the second describes the purpose and use of midi (musical instrument digital interface). in chapter 11 the author introduces some basic material on cd-rom drives and their associated optical discs. this is followed in chapter 12 by a fairly detailed description of the various multimedia and hypermedia electronic books that make up the microsoft bookshelf cd for windows (jhe american heritage dictionary, bartlett's familiar quotations, the concise columbia dictionary of quotations, the concise columbia encyclopedia, hammond's atlas, roget's electronic thesaurus, and the world almanac and book of facts). further examples of multimedia cd books are described in chapter 22. the visual mode of communication is considered in more detail in chapters is to 20. these cover a wide range of topics such as fonts (vector, raster and outline) and their use in both textual and multimedia documents (created by write and paintbrush), scanners and their applications (for creating image collections and ocr work), digital electronic cameras, and the use of bit-map images as wallpaper for use in windows screens. the 'theory' of video is covered somewhat superficially in chapter 18, while the next two chapters treat animation. some examples of animation and video clips from the companion discs are used to illustrate the kind of thing that can be done with an animation package in this case, autodesk animator. the last three chapters deal with future possibilities (such as virtual reality, new ways of publishing information in multimedia documents, and the potential utility of multimedia techniques in education). chapter 22 gives a very useful overview of a number of currently available multimedia cd publications such as multimedia beethoven, survey of western art (which contains over 1,000 images and 300 biographies), sherlock holmes, consulting detective (which embeds 77 reviews motion video), time table of history, science and innovation, mammals a multimedia encyclopedia, compton's multimedia encyclopedia, and various hypertextual cd applications such as ebert's movie home companion. the final chapter describes some of the ways in which sound effects can be incorporated into other windows applications such as spreadsheets, memos, electronic mail and applications programs using the windows ole (object linking and embedding) facility. what one has here is therefore a good overall introduction to the topic of multimedia. the material is sometimes of a technical nature, but the book itself is written in a way which makes that material understandable even to the novice. in fact, it should provide a helpful exposure for readers who are coming at this topic for the first time, and who require a guided exploratory approach to multimedia. philip barker, university of teesside producing teaching materials: a handbook for teachers and trainers by henry ellington and phil race, 2nd edition, london, kogan page, 1993. isbn: 07494-0393-4. neither of the reviewers of this book was familiar with the first edition. that was no disadvantage books should be current, update or not. as the authors recognize in their introduction, preparing a second edition raises the vexatious question of what to leave out from the first, published in 1985. since then, document preparation and the production of teaching materials has undergone a technological revolution at all levels of education and training, and is still running. there are signs that the baggage from the last decade has slowed the authors down in their attempts to keep up. in most areas there is an admirable linkage between the technology and the learning objectives. a good example is the section of chapter 2 on printed materials which deals head-on with the current interest in open and flexible learning, provides definitions, and asks the pertinent questions anyone considering a venture in this area should be able to answer. as is appropriate for what is still the most widely used medium, print is covered in some detail. but do we still need to know how a stencil duplicator or even off-set litho work? chapter 3 discusses the use of non-projected display materials, posters, flipcharts, and magnetic boards. it includes a section on how a chalkboard should and should not be used. we are not sure which concerns us most; that this section should be in the book at all, or that teachers at all levels in education actually need the instruction. the production of still projected display materials is detailed in chapter 4 with some excellent reference data on layout of ohts and 35 mm slides. there is also a useful discussion on the management of a library of these materials. production of audio materials is looked at in some depth in chapter 5. the section on design and management of the materials is practical and to the point, though some of the technical aspects as presented would serve to confuse and frighten off certain teachers who might be considering the use of this medium. it was delightful, however, to see chapter 6 given over to the production of tape-slide material, which provides an effective linking of the techniques discussed in the preceding chapters. video technology is dealt with only threequarters of the way through the book in chapter 7, which might be an appropriate allocation of space for those working in primary and secondary education, whose funds are tightly constrained. for further and 78 alt-j volume 1 number 2 higher education, however, with many institutions now providing audiovisual services centrally, the teachers might feel shortchanged. with millions being spent on the teaching and learning technology programme, chapter 8's 30 pages on the role of computers in the classroom is miserly, given the multitude of applications. many of the photographs in this section already look dated, and in some cases risible, such as those on two pages where essentially the same configuration is first captioned as a multimedia workstation and then as being used for computer conferencing. with computer technology advancing so fast in the time it takes to read a book, let alone in the time it takes to write one, the authors clearly had difficulties. the book has thus tended to concentrate on the principles and concepts of using computers in different teaching and learning situations. it looks at some concepts like multimedia and authoring languages, and at some of the new technologies, including cd-i and electronic books. the chapter, and book, ends with a couple of case studies. somewhere along the way, the balance between the treatment of the old and new technologies has gone astray. the outcome is a book whose primary use would be as a student reference to how things used to be. mike kibby and stephen welsh, university of strathclyde interactive multimedia promise and pitfalls, edited by colin latchem, john williamson and lexie henderson-lancett, kogan page, london, 1993. isbn: 07494-0815-4. virtually all forms of learning depend on effective communication. multimedia approaches to knowledge transfer are attractive because the limitations and inadequacies of one channel of communication can usually be compensated for by the advantages of another. also, introducing interactivity into communication mechanisms gives the recipient of a message considerable control over the nature of the information received and how it is presented. for reasons such as these, interactive multimedia would seem to offer significant potential for the design and implementation of effective and efficient communication mechanisms. this book explores some of the possibilities, and describes some current approaches to using interactive multimedia in an educational setting. the book was originally conceived as a result of an international conference held in january 1992 in perth, western australia, and several of the papers presented at this conference are included, along with others that have been specially commissioned. there are 12 chapters, produced by an international group of authors from australia, canada, the netherlands, the uk and the usa. all are well-written, and each contains a useful list of references. a glossary, author index and subject index are also provided. the opening chapter, written by the editors, gives an overview of interactive multimedia, outlining its basic nature and discussing the role of instructional design and the problems of introducing new technologies into learning environments. this is followed by a chapter by nigel paine and philip mcara which is mainly technical. it offers a summary of current developments in interactive multimedia, such as authoring systems, mpc, avi, cd-rom, dvi, cd-i, cd-rom xa, photo cd, interactive video and virtual reality. in chapter 3, alexander romiszowski discusses the problems of developing interactive multimedia courseware and communication 79 reviews networks to support teaching and learning. in doing so, he raises a number of relevant design issues, such as the dichotomy between the instructional and informational roles that interactive multimedia can play. chapter 4 (by thomas reeves) discusses the current lack of research foundations to support the use of interactive multimedia, then goes on to describe some possible directions of future development based on constructivism, exploratory case studies and computer modelling. in chapter 5, stephen heppell outlines the historical development of educational computing in terms of four overlapping stages, then discusses some of the changes needed to make it more pedagogically effective. some interesting illustrations taken from the xploratorium are presented. peter olaf looms continues the educational theme in chapter 6 by dealing with the use of interactive multimedia at the schools level, describing the current state of the art in both north america and europe, and concluding by discussing some obstacles that need to be overcome before interactive multimedia will become widely used in schools and colleges. in chapter 7, jenny preece describes multimedia and hypermedia and their potential for the support of interactive learning, giving special consideration to human factors issues. the contribution by william tan and ann nguyan (chapter 8) treats lifecycle costing models for interactive multimedia systems, then compares the cost-effectiveness of three different approaches to providing computerbased instruction with conventional instructor-led classroom presentations. although the costings used relate to australian experience, the models would be generally applicable to other situations and the same is to some extent true for chapters 9 and 10, where two interesting australian projects are described. lyn henderson outlines the difficulties involved in overcoming cultural problems using interactive multimedia methods (based on macintosh computers) within a remote-area teacher education programme (ratep) in north queensland. in chapter 10, ian conboy deals with the use of audiographic telematics in schools in order to improve retention rates. this project (called max) uses macintosh computers, fax transmission and audio-conferencing methods in order to create an 'electronic classroom' made up from participating groups distributed geographically throughout the state of victoria. the last two chapters (11 and 12) deal respectively with interactive multimedia in libraries, and the creation of information products based on electronic publishing. in chapter 11, john frylinck and david raitt review current technology, its potential roles in libraries, and ways in which libraries may have to adapt in order to accommodate interactive multimedia products such as electronic books and multimedia encyclopedias. the final chapter by eric lugtigheid describes a forward-looking canadian venture called the jean talon project, intended to produce a range of multimedia educational products dealing with various aspects of canada: its political system, cultures, scientific achievements, geography, and so on. i found this book readable and extremely interesting. although some parts of it have a necessarily technical orientation, on the whole most of it deals with conceptual issues involved in using interactive multimedia within educational and training settings. taken together, the chapters form a wellbalanced whole, adequately reflecting the current state of the art in interactive multimedia from an educational perspective. philip barker, university of teesside 80 (page number not for citation purpose) 1 *corresponding author. email: inabl@openu.ac.il research in learning technology 2023. © 2023 r. kasperski et al. research in learning technology is the journal of the association for learning technology (alt), a uk-based professional and scholarly society and membership organisation. alt is registered charity number 1063519. http://www.alt.ac.uk/. this is an open access article distributed under the terms of the creative commons attribution 4.0 international license (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 research in learning technology vol. 31, 2023 original research article analysis of emergency remote teaching in formal education: crosschecking three contemporary techno-pedagogical frameworks ronen kasperskia, erez poratb and ina blauc* asha’anan religious college of education, gordon academic college of education, haifa, israel; bthe nb haifa school of design, haifa, israel; cdepartment of education and psychology, the open university of israel, ra’anana, israel (received: 25 december 2022; accepted: 31 march 2023; published: 12 may 2023) during the covid-19 pandemic outbreak many countries around the world were forced to turn to emergency remote teaching (ert) and upscale the use of digital technologies for learning, teaching and assessment. the current study analysed field reports from 89 elementary and secondary hebrew-speaking and arabic-speaking information and communication technology schools in israel, representing the cultural, ethnic, and religious diversity of the education system. the qualitative analysis of the collected data was based on three well established contemporary models of technology integration and digital competence of educators (digcompedu): the international society for technology in education, technological pedagogical content knowledge and digcompedu. the analysis (n = 872 statements) yielded aspects in the teachers’ reports that correspond with the theoretical models, alongside aspects that extend these models to ert and aspects that were missing from the reports. finally, based on our findings and previous work we suggested a comprehensive framework for ert that can be used to design teachers’ professional development necessary for effective remote teaching in both emergency and routine times. keywords: emergency remote teaching; covid-19 pandemic; information and communication technology; e-learning; online learning; digital pedagogy introduction emergency remote teaching (ert) refers to the sudden temporary shift of instructional delivery to an alternate delivery mode due to crisis circumstances. in most cases it involves the use of fully remote teaching that would otherwise be delivered faceto-face or in blended or hybrid courses. the primary objective in an emergency crisis is to provide temporary access to instruction (hodges et al. 2020). the outbreak of the covid-19 pandemic has created one of the most challenging times in the global community (world health organization 2020). as part of an effort to limit the spread of the pandemic, over 190 countries worldwide closed their schools and switched to ert for approximately 1.6 billion students (unesco 2020b). despite the challenges presented by the transition to ert, the world has witnessed an important technological transformation. however, there is an important distinction mailto:inabl@openu.ac.il http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2982 r. kasperski et al. 2 citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 (page number not for citation purpose) between ert and online learning, and even though policy makers and education officials mistakenly refer to these terms interchangeably (hodges et al. 2020), high quality online instruction is not something that can be launched overnight. thus, the purpose of the present study was to analyse ert during this unique and unprecedented worldwide pandemic based on contemporary techno-pedagogical frameworks in order to propose a comprehensive framework for ert. frameworks for technological development in context teacher education research indicates that the development of technology-based competencies, appropriate pedagogical techniques, and teaching strategies is pivotal for effective remote learning (munoz-najar et al. 2021; rivera et al. 2021). while much research on the development of technology-based competencies during covid-19 has been accumulated over the past 2 years (avidov-ungar et al. 2023; lien et al. 2022; shamir-inbal and blau 2021; valeeva and kalimullin 2021; zhou and li 2020), only a handful of studies, however, attempted to analyse ert on the basis of theoretical models such as  technological pedagogical content knowledge (tpack) (decoito and estaiteyeh 2022) or the digital competence of educators (digcompedu) standards for educators (dias-trindade et al. 2021). in the present study we adopted three comprehensive conceptual frameworks that address techno-pedagogical development of teachers in routine times and examined their applicability to ert: the tpack model (koehler and mishra 2009), the international society for technology in education standards for educators (iste 2017), and the european framework for the digcompedu (2017). the tpack the model suggested by koehler and mishra (2009) asserts that effective teaching with technology practices depends on content knowledge (ck), pedagogical knowledge (pk), and technological knowledge (tk). the three core components of knowledge constructs are strongly interconnected and combine in various ways within the tpack framework. technological pedagogical knowledge (tpk) describes teachers’ understanding of how particular technologies can change both the teaching and learning experiences by introducing new pedagogical affordances and constraints. technological content knowledge (tck) involves understanding how the subject matter can be communicated via different technological tools and considering which specific tool might be best suited for specific subject matters or classrooms. the iste standards for educators the iste standards conceptualise seven key standards and best practices for educators that focus on using technology to empower student learning experience and support teachers’ technological and pedagogical competencies in the digital age. (1) the learner standard views educators as lifelong learners seeking evidence-based practices that leverage technology to improve student learning. (2) the leader standard encourages educators to seek out opportunities for leadership by engaging with education http://dx.doi.org/10.25304/rlt.v31.2982 research in learning technology citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 3 (page number not for citation purpose) stakeholders, looking for ways to capitalise on resources in order to provide students with equitable access to educational technology, digital content, and learning opportunities, as well as modelling others in the adoption of new digital resources and tools for learning. (3) the citizen standard states the importance of educating students to use technology positively and socially and to contribute to and responsibly participate in the digital world. (4) the collaborator standard encourages professional growth through meaningful collaboration and interactions with colleagues, as well as co-learning with students to discover and use new digital resources and technology. (5) the designer standard refers to the ability to design authentic, learner-centred activities and learning environments and to apply instructional design principles that foster independent and personalised learning. (6) the facilitator standard encourages educators to foster opportunities and the culture of effective use of technology and learning strategies in digital platforms. finally, (7) the analyst standard highlights the importance of understanding and using data to guide instruction and support students in achieving their learning goals. the european framework for the digcompedu the digcompedu (redecker 2017) framework aims to capture and describe six different areas of digital capability that are essential for educating in the 21st century: (1) professional engagement, which refers to the use of digital technologies for communication, collaboration, and professional development. (2) digital resources, which refers to sourcing, creating, and sharing digital resources. (3) teaching and learning, which refers to managing and orchestrating the use of digital tools in teaching and learning. (4) assessment, which refers to using digital tools and strategies to enhance formative and summative assessment through a diversity of assessment formats and approaches. (5) empowering learners, which refers to using digital technologies to enhance inclusion, personalisation, and learners’ active engagement. (6) facilitating learners’ digital competence, which refers to enabling learners to creatively and responsibly use digital technologies for information, communication, content creation, wellbeing, and problem-solving. research goals and questions this study aims to analyse teachers’ reports of the transition to ert based on three well established contemporary models of technology integration and digcompedu, the iste, tpack and digcompedu, in order to propose a comprehensive framework for ert. the following research questions were explored: (1) which aspects of iste, tpack, and digcompedu knowledge and skills relating to the integration of technology during ert were reported? (2) which aspects emerging from the data are beyond the iste, tpack, and digcompedu models? method participants in the current study, we analysed 89 fully detailed written reports from schools across the state of israel. the reports represent the perspectives of schools’ information and communication technology (ict) coordinators, teachers, and http://dx.doi.org/10.25304/rlt.v31.2982 r. kasperski et al. 4 citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 (page number not for citation purpose) principals from various k-12 levels (kindergartens, n = 5, 5.68%, elementary schools, n = 45, 51.14%, and secondary schools, n = 38, 43.18%) and also the cultural, ethnic, and religious diversity of sectors in the israeli education system: regular and special education, including hebrew-speaking schools (secular, religious, and ultra-orthodox) and arabic-speaking schools (arab, druze, and bedouin). all of the schools had participated in the national ict program for several years and thus, were quite experienced in technology-enhanced learning in classroom setting. instruments and procedure the study was approved by the institutional ethics committee. during the lockdown from march 2020 to june 2020, the ministry of education (moe) issued a call to all the regional ict coordinators to collect field reports that would describe challenges and successes, how the rapid curriculum transformation was established, and the educational continuity was maintained. the schools provided, through ict coordinators and school principals, in-depth and detailed information about special projects which were carried out during this period, and some schools shared learning outcomes from these projects on the portal. the original data obtained from participants was in hebrew and the themes and representative quotes in the results section were translated into english by the researchers. all reports were analysed by content analysis based on the thematic analysis approach (nowell et al. 2017), utilising axial coding, selective coding, and establishing hierarchies between the categories. at the initial coding stage, as many codes as possible were generated inductively from each section of the dataset and labelled as similar to the data as possible (charmaz 2006). the major themes that emerged from the data were: (1) aspects in the teachers’ reports that correspond with the theoretical models, (2) aspects in the teachers’ reports that extend these models to ert, and (3) aspects from the theoretical models that were missing from the reports. once categories began to develop in the intermediate phase of coding, relationships between the categories were identified and basic information was transformed into more abstract concepts. the data was analysed according to the competencies definitions from the theoretical models. some new categories that were identified from the data, extended these models to ert, alongside aspects from the theoretical models that were missing from the reports. following several iterations, final categories and subcategories were determined. in the last phase of advanced coding, the categories became abstract and represent the participants’ stories reduced into conceptual terms. these interrelated concepts represent the relationships between the core category of each research question and more detailed subcategories (corbin and strauss 2007), see tables 1 and 2 in the results. finally, based on this conceptualisation, a theoretical model of the phenomenon (birks and mills 2015) was suggested and explored in the last research question (see figure 1). top-down and bottom-up data coding patterns from 25% of the reports were established independently by three researchers. disagreements between them were resolved through discussion. the coding of the remaining reports was conducted based on the established coding. http://dx.doi.org/10.25304/rlt.v31.2982 research in learning technology citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 5 (page number not for citation purpose) t ab le 1 . t h eo re ti ca l an al ys is o f re m o te t ea ch in g ac co rd in g to t h e is t e , t pa c k , an d d ig c o m p e d u m o d el s (n = 8 72 ). t h em e % o f th e st at em en ts r ep re se n ta ti ve q u o te s p h as e 1: t ea ch er s’ a cq u is it io n o f te ch n o lo gi ca l k n o w le d ge ( n = 2 42 , 27 .7 5% ) e d u ca to rs a s le ar n er s (i s t e ); d ig c o m p e d u a re a 1: p ro fe ss io n al e n ga ge m en t; t ec h n o lo gi ca l k n o w le d ge (t pa c k : t k ) n = 1 04 , 42 .9 7% ‘t ea ch er d ev el o p m en t w as c o n d u ct ed e ve ry o th er w ee k v ia z o o m i n t h e fo rm at o f tw o s es si o n s: o n e se ss io n f o r le ar n in g h o w t o u se d ig it al t o o ls (b as ed o n a t ea ch er s’ n ee d s su rv ey ) an d a n o th er f o r te ac h in g h o w t o u se th es e to o ls e ff ec ti ve ly .’ (s ch o o l 41 ) e d u ca to rs a s co ll ab o ra to rs ( is t e ); n = 1 37 , 57 .0 3% ‘w e p ra ct ic ed p ee r le ar n in g fo r th e p u rp o se o f tr ai n in g th e te ac h er s fo r d is ta n ce l ea rn in g – th e te ac h er s w er e d iv id ed i n to l ea rn in g p ai rs – o n e te ac h es w h il e th e o th er w at ch es a n d s u p p o rt s. g ra d u al ly t h e st af f d ev el o p ed i n d ep en d en t re m o te t ea ch in g sk il ls .’ (s ch o o l 51 ) p h as e 2: r em o te /o n li n e t ea ch in g – t ec h n o -p ed ag o gi ca l ap p li ca ti o n ( n = 2 80 , 32 .1 % ) e d u ca to rs a s d es ig n er s (i s t e ); t ec h n o lo gi ca l c o n te n t k n o w le d ge ( t pa c k : t c k ); t ec h n o p ed ag o gi ca l k n o w le d ge ( t pa c k : t p k ); d ig c o m p e d u a re a 2: d ig it al r es o u rc es ; d ig c o m p e d u a re a 3: t ea ch in g an d  l ea rn in g m an ag in g n = 1 80 , 64 .2 9% ‘f o r te ac h in g n ew m at er ia l, t ea ch er s p re p ar ed n ar ra te d p re se n ta ti o n s w it h li n k s to s u p p le m en ta ry m at er ia ls a n d a ss ig n m en ts , si m il ar t o a d ig it al co u rs e. ’ ( s ch o o l 1) ‘t o s p ar k t h e st u d en ts ’ i n te re st , i p re p ar ed a n e xp er im en t in s ci en ce s tu d ie s th at t h ey c o u ld f o ll o w e as il y vi a z o o m . it w as a g re at s u cc es s. i w as e xc it ed to s ee t h e en th u si as m i n t h ei r ey es – t h ey w er e to ta ll y en ga ge d . i th in k t h at th e ca re fu l d es ig n o f th e le ss o n i s w h at m ad e th e d if fe re n ce .’ (s ch o o l 37 ) e d u ca to rs a s fa ci li ta to rs ( is t e ); t ec h n o -p ed ag o gi ca l k n o w le d ge ( t pa c k : t p k ); d ig c o m p e d u a re a 5: e m p o w er in g l ea rn er s’ u se o f d ig it al t ec h n o lo gi es n = 9 1, 3 2. 5% ‘t h e te ac h er i n co rp o ra te d d if fe re n ti al l ea rn in g b y d iv id in g th e st u d en ts in to b re ak o u t ro o m s vi a z o o m , w h ic h a ll o w ed t h e te ac h er t o t ak e th e ro le o f a fa ci li ta to r w h o m ov es b et w ee n r o o m s an d p ro vi d es a d if fe re n ti al re sp o n se t o a d d re ss t h e d if fe re n ce s b et w ee n t h e le ar n er s. ’ ( s ch o o l 55 ) e d u ca to rs a s an al ys ts ( is t e ); t ec h n o -p ed ag o gi ca l k n o w le d ge ( t pa c k : t p k ); d ig c o m p e d u a re a 4: a ss es sm en t n = 9 , 3. 2% ‘d at a re ga rd in g th e le ar n in g p ro gr es s o f ea ch s tu d en t w as c o ll ec te d i n t h e g o o gl e cl as sr o o m i n o rd er t o c re at e an i n d iv id u al p ro gr am t h at a d d re ss ed d iv er si ty a n d e xi st in g n ee d s. ’ ( s ch o o l 47 ) s tu d en ts ’ d ig it al c it iz en sh ip ( is t e ); l ea rn er s’ d ig it al c o m p et en ce ( d ig c o m p e d u , ar ea 6 ); – e d u ca to rs a s le ad er s (i s t e ) – is t e , in te rn at io n al s o ci et y fo r t ec h n o lo gy i n e d u ca ti o n ; t pa c k , t ec h n o lo gi ca l p ed ag o gi ca l c o n te n t k n o w le d ge ; d ig c o m p e d u , d ig it al c o m p et en ce o f e d u ca to rs ; t c k , t ec h n o lo gi ca l co n te n t k n o w le d ge ; t p k , t ec h n o lo gi ca l p ed ag o gi ca l k n o w le d ge . http://dx.doi.org/10.25304/rlt.v31.2982 r. kasperski et al. 6 citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 (page number not for citation purpose) t ab le 2 . a sp ec ts t h at e xp an d t h e is t e , t pa c k a n d d ig c o m p e d u m o d el s (n = 8 72 ). t h em e % o f th e st at em en ts r ep re se n ta ti ve q u o te s r eo rg an is at io n o f ed u ca ti o n al f ra m ew o rk t o e r t ) n = 9 8, 1 1. 24 % ) a d ju st in g th e sc h ed u le n = 7 5, 7 6. 53 % ‘w it h t h e o u tb re ak o f th e cr is is , w e co n d u ct ed r ap id a d ju st m en ts t h at w o u ld a ll o w u s to m an ag e th e d is ta n ce le ar n in g p ro ce ss es . f ro m t h e o rg an iz at io n al p er sp ec ti ve , w e p re p ar ed a n o n li n e ve rs io n o f th e sc h o o l sc h ed u le an d l ea rn in g m o d es ( sy n ch ro n o u s, a sy n ch ro n o u s, o r th ei r co m b in at io n – b le n d ed l ea rn in g) . w e d iv id ed t h e o n li n e ch an n el s ac co rd in g to s u b je ct m at te r. ’ ( s ch o o l 55 ) b ri d gi n g th e d ig it al ga p ( eq u al it y) n = 9 , 9. 18 % ‘t ea ch er s li st ed s tu d en ts w h o d id n o t h av e p ro p er m ea n s fo r d is ta n ce l ea rn in g. t h e sc h o o l p ro vi d ed l ap to p s to a ss is t th es e st u d en ts .’ (s ch o o l 19 ) ‘t ea ch er s en co u ra ge d k n o w le d ge ab le s tu d en ts t o g u id e th ei r p ee rs o n h o w t o s u b m it a ss ig n m en ts o n m ic ro so ft t ea m s. ’ ( s ch o o l 61 ) p ro vi d in g ac ce ss to v ir tu al l ea rn in g p la tf o rm s n = 1 4, 1 4. 29 % ‘w e d ec id ed t o m an ag e th e re m o te l ea rn in g vi a m ic ro so ft 3 65 . t o t h is e n d , al l 10 00 s tu d en ts a n d 1 70 te ac h er s w er e co n n ec te d t o t h e sy st em . s tu d en ts a n d t ea ch er s h ad t o d o w n lo ad t h e p la tf o rm t o t h ei r p er so n al co m p u te rs a n d w er e gi ve n a u se rn am e an d p as sw o rd t o a cc es s th ei r ac co u n ts .’ (s ch o o l 46 ) s u p p o rt in g so ci al e m o ti o n al n ee d s (n = 2 52 , 28 .9 % ) s tr en gt h en in g th e re la ti o n sh ip s am o n g te ac h in g st af f n = 3 5, 1 3. 89 % ‘t h e p ri n ci p al g av e u s th e fe el in g th at s h e b el ie ve d i n o u r ab il it ie s. s h e tr u st ed u s to s u cc ee d a n d b e m ea n in gf u l to o u r st u d en ts d u ri n g th e p an d em ic . t h e se n se o f b el o n gi n g h as g ro w n s tr o n ge r d u ri n g th is t im e. t h er e h as b ee n c o n si d er ab le s h ar in g o n s o ci al n et w o rk s, a n d t h e te ac h er s al l fe lt t h at w e w er e in t h e sa m e b o at a n d t o ge th er w o u ld o ve rc o m e th e cr is is .’ (s ch o o l 37 ) e m o ti o n al s u p p o rt fo r st u d en ts a n d fa m il ie s n = 1 02 , 56 .0 3% ‘e m p h as is i s p la ce d o n c o m m u n ic at io n w it h s tu d en ts t h ro u gh a ll c h an n el s; t ea ch er s re sp o n d t o s tu d en ts a n d su p p o rt t h em b o th a ca d em ic al ly a n d e m o ti o n al ly o n w h at sa p p, w h ic h i s al w ay s av ai la b le .’ (s ch o o l 5) ‘t h e sc h o o l co u n se lo r an d p sy ch o lo gi st c o n d u ct ed a n xi et y w o rk sh o p s fo r p ar en ts a n d s tu d en ts t o a ll o w a n em o ti o n al d ia lo gu e th ro u gh w h ic h t h e ch il d re n c o u ld e xp re ss t h ei r co n ce rn s an d r ec ei ve c o p in g st ra te gi es d u ri n g th e lo ck d o w n .’ (s ch o o l 12 ) e xt ra cu rr ic u la r p ro je ct s, i n it ia ti ve s, an d r ec re at io n al ac ti vi ti es n = 1 15 , 45 .6 3% ‘t h e st u d en t co u n ci l, i n c o ll ab o ra ti o n w it h t h e so ci al s tu d ie s te ac h er s, t h e in n ov at io n t ea m , an d t h e ic t co o rd in at o r, l au n ch ed “ c o ro n a l iv e” t o b ro ad ca st s p ec ia l ev en ts a n d h o li d ay s in r es p o n se t o s tu d en t n ee d s th at a ro se f ro m t h e d ay s o f lo ck d o w n o r q u ar an ti n e. i t w as a m u lt id is ci p li n ar y p ro je ct ( so ci al s tu d ie s, h is to ry , ge o gr ap hy , co m p u te rs , en tr ep re n eu rs h ip , so ci al e d u ca ti o n , et c. ) an d i n cl u d ed s tu d en ts , te ac h er s, p ar en ts , an d sp ec ia l gu es ts . t h is w as b ro ad ca st ed t o t h e ge n er al p u b li c. ’ ( s ch o o l 36 ) ‘t h is y ea r th e p ar en ts ’ c o u n ci l tu rn ed o u r an nu al p ar en ts a s t ea ch er s ev en t in to a v ir tu al c o n fe re n ce t h at w as h el d t h ro u gh o u t th e w ee k a n d c o n si st ed o f d iv er se l ec tu re s an d w o rk sh o p s. ’ ( s ch o o l 43 ) is t e , in te rn at io n al s o ci et y fo r t ec h n o lo gy i n e d u ca ti o n ; t pa c k , t ec h n o lo gi ca l p ed ag o gi ca l c o n te n t k n o w le d ge ; d ig c o m p e d u , d ig it al c o m p et en ce o f e d u ca to rs ; e r t , e m er ge n cy r em o te t ea ch in g; i c t , in fo rm at io n a n d c o m m u n ic at io n t ec h n o lo gy . http://dx.doi.org/10.25304/rlt.v31.2982 research in learning technology citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 7 (page number not for citation purpose) results aspects of educators’ tk and skills addressed by the iste, tpack, and digcompedu frameworks similar to perla et al. 2018 and desantis (2016), analysing the data we found that there is some overlapping of educator competencies between the three theoretical models described by the tpack, iste, and digcompedu models (table 1). as can be seen in table 1, phase 1 consisted of teachers’ acquisition of tk (n = 242, 27.75%). in this category we found an overlapping between the three models: educators as learners (iste), professional engagement (digcompedu, area 1), and technological knowledge (tpack: tk). all these dimensions relate to the acquisition of fundamental tk and skills in how to use new technologies to teach. the need to learn synchronous and asynchronous digital tools was paramount for communicating and collaborating with colleagues as well as for teaching remotely. the second category refers to educators as collaborators (iste). teachers had opportunities to work together with fellow teachers, supporting each other and sharing experiences specifically related to gaining tk. phase 2 in table 1 refers to remote/online teaching – techno-pedagogical application (n = 280, 32.1%). the first subcategory corresponded with each of the three models: educators as designers (iste); technological content knowledge (tpack: tck) and techno-pedagogical knowledge (tpack: tpk); digital resources (digcompedu, area 2); teaching and learning managing (digcompedu, area 3). all of the above digital skills are integrated to build the teacher’s comprehensive techno-pedagogical skill. the techno-pedagogical design process of instruction and learning consists of selecting the proper technological tools to achieve the learning goals, manage the learning process, and incorporate and provide the right digital resources. therefore, together all of these competencies provide a comprehensive coverage of the digital skills that teachers should master in their teaching. the second subcategory refers to the overlap between educators as facilitators (iste), techno-pedagogical knowledge (tpack: tpk), and empowering learners’ use of digital technologies (digcompedu, area 5). through fostering techno-pedagogical approaches to learning, educators promoted independent and collaborative work, student learning strategies in digital platforms, and other digital skills. the third category refers to the overlapping of educators as analysts (iste), assessment (digcompedu, area 4), and techno-pedagogical knowledge (tpack: tpk). a proper techno-pedagogical design should incorporate the right assessment methods and the use of student learning behaviour and achievement data to support instructional decision-making. aspects of ert that expand the iste, tpack, and digcompedu models as can be seen in table 2, two major aspects emerged from the data analysis and elaborate the existing theoretical frameworks of technological development: reorganisation of the educational framework to ert (n = 98, 11.24%) and supporting social emotional needs (n = 252, 28.9%). in terms of reorganisation of educational framework to ert, we found the following subcategories: adjusting the schedule, bridging the digital gap (equality), and providing access to virtual learning platforms. in terms of supporting social emotional needs, we found the following subcategories: strengthening the relationships among staff, emotional support for students and families, and extracurricular projects, initiatives and recreational activities. http://dx.doi.org/10.25304/rlt.v31.2982 r. kasperski et al. 8 citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 (page number not for citation purpose) discussion with the closure of the education system during the covid-19 pandemic, schools were required to provide students with remote education routines that would replace the traditional face-to-face instruction (manca and meluzzi 2020). the current study analysed theoretically how ict-integrated schools, who often used technology for teaching and learning, coped with this sudden transition to ert. the analysis was conducted on the basis of three well established contemporary models of technology integration and digcompedu (the iste, tpack, and digcompedu), in order to propose a theoretically based, comprehensive framework for ert that can be applicable in emergency times. figure 1 illustrates the theoretical analysis and presents aspects in the teachers’ reports that correspond with the theoretical models and domains that extend these models to ert. we also identified domains from the theoretical models that were missing from the reports and should be the focus of teachers’ professional development in order to improve their practice and to develop and apply the skills and knowledge necessary for effective remote teaching. figure 1. comprehensive framework of effective ert based on the iste, tpack, and digcompedu models. ert, emergency remote teaching; iste, international society for technology in education; tpack, technological pedagogical content knowledge; digcompedu, digital competence of educators. • educators as learners (iste); digcompedu area 1: professional engagement; technological knowl edge (tpack: tk) • educators as collaborators (iste) • educators as designers (iste); technological content knowledge (tpack: tck); techno-pedagogical knowledge (tpack: tpk); digcompedu • educators as facilitators (iste); techno-pedagogical knowledge (tpack: tpk); digcompedu area 5: empowering learners use of digital technologies • strengthening the relationships among staff • emotional support for students and families • extracurricular projects, initiatives, and recreational activities educators as leaders (iste) promoting student digital citizenship (iste); the digital capabilities of learners (digcompedu, region 6); technological knowledge (tpack, tk) educators as analysts (iste); technopedagogical knowledge (tpack: tpk); digcompedu zone 4: assessment aspects from theoretical models missing from teacher reports elaborating the theoretical models phase 2: supporting social emotional needs phase 2: teaching the curriculum remotely phase 1: instituational reorganization phase 1: teacher acquisition of technological knowledge effective ert corresponding with theoretical models • adjusting the schedule • bridging digital gap (equality) • providing access to virtual learning platforms http://dx.doi.org/10.25304/rlt.v31.2982 research in learning technology citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 9 (page number not for citation purpose) phase 1: reorganisation and teacher acquisition of tk the first phase in initiating the online educational framework consisted of reorganising and acquiring tk and skills. in terms of reorganisation, schools needed to plan and provide an online version of school programs consisting of schedule adjustments, curriculum adaptation, learning modes (synchronous, asynchronous, blended learning), and suitable learning platforms. these findings correspond with previous findings regarding the global shift to remote online learning (lien et al. 2022; valeeva and kalimullin 2021; zhou and li 2020). because all distance learning planning should be guided first and foremost by concern for equality and inclusion (miao et al. 2020), similar to reports from other countries (cerna et al. 2020), equality issues were raised and addressed in the present study. our findings revealed that teachers mapped the technological infrastructure and internet access available at their students’ homes and assisted those who did not have proper technology means for remote learning. this included taking steps to increase the accessibility of teaching materials for students with low digital literacy who were facing technology know-how obstacles beyond the lack of necessary equipment and internet access. for instance, teachers encouraged students with tk to aid students who were less technology-oriented in their nature. in terms of teacher acquisition of tk, school principals and ict coordinators held weekly meetings to provide the teaching staff with technology training. according to global reports, only a few teachers were able to conduct the transition to ert quickly and effectively, even those who were well experienced in the use of educational technologies (world bank group 2020). previous research shows that teachers felt unprepared with the sudden move to online teaching and struggled with supporting students in engaging in remote learning, with developing new skills, with transitioning their lessons to a remote format, to use new platforms, applications, and teaching tools, and how to adjust their teaching strategies and adapt their pedagogy in effective ways (avidov-ungar et al. 2023; bond 2021; leech et al. 2020; shamir-inbal and blau 2021). in this context, hodges et al. (2020) stressed that merely going online is not enough to meet the criteria for effective distance learning – a goal which requires thoughtful instructional design and systematic planning. phase 2: remote online teaching and supporting social emotional needs the two pillars of the ert education framework, due to the suspension of classroom instruction, were the remote online teaching and supporting social emotional needs. in terms of remote online teaching, we found an emphasis on two subcategories (i.e. design and facilitate), similar to miao et al. (2020) who argue that when delivery technologies are in place, teachers are in the frontline to design and facilitate learning activities, monitor and evaluate students’ home-based distance learning processes, adjust their learning management accordingly, and assess students’ achievement of learning outcomes. we will elaborate on these two subcategories in the following paragraphs. teachers referred to assessment to a lower extent (see the section entitled ‘aspects from the theoretical models that were missing from the reports’). the design and facilitate subcategories are vital components of digital teaching and thus represented in all three digital models: educators as designers (iste); technological content knowledge (tpack: tck); techno-pedagogical knowledge http://dx.doi.org/10.25304/rlt.v31.2982 r. kasperski et al. 10 citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 (page number not for citation purpose) (tpack: tpk); digital resources (digcompedu, area 2); teaching and learning managing (digcompedu, area 3); educators as facilitators (iste); techno-pedagogical knowledge (tpack: tpk); and empowering learners’ use of digital technologies (digcompedu, area 5). all these domains relate together to the thoughtful process of defining, designing, and executing optimal remote learning and teaching experiences that engage students and are driven by student needs and interests. teachers found videoconferencing tools helpful, because they enable content and assignment delivery, feedback, and vital communication (chimelis 2020). according to lowenthal et al. (2020), live synchronous videoconferencing best approximates the traditional classroom approach to teaching. instructors can quickly clarify problems, stimulate social learning, and, in this way, reduce the feeling of isolation. effective online learning is a by-product of cautious instructional design and planning, and numerous considerations, such as identifying the content, types of interaction, learners’ characteristics, social and cognitive aspects, learner support, all have an impact on the quality of the instruction (hodges et al. 2020). thus, many resources were allocated to socio-emotional support to alleviate the mental burden experienced by the students, their families, and teachers alike during the closure (fegert et al. 2020; wango et al. 2020). bozkurt et al. (2020) described the pedagogy of care which emerges during times of anxiety and trauma. accordingly, in times of crisis educators allocate resources to address their students’ vulnerabilities, allowing them to be more attuned to their individual needs. as for the teachers, the mutual efforts to close technological and pedagogical gaps through weekly school organised sessions contributed to the relationships among staff and brought teachers together. aspects from the theoretical models that were missing from the reports we also identified digital capabilities that barely appeared in the reports, such as educators as analysts (iste) and assessment (digcompedu zone 4). overall, the assessment category was represented less in teachers’ reports (n = 31) and, of these statements, teachers mostly referred to traditional evaluation processes such as checking homework and classroom assignments (n = 22). they rarely referred to the use of digital tools and digital learning data to conduct their instruction modes and provide personalised learning as described in both iste and digcompedu (n = 9). our findings are supported by previous research showing that while rapidly transitioning to online teaching, teachers replicated their face-to-face teacher-led traditional teaching and learning into the online environment with digital tools without making instructional adjustments to the online setting (shamir-inbal and blau 2021). one important component that was completely lacking from teacher reports was promoting students’ digital citizenship (iste) or facilitating learners’ digital competence (digcompedu, area 6). the citizen standard (iste) states the importance of educating students to use technology positively and socially, contributing to and responsibly participating in the digital world. the definition of digcompedu refers to enabling learners to creatively and responsibly use digital technologies for information, communication, content creation, well-being and problem-solving. these two dimensions are connected, since using digital technologies safely and responsibly as well as using digital technologies for communication, collaboration, and civic participation, lead to smart digital citizenship behaviour. although teachers were briefed on online safety and internet security to protect children’s privacy and allow for the http://dx.doi.org/10.25304/rlt.v31.2982 research in learning technology citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 11 (page number not for citation purpose) proper operation of distance learning, there were no references to teaching students safe online citizenship. these areas should be developed in teacher training and professional development courses to develop the skills and knowledge necessary for effective remote teaching. another aspect that was missing from teacher reports was educators as leaders (iste). the leader standard encourages educators to seek out opportunities for leadership by engaging with education stakeholders, looking for ways to capitalise on resources in order to provide students with equal access to educational technology, digital content, and learning opportunities, as well as modelling others in the adoption of new digital resources and tools for learning. aside from providing students with equal access to educational technology, which was a threshold condition for distance learning, the reports did not reflect leadership and seeking collaboration with stakeholders in education or resource mobilisation. it should be taken into account that the teachers themselves also experienced an emotional burden and anxiety from the pandemic, as they and their families were exposed to covid-19 just like the rest of the population, so it is understandable that they did not engage in activities beyond what was needed for their teaching. conclusions while most of the previous reports on ert have focused mainly on the lack of technological infrastructure (alvarez 2020; cullinane and montacute 2020; sen and tucker 2020; unesco 2020a), the current study focused on schools that were already technologically equipped for the transition. these settings enable to identify techno-pedagogical characteristics associated with effective ert and explore the promise of the pandemic to upscale the use of educational technologies for learning, teaching and assessment. in accordance with hodges et al. (2020), who differentiated between ert and quality online learning, our data suggests that, after initially adjusting school schedules to synchronous, asynchronous, or blended lessons, these schools were able to grow from technical issues to techno-pedagogical teaching and learning, while addressing the social emotional needs of both students and teachers. as one school so beautifully articulated: ‘the covid-19 crisis has brought not only challenges, but also opportunities that should not be ignored, but rather promoted and learned from later on’ (school 70). adopting three well established contemporary models of technology integration and digcompedu, the iste, tpack and digcompedu, to analyse teacher reports has led to the identification of aspects that correspond with the theoretical models, aspects that extend these models and adapt them to ert. some aspects from the theoretical models that were missing from the reports should be the focus of teachers’ professional development in order to improve their practice and develop and apply the skills and knowledge necessary for effective remote teaching. consistent with previous studies (avidov-ungar et al. 2023; hodges et al. 2020; kasperski et al. 2022; munoz-najar et al. 2021; porat et al. 2018; rivera et al. 2021; shamir-inbal and blau 2021), we call upon education policy makers to provide extensive teacher training in technology and pedagogy to support high quality online teaching. teacher training should address digital competencies such as analyst/assessment, students’ digital citizenship (iste and digcompedu), and digital leadership (iste). http://dx.doi.org/10.25304/rlt.v31.2982 r. kasperski et al. 12 citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 (page number not for citation purpose) limitations and future directions the main limitation of this study is its self-report methodology. it would be worthwhile for future studies to triangulate school reporting with observations of the actual behaviour of teachers and students during online lessons in order to improve the credibility of the qualitative findings (jonson et al. 2020). in addition, the schools participating in this study previously took part in the national ict program. as such, the reports reflect fewer technological challenges and shed more light on the pedagogical aspects and practices of schools experienced in technology-enhanced learning. lastly, the unit of analysis in this study was the school, and thus, we did not distinguish between the ert of younger and older students. the results of the future of school’s survey (2020) indicated that younger students had more difficulty switching to ert than older students. therefore, future studies may choose a different unit of analysis in order to conduct quantitative comparisons of the findings in various age groups. disclosure statement the authors declare that they have no conflict of interest. data availability statement these data were derived from the following resources [in hebrew] available on the public domain: israeli ministry of education portal https://pop.education.gov.il/ online-learning/teachers-reported/ contribution statement all the authors contributed to conceptualisation of this paper and to the data analysis. the first author contributed to writing the paper draft, while other authors contributed to its editing. all authors read and approved the final manuscript. acknowledgements the authors are thankful to ms. orit avdiel for her help in initial data coding. references alvarez, a. j. (2020) ‘the phenomenon of learning at a distance through emergency remote teaching amidst the pandemic crisis’, asian journal of distance education, vol. 15, pp. 127–143. avidov-ungar, o., hadad, s., shamir-inbal, t. & blau, i. (2023) ‘formal and informal professional development during different covid-19 periods: the role of teachers’ career stages’, professional development in education, pp. 1–17. doi: 10.1080/19415257.2023.2174163 birks, m. & mills, j. (2015) grounded theory: a practical guide, sage, los angeles, ca. bond, m. (2021) ‘schools and emergency remote education during the covid-19 pandemic: a living rapid systematic review’, asian journal of distance education, vol. 15, pp. 191–247. bozkurt, a., et al., (2020) ‘a global outlook to the interruption of education due to covid-19 pandemic: navigating in a time of uncertainty and crisis’, asian journal of distance education, vol. 15, pp. 1–126. http://dx.doi.org/10.25304/rlt.v31.2982 https://pop.education.gov.il/online-learning/teachers-reported/ https://pop.education.gov.il/online-learning/teachers-reported/ http://10.1080/19415257.2023.2174163 research in learning technology citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 13 (page number not for citation purpose) cerna, l., rutigliano, a. & mezzanotte, c. (2020) the impact of covid-19 on student equity and inclusion: supporting vulnerable students during school closures and school re-openings, organisation for economic co-operation and development, [online] available at: https://read.oecd-ilibrary.org/view/?ref=434_434914-59wd7ekj29&title=theimpact-of-covid-19-on-student-equity-and-inclusion charmaz, k. (2006) constructing grounded theory: a practical guide through qualitative analysis, sage, thousand oaks, ca. chimelis, r. (2020) ‘zoom isn’t perfect, but educators are learning how to make it work’, mass live, 01 july, [online] available at: https://www.masslive.com/news/2020/07/zoom-isnt-perfect-but-educators-are-learning-how-to-make-it-work.html corbin, j. & strauss, a. (2007) basics of qualitative research: techniques and procedures for developing grounded theory, 3rd edn, sage, thousand oaks, ca. cullinane, c. & montacute, r. (2020) covid-19 and social mobility impact brief# 1: school shutdown, research brief, sutton trust, [online] available at: https://www.suttontrust.com/ wp-content/uploads/2020/04/covid-19-impact-brief-school-shutdown.pdf decoito, i. & estaiteyeh, m. (2022) ‘transitioning to online teaching during the covid19 pandemic: an exploration of stem teachers’ views, successes, and challenges’, journal of science education and technology, vol. 31, pp. 340–356. doi: 10.1007/ s10956-022-09958-z desantis, j. (2016) ‘investigating the relationship between tpack and the iste standards for teachers’, issues and trends in educational technology, vol. 4, no. 1, pp. 16–30. doi: 10.2458/ azu_itet_v4i1_desantis dias-trindade, s., ant, j. & ferreira, a. g. (2021) ‘evaluation of the teachers’ digital competences in primary and secondary education in portugal with digcompedu checkin in pandemic times’, acta scientiarum. technology, vol. 43, pp. e56383–e56383. doi: 10.4025/ actascitechnol.v43i1.56383 fegert, j. m., et al., (2020) ‘challenges and burden of the coronavirus 2019 (covid-19) pandemic for child and adolescent mental health: a narrative review to highlight clinical and research needs in the acute phase and the long return to normality’, child and adolescent psychiatry and mental health, vol. 14, pp. 1–11. doi: 10.1186/s13034-020-00329-3 future of school. (2020) progress report on crisis schooling: national survey of america’s teachers, 19 june, [online] available at: https://www.futureof.school/fos-webi-replay/ progress-report-on-crisis-schooling-national-survey-of-americas-teachers hodges, c., et al., (2020) ‘the difference between emergency remote teaching and online learning’, educause review, 27 march, [online] available at: https://er.educause.edu/ articles/2020/3/the-difference-between-emergency-remoteteaching-and-online-learning international society for technology in education (iste), (2017) iste standards, [online] available at: from https://www.iste.org/standards. johnson, j. l., adkins, d. & chauvin, s. (2020) ‘a review of the quality indicators of rigor in qualitative research’, american journal of pharmaceutical education, vol. 84, no. 1, p. 7120. doi: 10.5688/ajpe7120 kasperski, r., blau, i. & ben-yehudah, g. (2022) ‘teaching digital literacy: are teachers’ perspectives consistent with actual pedagogy?’, technology, pedagogy and education, pp. 1–21. koehler, m. & mishra, p. (2009) ‘what is technological pedagogical content knowledge (tpack)?’, contemporary issues in technology and teacher education, vol. 9, pp. 60–70. leech, n. l., et al., (2020) ‘challenges of remote teaching for k-12 teachers during covid-19’, journal of educational leadership in action, vol. 7, p. 1. lien, c. m., khan, s. & eid, j. (2022) ‘school principals’ experiences and learning from the covid-19 pandemic in norway’, scandinavian journal of educational research, pp. 1–16, advance online publication. doi: 10.1080/00313831.2022.2043430 lowenthal, p., et al., (2020) ‘thinking beyond zoom: using asynchronous video to maintain connection and engagement during the covid-19 pandemic’, journal of technology and teacher education, vol. 28, pp. 383–391. http://dx.doi.org/10.25304/rlt.v31.2982 https://read.oecd-ilibrary.org/view/?ref=434_434914-59wd7ekj29&title=the-impact-of-covid-19-on-student-equity-and-inclusion https://read.oecd-ilibrary.org/view/?ref=434_434914-59wd7ekj29&title=the-impact-of-covid-19-on-student-equity-and-inclusion https://www.masslive.com/news/2020/07/zoom-isnt-perfect-but-educators-are-learning-how-to-make-it-work.html https://www.masslive.com/news/2020/07/zoom-isnt-perfect-but-educators-are-learning-how-to-make-it-work.html https://www.suttontrust.com/wp-content/uploads/2020/04/covid-19-impact-brief-school-shutdown.pdf https://www.suttontrust.com/wp-content/uploads/2020/04/covid-19-impact-brief-school-shutdown.pdf https://doi.org/10.1007/s10956-022-09958-z https://doi.org/10.1007/s10956-022-09958-z https://doi.org/10.2458/azu_itet_v4i1_desantis https://doi.org/10.2458/azu_itet_v4i1_desantis https://doi.org/10.4025/actascitechnol.v43i1.56383 https://doi.org/10.4025/actascitechnol.v43i1.56383 https://doi.org/10.1186/s13034-020-00329-3 https://www.futureof.school/fos-webi-replay/progress-report-on-crisis-schooling-national-survey-of-americas-teachers https://www.futureof.school/fos-webi-replay/progress-report-on-crisis-schooling-national-survey-of-americas-teachers https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remoteteaching-and-online-learning https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remoteteaching-and-online-learning https://www.iste.org/standards https://doi.org/10.5688/ajpe7120 https://doi.org/10.1080/00313831.2022.2043430 r. kasperski et al. 14 citation: research in learning technology 2023, 31: 2982 http://dx.doi.org/10.25304/rlt.v31.2982 (page number not for citation purpose) manca, f. & meluzzi, f. (2020) strengthening online learning when school are closed: the role of families and teachers in supporting students during the covid-19 crisis, [online] available at: https://read.oecd-ilibrary.org/view/?ref=136_136615-o13x4bkowa&title=strengthening-online-learning-when-schools-are-closed miao, f., et al., (2020) ensuring effective distance learning during covid-19 disruption: guidance for teachers, [online] available at: https://unesdoc.unesco.org/ark:/48223/ pf0000375116 ministry of education. (2020) reports from the field, [online] available at: https://pop.education.gov.il/online-learning/teachers-reported/ muñoz-najar, a., gilberto, a., hasan, a., cobo, c., azevedo, j. p. & akmal, m. (2021) remote learning during covid-19: lessons from today, principles for tomorrow, world bank. nowell, l. s., et al., (2017) ‘thematic analysis: striving to meet the trustworthiness criteria’, international journal of qualitative methods, vol. 16, no. 1, pp. 1–13. doi: 10.1177/ 1609406917733847 perla, l., agrati, l. s. & vinci, v. (2018) ‘the ‘supply chain’ of teachers’ digital skills training’, in the tpack traceability in the teachers’ trainers. grial research group (ed.), proceedings teem’18 sixth international conference on technological ecosystems for enhancing multiculturality, salamanca, spain, 24–26 october 2018, pp. 604–612. porat, e., blau, i. & barak, a. (2018) ‘measuring digital literacies: junior high-school students’ perceived competencies versus actual performance’, computers & education, vol.  126, pp. 23–36. redecker, c. (2017) european framework for the digital competence of educators: digcompedu, y. punie (ed.), eur 28775 en, publications office of the european union, luxembourg. rivera, r., et al., (2021) ‘planning engaging, remote, synchronous didactics in the covid-19 pandemic era’, jmir medical education, vol. 7, p. e25213. doi: 10.2196/25213 sen, a., & tucker, c. e. (2020). social distancing and school closures: documenting disparity in internet access among school children, [online] available at ssrn: doi: 10.2139/ssrn.3572922 shamir-inbal, t. & blau, i. (2021) ‘facilitating emergency remote k-12 teaching in computing-enhanced virtual learning environments during covid-19 pandemic-blessing or curse?’, journal of educational computing research, vol. 59, no. 7, pp. 1243–1271. unesco. (2020a) global education monitoring report summary 2020: inclusion and education: all means all, unesco, paris. unesco. (2020b) policy brief: education during covid-19 and beyond, august, [online] available at: https://www.un.org/sites/un2.un.org/files/sg_policy_brief_covid-19_and_education_august_2020.pdf valeeva, r. & kalimullin, a. (2021) ‘adapting or changing: the covid-19 pandemic and teacher education in russia’, education sciences, vol. 11, p. 408. doi: 10.3390/educsci11080408 wango, g., wairire, g. & kimamo, c. (2020) ‘anxiety and emotions of covid-19: the emotional underpinnings of dealing with the corona virus disease’, journal of psychology and behavioral science, vol. 8, no. 2, pp. 1–18. world bank group. (2020) rapid response briefing note: remote learning and covid-19 outbreak, 16 march, [online] available at: http://documents1.worldbank.org/curated/ en/266811584657843186/pdf/rapid-response-briefing-note-remote-learning-andcovid-19-outbreak.pdf world health organization. (2020) report of the who-china joint mission on coronavirus disease 2019 (covid-19), world health organization, geneva, [online] available at: https://www.who.int/docs/default-source/coronaviruse/who-china-joint-mission-on-covid19-final-report.pdf zhou, l. & li, f. (2020) ‘a review of the largest online teaching in china for elementary and middle school students during the covid-19 pandemic’, best evidence in chinese education, vol. 5, pp. 549–567. doi: 10.15354/bece.20.re040 http://dx.doi.org/10.25304/rlt.v31.2982 https://read.oecd-ilibrary.org/view/?ref=136_136615-o13x4bkowa&title=strengthening-online-learning-when-schools-are-closed https://read.oecd-ilibrary.org/view/?ref=136_136615-o13x4bkowa&title=strengthening-online-learning-when-schools-are-closed https://unesdoc.unesco.org/ark https://pop.education.gov.il/online-learning/teachers-reported/ https://pop.education.gov.il/online-learning/teachers-reported/ https://doi.org/10.1177/​1609406917733847 https://doi.org/10.1177/​1609406917733847 https://doi.org/10.2196/25213 http://dx.doi.org/10.2139/ssrn.3572922 https://www.un.org/sites/un2.un.org/files/sg_policy_brief_covid-19_and_education_august_2020.pdf https://www.un.org/sites/un2.un.org/files/sg_policy_brief_covid-19_and_education_august_2020.pdf https://doi.org/10.3390/educsci11080408 http://documents1.worldbank.org/curated/en/266811584657843186/pdf/rapid-response-briefing-note-remote-learning-and-covid-19-outbreak.pdf http://documents1.worldbank.org/curated/en/266811584657843186/pdf/rapid-response-briefing-note-remote-learning-and-covid-19-outbreak.pdf http://documents1.worldbank.org/curated/en/266811584657843186/pdf/rapid-response-briefing-note-remote-learning-and-covid-19-outbreak.pdf https://www.who.int/docs/default-source/coronaviruse/who-china-joint-mission-on-covid-19-final-report.pdf https://www.who.int/docs/default-source/coronaviruse/who-china-joint-mission-on-covid-19-final-report.pdf https://doi.org/10.15354/bece.20.re040 an interactive approach to learning economics: the winecon package jean b. soper and alexander b. macdonald department of economics, leicester university, uk abstract under the tltp initiative, the economics consortium is developing an interactive computerbased learning package called winecon. the package is directed at first-year economics undergraduates, particularly those taking economics as a supplementary course. using recent technological developments, the aim is both to facilitate a further increase in student numbers without a proportionate increase in teaching staff, and to provide a better method of student learning. some key elements of winecon are set out in this paper and demonstrated by screens produced at leicester university. methods of presenting textual information that give the user control over accessing it are described. for learning difficult concepts, a visual active learning approach is discussed. it involves user interaction and step-by-step analysis. the importance of flexibility and choice is emphasized, and the capacity of the computer to assist in deepening and consolidating learning is shown. introduction winecon is a software package offering a new and interactive approach to learning economics for students taking a first-year university course. it is being developed by the economics consortium as part of the teaching and learning technology programme (tltp) set up in the uk by the higher education funding councils. the authoring package chosen by the consortium is asymetrix toolbook which runs under windows, and the consortium software contains many graphical user-interface features that will be familiar to any windows user. students using winecon work at their own pace, and can adopt either an inquisitive or an acquisitive approach. they progress through colourful and varied computer screens, choosing what they would like to look at next, and whether they wish to receive further information. different types of feedback are provided to students about their progress. some screens change their appearance to indicate what the student has done. others indicate whether an answer or a curve movement is correct, and provide either a hint or the correct answer. menus of different styles in different contexts allow students to select the particular aspect of a topic they wish to study, and pop-ups activated by buttons present key concepts in an arresting way. economic analysis is explained with animated graphics, and diagrams are built up in stages 14 alt-j volume 2 number 1 adding one curve at a time. one of the advantages in developing the package as part of a consortium is cross-fertilization of ideas and cross-checking of the work produced. materials for winecon are being produced separately at different sites, with central coordination and leadership from the university of bristol. a common look-and-feel has been achieved by providing all developers with a core template which customizes toolbook to standards agreed by the consortium, and which also provides additional facilities. under this, for example, all screens use a grey textured background and have the same set of buttons in the bottom window bar. these allow the student to access utilities such as the glossary, and to navigate through the package, moving either to an adjacent screen or to a higher-level menu. the topics described here were authored at leicester university, and the screens illustrate particular principles of common interest to many different university disciplines which we were concerned to address. (the screens are taken from topics in both the welfare economics and regulation of markets and market failure modules, two of the three modules being developed at leicester.) objectives the objective of the funding bodies is to harness technological potential so as to enable an increase in student numbers to take place without an accompanying proportionate increase in the number of teachers. members of the economics consortium envisage that the computerbased learning package they are authoring will replace some lectures, but that its greater impact will be in substituting for labour-intensive tutorials and classes, and in checking and assessing students' progress. equally importantly, however, the consortium regards the software design process as an opportunity to develop better methods of learning. understanding and retention of material is greater when students are actively involved in the learning process. students using winecon cannot be passive observers, because progression through each screen and on to the next requires some meaningful user-interaction such as clicking with the mouse or typing appropriate responses. to prevent responses from becoming automatic, the designers of winecon have required a considerable variety of types of userfigure 1: presenting textual information method 1 01993. tltp economics consortium types of tax in the u.k. 7a* revenue can be raised by the government :*f the aaivhipg nr w*i=cti u k rates sppiy zan be identified thd tniea t-nns of tax :n ins :j k are escribed belfnv to ;)£ noie fifomalion shou! each r » ciick the siiscort g-een circle direct taxes these are taxes which are levied directly on individuals and firms and are payable on earnings from labour, rent, dividends and interest ® personal income tax d^ national insurance ® corporation tax © capital gains tax t wealth taxes these are taxes levied on wealth itself rather than the income which is derived from wealth. paid either on general assets or specific assets acquired. o inheritance tai council tax indirect taxes these are taxes levied on sellers when goods and services are purchased and are thus paid indirectly by individuals and firms when they buy from the sellers ® value added tax o excise duties © customs duties 15 jean soper and alexander macdonald an interactive approach to learning economics 01993. tltp economics consortium requirements of a good tax system: as discussed in the topic "equity & resource allocation" people earning the same level of income and with the same personal circumstances should pay the same level of income tax. taxes should be fairly apportioned so as to redistribute income from the rich to the poor. to achieve this the system must address the issue of ability to pay. it would seem equitable if those who can afford it pay more than those who cant card eight m ssisbi s i figure 2: presenting textual information method 2 interaction, some of which are especially appropriate to particular aspects of the learning process such as recall and integration of material. in universities, the main body of course material has traditionally been presented in lectures, where gifted teachers may have stimulated and inspired their students. a computer perhaps lacks the ability to inspire, but it can present all the basic material in an interesting way. the authors of winecon aim to set out the main points clearly and to present information in a way that is easy to absorb. an acquisitive student will progress steadily through the material on a screen, and move on to the next screen of the topic. the software will introduce the main ideas, help students through difficult aspects and suggest avenues for further study both within and outside the package. moreover, the advantage of the computer is that it provides a visual approach where the inquisitive student can construct and deconstruct diagrams or graphs, and immediately observe the effect of doing so. the comparative advantage of the computer over the lecture, textbook or tutorial is this dynamic element which proves so elusive to other media. the principles involved are an extension of those formulated in developing the use of spreadsheets in the learning process. it was seen then that by working through a calculation one step at a time the student gained insight into it (soper and lee 1990). flexibility extends the life and widens the usefulness of a software package. the designers of winecon are building flexibility into it in various ways. screens are being designed to accommodate the updating of information as this becomes necessary, and to give students many alternative access points. lecturers will be able to customize the package to suit their particular lecture course. these features will widen winecon's appeal, and ensure it can offer something to people with different backgrounds in economics. this is further enhanced by the inclusion of more and advanced buttons. the more button will give the student access to another layer of information on a specific topic. by contrast the advanced button will allow those students with a strong background in economics to explore topics in greater depth, typically one which is beyond the requirements of the" mainstream first-year course (for example, the diagrammatic analysis may be made more precise using calculus). 16 alt-j volume 2 number 1 presenting textual information the design of screens such as those shown in figures 1 and 2 is driven by two complementary factors: user-interaction and the physical size of the screen. a concern of the winecon authors is that in those cases where topics are particularly textual, the package should not take the form of an electronic book. thus, where there is a large volume of text to be presented which cannot be separated between screens, it should be set out in a way which allows the user to access it in logical chunks. this requires the text to be broken down into meaningful sections, the sum of which, when combined with the control the user has over access, is hopefully worth more to the student than just the whole text. naturally, the physical size of the screen imposes some constraint on how these elements of text can be presented by the software author. in figure 1 the sheer volume of material requires a pop-up mechanism so that the bulk of the information, which is stored 'behind the scenes'" can be accessed one part at a time. in figure 2, the volume of material permits it all to be physically on the same screen, but in such a way that the user has to get involved with the package in order to read it. on both screens, the user has a visual reminder of exactly what information has been accessed. figure 1 is a screen-shot from the topic on the structure of taxation. this page gives students information about the types of taxes in the uk. the screen explains what direct taxes, wealth taxes and indirect taxes are, giving the major examples of taxes in each category. the student can click the green circle button adjacent to a tax name to find out more about that tax, and a black centre will appear in the circle, indicating that the student has displayed the further information about that tax. the student can move to a new screen at any time. if later the student returns to this screen, then provided that he or she has not quit the package in the meantime, the record of what has previously been studied will still be available. careful planning of the screen design and programming will make the winecon package flexible. this is particularly important with screens such as that shown in figure 1 where the information will require frequent updating. information which is likely to change often, such as tax rates and allowances, will be held in a database for easy updating (subject to the obvious constraint of volume of data). the whole screen is designed so that any major figure 3: clear presentation of contrasting definitions 01 993. tltp economics consortiun types of equity: • horizontal equity horizontal etraity is concerned with &e identical treatment of identical people this roles out discrimination on the basis of race or sex between people whose economic characteristics an" identical. vertical equity vertical equity refers to &e different treatment of different people with the aim of reducing the consequences of these differences. this includes the "robin hood" principle of taking from the rich and erring to the poor, i 17 jean soper and alexander macdonald an interactive approach to learning economics changes, such as the addition or removal of tax examples, can be implemented using only one access point. for users in countries other than the uk it will be possible to substitute an alternative screen relevant to their country for that shown in figure 1. an alternative method of presenting textual information in organized and easy-to-assimilate chunks is shown in figure 2. here, a card-index type approach is used, with the information about each requirement of a tax system being available on one or two cards. the user clicks a card to view it, and its title bar then changes from cream to green, thus creating a record of what has been studied. even in screens which are concerned with presenting text, students are involved in the learning process, and have a choice about the material presented to them. the package is therefore useful to students with different amounts of prior knowledge, and the choices it offers give easy access to selected material for revision purposes. the screen shown in figure 3 presents two definitions. typically, definitions in winecon are accessed at the click of a standard definition button on the first page of each topic. in this case, the topic is based on two definitions of equity which are presented together so that they jointly form the focus of the screen, rather than being supplementary information to be accessed via the click of a button. the screen in figure 3 illustrates the design objective that the software should draw attention to important points, present them clearly, and compare and contrast alternatives. another feature of winecon, implemented via the core template, is the 'roladex' mechanism for presenting voluminous text. roladex is the metaphor adopted by the economics consortium to describe its linear-access card mechanism (unlike the hypercard system, roladex does not facilitate random access of cards). roladex can be seen in figures 4a and 4b, at the upper left-hand side of both screens. with this mechanism, the user clicks the appropriate button to access sections of text as a linear progression of cards. in figure 4b, the student is reading the sixth card out of the total of the six cards available. as with the other card mechanisihs, large sections of text are presented in smaller, more readable divisions. a particularly attractive feature of the roladex mechanism is that movement from one card to the next can trigger another event on the same screen. in this way, a click tratton t below and observe what happeas lathe allocations this morement is a pareto teen nude better off, no one has bean allocation of goods, a , • jt] | figure 4a: learning the theory by example 18 alt-j volume 2 number 1 student can progress through several cards of text which relate to the same diagram while that diagram remains on the screen. at the same time, additions can be made to the diagram (or things can be removed), such as the plotting of a new equilibrium point or the introduction of a new curve or graphic. the methods chosen by the economics consortium for presenting textual information recognize that a computer screen is not necessarily a suitable medium for conveying the written word. care is being taken to improve readability by separating the text into logical divisions, and by using sans-serif fonts for clarity. when continuous text is printed, a serif font is generally recommended as assisting the flow of reading. on screen, however, the extra complexity of the serif font makes the letters somewhat disjointed, and so the text becomes difficult to read. the fonts being used by the economics consortium are standardized in the core template. they comprise arial fonts in three different sizes and faces for different circumstances, for example titles, headings, graphs and continuous text. interactive learning of concepts a concept which is difficult to grasp can first be demonstrated as an interactive example on a computer. with the screen shown in figures 4a and 4b, students investigate the pareto criterion for comparing different allocations of two goods between two people. before entering this screen, the student is given the opportunity to customize the example by changing the names of the people in it. these names then replace the default names in the text and on the graph. on the screen shown in figure 4a, the user is asked to click button i to see an example of a pareto improvement. this is demonstrated by both the marker at point a turning red and moving from point a to point b, while simultaneously the pie chart at the bottom of the screen slowly changes to reflect the new allocation of goods between the two people. progressing through the text, the student is instructed to click the remaining buttons to discover more about pareto improvements. when button v is clicked, as instructed iri the screen shown in figure 4b, the point moves from a to e, which is on the same line as b, and the pie chart changes accordingly. it might superficially seem that another pareto improvefigure 4b: learning the theory by example 01993. tltp economics consortium {example-' • \ ---' ' fina% e t a * buttonv below, i ",* ths change of ̂ locations t a n nrt be , emtipsfa* using itepsretocrtfenon , t h o u g h there has been sn mckmae '• efficiency, laura b s s been made taber i oflwhtistzigjyisnwtieworse off , * allocation of good?, e 19 jean soper and alexander macdonald an interactive approach to learning economics nient has occurred. this, however, is not so. as the text explains, one person, ziggy, has been made worse off, and therefore the two points a and e cannot be compared using the pareto criterion. . another illustrative example is shown in figures 5a and 5b. these screens incorporate a further design principle of the package, namely increasing the depth of learning by checking the student's understanding of concepts as they are applied in a new context. the two screens shown form part of a set in which the student views four possible alternative allocations of a cake between two identical people, and is asked to respond to questions about each allocation by clicking the yes or no button. the separate ideas of equity and efficiency should already be familiar to the student, but these screens require them to be brought together. in figure 5a, the user has clicked the button allocation one, which causes a whole cake to divide and then be allocated as shown. in this case, the student has answered the first question by clicking either yes or no (the corresponding pair of buttons are now greyed out), and winecon is now ready to receive the answer to the second question. figure 5b shows allocation two when the student has revealed the new allocation and answered the question concerning equity. regardless of what the student chooses, the correct answer appears in the allocation status box and subsequent correct answers are added to describe the allocation. to help the student answer the questions, the box at the top right of the screen records numerically the allocations of cake for each example. when the student has completed all the allocations, feedback is provided about how many answers were correct, and an opportunity to repeat the questions is given. monitoring and feedback interactive software such as winecon is designed to involve students in their own learning in a variety of ways. one of these is their responding to the feedback which the computer offers them about their progress. with the screens shown in figures 5a and 5b, the student receives immediate feedback about the correct answer to each question. each is used in constructing a final answer, and the screen designers felt it important that students should have the correct information, although o1993, tltp economics consortium '.horizontal equity-examples;;" • .: . c w ^ f b e allocation ofiniscakebetweeoiwo ' i d e n t i c a l s a n n f o . . "•' . ' : ' . ' • " . " allocation: person 1 pfrson 2 one 1/3 1/3 two three four allocation one this allocation is equitable this allocation is efficient lyesj i "no allocation status equitable because both have the same amount of cake figure 5a: simple presentation of hard-tograsp concepts 20 alt-j • volume 2 number 1 they may not study it as it is received. for acquisitive students proceeding through the steps on the screen, it is helpful that they are told at the end how many of their answers were correct. this will alert them to the state of their understanding of the topic. we have considered the possible provision of rewards for full marks on questions sets. machines with sound cards might play a tune of the student's choice (assuming headphones are also provided) or some fun item might appear on the screen, possibly a game. for less successful students, we have discussed whether some humorous portrayal of their deficiencies might make them feel prepared to try again. further consideration will be given to feedback as we produce the test module, which is still under development at the time of writing. another aspect of computer-based learning is computer monitoring. the computer can record what individual students do, so that an account of the screens they have accessed, the information they have revealed and the answers they have given to questions can be available to them. it could also be made available to the students' tutors, but whether it should be is a matter for debate which should perhaps be decided separately at each institution that installs the software. computer monitoring is something the consortium plans to consider in more detail once the main software development phase is nearing completion. step-by-step analysis of complex material much economic analysis is presented by means of graphs, and students have to understand what each curve on a graph represents. one of the advantages of a computer is that it can show in a matter of seconds how a particular graph is built up, whereas a textbook would require several pages of text to describe it. figure 6 shows the start of a diagram for a perfectly competitive industry. the screen invites the user to click the five buttons in sequence, which builds up the graph one curve at a time. after the first button has been clicked, the screen displays the industry demand curve as shown in figure 6, with an explanation about the curve appearing in the box to the side of the buttons. the student can watch the graph being built up until the long-run equilibrium position of a perfectly competitive industry is found. for those already familiar with the basic curves, a show curves button is provided. figure 5b: simple presentation of hard-tograsp concepts 01993, tltp economics consortium horizontal jecfuity one cakb between 1wo • allocation two allocation: pfrson1 person 2 one 1/3 1/3 two 1/2 1/3 three four bartons . , / fedn* fc? rickino tip spprop'iote this allocation b equitable and more efficient than allocation one i t « a pareto optimal allocation unallocated allocation status inequitable as one person has more fake than the other es naniia i h 21 jean soper and alexander macdonald an interactive approach to learning economics ©1993, tltp economics consortium perfectly competitive industry westat) wisfcapelfccttyb •fm a jjmdei of ste tri<taa»sr *>„ joag m oftdsbnam-, 1 >',"* dick flw'sw fewsos tetowin • j industry ui'mand j industry supply industry murqirm perfectly competitive industry industry averngr cast equilibrium industry demand is a standard demand curve. figure 6: adding curves to build a graph the computer-based learning approach is more visual than the textbook one, and it is also flexible to the needs and wishes of the student. guiding the student one step at a time through complex material is the approach that a lecturer or tutor would use when teaching the subject live. an economics teacher would also indicate points, distances and area on graphs that have particular interpretations. the computer can do this also. we have already noted that different colours can be used, but the screen shown in figure 7 has a further method. as the student clicks each button in turn to reveal the definition of a concept, the area or distance which represents that concept on the graph flashes to attract the student's attention and ensure that it is correctly identified. in the screen shown in figure 7, the student has clicked the consumer surplus button, and then on clicking the producer surplus button, the text shown appears one step at a time while simultaneously the corresponding lines and axis labels flash. the conclusions of economic analysis are often deduced by comparing two contrasting diagrams. when a perfectly competitive industry is monopolized, there are many points of comparison to be drawn. figure 8 shows a screen where the student observes the differences between the two industries. having clicked buttons (obscured in the screen-shot) to reveal the competitive equilibrium and set the condition required to determine production under a monopoly (the mr curve, which diverges from ar, is needed), the student then progresses through the buttons to reveal changes in price, quantity and surpluses as a result of monopolization. as each button is clicked, a pop-up completes the button caption so as to explain what is happening in the diagram as each component is revealed. screen design aspects of screen design relating to readability, flexibility and choice are discussed in the earlier section of this paper on presenting textual information. some more general issues of screen design will now be addressed. 22 alt-j volume 2 number 1 the economics consortium recognized at an early stage of production that a degree of consistency between screens would be desirable, for both aesthetic and practical reasons. the former are concerned with giving unity to the package as a whole, and the latter with ensuring that students feel in control of winecon. in particular, students need to know firstly where to begin on a new screen, and secondly when they have completed it and should move on to the next. there are also good reasons, however, for believing that a variety of screen styles is required. different developers author in different ways, each of which may have its own advantages, so allowing developers freedom to try out their ideas is likely to enrich the package as a whole. consistency between sites is not necessarily jeopardized by this, because ideas originating at one site are often adapted at another, with the janet network facilitating frequent exchange of programming code. as educators, we are also concerned to produce a package which will maintain students' attention and prevent them from predicting exactly what will follow next, which is a further argument for variety between screens. the economics consortium's approach to this multi-faceted question is a fairly minimal amount of standardization comprising such items as icons, fonts, a screen background, some buttons and commonly required graphics. the scope for variety that is still available to developers is illustrated by the considerable variety of styles exhibited even in the limited number of screens depicted in this paper. a major design consideration is to partition the screen in such a way that the eye is not overworked. in winecon, screens are often divided into a number of unequal sections which serve different purposes. frequently, a large part of the screen is devoted to presenting the textual analysis, with a similarly sized area containing the diagram. a small portion of the screen is typically given up to user instruction. throughout winecon, such areas are identified by professor fields, an example of which can be seen in the top left-hand corner of the screen in figure 8. by this means, the student'will quickly become familiar with where to look for screen instructions, and additionally the student can choose the identity of the professor during the initial customization of winecon. another different use for which a small section of the screen may be required is to store information or present it in a different format for reference purposes. alternatively, a set of buttons with which the student controls the figure 7: attracting the user's attention by flashing objects on screen 01993. tltp economics consortium dick the buttons below in tain. consumer surplus consumer surplus is the area dbiwben file demand cijnre and unit price, pc, and is shown by the red triangle. first find price per unit, pc then find cost per unit, cc and the number of units produced, qc producer surplus 1st (pc-ccjxqc (price per unit cost per unit) multiplied by the number nf units. butaspccc .then pc-cc-fl and so producer surplus is zero. explanation consumer, producer & social surplus in competition. 23 jean soper and alexander macdonald an interactive approach to learning economics 01993. tltp economics consortium changss resulting from monopolisation l'mnplfilp ihi* klnlrmfsnni hi inw by ihietanq ihc .£* i iiitp^innilini) hullnm vaaer mnnouoly. uunnmyis j i consumer surplus is j j loss ol social kurjilus m i | prict* is j i producer surplus is i 1 i sonibl suriilus isunit c o m is j figure 8: step-by-step approach to presenting analysis action of the screen may be grouped together in one part of the screen as in figures 4a and 4b. on a simple screen such as that shown in figure 3, where instructions and buttons are not required, the user is assumed to read the material in the conventional way from left to right and from top to bottom. the fundamental concern here is with balance. the division of the screen may be most important on screens which contain purely textual information. as with the printed word, the less clutter on a page the more willing is the reader to engage that page. thus, although a screen may contain vast amounts of information, it should be presented so that superficially it appears simple and uninvolved, thus capturing the'user's interest at first sight. colours can also be used to attract the user's attention, but care must be taken to ensure that the student is not entering the world of psychedelia. if a strong colour is to be used to highlight a particular aspect of analysis, this should be done subtly so that the eye is naturally drawn to that object or area of the screen. if the whole screen is a wash of vibrant colour, although such a screen will undoubtedly capture the user's initial interest, the power of colour in retaining that interest throughout the analysis is lost as the user's eyes become tired when continually faced with strong screen colours. the economics consortium has chosen a neutral grey colour for screen backgrounds, allowing developers a wide choice of colours that stand out against it. the use of a subtly textured grey background contrasts well with metallic buttons and dithered backdrops for graphs and diagrams. in this way, the student does not have to deal with the author's overworked imagination but can concentrate on the analysis .being presented. flexibility and choice very large textbooks are off-putting to students and expensive to produce. authors of books therefore have to choose between alternative presentations of material, and teachers usually find that even the textbook they like best does various things in ways they would not choose. the volume of material being stored in winecon is not immediately apparent to the student, 24 au-j volume 2 number 1 and where it seems likely to be useful, the package will include alternative presentations. one of the changes resulting from monopolization which the screen shown in figure 8 derives, is a measure of the deadweight loss of monopoly. figure 9 explains that the deadweight loss can also be measured in a different way, and asks the student to investigate why the measures are the same. a student who does not wish to compare the two measures can easily proceed to the next screen by clicking the appropriate screen-movement button. and lecturers who wish their students to study only selected screens will be able to customize the package for their course and exclude specified sections. all the screens in winecon will have a unique name so that after customization each screen can still be identified and thus cross-screen references maintained. students sometimes have difficulty in picking out relevant material from a textbook. with lecturer customization, a computer-based learning package can be tailored to a particular course. in addition, the menu mechanism allows the student to navigate through the package to a particular page. each module has a standard style top-level menu listing the topics available in that module. on clicking a topic button, the user enters that topic and is confronted with a second-level menu specific to the topic, one of which is shown in figure 10. this menu allows the student to go straight to a named page within the topic by clicking the page name. from any screen within a topic, the student can click a button in the bottom window bar to return to the second-level menu. from this page, the student can click the module map button to return to the top-level menu of topics. this system allows students to have total control over where they start within a module. figure 10 shows the menu of choices for the welfare effects of monopoly topic. students are encouraged to click the roladex button to reveal them one at a time, as a lecturer would do with a real overhead projector. a reveal all button is also provided, however, and this gives fast access to particular sub-topics when required, for example for revision purposes. the menu system, together with the glossary to be included in the finished version of the software is expected to provide students with easily accessible self-help facilities where they figure 9: using the existing diagrams to present an 'alternative view' 01993. tltp economics consortium deadweight loss of monopoly an alternative measure ai wy have ssen, tne cp<idw?.ghl lass wtitr, ojlpul ii. restricted from qr in qm ca^ ae msasijrod by the :aa area ttf change in got.a! surplus whnr. p •s restricted from qc to oncan atee be by tl.p green area tha purple upper trinnqlh the yellow lnwt?r tnanqle deaijwmqm loss is the same whether measured as the: green area. i 0 r red area why? m b i b mm 25 jean soper and alexander macdonald an interactive approach to learning economics 01993. tltp economics consortium welfare effects of monopoly • introduction • model comparing monopoly & competition • consumer, ft-oducer & social sarpms in competition • consumer, producer & social surplus under monopoly • changes resulting from monopolisation • deadweight loss of monopolyan alternative measure • monopolisation of a constant cost industry • summary aide on a title to go straight to it or proceed through the topic using the next page button ciick tfe salads*..,,' buttons .apposite jo * re»'3liiieissues.:. rcvcnl all (h hiisi b i figure 10: the menu of choices for the topic welfare effects of monopoly can get answers to their individual problems. as tutors are increasingly busy, and computers are more widely available, we anticipate students choosing to go to winecon with their difficulties. they may even find winecon more approachable than their tutor because they know it will not look down on them when they ask for help on points which all the other students seem to understand. depth of understanding one of the roles of tutorials in conventional teaching is to allow student to discover the extent of their knowledge of a topic. economics tutors may ask their students to draw for themselves diagrams that have been presented to them. winecon checks students' understanding of diagrams by asking them to indicate how they should be built up, and by requiring them to move curves to their correct position. if they are wrong, they are invited to try again and if necessary may be given some suggestion on how to proceed. should they wish to, they can try the analysis again and again. another possible advantage over a tutor is that the computer's patience is inexhaustible. before encountering the screen shown in figure 11, a student is expected to have learned about equilibrium in perfect competition and monopoly from screens such as those in figures 6 to 9. the screen shown in figure 11 asks the student to indicate which curves are required to determine the equilibrium position of a perfectly competitive industry. necessary curves appear in the diagram when the buttons for them are clicked, while other buttons activate popup error messages containing hints about what to do. students must next demonstrate that they understand the concept of the industry choosing to operate where marginal cost equals marginal revenue, because the ordinate marked on the graph must be dragged to that point for the analysis to proceed. if it is correctly positioned, the ordinate disappears, the construction lines are drawn in and price and output are marked pc and qc respectively. otherwise, the ordinate returns to its original position and a message appears saying that the position chosen was wrong, and inviting the student to have another attempt. having correctly positioned the ordinate, the student then proceeds to determine monopoly equilibrium, and to position the 26 alt-j volume 2 number 1 long-run average cost curve using the same method. screens such as this apply the familiar windows drag-and-drop procedure to involve students in creating graphs. this method of learning is particularly attractive as it allows the student to experiment with ideas in the absence of time pressure, and, more importantly, to make mistakes and learn from them without fear of being derided by their peers. in this way, it is hoped that the student can gain further understanding of concepts and perhaps more confidence in experimentation with what would otherwise be presented as fact. once the somewhat complex diagram shown in figure 12 has been built up, the student steps through the analysis of profits in both industries by using a multiple click button, the caption of which changes each time it is clicked. the formulae used are displayed in the form in which they would be hand-written by the student. further development and pilot testing although the formula displayed in figure 12 is for explanatory purposes only, we are designing a mechanism with which the student can answer numerical questions (particularly those requiring algebraic solutions) by selecting terms from a palette. these responses will be converted by winecon's program into a true mathematical expression so that they can be evaluated algorithmically. such a system will typically be used in the mathematics for economists module (the third module in production at leicester). the attractive feature of this palette mechanism is that it does not require students to have to learn a new method for writing mathematical expressions (for example, 4 * x * x instead of 4x2) in order to provide answers to these types of questions. another part of the package, which is still at the development stage, is the test module. this will contain questions on all parts of the package that are suitable for computer marking. these questions will be contained in a database so that lecturers will be able to add. their own questions and/or customize those which already exist. with numerical questions, we expect to provide different data randomly to different students. we hope that winecon will generate discussion among students, but we do not want them simply to be able to type in answers that figure 11: revising the theory with drag-anddrop graphics ©1993. tltp economics consortium output, price and profit m competition and monopoly fat aujr t w v e s you require . to determine enumsrairo itjen <&ck a**d drag tfra < vertical sue # 9 to mark' ,«r 3 read tits roladex card first m. mmm'mm 27 jean soper and alexander macdonald an interactive approach to learning economics 01993. tltp economics consortiu output, price and profit in competition and monopoly • 'thw ctffectly position the . average east f a q t u w e if this is long rim. equilibrium' by dickiji^ and dragging s t .seas t&jeffiaf j>nrifc»iraifc,\ 1 m competition aw)" •";-.'".';'• 3-w:3; i 1 " each unit of output gnnn a profit of (p a q :' in monopoly h > ac so profit-(pm acnq'flm figure 12: stepping through the analysis with multiple choice buttons someone else has calculated. the economics consortium recognizes that winecon is likely to be more successful the greater is the involvement in its development of those who may use it. hence it welcomes associate members who provide regular feedback on the package as it is being produced. user-evaluation is also considered important. some of the material presented in this paper was trialled with a small number of leicester students in june 1993, and selected tutorial groups are using it to write assignments in january 1994. a large trial involving around 200 students is also in progress at another development site. it is too early to be sure how the intended users will react to winecon, but the initial responses have been very positive. typically, students feel that when using the computer they are more in control of the information presented to them, and thus feel that there is less pressure involved with the process of learning. the suggestions of associate members and the information gained from the student trials will all have a bearing on the revisions made as the final version of winecon is produced. the work of the consortium is some way near to completion, with winecon projected to be generally available in october 1994. in total, winecon will consist of 22 modules covering all the requirements, for introductory economics, including skills-based modules in data handling and mathematics. conclusion the most likely application of winecon is in replacing some tutorial classes. students will be able to cover material dealt with in lectures in their own time using the software. each module will explain the key aspects of the theory and allow the student to explore 'what happens if ..?•' situations. the software will then test them on how well they understand the material. by using winecon, it is hoped that students will feel they will have greater control over their learning process. not only will they be able to explore areas of interest using winecon, but those students who perhaps feel overwhelmed with the amount and speed of 28 alt-j volume 2 number 1 information which has been presented to them in lectures can use the package to learn the basic principles and concepts at their own pace and in a way which is more readily accessible than it would be with a conventional text book. finally, it is hoped that the methods of presentation which have resulted from the development of winecon will be widely applicable and useful to developers of computer based learning material in a variety of disciplines. acknowledgement the authors would like to thank an anonymous referee for helpful contributions and suggestions. associate members to become an associate member of the tltp economics consortium, contact: phil hobbs project director, economics consortium centre for computing in economics university of bristol 8 woodland road bristol bs8 1tn tel: 0272 288478 email: phil.hobbs@bristol.ac.uk references ayres, m. (1993), 'don't you just love being in control? interactive features of the winecon courseware', computers in higher education economics review, 20 (november), 27-34. burke, p. and soper, j.b. (1993), 'steps in learning about consumer surplus using asymetrix toolbook', computers in higher education economics review, 19 (may), 8-13. kelmanson, m.a., maunders, s.b. and cheng s.y. (1993), 'a mathematica-based cal matrix-theory tutor for scientists and engineers', association for learning technology journal, 1, 2, 27-39. kulik, j.a. and kulik c.c. (1988), 'timing of feedback and verbal learning', review of educational research, 58, 1, 79-97. macdonald, a.b. and soper, j.b. (1993), 'learning about the production possibility frontier (ppf) with asymetrix toolbook', computers in higher education economics review, 19 (may), 3-7. senge, p.m. (1990), 'the leader's new work: building learning organizations', sloan management review, (fall), 7-23. soper, j.b. and lee, m.p. (1990), statistics with lotus 1-2-3, 2nd edition, bromley (kent), chartwell-bratt. soper, j.b. and macdonald, a.b. (1993), 'interactive learning with winecon', computers in higher education economics review, 20 (november), 23-26. tltp economics consortium (1993), economics briefing, issue 1 (spring). 29 changing the role of tutors in distance education with information and communication technologies adrian kirkwood, ann jones and anne jelfs institute of educational technology, the open university the open university plans to make more extensive use of information and communications technologies (icts) for distance teaching and learning and for administrative contacts between students, tutors and the university's headquarters. this paper reports on a survey of the tuition and counselling (tac) staff, most of whom work only part-time for the ou. it established the extent to which tac staff currently have access to and familiarity with icts and their perceived needs for training and other forms of support for its effective use. the paper discusses the possible impact on tac staff of the greater use of new technologies in their ou work, and the organisational and pedagogic changes that may ensue. introduction the open university (ou) is a large, complex organization offering many types of course to a wide variety of adult learners. at the time of writing (1995), the university has over 200,000 distance learners, of whom more than 100,000 are taking undergraduate courses. the ou offers more than 140 undergraduate courses and many other courses in health and social welfare, management, community education, etc. there are about 7,700 tuition and counselling (tac) staff throughout europe. individual members of the tac staff provide student support as study counsellors and/or tutors for one or more of the ou's courses, and act as a student's local point of contact with the ou. they are usually contracted to work only part-time for the ou, and many are employed fulltime or part-time at another university or educational institution. within the ou system, there exist only limited opportunities for contact between students and their tutors. infrequent tutorial sessions provide the main opportunities for face-to-face interaction between tutors and students, although the telephone enables the discussion of problems and queries. communication by post forms a vital part of ou studies, with assignments being submitted by students for comment and grading by their tutors, who subsequently send back their responses to students. 35 adrian kirkwood et al changing role of tutors in distance education with information and communication technologies since 1988 a small number of courses have involved students and tutors in computer-mediated communication (cmc), and there have also been some small-scale experiments with cmc for students and tutors located in certain 'remote' areas (mason, 1995a). however, from 1996 there are plans for more widespread access to cmc and other online facilities for those with a modem, etc., and very soon the ou will offer online facilities for administrative purposes. access to information and communications technologies a survey of tac staff was undertaken in march 1995. questionnaires were sent to one-third of the tac staff, with full-time ou staff being excluded from the sample. a total of 2,044 responses were received (78.5% of the sample). when the data were analysed, differences between groups (age, gender, type of course tutored, etc.) were examined (kirkwood et al, 1995). the survey sought to determine: • the extent of access tac staff have to information and communications technologies (icts); • the types of equipment available to them and the location; • their attitudes towards using a computer for their ou work; • in what ways (if any) they had used a computer for their ou work; • their perceived needs for support and training in the use of icts. the survey found that, overall, a very large proportion of the tac staff has access to a computer, with only 12.3% having no access at all. if, however, we consider those who do not tutor courses that require students to use a computer, the proportion without access rises to 16.0%. the gender differences in both the quantity and quality of access that we have found with successive student surveys (jones et al, 1992) is repeated in the data for tac staff, i.e. men have better access than women. the type of computer most frequently used is a pc with windows, although there are many using other systems, especially apple macintosh and amstrad pcw machines as well as pcs without windows. less than 20% of those with a computer at home have a modem; about the same proportion have a cd-rom drive. very many tutors have already used a computer in connection with their ou work, even though only a small number of courses expect tutors to use one. many of the survey respondents felt that the ou should provide support for their use of icts and not simply in terms of financial assistance to help cover the costs of acquiring hardware or peripherals (a modem, cd-rom drive, etc.) or of online working. a large proportion felt that they should be trained in the use of software appropriate for their ou work. issues for discussion we believe that the results of the survey raise three issues which are particularly important to the ou, but which also have a wider relevance. expected computer literacy and competence the first of these issues is the level of computer literacy and competence that tac staff will be 36 alt-j volume 4 number i expected, in future, to have. currently, tac staff are selected on the basis of subject expertise (i.e. their knowledge and skills related to the subject of the course) and also on their relevant teaching expertise (i.e. their experience and skills of tutoring adult students both face-to-face and at a distance). knowledge and skills about using computers are necessary only, and indeed relevant only, when tutors are being considered for courses where these form part of the content (e.g. computing or information technology). the issue, therefore, is what the effects will be on recruitment and retention of tac staff if wider use of computers for ou teaching and administration leads to an expectation of access to icts and skills in their use. this is of particular concern if the expectation is applied in respect of courses that do not require students to use computers as well as those that do. might tutors without such skills find themselves disadvantaged even though they fulfil the criteria in terms of subject and teaching expertise? as has been discussed elsewhere (kirkup and jones, 1996), there is the real danger that while opening up new possibilities for some, increased use of icts closes down possibilities for others and for tac staff, as for students, those affected adversely are often groups who are already disadvantaged. who gains most from wider use of icts by tac staff? does the open university benefit from the computing knowledge and skills acquired by tac staff at home or at their main workplace? do the tac staff seek to gain experience and skills in their use of icts for ou activities that can be transferred to their domestic or main work situation? currently, the ou benefits from the general computing competence and skills possessed by tac staff. however, very many of the tutors with access to a computer feel that they need training in the use of software and applications, ranging from about 20% wanting training for word processing, to more than 50% seeking guidance for email and computer conferencing. a survey was recently conducted of tutors for the new (1995) ou course information technology and society (thd204), which requires students and tutors to use a computer for a number of applications, including cmc and accessing resources on a cd-rom disc. it found that many hope to increase their it skills in the process of tutoring the course: 'it appears that many tutors are getting their main training in the use of information and communications technologies (icts) for teaching through their ou work!' (kirkup etal, 1995, p. 17). within the wider context of uk higher education, the adoption and use of icts for teaching and learning may be less well developed than many enthusiasts expect. research on the transformation of higher education in the uk and the role of icts (based upon interviews with staff and students in more than 30 universities and colleges) suggests that with the exception of word processing and email (both internally and inter-institutionally) much, if not most, of higher education remains untouched by icts (albury, 1995). how will the tutor's role change with the increased use of icts for ou work? ou students' overall workload has tended to increase with the introduction of home-based computing as they learn to use the equipment as well as the subject being studied (jones et al, 1992). it is highly likely that tac staff will experience a similar increase in workload, at least initially. currently, the workload for ou tutors tends to be intermittent, in that it is greater when assignments are due and at the time of group tutorials than at other times. if the use of cmc 37 adrian kiriwood et ol changing role of tutors in distance education with information and communication technologies were introduced more widely, tutors would spend time throughout the year working online, and the workload would tend to be more continuous. the potential for online working and cmc might set up unrealistic student expectations about virtually unlimited levels of contact. as mason (1995a) has pointed out, increased electronic communication can lead to a particular paradox: it offers the potential for providing more learners with up-to-date information (which could reduce teacher resources); however, it also sets up an expectation of fast personal interactive feedback (which requires more teaching resource). although students currently have open telephone access to their tutors, social conventions have developed to regulate contact time, for example, no telephone conversations at 1.00 a.m. if tutors are to establish boundaries of the type that exist in telephone contact, and through that reduce their computerconferencing overload, they need to announce to their students when they will be logging on (mason, 1995b). tutors might have to make additional purchases to achieve the required computer specification, and spend more on telephone costs; both of these will have an impact not only on the domestic budget, but also on the use of shared resources. tutors are therefore likely to welcome methods of reducing the online costs, both in terms of finance and time. there are at least two changes to current practice that could achieve such a cost reduction. firstly, conferencing could be used as a many-to-many delivery system in addition to the more frequently used one-to-many system. secondly, students could be encouraged to take ownership of the conferencing. both these changes, however, would require tutor training in conferencing, and would also assume a change in the nature of tutoring such that it would become a blend of course delivery and course support not previously undertaken by ou tutors. one approach, for example, may be for course teams to provide tutors with structured materials guiding tutor conferences which include questions to pose to students at intervals throughout the course. such training would respond to tutors' expressed needs, and would go some way to solving the electronic communication paradox, and also ensuring that the opening up of possibilities for some is not done entirely at the risk of disadvantaging others. in conclusion as the ou moves towards greater use of information and communication technologies for teaching and student support, the role and the needs of the tac staff (most of whom work only part-time for the ou) must be re-assessed. the issue is not simply one of ensuring that tac staff have access to appropriate equipment without particular groups being disadvantaged or undue additional expense being incurred by all. the university must enable tac staff to make effective use of technologies appropriate to their teaching and student support activities. in time, icts will not simply replace existing means of delivery and communication: they will make possible different forms of educational interaction. in the case of certain courses or programmes of study, this might involve the generation of new models of distance education, in which current distinctions between established patterns of course production, presentation and student support become untenable. references albury, d. (1995), 'transformation or expansion: icts and the future of higher education', 38 aa-j volume 4 number i paper presented at the pict international conference on the social and economic implications of information and communication technologies, london, 10-12 may. kirkup, g. and jones, a. (1996) 'new technologies for open learning: the superhighway to the learning society' in raggatt, p., edwards, r., and small, n. (eds), the learning society: challenges and trends, london, routledge. kirkup, g., jelfs, a. and heap, n. (1995), 'thd204 information technology and society: first survey of tutors', report no. 54, programme on learner use of media, institute of educational technology, the open university. kirkwood, a., jelfs, a. and jones, a. (1995), 'teaching and counselling staff and computing: overall findings of 1995 survey', report no. 64, programme on learner use of media, institute of educational technology, the open university. jones, a., kirkup, g. and kirkwood, a. (1992), personal computers for distance education, london, paul chapman. mason, r. (1995a), 'the mint project in the context of future directions for computer conferencing in the open university', report no. 207, centre for information technology in education, institute of educational technology, the open university. mason, r. (1995b) 'computer conferencing on a423: philosophical problems of equality', report no. 210, centre for information technology in education, institute of educational technology, the open university. 39 retraction with face saving: modelling conversational interaction through dynamic hypermedia anthony baldry*, marco piastra and roberto bolognesi** * university of pavia, italy ** cares (research institute), pavia, italy abstract this paper describes rwfs (retraction with face saving), a hypermedia application which models an interview between a lawyer and his client a lorry driver facing court charges of reckless driving. at one level rwfs takes the form of a sophisticated game in which different outcomes to the interview are possible according to the learner's degree of skill. at another level, rwfs is designed to encourage the language learner's awareness and understanding of the pragmatic features of conversation. rwfs runs on hypercontext, a hybrid hypertextlexpert system developed in pavia by two of the authors, marco piastra and roberto bolognesi, and which supports dynamic hypermedia units. hypercontext's dynamic linking capacity plays a vital role in simulating significant conversational features such as the conditioning of a current move in the conversation by information acquired much earlier in the course of the interview. in this connection, the paper discusses the contribution of rmci (re-usable model of conversational interaction), a re-usable application-independent applied model of interaction on which the game is based, and which links a tactical level (the conversation) to a metalevel which provides a moveby-move commentary on interactional theory. in its turn, rmcfs metalevel is linked to a strategic level which interprets the structure of the conversation in terms of a pyramid-like hierarchy of increasingly abstract theoretical concepts. introduction assume a lawyer is interviewing a client. for the participants in the interview, the interaction will probably be seen in its simplest form, namely as achieving certain goals and overcoming obstacles that would otherwise prevent these goals from being achieved. for an outside observer, examining the conversation will be a more complex process that takes into account the interactants' behaviour, their social status, and knowledge and perception of each other, as well as their perception of what it is possible and appropriate to say and do at any point in the conversation. although many excellent theoretical models exist that interpret these characteristics (see, for instance, sinclair and coulthard, 1975; coulthard, 1977; de 27 anthony baldry, marco piastra and roberto bolognesi retraction with face saving beaugrande and dressier, 1981), an application-independent applied model of interaction that can be used in a variety of computer applications to encourage a student's awareness and understanding of the pragmatic features of conversational encounters still needs to be developed. since its educational success depends on its potential to interpret the work being done in a conversational encounter, such an application must contain a metalevel that systematically comments on the tactical level, i.e. what is actually happening in the conversation. in its turn, the tactical level must be related to a higher level, the strategic level which characterizes goals, constraints, identities and other ingredients in the encounter in terms of specific theories of interaction. the resulting application will thus provide a commentary that interprets the tactics that the participants adopt in pursuit of their conversational goals, but which in its turn is linked to a more theoretical account of how the participants' tactics are achieving these goals and coping with the social and behavioural conditions placed on their attainment. rwfs (retraction with face saving) is designed to fulfil these objectives. it simulates a lawyer-client consultation in which the goal of the lawyer (lw) is to get his client, a lorry driver (ld), to rectify a misleading statement made to the police following a road accident. the description given in this paper relates to the tactics needed to complete the first stage in the game, namely the lw gets the ld to agree to retract his statement without humiliating him in the process. as the introductory case notes (figures 1 and 2) show, rwfs implements the strategic level in terms of intentionality and acceptability (de beaugrande and dressier, 1981; searle, 1983; searle et al, 1992). the simulation takes the form of an adventure game with rules that must be respected. in order successfully to negotiate the individual stages in the lws conversational plan, the student must respect the conditions placed on the lw in the conversation by the ld's personality, by the social conventions that operate in a lawyer-client consultation, and more generally by the structure of conversation itself. the student is required to follow a tactical progression that keeps the ld's willingness to participate in the dialogue alive. the tactical progression consists of a series of moves made by the lw (a multiple-choice option selected by the student). the lw has to adopt tactics that fall within the range of (psychological) acceptability to the client, since each move causes the ld's degree of 'one-up/one-down' domination (watzlawick et al, 1967), co-operation and trust (grice, 1975) to be modified. as figure 3 indicates, there is no single winning path through the materials. in order to proceed to the second stage in the game, a student must achieve the interactional goal (the ld's admit), respecting social conventions and norms, in particular the need to preserve the ld's face. visual and textual clues relating to the ld's reactions (and hence to the student's skill in handling the interview) are given as the game proceeds. with the exception of three 'fatal' moves, marked exit in figure 3, no single move will cause the ld to seek another lawyer, and provided the tactics fall within a range of acceptability to the ld, the first stage in the game can be taken to a successful conclusion. there are many possible outcomes, each associated with different conversational and psychological states. some of these outcomes mean that the student wins the game; others that he or she has failed. five basic problems face the researchers wishing to model a simulated interview which, though implemented as an adventure game, still remains sophisticated enough to act as an introduction to conversational theory for students of linguistics: (i) scripting the simulated interview in such a way as to provide for convincing alternative conversational states during the proceedings, as well as a convincing outcome; 28 alt j volume 2 number 2 drawing up a conversational strategy you are a lawyer (lw). a few days ago you were contacted by a lorry driver (ld) involved in a road accident. while you are waiting for him to arrive, you decide your conversational strategy and tactics. you suspect that your client will lose his licence, at the very least. you also realize that the statement he has made to the police is inaccurate and must be rectified. by all accounts, he is clearly an irascible person so you must tread carefully. you must do four things in this interview: (1) make sure your client tells you the truth about the accident; (2) get your client to withdraw from his current position; (3) tell your client the truth about his licence, if he asks you; (4) show the client various documents at the right time and in an appropriate order: these include police reports, medical examinations, newspaper reports, statements and documents relating to previous cases. dividing up the conversation into sub-goals and sub-sub-goals in order to proceed to the second sub-goal you must successfully complete the first subgoal: admit. this requires the ld to recognize that certain passages in his statement are not in his favour. this is divided into four sub-sub-goals. each establishes a desired state whose achievement requires you to adopt specific tactics. your plan is to establish: (1) a starting point in the discussion: by selecting which documents to leave in full view on the table, you decide to reveal how much you know about the case; by setting the declared (as opposed to real) shared information, you also implicitly establish your subsequent tactics by choosing one of four initial conditions: hide newspaper; hide statement; hide both; hide nothing. your choice of initial condition will influence the dialogue, but you should attempt to manipulate these conditions to your advantage as the game goes on. (2) recognition of stress by the ld: the ld's actions (and in particular his statement) were the result of the stressful circumstances at the time; you use the newspaper article and/or the information it contains to achieve this recognition; decide your tactics carefully. (3) an acceptable pre-retraction state: namely the idea that because the statement was given under stressful circumstances, it contains errors and ambiguities; you get the ld to agree that a misleading statement is understandable but regrettable; you use his statement but what are your tactics in using it? (4) an admission that retraction is necessary: having established that (a) the statement was given under stressful circumstances almost under duress and that (b) there are inaccurate and misleading details in the statement, your tactics now require you to get the ld to agree to retract; you decide to play your ace, showing the ld the confidential report and/or the information it contains in an appropriate way how in practice will you play this ace? the phone rings and your receptionist tells you that your client is on his way to your office figure 1: case notes: the lawyer's conversational plan 29 anthony baldry, marco piastra and roberto bolognesi retraction with face saving coping with the lorry driver's conversational plan it will be hard to stop the client from trying to impose his own conversational plan on you. he wants to find out whether he will lose his licence. he is not an easy client, and there will be times when he will be dominating, uncooperative and highly suspicious the very opposite of what you want. the trick is to make sure these attitudes are kept within an appropriate range so that you keep the conversation going, steering it towards your own conversational goals. you cannot simply ignore your client or simply lay down the law with him; any conversation is a process of negotiation. in this chess-like game, you need to work out tactics that take into account the possible responses that your client will make both immediately and at a later stage in the conversation as the result of your current move. anticipating the lorry driver's probable reactions good and bad moves are defined in terms of their acceptability or otherwise to your client. the range of acceptability is defined numerically in terms of three parameters: dominance, co-operation, trust. these function independently of each other, and each move you make will cause the current state of each parameter to go up or down or stay the same. ideally, you want dominance to decrease, co-operation and trust to increase. the feedback from the client his reaction to your move will, of course, influence and become part of the subsequent conversation. use your common sense and knowledge of the previous, current and probable subsequent states in the conversation to keep the conversation going. figure 2: case notes: the lorry driver's conversational plan and likely reactions (ii) getting the computer to simulate the behaviour of one of the parties in a two-party conversation, and that party's reactions to the conversational tactics and strategies adopted by the other party, played in a chess-like game, by the student; (iii) capturing the dynamic nature of conversation, a process in which the decisions made by the parties place constraints on their subsequent conversation; (iv) defining mechanisms by which to associate an amusing game to rather more serious theories of conversational interaction; (v) creating a re-usable model of conversational interaction that can be used as a basis for subsequent related applications. because of the complexity of these issues, and restrictions on space, most of what follows relates to the tactical level, and only a brief reference is made to the metalevel and the strategic level. rwfs and the definition of the tactical level scripting and the problem of modelling dialogue pathways in hypertext as mentioned above, the first problem relates to the scripting of an interview in such a way as to provide for convincing alternative conversational states during the proceedings, as well as a convincing outcome. figure 3 gives an overview of the tactical level (limited to the first subgoal) which shows that the set of possible hypertext pathways coincides with an equivalent set of dialogue pathways: the user impersonates one of the two interactants and at each step 30 altj volume 2 number 2 makes a choice as to the next move to make along the dialogue pathway. the nodes are divided into four sections corresponding to each of the four sub-sub-goals (hence nodes lw 1.1,2.1,3.1,4.1 etc.). simulating the other party's behaviour rwfs relies on a working memory, a means by which the history of the moves along the dialogue pathway and the current state of the exchange between the two interactants are recorded. as indicated in the second paragraph of figure 2, simulating the other party's behaviour is achieved in terms of three parameters co-operation, dominance and trust that change the state of the working memory in such a way that a visual and textual response can be given to the student. these parameters are expressed as conditions associated with each step taken along the dialogue pathway. in figure 3 they are marked respectively as c, d, t; the symbols + and represent changes in their values associated with a given choice of move in the dialogue. in other words, the ld's linguistic-pragmatic interpretation of the lw's moves is associated to each step in the dialogue. but this association is carried out dynamically. in order to understand the concept of dynamic management of dialogue, a key concept in rwfs is the following theorem: a conversational state = the current hypertext node + the state of the working memory when an application is authored in hypercontext, rules are associated by the author to every possible hypertext move. these rules provide for the updating of the working memory. the history of the student's passage along a dialogue pathway recorded in the working memory is in reality a synthesis of the major events which may affect the entire history (previous, current and subsequent) of the dialogue. the working memory update is not directly perceived by the user, but its effects are constantly present. the modelling of the conversational and psychological state of the other party in rwfs, i.e. the ld, is managed through the working memory. hence the presentation of the ld with an angry or happy face depends entirely on the state of the working memory. similarly, as figure 3 shows, the initial conditions (hide newspaper, etc.) are recorded in the working memory (+hn or +hs, etc.). a question mark indicates a consistency check. in other words, according to the way the game is played, the state of the working memory will be checked against the rules associated with a particular node. conversational exchanges and moves the best-known theoretical model of conversational interaction is the rank-and-scale model associated with coulthard, sinclair and their colleagues (sinclair and coulthard, 1975; coulthard, 1977, 1981, 1992) which provides for five levels of analysis, and in which each higher level consists uniquely of elements taken from the level immediately below. thus an interactional encounter contains one or more transactions, a transaction one or more exchanges, an exchange one or more moves, a move one or more acts. the tactical level of rwfs is systematically and exclusively defined in terms of exchanges and the moves they consist of. a hypertext node in the tactical level corresponds to a move in the rank-and-scale model. as figure 3 shows, the symbol • represents an lw node, and hence an lw move, while the symbol q, represents an ld node, and hence an ld move. all the exchanges in the current version of rwfs are made up of an lw move followed by an ld move. 31 anthony baldry, marco piastra and roberto bolognesi retraction with face saving the concept of different conversational states despite apparently identical circumstances like other hypertext units, the nodes in rwfs contain text and images. these are static nodes: the texts, the images and their composition do not change as the student progresses along a dialogue pathway. the nodes in the tactical level are not accessible in a completely free way: the sequences linking nodes at the tactical level (see figure 3) are the only sequences which form a coherent dialogue pathway. the coherence of the pathways at the tactical level is guaranteed by the arrangement of the hypertext links and by the consistency rules affecting these links. the consistency rules are established by the author (in the same way as specified above under simulating the other party's behaviour, namely by associating rules to every possible hypertext move). in most nodes, the user can choose between different moves. the consistency rules make the links dynamic, thus governing access between the nodes. to simplify the work of the author, different dialogue pathways may share one or more hypertext nodes. this means that there is no need to author each dialogue pathway separately. as mentioned above, the way in which a node is accessed determines the state of the conversation in that node and the possible continuation of the conversation subsequently. thus the user may find himself or herself in apparently similar circumstances (i.e. the presentation of the same text and images) and yet be faced with conversational states that are entirely different. take, for example, the node marked lw 2.1 (show the article). this may be accessed from: (a) node ld 1.1.2(1); (b) node ld 3.1.1(1) (c) node ld 3.1.1 (2). in contrast to case (a), access from (b) or (c) entails a reversal of the order in which the subsub-goals are achieved, with sub-sub-goal 3 preceding sub-sub-goal 2. in other words, accessing the same node through two different pathways means having two different states in the working memory. dynamic interpretation the previous section describes the tactical level. this provides the game component. but we need something that associates an amusing game to rather more serious theories of conversational interaction and that views a conversation as a dynamic, on-going process in which the decisions made by the parties will place constraints on their subsequent conversation. how does a researcher keen to create an applied model of interaction go about this particular task? the need for a connection between the game level and the theoretical models clearly there is a need to link up the tactical level to the strategic level in a context-dependent way in order to establish what specific aspects of conversational theory are being illustrated at a given point in a conversation. but it is also necessary to organize the strategic level in a noncontext-dependent way such that the student can browse through the materials and gain a general overview of the conversational theory being presented. how can these conflicting principles be resolved? the strategic level itself is organized as a pyramid in which the characterization of, for example, sub-sub-goals is the lowest level, linked to a higher set of subgoals, in turn linked to a still higher level of goals, and so on until the top of the pyramid is 32 altj volume 2 number 2 jsji 1 o -ar o « 3 o *| iii iiph | go|i = !t l, gcblo § i | i. i i°is o 3 -ts o • _ t i ? sigoi goi ii • " j g i 3 figure 3: possible pathways: a map of the four sub-sub-goals 33 anthony baldry, marco piastra and roberto bolognesi retraction with face saving reached. students can browse through the pyramid at will. however, access to the strategic level can be gained only from the tactical level and this link is made to specific points in the strategic level, the point of access changing as the game proceeds. access to a specific strategic level node is achieved by clicking on a highlighted keyword in the metalevel commentary associated to a particular move. it may well be that a different choice made at the same node will require access to a different point in the strategic level. this is one reason why the link between strategic and tactical levels is made through a separate level: the metalevel. providing a dynamic interpretation environment the link between a move and a strategic level node is achieved by associating interpretive hypertext nodes (the metalevel commentary) to the dialogue pathway nodes (the tactical level). this association also has to be dynamic to work properly. access to the interpretive nodes depends on the state of the conversation, i.e. on the node reached in the dialogue pathway and the state of the working memory. the interpretive nodes explain, in a context-dependent way, the theoretical significance of a conversational state vis-a-vis a possible model of dialogue interpretation. because of the complexity of authoring such a representation of interaction, where a change at one level has consequences at the other levels, it was decided in the early (prototype) design stages to consider a single feature as primary at all three levels, from a theoretical standpoint. the choice fell on the exchange, as this ensures the necessary focus on the relationships between the participants in a conversational encounter. thus, although the tactical level is organized in terms of moves and exchanges, the links with other levels are achieved in terms of exchanges. other interactional features, such as the individual moves contemplated by each participant, their speech acts, their decisions as regards turn-taking and next-speaker selection, changes in mood and body language, are considered essential but subordinate. with its focus on the notion of exchange, the proposed applied model of interaction takes changes in the state of the shared information, the mutual recognition of the roles and identities of participants and their mutual perception of other participants' moods into account. more significantly, the model captures the constraints imposed on subsequent exchanges by the previous history of the conversation. connecting the tactical level to the strategic level through the metalevel in just the same way as the dialogue pathway nodes collectively make up the tactical level, so the interpretive nodes collectively make up the metalevel. the strategic level consists of hierarchically organized hypertext nodes in which the author describes and interprets the sequences in a dialogue pathway according to the chosen theoretical model. the strategic level, moreover, has the goal of justifying the alternative sequences of moves and exchanges in the dialogue pathways and illustrating the restrictions on the possible continuation introduced by the choices made. the interpretive nodes (i.e. the metalevel) serve to correlate each accessible conversational state with the theoretical explanations contained in the strategic level. model re-usability even authors may well get lost in hyperspace the construction of a hypermedia unit like rwfs can require the author to create hundreds, possibly thousands, of different nodes. the organization into three distinct levels helps the author to dominate the complexity of the application. within certain limits the three levels can 34 alt j volume 2 number 2 be subdivided and managed separately. each level can be further organized (e.g. a flow chart containing possible dialogue pathways for the tactical level, a hierarchical pyramid structure for the strategic level). hence the conception of rwfs as being based on a more abstract application-independent applied model of interaction denominated rmci (re-usable model of conversational interaction). the rwfs paradigm can be re-used thanks to the rmci, the material contained in rwfs can be re-used in other units. the metalevel nodes act in such a way as to separate the tactical level and the strategic level, thus enhancing the possibility of re-use. the tactical level may be interpreted in terms of one or more theoretical interpretive models, and vice versa the strategic level can be linked to different tactical levels. thus, for example, although the current model of interaction is heavily geared towards an understanding of the concept of exchange, there is no reason why the tactical level could not be linked to an entirely different theoretical interpretation. indeed, the reversal in the order in which sub-goals are tackled (see above under conversational exchanges and moods) indicates rmci's capacity to manage whole chains of exchanges as a separate block. in other words, rmci already contains the wherewithal to handle conversational units higher than the exchange. it may well turn out to be the case that rmci can be used in conjunction with end-applications that require, for example, the incorporation of coulthard and sinclair's concept of transaction (sinclair and coulthard, 1975; coulthard, 1992). rwfs, rmci and hypercontext although both rwfs and rmci represent a simplified interpretation of interaction, many of their features cannot be captured with the current generation of commercially available hypertext tools. the design criteria of hypercontext, a hybrid hypertext/expert system, are such as to take into account the complex work of tracking and updating changes that occur at all three levels in the applied model of interaction. this is in itself a complex game of mutually conditioning and interlinked actions. since the description of rmci and hypercontext in this paper is necessarily reductive, readers are referred to the technical report on hypercontext (piastra and bolognesi, 1993). it is hoped to publish a fuller account of hypercontext's potential value in applied linguistics at a later date. this will indicate how important it is to develop re-usable applications that represent an intermediate level between an authoring system and an end-user application. conclusion the description given above is technical, but shows how dynamic hypermedia can present conversation in a holistic way and from many different viewpoints an essential requirement if conversational theory is to be tackled properly. in this respect, the goal of rwfs is to encourage students to acquire the following skills: • relating the surface structure of the conversation (the tactical level) to the underlying belief and behavioural systems (accounted for in the strategic level); • the constraints social and linguistic placed on the interactants; • the changing relationship between the interactants and the changes in the roles that they play; 35 anthony baldry, marco piastra and roberto bolognesi retraction with face saving • changes in the shared information as the conversation proceeds; • turn-taking and next-speaker selection; • other factors affecting the course of the conversation such as the need to preserve face. we may view these skills as relating, at least in part, to a student's metapragmatic competence taken here as being a student's ability to describe what is going on in a conversation and relating it to theoretical models of interaction. such skills can be secured only with the develppment of specialized techniques and frameworks in which they can be practised. a dynamic hypermedia model of at least some aspects of conversation seems to be effective. in pursuing the goal of creating suitable computer-based techniques and frameworks, research is needed in specialized melds, like the one proposed here, which entail the modelling of the interplay of many linguistic phenomena from many different standpoints. this project thus represents a specialist approach to multimedia in which the relationship of data to a critical and theoretical interpretive framework is an overriding consideration. but will it help students? how do students react to the view of pragmatics as presented in rwfs? will they find it as dull as ditchwater or will it help them to pin down the eel-like, multifaceted interplay of factors in conversation which are so hard to cope with? it must be stressed that rwfs is an alpha-prototype application, based on an alpha-prototype applied linguistics model of interaction which in turn runs on an alpha-prototype system whose educational validity is in the alpha-stage of evaluation. the application has all the weaknesses of a prototype, most seriously that it does not use authentic interactional data. it is also a heavily simplified model of face-to-face interaction, which takes the form of an interview made up of a series of exchanges, each consisting rigidly of a client move followed by a lawyer move. the work of experimentation and validation of the application may well indicate a need for changes to be made. but, in focusing on pragmatic principles rwfs does make students more aware of the nature of interviews as a social activity; it does relate interviews to other types of conversational interaction; it does focus on the asymmetries that occur in interviews, and it does highlight the consequences when a particular exchange sequence is adopted. it is a start (but only a start) to the complex work of realizing the potential of computers to relate the study of language more fully to social behaviour and cross-cultural differences. acknowledgements the authors wish to thank lavinia merlini barbaresi, university of pisa, and carol taylor torsello and franco crevatin, university of trieste, for their help during the development of rwfs. part of the applied research and experimentation with rwfs is being undertaken within, the apprendimento linguistico in ambiente ipermediale project, murst 40%. references coulthard, r.m. (1977), an introduction to discourse analysis, london, longman. coulthard, r.m. (1981), studies in discourse analysis, london, routledge. coulthard, r.m. (1992), advances in discourse analysis, london, routledge. de beaugrande, r. and dressier, w. (1981) introduction to text linguistics, london, longman. grice, p. (1975) 'logic and conversation' in cole, p. and morgan, j. (eds) syntax and semantics iii: speech acts, new york, usa, academic press, pp. 41-58. 36 altj volume 2 number 2 piastre, m. and bolognesi, r. (1993), a brief description of hypercontext, technical report 001-93, cares, pavia, italy. searle, j.r. (1983), intentionality. an essay in the philosophy of mind, cambridge university press. searle, j.r. et al (1992), (on) searle on conversation, philadelphia, usa, john benjamin. sinclair, j.m. and coulthard, r.m. (1975), towards an analysis of discourse, oxford university press. watzlawick, p., helmick, b.j. and ackson, d.d. (1967), pragmatics of human communication. a study of interactional patterns, pathologies and paradoxes, new york, usa, norton. 37 reviews edited by philip barker authoring-systems software for computerbased training, edited by william d. wilheim, educational technology publications, englewood cliffs, new jersey, usa, isbn: 087778-274-1, 1994. traditionally, the development of computerbased instructional materials has relied on the use of programming languages and computer programmers. this has often made development prohibitively difficult, time-consuming and expensive. it has also meant that it has been difficult for enthusiastic subject experts to develop their own teaching materials. in order to address these problems a wide variety of authoring systems have been produced which allow the rapid development of instructional materials by reducing the amount of low-level coding necessary for their implementation. a problem which arises from this proliferation, however, is that subject experts find it hard to identify the best tool for a particular development task. this book has been written essentially with the aim of providing potential educational software authors with a brief background to instructional design, an overview of some currently available authoring systems, and a means by which an informed choice of authoring system can be made. the preface (outlining the content and objectives, of the book) is followed by the main body which is divided into nine chapters. the first chapter outlines an approach to instructional design. the method follows a traditional software life-cycle model, though detail has been included to apply the model specifically to the authoring of computer-based learning materials which may employ multimedia techniques. chapters 2 through 8 each address a particular authoring environment. the aim here is to provide an overview of some of the more popular authoring environments namely hypercard, toolbook, linkway, spinnaker plus, authorware professional, quest and tencore. although each chapter is written by a different author, the topics covered within each are relatively consistent. notably, each chapter provides specific examples of the authoring process, and some short illustrative code segments are provided where appropriate. in addition, screen-shots of the types of product which may be developed using the various systems are presented. this allows potential authors to visualize the style of their future product even before a prototype has been created. the final chapter looks at the process of concurrent authoring. this is the idea that an authoring tool can be made which will allow the seamless integration of training materials within any software application. in this way, training facilities can be integrated into existing computer applications so that users can access real-time on-the-job training. such training can 126 alt-j volume 3 number i be used to assist in the completion of tasks currently being undertaken, and can also be used within a pure training context. these justin-time training techniques have become known as electronic performance support systems and there is a growing need for tools to facilitate their production. two useful appendices are provided at the end of the book. the first of" these provides a checklist to assist in choosing an appropriate authoring system for a particular task, and although rather simple, the checklist does detail some of the important features to look out for when choosing an authoring environment. the second appendix provides a list of authoring systems and addresses from which to obtain further information, or the software itself. although useful for prospective authors of computer-based learning materials, books of this kind tend to date rapidly, the result of constant upgrades being made to software of all kinds. as a result, the authoring environment descriptions which make up the bulk of the book will rapidly cease to describe accurately the current features and facilities offered by the systems treated. it tends to be the advanced features, which may well have a crucial impact on the choice of authoring environment, that become outdated. indeed, this has already happened in the case of both hypercard and toolbook. nevertheless, the basic strategies adopted by such systems remain the same, and therefore value is retained. stephen richards, university of teesside designing instructional text by james hartley, third edition, kogan page, london, isbn 07494-1037-x, 1994. this book is intended to give general guidelines for the production of instructional materials. three main areas are covered: the design and layout of instructional materials, the language in which they are written, and how to evaluate the finished products. the first five chapters cover the traditional typographic conventions which evolved before the advent of the desktop-publishing era. factors affecting the choice of page size, type size and typeface are discussed in detail; effective use of white space is also described. following the basic rules outlined in these chapters would enable readers to produce materials free of the amateurish horrors which have become, sadly, all too familiar. one useful addition to this part of the book might have been a discussion of the advantages and disadvantages of different binding methods. chapter 6 goes on to discuss the production of instructional text in terms of clarity and organization. where possible, recommendations and conclusions are drawn from actual research studies. hartley is careful to stress the difference between improving the appearance of text, and improving the degree of reader comprehension. the principles outlined are then illustrated by means of worked examples in chapter 7, while chapters 8 and 9 consider ways of enhancing instructional text by means of diagrams, illustrations, tables and graphs. displaying numerical data by graphical means frequently results in readers being misled; this may be intentional or unintentional on the part of those presenting the information. although this book provides guidelines to improve the legibility of graphs, there is, unfortunately, little discussion of the relationship between numerical significance and appearance. this is an important omission, now that computer charting programs enable users to present the same set of data in a variety of ways at the touch of a button. chapters 10 to 13 cover specific topics: the design of forms and questionnaires; text for the visually impaired; text for older readers, and electronic text (i.e. screen design). the chapters focusing on the visually impaired and the older reader are particularly useful, as this material is often omitted from mainstream textbooks. chapter 14 discusses the evaluation of instructional text in terms of content, presentation and teaching effectiveness. some examples of checklists for assessing the layout and technical quality of school text books are provided, along with measures to gauge the suitability of a text for its intended readers (readability indices and cloze tests, for example). hartley rightly emphasizes the limitations of such measures, yet illustrates their usefulness in the decisionmaking process by means of case studies. overall, the book presents a great deal of useful 127 reviews information and illustrative examples, and there are ample references to primary source material for those requiring more in-depth knowledge. the clarity of the presentation would be improved if the separation of figures from their corresponding reference in the text could be avoided, and the inclusion of section numbers would also be helpful. yet despite these minor reservations, i would have no hesitation in recommending this book as a useful resource for anyone involved in the design of instructional materials. lorraine warren, university of humberside preparing materials for open and distance learning: an action guide for teachers and trainers by derek rowntree, kogan page, london (in association with the institute of educational technology, open university), isbn: 0-7494-1159-7, 1994. the author of this guide needs no introduction. he has contributed extensively to the field of educational technology, curriculum development, and open and distance learning. the delivery of learning has changed over the last 25 years with the development of open learning, flexible delivery courses, and the widening of student access to courses and institutions. this change has resulted in the need for materials to be available in a more diverse and flexible form in order to be sympathetic to different learner characteristics and to promote deeper learning. this guide is aimed at those who are about to design and produce largely print-based materials and/or adapt existing materials for particular students. it is not, in my view, to be read from cover to cover, but to be used as a structured guide for the preparation, production and adaptation of materials, and for the support of learners using such materials. there are many useful checklists that can be adapted for existing courses whether face-to-face or at a distance. the guide is divided into four main sections: an introduction plus sections (called stages) on planning materials, preparing for writing, and writing and re-writing. the introductory section lists the types of materials that could be used from print-based materials to technology-based materials, and includes a comprehensive section of sample pages from existing flexible learning courses. stage 1 deals with planning, and contains some useful suggestions and tactics for finding out about intended recipients, because the belief is that this information will directly dictate presentation and design strategies. the question of framing objectives is considered briefly but concisely, together with how to derive content from these objectives. and advice is given on making the material more learner-centred, on choosing media, and on supporting students, with some useful checklists on where this support might come from. in stage 2, there are suggestions and checklists on time management, on scheduling work, and on determining what resources might be available to enable you to develop your own materials. sequencing of learning is considered, together with suggestions on developing activities for students, and the types of feedback you might use. there are also suggestions on the physical layout of printed text with the aim of allowing students to access information and ideas .quickly (rowntree refers to making the prose 'clearer' and more 'cheering' to the reader). there is stress for the need to include meaningful examples in the text, with a good list of possible sources, although it is somewhat surprising that assessment is not really considered here (or indeed in stage 1), and is dealt with in the last section on writing and rewriting. inclusion here would have emphasized the need to consider assessment very early in the development of flexible learning materials (or any other learning materials for that matter). stage 3 contains some useful tips and advice on language style and how to get started, how to check for readability, and for completing and editing for first draft. and finally on piloting: write + try it out + improve it. although the section on further reading could have been more comprehensive, it is sufficient for the purposes of this guide, and there are references to the texts throughout at appropriate points, though i did find the use of some abbreviations rather irritating. 128 alt-j volume 3 number i i certainly think the guide is interesting, and indeed could be invaluable for those considering developing flexible learning materials. gareth jones, university of teesside teach yourself with open learning by derek rowntree, kogan page, london, isbn: 0-74941153-8, 1993 this is an extremely practical and usable book, again by rowntree. while generally aimed at readers who are new to open learning, it has much wider applicability, offering much to those already experienced or, indeed, to any educationist interested in preparing openlearning materials. interestingly, the book is itself written in the form of an open-learning text. each chapter is focused, with very clear, boldly titled sections with specific objectives. the nine chapters cover topics such as 'you and your learning', 'what open learning can offer you', 'how does open learning work', 'choosing your open learning programme', 'getting organized', 'how to tackle your package', 'getting help from other people', and 'getting value for money'. the style of each chapter is interactive, with simple activities where the reader is required to respond to check boxes or make written responses to questions. the objectives of the book are clearly stated at the beginning, and the reader is invited to identify appropriate objectives. this involves the reader immediately. however, although there is no requirement to study all the material in sequence, and dipping into chapters and sections is quite possible, this is a book that demands a certain degree of reflection and commitment. the style and layout are light and informative, and the book is easy and fun to read. for example, chapter 4 takes the reader into topics on course evaluation, what one might hope to gain from an open-learning course, how such material might be presented, the use of different media, and possible learning techniques. cleverly, the author does this partly by making explicit the very techniques used in writing the book itself, as well as including nine sample pages from a diverse range of open-learning courses. the book contains a list of useful addresses and a short bibliography, but it is in any case immensely informative without being a factbook as such. it raises issues and encourages valuable reflection and self-evaluation. it is also a manual for constant support and revision as well as a one-off read. chapter 5, for example, deals with specific providers of open-learning packages, and has an excellent section on talking to providers. it is suggested that the book itself with checklists and readers' notes might even be taken to an interview with a course provider. chapter 6 includes clear and useful directives on time planning, and introduces the concept of a learning diary to record both personal reflections and questions to be raised with a tutor or friends. chapter 7 deals with readers' real fears about coping with materials, and offers some very useful tips on subjects such as making materials on your own by annotation and talking about the course. it also has an excellent section on learning from a workbook through the stages of previewing, reading and reflecting. this book must be regarded as an excellent resource to be used not only as an introduction to open learning, but also throughout any course for reminders and practical support. it could certainly make the difference between success and failure on an open-learning course, and for this reason is highly recommended. terry king, university of portsmouth 129 needs before means: the dialectics of learning and technology anny king and andrew honeybone language centre, university of cambridge the general argument advanced in this paper is that in the changing context of present-day higher education it is vital that our educational purposes and student needs are clarified before decisions are taken about the means, including the use of learning technology, of satisfying those purposes and needs. the development of a critical understanding is still seen as the central purpose of higher education even in the context of a more vocationally relevant mass higher education. it is argued here that dialogue is the key to critical learning based on a process of dialectical communication. the task then is to construct an understanding learning environment which fosters interaction between students, staff and resources, reconciling individual needs with collective purposes. the specific role of learning technology as a means of encouraging dialogue within a learning environment is illustrated through examples of language learning such as tltp cks33 and the race hipernet project. through a dialectical process, the appropriate use of learning technologies in meeting students' changing needs can be progressively refined introduction higher education and society in general are changing rapidly. the student body is more diverse and larger. arguments rage about the vocational relevance of higher education. the unit of resource is declining. the way ahead appears fraught with difficulty. however, computer-based technology is advancing at breakneck speed. is it the 5th cavalry charging to the rescue of the beleaguered educational troops? the argument that will be advanced in this paper is that such a view is simplistic. the key issue is that pedagogic purposes and individual student needs should be first defined so as to identify the best means of meeting those purposes and needs. the general argument that dialogue based on dialectical communication lies at the heart of the learning process will then be applied to the use of learning technology in language learning. it will be demonstrated how, through a dialectical process, the appropriate use of learning technologies in meeting students' changing needs can be progressively refined. 4 alt-j volume 4 number 2 the changing context of higher education three aspects of change affecting higher education can be noted. first, the move to mass higher education (trow, 1987) has been associated with a greatly increased diversity of students. therefore there is an added requirement to clarify the educational needs of individual students in relation to their past learning and experience and their particular motivation for entering higher education. the means of meeting those needs may also differ from those traditionally used in higher education not only because the needs have changed but also because old methods may be inappropriate with larger student numbers (or too expensive with a declining unit of resource) even before consideration is given to the new means available through the development of learning technology. the second aspect of change influencing higher education is the continuing debate about the relative importance of vocational and academic purposes in higher education. participants in the debate have tended to adopt an either/or position, whereas the view adopted here, following such writers as neave (1992), nuttgens (1988) and reeves (1988), is that there need be no fundamental conflict between higher education for academic purposes and that with more vocational objectives. the reasons for this view will be elaborated shortly. the third element of change is the pace of change in modern society which means that, coupled with the continuing nature of that change, much of the content of present courses will be rapidly outdated (sadytzky and bereday, 1977). more specifically, the means of communication are rapidly changing with the growing centrality of the audio-visual culture of television and personal computers. the central purpose of higher education the effect of these changes on the main purpose of higher education depends very much on our broader conception of society. if our conception is of a genuinely democratic society related to popper's notion of an open society (magee, 1973), then an essential ingredient of such a society is a mass higher-education system that encourages its participants to develop their own understanding based on critical enquiry and independence of thought (honeybone, 1994). from an academic perspective, this has long been seen as the hallmark of the higher forms of education (barnett, 1990). from a vocational viewpoint, a similar conclusion can be reached, particularly in the previously mentioned context of rapid change: if the individual does not develop critical understanding, the value of higher education to the economy will be short-lived as the specific content of study becomes outdated and the individual is lacking the critical skill of thinking through how best to act in new situations. but some clarification of the word understanding is required. it is a word used with different meanings ranging from 'reproducing content' to 'developing an individual conception' (entwistle and entwistle, 1992). in the present context it is being used with the latter meaning, that is in the sense of 'conceptual understanding' which involves making connections and weaving 'bits of knowledge into an integral and cohesive whole' (nickerson, 1985). however, that definition introduces another word with diverse interpretations, namely knowledge. in this paper, the word is used in the wider sense 5 army king and andrew honeybone needs before means: the dialectics of learning and technology adopted by eraut (1992), encompassing propositional, process and personal knowledge. in this formulation, knowledge is something that is being actively worked on rather than passively received by the learner. the learning environment following the seminal work of marton and saljo (1976a and b), it is argued that conceptual understanding can best be achieved when students adopt a deep (learning as transforming) rather than a surface (learning as reproducing) approach to learning. the aim, then, is to develop a learning environment that will encourage students to adopt a deep approach and strive for conceptual understanding. such a learning environment will include the complete setting for learning provided by a department or institution, including the curriculum, the way the curriculum is taught, the way it is assessed, the material resources to support learning, including learning technology, and the type of interaction or dialogue between staff and students. this interaction, and indeed the interactions among students themselves and also with the material resources, can be seen as central to the development of understanding: deep learning leading to conceptual understanding through dialogue. however, such dialogues are mediated by a number of influences: the needs, learning styles (kolb, 1984) and experience of individual students, the learning and teaching styles and experience of staff, and the resources and technology (the means) available. needs should come before means, but with those individual needs being reconciled with the collective purpose of higher education. this argument will now be developed using language learning (within a communicative approach) as an example. we learn to comprehend the world, to understand it better. at every stage, we need to communicate our interpretation of our comprehension of the world (to start with, comprehension of our immediate environment, then of a more distant one) to ourselves and to others. in order to do so we use language. this paper, then, considers the particular role of learning technology in aiding dialogue in language learning. dialogue as a model of communication the word dialogue comes from the greek dialogos. it is in its first meaning a conversation between two people, encompassing the notion of critical enquiry (plato, 1965; russell, 1991) in which the learner engages in a process of questioning conventional knowledge. dialogue cannot be divorced from social interaction and, as face-to-face conversation, it can be seen as the basic model from which all other forms of human communication ultimately derive (good, 1995). in a dialogue, both the speaker and the listener are actively engaged collaboratively in a reciprocal activity (brown, 1986). (brown was writing in the context of language learning, but similar arguments have been advanced for the centrality of dialogue in the sphere of learning in general see gibbs, 1992.) both participants in the dialogue are, in turn, speaker and listener, and as such co-operate actively in constructing a coherent interpretation of each other's texts. together they construct the communicative message. 6 volume 4 number 2 such communication, i.e. the construction of a joint discourse, is based on interaction encompassing both receptive (particularly listening) and productive (particularly speaking) skills. such an interaction is today understood to be made up of many elements encompassing both the linguistic and social/cultural sides, and is viewed as central to language learning. it takes into account the knowledge and competence of many areas obviously linguistic competence (lexical, grammatical, semantic and phonological) and socio-linguistic competence (politeness conventions, registers), but also general competences such as knowledge of the world , socio-cultural knowledge (knowledge of society and culture of the target community), and inter-cultural awareness (awareness of the differences between the world of origin and the world of the target community). intentionality thus, as brown (1986) argues, dialogue is 'a risky business'. at every stage of the construction of the message, both participants make choices on what seems to be the most appropriate interpretation of each other's texts, and they construct appropriate mental representations. they do so in order to allow the common message to be woven as successfully as possible. here it might be appropriate to explain the notion of intentionality in a dialogic interaction. if we subscribe to the widespread belief that we learn languages to communicate, then it is legitimate to assume that intentionality plays an important role in successful communication, intentionality not just on the part of the speaker but also on the part of the listener (brown, 1986). just as the speaker speaks with intent, so the listener listens with intent (the way in which learning technologies may be able to accommodate the variations in intentions of individual learners will be considered later.) indeed, it could be argued that if the listener is not intent on playing his part fully in such an interaction and chooses not to co-operate, the result would be at best misunderstanding, and at worst breakdown in communication. some linguists believe that misunderstanding rather than understanding is common in communicative interactions (spolsky, 1995), and that its wide prevalence signals the difficulty of understanding. context if the dialogic process leading to understanding is fraught with difficulty, it is not an impossible task. there are parameters which can help to ensure a more successful communication. one such parameter is context context here is used in its wider interpretation encompassing the notions of background, mutual and shared knowledge (shank and burnstein, 1985). communication is seen as 'the reduction of uncertainty, where speakers fill in the gaps in each other's knowledge, negotiating meaning and modifying their own knowledge interpretations as the discourse progresses' (brown, 1989). in such approaches, context helps the listener to predict what the speaker is likely to say. the more the participants know about each other, the closer their social background, the more similar their use of language, and the more mutual the knowledge they share, the more successful the communication. in a familiar setting, with a familiar speaker, on a familiar topic, the listener tends to find that many of her or his predictions are correct. in those circumstances, meaning is negotiated and understanding is developing, thus a more successful discourse progresses. genres of language use of course, not every communicative interaction has the same level of'risk'. for example, 7 anny king and andrew honeybone needs before means: the dialectics of learning and technology provided you are a reasonable intermediate speaker of french (gcse-type level), it is unlikely that you would go into a bakery in france with the intention of buying 'deux croissants' and that you would come out carrying a dozen cakes. apart from the familiarity of the topic and the setting and predictability of the language, the genre of language needed is not conceptually very demanding. in the sphere of language learning, the different genres of language used at any particular point in a communicative task can lessen or heighten the difficulty of understanding, and thus whether the message is successfully elaborated or not. the ease or difficulty of understanding is linked to the amount of cognitive load imposed on the participants in an interaction. the less the cognitive load, the easier the understanding. the more demanding the cognitive load, the more difficult the understanding. in language learning terms, the simplicity or complexity of language needed in such interactions must always be taken into account. the language needed in transactions such as the one described above is conceptually less demanding than the language needed to engage in an argumentative discussion of sartre's assertion that tenfer, c'est les autres'. a brief explanation of what is meant by genres of language may serve to clarify the point there are broadly speaking four genres of language: procedural, transactional, narrative and argumentative. procedural language deals mainly with instructions (how to do something, how to get somewhere etc.), and it is barely necessary to understand the language in a linguistic mode, in the sense of being able to repeat it back or constructing a summary of what has been said (brown, 1995). transactional language deals mainly with everyday-life situations (buying something, asking for information, requesting a service), and it is highly predictable and familiar. the language needed is functional, and covers certain structures and forms of language. in spatial and temporal terms it is usually a here-and-now affair. narrative language is found in reports, historical documents, anecdotes, the planning of future events as well as novels or short stories. it involves the ability to understand content in a temporal sequence of events, in a given spatial domain (brown, 1995). understanding a narrative is more demanding conceptually since the listener must carry in her mind at all times a reconstructed interpretation of the sequence of events both spatially and temporally, and try and make sense of the causality and intentionality of relationships. finally there is the language of arguments, explanations, justifications, theorizing. compared with procedural or transactional languages (and to a certain extent narrative language), argumentative language has less connection to the real, physical world. it is more abstract and complex, and fewer concrete props are available to the listener engaged in such a dialogue. the cognitive load on the participants is even greater than in narratives. understanding a theory, an argument or a justification is mainly done through the medium of linguistic input. both participants in such a dialogue must infer a lot in their interpretations of each other's constructs and their relationships. 8 alt-j volume 4 number 2 the communicative approach in language learning in the past, the academic study of foreign languages was primarily to allow learners to read literature in the original text. as such, the learning of language was primarily the learning of the written form of language. and yet understanding the spoken language is one of the requirements for language learning in order to make it your own (what linguists call language acquisition). the move towards a wider view was advanced in the 1960s by the pioneering work of lado who distinguished four skills in language learning: listening, speaking, reading and writing (lado, 1964). in the 1970s, building on lado's ideas, the council of europe's threshold level made the case for a communicative approach to language teaching (council of europe, 1976). such an approach promotes language learning as a social activity; it has a functional view of language learning, and emphasizes the social roles of social interactants. similar ideas on the importance of social interaction have been applied to learning in general, for example in the advocacy of problem-based learning (margetson, 1994). a communicative approach is based on three fundamental principles. the first and central one is that of learners' needs; the second one is that of learner-centredness, and the final principle is the primacy of the functionality of language use over its form. communication with others can be through the medium of oral or written discourse, and since the development of new technologies, it can be both oral and written simultaneously. it is precisely this simultaneity of access to both forms of language that gives new technologies a vital place in the language-learning process. dialectical communication and new learning technology building on the original definition, dialogue can be interpreted more widely to include dialogue with self and dialogue with technology whatever the technology be it a text, video or hypertext. it is in this wider interpretation that dialogue can be seen as the key to learning (mayes, 1995), and one which can be transformed by technology (good, 1995). however, it would be simplistic to suggest that a more technologically-based dialogue would necessarily help to maintain, in the present move to mass higher education, an appropriate critical perspective. for that perspective to be maintained (and maybe enhanced), new means of dialogue must ensure a continuing interaction between the participants, leading to qualitative change, i.e. greater conceptual understanding. that, it is argued, can be aided by a particular form of dialogue based on hegel's notion of dialectical communication, that is 'the development of thought which tends towards a synthesis (union of contradictions) striving continually to resolve oppositions between each thesis (affirmation) and its antithesis (negation)' (larousse 1986). through such a dialectical interaction, students construct their own understanding and interpretation of knowledge. and that could be a key test of the appropriate use of learning technology in higher education: does it aid understanding and interpretation? non-reciprocal/reciprocal in answering this question, the distinction between reciprocal and non-reciprocal communication is helpful (widdowson, 1978). in an oral/written dialogue between two 9 ann/ king and andrew hone/bone needs before means: the dialectics of learning and technology people, both participants are overtly and actively engaged in a reciprocal activity (speaking to one another or writing letters to one another). however, in listening to the radio, watching tv or reading an article, while the listener/viewer/reader may still be actively engaged, the dialogue is a covert and internal one with the producer's/writer's message. it is a non-reciprocal activity. the latest developments in learning technology are overcoming the restrictions of nonreciprocal communication, and thus are better able to aid understanding and interpretation. new technologies are intrinsically non-linear, allow more choice and justin-time access to the learner, and with their never-ending supply of integrated resources can make learning in general and learning a language in particular a virtual reality. these five points of non-linearity, choice, just in time, integration and virtuality which multimedia brings to the learning process, can make language learning a richer, freer, more satisfying and ultimately qualitatively different learning experience. they will shortly be considered in more detail. linearity/non-linearity it is non-controversial to say that the written form of language is linear. on the one hand, the medium (print) demands it. on the other, the writer writes her books/articles with the reader in mind. in that sense, the writer takes every step necessary to produce a printed text with the following elements in mind: shared knowledge of the world and of the topic in discussion, of the reader's reactions, and of the forms and structures of the language. she intends the message to be understood: it is perhaps in this light that krashen's 'comprehensible input' should be primarily viewed (krashen, 1985). therefore, the writer organizes the arguments and presents them in a way that conventions dictate and that is both clear and logical, taking, so to speak, the reader gently by the hand. this is not to say that the reader does not stop here and there to dispute/counter-argue with the writer's message some readers do. indeed, this internal dialogue with the writer's message, this deconstruction and reconstruction of the message, is vital to deep learning (honeybone, 1994). in language-learning terms, this process is inherent to the learning of the foreign language since each person's construct-language must be, to some extent, an individual construct (brown et al, 1994), as indeed it can be argued for learning in general (sutherland, 1992). however, with the linear written text, the learner's attempts to construct his own message is made more complex by the fact that language is just one element of communication; context (i.e. mutual and shared knowledge, background knowledge, scene-setting) as explained above is also important. it is more difficult to grasp context in a printed text, since meaning is approached solely or mainly through the written forms of language; thus, meaning must be constructed before an overall context can be revealed. on the other hand, visual texts (i.e. on the whole, spoken language and moving images) usually transcend language and give to the learner direct access to context. they are intrinsically non-linear. the language of a visual text is more diverse than that of a written text. it is made up of words (linguistic elements as in the written text), images (giving context and settings) and technique (angle of shots, music, pauses, rhythm of sentences, visual effects, colour, rhythm of montage) (compte, 1993). this new technology, by increasing access to visual texts, can aid learning. 10 alt-] volume 4 number 2 choice and just-in-time concept within the same environment, the learner has access to a wide variety of texts video, audio and print. he can browse through them and choose what corresponds to his needs in order to construct his own interpretation. the 'just in time concept' of retrieval allows the learner greater choice in his learning. it could be argued that multimedia gives the learner a richer, more comprehensive internal dialogue when he looks in the software for answers to his questions. by browsing through the multimedia, incidental learning is potentially greater than if the learner were just reading a text or listening to a tape. however, there is the danger that the very richness and variety of the choice may lead to cognitive overload (rheaume, 1993). integration learning technology may help language learning further by bringing together within one environment different media print, audio and video thus making skills-integration easier and more realistic for the language learner. as previously noted, a language learner needs to be competent in all four skills (speaking, listening, reading, writing), although not all learners need all four skills; or, to be more precise, not all learners need to develop all the skills at the same level of proficiency (king, 1993). skills are rarely acquired in isolation. in most commonly encountered contexts, they are integrated indeed the speaking and listening skills are at the basis of dialogue. reading and speaking (or vice versa) and listening and writing (or vice versa) are acquired together since in almost all real-life situations they will be used together. different skills (needs) need different resources (means). analysing learners' needs leads to analysing available resources in order to find out which resources serve which skills best. virtu ality multimedia offers the learner not just images and sounds of the country of the target language but also opens many doors to make her feel already 'at home'. she can go to the louvre and glimpse the mona lisa at the press of a button. she can, via a forum, ask a student in a university of the target language to tell her how life is in this university prior to her taking a course in this same university. the examples are numerous. for all these reasons, developments in technology over the last 10 years have great potential to stimulate learning.. furthermore, such technology builds on our learners' extensive televisual culture in our world of images (porcher, 1976) and on their fastdeveloping computer literacy acquired before entering higher education. the dialectics of learning and technology once needs have been analysed, the question that arises is: which means for which needs? our thesis is that as language-learning needs change and technology develops, so does the dialectical relationship between needs and means, i.e. between learning and technology. the nature of this relationship is shown in figure 1. the thesis that earlier call (computer-assisted language learning) was good for the development of reading skills is countered by the antithesis that highlights the fact that it was not suitable for other skills. a synthesis is found in later call which allows the development of not just reading skills but also to some extent listening and writing skills (with writing skills only in a closed context but not developed as a creative, productive anny king and andrew honeybone needs before means: the dialectics of learning and technology leverage • hipemet across europe thesis antithesis ethnocentric system within cu conceptual synthesis hipemet (integrated open system) multimedia + dialogue (community of learners) thesis antithesis but restricted dialogue: with self and media narrative synthesis multimedia workstation. skills integration + resources inegratlon thesis antithesis 3ut stops short of skills integration. each resource accessed separately synthesis tltp cks33 -3 media + integrated call incorp, all 4 skills thesis procedural/ transitional antithesis but speaking missing. writing closed synthesis later call reading/writing(closed)/listening thesis call: interactive with text, instant feedback, improves learning antithesis but not suitable for other skills. speaking/listening/ writing figure i: representation of dialectical relationship between needs and means, between learning and technology skill). this in turn becomes a new proposition (or new thesis) which is challenged by a new antithesis that stresses the fact that in later call, writing is still closed and speaking as a skill is still missing. 12 alt-j volume 4 number 2 and so the dialectical nature of learning and technology develops (figure 1), leading to a new argument (synthesis) exemplified by tltp cks33, a multimedia project using the 'traditional' media of print, audio and video with contextualized and integrated call. cks33 allows for not just a wider integration of media but also for more developed writing skills. but this in turn leads to the counter argument (antithesis) that cks33 stops short of skills integration which a multimedia workstation (new synthesis) can now provide. this new thesis soon highlights the fact that a multimedia workstation restricts dialogic communication since dialogue takes place only with self and with the technology. this leads to the new proposition (synthesis) of hipemet (high performance network) which is an integrated open system using the infrastructure of the granta backbone in cambridge (fibre-optic network) and which aims to develop a community of learners. hipernet (a race project) puts dialogue between learners first by encouraging a taskbased approach to language learning through video-conferencing while allowing learners to enter into a dialogue with technology through the multimedia learning tools at their disposal (bbc french means business, a multimedia glossary). leverage (learn from video extensive real atm gigabit experience)1, building on hipernet, aims to overcome the ethnocentricity of hipernet (system operating solely within cambridge university) by taking it across europe. our thesis is further strengthened by taking into account the four genres of language learning as well as the four skills. earlier technology suited the less complex genres of language use, and it is only lately that learning technology has been able to play an important part in the learning of the more complex genres. procedural and transactional genres of language can be served by call, and particularly by later call, since this genre of language (as previously argued) is highly predictable. the narrative/discursive genre of language (to be understood at post-gcse level) can be served by a multimedia environment since context is paramount for narrative understanding and, as previously argued, the visual text gives the learner direct access to context. multimedia encourages the learner to follow his own train of thought or, to be more precise, allows the learner to jump from one thought to another, come back to the first one and then wander around the hypermedia led by his curiosity and sense of discovery. thus the technology is encouraging a more independent style of learning which motivates the learner and enriches the learning process. as he makes more and more decisions about how to fulfil his language-learning needs, the learner uses the technology which is the more appropriate to those needs. conceptual/argumentative language, as stated earlier, is above all abstract with fewer concrete props or links with the real, physical world. conceptual language is used when communicating arguments, explanations, justifications, theories. within this framework 'communication' can be seen as encompassing the notion of plato's critical enquiry in which the learner engages in a process of questioning conventional knowledge. in this case 'dialogue' could be seen as the highest form of learning, and, with the latest developments in technology, such a dialogue, mediated by technology, can be with others peers and teachers. until recently, dialogue using learning technology solely involved the learner communicating with a system (call, tltp cks33). now, using developments in data communications, such as broadband technology, the learner is able 13 anny king and andrew honeybone needs before means: the dialectics of learning and technology to communicate, both with the technology and through the technology, with a wider community of learners as is the case with hipernet and leverage. the learner is no longer restricted to the language of the teacher or the texts provided, but can herself look for texts (both written and visual) that are appropriate to her needs. furthermore, she can have access to peers of the target language, making learning the target language a real and fulfilling experience, asking real questions to get real answers, and not having the teacher ask her questions which the learner knows very well that the teacher already knows the answers to. conclusion • these examples of the use of learning technology in language learning demonstrate that once the appropriate use of technology has been initially identified and then put into practice, if successful, the learning technology will itself help to refine further both purpose and needs, leading in turn to further revisions in its use. a similar analysis could be undertaken for other areas of learning. but the centrality of dialectical communication in higher education goes beyond its use in clarifying the most appropriate pedagogic role for learning technology. it also epitomizes the approach which both staff (as expert learners) and students (as novice learners) should adopt towards their subjects. as montaigne, the sixteenth-century french philosopher, put it: c'est temoignage de erudite et d'indigestion que de regorger la viande comme on i'a avalee. l'estomac n a pas fait son operation s'iln'afait changer la/agon et la forme a ce qu'on lui avail donni a cuire. (it is a sign of a bad digestion to bring up food in the shape it was swallowed. the stomach has not done its work if it has not changed the nature and form of what it is given to eat.) and that, maybe, is how we should be evaluating our use of learning technology: is it enriching our students' diet and helping them to digest their studies and nourish their own understanding? note 1 leverage is a project within the acts programme (advanced communications technologies and services) made up of eight european partners aiming to encourage greater international co-operation by establishing inter-country links between northern and southern european users, i.e. staff and students from cambridge university, int (institut national des telecommunications) in paris, and universidad polytechnica de madrid. references barnett, r. (1990), the idea of higher education, buckingham, srhe and open university press. 14 ki-j volume 4 number 2 brown, g. et al (1984), teaching talk strategies for production and assessment, cambridge, c.u.p. brown, g. (1986), 'investigating listening comprehension', applied linguistics, 7 (3). brown, g. (1989), 'making sense: the interaction of linguistic expression and contextual information', applied linguistics, 10 (1). brown, g. (1995), 'modes of understanding' in brown, g. et al (eds), language and understanding, oxford, o.u.p. compte, c. (1993), la vidio en classe de langue, paris, hachette. entwistle, a. and entwistle, n. (1992), 'experiences of understanding in revising for degree examinations', learning and instruction, 2 (1), 1-22. eraut, m. (1992), 'developing the knowledge base: a process perspective on professional education' in barnett, r. (ed), learning to effect, buckingham, srhe and open university press. gibbs, g. (1992), improving the quality of student learning, bristol technical and educational services ltd. good, d. (1995), cognitive technology: the human interface, amsterdam, north holland (in press). honeybone, a. (1994), understanding environments: environments for learning in learning about the environment, unpublished ma dissertation in higher and professional education, institute of education, university of london. king, a. (1993), languages for work and life: a european survey on common needs in vocational oriented language learning, council of europe/university of cambridge. kolb, d.a. (1984), experiential learning. experience as the source of learning and development, englewood cliff nj, prentice-hall. krashen, s.d. (1985), the input hypothesis: issues and implications, london, longman. lado, r. (1964), language teaching: a scientific approach, new york, mcgraw-hill. magee, b. (1973), popper, london, fontana press. margetson, d. (1994), 'current educational reforms and the significance of problembased learning', studies in higher education, 19 (1), 5-19. marton, f. and saljo, r. (1976a), 'on qualitative differences in learning: i outcome and process', british journal of educational psychology, 46, 4-11. marton, f. and saljo, r. (1976b), 'on qualitative differences in learning: ii outcome as a function of the learner's conception of the task', british journal of educational psychology, 46, 115-27. mayes, j.t. (1995), learning technology and groundhog day: hypermedia at work, practice and theory in higher education, conference handbook, university of kent. montaigne, m. (1926), les essais, paris, conard. 15 anny king and andrew honeybone needs before means: the dialectics of learning and technology neave, g. (1 992), 'on instantly consumable knowledge and snake oil', european journal of education, 27 (2), 5-27. nickerson, r. (1985), 'understanding understanding', american journal of education, 93 (2), 201-39. nuttgens, p. (1988), what should we teach and how should we teach it? aims and purposes of higher education, aldershot, wildwood house. pinker, s. (1994), the language instinct, london, penguin. plato (1965), the republic, harmondsworth, penguin classics. porcher, l. (1976), introduction a une semiotique des images, paris, credif/didier. reeves, m. (1988), the crisis in higher education. competence, delight and the common good, milton keynes, open university press. rheaume, j. (1993), 'l'enseignement des hypermedias pedagogiques', third european congress on 'hypermedias et apprentissages', lille (no publisher), (24-25 march). russell, b. (1991), the history of western philosophy, london, routledge. sadytzky, n.o.o. and bereday, g.z.f. (1977), 'mass higher education' in knowles, a.s. (ed-in-chief), the international encyclopaedia of higher education, 2763-80. shank, r. and burstein, m. (1985), 'artificial intelligence: modelling memory for language understanding', in t. van dijk (ed.), handbook of discourse analysis, vol.1, california academic press. sutherland, p. (1992), cognitive development today. piaget and his critics, london, paul chapman. trow, m. (1987), 'academic standards and mass higher education', higher education quarterly, 41 (3), 268-92. widdowson, h.g. (1978), teaching language as communication, oxford, o.u.p. 16 communication skills: a new strategy for training shane a. gordon and paul garrud behavioural sciences section, department of psychiatry, university of nottingham, medical school a new five-year course in communication skills for medical students has been developed at nottingham medical school in response to recommendations from the general medical council and a large body of research. as one foundation for this course, a multimedia cal package was developed to aid acquisition of basic skills and associated knowledge. the cal course uses extensive digital video for illustration, and relies heavily on interaction for learning. evaluation of the cal course has used a variety of methods — pre/post tests, randomized trials, and cohort comparisons. given the investment needed to introduce learning-technology-based courses, initial evaluation has been encouraging and has been used to improve later versions of the material. why do we need a new strategy? in 1993 the general medical council (gmc) published tomorrow's doctors, a set of recommendations for medical education. much of this document was concerned with the training of communication skills and how this could be improved. this recommendation follows decades of evidence about the importance of communication from many widely respected medical teachers from every discipline: doctors can discharge (their) important tasks effectively only if they possess the relevant skills. unfortunately, many do not appear to acquire them during their professional training. (maguire, 1981) there appears to be a failure sometimes to notice what is really being said... the doctor avoids the acute discomfort of being aware of a problem in which he would rather not get involved. (norell, 1972). particular deficits in the skills of both medical students and qualified doctors were highlighted, often showing that communication with patients got worse as training progressed! it would appear that, as medical students move through their training, a certain degree of their innate ability to communicate with mothers of sick children seems to have been altered by their desire to obtain factual information. (heifer, 1970) 15 shane a gordon and paul ganvd communication skills: a new strategy for training this points to a failure of the traditional 'apprenticeship' model in teaching these particular skills. several studies have now been conducted which show that specific training methods can produce gains in particular skills (truax and carkhuff, 1967; ivey, 1971). research has identified certain basic tasks and approaches which lead to unproved communication. these include addressing the concerns of patients other than their immediate physical complaint, explaining terms and procedures in lay language, dealing with patients' anxiety, embarrassment and anger, and improving patients' recall and retention of information and advice (balint and norell, 1973; evans et al, 1993; ley et al, 1976; mackenzie, 1975; scott et al, 1991; uhlemann et al, 1982). attention to these details, especially the last, has also been positively correlated with patients' compliance with advice and medication regimes. tomorrow's doctors outlined 14 specific recommendations for improving doctors' communication with patients. we have based our new strategy around this core. also of increasing importance are diminishing staff-to-student ratios from 1:8.4 in 1970 to 1:11.3 in 1989 (aut, 1991) a trend which has continued. ever tighter budgets, within which higher education is forced to operate, also have a big impact. medical schools are also faced with the challenge of an ever more diverse student body to cater for: for example, greater numbers of european union and overseas students. it is hoped that the new strategy will prove both more efficient in achieving its goals and more flexible in its provision for different modes and styles of learning. the new strategy at nottingham a new five-year course in communication skills has been designed to replace traditional teaching. this course integrates taught classes, seminars, clinical experience and a cal package. this cal is intended to provide basic knowledge and experience of the use and improvement of communication skills. our medical students arrive at nottingham from a wide variety of cultural, social and educational backgrounds, and consequently there is considerable variation in their experience of communicating with different types of people and their level of skill. the cal package is planned to give them all enough knowledge and awareness of the subject so that they can begin to improve. the cal material incorporates a selection of instructional methods and styles such as mental practice, modelling, and drill and practice, providing immediate feedback on the student's actions. it provides a safe environment for students to learn and practise new skills, or improve existing ones. the achievement of effective communication skills is assessed using the objective structured clinical exam (osce) format. communication skills are assessed objectively against validated behavioural checklists as an integrated part of a series of clinical tasks the students are required to tackle. how are the new strategies incorporated into the cal package? the cal package consists of seven, 30+ minute units which are based on the gmc recommendations. these units are: • opening an interview; • eliciting patients' views and concerns; 16 alt-] volume 4 number i • dealing with negative emotions; • principles of communicating with patients whose first language is not english; • summarizing and clarifying information; • exposition of information and procedures; • closing an interview. this is not intended to be a comprehensive list, but these topics are relevant in the majority of medical interviews and are commonly done poorly. each topic is presented in a structured fashion, looking first at the importance of the topic to the patient and the doctor (see figure 1). it is then broken down into relatively straightforward tasks such as identifying verbal and non-verbal cues and suggested appropriate principles of dealing with the situation. the suggestions are not encyclopaedic, and the students are encouraged to analyse the material in ways which help them develop their own practices. ¥>:w::>:::::::w why is it important? presentation to patients or colleagues @ checking for correctness and completeness figure /: a screen from the unit on summarizing and _ . . . . clarifying information, • aid to memory illustrating the small amount of text used and the structure of the information in order to avoid information overload, the amount of text on the screen at any one time is kept to a minimum. important details are presented as bullet points which can be clicked to play a voice-over with more information (or displayed as text if the user wishes). each unit is kept relatively short and focused: students should finish each unit in about 30 minutes, well within their attention span. interaction and discovery learning (where the student alters variables and predicts results) are used to encourage analysis of situations and practice of appropriate courses of action an example is shown in figure 2. most importantly, large amounts of video footage have been incorporated to illustrate situations and provide models for the students see figure 3. these films were produced using actors and actresses, but the scenarios are based on our analysis of lengthy recordings of real consultations. 17 shane a gordon and paul ganvd communication skills: a new strategy for training for a history summary. for a management plan,. click on 4 phrases you would like to include in the transcript. click the "done" button when you have finished. synopsis of interview: mr. willis has pains in his chest and is worried that he might have cancer. he has had them for about four weeks. he has recently moved house. hfc is 56 years old. the pains are stabbing and sudden in onset. he is a farmer and is married with three children. he has never smoked and drinks about 3 pints at weekends. the pain does not get worse with exercise. your summary of mr. willis1 complaint: he has has pains in his chest. he has had the pains for about four weeks. reset what do 1 select for a summary? figure 2: a screen where the user is asked to carry out a task identifying key information for a summary how are we evaluating this? our first evaluation of this type of training was in 1993 when we compared two methods of teaching first-year medical students about non-verbal communication (garrud et al, 1993). the study sample was large (n = 151), and subjects were randomly allocated to either a traditional lecture or a cal package on the same material. the content of the cal was agreed by the lecturer to cover the same material. all the subjects were then given a surprise test on their knowledge of non-verbal communication. the test was set by the lecturer and the cal authors jointly. the package was used by 60 of the students allocated to it (80%), and the lecture was attended by 61 students (80%). median usage of the cal package was for 45 minutes (range 15-120 minutes) with over half using it only once. test scores ranged from 1.5 to 7.9 (maximum = 10) with a normal distribution. mean scores were: cal group 5.2; lecture group 4.4 a reliable superiority of cal over the lecture (p<0.01). there was also a strong correlation within the cal group between score and duration of use (pearson correlation coefficient = 0.45, p = 0.0003). overall academic achievement was compared for the two groups using end-of-year examination scores no significant difference was found (p = 0.32). 18 alt-] volume 4 number i figure 3: an illustration of the use of digitized video in the course _ ^ analysis of student evaluation questionnaires revealed that 95% of students found operating the program easy. the majority of cal users found that it enabled learning well, and was a good use of their time. during the development of the communication skills cal course, we produced a prototype unit and evaluated it using 11 volunteer students. performance was analysed using a structured, 11-point scheme (similar to those used in the end of year osces). we also videotaped students using the cal package, and used this data (in conjunction with data automatically recorded by the package) to analyse the ergonomics. although the sample was too small to show any significant results, the ergonomic data allowed us to modify the packages to avoid commonly encountered problems such as difficulty in closing inset boxes to return to the main screen. it is intended that a larger pre/post test study (n>20) will be carried out on the full course. the data from this will be compared with students' performance on other measures such as transcripts of real patient interviews which they have conducted for other purposes, and their results in the end-of-year examinations. by the end of the year, all students in the first year (n = 170) will have had the opportunity to use the cal course. we intend to analyse student performance in the end-of-year osces to examine the relationships between duration of use, number of uses and overall marks. results will also be compared to last year's cohort who were not exposed to the cal. this will provide some medium term ( 4 months) data. finally, by monitoring students' use and re-use of the cal course we will gain some insight into the lasting popularity of the material as a learning resource. 19 shone a gordon and paul ganvd communication skills a new strategy for training conclusion in developing a new strategy to cope with the changing nature of medical education, we have found that computers can offer many advantages over traditional methods of teaching and learning. the key benefits of cal for training communication skills are: • a safe environment to practise communication where mistakes are not made with real patients; • a student-centred resource, allowing individual students to work on the aspects relevant to them at their own pace; and • the use of multimedia exemplars, which the student analyses and interacts with. our evidence so far suggests that the cal resource helps students to improve their communication in two ways learning simple plans and procedures to tackle common communication situations; and developing confidence to practise their skills in clinical settings. there are also costs and limitations. this development used £10,000 in funding, and took two years from inception to delivery. the cal course has yet to prove its cost effectiveness over traditional methods (cf. gibbs, 1993). it is not intended that cal should replace all traditional teaching on the course. it is another weapon in the educator's armoury and a powerful one, but it needs careful integration into the overall structure of a course. overall, we feel that cal provides sufficient added value to our teaching that it justifies its use in this course. references aut (1991), newsletter, nottingham association of university teachers. balint, e. and norell, j.s. (1973), six minutes for the patient, london, tavistock publications. evans, d.r., hearn, m.t., uhlemann, m.r. and ivey, a.e. (1993), essential interviewing, california, brookes/cole publishing co. garrud, p., chapman, i.r., gordon, s.a. and herbert, m. (1993), 'evaluation of a computerassisted learning package on non-verbal communication', medical education, 27, 474-8. gibbs, g. (1993), 'how do we measure efficiency gains?', teaching and learning technology programme (tltp) forum. heifer, r.e. (1970), 'an objective comparison of the pediatric interviewing skills of freshman and senior medical students', pediatrics, 45, 623-7. ivey, a.e. (1971), microcounselling: innovations in interviewing training, springfield il, charles c. thomas. ley, p., whitworth, m.a., skilbeck, c.e., woodward, r., pinsent, r.j.f.h., pike, l.a, clarkson, m.e. and clark, p.b. (1976), 'improving doctor-patient communication in general practice', journal of the royal college of general practitioners, 26, 720-4. mackenzie, d.d. (1975), how to interview, british institute of management, halifax, edward mortimer ltd. 20 alt-j volume 4 number i maguire, p. (1981), 'doctor-patient skills' in argyle, m. (ed), social skills and health, london, methuen. norell, j.s. (1972), 'changing perspectives in general practice' in patient centred medicine, london, the balint society. scott, g.s., barrows, h.s., brock, d.m. and hunt, d.d. (1991), 'clinical behaviours and skills that faculty from 12 institutions judged were essential for medical students to acquire', academic medicine, 66, 106-11. truax, c.b. and carkhuff, r.r. (1967), toward effective counselling and psychotherapy: training and practice, chicago, aldine. uhlemann, m.r., stone, g.l., evans, d.r. and hearn, m. (1982), 'evaluation of microtraining modifications: implications for paraprofessional training within community counselling agencies', canadian counsellor, 16, 115-21. 21 learning styles: individualizing computer-based learning environments anne groat and tim musson napier university in spite of its importance, learning style is a factor that has been largely ignored in the design of educational software. two issues concerning a specific set of learning styles, described by honey and mumford (1986), are considered here. the first relates to measurement and validity. this is discussed in the context of a longitudinal study to test the predictive validity of the questionnaire items against various measures of academic performance, such as course choice and level of attainment in different subjects. the second issue looks at how the learning styles can be used in computer-based learning environments. a re-examination of the four learning styles (activist, pragmatist, reflector and theorist) suggests that they can usefully be characterized using two orthogonal dimensions. using a limited number of pedagogical building blocks, this characterization has allowed the development of a teaching strategy suitable for each of the learning styles. further work is discussed, which will use a multi-strategy basic algebra tutor to assess the effect of matching teaching strategy to learning style. introduction while the need to adapt teaching to the needs of a student is generally acknowledged (see corno and snow, 1986, for a wide review of the literature), little is known about the impact of individual learner-differences on the quality of learning attained within computer-based learning environments (cbles). what evidence there is appears to support the notion that individual differences have implications for the degree of success or failure experienced by students (ford and ford, 1992) and by trainee end-users of software packages (bostrom et al, 1990). the problem is to identify the way in which specific individual characteristics of a student interact with particular features of a cble, and how the interaction affects the quality of the resultant learning. teaching in a cble is likely to require a subset of teaching strategies different from that subset appropriate to more traditional environments, and the use of a machine may elicit different behaviours from those normally arising in a classroom context. of the candidate-factors influencing learning like personality, motivation, cognitive style, level of ability, and learning style it is the last that we consider here. for this work, 53 anne groat and tim musson learning styles: individualizing computer-based learning environments learning style is defined as a preference for processing information in a particular way when carrying out a learning activity; it is the observable behaviour that arises from a person's underlying personality, motivation, cognitive style and ability, and which is stable over a variety of situations. there is some disagreement about the stability, and therefore usefulness, of learning style. for example, entwistle (1988), pask (1976) and schmeck (1983) argue that individual students tend to be consistent in their approach to learning, but on the other hand laurillard (1984) and ramsden (1979) consider that a student's perception of a particular situation is of overriding importance in influencing what and how a student learns. the position adopted here is that each student's approach to learning is determined both by a relatively stable entity called learning style, and by more situation-specific concerns. determining learning style there are two basic approaches to modelling a student's learning style for use within a cble. it can be done dynamically during the course of a tutorial session. for the student, this has the advantage of being non-intrusive, but it is expensive in terms of system resources, and it is unlikely that a student would use a small tutoring system for long enough to enable it to infer what learning style is being used. an alternative is to model the student's behaviour by using stereotypes. for a small system this is a potentially useful approach that can substantially simplify the problem of modelling a student. the main disadvantage is that it requires the student to answer questions before starting work, and so runs the risk of alienating its target audience. the aim should be, therefore, to ask as few questions as possible, choosing those with good predictive power. the most radical approach is to ask only one question. this has been explored to some extent by ford (1985) and by clarke (1993). in each case, ford's study preference questionnaire was administered to a group of postgraduate students. one item was identified that differentiated, at an acceptable level of significance, between students on the holist/serialist dimension. the problem is that each attempt identified a different item, a result that clarke attributed either to the interference arising from the reliability analysis or to sample characteristics. whatever the reason, it highlights the difficulty of obtaining a reasonable balance between predictive value and user acceptability. acceptance of a time-consuming procedure might be enhanced if the investment of time and effort on the student's part was made worthwhile. knowing what one's learning style is can be shown to be beneficial not only within computer-based learning environments, but also in the wider educational context. knowledge about how one learns has been shown to lead to more effective learning (pennell, 1985). the instrument that was chosen to assess learning style is based on kolb's experiential theory of learning (kolb, 1984). this theory considers that learning is more than just a cognitive process; it is a series of experiences that involve cognition, and the learner is seen as moving iteratively through four stages. there are, however, problems with kolb's learning styles inventory (lsi): with the psychometrics (freedman and stumpf, 1978) and with the construct and face validity (wilson, 1986). although the underlying theory is considered helpful, the shortcomings of the lsi prompted honey and mumford (1986) to devise a questionnaire for measuring learning style. their learning styles questionnaire (lsq) is based on self-reported behaviour, and their learning styles differ in some respects 54 volume 3 number 2 from kolb's. the problem that triggered this investigation was how best to provide adults returning to higher education with the basic skills in algebra to enable them to pursue their studies successfully. such people are likely to have spent a number of years outside formal education. we can therefore make no assumptions about their knowledge of their own study habits, and a questionnaire that refers directly to behaviour in an academic setting is not likely to be useful. because the lsq asks about behaviour and beliefs in the workplace, in particular within a managerial context, it was considered that it would be more meaningful to the target population. the learning styles questionnaire the lsq consists of 80 items which the respondent is required to check either positively or negatively. there is no time limit for completing it and the respondent is assured that there are no 'right' or 'wrong' answers. the questionnaire seeks to identify general trends, and so gives equal weight to all answers; no one answer has more significance than any other. most of the questions are concerned with a person's normal behaviour in an information-processing and decision-making environment, for example: / thrive on the challenge of tackling something new and different, lam careful not to jump to conclusions too quickly. i like to relate my actions to a general principle, lam keen to try things out to see if they work in practice. but a few are concerned with beliefs, for example: it is more important to enjoy the present moment than to think about the past or future. i believe that rational, logical thinking should win the day. on completion, the questionnaire is scored, the respondent is provided with the result and is given the information necessary for its interpretation. four learning styles are identified. these are activist, reflector, theorist, and pragmatist. activists want to learn something new, like immediate involvement in activities, and enjoy the challenge of problem-solving. reflectors prefer to gather and assimilate as much information as they can from as many sources as possible, and like time to consider; they do not like to commit themselves to a course of action before they are ready. theorists like to explore complex ideas and concepts, and prefer to use their observations and experiences to build their own models and theories. pragmatists want to learn techniques, and like to practise and experiment; they prefer to address real-world problems. the authors have documented the reliability of the questionnaire, and while they admit that its construct validity is difficult to prove, they claim its face validity has rarely been questioned by those who have completed it. allinson and hayes (1988), however, failed to find evidence to support the construct validity of the four factors identified by honey and mumford. instead, they distinguished two factors which they called analysis and action, and which they considered to be statistically justifiable. their subject population was drawn mostly from managers in the civil service and public enterprises, and this may have reduced the range of likely behaviours. doubts have been cast upon the questionnaire's ss anne groat and tim musson learning styles: individualizing computer-based learning environments predictive value by allinson and hayes (1988) and by fung et al (1993). in both cases, however, there are problems with the interpretation of the data. in the work of allinson and hayes, the correlation between the scores on the questionnaire and the students' results (from both examinations and project work) did not reach significance. they attribute this, at least in part, to inconsistencies in the assessment procedures. in the work of fung et al, the students were asked to rate their preferences for items presented in a list of learning activities; in other words, actual student performance was not assessed. neither of these studies provides grounds for rejecting the lsq as a useful indicator of learning style. all that can be inferred is that we must use it with caution until more is known about its effectiveness. computer-based learning environments have the potential to address the issue of consistency in assessment. once the assessment process is devised, it is the same for everyone; the effects of variation in assessor behaviour are eliminated. the nature of the assessment process may, of course, favour some learning styles over others. this may or may not be desirable, but the issue highlights another reason for attending to learning style. the relationship between assessment design and learning style can be made explicit, and decisions about assessment procedures are therefore available for scrutiny. from learning styles to teaching strategies empirical work in the classroom has indicated that effective instructional design requires knowledge of how people learn (van der veer and beishuizen, 1986; bernaert, 1978, reported in van der veer and beishuizen, 1988). the work of snow (1989) on aptitudetreatment interaction demonstrated that differences in instructional treatment moderated the learning differences between learners of high and low general ability. others have made a strong case for adapting teaching to learner needs over a range of cognitive and personality styles (ash, 1986; corno and snow, 1986), and, using a computer-based task, ambardar (1984) demonstrated significant differences in search behaviour between field-dependent and field-independent types. the converse, that preferred learning style has a decisive influence on what people choose to learn, has been observed. there is a tendency for people to select for study at college level those subjects that fit with their learning style (witkin et al, 1977; entwistle, 1988). how can the learning styles identified by the lsq be used to generate teaching strategies? of the key aspects of the learning styles described by honey and mumford, two can form orthogonal, bi-polar dimensions. one dimension has at one extreme the preference for dealing in the concrete, a need to link the domain material with real-life experiences; and at the other the preference for dealing in the abstract, for symbolic manipulation and making links within the domain. this distinction has been proposed by gregorc (1979), and as 'visual' versus 'verbal' thinking by stenning et al (in press). on this basis, activists and pragmatists prefer the concrete, and reflectors and theorists prefer the abstract. the other dimension is concerned with the challenge/safety dichotomy. recent work (newstead, 1989; stenning and cox, personal communication to the authors of this paper) has demonstrated marked differences in behaviour, called 'rashness' and 'hesitancy', along what appears to be a similar dimension. compared to the previous dimension, this one is more difficult to interpret, and possibly also to implement. honey and mumford suggest that activists and theorists tend to seek challenge, whereas the 56 alt-j volume 3 number 2 other two learning styles prefer safety. pragmatists prefer to be shown what to do and be given ample opportunity to practise, and reflectors work best when given sufficient information to assimilate and time to reflect upon it. the learning styles can be placed on a graph formed by the intersection of the two axes (figure 1). consideration of the graph can be used to define appropriate teaching strategies thus: figure i: learning styles concrete pr safety re ac challenge th abstract pr = pragmatist th = theorist re = reflector ac a activist concrete versus abstract the choice of graphical or textual representation of a concept has implications for the way it is processed and communicated. graphics enforce specificity, or at least restrict the level of abstraction possible, whereas text enables expression of abstract ideas (stenning and oberlander, in press). perceptual and spatial relationships are better depicted by graphics, logical and temporal relationships by text (larkin and simon, 1987). the preference of learners for concrete or abstract processing, and for attending to relationships they find meaningful, can be addressed by providing either a graphical or a text-based environment. safety versus challenge 'safety' can be provided by a highly structured environment, one that teaches the subjectmatter in small incremental steps, and which provides ample opportunity either for observing what is to be done (reflector), or for putting into practice what has been demonstrated (pragmatist). meta-information for the learner can consist of what has been done and what still has to be done, but the learner is given no opportunity to alter 57 anne groat and tim musson learning styles: individualizing computer-based learning environments the order of events. for 'challenge', what is required is the opportunity to attempt more demanding tasks as soon as the learner wants to tackle them, whether or not a teacher (or cble) would agree. essentially, what is happening is that the learner is free to try out a course of action, or to test a hypothesis; in other words, to experiment in whatever way suits. the learning environment should, of course, allow back-tracking when the step taken proves to be too large. this implies that the learner should have complete control over the learning environment. designing the system once the teaching strategies have been derived from the learning styles, they have to be translated into teaching methods that are possible in a cble, at a level of granularity suitable for the design and implementation of a system. chen (1993, pp. 299-300) has developed a methodology for classifying pedagogical methods, most of which are applicable to a small system. such methods can provide the basis for designing the learning environment, and for analysing user actions. her basic pedagogical methods, with some minor adaptations, are: 1. setting goals: the knowledge or skills the learner is expected to acquire during the lesson; 2. providing instructions: uninterrupted presentations of any type of knowledge through text; 3. providing demonstrations: illustration of a particular task, a worked example; 4. providing explanations: the rationale behind a particular action; 5. presenting tasks: activities presented through text or graphical representations; 6. asking questions: questions requiring specific answers are presented; 7. providing working spaces: opportunities to interact with the program, the learner being able perform a task only when a corresponding working space is provided; 8. providing examples: the designer relates a learner's daily experience to the present learning; 9. providing reminders: the designer presents key words, such as commands, or key concepts, that a learner needs during the performance of a task; 10. providing hints: implicit cues are embedded in the task to be performed; 11. providing evaluation and feedback: the learner's performance is evaluated and feedback is provided accordingly. implementation of all of the pedagogical methods is possible within the proposed system, and each is relevant to one or more of the learning styles. all teaching strategies will have relevant goals set and stated, and appropriate evaluation and feedback provided; tasks will be presented and workspace will be provided. thus each learning style will be targeted with a teaching strategy consisting of the core pedagogical methods (those concerned with goals and feedback possibly implemented differently to match learning style), with different combinations of the others. 58 alt-] volume 3 number 2 for the 'concrete' strategies (pragmatist and activist learning styles), graphics will provide the medium of instruction (where the use of graphics will explicitly link the material to be learned with examples from the everyday world of experience), and for the 'abstract' strategies (theorist and reflector) text directly relating to symbolic manipulation will be used. for 'safety' strategies (pragmatist and reflector), a tightly structured environment will be provided, and for the 'challenge' strategies (activist and theorist), no structure will be imposed. instead information about the contents of the system, and recommendations about what might be an appropriate activity, will be available. each teaching strategy will require its own particular subset of the pedagogical methods. the teaching strategies for each of the learning styles are as follows: concrete-safety learners will have graphical examples and demonstrations provided, to tie in with the need to learn how to do things. only after students have been shown how to do a task will they be given the opportunity to perform similar tasks for themselves. concrete-challenge learners will also have the material presented in graphical form, to allow opportunities for direct hands-on experimentation. these students will be given no explicit instructions; instead, tasks will be so presented that hints about what to do will be embedded in the material. information about the content and organization of the system will be provided graphically. abstract-safety learners will be text-based, and instructions, demonstrations, and reminders will be given before students are required to carry out a task. abstract-challenge learners will also be text-based and, within an unstructured environment, students will have access to explanations and instructions. they will be provided with textual information on the content and organization of the system. figure 2: teaching strategies graphics provide demonstrations provide examples structure provide instructions provide explanations provide reminders provide hints present information about structure and content graphically freedom provide explanations provide instructions present information about structure and content textually text 59 anne groat and tim musson learning styles: individualizing computer-based learning environments the pedagogical methods can be placed on a graph of teaching strategies analogous to the one above of learning styles (figure 2). multi-strategy cbles? further work will address two issues: • whether it is worthwhile to develop a cble with relatively independent teaching strategies, that is, whether the extra effort involved in developing a multi-strategy cble is justified by improvement in student learning; and • to what extent the lsq is a useful tool for measuring learning style, particularly with reference to its predictive validity. the two issues are linked. the question is not so much whether learning style exists; most teachers believe so, and behave as if it were so, by adapting their teaching to the needs of their students, most noticeably in one-to-one situations. the problem is how to assess learning style. the identification and measurement of useful learning styles are critical to the development of the teaching strategies in a cble. honey and mumford's lsq is a candidate, despite the doubts about its predictive validity and concern about its length. factor analysis of the individual items rather than of the sub-scales has been suggested by allinson and hayes (1990); this can address questions about both predictive validity and length. while the algebra tutor is being developed, a pilot study is planned. the lsq will be administered to first-year students so that the correlation between the questionnaire items and the students' progress (in terms of course choice and measures of success, for example courses passed and qualifications if any obtained) can be considered. the relationship between learning style and teaching strategies will be examined in a more controlled manner using the algebra tutor which will present the same subject matter differently according to the four teaching strategies. using a four-by-four within-groups experimental paradigm, the effect of the teaching strategies on each learning style will be assessed. it is hoped that analysis of both field and experimental data will provide some insight into the difficulties and benefits of developing a multi-strategy cble, and also help to determine whether refinement of the lsq should be pursued. references allinson, c.w. and hayes, j. (1988), 'the learning styles questionnaire: an alternative to kolb's inventory?', journal of management studies, 25 (3), 269-81. allinson, c.w. and hayes, j. (1990), 'validity of the learning style questionnaire', psychological reports, 67, 859-66. ambardar, a.k. (1984), 'human-computer interaction and individual differences' in salvendy, g. (ed), human-computer interaction, amsterdam, elsevier science. ash, b.f. (1986), identifying learning styles and matching strategies for teaching and learning, eric document ed 270 142. bostrom, r.p., olfman, l. and sein, m.k. (1990). 'the importance of learning style in end-user training', mis quarterly, 101-19. 60 an-} volume 3 number 2 chen, m. (1993), 'a methodology for characterizing computer-based learning environments' in peg '93: proceedings of the seventh international peg conference, edinburgh. clarke, j.a. (1993), 'cognitive style and computer-assisted learning: problems and a possible solution', association for learning technology journal, 1 (1), 47-59. corno, l. and snow, r.e. (1986), 'adapting teaching to individual differences among students' in wittrock, m.c. (ed), third handbook of research on teaching, new york, macmillan. entwistle, n. (1988), styles of learning and teaching, london, david fulton. ford, n. (1985), 'learning styles and strategies of postgraduate students', british journal of educational technology, 16 (1), 17-26. ford, n. and ford, r. (1992), 'learning strategies in an "ideal" computer-based learning environment', british journal of educational technology, 23 (3), 195-211. freedman, r.d. and stumpf, s.a. (1978), 'what can one learn from the learning style inventory?', academy of management journal, 21 (2), 275-82. fung, y.h., ho, a.s.p. and kwan, k.p. (1993), 'reliability and validity of the learning styles questionnaire', british journal of educational technology, 24 (1), 12-21. gregorc, a.f. (1979), 'learning/teaching styles: their effects' in national association of secondary school principals (eds), the modern american college, san francisco ca, jossey-bass honey, p. and mumford, a. (1986), the manual of learning styles, maidenhead, peter honey. kolb, d.a. (1984), experiential learning: experience as the source of learning and development, eaglewood cliffs nj, prentice-hall. larkin, j.h. and simon, h.a. (1987), 'why a picture is (sometimes) worth ten thousand words', cognitive science, 11, 65-100. laurillard, d. (1984), 'learning from problem-solving' in the experience of learning, marton, f., hounsell, d. and entwistle, n. (eds), edinburgh, scottish academic press. newstead. s.e. (1989), 'interpretation errors in syllogistic reasoning'. journal of memory and language, 28, 78-91. pask, g. (1976), 'styles and strategies of learning', british journal of educational psychology, 46, 128-48. pennell, l. (1985), 'academic intervention program: applying brain and learning style concepts', theory into practice, 24 (2), 131-34. ramsden, p. (1979), 'student learning and perceptions of the academic environment', higher education, 8, 411-28. schmeck, r.r. (1983), 'learning styles of college students' in dillon, r.f. and schmeck, r.r. (eds), individual differences in cognition, vol. 1, new york, academic press. 61 anne groat and tim musson learning styles: individualizing computer-based learning environments snow, r.e. (1989) 'aptitude-treatment interaction as a framework for research on individual differences in learning' in ackerman, p.l., steinberg, r.j. and glaser, r. (eds), learning and individual differences: advances in theory and research, new york, freeman. stenning, k., cox, r. and oberlander, j. (in press), 'contrasting the cognitive effects of graphical and sentential logic teaching: reasoning, representation and individual differences', language and cognitive processes. van der veer, g.c. and beishuizen, j.j. (1986). 'learning styles in conversation a practical application of pask's learning theory to human-computer interaction' in klix, f. and wandke, h. (eds), man-computer interaction research macinter 1, amsterdam, north holland elsevier science publishers. van der veer, g.c. and beishuizen, j.j. (1988), 'computers and education: adaptation to individual differences' in van der veer, g.c., green, t.r.g., hoc, j.-m. and murray, d.m.(eds), working with computers: theory versus outcome, london, academic press. wilson, d.k. (1986). 'an investigation of the properties of kolb's learning style inventory', leadership and organization development journal, 7 (3), 3-15. witkin, h.a., moore, c.a., goodenough, d.r. and cox, p.w. (1977). 'field-dependent and field-independent cognitive styles and their educational implications', review of educational research, 47, 1-64. 62 editorial in this special conference issue of altj, a number of recurrent themes emerge. these include the integration of information and communication technology systems to assist in the teaching of a given higher-education curriculum. a number of papers address the questions which are on all our lips, such as: how, when and where should we embark on such a teaching exercise and, perhaps more importantly, at what cost? ryan et al, zhao et al and lavelle offer useful insights into the setting up of computer-based resources for the teaching of economics, design and contract law, while mozzon-mcpherson's study demonstrates how internet resources can enhance the process of language learning. however, an interesting finding from many of the teaching experiments is that in spite of students' enthusiasm for these new teaching technologies, they do not wish to see the traditional lecture totally replaced by computerlearning sessions. students recognize the advantages of working at their own pace, but still feel more comfortable with the tried and trusted lecture. they do not want to be distanced from their tutors, and the students from the study by ryan et al suggested that their cbl sessions should in future be supervised by subject specialists. harding et al warn that the introduction of computer-based learning materials should be considered as an 'add-on cost' to course production, and that the benefits of using such materials should be seen more in terms of an 'enhancement of education rather than a replacement of older methods'. this opinion is substantiated by allinson who suggests that the application of 'comprehensive multimedia implementations for the teaching of practical social psychology skills can be viewed as an augmentation of traditional methods and not merely an imposed substitute'. how to promote and support the appropriate use of technology in teaching and learning was the remit of the education technology service of the university of bristol, and longstaffe et al describe the activities of such a working support-service, while harding et al argue for the implementation of a simple courseware-management system if co-operative cross-platform courseware development is to be sustained. another question which is prompted by this set of papers is: how can we evaluate our 3 den/se whitelock editorial computer-based learning systems once we have built them? the formative evaluation carried o out by tosunoglu et al included a number of different measures. they utilized preto post-test change scores to assess learning gains, and also conducted a series of interviews supported by a number of questionnaires to gain insight into the subject attitudes towards the software. gordon and garrud employed an array of evaluation techniques including preand post-tests, randomized trials, and cohort comparisons. the summative evaluation of pollock et al of their first-year mathematics cal course for the bteched students shows that students' examination results were on a par with the previous year which was taught by traditional methods. a combination of formative and summative evaluations are advocated to understand the benefits of computer-based learning systems. a further theme which emanates from these papers is the role of information and communication technologies for distance learners. the study by whitelock et al forms part of an ongoing research programme to investigate students working with a computer at a distance with audio only, and then video links. their work suggests that rather than being an inferior substitute to working side by side, it seems that the audio condition has some advantages. wilson and whitelock's paper emphasizes the importance of minimizing information overload for both tutors and students (a computer-conferencing system is being used to teach a computer science course to distance learners), while kirkwood et al examine the organizational and pedagogical changes that may ensue with the more extensive use of it. one issue that was only touched upon was the use of it to promote collaborative learning. we know that it can provide an environment that supports group work, and finding ways to investigate laboratory results into curriculum packages presents us with future challenges. denise whitelock, the open university 4 computer-based laboratory simulation: evaluations of student perceptions norrie s. edward school of mechanical and offshore engineering, the robert gordon university, aberdeen providing resources to meet the needs of oil workers who miss blocks of an engineering course was the motivation for producing computer-based simulations of laboratory equipment. this paper reports on student perceptions of various aspects of the package. the factors are grouped into (i) motivation and support, and (ii) presentations and interaction. a schematic representation of the controls and instrumentation was used. two classes, engineers and non-engineers, were the pilot groups. the engineers clearly preferred laboratories, whereas the non-engineers were just as happy with the simulation. the results of the survey suggest that while computer-based simulation may be an alternative to laboratories, even the best alternative, much is lost as a result. practical appreciation and team-working skills are not well developed the schematic presentation is easy to use, but gives the student little feel'for the operation of a real plant. introduction laboratory experimentation in engineering is an essential part of the three main components in an engineer's formation. the theoretical constructs and models are imparted in lectures and tutorials. workshop hands-on activity allows the student to acquire an understanding of the interaction of design and manufacture, and the constraints both impose. characteristics of plant are investigated through experiment, and this aids the learner's understanding of the limitation of models in predicting performance. the learner also gains an appreciation of the nature of errors and of the construction of plant. but while the oil industry has brought prosperity to the northeast, it has also brought unique educational demands: the working arrangements place severe restrictions on part-time student attendance. technicians work a block of two to four weeks offshore, followed by a similar period of leave. different companies have different arrangements, and shift-change days. a review of the literature (see boud et al, 1984) suggests that the main aim of laboratory work should be to teach inquiry methodology and experimental design. it is often also 41 nome s. edward computer-based laboratory simulation: evaluations of student perceptions reported that students do not enjoy the activity. in contrast, our staff at the school of mechanical and offshore engineering are clear that the main aims of the work are to establish the links between theory and practice, to aid visualization, and to develop teamworking skills, and our students, in common with carter's (1980) findings, enjoy this aspect of their education. whatever the true outcomes of laboratories, the activity is seen as essential by our staff and students alike. alternative provision, such as directed reading to allow part-time students to stay abreast of theory, is not difficult to arrange, and most of them gain a knowledge of plant construction through daily contact in their working lives. but this is not to say that they are afforded the opportunity to investigate the characteristics of the equipment. rather, the objective will often be to maintain a steadystate operational condition. we therefore felt that the provision of an alternative to laboratory work was essential. the evaluation reported in this and another paper (edward, in press) set out to test the contention that in comparison with laboratories, cognitive learning would be as high with a simulation, but that practical appreciation would suffer. the opportunity was taken to carry out what flagg (1990) describes as 'formulative evaluation' based on our students' perceptions of the package, i.e. 'the systematic collection of information for the purpose of informing decisions to design and improve the product'. reeves (1992) avers that the effectiveness of a package is constrained by (i) the design of the user-interface, and (ii) the motivation and expertise of the user. it is on these and related factors that this paper reports. since students may miss either or both of the laboratory experiment and the relevant theory, it was decided that any alternative should provide both. the final package was multiple media. the theory was covered in an interactive text-based workbook. a separate laboratory procedure sheet was also provided so that students who had attended the lectures did not have to work through the book. as an aid to visualization, a short video showing the plant in operation was produced. the heart of the package was a computer-based operational simulation. this was interactive, and allowed users the same type of control as they would have on the actual plant. similarly, the readings provided were based on actual experimental performance data from the machine, a centrifugal pump being the example examined here. using real data allowed representative systematic error to be included. random error was approximated by analysing repeated readings and applying an appropriate randomization factor to each of the displayed outputs. a desirable package would be 100% faithful to actual performance, would include effects such as sound, would have a photographically realistic interface, and would be controlled in a manner identical to the real plant. the first of these would have required an enormous database, but the inaccuracy introduced by limiting the database is very small. sound, though desirable, was not felt to add much, and at the time would not have been available on most computers. the realism is important. does the display have to look like the equipment for the user to gain the most from the experience? must a dial gauge be so represented, or can it be replaced by a digital output? must a valve be opened by clicking and dragging a lever, or can an indexing button be used? in the interests of reducing the production time* and of ensuring rapid response and portability to relatively lowperformance machines, compromises had to be made. a diagrammatic interface was used 42 alt-j volume 4 number 3 (figure 1). (in passing, it is worth noting that our gas turbine simulator uses a much more realistic interface.) these compromises were not made lightly and, as discussed below, i have separately been investigating their importance. presentation window file valve opening control valve .5 + 10 1 0 speed control figure /: screen representation of the pump controls and instruments evaluation of the centrifugal-pump simulation was conducted by running the package with a pilot group of students. students were drawn from two classes. one class (30 students) was composed of mainstream engineers. the other (26 students) was composed of cross-disciplinary technology and business undergraduates. means were compared using an independent t-test with a <.o5 level considered to be significant. few significant differences were detected between the two classes, and they are generally reported here as a single cohort. they were divided into two groups of 28, one of which carried out the actual laboratory experiment while the other obtained their results from the computerbased simulation (cbs). most of the aspects of the experience reported here relate only to the simulation, and for them the relevant sample size is 28. evaluation was by means of questionnaires administered to all students, and follow-up interviews. the questionnaires 43 nome s. edward computer-based laboratory simulation: evaluations of student perceptions used five-point likert scales. space was provided for free-form comments, with suggestions being made that respondents might wish to comment on matters such as ease of use, help facilities, difficulties with operation, and so on. the response rate was 84% overall, with the cbs group return-rate at 89%. ten formal audio-taped interviews were conducted, and around the same number of ad-hoc informal interviews were written up afterwards. in the formal interviews, a semi-structured questionnaire was used. quite a range of factors was investigated, and i have grouped these factors into three main categories. these are the learning outcomes, the presentation and interface, and finally motivation and support. i have reported elsewhere on the learning outcomes (edward, in press). in this paper, i report on the findings on the remaining factors. table 1 gives the means and range for each of the factors discussed in this paper. factor ease of use of the package (general section)* ease o f use of the package (simulator section) realism of the simulation attractiveness of the screen presentation length of the package effectiveness of simulation as a laboratory alternative sufficiency of interaction provided guidance on use of the package (general section) guidance on use of the package (simulator section) guidance on writing report sufficiency of feedback on performance motivational impact of the simulation preference for independence mean 3.91 3.88 3.56 3.41 2.71 3.18 2.67 3.18 2.76 2.95 2.95 3.29 3.28 (n = 28 for all factors. * indicates low t o high transposition of results) range 1-5 1-5 2-4 1-5 1-4 1-5 1-4 1-5 2-4 1-5 1-4 2-5 1-5 table i: analysis of student perceptions of the computer-based simulation package presentation and interface ease of use students rated how easy they had found the simulation to operate, both in the general questionnaire and in the one on the cbs itself. the scales were reversed, but after transposing one set so that a low score indicates that it was found difficult to use, the two mean scores were almost identical at 3.88 and 3.91. a total of 46% of the respondents rated the package easy to use. students did report some difficulties, but these generally were due to faults in the program which were readily remedied. their most compelling criticism was of the quality of on-line help. the laboratory group found the actual plant slightly more difficult to use (mean 3.68), but interviews suggested that this was welcomed. the plant is fairly complex and the students enjoyed operating it. 44 alt-j volume 4 number 3 screen presentation % 20 10 very poor 2.00 3.00 4.00 screen presentation figure 2: students' appraisal of the representation of the plant on screen very good because of the compromises made in the representation of the apparatus, as reported above, i was anxious to see user-reaction to the result. it was something of a relief that the mean rating was 3.41, and only 16% of students selected a rating below the mid-point. but the students were somewhat more circumspect about the realism of the simulation. no student in either class used the extremes of the scale. here also the overall mean of 3.56 suggests that a suitable approach has been adopted. complacency must be avoided, however, as one or two comments were either unfavourable or uninformed. in the latter category, one student welcomed the simulator's ability to repeat the results 'without error', having failed to appreciate the distinction between random and systematic error. one student who had previously used a simulation said that he preferred its approach which featured optional controls offering different levels of realism from alpha-numeric to a fully realistic representation of the plant. length although length was included in the assessment of the simulation, some students obviously treated it as relating to the overall experience. even for the simulation, a definition of this may have proved difficult because users had latitude in use of the various sections, repeat tests and self-assessment questions (saqs). some students said they felt that even more choices should have been available. some wished to be free to escape from an saq if they felt it was inappropriate. another said he would have liked more depth to the theory and more testing saqs. it was pointed out that these were provided in the workbook. he said: 'i did read the workbook but i'd have preferred to have it all in one place when i was at the computer. also i would have liked to be able to go back to the theory when i got stuck.' the laboratory group reported that on average they had spent 100 minutes on the test, while those using the cbs approach spent only 74 45 nome s. edward 'computer-based laboratory simulation: evaluations of student perceptions minutes (see figure 3). overall, despite the shorter times recorded as being spent on the cbs than on the laboratory, students tended to feel that the package was rather long. they rated it from too short to much too long, but the mean of 3.9 shows that generally it was found slightly long. it suggests that more options on routing would be welcomed by the students. 40, 30, 2010< 0 i fin n • . 1in experimental method •laboratory • cbs 30 35 45 50 60 70 80 85 90 100 120130180 time spent on activity (minutes) figure 3: length of time spent by students of both groups on the activity effectiveness of the simulation the group were asked how effective they had found the pump simulation, and how effective they felt the general approach would be as a laboratory substitute. this was one area in which there was a significant different between the two classes who formed the pilot group, though the small sample sizes make interpretation of the difference speculative. the mechanical engineers did not in general feel that the simulation was an effective alternative. as one put it: 'it [cbs] wouldn't give you practice of hardware and setting things up properly, running it and taking good readings.' another said: 'it rather defeats the purpose of doing lab work.' the cross-disciplinary technology and business class were more appreciative. one called it the 'perfect alternative', and another described it as 'semi hands-on'. the difference may reflect the engineer's identification of working with plant as 'what an engineer does'. interaction the appropriateness of the degree of interaction to students' needs is related to routing. a low score indicated that students would have liked more interaction, and as the overall mean was 2.67, this appears to have been the case. most students rated the factor between 'much too little' and the mid-point. their comments suggest that they were referring mainly to the theory section where a few students said they would have welcomed more depth being available. one more ambitious individual said: 'i would have liked to have 46 au-j volume 4 number 3 percent much too little 2.00 3.00 extent of interaction figure 4: student perceptions of the adequacy of the interaction provided 4.00 tried other pump speeds so i could have done an ndg (non dimension group) calculation'. the conclusion is that more and better detail and interaction would be desirable. motivation and support guidance on use and report writing, and feedback on performance as these topics are obviously related they will be considered together: they gauge the level of support the students felt they had received. the simulation was run in somewhat contrived circumstances, but every effort was made to confine support to dealing with operational problems. it must also be recognized that the so-called hawthorne effect may have been invoked: the students were aware that they were the pilot group for a novel methodology. a rather important factor if the simulation were to be used independently is guidance on the use of the package. again, a question on this was included in the general section of the questionnaire, and in the section specific to the simulator. responding to the general question, many of the simulator group felt better guidance could have been beneficial, although the overall mean ranking of 3.18 was moderately favourable. (the laboratory group felt that guidance was very good, with a mean of 3.63.) there were teething problems with getting into the package which will be readily resolved, but of more concern were expressed difficulties in running the simulation: 'i went off in the wrong direction at the start, and didn't have time to complete it. if it was a bit clearer as to what to expect and what to do . . . ' in response to the question in the simulator section, rather strangely the rating for the combined group dropped from 3.18 to 2.76 (significant at the 5% level). admittedly this was due to many of the students rating guidance as 'too little' but none as 'much too 47 nome s. edward computer-based laboratory simulation: evaluations of student perceptions little'. the most frequently cited reason given was shortage of on-line help, particularly during the operational simulation. • the responses on guidance on report writing are rather disappointing, since the workbook activities were intended to lead the user through the report. laboratory groups from both classes actually rated the guidance on report writing higher than did their counterparts on the cbs. one cbs user said: 'i knew what the procedures were on the day, but wasn't sure how to apply them or what they were for'. the overall means were 3.32 for the laboratory groups, and 2.95 for the cbs groups. (what the implications are is unclear, but the topic is discussed below.) this contrasted with the responses on feedback on performance where the respondents reported better feedback from the simulation than the laboratory. the mechanical engineers were only slightly higher at a mean of 3.00 as against 2.92, but 89.7% of the combined cbs group rated feedback at or above the mid-point. the non-engineering laboratory group gave a significantly lower rating in the experiences, with a mean of 1.86. only one student in this group even rated feedback at the mid-point. the technology and business simulation group mean at 2.86 was significantly higher than the laboratory group, but still more than a quarter of the students rated this factor as 'poor' or 'very poor'. it seems that more can be done to enhance feedback, a point addressed below. motivation asked how motivational the experience was, the two classes responded rather differently (see figure 5). overall, the mean for the laboratory group was similar to that for the cbs group (3.21 as against 3.29). in the case of the mechanical engineers, there was little difference between the two mean ratings, with the laboratory rating being slightly the higher. the non-engineers, however, produced a mean of only 2.71 for the laboratory, compared with a mean of 3.25 for the simulation. count .5 experimental method • laboratory • cbs 2.00 3.00 4.00 figure 5: student perceptions of motivation of method 48 adj-j volume 4 number 3 to an extent at least, this may again reflect the engineers' self-identification. they were much more adamant that the missing element in the cbs was hands-on, which they felt was both necessary and enjoyable. overall, however, both the ratings and the students' written and oral comments make it clear that the non-engineers did not find the laboratory experience motivational. this calls into question the nature of this type of activity for this less technical class. are the right objectives being pursued? is the right approach being adopted? is enough time being given? further, the question of how motivational students had found the experience begs the question of what motivated them. were they motivated by the experience per se, by the opportunity to gain knowledge, or by the need to satisfy an assessment criterion? this factor will warrant fuller investigation, but might be related to the students' motivation for taking the course. independence a feature of laboratory work is that there is always a lecturer supervising who will offer advice or answer queries. the simulation is intended to be used without this support. students were asked whether they preferred to have a lecturer present or to work independently. the results were rather surprising. those who had done the actual laboratory declared a significantly higher preference for having a lecturer present (mean 2.5 as against 3.28). analysis of the figures produced more surprises. both groups of nonengineers had a mean of 3.0, the mid-point of the scale. the significant difference was due to a very marked difference between the two groups of engineers. the laboratory group much preferred the support of a lecturer (mean 2.00), whereas the cbs group declared a preference for independent working (mean 3.45). interviews did little to explain this difference. students generally said that the information they were given about the laboratory was sufficient, and that they rarely needed to consult the lecturer: 'we usually know what we have to do. we just get on with it and take our readings. the lecturer helps if we ask, but mostly we don't need to.' the cbs group, on the other hand, made little comment, though one student did say that he felt he had to find out how the simulation was operated by trial and error. probably, although asked about their general preference, students responded in the context of the activity they had experienced. help one question not specifically asked was the students' opinion of the adequacy of on-line help. in retrospect, this important issue should have been included: it was the most quoted topic of concern to the users. help provision is an important issue which is seldom given the attention it deserves; both the strategy and the presentation of assistance may greatly affect the effectiveness of a package. students generally felt that insufficient help was available through the computer (although they agreed that most of it was available in the accompanying workbook). one student said: "the instructions weren't clear. you sit down and you do something. you think: what am i to do now? nothing tells me. it's trial and error, like varying the speeds and things. see what happens.' student comments have alerted me to the crucial role which the help provision plays in the success of a package. i plan to enhance the provision in the present packages, and to devote significantly more attention to it in future designs. 49 nome s. edward computer-based laboratory simulation: evaluations of student perceptions correlations few significant correlations were found which related to the factors considered in this paper. with regard to factors related to support, the feedback the students felt they had received proved to be significantly correlated (p > 0.05) with quite a number of other factors. it was very strongly related to the motivational influence on the student (p. =.000), but negatively correlated with the time they spent on the package. this tends to suggest that feedback led to the desire to get more out of the experience, and hence to spending longer investigating the scope of the package. respondents had been asked to rank their objectives in pursuing the degree course. the options offered were broadly in two groups: (i) extrinsic, e.g. pursuit of a qualification, improving employment prospects, and (ii) intrinsic, e.g. increasing knowledge, challenge. feedback also tended to be rated higher by those who had ranked the pursuit of a qualification lowest, although whether this suggests that the extrinsically motivated students merely wanted results with the minimum of effort is debatable; it could equally be argued, given an opposite correlation, that those seeking knowledge had proved to have been less easily satisfied with the feedback given. guidance on use, and guidance on the report and ease of use, were positively correlated. guidance on use was negatively correlated with the desire to gain knowledge as a motivational factor. taken together with the fact that the easier students found the package to use, the more time they spent on it, it could be suggested that those seeking knowledge benefited from feeling better guided and finding the package easier to use in seeking deeper understanding. a more jaundiced connotation could be that, since it is not correlated with their knowledge gain, either their expectations were higher, or they found their search fruitless. disappointingly, few correlations were detected in any of the measures on the simulation itself. the effectiveness of the simulation correlated positively with the usefulness of the workbook, which suggests that those who found the workbook most informative may then have been better prepared for the simulation. students were asked a general question on the suitability of cbs as a laboratory alternative. this factor, as might be expected, correlated positively with their perceptions of the effectiveness of the pump simulation. much less to be expected was the negative correlation between perceptions of the general suitability of simulation and of the effectiveness of guidance on this simulation: a positive correlation would have evoked no surprise. the only other correlation of note was between the length of the simulation and the ranking of seeking a qualification as the students' reason for study, which again suggests that the more extrinsically motivated were looking for the shortest route to satisfying the assessment criteria. discussion a number of students commented on the value they placed on the summary theory section. they felt that the saqs helped to reassure them that they understood the topic. this section, they felt, could usefully be extended. this corresponds with the findings of coleman et al (1994) when evaluating an application of cal with electronics students. motivation is promoted by keeping learners informed of their progress (see, for example, ritchie and garner, 1995). motivation has also been related to prior preparation. colgan et al (1994) found that only 50 aa-j volume 4 number 3 31% of respondents wanted to use cbt again, and inferred that preparation must be enhanced. my finding, that those who made most effective use of the workbook which contained the preparation also performed best in simulation, perhaps supports this. motivation is likely to be a crucial factor in an open-learning context. k. bateson and w. simpson, of the university of surrey, reported in 1995 (in a paper given at the cal95 conference learning to succeed at the university of cambridge) that allowing electronics students unrestricted freedom to explore systems behaviour was intensely motivating. catteral and ibbotson (1994), however, found that students' motivation declined over time, and concluded that cal is useful for preparatory work but too timeconsuming to remain motivating. i feel that a well-presented package without unnecessary time-consuming effects, and with what phillips and moss (1993) have described as 'an attractive interactive manner', can be stimulating and can remain motivating. i used a multiple-media approach, and found that motivation was related to feedback and the effectiveness of the workbook. as i mentioned above, on-line help was pinpointed by a number of students as an important motivational factor. whiting (1989) found that an easily used screen presentation reduced the need for what in his application was largely off-line help. he suggests optional presentations to cater for different learning approaches. a thought-provoking paper by pengelly (1993) on motivation and on-line guidance is concerned with developing domain reasoning, monitoring, and reflection levels of learning. he suggests a broad model for developing support by monitoring the search paths of the users, and by analysis inferring their reasoning processes. he points out that a human teacher becomes sensitive to a student's moods, motivation, compliance, etc., and modifies his or her approach accordingly. in some way, the computer must emulate this if effective prospective help is to be provided. screen presentation is an important issue. a declared objective of laboratories is to provide an appreciation of the plant, and presentation obviously has a bearing on this. the attractiveness of the display, ease of interpretation and use of effects are all likely to influence a user's approach to and evaluation of the product. respondents would have welcomed more realism: 'the gauges and that sort of thing. sound and sight has a lot to do with things. if you're looking at things, it does make a difference from seeing it on screen.' in general, however, the students found the display clear and attractive, and were at ease using it, something which they welcomed. few if any researchers have published systematic evaluations of the importance of realism in simulation. my own work (edward, 1996) revealed that cognitive learning was perceived to be lower and practical appreciation was poor in a diagrammatic presentation compared to a realistic one. waddick (1994) found that students could as effectively learn spectrophotometry on a realistic simulation as on real equipment. leary (1995) exploited the potential to the full by allowing the internal operation of plant to be viewed by the user. although there are some benefits in using, as in the pump, a restricted diagrammatic presentation, i must concede that for engineers no simulation can take the place of laboratories. that said, if a simulation is required for whatever use, i now believe that it is worth making it as realistic as possible. adumbra (1994) also advocates interaction which he avers keeps the learner 'awake'. our si nome s. edward computer-based laboratory simulation: evaluations of student perceptions simulation is, of course, very interactive, all controls being operated by the student. students also appreciated the interaction and feedback in the theory section, and wished this to be expanded. this corresponds with the findings of catterall and ibbotson (1994), although the need diminished as their students gained confidence (they wanted more control over the interaction). catterall and ibbotson reported at alt-c '94 in hull that their students sought control over routing and number of attempts at questions. this is an aspect which i propose to address in future simulations. whiting (1989) found that where ease of use and interaction were highly rated, the learners had less need of external support. whether the workbook can eventually be absorbed as an interactive section of the package is debatable, but it is a useful long-term objective. conclusions i am reluctant to use the word 'successful' about the results because at best the simulation was a qualified success. i feel that a sound start has been made, but that my evaluations have revealed scope for improvement. my objective was to produce an alternative for students who miss laboratories. i do not feel it provides an equivalent experience. although the simulation was clear and easy to use, practical appreciation was diminished. despite completing the cbs more quickly, students still wished it to be shorter this suggests that, in contrast to laboratories which engineers enjoy, it was viewed as a means of satisfying an assessment criterion. my aim now is to enhance the package, and produce what solomon (1993) describes as a simulation which 'take[s] the best of the experiential and combinefs] it with more traditional learning methods [. . .] [to] enhance but not replace traditional learning techniques'. references adumbra, c. (1994), 'enhancing student experience of computer-aided learning packages', proceedings of the conference on computer aided learning in engineering, sheffield, september 1994, sheffield: university of sheffield. boud, d., dunn, j. and hegarty-hazel, e. (1984), teaching in laboratories, guildford: srhe & nfer nelson. carter, g., amour, d. g., lee, l. s. and sharpies, r. (1980), 'assessment of undergraduate electrical engineering laboratory studies', iee proceedings, a.460. catterall, m. and ibbotson, p. (1994), 'the development of a low technology marketing cbt', account, (6) 1. coleman, j. n., kinniment, d., burns, f., and butler, t. (1994), 'teaching in a third of the time: a successful application of computer-aided learning in degree-level electronics', proceedings of the conference on computer aided learning in engineering, sheffield, september 1994, sheffield: university of sheffield. colgan, n., mcclean, s. and scotney, b. (1994), 'computer-based teaching and evaluation of introductory statistics for health science students: some lessons learned', association for learning technology journal, 2 (2), 68-74. 52 alt-j volume 4 number 3 edward, n. 'evaluation of computer-based laboratory simulation, computers and education (in press). edward, n. (1996), 'screen presentations in laboratory simulations as perceived by students', proceedings of the sera (1995) conference, glasgow: university of strathclyde. flagg, b. (1990), formative evaluation for educational technologies, hillsdale nj: lawrence erlbaum. leary, j. (1995) 'computer-simulated experiments and computer games: a method of design analysis', association for learning technology journal, 3 (1), 57-61. pengelly, m. (1993), 'computer-based assessment to support the learner not the assessor', teaching and learning technology programme workshop on assessment of learning in higher education, workshop papers, sheffield, november 1993 (isbn 1-85889-086-1). phillips, t. and moss, g. (1993), 'can cal biology packages be used to replace the teacher?', journal of biological education, 27 (3), 213—16. reeves, t. (1992), 'evaluating interactive multimedia', educational technology, may, 47-52. ritchie, g. and garner, p. (1995), 'computer-based laboratory tutorials', iee colloquium: computer-based learning in electronic education, digest no. 1995/098, p. 13/1-2, london: iee. solomon, c. (1993), 'simulation training builds teams through experience', personnel journal, june. waddick, j. (1994), 'case study: the use of a hypercard simulation to aid in the teaching of laboratory apparatus operation', etti, 31 (4), 295-301. whiting, j. (1989), 'an evaluation of some common cal and cbt authoring styles', etti, 26 (3), 186-200. s3 learning style and learning strategies in a multimedia environment pat paterson and john rosbottom department of information science, university of portsmouth there is a growing realization that it may be expeditious to combine elements from different theories of learning when trying to derive a coherent and usable policy towards computermediated learning. consideration of the subtle distinction between computer-aided learning (cal) and computer-aided instruction (cai) conform the basis of a possible classification of computer-mediated learning, and hence of multimedia tools. this classification enables the development of a continuum upon which to place various strategies for computer-mediated learning, and hence a means of broadly classifying multimedia learning tools. it is also important to consider the characteristics of the user when deciding whether to employ interactive multimedia as a tool for learning. learning style is predisposition to behave in a particular way when engaged in the learning process. learning strategies are the methods employed by the learner to achieve learning. taking the work of pask (serialistlholist) and kolb (learning style inventory) as starting points, research is being done to investigate links between learning style and strategy and multimedia. such knowledge could prove invaluable when attempting to design multimedia learning materials for the widest possible educational use. introduction two important factors need attention when contemplating how and where to use the new computer-based learning technologies. firstly, in order to select the right kind of software, it is necessary to have a clear picture of the desired outcomes of a particular learning situation. there are many varieties of interactive multimedia available, and matching the tool to the job should be an early consideration. of equal, if not greater importance, is the question of whether, for a particular learner, interactive multimedia is the most appropriate learning tool. if there is too great a mismatch between the learning styles and strategies of a particular learner and the given learning environment, learning may be inhibited rather than enhanced. 12 alt-] volume 3 number i the right tool for the job? behaviourist theories of learning have been largely overshadowed since the 1960s, when modern cognitive psychology began to gain in importance. generally, behaviourism relies on a definition of learning as an observable change in behaviour not caused by physical development of the subject, and as such disregards internal learning by a process of maturation. the rationalist/cognitive approach places the main emphasis on the acquisition, manipulation and recall of abstract symbols, implying a greater interest in the mental processes involved with learning. it would be simplistic to assume that theories of education rely exclusively upon either of these extremes. however, they do offer a convenient starting point upon which to build a framework wherein to establish the capabilities of interactive multimedia as a tool for learning. table 1 shows the evolution of the two main strands of computer-mediated learning. cal implies an active role for the student, who is more in charge of the learning process, and hence has greater freedom in the management of this learning. cai suggests that the student is the recipient and adopts a more passive role. table i: table io show the relationship between computer-mediated learning and learning theory • theories of education instructional approach learning approach behaviourist cognitivist skinner, presse/... piaget, brimer, gagne computer aided instruction • computer aided learning • ^ classification by amount of learner freedom • this framework enables the development of a continuum on which to place various strategies for computer-mediated learning, since as the use of computers as learning tools matured, it resulted in a wide diversity of applications. some of these are shown in table 2. table 2: table to show relationship between strategies for computer-mediated learning computer aided instruction computer aided learning ^ classification by amount of learner freedom • drill and practice case studies all open styles of learning equipment simulations experiment simulations rule-based problern solving some limited problem solving business and strategy games exploration of information bases routine testing creative composition 13 pot paterson and john rosbottom learning style and learning strategies in general, cai leads to a single learning outcome while cal encourages a variety of learning outcomes. intelligent tutoring systems, which have been produced as an application of artificial intelligence, develop elaborate models of the expert and the student but are focused on the support of a single learning outcome. cognitive tools (jonassen, 1992), on the other hand, are environments that support a wide range of learning tasks in almost any content domain. between the two extremes, lie microworlds (kommers, 1992), where a more closed environment offers apparently free exploration but limits the conceptual domain in order to focus the attention of the learner. the placing of learning outcomes and environments within the same framework as before is shown in table 3. table 3: table to show learning outcomes and environments computer aided instruction computer aided learning < classification by amount of learner freedom • single learning outcome multiple learning outcomes intelligent tutoring systems microworlds cognitive tools the steadily-increasing range of interactive multimedia applications and authoring environments can also be placed on the same continuum between cai and cal. it is cost-effective, both in money and time, to try and match a particular learning situation with a suitable multimedia environment, since much of the complexity of some of the more sophisticated cal applications may be wasted on a simple cai implementation. table 4: table to show placing of some commercial ¡mm software computer aided instruction computer aided learning • ^ classification by amount of learner freedom • self-contained self-contained but allowing completely open closed environments limited links to restricted environment supporting range of external sources almost unrestricted links eg hypercard supercard, toolbook, authorware, guide... microcosm . . . ; table 4 shows the placing of some of the many different interactive multimedia environments onto the same framework as earlier. \ {the right job for the tool? learning style can be considered as a predisposition to adopt a certain kind of behaviour 14 alt-j volume 3 number i when engaged in the learning process. for example, table 5 shows an overview of learning modes as suggested by kolb (1984). table 5: overview of learning modes (kolb)] m o d e concrete experience reflective observation abstract conceptualization active experimentation focus involvement in experiences understanding the meaning of ideas and situations using logic, ideas, concepts actively influencing people and changing situations emphasis feeling as opposed t o thinking understanding as opposed to practical applications thinking as opposed to feeling practical application as opposed t o reflective understanding this behaviour is influenced by the ways that learners process information, and how they react to certain learning situations and environments. this will result in certain preferences and tendencies to adopt particular habits or patterns, known as learning strategies. these patterns or habits may be liberating or inhibiting, depending on the learning situation, and therefore an understanding of the learning style of the learner is desirable, both for the learner and for the author of learning materials. once this information is gained, guidance and advice on the adoption of the most appropriate learning strategies may be formulated. there are several components of learning style, usually categorized as cognitive, affective or environmental (e.g. smith, 1983; kolb, 1984; coates, 1991). the cognitive component encompasses the internal processes of learning. the affective component pertains to the feelings of the learner, and as such includes aspects such as the measure of structure and authority within the learning situation, which affect the amount of autonomy the learner has, and the expectations and motivation of the learner. the environmental component ranges from physical considerations of comfort to more abstract influences resulting from emotional support. the adoption of a particular learning style may be influenced by the learning task in progress, or by psychological predisposition, or both. where this produces a conflict, it may be reasonable to assume that the learning process is impaired. learning strategies are the particular habits or patterns espoused when engaged in the learning process. the proficiency of different learners in learning, reasoning and problemsolving varies widely, and so do the strategies they evolve to carry out these activities. hayes (1985) raises the possibility that there may be several hundred plausible learning and thinking strategies, and identifies at least fifty different strategies that he himself presents in a basic learning-strategies course for students. many researchers (e.g. hartley & davies, 1976; rigney, 1978; tessmer & jonassen, 1988; coates, 1991) have examined 15 pat paterson and john rosboffom learning style and learning strategies f 1 secondary j j v . : 1 meta-leamlng i / \ ( primary "| x , , 7 1 information processing j \ . , jc cognitive \ . \ v afledive environ/ íliéma} • 7 ( active study ̂ / primary y? '.' | preparation/execution | s* y secondary j ^ s / figure i: relationship between components of learning style and primary and secondary learning strategies the field of learning strategies, which has emerged as part of the cognitive science revolution in instructional technology. successful learning strategies enable learners to manage their own learning process and integrate new learning into their own existing cognitive structures. unsuccessful or inappropriate learning strategies may inhibit learning (pask, 1976; ford, 1985). although there is general agreement on what learning strategies are (how one uses one's head when learning) and what they do when successful (help one to learn), the way they are viewed varies considerably. nevertheless, despite the large number and wide variety of learning strategies, it is possible to group them according to how they affect the learning capacity of the learner. the major grouping is into primary strategies, which act on the information to be learned, and secondary strategies, which provide support for the learner. primary strategies include information processing strategies, which have more influence on cognitive aspects of learning style, helping to enhance mental organization 16 alt-j volume 3 number i and recall of material, and active study strategies, which often involve physical activities such as note-taking and seek to improve the affective and environmental components of learning style. secondary strategies help the learner to establish a suitable mental and physical state for learning and to monitor progress as learning proceeds. these secondary strategies may be considered as meta-learning strategies and preparation/execution strategies (tessmer & jonassen, 1988). the relationship between the cognitive, affective and environmental components of learning style, and the broad groupings of learning strategies which may be favoured is shown in figure 1. some experimental results a preliminary study has been.carried out in order to identify some of the links between learning style, learning strategy and the use of interactive multimedia as a tool for learning. the experimental group consisted of course members for the msc in information systems, which is a conversion course for students with little background in it, but a wide range of non-it experience. kolb's learning style inventory (kolb, 1984) was used to investigate the learning style of each student. learning strategy for each was broadly assessed by means of the single question identified by clarke (1993) as a possible indicator of serialist/holist tendencies. the application used as a vehicle for the study was an interactive guide to the use of a complex telephone console (paterson, 1990). four different navigational tools were available. these were: • representational: a graphic of the console and keyboard. • schematic: a hierarchical map showing the structure and subdivisions of the information available. • descriptive: a verbal list of information about keys offering more detail than the map bot reflecting its structure. • menu: a pull-down menu listing the high level subdivisions of the information. students were allowed to browse the application freely until they were confident that they understood its structure. they were then given a question sheet which grouped questions under six named section headings and asked that these sections be searched to find a total of sixteen specific pieces of information. no time restrictions were applied, but all students completed the work to their own satisfaction within one hour. an analysis of the total number of pages visited during the search for information was carried out so as to establish the potential validity of the data. for 28 subjects, the range was 70 pages (13 min, 83 max), the mean was 47.93, mode was 55.00 and median was 47.50. it was therefore decided to exclude the lower quartile (<33 pages) from analysis of preference for navigational tools on the grounds that the search for information was too cursory to produce a meaningful pattern. 17 pat poterson and john rosbottom learning style and learning strategies analysis of paths taken through the interactive-multimedia application shows distinct preferences by different users for different navigational tools. representational (graphic/picture of telephone console), schematic (map), descriptive (verbal listing of keys on console) and menu (pull-down list) tools were all freely available. in order to compare preferences, ratios of visits between pairs of tools were calculated from the original totals of visits. boxplots were made of the ratios console/keylist, console/map and keylist/map (figure 2). f/gure 2: boxplot to show comparisons between preferences for navigational tools the horizontal line at ratio = 1 represents equal preference. it can be seen that the console was clearly preferred over both the keylist and the map, and that there was a slight preference for the keylist over the map. this result was unexpected, since the most commonly provided navigational tool is usually the map. . as described above, it was decided to exclude the lower quartile (<33 pages) from analysis of preference for navigational tools. this raised the question as to whether there was any discernible consistent reason for the cursory nature of the search as performed by these subjects. scatter plots of combined learning-style scores obtained form the kolb analysis were plotted against number of pages searched, and the result of the active-reflective versus pages searched may be significant (figure 3). it can be seen that for all subjects whose searches were considered cursory, their 18 alt-j volume 3 number i a c t i v e r e f i e c t i v e 10-r 8-> 6'4 " 2 " 2 4 " -6.. 8 . . -1o-14. 10 h 20 30 40 5 0 60 70 80 90 number of pages searched figure 3: scatterpbt of combined active-reflective learning style (kolb) and number of pages searched orientation towards reflective observation, which emphasizes understanding as opposed to practical application, outweighs their orientation towards active experimentation (that is, their combination score on the active-reflective axis is <=0). the relevance of this is a matter for conjecture, and needs much more investigation, but it might correspond to a reluctance to delve too deeply into an unfamiliar computer environment such as interactive multimedia. a further exploration of the data suggested an investigation of correlation of learning strategy with preferred navigational tool. a snapshot picture of serialist/holist tendency for each student was obtained using the following question (clarke, 1993): when i'm reading a book (or other information source) for my studies, i prefer to spend quite a long time skimming over and dipping into it to get a clear picture of what it's about and how it will be relevant when i'm reading a book (or other information source) for my studies, 1 2 3 4 5 i prefer to get quite soon into a fairly detailed reading of it once i know it's going to be useful, in the knowledge that its precise relevance and contribution will become clear from a detailed reading while plots of console/keylist and console/map ratios showed an even spread across breadth-first depth-first strategies, the keylist/map ratio (which indicated no real preference for either tool see figure 2) was more interesting. figure 4 shows the keylist/ map ratio plotted against breadth-first/depth-first learning style preferences. 19 pat paterson and john rosbottom learning style and learning strategies k 4 + e y i 3+ i s t 2 t m a of p -1, breadth first depth first o 1 learning strategy figure 4: plot to show relationship between keylist/map ratios and preferred learning strategies the keylist is a descriptive tool which offers more detail than the map, and seems to be preferred by those who adopt a global approach to problem solving. since a map is usually assumed to give the best overview, this result was somewhat unexpected. however, it may be that on this occasion the detail given in the keylist gave the better overview, and that the map encouraged a more linear approach. future work clearly, this preliminary exercise has produced more questions than it has answered. the study is being extended to include field dependence/independence tests (witkin et al, 1977) and further experiments are in progress. it is possible that for some students, the increased use of interactive multimedia as a learning tool may be a potential handicap instead of the expected advantage. references clarke, j. (1993), 'cognitive style and computer assisted learning: problems and a possible solution', association for learning technology, 1, 1, 47-59. coates, m. (1991), open teaching toolkit: learning how to learn, milton keynes, open university. ford, n. (1985), 'learning style and strategies of postgraduate students', british journal of educational technology, 16, 1, 65-77. 20 an-) volume 3 number i hartley, j. and davies, i.k. (1976), 'preinstructional strategies: the role of pretests, behavioural objectives, overviews and advance organisers', review of educational research, 46, 2, 239-65. hayes, j. (1985), 'three problems in teaching general skills' in chipman, s.f., segal, j.w. & glaser, r. (eds), thinking and learning skills, vol 2, research and open questions, new jersey, lawrence erlbaum, 391-405. jonassen, d.h. (1992), 'implementing cognitive tools' in kommers, p.a.m., jonassen, d.h. & mayes, j.t. (eds), cognitive tools for learning, nato asi series, f81, berlin, springer-verlag, 249-50. kolb, d.a. (1984), experiential learning, new jersey, prentice-hall. kommers, p.a.m. (1992), 'microworlds: content dependent cognitive tools', in kommers, p.a.m., jonassen, d.h. & mayes, j.t. (eds), cognitive tools for learning, nato asi series, f81, berlin, springer-verlag, 249-50. pask, g. (1976), 'styles and strategies of learning', british journal of educational psychology, 46, 128-48. paterson, p. (1990), development and evaluation of a hypermedia training manual for the islx operator console, msc dissertation, university of portsmouth. rigney, j.w. (1978), 'learning strategies: a theoretical perspective', in o'neil, h.f. jr. (ed), learning strategies, new york, academic press, 165-205. smith, r.m. (1983), learning how to learn, milton keynes, open university. tessmer, m. & jonassen, d. (1988), 'learning strategies: a new instructional technology', in harris, d. (ed), world yearbook of education 1988: education for the new technologies, london, kogan page, 29-47. witkin, h.a., moore, c.a., goodenough, d.r. & cox, p.w. (1977), 'field-dependent and field-independent cognitive styles and their educational implications', review of educational research, 47, 1, 1-64. 21 new technology for interactive cal: the origami project robert harding * steve lay* peter robinson** dan sheppard** and richard watts** *department of applied mathematics and theoretical physics, university of cambridge **computer laboratory, university of cambridge origami is a three-year epsrc project that forms part of a general research programme on humancomputer interaction. the goal of this research is to investigate and implement new methods for human-computer interaction, and to apply and evaluate their use. the research centres on the digitaldesk, an ordinary desk augmented with a computer display using projection television and a video camera to monitor inputs. the digitaldesk allows electronic and printed documents to be combined to give richer presentation and interaction possibilities than are possible with either separate medium. this paper examines the implications of such a system for cal, and presents two prototype applications that demonstrate the possibilities. introduction for many years, almost since the dawn of computing, people have speculated about the paperless society. each new technology is heralded as the one that will finally take us to the fully electronic utopia. however, the use of computers as document preparation tools has actually increased the number of paper documents. the latest mass-market development, the internet, is unlikely to buck this trend even though the tools used seem to encourage on-screen browsing. the problems inherent with screen presentation have not gone away, while the amount of information available to the average computer user is increasing all the time. education is no exception; students too are encouraged to create documents as part of more active learning programmes, building an awareness of the importance of presenting their own ideas to others as well as helping to meet targets for the integration of technology into teaching. but cal is about more than just having better tools to create more attractive documents. 6 air-j volume 5 number i the arrival of multimedia-capable computers has provided an opportunity for the integration of new media types into documents. these media types range from sound, video and typed text to fully interactive computational tools such as virtual worlds. however, multimedia technology has its problems: (i) video frame rates and resolution are often low; (ii) audio quality is often poor; (iii) text is hard to read from computer displays; (iv) the document must be viewed with a suitable computer. these are problems of perception, rooted in our experience of existing single-medium documents. we have got used to broadcast video, cd audio, good-quality colour imaging and high-resolution text rendering. it seems likely that the advances being made in software and hardware will fairly quickly remove the barriers presented by (i) and (ii) through better compression and more bandwidth. the typical display and speaker system of a modern multimedia pc are already good enough to deliver the fruits of these advances. on the other hand, solutions to (iii) and (iv) do not appear to be just around the corner. this threatens to divide multimedia documents into two types: those that are delivered in a virtual world (by computer) and those that are delivered in the real world (by paper). the typical solution to this division is to create a metaphor which reduces the gap between the real and virtual worlds hence the desktop metaphor. many cal experiments have involved placing material which is available to the student in printed form online in an attempt to take advantage of the added features that a computer text can have for example hypertext links and search engines for navigation as well as using this as the starting point for adding multimedia content to the original publication and linking it to computer-based instructional resources such as self-assessment engines. see rush (1995) for a recent example. digitaldesk the philosophy of the origami project is that a significant improvement to this situation can be made by using a human-computer interaction technique known as computeraugmented environments (robinson, 1995) to allow publications which sit uneasily in the computer domain to have the best of both worlds. rather than create virtual books, we seek to unite the value-added functionality of computer-based texts to real paper documents and ultimately to the student's working environment itself via the digitaldesk (wellner, 1991; wellner, 1993). in the 1970s, xerox parc developed the desktop metaphor which made computers easier for many people to use by making them look and act like ordinary physical desks. electronic documents could be manipulated like paper documents, but the computer added powerful new capabilities. recently, a joint project between the computer laboratory at the university of cambridge and rank xerox europarc resulted in the development of brightboard (stafford-fraser and robinson, 1996), a computer-augmented whiteboard, and prototype digitaldesks (wellner, 1994). instead of making the 7 robert harding et a\ new technology for interactive c a l the origami project computer active paper desk surface figure /: the digitaldesk is built around an ordinary physical desk figure 2: the anatomy of trie digitaldesk computer act like a desk, the digitaldesk makes the desk act like a computer. with a digitaldesk, the user can do things with ordinary paper that normally can be done only with electronic documents on a screen. the digitaldesk is built around an ordinary physical desk and can be used as such, but it has extra capabilities. a video camera is mounted above the desk, pointing down at the work surface. this camera's output is fed through a system that can detect where the user is pointing (using an led-tipped pen) and it can recognize the documents that are placed on it. the more advanced version also has a computer-driven projector mounted above the desk enabling electronic objects to be projected onto real paper documents, thus removing the burden of having to switch attention between screen and paper and allowing additional user-interaction techniques. it will be some time before such systems become widely available, but workstations with built-in support for video input have been available for many years and have now even found their way into the personal computer market. • the techniques employed by the digitaldesk take advantage of the fact that most written materials are already created in electronic form. sadly, much of the richness of this form is thrown away when the document is printed, but the digitaldesk attempts to utilize this extra information to augment the paper copy. educational applications the digitaldesk is an ideal way to advance the computer-illustrated text (cit) concept (harding and quinney, 1990) which sought to combine the strengths of the traditional printed textbook with those of the computer as an educational medium. the earliest cits were printed books with software, the use of which was an integral part of the learning strategy of the package as a whole. they covered subjects in mathematics, computing and 8 aa-j volume 5 number i physics, areas where screen presentation is particularly difficult but where interactive graphic displays and dynamic computation can considerably enhance the learning activities. of course, computers have come a long way since that series was published. the renaissance project (harding et al, 1995) experimented with electronic delivery but more recently calculus connections (harding and quinney, 1995), which involves video-clips, sounds and sophisticated computer animation, is still centred on the printed course book. when graphical user-interfaces were first introduced to desktop computers they were principally aimed at document generation. in order to help the user transfer existing skills in handling paper documents to the computer, the desktop metaphor was invented. an integral part of the resulting systems was the printer which allowed the user to transfer the finished document back to the real world, throwing away properties (such as the ability to edit) which were no longer needed. although document generation is an important part of educational computing, the real explosion in computer use has come from the ability to interact with multiple media types, to gain access to information resources that exist outside the user's immediate 'desktop', and to participate in new styles of communication. as before, real-world metaphors have been applied in order to assist the use of these new applications. however, the problem with these new media types is that printing them causes them to lose properties which are still useful. the digitaldesk is effective for these applications because its purpose is to retain these useful properties and, where possible, to add them to documents which were not even created on the computer. working prototypes to illustrate the digitaldesk's contribution, two illustrative scenarios are presented here. these scenarios represent possible uses of software that has already been developed for the digitaldesk as part of the current origami project. the first covers a broad range of computing applications and is used to set the scene for the second, an example aimed specifically at education. augmenting paper documents consider a user at the digitaldesk browsing the world wide web, with the current page projected onto the desk surface in front of her. she selects links to follow by pointing with a special 'active' pen and pressing a button located near the tip. having discovered a page of interest, she uses the projected print button to create a hard-copy with all the familiar and desirable qualities of paper: readability, tactile sensation and portability. she takes this paper copy away with her to read on the way home. the next day she returns to her desk with the paper copy of the page she printed the day before, marked up with her own notes. when she places it on the digitaldesk it is recognized as the page that was printed the day before, suitable browsing software is launched and the same set of buttons that are available when browsing the electronic version are projected onto the header of the paper version and the links automatically become active. she can then continue to work with the document. computer-aided assessment now consider a mathematics student working with his course book at the digitaldesk. the software which accompanies the course book is automatically launched when the book is 9 robert harding et al new technology for interactive cal the origami project the expresses ax* * tx* e. where a, b mi e ere conitntt. u caoed • q function of xthe bigheit power of the vtriifcle x ii the tecond. iffuch m ezpreoic if plotted tgiiiut x. the resulting curve wil hare «ne of the * tupet shown below: •.positive a.fiegtfm the tign of a determtnei which w>? «p the carve i*. considerthe eqa«ion y = x 1 3x + 1 titnlttiat ac » * » « for j»for »ehiei of x between 0 and 4 gives the following titoe*: xo 1 2 3 4 y20 0 2 6 wh«t bvpeni between z l«rf x "2? caml*iftgj> for j 1h » r » « w. we cm now me thwe *4uef to ntke • sketch of the carte. note that the vatuet of x m which the curve crone* the x-«xii we unptj die potati which y = 0. they »re therefore fce tohitioiu of the quadrtbc eqntfioa: l . *" v : • • • figure 3:7?ie d/g/to/desk brings paper publications to life first placed on the desk. during a section on curve-sketching, he reads about polynomials. the generic equation of a quadratic polynomial is given with spaces for the values of the coefficients and an empty box underneath for plotting the graph. the software projects default values and draws the graph into the box. however, it also projects controls alongside the coefficients to allow him to change these values while observing the corresponding change in the graph (perhaps in response to a rhetorical question like: what happens if you change c?) on the next page of his course book, he is faced with an assessment exercise. this time the polynomials are fixed and he must draw the curve into the box (the active pen does of course have a real nib for writing). when he has finished, he clicks a projected button to ask the computer to assess his sketches. the sketches are analysed for features such as maxima, minima and axis crossings, and marked accordingly. future developments hardware and software have already been developed to meet the requirements of the above scenarios. for the future, the digitaldesk provides exciting possibilities for the development of educational computing projects. however, it is a research project; it will be some time before computer-augmented environments make their way onto the average laboratory desktop. ten years ago, the same could have been said for desktop video. in the meantime, when designing applications, it is useful to break down the technology into several levels of increasing technical sophistication: • printed materials only; • printed materials augmented by traditional tools such as pen and paper, hand calculator, etc.; 10 alt-j volume 5 number ) • printed materials augmented by a computer system (with keyboard and pointing device); • printed materials augmented by a computer system fitted with a page-recognition device (possibly driven by a hand-held device such as a simple bar-code reader); • printed materials augmented by a computer system with page recognition and a pen/finger-driven locator; • printed materials used in the context of the fully featured digitaldesk with page recognition, locator tracking and projection. materials that are flexible enough to be used at any of these levels would be useful even if everyone had access to a fully functional digitaldesk, since there are bound to be times when the complete system is not available. by designing computer-aided software packages with this in mind, it is hoped that a greater continuity can be provided between the student's experience in the computer laboratory and his or her own private time spent revising and reflecting on the subject matter. acknowledgements the original digitaldesk was built by pierre wellner, and brightboard by quentin stafford-fraser (1996), both research students sponsored by rank xerox in the computer laboratory at the university of cambridge. current work on animated paper documents is sponsored by the epsrc under grant gr/j65969. references harding, r.d. and quinney, d.a. (1990), 'computer illustrated texts (cits) for teaching numerical analysis', computers and education, 15 (1-3), 205-11. harding, r.d., lay, s.w., moule, h. and quinney, d.a. (1995), 'multimedia interactive mathematics courseware', computers and education, 24 (1), 1-23. harding, r.d. and quinney, d.a. (1995), calculus connections, chichester: wiley rush, d.g. (1995), 'do we need hypermedia and books in engineering education?', proceedings of hypermedia in sheffield '95, university of sheffield, 284-90. robinson, p. (1995), 'virtual offices', proceedings of royal society discussion meeting on virtual reality in society, science and engineering, bt publication srd/r5/1, july. stafford-fraser, q. and robinson, p., (1996) 'brightboard a video augmented environment', proceedings of the acm conference on human factors in computing systems, vancouver, april. stafford-fraser, q. (1996), 'video augmented environments', ph.d. dissertation, university of cambridge computer laboratory. wellner, p. (1991), 'the digitaldesk calculator tangible manipulation on a desk-top display', proceedings of the acm symposium on user interface software and technology, hilton head, november. robert harding et al new technology for interactive cal the origami project wellner, p. (1993), 'interacting with paper on the digitaldesk', communications of the acm, 36 (7), july 1993. wellner, p. (1994), 'interacting with paper on the digitaldesk, technical report 330' (ph.d. dissertation), university of cambridge computer laboratory. 12 design issues in the production of hyper-books and visual-books nadia catenazzi*. monica landoni* and forbes gibb** * environmental informatics unit, tp.440 cec joint research centre, 21020 ispra (va), italy ** department of information science, university of strathclyde, ljvingtone tower, 26 richmond street, glasgow g1 1xh abstract this paper describes an ongoing research project in the area of electronic books. after a brief overview of the state of the art in this field, two new forms of electronic book are presented: hyper-books and visual-books. a flexible environment allows them to be produced in a semi-automatic way starting from different sources: electronic texts (as input for hyper-books) and paper books (as input for visual-books). the translation process is driven by the philosophy of preserving the book metaphor in order to guarantee that electronic information is presented in a familiar way. another important feature of our research is that hyper-books and visual-books are conceived not as isolated objects but as entities within an electronic library, which inherits most of the features of a paper-based library but introduces a number of new properties resulting from its non-physical nature. introduction the evolution of publishing tools from movable print to digital typesetting controlled by computer has been principally aimed at improving document quality and lowering unit costs rather than responding to readers' needs. and a by-product of these improvements is the production of enormous quantities of printed paper which are never read and which have limited justification in being defined as information. although the computer is largely responsible for this information glut, it can and should be turned to assist the reader by providing a consultation environment alongside the production system, where information can be presented in a more attractive and more functional way than on paper. to ensure this result, it is necessary to provide some conversion tools which are able to translate information acquired from different sources into an electronic environment, 40 alt-j volume 1 number 2 and at the same time to offer a powerful and meaningful paradigm for information representation. the result is an object called the electronic book, a generic term which is becoming widely used, and which encompasses a large family of implementations where the features of a conventional book are mapped into an electronic system to various degrees. the route followed by the authors of this paper is to maintain as closely as possible the familiar metaphor of the book. the result is two complementary proposals, hyper-books and visual-books, both based on the same philosophy but with specific features related to the type of original documents they manage: electronic texts and paper books respectively. electronic books a working definition of an electronic book has been given by barker (1991) as 'a collection of pages of electronic information which can be displayed on a suitable viewing screen.' however, this definition is very general and encompasses various interpretations. our approach to electronic book design is strongly related to the book metaphor. books are the traditional repositories of information. people know how to read them, how to use a table of contents, how to use an index, etc. by maintaining the same model on screen, people's access to electronic information can be facilitated. instead of formulating complex queries which are used to retrieve books based on, for instance, a document surrogate such as an inverted file, a representation of the book itself is available which can be consulted like a physical book. this approach helps to overcome some of the limitations inherent in electronic support, such as the loss of the physical aspects of the book (for example its thickness) which can be used as an orientation clue. nevertheless, there is another important group of difficulties which are connected to technological shortcomings, mainly the limited size and resolution of display screens, that makes reading an electronic book less pleasant than a paper book. design issues the communication of knowledge is a basic requirement for all types of books, and particularly for electronic books where methods of information and knowledge delivery are different from those used in conventional paper books. essential to electronic book design is an understanding of user profiles and of the purpose of the book, which determine the choice of interaction strategies and types of interface. it is also important to consider the design processes used for paper books as possible guidelines for electronic-book production, while recognizing that the different nature of electronic books makes it necessary to develop specific procedures. at least two different levels of analysis have to be considered, as suggested in barker and manji (1991): a basic one which deals with features related to hardware, software and interfaces; and a conceptual one which considers the user's point of view. at the first level of analysis, the attention of the designer is focused on practical aspects of the system, such as the study of interface features and media involved in book representation, and the need for various design templates. in conceptual design, models and paradigms for book representation are considered in order to provide an environment where the user can interact with the information in a natural way. particular attention is given to the design of empty templates for books and pages which the author will fill in following a relatively simple process. the conceptual design phase also provides a justification for the choices made in the 41 nadia catenazzi et at ' design issues for hyper-books and visual-books interface design. in this context, it is useful to distinguish between various categories of electronic books, because different books have different design requirements. there are different ways of classifying electronic books according to different criteria. barker (1991) defines a taxonomy of electronic books based on two criteria: the nature of information, and the types of services they are able to provide. collis (1991) classifies electronic books as reference books and text books, a distinction based on different usage. another important criterion for classifying electronic books is based on the kind of presentation offered. each presentation is dependent on a particular interpretation of the concept of the electronic book. these categories are not necessarily exclusive. there is a large group of electronic books, called portable electronic books, whose main purpose is to reproduce portability (feldman 1990; lande 1991). in fact, one of the most appealing features of paper books is the possibility of carrying them everywhere and reading them comfortably. the electronic equivalent has many drawbacks: limited screen size, limited resolution, etc. at the moment, portable electronic books have made their most significant impact in the area of reference publications such as dictionaries and thesauri. a further group of electronic books is more focussed on the logical structure rather than on the physical nature of the book: information is usually displayed in scrolling windows, which may overlap, and the idea of the page as a static block of text is usually not considered. this is the case with dynatext (ebt, 1990) and superbook (egan et al, 1989). both of these provide full-text indexing, links, navigation and orientation through a dynamic table of contents, and a multi-window text display. an interesting aspect of these two systems is the fact that they provide the capability of automatically importing electronically available text finally, there is a group of electronic books which emphasizes the physical aspects of the book but also considers its logical aspects. this, in our opinion, is the most complete and satisfactory solution because it considers logical structures together with physical clues (for example, pages, book thickness), both of which are indispensable for recognizing an object as a book. this approach becomes particularly important when a paper version exists of the same document and readers are familiar with it. benest (1987; 1990) presents interesting examples of this kind of electronic book; a book is presented as an open document in which pages may be turned and flicked through. the thickness of the book is indicated by an active black band along the side and the bottom of the book. the text page may be annotated and highlighted. book markers also may be inserted. vortex (burril, 1989) is another example of an electronic book in this category, but one which proposes different solutions. the most original idea here is. concerned with how to solve the problem of providing orientation facilities within the book. in order to indicate the reader's position in the book, the solution adopted in vortex is the idea of closed pages. the text is displayed between two sets of vertical lines which represent the thickness of pages before and after the current page. section names are indicated between these lines and may be selected in order to go directly to that page. electronic books in the context of electronic libraries the increasing interest in electronic books and the availability of practical examples of them indicates that using them could become a common practice. from this perspective, an obvious 42 alt-j volume 1 number 2 and very useful application is in the context of an electronic library which could be seen both as a personal library (for organizing personal collections of electronic books), and as a public library (for delivering the full text of documents online). the library could be accessed remotely from any location through a network using personal or portable computers. electronic books do not suffer from the constraints of their physical counterparts, and can be easily replicated as many times as required. the library itself has infinite space: there is no limit to the number of books it can contain. many of the operations normally performed in physical libraries do not exist in electronic libraries (binding and preservation, shelf maintenance etc.), although there may be new administrative functions. the computer gradually replaces the librarian's role as intermediary between users and the library. another player, the information manager, emerges in the electronic library environment; (s)he is responsible for the development and the maintenance of the system, and uses expertise drawn both from library and information science. when organizing electronic libraries, most of the technical operations performed in traditional libraries have to be reconsidered. one of the main issues is the acquisition and processing of material. material can exist in different formats, and conversion tools are usually essential parts of the acquisition process. another issue is concerned with the cataloguing of the material in order to store it in the library database. several issues are concerned with public services: how to provide copies of documents; how to display them; how to inform users; how to assist users; how to manage search and retrieval taking into account gateways to other library systems; how to manage loans, which can be interpreted as temporary provision of copies or free-copy delivery according to the system strategy; and how to protect copyright. two new forms of electronic book. the authors of this paper are currently working on a research project whose purpose is the design and development of an environment for building electronic books. referring to the classification above, our electronic books belong to the final category, as both the physical and logical aspects of the book are taken into account. this choice has been based on our view of user-needs (readers need to consult a book on the screen and read portions of it, not just find an entry in a dictionary) and of our selected application area (scientific publications). for that reason, large screens (at least 19 inches) are an important feature in the design of our electronic books. at the moment, low resolution is one of the main obstacles to acceptance of reading electronically displayed text, but technological developments will ensure that this barrier will be removed within a short period of time. our electronic books are conceived not as isolated objects but as integral parts of an electronic library, and some of their properties are dependent on this feature. users typically find a book after a bibliographic search or browsing through library shelves. they then consult it in order to see if it is what they are looking for and, if the book seems to be interesting, they borrow it. our electronic books are designed to allow this kind of approach: browsing, followed by personalization, and finally printing in hard copy. another important aspect of our research is that in addition to the interface-design issues, a flexible publishing environment is provided in order to produce electronic books in a semi-automatic way. more specifically, the design of two different environments is involved: the first consists of a set of tools which allow the conversion of existing electromc texts into 43 nadia catenazzi et al design issues for hyper-books and visual-books electronic books (hyper-books); the second aims to define tools for the conversion of paper books into their electronic counterparts (visual-books). hyper-books more and more information is now available in electronic form, but it is not presented in a natural and user-friendly way. the hyper-book proposes an alternative for representing electronic text which overcomes the problem of limited familiarity with text representation, and defining an environment where electronic books are generated using electronically produced text. an important feature of such electronic books is the possibility of accessing text in a non-linear way, following the hypertext philosophy. it is obviously for this reason they are called hyper-books. the environment where they are produced is known as a hyper-book builder. hyper-books maintain many of the features of paper books and present information in a format which is as close as possible to the paper book format. in addition, other capabilities can be provided such as indexing, links, and history mechanisms. a hyper-book is presented as an open document with the two pages side by side in view. the page contains different components: the footer, the header, and the text itself. the text may contain notes, pictures or other non-textual components. a different font style is used for distinguishing active areas of the text (i.e. links). the hyper-book supports several book-control functions such as turn page and close book, and a number of reader tools: orientation, personalization, searching, history, navigation and printing facilities, which are described in turn below. in addition, a number of editor tools are provided (see figure 1). these are used during the building process of hyper-books, but are not visible when the hyper-book is made available to the reader in its final version. ha change font chaste size ncvefieh bender sereitet | v h e n end v h e n they need them. the library is conceived as • service, not just as a place [ m u £ 9 ] d r a v a r e h o u s e v h e r e f e l f l } m r i a n b only * custodian. the library services include providing infbxmatioii, cfaculaiing and netvoiking documents to other tibiaries and users. nowadays many hhrarjes use electronic suppon. h o v e v e r , the global automation of me ttnaiyi&clodmc the delivery of fan t e n o n line to stffl in an experimental phase. this report illustrates the main features of an electronic library, through the evolution of the paper waary a w a r d s me fun text eiecteonic ĵ dxun̂ . s o m e projects deajin? v t t h fan text electronic libraries v f l l be 2. library automation in the past, kbiaxfes v e x e places v h e r e documents v e i e preserved and coseccd [mcg91]. the access to the material v a s restricted to a small p o o p of people. the concept of fte ifbxaxy as a senrice fox a mmwmnjiy of 03 cts cmeiged later. t v o factus v e m responsible: the invention of printing movable v p e , v h i d i adoved multiple coptes of a document to be produced and the fact that more people v e i e literate. therefore, the library as ttb today to the result of a literacy process v h f c h causes a n increasing need t ot a service v h i c h provides i&fonnsuoil |jirip^pftftx jtp^qj o f b o r a x i e s e x i s t ^^itti figure 1: the hyper-book template: designer and reader services 44 alt-j • volume 1 number 2 the obvious method of providing orientation in a book is the table of contents. in addition, paper books use running headings and the thickness of the paper as orientation clues. some electronic books show the table of contents at all times (egan et al, 1989; catenazzi, 1991) with a marker on the current position. however, this solution does not respect the book metaphor. hyper-books provide a table of contents only in the preliminaries, with the possibility of jumping there by clicking on a button from any part of the book. many approaches have been adopted in electronic books for indicating the user's position within the book, such as the closed pages (burril, 1989) and black bands (benest, 1990) mentioned above. hyper-books provide black bands as well as running heads. a black band, simulating the thickness of the book, is composed of vertical and horizontal lines along the bottom and outer sides of the book, proportional to the number of pages currently opened. it is possible to make large jumps in the book by selecting the black band in a specific place. hyper-books provide several kinds of personalization: annotation, highlighting and bookmarks. these have the advantage that they may be superimposed on the page without altering the original copy of the book for another user. the types of annotations provided by hyper-books are essentially typing and graphic annotations. in addition, it is possible to write notes in an overlapped window associated with the text. a bookmark is displayed as a small rectangle inserted on the top side of the book, with a number indicating the page where it has been created. bookmarks may also be removed. another opportunity offered by hyper-books is the possibility of copying pieces of text and saving them in a personal clipboard. as far as searching facilities are concerned, the index and table of contents represent the two main methods of finding useful material in a physical book. hyper-books provide both of these, as well as full text indexing. the result of a search using full text indexing is visible in the main body of text through highlighted search terms, as well as in the table of contents as the numbers of search-term frequencies for each entry. this approach guarantees that contextual information is always available (egan et al, 1989). when consulting electronic books, it is useful to remember the path followed during the reading process. this can concern all visited pages or only those which have been marked with bookmarks. hyper-books provide history mechanisms for both cases. there are essentially two ways of navigating in a book: hierarchically (from the table of contents down to the units of text) and transversally (references across the text). hyper-books provide both these methods. transversal links include internal links (reference across the document) and external links (links across the library). the third way of moving in the text is the classical linear way. the hyper-book offers the possibility of going back and forward through the pages of the book, as well as flicking through pages. the two pages currently in view, or selected parts of the book, may also be printed in hard copy. all the typographical information of the hyper-book is preserved in the paper copy. not all the facilities described above are necessarily provided to the reader. most depend on the choices made during the building phase, to which we now turn. hyper-book builder the hyper-book builder is an environment in which hyper-books are produced. two activities are basically supported: design of the hyper-book interface (designer's role) and interpretation 45 nadia catenazzi et al design issues for hyper-books and visual-books and presentation of electronic texts according to that interface. hyper-book production presents many analogies with the production process of a paper book. essential players in the production of paper books are author, editor, designer and publisher. the roles which are mostly reproduced in this context are that of the designer and that of the editor. the text is assumed to be in its final version, and the author is not considered, the editor and the designer collaborate to produce the definition of typographical aspects. main issues related to the hyper-book builder an electronic text may be available in several formats, the simplest being a flat ascii text. however, there are many programs for writing and manipulating text, each of which stores it in a specific format using a markup language. markup gives information about the logical structure and/or the physical appearance of a document. markups can provide useful information when automatic treatment of the document is required, and it should therefore be exploited in the process of translating the electronic text into the electronic book. each markup has specific features, and different translation rules have to be defined for different markup languages. in order to realize our first prototype hyper-book, sgml (standard generalized markup language) was chosen as the markup language. sgml offers a number of advantages over other markup techniques such as oda, interleaf or scribe, or specific markup languages used by word-processors. sgml was accepted as an international standard (iso 8879) by the international organization for standardization in 1986. macleod (1991) describes it as a system for specifying markup languages rather than as a markup language itself. it allows for the definition of different types of documents, so that a document type definition (dtd) formally defines which elements are allowed in a document and how they are organized. the language is deviceand application-independent (feeney, 1989). only the logical structure is indicated by the markup, and the same document can be presented in many different ways. this represents an additional advantage of using sgml when, for instance, the same document has to be accessed on different machines. sgml is thus a standard and has all the advantages of an open system, which solves the problem of exchanging electronic documents, something which is extremely important in the context of implementing an electronic library. another advantage of using sgml is the availability of ready-developed software for handling it. there are sgml parsers, which verify that a document conforms with its dtd. translators also exist from other formats into sgml, and the system can be easily expanded. many initiatives and projects are promoting the use of sgml in the usa and, more recently, in europe. there are good possibilities that it will be more and more widely used. as for design principles, considered in terms of types of interface, styles of interaction and facilities and services provided to the users, these may be partially deduced from traditional book design principles and from contemporary experiments with electronic books. many aspects of paper book design are also valid for electronic books, although some constraints on the one hand, and more capabilities on the other, contribute to create a very different object. the size of the book, for instance, is constrained by the screen dimensions: the font size should be bigger on screen than on paper in order to be legible. the study of book design issues is particularly important, because the hyper-book builder has to provide a number of empty templates. these are pre-formatted empty book structures and pages which have to be filled in by the text, as will now become clear. 46 alt-j volume 1 number 2 using the hyper-book builder hie purpose of the hyper-book builder is to provide an environment where the user (at this stage the user is the creator of the book) can build his or her hyper-book through a set of tools described below. a hyper-book is generated by starting from empty templates and a text in sgml or ascii form. a template is a model of a hyper-book; it represents it in terms of structure, format and reader services. the contents are not included. the term structure always refers to the logical structure. the format is relative to the presentation of the text and to its appearance on the screen. the structure of the hyper-book is determined by the dtd (document type definition) if the input text is in sgml; no structure is defined for ascii texts. the format specifies a number of layout features: book size; page size, margins, footer and header; typefaces to be used in different parts of the book; notes; link activation process; design of specific pages (title page, contents page, index, references, etc.). the reader services include a number of facilities, as described above (orientation, personalization, searching, etc.) which, during the book generation phase, may be defined by the book designer (editor) for the reader. in figure 1, the editor panel contains a list of services that the designer can select for the reader. once a service has been selected, corresponding buttons appear in the reader tools panel area. a number of predefined templates are provided, while a set of control functions enables the user to modify some aspects of the book format and also some reader services, taking into account the reader and the application features. in order to generate hyper-books, these empty templates are filled in by the text. the book can then be further modified. different steps are involved in the process of generating hyper-books. the first consists of the automatic translation of sgml texts into the hyper-book format. a template is chosen from the set of templates valid for a specific dtd before starting the process. the translation is a two-step process: the parsing process which verifies that a document conforms with its dtd, and the application of rules which map the different elements of the document into a format specified by the template. if the text is available in ascii form, it will be simply inserted and paginated in a template. the next step consists of manipulating the resulting book in order to improve its appearance. the changes deal with the templates and the text itself. the resultant text is characterized by containing active areas (links). the user is provided with the ability to modify the link structure: new links may be added, and existing ones may be deleted or changed. in order to complete the generation process, full text indexes may be provided. there are several reasons which justify the choice of full text indexing: firstly, people often remember exactly a word in a text, for instance a date or a name, and they need a mechanism for finding that word; secondly, the cost is relatively low compared to keyword indexing, and the indexing process can be automatically implemented. when the generation process is complete, the book is ready for the end-user, all the tools used in the generating phase having been removed. our prototype hyper-book is under development on a macintosh iix; the screen size is 19 inches and the resolution is 75 dpi (dots per inch); the software platform is hypercard 2.0. 47 nadia catenazzi et al design issues for hyper-books and visual-books the sgml software currently used is xgml translator, a tool for specifying conversion produced by exoterica corporation. figure 2 shows the conversion process. dtd x represents a given dtd. t l , t2, t3 are templates defined for that dtd. text and template manipulation figure 2: the process of converting electronic texts into hyper-books visual-books the visual-book represents a particular interpretation of the electronic book, based mainly on the visual aspects of a conventional paper (real) book: its physical features, such as dimensions, thickness, page form and general design style. the visual-book is the result of a process of conversion of existing paper books into electronic form. the visual-book project is part of a more general project called the virtual electronic library, under development at the environmental informatics unit of jrc at ispra, italy; the 48 alt-j volume 1 number 2 visual-book is a practical approach to real-book translation. the physical aspects of the real book are the keys to the construction and representation of the visual-book. the focus is on the book's appearance, interpreted as a way of conveying the cognitive background the reader already has. book functionalities are studied in relation to the book's use, so information is presented in a natural and familiar way, and at the same time the reader's activities are supported hi a comfortable environment. an important point in visual-book development is the choice of which original paper book to translate. not every kind of publication is suitable for electronic translation. the computer has some limitations when compared to paper support, such as portability and ease of reading, but at the same time offers advantages, related to its active role. an electronic environment allows changes and updating of original information, provides different views/reading of the same document, integrates multimedia sources of information, permits interchange of data, and offers software support online. all these facilities, while being useful for some types of publication, are not appropriate for every kind of book; different kinds of reading requirements make electronic translation more or less useful for the reader. in particular, books which are commonly read in their entirety and in a linear way, such as novels, are not as suited to electronic translation which can generate confusion and disturb the reading. indeed, the main requirement of such books is that of legibility, and different views can disturb the reading flow. furthermore, if the active environment offered by the computer is not exploited, the screen is reduced to being no more than a surrogate for a role that paper can play better. the relation between book and reader in this context is very close, and the presence of the computer is more a hindrance than a help. on the other hand, publications which are consulted rather then read, such as encyclopedias, manuals, scientific documents, and in general any reference books, and which suggest a different kind of reading geared to problem solving, are more suited to electronic support. the reader of these types of publication is more involved in using their contents than in simply reading them. for this reason, every attempt to enrich and exploit the existing information is useful to the reader, and electronic support plays an important role by offering more facilities than the paper version. another specific and interesting application area, especially for visual-book translation, is that of ancient manuscripts, where page image is often more relevant than content, and the original document is too precious and fragile to be consulted and read as a normal document. when looking for a meaningful application for the visual-book, two areas have therefore been considered: scientific documentation and ancient manuscripts. technological constraints have made parameters such as the book's dimensions, printing and paper quality particularly relevant when choosing an application. the area we are examining at present is that of scientific publication, this being the most promising common application for visual-book and hyper-book. basic concepts of the visual-book the main idea of a visual-book is to offer the reader an object as similar as possible to the paper book. an important principle in visual-book design is therefore to determine which paper book features are most useful and familiar, and to reproduce them hi an electronic environment, particular attention being paid to the functionalities a visual-book could acquire from electronic support, and to the choice of those which would be most useful and consistent 49 nadia catenazzi et at design issues for hyper-books and visual-books with the specification of the book concept. one of the main advantages of electronic support is its dynamic nature which allows the modification and updating of data. on the other hand, a set of functionalities already available in paper books but not currently in their electronic translation are widely used by conventional readers, functionalities such as bookmarks, notes in the margins or elsewhere in the text, highlighting interesting parts, easy access to pages frequently consulted following different clues, information about the ratio between pages already read and those remaining, control of the reading progress, browsing, and searching for interesting sections. the translation of the paper book into a visual-book has to take into account all these functionalities with respect to the available technology. visual-book features the visual-book is composed of a set of visual-pages, i.e. images of original pages connected by the same relations as in the original book. page images are shown on the screen following the book metaphor: a pair of images, the left and the right pages, are displayed in a graphical replica of a book, maintaining as much as possible the original size and shape. this kind of representation solves, in a natural way, problems such as: orientation: maintaining the same aspect as in reality, so the reader has a familiar clue while navigating the visual pages. the page number, the page location, the position of pictures and graphics and other visual clues are consistent with the real-book version, thus overcoming orientation problems which may be caused by the browsing facilities offered by a dynamic medium. personalization: this is more a technical than a conceptual problem. in fact highlighting, underlining, and adding notes are all possible and naturally interpreted (maintaining, once more, the same appearance as on the paper) in the visual-book but, at this development stage, the difficulty lies in managing pictorial objects overlapping written text. searching: this is based on the table of contents and indexes already present in the original book. there is no text indexing because of the pictorial nature of the visual-book. another important strategy in visual-book searching is serendipity, the art of finding relevant or interesting information which you were not specifically looking for. by maintaining the real-book appearance, it is possible to reproduce in the visual-book the same effect as in real-book browsing. history: this can be maintained as a visual map of the visited pages, using suitable icons to represent them. similar work an interesting interpretation of the visual-book concept is the one developed by the library of congress research service (arms, 1990). the idea is to merge the existing index file with the document image file, and at the same time provide the document image display (soft copy) and the document hard copy (printed version). the objective of this experiment is to evaluate the impact of sophisticated electronic technologies for translating documents to optical digital storage and retrieving them in order to preserve and manage library printed material. at the same time, it is interesting to examine the effects of the technology on copyright, royalties and the publishing community. an interesting feature, when developing the visual-book, is the 50 alt-j volume 1 number 2 acquisition module and the connected compressor and decompressor module. the storage strategy is based mainly on optical-memory technology. another possible interpretation of the visual-book, as a reconstruction of a real document in an electronic medium, is the work developed by inist the institut de l'lnformation scientifique et technique (gourd, 1991). this organization manages a large number of written scientific publications, after scanning them in electronic format. the system used for document management is geac (the automated library cataloguing system used in some uk universities). digitized documents are then presented in hard-copy form when a request is made, thus allowing the system to by-pass presentation problems. the importance of this approach lies in the solution to storage problems and, in general, the addressing of the problem of managing large document collections. another similar system is elnet (king, 1991), a database in japanese composed of 34 newspapers and more than 2s0 journals, all digitized on optical support, and available as hard copy. it has also faced the problem of data storage, and the solution (optical storage) is close to the one chosen by inist. this system is part of a family of similar work all aimed at information delivery through fax or equivalent tools, starting from a paper original, and avoiding display problems by only providing hard copies of the original document. visual-book builder the chosen interpretation of the visual-book is based on the collection of page images organized in a way which simulates the real-book structure. the elementary model is composed of pages, interpreted as pictorial objects and the relationship between the previous and following pages, so as to allow them to be turned. it is possible to isolate at least three main components in visual-book construction: the photo laboratory, the pictorial database, and the authoring environment. the module that allows page-image acquisition is the photo laboratory. currently we realize the acquisition by testing different grey levels during the scanning process. image processing can be performed using software such as photoshop on a macintosh. the idea is to define the command sequence leading to a good-quality final image. this is the basic step toward producing a complete automated image-processing module. theoretical problems arise in defining the concept of legibility, but this stage is mainly based on testing available technology for text image-processing, which includes scanning, compressing and filtering the text image. at the present status of development, we acquire pages through a datacopy scanner, using 64 grey levels with a resolution of 100 dpi. the set of books used for our experiment is composed of elements made of less then 100 pages, and of pocket size. paper quality is average, in order to reproduce a realistic example, and style and typographical components . have been chosen to assure a good level of legibility, i.e. the written text has to be clearly printed and its dimension must allow for comfortable reading. the choice of the scanner has been determined by the quality and properties of the objects to be scanned (black and white text) as well as by the limitation in screen resolution (75 dpi against paper with more than 2,000 dpi). black and white text does not require a sophisticated colour scanner, which is usually more suitable for photo acquisition, but several grey levels are useful in order to make displayed text more legible. different scanners, according to these 51 nadia catenazzi et al. design issues for hyper-books and visual-books requirements, have been tested, and other attempts will take place as and when new products appear on the market. another evolutionary area is that of monitors, where various experiments are also foreseen in parallel with the development of display technology. for this reason, the acquisition phase has to be considered as a changeable module, guaranteeing standard output as the basis for the rest of the project. after this step, images are saved as (macintosh) pict files in order to be opened by photoshop as well as by hypercard. pict is an almost standard format, and guarantees that resulting data is accessible by a large number of applications. at this stage, storage problems are not considered, so compression techniques are not applied to these picture files. at the same time, observations have resulted in a set of experimental values for acquisition parameters such as the grey levels and the dpi in order to obtain a good, readable image. the result displayed on the screen, a 21 inch colour monitor with a 75 dpi resolution, is legible and clear but not so much as to make the user prefer to read the electronic version instead of the original paper one. the purpose of the visual-book project is first and foremost an investigative one, and results are used as a starting point for exploring other related concepts. as for the pictorial database, a picture repository has been created in order to collect the results of the photo laboratory. a hierarchical organization is given to pictorial objects which reconstructs the book's original structure. in addition, a file containing details of the logical structure of the book is constructed during image-book creation. the purpose of this descriptive phase is to realize a prototype as independent as possible of particular kinds of book presentation and structure. currently, the emphasis is on developing an image storage strategy. a library composed of visual-books requires a large amount of memory in order to keep and manage an appropriate quantity of visual-books. the most promising approach is the one followed in the set of projects mentioned above, i.e. optical storage. at the same time, software and hardware compression techniques can be applied to the picture files, and in particular those solutions suitable for black and white images of text. the authoring environment is the module that allows an image book designer to reconstruct the visual-book on the screen. the basic requirement for this environment is to maintain the original pagination of the document, and perform it in a (semi-) automatic way, following jthe description defined in the pictorial database construction. this offers a natural and well-known access key to the document which already exists in paper form, giving a consistent way to browse and read. some other reader services have been added around the simple image book: personalizing, searching and history, developed under hypercard. page pictures are inserted in the correct position within the book at run time by opening the corresponding picture file and showing it on the top of the hypercard stack whose background mimics an appropriately open book. the book is presented using two templates: the first simulates the cover page, the second imitates two open pages. a lateral band suggests the current position while reading or consulting the book. arrows are shown on the screen for turning the visual-pages backwards and forwards. the table of contents is interpreted as text (using ocr or inserting it manually) that is translated into active references for readers looking for particular subjects. the index is a more complex object. many conventions have to be exploited in order to translate in a meaningful way the classical paper index: references such as see also or stylistically emphasized items need particular care. 52 alt-j volume 1 number 2 hyper-books and visual-books in an electronic library the natural evolution of our prototype is the organization of hyper-books and visual-books into a library, and their distribution on line. the resulting electronic library is called the virtual electronic library (vel). the acquisition problem has already been solved: hyper-books and visual-books are the result of this process. the next step consists of organizing and making them accessible. online catalogues are the main access method in the vel. the cataloguing process could be automated for hyper-books by exploiting information provided in their original markup. in addition to the features described above (infinite copies, infinite space), the vel presents a number of innovative features. hyper-books are not isolated objects, and links exist between documents. visual-books provide a limited link capability but they also offer a powerful tool for the complete conversion of paper books into electronic support, thereby solving problems of storage and maintenance of large collections of paper books. intelligent tools are being developed by participants in our project in order to simulate the role of the librarian as consultant: help in formulating bibliographical queries, help in understanding the result of a search, general information about the system. the concept of loan in an electronic environment is of course completely different from what one usually encounters: loan in this context is comparable to having access to the electronic library material. as documents can exist in as many copies as requested, books are always available. once the book has been accessed, the user can put it in his or her personal library and it can then be manipulated: annotation and cut and paste operations can be allowed, just as if the book were the user's property. a charging system may need to be defined in order to manage the purchase of books. once the book has been bought, the user becomes its owner. users have different interests and objectives. by observing user-behaviour, a vel will be able to construct individual user profiles. knowledge about user-needs may be used in order to optimize retrieval strategies and acquisition policy. such reader profiles can be used to notify new publications of interest to the user, and to rank documents, identified during search, in order of relevance. reader profiles can also be used to filter library material and reduce the confusion caused by an excess of information, or to create contacts among users sharing the same interests. conclusion during the first stage of development, a number of difficulties have been experienced. some of them are related to the difficulty of finding sgml documents and their paper counterparts to be used as test applications. other problems are more dependent on the development environment: hypercard. these are essentially the limited text-processing capability (for example, text cannot be justified), limited graphics management (images are currently included using external functions), inefficiency in performing heavy computational operations (repagination, thickness redrawing) and in managing grey-level images. future versions of hypercard should improve text-formatting capabilities, and speed up executions thanks to a compiled programming language. display problems connected with limited monitor resolution will be solved by the evolution in this technological field, while storage aspects will be faced in the next development phase. an evaluation strategy will be exploited in order to formalize the real impact of our prototypes. 53 nadia catenazzi et at design issues for hyper-books and visual-books finally, the actual publishing of hyper-books and visual-books is extremely important in the context of building an electronic library, and this represents the future objective of our work. references arms, c. (1990). 'the context for the future', in arms, c. (ed)), campus strategies for libraries and electronic information, bedford ma, digital press. barker, p. (1991), 'interactive electronic books', interactive multimedia, 2, 1. barker, p. and manji, k. (1991), 'designing electronic books', educational and training technology international, 28, 4. benest, i.d., morgan, g. and smithurst, mjd. (1987), 'a humanised interface to an electronic library', in bullinger, h.-j. and shakel, b., human-computer interaction interact' '87, north holland, elsevier. benest, i.d. and duric, d. (1990), 'some design issues in the automated office metaphor', conference proceedings of the european x-window system user group (guildford), cambridge, uk european x user group. burril, v.a. and ogden, j.a. (1989), 'vortext: the hard-backed screen', electronic publishing, 2, 3. catenazzi, n. and argentesi f. (1991), 'super-library: an environment for the conversion of linear text into hypertext', riao 91 (recherche d'information assistée par ordinateur) conference proceedings, barcelona. collis, b a . (1991), 'the evaluation of electronic books', educational and training technology international, 28, 4. ebt (electronic book technologies inc., providence ri, usa) (1990), dynatext: electronic book indexer/brower (descriptive leaflet). egan, d.e., remde, j.r., gomez, l.m., landauer, t.k., eberhardt, j. and lochbaum, c.c., (1989, 'formative design-evaluation of superbook', acm transactions on information systems, 7, 1. feeney, m. (1989) 'sgml', library and information briefings, june, pp. 99-115. feldman, t. (1990), 'the emergence of the electronic book', british national bibliography research fund report no. 46, london, bnbrf and the publishers association. gourd, f. (1991), 'il sistema interattivo di fornitura di documenti numerizzati all'inist', convegno smau/cnr', la biblioteca elettronica, meeting of 3 october, milan (unpublished summary of presentation). king, s.v. (1991), 'ii sistema telnet', convegno smau/cnr, la biblioteca elettronica, meeting of 3 october, milan (unpublished summary of presentation). lande, n., (1991), 'toward the electronic book', publishers weekly 238, 42. macleod, i.a., barnard, d.t., hamilton d. and levison m. (1991), 'sgml documents and non-linear text retrieval', in riao 91 (recherche d'information assistie par ordinateur) conference proceedings, barcelona. 54 piloting a new approach: making use of technology to present a distance learning computer science course tina wilson and denise whitelock institute of educational technology, the open university computer-mediated communication (cmc) systems have been described and evaluated in a number of ways by different researchers in the field. this paper proposes that computer conferencing systems should be designed to encourage students to participate in three dimensions previously treated by separate researchers. these can be summarized as a knowledge dimension, a social dimension and a motivational dimension. this paper reports on how one particular conference, that of m205-stile, was constructed to take account of these dimensions and to facilitate students' computer-supported cooperative learning. introduction teaching projects which make use of new technology are becoming of interest to all academic institutions in the uk due to economic pressure to increase student numbers. cmc (computermediated communication) such as computer conferencing appears an attractive solution to higher education's 'numbers' problem, with the added benefit that it is free from time and place constraints. researchers have discussed cmc from a number of different perspectives, for example mason and kaye (1989) describe a cmc system as a system of interactivity between tutors, students, resources and organizational structure. steeples et al (1993) consider cmc in terms of group cohesion, modes of discourse and intervention strategies to stimulate and structure participation. goodyear et al (1994) discuss the just in time (tt-based) open learning (jttol) model in terms of a set of educational beliefs, role definitions, working methods and learning resources, together with a definition of infrastructure requirements for cmc. shedletsky (1993) suggests that a cmc should be viewed in terms of an 'intrapersonal communication' model, while mayes et al (1994) identify three types of learning which is mediated by telematics, that is, learning by conceptualization, construction and dialogue. other researchers, such as velayo (1994), describe the teacher as 'an active agent', and present a model for computer conferencing which neglects the social aspect of cmc, while berge (1995) mentions the importance of social activity between students and the importance of the 58 alt-j volume 4 number i role of the moderator. from these accounts, there appear to be a number of dimensions which can be used to evaluate cmc. not all researchers emphasize the same dimensions; however, this paper proposes that computer conferencing systems should be designed to encourage students to participate in all three of the following dimensions. these can be summarized as: (a) a knowledge dimension (includes domain and meta knowledge); (b) a social dimension; and (c) a motivational dimension. we will focus on the design and management of a conference m205-stile, which was employed by the open university's computer science department to supplement the teaching of an undergraduate course. the conference was structured into sub-conferences to take account of the above three dimensions and to facilitate students' computer-supported collaborative learning. this paper reports on the construction of the cmc environment, but more importantly how the three dimensions can be sustained through the role of a person whom we have called an interactive media facilitator (imf). this implementation is part of the 'students' and teachers' integrated learning environment' (stile) project. the stile project the stile project started at the open university at the end of march 1994, and set out to pilot the use of conferencing facilities on m205, the fundamentals of computing course. the main aim of the project was to improve the presentation of the course by offering online facilities to a group of 110 students taking the course in 1995 (february to november). m205 usually has a student population of over 3,000, and is presented to the open university's distance learners as text and video material. for the students taking part in the project, the extra commitment was to use the online facilities for 'tutorial-type support', to access extra course materials, and to interact with their tutor and fellow students. the students who took part were selected from volunteers who were interested in participating, had suitable equipment, for example a 386 pc to a particular specification, a telephone line, and were prepared to pay their own telephone bill. for further information on the selection of tutors and students see wilson (1995). the m205-stile computer conference over 100 students used softarc's firstclass conferencing system to engage with their fellow students and tutors. sub-conferences were set up at various levels within m205-stile where the students could socialize, work in tutorial groups, consult with the course team, and access extra course material (see figure 1). it is even at a first glance obvious to see how the social and knowledge dimensions of the conference have been catered for. the tutorial conferences and the help conference provide the knowledge dimensions, while the general meeting area and the student only area provide the social dimension. this paper concentrates on the help and meeting place conferences. we anticipated that the motivational aspects of the conferencing system would become evident through the changing nature of the conference itself, and that the introduction of an independent imf would have a significant contribution to the motivational aspects of the conferences overall. 59 tina wilson and den/se whitelock making use of technology t o present a distance learning computer science course • materials notices messages the desktop mailbox d-d misc confs r m205 stile o ft resource meeting area " e l p development d tutorials course material messages npn.,n on-line material | messages figure /; conference structure for m205-stile the role of the interactive media facilitator in order to create a learning environment for the students without overloading the tutors, it was important to consider how to minimize the perceived workload. goodyear et al (1991) found that tutoring via email worked well but that the 'quality and quantity of interactions on matters of course content' did not meet initial expectations. to ensure a work focus, the overall control and responsibility for all conferences within m205-stile was allocated to an independent controller, the imf. with this strategy, the tutors were left free to engage in the tutorial conferences and to dip into the other conferences as they wished. in order to reduce tutoroverload associated with reading social-type messages, the work-related dialogue was separated from chat, not only by creating separate conferences but also giving the responsibility of the chat conferences to the imf. there is always a need for a moderator in conferencing to stimulate conversation and watch out for trouble, etc. berge (1995) describes the role of the moderator in computer conferencing as involving 'special technical and educational responsibilities and powers'. the role of the imf however is different. the imf is responsible for (a) constructing the environment; (b) acting as a facilitator and motivator; and (c) encouraging tutors and students to utilize the medium in order to take full responsibility 60 alt-j volume 4 number i for their teaching and learning. in the future the imf's role will develop across different types of media, but in this specific implementation, the imf facilitated distance learning students' use of conferencing and the world wide web. the imf, in this instance and in the future, did and would develop teaching materials to guide the students in their use of new technology, in this case to bring them online. the imf was the overall controller or administrator for all conferences (including social conferences), and guided the tutors and students in how to use the system for teaching and learning. tutors and students have varying levels of experience in, and abilities at, mastering new technology, and the imf needed to encourage the learner not to be frightened of the technology. this was achieved partly by monitoring and encouraging activity in all conferences relevant to the course, directing the students to the different conferences at the right time, setting up new conferences and access permission as required. the imf looked out for developing needs for conferences, sent encouraging personal messages to the students and tutors, and kept a watchful overview to spot problems before they arose. the imf was centrally based and the lynch pin (contact person) for students and tutors alike to contact when they first came onto the system and in cases of problems throughout the year. we would recommend that in the future the imf might be seen as part of the course team (that is, part of the group of people putting together the distance-learning course), taking overall responsibility for the online side of course production, that is, preparing relevant documentation to enable the students to access the facilities, and then following the role through to creating the environment and facilitating the tutor and student participation. in this way, the imf would be aware of the learning aims of the course, while supplying expertise in how the technology could be used to meet objectives in both the design and presentation of the course. the use of new technology would therefore be integral to the course and not just a bolton after-thought. the imf could take control of a number of courses, but work overload must be avoided if the imf is to fulfil all his or her duties. stile participants of the 110 students taking part, 24 were female and 86 were male. of the 24 female students, nine had used a modem before, and is had not. six of the female students had used conferencing software, and one had used internet mail. of the 86 male students, 50 had experience of using a modem, and 36 did not. sixteen of the male students had experience of using conferencing software, and 15 had experience of sending internet mail. the gender of the nine tutors taking part in m205-stile was six males to three females. one of the tutors had previous experience in conferencing. two had experience of using modems. six had varying experience in using email. all used computers as part of tutoring m205, and all used computers for their occupation. preliminary findings the presentation of the course runs from february through to october, and requires the students to submit eight tutor-marked assignments and to sit an examination at the end of the course. to gauge what type of activity the students were involved in and how it changed, we have analysed the number and type of messages in two sub-conferences, which were the meeting place, a social conference, and m205-help, a work-focused conference. the period 61 tina wilson and denise wh/te/ock making use of technology t o present a distance learning computer science course social general problem help with messages computing solving the system subject matter in the 'meeting place' figure 2: type of messages found in the meeting place >*o k_01 xi e 3 z 0) s0» co 0) dl <d (0 09 e 30201001 ^ ^ ^ ^ ^ ™ ' l$js$$bjli§»| gbb8bs9e8! ""hill rim | n 1 1 student m ———— ap r messages ma y to the ju n imi 'meeting au g place1 figure 3: number of student messages posted to the meeting place per month monitored was between february and august, 1995. these two conferences were compared since there was no restriction on who could access either conference. social aspects the social dimension of the conference was apparent from the beginning, since we found that students tried to get to know the system, and in so doing started to learn about each other. the first month was a critical period for social activity before the focus on course-content became more important. looking at the social conference, that is the meeting place, the majority of student messages are for fellow students. not much talk is directed to their tutor. student messages to their fellow students accounted for 47.6% of messages, while student messages to tutors only accounted for 5.6%. there is very little talk about general computing, problemsolving or about help with the system itself; in fact, as one would expect, most of the messages were of the 'getting to know you' type (see figure 2). the pattern of student interaction from february to august in the meeting place is shown in 62 alt-] volume 4 number i figure 3. more messages were recorded in february, while there was only one in august this suggests that in the first month the students were using the system from the earliest opportunity to make contact with each other and their tutor. there was no need to talk about work at this stage, and we found that students discussed work-related issues later in the course (see below, 'knowledge dimension'). students introduced themselves to each other in a number of different ways, but a typical early social message is illustrated by the following example. 8 feb hello everyone, i have been having fun working my way around the icons! i'm from barry so perhaps i'm within yelling distance ofmerthyr!! anyone else in the s. wales area about? good luck everyone. happy clicking! knowledge dimension the m205-help conference was ah area set aside specifically for daily help with solving computing problems. members of the course team who originally wrote the course material were available here to answer queries. the tutors were also invited to 'chip in' if they could. the majority of messages in this area were student-to-student interaction. very little talk was directed to the course team or tutor (see figure 4). a message-approval feature was available with the software but was not adopted in either this conference or the meeting place conference. this meant that whatever message a student put up, the other students could see it immediately and respond without waiting for the course team to reply. whether the messageapproval feature was used or not could have a great bearing on the flow of messages. some students have reported frustration with other conferences when new messages were posted but not readable because they had not been approved. of the total number of messages in this conference, 69% were generated by the students themselves. again the majority of student messages were for their fellow students, rather than the tutors. student messages to their fellow students accounted for 56.8% of messages, while student messages to tutors only accounted for 7.2%. when the message content was analysed (see figure 4), there were more messages of a problem-solving nature than about using the system, etc. the m205-help conference had been designed as a problem-solving area. the imf sent messages to encourage the students to use this conference to ask the course team and their fellow students for help. when the messages from the m205-help conference were analyzed over time (february to august see figure 5), very few messages were found for the month of february. our data indicates that there was more activity in the meeting place during this first month (see below, 'social aspects'). since the messages there were more like introductions, we would suggest that the students at this time were busy getting to know each other and the system rather than solving computing problems. the number of messages, however, increased in the m205-help conference for the month of march, and remained steady through april, may and june. in july, the number of messages increased, and this could be explained by the fact that the students needed to submit a tutor-marked assignment by 1 august. upon closer inspection of the nature 63 tina wilson and denise wn'itelock making use of technology to present a distance learning computer science course (0 cd a? (0 cd e "c cd " d to t ^ o t _ cd x3uu 3 z 60 -i 50 • 40 • 30 20 10 0 m j | | s o ci i1• co cd a i e s s a g e n e ra l co m p u tin g e n> s o (1 « ii il|3**l h el th e subject matter in 'm205 help' figure 4: type of messages found in m205-help figure 5: number of student messages posted to m205-help per month of the message content, students were asking each other for help with respect to these assignments. the majority of messages in the m205-help conference were related to students aiding each other with specific problems which could be conceptualized, as shown in the first message below, or as technical problems see message of 9 april. 64 alt-j volume 4 number i 8 april hiandie from your reply you still seem a bit confused about how to use the two arrays 'previous' and 'newreadings'. i apologise if this is not the case but if it is, then here is another hint. when you get your electricity bill at home, you get charged for the units used that is the difference between the previous reading and the new reading. 9 april you can also save the files you want to keep to a floppy (zero the volume and then rescan volumes so you get it up in your list), then you can just delete the file off of your user volume. motivational aspects the role of the imf was envisaged as sustaining student motivation throughout the duration of the course by responding quickly and effectively to students needs. the students were instructed to direct their subject specific issues to their own tutor's tutorial conference or to the m205-help conference. in the latter, they could expect help from their fellow students and three members of the course team. however, we have found that the imf has been asked about specific help with programming difficulties as well as technical problems with operating the cmc. a further role also emerged, that of counsellor. as a case in point, the imf was asked about his competence as a pascal programmer, and in the ensuing conversation it became evident that the student was struggling on from one tutor-marked assignment to the next, continually reflecting on 'should he stay on the course'. in fact, this student and others have stated that they would have abandoned the course if m205-sttle was not providing support for their study. with m205-sttle, they felt they could see whether other students were experiencing the same difficulties as themselves. one such student has developed a learning strategy whereby he follows the questions asked by another student who orchestrated the difficulties. in effect, he was shadowing the other student and relying on the latter to obtain essential information for his own understanding. the students also expressed their belief in the conferencing system to provide pertinent help, and hence sustain their motivation and interest: 13 feb hi george, from the aberdeen area. i'm looking forward to the course and am all in favour of this type of contact with other students and the tutor. hopefully i will not need to use this method of communication with you too often (wishful thinking i.e. the course is easy) but if i do then i'm sure that i can say what i'm stuck with more clearly via e-mail than over the telephone. and i won't hesitate to use this method. at the beginning of the course, all of the students were sent a questionnaire about how interested they were in using m205-sttle to help their study. a response rate of 50% was obtained, and the majority of students who replied felt motivated to use the system to aid their understanding of the computing subject matter (see figure 6). _ tina wilson and denise whitelock making use of technology t o present a distance learning computer science course scale of 1 to 5. 5= very interested figure 6: students' interest in using the system at the beginning of the project summary of findings our preliminary findings suggest that the conferences were being used to explore coursecontent, although some social 'chit-chat' still took place. many students have been looking on from the sidelines, i.e. not leaving messages, but evidence from student comments suggest that they have found it very helpful to watch the question-and-answer sessions, for example in the m205-help conference approximately five questions per week were asked, but approximately 35 students read each of these messages. students were using the system to find out what happened at any face-to-face tutorial they missed, and were also obtaining help with their m205 coursework. although the tutorial conferences are not discussed in detail here, some were busier than others. students have reported that they have found it very helpful to have read-only access to all the tutorials apart from their own. it will be interesting to see in a later analysis if the tutorial conferences are used more for work-related dialogue as the course gets progressively harder. planned follow-up one important finding to date has been that the tutors were not complaining of being overloaded with student messages, and follow-up studies will involve discussing with the tutors the effectiveness of the conference structure and how the role of the imf reduced their perceived workload. the students were very interested in using m205-sttle in the early part of the course; a follow-up survey to be conducted in december will indicate how well they were motivated to use the system throughout the presentation of the course, and especially for their revision. conclusions this paper proposed that student interactions while using a cmc could be monitored in the three dimensions summarized in our introduction: a knowledge dimension (which includes domain and meta-knowledge), a social dimension, and a motivational dimension. 66 alt-j volume 4 number / by comparing the social and help conferences (see figures 2, 3,4 and 5), it can be seen that the social dimension was more important at the beginning. the critical period for the students to get to know each other and the system was one month. only then did they make use of the system for their coursework. it can be seen that use along the knowledge dimension then increased. although specific conference areas, for example m205-help and the tutorial conferences, were set up to deal with domain-specific interactions, the numbers of messages were deceptively small. this was due to the fact that more students were reading the messages rather than generating their own messages. students said that because they had read-only access to all tutorials, they were obtaining enough information about their assignments by scanning all the messages. although only five or six messages were posted per week to the m205-help conference, these messages were, surprisingly, read by more than 30 students. the motivational dimension became apparent from the students' enthusiasm to use m205-stile to aid their study not only to describe their problems but also to read messages left by other students. m205-stile was set up to minimize information overload for both tutors and students. an important factor could be the imf's overall responsibility for designing the conference and sub-conference structure and guiding the tutor and student activity without intervening too much. the imf designed conference areas to promote work-related dialogue, and to encourage the students to help each other in a supervised environment. the latter point is important because without expert computer-science onlookers, the students could have confused each other. the imf also played a role in promoting work-related dialogue by responding quickly to student problems with the technology itself. acknowledgements the research described in this paper was undertaken in the computing department, faculty of mathematics and computing at the open university, as part of the stile project. the project was supported by a grant from the teaching and learning technology programme (tltp). references berge, z. l. (1995), 'facilitating computer conferencing: recommendations from the field', educational technology, 35 (1). goodyear, p. and steeples, c. (1994), 'computer-mediated communication in the professional development of workers in the advanced learning technologies industry' in eccleston, j., barta, b. and hambusch, r. (eds), the computer-mediated education of information technology professionals and advanced end-users, amsterdam, elsevier. goodyear, p., johnson, r. and steeples, c. (1991), 'training of courseware designers at a distance', proceedings of 8th international conference on technology and education, toronto. mason, r. and kaye, a. (1989), mindweave, oxford, pergamon press. mayes, t., coventry, l., thomson, a. and mason, r. (1994), 'learning through telematics. a learning framework for telecommunication applications in higher education7, the institute for computer based learning and the open university, report ref. e31/e33 for british telecom. 67 tina wilson and den/se wwtelock making use of technology t o present a distance learning computer science course shedletsky, l. (1993), 'minding computer-mediated communication: cmc as experiential learning', educational technology, 23 (12). steeples, c., goodyear, p. and mellar, h. (1993), 'flexible learning in higher education : the use of computer-mediated communications', computers and education, oxford, pergamon press. velayo, r. s. (1994), 'supplementary classroom instruction via computer conferencing', educational technology, 34 (5). wilson, t. (1995), are you able to take part or not? selecting tutors and students for the m205-stile project, internal computing department report for the open university, technical report no. 95/08. 68 microwave workshop for windows colin white division of physics, university of portsmouth this paper describes microwave workshop for windows, a suite of three programs which have been designed in the division of physics in the university of portsmouth to perform the dual function of teaching aid and providing a circuit-design utility in the field of microwave engineering. introduction a suite of three programs has been developed to support the teaching of microwave theory and design. a secondary function of the package is to support microwave engineers by providing a library of utilities to assist their design function. all three programs were written in visual basic and are aimed at supporting both tutor-directed and student-centred learning methodologies. the development team consisted of three final-year degree students. program 1 is smith chart. the smith chart is one of the most important tools for the microwave engineer, yet students find the concepts of its application difficult to grasp using traditional teaching methods. students can choose between running an animated tutorial or using the chart, in stand-alone mode, to solve problems. program 2 is microwave laboratory. this is a virtual microwave laboratory and includes all the waveguide components one would reasonably expect to see in such a laboratory. the components can be connected in any combination, and a generator and detector can then be connected to the completed circuit's remaining ports. important parameters can be varied by the student, and results in terms of power or voltage standing wave ratio (vswr) can be observed. program 3 is microwave device technology, and is a hypertext interpretation of lecture course notes describing a wide range of devices at three levels of complexity. level 1 s7 colin white microwave workshop for windows eu iocls jftew b«lp raaxtma r u t e n ; digna irtfoafl (fi atmnurfjen vswr 103.50 i fulael golf 109.« \t |0«5i kj 1-0.5731 impedance mode figure i: smith chart reflects the requirements of a higher diploma unit, level 2 corresponds to final-year degree level, and the highest level supports postgraduate teaching within the institution. all three programs include extensive help files which describe not only program operation but also context-sensitive microwave theory and device descriptions. smith chart the currently accepted form of the smith chart was developed by philip h. smith in 1936, and is now extensively used by microwave and radio-frequency engineers to aid design work. the chart is a two-dimensional graph with real and imaginary axes (argand diagram) on which can be mapped complex impedance points such as those which may exist along a high-frequency cable or transmission line. the chart may be used to calculate the value of the impedance at the end of the line (the load impedance), or alternatively the length of a transmission line necessary to achieve an impedance match. the chart does not provide answers which could not be derived by someone of a good mathematical grounding armed with a calculator; rather, it performs the function of a special-to-type slide rule to speed up such calculations. the individual steps required to perform the calculations using the chart are actually quite trivial. the difficulty students have in grasping the operations is, i believe, due in part to their inability to equate real and measurable quantities, such as voltage and power which exist along a transmission line, into coordinates and lines on the somewhat abstract concept of a chart. indeed, some students successfully complete calculations on the chart by rote, with little knowledge of the underlying philosophy. the students' cognitive difficulty is further compounded by the fact that the two axes of the chart are circles and segments of circles respectively, and they are also highly non-linear. finally, the confusion 58 alt-} volume 3 number i is exacerbated by virtue of one of the major attributes of the smith chart: each coordinate point represents firstly a complex impedance value, secondly the value of the reflection on the transmission line (the vswr) and finally the distance of the point on the line from the line's termination. it is small wonder that students have trouble when faced with a real problem to solve, and given only a blank, black-and-white chart which is remote from any real hardware. in such situations a coloured cal animation becomes an extremely powerful learning tool. the smith chart tutorial takes the student in a step-wise manner from construction of the chart itself through to incremental problem solving. students are at all times in control of the progression of the program, and hence their pace of study. colour is used to highlight relevant areas of an otherwise invariable chart, and this is mirrored with highlighted points along a drawing of an actual transmission line. movement around the chart corresponds to movement along the transmission line, and student-selectable animations are used to good effect to illustrate these aspects. the, smith chart utility is a fully operational chart on which all functions that would normally be carried out with a paper chart, pencil, ruler and compasses (and probably rubber!) can be performed by keyboard and mouse-click and displayed as accurately as the screen resolution will allow. a magnify button is included for situations where screen resolution is a limiting factor. results are as accurate as the computer's arithmetic precision has been set, rather than governed by the thickness of the pencil lead, and it is, consequently, orders of magnitude better than the traditional method. as the cursor is moved around the chart by the mouse, the corresponding impedance value is updated in real time. this dynamic nature, again, aids the students' understanding of the chart as they acquire a better feeling for the impedance values in different areas of the chart than they ever would by merely reading off the scales. finally, although unnecessary in the normal course of events, a calculator button is included for emergencies. although this is not the only software implementation of the smith chart available, it is the only one designed primarily for teaching purposes. it is also the most intuitive and, perhaps most significantly, the only implementation which runs under windows. microwave laboratory microwave laboratory is an accurate visual representation of a real teaching microwave laboratory. all the common components are represented. they are taken individually, as required, from the virtual component store, and drag-dropped onto the virtual test bench where they are automatically connected together if dropped adjacent to another component. in this manner, a complete circuit may be built up, and the input/output ports are connected to the microwave generator and receiver respectively. these, again, are accurate representations of commercially available equipment. devices capable of parameter variation are adjusted using a control pane launched by double-clicking the device. to aid the construction of an elegant arrangement of the circuit on the workbench, any device can be reoriented in 90° steps by pressing the rotate button. further, any device can be deselected by dragging it to the mac-style dustbin. 59 colin white microwave workshop for windows once a circuit has been constructed, the generator power and frequency can be selected and then switched on. first, the circuit is checked for validity by confirming that the workbench contains one launcher, one or two detectors, no open ports and no loops. each device in the circuit defines a square array-type variable of order equal to the number of ports. this device array fully defines the device characteristics. a matrix multiplication routine calculates the power at the detector and displays it on the virtual amp in digital and analogue format. if a standing wave detector has been selected, power is also displayed graphically in a pane below the workbench, as a function of length along the transmission line. the usual windows menus or short-cut keys allow any useful circuits to be saved and then opened at a future date or, indeed, to be printed. certain common devices are not available in the device store. there are, however, sound pedagogic reasons for this. by way of example, consider the wavemeter. this is a device used to measure very accurately, but with much time and effort, the output frequency of the generator. it is obvious that the frequency accuracy of the virtual generator is defined by the program's mathematical precision and is certainly orders of magnitude better than the real wavemeter. there is, therefore, no reason to specify a wavemeter and, perhaps more importantly, the measurement of accurate frequency can detract from the main aims of the experiment. this laboratory simulation will model all the usual microwave experiments quickly, efficiently and accurately, but it important that students do not perceive it as the real world. the program is intended to support traditional experimental procedure but not to replace it. typically, the student is given some time on the package, constructing the circuit and measuring relevant parameters before doing it for real. by this means he or she acquires an overview of what each device does, what happens when they are connected in a particular configuration, and the particular measurement technique to be learned by the experiment. an important part of the learning strategy is in the interpretation of measurement discrepancy between the virtual and the real laboratories. microwave device technology the third program is simply a hypertext implementation of microwave course notes. the notes reside in three levels representing hnd, final level degree and msc teaching. the level can be switched at any point in the device descriptions to enable a lower-level student to read about a particular device at a higher level, if required (or even desired!). the program has been used in a traditional lecture environment using a laptop and overhead tablet as a cleaner alternative to chalk. students have access to the notes on the network and they can print out any or all of the levels, as required, for study remote from a terminal. future work future work will involve both improvements to the existing programs and also additions of new programs to the workshop suite. 60 alt-} volume 3 number i the smith chart presently supports only calculations on one impedance point. occasionally it is necessary to work with two points and, for instance, calculate the impedance difference. a future version will incorporate this facility. the microwave laboratory can support only devices specified in the device store. it is a trivial addition to include a user-defined black box which may be connected on the workbench. in this case, by double-clicking on the black box, the user may write directly to the defining array the parameters of some device not included in the store. however, a non-trivial addition to the microwave laboratory could be the inclusion of a swept-frequency generator and amplitude analyser. this would open up a range of more sophisticated experiments involving frequency response measurement techniques. the microwave device program will soon have device diagrams reinforced by colour scanned images of the devices. as for additional programs, a microstrip version of the laboratory simulator is planned which is thought to be of interest to commercial companies, and an antenna design cal package and design utility. 61 calt586679 89..100 undergraduate paramedic students’ attitudes to e-learning: findings from five university programs brett williams*, malcolm boyle, andrew molloy, richard brightwell, graham munro, melinda service and ted brown monash university, frankston, victoria, australia (received 27 january 2011; final version received 28 april 2011) computers and computer-assisted instruction are being used with increasing frequency in the area of undergraduate paramedic education. paramedic students’ attitudes towards the use of e-learning technology and computer-assisted instruction have received limited attention in the empirical literature to date. the objective of this study was to determine paramedic students’ attitudes towards e-learning. a cross-sectional methodology was used in the form of a paperbased survey to elicit students’ attitudes to e-learning using three standardised scales. convenience sampling was used to sample a cross-section of paramedic students at five universities during semester 1 of 2009. the scales used were: the computer attitude survey (cas), the online learning environment survey (oles), and the attitude toward cai semantic differential scale (atcaisds). there were 339 students who participated. approximately onehalf (57.7%) were female and most (76.0%) were under 24 years of age. moderate results were noted for the cas general and education subscales. the cas results were broadly corroborated by the oles, although a statistically significant difference between participants preferred and actual results on the oles computer usage subscale identified that participants would prefer to use computers less than they actually do. similarly, the atcaisds found participants were largely ambivalent towards computers. as paramedic degree programs continue to emerge and develop, careful consideration should be given to the usability and utility of various e-learning approaches. keywords: e-learning; paramedic; higher education introduction paramedic education and training in australia and new zealand has undergone significant change over the past decade, moving from its former vocational roots to tertiary-level education. although the first university-based paramedic bachelor degree course began over 10 years ago in australia, the past five years have seen the number of universities offering paramedic programs increase dramatically. presently, there are 14 higher education institutions offering either undergraduate or postgraduate entry-level courses. while each of these programs is on-campus, all of these programs still integrate some form of e-learning into their pedagogy. for example, this might include whole units of study delivered via online only, such as clinical fieldwork rotations, or simply elements of blended learning spread *corresponding author. email: brett.williams@monash.edu research in learning technologyaquatic insects vol. 19, no. 2, july 2011, 89–100 issn 0968-7769 print/issn 1741-1629 online � 2011 association for learning technology doi: 10.1080/21567069.2011.586679 http://www.informaworld.com throughout programs, using programs such as blackboard or moodle. specific elearning examples include sophisticated online virtual patient simulations, formative online quizzes and self-evaluation, and problem-based learning provided via online modules. although this transition to university-based education has been an important part of the professionalisation of the paramedic discipline, the rate at which these programs have been established has led to the existence of a level of uncertainty about general e-learning teaching approaches or students’ perceptions and attitudes to e-learning. computers, computer-assisted instruction (cai) and e-learning are being used with increasing frequency in the area of undergraduate and postgraduate paramedic education. e-learning refers to internet-based technologies that are used to enhance teaching (ruiz, mintzer, and leipzig 2006). cai does not necessarily use the internet, but typically incorporates audio, multimedia, and student interaction (rouse 2007). as the internet is increasingly used to integrate cai initiatives, the terms e-learning and cai are becoming synonymous (ruiz, mintzer, and leipzig 2006). examples of cai include initiatives such as dvds and simulators that allow paramedic students to witness and discuss a range of authentic scenarios, ensuring students experience a breadth of scenarios during their coursework and while on clinical placement to assist them towards work readiness (williams, brown, and archer 2009; taber 2008). more generally, e-learning can potentially help encourage students’ critical thinking, self-reflection, multiculturalism, team work and interprofessionalism (harden 2008). when e-learning strategies are implemented, the role of the learner often changes from a passive one to one who is self-directed and takes responsibility for their own learning (williams 2006), indicating that the role of the educator must change to support student’s learning. the integration and use of educational technology and e-learning strategies are being utilised with increasing frequency in the higher education sector. while this integration is simply to avoid falling behind in a world where computers have become ubiquitous; the implementation of such teaching methods should be carefully considered (govindasamy 2002). too often e-learning initiatives are seen as a pedagogical panacea in higher education. moreover, these initiatives are often not based on educational theory, leading to unsuccessful, unrewarding and uninspiring experiences for students and educators alike (harden 2008). when e-learning strategies and other educational technologies (i.e. wikis, audience response systems) are used appropriately, it is argued these tools offer better experiential learning and greater student empowerment, which is particularly important given the emergence of the ‘net generation’ (maag 2006). initiatives that involve e-learning must meet the needs of students. e-learning must contribute to students learning by providing appropriate content, but to be truly effective it must be tailored to students’ prevailing attitudes and expectations. the student must be the focus, not the technology (roe, carley, and sherratt 2010). attitudes towards computers have changed significantly over the past two decades as computers have become omnipresent; for instance, there is no longer a difference between genders in general attitudes towards computers (popovich et al. 2008). the major factor affecting attitudes today is anxiety; with there being a negative relationship between anxiety about computer and attitudes towards computers (popovich et al. 2008; hegney et al. 2006; webster et al. 2003). in other words, 90 b. williams et al. e-learning needs to be appropriate for students’ level of ability with computers and using technology, otherwise it will be frustrating (link and marz 2006). understanding students’ attitudes towards e-learning facilitates the development of appropriate educational initiatives. paramedic students’ attitudes towards the use of e-learning technology and cai have received limited attention in the research literature to date. as such, the objective of this study was to determine paramedic students’ attitudes towards e-learning. to do so, a survey of undergraduate students enrolled in paramedic degrees across australia and new zealand was conducted using three standardised self-report scales. method design a cross-sectional methodology using a paper-based questionnaire encompassing three standardised scales and a demographic section. population a cross-section of students enrolled in emergency health (paramedic) undergraduate degrees from five universities across australia and new zealand during semester 1 of 2009 was surveyed. a convenience sample of students from monash university (mu), auckland university of technology (aut), charles sturt university (csu), edith cowan university (ecu) and queensland university of technology (qut) was used. instrumentation the computer attitude survey (cas) is designed to assess participants’ attitudes towards computers (startsman and robinson 1972). the scale consists of 26 items, each rated on a five-point scale (1 = strongly agree, 5 = strongly disagree). negative worded items are reversed before undertaking analyses. the first 16 items make up a subscale concerning attitudes towards computers in general (casg). the remaining 10 items concern attitudes towards computers in education (case). low scores for both sub-scales are indicative of a positive response towards computers. the cas has previously demonstrated reliability and validity (wagman 1983; startsman and robinson 1972; steele et al. 2002). the online learning environment survey (oles) is a test that assesses participants’ actual and preferred experience with components of an online learning module they have recently completed (pearson and trinidad 2005). the scale consists of 54 items rated on five-point scales (1 = almost never, 2 = seldom, 3 = sometimes, 4 = often, 5 = almost always). from the items nine scales are derived, which are outlined in table 1. internal consistency has been reported by trinidad, aldridge, and fraser (2004) as ranging from 0.86 to 0.96. content validity and face validity were established through a detailed literature review and input from content knowledge experts. construct validity was established through a factor analysis (pearson and trinidad 2005). the attitude towards cai semantic differential scale (atcaisds) is a scale used to elicit participants’ attitudes towards the use of computers (allen 1986). on research in learning technology 91 seven-point scales, participants are asked to select which word they err towards on 14 differential descriptive terms (e.g. rigid or flexible, stimulating or boring). the content validity, factor structure and internal consistency of the atcaisds have all been reported as satisfactory (allen 1986; brudenell and carpenter 1990). procedures at the conclusion of lectures for each year level of each of the universities, students were invited to participate in this study on a voluntary basis. students were provided with an explanatory statement and were informed that participation was voluntary and anonymous. a non-teaching member of staff facilitated the administration of the questionnaire containing the cas, oles, atcaisds, and table 1. descriptions and examples of the nine oles subscales. subscale number of items description sample item computer usage 6 the extent to which students use their computers as a tool to communicate with others and to access information “i use the computer to ask the teacher questions” teacher support 8 the extent to which the teacher helps, befriends, trusts and is interested in students “the teacher gives me valuable feedback on my assignments” student interaction and collaboration 6 the extent to which students have opportunities to interact with one another, exchange information and engage in collaboration “i share information with other students” personal relevance 5 the extent to which there is a connection between students’ out-of-school experiences “i can relate what i learn to my life outside of this class” authentic learning 5 the extent to which students have the opportunity to solve real-world problems that are authentic “i work on assignments that deal with real-world information” student autonomy 5 the extent to which students have opportunities to initiate ideas, make their own learning decisions, and the locus of control is on the student “i make decisions about my learning” equity 7 the extent to which students are treated equally by the teacher “i am treated the same as other students in this class” enjoyment 6 the extent to which students are satisfied with their e-learning environment “i would enjoy my education if more of my classes were online” asynchronicity 6 the extent to which the discussion forum promotes reflective thinking and the posting of messages at times convenient to the students “i read the posted messages at times that are convenient to me” 92 b. williams et al. demographic questions, and arranged for collection of the completed questionnaires. the questionnaires took approximately 20 minutes to complete and consent on the part of the students was implied by completion and submission of the questionnaire. ethics approval was obtained from each of the five participating universities’ human ethics committees. data analysis the statistical package for social sciences (spss, version 18.0) was used for data storage, tabulation, and the generation of descriptive statistics. means were used to describe the descriptive data and the analysis of variance with tukey honestly significant difference (hsd) post hoc test was used to determine the relationship between the three scales and each university. all tests were two-tailed unless otherwise stated, and results are considered statistically significant if p < 0.05. results demographics a total of 339 students participated in this study from five different universities in australia and new zealand. the two universities with the largest representation in the sample were mu (n = 116) and qut (n = 116). slightly more than one-half (57.7%) of the participants were female and the majority were under 24 years of age (76.1%). there was a representation of students from each year of the respective universities’ courses; however, first-year and second-year students had the largest representation in the sample. the complete demographic profile of the participants is presented in table 2. computers participants were asked a number of preliminary questions related to the use of and access to computers. the overwhelming majority owned a computer (94%), and had unlimited access to both a computer (97.1%) and the internet (97.1%). the computers that participants have access were mostly new (<1 year, 17.4%; 1–2 years, 47.8%). when asked whether they had plans to purchase a new computer, most either had no plans (48.7%) or indicated they possibly would 18 months into the future (32.0%). that is, one-third had an intention to upgrade their computer sometime into the future. thus computers are accessible by effectively all the students sampled. indeed, approximately three-quarters (72.1%) of the participants reported over 60% or more of their academic work already involves the use computers in some capacity. computer attitude survey the casg subscale has a range from 16 to 80. the participants overall reported an attitude almost exactly in the middle between positive and negative (mean = 49.2, standard deviation [sd] = 4.9). the case subscale has a range of 10 to 50. participants’ attitude towards the use computers in education was also in the middle between positive and negative (mean = 31.0, sd = 5.2). research in learning technology 93 a statistically significant difference between participants from the different universities and their general attitude towards computers was found, f(4, 311) = 40.4, p < 0.0001. a post hoc analysis using tukey hsd found the mean scores for participants from ecu (mean = 44.7, sd = 3.0) and qut (mean = 45.7, sd = 4.4) were significantly lower (better attitude) than the other three universities. an g2 value of 0.34 suggests the effect size is large. an analysis of variance also found a statistically significant difference between the participants from the different universities in the attitude toward computers in education, f(4, 314) = 21.6, p < 0.0001. a post hoc analysis using tukey hsd again found ecu and qut to have statistically significantly higher means (worse attitude) than the participants from the other three universities. the effect size for the difference between participants, using g2, was large at 0.22. the mean scores for both the casg and case subscales are presented in table 3. table 2. participant profile. number percentage university monash university 116 34.2 auckland university of technology 35 10.3 charles sturt university 49 14.5 edith cowan university 23 6.8 queensland university of technology 116 34.2 total 339 100 year first year 117 34.5 second year 116 34.2 third year 12 3.5 first-year double degree 66 19.5 second-year double degree 12 3.5 third-year double degree 5 1.5 fourth-year double degree 10 2.9 not stated 1 0.3 total 339 100 age 15–19 years 132 38.9 20–24 years 125 36.9 25–29 years 26 7.7 30–34 years 28 8.3 35–39 years 10 2.9 40–44 years 12 3.5 44–49 years 4 1.2 >50 years 1 0.3 not stated 1 0.3 total 339 100 gender female 195 57.5 male 143 42.2 not stated 1 0.3 total 339 100 94 b. williams et al. online learning environment survey the overall results for the oles are presented in table 4. paired-sample t tests found statistically significant differences between the actual and preferred results for the computer usage, personal relevance, authentic learning, and asynchronicity subscales. the g2 statistic, however, shows that the magnitude of the difference between participants’ actual and preferred results was only large for the computer usage subscale. on this subscale, participants overall indicated that they would prefer slightly less use of computers in their university education. a statistically significant difference was found between participants from the different universities and their actual and preferred mean scores for the nine subscales. these results are presented in table 5. the two subscales with the largest magnitude of difference between the mean scores reported by participants from different universities, as measured by g2, was preferred teacher support (g2 = 0.45) and preferred student autonomy (g2 = 0.26). for teacher support, participants from ecu (mean = 17.4, sd = 4.4) would prefer more teacher support than those participants from mu, aut and csu. the participants from qut would prefer even more, again (mean = 20.6, sd = 5.5). it is of note that a statistically significant difference was noted between the actual teatable 3. computer attitudes survey results. university general (casg) education (case) mean sd mean sd monash university 51.4 4.2 28.9 5.2 auckland university of technology 50.7 3.6 28.5 4.3 charles sturt university 52.0 2.9 29.8 4.6 edith cowan university 44.7 3.0 33.2 3.0 queensland university of technology 45.8 4.4 34.0 4.3 overall 49.2 4.9 31.0 5.2 note: possible scores range from 16 to 80 for general and 10 to 50 for education table 4. overall participant actual and preferred for oles subscales. subscale actual preferred p value g2mean sd mean sd computer usage 15.2 3.8 13.2 3.8 0.000 0.24 teacher support 16.1 5.6 15.3 6.1 0.104 0.01 student interaction and collaboration 11.4 4.6 11.2 4.2 0.285 0.00 personal relevance 10.0 3.6 9.5 3.3 0.013 0.02 authentic learning 9.7 3.4 9.0 3.3 0.000 0.04 student autonomy 8.8 3.1 8.4 3.2 0.078 0.01 equity 11.0 4.3 10.7 4.1 0.114 0.01 enjoyment 15.4 5.0 15.2 5.0 0.130 0.01 asynchronicity 17.3 6.0 16.6 6.6 0.002 0.03 note: paired-sample t test used to determine statistical significance. research in learning technology 95 cher support for participants from each university, with participants from ecu and qut reporting lower mean scores, indicative of less teacher support (table 5). with regard to student autonomy, participants from ecu (mean = 11.13, sd = 3.0) and qut (mean = 10.2, sd = 3.4) reported that they would prefer significantly more autonomy than the other three universities. it is of note that there was a statistically significant difference in the actual student autonomy means by university. participants from ecu reported a lower mean score for actual student autonomy, although not by a statistically significant margin. participants from qut, however, reported a statistically significant lower mean score for actual student autonomy (table 5). attitude towards the cai semantic differential scale overall the participants’ mean responses were close to the midpoint of four for each semantic differential item. there was, however, one standout differential scale item rating; participants erred toward the term ‘personal’, as opposed to computers being ‘impersonal’. the overall results are presented in table 6. comparing the responses of participants from different universities with the 14 semantic differential rating items found statistically significant differences for all but the last differential scale item, ‘time saving–time consuming’; for this differential rating item, participants from all five universities overall erred to a small degree toward computers being ‘time saving’ (see table 6). post hoc analyses found that in most cases it was participants from ecu or qut whose opinions on the semantic differential scale items that differed. in most cases, participants from ecu and qut reported more negative attitudes towards computers, with the most striking differences being their perception of computers as more inefficient, whereas participants from the other three universities viewed table 5. oles subscales tabulated against universities. mean scores overall p value g2 computer usage actual 15.2 13.2 17.0 14.7 15.1 15.2 0.000 0.06 preferred 13.5 12.5 14.0 11.4 13.2 13.2 0.053 0.03 teacher support actual 18.3 17.9 16.9 12.8 13.4 16.1 0.000 0.17 preferred 12.6 11.6 11.0 17.4 20.6 15.3 0.000 0.45 student interaction and collaboration actual 12.7 14.1 12.4 9.3 9.2 11.4 0.000 0.16 preferred 10.5 11.3 10.0 12.7 12.0 11.2 0.005 0.04 personal relevance actual 10.6 11.9 11.3 9.1 8.3 10.0 0.000 0.15 preferred 8.6 9.3 9.1 11.9 10.2 9.5 0.000 0.08 authentic learning actual 11.0 10.5 10.2 9.4 8.0 9.7 0.000 0.15 preferred 8.3 7.8 7.5 11.5 10.2 9.0 0.000 0.15 student autonomy actual 9.5 9.4 9.5 8.3 7.5 8.8 0.000 0.09 preferred 7.4 6.7 6.9 11.1 10.2 8.4 0.000 0.26 equity actual 11.3 12.8 12.1 10.0 9.9 11.0 0.001 0.06 preferred 9.7 9.0 9.7 11.2 12.4 10.7 0.000 0.10 enjoyment actual 15.6 16.3 17.4 12.8 14.6 15.4 0.001 0.06 preferred 14.6 13.6 15.6 14.9 16.1 15.2 0.065 0.03 asynchronicity actual 17.2 15.8 16.9 16.9 18.2 17.3 0.332 0.01 preferred 14.7 13.7 14.2 18.3 20.4 16.6 0.000 0.18 96 b. williams et al. computers as more efficient. the mean scores for each semantic differential item, tabulated against the participants’ universities are presented in table 6. discussion the majority of students in this study have access to, and use, a computer that is connected to the internet. this is a reflection of the ubiquity of computers and the internet in today’s society. computers have also become an essential tool in university life with most assignments requiring a computer to be completed. indeed, the vast majority, if not all, of essays are typed nowadays; many library resources are available online, and in recent times pen-based computers (known as tablets) have been introduced for students to take notes and interact in lectures. although the integration and utilisation of e-learning is rapidly changing and evolving, as are student expectations, the reality is that, despite the available educational technology, e-learning does not automatically improve teaching and learning. gaining a better understanding of students’ attitudes towards e-learning will provide educators with greater insight and likelihood to develop strategies that match students’ needs. the ubiquity of computers is evident in the cas results. participants had moderate views towards computers in general (casg). no enthusiasm for computers was evident in the results. an analysis of the attitudes towards computers by university found participants from ecu and qut had significantly better attitudes towards computers in general than participants from the other three universities. the attitudes of participants from ecu and qut towards computers in education (case) were, however, more negative. overall, participants’ attitudes towards computers in education were also moderate. these findings suggest further research should be undertaken. for example, are these results related to units of study that use a blended approach, combining face-to-face with e-learning? or are e-learning platforms such as blackboard or moodle simply used by teaching staff as digital repositories, offering students little engagement or synchronous communication? the oles results were similar to the casg/case results; however, caution is advised when interpreting the results of this standardised scale. the first subscale computer usage, for instance, does not enquire into all the uses a student may make of a computer. the most noticeable omission is an item concerning the use of table 6. semantic differential differences between participants from different universities. semantic differential mean scores overall p value 1 = rigid, 7 = flexible 4.10 4.06 4.55 2.87 2.42 3.53 0.000 1 = useful, 7 = useless 2.64 2.71 2.76 3.45 4.11 3.18 0.000 1 = stimulating, 7 = boring 3.71 3.97 4.12 2.23 2.96 3.46 0.000 1 = meaningless, 7 = meaningful 3.09 2.85 3.94 4.05 4.24 3.62 0.000 1 = pleasant, 7 = unpleasant 3.49 3.50 3.92 2.82 3.76 3.60 0.016 1 = valuable, 7 = worthless 2.83 2.79 2.88 3.45 3.87 3.20 0.000 1 = creative, 7 = unimaginative 3.75 3.62 3.80 2.32 3.13 3.45 0.000 1 = impersonal, 7 = personal 5.41 4.88 5.88 4.27 4.25 4.99 0.000 1 = efficient, 7 = inefficient 2.65 2.59 2.65 5.86 5.18 3.65 0.000 1 = inappropriate, 7 = appropriate 3.24 3.21 3.14 2.32 3.11 3.12 0.045 1 = comfortable, 7 = uncomfortable 3.08 3.03 3.24 2.77 3.64 3.25 0.011 1 = nonthreatening, 7 = threatening 2.52 2.44 2.80 2.59 3.47 2.85 0.000 1 = overpowering, 7 = easy to control 3.10 2.94 3.69 2.23 3.05 3.10 0.002 1 = time saving, 7 = time consuming 2.95 3.59 3.53 3.14 3.18 3.19 0.096 research in learning technology 97 computers to conduct research for assignments. the scale also enquires into frequency rather than the participants’ agreement with items, and in doing so confounds the interpretation of how often something is done irrespective of computers and how often something is done with a computer. how participants are answering is, therefore, not explicitly clear. the strength of the oles, however, is the comparison between participants’ actual and preferred attitudes and opinions. a number of statistically significant differences were noted; however, only the computer usage subscale had a large magnitude of difference. it was noted that participants indicated they would prefer to use computers less than they actually do for university-related work. these findings provide important information for the paramedic discipline, particularly as course accreditation of national programs will soon take place. this will provide curriculum designers with a better insight into ways of improving students’ attitudes towards cai and also develop better pedagogical strategies involving teaching using educational technologies. an analysis of participants’ responses to the oles by university found two important differences. participants from ecu and qut would prefer more teacher support and more student autonomy than the participants from the other three universities. it appears that the participants from ecu and qut are not as confident with computers as their peers from the other universities. a negative relationship between computer anxiety and attitudes towards computers has been found elsewhere (popovich et al. 2008). such an explanation would adequately explain the difference between participants from ecu and qut and the other three universities. the findings of the semantic differential scales corroborate the results of the cas and oles. overall the mean score for each differential item was around the midpoint of four, erring towards the more positive word. the only differential item to stand out was the personal–impersonal sematic scale, where participants indicated that computers were personal. this perception is an important area for paramedic education and training, particularly as e-learning offers greater scope providing authentic learning activities (e.g. patient scenario software, or virtual learning spaces that are otherwise more difficult to achieve). thus, if students view computers as personal then this may provide the context for better student engagement, and better intended and observed learning outcomes. an analysis of the semantic differential scale ratings by university found the differences were predominantly between participants from both ecu and qut and the other three universities. consistent with the results of the cas and oles, participants from ecu and qut generally had worse attitudes towards computers, most noticeably in regard to their considering them inefficient. participants from mu, aut and csu considered computers to be efficient. this difference is consistent with the notion that participants from mu, aut and csu are more confident in their ability to use computers. there may be some differences in the orientation and training sessions that students are provided with at each of the universities. as well, there may be differences in the number of computers available for student use and the accessibility of computer laboratories to students (e.g., weekends, late at night) at each university. these may be factors that account for part of the difference between aut and csu compared with the other three universities. the sample of paramedic students from across australia and new zealand is a strength of this study, despite convenience sampling having been used. it is recommended for future study that efforts to understand the needs and wants of paramedic 98 b. williams et al. students with regard to computers delve into more specific aspects. there appears to be significant differences between attitudes of students from different universities, although none exhibit unbounded enthusiasm for computers. this would suggest that the manner in which computers are taught, the availability of computers, access to specific computer programs (e.g. the statistical program spss) and the use of computers varies with measureable effects on students. this study is limited in its ability to delve into specifics, and therefore requires further investigation. this sample of paramedic students appears ambivalent towards computers and, as such, efforts to engage them and maximise the utilisation of computers needs to address their concerns and needs. by not doing so, faculties risk spending significant sums of money implementing e-learning initiatives that could potentially go underutilised and offer poor learning opportunities. consequently, as paramedic-orientated degree programs continue to emerge and develop, careful consideration should be given to the usability, utility, and sustainability of e-learning approaches. conclusion this study has provided significant foundational data for the australian and new zealand paramedic discipline, particularly as the body of knowledge in e-learning pedagogy is limited. results from this study indicate that paramedic student attitudes towards computers in general and computers in education found participants to have moderate attitudes, but a degree of ambivalence was evident. consequently these participants did not indicate an interest in computers for computers’ sake. with such attitudes, students are unlikely to utilise e-learning initiatives to their maximum unless they meet their needs. as paramedic-orientated degree programmes continue to emerge and develop, careful consideration should be given to the usability and utility of e-learning approaches and overall integration into curriculum. acknowledgements the authors would like to acknowledge the paramedic students from the participating universities who took part in the study. references allen, l. 1986. measuring attitude toward computer assisted instruction: the development of a semantic differential tool. computers in nursing 4, no. 4: 144–51. brudenell, i., and c.s. carpenter. 1990. adult learning styles and attitudes toward computer assisted instruction. the journal of nursing education 29, no. 2: 79–83. govindasamy, t. 2002. successful implementation of e-learning pedagogical consideration. the internet and higher education 4: 287–99. harden, r.m. 2008. e-learning – caged bird or soaring eagle? medical teacher 30: 1–4. hegney, d., r. eley, e. buikstra, t. fallon, j. soar, and v. gilmore. 2006. australian nurses access and attitudes to information technology – a national survey. studies in health technology and infomatics 122, no. 6: 88–92. link, t.m., and r. marz. 2006. computer literacy and attitudes towards e-learning among first year medical students. bmc medical education 6, no. 34. maag, m. 2006. podcasting and mp3 players: emerging education technologies. computers, informatics, nursing 24, no. 1: 9–13. research in learning technology 99 pearson, j., and s. trinidad. 2005. oles: an instrument for refining the design of e-learning environments. journal of computer assisted learning 21, no. 6: 396–404. popovich, p.m., n. gullekson, s. morris, and b. morse. 2008. comparing attitudes towards computer usage by undergraduates from 1986 to 2005. computers in human behavior 24: 986–92. roe, d., s. carley, and c. sherratt. 2010. potential and limitations of e-learning in emergency medicine. emergency medicine journal 27: 100–4. rouse, d.p. 2007. computer-assisted instruction: an effective instructional method. teaching and learning in nursing 2: 138–43. ruiz, j.g., m.j. mintzer, and r.m. leipzig. 2006. the impact of e-learning in medical education. academic medicine 81, no. 3: 207–12. startsman, t.s., and r.e. robinson. 1972. the attitudes of medical and paramedical personnel toward computers. computers in biomedical research 5: 218–27. steele, d., j. palensky, t. lynch, n. lacy, and s. duffy. 2002. learning preferences, computer attitudes, and student evaluation of computerised instruction. medical education 36, no. 3: 225–35. taber, n. 2008. emergency response: elearning for paramedics and firefighters. simulation & gaming 39, no. 4: 515–27. trinidad, s., j. aldridge, and b. fraser. 2004. development and use of an online learning environment survey. paper presented at the annual meeting of the american educational research association aera, april 12–16, san diego, ca. wagman, m. 1983. a factor analytic study of the psychological implications of the computer for the individual and society. behavioural research methodology and instrumentation 15: 413–19. webster, j., j. davis, v. holt, g. stallan, k. new, and t. yegdich. 2003. australian nurses’ and midwives’ knowledge of computers and their attitudes to using them in their practice. journal of advanced nursing 41, no. 2: 140–6. williams, b. 2006. qualitative analysis of undergraduate paramedic students’ perceptions of using case-based learning in an online learning environment. journal of emergency primary health care 4, no. 3. williams, b., t. brown, and f. archer. 2009. can dvd simulations provide an effective alternative for paramedic clinical placement education? emergency medicine journal 26: 377–81. 100 b. williams et al. supporting the active learning of collaborative database browsing techniques m. b. twidale, d. m. nichols, j. a. mariani, t. rodden and r sawyer computing department lancaster university we describe the implications of a study of database browsing behaviour for the development of a system to support more effective browsing. in particular we consider the importance of collaborative working, both in learning browsing skills and in co-operating on a shared information-retrieval task. from our study, we believe that an interface to support collaboration should promote the awareness of the activities of others, better visualization of the information data structures being browsed, and effective communication of the browsing process. introduction the proliferation of databases has led to a growing demand for skills in navigating and manipulating them. these skills are important not just in order to undertake study and research in higher education, but as an important transferable skill likely to be expected by industry and commerce of all graduates. we are developing a system that can more effectively support the browsing of library databases. in particular we are interested in developing an interface to support the process of collaborative browsing. we have made a study (twidale et al, 1994) of the requirements of such an interface, chiefly to inform our design intuitions. the study has involved a substantial literature survey, interviews with subject librarians, and small-scale in-depth studies of authentic browsing activities undertaken by library users. the latter involved observing remote collaboration using existing simple tools including unix talk in order to reveal the problems that a more sophisticated system should address. the use of library resources is stereotyped as a solitary activity, with hardly any mention in the library science and information retrieval literature on the social aspects of information systems. however, our study indicates notable collaboration, with users consulting both library staff and each other. informal computer-based collaboration already exists through sharing or leaning over terminals and pointing at screens. 75 m. b. twiddle et al. database browsing techniques traditionally, as computer scientists, we have designed databases to appear as single-user systems. we aim to build a system to support more active collaboration. browsing whereas some information scientists use the term browsing in a very restricted and precise manner, we use it to describe all search activities that involve a level of imprecision about the desired outcome of the search (bates, 1989). a critical feature is that users refine their requirements over time as more information becomes available and set subgoals to satisfy them. the literature on browsing backs up our observations of its importance to library users as well as the problems that novices have. researchers have stressed the importance of serendipity and its effective exploitation in the browsing process (rice, 1988). but it also seems that expertise makes you luckier! this may be because an expert is better able to recognize and exploit the accidental find. there are two types of relevant expertise: generic knowledge of browsing techniques, and local knowledge of the subject area being explored. in addition, browsing involves two different kinds of skill: tactics, which are actions relating to the composition of a single query, and strategy, which involves actions in managing a sequence of queries. examples of tactics are the use of boolean operators, keywords, wildcards and indexes. strategic skills include choosing the right database, restricting or widening a search, and managing your evolving set of goals and subgoals. browsing tactics involve knowledge that can be unique to a single database system, and their transfer between the many and constantly evolving systems that learners use is difficult. this is a frequent complaint both of users and of information professionals (yee, 1991). by contrast, the strategies, being more abstract, transfer better and are the kinds of generic transferable skills that students should acquire. unfortunately, this very abstraction can also make their acquisition all the more complex and their consideration may be masked by the more visible issues of tactics. learning about the system novices may have misconceptions about the browsing process. these can include an inappropriate overview or metaphor of the search process. an example we observed was considering an online public access catalogue (opac) system as analogous to an intelligent colleague. associated with such a misconception will be a set of beliefs, such that results are presented in some order of importance, or that a 'common sense' filter will have been applied. for example, a novice with the 'intelligent colleague' misconception would find it bizarre that a search on the word browse will return references to the behaviour of reindeer, and consequently distrust any use of the search system as being so stupid as to be utterly flawed. information systems are complex, and it is entirely appropriate that novices learn a minimal subset of the available facilities to enable them to do some limited searching. unfortunately, this can lead to users who are confident in their use of the system but who have settled into sub-optimal work patterns. examples of these are: getting several 76 alt-] volume 3 number i hundred hits and reading them all, failing to look at all the results when there are a reasonable number, and failing to pursue a lead or revise a strategy. users may settle into these patterns either because they are unaware of the more sophisticated methods or because they choose not to use or practise them, due for example to cognitive overload (rudd & rudd, 1986). therefore, in addition to providing them with an introduction to that subset, we must consider issues of how they can continue their learning of more advanced features. the process of continuing learning involves experimentation, blind trial and error, asking experts for help, informal learning from colleagues, and advice being volunteered by experts as a part of a consultation about a domain issue. potential losses from computerization computerized library databases offer features not available in traditional library information structures (such as keyword searching and the ability to work remotely), but they may also lose some useful features. these include the spatial nature of the way information is presented in a library: the ability to walk round the bookstacks and to exploit our powerful spatial memory abilities (beheshti, 1992). the organization of bookstacks can also facilitate browsing: a searched-for book on a shelf will have related books near to it that can be equally or even more relevant to the user's vague and constantly evolving information needs. an electronic view of information is remarkably impoverished compared with the vast amount of peripheral information of real books. these include the ease of seeing whether a book is brand new, well thumbed, borrowed a lot, or ancient but never consulted (mitev, 1989). people also have well-developed spatial awareness and memory skills that can be exploited in navigating a physical library and in supporting the retrieval of information by its position (chang & rice, 1993). however, not everything is easily accessible in a physical library. for example, at lancaster the librarianship section is not on open access, and one cannot physically browse the books on browsing! furthermore, one naturally cannot browse the shelves for books that others have borrowed. even the old-fashioned cardindex file can convey peripheral information from nearby cards, the colour and degree of ageing of the card and even how dog-eared it is. such physical information structures enable the browser to get a sense of the size of the related information. collaboration issues it is not just the inanimate contents of a traditional library that convey useful information. people can also be useful to a browser. one can observe and learn from the browsing techniques of others, discuss issues with co-learners or with subject experts, and also be aware of the activities of others that may be of interest and relevance to one's own work. for example, upon seeing a colleague in an unexpected part of the library, one might choose to ask what she has found there. similarly, upon seeing someone in 'your' area, you may decide to introduce yourself as someone also interested in that field. a computerized library that is accessed remotely will lack these advantages unless we take steps to re-introduce them into the system. 77 a i ß. twiddle et al. database browsing techniques research in computer supported co-operative work employs a useful classification of collaboration (rodden, 1991). collaboration may be remote or co-located, as well as being synchronous or asynchronous. in conventional libraries, we can consider most cooperation to be co-located and synchronous, but the computerization process makes the other permutations possible, while offering new opportunities for the first. we envisage a number of scenarios in which the system being developed might be used in an educational context. in all cases, the interaction may be synchronous (participants working at the same time) or asynchronous (participants leaving messages for each other): • expert consultancy. a learner is browsing the database and decides that she is not making the progress she would like. in a physical library, she might go and talk to the subject librarian. for the database use, she may wish to communicate by telephone or email (or even ultimately by video link). the expert calls up a representation of the student's browsing history and composes suitable advice. this advice can be specific, general and remedial. it can help to solve the current task, explain a generic browsing technique and also correct any apparent misconceptions. as part of the explanation, the expert passes on an annotated browsing procedure which the student can view and even use on her terminal. • a 'language laboratory' for database skills. a number of individual learners are browsing a database as part of a practical class on database-browsing skills. they may all be in the same teaching room, or working remotely on their own terminals elsewhere. the expert can observe the browsing activities of the students in turn, and offer advice as necessary. • collaborative browsing for learners. small groups of learners who have a similar browsing task make use of the awareness mechanisms to monitor and discuss the progress and activity of each other. this is known to be a useful learning activity both for the questioner (who asks questions such as 'why are you doing that?'), and for the respondent who has to reflect on her action in order to generate a suitable explanation. • serendipitous meetings. if the groups in the case immediately above are made larger, database awareness can be a useful tool for supporting serendipitous meetings of the kind that naturally occur in physical libraries where strangers meet by browsing the same bookstack and finding common interests for collaboration, or when finding that a book they want is already being used by someone else, who therefore might be worth talking to. now, one can become aware when others are, or have been, browsing the same parts of the database. we are working on an interface to support collaborative browsing. our study will inform the initial version of our interface, and one of our findings we believe to be particularly important. collaborative working implies a need to share information: both the end product (the hits) and the process (the search strategy/tactics). similarly, there is a need to share this information with the librarians, for whom inspection of the search process can 78 volume 3 number i reveal not only gaps in the users' browsing techniques but also an indication of their degree of searching sophistication. in addition, an externalized representation of the search process reduces cognitive load and facilitates reflection, a vital component of learning. an initial analogy would be the unix history list, but the actual representation will need to be considerably more sophisticated and flexible. conclusion it seems that collaboration issues have not been addressed in opac design. we believe that an interface that takes them into account in providing visualization features will greatly facilitate both the learning and exercise of browsing skills. acknowledgements financial support for the research reported in this paper was provided by a grant from the jisc new technologies initiative. references bates, m.j. (1989), 'the design of browsing and berrypicking techniques for the online search interface', online review, 13, 5, 407-24. beheshti, j. (1992), 'browsing through public access catalogs', information technology and libraries, 11, 3, 220-8. chang, s.j. & rice, r.e. (1993), 'browsing a multidimensional framework', annual review of information science and technology, 28, 231-76. mitev, n.n. (1989), 'ease of interaction and retrieval in online catalogues: contributions of human-computer interaction research' in hildreth, c.r. (ed), the online catalogue: developments and directions, london, the library association. rice, j. (1988), 'serendipity and holism: the beauty of opacs', library journal, 113, 3, 138-41. rodden, t. (1991), 'a survey of cscw systems', interacting with computers, 3, 3, 31954. rudd, j. & rudd, m.j. (1986), 'coping with information load: strategies and implications for librarians', college & research libraries, 47, 4, 315-22. twidale, m.b., nichols, d.m., mariani, j.a., rodden, t. & sawyer, p. (1994), issues in collaborative database browsing, internal departmental report, computing department, lancaster universiiy. yee, m.m. (1991), 'system design and cataloguing meet the user: user interfaces to online public access catalogs', journal of the american society for information science, 42, 2, 78-98. 79 editorial educational technology and the traditional lecture i was recently invited to give a lecture at the opening of a new high-technology lecture theatre at leeds metropolitan university. it is one of the best examples of its kind i have seen. its impressive features include hi-fi surround sound, an enormous back-projected screen giving superb picture quality from either a vcr or directly from a computer for live demonstrations, online facilities, the latest remote-control slide-projection equipment, complete lecturer's control panel, and several nice touches such as automatic dimming of the auditorium lights when play is pressed on any of the hidden video playback machines. the overhead projectors and their screens are of the best quality and correctly positioned for the clearest possible display. there are also video-link facilities for spill-over into a secondary lecture theatre which itself is well fitted out in presentational equipment. clearly, this set-up involved major expenditure. it was therefore presumably discussed at great length before the decision concerning such a long-term commitment was taken. but a commitment to what? to the use of technology in education, obviously, but also to the notion of the stand-up-and-deliver lecture. typically, computer-assisted instruction involves a single student or small group of students sitting in front of a monitor, interacting with some software and self-pacing their learning. the traditional lecture represents the very opposite of this approach: large numbers of students taking notes, with interaction being at best limited and at worst non-existent, and with the pace of proceedings depending almost entirely on the lecturer's judgement. now, i am involved in a shared final-year course in my own university which attracts large numbers of students and for which the traditional weekly lecture is the main communication medium, and i have just attended a staff-student committee meeting at which some criticism of this course was voiced by students taking it. it was not that these students complained of the course being taught in a less than innovative way, but rather that they were being asked to do too much on their own outside class hours, in particular studying secondary topics which cannot be covered in only 25 or so lectures plus some tutorials. in my long experience, university lecturers have always grumbled that students wish to be spoon-fed. but even taking into account any rose-coloured view of the past i may have, i would say that the situation has, if anything, worsened over the past decade or so. these days, we might tend to put the problem accepting that for some teachers in some disciplines, it may not actually be a problem in terms of students preferring passivity to interactivity, or at least we might extend the problem to those terms. for this reason, i am a little sceptical when it comes to the practice about the claimed benefits of courses offering the possibility of experiential and discovery-based learning in higher education. it almost goes without saying that the intentions are good, but the practice, if it is not precisely the streets of hell, is paved with obstacles, not the least of which is a widespread desire among university students to have files full of lecture notes. having lots of neatly filed-away information appears to be satisfying in itself, such that learning can be postponed with a clear conscience until the time comes for revision. and this is true not only in subject-areas where large amounts of factual information have to be delivered. for instance, it is equally true of translation courses in modern languages, where fair copies of translations may be given out by a teacher, to be filed away by the students for future reference, whatever that teacher's 2 aa-j volume 2 number 1 exhortations that such fair copies should form the basis of a regular review of errors, in the students' own time, so that learning is continually maximised. it is equally true of certain science, social-science and philosophical disciplines where the understanding of concepts outweighs factual knowledge, and where notes about a lecturer's view of a multi-sided problem may not be thought about but rather, simply, filed for later consideration as the examination approaches. all this, of course, is part of the traditional criticism of the traditional lecture, but the advent of usable educational technology has made the issue even more poignant not all educational technology is in a primitive stage of development. we have with us, here and now, many tools which allow genuine self-paced exploratory learning at a high level, without those almost insurmountable barriers of a computer program crashing, or occasionally being made to look inadequate (or plain stupid). why, then, does educational technology have to be promoted so vigorously by those who believe in its advantages? indeed, why does alt need to exist? why not simply an association for learning? after all, there exists no association, as far as i am aware, for the promotion of good practice in word-processing, or sending faxes, or using the telephone. two answers to such questions are often given. the first is that educational technology involves considerable financial investment, so that those who decide on how much to spend on what, need to be convinced of its cost-effectiveness. the second is that many students are technophobes. as to the former the question of cost it is plainly true that decision-makers within higher education have to be persuaded that educational technology involves a worthwhile return, both financial and pedagogical. but that argument, taken on its own, can be easily won, and once it has been won, what seemed at first impossible in terms of commitment to spending even in times of cash shortage can actually be achieved. there is also clearly some truth in the existence of student technophobia, but again this can be overcome, and is being overcome in many disciplines in many institutions. no, in my view, the most fundamental obstacle to the acceptance of student-centred learning, with or without the use of technology, is student attitudes to university courses. i would not claim that this is true in all subject-areas in all universities, but i have my own experience to draw on, as well as that of colleagues both in my own institution and many other institutions. the experience of many teachers who have tried educational software is that student reaction is favourable indeed, we have seen that direction clearly emerging in certain contributions to this journal. this must give us cause for optimism. there remains nevertheless the ominous presence of the satisfying, if dusty, file of lecture notes. in a word, somehow we have to change student attitudes to learning, and that is the major challenge, a far greater one than changing the attitudes of students and financial decision-makers to technology. if we succeed in changing such attitudes to learning, what of the traditional lecture? it seems to me that leeds metropolitan university will have got it right on this score provided that the new lecture facilities are accompanied by other technological facilities for small-group learning. but, whatever other facilities are available, the high-technology lecture theatre will in any event have its rightful place. if used well and really good facilities tend to generate their own use (while poor, or even merely adequate, facilities result in technological hardware gathering dust) it allows.for brighter lectures. and since the real purpose of a lecture is to stimulate thought and further study rather than being merely a method of communicating information, having the technological means of making a lecture more interesting, and hence more thought-provoking, can only enhance the learning process. gabriel jacobs 3 computer-based teaching and evaluation of introductory statistics for health science students: some lessons learned nuala colgan, sally mcclean and bryan scotney department of mathematics, university of ulster abstract in recent years, it has become possible to introduce health science students to statistical packages at an increasingly early stage in their undergraduate studies. this has enabled teaching to take place in a computer laboratory, using real data, and encouraging an exploratory and research-oriented approach. this paper briefly describes a hypertext computer based tutorial (cbt) concerned with descriptive statistics and introductory data analysis. the cbt has three primary objectives: the introduction of concepts, the facilitation of revision, and the acquisition of skills for project work. objective testing is incorporated and used for both self-assessment and formal examination. evaluation was carried out with a large group of health science students, heterogeneous with regard to their it skills and basic numeracy. the results of the evaluation contain valuable lessons. rationale traditionally, the teaching of introductory statistics to health sciences students has been based on a series of lectures in conjunction with practical classes using an electronic calculator. the use of a statistical package such as bmdp or glim has usually been restricted to advanced classes encountering more sophisticated techniques. however, the increasing availability of user-friendly packages designed for teaching, such as minitab, has meant that it has become possible to introduce elementary classes to a statistical package at an early stage (stephenson, 1990). in turn, this provides the opportunity for analysis of real data right from the outset of the student's statistical learning, giving experience of problems as they arise in practice (singer and willett, 1990). the use of computers in nursing education is considered to be particularly important as a means of preparing nurses for an increasingly computerized workplace (yucha and reigeluth, 1983). a teaching and learning approach which combines the use of statistical software with computer-based tutorials provides flexibility with respect to tailoring information to the specific needs of individual types of users, and allows the incorporation of user68 alt j volume 2 number 2 interaction (richards et alt 1991; barker and manji, 1991; bishop et al, 1992). the computer based tutorial (cbt) we have developed at the university of ulster is concerned with descriptive statistics and introductory data analysis using minitab, and is part of a suite of cbts developed under the funding of the enterprise in higher education (ehe) initiative. the tutorial is targeted specifically at health sciences students, and uses data sets relevant to their subject of study as examples. the tutorials are structured so that they can be used for directed learning and assessment to introduce concepts, and student-centred learning to facilitate revision and the acquisition of skills for project work. objective testing is also incorporated into the tutorial which may be used for either selfassessment or formal examination. content the cbt introduces the basic ideas and applications of descriptive statistics, with reference to minitab in applying statistical techniques. it aims to provide the student with an intuitive understanding of statistical concepts, and a variety of relevant examples are used to illustrate the techniques. the tutorial is designed to develop the student's ability to organize, present and interpret data in a number of ways such as tables, graphs and summaries. in addition, it aims to introduce the student to the concept of statistical inference, and enable him or her to carry out and interpret simple hypothesis tests. various screens within the tutorial include output from minitab, with an explanation button, or a short note which may be expanded to give additional information. an example is presented in figure 1, illustrating the minitab command desc which is used here to produce summary statistics for the pulse rates of respondents who did not run before their pulse was measured. the explanation button provides information on how to calculate the median and semi-interquartile range. for the non-runners mtb>descc17 produces: n mean c17 57 0.105 min max c17 -14.000 10.000 median 0.000 q1 -2.000 trmean -0.137 03 2.000 jggicirifiii stdev 4.1 el semean 0.551 flsjiswtlohiij figure la typical screen 69 nuala colgan, sally mcclean and bryan scotney computer-based teaching on completion of a section, the student may undertake to answer a series of questions for self-assessment. these may require a textual answer, or a choice of one or more appropriate answers from a selection provided. delivery the tutorial has been developed using version 3.1 of the authoring system guide running under windows and under the management of a novell network which allows the network manager to collect individual student answers from the respective work-stations, and to present them in a file for marking by the class instructor. in our trials, the students were introduced by the lecturer to the system prior to starting to use the cbt, initially in a lecture theatre, and subsequently in a computer laboratory. each topic was introduced verbally to the students, as a group, before they began to use the appropriate section of the tutorial. in order to facilitate independent learning, the cbt was accompanied by a printed booklet which explained the essential features of the system. the students were allowed to work through the tutorial during class, and in their own time, and both individual and group use of the system was encouraged. the use of the cbt was piloted with a group of 70 nursing students as part of an introductory course on research methods. these students were judged to be particularly appropriate for introduction to the subject via a cbt as they provided a means of allowing a large heterogeneous group with regard to it skills and basic numeracy an opportunity to familiarize themselves with the material at their own pace. previous cohorts had expressed dissatisfaction with the course as they found the class size too large and the topic difficult to master, and expressed a lack of conviction that either statistics or computing was needed for nurse training. it was hoped that the cbt could help to overcome some of these problems. prior to the introduction of the tutorial, students were taught using a combination of formal classes and computer-laboratory sessions. although it was deemed essential for future project-work that they learned to use the computer for statistical analysis, the laboratory classes often proved problematic as a few students with learning difficulties tended to monopolize the lecturer's time at the expense of the more able students. under the new system, students used the software as part of a series of one-hour classes held in two inter-connected 30-workstation networked laboratories. during class, the lecturer, assisted by a demonstrator, was present to answer questions and trouble-shoot. students were allowed to access the software outside formal teaching times. evaluation at the end of the course, the students were evaluated by means of a confidential questionnaire in order to ascertain how useful they had found computer-based learning, and their views on the system. the questionnaire is presented in appendix 1. it was completed by the 70 students of whom 67% had managed to complete all four sections of the tutorial, while 17% had completed three out of four during class. 11% had completed two sections, and 5% had completed only one. in all, nearly three quarters of the group managed to complete them during the class, so the volume of material was about right for the amount of time allocated during timetabled contact. of those students who completed all four sections of the tutorial, 74% often or always completed them during the 70 alt j volume 2 number 2 timetabled classes. overall, 73% never attempted the tutorial outside class. this aspect of students' use of the cbt may be partly accounted for by the fact that 23% of the class were part-time students for whom on-campus time was severely limited, but it is still rather disappointing. in all, 67% of students found the cbt helpful and 33% found them unhelpful. there was no association between success at completing the tutorials and whether a student found them helpful, with 65% of students who failed to complete all four sections finding them helpful, as opposed to 67% of those who completed all four of2 = 0.038 with 1 degree of freedom). all (27%) students who attempted the tutorial outside the timetabled classes reported that they found it more useful outside than during class. overall, the response as to how useful the students found the tutorial was again disappointing. 64% said that they did not feel discouraged if they scored poorly on the end-of-section self-assessment tests, and 37% said that they found the tutorial beneficial for reinforcing understanding, 11% for increasing confidence, 54% for providing explanations, 20% for providing a change from minitab, 39% for reviewing material, 50% for testing understanding. on the other hand, only 3% said that they found the tutorial beneficial for providing understanding. furthermore, 33% reported that they did not enjoy the tutorial, as opposed to 47% who did not mind it. only 20% reported that they positively enjoyed it, and only 31% thought they would use computer-based instruction again if given the opportunity, while 44% would not use it. the remaining 25% were unsure. 32% preferred to work alone while 68% preferred to share a computer and discuss the answers with someone else. the multiple-choice questions were found useful by 68%, but only 22% regarded the instructional level to be appropriate to their needs. 55% of students did not find the instructional booklet useful, with no association between this comment and whether they had worked alone or together (x2 = 6.45 with 3 degrees of freedom) or whether they had used the tutorial outside the timetabled classes (g2 = 0.053 with 1 degree of freedom). however, a majority (55%) thought that the tutorials were a useful revision aid, with a further 33% unsure, and only 12% who did not think they were useful for revision. the evaluation questionnaire offered the students the opportunity to raise any additional issues; ones which were frequently mentioned, and which are discussed in the next section, were that the classes were too large, that they were inadequately experienced in using computers, that they preferred ordinary lectures, and that they would like a more comprehensive induction programme prior to using the cbt and minitab software. discussion in terms of completing the cbt, the students successfully used the tutorial, with two thirds of them completing all four sections and three quarters of them completing it within timetabled classes, and it is encouraging to note that many students found it helpful, particularly as a means of increasing understanding and as a revision aid. what is more, two thirds of the students found the tutorial helpful, or did not mind it. however, the figure of 44% of students who stated that, if given the chance, they would not use computer-based learning again has provoked us seriously to examine the shortcomings in both the delivery of the tutorial and the way it is embedded in the course. 71 nuala colgan, sally mcclean and bryan scotney computer-based teaching some of the negative comments are concerned with problems of the course rather than the delivery mechanism. for example, students expressed a dissatisfaction with large class sizes, which meant that individual assistance was limited. the most frequently voiced concern, however, was that students felt that much more initial explanation was required from the lecturer of both the structure of the tutorial and its statistical content. some students expressed a preference for a short verbal introduction at the beginning of each laboratory session; others felt that the whole of the first few classes should follow a traditional lecture/discussion format to establish the teaching and learning of the statistical content, and only then should work be predominantly laboratory based. despite the fact that the use of minitab and the cbt requires very little knowledge about the machine or its operating system, many students actually saw the use of computers as a 'complication' with which they felt unable to cope until they had mastered the statistical material by more traditional means. this was certainly at variance with our desire to enable students taught in a computer laboratory to have immediate hands-on experience of the methods being taught, and also with our hope that thus liberated from the drudgery of calculators, they might focus their attention on the statistical methods under consideration. we had anticipated that the teaching method would enable the students, from the outset, to adopt a more exploratory and research-oriented approach with a focus on methodselection rather than calculation. but many students felt that their own limited knowledge of computers was an obstacle to adopting such an approach. while some students felt that by being encouraged to develop a more critical approach to the use of statistics they would be better equipped to cope with project work, they felt that the tutorial would be of greater benefit at a later stage in the course in relation to clinical practice. a small number of students still expressed a lack of understanding as to why they needed to use the computer at all. the clear lessons in all this are that in future years we shall focus more on motivating students at the beginning of the course, and on providinga more structured introduction to the use of the computer and the use of computer-based learning. this may in part be achieved by accepting the need for the students to undertake the learning of basic statistical concepts by more traditional means prior to their exposure to computer-based approaches. together with a reorganization of delivery which would permit greater access to the cbt outside of timetabled classes, the students would thus 'gain control of their learning' (isaacs, 1990). in any case, the entire approach must be underpinned by a commitment to the relevance of computer-based statistics to both health science research and clinical practice. references barker, p. and manji, k. (1991), 'designing electronic books', educational and training technology international, 28, 4, 273-80. bishop, p., beilby, m. and bowman, a. (1992), 'computer-based learning in mathematics and statistics', computers in education, 19, 1/2, 131-43. issacs, g. (1990), 'course and tutorial cal lesson design: helping students take control of their learning', educational and training technology international, 27, 1, 85-91. richards, s., barker, p., giller, s., lemont, c. and manji, k. (1991), 'page structures for electronic books', educational and training technology international, 28, 4, 291-301. 72 altj volume 2 number 2 singer, j.d. and willett, j.b. (1990), 'improving the teaching of applied statistics: putting the data back into data analysis', the american statistician, 40, 3, 223-30. stephenson, w.r. (1990), 'a study of student reaction to the use of minitab in an introductory statistics course', the american statistician, 44, 3, 231-5. yucha, c. and reigeluth, c.m. (1983), 'the use of computers in nursing education, practice and administration', computers in education, 7, 4, 223-6. appendix 1 guide computer-based tutorial it would be very helpful to me if you would provide some feed-back on the guide computer-based tutorial which we have been using during the lab sessions in the unit ns106. your views will help me to plan the development and use of computer-based tutorials in the future. i would appreciate you spending just a few minutes answering the questions below. your answers will be anonymous. please circle the appropriate answer in each case 6. in which of the following ways do you feel that the guide tutorial is beneficial? (circle as many as you want) reinforcing understanding increasing confidence providing explanations providing a change from minitab reviewing material testing understanding providing interest other (please list) 1. the tutorial contained four topics in the main menu: 1 summarising data; 2 manipulating & describing data; 3 associations for nominal data; 4 scatterplots & correlations. how many of the four topics have you completed? 1 7. in what ways do you feel that the guide tutorial is not beneficial? (please list overleaf as many as you want) 8. d o you enjoy the guide tutorial? very much quite a bit don't mind not much not at all 2. have you completed the guide tutorial topics during the timetabled classes always often occasionally never 3 . have you attempted the guide tutorial outside the timetabled classes always often occasionally 4. how helpful do you find the guide tutorial? very fairly not very not at all 5. d o you feel you get more out of the guide tutorial during the timetabled classes or outside the timetabled classes? during outside 9. if you score poorly on a multiple choice part of the guide tutorial, do you feel discouraged ask for help try again review the material (you may wish to circle more than one) 10. in the multiple choice part of the g u i d e tutorial, how easy do you find it to understand what the questions are asking you to do (as distinct from how easy they are to answer)? very easy easy ok usually occasionally usually difficult difficult 11. d o you prefer to work on the guide tutorial at a computer on your own, or do you prefer to share a computer and discuss the answers with someone else? alone share 73 nuala colgan, sally mcclean and bryan scotney 1 2 4 0 are you a student on bsc nursing bsc prof. dev. nursing other (please specify) (b) which year of your course are you in? 1 2 3 (c) are you full-time or part-time? full-time part-time computer-based teaching 13. please indicate your level of agreement with the following statements about the guide tutorial by circling 5 , 4 , 3 , 2 o r l strongly strongly agree disagree (a) material is well 5 4 3 2 1 organised (b) information on screen 5 4 3 2 1 is clearly presented (c) instructions on screen are easy to follow (d) multiple choice exercises are useful (e) the instructional level was appropriate to my needs (i) i used the accompanying booklet a lot (g) the accompanying booklet was very useful (h) the guide tutorial is a useful revision aid (1) i would use computerbased instruction again if given the opportunity 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 14. if you have any additional comments, please write them overleaf thank you for completing the questionnaire. bryan scotney (7/5/93) 74 establishing a support service for educational technology within a university j. a. longstaffe, p. j. williams, k. d. whittlestone, p. m. hammond and j. edwards educational technology service, university of bristol this paper covers some of the issues associated with the support of learning technology within an institution. it describes the activities of a working support service, and highlights approaches to some of the academic, technological, political and management issues that have helped it perform effectively over a four-year period. introduction over the past decade it has become increasingly apparent that computer technology is extremely effective in the support of teaching and learning. it has also become obvious that without proselytization and support, the adoption of this new method of teaching is patchy and frequently inappropriate. the raising of awareness, the training of staff and the provision of informed advice and support are necessary to facilitate the appropriate development of technology-supported learning within an institution. the support paradigm which has emerged so far has two parts. the first consists of national discipline-based centres. these exist mainly within the computers in teaching initiative (ctt) network and possess specialist subject knowledge as well as a knowledge of good practice within the academic field in question. although personnel in the cti centres have an ideal constellation of the necessary skills and knowledge, they are not available to academic staff in an individual institution on a day-to-day basis. there is therefore a need for local institutional support complementary to that supplied by national centres. jonathan darby, in his comprehensive picture of uk learning technology in 1992 (darby, 1992) refers to the fact that although a number of universities have developed their own cbl (computer-based learning) units, with a few notable exceptions these are often limited in scope and frequently concentrated solely on it skills. the university of bristol has drawn on the early experience of units such as the institute for computer based learning (icbl) at heriot-watt university, but has also looked closely at the experience and needs of individual academics as gathered by the staff of the cti centre for medicine since its inception in 1989. 85 j. a bongstaffe et al establishing a support service for educational technology within a university this paper seeks to encapsulate some of the structures, strategies and experiences which have arisen from the first four years of activity in a relatively modestly funded university-based educational technology service. structure and activities in order to achieve its remit of 'promoting and supporting the appropriate use of technology in teaching and learning', the educational technology service (ets) of the university of bristol undertakes three categories of activity, namely service, focused and academic. service activities raise awareness and inform client staff by means of a newsletter, an electronic mail list, a seminar programme, open days and a comprehensive advice service which includes technology, instructional design, project design and management, proposal construction and the use of global networks such as the internet (browning and williams, 1995). focused activities consist of support for a number of defined learning projects each year. this has included evaluation of learning needs, acquisition of learning software and resources, and the production of learning material by university staff and its integration into the curriculum. these projects are carried out in collaboration with individual departments and are aimed at producing a measurable change in the quality and efficiency of a particular course or module. productivity tools such as authoring templates for the production of interactive learning material and software for image digitization, storage and manipulation are developed to support these activities. academic activities cover the normal range of research and publication. these projects serve to expand the range of staff and expertise available to the university, and many form collaborations with other institutions. academic issues over the four years of the existence of the ets, a number of issues and strategies have proved important. this paper concentrates on a few of the more significant ones; others are discussed in longstaffe et al (1995). in terms of academic issues, it is important to understand some aspects of the culture and the people involved in higher education before considering some of the approaches that it might be useful to adopt. there are three major factors in the university environment that must be taken into account when supporting learning technology. these are described here as individualism, teaching paranoia and time pressure. individualism most teachers in universities have grown up in a tradition which involves the preparation of a highly personal course, with much reference to printed material, but little to academic colleagues. learning technology does not generally lend itself to such an individual approach. it is more effective when produced by teams or consortia which are likely to possess the mix of skills necessary. a collaborative attitude to the production of learning materials needs to be fostered by the staff of a support service. locally, this has not only had the effect of generating 86 alt-j volume 4 number i better-quality materials than in projects driven by individuals, but has also made materials more generally acceptable. the advantages of the team approach are further supported by duguid (1995). teaching paranoia university teachers often have little or no formal training in teaching or educational theory. most academics are aware of this and, as a result, are not unnaturally reluctant to expose themselves by preparing material which will be more public than their usual lecture or tutorial. it is necessary to provide sympathetic and constructive support which will include tactful training in aspects of instructional design, course design, evaluation and so on. it is becoming apparent to the authors of this paper that as teachers become involved in the production of materials of this sort, their criticism of similar work becomes more constructive. time pressure the entire university culture is under pressure from reductions in funding, increased student numbers and administrative load. this is not an easy environment in which to expect the investment of large amounts of staff time. fortunately, there are always enthusiasts willing to commit quite unreasonable amounts of time to projects. tactful direction and support of these enthusiasts has produced good results locally. in addition, strategies such as the use of sabbaticals and the student production of computer-aided learning have worked extremely well in individual cases. student production of cal is a regular feature of the veterinary anatomy curriculum at bristol, and has produced a measurable increase in performance in conventional examinations (garvin and carrington, 1995). differing levels of staff expertise it is important to remember that no university academic population is uniform with respect to expertise in learning technology. we find that our 'clients' divide into sophisticated, knowledgeable and naive groups. in 1991, a small cadre of sophisticated staff were already producing and delivering material with a degree of expertise. support targeted to this group has been specialized, such as help with image recording and manipulation, cd pressing and the digitization of video. with good contacts, it has also been possible to help the departments in question with locating, interviewing and appointing staff appropriate to particular jobs. a larger and growing group comprises those who are knowledgeable about cal and who are engaged in its production or at least sufficiently motivated to begin doing so. we have found that this is the most important group to target, and the one which has produced the most results per unit-effort. understandably, the majority of teaching staff can still be termed naive with respect to learning technology, although they may have a range of it literacy. the ets has raised awareness with a departmental seminar programme on the basics of cal backed up with printed and electronic information, workshops and demonstrations. visits from relevant cti centres have created a very useful start with several departments. once a naive department starts to move towards the knowledgeable, assistance with the formation of cal committees, the planning and design of learning projects and the construction of effective grant proposals have also been shown to be successful strategies. 87 j. a longstaffe et ol establishing a support service for educational technology within a university technological strategies these strategies are not so much about the technology itself, but constitute ways of coping with the technology. it is important, for instance, to be able to advise on how close to stay to the technological cutting edge: too close, and a project can get bogged down in bugs and incompatibilities; too far away, and opportunities can be lost. guidelines can be broken in both directions, but it is important to have them first. a second technological strategy is to try to avoid the 'high-tech fascination' that seems to lure many enthusiasts into committing too much energy to ultra-sophisticated prototypes which never see the light of day. the 'keep it simple' approach is vital. another observed problem is the failure of clients to understand, or technologists to communicate, the nature of the software development cycle. any piece of developing software must undergo numbers of iterations between producers and computer-tolerant users before it becomes effective. this is often extremely time-consuming for the users, who must be briefed accordingly. too often, projects have been seen to under-achieve because of unreasonable promises on the one hand and unreasonable expectations on the other. other technological strategies include coping with conflicting delivery platforms, keeping data separate from authoring structures, and finding ways of coping with the moving technological goal posts. political and management issues ways to relate to university senior management and to complementary units such as computing services and libraries must be devised, and the political and financial location of the service established. the location of the ets within a school of education has been successful both in terms of available expertise and credibility. one important concept to bear in mind is that of the 'multiplier effect'. the focused training and support of a relatively small group of individuals will create a growing body of expertise and models of good practice which in turn support and inspire others. our own dental school has progressed in this way over five years from a single enthusiast to the replacement of 98 hours of the undergraduate syllabus by cal materials. the bristol dental school is now regarded as a national leader in the field. a service of this type must also manage itself in a highly conscious and flexible manner. most of its activities will not have been performed before, and it is vital that reviews of policy and direction are frequent and effective. the ets has developed a set of activities which are designed to effect its mission statement. these activities form a team job description which devolves into individual job descriptions. all activities are subject to an annual review which is informed by monthly brainstorming sessions. these focus on a particular aspect of servicefunction or client-need. quality and value for money are important measurements for senior management to use in forming opinions about the optimal placement of funding. in order to achieve some sort of measurability, we return to the purpose which attaches to each group of activities, and then check to see how well those purposes are being met. parameters such as departments producing cal, departments delivering cal, student contact hours and staff time saved are useful quantity indicators. quality can be judged by staff and student feedback and assistance with rapid evaluation protocols (edwards, 1995). 88 alt-] volume 4 number i organizational models for institutional and departmental support within the university the ets, based academically and politically within the school of education, relates closely on a day-to-day basis to the computing service, staff development, the library, the school of education and other units such as audiovisual services and the centre for video publishing. staff training activities have, as a result, become virtually seamless, with staff from all units working in a collaborative fashion. with this sort of interdepartmental co-operation, it is becoming easier to identify training needs and satisfy them using knowledge and expertise located in the various units. we have compared our activities and approach with other universities, notably aberdeen, heriot-watt, nottingham and southampton. it is refreshing to note that, in general, they are grappling with similar problems in much the same way. staff awareness, training and advice are dealt with in a 'service' manner with the production, acquisition, evaluation and integration of learning material being supported on a project basis. comparing our own approach to the complex relationships between academic and technological support units with those of other universities, it has become obvious that while a number of intra-university organizational models exist (ranging from the discipline-based initiatives at the university of aberdeen to the more institutional-based approaches of nottingham, heriot-watt and southampton), the problems, strategies and solutions remain very similar. the key element seems to the effectiveness of collaboration and awareness between units rather than any single organizational model. relationship to national initiatives the most significant and useful initiative in terms of the needs of the ets and its client academics is the the discipline-based cti network. the ability to call in a ctt centre with its discipline-based expertise, its networking and its knowledge of relevant materials expands the ability of a local service to create a positive effect in its community. a cti visit to a relevant department has on several occasions been the trigger which has led to its serious involvement in learning technology. ets staff have contributed to such visits, assuring personnel of the ongoing support and activities available locally. ( in addition to the cti, the development of the tltp regional centres, as well as initiatives such as the scottish learning technology development initiative (ltdi), clearly have a further positive effect. provided that proper liaison is established between national and local initiatives, the synergy developed can only help with the enormous task of effecting the cultural change within higher education which is so necessary to its future. integration of the work of the service with individuals and departments the section above deals with the relationship between the ets and other learning-support units within the institution. turning to the ways in which at the grass roots the ets integrates its work with individual academics and departments, some of our experiences may be useful. with respect to focused or departmental projects, we have found academic ownership of the project to be essential. with assistance in project management, our most successful projects (in the department of professional legal studies, in dentistry and in veterinary anatomy, for 89 j. a longstaffe et o\ establishing a support service for educational technology within a university instance) have been clearly owned and driven by academics within the departments. on the down side, this has sometimes led to isolationism by departments, and unwillingness to involve ets staff other than as service providers. this, in turn, has led to the consequent reinvention of some wheels. support and enthusiasm from a departmental head is an extremely powerful factor in the success of learning projects, even if this person is not directly involved. conversely, negativity from senior personnel has led to loss of morale and momentum in some cases. one example of academic staff 'directed' to produce cal material led to many delays in the production of that material, extra work for ets staff, reluctant communications and severe delays in implementation. another stimulus for successful projects has been the award, by the university, of a number of grants totalling £100,000 per annum to departments for the development of 'excellence and innovation in teaching and learning'. the participation of the ets in, first the structuring of proposals for this money and subsequently support for successful projects, has had a strong influence on both the high proportion of money awarded for technology-based solutions and the ultimate success of these projects. another stratagem is diat of constant personal contact and encouragement. many busy academics, left to their own devices, will procrastinate, delaying a cal project to a severe degree. gentle support, follow-ups, reminders and offers of help mean that project development stays high on personal agendas, making sure that academic time is made available within a reasonable timetable. finally, one of our most successful strategies has been to produce a simple template associated with one of the commercial authoring systems (asymetrix toolbook. we support this with one-day introductory and follow-up seminars. this template has now been used to produce some hundreds of tutorials in use with students in disciplines as separate as dentistry, anatomy, law and modern languages. evaluation of the impact of the service this has not been an easy task. because it is essential that academic departments have ownership of the various learning projects, it is difficult to apportion credit between the support service and the client department. in the early years, the impact can be measured, as in figure 1, by demand indicated by numbers of departments advised or trained. another useful measurement in the early stages is departmental involvement in the production and delivery of technology-based learning materials. we have more recently introduced such parameters as student contact hours and staff time saved. it is our experience that these measurements need to be planned for in advance: it does not usually occur to staff responsible for the development and delivery of learning material diat the measurements could be important. the ets is also assisting departments to evaluate their own and imported materials incorporating student feedback and attitudes. at this stage, most of die evaluations are informal, with lack of staff time proving a barrier to more formal processes. however, they are positive, and staff are becoming more evaluation-aware, and planning better monitoring in the future. finally, and encouragingly, we are beginning to receive positive and complimentary feedback 9 0 • • -s volume 4 number i m departments advised by ets • departments producing cal • departments delivering cal 10 1990/1 1991/2 1992/3 1993/4 1994/5 figure ! from the national teaching quality assurance (tqa) visitations (for example, in one of our modern language departments) as well as from external examiners (for example in veterinary anatomy). conclusion the impact of the ets has been considerable. the number of departments advised has risen to 45 in the last four years. in the same period, the number of departments producing cal materials has increased from 3 to 30, and those delivering some form of cal to their students has increased from 3 to 21 (see figure 1). it is important to remember that a support unit in learning technology is not just a technical or even an academic service, but an agent to effect and underpin cultural change within an institution. it is also worth noting that activities in support of the adoption of learning technology have a number of unanticipated and significant side-effects. client academics have demonstrated improved teaching ability, have developed project design and management skills, have structured collaborative teaching, and have acquired it skills. transferable skills such as these are already seen to be important in students, and their development is of course equally important in staff. references browning, p. and williams, j. (1995) 'the geology@bristol experience', active learning, 2, 34-8. darby, j. (1992), 'computers in teaching and learning in uk higher education', computers and education, 19(1/2), 1-8. duguid, k.p. (1995), 'the team approach to the design of computer-assisted learning packages in medicine', journal of audio-visual media in medicine, 18 (2), 53-8. 91 j. a longsto/fe et al establishing a support service for educational technology within a university edwards, j. (1995), 'judging the usefulness of courseware', journal of audio-visual media in medicine, 18 (2), 83-4. garvin, g.p. and carrington, s.d. (1995), 'student-authored computer-assisted learning: the teaching and learning outcomes of a programme of assessed group project work producing hypermedia peer-tutorials in veterinary anatomy', proceedings of the third cticm conference on computers in medical education, bristol, cticm (in press). longstaffe, j.a., whittlestone, k.d., williams, j., hammond, p.h. and edwards, j.d. (1995), supporting learning technology in higher education, itti monograph series, sheffield, cvcp/usdu. 92 alt-j volume 2 number 1 intelligent approaches to performance support philip barker, stephen richards and ashok banerji interactive systems research group, school of computing and mathematics, university of teesside, uk abstract electronic performance support systems provide an important method of meeting on-demand educational and training requirements. they also provide efficient and effective ways of enabling the knowledge and expertise within an organization to be shared. this paper discusses the design of a distributed electronic performance support system and the ways in which 'intelligent agents' based on expert systems and neural networks can be used to locate and share distributed expertise. a case study illustrating our approach to the implementation and use of intelligent agents is presented. introduction the performance of virtually all organizations depends on a wide range of internal and external factors that can interact with each other in many complex ways. one very important aspect of performance is competitivity the ability to perform organizational tasks in ways that are more efficient and more effective than those of other organizations. research has shown that the competitivity of any given organization in the manufacturing and services sector largely depends on four critical factors (churchman and cerhulst 1963; beishon and peters 1972): (1) a flexible management structure that is able to take advantage of the latest decision-making tools; (2) the availability of appropriate facilities to enable the organization to perceive and understand changes and trends in market forces; (3) an adaptable and responsive workforce; and (4) the ability to make information and knowledge available at critical points of need. such aspects of organizational activity are, obviously, also broadly applicable to educational institutions, not least since each aspect involves people and human behaviour. in order to be successful and competitive, it is therefore imperative that an organization provides appropriate opportunities and facilities to enable its workforce to perform jobs to maximum effect in an ergonomically pleasing way. one approach to realizing this objective is through the use of various types of performance support system. a performance support system is essentially a 'job aid' that enables its users to improve the efficiency and effectiveness with which they are able to complete particular tasks. a simple example of a performance support system is a pocket calculator. word-processing systems and 63 philip barker et al. intelligent approaches to performance support desktop-publishing packages are further examples: they significantly improve the efficiency with which their users can create documentation. email and computer-supported co-operative work (cscw) techniques are also important performance aids since they enable geographi1 cally distributed organizations to undertake rapid inter-site and interpersonal communication, to share intellectual and managerial resources, and to participate in collaborative problemsolving at a distance. high-performanee, general-purpose, multimedia, interactive, computer-based workstations now provide a powerful environment within which to implement a new type of sophisticated electronic performance support system (epss). such systems can be designed to perform a number of generic job-aiding functions such as the provision of help and advice, the sharing of organizational expertise, and access to large volumes of information and documentation. also, in situations where it is appropriate, an epss can provide on-demand instruction at a particular point of need. this latter function can be achieved by means of a just-in-time (jit) paradigm which can be realized through interactive on-the-job training techniques. jit training methods involve providing instruction to an individual as and when it is needed in order to perform a particular job function effectively. the concept and potential of epsss is well-documented in the literature (see, for example, gery 1991; barker and banerji 1993; and whitlock et al 1993). one important aspect of epss design and implementation is the provision of organizational documentation in electronic form. this requirement is necessary for a number of practical reasons, but primarily because of the large volumes of information that are involved and the ws. = multimedia workstation isdn = integrated services digital network electronic books experts employees types of electronic book: 1: common shared books 2: private local books 3: external public books electronic books experts experts v j / employees electronic books figure 1: a distributed electronic performance support system 64 au-j volume 2 number 1 ease with which electronic documentation can be accessed, updated and shared. as part of our epss research we have therefore been exploring the problems involved in converting conventional paper-based documentation into electronic form. as an alternative publication medium we have been using cd-rom to produce a series of electronic books for use with performance support systems (barker 1991; barker 1992). a range of different types of publication have been produced, containing textual information, multimedia information (text, pictures and sound) and hypermedia material. these electronic publications embed the information necessary to facilitate many different aspects of organizational activity, and a range of storage and retrieval techniques has been used in order to provide access to the information held on the cd-roms. our previous research into generic epss structures has been described in detail elsewhere (barker 1991; banerji 1993). this paper now describes the ways in which we have been using electronic books to provide intelligent approaches to performance support within a distributed epss facility. the basic type of system we have been considering is illustrated schematically in figure 1. the implementation depends on the creation of three basic types of electronic-book publication: common shared books, private local books, and external public books. within our system, 'intelligent agents' are used to keep track of an organization's expertise within different problem-solving domains. these agents can be used to locate sources of relevant expertise, and to facilitate the transfer of this expertise (in electronic form) to the required point of need. this transfer can be accomplished using a dedicated network or, for example, the integrated services digital network (isdn) when large amounts of data have to be transferred rapidly over public telephone lines (for a technical overview of isdn, its limitations and practical potential within education and training, see jacobs 1994). development tools for intelligent agents a range of different artificial-intelligence techniques can be used to develop intelligent agents for use within distributed epsss. two of the most important types of generic.tool we have been using in our work are expert systems and neural networks. an expert system is a software tool containing domain expertise which can be used to solve problems within a particular subject area. most of the commonly available expert systems model domain knowledge in terms of collections of 'if-then-else' rules (hart 1986). essentially, these rules constitute a decision tree which, given the necessary input parameters, reduces a large problem-space to more manageable proportions. the rules then provide relevant sources of advice about the problem being solved. the major limitation of expert systems, however, is that they cannot learn: all the rules used have to be coded into an expert system by human experts before it can be used to solve problems. a neural network system is a computing facility modelled on the basic way in which the human brain is thought to work (beale and jackson 1990). such systems consist of a large collection of artificial neural elements interconnected in various ways in order to solve particular classes of problem. the interconnections within a given neural network are formed by a process of training. during its training phase, a neural network learns what connections are necessary in order to solve a given type of problem. once the training phase is complete, a trained neural network can be put to work within the problem domain for which it has been created. 65 philip barker et al. intelligent approaches to performance support in many situations, tools can be constructed which depend on the useful features of both expert system and neural-network technologies. this is the approach we have adopted in order to create the intelligent agents used in the case study described below. in the example presented, the expert system is used to embed rule-based 'common-sense' knowledge, while a set of neural networks is used to encode 'deep' compiled knowledge. each neural net represents a different conceptual view of a knowledge domain within a particular distributed epss facility. obviously, within the distributed epsss we are developing, the expert system and the neuralnetwork components have to be integrated in a transparent and seamless manner. they also have to be combined with those other components of the epss that are needed for the particular application being considered (email, text retrieval, simulation, computer-based instruction, and so on). we will now describe in more detail the way in which these objectives are realized. case study views of knowledge the work we have been undertaking is aimed at finding ways of sharing expertise and knowledge within a distributed epss facility similar to the one illustrated in figure 1. the particular aspect we have been exploring relates to the sharing of different views of a common knowledge domain within a distributed environment. the basic structure of the system we have been developing is illustrated schematically in figure 2. in this particular work, the knowledge pool consists of large collections of text-based material that have been published in the form of electronic books on cd-rom. terminal communications facility epss shell expert system neural network intelligent agents window data base full-text retrieval browsers knowledge pool (electronic books) figure 2: use of intelligent agents in an epss facility 66 alt-j ' volume 2 number 1 fundamental to the basic epss shell is its system database, used to organize and control access to all the underlying information. a user has full access to all the available material through the mechanisms associated with conventional computer-based full-text retrieval (heaps 1978), book indexes and book contents lists. of course, if necessary a user could also simply browse through the material in the knowledge pool using a range of different browsing techniques, but because of the significant volumes of information involved this approach would be time-consuming and inefficient the full-text retrieval method is difficult to direct with regard to context. this can be both an advantage and a disadvantage. it is an advantage in the sense that all information associated with a given set of keywords is retrieved. on the other hand, it is a disadvantage in that high degrees of relevance are often difficult to achieve; that is, the problem with full-text retrieval is that it can retrieve a substantial volume of information which is irrelevant, even when boolean search techniques (and, or etc.) are used. within the system we have been developing, users are able* to initiate both 'direct' or 'assisted' information access. direct access involves invoking either the full-text search engine, the book browser, or a direct query to a specified neural net (that is targeted towards a particular part of the knowledge pool). such direct access depends on a user having a clear idea of both problem formulation and knowledge-pool organization. while this may be feasible with small amounts of knowledge, it can become a problem when access to information from a large corpus is required. an alternative strategy involves a user seeking guidance from an expert system for direction as to the most suitable access method based on his/her current orientation to a particular problem. in this situation, the expert system uses basic rules about the knowledge pool and the user's problem definition in order to identify the strategy most likely to yield the desired information. if a user chooses an approach based on a neural network, the expert system will access the system database in order to obtain the necessary information (keywords and their associated weightings) to supply to the neural-network engine. because people differ in their views of knowledge, different views of the knowledge pool can be constructed by changing the weightings assigned to the keywords. this information is used to create a new window (or 'view') onto the knowledge pool through the production of a new net (see figure 2). the resulting net, with particular values of keywords and strengths, constitutes the compiled version of various experts' knowledge. these nets provide multiple views of the information for instance, from the perspective of a particular type of expertise, context, interest or problem area. we are currently using two strategies to create the information needed to train our neural networks. one of these is automated, while the other (manual method) depends on the encoding of knowledge by an expert. the automated approach uses software tools, which we have developed, in order to identify potential keywords and calculate their frequency within the various sections of a document. these frequencies are then used to calculate weightings and produce a training set for the neural-network engine. a typical training set could be as follows (where kw = keyword and w = weighting): input: kwl,w3; kw2,w2; kw4,w5; output: a input: kwl,w5; kw3,w4; kw5,wl; output: b the expert encoding technique relies on an expert to input relevant keywords and/or phrases, along with respective weightings, into our training-set generation tool. the latter technique 67 philip barker et al. intelligent approaches to performance support requires greater production time and effort but provides increased accuracy. when using the delivery system, a net appropriate to the current task is selected either manually by the user or automatically by the expert system. information about the current problem, consisting of keywords and relative weightings from the user, is then encoded into a query. this query is passed, along with the appropriate neural-net information from the database, to the neural network engine. information is returned in the form of a list of locations in order of relevance to the current problem. direct access is then available to the locations suggested. only on the occasion when such a strategy fails will the user need laboriously to access every instance of a particular set of keywords using the full-text retrieval engine, a major advantage of a neural-net approach is the potential for the various networks to continue learning about the knowledge pool from interaction with expert users. keywords can be added or removed, weightings can be adjusted, new knowledge can be added to the pool, and new training sets can be produced which allow the first-time hit rate of the networks to be improved. in this way, an information environment is created which can learn and adapt to constantly changing environments. obviously, such a strategy is also very useful when organizational expertise is distributed. for example, in on-the-job situations, information requirements are often immediate. if a particular individual does not have the necessary expertise, then very often he/she will consult the organization's expert in that particular field. in a situation where the expertise is at a remote site, or simply unavailable, efficiency is obviously diminished. even in cases where the remote expert is available, it can take some time for that expert to locate the particular information required. the distributed system we are developing goes some way to reducing such problems. when a query is made to an expert, the expert has access to a number of neural nets providing windows on the knowledge pool which are tuned to particular perspectives or sets of problems. this allows the expert quickly to obtain the required information and respond promptly to the remote request. in addition, the expert's neural nets can be sent to the remote sites whenever any new learning has taken place. this means that in the absence of an available expert, any individual can formulate a query based on a combination of the expert system and the relevant expert's neural nets, providing the potential for non-experts to solve problems independently outside their knowledge domain. such immediate access to essential information can greatly improve both individual and organizational efficiency. the prototype system which has been built to implement the model described above and depicted schematically in figure 2 runs under windows 3.1 on a 486 pc. the epss shell is written in knowledgepro for windows (version 2.21). knowledgepro also provides the environment for the creation of the expert-system component of our system (barker 1989). the neural networks used within our prototype have been developed using a low-cost system called neuroshell (ward systems 1990). full-text retrieval facilities are available through the status information retrieval package (harwell computer systems 1991). depending on the particular tasks that have to be carried out, the epss shell can invoke the full-text retrieval engine or the neural network facility or use its own inferencing mechanisms in order to take appropriate action. conclusion working together, expert systems and neural networks can offer a powerful set of tools for the implementation of intelligent agents for handling information retrieval problems within an epss. this paper has described some of the ways in which we have been using these tech68 alt-j volume 2 number 1 niques for sharing expertise in a distributed epss facility. the methods we have described have been tested using relatively small textual-knowledge corpora, and they seem to work successfully. the next phase of our work will extend these methods to large collections of multimedia knowledge. note an earlier version of this paper, here significantly updated, was presented at the international computing congress held in.hyderabad, india, in 1993 (barker and banerji 1993). references banerji, a.k., designing electronic performance support systems, draft ph.d. thesis, school of computing and mathematics, university of teesside (in progress, 1994). barker, p.g. (1989), 'knowledgepro: a review and assessment', engineering applications of artificial intelligence, 2, 4, 325-38. barker, p.g. (1991), electronic books, special edition of educational and training technology international, 28, 4, 269-368. barker, p.g. (1992), 'electronic books and libraries of the future', the electronic library, 10, 3, 139-149. barker. p.g. and banerji, a.k. (1993), 'multimedia environments for epss development', in balagurusamy, e. and sushila b. (eds), proceedings of the indian computing congress (icc '92) innovative applications in computing, new delhi, tata mcgraw-hill publishing company, pp. 423-30. beale, r. and jackson, t. (1990), neural computing an introduction, bristol, adam hilger. beishon, j. and peters, g. (1972), systems behaviour, london, open university press. churchman, c.w. and cerhulst, m. (1963), management sciences: models and techniques, new york, pergamon. gery, g.j. (1991), electronic performance support systems how and why to remake the workplace through the strategic application of technology, boston ma, weingarten. hart, a. (1986), knowledge acquisition for expert systems, london, kogan page. harwell computer systems (1991), status user's manual, oxford. heaps, h.s. (1978), information retrieval computational and theoretical aspects, new york, academic press. jacobs, g. (1994), 'isdn: theoretical specification, practical potential' in mason r. and bacsich, p. (eds), isdn applications in education and training, london, institution of electrical engineers. ward systems (1990), neuroshell user's manual, 3rd edition, maryland va. whitlock, q. et al (1993), 'hypermedia and performance support', to appear in proceedings of ed-media '93, orlando fl. 69 multimedia courseware: never mind the quality how much will it cost to develop? i. m. marshall, w. b. samson and p. i. dugard software quality group, department of mathematical & computer sciences, university of abertay, dundee this paper evaluates multimedia courseware costing techniques such as the us airforce interactive courseware method (golas, 1993), cbt analyst (kearsley, 1985), ceac (schooley, 1988) and meem (marshall, samson, dugard, & scott, 1994) against the data from ten multimedia courseware developments. the relative error and mean absolute relative error (mare) are calculated to allow comparison of the different methods. introduction baker (1994) humorously described the failure of teachers, companies, organizations and governments over the last twenty-five years to deliver the volume of courseware which would spark the active-learning revolution. hardware now exists to deliver multimedia, but the cost of developing quality courseware remains high. what chance does multimedia-based active learning have of widespread adoption if developers cannot reliably estimate the development effort of multimedia courseware? this paper presents expert estimation of development effort to learner time ratios found in the literature, before investigating four alternative methods for estimating multimedia development effort. the results of the estimates are then compared against the actual project data. estimation of development effort the range of development efforts reported in the literature to produce one learner-hour of multimedia courseware is presented in table 1. analysis of the projects included in table 1 indicates that the values cover a wide range of different types of courseware from simple drill and practice exercises (jay et al, 1987) through to high-fidelity multimedia simulations (golas, 1993). this range is reflected in the range of estimates senbetta (1991) found when experts were asked to estimate no alt-j volume 3 number i table i: estimates of courseware development effort data collection method anecdotal evidence author experience expert estimation expert estimation development effort required to deliver one hour of learner time lowest 50 85 30 1:1 highest 350 300 1380 4000 author (jay, bernstein, & gunderson, 1987) (gery, 1987) (golas, 1993) (jay, bernstein, & gunderson, 1987) different courseware-development effort from detailed specifications. the development effort estimates varied by up to 500% from the minimum to the maximum estimate for the same specification. us airforce interactive courseware method (usaf i c w ) golas (1993) developed an interactive courseware estimation method for the us air force based on expert opinion of the factors which affect development effort. the starting point is a best-case estimate which is made for the level of course and type of behaviour to be delivered using the criteria listed in table 2. these values are best-case estimates which are then increased by the appropriate number of developer-hours for each factor described in table 3. table 2: best cose estimate for interactive courseware type of training level of presentation knowledge skill attitude i basic 30 75 200 ii medium 75 125 250 iii high 200 400 600 the model has been reviewed and revised using expert opinion but unfortunately no information exists about external validation using real courseware data. table 3: factors affecting best case estimates factor 1 n o 'in-house' subject matter experts; must rely solely on the use of customer subject matter expertise 2 subject matter is highly complex 3 instructional content is unstable. systems for which interactive courseware is being developed are emerging. tasks for interactive courseware are constantly changing increase effort for each learner hour 35 100 • 100 i i i /. m. marshall et al. multimedia courseware factor increase effort for each learner hour 4 inadequate documentation. no training needs assessment was performed. no task analysisor learning analysis data. technical manuals are non-existent or are not helpful 20 5 total interactive courseware length less than 100 learner hours 20 6 interactive courseware developer is not familiar with interactive courseware software/authoring systems 15 7 interactive courseware developer is not familiar with target audience 10 8 best commercial practices are not acceptable for video, graphics production and software development 50 9 inexperienced project team: interactive courseware designer inexperienced 80 interactive courseware manager inexperienced 100 interactive courseware programmer inexperienced 60 10 using a beta version of interactive courseware software 80 11 no prototype exists, no agreement 'up front' on design strategy, no standardized development process followed 50 12 customer is not using objective and consistent acceptance criteria. customer unsure of what is wanted and does not communicate with developer 50 13 required resources are not in place at start of project 20 cbt analyst kearsley's (1985) cbt analyst estimates development effort by asking questions about the courseware to be developed. based on the answers to twenty-two questions, the software produces approximate development effort per learner-hour. table 4 shows the twenty-two questions asked to estimate the courseware development effort. this result is then modified by the three composite rules described in table 5. table 4: cbt analyst's base constraints questions question 1 2 3 4 5 6 7 8 what type of cbt do you plan to develop? (tutorial, simulation, testing or embedded) how complex is the learning task the cbt course is to be developed for? will colour graphics be used? will interactive video or audio be used? how will the courseware be developed? does a library of cbt routines and graphics exist or does all programming have to be done from scratch? how much cbt experience does the designer or design team have? how much experience does the developer/programmer have with the authoring language or system being used? lowest value 0 0 0 0 0 -5 +1 -5 highest value +5 +2 +5 +5 +3 0 +5 0 112 alt-j volume 3 number i question 9 is this a new or existing course? 10. is the subject matter for the course available or is it in the process of being developed? 11. is the cbt course being developed for internal use or will it be sold commercially? 12. what kind of branching will the course involve? 13. will the answer analysis be simple or complex? 14. what kind of response will the course involve? 15. how much learner control will the program have? 16. what percentage of the course do you anticipate having to revise each year? 17. does a well defined storyboard exist for the cbt course to be developed? 18. if the cbt is to be developed by a team, does this team have previous experience developing cbt courses together? 19. do written standards, guidelines, or procedures exist for cbt development and are they followed? 20. is the development effort being managed by an individual with past experience of managing cbt projects? 21. is there a single individual responsible for approving the course and revisions to be made? 22. how would you describe the motivational level of the designer/developer? lowest value 0 0 0 0 0 0 0 0 -5 0 0 0 0 -10 highest value +5 +5 +5 +5 +5 +5 +5 +3 0 +5 +5 +5 +5 0 table 5: cßt analyst's composite rules composite rule 'unknown' rating in questions new score rule 32 inadequate cbt specification 1,13 and 15 rule 33 human factors unknown 20,21 and 22 rule 34 experience unknown 7 and 8 +10 +10 +5 cbt analyst then uses this result to select an estimated development effort using the values in table 6. table 6: cbt analyst's threshold values and development effort threshold values -9999 to 0 1 to 20 21 to 50 51 to 9999 development effort per hour of learner time under 100 100-200 200-400 500+ 113 /. m. marshall et al. multimedia courseware the upper limit of 500+ developer hours limits the usefulness of the estimate produced, but it does provide a consistent method which is simple to use. cost estimating algorithm for courseware (ceac) ceac (schooley, 1988) estimates both courseware development effort and cost. estimates are based on project and organization specific inputs as well as an internal database of courseware development data. the software uses the following equation to calculate the development effort by summing the contribution of tutorial, drill and practice, simulation and certification test elements to the project. certification test development time = x (cfx caxlt)x(tmxefxsfxdvx(l -lsf)) tutorial where: cf = ca = lt = tm = courseware fraction cbt advantage lecture equivalent time teaming multiplier ef = sf = dv lsf = experience factor sophistication factor database values library saving fraction schooley found that estimates were within 20% of the actual figure on six of the twelve projects evaluated. ceac's main strength is the range of factors which contribute to the estimate of development effort. however, the internal database is constructed from linear projections of a limited number of data points. multimedia effort estimation model (meem) the authors of this paper are currently involved in a project to develop a multimedia costestimation model. using courseware-development data from fourteen projects has allowed 85% of the variation in development effort to be explained by the use of four grouped cost drivers (marshall et al, 1994). the model involves rating individual cost drivers under the following groups: • course difficulty (cd) • interactivity (in) • development environment (de) • subject expertise (se) the individual cost driver is based on expert opinion of key factors which contribute to development effort. at present there are too few projects to generalize these results, but it does indicate that statistical analysis of development data can form the basis of a costestimation model. 114 volume 3 number i comparison of estimates the data from ten of the meem projects were used with the three courseware-estimation methods previously described. table 7 presents the estimates produced by each model, along with the actual development effort and the results from meem. the ten projects selected each had an estimated learner-time of one hour. table 7: comparison of estimates and actual development effort project a b c d e f g h i j actual development effort 80 100 100 180 200 220 250 320 400 500 usaficw 165 420 50 85 420 165 120 120 420 590 ceac 349 319 174 137 879 172 191 174 319 3199 cbt analyst 100-200 200-400 100-200 100-200 2oíw0o 100-200 200-400 200-400 20ím00 500+ meem 128 186 97 125 247 128 275 306 367 487 schooley (1988) used a measure called relative error to determine the accuracy of the ceac estimation method. the following equation shows the relative error for development effort. relativee = actual effort estimated effort actual effort the relative errors for the four estimation methods for the ten projects are shown in table 8. cbt analyst's results are divided into low and high values to indicate the range of values produced by this tool. because the relative error can be greater or less than zero, mean relative error (mre) would no be a useful summary. taking the absolute value provides a more useful summary measure. table 8 shows the mean relative error (mre) in addition to the relative error for each projects. meem produces a mare of 37%, but this is not surprising because the same data was used in the statistical analysis. cbt analyst (low) produces a mare of 43% with the ten projects used. despite its relative age, it produces on average more accurate results than the other newer estimation models with this data set. 115 /. m. marshall et al. multimedia courseware table 8: relative error of estimated development effort project a b c d e f g h i j mare usaficw % 106 5 -50 -53 320 -25 -52 -62 110 18 79 ceac % 336 -20 74 -17 219 -21 -23 -45 340 540 164 cbt analyst (low) 125 -50 0 ^44 100 -55 -20 -37 0 0 43 cbt analyst (high) 150 0 100 11 300 -9 60 25 100 ? 83 meem v . -60 8 3 30 187 42 -10 4 -24 3 37 conclusion the four models used to estimate the development cost of multimedia courseware produced mare results which range from 37 to 164%. these results support kitchenham's (1992) concerns about general software cost-estimation models, and indicate the need for specialist models related to the development environment. further research is underway to collect data to assist in the development of multimedia costestimation models, and the rigorous validation of existing models. it is only with the development of a reliable method of estimating development effort that multimedia courseware can hope to sustain the active learning revolution. references baker, j. (1994), 'one man and his dog', interact, 1, 3, 16-17. gery, g. (1987). making cbt happen: prescriptions for successful implementation of computer-based training in your organisation, boston, ma, weingarten. golas, k. c. (1993), 'estimating time to develop interactive courseware in the 1990s', paper presented at the interservices industry training and education conference, orlando fl. jay, j., bernstein, k., & gunderson, s. (1987), estimating computer-based training development times (ari technical report no. 765), research institute for behavioural and social sciences. kearsley, g. (1985), 'the cbt advisor: an expert system program for making decisions about cbt', performance and instruction, 24, 9, 15-17. 116 aa-j volume 3 number i kitchenham, b. a. (1992), 'empirical studies of assumptions that underlie software costestimation models', information and software technology, 34, 4, 211-18. marshall, i. m., samson, w. b., dugard, p. i., & scott, w. a. (1994), 'predicting the development effort of multimedia courseware' information and software technology, 36, 5, 251-8. schooley, r. e. (1988), 'computer-based training (cbt) cost estimating algorithm for courseware (ceac)', proceedings of the interservices industry training systems conference, 319-28. senbetta, g. (1991) an inquiry of time and cost estimating for computer-based training courseware design and development as determined by modified delphi method, phd thesis, purdue university. 117 editorial distinguishing features by chance or probably not so, as i shall explain in a moment the papers which appear in this issue of alt-j all deal, directly or indirectly, with investigations into how students interact with computers, and how they might learn most effectively. scott grabinger's article the longest we have ever published, going way over the normal limit of 5,000 words is a comprehensive overview of reals (rich environments for active learning), a discussion of many aspects of educational theory, and a wide-ranging review of the relevant literature. the referees for this article made almost identical comments: it fits within the criteria set out in the guidelines for authors, it is worth publishing for both its new perspectives and its review of current thinking, and it cannot be cut down to the normal word limit without losing most of the value of what it says. grabinger's bibliography alone will, i hope, prove a valuable research resource in its own right. the piece by c. k. ramaiah and colleagues presents the results of some interesting experiments concerned with student skills in handling new technology; they thus tangentially encompass learning theory. oleg pilipenko and helena komissarova do so more directly in dealing with the ways in which learners interact with computers. gervase phillips, who proposes that the distinction between the activities of history students and programmers is artificial, touches on the same theme. anne groat and tim musson deal with the fascinating question of learning styles. and the report by naidu and colleagues, on ways of improving instructional effectiveness with computer-mediated communication, naturally involves learning theory. even in the book reviews in this issue, there is a bias towards educational theories and how we can best put them into practice. these facts are the result of two interlinked things. the first is that while theorizing about education is hardly new, we are seeing more and more of it in the area of educational technology so much so that at alt-c '95, i heard more than once the view expressed that we ought to be getting on with the practice of education and leaving the theory to those who live in the clouds. what a relief, i was told by one delegate, to come to a conference where the vast majority of presentations were concerned with people's actual 2 alt-j volume 3 number 2 experiences as practitioners, as opposed to airy-fairy, holier-than-thou pronouncements, in jargon-ridden and often impenetrable language, about this or that obscure theory. i quote: 'let them actually produce something and try it out in the real world of the tutorial room or computer lab; only then will they be worth listening to and we might then also understand what they're talking about.' and aimed straight at me by another delegate: 'alt-j should be publishing more about what alt members have actually done, and are actually doing'. well, the second thing is that referees for alt-j, following the guidelines sent to them, tend to reject papers which merely describe what people have done, are doing or are intending to do, unless such papers say something really new and, paradoxically, genuine innovation is actually quite rare in our highly innovative field. most purely descriptive papers report the implementation of a piece of software, and perhaps some student reactions to it, with no worthwhile analysis or independent judgement of it. students may say they like a piece of cal software, but that is not the same as saying that they have learned much from it, or have learned in some better way from it, or have learned anything from it that they could not have learned by some other less expensive means (you can buy a lot of books with the amount spent on developing some cdroms). assessment of the value of cal must be based on comparisons with alternatives. in three years as editor of this journal, i can count on the fingers of one hand the number of papers i have received which make even an indirect attempt at such comparisons using proper research methods (as opposed to 'feelings', or vague and sometimes dubious anecdotal evidence). of course, for certain aspects of cal, it is hardly necessary to prove the case. distance learning is one example. here, the balances between the costs and the benefits of sending digital data down telephone lines instead of having face-to-face contact are often fairly clear, and in any case (as with the open university) usually carefully assessed, before, during and after implementation. furthermore, the justification for using communications technology, whatever the cost, is often selfevident; indeed (as at the open university), such technology is sometimes indispensable. but there are other areas where one wonders if the team of people who have written a piece of cal software have not started from the assumption that cal is a good thing (or, worse, have received funding and to hell with whether it is or is not), never questioning its value when set against its costs except for the occasional nod in the direction of real-worldliness costs not just of development, but also costs to an institution persuaded to spend large amounts of money on equipment to deliver the software, and an army of technicians to keep it all up and running. let me emphasize that i am not arguing the case against cal. i am hardly likely to do that, given my belief in its huge potential in so many aspects of university activity. i am arguing for a level of selfassessment on the part of those involved in it. a paper which does no more than describe what a development team has done, with no real analysis, nor showing any evidence of rigorous methodology in its judgements (if they are there at all) does not advance our cause. if we do not look critically at what we do, if we do not present incontrovertible evidence of the benefits of cal set against its cost (and, if necessary, be honest about its failures), others who know little about the subject will eventually take a cool look at what we are doing and conclude that we do not know either. thus, we have an issue of alt-j which is biased towards learning theory, and which, i 3 gabriel jacobs editorial think, reflects the view that the use of educational technology is an intellectual challenge as well as a practical means of teaching and learning. rightly so. we are academics. we should doubt all claims of success, all evidence which has not been subject to critical scrutiny. nearly all the papers which the referees rejected for this issue failed to take a self-critical look at what they described (described rather than analysed). where were the control groups of students who had learned the same material delivered by some means other than a computer (difficult, i know, but not impossible)? where was the testing of a theory? where were the cost-benefit analyses? where was the research methodology? where were the papers which discussed why students disliked an implementation? do all cal implementations result in favourable student reaction? so it is not by chance after all that this issue contains so much theory, even if much of it is inextricably linked with practice. theory inevitably involves intellectual activity, and automatically attracts analysis, critical assessment, and detached evaluation of evidence. the referees have reacted accordingly. gabriel jacobs 4 effecting institutional change: the impact of some strategic issues on the integratiye use of it in teaching and learning audrey mccartan* and catherine hare** *university of durham and **university of northumbria at newcastle this paper addresses the effective implementation of change through the identification of issues associated with three key institutional areas which, together, will ensure the effective'integration of multimedia technologies into teaching and learning. these are the need for a firm commitment of support at the institutional level, the development of an institutional strategy, with related staffdevelopment policies and programmes, and provision for students and academic staff to. acquire enabling it skills through resource-based learning as a means to maximize the benefits to be gained from the use of these new technologies. the essential conditions which ensure effective implementation communication between all stake-holders, leadership by senior managers, responsive central services and a mixture of funding arrangements -r are discussed the changing higher-education environment an increasing number of multimedia-type products are being developed for the highereducation market. this development is gathering pace as the teaching and learning technology programme (tltp) projects deliver their courseware. tltp arid the fund for the development of teaching and learning (hefce circular 29/95) are initiatives through which the integration of multimedia, technologiesinto hjgherreducatiofi institutions has been actively promoted through the allocation of nationally administered funding. in a rapidly changing higher-education environment, there is an emphasis on widening access for both school-leavers and. mature students, modularization of course structures and diversification of course content, increasing costs and the diminishing unit of resource. in a speech to the cvcp, the minister for higher education (11 -december 1995) stated that 'new methods, of teaching, especially the growing potential of information technology,, offer scope for further improvements in productivity'. the funding councils have indicated a number of areas where they believe the (use of technology can have substantial benefits. the. intention is to help institutions respond . . ^, . _ audrey mccartan and catherine hare effecting institutional change: the impact of some strategic issues on the use of it effectively to this changing environment while continuing to promote and maintain quality provision within a context of public accountability. the considerable increases in computer power and functionality which have been developing during the same time, the introduction of distributed systems, networks, and the increased availability of computers, have contributed to a growing awareness and use of the new technologies in the management, delivery and assessment of teaching and learning. it is apparent that, at the present time, many academic staff are not poised to make effective use of multimedia products in academic courses, nor are students equipped to exploit computer-mediated learning materials. experience at both national and local levels suggests that the availability of multimedia learning-support materials alone is not sufficient to ensure their widespread use (mccartan et al, 1994). the popularity of the recent programme of regional workshops, which were conducted within the delivery of the tltp project it in teaching and learning: a staff development pack, demonstrated the need for staff development programmes in the uses of it in all aspects of curriculum design and assessment. the importance of staff development was underlined again recently in the higginson report (fefc, 1996) which recommended a sector-wide initiative to enhance the professional skills of staff in the further-education sector, and to alert them to the opportunities presented for more effective teaching and learning using modern technologies. some examples of the development of institutional teaching and learning strategies incorporating the integrative use of it and the implementation of specific programmes designed to improve the it skills-base and to promote the integrative use of it in teaching and learning were demonstrated and discussed at a forum in 1994 (mccartan and hare, 1994). the most successful developments which actively support learning included a partnership between academic staff, on the one hand, and central services on the other, both focusing on the use of it as a mass teaching, learning and information storage and retrieval medium. in this paper the significance of the impact of the institutional context on the existing gap between the aims and objectives of current developmental projects, and the ability and readiness of staff and students to capitalize on their benefits, is examined through the identification and analysis of some critical institutional and strategic factors. these provide the framework which determines the development and implementation of institutional models of it training and development. development of the institutional models the need for institutional support to encourage the integration of new technology into the academic curriculum was apparent from the report of the information technology training initiative (itti) funded skills levels project (hodgson et al, 1994). sixteen universities had participated actively in the project for which representative members of each institution's senior officers, academic and other staff had formed a working party to undertake an internal consultation exercise. the membership and remit of the group was determined by the priorities of each institution within the broad aims of the project. the outcomes could be used, therefore, as much to further institutional development as to contribute to the identification of trends across the consortium. there were strong indications from the participating universities, the associated programme of regional an-j volume 4 number 3 workshops and from the sales of the report, that the project had proved to be both relevant in its focus on individual institutional situations and timely in its consideration of the development needs of all staff in respect of the integrative use of it in their job functions. the tltp project core it skills for teaching and learning in higher education: tools for the development of a national framework (hodgson et at, 1995) has applied similar methodological procedures to assess the impact of key institutional factors on the formulation and implementation of teaching and learning policy. about 30 institutions have been included in the nation-wide consultation process through the collection of extensive experiential data. detailed institutional information has been gathered from nine universities, selected to be representative of the whole sector, with a number of 'old' and 'new' universities, and including a number which had expressed a wish to join the consortium as a means of benefiting their own development. a working group was formed in each institution consisting of a wide range of staff representing different functions and including senior officers. given institutional diversity, it was not considered appropriate to ask institutions to return the information in strict questionnaire format. documentation was provided which indicated the areas in which responses were considered to be most helpful (see figure 1), but institutions were free to add headings they considered relevant to their own exercise. contact was maintained throughout with the project team, and progress monitored. some additional visits, reviews and some telephone interviews were all arranged as a result of particular circumstances. the procedure proved to be both responsive and flexible to changes which occurred in the institutions during the course of the consultation exercise. this has meant that the information gathering has taken place over an extended period, and in some cases the original completion has had to be updated in the light of subsequent events. the involvement of the project over a period of two years with each participating institution has demonstrated the significant impact of change as a function of various external and internal pressures. no institution was left untouched. participation in the project's activities has prompted institutional responses to situations which have been addressed through bringing together a range of professional expertise, which would otherwise be channelled through a number of different and, perhaps, mutually exclusive reporting routes within the university. a result in one institution is a new student it-skills initiative which aims to increase the provision of it-skills training for students in the university by encouraging departments, or groups of departments, to introduce subjectoriented it-skills modules. another university used the collection of the institutional information as an opportunity to enhance communications between staff with disparate responsibilities, and to urge forward an institutional approach to the development of teaching and learning and it strategies. the project team provided feedback through customized workshops delivered locally to the university on its responses in relation to its declared priorities. the result of the compilation of all the institutional information is a chronicle of the complexities and interconnectedness of organizational change, and the evolution of institutional structures. the information has been developed into a series of institutional models which describe the patterns of structures in place in the mix of higher-education institutions covered by the project. these illustrate the dynamic relationship between institutional culture and strategies for change and development, and the key areas of implementation for the integrative use of it in teaching and learning. 23 audrey mocartan and catherine hare effecting institutional change: the impact of some strategic issues on the use o f it allocation of resources in the institution funding of academic departments/sections funding of it, library and other services it services sections involved functions it strategies management monitoring of policies teaching and learning policies management monitoring of policies relationship between it and teaching and learning links externally funded projects it and developments in teaching and learning staff development function institutional policy management monitoring of policies funding it training academic staff other staff students future developments figure i: institutional information institutional models to support the integrative use of it in teaching and learning the models do not relate to any single institution, but rather bring together the characteristics which have been identified in a number of universities of similar types. they focus on the key factors relating to it in teaching and learning, presenting the interrelationship between the centre, the executive framework and the support services of the institution in an attempt to assess the most effective arrangement for integrating it in teaching and learning. there are basically three models (illustrated in figure 2). i . traditional 'passive' model : as the name suggests, this model is characteristic of the traditional universities typified by strong central services, e.g. it services and staff development, funded by top-slicing and operating independently. in addition to the top-slicing, other funding allocations are normally on a formula basis. . alongside the central services, and running in parallel, is the committee structure. the committees of which the most important will be chaired by pro vice chancellors, deans of faculty or senior professional staff, are a well-established mechanism for policy and strategy development. however, in practice there is evidence that they are now more a vehicle for implementation, with policy and strategy being developed by small groups of senior staff. their composition will.vary either by a process of election or rotation resulting in a lack of continuity which can hamper, sustained development in an area such as the integration of it into teaching and learning which is often characterized by extreme polarization of views as to its merits and benefits. both the central services and committees dispense their 'service' or authority to the users 24 volume 4 number 3 i. traditional "passive" 2. "active" top-slicing to support central_ services central depts/ individualsj teaching & learning committee n" service 7t\ committees departments v ' t 3. modern "proactive" strong links between it and teaching and learning senior manager responsible for it and/or teaching and learning staff or student user l i n e of communication figure 2: institutional models to support the integrative use of it in teaching and learning acting as one-way, top-down conduits for institutional policies and strategies which tend to be rather weak. there is no single individual at a senior position in the centre whose prime and constant responsibility is it and/or teaching and learning. this model in the past provided reasonable technical infrastructure (that is, from computer board monies) together with development funding. however, nowadays the harsh reality is a major reduction in funding combined with the introduction of devolved budgets and associated devolved responsibilities. this has resulted in central services refusing to take on additional services unless they get paid to do so. the departments individually make choices about how they support it in teaching and learning in terms of equipment, staff development and student it-skills programmes. they often duplicate activity with initiatives repeating work already undertaken and evaluated in other departments. this is particularly true in the case of documentation which could be effectively and efficiently co-ordinated at the centre. there are, however, pockets of activity led by enthusiasts often with funding from institutional initiatives. but their best efforts do not get a chance to propagate because of the rigidity of the system, the absence of effective communication channels, and the lack of a champion at a senior level. despite some users being well served for specific applications, there is no recognition of the developing importance of it in teaching and learning because there is no single focus at the institutional level, and no forum for discussion and exchange of ideas. the communication is all from the top down, with only a relatively small number of people 25 audrey mccartan and catherine hare effecting institutional change: the impact of some strategic issues on the use of it involved. the rigidity of the structure, which in the past was a strength when change was a rare phenomenon, now has become a weakness in providing an appropriate environment for the integration of it in teaching and learning which, because of its pervasive nature, needs to cut across what remain parallel and independent activities. 2. 'active' model this model, with its three sub-models which have either committees, departments or a group of senior staff at the second level of management, can be found in both traditional and new universities. central services remain relatively strong but participate actively in serving the needs of their users. in each of the sub-models there is effective and efficient two-way communication with the users, and a direct feedback loop to a senior individual whose prime responsibility is it and/or teaching and learning. in this model, devolved funding is the norm, and results in devolved structures and responsibilities which work together effectively to maximize the benefits and efficiency gains from the funding arrangement. it is likely that departments will buy into central services which might include access to the network, provision of training and/or training materials for staff and/or students. the committee structure may in some cases retain an important role, but the structure will allow for the bringing together of the teaching, information and technology issues because of more opportunities for communication and exchange of ideas between and across levels of management. the whole culture and communication infrastructure actively involves more people across the different levels, thus providing a robust framework with in-built flexibility. the greatest strength of this model is the senior individual who embraces technical expertise with an understanding of its implications for the management of all aspects of the university's business, together with the vision to fulfil the developmental needs of all staff in respect of the integrative use of it in their job functions. this feature can, however, also be the greatest weakness of this model, particularly in the new universities where senior staff are in promoted and therefore semi-permanent posts and may not have the mix of knowledge, skills and vision required. in the traditional universities there is also a potential weakness but for a very different reason: the senior individual will hold the post only for a fixed term of office, which is good if the person lacks vision but poor if the person has vision and moves on. there is another potential weakness in this model where resources at the centre are not adequate to maintain and consolidate the central services. this situation is especially prevalent in the new universities which are still striving to catch up in terms of it infrastructure without any additional funding to compensate them for the computer board monies from which the traditional universities benefited in the past. 3. modern 'proactive' model this model to date seems to appear only in the new universities. there is an additional funding element to top-slicing and formulae: bid funding. central services are no longer large and tend to be integrated. they also operate on a service model supported by service-level agreements. devolved budgeting and responsibilities are very much the norm, with some buying in of external services, including training and documentation. the committee structure is very weak, with management being implemented via a hierarchy of management groups composed in the majority of academics who have been promoted to management posts. 26 alt-] volume 4 number 3 the institutional policies play a key role of informing, directing and providing a monitoring device and/or performance measures for all developments. they hold together what is a very distributed and therefore potentially fragile structure. the departments/ schools become key nodes implementing institutional policies, often by the appointment of academic staff to some responsibilities for it in teaching and learning, including staff development, and also for taking feedback from the consumers/users. the system is thus responsive to user-need. there will be strong departmental projects, including participation in itti and tltp projects. the central services act as the hub of the network, but communication can break down as neither the hub nor the nodes want to invest their resources, which are often limited, in maintaining the links, and the nodes in some cases may even choose to buy in services from outside rather than pay into central provision. a recognition of the importance of the strong linking of it into teaching and learning is a significant feature of this model. this is predicated on strong networking between all staffs. the impetus for change, however, also needs a strong and enthusiastic individual at the top who ideally has responsibility for both it and teaching and learning. a key focus is the provision of a range of proactive, integrated services, providing expertise and support to underpin the implementation. the proactive model thus involves the greatest number of people with communication top down, bottom up and across the levels. it is the model which applies to a greater number of institutions than the other two models. the approach adopted by" each university depends very much, however, on the institutional context, and progress will be determined by the strength of the imperative to integrate it into teaching and learning. the progression from model to model does not need to be incremental. it is possible to leapfrog a stage. the impetus/movement for change comes when it is linked into teaching and learning, and this is reflected in the institutional policies and strategies. effective implementation of change institutional change is a continual evolutionary process, especially in the use of it where institutions are urged to exploit its use in the management, delivery and assessment of teaching and learning. the different models have shown varied responses to the integrative use of it in teaching and learning. the most recent manifestation of this imperative can been seen in the teaching quality assessment (tqa) exercise which both explicitly and implicitly makes requirements on teaching departments in terms of it management and integration issues. this imperative to integrate will be interpreted in different ways depending on the particular institutional context. the evidence, as highlighted in the detailed expositions of the passive, active and proactive models from the consortium institutions, which represent varied traditions, missions and executive frameworks, shows the need to fulfil the following essential conditions to ensure effective implementation: • communication between all stake-holders (all personnel and students); • a senior manager in charge of it and/or teaching and learning; • responsive central services; • a mixture of funding arrangements. 27 audrey mccortan and catherine hare effecting institutional change: the innpact of some strategic issues on the use of it references fefc (further education funding council) (1996), report of the learning and technology committee (higginson report). hodgson, m., mccartan, a. and hare, c. (1994), a framework for the development of core it skills for university staffs, cvcp/usdu publications. hodgson, m., mccartan, a., clayphan, r., hare, c , and jezierski, g. (1995), teaching and learning in higher education: the integrative use of it within the institutional context, university of durham. mccartan, a., watson, b., lewins, j. and hodgson, m. (1994), 'enabling learning through technology: some institutional perspectives', alt-j, 3 (1), 92-7. mccartan, a. and hare, c. (eds.) (1994), enabling technologies for teaching and learning: national perspectives and futures, oxford: ctiss publications. 28 enabling learning through technology: some institutional imperatives audrey mccartan, barbara watson, janet lewins and margaret hodgson education and training area, it service, university of durham this paper considers the importance of the institution as the dynamic interpretative element on which will depend the successful integration of the learning technology developed through our national initiatives into the academic curricula of higher-education institutions. based on our experience of working on teaching technology programmes, within the framework of national and institutional initiatives, it is evident that the establishment of an institutional strategy, and its implementation in a supporting university-wide programme of staff development and training, together with strong commitment at the senior managerial level, are imperatives which determine the successful integration of learning technology within academic institutions. introduction the imminent completion of many teaching and learning technology programme (tltp) projects means that a considerable number of courseware deliverables will soon be available to higher-education (he) institutions. the higher education funding council's intention in funding the programme (hefce circulars, 8/92, 13/93) was to ensure their integration into academic curricula by providing institutions with an opportunity to review their 'teaching and learning culture' with regard to the embedding of learning technology within their institutional practice. two recent workshops, conducted with a representative sample of newly appointed academic staff in connection with the evaluation of materials to be included in a staff development pack whose purpose is to encourage the use of it in teaching and learning (tltp project 7), strongly suggested that the availability of courseware alone was insufficient to ensure its integration into educational practice. the establishment of enabling mechanisms at the institutional level, as well as within departments, was crucial to ensure the effective use of learning technology. 92 alt-] volume 3 number i institutional commitment the priorities of the national teaching technology initiatives which have been funded during the last decade (cti, itti, tltp) have reflected the increasing capacity and functionality of hardware and software as well as the growing emphasis on curriculum development. there has been a development from computer-led initiatives to pedagogically driven priorities. it is within the institutional context that these nationally identified objectives, institutional policy and implementation meet, where policy is developed and programmes delivered. the institution is the dynamic interpretative element working between the national and local levels, yet to date the institutional perspective has been the least regarded. in a recent itti project, which investigated the core it skills which were required by university staffs in order to execute their job functions, the institutional context, particularly the strength of institutional support, was found to be the most significant factor influencing implementation and practice. the report (hodgson et al, 1994) was the result of a nationwide consultation exercise in which about a quarter of uk universities participated. initial advisory discussions were conducted with representatives of the variety of interests which contribute to the formulation of it and teaching and learning policy it managers, staff development advisers, teaching and learning co-ordinators, and senior managers. sixteen universities, drawn equally from the former ufc and pcfc sectors, went on to form internal working parties to identify which core it skills were required within their institutional context amongst academic, administrative and clerical staffs, thus enabling them effectively and efficiently to operate their job functions consistently with institutional priorities. each institution collected relevant background information about itself, then defined and conducted the business of its working party in order for the results to be most informative for its own forward planning as well as reporting back to the project. the institutional reports indicated that there was an already pervasive use of information technology across institutions and that its application in all areas of a university's business was rapidly increasing. generally, staff had a positive attitude to the use of information technology, and expected to be offered the opportunity to develop their skills further. current training provision was, however, often patchy and uncoordinated and was likely to be supplyrather than needs-driven. it was not always related to institutional needs. effective information technology training provision was most likely to arise from firm commitment at the institutional level. the foundation of this commitment was the development of an institutional information strategy with associated faculty or departmental strategies. these strategies would incorporate responsibilities for the installation of up-to-date technology, made widely available across distributed networked systems, together with the provision of appropriate training opportunities. there needed to be a clear relationship between institutional strategy, staff development and training programmes. it was essential for this relationship to be reflected in effective resource allocation mechanisms, especially in a devolved budgetary organization. for the delivery of training programmes, a structured approach was preferred. this might — audrey mccartan et al. enabling learning through technology be based on the identification of core skills, related to particular job functions. a 'bricks' approach, where the skills were defined in terms of levels, would enable training provision to link the acquisition of particular skills and levels to specified institutional functions. the strong staff development interest would ensure coherent and co-ordinated provision for individual members of staff while supporting the effective implementation of institutional planning. in order to ensure responsiveness to user needs, the programme would encompass a variety of learning opportunities. although traditionally these have mainly been provided through taught courses with supporting documentation, new learning methods and mixed media were now available to enable increased flexibility and customization in delivery. the recognition of informal methods of learning, such as training selected staff to undertake limited training responsibilities within their section, would maximize the use of relevant expertise and experience. an identified group of staff for whom particular provision was urgently required was senior managers. they were responsible for the formulation of policy and its implementation. they needed to be aware of, if not actually competent in the use of, the wide ranging power of modern computer networks and its impact on institutional practices, as well as the implications of the pervasiveness of technology for future planning, including job functions (tltp project 45). teaching technology initiatives the institutional imperative emphatically to support staff development within the implementation of a university-wide policy to encourage, support and develop the use of information technology in teaching and learning has been implemented at the university of durham through two teaching technology initiatives posts. these staff work in partnership with academics to review their course structure, content, delivery, assessment and evaluation methods in respect of the appropriate and effective use of it. they coordinate the dissemination of knowledge and increase awareness about developments in computer-aided learning. the awareness program includes university-wide and departmental presentations and hands-on sessions, staff development sessions, contributions to text-based newsletters, and running mail groups. development work in departments includes advice, information and the development of substantial courseware projects. to date, courseware has been developed successfully for projects in archaeology, chemistry, education, geography and spanish. selecting software for successful courseware the success of courseware in achieving educational objectives is in part dependent on selecting the most appropriate development tool. the learning objectives must be analysed* then clearly defined in order to evaluate which software would be most suitable to achieve these educational goals. consequently, at the university of durham, courseware has been developed in: 94 alt-j volume 3 number i • the core spreadsheet application quattro pro for windows • the assessment package question mark professional • the authoring package toolbook learning errors in measurement in chemistry was developed using the universityrecommended spreadsheet, quattro pro for windows, to utilize its spreadsheet functions. the first-year chemistry students start by acquiring a foundation knowledge of quattro pro, using the self-study guides of the computer literacy programme. the students need to understand how to navigate through quattro pro in order to be able to use the full potential of the courseware. they then use the cal material in supervised practical sessions as an integrated part of their course. students read the explanatory text, and are then directed to pages of exercises. they learn interactively by calculating their own values for the exercises and entering them into the spreadsheet. the built-in spreadsheet functions are used, through macros, to make the graphs change according to their values and to check their answers. the students click the am i right? button to find out if their answers are correct or need modifying. the courseware is based on random numbers, so the students all have different values for their exercises, and are able to start again with new values if they wish. there is a summary page which holds all the values entered by students and the students are requested to give a hard copy of this page to the lecturer. question mark professional is an easy-to-use computerized assessment and examination program. its strength lies in its analysis and reporting facilities. as part of the preparation for a departmental test, a general knowledge test has been written to enable the figure f: errors ¡n measurement developed in quattro pro. th« general equation for this c u m it shown next to the histogram. your task is to adjust the values of the m e i n and standard deviation parameters (in the blue boxe») so that the curve ftt'the measured data as well as possible. notice what happens to the curve when you change parameters. changing the value of the mean shifts the distribution along the x-aiis. increasing the value of the standard deviation broadens the distribution. change them slowly i t first, perhaps by 0.s to see what happens. w h e n you think you have a good fit, cuck on the "am i right?' button. scroll down to the next chart and try again with th« next sel of data. 100 data values enter your values into the blue eels watch the curve changa ! £\ a s i right? fitting the normal dfetrfbutjon 100 d a t a points 8jw) 10dc0 12x00 11000 1gj0oo x your value of th» sd bnttoo good. try again. m i l>l\tent\dirtrbrfffiu'me«rlsd.se/pio8_eiron/ lal[«wt 95 audrey mccartan et al. enabling learning through technology figure 2: general knowledge test question mark professional students to attempt the different question types available in question mark. at the end of a departmental test, the students were asked if they found the software easy to use, and 97% found it 'easy' or 'very easy'. this positive result is partly due to the students being able to become familiar with the package through the general knowledge test. until recently, spanish syntax had been taught using only lecture notes. students had found this topic difficult to grasp, and the aim of the spanish syntax project is to use the hypermedia facilities of toolbook to enhance the learning of the subject material through interactive examples and tests. colour is used consistently throughout the courseware to identify the different syntactical units of sentence structure. there are different types of exercises, including tree diagrams, following each explanation of the syntax. in many of the exercises, the students click the am i right? button to find out if they have completed the exercise correctly. if they have not answered correctly, they are given hints, e.g. on vocabulary or abbreviations. after two attempts, the answer is given to them. this is to allow the students to be able to learn through their mistakes and progress to the next question. the students use this cal material as an integrated part of their course. conclusion firm institutional commitment is imperative to enable learning through technology. a clear relationship between an information technology strategy, staff development and training provision, supported by appropriate resource allocation mechanisms, is essential to ensure coherent and co-ordinated provision for all staff. 96 alt-] volume 3 number i sentence structureexercises art n pr adj n las páginas de este hbro cop y y-jartsub comf están manchadas vou musí use the tab kevhetu'ee« amimiatiohboxes figure 3: spanish syntax toolbook selecting the most appropriate software for the development of courseware has resulted in successful and enjoyable learning. all the courseware and computer-based assessment is delivered on the campus-wide novell network, so the students are able to progress at their own pace and repeat sections and exercises. references computers in teaching initiative (1984), university grants committee. core it skills for teaching and learning in higher education: tools for the development of a national framework, tltp project no. 45. hodgson m., mccartan a. & hare, c. (1994), a framework for the development of core it skills for university staffs, cvcp/usdu publications. information technology training initiative (1991), universities' funding council. it in teaching and learning: a staff development pack, tltp project no. 7. teaching and learning technology programme phase 1 (1992), hefc(e), circular 8/92, march. teaching and learning technology programme phase 2 (1993), hefc(e), circular 13/93, april. 97 editorial caution: changing education, changing technology the theme of our^very successful alt-c 95 was changing education, changing technology. the papers which made up the conference (a small selection of which are published in this special issue of alt-f) would have reflected this theme even if it had not been chosen by the conference committee, since both educational thinking and educational technology are so patently in flux. educational theory over the centuries has shown wild swings between, on the one hand, an approach based on rote learning and step-by-step dispensing of knowledge (politically safe and therefore the traditional route) and, on the other, one based on discovery and experiential learning (nearly always rejected by the establishment, at best after trial periods). at present, the direction of the pendulum is towards active learning, exploratory learning and so forth, and the wobbles i detect from time to time probably do not indicate an imminent swing in the opposite direction among educationists, since these wobbles are mostly caused by politicians. running parallel with this move towards learning centred on partnership, interactivity and discovery are the developments in learning technology with which alt is so deeply concerned. but user-expectations are still largely ahead of what the technology can actually deliver. and herein lies the heart of the problem of the relationship between changing education and changing technology. current educational thinking makes ever-increasing demands on a technology which, as often as not, is unequal to the task. hence the lack of fullscreen, full-motion, fully interactive video in multimedia cal software. hence the inability of most examples of such software to distinguish between a serious student error and little more than a slip of the finger on the keyboard, let alone to see subtle implications in errors, implications which human teachers might easily sense. hence the difficulties encountered in a number of internet-related learning schemes: the supposed information superhighway can in reality be more like an information b road. so while i was hardly astonished that the majority of the papers given at the conference dealt in one way or another with changes in technology and/or reflected present trends in educational thinking, i was a little surprised that so few of them directly addressed what for me 5 gabriel jacobs editorial was the central implication of the conference's theme: the complex and potentially uneasy relationship between the two kinds of change. many papers described pieces of high-risk software high-risk because it teeters on the edge of the capabilities of the technology available to most people, and because its effectiveness is as yet unproven. the history of information-delivery technology shows that as soon as a new form of it appears (chalk and slate, the printing press, tape recorders, overhead projectors, video tapes, computer-controlled presentations, data communications . . . ) , teachers try to take advantage of its educational possibilities, with varying degrees of success. there is of course nothing intrinsically wrong in the desire to tread new ground; after all, if columbus had waited until it was perfectly safe to travel by sea, he would not have found america. on the other hand, we are dealing here not with, say, medical research where doctors try out risky techniques on patients who would otherwise die because there is no alternative treatment, but rather with an area offering a whole range of well-tried methods. changing education is not always a consequence of changing technology, nor vice-versa. the links between the two, inextricable though they are, are far from straightforward, and we who consider ourselves at the forefront of educational thinking should constantly be on our guard against the dangerous view that they must necessarily move forward hand in hand. who dares wins may be the motto of the sas, but we are not in their business. gabriel jacobs 6 computer-simulated experiments and computer games: a method of design analysis jerome j. leary department of mathematical sciences, university of brighton this paper describes a new research programme to design computer-simulated experiments in the field of fuels and combustion, and describes a method of categorization based on a taxonomy proposed by gredler. the key features which enhance science content and process skills are identified the simulations are designed to be as realistic as possible, and are built using threedimensional computer-aided design, rendering and animation tools, with the intention of creating an interactive virtual laboratory on the computer screen. a number of computer games are also categorized against the computer simulations and the same taxonomy for comparison. the paper then describes how designers of computer simulations can add to their own learning by retrospectively analysing their own simulations. background through the new modularization of the undergraduate science degree at the university of brighton, larger numbers of students are choosing to take some science modules which include an amount of laboratory practical work. indeed, within energy studies, the fuels and combustion module, for which the computer simulations were written, has seen a fourfold increase in student numbers from twelve to around fifty. fitting out additional laboratories with new equipment to accommodate this increase presented problems: the laboratory space did not exist; fitting out the laboratories with new equipment would involve a relatively large capital spend per student for equipment that would be used infrequently; and, because some of the experiments use inflammable liquids and gases, additional staff would be needed for laboratory supervision. however, the option of running a science-based module such as fuels and combustion with students doing fewer practicáis was not desirable from an educational view, since experimentation plays a key role in enabling students to develop science process skills (gibbs & jaques, 1990), and a reduction in the amount of experimentation would have a consequential effect on students developing those process skills. the necessity of teaching 40 alt-] volume 3 number i the maximum amount of science content and science process in a cost-effective manner leads to consideration of computer simulations as a supplemental means of instruction. indeed, the opportunity was seen that the total number of experiments that an individual student performed could be increased with the use of computer simulations. much research has already confirmed that simulations are no less a viable means of instruction than any other form of instructional approach (for example, hedlund et al, 1986; sherwood et al, 1984), and in fact studies have shown that students achieved as well, if not better, on the post tests through computer simulations. magín and reizes (1990) suggest there is considerable unexploited potential for computer simulations in education in the field of laboratory experiments, but in reality there is little computer simulation software available for undergraduate teaching in the field of science. in 1992, this dearth lead to the instigation of a research programme by the author of this paper in computer simulations within the university (leary, 1994). the latest three-dimensional computeraided design, rendering and animation tools would give the experiments a sense of realism. the experiments it was initially planned to develop eight computer-simulated experiments in the field of fuels and combustion. this phase is now complete and the simulations are currently being tested on second-year energy studies students. the students use the simulations to: • determine the gross and net calorific values of natural gas using a boy's calorimeter; • determine the upper and lower flash points of liquid hydrocarbon fuels using the pensky martens apparatus; • measure the relative density of liquid hydrocarbon fuels by the balance method and with hydrometers; • determine the viscosity of liquid hydrocarbon fuels by using a redwood viscometer; • determine the fractions of light hydrocarbons in gasoline using a distillation column; • measure the pollutant emissions from a wankel engine; • determine the constituents of coal and efficiency of combustion by measuring the products of combustion in a fluidized bed boiler; • investigate diesel-engine efficiency and diesel knock by varying fuel injection profiles. classification of simulations before undertaking any redesign or refining of the initial eight computer-simulated experiments, or indeed developing any further simulations, it was felt necessary to perform an analysis to identify the factors involved in optimizing the success of the learning outcome and, thereby, add to the designer's own learning. as a comparison, some popular computer games would also be analysed on the same basis. the model chosen for the analysis was gredler's taxonomy (gredler, 1986) of computer simulations which identified four categories of simulation: 41 ¡eróme j, leary computer simulated experiments and computer games 1. structured questions with associated graphics. the learner views the simulation, then answers specific questions. these can be either drill-and-practice or tutorial type exercises. 2. assignment of variables with results exercises. one or more independent variables are set by the learner and the results on the dependent variable or variables are observed through several trials. the nature of this simulation is cyclic, feedback being related to variables affecting other variables. 3. diagnostic simulations. this type of simulation usually involves applied decisionmaking in a professional content area leading to the development of 'expert knowledge'. a decision variable is selected by the learner after several episodes, and there is usually one large scenario. 4. group-interactive. this is applied decision-making in a real social context. distinguishing the correct approach in context is the principal learning outcome. there is no right or wrong answer in this case, and the results of decisions are difficult to determine. none of the laboratory experiments falls into this category, so the category is not considered further. computer games the following computer games were chosen arbitrarily on the basis of being commercially successful and representing a cross-section: • microsoft flight simulator • pong • simcity • doom anal/sis of the taxonomy table 1 identifies the fit between the eight computer-simulated experiments described above, the four computer games, and the taxonomy proposed by gredler (1986). the descriptions of type, or categories, are listed on the left, and the shaded boxes identify those items which match each of the four classifications. if there is a match between a computer simulation or a computer game and a category, a star is used to indicate this. then, in looking at the data in each of the columns, it is a matter of identifying a match between each column and each of the columns with shaded boxes. if the'pattern is the same, the category of simulation is clearly identified as being either of the four types. the taxonomy splits the category-structured questions with associated graphics into two other categories: drill and practice, and tutorial. the conclusions that can be drawn from the analysis of the simulations in the table are that: • there is a weak but marked correlation between both the tutorial and variable assignment categories and the majority of the simulations; 42 nature of situation presented i.i 1 1 1.3 1.4 specific problems each requiring a particular answer • a continuing situation that includes two or more variables [ complex real-world problem that takes new directions in response to in | prt" use of visuals and data | | less« ii a1 n.a2 h.a3 >n preparation visual presentation of a real-world problem • graphic presentation of a set of data i -'v feedback to students ii.b1 ii.b2 ii.b3 ii.b4 identifies right and wrong answers provides clue's to errors in student's selected strategy identifies outcomes of student's assignment of values to variables 1 describes new direction in a complex problem that follows student's inp nature of instructional process hi.1 m.2 in.3 h1.4 iu.5 solve a set of discrete, specific problems identify new examples of concepts or rules disco ver the optimum strategy to solve a problem 1 discover the set of values identified as optimum for several variables i implement the knowledge acquired in a particular subject area v ar . a ss ig n . i u t | • experiment number 1 * r 4ü * ** 2 * 3 * * 4 * * * 5 * * * 6 * 7 * * 8 * •—• * * * * * * * • h h condition i l l optional table i: classification of computer simulations and computer games jerome j, leary computer simulated experiments and computer games • there is strong use of visuals in both demonstration of items and presentation of the running experiments; • the simulations provide only weak feedback and do not identify errors in the student's selections; • simulation number 3 shows marked characteristics of the drill-and-practice class; • the nature of the instructional process of simulations 7 and 8 is akin to the diagnostic class. this analysis provides an extremely useful agenda for further design work on the simulations with the issue of weak feedback heading the list. it also provides an insight into the timing of the use of the simulations within the fuels and combustion module as a whole, as it is apparent from the taxonomy that simulations 1, 7 and 8 are partly diagnostic in nature and require some prior knowledge, that simulations 2, 4, 5, and 6 are mainly variable assignment and require minimal prior knowledge, while simulation 3, as a drill-and-practice type, can be used without any prior knowledge of measuring relative density. lastly, the analysis also provides a level of comfort as it confirms that about half of the simulations fall into the assignment of variables class, which clariana (1989) indicates do result in improved learning of specific content and also of more generalized cognitive processes. the games were also analysed using gredler's taxonomy and the results are also shown in table 1. it is quite interesting that there is a strong correlation in all cases with variable assignment and diagnostic categories. the emphasis in these games is providing an environment in which players can test their skill, be it landing an aircraft, playing ping pong, managing a city or doing battle, and although the teaching of these skills is not a primary aim of the software commercial success is the development of the skills does occur, and the players in all cases do improve their performance and increase their knowledge. however, although the analysis using the taxonomy provides much useful information, it leaves unanswered some questions concerning the effectiveness of the simulations. this is because these questions primarily concern learner understanding rather than typological issues of context and feedback, and are, therefore, outside the ambit of gredler's taxonomy. when addressing issues of learner understanding there is, unfortunately, no single reference point like gredler's taxonomy, and a comprehensive view of the issues is gained only by studying relevant literature, some of which is discussed below. they include areas such as enhanced reality, learner guidance or learner control, level of questioning, degree of fascination, and transfer. further issues enhanced reality. the eight simulations developed for the fuels and combustion module are, thanks to the latest 3d graphics tools, quite realistic, and this is borne out by student feedback. realism is considered important as many of the undergraduates on the module have no prior experience of what some fuel-testing apparatus looks like. thus, important features have been enhanced using animations and cut-away views. for example, 44 alt-j volume 3 number i figure i: boy's calorimeter simulation simulation number 1, which uses a boy's calorimeter (figure 1) to determine the gross and net calorific values of gaseous fuels, shows a cut-away view of the inner workings of the calorimeter, allowing the student to view the direction and magnitude of water flows, gas flows and the combustion flame. this view conveys additional information which would not be available in a conventional laboratpry. furthermore, where possible, taskirrelevant cues have been eliminated or reduced. again using simulation number 1 as an example, the experiment would, in reality, involve a mass of wiring, connecting thermocouples and other components to temperature meters, whereas the simulation shows only simple digital thermometers. the simulations also highlight features that may not be visible in reality, such as heat flows, and show magnified views, such as a single oil droplet burning, and slow down time so that fast processes, such as combustion occurring in a diesel engine, can be studied in detail in slow motion (figure 2). successful implementation of these features was a key design consideration, and the ability of the eight simulations to aid the students' understanding more effectively than traditional laboratory-based experiments is therefore deemed to be high. learner guidance or learner control. it has been shown, for example by vockell & rivers (1984), that guided discovery computer simulations are generally better than the exploratory computer simulations, particularly in the development of general problemsolving skills. this is contrary to the commonly held view that free exploratory discovery is best for learning science process, and that guided discovery is best for learning science content. in any case, simulations of a diagnostic nature require prerequisite knowledge and provide a test environment to try out that acquired know-how. in other types, it is more difficult to determine in advance exactly how much learner-guidance to include in each of the simulations, so the method for deciding this is an interactive one based on student feedback. the amount of student questioning is monitored and used to include or remove additional information. 45 jerome j, leaiy computer simulated experiments and computer games figure 2: diesel engine simulation level of questioning. this is aimed at stimulating reflection on what has occurred during each experiment. a correct level of questioning is important because reflection is needed to create a deeper understanding, and in order to develop cognitive processes in the learner. in a study by merrill (1985), the use of higher-level questions in computer simulations was supported, but not lower-level questions where individual attributes of a concept are treated in isolation. in the simulations described in this paper, students are asked a set of higher-level questions at the end of the experimentation phase to encourage reflection. in diagnostic simulations and computer games, students are already working at a deeper level, testing and refining their mental model continuously, and so additional questioning is less appropriate. fascination. the level of difficulty of a simulation is important. ideally, there should be an exact match between the learner's ability and the difficulty of the simulation to stimulate interest. if a simulation is too easy it will result in boredom and poor skill development, whereas if a simulation is too difficult it will cause anxiety in the learner and a failure to understand or complete the task. however, if there is an exact match, the learner is more likely to proceed with maximum concentration, or even become fascinated with the task, at which point learning occurs with consummate ease (mayhew 1994) as the student becomes addicted. with computer simulations, it is much easier to match the difficulty of the task with the level of competence of the learner. some computer games do this particularly well with different skill levels. with real experiments, this process is limited by physical constramts. student feedback suggests that the level of difficulty for one or two of the experiments is too low, in particular simulation number 3. table 1 confirms that this simulation belongs to the drill-and-practice class, which are routine in nature, and therefore we can conclude that simulation number 3 needs to be redesigned to make it more challenging. validation. one desirable learning outcome from any experimentation is development of skill in validation of results. rather than trusting implicitly the output of the experiment, as students are more likely to do with computer generated results, in these experiments 46 alt-} volume 3 number i students are made aware that they should treat the measurements from the computer simulations with the same degree of caution as they would measurements from equivalent real experiments. several processes are incorporated into the simulation to ensure that good results can be obtained only with careful procedure. for example, a mathematical model which is dynamic, with built-in time delays, thermal inertias and so on; a measurement variation or 'noise' added to measurements; and, thirdly, variations introduced into the parameters of the model so that it changes gradually over time in a random manner. the level of these contingencies actually built in determine the development of experimentation skills and lead to a fuller appreciation of the nature of empirical investigation. transfer. the above issues directly affect transfer, that is, the ability of a student to use knowledge in either a similar context (near transfer) or a different context (far transfer). this is tied in with whether the student is gaining a surface knowledge or a deeper understanding (entwistle, 1993). to perform far transfer, the student must acquire a deeper understanding, and in general simulations of a diagnostic class are better at this than drill and practice. however, other issues are also important, for example enhancing reality, providing an optimum level of learner guidance, generating higher level questions, and making the simulations fascinating all help the student achieve a deeper understanding and obtain skills in both near and far transfer. conclusions it is natural for designers of simulations to want to undertake some form of analysis of their work to identify how their own learning can be improved. more effective designers lead to more effective simulations. educational software also has a lot to learn from computer games in their potential to support that old adage that learning can be fun. gredler's taxonomy is a useful tool for assessing the context and feedback characteristics of simulations and comparing them to other types of software, such as computer games, and it can provide useful pointers for retrospective design work. an assessment of the effectiveness of characteristics beyond typological considerations is, however, much more difficult as there is no single reference point as yet that provides a model for comparison. the analysis of issues such as enhanced reality, learner control or program control, level of questioning, fascination, and transfer, is, therefore, judgmental and often subjective. as designers' judgement about the effectiveness of their own simulations may be clouded by their close proximity to the development work, other judgements such as the students' feedback should be taken into account. despite the problems associated with the assessment of learner understanding, the exercise outlined in this paper encourages the designer to take a holistic view, and provides criteria which will tend to make his or her work more effective. references clariana, r.b. (1989), 'computer simulations of laboratory experiences', journal of computers in mathematics and science teaching, 8, 2, 14-19. 47 jerome j, leary computer simulated experiments and computer games entwistle, n. (1993), recent research on student learning and the learning environment, centre for research on learning and instruction, university of edinburgh. gibbs, g. & jaques, d. (1990), labs and practicals, oxford, educational methods unit, oxford brookes university. gredler, m.b. (1986), 'a taxonomy of computer simulations', educational technology, 26, 4, 7-12. hedlund, d.e. & casolara, w.s. (1986), 'student reactions to the use of computerised experiments in introductory psychology', educational technology, 26, 3, 42-5. leary, j.j. (1994), 'computer simulated practicals', paper presented at the 29th international conference of the association for educational and training technology (napier university, edinburgh, 11-13 april). magin, d.j. & reizes, j.a. (1990), 'computer simulation of laboratory experiments: an unrealised potential', computers and education, 14, 3, 263-70. mayhew, j. (1994), motivation, interest and learning, school of law, social work and social policy, liverpool john moores university (forthcoming). merrill, j. (1985), 'levels of questioning and forms of feedback: instructional factors in courseware design', paper presented at the annual meeting of the american educational research association (chicago il, march 4). sherwood, r.d. & hasselbring, t. (1984), a comparison of student achievement across three methods of presentation of a computer based science simulation, technical report series, report no. 84.1.5 (eric document reproduction services no. ed 262 750). vockell, e.l. & rivers, r.h. (1984), 'computerised science simulations stimulus to generalised problem solving capabilities', paper presented at the annual convention of the american education research association (68th, new orleans la, april 24). 48 reviews lumsdaine, a.a. and glaser, r. (eds) (i960;, teaching machines and programmed learning, washington dc, n.e.a. pressey s.l. (1926), 'a simple apparatus which gives tests and scores and teaches', school & society, 23, 373-6. unwin, d. (1969), media and methods: instructional technology in higher education, london, mcgraw hill. wilson, b. g. (1996), constructivist learning environments: case studies in instructional design, englewood cliffs nj, educational technology publications. judith jeffcoate, multimedia in practice technology and applications, bcs practitioner series, prentice-hall international, 1995. isbn: 0-13-123324-6. £24.95. ray welland concludes his editorial preface to this book with the sentence: 'if you want to find out what multimedia is and what it has to offer this is an excellent place to start'. the comment reflects the major strength of this book, which does provide an excellent starting place for anyone wanting to learn more about multimedia. the author explains what multimedia' is today and the technology involved, gives example applications, and projects current trends forward to explore the use of multimedia in the future. she clearly states the perceived audience for the text as 'business managers and project leaders who need to introduce multimedia technology into their applications'. a guide to using the text is provided: parts i and iii are suitable for readers with no technical knowledge; part ii requires familiarity with computer architecture, and part iv is most suitable for managers planning a multimedia project. these guidelines are useful, but the text as a whole will appeal to anyone who has an interest in learning about multimedia. technical knowledge is advantageous in some sections, but the author takes time to explain clearly, and uses examples to illustrate points. the book is very well structured. the author provides an excellent summary of the content in the preface. this clarity of structure continues throughout the text; for example, each chapter begins with a brief outline of its contents and concludes with a summary and references. multimedia is a subject engulfed by acronyms and, to prevent confusion, the author provides a detailed glossary which at times proves to be very useful. a bibliography and a full index are further resources which the reader can draw upon. part i, 'multimedia today', begins by painting a scenario of a modern family using new technology in their everyday lives, then projects the same family into the year 2001, highlighting the role technology may play in their lives in the future. part ii, 'technology', deals with the complex areas that make up multimedia, and is the most technical section of the book, dealing with different platforms, development tools, storage methods and the components that make up a truly multimedia application such as images, audio and video. the chapters focusing on the different media are divided into sections describing applications, capture, compression and standards, as well as detailing information specific to each medium. a history of the media is provided, enabling the reader to see, for example, .how standards have evolved. this technique of placing information within its historical context is used throughout the book, and is vital for a reader who may be coming to the topic for the first time. part iii returns to the theme with which the book began: 'applications'. it focuses on present use and future trends. again the author places the use of multimedia in its historical context: the early use of multimedia was largely for off-line training and education using interactive videodisk and standalone players. 72 alt-j volume 4 number 2 chapters in this section detail multimedia uses in training and education, kiosks, image processing, the office and the home. the author draws on real world examples. part iv, "the impact of multimedia', highlights other issues that are important in the development of applications, discusses the human-computer interface, tackles legal issues, and identifies steps forward in computing that will influence the development process such as collaborative computing and object-oriented design. the author says in the preface that she faced a complex task in writing the book 'because multimedia is not a single technology'. the technology and applications are explained well, and recent research is used effectively to highlight present and future trends. however, a reader looking for greater detail on a topic will have to explore other texts. i would nevertheless recommend this book to anyone interested in, or already involved in, multimedia. it is such a diverse area that we can all benefit from what this book provides: an insight into the parts that make up the whole. julia little, university of wales, aberystwyth david boud, enhancing learning through selfassessment, london, kogan page, 1995. isbn: 0-7494-1368-9.247 pages, paperback, £19.95. at the heart of learning technology is a complex web of ideas on learning; how teachers assess it and how learners can influence it. david boud's book provides us with a rich source of inspiration that should influence any learning technologist. central to education and training is a need to ensure that students benefit from their time engaged in education and what we think of as learning. after an effective class, learners can do and say things they could not have done before. they feel they know something and they are aware of their learning. we have only a slender conception of the mysteries of consciousness of performance at present. how is it that we are conscious of what we know, and how is it possible to influence our knowing or not knowing? often when, learners or trainees are asked 'what did you learn from such and such an intervention?', they can only answer 'nothing!'. to the courseware designer this is a cruel attack. the careful design of screens, the consultation with experts and the attention to the latest multimedia techniques will lead to knowing and an awareness of knowing. not only will the learner know, but others will, through assessment, be able -to judge the intervention as effective. sadly, this does not always occur. although we have a very effective examination and assessment system in education and training, designers often feel that assessment is not for them. assessment is external to the process of design. assessment smacks of formal education schools, key stages, highers and a levels: the external artefacts of a quality control system that we all have to suffer. boud's book addresses issues of assessment and examination, not from the outside the observer's view but from the learner's point of view. learning is a self-regulating process. admittedly, learners do not always see it in this way. often they feel the only function of education is to get them to the examination room so that they can display their ignorance. learning technology must address this problem, and this book is an essential guide to the way we can produce more effective and efficient learning experiences. at the heart of boud's excellent survey of ideas on self-assessment is reflection or automonitoring. good learners are aware of what they know and how they come to know. learners who fail not only seem unaware of what it is they should know, but what they can do about it. boud has provided evidence and stories to give designers plenty of thought. the book is not a single narrative but a number of multiple perspectives on self-assessment. boud can speak with authority: he has been writing about reflection on learning for many years. currently he holds the position of head of the school of adult and language education in the university of technology in sydney. he worked in the uk in the 1970s, and after a sojourn in perth, moved to the university of new south wales where he developed ideas in staff development. he is a fellow of the society for research into higher education and 'weel kent' across higher education. the book consists of 17 chapters. in seven of these, boud is ably assisted by other practitioners and experts in higher education. the book not only provides the theory and research with an extensive bibliography but also good examples of how the ideas are put into practice. boud has divided the book into four parts. after "theory and examples', part 73 reviews iii makes compelling reading as it deals with issues relating to the effect of self-assessment on formal examination processes. the final section considers, in a down-to-earth way, the practical implications for self-assessment. here, i would like to have seen more attention to the way courseware designers learning technologists can make a positive impact on learning. too often i feel that designers are over-concerned with presentation. they know about networking and multimedia, but there is a 'correctness' about shying away from assessment. orthodox design seems to avoid assessment as it is 'othercentred': something that the system will do to the learner. if only we could build into courseware, and other instruction, opportunities for learners to reflect on their competencies, we would be making a significant contribution to success in life-long learning. one simple example should suffice. feedback on performance is essential for reflection to occur. feedback has been shown to have a significant influence on the attitudes of learners to instruction. boud provides an excellent checklist on offering feedback, and although the advice is general and global, if designers were to read this and to do something about it, learners would be better able to reflect on and influence their own learning. boud says: 'be timely; be realistic; be specific; be descriptive; be non-judgmental; be diligent' all things that good designers should be. often the best advice we can get comes from an unexpected quarter. if we feel that assessment is only to do with authority and quality control, we miss an important opportunity. self-assessment is at the heart of learners owning the ability to influence their own success. designers should thus read and pay attention to david boud in this excellent book. he is an acknowledged master of putting difficult ideas in straightforward terms. acquire this book and make use of it. ray mcaleese, heriot-watt university ian mcgill and liz beary, action learning, london, kogan page, 1995. isbn: 0-7494-15347. second edition, 274 pages, softback, £19.95. this guide sets out to provide a practical approach to action learning. aimed at readers seeking to develop their personal learning through participation in action learning groups, as well as those wishing to introduce the process into a wider educational programme. the underpinning principles upon which action learning is based are discussed in sufficient detail to provide a basic grounding, but the main strength of the guide lies in its focus on the successful practical implementation of action learning sets. this includes not only guidelines for implementing an introductory workshop, but also for the range of skills required by set members to enable them to participate effectively as action learners. the authors aim to demystify action learning, and reveal its strengths as a method of personal learning and development. the book is divided into three sections. in the first, the focus is on the meaning of the term action learning, and the presentation of an understanding of what it is to be an action learner. this is then developed in the second section, which looks at effective set membership and the types of skills needed to ensure that the individual is able to participate fully in the action-learning process. the final second places action learning in a number of professional and educational settings, and shows how it can be used to enhance personal development and learning in those contexts. in part i, action learning is described by the authors as the 'process of learning and reflection that happens with the support of a group or "set" of colleagues working with real problems with the intention of getting things done'. in chapter 2, action learning and the role of the action-learning set are defined, with particular emphasis on the importance of the relationship between action and reflection. action learning is based on the principles of kolb's experiential learning cycle, which embodies the idea that systematic reflection on prior experiences can help in the generation of future ideas, plans and actions. a useful distinction is made between action learning and action research in order to avoid any potential confusion between the two approaches, given that both are closely aligned with models of experiential learning. chapter 3 describes how an action-learning set is formed, and how it works at a structural or organizational level, setting the scene for the two following chapters which describe in detail what it means to be a set member and a set facilitator. the concluding chapter in this section is an examination of different types of action learning. in part ii, consideration is given to the way in which action learning can be introduced to potential participants. this section is very much 74 alt-} volume 4 number 2 about getting action-learning sets underway, and a large part of it is devoted to presentation of a format for an introductory workshop. this includes a schedule and a set of suitable supporting activities. the remainder of this section describes the interpersonal and learning skills required by set members, which will enable them to participate fully in the activities of the group, and for the set to function effectively. the final part of the book looks at the potential of action learning in the settings of continuing professional development, management development and higher education. the concluding chapter in this section addresses the potential contribution of action learning to individual, organizational and social change. in the final chapter of the book, the authors express a concern that by writing a how-to text, they run the risk of being too prescriptive and of presenting • a definitive approach to action learning. this is not the case. what they do present is a clear exposition of the underlying philosophy of action learning, with a sense of the boundaries within which it can be shown to work effectively. the book presents action learning in a clear and logical way, with a good balance between attention to the underlying principles and the practicalities of being a set member and/or facilitator. this book is thus a valuable resource, which will be useful to a wide range of readers, from those interested in finding out a little more about what action learning offers, to those seeking more practical guidance in the facilitation of, or participation in, an action-learning set. the authors convey not only a genuine belief in and commitment to action learning, but also an enthusiasm for the contribution it can make to personal development and learning. it is clear that this book is written by two people who care about action learning, and who wish to share the insights gained from their own professional practice with a much wider community. susan fowell, university of sheffield higher education 1998 transformed by learning technology, edited by joyce martin, jonathan darby and bengt kjoellestroem, published jointly by cttss publications (cti support service, university of oxford) and the swedish council for the renewal of undergraduate education, university of lund, 1994. isbn: 0-9513896-6-1. softback. although the art of prophecy is problematic; time tells whether our view of the future has been rooted in the present. the art of prophecy is problematic because most seers are on a hiding to nothing. we live in a time of change, and merely predicting that change will come about is a truism. the issues of interest are the rate of change, its nature and the vehicles by which it arrives on our doorstep (or gets lost en route). those involved in the promotion of it and other forms of resource-based learning are frequently driven by evangelical zeal, and are often at a loss to understand why their enthusiasm for all things digital is not shared by others as they proclaim their millenarian visions. on the one hand, technophiles tend to dismiss sceptics as luddites and dinosaurs, while those of a more moderate disposition simply assume that training is the panacea for those poor myopic souls whose narrow conceptual horizons condemn them to a sequestered life in the technological void. both sides, paradoxically, usually view the digitization of the world as inevitable. the only question is how long the resistance can be maintained. this book is a brave attempt to operationalize a view of the future. how well does it do it? unfortunately, not well. even though its scope is specific, comparisons with negroponte's being digital, and even bill gates's view of the road ahead, are inevitable. both those books were written by people with a serious vision (which we may or may not share), but, by way of contrast, higher education 1998 is less a vision of what life will be like on the campus two years from the millennium than a series of mechanistic extrapolations. in other words, it has been assumed by the contributors (and certainly the editors) that we will be doing much as we do now, only more rapidly and with better technology: more work at home, sandwich-style study, extensive use of email, and a centralized network providing communications and learning resources. the change conceived is, from both an academic and organizational perspective, incremental rather than qualitative. the event from which this book derives was the first (and perhaps last) international cti jamboree. the computers in teaching initiative had been set up in the uk in 1985 by the then universities' computer board, to promote the use of computers in university teaching. the first of two phases centred on the funding of 139 software development projects, many of. which 75 reviews quickly sank without trace. in the second phase, it was recognized that software did not implement itself, so human beings were funded to operate in academic disciplines through a dissemination network of (currently 24) cti subject-based centres, most of which were established in 1989 and which remain in place. since they began, all cti centres have published journals and newsletters, undertaken reviews of courseware, provided an advice service, visited departments within their disciplinary constituency, and organized workshops and conferences. there have been two attempts to establish the cti model beyond the uk: in eire and sweden. of the former we know little. prospective founder members of the swedish cti attended the uk cti 1992 spring forum in stirling; and in the following year a joint conference was held in lund for members of the uk cti and colleagues from sweden keen to establish a similar organization in their country. the theme of the conference was a futuristic one: 'higher education 1998 transformed by learning technology'. a scenario was set out, and the question asked: 'how did we get from there (1993) to here (1998)?'. the conference took the form of formal papers by invited speakers, case studies, and participant-work in small groups looking at various problems. the fruits of all are represented in the proceedings published here. the task for the conference delegates was to undertake a retrospective look 'back' over five years and to analyse the processes that marked the changes posited to be in place. at the time of publication of this review, we are now more than halfway 'there'. so with the benefit of partial hindsight it is possible to begin to assess how accurate were the original vision and the integrated analysis. learning technology is not just perceived by its proponents as being an application in search of a solution. it has much loftier ambitions: nothing less than cultural change. inherent in this assumption is the view that because the adoption of information technology has already brought about a considerable degree of change in society generally, linking together higher education and computers generates an inexorable facility for widespread change in universities. unfortunately, it zealots seem remarkably naive as to just how much de facto opposition there is to the use of computers within teaching contexts in higher education. granted, there is little likelihood of midnight raids by modern-day torch-bearing luddites bent on physical destruction of computers, but that probably has more to do with the fact that most of them do not know where the bulk of computers are kept. most of the 'opposition' seem quite content to do nothing; given that only a small (though growing) section of the lecturing community has come around to the idea of using it in teaching. this suggests that the status quo says more about those doing the promoting than those being addressed. many lecturers teach it, rather fewer teach with it. i was disappointed to find the book unconvincing, lacking in sedulous argument, conceptually disjointed, badly written in some parts and simply execrable in others. this is not to impugn the editors for the variable quality of the writing, although the imperious constraints of a publishing deadline no doubt occasioned hasty editing in some sections of the work. the chapters hang very loosely together all too typically in edited works these days. the worst example is of the final case study on interactive visual electromagnetics, leading straight into the section reporting on the outcome of smallgroup discussions, without any signposting for the reader to prepare him or her for what is to come, as shown by the first sentence of the next chapter: "there was a consensus in the group t h a t . . . ' . anyone who missed the second paper, which set the stage, would be totally confused. even those who do not use hypermedia do not necessarily start at chapter 1 and read books in an invariable linear sequence. if the key to rooting the future in the present is dependent on international co-operation, all will come to naught. both the irish and swedish ctis have failed to take off. a recent survey showed that, except in one cti centre, no active co-operation now exists beyond the exchange of journals. the uk cti centres have some sense of history, and many of their papers do provide an introspective temporal perspective on developments. unfortunately, many of the swedish papers are simply descriptive snapshots of the 'look what we are doing' variety. more importantly, the book fails to take into account the two insuperable obstacles to change: the combined lack of funds, and the erroneous assumption endemic in both the swedish and us contingents that once the technology is in place, everyone and his dog will flock to make use of it. this is a myth, and many of us who were there were rather surprised to learn that many (but not quite all) of the swedes could not 76 volume 4 number 2 believe that there was a widespread degree of resistance in higher education. neither could they grasp something we knew only too well in the uk: the absence of a common will to implement the necessary structural and organizational changes in institutions and curricula. conservatism in academia is so pervasive as to be ubiquitous, even inherent, and the lack of reward to overcome inertia is a vital catalyst; without it, the vision of the future promoted here remains in the future. another missed element is the axiom that those who neglect to learn the lessons of history are condemned to repeat them. very few of the participants here had any sense of past development, something being mirrored in many tltp projects. it is almost as if new groups arrive on the scene every two years or so, totally ignorant of what has transpired to date. beware the it enthusiast: his/her evangelical fervour automatically seems to block out the past, almost as if the past never existed only a future to which they strive. technological upgrades seem to make more than equipment obsolete: there also seems to be a concomitant vaporization of modes of use. the major flaw is made explicit in the introduction to the final paper: 'funding and technical obstacles were not allowed to affect the group's discussions, so the picture that emerged [. . .] was the perfect scenario for the student in 1998 sufficient finance was available and all technical problems were resolvable'. worse is to follow: '[. . .] it is important that there is a sound it strategy led by computer services within all institutions.' and we are not done yet: "the university's administration will also make use of it for it will help overcome scheduling difficulties created by the increase in student numbers.' with flawed thinking like that, no wonder nothing substantial emerged. the future remains open. next, please. john castleford, university of leicester jim sterne, world wide web marketing, new york, wiley, 1996. isbn: 0-471-12843-0. 331 pages, softback, £15.95. this is the best book i have read on using the world wide web (the web) as a marketing device. it is profusely illustrated with sample web pages, and it ranges from discussions of detailed issues on layout, buttons, and graphic design, to measuring your impact, establishing your criteria, and making your site fun. even if you use the book just to get an overview of the variety of information sources on the web and how people have used it for different aims, it is well worth looking at. and if you are particularly interested in promotion, marketing or other related value-added services, you probably need go no further than this book. there are 14 chapters. they cover a very good general introduction to the internet, and then focus on the web. the author looks at graphics, feedback, interactivity, value-added marketing, gaining attention, measuring success, starting points and, finally, adds a very interesting chapter which looks towards the future and discusses where the web might be in a year or two. the author claims that the book is written by a marketer for marketers. i am sure that is true, but there is also a lot of useful information that a non-marketer will get from it, and it will be a very good index against which to judge marketers' advice. anyone who has a web site which is essentially providing information will have plenty of indices against which to judge that site. and the book will provide a good way of seeing through marketers' recommendations and establishing your own objectives (as well as of borrowing ideas). sterne puts some very interesting information in his appendices. he gives a run-down on a range of discussion groups all to do with marketing, including academic marketing. he then lists a whole range of marketing-oriented web sites that he recommends, and finishes the book with a useful index which takes you through most of the sites he has mentioned, the companies, the concepts and so forth. even though the web is doubling in size every 53 days and is changing radically as it grows, this book is still up to date, and has a large amount of valuable data. all the way through we get the distinct impression that sterne knows what he is talking about. his style is straightforward and, what is more, every page is displayed from netscape navigator on a macintosh (the fact that the book, so very much focusing on business, uses a macintosh as the delivery engine, appealed to me!). if i have a criticism, it is that in describing pages which are displayed, sterne sometimes states the obvious, and there is the occasional over-use of 77 reviews exclamation marks. but on the whole, this is a well-written and attractive book, and oversized to allow for the large use of graphics. indeed, if you do not want to listen to sterne's own account, just by looking in detail at the illustrations, you can get a fairly clear idea of the range of uses of the web, and some of the clever ideas which have been picked up by a range of large and small companies. i suppose i can sum up the book by quoting the final paragraph: 'so go surf the web. assemble a team. read some books. get some tools. write a style guide. get started. get started soon. it will be much harder to catch up later'. the book is a good and fast read. nigel paine, chief executive, scottish council for educational technology fred t. hofstetter, multimedia literacy, new york, mcgraw-hill, 1995. isbn: 0-07-9119565.360 pages, paperback, plus cd-rom, £20.42. this book is published as an a4-size (american) edition with an accompanying cd-rom. the disc is intended for use on mpc (multimedia personal computer)-compliant systems. the 40 chapters that make up the book are organized into seven basic parts. of these, the first four (chapters 1 to 17) are devoted to understanding multimedia, a survey of its applications, selecting suitable hardware, and looking at imminent and long-term future developments. the remaining three parts (comprising chapters 18 to 40) each present a hands-on tutorial using the various software resources presented on the cd-rom. these tutorials cover multimedia tools and techniques, developing a simple multimedia application, and creating advanced' multimedia packages. the chapters begin with a statement of objectives which specify what readers should be able to achieve on completion of their associated study units. apart from those in part 6, every chapter includes a set of exercises which provide various opportunities for involvement in hands-on activities using the cd-rom. the 'theory' section of the book (parts 1 to 4) includes introductory chapters that deal with the various mpc specifications, a multimedia object taxonomy, and descriptions of various application areas such as education, entertainment, business and industry, government and medicine. the sections that deal with corporate training, just-in-time training, medical training and education are particularly interesting, and relevant to those involved in the pedagogic aspects of computer use. subsequent chapters in this section deal with a variety of different topics including application development packages (powerpoint, harvard graphics, toolbook, podium see below -, authorware professional and quest), competing multimedia standards ('multi-multimedia'), multimedia pc components, checklists for buying a multimedia computer, and how to configure mpc systems. some thought is also given to possible future developments in these areas and the legal issues that multimedia technologies create. the book therefore considers recent developments in electronic publishing, the impact of the information superhighway, virtual reality and emerging technologies. various societal issues associated with multimedia (and the internet) are also discussed. throughout the theory part of the book, the author makes frequent references to demonstration material contained on the cdrom. thus, after reading about a topic, system or package, you can follow up these references and use demonstration versions of the software discussed in the text. the hands-on section of the book is based on the use of a multimedia authoring tool called podium. this tool is available for purchase from the university of delaware, but a cutdown version of the system is supplied on the cd-rom for use with the tutorials. the first eight of these tutorials (chapters 18 to 25, which make up part 5 of the book) deal with the use of screen-based text, simple graphics (based on bitmap images), triggering events, waveform audio, cd audio clip-making, midi sequencing, digital video recording, and videodisc clip-making. for those who are new to multimedia techniques, the practical activities embedded in these chapters would serve as a useful means of developing the necessary background skills. in part 6 (chapters 26 through 30), the tutorials lead readers in a stepby-step fashion through the development of a simple multimedia application that deals with the history of flight. the essential steps involve creating a pictorial menu (using images of aeroplanes taken from the cd-rom), and linking these to related multimedia information (again taken from the cd-rom). the ten chapters that make up the final part of the book are used to illustrate the stages involved in creating a more advanced multimedia application. the case study that is chosen deals with the 78 alt-] volume 4 number 2 use of various information-superhighway services such as ftp, telnet, gophers and news feeders. within this section, readers are taken through the steps involved in creating the basic package using podium. a bill-board metaphor is used as the basis for the end-user interface; this is also used for linking in the various internet resources that are called upon. i found this book exciting and easy to read (it contains a multitude of useful illustrations, all of which are in colour and of an extremely high quality). the book is very readable for two reasons: first, because it does not delve too deeply into the highly technical detail of multimedia; and second, because it gives comprehensive explanations of all the steps involved in undertaking multimedia development tasks. because it uses a friendly hands-on environment for teaching, this book would undoubtedly make a useful starting point for those who are computer literate and want to find out more about creating and publishing multimedia learning resources. although the book can be read independently of its cd-rom, the latter certainly adds value to the learning experience. the polymedia approach to publishing (paper and cd-rom) taken here is, not surprisingly, becoming an increasingly popular way of providing active learning experiences. philip barker, university ofteesside j. brown, r. earnshaw, m. jem and j. vince, visualization using computer graphics to explore and present information, new york, wiley, 1995. isbn: 0-471-12991-7. 287 pages, paperback, plus cd-rom, £35.00. to paraphrase the authors' own words: the publication is a non-technical guide which provides a brief background on the field of visualization the use of computer graphics to explore and present information. it is intended for it professionals who need to evaluate or implement a computer-graphics or visualization facility. the nine chapters lead the reader from an introduction to the concepts of visualization, through the commercial possibilities of its use, the range of currently available hardware and software and a number of case studies, to a cautious vision of likely developments in the near future. throughout, there are screen-shots of visualization applications in use and references to specific software producers and their products. chapter 1 gives a brief and rather spasmodic history of the ideas behind visualization, starting with mankind's first use of maps and mentioning the discovery of x-rays on its way to bringing in computer graphics and virtual reality. chapter 2 discusses the global markets for visualization applications in computer aided design, geographical information systems and visual data analysis. estimates of the current size of the markets and their likely growth are given as quotes from various commercial sources. the chapter continues with a brief discussion of the issues to be addressed by a company entering the field. comparisons are made between custom and off-the-shelf software, in-1iouse development and outsourcing, and the adoption of iso versus de facto standards. the third chapter covers the range of possibilities of computer-generated images and the ways they can be produced. methods of creating visualizations from subroutine and object libraries, from fourth-generation languages and from visual programming tools are catalogued. the whole range of visualization output is covered, from 2-d charts and graphs to animated, rendered, 3-d models. important concepts are introduced viewports, interpolation, contours, iso-surfaces, vector field visualization and geometric modelling, but, in common with the rest of the book, the descriptions are superficial, concentrating on breadth of coverage rather than detailed explanation. in chapters four and seven, the concepts introduced are revisited in terms of particular application fields. software requirements for computer aided design, finite element analysis, computational fluid dynamics, geographical information systems and mathematical visualization are analysed in chapter 4, and suitable visualization products for these fields are listed. there is also a discussion of typical hardware platforms from pcs to supercomputers, and of image-related output peripherals. chapter 4 concludes with a buyer's guide, a table showing the features of a number of current visualization and graphics products (for those interested, a more comprehensive list is given in the november 1995 edition of ieee spectrum). over 20 short 79 reviews visualization case studies are described in chapter 7. the rationale for using visualization techniques is given in each case, and sometimes the software and hardware used is mentioned, but there is no other technical detail. the other two chapters in the book contain an uncharacteristically detailed section on the use of colour and a history of guis, with some explanation of the structure of x-windows and motif, and a description of the internet. the accompanying cd-rom contains digitized copies of all the images in the text, some of the tables, and 16 quicktime movies showing visualization in action. none of these are explicitly referred to in the text, but they are organized into directories which match the chapters of the book. the cd is dual-format, readable by pcs and by apple macintoshes. there are two viewers included: adobe acrobat (although the buyer's guide is the only document in this format), and apple quicktime for viewing the digitized videoclips. the images and figures from the book are stored in tiff format, and no viewer is provided for them. tiff viewers are readily available, and are built into some word processors, but it does seem rather inconsiderate not to provide a way of looking at the bulk of the content of the cd (had the images been stored in jpeg format, they could have been seen with the supplied quicktime viewer).the images and movies are drawn from commercial visualization examples. the movies, many with sound, are mostly publicity or demonstration material produced by large corporations, and show very slick examples of high-quality computer graphics. it is questionable whether digitized video is the best medium to use for these examples, but i suspect that the novelty value of a book with an accompanying cd-rom is still enough to override such practical considerations. nevertheless, distributing a video tape with the book would be a better way to experience the high quality of state-of-the-art visualization. the quality would be much better, and the examples could more easily be shown to a large audience. of course, there are reasons for using cd instead; they are cheaper to produce than video tapes, more compact, and only one version is needed to cover the main formats. i felt, though, that the authors have not made the best use of the possibilities of the cd-rom medium. they could have included demonstration versions of visualization software, some of which the manufacturers are falling over themselves to give away, or simple simulations to illustrate some of the concepts introduced: visual programming, choice of colour in visualization design, and 'walking through' data, for example. they have missed the opportunity to convey the sensation of interacting directly with data, which is surely the biggest selling point for visualization tools. as for possible use of the book in education, it is too light on detail to be used as a visualization or computer-graphics textbook. it may have a place, however, as a primer for students who are about to embark on visualization or multimedia projects. the publication's broad taxonomy of applications and the state-of-the-art examples it provides may inspire them to explore more widely the possibilities of the field. richard storer, university of paisley andrew bradbury, successful presentation skills, london, kogan page, 1995. isbn: 0-7494-17498.107 pages, paperback, £6.99. this book is a new addition to the well-known kogan page series of guides for business managers, 'better management skills'. like all books in the series, the intention is to provide a practical but comprehensive coverage of the area in a readily accessible and easily read form. since effective presentation, the topic of this particular book, is informed by a wide range of quite complex material, including studies of communication theory, learning styles, body language, visualization, and memory, the author has had a particular challenge. readers need to be able to understand enough of the theoretical ideas that justify the suggestions, and to see these in an integrated framework to gain confidence for implementation. hence, within a text of just over 100 pages, there are 14 very short chapters, each one covering a range of different ideas leading to practical suggestions. the book begins with two basic messages: that everyone in business should realize the importance of presentation for success, and that everyone can learn how to make a successful presentation. in the following chapters, the author suggests how this may be achieved. 'how we communicate' includes a repeat of the message that needs to be repeated so many times the importance of non-verbal signals 80 alt-j volume 4 number 2 (93%) to verbal content (7%) in presentations. systematic ways of building up confidence are outlined in the next two chapters. later chapters deal with objectives, structure and organization. the critical thing to identify is the objective. once this is clarified, the appropriate structure and organization can be decided. the chapter on knowing your audience stresses the importance of putting the customer first. here, the treatment of learning styles may be cursory, but is sufficient to stress the need to consider them in both the preparation of material and how questions are to be handled. there are two chapters on fitting the words to the presentation frame. the first deals with how many points you can hope to cover within given time-slots, and offers advice on how to adapt materials. the second emphasizes the importance of the conclusion. presenters often pay a great deal of attention to their introduction, but neglect the need to provide a good ending. a following chapter deals with the critical first 60 seconds of a presentation. a joke? a question? an anecdote? what helps? what does not help? this topic provides a useful way into the two chapters that deal with visual aids. the first of these reviews the advantages and disadvantages of chalkboards, whiteboards, flipcharts, overhead-projector transparencies, slides, computer-based displays, video and film. this is followed by a brief treatment on designing effective visual aids. a welcome chapter on the setting up of display facilities provides a reminder to the presenter not to forget to consider the arrangements for the audience as well. as in all the chapters, there are useful tips: for example on how to deal with situations where you have a large number of seats and only a small audience, and when and where to have refreshments served. why, when and how to deal with questioners and hecklers is the subject of the next chapter. this topic not always covered in texts on presentational skills, yet it can be one of the aspects of a presentation about which the novice presenter is most nervous. the final chapter looks at personal presentation, and once again the message is the importance of paying attention to how you say what you want to say as well as what you want to say. the whole text is written in a very lively manner with ample use of boxes, bullet points, review questions, lists of key ideas, tips and mnemonics. i particularly liked panic, when called on to make an unexpected presentation. panic unpacks most helpfully as purpose, audience, need, information, communication. sometimes, however, the reader may feel the need of a further glossary. the instruction to 'indicate whether batteries are included' translates into letting your audience know what their follow-up action or success criterion will be. such examples illustrate how the book is intended for a business readership. this point does raise some minor concerns. first, such a readership may readily accept the 'sound byte' approach, but not all readers will. snappy headings are used over-abundantly not only for chapter headings but for sections within chapters, and for some readers, such a treatment may jar. secondly, some academic readers may wince at the summary treatment of their particular area of academic interest. there is a brief basic bibliography appended. but the skills of effective communication are common to business and academic presentations. this book contains many helpful practical suggestions on presentation which are relevant to readers who present research papers, lead seminars, give lectures and run workshops. and it must be admitted that although there is currently a lot of emphasis on communication skills in the academic field, the presentation of the subject matter, if addressed at all in the standard course texts, tends to concentrate on written reports. this book can help to fill the gap. dianne phillips, manchester metropolitan university j. ceserani and p. greatwood, innovation and creativity, london, kogan page, 1995. isbn: 07494-1593-2.192 pages, paperback, £12.95. this is one of the kogan page 'fast-track mba* series which aims to present core subjects from a typical mba syllabus in a lively and accessible way. the claim is that the series provides texts that have a 'practical, actionoriented style which involves the reader in self-assessment and participation'. the intended readership is managers wanting either to refresh or further to develop their capabilities, or those actually undertaking study for an mba who should find that the series provides a solid grounding in the subjects to be covered. at the 81 reviews outset, the book states that it is 'not a book', but rather a prompt to a journey of exploration. indeed, the reader is encouraged to read the main sections in any order. part 1 introduces the 'toolkit'. some analysis of the importance and extent of innovation in business is reported. for example, some 80% of us companies say that innovation is very important to their business (though no evidence is offered of the situation in other countries). only 4% of these companies say that they are good at innovation. sales, profits, and market share were all found to be improved in innovative companies. the advice which follows is an attempt to encourage forms of thinking which may lead to innovation. it often does this by phrases such as 'you are creative'; 'you are an innovator'; 'take this thought into yourself. albert einstein is quoted several times. there are also many short examples of ways in which new thinking has led to product innovation. there are maps and diagrams, some of which are superfluous. there is helpful advice on how to run meetings in order that unconventional or even silly suggestions can be considered without participants feeling uncomfortable or threatened. a structure for creative problemsolving is then proposed. part 2 purports to demonstrate a map of the territory of innovation. however, it differs from part 1 only in some details. one of these lies in considering the participants in a meeting as a cast of jungian characters, though it is doubtful whether the descriptions accurately mirror jung's theory of psychological types. the level of subtlety introduced clouds any practical benefits: i tried viewing a (real) problem from several of these angles and found the distinctions impossible to apply. part 3 is entitled 'the author is you', which means that after being told that creativity is like sex, there follows 30 pages of simple questions with blank boxes for the answers. in attempting to teach creative thinking, this book has several flaws. it is a slim volume with many pages that convey very little information. for those who require structure, it does not offer a coherent and systematic methodology; indeed it confuses by a plethora of simplistic examples backed up by gimmicky phrases. many are entertaining, some are useful, but taken as a whole there is little of substance. it is not 'not a book'. it is simply a book which is not academic: it appears to be written for people in a hurry, and who are not prepared to concentrate on gaining a deeper understanding of the subject. it has surface but no depth. perhaps people studying for an mba are like that, but i doubt it. the authors have a background in training business personnel using these techniques. perhaps this is the book's main deficit: in direct face-to-face training, trainees may be led to gain an understanding through several differing exercises of a more ideational and playful form of thinking which may lead to innovation. in the hands of a skilled trainer, these disparate threads can be pulled together. in this book the threads are left hanging. it is strange that though the text often uses examples based on manufacturing industry, it neglects the use of those who are skilled (and talented) in creative thinking, for example industrial designers. now that really would be innovative. david darling, edge ltd. hartlepool julie cotton, the theory of assessment: an introduction, london, kogan page, 1995. isbn: 0-7494-1709-9.160 pages, paperback, £14-95. towards the end of this book, the author julie cotton advises assessors to be 'positive first'; and like much of the practical advice contained in the theory of assessment, i propose to take it. the book provides much valuable practical advice on how to construct effective and suitable modes of assessment; but as a discussion of the theory of assessment, it is frankly disappointing. to some degree this criticism arises directly from one of the book's greatest strengths. the author describes the book as 'an introductory book with suggestions for activities to help you apply the theory to your work'; and this is exactly what it is. the difficulty, if it is one, is that it is stronger on application than theory. for example, in the introduction the author writes: 'assessment is not a bolt-on extra to teaching and training but an integral part of the planning, preparation and delivery. indeed, good assessment is a consolidating tool within the learning process'. one can only concur; but this remains an assertion rather than a developed theoretical position. nonetheless, it is one of the fundamental assumptions of the book, and much of what is valuable in the book arises from it. the book is divided into 10 chapters each of which covers some issue or issues that arise 82 an-j volume 4 number 2 from considering the role of assessment in education and training. each chapter carefully sets out several 'concepts', the development of which is punctuated by short sections that invite readers to 'stop and reflect', or by some 'activity' designed to make readers evaluate their own practice as assessors. thus at one point in chapter 2, 'traditional methods', the reader is asked to reflect on the question: 'do you think that regular assessment encouraged or discouraged your progress during formal study?'; and later to perform the following activity: "the next time you set an "end" test paper, write it in the form of a structured paper with a clear marking scheme and write a student feedback form. check to make sure that the learners get better feedback from this experiment.' pursuing these strategies as a reader may not exactly lead one to theorize one's modes of assessment, but it can certainly provide a fruitful way of reflecting upon them. chapters 1 to 6 consider a range of testing strategies and methodologies, including various forms of objective, standardized, and psychometric tests. in general, the discussions are most fruitful when dealing with modes of assessment that do not invite open-ended answers. there is comparatively little consideration given to the theoretical or indeed practical issues that arise in assessing, for example, open-ended essay-type questions. of course, much of the general discussion of such issues as 'reliability and validity' (chapter 6) applies equally to open-ended examination situations and more closed objective testing. furthermore, since many readers of alt-j are likely to be involved in some form of computeraided learning where open-ended assessment is notoriously difficult anyway, this concentration on more closed forms of examination and testing is by no means a serious fault. for example, chapter 5, 'statistics and records', provides an eminently readable and helpful introduction to the intelligent use of statistical analysis within an assessment situation. certainly, there is much there that could be used by cal/cbt developers to help them devise intelligible modes of representing data gathered about student performance in computer-based learning environments. similarly, much of cotton's practical advice about how to devise various forms of tests applies equally to computer-based and paper-based modes of assessment. however, given the current interest in computer-based assessment, viz. the recent edition of active learning devoted to the subject, it is somewhat disappointing that cotton only gives it a single page of consideration. the book has a strong bias towards 'competency-based learning'. the subject is introduced in chapter 3, and much of what follows seems to accept that this is the most appropriate form of assessment. again one is inclined to concur, but the arguments in favour might have been more strongly developed. for example, in chapters 7 and 8 which deal with external and internal verification respectively, the author accepts the nvq approach as an example of good practice. at the very least, this is a contentious claim, but it is not debated as such. no doubt this is partly because of the author's considerable expertise in the field of nvqs, but it also appears to be partly a function of her acceptance of the competencebased model. nonetheless, for those readers involved in developing or teaching within the nvq framework, these two chapters should prove invaluable. indeed, for anyone interested in nvq, the whole book is a valuable introduction. the final two chapters, 'trends in assessment' and 'self-evaluation and appraisal' are the least interesting in the book, and seem, after the detailed analysis of nvq, to come as something of an afterthought. even the proposed 'activity' with which the book closes falls somewhat flat even though one could not argue with the appositeness of ending this book with the words: 'write your own appraisal form as selfevaluation. what better to finish this book than to fill in this form for yourself to start your own system of self-evaluation!' one final criticism that is in no way the responsibility of the author. the book did not survive the process of review. when i was just . over half way through, pages began to fall out. this is not what one expects from a publisher as respectable as kogan page or from a book priced at £14.95. bruce douglas ingraham, university ofteesside 83 co-operative cross-platform courseware development r. d. harding, s. w. lay and h. moule department of applied mathematics and theoretical physics, university of cambridge the ukmcc (uk mathematics courseware consortium) is a consortium funded under tltp (training and learning technology programme) to produce courseware for service mathematics teaching, using the sefi (société européenne pour la formation des ingénieurs) syllabus. there are agreed courseware design guidelines and a simple courseware management system which allows cross-referencing. courseware is divided into modules, with an author as implementer for each. on any one hardware platform, a variety of authoring languages is possible. across hardware platforms, the design guidelines ensure that conversion is possible, and will preserve look and feel. we argue here that these arrangements provide a basis for continued co-operation between authors and future development as the technology changes.. introduction of all the tltp projects, the ukmcc (uk mathematics courseware consortium) has involved the largest number of institutions in authoring courseware. this was part of a deliberate attempt to set up an 'open courseware' system which can continue to be adapted and grow even after the end of tltp funding. in turn, such an objective requires a cross-platform approach to courseware development. even when considering the academic community across a single subject at just one time, there will be a variety of hardware and software that people will want to use, but a ghost of a case can be made out for attempting some sort of standardization. when change over time is considered, it is inconceivable that hardware and software platforms will remain unchanging, and one way to manage changing standards is to have a courseware structure that permits variety. this allows different parts of a collection of materials to evolve at different speeds. the ukmcc is writing mathematics courseware, but all the issues raised above will arise in many other disciplines. this paper will also deal briefly with three other issues, which although general in themselves may require different solutions in different disciplines: • who should write courseware, academics or programmers? 22 alt-j volume 4 number / • how should courseware be used? • what is the future of courseware? the ukmcc even before the tltp was announced, the learning-technology community in university mathematics was considering ways to secure wide involvement in courseware writing. this was a result of a large collaboration under the older cti (computers in teaching initiative) (harding, 1993), in turn leading to the setting up of the cti centre for maths and stats (cticms) at birmingham. two pilot projects (maths 2 cd-rom, 1989-1991; harding, 1994) had gathered sufficient experience and demonstration materials to attract recruits for the ukmcc. over 30 authors (see acknowledgements) from 26 institutions joined together to form the ukmcc, and an early account of the ukmcc is given in beilby (1993). phase 1 of the ukmcc effectively spanned two and a half years from autumn 1992 to spring 1995. during this period, a basic authoring style was established and 30 modules were commissioned from authors, to cover the basic topics in the sefi (socie'te' europe'enne pour la formation des ingenieurs) syllabus. the sefi syllabus is primarily meant for engineering mathematics, but it is a comprehensive international standard and has met with general approval amongst members of the ukmcc as a suitable syllabus for service course mathematics in the sciences. phase 2 is nominally from april 1995 to the end of march 1996, during which time a further 20 modules are planned, although because of the way that authors plan their time, these will not all be complete until september 1996. a module covers approximately the equivalent of four to five hours of conventionally lectured material. who writes courseware? in planning a project to deliver such a large amount of material, there are two poles to the strategies that might be used. one is to use academics as advisers or specifiers of materials and have the implementation done by technical staff. the other pole is to have the academics as implementers. the ukmcc opted for a policy that is closer to the latter pole (authors as implementers) although resource centres specializing in each of the main platforms have been set up to provide technical support to authors. the ukmcc s main reasons for leaning towards the authors-as-implementers policy were partly to harness the enthusiasm that already existed amongst some academics for courseware authoring, and partly to help increase the level of experience in the academic community in a way that would be retained for the academic world at the end of the project. an additional reason is an assumption that academics are more likely to use courseware when they have contributed to it themselves, and know that they could change or adapt it themselves as and when they want. mathwise and the courseware manager one consequence of the authors-as-implementers policy is that any project adopting it must either be prepared to take a draconian line on which authoring platform to use (in regard to both hardware and software), or be prepared to accommodate the usual range of prejudices and 23 r d. harding s. w. lay and h. moule co-operative cross-platform courseware development preferences amongst academics in computing matters. to involve as much of the subject community as possible, the ukmcc wanted to accommodate almost any preference, but there were two provisos. the first was that there had not to be so much diversity that student users were confused (by different interface conventions for example), nor had this to hinder courseware written by one author being referenced from materials written by others. the second proviso was that in a project with finite resources it was not feasible to offer support for an unlimited variety of platforms. the consensus that was reached within the ukmcc was to use pc and macintosh hardware, and toolbook, hypercard and authorware as software for authoring. software was thought to be of greater long-term significance than hardware choice, as industry is developing hardware which will run any major operating system suitable for personal systems, and we are also seeing more packages flike authorware) which work across platforms. to contain the natural diversity of authoring styles to an acceptable level, and to create a look and feel to be the hallmark of ukmcc modules, a document known as the courseware design specification was produced. this was based on the experience of the renaissance (harding et al, 1995) and nuffield (harding, 1994) projects, developed in discussions with all ukmcc partners. plans were drawn up for a supporting environment within which modules would run. the whole system of environment plus modules was given the name mathwise. figure 1 shows a typical screen from a mathwise module: solutions of linear equations. cramer's rule cramer's rule provides a systematic procedure for the solution of systems of linear equations. the solution method makes it very easy to count the number of solutions. this is discussed in the next leaflet test here is a matrix equation the corresponding planes are described by one of the options in the table belov. click the correct one. all 3 planes cross at & point ah3 planes coincide. 2 planes that coincide cross & fluid plane. an 3planes tore a. of intersection. 2 planes coincide: the other is parallel. the 3 planes are parallel one plane cuts 2 parallel planes. all 3 planes are distinct and the lines of intersection are paiadet fill in the determinants of a, a!, a 2 and a s . to save you time, click on each of the vhite boxes and a determinant calculator vill appear. det(a) = det(aih ] dat(aj)-[ ] nev question mark question page 1 of 1 det(a3 > i 1 figure i: cramer's rule 24 an-} volume 4 number i figure 2: courseware hierarchy mathwise courseware database leaflet 1 leaflets leaflet 1 leaflet 2 the ukmcc's original plans were to restrict authors to toolbook, hypercard or authorware, and use the built-in facilities of these systems to cross-link between documents of different type. however, it became clear at an early stage that none of these built-in facilities would allow cross-linking to be done in a fully satisfactory manner. this led us to design a separate piece of software which can link together documents written under any application (on the macintosh for example, such documents could be hypercard stacks, authorware packages, microsoft word files, maple or excel documents, etc.). this software we call the courseware manager; distinct versions are needed for pc and the macintosh, but it has essentially the same functions on each. a fuller description has been published elsewhere (harding et al, 1995). to be usable under the courseware manager, a piece of courseware must conform to the simple structural rules contained in the courseware design specification, as illustrated in figure 2. there is now a tltp-supported working group, epoc (enabling the provision of open courseware), with one of us (lay) as a member (url: http://www.amtp.cam.ac. uk/icrd/epoc/). once a generally agreed open standard exists, mathwise will want to conform. such a standard (mathwise's own, then a fully open standard in the future) underpins the achievement of many of the ukmcc's objectives. although for the present, ukmcc authors may prefer to stick with our major choices of authoring packages because help and advice is available for them, there is no technical reason why materials authored using other systems should not be included. in phase 2, one author will be using supercard, and in the future authors can migrate to newer and better systems as they become available. older courseware will remain compatible (although of course there is no magic way that it can improve itself!). user sites are able to customize mathwise by removing, adding or replacing any courseware component that is visible to the operating system as a single file or collection of files. this r d. harding s. w. lay and h. moule co-operative cross-platform courseware development process may, of course, require some authoring work by the user site, and the use of the courseware manager's administrative facilities. learning style mathwise can be used in any of the following ways: in place of lectures, as partial replacement for lectures, as an addition to lectures, or as a library resource. institutions will need to decide how to manage the use of these materials, and incorporate advice and directions for use into their standard procedures. students' access to the materials also needs consideration. there could be central or distributed labs, with scheduled hours, bookable individual seats, or just free access. there could be network access via students' own machines in their halls of residence. our courseware design and the courseware manager itself are intended to make it easy for lecturers to configure mathwise modules, and any other courseware materials required that conform to the courseware standard. the future of courseware the basic character of any body of courseware, its ease of use, modularity, availability across campus, adaptability to local conditions, etc., is inextricably linked to the style in which courseware will be used. unfortunately, it seems to us that the tltp has not addressed the issues adequately: not only should the demand for courseware be addressed (as discussed further below) but, in addition, career issues such as incentives and support for staff to introduce new techniques into their teaching need to be considered. improved effectiveness of learning has been a major objective of educational computing since cal first became a technical possibility. however, there is very little evidence from experience or in the literature that computers can replace human teachers in higher education, and the evidence points to computing costs being add-on, and the benefits being enhancement of education rather than replacement of older methods. hooper's final report (1977) on the ndpcal nearly 20 years ago made this clear, and although since then the technology has made huge advances, the basic modes of use of computers in teaching and their likely effectiveness has remained much as summarized by hooper. it is disheartening that either the lessons of the ndpcal were not known to tltp policy makers, or that no public debate was conducted to explain why it was thought that dramatic changes were imminent. supporting the authoring of courseware is only half of the equation, the supply side. consortia like the ukmcc have no control over the adoption and use of their products. it is necessary to make an informed guess at all the ways institutions will use courseware and try to write appropriate materials, but no project can guarantee that materials will be adopted. there needs to be a demand-side policy as well, and it is here that the signals from hefce are most confusing. assurances have been given that successful efforts in courseware development will be recognized in research and teaching assessment exercises, but there is scant evidence that lecturers feel there has been any change in the link between their efforts in this direction and career development. perhaps alt has a vital role to play here? 26 volume 4 number i acknowledgements david asche, neil challis, jean cook, douglas curran, maureen foster, martin greenhow, david griffel, jon greenman, david harding, pat hawkshaw, graham hendry, stephen hibberd, david hodgkinson, john kingston, ron knott, eric lawrence, duncan lawson, michael mccabe, george mcguire, diana mackie, walter middleton, john o'connor, enid pinch, neil pitcher, david poutney, douglas quinney, peter rangecroft, edmund robertson, douglas siviter, peter soan, jim tabor, alan thomas, geoff wood and mike yates. references beilby, m. (1993), 'mathwise: a learning environment (part 1)', maths & stats, 2 (5) (november). harding, r.d. (1993), 'computer interactive texts' in proceedings of 4th annual conference on technology in collegiate mathematics (reading, ma, addison-wesley), 132-8. harding, r.d. (1994), 'nuffield interactive mathematics courseware', computers and education, 22 (1/2), 73-82. harding. r.d., lay s.w., moule, h. and quinney, d.a. (1995), 'multimedia interactive mathematics courseware', computers and education, 24 (1), 1-23. harding, r.d., lay, s.w. and quinney, d.a. (1995), 'a consortium approach to courseware design in mathematics', computers and education (proceedings of cal 95) (in press). hooper, r. (1977), national development programme in computer assisted learning. final report of the director, london, council for education technology. maths 2 cd-rom (1989-1991), published by ultralab, anglia polytechnic university, produced under the renaissance project, sponsored by apple computer uk. 27 a computer-aided continuous assessment system b. c. h. turton school of engineering university of wales cardiff a high-quality assessment system should have the following attributes: rapid feedback to the students, appropriate and detailed feedback, and an effective grading system which provides an accurate overall grade as well as information which identifies the student's weak areas. as stafflstudent ratios worsen, providing such a system will become more difficult and consequently computer assistance in this task is becoming more attractive. this paper describes a computer-aided assessment (caa) system based on a modified version of the multiple-choice questionnaire. the caa has been designed to be used in continuous assessment, with features that discourage plagiarism and provide appropriate feedback over a hundred students were tested using this caa and the results were compared with a more traditional assessment system. in addition, questionnaires were used to assess the student's reaction to the caa. the results were highly satisfactory, and a more advanced version of the original software is under consideration. introduction universities within the united kingdom have had to cope with a massive expansion in undergraduate student numbers over the last five years (committee of scottish university principals, 1993; cvcp briefing note, 1994). in addition, there has been a move towards modularization and a closer monitoring of a student's progress throughout the year. since the price/performance ratio of computer systems has continued to improve, computerassisted learning (cal) has become an attractive option. (fry, 1990; benford et al, 1994; laurillard et al, 1994). to this end, the universities funding council (ufq has funded the teaching and learning technology programme (tltp). however universities also have a duty to assess as well as to teach. this paper describes a computer-aided assessment (caa) system capable of assisting in grading students and providing feedback. in this particular case, a continuously assessed course (low-level languages) of over 100 students is considered. typically, three man-days are required to mark one assessed piece of coursework from the students in this class. any feedback on how the questions were dealt with by the student are of necessity brief. most of the feedback is provided in a tutorial session that covers the pitfalls encountered by the majority of the students. 48 alt-j volume 4 number 2 a caa solution was sought which covered the following points: • rapid feedback; • appropriate feedback; • breakdown of the strengths and weaknesses of the student; • breakdown of the strengths and weaknesses of the class • a system that makes plagiarism difficult; • effective assessment of the student; • reasonable resource implications. rapid and appropriate feedback is clearly necessary for a course lasting only 12 weeks. any assessment method will obviously be more useful for teaching purposes if it can form a profile of the student and the class. plagiarism must be guarded against since continuous assessment precludes examination conditions. all these conditions must be met within the resources available. the computer-aided assessment system in order to facilitate rapid marking, a fixed set of solutions was deemed to be the only cheap and effective means. traditionally, this has been thought to be a rather limited system as feedback can be poor. in order to combat this problem, each available choice within a question was designed to capture a particular misunderstanding of the subject. this discipline was highly informative as it requires the examiner to consider the common errors made by students. by storing information on common mistakes, a detailed explanation can be given to the student which deals with the specific area of misunderstanding that led him or her to choose a particular, inappropriate answer. an explanation of the correct answer is also given to ensure that those who have guessed the answer can still benefit from the correct explanation. nam« . paper no. sfiih«ntno.. student / sits student exam paper class paper no. contains ^ oue.stion no. 1 / paper questions *" are on answer textveracity question category j isoftyj question / "category no. .category text je [ouestion no. .question text •.category no. has the answer answer answpr nn ..question no figure /: entity relationship diagram for caa t explanation 49 8. c h. turton a computer-aided continuous assessment system each question must also be classified so that a summary can be made of the student, and classes, for different categories. such classification enables the computer to create summaries for students and classes. figure 1 shows an entity/relationship diagram for the situation as described so far. the key disadvantage of this system is its susceptibility to plagiarism. unlike the case with traditional assessments, plagiarism would be impossible to detect between students taking the same paper. an alternative would be to set different questions for each student, but it would be impractical to design a paper with 10 different questions for over 100 students. one solution would be a bank of multiple-choice questions from which 10 questions are selected for each student; however, this bank would have to be very large, of the order of 100 questions, to ensure that on average any two papers would have no more than one question in common. in addition it would be difficult to ensure that a large range of questions maintained roughly the same difficulty and subject coverage. the solution adopted was to have a small bank of questions from which 10 were selected for each student. each question has a number of different ways of being asked, a number of correct answers, and a number of false answers. the question asked of the students is one instance of the text for that question with one true answer, a number of randomly selected false answers from the set of false answers to that question, and a 'don't know' answer. this combines the advantages of having a small set of questions yet ensures a variety of combinations such that plagiarism is discouraged. for example, a simple question designed to check a student's ability to perform base conversion could be stored as follows: choice question version 1: which value is equivalent to the number 1125. question version 2: am 0123 = 0, select zfrom the choices below. answer (true) 1000002 answer (true) 447 answer (false) 11210 or 101210 (student failed to perform conversion, guesswork) answer (false) 3323 (student failed to notice digit range 0-2 for base 3) answer (false) 25g (student transposed the digits) question there are twelve combinations for a simple three-option multiple-choice questionnaire; one possibility is: x-10123 = 0, select zfrom the choices below. (a)256 (b)447 (c)3323 50 alt-j volume 4 number 2 assume q questions are required for m students, with c choices per question not including 'don't know'. a formula will be given for each case followed by a numeric result assuming 100 students, 10 questions and four choices. scenario 1: entirely different questions for each student probability of two papers not duplicating any question number of questions to be prepared = q.m number of answers to be prepared = q.m.c = 1.00 example = 1000 questions example = 10000 answers scenario 2: a bank of questions {b questions in the bank, assume 6=100 for the example) probability of two papers sharing s questions, = results for the example numbers. number of questions to be prepared = b number of answers to be prepared = b.c i bcq. figure 2 shows the example =100 questions example = 400 answers (4 per question) •a i 0 1 2 3 4 5 6 7 8 9 1 0 no. of shared questions figure 2: probability that two papers will share zero or more questions scenario 3: in this scenario, a distinction will be drawn between the whole question (questions and choices) and the text of the question itself (not including choices). the whole question will be referred to as the 'question', and the text of the question itself will be referred to as the 'question-text'. in this scenario, there is a bank of b questions with n versions of the question-text for each question. there are t true and / false versions of answers to that question from which c choices are given to the student, only one of which is true. a variety of question-texts and answers can be used to create a single question with c choices of answer. clearly some combinations of answers and question-texts will be so similar that the questions are for practical purposes identical, while other combinations will be very different. in order to provide a fair comparison with the other techniques, questions which are very similar should be considered to be the same question. two methods of grouping questions together as being essentially identical can be defined: 51 8. c h. turton a computer-aided continuous assessment system option (i): if two questions share the same 'true answer' choice and the same questiontext, they will be deemed to be essentially the same question. option (ii): if two questions share the same question-text and choices, in any order, they will be deemed to be essentially the same question. the first option assumes that a student is cheating by acquiring someone else's 'perfect' paper in addition to his or her own, and so can copy where the question-text and the true answer appear on both papers. the second option is strictly correct in that the questions are different if the choices or question-text are different. the author of this paper prefers option (i) as a measure of similarity, as it more closely reflects the possibility of plagiarizing another student's work. in the equation that follows, the number of distinct versions of question is denoted by the symbol v. this number can be calculated as follows: for option (i) v=t.n; for option (ii) v=t . (^i)c/. n (see figure 3). the value of v will be substituted in the probability formula given later. i •8 -option i) option ii) •40 1 2 3 4 5 6 7 8 9 1 0 no. of shared questions figure 3: probability of shared questions {b=25, q=io, v=4, option i) v=4, option ii) v=80} the formula for calculating the probability of two papers having i questions in common, of which s are the same variant and consequently can be easily copied, requires some explanation. figure 4 represents the simple case where there is only one version of each question. in this case^ the formula is simply g ^ c ^ . qcs)/hcr the reader must remember that there is a bank of b questions of which q are on the paper which is being used for plagiarism purposes. so b-qcq-s represents the number of ways of choosing (q-s) non-duplicated questions from the possible (b-q) non-duplicated questions in the question bank. qcs represents the number of ways of choosing j-duplicate questions from the set of q questions which could be duplicated. so the total number of ways of producing a paper which has exactly s duplicated questions is (j^qcq_s. ?cv). since the total number of ways 52 alt-j volume 4 number 2 bank of b questions 1 1 1 , q questions on the first paper which the . 1 1 ; s same questions i a student is attempting to copy from paper 1 1 ' ! * | j paper 2 q questions on the paper to be completed figure 4: pictorial representation of questions chosen from a bank of producing q questions from b is jc9, the probability of doing so by chance must be (j>-qcq-s • qcs)fbcqin scenario 3 this is complicated by the fact that even if a question is duplicated, it may be a different variant to the one chosen on the original paper. consequently, it will be assumed that of the q questions, i are the same question and s of them are the same variant of the same question. bank of b questions i » 1 m . i .•* q questions on the first paper which the student is attempting to copy from \ l 1 i shared questions ^-shared questions same version paper 1 • ; | v ] paper 2 q questions on the paper to be completed figure 5: pictorial representation of questions chosen from a bank with different versions of question the formula must now be split into three sections: the number of ways of choosing (q-i) non-duplicated questions from (b-q) questions, multiplied by the number of ways of selecting (is) duplicate questions but different variants from q questions, times the probability of that event, multiplied by the number of ways of selecting s questions from the remaining (q-(i-s)) questions, times the probability of that event. now the equation must be summed for all possible values of /, and divided by the total number of ways of choosing q questions from b. this gives a probability of any two papers sharing s 'identical' questions of: 6. c h. turton a computer-aided continuous assessment system assume for this example that there are two versions («=2) of the question-text for each question, two true answers (f=2), and six false ones (f=6) from which four options (c=4) are chosen. finally, let the bank have 25 questions (6=25). a particular question may only appear once on a particular paper, irrespective of the way it is expressed. number of questions to be prepared = b.q example = 50 questions (25 questions each with two different question-texts) number of answers to be prepared = b.(t+f) example = 200 answers (eight for any one question, six false, two true) in order to ensure that the frequency of a particular choice being on a question is the same for both true and false answers, t <=f/ (c-1) which is true for the example given. once this bank of questions and answers has been created, there is no requirement to create a new bank for the next year, as comparison of papers between students from different years will not easily yield the correct answers. this method of producing question papers from a bank of questions can be organized as depicted in the entity relationship (e/r) diagram shown in figure 6. the key differences between the previous e/r diagram and this one is the different versions of texts for questions, multiple versions of answers, and an individual paper for each student. r^ateg orv no. category texlanswer text. explanation— answer question no. paper question no» t paner no. ident choicejfo "^ sou student examined by student answers \ ientnc which picks answer no _veracity s contained in \ panri-nn paper/answer options f \ is answered by par«rno has the choice: paper paper/question >papername is written as / question category categorises ^ 1 \ has the variants "^ question text ppper question no. ^question choice no. ^answer no. paper no. .question no. paper question no. \juestion version no. question no onestion version no. question text 7 v category no. qimstion no. figure 6: entity relationship diagram for individual multiple-choice question papers implementation the software for this project was written in borland c++ by p. morgan (uwcc) using a modified version of the earlier e/r diagram which could be implemented more efficiently. each entity was stored as a separate file. the modified e/r diagram is depicted in figure 7 using the same notation as found in figure 6. a breakdown of the menu structure can be found in figure 8. 54 alt-] volume 4 number 2 answer textexplanation question no. answer paper question no., student no_ student choice no." student answers n a m e _ j class 1 which picks _answer_njj |__veracity is contained in paper/answer options produces student no. has the choices students sits student no. _pat«tquestion no. kouestion choice no. "answer no. .student no. ^taper/question is written as category no. 1 category textj question category categorises [.question no. .paper question no. .ouestionjjo, question text [xategory no. .question text ition version figure 7: modified version of an individual multiple-choice question paper figure 8: menu structure for an implementation 55 8. c h. turton a computer-aided continuous assessment system in order to allow a high degree of flexibility in the system, the mark scheme can be selected from a number of alternatives. for example: 1 correct answer 1 correct answer 0 false answer 0 don't know —l/(c—1) false answer 0 don't know where c is the number of choices for a multiple-choice answer. • 1 correct answer 1 false answer 0 don't know in this paper the results are based on the second option, which is a negative marking scheme. results one hundred and thirteen students were tested using ten questions from a bank of fifteen questions. each question had the following choices: one question-text from two choices, one true answer from two choices, five false answers from seven choices and a don't know option. questions were allocated in a random order on the paper, and choices allocated randomly within a question. the probability of a number of questions (s) being duplicated when comparing two individual papers agreed with the analytic result given earlier (see figure 9). the two options given in this figure relate to two separate ways of defining 'different' papers. if the definition for option (i), as described earlier in this paper, is used, then the probability of finding less than or equal to three shared questions when comparing two papers is over 90%. alternatively, the definition in option (ii) would describe the chance of finding one or fewer shared questions between two papers as over 90%. both definitions are reasonable, therefore both results are given. an example of the type of question produced for the student's questionnaire is given in figure 10. l • 0.9 -r 0.8 --" 0.7 0.6 -j'. 0.5 0.4 0.3 0.2 0.1 0 -option i) option ii) i 3 4 5 6 7 n o . of shared questions 10 figure 9: probability of shared questions {b=l5, q=io, option i) v=tn=4, option ii) v=t(c-l)cfn=84] 56 au-j volume 4 number 2 students name : mr xxxx xxxxx xxxx hs10 class name : lll93 ) marking scheme : [right = (+1.0) ] [wrong = (-0.2)][dont know = (0.0] [question d ] given the following machine code and the table below, state the number placed by the program at location $42, given that memory location (0041) program:(7f 00 40)(de 40)(a6 50)(97 42)(3f) 3 hex table: memory address (hex) 0050 0051 0052 0053 0054 0055 0056 0057 entry (hex) | entry (decimal) 00 01 04 09 10 19 24 31 i 0 1 4 9 16 25 36 49 answer answe: answe: answe: answe: answe: (1) (2) (3) (4) (5) (6) output from program = (10) hex output from program = (31) hex output from the program is impossible to determine as it is outside the range of the table. output from program = (36) decimal output from program = (34) decimal output from program = 1001 binary. answer (7) (dont know) figure / 0: example question from the multiple-choice questionnaire six categories of question were available from which to generate the report. these categories were used to identify 'weak' areas for tutorial work. finally, the results were correlated with those previously obtained from a traditional examination, and they produced a correlation coefficient of 0.923 for 111 degrees of freedom and two variables. this indicates a high degree of positive correlation between the results. feedback was also obtained from the students. they were not told how the questionnaires were generated, so the plagiarism question refers to their impression from their papers, not their understanding of the technique. an example of the questionnaire is given in figure 11, with shaded boxes to indicate the percentage of students choosing the option indicated. figure 12 shows the percentage results graphically. in addition to the results shown in the figures, the following comments, derived from the comments section of the questionnaire, are worth highlighting. • students found that multiple-choice questions specifically designed with options that tested their ability to understand subtle differences and exploited common student misunderstandings were very difficult to answer. one commented that this forced them to go to the textbooks. 57 b. c h. turton a computer-aided continuous assessment system multiple choice questionnaire (14/1/94) the following questionnaire is designed to elicit your reaction to using multiple choice papers with mixed questions and computer generated explanations. please tick one box and place any relevant comments at the bottom of the page. did you find the multiple choice paper did you prefer the mcq paper to traditional methods in your opinion would plagiarism be did you think the feedback was very difficult reasonable easy very difficult easy much preferred very difficult preferred made little disliked very difference much disliked difficult moderate easy very easy much better better than average poor very poor than normal normal did you think that the speed of response (feedback time) was excellent good reasonable poor very poor i i do you think as a teaching tool this form of mcq was excellent good reasonable poor very poor j do you think as an assessment tool this form of mcq was excellent good reasonable poor very poor comments key indicates percentage of students ticking the option shown (50% uptake of questionnaire) j 10% 1 5% | 0% figure / / : student questionnaire 58 alt-j volume 4 number 2 difficulty preference plagarism feedback response teaching assessment question choices figure / 2: bar chart showing the percentage results for the questionnaire • the assessment was criticized for using negative marking and not being able to give part-marks for working out. this indicates that the students' opinion on the multiplechoice questionnaire as an assessment tool may partly be based on their opinion of negative marking. • the computer-generated answers only referred to one version of the question text. some students found this confusing when trying to understand the feedback papers. conclusion according to the questionnaire results, the class seemed to prefer this form of assessment despite finding it difficult. plagiarism did not seem to be a problem, and the response time was good. however, they would have preferred a system that did not use negative marking, and would have liked clearer explanation sheets. of particular interest is the students' opinion of this caa as a teaching tool. students believed this caa system was a better teaching tool than assessment tool, according to the questionnaire. this statistic is backed up by comments on how the caa paper forced them to study their books and which also indicated their dislike of an assessment method that used negative marking. clearly, this particular study achieved its goals and provides an effective means of assessing students and providing effective feedback. however, three man-days were spent on developing the questions by a lecturer experienced in setting appropriate questions for this subject. so no saving of time can be realized unless very large classes (>100) are assessed or the questions can be re-used. due to the nature of this system, such questions could be re-used provided a student did not collect multiple explanation sheets from a previous year. arguably, a student who reads through explanations for many different forms of questions and answers is likely to spend more time learning than the one who studies his or her course notes. naturally, the explanations could be withheld, thus treating the students from several years as one class for the purposes of the results obtained. but this defeats the feedback aspects of the system. certainly, this system does 59 b. c h. turton a computer-aided continuous assessment system not represent a perfect solution, as considerable effort has to be put into the 'false' answers in order properly to test and inform the students. nor can this be used as.a system to replace traditional assessment, as traditional assessment and tutorials provide the means for identifying common misconceptions. nonetheless it is a powerful new tool in the hands of a lecturer. the next stage in developing this system will be to provide a networked version to remove unnecessary paperwork and to rewrite the system so that it becomes part of a standard database. these developments should allow greater flexibility and greater ease of use. references committee of scottish university principals (1993), teaching and learning in an expanding higher education system: executive summary', the ctiss file, 15, 5-6. cvcp briefing note (1994), funding of undergraduates and university teaching, london, cvcp publications. benford, s., burke, e., foxley, e., gutteridge, n. and mohd zin, a. (1994), 'early experiences of computer-aided assessment when teaching computer programming', association for learning technology journal, 1 (2), 55-70. fry, j. (1990), 'the database format question: an alternative to multiple choice and free format for computer-based testing', computers and education, 14 (5), 391-401. laurillard, d., swift, b. and darby, j. (1994), 'academics' use of courseware materials: a survey', association for learning technology journal, 1 (1), 4-15. 60 alt-j, research in learning technology vol. 15, no. 1, march 2007, pp. 1–2 issn 0968-7769 (print)/issn 1741-1629 (online)/07/010001–02 © 2007 association for learning technology doi: 10.1080/09687760601129489 editorial making connections: reflection, feedback and support taylor and francis ltdcalt_a_212880.sgm10.1080/09687760601129489alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis151000000march 2007janesealej.k.seale@soton.ac.uk hansell (1974) conceptualized seven different connections or attachments that we have to the world in which we live: attachments that consist of connections to other people, to a social role as well as to a system of meaning. for many, learning is at the heart of the process of making connections, particularly meaningful connections (horton, 2000). if this is true, then it leads us to consider the extent to which educators, the systems they work in and the tools they use have a role to play in helping learners make those meaningful connections. for example, scherer (2002, p. 5) suggests that educational systems have a responsibility to: […] connect individuals to other people, groups and social roles; provide the knowledge necessary to be a productive member of society; and help individuals to achieve a sense of identity, purpose and meaning. defining and understanding how educators do this in a technology-rich society is a challenge that some learning technology researchers and practitioners are addressing. in this issue of alt-j we have six papers that focus in some way on the supportive or facilitative role that technology and teachers can play in helping learners to make connections between: ● prior knowledge and understanding, and new conceptions and experiences (masikunas et al.; handley and cox; moule). ● old and new practices or materials (vallence; bates et al.). ● learning and teaching activities and learning outcomes (ellis et al.). three key themes emerge from these six articles: feedback, reflection and support. masikunas et al. explore the use of electronic voting systems in lectures and note that, for some students, the feedback and reinforcement provided by electronic voting systems helps students to re-think their understanding and challenge the meaning they had construed and derived. as part of their examination of learners’ perceptions of model answers in online feedback, handley and cox conclude that students need to be supported in their efforts to build on existing knowledge and experiences and to reconsider what they perceive to be the relevant conceptual and practical issues. moule offers an alternative to salmon’s five-stage model of learning and teaching online that embraces both instructivist and constructivist approaches to learning. the model is based on the metaphor of a ladder, where the higher rungs of the ladder 2 editorial represent the need to encourage and support learners to think creatively, problemsolve, update knowledge and skills, and develop critical thinking skills. vallence outlines an ‘ict enabled’ method for collecting evidence about teachers’ pedagogical beliefs regarding the use of information and communications technology (ict). the teachers in question were pre-service teachers who were supported through the use of synchronous networked tasks to reflect on their existing practices in relation to ict and consider ways of integrating ict into their practice. bates et al. explore the attitudes of academics, support staff and managers to the potential benefits of using online repositories for teaching and learning. one key benefit that was identified was that of receiving feedback from others about how their materials had been used in order to help them improve the quality of the materials. ellis et al. explore students’ experience of face-to-face and online discussion and the implications this has for supporting students in making sense of and reflecting on their learning experience. they argue that, for many learners, discussion activities can be fragmented from more meaningful learning when the learners lack an awareness of the associations between discussion and the learning outcomes of the course or task. the concept of connectivity is not a new one in the field of learning technology. it has been central to our understanding of how learning technology can promote an interaction between learners, tutors and resources (for example, jones, 2004). connectivity is beginning to be central to discussions regarding the need to connect the use of learning technologies to theories of how we learn and how we should design for learning (for example, wiley, 2001). the papers in this issue of alt-j challenge us to think about connectivity in a different way, where learners are not necessarily connected to other learners or other resources but are connected to other concepts and understandings that help them make sense of and extract meaning from their online learning experiences. jane seale university of southampton, uk references hansell, n. (1974) the person-in-distress: on the biosocial mechanisms of adaptation (new york, behavioural sciences press). horton, s. (2000) web teaching guide: a practical approach to creating course web sites (new haven, ct, yale university press). jones, c. (2004) networks and learning: communities, practices and the metaphor of network, alt-j, 12(1), 81–93. scherer, m. (2002) connecting to learn: educational and assistive technology for people with disabilities (washington, dc, american psychological association). wiley, d. (2001) connecting learning objects to instructional metaphor, and a taxonomy, in: the instructional use of learning. available online at: http://www.reusability.org/read/chapters/ wiley.doc (accessed 15 november 2006). the role of learning technology in planning change in curriculum delivery and design colin harrison anglia polytechnic university, uk abstract this paper is a discussion of some aspects of the application of learning technology to the design and delivery of flexible learning packages, which can help make the most effective use of the most valuable resource in a university its staff. principles of good course design, preparation and use are exemplified in the approach taken towards flexible learning packages at anglia polytechnic university. introduction learning technology is not chiefly concerned with using computers, video recorders, audio tapes or any other form of technology. rather, it is concerned with a systematic approach to the planning and delivery of the educational experience. in other words, it should be more concerned with the design of the whole learning experience than with any particular delivery medium. from good design it should become clear how best to deliver a quality learning experience. this paper will therefore not be about the technology but more about the process and the initiatives that we in higher education are launching to share that planning and delivery process. but since the principles of learning technology have been around for many years, why suddenly bring it back to the front burner and ask that colleagues reassess its place in strategies for teaching and learning? simply because it offers us a flexible model that will aid us not only in coping with the increase in student numbers but will also improve the learning experience of students. the pressures that have led us to look at new (or not so new) ways of designing the learning experience can, at one level, be said to stem from the major growth in higher education encouraged by the government in the 1990s. the target then set, of increasing the participation rate of 18-19 year-olds from one in five to one in three by the year 2000, was used by the universities as a pathway to a rapid growth and a way of increasing income. the year 2000 target is already in sight and we have a slow-down imposed on us. we have 30 au-j volume 2 number 1 also seen a jump in the numbers of mature students in higher education, a 70% increase in the 10 years up to 1990. these two factors more students leading to greater staff-student ratios, and an increase in the mature mix has forced all universities to look at ways of planning their educational programmes to preserve the best experience, to provide a flexible pattern of attendance for mature students and to allow staff to survive all the pressures. against this background of increases in participation in higher education, and our desire to maintain quality and widen access, particularly within the wider region that many of the new universities in the uk serve, we face the pressure of having to deliver even further efficiency gains; and all this without increased resources for the university or our students. so a rethink is called for in how we make the most effective use of the most costly resource in the university the staff. we need to adopt new approaches to teaching and learning. there are also positive, pedagogic, reasons for change. there is real evidence that more independent study, flexibility of pace and place, and the appropriate use of technology can produce better results and enhance the transferability of intellectual and personal skills that is sought by employers. it allows for greater access via franchises to regional colleges, it supports mature students, and it allows more options to run each year regardless of student numbers. all the evidence is that in facing up to these pressures for change we (as a sector) seem to be moving in three distinct stages. these have been defined by wagner (1993) as follows: • a first stage is usually more intensive use of existing methods resulting in classes getting larger, options reduced, the hours that students are taught being reduced and the hours that staff teach increased. • the second stage is change but not fundamental change. for example it is realized that seminar groups of 25 begin to challenge the fundamental purpose of seminars. so, some rethinking occurs as to how to make the most cost effective use of both the most valuable and most costly resource in higher education the time of an academic. this problem has been characterized as the division between 'mass time' and 'quality time'. in higher education the tendency has been to increase mass time at the expense of quality time. • the third stage is to face the challenge and to turn mass time into quality time and make the whole of higher education a quality experience. this involves the application of appropriate learning-technology skills to the design of the syllabus, the course materials, and the assessment procedures. it takes on board the use of the most appropriate (not necessarily the sexiest) form of delivery. it will always tempt the technocratic mind to think that technology will be able to do what people do, better than they do it already. over the past two decades technology has been pushed forward as a way of replacing/reducing the role of the teacher. but materials based without good cause on a particular technology platform fail in two ways. first, like bad teachers they are inflexible, unadaptive and tiresome; secondly, they provide no exciting stimulus to trigger the creative interest of the student. perhaps we all need to listen to a colleague of mine, professor steve hepple, who often calls for us to begin 'delighting the student' with our courses. so what is it that we should focus our creative abilities on that will make the most effective use of our time and talents, and that will 'delight' our students? i suggest that there are three 31 colin harrison the role of learning technology in planning change broad issues we must attack quickly if we are to enter "into stage 3 of wagner's list. these issues are: • the role of the teacher in relationship to the student; • course design and presentation; • assessment models. clearly, these issues_are not totally separate, but lead on from one to the other with considerable overlaps in some cases. however, it is helpful to address them as steps in a process, and to apply an educational technologist's eye to them. the role of the teacher in relationship to the student in my paper 'teaching and learning a strategy for anglia' (harrison 1993), i suggest that 'a university education is a combination of complex and interacting learning experiences including, inter alia, formal lectures, tutorials and seminars, work based learning, resource materials, libraries and computer access, student support services, accommodation and the assessment procedures'. the provision of an affordable experience for students depends on a careful analysis of requirements at discipline, course, institutional and employer levels. it must include an analysis of the potential and actual contributions of learning technology to teaching and learning in higher education. my own university is one that prides itself on the quality and expertise of its teaching and learning support staff. in many parts of the course structure, students have the opportunity for close contact with all these staff through tutorials, seminars, lectures and learning support facilities. as part of its continuing evaluation, anglia polytechnic, like all universities, is reviewing its teaching and learning methods. we believe that by allowing students more freedom to pace their study and by arranging modules with fewer rigidly fixed meeting points, they can organize their own learning more effectively. the development of credit accumulation and modular systems, a fundamental part of anglia strategy, builds into our courses flexibility for both subject selection and timescales for completion of courses. to achieve these changes, anglia has developed new approaches to learning. in doing this, the university has cognizance of the interaction between teaching, the formal learning that stems from the organized structure of the planned teaching process, and the informal learning which occurs as a result of and which is encouraged by the flexible study methods and student-group interaction that flexible learning systems engender. this approach starts with the way the university clearly states the learning outcomes of undergraduate and postgraduate study, and of the individual modules, so that students can easily see what they have to achieve. staff can use these outcomes to develop assessment methods to test the outcomes and competencies more accurately. the university can also use this foundation to produce planned flexible study materials that will allow some parts of the course to be given to-students early in their year so that they can organize their work-flow independently. some of these materials will contain advice on group activities, and most will contain seminar and tutorial meetings that can be flexibly arranged by the students. at the same time, equal attention will be given to ensuring that the best practice associated with what has been called 'traditional' teaching is developed and incorporated into the strategy. 32 au-j volume 2 number 1 the following represents the heart of the teacher's role: knowing: • the subject; • the students and their needs; • the skills of presenting and managing the materials and processes of learning; and • the best ways of assessing and providing feedback on the learning that is taking place. nothing in this suggests that teachers should never stand up in front of a class nor does it suggest that the main function of a teacher is to stand up in front of a class. rather, it says that the teacher is the best person to design the course, validate the information content, design the assessment, and work with the learners during the delivery phase. how the content is delivered must vary from module to module and course to course. course design and preparation . as part of this year's teaching and learning developments at anglia, as pro-vice chancellor (teaching and learning) i have called for volunteers to suggest modules they believe can benefit from being presented in a more flexible form. i have no preconceived ideas about that form, though i expect that the most popular medium will be print on paper. from research i have recently undertaken, i have developed an approach to the structure of packaged materials that will, i believe, aid their rapid development, and ensure that we provide our students with a helpful consistency of approach. however, i do not think that anglia, or any university, can further impose upon the goodwill of its staff by expecting them to undertake the development of flexible materials and continue to carry their normal work loads. because of this, i have recommended that monies be made available to buy out staff from normal duties to allow them safeguarded time to work with trained colleagues in the production of flexible learning materials. a new style for flexible learning packages . the planning process applying the techniques of learning technology to this process, one must first look at the content of the module and what one is trying to achieve from it. a number of decision points are discussed below that show that the package should be content-led rather than a decision being taken at the outset to base the package on a particular technology platform. the key decisions relate to the aims and learning outcomes of the course. once these are in place, the best way of delivering them can be more reasonably assessed. in using a particular delivery medium, it is important to consider the availability of the mechanism for 'playback' to the student who may be working at home or in some other remote location from the college where the package was designed. write, buy, adopt before jumping into developing new material, it is worth looking around for existing packages that can easily be adopted/adapted to meet our needs. material from the open learning foundation or the open university are examples. 33 colin harrison the role of learning technology in planning change content write or use textbook, etc. a decision point is reached. does one write the materials oneself and give readings that students can follow up? does one build the module around a single book, video, film or whatever, tell the student to buy it, then direct students to use selected passages from it? does one mostly write the material, give readings, and include in full some extracts that students might find difficult to find? copyright regulations and necessary payments to rights holders must be considered. therefore list the decisions: • set textbook, student to buy • recommended readings not included • selected reading to be included • video/audio/computer materials to be produced. it will be important to talk to the library staff about stocking the necessary titles, and immediately to discuss media support with one's educational technologist. module overview it will be necessary to prepare a brief statement that will later be used to guide the intending student in gaining an overview of the content. at this stage, it guides the later development of the module. academic level will the module be at introductory level, at a higher undergraduate level, or at a postgraduate level? the approach needs to be set from the start. aims of the module deciding on the aims of the module is a major task, and each aim should be large enough so that 10 or more learning outcomes can be derived from their content for trialling and assessment purposes. for example, learners should be able to: • explain the notion of research design and how it relates to the decision problem; • construct a questionnaire; • do some important tests for examining the statistical significance of differences; • carry out, using computer-based procedures, a statistical analysis of a reasonable data set. note that each of these aims must be able to sustain many pages of explanatory text and so forth, and provide material for assessment questions. these questions may, in the end, be selfassessed by the student, computer-assessed or tutor-assessed. from each of the aims one will need to develop the appropriate learning outcomes. each aim and its group of learning outcomes will then form a unit of "study within the module. learning outcomes next, one takes the aims one by one, and one develops the appropriate number of outcomes to cover the achievement of each of them. it is often the case that 10 or so learning outcomes may be required to cover a single aim. the validity of an outcome will be tested by quickly attempting to devise a dozen or so true or false questions which, while they may never be used in the actual learning situation, will show if it is possible to test/assess completion and understanding of the outcome. the reasons for different assessment aims is discussed below. 34 ait-j volume 2 number 1 assessment models atkins (1993) identifies several reasons for assessing students. as he puts it, they include: • establishing the level of achievement reached at the end of a course or module; • establishing progress during a course and to give feedback on it; • diagnosing strengths and weaknesses leading to remedial action; • consolidating work done so far a learning experience in itself; • motivating students; • predicting a student's likely performance level in the future; • determining whether a student is 'safe to practice'; • conforming to the requirements of external regulatory bodies; • giving individual staff feedback on the effectiveness of their teaching; • obtaining information on the effectiveness of the learning environment (and methods of presentation); • monitoring standards over time. these headings noted by atkins provide a very useful guide to the range of questions one should plan to set so that various dimensions of the assessment can be met. assessment can be built up in many ways, and to some extent our decisions will depend on how quickly we can provide a computer-based learning management system to underpin the use of flexible materials. however, three forms of assessment must first be considered. first, there will be student-marked feedback questions designed to give assurance that the learners are on the right lines. these questions will test only the outcomes of one unit. secondly, there will be more complex multiple-choice questions that should be computergenerated and marked from a custom-designed databank. initially, these may have to be marked by a tutor, or the students trusted to mark them themselves and to hand in the marked papers, or marked by a computer using a midway mark sensing card to collect answers. the questions may test integration of learning across two or more units of the module. answers can be placed as an appendix to the module or issued later (though this creates additional work). thirdly, there will be assignments that draw content from the whole module or a major part of it, and that will test integration and understanding of the content or ability to perform skills. while these could be delivered by the computer, they will be tutor-marked. an examination may replace an assignment. this summative and integrating assessment is an essential part of the learning process and should not be seen as 'just another assessment'. it will help 'make thinking and problem solving skills explicit, use holistic tasks, use occupationally relevant simulations' (thinking and learning at work 1993). the more assessments integrate learning, the better the course and the more helpful the assessment. 'learners constantly fed on a diet of "chopped up tasks" may sometimes never learn to chew or integrate all of the sub skills into one whole complex task.' (thinking and learning at work 1993). the volume of assignments must be related to the need for assuring the student and then ourselves that they have understood the content and, if appropriate, that they can perform the tasks. it should not be excessive and should be paced across the whole module. the weighting of assessments must be a matter of debate according to the subject. in designing the way assessment is treated in the packages, i confess that i lean towards a model of 'mastery' or 35 colin harrison the role of learning technology in planning change 'competence'. by this, i mean that the student achieves an acceptable level of performance when tested against a set of predetermined criteria or standards. the activity can require intellectual, personal or practical achievements. they may be tested in a variety of ways. in australia, i saw an excellent computer system that underpins the deakin university flexiblelearning operations. it can be used just to assess, deliver, monitor, record (and so on) flexible modules, or to deliver and mark assessments for any or all students in the university. known as the computer managed learning (cml) package (stubbs, 1993), it is a vax/vms-based system owned by cbts and campus america and costs in the order of £15,000 £20,000. it comes in two versions: the cml, which is the one used at deakin university, and the cml/ cis, a new release containing extra tools to help with the setting of questions at various levels of complexity. it apparently uses a version of bloom's taxonomy to aid question design. it has the facility to control student progress through courses by releasing tests and assignments, to block progress if defaults are met, to update printed materials as students reach that part of the course, and to accept 'written' answers and pass them across to staff for on-screen marking and feedback. this is currently done via vax mail, but the latest release of the software has the facility built within the structure. staff can then update student marks in real time. computer-marked assessments are recorded automatically and students are allowed to pass to the next assessment point as soon as they are ready. it has a wide range of management information. this includes reports on screen or paper about the progress of groups, individuals, questions and how they are answered, cohorts, etc. all the staff who used the package were clear that it was rock solid, and they never had software problems. 135 users worldwide use cml, some 35 in australia. of those in australia, three have more than 5,000 students using the system. a tool like this is an essential part of supporting staff in their coping with the increasing volume of students that we seem destined to have on our campuses. i guess we could each write our own version, but frankly my advice is to join a club, be it this one or one of the others currently appearing in the market place. building in the teacher flexible learning does not imply that the students need be totally 'on their own', and tutorials can be built in as one deems appropriate. students should have some choice of time and day, something which allows a package to have the teacher at the centre but also provides freedom of choice in attending a centre. recent findings of what students enjoy most about higher education show that they like smaller group tutorials but can happily do without large-scale lectures. this facility could well be extended to distance students, with careful planning, and be delivered by telephone, electronic mail or video conference. there are well-qualified staff in regional colleges who can be the local expert to help students using a flexible package. franchising of courses provides for this local support very well indeed. these 'help' points need to be placed in the text of modules with details of how they will work and the location. copyright it is vital to list all material to be reproduced within the learning package. an educational technologist will then be able to advise on any problems and costs, and will help get clearance for its use. recent changes in the copyright licence mean that additional page charges apply if more than four copyright extracts are packaged together for five or more students. with a 36 alt-j volume 2 number 1 minimum charge of 8 pence per page per copy, this adds significantly to production costs (are publishers about to kill the goose with such high charges?). evaluation once designed and written, all modules will need to go through a peer-group evaluation process. necessary revision can then be fed back into the information, design, and layout contents. the final stage will be the testing under user-conditions with even greater feedback being elicited from the students than one would usually seek. flexible material, by its nature, needs to be open and accessible to students and to encourage them to return to the study. therefore, the evaluation process must be more than just test results. conclusion in summary, flexible packages offer many attractive advantages. they can retain the teacher at the centre of the experience. they offer greater choice not only of time, but place and pace of study. they offer a way of underpinning summer schools, since materials can be pre-issued to clients so that the short time they have is used to full advantage. they allow very rapid response to industrial needs. they build on the franchised regional links and its strengths. they provide real flexibility for students who cannot atte.nd at fixed times for all sorts of reasons. finally, they offer choice when numbers for a topic may be low. references atkins, m.j. (1993), assessment issues in hes, london, department of employment. harrison, c. (1993), teaching and learning: a strategy for angliat cambridge, anglia polytechnic university. stubbs, n. (1993), 'unmet demand and over-enrolment problems can be met with cml', paper delivered to australian universities, available from noel stubbs, cbts (australia) pty ltd., 62 elgin street, carlton, victoria, 3053 australia. thinking and learning at work, autumn 1993, department of employment. wagner, l. (1993), 'the economic and pedagogic imperative', using telematics in education and training, march. 37 rich environments for active learning: a definition r. scott grabinger and joanna c. dunlap university of colorado at denver rich environments for active learning, or reals, are comprehensive instructional systems that evolve from and are consistent with constructivist philosophies and theories. to embody a constructivist view of learning, reals: • promote study and investigation within authentic contexts; • encourage the growth of student responsibility, initiative, decision making, and intentional learning; • cultivate collaboration among students and teachers; • utilize dynamic, interdisciplinary, generative learning activities that promote higher-order thinking processes to help students develop rich and complex knowledge structures; and • assess student progress in content and learning-to-learn within authentic contexts using realistic tasks and performances. reals provide learning activities that engage students in a continuous collaborative process of building and reshaping understanding as a natural consequence of their experiences and interactions within learning environments that authentically reflect the world around them. in this way, reals are a response to educational practices that promote the development of inert knowledge, such as conventional teacher-to-student knowledge-transfer activities. in this article, we describe and organize the shared elements of reals, including the theoretical foundations and instructional strategies to provide a common ground for discussion. we compare existing assumptions underlying education with new assumptions that promote problem-solving and higher-level thinking. next, we examine the theoretical foundation that supports these new assumptions. finally, we describe how reals promote these new assumptions within a constructivist framework, defining each real attribute and providing supporting examples of real strategies in action. introduction in today's complex world, simply knowing how to use tools and knowledge in a single domain is not sufficient to remain competitive as either individuals or companies. people must also learn to apply tools and knowledge in new domains and different situations. industry specialists report that people at every organizational level must be creative and flexible problem solvers (lynton, 1989). this requires the ability to apply experience and 5 r. scott grabinger and joanna c dun/op rich environments for active learning: a definition knowledge to address novel problems. consequently, learning to think critically, to analyse and synthesize information to solve technical, social, economic, political, and scientific problems, and to work productively in groups are crucial skills for successful and fulfilling participation in our modern, competitive society. the purpose of this article is to describe and organize the shared elements of rich environments for active learning, or reals, including the theoretical foundations and instructional strategies to provide a common ground for discussion. reals are based on constructivist values and theories including 'collaboration, personal autonomy, generativity, reflectivity, active engagement, personal relevance, and pluralism' (lebow, 1993, p. 5). reals provide learning activities that, instead of transferring knowledge to students, engage students in a continuous collaborative process of building and reshaping understanding as a natural consequence of their experiences and authentic interactions with the world (goodman, 1984; forman and pufall, 1988; fosnot, 1989). advocating a holistic approach to education, reals reflect-, the assumption that the process of knowledge acquisition is 'firmly embedded in the social and emotional context in which learning takes place' (lebow, 1993, p. 6). need for educational change changing society education is receiving increasing pressure from changing global economic circumstances and complex societal needs. yet, according to lynton (1989, p. 23), 'at this time [...] education is far from fully contributing to the economic well-being of this country [united states]'. public and private institutions are demanding employees who can think critically and solve a range of problems, who can move easily from task to task, and who can work efficiently and effectively in team situations; yet they claim that those people are difficult to find. the us education system, to its credit, is neither deaf to this plea nor ignorant of the need to change how and what instruction is delivered in the classroom. calls for restructuring the way students learn come from a variety of institutions including the american association for the advancement of science (1989) and the national council of teachers for mathematics (1989). educators agree that students must learn to solve problems and think independently (feuerstein, 1979; mann, 1979; segal et al, 1985; linn, 1986; resnick and klopfer, 1989; bransford et al, 1990). the challenge for educators is to utilize strategies that teach content in ways that also help to develop thinking, problem-solving, metacognitive, and life-long learning skills (bransford et al, 1990; savery and duffy, 1994). weaknesses within the current system there is considerable evidence that today's students are not particularly strong in the areas of thinking and reasoning (resnick, 1987; nickerson, 1988; bransford et al, 1991). bransford et al (1990, pp. 115-16) state that the 'basic problem is that traditional instruction often fails to produce the kinds of transfer to new problem-solving situations that most educators would like to see'. conventional instruction often utilizes simplified, decontextualized examples and problems, leading to an inadequate understanding of and 6 alt-] volume 3 number 2 ability to apply the knowledge acquired. students often are not exposed to examples and problems that make knowledge relevant to them (collins et al, 1991). instead, students are asked to solve problems that cause them to wonder: 'why do i need to know this?'. because the information presented to students has no relevance or meaning for them, they tend to treat new information as facts to be memorized and recited rather than as tools to solve problems relevant to their own needs (bransford et al, 1990). this, unfortunately, leads to inert knowledge knowledge that cannot be applied to real problems and situations. inert knowledge research shows that knowledge learned but not explicitly related to relevant problemsolving remains inert (whitehead, 1929; perfetto et al, 1983, ctgv, 1993c). whitehead first coined the phrase inert knowledge in 1929 to refer to knowledge acquired in abstract circumstances without direct relevance to the learner's needs. inert knowledge is not readily available for application or transfer to novel situations (for a review of transfer research, refer to clark and voogel, 1985; butterfield and nelson, 1989). it seems that our educational system is focused on producing inert knowledge. the cognition and technology group at vanderbilt (ctgv) (ctgv, 1993c) specifies the following flaws in our conventional approaches to schooling and teaching that lead to inert knowledge: • there is a constant battle of breadth versus depth and breadth usually wins. we (educators) tend to fill our students with facts, and leave no time for dealing with topics in depth. we expect our students to remember dates, formulae, algorithms, quotations, and whole poems, yet show little practical use for that knowledge despite the fact that we know our students have difficulty transferring the knowledge. robertson (1990) states that: students who rely on memorized algorithms for solving problems typically do not perform as well on transfer problems as do students who rely on an understanding of the underlying concepts, (p. 253) • we consistently rely on decontextualized instructional strategies. in our desire to cover as much material as possible, we focus our instructional activities on abstract basic skills, concepts, and technical definitions. we believe that decontextualized skills have broad applicability and are unaffected by the activities or environments in which they are acquired and used (brown et al, 1989). however, when we do this, students do not learn when to apply those skills or within what kinds of contexts they work. we do this despite a large body of evidence that indicates that abstracted skills are seldom transferred from one domain to another (clark and voogel, 1985; butterfield and nelson, 1989). we fail to realize that abstracted skills have no contextual cues to relate to new situations. • when we do provide practice for our students, we give them arbitrary, uninteresting, unrealistic problems to solve. the example of story problems in mathematics is overused. we can also find examples of unrealistic and over-simplified problems in the sciences, languages, and social studies. again, we do this in the mistaken belief that we must emphasize decontextualized skills that are applicable everywhere. yet these unrealistic problems have no meaning to the students who then fail to find any contextual cues to relate to problems they may encounter in their lives. 7 r scott grabinger and joanna c dunlap rich environments for active learning a definition • we treat students passively for 12 to 16 years, rarely giving them the opportunity to take responsibility for their own learning, to explore ideas of their own choosing, to collaborate with one another or with teachers, or to make valuable contributions to the learning of others. they do not learn to takl'charge of their own learning, nor do they learn the skills necessary to become life-long learners and daily problem-solvers. to the preceding four items cited in the ctgv article, we add a couple more to the list of conventional educational practices that foster the production of inert knowledge: • students are not evaluated in authentic ways. after teaching in decontextualized ways, we test in the same ways. we do not look at actual performance but use paper-pencil tests to measure the quantity of knowledge learned. we fail to examine the quality of their thinking or problem-solving. • finally, our current school practices often have negative effects on the morale and motivation of students. perelman (1992) states: students are forced to compete to achieve as much as they can within the periods of time allotted for each activity. this design requires that most students fail or do less well most of the time so that a minority of them can be labelled 'excellent.' the main functional focus of the system is not 'learning,' it is 'screening out', (p. 72) this has a tremendous effect on society. drop-out rates are higher than our society can afford. we have created an evaluation, testing, and grading sub-structure that helps perpetuate the view that education is a game that has few winners and many losers. this game teaches our students to focus on tests and grades rather than on problem-solving, applying knowledge, and working collaboratively in a risk-free environment. the best students learn early on that they succeed best by working by themselves as quickly as possible. they learn to beat the tests and win the game. erroneous assumptions we begin to change these conventional practices by calling into question some of our basic assumptions. berryman (1991) states that the educational practices described above stem from five erroneous assumptions about learning that have governed education since the beginning of the industrial age. he holds that we often assume incorrectly that: • people easily transfer learning from one situation to another if they have learned the fundamental skills and concepts [decontextualized]; • learners are 'receivers' of knowledge in verbal forms from books, experts, and teachers; • learning is entirely behaviouristic, involving the strengthening of bonds between stimuli and correct responses; • learners are blank slates ready to be written upon and filled with knowledge; and • skills and knowledge are best acquired independent of realistic contexts. to begin to address the issues of transfer and instructional methods to meet employer and societal needs, reasoning and problem-solving skill development must be a part of an interdisciplinary program of study in education (lynton and elman, 1987) a program 8 alt-j volume 3 number 2 or environment that places students in situations where they can practise solving problems in a meaningful and constructive manner. w e need to look at other ways one view of an alternative framework comes from researchers who are beginning to emphasize the importance of anchoring or situating instruction in meaningful problemsolving environments (ctgv, 1993c). the cognition and technology group at vanderbilt is a leader in the development of alternative frameworks of instruction and schooling. the group posits two significant changes. first, we as educators must establish new goals for learning. we must move from emphasizing decontextualized reading and computational skills to developing independent thinkers and learners who engage in life-long learning. this does not mean that we abandon the important skills of reading and computation; it means instead that we should be teaching reading and computation within more situated contexts that demonstrate the value of those skills. new assumptions about learning second, in contrast to our long operative conventional assumptions (see above), we must base our teaching on new assumptions about the nature of thinking, learning, and instruction. we must accept that [...] the mere accumulation of factual or declarative knowledge is not sufficient to support problem solving. in addition to factual or declarative knowledge, students must learn why, when, and how various skills and concepts are relevant. (ctgv, 1993c, p. 79) effective problem-solving and thinking are not based solely on motivation and knowledge of thinking strategies, but also on well-organized and indexed content knowledge. learners must have rich knowledge structures with many contextual links to help them address and solve complex problems. this means that instead of trying to abstract a set of decontextualized general skills, we must make our teaching as contextualized as possible to provide as many possible links with other domains as possible. therefore, we propose the following 'new' assumptions about learning and teaching: • people transfer learning from one situation to another with difficulty. learning is more likely to be transferred from complex and rich learning situations. rich learning activities help students think deeply about content in relevant and realistic contexts (ctgv, 1993c). • learners are 'constructors' of knowledge in a variety of forms. they take an active role in forming new understandings and are not just passive receptors. • learning is a collaborative process. students learn not solely from experts and teachers, but also from each other. they test ideas with each other and help each other build elaborate and refined knowledge structures. • learning is cognitive, and involves the processing of information and the constant creation and evolution of knowledge structures. we must focus on and make visible thinking and reasoning processes. we are not suggesting abandoning the teaching of 9 r. scott grabinger and joanna c dunlap rich environments for active learning: a definition content to teach only thinking and reasoning, because 'knowledge of concepts, theories, and principles [. . .] empowers people to think effectively' (bransford et al, 1990, p. 115). • learners bring their own needs and experiences to a learning situation and are ready to act according to those needs. we must incorporate those needs and experiences into learning activities to help students take ownership and responsibility for their own learning. • skills and knowledge are best acquired within realistic contexts. morris etal{\ 979) call this 'transfer appropriate processing.' transfer appropriate processing means that students must have the opportunity to practise and learn the outcomes that are expected of them under realistic or authentic conditions. • assessment of students must take more realistic and holistic forms, utilizing projects and portfolios and de-emphasizing standardized testing. educators are increasingly aware that conventional achievement and intelligence tests do not measure the ability of people to perform in everyday settings and adapt to new situations (ctgv, 1993c). a discussion of the foundations for these assumptions and examples of their implementation makes up the rest of this article. rich environments for active learning definition of reals we must implement a number of strategies in order successfully to adopt the new assumptions about thinking, learning, instruction, and achievement described above. the adoption of these strategies creates learning environments that we call rich environments for active learning (reals). reals are comprehensive instructional systems that (dunlap and grabinger, 1992,1993; grabinger and dunlap, 1994a, 1994b): • evolve from and are consistent with constructivist philosophies and theories; • promote study and investigation within authentic (i.e. realistic, meaningful, relevant, complex, and information-rich) contexts; • encourage the growth of student responsibility, initiative, decision-making, and intentional learning; • cultivate an atmosphere of knowledge-building learning communities that utilize collaborative learning among students and teachers (collins, 1995); • utilize dynamic, interdisciplinary, generative learning activities that promote high-level thinking processes (i.e. analysis, synthesis, problem-solving, experimentation, creativity, and examination of topics from multiple perspectives) to help students integrate new knowledge with old knowledge and thereby create rich and complex knowledge structures; and, • assess student progress in content and learning-to-iearn through realistic tasks and performances. it is important to note that two of the overall defining characteristics of learning environments are integration and comprehensiveness (hannafm, 1992). hannafin 10 alt-j volume 3 number 2 describes integration as a process of linking new knowledge to old, and modifying and enriching existing knowledge. integration enhances the depth of learning to increase the number of access points to that information. as we described earlier, learning environments must go beyond general, abstracted learning to include specific learning. goldman et al state that: these environments are designed to invite the kinds of thinking that help students develop general skills and attitudes that contribute to effective problem solving, plus acquire specific concepts and principles that allow them to think effectively about particular domains, (p. 1) comprehensiveness, the second defining characteristic, carries this notion further. it refers to the importance of placing learning in broad, realistic contexts rather than in decontextualized and compartmentalized contexts. real learning strategies, then, guide and mediate an individual's learning, and support the learner's decision-making (hannafin, 1992). themes are used to help organize learning around interdisciplinary contexts that focus on problem-solving or projects that link concepts and knowledge to focused activities within the environment (hannafin, 1992). what a real is not because the phrase learning environment is broadly and carelessly used in educational literature to describe everything from schools to classrooms to computer microworlds to learning activities to air conditioning and furniture, we shall try to clarify what a real is not before examining the attributes in more detail. we attempt to make the case that a real is a more accurate description of what people generally mean when they use the term learning environment. first, a real is not a delivery technology like video, cd-rom, or audio tapes. clark (1994) defines delivery technologies as those that draw on resources and media to deliver instruction, and affect the cost and access of instruction. media technologies can be integral components of reals and, in fact, usually are. however, a real is not limited to any specific media, but instead is an assortment of methods and ideas that help create an environment that promotes and encourages active learning. clark's point is important from a research standpoint because instructional methods are often confounded with media in research, and he argues strongly and convincingly that it is instructional methods, not media, that influence learning. he contends that any necessary teaching method can usually be designed in more than one medium. although there are varying degrees of acceptance and disagreement with clark's point of view (for example, jonassen, 1994b; kozma, 1994), a real is a set of instructional methods designed on the assumptions that media are tools for students and teachers' to use, and that the learning that occurs within the environment is founded on the activities and processes that encourage thinking and reasoning, not the media that deliver information. second, do not confuse reals with computer-based microworlds or learner-support environments (allinson and hammond, 1990). computer-based microworlds are computer programs that are designed to apply constructivist theories. examples include case-based applications, simulations, intentional learning environments, and some hypermedia resources. developers of microworlds often refer to their programs as learning environments because they often attempt to simulate, on a smaller simplified ii r scott grabinger and joanna c dunlap rich environments for active learning: a definition scale, realistic environments. however, we contend that this limits the concept of learning environment. learning environments, and especially reals, are much more comprehensive and holistic than individual computer applications. although some "computer-based applications use constructivist ideas quite admirably see especially the strategic teaching framework (duffy, 1992; duffy, in press), and the' transfusion medicine modules (ambruso and the transfusion medicine group, 1994) they are not learning environments in the sense that reals are. to create reals, teachers must involve their students, parents, administrators, and colleagues in planning and implementing strategies that encourage student responsibility, active knowledge construction, and generative learning activities on a large scale and in a variety of methods and forms. microworlds may play a role in a real through the delivery of information, practice, finding and presenting information, stimulation of high-level thought processes, promotion of collaboration, or exploration; but reals involve many more activities and demand much more flexibility than can probably ever be contained in a single computer program. a real is, then, a learning community that 'includes the content taught, the pedagogical methods employed, the sequencing of learning activities, and the sociology of learning' (collins et al, 1991, p. 6). finally, a real is not the physical attributes of the classroom. it is not about air conditioning, desks, lighting, or ergonomics. while these environmental factors are critical in creating a smooth-operating learning environment, they are not included in our definition or reals (for information about this topic, see gayeski, 1995). foundation of reals reals (i.e. constructivist learning environments, information-rich learning environments, interactive-learning environments, or knowledge-building learning communities) are not new to education. we can go back to socrates and see that he used problems and questions to guide students to analyse and think about their environments (coltrane, 1993). rousseau prescribed using direct experience (farnham-diggory, 1992). in the early 1900s, john dewey (1910) proposed student-directed reforms and experiential learning. bruner (1961) advocated discovery or inquiry learning around realistic problems. the notion that students should learn through practice, application, and apprenticeship has been with us for centuries. it was not until the industrial age, when we needed places to store children until old enough to work on assembly lines, that we began trying to mass produce replicable results. yet the last five to ten years have seen renewed emphasis on reforming schools and teaching practices to replace our educational production lines with classrooms that teach people to think and solve problems. this current effort at renewal revolves around a set of ideas and theories referred to as constructivism. characteristics of constructivism the class of theories that guides the development of reals is called constructivist theories (clement, 1982; bransford and vye, 1989; minstrell, 1989; resnick and klopfer, 1989; schoenfeld, 1989; bednar et al, 1991; duffy and bednar, 1991; perkins, 1991; scardamalia and bereiter, 1991; spiro et al, 1991). fundamentally, constructivism asserts that we learn through a continual process of building, interpreting, and modifying our own representations of reality based upon our experiences with reality (jonassen, 1994c). — alt-] volume 3 number 2 wheatley (1992) elaborates on the importance of this idea not just in learning but also in the innovation and improvement of society: learning (innovation) is fostered by information gathered from new connections; from insights gained by journeys into other disciplines or places; from active, collegial networks and fluid, open boundaries. learning (innovation) arises from ongoing circles of exchange where information is not just accumulated or stored, but created. knowledge is generated anew from connections that weren't there before. when this information self-organizes, learning (innovation) occurs, the progeny of information-rich, ambiguous environments, (p. 113) ttworetical foundation characteristics of constnjcuvum attributes of reals instructloni t t n t e g i m t o support reals construct! vbt view of teaming j_ active knowiedfl* construction and •volution mdixtd knowudg* c s j studtnt responsibility indlnftl«lly« 6*n«nttv* wsi iiuiu strategics reciprocal tticning authentic u«n*ig contexts cognlttv* spprtntlcvstilp authentic tssigmmt stntigtes instruction figure i: real definition: framework lttmingki daslgn collaboration and social negotiation ofnteanlno co-openttv* suppoft proolefli^wweo leamtno cognuv* fitxibilly truory to develop learning environments that encourage the type of learning described by jonassen, wheatley, and others, reals reflect the theories and philosophies of constructivism. in fact, the three main characteristics of the constructivist view of learning govern our design of reals (see figure 1). the first characteristic of constructivism is the notion that knowledge is not a product to be accumulated but an active and evolving process in which the learner attempts to make sense out of the world (gurney, 1989). brown et al (1989) illustrate this idea: a concept, for example, will continually evolve with each new occasion of use, because new situations, negotiations, and activities inevitably recast it in a new, more densely textured form. so a concept, like the meaning of a word, is always under construction, (p. 33) 13 r scott grobinger and joanna c dunlap rich environments for active learning: a definition the second characteristic is the notion that people conditionalize their knowledge in personal ways (gurney, 1989). that is, they acquire knowledge in forms that enable them to use that knowledge later. bransford et al (1990) state that: [...] there are large differences between knowing something and spontaneously thinking to do it or use it when one is engaged in an actual problem-solving situation, (p. 122) knowledge, then, is 'indexed' to the contexts in which we encounter it. we are unlikely to use knowledge that is decontextualized because we can see no relevance no contextual cues cause us to remember that knowledge because it was taught without cues. a person who indexes and conditionalizes knowledge knows when to apply that knowledge. a person who learns in a decontextualized way often is not aware that he or she has the applicable knowledge to solve a problem. in other words, knowledge acquired is inert. students must acquire concepts and theories in ways that help them use the information later on, and appreciate the value of that information. brown et al (1989, p. 36) describe this process as indexicalizing knowledge. they mean that rich involvement in realistic and relevant problem-solving allows learners to develop many broad and deep indexicalized representations that enable them to apply knowledge more spontaneously to new situations because they can compare a known and relevant situation with a new situation. the more links there are across related knowledge structures, the more likely students are to apply that knowledge. the more closely the learning context resembles the actual context, the better people will perform. tulving and thompson (1973) refer to this as 'encoding specificity,' which holds that successful retrieval of information is enhanced when cues relevant to later retrieval of that information are encoded along with the material learned. it is important to note that constructivists contend that these rich links cannot be developed in decontextualized learning activities; rather, learning must be placed in realistic contexts that provide cognitive conflict or puzzlement, and determine the organization and nature of what is learned (savery and duffy, 1994). white (see robertson, 1990) theorizes that there are two kinds of links that need to be developed while learning: internal and external. internal associations are connections among the criterial attributes of a principle. internal associations reflect the learner's understanding of the concept. external associations refer to connections between the principle and everyday experiences or context, and indicate the 'usability' of a concept. learning to solve problems requires both kinds of links. our schools are good at building the internal links, but poor at providing the external links. the third major characteristic of constructivism is the importance of collaboration and social negotiation of meaning. learning happens within a social context. conceptual growth comes from sharing our perspectives and testing our ideas with others, modifying our internal representations in response to that process of negotiation (bednar et al, 1991). common understandings and shared meanings are developed through interaction among peers and teachers. this, is the cultural aspect of knowledge. the activities of a domain are framed by its culture. their meaning and purpose are socially constructed through negotiations among present and past members. activities . thus cohere in a way that is, in theory if not always in practice, accessible to members who move within the social framework. these coherent, meaningful, and purposeful 14 alt-j volume 3 number 2 activities are authentic, according to the definition of the term we use here. (brown et al, 1989, p. 34) this social aspect of constructivism is important on an individual level as well as cultural level, for collaborative interactions allow us to test the viability of our understandings, theories, and conjectures (savery and duffy, 1994). reals embody the characteristics of constructivism, but how? what are the critical attributes of reals that support a constructivist view of learning? the next section of this article discusses each of the criterial attributes of a real: a. student responsibility and initiative. b. generative learning activities. c. authentic learning contexts. d. authentic assessment strategies. e. co-operative support. we will illustrate each attribute with examples of functioning reals utilizing the strategies of reciprocal teaching, cognitive apprenticeship, anchored instruction, cognitive flexibility theory, learning in design, and problem-based learning (pbl). the main attributes of reals a. student responsibility and initiative the first attribute of reals is that they are student-centred. student-centred learning environments place a major emphasis on developing intentional learning and life-long learning skills. these skills include the abilities to construct higher-order questions to guide learning, reflect on consequences and implications of actions, and monitor and modify personal cognitive activity. this attribute is critical because students cannot actively construct and evolve their knowledge structures without taking responsibility and initiative for their learning. intentional learning scardamalia and her colleagues (1989) noticed that passive or immature learners have certain characteristics that prevent them from becoming skilful problem solvers. first, immature learners tend to organize their mental activities around topics rather than goals, promoting decontextualization and the inability to see the relevance of the learning activity to their lives. second, they tend to focus on surface features, and do not examine a topic in depth. third, they tend to work until a task is finished. they do not take time to examine the quality of their work, nor do they take the time to consider revisions to their finished product or to the strategies utilized to complete the task. when the task is finished, they forget whatever learning took place during their work on the task. finally, they think of learning in an additive fashion rather than transforming and enriching their existing knowledge structures. these characteristics are, in essence, the product of the conventional kind of schooling that we described earlier. these behaviours prevent students from transferring their is r scott grab/nger and joanna c dun/op rich environments for active learning a definition knowledge to new problems because they have learned content and strategies in a decontextualized context. palincsar (1990) states that: to achieve transfer it is necessary to attend to the context in which instruction and practice occur; transfer is likely to occur to the extent that there are common elements between the situation in which the children are learning this tactic and the situations in which such a tactic would be useful, (p. 37) palincsar, scardamalia, and bereiter are leading proponents of the conviction that students must be taught to take more responsibility for their own learning to enhance the likelihood of transfer. they refer to this concept as intentional learning, or 'those cognitive processes that have learning as a goal rather than an accidental outcome' (bereiter and scardamalia, 1989, p. 363). palincsar and klenk (1992) state that 'intentional learning, in contrast to incidental learning, is an achievement resulting from the learner's purposeful, effortful, self-regulated, and active engagement'. to be intentional learners, students must learn to learn as well as accrue knowledge. teaching, too, takes on a revised role, for to teach for intentional learning means to cultivate those general abilities that facilitate life-long learning (palincsar, 1990). the main skills involved in teaching students to be more intentional are questioning, self-reflection, and metacognition, 'or the awareness and ability to monitor and control one's activity as a learner' (brown et al, 1983, p. 212). questioning scardamalia and bereiter (1991) believe that one of the first steps in developing intentional learners is to help students take more executive control over what they decide to learn through the development of higher-order questioning skills. they point out that in a typical classroom, teachers ask all the questions. therefore, teachers are the ones engaged in the question generation process, which involves important high-level thinking skills and executive control decisions the skills students need to develop actively to construct and evolve their knowledge structures. the students are not given an opportunity to ask questions and, therefore, do not learn to perform the analysis activities related to question generation. research by scardamalia and bereiter (1991) indicates that students can learn to ask questions to guide their knowledge-building, thus assuming a 'higher level of agency' and more ownership for their learning. in a student-centred real, students are given more executive control over their learning to enable them to take more ownership, to find more relevance and authenticity, and to learn life-long learning skills. self-reflection a second skill in intentional learning is self-reflection. 'self-reflection implies observing and putting an interpretation on one's own actions, for instance considering one's own intentions and motives as objects of thought' (von wright, 1992, p. 61). von wright writes that self-reflection involves the abstraction of meaning and is an interpretative process aimed at the understanding of reality. to understand the world in different ways involves modifying our conceptions of the world and our place in the world. it involves thinking about reality in alternative ways. von wright goes on to describe two levels of reflection. one level of reflection is the ability to think about consequences and implications of actions. a second level of reflection is the ability to think about oneself as — _ _ alt-j volume 3 number 2 an intentional subject of one's own actions and to consider the consequences and efficacy of those actions. this involves the ability to look at oneself in an objective way, and to consider ways of changing future actions to improve performance. the second level of reflection also involves metacognitive learning skills. metacognitive skills 'metacognitive skills refer to the steps that people take to regulate and modify the progress of their cognitive activity: to learn such skills is to acquire procedures which regulate cognitive processes' (von wright, 1992, p. 64). metacognitive skills include taking conscious control of learning, planning and selecting strategies, monitoring the progress of learning, correcting errors, analysing the effectiveness of learning strategies, and changing learning behaviours and strategies when necessary (ridley et al, 1992). these abilities interact with developmental maturation and domain expertise. immature learners have not acquired the ability to regulate or modify their cognitive activity; they may have learned a single strategy, such as memorization, and default to that one strategy in all situations. • studies show that the use of metacognitive strategies can improve learning skills, and that the independent use of these metacognitive strategies can be gradually developed in people (brown, 1978; biggs, 1985; weinstein et al, 1988). blakely and spence (1990) describe several basic instructional strategies that can be incorporated into classroom activities to help students develop metacognitive behaviours: • students should be asked to identify consciously what they 'know' as opposed to what they 'do not know'. • students should keep journals or logs in which they reflect upon their learning processes, noting what works and what does not. • students should manage their own time and resources, including estimating time required to complete tasks and activities, organizing materials and resources, and scheduling the procedures necessary to complete an activity. • students must participate in guided self-evaluation using individual conferences and checklists to help them focus on the thinking process. real strategy: reciprocal teaching one of the manifestations of a real that emphasizes the development of intentional learning skills described above is reciprocal teaching. the context of reciprocal teaching is social, interactive, and holistic. palincsar and klenk (1992) used reciprocal teaching with at-risk first-grade students to develop reading skills. palincsar and klenk (1992) describe reciprocal teaching as: [. . .] an instructional procedure that takes place in a collaborative learning group and features guided practice in the flexible application of four concrete strategies to the task of text comprehension: questioning, summarizing, clarifying, and predicting. the teacher and group of students take turns leading discussions regarding the content of the text they are jointly attempting to understand, (p. 213) these strategies are the kinds of intentional learning strategies that encourage selfregulation and self-monitoring behaviours. 17 r scott grabinger and joanna c dunlap rich environments for active learning: a definition the relationship of reciprocal teaching to reals is founded on three theoretical principles, consistent with the characteristics of constructivism described previously, based on the work of vygotsky (1978) as described by palincsar and klenk (1992). the first principle states that the higher cognitive processes originate from social interactions. the second principle is vygotsky's zone of proximal development (zpd). vygotsky described the zpd as: [. . .] the distance between the actual developmental level as determined by independent problem-solving, and the level of potential development as determined through problemsolving under adult guidance or in collaboration with more capable peers, (p. 86) reciprocal teaching is designed to provide a zpd in which students, with the help of teachers and peers, take on greater responsibility for learning activities. finally, vygotsky's third principle advocates that learning take place in a contextualized, holistic activity that has relevance for the learners. how, then, does reciprocal teaching work? the process begins with a text that a class reads silently, orally, or along with the teacher depending on the skill level of the students. following each segment, a dialogue leader (students take turns) asks questions that deal with content or 'wonderment' issues. the questions often stimulate further inquiry. the other students respond to the questions, raise their own questions, and, in cases of disagreement or confusion, re-read the text. the discussion leader is responsible for summarizing and synthesizing the reading and discussion, and clarifying the purpose of the reading. the leader also generates and solicits predictions about the upcoming text to prepare for meaningful reading of the next segment. the teacher must model the appropriate behaviour and provide scaffolding to sustain the discussion. (the preceding description is taken from palincsar and klenk, 1992.) the students, then, are involved in the higher-level thinking and decision-making activities that usually fall within the realm of the teacher. with the help of the teacher, students share a zpd where they can learn the questioning, summarizing, clarifying and predicting activities so integral to metacognitive awareness. finally, why does reciprocal teaching work? collins et al (1991) posit the following reasons for its success (and, in a broader view, for the success of reals): • the reciprocal teaching model engages students in activities that help them form a new conceptual model of the task of reading. they see reading as a process that involves reflection and prediction rather than just the recitation of words. they learn to make what they are reading relevant to their needs and to monitor their progress and strive for clarification. • the teacher and student share a problem context while the teacher models expert strategies that the students learn to use independently. • scaffolding is crucial in the success of reciprocal teaching. 'most importantly, it, decomposes the task as necessary for the students to carry it out, thereby helping them to see how, in detail, to go about it' (collins, et al, 1991, p. 11). • finally, students learn the self-monitoring activities and thinking processes involved in critiquing and improving their work. 18 alt-] volume 3 number 2 b. generative learning activities the second requirement of reals is that students engage in generative learning activities. people who learn through active involvement and use tools build an 'increasingly rich implicit understanding of the world' (brown et al, 1989, p. 33). generative learning requires that students 'engage in argumentation and reflection as they try to use and then refine their existing knowledge as they attempt to make sense of alternate points of view' (ctgv, 1993b, p. 16). studies indicate that knowledge is more likely to be active and used when acquired in a problem-solving mode rather than in a factual-knowledge mode (adams et al, 1988; lockhart et al, 1988). the concept of generative learning is an extension of the constructivist characteristic of actively constructing knowledge; students cannot construct or evolve their own learning without generating something through active involvement. generative learning requires a shift in the traditional roles of students and instructors. students become investigators, seekers, and problem solvers. teachers become facilitators and guides, rather than presenters of knowledge. for example, rather than simply learning what objectives and goals are, students in a teacher-education class generate lesson plans and objectives and then manipulate and revise them to solve new teaching problems. ideally, they test their objectives, goals, and strategies in actual practice-teaching situations. in generative learning, students apply the information they learn. generative learning activities require students to take static information and generate fluid, flexible, usable knowledge. generative learning, then, means that students are involved deeply and constantly with creating solutions to authentic problems via the development and completion of projects. a real strategy that relies heavily on projects is cognitive apprenticeship. real strategy: cognitive apprenticeship cognitive apprenticeship is modelled after the traditional apprenticeship way of learning arts and crafts. it incorporates elements of traditional apprenticeship and modern schooling. in traditional apprenticeship, the processes of an activity are visible and involve learning a physical and outwardly observable activity (collins et al, 1991). the expert shows an apprentice how to perform a task, then watches and coaches as the apprentice practises portions of the task, and finally turns over more and more responsibility to the apprentice until the apprentice can perform the task alone (collins et al, 1991). traditional apprenticeship deals with processes that are easily visible because they involve skills and produce visible products. the goal of cognitive apprenticeship is to make processes that are normally invisible visible. in schooling, the process of thinking is usually invisible to both students and teachers. for example, the practices of problem-solving, reading comprehension, and computation are not visible processes (collins et al, 1991). brown et al (1989) point out that the term cognitive apprenticeship emphasizes that apprenticeship techniques can reach beyond observable physical skills to the kinds of cognitive skills associated with learning in schools. in a cognitive apprenticeship environment, the teacher attempts to make visible the thinking processes involved in performing a cognitive task. the teacher first models how to perform a cognitive task by thinking aloud. then the teacher watches, coaches, and provides scaffolding as the students practise portions of the task. finally, he or she turns over more and more responsibility to students, and fades coaching and 19 r scott grabinger and joanna c ounlop rich environments for active learning a definition scaffolding until they can perform the task alone. 'cognitive apprenticeship supports learning in a domain by enabling students to acquire, develop, and use cognitive tools in an authentic domain activity' (brown et al, 1989, p. 39). the differences between traditional and cognitive apprenticeship (collins et al, 1991) are important because they indicate where the effort must be placed on the instructional design of learning activities. first, in traditional apprenticeship the task is easily observable. in cognitive apprenticeship, the thinking processes must be deliberately brought into the open by the teacher, and the teacher must help students learn to bring their thinking into the open. second, in traditional apprenticeship, the tasks come from work, and learning is situated in the workplace. in cognitive apprenticeship, the challenge is to situate the abstract goals of school curriculum in contexts that make sense to students. third, in traditional apprenticeship, the skills learned are inherent in the task. in schooling, students learn skills that are supposed to transfer to different tasks. in cognitive apprenticeship, the challenge is to present a range of tasks to encourage reflection and to identify common transferable elements across tasks. the goal is to help students generalize and transfer their learning through conditionalized and indexed knowledge (related to the second characteristic of constructivism). cognitive apprenticeship and generative learning are closely linked because the process of making cognitive processes visible means that students must create or generate things that represent those processes. teachers must create work and tasks that represent the process of solving a problem, writing, or computation in addition to products. to examine the development of student thinking, an english teacher may ask for questions, themes, concept maps, and outlines before students begin writing. mathematics teachers are often notorious for telling students: 'i want to see your work, not just the answer,' so they can look for errors in the thinking process. generative learning is one of the simplest features of a real. it simply demands that students produce something of value. it is probably the most exciting part of a real because students work on projects and tasks that are relevant to them and to their peers. it keeps students busy and happy or active while helping them construct and evolve their knowledge structures. c. authentic learning contexts the third attribute of reals is that learning takes place within an authentic context. an authentic task, activity, or goal provide learning experiences as realistic as possible, taking into consideration the age and maturation level of the students and environmental constraints such as safety and resource availability. an authentic context incorporates as much fidelity as possible to what students will encounter outside school in terms of tools, complexity, cognitive functioning, and interactions with people (williams and dodge, 1992). therefore, creating an authentic learning context requires more than just presenting students with realistic problems or situations it also means that students must address the problems or situations realistically as well (honebein et al, 1993). authenticity is important to reals for three reasons. first, realistic problems hold more relevance to students' needs and experiences because they can relate what they are 20 aa-j volume 3 number 2 learning to problems and goals that they see every day. therefore, it encourages students to take ownership of the situation and their own learning. second, because the situations students encounter during learning are authentic and reflect the true nature of problems in the real world, it develops deeper and richer (indexicalized and conditioned) knowledge structures, leading to a higher likelihood of transfer to novel situations. finally, because complex problems require a team approach that provides natural opportunities for learners to test and refine their ideas and to help each other understand the content, it encourages collaboration and negotiation. real strategy: anchored instruction one of the ways to create authentic instruction in a real is to anchor that instruction in a realistic event, problem, or theme (ctgv, 1990, 1992a, 1992b, 1993a, 1993d). anchored instruction is fixed within a real-world event that is appealing and meaningful to students (bransford et al, 1990) and involves complex contexts that require students to solve interconnected sub-problems. because students are encouraged to work together to solve these complex problems, they are exposed to multiple perspectives in an environment that gives them an opportunity to test out their ideas, solutions, and processes (ctgv, 1992b). at the heart of the model is an emphasis on the importance of creating an anchor or focus to generate interest, and to enable students to identify and solve problems and pay attention to their own perception and comprehension of these problems' (bransford et al, 1990, p. 123). in anchored learning situations, students develop component skills and objectives in the context of meaningful, realistic problems and problem-solving activities. these complex contexts are called 'macrocontexts' (williams and dodge, 1992, p. 373). addressing a key characteristic of constructivism indexed knowledge acquisition the primary goal of anchored instruction (and reals) is to overcome the problem of inert knowledge. for example, students in an instructional design and development class work in teams with actual clients to develop instruction that will be delivered to another group of students. they must define the problem, identify resources, set priorities, and explore alternative solutions with the same skills and abilities that are required during realistic, outside-theclassroom problem-solving and decision-making activities. this is in direct contrast to the way students develop component skills and objectives in a more conventional classroom environment by working simplified, compartmentalized, and decontextualized problems. simply stated, it is the difference between providing meaningful, authentic learning activities and 'i'm never going to use this' activities. anchored instruction shares many features of programmes that are case-based and problem-based (barrows, 1985; spiro et al, 1991; williams and dodge, 1992). the idea is to let learners experience the intellectual changes that experts feel when modifying their own understandings from working with realistic situations (ctgv, 1992b). effective anchors are intrinsically interesting, fostering ownership, and helping students notice the features of problem situations that make particular actions relevant (bransford et al, 1990). the ctgv (1991) uses the following design principles when creating anchored instruction. first, they use a video-based presentation format because of the dramatic power of the medium and because of the use of multiple modalities, realistic 21 r. scott grabinger and joanna c dunlap rich environments for active learning a definition imagery, and omnipresence in our culture. second they present a problem using actors and a narrative format for interest. third, the problem solution requires a generative learning format in which students must identify pertinent information in the fourth feature, embedded data design. fifth, the problem is complex, with the possibility of multiple solutions, and requires a team approach. sixth, they use pairs of similar problems in different contexts to enrich the indexicalization of knowledge structures. finally, they attempt to draw links across the curriculum to enhance the relevance of the problem. one of the ctgv's projects in anchored instruction is the jasper woodbury series (ctgv, 1992b). jasper is a video-based series designed to promote problem-posing, problem-solving, reasoning, and effective communication. each of jasper's adventures is a 15to 20-minute story in which the characters encounter a problem that the students in the classroom must solve before they are allowed to see how the movie characters solved the problem. the jasper series helps students learn to break a problem into parts, generate sub-goals, find and identify relevant information, generate and test hypotheses, and co-operate with others. real strategy: cognitive flexibility theory cognitive flexibility theory (cft) also addresses the need for authentic learning contexts in order to help learners develop conditionalized and indexicalized knowledge structures (spiro et al, 1991; jacobson and spiro, 1992).' essentially, the theory states that cognitive flexibility is needed in order to construct an ensemble of conceptual and case representations necessary to understand a particular problem-solving situation. the idea is that we cannot be said to have a full understanding of a domain unless we have the opportunity to see different case representations (borsook and higginbotham-wheat, 1992, p. 63). cft attempts to teach content in ill-structured domains, that is, in domains where the knowledge-base is so vast and complex that multiple solutions to problems are possible and likely. there are no clear-cut answers in ill-structured domains, so simple algorithms often fail. ill-structured domains include law, medicine, and education. so, cft emphasizes the following instructional strategies to help learners develop rich and deep knowledge structures (jacobson, 1994): • cft uses several cases and rich examples in their full complexity. one of the tenets of cft is to avoid over-simplifying knowledge and examples because this leads to future misunderstandings that are difficult to change. • cft uses multiple forms of knowledge representation, providing examples in several kinds of media. cft encourages students to look at knowledge in several ways and from several perspectives. • cft links abstract concepts to case examples and brings out the generalizable concepts and strategies applicable to other problems or cases. • to avoid the mistakes of over-simplification, cft presents a number of examples to make apparent, rather than hide, the variability of concepts and themes within the domain. 22 an-] volume 3 number 2 although authentic learning contexts help students to develop knowledge that can be transferred and applied to new problems and situations, the fact that students are engaged in authentic activities creates some unique assessment problems. these issues will be examined next. d. authentic assessment strategies going hand in hand with the need to develop authentic learning contexts, the fourth real attribute is the use of authentic assessment strategies to evaluate student performance. conventional schooling relies on standardized and paper/pencil tests to measure the quantity of knowledge that students have accrued. but traditional tests, written reports, and grading schemes are inappropriate measures (frederiksen and collins, 1989), time-consuming to administer and score (williams and dodge, 1992), and poor indicators of how students will perform in actual problem-solving conditions. williams and dodge also state that students are often assessed on skills different from ones that are taught, and experience problem-solving assessments that tend to be subjective. testing and assessment must recognize the importance of the organization of the knowledge-base and its connectedness to contexts. wiggins (1989) contrasts authentic tests, which he describes as contextualized, complex intellectual challenges against multiple-choice measures that he describes as fragmented and static. according to wiggins, authentic tests include the following criteria: • the intellectual design features of tests and evaluation tasks must emphasize realistic complexity, stress depth more than breadth, include ill-structured tasks or problems, and require students to contextualize content knowledge. • standards of grading and scoring features should include complex multi-faceted criteria that can be specified and that are reliable across multiple scorers. what constitutes a high level of performance should be explainable to students and teachers before they take the test. teachers often claim that criteria are subjective, but this is seldom the case; most criteria can be described with some thoughtful effort. • tests and evaluations must be diverse, and must recognize the existence of multiple kinds of intelligences. in terms of fairness and equity, evaluations and assessments should allow students to use their strengths within areas where their interests lie. real example: learning in design carver et al (1992) elaborate on this theme by proposing an extensive list of behaviours needed by students in a real. in their particular manifestation of a real, they consider the classroom a design community in which students design instruction for other students, documentaries for local media, and other exhibits for the community. their program has the same goals of high-level thinking, reflection, and transfer as the other real strategies described: the instructional virtues of these design experiences include the opportunity to develop and coordinate a variety of complex mental skills, such as decomposing a topic into subtopics, gathering data from a variety of sources, organizing diverse and often contradictory information, formulating a point of view, translating ideas into a presentation targeted at a particular audience, evaluating the design, and making revisions based on the evaluations. (carver ef a/, 1992, p. 386) 23 r. scott grabinger and joanna c dunlap rich environments for active learning: a definition again, there are several parallels with the other examples that we have discussed. their real focuses on complex mental skills; analysing, comparing, and manipulating information; working on authentic, community-based tasks; and working with others. to evaluate fairly students working in this environment, teachers need a clear specification of the skills students need for design tasks and prescriptions for how teachers can effectively support those skills (see also agnew et al, 1992). specification of skills and prescriptions of support are two parts of assessment that must be linked for fair assessment. if a skill cannot be supported by the teacher or some kind of scaffolding technique, then it cannot be fairly evaluated. it may, in fact, be outside the zpd (zone of proximal development) and beyond the current capability of the student. one of the teacher's jobs in a real is to specify skills and performances that can be supported so the student can grow in ability. carver et al (1992) break the important behaviours for their environment into: • project-management skills, including creating a timeline, allocating resources, and assigning team roles; • research skills, including determining the nature of the problem, posing questions, searching for information, developing new information, and analysing and interpreting information; • organization and representation skills, including choosing the organization and structure of information, developing representations (text, audio, and graphics), arranging structure and sequence; and juggling constraints; • presentation skills, including transferring their design into media and arousing and maintaining audience interest; and • reflection skills, including evaluating the process and revising the design. their criteria work for their learning environment. other models may need to revise some of the specifics, though the five main categories provide an excellent starting place in specifying skills targeted for assessment in authentic assessments. goldman, pellegrino, and bransford's (1994) discussion of assessment in the ctgv jasper series suggests the following assessment areas: assessment of complex mathematical problem-solving (mathematics is the content domain of jasper), measures of group problem-solving performance, assessment of extensions into other areas of the content area, and assessment of cross-curricular extensions. although conducting an assessment for a real is more work than conventional assessment, it is also an integral part of the learning process rather than a periodic quantifiable measure. authentic assessment provides feedback and information that is useful for planning future learning activities. assessment in reals means that we have to consider more varied techniques. neuman (1993), in her work with the perseus hypermedia program, suggests several alternatives. first, she suggests that teachers use more observations including evaluator observations of performance processes, think-alouds by students, and automatic transaction monitoring. second, she suggests using interviews of students, instructors, and staff using both questionnaires and focus groups. finally, she suggests using document and product analysis including assignments, syllabi, essays, journals, paths, reports, documentation, and presentations. 24 alt-j volume 3 number 2 e. co-operative support the fifth and final characteristic of reals, and one that is an important feature of almost all the other real strategies described above, acknowledges the transactional nature of knowledge and suggests that a shift be made to focus on social practice, meaning, and patterns (roth, 1990). 'all cooperative learning methods share the idea that students work together to learn and are responsible for one another's learning as well as their own' (slavin, 1991, p. 73). working in peer groups helps students refine their knowledge through argumentation, structured controversy, and the sharing and testing of ideas and perspectives. additionally, students are more willing to take on the extra risk required to tackle complex, ill-structured, authentic problems when they have the support of others in the co-operative group. co-operative learning and problem-solving groups also address students' needs for scaffolding during unfamiliar learning and problemsolving activities; therefore, with the support of others in the group, students are more likely to achieve goals they may not have been able to meet on their own. constructivists argue that co-operative learning and problem-solving groups facilitate generative learning. some of the generative activities that students engage in co-operative groups include (brown et al, 1989): • collective problem-solving. groups give rise synergistically to insights and solutions that would not come about individually. • displaying multiple roles. group participation means that the members must understand many different roles. they also may play different roles within the group to gain additional insights. • confronting ineffective strategies and misconceptions. teachers do not have enough time to hear what students are thinking or how they are thinking. groups draw out, confront, and discuss both misconceptions and ineffective strategies. • providing collaborative work skills. students learn to work together in a give-and-take interaction rather than just dividing the workload. research indicates that co-operative learning, when implemented properly, is highly successful. slavin (1991) provides the following four summary statements regarding research findings in co-operative learning: • successful co-operative-learning strategies always incorporate the two key elements of group goals and individual accountability. • when both group goals and individual accountability are used, achievement effects are consistently positive. slavin's review found that 37 of 44 experimental/control comparisons of at least four weeks' duration found significantly positive effects for cooperative group methods with none favouring traditional methods. • positive achievement effects are present to about the same degree across all us grade levels (2-12), in all major subjects, and in urban, rural, and suburban schools. effects are equally positive for high, average, and low achievers. • positive effects of co-operative learning are consistently found on such diverse 25 r. scott grabinger and joanna c dunlap rich environments for active learning: a definition outcomes as self-esteem, inter-group relations, acceptance of academically handicapped students, attitudes toward school, and ability to work co-operatively. real strategy: problem-based learning problem-based learning (pbl) embodies the definition of a rich environment for active learning, and incorporates all attributes previously described. pbl is 'the learning that results from the process of working toward the understanding or resolution of a problem' (barrows and tamblyn, 1980, p. 18). pbl found initial acceptance in the medical field, and has grown to become a major learning system for a number of medical, law, and business schools. it is also being adapted for use in secondary schools and corporate training environments (dunlap, 1995). pbl reflects the real attribute that knowledge is constructed rather than received, for it is based on the assumption that knowledge arises from work with an authentic problem (coltrane, 1993). benor (1984) states that: problem-based learning in the context of medical education means self-directed study by of [sic] learners who seek out information pertinent to either a real-life or a simulated problem. the students have to understand the problem to the extent that its constituents can be identified and defined. the learners have then to collect, integrate, synthesize and apply this information to the given problem, using strategies that will yield a solution, (p. 49) how does pbl work? savery and duffy (1994) describe four characteristics of pbl. first, pbl environments include the learning goals of realistic problem-solving behaviour, self-directed learning, content knowledge acquisition, and the development of metacognitive skills. second, savery and duffy state that pbls are based on problems that are generated because they raise relevant concepts and principles that are authentic. problems must be authentic because it is difficult to create artificial problems that maintain the complexity and dimensions of actual problems. recall that we encountered the need for complexity in the real strategies of anchored instruction and cognitive flexibility theory. realistic problems also have a motivational effect. they tend to engage learners more because they want to know the outcome of the problem. when the learning context is similar to the situation in which the learning is to be applied, learning transfer is more likely to occur. therefore, we see the continual reference to the necessity for transfer in pbl. third, the actual presentation of the problem is a critical component of pbls. problems are encountered before any preparation or study has occurred (barrows, 1980). the problem must be presented in a realistic way that encourages students to adopt and take ownership for the problem (barrows and tamblyn, 1980; savery and duffy, 1994). work on the problem begins with activating prior knowledge to enable students to understand the structure of the new information. learners state what they already know about the problem domain. they use that knowledge to form hypotheses or ideas about potential solutions. we also see this principle emphasized in intentional learning, because students must ask themselves what they know about a subject before creating learning plans. the data must be embedded in the problem presentation (refer back to the example of anchored instruction) but must not highlight the critical factors in the case. students must 26 alt-) volume 3 number 2 make their own decisions about what is critical because that is cognitively authentic it reflects actual job performance (savery and duffy, 1994). fourth, the facilitator has a crucial role comparable to the roles described in anchored instruction and reciprocal teaching. the facilitator interacts with the students at a metacognitive level, helping them ask the right questions and monitor their own progress. facilitators avoid expressing opinion, giving information, or leading to a correct answer. their role is to challenge the students, and help them reflect on what they are learning. (savery and duffy, 1994). co-operative learning is a critical component of pbl for it is used from the beginning to the end of the problem-solution process. members of the group listen to the problem presentation together. they analyse the problem's components, recall what they know, hypothesize, consider possible resources, and choose directions to go. they test and help each other. they work together on the solutions and reach consensus on final actions. the entire process from beginning to end is co-operative. co-operative learning is also used for its motivational factors. problem discussion also increases motivation by gaining and maintaining student interest (attention), by relating the learning to student needs or helping students to meet personal goals (relevance), by providing conditions conducive to student success (confidence), and through the motivation provided by that mastery of the task(s) (satisfaction) (coltrane, 1993, pp. 12-13). pbl is the epitome of the real constructive learning process. students work with problems in a manner that fosters reasoning and knowledge application appropriate to their levels of learning. in the process of working on the problem and with their peers, students identify areas of learning to guide their own individualized study. the skills and knowledge acquired by this study are applied back to the problem to evaluate the effectiveness of learning and to reinforce learning. the learning that has occurred in work with the problem and in individualized study is summarized and integrated into the student's existing knowledge structure. conclusion times have changed. people now need to be able to think flexibly and-creatively, solve problems, and make decisions within complex, ill-structured environments. given these changes, our assumptions on learning and education are out-dated, forcing us to modify our assumptions based on current theoretical views of learning. these new assumptions, supported by the theories and philosophies of constructivism, require different instructional methods, techniques, and strategies than have been conventionally used in classroom settings. reflecting a constructivist view of learning, reals provide a way for us to address these new assumptions in order to meet the educational demands of a changing society. we have looked at each of the five main attributes of reals that support the goals of constructivism: (1) student responsibility and initiative, (2) generative learning activities, (3) authentic learning contexts, (4) authentic assessment strategies, and (5) co-operative support. each real attribute builds upon and uses the others. none of the attributes are 27 r scott grabinger and joanna c dunlap rich environments for active learning: a definition mutually exclusive, and no one attribute is more important than another; you cannot implement one feature without incorporating the others to some degree. in effect, the attributes of reals mirror the comprehensive, integrated, and holistic nature of reals. the characteristics are symbiotic, with one feature both supporting and needing the others to create a successful rich environment for active learning. references adams, l., kasserman, j., yearwood, a., perfetto, g., bransford, j. and franks, j. (1988), 'the effects of facts versus problem-oriented acquisition', memory and cognition, 16, 167-75. agnew, p., kellerman, a., and meyer, j. (1992), 'constructing multimedia: solutions for education', paper presented at the 34th annual international conference of the association for the development of computer-based instructional systems, norfolk va. albanese, m.a. and mitchell, s. (1993), 'problem-based learning: a review of literature on its outcomes and implementation issues', academic medicine, 68 (1), 52-81. allinson, l. and hammond, n. (1990), 'learning support environments: rationale and evaluation', computers in education, 15 (1), 137-43. ambruso, d. and the transfusion medicine group. (1994), transfusion medicine modules [computer programs], denver co, the bonfils blood center. american association for the advancement of science (1989), a project 2061 report on literacy goals in science, mathematics, and technology, washington dc, aaas. barrows, h.s. (1985), how to design a problem-based curriculum for the preclinical years, new york, springer-verlag. barrows, h.s. and tamblyn, r.m. (1980), problem-based learning: an approach to medical education, new york, springer publishing company. bednar, a.k., cunningham, d., duffy, t.m. and perry, j.d. (1991), 'theory into practice: how do we link?' in anglin, g.j. (ed), instructional technology: past, present, and future (pp. 88-101), englewood co, libraries unlimited. benor, d.e. (1984), 'an alternative, non-brunerian approach to problem-based learning' in schmidt, h.g. and d. voider, m.l. (eds), tutorials in problem-based learning: new directions in training for the health professions (pp. 48-58), assen/maastricht, van gor cum. bereiter, c. and scardamalia, m. (1989), 'intentional learning as a goal of instruction' in resnick, l.b. (ed), knowing, learning, and instruction: essays in honor of robert glaser (pp. 361-92), hillsdale nj, lawrence erlbaum. berryman, s.e. (1991), designing effective learning environments: cognitive apprenticeship models, eric document 337 689, 1-5. biggs, j.b. (1985), 'the role of metalearning in study processes', british journal of educational psychology, 55, 185-212. blakey, e. and spence, s. (1990), developing metacognition, eric document 327 218, 1-4. 28 alt-j volume 3 number 2 borsook, t.k. and higginbotham-wheat, n. (1992), 'the psychology of hypermedia: a conceptual framework for r and d', paper presented at the 1992 national convention of the association for educational communications and technology, washington dc. bransford, j.d., sherwood, r.d., hasselbring, t.s., kinzer, c.k. and williams, s.m. (1990), 'anchored instruction: why we need it and how technology can help' in nix, d. and spiro, r. (eds), cognition, education, and multimedia: exploring ideas in high technology (pp. 115-41), hillsdale nj, lawrence erlbaum. bransford, j. and vye, n.j. (1989), 'a perspective on cognitive research and its implications for instruction' in resnick, l. and klopfer, l.e. (eds), toward the thinking curriculum: current cognitive research (pp. 173-205), alexandria va, ascd. bransford, j., goldman, s.r. and vye, n.j. (1991), 'making a difference in peoples' abilities to think: reflections on a decade of work and some hopes for the future' in okagaki, l. and sternberg, r.j. (eds), directors of development: influences on children (pp. 147-80), hillsdale nj, lawrence erlbaum. brown, a.l. (1978), 'knowing when, where, and how to remember: a problem of metacognition' in glaser, r. (ed), advances in instructional psychology, hillsdale nj, lawrence erlbaum. brown, a.l., bransford, j.d., ferrara, r.a. and campione, j.c. (1983), 'learning, remembering, and understanding' in flavell, j.h. and markman, e.m. (eds), vol. 3, handbook of child psychology: cognitive development (pp. 177-266), new york, wiley. brown, j.s., collins, a. and duguid, p. (1989), 'situated cognition and the culture of learning', educational researcher, january-february, 32-42. bruner, j.s. (1961), 'the act of discovery', harvard educational review, 21-32. butterfield, e. and nelson, g. (1989), 'theory and practice of teaching for transfer', educational technology research and development, 37 (3), 5-38. carver, s.m., leherer, r., connell, t. and erickson, j. (1992), 'learning by hypermedia design: issues of assessment and implementation', educational psychologist, 27 (3), 385404. clark, r.e. (1994), 'media will never influence learning', educational technology research and development, 42 (2), 21-9. clark, r.e. and voogel, a. (1985). 'transfer of training principles for instructional design', educational communication and technology journal, 33 (2), 113-25. clement, j. (1982). 'algebra word problem solutions: thought processes underlying a common misconception', journal of research in mathematics education, 13, 16-30. ctgv (cognition and technology group at vanderbilt) (1990), 'anchored instruction and its relationship to situated cognition', educational researcher, 19 (6), 2-10. ctgv (cognition and technology group at vanderbilt) (1991), 'technology and the design of generative learning environments', educational technology, 31, 34-40. ctgv (cognition and technology group at vanderbilt) (1992a), 'anchored instruction 29 r. scott grabinger and joanna c dunlap rich environments for active learning: a definition in science and mathematics: theoretical basis, developmental projects, and initial research findings' in duschl, r.a. and hamilton, r.j. (eds), philosophy of science, cognitive psychology, and educational theory and practice (pp. 244-73), new york, suny press. ctgv (cognition and technology group at vanderbilt) (1992b), the jasper series as an example of anchored instruction: theory, program description, and assessment data'. educational psychologist, 27 (3), 291-315. ctgv (cognition and technology group at vanderbilt) (1993a), 'anchored instruction and situated cognition revisited', educational technology, 13 (3), 52-70. ctgv (cognition and technology group at vanderbilt) (1993b), 'designing learning environments that support thinking' in duffy, t.m, lowyck, j. and jonassen, d.h. (eds), designing environments for constructive learning (pp. 9-36), new york, springer-verlag. ctgv (cognition and technology group at vanderbilt) (1993c), 'integrated media: toward a theoretical framework for utilizing their potential', journal of special education technology, 12 (2), 76-89. ctgv (cognition and technology group at vanderbilt) (1993d), 'the jasper series: theoretical foundations and data on problem solving and transfer' in penner, l.a., batsche, g.m., knoff h.m. and nelson, d.l. (eds), the challenges in mathematics and science education: psychology's response (pp. 113-52), washington dc, american psychological association. collins, a. (1995), 'learning communities', presentation at the annual conference for the american educational research association, san francisco ca, april, 1995. collins, a., brown, j.s. and holum, a. (1991), 'cognitive apprenticeship: making thinking visible', american educator (winter), 6-11, 38-46. coltrane, l. (1993), 'an overview of problem-based learning in medical education' (class paper). dewey, j. (1910), how we think, boston, heath. duffy, t.m. (1992), the strategic teaching framework, bloomington, in. duffy, t.m. (in press), 'strategic teaching framework: an instructional model for learning complex, interactive skills' in dill, c. and romiszowski, a. (eds), encyclopedia of educational technology, englewood nj, educational technology press. duffy, t.m. and bednar, a.k. (1991), 'attempting to come to grips with alternative perspectives', educational technology, 31 (9), 12-15. dunlap, j.c. (1995), 'using constructivist training environments to meet long-term strategic training needs', paper presented at the annual conference of the association for educational communications and technology, anaheim ca. dunlap, j.c. and grabinger, r.s. (1992), 'designing computer-supported intentional learning environments', paper presented at the annual conference of the association for the development of computer-based instructional systems, norfolk va. dunlap, j.c. and grabinger, r.s. (1993), 'computer-supported intentional learning 30 alt-j volume 3 number 2 environments: definition and examples', paper presented at the annual conference of the association for educational communications and technology, new orleans la. farnham-diggory, s. (1992), cognitive processes in education (second edition), new york, harper collins. feuerstein, r. (1979), instrumental enrichment, baltimore md, university park. forman, g. and pufall, p. (eds) (1988), constructivism in the computer age, hillsdale nj, lawrence erlbaum. fosnot, c. (1989), inquiring teachers, inquiring learners: a constructivist approach for teaching, new york, teacher's college press. frederiksen, j.r. and collins, a. (1989), 'a systems approach to educational testing, educational researcher, 18, 27-32. gayeski, d.m. (ed) (1995), designing communication and learning environments, englewood cliffs nj, educational technology publications. goldman, s.r., pellegrino, j.w. and bransford, j. (1994), 'assessing programs that invite thinking' in baker, e. and o'neill, h. f. j. (eds), technology assessment in education and training (pp. x-y), hillsdale nj, lawrence erlbaum. goldman, s.r., petrosino, a., sherwood, r.d., garrison, s., hickey, d., bransford, j.d. and pellegrino, j.w. (1992), 'multimedia environments for enhancing science instruction', paper presented at the nato advanced study institute on psychological and educational foundations of technology-based learning environments, kolymbari, greece. goodman, n. (1984), of mind and other matters, cambridge ma, harvard university press. grabinger, r.s. and dunlap, j.c. (1994a), 'implementing rich environments for active learning: a case study', paper presented at the annual conference of the association for communications and technology, nashville la. grabinger, r.s. and dunlap, j.c. (1994b), 'technology support for rich environments for active learning', paper presented at the annual conference of the association for communications and technology, nashville la. gurney, b. (1989), constructivism and professional development: a stereoscopic view, eric document ed 305 259, 1-28. hannafin, m.j. (1992), 'emerging technologies, isd, and learning environments: critical perspectives', educational technology research and development, 40 (1), 49-63. honebein, p.c., duffy, t.m. and fishman, b.j. (1993), 'constructivism and the design of learning environments: context and authentic activities for learning, in duffy, t.m., lowych, j. and jonassen, d.h. (eds), designing environments for constructive learning, hillsdale nj, lawrence erlbaum. jacobson, m.j. (1994), 'issues in hypertext and hypermedia research: toward a framework for linking theory-to-design', journal of educational multimedia and hypermedia, 3 (2), 141-54. 31 r. scott grabinger and joanno c. dunlop rich environments for active learning a definition jacobson, m.j. and spiro, r.j. (1991), 'hypertext learning environments and cognitive flexibility: characteristics promoting the transfer of complex knowledge', paper presented at the international conference on the learning sciences, evanston il. jacobson, m.j. and spiro, r.j. (1992), 'hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: an empirical investigation', paper presented at the annual meeting of the american educational research association, san francisco ca. jonassen, d.h. (1994b), 'sometimes media influence learning', educational technology research and development, 42 (2). jonassen, d.h. (1994c), 'thinking technology: toward a constructivist design model', educational technology, 34 (3), 34-7. kozma, r. (1994), 'media attributes', educational technology research and development, 42(2). lebow, d. (1993), 'constructivist values for instructional systems design: five principles toward a new mindset, educational technology research and development, 41 (3), 4-16. linn, m.c. (1986), establishing a research base for science education: challenges, trends, and recommendations, report of a national science foundation national conference, university of california. lockhart, r.s., lamon, m. and gick, m.l. (1988), 'conceptual transfer in simple insight problems', memory and cognition, 16, 36-44. lynton, e. (1989), higher education and american competitiveness, national center on education and the economy. lynton, e. and elman, s. (1987), new priorities for the university, san francisco ca, jossey-bass. mann, l. (1979), on the trail of process: a historical perspective on cognitive processes and their training, new york, grune and stratton. minstrell, j.a. (1989), 'teaching science for understanding' in resnick, l.b. and klopfer, l.e. (eds), toward the thinking curriculum: current cognitive research (pp. 129-49), alexandria va, ascd. morris, c.d., bransford, j.d. and franks, j.j. (1979), 'levels of processing versus transfer appropriate processing', journal of verbal learning and verbal behavior, 16, 51933. national council of teachers of mathematics (1989), curriculum and evaluation standards for school mathematics, reston va, nctm. neuman, d. (1993), 'evaluation of the perseus project', paper presented at the 1993 national conference of the association for educational communications and technology, new orleans la. nickerson, r.s. (1988), 'on improving thinking through instruction', review of research in education, 15, 3-57. 32 alt-} volume 3 number 2 palincsar, a.s. (1990), 'providing the context for intentional learning', remedial and special education, 11 (6), 36-39. palincsar, a.s. and klenk, l. (1992), 'fostering literacy learning in supportive contexts', journal of learning disabilities, 25 (4), 211-25. perelman, l.j. (1992), 'living in the gap between old and new: managing transitions', paper presented at the technology in education conference, steamboat springs co. perfetto, b.a., bransford, j.d. and franks, j.j. (1983), 'constraints on access in a problem solving context', memory and cognition, 11, 24-31. perkins, d.n. (1991), 'what constructivism demands of the learner', educational technology, 31 (9), 19-21. resnick, l. (1987), education and learning to think, washington dc, national academy press. resnick, l.b. and klopfer, l.e. (eds) (1989), toward the thinking curriculum: current cognitive research, alexandria va, ascd. ridley, d.s., schutz, p.a., glanz, r.s. and weinstein, c.e. (1992), 'self-regulated learning: the interactive influence of metacognitive awareness and goal-setting', journal of experimental education, 60 (4), 293-306. robertson, w.c. (1990), 'detection of cognitive structure with protocol data: predicting performance on physics transfer problems', cognitive science, 14, 253-80. roth, w.-m. (1990), collaboration and constructivism in the science classroom, eric document 318 631, 1-39. savery, j.r. and duffy, t.m. (1994), 'problem based learning: an instructional model and its constructivist framework', educational technology (august). scardamalia, m. and bereiter, c. (1991), 'higher levels of agency for children in knowledge building: a challenge for the design of new knowledge media', the journal of the learning sciences, 1 (1), 37-68. scardamalia, m., bereiter, c., mclean, r.s., swallow, j. and woodruff, e. (1989), 'computer-supported intentional learning environments', journal of educational computing research, 5 (1), 51-68. schoenfeld, a.h. (1989), 'teaching mathematical thinking .and problem solving' in resnick, l.b. and klopfer, l.e. (eds), toward the thinking curriculum: current cognitive research (pp. 83-103), alexandria va, ascd. segal, j., chipman, s. and glaser, r. (eds) (1985), thinking and learning skills: relating instruction to basic research, vol. 1, hillsdale nj, lawrence erlbaum. shank, r.c. (1990), 'case-based teaching: four experiences in educational software design', interactive learning environments, 1 (4), 231-53. slavin, r.e. (1991), 'synthesis of research on cooperative learning', educational leadership, 48 (5), 71-82. 33 r scott grab/nger and joanna c dunlap rich environments for active learning a definition spiro, r.j., feltovich, p.l., jacobson, m.j. and coulson, r.l. (1991), 'cognitive flexibility, constructivism, and hypertext: random access instruction for advanced knowledge acquisition in ill-structured domains', educational technology, 31 (5), 24-33. tulving, e. and thompson, d.m. (1973), 'encoding specificity and retrieval processes in episodic memory', psychological review, 80, 352-73. von wright, j. (1992), 'reflections on reflection', learning and instruction, 2, 59-68. vygotsky, l.s. (1978), mind in society, cambridge ma, harvard university press. weinstein, c.e., goetz, e.t. and alexander, p.a. (1988), learning and study strategies: issues in assessment, instruction, and evaluation, san diego ca, academic press. wheatley, m. (1992), leadership and the new science, san francisco ca, berrettkoehler. whitehead, a.n. (1929), the aims of education and other essays, new york, macmillan. wiggins, g. (1989), 'a true test: toward more authentic and equitable assessment', phi delta kappan, 70, 703-13. williams, m.d. and dodge, b.j. (1992), 'tracking and analyzing learner-computer interaction', paper presented at the 1992 national conference of the association for educational communications and technology, new orleans la. williams, s.m. (1992), 'putting case-based instruction into context: examples from legal and medical education', the journal of the learning sciences, 2 (4), 367-427. 34 reviews edited by philip barker web-teaching a guide to interactive teaching for the world-wide web by david w. brooks, new york: plenum, 1997. isbn: 0-306-45552-8. paperback, 214 pages. $30. as a result of various paradigm shifts, the world wide web (the web) is changing both what we teach and how we teach it. consequently, from the point of view of distributing resources, we are now experiencing a significant change of emphasis from instructor push towards student pull. web-teaching is a book about teaching, especially interactive teaching, using the web as a communications medium. it deals with the kinds of hardware, software and networks commonly used on the web to deliver and support instruction and learning. overall, it has two basic thrusts: first, it gives descriptions of what is possible on the web; second, it identifies instructional strategies that are likely to be effective. the book is organized into fourteen chapters, a reference section, a glossary, a software list and a set of urls that are cited in various parts of the text. chapter 1 offers a basic introduction, both to the contents of the book and to web technology. in chapter 2 the author reviews • relevant dimensions of learning theory and discusses potentially useful research on teaching and learning. chapters 3 to 7 provide an overview of what is currently possible on the web. these chapters are intended for those readers who are starting out without much knowledge about technology beyond wordprocessing and email. the topics covered include multimedia, web-ready materials, and using images, movies and other media. unfortunately, using the web for teaching and learning does not automatically guard against the creation of purely passive learning resources. therefore, those wishing to use this type of technology to best advantage need to become aware of strategies for introducing interactivity. chapters 8 to 10 address some of the major issues involved in developing interactive learning. chapter 8 focuses on strategies for supporting both student-student and studentteacher discussion and debate via the web. chapter 9 describes the use of html (hypertext markup language) forms for the support of automated interactive assessment, while chapter 10 deals with mechanisms for the promotion of self-regulated learning (or studentmanaged learning). in the following three chapters (11 to 13), the author focuses on various management topics related to the creation and maintenance of web sites. as well as treating hardware and software issues, in these chapters he discusses the use of cd-rom, intranets for electronic course delivery, access control and security, and protection of a web site. the final chapter deals with ways to use web software to create multimedia lectures for in-class use, and provides many useful suggestions for setting up multimedia rooms. according to the author, the intended audience for the book includes teachers and trainers in 63 reviews high school, college, graduate school, in-service education, professional development, adult education and industry. one would therefore expect it not to be over-technical, and indeed this is the case. but it contains a lot of useful information, and is easily readable. in fact, its only (minor) shortcoming from the point of view of windows users is that all the discussion and examples relate to the apple macintosh. despite this, i am sure that windows users will find the techniques and principles of interest. philip barker, university of teesside designing competence-based training by shirley fletcher, london: kogan page, second edition, 1997. isbn: 0-7494-2196-7. paperback, 86 pages. £15.99. this book makes an excellent, provocative addition to kogan page's practical trainer series. it is aimed at trainers who 'want to take a fresh approach and move away from traditional forms of training and development', and there is much to gain from it, for both the novice and the already-committed. for trainers and training manager it identifies opportunities to focus programmes on training outcomes rather than the delivery of content. it also offers practical frameworks to develop programmes using a functional analysis approach rather than traditional task analyses. the functional approach concentrates on 'identifying the purpose of the organisation and its occupational roles together with the standards expected of individual job holders'. the results are that the responsibility for training is moved onto the individual trainee. implications are that the training has to take into account learning styles, learning strategies and individualized learning programmes. one of the strengths of outcome-based training and development is that it is possible to employ as much or as little as you want at any time to augment existing courses or change whole programmes. this book is a first-rate example of such a flexible approach, making individual processes simple enough to understand and apply but treating them with sufficiently thoroughness to facilitate the development of whole frameworks. it is essential reading for anyone who is planning to adopt competencebased training techniques in an organization. it is also excellent reading for those who wish to find out about such training, whether they require in-depth procedures for implementation 64 or an overview of the whole topic. the book is carefully structured, guiding the reader through five chapters beginning with 'what is competence-based training?', continuing with chapters addressing standards and training design, and finishing with 'assessment, evaluation and accreditation'. each chapter is headed by what the book terms a 'summary', but which is really a list of expected outcomes for the chapter. a far better summary is provided by the 'review' at the end of every chapter. i can highly recommend the excellent foreword and preface. some of the early chapters pack an astonishing amount of information into just a few pages: several scans might be required to assimilate some of this material, particularly if you are unfamiliar with the jargon (there is a glossary at the end of the book). chapters 1 and 2 provide solid background information, essential to the understanding of the rest of the book. chapter 2 includes some interesting comparisons between us and european systems, and a brief discussion of credit and credit accumulation in the context of the qualification framework. chapter 3 introduces some useful examples of real competence-based systems, and devotes much space to explaining how to carry out a functional analysis to arrive at your own competence standards. such detail makes heavy going for newcomers; although a certain amount of explanation is necessary, more • emphasis might perhaps have been placed on applying the existing, nationally recognized standards now available for many occupational areas. some large organizations (for instance, bt) use these national standards but augment them with their own and therefore make extensive use of functional analyses. chapter 3 is rounded off with two case studies featuring whitbread (cellar service) and bt. these case studies are invaluable, and i was disappointed not to see others used elsewhere. throughout the text there are numerous bulleted lists of activities and topics for consideration but only in chapters 2 and 4 are proper checklists supplied. the provision of such checklists should have been more consistent as there are very many useful activities buried in the text as bullet points. if you are embarking on the development of competency-based training systems, i would recommend the extraction of all these lists in their appropriate context. chapter 4 contains some examples of the sorts aa-j volume 5 number 3 of paperwork that can be used to develop and support competence-based training systems. in particular, the 'training design' and 'training method' matrices looked useful, but i confess to being a bit confused about their exact implementation. as with chapter 4, chapter 5 on assessment, evaluation and accreditation, could have usefully been enhanced with more examples of the resources which can be employed to enable these activities. although the final impression might be one of burgeoning paperwork, the underpinning structure provided by the use of (for example) planning and evaluation forms is critical to competence-based training systems. the discussion of evaluation in chapter 5 is so critical to the whole ethos of competence-based systems that i was surprised that the chapter was so short. evaluation is also missing completely from the key figure 1.1 in chapter 1, 'model for competence-based training design'. this is rather unfortunate since, without evaluation, there is no test of competence and therefore no complete justification of the system! as the author herself states, training is only effective if it produces results. evaluation against the expected outcomes of the training is the only way to measure results. this demands a clear understanding of the expected outcomes (why are you doing the training, how is it going to be applied) and a workable method of evaluation. once the standards are clear and recording mechanisms are in place, assessment and accreditation can be offered as an integrated option. the nvq system in the uk can work well within this context, despite some bad pr. there is much interest in competence-based systems, not necessarily under that name, particularly in the higher-education sector with the recent ncihe report recommending the integration into academic curricula of key skills and planning and recording of achievement. from page 1 this book helps dispel some potent myths about competence-based training which may help propagate its use not just in a work environment but also in higher education at all levels. give betts, university of exeter adult learning: a reader edited by peter sutherland, london: kogan page, 1996. isbn: 07494-1971-7. hardback, 208 pages. £35.00. the stated aim of this collection, edited by peter sutherland who is a specialist in adult learning and cognitive development, is to explore approaches to teaching and learning appropriate to traditional and mature students in higher education. the contributions are arranged into six sections. section i (on cognitive processes) has three chapters. jack mezirow reviews contemporary paradigms of learning and the current debate on the nature and meaning of learning. gillian boulton-lewis explores human information processing and aspects of memory, cognitive development with age and the implication for adult learning. barry dart considers metacognition and the use of metacognitive strategies by adult learners in higher education, stressing the importance of reflecting upon, understanding and controlling learning in the context of complex cognitive activities and selfdirectedness. section ii (on learning and education) has four chapters. paul hettich reviews four epistemological theories of intellectual growth in adults. ronald r schmeck examines student individuality and orientation to work, and its influence on learning tactics, pedagogic and andragogic approaches to teaching. glyn owens looks at behaviourist approaches to adult learning. peter sutherland (the editor) argues that the concepts of constructivism ('constructing a version of reality from [...] experience') and experiential learning ('constructivism in an adult context') can be blended into a fruitful union, particularly where the teacher adopts andragogic strategies with students, negotiating assignment topics based on their prior experiences so as to enhance and extend established constructs. section iii (on adult learning in small-group situations) contains only two contributions. ian mowatt and gerda siann promote the benefits of learning in small groups, advocating the consideration of knowledge, ability and skills of the group participants, the purpose to which the desired learning is directed, and the extent to which the needs of the individual members are met by the group dynamics, and emphasis on the role of the group as an instructional tool not as a recreational activity in determining the quality of the learning outcomes achieved. keith topping considers the increased interest in peertutoring in post-16 education and its effectiveness as an alternative to traditional curriculum delivery, and therefore as a strategy 65 reviews for 'doing more with less'. peer-tutoring is valuable tool in the repertoire of adult educators, promoting learner-managed learning and learner empowerment. topping stresses the need for caution before seizing peer-tutoring as a universal, undifferentiated and instant panacea to the ills of higher education. section iv (on the context of education) also has only two contributions. hitendra pillay identifies employer expectations and requisite skills for effective performance in the workplace, and in particular the need for employees to be crossskilled, high on up-skilling and able to engage in high-order thinking, a 'cocktail of abilities [...] known as multi-skilling' which can be developed through training programmes and which 'explicitly illustrate the connection of scientific principles and mathematical knowledge to work practice'. loraine blaxter, christina hughes and malcolm tight present as 'analytical vignettes' the results of two projects, one looking at parttime degree students, the at adults in six medium-to-large organisations. three broad conclusions are reached: (i) that while many adults reject education in their early youth, full-time education later assumes a more important, even central, role; (ii) that the links between education and other aspects of life (work, family, leisure) are many and varied, and change over time, and while these links are influenced by others (employers, partners), the real meaning comes from the individual; (iii) that the success of cultivating a learning culture must be judged by the extent to which individual perceptions are changed. section v (on the particular context of the mature student in higher education) has four chapters. elizabeth beaty and colleagues explore interviews with female open university students against a hierarchy of six different views of learning which culminate in personal transformation to achieve increased competence and capability, confidence through the adoption of multiple perspectives and personal control to 'take hold of life and make it go your way' mezirow's emancipation (section i). there follow three papers with varied approaches to the issue of age as the dominant variable in the mythology of perceived differences between mature and traditional students in higher education. john richardson rebuts the prevalent stereotype of mature students as inherently problematic on three grounds: that there is no evidence of a deficit of study skills in mature students; that the deep approaches to learning of mature students are in fact compatible with the mission of higher education; that there is no 66 evidence to support the idea of age-related deficits in the intellectual capacity requisite for higher education. james hartley and mark trueman reinforce the view that mature students are as variable a commodity as any other students with differing goals for participation and social situations, all of which impact on performance. stephen newstead and colleagues identify three motivational categories to study (stop-gap, means to an end, personal development) which dominate age in influencing differences between mature and traditional students. section vi (on implications for teaching) contains just one contribution by peter sutherland himself. drawing the collection together, sutherland uses the dichotomous model of teacher as pedagogue and andragogue to explore the previous contributions in terms of their potential to develop metacognitive skills and deep learning, and seeks to identify techniques which promote reflection learning diaries, peer tutoring, etc. the mission of higher education is to develop deep approaches to learning, and requires the development of metacognitive strategies which mature learners participating for personal development are often able to demonstrate more easily than their traditional-age counterparts. higher education today is played out against rising staff-student ratios and a decreasing unit of resource which mitigate against a focus on the individual student. this book seeks to promote understanding and awareness of some of the potential strategies by which educators of adults can seek to facilitate the connection between theory and practice that promotes adult learning and multi-skilling. it is a reader, not a primer, and will, in part, be really useful only for those who have already started to consider the benefits and issues of interacting with mature students and who, even then, need to be equipped with a fairly powerful dictionary. adult learning is, however, as its dust cover suggests, 'an internationally accessible collection', and will undoubtedly benefit all those who teach adults. eleri jones, university of wales institute, cardiff design for multimedia learning by tom boyle, london: prentice-hall, 1997. isbn: 0-13242215-8. hardback, 240 pages. £29.95. design for multimedia learning covers all aspects of the area from trends in multimedia alt-j volume 5 number 3 learning theory to conceptual and presentation design, along with information about multimedia-resources project development and evaluation. the book comes somewhere between a 'how to' technical manual and an overview of current academic thought in the field. in the area of multimedia design, a broad cover of subjects is inevitable, and getting the balance right between depth and breadth cannot have been easy, but this book manages it. its four main parts addresses the principal elements of the subject in an informed manner. part 1 charts the background to the 'multimedia learning revolution', explaining the reasons why multimedia learning is a growth area. developments in student-centred learning are cited, along with software developments such as the web and affordable multimedia hardware, as reasons for this. part 2 looks at strategic approaches to educational multimedia. a critique of current constructivist learning theory is made, and a history of its roots in the procedural-based approach of cai (computer-aided instruction). the author asks if this new approach is 'too woolly', and provides an interesting discussion about the current conceptual design of multimedia learning environments, citing the establishment of context in learning environments as crucial for learning advances. illuminating points are made about using paradigms from film and linguistics for multimedia. in film, for example, 'communication spaces' are linked in the 'temporal domain', and similar links can be used in multimedia. the complexity of interactive media, and the importance of maintaining coherence through user-centred design, are recognized. the chapter dealing with 'design action potential' approaches the structuring of content, and is the real heart of the book. it contains pertinent observations on what is perhaps the hardest part of constructing a multimedia learning environment: the structure, especially for the novice designer. different ways of thinking about structure in terms of hierarchy, networks and competency are explored in some depth, with helpful diagrams. however, some of the discussion is made less elegant by the unfortunate term 'design action potential'. action potential has a specific meaning in biology; in this context it has a very different meaning of 'potential for action'. elsewhere, the author describes this as the available 'design options', which is perhaps a more helpful phrase. in part 3, a holistic perspective on presentation design recognizes the close interrelation of and the structuring and conceptual design of learning environments. comments about the principles of composition in terms of tone, colour and balance are made, along with the importance of creating a 'flow' of attention for the user. the description of multimedia toolsets, media capture, editing and authoring tools will date much more rapidly than other parts of the book. it is, for instance, misleading for a new developer to read that 8 mb of ram is specified for a development machine (the specification cited is that of 1995) however, there is useful background information about the nature and use of sound and digital video which should be universally applicable. part 4 focuses on the area in which many multimedia projects fail: their management and the evaluation cycle. it outlines the iterative prototyping and evaluation process, and uses the example of the dynamic systems development method as a flexible method of project development. this involves the analysis and specification of objectives, a vital but difficult part of the process. strategic views of evaluation are covered, there are notes on how to go about developing questionnaires, and observation methods are described. delivery scenarios are explored in terms of the total environment that the software will be used in, and the importance of good packaging, installation and documentation are summarized. this book, covering every aspect that a designer should consider, will be useful for anyone new to developing multimedia learning, although, the focus unfortunately tends to waver in places: some things are skirted over, and sometimes aspects that would be better placed in a technical manual are gone into in detail. on occasion, too, the figures and screen shots used to illustrate points are fairly dry, and sometimes even misleading. for example, in the chapter on presentation design, the importance of composition on the screen is discussed, along with important notions of directing and maintaining attention for the student. the importance of balance and the use of a screen grid to create consistency is addressed and a figure showing blocked out areas for two identical work areas on either side is used. splitting the picture in half is visually difficult unless elements are balanced with stronger elements on one side or another. the author does mention balancing the screen using 67 reviews pictures, but the example simplifies the notion of composition too much and could confuse the novice reader. the subsequent example from the dove program is better, with its use of real images. however, how the different elements interact through colour, movement, tone and proportion is not properly explained. again, in the section on use of animation, the text accurately describes the importance of using animation in context to enhance the learning objectives, but the example given is poorly drawn and badly laid out. in fact, with the exception of the liverpool john moores cytofocus program and some other examples, the illustrations in the book do not fully sample the now growing number of high-quality materials available in the education and commercial arena. surely a book about good design should be an exemplar, and use only the best examples to inspire and illuminate this subject for multimedia designers old and new? but perhaps this is a harsh criticism of what is, on the whole, a very useful book, and one which will be a welcome addition to a learning technologist's bookshelf. emma candy, university of edinburgh researching into learning resources in colleges and universities by chris higgins, judy reading and paul taylor, london: kogan page, 1996. isbn: 0-7494-1771-4, paperback, 156 pages. £14.99. this book is part of the practical research series edited by kate ashcroft, aimed at teachers in further and higher education who are starting research into various aspects of teaching and learning. it is designed to support both 'reflective practitioners', taking a critical look at their own professional work, and those evaluating and researching education more generally. the books in this series, according to the editor's introduction, combine practical advice in research methods and design along with suggestions for likely research topics. this means that this relatively slim volume combines some aspects of a research methods manual with an introduction to some of the educational literature pertinent to the use of learning resources. the book starts with a lengthy introduction by the series editor in which she outlines the philosophy of the series. this is based on the idea of 'insider research' researching the activities and institution in which one works (or is otherwise involved). while this brings many 68 advantages, such as good background knowledge and easy access, it also demands a certain distancing, an open-mindedness, and commitment and responsibility from the researcher. often such research involves just one researcher, part-time, and the book essentially addresses the needs of such people. as ashcroft points out, this commonly means the research is small-scale, and even 'quick and dirty', and it is important not to claim spurious objectivity for results. on the other hand, it can be treated as action research, in which results are fed back rapidly into (one's own) practice, and the research activity adjusted to accommodate the changes. the following six chapters are the meat of the book. they cover the resource environment, the institutional context, the link between teaching and resources, the student as stakeholder, and evaluation of the resource environment. as can be seen from this list, the contents are examined in a framework derived from the subjects being researched, rather than as is typical in research methods books around specific methods and strategies. throughout these chapters, there is copious use of boxes and sidebars which contain many practical suggestions for research and evaluation activities. in chapter 2, after introducing the concepts of resource-based, open, distance and flexible learning, the authors discuss a range of resources based on new communication and information technologies (c&it), along with more traditional forms. the former includes the internet, multimedia and conferencing; the latter, reading lists, laboratories, audio-visual materials and libraries. these are dealt with only in outline, but as in other chapters, there is good reference to further reading. chapter 3 examines the wider, institutional framework within which resources are delivered. in quick succession, the authors consider structure and culture in institutions and their responses to recent changes. they present three methods of analysis based on systems analysis, open systems and swot, and then discuss support staff, staff training and collaboration between institutions. in contrast with the preceding chapter, chapter 4, on resources and teaching issues, takes a more theoretical tack. it begins with a typology of cal approaches which are explained by the following sections on various theories of learning, such as behaviourism, cognition and alt-} volume 5 number 3 learning styles. in consequence, the suggested research tasks are much better grounded and address real conceptual issues. chapter 5 turns attention to the student's view of things and, especially, to their needs and the availability of resources. groups that receive particular attention are part-time students, distance learners and disabled students. this chapter contains the only (brief) reference to cost-effectiveness. this is a pity, given the emphasis now placed on the issue by many institutions and funding bodies, not least the teaching and learning technology programme phase iii. the last substantive chapter examines the more general resource environment. rather oddly, this starts with a discussion of approaches to learning (deep, surface, etc.) which would seem to belong with the discussion of theories of learning in chapter 4. there then follows a useful discussion of teaching strategies including reference to laurillard's conversational approach. the book ends with another substantial chapter by the series editor. this covers the issues of writing and submitting research for publication. the chapter contains a very useful listing of educational publishers, educational journals (including, of course, alt-j) and other education-related journals. does the book succeed in its attempt to review of research approaches and to consider the substantive issues resource-based learning raises? in the end, my feeling is that it does a better job of the latter than the former. although the authors make plenty of reference in the annotated reading lists, and in many of the boxes of research tasks, to further reading in research methods, i think that too often they give the impression that research is mere observation or data collection. there is little reference to careful analysis or to issues of reliability, validity, generalizability, and so on. this is in marked contrast with the last chapter which gives the impression that reflective practitioners, who seem to be the readers addressed by most of the book, should be getting their research published in academic journals. these journals are likely to demand more than the mere stories which many of the research task boxes refer to. perhaps, though, it is the last chapter that is out of line. most of the book, in my view, forms an excellent introduction to any teacher or lecturer reflecting on their practice for the first time, especially if they have no background in educational or learning theory. in fact, if the proposal of the dealing report for teacher training in higher education is acted upon, the book could be very timely in meeting the needs of teachers who might have to produce reflective portfolios for assessment. graham r. gibbs, university of huddersfield facing up to radical changes in universities and colleges by steve armstrong, gail thompson and sally brown, london: kogan page, 1997. isbn: 0-7494-2129-0, paperback, 186 pages. £18.99. the past 10 to 15 years have seen significant changes in higher education, and there have been many factors which have had (and continue to have) an impact on teaching and learning. factors such as the level of national investment in education in real terms, huge growth in student numbers, the change in the type of student entering higher education, material being taught, unitization programmes, and the technology available for teaching, have all had an effect on the nature of teaching and learning. in fact, the change in the nature of higher education has meant that a veritable revolution has been taking place. the book is published as part of the staff and educational development association's (seda) development series, and it grasps the radical nature of the changes in higher education and tackles clearly and concisely issues very close to the heart of anyone working within it. there are problems and issues examined and discussed which will ring true with practitioners in higher education all over the country. one of the main benefits of texts like this one is the realization for the reader that he or she is not alone. that in itself is reassuring, and gives the book great value. the book, organized into four main sections, has 18 chapters, each written by a different author, and the range of authors gives the book a valuable breadth of experiences and opinions. taken as a whole, it provides an interesting spectrum of perspectives, all moving towards the central theme, namely that since there is a huge and significant change taking place in higher education, practitioners have much work to do in order to manage it. the book is clearly written, and the notes and reference sections at the end of each chapter are very useful as indicators for further study. 69 reviews after a very perceptive introduction from gail thompson, who discussed the current situation in higher education in the uk, the first section (chapters 2 to 5) deals with the use of technology to support teaching and learning. it examines many of the concepts and issues raised in the use of information technology as a means of dealing with the changes in higher education. the blend of chapters very nicely links practical examples with educational theory. the theme is positive throughout, and perhaps examination from another point of view (the dangers of information technology) would have given the section even more balance. the second section (chapters 6 to 10) deal with the strategies and policies in use in higher education. again, there is an interesting mix of practical example and theoretical discussion, and chapter 10 introduces an interesting perspective: that of education managers. the third section (chapters 11 to 14) concentrates on dealing with the changes in the student population. this is a fascinating topic of discussion, where a tremendous amount of work will need to be carried out in the next few years if higher education is to prosper. the contributors to this section offer an insight into many of the current student issues. the concept of an educator moving from an expert to a facilitator, tackled in chapter 14 is particularly interesting. in chapters 15 to 17 (most of the final section) there is a discussion of some of the measures available to lecturers in coping with the changes in education. these are valuable insights based on experience. the concept that the radical changes in education will have to be met by radical changes in approach to teaching and learning is a message which should become very clear in the next few years. chapter 18, by sally brown, nicely pulls the whole book together in an interesting and informative manner. overall, this book is an excellent addition to the growing library in the staff and educational development series. it is readable, and tackles many of the issues and concerns shared by practitioners in uk higher education. the contributors have used their experience to discuss and analyse the current position of higher education, and to develop that understanding into a means of coping positively with a future fraught with hurdles and difficulties. it is a text everyone working in higher education should read if only to realize that the problems they are facing are faced by others, and that 70 people are putting forward ideas which will allow us to cope with radical change. alastair irons, university of northumbria at newcastle technology-based training a comprehensive guide to choosing, managing, implementing and developing new technologies in training by serge ravet and maureen layte, london: kogan page, 1997. isbn: 0-7494-1975-x. hardback, 306 pages. £35.00. 'technology' and 'training' are words which are used in many and varied contexts. therefore, the task of writing a single text to synthesize current information about the uses of technology in training is enormous, and brings to this book both its principal strength and its principal weakness. as a broad introduction to technology-based training (tbt), the coverage tackled by the authors is commendable, and there are plenty of pointers to examples and further information. there need to be. anyone seeking depth of knowledge may feel disappointed and frustrated with the lack of detail. the book has four sections, each divided into manageable chunks, with good use of tabulated summaries and illustrated with case studies or examples easily identified by their boxed presentation. but exactly what the authors intend or mean by training is not specified; the emphasis seems to be on training within a work context, but many examples and case studies are drawn from formal educational settings. the first section looks at the context of technology and the impact it has on everyday life, and at its advantages as a training medium, the list presented being familiar and unadventurous. the term 'computence' is introduced to mean the competence of the computer to execute certain tasks. section 2 discusses the merits of using multimedia as a training medium. the internet is described, and chapter 5 includes a full-page introduction to hypertext aimed at novices, but by chapter 7 the reader is expected to understand object-orientated programming, java and virtual reality, each of which are discussed with little or no explanation beyond the terms themselves. chapter 8, however, is most welcome, addressing issues so often omitted from texts in this area: using technology to empower and facilitate access to learning for individuals with disability. an-j volume 5 number 3 the remaining two sections look at the issues surrounding the design and development of computer-based training packages. important design factors are identified, and the skills needed among all members of a design team are clearly presented. legal considerations, such as copyright and data protection, are mentioned in different chapters of this section, but once again detail is somewhat scarce. chapter 14 describes a typical multimedia workstation, starting with definitions of the role of the cpu, rom and ram, moving on through modems (apparently the only means of getting access to the internet) to a basic list of common input devices. the following chapter deals with network issues, mentioning bandwidth, computer conferencing and different network structures. authoring tools are compared in general terms, and a checklist of useful features presented for consideration. in section 4, on design principles and methods, five layers of tbt are defined and described: model, process, knowledge, resources and guide, which all interact with a userinterface. each element is discussed in turn, with general issues, particularly simulations, being tackled at the end. the book finishes with a series of short information sections: useful addresses (all but one in the uk), more than eight pages of useful urls, grouped into sections (any list of this nature is always likely to be incomplete and out of date, but it seems peculiar that tltp and cti merit no more than a passing mention, and that there is only a single contribution under the heading of computer-assisted assessment); authoring and development tools (listing the principal features of each); a glossary of terms; a bibliography; and suggestions for further reading. the case studies and examples presented throughout the book are generally useful additions, but i was frustrated by the attempt to cover both sides of the atlantic. i found myself wondering whether the book is aimed at a us or a european market. however, the book provides a useful introduction to the use of computers as a training medium (even if i remain to be convinced that tbt in education and commerce can really be tackled as a single issue). it is written in a clear, flowing and relaxed style, and is nicely laid out. it is likely to prove most appropriate to those with an early and general interest in the uses of technology for training, rather than those who already have a professional involvement. the claim on the cover that it is 'a comprehensive guide to choosing, implementing, managing and developing new technologies in training' may be justified, but perhaps it would have more appropriate to present such a guide in four separate, detailed volumes, where the many serious and interesting issues could have been addressed in a more challenging and more memorable form. nora mogey, heriot-watt university implementing computer-supported co-operative learning by david mcconnell, london: kogan page, 1994. isbn: 0-7494-1237-2. hardback, 220 pages. £34.00. this book discusses the use of computers to create collaborative environments that support the process of learning a very topical subject since, certainly within both further and higher education, there is a great deal of interest in how to support students who ate learning independently or at a distance. the role that technology can play in independent learning is the subject of much debate, both in terms of the underpinning infrastructure needed and the pedagogy and educational approaches which it could enable. i was keen to see what this book, by someone with extensive experience in this area, could add to my understanding of the subject. i was somewhat disappointed, but this statement requires some qualification. the book is nicely structured, and each chapter carefully introduces the topic to be developed in the context of what has gone before.'the reader is never in any doubt where the specific chapter is going or how it relates to the rest of the book. i found this particularly helpful since the book itself is broad ranging and, at times, i wanted to be more selective about the topic i was following. chapter 1 introduces the concept of cooperative learning, not from a technology perspective, but focused on the learning process, the way in which individuals can work as a group, and the possible beneficial outcomes of making learning a group activity. there is a smattering of educationalist jargon, but not enough to make the thrust of the argument inaccessible to a lay reader. the stage having been set, chapter 2 goes on to discuss the technologies which can support cooperative learning, particularly group working. 71 reviews mcconnell discusses the possible classification of computer-supported co-operative learning (cscl) into a number of different categories based on the form of the interaction, the geographical location of the users and the degree of structure imposed by the system. a point which mcconnell makes, and with which i totally agree, is that the real costs of implementing cscl lie in the development of the pedagogy and the development of materials rather than in the computers themselves. the weakness of this chapter, and also the later sections of other chapters that illustrate the use of the technology, is that the book was published in 1994 and the technologies referred to are already very out of date. chapter 3 looks at the way in which people work together in online environments, either as peer groups or in a tutor-student relationship. the chapter is based on the author's own experiences, and gives a useful insight into the dynamics of working in this way. particularly interesting is the analytical comparison of the strengths and weaknesses of face-to-face and virtual student groups. having covered these important issues the author goes on, in chapter 4, to discuss the different ways in which courses need to be designed in order to maximize their success when delivered using cscl. this chapter draws heavily on research into the differences between traditional (didactic) and co-operative learning and, rightly, concentrates on examining the learning process in a generic way rather than using technology as the driving force for redesigning courses. the problem with this approach, however, is that mcconnell is then in competition with other, more established, authorities who publish books on the learning process. chapter 5 allows the author to describe the use of cscl within an open-learning ma in management learning. this case-study approach allows us an insight into the attitudes and responses of the students engaged in the course, and is a clear and informative example of theory into practice. it also looks at the role of the online tutors and their experiences of teaching in this new medium. the limitation, again, is that the case study is based on out-ofdate technology, although much of the discussion still has relevance. for the same reason chapter 6 is largely redundant, dealing as it does with 'recent' developments in the technologies for cscl. the final chapter looks at ways of researching groupwork in cscl environments. the author, quite rightly, considers that there are a number of important questions about cscl which need to be answered, and proposes a structure in which this could be accomplished. granted, much of this has been attempted since the book was published, but it is interesting to look down the list of questions and note how many still do not have definitive answers. so, i found this book interesting, but disappointing. the discussions concerning the way in which courses need to be designed for cscl, and the underlying pedagogy of such an approach to open, independent learning are useful, but can largely be found in any of the recent books dealing with open and flexible learning. the discussions about the technology itself are now out of date and potentially misleading to the uninformed. the title of the book implies a practical guide to the use of cscl although in reality it is a useful review of the issues surrounding its use. of course, it is not fair to the author to be too critical of a book which published three years ago and which draws on work that he had previously published elsewhere the year before that. however, taken overall, i could not really recommend this text as a good starting point for someone interested in developing courses which use cscl. c.j. wookton, university of hull calt586675 129..142 peer mentoring – is a virtual form of support a viable alternative? joanne smailesa* and pat gannon-learyb aacademic services, northumbria university, newcastle upon tyne, uk; bbede research & consultancy uk, newcastle upon tyne, uk (received 15 july 2009; final version received 3 march 2011) support systems are vital for university entrants and one established means of support is peer mentoring, which has the potential to improve student engagement and retention. peer mentoring models are generally based on face-to-face contact. however, given the increasing number of higher education institutions using social media, might online models be beneficial in a peer mentoring context? this article describes a literature review and case study that considers the advantages and disadvantages of three potential virtual models to facilitate a peer mentoring scheme. the case study, undertaken at northumbria university, uk, involved an investigation of mentoring needs and current usage of electronic media where special attention is afforded to a diverse student body. the three models discussed are virtual learning environments (vle), social networking sites and virtual worlds. we find that the vle is established within institutions but lacks excitement; social networking is popular particularly with younger students but there may be resentment if this appears to be appropriated by the institution; whilst virtual worlds are unfamiliar to many students and require advanced skills to use successfully. based on these findings the social networking model is now being run as a pilot study by business programmes at northumbria university. keywords: diversity; peer mentoring; student support; virtual learning environments introduction it is well recognised that students’ sense of identification with their peers is both important to their success and complex (haythornthwaite 2008). eggens, van der werf, and bosker (2007) note that personal networks affect student attainment, and clifton et al. (2004) observe how peers give an individual the sense of coping and hence perceived control over academic progress. harvey, drew, and smith (2006) noted how students from lower-income families have less peer support to draw on, and go on to suggest that there is some correlation between socio-economic groups, first-year grades and the possibility of withdrawing from study (especially where family problems intervene). peer mentoring is a well-established method employed by higher education institutions to assist in the integration of students into university. the topic attracts a plethora of literature that is reviewed on a regular basis (jacobi 1991; ehrich, *corresponding author. email: joanne.smailes@northumbria.ac.uk research in learning technologyaquatic insects vol. 19, no. 2, july 2011, 129–142 issn 2156-7069 print/issn 2156-7077 online � 2011 association for learning technology doi: 10.1080/21567069.2011.586675 http://www.informaworld.com hansford, and tennent 2004; terrion and leonard 2007; crisp and cruz 2009). peer mentoring involves more experienced students supporting new students during their academic and personal development, and has been found to be effective in: improving the first-year experience of students (see, for example, watson 2000; hensen 2003; mee-lee and bush 2003; farrell et al. 2004; tariq 2005); increasing student retention (see, for example, packham and miller 2000; boud, cohen, and sampson 2003; muldoon 2008); improving achievement, with some suggesting peer mentoring users gain higher mean grades than non-users (see, for example, congos, langsham, and schoeps 1998; ashwin 2002; 2003); and improving the interpersonal skills of the mentors, which leads to enhanced employability (see, for example, congos and stout 2003; mclean 2004; muldoon 2008). however, there are few studies that have specifically considered the role of peer mentoring in terms of student diversity. best, hajzler, and henderson (2007) reported on how peer mentoring had been used as a technique for the improvement of the international student transition from one learning culture to another, and devereux (2004) reports on the use of peer mentoring as a way to enhance intercultural relationships. the key success factors consistently identified include appropriate mentor training and the inclusion of peer mentoring as part of the formal curriculum (jacobi 1991; ehrich hansford, and tennent. 2004; terrion and leonard 2007; crisp and cruz 2009). in the 1990s, the concept of a ‘community of practice’ (cop) was one of the most important developments in educational theory (lave and wenger 1998). of particular note was the recognition that a cop is not an exclusively academic notion (wenger, mcdermott, and synder 2002). the concept of a cop refers to the process of social learning that occurs – and shared socio-cultural practices that emerge and evolve – when people who have common goals interact as they strive towards those goals (lave and wenger 1998). in a community of learners the first purpose is not practice but rather the advancement of learning (wubbels 2007). these two communities (of practice and of learning) converge when the learning experience combines with socio-cultural development of students, as in the case of support mechanisms such as peer mentoring that aid the enculturation process. smith and bath (2006) suggest that as peer interactions, including the social, are essential determinants of graduate outcome, the notion of a learning community should be reinforced within any innovations for supporting student learning. essentially, peer mentoring, which literature indicates is a sound principle, is designed to initiate cops amongst student cohorts. however, attempts to introduce traditional models of peer mentoring at northumbria university have had mixed fortunes. recruitment and training of mentors, across a number of disciplines, has always been successful but, despite adhering to advice drawn from literature, first-year student attendance was often poor. to investigate this further, research via questionnaire and interview was conducted to identify whether there was actually a demand for peer mentoring from the students. northumbria university has a diverse student group of over 26,500 students. onequarter of students are over the age of 25, 23.0% of are on part-time programmes and 13.1% are classed as international (the majority of whom originate from asia). questionnaire responses received from a representative sample of the diverse cohort (n = 309) showed there was a clear demand for peer mentoring as a concept, with only 5% of respondents stating that they never envisaged requiring the use of a peer 130 j. smailes and p. gannon-leary mentor. however, students felt that the traditional face-to-face contact model was unnecessary, and 75% of respondents stated that virtual contact with a peer would be sufficient (smailes et al. 2008). it was proposed that additional investigation to find a suitable virtual medium would be undertaken, and it is this scoping study that is the principal basis of the present paper. peer mentoring and virtual formats in a report commissioned by the joint information systems committee (jisc 2008) the two most regular computer-based activities students undertake are use of university systems that support their learning (i.e. virtual learning environments [vles]) and accessing social networking sites. vles provide an integrated set of tools such as folders, email, quizzes and discussion boards to support learning (svane and smedley. 2004). social networking sites provide space allowing individuals to create personal profiles and social commentary that can be publically shared (boyd 2006). additionally, in an eduserv report kirriemuir (2008) examines the increasing number of uk universities that have a presence in virtual worlds. a prime example being second life, where users become residents known as avatars who interact either in fantasy or reality-based environments. students make intensive use of online media including socialisation tools in their social lives (cheng and vassileva 2006) but the same technologies, with the exception of email, are less used for academic-related and support work. there is little evidence of the existence of uk programmes of peer mentoring in virtual formats. the nearest cases being an australian example (dewart et al. 2003), in which email was used as the principal form of communication between mentor and mentees, and the work of davis and bullen (2004), a subject-specific example where a vle was used to match year four students as mentors to year one students on a construction management project. however, there is increasing interest in the potential of virtual support mechanisms. for example, madge et al. (2009) explores social networking and observed how pre-registration engagement with the social networking site facebook influences students’ post-registration social networks. students reported joining facebook as a means of making new friends at university, as well as keeping in touch with friends and family at home. once at university, facebook formed part of the ‘social glue’ that helped students settle into university life but was just one aspect of students’ more general social networking, both face to face and virtual. students thought facebook was used most importantly for social reasons, not for formal teaching purposes, although it was sometimes used informally for learning purposes. selwyn (2009) presents an in-depth qualitative analysis of the facebook ‘wall’ activity of 909 undergraduate students in another uk university. analysis of these data shows how much of students’ education-related use of this social networking application was based around critiquing of/reflecting on learning experiences, exchange of factual information about teaching and assessment requirements and instances of requests for support (actual and moral) with regards to assessment or learning. facebook was seen as place to discuss their personal identity as a student outside the confines of formal academic environments and therefore would appear relevant for undergraduates making the transition from school to higher education. research in learning technology 131 glasgow caledonia university’s initiative on emergent technologies for learning is currently exploring how the second life virtual world can help students become more effective learners. in their i-campuss (internet college articulation and mentoring project for prospective university students), student mentors use second life to interact with first-year (and even prospective) students to explain things about schools and services that new students need to know. this service is specially aimed at international students (glasgow caledonia university 2009a). in order to capitalise on the positive benefits of peer mentoring as a method of engaging students in communities of practice and to counteract the issues associated with mentee engagement in face-to-face programmes, this paper will examine three virtual formalised models of peer mentoring and scrutinise evidence both from the literature and using a case study from northumbria university’s student body to explore the arguments both for and against the connections between communities of practice and the three models proposed. furthermore, as technology has the potential to begin the mentoring process before students arrive for the first-year induction period, capability of the technologies to do this will also inform the decision-making process. in this instance, the three technologies under consideration are, respectively: northumbria university’s established learning and support tool, the vle blackboard; a social networking site (e.g. facebook); and the virtual world second life. in the former case, selected due to its long-term (since the late 1990s) establishment at the university; the latter formats chosen on the emerging interest of their use as a support mechanism (glasgow caledonia university 2009b; madge et al. 2009; selwyn 2009). methodology the first phase of the research methodology comprised a desk-based review of relevant literature that analysed and synthesised the outcomes of existing research and studies in the areas of interest (i.e. peer mentoring, communities of practice and new technologies). in doing this, a modified version of a systematic literature review was employed. the researchers were engaged in searching, and together they built an endnote library based on the retrievals, sharing selection and assessment criteria and key search terms (see table 1). online searches were used and these concentrated on a federated library search engine (especially databases in computing and business information systems, information and communication studies, and education). the researchers independently conducted searches around different areas of the project and then merged their resultant libraries, subsequently eliminating duplicates, whilst acknowledging that the existence of duplicates might add weighting to the value of the references since they had been retrieved via independent searches using different search strategies. the rationale for this approach was that it would ensure the selection of relevant, quality work from what was available using an efficient technique that minimised bias. articles found were fully reference-checked by the researchers and briefly summarised. the contents of the endnote library were analysed to develop an in-depth understanding of the current state of the topics under study and, based on the analysis, subject terms were derived and applied in order to be able to categorise the contents under subject headings. the final endnote library contained more than 250 references. the ultimate selection of references to include 132 j. smailes and p. gannon-leary in the study was made on the basis of the appropriateness of the categorisations to the aims of the research. final selections were evaluated by the researchers to improve the reliability of inclusion and, where differences arose, consensus was achieved by means of discussion. the second phase of the research methodology involved a community consultation via a traditional survey approach in order to validate and extend the findings from the literature review. this was undertaken in order to establish current usage of social media within the context of one higher education institution, northumbria university, and to explore the potential for developing a peer mentoring system via different social media. the survey method was chosen because it is fast and relatively inexpensive, whilst enabling the establishment of a baseline. for this exercise all of northumbria university’s on-campus students formed the population of interest and would be eligible to take part in the survey. after securing the appropriate ethical clearance, surveys were distributed both by hand (in social areas) and via the internet (link provided in the students’ union electronic newsletter) using surveymonkey. responses to the survey were received from 451 northumbria university students. around one-third of the respondents were male and two-thirds female; the majority of respondents (80%) were in the 19–24 age group and responses were received from students in all of northumbria university’s nine academic schools and across all levels of study (i.e. foundation level to postgraduate). over 20% of the respondents were non-uk students and 11% declared a disability – the majority of whom cited dyslexia. in terms of socio-economic status, over 50% of the cohort were first-generation students and around 40% of students noted they were the first table 1. search terms. academic + persistence new + entrants attendance peer + assisted + learning attrition peer + educators blackboard peer + learning, cal or computer aided [or assisted] learning peer + support collaborative + learning peer + teaching collaborative + technologies peer + mentoring communication⁄ retention communities + learn⁄ second + life communities + practice social + integration course + completion social + media first + year + experience social + networking emotional + competence socialisation enculturation students facebook student + attrition high + risk + students student + experience higher + education supplement⁄ + education or instruction integration technolog⁄ interaction undergraduate⁄ intercultural + skills universit⁄ interventions virtual + communit⁄ mentor virtual + learn⁄ + environment motivation virtual + worlds networking note: ⁄indicates truncation. +/or indicate use of boolean operators. terms were used in various permutations. research in learning technology 133 of their sibling group to enter higher education. in all cases, the responses received were deemed to be representative of northumbria university’s student cohort. results and discussion as a starting point, students were asked what most accurately reflected their access to the internet. the vast majority of students (82%) stated they had full private access to the internet in their place of residence, with a further 12% saying they had frequent shared access. furthermore, 79% of respondents indicated they have been internet users for five years or more and 68% of respondents stating they had email accounts for five years or more, both observations being consistent across the different age groups surveyed. these figures indicate that students are both experienced users of email and the internet, substantiating the viability of introducing virtual media in the peer mentoring process. virtual learning environments vles typically provide an integrated set of tools for learning and teaching (svane and smedley 2004). there are numerous vles in operation but with over 5000 colleges and universities using webct or blackboard these are viewed as the market leaders (maclaren 2004). northumbria university’s vle (blackboard) has been in operation since the late 1990s. it is extremely well integrated into teaching and learning, with around two-thirds of survey respondents saying they access the vle on a daily basis and a further 30% stating they access the vle on a weekly basis. this was the case across all demographic factors; that is, age, international students, disabled students, part-time students and first-generation students (students who are the first in their family to attend university). burge (1994) comments that as a communication tool a vle provides students opportunities to engage in social interactions with their peers alongside course-related conversations, with sole and lindquist (2001) amongst others (gibbs 1999; arbaugh 2000) noting the potential for vles to break down some social barriers found in the classroom, such as gender and nationality. there is some evidence that cops using discussion lists and vles have some benefit, particularly within professional practice (ebenezer et al. 2003; goos and bennison 2008). in an extensive report reviewing northumbria university students’ use of the vle conducted by gannon-leary and turnock (2007), similar positive comment was received: i agree that i could not successfully complete a module without the vle site. i use vle to allow me to interact with other students and lecturers via the discussion boards and it often helps when i am struggling with work, especially when i know others are also finding it difficult. ellaway, dewhurst, and mcleod (2004) used wenger’s learning architecture framework to propose a method of evaluating a vle as supporting an existing cop, and found the value of the vle is not in the software itself but dependent on its use and hence exists “in a ‘blended’ relationship with human activities” (ellaway, dewhurst, and mcleod 2004, 127). maclaren (2004) comments that the commonest model in undergraduate teaching is that of a ‘transmission’ model, with conole and de laat (2006) observing that students were unenthusiastic about vles, citing 134 j. smailes and p. gannon-leary them as an unimaginative repository for materials that was often poorly structured and used on an ad hoc basis. this was clearly illustrated through comments received in relation to the aforementioned northumbria report by gannon-leary and turnock (2007): the vle is certainly an effective addition to teaching in terms of the facility to post learning information and material. sometimes it is an interesting addition to teaching. it depends on the lecturer and how they use it. it also depends on the student cohort. for example, one of my lecturers started a discussion board this semester which i thought was a really good idea and a good way to stimulate interest but no one bothered to post anything there! i found that the vle was not very exciting to the typical student that feels they have better things to do with their time. unless the students are excited by the topic (which they should be) the vle does not give off the impression of keeping people’s attention for a long time. finally, students are not usually given access to the vle until after enrolment, which means the virtual peer mentoring environment is unlikely to be available for use until after the induction period. therefore it is suggested that the very nature of institutional control of a vle is likely to lead to difficulties in facilitating or engaging with social communities of informal and impromptu learning (o’hear 2006). social networking sites a social networking site is defined as one with profiles, and regular commentary on those profiles with a varying but publicly available social network (boyd 2006). liccardi et al. (2007) note how, owing to the perceived alignment with social– cultural theories of learning, social networking sites and their role within the students’ learning experiences have an increasing and important role. several researchers (phipps 2007; cain 2008; ryberg and christiansen 2008) have identified the potential for creating social learning environments that offer ‘social communities of practice’. evidence suggests access to social networking sites is a day-to-day occurrence (bausch and han 2006). as noted by tufekci (2008), multiple us studies have found that significant proportions of students (>80%) have a social networking profile where it was found that around one-half considered themselves to be heavy users of such sites. when considering what drew students to such social networking one of tufekci’s (2008, 558), interviewees commented how they took “the work out of meeting people” as students were busy people and this cuts out one thing to do whilst “giving you the benefit you desire”. conversations surrounding social networking sites often make assumptions that in reality this is facebook. however, as noted by boyd and ellison (2007), social networking engagement is culturally dependent – a particularly important consideration for the case study at northumbria university, since 13.1% of students there are classed as international (the majority of whom originate from asia). therefore, it was felt that differences in usage by diverse students merited investigation. to investigate whether cultural differences exist, students at northumbria university were asked about their use and knowledge of a number of social networking sites established as being used research in learning technology 135 globally although this list is not exhaustive, table 2 clearly illustrates that facebook is indeed the most widely used at northumbria university with over threequarters of the respondents noting that they access it on a daily basis. this figure is 10% more than those accessing the vle daily and over 70% more than the daily use of the second most popular social networking site – myspace. when examining the use of facebook across the different diverse cohorts, international students were the group of particular interest and no significant differences were found in terms of facebook use. nevertheless, they were more likely to also use an additional social networking site on a regular basis – orkut being popular with students from india. however, age was one factor where differences were observed, with students aged over 30 reporting that they were less likely to use any form of social networking. for all other demographic factors, social networking via facebook was dominant. however, kolek and saunders (2008) note how a number of us universities have increasing concerns in relation to social network sites. in their own study, which analysed student use of facebook, it was revealed that only small proportions of students restrict access to their individual postings on the university network. many students gave full disclosure of personal information, including photographs. it was observed that many of these individual postings could have “potential negative consequences for individual students”, amplified by us media cases of sexual assault, stalking, and so forth, and a number of photographs displayed (e.g. alcohol fuelled) could cause “potential damage to an institution’s image” (kolek and saunders 2008, 5). in a separate study, lipka (2007) comments on the complex difficulties that can arise through facebook relationships; in particular, staff–student connections. firstly, students in lipka’s study expressed the view that the site was their domain and as such needed free expression without worrying what professors might think. this is corroborated by monteith and smith (2001) and jisc (2008), the latter table 2. social networking knowledge and use at northumbria university 2008. social network use daily (%) use weekly (%) occasionally or rarely (%) don’t use/not heard of it (%) facebook 76.2 13.1% 4.2% 6.5 myspace 5.0 5.5 32.0 57.6 bebo 3.8 3.1 17.1 76.1 orkut 2.4 0.7 1.4 95.4 twitter 1.4 1.4 4.6 92.5 friendster 0.7 1.2 3.9 94.2 hi 5 0.7 1.2 8.6 89.5 xiaonel 0.7 1.2 0.9 97.1 faceparty 0.7 0.2 5.8 93.2 xanga 0.5 0.2 2.2 97.1 tagged 0.5 1.4 7.2 90.9 xing 0.2 0.2 1.5 98.0 skyrock 0.2 0.2 2.2 97.3 cyworld 0.2 0.2 0.7 98.7 friends reunited 0.2 0.7 10.6 88.5 second life 0.0 0.2 2.6 97.1 136 j. smailes and p. gannon-leary reporting that although there is a “wide use of social networking students struggle how to see it can be used in learning” (jisc 2008, 10). lipka (2007) additionally warns that students may also expect special privileges in the name of being a facebook ‘friend’ citing requests such as asking to swap classes or assignment extensions as examples. furthermore, carnevale (2006) warns that one us college fell foul of facebook’s terms and conditions, which exclude an organisation setting up a user account for a group or entity; thus in creating a virtual mentor model, great care needs to be taken to ensure ownership lies with the mentors and not the institution per se. in order to offset potential personal dangers and the reflection of some student entries on an institution, kolek and saunders (2008, 18) recommend that institutions at the very least “should develop clear policies and procedures for the use of facebook and other social networking sites in (the name of) official institutional business” as well as use the induction process to warn students of the dangers of personal disclosure. if institutions begin to consider policies in relation to social network site use, students may begin to view the legitimate educational uses the university does support as imposed and hence suffer from the same negativity as the increasing current view of vle use. the group basis on which social networking sites like facebook operate, as suggested miller and jensen (2007), focus on predefined activity as opposed to learning within a community. in traditional models of peer mentoring, the mentors have at least one full year of experience at university. however, if considering a basic dictionary definition of a mentor as a wise giver of advice, a student could pass on advice arising from their own personal discoveries at any time. given that a number of northumbria university students reported that their facebook use began before entering higher education, where 73% had registered with pre-university email addresses, this process of passing on advice could also begin before the traditional induction periods. virtual worlds virtual worlds (e.g. second life), also referred to as three-dimensional multi-user virtual environments, have existed in some form since the early 1980s (warburton 2009). closely associated with computer gaming, virtual worlds have become of increasing interest to educators (peachy 2007). one popular example is second life created by linden labs in 2003. users who create avatar characters can purchase land and other items to create their own interactive graphic environments. kirriemuir (2008) points out that second life creators claim that such spaces offer a community-based environment that mirrors reality – able to combine social cops with a personal learning environment, creating a ‘learning network’ that enables social and collaborative experiential learning. several uk universities have purchased land within the second life grid to host their own support and learning environments (shepard 2008), the basis of purchase being that the fantasy environment will allow real academic freedom for discussion. salmon believes that with the use of second life avatar characters “there are not going to be the usual discrimination issues of the face-to-face environment . . . the student and the teacher are on the same level” (salmon quoted by shepard 2008). however, wetsch (2008) observes students’ perception of a greater learning curve required to operate at a basic level within second life and browne et al. (2008) research in learning technology 137 note that staff skills were the greater challenge in incorporating virtual world technologies into learning. salmon (2009) recognises that in second life strong scaffolding and support is likely to be required, but believes that if these barriers are overcome second life “is low cost and high value for learning” (salmon 2009, 535). oliver and carr (2009) applied wenger’s model of participation in cop to couples that play world of warcraft – a virtual world. they found analogies with distance learners, concluding that similar problems of retention faced by students studying at a distance (i.e. insufficient time to engage in activities and hence lack of socialisation) were as just as likely to occur when engaging in virtual world activity. warburton (2009) detects parallels within second life, the vast choice of communities deters casual use. navigation, codes of etiquette, stigmatisation of a ‘newbie’ can lead to an isolating experience (boostrom 2008). cheal (2007, 204) has pronounced “that second life is not simply the latest online fad, but part of a continuum of instructional technology tools that corresponds to twentieth and twentyfirst century developments in educational theory”, with gartner inc. (2007) predicting that 78% of internet users will have a “second life” by 2011. the current picture at northumbria university indicates knowledge and use of second life is much less than this. as illustrated in table 2, only 12 of the 451 respondents indicated they used second life, most rarely, and all users were in the 19–24 age group. notably, the only two regular users were international male students at newcastle business school from hong kong and china. of the remaining 439 respondents, one-third have heard of second life but do not use it but, more surprisingly, two-thirds of the sample said they had no knowledge of the application. warburton (2009) also recognises that although the second life avatar characters do have real-life profiles they are limited and as such do not provide the level of social discovery that social networking sites like facebook can provide. finally, edirisingha, salmon, and nie (2009) note that with any e-learning materials an academic as second life designer and moderator will be crucial in ensuring student engagement and success. therefore, it may suffer from the same disadvantages as those currently experienced by the vles. conclusions the traditional face-to-face peer mentoring model has had limited success at northumbria university. nonetheless, as this paper reports, previous research with students established that there was a demand for peer mentoring but that such demand was for ‘needs-based’ access to a mentor and that virtual contact with a mentor would be readily accepted. students were found to have frequent access to the internet and hence would be able to access materials on any of the three virtual mentoring platforms under consideration. the vle offers an established technological platform for mentoring since it is used regularly (in the majority of cases daily) by students. however, it lacks excitement, its use is directed by tutors post induction and it is mostly focussed on a repository role rather than that of a learning community. therefore a vle would be unlikely to engage students in peer mentoring activities. 138 j. smailes and p. gannon-leary social networking sites are heavily used by students and have popular appeal. there are a vast number of social networking sites available, the use of which can be culturally dependent. the primary research data (table 2) established that facebook was extensively and regularly used by all including international students – an initial concern. however, older students may be less likely to fall into the heavy user category. the potential for misuse and abuse indicates a potential necessity to introduce policies regarding facebook usage, but this could have a negative impact on students who may resent the appropriation of one of their mechanisms for socialising by the institution. virtual worlds such as second life are increasingly being used by academic institutions. they offer a fantasy environment that has the potential to afford users academic freedom and break down barriers between academic staff and students, mentors and mentees. currently second life is unfamiliar to most students and literature indicates that the skills associated with a virtual world environment have a greater learning curve on both the part of the designers and users. the basis under which second life operates is likely to require institutional purchase, and hence design of materials, like a vle, is likely to come under institutional control. the evidence presented here indicates that a model involving social networking sites has the greatest potential for supporting peer mentoring. this is due both to its immense popularity amongst the student body alongside the ability for mentoring to commence before formal student induction. it is proposed that, as mentors are students themselves, the site is unlikely to be negatively perceived as being under institutional ownership. however, in line with any mentoring programme, mentor training is essential and this instruction needs to include guidelines on insuring facebook is used in a secure and professional manner. agreement has been established with one of northumbria’s academic schools to work with the authors to further explore a social networking-based model of peer mentoring. the pilot study involving students on marketing and business programmes will be analysed using participatory action research methods to ensure the mentoring process and materials provided warrant continued student engagement. it is expected that the outcomes from this pilot will inform a future paper establishing a new dimension in peer mentoring practice. references arbaugh, j.b. 2000. virtual classroom characteristics and student satisfaction with internetbased mba courses. journal of management education 24, no. 1: 32–54. ashwin, p. 2002. implementing peer learning across organisations: the development of a model. mentoring and tutoring 10, no. 3: 221–31. ashwin, p. 2003. peer facilitation and how it contributes to the development of a more social view of learning. research in post-compulsory education 8, no. 1: 5–17. bausch, s., and l. han. 2006. social networking sites grow 47 percent, year over year, reaching 45 percent of web users. http://www.netratings.com/pr/pr_060511.pdf. best, g., d. hajzler, and f. henderson. 2007. communicating with chinese students offshore to improve their transition and adjustment to australia – a pilot program. journal of academic language and learning 1, no. 1: a78–90. boostrom, r.e. 2008. the social construction of virtual reality and the stigmatized identity of the newbie. journal of virtual worlds research 1, no. 2: 1–19. boud, d., r. cohen, and j. sampson. 2003. peer learning in higher education: learning from and with each other. london: kogan page. boyd, d. 2006. social network sites: my definition. http://zephoria.org/thoughts/archives/ 2006/11/10/social_network_1.html. research in learning technology 139 boyd, d.m., and n.b. ellison. 2007. social network sites: definition, history, and scholarship. journal of computer-mediated communication 13, no. 1: 210–30. browne, t., r. hewitt, m. jenkins, and r. walker. 2008. survey of technology enhanced learning for higher education in the uk. oxford: ucisa. burge, e.j. 1994. learning in computer conferenced contexts: the learners’ perspective. journal of distance education 9, no. 1: 19–43. cain, j. 2008. online social networking issues within academic and pharmacy education. american journal of pharmaceutical education 72, no. 1: 10. carnevale, d. 2006. college tries to be cool but runs afoul of facebook. chronicle of higher education 153, no. 14: 39. cheal, c. 2007. second life: hype or hyperlearning? on the horizon 15, no. 4: 204–10. cheng, c., and j. vassileva. 2006. design and evaluation of an adaptative mechanism for sustained educational online communities. user modeling and user-adapted interaction 16: 321–48. clifton, r.a., r.p. perry, c.a. stubbs, and l.w. roberts. 2004. faculty environments, psychosocial dispositions and the academic achievement of college students. research in higher education 45, no. 8: 801–28. congos, d., and b. stout. 2003. benefits of si leadership after graduation. research and teaching in developmental education fall: 1–9. congos, d., d. langsham, and n. schoeps. 1998. inside supplemental instruction sessions: one model of what happens that improves grades and retention. research and teaching in developmental education 15, no. 1: 47–61. conole, g., and m. de laat. 2006. methodological issues with researching the student learning experiences. presentation to the jisc pedagogy experts group meeting, february, in birmingham, uk. crisp, g., and i. cruz. 2009. mentoring college students: a critical review of the literature between 1990 and 2007. research in higher education 50, no. 6: 525–45. davis, p., and p. bullen. 2004. putting it together: supportive strategies for learning and assessment using peer mentoring. paper presented at the 3rd asia-pacific conference – diversity and equity in continuing education and lifelong learning, september 1, in curtin university of technology, perth, australia. devereux, l. 2004. when harry met sarita: using a peer-mentoring program to develop intercultural wisdom in students. paper presented at the higher education research and development society of australasia: transforming knowledge into wisdom: holistic approaches to teaching and learning, july 4–7, in miri, sarawak. dewart, h., d. drees, p. hixenbaugh, and d. williams. 2003. electronic peer mentoring: a scheme to enhance support and guidance and the student learning experience. paper presented at educause 2003, may 6–9, in adelaide, australia. ebenezer, j.v., f. lugo, b. beirnacka, and a. puvirajah. 2003. community building through electronic discussion boards: pre-service teachers’ reflective dialogues on science teaching. journal of science education and technology 12, no. 4: 397. edirisingha, p., g. salmon, and m. nie. 2009. 3-d multi-user virtual environments for socialisation in distance learning. paper presented at the association for learning technology conference 2008, september 9–11, in university of leeds, leeds, uk. eggens, l., m.p.c. van der werf, and r. bosker. 2007. the influence of personal networks and social support on study attainment of students in university education. higher education 55, no. 5: 553–73. ehrich, l.c., b. hansford, and l. tennent. 2004. formal mentoring programs in education and other professions: a review of the literature. educational administration quarterly 40, no. 4: 518–40. ellaway, r., d. dewhurst, and h. mcleod. 2004. evaluating a virtual learning environment in the context of its community of practice. alt-j, research in learning technology 12, no. 2: 125–45. farrell, h., c. pastore, n. handa, j. dearlove, and e. spalding. 2004. initiating the battlers. paper presented at isana conference 2004, december, in melbourne, australia. gannon-leary, p., and c. turnock. 2007. student use of the elearning portal. internal report for elearning enhancement group, northumbria university at newcastle upon tyne. 140 j. smailes and p. gannon-leary gartner inc. 2007. gartner says 80 percent of active internet users will have a ‘second life’ in the virtual world by the end of 2011. http://www.gartner.com/it/page.jsp?id=503861. gibbs, g.r. 1999. learning how to learn using a virtual learning environment for philosophy. journal of computer assisted learning 15, no. 3: 221–31. glasgow caledonia university. 2009a. i-camppus (internet college articulation and mentoring project for prospective university students). http://caledonianblogs.net/sohsecondlife. glasgow caledonia university. 2009b. initiative on emergent technologies for learning. http://www.gcu.ac.uk/cuthere/projects.html. goos, m.e., and a. bennison. 2008. developing a communal identity as beginning teachers of mathematics: emergence of an online community of practice. journal of mathematics teacher education 11, no. 1: 41–60. harvey, l., s. drew, and m. smith. 2006. the first-year experience. briefing paper: overview for higher education policy makers and practitioners. york, uk: higher education academy. haythornthwaite, c. 2008. learning relations and networks in web-based communities. international journal of web based communities 4, no. 2: 140–58. hensen, k.a. 2003. impact of supplemental instruction: results from a large, public, midwestern university. journal of college student development 44, no. 2: 250–9. jacobi, m. 1991. mentoring and undergraduate academic success: a literature review. review of educational research 6, no. 4: 505–32. joint information systems committee. 2008. great expectations of ict, how higher education institutions are measuring up. http://www.jisc.ac.uk/publications/publications/greatex pectations.aspx. kirriemuir, j.a. 2008. a spring 2008 ‘snapshot’ of uk higher and further education developments in second life. http://www.eduserv.org.uk/~/media/foundation/sl/uksnap shot052008/final%20pdf.ashx. kolek, e.a., and d. saunders. 2008. online disclosure: an empirical examination of under graduate facebook profiles. naspa journal 45, no. 1: 1–25. lave, j., and e. wenger. 1998. communities of practice: learning, meaning, and identity. cambridge: cambridge university press. liccardi, i., a. ounnas, r. pau, e. massey, p. kinnunen, s. lewthwaite, m. midy, and c. sarkar. 2007. the role of social networks in students’ learning experiences. acm sigcse bulletin 39, no. 4: 224–37. lipka, s. 2007. for professors, ‘friending’ can be fraught. chronicle of higher education 54, no. 15: 1. maclaren, i. 2004. new trends in web-based learning: objects, repositories and learner engagement. european journal of engineering education 29, no. 1: 65–71. madge, c., j. meek, j. wellens, and t. hooley. 2009. facebook, social integration and informal learning at university: ‘it is more for socialising and talking to friends about work than for actually doing work’. learning, media and technology 34, no. 2: 141–55. mclean, m. 2004. does the curriculum matter in peer mentoring? from mentee to mentor in problem-based learning: a unique case study. mentoring and tutoring: partnership in learning 12, no. 2: 173–86. mee-lee, l., and t. bush. 2003. student mentoring in higher education: hong kong baptist university. mentoring and tutoring: partnership in learning 11, no. 3: 263–71. miller, s.e., and l.a. jensen. 2007. connecting and communicating with students on facebook. computers in libraries 27, no. 8: 18–22. monteith, m., and j. smith. 2001. learning in a virtual campus: the pedagogical implications of students’ experiences. innovations in education and teaching international 38, no. 2: 119–32. muldoon, r. 2008. recognising and rewarding the contribution and personal development of peer supporters at university. journal of further and higher education 32, no. 3: 207–19. o’hear, s. 2006. e-learning 2.0 – how web technologies are shaping education. http://www. readwriteweb.com/archives/e-learning_20.php. research in learning technology 141 oliver, m., and d. carr. 2009. learning in virtual worlds: using communities of practice to explain how people learn from play. british journal of educational technology 40, no. 3: 444–57. packham, g., and c. miller. 2000. peer assisted student support: a new approach to learning. journal of further and higher education 24, no. 1: 55–65. peachy, a. 2007. education in second life. http://www.open.ac.uk/cetl-workspace/cetlcon tent/documents/48845a25f07ab.pdf. phipps, l. 2007. web 2.0 and social software: an introduction. http://www.jisc.ac.uk/media/ documents/publications/web2socialsoftwarev1pdf.pdf. ryberg, t., and e. christiansen. 2008. community and social network sites as technology enhanced learning environments. technology, pedagogy and education 17, no. 3: 207–19. salmon, g. 2009. the future for (second) life and learning. british journal of educational technology 40, no. 3: 526–38. selwyn, n. 2009. faceworking: exploring students’ education-related use of facebook. learning, media and technology 34, no. 2: 157–74. shepard, j. 2008. it’s a world of possibilities, the guardian, may 8. http://www.guardian. co.uk/education/2007/may/08/students.elearning. smailes, j., p. gannon-leary, c. laing, and l. conniss. 2008. virtual mentor: an innovation in student support. paper presented at making connections conference, november 6, in london, uk. smith, c., and d. bath. 2006. the role of the learning community in the development of discipline knowledge and generic graduate outcomes. higher education 51, no. 2: 259–86. sole, m., and l. lindquist. 2001. enhancing traditional, televised and videotaped courses with web-based technologies: a comparison of student satisfaction. nursing outlook 49, no. 3: 132–9. svane, t., and j. smedley. 2004. encouraging reflection using a bespoke virtual learning environment. paper presented at conference of the australian association for research in education, november 29–december 2, in melbourne, australia. tariq, v.n. 2005. introduction and evaluation of peer-assisted learning in first-year undergraduate bioscience. bioscience education e-journal 6, no. 3: 1–19. terrion, j.l., and d. leonard. 2007. a taxonomy of the characteristics of student peer mentors in higher education: findings from a literature review. mentoring and tutoring: partnership in learning 15, no. 2: 149–64. tufekci, z. 2008. grooming, gossip, facebook and myspace. information, communication and society 11, no. 4: 544–64. warburton, s. 2009. second life in higher education: assessing the potential for and the barriers to deploying virtual worlds in learning and teaching. british journal of educational technology 40, no. 3: 414–26. watson, j. 2000. peer assistance support scheme (pass) for first year core subjects. paper presented at the fourth pacific rim – first year in higher education conference: creating futures for the new millennium, july 5–7, in queensland university of technology, brisbane, australia. wenger, e., r. mcdermott, and w. synder. 2002. cultivating communities of practice: a guide to managing knowledge. boston, ma: harvard business school press. wetsch, l.r. 2008. the ‘new’ virtual consumer: exploring the experiences of new users. journal of virtual worlds research 1, no. 2: 1–2. wubbels, t. 2007. do we know a community of practice when we see one? technology pedagogy and education 16, no. 2: 225–33. 142 j. smailes and p. gannon-leary alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 173–174 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020173–02 © 2007 association for learning technology doi: 10.1080/09687760701482358 discussion section the tipping point: a response pam moule* university of the west of england, uk taylor and franciscalt_a_248112.sgm10.1080/09687760701482358alt-j research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis152000000june 2007pammoulepam.moule@uwe.ac.uk i am grateful to gilly salmon for providing further personal insight into the five-stage model for e-learning, reviewed in my recent paper (moule, 2007). professor salmon plots the development and use of the model, first conceived some 12 years ago, and encourages us to reflect further on a model that has been so widely adopted. the longevity of its use in a fast-changing field is testament to its appeal to educators, developers and learners. it is clear that a number of alt-j readers will know of, and have used, the model and may want to express thoughts on its current applicability, as salmon invites. in challenging the model with the proposed ‘e-learning ladder’ (moule, 2006), i am also encouraging a discourse about the pedagogy of e-learning at a time when technology continues to offer great scope for further development through virtual learning environments and web 2.0 technologies. i believe it is important to encourage academic staff engagement in designing for online pedagogy, although i support salmon’s suggestion that this remains far from easy. too frequently i encounter concerns relating to an information and communications technology-driven pedagogy, where academics struggle to ensure learning outcomes are met through the available computer technology. these issues are explored in a recent publication (shalnti, 2006), where it is suggested that technological developments, including mobile devices, continue to be defined in relation to delivery capability rather than accounting for different learning needs. the e-learning ladder presents my construction of a conceptual model that acknowledges the range of e-learning approaches available to academics and e-learning designers. it shows how learning might be positioned from instructivist through to constructivist, giving examples of the types of e-learning that might be located within. these ladder ‘rungs’ could be developed further to take account of the range of web 2.0 technologies now available, particularly in respect of m-learning. the ladder ‘sides’ represent a number of the requirements to support e-learning identified through the research. these i feel might also be enhanced, to include, for example, the role of gender in e-learning. it is my intention to develop this through further research and use, and hope that others in the e-learning community will engage in this as well. i * university of the west of england, faculty of health and social care, blackberry hill, stapleton, bristol bs16 1dd, uk. email: pam.moule@uwe.ac.uk 174 p. moule developed the model to take account of learner diversity and pedagogic innovation. it was my intention that it should support academics and designers in thinking through the learning approach, on an instructivist to constructivist continuum, to identify how the technology could enable achievement. i wait to hear your thoughts and feedback. references moule, p. (2006) e-learning for healthcare students: developing the communities of practice framework, journal of advanced nursing, 53(3), 370–380. moule, p. (2007) challenging the five-stage model for e-learning: a new approach, alt-j, 15(1), 37–50. shalnti, g. (2006) application of new technologies: nurse education, in: s. glen & p. moule (eds) e-learning in nursing (basingstoke, palgrave). reviews edited by philip barker integrating information technology into education edited by deryn watson and david tinsley, london, chapman & hall, 1995, isbn: 0-412-62250-5, 316 pages. in these days of research-assessment exercises and cost-cutting measures, if you have not arranged beforehand for the proceedings of your conference to be published, you can expect fewer proposals for papers from experts at the cutting edge of their subject. many departments will not normally fund attendance at conferences, especially those held abroad, unless there is a concrete outcome which can be added to the departmental publications list. this is why, over the last few years, we have seen such an increase in the number of edited conference papers. unfortunately (and i certainly do not include this current issue of alt-j in this statement, because the selection procedure has been really quite cruel), many of these proceedings are of low overall quality. the is no doubt partly because once your paper has been accepted for the conference, you want as much of a guarantee of publication as possible before you turn up and give it, and conference organisers are thus under pressure not to give the copper-bottomed guarantee they cannot of course give, but at least to keep you, as far as possible, in promise land. this situation tends to lead to diluted proceedings. it was therefore a pleasant surprise to find that this set of proceedings the result of the international federation of information processing working conference, held in barcelona last october, on integrating technology into the curriculum at secondary-school level was sufficiently focused oh a main topic (essentially, how do we make it as 'invisible' as tv?) to produce many very thoughtful papers. not only that, but the editors have clearly been at the contributions, prior to publication, with a not-so-gentle editing hand, such that they all end up well structured and well written (it would be hard to believe that they were all received in their final published state). the conference was truly an international one; that is, not one organized from the uk in some foreign town to give the event an international dimension and at which most participants are actually british. contributors at this conference came from all over the world, and together were able to give a genuinely global perspective on the problems of integrating it into schools. many of these problems are identical in the world of higher education, so these proceedings will be of considerable interest to alt members. topics range from learners' expectations and their achievements when relying on it, through the role of the teacher in an integrated it environment, to specific instances of success, part-success and even failure in integrating pieces of software into the classroom. there are 23 major (invited) papers, eight short papers, and nine so-called focus-group reports. the latter are summaries of round-table discussions, and in some ways represent the most interesting part of the proceedings, since they tend to highlight practical problems. but, in any case, such practical problems are everywhere present in the papers, whose themes cover topics such as coming to terms with teachers' concerns, the role of software in changing the conceptual basis of subjects, the role of it itself as a subject of study and its role in fostering inter-disciplinary activities, and the need for national government initiatives. it would be unfair to pick out one paper rather than another as being more important to, or more 106 volume 4 number i representative of, the proceedings as a whole, but alt members will be particularly interested in a number of contributions. these include collis on societal and organizational influences of networking, olson on the classroom ethos, van weert on integrating it generally into education (including higher education), graf on interdisciplinary and inter-cultural intentions, teague and clarke on some proposed historical reasons for gender differences in interest in computers, and aston's short paper on the british approach to integrated it. gabriel jacobs, university of wales swansea the simulation and gaming yearbook. volume 3: games and simulations for business edited by danny saunders, london, kogan page, isbn: 0-7494-1617-3, 1995, 331 pages. this series of annual volumes succeeds simulation/games for learning, the quarterly journal published from 1970 to 1992. like the journal before it, the yearbook is compiled by sagset (the society for the advancement of games and simulations in education and training) which has about 150 members, mainly educationists and trainers in the uk. volume 3, the 1995 yearbook, contains 29 papers arranged into six main sections: activities and ideas; business applications; transforming the curriculum; development and design; the finland symposium; and computer applications. a seventh section lists contacts (including sagset), publishers and recent articles. however, much of the content transcends this simple classification, with themes and ideas recurring across sections. nor is the content confined solely to the topic of the book's subtitle the environment, mathematics, primary education, english teaching, higher education and theoretical issues feature as well as business. games and simulations provide frameworks of rules and roles with the aim of promoting live, vividly interactive learning experiences, usually through group work, sometimes with computer support. as jacques points out (p. 27), 'the magic ingredient is undoubtedly the imagination, triggered by a sense of play'. although there are ways that we in alt can incorporate such elements into the computer-based learning software we design, we probably have to accept there is little chance of creating such high levels of interaction for students working one-to-one with computers. some of the ideas described in the book make no use of computing technology at all. a simple but useful example is the jigsaw induction activity described by king and saunders (chapter 6). rather than just giving information to new students verbally or on paper, it is broken down into separate information cards handed out one per student. all students are then given a questionnaire which they can complete only by interacting with others. more sophisticated is walford's caribbean fisherman (chapter 20) with 30 years of development behind it. teams decide where to fish, whether to risk dangerous currents in order to catch larger fish, whether to run a holiday hotel or make pots rather than fish at all, and so on. it seems doubtful that a computer-based version could ever match the flexibility and spontaneity of an experienced human facilitator, for example by instantly inventing new rules for fishermen who unexpectedly ask whether it is possible to run tourist trips. these kinds of activity, when organized well, are clearly very motivating. thiagarajan (chapter 5) goes one step further in arguing mat the ultimate motivator is money, and describes role play games in which actual cash changes hands. these games employ devices such as secret ballots in order to create conflicts between self-interest and commoninterest in competing teams. he writes: '[...] cash games arouse intense emotion, including interpersonal hostility. as a facilitator, you should be ready to abort the game if things appear to be getting out of control'. of course, games and simulations are not just for fun, they promote experiential learning. for example, thiagarajan's cash games are intended to bring out concepts such as co-operation, confrontation, escalation and fairness, and to exercise skills such as negotiation, persuasion and building trust. several contributors attempt to place games and simulation within a context of educational theory. kolb's experiential learning cycle is discussed in several places. another recurring theme is the importance of debriefing for transforming experience into positive and lasting learning. debriefing usually involves discussion in which the facilitator's main dilemma is to maintain balance between structure and free flow of comment. crookall points out that debriefing is the most neglected stage of simulation and gaming: 'compared to the amount that has been written about other aspects of simulation/games, very little has been written 107 reviews about debriefing' (p.10). the book goes some way towards making good this deficit. also running through several papers is the issue of the nature of education and its future, especially higher education. all contributors see simulations and games as a means of improving learning quality. evans and jewell (chapter 17) answer sceptics who might not regard 'games' as a serious academic endeavour. cudworth (chapter 19) deplores the stigma of 'abnormality' placed upon creative teaching and the hold of conformist tradition upon higher education, which still relies on lectures as the main form of learning. yorke and mccormick (chapter 14) report a workshop in which senior university staff planned institutional strategies beyond 2001 in order to submit proposals for an imaginary funding initiative. changes to the nature of teaching and learning featured heavily in the strategies created. computing technology is by no means absent from the book; it plays a part in more than half of the games and simulations described, especially in business games in which teams manage simulated organizations competing in simulated marketplaces over a series of time periods. the main role of the computer here is in evaluating performance at the end of each time period. examples include fawcett's account of a team game for hospitality students who manage restaurants competitively (chapter 11) and gray's cost-accounting game which simulates competition between manufacturing companies (chapter 12). burgess (chapter 7) traces the evolution of these business games, and the gradually increasing importance of computers within them. he sees this trend continuing into the future, with expert systems and decision-support systems to aid both organizers and players, and multimedia to give greater realism. gray's cost-accounting game actually runs on microcomputer networks, and he sees no reason why games could not be run with 200 teams and 1,000 globally distributed participants. i found the quality of papers variable. some seem a little out of place in the collection. one or two would have benefited from strict editing for length. there is no index. but these things aside, the book provides a rich source of ideas from experienced designers and users of educational simulations and games. we in alt, who sometimes perhaps focus too narrowly on the design of software rather than the design of broader learning experiences involving computers, might be able to use some of these ideas. robert ward, university of huddenfield simulations a handbook for teachers and trainers by ken jones, london, kogan page (third edition), isbn: 0-7494-1666-1, 1995. as a teacher, how well do you prepare your students for the workplace? few would deny their ability to convey the basics of their own special subject. however, the social context in which professional graduates find themselves involves a variety of different behavioural skills, often beyond the technical aspects of the degree subject they have read. in attempting to prepare students for a life in a workplace that is dependent on communication and negotiation skills, where they are largely selfmotivated, and perhaps expected to perform single-handed in strategic positions, benefit can often accrue from the experience of a simulation, described by ken jones as a classroom event in which (1) the participants have functional roles survivor, journalist, judge, fashion designer, prime minister; and where, (2) sufficient information is provided on an issue or problem to enable the participants to function as professionals, using real-world ethics. if an appropriate level of autonomy is granted to the participants, such events permit powerful learning by one's mistakes, making them well worth pursuing in most learning areas. and the immense effort of creating a satisfactory simulation can be relieved by use of published simulation material: jones refers to several sources, including his own previous work in this area. following a thorough grounding in the characteristics of simulations (where the objective is to experience professional activities) and contrasting these with those of a game (where the object is generally entertainment), jones goes on to describe how simulation was first used in war and business before being transferred to the educational arena. personal development and social/cultural awareness are other applications that can benefit from a classroom simulation. indeed, it seems difficult to imagine an area of learning that would not benefit by this immersion technique. for anyone with an interest in design, choice or use of these activities, this little book holds a real wealth of information in an entirely readable format. while many authors have attempted to formally assess the value of simulation in education, there is arguably more to be gained from the flip-side: using simulation in assessment 108 alt-j volume 4 number i rather than assessing simulation. a major role of classroom simulation lies in student-evaluation for example, of oral skills and behaviour. however, one note of caution: a section on computer-assisted simulation presents this only as a supporting device for the perceived primary role of simulation: human interaction, not as a costeffective technological simulation of complex systems in training prospective users. jones concentrates on the social and behavioural aspects rather than the recent innovations of simulation in physiology or surgery; situations where the ability to learn by one's mistakes has naturally been somewhat problematical and limited to unfortunate animal work. a reader of modem simulation will find little here to inspire confidence in computer technology providing virtual learning environments for those heading for careers in control of large or vital systems such as trains, boats and aeroplanes, power-generation plants, hospitals, or even electronic design. perhaps this is due to a risk of over-simplification in simulations of this type, and the relatively recent introduction of computersimulation and modelling techniques for such specialized training functions. nevertheless, if you teach any subject, and hope in vain that the students will somehow absorb the socalled transferable skills of communication negotiation and/or the art of listening, then this book will be of considerable value. george kernohan, queen's university, belfast design and production of multimedia and simulation-based learning material edited by ton de jong and luigi sard, dordrecht (netherlands), kluwer academic publishers, isbn: 0-7923-3020x, 1994, 253 pages. this book is a collection of papers by different authors chiefly from within the delta (developing european learning through technological advance) programme, which aims to facilitate the use of information technology in learning through the development of more effective design, production, and delivery. thirty projects are funded under delta, and eight of these are covered in this book, though in order to provide a broader view of the current status of research into learning and technology within europe, three other projects which lie outside the delta programme are also included. the book has an introduction, followed by 11 chapters, each focusing on a different project. in the introduction, de jong and sarti focus on current trends in the design and production of computerbased learning material. design and production are defined, and related to the contributions presented later in the book. a distinction is also made between 'general' courseware and simulationbased training material. again, the concepts are related to the individual contributions made later in the book. the introduction ends with a brief description of each of the contributions which make up the work. this is useful in a book of this kind for several reasons: it provides a conceptual framework for the reader, and it allows the reader to rapidly gain a detailed overview of the contents of the book, and to focus in on areas and issues of specific interest. such a useful introduction is all too often omitted from books consisting of collections of papers. the range of projects covered in the book is, diverse, but all address the issues of design and production at some level, and a number of papers deal with both these issues at a significant level (barker; van rosmalen; de jong et al; verreck and weges). barker, for example, outlines models for both learning design and for interactive learning environment implementation. other papers tend to centre primarily on either design or on production issues: a strong emphasis on design issues is particularly notable in some contributions (grandbastien and gavignet; mispelkamp and sarti; and tait), while a focus on production can be found in others (ulloa; benamou and celentano; busch et al; and hartel). a key problem with books created from collections of papers is the lack of continuity and structure. authors' styles vary, and they are producing essentially stand-alone pieces of work. this means that such books are less amenable to a 'cover to cover' approach, and considerable effort is required of the editors in order to structure and sequence contributions. although in this particular book some structure is provided by the introduction, it would have been useful if some form of sub-sectioning had been applied so that readers could more immediately focus into areas of interest to them. sub-sections are not used and, as a result, the order in which papers are organized is not immediately obvious. although all contributions are project-based, it is interesting that they successfully address underlying conceptual issues and problems. as a result, this book will be of general interest to anyone involved in the design and production of computer-based learning materials. stephen richards, university of teesside 109 reviews how to motivate people by twyla dell, london, kogan page (second edition), isbn: 0-7494-18230,1995, 96 pages. this is one of the books in kogan page's better management skills series. it is designed to be used as a self-study guide, and offers managers a series of practical tips on how to increase employee commitment it centres on maslow's hierarchy of needs, and the theoretical base on which the book is founded is thus very narrow. dell does not deal with the work of many of the motivation theorists which would be expected to be part of a general text on the subject (e.g. herzberg, mcgregor, mcclelland), and because of this the book cannot be considered an in-depth treatment. however, through the clever use of activities and exercises, dell brings the reader face to face with the realities of their personal management style and the expectations of their subordinates. the book has four parts, which lead from theory to practice. one immediate criticism is of part 3, entitled 'five steps to success'. it is wise to be somewhat sceptical of any book which purports to have the answer to management success. perhaps it would have been better for dell to use a more considered approach, drawing readers into full knowledge of the complexity of motivation theory, and to introduce them to the range of methods available to create an atmosphere within an organization where motivation is encouraged. another possible flaw is the assumption that employees are capable of achieving -100% productivity in the workplace. i have the niggling feeling that if we could get 100% effort out of staff in the academic workplace, they would be unlikely to give the extra 50% to 60% which is necessary for the survival of higher education. this highlights an obvious weakness in the book that of how to motivate staff where a significant proportion of their effort depends on goodwill a weakness no doubt brought about by the author's experience in running a commercial company where motivation factors (e.g. bonuses, perks and promotion) are more readily available than in education. but perhaps it is too much to expect dell's book to cater for all eventualities. part 1 ('close the commitment gap') introduces the reader to maslow's hierarchy of needs, and begins the process of applying this knowledge to the reader's specific circumstances. the exercises in this section are geared to creating awareness of how one, and one's subordinates, are performing. part 2 ('how to empower staff) seeks to convey the three phases of empowerment, which in the opinion of dell are 'to give someone the power to: judge act command*. as with part 1, the practical exercises are immensely useful, enabling readers to understand their particular management style, and how they might best empower those members of staff for whom they are responsible. part 3 consists of those 'five steps to success'. although 'instant success' books of this nature abound in the area of management, this text does contain some very practical wisdom. the five steps which, in dell's opinion, will lead to success are: learn to lead; examine expectations; act as though you care; respect employees as professionals; and never stifle personal growth not exactly 'earth shattering', but they do offer some common-sense tips towards creating a more contented, productive workforce. having confronted the reader with the basis of motivation theory and its application in the workplace, part 4 ('practise what you've learned') attempts to ensure that what has gone before is not forgotten, but is put into practice. this book is aimed at anyone who has staff who are under-producing in the workplace, either by an act of will, or by being prevented from doing so by circumstances or lack of training. the practical, well-thought-out exercises are useful in forcing managers to be honest with themselves in establishing reasons for lack of motivation. although they are designed to be used by individual managers, some would also be extremely useful in gaining the opinions of the workforce. this would require some re-wording, but as all the exercises are brief, it would not present too much of a problem. the book would have benefited enormously, however, from the addition of a bibliography, which would have added weight and authority to the arguments postulated. readers would then have become aware that the ideas are based on sound theory, as well as practical experience. brian sloan, napier university, edinburgh evaluating and assessing for learning by duncan harris and chris bell, london, kogan page, isbn: 0-7494-1301-8, 1994, 204 pages. good teaching requires evaluation. this means measuring how well the teaching is going, and identifying areas that need to be improved. evaluation can also identify what the learners need 110 volume 4 number i to know. but an easy criticism of any book on evaluation is to ask what evaluation the book itself has undergone. how well have the authors addressed the needs of the readers? did they elicit the views of readers? it would seem that the ideas harris and bell present have been developed through discussion and criticism with teachers and learners. each section has a discussion on 'difficulties and criticisms', so their ideas do indeed seem to have been evaluated. and the book is in its second edition, so there has been plenty of time for evaluation. the structure of the work has been retained, but it has also been built upon: this second edition has more pages, case studies have been expanded, and sections rewritten and extended. this has improved an already good book. the authors anticipate that the readers of the book will include mature students on advanced courses, tutors of advanced courses, teachers or groups of teachers undertaking curriculum development, and groups of trainers undertaking development of training material. there are five sections, each with some suggestions for group activities, the discussion on difficulties and criticisms mentioned above, and a useful annotated bibliography. harris and bell attempt to cater for different types of learners by providing overviews in the form of concept maps, and also a conventional table of contents. the book is centred on the needs of the learner, providing practical, well presented ideas on how to assess and evaluate learning. this is not, however, a how-to guide; rather, it is more of a discussion of the issues necessary for an effective learning environment much of the discussion is on the context of learning, with some advice on how to assess or evaluate. section 1 is devoted to the needs of the learner. the types of learner follow honey and mumford's 1982 classification of 'activist, pragmatist, reflector, and theorist'. harris and bell describe how each type of learner would learn best, then go on to describe different types of learning. there are many ways of describing these, but whatever the terminology used, most readers will be familiar with memorizing, decoding, and creating. the type of learning i was not familiar with was termed 'loving'. this is the learning that occurs in groups, and harris and bell suggest that it is necessary for the individual and other members of the group to identify each learner's needs rather than depending on learning that is driven by the teacher. section 2 deals with evaluation, and fitting the system to the needs of the learner. it is where the underlying theme of the book is discussed. needs analysis will help to avoid a mismatch between the learner and the content or the approach to learning. one of the methods discussed is the delphi procedure, a way of collecting information and combining ideas from a number of individuals. the information collected is fed back to the group to get more information and to refine ideas, the process continuing for as many iterations as necessary. in section 3, the authors address the topic assessing for learning, and for me this section was the core of the book. i wanted to find out how to assess learning, and i was looking for techniques that i could use to identify learning gains, techniques that would show that the learners' ideas had changed, that their understanding had developed, or even that knowledge had been gained. however, the section focuses on reasons for assessment. it seems that learning is more complex than my ideas would allow. cook-book techniques to assess learning are likely to miss important points, because issues central to learning have not been addressed. harris and bell discuss why, how and what you should assess. they assert that assessment should be focused on active involvement by the learner on learning. their philosophy is to feed back information to the learners, assisting them in learning and giving them responsibility for their own learning and assessing. in section 4, entitled 'role of learners and teachers', harris and bell adopt a musical metaphor for teaching: composer, conductor, performer, critic; and summarize the learners' role as receiver, detective, generator, and facilitator. they take the view that much of the communication between teachers and learners should be through the medium of assessment and evaluation. section 5, 'meeting the needs of the learner', discusses different forms of teaching and learning. harris and bell develop a model in which there is two-way communication between teacher and learner and also between learners. they stress the importance of group work. the learning context and evaluation are intertwined: the context will limit the evaluation that can be applied. this book discusses the context, and explains appropriate evaluation and assessment. an ideal learning situation for harris and bell would be one in which the teacher negotiates the learning task and assessment strategy, and resolves between them the best approach for the learner. i l l reviews a ripping good yarn? no, but it is clear and readable, with case studies and different approaches to revive waning interest i would recommend it for anyone who is interested in improving teaching and learning. john milne, university of aberdeen html manual of style by l. aronson, emeryville ca, ziff-davis press, isbn: 1-56276-300-8, 1994, 132 pages. the internet and world wide web (www, or the web) have now become valuable learning, teaching and training resources. consequently, over the last few years books on these topics have become very popular. essentially, these deal with three basic issues: the underlying theoretical and technical matters relating to using distributed networks; accessing internet and web services; and web publishing. an important aspect of publishing material on the web is the use of an appropriate markup language so that search engines and web browsers (such as lynx, cello, mosaic and netscape) can access the information and interpret it in the correct way. the markup language used on the web is html (hypertext markup language) which, through the use of an appropriate dtd (document type definition), is essentially a subset of the iso 8879 standard sgml (standard generalized markup language). obviously, anyone wishing to publish material on the web will find some knowledge of html useful. this monograph provides a helpful introduction to the topic. the book contains just four chapters, plus three appendices and a subject index. the opening chapter ('what is html?') is very short and provides a brief introduction to html, hypertext and hypermedia, mosaic, the web and the internet. chapter 2 is more substantial, and is devoted to the html (level 2) language itself. it covers such issues as html syntax, formatting tags (for headings, paragraphs, lists and styles), anchors and links, images, and forms (which facilitate input from users). in chapter 3, the author discusses how to write html documents. the chapter is organized into four major sections which together cover such issues as general principles, identifying good html style, how to create a home page, and the conversion of existing documents into html format. the final chapter is by far the largest, and is devoted to a series of eight case studies. in each case study, the author describes various existing web pages accessible, then goes on to discuss the html encoding underlying their appearance. the examples presented in this chapter include a personal home page, typical home pages for small and large organizations, an electronic brochure, a web survey, and pages for a number of subject guides and specialisms. the site addresses (urls) for the examples are given and, surprisingly, they can all be accessed relatively easily. i used a fullscreen, text-based browser (lynx) to access them. indeed, i explored the home pages described in the book using an old dos-based computer (an amstrad 1640) as a remote terminal running at 1200 baud (using vt100 emulation). not surprisingly, however, most of the pages (except the web survey) were in many ways different from those described by aronson. obviously, since the book was originally published, much of the material has been updated and restructured. i was interested to find (p. 107), the example used to illustrate the home page for the san francisco bay area chapter of the acm's special interest group on computer-human interaction (baychi) was advertising a talk entitled 'too much hypertext or too little?' by jacob nielson (to be given on tuesday, 11 october 1994). when i accessed this same page a year later (while writing this review) i was excited to see the baychi page advertising a lecture to be given by two of my hci 'heroes' (william buxton and stuart card) entitled 'radical human-computer interaction' scheduled for tuesday, 10 october 1995 at 7.30 pm! wouldn't it be lovely if the internet could be used to transmit this lecture 'live' to interested people in europe? unfortunately, herein lies some of its basic limitations. maybe the future information superhighway will provide enough bandwidth for this type of thing to happen. who knows? in addition to the case study examples described above, the three appendices at the end of the book are also very useful. the first provides a quick reference guide to html (based on the html 2.0 dtd specification). the second gives an overview of html+, describing many of the features and extensions that are to appear in new versions of the language. the third appendix gives details of various html resources (such as guides and references, browsers, conversion tools and listservers), many of which can be obtained using the web itself. unfortunately, since the book was published, the info.cem.ch documents have been relocated to www.w3.org. this meant that my initial attempts at accessing some of the guides and references failed when i entered the urls quoted by larry aronson. 112 alt-j volume 4 number i overall, although the book has some very minor limitations (such as out-of-date urls and the occasional spelling error), i found it quite interesting and fairly easy to read. the author is thorough in his explanations and has chosen some good examples in order to illustrate the points he wanted to make. it is my belief that the text will make a useful first reader for those wishing to launch themselves into markup techniques and web publishing. it is also a book that i would definitely recommend to any student who wants to find out about html and what it has to offer. philip barker, university of teesside film and television in education edited by chris dry, london, chapman & hall (bluprint), 1995, isbn: 1-857130-16-2, 341 pages. this is the second edition of the handbook of the british universities rim and video council (bufvc). the first edition dates back to 1991; this new one is more extensive. it consists of about 100 pages of short articles on various aspects of film and video as educational media, including (but by no means limited to) the new digital-video technologies of the last few years, and about 200 pages of directory. the directory consists of reference entries on a wide variety organizations whose business, in one way or another, is audiovisual, and on distributors of film, video and related media, classed by subject. there are also lists of sources of multimedia products, library suppliers of audio-visual materials, equipment suppliers, european audio-visual projects, and courses and events. supplementary information includes a section on media legislation and reports, and one on the activities of the bufvc. in all, then, this work offers a pretty wide coverage of the field, and could be very useful to developers of courseware, who would do well to keep it on their shelves. for instance, if you need some topquality sound recording, some 1-inch video mastering, some graphic design, some professionally produced video sequences, or just the loan of a good camcorder, you will certainly find good sign-posts in its pages, as you will if you want to know your nearest audio-visual centre (university or otherwise) and precisely what it has to offer. equally, if you need to know exactly what a particular audio-visual organization can do for you, from the arts council of great britain to the writers and scholars educational trust, you will find enough to give you a fairly good idea, as well (of course) as all relevant contact details. alt is there too, i am very glad to say. also of interest to many alt members will be the lists of uk distributors (116 pages, making up the largest section of the directory) and what they can offer everything from educational cd-roms and videos for sale or hire (there is a good list of available catalogues, too) to printed publications on resources. the articles preceding the directory are of varying interest to educationists. some, meant of course for non-specialists, merely re-state themes (such as interactivity and technology) which will be well known to alt members. others, however, offer interesting insights into aspects of educational technology not often dealt with in journals and newsletters about computers and education, such as the future role of educational television, or the history of moving pictures and education (a short but excellent contribution by murray weston, the director of the bufvc), or how to research film and television collections in europe). thus, this handbook, while primarily of use a reference work, also offers a good overview of audio-visual aspects of educational technology. it is certainly worth adding to one's collection, and is the sort of thing which will come in useful when you least expect it. gabriel jacobs, university of wales swansea multimedia technologies and future applications edited by damper, r.i., hall, w. and richards, j.w., london, pentech press publishers, 1994, isbn 0-7273-1320-7, 267 pages. this book is the proceedings of the ieee (institute of electrical and electronics engineers) international symposium on multimedia technologies and future applications, held in 1993. as such, it is an eclectic collection of papers on the development of multimedia technologies and their applications, ranging from theoretical overviews to highly technical reports. it consists of 19 papers by different contributors from industry and academe, representing a wide variety of backgrounds, from engineers to linguists, computer scientists to broadcasters, publishers to human-factors experts. this may seem a disparate group, but in fact it actively reflects not only the breadth of the appeal of multimedia technologies and techniques at the moment, but a general consensus in almost all disciplines that we are on the brink of a multimedia revolution. it also demonstrates the often claimed 'interdisciplinary' nature of multimedia in that the techniques perfected in one 113 reviews discipline are often applied in other, quite different, areas. the papers are divided up into four sections. each contains a helpful introduction which summarizes the content of the papers which follow, and which help to put them into the context of the work as a whole. these introductory chapters are a very useful addition when material which is comparatively so varied in scope is represented in one publication. another useful aspect is the inclusion of an introduction or abstract at the start of each individual paper. i have one minor quibble, however, with the lack of chapter numbers within the individual sections. this makes crossreferencing rather difficult the editor of section 1 himself confused the sequence of two chapters in his introduction. the collection begins with an introductory and general foreword looking at the implications of multimedia in publishing, and comparing the information revolution which is facilitated by multimedia to the technological revolution created by gutenberg. this is a concise and clear introduction to the many political and social issues intrinsic to the 'multimedia revolution*. part i, on core technical issues, is primarily concerned with data compression, particularly with reference to image quality and speed of delivery. although at first frankly terrifying in its technical complexity to the reviewer with a non-technical background, it quickly becomes apparent that issues of data, image and video compression are something all of us involved in multimedia development need to be appraised of, and this collection provides an excellent overview of the problems and some possible solutions. the first paper presents an identification of the components necessary for a systems approach to solving the problems of standardization across machines and networks using multimedia communications. having identified with the systems analyst, recognizing that their concerns are of interest to all involved in multimedia development, we can more confidently approach the following papers. these include a new technique for high speed image coding, an overview of design issues pertinent to the development of a low bit-rate audio/data/video multimedia system, an exploration of an algorithm for image coding which could help solve problems of storage and compression of images, and a practical technique for image processing with an origin in pure mathematics which is compared to more conventional coding methods based on simulation results. the final paper in this section describes acom replay, software-only full-motion video from cd-rom, a patent-pending technique which has the potential to be used in the production of low-cost educational software. all six papers in this section deal with highly complex technical issues in an clear and easy-to-follow manner. they are as good a summary of complex techniques as i believe will be found in one volume. part ii, on audio-visual delivery systems addresses implementation problems inherent in audio-visual systems in the digital domain, especially processing power and bandwidth. the first paper describes a collaborative project to develop a multimedia environment which will allow high-quality video, data and audio connections between remote sites using isdn, supporting a multimedia system in both local and remote networked environments. this is followed by an overview of a technique to store high-quality picture information in a way that is compatible with the needs of distributed multimedia environments. this paper characterizes the components of, and interactions within, a multimedia environment. the final paper in this section, entitled 'the multimedia microwave oven' is the perspective of a television producer looking at the new multimedia communications systems which are set to replace what was not so long ago the only audio-visual system in use broadcast television. this is a reflective paper in the 'where are we and how did we get here?' mould. the author (macgregor) reflects upon the political, social and cultural implications of the adoption of multimedia technologies, and surmises the infrastructure required before the 'multimedia revolution' which we are constantly reminded is upon us can really begin. part hi, entitled human-computer interface, looks at the 'human' issues how people interact with technology rather than the technical concerns of the previous sections. all the papers in this section address the issues involved in creating high-quality multimedia materials, paying particular attention to questions of why and for whom this material is created. in his thoughtful paper, 'not a cottage industry: understanding the scale of multimedia production', bruce ingraham addresses the myth that multimedia is a quick and easy solution to educational problems. as the developer of france interactive, ingraham is well placed to realistically assess the real cost and scale of developing intelligent educational multimedia. his argument is that the investment in educational technology must include an investment in the structures, skills and 114 alt-j volume 4 number i tools to support its use for real benefits to be gained. other papers in this section report on specific ways in which human interaction can assist in the development of multimedia technologies, including voice and facial image integration for person recognition, interaction between a patient and psychiatrist to analyse the use of video telecommunications, the way in which users interact with audio and visual information in the workplace, and the perceptions of language, sound and vision which may contribute to the development of formal and effective multimedia applications. part iv, applications, presents examples of applications in use at the moment, any or all of which may well become prototypes. all represent examples of good practice in the use of multimedia for specific tasks, practical or pedagogical. again, contributors come from industry and academia. there is a description of a system in use by lloyds register to enable users to make quick and interactive decisions about damage on board a ship, and a contribution describing the microcosm project, addressing media integration issues within open-media systems. the papers in this section illustrate what wendy hall herself says in her introduction to this section: multimedia applications are today much more than simply presentation systems. multimedia information systems are merging with computer technologies to the extent that soon the two will be inseparable and interchangeable. multimedia technologies is a timely collection, one which demonstrates the huge impact of multimedia technologies on so many disciplines. it demonstrates clearly that there is more to multimedia than theory that there is a need for the techniques used in multimedia development to catch up with the technology available. multimedia is a term which is bandied around to the extent that it seems to mean all things to allcomers few of us stop to think about the technical issues which are so crucial in successful development and implementation of multimedia. this collection makes no claims to being exhaustive, but is a valuable snapshot of the state of play in the technological developments which make it possible for us all to look to the use of multimedia in our own disciplines. lorna hughes, arizona state university 115 educational software reflecting two philosophical approaches to ethics education marike hettinga* and betty collis** * now at department of social studies, noordelijke hogeschool leeuwarden, postbus 1018, 8900 ca leeuwarder. the paper refers to work done prior to taking the appointment at the noordelijke hogeschool leeuwarden.1 ** faculty of educational science and technology, university of twente, postbus 217, 7500 ae enichede; email: collis@edte.utwente.nl. abstract ethics education can vary considerably in its instructional strategies based on differences in the theoretical positions underlying the approach to moral development being stressed. two such approaches are the 'justice' approach as exemplified by kohlberg's six stages of moral development, and the 'care ethic' approach as exemplified by gilligan's work on empathy as a base for moral decision-making. each of these approaches can be demonstrated through different instructional strategies in the ethics education course, but each strategy is often difficult to execute in practice, given time and resource constraints. introduction problems in the delivery of ethics education ethics education in university is not a field in which one can easily imagine extensive use of educational software. there are, for instance, no drill-and-practice programs in ethics education because there are no unambiguous answers to ethical problems. that is what ethics is all about: complex moral dilemmas without single solutions. ethics courses usually offer theories from great philosophers about how to deal with moral dilemmas. and usually they offer students a chance to deal with a dilemma themselves. it is in this second part of ethics education where didactic difficulties often arise. in order to deal with a dilemma, the students need to be able to discuss this dilemma with fellow students. therefore, a class oriented toward such discussion should not consist of more than a small number of students, perhaps 20. this limit is in many higher-education settings exceeded with the consequence that only a small number of students get a chance to speak up. cost cutbacks restrict the number of tutor-led seminar groups that can be formed out of large lecture groups. 4 alt j volume 2 number 2 another difficulty arises in the experience of a real dilemma. when does a dilemma offer a real-life experience? can a description of a case in a few lines be called a real dilemma? in other words: what is needed to get the empathy and engagement of a student in a complex case? two projects involving educational software for ethics education there appear to be useful possibilities for educational software in the context of these two difficulties. one application of educational software could be to provide a substitute for the dialogue aspects of face-to-face small-group discussions, and a second application is to create an electronic learning environment in which students are able to experience a real-life moral dilemma from a variety of perspectives and with a number of sensory modalities. in this paper, we will describe two projects for ethics education in which we have been involved, and which were concerned with the design, development, and evaluation of educational software for the two types of applications described above. the first is the colada project of the educational centre at the university of twente in the netherlands (de mink, 1991). in this project, software is created which offers students the opportunity to participate in a heated discussion with electronic partners about a described case involving an ethical dilemma. the second project is called the theoria project (watkins, 1992). it is being carried out by the center for the advancement of applied ethics at the carnegie mellon university in pittsburgh, pennsylvania, usa. the emphasis in the software of the theoria project is not on discussion or rational arguments, but on being able to offer students a close-to-real-life experience of a case involving a complex ethical dilemma. the conceptual backgrounds of these two projects seem to be contradictory: colada stresses rational discussion, while theoria emphasizes an increase in students' sensitivity and empathy. however, in ethics education both approaches are necessary. therefore, each project addresses a different difficulty in the delivery of ethics education in the higher-education setting. in this paper, we will describe the conceptual background as well as examples of the software developed in the two projects, and give an evaluation of the software from different perspectives of educational effectiveness. furthermore, we will raise the question of combining both conceptual orientations in one electronic learning resource. this could be an efficient way to deal with the two difficulties compensating for lack of opportunity for intensive face-to-face discussions, and inadequate exposure to the complexity of an ethical dilemma in the delivery of ethics education that were noted above. the colada project kohlberg and the six stages of moral development the theoretical background of the colada project at the university of twente in the netherlands lies in the philosophical position called the 'ethics of justice' or of reason. a central person associated with this approach is lawrence kohlberg (1981). his theory is 5 marike hettinga and betty cottis ethics education and educational software built around the concept of a fixed and universal sequence of moral development. according to kohlberg, this moral development can be divided into six stages. every person passes through a number of these stages. however, their rate of velocity of evolution may differ and so also their final stage of development. some people reach the highest stage while others do not get any further than, for instance, stage 3. kohlberg's six stages are briefly indicated in table 1. table 1: kohlberg's six stages stage description stage 1 right is literal obedience to rules and authority (parents), avoiding punishment, and not doing physical harm. stage 2 right is serving one's own or other's needs and making fair deals in terms of concrete exchange: "you scratch my back and i'll scratch yours'. stage 3 the right is playing a good (nice) role, being concerned about other people and their feelings, keeping loyalty and trust with partners, and being motivated to follow rules and expectations. stage 4 the right is doing one's duty in society, upholding the social order, and maintaining the welfare of society or the group. stage 5 the right is upholding the basic rights, values, and legal contracts of society, even when they conflict with the concrete rules and laws of the group. stage 6 this stage assumes guidance by universal ethical principles that all humanity should follow. principles are universal principles of justice: the equality of human rights and respect for the dignity of human beings as individuals. kohlberg, 1981, pp. 409-12. how did kohlberg come to the definition of these stages? the stages reflect a division of the kind of arguments people use when confronted with a moral dilemma. kohlberg did research in many different contexts and cultures, and concluded that there were six different and sequentially more adequate ways of dealing with an ethical dilemma (kohlberg, 1981). the six stages can be placed in a hierarchical order. each stage reflects a higher degree of abstraction and is more widely applicable than the former; thus according to kohlberg, each stage is ethically more adequate than those that have preceded it. the arguments used in kohlberg's highest stage are based on what are defined as the principles of justice: the equality of human rights and respect for the dignity of human beings as individuals (kohlberg, 1981). these principles are universal. they can be rationally derived and argued. sensitivity and emotion should not play a role in moral decision-making, since such feelings can differ from person to person, and may be unable to rise above the specifics of a certain situation. so, according to kohlberg's approach to ethical development, feelings and emotions will never lead to universal ethical principles. universalization is the central issue in the highest stage. a guideline here is the golden rule: do unto others as you would have them do unto you. 6 alt j volume 2 number 2 ethics education reflecting kohlberg based on this theory, kohlberg developed an educational method which he called interactive progressivism (kohlberg, 1981). the essence of this method is that the individual's progress through the different stages of moral development needs to be stimulated. since this development follows a natural sequence, ethics education should be free of indoctrination. moral development can be stimulated through interaction between the student and the environment the student should actively deal with moral situations. this can be done by confronting the student with a dilemma for which he or she does not have an immediate solution. thus his or her present stage of development is not adequate. next, the students should be confronted with arguments from a higher stage than their current level. the students will experience that these arguments are more adequate to deal with the problem than are arguments of a lower stage. intense discussion with a skilled mentor is a good means for confronting students with arguments from a higher stage. educational software reflecting kohlberg's philosophy of ethical development the colada shell for software to engage the student in dialogue about an ethical dilemma the colada project being carried out at the educational centre at the university of twente is based on the philosophy of justice and on kohlberg's conception of a fixed sequence of stages of moral development. the centre has been contracted to develop educational software for ethics education based on the educational method of kohlberg colada 2.30 e demo-version (c) p. hoatert 21-3-1994 12il8t51 -wastedyou are the head of the marketing department of a chemical i i round company. on a certain day you diacover by accident that your ii 1 company dumps on a regular basis large amounts of toxic waste on a dumping site. although this happens secretly, it happens by order of the direction. i i tutor ii oh fchat do you think you should do? nothing? or trying to stop these toxic vaste dumps? select your choice beneath. several colleagues will react on your choice. select 1 1 i will not undertake any action. 2 i have to try to stop these dumps. 3 it depends if i undertake any action. 4 i am ready to decide. scroll : pgup-pgdn / f-1 . f4 tutor on f5 quit f10 colada figure 1: the structure of colada 7 marike hettinga and betty collis ethics education and educational software colada 2.30 e demo-version (c) p. mostert 21-3-1994 12:19:55 -wastedyou are compan compan dumpin order what d these select the head of the marketing department of a chemical -compute hot any action? you are the witness of a crime which has far-reaching consequences for the environmenti round —12 -tutoryou have to deal with a passionately reacting colleague who obviously condemns you for undertaking no action. tor n ect 1 scroll : pgup-pgdn / t"t ready : spacebar figure 2: the discussion within colada colada 2.30 e demo-version (c) p. mostert 21-3-1994 12:20:24 -wastedyou are the head of the marketing department of a chemical company. on a certain day you discover by accident that your company dumps on a regular basis large amounts of toxic waste on a dumping site. although this happens secretly, it happens by order of the direction. what do you think you should do? nothing? or trying to stop these dumps? select your choice beneath. several colleagues will react on ; i round 3 tutor on select 6 1 5 3 4 5 6 your decision in my opinion... as the responsible person for the marketing policy i i i should tell the direction about this problem. should do everything in my power to prevent damage the environment. i would not undertake any action if it concerns just amount of waste. i am ready to decide. think on a small scroll : pgup-pgdn f1 cancel f5 quit figure 3: the final decision 8 altj volume 2 number 2 and reflecting the problem of inadequate opportunities for students to engage in face-toface intensive discussions with a skilled mentor, given increasing class sizes and fixed staff resources. the software is being produced through the use of a relatively simple authoring package, also called colada 2. this package provides a shell into which a brief textual description of an ethical dilemma can be entered, and response options reflecting different stages of moral development can be offered to the respondent. following each choice of response by the student, a new set of options and associated 'discussion' can be offered by the software. this means that within a preset didactical structure, the contents can be varied to reflect as many different ethical dilemmas as the instructor wishes the students to consider. the following description shows the general structure of the colada program, through one of the cases developed by the first author of this paper. an example of an ethical dilemma presented by colada the student is shown a case containing a moral dilemma, in this example a dilemma concerning secretly dumped chemical waste (figure 1). the students are asked to imagine that they are one of the central figures in this dilemma, and that they will have to make an ethical decision in the case. the students can choose one of a number of response possibilities. next, another 'person' from the case will react to the response choice of the students (figure 2). this process, involving pre-programmed branching options, can be continued a number of times. if the students wish, they can also ask for a reaction from the 'tutor'. the tutor's reactions have been entered by the instructor previously, and give, for instance, an explanation about the arguments that were used, or give a theoretical perspective. when these reactions are removed from the screen, the students can engage in further discussion by reacting to the comments made by the 'discussion partner' in the program. they are able to do this by choosing from a new number of pre-programmed, multiple-choice possibilities. whenever the students want to stop the discussion, they choose the option ready to decide... a new box will appear in which the students will type a description of their final decision (figure 3). after this, the students are shown an explanation of the case. if they want to, they can print their decision and the explanation. evaluating the use of the colada program the first author of this paper participated in, or carried out in full, various evaluations of different cases entered into the colada shell. one evaluation focused on the educational design of the software itself, and another on student reactions to the program as a learning experience in ethics education. in addition, another evaluation focusing on the effectiveness of the program on advancing students' levels of moral development was also being carried out, and its preliminary results were made available to the authors. educational design aspects of the program we studied programs generated by the colada shell according to various features relating to usability (nielsen, 1993), user-interface design (shneiderman, 1987), and instructional design (alessi and trollip, 1985). these perspectives related to flexibility and adaptability, user-friendliness, feedback adequacy, and instructional-design choices. 9 marike hettinga and betty collis ethics education and educational software the evaluation is described in detail elsewhere (hettinga, 1993a,b); some of its major conclusions follow. • the flexibility of the program is good. the contents can be varied in many ways in that the instructor can in theory enter as many different cases as he or she wishes. • however, this content flexibility does not automatically imply that the instructional flexibility of the program is also good. the content may be flexible, but it must be entered according to a very complicated and inflexible structure. instructors who want to adapt the program to cases relevant to their specific situation will not have many technical problems, but they might find the requirements of the structure itself too complicated to realize. partly, this is because of the considerable time needed to think through many different response options, each apparently representing different stages of moral development, and to express each option in parallel form in a string whose length can have only limited variability. • from the learner's perspective, a number of aspects of the program were not userfriendly. it is not always clear how one should go on in the program and the help function needed considerable improvement. however, these points could be changed within the shell program itself, with some effort. • the feedback options offered by the colada shell are good. the 'discussion partner' reacts first, and second (if it is requested) the 'tutor' reacts. this structure gives the possibility of dividing the reactions offered by the program into concrete case-based reactions (given by the 'discussion partner') and abstraction reactions (given by the 'tutor'). the 'tutor' could for instance refer the student to relevant literature. all of this potential, however, depends on the skill of the instructor in entering appropriate dialogue into the shell, a process (as noted above) that is timeconsuming and conceptually challenging. • the layouts of the screens are functional, but of limited visual appeal. the screen is divided into a few boxes, each with a pre-determined content. this gives clarity to the students. there is, however, one box which could be emphasized more. this is the box at the bottom of the screen containing the possible command options. in its current presentation, the box could be overlooked (see figures 1,2 and 3). student evaluation of the program as a second approach to evaluation, we asked a sample of six students from the faculty of business studies at the university of twente to work with a case entered in the colada shell involving the production of a malfunctioning and therefore dangerous laser drill. after discussing this case in the colada shell, they participated in an interview about their reactions to the use of the program as a learning experience. we also asked the students to indicate if they thought they had learned something from the experience. a summary of the student reactions follows; as before, a detailed review of this aspect of the evaluation is available in hettinga, 1993a,b. • the students thought they had strengthened their ability to discuss ethical problems and to find convincing arguments through the experience of interacting with the program. besides that, the students said that the experience of the detailed dialogue offered by the program gave them a better idea about what ethical dilemmas really are and how they can be seen from different levels of development. finally, the 10 alt j volume 2 number 2 students indicated that they thought they had learned an approach to rational discussion which they could use in the future. effectiveness evaluation an additional evaluation was also carried out that looked more specifically at the effects of the program relative to students progressing to a higher level of kohlberg's moral development. this evaluation was carried out by the developer of the program. in this study, students worked about one and a half hours with the program. a few weeks before and a few weeks after this session, they gave a written reaction to the same case. the arguments of the students were analysed, and each placed in one of the six stages of kohlberg. this classification of arguments made it possible to compare the arguments the students used before their experience of the program and the arguments they used after working with the program. this comparison showed that indeed a number of students did move to a higher stage of moral development in their decisions, although the change was not statistically significant (p>0.5, de mink, 1991). a careful conclusion from this evaluation, which is still in progress, is that working with the program may have a positive effect on the students' ability to reason about moral dilemmas. the theoria project theoretical framework for the theoria project: the care ethic the theoria project at the center for the advancement of applied ethics at carnegie mellon university has a different theoretical background from the colada project. the central theme in the philosophical approach underlying the theoria project is that of care for the other person (covey, 1993). the needs of the specific person in the specific moral situation is the central issue: universalization is not a goal and should even be avoided. instead of universal and abstract rules and values, the details of a case are of critical importance. a central person in this theoretical approach is carol gilligan (1982). she claims that kohlberg looks at moral conflict too much from the viewpoint of rights. a moral conflict is in his eyes a conflict between rights. resolving the problem is a matter of bringing up arguments in favour of one person's rights compared to those of another. gilligan defines a moral problem, however, as a conflict of responsibilities involving the well-being of people (gilligan, 1982). resolving the problem is not just the weighing of rights. one needs to know the case thoroughly to decide which responsibility should be given priority above another in a particular context. to get to know the case thoroughly, one should gather as much detail about it as possible. following this, sensitivity to the particular context is needed. one needs to be open to the details, feelings, and emotions of those involved with regard to the situation. ethics education reflecting the care ethic ethics education based on this theoretical viewpoint should teach students how to empathize in cases which show a moral dilemma. students should be taught to be sensitive to the details and emotions involved in a case. a requirement for ethics education according to this method is, first of all cases, which offer the details and 11 marike hettinga and betty collis ethics education and educational software emotions of a complex moral dilemma: cases with 'thick descriptions' (rorty, 1989). a second requirement is that students should be taught how to deal with all the details and emotions from a case when confronted with these thick descriptions. vivid and thick descriptions of cases may not spontaneously bring up the possibilities of a computer as a learning resource, particularly if one's experience with educational software has been with structured and choice-restricted shells such as colada. however, modern multimedia software certainly makes thick descriptions and, as a presented consequence, empathy possible. within the theoria project various programs have been designed and developed with these premises. we were involved with two of them, one during an evaluation of use in practice, and for the other, as a designer and developer. we will describe here the one in which we were involved in an evaluation: 'a right to die? the case of dax cowart'. 'a right to die?': multimedia software for ethics education this program contains textual information as well as audio-visual material about the true case of a man named dax cowart (user manual, 1990). in 1973, donald (dax) cowart suffered burns over 65 per cent of his body, and was left blind and maimed when his car, parked over a leaking gas line, exploded. for 17 months, as he underwent agonizing burn therapy, he pleaded with doctors and nurses to stop the treatments and let him die. actual visual footage was taken of the dax cowart case, including interviews with the victim, his family, and his medical attendants. this video footage has been worked into the multimedia learning program. the program starts with a main menu (figure 4) from which all the sections of the program can be accessed. main menu press one of the buttons below to proceed with your investigation of dax's case. you can also "restart" or "quit" the program. an introduction show me a brief introduction to dax's case guided inquiries let's explore the facts and issues of dax's case. archives let me see the library of video segments about dax's case. open the notebook restart this program quit this program figure 4: the dax case main menu 12 altj volume 2 number 2 issues menu press the buttons below to begin exploring the issues f"t of dax's case. then press "return to inquiries menu" ^ to register vour final position on dax's case. mm-,mor*h»1p medical professionals' obligations thoughts of dax's health care providers about the case. pain of treatment issues about the painful nature of dax's treatment. quality of life issues thoughts of dax's quality of life before and after the accident. patient's rights and capacities issues surrounding dax's rights and ability to administer them. return 1(1 quit lii to inquiries m e n u l i i t h i s program figure 5: the issue of the audio-visual material archives menu press a button below to see that section of the archives. when you finish this section, you can "ft t main menu" or "quit tf»« program." mor* h«1p ";:h:£::::£s:s:::s;:::h:::seh:s:£^ the case take a look at the facts surrounding dax's case. the principals see videos organized by the principal people of the case. the issues see the video library organized by the issues of the case. open the notebook return to the main menu restart this program quit this program figure 6: the archives menu of self-directed exploration 13 marike hettinga and betty collis ethics education and educational software by selecting an introduction, a visual montage of dax cowart before and after his accident, and a narrative summary of the case, are presented. in addition, the introduction poses the task: to come to a reasoned position on whether dax cowart should be released from treatment to return home, in all likelihood to die, as he requests. the guided inquiries option leads the user through the narrative and audio-visual material. the guided tour starts with a review of the facts of the case and asks the user for an initial position. second, the videos are shown. they are arranged by issue (figure 5). after each video fragment the opportunity is given to reflect on what one has seen, and to affirm or revise the preliminary judgement on whether dax should gain his release. these reflections help the student deal with the complexity of the case. after these video fragments, the student is asked to give a final position. when this position is registered, a final video segment is presented for the student to ponder. after this, the student is returned automatically to the main menu. the third option of the main menu is called the archives (figure 6). this section allows one to move independently through the case narrative and all audio-visual material. in the archives, the user is not prompted for a position. the audio-visual materials are organized around the issues and principal characters of the case. the purpose of the archives is to provide the means to carry out self-directed exploration of the facts and the issues of the case. evaluation of 'a right to die?' 'a right to die?' was also evaluated by us on its educational aspects (hettinga, 1993b,c). the following are the main findings of the evaluation. • the program was very user-friendly, partly thanks to the design of its graphical user interface. • an important aspect of a large hypermedia program such as this one, is the support offered for navigation through the program (reeves and harmon, 1993). a user for instance needs to know where in the program he or she is. this has been done well in 'a right to die?'. besides that, the program offers the possibility of a guided tour. • however, another option, important for a large program (reeves and harmon, 1993), is not offered by 'a right to die?'. this function is called mapping. through a graphical representation of the program a map the user can see where he or she is, and where he or she has been before. mapping would be a positive addition to 'a right to die?', since the program is large and therefore a user may lose orientation. • another aspect we evaluated was the screen layout. the screen design is quite good in 'a right to die?'. its layout is clearly and functionally structured, and it follows a consistent plan throughout. based on this evaluation, we concluded that the educational-design aspects of the program are good. in addition, an extensive summative evaluation of the program has been done by an instructional technologist (caae, 1992). he compared three treatments of the case: with interactive video, linear video, and with text only. at the time of writing, all the data has 14 altj volume 2 number 2 been gathered but the analysis has not yet been completed. the data was collected via questionnaires responded to by students in courses at carnegie mellon where the software is being used. the questionnaire, in addition to items relating to personal information about the respondents, included a defining issues test (a standardized test of moral development), preand post-tests, videotaped think-aloud protocols, log files and delayed interviews. using the personal information and the preand post-tests, one aspect of the evaluation has been analysed completely: the students' change of attitude regarding the question: 'should dax be granted his request to die?' the students' attitudes indeed did change significantly. it appears, however, that there was no significant change due to the three different treatments. the attitude changes might indicate that the students were better able to empathize in the case after instruction than they had been before instruction. this conclusion was supported by the interviews held with students. the students realized that the case was much more complicated than they had thought in advance. they were touched by the details and emotions of the dilemma. conclusion: can we combine philosophical approaches in one software product? the description of these projects shows how valuable both kinds of educational software can be for ethics education. the colada programs teach students how to reason about moral dilemmas, and to approach these dilemmas as abstractly as possible. the theoria programs, on the other hand, teach students to empathize in a case. students should be sensitive towards the details and emotions of a specific case. the approaches seem contradictory. we think, however, that both ways of dealing with a moral problem are necessary, both in private as well as in professional life. they are complementary. students should therefore be taught both approaches. preferably, they should be shown how to use both approaches on one and the same case. how can both approaches be integrated in one program? as a starting point in answer to this question, take 'a right to die?'. abstract discussions are not included in this program. how could they be added to the content? a new section should be created in which the topic can be released from the specific context and discussed in abstract terms. we suggest that students be asked to describe the government policy on euthanasia. what kind of policy would they want and why? they can be confronted with arguments against their policy. this could be done in a discussion form such as that offered by colada. it is important that students reflect on the way the specific case with all the details and emotions affects the choice of an abstract policy. the best reflection on this point would be reached when students starting with the program discuss the abstract issue first. they approach the topic mostly by rationality; then they can be confronted with the detailed and rich case. at this point, their sensitivity is activated. when talking rationally and abstractly about the euthanasia topic, they did not realize the complexity of a real-life case. discussing the topic rationally and abstractly leaves a distance between the students and possible cases. however, when confronted with a real-life case, they empathize with it. the third part should consist of another abstract discussion about the policy on euthanasia. a final discussion should try to make the students aware of the differences between the abstract and the concrete, and of the ways both approaches can affect each other. 15 marike hettinga and betty collis ethics education and educational software acknowledgements our thanks are due to the following individuals, who hosted the research opportunities: drs frank de mink of the educational centre, university of twente, and dr robert cavalier of the center for the advancement of applied ethics, carnegie mellon university. notes 1. ir. (engineer) marike hettinga now works as a member of a development team for multimedia learning resources for a higher-education institute in the netherlands. dr betty collis is a faculty member in the faculty of educational science and technology at the university of twente in the netherlands. this paper reflects work done in fulfilment of the graduate (engineer's) degree by the first author previous to her employment under the supervision of the second author under a special collaborative arrangement between the faculty of philosophy and social studies and the faculty of educational science and technology. 2. developed at the faculteit der rechtsgeleerdheid, rijksuniversiteit limburg. for information, contact fokke fernhout. references alessi, s.m. and trollip, s.r. (1985), computer-based instruction, englewood cliffs nj, usa, prentice hall. caae (1992), 'the dax project, review of the design of the study', internal report on an evaluation, center for the advancement of applied ethics, carnegie mellon university, pittsburgh, usa. covey, p.k. (1993), 'of balloons and bicycles, multimedia and ethics', paper for the 1993 computers & philosophy conference of the carnegie mellon university, pittsburgh, usa. gilligan, c. (1982), in a different voice: psychological theory and women's development, harvard university press, cambridge, mass., usa. hettinga, m. (1993a), 'het colada-project: bedrijfsethiek op de computer, verslag van een stage bij het onderwijskundig centrum', unpublished report on an internship at the educational centre, university of twente, the netherlands. hettinga, m. (1993b), 'de computer als ethicus, een onderzoek naar ethiek-onderwijs en educatieve software vanuit de ideeën van de rechtvaardigheidsethiek en de zorgethiek', unpublished master's thesis, university of twente, the netherlands. hettinga, m. (1993c), 'the theoria project: compelling ethical multimedia, report on an internship at the center for the advancement of applied ethics', unpublished report, university of twente, the netherlands. kohlberg, l. (1981), essays on moral development, volume 1: the philosophy of moral development, san francisco, usa, harper and row. mink, f.b. de (1991), 'cases met coo bij ethiek-onderwijs', paper presented at the 16 altj volume 2 number 2 onderwijs research dagen 1991, symposium leren van abstracte begrippen, university of twente, the netherlands. nielsen, j. (1993), usability evaluation, boston, usa, academic press. reeves, t.c. and harmon, s.w. (1993), 'systematic evaluation procedures for instructional hypermedia/multimedia', paper presented at the annual meeting of the american educational research association, atlanta, georgia, usa. rorty, r. (1989), contingency, irony and solidarity, cambridge university press. shneiderman, b. (1987), designing the user interface, reading ma, usa, addisonwesley. watkins, b.t. (1992), 'videodisks bring "dimensions of emotion" to ethics education', chronicle of higher education, march 4. 17 natural-language processing applied to an its interface enrico fischetti and antonio gisolfi dipartimento di informatics, university of salerno, italy abstract the aim of this paper is to show that with a subset of a natural language, simple systems running on pcs can be developed that can nevertheless be an effective tool for interfacing purposes in the building of an intelligent tutoring system (its). after presenting the special characteristics of the smalltalk/v language, which provides an appropriate environment for the development of an interface, the overall architecture of the interface module is discussed. we then show how sentences are parsed by the interface, and how interaction takes place with the user. the knowledge-acquisition phase is subsequently described. finally, some excerpts from a tutoring session concerned with elementary geometry are discussed, and some of the problems and limitations of the approach are illustrated. introduction this paper deals with the way in which users and computers can interact by means of natural language. in particular, we focus our attention on the development of a restricted and simplified, but still potentially useful, natural-language interface for an intelligent tutoring system (its). although natural-language processing technology has become flexible enough to assist in certain problem-solving activities, only powerful systems such as intellect (aic, 1986) are currently able to tackle it effectively. yet even such systems are subject to severe limitations (e.g. distinguishing between nouns and verbs), and consequently the problem arises of considering to what extent simple systems handling only a subset of a natural language, and running on inexpensive machines, can be effectively used for interfacing purposes. so far, many systems for understanding natural language have been developed, and their advantages and disadvantages have been investigated for some time (see, for example, 38 altj volume 2 number 2 morik, 1984; rich, 1985; ogden, 1986; shneiderman, 1986). the most potent argument in support of the use of natural language is, of course, its flexibility and 'naturalness'. the merits of natural-language interfaces are well documented as a result of a number of field trials (turner et al, 1984; jarke et al, 1985; capindale and crawford, 1990; slator et al, 1986). but one should remember that 'normal* language is not always the best way of describing things. in fact, activities requiring mathematical skills (e.g. creating shapes or solving equations) benefit from the use of a restricted formal language that allows the avoidance of ambiguities inherent in natural languages. on the other hand, when question-answer tasks are to be tackled, natural-language interaction is clearly better suited. intelligent tutoring systems although the goal of a system capable of entirely autonomous teaching is still very far off, recent advancements in both hardware and software technologies mean that systems can now be developed that offer partial but nevertheless potentially effective solutions to problems of flexibility, and that can act as reliable partners for human teachers. it is apparent that the overall result of bringing together multimedia devices and al technology will represent a significant improvement in the quality of interface systems available for itss (barker and tucker, 1990). itss originated during the 1970s, and have emerged, among other things, as tests for artificial intelligence (al) concepts. computer-assisted instruction (cai) systems, despite their success in certain environments, suffer from an inherent rigidity, being unable to adapt themselves to the characteristics of specific students. but al techniques can be profitably used to improve the performance of cai systems. itss take into consideration individual differences in students, and provide more individualized instruction. this adaptation to individual students has been made possible through the development of certain modules within an its (poison and richardson, 1988; mcfarland and parker, 1990). the most generalized architecture includes four functionally distinct, though linked, modules: • database module, containing knowledge about the topic to be taught, and holding the data created while the its is in operation; • student module, implementing the student model, and tailoring the system behaviour to the student's needs; • teaching module, implementing adequate tutoring strategies according to the information obtained in the student module, and controlling the topic to be taught, and how and when it should present it; • interface module, managing student interaction with the system, and supporting the building phase of the knowledge base by the human expert. the role of the interface module is of critical importance (see brown, 1989; laurel, 1990), since it processes the information flow from the user to the system, and vice versa. thus the presentation of a topic will be made more or less understandable according to the quality of the interface. the knowledge base is usually built by expert al programmers, but an important issue is constructing the knowledge base so that a teacher-expert in the field can modify it, even 39 enrico fischetti and antonio gisolji natural-language processing though he or she may lack programming experience. for this, our interface module manages the knowledge acquisition phase so that the its is easily maintainable. the smalltalk/v programming environment the object-oriented programming (oop) paradigm allows systems to be modelled in terms that attempt to match human thinking and language, and considering that computer-based tutorial activity also attempts to simulate the way in which human beings tackle teaching and learning, the importance of the oop approach with respect to its research is clear. the essence of the approach is that software systems are modelled in terms oi objects and actions on objects by means of messages. an object is essentially a set of data and methods (the procedures that act on this data). in defining a type of object, a class is obtained whose elements share some characteristics. the classes are hierarchically organized so that their characteristics can be inherited by sub-classes. we decided to implement our interface module in smalltalk/v, a dialect of smalltalk developed by digitalk inc. one of whose main advantages is that it runs on pcs. it also offers very rapid coding and testing, and provides a wide range of hci (humancomputer interface) facilities. in particular, the graphical interface is very friendly, the user interacting with the system mainly by means of windows and menus; friendly interfaces combined with powerful facilities are an essential part of its development, as has been shown many times (or-bach and bar-on, 1989; hobbs, 1990; gisolfi and moccaldi, 1990; gisolfi and moccaldi, 1992; fischetti and gisolfi, 1992). moreover, employing oop in its development is useful for diagnosis. each concept is implemented as an object, and objects are in turn hierarchically organized so that the attributes needed for diagnosis can be inherited. smalltalk/v is very economical in code: an impressive amount of activity results from a few lines of code, and one can re-use most lines of prewritten code. further, editing and correcting errors is accomplished very easily. of course, oop systems are not disadvantage-free (see jones and thorne, 1988); for instance, dynamic binding allows software rules to be compiled, linked and loaded incrementally, but requires complex structures for its implementation and additional time for execution. we can give here, in describing our work, only a vague feel for the unique features of smalltalk that make it quite different from conventional languages. newcomers to the language may be helped by standard texts on the subject, e.g. meyer (1988) and digitalk (1988). the architecture of the interface module our interface module consists of two distinct sub-modules: expert and user. the expert sub-module manages knowledge acquisition and allows the human expert to build the knowledge base of the its so that it can subsequently be consulted by the student by means of the user sub-module. the hierarchical characteristics of the smalltalk/v language are exploited: thus, we have defined a sub-class of the class object, called modules, and have linked the classes interface, engine, expert interface and user interface as depicted in figure 1. although the underlying principles of the interface are simple, their translation into 40 alt j volume 2 number 2 figure 1: hierarchical structure of the interface software brings with it several unavoidable technicalities that use the advanced features of smalltalk/v. it is not possible in an outline paper of this kind to go into the details of these technicalities, and we confine ourselves here to some conceptual aspects. the class engine is implemented by means of the minimal functions of a database module, so that the knowledge base can be built fairly easily. in smalltalk/v terms, the knowledge base consists of a set of objects having an internal state, as defined by their characteristics (instance variables), and a set of specific methods, for example those defined for the classes string, dictionary and set. the knowledge base evolves gradually as its elements are modified: after some initial conditions have been satisfied, changes in the knowledge base are related to the inferences carried out by the database module. as the interface has to process natural language, its dictionary plays a vital role. the nonspecific part of the dictionary is built-in: the human expert has only to add information specifically related to the topic to be taught. the expert is guided in this activity by appropriate windows and menus present in the smalltalk/v environment. all the terms inserted by the expert are added to the basic dictionary. the subset of the english language recognized by the interface incorporates nouns, verbs, adjectives, conjunctions, articles and prepositions. the first three of these (nouns, verbs, adjectives) contain objects to be considered as 'meaningful' (a semantic value is associated with each of them), whereas the other three (conjunctions, articles and prepositions) are used to link together elements of the 'meaningful' classes. building the knowledge base has the side-effect of adding to the interface all the elements it needs to carry out its function. our prototype its is intended for a tutoring activity in elementary geometry. the human expert has first to specify what objects are involved (triangles, rectangles, etc.), and then for each object its syntactic features (singular, plural, gender in the case of italian) and its semantic ones (subclass of ...). other information is also linked to each object, for instance the number of sides, the perimeter, the area, and what operations are allowed for the object. for example, operations such as 'area measurement' or 'verify the equivalence' are suitably linked to smalltalk/v methods, then saved in the appropriate class. the goal of the interface is to associate a semantic value with the sentence typed by the user, something which can be tackled only if the process is adequately supported by 41 enrico fischetti and antonio gisolfi natural-language processing parsing algorithms able to check the syntactic and semantic validity of the sentence. the analysis carried out by the user-interface, faced with the sentence x, can be summarized as follows: (a) does x belong to the interface language? (b) what is the type of x? (c) what is the semantic value of x? the following section shows how sentences are parsed and these questions answered. parsing sentences the process which a sentence typed by the user undergoes can be briefly described as follows: the interface module first analyses the sentence to check whether all its elements belong to the dictionary. each acceptable word is stored in the dictionary along with its synonyms, allowing the maximum available flexibility, and therefore the maximum available freedom for the user. although the general structure of parsable sentences is subject-verb-object, the interface can also cope with more complex sentences where there are verbs indicating requests. for example, given the sentence 'rome is the capital of italy', the interface recognizes the noun 'rome' as the value of the instance variables capital and italy as instances of the class nation. if the check is positive, the next step involves a diagnosis as to whether the sentence is the answer to a tutor question or a user's question, and the dialogue state has to be changed accordingly. now, the parsable sentences can be classified as one of: (a) declarative (b) direct interrogative (c) indirect interrogative direct interrogative sentences are recognized by the presence of the final question mark, a crude but workable solution. user's answers are expressed by declarative sentences. indirect interrogative sentences express an information request by the student. as their structure is similar to that of declarative sentences, we have introduced into the dictionary a set of typical expressions whose presence induces the interface to assign the sentence to the indirect interrogative class. these expressions are, for example: tell me, show me, explain, and so on. the interface manages user's questions by means of the prompter, a built-in smalltalk/v window. the sentence typed by the student is stored in the global variable line, and then the parsing process starts. a crucial problem to be tackled is the correct management of sudden changes in the dialogue state. a dialogue between two individuals may be composed of questions and answers, but it is hardly realistic that the roles 'asking-answering' of individuals carrying on a question-answer dialogue remain static, something which happens only in quiz-like sessions, and one has thus to consider that, for example, when the student is asked a question, instead of answering, s/he might reply with another question. to cope with such situations, the binary variable dialogue-state is introduced, whose value changes according to the circumstance that a question or an answer is expected from the student. 42 alt j volume 2 number 2 thus, if the dialogue-state indicates that the student's answer is expected, and an interrogative sentence is typed, the unanswered question is saved into a stack, the dialogue-state switched, the student's question answered, the tutor's question popped from the stack, and the dialogue-state switched again. associating a semantic value with a sentence x has proven to be one of the most difficult problems. we have tackled it by supposing that the student can ask five types of question, schematically classified as follows: (1) tell me if... (2) tell me h o w . . . (3) tell me w h y . . . (4) tell me w h i c h . . . (5) tell me how m a n y . . . the keywords: if, how, why, which and how many are stored into set instances, and are declared as global variables in the basic dictionary. it is worth noting that the performance level of the interface is strictly related to the characteristics of the smalltalk/v environment. this carries with it some disadvantages, but on the positive side, this means that a hierarchy of classes can be developed, each linked to a type of object. in this way, all the knowledge related to the object is contained in a unique class, including internal variables and methods, so that if a particular class is selected, one is not obliged to deal with the entire knowledge base: the methods of the selected class are used to investigate its internal variables and related instances. but one has also to remember that the interface is not a module independent from the others making up an its: indeed, there is no sharp dividing line between its modules, and managing questions requires suitable interactions between the interface and the inference engine of the database module. the first type of question ('if) is a confirmation request. of course, it can also be expressed, like other classes of sentences, by a direct question ('is a triangle a polygon?'). the interface has simply to check that the relationship between the objects is present and, if this holds true, the student's hypothesis is confirmed. the second type of question ('how') is concerned with requests for explanations about the correct way of executing a specific operation. in this case, the answer requires that the methods introduced by the expert are suitably communicated to the student. for example, the sentence 'tell me how the perimeter of the triangle is measured' activates the inference engine to draw the symbolic code associated with the object 'triangle' and the operation 'measurement of perimeter'. the third type of question ('why') is concerned with requests for explanations about the logical steps that have led to the current state. for example, in the sentence: 'why does the measure of the area equal 70?', a smalltalk/v method will show the steps the current situation depends upon. the fourth type of question ('which') involves situations where the student wishes to know which objects satisfy the constraints present in the question. if, after the preposition 'of, we find the name of the class to be investigated, synonyms management allows the phrase 'surface of the triangle' to be recognized by the system as equivalent to 'area of the 43 enrico fischetti and antonio gisolfi natural-language processing triangle', and since 'area' is one of the instance variables of the class triangles, the object triangle is scrutinized to get information about area measurement. the reasoning can be iterated, and a similar process investigates the structure of the more complex phrase: 'measurement of the area of the triangle'. the basic structure of the system allows for several kinds of mistake to be detected. for example, it is easy to verify that the sentence 'measure the area of the side' is incorrect because there is no instance variable associated with area in the class side. the last type of question ('how many') can be managed in a similar way to the previous one, but the system has additionally to look for all the objects that satisfy the conditions of the question. thus, in our architecture, the knowledge base can be regarded as a set of classes, hierarchically organized according to an expert's requirements. each class is associated with a class variable of type set in order to contain the instances created during the processing phase. one can declare, for each class, a specific sub-class named exercises, whose instances are all the exercises concerning that class. the student model benefits from this approach as the analysis concerning student's knowledge can become more specialized as lower hierarchical levels are attained. in this way, the interface becomes more powerful because a sentence can be analysed semantically without using the inference engine. for example, diagnosing that the phrase 'area of the line' is meaningless would be immediate, although the sentence is syntactically correct. operating the knowledge-acquisition interface as mentioned earlier, the dictionary required by the interface consists of a non-specific part which includes basic information (articles, prepositions, auxiliary verbs, request verbs, and so on), and a specific part consisting of information related to the particular topic to be taught. figure 2 depicts the general structure of the dictionary and its functional links. dictionary interface ooo o b i e c t 8 i expert dictionary verbs -• | engine | i knowledge base classes i membership rules instance variables set of instances exercises smalltalk methods symbolic code (for each dass) figure 2: structure of the dictionary and functional links 44 altj volume 2 number 2 the structure of the first part of the dictionary is relatively simple. instances of the class set, declared as global variables, are used to contain articles, prepositions, conjunctions and a collection of typical terms that characterize interrogative sentences, such as 'tell me' or 'show me'. the second part of the dictionary is more complex because the information furnished by the human expert about the topic to be taught has to be organized in a suitable way. we have defined two basic classes things and actions that represent the objects and the operations on them respectively. an instance of things is characterized by the name (noun), its gender (in italian), its plural and a set of associated synonyms. in a similar way, an instance of actions consists of the tenses of a verb and a set of synonyms. all the instances of things are stored in a set called objects; the instances of actions are stored in a set called verbs. for example, as elementary geometry is the topic to be taught, the class things will include triangle, rectangle, circle, side, area, and so on. in turn, in the class actions, we have verbs such as compute, measure and compare. the knowledge acquisition phase can be sketched as follows. first, the expert is requested to specify the list of objects the topic consists of. each topic requires several specifications because part of the information is required by the interface and another part by the inference engine. the expert has to define the syntactic features of the object, i.e. singular, plural, gender and synonyms. consequently, the instance variables get their value, and the resulting object is added to the set objects. subsequently, the relation 'is a' is considered, and in such a way classes get hierarchically organized. for instance, one has (the asterisk denotes that the class has no super-class): perimeter * area * polygon * square polygon triangle polygon right-angled triangle triangle figure 3 depicts the hierarchical structure of the geometric classes. figure 3: hierarchical structure of the geometric classes 45 enrico fischetti and antonio gisolfi natural-language processing in order to define the classes, the expert has to specify the features of the objects, and these characteristics become the instance variables of the classes. for example, a polygon is characterized by the number of sides, formulas for evaluating area and perimeter, and so on. more specifically, to manage the peculiar features of the objects, for example triangles and quadrilaterals, two global variables are defined: features and differences. in such a way, the expert specifies which features are shared by objects and which characteristics are peculiar to each object. in particular, the variable differences contains a set of rules that characterize the objects belonging to that class. in other words, the values contained in such a variable represent the necessary and sufficient conditions for an object to belong to that class: a polygon is a triangle if, and only if, three sides are present when the building phase is completed from the syntactic point of view (gender, synonyms, singular, plural), the expert has to specify which operations are allowable for these objects, and this happens by means of sentences having the general structure verbcomplement. first, each sentence is analysed with respect to its compatibility with the objects present in the dictionary, and if no discrepancy is recognized, the sentence is stored in the dictionary. the activity of the interface is twofold: while the dictionary is suitably enlarged, the engine receives the necessary information to define the methods related to the instances of the classes. this task is by no means trivial; for example, the complement typed by the expert might be unknown to the dictionary, so that the operation will be utterly meaningless. if this is not the case, and the complement is a known object, the verb is added to the set verbs, after having been declared as an instance of the class actions, and its conjugation and synonyms having been specified. if the complement is unknown, the expert is requested to re-type the name of the object, and if it is still unknown, the interface asks whether the new object is to be added to the dictionary. if the answer is yes, the complement is added, through the above-mentioned procedure, to the set objects, and the action to the set verbs. of course, the expert can alternatively express the complement in a different way if he or she does not want to introduce a new term. the above-mentioned operations are carried out for each object declared as a class. when all the required information is collected, the interface prepares the smalltalk/v code representing the methods that operate on the data. thus each class gets associated with a set of relevant information. for example, the class right-angled triangles includes slots for several pieces of data: the names of the sides and vertices of the triangle, the lengths of its sides, and the measurement of its angles, features such as 'number of sides = 3' and 'one angle is 90°', the symbolic code for evaluating its area, perimeter, catheti and hypotenuse. at the end, the dictionary is ready to start the consulting phase. operating the user interface a student activates the its and starts interacting with the system. suppose that the student types: 'tell me how the area of the triangle can be measured' the words this sentence consists of are recognized by the interface, and thus the sentence is reduced to a standard form: 46 altj volume 2 number 2 'tell me how area of triangle is measured* the system displays this to the student, and asks for confirmation. if the sentence is not confirmed, the interface asks for a new formulation of the question. otherwise, the module recognizes that this sentence expresses an information request thanks to the term 'tell me', while the presence of the adverb 'how' allows it to recognize that the student wants to know how a certain operation can be carried out. the operation is 'area measurement', and the presence of the phrase 'of triangle' allows the system to recognize the relationship between the operation (measurement), the feature (area) and the class (triangle). at this point, the interface is able to ask the engine to draw up the symbolic code associated with the method 'area measurement' defined for the class triangles. this looks as follows: [engine new drawfrom: triangle codof: area measurement] interface: measurement of area of triangle: base * height / 2 suppose now that the student types: 'measure the area of the quadrilateral abed' as the noun 'quadrilateral' is followed by another noun, the interface is able to recognize that the request 'measure' refers to some instance of the class quadrilateral. thus the set of instances of this class is investigated to verify that 'abed' exists, then the data is fetched. if 'abed' is not an instance, the interface asks the student whether it has to be defined. if the answer is yes, the interface calls the engine in order to declare a quadrilateral. the related method gives the appropriate values to the instance variables, provided that the rules present in differences are not contradicted, and finally the new instance is added to the instance of quadrilateral. then the method specific for area measurement is activated: [engine new activate: area measurement in: quadrilateral for: abed]. suppose that the interface is faced with the following statement: 'tell me if ptr is a polygon' this question is a request about a possible class membership. suppose that 'ptr' is an instance of triangles but not of polygons. the interface asks the engine to look for such instance: [engine new search: ptr relay: is a] and starting from the circumstance that a triangle is a polygon, the question is answered affirmatively. finally, to give an idea of how the system works in practice, here is an excerpt from a typical student-interface dialogue: s: tell me how the perimeter of the tide can be measured i: i don't understand "tide" s: sorry, how do you measure the perimeter of the side? i: "measure perimeter" is not defined for side. 47 enrico fischetti and antonio gisolfi natural-language processing concluding remarks we are all too aware of the constraints of our interface, but it is apparent that languageprocessing capabilities which will be satisfactory in certain circumstances can be achieved by means of a simple conceptual architecture supported by a powerful programming environment. field tests so far carried out show that users generally have a positive attitude towards the interface. most errors made by users are directly related to restrictions in the language, and when a mistake occurs, feedback helps the user to understand the language limitations of the interface, so that errors are subsequently avoided. of course, this kind of positive conclusion has to be drawn cautiously: far more extensive evaluation is required, and we are now actively engaged in testing, evaluating and improving the interface. references digitalk (1988) (anon.), smalltalk/v 286 tutorial and programming handbook, los angeles, digitalk inc. aic (1986) (anon.), intellect reference manual, waltham, educational technology publications. barker, j. and tucker, r.n. (eds) (1990), the interactive learning revolution: multimedia in education and training, london, kogan page. brown, j.r. (1989), programming the user interface principles and examples, reading, ma, usa, addison-wesley. capindale, r.a. and crawford, r.g. (1990), 'using a natural language interface with casual users', international journal of man-machine studies, 32, 341-62. fischetti, e. and gisolfi, a. (1992), 'design and development of the student module of an its', interactive learning international, 8, 201-11. gisolfi, a. and moccaldi, g. (1990), 'an its for the factorization of algebraic expressions in smalltalk/v', educational and training technology international, 27, 4, 406-13. gisolfi, a. and moccaldi, g. (1992), 'the student module in posits, a tutor for the factorization of algebraic expressions', journal of artificial intelligence in education, 3 , 3, 347-58. hobbs, d.j. (1990), 'second-level design of a knowledge-based educational advisor', educational and training technology international, 27, 2, 216-23. jarke, m., turner, j.a., stohr, e.a., vassiliou, y., white, n.h. and michielsen, k. (1985), 'a field evaluation of natural language for data retrieval', ieee transactions in software engineering, se-ii, 1, 97-113. jones, a.c. and thorne, m.p. (1988), 'smalltalk as a vehicle for intelligent computer assisted learning', proceedings of the 5th international conference on technology in education, march 1988, london, cep consultants. laurel, b. (ed) (1990), the art of human-computer interface design, reading, ma, usa, addison-wesley. 48 altj volume 2 number 2 mcfarland, t. and parker, r. (1990), expert systems in education and training, englewood cliffs, usa, educational technology publications. meyer, b. (1988), object-oriented software construction, englewood cliffs, usa, prentice-hall. morik, k. (1984), 'cumers' requirements for natural language systems: results of an inquiry', international journal of man-machine studies, 21, 401-14. ogden, w.c. (1986), 'implications of a cognitive model of database query: comparison of a natural language, formal language and direct manipulation interface, sigchi bulletin, 1 8 , 2 , 5 1 4 . or-bach, r. and bar-on, e. (1989), 'probit: developing an intelligent tutor', proceedings of the 4th international conference on ai and education, amsterdam, 185-92. poison, c.p. and richardson, j.j. (eds) (1988), foundations of intelligent tutoring systems, hillsdale, usa, lawrence erlbaum. rich, e. (1985), 'natural language understanding: how natural can it be?, ieee conference on ai applications, 372-7. shneiderman, b. (1986), 'seven plus or minus two central issues in human computer interaction' in mantel, m. and orbeton, p. (eds), human factors in computing systems: chi '86 proceedings, 343-9. slator, b.m., anderson, m.p. and conley, w. (1986), 'pygmalion at the interface' comm. acm, 29, 7, 599-604. turner, j.a., jarke, m., stohr, e.a., vassiliou, y. and white, n. (1984), 'using restricted natural language for data retrieval' in vassiliou, y. (ed.), human factors and interactive computer systems, norwood, usa, ablex, 163-90. 49 editorial is there a way? i am currently a member of a working party set up in my own university to look into a range of it matters, including learning technology, with the aim of producing a mediumterm institutional plan. at many of the meetings i attend, i hear about the urgency of focusing our cal effort, but the conviction around the table is often tempered by lecturers' complaints that the off-the-shelf courseware they have tried either does not work well, or does not fit their particular needs, or both. so a suggestion is made: we should move in the direction of developing our own high-quality educational software tailored to our individual requirements. and since these requirements are very diverse over the whole campus, we should establish a centre for educational technology, a courseware resources and advice unit, a virtual learning development laboratory, an institute for computer-based academic practice . . . call it what you will. it should be staffed by experts who can advise departments and produce for them, or help them to produce, the exact software they require. it should be supported by a battalion of technicians, and should not only be equipped with white-hot multimedia but also backed by sufficient financial resources to ensure continuous upgrades so as to remain in a permanent state of state-of-the-art. the bank balance is not as healthy as it might be (whose is?), but the university management must nevertheless somehow be convinced of the necessity of spending money on the project. i wonder how many alt members recognize the scenario. i know some who certainly will be familiar with at least parts of it. most he institutions utter noises about the need to make as much use as possible of computer technology in teaching, but in a significant number of them the decision-makers are still unwilling to commit money while they are being given contradictory messages. they are informed on the one hand that the future success of the institution may depend on a large-scale implementation of learning 2 alt-j volume 4 number 3 technology, and on the other that comparatively little educational software is being used within curricula because what is available is so often defective or otherwise unsuitable. in their minds, the two arguments do not converge on an expensive, wide-ranging programme of courseware development. part of the thinking behind the tltp initiative was that it would address this problem: that is, that it would bring together expertise, thus avoiding duplication of effort, and that the targeted software produced would be made available to appropriate departments in institutions nation-wide. but tltp has had less success than was hoped for in seeing the resulting products actually used, let alone relied upon as indispensable teaching resources. i do not believe that this situation is primarily due to a xenophobic not-invented-here syndrome, but rather to the fact that most tltp products genuinely fail to measure up precisely to course requirements and standards, for whatever reasons. i have dealt with essentially the same issue from a different angle in an editorial in this journal more than three years ago, and am no further on than i was then in being able to propose a practicable solution to what is, for alt, a major strategic problem. by the time this editorial appears, we shall have had alt-c 96, where we shall doubtless have heard paper after paper enthusing directly or indirectly about the benefits of learning technology. but the authors of those papers will be preaching to the converted and the ready-to-be-converted. what should we do to convince doubting thomas? gabriel jacobs 3 resource-based learning strategies: implications for students and institutions malcolm ryan,* julian wells** alan freeman** and george hallam** *academic development group, university of greenwich **school of social sciences, university of greenwich this paper reports some findings from a project in implementing resource-based learning in economics, and identifies some implications for students and institutions. these include student responses to a midsemester evaluation and the views of the project team. the latter have been informed by action research which sought to recognize students' individual differences, employ active learning methods and, above all, integrate it into the curriculum. while innovative strategies are clearly welcomed, students show strong attachment to some traditional methods. most of those who suggested changes to the range of activities asked for reinstatement of at least some lectures, generally as additions to existing activities. implications include the need for students and staff to acquire a wide range of new skills, for large-scale curriculum review if new learning technologies are to be fully integrated, and the need to acknowledge that, given student and staff perceptions of change, the process may be long and costly. introduction in its strategic plan, the university of greenwich envisages a significant shift to resource-based learning (rbl). enterprise in higher education (ehe) has funded five pilot rbl projects during the past year, including one in introductory economics. the project was managed by three lecturers in the school of social sciences, supported by an academic development officer. learning outcomes were completely revised, and a range of assessment strategies, including computer-based tests, was identified. a resources guide was produced which identified the materials and activities that would enable students to achieve the learning outcomes. a number of innovations were adopted, including: • computer-based curriculum delivery, assessment, and student evaluation of the course; • an open approach to assessment; • abolishing lectures in favour of a diverse range of teaching and learning activities. a mid-semester computer-based evaluation was conducted using question mark designer, with the specific intention of providing data to inform future curriculum revision. 93 malcolm ryan et d resource-based learning strategies: implications for students and institutions rationale for the adopted approach previous work had shown that where computer-based methods were simply 'bolted-on' to existing teaching, neither students nor lecturers used them effectively, if at all (freeman and wells, 1994). the team felt that to be successful, ft needed to be an integral part of the programme (including its assessment), and that continuing curriculum review was essential. accordingly, we began by identifying the relevant knowledge, skills and concepts needed by students at an introductory level, and turned them into clearly stated learning outcomes. this work entailed staff development on the part of the three subject specialists, and hence support from the academic development officer. to facilitate testing of the learning outcomes, a range of assessment strategies was identified, in which the benefits of providing students with informative and frequent feedback were recognized (see rowntree, 1992a, 24-7,162). it was noted by the team that individual difference affects the way students learn (tomlinson, 1985; messick, 1978, vi, 300, 310); adoption of a range of teaching and learning strategies was therefore paramount. the nature of the subject and diversity of available resources made management of teaching and learning situations an important issue. consequently, it was decided that a guide matching resources to outcomes and assessment would be an invaluable aid to students and staff. the unit was introductory and university-wide for students taking degrees other than economics or business studies but needing a basic understanding of economics. we could not assume previous exposure to the subject, and general background varied widely. the previous project had shown that we should insure against unexpected technical problems by timetabling supervised computer workshops. a simple numerical problem then arose. we had 90 students and the maximum group we could place in the computer laboratory was 30. moreover, our timetable called for three consecutive teaching hours. we therefore had to divide the class into three groups of 30 and process them one at a time; the remaining innovations were driven by this and the fact that just three subjectspecialist teachers were available. we could have dropped traditional seminars; but we decided not to, so that we could discuss with the students their laboratory work and associated learning outcomes. thus, we tacitly recognized that computer-based learning on its own could not, at least so far, achieve the outcomes of the unit. the same constraints suggested the most radical (in a local context) innovation which we made: to abandon lectures and substitute a timetabled hour of self-directed study. we knew from the previous work mentioned above that laboratory sessions for 30 students were likely to be unmanageable unless two teachers were on hand, and in each hour one member of staff was always needed to conduct the seminar. hence scrapping lectures was not something we set out to do for educational reasons; we simply ran out of hours and staff. mid-semester evaluation as question mark designer was used as part of the programme of assessment of the students, its use for student evaluation of the course seemed both appropriate and convenient. accordingly, the software was used to administer a mid-semester questionnaire on student 94 alt-j volume 4 number i response to the course. reaction to the computer-based activities was very positive, with 79% of respondents stating that they found them very useful as a learning aid. when asked to indicate which of the strategies they would most like retained in the unit programme, 70% cited the supervised computer workshops. reaction to the computer-based assessments (which contributed to students' final grades) were also positive, and there was evidence to suggest that students used them as a learning device (see below). open assessment was welcomed by students, with 90% favouring publishing examination questions in advance. the discussion, planning, structured feedback and sharing of essay plans, which also contributed to the final grade, was regarded as very useful by 84% of respondents. however, asked what changes to the unit students would like, 94% wanted formal lectures added to the existing range of activities. comment the very positive response to computer-based activities is significant, and justifies the team's view that embedding it into the curriculum as a means of acquiring skills and concepts is sound. however, it was noted that some students spent considerable amounts of self-directed study time on the computer activities, yet many were unable or perhaps unwilling to complete some of the tasks which were needed as part of the seminar programme. this suggests that students found that the additional tutor-contact gained from attending workshops supported by specialist staff was motivating, but despite extra tuition some failed to identify the role of the computer-based activities in preparing for seminars. we had previously discovered that it was difficult to disentangle acquisition of the it skills needed for a particular activity from understanding the economic concepts. also, experience with computer-aided learning packages suggested that students are reluctant to use the software outside scheduled classes, partly because they find it technically and logistically difficult to do so. this, plus our strong belief that such activities need to be supervised by a subject-specialist with computer experience (freeman et al, 1995), were the reasons for treating workshops as part of teaching time and staffing them with economists. staff scepticism about multiple-choice questions was partly dispelled by the way students used the computer-based tests as an aid to learning. it would have been possible to score 100% in the first of these tests merely by recording the feedback given by the program and entering this at the second attempt. but students did not 'cheat' in this way; instead, they repeated the test until they achieved what they regarded as a satisfactory score. these repeated attempts, the resultant improvements in scores, and the summative nature of the test, suggest that the students were motivated both to learn and succeed (see rowntree, 1992a, 22-4). thus the use of computers in assessment brought about a fundamental change in the relationship between the students and the test essentially, by placing the students in control of when and how often they took it (see laurillard, 1988,218). as well as the qualitative data generated by the students' evaluations, we have a substantial body of quantitative data resulting from the number, variety and frequency of assessments. preliminary results (wells et al, 1995) suggest that open assessment, even when students can explicitly practise for it, has no less power to discriminate between different levels of achievement than traditional methods (in the examination there were 16 upper seconds, 23 _ malcolm ryan et o\ resource-based learning strategies: implications for students and institutions lower seconds, 23 thirds, 13 fails, and no firsts). in fact, comparison of results from the course essays and the students' attempts on the same topics in the examination actually shows a small decline in average performance (table 1), thus undermining the idea that open assessments are less rigorous. moreover, there is evidence of improvement on the part of students whose essay results were weak: of the 22 students who scored less than the median essay mark of 59,15 did better on the same question in the examination, while of the 27 who scored 59 or more, only five improved on their essay mark (and 15 of the 27 declined by more than 10 marks). table i: comparison of results from course essays and examination at the first session of the unit, students were told it was resource-based and given documents including an introduction to the unit, a timetable of the weekly sessions, the intended learning outcomes, the final examination questions, and advice on how to organize their work. this amount of information was unusual for a taught unit, but was felt to be necessary if students were to be supported and guided through this new experience (see rowntree, 1988, 11, 163). however, evaluation suggests that this provision was ineffective. the students' desire for traditional lectures appears to be grounded in the view articulated by one student: 'what i did not like about the course is that i was expected to find out everything myself instead of being told'. an important issue here is that students wanted lectures in addition to the other resources provided, a preference for a greater emphasis on tutor-led activity clearly implying that they wanted more direction and less autonomy and control (see rowntree, 1992b, 62). they felt that lectures would be useful for defining terms and providing an overview, reviewing relevant book chapters and identifying examination topics. classroom experience and informal discussion with students suggest that some felt threatened by an approach which demanded not only that they become responsible for their own learning, including planning and managing their time effectively, but also that they use one set of learning opportunities (the computer workshops) to provide material for another (the seminars). some students admitted to liking the approach, but found it strange compared to their previous experience of the institution, where the formal lecture followed by a discussion seminar was the norm. the novelty of this unit's approach might have been both an advantage in terms of motivating students but a disadvantage insofar as it posed a potential threat to their psychological safety. this desire for more tutor contact rather than less has serious implications in a highereducation culture currently seeking to reduce contact time. the double-edged sword of novelty might also pose problems in an institution wanting to promote rbl. if the approach is to gain wider acceptance, then both these issues need further investigation. some conclusions and observations if the adoption of rbl is viewed as an opportunity to review the curriculum, then it may also be used as a vehicle for embedding it into that curriculum. our experience shows that in order _ alt-j volume 4 number i to be effective, any such activity must be appropriately supported and supervised, and the outcomes need to contribute to the final grade. '; it is clear that while students may welcome some innovations, they also wish to retain traditional approaches, and that they value contact with tutors. if students perceive lectures as important and necessary, then an approach which precludes them may not be popular. an institution wishing to move towards rbl will thus need to consider these perceptions and how best to accommodate them. compared with more traditional approaches, and in common with open and distance learning, students and staff need a different, and wider, range of skills., if students are required to use new learning technologies, they need to be competent users of software, hardware and information, and to be able to manage their time effectively. institution-wide use of such technologies can occur only if all undergraduates possess or have the opportunity to acquire this vast array of skills. one possible approach is a core programme in information and study skills and computer literacy for all undergraduates. an alternative which we favour is to embed these skills into the curriculum of all subject areas so that they become contextualized, as they did within this project. furthermore, our experience teaches us that staff engaged in this approach need to be able to redesign and evaluate curricula, understand new learning technologies and employ them appropriately, construct a range of learning and assessment activities to meet outcomes, design teaching and learning materials, provide tutorial support, and operate effectively in teams. while one may argue that these are the skills all teachers in higher education require, in our experience rbl makes the full range of these skills essential and not merely desirable. this raises the issue of staff development (see rowntree, 1992b, 260-63) even where these skills already exist, the time required to prepare the project described above exceeded that for a standard lecture and seminar programme. innovation is an expensive exercise (which in our case was funded by the ehe). unless sufficient funding can be found it is unlikely that thorough and continuing curriculum review can be sustained within the current cost-cutting regime. further infrastructural implications include the physical design of it facilities so as to permit discussion and other group activities in conjunction with computer work, the technical support needed to ensure that hardware is configured correctly and software operates as intended, and the training of it support staff to play a pedagogical rather than merely technical role. the significance and role of student feedback, while currently a popular issue and target for automation, also needs to be examined further in the light of what it revealed about our students and their views on our approach. finally, further investigation is needed into how an institution sets about 'mainstreaming' innovation so that it becomes the rule rather than the exception. there are, in our view, many obstacles to this, not least of which are prevailing organizational cultures. cultural shift is a long-term goal of enormous scale, but to adopt rbl in an otherwise alien culture is likely to be unproductive, if not foolhardy. 97 malcolm ryan et al resource-based learning strategies: implications for students and institutions references freeman, a. and wells, j. (1994), computer-aided learning in economics, enterprise in higher education unit, university of greenwich. freeman, a., hallam, g., ryan, m. and wells, j. (1995), 'teaching economics with the aid of computers', paper presented at the caleco conference, bristol, september 6-7, 1995. laurillard, d. (1988), 'computers and the emancipation of students: giving control to the learner' in ramsden, p. (ed), improving learning -new perspectives, london, kogan page. messick, s. and associates (1978), individuality in learning, san francisco ca, jossey-bass. rowntree, d. (1988), teaching through self-instruction, london, kogan page. rowntree, d (1992a), assessing students: how shall we know them', london, kogan page. rowntree, d (1992b), exploring open and distance learning, london, kogan page. tomlinson, p. (1985), new directions in educational psychology -1: learning and teaching, london, falmer press. wells, j., ryan, m., freeman, a. and hallam, g. (1995), 'assessing innovation and innovation in assessment', paper presented at the edineb conference, university of uppsala, august 17-20, 1995. 98 technology for social work education tom hopkins* and david colombi** *institute of health and community services, bournemouth university **centre for human service technology, university of southampton the intention of this paper is to examine aspects of the role of information technology in social work education in relation to existing developments within an international context, conceptual issues concerning the application of cal to the teaching of social work, and the implication of these issues for the development of integrated teaching modules in interpersonal skills and research methods, together with some of the practical issues encountered and solutions being adopted the context for the paper is joint work by the authors as members of the procare project, a partnership between southampton and bournemouth universities, and part of the uk government-funded teaching and learning technology programme (tltp) in higher education. procare is developing courseware on interpersonal skills and on research methods for use in qualifying-level social work and nursing education. while the emphasis is on the social work version of the interpersonal skills module, limited reference is made to the nursing component and the differential approaches that proved necessary within the subject areas under development. social work and it developments the development of technology in social-work education is set not only in the broader context of the use of technology in higher education generally, but also in the parallel context of the use of technology in social work practice. the latter is important for a subject area with two primary characteristics: the need to acquire not just academic knowledge but also professional expertise and understanding of the aims, methods and values of social work; a strong emphasis on practical preparation for work in social work agencies, which is expressed through the universal use of practice placements within agencies as a major component of social-work professional education and training. within contemporary social work practice, it is possible to identify several broad themes concerning the use of new technology: 17 tom hopkins and david colombi technology for social work education a low level of overall penetration into the work of social work agencies; a primary focus on developing basic client information systems; a preoccupation with the needs of managers and local and central government for financial and statistical data used for accountability and policy purposes. as a consequence, there has until recently been little expectation by employers that those completing professional social-work training will be computer literate, although that situation is now changing. those who do acquire such skills may experience frustration at the limited opportunities within agencies to develop and apply them in their work with clients. for social work courses the issues are, then, primarily about how technology is used for educational purposes. however, for some educators there is a secondary objective of contributing to initiatives which may indirectly help to reshape social-work practice through diffusion, example and opportunity. the barriers for social-work education include many shared with other disciplines, but also some that are unique, or at least less common. those that are familiar, applying across most or all disciplines, may be characterized as: structural: organizational resistance and/or antipathy towards new technology, arising from an unwillingness to contemplate organizational changes which the use of new technology requires. developmental: failure or unwillingness of course planners, managers and instructors to consider the potential of new technology in relation to curriculum needs and requirements; identifying the place of computer-assisted learning (cal) in the curriculum and developing the competence and confidence of teachers to use cal and adapt teaching styles to a cal environment. conceptual: confusion about the nature of the activity whether cal, computer-based learning (cbl) or computer-assisted instruction (cai). the term cal is used here with an emphasis on learning rather than instruction; and on computer assistance rather than the more fully entrenched role implicit in cbl. practical: availability of and access to required technology, quality software and appropriate computer teaching environments (including educational, technical and administrative supports); the second aspect depends on the extent to which software is designed to be used entirely by students working alone, or as part of a teaching and/or other shared learning experience. philosophical/ideological: individual resistance by staff, through apathy, anxiety or hostility, including the 'not invented here' syndrome; also, collective resistance by academic (and some professional) groupings to new technology having other than a minor, adjunctival role to play in education for human-service professions. personal: individual resistance by academics and practitioners to new technology in professional education and training, largely out of fear of job change, relocation or loss; these views are shared by some students who see the introduction of new technology as less attractive than conventional instruction, and which may require more effort on their part (see hopkins, 1996). 18 alt-] volume 4 number 2 less familiar barriers include: professional: the imprecise nature of the knowledge base of social work that makes it less amenable to computerization than more structured and precise knowledge-based systems, such as science, law or medicine. ethical: issues of client confidentiality, together with concerns about the extent to which computers may be perceived as dehumanizing or undermining the quintessentially personal interactions which characterize much social work. on the positive side, there is an international co-operative endeavour to contribute, primarily at an academic level, but also at a practitioner level, to change. this work finds its strongest collective expression though a series of international conferences known as husita (human services information technology applications), and through the work of the european network for information technology in the human services (enith). these endeavours provide a means of meeting, sharing and networking, and the focus for publication of conference papers and other texts, such as international comparative studies (colombi, 1993; steyaert 1996). the educational dimension to this work is an important one amongst other aspects that include concerns about the needs of social-work practitioners, service users and other citizens, combined with emphasis on ethical themes of empowerment, disability, discrimination and confidentiality (glastonbury and lamendola, 1993). the educational dimension includes development of simulation software in the areas of child protection (schoech, 1996) and discrimination (seabury, 1993), and information software in the areas of child protection (child care information system from ssradu), care management (unlocking care management from ccetsw), social work literature (caredata from nisw) and social work and technology (social work and information technology from chst). within the social work education dimension to this co-operative work there are, inevitably, both similarities and differences. the differences are exemplified by development within the netherlands, belgium and some other continental european countries of the concept of'social informatics' (visser, 1996). in these european contexts, the concept is dealt with as a taught course on the curriculum (roosenboom 1993), rather than the more experiential introduction to using technology as an educational medium across all aspects of the curriculum, found in british and north american higher education. cal for social work: some conceptual issues and concerns in comparison with other forms of non-traditional instruction print-based, video, audio etc. cal has yet to make much impact on social-work education delivery for reasons outlined earlier. together with other issues beyond the scope of this paper, these constitute a formidable set of barriers and obstacles which advocates of cal for socialwork education must address. however, at the very centre of the debate about the use of cal for social-work education and training lies a much more fundamental set of issues. put simply, these are concerned with the ability of cal to offer and deliver a teaching and learning experience which is as effective as more established modes. 19 tom hopkins and david cobmbi technology for social work education in order to challenge some of the criticisms which are made of cal for social-work education, it is necessary first to understand something of the conventional practices of mainstream social-work education, and the values, attitudes and beliefs that underpin them. based on a view that the nature of social work is contested and contestable, and that these complexities and ambiguities must be reflected in the curriculum, the main characteristics of contemporary educational practice in this field are that: most learners already possess some relevant knowledge and skills; teaching and learning must be, at least in part, experiential (real or simulated); education and training methods should be learner-centred; learners should be involved in identifying and managing their own learning needs; instruction should include, wherever possible, elements of group-based learning; staff should act primarily as facilitators and enablers of learning. while not each and every class or study-session of a social work course will incorporate every aspect of the above, learners and staff will, overall, share a view of the education and training process to which this description closely corresponds. it is not difficult to see that a central tenet of contemporary social-work education is interaction, between teacher and learner, and amongst learners themselves. thus, any new instructional medium has not only to contend with the set of macroand mezzo-level issues outlined at the beginning of this section, but must then prove itself in relation to a set of micro-issues specifically concerned with the nature and experience of teaching and learning. put at its simplest and most challenging new modes of education and training must be able not only to match the content of conventional offerings but also their process and outcomes. this would be challenge enough to cal if its use was to be simply adjunctival, complementing or supplementing mainstream methods of instruction. the literature of cal's recent history in higher education generally, and specifically in social work, shows this is its primary function; staff recommend rather than require student use. however, increasingly british higher-education funding support for "cal development is linked with notions of 'greater efficiency', which translates into cost-savings, particularly of teacher-learner contact time. thus, cal for social-work education and training must, if it wishes to be taken seriously as a mainstream method of instruction, be designed primarily as a replacement for, rather than a supplement to existing delivery. put another way, to be acceptable to both educators and learners, cal must offer and deliver learning experiences at least as effective as those currently in use. in trying to respond to this challenge, the temptation is to look first for technological solutions, for example by designing cal materials that are sophisticated, incorporating sound and graphics, possibly video, perhaps in cd-rom format, in the belief that these will in some unspecified way compensate learners for the absence of more conventional features of teaching and learning, particularly that of engaging with the content of the material. yet, most social work departments currently lack access to the computer 20 alt-j volume 4 number 2 hardware required to run such programs, or on such a small scale that use of cal material could never be more than adjunctival. while multimedia features play an increasingly important role in cal, greater technological sophistication alone will not redeem a program which lacks a clear and coherent teaching and learning strategy. effective software requires interactivity both at the level of design of how imaginatively users can access and be presented with material, and at the level of how they are helped to respond to the content. thus the most appropriate and fruitful response to the challenge of developing cal materials for social work lies initially in a deeper understanding of what constitutes effective teaching and learning, and how the content, structure and process of cal materials can be designed to deliver it. developing the interpersonal skills module we turn now to the specific work within the procare project team of creating social work and nursing modules in the field of interpersonal skills (ips) under the headings of design and structure, content, functioning and implementation. design and structure curriculum design within uk higher education has, in recent years, become an extremely complex task. where once curriculum content and structure were the sole province of academics, a range of influences, some competing, now play a significant part in shaping the planning of courses and programmes (for a detailed consideration of this area, see hopkins, 1995). of most significance here are: the near-universal modular restructuring of higher-education courses; the shift from an input-driven to an outcomes-led curriculum model; attempts to create interdisciplinary and interprofessional education. the first of these led to an early decision that cal materials would be designed to fit the modular course structures now commonly found in higher education throughout the uk (for a discussion in relation to social work, see storan, 1992). the decision called for materials to be designed within a context of prescribed module sizes, expressed in credit points, with indications of likely overall 'student effort hours' required for successful completion. this would assist social-work tutors who wished to consider the use of procare materials as part or whole replacement for existing teaching arrangements. the second factor was significant for program design in that it required cal materials to state clearly and consistently the objectives of each module (and in turn, each unit and session), and how the learner would achieve them. the third factor is of greatest significance for this paper, in that it speaks to the professional contexts of social work and nursing, and thus, by implication education and training in preparation for it. it has two major strands, which briefly may be characterized as follows: that it is both possible and desirable to blur or erase existing boundaries between professional groupings, with the objective of creating interprofessionally-skilled 21 tom hopkins and david colombi technology for social work education practitioners (see weinstein, 1994); that a) may be achieved while simultaneously embracing the increasing shift towards 'specialism' within professional practice. in designing the interpersonal skills and research methods modules, these two factors manifested themselves in the following ways. interprofessionalism. our early belief that it would be relatively easy to design materials for interprofessional use was challenged by educators and students alike. this led us to consider a choice between separate versions for social work and nursing, or a combined program offering choices about specific social work or nursing case examples. in practice, it proved necessary to develop separate versions for interpersonal skills, given the more fundamental differences involved between the two professions, allied to practical issues about developing and evaluating the source materials. for research methods, the differences between the professions were less critical and a single module approach was developed which allowed the user to choose at any stage between being offered social work or nursing case examples, specialism. here, the central question for development concerned whether to meet needs of specific groups in and related to the two professions, such as staff working in residential settings; in special educational needs; in the fields of disability and psychiatry; or in voluntary and private-sector settings. a decision was made to provide materials of a generic nature, while earmarking them for future development once basic versions of the programs are stabilized. content with the primary focus on content in mind, in seeking to create a framework for development, the following elements of curriculum philosophy and design were conceived of as central: what are the overall aims of the module? what are the objectives of the module, expressed as competencies to be achieved? what knowledge and skills must learners acquire in order to become competent? what knowledge and skills might learners already possess, and how can they be accessed? what might constitute an effective teaching and learning process? what might constitute an effective module structure? discussions of these issues amongst project staff resulted in a design for the ips module consisting of eight units, which together cover the key phases of contact between social workers and their clients, viz: communication and ips making contact assessment planning implementation _ alt-j volume 4 number 2 maintaining contact endings evaluation this structure is complemented by two key tools, described later, of a notepad facility and a skills bank. most of the eight units above consist of two cal sessions; each session, following a short introduction and objective setting stage, includes the following elements, in a similar sequential format. context activity. these require learners to begin each session by identifying, through recall and review, past and/or current relevant knowledge and practice skills. this activity also helps orientate the learner towards the new information and other activities encountered within the session (see figure 1). interpersonal skills module social work jze eile edit search navigate annotations bookmarks vjvindow help kirnexit £ q 4.1 context activity 1 hink about somewhere you have recently worked and then answer the following questions: • if planning for service was carried out, what form did it take? • what was planning trying to achieve? • who did planning involve? • if users were involved, how? • what skills did you and/or other workers use in planning? • if planning wasnt undertaken, but in your view should have been, what skills do you think it would have required? contents | a. planning session 1 session objectives the nature & purpose of planning context activity brief recap from assessment key steps & skills for planning review and integration activities reed and search session 2 notebook ] s k i l l s b a n k j figure i teaching content. information is presented in the form of major theoretical perspectives, key concepts, critical issues, contrasting viewpoints, etc. much as in conventional teaching, it seeks to engender a sense of enquiry; critical thinking, and a willingness to live with the ambiguities of professional practice (see figure 2). 23 tom hopkins and david colombi technology for social work education interpersonal skills module social work d i e edit search navigate annotations bookmarks window help kirnexit 4.1 key steps < skills for planning a m o n g s t the general skills of planning are several you have already encountered in the previous sessions. these include: listening, attending, observing and interpreting non-verbal behaviour, clarifying, timing and pacing, and identifying user resources. if you wish to refresh your understanding of any of them revisit them in the communications and interpersonal skills unit under types of communication and in the making contact unit under the skills for making initial contact. in addition to the above skills are some which though not exclusive to planning have a particular part to play in it. they are described in detail below in relation to three key steps to effective planning. step 1: developing a more desirable scenario step 2: clarifying the consequences of new scenarios step 3: making choices and commitments contents 4. planning session 1 session objectives the nature & purpose of planning context activity brief recap from assessment key steps & skills for planning review and integration activities read and search session 2 n o t e b o o k \ s k i l l s b a n k | figure 2 review and integration activity. as its name suggests, this activity requires students to work at integrating existing and new learning through a practice-focused exercise such as the one in figure 3. integration and preparation activity. students are required to use the learning they have acquired during the session (and preceding ones) in preparation for real or simulated practice. learners may be asked to undertake a range of tasks, including: searching for evidence of specific interpersonal skills embedded in case dialogue; planning for handling hypothetical interactions between workers and clients; proposing alternative ways of dealing with complex interpersonal communications between clients and workers; undertaking self-assessment of their own interpersonal skills, and identifying ways and means of improving them. as the above illustrates, within the ips module, structure and process are inextricably linked. laying out the various elements of the curriculum strategy helps learners to develop a familiarity with the way in which the module works in educational terms, while also recognizing that all effective learning involves a process and outcome. students are also prepared for their use of the module by a user's guide which explains, in a non24 alt-] volume 4 number 2 interpersonal skills module social work hie edit search navigate annotations bookmarks window help jz m exit crffc) 4.1 review and integration activity 1 this task focuses on the dialogue between the worker and ms. smith which you have been following throughout the session. the complete dialogue is shown again but this time go through looking for evidence of the following interpersonal skills being used by the worker. when you think you have identified one, click on the sentence to select from the list shown and see if you are right. (nb: each skill may be found more than once.) recall and review goal setting focusing elaborating empowering clarifying mobilising progressing summarising identifying user resources worker: "well mrs. smith, when w e met last week it was clear that tom's behaviour was proving too much for you and his brothers and sisters to cope with. contents | a. planning session 1 session objectives the nature & purpose of planning context activity brief recap from assessment key steps &. skills for planning review and integration activities read and search session 2 notebook | s k i l l s b a n k j figure 3 jargonistic way, how the modules were conceived, how they are intended to work, and how to obtain help if problems, either technical or educational, are encountered. functioning the way that software functions is shaped by a variety of factors including the specification of computers available to users, the programming and design time and skills available, the program size planned, and the authoring system(s) and environment used. the first of these factors was discussed earlier as a significant limiting factor for implementation in most social-work educational settings. other key aspects considered here are navigation, hypertext, special tools and authoring system(s). navigation. the requirements were that users have a clear overall view of the program, and their current position and progress, combined with freedom to move around the program at will rather than along pre-set paths. this presents dilemmas in that the knowledge incorporated in each program is incremental and cumulative so that 'naive' users need to understand earlier sections to gain from later ones. that the material has a logical progressive structure is, however, no reason to provide a straitjacket; just as books allow users to browse, flip, dip and peruse as well as to read through from start to finish, so should cal. hypertext. use of hypertext facilities provides definitions, explanations, answers to questions and reference information in the form of pop-up notes. it is also used for 25 tom hopkins and david cobmbi technology for social work education making links between different sections of the program so that users can move from one topic of interest to another in a non-linear fashion. hypertext is one aspect of interactivity in which the user makes decisions and choices rather than just being a passive respondent. special tools. the notepad facility referred to earlier allows students to undertake written tasks, to make notes, and to copy key materials for use in assignments and seminars. it is context-sensitive with relevant text already included, and notes can be stored, printed or transferred.to a word processor of choice. the skills bank is an index of interpersonal skills referred to in the program with a brief description of each. students are helped to understand that most skills are of a generic nature, with applicability in a variety of practice situations. other interactive features include demonstrations of specific humanservice tools and techniques, such as ecomaps and genograms. authoring systems. given that both modules are largely text-based and offer less scope that many other subjects for visual representation, easy handling of text and use of hypertext facilities were important criteria. this led to the choice of the windows-based guide program for the cal authoring of interpersonal skills, with some visual basic add-ons for features requiring particular forms of representation or interactivity. for research methods, this was switched to use of a visual basic environment which accessed guide text windows with the aim of achieving a more professionally designed product, a move from scrolling to paging, more integrated and systematic accessing of special features such as the notebook, glossary and other add-ons, and a more promising development route for multimedia and extended graphics. implementation implementation has involved establishing test and trial sites within selected social-work courses for evaluating staff and student reactions to ips. other less formal feedback has been through demonstration at workshops and conferences. the programme for formal evaluation was initiated by a training the tutors event, at which social work educators proposing to introduce the module in their work with students were offered an intensive 24-hour orientation workshop. this proved invaluable in a number of respects in that it: created a 'safe haven' for educators and procare staff to discuss the potential and limitations of ips materials; offered an opportunity for shared creative thinking on how to implement module-use in different settings and circumstances; generated the basis of a tutor-use network for ongoing support and development. initial reactions to the ips module have been encouraging, with important lessons learnt about content, structure, presentation, and how the program is used. of particular interest have been the evaluation comments endorsing the project team's decision to provide a program requiring a low level of technology, and which appears to have been easy to install and maintain, both stand-alone and networked. currently, within test and trial sites, the primary focus has been on use of ips as an additional learning resource rather than as a complete replacement for existing teaching. as tltp materials are made available at no cost to uk higher-education institutions, this does not constitute a 'product acceptability' problem at this stage. however, it may 26 alt-j volume 4 number 2 become a critical issue if a move is made to offer procare materials on a more commercial basis; there, economic viability may well be determined, in large part, by its potential to replace rather than supplement teaching. references adams, r. and hopkins, t. (1996), 'towards a systemic approach to the development of open and flexible learning environments: studies of implementation in the united kingdom' in fandel, g., bartz, r. and nickolmann, f. (eds), university level distance education in europe, berlin, deutscher studien verlag. centre for human service technology (1996), resources guide: special issue of new technology in the human services, 8 (4), chst, university of southampton. colombi, d.p., rafferty, j. and steyaert, j. (1993j, human services and information technology; a european perspective, enith (isbn: 0-854-32476-3). glastonbury, b. and lamendola, w. (1992), the integrity of intelligence a bill of rights for the information age, basingstoke, macmillan. hopkins, t. (1995), 'old wine in new bottles? issues in cal curriculum design', new technology in the human services, 8 (2). hopkins, t. (1966), using open learning in social work: a guide to implementation, open learning foundation (in press). roosenboom, p. (1993), 'the dutch "national" curriculum computer applications for schools of social work' in liederman, m., guzzetta, c., struminger, l. and monnickendam, m., technology in people services, new york, haworth. seabury, b. (1993), 'interactive video programs: crisis counselling and organisational assessment' in liederman, m., guzzetta, c., struminger, l. and monnickendam, m., technology in people services, new york, haworth. schoech, d. (1990), human service computing, new york, haworth. schoech, d. (1996), 'multimedia training for child protective social workers: results of initial development and testing' in rafferty, j., steyaert, j. and colombi, d.p., the human services in the information age, new york, haworth. storan, j. (1992), credit accumulation and transfer for the diploma in social work, ccetsw. steyaert, j., colombi, d.p. and rafferty, j. (1996), 'human services and information technology: an international perspective', arena (june). visser, a. (1996), 'computers in education: added value leading towards better quality' in rafferty, j., steyaert, j. and colombi, d.p., the human services in the information age, new york, haworth. weinstein, j. (1994), sewing the seams of a seamless service, ccetsw. details of all computer programs referred to in the text are found in the centre for human service technology (1996), resources guide, referred to above. 27 knowledge tree: putting discourse into computer-based learning t. j. brailsford, p. m. c. davies, s. c. scarborough and w. j. trewhella department of life science, university of nottingham most cbl materials currently in use model only the declarative aspects of the learning process. if such courseware is used without careful planning, this can be dangerous because one of the most fundamental aspects of education is the dialogue that occurs between teachers and the students. traditionally, this has taken place in informal discussions as well as in formal small-group learning sessions such as the conventional tutorial. however, as the student-staff ratio increases, so does the opportunity for this type of personal dialogue decrease. modern networking technology offers a huge potential to add discourse to cbl, but there are many pedagogical problems involved with the intrinsically ephemeral and anarchic nature both of the internet and of most conferencing or bulletinboard systems. in this paper we describe a software system called knowledge tree (kt) which we have developed to address some of these issues. kt combines a hierarchical concept-oriented database functionality with that of a usenet-style bulletin board using this, a knowledge garden may be developed for any subject area. these each contain a hypermedia database of frequently asked questions, together with answers provided by subject experts. there is provision for inter-student discussions of problems and issues. when students ask new questions these are automatically emailed to a relevant subject expert (determined by a subject-specific concept thesaurus). the answer is then placed in the database which eventually grows to become a valuable teaching resource. kt is disciplineindependent as the concept thesaurus can be changed to encapsulate any domain of knowledge. we have used it in support of conventional lecture courses, as an important component of a multimedia course, and for general it support. these examples illustrate the role that this system can play both in basic information provision, and in facilitating the discussion of deep issues. introduction the learning process one of the most fundamental aspects of education is the dialogue that occurs between teachers and the students. laurillard (1993) has discussed this in depth, and has concluded that any teaching strategy should make provision for a dialogue which reveals the conceptions of both the teacher and the student. this allows the teacher to analyse the students' understanding, and to use this information to form the basis of subsequent dialogue. 19 t.j. brailsfbrd et at knowledge tree: putting discourse into computer-based learning traditionally this process has taken place in informal discussions, as well as in formal small-group learning sessions such as the (old-fashioned) tutorial. a real problem that is currently facing education is that as the student-staff ratio increases, so the opportunity for personal dialogue decreases. in many university departments, tutorials have much less pre-eminence than they once did because they are often not perceived as being a costeffective form of teaching in comparison with declarative methods such as lecturing. it-based teaching strategies computer-based learning (cbl) is currently being widely promoted as the saviour of teaching quality in the face of diminishing resources. although it is true that cbl has this potential, there is a real danger that if it-based educational strategies are not implemented very carefully they could damage the learning process. the majority of courseware currently available is primarily declarative, embodying the 'show and tell' learning model. when this is used as part of a carefully designed teaching strategy, such material can be extremely valuable, but it must be remembered that it models only one type of teaching. the learning process can be viewed as a perpetual cycle of three phases, each of which can potentially be reproduced in courseware (mayes, 1993; mayes and neilson, 1995). this starts with the initial conceptualization of the subject material from declarative sources (for example, lectures or books). there are then phases of construction (for example, essay writing or problem solving) and dialogue. students who fail to perform adequately in the construction or dialogue phases can be given the assistance they require to improve their conceptualization. most pieces of courseware are intended for the primary exposition of subject material, and thus tend to be declarative in nature. this is what mayes and neilson (1995) have termed primary courseware. secondary courseware uses the constructionist model of learning, and provides tools and objectives for students to enable them to manipulate resources (i.e. the students effectively become the authors of their own courseware). tertiary courseware provides distributed learning support, and in cal it is this which can create the dialogue that is so essential to effective learning. collaborative learning barrett (1989) has pointed out that when computers are used in education, the social processes of interaction and collaboration that occur during learning should be modelled in the computer. collaborative group-learning is a well established educational strategy, which is readily amenable to extension and enhancement by it (collis, 1994). in a study of a computer-supported collaborative learning system, alavi (1994) found that students attained a higher level of skills development and self-reported learning than did their conventionally taught colleagues. an important aspect of this type of collaborative learning is that not only is there an opportunity for dialogue between students and their teachers, but there are also opportunities for discussion amongst peers. a major feature of the jitol (just-in-time open learning) project is that peer discussions are used to create a hyperstack of public knowledge (boder, 1992; goodyear and steeples, 1992). the rationale of" this is a process that boder (1992) calls the 'reification of interactions'. this involves new knowledge being acquired by a recursive process of debate which stems from previous knowledge. this newly acquired knowledge is then used as the basis for continuing debate. 20 volume 5 number i the recent massive expansion of global networking technology has provided a number of new opportunities for learning through dialogue. automated email distribution lists and usenet news provide a means of distributed electronic debate. there are, however, problems with using such systems for teaching. email lists are an effective means for distributing information to a large audience, but they tend to be a clumsy forum for debates. if an individual subscribes to a significant number of lists, he or she will be continually bombarded with parts of discussions on a variety of subjects. usenet is rather better on this score, but its intrinsically anarchic and ephemeral nature inevitably pose serious problems if it is used for teaching. the answer garden approach to learning a number of systems, of varying levels of sophistication, have been developed that address some of these problems. the most interesting and innovative of these has been the answer garden (ackerman and malone, 1990; ackerman, 1994). answer garden was not originally designed for use in education, but rather for the distribution of information from an organization's 'memory'. this 'organizational memory' is that diffuse pool of authoritative knowledge present in an organization but not in any individual. answer garden was designed to provide a means to tap this pool, and it was envisaged that this would be used for purposes such as technical support or customer hot-lines. in such environments there is a continual stream of questions, most of which recur frequently, but there are always some that are novel. answer garden presents users with a branching network of diagnostic questions about their problem. if there is an appropriate answer in the database, this will be found; but if there is no such answer, an email will automatically be sent to a relevant subject expert. both the question and the answer are eventually incorporated into the database which thus grows 'organically'. in our view, this basic approach to knowledge dissemination is also highly applicable to education. however, there are a number of ways in which the original answer garden is not ideal for educational use, the most important of which is that each user is fundamentally isolated. he or she can see questions that others have asked in the past, but has no knowledge of who else is currently using the system or what they are discussing. if used for teaching, answer garden would provide a student-teacher dialogue, but no peer-peer interaction. in order to address some of these issues we have developed a software system called knowledge tree (kt) inspired by the answer garden concept but extended in a number of important ways. we have been using this system in undergraduate teaching since early 1994. knowledge tree: a collaborative learning tool for networks the kt software combines a bulletin board (bb) functionality with a hierarchical concept-oriented database to provide a structured environment to host and archive discussions at both a student-student and student-teacher level. using kt, a knowledge garden may be developed for any subject area, and these each contain a database of frequently asked questions together with live discussions of problems and issues. 21 t.j. brailsfbnj et al knowledge tree: putting discourse into computer-based learning subject kxpert figure /: the knowledge tree system model the system model is shown in figure 1. kt is divided in to two distinct but related components called the forum and the knowledgebase. the forum contains primarily student-student interaction and is essentially a threaded bb where students can ask questions and participate in ongoing debates. new questions asked in the forum are automatically emailed to relevant subject experts, and together with the answers they are archived in the knowledgebase, which is effectively a subject-specific hypermedia database. epistemology each garden has its own subject-specific thesaurus consisting of a hierarchy of concepts that is displayed as a folding 'outline'. in the forum, students use this to attach a concept to their question, and in the knowledgebase it forms a browser used for navigation. the thesaurus uses a system of upward inheritance to provide a powerful means of structuring any subject domain. for example, using the simple thesaurus shown in figure 2, any concept b rowser;; * plants animals * invertebrates =• vertebrates •=> mammals •= carnivores 4-dogs •=cats lion tiger cheetah etc... figure 2: an example of a simple concept thesaurus to illustrate the use of inheritance 22 alt-j volume 5 number i question with the concept of 'tiger' attached to it would be found under carnivores or cat, but not under dog. dialogue in the garden in the forum, students can see the questions their peers have asked, and can comment on them or answer them (see figure 3). this results in discussion threads similar to those found in usenet or most bb systems. articles in a thread are not automatically deleted, but instead the entire thread is removed once it is obsolete. this avoids the problem of decapitated threads that bedevils systems such as usenet: frequently, users find they can read part of a thread, but not the original posting. this can lead to the vague ambling from one subject to another that is so common in news groups; it can be entertaining and sometimes informative, but it is not ideal for the type of debate that should be encouraged in education. while reading a thread in the forum, the students may press a button and jump to the appropriate knowledgebase article (see figure 4). kt thus supports dialogue both between peers, and between students and their teachers. the forum provides an area of dynamic debate, but knowledgebase articles are permanent. thus the garden grows with time, eventually to become a valuable resource. 'edit" help knowledge garden: biological anthropology knowledgebase about this author scntoc lecturcf srare any set criteria to distmgufsti a "graafe' and department of u e science. b o * size *>bod> (ona/ihape skin colour skeletal anitaa? 4u uravenx; of nottingham. gradte or robust? |t) vertebral epipwaaa •s-giofe * betutvww/fonctwn •evolution •knowledge fiaxlen no lmequiyocbl, quantitative crib began with the discovery of clea from a guksiws and a rabostui sample saes have g r ' d the dtstottion [ih< become somewhat bt example) predicted be alometnc equations cold adaptation. at rahuil? f i t brec3tn.iiobu.ttp} ktal.ni.pl.? (11 m4e:nipple?r.2) but theft robuttntts but more tf'bubtf of oxtinct robusttwnriidtjidm south afhca. , an a m ' : hipiesiknof a •v 3 .' they ragrt nave tppise , <-'', atastatop^xabviiiblm anodataltaunal(on3.«j9g«tsavannas . ' 2 hat**. qpenfonn figure 3: part of the knowledgebase from a biological anthropology knowledge garden the knowledge tree in use we have used kt in quite a number of different applications for three cohorts of students. it provides ancillary support for two conventional lecture courses, and forms a 23 t.j. brailsford et al knowledge tree: putting discourse into computer-based learning die e<>» help knotttedge g a r d e n : information technology knowledgebase j * mt: "hardware ibm pc a r t * * mac cecsmtttm when fetehtaa tinoe presraras far fee uracorti they are often compressed j : .tar etc, which ones matter,, how tfo t uncompress them, tan i tor these sr» ttie tmicor*i? forum woidplocamra spmaddi bianmnb ctmeeptsaarch ascii feagek uf t l •clock «pe»<m1) •colour <rapr>i($ -£rickat£r«pknr«t*»(4) " f « p d j21 •wow midi «e*p«e 7 <1] tr«enie<bad«|2) •mat dkimtmitt (w «a«tmtt (31 • d t t ) te there a smpte way of redirecting e-mait trom vmet to t pegasus maif? also can one recover past messages from > vme( tntaf5egasu$..;for instance for prmwig purposes. figure 4: part of the forum from an information tedino/og/ knowledge garden major component of a module based entirely around cbl. in addition, it is used to support general it queries in our department. some long threads have been developed about quite deep issues, and some of the student contributions have been of extremely high quality (in a number of cases we have transferred student answers directly to the knowledgebase). one of the main aims of kt is to lighten the burden of responding to student problems, as queries automatically go to the most appropriate available expert. once a query is answered, this is publicly available until it is no longer appropriate. a good answer should promote reflection and further study rather than provide a package of selfcontained information. in the case of universities, the experts are likely to be academic staff, although this need not be the case. it would be quite feasible to make use of graduate students or teaching assistants to provide answers. there are also potentially many more ways of using kt than we have yet tried. the effort that some students have put into writing articles in the forum has convinced us that the system has potential as a constructionist tool as well as a learning-support tool. it would be an interesting experiment to run a student-administered garden, where groups of students are allocated a subject domain for which they have to research and create their 'official' answer. it might even be possible for students to create their own gardens from scratch for each other, as building a good thesaurus requires quite a deep understanding of the subject material. another possibility would be for teachers to ask questions in the forum to seed 24 alt-} volume 5 number i student debate. kt can also be used on various different scales. on a large scale it can provide support for an entire course and cover many different topics. it is equally usable on a small scale, and would provide an interesting alternative to essay-writing in an electronic equivalent of a tutorial group. integrated courseware and the future of cal we have found that kt can provide useful support for conventional teaching methods, and can be a valuable addition to conventional declarative cbl in that it can offer the staff feedback and peer debate that cbl usually lacks. the real potential of this approach to learning will only be realized, however, with fully distributed systems. so far, kt has been used only to support collaborative learning on local-area networks. in the near future, we hope to run kt across wide-area networks, and this opens the interesting possibility of inter-institutional collaboration to provide distributed learning support. thus instead of a garden being tied to a single course in a single institution, it could be used to support similar courses running concurrently in several different institutions. students would thus be able to participate in debates with students in other institutions, and the subject experts would be drawn from the staff of all of the participating courses. this would make the most effective use of available staff resources, and provide a dynamic learning experience for the students. an interesting project that has implemented a distributed approach to the answer garden principle is the answer web system (mayes and neilson, 1995). this is effectively an implementation of the original answer garden concept on the world wide web. the most important aspect of this is that it can be used in an environment where the world wide web is an important element of teaching, and thus it will be seamlessly integrated into other courseware. we hope eventually to integrate knowledge gardens with more conventional declarative courseware, so that they simply become a tool in a learning-support environment. this raises the interesting possibility of a type of 'living hypertext', where a link in conventional courseware (possibly automatically generated) could take the student to a relevant live discussion in which they could participate. this could, in principle, provide composite courseware that would contain the primary educational material, but would also provide tools allowing students to collaborate with their peers and interact with teachers. courseware of this type would then be much more capable of fulfilling the entire learning process than most courseware available today, and it could thus be more safely used as a core component of teaching lectures and conventional teaching methods are likely to remain important for the foreseeable future, but courseware of the type described above could easily teach the mundane parts of many courses. lectures could then become the inspirational performance they should be, rather than the mental filling stations they all too often become. acknowledgements and further information we are most grateful to don whittle and craig stewart for their help during the development and trials of knowledge tree. we should also like to thank wendy hall, 25 t.j. broitsford et al knowledge tree: putting discourse into computer-based learning terry mayes, brian shields and barry wood for many useful discussions, bourne robinson for programming , and the ibis team for their support. this work was funded by the itti and tltp. further information about the knowledge tree software can be obtained on the world wide web at http://ibis.nott.ac.uk/software/kt.html. references ackerman, m.s. (1994), answer garden: a tool for growing organizational memory, ph.d. thesis, cambridge ma: massachusetts institute of technology. ackerman, m.s. and malone t.w. (1990), 'answer garden: a tool for growing organizational memory' in proceedings of the acm conference on office information systems, new york: acm, 31-9. alavi, m. (1994), 'computer-mediated collaborative learning an empirical evaluation', mis quarterly 18 (2), 159-74. barrett, e. (1989), 'thought and language in a virtual environment' in barrett, e. (ed.), the society of text, cambridge ma: mit press. boder, a. (1992), 'the process of knowledge reification in human-human interaction', journal of computer assisted learning, 8, 177-85. collis, b.a. (1994), 'collaborative learning and cscw research perspectives for interworked educational environments', ifip transactions, a computer science and technology, 46, 81-104. goodyear, p. and steeples c. (1992), 'it-based open learning: tasks and tools', journal of computer assisted learning, 8, 163-76. laurillard, d. (1993), rethinking university teaching: a framework for the effective use of educational technology, london: routledge. mayes, j.t. and neilson, i. (1995), 'learning from other people's dialogues questions about computer-based answers', ifip transactions, a computer science and technology, 61, 31-47. mayes, j.t. (1993), 'commentary: impact of cognitive theory on the practice of courseware authoring', journal of computer assisted learning, 9, 222-8. 26 italian via email: from an online project of learning and teaching towards the development of a multi-cultural discourse community marina mozzon-mcpherson university of hull this paper seeks to illustrate how the use of internet resources (specifically email and the web) can affect and enhance language learning and cultural understanding, modify the learning environment, reduce the barriers which time, space and societal differences may create, be a source of motivation, and redefine the role of teachers and learners. although it is based on an on-going project, it already provides practical evidence of some advantages email and internet resources can bring to the language learner and to the teacher. a detailed evaluation of the language outcomes is under way, but incomplete at the time of writing. this paper is nevertheless more concerned with other variables of language learning and teaching which the author considers fundamental to reach a successful degree of language use. introduction: the nature of the project the project originally started in january 1994 and involved the creation of an email-based interactive italian-speaking community (isc) in the form of a discussion list ('due chiacchiere in amicizia'). it was initially limited to on-campus learners of italian or 'real' italians. when the project started, the aims were: • to provide online support for students who needed extra help in the learning process; • to support learning motivation through one-to-one communication with the tutor; • to provide the opportunity for additional language practice; • to encourage students to reflect on both the language process and content; • to provide a 'special niche' where learners from different nationalities, backgrounds and experiences could 'meet' with italians and have the opportunity to explore the socio-cultural dimension. the first period was left very open to permit familiarization with the use of email (not everybody was a regular email user). i provided short texts to be read, containing particular 40 volume 4 number i grammatical points, and produced language exercises (gap filling, multiple choice questions, question and answer, etc.). the italian contribution varied from participation in story-writing activities to riddles to solve and questions to answer. initially, i selected topics which were relevant to what the learners were doing in class (europe and being european, past experiences, holidays, politics, reasons for learning italian, etc.). occasionally, i suggested some web sources (other discussion lists, newsgroups, etc.) where they could learn more about current affairs or cultural issues, or keep up to date with the reality of their country. although the participants responded positively to the activities as reflected in a substantial increase in the number of messages after the first three weeks 1 sensed that the reaction was more from a sense of duty than from genuine interest. apart from the story-writing activities and the riddles which they liked, the learning environment was not leaving enough space for 'meaningful communication' (see brown, 1980; and halliday, 1985). the first four of the five aims mentioned above were satisfactorily achieved by the project as it stood. however, the creation of a real interactive community with its own dynamics was far from being achieved. it resembled closely a replica of communicative teaching techniques on screen. the influence of the teacher (myself) was too evident, and this showed in the formality of some messages and the type of requests. in january 1995, membership was therefore opened to other learners of italian and italians globally. absolute beginners were allowed to join in as silent participants and when or if they felt ready, they could give their profile or ask for clarifying questions. consequently, the isc became more diverse. membership included canada, spain, america, germany, france, ireland, brazil and poland, to mention only some, while the 60 or so members (gained in the first six months) were equally mixed in terms of age, sex and occupation, and included the participation of two blind students. for these latter members, email represented a comparatively simple, economic and less discriminatory system which allowed them to communicate with the rest of the world. because their world already contemplates a different visual dimension, the 'cluelessness' of email put everyone in the same position. while in a face-to-face situation, they may have been disadvantaged by the use of visual clues; in this 'blind' communication, the only clues were coming from within the texts. as an active member commented: 'email gives me the chance to be heard for what i have got to say, regardless of my appearance'.1 as a matter of fact, the two blind participants played a very active role in stimulating discussions. a group dynamics developed, the shape of the messages changed, the tone relaxed, the messages became longer, and the language style more adventurous. the relationship between the participant, the others and the screen became strongly directed by the individual's agenda and satisfaction of perceived needs, interests and wants. one student noted that he would rather spend more time discussing issues on italian life and culture than go to lectures which had no bearing in his real daily life. another pointed out that 'it was not discussing for the sake of language learning'. the exchanges became more meaningful and carried the richness of the individuals' identities. in some cases this meant diverting discussions from italian-related topics to more personal concerns about the world and the way things were going, their experiences of cultural shock, and political and social beliefs. the next stage was the request by a participant to forward to the isc a brief personal introduction which could allow the rest of the group to empathize with their fellow learners. 41 marina mozzon-mcpherson italian via email: from an online project towards a multi-cultural discourse community the creation of a folder containing all these profiles proved an important step. this file was given to all new isc participants on joining. the personal profiles helped to create lines of interests, and allowed the integration of new meanings and recognition of a new set of beliefs, making sure that the wit and irony of certain participants could not be misunderstood. this can be clearly illustrated by the incident i now describe. a participant asked a question concerning the current turbulence in italian politics with particular reference to the magistrates' attitude towards the state and the mafia. a member from sicily, doing research in a german company in munich, had just joined. his vivid images and experience of the bitter fight magistrates are conducting against the mafia, and his position as newcomer, caused him to reply with vehemence (see guiora et al, 1972). the misunderstanding was not at language level but at cultural level. it transpired that none of the other italians reacted aggressively because they had already adjusted their ability to 'read between the lines', to negotiate meaning regardless of accuracy, and they were already familiar with the dynamics and socio-linguistic errors of the isc. interactions went from general to specific, from personal to topical and involved a range of skills to tackle the texts and extract information. some reacted immediately, some preferred to think about the topic and document themselves, some put more weight on their language improvement, some by-passed the language issue to concentrate on the meaning. distinguishing factors previous work in the area of technology and language learning (garret, 1991; ncet, 1991; mathiesen, 1993) has examined the potential use of email interaction to encourage students to use computers in a realistic context and enhance their communicative and thinking skills, and the last decade has seen an enormous increase in the use of this communication system for language learning. my project follows the work done by austin and mendlick (1993), kroonenberg (1994) and eck et al (1994), in exploring the potential of email for language learning. half-way through the first year, a number of factors changed and contributed to distinguishing this project from others such as those above. these changes include the group composition, the aims and objectives, the disappearance of the teacher and the development of a community spirit. group composition while the majority of the projects involving email tend to focus on entire classes, 'due chiacchiere in amicizia' consisted of a very heterogeneous, global group far from any comparison with a class. what evolved was an isc which consisted of natives, non-natives and semi-bilinguals from all over the world, and the classroom or campus network shaped into a completely different environment, unexpected to the author of this paper as well. this created a very special and authentic setting within which language and social interaction took place and still takes place. the variables between classroom and email environment are many: the setting of the interaction (physical distance, lack of visual input), the type of communication acts (written communication), the language use opportunities (continuous and asynchronous) and the affective domain (the level of empathy, inhibition, self-esteem, extroversion and 42 ait-) volume 4 number i aggression and motivation). this is another good reason to detach the email environment from classroom comparison and create a new natural environment of authentic interactions. i do not intend to make the assumption that one system may or will substitute for the other, but that language learning is a complex phenomenon which involves social, emotional and intellectual dimensions, and any environment which can help the learning to happen should be tried and welcomed. with kohn (1994), i share the belief that one of the ways perhaps the most significant and effective in which language learners acquire a considerable and vital part of their communicative linguistic competence, is through social interaction. it is through the learner's contact with the 'discourse community' (be it the workplace, the holiday resort, the university campus or whatever) that learners can measure their competence and modify both language needs and their cultural differences according to the challenges of their environment. the degree of proficiency achieved in the language will then be affected by the extent to which the learner wants to 'acculturate' to the other language group (schumann, 1976); it will depend, therefore, on factors other than simply appropriating words or grammatical rules. although email interaction is text-based, it was very interesting to try to read aloud some of the interactions produced by the semi-bilingual participants. it was my impression that you could easily identify from the spelling of certain words the region of origin of the parents. in particular, the use of double consonants ('sonno un italiano-americano' for 'sono un italianoamericano'; 'mottore' for 'motore') or the substitution of a consonant or vowel ('strata' for 'strada'; 'genetori' for 'genitori'). there were also particular structures ('lui aiuta a sua figlia', 'voglio a praticare a lingua', etc.). this is not the place for a detailed linguistic analysis of the language, but a positive effect email interaction had on the semi-bilingual was the development of language awareness (sometimes hindered by very fluent spoken language) and improvement of written communication. this initial impression needs to be verified by further research. input, interaction and motivation clearly play a major role in language learning in and outside the classroom, and played a key role in the development of the community spirit. the increase in frequency and length of messages, and the cosmopolitan environment, stimulated a further increase in frequency, length, accuracy and motivation. as noticed by a participant, a beginner in italian, 'i am talking to real italians in italy! wow!'. in this situation, errors had a positive role to play, and there was no such thing as making a fool of yourself, which in a face-to-face situation may be a cause of anxiety. participants started correcting each other (native to nonnatives), and members of the community expressly asked to send corrected messages back to the main conference for everyone to benefit from. the aims and objectives the change of the group composition partially affected the original aims of the project in that much more emphasis was put on meaning than on language with the intervention of affective variables; this was less evident in the first stage. although the four major components of the communicative approach (savignon 1983) were still strongly present, the concern moved towards the content of the communication. participants were still working on their grammar (linguistic competence) or their ability to use appropriate writing style to the context (sociolinguistic competence). they were above all concerned with their ability to interpret sentences into a meaningful world and achieve coherent texts which are relevant to the given context especially after a few misunderstandings (discourse competence) and their ability to devise strategies to compensate for imperfect knowledge, for example seeking clarification, repetition, 43 marina mozzon-mcpherson italian via email: from an online project towards a multi-cultural discourse community paraphrasing and silence (strategic competence). as previously indicated, email provided an ideal environment to support the development of the these competences. the attempt to create a real, new type of language learners' community, with its own internal dynamics, and to avoid a replica of a classroom environment, was a successful point of the project. if we agree with some of ausubel's theory (ausubel, 1964) that it is not trie frequency of stimuli or the number of times spent practising that influence learning but the meaningfulness, then the development of a natural communicative setting was a powerful factor. the intensity of the interactions certainly has an important effect on memory retention (brown, 1980), and has good effects on vocabulary expansion. one member commented on how she improved her speaking skills (in terms of vocabulary richness) and overcame her lack of confidence when in discussion. the disappearance of the traditional teacher although the teacher can create a rich environment within which authentic communication can take place, research into the learning process has shown that language learning is a highly complex and individual matter (oxford, 1990; wenden, 1991). taking personal responsibility for the learning, and becoming aware of the mechanisms involved, are strictly related to the achievement of high proficiency. language teaching in the context of this project consisted of setting up the right conditions for interactions to take place and promoting induction work and personal development of strategies to tackle language-focused issues. in this respect, an obscure and difficult grammatical point like the subjunctive was more easily clarified by its recurrence in nearly every message than through a presentation of the rule. one aspect of this approach is the effective application of 'learning by communicating' which stimulated subsequent 'induction work'. this is exactly what happened in the course of the project, where students started to ask questions about new structures. one, in fact, was the subjunctive nearly always present in each text, and new or obscure to many non-native speakers. through examples extracted from real conversations and help from the natives, some managed to extrapolate the rule and act as 'miniconcordancers', creating for themselves a portfolio of useful authentic examples of uses (widdowson 1990; schultz, 1991). there was a noticeable shift from the use of paraphrasing techniques, to avoid the use of such a mood, to the correct use of it. the gradual withdrawal of the teacher as the main language input maker, and only feedback provider, also opened up opportunities for dominant personalities to mould the nature of the discussions. if the communicative aspect of language learning is a vital component of current teaching methodology which tries through group-work activities, simulations and authentic materials (satellite tv programmes, newspapers, etc.) to replicate the native speakers' world, this attempt to replicate 'authenticity' is not without weaknesses. the biggest one is the reduction of language learning to sets of functions and scripts ready for performance, something which tends to limit the ability to react to unscripted language and cultural meanings. after all, life does not always fit the script. it is in the attempt to overcome this weakness that the project positively created a fluid environment of unfiltered, unscripted language and culture. 44 alt-j volume 4 number i the development of a community spirit such development shares on a much smaller scale many of the stages one can see in the community language learning model of rogers (1951), in that the affective and social dynamics were valued, the teacher was seen more like a counsellor and mediator/translator, the style and tone of the conversation was in the hands of the group, the language was inductively analysed, and 'defensive' mechanisms were made unnecessary by the supportive environment. furthermore, the counsellor became a 'client' of the community. as brown (1980) has highlighted, the community language learning model is not without problems, and i was aware that it needed adapting. for this reason, the first stage of the project was very much directed, and only at a later stage was community language learning introduced. however, in order for the group dynamic processes to set and develop effectively, group work should preferably provide opportunities for learners to explore the socio-cultural dimension of language, and prepare and engage then in intercultural communication' (eck et al, 1994,45). eck's remarks lead to another major aspect of this project: culture learning. the emphasis on active use of the language as a technique for learning urged by the communicative language teaching approach has consequently demanded more authentic sources of language, hence the exposure to the language produced by the natives in their natural environment. as seelye rightly stated: if he [the learner] avoids contact with native speakers and if he lacks respect for their world view, of what value is his training? where can it be put to use? what educational breadth has it inspired? (seelye, 1974,21) seelye's questions effectively exemplify my project's philosophy, and highlight the impossibility of distinguishing language from culture. the isc offered a panorama of cultures within the culture, of points of view and attitudes (the sicilian participant represented a different culture from the girl from bergamo or the professional from verona). in this project, an interesting dimension was added by the italian erasmus students who used the environment to escape their inability to express themselves in english, and attempted to compensate for this temporary loss of identity by sharing their cultural values and beliefs with the isc. as an italian participant said, 'i have more to say than i'd like a pound of tomatoes', indicating her frustration at not being able to express her real social, emotional and intellectual self. as cortazzi and jin (1993, 24) have noted 'the lack of shared schemas in interaction are more likely to lead to communication breakdown than differences and difficulties at the level of linguistic code' the isc provided the framework within which widely dispersed and disparate participants could utilize their varied strengths in terms of culture and language, and benefit from the ensuing synergy. while teichmann (1994) argues that 'confrontation with the reality of a different culture can lead at first to a reinforcement of all the prejudices previously cultivated', this paper posits that, in the long term, email and similar media may help overcome stereotypical attitudes, and possibly help a fruitful appreciation of cultural values through a process of 'acculturation'. this is a long-term goal of the project, which will include more sophisticated cultural activities. byram (1989) provides practical ideas to look at stereotypes, prejudices and culture teaching activities. in my experience on the project, when people got together and really talked about what was significant for them, the whole team became more 45 marina mozzon-mcpherson italian via email: from an online project towards a multi-cultural discourse community than the sum of the individuals, and it gave birth to a lively multi-cultural community. such an environment could be positively exploited for preparation for life abroad, either for work or study reasons, and help students to prepare from the culture shock that comes from thinking that 'everyone in the world should be like me' (adler, 1972). project evaluation should the success of the project be measured through the 100% renewal of the subscription for 1995/96? what is perhaps more subject to a critical evaluation is the appearance of a learning community which embraced at least three types of interaction: • between native speakers; • between natives and non-natives; • with semi-bilingual members. further evidence of the development of an internal culture can be seen in the change of style of the messages. messages came to be written and addressed not to me but to 'cari amici della lista' ('dear friends of the list'), or to a particular person, the topics were suggested by the participants, and the length of the messages doubled. after a few incidents, a new level of caution arose, not linked to grammatical concern but to a respect for the other's culture and hence a more careful search of the correct word to exemplify the concept one wanted to express. language and cultural awareness for some members started to be seen as a unity. the position of some semi-bilinguals in the group was interesting. they felt exposed to two cultures, neither of which they necessarily understood deeply. for this particular group, some exchanges were clearly enlightening. despite the asynchronous level of the interactions, the distance and the lack of visual signs, i found it amazing how clearly 'silences' due to nonunderstanding could be perceived, and it was in these situations that my role again became of relevance. this online italian environment was now allowing everybody to be a teacher in proportion to their knowledge and created a micro-society where strong personalities not necessarily the same strong ones you will have in face-to-face communication appeared and moderated discussions. in several instances, other italians corrected language mistakes, and justified their corrections. in some other situations, students took the initiative and re-orientated the conversation. particular expertise came to light, and in order to let the community express itself, i had to learn to step back and not to interfere. i was there to foster this expertise and if necessary to moderate the exchanges, to suggest readings and other sources of useful information, to correct grammatical mistakes (which i regularly did and sent back to the isc with the member's permission), to gauge the level of the discussion, and to make sure that even simple basic questions could still be asked. the exploitation of other web facilities, for example the televideo-news online or newsgroups, or surfing through italy, helped to create a purposeful and authentic environment. some members started asking for information connected with their research areas, etc. i myself discovered places on the web i did not know, thanks to a computer expert actively contributing to the list who regularly updated the list with new web sources of information for italian. 46 alt-j volume 4 number i it was particularly fascinating to observe the gradual linguistic changes (in terms of accuracy and fluency) which some members developed in the course of the year. because of the nature of the medium which supports written interaction learners started with very short and simple messages (on average about four to five lines), and gradually moved to more complex ones. moreover, the speed with which one could receive feedback contributed to creating in the participants this ambiguity of 'written conversation', the illusion of being in a face-to-face situation, and hence sending a message to a 'real' audience online at the time of the dispatch. it was not always like that, however, and sometimes there was a sense of frustration at the 'message in the bottle' which nobody collects. problems also arose over turn-taking or misunderstanding of the rules. another aspect of this type of communication was the apparently incoherent and undisciplined arrival of the messages, which because of the lack of a subdivision per topic was very difficult to control. it was real communication without filters. and, as grenfell (1994) has pointed out, 'real language is just not that tidy.' in most cases, the native speakers were using typical italian structures, vocabulary and metaphors which denied the non-natives any chance of slow, artificial modification of the style in order to make themselves understood. but this encouraged and challenged the participants to go further and ask relevant questions. all of a sudden, they found themselves part of an italian environment, unlike the simulations they may have experienced in a class. certain structures were absorbed by observing how the italians were using them: vocabulary used by the italians reappeared in the non-natives' replies, applying what ellis (1991) has brilliantly called the 'scaffolding' process. email helped to contextualize hitherto unknown written words, and hence enabled learners to remember them vividly and use them actively. what was difficult, and what other more sophisticated computer-conferencing systems (such as firstclass) can overcome, was the inability to sort out messages automatically and allow the flow of the thread. this meant the participants had to scan through all the messages sent to the list to see which one they wanted to reply to. this was particularly time-consuming for the two blind students. in the second phase, the list has now been subdivided in various conferences, each with a thread one can follow. the conferences currently active are on politics, tourism, cultural issues and language. the attraction of email was its potential to engage the imagination of the participants and create positive interactions where they not only moved into new areas of knowledge but lived this experience as a development of their selves. without this empowerment, language learning risks being reduced to mechanical and technical competence acquisition, and hence subordinate to the technology. as per brown (1980), 'the human being is a social animal and the chief mechanism for maintaining the bonds of society is language'. some teaching methods tend to over-simplify or disregard the social factor as relevant for language learning by not recognizing the complexity of the relation between language and society, and language and culture. to date there is no telecommunications system which effectively can reproduce such a complexity. this shows another possible limitation of email in comparison with other telecommunications applications: it does not give space for body language, and has to find surrogates in the use of 'emotions', not always very effective or expressive. nevertheless, any incorrect interpretation of a text caused by the inability to recognize certain socio-linguistic, discursive, grammatical or strategic aspects of the language, provided the opportunity to become aware of these aspects. unexpected or inappropriate responses generated awareness of 47 marina mozzon-mcpherson italian via email: from an online project towards a multi-cultural discourse community language phenomena. once again, this could be seen as a positive aspect of language learning which the email environment was able to develop. conclusion this attempt, which needs further refining, seems to have proved successful in that learners of italian gathered together regularly and compared notes on both cultural and linguistic issues. demand was such that membership had to be closed to foster the development of the existing community spirit. however, from september 1995, new universities in both italy and elsewhere have joined in, with between 5 and 20 participants from each institution. team teaching, which was a missing element of the first phase, has been introduced by inviting lecturers of italian from the other participating universities to moderate particular conferences, and some of the 'old' participants have assisted me in the development of the new framework. a conference has been specifically devoted to language issues, and there are 'surgeries' for learner training for members to 'attend' when in need of clarification on specific points. during this year and a half of interaction, i managed to store and record a huge amount of texts which transform what could have been a conversation lost in space into something durable and relevant. the texts constitute an invaluable collection of authentic interaction which could be re-used to stimulate future discussions. these texts need to be re-analysed in the light of experience gained thus far, and this year's new developments. while language skills can be measured and quantified, cultural interactions seem to be more difficult to quantify. however, further work utilizing henri's analytical model highlighting five main dimensions (participative, social, interactive, cognitive and meta-cognitive henri, 1992) may prove more useful in ascertaining if this 'invaluable collection' lends itself to further analysis. open issues remain the impact on cultural awareness, vocabulary acquisition and reading speed, learning styles, and teacher training to cope with the new counselling and mediating role. some of these things are currently under scrutiny in the second phase of the project. note 1. the participants' observations and comments reported have been translated into english by the author of this paper. references adler, p.s. (1972), 'culture shock and the cross-cultural learning experience' in readings in intercultural education, volume 2, pittsburgh, intercultural communication network. austin, r. and mendlick, f. (1993), 'email in modern language development', re-call, 9, 19-23. ausubel, d.a. (1964), 'adults versus children in second language learning: psychological considerations', modern language journal, 48, 420-4. brown, h.d. (1980), principles of language learning and teaching, london, prentice hall. 48 alt-} volume 4 number i byram, m. (1989), cultural studies in foreign language education, clevedon, multilingual mattes. byram, m. and esarte-sarries, v. (1991), 'investigating cultural studies' in foreign language teaching, clevedon, multilingual matters. cortazzi, m. and jin, l. (1993), 'cultural orientation and language use in thompson, l., graddol, d. and byram, m.(eds), language and culture, clevedon, multilingual matters. eck, a., legenhausen, l. and wolff, d. (1994), 'assessing tele-communication projects: project types and their educational potential' in jung, h. and vanderplank, r. (eds), barriers and bridges: media technology in language learning, london, peter lang, 45-62. ellis, r. (1991), second language acquisition and language pedagogy, clevedon, multilingual matters. friedlander, a. (1990), 'composing in english: effects of a first language on writing in english as a second language' in kroll, b. (ed), second language writing: research insights for the classroom, 109-25. garrett, n. (1991), 'technology in the service of language learning: trends and issues', the modern language journal, 75, 74-83. grenfell, m. (1994) 'communication: sense and nonsense' in teaching modern languages, london, routledge, 54-60. guiora et al (1972), 'empathy and second language learning', language learning, 24, 287-97. halliday, m.a. (1985), spoken and written language, victoria, deakin university press. henri, f. (1992), 'computer conferencing and content analysis' in kaye, a.r. (ed) collaborative learning through computer-conferencing, the najaden papers, nato asi series, series f: computer and systems sciences, vol. 90, 115-36. kohn, k. (1994), 'distributive language learning in a computer-based multilingual communication environment' in jung, h. and vanderplank, r. (eds), barriers and bridges: media technology in language learning, london, peter lang, 31-43. kroonenberg, n. (1994), 'developing communicative and thinking skills via electronic mail', tesol journal, winter, 24-7. mathiesen, a.b. (1993), 'electronic communication media and second language learning', oncall, 7 (3), 15-20. ncet (1991), on-line electronic mail in the curriculum, lingua delta conference proceedings, 1993. oxford, l.r. (1990), language learning strategies, london, newbury house publishers. rogers, c. (1951), client-centered therapy, boston, houghton mifflin. savignon, s. (1983), communicative competence: theory and classroom practice, reading ma, addison-wesley, 37. schulz, r.a. (1991), 'second language acquisition theories and teaching practice: how do they fit?', modern language journal, 75. 49 marino mozzon-mcpherson halian via email: from an online project towards a multi-cultural discourse community schumann, j.h. (1976), 'social distance as a factor in second language acquisition', language learning, 19, 135-43. seelye, h.n. (1974), teaching culture: strategies for foreign language educators, skoki il, national textbook company. swarbick, a. (1994), teaching modern languages, london, routledge. teichmann, v. (1994), 'an interdisciplinary project orientation using telecommunications media in foreign language teaching' in jung, h. and vanderplank, r. (eds), barriers and bridges: media technology in language learning, london, peter lang, 63-8. wenden, a. (1991), learner strategies for learner autonomy, london, prentice hall. widdowson, h.g., (1990), aspects of language teaching, oxford, oup. 50 alt-j, research in learning technology vol. 17, no. 2, july 2009, 75–88 issn 0968-7769 print/issn 1741-1629 online © 2009 association for learning technology doi: 10.1080/09687760903033058 http://www.informaworld.com students make a plan: understanding student agency in constraining conditions laura czerniewicza*, kevin williamsb and cheryl browna acentre for educational technology, university of cape town, cape town, south africa; bhigher and adult education studies and development unit, university of cape town, cape town, south africa taylor and francis ltdcalt_a_403478.sgm (received 11 may 2008; final version received 13 november 2008) 10.1080/09687760903033058alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2009association for learning technology172000000july 2009professor lauraczerniewiczlaura.czerniewicz@uct.ac.za drawing on archer’s perspectives on the agency/structure relationship, this paper explains situations where students in varied, challenging circumstances find ways to negotiate difficult conditions. it reports on a 2007 study undertaken through a survey at three quite different universities in three south african provinces, addressing inter-related questions on access and use. our findings are that on-campus access is generally reported favourably, and off-campus access is problematic and uneven. there is a cluster of students using their cell phones to access the internet, and using their cell phones for academic purposes, and this is true across socio-economic groups (segs). it is especially striking that students from low segs do so. the findings show the choices students are prepared to make and the strategies which they find in order to engage online or access the internet to support their studies. archer’s nuanced approach to agency and structure helps us begin to make sense of the way that students exhibit a more complex and nuanced way of engaging with the availability of different kinds of technologies, as well as making considered decisions about using ubiquitous technologies in unexpected ways and for purposes for which they may not have been intended. her concept of reflexivity provides a way of describing how those choices are made in relation to structural conditions and enables us to explain how students are ‘persons’ showing an inventive capacity to circumvent the constraints imposed by structures. keywords: students; archer; agency; higher education; access; use; icts; information and communication technologies; south africa; cell phones introduction the context of the study is a south african higher education system clearly committed to preparing university students for participation in the knowledge society, as is evident in numerous policy documents.1 however, the response to this rapid worldwide social and economic transformation has occurred simultaneously with the substantial restructuring of a fragmented, divided and unequal sector, the legacy of racially demarcated and differentially resourced apartheid institutions (department of education 2001; gillard 2004). additionally, social demands on south african higher education institutions have intensified in recent years. increased participation by a diverse range of students has *corresponding author. email: laura.czerniewicz@uct.ac.za 76 l. czerniewicz et al. resulted in massification of the sector within a context of limited or even reduced funding (maasen and cloete 2002). as is the case internationally, there are both more and different students entering the sector.2 the study during 2007, we conducted a survey at three quite different residential universities in three south african provinces. the 2250 student respondents completed a detailed questionnaire relating to their access to information and communication technologies (icts), including cell phones, both on and off campus, and their academic and social uses of that technology. it was based on previous research conducted in 2004 in the western cape of south africa (czerniewicz and brown 2006). the print-based questionnaire included qualitative (open-ended questions) and quantitative (ranges) data for analysis. our sample size is congruent with other studies of higher education student access to and use of icts such as pew research center and educause center for applied research (ecar) (salaway and borreson 2007; horrigan 2008). while the sample is representative of the national student population in demographic terms, we are cautious about assuming that the sample generalises beyond those students sampled and consider our findings indicative rather than conclusive. the research addressed inter-related questions on access and use. we were interested in identifying enablers and constraints both generally and in relation to actual use. we wanted to know how students use icts for social and academic purposes, and were keen to identify distinct student clusters should they exist. the three institutions from which the respondents were drawn are a mixture of traditional and comprehensive (universities merged with former ‘technikons’) institutions from rural and urban settings. institution 1 is a small (8657 students) traditional english language institution that has been disadvantaged historically. institution 2 is a medium-sized (17,500 students) historically advantaged afrikaans language institution and institution 3 is a large (24,061 students), recently merged mix of historically advantaged and disadvantaged institutions of mixed languages. the diversity from this mix provided insight into a highly differentiated student body, varied contexts, different infrastructures and historically distinct backgrounds, thus providing a rich data set. in the light of the complex historical south african higher education context, we had a particular interest in diversity and student background, and to this end constructed an index of socio-economic group (seg).3 our sample had roughly a third of students from low, average and high seg backgrounds respectively. these groups were not, however, spread evenly across the three participating institutions. students also spoke a variety of home languages with the majority (52%) being a south african language of african origin (predominantly isixhosa, sesotho and setswana) followed by afrikaans (34%) and then english (9%). we employ a ‘thick notion’ of access incorporating different kinds of access: technological, practical, social and contextual (see czerniewicz and brown 2006). in this paper we focus on the findings regarding technological access, i.e. physical access to computers and other hardware such as cell phones, and practical access which means control over when and to what extent technologies are used. this notion of access also includes concepts such as ease of access and adequacy of technology. we differentiated between social and academic uses, drawing on laurillard for a typology of academic uses (laurillard 2002). alt-j, research in learning technology 77 findings of the study in this section, we report on the findings of the study with regards technological access. we report briefly on access to computers on campus, and then focus on offcampus access. we report on cell phone access and use for academic purposes. the respondents were all residential, not distance education students. our focus is on types and conditions of off-campus access of residential students. on campus access to computers is reported very positively with 1739 of the 2195 respondents describing it as easy or very easy. this is an important finding for institutional decision-makers who may be reconsidering on-campus investment strategies. it is particularly relevant in the light of the findings that off campus access is reported to be varied and challenging. it is however significant to note that even given this generally positive response, the data suggest that students consider off-campus access essential. it is the nature and complexity of this off campus access that is of particular interest in this paper. just over a third of respondents report a high degree of access off campus (figure 1), but this is uneven across institutions. the institution with a large proportion of students from low socio-economic groups also has a large proportion of students with no off-campus access to icts. figure 1. off campus access. student responses suggest that those from low socio-economic groups are more likely to have no or very low access (e.g. a shared computer outside the home and/or not being primary users of available computers) than students from average or higher socio-economic groups (as shown in table 1). it is nonetheless significant that just over a fifth of those with no off-campus access are from high socio-economic groups, while for the low socio-economic groups the converse is the case, with just over a fifth of those with high access being from these groups. in contrast to computer access, cell phone ownership is pervasive (98.5% of students report owning one) and ownership is not socially differentiated. figure 1. students’ access to icts off campus. 78 l. czerniewicz et al. an interesting finding is that despite the cost implications, and the fact that students from a high socio-economic group have other kinds of access to the internet4, the spread of internet access via cell phone is remarkably even across socio-economic groups (figure 2). figure 2. students who use cell phones to access the internet off campus by socio-economic group. the fact that cell phones are increasingly being used for internet access in other sectors is unsurprising (an increase from 58% to 82% was reported from 2006 to 2007 in the south african corporate sector (marsland 2007)), but it is unexpected in the student sector, especially where so many students are under severe financial pressure. we asked those students who report some access to the internet off campus how they connected to the internet. the largest group of students reported using their cell phones rather than other means (figure 3).5 figure 3. type of connection off campus. of particular interest is the fact that the most frequent form of internet access for students from low socio-economic group is their cell phones (figure 4). figure 4. type of connectivity off campus. this raises questions about why students are using their own cell phones to access the internet at their personal expense, in conditions where they are generally very cash strapped. where does the value lie? in particular, why would students from low seg backgrounds be prepared to do so? given that students report accessing the internet via their cell phones, this suggests that their use is quite purposeful, and in support of their studies. this supposition is supported by responses from a question specifically about cell phone use where a table 1. off campus access by socio-economic group (seg). low seg average seg high seg none 170 (52%) 80 (25%) 79 (23%) low 172 (48%) 111 (31%) 72 (21%) avg. 244 (35%) 201 (31%) 232 (34%) high 167 (22%) 242 (34 %) 326 (44%) 136,38% 124,34% low seg average seg high seg 102,28% figure 2. students who use cell phones to access the internet off campus by socio-economic group. alt-j, research in learning technology 79 400 300 219 170 355 72 29 200 100 0 dial up broadband cellphone wireless satellite figure 3. type of connection off campus by socio-economic group (seg). 140 120 100 80 60 40 20 0 dial up low seg 43 47 124 14 22 31 17 7 6 102 136 55 59 76 99 average seg high seg broad band cell phone wireless satellite figure 4. type of connectivity off campus. 80 l. czerniewicz et al. significant group said they often used cell phones for something to do with their studies (40%) and just over a fifth (22%) said they hardly ever did. these findings suggest strategic use of available resources. we considered related data to track the relationship between access and types of use. given that cell phone use is so high, and that so many students report using them for something to do with their studies (table 2), our attention was focused on cell phone use, in general and in terms of socio-economic group. the general assumption is that cell phone use is primarily social, and indeed this is a finding in this study. yet the fact that there is a group of students undistinguished by socio-economic background who use cell phones for academic purposes is of special interest and worthy of investigation. almost a fifth of those reporting on academic use of cell phones say that this is 40–80% of their cell phone use, a sizable group who are making deliberate choices. this is striking when contrasted with reports that only two out of 14 south african universities, who participated in a brief survey in late 2007, were exploring the use of mobile technologies to support teaching and learning (brown et al. 2008). figure 5. academic use of cell phones. table 2. academic use of cell phones. hardly ever sometimes often (n) receive information from my university via my cell phone 884 (42%) 908 (43%) 325 (15%) 2117 use my cell phone for something related to my studies 462 (22%) 817 (38%) 841 (40%) 2120 400 300 200 100 0 low seg 227 343 138 32 228 275 103 15 215 348 121 12 average seg high seg no cell phone use <40% cell phone use bt 40-80% cell phone use >80% cell phone use figure 5. academic use of cell phones. alt-j, research in learning technology 81 of particular interest is the fact that 59 of the students spend almost all (80–100%) of their cell phone time on academic activities. while this is not a large percentage of the sample, the existence of such a cluster of responses is of note. when considered by seg, it is even more noteworthy that of those 59 students, half (32) come from low seg backgrounds. before discussing how one might make sense of students’ access choices, it is worth considering potential drivers of their choices. thus, we see that 63% of the respondents report that more than half of their courses use icts as part of the teaching and learning. in addition, over two-thirds of the respondents whose courses used icts said that their ict activities counted for marks – 1379 said yes in response to this question, while 613 respondents said no. these findings raise intriguing issues about the choices available to students, the enabling and constraining conditions in which they operate, the drivers for their use, the choices they make and the basis on which they make their choices. discussion in brief, we observe a situation where on campus access is generally reported favourably, and off-campus access is problematic and uneven. there is a cluster of students using their cell phones to access the internet, and using their cell phones for academic purposes, and this is true across socio-economic groups. the findings indicate that some students are especially strategic and committed with regard to access, especially to the internet. the spread of internet access via cell phone is remarkable, even when students have other options available to them. the use of cell phones for internet access by students from low segs is particularly striking in the light of their often severely constrained circumstances and the relatively high cost of cellular access both in terms of cell phones with this capacity, and actual usage costs. how does one make sense of these findings? what makes it worth it for these students, especially those from poor backgrounds to make choices to use technologies in ways which require a sacrifice on their parts? why is it that in these difficult conditions, so seriously constraining in real ways, some students are able to overcome structural challenges which would seem to determine their actions? and how has it come about that students are using their cell phones for academic purposes when this was not the original intention of these tools, which one assumes are generally used for social purposes? the data from the study reported on in this paper cannot on its own inform deep explanations. we can only offer tantalising pointers provided by the responses to the table 3. number of courses which use icts. frequency (n) % none 165 8 very few 644 29 about half 455 21 most 692 32 all 227 10 (n) 2183 82 l. czerniewicz et al. open ended questions. it will therefore, inevitably, suggest further research to provide answers to the ‘why’ questions suggested by the clusters of responses seen in the findings. the most common interpretation of the findings in this paper would be provided by the logic of the ‘digital divide’. this has often served as a way to identify and perhaps describe some consequences of social structural constraints on, and enablements of, ict usage by students in developing countries and assumes that lack of physical access to icts is based on social constructs such as race, ethnicity, age, income etc. (warschauer 2003). these are determinants over which individuals have no personal control, and where inclusion or exclusion is socially determined. thus such a binary notion leaves individuals trapped into socially deterministic modes of reaction, dependant on ‘outsiders’ to liberate them by providing them with access to icts. agency is suffocated under the weight of social structure. however, what emerges from the research reported on in this paper, resists the simple reduction of the digital divide, which tends to deprive students of agency. rather, the findings indicate that the students ‘make a plan’, a metaphor drawn from the south african afrikaans expression ‘maak ’n plan’ which connotes an ability to respond smartly to adverse circumstances. it is clear that students exhibit a more complex and nuanced way of engaging with the availability of different kinds of technologies, as well as making considered decisions about using ubiquitous technologies in unexpected ways and for purposes for which they may not have been intended. certainly the availability of ict technologies creates such structural constraints and enablements, and without them the students would not be in a position to ‘make a plan’. the social realist point, however, is that the digital divide ‘haves/have-nots’ description does not exhaust the explanatory possibilities, it merely describes the objective circumstances in which students-as-agents must live. indeed, as other researchers have noted, “though much research sheds light on the structure–action relationship, it does not illuminate users’ biographical realms and reflexivities. in consequence, some genuine motives in ict-related practices remain poorly understood” (de vaujany 2008). the literature on icts in education in general and in developing countries in particular provides numerous examples of studies of social conditions. in developing country contexts the ways that computer use is especially constrained by circumstance have received considerable attention. given the real and demanding challenges faced by students in such contexts, this is understandable. the problem is that this linear, determinist approach assumes that by creating enabling conditions, actions will automatically change. this view leads to strategies premised on the belief that if we build it they will come. a project which preceded this research found that while ict use is constrained by lack of access, it is not necessarily enabled by access (czerniewicz and brown 2009). the findings reported here suggest that even in constraining circumstances, students find ways of ‘overcoming the odds’ to meet their objectives. thus on both counts, the power of structure to determine action is challenged. what is needed is a theoretical perspective which provides a way to examine the social reality of students’ use of icts by creating ontological, methodological and theoretical space to analytically separate structure from agency. as with de vaujany, although independently from him, we suggest that margaret archer’s social realism provides just such a theoretical lens (archer 1995, 2000, 2002, 2003, 2007). alt-j, research in learning technology 83 like giddens (1996), archer rejects the old sociological dilemma of structural determinism versus voluntarism by claiming the mutual dependence of structure and agency. however, in contrast to giddens’ structuration theory which collapses structure and agency together in the action of ‘social interaction’ (giddens 1996, 101), archer’s social realism provides the basis for an ‘analytical dualism’ based on the recognition that structure and agency are sui generis different kinds (archer 1995). analytical dualism is a ‘theoretical necessity’ (and not a philosophical dualism) that enables the examination of the processes of structural elaboration (structuring and re-structuring) that occur over time (archer 1995, 157). rooted in what archer terms an ‘emergentist ontology’ (objects possess powers brought about by the conjunction of two or more elements, but these powers cannot be explained by reduction to their components), social realism recognises social reality as stratified: agency cannot be reduced to structure, nor vice versa. furthermore “structure necessarily pre-dates the action(s) which transform it … and … structural elaboration necessarily post-dates those actions which have transformed it…” (1995, 157). agents thus do not produce and reproduce society, but rather ‘reproduce or transform’ society (archer 1995, 140): but always on the basis of something that was already there, albeit the result of “the activities of the long dead” (archer 1995, 143). this combination of temporal distinction and emergence permits freedom from social determinism, individualist voluntarism and the elision of structure and agency in action. this freedom allows for the practical examination of each of the strata in sociological analysis, without denying their interdependence. as archer herself notes, social realist theory is useful for understanding the ways that the causal power of social forms is mediated through social agency (archer 2003). thus the notions of constraints and enablements themselves imply agential powers. only because people envisage particular courses of action can one speak of their constraint or enablement, and only because they may pursue the same course of action from different social contexts can one talk of their being differentially constrained or enabled (archer 2003, 4). the particular value of archer’s work is her interest in the relation between agency and structure from the perspective of the agent, or the person. in summary, she argues that the process of mediation between structure and agency must be considered as entailing three stages, which capture the interplay between objectivity and subjectivity, as follows: (1) structural and cultural properties objectively shape the situations which agents confront involuntarily, and possess generative powers of constraint and enablement in relation to; (2) agents’ own configurations of concerns, as subjectively defined in relation to the three orders of natural reality – nature, practice and society; (3) courses of action are produced through the reflexive deliberations of agents who subjectively determine their practical projects in relation to their objective circumstances. (archer 2003, 135) this provides a useful way of reading the findings of the study. in particular, it points to an examination of the ways that courses of action are produced through the reflexive deliberations of agents who subjectively determine their practical projects in relation to their objective circumstances. furthermore, for anything to exert the power of a constraint or enablement it has to stand in a relationship such that it obstructs or 84 l. czerniewicz et al. aids the achievement of some specific agential enterprise or ‘project’. “a project involves an end that is desired … and some notion … of the course of action with which to accomplish it” (archer 2003, 6). thus, in the case of this research, the conditions of access off campus for students are real constraints, just as the backgrounds of those from low segs genuinely locate them in challenging circumstances (point 1, above). however, students’ experience of these constraining factors are mediated by their own constellations of concerns (point 2), and what students actually do occurs as a result of individual reflexive processes in which their concerns are considered in relation to the objective reality of structural enablers and constraints (point 3). as mentioned, a number of papers address what archer describes as the first stage of examination, the way(s) in which structural emergent properties shape “our situations such that they have the capacity to operate as constraints and enablements” (archer 2003, 132). this the digital divide does well. the next stage of examination considers the ways in which “… agents, in virtue of their reflexivity … deliberate about the circumstances in relation to their own concerns. agential subjectivity reflects upon societal objectivity” (archer 2003, 133). this does not suggest that agential subjectivity is infallible, but it is through this examination of the objective circumstances in which we are placed, and which are ‘not of our choosing’, that we prepare for the next stage in the process: … we consult our projects which were deliberatively defined to realise our concerns; and we strategically adjust them into those practices which we conclude internally (and always fallibly) will enable us to do (and be) what we care most about in society… this final stage of mediation is indispensible because without it we can have no explanatory purchase upon what exactly agents do. deprived of such explanations … [we have] … to settle for empirical generalisations about ‘what most people do most of the time…’ (archer 2003, 133) while this seems obvious, it challenges the adequacy of simple have/have-not explanations to explain ict usage by students. because agents can and do reflexively engage in projects which arise out of their concerns, society remains an open and contingent system (archer 2003). socially deterministic explanations remain blunt instruments of explanation, because they do not “…distinguish between the existence of structural properties and the exercise of their causal powers” (archer 2003, 7, original emphasis). using archer’s language, one would ask, ‘what is the “project” in which students are engaged?’ are there any indications or pointers in the existing data? then, noting the constraints which realistically limit students’ activities in terms of limited, unavailable and inadequate off-campus access, how are the causal powers activated of those students who are not passive recipients of their circumstances in which they find themselves? access is difficult, but there are benefits, and there are some indications in this data of the specific ways in which students find icts and the internet useful. thus, they are convenient (“don’t have to pile huge volumes of books – hard copies – on my desk. all that can be retrieved from micro chips”, 1–268) and they save students from potentially embarrassing situations (“you don’t always have to confront your lecturers as you can get most answers online”, 1–62). in addition, cell phones have uses beyond the obvious, i.e. “[i] use a calculator on the phone” (1–114), and “[i] save facts on my phone (text)” (2–672). alt-j, research in learning technology 85 however, there is a larger project suggested in some of these responses that goes beyond such instrumentalist expressions of use. comments such as the following suggest a life project: “i feel computer skills are very essential as we are headed towards a strongly computerised civilization” (1–283); “icts form part of our modern lifestyle, it is the new ‘language’ everybody understands” (3–306); “it gives me the skills i need for future use” (3–197); “it is so great because it makes someone marketable” (2–56); and indeed, “i only do it because it will benefit me in the future, otherwise i hate computers” (1–468). while data of this kind can only be preliminary, the comments above suggest that computers are the means to a ‘better’ life, to success in the market place and possibly to future financial security. the final student comment above is fascinating because it casts technology in such a negative light, yet the student’s commitment to their project, their investment in their future, is a powerful driver. this exemplifies archer’s assertion that “… the activation of causal powers is contingent on agents who conceive of and pursue projects upon which they would impinge” (archer 2003, 7). archer also notes that when a project is constrained “… during its execution agents can act strategically to discover ways round or to define a second best outcome …” (2003, 6). in this study, the system is challenged by a student creatively evading the constraints imposed by the objective reality in which the student finds him/herself: “… using a blue tooth phone with airtime will help you access to ict through internet” [sic] (2–128). students also exhibit their agential power by circumventing structural constraints. as archer notes “… the influences of constraints and enablement will only be tendential because of human reflexive abilities to withstand them and circumstantially to circumvent them …” (2003, 7). in this case challenges to the system occur, for example, through a student ignoring the affordances of icts: “as daar geen ander alternatief is nie is dit goed. maar ek persoonlik verkies boeke as ’n rekenaar” [“if there is no other alternative it [ict] is good. but i personally prefer a book to a computer”] (3–695). in this case the student’s concerns result in him/her engaging in the project of studying in such a way that – at least when there is an alternative (under his/her own description) – that neither structural enablements nor constraints are activated. the emotional investment is of note here. as mutch notes (2007, 1128), archer places considerable emphasis on the emotional commitments of the person who is conceived as a strong evaluator of moral projects (mutch 2007). the focus then is on the end not the means as the prime concerns of persons who engage in and reflectively on their personal projects. the reflection takes place in the form of an internal conversation, and it is this which suggests why some persons come into evade or challenge structures. ignoring or evading the constraints on a project, however, carries “objective opportunity costs” (archer 2003, 136). for the student respondents evading the constraints of limited off-campus access these opportunity costs translate very literally into time, effort and resources: “i have to travel for 30 kilometers and pay ten rands for the taxi fare and after that i still have to pay 10 rands for 30 minutes using the internet” (1–93).6 these opportunity costs arise in part because the decisions that students (and indeed all social actors) make are made under their own descriptions and as such are fallible. however, they arise also because the decisions are made from within the situations not of our choosing in which we unavoidably find ourselves as agents 86 l. czerniewicz et al. (“collectivities sharing the same life chances”: archer 2003, 133, n. 2). thus: “when you go to the lab, you get there everyone busy with computers others are waiting and you have to decide whether you go or skip the lecture” (1–76). as archer notes, the costs of engaging in a project are differential across groups. thus for students from high segs, the cost-to-the-student of accessing the internet is lower than for students from low segs, but as she notes, “such pricing is … objective and failure to allow for it can simply derail strategic action” (2003, 136). what of that specific cluster of 32 students from low segs who use their cell phones for academic purposes? a closer look at the data indicates that this group experiences the most difficult conditions on campus (one of them comments that “the lab is infested with people”, 1–57); that 10 students report sharing the computer they have access to off campus with four or more people, and 25 describe their off-campus use as difficult/very difficult. they may be driven to use expensive cell phone time by course requirements (24 say their ict course activities count for marks); or they may be motivated by their bigger projects, their longer-term life concerns. all students have the “properties and powers to monitor their own life, to mediate structural and cultural properties of society” (archer 2000, 19). these particular students have made sacrifices and trade offs. they are from poor backgrounds, yet they choose to use their limited funds for academic purposes. archer’s concept of reflexivity provides a way of describing how those choices are made in relation to the structural conditions when she points out that the subject is conducting an endless assessment of whether the price to be paid for devotion to a project is worth paying and whether the price which was paid for subordinating and accommodating other concerns is still one with which the subject can live (archer 2002, 19). these students are ‘persons’ showing an inventive capacity to circumvent the constraints imposed by structures. it is as persons with personal and body embedded life histories that their biographical accounts can be told. the level of description to be pursued extends beyond that of actors who are related to a social group with specific interests and strategies, and beyond agents with shared cultural economic and demographic features (archer 2003). it is these biographies, these personal accounts which further research must uncover. conclusion this paper set out to report the findings of a study of student access and use informed by archer’s social realist theory, particularly pertinent in the light of the constraining and enabling factors reported on and the ways that students engage with them. while the research methodology did not allow for deep explanations, the data clarified the possibilities. particularly noteworthy are the findings which show the choices, even sacrifices, students are prepared to make and the strategies which they find in order to engage online or access the internet. the areas for further research are foregrounded both by the data and by the theoretical dimensions of archer’s work. archer provides us with a means to show how students as persons are able to find ways to evade, endorse, repudiate or contravene enablements and constraints on their projects (archer 2003, 131). the students are influenced by, but not determined by, the barriers they face. they are not influenced in the same way by social structures simply by being part of a broader group, nor do they respond in the same way. similarly, while conditions can profoundly influence the attitudes and behaviour of actors, all students do not experience an equal measure of freedom to respond. alt-j, research in learning technology 87 biographical studies will be valuable to specify the causal mechanisms which underlie different agential responses. archer (2007, 93) also offers a typology of reflexivity shaped by the interplay between the social situation and the personal projects of the agents. this is a sociological concept of reflexivity albeit one underpinned by the cognitive affordances possessed by embodied agents. indeed as, mutch (2007, 119) notes, differential access to resources may be of considerable significance to this interplay. two of these four reflexivity categories present a promising lens through which to view the findings of this study and their further investigations: the autonomous reflexives who sustain internal conversations leading directly to action, and the fractured reflexives whose internal conversation may lead to disorientation, rather than purposeful action, with life chances being determined to a large extent by their involuntary social positioning. the findings reported in this paper have only hinted at the latter category, those fractured reflexives who remain victims of circumstances, but the data points decisively to a cluster of students whose behaviour would define them as autonomous reflexives. the strategies described in this paper show how this group has been able to circumvent the constraints that would normally limit their projects, by using cell phones for academic purposes, and by making strategic choices in favour of their long-term futures. why they do so, how they are unique, how those interplays of context, social identity and personal identity play out – such explanations would require individual student stories. archer’s theory of the relationship of agency and structure would provide a rich and valuable framework to deepen our understanding of these important biographical accounts. notes 1. national plan for higher education (department of education 2001), the national research and development strategy (department of arts culture science and technology 2002), the national research and technology foresight ict report (department of science and technology 2000), and the white paper on e-education (department of education 2003). 2. students’ enrolments increased by 30% from 569,000 in 1995 to 744,489 in 2004. the ratio of black student enrolments increased from 52% in 1993 to 74% in 2004 (hemis 2003; council on higher education 2004; hemis 2004). 3. the socio-economic group index was calculated based on a cumulative score of three items: (1) occupation of primary breadwinner; (2) highest education level of primary breadwinner; (3) if respondent was the first person in their immediate family to go to university. 4. high socio-economic group students use a mix of dial-up (29%) and broadband (17%). 5. students chose one answer, their most dominant form of connection. 6. these (literal) costs reflect 2007 figures in south african rand. it is, however, important to note that costs are not restricted to material costs, but include such costs as relationships, self-concept, opportunity etc. so for this student, costs may include time with family and friends. references archer, m. 1995. realist social theory: the morphogenetic approach. cambridge, uk: cambridge university press. archer, m. 2000. being human: the problem of agency. cambridge, uk: cambridge university press. archer, m. 2002. realism and the problem of agency. journal of critical realism 5, no. 1: 11–20. 88 l. czerniewicz et al. archer, m. 2003. structure, agency and the internal conversation. cambridge, uk: cambridge university press. archer, m. 2007. making our way through the world: human reflexivity and social mobility. cambridge, uk: cambridge university press. brown, c., h. thomas, a. van der merwe, and l. van dyk. 2008. the impact of south africa’s ict infrastructure on higher education. in proceedings of the 3rd international conference of e-learning, ed. d. remenyi, 69–77. cape town, south africa: academic publishing. czerniewicz, l., and c. brown. 2006. the virtual möbius strip. research report series 1. cape town, south africa: university of cape town. http://www.cet.uct.ac.za/virtualmobius czerniewicz, l., and c. brown. 2009. a virtual wheel of fortune? enablers and constraints of icts in higher education in south africa. in bridging the knowledge divide: educational technology for development, eds. s. marshall, w. kinuthia and w. taylor. charlotte, nc: information age publishing. de vaujany, f. 2008. capturing reflexivity modes in is: a critical realist approach. information and organisation 18, no. 1: 51–72. department of arts culture science and technology. 2002. south africa’s national research and development (r and d) strategy. pretoria, south africa: government printers. department of education. 2001. the national plan on higher education. pretoria, south africa: government printers. department of education. 2003. white paper on e-education: transforming learning and teaching through ict. pretoria, south africa: government printers. department of science and technology. 2000. national research and technology foresight ict report. pretoria, south africa: government printers. giddens, a. 1996. in defence of sociology. essays, interpretations and rejoinders. cambridge: polity press. gillard, e. 2004. report on the colloquium on 10 years of democracy and higher education change. http://www.che.ac.za/documents/d000085/gillardcolloquiumreport-4february 2005.pdf horrigan, j. 2008. mobile access to data and information: pew internet and the american life project. http://www.pewinternet.org/pdfs/pip_mobile.data.access.pdf laurillard, d. 2002. rethinking university teaching. 2nd ed. london: routledge falmer. maasen, p., and n. cloete. 2002. global reform trends in higher education. in transformation in higher education. global pressures and local realities in south africa, eds. n. cloete, r. fehnel, p. maassen, t. moja, h. rerold, and t. gibbon, 13–58. cape town, south africa: juta and company. marsland, l. 2007. the tipping point for mobile is near. http://www.bizcommunity.com mutch, a. 2007. reflexivity and the institutional entrepreneur: a historical exploration. organization studies 28, no. 7: 1123–40. salaway, g., and j. borreson. 2007. the ecar study of undergraduate students and information technology, 2007. educause center for applied research. http://net.educause.edu/ir/ library/pdf/ers0706/rs/ers0706w.pdf warschauer, m. 2003. demystifying the digital divide. scientific american 289, no. 2: 42–7. online learning for design students zhengmai zhao, john cook and nick higgen de montfort university, leicester this paper describes the development of a web-based learning resource for design students at de montfort university, and presents the results of a study to investigate the effectiveness of this system. some issues regarding further improvements to the online resource are also discussed. introduction during the last five years, many changes have taken place in undergraduate design education in the uk. the main force behind these changes has been the dramatic increase in the number of students without a commensurate increase in teaching staff. typically, many design departments are now teaching at student-staff ratios of up to 50:1 compared with the 10:1 typical of design courses only a few years ago. with the more recent reductions in highereducation funding, combined with the added pressures on staff to raise their research profile and also become income generators, the problems relating to increased student numbers have been further exacerbated. the authors of this paper believe that in the current educational climate, the only way to maintain and promote the quality of the educational provision is to make teaching and learning more productive and efficient by harnessing modern technology. earlier this year, we developed an online learning resource on a lecture entitled contemporary crafts, based on a platform provided by a tltp project, stile (students' and teachers' integrated learning environment), and put it into use for an undergraduate introductory design module, contextual basis of design. the work was initiated with two objectives: 1. it was expected that an intensive short-term effort to build the online resource would bring about significant long-term benefits by reducing both the costs and staff time involved in the management, delivery and updating of the course materials. 2. it was intended that student support would be improved by providing them with flexible access to learning resources which included lecture notes, bibliography, lecturer information, and a large number of images with explanations. these visual materials 69 zhengmai zhao et al online learning for design students were regarded to be particularly important for students to better understand the subject matter, but students had previously had only a limited access to them. in order to improve the learning resource itself, and to find the most appropriate way to incorporate it into existing pedagogical practice, an exploratory study was carried out. in this paper, a description of the online resource will be provided, along with a report on results from the study, and a discussion of some issues raised from the initial experience. the online resource the world wide web provides an excellent platform for developing, organizing and disseminating learning resources due to its unique combination of features: universal readership, hypertext and multimedia capabilities (berners-lee et al, 1994). once information is made available on the web, it is accessible from any computer connected to the internet world-wide, which allows a highly flexible access to the learning resource. documents on the web can be organized in hypertext form. this means two things: (1) the provider can build a resource in a non-linear fashion, and expand the resource by integrating new material using hypertext links; and (2) the reader can escape from the sequential organization of the pages to pursue a thread of his or her own, something which offers unusually high levels of learner control. the multimedia capability of the web makes it possible to incorporate resources in a variety of forms including text, visuals, sound, and graphics. it is due to these attractive features that the educational community has been putting great efforts to make learning resources available through the web (blumberg, 1994; rebelsky, 1994). stile (ruggles et al, 1995; zhao et al, 1995), one of tltp phase i projects funded jointly by the four uk highereducation funding bodies (hefce, hefcw, shefc, and deni), is more ambitious in making educational resources available through the web. it involves four uk universities, initially bringing together the expertise of the university of leicester and loughborough university of technology, and then being extended in its second year to de montfort university and the open university. stile aims to establish a powerful web-based learning environment with three main features: • the provision of a large, extensible resource-base with special reference to images covering a broad range of disciplines; • the means to locate and access relevant information efficiently; and • the means to facilitate a more flexible approach to learning, including group work and distance and collaborative learning. we decided to develop an online resource within the stile framework due to its simple model, consistent interface and range of tools which makes both the access to and the provision of information relatively easy. additional important reasons for adopting stile were that the stile software is freely available to all uk higher-educational institutions, and that the project provided the necessary technical and financial support. the conceptual model of a stile resource-base can be thought of as a network of pieces of information. each piece of information is either a resource (something to be found and used) or indexing information (something to assist in finding the desired resource). the indexing information is also referred to as topics. a topic represents a particular subject area. subject 70 volume 4 number i areas can be anything general, specific, material or abstract. in terms of this particular study, topics include contemporary crafts, craftperson, slides of contemporary crafts, and glass. the resources are actual teaching and learning materials. in this study, images and bibliographies are considered as primary resources. figure 1 illustrates the stile abstract model, where the shaded circles represent topics which form a topic net, the boxes represent resources which hang on the topic net, and the dashed arrows represent links to external relevant information. the resource-base holds a hierarchical structure, with more general topics being placed at a higher level in the hierarchy. figure i: the stile abstract model it is rather natural that the stile model is realized on the web because both stele and the web consist of networks of linked information. each stile topic is represented as an html (hypertext markup language) document, each resource is represented as a file the web can handle, and links of information are represented using hypertext links. navigation of the stele resource-base can be performed by following the hypertext links. this method of locating information is simple, but can be slow since it proceeds only one step at a time through the network. more effective navigation of the topic net is possible using the stile navigation tools. one such tool allows navigation at the topic-title level rather than the actual topic document level. the major advantage of this functionality is that it makes parallel navigation possible, i.e., more than one link can be followed at a time, thus speeding up the process of locating information. another useful navigation tool is the topic-title search engine. given a word or phrase, the search engine will gather all topics in the current resource-base which include the given word or phrase in their titles, and display them in the order of relevance. the process of building the stile resource-base featured in this study began with design of the structure. this work included the identification of the indexing information (topics), the 71 zhengmai zhao et al online learning for design students actual information the students would require (resources), and the structure of the topic network. then topic information, in plain-text format, was marked up using an extremely simple but function-limited markup language called trivial metalanguage, and converted into the topic network using the stile batch conversion tool. following that, the stile browserbased editor was used to link resources (mainly images in gif format) into the topic net, and to make necessary modifications to the resultant resource-base. the combination of batch conversion tool and interactive editor enabled a standard stile resource-base to be built without the need to learn how to write html documents. evaluation objectives the main purpose of the evaluation was to see how the online resource on the lecture entitled contemporary crafts worked in a real educational situation so as to improve both the system itself and the way it is integrated into pedagogical practice. due to the ill-defined goal of the evaluation, an exploratory evaluation strategy was employed. more specifically, an attempt was made to examine the effectiveness of the resource-base in terms of the product of using the resource-base, the process of interacting with the system, and the attitude towards the innovation. participants twelve out of 168 undergraduate students who were taking the introductory design course contextual basis of design were requested to participate in the study. they were chosen because they were assigned the seminar presentation topic concerned with the subject of contemporary crafts (all other students were allotted seminar presentations against lectures not on the web). it must be noted that the allocation of seminar topics was conducted randomly. each chosen student used the resource-base, completed questionnaires, and attended interviews. procedure shortly before the lecture on contemporary crafts, the chosen students were introduced to the online resource as a study reference in addition to the lecture. they were each provided with flexible access to the resource through about 20 pcs with high bandwidth connection to the internet in the university library. since the online material was intended as a flexible learning resource, the students were supposed to access it whenever they wanted. obviously because of this, it was impractical to observe in person how the students exactly interacted-with the system. as an alternative, an attempt to elicit such information was made using a combination of selfreporting and an automated user-access log. at the very beginning of the study, a questionnaire developed by draper et al (1994) was used to measure the students' previous experience with computers. the students were also given another questionnaire that they were required to hand in by the end of the evaluation. this questionnaire, adapted from shneiderman's quis tool (shneiderman, 1992), covered a range of issues including use of the system, overall reactions to the system, system capabilities, process of learning to use the system, and efficiency of using the system. every student who took the course was requested to give a seminar presentation on an allotted topic about a month after the related lecture was given. the assessment data of all 72 alt-j volume 4 number i students' presentations was available for the subsequent comparison. after all the chosen students made their presentations, structured interviews were conducted. results and discussion according to the students' self-report, they used the online resource twice on average, and the average length of each session was 25 minutes. these figures were consistent with the data automatically captured in the server log file. the students reported that they used the resource for reviewing the lecture, preparing for seminar presentation, and seeking new ideas. they also reported that curiosity of this new learning technology served as an important initial driving force for them to start using the system. analysis of questionnaires and interview protocols showed that the overall reaction to the system was generally positive. the system was regarded by the students as being useful, enjoyable to use, and easy to use once they had mastered the basic skills needed to use the system. most of the students found it easy to find the information they wanted, and to navigate around the system. however, their responses towards how easy it was to learn to operate the system were rather diverse. a further investigation of the students' computer experiences suggests a link exists between their skills of using computers and their response to this issue. it would appear that those students who felt more confident with computers found it easier to learn to operate the system. the students listed a range of benefits from the system. first, the system served as a backup to the lecture, which gave the students an opportunity to review the lecture in more detail. more interestingly, some of the students said that having known the lecture was online, they could concentrate better on the interaction with the tutor during the lecture itself without being worried about taking detailed notes, and that it had the potential to significantly change the way in which they would respond to lectures in the future. second, the system provided greater flexibility of access to the material, with respect to time, place, pace, and sequencing. the students liked the possibility of accessing the material in their own time and at a place preferable to them. they did not have much choice of place, since none of them had access to the web anywhere except in the university library. however, they saw this as a great potential benefit of the system. they appreciated the control provided by the system in terms of the pace and sequencing when browsing through the material, especially in the investigation of the visual material. access to primary materials, images of crafts for instance, is vital for design students to obtain artistic feeling, critical thinking, and creative inspiration. in this particular case, the lecturer had a large collection of slides and only one hour to show them. with the system, the students were now happier to have more flexible access to them. as one of students commented during the interview: 'it's just good to get down to a little bit and study it and then go on to the next when you're ready, rather than have a quick look and just walk out'. in addition, some of the students felt that they became more engaged in the learning process when interacting with the system than when sitting in lectures. finally, the students, especially those with little computer experience, felt more confident in learning with technology-based systems after using the system. the overwhelming dissatisfaction expressed by the students was with the slow response rate of the system. the length of time to display an image, with an average size of 50kbytes, sometimes became intolerable due largely to the low capacity of the university network 73 zhengmai zhao et al online learning for design students service. in addition, a richer resource-base was strongly demanded for future use. the system did not include links to any materials by others because the subject-author could not find any relevant external resources. a number of improvements were also suggested by the students including the provision of more explicit online help, a full-text search engine, a printing-out facility, and a multiple-windows capability so that comparisons between images could be made more easily. every student's seminar presentation was monitored by using a common mark sheet, and a group discussion about its positive and negative aspects was carried out immediately after each presentation. the presentations by the students chosen to participate in our study seemed better than those of the remaining students. ninety-one per cent of the chosen students obtained a grade of a or b, while only 82% of the remaining students reached such a grade. in contrast to 2 1 % of the remaining students receiving an a grade, the percentage of the chosen students receiving a top grade was as high as 36%. however, the results of the mann-whitney u test show that there is not a significant difference in the grades of seminar presentations between the chosen students and the remaining students (na= 11, nb= 144, u = 592, z = 1.39<1.96). as mentioned early in this section, an attempt was made to obtain the information in relation to the students' interaction with the system by using a combination of self-report and automated user-access log. it was found that the students could explain the basic strategy with which they dealt with the system, e.g., systematically covering all contents or selectively picking up points which looked interesting to them. however, they were not able to recall the detail of their behaviour. for example, they had difficulty detailing where they got lost even though they reported disorientation in their navigation, which could be very useful information in terms of improving the structure of the material. moreover, it was not realized until quite late that the information captured in the web server access-log file was not adequate to determine the students' actual navigational tracks because of web client-side cacheing. simply speaking, a web browser, which is the interface to the system, can store in its local cache a certain number of web pages previously requested by its users. every time a request for a web page is made by a user, the browser checks to see if the page is in its cache. if not, the browser sends a request over the network and then stores the received page in the cache. otherwise, it picks up the page from the cache without requesting the server to provide that page. in such a case, the server access-log file will not include this user request. obviously, the cacheing function is able to save time and cut down traffic over the network. however, it posed a problem for this study in its attempt to find how the material on a server is accessed by a user. the evaluation effort described so far concentrated on only one, but probably the most important one, of our objectives for using the online resource, that is, to facilitate student learning. the other longer-term objectives were to reduce the cost and staff time spent on the management of course materials. the related questions concerned how easy it was for the tutor to put original resources into the system and to update them afterwards. compared to the evaluation of student use of the system, the assessment of ease of use for the tutor to deliver resources through the stile platform was more straightforward. experience from this study showed that the maintenance of the resource-base was relatively easy through the use of the stile browser-based editor. however, it was still not easy, especially for a non-unix user to build a stile resource-base even though html programming skills were not necessary. as described earlier, to build the resource-base, a bulk conversion of our topic network was carried out from a marked-up file using the stile batch conversion tool. for a non-unix 74 alt-j volume 4 number 1 user, extra skills such as file transfer over the network and a basic working knowledge of unix are needed to fulfil this task. this is because the current stele conversion tool works only under the unix environment. once the topic network was established, the resources (mainly images of crafts) were mounted onto the topic-net using the stele browser-based editor. this task was easy but tedious because there were a quite large number of images which had to be linked into the topic-net one by one. there are two suggestions to ease this difficulty. first, to improve stele's authoring facility to allow platform-independent mass conversion enabling the system to convert not only pure topic networks but also to include references to resources from bulk data. second, for potential stele resource providers who are computer illiterate, to improve their computing skills simply because it is unrealistic to expect to gain benefit from a new technology without a basic technical understanding. conclusions and further work the online resource-base has proved to be successful in demonstrating its great potential in meeting our proposed objectives, that is, improvement of the support for design teaching and learning. the students revealed in the study a great appreciation of the high degree of learner control provided by the system, and expressed a strong demand for more information online concerning the subject area. from the tutor's perspective, the maintenance of the existing resource-base has been made relatively easy by the use of the stele browser-based editor although the current conversion tool still needs real improvement. nevertheless, this experience has also raised a range of issues that need to be thoroughly addressed before the innovation can be fully applied to existing teaching and learning activities. some of these are highlighted below, and might be of general interest to those in a similar situation. the university's it infrastructure particularly for online access to learning resources needs to be urgently improved, which includes updating the network capacity and the establishment of specialized computer labs for online resource delivery. this has an implication for the way in which the university allocates its resources. a shift from teaching to learning, and a consequent shift in resources from teaching resources to learning resources, might be essential. new strategies and skills need to be developed for both students and tutors. for students, consulting information from an online system is different from consulting information from conventional printed materials. likewise, for tutors, delivering information in hypertext form requires different abilities from those they have already developed. teaching styles need to change in order to make effective use of such innovations. in this case, it might be possible to rely heavily on the online system for information delivery so that more staff time can be devoted to actual student contact in helping students develop the practical skills of designing and making things. guidance on the use of copyright materials should be made available to the academic staff who want to provide educational resources online. unless the materials are owned by the provider, copyright clearance is necessary for their use or even their adaptation. the lack of knowledge of dealing with copyright appears to be a restraining force in making educational resources available online. further work on the use of the online learning resource will concentrate on three areas. the first area of work will be to include more information in the resource-base. the contents of the 75 zhengmai zhao et al online learning for design students resource-base will gradually expand to include the whole module of contextual basis of design; links to relevant external resources will be added; materials such as previous students' seminar presentations and transcriptions of group discussions may be integrated; and the use of digital video, animation, and 3d objects will also be explored. the second area will involve encouraging students to become not only information consumers but also information providers, and to make use of communication facilities supplied by the stile platform. finally, the possibility of replacing facts-based lectures with a well-built online resource system will be further investigated. acknowledgements the authors wish to express their gratitude to all members of stile project team and those students who participated in the study described here. references berners-lee, t., cailliau, r., luotonen, a. nielsen, f. and secret, a. (1994), 'the worldwide web', communications of the acm, 37 (8), 76-82. blumberg, r. b. (1994), 'an electronic science/math/history resource for the www', the second international www conference: mosaic and web, chicago, illinois, usa, october 17-20, 1994, http://www.ncsa.uiuc.edu/sdg/it94/proceedings/educ.html. draper, s. w., brown, m. i., edgerton, e., henderson, f. p., mcateer, e., smith, e. d. and watt, h. d. (1994), 'observing and measuring the performance of educational technology', technical report, tilt project, university of glasgow. rebelsky, s. a. (1994), 'a web of resources for introductory computer science', the second international www conference: mosaic and web, chicago, illinois, usa, october 17-20, 1994, http://www.ncsa.uiuc.edu/sdg/it94/proceedings/educ.html. ruggles, c., underwood, j. and walker, d. (1995), 'stile: a hypermedia approach to the provision of a flexible, interdisciplinary resource base for teaching and learning', innovations in education and training international, 32 (3), 209-19. shneiderman, b. (1992), designing the user interface: strategies for effective humancomputer interaction (2nd edition), reading ma, addison-wesley. zhao, z. and patel, d. (1995), 'an online resource-based learning environment' in jonassen, d. and mccalla, g. (eds), proceedings of icce 95 international conference on computers in education, singapore, 5-8 december 1995, 315-23. 76 improving instructional effectiveness with computer-mediated communication som naidu, john barrett and peter olsen university of southern queensland, australia this study explores the use of asynchronous computer-mediated communication (cmc) in the delivery of instructional content, and points up the interaction among learners, as well as between learners and instructors. the instructional content in the project described was available to learners online as microsoft word documents, with email being used for communicating within the student group. many students, as well as some of the instructors, felt uncomfortable with the flexibility and openness that a cmc environment allowed. however, once familiar with this process of instruction and interaction, learners were able to work consistently at their own pace, and understand that instructors are interested in every individual learner's opinion and in the collective views of the group. it was evident that a cmc-based instructional delivery system, when carefully planned, has the potential to facilitate that outcome, and to improve instructional effectiveness. introduction a critical question that often faces educational technologists is how to deliver excellence in teaching and subject-matter content to learners. a corollary to this question is how and what instructional technologies can be brought to address this search for particular contexts? while there is a wide range of instructional delivery technologies we can choose from, our choice must be carefully considered. the number of factors to be considered are too many to list here, but must include a cognizance of the nature of the content or skill that comprises the subject matter of instruction, the learners, the time and their place of study, and the costs of the delivery mode, both for the learners and the institution. this paper reports our experience of the first phase of a three-phased integration of computermediated communication (cmc) aimed at improving instructional effectiveness. interest in the application of some form of cmc in the enhancement of teaching-learning environments is currently widespread. this has led to the emergence of a growing body of literature on various aspects of cmc-based instructional delivery systems (for example, mason and kaye, 1989; harasim, 1993; mason, 1993; wells, 1993). while there exists in this literature a great deal of information on the hardware and software requirements for 63 som naidu et al improving instructional effectiveness wirh computer-mediated communication such delivery systems, and numerous reports of applications in a wide variety of contexts, relatively little attention is focused on approaches to the integration of cmc in teachinglearning environments. the project reported in this paper is about the design of instructionally effective cmcbased teaching-learning environments. the emphasis here is not on the hardware or the software requirements but on the organization and presentation of subject matter, the designed activities, and human factors which are often, in our experience, at the heart of the success or failure of such projects. a phased-integration of computer-mediated communications technology is described and also recommended to overcome an all-toopervasive mindset about teaching that is a carry-over from the conventional face-to-face classroom instructional situation. teaching-learning orientation a considerable amount of research exists in favour of teaching-learning designs that engenders collaboration and interaction among the peer group (ide et al, 1981). there is evidence, for example, that cooperative learning benefits learning for all except the most concrete, repetitive tasks with effect sizes as high as .8ocr (johnson and johnson, 1974). allowing learners to exercise adaptive control over their learning process is also reported as having positive impacts on learning (hannafin and colamaio, 1988). cmc-based teaching and learning systems are characteristically designed to empower learners by allowing them a greater degree of adaptive control over their learning environment which is not feasible in conventional systems. this facility refers to the flexibility that learners may have in exercising as much control and autonomy over their learning, as and when necessary. the hypothesized advantages of this electronic teaching-learning environment over the conventional system were several. for the learners it was intended to encourage them to move from: • a defined learning space to an open, richer global resource base; • an instructor controlled learning environment (that of a conventional classroom) to a collaborative and co-operative learning context; and • being uncritical recipients of content to dynamic and participant explorers of the knowledge base. the orientation of this project is derived from increasing evidence in favour of cmcbased collaborative learning and instructional environments. its basic philosophy of flexible access and an instructor-learner negotiated teaching and learning process is at the heart of open learning systems, and increasingly now of conventional face-to-face teaching-learning environments which are also aiming for more flexible and adaptive formats (see the concept illustrated in figure 1). figure 1 is a graphic representation of the teaching-learning environment that was designed and implemented as part of this study. notice that the focus in the design is on integrating cmc in the teaching-learning environment as one of the resources available to the teacher and the learner. students enter this instructional environment with variable 64 an-j volume 3 number 2 figure i: integrating cmc in the teochingleaming process learner cmc adtpdva control of leaning •uuctures/capability levels of prior knowledge in the use of cmc as a teaching and learning delivery medium, and also with variable levels of motivation to use it. upon entry, they are confronted with an instructional environment that comprises the instructor, subject-matter content and cmc-based learning activities, conventional group lecture sessions, tutorial assistance, and other forms of individualized and group support. the goal of this instructional scenario is to allow the learner a level of flexibility that will enable adaptive control of the instructional environment. for the learner, this means seeking out and receiving as much instructional direction as is necessary, or assuming as much flexibility and freedom as one desires. such autonomy should also allow the learners to use the instructional resources in a manner that will suit not only their place and pace of study, but also their learning styles and approaches to learning. the intended outcomes of this instructional environment are several. these include foremost, for the learner, changed cognitive structures. this means a dramatically new and improved way of relating to content, storing it in memory and retrieving it. for the instructor, this arrangement implies new and improved ways of delivering content, and also improved ways of interacting with learners. the instructional delivery system also facilitates the arrangement of subject-matter content in new and interesting dimensions. content can be organized in hypertext formats and include hypermedia to facilitate a richer and resourceful instructional and learning environment. all of these outcomes in the end lead to the creation of a new and improved instructional system that is more flexible, richer, and a realistic teaching and learning environment. context and subjects the project reported in this paper was carried out in the context of a second-year unit in the bachelor of education program in the faculty of education at the university of southern queensland (australia). the unit titled teaching-learning studies focused on the classroom teacher and related issues like planning for teaching, instructional strategies, and classroom management. the project ran for a full 13 weeks in semester 2, 1994. subjects in the study were all the students enrolled in the unit. 65 som naidu et al improving instructional effectiveness wirh computer-mediated communication design in order to ensure systematic integration of cmc in the teaching and learning context, and to cope with the human factors pervasive in the situation, a three-phased model for implementing cmc was devised (see figure 2). 3 2 1 a document source integrated mm hypertext documents microsoft word 3 2 1 b communications desktop video + audiographics computer conference (groupware) microsoft mail figure 2:7?ie three-phased cmc implementation there are two components to each phase. the first component (a) concerns the organization and presentation of the subject matter. at the first level, all subject-matter content can be created, stored and retrieved as word-processed files. at the second level, the word-processed files are presented as hypertext documents to facilitate such uses as complete document search, a linked hypertext-type lesson structure, zooming-in and magnifying components, copy and paste to other documents, and insert 'post-it' notes and 'bookmarks', etc. the third level extends the hypertext document to include interactive multimedia and activate other programs such as a statistical package. the second component of the system, the communications dimension (b), is similarly organized into three layers with the first level operating on electronic mail. the second level includes text-based conferencing systems and/or groupware, while the third level includes audiographics and/or desktop video conferencing. this system provides for a 'stepwise' progression allowing advancement up the layer on one dimension while operating at a lower level in the other dimension (that is, hypertext documents and email a2 bl), or integrated multimedia documents and conferencing a3 b2). the building of these steps is dependent on the technical infrastructure, skill level of students and the teachers, and more importantly, on the nature of subject-matter content and the desired interactions/activities between and among all participants. figure 3 represents the implementation model reported in this paper. the current project was designed to represent the first phase of this model. core content in the model is held on a file server at level al as microsoft word documents. level a2 uses cd-rom as a storage device while level a3 utilizes greater mass storage devices such as interleaf and worldview. communication at level 1 was via microsoft mail. at level 2, computer conferencing is envisaged, and at level 3 we are looking at audiographic communication. alt-) volume 3 number 2 figure 3: usq implementation model 3 2 1 a document source i-con author* interleaf* woridview interleaf* worldview microsoft word 3 2 1 b communications smart 2000 computer conference microsoft mail network configuration the phase 1 network configurations included 486 33 eisa machines running windows 3.0 with microsoft word 2.0 and microsoft mail. the core lecture content, including the learning activities for each week's study, was input as word documents. all supporting reading materials were scanned and also included as word documents. students were required to log on, read each week's material, and then respond to the required learning activities via microsoft mail. students in the project were registered on the local-area network individually, as well as under an alias which was the unit number. a user could therefore send a message to another directly by addressing a message to that individual, and also to the whole group by addressing messages to the alias. learning activities learning activities were generated by the instructor on a weekly basis. these activities were carefully designed to focus on the subject-matter content that was being covered during that week. in the first week, for example, students examined the topic of 'direct instruction' as an instructional strategy. students were exposed to a discussion on the topic by the instructor. this was accessible as a word document. after reading and synthesizing this material, students were required to respond to the learning activities that were generated by the instructor. some activities required further reading, others required some field work, and others required summarizing or synthesizing. on-line response patterns responses to these instructor-generated learning activities were transmitted via email to the student-group alias, which meant that it went out to all the students registered under that alias, including the instructor. each student's response was therefore received by all the others in the group. individuals could comment on all the responses that were made, and especially those that raised controversial or interesting issues. the instructor was in a position to observe the transactions that were going on via email and make his own contributions to the discussion. he would make his own contributions and also provide feedback to the student commentaries. students could also send messages to individuals in the group, which would then comprise private communications as neither the 67 som naidu et al improving instructional effectiveness wirfi computer-mediated communication instructor nor anyone else in the group could have access to that message, unless such a message was copied or forwarded to a third or fourth individual in the group. data sources the objective of data collection was focused on the use and utility of the cmc-based delivery environment to the stakeholders (that is, instructors and learners). hence mostly ethnographic data-gathering techniques were employed. these included in-depth interviewing of learners during and after the process, and also analysis of the texts from the online interactions. questions asked, focused on various aspects of the delivery system and their interface with it. these included training in the use of email; the benefits and disadvantages of the delivery mode; the changed role of the instructor; the technologyhuman interface in online communication; impacts on learning outcomes; and problems with reading text online and other concerns relating to online communication. all interview protocols were tape-recorded and transcribed for further analysis. analysis of data the interviews were conducted using the following specific headings: • training for the project; • instructional delivery system; • changed role of the instructor; • technology-human interface; • learning experiences. observations preparatory training learners reported being generally satisfied with the training that was provided in the use of email and online communication at the start of the project. there was a view that the training could have been more effective if it had been focused on specific needs of learners with variable levels of expertise with the email software and online communication. most felt that a list (in print form) of the most common features of microsoft mail would have helped a lot. some of the learners found certain components of the training redundant, and which could therefore have been eliminated with better design, delivery and coordination of the training programme. the instructional delivery system participants felt that, while this new mode of delivery was 'different', it offered them much needed flexibility in the way of their place, pace and time of study. the general disposition was that this was a more time-efficient way of making content and instructor expertise more accessible to learners. changed role of the instructor the use of cmc in instructional delivery meant a significant change in the role of the human teacher. while cmc allowed the instructor a greater degree of interaction with 68 fia-j vo/ume 3 number 2 individual students in the class, it also increased the instructor's workload. even though learners realized that they were in contact with their instructor online 24 hours a day, the lack of face-to-face contact with the instructor in a cmc-based instructional environment was missed by the students. there was an overall preference for retaining some face-toface contact in this predominantly electronic teaching-learning environment. technology-human interface for the computer-literate, a cmc-based teaching-learning environment posed little problem. for the computer-illiterate however, this delivery mode was a source of much anxiety. the experience from this project revealed that several factors, if not carefully managed, could lead to the failure of such systems. these include issues relating to moderation of discussions, careful management of dominant personalities, and use of acceptable online conversation protocols. the asynchronous nature of the medium meant that a response from the intended recipients of a comment was not available or 'visible' immediately as is the case in face-to-face contexts. a few of the participants found it rather frustrating having to wait for 24 or more hours to get some response on their comments and queries. some students found reading and composing on-screen somewhat difficult. this was not unexpected. initially, students preferred to read offline, compose their message on a word processor, then send the message as an attachment to an email message. this was a very time-consuming and tedious exercise that detracted from one of the obvious advantages of cmc systems, which is the spontaneity with which messages can be received and responded to. with some practice, however, students were able to read comfortably onscreen and also compose messages online. learning experiences a majority of the students felt that one of the significant outcomes of the cmc-based delivery system was the shift in the degree of control learners could exercise in their learning. in the conventional face-to-face context, their instructors made a lot of the decisions about how and when things were supposed to happen. in the cmc-based system, the learners took over a greater degree of responsibility of their learning patterns, determining for themselves how and when things would be done. in some cases there was a commensurate increase in the amount of time students spent on studying. students enjoyed the opportunity to interact with their peers in an on-going manner without being constrained by the hours of a lecture and tutorial session. students reported getting more 'involved' in their study. there were some negatives as well. one of the problems that seemed to concern students about online communication was that of being misunderstood and not able to defend or explain one's comments before the others were already in an attack mode. in a face-toface mode, if one felt that a comment was being misunderstood or misrepresented, there was the opportunity to correct it right there and then. in a cmc-based delivery mode, however, once a message had been posted, it was out in the open for its recipients to make whatever out of it, and it was only after some time, usually after a day or so, that one could correct oneself or clarify the misunderstanding of the others. an advantage of the possibility of this occurring was that learners were forced to think through their ideas and comments a lot more clearly before broadcasting these online. some students found that in so doing they were improving their reading, writing and thinking skills. 69 som naidu et ol improving instructional effectiveness wirti computer-mediated communication discussion a cmc-based delivery system is fundamentally different from a face-to-face teachinglearning environment. it carries with it, among other things, a uniquely different student and instructor mindset. like any other instructional delivery system, a cmc-based teaching-learning environment has its strengths and weaknesses for learning and teaching outcomes. its success, in these terms, is dependent on a number of design considerations that are unique to it. these considerations and their implications on teaching and learning in a cmc-based instructional delivery system are discussed in the remainder of this paper. implications for system management an integral part of the success of a cmc-based instructional system is the function of the system manager. this is a person who is responsible for network upkeep and maintenance. teaching and learning using cmc requires maintenance and support of email communications, word processing and transferring of files and material from wordprocessed documents to mail and vice versa. many things do and can go wrong in this process, especially when dealing with students and instructors who lack much experience with operating in this mode. it must be understood by all parties concerned that there is a protocol for online communication. these have been widely documented in the literature on cmc-based instructional systems (see harasim, 1993). stakeholders must be given initial training in these and their use with continuous support and reinforcement throughout the process. it must be understood that there will be a slow start, and that as students and instructors become familiar with the delivery medium, the pace will pick up. this underscores the need for that front-end training and on-going support to be carefully planned and executed. implications for the instructor this refers to the perceptions that instructors hold about their roles as teachers. usually these perceptions are a result of their own training as teachers, or requirements of their institution, and years of work experience in conventional or other educational systems. the principal tasks of teachers in conventional face-to-face instructional systems, for instance, are to prepare lectures, deliver them, hold small group tutorial sessions, and mark written assignments. a cmc-based teaching-learning environment prescribes a different set of tasks for the instructor. these include, among other things, the development of the core content in advance of the actual instruction, presenting these online, and communicating with students on a continual basis in an asynchronous mode. the instructor takes on a facilitator's role, reading and responding to student's work and clarifying any misconceptions as they arise. the instructor is no longer tied down to lecture or tutorial times. all of this happens asynchronously and in a 'fluid' teaching-learning context. for some instructors this may mean additional work which may lead to the fear of being eternally 'on the job'. the truth is that a cmc-based instructional environment, if carefully planned and executed, has the potential to save time and effort for the instructor time which previously had been taken up by routine and repetitive tasks such as grading paper-based work and answering generic types of queries. 70 alt-j volume 3 number 2 many instructors also find this to be a rather 'loose' arrangement in which they relinquish their control over how students study. this is not always acceptable to many instructors who believe in retaining greater control of their classes. in the academic community, some of these perceptions are deeply ingrained and very resistant to change. much of this resistance, especially towards new delivery technologies, can also be attributed to a lack of confidence on the part of instructors in their abilities, and a fear of having to learn a new skill. a cmc-based instructional delivery system poses this kind of threat to most instructors, especially those who have little or no proficiency in working with students online. our experience tells us that these are serious considerations that must be carefully planned for. instructors have to be introduced to the radical shift in the mindset or orientation towards their teaching function in a cmc-based environment. unless this shift in their mindset has taken place, success is far from certain. implications for the learner students, also, hold particular conceptions and a mindset about their own roles as learners and that of their instructors. like instructors, they need to understand that a cmc-based teaching-learning environment shifts the bulk of the responsibility for learning onto themselves. this requires recognition of that responsibility on their part, and the assumption of an active role in the learning process. students must understand that the instructor is no longer going to drive the learning for them, that they must learn to drive it for themselves. importance of front-end training while an increasing number of the current generation of students are computerand internet-literate, there are still many among them who have no or only negligible experience with computers and electronic networks. experience derived from this project suggests that a carefully planned front-end training is imperative for both students and instructors involved in any cmc-based teaching-learning arrangement. such a training program must be based on a thorough assessment of the needs of the participants. it is likely that these needs will be variable which would then necessitate individualized or small-group attention for particular or all aspects of the training. yet while front-end training is a must, it is also certain that training up-front in the use of cmc will be insufficient. training and continuing assistance will be necessary throughout the duration of such a project. therefore, help with online instruction and the equipment will need to be available on a continual basis, either online or otherwise. outcomes of cmc-based instruction a cmc-based instructional environment is fundamentally different from the conventional classroom-based system in very many ways. the difference between the two modes of instruction is most explicit in the manner in which content is delivered to the learner, the communication patterns between tutors and students and among the students themselves, the manner in which students access additional learning resources, and the manner in which assessment of learning is carried out. most instructors who have been exposed to a cmc-based instructional system argue, although not initially, that the mode has had a significant and lasting impact on their overall approach to teaching and 71 som naidu et at improving instructional effectiveness wirh computer-mediated communication learning. our experience suggests that this kind of an impact is manifested in several ways. some of the more visible ones are discussed in the following sections of the paper. expanded resource base currently, the integration of cmc in instructional systems may include any one or more of a growing number of applications. these include access to electronic mail, bulletin boards, databases, networked cd-roms, network newsgroups, electronic discussion lists and computer conferences, file transfer protocol (ftp) facilities, and global search tools such as gopher, telnet, knowboots, netfind, finger, and archie. this list tends to be expanding all the time. with the help of an average personal computer and a communication connection, these applications have the potential of bringing to the learners an expanded learning-resource base which would be otherwise beyond their reach. the learning resources that these applications can bring to the learners include, inter alia, libraries and other sources of information at remote sites, access to international experts and research sites, other students, colleagues and peers, and access to relevant discussions that might be taking place in other parts of the world. our experience and those of others (ferris and roberts, 1994) show that access to an expanded learning resource base that is carefully selected can mean that learners are not only encountering more useful material than that specified in the curriculum, they are also covering the specified content and more material more rapidly than is possible in conventional systems. learners shift from being passive receptacles to being active participants in the search for knowledge. moreover, they learn how to acquire and use knowledge, and instructor-roles shift from dispensers of information to producers of environments which allow learners to learn as much as is possible on given topics. co-operative learning environment conventionally, co-operative learning refers to instructional and learning environments which are characterized by increased interactions among individual learners or among small groups of learners. this form of learning has its limitations no doubt, but its positive contributions to learning and instructional outcomes are reported to far outweigh its limitations or difficulties (johnson and johnson, 1984). a computer-mediated learning and teaching environment which depends upon interaction between databases, instructors, tutors and students, and also among students, is by definition a co-operative one. computer-mediated co-operative learning and teaching has been shown to enhance learning outcomes in many different ways, including improvement in the quantity and quality of the learning experience (see collis, in press). computer-supported co-operative instruction is also reported to have benefited the instructors. ferris and roberts (1994) report that as a result of the experience in the use of cmc in their instructional environments, instructors began to take a larger and more systems-oriented perspective of their contributions in the school, meeting each week in teams to plan an integrated effort. problems that arose were dealt with by the teams rather than by the one instructor in the conventional system. almost subconsciously, the roles of instructors in their classrooms got transformed into a more collaborative one. authentic learning environment a computer-mediated learning and instructional environment is capable of reaching out to resources in remote locations, including people and sites, for relevant information that — alt-) volume 3 number 2 would otherwise be impossible to access by most students in the time that is usually available to them. as such, a cmc-based instructional system is capable of facilitating 'authentic' learning by enabling access to environments separated in time and place from one's home base. realities can be created, in cyberspace in a manner of speaking, by electronic access to people in situ in different countries, cultures, systems and with access to libraries, databases and also discussions raging on the internet. authentic assessment environment similarly, a cmc-based instructional environment is potentially capable of an assessment system that is authentic and dynamic in nature. assessment can be authentic in the sense that it is situated and contextualized rather than contrived. it can be dynamic in that assessment is continuous and pegged to what students are doing in terms of their interactions and activities online. a variety of instructional strategies are open to instructors in cmc-based instructional systems such as setting up discussions, debates, tasks, project work etc. to which students are required to contribute. these contributions are asynchronous, and can be assessed by the instructor on an individual basis, including the provision of individualized as well as group feedback. flexible learning flexibility is a characteristic feature of a cmc-based learning and instructional environment. the integration of some form of cmc in teaching and/or learning means that the communication channels between the instructor and the students, and among the students, are not only open all the time, but asynchronous. with communication channels open, teachers and students have access to one another at any time of the day rather than waiting for the lecture, tutorial or consultation times to raise a query or share some interesting thought as these are occurring in the process of one's study. messages, questions and contributions by students or teachers can be left on the network asynchronously, meaning at different times and from different locations. open and asynchronous communications between the instructor and the students are the hallmarks of a flexible learning arrangement. a cmc-based learning and instructional system is open also in terms of the many instructional strategies it can accommodate., these include individual searches of online databases, journals, libraries, and discussion groups. one-to-one communication may take the form of correspondence study, learning contracts, apprenticeships, and internships between the instructor-student, the student-expert, and also between students. one-to-many online communication may include lectures, symposiums, and panel discussions. many-to-many techniques can include debates, simulations/games, role plays, case studies, discussions, project-based work, brainstorming, delphi and nominal group techniques, forums, and cognitive networking/mapping. this list of online activities is by no means exhaustive. it is an indication of what is possible in a cmc-based instructional environment with a little bit of imagination and creativity, mostly as part of instructional design and development. learning skills development a cmc-based learning environment places the responsibility for learning more than ever in the hands of the learner. in so doing, the learner is greatly empowered, and also placed in a position of greater control of not only the amount but the quality of his or her 73 som naidu et al improving instructional effectiveness wirti computer-mediated communication learning. this autonomy allows learners (especially the enterprising ones) to explore, experiment, take risks and venture beyond that which is necessary. a natural outcome of this kind of initiative on the part of learners is enhanced learning skills relating to the search for and acquisition of knowledge. in conventional systems of instruction, much of this kind of autonomy is not possible due to a greater degree of instructor-control of learning, and also because of the lack of resources in the learners' immediate learning environment. with the help of cmc, that learning environment is now much larger and more accessible, allowing for a richer and larger resource base for the learners as well as the instructors. concluding comment this project was implemented to ascertain, inter alia, factors that needed addressing when building a cmc-based learning and instructional delivery system. it was evident, from our experience, that in the initial stages students as well as instructors require an intensive training program to familiarize them with basic operations such as logging on and the use of passwords, moving between applications, such as from word to mail, and how to save and store material for future reference. many students and instructors are still locked into the time and place concepts associated with lectures and tutorials. the issues of flexibility and openness must be carefully introduced to all stakeholders in such a project. once familiar with this process of instruction and interaction, learners are able to work consistently at their own pace, and realize that instructors and tutors are interested in every individual learner's opinion and also in the collective views of the group. it is evident that a cmc-based instructional delivery system has the potential to facilitate that outcome as well as improve instructional effectiveness. acknowledgement the authors are grateful to the committee for advancement of university teaching (caut) for financial support in the conduct of this study. in addition, we thank john elms and ken woolford (faculty of education, university of southern queensland) for their support. references collis, b. (in press), 'cooperative learning and cscw: research perspectives for internetworked educational environments' in lewis, r. (ed), lessons from learning, amsterdam, north holland. ferris, a. and roberts, n. (1994), 'teachers as technology leaders: five case studies', educational technology review, 3, 11-18. hannafin, m.j. and colamaio, m. (1988), 'the effects of variation of lesson control and practice on learning from interactive video'. educational communication and technology journal, 35 (4), 203-12. harasim, l.m. (1993), global networks: computers and international communication, cambridge ma, mit press. _ an-] volume 3 number 2 ide, j.k., parkeson, j.a., haertel, d.d. and walberg, h.j. (1981), 'peer group influence on educational outcomes: a quantitative synthesis', journal of educational psychology, 73 (4), 472-84. johnson, d.w. and johnson, r.t. (1984), 'cooperative small-group learning', curriculum report, 14, 1-6. johnson, d.w. and johnson, r.t. (1974), 'instructional goal structure: cooperative, competitive or individualistic', review of educational research, 44 (2), 153-66. mason, r. (1993), computer conferencing. the last word, bc canada, beach holme publishers. mason, r. and kaye, a. (1989) (eds), mindweave: communication, computers and distance education, oxford, pergamon press. wells, r (1993), computer-mediated communication for distance education: an international review of design, teaching, and institutional issues, research monograph no. 6, pennsylvania state university college of education pa, acsde. 75 editorials alt-c 97, university of wolverhampton in this special conference edition of alt-j, i would like to say that we have moved forward. and in many respects we have. the technology is progressing at a pace which is increasingly difficult to comprehend. the software 'revolution' continues unabated to produce increasingly complex integrated applications. these daunting developments increase the pressure on academics, their institution and inevitably the student. to this we can add the problem of decreasing finance to universities and educational establishments in general. we, as practitioners in learning technology, must utilize the best features of communications and applications to reduce the pressures on ourselves and our colleagues. are we succeeding? what is success? how is it monitored, measured and evaluated? to use the convoluted technology and its associated infrastructure, and to exploit the opportunities afforded to us by the internet and advanced communications, is no mean feat. the papers presented at this year's conference, under the banner virtual campus real learning, presented some apposite developments in the field of learning. this is the crux of any development. it is very easy to blame the technology, or worse still see the technology as the goal of the research development process. we need to be mindful of what we as learning technologists are endeavouring to achieve. is it interaction (something as yet not really achieved on the internet)? is it administration? should we as academics be researching into this area? is it simply to occupy? as academics can we afford to teach students on a one-to-one basis? what am i suggesting is: yes, we have come a long way but there is a long way to go still. in fact, arguably as we consider that we have reached the end, the chequered flag has not only moved on but it has probably moved tangentially to where we wanted or felt that it could go. i quote from an article by richard hooper published in 1975: the computer [...] displays to the learner another frame of text depending on the particular response the learner has typed in to the questions. each frame of text is 3 editorial selected by the computer to match the performance of the individual under tuition. the computer is said to be adaptive to the individual learner. we are still striving for this, and now with 'virtuality'. the title of the conference suggested an unreal learning world. i can cope with this as long as the learning and its associated experiences are real. the papers at the conference addressed many of the issues of learning in a world of communications technology. the conference presented many opportunities for collaborative networking of research and development ideas. for this particular issue we received 52 submissions for publication. yet again, the referees found it difficult to extract the few accepted for this edition. we have increased the number accepted last year to 14 this year, and hope that the diversity of approach, interest and development reflects what is currently happening within our respective institutions. i look forward to alt-98 and the connected world. john o'donoghue, chair, alt-c97 reference hooper, r., and toye, i. (eds), computer-assisted learning in the united kingdom, london: ncet, 1975. real learning: the obstacle virtual campus real learning. nothing wrong with the first phrase: we are probably moving (if falteringly) towards one. nothing wrong with the ideal behind the second phrase. who would argue that false learning is better than real learning, that superficial understanding (or none at all), resulting in being able to repeat things parrot-fashion in an examination or a project dissertation, is better than deep understanding? it is like asking people if they prefer war to peace. ask, in a student feedback questionnaire, whether respondents prefer to have at university (i) a real learning experience, one which inculcates critical reasoning and judgement, or (ii) an experience of learning lecture notes by heart ready for regurgitation, and see what the answer is. i know, because i have done it (more or less). nearly all students say they want the real thing, but they also want above all high marks. are the two incompatible? they should not be. there is a vanishingly vast literature on learning styles, environments for active learning, experiential learning, discovery-based learning, student-centred learning... i could go on. yet what i find among real students (and i know that neither i nor my institution are alone: very far from it) is something different. here is a revealing little example. a colleague and i teach a course jointly. recently, i gave some lectures on electronic commerce, in which i tried to get away from the hyperbole surrounding it as far as the internet is concerned, in order to offer a more detached vision of its future. my colleague had lectured a week or two before on a related topic, and had taken a far more optimistic line than mine. at the end of my lecture, a student asked who would be marking the examination question, if there was one, on electronic commerce and the internet. when i asked why, he replied that, of course, he wanted to tailor his answer to fit the marker's view. i can perfectly understand his attitude, as also his rather sceptical reaction to my 4 alt-j volume 6 number i subsequent (automatic-pilot)) comment that reasoned, structured arguments would always be acceptable whatever the marker's own opinion of the matter, and that this was true of all examinations. but he was, plainly, overwhelmingly more concerned with his mark in the examination than with the arguments themselves. any tactic which might lead him to a higher mark was one to be adopted: in the real world, he was clearly less interested in real learning than in any kind of learning which would lead to a high assessment of his performance; and, from his standpoint of future employment, etc., understandably so. could one reasonably argue that this student's learning might have had nothing to do with his examination strategy? that he might well have really learned but that he had decided to put this real learning behind him at examination time? i doubt it. it seems to me that his attitude betrayed a typical young-student learning experience, focused on gaining high marks and to hell with self-edification which at best might be a by-product of study. the way out of this problem is, of course, to assess students on real learning. there is less of a literature on that topic. we need not only, in the words of diana laurillard, to rethink university teaching, but it seems to me, more so, since re-thinking teaching would inevitably follow from it to re-think university assessment. perhaps the virtual campus will really make a difference. gabriel jacobs 5 editorial quality and targets i am sure that few people know how alt started, and i suppose that even fewer care. but how our association began relates to a basic issue which i wish to address in this editorial. alt began with a meeting of three people. graham chesters and i went to see jonathan darby to ask if he wanted to be involved in launching a new journal in the area of cal. the idea of an association grew out of our discussions (as i remember, jonathan darby thought of the name). the three of us could see that, beyond the cti centres, learning technology was creating a loose network of interested parties within higher education, and we decided that this network would benefit from a formal structure which an association could provide. shortly afterwards, at our first meeting proper of the newly formed executive committee, it was evident that all involved were generally optimistic about the use of computer-based learning technology within higher education. on the other hand, we were also very conscious of the resistance to learning technology already being encountered in many higher-education institutions, partly for reasons related to technophobia and fear of job losses, more so because of some poorly designed cal products that did not fit learners' requirements and/or expectations (not to mention the expectations of university managers). all the papers and many of the reviews in this issue of alt-j are concerned with this matter to some extent, and two papers directly so. king and honeybone ask if cal can be 'the 5th cavalry charging to the rescue of the beleaguered educational troops'; not, they say, unless student needs are understood and the very purpose of university education clearly defined and accepted. hopkins and colombi are convinced that at the centre of the debate lie issues 'concerned with the ability of cal to offer and deliver a teaching and learning experience which is as effective as more established modes'. 2 alt-j volume 4 number 2 most higher-education institutions nowadays are either (at least in certain departments) committed to or looking seriously at the use of cal, for one or more of four reasons. the first is that staff-student ratios have risen. the second is that modularization has brought increased pressure. the third is that it finally seems that the technology can (just about) meet user-expectations. the fourth is that not being able to boast the wide use of cal carries with it the risk of appearing old-hat. but the fact that orsini-jones and jones, again in this issue of alt-j, can talk of 'the recent backlash against information technology which is becoming increasingly fashionable' in itself indicates that some of the fundamental problems of promoting cal, of which that original alt executive committee was only too aware, have still not been resolved. has alt, in four years, achieved anything beyond what we hope are excellent information services, useful workshops, exciting conferences, valuable briefing sessions, and so forth? if these are truly achievements, they are not to be belittled. but on the question itself, i claim not the 5th cavalry but the 5th amendment. what i will say is that i believe in the effectiveness of good cal, but that we continue to climb a steep rock-face as its advocates, and that we remain in mortal peril of crashing to the ground unless we face up squarely to the doubts about cal deeply embedded in many university teachers (even some of those involved in it). it is not that alt has failed to play its part in convincing higher-education institutions to go down the technological route on the contrary. it is that not delivering quality and well-targeted goods, and thereby fuelling doubts and alienating decision-makers, may have disastrous consequences. gabriel jacobs 3 learning morphological phenomena of modern greek an exploratory approach y. kotsanis, *a. kokkinos dimitrios, *a.g. manousopoulou and *g. papakonstantinou *national technical university of athens this paper presents a computational model for the description of concatenative morphological phenomena of modern greek (such as inflection, derivation and compounding) to allow learners, trainers and developers to explore linguistic processes through their own constructions in an interactive open-ended multimedia environment. the proposed model introduces a new language metaphor, the 'puzzle-metaphor' (similar to the existing 'turtle-metaphor' for concepts from mathematics and physics), based on a visualized unification-like mechanism for pattern matching. the computational implementation of the model can be used for creating environments for learning through design and learning by teaching. introduction educational technology is influenced by and closely related to the fields of generative epistemology, artificial intelligence, and the learning sciences. relevant research literature refers to the term constructionism (papert, 1993) and exploratory learning (disessa et al, 1995). constructionism and exploratory learning are a synthesis of the constructivist theory of piaget and the opportunities offered by technology to education on thinking concretely, on learning while constructing intelligible entities, and on interacting with multimedia objects, rather than the direct acquisition of knowledge and facts. these views are based on the approach that learners can take substantial control of their own learning in an appropriately designed physical and cultural environment (harel, 1991). in parallel, most of the studies of the vygotskian framework focus on the role of language in the learning procedure, considering conceptual thought to be impossible outside an articulated verbal thinking. moreover, the specific use of words is considered to be the most relevant cause for childhood and adolescent differentiation (vygotsky, 1962). these approaches offer important pedagogical ideas for the creation of powerful computational principles, such us procedures and interactive objects (pea, 1992). they 29 v. kotsanis et a\ learning morphological phenomena of modem greek an exploratory approach can be used in a flexible, reusable and modular way, and are usually referred to as the lisp-derived logo-like environments (hoyles et al, 1992; georgiadis et al, 1993; disessa et al, 1995). moreover, they concretize the argument about how different languages (spoken and programming) can influence cultures that grow up around them (papert, 1980). independent from educational technology, in the area of language technology naturallanguage processing systems have attempted to encode linguistic information and to endow computers with human language capability. a unification-based grammar formalism has been widely used in a variety of these systems as a pattern-matching technique for different purposes. definite-clause grammars from the logic programming field, generalized phrase-structure grammars and typed-feature structures from the computational linguistics field are examples of formalisms based on unification (shieber, 1986; carpenter, 1992). specific morphological processing systems are used in a fairly broad range of technological applications such as word processing, speech and language applications, and machine translation. these systems usually include a morphological processor that uses one or both of the following operations: • an analyser, to recognize the combination of morphemes that form a word and/or the morphosyntactic features associated to the word; • a synthesizer, to generate a well-formed word from its morphemes and/or the morphosyntactic features. many systems and applications which take this direction have been presented (sproat, 1992), and specifically for the greek language (ralli, 1987; kotsanis 1991; markopoulos, 1994). the 'two-level morphology' theory, with its kimmo implementation, has been by far the most successful and best-known general model of computational morphology (koskenniemi, 1983; antworth 1990). however, few references exist in the development of learning tools for modelling educational linguistic processes in an exploratory way (sharpies, 1986; ohmaye 1992). the objective of our approach is to provide learners with a powerful and natural learning environment to study morphology and words more generally. learners are provided with tools to develop educational environments for language. this approach presents a computational model for the description of concatenative morphological phenomena of modern greek (such as inflection, derivation, compounding it can be extended to other languages as well) by allowing learners, trainers and developers to explore linguistic processes through their own constructions in an open-ended interactive multimedia environment. the proposed model introduces a new language metaphor, the 'puzzle metaphor' (similar to the existing 'turtle-metaphor' for concepts from mathematics and physics), based on a visualized unification-like mechanism for pattern matching. the computational implementation of the model can be used for creating environments for learning through design and learning by teaching, based on the experience that learners prefer to choose the role of the developer rather than the role of the user. the puzzle metaphor metaphors are potentially important components of educational learning environments, 30 alt-j volume 4 number 3 for providing dynamic models of systems that learners can explore and study. the turtlemetaphor, one of the most well-known, which uses the kinematic image of a curve as a moving point, is based on deeply rooted intuitions concerning body motion. it can be used to generate rich mathematical and science environments, offering a visually attractive and comprehensible introduction to programming (papert, 1980; hoyles et al, 1992). this mathematical-oriented metaphor is widely and internationally used for a broad range of ages and activities (kynigos, 1992; georgiadis et al, 1993; blaho et al, 1994; disessa et al, 1995). however, there are almost no references for a similar language metaphor (tinsley et al, 1995; vosniadou et a/, 1995), with the exception of the phrasebooks and boxes, a general oriented language microworld for linguistic explorations (sharpies, 1986). by focusing our interest on morphological phenomena, the most adapted interpretation is based on the approach that words are built up from smaller meaningful units, namely morphemes. morphological analysis (recognition) is concerned with retrieving the structure of morphemes that form a word. morphological generation is concerned with producing an appropriate word-form from some set of morphosyntactic and semantic features (sproat, 1992). important for performing these tasks is that inflected, derived or compound words are built up via the successive application of word-formation rules, which are similar to syntactic rules for sentence formation. for example, we can define the following simplified rule for the combination of an infinitive verb with the nominalizing er morpheme (bear, 1986) such as readreader. noun —» verb nominal <noun number> = singular <verb form> = infinitive <nominal string> = 'er' general word-formation rules can be expressed as a context-free grammar of the form: word -» stem inflectional_ending stem -» stem derivational_suffix our linear puzzle-metaphor concept introduces a slightly modified view of a wordformation mechanism. the puzzle metaphor, underlying the context-free grammar, does not express the relation between the category of each morpheme (e.g. stem, ending) but the position (from now on called puzzle-type) that a morpheme has inside the word (w): r: complete morphemes that do not concatenate, s: beginning morphemes that are concatenated only from the right, d: intermediate morphemes that are concatenated from both right and left, i: ending morphemes that are concatenated only from the left. thus our context-free phrase structure of the word-grammar model (for word w recognition and generation) consists of the following three (and only these three) wordformation rules: 31 v. kotsanis et al learning morphological phenomena of modem greek an exploratory approach w > s i s > s d the above production rules generate only the following listed sequence of morphemes: r, s i, s d i, s d d i, s d d d i , . . . (further down),for example, the greek words karw (under, down), (graphic) and •nepiypa^ncq (descriptive) are analysed as: s i: [napa]s s d e : s d d e : any morpheme can belong to more than one puzzle-type. table 1 contains examples of various morphemes of the greek language. r s symbol i position in the word xyz xyz-xyz-xyz j r ? p»ww.-.l.w?g} n. category functional word non-inflected non-inflected prefix stem prefix (particle) augmentation prefix stem derivat. suffix synthetic vowel augmentation non-inflected inflect, ending examples of morphemes 0, n, (the article), eueic; (we) nepi (about, for) (metro) nepi-ypa<i)-rt (description) ypa$-ik-ri (graphic/al) a-ypaip-oc; (unwritten) e-ypa<p-a (i was writing) auto-nepi-ypam1! (autodescription) nepi-ypa^-rt (description) ypaqmk-otnt-a (picturesqueness) acnv-o-ypaip-l-a (x-ray) nepi-e-ypaip-a (i was describing) n a p c f x a r o (further down) ypcxjhfi (writing) table i: examples of puzzle-types and morphemes ; the building blocks of the metaphor the above mentioned context-free grammar describes which morphemes may combine with which. to determine the appropriate order of morphemes within words, we have to define for each morpheme a set of attribute-value pairs (which are user-defined and constitute a feature structure) containing several morphosyntactic pieces of information, for example word category (part of speech), gender and voice. these pieces of information can also have semantic characteristics, for example 'consists *of. the determination of the attribute-value pairs does not restrict the underlying morpheme but 32 aa-j volume 4 number 3 rather extends it to concatenate with the 'valid' next and/or previous morphemes (pentheroudakis et al, 1993). this means that each of the three puzzle-types (s, d, or i) can be used as a previous or next puzzle-type (sketched with dotted-lines in the figures that follow). in figure 1 two morphemes have been determined. the first morpheme, the stem /3aa (of the word fsdo-t) basis) is of puzzle-type s and has an associated feature structure. it can be concatenated with the associated feature structure of the inflectional ending tj, which is a morpheme of puzzle-type l gaot cat; noun **~\ inflcat: n_eiq gender: fern accent: 2 type: s t e m ^ / t y p e : infl_end cat: {noun, gender: fern accent: {1.2.3} type: stem nom umber: sing type:infl_end figure i: definition of basic building blocks based on the above analysis, we have determined nine combinations of puzzle-types that constitute the basic building blocks of the model. no 1 2 3 4 5 6 7 8 9 symbol r 0si 0sd sdi sdd ddi ddd d i 0 s i 0 prev-morphemencxt ix id >> > 77t? $n examples jiovo (only) 0 06o-n (basis) 0 bao-ik-n. (basic/al) bachk-fi (basic/al) sckmk-otnt-a (basicity) uovo-bao-u«-a (monobasic) uovo-bao-tk-otnt-a (monobasicity) bao-ik-n 0 (basic/al) frbo-eiq 0 (bases) table 2: examples of basic building blocks 33 v. kotsanis et al learning morphological phenomena of modem greek: an exploratory approach any morpheme can belong to one or more of the above building blocks. figure 2 demonstrates how complex building blocks can be created. the morpheme j8aa, the stem of the word fsio-q basis, or ^aa-ik-q basic, can be an s and d puzzle-type at the same time. the morpheme «? is the plural inflectional ending of the word pdoeis bases, and can be attached to an s or d puzzle-type. jjaa " s , cat: noun \ gender: fem jaccent 2 ^/ type: stem c s,d l*"*vinflcat: n_eiq : / t y p e : infl_end i r •-,. cat: noun > \ gender: fem < < accent: 2 • / t y p e : {stem ;•*• d«r_mjfrix} s chs ^ ^^nmflcat: il$«; \case: ncm inumbenpiur ^/type:jnnj3nd i j figure 2: definition of complex building blocks the unification process and the associated feature structures to establish a valid concatenation of two morphemes (beyond the rules of the contextfree grammar), a pattern-matching mechanism is used. this unification mechanism is based on the notion of combining the information from two feature structures to get a feature structure with all of the information of both (shieber, 1986). table 3 contains examples of how the unification mechanism actually works (note that a value of an attribute-value pair, which consists of a feature structure, can also be a feature structure). feature structure a cat: noun cat: noun gender: fem cat: noun cat: noun cat:np agreement: [num: sing] feature structure b gender: fem cat: noun cat: adj cat: {noun, adj} agreement: (pers: 3rd] agreement: [pers: 3rd] table 3: unification examples (curly braces {} denote disjunction^ unification a u b cat: noun gender: fem cat: noun gender: fem cat: noun cat:np agreement: [pers: 3rd] agreement: [num: sing pers: 3rd] • 34 ml-) volume 4 number 3 the unification mechanism is performed only while parsing similar morpheme categories. for example, in a puzzle-type x=s or d or i, unification will be applied only between the two morphemes where the first belongs to puzzle-type x and the previous or next of the second belongs also to puzzle-type x. the morpheme of figure 3, fiaa, can be concatenated with the inflectional ending 17 if the latter is an 'inflectional ending' (value of the 'type' attribute) and has an attribute 'inflectional category' with the value 'ij_ei?'. the derived inherits the values: noun, feminine, singular, nominative, and is stressed at the penult (value 2 of the attribute 'accent'). the morpheme j3a<r cannot be concatenated with a morpheme which has different values for the attributes 'type' and 'inflectional category'. figure 3: the unification process of two building blocks attribute-value pairs, depending on their function, can be classified in two groups with the following characteristics: inheritance or concatenation of values. as soon as the unification mechanism is started, the resulting allowed morpheme pairs contain a set of attribute-value pairs. in the wordformation process, the features of the word are inherited from the head of a morphological constituent. if, for example, the unification mechanism generates: the word will inherit the attribute 'cat' (part of speech) with value 'adjective', since the right-hand head rule dominates and is frequently cited in morphology (selkirk, 1982; sproat 1992). moreover, there are a number of cases which appear to have left-headed morphological constructions which have to be defined to the inheritance process (for example, the non-inflected prefix, ava, when attached to the word fido-q for the formation of the word avd^aat) ascension, shifts the accent from the second to the third syllable from the ending of the word). if the unification generates (in the above example): lpaaltemmderivational^ix™inflectional ending 35 v. kotsanis et al learning morphological phenomena of modem greek: an exploratory approach then the attribute 'type' of the word (with the values 'stem', 'derivational_suffix' and 'inflectional_ending' for the three morphemes respectively) is derived by concatenating the values; in this case the value would be 'stem & derivational_suffix & inflectional_ending'. values: every value of an attribute-value pair can be either a string (text), a sound, a bitmap (picture), an animation, a video or a user-defined procedure which will start a computation or will return a value. the handling of multimedia objects and user-defined procedures as values in a feature structure constitutes a powerful environment which covers a broad range of requirements (for example, handling of unusual morphological phenomena or audio-visual wordformation from audio-visual morphemes). implementation the suggested model has been implemented using the following environments: i. prototyping using prolog to test the expressive power of the proposed formalism and to determine if the models generate all possible words of the given grammar (completeness), and if the words are 'valid' in the underlying vocabulary (soundness) (gazdar 1989), a pilot system has been developed using prolog which represents morphemes as follows: s t r ( morphemename, listoflettersinmorpheme ) . morf( morphemename, typesofmorpheme, attributesofmorpheme ) . next ( morphemename, typesofnextmorpheme, attributesofnextmorpheme ) . prev( morphemename, typesofpreviousmorpheme, attributesofpreviousmorpheme) . the t y p e s o f . . . fields are lists consisting of symbols from [r, s, d, i], while a t t r i b u t e s o f . . . fields are lists of the form [[attr-l,[value-l-l,valuel-2, . . .]], [attr-2,[value-2-l,value2-2,...]],...]. multiple n e x t () and p r e v () facts are allowed for one morpheme. the distinction between the name of the morpheme and the actual list of letters that constitute it allows us to encode morphemes with similar behaviour under a single name. an example database would be: str( bas, [%j3','a'# 'a'] .). morf( bas, [s,d], [ [type,[stem]], [cat,[noun]], [gen,[fem]], [accent,[2]] ] ). next( bas, [d], [ [type,[der_suffix]] ] ). next( bas, [i], [ [type,[infl_end]], [inflcat,[c_ae6]] ] ). str( ik, [%i','k'] ). morf( ik, [d], [ [type,[der_suffix]], [cat,[adj]], [accent,[1]] ] ). prev( ik, [s,d], [ [type,[stem]], [cat,[noun,adj]] ] ). next( ik, [d], [ [type,[der_suffix]], [cat,[noun]] ] ). next( ik, [i], [ [type, [inf l_end] ], [ inf lcat, [os_t/_c] ] ] ). stx( i, pi?'] ). morf ( i , [ i ] , [ [type, [infl_end] ] , [ i n f l c a t , [oy_i?_oij_£is] ] , [numb, [sg] ] , [case, [ n o m , a c e , v o c ] ] ] ) . 36 alt-) volume 4 number 3 prev(i,[s,d],[[type,[stem,der_suffix]],[cat,[noun,adj]],[gen,[fern]], [accent,[1,2,3]]]. the above database and its associated grammar accept (generate) the following two, and only two, valid greek words: (80017 [bas,i] [bas,ik,i] the current version of the prototype contains a lexicon of 200 different morphemes which generate about 5,000 valid words (nouns and verbs). at this point we should mention that right recursion of the grammar terminates normally, without the need to define a value for the maximum allowed depth which is directly proportional to the maximum number of concatenated morphemes. 2. educational logo-like multimedia environment after receiving the outcome of the previous mentioned prototype in prolog, we started implementing the proposed model by developing a microworld using a logo-like interpreter (based on comenius logo: blaho et al, 1994). this interpreter is equipped with new powerful primitive procedures and data types. moreover, it works under the familiar windows environment and takes full advantage of today's multimedia capabilities. r sb i d "•-, cat: {noun, adj} t / type: stem ik cat: adj accent: 1 inflcat: oq_n_o suffix i. it v»?4{/.?. ,fi*. "(* '> j l 3 i figure 4: working interactively with the 'makemorph' primitive using the proposed microworld, the user will be able to work in a friendly windows environment for handling the data (morpheme objects), and to recognize and generate valid morphemes and/or words. further, the microworld allows or helps the user to group his or her data. the implemented microworld introduces the following six new primitives: 37 y. kotsanis et a\ learning morphological phenomena of modem greek: an exploratory approach makemorph: create basic or extended morphemes getmorph: search for and modify an existing morpheme getword: recognize and generate a valid concatenation of morphemes and build up a valid word setpair: define and/or modify attribute-value pairs and their characteristics loadbase: use an existing or user-defined knowledge base savebase: save a knowledge base all primitives have two modes of operation: graphical and command-line. the graphical mode of operation is intended for the visual interaction with the various morpheme objects. the command-line mode is intended to provide users with primitives to implement their own or extend existing lexical application. the system also contains special-purpose attribute-value pairs for executing video, sound, image, music, and userdefined procedures. in its present form, the system is being tested by a small group of educators and students. it will be enhanced with the data of an educational multimedia dictionary developed at doukas school in athens (kotsanis et al, 1996) and will be tested in classroom activities of the same school. conclusions the suggested model and its educational value constitute a learning-by-doing environment which simplifies functions and lexical representations without neglecting the expressive power of linguistic processes. furthermore, it can introduce the idea of grammars and parsing to secondary-level students. the open-ended design of the environment gives the opportunity easily to develop audio-visual lexical applications for students by using different teaching architectures (schank, 1994) and enhancing the kernels of authoring environments. in this unified environment (for use and development), learners, trainers and developers have access to and can reuse the same resources and data. for example, students will be able to design a spelling checker or express orthographic rules on their own. this environment can be enhanced in many ways. first, it can be modified to allow more than one next or previous puzzle-type (s, d, or i) while defining morphemes. this results in the ability to describe long-distance dependencies where the existence of one morpheme is allowed by another morpheme which is not adjacent to it (for example, joy, *joy-able, en-joy-able: sproat, 1992). it can also be extended to allow the description of phonological phenomena (antworth, 1990). further, all lexical components of the model can be changed to graphical objects or even objects in space, so that it can be used by children at the primary educational level. finally, in order to enhance the existing graphical representation of the puzzle-types beyond their two-dimensional movement, they can be moved and rotated in three-dimensions and their shapes changes (userdefined or selected from the existing library). 38 alt-) volume 4 number 3 acknowledgements the authors would like to thank s. piperidis, a. giabris, y. maistros, c. kynigos, p. giakoumis, n. filippaki, a. triantafyllou, g. markopoulos, g. drivas and c. doukas for their contribution and support in the above study. references antworth, e. (1990), pc-kimmo: a two-level processor for morphological analysis, occasional publications in academic computing 16, summer institute of linguistics, dallas ta. bear, j. (1986), 'a morphological recognizer with syntactic and phonological rules', coling-86 (association for computational linguistics). blaho, a., kalas, i. and matusova, m. (1994), 'environment for environments: new metaphor for logo', in exploring a new partnership: children, teachers and technology, philadelphia pa: ifip transaction a-58. carpenter, b. (1992), the logic of typed feature structures: inheritance, (in) equations and extensionability, cambridge: cup. disessa a., hoyles, c. and noss r. (1995), computers and exploratory learning, nato asi series f:146. georgiadis, p., gyftodimos, g., kotsanis, y. and kynigos, c. (eds.) (1993), logo-like learning environments: reflection and prospects, proceedings of the fourth european logo conference, university of athens and doukas school. gazdar, g. and mellish, c. (1989), natural language processing in prolog, reading ma: addison-wesley. harel, i. (1991), children designers, cambridge ma: mit media laboratory. hoyles, c. and noss r. (eds.) (1992), learning mathematics and logo, cambridge ma: mit press. koskenniemi, k. (1983), two-level morphology: a general computational model for word-form recognition and production. ph.d. thesis, university of helsinki. kotsanis.y. and maistros, y. (1991), 'describing morphological phenomena of modern greek using a unification grammar formalism', information systems, 16 (6). kotsanis, y., raptis, k. et al (1996), user needs analysis for educational multimedia dictionary, tech. report no 2.2.2, doukas school, athens (research programme dialogos, co-ordinator ilsp, supported by the grant epet ii, 715, gsrt). kynigos, c. (1992), 'the turtle metaphor as a tool for children's geometry', in hoyles c. and noss r. (eds.), learning mathematics and logo, cambridge ma: mit press. markopoulos, g. (1995), 'two-level morphology in modern greek', proceedings of the second international conference of greek linguistics (september 1995), salzburg. ohmaye, e. (1992), simulation-based language learning: an architecture and a multimedia authoring tool, ph.d. thesis, northwestern university. 39 y. kotsanis et ol learning morphological phenomena of modern greek an exploratory approach papert, s. (1980), mind-storms, children, computers and powerful ideas, new york: basic books. papert, s. (1993), the children's machine: rethinking school in the age of the computer, new york: basic books. pea, r. d. (1992), 'distributed multimedia learning environments: why and how', interactive learning environments. pentheroudakis, j. and vanderwende l. (1993), 'automatically identifying morphological relations in machine-readable dictionaries', proceedings of the ninth annual conference of the uw centre for the new oed and text research. ralli, a. and galiotou, e. (1987), 'a morphological processor for modern greek', proceedings of the third european chapter of the association for computational linguistics (april 1987). selkirk, e. (1982), the syntax of words, cambridge ma: mit press. schank, r. c. (1994), 'active learning through multimedia', multimedia ieee, spring. sharpies m. (1985), cognition, computers and creative writing, chichester: ellis horwood. shieber, s. (1986), an introduction to unification-based approaches to grammar, chicago: university of chicago press. sproat, r. (1992), morphology and computation, cambridge ma: mit press. tinsley, j. d. and van weert t. j. (eds.) (1995), liberating the learner. proceedings of the sixth ifip world conference on computers in education (wcce '95). vosniadou, s., de corte, e. and mandl, h. (eds.) (1995), technology-based learning environments, nato asi series f:137. vygotsky, l. s. (1962), thought and language, cambridge ma: mit press. 40 alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 143–153 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020143–11 © 2007 association for learning technology doi: 10.1080/09687760701470973 students and instant messaging: a survey of current use and demands for higher education gijs de bakker*, peter sloep and wim jochems fontys university of applied sciences, the netherlands taylor and franciscalt_a_246976.sgm10.1080/09687760701470973alt-j research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis152000000june 2007gijsde bakkerg.debakker@fontys.nl instant messaging (im) is the term used to describe the technology through which ‘users can set up a list of partners who will be able to receive notes that pop up on their screens the moment one of them writes and hits the send button’. while early use could be described as mainly for fun, im today is a serious communication medium. remarkably, it seems that educational institutions have been doing very little with it, while several studies indicate that it could indeed be a valuable tool in education. as a first step towards a better understanding of the educational use of im, we want to gain insights in how students currently use im and what opportunities they themselves see for the medium. to that end we conducted a survey among students of the fontys university of applied sciences in the netherlands. a large majority of the participating students indicated using im for their studies. also, when asked about their demands for a possible educational implementation, the majority were positive. introduction instant messaging (im) is the term used to describe the technology through which ‘users can set up a list of partners who will be able to receive notes that pop up on their screens the moment one of them writes and hits the send button’ (castelluccio, 1999, p. 34). im contrasts with synchronous chat in this respect, since this is usually organised through publicly accessible chat rooms. dating back to the moment the first computer networks were available in the 1970s, messaging experienced significant growth in the late 1990s via the rapid growth of aol’s instant messenger in the united states and the msn messenger system in europe, both of which provided free consumer applications for im. while early use of the tool could be characterised as being for fun, im today is more usually considered to be a serious communication * corresponding author. eindhoven school of education: fontys university of applied sciences, technical university of eindhoven, traverse 3.42, den dolech 2, 5612 az eindhoven, the netherlands. email: g.debakker@fontys.nl 144 g. de bakker et al. medium. especially among young people, it has become one of the primary communication means (pew internet, 2004, 2005; qrius, 2005), with its own culture and language. commercial organisations are aware of this development as well, but in our experience educational institutions have been very reluctant about adopting this medium. for example, many schools consider instant messaging as a mere fun tool for children in their spare time, and some even ban it from any school activity. also, the research field of educational technology shows only a small number of studies on im, as compared with other tools such as web-logging and communities. several studies indicate that im can indeed be a valuable tool in current education, which has changed its approach from being top-down organised to a more bottom-up approach based on social–constructivist theories and competence-based education. in this new educational model, im could for example be used by students to practice their language proficiency skills with distant fellow-students (coniam & wong, 2004). other possible implementations are synchronous tutor guidance through im or students using im to reflect on fellow-students’ work. social developments, and especially the current ict adoption wave among youth (prensky, 2001), should influence the choices one makes regarding the implementation of the medium in or outside our classrooms. the few studies conducted on im up until today, however, use relatively small sample sizes and have an explorative nature (for example, nicholson, 2002; coniam & wong, 2004). as a first step towards a better understanding of the possibilities for educational use of im, we want to gain insights in how students currently use im and what opportunities they see for the medium themselves. these insights are valuable, in order to be able to develop software that answers to user needs. to that end we conducted a survey among students of the fontys university of applied sciences in the netherlands. this article reports the results of this study. instant messaging defined for those unfamiliar with instant messaging, castellucio’s definition mentioned in the introduction might not be sufficient. grinter and palen use the following, more elaborate description: im systems support internet-based synchronous text chat, with point-to-point communication between users on the same system. a window is dedicated to the conversation, with messages scrolling upward and eventually out of view as the conversation ensues. im also supports group chat, with users inviting others to join them in a specified ‘room.’ some systems, such as aim and icq, make some chat rooms public. in some im systems, pictures and urls can be included in the messaging. colors and fonts are personalizable. ‘buddy’ lists display information about im cohorts. buddies’ on-line handles (usernames) are displayed, along with indicators of activity (usually as a function of input device use) and availability (as inferred by activity and as stated explicitly by user-specified settings). buddies can be sorted into user-defined categories such as ‘friends,’ ‘family,’ ‘co-workers’ and so forth. (2002, p. 21) in this description, im is limited to text-based communication. however, most im systems now offer audio and video chatting functionalities as well. also, im is not students and instant messaging 145 limited to one-to-one communication. users can add additional users to create group conversations. in most cases, the im communication is handled through a software application installed on a users’ computer. the majority of this software is free. popular messaging systems are msn messenger (which recently migrated to windows live messenger), aol instant messenger, yahoo! messenger, skype, google talk and icq. this last one initiated a popularity boost among internet users in the second half of the 1990s. however, most of the current im systems are (partly) based on an older online chat medium: internet relay chat. together, wikipedia estimates that im systems currently have over 250 million active users. most systems use their own protocol, which prevents them from being interoperable. so users with an aol account cannot communicate with msn users. several attempts, for example by the internet engineering task force, to adopt a single, open, newly developed standard protocol have failed; it is only recently that yahoo and windows live have opened up their protocols for each other. interoperability will probably be one of the main challenges for im’s lasting success. social developments: youth and instant messaging the number of active im users (approximately 250 million worldwide) shows the success of the medium in the short period (since the late 1990s) it has been available to the public. especially among teenagers, im has become one of the most important communication means. more than 70% of today’s youth uses im, as surveys in the united states (12–17 year olds) and the netherlands (6–29 year olds) show (pew internet, 2005; qrius, 2005). according to the american research, only 44% of the adults used the medium. furthermore, the dutch survey showed that im shares first place with email as the most popular activity for teenagers when online. the medium is used on a much more serious level than many adults might think. through im, teenagers communicate with their friends, make appointments, date (pew internet, 2001) and collaborate on school tasks (grinter & palen, 2002). teenagers do about everything online through im, since ‘the buddy list is teens’ social world’ (boneva et al., 2006). the rising popularity of social networking websites such as facebook and myspace (pew internet, 2007), which incorporate im functionality into their systems, makes it even easier for teenagers to meet new social contacts online. grinter and palen (2002) point out that examining this development of im usage among teenagers provides valuable insights. im is the first and most successful form of social software that has entered into the public’s lives. studying the way in which the younger generation uses it, teaches us about its ‘role in domestic ecology’; also, the ‘communication habits they develop now may indicate what we can expect from them as adults’ (grinter & palen, 2002, p. 22). instant messaging and education the studies on im use mentioned indicate that youths already use the medium for educational activities. as already indicated, however, education has been neglecting 146 g. de bakker et al. the serious medium it has grown up to be. this might indicate that im would not be a suitable medium for education, but there are strong arguments to the contrary. for example, farmer (2005) conducted an im experiment among students, concluding that they had a positive attitude towards the medium. however, empirical results are available as well. a survey among students of the syracuse university school of information studies in new york showed that students benefited from the use of im as a tool for socialisation with fellow students outside lectures (nicholson, 2002). several studies have shown the value of the implementation of im as an online library referencing service (foley, 2002; andrews, 2004; cummings & guerlain, 2004; fagan, 2004; johnson, 2004). coniam and wong (2004) tested im as a tool for language proficiency training between students from different countries. although the measured quantitative differences were not statistically significant, a qualitative analysis showed improvements in the language proficiency skills of the participating students. hrastinski (2006) showed that adding im to an asynchronous distance learning course stimulated student participation. however, to enable education to determine the real benefits of im, more empirical evidence is needed. our future research will focus on identifying and exploring opportunities for im in class. to avoid the pitfall of implementing information and communication technologies solutions that are not sufficiently based on actual demands, we first want to have better insights in how students currently use im and what their wishes are for its use in education. for example, if students would state that they are willing to use im for consultation with teachers, but not for online collaboration with fellow students, we should take that into account in the design of an experiment of educational im applications. previously, several surveys have been conducted on im use among youth (pew internet, 2005; qrius, 2005). however, these studies did not provide data on educational use. at the same time, the few surveys conducted specifically on educational im use (grinter & palen, 2002; nicholson, 2002; farmer, 2005; boneva et al., 2006), and the experimental studies on im use available (andrews, 2004; coniam & wong, 2004; fagan, 2004), used relatively small sample sizes and, in most cases at least, consisted of user experience and appreciation data. a larger study therefore is needed to get a better indication of students’ im use. such a study should also gauge students’ demands for the implementation of im in their studies, as such data are also so far conspicuously lacking. to serve these ends, a survey was conducted among students at a dutch institution for higher education. the study participants all participants were students at the fontys university of applied sciences in the netherlands. a number of its institutes were willing to forward a request mail to their students. participants had to fill in their institutional personal identification number at the start of questionnaire, to prevent them from completing the online form more students and instant messaging 147 than once (they might want to do so in order to have a better chance at winning the lottery prize made available to them for maximising the response). eventually, 376 male and 405 female students participated. they were aged 16–57 years. the participants came from various studies in arts, science and humanities. materials and procedures based on the motive for this initial im study, we analysed students’ perspectives rather than examining what they are actually doing. a questionnaire with mainly multiple-choice questions was set-up. the questionnaire was based on questionnaires used in previous studies on im use (pew internet, 2001; grinter & palen, 2002; boneva et al., 2006), with additional questions added. the questionnaire was published online on a separate web space of the fontys institute’s website. students received an email with an explanation of the study, and the request for completing the online form. also, a news item was published on the institute’s intranet to attract more people to the survey website. in total, approximately 4500 students were approached. some of these received an email, which resulted in an initial response of 668 completed forms. after the news item was published on the institute’s intranet website, an additional 113 responses came in, providing a total of 781 responses. thus, the response rate was 17%. this is a fairly low percentage, which we feel is due to the non-committal way respondents were approached. results the results are organised in terms of the specific aspects of im usage we were looking for: im use frequency patterns, technological specifications of users’ environments, social aspects of im usage and school use. finally, the results of the specific questions asked on students’ views on educational implementation of the medium are described. in order to find out whether generation differences show different results, the data were analysed using a pearson product moment analysis. instant messaging use frequency patterns the survey shows that 96% of all respondents used im. in this group, 74% indicated they used im on a (nearly) daily basis: five to seven days a week. these data correspond to a previous dutch im survey (qrius, 2005) and american survey results on internet use (pew internet, 2005). female students used the medium more often than their male colleagues. it could be argued that im use develops over time. forty-eight per cent of the respondents using im indicated that they used the medium more often compared with the first time they used it. however, at the same time, 34% used it less often. possible differences between disciplines were also examined. as the disciplines were so diverse, we were only able to compare science students (e.g. physics, natural sciences, computer science, etc.) with the rest of the student population (e.g. history, 148 g. de bakker et al. languages, economics, etc.). technological tools such as im are at first often mainly used by technophiles, before becoming widespread among a larger public. in contrast with what people might expect, however, the science students (often considered to be the technophiles compared with other students) did not use im more intensively than the other students, as table 1 shows. im is usually used in between other computer activities (65%). when taking the time for it, only 24% of chats last longer than one hour. also, most participants indicated that on an average day they do not talk to more than 10 people (95%). the majority of the communication is done in separate conversations; only 3% of the respondents stated they used the group conversation functionality in their im system regularly. it can be argued that im is characterised by short sessions and a fragmented use throughout the day. although the questionnaire had options for respondents to note that they were not using im at all, very few (8%) did. this prompts the question whether im users are more than proportionally present among the respondents (selection bias). as it is to be expected that people who actually use the medium already respond differently from those who do not, one should only carefully extrapolate results to the entire population. this issue should be taken into account in future research. however, other surveys, which used respondent approaching procedures that guaranteed a more balanced population, show similar im use figures (qrius, 2005). a survey specifically on im use in the united states showed lower use frequency percentages (pew internet, 2004), but this used an adult sample only. technological specifications of users’ environments in the netherlands, im is often being referred to as ‘msn-ing’, since msn messenger/windows live messenger is the most popular service in the country. the survey results confirm this, 99% of the im-using respondents uses this messaging system. almost one-quarter of all students (24%) also use skype. other less frequently used systems are icq, google talk, internet relay chat, aim (the most used system in the united states) and yahoo messenger. mac users also mentioned table 1. im use frequency of science students, compared with the rest of the population percentage of science students percentage of the rest of the population no answer 0 1 never 3 2 once a month 5 2 once a week 7 5 two to four days a week 15 16 five to seven days a week 69 73 do not know 1 1 students and instant messaging 149 ichat, adium and gaim, all of which are multi-protocol applications (i.e. through these applications, users can use several im protocols). when asked about their computer facilities, students indicated having the following hardware: headset (40%), webcam (58%), microphone (53%) and speakers (92%). however, the availability of hardware does not mean that it is actually being used for im purposes. the majority of respondents mainly use text chatting. voice and video chat are being used ‘sometimes’ by 33% and 44%, respectively, and more than onehalf of the respondents ‘never’ use either. two places are most popular for using im: at home on a students’ personal computer (78%) and in school (69%). other places are at home on a shared computer (42%), at a friend’s house (18%) or at work (11%). only 8% of the students use im on a mobile device. social aspects of instant messaging usage most students have quite an extensive buddy list (im contact list). one-third of all students have more than 100 friends in their messenger list. also, people who use im more regularly have a larger contact list. although these are impressive figures, danah boyd argues these online friends have a different status to real-life friends (boyd, 2006). a large number of students (36%) indicated that they would have a hard time missing the availability of im. students tend to treat the medium rather freely. sixty-eight per cent of all respondents have ignored messages in the past, and 88% have even blocked contacts in their buddy list (making it impossible for those contacts to see or talk to them). conversation topics are very diverse. table 2 presents the survey conversation topic categories and the percentage of participants using them. school use and students’ views on educational implementation of im the majority of students (89%) used im for one or more study activities; 67% even used it for five or more of the study activities mentioned in the questionnaire. when table 2. im conversation topics (excluding educational use) and dutch higher education students using them (%) topic percentage of students using topic talking casually to family and friends seen rarely 82 talking casually to family and friends seen often 82 making appointments with family and friends 77 discussions with family and friends 28 playing games 16 discussing things you do not dare to tell face to face 14 starting or ending relationships 8 other 5 150 g. de bakker et al. asked whether the students wanted im to be implemented in their education, most students (86%) indicated they wanted to use im in their education. almost one-half of the students mentioned more than five activities they would like to see in educational im. table 3 presents the educational im activities mentioned in the survey and the percentages of students using them already personally. the last column represents the percentage of students that would like to see these activities being implemented in their education. differences between actual and desired use are small, with the exception of teacher guidance. generation differences the age of the respondents ranged from 16 to 57 years, but the majority was 19–21 years old (43% of the respondents). the survey showed some interesting correlation effects between certain aspects of im use and demands, and the age of the respondents. we will limit ourselves to describing only the correlations that proved significant (α = 0.01). in general, younger students are more intensive im users. they used the medium more often that the older students (r = −0.389), had more buddies (r = −0.429) and used it when connected to the internet more often (r = − 0.283). also, they talked to more people in one day (r = −0.326). when looking at technical specifications, we did not find a significant effect of age. however, younger students seem to replace text-based messaging by audio and video alternatives more often (r = −0.205). when asked whether the respondent would like to make more use of im, it turned out that the older students were more interested in doing so (r = 0.194). on the educational use of the medium, we did not find major differences between ages, apart from the younger students discussing school tasks more with their fellow table 3. educational im activities and demands of dutch higher education students educational activity percentage of students using personally percentage of students that would like to see implementation discussing school tasks with fellow students 85 85 sharing files (such as report concepts or sketches) 71 74 cooperating on school tasks with fellow students 63 76 discussing course material with fellow students 61 59 gathering content, such as course material 44 55 reflecting on fellow students’ work 36 45 teacher guidance 1 34 other – 1 students and instant messaging 151 students (r = −0.076). this result, however, was only significant at a lower level (α = 0.05). educational demands did show some interesting differences. in general, a larger number of the younger students would like to see an educational implementation of im in the future. four out of the nine categories of educational im use students could chose from for indicating their use for those purposes showed a negative correlation with age. those categories were ‘discussing school tasks with fellow students’, ‘cooperating on school tasks with fellow students’, ‘gathering content, such as course material’ and ‘sharing files (such as report concepts or sketches)’. also, the option ‘i do not want to use im in my study’ showed a positive correlation (r = 0.152), which indicates that older students are less willing to use the medium for educational purposes. when looking at the activities more specifically, the younger students would like to have more task discussions (−0.100), collaboration (−0.101), information gathering (−0.116) and file-sharing (−0.142) through im. conclusions and discussion in general, people who make more use of im also use it longer, have a larger buddy list (im contacts), and so on. thus, the adoption wave does not seem to concentrate on specific aspects of im usage. perhaps one of the most interesting findings of this study is that younger students use im more intensively than their older peers. younger students are also keener on trying new possibilities of the medium. this indicates that the adoption of the medium is still developing. it would be interesting to see how this development continues. will im become the primary communication medium for everybody? a dutch youth survey shows this is already the case for a certain age group (qrius, 2005). it could also be that age plays a big role in the adoption of the medium. it might be a youth tool people tend to grow out of. this is a question that can only be answered as the current im generation grows older. looking at the recent corporate im implementation wave, and the indicated usefulness of the medium in business contexts (nardi et al., 2000), the latter it is not to be expected (pauleen & yoong, 2001; perry et al., 2001). as noted previously, the survey might have attracted im users mainly. this should be taken into account for future research. we could test our findings among a more naturally varied group of students. as this study was part of a research project into educational im implementation possibilities, in future studies we will be analysing im conversations more closely in order to gain insights into what educational im conversations consist of in terms of content, communication style, and so on. in a later stage, we will also develop new applications or plug-ins for existing im systems, which make educational use of im more fit for current education, looking at recent changes such as self-directed learning. an example of such a new application is a peer-tutoring system, which enables students to get instant support from relevant fellow-students. this not only enables them to get their questions answered more quickly, it also reduces the workload for teachers, which has increased due to the introduction of the social–constructivist theories in learning (de vries et al., 2005). 152 g. de bakker et al. above all, the survey shows that education cannot simply ignore the importance of im. today’s students embrace the medium, and would like to see it being implemented in their learning environment. early empirical research has shown that im might be a useful tool, but the simple fact that students demand the medium’s implementation should be enough reason for education to consider that. the fact that only one-third of the population indicates that they would like to see teacher guidance being available via im should be considered in such implementations. this might indicate that students are interested in the self-organised bottom-up nature of the medium, which should be preserved when institutionalising it. references andrews, k. w. (2004) what are they asking and why are they asking it? usage patterns of an aol instant messenger chat reference service. unpublished master thesis, university of north carolina, chapel hill, nc. available online at: http://etd.ils.unc.edu/dspace/retrieve/ 461/kristinandrews.pdf (accessed 10 august 2006). boneva, b., quinn, a., kraut, r., kiesler, s., cummings, j. & shklovski, i. (2006) teenage communication in the instant messaging era, in: m. b. r. kraut & s. kiesler (eds) domesticating information technology (oxford, oxford university press), 201–218. boyd, d. (2006) friends, friendsters, and top 8: writing community into being on social network sites, first monday, 11(12). available online at: http://www.firstmonday.org/issues/issue11_ 12/boyd/index.html (accessed 21 december 2006). castelluccio, m. (1999) e-mail in real time, strategic finance, 81(3), 34–37. coniam, d. & wong, r. (2004) internet relay chat as a tool in the autonomous development of esl learners’ english language ability: an exploratory study, system, 32(3), 321–335. cummings, j. & guerlain, s. (2004) using a chat interface as an embedded secondary tasking tool, paper presented at the human performance, situation awareness and automation technology ii conference, daytona beach, fl. available online at: http://web.mit.edu/aeroastro/www/people/ missyc/pdfs/cummingshpsaa04.pdf (accessed 10 august 2006). de vries, f., kester, l., sloep, p., rosmalen, p. v., pannekeet, k. & koper, r. (2005) identification of critical time-consuming student support activities in e-learning, research in learning technology (alt-j), 13(3), 219–229. fagan. (2004) creating an instant messaging reference system, information technology and libraries (ital), 20(4), 202–212. farmer, r. (2005) instant messaging, educause review, 40(6), 48–62. foley, m. (2002) instant messaging reference in an academic library: a case study, college and research libraries, 63(1), 36–45. grinter, r. & palen, l. (2002) instant messaging in teen life, paper presented at cscw’02, new orleans, la, 16–20 november, pp. 21–30. hrastinski, s. (2006) introducing an informal synchronous medium in a distance learning course: how is participation affected?, the internet and higher education, 9, 117–131. johnson, c. m. (2004) online chat reference, reference & user services quarterly, 43(3), 237–247. nardi, b. a., whittaker, s. & bradner, e. (2000) interaction and outeraction: instant messaging in action, paper presented at the computer supported cooperative work conference, philadelphia, pa, 2–6 december. nicholson, s. (2002) socialization in the ‘virtual hallway’: instant messaging in the asynchronous web-based distance education classroom, the internet and higher education, 5(4), 363–372. students and instant messaging 153 pauleen, d. j. & yoong, p. (2001) facilitating virtual team relationships via internet and conventional communication channels, internet research: electronic networking applications and policy, 11(3), 190–202. pew internet (2001) pew internet project report: teenage life online (washington, dc, pew). available online at: http://www.pewinternet.org/pdfs/pip_teens_report.pdf (accessed 15 august 2006). pew internet (2004) pew internet project report: the internet and daily life (washington, dc, pew). available online at: http://www.pewinternet.org/pdfs/pip_internet_and_daily_life. pdf (accessed 16 march 2007). pew internet (2005) pew internet project report: teens and techonology (washington, dc, pew). available online at: http://www.pewinternet.org/pdfs/pip_teens_tech_july2005web.pdf (accessed 14 march 2005). pew internet (2007) pew internet project report: social networking websites and teens: an overview (washington, dc, pew). available online at: http://www.pewinternet.org/pdfs/pip_sns_ data_memo_jan_2007.pdf (accessed 16 march 2007). perry, m., o’hara, k., sellen, a., brown, b. & harper, r. (2001) dealing with mobility: understanding access anytime, anywhere, acm transactions on computer–human interaction, 8(4), 323–347. prensky, m. (2001) digital natives, digital immigrants, on the horizon, 9(5), 1–6. qrius (2005) jongeren 2005, youth research report (amsterdam, qrius). available online at: http:// www.hetjongerenonderzoek.nl/pdf/inleiding2005(kort).pdf (accessed 18 august 2006). alt-j, research in learning technology vol. 17, no. 3, november 2009, 161–172 issn 0968-7769 print/issn 1741-1629 online © 2009 association for learning technology doi: 10.1080/09687760903247617 http://www.informaworld.com towards an understanding of the virtual context in mobile learning sarah corneliusa* and phil marstonb aschool of education, university of aberdeen, uk; bcentre for learning and teaching, university of aberdeen, uk taylor and franciscalt_a_424935.sgm (received 14 january 2009; final version received 2 july 2009) 10.1080/09687760903247617alt-j research in learning technology0968-7769 (print)/1741-1629 (online)original article2009association for learning technology1730000002009sarahcorneliuss.cornelius@abdn.ac.uk much of the literature theorising mobile learning emphasises the importance of context with physical and social contexts identified to date. our work to design and implement authentic mobile simulation activities using sms text messaging suggests that a third context may also be important. this is a virtual context, the learner-created cognitive space within which the activity takes place. the arrival of a message during a simulation activity will disrupt the real world physical and social contexts in which the learner finds themselves and transfer them into the virtual context. we argue that this disruptive power of the mobile device to shift the user’s presence may be one of the distinguishing characteristics of certain mobile learning applications. this paper will explore the idea of the virtual context and discuss the role of issues of context, presence and disruption, illustrating these with reference to a case study of mobile simulation learning using sms text messaging. keywords: mobile learning; sms; simulation; virtual context; presence introduction the landscape of mobile learning is still being sculpted as innovative applications emerge. the use of mobile phones is now almost ubiquitous amongst students in many parts of the world (in the uk as many as 99% of undergraduates already have one – margaryan et al. 2008) and one of the main uses to which they put their phones is sms (short messaging service or text messaging). sms provides “anywhere, anytime” (geddes 2004, 1) access to learners and a channel for communication with which they are generally familiar and comfortable. sms-related applications have to date covered a number of areas: such as providing administrative information (e.g. duvall et al. 2007), offering revision and test materials in the form of short quizzes (e.g. ogino et al. 2008), facilitating language learning (e.g. kukulska-hulme and shield 2008; cavus and ibrahim 2009), and improving feedback and interaction in lecture-related applications (e.g. kinsella 2009; wang et al. 2009). this paper focuses on simulation activities using sms text messaging. simulations have a wider history in mobile learning, not always employing sms. one category of simulations is the participatory simulation (colella 2000), in which learners “act out key parts in an immersive recreation of a dynamic system” (naismith et al. 2004, 13) and their actions have an effect on progress of the simulation. examples implemented *corresponding author. email: s.cornelius@abdn.ac.uk 162 s. cornelius and p. marston with mobile technologies include simulations representing the spread of a virus (colella 2000) and a game to encourage the development of a conceptual understanding of animal behaviour (facer et al. 2004). in the augmented reality simulation environmental detectives (squire and klopfer 2007) learners play the role of environmental scientists conducting investigations into a simulated chemical spill. the area of simulation of particular interest to our work is that of authentic real-time decision-making scenarios – situations in which the use of a mobile device is authentic and appropriate. in these situations (e.g. disaster management, mentoring, medical emergency) a decision maker is ‘on the move’ so mobile phone contact would be appropriate. rapid decision making is required, with decisions, which will affect the progress and outcome of the situation, communicated by mobile phone. indeed sms text messaging has become important in real-world emergency situations (e.g. bbc 2008), and despite potential difficulties such as the problem of guaranteeing message arrival (latimer 2008) it is likely that it will remain an important tool for decision makers. simulations have particular relevance for the development of work-related skills, and have been applied in non-mobile learning, such as for the development of practical skills in geo-spatial data handling using a ‘virtual placement’ in the geosciences (cornelius, medyckyj-scott, et al. 2008). the virtual placement, which supports learners playing the role of a research assistant undertaking an assessment of the visibility of wind farms for a consultancy company, also incorporates a real-time element, and email is used to provide a channel of communication between members of a virtual team and the student. in mobile learning this type of simulation is less well represented to date and difficult to place easily into classifications of applications such as those developed by naismith et al. (2004), patten, arnedillo sánchez, and tangney (2006) or frohberg, göth, and schwabe (2009). winters (2006) briefly reviews the conceptual development of mobile learning. he considers that early work to develop definitions of mobile learning focused on the technology, then, the mobility of the learner came to the fore. more recently mobile learning has been considered as communication in context building on reinterpretations of activity theory (sharples 2005) and laurillard’s conversational framework (laurillard 2002). in line with this trend, although drawing on activity theory alone, wali, winters, and oliver (2008, 55) offer a definition of mobile learning as “learning that occurs as a result of pursuing learning activities that are directed towards achieving the same objective across multiple contexts (both social and physical)”. frohberg, göth, and schwabe (2009) also draw on activity theory to evaluate and categorise mobile learning projects, identifying six factors (context, tools, control, communication, subject, and object) that can be used to describe mobile learning projects. they consider the most significant of these factors to be context, but restrict their interpretation of context to “where the learning takes place” (7). context has also been a concern of other researchers including yang (2006), kurti, spikol, and milrad (2008) and hansen and bouvin (2009). the focus of these studies has been context awareness, which allows mobile devices to respond automatically based on users’ location. in this paper we revisit the idea of multiple contexts, adding to the contexts suggested by wali, winters, and oliver (2008), kurti, spikol, and milrad (2008), hansen and bouvin (2009) and frohberg, göth, and schwabe (2009) a ‘virtual context’ that aids conceptualisation of mobile simulation activities. a case study which illustrates the role of the virtual context in simulation activities is provided and the concept discussed in the light of other perspectives on presence and disruption in mobile communication. alt-j, research in learning technology 163 towards the virtual context the notion of context is important in recent conceptualisations of mobile learning. sharples, taylor, and vavoula (2005, 5) describe context as “a dynamic entity, constructed by the interactions between learners and their environment” and wali, winters, and oliver (2008) highlight the role of multiple contexts in mobile learning, identifying two particular contexts – the physical (the ‘environment’ which surrounds the user) and social (that which arises from constructive interaction between people and technology). wali, winters, and oliver argue for the application of engeström’s activity theory framework to mobile learning in which the social context is considered through rules, community and division of labour. this is at odds with sharples, taylor, and vavoula’s (2007) suggestion for the distinction between two layers representing technological and semiotic components of the system. it may be the case that mobile phones and sms messaging are “tools [that] when used, become an extension of the body” (gibson 1979 cited by rettie 2005, 25), so for student users a separate technological layer is no longer necessary. in line with wali, winters, and oliver (2008) and frohberg, göth, and schwabe (2009), we would argue that a simplified version of the activity system allows effective analysis of mobile learning activities. frohberg, göth, and schwabe (2009, 7) consider four types of context: independent, formalised, physical and socialising. they suggest context is independent when “the current environment of the learner has no relationship to their current issue of learning”. this context is considered of low pedagogic ambition and low complexity. however, this low ranking does not allow for the use of an independent context as part of a pedagogic design, for example to represent an authentic virtual context for learners participating in a mobile simulation activity. when a simulated event occurs the student has no control over the time or place in which they have to address the problem and thus the physical location of the student is, by design, not relevant to the learning activity, but the fact that the location is not relevant is important. further up frohberg, göth, and schwabe’s (2009) scale of complexity and pedagogic ambition is their notion of physical context. here, “the place of being is relevant for the learning issue” (8). in a mobile simulation there is a place that is relevant to the activity, but it is a fictional or virtual representation of a (possibly real) place. actual physical presence in that place is not necessary to engage with the activity. participants are ‘transported’ to the virtual place by their imagination and the experience of ‘presence’ (rettie 2005). this is quite different to frohberg, göth, and schwabe’s (2009) notion of physical context that (in common with much of the literature on mobile learning) requires interaction with a physical space, such as a museum, gallery or field location. thus, in addition to the four contexts suggested by frohberg, göth, and schwabe (2009) there seems to be a need for an additional context in this framework where significant and sophisticated learning activities take place and where the physical location of the learner is not relevant. wali, winters, and oliver (2008, 56) define context as “a combination of the properties of the physical location where the learning activity takes place and the rules and division of labour within the community that the learner belongs to” and argue that it is context crossing – changes in both the physical and social contexts – that distinguishes mobile learning from static learning. context crossing is certainly associated with mobile simulation activities. in such an activity the physical context is that which surrounds the user at the time when they receive a message. this may be a classroom, 164 s. cornelius and p. marston workplace, home or any other location, which will have an associated social context. as time progresses the learner will move (or cross) to a different physical and social context, where the next message will be received. however, the wali, winters, and oliver (2008) model does not allow for representation of the context of the simulation scenario itself which could be a different geographical location in which an event is unfolding. they do, however, suggest that mobile learning may help to create contexts that cannot be created with conventional learning devices, as it certainly does in the simulation case. thus an additional context, which we have termed the ‘virtual context’, can be identified. this context is intimately associated with the learning activity and it is returned to (crossed to) by the learner each time they engage with the activity. the virtual context has physical properties (physical features and perhaps a geography of its own – real or conceptual – so that it can be regarded as a place or a space) and the learner belongs to an associated virtual community (the community associated with their role in the scenario) with rules and a division of labour. key to this notion of virtual context is the concept of place as described by rettie (2005). she describes the potential for new forms of communication such as mobile phones to “change perceptions of time and place” and the disruptive nature of mobile phones can be expressed as “the experience of presence between parallel interactions, creating absent presence in the face-to-face interaction, as presence is diverted to the phone interaction” (rettie 2005, 16–7). as an illustration, imagine, for example, a simulation activity where the student is required to take the role of a decision maker managing the emergency services in response to a volcanic eruption on in the island of sicily. the student may be located at their normal place of study, home, place of work or in a social space – these are possible physical contexts which all have their own associated social contexts (rules and division of labour). however the simulation activity requires the student to also consider the physical location of the volcano (sicily) and this becomes the virtual context. the cultural and organisational characteristics of sicily provide the social context (rules and division of labour) to accompany this virtual context. in order to complete the simulation activity, the student will cross physical contexts and also cross into and out of the virtual context. regardless of their physical context, during their engagement with the simulation their attention will be on the virtual context (sicily). the framework for analysing mobile learning provided by wali, winters, and oliver (2008) has been adapted to include this idea of the virtual context (figure 1) figure 1. the physical and virtual contexts (adapted from wali, winters, and oliver 2008). the development of a virtual context for mobile simulation activities has some significant advantages: it remains persistent throughout the learning activity; it requires cognitive engagement and may enhance the authenticity of a scenario; and it provides a disruptive element to learning. once established, the virtual context remains persistent for the duration of a mobile activity. whilst the physical context within which the learner finds themselves may change (work, home, university, etc.), the virtual context endures. for the duration of a simulation activity the learner ‘takes with them’ the virtual context. it will have its own rules for social interaction – for example responses may be required in a particular format and/or within a particular timeframe. thus during a mobile simulation activity familiarity and comfort with the virtual context can be developed and maintained to encourage engagement with the activity. a virtual context can be based on a real physical environment, or be a completely imaginary space. a certain degree of cognitive investment is required by the student alt-j, research in learning technology 165 to help them ‘imagine’ all the characteristics of the virtual context. the features of the simulation environment will be created within the student’s imagination and the impact of their decisions will have to be visualised. the experience is similar to that required to imagine a world from reading a book rather than the more structured and ‘helpful’ approach of a virtual world or real life role play. baños et al. (2005) found that using imagination to construct a sense of place could provide a greater sense of reality than using virtual reality. the use of a mode of communication (such as sms) that is indistinguishable from that which would be used in a real world situation enhances this experience of place. placing the student in a first person role increases their immersion, making them feel part of the activity rather than a witness to an avatar. this additional cognitive engagement may be beneficial for the learner, requiring deeper thinking and involvement. one of the features which this virtual context carries with it is the ability to disrupt. when a learner receives a message from a simulation, it disrupts their current physical and social contexts. it will transport them, albeit temporarily, to the virtual context. the opportunity for disruption is enhanced when simulations are carried out in real time, with messages arriving at possibly unexpected or inconvenient times. this disruption and repeated return to the virtual context offers opportunities for microlearning and at the same time retains engagement with the task over an extended period of time. it is this ability to impinge on the learner’s current context which appears to be one of the defining characteristics of mobile simulations. for example, a hard copy book also provides the advantages of retaining a virtual context and a requirement for cognitive engagement. it also allows the reader to return to the virtual context repeatedly. indeed there are books – a well known example is the warlock of firetop figure 1. the physical and virtual contexts (adapted from wali, winters, and oliver 2008). 166 s. cornelius and p. marston mountain (jackson and livingston 1982) – which require decision making in line with the model used for simulation activities. however, a book will be read at the reader’s convenience, rather than at a time dictated by the simulation and the book will not disrupt the reader’s other activities. other engaging technologies from a games background which offer a similar level of disruption include the tamagotchi®, which pesters the ‘owner’ to keep it alive and well. a real-time authentic decision-making scenario a flood disaster simulation is outlined below as a case study of a real-time authentic decision-making scenario which has been tested and evaluated. full details of the design and evaluation of a pilot version of this simulation are reported in gemmell, finlayson, and marston (in press). following the success of this pilot, twenty three final year undergraduates studying applied geomorphology participated in the simulation in 2008. a mixed methods study, involving learners, tutors and technologists, was undertaken to gain a holistic view of the activity and investigate learners’ experiences. the study, informed by an activity theory perspective, aimed to uncover issues related to control, context and communication. preliminary results were reported in cornelius, gemmell, and marston (2008). seventy percent of the cohort responded to a detailed questionnaire after the simulation was completed, and two students, the tutor and educational technologist were interviewed. participation throughout was voluntary and it was disappointing that additional students could not be recruited for interviews. comments from the two volunteers may be unrepresentative of the group as a whole, however, they are considered valuable in providing an insight into the experiences of learners who had a positive experience of the simulation. the scenario is outlined first below, and findings from the evaluation are then discussed in the light of issues raised by the virtual context. the flood disaster simulation during the flood disaster simulation learners were asked to imagine that they were the manager of civil defence utilties in a french town. the town was given a fictional name for the simulation, but the scenario was based on a real event which took place in a real location. a local manager would have some understanding of the geographical location, and learners were provided with background information on the area in advance of the simulation. as the simulation began, local weather conditions were changing and the manager had to respond to changes to prevent a flood disaster. their mobile phone was their most important channel of communication and as the situation unfolded they received updates about the situation and communicated decisions on what action they wished to take by sms text message (figure 2). the simulation ran in real time over a 72-hour period and the use of mobile phones allowed the ‘manager’ to be contacted anywhere and make quick decisions to help control the situation. figure 2. sms messages used during the flood disaster simulation. (a) information message; (b) message requiring decision response from student. the simulation provided a personalised experience for each learner as the decision they made at each ‘event horizon’ determined their pathway through a decision tree to an appropriate scenario end point. decisions had to be made within a two hour period of receiving information, and if this deadline was missed the system implemented a decision on behalf of the student. this reflects the real life situation where, even if the manager were unavailable, a decision would have to be made in such a rapidly developing scenario. alt-j, research in learning technology 167 the activity encouraged rapid assessment of information and the application of prior theoretical knowledge to a practical context. if learners required further information to support their decision making specific requests had to be made to the tutor, who played the role of a representative from civil defence hq. the virtual context in the flood disaster simulation the virtual physical context for the flood disaster simulation was based on a real geographical location, and although learners were provided with some background on the area, they still had to imagine many of the aspects of the environment. this was particularly important once the simulation was underway and their decisions started to invoke changes in the environment. the virtual social context was in part based on rules and norms established by the tutor and those associated with the role being played, but elements were also created in the learners’ imaginations when they considered the effect the flood had on the imaginary residents of the town. although, in general, questionnaire respondents were neutral about whether the simulation was a realistic experience, they were more likely to disagree with the statement that the activity was artificial and did not reflect what would happen in real life. some of their criticisms related to elements of the social context – for example, they felt there was insufficient information in some text messages and would have liked the opportunity to select from more than two possible options for action, or even to suggest their own actions. however, evidence from interviewees also suggested that engagement with the physical and social elements of the virtual context was beneficial for their learning: i think [the simulation] helped to reinforce how these things go through […] you think it’s just common sense sometimes – a flood’s going to come, put out some figure 2. sms messages used during the flood disaster simulation. (a) information message; (b) message requiring decision response from student. 168 s. cornelius and p. marston sandbags – you don’t really think of is it going to cost money, about time, is it going to affect people, what’s going to happen downstream. (female student 1) it was good to look at a real life situation rather than just a vague sort of thing…you hear the numbers and you hear about floods and things but it doesn’t really mean anything. whereas when you are actually having to prevent death by the text message then you have to…you do look at the situation slightly differently. (female student 2) the quotations provided above also illustrate that the interviewees demonstrated cognitive engagement and commitment to the scenario: they give an indication of ‘taking a different view’ of the scenario and the adoption of a wider perspective than might be obtained by other types of activities. there was also evidence of emotional engagement. although not representative of all learners in the group the interview respondents both highlighted their ‘involvement’ with the activity, one commenting that it was a good way to get involved in such a scenario, and the other that being involved encouraged them to do more work. other emotions were evident, for example the tutor noted that some students suffered a ‘mild case of trauma’ because mixups with messages led to them wiping out most of the population they were supposed to be defending: they would come here saying ‘i just feel frustrated, i thought i’d done my best but i’ve killed 50 people’. telling them they could do it all over again and not to worry about it partly reassured them, but i think, for some of them they thought ‘now i’ve got a second chance to kill 50 people’. (tutor) these comments suggest a level of role engagement which the tutor admitted he had not expected. a wide range of emotions were reported by learners in questionnaire responses, both positive (enjoyment, engagement, involvement) and negative (annoyance, frustration), although the negative emotions were mostly prompted by technical issues (including problems with the service providers’ network, and some participants returning messages in inappropriate formats) which impacted on preferred decisionmaking strategies and took the progression of the scenario out of the learner’s control. some learners clearly did not engage fully with their roles, making decisions on the basis of ‘common sense’ or ‘instinct’ or even following a conviction that ‘option 2 was always the right answer’. differences in the level of engagement with the scenario and the role are perhaps inevitable in any group of learners, but there is evidence that most learners were engaged by the activity (for example 60% ‘strongly agreed’ or ‘agreed’ that they looked forward to messages arriving). the simulation interrupted learners’ day-to-day activities in a variety of physical locations (including at their place of residence and workplace), yet the real time element was viewed positively and appears to have enhanced engagement with the activity. all of the questionnaire respondents ‘strongly agreed’ or ‘agreed’ that they ‘enjoyed the real time aspect of the simulation’ and were happy to receive messages which disrupted their lives outside normal ‘working’ hours. only one questionnaire respondent indicated that the activity intruded negatively on other aspects of their life. one respondent commented that “you felt more involved because you didn’t know when you were going to get the updates – that was fun,” whilst another “liked the way it brought together university and the outside.” one interviewee reported that a group had been sitting together waiting for the first message “we were all like ‘i’ve got a message.’ it was exciting.” alt-j, research in learning technology 169 the positive attitude to the disruptive nature of the activity may be enhanced by the fact that the simulation was seen as a novel and innovative activity, particularly in terms of the approach to assessment. for assessment purposes students were required to reflect on and justify the decisions they had made during the simulation in diary format. this was widely applauded as ‘not just another boring essay’. however, whether this approach would have an enduring positive effect is brought into question by the comments of one of the interview respondents. she cautioned against too much use of mobile phones for learning in general to prevent boredom and a less positive response – ‘oh for goodness sake, not another one’ – to the disruption involved. discussion the notion of virtual context is not explicitly accounted for in wali, winters, and oliver’s (2008) definition of mobile learning, and not at all in the work of frohberg, göth and schwabe (2009). it may not be important in all mobile learning applications. however, it does appear to have a role to play in the conceptualisation of mobile simulation activities such as those described above. the virtual context may also assist conceptualisation of some participatory simulations where learning is not constrained to take place in a particular physical location. in particular the virtual context helps to highlight the important role and potential of the disruptive nature of intrusive communication channels for learning activities and the power of a textual narrative to stimulate the construction of an imaginary virtual world. the disruptive power of mobile devices to transport a user away from their current context warrants further investigation. in the flood disaster simulation the real-time and disruptive aspects of the activity appear to have had an impact on enjoyment and engagement. considering this further within the framework of context crossing (wali, winters, and oliver 2008) or the notion of ‘frame-switching’ between real and virtual worlds (bell 2009) may have value. loewenstein’s (1994) ‘gap theory’ of curiosity may also shed light on learners’ reactions to unexpected messages which compel them to engage and investigate something in greater depth. other work on interruption and the blurring of boundaries between contexts includes that from the design of mobile and sms games and demonstrates the complexity of issues associated with the juggling of interactions and management of disturbances created by sms messages (e.g. tolmie et al. 2008). the issue of presence also deserves further exploration in connection with teaching and learning. baños et al. (2005, 90) note that “many authors assume that a person feels present in an environment when his/her cognitive processes lead to a mental representation of a space” and in a study of participants in imagined and virtual spaces suggest that imagination is a good procedure to elicit a sense of presence, although some users may have difficulties in maintaining this sense of presence. in particular “it could be difficult for people to suppress for a long time the actual physical environment…in favour of an alternative and cognitive environment” (98), though this may not be such an issue with an interface such as sms, which is used for normal day-to-day interactions and which was preferred over email options offered to the flood disaster participants. the use of such a familiar interface may reduce the cues for un-reality that hinder the suspension of disbelief found in 3d virtual worlds. klopfer (2008) also suggests that creating a sense of presence is difficult, but cites research by others (salzman et al. 1999 and winn et al. 2002), which suggests that increasing presence is associated with increased learning in virtual 170 s. cornelius and p. marston learning environments. such issues warrant further research in connection with the use of sms for learning during simulation activities. for example, work to investigate learners’ ability to develop and maintain a sense of place and presence using text prompts alone would be interesting. other sms simulations drawing on the model used for the flood disaster simulation are under development and in testing. in particular a contrasting scenario, developing an appropriate mentoring relationship with learners on an adult literacy qualification, is providing further insight into the virtual context and associated issues. in this example the virtual context is completely imaginary. there are no physical features about which information can be provided, and the characteristics of the mentee must be imagined by the mentor. this reflects the real world situation in which a new relationship must be developed with someone one has never met. the social context is informed by the mentor’s own professional norms and expectations, and personal preferences, although there are some restrictions imposed by the simulation design on the time limits and nature of responses. there is the potential for this simulation to be more disruptive to learners than the flood disaster example, since participants are all working professionals with other demands on their time and attention. this simulation and others at the early stages of design in the areas of medicine and education will permit further exploration of issues of context, cognitive engagement, persistence and disruption. there may be other areas relevant to teaching and learning where the development of a more robust virtual context may be beneficial for learners. for example, activities involving the use of sms in language learning such as vocabulary practice, may benefit from the design of virtual context (e.g. school or hospital) to which vocabulary is relevant and within which practice can take place. disruptive return to this virtual context may encourage microlearning through repetition and reinforcement. similarly the use of a virtual context in connection with stories or other narratives may help to engage learners in other situations and disciplines. conclusions the work on which the concept of the virtual context has been developed is necessarily limited in scope and duration. however, the notion of virtual context allows mobile learning to be de-coupled from the restrictions of the physical context, at least in the case of simulations using sms, such as the flood disaster management example presented here, and is providing a useful framework for the development of new sms based simulations in other authentic decision making scenarios. the challenge for designers is to provide a virtual context which is realistic and persistent (physically and socially) within which the learner can play an authentic and engaging role. the relevance of the virtual context to other mobile learning applications requires further investigation, as there appear to be pedagogical and other benefits of using a virtual context to engage learners. in particular the pedagogic benefits of stimulating learners’ imagination so that they construct their own learning spaces to help engagement with learning activities could be beneficial to the development of mobile learning. consideration of the virtual context may have a role in encouraging deeper reflection and knowledge application, analysis, synthesis and evaluation that will meet the demand from frohberg, göth, and schwabe (2009) for mobile learning applications with ‘higher pedagogic ambition’. alt-j, research in learning technology 171 acknowledgements the original concept for the flood disaster simulation was developed at the university of aberdeen by alastair gemmell (school of geosciences) and phil marston with programming done by ian finlayson (directorate of information technology). chris aldred (school of education) has designed the mentoring case study scenario. thanks are also due to steve sidaway at txttools.co.uk for assistance with technical aspects of the messaging system used, and to questionnaire and interview respondents. references baños, r.m., c. botella, b. guerrero, v. liano, m. alcaniz, and b. rey. 2005. the third pole of the sense of presence: comparing virtual and imagery spaces. psychnology journal 3, no. 1: 90–100. bell, d. 2009. learning from second life. british journal of educational technology 4, no. 3: 515–25. british broadcasting corporation (bbc). 2008. surgeon saves boy’s life by text. bbc news. http://news.bbc.co.uk/1/hi/health/7761994.stm. cavus, n., and d. ibrahim. 2009. m-learning: an experiment in using sms to support learning new english language words. british journal of educational technology 40, no. 1: 78–91. colella, v. 2000. participatory simulations: building collaborative understanding through immersive dynamic modelling. journal of the learning sciences 9, no. 4: 471–500. cornelius, s., a. gemmell, and p. marston. 2008. real-time simulation on the move: the learner context. paper presented at online educa 08, 14th international conference on technology supported leaning and training, december 3–5, in berlin, germany. cornelius, s., d. medyckyj-scott, d. forrest, a. williams, and w. mackaness. 2008. the virtual placement – an alternative to the traditional work placement in the geographical sciences? journal of geography in higher education 32, no. 2: 287–302. duvall, j.b., m.r. powell, e. hodge, and m. ellis. 2007. text messaging to improve social presence in online learning. educause quarterly 30, no. 3: 24–8. facer, k., r. joiner, d. stanton, j. reid, r. hull, and d. kirk. 2004. savannah: mobile gaming and learning? journal of computer assisted learning 20, no. 6: 399–409. frohberg, d., c. göth, and g. schwabe. 2009. mobile learning projects – a critical analysis of the state of the art. journal of computer assisted learning 25, no. 4: 307–31. geddes, s.j. 2004. mobile learning in the 21st century: benefit to learners. the knowledge tree, edition 6. australian flexible learning network. http://knowledgetree.flexiblelearning.net.au/edition06/download/geddes.pdf. gemmell. a., i. finlayson, and p. marston. in press. first steps towards an interactive realtime hazard management simulation. journal of geography in higher education. hansen, f., and n. bouvin. 2009. mobile learning in context – context aware hypermedia in the wild. international journal of interactive mobile technologies 3, no. 1: 6–21. jackson, s., and i. livingston. 1982. the warlock of firetop mountain. london: puffin books. kinsella, s. 2009. many to one: using the mobile phone to interact with large classes. british journal of educational technology 40, no. 5: 956–8. klopfer, e. 2008. augmented learning: research and design of mobile educational games. cambridge, ma: mit press. kukulska-hulme, a., and l. shield. 2008. an overview of mobile assisted language learning: from content delivery to supported collaboration and interaction. recall 20, no. 3: 271–89. kurti, a., d. spikol, and m. milrad. 2008. bridging outdoors and indoors educational activities in schools with the support of mobile and positioning technologies. international journal of mobile learning and organisations 2, no. 2: 166–86. latimer, d. 2008. text messaging as emergency communication superstar? nt so gr8. educause review 43, no. 3: 84–5. laurillard, d. 2002. rethinking university teaching: a conversational framework for the effective use of learning technologies. 2nd ed. routledge-falmer: london. loewenstein, g. 1994. the psychology of curiosity: a review and reinterpretation. psychological bulletin 16, no. 1: 75–98. 172 s. cornelius and p. marston margaryan, a., d. nicol, a. littlejohn, and k. trinder. 2008. students’ use of technologies to support formal and informal learning. in proceedings of world conference on educational multimedia, hypermedia and telecommunications, 4257–66. chesapeake, va: aace. http://www.editlib.org/p/28976. naismith, l., p. lonsdale, g. vavoula, and m. sharples. 2004. literature review in mobile technologies and learning. report 11: future lab series. http://www.futurelab.org.uk/ resources/documents/lit_reviews/mobile_review.pdf. ogino, t., m. suzuki, f. kusunoki, and i. hatono. 2008. cellular phone with gps based learning support system for field studies. in proceedings of world conference on educational multimedia, hypermedia and telecommunications, 1812–7. chesapeake, va: aace. http://www.editlib.org/p/28620. patten, b., i. arnedillo sánchez, and b. tangney. 2006. designing collaborative, constructionist and contextual applications for handheld devices. computers & education 46, no. 3: 294–308. rettie, r.m. 2005. presence and embodiment in mobile phone communication. psychnology journal 3, no. 1: 16–34. salzman, m.c., c. dede, r.b. loftin, and j. chen. 1999. a model for understanding how virtual reality aids complex conceptual learning. presence: teleoperators and virtual environments, 8, no. 3: 293–316. sharples, m. 2005. learning as conversation: transforming education in the mobile age, in proceedings of conference on seeing, understanding, learning in the mobile age, 147–52. budapest, hungary. http://www.eee.bham.ac.uk/sharplem/papers/theory%20of %20learning%20budapest.pdf. sharples, m., j. taylor, and g. vavoula. 2005. towards a theory of mobile learning. in proceedings of mlearn 2005 conference, ed. h. van der merwe and t. brown. cape town: south africa. http://www.mlearn.org.za/cd/papers/sharples-%20theory%20of% 20mobile.pdf. sharples, m., j. taylor, and f. vavoula. 2007. a theory of learning for the mobile age. in the sage handbook of e-learning research, ed. r. andrews and c. haythornthwaite, 221–47. london: sage. squire, k., and e. klopfer. 2007. augmented reality simulations on handheld computers. journal of the learning sciences 16, no. 3: 371–413. tolmie, p., a. crabtree, t. rodden, and s. benford. 2008. “are you watching this film or what?” interruption and the juggling of cohorts. in proceedings of the acm 2008 conference on computer supported cooperative work, 257–66. new york: acm. http:// doi.acm.org/10.1145/1460563.1460605. wali, e., n. winters, and m. oliver. 2008. maintaining, changing and crossing contexts: an activity theoretic reinterpretation of mobile learning. alt-j, research in learning technology 16, no. 1: 41–57. wang, m., r. shen, d. novak, and x. pan. 2009. the impact of mobile learning on students’ learning behaviours and performance: report from a large blended classroom. british journal of educational technology 40, no. 4: 673–95. winn, w., m. windschitl, r. furland, and y. lee. 2002. when does immersion in a virtual environment help students construct understanding? in proceedings of the international conference of the learning sciences icls 2002, ed. p. bell and r. stevens. mahwah, nj: erlbaum. winters, n. 2006. what is mobile learning? in big issues in mobile learning. report of a workshop of the kaleidoscope network of excellence mobile learning initiative, ed. m. sharples, 7–11. nottingham: learning sciences research institute, university of nottingham. http://www.lsri.nottingham.ac.uk/publications_pdfs/big_issues_ report_published. pdf. yang, s.j.h. 2006. context aware ubiquitous learning environments for peer-to-peer collaborative learning. educational technology and society 9, no. 1: 188–201. iterative student-based testing of automated information-handling exercises c.k. ramaiah*. mubarak sulaiman** and a.j. meadows*** *desidoc, metcalfe house, delhi, india ** department of library and information science, king saud university, riyadh, saudi arabia ***department of information and ubrary studies, loughborough university of technology the continuing rapid changes in how information is handled via computers means that training in computer-based information handling must itself undergo continuous modification. the investigations reported here examine the use of student feedback to improve presentation of hands-on exercises. student responses appear to be reasonably consistent across both time and different student backgrounds. for the groups examined here, such factors as age and sex appear to have little effect on responses. the only significant factor is duration and extent of prior computer experience. problems in hands-on exercises noted by the students therefore tend to have a continuing applicability, which suggests the value of iterative feedback in the provision of such exercises. introduction much laboratory teaching of information-handling involves students in evaluating information provided either online or via a computer package. a lecturer can help students carry out these tasks in a variety of ways. in particular, it is customary to provide students with hand-outs, and there is good evidence that such hand-outs are a valuable resource, especially for lower-ability students (see, for example saloman, 1979). in many of these exercises, students are passive receivers of information, in the sense that they assess the information but do not change it. however, it is sometimes possible to use student feedback to change the original input. in this case, the users' mental models of the system can be employed to modify the user-interface set up by the original designer (see moran, 1981). a number of experiments have been carried out in the department of 35 ck. ramaiah et al iterative student-based testing of automated information-handling exercises information and library studies at loughborough university to examine how computer interfaces and instruction sheets used in teaching can be improved by student feedback. the present paper discusses examples of this work to help suggest both the factors to be taken into account and the sorts of changes involved. our approach has been based on the concept of 'iterative usability testing', the value of which has recently been emphasized by shneiderman (1993). the teaching/learning model envisaged here is essentially teacher-focused, and supposes a teaching environment in which both hand-outs and practical assistance are provided (joyce and weil, 1980). however, part of our experiments involves student groups of differing sizes, so that the results also have some implication for group-focused teaching. the experimental approach is task-based, depending on the provision of well-structured tasks for students to perform at three levels. these levels are (1) simple (finding factual information from a hypertext database); (2) complex (evaluating the presentation of information via an electronic bulletin board); (3) design (setting up a hypercard stack). for this work, teaching sessions with second-year students of information and library studies were employed. experiments in categories (1) and (3) were carried out with the same groups of students, and are discussed first below. hypertext experiment the exercises here were based on hypercard. the first required the retrieval of factual information from a disk produced at drexel university in the usa, which described the university and its facilities. the second involved a simple customization exercise to produce a daily diary. an initial pilot study was carried out with six students to give some estimate of the likely minimum teaching time for the exercise, and how it should be organized. first study the first full study involved 41 students who had no previous experience of hypertext. two-thirds were 20 years old or less, four-fifths were female, and most had at least three years of computing experience. for teaching purposes, the students were randomly assigned to one of 10 groups, each consisting of three to four students. (the number per group was determined by the number of machines available in the laboratory. however, since more machines were on order for the following year, this offered the chance of experimenting with different-sized groups.) the student in the group who did the inputting was changed at specified intervals, and the groups were instructed to discuss each step. the groups were separated into two sets, each of five groups. the first set was given a short introduction to hypertext (specifically hypercard), and then went hands-on immediately. the second set was exposed to a much more extensive demonstration and discussion of hypertext and hypercard before going hands-on. all groups were given the same hand-outs, covering the background to hypertext, to the specifics of hypercard, and to the exercises. in the drexel disk experiment, note was taken of the time required to retrieve the information and the number of false steps involved. students were also asked to complete questionnaires regarding any problems they had encountered. similarly, in the diary 36 ait-j volume 3 number 2 customization exercise, note was taken of how far each group had progressed in the time available, and students were again asked to record any problems encountered. no significant differences were found between the two sets in terms of time required. the same was true of most on-screen problems (for example, interpretation of menu options, or of icons). however, there were significant differences in terms of moving between screens, or (in the second exercise) of transferring items. for the drexel disk, the less wellprepared set recorded twice as many problems in this category, and for the diary exercise, 50% more as compared with the better-prepared set. for students who had not encountered a mouse before, the physical handling of a mouse proved to be a major (though rapidly surmountable) problem, regardless of the set. second study changes were next made in the organization of the teaching and the hand-out materials in the light of student feedback from the first study. the customization exercise was simplified; the questions on information retrieval, which had previously been in random order, were re-ordered in terms of hardness (as perceived by the students); and a hand-out was prepared regarding use of the mouse. students had indicated that three to four participants per group was too many, and that differences in computing experience were particularly irritating when discussing responses. for the repeat study, two thirds of the students were grouped into pairs with similar computing backgrounds; the remainder worked on their own. there had been adverse comments on the duration of both the long and short introductions to hypertext and hypercard; in the revised exercise, all students were given the same intermediate-length introduction before going hands-on. table i: groupings used in the second study set characteristics a female; similar ages; familiar with macintosh b female; range of ages; not familiar with macintosh c male; similar ages; n o t familiar with macintosh d male and female; range of ages; familiar with macintosh the teaching exercise was now repeated with the next cohort of second-year students 45 altogether who had again not had prior exposure to hypertext. in this case, a half were 20 years old or less and two-thirds were female rather different from the first group but, again, most had three years or more of computing experience. this time round, the students were divided into four sets. in contrast to the first study, these sets were selected to have certain group characteristics in common, as listed in table i. the first study had suggested that computing experience, particularly in the use of apple macintoshes, was the most important factor relating to the accuracy and speed with which the hypercard exercises were carried out. membership of these four sets was therefore determined on the basis of familiarity with macintoshes, in order to test how significant this effect was. it was decided to investigate, at the same time, how useful the hand-outs were. sets a and d were provided with detailed hand-outs, but were expected to work in the laboratory mainly on their own. sets b and c were given condensed hand-outs, but could readily call on oral advice in the laboratory. 37 ck. ramaiah et a\ iterative student-based testing of automated information-handling exercises very useful fairly useful not very useful assessment of hand-outs medium-length hand-outs 20% 66% 14% first study detailed hand-outs 35% 57% 9% second study condensed hand-outs 9% 59% 32% table 2: evaluation of band-out maten'al in general, respondents in the second study indicated an appreciably higher level of satisfaction with the exercises than those in the first study. in terms of ability to cope with the exercises, there were significant differences between sets a and d, on the one hand, and sets b and c, on the other. though, overall, problems in handling a mouse were reduced compared with the first study, the level was significantly higher for sets b and c than for a and d. there was a smaller, but still clear-cut difference between these sets in terms of problems with interpreting icons: sets b and c also found this more difficult. in addition, sets b and c encountered more problems in moving between screens and in using paint tools. the level of difference can be illustrated by the time taken by students to complete the customization exercise. they were allotted 90 minutes of class time for this (otherwise the project had to be completed in their own time). almost all the students in set d (which contained those with most computer experience) completed the exercise in this time, as did over a third of the students in set a. only a quarter of the students in sets b and c finished within the time limit. by way of contrast, the responses from all sets indicated that working in pairs was generally preferred to working alone. students were asked to evaluate the hand-out material they were given. their responses to the question of content are given in table 2. discussion of these responses with the students indicated that the provision of detailed hand-outs was regarded as desirable by all groups, regardless of computer experience or of the amount of other assistance that was available. the revised introduction to the course received a higher rating than either the long or short introductions used in the first study. electronic bulletin board experiment an initial study was also made here in order to obtain a general view of how students reacted to different kinds of electronic bulletin board, and what problems they found in using them. to this end, 47 students were asked to access and report on four readily available bulletin boards. the method was to ask them to respond to a series of questions which could be answered via the information on each bulletin board. one of the bulletin boards was bubl (the bulletin board for librarians). it was decided to concentrate on this, paying particular attention to the section that contained practical exercises designed to provide training in the use of online services available over janet. the question at issue was how well the students felt that this section of the bulletin board had been laid out for training purposes. in studying this question, the students obtained practice in the 38 alt-) volume 3 number 2 evaluation of bulletin boards while experimenting with online training. the bubl administrators had agreed to try and modify this section in the light of student feedback. two successive studies were therefore planned. the first cohort of students would carry out their assessment of the bulletin board. the section concerned would be modified on the basis of their comments, and then reassessed by a second cohort of students. first stud/ the first cohort consisted of 41 students, of whom some two-thirds were female and threequarters under 25; a half had more than three years of computing experience, and a half had accessed electronic bulletin boards before. the students were divided into four sets, depending on their previous computing experience. each set had four sessions devoted to bulletin boards. in the first two, the students worked in pairs and received a general introduction to electronic bulletin boards. in the next two, they had the opportunity to work individually on bubl. during the latter sessions, the students answered (by electronic mail to the coordinator) a series of questions relating to the layout and design of the interface, ease of absorbing the contents, and ease of navigation. the students' responses indicated that they encountered most problems in the area of layout and design. for example, they noted a need for better spacing on-screen between instructions and other types of text, a need to use highlighting (or some similar method) for distinguishing between different types of heading, and a need to differentiate between levels of difficulty in the practical exercises (for example, basic, intermediate and advanced). second study changes were made to bubl during the intervening summer vacation. then the material was re-evaluated by a new cohort of 54 students. their characteristics were very similar to those of the first group except that, owing to a change in the pattern of teaching, only some 15% had previously accessed bulletin boards. the teaching methods and questions asked were the same as in the previous year. a comparison of the comments from the first and second studies indicates that both cohorts were looking for similar factors when using the bulletin board. however, the respondents in the second study clearly preferred the revised layout and design of the interface. for example, the proportion of students who suggested the need for changes in the training section to make it more user-friendly fell from 84% to 27%. at the same time, student responses on the second time round also implied that the overall ease of use of the system had decreased. further feedback was sought on this apparently contradictory result: it was found that the problem lay in a major increase in the amount of information available via bubl from the first to the second year (this was reflected in the fact that the main menu contained 15 items in the first year and 25 in the second). in consequence, though the material was better presented on-screen in the second year, it often took more time and effort to track down any particular item of information. there were no systematic differences, in this case, relating to the students' computer experience. conclusions the participants in these experiments formed a homogeneous group in the sense that they were second-year students in the same department. however, they had both a range of computing backgrounds and widely differing levels of interest in computer-based systems. from this viewpoint, they can be seen as a reasonably typical student group. the fact that 39 ck. ramaiah et al iterative student-based testing of automated information-handling exercises the types of problem they encountered in these experiments appeared to be common to all suggests that, regardless of experience and motivation, the whole group had a similar mental model of what they were trying to do. indeed, from their responses, it was clear that they envisaged the computer as primarily an information provider. in terms of personal characteristics, there was no correlation between student responses and sex or age. there was a slight suggestion in the first investigation that students with a science background experienced rather fewer problems than those with a humanities background, but this was submerged by differences in computer background. experience in using a macintosh was clearly a factor, to the extent that it made sense to teach hypercard to macintosh users and non-macintosh users separately. by way of contrast, using and evaluating the training section of bubl was not significantly influenced by computer experience. the reason appears to be that, although some students had had prior experience with bulletin boards, none had previously been involved with online training. prior computer experience in the first investigation actually did not affect the range of problems that a student encountered: it did affect the number of errors made in carrying out the exercises. it is useful here to draw a distinction between two types of error: 'mistakes' and 'slips' (norman, 1983). computer users begin with some kind of intention when they start using a system. a 'mistake' occurs when there is some error or deficiency in this intention. a different type of error occurs when the intention is correctly formulated, but something goes wrong in the attempt to implement it. such an error can be referred to as a 'slip'. in the studies reported here, where conditions are fairly well defined, novice and experienced computer users tend to make similar 'mistakes', but the former make considerably more 'slips'. in working on these exercises, the student preference was clearly for operating in pairs, with both members having the same computer background. whatever their level of computer experience, students preferred to have their hand-outs as detailed as possible. at the same time, they preferred the introductory lectures and demonstrations to be of moderate length, simply providing a framework which would allow them to appreciate the overall nature of an activity before they went hands-on. students who participated in the second experiments were not told what modifications had been introduced as a result of feedback from the first experiments. their responses indicate that the modifications generally improved the acceptability of the teaching/ learning interface involved. so far as the present type of information-acquiring and information-evaluating activities are concerned, it appears that iterative feedback can improve computer-based teaching exercises. this result seems to apply to all three levels of task distinguished in our introduction: simple, complex and design. references joyce, b.r. and weil, m. (1980), models of teaching, englewood cliffs nj, prenticehall. moran, t.p. (1981), 'the command language grammar: a representative computing system', international journal of man-machine studies, 15, 3-50. norman, d.a. (1983), 'design rules based on analysis of human error', communications of the acm, 26, 254-8. 40 alt-j volume 3 number 2 saloman, g. (1979), intuition of media, cognition and learning, san francisco, josseybass. shneiderman, b. (ed) (1993), sparks of innovation in human-computer interaction, norwood nj, ablex. 41 alt-j, research in learning technology vol. 16, no. 3, september 2008, 207–220 issn 0968-7769 print/issn 1741-1629 online © 2008 association for learning technology doi: 10.1080/09687760802526723 http://www.informaworld.com machinima interventions: innovative approaches to immersive virtual world curriculum integration andrew john middleton* and richard mather learning and teaching institute, sheffield hallam university, sheffield, uk taylor and francis ltdcalt_a_352840.sgm (received 1 april 2008; final version received 8 september 2008) 10.1080/09687760802526723alt-j, research in learning technology1466-4658 (print)/1743-7903 (online)original article2008taylor & francis163000000september 2008mr andrewmiddletona.j.middleton@shu.ac.uk the educational value of immersive virtual worlds (ivws) seems to be in their social immersive qualities and as an accessible simulation technology. in contrast to these synchronous applications this paper discusses the use of educational machinima developed in ivw virtual film sets. it also introduces the concept of media intervention, proposing that digital media works best when simply developed for deployment within a blended curriculum to inform learning activity, and where the media are specifically designed to set challenges, seed ideas, or illustrate problems. machinima, digital films created in ivws, or digital games offer a rich mechanism for delivering such interventions. scenes are storyboarded, constructed, shot and edited using techniques similar to professional film production, drawing upon a cast of virtual world avatars controlled through a human–computer interface, rather than showing real-life actors. the approach enables academics or students to make films using screen capture software and desktop editing tools. in student-generated production models the learning value may be found in the production process itself. this paper discusses six case studies and several themes from research on ideas for educational machinima including: access to production; creativity in teaching and learning; media intervention methodology; production models; reusability; visualisation and simulation. keywords: digital media; immersive virtual worlds; machinima; simulation; media interventions introduction immersive virtual worlds (ivws) have attracted the attention of educators over recent years, especially since the release of second life, one of the most popular of several internet-based virtual worlds (littleton and bayne 2008). the value and attraction of ivws to education is in the ease they promise the general educational user for creating simulations in an immersive, abstracted space, offering students the opportunity to synchronously encounter authentic experiences involving other people, objects and environments. interest in simulation is growing amongst educators who believe experiential learning has a role in engaging students (aldrich 2005; dickey 2005; de freitas 2006) and that the visualisation of ideas accommodates a range of learning styles and teaching strategies (burbules 1999). the opportunities offered by ivws are counterbalanced by barriers to access including: lack of academic technical literacy and commitment to learn new technology; academic inertia; the threat to academic freedom and autonomy; usability; reliability; affordability; *corresponding author. email: a.j.middleton@shu.ac.uk 208 a.j. middleton and r. mather compatibility; and support requirements. will academics risk hours of their time in developing teaching on the chance that the technology enhances the curriculum (roberts 2008; collis and moonen 2002; shephard 2003)? the authors have previously explored the potential of digital media in the context of student-centred pedagogies and this has resulted in various ‘media intervention’ curriculum design methodologies, including audio feedback, educational podcasting models, studentgenerated video and digital storytelling. in this development media intervention techniques are connected with ivws through the process of making digital video learning objects on ivw virtual film sets. the collaborative design of films made in ivws, known as machinima, proved to be a rewarding exercise, leading to the generation of six storyboards and a better understanding of the benefits, challenges and potential of educational machinima. this paper presents six short case studies of design conversations with academics, representing a range of disciplines, towards finding more widespread applications for ivws in engaging learners in higher education curricula. background lombardi (2007) notes that developing authentic learning activities enriches learning and that such activities need not be limited to real-life locations and practice, as authenticity can also be found through the design of blended learning environments, which yoon and lim (2007, 475) describe as “a purposeful mix of delivery media”. the concept of media intervention (bradley, middleton, and nunnington 2006) proposes such a mix. media intervention methodology also encourages wider consideration of locally developed digital media in higher education by reducing the barriers created by technical processes that are perceived to be overly complicated (collis and moonen 2002). instead, it encourages academics to view digital media objects as simple resources deployed with the intention of promoting active, student-centred learning designed to set challenges, seed ideas, or illustrate problems so as to quickly engage and affect the learner. media interventions may be rapidly deployed in response to emerging needs, or as student responses to emerging knowledge. their successful deployment, however, is dependent upon ease of production and integration. media intervention can encompass the re-use of existing media objects as well as the production of new objects by staff, students, technicians, or external suppliers. the value of the media is found in its currency, authenticity, specificity, repeatability and occasionally in its constructionist value (papert 1986; kafai 1996), rather than in its production value. the asynchronous nature of these media objects heightens their potential learning impact by letting the student user determine their timely application (middleton 2008). this results in digital media moving away from transmission and towards conversation (laurillard 2002). in the age of user-generated grass roots video and podcasting (horizon project 2008), digital media become viable channels in extending the virtual learning environment and, through media intervention, promote change in the teacher’s role towards that of guide, seeder of ideas, and motivator. ivws, such as second life, allow the user to create props and scenery and, if necessary, program them with behaviours. such objects can be small (e.g. buttons on a mobile phone) or huge (e.g. office blocks). machinima have been described as “animated filmmaking within a real-time virtual 3d environment” delivered as digital video (daly-swanson 2007). machinima production first emerged in 2001 in the gaming community and has since been featured in film festivals and alt-j, research in learning technology 209 other public presentations. films are made by orchestrating and recording avatars and objects in real-time. the avatars, animated 3d representations of people or other beings, are manipulated as virtual puppets (lowood 2006) by real people through computer interfaces. a film with a large cast may therefore require several puppeteers (nitsche 2005). machinima recording involves screen-capture, with the resultant video footage edited using desktop software. sound-tracks and effects can also be applied at this stage. finally the films are exported to a digital video file. machinima production follows the same procedures, and uses similar techniques, as real-world filmmaking: scenes are storyboarded, shot, edited and processed. this can be brief, requiring just one or two people, or extensive. films can be sophisticated productions or, at their simplest, can be rudimentary cctv-type recordings of whatever happened whilst ‘the camera’ was switched on (call 2005). ivws bring the film set and the cast to the producer and this can make it attractive to educationalists where a lack of access to budgets, people, locations and time contribute to the inaccessibility of educational video production (bardzell et al. 2006). for example, second life is a widely accessible online environment that supports multi-user engagement. in addition it can be used for free and provides tools to enable the user construction of interactive objects. users can also import images to speed up set development. compared to real-world filmmaking, machinima production is cost-effective, timeefficient and offers producers a large amount of creative control (academy of machinima arts & sciences website 2005). machinima and simulation, used together, therefore offer functionality to academic innovators that until now has been out of reach and so creatively unexplored. recent interest in second life from educators (kirriemuir 2007) suggests that although ivws promise many opportunities for synchronous interaction and learning, machinima production results in objects that have an asynchronous value. machinima can be produced using various 3d environments. however, bardzell et al. (2006), in an evaluation of machinima platforms, conclude that second life (as an example of their ‘hybrid games’ category) offers a range of functionality that makes it better suited than either ‘pure games’ or ‘pure machinima’ platforms for machinima development. games such as the sims and the movies do not offer the freedom to customise sets, props and camera angles that can be found in second life, and machinima platforms such as moviestorm (http://www.moviestorm.co.uk), virtual stage (http://www.dakinewave.com/ virtualstage/index.htm) and machinimation (http://www.fountainheadent.com/n.x/fe/ home/productions/machinimation) are promising, though complex or too limited to support the diversity of educational ideas as discussed here. as ivws become more familiar to students, they may become widely accepted as creative production spaces for student group work. filmmaking provides a useful framework in which to set a collaborative assignment and, as a learning space, may encourage participants to apply a range of interpersonal and organisational skills. its narrative nature connects to emerging interest in educational digital storytelling (jenkins and lonsdale 2007) and other techniques that promote the student voice. carr (2007) describes how the us navy is capturing simulation activity for review, finding it to be safe and depersonalised and produced “at a fraction of traditional audio/video production costs”. the second health project (second health, website and online machinima at http://secondhealth.wordpress.com) has created a second life presence and a series of machinima, allowing service users to experience proposals for london healthcare provision. machinima production is also used in schools to encourage student autonomy and engagement (czarnecki 2007). elsewhere large groups are offered diverse experience and 210 a.j. middleton and r. mather skill development through the creation of sets, costumes, stunts and special effects that, otherwise, would not be possible (joseph 2007). methodology the aim of this ongoing work is to discover whether machinima produced in ivws can be used to engage learners across the higher education curriculum. a concept of machinima media intervention has been developed and used to underpin this research. outline information about ivws, machinima and media interventions was included in an e-mail invitation to selected academics who were also informed that they had been specifically chosen to contribute on the basis that they had demonstrated an interest in academic innovation in previous work. this enabled the current research to acknowledge and build upon prior success strategies employed by those taking part. together they represented a wide range of disciplines. semi-structured interviews with academic participants were carried out to create case studies around a collaborative task of completing a design specification and storyboard for the production of machinima. the interview method may be more usefully described, therefore, as design conversations. this approach was devised to engender a creative attitude amongst participants and to encourage consideration of the pedagogic opportunities offered by media interventions. it was also intended to dissuade participants from dwelling on the technical barriers, real or imagined. six academics agreed to participate. some had heard of second life, but none had used it or were familiar with the machinima or media intervention concepts. the invitation explained that part of the design sessions would include an induction to second life, ivws in general, machinima and the media intervention concept. the first 20 minutes in each conversation were used to introduce the various concepts and technologies. this was followed by a demonstration of between five and 10 short machinima (appendix 1), each selected to illustrate facets that could be used to inform the design process. first, a short machinima with a strong narrative and high cinematic value was shown. this was followed by short youtube (http://www.youtube.com) video clips that described second life, methods of constructing objects, collaborative activity and educational applications for ivws and machinima. using youtube clips to introduce concepts was a useful technique and served to demonstrate how media can generate discussion. following familiarisation, participants were asked to design a machinima in collaboration with the authors. informed and inspired by the induction, design conversations focused on the completion of a design specification proforma. discussion of wider issues was also encouraged. each conversation was limited to two hours, which provided a useful design constraint. the academic participant was joined by both authors: one who completed the specification, clarifying the design idea through its documentation; the other responsible for encouraging creativity, reviewing understanding of the related concepts used to inform the design, and ensuring that the conversation covered the intended scope of the research. data were captured on the proforma and in audio recordings of the conversations. six common themes emerged from analysis of the full transcripts, as discussed in the results section. case studies results analysis of transcripts of the design conversations resulted in the identification of six themes. alt-j, research in learning technology 211 table 1. case study descriptions. 1. psychology module: forensic psychology participant: confident user of ict; enthusiastic about trying new technologies if the pedagogical benefits are clear; some knowledge of ivws. machinima idea: a car crash involving two vehicles, seen by two witnesses, one of whom has an obstructed view. the crash is shown from various angles, and may include details and commentaries. more detail is released to students over time, in increasingly longer video clips, as the module progresses. application: used in lecture to stimulate thinking and discussion, and to challenge student perceptions. students could be invited to manipulate the scene themselves. intended impact: perceptions challenged; familiarity with perception theories; theories of bystander intervention and eyewitness testimony. reusability: other subject areas (e.g. criminology, sociology, transport design, urban planning, physics, engineering). other ideas: global real estate and architectural projects that cover a range of global perspectives, with different people in different locations. 2. sport module: gymnastics, exercise prescription participant: confident user of ict; enthusiastic about trying new technologies if the pedagogical benefits are clear; has innovated with digital media; some knowledge of ivws. machinima idea: students populate a pre-designed virtual gymnasium with pre-designed equipment recording their layouts as short video clips; commentary to capture their reasoning. application: as a formative assessment. intended impact: knowledge of how to set up an exercise environment; exploring an environment in which they can experiment. reusability: student-submitted machinima added to a library of examples for future use. the fitness space and equipment could be used in a number of other ways. other ideas: setting up a biomechanical experiment; instructional videos of how to use equipment with voiceovers in different languages; setting up of fitness activities for specific scenarios, e.g. training elite athletes or rehabilitation from an injury. simulating scenarios which might be ethically problematic, e.g. interviewing children about obesity; rehabilitation. 3. business and technology module: computing foundation degree participant: confident user of ict; enthusiastic about trying new technologies if the pedagogical benefits are clear; some awareness of ivws; innovates in interdisciplinary teaching and blended learning. machinima idea: an avatar, assuming the role of managing director of a major company, gives a briefing for a group of students. application: used in a real-world group assignment briefing supporting an educational simulation. the machinima clip is used as a surrogate managing director. intended impact: professional development. 212 a.j. middleton and r. mather table 1. (continued). reusability: re-use by replacing avatar voiceovers in the machinima; across disciplines where insight to boardroom activity is useful. other ideas: creation of built environments (e.g. an office location), which students have to fit out, recording their solutions as machinima. alternatively the students could be given existing video clips of the space to which they have to add captions, text descriptions, or voiceover. 4. criminology module: criminal justice systems participant: enthusiastic about blended learning; basic ict skills and a little intimidated by technology and the time it takes to learn new skills. machinima idea: depiction of a crime, a subsequent arrest, and resultant police interview. application: a stand-alone video clip describing a common crime and the processes of the criminal justice system (cjs). eventually replacing lectures by encouraging student engagement in other ways. intended impact: knowledge of the cjs; ability to discuss an offender’s journey through the cjs in relation to the various agencies; awareness of the strengths and limitations of the cjs. reusability: re-use at different levels of study. other ideas: a series representing the whole criminal justice system. 5. teacher education module: pgce learning and skills sector participant: experienced and confident user of ict; enthusiastic about innovation in teaching and learning; some knowledge of virtual worlds including one used for language learning. machinima idea: classroom behaviour management – short films of classroom scenarios, including disruptive students and how to deal with them. a teacher begins a lesson by introducing a task. as the students begin to engage, one of them becomes unruly. the machinima would use recordings from real classrooms. application: in class to promote discussion; as a reference for education students who are out on teaching practice. intended impact: increased confidence and skills in classroom management. reusability: re-use of locations for all levels of teacher education. other ideas: education policy; modelling of classroom-based research activity. 6. microbiology module: pathogenesis of infectious disease participant: experienced in the use of ict and blended learning; enthusiastic about teaching with a wide range of media; some knowledge of virtual worlds. machinima idea: interaction between cells – the ‘battle’ between bacteria with virulence factors and the human defence system. this is usually very difficult to visualise. application: machinima available in the vle or used in taught sessions. intended impact: deeper understanding of the interaction between bacteria and the human defence system. alt-j, research in learning technology 213 1. access to the means of production and other benefits of machinima production when participants were asked why machinima may be more useful than other approaches, several explained how developing their ideas using real-life footage would require high budgets and access to a crew and locations. in the business scenario the academic explained that a new office construction could be a fantastic learning resource, but the building’s state and site personnel change continuously. this project could become cost-effective by developing an ivw simulation and capturing interactions as machinima. lack of access to physical locations contributed to ruling out video as an option in five cases and in the sixth, psychology, filming a car crash was not feasible. access to professional production facilities is also problematic for academics and students alike, though many universities are developing student-generated media project-based learning initiatives (bramhall, radley, and metcalf 2008; hung, keppell, and jong 2004). the sport academic explained that, “time is pretty expensive”. this concern was shared by others who agreed that finding time to carry out work with traditional media is problematic and that productions involving other people introduce dependencies and inflexibilities. virtual world filming typically requires less crew and cast and can fit alongside other commitments more easily. time pressures around the use of physical space and equipment can distort the quality of learning and its assessment, according to the sport tutor who described how students are expected to lay out the gym in a specific time slot for an assessment on decision-making about spatial solutions. if they make a mistake they have little opportunity to rectify it. in an ivw students can prepare in their own time and produce commented machinima of their decision-making process. it becomes easier for them to try different configurations of equipment, identify hazards and optimise their solutions. the real-world activity required unnecessary physical effort, and focused assessment on the product rather than on the decision-making process, whereas the commented machinima would capture everything. finally, machinima production techniques offer access to tools that allow rapid, rich media responses. if a set is already established the production of a visual response is within the grasp of both students and academics, and this can be useful in offering visual feedback, especially where digital media-enhanced pedagogies are already used. the business tutor appreciated that student groups would respond differently to the machinima of the managing director’s briefing and noted the potential to generate further machinima in response to their initial submissions. 2. creativity in teaching and learning all six participants readily engaged with the design methodology and enjoyed the opportunity to work creatively. they demonstrated familiarity with the grammar of film, quickly accepting suggestions, and proposing their own. they introduced visual techniques, such as mixing real and virtual footage. in the education case study, for example, the value of a table 1. (continued). reusability: the film could be used for other subject areas, e.g. nursing where there is a need for more basic understanding of microbiology. some of the modelled components (bacteria, cells, etc) could be re-used in other films. other ideas: none. participant was not keen on the idea of virtual lab machinima. 214 a.j. middleton and r. mather real-world audio track demonstrating classroom disruption mixed with ivw footage was seen as useful. the case studies each highlight one of several ideas that were generated quickly. often many and varied ideas were explored. the conversational approach, anchored to the design proforma, was effective in generating and refining ideas. 3. the application of media intervention methodology participants readily understood and valued the concept of media intervention, where digital media is used as a “catalyst for learning” (psychology academic), though all thought in terms of series, or sets of machinima rather than individual objects. in microbiology the academic believed machinima “interspersed in a lecture” would engage and interest students more than current methods and would be more “stimulating and exciting”. classroom disruption is a difficult topic to address; techniques such as role play often fail to capture the chaos. the education academic felt that media interventions depicting difficult situations could generate empathy and help students to predict such behaviour whilst on placement. they could serve as useful reference points when communicating their own placement experience to others and as discussion points in group decision-making. the lecturer, recognising the catalytic potential of media said, “you don’t need to look for solutions necessarily within the media itself”. 4. models for the generation of educational machinima none of the academics interviewed proposed student-produced machinima ideas without prompting. however, once the potential had been introduced, it generally found favour. this response from the criminology academic was typical: that’s really challenging me now. well it would be a good test of their learning. it would be a good learning experience, because i’m saying you need to get all those cues in. well that would be a really, really hard thing for them to do. it would be something i’d have to think about. they’d need to show their learning in terms of, not the technical construction, but in terms of have they really understood what the role of the crown prosecution service is. well it wouldn’t just be through having a prosecutor and a pile of files there. it would be about showing that they’d made the decision about the evidential test and the public interest test, and that they showed their independence from the police, and they were thinking forward to how the court process would go. so yes, if they showed that in the machinima then i would say that’s a demonstration of the conceptual knowledge i want them to have as criminology students. yes. ooh, that’s pushed me a bit! though the research was primarily interested in the generation of ideas, it was inevitable that the practicalities of production would be raised. none of the academics saw it as their role to construct the machinima themselves, leaving three options: ● a team of learning technologists; ● their own students; and ● technical students (e.g. from computing courses). several variations emerged for each of these. the basis for engaging students was particularly interesting, with options ranging from paid employment or cv recognition for the production of machinima products, through inter-disciplinary learning collaborations, to students capturing the process of their own learning in ivws. in microbiology, for example, alt-j, research in learning technology 215 it was suggested that the students could be the content experts briefing an authentic project for visualisation students and that this would provide a rare opportunity for authentic interaction in the subject with students learning how to articulate ideas to laypeople. the psychology academic commented that for ivws to become accessible to all students as production environments, they would first have to feel “ordinary”: i don’t want my students to spend a fortnight trying to work out how to use the software and a day trying to put the psychology into it. there was some concern that the process might become a distraction and be time consuming. following a similar experience of working with student producers, one academic noted that the supervision role had been arduous. the sport lecturer wondered whether all students would be comfortable operating in such environments, although reflected on other student-generated media assignments that had worked for him following half-hour induction sessions. he thought that a library of student-generated machinima could become useful over time. rapid student machinima production (i.e. simple action capture) offered opportunities for education and sport where machinima could drive class discussion. on several occasions it was noted that the design process (e.g. discussion, script-writing, planning), rather than the filming process itself, could provide great value for collaborative learning. 5. reusability many of the respondents recognised that the reusability of the virtual film set itself was more useful to them than the resultant machinima. in the sport case study, for example, there is no need for each student to build a gym and its various equipment; their task is to lay out and describe the use of the space. in the microbiology example the lecturer explained: once you’ve produced a typical bacteria with its virulence factors … you could cut and paste different virulence factors onto the basic organism, and make it slightly different. reusability may also be found by offering students machinima stems that they are tasked with completing by introducing variables (e.g. decisions, viewpoints) and conclusions. this approach arose in the criminology, sport, psychology and microbiology discussions. in sport and microbiology it was suggested that students could contribute to the creation of commented machinima content libraries, adding a degree of authenticity to the assignment, which could be used by current and future students. 6. visualisation and simulation virtual worlds are visually immersive environments capable of supporting independent and social exploration. visualisation is a key aspect of all the cases designed in this machinima study: in offering multiple perspectives of an event; in making records of a plan; through observing artefacts and evidence in an environment; in representing and developing the characters of protagonists; and in representing concepts visually. in addition, machinima can support the visualisation of relationships, processes and activities, including those that are inaccessible to students in the real world. the verisimilitude may be limited, for example in the unrealistic movement of avatars, but all participants still valued the opportunity to visually represent their ideas. in microbiology 216 a.j. middleton and r. mather the tutor explained, “microscopes can only look at living organisms at a very limited magnification of little spots … very two dimensional and not very interesting at that scale”. discussion at this early stage the results are encouraging, especially with respect to the attitude of those who took part. participants demonstrated that they were able to think creatively and cinematically, visualising how one shot would follow another. the specification and storyboarding of ideas proved to be a valuable and enjoyable exercise. participants were initially asked to develop one idea, though often many ideas were generated in the design conversations. even once the topic had been chosen, all participants were keen to keep hold of related ideas, noting that once the set has been dressed more use should be made of it. discussion about simplified production and access initially suggested that educational machinima production may be within the grasp of academic producers, enabling end-user independence and flexibility, though this may be inhibited where ideas are dependent upon large casts of avatar operators. however, it is disconcerting to note that, even amongst the selected innovators, there was clearly a reticence to imagine themselves creating their own machinima, even where the benefits had been enthusiastically discussed. this may be because they have never considered themselves as media producers before. nevertheless, it indicates that the perceived barriers to such useful technologies are still too high for most. inter-disciplinary production, drawing upon and developing different areas of student expertise, was an idea that emerged as a production solution, though again comes with risks and an organisational overhead. 3d visualisation, even when captured in 2d digital films, is an attractive quality of machinima to some academics; finding different viewpoints for an event or experiencing microscopic phenomena in-the-round cannot be easily achieved by other means. the name second life indicates the notion of abstracted life, but the abstraction of rendered life in ivws requires an acceptable degree of verisimilitude and fidelity so that users can make sense of, and believe in, what is happening (boyle 1997). on the other hand aldrich (2005, 81) notes how “simulation elements selectively represent objects or situations”, and perhaps this indicates how these ivw media objects should be understood too. this abstraction, and its capture in machinima, appear to be acceptable in the cases discussed here, but may compromise other applications. the topic of machinima reusability resulted in the focus shifting from reusable films towards reusable film sets, scenery and props. by establishing such material collections in the in-world ivw libraries, the technical and administrative barriers can be lowered: series or sets of machinima in related and unrelated disciplines become much more likely if such objects are already available. this priming activity may involve buying in properties, expert technical contributions, or extra-curricular opportunities for students. priming approaches such as these would also help to address perceptions of technical barriers and the need for academics or their students to find time to develop their set and object construction skills. the authors have subsequently developed some of the scenarios described here with a minimal budget for props. where it has not been possible to make items, free objects have been found in the second life library, images have been uploaded as scenery textures, free items have been found elsewhere, and occasionally items have been purchased from in-world stores (e.g. a suit for the managing director in case study 3). once acquired, such props have been re-used. reusability should be considered with any content production. machinima are selfcontained, highly granular, digital videos and, technically, would appear to be highly reusable. alt-j, research in learning technology 217 pedagogically their reusability may be limited by over-contextualisation, for example where content is targeted at a particular subject, educational level (currier and campbell 2002) or by particular cultural and pedagogical approaches. designing reusability into any learning content can be difficult (collis and strijker 2004). machinima objects enable students to view them more than once, store and share them, modify them, and access them on a range of devices. the idea of reusable machinima stems, requiring completion by students, may offer opportunities for constructionist pedagogies. the machinima designs developed in this study were intended to be around three minutes long, recognising that high-impact objects are more likely to affect learning. brief media objects can be valuable informing components in an active learning landscape, whilst longer videos require a different type of learner engagement. beyond the case studies discussed here, there remains potential for other applications. for example, none of the participants proposed the use of machinima as a medium for reflective digital storytelling (jenkins and lonsdale 2007), though the linear nature of film is suited to such student narratives. machinima could be used to present a visual identity to a real-world voice, as in the hypothetical case of child testimony that would otherwise not be ethically acceptable. conclusion the scenarios considered here suggest that ivw machinima can find value across the curriculum, with many of the benefits being widely applicable. the six themes discussed here are by no means exhaustive; for example, understandings about learning through a captured, abstracted environment could be developed. the idea of machinima media intervention was readily accepted, resonating with awareness of other methods of intervention to promote student-centred learning. this simple use of media indicates how machinima production may be technically feasible, though development models for educational machinima are still required and suggest the need for further research. references academy of machinima arts & sciences website. 2005. the machinima faq. http://www.machinima. org/machinima-faq.html. aldrich, c. 2005. learning by doing: a comprehensive guide to simulations, computer games, and pedagogy in e-learning and other educational experiences. san francisco: pfeiffer; chichester: john wiley. bardzell, j., s. bardzell, c. briggs, k. makice, w. ryan, and m. weldon. 2006. machinima prototyping: an approach to evaluation. proceedings of the 4th nordic conference on humancomputer interaction: changing roles, vol. 189, 433–36, oslo, norway. bradley, s., a. middleton, and n. nunnington. 2006. media interventions: designing video and audio to stimulate learning activity. paper presented at diverse (developing innovative video resources for students everywhere) conference: enriching e-learning using video and multimedia (both face-to-face and at a distance), glasgow. bramhall, m., k. radley, and j. metcalf. 2008. users as producers: students using video to develop learner autonomy. proceedings of engineering education 2008, international conference on innovation, good practice and research in education, 14–16 july 2008, at loughborough university, uk. burbules, n. 1999. technology in education: who, where, when, what & why? international journal of educational technology 1, no. 1. http://www.ed.uiuc.edu/ijet/v1n1/v1n1feature.html call, e. 2005. making machinima in second life. http://static.secondlife.com/_files/making_ machinima.pdf carr, d. 2007. machinima and education. http://www.futurelab.org.uk/resources/publications_ reports_articles/web_articles/web_article794. 218 a.j. middleton and r. mather collis, b., and j. moonen. 2002. flexible learning in a digital world: experiences and expectations. london: kogan page. collis, b., and a. strijker. 2004. technology and human issues in reusing learning objects. journal of interactive media in education 4, special issue on the educational semantic web. http:// www-jime.open.ac.uk/2004/4/collis-2004-4-disc-t.html. currier, s., and l. campbell, l. 2002. evaluating learning resources for reusability: the dner & learning objects study, in winds of changing in the sea of learning, proceedings of the 19th annual conference of the australian society for computers in tertiary education (ascilite), auckland, new zealand, 8–11 december 2002, ed. andy williamson, cathy gunn, alison young and tony clear. auckland, new zealand: unitec institute of technology. czarnecki, k. 2007. library services on the teen grid. in proceedings, second life education workshop 2007, part of the second life community convention, at the chicago hilton, august 24–26, ed. d. livingstone, and j. kemp. http://www.simteach.com/slccedu07 proceedings.pdf. daly-swanson, b. 2007. second life machinima for libraries: the intersection of instruction, outreach and marketing in a virtual world. paper presented at the world library and information congress, 73rd ifla general conference and council, august 19–23, in durban, south africa. http://www.ifla.org/iv/ifla73/papers/133-dalyswanson-en.pdf. de freitas, s. 2006. using games and simulations for supporting learning. learning, media and technology 31, no. 4: 343–58. dickey, m. 2005. three-dimensional virtual worlds and distance learning: two case studies of active worlds as a medium for distance education. british journal of educational technology 36, no. 3: 439–51. horizon project. 2008. the horizon report 2008 edition a collaboration between the new media consortium and the educause learning initiative. http://www.nmc.org/pdf/2008-horizonreport.pdf. hung, v.h.k., m. keppell, and m.s.y. jong. 2004. learners as producers: using project based learning to enhance meaningful learning through digital video production. in beyond the comfort zone: proceedings of the 21st ascilite conference, december 5–8, in perth, ed. r. atkinson, c. mcbeath, d. jonas-dwyer, and r. phillips, 428–36. http://www.ascilite.org.au/conferences/ perth04/procs/hung.html. jenkins, m., and j. lonsdale. 2007. evaluating the effectiveness of digital storytelling for student reflection. in ict: providing choices for learners and learning. proceedings ascilite singapore 2007. http://www.ascilite.org.au/conferences/singapore07/procs/jenkins.pdf. joseph, b. 2007. global kids, inc.’s best practices in using virtual worlds for education. in proceedings, second life education workshop 2007, part of the second life community convention, at the chicago hilton, august 24–26, ed. d. livingstone, and j. kemp. http://www.simteach.com/ slccedu07proceedings.pdf. kafai, y.b. 1996. learning design by making games: children’s development of design strategies in the creation of a complex computational artifact. in constructionism in practice: designing, thinking, and learning in a digital world, ed. y.b. kafai, and m. resnick, 71–96. mahwah, nj: lawrence erlbaum associates. kirriemuir, j. 2007. an update of the july 2007 “snapshot” of uk higher and further education developments in second life. http://www.eduserv.org.uk/foundation/sl/uksnapshot092007. laurillard, d. 2002. rethinking university teaching: a conversational framework for the effective use of learning technologies, 2nd ed. london: routledgefalmer. littleton, f., and s. bayne. 2008. virtual worlds in education. the higher education academy subject centre for education, newsletter no. 10, spring 2008. http://escalate.ac.uk/4453 lombardi, m.m. 2007. authentic learning for the 21st century: an overview. eli paper 1, educause, may 2007. http://www.educause.edu/ir/library/pdf/eli3009.pdf. lowood, h. 2006. high-performance play: the making of machinima. journal of media practice 1: 25–42. middleton, a. 2008. audio feedback: timely media interventions. in proceedings of the third international blended learning conference, ‘enhancing the student experience’, june 18–19, at the university of hertfordshire, uk, 15–27. nitsche, m. 2005. film live: an excursion into machinima. in developing interactive narrative content: sagas_sagasnet_reader, ed. b. bushoff, 210–43. munich: high text verlag. alt-j, research in learning technology 219 papert, s. 1986. constructionism: a new opportunity for elementary science education. a proposal to the national science foundation, cambridge, ma, massachusetts institute of technology, media laboratory, epistemology and learning group. roberts, c. 2008. implementing educational technology in higher education a strategic approach. the journal of educators online 5, no. 1. http://www.thejeo.com/archives/volume5number1/ robertspaper.pdf shephard, k. 2003. questioning, promoting and evaluating the use of streaming video to support student learning. british journal of educational technology 34, no. 3: 295–308. yoon, s.-w, and d.h. lim. 2007. strategic blending: a conceptual framework to improve learning and performance. international journal on e-learning 6, no. 3, 475–90. 220 a.j. middleton and r. mather appendix 1 machinima used in the research process a whale of a tale http://www.youtube.com/watch?v=q_54jgose0g&feature=user a short film about a fishing trip, used to illustrate the potential of second life for creating machinima. what is machinima? http://www.youtube.com/watch?v=dplzkur06vo&feature=related a movie where an avatar explains the concept of machinima. used to introduce participants to machinima. second life: get one http://www.youtube.com/watch?v=xxj8rtw4ev4&nr=1 a promotional movie about second life. used to introduce participants to second life. architecture on the double http://www.youtube.com/watch?v=r7bxu2ozxo0&feature=user a film that shows the construction of a house by a group of avatars. used to show how second life can be used collaboratively. what is a prim? second life video tutorial http://www.youtube.com/watch?v=8tlcxwr-a2s a quick description of what prims are, used to show participants the basic construction components of second life. fruit fly genetics experiment http://www.youtube.com/watch?v=08f1rg-d1ua a simulation of a fruit fly genetics experiment, to illustrate a lab setting, and the use of an hud (head up display) to guide users through the experiment. heart murmur sim http://www.youtube.com/watch?v=xjy2iwbzop4 a short film that shows a simulation of various heart conditions. this demonstration shows how students can interrogate a virtual world audio enhanced simulation. second health: emergency and specialist care http://www.youtube.com/watch?v=mp8iyxzav-u&feature=related a machinima explaining how paramedics can respond to someone having a heart attack. mixing 3d virtual world footage with other media including graphics and video. the machinima provides an overview of services and processes to the layperson. alt-j, research in learning technology vol. 14, no. 1, march 2006, pp. 117–129 issn 0968-7769 (print)/issn 1741-1629 (online)/06/010117–13 © 2006 association for learning technology doi: 10.1080/09687760500479811 accessibility and adaptability of learning objects: responding to metadata, learning patterns and profiles of needs and preferences steve green*a, ray jonesb, elaine pearsona and stavroula gkatzidoua auniversity of teesside, uk; blondon metropolitan university, uk taylor and francis ltdcalt_a_147964.sgm10.1080/09687760500479811alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2006taylor & francis141000000march 2006stevegreens.j.green@tees.ac.uk the case for learning patterns as a design method for accessible and adaptable learning objects is explored. patterns and templates for the design of learning objects can be derived from successful existing learning resources. these patterns can then be reused in the design of new learning objects. we argue that by attending to criteria for reuse in the definition of these patterns and in the subsequent design of new learning objects, those new resources can be themselves reusable and also adaptable to different learning contexts. finally, if the patterns identified can be implemented as templates for standard authoring tools, the design of effective, reusable and adaptable resources can be made available to those with limited skills in multimedia authoring and result in learning resources that are more widely accessible. introduction drawing on our experiences of creating and using learning objects for teaching java as a first programming language (the java project), we will: ● explore the potential of accessibility profiles in the context of the development of learning objects. ● demonstrate how anonymous user profiles can influence the design patterns and templates of learning resources and how new accessible learning objects can be derived from them. * corresponding author. accessibility research centre, school of computing, university of teesside, middlesbrough ts1 3ba, uk. email: s.j.green@tees.ac.uk 118 s. green et al. by focusing on user profiles and design patterns we will argue that accessible (and usable) learning objects can be achieved by designing for adaptability. this is a departure from the well-documented and frequently proposed approach to designing for accessibility: that of universal design (sometimes called design for all). it does, however, reflect current developments. for example, the ims (2004) ‘accessforall’ specification currently proposes an adaptability model for digital resources and services, which attempts to match resources and services to users’ needs and preferences. although the motivation of the ‘accessforall’ specification is primarily to ensure the accessibility of resources, it also, almost as a by-product, ensures that resources are more usable and better designed. the new proposals move away from the view that web resources can be sensibly designed to meet the needs of everyone through strict adherence to standards and universal design principles. there is a new recognition that addressing all issues of accessibility, multiculturalism and language in a single standard could never be realised: one group or other would inevitably be excluded. however, we argue that the idea that resources should be adaptable (and adapted) to the needs of individuals or groups is eminently sensible. in exploring issues of accessibility, adaptability and learning objects, the term ‘accessibility’ is used to mean availability to a wide audience including those with specific needs; ‘adaptability’ is used in the context of the ease with which a digital resource can be modified to meet different user requirements, and the terms ‘learning patterns’, ‘learning objects’, ‘resources’ and ‘components’ represent a loose hierarchy of reusable elements that together form a learning solution. metadata and profiles of needs and preferences dublin core metadata dublin core (dc) in its simplest form is an interoperability standard for catalogue information on digital resources. books, journal articles, webpages, videos, digital images, learning objects or information relating to almost anything anyone might wish to reference could be considered a digital resource. the dublin core metadata initiative1 proposes an abstract model consisting of a description made up of one or more statement about a single resource and optionally the unique resource indicator of the resource being described. each statement is made up of a property and a value (or value unique resource indicator). one advantage of dc over other standards is that it is relatively simple, consisting of only 15 core properties, and consequently forms the basis of many other standards particularly for archives or digital repositories (e.g. the open archive initiative [2004] metadata harvesting standard). additionally, the core properties or elements are fairly obvious things such as title, subject, date, creator, contributor, and so on. dc statements can be encoded directly within the object or resource itself typically if it is an xhtml document. more commonly a separate catalogue is held to contain the dc metadata elements and reference information. essentially, the dc elements accessibility and adaptability of learning objects 119 typically form a separate database often referred to as a resource catalogue. in the case of the ‘java project’, learning object references and metadata are held in a learning resource catalogue (lrc3). accessibility profiles as well as knowing what resources they are dealing with and their properties, resource creators may also be interested in characterising users so they can match available resources to them. this is often referred to as user profiling. user profile information can be used for accessibility purposes, but in many instances context or preference may be equally important. for example, a car driver needing access to web or pda information may be in a similar position to a blind user in that neither can handle primarily visual material but could probably usefully access audible descriptions or instructions. consequently it may be possible to identify this profile as a non-visual or auditory profile. equally, other contexts may require large text (e.g. visual presentations to large audiences or to someone with a visual impairment). a non-auditory profile might mean that audio material needs a transcription and video needs captions or subtitles. in broad terms there is a need to be able to define a user’s contextual profile as a set of requirements for services and resources. typically the profile will define the user’s human–computer interaction requirements in terms of visual, auditory and tactile components. the three main elements of the profile will be: 1. display or output (typically visual but could be an auditory screen reader or tactile braille display). 2. control or input (typically keyboard and mouse but could be switches, touchscreen, joystick tactile devices or an auditory voice recognition system). 3. content (primarily visual, auditory media or textual components that can be read or transformed into auditory components by a screen reader). in the ims ‘accessforall’ proposal (ims, 2004; nevile, 2005), an ‘adaptability’ element is employed to identify a set of user needs and preferences. this is considered an important enough extension by the dublin core metadata initiative to consider incorporation into the dc standards itself. profiles of needs and preferences the ims ‘accessforall’ initiative also proposes an anonymous profile of needs and preference (pnp). the profile is anonymous in that there is no need to know who the user is or why they require the specified support. also, choices and options are considered just as important as absolute needs; for example, a user might express a preference for braille output, but with an indicator that auditory substitution is also acceptable but visual elements are unacceptable. another profile might express the need for large text or a screen reader. in essence this is encoded using a predetermined structure and vocabulary (see figure 1). 120 s. green et al. figure 1. ims ‘accessforall’ pnp vocabulary the proposed structure and vocabulary encourages the designer to check and build for adaptability. for example: is the display fully accessible to a screen reader? is mouse emulation or alternative pointing catered for? does the content cater for personal style sheets? these factors and more can be coded into the adaptability element using the items identified in figure 1. the important point is that, in principle, if digital resources are associated with rich metadata and a detailed profile of needs and preferences are available, there exists the necessary pre-requisites for designing and developing accessible applications through adaptable resources. before considering in detail the process applied, the following section takes a closer look at the application area under consideration. screenreader screenenhance textreadinghighlight braille tactile visualalert display structuralpresentation keyboardenhance onscreenkeyboard alternativekeyboards mouseemulation alternativepointing voicerecognition structuralnavigation control codeinput alternativestovisual alternativestotext visualtext alternativestoauditory learnerscaffold personalstylesheet content extratime figure 1. ims ‘accessforall’ pnp vocabulary accessibility and adaptability of learning objects 121 learning objects and metadata criteria for reusing learning objects the arguments in favour of the reuse of learning resources include economic ones (downes, 2001) and those of quality (jones, 2004). however, in order for learning resources to be effectively reused they should be designed with reuse in mind: they should be cohesive and decoupled from other resources (boyle, 2003). in other words, a reusable learning object should be concerned with a single topic (i.e. cohesive) and not be unnecessarily linked to (i.e. decoupled from) external resources; thus, a learning object should be ‘an independent and self-standing unit of learning content’ (polsani, 2003, section 2.2). in order to be as widely reusable as possible, learning objects should also be context free. however, freedom from a particular learning context may reduce their pedagogical effectiveness; therefore reusable learning objects, while being free from any particular context that would restrict their reuse, should still be adaptable to the specific context in which they may be used (jones, 2004). patterns expert practitioners use their experience of solving problems in the past to build on and create new solutions in new situations. one thing expert designers know not to do is to solve every problem from first principles. rather, they reuse solutions that have worked for them in the past. when they find a good solution they use it again and again. such experience is part of what makes them experts. (gamma et al., 1995, p. 1) these reusable solutions that gamma et al. refer to are design patterns. each pattern describes a problem which occurs over and over again in our environment, and then describes the core solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice. (alexander et al., 1977, p. x) here, alexander et al. were referring to patterns in architecture and town planning, but core solutions in any area of design can be described in terms of patterns. it is these patterns that will be extracted from the learning objects referred to earlier for reuse in creating new learning resources. the aim in doing this is to help learning object designers by providing them with a set of design ideas in a structured way that will clearly articulate the design problem and its solution (goodyear, 2005). this will enable the designer to produce effective and adaptable learning resources in an efficient manner. in simple terms, therefore, a pattern defines a common problem and proven solution, which an experienced designer may choose to adopt. in practice, patterns are implemented as standard sequences of learning components (here ‘pages’) compounded from learning objects and resources. they are generally implemented as a selection of reusable frameworks with slots for the learning 122 s. green et al. components. for example, a particular type of learning activity might best be presented using a sequence composed of a text-based explanation of the principles involved, an illustrative animation, a review question and a practical exercise. this learning pattern might be presented in a framework with four separate pages each embedding a separate learning object. this learning pattern forms a solution to a learning event, which can be reused in designing an appropriate learning sequence. practical application: the java project the design and successful use of learning objects to complement a first-year degree course in the programming language java is well documented as part of an innovative blended learning approach to the teaching of programming to first-year computing students (boyle et al., 2003). the learning objects were designed by a team of subject experts and a multimedia developer who authored them in flash. each learning object was self-contained and was intended to be used either as a stand-alone resource or as an embedded link within a compound learning object. each object was made up of a sequence of frames, which themselves consisted of smaller, selfcontained, multimedia content. the aim of each learning object was to explain a particular programming concept in the java programming language through the use of examples. the main learning objects, their content and their sequencing were designed to follow a particular pedagogical method. the effectiveness of the learning resources was recognised by being one of a small number of recipients of the european academic software awards in 2004.2 analysing learning objects the java project implemented a simple sequence of ‘pages’ of multimedia content. each of the pages was of a particular type, or design, and served a specific purpose. each learning object comprised a similar sequence of pages. user control over the sequence consists of simple forward, back and rewind navigation buttons. in a typical learning object, the first page provided the title and a short description of the topic to be covered. this was followed by pages of different types from simple static text and images to pages that consisted of synchronised sequences of animations and text. the general format of each sequence was: 1. the title page. 2. the concept page. 3. the example page. 4. the detailed explanation page. 5. the practice page. a typical sequence consisted of at least five steps employing five basic page types. in some learning objects, steps 3 and 4 were repeated with different programming examples and, often, there was more than one practice page. accessibility and adaptability of learning objects 123 learning objects as patterns a major aim of our work is to provide design patterns that will enable new learning resources to be created from existing successful resources. from the analysis of the learning objects used in the java project, a pattern can be detected in the sequence of pages that make up each learning object. the individual page types can also be regarded as patterns that describe a particular learner activity. thus the main pattern may be termed the sequencer pattern and it can be regarded as a compound pattern (i.e. one that is made up of other patterns). buschmann et al. (1996) define design patterns, in part, as describing a particular recurring design problem that arises in particular design contexts and presents a wellproven scheme for its solution. the learning object components, described here, have been extracted from existing successful learning resources and they are described in a technology neutral manner, thus they can be considered as design patterns. however, buschmann et al. also suggest that patterns should be described in a consistent and structured manner in terms of their context (the situation that gives rise to the problem), the problem itself and the solution to that problem. this advice follows the lead of alexander et al. (1977), who describe their patterns in a similar manner and who refer to a collection of such patterns as a pattern language. it is also important that the learning object components that are defined using these patterns fully take into account the need for reuse and adaptability. thus these aspects need to be addressed in any documentation of a pattern. (an example of such documentation (as adapted by the authors) can be found in appendix 1.) the analysis of the original learning object results in the identification of a pedagogical pattern that represents a simple framework into which a series of activities can be located. the implementation of the pattern is a simple page sequencer that could be used for any simple linear sequence of activities. the pedagogy is thus not enforced; it is left up to the designer to select the appropriate activities and the correct ordering of those activities. the activities themselves are also patterns and are implemented as templates into which appropriate content can be inserted. this content can also be accompanied by metadata that help search engines find it, define its properties and indicate its level of adaptability. based on these metadata, an automated service can be identified to adapt the learning object components or patterns to different user requirements. discussion: towards adaptable learning objects learning object metadata the provision of metadata was not of prime concern during the initial development of the learning objects in the java project. however, dc metadata are typically held within a separate catalogue. consequently, identifying the main features of the resource, its learning objectives, media content, author and other relevant details can be a second stage refinement based directly on a learning object or learning pattern description. depending on the context in which the object is expected to be used, a 124 s. green et al. range of descriptors may be indicated. in our context those descriptors that have appropriate matches to the anonymous pnps (and which will generally appear in the newly proposed ‘adaptability’ tag) are liable to be the most relevant. application profiles for the purposes of this discussion we accept the extension of the definition of accessibility beyond disability, and define the relationship between a user and a resource as accessible when the characteristics of the resource as delivered match the user’s needs and preferences (nevile, 2005). the definition of accessibility implied here is that the relationship between the user and the resource is one that enables the user to make sensory and cognitive contact with the content of the resource (ims, 2004). according to the ‘accessforall’ statement the term disability has been re-defined as a mismatch between the needs of the learner and the education offered, and it is therefore not a personal trait but an artefact of the relationship between the learner and the learning environment or education delivery (cooper et al., 2005). accessibility, therefore, is the ability of the learning environment to adjust to the needs of all learners and is determined by the flexibility of the environment (with respect to presentation, control methods and access modality) and the availability of adequate ‘alternative-but-equivalent’ content (heath et al., 2005). the needs and preferences of a user may arise from the context or environment the user is in, the tools available (e.g. mobile devices, assistive technologies such as braille devices, voice recognition systems, or alternative keyboards, etc.), their background or a disability (physical, cognitive or sensory). according to the ‘accessforall’ vocabulary, descriptions of needs and preferences are separated into display, control and content characteristics (as already described). the user’s descriptions of their needs and preferences may change according to the context or occasion. accessibility service in order to achieve an accessible relationship between the resource and the user, descriptions of user needs and preferences are checked against descriptions of resource components until they match. this process involves a description of a user’s control, display and content needs and preferences being matched with a description of the components of the learning object (nevile et al., 2005). the delivery of the appropriate component will form an accessible relationship between the user and the learning object. according to the ‘accessforall’ metadata overview, accessible systems should be able to adjust the user interface of the learning environment, locate needed resources and alter resources properties to match the needs and preferences of the user. this may involve the substitution, augmentation or transformation of components of the resource such as changes in sensory modality. for the purposes of this paper we will refer to an abstract transformation, augmentation and substitution service (tass), which is geared to our specific learning object application. however, this can be viewed as a special instance of an ‘accessforall’ service. accessibility and adaptability of learning objects 125 an example of a replacement occurs when a user accessing a learning object requires a vision-free access to the resource, and therefore need alternatives to the visual content contained in the primary resource of the learning object. as stated previously, the profile of this user may actually be the same as the profile of a sighted user accessing the learning object on a pda while driving: the user needs to access the learning object using non-visual techniques. for this relationship to be accessible it is necessary to replace the visual element of the learning objects with components that match the user’s preferences of vision-free access. it is also often the case that the original content of the resource has to be supplemented, for example with the availability of a dictionary or captions, for an aural component. the process of retrieving and presenting accessible learning objects employs an adaptation of the ‘accessforall’ application service or tass. this is presented in figure 2. figure 2. tass: transformation, augmentation and substitution service in simple terms, the tass service is triggered by the user making a request through the learning resource catalogue to the learning object repository. the tass service checks the catalogue for the objects accessibility element and compares it with the user’s profile in the pnp repository. appropriate transformations, augmentations and substitutions are applied either directly through the local tass managed resources or indirectly through a global ‘accessforall’ service. in principle, transformation, augmentation and substitution may occur at any level but would typically take place at the component or learning pattern level. where a learning object in itself is not considered accessible, its constituent components are examined. if all individual components can be transformed, augmented or access4all service search engine tass user learning object repository pnp repository learning resource catalogue figure 2. tass: transformation, augmentation and substitution service 126 s. green et al. substituted then the learning object can be reconstructed as an accessible object. in certain cases, needs or preferences may require a substitute learning pattern. the learning object can then be recreated from its constituent components but employing an alternate pattern. while the primary concern is one of accessibility of resources, this feature could simply provide learners with learning patterns more suited to their personal learning styles. conclusions and future work drawing upon the work of the java project, the current research looks at ways to enrich and enhance these learning patterns in order to develop not only pedagogically effective, reusable but adaptable learning objects. the next stage of the research is to develop a tool to assist academics, without previous programming experience, in creating reusable, retrievable and adaptable learning objects. by incorporating these learning design patterns into a tool, an instrument can be developed to help the authors (in this case university tutors) to design and develop learning objects. this tool will also provide the opportunity for the author to assign dc metadata to the learning object in as simple a way as possible: this may include methods for intelligently inferring metadata, in order to make the process easier and quicker for the resource creator. an interesting challenge in terms of this research is to deal with the integration of ‘accessforall’ and dc vocabularies. the development of an application profile for a set of user accessibility needs and preferences using the ‘accessforall’ vocabulary can be easily discovered using the dc records. however there is considerable disagreement on whether the pnp, which is a metadata record itself, should be integrated within the dc framework. further work also needs to concentrate on the recursive nature of accessibility matching. the matching process (figure 2) involves having a first ‘main’ resource identified and then tested against a pnp, while it would be more efficient to first decompose the resource to find an alternative to one component and to transform, augment or substitute that component, before re-assembling and re-testing the resource. another important issue, which has yet to be resolved, is who is responsible for the substitution or augmentation service. it might be reasonable to expect a tutor employing a short video element to provide a transcript, but if a user profile expresses a preference for british sign language can we really expect this service to be provided by the content creator? the obvious conclusion, therefore, is that the provision of the adaptation service needs to be a collaborative venture, with additional components or alternative learning patterns provided by those with the skills to do so. in essence this can lead to the creation of a wide range of additional or alternative learning objects; some of which may be considered true alternate resources, and others as virtual objects in that they may have no physical existence but are supplied on-demand through transformations and advanced searches in response to a new pnp. in conclusion, the newly emerging ims and dc adaptability and accessibility standards and the proposed pnps are set to have a profound impact on the way we view accessibility and adaptability of learning objects 127 the creation of digital content as well as the way it is presented to us. this paper suggests an approach for designers of learning objects and learning patterns to respond to this challenge. assuming that appropriate tools can be created to allow non-technical tutors to create learning objects based on these patterns, and assuming that rich metadata can be collected or inferred from those resources, then learning objects can be developed that are truly adaptable. the process of carrying out this future research and applying it to this application area will give a clearer indication to those developing the standards of whether their approach is fundamentally sound and where refinements are needed. notes 1. dublin core metadata initiative: http://www.dublincore.org 2. european academic software awards: http://www.bth.se/llab/easa.nsf references alexander, c., ishikawa, s. & silverstein, m. (1977) a pattern language: towns, buildings, construction (new york, oxford university press). boyle, t. (2003) design principles for authoring dynamic, reusable learning objects, australian journal of educational technology, 19(1), 46–58. boyle, t., bradley, c., chalk, p., jones, r. & pickard, p. (2003) using blended learning to improve student success rates in learning to program, journal of educational media, special edition on blended learning, 28(2–3), 165–178. buschmann, f., meunier, r., rohnert, h., sommerlad, p. & stal, m. (1996) pattern-oriented software architecture (chichester, john wiley). cooper, m., treviranus, j. & heath, a. (2005) meeting the diversity of needs and preferences—a look at the ims accessforall specifications’ role in meeting the accessibility agenda efficiently, paper presented at the accessible design in the digital world conference. available online at: http://ewic.bcs.org/conference/2005/accessible/workshops/workshop3.htm (accessed 4 january 2006). downes, s. (2001) learning objects: resources for distance education worldwide, international review of research in open and distance learning, 2(1). gamma, e., helm, r., johnson, r. & vlissides, j. (1995) design patterns: elements of reusable objectoriented software (addison-wesley). goodyear, p. (2005) educational design and networked learning: patterns, pattern languages and design practice, australian journal of educational technology, 21(1), 82–101. available online at: http://www.ascilite.org.au/ajet/ajet21/goodyear.html (accessed 4 january 2006). heath, a., treviranus, j. & nevile, l. (2005) individualised adaptability and accessibility in e-learning, education and training part 1: framework. available online at: http://jtc1sc36.org/doc/ 36n1024.pdf (accessed 4 january 2006). ims (2004) ims accessforall metadata specification. available online at: http://www.imsglobal.org/ accessibility/(accessed 4 january 2006). jones, r. (2004) designing adaptable learning resources with learning object patterns, journal of digital information, 6(1). available online at: http://jodi.tamu.edu/articles/v06/i01/jones/ (accessed 4 january 2006). nevile, l. (2005) anonymous dublin core profiles for accessible user relationships with resources and services, paper presented at the international conference on dublin core and metadata applications, madrid, spain. 128 s. green et al. nevile, l., cooper, m., heath, a., rotherberg, m. & teviranus, j. (2005) learner-centred accessibility for interoperable web-based educational systems, paper presented at the 1st international workshop on interoperability of web-based education systems, 14th international world wide web conference, chiba, japan. available online at: http://l3s.de/∼olmedilla/events/ interoppapers/paper08.pdf (accessed 4 january 2006). open archive initiative (2004) open archive initiative metadata harvesting standard version 2.0. available online at: http://www.openarchives.org (accessed 4 january 2006). polsani, p. (2003) use and abuse of reusable learning objects, journal of digital information, 3(4). available online: http://jodi.ecs.soton.ac.uk/articles/v03/i04/polsani (accessed 4 january 2006). appendix 1. extracts from a ‘learning objects for programming’ pattern language the following descriptions form a part of a small patterns catalogue. they are similar to the format used by alexander et al. (1977) as adopted (and slightly adapted) by goodyear (2005) but with the addition of an entry on reuse. the detailed explanation page pattern context. when a new programming concept is to be learnt it is valuable to be able to see an example program, or program fragment, and to have its operation explained in an interactive way. problem. in order to properly illustrate a programming concept or structure it is valuable to ‘walk through’ an example. in a teacher-led class this is easily achieved but for self-study a dedicated resource needs to be designed that allows the learner to control the pace and progress through the ‘walk-through’ process. solution. using a multi-frame approach, an example program, or program fragment, is displayed in one frame where its execution can be simulated by highlighting each section of the code in execution order. in a second window, the effect of the program execution is demonstrated via an animation and in a third one, an explanation of the execution is given. the animation illustrates the code being executed by highlighting the appropriate parts of the code that are being executed. at the same time the animation is also stepped through and the text window changes at each step to describe the action of the particular line, or section, of the program code. thus there are three parts of the accessibility and adaptability of learning objects 129 page that are cycled through in synchronisation: the code, the commentary and the animation. the learner controls the operation via a button in the explanation window: this starts the synchronised animation. reusabilty. (e.g. an explanation of a programming concept could be supported in two different programming languages by simply replacing the program element in one frame) related patterns. this pattern may be implemented by the frameset controller and frameset components. alt-j, research in learning technology vol. 15, no. 1, march 2007, pp. 37–50 issn 0968-7769 (print)/issn 1741-1629 (online)/07/010037–14 © 2007 association for learning technology doi: 10.1080/09687760601129588 challenging the five-stage model for e-learning: a new approach pam moule* university of the west of england, uk taylor and francis ltdcalt_a_212890.sgm10.1080/09687760601129588alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis151000000march 2007pammoulepam.moule@uwe.ac.uk the five-stage approach to e-moderating has provided a coherent model upon which to base online learning design in higher education. however, despite its growing popularity, there are concerns that the model is becoming a dominant discourse, being adapted as a template for the design of all online teaching and learning, to the exclusion of other ideas. it is suggested that the five-stage model may not be the panacea it appears and alternative models of e-learning cannot be ignored. this paper reviews the five-stage model and contrasts it with a new conceptual model, ‘the e-learning ladder’, conceived as part of research with healthcare students in the higher education setting. introduction the integration of e-learning into higher education pedagogy has occurred against a backdrop that includes the development of a unified e-learning strategy (department for education and skills, 2005) and an increased focus on setting priorities for elearning research. currently, the uk joint information systems committee is funding an e-learning programme.1 this includes an e-learning and pedagogy strand with a focus on both designing for and understanding learning, including consideration of learner participation in, and experience of, e-learning (beetham, 2005). one of the most popular process-based online teaching and learning models is that conceived by salmon (2000, 2003). it is designed to demonstrate learner participation in online learning and the role of the e-moderator or online facilitator. this fivestage approach ‘model of teaching and learning online’ provides a model that is easily understood and employed by a number of e-moderators within higher education, although it may not be the panacea it appears. this paper reviews salmon’s five-stage model and contrasts it with a more embracing conceptual model of e-learning, ‘the e-learning ladder’, developed as part of doctoral research completed within a higher education setting (moule, 2005, 2006a). * university of the west of england, faculty of health and social care, glenside campus, blackberry hill, stapleton, bristol bs16 1dd, uk. email: pam.moule@uwe.ac.uk 38 p. moule the ladder design was based on the interpretations of research that asked whether students of the healthcare professions could develop the essential components of a community of practice (wenger, 1998), while engaged in online learning. as such it represents the learner’s experiences of e-pedagogy. it reflects an interpretation of elearning that encompasses instructivist, where the learner takes a more passive role learning from provided knowledge, through to constructivist learning approaches, seen as learner focused, and endorses the building of learning on previous experience and learning (peters, 2000). e-moderating: the five-stage model developed following research into online education and training with the open university, salmon’s (2003) model features five stages (see table 1) presented as a flight of steps. each stage identifies technical and e-moderating skills required, with an interactivity bar running along the steps that indicates varying amounts of interaction expected between the participants at each stage. this is seen as greatest towards the end of stage three (information exchange), throughout stage four (knowledge construction) and into stage five (development). participants are expected to progress through each of the five stages as part of online networking and group working. the structure is designed to support a constructivist approach to learning. this constructivist model of e-moderating provides a framework with clear progressive stages that can support the design and facilitation of online courses. it received a number of uncritical citations in the united kingdom when initially published (good, 2001; barker, 2002, bennett & marsh, 2002; thorpe, 2002). its clarity has generated wide appeal, and consequently the model has been applied in structuring online programmes (lisewski & joyce, 2003; hughes et al., 2004). it should be recognised, however, that the e-moderating model was developed from experiences of facilitating online networking and group working. its principle purpose is to provide a model for e-moderators to support student engagement and learning online, employing constructivist pedagogic theory (salmon, 2003). consequently, it is limited because the variety of e-learning approaches available for use within table 1. an interpretation of the five-stage model (salmon, 2003, p. 29) five stages of the model technical skills e-moderating activity 1. access and motivation setting up system and accessing welcoming and encouraging 2. online socialisation sending and receiving messages familiarising and providing bridges between culture, social and learning environments 3. information exchange searching, personalising software facilitating tasks and supporting use of learning materials 4. knowledge construction conferencing facilitating process 5. development providing links outside closed conference supporting—responding challenging the five-stage model for e-learning 39 computer-mediated communication is neglected and the range of learning theories available is ignored. not all e-learning occurs as part of a community; learning from a cd-rom or from interactive web-based packages can be an individual and instructivist activity. the learners would be gaining objective information, such as through the basic life support cd-rom (moule, 2002), however, are seen as passive recipients of knowledge (armitage & o’leary, 2003). while constructivist online learning communities are often adopted in higher education (wong et al., 2003), the five-stage model has not reflected the potential available to use e-learning as part of an integrated approach that includes face-to-face delivery. indeed, chowcat (2005) found the model ineffective when used to train those mentoring head teachers in the united kingdom. chowcat suggested the model assumes an exclusive online environment will be used to support a course run to a specific timeframe of one week for each stage. the course structure proved difficult to sustain in the school year, with chowcat concluding that the model may not readily transfer to different learning situations. there are further concerns that the model is dominating discourse in learning technologies, being seen as a template for the design of all online teaching and learning environments regardless of the context. there is a broad concern that the reification of models of learning and teaching, while meeting organisation needs for transferable, multi-use products, will dominate and stifle professional practice development. for example, lisewski and joyce (2003) presented difficulties seen when using the fivestage model as a template for an e-moderating training course. the course failed to take account of individual learning styles and the rigid application of its design was difficult to challenge. this led them to highlight the dangers of reification. objectifying the model transforms it to a product that is seen as a ‘one size fits all’, for use across any number of teaching and learning contexts. paradoxically, reification of the model undermines its ethos, which seeks to support reflective practice. through slavishly applying the model as a rigid course, any opportunities to develop flexibility and reflexivity are lost. further research by jones and peachey (2005) presents a critical review of the fivestage model. their main criticisms are levelled at the stages of socialisation. jones and peachey included face-to-face contact as part of the initial stage. they were unable to determine whether an ‘appropriate’ level of socialisation was achieved here, as little guidance is offered that allows such measurement. furthermore, activities in stage two were subdued, with a reduced number of postings seen. this led jones and peachey (2005) to suggest that stage two may be omitted if initial socialisation in stage one has been effective. the findings also suggest those employing the model might need further guidance to enable judgements about the readiness of participants to progress through the stages of the model. e-learning ladder: theoretical development the e-learning ladder was conceived as part of research exploring whether the essential characteristics of communities of practice (wenger, 1998) develop in higher 40 p. moule education online learning environments (see moule, 2005, 2006a, b). conducted in a university using a case-study approach (yin, 1994), data were collected from healthcare students taking part in a six-week online module. from the sample of 15, five students completed online diaries, three students were interviewed and a group of seven students were involved in an online discussion board. an increasing number of studies are applying wenger’s (1998) theory to online learning contexts as interest in constructivist approaches to e-learning develop. conducted in america, rogers (2000) sampled 26 teachers and administrators participating in a three-week workshop, ‘teachers of english as a second or other language’. online dialogue was analysed, suggesting the presence of wenger’s (1998) elements of mutual engagement, joint enterprise and shared repertoire. a further study of three software engineering students in the united kingdom is presented by chalk (2001). based on limited analysis of team documentation and responses to a ‘hidden’ examination question, chalk (2001) implied students’ online experiences constituted a community of practice. both case studies have a number of weaknesses and the results should be interpreted with caution. in particular, the small sample sizes and limited periods of engagement weaken the conclusions. indeed, hildreth (2004) presents a further case study of an online community of practice, highlighting some of the problems and issues involved in trying to build relationships across dispersed boundaries. wenger’s (1998) theory has also been employed in the evaluation of a virtual learning environment, built to support an undergraduate medical course (ellaway et al., 2004), identifying aspects of the learning environment that could be improved. earlier research used the theory to review online working, with somekh and pearson (2000) analysing electronic interactions among researchers during a european project that considered children’s representation of information and communication technology (ict). dispersed working seemed to confound the negotiation of joint enterprise, with deadlines often being missed. different shared repertoires and histories of research methods affected mutual engagement, as participants were unable to develop shared understanding of the action research approach used. more recently, seale (2004) applied the theory to the interpretation of a literature review that explored the enterprise of accessibility practices in e-learning. the findings highlighted the need for learning technologists to share accessibility practices and focus on the process of accessibility, rather than the product. finding similar results to rogers (2000) and chalk (2001), elements of mutual engagement, joint enterprise and shared repertoire emerged from the analysis of the textual data collected in this study. the results suggested healthcare students were able to develop a community of practice, although this was not uniformly seen (moule, 2005, 2006a). it was apparent that operating within a community of practice online held complications arising from the e-based context of learning (moule, 2006a). the emerging issues were related to access, trust development, the need for technical skills and support, facilitation issues, group working and the effects of longevity of participation. these issues were extracted to develop the ‘supports’ for the ‘rungs’ of a conceptual model for e-learning—the e-learning ladder. challenging the five-stage model for e-learning 41 the e-learning ladder—a conceptual model in contrast to the five-stage model, the e-learning ladder (moule, 2005) acknowledges a range of learning approaches, starting at the bottom ‘rung’ with an isolated approach to learning that might be termed as instructivist and moving through the ‘rungs’ ending with constructivist, or interactive learning approaches (see figure 1). through the inclusion of different learning approaches from isolated through to interactive learning, it acknowledges that not all e-learning is constructivist or course based and reflects the range of e-based learning materials available. although not explicitly expressed, there is an expectation that the e-resources introduced might be used as part of a blended learning approach or incorporated within classroom deliveries. for example, learners using cd-roms or web-based materials to gain essential knowledge to support skill development are likely to attend a face-to-face skills practice session to develop the practical skills to compliment theoretical learning. figure 1. a conceptual model of online learning: the e-learning ladder the structure of a ladder is not meant to imply that students must move up through each ‘rung’ to reach the top; rather, it is a presentation of a conceptual ladder of learning. it shows how learning might be positioned at instructivist to constructivist levels. although presented hierarchically in a ladder structure, it is intended that the ‘rungs’ should be viewed as presenting flexible pedagogies, which inter-link. it is additionally suggested that the ‘rungs’ might be developed over time to allow the incorporation of figure 1. a conceptual model of online learning: the e-learning ladder 42 p. moule new methods of e-delivery, such as the expanding use of mobile technologies, to support learning in higher education. the ladder ‘rungs’ the initial ‘rung’ includes information gathering that might encompass using bibliographic databases to search for material and accessing course notes. rusby’s (1979) classification of software would see this level as emancipatory, where the computer acts as an aid to learning but is not central to the learning process. it enables the student to access material that might support their learning. the second ‘rung’ incorporates the use of interactive media for learning, such as cd-roms. the level of interaction is limited to the learner working with the educational media. this would be seen as the instructional paradigm (rusby, 1979), where information is presented, broken into steps to aid learning. this reflects a behavourist approach to learning pioneered by skinner, where specific steps in the process must be learned and followed (nye, 1996). one example is a basic life-support cd-rom developed to support competency development. it includes the steps in the skill, and tests assess the learner’s knowledge of the topic (moule & gilchrist, 2001). materials at this level might also include those with simulations and animations to explore the real world, viewed as the revelatory paradigm (rusby, 1979). a statistical program allowing learners to change the values of variables (adams, 2004) provides one example. these packages engage learners in exploratory learning and ‘mental model building’, a constructivist technique that also features in the higher ‘rungs’ and as such is an example of the inter-linking of pedagogies between the ‘rungs’. materials can be further developed to meet rusby’s (1979) definition of the conjectural paradigm. these materials provide greater scope for learners to set their own learning parameters; for example, through conducting virtual experiments (bell, 1999). the higher ‘rungs’ encompass learning approaches that would be seen as constructivist. they encourage the learner to think creatively and problem-solve, update knowledge and skills, and develop analysis, critical thinking and evaluation (adams, 2004). the final three ‘rungs’ of the ladder include aspects of interaction and engagement, although they suggest an incremental development of this, with the greatest scope for learning through interaction and discussion being placed within the higher rungs. these ‘rungs’ support a social constructivist approach to learning based on the theories of vygotsky (1978), where learning is constructed through social interaction. video conferencing on the third ‘rung’ allows synchronous transmission of learning materials. the learner will engage by listening to delivery and may interact through questioning and verbal discussion. this may support debate, argument and reflection in knowledge creation, also seen as present within the next ‘rung’. the fourth ‘rung’ presents discussion boards and email. these allow students to interact online through asynchronous communication as the fifth ‘rung’. this approach gives scope for student-led debate and interaction that contributes to knowledge creation. the challenging the five-stage model for e-learning 43 virtual classroom enhances communication opportunities, providing a vehicle for synchronous communication, used by the students for decision-making. this is viewed as enhancing opportunities for effective online communication, essential to knowledge development. at the top of the ladder is communities of practice (wenger, 1998), where interaction among learners is vital to the development of mutual engagement, joint enterprise and shared repertoire. the development of identity is dependent on engagement in the community, with knowledge and learning being constructed through understanding gained within the community. engagement in a community might require the student to employ all of the learning approaches identified in the lower ‘rungs’. this can include drawing on databases to access information to support the community endeavour and engaging in the use of discussion boards, email and virtual classroom to communicate, problem-solve and support decision-making with community members. the ladder ‘sides’ the ‘sides’ of the ladder demonstrate the support needed for learners to access the ‘rungs’. these results are drawn from the study data (moule, 2005) and are presented here as verbatim quotes labelled db1–db7 (discussion board members), d1–d5 (online diaries) and as attributed to interviewees i1–i3. there is the potential for support to increase as the student climbs the ladder, depicted by the arrows at the ‘sides’. learners may require greater support in some areas as the technological complexity of the online environment increases, although this need not be uniform. for example, as part of the research study not all the learners experienced difficulties in the use of virtual classrooms and discussion boards, requiring the development of ict skills. this said, often those with some computer familiarity are required to develop ict skills in certain contexts. andrusyszyn et al. (1999), in evaluating computer conferencing, found that despite nurses having access to home computers, the internet and using email, they needed technical guidance in the use of computer conferencing technology. access. access coupled with motivation forms the first stage of salmon’s (2003) model. at this stage salmon refers to the need for e-moderators and participants to gain access to the online system. it is suggested that participants may need some technical support at this point to facilitate access. the e-learning ladder also includes access as a potential issue, but relates it to all e-based learning, suggesting that access issues might increase for some users as the technological complexity of the online environments intensify. access is therefore not viewed merely as initial but as ongoing issue, as one student without computer access at home highlights. i think i lost nearly two stone walking into uni, that’s the biggest advantage of online learning. (db1) access to appropriate ict hardware, software and network connections is required to support e-learning, and is often problematic to learners. 44 p. moule just to let you know that this weekend the uni computer system is shutting down—so i have the weekend off. (db3) radiographers have been unable to access bb [blackboard] for a number of days. (d3) an earlier study highlights access difficulties seen for higher education students, supporting these findings. mcmahon et al. (1999), sampling 835 first-year students in belfast, found students experienced access difficulties that proved to be a barrier to computer use. a further concern is access for those with disabilities. difficulties can exist for the visually impaired, colour blind, deaf and those unable to operate a mouse. such disabilities can affect individual ability to engage with the computer and access visual imagery such as graphics and colour and audio presentation.2 all educational institutions need to comply with the disability discrimination act 1995 (her majesties stationery office, 1995) and special educational needs and disability act (her majesties stationery office, 2001). technical skills. the need for increased technological skill development among elearners was identified by andrusyszyn et al. (1999) in an evaluation of computer conferencing that confirmed the need for technological guidance in the use of technology. boyle and wambach (2001), curtis et al. (2002) and hong et al. (2003) comment on a lack of confidence and fear as barriers to using technology for online learning, following research with postgraduate students studying in higher education. indeed, students in this study noted, i’m … an adult nurse who is very scared about ip3 [online module]. (db7) for me personally, getting to grips with the it side of it. my confidence was one or two …’ (i1) salmon (2003) reflects on the possible need to support participants with technical skills at different stages of the model, although referring to these as technical support. the ladder demonstrates that the technical skills of learners will need consideration across a range of e-learning materials and deliveries, with the possibility of the need for such skills to increase as the engagement in e-learning becomes more complex. this was seen in the research results as students familiar with computer use at the lower ‘rungs’ found use of components of the virtual learning environment difficult. there were no problems with going online, although it took me ten minutes to work out how to use the virtual classroom. (d1) the ladder separates technical skills from technical support, however, highlighting the need for ongoing support for computer-based learning, whether it employs instructivist or constructivist learning approaches. technical support. not only do learners potentially need support with initial access to hardware, learning resources and networks, but also issues can arise with ongoing challenging the five-stage model for e-learning 45 technological delivery. cooper and mcconnell (2000) identified that technical issues were one of the least favourite aspects of a web-based tool supporting physiotherapy student learning in scotland. a number of students involved in this research experienced problems with password access to the virtual learning environment (moule, 2005) and further examples of the need for ongoing technical support were highlighted. sometime during the day it was difficult to get into the blackboard. a message kept saying that the request to get into the blackboard was timed out. (d1) after entering the v-classroom about twenty mins into the conversation i was thrown out of the classroom … this also happened to other members. (d2) it is suggested that the need for technological support may increase with the complexity of the learning environment being used, although some may have greater ict skills than others, eliminating the need for technical support. this does not detract from the need for educational providers to ensure technical support is available and accessible. atack (2003) provides an example of users failing to ask for technical help when needed, which stemmed from their lack of ict knowledge. facilitation. bender (2003) suggests that the role of the facilitator is important in supporting initial socialisation and ongoing maintenance of online groups. salmon’s (2003) model focuses on the role of the e-moderator, identifying the different emoderating skills required in each of the five stages. this component of the model seeks to enable e-moderators, offering practical guidance to those facilitating student engagement in online learning. the e-moderator role is further supported through a second publication (salmon, 2002) that concentrates on e-activities. a number of international studies conducted in higher education settings identify student dissatisfaction with online tutor support (hara & kling, 2000; monteith & smith, 2001; hong et al., 2003). this discord seems to reflect the frustration felt by students, who identify a lack of online interaction with their facilitators. monteith and smith (2001) suggest this may be related to difficulties experienced by moderators of constructivist approaches to learning who are unsure when they should interject. facilitation is highlighted as a necessary support for all e-learners throughout the ladder, including those engaged in constructivist learning approaches. a number of students looked to the facilitator to address specific questions such as one enquiring about the style of presentation needed for the assessment. i was wondering about the use of the first person … (db2) it is felt that online facilitation will be increased when employing these learning approaches, but that there may still be a need for facilitation for those using interactive learning media or gathering web-based information. the facilitation need not always be conducted in the online context, but may relate to the preparation of participants for learning and any subsequent activities or learning that relate to the use of materials or development of skills that might occur in a face-to-face setting. in 46 p. moule other words, the facilitation may occur as part of a blended approach to learning, where the application of e-based learning or information is facilitated. group working. the e-learning ladder includes group working and composition as one of the ‘sides’. group composition was seen as important to the functioning of the communities of practice as it was felt to impact on group working. the discussion group members had pre-established friends in the group. this group worked effectively, developing shared repertoire and forming relationships during the module. i agree that we should follow the learning outcomes, then maybe adapt some of them to our group’s needs. (db4) other students also felt they had developed relationships within their groups, with one stating: two days before submission people were putting up supportive messages. (i2) previous knowledge of group members has been found to aid group functioning. as part of australian research with 114 business students, andrews and schwarz (2002) found learning behaviours were linked to team performance, with those teams where members already knew each other performing more highly. the initial phase of the research also saw time spent on social learning as individuals tried to socialise into the group, although for some online socialisation was problematic. for example, one student felt that relationship-building for her was best achieved in a face-to-face environment: for group working for me, i would like to know the people i am working with. (i3) effective learning online requires the support of underpinning social processes, as team relationships have an impact on performance. joiner (2004) also asserts that the assumed advantages of constructivist online learning are being challenged, with recent research showing there is no evidence of equal participation. indeed one student commented on such issues. there only seem to be myself and two others who are pulling our weight! (d5) there is a need to create effective social collaboration and networking between members as computer-mediated learning will not automatically lead to effective collaboration. the five-stage model (salmon, 2003) includes the interactivity bar, which assumes interaction between the e-moderator and group members. this is also implied through the engagement expected throughout the stages, including online socialisation and information exchange. salmon (2003) also discusses the use of structured exercises and assessment that encourage student participation and group functioning. group working is obviously a fundamental requirement for constructivist approaches to learning, and consideration of initial group composition or socialisation processes will aid its development. the ladder includes group composition and working as a facet for consideration across different learning approaches that might be employed. although the need for effective group functioning is seen as increasing as more constructivist approaches are used, there is recognition that students may be challenging the five-stage model for e-learning 47 using interactive learning materials or gathering information as part of a group, or indeed use these approaches to inform a community of practice. longevity of engagement. longevity of engagement is a feature affecting the online learning environment, particularly if a constructivist learning approach is used. indeed, those employing the five-stage model have commented on the difficulties of use in the short time-frame proposed (lisewski & joyce, 2003; chowcat, 2005). the period of engagement is of particular concern where there are expectations that a community of practice will emerge (moule, 2005, 2006a). the role of shared histories in the development of a community of practice, it is suggested, will render unfeasible their development in short term courses (fowler & mayes, 1999). a shortened period of engagement can affect the commitment of the participants, as seen in this research where one student preferred to maintain contact with existing peers rather than invest in a short lived community. the majority of the information i am getting is obtained from fellow students not in my [online] group. (d5) it will also adversely affect the development of shared repertoire, which will emerge over a long period of engagement. given these concerns, longevity of engagement is a component of the ladder, seen as increasing in emphasis at the top of the ladder, when participants are engaging in more constructivist learning approaches. the ‘side’ representing engagement extends the full length of the ladder, acknowledging the need for participant engagement at every level of learning. conclusions this paper has offered a critical review of the five-stage model, comparing it with the e-learning ladder and highlighting the different interpretations of e-learning presented in each. discussions have also identified areas of similarity seen, where both the five-stage model and the e-learning ladder include the factors necessary to enable engagement in e-learning environments such as access and technical support. in presenting such dialogue the paper makes a contribution to the current agendas that seek to understand learner participation in, and experiences of, e-learning. the discussion challenges the view that e-learning in higher education must reflect a constructivist approach to learning, reinforcing the opportunities for e-learning to support instructivist approaches, blended learning and classroom supported delivery. the e-learning ladder is offered as a conceptual model that provides one interpretation of e-learning, which in the future might be expanded to incorporate the potential of mobile technology deliveries within higher education. notes 1. see www.jisc.ac.uk/elearning 2. see www.techdis.ac.uk 48 p. moule references adams, a. (2004) pedagogical underpinnings of computer-based learning, journal of advanced nursing, 46(1), 5–12. andrews, t. & schwarz, g. (2002) preparing students for the virtual organisation: an evaluation of learning with virtual learning technologies, educational technology and society, 5(3), 54–65. available online at: http://ifets.ieee.org/perodical/vol_5_2002/andrews.html (accessed 11 march 2003). andrusyszyn, m., iwasiw, c. & goldenberg, d. (1999) computer conferencing in graduate nursing education: perceptions of students and faculty, the journal of continuing education in nursing, 30(6), 272–278. armitage, s. & o’leary, r. (2003) a guide for learning technologists (york, learning and teaching support network generic centre). atack, l. (2003) becoming a web-based learner: registered nurses’ experiences, journal of advanced nursing, 44(3), 289–297. barker, p. (2002) on being an online tutor, innovations in education and teaching international, 39(1), 7–13. beetham, h. (2005) e-learning research: emerging issues?, alt-j, 13(1), 81–89. bell, j. (1999) the biology labs on-line project: producing educational simulations that promote active learning, interactive multimedia electronic journal of computer enhanced learning. available online at: http://imej.wfu.edu/articles/1999/2/01/index.asp (accessed 5 june 2003). bender, t. (2003) discussion-based online teaching to enhance student learning. theory, practice and assessment (sterling, va, stylus publishing). bennett, s. & marsh, d. (2002) are we expecting online tutors to run before they can walk?, innovations in education and teaching international, 39(1), 14–20. boyle, d. & wambach, k. (2001) interaction in graduate nursing web-based instruction, journal of professional nursing, 17(3), 128–134. chalk, p. (2001) learning software engineering in a community of practice—a case study, paper presented at the cal 2001 conference, warwick university. available online at: http:// www.ics.ltsn.ac.uk/pub/conf2001/papers/chalk.html (accessed 24 may 2004). chowcat, i. (2005) models of e-learning: the importance of context, paper presented at alt-c 2005 12th international conference, manchester university, uk. cooper, v. & mcconnell, m. (2000) development of a web-based learning tool for undergraduate health professions studying applied anatomy, alt-j, 8(1), 62–69. curtis, e., hicks, p. & redmond, r. (2002) nursing student’s experience and attitudes to computers: a survey of a cohort of students on a bachelor in nursing studies course, information technology in nursing, 14(2), 7–17. department for education and skills (2005) the e-strategy—harnessing technology: transforming learning and children’s services (london, dfes). available online at: http://www.dfes.gov.uk/ publications/e-strategy/exec.shtml (accessed 7 october 2005). ellaway, r., dewhurst, d. & mcleod, h. (2004) evaluating a virtual learning environment in the context of its community of practice, alt-j, 12(2), 125–145. fowler, c. & mayes, j. (1999) learning relationships from theory to design, alt-j, 7(3), 6–16. good, m. (2001) on the way to online pedagogy, in: j. stephenson (ed.) teaching and learning online: pedagogies for new technologies (london, kogan page), 166–174. hara, n. & kling, r. (2000) students’ distress with a web-based distance education course: an ethnographic study of participants’ experiences (bloomington, in, center for social informatics, slis, indiana university). available online at: http://www.slis.indiana.edu/csi/wp/wp0001b.html. (accessed 14 october 2005). also in information, communication and society, 3(4), 557–579. her majesties stationery office (1995) disability discrimination act 1995 (dda) available online at: http://www.hmso.gov.uk/acts/acts1995/1995050.htm (accessed 7 january 2005). challenging the five-stage model for e-learning 49 her majesties stationery office (2001) special educational needs and disability act 2001 (senda). available online at: http://www.hmso.gov.uk/acts/acts2001/20010010.htm (accessed 7 january 2005). hildreth, p. (2004) going virtual: distributed communities of practice (hershey, pa, idea group publishing). hong, k.-s., lai, k.-w. & holton, d. (2003) students’ satisfaction and perceived learning within a web-based course, educational technology and society, 6(1). available online at: http:// ifest.ieee.org/periodical/vol_1_2003/hong.html (accessed 11 march 2003). hughes, m., ventura, s. & dando, m. (2004) on-line interprofessional learning: introducing constructivism through enquiry-based learning and peer review, journal of interprofessional care, 18(3), 263–268. joiner, r. (2004) supporting collaboration in virtual learning environments, cyberpsychology and behaviour, 7(2), 197–200. jones, n. & peachey, p. (2005) the development of socialization in an on-line learning environment, journal of interactive online learning, 3(3), 1–20. available online at: http://www.ncolr.org/jiol/ issues/showissue.cfm?volid=3&issueid=12 (accessed 11 november 2005). lisewski, b. & joyce, p. (2003) examining the five-stage e-moderating model: designed and emergent practice in the learning technology profession, alt-j, 11(1), 55–66. mcmahon, j., gardner, j., gray, c. & mulhern, g. (1999) barriers to student computer usage: staff and student perceptions, journal of computer assisted learning, 13(4), 302–311. monteith, m. & smith, j. (2001) learning in a virtual campus: the pedagogical implications of students’ experiences, innovations in education and teaching international, 38(2), 119–127. moule, p. (2002) evaluation of a basic life support cd-rom: its effectiveness and user experiences, educational technology and society, special edition evaluation of learning technologies in higher education, 5(3), 163–174. available online at: http://ifets.ieee.org/ periodical/vol_3_2002/v_3_2002.html (accessed 7 october 2005) moule, p. (2005) e-learning for healthcare students: developing the communities of practice framework. ed.d. thesis, university of the west of england, bristol. moule, p. (2006a) e-learning for healthcare students: developing the communities of practice framework, journal of advanced nursing, 53(3), 370–380. moule, p. (2006b) e-communities, in: s. glen & p. moule (eds) e-learning in nursing (basingstoke, palgrave). moule, p. & gilchrist, m. (2001) an evaluation of a basic life support cd-rom, health informatics journal, 7(1), 28–35. nye, r. (1996) three psychologies. perspectives from freud, skinner and rogers (new york, brooks/ cole publishing). peters, m. (2000) does constructivist epistemology have a place in nurse education?, journal of nursing education, 39(4), 166–172. rogers, j. (2000) communities of practice: a framework for fostering coherence in virtual learning communities, educational technology and society, 3(3), 1–12. available online at: http:// ifets.ieee.org/periodical/vol_3_2000/eo1.html (accessed 11 march 2003). rusby, n. j. (1979) an introduction to educational computing (london, croom helm). salmon, g. (2000) e-moderating: the key to teaching and learning online (london, kogan page). salmon, g. (2002) e-tivities (london, kogan page). salmon, g. (2003) e-moderating (2nd edn) (london, routledge falmer). seale, j. (2004) the development of accessibility practices in e-learning: an exploration of communities of practice, alt-j, 12(1), 51–63. somekh, b. & pearson, m. (2000) inter-cultural learning arising from pan-european collaboration using electronic communications: sub-text or substance?, paper presented at the british educational research association annual conference, cardiff, uk, september. thorpe, m. (2002) rethinking learner support: the challenge of collaborative online learning, open learning, 17(2), 105–115. 50 p. moule vygotsky, l. (1978) mind in society (cambridge, ma, harvard university press). wenger, e. (1998) communities of practice (cambridge, cambridge university press). wong, s. l., jahil, h. a., ayub, a. f. m. & hong, t. s. (2003) teaching a discrete information technology course in a constructivist learning environment: is it effective for malaysian preservice teachers?, the internet and higher education, 6(2), 193–204. yin, r. (1994) case study research. design and methods (2nd edn) (london, sage). evaluation in a project life-cycle: the hypermedia camille project thierry chanter université de clermont ii and université de paris v in the cal literature, the issue of integrating evaluation into the life-cycle of a project has often been recommended but less frequently reported, at least for large-scale hypermedia environments. indeed, cal developers face a difficult problem because effective evaluation needs to satisfy the potentially conflicting demands of a variety of audiences (teachers, administrators, the research community, sponsors, etc.). this paper first examines some of the various forms of evaluation adopted by different kinds of audiences. it then reports on evaluations, formative as well as summative, set up by the european camille project teams in four countries during a large-scale courseware development project. it stresses the advantages, despite drawbacks and pitfalls, for cal developers to systematically undertake evaluation. lastly, it points out some general outcomes concerning learning issues of interest to teachers, trainers and educational advisers. these include topics such as the impact of multimedia, of learner variability and learner autonomy on the effectiveness of learning with respect to language skills. introduction this paper reports on a series of evaluations undertaken in the countries which participated in the camille project.1 the principal aim of this european project has been the development and delivery of hypermedia courseware in dutch, spanish and french. the courseware encompasses the training of general linguistic competencies for beginners (dutch and spanish) as well as competencies related to the use of language for specific purposes (french). the target audience includes students in science or business, and technicians or engineers from smes (small and medium enterprises small businesses). this report may be of interest to two kinds of reader of this journal, as follows. • each one of our packages exploit the full range of hypertextual and multimedia facilities currently provided by standard computing platforms. furthermore, each package offers learners a large-scale learning environment capable of supporting autonomous study. consequently, these preliminary outcomes relating to the way 54 alt-j volume 4 number 3 camille has been practically used by learners and to its effectiveness are of potential interest to teachers, trainers and educational advisers. • the various experiments conducted by the teams and integrated into the process of software development will be of interest to computer-aided learning (cal) developers in general. indeed, within the cal literature, the issue of integrating evaluation into the life of a project, i.e. either in the course of the development or at the end of it, has often been recommended but much less frequently reported, at least for this type of environment. the paper discusses the constraints, advantages and drawbacks of actually adopting such a procedure. in order to make clear both the nature of the experiments undertaken within camille and the significance of the results obtained, a brief preamble on evaluation is necessary. the term evaluation is widely used by various groups connected with computer-assisted language learning (call) but frequently approached from very different perspectives, and this can leave the reporting of results open to misinterpretation. at one end of the spectrum there is an increasing pressure on researchers and developers to adopt more methodological and scientific procedures, and, at the other end, educational advisers and executives constantly require concrete and positive results before extending their support to call. camille is one project, among an increasing number of others, which has had to try to make these potentially contradictory viewpoints coexist. below i describe various aspects of evaluation in language learning and in cal. after this, i set out the initial requirements and achievements of the camille project and introduce the common features of the different experiments. this is followed by detailed evaluations made in two countries, and a report of the main general outcomes, summing up our experience of managing evaluation as an integral part of a project life-cycle. preamble on evaluation in order to delimit the framework adopted in this research, this section presents the principal functions of evaluation, the initial questions in the design process, the overlapping forms of evaluation, and the evaluation procedure. functions of evaluation for almost thirty years, a distinction has been frequently made between two principal functions of evaluation: formative evaluation and summative evaluation. this distinction exists in language teaching (lussier, 1992) as well as in cal (knussen et al, 1991; demaiziere and dubuisson, 1992; mark and greer, 1993), but they are differently interpreted. in language teaching, formative evaluation consists in regularly diagnosing the learner's state of knowledge, abilities, attitudes. it is undertaken for learners in order to let them know their current position with respect to their final goals; and for teachers to gain information that may lead them to adjust and adapt their teaching before the end of the course. in cal, formative evaluation also occurs before the end of the implementation phase. it is intended to help the designers review their progress towards achieving the goals of an educational innovation. it is set up by designers, and involves a few learners who are carefully observed in order to assess whether they use the software as intended. 55 thieny chanter evaluation in a project life-cycle: the hypermedia camille project such aspects as interface, human-machine interaction, learner strategies, hardware configuration and computing architecture are observed with rather informal methods. this process brings both detailed and general information, which may lead to surface changes (correction of bugs) or more profound changes in the design and the development. it also provides insights into the way the courseware will perform when integrated into a real-life learning situation. in language teaching, summative evaluation comes at the conclusion of a course, or a programme, in order to measure the level of proficiency acquired by a learner with respect to normative goals explicitly fixed by the learning institution. it is a global measure which compares the performance of learners. it is intended to certify learners in order to give them credits, to recommend an orientation, or to check the effectiveness of the course or programme. in cal, summative evaluation is concerned with the evaluation of completed systems. its purpose is to measure the effectiveness of an innovation in terms of its stated aims. it is intended for trainers, centres and designers to assess the suitability of the software for certain tasks and users, or to compare it with other products already in use. in both cases, summative evaluation has to be undertaken in real-learning settings, and to involve a larger number of subjects than formative evaluation. since the central topic of this paper is the role of evaluation in developing a software package, i will adopt the cal standpoint rather than that of language teaching. moreover, since the computing environments developed in the camille project have been designed to support autonomous learning, some aspects of the language-teaching model would be inappropriate. however, beyond the discrepancies between cal and language-teaching models of evaluation, there is a common feature which distinguishes them from the issue of assessment. evaluation is not a judgemental but a decision-making process. since outcomes may be interpreted by various audiences (e.g. designers, teachers, institutions) in order to make lasting changes, the framework for setting up an evaluation and its procedure will be examined hereafter. initial questions in the design of an evaluation evaluating a language program, or any piece of call software, is a complex process. there follow some key questions (taken from nunan, 1992, chapter 9) that should be answered before starting any evaluation. objectives: what is the purpose and who is the audience of the evaluation (for whom is it made)? methodology: what principles of procedure should guide the evaluation? what tools, techniques, and instruments are appropriate? material constraints: who should carry out the evaluation? when should it be carried out? what is the time-frame and budget? release: how should the evaluation be reported? it may seem obvious that it is extremely important to clarify, from the beginning, the goals of the evaluation. however, it is not a straightforward task. let us consider an innovation. relationships are not clear at all between the original working hypotheses of designers, the actual achievement, and the selection of precise experimental variables: a 56 ait-j volume 4 number 3 shift may have appeared between the starting and end points; an innovation may have unexpected effects (it may not raise the level of proficiency, but the learner's motivation); comparisons with other existing learning environments may be problematic simply because they are so different. for example, determining a scale for measuring effectiveness with respect to communicative goals and specific purposes was an expected outcome, in itself, in the camille project. fixing the objective of an evaluation is again not always easy when the audience is diversified: designers, teachers, administrators, and funding bodies often have different perceptions. if learning objectives need to be elicited, they also need to be associated with precise forms of evaluation which are themselves associated with different methodological approaches. below, i extract overlapping forms of evaluation from one (knussen et al, 1991) out of many possible presentations. • experimental. a limited number of clearly defined variables are scientifically measured, usually based on statistical inferences. the laboratory is the traditional setting for evaluations which generally have a formative function. if such a form is considered as more scientific, its relevance to real learning settings is problematic. • research and developmental. the purpose is to apply quasi-experimental methodologies, including preand post-tests, in situations closer to real learning settings. this form, which many evaluations of cal systems try to adopt, also requires clear statements of measurable objectives. they may be easier to guarantee in scientific or industrial environments than in educational ones. they more often concern summative than formative functions. • illuminative. isolating variables and associated parameters, as well as quantifying measures, is hard to achieve in real learning settings, especially if estimation of the impact of social factors and the participant's views on the meaning of educational innovations are at stake. consequently, methods, essentially qualitative and usually based on observations and interviews, are applied to 'illuminate' important factors rather than to test hypotheses. associated pitfalls here range from the risk of observers' obtrusiveness to findings which cannot be generalized to apply to other settings. • teacher as researcher. since teachers play a prominent role in the integration of call systems into the curriculum, it seems natural to let them take charge of the evaluations. biases (e.g. subjectivity, role-conflict, work overload) introduced by this form suggest that it should be used only in addition to other approaches. • case studies. understanding the effects of situational and personal factors in the use of innovative software is generally based on the detailed study of a restricted number of learners. however, a generalization of the findings to other situations may be difficult. in camille, two forms of evaluation have mainly been used: the research and developmental one, and the illuminative one. once the form of the evaluation has been determined, material constraints need to be appreciated before performing the evaluation procedure. the first step of the procedure consists in designing the whole evaluation. the initial task of the second step is the 57 thierry chanier evaluation in a project life-cycle: the hypermedia camille project construction of the instruments: materials for the tests, questionnaires and forms, and extra materials for the control group, if necessary. data collection and analysis follow, according to the methodological approach chosen. the third step, drafting the report, learning and deciding from it, may not be the last one. in formative evaluation, immediate decisions may be taken, followed by changes which then will be measured a second time. the variety of tasks and their co-ordination create genuine obstacles for the successful completion of a project. who will the evaluators be? what is the time-frame and the budget? this may explain why many call developers seem reluctant to include an evaluation procedure as part of their project. the last issue, raised in the initial questions, refers to the release: how is the evaluation to be reported? on one hand, evaluation is often described as a public act which should be open to inspection. on the other, unsatisfactory findings, and/or disagreements between participants, may impede the publication of a final report. alternatively, interesting findings may be over-generalized if the final conclusions are not clearly delimited. general aspects of the camille evaluation the european lingua camille project started in 1993 and will finish this year. descriptions of the project and of its theoretical standpoints can be found in ingraham et al (1994); chanier (1996); and pothier (in press). in this section i recall only its initial requirements and its main achievements. from there, i examine the purposes and audiences of the various evaluations, the common features shared by the different experiments, and details of evaluations. initial requirements of the whole project the camille project is aimed at conducting a large-scale experiment touching on issues arising from both pedagogical and software-engineering viewpoints. from a pedagogical viewpoint, hypermedia technologies are often presented as an opportunity to enhance language learning. although these factors are often assumed to play an important role in the acquisition process, as yet there have been no large-scale experiments based on their use. camille was thus seen as an opportunity to undertake such an examination in a multi-cultural environment. the objective was the construction of an environment that would provide learners with all the tools and information, short of a live teacher, that they might need to undertake a specific level of course in the target language. one consequence was the integration of books/resources (on lexicon, culture, function, grammar) with the textbook (the course proper) on the same desktop. another consequence was the mode of its use and of its integration into a whole curriculum: camille was designed to be used by well-motivated adults, who may or may not be engaged in formal education or training and who may or may not have access to a tutor. thus the emphasis was on autonomy. from a software-engineering viewpoint, hypermedia programming tools are often recommended as an opportunity to speed up courseware development, and therefore make cal a realistic complement for training learners in and out of the academic world. but production of courseware in hypermedia also dramatically increases the number of skills required, and up to now our experience in reusing modules of software, or shared knowledge 58 volume 4 number 3 for large-scale software, is still very limited. the camille project was supposed to help to gain a clearer understanding of trans-national courseware development. as a starting point, it was decided to use a common template for development, a template which consisted of a software and hardware platform, created by our british partners in 1991/92. the effectiveness of the software-engineering viewpoint was enforced by the decision to launch a commercial release at the end of the project, i.e. at the beginning of 1996. main achievements a few months before the conclusion of the project, the main courses finalized or near completion are as follows. • espanol interactivo, interactif nederlands, and france interactive. these three packages are respectively designed for the training of general linguistic competencies for beginners in spanish, dutch, and french, and developed in spain, the netherlands, and the uk. • travailler en france. this package has been designed for the training of competencies related to the use of language for specific purposes (lsp) for intermediate-advanced level french, and developed in france. each package includes two cds which run on a standard, basic hypermedia pc platform, namely the international standard mpc2. this has the minimal equipment to play fullmotion video, and offers good quality for recording and playing sounds. each disc gathers approximately 30 minutes of original video, plus other oral, graphical and textual data, on top of which are built resources, and several dozen activities, which offer more than 20 hours of study to the learner. while, at present, debugging and some coding processes are still under way, camille partners are fixing the legal aspects in order to start the commercial release of the most advanced courses. common features of the evaluations following the general framework discussed above, i review here the common features of the evaluations undertaken by all the camille partners. three kinds of audiences with their respective purposes can be distinguished. the first encompasses the european union and publishers, as external (to the call community) actors which intervene in the project life. the former (the eu) partially funded camille (actually for less than a quarter of the total budget), added its own requirements, and annually examined achievements before deciding any extension of funding. the latter (the publishers) have recently undergone internal restructuring in order to be prepared to release multimedia software. most limit the major risks linked to innovations by expecting developments to be supported by small, recent private ventures. furthermore, they are not accustomed to dealing with academic institutions. for them all, evaluations were intended to assert our reliability, by proving that learners could turn their hand to our courseware in real settings, by convincing them that academics could challenge private companies and be more transparent when performing evaluations as public acts open to inspection. the second kind of audience is the call community, which includes teachers and 59 thierry charter evaluation in a project life-cycle; the hypermedia camille project researchers. the pedagogical perspectives outlined above needed to be made explicit. the purpose here is twofold: firstly, measuring what kinds of language skills multimedia technologies can help practise, what sorts of learning strategies are performed in hypermedia environments, and how effective autonomous learning is in various settings; secondly, scaling effectiveness with respect to communicative goals and specific purposes. the latter point refers to the problem of finding criteria by which educational objectives can be measured: how can we assess the learner's ability to master knowledge and skills, mobilized around the specific purposes of each piece of courseware, to transfer them and create new pieces of discourse (cf. de landsheere's trilogy, 1984)? developers constitute the third kind of audience. the purpose here was to appreciate to what extent formative evaluation is necessary for adapting and debugging the learning environment, to perform summative evaluation to clarify the software goals (i.e. exactly identify what can be measured) and to determine the constraints and overheads brought upon the whole project. the different camille research teams set up evaluations, located either in their own institutions or in neighbouring ones. this happened over 14 months (1994-1995) during implementation, or at the end of large parts of it. no extra budget, nor extra human resources, were available. the results of these evaluations are being reported in three different ways: the final report to the european union, conferences such as eurocall (emery et al, in press) and academic papers. details of the evaluations in the camille project, objectives and methodologies varied from one research team to another. as an illustration, in this section i detail evaluations undertaken on interactif nederlands and travailler en france. results drawn from france interactive and espanol interactivo are included in the general outcomes presented in the next section. evaluation at hebo (de haagse hogeschool, the netherlands) the dutch camille team performed both formative and summative evaluations. the formative side of the evaluation was designed as a two-round experiment. as soon as data was analysed, changes were made and new experiments were based on the modified software. the aim of the summative evaluation was to compare the software with local classroom learning. this second side directly interested the hebo managers and the local teachers. the school supervised more than a thousand university dutch or foreign students who needed intensive training in several languages for professional purposes (legal or business). it offered a strong integration of call into the curriculum: nearly 50% of the students' work time in language learning was organized around free access to computers. heads of the school consider the familiarization with the dutch language and society by foreign students as an important factor of integration into a country where they are spending several years. of course, dutch is not a 'survival' language (learners can easily talk english and be understood by anyone in everyday life), but attendance in dutch classrooms is strongly encouraged, though not mandatory, and learners' credits can easily be transferred. it was thus decided that learners who learned dutch only through interactif nederlands would take the same oral examination as the other learners of dutch who attended classroom sessions. 60 aa-j volume 4 number 3 the experiment took place at the hebo in the multimedia, free-access room. evaluators used a network version of the software on computers with the recommended hardware configuration. sixty local students were involved, on a voluntary basis. they were true beginners in dutch, but experienced language learners (dutch often being their third language) had a low motivation for learning dutch, and only basic experience with computers. learning tasks were organized around half of the software, which represented 30 hours of work, distributed over 10 weeks, with free-access conditions. learners had to fill in questionnaires and were interviewed at the end of each session. evaluators also made non-systematic observations. data from 14 students was analysed for the formative part of the evaluation. this analysis will not be detailed here, but the lessons evaluators learned from this experiment will be mixed with the other general outcomes in the next section. the final examination was organized by the usual dutch teachers, not by the evaluators. marks and teachers' comments on the call group showed that results were neither better nor worse than usual. since the timing and the assessment procedures were the same as for the live course, the software would appear to be efficient in this sort of situation and with these types of learners. evaluations in cavi lam (france) formative and summative evaluations of module 1 of travailler en france were organized at two different stages of the project: the formative evaluation in october 1994, at the very end of the development of the prototype of module 1, and the summative one in june 1995, after changes and debugging had been finished on module 1 and while module 2 was under development. before considering the details of these evaluations, it will be helpful to consider certain common features local students, who were following full-time language training periods of 1 to 6 months in length, participated in the evaluation. they were between 21 and 47 years of age, with an average age of 25, coming from various continents and cultures. all were intermediate (200 hours) or advanced (400 hours) learners of french, with french often being their third language. they had good professional motivation for learning, either because they already had a job, or were seeking work where the mastering of specific skills in french was important, or because they wanted to attend french universities. they had a mixed experience with computers, some being almost computer-illiterate as they came from countries where computers are not part of the work or study environment. both evaluations were undertaken with module 1, where the specific purpose is to learn how to apply for a job in france. this makes a noticeable difference from other camille courses which are for general purposes (chanier, 1996). the module is built around one main task: making a job application. knowledge bases and activities allow learners to fulfil the task and immerse them in a socio-cultural context which determines the architecture of the software. the story-line of the module presents two characters who are very different in nature and who encounter a series of representative situations, for example how to find appropriate information and acquire experience in the employment market; how to write a letter of application and a cv in the french way; how to make an appointment on the telephone; how to handle an interview. linguistic knowledge and activities have been designed from the task context, but do not have top priority. the learning tasks require a total amount of 20 to 25 hours' work over three weeks. the 61 thierry oionier evaluation in a project life-cycle: the hypermedia camille project learners used the software during the time usually allocated for practical work in their training, and had further opportunities for free access. formative evaluation one purpose of the formative evaluation was to measure the performance of the courseware. the second one focused on how effectively the kinds of activities and resources available matched the learners' strategies and interests. the sample population was limited to five volunteers because we wanted one of our observers always to be present. they could work alone or in a group. the observer, who acted in a non-obtrusive way, either videoor audio-recorded all the sessions, and took detailed notes on the learners' moves, selections and timing. learners filled in preand post-questionnaires and had a form to fill in at the end of every session, followed by a short interview. through this procedure we were able to collect detailed information about the learners' behaviour and reactions as well as their (positive) comments. all this helped us to make subsequent adaptations. details are discussed below, but one point is worth mentioning at this stage because it relates to the lsp aspect of our software. even when learners were not directly, personally concerned with seeking a job, they all (even subjects of the summative evaluation) indicated that the experience provided important discoveries concerning sociocultural aspects of the target-language country, and of its everyday native language. apprehending variations in the target language and links between language and complex situations encountered daily by natives is an efficient way of raising language awareness; as such, it is an important aspect of second-language learning. summative evaluation the purposes of the summative evaluation were threefold: • assessment of the suitability of the first lsp courseware with respect to the local learners, • comparison with autonomous (audio + paper) learning, • measurement of the impact of hypermedia call on vocabulary learning. for this second experiment, the audience was not limited to the project team. the cavilam staff were also interested in the outcomes, and took over the supervision of the learning task, acting as counsellors. the project team only handled the various tests. subjects were divided into two groups on a voluntary basis: group 1 (gl), the paper-and audio-based group, comprised six people; group 2 (g2), the call one, seven. for gl we extracted large parts of textual data contained in the software activities and resources, and all the sounds of the dialogues. they then had a document and audio-cassettes to work with. they also had access to paper-based dictionaries available in the language laboratory. we prepared preand post-questionnaires, the post-questionnaire contents being different for gl and g2. we also translated into french and administrated the sill (strategy inventory for language learning test: ehrman and oxford, 1990) which allows learners clearly to indicate which sort of strategies they usually apply when learning a language generally. results show to what extent they use (and are aware of using) appropriate strategies for remembering more effectively, using mental processes, compensating for missing knowledge, organizing and evaluating their learning, managing emotions, and 62 alt-j volume 4 number 3 learning with others. subjects also passed a preand a post-test on vocabulary (preand post-tests were identical) and a post-test to assess communicative competence in the same domain. for the latter one, called the main post-test, we created original aural and textual materials. subjects had to write their answers and essays. the main post-test had three parts: an aural comprehension of an interview which included subjective appreciation of the applicant's situation; a comprehension and a written production of part of the exchanges in a dialogue on the telephone; and the writing of a letter of application for a post-profile described in an advertisement. this test was not ready when the experiment started, so we could only use it as a post-test. the two groups appeared not to be equally balanced. analysis of subjects' answers in g2, the computer-based group, showed that they used more varied strategies and were more self-conscious of the way they usually learned. they proved to have better lexical knowledge than gl in the pre-test. both groups progressed in this domain, gl slightly more than g2. this may not be very surprising since the lexical test was difficult (the emphasis was put upon the relationship between words and phrases, and collocations; semantic relationships, grammatical structures and relational constraints of lexical phrases were required to be understood). within this context, progression in subjects with lower-level knowledge is easier. as regards the main post-test, there was not much difference between gl and g2. this result is not easy to explain since samples were limited in both groups. however, we noticed that gl behaved as if they were competing against g2. the learners in gl did, however, have to find by themselves extra resources which were easily available in the software: for example, we observed that gl learners frequently used dictionaries. gl strongly protested against their learning materials which they found boring, while g2 found much interest in the software. learning may have been a harder process for gl, but both groups satisfactorily learned, and passed their exams (vocabulary and main post-tests), which was what we were expecting. conclusions general outcomes multimedia the activities which the learners rated most highly were the video-based and the audiobased activities, in order of preference. when asked to evaluate the activities upon quality alone, this order of preference was reversed. when learners found quality of sound unsatisfactory, they expressed their view strongly, although they never complained about the definition or size of the video material. this result supports our original decision to use the basic mpc2 standard, since limitations to the quality of video are less important given the range of functions we assigned to video. in camille, as in other call environments, video is primarily used: • to put language into context, thus to raise motivation; • to support the interpretation of the linguistic contents of utterances: in simulation activities, looking at the speaker's face may bring information on the pragmatic contents of the message (happiness, irony, discontent, etc.), and when pronunciation activities are essential, as in the first lessons of the dutch course, focusing on the speaker's lip movements facilitates comprehension and production of phonemes. 63 thierry chanter evaluation in a project life-cycle: the hypermedia camille project for such functions, the video supports sound. this means that when use of video is suppressed (for example, in some telephone-based activities where we wanted to increase the level of difficulty), the linguistic content is still comprehensible, provided that the quality of the sound is very good. however, learning a language is not reducible to purely linguistic knowledge. kinesics and proxemics are also very important (feldman and rive, 1991). in real communication settings, the hearer not only interprets the speaker's message from its linguistic content, but also from his/her gestures, location, etc. in situations such as interviews or negotiations, the issue not only relies on what is said but also fundamentally on the predisposition of the various parties a predisposition which will, be interpreted according to a protocol of behaviour and gestures. in foreign-language learning, these aspects are never neglected in live courses. if we want to do as well in call, we need, to study other functional uses of video. in one module of the french for business courseware, we have designed three activities on gestures, which can either support the verbal message or completely replace it. however, no experiments have yet been made concerning this new type of activity because its development was completed only after the evaluation phase. as regards sound, the results of our experiments showed that we had underestimated the potential of simple technologies which allow recording and producing sounds of high quality. in camille, it is possible for the learner to record him/herself in almost every activity. in some of them, self-recording is an accessory, but in others (like simulation activities) it is essential. experiments showed that even if all learners regarded it as important to have self-recording facilities, there was a large discrepancy between the way they claimed to use these resources and the extent to which they actually did. this can be explained by learners' lack of self-assurance, and by the lack of explicit stress in the first versions of our software on this important and preliminary step for the support of oral production skills. we have now switched to simple solutions such as adding signposts and interactive comments in relevant activities, and in the general learners' follow up. in fact, the cal environment must indicate to every user the importance of adopting effective, interactive strategies, such as re-recording oneself several times and making a (subjective) comparison with the model (as we observed some learners actually doing). learner variability in all the questionnaires, learners almost unanimously expressed their preference for interactive activities compared to more passive ones, but they disagreed about which ones they considered better (with the exception of simulations, which were always highly rated). learners also often stressed the fact that even if they found communicative activities attractive, basic linguistic activities, on grammar or vocabulary, should not be forgotten. in the case of interactifnederlands, for example, this led to an adjustment of the balance between both types of activities by adding new, more linguistically oriented, tasks. the learners' reaction was not necessarily a plea for activities of a 'traditional' nature. linguistic activities can be designed in new ways. thus learners found our presentation of vocabulary knowledge in lexical networks in travailler en france very appealing. learner variability appeared not only in opinions but also in ways of working with the 64 alt-] volume 4 number 3 courseware. learners followed very different routes in the scheduling of their overall work: some undertook activities strictly in the order suggested by our presentation; others took a quick overview of the whole contents and of the various kinds of activities (which were signposted), then started with the ones they preferred. learners also performed activities in very different ways, some trying to finish them quickly without paying much attention to instructions or without looking at the associated resources (they generally then got stuck and had to restart), others self-monitoring their task by first carefully considering in which order to proceed, looking at the cues and available resources. some were systematic, relying on repetitions of self-recording and exhausting the various possible alternatives. some systematically took notes before actually performing any activity. some verbalized their thoughts and reactions, whereas others were almost completely silent. when group-work occurred, and when skills were complementary inside the group, effective collaboration took place with one taking over the interaction with the system, while a second controlled the planning, or negotiated the knowledge. this learner variability is an important positive outcome. disagreement on attractiveness of activities showed that everyone found their own interest. variations in the way of using the software happened according to learners' personal characteristics. whatever our wishes may be in expecting learners to follow a particular route, individual variability remains the rule in language learning (ellis, 1994). one of the advantages of multimedia learning environments is the support they can give to these individual variations by offering different types of activity, practice of different linguistic skills, flexible navigation, access to resources of various kinds, and note-taking. autonomy camille has been designed for an audience of learners who are typically clients, professionals with clear demands and for whom flexibility and swiftness are essential criteria. sample populations involved in our evaluations mostly corresponded to this profile. furthermore, nearly all were experienced learners, either advanced learners of a second language, or beginners in a third. they were aware of their own preferred learning strategies, and used software in an autonomous way, evaluators and teachers, when present, being merely observers. from the learners' answers, and from our own observations, it is possible to underline the points which follow. • when we developed our software, we recognized the need to distinguish between activities and resources, but also the need for resources to be tightly linked to activities in order to make essential extra knowledge readily available within self-contained courses (chanier, 1996). the fact that learners did use these extra resources suggests that significant time and energy should be allocated to their development in such hypermedia environments. • software can be self-contained, but learners will still be looking for discussion and feedback with experts. it is still an open question whether these experts should be teachers acting as guides or counsellors, or native speakers. • self-access has been, as far as possible, the rule. learners have made it a basic requirement. insufficient provision of equipment and flexible access time within institutions may jeopardize the whole learning procedure. 65 thierry charter evaluation in a project life-cycle: the hypermedia camille project • autonomous learning situations have been explored only in training institutions. learners said they were willing to work alone, and to work at home. we have yet to investigate how this might affect learning outcomes. experimenting with access at work is yet another possible approach that should be considered. not surprisingly, the types of learners with whom we were concerned reacted very positively. they appeared to master the essential three domains for managing one's own learning (holec, 1990): methodological aspects, linguistic aspects and cultural background. we have collected no data for generalizing these outcomes to other types of learners. the experiment undertaken in teesside with true second-language beginners, lacking the self-assurance and motivation, was not conclusive. blin (in press) has also remarked that an insufficient level of confidence in using computers for language-learning purposes (which never appeared to be a problem with our experienced learners) may represent a major element in the learner's decisions to under-use computers as opposed to other materials in self-access centres. effectiveness and language skills it is now time to come back to the question of the effectiveness of such hypermedia software with respect to the four language skills. as pointed out above, the technology we relied on is more adequate for practising aural (listening) and written comprehension than aural and written productions. learners passed two summative tests, as described earlier, in hebo and in the cavilam. the former test was completely based on aural skills and thus included aural production. the latter test encompassed aural comprehension, written comprehension and written production. in order to appreciate these results correctly, it should be remembered that the evaluations involved small samples, related to specific types of learners, and in both places quality of results was not much better than that of more traditional approaches, live courses or audio-cassette methods. this quality is satisfactory because we were not expecting computing-learning environments to be much more efficient, but to represent an effective alternative which can be taken into account in autonomous learning situations, an alternative which possesses other advantages discussed in the previous sections. another open question is whether or not our results can be generalized to all experienced learners. evaluation as part of the development process the main goal of formative evaluation is to measure the performance of the courseware. it is a necessity for adapting the software, for debugging it, and for collecting essential information on timing etc., information which can then also help in preparing the usermanual delivered with the software. the procedure must involve real learners belonging to the target audience, and should be set up long before the end of the development. the elapsed time between the final release of the software and the evaluation phase is often as long as the duration of the development of the first version of the software which served in the evaluation. in general, a reduced protocol is sufficient, but if research questions are at stake, an extended protocol is necessary for setting up case studies. the whole evaluation procedure then becomes much more complex. the purpose of summative evaluation is to measure the effectiveness of the courseware in terms of its stated aims. we have pointed out several caveats: the summative evaluation is 66 alt-] volume 4 number 3 time-consuming; it requires adequate means for achieving it; many partners are often involved; and its results or its abandonment may be used against the project. since it represents a real risk for the whole project, the first question which should be answered before making any decision is: who are the audience? who really wants to know the outcomes? nevertheless, the organization of summative evaluations by project teams should happen more frequently. they are important for the research and pedagogical communities for deontological reasons, as follows. • they help to clarify the functional differences between the various sorts of software reports and the evaluation reports. for most software, the only accessible reports are commercial reports, written by publishers, or software reviews, written by external teachers or researchers. these reports may bring useful information, but they have the disadvantage of often being labelled as 'evaluations'. confusion with reports based on experiments involving real learners and following a methodological procedure should be avoided. • they minimize over-generalizations, either pro or con. an evaluation has specific aims. results can be interpreted only with respect to the restricted parameters which have been tested. unfortunately, papers are still published which either present evaluations as being aimed at definitely stating the superiority of cal over other learning methods, without defining parameters such as types of learners, types of skills, levels of proficiency, nor actual learning situations, or which, when they make explicit their restricted purposes, do not incorporate any detailed information. it is then impossible for readers correctly to interpret their results, and to undertake other evaluations in order to verify them. • they reinforce the idea that, in an evaluation, not only the software may be tested, but also the learning situations. a limited piece of software can be very useful, and on the contrary, a wonderful language package can be misused, depending on its integration into the curriculum, its access conditions, hardware configurations, etc. • they may offer instruments for measuring various aspects of so-called communicative competence. such references support the dialogue between designers and the second language acquisition community. potentially, a summative evaluation is also of direct interest for the project team itself. it represents an efficient way of clarifying the final aims of the software, and of estimating the inevitable shift between the initial hypotheses and the reality of the achievements. measuring tools make it possible to elicit how and on what grounds designers want their innovation to be estimated. since evaluation is a cumulative process, it forms a starting point from which other researchers are able to set up new experiments in order to extend the initial measures. tests can also be adequately joined to the software delivery in order to let learners evaluate themselves at the end of their training period. note 1 camille (which stands for computer-aided multimedia interactive language learning environment) has partly been financed by the european lingua programme. members of the camille consortium are: the university of teesside; universite blaise pascal and universite d'auvergne, clermont-ferrand; de haagse hogeschool, the hague; and universidad politecnica, valencia. 67 thierry chanier evaluation in a project life-cycle the hypermedia camille project acknowledgements i would like to thank all the teachers and researchers who participated in the camille evaluations, particularly ana gimeno sanz in universidad politécnica de valencia, jan brouwer in de haagse hogeschool, janina emery, chris emery and bruce ingraham in the university of teesside, the cavilam staff in vichy, maguy pothier, paul lotin and jérôme oilier in the université of clermont ii. references blin, f. (in press), 'integrating call in the negotiated learner-centred curriculum: a case study', eurocall '95 conference, valencia, spain, september, 1995. chanier, t. (1996), 'learning a second language for specific purposes within a hypermedia framework', computer-assisted language learning, 9 (1), 3-43. de landsheere, v. and g. (1984), définir les objectifs de l'éducation, paris: puf. demaizière, f. and dubuisson, c. (1992), de l'eao aux ntf: utiliser l'ordinateur pour la formation, paris: ophrys. ellis, r. (1994), the study of second language acquisition, oxford: oup. emery, c , ingraham, b., chanier, t. and brouwer, j. (in press), 'creating interactive multimedia callware: the camille experience', eurocall '95 conference, valencia, spain, september, 1995. ehrman, m. and oxford, r. (1990), 'adult language-learning styles and strategies in an intensive training setting', modern language journal, 74 (3), 311-27. feldman, r. s. and rimé, b. (1991), fundamentals of non-verbal behaviour, cambridge and paris: cup and editions de la maison des sciences de l'homme. holec, h. (1991), 'autonomie et apprentissage auto-dirigé: quelques sujets de reflexion', in les auto-apprentissages, actes des 6èmes rencontres de l'asdifle, paris: les cahiers de l'asdifle, 2, pp. 23-33. ingraham, b., chanier, t. and emery, c. (1994), 'camille: a european project to develop language training for different purposes, in various languages on a common hypermedia framework', computers and education, 23 (1/2), 107-15. knussen, c , tanner, g. r. and kibby, m. r. (1991), 'an approach to the evaluation of hypermedia', computers and education, 17 (1), 13-24. lussier, d. (1992), valuer les apprentisages dans une approche communicative, paris: hachette. mark, m. a. and greer, j. e. (1993). 'evaluation methodologies for intelligent tutoring systems', journal of artificial intelligence in education, 4 (2/3), 129-53. nunan, d. (1992), research methods in language learning, cambridge: cup. o'malley, j. m. and chamot, a. u. (1990), learning strategies in second language acquisition, cambridge: cup. pothier, m. (in press), 'travailler en france: un environnement informatique hypermedia pour l'auto-apprentissage sur objectifs spécifiques', revue de phonétique appliquée. 68 alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 103–117 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020103–15 © 2007 association for learning technology doi: 10.1080/09687760701470924 the influence of communication technologies and approaches to study on transactional distance in blended learning steve wheeler* university of plymouth, uk taylor and francis ltdcalt_a_246971.sgm10.1080/09687760701470924alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis152000000june 2007stevewheelers.wheeler@plymouth.ac.uk this paper explores the role played by communication technologies and study orientations in the amplification and reduction of transactional distance in blended learning. factor analysis and structural equation modelling of different communication modes (face to face, email and telephone) revealed that students experience at least some transactional distance when separated from their tutors. email was found to facilitate the highest levels of immediacy of dialogue for most students. the conclusion is that strategic students are best placed to benefit from blended learning, and that the effects of transactional distance could be analysed more deeply if two subvariables of dialogue were recognised. these are social presence (the perception of connectedness between students and their tutors) and immediacy (the temporal effects of dialogue). introduction learning technologies can provide ‘any time, any place’ learning opportunities (wheeler & vranch, 2001). reflecting this trend, the term ‘blended learning’ describes incorporation of distance and local learning activities. improved communication bandwidth and greater access to computers have blurred the boundaries of contemporary practice. consequently, the term ‘distance education’ may now be redundant, as geographical distance should no longer be a barrier between teacher and learner (dabbagh, 2004). instead, gaining a better understanding of psychological distance may hold the key to the successful teaching of distributed students. perceptual distances have profound effects on learners who are separated for long periods from their tutors. one-half of all distance learners fail to complete their * faculty of education, university of plymouth, drake circus, plymouth pl4 8aa, uk. email: s.wheeler@plymouth.ac.uk 104 s. wheeler studies (ross et al., 1991; moore & kearsley, 1996; simpson, 2004). lack of dialogue may heighten perceptions of distance, which may contribute toward such poor retention rates (dron et al., 2004). perceptual distance should therefore assume importance as a research focus. the means through which perceptual distance can be empirically measured and analysed constitutes the methodological basis of this study. moore’s (1973) theory of transactional distance was applied as an explanatory framework, with perceptual distance predicted by the variables of dialogue, structure and student autonomy. the study utilised entwistle’s (1981) approaches to study inventory and two additionally devised likert-scale inventories to measure the variables predicted to influence transactional distance. structural equation modelling was applied to enable visualisation of the relationships between these variables, and to assess the statistical power and validity of the research findings. transactional distance theory according to transactional distance theory (moore, 1993), perceptual distance is present in every educational transaction. the transactional distance between student and tutor, whether based upon perceptions of intellectual differentials, demographic variables such as age or gender, or cultural distances such as language, may create the potential for misunderstandings to occur. transactional distance theory predicts that psychological distance between students and tutors varies as a result of the structure and dialogue mix within transactions. if structure is high, dialogue decreases; whereas if dialogue is high, structure decreases. moore’s theory predicts that greater presence of dialogue in the learning transaction should reduce transactional distance, engender better understanding and lessen the potential for misunderstandings to occur. this study focuses on how communication technologies may mediate psychological distance. blended learning communication is represented across three modalities—the baseline measurement of face-to-face communication, and two distance-based technologies: telephone and email. research hypotheses the study set out to explore two hypotheses. the first was that students whose dialogue with their tutors is mediated by technology should report greater (i.e. more remote) transactional distance than students whose dialogue with their tutors is conducted through face-to-face communication, due to the reduction of social cues and a decrease or inhibition in the richness of communication that would normally be expected to be present in face-to-face dialogue. such reductions should amplify the potential for misunderstandings to occur and for a raised perception of distance due to a lessening of social presence. the second hypothesis was that distance learners should report greater transactional distance of communication with their teacher if dialogue is mediated through asynchronous text-based technologies. this may result in less satisfaction with the transactional distance in blended learning 105 learning experience due to a greater potential for misunderstandings to occur. where synchronous technologies are employed, distance learners should report less (i.e. closer) transactional distance, leading to the potential for less misunderstanding to occur during dialogue. this could be due to the tendency of synchronous forms of communication to sacrifice structure in favour of greater dialogue. synchronous technologies were also thought to provide more immediacy of dialogue, providing remote students with rapid support and guidance. sample in an opportunistic sample, 348 students enrolled on a variety of continuing professional development courses for teachers participated in the study. the participants studied across a broad spectrum of study modes mixing face-to-face on-campus learning with varying amounts of directed study undertaken both at home and in the workplace. participants were predominantly female, with 307 females and 38 males in the sample (three participants declined to report their gender) (table 1). this gender bias is consistent with similar recent studies where females constitute the largest percentage of students studying on humanities-based and liberal arts-based courses, such as teaching in the united kingdom (goodyear et al., 2003). as is common in continuing professional development, the sample was predominantly mature, comprised mainly of fully employed, part-time students. method participants completed a five-point likert-scale questionnaire, containing entwistle’s (1981) approaches to study inventory, the student support inventory (wheeler, 2002) and the communication mode questionnaire (see http:// www2.plymouth.ac.uk/distancelearning/). the student support inventory measured students’ perceptions of expected and received support and the communication mode questionnaire measured perceptions of dialogue, structure and transactional distance in different communication modes. data were collected over a two-and-ahalf-year period, as each student cohort commenced study. the questionnaire also contained demographic questions, and asked about expected tutor support. table 1. participant sample descriptive data age (years) number of questionnaire respondents for each mode male female missing data mean standard deviation face to face email telephone total 38 307 3 40.8 8.15 237 85 72 106 s. wheeler structural equation modelling the first stage of the structural equation modelling process required the creation of a model that would represent the theoretical interrelationships between the proposed variables of structure, dialogue and study approaches in all three modes of communication. through factor analysis, a first-order path model was constructed comprising of three latent variables. exogenous model figure 1 represents the proposed full ‘three-factor’ exogenous model. the ovals represent the three latent variables, while the rectangles represent the questionnaire items as observed, dependent variables that contribute to the latent variables. this was a theoretical model used as a template for the actual data, enabling visualisation of the potential relationships between observed and latent variables (factors). figure 1. theoretical full three-factor model used for confirmatory factor analysis following confirmatory analysis using eqs software, one item was removed from the model (asi11b—relating new learning to ‘real life’) as its inclusion reduced goodness of fit. the revised full three-factor model for approaches to study is illustrated in figure 2 complete with estimated factor loadings. figure 2. revised full three-factor face-to-face model with standardised solution single-headed arrows show standardised factor loadings, and double-headed arrows represent the inter-factor correlations. the double-headed arrow joining two of the variables on the extreme left of the model indicates a correlation between two error terms. factor loadings are equivalent to standardised regression beta coefficients, which were estimated using maximum likelihood estimation (miyake et al., 2000). asi1b asi18b asi13b asi4b asi22b asi12b asi6b asi29b asi11b strategic autonomy surface figure 1. theoretical full three-factor model used for confirmatory factor analysis transactional distance in blended learning 107 factor loadings, correlations and beta coefficients significant at the 0.05 level and below are indicated by asterisks. the fit indices for this revised model were within acceptable limits, producing a statistically non-significant χ2(16) = 21.962, p < 0.186, indicating that the model’s predictions did not significantly deviate from the expected data pattern. the comparative fit index (cfi = 0.97) and the normed fit index (nfi = 0.95) were clearly above the generally accepted cut-off point of 0.90 (byrne, 1994). strategic approaches to study do not preclude the use of surface approaches if required; for example, during intense revision for an impending written examination. this may explain the significant correlation between the surface and strategic factors (r = 0.43, p < 0.05). strategic learners can manage their own personal study routines better, and require less tutor input, developing skills to manage course content and monitor their own progress. they have better self knowledge and know how to evaluate the outcomes of their own learning to inform themselves of future directions for study (weinstein & van mater stone, 1996). the strategic label was thus applied to the factor explaining the revised items grouping. endogenous model the factor structure for the first theoretical model represents the face-to-face modality of communication between student and tutor, and also by extension the other communication modalities to be tested. 0.20* 0.75* 0.69* 0.47* 0.67* 0.20* 0.40* 0.27* 0.95* statistics: cfi = 0.97, nfi = 0.95, 2 (16) = 21.962, p = 0.186 rmsea = 0.038 surface strategic autonomy organising study time changing study conditions prompt at starting study precise study instructions more interested in qualification reading just enough important to do well need a lot of concentration -0.20 0.43* -0.15 figure 2. revised full three-factor face-to-face model with standardised solution 108 s. wheeler the second factor is an aspect of dialogue, based upon the work of shin (2003), and represents transactional presence. the third and final factor represents items related to immediacy, a proposed affordance of the technology mediated process of dialogue between teacher and student. the one-way arrows between factors represent the strength and direction of the assumed causal relationships between latent and dependent variables, and are measured as regression or beta coefficients. the two-headed arrows are correlations, indicating the strength of association between the latent variables. the next stage in the structural equation modelling process was to fit the complete model together by combining the exogenous model together with an endogenous model into a full face-to-face modality pathway matrix and then running a third, general, model fit analysis as shown in figure 3. figure 3. structural equation model (modality model: face to face). χ2(120) = 171.08, p < 0.001, cfi = 0.92, rmsea = 0.05. the eqs software analysis showed that the chi-squared value was statistically nonsignificant, at χ2(28) = 39.301, p = 0.145, indicating that the model’s predictions did not transgress data pattern boundaries. both the cfi (0.98) and nfi (0.98) were above the recommended 0.95 limit and were thus deemed acceptable. the root mean square error of approximation (rmsea) value achieved a good fit at 0.50 so the model was accepted as the baseline face-to-face model and used for all subsequent comparisons between communication modes. error terms and correlations were omitted from the above path model diagram to simplify its presentation. similar models were created to represent participants’ perceptions when using telephone or email. all data from these path models are found in table 2. organising study time controlling study conditions prompt at starting study precise study instructions more interested in qualification reading just enough important to do well need a lot of concentration immediacy presencesurface strategic autonomy -0.35* 0.95* -0.27 -0.37 -2.71 2.61 3.15 2.15 -2.29 structure 2.98 -0.29 0.76* 0.67* 0.47* 0.52* 0.12 0.44* 0.31* 0.45* quicker tutor response f2f f2f tutor contact essential tutor contact improved quality tutorcontact gave structure best communicate my needs f2f 0.77* 0.75* 0.86* 0.52 0.72 0.65 0.62 talking f2f reduced distance less isolation through f2f talk talking f2f lessen misunderstandings time to reflect on learning f2f f2f tutor accessibility 0.72* 0.52* 0.28* -0.11 figure 3. structural equation model (modality model: face to face). χ2(120) = 171.08, p < 0.001, cfi = 0.92, rmsea = 0.05. transactional distance in blended learning 109 results the full baseline pathway model for face-to-face tutor support represents student perceptions of course structure, social presence, and immediacy of dialogue. of these endogenous factors, the first represents moore’s concept of ‘structure’ and is measured through student evaluations of how important they perceived tutor contact, and whether this contact improved the quality of course delivery. also contributing to the factor of structure is the extent to which student felt dialogue with the tutor imposed structure upon their learning. the second factor of social ‘presence’ representing students’ perceptions of connectedness to their tutors is derived from their responses to questions relating to how they could best communicate their study needs, their perceptions of distance reduction, amelioration of misunderstandings and perceptions of isolation. the third factor represents perceptions of immediacy in dialogue. immediacy represents temporal perceptions such as the accessibility of tutors and the affordance of the technology to enable reflection on learning or gaining quick responses. following the successful fit of the baseline face-to-face modality model, two additional pathway models were constructed to represent students’ perceptions of communication with their tutors using email and telephone. each was constructed table 2. summary of coefficients factor pathway face to face telephone email exogenous factor correlations autonomy/strategic −0.27 0.15 −0.09 autonomy/surface −0.35* −0.29 – strategic/surface 0.95* 0.69* – regression (β) coefficients to endogenous factors autonomy/structure −0.37 −0.24 0.32* autonomy/presence −0.29 −0.25 0.31* autonomy/immediacy −0.11 −0.14 0.49* strategic/structure 2.61 0.92* 0.98* strategic/presence 3.15 1.36 0.92* strategic/immediacy 2.15 1.36 0.77* surface/structure −2.71 −0.75* – surface/presence 2.98 −1.45 – surface/immediacy −2.29 −1.26 – χ2 171.08** 144.27 169.51** degrees of freedom 120 119 127 χ2/degrees of freedom 0.09 1.21 1.33 rmsea 0.05 0.05 0.06 cfi 0.92 0.94 0.93 significant at *p < 0.05 and **p < 0.01. 110 s. wheeler with identical pathways, and tested individually for goodness of fit. lagrange multiplier (lm) tests were similarly performed on each model to ascertain the best fits for each. discussion in this study, several parameters were taken into account including preferred approaches to study reported by students measured against the amount of structure, social presence and immediacy perceived by learners to have been present within each of the three tutor support modes. the three factors were constructed from items in the communication mode questionnaire. although the model was identified in faceto-face mode with a cfi of 0.92 and in telephone mode with a cfi of 0.94, the cfi for email mode was well below acceptable limits. however, after this model had been adjusted by convergence of two of the factors (surface and strategic approaches to study) into one (strategic), a more acceptable model cfi fit was obtained at 0.93. the need to converge factors in email mode exposed an interesting effect relating to the nature of strategic study approaches. convergence of the two factors was justified through a consideration of strategic learning approaches that embrace both deep and surface approaches. students adopt strategic approaches depending upon the demands of the learning activity, switching between them according to need and expediency (entwistle, 1981). it is plausible to surmise from the need to converge factors that students using email as a means to communicate with their remote tutors may rely more on strategic approaches to study than they do when they use more familiar telephone and face-to-face communication methods to access tutor support. they may need to adopt more flexible approaches to communication with their tutors due to the unfamiliar and less predictable nature of the technologies. the average age of students in the sample was 40.8 years, ensuring that the majority of participants in the study would not have used computer-based applications such as email at school, and would be less habituated in using them than the more familiar telephone. there may also be other variables influencing this result, such as the comparatively more limited access participants have to these new technologies, and the possibility of lack of technical skill and confidence in their effective use. the factor convergence adjustment will therefore be taken into account in this analysis of the three tutor support modalities. it was important that, once a model fit had been established for the baseline face-to-face measurement model, the subsequent comparison models exhibited identical features including pathways, factors, and the same array of modality specific variables. model parameters in this causal model, autonomous learners tend to perceive more rigidity or structure in their courses (β = −0.37, not significant) than their surface learner counterparts (β = −2.71, not significant). furthermore, as autonomous behaviour increases, so structural perceptions of the course will decrease. this result is expected, given that transactional distance in blended learning 111 autonomous learners impose their own structure and therefore demand less structure from their tutors than surface learners. the strongest contributing measure to the surface factor is the need for precise study instructions (β = 0.52, p < 0.05). conversely the strongest item loading onto the autonomy factor is an item relating to students’ need to control the learning process through self-organisation of time (β = 0.76, p < 0.05). this is the basis upon which this model and its companion models are founded. the central parameters are thus used as predictors of student responses in any given factor context. for specific discussion of the models we explore immediacy in dialogue and how it may contribute toward a clearer understanding of the possible causes and effects of transactional distance. immediacy as previously discussed, immediacy in this study refers to the extent to which students feel they can gain quick access to their tutors, and their perception of the timeliness of tutor responses. immediacy is established in face-to-face settings through nodding, smiling and other non-verbal behaviours such as eye contact and gaze. it can also be characterised in paraverbal utterances such as backchannelling and other confirmatory utterances. such behaviours are often subconsciously managed (argyle & dean, 1965), but if they are absent or poorly mediated through the communication medium, students will notice. immediacy also seems to at least partially to encompass the notion of the student’s perception of ‘connectedness’ to the tutor as mediated through the appropriate technology, although this effect is perhaps better represented as a feature of social presence. immediacy has been identified as an important measure of psychological distance, and could therefore be associated with perceived social presence in distance settings, as proposed by argyle and dean (1965). it is likely that a perception of lack of immediacy, resulting in a feeling of reduced social presence, may amplify the detrimental effects of transactional distance. de kerckhove (1997) suggests immediacy is an artefact of the increased capabilities of communication technologies to facilitate interaction. the ability, for example, of tutors to respond quickly or even ‘instantly’ to students’ requests for help is extremely variable. if perceptions of immediacy can be increased or enhanced using technologymediated communication, however, we could surmise that face-to-face environments might take second place to that technology in the preferences of some students. if this were possible to achieve, it would represent the utopian ideal of distance learning actually surpassing the equivalency afforded by face-to-face learning (simonson et al., 2000). less variable would be the tutor’s ability to respond quickly to provide feedback on a student’s work, but sending a quick acknowledgement of receipt would still be possible and indeed even desirable (collis & moonen, 2002) as a means of establishing some form of immediacy in the mind of the student. either way, distance learners generally agree that immediacy of responses from their tutors encourages them to keep on track with their studies, and provides much needed impetus and 112 s. wheeler motivation to persist with study. indeed, timely responses and feedback from tutors may constitute the most significant predictor of distance learner satisfaction (tallman, 1994). immediacy may also directly influence feelings of intimacy in relationship building between student and tutor. gunawardena (1995) has argued that immediacy can convey closeness and ‘warmth’, while lack of immediacy often connotes aloofness or ‘coldness’ (wallace, 1999), indicating that there is a clear relationship between immediacy and intimacy. gunawardena and zittle (1997) also connect intimacy and social presence, arguing they are measures of the amount of ‘psychological distance that a communicator puts between him or herself and the object of his/her communication’ (p. 9). gunawardena and zittle consider that tutors can convey this intimacy as an immediacy of communication through both verbal and non-verbal means, the latter usually through dress codes, facial expressions, and so on. patently, such non-verbal expressions of immediacy are best conveyed through communication in face-to-face mode. immediacy can be a feature of either the communications technology itself or the skills and attributes of the communicator (short et al., 1976; whittaker, 1996). it is of course entirely possible that immediacy may be an artefact of both the individual’s skills and the affordance of the technology. notwithstanding this discussion, immediacy should be considered an important predictor of student satisfaction, and could also therefore be deemed a key predictor of transactional distance. it is apparent from the results of the structural equation modelling of the modality hypothesis that out of the communication modes email is associated with the greatest perception of immediacy of response from the tutor for autonomous students at β = 0.49, p = 0.00001 (table 2). this result runs counter to hypothesis two, and is a somewhat surprising finding, being the only asynchronous communication modality chosen for analysis within this study. however, if de kerckhove and others are correct, and technology-mediated communication has the capability to enhance communication characteristics such as immediacy, this finding makes sense. there is a caveat to this finding, however. this result could be confounded by the inclusion of the ‘time to reflect’ questionnaire item within this factor. time to reflect is important for many students, particularly those who approach study in a less impulsive manner. students who need time to reflect may prefer the asynchronous affordance of communication through email when compared with synchronous modes of communication such as the telephone. the presence of the ‘time to reflect’ item within the immediacy factor may be a misleading element. future questionnaire design would need to take into account the possibility that there may be two separate aspects of immediacy that influence a student’s perception of how quickly the tutor responds. it is probable that immediacy of tutor response and the ability to defer immediate response in favour of greater time for reflection are two separate constructs. these results may also be due to an increased usage of email to communicate in recent times, with the telephone being relegated to second place in the working life of many. longer, more detailed, email messages can be composed and sent, and there is the facility for tutors to program automatic acknowledgements of message receipt transactional distance in blended learning 113 into their systems. email use can also be creative, providing users with a rich repertoire of text-based expressions, emoticons and other devices with which to communicate (wheeler & nistor, 2003). such variables should be acknowledged in any analysis of this type. both the student autonomy (β = 0.49, z = 9.457, p = 0.00001) and strategic approaches to study factors (β = 0.77, z = 4.542, p = 0.00001) for email mode show statistically significant beta coefficients. these results indicate that students who can direct their own study will perceive more immediacy from email communication than their less autonomous counterparts. furthermore, those who are able to approach their learning in a strategic, flexible and versatile manner will perceive greater levels of immediacy in email-mediated communication. similarly with students who need more time to contemplate their learning, the findings suggest that students scoring high in personal autonomy tend to see email as offering a more effective facility with which to reflect upon the learning process. they may also view it as a better utility through which to receive valid and more considered tutor responses in comparison with other modes of communication. the affordance of permanency of email through archiving may advance dialogue beyond real-time events for these students. strategic learners within the sample may also consider that email provides greater accessibility to the remote tutor than other modes of communication. co-present interactions can be rushed or deferred due to pressure of time, environmental distraction or other communication constraints. email communication would not be directly affected by these constraints. further research into these possibilities should be considered as this is perhaps one area in which distance students might regularly benefit over their traditional on-campus counterparts. students receiving timely tutor responses by email could be more predisposed to perceptions of immediacy than those who experienced longer waits. another factor worthy of consideration is the perceived urgency or importance of the message sent by the student to the tutor. if students perceive that messages they have sent to their tutors are urgent, yet they receive no response, or they receive a response they consider to be unduly delayed, more remote transactional distance may be experienced than if they perceived their messages to be less urgent. transactional distance could also be a function of the students’ perceptions of the urgency or importance of their message, regardless of the media or technology used to mediate the communication process. asynchronous communication systems such as email should therefore fare less well than synchronous systems as a means of conveying urgent messages. social presence this study investigated the link between transactional distance and social presence. similarly to transactional distance, social presence is a perceived aspect of technology-mediated communication, and both are hypothesised to have their basis in dialogic processes, so the theoretical basis seems feasible. as social presence is indicative of feelings of connectedness, less autonomous students who are more dependent on their tutors should seek higher levels. however, 114 s. wheeler in this sample, a significant finding is observed for autonomous learners. an examination of the results shown in the summary of coefficients in table 2 reveals that the email communication yields the steepest beta coefficient for autonomous learner perceptions of social presence (β = 0.31, p < 0.05). this result runs counter to hypothesis two, supporting earlier findings of increased perception of immediacy within email mode. autonomous students in the sample may have perceived face-to-face conversation with their tutor to be dialogically the poorest, thereby engendering a more remote transactional distance. due to their high levels of autonomous behaviour, such students may be more comfortable studying remotely from their tutors, preferring to communicate in a manner that enables them to maintain a level of control. email certainly provides such learner centric affordances. conversely, surface studiers reported much lower levels of social presence in telephone mode (β = −1.45, not significant) and much higher levels of social presence face to face (β = 2.98, not significant). this result is more in line with expectations, given that surface learners tend to rely more on direct instruction and tutor feedback than their more autonomous peers. communication over telephone would therefore be less preferable for surface learners than a face-to-face tutorial. however, these results should be treated with some caution, as the beta coefficients are statistically non-significant. this result appears to support the earlier finding that immediacy was perceived by participants to be higher in email mode. although the two are associated, perhaps a distinction between immediacy and perceived social presence should be clarified. immediacy has already been defined as relating to the learner perception of connectedness and accessibility to the tutor. it also has elements of reflection on learning and is therefore essentially temporal. conversely, social presence focuses more on the affordances of the communicating medium, and perceptions that the student is communicating with a real person rather than with a mute technology. social presence defines the qualitative difference between students being able to communicate their needs from a distance, thereby reducing social isolation, as opposed to the ease of accessing tutor support and the gaining of quick response times. characterising two distinct aspects of dialogue in the form of immediacy and social presence may be helpful to determine its function in transactional distance better. structure structure has been defined as the extent of the constraints imposed upon students through the way a course is delivered, managed and evaluated. in this context, study can be characterised by its level of flexibility or rigidity. lower levels of structure were reported by strategic learners in telephone mode than in other modes (β = 0.92, z = 2.671, p = 0.003), with the highest levels of structure being reported in face-to-face contexts (β = 2.61, z = 1.503, p = 0.06, not significant). students therefore reported more flexibility in telephone mode than in face-to-face contact with their tutors. while the face-to-face result should be treated with caution, the finding in telephone mode is statistically significant and suggests that this mode of communication may transactional distance in blended learning 115 impose a lesser level of structure upon strategic distance learners than any of the other modes. this is possibly due to an increase in the dialogue that is inherent in telephone communication. if moore’s transactional distance theory is correct, the telephone should produce the least transactional distance for strategic distance learners due to its greater potential to reduce structure. telephone-mediated dialogue may subvert structure due to its often informal and open-ended nature. it should also be noted that the telephone is probably the most easily accessible and familiar of the technologies for the majority of the participants. analysis of email communication also yielded a statistically significant positive regression (β = 0.98, p < 0.05) from strategic approaches to structure, which suggests that this technology would actually impose greater levels of structure for strategic distance learners. if moore’s theory is correct, email would actually create a more remote transactional distance for strategic learners than other modes of communication, with the telephone emerging as the technology that is least likely to impose structure. in the original modality hypothesis, co-present tutorials were predicted to evoke the least amount of reported transactional distance, on the basis that the level of dialogue would be richer. if dialogue is richer in face-to-face communication, then close transactional distance would be experienced; while if high levels of structure were imposed, remote transactional distance would result. it is clear from an examination of the covariance matrix that face-to-face mode is actually one of the most effective modes in imposing structure upon strategic distance learners (β = 2.15, z = 1.484, p = 0.06, not significant). some autonomous learners in the sample may have been able to subvert the structural effects of the face-to-face encounter more effectively than in most of the other communication modes (β = –0.37, z = –0.602, p = 0.27, not significant). autonomous learners may have achieved this by proactively instigating more dialogue with their remote tutors, which would decrease structure. however, this is conjectural as, again, these results were far from conclusive and yielded no statistical significance. what also remains unresolved is whether the imposed structure on study actually increases transactional distance. many commentators suggest that this is the case, but doubt remains (vrasidas & mcisaac, 1999), and there is also a dearth of concerted research into this effect within multiple-mode learning environments. it may be beneficial to approach this problem by viewing structure, perception of social presence and immediacy as three separate constructs, each of which may contribute toward transactional distance. moore’s original configuration of structure and dialogue remains intact, but with dialogue consisting of at least two subfactors, social presence and immediacy. these are therefore proposed as revised predictors of transactional distance. conclusions there are several limitations to this study. firstly, the opportunistic sample comprised several cohorts of students studying in a variety of blended modes, at various levels 116 s. wheeler from foundation degree to postgraduate study. the statistical method was not designed to investigate the nuances and subtleties of individual student experiences across such a range of levels. secondly, the mix of communication modes provides little knowledge about student perceptions for those who engaged in both modes of distance communication. it would be unwise to attempt to generalise these findings across all blended learning experiences, but the results nonetheless raise some interesting issues. statistically significant findings related mainly to autonomous and strategic learners. email seems to provide the highest level of immediacy of dialogue for learners who are proactive and who control their own learning activities (p = 0.00001). email also provides the greatest amount of social presence for autonomous students (p < 0.05). however, email is also likely to impose the highest amount of structure on strategic learners, and if this is at the expense of dialogue then it is likely to create a more remote transactional distance for this type of learner (p < 0.05). finally, the telephone tends to impose the least amount of structure on strategic learners (p < 0.05), which may lessen transactional distance. it can be concluded from these findings that students experience at least some elements of transactional distance when separated from their tutors. structure can be imposed by communication technologies to varying degrees, with the telephone appearing to produce the lowest level. dialogue can be mediated through a variety of technologies but for this sample email appears to facilitate the highest levels of immediacy, and so should be considered an important communication technology for tutors to develop within any blended learning strategy. the future success of blended learning will rely heavily on technology-mediated communication, but even more on the skills and knowledge of responsive tutors. references argyle, m. & dean, j. (1965) eye contact, distance and affiliation, sociometry, 28, 289–304. byrne, b. m. (1994) structural equation modeling with eqs and eqs for windows: basic concepts, applications and programming (london, sage). collis, b. & moonen, j. (2002) flexible learning in a digital world: experience and expectation (london, kogan page). dabbagh, n. (2004) distance learning: emerging pedagogical issues and learning designs, quarterly review of distance education, 5(1), 37–49. de kerckhove, d. (1997) the skin of culture: investigating the new electronic reality (london, kogan page). dron, j., seidel, c. & litten, g. (2004) transactional distance in a blended learning environment, alt-j: research in learning technology, 12(2), 163–174. entwistle, n. j. (1981) styles of learning and teaching (chichester, john wiley & sons). goodyear, p., asensio, m., jones, c., hodgson, v. & steeples, c. (2003) relationships between conceptions of learning, approaches to study and students’ judgements about the value of their experiences of networked learning, alt-j: research in learning technology, 11(1), 17–27. gunawardena, c. n. (1995) social presence theory and implications for interaction and collaborative learning in computer conferencing, international journal of educational telecommunications, 1(2/3), 147–166. transactional distance in blended learning 117 gunawardena, c. n. & zittle, f. j. (1997) social presence as a predictor of satisfaction within a computer-mediated conferencing environment, the american journal of distance education, 11(3), 8–26. miyake, a., friedman, n. p., emerson, m. j., witzki, a. h., howerter, a. & wager, t. d. (2000) the unity and diversity of executive functions and their contributions to complex ‘frontal lobe’ tasks: a latent variable analysis, cognitive psychology, 41, 49–100. moore, m. g. (1973) towards a theory of independent learning and teaching, journal of higher education, 44(12), 661–679. moore, m. g. (1993) theory of transactional distance, in: d. keegan (ed.) theoretical principles of distance education (new york, routledge), 22–38. moore, m. g. & kearsley, g. (1996) distance education: a systems view (belmont, ca, wadsworth). ross, s. m., morrison, g. r., smith, l. j. & cleveland, e. (1991) an evaluation of alternative distance tutoring models for at-risk elementary school children (eric document reproduction service no. ed 335 009). shin, n. (2003) transactional presence as a critical predictor of success in distance learning, distance education, 24(1), 69–86. short, j., williams, e. & christie, b. (1976) the social psychology of telecommunications (london, john wiley). simonson, m., smaldino, s., albright, m. & zvacek, s. (2000) teaching and learning at a distance: foundations of distance education (upper saddle river, nj, merrill). simpson, o. (2004) the impact on retention of interventions to support distance learning students, open learning, 19(1), 79–95. tallman, f. d. (1994) satisfaction and completion of correspondence study: the influence of instructional and student-support services, the american journal of distance education, 8(2), 43–57. vrasidas, c. & mcisaac, m. s. (1999) factors influencing interaction in an online course, the american journal of distance education, 13(3), 22–26. wallace, p. m. (1999) the psychology of the internet (cambridge, cambridge university press). weinstein, c. e. & van mater stone, d. (1996) learning strategies and learning to learn, in: e. de corte & f. e. weinert (eds) international encyclopedia of developmental and instructional psychology (oxford, pergamon), 419–423. wheeler, s. (2002) student perceptions of support in distance education, quarterly review of distance education, 3(4), 419–429. wheeler, s. & nistor, n. (2003) human behaviour in the online sub culture, in: n. nistor, s. english, s. wheeler & m. jalobeanu (eds) toward the virtual university: international online perspectives (greenwich, information age). wheeler, s. & vranch, a. t. (2001) building for the future of educational telematics: models, foundations and frameworks, international journal of engineering education, 17(2), 145–152. whittaker, s. (1996) talking to strangers: an evaluation of the factors affecting electronic collaboration, in: proceedings of conference on computer supported collaborative work (new york, acm press), 409–418. alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 175–180 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020175–06 © 2007 association for learning technology doi: 10.1080/09687760701482523 discussion section tipping into the abyss: with more than a virtual parachute? chris tompsett* kingston university, uk taylor and franciscalt_a_248129.sgm10.1080/09687760701482523alt-j research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis152000000june 2007christompsettc.p.tompsett@kingston.ac.uk; cptompsett@king.ac.uk any application of information and communication technology in education (icte) sits, at times uncomfortably, at the intersection of three key disciplines: technology, education and sociology (including reflexivity). to confuse matters, any specific study may need to take account of specific knowledge within subdisciplines, such as organisational management and technology transfer, and of knowledge within the domain of application (e.g. nursing, social work, fashion, etc.). researchers must build a consistent model of knowledge that can integrate disparate methodologies, research goals and even conflicting interpretations of the same terminology. without this, the icte research field will be dominated by what is simply novel, irrespective of the relevance of particular changes to educational practice. if existing models in this field are as limited as suggested by moule, when should lecturers and teachers, with no motivation to use technology for its own sake and no additional financial support, review progress in this field for effective examples of innovative practice, let alone wide-scale change? on most of the criteria that could be introduced to compare two papers, the views of moule and salmon appear almost diametrically opposed and a detailed comparison would seem of limited value. instead, this paper asks a more fundamental question: what could be the basis within this research community for establishing coherence within the field and ensuring that research can justify actual changes in educational practice? introduction providing a critical analysis of research literature that lies between the views of moule (2006, 2007a, b) and salmon (2000, 2007) would demand far more space than this brief commentary will allow, but, as the intention is to foster debate and discussion, it makes more sense for detailed consideration of individual aspects to be considered by later contributors. the present article, instead, is intended to suggest that a number of critical factors must be considered in any subsequent discussion to ensure that the debate fosters developments in this field towards the establishment of reliable outcomes for educational practitioners to adopt (cf. hargreaves, 1996), rather than experimental or innovative, practice (rogers, 1962). questions will be examined * kingston university, learning technology research group, penrhyn road, kingston upon thames, london kt1 2ee, uk. email: cptompsett@king.ac.uk 176 c. tompsett necessarily in turn, within the dialogical format of this paper, but that oversimplifies the underlying theme. questions regarding the construction of knowledge itself are inherently circular: the processes of research and education are implicitly entwined with the nature of knowledge and how it can be learned. some further issues can only be raised; in relation to these, pointers are given towards more detailed discussions that exist elsewhere. the intention is therefore to create a ‘playground’ within which colleagues can discuss these two papers, always allowing, if need be, for others to remodel the playground itself. remember education? the rate of technical development in information and communication technology produces an ephemeral research agenda in icte. complex and critical research questions are raised with one particular technology but disappear as researchers gain funding to test new ideas without resolving the issues that limit the potential value of previous work to educational practice. the current interest in m-learning, for example, either acts as a replacement for wired technology (a technical rather than educational problem) or else reduces the emphasis on visual attention, when full-size screens and existing interface devices remain hopelessly inadequate in many subjects. while such work is necessary in the first stages of development, the constant changes in the research agenda leave much started and little complete. even where research in icte works specifically within an educational environment, and suggests that educational improvement has been achieved, it is far too common for ‘improvements’ to be attributed to the technological change rather than the many other educational changes that may also have taken place. these improvements are, of course, invalidated, if the measures of improvement themselves are different from those already used (macfarlane, 2004). if there was little disparity between this community’s conception of icte, and that of those in education, there would be little need for macfarlane’s suggestion that all research teams should include both educators and technologists. this disparity is emphasised when distinctions that would be understood in education become lost within this research community. when juxtaposed with instructivism, constructivism appears as a coherent position but, when applied within education, it hides a multiplicity of meanings. laurillard (2004), for example, suggests 13 different models of constructivist education. some interpretations may provide subtle distinctions, or reflect a change in context, but other differences entail a radically different model of knowledge, learning and education and the interrelationship between these concepts. laurillard’s selection would not cover the full set covered in icte, let alone education. activity theory (engeström, 1991; engeström et al., 1999) would need to be included, as it is a key socio-constructivist position that remains distinct from but complementary to the views of vygotsky, lave and wenger (see lave & wenger, 1991, p. 31). similarly, chomsky would need to be added to complement the views of piaget and papert (see piattelli-palmarini, 1980). researchers in icte might also benefit from a number of systematic reviews and meta-studies that have been published more extensively within the education field. tipping into the abyss 177 the benefits of peer-group discussion and networked learning are clearly difficult to establish (for example, andrews et al., 2002; goldberg et al., 2003; hogarth et al., 2005). a careful analysis of these reviews/studies provides interesting examples of what not to repeat, what is required to establish educational benefits and to provide benchmarks for the standard of research that can be achieved. why sociology? for those who are striving to integrate the most recent innovations in icte, such as podcasts and the semantic web, it may also seem unusual to insist that due attention is given to sociological issues in research. however, the reasons to do so increase from year to year. for the educationalist, ‘blended learning’ and ‘work-based learning’ blur the self-evident separation (for those in educational practice) that students are either studying, working or ‘not studying’ (i.e. living). for the technologists, the increasing focus on delivering solutions through technology to the new generation of technologically aware ‘screenagers’ (watson, 2006, p. 10) depends on understanding the sociological function of such technology. the interest that each student may show in self-selected web-logs or the individual collection of performances that are stored on an mp3 player may well be remarkable, but there is no socio-cultural justification that the same interest will transfer when the freedom to choose content is removed and/or dictated by some other agent. even if the same enthusiasm were to be transferable, researchers seem unwilling to discuss the inherent limitations that restrict the quality and range of content when compared with alternative modes of interaction and delivery (with or without technology). at the very least, a socio-technical approach (checkland & scholes, 1991) is required to understand the way in which each group of users (educators, support staff and students) are likely to react unless ‘required’ to do so. in the latter case, an even wider research framework is needed to understand how such ‘requirements’ may be legitimated or flouted (see, for example, latour, 1992). selecting an appropriate critical framework for sociological modelling may seem arbitrary, when first encountered, since some of the most critical terms, such as ‘community’, are defined in contradictory ways. in exchange theory (and also action research), the characteristics of a community must be determined by the choices made by each of the individuals. in a functionalist model, a community may acquire independent characteristics above and beyond those that follow from its members. if we include socio-constructivist models, whether vygotskian, activity theoretical or communities of practice (respectively wertsch, 1985; engeström, 1991; wenger, 1998), then the primary focus on the individual over the community is reversed. wenger (1998), for example, only addresses individuality after five chapters on practice. each of these models determines the objects that can be observed within the world, their potential properties and the possible relationships between them. these then determine how knowledge is established and shared, the relevance of existing research to the current problem and, from this, the extent to which any results from this project may be relevant to future projects. if ‘communication’ is only researched in terms of 178 c. tompsett communication within a discussion group but cannot capture informal discussions and conversations, then the researchers will be blind to interactions that could be far more significant in educational terms (for example, steyn & cronje, 2006). if the behaviour of scientists ‘in the laboratory’ is observed to the same level of detailed analysis as other cultural groups, it becomes clear that social controls and unscientific arguments are as common and entrenched as in any other hierarchical professional group. the posthoc, rationalised descriptions that are published in scientific journals simply report what a scientist is allowed to report (latour & woolgar, 1986). as some critical terminology (e.g. community, constructivist) has markedly different connotations depending on the approach chosen, it is incumbent on those reporting the research to ensure that these differences are presented coherently within the published literature. failing to do so can, in extreme cases, lead to terms becoming meaningless; for example, with the concept of the delphi methodology in decision support (dalkey & helmer, 1963; linstone & turoff, 1975; hasson et al., 2000; mitroff & turoff, 2002). avoiding this problem requires that there is consistency between the problem, model and methodology within the research—and clarity of reporting for the relevant readership. a pick-and-mix approach, or lack of awareness of the issues that need to be managed within almost all of the approaches, opens the suggestion that such research is unaware of the critical issues that affect learning within a technological society. reflexivity although there is insufficient room to review all of the wide range of issues that are considered within a properly conducted sociological study, one particular issue cannot be ignored—since it is introduced as soon as a study expects to account for any factor that influences human behaviour beyond the purely clinical. if a researcher believes that some set of properties, personal or social, affects how subjects report, learn and reflect new learning in their behaviours or interactions with others, then the same range of factors could influence the researcher(s) themselves—the principle of reflexivity (ashmore, 1989). at the simplest level this requires that the researcher is aware of, and accounts for, their own influence on the research processes (reflective self-awareness—the researcher as a non-independent observer; ashmore, 1989, p. 32). at a more fundamental level (and frequently missed in most such studies), it is generally unsound to conduct research in which the relationship between actions, learning and knowledge that is expected to apply to any subjects is distinct from that applied to researchers themselves (reflective self-reference; p. 32). merrill et al. (1996) proposed a laboratory study comparing individual with group-based learning, but failed to note that some aspects of group inter-activity might be lost when informal interactions were transferred to artificial environments and scheduled times so that they could be ‘observed’. alsop and tompsett (2006) provide a more detailed analysis of this issue in the case of phenomenography. in icte, as well as in education, additional levels of inconsistency may be introduced between knowledge in the domain, models of assessment and models of learning tipping into the abyss 179 (wenger, 1998, epilogue)—and even between course design and models of student learning. dirckinck-holmfeld et al. (2004) provide one of the few cases within the community of practice model in which consistency has been achieved. in contrast, zuber-skerrit (1992) provides some cases within the action research model where course designers clearly apply one model to themselves and a different one to their students. some contradictions may be unavoidable, but that does not remove the need to recognise that they exist and to address them within the research process. conclusions the differences in perspective, approach and scope of the work of gilly salmon and pam moule cover a considerable range of viewpoints regarding the nature of research in this field, and there is a risk that a continuing discussion might result in revealing differences within the community that are interesting from the ‘intellectualist’ position (oancea, 2005) but that are of little value to those engaged in educational practice. for those in educational practice, even if research progresses beyond the simply technological, innovative work is seldom pursued to the point at which effectiveness is sufficiently proven to justify a change in practice (alsop & tompsett, 2007). these two opposing views provide an opportunity to discuss and reassess the future of research in this field. if there are to be rules for the debate, then this author would argue strongly that the aims and principles of education and sociology need to become embedded as centrally as technology is at present. without that, research in this community will become increasingly self-focused, leading to improvements in a virtual reality that are incapable of influencing educational practice in a real world. references alsop, g. & tompsett, c. (2006) making sense of ‘pure’ phenomenography in information and communication technology in education, alt-j, 14(3), 241–259. alsop, g. & tompsett, c. (2007) from effect to effectiveness: the missing research questions, education, technology and society, 10(1), 28–39. andrews, r., burn, a., leach, j., locke, t., low, g. & torgerson, c. (2002) a systematic review of the impact of networked ict on 5–16 year olds’ literacy in english (london, eppi-centre, social science research unit, institute of education). ashmore, m. (1989) the reflexivity thesis: wrighting sociology of scientific knowledge (chicago, il, the chicago university press). checkland, p. & scholes, j. (1991) soft systems methodology in action (chichester, john wiley and sons). dalkey, n. & helmer, o. (1963) an experimental application of the delphi method to the use of experts, management science, 9(3), 458–467. dirckinck-holmfeld, l., sorensen, e. k., ryberg, t. & buus, l. (2004) a theoretical framework for designing online master communities of practice. networked learning 2004 (lancaster, lancaster university). engeström, y. (1991) activity theory and individual and social transformation, multi-disciplinary newsletter for activity theory, 7/8, 6–17. engeström, y., miettinen, r. & punamäki, r.-l. (1999) perspectives on activity theory (cambridge, cambridge university press). 180 c. tompsett goldberg, a., russel, m. & cook, a. (2003) the effect of computers on student writing: a metaanalysis of studies from 1992 to 2002, the journal of technology, learning and assessment, 1. available online at: http://http://www.jtla.org (accessed 12 april 2007). hargreaves, d. h. (1996) teaching as a research-based profession: possibilities and prospects (london, teacher training agency). hasson, f., keeney, s. & mckenna, h. (2000) research guidelines for the delphi survey technique, journal of advanced nursing, 32(4), 1008–1015. hogarth, s., bennett, j., campbell, b., lubben, f. & robinson, a. (2005) a systematic review of the use of small-group discussions in science teaching with students aged 11–18, and the effect of different stimuli (print materials, practical work, ict, video/film) on students’ understanding of evidence. research evidence in education library (london, eppi-centre, social science research unit, institute of education). latour, b. (1992) where are the missing masses? the sociology of a few mundane artifacts, in: w. bijker & j. law (eds) shaping technology/building society: studies in sociotechnical change (cambridge, ma, mit press), 225–258. latour, b. & woolgar, s. (1986) laboratory life: the construction of scientific facts (princeton, nj, princeton paperbacks). laurillard, d. (2004) realising the vision for e-learning (london, bett). lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation (cambridge, cambridge university press). linstone, h. a. & turoff, m. (eds) (1975) the delphi method: techniques and applications (reading, ma, addison wesley inc.). macfarlane, a. (2004) e-pedagogy—conference review, paper presented at alt-c 2004, exeter, uk. merrill, d. m., drake, l., lacy, m. j. & pratt, j. (1996) reclaiming instructional design, educational technology, 36(5), 5–7. mitroff, i. i. & turoff, m. (2002) iib. philosophical and methodological foundations of delphi, in: h. a. linstone & m. turoff (eds) the delphi method: techniques and applications (new jersey institute of technology). available online at: http://is.njit.edu/pubs/delphibook/ (accessed 29 march 2007). moule, p. (2006) e-learning for healthcare students: developing the communities of practice framework, journal of advanced nursing, 54(3), 370–380. moule, p. (2007a) challenging the five-stage model for e-learning: a new approach, alt-j, 15(1), 37–50. moule, p. (2007b) the tipping point—a response, alt-j, 15(2), 173–174. oancea, a. (2005) criticisms of educational research: key topics and levels of analysis, british educational research journal, 31(2), 157–183. piattelli-palmarini, m. (ed.) (1980) language and learning: the debate between jean piaget and noam chomsky (cambridge, ma, harvard university press). rogers, e. m. (1962) diffusion of innovations (new york, the free press). salmon, g. (2000) e-moderating: the key to teaching and learning online (london, kogan page). salmon, g. (2007) the tipping point, alt-j, 15(2), 171–172. steyn, d. & cronje, j. (2006) interpreting cross-cultural blended teaching and learning along hofstede’s cultural dimensions, paper presented at e/merge 2006: learning landscapes in southern africa online conference. available online at: http://emerge2006.net (accessed 1 march 2007). watson, d. (2006) whatever happened to the student experience?, academy exchange, 4(summer), 9–11. wenger, e. (1998) communities of practice: learning, meaning and identity (cambridge, cambridge university press). wertsch, j. v. (1985) vygotsky and the social formation of the mind (london, harvard university press). zuber-skerrit, o. (1992) professional development in higher education—a theoretical framework for action research (london, kogan page). calt586678 101..116 learners’ perceived information overload in online learning via computer-mediated communication chun-ying chena*, susan pedersenb and karen l. murphyb acenter for general education, national taichung university of education, taiwan; bdepartment of educational psychology, texas a&m university, college station, texas, usa (received 1 december 2010; final version received 8 april 2011) many studies report information overload as one of the main problems that students encounter in online learning via computer-mediated communication. this study aimed to explore the sources of online students’ information overload and offer suggestions for increasing students’ cognitive resources for learning. participants were 12 graduate students from two online courses in the united states. their learning experiences in both online discussions and on the course website were explored through semi-structured interviews. they also completed a background questionnaire that assessed three constructs that limit learner readiness and are likely to lead to online students’ perceived information overload: inadequate prior knowledge, inadequate english proficiency, and lack of technical skills for participating in computer-mediated communications. the findings suggest that varied learner characteristics led some students to be more susceptible than others to information overload. emerging data-driven risk factors were: lack of efficiency in reading from computer screens, visual and auditory learning preferences, and time constraints. difficulties associated with students’ perceptions of information overload are addressed and implications for course design are offered. keywords: information overload; computer-mediated communication; cognitive load theory; online discussion introduction with increases in internet communication technologies, online learning has grown rapidly through the use of computer-mediated communication (cmc). cmc uses telecommunication technologies such as email, real-time chat, computer conferencing/online discussion systems, and online databases to support human communication between spatially separated learners (jonassen et al. 1995). although cmc can support teaching and learning by making information and communication easily accessible via computer networks, one of the main problems caused by the medium is information overload (io) (burge 1994; eastmond 1995; harasim 1987; kear and heap, 2007; paulo 1999; vonderwell and zachariah 2005). io becomes a problem when students simultaneously face having to acquire the technical skills necessary to participate in cmc and manage a large volume of information both on *corresponding author. email: cychen666@gmail.com research in learning technologyaquatic insects vol. 19, no. 2, july 2011, 101–116 issn 2156-7069 print/issn 2156-7077 online � 2011 association for learning technology doi: 10.1080/21567069.2011.586678 http://www.informaworld.com the course website and through computer conferencing (harasim 1987). numerous studies have investigated the problem of io in the disciplines of organisation science, marketing, accounting, management information systems, business management, and psychology (for reviews, see edmunds and morris 2000; eppler and mengis, 2004; klausegger, sinkovics, and zou 2007). yet little research has been dedicated to investigating the problem of io per se in online education as other disciplines have done. this study thus aimed to explore the sources of online students’ io and accordingly offer suggestions for increasing students’ cognitive resources for learning. theoretical framework cognitive load theory cognitive load theory may provide a basis for understanding the concept of io in this study. the theory consists of the cognitive architecture including memory and schema together with the information structures and their instructional implications that emphasise working memory constraints as determinants of instructional design effectiveness (sweller, van merriënboer, and paas 1998). knowledge is stored in long-term memory in the form of schema that acts as a single element. the sub-elements or lower-level schemas that are incorporated in the higher-level schemas no longer require working memory capacity. understanding occurs when high-elementinteractivity material can be held simultaneously in working memory, and schema construction is the mechanism of this understanding process (sweller, van merriënboer, and paas 1998). cognitive load can arise from three sources: intrinsic, extraneous, and germane cognitive load. intrinsic cognitive load is intrinsic to the learning material, which has high element interactivity. extraneous cognitive load is unnecessary load caused by inadequately designed instruction. germane cognitive load comes from the effort that contributes to schema construction. while effective instructional design decreases extraneous cognitive load and increases germane cognitive load, increasing germane cognitive load will only work if the total load stays within working memory limits. effective instructional techniques include the goal-free effect; the worked example effect; the completion problem effect; the split-attention effect; the modality effects; the redundancy effect; and the variability effect (for details, see sweller, van merriënboer, and paas 1998). the first three instructional techniques are involved in problem-solving, in areas such as mathematics and science. the remaining four techniques are used in the presentation design of the learning material. valcke (2002) updated the current cognitive load theory by positioning prior knowledge and metacognitive load in the conceptual framework (figure 1). valcke stressed the contribution of prior knowledge in schema construction. moreover, valcke argued that learners invest effort in schema construction as well as in the monitoring of this activity, so the overall germane cognitive load is to be linked to the latter activity called metacognitive load. information overload versus cognitive overload memory and schema are key concepts of cognitive load theory that relate to the information-processing approach to learning (valcke 2002). the human 102 c.-y. chen et al. information-processing model thus provides a basis for understanding how io and cognitive overload could possibly interfere with the cognitive processes (i.e., attention, storage, and retrieval) and metacognitive processes (that are responsible for guiding and monitoring the three cognitive processes) required for learning; that is, knowledge construction. accordingly, the distinction between the two terms is illuminated. as shown in figure 2, io and cognitive overload may occur in any of the cognitive processes as well as metacognitive processes to interfere with student learning. io usually occurs in the attention process when an individual experiences any interference from the environment resulting in information loss, due to the limited figure 1. valcke’s (2002) updated model for cognitive load theory. figure 2. information overload vs. cognitive overload. research in learning technology 103 capacity of both the sensory memory and the working memory. alternatively, cognitive overload happens in the storage and retrieval processes, resulting in the inability to connect new information to prior knowledge. when cognitive and metacognitive processes cannot function well for certain purposes, humans fail to construct knowledge. thus io seems to be the precursor of cognitive overload. there is an overlap between the two in that both emphasise limits in the capacity of working memory. the overlap may help clarify the common explanation that both terms refer to the same theory emphasising constraints on human working memory. moreover, io includes sensory memory that was omitted by cognitive load theory (sweller, van merriënboer, and paas 1998; valcke 2002). in brief, cognitive overload is the load imposed on students during content learning, whereas io is the ‘noise’ preventing students from learning content. io caused by cmc in online learning is therefore in this study defined as the point at which a learner’s capacity of sensory memory and working memory are exceeded, and the excessive information and stimuli from the cmc learning environment interfere with content learning. thus, the concept of io could be linked to the concept of extraneous cognitive load. potential contributors to online students’ perceived information overload literature was reviewed to identify elements likely to contribute to online students’ perceived io, and revealed four dimensions of potential contributors: limited learner readiness, quantity of information, quality of information, and medium interface. first, three elements that limit learner readiness and are likely to lead to students’ perceived io are a lack of the technical skills for participating in cmc (paulo 1999), inadequate prior knowledge (valcke 2002), and inadequate english reading/ writing proficiency (eastmond 1995). in cmc, students have to possess computer skills and computer conferencing skills in order to perform cmc tasks such as participating in online discussions and interacting online with others. cognitive load theory emphasises the contribution of prior knowledge in schema construction that functions to reduce working memory load. moreover, the ability to read and write well is necessary in the text-based cmc environment. students with inadequate language proficiency may suffer reduced processing ability if the capacity in their working memory is exceeded by the tasks of reading and writing in english. second, three sources regarding quantitative components of information probably lead to students’ perceived io: large volumes of information from the internet, and from the online discussion systems along with demands of course readings (harasim 1987; vonderwell and zachariah 2005). in computer conferencing, active discussions generate heavy amounts of messages, and this problem increases with the group size. third, certain qualitative components of information help explain obstacles to processing information: difficult learning materials (sweller 1994), text ambiguities (trevino, lengel, and daft 1987), and redundant information (mason and kaye 1990; schwan, straub, and hesse 2002). difficult learning materials reflecting the inherent complexity of course content impose an intrinsic cognitive load (high element interactivity) on students. sweller suggested that extraneous cognitive load that interferes with learning is a problem only under conditions of high cognitive load caused by high element interactivity. text ambiguities cause students to misinterpret information. text communication is a lean medium; it lacks the social cues (i.e. facial expressions, body language, and intonation) that aid participants’ 104 c.-y. chen et al. clarification and understanding of discussions. students may experience text ambiguities while reading the learning material or conferencing messages. redundant information refers to repetitive, irrelevant, or trivial information. repetition of ideas may occur naturally due to the asynchronous attribute of the medium. for instance, while one is composing a message in computer conferencing, other students may be writing the same thing simultaneously (burge 1994). last, interfaces of computer conferencing and online databases both can lead to learner disorientation and cognitive overload (conklin 1987; destefano and lefevre 2007; hiltz and turoff 1985). in computer conferencing, multiple simultaneous threaded discussions (a non-linear structure of discussions in which each topic is the starting point for a branch of responses and may be sorted by topic, date or author) can contribute to message fragmentation and be confusing to navigate and respond to. further, multiple conference spaces can lead to disorientation. as for the hypertext interface of online databases, large numbers of links may overload and distract users with navigation and informational choices. this study aimed to explore the sources of online students’ io and accordingly offer considerations for increasing students’ cognitive resources to focus on content learning. notice that we dealt with perceived io rather than real io because it is difficult to measure the real amount of information that an individual attends to and processes. two questions helped to guide this research: 1. what learner characteristics cause students to be more likely at risk of io? 2. what difficulties contributing to their perceptions of io do students experience when they learn through the medium of cmc? method online courses and participants two online courses at a large university in the united states were selected purposefully because both: were a semester-long graduate-level course at a college of education; integrated computer conferencing as a component of course activities as a replacement for traditional classroom discussions; and required students to participate in online discussions. the instructional context of each class included a website and a computer conferencing system. the conferencing system offered the non-linear structure of threaded discussions. the learning activities required in the two courses were similar: both included small-group discussions, individual projects and critiques, and final papers. participants were 12 graduate students from two online courses in the united states. of the 12 participants (five females and seven males), one-half were master’s students and one-half were doctoral students. five participants were full-time students, whereas seven were employed full-time. students with different ranges of experience with technology use and online courses were selected purposefully. data collection a mixed-method approach, with an emphasis on qualitative aspects, was chosen for the study to provide comprehensive descriptions of the phenomenon of io in educational cmc, and to enable cross-method triangulation. two data sources were research in learning technology 105 used in the study: semi-structured interviews and a background questionnaire. interviews were used primarily to seek answers to both research questions; the quantitative data obtained from the questionnaire were used to support and verify the qualitative findings obtained in the interviews regarding research question one. the literature review, as summarised in the earlier section ‘potential contributors to online students’ perceived information overload’, helped to inform the development of both the questionnaire and the interview protocol. the questionnaire was administered during the orientation meetings. interviews were conducted during the fourth and fifth weeks of the semester because the literature indicated that io is more pronounced early in a course, particularly during the first five weeks (harasim 1987). semi-structured interviews interview questions (table 1) were designed to examine the answers to research questions by exploring interviewees’ learning experiences both in online discussions and on the course website. additional probing questions were prepared to identify how the quantity of information, quality of information, and interfaces of cmc contributed to interviewees’ perceptions of io. the interviewees were guided to talk about their learning experiences in the online course they were taking. all interview table 1. interview protocol. question 1. tell me about your learning experiences in online discussions. 2. how do those difficulties influence your learning? probe: quantity of information (a) how do you feel about the numbers of messages posted in online discussions? quality of information (a) do you ever encounter messages that you feel hard to understand? (b) do you ever encounter messages that seem ambiguous and that influence your learning? for example, in this sentence “mary told sue that she had won the beauty contest”, it is not clear who won the beauty contest – mary or sue. (c) do you ever encounter messages that seem trivial and that influence your learning? (d) do you ever encounter messages that seem repetitive and that influence your learning? for example, someone may make the same or similar comments as others. medium interface (a) how do you feel about the structure of how messages are arranged in online discussions? (b) how do you feel about the multiple conference spaces in online discussions? (c) do you ever feel information is fragmentary and that it influences your learning? 3. tell me about your learning experiences from the course website. 4. how do those difficulties influence your learning? probe: quantity of information (a) how do you feel about the quantity of information on the course website? quality of information (a) how do you feel about the structure of how information is arranged on the course website? (b) do you ever encounter information that you feel hard to understand? medium interface (a) how do you feel about the number of links on the course website? (b) do you ever feel information is fragmentary and that it influences your learning? 106 c.-y. chen et al. transcript analysis was consistent with the constant comparative method (lincoln and guba 1985). the constant comparative method is an inductive data analysis, which uses the specific raw data of transcripts to generate abstract categories. the analysis took the form of successive iterations involving the procedures based on lincoln and guba’s techniques of unitisation and categorisation. the iterations were repeated until no new patterns emerged. data collection and analysis continued until the categories were saturated (i.e. definitions of categories were well defined). to ensure credibility of the findings – that is, the trustworthiness of the data (lincoln and guba 1985) – extensive member checking with the interviewees was conducted throughout the study. background questionnaire three constructs that limit learner readiness and probably lead to online students’ perceived io were identified from the literature: inadequate prior knowledge, inadequate english proficiency, and lack of technical skills for participating in cmc. accordingly, the background questionnaire used to identify students who were at risk of io consisted of four parts: (i) prior knowledge, (ii) english proficiency, (iii) online course experience, and (iv) technology use experience. part i consisted of a series of questions (regarding students’ undergraduate and/ or graduate major and the number of relevant courses taken) designed to determine participants’ prior knowledge. part ii was a self-report survey that asked students to rank their english reading and writing proficiency as poor, fair, good, or fluent. part iii, an online course experience survey, inquired how many online courses participants had taken previously. part iv was a technology use survey. this self-report survey asked participants to identify their level of skill on a scale ranging from one (low) to four (high) for a range of tasks including basic computer operation, file management, file transfer, email use, web browser operation, internet use, computer conferencing, and information searching. this survey was adapted from harvell’s (2000) ‘background and experience of developers’ questionnaire, which was adapted from the bellingham public schools’ (1999) staff use of technology: 1999–2000 self-evaluation rubric. table 2 provides criteria to identify interviewees’ level of background knowledge/skills. the information provided in parts i and ii was used to identify each interviewee’s level of prior knowledge and english proficiency. the results of parts iii and iv were combined to identify the level of technical skill required to participate in cmc. accordingly, interviewees were identified as at risk of io if they possessed a low level of any one of these variables: prior knowledge, english proficiency, or technical skill required to participate in cmc. results characteristics of learners susceptible to information overload table 3 summarises interviewees’ reported readiness for learning in the current online course based on data obtained from the questionnaire, and their perceived io as revealed in interviews. this table shows that interviewees who were identified as at risk of io usually did indeed express difficulties related to io during the course. participants were identified as at risk of io if they possessed a low level of any one research in learning technology 107 of these variables: prior knowledge, english proficiency, or technical skill for participating in cmc. survey findings thus identified six students at risk of io. additionally, each participant’s reported difficulties related to io were classified into one or more of these areas in terms of quantity of information, quality of information, and medium interface. consequently, interview findings revealed that eight out of 12 students perceived io. two students (alan and carl) were identified as at no risk of io in the survey, but their interviews revealed their perceived io. in addition to the three constructs (as identified from the literature) leading students to be more susceptible to io, other variables emerged from the interviews. the emerging data-driven variables were lack of efficiency in reading from computer screens, visual and table 2. criteria to identify interviewees’ level of background knowledge/skills. part construct criteria level i prior knowledge previous undergraduate or graduate major was relevant to that person’s current major low: satisfied none had taken at least one course relevant to the current course medium: satisfied either one high: satisfied both ii english proficiency a native english speaker low: satisfied none indicated either fluency or possessing a good level of english proficiency medium: satisfied either one high: satisfied both iii online courses experience had not taken any online courses previously novice: satisfied first had taken more than two online courses previously moderate: satisfied none experienced: satisfied second iv technology use experience indicated an average level of technology use that was: novice: satisfied first � less than two � two � three or more moderate: satisfied second experienced: satisfied third iii and iv combined technical skills for participating in cmc identified as a novice online learner and technology user low: satisfied first identified as a moderate/experienced online learner and technology user medium: satisfied none high: satisfied second 108 c.-y. chen et al. t ab le 3 . s u m m ar y o f in te rv ie w ee s’ re p o rt ed re ad in es s an d p er ce iv ed io . s tu d en t (n = 1 2 ) l ea rn er re ad in es s a t ri sk o f io p er ce iv ed io p ri o r k n o w le d g e e n g li sh p ro fi ci en cy t ec h n ic al sk il ls fo r p ar ti ci p at in g in c m c q u an ti ty o f in fo rm at io n q u al it y o f in fo rm at io n m ed iu m in te rf ac e a la n h ig h h ig h h ig h n o u u b il l l o w m ed iu m l o w y es u u u c ar l h ig h h ig h h ig h n o u u d o ri s l o w m ed iu m l o w y es u u e ri c m ed iu m h ig h h ig h n o f ra n ce s l o w h ig h m ed iu m y es u u g ra ce l o w h ig h h ig h y es u h el en h ig h h ig h h ig h n o iv an l o w h ig h m ed iu m y es u u ja ck m ed iu m m ed iu m m ed iu m n o k ev in h ig h l o w m ed iu m y es u u u l il y m ed iu m h ig h m ed iu m n o n o te : a fi rs tn am e o n ly p se u d o n y m w as u se d to p ro te ct ea ch p ar ti ci p an t’ s id en ti ty . research in learning technology 109 auditory learning preferences, and time constraints. time constraints were mainly due to the demands of job and family responsibilities. io was usually a result of several variables rather than a single one. difficulties contributing to perceptions of information overload in cmc interview findings revealed a number of difficulties that contributed to students’ perceived io, and were organised into six broad categories: (a) connection problems; (b) navigation difficulties; (c) discomfort with online communication; (d) demands of ongoing discussions and readings; (e) difficulty in organising learning; and (f) problems understanding the text-based readings. connection problems students who lacked technical skills for participating in cmc were observed likely to encounter connection problems. the students’ connection problems were inability to log on to the conferencing system, and inability to access the electronic materials from the library. the first problem was usually due to the failure to set up the connection. the second problem was caused by a firewall. this firewall problem meant students were unable to access the electronic materials that were the required readings from the library. when they needed to pay attention to so much information at the beginning of the semester, students easily failed to attend to the connection instructions from the course website. consequently, students might fall behind at the beginning of the semester. one student encountering the firewall problem was behind in coursework for approximately five weeks, as exhibited by participating late in each week’s online discussion. navigation difficulties over one-half of the students, including experienced technology users, encountered navigation difficulties. those difficulties included disorientation when browsing the web and computer conferencing, and difficulty linking discussion messages. first, students’ disorientation on the web resulted primarily from external web resources, which were organised in a hypertext structure of more than three levels. second, perceived disorientation in computer conferencing was due to the multiple conference rooms. when they were still unfamiliar with the conferencing structure and were unaware of their target location, some students became frustrated by spending a lot of time trying to find the target conference. third, several students found it difficult to link discussion messages. some students could not understand from the screen which person had replied to which message. one student offered a vivid metaphor to describe his confusion when the online discussions got busy: “it was like you sat in a room and talked to four or five persons at a time. very confusing”. off-topic discussions in which someone changed the discussion topic without creating a new thread made navigation more difficult. 110 c.-y. chen et al. discomfort with online communication students’ technical skills for participating in cmc and efficiency (speed and comprehension) in reading from computer screens influenced their level of comfort with online communication. students who identified themselves as computer illiterate indicated that they needed to make a greater cognitive effort to figure out how to interact with the medium interface, while simultaneously having to process the discussion messages. two students said that they had to print out all course materials because they had difficulty reading so much from a computer screen. one of these students said that his slow speed in reading from computer screens, his lack of computer skills, poor typing skills, and job-related and family-related time constraints made him feel reluctant to participate online. this led to infrequent logins and, subsequently, the number of his unread discussion messages mounted. students reflected that online communication was a time-consuming process, so they appreciated the opportunities that classroom learning offered for immediate dialogue and immediate feedback loop. demands of ongoing discussions and readings over one-half of the students, especially those with insufficient prior knowledge, encountered this difficulty (i.e. numerous ongoing discussion messages along with the many resources provided on the course website). for students with full-time jobs and family responsibilities, this difficulty was a product of time constraints. however, some students finally developed information selection strategies to resolve the problem. they focused on information needed to fulfil course requirements and did not delve into extra resources unless time permitted. difficulty in organising learning over one-half of the students expressed difficulties in organising their learning. they indicated that organising learning was challenging when there were numerous ongoing learning activities, including continual online discussions. the following situations compounded this problem: demands of job, family responsibilities, or both; taking another online course at the same time; and novelty of the approach for first-time online learners who had to adjust to the differences of this mode of learning from the format of regular once-a-week on-site classroom learning. problems understanding the text-based readings students who identified themselves as slow readers and with visual and auditory learning preferences indicated that they had this problem. they could no longer rely on the instructor’s lectures and verbal discussions for help as they had previously in classroom learning. this diagnosis is consistent with the self-report of their english proficiency obtained from the questionnaire, which revealed their english competence to be at only a fair level. the variables of both inadequate language proficiency and visual and auditory learning preferences evidently disadvantaged students in a text-based learning environment. research in learning technology 111 discussion this study aimed to explore the sources of online students’ io and accordingly offer considerations for increasing students’ cognitive resources to focus on content learning. varied learner characteristics led some students to be more susceptible than others to io. those variables in turn probably led to students’ difficulties in one or more of the following different dimensions: medium interface, quantity of information, and quality of information. those difficulties, in turn, contributed to students’ overall perceptions of io. io was usually a result of several variables rather than a single one. moreover, over one-half of the students encountered both problems (i.e. demands of ongoing discussions/readings and difficulty in organising learning) in the dimension of quantity of information, and navigation difficulties in the dimension of medium interface. the importance of prior knowledge was confirmed in this study. as indicated in table 3, students with low levels of prior knowledge (i.e. bill, doris, frances, grace, and ivan) were not only identified as at risk of io but also revealed their learning difficulties associated with io from their interviews. it was further observed that those students tended to encounter both problems in the dimension of quantity of information (i.e. demands of ongoing discussions/readings and difficulty in organising learning). high levels of prior knowledge imply that schemas are readily available in long-term memory and these schemas serve as advance organisers that help to interpret sensory information and link it to the existing schema (valcke 2002). the available schemas thus provide executive guidance to the selection and processing of incoming information and thereby assist learners in their organisation of the learning activity (kalyuga 2009). it should be noted that some students with high levels of prior knowledge (i.e. alan, carl, and kevin) also perceived io, as revealed from their interviews. the variables that possibly led them to experience difficulties associated with io included inadequate english proficiency, visual and auditory learning preferences, lack of efficiency in reading from computer screens, and time constraints. cognitive load theory thus neglects characteristics of learners other than prior knowledge that should also be considered in certain learning contexts such as cmc. rouet (2009) similarly emphasised the need to control other variables like individual characteristics and the task setting, not just learners’ prior knowledge of the content area, in order to obtain reliable assessments of cognitive load and learning outcomes. however, to deal with the problem regarding demands of ongoing discussions/ readings, several students relied on external means to identify relevant information to fulfil course requirements. the external means they mentioned included assignments, learning objectives, and overviews of the module content. those external means were equivalent to the schemas that serve as advance organisers. the instructor can design such advance organisers or knowledge maps to provide students executive guidance as to what information to process, and how and in what sequence to process it. students identified as either novice online learners or having time constraints, including students with high prior knowledge, were observed to be more susceptible to the difficulty of organising learning. those cases offer some considerations for online instructors. the literature noted that time management is an important factor in the success of most distance students, particularly those individuals in mid-life who are busy with the demands of job and family (moore and kearsley 1996). 112 c.-y. chen et al. such students need guidance to help them organise learning. likewise, first-time online students need guidance in time management to adjust to the differences between online learning and on-site classroom learning. online instructions that are designed and implemented from a constructivist perspective rely heavily on students to manage their own learning tasks and engage in interaction with peers and content (vrasidas 2000). regarding the medium interface, the literature suggested consistently that students’ navigation struggles are inevitable while browsing hypertext systems (destefano and lefevre 2007). in the current study, students’ reflections regarding the levels of interface layers may have implications for interface design. two students who possessed a high level of computer competence indicated that, in their personal experience, people got confused easily when the hypertext structure was organised more than three levels deep. similarly, other students remarked that they preferred a linear as opposed to a non-linear style for browsing web pages. when the number of interface layers exceeds three levels, users may feel that the information is fragmentary. the observations parallel the reports of earlier studies on hypertext environments that favoured linear presentation of material over hypertext formats because the linear formats provided more structure with less disorientation (jonassen and wang 1991; nelson and joyner 1990). similarly, it has been reported that while navigating hypertext, students read the material in a more sequential manner and default to use of print-based text reading strategies to minimise cognitive load (niederhauser et al. 2000). the cognitive demands of the threading structure of computer conferencing, particularly the fragmentation of information caused by the non-linear discussion format, were evident to most students in both classes. fragmented information creates a split-attention effect (sweller, van merriënboer, and paas 1998) that may occur in computer conferencing when students have to integrate messages from different contributors regarding different topics. off-topic discussions – when the line of discussion loses its coherence – create difficulties in linking messages belonging to a particular topic. in order to help students process online information more effectively, the instructor or discussion facilitators can offer guidance at the beginning of the course on how to navigate messages effectively by utilising system functions such as the selecting, sorting, and summarising features of computer conferencing systems. several suggestions have been made to avoid fragmentary information and confusion in online discussions (hewitt 2001; salmon 2000). implications for course design this study has several implications for course design. they consist of: readiness for online courses, design issues for the course website, and design issues for online discussions. first, readiness for online courses is essential for reducing students’ cognitive demands from learner–interface interaction (hillman, willis, and gunawardena 1994), as well as ensuring their familiarity with course structure and expectations. the following suggestions will assist online educators in ensuring students’ readiness: � most students in the present study indicated that the orientation sessions were an immense help in resolving technical problems, as well as offering an research in learning technology 113 opportunity to understand the course requirements and the instructor’s expectations. thus, an orientation meeting may be the key to preparing students for the online course. � an orientation session alone may not be sufficient to prepare students for learning online. students may require more time to become familiar with the online environment. two weeks is the period of time that most student participants in conrad’s (2002) study recommended for sufficient preparation. � in order to give students enough time to resolve technological problems, the workload at the beginning of the semester should not be too demanding. � it is important for the instructor to provide students with guidance in time management at the beginning; those with work or family responsibilities and novice online learners particularly need guidance to help them organise learning. second, this study suggests ways that the design of the course website might help manage io: � when having difficulties dealing with the quantity of information on the course website, several students noted that they relied on external means to identify relevant information to fulfil course requirements. the external means they mentioned included assignments, learning objectives, overviews of the module content, and fundamental concepts. the instructor can make use of external means to assist students with insufficient prior subject knowledge in identifying relevant information. � some students, particularly those learners with certain visual and auditory learning styles, commented that when they could not understand something on their own, they tried to interact with peers on the telephone, in person, or by email. such learners can be encouraged to find learning partners. in addition, online examples of previous or current students’ assignments would assist students’ accomplishment of the learning tasks. � to avoid navigation difficulties, most students in this study indicated that they preferred a linear as opposed to a non-linear style for browsing web pages. in addition, a web page containing more than three levels easily confused students. third, this study also suggests ways that the design of online discussions might help manage io: � first, the instructor or discussion facilitators can remind students how to navigate messages effectively when discussions become very active. second, the instructor or discussion facilitators can weave and summarise messages at the appropriate time. third, the instructor or discussion facilitators can reorient the discussions when they stray off-topic. finally, the instructor or discussion facilitators can provide immediate guidance to students who are not on the right track. � at the beginning of the course, the instructor or discussion facilitators can offer guidance and instructions on how to navigate messages effectively by utilising the system functions such as the selecting, sorting, and summarising features of the computer conferencing systems. 114 c.-y. chen et al. � a deep level of non-linear style of conferencing structure is not preferred. most students responded that when the levels of conferences went beyond three, they became easily confused. conclusions it should be noted that the results of the current study are based on a small sample of student volunteers, and although quantitative data from a questionnaire were used, the emphasis is on qualitative findings. this study is limited to the context and setting of the two online courses, which needs to be taken into consideration to make transferability judgements. recommendations for further investigations follow. first, this study focused on a specific group of graduate students taking reading and writing intensive classes in the field of education. similar studies could be carried out with different target learners such as undergraduate students, with learners majoring in different kinds of subject matter, or with students learning in different domains (e.g. scientific and technological fields). second, this study examined a particular technology format (text-based cmc). a similar study could be conducted in online courses incorporating different technologies such as audio, video, or multimedia technologies. last, most research so far focuses on the external management of cognitive load by means of optimal instruction, but neglects consideration of learners’ internal management strategies (bannert 2002). in accordance with bannert’s call, future research may involve investigation on how learner characteristics and learning strategy use, particularly, metacognitive strategies, moderate perceived io or cognitive load to achieve effective learning in different learning contexts. references bannert, m. 2002. managing cognitive load – recent trends in cognitive load theory. learning and instruction 12: 139–46. bellingham public schools. 1999. staff use of technology: 1999–2000 self-evaluation rubric. http://www.bham.wednet.edu/tcomp.htm. burge, e.j. 1994. learning in computer conferenced contexts: the learners’ perspective. journal of distance education 9, no. 1: 19–43. conklin, j. 1987. hypertext: an introduction and survey. ieee computer 20, no. 9: 17–41. conrad, d.l. 2002. engagement, excitement, anxiety, and fear: learners’ experiences of starting an online course. american journal of distance education 16, no. 4: 205–26. destefano, d., and j.a. lefevre. 2007. cognitive load in hypertext reading: a review. computers in human behavior 23: 1616–41. eastmond, d.v. 1995. alone but together: adult distance study through computer conferencing. cresskill, nj: hampton. edmunds, a., and a. morris. 2000. the problem of information overload in business organizations: a review of the literature. international journal of information management 20: 17–28. eppler, m.j., and j. mengis. 2004. the concept of information overload: a review of literature from organization science, accounting, marketing, mis, and related disciplines. the information society 20: 325–44. harasim, l.m. 1987. teaching and learning on-line: issues in computer-mediated graduate courses. canadian journal of educational communication 16, no. 2: 117–35. harvell, t. 2000. costs and benefits of incorporating the internet into the traditional classroom. unpublished doctoral diss., texas a&m university, college station, tx. hewitt, j. 2001. beyond threaded discourse. international journal of educational telecommunications 7, no. 3: 207–21. research in learning technology 115 hillman, d.c., d.j. willis, and c.n. gunawardena. 1994. learner interface interaction in distance education: an extension of contemporary models and strategies for practitioners. american journal of distance education 8, no. 2: 30–42. hiltz, s.r., and m. turoff. 1985. structuring computer-mediated communication systems to avoid information overload. communications of the association for computing machinery 28, no. 7: 680–9. jonassen, d., m. davidson, m. collins, j. campbell, and b.b. haag. 1995. constructivism and computer-mediated communication in distance education. american journal of distance education 9, no. 2: 7–36. jonassen, d.h., and s. wang. 1991. conveying structural knowledge in hypertext knowledge bases. paper presented at the annual meeting of the association for educational communications and technology, february, in orlando, fl. kalyuga, s. 2009. knowledge elaboration: a cognitive load perspective. learning and instruction 19: 402–10. kear, k.l., and n.w. heap. 2007. ‘sorting the wheat from the chaff’: investigating overload in educational discussion systems. journal of computer assisted learning 23: 235–47. klausegger, c., r.r. sinkovics, and h. zou. 2007. information overload: a cross-national investigation of influence factors and effects. marketing intelligence and planning 25, no. 7: 691–718. lincoln, y.s., and e.g. guba. 1985. naturalistic inquiry. newbury park, ca: sage. mason, r., and t. kaye. 1990. toward a new paradigm for distance education. in online education: perspectives on a new environment, ed. l. harasim, 15–38. new york: praeger. moore, m.g., and g. kearsley. 1996. distance education: a systems view. belmont, ca: wadsworth. nelson, w.a., and o.j. joyner. 1990. effects of document complexity and organization on learning from hypertext. paper presented at the annual meeting of the eastern educational research association, february, in clearwater, fl. niederhauser, d.s., r.e. reynolds, d.j salmen, and p. skolmoski. 2000. the influence of cognitive load on learning from hypertext. journal of educational computing research 23, no. 3: 237–55. paulo, h.f. 1999. information overload in computer-mediated communication and education: is there really too much information? implication for distance education. unpublished master’s thesis, university of toronto, canada. rouet, j.f. 2009. managing cognitive load during document-based learning. learning and instruction 19: 445–50. salmon, g. 2000. e-moderating: the key to teaching and learning online. london: kogan page. schwan, s., d. straub, and f.w. hesse. 2002. information management and learning in computer conferences: coping with irrelevant and unconnected messages. instructional science 30: 269–89. sweller, j. 1994. cognitive load theory, learning difficulty and instructional design. learning and instruction 4: 295–312. sweller, j., j.g. van merriënboer, and f.g. paas. 1998. cognitive architecture and instructional design. educational psychology review 10, no. 3: 251–96. trevino, l.k., r.h. lengel, and r.l. daft. 1987. media symbolism, media richness, and media choice in organizations: a symbolic interactionist perspective. communication research 15, no. 5: 553–74. valcke, m. 2002. cognitive load: updating the theory? learning and instruction 12: 147– 54. vonderwell, s., and s. zachariah. 2005. factors that influence participation in online learning. journal of research on technology in education 38, no. 2: 213–30. vrasidas, c. 2000. constructivism versus objectivism: implications for interaction, course design, and evaluation in distance education. international journal of educational telecommunications 6, no. 4: 339–61. 116 c.-y. chen et al. alt-j volume 4 number 2 reviews edited by philip barker seels, barbara b. and richey, rita c , instructional technology: the definition and domains of the field, washington dc, association for educational communications and technology, 1994. isbn 0-89240-072-2. editorial note this review differs from others we have so far published in alt-j. in order to offer a multiple perspective on a contentious area, we approached three reviewers. we then asked barbara seels and rita richey, in the light of the reviews we received, to reconsider their position set out in their book. in addition, ray mcaleese (heriotwatt university) has commented both on the reviews and on the reactions to them by seels and richey. we welcome any comments you may have. the order in which the reviews are published here has been determined by the extent to which the reviewers offered a description of the book's contents. this review, written by the foundation director of the institute for education technology (iet) at the open university, reflects the ou'spre-eminence in course design. the iet was an early pioneer of collaborative course teams by including an educational technologist as well as a subject specialist. david hawkridge, institute of educational technology, the open university email: d.g.hawkridge@open.ac.uk what is instructional technology (or, for that matter, educational technology)? that is a question asked repeatedly by lay people when they first hear the term, and it can be embarrassing, even tiresome, to have to explain. it is therefore entirely appropriate that aect (association for educational communications and technology), founded in the usa, should have fostered this plain book about changes since 1977 in instructional technology as perceived by practitioners in the usa. 61 reviews aect does not pretend to be an international organization, and its committee on definition and terminology consulted widely within the usa, not beyond. nevertheless, all who work in the broad field of applying technology in education can benefit from reading this book. the term instructional technology has no precise equivalent in europe. educational technology overlaps with it, as seels and richey note, but is it closer to tecnologie didattiche (italian), unterrichtswissenschaft (german) or tecnologia educativa (spanish)? the definition eventually agreed by aect in 1994 is that instructional technology is the theory and practice of design, development, utilization, management and evaluation of processes and resources for learning. seels and richey tackle the issues of definition rather thoroughly, as we would expect in 124 pages. drawing on a modicum of recent writings, they analyse in detail and comment upon the five domains in the definition. fair enough, but then they discuss the sources of influence on instructional technology. here they correctly note the major influence of positivist science, particularly behaviourism, without giving enough space to criticisms of its failures. in my opinion, they overestimate the extent to which theory has determined practice: a great deal of instructional technology in practice is still atheoretical, much like a craft. having said that, however, am i right in saying that nowhere is there an exposition of the ontological and epistemological roots of instructional technology? if so, that is disappointing in a reference work like this. was it left out because committees do not like such serious and abstract discussion? or because nobody wants to know about it? seels and richey describe briefly and uncritically certain aspects of practice and the practitioners. in their efforts to systematize instructional technology, they seem to focus their attention on what they believe they have seen rather than on what might be best practice. their view of the practitioners is clear but limited when compared with, say, eraut's book developing professional knowledge and competence (1994), which draws on schon's ideas. finally, seels and richey examine the role and implications of the definition. it will aid communication and build a sense of community, they say, because it will add to shared understanding. they also suggest that the new definition, because it is different from the 1977 one, will aid new agendabuilding. the chief difference, for me, is that in 1977 instructional technology was defined as a practiceoriented problem-solving process, while in 1994 the definition is oriented towards both theory and the american national perspective does come over in the book, in spite of aects attempt to be more international. there is a degree of parochialism in the citations, and the flavour of the book defines it as american. i am glad the reviewer has made the point about craft. many activities are craft-like in their eclectic use of multiple paradigms and atheoretic assumptions. i do not feel that craft should be taken to imply 'lacking professionalism'; rather, it should locate the practices in the authentic world of learners and designers students and teachers. 62 an-} volume 4 number 2 this must be getting at the heart of the matter. practice leads to better practice and the development of criteria for producing such practice. these criteria must in time become an integrated body of knowledge, a theory of practice. therefore, one would imagine that as the practice-based approach becomes more useful, the acquired procedures and knowledge contribute to a theory. the question one is left with when one reads the seels and richey book is 'has learning technology come far enough?' one can not but be impressed with a code of ethics. the aect code certainly makes one think about the responsibilities designers have to ensure that knowledge is made accessible and is not hidden by a particular design philosophy. as historical gate-keepers, teachers and designers have this ethical role with regard to how learners acquire skills and competencies. as ncet is the official government-sponsored agency that supports, applies and advises on what is known about educational technology, one would expect this review to take a utility view. ncet plays a middle-of-the-road role between the academic community and the marketplace. abstract: no. academic: yes. if theorybuilding is not part of a discipline and profession, i do not know what is. i suspect margaret bell is more concerned with the utility of such a framework than how it contributes to the professionalization of the field. practice. seels and richey say that the change implies the need for increased research and theory construction unique to this field, and decreased reliance upon the products of the research and theory of other fields. for me, there's the rub. the boundaries of instructional technology are fuzzy, thank goodness, and in its eclectic approach should lie its strength. it cannot stand apart. it must change with changes in the arts and sciences it depends upon. for good measure, seels and richey add a glossary. the terms selected indicate well the book's lusty positivist bias. three appendices provide data of variable potential value. do you really want 40 other glossaries of instructional technology? the directory of associations and their publications may be useful to students and practitioners, but is already looking dated. the aect code of ethics makes interesting reading, but i probably would not be able to sign up to it because i do not know how to represent accurately and truthfully the facts concerning educational matters. the facts change and need definition and redefinition. so does instructional technology. margaret bell, chief executive, national council for educational technology email: margaret_bell@ncet.org.uk i should make it clear that i am not an academic, and am not familiar with the systems and processes which are of relevance and important in academia. i looked at this book in terms of its practical use in progressing the development of technology-based learning materials. i would be interested to know if the contents have been tested with a commercial producer of instructional technology, and if so, whether they were found to have made a contribution. i am not convinced of the value of building an abstract academic subject of instructional technology. understanding the effectiveness of instructionaltechnology materials, the full potential of them and the complexity of their design, is a living topic, one which continues to grow and develop rapidly. in my opinion, this approach could, at worst, hinder that natural progression and, at best, rapidly become out of step with what is happening on the ground. chapter 1 left me with only one question: 'so what?'. why do we need a definition which is so carefully worked and in so much detail. understanding of this area is in a fluid state, the technology is developing rapidly, new opportunities are opening up, and our understanding of technology's application to learning is growing. this continued development and change 63 reviews cannot be tamed, controlled or better understood by a definition; neither should progress be constrained by it. there is no evidence that the development of instructional technology has been hampered through lack of definition. throughout this chapter there is, in my view, too much emphasis on instruction and teaching; not enough on learning. the main impact of the introduction of technology into the learning process is that students have more opportunity to develop into active learners. it changes the focus from the teaching to the learning process. in designing computer-based learning materials, we are challenged much more by understanding the necessary support to learning than in the required support to teaching. is it the lack of taxonomy that impedes communication and discourages transfer of expertise from one technology to another? in any case, does the taxonomy suggested in chapter 2 not need to change to accommodate advances in technology? the management domain does not appear to include learner management. this is a very important part of technology-based learning, and may provide the key to effective open and flexible learning. in the development domain, what is meant by integrated technologies? the integration of what? cbt is an integration of approaches; it integrates the intelligence of the computer with text, graphics, etc., and now increasingly with pictures and sound. cbt and integrated technologies are surely part of the same continuum. i am not sure that there is a common understanding of the terminology used to describe the domains. i agree with the authors that integrated systems design is an iterative process, but where, in the four areas of the domain of design, is there consideration of exploitation of the media? new media allow approaches not previously possible. how are these identified and integrated into the design? my experience is that new technologies require a re-think of learning processes. learning styles (cognitive, etc.) should be recognized and selected, but there is no reason why more than one should not be accommodated within a single product, adapting to the learner's requirements as identified and demanded. in the domain of development, i am not sure why print is considered the foundation of all others. some products do not have print. print, audio-visual media and cbt are all components of integrated technologies. the key characteristics of computerbased materials are not those that are visible. you cannot, therefore, categorize/define the media by the visible characteristics. how will the new electronic there is evidence that the imprecise definition of the field affects the way learning technologists are integrated into the academic community (cummings, 1996). surely learning technologists need some form of accreditation? that makes sense. the book does seem to avoid, if not miss, a perspective from the learner. of course, the authors set out to examine the design of learning opportunities. one might argue that the whole emphasis should be learning technology, not instructional technology. taxonomies do seem to have an 'academic' flavour. what would be useful in the book is to have more instances and examples. with a classification and instances (for example, 'positive' and 'negative'), the reader might make more use of the framework. for an early taxonomy, readers should look at bretz (1971) or heidt (1978). this seems very important. the mean-endmeans dialectic of instructional design often alerts the designer to new media opportunities, and such opportunities often lead to radical new designs. volume 4 number 2 the issue here, to my mind, is that of convergence. there is an inevitable convergence of all forms of technology. this will lead to less emphasis on media classification and selection (see above). to some it may be a pity that the unique characteristics of some media are lost, for instance the entertainment aspect of early film and the fun of early simulation games. no, but practice or utility are not sufficient attributes for a technology of learning or instruction. this is a pity. i sense that there is a growing approach in education to the utility of skills for the present (being fit for the job) as opposed to an investment in generic skills (of use in some unknown setting). commercial instructional technology may have to be more 'immediate' in its design philosophy. communication technologies be categorized, and what will be the result of the convergence of telecommunication and multimedia? the main problem with this chapter, as i see it, is that it does not recognize that the relationships between and within the processes are more important than the component parts. in interactive design, the relationships and links are more important than the individual components. i do not believe the instructional design process can be understood through a reductionist approach. the principles of cognitive science are not confined to the development of cbt. (again what is the difference between cbt and integrated technologies?) even in cbt there must be some visual presentation of material, and most computers now have sound output. the concepts of media utilization and media selection are being overtaken by the digitization of the full range of media, and this is being driven by the increasing availability and bandwidth of electronic communications. we are moving rapidly to a situation where all media will have a common form; different types of information will be selected and combined for specific purposes. in chapter 4 (the practice of instructional technology) the authors state that 'practice has had more influence than theory'. i agree entirely. but i was very interested in the assertion that instructional technology has moved from a craft via a profession to a field of study. is it that mature? if so, has it followed that path? can the field of study be separated from the profession and practice? i would suggest that the developments have been, and still are, so rapid that a different approach may be needed. new stages in the history of instructional technology have often brought new practitioners; not necessarily learning from their predecessors. for example, the introduction of interactive video brought a number of video producers into the field who made many of the mistakes evident in the early years of cbt. commercial producers work in a competitive environment; this can cause resistance to sharing experience and expertise. my understanding of the link between a field of study and a profession is that the route to the profession is established through academic achievement; i think there is little evidence of that in commercial instructional technology. the implications in the last chapter are not positive ones. trying to capture and define the process has led to a feeling of stagnation, inflexibility and immobility in an area that is at the core of the changes impacting 65 reviews on society. this is a dynamic, iterative process; it cannot be captured in a static medium. this area of work and thought may need a different approach. i was left wondering about the purpose of the book. i assume it is to carve a position for instructional technology in academia; perhaps to fit educational technologists into the academic hierarchy. it may also be driven by a need to create a recognized profession and professionalism in this area. all of those reasons are valid and probably worth pursuing. however, they are not on my agenda, nor do they contribute to my personal or corporate objectives. i am not qualified to comment on what a group of scholars agree to agree on. derick unwin, formerly head of educational research and development at queensland university of technology, brisbane, australia email: eemcat@ozemail.com.au as one who in recent years has withdrawn from dayto-day involvement with instructional/educational technology (edtech), i was particularly interested to see if this latest attempt to define and delimit would appeal to me as a non-practitioner. i have to report that little has changed over the years: all is still enmeshed in a thicket of brer rabbitry. this is almost certainly inevitable: the academic leaders in the field have always tried to draw together threads which resolutely resist any such taxonomic weaving. in their seminal work defining the scope of educational technology lumsdaine and glaser (1960) even included arcane devices such as one for improving firearm trigger-work as an example of the genre. the 1960s impetus was very much geared to the concept of the teaching machine, with a generous overlay of psychological justification. since that time, each and every promising candidate and some not so promising from the worlds of entertainment, military training, business management, computer technology, etc. has been press-ganged into educational technology, along with all the associated jargon and anecdotal evidence. not surprisingly, the notion of what educational technology actually is becomes a little hazy along the way. one solution would be to set out an operational definition; i.e. a statement of what educational technology really does. the problem with this approach is that it yields slim pickings: few of us could put hand on heart and list a series of benefits (to learners not to educational-technology professionals!) which have arisen as a result of all the educational technology developments since, say, 1954. in fact, an this view is surely a little too antischolarship. clearly, the real-world trainers need results not theories, but is not the approach that education supports of providing a generalizable explanation of what to teach, as well as how to teach, worth maintaining? this review should be influenced by the fact that derick unwin was one of the founding fathers of programmed learning in the uk he has seen the emergence of the technology of instruction and education over 30 years. of course, it may be that with such a historical perspective he may yearn for the values of early uk-based programmed learners. one must read the enthusiasm of the early programmed learners to wonder where things went wrong. for example, annett (1964) writes of'putting teaching on a firmly scientific basis and from this develop a new technology adequate to meet the rapidly growing demands for training and education. it is fair to comment that this reviewer had his own attempts to classify media and provide guidelines for its use (unwin, 1969). 66 alt-j volume 4 number 2 the word academic has such a pejorative ring. perhaps there is a free masonry to the profession. certainly to the uninitiated, design models that draw on 'epitomes' (rigileuth) and 'algo-heuristic' paradigms (landa) have the feel of arcane practice. there must be strength in having denotive models and practices ifand only ifthey are used and are found to be useful. unbiased observer of the educational scene might feel quite justified in concluding that the output of the educational systems in western countries has improved little, if at all, in spite of the staggering increase of available technological resources, and huge investments in the design of materials. it is noteworthy that at the cutting edge of instruction, e.g. the provision of 'tutorials' for computer software, despite the use of interactive cd-rom and other 1990s reincarnations of sidney pressey's teaching machine (pressey, 1926), we frequently encounter absolutely abysmal teaching materials. nor does the quality of instruction encountered in universities of the air seem to have advanced much further than the 'tell 'em what you're going to tell 'em, then tell 'em, then tell 'em what you've told 'em' school of educational communication. instructional and educational technology can be usefully compared to the entertainment industry. both are concerned with communication, and both have evolved from very basic uses of human skills and rudimentary technology, through to exploitation of the latest electronic and optical techniques. both sets of endeavours employ a smorgasbord of experts, and in each case there is an end product, hopefully tailored to the requirements of the consumer. why is there no equivalent in educational technology of internationally recognized producers, directors, performers, authors, photographers, and so on? in fact, the only celebrities in educational technology are not practitioners at all: they are prominent academics whose main hands-on familiarity with educational technology is using a word processor to write lecture notes for their own courses. without being discourteous, one might diffidently suggest that educational technology as defined and described in the volume under review has effectively been hijacked by an academic masonry, fiercely defensive of its rituals and esoteric protocols. in this context one can only admire the thoroughness of seels and richey: in a mere 124 pages (plus appendices, etc.) they have visited every possible aspect of their subject matter. this is a t our deforce in its coverage of the liturgy of academic educational technology in 1994, a veritable catechism for the faithful. there is no doubt that members of educational technology academia will find much to applaud in this rigorous exposition of the minutiae of their tools of trade. perhaps fortunately, nothing here is likely to have much effect on learning and teaching. 67 reviews barbara b. seek and rita c. richey: a considered reply (april 1996) email: bseels+@pitt.edu & rrichey@cms.cc.wayne.edu a mature field demands a complementary and continuous relationship between practice and theory. instructional/educational/learning technology has progressed to t h e ' point where both are well established. when a book intended to explain this interaction of practice and theory is examined only from a practitioner orientation, distortions can occur. our book describes the evolution of our field. the book was not intended to be evaluative; therefore, positions were not taken about one theoretical orientation or another. apparent emphases only reflect historical development. depending on the setting in which one works, there is a tendency to feel that more attention should have been paid to a specific theory. for example, many people in schools voice a concern for more attention to constructivism (duffy and jonassen, 1991; wilson, 1996). conversely, many working in corporate training environments would prefer more emphasis on performance technology (geis, 1986). these controversies reflect the healthy diversity of the field. we deliberately included alternate perspectives, and viewed the field as broad. and inclusive. one important area of discussion, as the project progressed, was the choice of domains and their components. we anticipate that both the subcategories and the domains will grow and change as the field develops. furthermore, the taxonomy presented in the book represents the points of view of a group of people at a given period in time, and continued analysis and discussion is desirable. for example, we think the recommendation of the addition of learner management is valuable. it is true that the field has expanded historically primarily through changes in practice. the increasing demand for practice has created pressure for the many academic programmes in the field. in the united states, most practitioners either enter the field from academic programmes, or enter an academic programme after obtaining a position in the field. there are about 50 doctoral programmes here, and more than twice that number with master's and six year programmes. in addition, there are over 50 graduate programmes in educational computing. these programmes are listed annually in the educational media and technology yearbook published by libraries unlimited. there is also pressure for synthesis from over 40 years of extensive research, including the development and the authors of the book were given the opportunity of reading the reviews above and of commenting. they were asked to extend the debate rather than make counter-claims or concentrate on justifying their own approach. this was, then, the starting point. the field has come far enough to merit careful scrutiny, and one test that needs to be applied is 'can the pragmatics of practice lead to a sustainable theory?' of course the book like any volume is time-stamped. but the question here is: 'has the field come far enough to merit a defining treatment?' it is not a question of how seels and richey view it: it is instructional technology that is under the microscope. the impressive influence of the id 'free masonry' is not to be ignored. surely the graduates of such programmes represent a critical audience for the value that a training in instructional technology provides? 68 alt-} volume 4 number 2 this is where one wonders about the influences of instructional theorists on the aect definition. although there is ample citation given to theorists such as rigileuth's (1983) elaboration theory, such theories are not instantiated in the aect definition. one is left with a question about the atheoretic view this definition of instructional technology has. i do not see an attempt to synthesize merrill's component display theory with gagne-briggs' events of instruction. perhaps aect and its expert group could not reach a negotiated settlement based on the conflicting theories of instruction. the authors are a little self-conscious here. the reviewers have not necessarily found that the book has no use; rather, one senses that they feel that the field does not merit such treatment. this is where the atlantic divide makes most impact. it may well be that there is a considerable influence felt in schools, training organizations and even business due to instructional technology in the usa. the question one is left with is: 'why has this not happened in the uk?'(see the reference to the development of programmed learning above). i would like a citation here to corroborate the assertion that instructional theory has an effect on teaching and learning. dissemination of theory. the existence of this research and theory is documented by the forthcoming handbook of research on educational communications and technology edited by david h. jonassen and being published by macmillan. moreover, pergamon has just released the second edition of the international encyclopaedia of educational technology edited by tjerd plomp and donald p. ely, which emphasizes the field's rich theoretical history. while we do not feel adequate to describe the situation in the united kingdom, in the united states the areas of theory and practice are equally dynamic. here, academic-programme faculty rightfully see themselves as training practitioners as well as those who will teach in higher education settings and do research. it is seldom that one teaches or does research here without also developing materials and consulting on design. it seems likely that this orientation to academic programmes was not shared by the reviewers. consequently, cross-cultural differences between our two countries lead to conflicting viewpoints. we would like to explain why the book was written because the reviewers seemed to feel the project was neither needed nor useful. the aect has an international focus. it has members from 42 countries, including a large contingent from canada and a wellestablished international division. many of those who attend aect's annual convention are international members. this convention is regarded by many as one of the most dynamic convocations of professionals in the field offered today. aect has a long standing committee on definition and terminology which has published a consensus on the field's functions and identity in 1963, 1972,1977, and now in 1994. the committee involved with the 1994 redefinition effort, which was an attempt to update language and description about the field, decided to focus the effort nationally in order to make the task manageable. however, several members of the committee were active internationally, and one member was from canada. one of the reasons aect makes a continuous commitment to the process of redefinition is to improve communications with the many organizations that contribute research and practice to the field and that use instructional technology. as for the point that the field and its theory have no effect on teaching and learning, research does not support this. research and theory help practitioners justify their roles and predict the impact of specific strategies and techniques. this is especially important in an era of limited resources. it is much more difficult to defend programmes and jobs without a consensus on identity 69 reviews and areas of contribution. it takes the joint efforts of practitioners and academicians to sustain a field and profession. the book is intended for all those in the field: researchers, academicians, and practitioners. hopefully, everyone finds a home, though clear boundaries are established and not set in concrete. nevertheless, the reviewers are right in stating that academic programmes led the redefinition effort, and that they need the book more than practitioners. what is missing from this observation is that in today's world, practitioners need the theory disseminated by academic programmes in order to collaborate and be competitive, just as academic programmes need interaction with practice in order to make dynamic contributions. every year in the united states, a loosely organized group of academic professors in instructional technology meets at a lake near indiana university in bloomington. attendance is usually about 120. typically, practitioner representatives also attend this meeting, and there have been representatives from the international arena. the redefinition effort was successful because of support from this group (professors in instructional design and technology). it is not surprising that there is more interest among academic programmes than among practitioners. the definition does not change the tasks or functions of a practitioner. the impact is greater in academic programmes because redefinition can have a direct effect on (a) teaching and learning about instructional technology, (b) reporting and synthesizing research and theory, and (c) communicating across disciplines and areas. the domains and sub-categories can illuminate gaps in theory that need to be addressed and thus help build support for research agendas. other issues arose during the redefinition project such as whether to use the term instructional or educational technology, and whether the field is a discipline. instructional was chosen because in the usa the field of instructional technology is more active in corporate and government settings than in schools, and educational implies schools only. this term also allowed us to begin the definition with the word instruction and end it with the word learning, which many felt reflected the field's focus more than the word educational. nevertheless, we continue to be concerned about the different usage internationally, although we treat the terms as synonymous. a further discussion of these issues and the redefinition process is found in a case study we wrote for the 1994 educational media and technology yearbook. we would like to conclude by responding briefly to a this must be the test. can such a definition help the practitioner? this is the question that all three reviewers have been asking. i had hoped for a stronger case to be made for the expertise of instructional designers. the book does seem to avoid the evidence that particular instructional strategies can and do make for effective learning situations. do i want a theory of design or a theory of learning? i feel that there are too many learning theories to be of any practical use to courseware or learning designers. terms such as instructional technology can be seen as pejorative in the uk. the same is true of itinformation technology. i fail to see, in its many definitions, the effective use of 'information' when we use this term as being synonymous with educational technology in uk higher education. it was probably for this reason that the association for learning technology adopted the term learning technology. 70 alt-] volume 4 number 2 i am left a little disappointed. i had hoped that after being in the field for 25 years, i could begin to use the description of 'educational technologist'. in fact, i find the more relevant 'learning technologist' still difficult to sustain. i feel that the field should have progressed more. seels and richey have brought a scholarly treatment to selective elements. i would have liked them to address the integration of instructional theories. i feel that such a bottom-up approach would have been more useful than the top-down approach adopted by seels and richey. few specific points raised by the reviewers, starting with margaret bell's comments. we agree that a definition cannot and should not contain or control or even constrain the field. we believe, however, that it can improve communication. each aect definition has become part of the historical record of our field. one way in which the field is hampered by lack of currency in definition is in defending jobs and programmes. we also agree that many aspects of the definition need elaboration and discussion. the committee always intended to expand the debate internationally once a national consensus was achieved and opportunities were available. the book contains visual representations which emphasize important relationships among domains and between theory and practice. the authors take the position that a field of study cannot be separated from the profession or practice. the book relates practice to the knowledge bases of the field. finally, we address specific points made by derick unwin. the field has progressed beyond purely operational or practice-based definitions. these are fine but are not adequate to meet all the field's needs for definition. hopefully, one benefit of taxonomic theory presented in the book is the continued progress of a profession that provides support for learning. if nothing in teaching or learning has or will change due to instructional technology, then neither practice nor theory in the field are justifiable. this is patently not true as the demand for instructional technology and the performance of instructional technologists, in the usa at least, documents. references annett, a. (1964), editorial, programmed learning, 1 (1), 2-4. bretz, r. (1971), a taxonomy of communication media, englewood cliffs nj, educational technology publications. cummings, l.e. (1996), 'educational technology a faculty resistance review', educational technology review, 5 (winter), 18-20. duffy, t.m. and jonassen, d.h. (1991), 'constructivism: new implications for instructional technology', educational technology, 31 (5), 7-12. geis, g.l. (1986), "human performance technology: an overview' in smith, m.e. (ed.), introduction to performance technology, vol. 1, washington dc, national society for performance and instruction. heidt, e. (1978), 'media classification', in unwin, d. and mcaleese , r. (eds), encyclopaedia of educational media commechnology, london, macmillan. 71 reviews lumsdaine, a.a. and glaser, r. (eds) (i960;, teaching machines and programmed learning, washington dc, n.e.a. pressey s.l. (1926), 'a simple apparatus which gives tests and scores and teaches', school & society, 23, 373-6. unwin, d. (1969), media and methods: instructional technology in higher education, london, mcgraw hill. wilson, b. g. (1996), constructivist learning environments: case studies in instructional design, englewood cliffs nj, educational technology publications. judith jeffcoate, multimedia in practice technology and applications, bcs practitioner series, prentice-hall international, 1995. isbn: 0-13-123324-6. £24.95. ray welland concludes his editorial preface to this book with the sentence: 'if you want to find out what multimedia is and what it has to offer this is an excellent place to start'. the comment reflects the major strength of this book, which does provide an excellent starting place for anyone wanting to learn more about multimedia. the author explains what multimedia' is today and the technology involved, gives example applications, and projects current trends forward to explore the use of multimedia in the future. she clearly states the perceived audience for the text as 'business managers and project leaders who need to introduce multimedia technology into their applications'. a guide to using the text is provided: parts i and iii are suitable for readers with no technical knowledge; part ii requires familiarity with computer architecture, and part iv is most suitable for managers planning a multimedia project. these guidelines are useful, but the text as a whole will appeal to anyone who has an interest in learning about multimedia. technical knowledge is advantageous in some sections, but the author takes time to explain clearly, and uses examples to illustrate points. the book is very well structured. the author provides an excellent summary of the content in the preface. this clarity of structure continues throughout the text; for example, each chapter begins with a brief outline of its contents and concludes with a summary and references. multimedia is a subject engulfed by acronyms and, to prevent confusion, the author provides a detailed glossary which at times proves to be very useful. a bibliography and a full index are further resources which the reader can draw upon. part i, 'multimedia today', begins by painting a scenario of a modern family using new technology in their everyday lives, then projects the same family into the year 2001, highlighting the role technology may play in their lives in the future. part ii, 'technology', deals with the complex areas that make up multimedia, and is the most technical section of the book, dealing with different platforms, development tools, storage methods and the components that make up a truly multimedia application such as images, audio and video. the chapters focusing on the different media are divided into sections describing applications, capture, compression and standards, as well as detailing information specific to each medium. a history of the media is provided, enabling the reader to see, for example, .how standards have evolved. this technique of placing information within its historical context is used throughout the book, and is vital for a reader who may be coming to the topic for the first time. part iii returns to the theme with which the book began: 'applications'. it focuses on present use and future trends. again the author places the use of multimedia in its historical context: the early use of multimedia was largely for off-line training and education using interactive videodisk and standalone players. 72 book reviews online philip barker1 human-computer interaction laboratory, university of teesside as the number of new academic books published each year continues to rise, such that it becomes evermore difficult to keep abreast of them in one's discipline, the book-review procedure takes on an increasing importance. this paper outlines the design and development of an automated system for handling book reviews. descriptions are given of some prototypes that have been developed for use on an intranet server and/or the internet. these systems, based on sgml and html, are briefly discussed and compared introduction many thousands of books are published each year, and even specialists find it difficult to keep abreast of new books in their disciplines, learning technology being no exception indeed, in our subject-area the situation is beginning to reach saturation point. the bookreview procedure facilitates selection: a good review will capture the essential content of a book, and will comment on its quality, style, level of presentation, appropriateness, and perhaps value for money. figure 1 shows the review process, its relationship to the production of books and learned journals, and the functional similarity between book reviews and abstracts of papers published in learned journals. abstracts are often archived in online databases or on cd-rom, in this way acting as an alerting and brief reference service. book reviews can be treated in the same way. this paper discusses the use of servers (internet or intranet) as a means of making them available to a global population. it also describes how such a facility could fit into a more general infrastructure for soliciting potential reviewers and drawing their attention to publications available for review. analysing book reviews in order to describe a system and the relationships between its component parts, it is useful to employ some form of meta-language or meta-notation in which each of the components of the system is represented as a meta-variable (alexander, 1987). the relationship between the meta-variables that define the top-level moieties of the system is then expressed using a series of meta-rules. the purpose of a meta-rule is to define a given meta-variable in terms 54 alt-j volume 5 number 3 figure i:the context of book reviews i observation i data and information 1 i documentation | research papers | chapters [ 1 | learned journals] | abstracts v i | publication media | of more elementary units, or indeed the axiomatic entities of the system being described. suppose, for example, one wished to describe the logical structure of a book using this technique. the first step would be to identify the book's building blocks. the second step would involve formulating a series of meta-rules that specify the relationships between the basic components identified. the structure of a text-book could thus be formulated in the following way: <text-book> ::= <front-cover> <body> <back cover> <body> ::= <contents-pages> <chapters> <index-pages> <chapters> ::= ... <chapter> <chapter> ::= ... <page> <page> ::= ... <text-line> <text-line> ::= ... <word> <word> ::= object (x) : x isin dictionary (y) in this example, the meta-variables that identify the basic building blocks are each contained within angle brackets (for example, <body>, <chapter>). each line defines a meta-rule, and within meta-rules various operators show how one meta-variable relates to others. three primitive operators have been used: the composite symbol ::= means 'is defined to be'; the... operator means 'is a replication of one or more'; the: operator means ss philip barker . book reviews online 'such that'. logical predicates are also used. a predicate takes a proposition about an object or relationship and determines its truth or falsity. in the book definition given above, two predicates have been used: 'object' and 'isin'. the 'object' predicate checks that a candidate object, x, exists and that it is of the correct type (that is, a <word> object). the 'isin' predicate checks for membership of the instance x in the dictionary y. if x is not found in the dictionary, the rule fails and the object x is rejected as a valid instance of <word>. this meta-notation is similar to that defined by the british standards institution (bsi, 1981). the definition of <text-book> presented above is not complete: the meta-rule sequences that define the meta-variables <front-cover>, <back-cover>, <contents-pages> and <indexpages> have been omitted. in a complete definition, these objects would need to be defined in terms of lower-level meta-variables or terminal (axiomatic) entities. but despite its incompleteness, the definition presented is sufficient to illustrate the basic principles involved. expressed in plain english, the definition of <text-book> given above would read as follows: a text book is made up of a front cover, a back cover and an inside (or body). the body is made up of a collection of pages. the pages at the front of the book identify its contents, those at the back make up the index, and those in the middle are organized into a series of chapters. each of the pages within a chapter is made up of a sequence of lines containing words. all the words used in any given line must exist within a dictionary whose name is y. how can this system be applied to the definition of a book review, which contains not only an assessment of the content and worth of a book, but also important items of information (attributes), such as the names of the author(s)/editor(s), the publisher, the place of publication, the series of which the book may form a part, its isbn, its cost, the number of pages it contains, its format (hardback, paperback), and so on? the metanotation approach used for the definition of a text-book above can be used to express relationships between these attributes, as follows: <book-review> ::= <book-detailsxkeywordsxreviewbodyxreviewer-details> <book-details> ::=<titlexauthor-listxpublisherdetailsxother-details> <publisher-details> : := <publisherxplace-of-publication> <other-details> ::= <date><isbnxbook-typexsize><cost> <book-type> ::= 'hardback' | 'softback' <keywords> ::= ... <keyword> <keyword> ::= isaword (x) & x isin dictionary (y) <reviewer-body> ::= ... <paragraph> <paragraph> ::= ... <line> <line> ::= ... <word> <word> ::= isaword (x) & x isin dictionary (z) <reviewer-details> ::= <reviewer-namexreviewer-affiliation> 56 alt-j volume 5 number 3 the process of specifying the structural units of a book review using a meta-notation paves the way for tagging a document using a markup language such as sgml (standard generalized markup language) or html (hypertext markup language). these markup tools form the basis for numerous online retrieval systems that can be made available on servers. the review process in order to achieve the development of an automated system for handling book reviews, it is necessary to consider in more detail some of the ancillary processes relating to the review procedure. a schematic representation of some of the more important supporting actors and processes involved in a reviewing operation, and their inter-relationships, is depicted in figure 2. figure 2: principal actors and processes involved in book reviews publisher consumers i | editors | secondary alerting services requests 1 | publication | <• | review | most journals have a person responsible for handling review material the reviews editor who is usually chooses and acquires review copies of books. sometimes these copies arrive unsolicited from publishers. more often, review editors have to ask to be supplied with copies of books relevant to their journals. review editors can build up a buffer stock of books for review, and reviewers have to be found for these and matched to appropriate books. this process can be initiated, for example, by listing titles available for review in the reviews section of the journal in question, or by sending out a printed supplement as an insert to that journal. alternatively, various electronic alerting mechanisms can be used, such as an email facility linked to a listserver. this is the case with alt-j, where a database list of the books received from publishers for review is built up and emailed to potential 57 philip barker book reviews online reviewers whose names are held in a listserver. people interested in reviewing books on the list then reply (by email) using appropriate code words to identify the book(s) they want to review. invariably, many bids are emailed back, and a selection mechanism, based on the judgement of the reviews editor, is then used to allocate a title to a particular bidder. another electronic alerting/awareness facility we have been exploring is based on the use of the web. a list of books available for review is maintained on a web server which potential reviewers can access. details of bidders and the books they select are automatically emailed back to the reviews editor. the user-interface also enables reviewers to make suggestions about other books that should be included in the list, and to make offers to review new books they are aware of. the main problem with this web-based approach is that someone has to be responsible for maintaining the online list of books for review. we are investigating ways of automating the process. one advantage of using the web for electronically processing book reviews is that the reviews can also subsequently be made available online to all those interested. towards an online review system in order to explore the possibility of creating an online database of book reviews which could be accessed via web, we created three prototype systems, one developed using sgml, the other two using html. the structure for <book-review> (see above) formed the foundation for each of the online implementations. it served as a basis for identifying search keys, stimulating thinking about system organization and screen layout; and specifying the display styles to be employed. the other main purpose of the meta-linguistic representation of <book-review> was as a means of identifying the various elements and entities that were subsequently to be used in the formulation of the sgml document type definition (dtd). for example, meta-variables such as <title>, <author>, <reviewbody> and <paragraph> mapped directly across to corresponding sgml tags, namely </title>, </author>, and so on. the importance of the dtd lies in the fact that it forms the underlying basis for consistency-checking when marked-up instances of <bookreview> are admitted into an electronic library-cataloguing system. during the initial stages of the project, three perspectives were identified as being important to the design process. first, that of system administrators who would need to set up and maintain an instance of the system at any given site. second, that of book reviewers who might use it to gain indirect access to material for review purposes. third, that of readers who would access the reviews. the ways in which these different perspectives influenced the system design and the functionality embedded in the user-interface are discussed below. basic system design as a result of surveying various potential users from each of the three categories identified above, we found that typical users of the system would wish to be able to retrieve a review from the database, do a library search to see if a book of interest was available in a local library, request a book either for purchase or using an inter-library loan service, see a list of book titles waiting to be reviewed, and perform system-maintenance operations. obviously, the detailed system implementation instigated at any given site would depend on the overall context in which a given instance of the system was to be used and the 58 alt-j volume 5 number 3 population it was intended to serve. it was therefore imperative that the design and implementation should lead to a flexible approach. one of the most important design tasks was choosing the search mechanisms to be used for retrieving reviews. from a needs analysis, we found that typical users would most likely want to search for a review based on a specification of relevant keywords. in a review, these might occur in the title of the book, in the body of the review, or as part of its <keyword> section if one has been used (we do not use one for alt-j, although we may do in future). in addition, some users suggested that retrieval by isbn and/or author(s) might also be useful. implementation three implementations of our online book-reviews system were undertaken, referred to hereaspl,p2andp3. pi was implemented using an integrated sgml development and display system called iads (interactive authoring and display system) produced by the automated publications division of the integrated material management centre of the us army (templeton, 1996). the system consists of three facilities: (1) a visual interactive editing system for authoring sgml documents; (2) a style sheet for specifying the visual appearance of documents; (3) reading and navigation tools (hyperlinks). we used this system to create an electronic library of book reviews and, in so doing were able to assess the merits of sgml for this type of application. our findings are briefly discussed later in this paper. p2 was created using hotmetal pro (version 2), a low-cost html editing system. it was implemented on a notebook computer as a proof-of-principle experiment, the main purpose of the prototype being to explore the potential of html markup techniques for this type of application. it was also used as a demonstrator to show potential users the kinds of facility a full implementation of the system would make available, and it provided the opportunity to obtain some valuable feedback from users. the implementation used simple html hyperlinking techniques in order to create a three-level menu-driven structure. the top level contained an introductory page which gave basic instructions and listed the search options available, and which embedded hotlinks to three second-level pages that were responsible for handling retrieval by author, title and keyword. the hotlinks embedded in the second-level html modules pointed directly to entries at the third level the book reviews themselves (which were all marked up in html). this prototype had a number of inherent limitations (see below) and a third prototype, p3, was developed in order to overcome them. p3 also used html but with more sophisticated techniques involving the use of the common gateway interface (cgi) standard (gundavaram, 1996). the design approach underlying the creation of this prototype was driven by the need to realize four basic objectives: (1) to avoid any requirement for editorial users to have any knowledge of markup languages; (2) to provide an easy-to-use system; (3) to add a search engine; (4) to produce a system that was easy to maintain. a series of cgi scripts and perl (searchengine) programs were mounted on a unix-based intranet server. the suite of programs had to perform the following processing tasks: (1) extract a series of keywords entered (by the user) by means of an online html form running on a client computer; (2) search a textual database of book reviews (held on the server) for instances of the keywords (using _ philip barker book reviews online and or or logic); (3) build a list of reviews containing the keywords entered as search terms; (4) deliver the list of candidate reviews to the client computer for perusal by the user; (5) allow users to look at individual reviews in their hit-list. as a result of the formative evaluation that was undertaken using prototype p2, a number of other facilities were identified as possible additional requirements for the p3 system. these included the provision of an online help facility, making available mechanisms for searching the university's online book catalogue (to check for the availability of books), and the provision of facilities to enable the electronic submission of inter-library loans and book-purchasing. each of the five primary functions mentioned above that users might wish to perform was represented by a iconic button embedded in the user-interface. various levels and types of help were available, depending on the type of user requesting it (editor, administrator, reviewer or reader) and his/her level of experience. in order to control the basic search process, users could enter a list of keywords into a text-entry box. the interface controls also allowed users to set various search parameters (such as case sensitivity and search logic) by choosing options from pop-down selection boxes. once all the search values had been set, a submit button transmitted the query to the server for processing. the hit-list resulting from a search was subsequently shown in a dynamically created html page which acted as a menu providing access to the reviews associated with the entries in the current hit-list. because p3, like p2, was a prototype, its interface also contained a feedback button, allowing users the option of sending back comments. throughout this project we were conscious of the need for evaluation of what was being developed. different methods were employed in order to undertake both formative and summative assessments of the p3 system including the ease-of-use, look and feel of the user-interface, the responsiveness of the system, and users' attitudes and perceptions about the potential utility of book reviews as an online information retrieval facility. the feedback button generated some simple questions relating to ease of use, levels of satisfaction with the search results and search time, and general usefulness of the system. the responses were aggregated in order to produce a cumulative response file that could be used to create dynamic histograms representing the views of those users who chose to submit some feedback. details of the evaluation have been documented in two student dissertations (bouvier, 1997; tan, 1997). discussion the results obtained from the evaluation, and our own experience of all three prototypes, allowed us to see a number of their shortcomings. sgml is undoubtedly a very powerful markup facility, but in contrast to html, we did not find it at all easy to use. furthermore, we found that we could not transport the <book-review> dtd (associated with the iads system) to other sgml browsers such as softquad's panorama pro. we also experienced substantial difficulties during our attempts to set up the types of search facility we wanted to use. p2 was rejected because it was a very simple implementation that did not contain any search mechanism, and all reviews had to be retrieved by menu selection. it was also awkward to maintain, and would undoubtedly have become cumbersome to use when a substantial number of book reviews were made available. 60 aa-] volume 5 number 3 there were three main shortcomings associated with prototype p3. first, the book reviews held in the online database were stored as plain text files, so lacked any attractive layout attributes. second, the underlying search engine that was used to retrieve reviews had to search all the files sequentially in the database for occurrences of the keywords associated with a query. we did not find this prohibitive in terms of search time, but with a very large database it is an issue which we may need to address. third, the system made extensive use of server-side processing involving a considerable number of data transfers between the server and the clients it serviced: the response time to queries could be subject to delays caused by other traffic on the network. although the use of a proprietary database system (such as oracle or dbase) might have been a way to proceed, we decided against the use of such packages, primarily because we wanted to generate implementations that were very easy to operate and install without being dependent on the availability of commercial software. future possibilities as we gain more experience with our online book reviews system and expose it to a greater number of potential users, so we begin to perceive more ways in which it could be used. we also begin to realize some of its current limitations, and foresee the need to review its design and implementation. it is anticipated that as larger numbers of book reviews need to be handled, we may have to redesign the search engine. a related issue will be the provision of mechanisms to enable users to handle large hit-lists. it is possible that this could be achieved by allowing users to specify how many hits they want to see, and/or by providing a range of selection algorithms (from which users can choose) that can be used to determine membership of the hit-list. for example, a user might wish to see only five reviews and that the hit-list should be based (say) on selecting those reviews which have been most often referenced by other users. an important overall future development of our system will be its integration into both the new digital library infrastructure at teesside, and our existing conventional library system. with this in mind, we intend to make three ancillary support services available: (1) mechanisms to search online catalogues in order to assess the availability of books; (2) facilities for the electronic submission of inter-library loans; and (3) methods to enable users to submit electronic library-purchase orders. other future development work we hope to undertake will include a re-assessment of the potential of sgml in environments other than that provided by iads, and an investigation of the potential role of java for the implementation of applets that will embed some of the system functionality described above. conclusion because of their importance as an information storage, dissemination and retrieval mechanism, it is all but unthinkable that books will become obsolete in the foreseeable future, even though, with the advent of low-cost electronic publishing mechanisms (particularly the internet), it is likely that some changes in their form, appearance and 61 philip barker book reviews online properties will take place (electronic books, for example, are now starting to become more widely available). given the richness of books as an information/knowledge resource, effective mechanisms are needed in order to retrieve details of them. although much more evaluation work remains to be undertaken, the initial studies outlined in this paper suggest that an online book-reviews system could form an effective mechanism for handling bookbased material. it would of course also be feasible to apply the concept to other types of publication such as sound recordings, films, videos, and multimedia productions. note 1. professor philip barker is alt-j reviews editor. references alexander, h. (1987), formally-based tools and techniques for human-computer dialogues, chichester: ellis horwood. bouvier, e. (1997), book reviews online, final-year bsc computer science dissertation, university of teesside. bsi (1981), bs-6154:1981 method of defining syntactic meta-language, bracknell, berks: british standards institution, technical standards ltd. gundavaram, s. (1996), cgi programming on the world wide web, sebastopol ca, usa: o'reilly. tan, c.m. (1997), hypermedia electronic books, draft phd thesis, university of teesside. templeton, r.s. (1996), 'the interactive authoring and display system (iads)', downloaded from http://shodan.redstone.army.mil/index.htm. 62 calt586676 143..154 educators’ perceptions, attitudes and practices: blended learning in business and management education vladlena bensona, deborah andersona and ann oomsb* akingston business school, kingston university, kingston upon thames, uk; bfaculty of health and social care sciences, kingston university and st george’s, university of london, london, uk (received 7 october 2010; final version received 28 april 2011) adoption of blended learning is a complex process for higher education institutions and academic staff. although the move towards blended learning is generally instigated at institutional level, factors determining its success and minimising resistance of faculty often emerge at delivery level. this paper explores adoption of blended learning practices in a business school at a university in the united kingdom. based on the interviews with a purposive sample of 16 academic staff members delivering 36 business modules, this case study explores the concept of blended learning from the academic staff’s perspective. a typology of three distinct approaches to blended learning – ‘technology is all’, ‘bolt-ons’ and ‘purely pedagogic’ – emerged from the data extending understanding of blended learning practices. a team of three researchers conducted the study. two of the researchers were academic staff members from the business school and one researcher was an academic staff member from another faculty within the same university. keywords: blended learning; business education; higher education; educational technology introduction the notion of blending various teaching methods to achieve an effective learning experience has been a subject of past and present exploration by academics. adding creative and innovative uses of technology to improve teaching practices have generated new opportunities for learning (for example, clark 2003). blended learning is envisaged to maximise the benefits of traditional teaching methods and online delivery. vaughan (2007) contests this assumption and stresses significant difficulties and risks associated with technology uses in learning and teaching. the problems that arise in developing blended learning modules include insufficient support, lack of time and resources for course development, risks associated with availability of technology and the necessity of acquiring new teaching and technology skills. evidence of challenges impeding the adoption of technology by instructors (derntl and motschnig-pitrik 2005) motivated the researchers to investigate approaches of blended learning at the module level (one module is equivalent to 15 or 30 credits) and explore the concept of blended learning from the perspective of academic staff at the business school at one post-1992 university in the united kingdom. *corresponding author. email: a.ooms@sgul.kingston.ac.uk research in learning technologyaquatic insects vol. 19, no. 2, july 2011, 143–154 issn 2156-7069 print/issn 2156-7077 online � 2011 association for learning technology doi: 10.1080/21567069.2011.586676 http://www.informaworld.com the definition of blended learning provided by williams (2002) as the combination of face-to-face and online learning has been contested by some researchers as insufficient because it involves ‘bolting on’ technology into traditional course as an add-on or extra content and ignores the need of rethinking the course design or pedagogy (bleed 2001; vaughan 2007). delivery of blended learning courses involves a move of a significant portion of the learning activities online. a blend of traditional methods (i.e. face-to-face classroom teaching), with technology-based instruction, including online communication, activities and delivery, entails significant changes to the course delivery; for example, leading to shorter ‘seat-time’ (garrison, kanuka, and howes 2002). these courses are distinct from ‘distance learning’ courses offered online or at a distance, as emphasised by dziuban, hartman, and moskal (2004). blended learning is significantly different to courses delivered in a traditional face-to-face classroom with a supplemental website. it is not implemented by a simple transfer of teaching materials online, but rather involves redesign of delivery and pedagogy (vaughan 2007). taking into consideration the arguments above, the definition of blended learning as a significant “integration of online and traditional face-to-face class activities” (alebaikan and troudi 2010, 50) is sufficient for the scope of this paper. this study aims to investigate perceptions, attitudes and practices related to blended learning in higher education and to identify factors that facilitate or impede adoption and implementation of blended learning at the delivery level at the business school at one university based in london, uk. the following research questions are addressed: (a) what are academic staff perceptions of blended learning? (b) what are academic staff attitudes towards blended learning? (c) what are academic staff current practices of blended learning? by exploring the staff perspective on blended learning, this case study can inform those managing, planning and taking initiatives in professional development that includes blended delivery. blended learning: benefits and risks persistent attention from researchers to blending technology with traditional delivery is related to practitioner beliefs that technology brings a greater level of effectiveness into learning and teaching. technology has had a significant influence on the ways students interact with their peers, faculty and transformed learning and teaching inside and outside of the traditional classroom. research suggests that information and communication technology can help remove geographical and situational learning barriers, offer better opportunities for learner and instructor interaction and lead to raising the quality of learning experiences (bates 2000; garrison and anderson 2003). a mix of different pedagogies and teaching strategies has been regarded as good practice for many years. lectures are no longer the standard, and teaching involves more classroom interaction, case studies, student group work and presentation, simulations and other types of learning activities (williams 2002). blended learning offers an effective platform for employing different pedagogical strategies and has 144 v. benson et al. the potential to maximise the advantages of both face-to-face and online learning (wu, tennyson, and hsia 2010). the studies conducted at the university of tennessee and stanford university show that blended learning can improve learning outcomes. these findings were confirmed by a study undertaken at two higher education institutions in the united kingdom (boyle et al. 2003). also in the united kingdom, a review of uk literature and practice commissioned by the higher education academy concluded that overall students are very positive about blended learning (sharpe et al. 2006). in addition, blended learning produced a stronger sense of community among students than traditional face-to-face or online learning (rovai and jordan 2004). students’ interaction and overall satisfaction have been shown to improve through blended learning (delacey and leonard 2002). research conducted by garrison and kanuka concludes that “blended learning is consistent with the values of traditional higher education institutions and has the proven potential to enhance both the effectiveness and efficiency of meaningful learning experiences” (2004, 5). finally, the role of instructor transforms from teacher to facilitator, as they take charge of learning the new technology, adapting the technology to pedagogy or even forging new pedagogical principles based on innovations in technology, as discussed in charles and anthony (2007). on the other hand, there is evidence of staff reluctance in adopting technology to support/replace faceto-face teaching (ooms et al. 2008). whether this reluctance is a result of disbelief in technology, lack of supporting resources or perception of lower quality, it can certainly influence the implementation of a blended learning strategy. academic staff encounter the challenge of how to effectively integrate technology in their teaching practices (arbaugh 2008). faced with a range of diverse tools including multimedia, mobile technologies, web 2.0 resources and services, and social networks, academic staff need to make informed decisions on how to use these technologies in their teaching. a willingness to accept new technologies could significantly affect the success of blended learning development. charles and anthony (2007) report that core faculty perceive the development of online-based activities as time-consuming and are more likely to shy away from technology-facilitated interactions. a preparation time of at least six months is suggested for blended learning integration with a great impetus on encouraging academics to apply blended learning pedagogy and exchange good practices from those experienced in the effective use of blended learning (charles and anthony 2007). as a result, institutions need to identify ‘early adopters’ and ‘laggards’ in the innovation process (rogers 1995) and develop strategies as to how blended learning may address their requirements in the changing educational landscape. blended learning developments take time, and the amount of work involved, even when given support by e-developers, can be underestimated by those staff who are new to blended learning (ooms et al. 2008). research indicates that the insufficient time due to contextual factors such as the structure of timetables and workload, and a lack of proper training can cause delays and ineffective use of technology in education (cuban, kirkpatrick, and peck 2001; pajo 2001). a combination of technological and pedagogic training is desirable (hannon 2008; oliver et al. 2004; higher education funding council for england. 2005), and academic staff development becomes most successful when supported by a range of strategies (bates 2000). lacking in the current literature are studies about educators’ perceptions of blended learning, their attitudes towards blended learning and their current blended research in learning technology 145 learning practices. faculty scepticism and misunderstandings about what blended learning includes are not uncommon and are factors that could hinder blended learning developments (ooms et al. 2008). this study aims to fill this gap in the literature by investigating perceptions, attitudes and practices of blended learning of academic staff in business higher education. research method a case-study methodology was used to investigate academic staff perceptions and attitudes towards blended learning and current blended learning practices. data were collected through semi-structured interviews with 16 academic staff. the focus of the qualitative interviews was to explore the practices of blended learning implementation at a business school in a uk university by unfolding the meaning of blended learning experiences from the staff perspective. in addition, secondary data, such as the faculty and university strategies on quality enhancement and blended learning were analysed, providing insight into the institutional context. the interviewing investigation comprised seven stages: thematising, designing, interviewing, transcribing, analysing, verifying, and reporting (kvale 1996). the first two stages were informed by the theoretical framework and secondary data providing an insight into the institutional context. three pilot interviews were conducted initially and were followed by some adjustments to the interview schedule. the interviewing method stage involved semi-structured interviews with academic staff from schools of business and law from a wide range of departments and subject areas. purposive sampling was used to sample 16 academic staff members from postgraduate and undergraduate programmes, in order to obtain a representative sample of class sizes; that is, from small (25 students and under) to large (100 students and over). a subset of 16 interviews of those academic staff engaged in undergraduate programmes and modules offered on the mba, where blended learning strategy was in the first year of its implementation, were selected. this subset of interviewed academic staff covered 36 modules. interviews were conducted at the university. the interviews were digitally recorded, transcribed, and then checked with interviewees for accuracy. primary analysis was conducted by focusing on specific questions relating to innovation, perceptions of blended learning and attitudes to technology. findings are reported by drawing upon (anonymised) quotations. the next stage of analysis developed a categorisation of the different perceptions leading to a range of approaches to blended learning, and involved identifying similarities and differences amongst participants. whilst this approach does not enable the research to make any claims about objectivity and generalisability, it does allow a detailed, in-depth understanding of the current perceptions, attitudes and practice relating to blended learning at this institution. findings what are academic staff perceptions of blended learning? analysis of the interview transcripts suggested that there is almost universal support for offering a range of learning resources to students. a perception that young 146 v. benson et al. people (students) like “to break things up” (participant 2) appears to have led to academic staff considering a variety of tools and techniques in the delivery of their modules. some participants saw this as “blended learning” that may or may not be information and communications technologies (ict) based: yes, i mean my blended learning does not necessarily have to do with ict, but i do use a variety of techniques in a sense; it’s partly group discussion, partly it’s lecture, partly it’s individual work, so it is blended although the blended is not mainly based [. . .] on ict. (participant 15) this perception of blended learning as more than just technology was a common feature of several interviews; upon answering a question on their understanding of blended learning: that blended learning is everything thrown in, not just information technology but it’s individual work, it’s group work, it’s quizzes, it’s questionnaires, it’s reading, it’s case studies, it’s role play . . . independent learning, you know, all of it. (participant 16) whilst most academic staff interviewed understood the inclusion of some technology implied in definitions of blended learning, for many of them blended learning was much wider than simply amalgamating a number of electronic and web-based tools. a number commented on the fact that they had always approached their teaching by considering variety in delivery. conclusions of the study confirm earlier findings (oliver and trigwell 2005) that blended learning is not understood uniformly by academic staff, and a mixture of its definitions exist in this case study. what are academic staff attitudes towards blended learning? despite concerns, participants demonstrated an appreciation of many benefits of an ict-based blended learning approach. for participant 16, blended learning: makes it more interesting. for the students [. . .] it also covers all learning styles because we don’t all learn in one particular way . . . and it gives different angles to things as well, it helps you see it from the other side. similarly, participant 14 commented: [. . .] i think one of the clear benefits of thinking of other ways of making the module more relevant, [is] more engaging, more motivating for the students. one theme that did emerge from a number of interviews was the idea that sophisticated technology could not be a substitute for poor teaching. the importance of execution was identified, for example, by participant 4: so we all set up discussion boards, but some people can use them very well. we all use case studies and presentations and some of us do them better . . . i think it’s more about the execution, not the novelty. i would look at blended learning more in terms of learning than support . . . you know, face-to-face learning and teaching rather than as a replacement for it. (participant 5) research in learning technology 147 results of the study demonstrate a prevailing perception of blended learning as employing a variety of tools and techniques in module delivery. for some participants, technology is simply a way of organising and managing information for large student numbers. according to the study, blended learning is not always perceived as a positive thing. whereas reports of successful examples of blended learning prevail, many participants find preparation time-consuming and supporting technology prone to failure. the majority of participants are more inclined to use ict in the areas where they can see it delivering distinct benefits, such as delivering efficiency in classroom communication and convenience to students. this research revealed that more technology-apt participants are more likely to experiment with technology and are less wary of the learning curve or potential failure. whilst it was rare for staff to be totally opposed to using technology, levels of experience varied greatly across the participants. at the one extreme, one participant had designed modules that ‘integrated’ computer-based and face-to-face techniques. participant 13 perceived blended learning as more than a hybrid approach in which two techniques are brought together without thinking about how they work together. for this participant, the use of the term ‘blended’ means: that it highlights the need to integrate them and to therefore think about the best way to actually teach this aspect of the module. and how i can bring together the face-toface and on-line aspects. (participant 13) in contrast to this, a singular view of purely technology focused approach emerged when participant 9 was asked about limitations of blended learning: none because you have to use your imagination to adapt blended learning methods to the material you are trying to teach. in this case, teaching methods and technology were used interchangeably – suggesting that pedagogy was overshadowed by a personal interest in technology. findings suggested that any barriers to developing blended learning modules were not, in the main, related to attitudes; in fact, virtually all participants claimed that they would collaborate with an e-developer. barriers appeared to be related to the perception of developing the ict-based aspects of blended learning as time-consuming: just initial set-up time really, rather than having to prepare a lecture and stand there and talk at the students for a couple of hours, obviously it takes more time. (participant 16) additional reasons for not incorporating more ict-based innovation into their delivery related to a lack of confidence that the technology would work: sometimes i use videos; sometimes it would be nice to just have a two minute clip . . . it just sums up exactly what you have said in a real context, and if i could manage that, or be confident that it would work, that would be the sort of thing, yes. (participant 8) 148 v. benson et al. despite concerns described above, participants demonstrated an appreciation of many benefits of a technology-enabled blended learning approach as providing exciting teaching opportunities as well as catering to a variety of learning styles. what are academic staff current practices of blended learning? implicit in the development of blended learning is the notion of innovation. as part of this research, participants were asked to describe any innovations they had implemented in their teaching. some were modest in their claims; participant 8, for example, began answering the question by stating “i don’t think i use anything innovative” but was one of the first to use electronic sign-up to assign students to tutorial groups, yet this was not even mentioned in the interview, suggesting that participant 8 did not perceive this as innovative, despite its clear benefits in managing student learning. others saw a lack of face-to-face contact time as a barrier to innovation in the classroom. participant 1, for example, reduced the number of student presentations as teaching time had become “too compressed”. some of the participants also noted how difficult it is to simply add tools such as a moderated online discussion board to a module and call it blended learning. for participant 14 this was a real situation that had left him feeling very negative. it also led him to suggesting that to successfully incorporate blended learning into a module does require a lot of administrative assistance. yet another barrier was noted by participant 6, who felt that use of learning management systems could make students lazy, in particular if the tutor was providing sources rather than “letting the student investigate for himself” (participant 6). in addition, it was possible to identify a group of participants who, whilst supporting a wide range of learning resources and approaches to teaching, saw technology as simply a way of organising and managing information for large student numbers. this group was also characterised by a common desire for more face-toface contact time with students (participants 1, 3 and 8). they saw technology as an important tool for clear communication: yes, i do try and make sure it is planned so they know what’s going on; they always love to know, obviously there’s a plan: the module handbook, what they do in each week, lectures and tutorials [. . .] that’s in the handbook and then, well the detail comes out on blackboard so they have always got information. and that . . . is important [. . .] because if you have got large numbers and they haven’t got good information, you are going to get 150 people asking you, which is a nightmare. so i think the module handbook and use of blackboard as a communication tool is vital in terms of handling the large groups. (participant 8) a dominant view of perceiving blended learning as a mixture of teaching practices has become evident. the application of a variety of pedagogical approaches in faceto-face teaching was used mainly to capture and retain learner focus and increase effectiveness of learning. levels of experience varied greatly across the participants, it was rare to find views that were totally against using technology. student-centred views have been encouragingly popular; however, face-to-face teaching was still seen as a more effective way of teaching than technology based. research in learning technology 149 discussion an issue emerging from both our research and other studies (for example, bonk and charles 2006) is the perception of developing blended learning materials as time-consuming. as the perception is not contested, and in fact has been acknowledged to reflect reality, it would seem sensible to adopt a strategy that attempts to realise the long-term benefits of investment in developing a blended approach. whilst benefits might include more efficient management and organisation of modules, they should also include distinct benefits in the form of improved student learning and enhancement of teaching quality. adoption of technology by academic staff may be influenced by a variety of factors. arguably, most academic staff seek some form of positive impact from embracing technology supporting learning and teaching. some try to achieve differentiation (daniel 1998), others associate quality enhancement with blended learning (delacey and leonard 2002). undoubtedly, adoption of technology has been associated with improving efficiency (porter 1985). in the context of this study the impact of technology on effectiveness of class communication, access to learning resources and general management of students has been widely reported. whilst not all academic staff are adventurous enough to embrace blended learning to its full potential, some see the positive impact of technology on making the most of their face-to-face teaching and utilising a wealth of resources in class or online. evidence from the study suggests that most academic staff adopted the use of multimedia to bring interactivity and interest into the classroom. a number of academic staff integrated group work and peer-learning with the use of discussion forums. the study revealed a significant range of technology tools successfully used by academic staff who witnessed a positive impact of blended learning on student performance. ultimately, the impact that blended learning has on students and module effectiveness appears to influence perceptions of blended learning by academic staff. finally, the choice of whether to adopt blended learning or not largely depends on the available resources and staff awareness about them. although many supporting resources, such as a learning management system, automated assessment tools, wireless networks and devices are available, not all academic staff choose to take advantage of them. others are overwhelmed by the variety of resources. for instance, one academic staff member deems it too time-consuming to venture out looking for new video case studies as the old non-interactive materials work just as well. in any case, availability of resources, including time, administrative and developmental support and technology, appears to define the way tutors adopt or shy away from blended learning. towards a typology of blended learning practices in order to understand the context of academic staff perceptions and attitudes influencing adoption of blended learning, it was important to establish the drivers of their teaching practices. key dimensions of technology and pedagogy, affecting adoption of blended learning strategies by academics, were linked to create twodimensional typologies (ritchie and spencer 2002) of approaches to blended learning. technological drivers impacting on academics’ approaches to teaching ranged from proven resources or technology that have been used for a significant period of time and are familiar to the majority of users through to technology so novel that 150 v. benson et al. the enthusiasm about its novelty may overshadow its lack of reliability or usefulness in the context of use. pedagogical drivers were viewed as ad hoc at one extreme, where methods of delivery were less thought through, through to a more strategic approach, where the choice of teaching methods was driven by the desire to achieve learning goals. analysis of the findings revealed the existence of three distinct types amongst the participants. based on their views of pedagogy and technology, as well as resources and perceptions of blended learning impact, it was possible to cluster participants according to their choice of drivers and expectations from blended learning. the following distinct approaches have emerged from our analysis: ‘technology is all’, ‘the bolt-ons’ and ‘purely pedagogic’. the vignettes below summarise the characteristics of each type: ‘technology is all’ this group was a small cluster of technology-driven academic staff who embraced each new tool with enthusiasm, striving to include them in their existing practice. they saw no limit to the possibilities of technology and, whilst acknowledging the importance of student learning, they were driven by the new technology rather than the strategic impact on student learning. ‘bolt-ons’ this group tended to use technology as a ‘bolt-on’ to improve their teaching (by more effectively managing student groups) or the students’ learning (by using technology to add variety to their delivery). they certainly did not regard technology as a substitute for teaching. indeed, for this group, the pedagogy was deemed more important than the need for technology. this group would typically draw upon examples of non-technological innovation in their teaching (participative workshops, guest speakers) to stress that they saw themselves as using a blended approach despite their reluctance to invest time in learning how to use technological tools. this was the largest group within the participants. ‘purely pedagogic’ a very small group within the participants appeared to start by thinking about pedagogy, then developing materials (both in-class and out of class) to ensure effective student learning. sometimes the solutions would involve sophisticated technology, other times innovations would be classroom based. this group did not express any concerns about embracing technology and learning about new tools, but, unlike the ‘technology is all’ group, would select their techniques on the basis of pedagogy and student learning rather than technology. whilst this paper does not seek to make judgements about the different typologies, one of the decisions that the institution must face is whether one type is preferable to another. is there a need to encourage staff towards the ‘purely pedagogic’ type and if so what are the implications for staff development? currently, at the research site, there are numerous workshops where academic staff can learn how to make podcasts, how to set up social networks, how to use electronic assessment, research in learning technology 151 and so forth. arguably this can provide the ‘bolt-ons’ with more tools, but does it encourage consideration of the underlying pedagogy? conclusions what are academic staff perceptions of blended learning? whilst most academic staff interviewed understood the inclusion of some technology implied in definitions of blended learning, for many of them blended learning was much wider than simply amalgamating a number of electronic and web-based tools and resources. a number commented on the fact that they had always approached their teaching by considering variety in delivery. results of the study demonstrate a prevailing perception of blended learning as employing a variety of tools and techniques in module delivery. some academic staff see technology as simply a way of organising and managing information for large student numbers. according to our study, blended learning is not always perceived as a positive thing. although reports of successful examples of blended learning prevail, many academic staff find the preparation stage time-consuming and supporting technology prone to failure. staff are more inclined to use ict in the areas where they can see that it delivers distinct benefits. such practices aimed at delivering efficiency in classroom communication and convenience of access to information for students. this research revealed that academic staff more adept with technology are more likely to experiment with technology and are less wary of the learning curve or potential failure. a rather extreme view of pedagogy as technology alone motivated some academic staff into experimenting with technology without appropriate pedagogical underpinning. what are academic staff attitudes towards blended learning? findings suggested that barriers to developing blended learning modules were unrelated to attitudes; in fact, almost all participants agreed that they would collaborate with an e-developer. although levels of experience varied greatly across the academic staff, it was rare to find views that were totally against using technology. barriers appeared to be related to the perception of developing the technology-based aspects of blended learning as time-consuming, including difficulties in locating relevant resources as so many are now available on the web. despite these concerns, many participants demonstrated an appreciation of the benefits of a technologyenabled blended learning approach in providing exciting teaching opportunities as well as catering to a variety of learning styles. what are academic staff current practices of blended learning? a dominant view of perceiving blended learning as a mixture of teaching practices has become evident. the application of a variety of pedagogical approaches in face-to-face teaching was used mainly to capture and retain learner focus and increase effectiveness of learning. beyond the classroom settings, discussion boards and podcasts, video materials and provision of resource access through the learning management system were among most widely reported tools that enabled casestudy-based teaching, action learning and facilitating group work. a significant 152 v. benson et al. number of participants voiced the view for more complex and engaging collaborative resources, such as games and simulations. there was a wish to extend the range of pedagogical and technological approaches if barriers could be overcome. recommendations from the academic staff perspective, blending technology and traditional teaching methods represents a significant challenge (vaughan 2007). current teaching practices involve substantial classroom interaction, case studies, student group work and presentation, simulations, and so on. blended learning as a seamless mixture of face-to-face and technology-facilitated methods of delivery appears to be well suited to address the needs of academic staff teaching varying numbers of students, in different classroom settings, addressing a range of learning styles and cultural differences. the study evaluated the current state of teaching practices and helped shed some light on the progression of implementation of a blended learning strategy at the business school. the study findings confirm previous research on staff perceptions that blended learning is time-consuming but that staff are inclined to use technology where they can see the benefits. hence any strategy implementation needs to address staff concerns and identify innovations likely to be beneficial in the specific context. factors affecting the integration of blended learning stemmed from the attitudes and perceptions of academic staff towards new teaching practices and technologies. the perceived impact, whether positive (e.g. increased efficiency of the communication process with students) or negative (e.g. extra administrative effort and time investment in content development), appeared to have the driving effect on blended learning adoption. availability of resources such as time, technology and support of e-developers also surfaced as determinants of positive attitudes towards adopting new teaching practices. it is conceivable that a study at other higher education institutions may yield different results; however, this case study certainly provides a useful insight into the factors facilitating or impeding implementation of the blended learning strategy from the academic staff perspectives. the challenge for institutions implementing blended learning strategies is to build on positive staff attitudes, provide appropriate resources and help staff develop blends of teaching practices and use of technologies that deliver the best outcomes in particular teaching and learning contexts. references alebaikan, r., and s. troudi. 2010. blended learning in saudi universities: challenges and perspectives. alt-j, research in learning technology 18, no. 1: 49–59. arbaugh, j.b. 2008. introduction: blended learning: research and practice. academy of management learning & education 7, no. 1: 130–1. bates, a.w. 2000. managing technological change. strategies for college and university leaders. san francisco: jossey-bass. bleed, r. 2001. a hybrid campus for a new millennium. educause review 36, no. 1: 16–24. bonk, c.j., and g. charles. 2006. the handbook of blended learning: global perspectives, local designs. san francisco: john wiley and sons. boyle, t., c. bradley, p. chalk, r. jones, and p. pickard. 2003. using blended learning to improve student success rates in learning to program. journal of educational media 28, no. 2–3: 165–78. research in learning technology 153 charles, d., and p. anthony. 2007. blended learning: research perspectives. needham, ma: sloan center for online education. clark, d., 2003. blend it like beckham. epic group plc. http://www.epic.co.uk/content/resources/white_papers/blended. cuban, l., h. kirkpatrick, and c. peck. 2001. high access and low use of technologies in high school classrooms: explaining an apparent paradox. american educational research journal 38, no. 4: 813–34. daniel, j. 1998. mega-universities and knowledge media: technology strategies for higher education. london: kogan page. delacey, b., and d. leonard. 2002. case study on technology and distance in education at the harvard business school. educational technology and society 5, no. 2: 13–28. derntl, m., and r. motschnig-pitrik. 2005. the role of structure, patterns, and people in blended learning. the internet and higher education 8, no. 2: 111–30. dziuban, c., j. hartman, and p. moskal. 2004. blended learning. educause center for applied research bulletin 7: 1–12. garrison, d.r., and t. anderson. 2003. e-learning in the 21st century: a framework for research and practice. london: routledgefalmer. garrison, d.r., and h. kanuka. 2004. blended learning: uncovering its transformative potential in higher education. the internet and higher education 7, no. 2: 95–105. garrison, d.r., h. kanuka, and d. hawes. 2002. blended learning: archetypes for more effective undergraduate learning experiences. report. university of calgary, learning commons. hannon, j. 2008. doing staff development: practices, dilemmas and technologies. australasian journal of educational technology 24, no. 1: 15–29. higher education funding council for england. 2005. hefce strategy for e-learning. statements of policy march 2005/12. http://www.hefce.ac.uk/pubs/hefce/2005/05. kvale, s. 1996. interviews: an introduction to qualitative research interviewing. london: sage. oliver, m., r. sharpe, j. duggleby, d. jennings, and d. kay. 2004. accrediting learning technologists: a review of the literature, schemes and programmes. alt accreditation project report 1. oxford: association for learning technology. oliver, m., and k. trigwell. 2005. can ‘blended learning’ be redeemed? e-learning 2, no. 1: 17–26. ooms, a., l. burke, t. linsey, and c. heaton-shrestha. 2008. introducing e-developers to support a university’s blended learning developments. alt-j, research in learning technology 16, no. 2: 111–22. pajo, k. 2001. barriers to the uptake of web-based technology by university teachers. journal of distance education 16, no. 1: 70–84. porter, m.e. 1985. competitive advantage: creating and sustaining superior performance. new york: the free press. ritchie, j., and l. spencer. 2002. qualitative data analysis for applied policy research. in qualitative researcher’s companion, eds. m. huberman, and m. miles. thousand oaks, ca: sage publications. rogers, e. 1995. diffusion of innovations. new york: the free press. rovai, a.p., and h.m. jordan. 2004. blended learning and sense of community: a comparative analysis with traditional and fully online graduate courses. the international review of research in open and distance learning 5, no. 2: 1–13. sharpe, r., g. benfield, g. roberts, and r. francis. 2006. the undergraduate experience of blended e-learning: a review of uk literature and practice. higher education academy. http://www-new2.heacademy.ac.uk/assets/york/documents/ourwork/research/literature_ reviews/blended_elearning_full_review.pdf. vaughan, n. 2007 perspectives on blended learning in higher education. international journal of e-learning 1: 81–94. http://www.thefreelibrary.com/perspectivesonblendedlearninginhighereducation.-a0159594390 (accessed september 28, 2010). williams, c. 2002. learning on-line: a review of recent literature in a rapidly expanding field. journal of further and higher education 26, no. 3: 263–72. wu, j.h., r.d. tennyson, and t.l. hsia. 2010. a study of student satisfaction in a blended e-learning system environment. computers & education 55: 155–64. 154 v. benson et al. alt-j, research in learning technology vol. 17, no. 2, july 2009, 73–74 issn 0968-7769 print/issn 1741-1629 online © 2009 association for learning technology doi: 10.1080/09687760903037109 http://www.informaworld.com editorial the agency of students, teachers and learning technologists taylor and franciscalt_a_403883.sgm10.1080/09687760903037109alt-j research in learning technology0968-7769 (print)/1741-1629 (online)editorial2009association for learning technology1720000002009francesbellf.bell@salford.ac.uk the articles in this issue reveal the commitment of alt-j to publish research that addresses the perspectives of a range of stakeholders in learning technology: in this case, students, teachers and learning technologists themselves. although the articles are wide-ranging in their scope and approaches, we can detect some important and common themes. the first of these is agency: learning technology research can fall into the trap of focusing on the technology that is offered rather than what people do with or without it. the second is online discussion: although computer conferencing has been used in education since the 1980s, student engagement remains a thorny issue for practice, and one that demands additional research. thirdly, the role of learning technologists remains in question: how can they support staff and students through their practice and research? czerniewicz, williams and brown’s article has been developed from the alt-c 2008 prize-winning paper on student agency: a study of their access (social, technical, practical and contextual) to information and communication technologies (icts) in three different universities in south africa. archer’s agency theory is used to explore from the student perspective the relation between agency and structure. their findings indicate the reflexivity of students’ responses to constraints they may encounter: for example, the use of cell phones by students in low socio-economic groups. the second article by skinner uses a theory from beyond the discipline of learning technology, namely community engagement theory, to investigate participation in online discussion in a level one undergraduate module. she concludes that engagement of all students from the start is necessary to build a strong sense of community. online discussion is also the focus of the next article, where enriquez critiques existing research frameworks for online discussion that rely on content analysis. she proposes the inclusion of genre analysis as a means of capturing the situated and emergent nature of online discussion as a communicative practice. these ideas are relevant to practitioners and researchers in learning technology. peacock et al. explore the changing roles of learning technologists, not only as researchers or practitioners in learning technology, but also as members of multidisciplinary research teams. the study explores whether such interventions can help to address any polarisation between practitioner and research roles for learning technologists. the bridge between research and practice is also a theme in the next article by schoonenboom, sligte and kliphuis. they report on the derivation and testing of guidelines for supporting re-use of digital learning materials: both from the literature and from practice case studies. their study not only validates the guidelines but also makes them more usable by readers. finally, middleton’s fascinating study of a podcasting pilot in higher education captures rich responses of staff and students. the findings capture the real possibilities 74 editorial of using audio and the contrast between reality and expectations; for example, audio is used to ‘soften’ the vle, rather than to feed audio (using the rss standard) directly to students. the work reported in this issue indicates the growing confidence of learning technology researchers, a willingness to reach beyond traditional boundaries. we can see this both in the grounded, people-centred approaches adopted, and in the inclusion of theories and methods from elsewhere: archer’s agency theory, community engagement theory and genre analysis. technology can offer new opportunities for learning, but only if practitioners, students and teachers can engage with it and understand its use in context. research in learning technology can contribute to this understanding. frances bell university of salford, uk reviews europe in the round cd-rom, guildford, vocational technologies, 1994. europe in the rotmdis a hypertextuall hypergraphical database of facts about europe, biased towards education and training, but taking in its stride many peripheral areas relevant to anyone considering living and/or working in, or even visiting, a member state of the eu population, economy, institutions, geography, climate and so forth all making up thousands of interconnected screens. these screens contain maps with a selection of zooms and hotspots, all kinds of facts and figures about available educational and vocational courses (presented both textually and, where appropriate, as graphs, bar charts and such like), details of life in each eu member state, details of cities and what they have to offer, and a lot more. there are also extra facilities, such as being able to set currency exchange rates or to create customized notebooks. the pc version received for review (on cdrom it is also available on floppy disks) has been written in toolbook. the macintosh version (the original) is based on hypercard. both were first published in 1991, and the design and structure of the software have changed little since then, but this latest issue contains more information, and is as up to date as software of its kind can be. improvements and additions include a new section on regional statistics, new sub-sections on work and finance for students, a new section on efta, information on the european job vacancy system, enhanced road maps, distances between cities, a powerful index, generally improved links between sections, and more use of colour. three years (1991 to 1994) is a long spell in the world of educational technology, yet the userinterface of europe in the round looks as fresh as it ever did. and perhaps not surprisingly, since many of the innovations it brought with it have become well-tried features of multimedia cal and reference software. the interface incorporates universal icon meaning and consistent positioning on the screen, never showing inactive icons, navigational clues provided by different transitions between screens, and the like, all of which shows how far-seeing the members of the design team were. the package lacks sound, though one cannot but agree with what jacquetta megarry (its producer) wrote not long after its initial publication (megarry, 1991), that adding sound would in turn have added little or nothing in terms of usefulness, while eating up computer memory and human and financial resources. one could also say that adding sound would be as risky today as it was in 1991: every extra multimedia element brings with it an increased chance of instability. i mention this because my practice these days when reviewing any multimedia package is to give it a tough test: as well as running it on a standard 'clean' machine (where there is rarely any problem, and indeed none with europe in the round), i run it too on a pc which conforms to mpc standards but which is also packed with all manner of cards and device drivers. i do this because not everyone has a perfectly clean machine. if a piece of software passes this test, i can be fairly certain of its robustness. few programs do, nor did europe in the round. i had severe installation problems, including some non-sensical messages (such as inadequate disk space when i had over 100 mb free), and when i eventually managed to install the software by ignoring the instructions in the manual, it crashed or hung on me with irritating frequency. the other side of this coin is, of course, that the test was particularly acid. furthermore, at the end of the manual is a troubleshooting section. this does not tell you how to get out of difficulties, but rather how to contact the publishers (far better than the usual advice about checking whether or not you have plugged in the computer). in other words, the publishers are happy to help with problems of installation or running. i should emphasize this praiseworthy attitude rather than the problems i encountered which were, after all, the result of difficult conditions. so i do. in any case, this is my sole major complaint. other complaints (such as the occasional use of the word card for screen, a long-standing hypercard hangover, or the fact that the program assumes a vga display and therefore does not use the full screen in svga mode on a large monitor) detract little from the overall impressiveness of the package. for it really is a 75 reviews veritable goldmine of information presented in a digestible form. indeed, why it has not yet been taken up by large numbers of public information services throughout europe is a mystery to me. it is certainly the best-presented information on opportunities for work and study in the eu, and its built-in facility for creating slide-show demonstrations seems tailor-made for public access: it should be available in the entrance halls of every educational establishment with european links. it is, very clearly, a tool which can be profitably used by students, whether in he or not, whatever they are studying, and whether or not they intend to study or work in europe. it also represents a fascinating case study in good practical cal design. gabriel jacobs, university of wales, swansea references megarry, j. (1991), 'europe in the round: theory into practice', interactive multimedia, 2, 3. multilingual multimedia bridging the language barrier with intelligent systems edited by masoud yazdanl, oxford, intellect books, 1993. isbn: 1871516-30-7. this book is edited by masoud yazdani, who in addition to having written the preface, has a chapter to himself, and is joint author of no less than four of the remaining eight chapters. yazdani is also the founder of intellect, the book's publisher. is this work therefore some kind of ego trip? well, no more than a singleauthor book is, and on the whole the work is certainly worth reading. the main theme is trans-language communication with the help of software tools such as grammar checkers and icon-based (nonlinguistic) systems. along the way, however, other themes pop up, some of which will be relevant to all readers of alt-j. the book opens with a descriptive piece about a prototype system for practising foreign languages in a restaurant, based around cartoons, sampled sound and hyperlinks. i have seen this program in action on a macintosh. it is impressive when demonstrated but, as the authors of the chapter admit, its most serious limitation is that it does not allow for any alteration of the material. the second chapter is concerned with a multilingual approach to databases intended to be accessed by speakers of different languages. it mostly says little other than comparatively obvious things: such databases ought to have multilingual interfaces, language-specific browsing facilities, non-linguistic systems of communication (such as icons, or video sequences without the use of language) where appropriate, and so on. however, the chapter does pick up when it treats cultural differences. chapter 3 is the first of two dealing with communication by icons and pictures. the idea of using graphics alone, to communicate ideas is studied in a fair amount of depth, and while the design of such graphics is fraught with all manner of difficulties (neatly outlined here), it is clear that there is considerable scope for using pictures in a multilingual environment provided one is careful about which pictures to use. by the time we reach chapter 5, on generalizing language tutoring systems, we have left the soft stuff and have entered the realm of clause analysis, syntactic representation and lexical processing. we are also introduced to linger, an expert-system grammar-checking program for would-be programmers with little programming experience. linger also forms the basis of chapter 6, while chapter 7 treats its successor known as el. the problems involved in checking grammar are formidable, and the two chapters (especially chapter 7) are at least useful in showing just how difficult it is to get a computer to 'understand' natural language. chapter 8 continues the theme of checking natural-language syntax by looking at some problems and possible solutions in this area when error-checking software is used with learners of english as a foreign language. there is a product review, together with a long comparison between the seven programs considered. the unsurprising conclusion is that all the programs fail, in different ways, in terms of the (rightly) exacting criteria used for evaluation. the final short chapter is a critical assessment of artificial intelligence as used in language teaching, again comparing some available products. for me, the downside of this book is that it appears to have been put together somewhat artificially. the unifying motif in the mind of the editor has clearly been that of addressing the problem of multilingual electronic 76 altj volume 2 number 2 communication, but the reader has to work hard to see some of the direct connections between chapters. furthermore, there is an appreciable disparity between material of interest to researchers but probably beyond beginners, and more basic material for relative novices. nevertheless, the book will be of interest both to those concerned with computer-assisted language learning and those concerned with making courseware in general available to speakers of different languages. the chapters on iconic communication will also interest courseware developers in all disciplines. gabriel jacobs, university college of swansea using records of achievement in higher education edited by a. assiter and e. shaw, london, kogan page, 1993. isbn: 0-7494-1111-2. this is the latest in the using... series of books, and it will be of particular interest to all in higher education who have responsibility for curriculum design and delivery. the material presented is drawn from the using records of achievement (roa) conference organized by the hec in 1992, and like that event the main thrust of contributions is very much directed towards educational practice, the enhancement of student learning and informed professional action. the origins of roa and the use of associated portfolios are rooted in two perceived deficiencies of the higher-education system. first, an acknowledgement of the limitations of conventional examinations in doing justice to the full range of learning achieved by students; and secondly a growing realization of the educational value to students of involving them more fully in the reviewing, monitoring and recording of their own learning. this latter stance has, of course, been at the centre of enterprise and capability programmes, although its importance is also to be found in a much older tradition which values autonomy in learning as one of the distinguishing features of university education. the publication is timely, and presents a nicely balanced blend of case studies, contextual issues and practical guidelines presented in an easy-toread format. both new and experienced staff will find much to interest them here, and the use of bullet-point listings works well for readers who want to engage with the text quickly. some 22 short chapters are presented under a number of organizing terms: • the use of roas and profiles for access, admissions and accreditation of prior learning; • the development of skills, both personal and professional; • the empowerment of learners through insight into their own motivations, needs and actions; • the facilitation and assessment of workbased learning. any reader pondering these issues will be interested in the cases and descriptions of practice from fellow-travellers who have already grappled with them and who have begun to formulate responses. it is particularly welcome to see the odd contribution from employers interleaved with those of academic staff, and salutary to note that others from very different contexts may have much of value to share with us. to this reader, the material certainly achieves its stated intention of 'providing a glimpse of what others are doing and what they have learned by doing it', and herein lies its paramount strength. an important sub-theme running through the variety of perspectives is the deployment of learning contracts and their importance as a mechanism for mediating the needs and interests of students, tutors and employers. first-hand accounts of the management of these processes provide valuable insights into the practical difficulties surrounding attempts to introduce innovation into higher-education institutions. if there is a deficiency in the book and this is a small one it is that the students' voice and perspective is not placed alongside those of the other key participants. this is an irony, given the commitment in many of the sections to a student-centred approach and learner empowerment. despite that reservation, however, this volume makes a significant contribution to what is an evolving area of work in many institutions, and as such is recommended as an excellent starting point for those considering similar initiatives. gus pennington, university of teesside 77 reviews handbook of educational technology by henry ellington, fred percival and phil race, 3rd edition, london, kogan page, 1993. isbn: 0-7494-0849-9. this publication gives a taste of a wide field, and is supported by an extensive glossary of terms used in educational technology, and a bibliography of recent articles and books. the authors go to some lengths to explain that by 'educational technology', they are emphasizing more the sense of 'the technology of education' than 'technology in education'. in this respect, they deal extensively with the way in which effective education may be crafted or engineered rather than considering the technological devices which might be employed. to this end, the book has chapters on basic educational strategies, educational objectives, instruction and learning techniques, assessment and evaluation, and includes a chapter on resource centres. each chapter is well structured with a basic but detailed outline of the topics under consideration, with their advantages and disadvantages. in the first chapter, the authors introduce a simple model of learning comprising the four stages of 'wanting, doing, feedback and digesting'. this is an interesting addition to the informative but rather dry chapter. the authors continue to work with this model throughout the book, often referring to it to support a particular instructional technique or educational approach. this helps the book to be more cohesive and rounded than it would otherwise have been, and will be very helpful to a reader with little or no exposure to theories of learning. the authors also posit a simple idea that there is a continuing trend from mass education, through individualized education, to group learning, something which is again used throughout to link otherwise diverse sections. however, the reader whose interest is in technological devices which can facilitate educational techniques, and moreover is looking for basic detail on the specific educational applicability of hypertext or hypermedia, groupware or cognitive strategies, or is keen to delve into the wider social or political aspects of the introduction of educational technology, will be disappointed. and the disappointment may be particularly acute in the last chapter where four out of seven of expected future trends in education have recently been under considerable attack by government ministers (two others are supported, but possibly for reasons of costsaving or social engineering, rather than in the interests of effective education). chapter 10 deserves a detailed look in the light of my overall criticism of a lack of being up to date: it offers a quite limited review of computers in education. i have not read the earlier editions of this handbook, but the chapter has a rather dated look-feel, with illustrations of a monstrously huge mainframe and of 'typical' textual computer interfaces capable of displaying only very basic graphics. although the end of the chapter seems to have been updated to include some sections on email, hypermedia, cd-i and photo-cd, these do not seem to hang together very well, and the chapter as a whole still seems out of date. unfortunately, in such a fast-moving field, the successive updating of a handbook first published in 1984 leaves an overall effect rather like that of an archaeological dig, where each era overlays another. a handbook on educational technology that has more on filmstrip projectors than hypermedia must be a candidate for a complete re-write. this is a suggested handbook for trainee teachers and other students of education or computers in education, and as a basic introduction i suppose it works reasonably well. supplemented as it is with a good bibliography, it will provide a fair starting point for any student, or indeed lecturer, with no previous exposure to educational theory. however, any novice in the field of educational technology would have to be encouraged to move on to more comprehensive texts as quickly as possible to avoid the over-simplified weltanschauung promoted by this publication. terry king, university of portsmouth the principles of screen design for computer based learning materials by alan clarke, 2nd edition, moorfoot, sheffield, learning methods branch, employment department, 1992. this book aims to inform authors of computerbased learning materials of findings in the research literature relevant to screen design. in reality, it draws on work that relates to information presentation in general, whatever the medium. there are chapters devoted to electronic text, colour, and graphics, with a collection of guidelines at the end of each chapter (the guidelines are also summarized at 78 altj volume 2 number 2 the end of the book). unlike many other sets of guidelines, this book does actually go into some detail of the research it draws on, rather than simply providing references. however, although this format allows the reader to make an assessment of how generalizable the guidelines may be, it does make it harder to find relevant advice, a problem compounded by the lack of an index. apart from minor niggles with the number of typographical errors, ungrammatical sentences, and the use of sexist language, i have other deeper concerns with this book. my first is that it contains no discussion of the design process itself. i found no mention, for example, of involving users in developing material, no mention of evaluation, and no mention of iterative design. although none of these things can guarantee producing good displays, they will certainly be at least as effective as guidelines in helping to avoid unusable ones. a second concern is that the book contains an uncomfortable amount of what might be termed naive psychology. for example, the assertion that 'long term memory seems to operate in a similar way [to short term memory]' (it doesn't), and the suggestion that miller's classic 'magic number seven' paper on the limits of short-term memory informs the decision of how many colours to use in any one display (it doesn't). this simplistic style of presentation does little to point to the importance of considering psychological factors when designing screen displays and learning materials (see, for example, helander 1988). a third concern is that the book is not comprehensive. seen simply as a collection of guidelines, there are surprisingly few of them (less than 80 in all). there are also surprising omissions from the list of references. for example, there is no mention of the classic work edited by smith and mosier (1986) who list over 900 guidelines (along with supporting references) for designing user-interface software, of which almost 300 are concerned with data display alone. finally, there is little discussion of the way particular authoring tools impose their own style on the finished product, or of the recommendations made in the interface style guides produced by apple and microsoft (for example, microsoft 1992). interestingly, clarke is careful to distinguish between guidelines ('hypotheses which [are] supported by evidence and dependable within limits') and principles ('guidelines which-[are] dependable in a wide range of situations, locations and with different groups of learners'). he concedes that much of the material presented in the book should be considered as hypotheses and guidelines, rather than principles, which makes one wonder about the title of the book. however, if you are looking for a few simple guidelines for a computerbased learning project, it may be a useful starting point, even if, in my opinion, smith & mosier or helander would almost certainly be better. overall, though, i have to confess to a certain scepticism that mechanically applying guidelines, whatever their source, will help anyone to design effective displays or learning materials. involving users at an early stage, conducting proper evaluations, and using a strategy of iterative design are far more likely to produce successful results. tony gillie, university of reading references helander, m. (ed) (1988), handbook of humancomputer interaction, amsterdam, north holland. microsoft (1992), the windows interface: an application design guide, microsoft press. smith, s.l. and mosier, j.n. (1986), guidelines for designing user interface software, the mitre corporation, report esd-tr-86-278. copies of this monograph are available free of charge from cambertown ltd, unit 8, goldthorpe industrial estate, goldthorpe, rotheram, south yorkshire s63 9bl. quote reference 50-ol121 when ordering. multimedia mania by d. paulissen and h. frater, michigan, usa, abacus software and dusseldorf, germany, data becker, 1993. isbn 1-55755-1669. it is well known that information is more effectively communicated, and messages more persuasively delivered, through the use of an appropriate mixture of text, sound and images. what is new about current approaches to multimedia is the technological opportunity that now exists for delivering such information via computers, but of course handling the various types of data involved is not easy. indeed, in order to complete a multimedia 79 reviews project successfully, specialized hardware and software are usually needed, as well as relevant experience. unfortunately, few (if any) of the currently available books on multimedia cover all the necessary techniques in a way that is suitable for newcomers to the field. multimedia mania is a two-part polymedia publication that attempts to fill the gap. it consists of thirteen chapters, three appendices and a subject index (all printed on paper) and a cd-rom containing a large number of demonstration and shareware programs. the first chapter briefly describes various multimedia applications, providing a useful introduction for newcomers. the second chapter describes the multimedia pc (mpc) standard and the various technologies (cdrom, audio, video, image acquisition, etc.) normally associated with multimedia. chapters 3 and 4 will be of interest to those involved in acquiring and installing a new mpc system. configuring a multimedia pc is often a tricky job particularly, for sound recording and playback. and in this context, the tips provided here will be found very helpful, especially with respect to the installation of sound cards. following on from this, the fifth chapter (on sound recording) describes the processes involved in making digital recordings using a pc. the accompanying cd-rom is used to expose readers to a sample music database developed using microsoft excel as a retrieval engine. it also takes readers through a hands-on session with the ezsound fx sound-editing utility. capturing and processing images are covered in chapter 6. here, the hijaak screen capture program (provided on the accompanying cdrom) is described in some detail anyone who intends to prepare a multimedia presentation will find this software useful. scanners and image processing are the other two important topics covered in the chapter, which also provides up-to-date information on kodak's photo cd. chapter 7 provides very good coverage of animation and its associated techniques. the examples presented include animation with toolbook and autodesk animator. examples of 3-d animation are also there, and again, the demonstrations provided on the cd-rom will be of help in learning about this topic. for those interested in midi, chapter 8 will be of substantial practical value: it contains a wealth of tips and guidance on musical applications of the mpc. sounds and pictures as essential components of multimedia presentations are the subject of chapter 9, the style and content of which make it of significant value for those starting off in multimedia. the wide range of demonstrations (based on mmplay, tempra show, authorware star, excel slide show, and picturebook professional) provide excellent hands-on experience. authorware star (which looks very similar to authorware professional) provides a useful way of sampling some of the features of a more sophisticated authoring system. the use of authorware star is further discussed in chapter 10 on authoring and hypertext, where methods for creating hypertext systems using microsoft's viewer are also described, as is the use of the multimedia development kit (mdk). digital video is covered in chapter 11 which is in some ways disappointing: although a number of different video cards are now available commercially, the authors of this book choose to discuss only the microeye video output card, which obviously this limits the scope of the treatment of video hardware. the softwarebased approach for handling video is illustrated, however, using microsoft's video for windows a far more widespread product which is discussed in some detail. and it is good to see that the authors have not omitted the important topic of virtual reality, treated in chapter 12. the last chapter, on visual basic, will be of particular interest to programmers who want to write code for multimedia applications. the examples on the cd-rom will help them explore and experiment with the mci (media control interface) commands within windows 3.1. the three appendices provide a glossary, an overview of various types of multimedia products, and a summary of the material held on cd-rom. the second of these gives a wide range of information sources that multimedia developers will be glad to have: it includes details of animation software, authoring systems, image editors, mpc upgrade kits, pc sound cards, sound editors, video editing systems, virtual-reality equipment, and suppliers of cd-rom magazines. this book will undoubtedly be a useful information resource for anybody wishing to enter the field of multimedia pc computing, and the tips it offers, as well as the cd-rom bundled with it, make it a good buy. ashok banerji and check meng tan, university of teesside 80 altj volume 2 number 2 designing usable electronic text ergonomlc aspects of human information usage by andrew dillon, london, taylor and francis, 1993. isbn: 0-7484-01130-x. increasingly, text is being viewed not in printed form but on a computer screen. this book is therefore a timely one, as it concentrates on the ergonomic and human-factors issues underlying the design of electronic text. it has ten chapters, a bibliography, a single appendix and a subject index. following a short introductory scene-setting chapter, the author moves straight into a discussion of usability and the potential utility of ergonomics and human factors within this area. this is followed by a review of the experimental literature on reading from paper and from screens. two basic strands are considered: outcome measures (speed, accuracy, fatigue, comprehension and preference) and process measures (eye movements, manipulation and navigational issues). the review is organized into three basic parts, covering reported differences between paper and screen, analysis of differences in terms of physical, perceptual and cognitive processes, and issues relating to text and task variables. in chapter 4, the author examines the value of existing human-factors literature to electronic document designers. the problems of applying this knowledge to the design and evaluation of an electronic-text system are then illustrated by means of a case study (an interactive document retrieval system based on a cd-rom called adonis). chapter 5 delves into the problems of classifying texts by means of repertory grid analysis using the focus program, and the implications of the results for electronic text design are then discussed. chapter 6 explores in more depth readers' interactions with two particular types of text academic journals and software manuals. the methodology used in the investigation is based on the wwh (why, what and how) approach to document usage). the chapter concludes with a discussion of the design implications of the research findings. the material presented in chapter 7 is concerned with the literature dealing with readers' impressions of structure and shape in information space (based on schemata theory), and how this might relate to navigational problems within electronic information. two simple experiments designed to investigate document structure are then briefly described. in chapter 8, the author proposes a framework which represents the ergonomic factors involved in using a text, and suggests the variables to consider when designing an electronic document. the framework consists of four interactive elements which reflect the issues dominating a reader's attention at various stages in a reading process. the components of the framework include the task model, the information model, a set of manipulation skills and facilities, and a serial reading processor. the validity and utility of the proposed framework are explored in chapter 9. this describes two experiments involving readers' use of an ordinary text (on paper) and electronic texts (both linear text and hypertexts) for information retrieval. various tools were employed to produce the electronic texts (hypercard, ties, a word processor and guide). the experiments were conducted in a usability laboratory, and verbal protocol techniques were used to analyse readers' behaviour. in the final chapter of the book the author summarizes and discusses what has previously been said, and then turns his attention to potential applications of the framework particularly for the development of systems based on the use of electronic text. i found this book quite useful. it draws together much of the documented scientific knowledge on reading processes, and reviews its worth in the context of comparing paper-based and screen-based visual interaction with text. although there are many more avenues yet to explore, the ideas contained here provide some useful starting points on which to base future research. philip barker, university of teesside cal into the mainstream: computerassisted learning (selected contributions from the cal 93 symposium, 5-8 april 1993, university of york), edited by m.r. kibby and j . hartley, oxford, pergamon press, 1994. isbn: 0-08-041945-3. it is somewhat difficult to choose the most useful way to summarize a book of selected contributions from a conference. the reader 81 reviews does not know if the contributions selected (in this case, 21) are meant to represent as faithfully as possible the range of interest areas and contributions presented at the conference, or if they are chosen on a basis of attempted balance among institutions or regions represented among the participants, or some other criterion such as a mix of 'old' and 'new' contributors. perhaps the standard is purely scientific: the editors' opinions of the most valuable entries. but because of this uncertainty, the reader cannot fairly approach the book as an inventory of british activities with computers in education, nor as conference proceedings. thus, how to proceed? by treating the book as a book, and thus expecting that the title is an organizer for the content? this approach does not help very much either, in that the reader entering the book through interest in its title gets very little sense of how much computerassisted learning has gone 'into the mainstream* of u k education after reading the entries. this said, perhaps the only way to address the book is as a hard-covered journal, with a collection of 21 articles offered without any overt editorial synthesis. that rather long-winded introduction is by way of explaining my choice of procedure for this review. i will give a general comment about the topics and settings addressed by the articles, then note some that i have found to be particularly useful. thus the overview of contents: 17 of the 21 articles are from u k author teams, one a team of mixed u k and non-uk authors, and three from other countries (canada, germany and finland). as to content and setting, my analysis is that: • three deal with software development methodology; • ten could be described as 'specific-case' studies situated at the university level, • seven of which i would most generally categorize as case studies of particular prototype packages, including descriptions of the theory of their designs and development processes and evaluations based on the author/designers' experiences with the inhouse use of the package, • one of which is a more cognitivelyoriented study of an experiment involving an existing package (a concept mapping tool), • and two of which relate, in different ways, to distributed learning (ou students' experiences with home computer use in a particular ou course, and experiences at lancaster with cmc); • four relate to experiences with children as they use a particular software environment; • one, from finland, studies a group of lower-secondary students and a legologo experience; • three, including one from canada, look at groups of primary students as they use either modelling software, or a prototype developed by the authors, or a cd-rom 'talking storybook'; • two describe case-study type observations about groups of students in initial teacher education at two institutions (articles 17 and 19 these articles were so similar in style and substance that i looked back twice to see if i was not reading the same article a second time); and • two describe, from large-scale and longitudinal perspectives, implementation experiences about computers in education. the latter two in the categorization i offer above are, in my opinion, the two articles which most substantially relate to the title of this book. they are the two i solidly recommend to anyone interested in what happens when largescale initiatives try very hard to stimulate computer use in schools. they are also two of the three articles from the collection i have already cited in my own writings, and will continue to use as valuable references, not only because they are based on broadscale samples of experience, but also because of the value of the insights they offer. the first of these two, whose title does not do much to suggest its contents, is by alan brown ('processes to support the use of information technology to enhance learning') and is based on an evaluation of the u k information technology teacher training development programme (1988-1992). it presents many rich and useful insights from seven projects focusing on the management of it use by teachers, and ways teachers use it to enhance learning in the classroom. the second, by john gardner et al ('learning with portable computers'), is a fascinating study of a broadscale attempt, 82 altj volume 2 number 2 involving nine schools and 235 pupils, to give students unlimited access to personal computers (at home and at school) over an entire school year, then to investigate the impact of this access on their performance in core school subjects. the article, whose findings deserve careful study by anyone interested in 'the mainstream', again points out the critical influence of the teacher in what happens with computers and learning. the third article i found particularly useful is by mike aston and bob dolden ('logiciel sans frontieres'), again a title which will not help the reader to identify its contents if seeing only a list of titles. this article is of less general interest than the ones by brown and gardner et al, but of particular value to those involved in software development from a broad perspective. the article summarizes some of the experiences from a number of european-level projects involving cross-national educational software portability. as for the rest of the articles, well, i certainly can say they are well-written, well-edited, and a number of them i have noted for myself because of their relation to my various research interests. but the reader will most benefit from them by seeing them in wider contexts. studies of the generic type: 'what a certain group of our students did with our prototype and what we have learned from the experience', for example, are most valuable when one can place such observations into the context of other studies involving similar cases in order to see where trends and insights transcend the particular case under description. for the reader interested in the various types of software or it-use situations described in the case-study-like articles in this book, there will certainly be good references to add to his or her collection. for example, because i am currently very interested in cmc, i quickly went to the article by steeples et al on the topic, and am pleased to add it to my cmc insights and experiences file. one accepts a journal as a collection of articles, and the articles included here are quite interesting. nonetheless, one would like a book, destined to live for years on library shelves, to go one step beyond the collection level to bring some sort of coherence in the articles for the reader who in the future seeks out the book because of its title. one hopes that those readers will find the articles by brown and by gardner and his colleagues, if they want a reflective summary of a wide set of experiences from the field, from mainstream practice. betty collis, university of twente, netherlands 83 note the content of this book also forms volume 22 (issues 1 and 2) of the journal computers and education. hyperprogramming: building interactive programs with hypercard by g. coulouris and h. thimbleby, woklngham, addison-wesley, 1992. isbn: 0-201-56886-1. this book is aimed at those wishing to develop hypercard applications and who already have some programming experience (for example, in a conventional third-generation language such as pascal or basic). the authors suggest that their work is suitable for a number of target reader groups: novice programmers, expert programmers, students, advanced students, and hypercard developers, and individual chapters are identified in the preface as being of particular relevance to each target reader group. a floppy disk containing hypercard stacks accompanies the book, and complements it by providing the hypertalk code discussed within it. in this way, readers with a macintosh and hypercard are able to run the programs and, more importantly, cut and paste both objects and code for use in their own programs (known as stacks in hypercard). there are a number of books about hypercard, but this one is quite novel in its approach. it goes straight into hypertalk (without lengthy description of the hypercard interface) and combines this with the pedagogic strategy of teaching by example. throughout the book, example code is used to introduce new hypercard and hypertalk topics to readers. a key benefit of this topic-based approach is that it presents material in problem-oriented ways. readers are thus able to discover hints and tips while learning about programming in hypertalk. they are also presented with solutions to some of the practical problems that might be encountered when they come to develop their own hypercard stacks. there are fifteen chapters, all oriented towards practical use. the first provides an immediate introduction to programming in hypertalk. this is followed by two chapters on the basics of working with hypercard and hypercard objects. the fourth chapter then describes the basic commands and control structures used in hypertalk programming. chapters 5 to 11 adopt the topic-based reviews approach mentioned earlier, with each chapter presenting a new theme. they take the reader through a wide variety of practical programming issues while presenting new syntax, structures, and tools. programming topics covered include graphics, timing, animation, menus and searching. chapters 12 to 14 deal with the concept of hypertext. a number of issues are addressed: design, authoring, reading (browsing), and so on. the final chapter presents ideas and strategies for undertaking a variety of projects in hypercard; the aim is to provide some indication of the flexibility and scope of environments like it. a wide range of problems are discussed, including the development of books, games and a simple spreadsheet. in addition, techniques for protecting software from tampering and virus attack are outlined. the two appendices are useful. the first provides lists of information about hypertalk (properties, commands, messages, constants, operators and functions). the second gives a set of comparisons between hypertalk and pascal, designed to allow readers with existing pascal experience to transfer rapidly between the two environments. this second appendix is very well done: the structures used in pascal are presented with the hypertalk nearest equivalent presented alongside. this is an excellent book for anyone wishing to program hypercard applications, particularly if they already have some programming experience. it also has some potential for people who wish to program in toolbook on the pc, an environment similar to hypercard. stephen richards, university of newcastle-upontyne stella ii version 3 (simulation and modelling for system dynamics), apple macintosh version. when stella was introduced several years ago, it provided the first opportunity for relatively unskilled workers to solve complex problems which normally would have required skills in algebra and calculus. stella was designed to model the behaviour of dynamic systems: that is, those in which the values of various components of a system (the system variables) vary in a complex manner with time. for example, the variations in populations in an ecosystem, or drug levels after taking a medicine, vary in a time-dependent manner which can be calculated from a physical model of the system. normally, the study of dynamic systems requires the use of calculus to set up and solve differential equations; these describe the interrelations between the quantities of interest setting up a dynamic model involves two steps; firstly, the identification of the interrelationships between the variables, and secondly, the writing of differential equations which embody these relationships in an algebraic manner. finally, the solution to the differential equations reveals the timedependent behaviour of the system variables. prior to the introduction of stella, there were a number of rather unfriendly programs available which assisted users in setting up the differential equations, and solved them numerically. solving the equations has always been considered the 'hard' part of the problem. unfortunately, if you do not have the skill or training to set up the model or equations, then that is the hard part, and finding the solution is irrelevant. stella allowed the user to define the problem simply by drawing boxes, which represented the time-dependent quantities in the system, and joining them by flows, which represented the inter-relationships between the quantities. the basic form of the system was defined by the structure of the diagram, and only very minor mathematics was required to define the flows. stella then transformed the model into a set of differential equations, and solved them, thus showing how the model system changed with time. the first version of stella was simple and in some ways did not conform to the usual macintosh interface; however, its idiosyncrasies were easily learnt, and it was widely used. in this form, it was capable of studying most of the problems which its users posed; as long as a simple box and flow model could be used to represent the problem, stella could provide an indication of the likely behaviour. despite this, there were a number of deficiencies: some fairly simple problems could not be broken down into a box-and-flow form, and there were limits to the presentation of the output, which was either a copyable string of numbers or a pictable graph. stella ii began to address these problems, and stella ii 3.0 adds yet more functionality. the new modelling tools added to stella ii allowed processes to be used which could only poorly be represented by boxes. these include the conveyor and the oven. a conveyor is a structure into which you can feed material at one end and it will appear at the other end after 84 altj volume 2 number 2 a delay. this is a much more realistic model of many processes. take, for example, a model we recently used to study the flow of material through the gastrointestinal tract. originally, we used a sequence of boxes to represent the stomach, small intestine, and large intestine. the problem with this model was that as soon as the stomach had emptied some material into the next box (the small intestine), it could be further emptied into the large intestine. of course, this is not a physically realistic model since the small intestine is a long tube and, as we know, it takes time for material to get to the far end. a conveyor is a much more realistic structure to model this behaviour. stella ii also allowed oneway or two-way flows, and most usefully introduced a non-negativity constraint. in the earlier versions, •accidental negativity of a variable had to be checked for by the model builder, which was most inconvenient. these additions, in our opinion, covered most of the requirements of the average user. stella ii 3.0 adds to both the computational and presentation areas. most of the changes have to do with presentation of the results. there are now two versions, stella ii 3.0 ordinary, and stella ii authoring. this is in line with many programs which now provide a full-featured version for developers, and a simplified version for end-users of the models. we were only able to test the authoring version, so cannot comment on the lesser facilities of the ordinary version. however, the claimed features of the authoring version over the other version include the ability to create stand-alone models fmicroworlds' ugh!) which can be locked so that students can explore their operation but not mess about with their structure. the authoring version also has a mapping mode to aid the construction of complex models, and an overview mode to display the broad structure of the model. this level allows the importation of graphics and quicktime animation (which we did not test). in this aspect, stella 3 (let us call it that) becomes more of a presentation package than a laboratory program, which will be useful for people who need to use simulations as a teaching or negotiation tool. the ability to create stand-alone models is most useful for teaching purposes (students may not need the full ability to create models). there are rather fewer computational additions to stella 3. the built-in function list is slightly longer with 11 new items. publish and subscribe are supported, which may allow some interesting connections to other calculation packages such as excel or mathematica. a valuable feature is the ability to plot several runs of a model on the same graph to allow comparison of changing a particular parameter. in previous versions of stella, different graphs had to be drawn and each plot locked before a parameter could be changed and the model re-run to produce the new graph. this made comparison very hard and necessitated printing each individual graph. one of the most serious difficulties with stella and stella 2 was that they were available only for the macintosh. 'is there a pc version?' was probably the most commonly asked question we encountered in our years of showing the program to our colleagues. stella 3 promises to provide model transferability between mac and pc; however a pc version was not available to us so we could not try it. the review version of stella 3 was provided on a single disc and occupied 1.3 megabytes of space. it loaded without difficulty onto a macintosh 2vi and a 2ci, both running system 7 with the usual heterogenous array of extensions, and claimed 1.5 megabytes of memory, although this could be reduced to 1 megabyte. we did not perform any timing comparisons with earlier versions, but the simple models we created ran smoothly and quickly, giving the same numerical results as those from stella 2. models created in stella 2 opened without difficulty in stella 3. despite the major additions, it must be said that the original version of stella has a simplicity which allows new users to grasp its functions rapidly, and will do virtually all that is asked of it. stella 3 is easy to learn if you have used earlier versions, but its many features can confuse novices. it is a common situation with all forms of software; for example, all we usually need is old macwrite and multiplan, but now we have word and excel, with massive functionality that is rarely accessed. who will buy stella 3? we would expect that people needing to present their models to audiences will find the additions invaluable, and particularly teachers associated with computerassisted learning in higher education. pc-based departments will benefit from the availability of the package on their machines. research users of modelling software will find new features in stella 3, but most of the time they will not need the glitz. clive and neena washington, university of nottingham 85 reviews note stella ii version 3 is available from cognitus systems, 1 park view, harrogate, yorks, hg1 5ly (tel: 0423 562622). it is available only to bona-fide educational users; others must buy the equivalent commercial package. prices are: authoring £399, standard £249; upgrades and lan packs are also available. the open learning handbook promoting quality in designing and delivering flexible learning by phil race, 2nd edition, london, kogan page, 1993. isbn: 0-89397-392-0. the transformation of higher education into a system adapted to servicing the needs of much larger numbers has created a demand for flexibility. distance and open-learning methods free students from constraints of time and place, and can allow more individualized feedback than traditional teaching programmes. however, there is also an emphasis on auditing for quality efficiency must be complemented by a concern for standards and excellence. this book is intended both for those already involved in the delivery of open learning, and those moving into the field for the first time. there are two main aims: firstly, to provide guidelines on good practice for those producing open-learning materials; secondly, to advise tutors and mentors on the effective support of open learning. the book is a readable, attractive reference text, packed with very useful checklists. the author is clearly drawing widely on his own experience, and although not an open-learning package itself, the book provides a feel for the subject by incorporating many of the features of open learning: each chapter begins with an abstract and a list of learning objectives, and some contain self-assessment questions. chapter 1 discusses the meaning of the term open learning, setting it in the context of the whole learning process. chapters 2 to 7 cover the production of open-learning materials, from the initial design stage through to marking student assignments. chapters 8 and 9 describe respectively the role of tutors and mentors. chapter 10 focuses on the integration of open-learning materials into traditional courses. the book concludes with an annotated bibliography and an index. it should be noted that the production, choice and use of non-print media is not covered. i would single out chapters 2 , 3 , 4 and 6, on the design and production of open learning materials, as being of particular merit. in chapter 2 ('designing for open learning'), the author takes a realistic approach, opening with a discussion about the relative merits of adopting or adapting existing materials, or starting from scratch. a full description of the difference between open-learning materials and traditional textbooks is given. the importance of piloting materials is highlighted, alongside suggestions for obtaining meaningful feedback from the piloting phase. a quality checklist is also provided. chapter 3 concentrates on the importance of showing learners where they are heading. syllabus content is now widely expressed (gnvq/nvq, for example) in terms of intended learning outcomes: competences, performance criteria and range statements. the section discussing these terms is therefore particularly relevant. chapter 4 focuses on the purpose and design of self-assessment questions, and the provision of poor, as well as good examples, is most helpful. chapter 6, on the design of tutor-marked assignments, benefits from covering not only expected topics, such as the development of marking schemes, but goes on to discuss how the delivery of constructive, supportive feedback might be achieved effectively in open-learning situations. chapter 5 moves on to the importance of adopting informal user-friendly language when producing open-learning materials, and the readability of this book as a whole demonstrates the efficacy of this approach. however, emphasizing readability above all else can have disadvantages: using an example from this book itself, improving flow by deliberately omitting literature references in the text could hinder those readers wishing to research specific topics in more detail (despite the inclusion of an annotated bibliography). in the same vein, the lack of section numbers gives an informal appearance, but is an irritation when making reference to the book. there is also the danger that 'friendly' language can so easily become patronizing chattiness; thankfully, the author is too experienced to fall into this trap very often. despite my minor criticisms, i would have no hesitation in recommending this handbook as a useful resource for those interested in the development of open learning materials, and those involved in tutoring open learners. lorraine warren, hull 86 hypermedia for language learning: the free model at coventry university marina orsini-jones and david jones centre for information technology in language learning, coventry university coventry university is pioneering the integration of hypermedia into the curriculum for the teaching of italian language and society with the creation of a package based on nerino rossi's novel la neve nel bicchiere. the novel was already in use as a basic course text, and developing a hypermedia package was felt to be the ideal way of creating a more stimulating means of access to it. the procedure used in creating the package is described, as are its contents, the ways in which the students use it and the tasks they are given to perform, the feedback from the students, and its impact on their performance. the testing of the prototype has helped in creating a new cognitive model: the free (fluid role-exchange environment) which functions as a fluid and interactive 'pool' where the three main actors, or act ants, ie. the learner, the lecturer and the computer, exchange roles. within the free, students were involved in the construction and evaluation of the courseware, as well as testing the various versions of the prototype. the development and use of hypermedia inside and outside the classroom has made it possible to change both the students' and the lecturer's attitude towards the material being learnt. however, the courseware does not seem to equip students sufficiently for essay writing, and this problem needs further investigation. introduction the tradition of incorporating call into the language-learning curriculum goes back to the early 1980s at coventry university, and since then has evolved in keeping with changes in the technology available (corness 1984; benwell 1986; orsini-jones 1987; corness et al 1992; orsini-jones 1993). coventry university is at present pioneering the integration of hypermedia into the curriculum for the teaching of italian language and society. the syllabus for a complete module of the ba modern languages and ba european studies degrees, which will count as l/8th of the students' programme for year 2, has been designed upon in-house produced hypermedia courseware. although the module has not yet run in its present version, the prototype of the hypermedia italian team's (hit) project la neve nel bicchiere has been used, in the 28 v alt-) volume 4 number 2 academic years 1993/4,1994/5 and 1995/6, with students taking italian as an option in the final year of their ba honours degree in modern languages. the testing of the prototype and its integration into the language-learning process have been very successful and have helped in creating a new cognitive model: the free fluid role-exchange environment. the original concept of the free arose from the observation that the very nature of hypermedia technology allowed students not only to consume the finished product, but also to play an active role in its creation and in its regular revision and re-elaboration. thus the experience of one cohort of students is fed back into the program for the benefit of the succeeding cohort (see acknowledgements). they cannot do this directly, but indirectly via the lecturer and the technical designer. this was an essential feature of the development stage of the project but, due to the flexible nature of the technology, there is no reason why this process should not continue indefinitely in the future. the project la neve nel bicchiere: why, what, where, when and how origins of the project the hit project was born out of two main needs. the first was of a pedagogical nature: the need to find suitable materials for students taking italian as a subsidiary language in the last year of their ba in modern languages. the course in languages at coventry is (or, at least, was at the time) area studiesrather than literature-based, but it was decided nevertheless to base much of the year's work on the study of a novel. there were several reasons for this decision: the wish to expose students to a different style of language from that encountered hitherto in newspaper and magazine articles; the coherence and continuity provided by a lengthy text; and the fact that works of fiction can sometimes provide a more lively, but nonetheless accurate, picture of social conditions than can the study of factual material. it was, of course, necessary to choose the novel carefully. it needed to be by a contemporary author, and to deal with the lives of ordinary people in a realistic way. it also needed to be of the right length to be covered in the 25 teaching weeks available. the novel would be used for the purpose of teaching the language as well as the history and society of italy. la neve nel bicchiere by nerino rossi (rossi, 1977) was eventually chosen because it fitted perfectly within the lecturer's requirements. the book, written in the first person and clearly autobiographical in origin, describes the historical and social conditions of an italian family over a period of 70 years (1880-1950). furthermore there is a film based upon it, and it is easy to access materials about emilia-romagna, the italian region in which the book is set. the book and film were used for two years. however, the students' performance in their seminar presentations of the chapters in the book was poor. students were finding using a book too difficult, and this reflected negatively on their ability to communicate effectively in italian. it was therefore necessary to find a way of making the book more user-friendly, as it was felt it was vital to keep it as the textbook, but the way it was being used was obviously not achieving the objectives set in the syllabus. secondly, there was a need of a more practical nature: the teaching hours for the italian — marina orsini-jones and david jones hypermedia for language learning: the free model at coventry university option were reduced from three to two per week after the introduction of modularization (1992). it was obvious that it would be impossible to cover the book in the existing format of lecture/seminar presentation/language work, particularly in view of the difficulties students were experiencing with the material. it was felt that developing a hypermedia package to be used both inside and outside the classroom would be the ideal way of creating a more stimulating means of access to the book for the students. creating the package the way in which the hit proceeded in building the package on screen was the following: in the first place, each screen was designed on paper by the lecturer creating the package, then submitted to the rest of the team for comments. once the screen was in its electronic version, feedback was asked of students too. their feedback was also used by the lecturer and the technical designer to further modify the package. all the texts but one (the chronology, see below) were first typed in word for windows, then transferred into toolbook. the whole of the text was transferred to the computer (manually, by two research assistants) after the issue of copyright was cleared with the publisher. in constructing the courseware, the hit tried to avoid making the mistakes outlined by khan (1995), i.e. producing courseware which is visually attractive but educationally weak: 'too many people put a linear book online, give it some bookmarks, and call it a hypertext; worse yet, they add a few scanned-in photographs and a soundtrack and call it multimedia' (p.99). such 'mistakes' occur when courseware designers follow mcluhan's dicta ( mcluhan and fiore, 1967) too strictly, and become infatuated with the medium rather than the message. in the course of the development of la neve, the hit became increasingly aware of the negative attitudes to multimedia which have developed among some colleagues involved in call, probably due to exposure to educationally weak products. bad multimedia products reinforce the old 'not invented here' syndrome, and encourage the recent backlash against information technology which is becoming increasingly fashionable see, for example, the various articles in the multimedia insert in the higher (such as the centre-page on 12th march 1995, pp. vi-viii). using the courseware the starting point of the students' activities is the text of the book (which they also have in hard copy). after the credits screens comes a summary page: by passing the pointer on the various buttons for the different chapters, a brief summary of each chapter appears on the left-hand side of the screen (figure 1). this was done in order to allow students to skim and scan for major topics in the book. by clicking on the buttons called capitolo 1, capitolo 2 (chapter 1, chapter 2) etc., students access the text of each chapter (figure 2). on this screen are two sets of buttons. on the right-hand side are the 'books' buttons which lead to information about: • italian history cronologia (figures 3 and 4); • a glossary activated via hotwords in the main text vocabolario; • a description for each character mentioned in the book, both fictional and historical personaggi non storicilpersonaggi storici; 30 alt-j volume 4 number 2 • geography of the region connected to the places mentioned in the book luoghi (figures 5 and 6); • the revision exercise for each chapter, in a multiple-choice format revisione capitolo. some pictures (historical characters, places, etc.) and some extracts from the film (chosen by the students in 1995/6) have been incorporated into the courseware too. the words in the glossary have mostly been chosen by students. feedback from students also affected the way the words in the glossary were explained: originally it was thought the glossary should be monolingual, but this idea encountered strong resistance from all cohorts of student users. the compromise solution is that there is a translation of each word in the glossary, but any further explanations are in italian. la neve nel bicchi figure i: summary page with buttons for each chapter in the book. summary of chapter i has been activated in this screen by clicking on the button capitolo i. fai click con h mouse sul capitolo che ti interessa. capitolo 1 introduzlone a! seguentl personaggi: venanzlo, marietta, medea. eneo c nullo. storia d'amore tra medea e eneo. descrizione de 'i due ponti'. capitolo? ii capitolo 13 capitolo 2 capitolo 8 capitolo 14 capitolo 3 capitolo 9 capitolo 15 capilolo 1 capilololo capitolo 16 capitolo 5 capitolo 11 capilolo 17 capitolo 6 capitolo 12 capitolo 18 c • o copyright coventry university. all rights reserved. no part may be copied or reproduced without written permission from the university. figure 2: screen with part of the text for chapter i: this is the main starting point for navigation into hypermedia. la neve nel bicchiere capirolo 1 scariolante, bracciante, mezzadro. ami, mezzadro deh'arciprete, che era on po' meno. foi bovaro e, infme, mezzadro regolare. insomnia una "camera" di totto rupetto. ii nonno venanzb l'avera pereorsa fra stenti, polenta e malaria. eppnre aveva le sembianze di nn gran "signore". chi vedesse la tna inunagine ovale julia laptde al camposanto di castenaso darebbe nn certo credito a ana voce che le nuore, dopo arerio contemplato dall'altro lato dellahnghissimatayola, andarano a ripetem van l'ahra durante il breve tmo che scariolante...: anche"scamolante",n.m.s., labourer whose work consists of pushing awheelbarrowfoll of earth and esci i appunti 1 cerca vocabolario seconda meta capitolo cronologia pcrsonaggi non siorici personaggi storici revisione capitolo 31 marina orsini-jones and david jones hypermedia for language learning: the free model at coventry university 1870-1880 comindano le opere di bonifies del delta del fiume po, presso il mare adriatico. cul partedpa nullo come scariolante. i carbonal anarchid (da cul discende don angelo] disboscano i'appennino. 1870/1871 it 20 settembre i'esercfto italiano occupa roma: ii papa plo ix <l dlchlara prigioniero e si rinchiude in vaticano. 1872 si svolge il primo sdopero generate del lavoratorl cu larga scala in 25 d t t i del nord e del centra. alia conferenza di rimini viene fondato 1b74 elezloni politlche: la destra storiea resta al potere. a bologna gli anarchid tentano una insurrezione die fallisce.1875 nasce venanzlo. figure 3: screen with main chronological index (cronologia): on the left are the fictional events, on the right, historical events. further information about characters mentioned is available by clicking on the buttons at the bottom of the screen (personaggi non storid fictional characters and personaggi storici historical characters). figure 4: example of the chronology: the years 1870-1875. if we return to the main screen (figure 2), on the left-hand side (at the bottom) are: • the search button cerca which works like any other search facility in a wordprocessor; • the notepad button appunti which enables students to take notes while they are working on the package and save them and/or print them out; • the close button esci exit. the help file istruzioni is still under development. in the academic years 1993/4, 1994/5 and 1995/6, the courseware la neve nel bicchiere was used by 50 undergraduates who approached the package in different ways, mainly in their guided-study time, according to the tasks assigned by their lecturer, to skim and scan the text(s) for information in order to: 32 alt-] volume 4 number 2 multimedia toolbook mapi 22.tbk pie edit iext page help figure 5: map of the emilia-romagna region, where the book is set with the towns and cities mentioned in the book. moweua j sostbkso »a1fok meoksft • * m < s » t roeaoncccxo • »c«siaeueu=o cltta e dttadine nel libra. per avere ulterior! informazlonl sui tuoghl fare dick con ii mouse dentro alia carina (in qualunque poslzlone). per sapeme dl piu sulla rcglone emilia romagna. fare dick con ii mouse sul bottone ibarra) "emilia romagna". multimedia toolbook mah1 22.1 bk figure 6: detailed description of one of the places mentioned in the book. p i e tjdlt i e x l eage help i due pontl campotto argenta ferrara i due pontl... questa locallta die at trova all'imboccatura del paesino di campotto (frazionc dl argenta. vedl) e 'protagonista' dl tuna la prlma parte del romanza. in quanta la prlma casa della famlglla dl cul esso tratta. risiede in una casa contadina di propriety dell'ardprete don angelo. quasi appogglata contra i'argine dl uno del due ponti (""picture of house with oxen cart. or snippet of film). i due pontl sono uns ii ponte sul torrente idice e i'altro ii ponte sul canale dl scolo della menata. marlena. la moqlie dl venanzlo. si a. prepare for seminar presentations; b. write essays about topics in the book la neve nel bicchiere; c. revise the book before the written examination; d. practise their grammar. here are some examples of tasks given during the academic year 1994/5: 33 marino orsini-jones and david jones hypermedia for language learning: the free model at coventry university a. prepare an oral' presentation on one of the fictional characters in the book (pair work), by searching the text of the various chapters (button cerca), by consulting the initial summary of each chapter, and by using the button personaggi non storici. you may even use the button cronologia to trace further information relating to the character in question in the historical context of the book. please provide wordprocessed hand-outs for your colleagues. b. compare the fictional representation of the first world war in the book with what is reported about it in the cronologia. the buttons personaggi storici, cronologia and luoghi can all be used for this purpose. c. analyse the major differences in the representation of the region emilia-romagna in the book and in the film. please refer to the button luoghi for this task. d. look for the sentences in which the following verbs occur in the gerund form in chapter 2: amando, pestando, aiutandosi, cercando, insistendo, passandole vicino, discuss in pairs the different meaning of each gerund according to the context in which it occurs, draw up some general rules on the comparison between the use of the gerund in italian and in english. you may use your grammar reference book. students can moreover listen to some of the texts, but there are no listening comprehension exercises in the courseware as yet (1996). there is, however, a reading comprehension exercise for each of the 18 chapters in the book. students receive training to learn how to use the courseware prototype in its different versions. students who want to use the prototype in their self-study time can easily do so as it is available on the network in three different open-access laboratories. to sum up, to date the project has helped students of italian to develop: • their ability to make cross-references between the different components of their syllabus for italian (i.e. history/language/present-day society); • their linguistic/communicative skills; • their information technology skills. the feedback and the outcomes the hit has been pleased with the feedback obtained both from the students using the prototype and from the teachers/lecturers who have seen it at call conferences and/or workshops. the general comment has been that they find it extremely easy to understand how to navigate the hypermedia environment in la neve. however, students varied in their initial reactions to hypermedia. in the academic year 1994/5, at the beginning of the first term, some were very reluctant to use the courseware, mainly because they did not have any it skills. for some, this issue was overcome after the four hours it training provided as part of the italian course; for others, extra hours of it instruction were needed (4-6) and were provided as guided-study time hours. by the end of the second term, even the most reluctant students had been converted to hypermedia mainly because their oral performance in italian had improved 34 an-} volume 4 number 2 dramatically. this change was noticed by the lecturer in charge of the course, who found that students who had used hypermedia presented their material in a more effective way than those who had used the book before the introduction of the courseware. the biggest achievements brought about by the use of hypermedia in the academic year 1994/5 the year in which the feedback was monitored in a more systematic way were as follows. • even the most computer-illiterate students had acquired word-processing skills: all students word-processed both the hand-outs for their seminar presentation and their assessed essays. • using the courseware la neve nel bicchiere had had an impact on the learning and presentation techniques of the learners. the structure of the courseware, which is topic-driven, was clearly reflected in the re-elaboration of the material carried out by the students, who gave highly structured and well-presented 'lectures' and made extensive use of various teaching aids (such as ohp transparencies, extracts from the film, pictures, charts, etc.). students, moreover, prepared and distributed to their colleagues good-quality hand-outs used for their presentations. • students enjoyed the courseware and were keen to contribute to its construction. theoretical considerations the main aim of the hit project was the creation of a learning environment which was filled with content consistent with the medium carrying it. the hit also wanted to involve students further in their 'learning contract', to enable them to participate in the construction of hypermedia on as equal terms as possible with their lecturers. the objective was a more democratic setting where roles could be easily exchanged. the starting point was the adoption of the constnictivist theory of learning (barker and proud, 1987; merrill, 1991; cunningham et al, 1993; barker, 1993), according to which individual students and groups of learners should be actively involved in constructing knowledge, rather than being mere consumers of other people's. because of its inherent flexibility, hypermedia lends itself to this approach to learning. it was of course necessary to prepare the ground for the introduction of truly integrated call materials into the classroom. both lecturers and learners needed briefing about their new functions. the cognitive model represented in the free (table 1) stemmed both from the experience of integrating hypermedia courseware and from that of keeping records of the students' and lecturers' feedback about the courseware. the free functions as a fluid and interactive 'pool' where the three main actors, or actants, i.e. the learner, the lecturer and the computer, exchange roles. the choice of the word actants instead of actors is deliberate. actants, which originated in french linguistics, is widely used in italian linguistics attanti. in etymological terms it is a noun which derives from a participle, an indefinite mood which better embodies the dynamism of the interchangeable roles the actants can assume in a hypermedia environment. actors would convey the idea of a more passive role. the hypermedia environment which is three-dimensional by nature enables the three actants to amplify and multiply their 35 marina orsini-jones and david jones hypermedia for language learning: the free model at coventry university various roles in a fluid web of cognitive relationships (represented by the arrows in figure 7). using hypermedia encouraged an on-going, open-ended redefinition of the 'imaginary borders' around the three above-mentioned actants. this is meant to be in keeping with the philosophy underlying the approach to the integration of it expounded by the more piaget-oriented cal theorists (see for example papert, 1980; varisco and mason, 1989; paterson and rosbottom, 1995). in the free model, the word tool is intended as meaning a flexible 'inanimate object'. it is applied only to the computer, which is thus similar to the computer in the role of 'pedagogue' as described by higgins (1983). in this role it serves both learners and lecturers. but the computer is not only a tool, as it can be a tutee too (students constructing hypermedia/lecturer constructing hypermedia) and/or a tutor (self-access, open-study). similarly, the lecturer can be both tutor and tutee (sitting back and learning from the learners) and the same applies to the learners. within the free, students were and are involved at the following levels: • testing, both inside and outside the language classroom, of the various versions of the prototype; • evaluation, via questionnaires in italian and english and weekly reports in both italian and english about the courseware; • construction: compilation of lists of hot-words to be included in the glossary of the electronic version of the book, writing of the help file for the courseware (in english), etc. free: elwdeole-ixchange environment actanti learner actant2 lecturer computer actant3 figure 7: graphic representation of the free model. 36 alt-j volume 4 number 2 the students' feedback, filtered by the lecturer and the technical designer (the experimental officer within the school vic tandy), feeds into the courseware which is then tested again by the students. further feedback is collected and incorporated as much as possible into the package and so on and so forth in a kind of circular paradigm of knowledge. in this way we can truly talk about tailoring the prototype according to students' needs. we must stress though that construction here refers to construction of contents, rather than programming. the process of student involvement was enhanced by the support of funding from both the enterprise in higher education initiative and the learning development systems funds within coventry university. funds were obtained, for example, to pay some students as research assistants so that they could keep a weekly diary some in italian and some in english of their use of the hit project and of their reactions to it. other students received funding to select the hot-words, and subsequently to check the accuracy of the english for each definition in the glossary. more recently, the allocation of some european funding (leonardo transnational pilot project for the dissemination of innovation in language teaching) has made it possible to pay some students to create the help file for la neve. needless to say, it boosts students' confidence to know that: a. their feedback is valued by the designers of the product; b. their feedback is incorporated, where possible, into the product; and c. they can earn some money by participating in the creation of the project. the vocational aspect of receiving funding for their contribution motivates undergraduates further, particularly in the present climate of diminishing grants. conclusion the experience of integrating the courseware via students' involvement has been positive. the development and use of hypermedia both inside and outside the classroom has made it possible to change both the students' and the lecturer's attitude towards the material being learnt. the aim of boosting students' communicative competence was achieved: it would appear that the integration of hypermedia is ideal for developing the skill of speaking in the foreign language, as well as that of taking notes in order to prepare for seminar presentations. it would also appear that the courseware la neve nel bicchiere has been effective in creating a new call model the free within which students, lecturers and computers can interact in a fluid and flexible way. however, the courseware and/or the module do not seem to equip students sufficiently for essay-writing. this skill needs to be further developed in order to avoid repetition, plagiarism and lack of originality. in 1993/4 there was the embarrassing recurrence of the same piece of text, reproduced word for word from the courseware in the essays in the written examination. in 1994/5 students were therefore warned repeatedly against plagiarism. this time the essays did not reproduce large chunks of texts from the courseware, but were mainly descriptive and not at all satisfactory. this compared unfavourably with the analytical quality of the seminar presentation and of the rest of their work during the year. 37 marino orsini-jones and david jones hypermedia for language learning: the free model at coventry university the latter issue will be investigated in a systematic way jointly with the english language unit at coventry university in academic years 1995/6 and 1996/7, in order to collect sufficient evidence as to why the students' written performance is not as good as their oral one. the following are the questions which will be addressed. 1. do students find it difficult to write essays in an analytical way both in english and in italian? if so, why? 2. did the fact that they had to hand-write the final exam paper up to academic year 1994/5 affect their ability to write a good essay? students had in fact worked with word processors and the hypermedia package all year, but were then required to do their written examination in a traditional setting. as indicated by martha pennington, in the talk she gave at the call and the learning environment conference in september 1995: 'writing the natural way: on computer', this could account for the substandard performance the students gave. in 1995/6 the first trial of an open electronic-book examination will be carried out during the may/june exam period, and the results will be compared with previous, hand-written, exam paper answers. 3. does the lecturer in charge of the module put too much stress on oral presentation skills and not enough on writing skills? these questions need to be addressed, and answers found, with hard evidence at hand (i.e. the students' exam-paper essays) in order to prevent a re-occurrence of the problem. the above issues do not diminish the value of the introduction of hypermedia, on the contrary, they are proof to the fact that its introduction was invaluable for the lecturers involved both in terms of self-assessment and of staff development. finally, using the courseware has also resulted in higher student motivation, which was shown by the response to the module based upon the courseware 301 lai italian language and society given by the students in 1994/5 in the evaluation questionnaires filled in at the end of the academic year. the module scored 5/5 overall (on a 1-5 scoring system), i.e. total student satisfaction. acknowledgements it was thanks to various brainstorming sessions with colleagues and friends in particular kathy courtney (tltp: project astcovea, coventry university) and frank knowles (linguistics, university of aston) that it was possible to arrive at the free definition of our model. we would also like to thank all the students of italian who used the courseware and actively contributed to its construction and development in academic years 1992/3, 1993/4, 1994/5 and 1995/6. references barker, p.g. and proud, a. (1987), 'a practical introduction to authoring for computerassisted instruction. part 10: knowledge-based cal', british journal of education technology, 18 (2), 140-60. barker, p.g. (1993), exploring hypermedia, london, kogan page. 38 alt-j volume 4 number 2 benwell, a. (1986), 'integrating the computer into a language course' in cameron, k.c. et al (eds), computers and modern language studies, chichester, ellis harwood. corness, p. (1984), 'integrating the microcomputer into language teaching', language monthly, 9. corness, p. et al (1992), tiger (translating industrial german): computer-aided learning package for german. modules: introductory, intermediate and advanced, coventry university. cunningham, d.j, duffy, t.m. and knuth, r.a. (1993), 'the textbook of the future' in mcknight, c., dillon, a. and richardson, j. (eds), hypertext: a psychological perspective, chichester, ellis horwood. curtis, f. (1993), 'computer discourse: language imprisoned or empowered?' in monteith, m. (ed), computers and language, oxford, intellect. kahn, h. (1995), 'effective cal: theory and practice', association for learning technology journal, 3 (1), 98-102. higgins, j. (1983), 'can computers teach?', calico journal, 1 (2), 4-6. mcluhan, m. and fiore, q. (1967), the medium is the message, new york, bantam. merrill, m.d. (1991), 'constructivism and instructional design', educational technology, 31 (5), 45-54. orsini-jones, m. (1987), call and italian: integrating a call component into an italian language course, unpublished ma dissertation, university of warwick. orsini-jones, m. (1993), 'a case for hypermedia in the teaching of italian' in wilkin, a. (ed), tuttitalia (the association for language learning journal for italianists), 8, 23-37. papert, s. (1980), mindstorms: children, computers and powerful ideas, brighton, harvester press/basic books. paterson, p. and rosbottom, j. (1995), 'learning style and learning strategies', association for learning technology journal, 3 (1), 12-21. rossi, n. (1977), la neve nel bicchiere, venice, marsilio. varisco, b.m. and mason, l. (1989), media, computer, società e scuola: orientamenti per la didattica in prospettiva multimediale e cognitivista, turin, sei. 39 designing software to maximize learning1 bridget somekh scottish council for research in education this paper starts from the assumption that any evaluation of educational software should focus on whether or not, and the extent to which, it maximizes learning. it is particularly concerned with the impact of software on the quality of learning. the paper reviews key texts in the literature on learning, including some which relate directly to software development, and suggests ways in which a range of learning theories can inform the process of software design. the paper sets out to make a contribution to both the design and the evaluation of educational software. introduction i take it as axiomatic that those involved in developing educational software intend that it should maximize learning. the evaluation of educational software, therefore, needs to focus mainly on this central issue. there are still, however, two ways in which 'maximization' can be measured: either in terms of the increase in the amount of learning, or in terms of the increase in the quality of learning. in either case, an important contributory factor in the evaluation of educational software is the amount of take-up in terms of purchase and use. the crude indicator of bsh (number of bums on seats for number of hours) is of some value, because if software is not used it cannot have any impact at all on learning. but it is an indicator which should only be used with caution because badly designed software used frequently presumably does less to maximize learning than well designed software used infrequently. i am concerned here with the design of software which maximizes the quality of learning. i will also deal briefly with the issue of sensitivity to the context of use, but the wider problem of the take-up and use of software, although important and interesting, is beyond the scope of this paper. the paper is based on a review of key texts in the literature on learning, including some which relate directly to software development. i hope it will make some contribution to both the design and the evaluation of educational software. 4 alt-] volume 4 number 3 learning theories there is no one recognized right way of thinking about learning. theories change over time and tend to reflect the attitudes and technologies of the period. the origin of the jesuits' emphasis on learning by heart lies in the scarcity of books in medieval times. this was later developed into a concept of 'training the mind' in which the habit of, and capacity for, memorization was seen as a prerequisite for scholarship. such a theory of learning had direct pay-offs. with so much information literally at the tip of their tongues, jesuit scholars became known for the acuity of their thinking and their administrative acumen, as well as their rigorous approach to matters of faith. in the twentieth century, the main debate has been between those following skinner, who see learning as 'conditioning' by means of a carefully planned sequence of 'stimuli' designed to elicit a set of desired 'responses', and those following the ideas of vygotsky and bruner, who see knowledge as 'a process rather than a product'(bruner, 1966, p.72), constructed by the learner as a result of experiences, critical reflection upon those experiences, and social interaction (e.g. discussion) (see prawat, 1991). the debate is clearly influenced by new technology which has greatly reduced the need for the memorization of facts, although the need to 'train the mind' continues in the longrunning debate on the need to develop 'transferable skills'. desforges (1989) rejects the confrontational stance between behaviourists and constructivists and sees a need for two kinds of learning. in his view, there are times when we need to learn apparently arbitrary facts (e.g. the sequence of elements in the periodic table), and there are times when we need to learn concepts (e.g. the concept of valence and co-valence in the molecular structures which make up those elements). although, as in this case, there is often an interconnection between these two kinds of knowledge, the learning process is quite different. as desforges says: 'the latter is knowledge constructed by human intelligence in interaction with and adaptation to the environment and unlikely ever to be understood unless reconstructed by the learner' (p.20). much early software of the drill-and-practice kind was designed according to skinnerian principles to support the acquisition of apparently arbitrary facts. underpinning this software is the notion that it is of no consequence that the periodic table is not in fact arbitrary the simplest way of embedding it as a tool that the learner can use with confidence is to teach it as if it were arbitrary. there is still a great deal of software that works on the basic principle of teaching facts and information as if they were unproblematic and then testing the students to see if they can put these facts and information to use at a fairly simple level to answer a question. it is very much more difficult to devise software which supports 'constructive' learning whereby concepts are internalized by the student and contribute from then onwards to his or her capacity for intuitive problem-solving. (it is also much more difficult to assess this kind of learning.) it follows that there are two key issues for software developers: 1. deciding on the right balance of emphasis between the two kinds of learning (given that one is much easier to achieve than the other and therefore cheaper); 2. once this decision is made, devising computer-mediated experiences capable of supporting the second kind (constructive learning of concepts). 5 bridget somekh designing software t o maximize learning the rest of this paper tries to throw light on the second of these issues. before moving on, however, it may be useful to explore what is meant by 'constructive' learning in a little more depth. macdonald and his colleagues (macdonald et al, 1976) provide some insights from their evaluation of the ndpcal programme, a large-scale initiative funded by the department of education and science in the 1970s. their five-scale typology of the levels of learning via cal contains 'operational definitions', as follows: key features of the types of interaction are: type a interactions: recognition: text-dependent; require matching of superficial features of information presented to information previously presented; non-productive. type b interactions: recall: text-dependent; superficial engagement of student with content; reproductive; combinatorial manipulation of syntactical and logical features of text. type c interactions: reconstructive understanding or comprehension: text-independent but discourse-dependent; involve semantic interaction with content; reconstructive; productive; involve comprehension of statements, concepts, or principles. type d interactions: global reconstructive or 'intuitive' understanding: experiential learning; discourse-independent; focus on structure of discipline; student as master of discipline; problem-solving; conservative of discipline. type e interactions: constructive understanding: student 'creates' fields of knowledge; discipline-independent; exploratory; problemfinding; domain-dependent. in terms of computer-software design, macdonald et al distinguish interactions of types d and e from the lower levels. type c interactions lie on the borderline between the lower levels and higher levels: they represent the lowest level of constructive understanding. in the case of interactions of types d and e, more complex opportunities are provided for the student to demonstrate learning, and evaluation of whether or not learning has occurred will require a far more complex judgement: the correctness or incorrectness of the response cannot be decided in terms of a simple discrepancy between response and text. thus, in these two types of interaction, the student demonstrates learning by much more complex acts of meaning production than in types a, b and c. research in artificial intelligence has focused on the attempt to design computer-user interactions at levels d and e. some progress has been made, but the problems remain rather intractable. for one thing, such interactions tend to require that the software has the ability to accept open-ended responses in natural language and respond to them appropriately. some of the most interesting al software has been developed to have the capacity to learn the user's working style and level of prior knowledge, in order to tailor the interactions to suit better the user's needs. work of this kind has been incorporated in intelligent tutoring systems, but the ability of such systems to promote constructive learning is not always clear, and their production costs are very high. 6 aa-j volume 4 number 3 the macdonald typology provides a useful monitoring device which a development team might use during the design stage. there are two ways in which software developers can focus their energies on ensuring level c 'constructive' learning interactions, and may be able to move towards promoting the higher d and e levels: 1. linking on-screen interactions with learning away from the computer. both in ndpcal and in more recent software development research (see harding, 1974; 1993), one fruitful way of extending cal interactions to level c and beyond has been by linking them with activities undertaken away from the computer. 2. use of simulations (using technological features such as graphical user-interfaces, high-resolution graphics and hypertext navigation). , macdonald et al point out that simulations have the potential to stimulate a student's thinking at level d by 'presentpng] the student with inferences from his [sic] responses, and thus creating] cognitive disequilibrium' (p. 12). mind maps or mental schema the theory behind constructivist learning is that each individual develops mental schema or 'mind maps' which serve to inform future thinking or action. these schema are fundamental to the way we understand all experience. as babies we begin to build up schema which enable us to distinguish a human face from its background. more abstract conceptualization involves the same process of constructing a meaning and pattern from a jumble of sensory information. these schema then enable us to function with confidence in a complex environment. as bruner puts it (1966, p.2): 'much of perception involves going beyond the information given through reliance on a model of the world or events that makes possible interpolation, extrapolation, and prediction.' effective learning depends on the creation of new schema, or on existing schema being revised, extended or reconstructed. the mind-maps theory has two implications for software development: 1. in order to learn, individuals need to be able to build on their existing schema. this points to a need for individualizing learning. learners need to exercise control over their learning so that its pacing and direction are guided by questions arising from their developing understanding. at a superficial level this can be achieved by giving the user control over what to learn and in what order by offering choices. since icon buttons have replaced the rather cumbersome system of layers of menus, it is now much easier to make this process appear genuinely exploratory and attractive to the user. however, at a deeper level it is arguable that software cannot address this need unless it contains the ability to 'model the user' (see above in the short discussion of artificial intelligence). neither the language, nor the complexity of the investigation of the concepts, nor assumptions about the learner's prior knowledge, can be adapted to the learner by offering choices in this way. a 'pre-test' which will act to select appropriate material for a particular user may be the best way of attempting to address this problem. 2. in order to learn, individuals need support in constructing new schema. 7 bridget somewi designing software to maximize learning this suggests the need for structure underpinning the software design. navigation is the centrally most important issue in developing software in a hypertext environment (laurillard, 1987). learners build their mental schema on 'where they go' in the software and 'what they do'. structure is largely a matter of sequencing the presentation of information or concepts, identifying sections, establishing links between sections and setting up a system for reviewing the structure both as a whole and in parts. a book provides such a structure (more or less well) through its table of contents, divisions into chapters, headings and sub-headings, etc. (an indication that we use the structure of text in a book to help develop mental schema is that we are often able to visualize the layout of a page when wishing to check on information.) in software development, structure is created for the learner through the approach taken to the key navigation issues of usercontrol and information sequencing. plowman (1992) has identified a problem with interactive multimedia that the learner's control over the 'path of disclosure' obscures the underlying structure which the authors have given the material. to some extent, therefore, there is a conflict between the simple solution of giving users choice as a means of individualizing their learning experience as appropriate to fit their existing schema, and the need to provide them with sufficient structure to enable them to adapt this as the basis of new schema. in practice, there is a need for a balance: there needs to be structure and choice, but the latter must be restricted as far as this is necessary to prevent the former from being unhelpfully obscured. an aid in constructing mind maps is to have advance knowledge of the ground to be covered (what ausubel and robinson, 1969, p.145, call 'an advance organizer'). there is then much greater likelihood that the individual can develop meaningful schema which link, where possible, with their existing schema. students are sometimes given an 'advanced organizer' in the form of an introductory lecture to a course or a set of learning objectives. in the same way, hypermedia or multimedia software can be used to present graphical overviews to serve as advance organizers. however, the process of developing schema is never mechanistic. providing an advance organizer, allowed by a series of learning experiences designed to cover each aspect of the conceptual field in turn, will not be sufficient. learning does not proceed incrementally along a linear path of increasing difficulty. opportunities for revision are important because it is unreasonable to expect students to remember everything they have learned on a future occasion. they need to revisit concepts, perhaps many times, until they are securely laid down in well developed mental schema capable of enabling intuitive decision-making. bruner (1960, pp. 52-4) addresses this need through his design for a 'spiral curriculum' in which learners are introduced to a series of concepts, each one in a number of different ways, over a period of time (specifically through different kinds of content involving slightly different approaches). this principle may be a relatively easy one to adopt in software design, although in practice designers usually rely on students using the software many times without variation and with the risk of monotony. another aid to developing schema which software can incorporate is a buzan-style onscreen note-making facility. this allows students to record the web of linkages which reflect the meaning they are constructing from using the software. given the facility continuously to extend and adapt this note-web, students can record their developing 8 alt-j volume 4 number 3 mental schema of the material, with the option of adding hand-written notes to the printout at a later stage. ' scaffolding learning for vygotsky and bruner is essentially a social phenomenon. it depends to a considerable extent on the ability to use cognitive tools such as language and other symbol systems. in vygotsky's theory, learners have limits to their current level of achievement, beyond which they are unable to go without the interventions of a teacher or peer. with the help of such interventions, learners can go beyond these limits into their 'zone of proximal development' (vygotsky, 1986, p. 187), and such experiences serve to extend their understanding so that next time they may be able to achieve this level of understanding without such mental and social 'scaffolding'. it is claimed that interactive software can provide a kind of mental scaffolding. at the lowest level, it can provide the incentive to continue to work on-task for longer periods without the intervention of a teacher. this may be little more than the plate-spinning phenomenon, whereby the student gets feedback on progress and is set a new task (although the mere setting and answering of questions may not achieve even this low-level effect because of its dullness). as a result of this recognition of the social nature of learning, many writers place considerable emphasis on the importance of discussion in facilitating learning. partly in recognition of this, and partly because shortage of hardware has dictated it, computer use in uk schools has become largely a group activity since the advent of the microcomputer in the early 1980s. many university students will therefore be used to this way of working which does appear to have many advantages. if software is designed for use by students without peer group support, it is important for development teams to bear in mind that interaction between learner and software must be livelier than it might otherwise need to be, because it is replacing, and not supplementing, learner-teacher or learner-learner interactions. situated learning, simulations and microworlds recently many researchers have placed emphasis on the need for learning to be 'situated' to enable the construction of schema (brown et al, 1989). this theory suggests that much learning is made more difficult because of the context of learning i.e. ideas have to be grasped through an abstract representation, part of which may be more to do with the context of learning (e.g. copying notes from the blackboard before the lecturer rubs them off) than the context of application. in contrast, when it is situated, the meaning of what is being learned is directly supported by the context of learning. an example might be learning physics in a research laboratory. some software provides an element of situated learning because it simulates an authentic situation, or alternatively provides a microworld in which the learner's experience is authentic within the frame of an alternative environment (for an early definition see papert, 1980). everything in the simulation or microworld is contained in a computer model which has an integrated set of rules and provides an integrated experience. in that 9 bridget somekh designing software t o maximize learning sense the metaphor and content are fused. sometimes the microworld involves the user in learning the microworld's language (e.g. natural language and syntax which the software accepts). always it involves the user learning the limits of the model. the metaphors which control content, simulation, interface and programming structure all contribute to the total effect of the microworld. nevertheless, there must always be a limit to the authenticity of computer-mediated experience. computer simulations by their very nature are presented through a computer interface. in terms of the theory of situated learning they may be very unhelpful if they necessitate the user learning too complex a set of rules governing the computer context of the simulation and unrelated to the simulated 'authentic' context. one of the most important features of situated learning is the idea of learning from expert colleagues who model appropriate behaviours and approaches to problems. if software has a strong emphasis on problem-solving and exploration, there is little room for expert modelling. the learner is on his or her own. this can be overcome by including some examples of problem-solution by posing problems and offering the option for a step-bystep solution with explanations, including graphical displays. this has been tried with some success by harding (1994, p.78). the danger of offering expert models, however, is that the user may learn formulaic responses instead of understanding the concepts. another important feature of situated learning is that what is learned can be applied immediately in a real situation to solve a problem. apart from the reservation about the intervening computer interface, this is where simulation software can be particularly useful. meta-cognition since 1983, salamon (1992) has used the construct aime (amount of invested mental effort), or 'mindfulness', in his work on learning. when writing about the use of computers to support students' self-study he says: while the open-endedness of tools is an opportunity for the awakening of mindful engagement and opportunity to sustain it, one has already to be mindfully inclined to take these opportunities. if young writers would seriously and intentionally attend to the expert-like guidance of the writing partner, or to the way variables affect each other in the systemic models of stella, they'd learn quite a bit. but, alas, only the already mindfully inclined, the ones who find thinking a pleasant challenge and who do not avoid the expenditure of mental effort, are the ones to intentionally expend the effort needed to notice the guidance or the inconsistency in the model's structure, (p. 13) at the lowest level, mindfulness is dependent upon motivation. but mindfulness is essentially a conscious cognitive activity which can be developed by means of critical selfreflection on oneself as a learner. for example, it is quite possible for an individual to have established habits of passive learning (acquired from prior training) so that he or she is unaware of the need to construct knowledge through interrogating ideas and trying to develop mental schema. becoming aware of this is the first step for these individuals in becoming mindful, and therefore more effective, learners. this has led writers like de corte (1990, p.73) to see a role for educational software in 'enhancing the acquisition of 10 alt-] volume 4 number 3 meta-cognitive skills and learning strategies through explication of, and reflection on [the learner's] knowledge (deficiencies and misconceptions versus strengths) as well as on their thinking methods and learning activities (powerful versus weak)'. it is fairly easy to build in explicit strategies to enhance meta-cognition. for example, pretests can include some element of analysis of an individual's learning style which can be fed back to the user. another low-cost solution is to provide an on-screen note-book of the type mentioned above for recording meta-cognitive reflections. this can be very effective if the student is subsequently required to use the notes in discussion with other students or the lecturer. motivation motivation is essential to learning. bruner (1966) points out that human beings are the only animals which depend on passing learning on from one generation to the next. we learn for a future purpose and not simply as a response to natural curiosity. this necessitates a large element of delayed gratification in learning, so that we cannot depend on natural curiosity alone to motivate learners. one common strategy for software developers is to present the user with a microworld which appeals to the imagination in some way as well as providing a simulated context for a range of human-computer interactions. now that the technology is becoming more advanced, there is increasing scope for microworlds to be delightful and absorbing and, at least initially, to have a strong impact on user-motivation. to move a student from initial motivation to sustained engagement with the task is difficult unless the student has already developed a capacity for sustained engagement. salamon's 'mindfulness', which bruner calls 'the will to learn', originates for most of us in a track-record of enjoyable and successful learning experiences. natural curiosity has in this way been nurtured to produce intrinsic motivation capable of sustaining our interest over a long period of delayed gratification. motivation is heavily dependent on' the learner's prior experience and attitudes. students are likely to bring with them a range of emotional blocks which undermine their confidence, reduce their ability to remain ontask for any length of time, and in this way obstruct their learning (what pirsig, 1974, pp.298-306, calls 'internal gumption traps' or 'value traps which block affective understanding'). some of these may relate directly to the computer itself as the context of learning (in this sense no computer-mediated learning can ever be 'situated'). many students may have had negative experiences of computer use, and others may perceive themselves as non-technology people so that using a computer threatens their own sense of self-worth grounded in their self-image (somekh and davis, forthcoming). software developers can try to overcome this by designing microworlds which counteract any hard technological image, but this will never be wholly satisfactory because microworlds carry with them their own social and value sets which are unlikely to be in tune with all users or even whole categories of users. 'flow' or cognitive engagement one of the great advantages of books, films, lectures, or any form of presentation which is uninterrupted, is that a learner's mindfulness can change gear into an intense level of ii bridget somekh designing software t o maximize learning concentration known sometimes as 'flow' (csikszentmihalyi, 1982), and sometimes as 'cognitive engagement' (kozma, 1991). in this state, the learner may lose all sense of the context of learning (time and place), the pace of learning accelerates, and there is greatly enhanced motivation which spills over into a sense of excitement and continuing reflection after the event. turkle (1984) gives many examples of how computers can induce this kind of intense concentration. in some of these, it is associated with the development of repetitive psycho-motor skills, employed under time pressure; in others, it is associated with the intellectual excitement of high-level cognition and creativity resulting from programming in non-linear languages such as logo. however, computer use can have very different consequences. for example, interactive software is unlikely to induce flow if the learner's thought processes are interrupted too frequently by the need to carry out tasks imposed by the computer. translation (or transformation) between symbol systems one of the main problems in any learning is the need to 'translate' concepts from a symbol system, such as language or numbers or graphical representations/models (bruner, 1966), into meaningful mental schema. these symbol systems are the basic tools of human intelligence our ability to use them is what differentiates us from other animals. when learners are asked to imbibe concepts from a textual explanation, they have to make a translation of that explanation in terms of their own experience, based on clues provided by the writer. the writer has attempted to encode a concept in a symbol system (text), and now the reader attempts to decode it and link it to his or her existing mental schema. content screens can make heavy demands of this kind on the learner. one of the most difficult problems facing software designers is how to get across sufficient content to enable learners to engage in problem-solving (dublin, 1988). until comparatively recently there has been an assumption, originating presumably in the poor quality of vdus, that the computer is not good at handling large quantities of text. but concepts which are linguistically constructed may be extremely difficult to impart in any other form. a related problem, which i have already touched on, is how to get across sufficient information about a microworld or the rules of a simulation to enable the learner to function intuitively within them. often such information is given in text form in a set of preliminary screens. with high-quality vdus and graphical environments, and the sudden increase in screen-displayed texts in cd-rom and multimedia, it is no longer necessarily true that computer software should be designed with a minimum of text. however, text does still pose a problem if it is to be incorporated on the screen alongside graphics, or if the software is designed for use by a group which will necessitate a larger font size. the need to reduce the number of words on the screen can lead to truncated text which does not present the concepts adequately. if the main focus of the software is to be some form of investigation or the exploration of a model, and if the concepts can only be delivered by means of considerable quantities of textual explanation, it may be best to produce printed materials to go alongside the software. the more interactive the software becomes, the more it is possible to allow learners to learn by exploring models, or building models, rather than by digesting text. however, designing software that supports this kind of learning requires a high level of creativity 12 alt-j volume 4 number 3 and painstaking planning. moreover, there are great differences between individual learners when it comes to the relative difficulty they experience in translating from different symbol systems. some learners will find it easier to translate from text to their mental schema than from graphical models to their mental schema. the problem of such translation for learners also includes 'transforming' meaning from one symbol system to another. a typical transformation problem for learners is the interpretation of graphs (mokros and tinker, 1987; smith, 1993). these are usually presented alongside a textual explanation. a graphical representation can communicate a range of detailed information in easily assimilable form, but its interpretation requires a sound understanding of the underlying concepts, e.g. of the likely relationships between the different kinds of information displayed in the graph. there is a marked tendency for teachers (and the producers of textbooks and educational materials) greatly to underestimate the difficulties that novices have with the interpretation of graphs. it is assumed that graphical representation will clarify and explain textual representation. often it simply overlays it with a second level of difficulty. a common error, which indicates the problem of transformation, is the novice's tendency to 'read' the graph as a picture (i.e. to see points plotted at the top of the page as more likely than those at the bottom of the page to signal a large quantity of something). many writers are agreed that the computer offers particular advantages in being able to display graphs dynamically so that the learner can either explore the relationships between the variables by experimentation, or be provided with a dynamic model of the transformation between text and graph (kozma, 1991); i.e. the user changes the text and the computer transforms this into changes in the graph. software can also be used to 'proceduralize' understanding by demonstrating how a problem can be investigated (and solved?) step by step, according to clearly defined rules (kozma, 1991). feedback and assessment there are always considerable problems in assessing learning adequately because cognitive processes are private, hidden and unique to the individual. these difficulties are compounded by the need to integrate assessment within a software package, because this usually precludes the possibility of the student giving natural-language responses to questions. yet, some element of feed-back or self-assessment is essential to the learning process, and both students and tutors tend to expect computers to provide assessment which records the student's progress and obviates the need to assess the work in some other way. > the main problem of computer-mediated assessment is that it is difficult to prevent testing from emphasizing the lowest a and b levels of the macdonald typology of interactions recognition and recall at the expense of even the c level of reconstructive understanding, simply because the former are so much easier to test than the latter. (an additional problem is that as soon as testing enters the realm of understanding of concepts there is likely to be difference of opinion among lecturers as to the 'correct' answer.) the need to provide a formal assessment option or facility for tracking students' progress for the benefit of providing lecturers with a record at a later stage has the danger of placing too high a constraint on the educational design of the software. it necessitates 13 bridget somekh designing software t o maximize learning the need for computer assessment of higher levels of learning. such assessment demands considerable sophistication in software design and programming. deciding against formal assessment is risky, however, given the deep acculturation of both university students and lecturers into study as an almost wholly assessmentorientated activity. there is the risk that students will view the software as an optional extra if it is not seen as-an essential part of the 'exchange of effort for grades' (doyle, 1979) which keeps them motivated and on-task. there is also the risk that lecturers will not recommend the software to students strongly because they know they will have no way of checking whether or not students have used it, and will therefore not be able to rely on the software as a substitute for any of their existing teaching. if bsh is to be the main performance indicator for software developers, project teams need to adopt a strategy to safeguard against this. one possibility is to produce supplementary, paperbased assessment materials to be used after completion of the modules. a note on the context of use teaching and learning in both schools and universities are highly routinized activities. lecturers and students expect to behave in certain ways (e.g. working at networked terminals in a computer room), and the structures of institutions are generally supportive of these routines (e.g. money is likely to be allocated to pay specialists to maintain the computer network and to purchase software by licence so that it can run on the network rather than on individual laptops). software which is designed to suit a different institutional context (e.g. demanding a more sophisticated network, or intended to be used on stand-alone computers as a more integral part of subject study in the science laboratory) poses challenges to these routine behaviours and institutional structures, and as a result it is necessary to provide very explicit guidance for users, and give considerable prior warning before asking individuals to undertake trialling. to guard against these kinds of problem, courseware should probably be designed so that it can be used in more than one kind of learning context. note 1 a version of this paper was presented at the conference of the association for learning technology (alt-c 94), university of hull, september 1994. references ausubel, d. p. and robinson, f. g. (1969), school learning: an introduction to educational psychology, london and new york: holt, rinehart and winston. brown, j. s., collins, a. and duguid, p. (1989), 'situated cognition and the culture of learning', educational researcher, january-february, 32-42. bruner, j. s. (1960), the process of education, cambridge ma: harvard university press. bruner, j. s. (1966), towards a theory of instruction, cambridge ma and london: the belknap press of harvard university press. 14 alt-] volume 4 number 3 csikszentmihalyi, m. (1982), 'towards a psychology of optimal experience', in wheeler, l. (ed.), review of personality and social psychology, volume 2, beverley hills ca: sage, pp. 13-36. de corte, e. (1990), 'learning with new information technologies in schools: perspectives from the psychology of learning and instruction', journal of computer assisted learning, 6, 69-87. desforges, c. (1989), 'understanding learning for teaching', westminster studies in education, 12, 17-29. doyle, w. (1979), 'classroom tasks and student abilities', in peterson, p. l. and walberg, h. j. (eds.), research on teaching: concepts, findings and implications, berkeley, ca: national society for the study of education, mccutchan, pp. 183-209. dublin, m. (1988), 'individualized instruction on the computer: the meaning behind the myth', interchange, 19 (2), 15-24. harding, r. d. (1974), 'computer-aided teaching of applied mathematics', international journal of mathematical education in science and technology, 5, 447-55. harding, r. d. (1993), 'computer interactive texts', in proceedings of the fourth international conference on technology in collegiate mathematics, reading, ma: addison-wesley. harding, r. d. (1994). 'nuffield interactive mathematics courseware'. computers and education, 22 (1/2), 73-82. kozma, r. b. (1991), 'learning with media', review of educational research, 61 (2), 179-211. laurillard, d. (1987), 'computers and the emancipation of students: giving control to the learner', instructional science, 16, 3-18. macdonald, b. and the uncal team members (1976), 'the educational potential of computer-assisted learning: qualitative evidence about student learning', paper prepared by uncal, the independent educational evaluation of the ndpcal, care, uea. mokros, j. and tinker, r. (1987), 'the impact of microcomputer-based labs on children's ability to interpret graphs', journal of research in science teaching, 24 (4), 369-83. papert, s. (1980), 'microworlds: incubators for knowledge', pp. 120-34 in mindstorms, brighton: the harvester press. pirsig, r. (1974), zen and the art of motorcycle maintenance, london: bodley head; reprinted london: corgi/transworld. plowman, l. (1992), 'an ethnographic approach to analysing navigation and task structure in interactive multimedia: some design issues for group use', in monk, a., diaper, d. and harrison, m. d. (eds.), people and computers vii: proceedings of hci 92, cambridge: cup. prawat, r. s. (1991), 'the value of ideas: the immersion approach to the development of thinking', educational researcher, 20 (2), 3-10. 15 bridget sometti designing software to maximize learning salamon, g. (1992), 'computer's first decade: golem, camelot, or the promised land?', invited address to division c, aera conference, san francisco, april 1992. smith, h. (1993), the use of computerized databases in critical inquiry by pupils aged 8 to 11 years, d. phil, thesis, university of sussex. somekh, b. (1989), 'the human interface: hidden issues in computer-mediated communication affecting use in schools', in mason, r. and kaye, a. (eds.), mindweave, oxford: pergamon press, pp. 242-6; reprinted in boyd-barrett, o. and scanlon, e., computers and learning, addison-wesley in association with the open university, 1991. somekh, b. and davis, n. (forthcoming), 'getting teachers started with it and transferable skills', in somekh, b. and davis, n. (eds.) (forthcoming 1997), using it effectively in teaching and learning: studies in pre-service and in-service teacher education, london, new york and toronto: routledge. turkle, s. (1984), the second self: computers and the human spirit, london, toronto and new york: granada. vygotsky, l. (1986), thought and language, cambridge ma: mit press. 16 approaches to knowledge-transfer systems oleg r pilipenko* and helena komissarova** * odessa state university, ukraine ** kiev institute of psychology, ukraine the use of computers for knowledge transfer does not result automatically in success: the efficiency of the process depends on the approach to instruction underlying the knowledge-transfer system in question. in this paper, the authors describe and compare computer technologies used for instruction in computer-aided instruction (cai) courses, electronic books, and knowledge-transfer systems based on heterogeneous information. they also deal with some psychological issues involved in dialogue, with the adaptability of knowledge-transfer systems, and with psychologically-based computerized instruction. introduction the use of computer technology in instruction is based on known approaches to instruction, and also promotes new approaches. contemporary problems in instruction tend to be determined by intensive information-flow and the short timescale of the relevance of information. computer-aided instruction courses historically, the primary method of teaching is the so-called 'nurnberg funnel' method. this is based on the assumption that knowledge in the teacher's head is linked to some 'correct' structure, and the objective of the teaching is to place facts and relations between them into the heads of a student with minimal distortion. obviously, this is a simplified approach to teaching, which nevertheless, under certain conditions, provides reasonably good results. however, its effectiveness depends heavily on the personality of the teacher, and it significantly underestimates the student's personality, thus decreasing the possibilities of creative learning. the introduction of the personal computer caused an exponential growth in the number of knowledge-transfer systems based on this approach, and the possibility of widely disseminating an outstanding teacher seemed to be one of its major advantages. 42 aa-j volume 3 number 2 knowledge-transfer systems of this kind are built as restricted models of a human teacher. traditionally, the teacher determines objectives and the means of instruction, makes decisions about the level of interaction, manages time, and evaluates the student's activity and performance. so, in human-computer dialogue which simulates the dialogue between a teacher and a student, the student is mainly a passive partner and the computer is an active, leading partner. this model often assumes that the learner knows almost nothing about the domain in question, nor about the ways to study it. under these conditions, the computer imposes the sequence in which the domain is to be studied, and thus determines the skeleton of the knowledge-transfer system. knowledge-transfer systems of this kind are often called computer-aided instruction (cai) courses. two major obstacles are encountered in implementing and using cai courses. many people use computers as tools in their everyday work, where the computer is considered a passive partner in the human-computer interaction. for such people, a contradiction appears when it comes to computer-aided learning: the computer which is thought of as a tool and a slave becomes an instructor and a master. under such conditions, people may feel uncomfortable because their role in the dialogue, now controlled by a computer, has changed (kommisarova, 1994). such discomfort may adversely affect students' motivation, as well their emotional state, and may decrease the effectiveness of computeraided learning. the second obstacle is that since learning depends so heavily on the personality of the teacher, it makes sense for one author (or a small group of authors) to design a cai course. this in itself is a constraint. electronic books according to current educational theory, direct transfer of knowledge from teacher to student is impossible. learning is a process involving the juxtaposition of, on the one hand, new facts and theories, and on the other, facts and theories already formed in the student's mind, a process which may result in augmenting or restructuring the latter. instruction within higher education relies heavily on such a process, and thus independent study, in which students can adapt the procedures associated with knowledge acquisition to their own cognitive processes, is encouraged. this approach is also effectively supported by modern technology in the form of electronic books which can be used to deliver information not only as text but also as images, sound and animation (see barker, 1991), with hypermedia links. knowledge-transfer systems of the above-mentioned type use a different model of human-computer interaction. here, the computer is considered to be a tool which immediately appears comfortable to use although the activity may be new. but there is a clear difference between using the computer as a tool in an everyday activity and as a tool for learning. the field to be studied and its internal structure may be terra incognita for the user. tables of contents and indexes do little to help a student who does not know the relevant terminology, nor how to begin. the student needs guidelines for an efficient acquisition of knowledge, and additional information about the logical structure of the knowledge-transfer system, as well as about the logical structure of the field in general and the potential links between information 'frames' within the knowledge-transfer system. the feeling of being an independent explorer of a knowledge-domain can increase 43 oleg p. pilipenko and helena komissamva approaches t o knowledge-transfer systems the motivation of a student, but he or she can easily be distracted and simply go off course (barker, 1990). this can prevent effective knowledge transfer and evoke a negative emotional attitude to the computer, or at least to computer-assisted learning. we believe that any knowledge-transfer system should incorporate both the knowledge and the techniques needed to use it. the teacher's knowledge about the sequence in which a student should ideally approach the information is of particular importance, and the student also needs to know why certain parts of it are more important than others. thus categorization of the links between information frames should be an essential feature of the above-mentioned kinds of knowledge-transfer systems. for example, we may specify links which describe a recommended way to learn the proposed material, or several different ways recommended by different teachers, or determined by different learning objectives. links to information frames relevant to the study of the current frame, and links between frames exploiting the same methodological approach, might also be assigned to particular categories of links. adding power to a knowledge-transfer system by means of dialogue dialogue can be used as a method of access to data contained in knowledge-transfer system. the dialogue scenario should generate information about why and how one should use the system. some authors of electronic books offer a sequence of questions and answers. in such cases, the roles of the participants are strictly determined: one is the active partner, the other is the passive partner, though this situation can be reversed the answering (passive) partner can momentarily take the initiative if he or she does not have enough information to answer the question. the answering partner may also take the initiative when the asking (active) partner fails to put the question in a way that is clearly understandable by the answering partner, and the latter may need to ask a series of questions in order to determine the precise demands of the asking partner. the changing of roles in partner dialogues of the kind under consideration here are discussed in more detail in pilipenko (1994). we believe that a reasonable balance between activeness and passiveness in such dialogues is vital for effective learning to take place when using knowledge-transfer systems. at each step of the human-computer dialogue, the system should provide the user with all necessary information, including information about its own built-in functionality. students feel comfortable if they feel they are in control, and if they can predict the overall behaviour of the knowledge-transfer system. knowledge-transfer systems delivering heterogeneous information electronic books lend themselves well to situations where information from different sources needs to be presented in a homogeneous form, though it is then important to explain the connections and differences between the viewpoints, and perhaps the terminology, adopted by different authors. however, a particularly interesting trend, based on the idea of the usage of heterogeneous information in knowledge-transfer systems, is that of so-called open hypermedia systems. a significant development related 44 volume 3 number 2 to this approach is the microcosm system developed at southampton university (davis et al, 1992). the microcosm environment allows information frames created with different applications to be incorporated into one system. markup information, which is used to designate links between information frames, is physically separated from corresponding frames. thus the documents included in the knowledge-transfer system are always available for modification within the applications which were used to create them. computer-based knowledge-transfer systems can thus be modified dynamically, and easily extended not only with new information frames, but also with new links. knowledge-transfer systems based on this approach can support several methods of structuring information simultaneously, and along with heterogeneous information frames, can contain a heterogeneous system of links, for example recommendations as to how and why to access information frames composed by different teachers or intended for students with different backgrounds or different objectives. learning supported with knowledge-transfer systems based on open hypermedia can also be underpinned by dynamic modifications made by a teacher while using the system, as he or she finds new links which can then be added: learning by doing is thus especially suited to such systems. new links, and possibly new information frames, can be created as a direct result of the search for information relevant to the problem in hand. shared computer books another way to use the unique potential of dynamically modifiable knowledge transfersystems lies in the idea of shared electronic books, that is, made available to several users simultaneously. shared electronic books can be open not only for joint access, but also for modification by all users. users can add new facts, new ways of handling facts already contained in the systems, and comments. a shared electronic book can become the joint creative output of many people. the potential of the world wide web is also interesting in terms of the development of shared electronic books. the most interesting feature of the world wide web is that there is no centralized maintenance or support. it is the result of the creativity of many people. on the other hand, the very absence of centralized support means that the web cannot perform the role of active partner in a human-computer dialogue: the information contained in the web is poorly structured (and searching for information can at present take an unacceptably long time). nevertheless, we believe that information has the strange ability to be self-structuring, a property which can manifest itself both in the intentional actions of a co-ordinated group of people, and in the chaotic activity of different people who do not even know of the existence of each other. we therefore expect that the increasing amount of information available on the web will not lead to its collapse, as many would argue, but rather to an improvement of its structure. adaptability of knowledge-transfer systems knowledge-transfer systems should ideally be able to adapt to student needs, including taking into, account different personal styles of knowledge acquisition. knowledgetransfer systems should also keep track of the dialogue between user and computer, and memorize which information frames have been accessed by the student, as well as 45 o/eg p. pilipenko and helena komissarova approaches to knowledge-transfer systems evaluating the student's progress and level of comprehension of the material placed in the system. monitoring dialogue traces is particularly important, since the student who is just starting to tackle a new subject can then have access to the ways in which it has been tackled by other students. in addition, the teacher (or group of teachers) responsible for the maintenance of the knowledge-transfer system can make appropriate corrections to the dialogue scenario if most the students are not using it efficiently (at critical points students may have to be guided more actively). conclusion the ultimate degree of adaptability in instruction is complete individualization. to achieve this, a computer-based knowledge-transfer system would have to take into account not only the intellectual background and progress of a student, nor only errors and general and specific performance, but it would also have to adopt a style of instruction precisely fitting the learning style of the student, including his or her cognitive and personality/psychological features such as holism/serialism, impulsivity/reflexivity, flexibility/rigidness, verbal/image thinking, introversion/extroversion, field-dependence, level of anxiety, and so forth. such a task would require a thorough probing of a student's personal characteristics with a series of psychological tests, and careful monitoring of his or her learning activity and performance. we do not foresee such knowledge-transfer systems in the near future. at present, only the first of the four accepted divisions of computer-based instructional systems (domain or field knowledge-base, pedagogical module, student module, and interface) have been developed to a satisfactory degree. theoretical and technical progress in interface construction is significantly poorer. the same applies to the pedagogical division, and the problems of student-module design are not even yet solved in principle. nevertheless, while waiting for progress in these areas, we should pay as much attention as possible when designing knowledge-transfer systems to individualization, and to the issues related to the dialogue between human being and computer. editorial note this paper was given at a nato asi (advanced studies institute) held in heriot-watt university in the summer of 1994. acknowledgements we should like to thank professor ray mcaleese and the british council for their assistance and support. references kommisarova, h. (1994), 'psychological peculiarities of man-machine communication in instructional systems' in: brouwer-janse, m.d. and t. l. harrington t.l. (eds), humanmachine communication for educational systems design, nato asi series, vol. f129, berlin, springer-verlag. 46 aa-j volume 3 number 2 barker, p. (1990) 'designing interactive learning systems', educational and training technology international, 27 (2). barker, p. (1991) 'electronic books', educational and training technology international, 28 (4). pilipenko, o.p. (1994), 'modeling a partner in a dialogue' in brouwer-janse, m.d. and harrington, t.l. (eds), human-machine communication for educational systems design, nato asi series, vol. f129, berlin, springer-verlag. davis, h.c., hall, w., heath, i., hill, g. and wilkins, r. (1992), 'towards an integrated information environment with open hypermedia systems' in lucarella, d., nanard, j., nanard, m. and paolini, p. (eds), proceedings of the acm conference on hypertext: echt '92, milan, acm. 47 alt-j, research in learning technology vol. 15, no. 1, march 2007, pp. 3–20 issn 0968-7769 (print)/issn 1741-1629 (online)/07/010003–18 © 2007 association for learning technology doi: 10.1080/09687760600837090 the use of electronic voting systems in lectures within business and marketing: a case study of their impact on student learning george masikunas, andreas panayiotidis and linda burke* kingston university, uk taylor and francis ltdcalt_a_183646.sgm10.1080/09687760600837090alt-j research in learning technology0968-7769 (print)/1741-1629 (online)original article2006taylor & francis143000000september 2006lindaburkelinda.burke@kingston.ac.uk this article presents a case study of the impact on student learning of introducing an electronic voting system (evs) into large-group lectures for first-year undergraduate students undertaking degrees in marketing and business systems. we discuss the potential for using evs-style interactive lectures in marketing and business programmes. we then describe how we planned the session and selected and implemented the evs system. we go on to present an evaluative research project, which was undertaken on the innovation using case-study methodology, and assess its impact on student learning. data for the evaluation were collected through questionnaire and focus groups with a sample of students. the data were analysed using thematic analysis. the findings show how students perceived the use of evs in large lectures and how their learning was affected. a ‘threefold typology’ emerged that explains how students related to the evs and how their perceptions of evs changed over time. the discussion links these findings to the literature on different paradigms of learning and teaching, using renshaw’s framework, and examines how the evs-style lectures promote deep and active learning within the constructivist, social constructivist and metacognitive learning paradigms identified in renshaw’s model. the conclusions show how the use of a userfriendly evs in large lectures motivates students, develops students’ cognitive and social learning skills, and improves learning effectiveness. introduction kingston university provides undergraduate degrees in business and marketing. a large numbers of students (500) undertake certain year one modules at the same time, which have traditionally been delivered through lectures. a decision was taken to test the use of an electronic voting system (evs)-style interactive lecture and * corresponding author. faculty of health and social care sciences, kingston university and st. george’s, sir frank lampl building, kingston hill campus, kingston-upon-thames kt27 7lb, uk. 4 g. masikunas et al. powerpointvote system (ppvote) in first-year marketing and business studies modules. this article begins with background information on how we proposed to use evsstyle interactive lectures to address issues raised by student evaluations. we then present the potential benefits of using evs-style lectures, how we believed we could make best use of evs as a tool for improving student learning and how the literature on evs supported our thinking. we go on to discuss how we prepared for and implemented the interactive evs-style lectures and the associated evaluative research to measure the impact on student learning and achievement. in most higher education institutions the lecture is the widely accepted method of knowledge transmission (jarvis, 2002); however, research shows that the traditional style of lecture is the least-favoured method of instruction (butler, 1992; bligh, 2000). bligh (2000) suggests that lectures are mostly useful for imparting information but not for developing powerful minds. this echoes boyer (1990), who suggested that: great teachers create a common ground of intellectual commitment. they stimulate active, not passive, learning and encourage students to be critical, creative thinkers, with the capacity to go on learning … (p. 24) as a teaching team we became aware of a range of problems with traditional lectures for first-year undergraduate students in marketing and business studies. our students were telling us that the traditional lectures were ‘a bit of a turn off’. internal feedback from our students led us to realise that our traditional lectures were not serving their purpose. several problems with the traditional face-to-face lecture format were identified from both students and lecturers, particularly with large classes: ● a general lack of student engagement during lectures, both with the material presented, and with their peers; ● large classes typically do not allow students an opportunity to present their opinions and to receive immediate feedback; and ● there was little opportunity for lecturers to test students’ understanding of material before moving on to new material. we undertook an analysis of our traditional lectures using bloom’s taxonomy of the cognitive domain (bloom, 1956). our students gave high scores to our lectures for bloom’s lower-order cognitive skills (knowledge and comprehension) and low scores for higher-order cognitive skills (application, analysis, synthesis and evaluation). our intention was that a new-style interactive lecture would score reasonably high for both sets of cognitive skills. the potential for using evs-style interactive lectures we decided that, by using evs as the basis for interactive lectures, we wanted to achieve a number of things in terms of enhancing student learning. we wanted to: evs in lectures within business and marketing 5 ● create a stimulating learning environment in the lecture room that would engage students through making them active rather than passive learners; ● enhance students’ overall performance in the year one modules; ● reduce the number of assessment retakes; ● improve retention rates; ● build self-confidence in the students through encouraging participation; and ● stimulate student–student interactions and student–teacher interaction in a way that led to a shared process of knowledge-building. mazur (1997) and dufresne et al. (1996) implemented evs into interactive lectures with the aim of developing students’ cognitive skills. mazur (1997) also wished to foster collaborative learning through integrating peer–student instruction as an integral element of an evs-style interactive lecture. others (for example, eliot, 2001) suggested that using evs in lectures is appropriate in subject disciplines that are perceived to be particularly challenging to student learning; for example, physical sciences, engineering, computing and statistics: … i was aware that it was a course that students have historically found challenging. further, in a lecture environment the students may be unwilling to volunteer information regarding their level of understanding of the material covered. (eliot, 2001, p. 80) boyle and nichol (2003) said that: the aim [of introducing evs-style interactive lectures] was to aid students’ transition into higher education, improve retention, give a better first year experience and introduce active learning and discussion concepts into large classes. (p. 43) lui et al. (2003) used evs to overcome the communicative limitations that large lectures traditionally create and to enable students to construct deeper meaning in terms of the content presented in lectures. others (for example, horowitz, 1988; hake, 1996; abrahamson, 1999) adopted student response systems in lectures with the aim of improving learners’ attention or increasing assessment scores, or to facilitate interaction between students in the lecture (sarah, 2002). hake (1996) concluded that interactive engagement in student learning was preferable to traditional classroom teaching and: … strongly suggests that the classroom use of ie [interactive engagement] methods can increase mechanics-course effectiveness well beyond that obtained by traditional practice. (p. 72) laurillard (1993) presented a persuasive argument for interactive lectures in terms of the essential role that feedback plays in student learning. she suggested that action on the part of the learner without some sort of feedback is unproductive and that receiving meaningful and useful feedback is important for student learning. plan for using evs in our marketing and business modules we proposed to use the evs-style lecture to: 6 g. masikunas et al. ● deliver interactive lectures that incorporated information giving, small-group learning to answer specific questions/complete group learning tasks, and group decision-making; ● use an evs that enabled students to communicate their answers to the group activity and instantly converted students’ responses (in the form of a histogram projected onto a screen at the front of the lecture room); ● use this process to provide formative feedback to all students through which they could judge their own responses against those of others and against the correct response provided by the teacher; and ● improve student concentration in lectures, promote deeper approaches to learning and enable students to develop a wider range of cognitive skills (bloom, 1956; renshaw, 1995). in drawing up our plans we were mindful of some barriers to its success, including potential concerns by students that less material might be covered in interactive evsstyle lectures, a concern that students might focus more on achieving better marks and that students might use the group-activity time in the lecture to socialise rather than complete the activity. selection of the evs system the evs system we selected needed to be congruent with promoting different types of learning and the aims we identified for this project. we wanted interactive evs-style lectures that would appeal to all types of learner styles, but that would especially promote the ‘deep active’ type (marton, 1994). we were mindful, too, of work by brown and brailsford (2004), who suggested that learners are adapters whose learning styles can change according to the learning environment. the evs system we selected would change the learning environment of lectures and needed to help us to bring together interactivity, group problemsolving following freshly taught material, and peer teaching and feedback to students. finally, we also took into account the work of other pioneers in the use of evs systems in teaching and learning (horowitz, 1988; dufresne et al., 1996; mazur, 1997; crouch & mazur, 2001; liu et al., 2003; draper & brown, 2004). introduction of the innovation after comparing different evs systems we elected to use the personal response system supplied by educue for our mass-attended classes. the systems purchase did require certain additional hardware and software conditions to be met. these were satisfied by the information technology provision in the lecture theatres across the university. there is a university-wide network linked to a proprietary virtual learning system, which feeds into classrooms where a data-projection system is run from a stand-alone computer linked to large screens or electronic white boards. this information-technology-enriched classroom met the hardware and software pre-requisite. evs in lectures within business and marketing 7 absence of this would have constituted a major barrier to adoption of the evs-style lecture. preparation for the innovation preparation involved a variety of events, including gaining funding approval, purchasing, staff training in the use of evs, practice sessions in our lecture halls using evs, and embedding dedicated evs-focused slides (with group work questions and possible answers) into the powerpoint presentation. practice sessions allowed us to load the special evs-dedicated software onto the computer in the lecture hall, ensuring that the sensor device could pick up responses from the remote controls used to signal student choice from the answers offered. the commitment involved and timing of lectures became significant elements of the plans for implementing this innovation. time was needed at the beginning of each lecture to set up the evs system in the lecture hall, distribute remote control panels, allow students to form small teams of three to five, undertake the integrated evs-type question and answer sessions, and ensure that the equipment was taken away at the end of the lecture session. we will come back to discuss how the use of time in the lecture changed as a result of moving to evs-style interactive lectures. implementation of the innovation suggestions for sequencing of learning activities in an interactive evs-style lecture vary in the literature. mazur (1997) and dufresne et al. (1996) propose different sequences in an evs-style interactive lecture. the major difference between the two lies in the extent to which it is the individual student who answers the question/ problem posed, or the small group who agrees an answer. mazur (1997) favoured an individual-student approach, while dufresne et al. (1996) proposed a group-wide sequence for providing the answers. nicol and boyle (2003) reported that students in one university found mazur’s peer instruction sequence to be more helpful to their learning in preference to the classroom-wide sequence advocated by dufresne et al. (1996). the two lecturers involved in this project created similar sequences, both of which are based on dufesne et al. (1996), but with some minor adaptations. figure 1 illustrates how evs operates within the following sequence of events in an interactive lecture. figure 1. the evs learning cycle and sequence the evs involves distribution of remote handsets and dividing the large group into small work groups of three to five students. the lecturer then introduces a new piece of module content that has a question/learning task posed, and a set of possible answers. each small group discusses the question/completes the task and, through collaboration, interactivity, group decision-making and discussion, agrees their answer as a group. in this process, students are encouraged to be involved in peer student teaching where individual group members explain their reasoning for selecting a given answer. as part of the discussion, the group evaluates each possible answer from the choices, before agreeing an answer. 8 g. masikunas et al. group answers are submitted by pressing the appropriate button on the remote handset, which is picked up by sensors linked to a computer that projects the results of all the responses. the voting process takes a very short time. a histogram is generated that displays on a screen in the front to the lecture room the overall outcome choices. at this point the lecturer can do a number of things: ● indicate the correct answer and give the reasoning to support it, followed by a classroom-wide discussion allowing team members to support/explain their choice. ● follow-on discussion, but without giving the correct answer, repeat the question allowing every team to answer again, permitting statistical comparison on any changes in voting. ● give the correct answer, with reasoning, followed by a summary. the lecture continues with new taught material, followed by a second evs sequence. the sequence presented in figure 1 can be repeated a number of times during the lecture. evs was first used in the lectures presented by the module leader for the business information systems module. approximately 240 students undertook this ��� ������ �� ��� ����� ����� � �� ����� �� ���������� � �� �� �� �� ����� �� ��� � ���� ������ ���� � �� �� ��� � ��� �������� �� �� �� ��� ��������� ���� ��� �� � ���� ������ ����� � �� ���� ��� � �� � � � ���� �� �� ��� ��� � �������� � ��� !�� ���� � ���� � ��� ���������� "���� �� ��� figure 1. the evs learning cycle and sequence evs in lectures within business and marketing 9 module together. other lecturers observed these sessions and shortly afterwards the evs-style lecture was adopted for the remainder of year one modules undertaken by the same students. the features that impressed the other lecturers, and that led them to adopt this approach, included the students’ response to this new style of lecturing and the way students engaged with each other, with the lecturers and with the course content, during the evs-style lectures. when we implemented the evs-style lecture we became aware that use of the lecture time changed with evs-style lectures. figure 2 shows the typical time allocation used in our traditional-style lectures as compared with the use of lecture time in evs-interactive style lectures. figure 2. typical time allocation for traditional lectures compared with evs-interactive style lectures the evaluation framework our research objectives with respect to student learning were, firstly, to measure students’ perceptions about the use of evs in large-group lectures, and, secondly, to identify and explore the specific factors that increased the effectiveness of learning in an interactive lecture using evs. additionally, we wished to examine students’ attitudes and feelings about evs lectures. we also wanted to examine the extent to which quantitative data (achievement and progression) correlated with qualitative data. a decision was taken to undertake an evaluative research project using case study design (alvarez et al., 1990; brearley, 1993; greenwald, 1991; yin, 1994; carney, 1995; tellis, 1997). selecting an evaluative research framework is important to ensure that an innovation is subject to rigorous scrutiny and measures the extent to which success criteria or proposed outcomes are achieved. this assures that educational ends precede technological means. data collection and analysis a case study as an evaluative research design lends itself to using a number of data collection methods. we focused the evaluative case study on measuring the impact of evs lecture time utilisation introduction 10% conclussion 10% interaction 36% lecture 44% traditional lecture time utilisation introduction 6% lecture 68% interaction 20% conclusion 6% figure 2. typical time allocation for traditional lectures compared with evs-interactive style lectures 10 g. masikunas et al. evs-style lectures on student learning and achievement. data were collected using survey questionnaires, focus groups and quantitative data related to assessment results. we undertook a survey of students that involved the distribution of a questionnaire to those students who attended the marketing and the business information systems modules. these survey questionnaires were distributed to students in week six of their modules. we also undertook focus group interviews with a sample of students to record and analyse their ideas, feelings, attitudes and perceptions about their evs experience and its impact on their learning. quantitative data about assessment results were obtained from minutes of examination board meetings. the survey questionnaire yielded both qualitative and quantitative data. the questionnaires and focus groups were analysed using thematic analysis (aronson, 1994). thematic analysis involved identifying patterns in the data and classifying data according to these, finding themes from within these and then determining which of these themes were key themes; that is, those that form a comprehensive picture of the collective experience of the stakeholders. in addition to thematic analysis of the qualitative data, we used the quantitative data about student performance in assessments to ascertain any differences in assessment marks for these students, compared with previous first-year students who undertook these modules through traditional lectures. findings the response rate for the survey questionnaire was 40% (n = 165). analysis of the survey questionnaires indicated that students perceived the effectiveness of their learning was improved through the evs lectures (mean response rate, 4.25 out of 5). thematic analysis of the survey questionnaires enabled us to identify the following themes that contributed to this improved learning: ● enjoyment of the evs-style lectures. ● consultation/participation. ● reinforcement feedback. ● team decision-making. ● student interactivity. ● student perceptions of the role of lectures. data from the focus groups confirmed that these were the key themes related to student learning and achievement. each of these themes are now discussed in turn. enjoyment of the evs-style lectures was a recurring theme from the survey questionnaires and focus groups. the majority of students expressed a high degree of satisfaction with the evs-style lectures. analysis of the survey questionnaires gave an aggregate score of 4.7 out of 5 for the question on enjoyment. lectures have come alive. before, in most classes, i think most students were not listening. some people sitting next to you did not want to know. evs in lectures within business and marketing 11 consultation/ participation was identified as a factor in improving learning, with a mean score of 4.35 out of 5. three types of consultation/participation emerged from the focus groups as having a positive impact on learning. students having the opportunity: ● to be active in the large group lecture; ● to discuss problems with others in their small group; and ● to discuss issues with the lecturer within the lecture, thus opening up a new channel of communication between lecturer and student and instigating a change in the student–lecturer relationship. reinforcement feedback as part of the learning process during the evs lectures emerged as another theme (mean score, 4.45). in one of several responses about reinforcement and feedback, a student commented that when the lecturer gave the correct answer after the evs activity, it was an extension of: … revising my understanding and … a function of learning reinforcement. what this seemed to say was that feedback and reinforcement from lecturers helped this student to re-think his understanding of the content and reinforced learning. students viewed the evs-style lecture as a means of providing them with feedback about newly taught material, and some came to see this as confirming or challenging the meaning they had constructed and derived. where their response was ‘correct’ they saw this as providing positive reinforcement, whereas ‘wrong responses’ helped to clarify a taught concept so that their state of knowledge was ‘on track’, which counts as another form of ‘reinforcement’. feedback and reinforcement within the evs lectures was very important to most students, and helped to develop their confidence, as the following quotes illustrate: further discussion or re-explanation was given to bridge the learning gap. it motivates to get feedback about what you learned. it gives us confidence, if the answer is right … and if you answer wrong you will see the correct answer later on and there’s a bigger chance for you to remember that answer. team decision-making was another emerging theme (mean score, 4.6) that had a positive impact on students’ perceptions of their learning. a number of reasons for this were identified by students in the questionnaires and focus groups. team decision-making gave students opportunities to look at different options for problemsolving and decision-making. … you get to know more about the other options … … you would think that this option would be right, but you think on the other option as well … team decision-making was seen as positive because of the opportunities it gave students to find out why each student believed certain answers to be correct. … you can ask a person in your group who has given the ‘a’ answer, ‘how do you come to the answer a’ … 12 g. masikunas et al. another recurring response was the positive effect of learning from others in groups (peer instruction). a third recurring response was that of developing self-confidence through joint decision-making. it [joint decision-making] builds your confidence. a small minority of students said that they would have preferred to have been given individual handsets so that they could work through the questions on their own rather than within a small group. this response tended to come from the earliest focus group. the issue of some students preferring to work independently rather than as part of a team did not appear in later focus groups. this may be due to the fact the early focus group took place prior to the first module assessment and assessment results from the first module demonstrated an improvement in results from previous years. in addition, if students were to tackle the problems posted in the evs lecture on their own, rather than in a group, the element of peer learning would be lost. the perceptions of the majority of students in this study were that peer learning was valuable and enhanced their individual learning. interactivity and the positive impact on learning that interactivity within the evsstyle lecture created was a theme that emerged from the survey questionnaires (mean score, 4.7). an overwhelming majority of students appreciated having the time to work in small groups during the large lecture, and preferred group working and group decision-making. most students felt that having the opportunity to interact in the lecture enhanced the learning experience. it was completely different from the way you assumed the lecture was going to be like. there was a group vibe; everyone conferred and you got to be in your group. there was a question in front of you and everyone had a discussion. students reported that the interactivity in class spilled over into their interactions with fellow students after the class was over. some students reported, however, that when other students did not take the group discussion/team decision-making time seriously, this detracted from their learning. students’ perception of the value of the role of lectures emerged as a theme from the focus groups particularly. the majority favourably compared the evs lecture with traditional lectures in terms of enhancing their learning effectiveness. analysis of the data, however, showed a very complex picture about how students perceive that their learning takes place. we are really learning more. we can really learn with our group. i do not have to spend ages re-reading my lecture notes to get it. … felt i really understood the basics; the concepts of marketing. before, i was not so sure. … if it [evs] helps you to know, then learning this way is good. in addition, the value of learning from fellow-students emerged from the data. you can join up with your friends in small groups. you learn from your mates. evs in lectures within business and marketing 13 however, a very small number of students expressed concern about optimum use of lecture time. … worried that we will not cover everything in time for the exams and we will get lower grades even though we enjoyed it. these students also expressed concern about the amount of ‘dead time’ in the evs lecture (e.g. time spent handing out and collecting in the remote voting devices). this appears to indicate a deep concern on the part of a small number of students about their overall performance in a module. those who expressed concerns related to optimum use of lecture time may have been demonstrating that their discomfort of new approaches to learning could adversely affect their level of achievement. in terms of assessment results, we found that student average aggregate performance improved for both modules in 2003/04 compared with the previous cohort of students. the student profile was similar across both cohorts and the same module leaders gave the lectures in both years. the variable was that in the 2003/04 cohort the lectures were interactive using evs. for both modules the average aggregate student marks rose, from 56% in the 2002/03 first-year cohort to 59% in the 2003/04 cohort. discussion from the data analysis we suggest that there may be a model emerging that depicts three stages experienced by students arising from a sequence of evs-style lectures. stage 1: early mass acceptance this occurred in the early period when evs was launched and where there was immediate and widespread acceptance. the vast majority of students described their initial experiences of evs in large-group lectures as highly satisfactory, and this may reflect their engagement with the new. there might also have been a halo effect arising from lecturer enthusiasm for evs. students made initial comparisons between the traditional lectures and evs lectures, appreciating the new benefits delivered by evs. the features that struck them as special to evs included interactivity, engagement, group and peer instruction, rapid feedback of student responses and better understanding. their attitudes and feelings towards the use of evs were strongly held and felt. in stage 1 the novelty factor is pervasive, and students express that this style of lecture relieves the monotony of traditional lectures. stage 2: reflection this occurs when students start a deeper reassessment of their newer blended learning environment created through the evs lecture. they clearly begin to articulate the difference between traditional and evs-style lectures and the effect that both have on their learning. they also begin to assess and analyse the strengths and 14 g. masikunas et al. weaknesses of both types of lectures. for some students, reservations begin to appear. evs lectures are still seen as enjoyable but some students begin to perceive distractions when other students do not take the group discussion time seriously. the euphoria of the first stage passes and students demonstrate a more reflective view in relation to evs as a process. as one student stated: the novelty wore off, but i still liked it. from our findings, students start to fall into a number of clusters according to their ideas, feelings, perceptions and attitudes: ● type 1: ‘must have it’. ● type 2: ‘need it but with minor reservations’. ● type 3: ‘sceptical about it but it needs to improve’. type 1 students are those who liked this way of learning. type 2 and type 3 students demonstrated more reflective and divergent thinking about evs. our focus groups indicated that students who had very fixed ideas about the traditional lecture, and who came from educational systems where there was an extremely strong belief in the ‘professor-model’, found it more difficult to accept evs in the large-group lecture. this will be the subject of further investigation in the future. students in the reflection stage (stage 2) demonstrated clear ideas about the characteristics of a good lecture. features they identified include: covers the material and covers it well. information is well-explained. one where you are interacting. depends upon lecturer style and charisma. stage 3: ‘divergence’ this is the stage when students become more sophisticated about evs, and is informed by the constructive reflections of the students who fell into types 2 and 3 in the previous stage. both these types are very clear about both their reservations and enthusiasm, but type 3 students go further by presenting constructive ways of overcoming any perceived defects or redefining new roles and applications for evs as an interactive tool. those students in type 3 appear to be protecting the evs ideology. in addition, the students who demonstrate divergence also showed evidence of highlevel skill development by applying marketing concepts to extended use of evs in lectures. they appeared to take marketing and customer orientation concepts and use them to: ● propose new uses for evs to meet other needs; and ● suggest product improvement to the existing use of evs. these students exhibited maturity in defining evs in terms of their perceived learning needs, and realised other educational needs which evs could be used to service. evs in lectures within business and marketing 15 the divergence stage symbolises a major shift in the way students think about evs and their own learning. most students, but particularly those who fell into type 3, want to be consulted, but indicate that they feel a part ownership of evs with the lecturers who adopted this interactive lecture method. those students in the divergence stage strongly defended the continued value of lectures as a teaching method, provided it included evs. with me there are a lot of external influences. if there were no lectures i would not learn anything. i particularly like it [the evs style lecture] when it is part of learning rather than part of assessment. i think prs [personal response system] should be made more interactive. this final quote indicates how this student reflected on the experience of the evs-style lecture. students in stage 3 went beyond merely defending the survival of the lecture, especially the evs-style lecture, to expressing the further potential of evs and identifying other areas where this technology could be applied. some suggestions that students in this group made for improving evs lectures included: ● simplifying the start-up and close-down of a evs lecture; ● equipping lecture halls with permanently fixed evs equipment; ● using evs to obtain individual responses to set questions in semester 1 lectures before moving to group interaction and group answers to set questions in semester 2 (or vice versa); and ● proposing different ways of allocating students to small groups with well-thought through reasons for doing this. in addition, students in stage 3 offered suggestions related to extending the use of evs to other learning activities. suggestions included: ● using evs as a formal testing mechanism linked to student performance in the module; ● using it as a revision tool before examinations; ● applying evs to modules that use problem sheets, such as in financial accounting and quantitative studies modules. we are exploring ways of using these suggestions in other learning activities. finally, students in stage 3 also went beyond merely identifying the characteristics of a good lecture. they offered suggestions as to how feedback and interaction in the evs-lecture could be expanded, including: ● enabling individual students to answer the set question and giving individual feedback on answers; ● exploring ways of using whole group interaction; ● greater use of mini-cases—using them as a pre-test and then post-test; and ● integrating results into tutorial work. 16 g. masikunas et al. our data did not provide an explanation as to why students in stage 3 made these suggestions. from their responses, however, we can assume that the impact of evs on their learning in interactive lectures has led them to conclude that its application in other learning activities would also be of benefit to them. our study begins to shed some light on what provides meaningful ways of learning from the students’ perspective. the literature related to learning theory indicates ways of mapping our findings against the different explanations and paradigms about how learning best takes place and how different teaching methods represent different understandings of learning. from the findings related to the themes of team decision-making and interactivity, and the comments by a small minority of students that they would rather have worked on their own on the problems set during the lectures, we recognise that some skills may be best learned independently rather than as a part of group learning activities. there is little in the literature about this and, therefore, this is a subject for future study and exploration. however, given the studies by mazur (1997) and dufresne et al. (1996) and their emphasis on the importance of group learning, we would be reluctant at this time to lose the group learning element of the evs lecture. different paradigms of teaching it appears that evs-style lectures combine a range of pedagogical approaches in an interactive learning experience. we therefore decided to try to map the students’ perceptions in our findings to a framework of different pedagogical approaches. renshaw (1995) suggests a framework of four models for defining and describing learning and teaching. these are outlined in figure 3. figure 3. contrasting approaches to learning and teaching (adapted from renshaw, 1995) what this suggests is that different explanations of learning and teaching determine the place of a range of pedagogical approaches in a programme of study. figure 3 also demonstrates differences between the traditional and the interactive evs lecture. traditional lectures sit firmly within renshaw’s transmission model, while the interactive evs lecture combines aspects of all four models. it is our contention that the evs-style interactive lecture provides learning opportunities within all four models of learning and teaching, and therefore promotes deep active learning in the students. in addition, the themes that emerged from our findings show that students’ perceptions of evs and its value span all four of renshaw’s models of learning and teaching. for example, the comments from students within the theme consultation and participation indicate the value that students place on constructivist/social constructivist learning. the social (and academic) community in which the student lives and works plays a central role in helping students to understand new concepts in a meaningful way. in addition, vygotsky (1978) emphasised the importance that other people in the student’s academic world have in developing the mental processes of that student. in addition, the theme from the student data, reinforcement and feedback, demonstrates the value students place on learning in a transmission model. the themes of evs in lectures within business and marketing 17 the transmission model learning is: teaching is: performing a task giving information correctly cumulative sequential receptive direct from the outside in structuring the environment practising/performing rewarding performance the constructivist model learning is: teaching is: personal setting challenging understanding tasks interpretative/ observing/ selective interviewing active supporting learner activity constructive reviewing/integrating helping learners to reconsider the metacognitive model learning is: teaching is: mindful engagement explaining meaning/experience the social constructivist model (socio-cultural model) learning is: teaching is: social joint activity assisted performance scaffolding creating dissonance managing learning modelling learning tasks strategically strategies reflecting and monitoring reflection supporting adapting and helping to apply transferring across contexts self-evaluating providing evaluation criteria participation interactive and guiding the co-constructive conversation self-regulating assisting in group membership joint construction of meaning evaluating with shared demonstrating values community/ professional values figure 3. contrasting approaches to learning and teaching (adapted from renshaw, 1995) 18 g. masikunas et al. interactivity and team decision-making indicate the importance that students place on social constructivist/metacognitive approaches to learning and teaching, where learning is influenced by social interaction and collaborative learning, and where cognitive skills are developed alongside content. the metacognitive model suggests that students need to develop and use strategies that enable them to monitor their own learning and understanding, and to consider alternative approaches to a task (renshaw, 1995). team decision-making in the evs-style lecture requires students to justify to the rest of the group their individual choice of answer to the problem under discussion, and to consider alternative answers and justifications from other group members. therefore, peer, team and collaborative activity are important metacognitive components of the learning process. through the evs-style interactive lectures we wanted to create a socialconstructivist learning environment, where communication between all team members (students and teachers) encourages collaborative learning and peer student teaching, and where students can develop higher-order cognitive skills. we also wanted to minimise ‘back row syndrome’, where students deliberately sit at the very back of the lecture room to avoid the need to engage with the lecturer or the lecture content. in summary, our study seems to indicate that using evs-style interactive lectures creates a deep active approach to learning within all four paradigms of learning and teaching. conclusions evs-style lectures contain features that promote active learning in lectures to large groups that were consonant with the beliefs of the two lecturers involved. these include the following: ● learning ought to be based on learning theory that is applied to the large lecture situation. ● students enjoy being active participants in an event that matches learning theory with teaching practice. ● electronic voting is a very effective and efficient method for providing rapid feedback that can be shared with students and their teachers. ● questions posed provide feedback to learners and teachers about the effectiveness of lecturer’s teaching and its effect on student learning. ● as an aid to instructional technology, evs is a relatively easy system to master, and provides a range of benefits that would be difficult to obtain by other methods. the interactivity in evs lectures overcame many of the barriers to student learning associated with the large classes taught in the traditional lecture format. the evs lecture opened up new dimensions for exploring effective pedagogy that provides a means of supporting interactive lectures to large groups of students and to ‘turn them on’ to learning. our findings show that our initial aims and objectives for using evs in lectures were met. evs in lectures within business and marketing 19 students, on the whole, perceive that evs-style interactive lectures are an improvement on traditional lectures. the quantitative analysis (assessment results) also indicates that evs-style interactive lectures can improve student learning and achievement. the identified themes, as well as the student responses to evs, when mapped against different models of learning and teaching, indicate that evs spans a range of pedagogical approaches. furthermore, students are not a homogeneous group in their perceptions about, and responses to, evs. this may raise issues about how the views of different types of students are managed. finally, this particular innovation led us to review our understanding of learning, and to appreciate the value of other paradigms of learning and how they help to explain why students reacted so positively to a transformed learning environment. references abrahamson, l. (1999) teaching with classroom communication system—what it involves and why it works, invited paper presented at the 7th international workshop ‘new trends in physics teaching’, puebla, mexico, 27–30 may. alvarez, m., binkley, e., bivens, j., highers, p., poole, c. & walker, p. (1990) case-based instruction and learning: an interdisciplinary project, in: proceedings of 34th annual conference (college reading association), 2–18. aronson, j. (1994) a pragmatic view of thematic analysis, the qualitative report, 2, 1. available online at: http://www.nova.edu/ssss/qr/backissues/qr2-1/aronson.html (accessed 8 march 2005). bligh, d. a. (2000) what’s the use of lectures? (2nd edn) (san francisco, jossey-bass). bloom, b. s. (ed.) (1956) taxonomy of education objectives. handbook 1: cognitive domain (london, longman group). bloom, b. s. (1984) the 2 sigma problem: the search for methods of group instruction as effective as one-to-one tutorial, educational researcher, 13(6), 4–16. boyer, e. (1990) scholarship reconsidered (nj, carnegie foundation for the advancement of teaching). boyle, j. t. & nicol d. j. (2003) using classroom communication systems to support interaction and discussion in large class settings, association for learning technology journal, 11(3), 43–57. brearley, d. (1993) the case study: threat or opportunity?, counsellor education and supervision, 33, 35–37. brown, e. j. & brailsford, t. (2004) integration of learning style theory in an adaptive educational hypermedia (aeh) system, paper presented at alt-c 2004, exeter, 14–16 september. butler, j. a. (1992) use of teaching methods within the lecture format, medical teacher, 14(1), 43–49. carney, c. (1995) teaching with cases in the interdisciplinary classroom: combining business language and culture, in: h. klein (ed.) world association for case method research and application (needham, ma, wacra), 117–127. crouch, c. h. & mazur e. (2001) peer instruction: ten years of experience and results, american journal of physics, 69(9), 2001, 970–977. draper, s. w. & brown, m. i. (2004) increasing interactivity in lectures using an electronic voting system, journal of computer assisted learning, 20, 81–94. dufresne, r. j., gerace, w. j., leonard, w. j., mestre, j. p. & wenk, l. (1996) classtalk: a classroom communication system for active learning, journal of computing in higher education, 7, 3–47. 20 g. masikunas et al. ellliot, c. (2001) using a personal response system in economics teaching, international review of economics education, 1(1) 80–86. available online at: http://www.economicsnetwork.ac.uk/ iree/1/elliot.htm (accessed 1 october 2005). greenwald, b. (1991) teaching technical material, in: c. christensen, et al. (eds) education for judgment: the artistry of discussion leadership (cambridge, ma, harvard business school), 193–214. hake, r. r. (1996) interactive engagement vs. traditional methods: a size-thousand student survey of mechanics test data for introductory physics courses, american journal of physics, 66, 64–74. horowitz, h. (1988) ibm study proves use of student response systems increases attentiveness, paper presented at the sixth conference of interactive instruction delivery for the society of applied learning technology (salt), orlando. jarvis, p. (ed.) (2002) the theory and practice of teaching (london, kogan page). laurillard, d. (1993) rethinking university teaching: a conversational framework for the effective use of learning technology (2nd edn) (london, routledge farmer). liu, t. c., wang, h. y. & chan, t. w. (2003) the features and potential of interactive response system, in: proceedings of international conference on computers in education (icce) (hong kong, ieee conference publishing services), 315–322. marton, f. (1994). phenomenography, in t. husen & n. postlethwaite (eds) international encyclopedia of education (oxford, pergamon), 4424–4429. mazur, e. (1997) peer instruction: a user’s manual (nj, prentice-hall). nicol, d. j. & boyle, j. t. (2003) peer instruction and class-wide discussion: a comparison of two interaction methods in the wired classroom, studies in higher education, 28(4), 477–473. renshaw, p. (1995) excellence in teaching and learning, in: b. lingard & f. rizva (eds) external environment scan (queensland, department of education). sarah, m. d. (2002) research into industry: four years of observations in classrooms using a network of handheld devices, in: proceedings of ieee international workshop on wireless and mobile technologies in education (los alamitos, ca, ieee computer society). tellis, w. (1997) introduction to case study, the qualitative report, 3, 2. available online at: www.nova.edu/ssss/qr/qr3-2/tellis1.html (accessed 1 october 2005). vygotsky, l. s. (1978) mind and society: the development of higher psychological processes (cambridge, ma, harvard university press). yin, r. (1994) case study research: design and methods (2nd edn) (beverly hills, ca, sage). alt-j, research in learning technology vol. 17, no. 2, july 2009, 101–113 issn 0968-7769 print/issn 1741-1629 online © 2009 association for learning technology doi: 10.1080/09687760903033066 http://www.informaworld.com discontent with content analysis of online transcripts judith guevarra enriquez* university of north texas – learning technologies, denton, texas 76207, usa taylor and francis ltdcalt_a_403479.sgm (received 8 may 2008; final version received 4 november 2008) 10.1080/09687760903033066alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2009association for learning technology172000000july 2009dr judithenriquezjudithenriquez@hotmail.com content analysis has dominated computer-mediated communication and educational technology studies for some time, and a review of its practices applied to online corpus of data or messages is overdue. we are confronted with complexity given the various foci, nuances and models for theorising learning and applying methods. one common suggestion to deal with the complexity in content analysis is a call for standardisation by replication or systematic research studies. this article presents its ‘discontent’ with content analysis, discussing the issues and concerns that surround the analysis of online transcripts. it does not attempt to resolve nor provide a definitive answer. instead, it is an open inquiry into another way of looking at online content. it presents an alternative or perhaps an extension of what we have come to know as content analysis. it argues for the notion of genres as another way of conceptualising online transcripts. it proposes two things: first that in performing transcript analysis, it is worthwhile to think how messages relate to a system of interactions that persists even beyond the online environment; secondly, there is an emergent and recurring metastructuring that is at work in online environments that is worth exploring, instead of imposing structures – models and frameworks that do not fit the emerging communicative practices of participants. keywords: genre analysis; content analysis; online transcripts; computermediated communication; educational technology research introduction content analysis (ca) is a research technique that is not singular. rather, it is multiple in both its definition and application whether these are qualitative, quantitative or a mix of both. in its application, determinist assertions of collaboration and community to the online aspect of learning have tended to confine it to the recorded online transcripts (i.e. messages), and consequently, the context of learning has been bounded in the design and use of a particular technology. consensus-bound discourses of collaboration and community have been reinforced by the research techniques used in transcript analysis. the storage of messages in database files made research data readily available for analysis. then in the ‘hands’ of researchers or data analysis software applications (e.g. nvivo), message transcripts become a collective of constructed knowledge produced by the participants. the knowledge reified in the transcripts is further investigated through ‘chunking’ of messages and assigning of codes. this technique or procedure is confronted with issues that are not easily resolved. *email judithenriquez@hotmail.com 102 j.g. enriquez most current educational technology studies emanate from a socio-constructivist background and pedagogical perspective. however, when we come to the constructs to be measured with ca, there are many of them. these include knowledge construction, online presence, interaction patterns or learning strategies (de wever et al. 2006). it is a widely shared view that the available transcripts are the key to understanding online learning. various models and tools have been developed to facilitate transcript analysis (e.g. henri 1992; gunawardena, lowe, and anderson 1997; garrison, anderson, and archer 2000). it is commonly the case that the recorded messages compel researchers and academics alike to ask questions where answers may be readily found within the message content. are the participants making statements or asking questions? is the discussion advancing beyond the sharing stage into negotiation and knowledge building (gunawardena, lowe, and anderson 1997)? is there evidence of cognitive presence, social presence (rourke et al. 1999) and/or teaching presence (anderson et al. 2001)? research findings from content analysis of transcripts offer, therefore, multiple descriptions regarding the cognitive and interpersonal characteristics of the technology. inevitably, this diversity blurs the purpose of transcript analysis. generally, its purpose is to provide a product of proof of what we have come to believe to be its educational value in the first place (e.g. to self-regulate learning, to build a learning community). the context of the transcripts – the technology itself, the physical setting and the conditions in which the transcripts are produced – is usually omitted in doing the analysis. although taxonomies have been developed to analyse the message content of teachers and students in a computer-mediated environment, little is still known about how learners communicate or interact online both as lurkers and participants in relation to the messages of their fellow students and that of their teacher. this article presents its ‘discontent’ with content analysis. it discusses the issues and concerns that surround the analysis of online transcripts. it does not attempt to resolve nor provide a definitive answer. instead, it is an open inquiry into another way of looking at online content. one may argue that using ca with other techniques and methods would deal with some of its limitations. however, the argument of this article goes beyond the need for mixed methods. we already know that. it explores the restricted manner on how we proceed to do content analysis itself. in which case, it presents an alternative or perhaps an extension of what we have come to know as content analysis. it argues for the notion of genres as another way of conceptualising online transcripts. it proposes two things: first, that in performing transcript analysis, it is worthwhile to think about how messages relate to a system of interactions that persists even beyond the online environment; secondly, there is an emergent and recurring metastructuring that is at work in online environments that is worth exploring, instead of imposing structures – models and frameworks – that do not fit the emerging communicative practices of participants. first, it reviews the frameworks and models and the unit of analysis in which popular ca is based. then, it proceeds to highlight its methodological issues, particularly in applying ca quantitatively, before offering genre analysis as an alternative in attending to the communicative practices of participants in online environments. the idea of using genres to study communication is not new. it has a rich tradition within the field of literary analysis (cf. bakhtin et al. 1986), and is emerging as a useful way to explain social action in cultural studies (cf. brown and duguid 1991). alt-j, research in learning technology 103 for more than a decade, it has been applied to the notion of organisational communications and specifically to online communications (e.g. orlikowski and yates 1994; yates, orlikowski, and okamura 1999) and most recently, in terms of weblogs (e.g. herring et al. 2005) and discussion forums (e.g. enriquez forthcoming). current methods and approaches content analysis has dominated computer-mediated communication (cmc) and educational technology studies for a while now. and there appears to be an unquestioned link between the socio-constructivist view of learning and the methods and approaches of educational technology research. this almost tells the story that the technology is inherently constructivist or collaborative in its design or that it may be designed to be collaborative. the most common theoretical perspective is knowledge construction1 usually espoused with notions of collaborative learning and learning community (e.g. järvela and hakkinen 2002; lockhorst et al. 2003; pena-shaff and nicholls 2004; veldhuisdiermanse 2002; veerman and veldhius-diermanse 2001; weinberger and fisher 2006) and the common variable is participation or interaction (e.g. bullen 1998; hara, bonk, and angeli 2000; henri 1992; hillman 1999; howell-richardson and mellar 1996; mcdonald 1998; murphy 2004; rourke et al. 1999). and many research projects undertaken had designed and structured social interaction (e.g. kearsley 1995; kreijns, kirschner, and jochems 2003; liaw and huang 2000; vonderwell 2002). furthermore, although the theoretical basis of such studies is considered to be established, a link between the theory and the methodology or methods is not always clear or present (de laat and lally 2004; de wever et al. 2006; rourke et al. 2001). models of online transcripts the automatic creation of machine-readable transcripts of interactions makes the use of cmc in performing learning tasks quite unique (harasim et al. 1995). it also provides researchers a ‘ready-made’ source of data. it is a widely shared view that the available transcripts are the key to understanding ‘online learning’. various models and tools have been developed to facilitate transcript analysis. for example, there is henri’s (1992) model. perhaps, it is the most popular one. it has been used in other studies, for example, hara, bonk and angeli (2000) and mckenzie and murphy (2000). henri has developed the most sophisticated cognitive analysis model for online interaction: explicit interaction through direct answer or comment; implicit interaction through indirect answer or comment; and independent statement. however, it has also been criticised, particularly by gunawardena, lowe and anderson (1997), who happened to have developed what is considered the next popular model of transcript analysis. they argue that henri’s model is limited in focusing on a teacher-centred instructional paradigm that is inappropriate in a constructivist environment based on a learner-centred construction of knowledge. they theorised that the active construction of knowledge is a movement through five phases in an interaction analysis model: sharing/comparing information; discovery and exploration of dissonance or inconsistency among participants; negotiation of meaning/co-construction of knowledge; testing and modification of proposed synthesis or co-construction; and agreement statement(s)/application of newly constructed meaning. 104 j.g. enriquez lastly, there is the community of enquiry model developed by garrison, anderson and archer (2000). this is a framework that classifies and identifies ‘presence’ in online transcripts in three ways: the cognitive, social and teaching presence of both teachers and learners. models and frameworks have been useful as ‘ordering mechanisms’ on how to represent the nature or characteristics of knowledge construction, for example, in terms of cognition for henri (1992) and presence for garrison, anderson and archer (2000). we have been so focused on evaluating the impact of computer-mediated environments on teaching and learning based on the educational value that we ourselves (or perhaps government initiatives and policies) have inscribed into the use of technologies. we have bounded the learning environment within the virtual (i.e. the technology itself) and omitted the ‘real’ learning environment in our analyses. besides, while (quantitative) ca may provide statistics regarding the number of messages, who contributes and how often and message coding of content, these have proved to be insufficient in indicating the impact of cmc usage in terms of meeting teaching and learning objectives. methodological issues of (quantitative) content analysis this section considers the methodological issues of quantitative ca based on the review of 19 commonly cited studies involving content analysis that have been published over the past decade reported in rourke et al. (2001); a review of 15 articles by de wever et al. (2006); and in a recent review of cscl conference proceedings (2001, 2002 and 2003) which included a total of 31 papers in strijbos et al. (2006). rourke et al. (2001) identified six fundamental issues of content analysis of conference transcripts: criteria of content analysis, research designs, types of content, units of analysis, ethical issues and software to aid analysis. their paper was introduced with a scenario: professor jones who had just completed her first university course delivered entirely online. it was a 13-week course that generated 950 messages, all captured in machinereadable format. almost instantly, she had data ready for analysis. she could test her hypothesis that online her students engaged in much higher levels of discourse and discussion than in face-to-face instruction. further, she was interested in investigating the impact of collaborative learning activity that was assigned in the middle of the course. how she hired two graduate students to do the analysis for her as she did not have much time to do the analysis herself. only to find out after two weeks, that the two coders failed to agree on the categorisations at 70%, the first one had 2032 incidents and the other only 635. and to add salt to injury, professor jones had learned that she could not use the ‘ready-made’ data from her online course as she had not asked the students’ consent. (rourke et al. 2001, 8) this scenario captured more than the six methodological issues identified and discussed in their paper. it also paints the general picture of the arrangements within which ca is used and the discourses and practices that frame computer conferencing in further and higher education. first, it was an online course. second, an online task is a ‘collaborative learning activity’. third, analysis was intended and usually confined within the transcripts. fourth, transcripts were perceived to be a ‘document of learning’, in this case, of higher order thinking, in an online environment. fifth, it was text-based and presumably asynchronous. alt-j, research in learning technology 105 the above scenario presents us with a familiar script of cmc regarding its flexibility: “the capacity of computer conferencing to support interaction among participants while providing for temporal and spatial independence creates a unique and valuable environment for distance, distributed, and lifelong learning applications” (rourke et al. 2001, 8). the focus on the issues identified by rourke and his colleagues was quantitative in nature. they discussed four criteria to which the resulting descriptions or relationships that are derived may be considered valid: objectivity, reliability, replicability and systematic coherence. in terms of research design, 18 out of the 19 papers they reviewed were descriptive. they brought to the fore manifest and latent content, wherein the former refers to surface content of transcripts that is easily observable, for example, the number of times students addressed each other by name (rourke et al. 1999) and the latter, less obvious constructs, themes or variables, such as those ones modeled in henri (1992), gunawardena, lowe and anderson (1997) and garrison, anderson and archer (2000) above. based on berelson’s (1952) definition, ca in communication research is “a research technique for the objective, systematic, quantitative description of the manifest content of communication” (1952, 519, cited in rourke et al. 2001, 4) and that for the sake of scientific objectivity, coding has to be restricted to manifest content (holsti 1969; riffe, lacy, and fico 1998). besides, as strijbos et al. (2006) admitted, latent variables are more difficult to replicate in the first place. however, this would not interest educational researchers and theorists much. their main interest with online transcripts has been on covert processes and latent variables relating to the social construction of knowledge (e.g. gunawardena, lowe, and anderson 1997). knowledge construction has been the basis for studies on higher-order cognitive processes, for example, critical thinking (e.g. bullen 1998; garrison, anderson, and archer 2001; newman, webb, and cochrane 1995). it is the most common ‘covert’ variable in those reviewed by rourke et al. (2001), de wever et al. (2006), and the analysis involved is argued to be inherently subjective and interpretative. so a preferred definition of content analysis might be that it aims “… to reveal information that is not situated at the surface [that is, ‘manifest’] of the transcripts” (de wever et al. 2006, 7). it is at this point that the objective and systematic content analysis defined by berelson (1952) runs into some issues as analysts move from manifest content analysis to latent content analysis, particularly when the variables under study are latent projective, for example, a variable ‘use of humour’ (rourke et al. 1999) does not reside in the surface meaning of the content, rather it resides in the interpretation of the coder drawing from his/her cultural background, age and personality type. the investigation of latent variables has usually been framed within frameworks and models as shown in the previous section. the work of henri (1992), which has been upheld and adapted in many other research studies, has been also been one of the most criticised. it was said to be subjective in rourke et al. (2001) and restricted and inadequate to judge the quality of interaction in meyer (2004). such criticisms raise the issue for some researchers, such as rourke and his colleagues (e.g. rourke et al. 2001; rourke and anderson 2004) and de wever et al. (2006), that content analysis still lacks rigour in terms of reliability, validity and replicability to foster scientific quality and status in the educational technology research 106 j.g. enriquez (e.g. e-learning, computer-supported collaborative learning, asynchronous learning networks). unit of analysis reconsidered rourke et al. (2001) identified five types of unit of analysis: sentence, paragraph, message, thematic, illocutionary. they said there seems to be a negative correlation between a reliable coding scheme and one that encompasses the construct under investigation. on the one hand, fixed units are objectively recognisable or manifest; however, they do not properly encompass the construct sought. on the other hand, a dynamic unit (e.g. unit of meaning) properly delimits the construct, but its latency invites subjective and inconsistent identification of the unit. furthermore, according to strijbos et al. (2006), there are four contextual constraints that affect the applicability of a unit of analysis smaller than a message: (1) object of the study: here the difference between manifest and latent variables is the main consideration. qualitative content analysis, which is the common one in educational technology research, refers to latent variables such as ‘knowledge construction’, that could not be directly observed from the transcript. this has to be inferred in retrospect. (2) nature of communication: here the difference between verbal and written communication and between synchronous and asynchronous media becomes important. in asynchronous text-based exchanges (e.g. forums), messages are usually compound sentences, and telephonic and oral styles are intermixed; while in chat rooms, messages are commonly short and resemble oral communication. (3) the collaboration setting: the collaboration setting or the type of task also influences the applicability of a unit. a task focused on a given topic of discussion leads to less coordinated exchanges. while in a project-based task coordination becomes important and so messages have to link and address the different requirements towards the completion of a project. (4) the technological tool: the tool influences the applicability of a unit. take, for example, the difference between a forum and a chat: messages in threaded forums tend to be longer and may focus on multiple issues, while chat postings evoke short statements usually regarding a singular topic and the frequency of exchange tends to be lower due to its asynchronous nature. to take into account the above contextual constraints, we need a level of analysis that enacts emerging affordances of knowledge in the system of interaction between the environment and the design of tasks using technologies and not in terms of predefined attributes. a unit of analysis must not focus on discrete blocks of content alone; emphasis must instead be given to the referential properties of content. aside from the fact that there seems to be no consensus on the unit of analysis, there is also no agreement on what index to use to test and report inter-rater reliability between two coders. de wever et al. (2006) identified at least nine.2 it does not stop there: there is also no established standard to judge the level of reliability: often a cut-off figure of 0.75–0.80 is used; others state that a value of 0.70 can be considered as reliable (neuendorf 2002; rourke et al. 2001). (de wever et al. 2006, 10) alt-j, research in learning technology 107 this further magnifies the issue that ca applied to online transcripts has failed to adhere to objectivity, reliability, replicability and systematic coherence that make quantitative research valid (rourke et al. 2001). this is not to say that quantitative ca is ‘doomed’ but that it has to be performed with thoughtful consideration. in fact, for those who would like to pursue the validity of quantitative ca, see rourke and anderson (2004) as this article follows a different lead. situated content analysis instead of breaking messages into blocks of semantic units, it is worthwhile to think how they may relate to a system of interactions, that is, the patterning that persists even beyond the online environment, which include the histories, constructs and habits of participants. there are variations in the context in which learning with technology is situated. according to baym (1995), there are five contextual elements that are rarely addressed. these are: (1) the outside environment in which the use of cmc is set. (2) the temporal structure (e.g. synchronous or asynchronous) of the group. (3) the infrastructure of the computer system. (4) the purposes for which cmc is used. (5) the characteristics of the groups and each member. these contextual elements structure the interaction between the technological environment and the pedagogical design that teachers and students may become involved in. before one can proceed with ca it is important to understand that recorded messages are limited in their ability to provide insight into the processes and practices that act upon student learning, regardless of whether they are collaborative or not. aside from the contextual constraints identified by strijbos et al. (2006) above, electronic engagements always require both reading and writing. ‘frozen’ electronic messages cannot possibly capture the interpretive moment. we rely on the written and the visible, because that is what we can monitor and control. genre analysis as an alternative cmc has significantly changed the manner in which we write and talk. online, we write to speak and we have written conversations. we have learned to write and talk in a singular communicative act (so it seems). and this point should be considered in our transcript analysis. it is argued here that we should ask, how do participants ‘write’ talk in an online environment? what are the communicative strategies that are adapted? for example, in a forum online texts are commonly organised into discussion threads. each thread is intended to visually depict a particular topic of conversation. however, as we know in oral conversation, in the process of communication, the conversation flows not because a particular topic is sustained, but because the interlocutors are able to in fact change the topic or refer to other things and able to take turns and repair breaks in the exchange. 108 j.g. enriquez in doing ca, the focus has always been confined within the online data retained in the electronic environment and the extent ‘chunks’ of messages fit into models and taxonomies we have developed given our educational value of knowledge construction, collaboration and community we ourselves have inscribed to online talk/text. it is about time that we analyse content not merely as ‘stuff contained’ but as situated and ‘continued’ in other places and technologies. it is for this reason that it becomes paramount that we take into account communication itself in the way we analyse online content. in this section, genres are described as an alternative way of analysing online transcripts. genres are considered as an analytical frame to consider language alongside the communication medium. a genre is a patterning of communication created by a combination of the individual, social and technical forces implicit in a recurring communicative situation. a genre structures communication by creating shared expectations about the form and content of the interaction, thus easing the burden of production and interpretation. (erickson 2000, 2) genres have identifiable form and purpose (orlikowski and yates 1994, 2002). these provide interlocutors with cues for electronic discourse. first of all, genres are socially constructed and shared (e.g. a discussion about assessment). its form refers to its medium (e.g. discussion board), its structural features (e.g. letter format) and its linguistic features (e.g. level of formality, or graphic devices). the heart of the matter is that a genre has a recognisable form, but the form is what best enables a purpose. so in focusing on its form, the question is ‘what purpose is being fulfilled?’ this purpose may be multiple. in different situations, participants draw from existing genre norms to accomplish a communicative action. the example provided by yates and orlikowski (2002) is that of choosing a letter template rather than an informal note genre for composing an e-mail message that is addressed to an unfamiliar international correspondent. in short, genres provide a template for interaction between members of a community. the particular genre template of a community is an important resource in facilitating efficient communication. in an online environment, individuals may draw on different genre norms out of habit and base genre norms on previous experiences to facilitate specific communicative act. secondly, genres are context dependent and interdependent. they shape, but do not determine the relational cues influenced by the technological environment, task design, previous genres used and the social relationships of those involved. people participate in genre usage rather than control it. one genre exists alongside others and is influenced by them. they are coordinated and combined to accomplish a specific purpose or communicative act (e.g. mulholland 1999; orlikowski and yates 1994). even though genres are dynamic entities that adapt to a change of circumstances, they develop regularities of form and substance. these regularities become established conventions and begin to influence all aspects of communication. prior studies in organisation and media science have shown that: (1) participants employ genres that accomplish the task at hand and the absence of certain genres provides information about their perception of the context of their interaction (e.g. orlikowski and yates 1994); alt-j, research in learning technology 109 (2) the genres used are initially and implicitly imported from the communicative practices use in other contexts (e.g. orlikowski and yates 1994); (3) there are ‘key’ participants who are able to explicitly shape or change the genres initially used (e.g. yates, orlikowski, and okamura 1999). in short, there are genres that serve as foundation blocks, but there are those that are specific to practice situations. if certain genres are the foundation of the genre templates for communicative interaction in an online environment, then it is important that these be supported at the very start of a course or activity. if other genres are important in a certain context of use only, then it is relevant for the tutor/teacher to know which contexts these might be and who the ‘key’ participants are. in an educational institution, the genres would include lectures, course programmes, assignments (essays), exams, etc. thus, in this case, the students would be likely to fall back on education-related genres and communicative practices that involved the use of electronic media (e.g. email, chat, text messaging, blogs). since genres are context dependent and interdependent in their social construction, they will always be identified by reference to conventions established in other related social activities in both physical and virtual settings. as such they are identified within transcripts, but also reflect the wider cultural contexts in which the transcripts were produced. in short, their textuality is relational – always simultaneously ‘in here’ and ‘out there’. the genre patternings become recognisable in their relatedness to other things in a recurring and meaningful fashion. as already mentioned, the use of genre analysis is not something new. however, it has not been widely used as a research technique in cmc studies. before concluding this paper, ‘snapshot’ illustrations of the genre patternings identified in blogs (see for details herring et al. 2005) and also those that were found in discussion forums (see for details enriquez forthcoming) are briefly described. genre analysis proceeds with two main foci: purpose and form. herring et al. (2005) used coding categories established in previous content analytic approach on web genres (e.g. homepages). they also used grounded theory approach to allow for the unique characteristics of a blog genre to emerge. they coded 203 blogs with a total of 44 elements, which included structural and temporal features. it is no surprise that blogs were found to serve multiple purposes: as filters, personal journals, k-logs and also as a mix of these. examples of structural features that were identified include: number of links and images, presence of a search feature, advertisements, calendar, archives, etc. at the blog entry level, body features such as images, links (to own website/s and/or blog/s, to others’ websites and/or blogs) were coded. contrary to the claim that blogs are link-centred filters of web content (e.g. blood 2002), the sample reflected low incidence of links in the entries, a reflection of the prevalence of personal journal-type blogs. blogs have features similar to homepages, asynchronous discussion forums and text-based forms of interactive cmc. in enriquez (forthcoming), a total of 174 postings from three discussion forums were coded. the initial categories for coding purpose and form in the postings were adapted from firth (2002). categories for purpose included individual comment, response or solicitation, group comment, response or solicitation. categories for form included presence or absence of a greeting and/or closing remark. as herring et al. (2005), a grounder theory approach was used to keep ‘in view’ emerging genre patternings and communicative strategies in the postings coded. for example, regularities and repetitions of individual and/or group comments, responses and 110 j.g. enriquez solicitations were signaled or invoked by changing the subject heading. the subject heading had been used in at least two ways in the forums: (1) as a headline that provided the reader with brief information of what the posting is about; and (2) as a turn-taking strategy to interrupt, elaborate or change the subject or topic ‘at hand’. however, there were instances when the subject heading was completely ignored. participants threaded their postings under a given heading not because they had something to say about the subject ‘at hand’, nor were they responding to the posting to which they were threaded. in fact, they were merely writing ‘talk’ after the ‘last’ or latest interlocutor in the forum. in short, it was a temporal structuring that was an attempt to adapt a turn-taking mechanism for written ‘talk’. conclusion in reviewing the frameworks and models and the unit of analysis in which popular content analysis is based, this article has described its discontent with transcript analysis as predominantly practised and applied to electronic content. it has highlighted the methodological issues of ca and has brought into focus communication itself in investigating the communicative practices of participants in online environments. this article presented the methodological issues surrounding the application of content analysis. it proposed genres as an alternative analytical frame for analysing online transcripts in cmc and educational technology studies. first, it suggested that our written conversations must be understood both within and outside the medium and the activity – that the act of ‘writing talking’ online is a confluence of many streams of activity richly equipped with tools, materials, experiences and purposes. this would allow us to understand learning as arising from a person’s historical and multiple relationships and conditions rather than from magnifying the visible design of technological environments and analysing learning or knowledge construction through recorded messages. secondly, we must turn our attention to communication or language itself. we have to understand what it means to ‘speak’ in writing. and with this intent, the ‘discontent’ with content analysis led to the identifiable form and purpose of genres as patternings of communication shared by participants in online environments (see enriquez forthcoming). this allows us to see the communicative strategies and cues that are used in electronic discourse. the importance of metastructuring technology (cf. orlikowski et al. 1995) through the interdependent and recurring patterns of genres becomes paramount in understanding the complex layering of online text. this emphasises that communication captured in online transcripts is not a mere carrier of content, but is an organising process crucial to what gets said and done and by whom. furthermore, the structures that emerge and are maintained become themselves additional resources, strategies and communicative cues for further organisation in communicative actions or practices (cf. crystal 2001; davis and brewer 1997; orlikowski and yates 1994). in future studies, i suggest that we draw upon genres to understand further the communicative practices that develop with the convergence of speech and writing online to accomplish a given learning activity or to fulfill an academic purpose or goal. acknowledgement i am grateful to the two anonymous reviewers for association for learning technology journal (alt-j) who provided useful feedback and suggestions on an earlier draft of this alt-j, research in learning technology 111 article. thanks to frances bell for her encouragement and input. lastly, i have to express my gratitude to martin oliver who has been a great adviser, a clear thinker who has helped me through the muddles of my own ‘complex’ intellectual meanderings. notes 1. ‘construction’ is another of those physical metaphors that easily slide through unnoticed. it suggests a building process, where larger things are created from smaller entities. knowledge creation would be different. it does not merely organise, it is amenable to growth other than [squ ]� + [squ ][squ ]� = [squ ][squ ]�[squ ][squ ]�. 2. these are: percent agreement, holsti’s method, scott’s pi, cohen’s kappa, krippendorff’s alpha, spearman rho, pearson correlation coefficient, lin’s concordance correlation coefficient and kupper-hafner index. references anderson, t., l. rourke, d.r. garrison, and w. archer. 2001. assessing teaching presence in a computer conferencing context. journal of asynchronous learning networks 5 no. 2: 1–17. bakhtin, m.m., m. holquist, c. emerson, and v.w. mcgee. 1986. speech genres and other late essays. trans. v.w. mcgee. austin: university of texas press. baym, n.k. 1995. the emergence of community in computer-mediated communication. in cybersociety: computer-mediated communication and community, ed. s.g. jones, 138–63. thousand oaks, ca: sage. berelson, b. 1952. content analysis for the social sciences and humanities. reading, ma: addison/wesley. blood, r. 2002. the weblog handbook: practical advice on creating and maintaining your blog. cambridge, ma: perseus publishing. brown, j.s., and p. duguid. 1991. organizational learning and communities-of-practice: toward a unified view of working, learning, and innovation. organization science 2, no. 1: 40–57. bullen, m. 1998. participation and critical thinking in online university distance education. journal of distance education 13, no. 2: 1–32. crystal, d. 2001. language and the internet. cambridge, uk: cambridge university press. davis, b.h., and j. brewer. 1997. electronic discourse. albany: state university of new york press. de laat, m., and v. lally. 2004. it’s not so easy: researching the complexity of emergent participant roles and awareness in asynchronous networked learning discussions. journal of computer assisted learning 20: 165–71. de wever, b., t. schellens, m. valcke, and h. van keer. 2006. content analysis schemes to analyze transcripts of online asynchronous discussion groups: a review. computers & education 46: 6–28. enriquez, j.g. forthcoming. genre analysis of online postings: communicative cues do exist online. in analyzing networked learning practices in higher education and continuing professional development, ed. l. dirckinck-holmfeld, c. jones and b. lindström. rotterdam, the netherlands: sense publishers. erickson, t. 2000. making sense of computer-mediated communication: conversations as genres, cmc systems as genre ecologies. paper presented at the 33rd annual hawaii international conference on system sciences, 3, 3011, january 4–7, in maui. firth, d.r. 2002. emergent online communities: the structuring of communicative practices over the internet. paper presented at 23rd international conference on information systems (icis), december 15–18, in barcelona, spain. http://www.business.umt.edu/ faculty/firth/ garrison, r., t. anderson, and w. archer. 2000. critical inquiry in a text-based environment: computer conferencing in higher education. the internet and higher education 2, no. 2–3: 87–105. garrison, d.r., t. anderson, and w. archer. 2001. critical thinking, cognitive presence, and computer conferencing in distance education. american journal of distance education 15, no. 1: 7–23. 112 j.g. enriquez gunawardena, c.n., c.a. lowe, and t. anderson. 1997. analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. journal of educational computing research 17, no. 4: 397–431. hara, n., c.j. bonk, and c. angeli. 2000. content analysis of online discussion in an applied educational psychology course. instructional science 28: 115–52. harasim, l., s.r. hiltz, l. teles, and m. turoff. 1995. learning networks: a field guide to learning and teaching online. london: the mit press. henri, f. 1992. computer conferencing and content analysis. in collaborative learning through computer conferencing, ed. a.r. kaye, 117–36. the najadan papers. berlin: springer-verlag. herring, s., l. scheidt, s. bonus, and e. wright. 2005. weblogs as a bridging genre. information technology & people 18, no. 2: 142–71. http://www.emeraldinsight.com/ 0959-3845.htm. hillman, d. 1999. a new method for analyzing patterns of interaction. american journal of distance education 13, no. 2: 37–47. holsti, o. 1969. content analysis for the social sciences and humanities. don mills, on: addison-wesley. howell-richardson, c., and h. mellar. 1996. a methodology for the analysis of patterns of participation within computer mediated communication courses. instructional science 24: 47–69. järvela, s., and p. hakkinen. 2002. web-based cases in teaching and learning: the quality of discussions and a stage of perspective taking in asynchronous communication. interactive learning environments 10: 1–22. kearsley, g. 1995. the nature and value of interaction in distance learning. http:// www.gwu.edu/-etl/interact.html. kreijns, k., p. kirschner, and w. jochems. 2003. identifying the pitfalls for social interaction in computer-supported collaborative learning environments: a review of the research. computers in human behavior 19: 335–53. liaw, s.s., and h.m. huang. 2000. enhancing interactivity in web-based instruction: a review of the literature. educational technology 40, no. 3: 41–5. lockhorst, d., w. admiraal, a. pilot, and w. veen. 2003. analysis of electronic communication using 5 different perspectives. paper presented at ord 2003, in heerlen, the netherlands. mcdonald, j. 1998. interpersonal group dynamics and development in computer conferencing. american journal of distance education 12, no. 1: 7–25. mckenzie, w., and d. murphy. 2000. i hope this goes somewhere: evaluation of an online discussion group. australian journal of educational technology 16: 139–257. meyer, k.a. 2004. evaluating online discussions: four different frames of analysis. journal of asynchronous learning networks 8, no. 2: 101–14. mulholland, j. 1999. email: uses, issues and problems in an institutional setting. in writing business: genres, media and discourses, ed. f. bargiela-chiappini and c. nickerson, 50–84. london: longman. murphy, e. 2004. recognising and promoting collaboration in an online asynchronous discussion. british journal of educational technology 35, no. 4: 421–31. neuendorf, k.a. 2002. the content analysis guidebook. thousand oaks, ca: sage. newman, d.r., b. webb, and c. cochrane. 1995. a content analysis method to measure critical thinking in face-to-face and computer supported group learning. interpersonal computing and technology 3: 56–77. orlikowski, w.j., and j.a. yates. 1994. genre repertoire: the structuring of communicative practices in organizations. administrative science quarterly 39, no. 4: 541–74. orlikowski, w.j., and j.a. yates. 2002. it’s about time: temporal structuring in organizations. organization science 13, no. 6: 684–700. orlikowski, w.j., j.a. yates, k. okamura, and m. fujimoto. 1995. shaping electronic communication: the metastructuring of technology in the context of use. organization science 6: 423–44. pena-shaff, j.b., and c. nicholls. 2004. analyzing student interactions and meaning construction in computer bulletin board discussions. computers & education 42: 243–65. alt-j, research in learning technology 113 riffe, d., s. lacy, and f. fico. 1998. analyzing media messages: using quantitative content analysis in research. mahwah, nj: erlbaum. rourke, l., and t. anderson. 2004. validity in quantitative content analysis. educational technology, research and development 52, no. 1: 5–18. rourke, l., t. anderson, w. archer, and d.r. garrison. 1999. assessing social presence in computer conferencing transcripts. journal of information technology education 14, no. 2: 50–71. rourke, l., t. anderson, d.r. garrison, and w. archer. 2001. methodological issues in the content analysis of computer conference transcripts. international journal of artificial intelligence in education 12: 8–22. strijbos, j.w., r.l. martens, f.j. prins, and w.m.g. jochems. 2006. content analysis: what are they talking about. computers & education 46, no. 1: 29–48. veerman, a., and e. veldhuis-diermanse. 2001. collaborative learning through computermediated communication in academic education. in euro cscl 2001, 625–32. maastricht, the netherlands: mcluhan institute, university of maastricht. veldhuis-diermanse, a.e. 2002. csclearning? participation, learning activities and knowledge construction in computer-supported collaborative learning in higher education. unpublished doctoral dissertation, university of wageningen, the netherlands. vonderwell, s. 2002. an examination of asynchronous communication experiences and perspectives of students in an online course: a case study. the internet and higher education 6: 77–90. weinberger, a., and f. fischer. 2006. a framework to analyze argumentative knowledge construction in computer supported collaborative learning. computers & education 46, no. 1: 71–95. yates, j.a., and w. orlikowski. 2002. genre systems: structuring interaction through communicative norms. the journal of business communication 39, no. 1: 13–35. yates, j.a., w.j. orlikowski, and k. okamura. 1999. explicit and implicit structuring of genres in electronic communication: reinforcement and change of social interaction. organization science 10, no. 1: 83–103. alt-j, research in learning technology vol. 16, no. 3, september 2008, 221–229 issn 0968-7769 print/issn 1741-1629 online © 2008 association for learning technology doi: 10.1080/09687760802526756 http://www.informaworld.com collaborative virtual gaming worlds in higher education nicola whittona* and paul hollinsb aeducation and social research institute, manchester metropolitan university, manchester, uk; buniversity of bolton, bolton, uk taylor and franciscalt_a_352843.sgm (received 1 april 2008; final version received 8 september 2008) 10.1080/09687760802526756alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2008taylor & francis163000000september 2008dr nicolawhittonn.whitton@mmu.ac.uk there is growing interest in the use of virtual gaming worlds in education, supported by the increased use of multi-user virtual environments (muves) and massively multiplayer online role-playing games (mmorpgs) for collaborative learning. however, this paper argues that collaborative gaming worlds have been in use much longer and are much wider in scope; it considers the range of collaborative gaming worlds that exist and discusses their potential for learning, with particular reference to higher education. the paper discusses virtual gaming worlds from a theoretical pedagogic perspective, exploring the educational benefits of gaming environments. then practical considerations associated with the use of virtual gaming worlds in formal settings in higher education are considered. finally, the paper considers development options that are open to educators, and discusses the potential of alternate reality games (args) for learning in higher education. in all, this paper hopes to provide a balanced overview of the range of virtual gaming worlds that exist, to examine some of the practical considerations associated with their use, and to consider their benefits and challenges in learning and teaching in the higher education context. keywords: virtual environments; game-based learning; constructivist learning environments 1. introduction the growing popularity of multi-user virtual environments (muves), such as second life, and massive multi-player online role-playing games (mmporg) like world of warcraft has led to an increased interest in the use of such technologies for learning. however, different types of collaborative virtual gaming environments have actually been used for many years, and this paper aims to explore the wider context of these worlds and their potential for learning. the focus here is unashamedly on the use of virtual gaming and multi-user environments with adults in the uk higher education sector. this is an area that the authors feel is often neglected in terms of research, or where assumptions are transferred from research with children or other groups; for example that games are a suitable educational tool because they allegedly lead to increased student motivation, which is not necessarily the case with adult learners, who may be time-poor and more strategic in their learning (beasley and crerar 2005). the authors acknowledge that research into the application of games and simulations for learning has a rich history, particularly in the military, corporate and health sectors, but *corresponding author. email: n.whitton@mmu.ac.uk 222 n. whitton and p. hollins the focus of this paper is primarily on the use of games to teach higher-level cognitive skills within higher education. this paper adopts an inclusive definition of an immersive virtual gaming world, so is not restricted to massively multi-user graphical online environments, but refers here to any computer-based environment that shares one or more game-like characteristics, including an environment that can be explored, a competitive element, a fantasy setting, goals, rules and outcomes, and interaction with others. the number of users in this broad definition can range from one to millions; as even single-user environments can still be collaborative, for example, by users working synchronously while using the environment. the fidelity of a virtual environment can vary from being entirely text-based, to using still images, twodimensional or three-dimensional graphics. using collaborative gaming environments to teach is not a new concept, with organisations such as the society for the advancement of games and simulations in education and training (sagset) dating back more than 35 years (van ments 1995). however, the ubiquity of online collaborative communities and use of large-scale virtual worlds, coupled with the ever-growing power of computing technology, have increased the potential of computer-based collaborative learning in a way simply not possible before. this paper will first consider the pedagogic benefits of collaborative game-based virtual worlds in relation to established constructivist theories of teaching and learning. it will then discuss some of the practical issues associated with using virtual worlds to teach in higher education. next, options for the development and creation of virtual environments are presented, and the paper concludes by discussing some of the benefits and challenges of using virtual worlds. 2. pedagogy in virtual gaming worlds collaborative virtual gaming environments can support a variety of pedagogic approaches and can be applied, for example, as both constructivist learning environments and as didactic instructional tools. this paper is concerned primarily with the constructivist perspective, which hypothesises that people learn by developing their own views about the world, by problem-solving and personal discovery. the design of student-centred online learning environments has been very much influenced by the constructivist perspective (e.g. grabinger, dunlap, and duffield 1997; land and hannafin 2000). honebein (1996) presents pedagogic goals of the design of constructivist learning environments in that they should: encourage students to take responsibility for their learning, including what and how they learn; provide multiple perspectives; create self-awareness of the learning process; make learning relevant and authentic; make learning a collaborative and interactive social experience; and use multiple modes of representation and rich media. immersive virtual environments can provide the opportunity for learners to explore and navigate worlds using a range of media types, with authentic and purposeful contexts for practising learning that can be transferred to the real world, and they can present a context for problem-solving and interaction with others. the simple project at the university of strathclyde graduate law school (hughes 2006) is a case that exemplifies this approach. it has created a ‘trading zone’ in virtual space, which provides students with bounded activities and ‘open fields of practice’ of self-directed activity in order to construct their own understandings. working collaboratively enables students to work to their strengths, develop critical thinking skills and creativity, validate their ideas, and appreciate a range of individual learning styles, skills, preferences and perspectives (mcconnell 2000; palloff and pratt 2003). participating in communities of practice provides a legitimate and ongoing way of learning alt-j, research in learning technology 223 from others as part of a group through apprenticeship and education in the context of the group norms, processes and identity (lave and wenger 1991). online gaming communities provide a similar platform for collaboration and the ability to learn with others. for example, studies of leisure users of massively multi-user online role-playing games (mmorpgs) have found evidence of collaborative learning, development of communities of practice (steinkuehler 2004) as well as the potential for learning a range of group skills, including the etiquette of meeting people, group management, co-operation and social interaction (ducheneaut and moore 2005). the constructivist perspective also puts forward the idea that students learn better by experiencing for themselves and discovering their own meanings from their experiences. gee (2003) argues that computer games reflect the experiential learning cycle (kolb 1984) in that students must examine the virtual environment, reflect on the situation to form a hypothesis, further investigate the virtual world and see what effect their action has. while it is true that this cycle maps onto learning within the game world, it does not necessarily provide students with scope for meta-cognitive processes and reflection outside of the virtual world. prensky (2001) argues that if games were used for learning, then “learning would happen almost without the learners realizing it, in pursuit of beating the game” (26). in the context of higher education, without the opportunity to support learners to understand the process, context and transferability of learning, the value of learning undertaken in this way is questionable. in relation to reflection, prensky (2001) argues that reflection does occur within games, only at a greater speed, or “twitch speed” as he defines it, although this assertion is controversial. it is important to recognise that learning in virtual environments may be part of a larger process and other activities – including formal and informal ‘affinity groups’ or communities, and reflective practice – often surround their use. although many first year undergraduates may have more in common with school-based learners, there is a growing representation of mature learners in higher education. these individuals typically have more work and personal responsibilities, caring commitments and greater life experience. adult learning theory (knowles 1998) argues that adult learning (or ‘androgogy’) is different from children’s learning and that the key characteristics of adult learners are that they need to know why they need to know something before they are willing to invest time and energy in learning it, they have a deep psychological need to be self-directing and to take responsibility for their learning, become ready to learn something when they need to know it to be able to use it in real-life, and are task-oriented in their learning. adult learning theory highlights the fact that motivation to learn is paramount and that learning activities must be purposeful for students to engage. this brings into question the acceptability of learning in virtual worlds with adult learners, who may perceive these environments as frivolous and time-wasting. there are misconceptions associated with play that might affect the acceptability of game-based learning, for both adults in higher education and children in school-based learning contexts (egenfeldt-neilson 2005), in that it is perceived by many as only for young children, as not being a respectable thing to do, and as an activity that is easy (rieber 1996) or that the learning is somehow inauthentic. although outside the scope of this paper, it is also recognised that immersive virtual worlds can be used in the behaviourist tradition as tools for skill development in higher education in the sciences or medical domains. while collaborative virtual gaming environments have the potential to engage students and support authentic, experiential and collaborative learning, there are many practical 224 n. whitton and p. hollins issues associated with their use in higher education, which are discussed in the following section. 3. virtual gaming worlds in practice there are many practical considerations when using virtual gaming worlds in learning and teaching in higher education. foremost, it is essential to have a clear educational purpose for their use, and not simply because they are thought to be motivational. while computer games may be motivating for some, this is often not the case for adult learners, who may be more strategic in what and how they learn and will look to learn things in the most appropriate way (knowles 1998). however, while adult learners may not be interested in learning with a game for its own sake, there is evidence that they will be happy to use them if they are perceived as an effective way to learn in that context (whitton 2007). matching the in-world outcomes with the intended learning outcomes from use is another key issue. if the use of the environment does not closely match the curriculum, then students may be learning something, but it may not be relevant to the course of study. built in to many entertainment games is the goal of learning to play the game itself, mastering the interface and discovering advanced features, which may detract from the learning value of the game. other practicalities such as the amount of time available both to develop or modify the gaming world and to use in teaching, space, whether a course is online or face-to-face, and resources available (e.g. access to computers with correct specifications) will also influence how and when gaming worlds can be integrated into teaching and learning. an issue often arises in university settings with gaming worlds being seen as inappropriate or with communications ports being automatically locked so that software can not be accessed; threedimensional immersive worlds also often require higher specifications of graphics cards or processors than are typical in university machines. university regulations may also affect how these worlds can be integrated and assessed, and impact on whether the use of a virtual gaming world is a supplemental activity, single or multiple use, or fundamental to the whole structure and design of a course; they can also affect whether, and how, learning in gaming worlds is assessed. it is particularly important to separate achievement in the gaming environment with achievement in the assessment so as not to disadvantage students who might be performing less well for reasons that are nothing to do with their level of learning (e.g. students who have problems navigating in a three-dimensional environment). welldesigned assessment can encourage the development and integration of reflective and collaborative activities, for example considering how progress through the game could have been different in different circumstances, or working with others to produce a reflective report. while gaming environments may provide experiential learning spaces, they do not necessarily provide students with scope for reflection and application of their learned knowledge and skills to the real world. activities such as debriefing and structured reflection are essential to ensure appropriate mastery of specified learning outcomes, and these activities can be structured outside the virtual world. as not all gaming environments are collaborative in nature, it is important to consider how communication with others can be built in to the learning package. ways of doing this include face-to-face discussion interspersed with periods of individual game play, two individuals playing the same game at the same time and discussing strategies, either face-to-face or using an online messaging facility, or asynchronous message boards supporting game play and problem-solving. alt-j, research in learning technology 225 whitton (2007) developed a set of practical criteria for the design of gaming worlds for learning and teaching in higher education, from a constructivist viewpoint, focusing on highlevel cognitive skills such as application, analysis and evaluation (bloom 1956). existing design guidelines in related areas were reviewed and synthesised and an evaluation of a number of popular internet games with potential for learning was carried out to examine their design characteristics. these two pieces of work were drawn together to produce a set of six criteria for the effective pedagogic design of educational games for students in higher education. (1) the environment should support active learning by encouraging exploration, problem-solving, and enquiry, providing opportunities to test ideas and gain feedback, practice and consolidation. opportunities for collaboration should also be provided and, as much as possible, game goals should align with learning outcomes. (this assumes, incorrectly, that collaborative learning is a prerequisite of ‘all’ learning; learning by rote or didactic approaches can be effective in complex skill and competency development in higher education. examples exist in the medical and engineering disciplines.) (2) the environment should engender engagement, with explicit and achievable goals, provision of a large world to be explored with a high level of interactivities, multiple pathways, and different ways in which success can be achieved. the world should stimulate the user’s curiosity and provide an appropriate level of challenge and control of the environment. this, however, assumes a level of competence of the course designer to develop ‘engaging’ immersive content and activities in the gaming medium, therefore a minimum requirement would be significant personal development planning for educators to support development of their gaming literacy or development competence. (3) the gaming world should be appropriate for the learning context, in that it fits with, or is at least relevant to, curriculum and assessment, is suitable for the subject area, matches the time available, and is personally relevant to and acceptable by the students. (4) the environment or associated activities should support and provide opportunities for reflection, allow students to debrief from the game and contextualise their learning, and highlight the process of learning. (this makes the assumption that reflection is a requirement of all higher education learning activities.) (5) the environment must provide an equitable experience for all users, taking into account and accommodating differing prior knowledge and experience of that world and similar environments. ideally it should allow for personalisation and customisation and provide equal opportunities for all students to participate. alternative pedagogic approaches should be available to students aligned to their personal learning preferences. (6) the gaming world needs to provide ongoing support, from initial orientation to early tasks that provide quick initial success, with a gradual introduction of increasing complexity, supported with help, hints or clues to ensure that the environment does not become limiting. the gaming experience should aim to extend players to their extreme level of competence. while these guidelines were developed from a constructivist standpoint, it is recognised that games can also be used effectively as instructional tools in adult learning contexts in further and higher education, adult and community learning, the corporate training sector 226 n. whitton and p. hollins and in military applications. for example, the most widely accessed ‘serious game’ is the americas army recruitment game with over 17 million downloads. in the game players are taken through a simulation of army basic training, career selection and into combat teams, and us army research suggests that there has been a marked decrease in the military training drop-out rates from those who have undertaken basic training in the americas army game prior to entry. in this section some of the issues associated with the practicalities of using virtual gaming worlds to teach have been discussed. a range of options exist for creating different types of virtual gaming environments and these are explored in the following section. 4. selecting a virtual gaming environment a number of options of virtual gaming environments are available to the educator, from using off-the-shelf games software designed for entertainment, to creating a bespoke environment from scratch. some school-based practitioners use existing commercial games in a teaching context, for example using the civilization game to teach history (squire 2005) and in the teaching with games project undertaken by futurelab and electronic arts (sandford et al. 2006). this has the advantage of enabling students to use a high-end product, which has been explicitly designed to be engaging; however many commercial games are also expensive. given the established business model where publishers commission development studios to develop content and pay ongoing royalties on unit (boxed) sales, developers insist, understandably, on royalty payments being applied against a minimum retail price, leaving little flexibility in respect of pricing for educational use. the technical architecture of commercial games makes the disaggregation of content almost impossible, an essential requirement if games are to be used in the time-constrained classroom environment. it can be difficult to exactly match the outcomes of the curriculum with those of the game so wrap-around briefing and debriefing activities are particularly important. commercial gaming environments may also have complex interfaces, extensive functionality and a long learning period, which may detract from the intended learning outcomes. the long time taken to complete many games may also be far greater than a typical teaching timetable, however even given these limitations, innovative approaches to the use of games in the classroom have emerged, for example the innovative use of the myst adventure game in english literature (rylands 2007). the growing trend towards modifying existing games software for use in education (de freitas 2007) may provide one way to address some of these issues. many gaming environments now come with a creation engine that allows teachers, or students, to develop their own add-ons such as new storylines or game areas. this reduces development time and required expertise – but is still not a trivial process – and can help to ensure the links between learning and gaming outcomes. a proliferation of available middleware and gaming engines, developed by the industry in order to negate spiralling development costs, are available to educators at increasingly affordable costs. for example many of criterion’s middleware tools are available to educators at no cost, and the havoc gaming engine has been used in a number of educational projects such as the virtual multi user learning environment (vmule) project, initially undertaken by the university of liverpool and further developed by futurelab (see www.futurelab.org.uk/projects/vmule). in parallel to this, open source tools and technologies such as opensim and croquet are available to educators, but do require a degree of technical competency. alt-j, research in learning technology 227 as well as the option of using entertainment games, there are also a number of commercial gaming environments specifically designed for learning. however, these can be expensive and difficult to customise. examples exist where the game exactly matches the subject area and learning outcomes and is suitable for students of a specific demographic (e.g. whitton and hynes 2006), but these are limited. a growingly popular option is the use of existing virtual worlds such as second life (e.g. childress and braswell 2006) or mmorpgs for learning. these massive online worlds allow the interaction of thousands of players in real-time in virtual spaces; they can manipulate objects, solve quests, create and join groups, or work with others. some worlds also allow users to create their own spaces and objects. issues with using these worlds include the lack of privacy from other users who can wander into teaching spaces, the bad reputation that has built up over certain types of behaviour in these environments (for example violence in mmorpgs and the overtly commercial nature and sexual behaviour in muves), and the high-specification machines and internet connections required to effectively run these applications. hollins and robbins (in press) have identified what they term the educational affordances of muves, namely: identity, space, activity, tools and community. another option is the creation of a bespoke game, such as the action-adventure game developed to teach basic literacy skills to adults (kambouri, thomas, and mellar 2006). this enables a designed match of learning outcomes with gaming outcomes for a specific user group. however, the problems associated with the design of bespoke education software involve the limited budgets compared to entertainment software, and how this affects the expectations of learners. jenkins (2002) argues that most educational software is of poor quality, badly edited and unprofessional. it is unlikely, however, that the amounts of money spent on commercial software will be viable in education, and it is important that resources are focused on ensuring that educational games for adults are well designed in terms of playability and learning. as well as viewing students as players of games (or merely consumers of content), they can learn by developing or creating games. rieber et al. (1998) argues that learning by building games can be an at least, if not more, effective way to learn than traditional methods, while gee (2003) argues that active, critical learning should lead to learners becoming designers, either by physically designing extensions to the game or by cognitively extending the game design and using that to inform their play. a solution to the problem of finding a gaming environment that is fit-for-purpose, customisable and relatively inexpensive could be the alternate reality game (arg). an alternate reality game is a type of interactive fiction that unfolds over a period of time, and requires puzzles to be undertaken collaboratively in order to progress the story. an arg consists of three elements: a series of challenges, an underlying narrative, and a collaborative community, and while all three elements can be facilitated online, many challenges take place in the real world, and may be collaborative or individual. args have been typically used for marketing and publicity (e.g. see rose 2007) but also have potential in education for providing purposeful and motivating collaborative activities to contextualise learning. alternate reality games have a number of advantages over commercial games or developing fully online games from scratch. they are lo-fidelity, combining a range of web technologies so, while they do require some expertise and creativity to design, they are far cheaper and faster to create than developing high-end software, and can ensure that specific learning outcomes are met. as they involve the integration of web technologies such as blogs, wikis, and social networking software, they provide users with the experience of different technologies. although args do require ongoing monitoring and moderating of 228 n. whitton and p. hollins the community, this task is often undertaken by enthusiastic ‘expert’ players who can also get involved with building and extending the game. examples of how args can be used in education are provided in the alternate reality games for orientation, socialisation and induction (argosi) project (see www.playthinklearn.net/argosi.htm) and the futurelab virtual savanagh project (see www.futurelab.org. uk/projects/savannah). 5. conclusions this paper has examined the use of collaborative virtual gaming worlds in the context of higher education, looking at the pedagogic advantages, practical considerations and issues of selecting an appropriate environment. in using these types of environment it is important to ensure that there is a sound pedagogic rationale, which is clearly communicated, and they are not simply being adopted for a perceived motivational effect. creating effective educational gaming worlds can be expensive, in terms of development time and expertise as well as financially, and it is essential to ensure that there are actual benefits to their use. creating ‘good’ games is extremely challenging as an undertaking in itself; even the commercial sector struggles to achieve a 20% success, using unit sales as the metric. the future for virtual gaming in higher education may be twofold: first, where collaborative activities in the constructivist philosophy may be based around the use of environments where a low (financial) cost of entry exists, such as alternate reality games or second life; and second, for skills development in photo-realistic simulation activities – such as currently undertaken in the military and medical sectors – where development budgets have historically been significantly higher. whether higher education practitioners will consider adopting commercial off-the-shelf (cots) products could be a subject of further research and analysis. the novelty effect of using such environments should also be considered – both from the perspective of teacher and student – as this may influence long-term acceptability and effectiveness. it is also crucial that these environments do not lead to unintended exclusion of some users, for example those individuals who find it difficult to navigate within threedimensional spaces. linking the use of these virtual worlds to appropriate and authentic assessment activities will also be one of the key challenges within the context of university education. however, there are extremely active teaching and research communities emerging around the use of gaming environments, the second life educational (sled) community being a case in point. in all, while there are, as highlighted in this paper, examples of the effective use of immersive gaming environments in the context of adult learning in higher education, there is a clear need for more ongoing and robust studies into the potential of different types of virtual worlds, studies that recognise the significant issue of the context of use. this will help to ensure that as innovative and effective practices emerge, they can be evaluated in the context of the student learning experience and integrated into the growing body of knowledge on learning in virtual gaming worlds. references beasley, n., and a. crerar. 2005. motivations for adults playing games. in proceedings of isaga 2005, june 27–july 1, in atlanta. bloom, b.s. 1956. taxonomy of educational objectives, handbook i: the cognitive domain. new york: david mckay co. inc. alt-j, research in learning technology 229 de freitas, s. 2007. learning in immersive worlds: a review of game-based learning. bristol: jisc. childress, m.d., and r. braswell. 2006. using massively multiplayer online role-playing games for online learning. distance education 27, no. 2: 187–96. ducheneaut, n., and r.j. moore. 2005. more than just ‘xp’: learning social skills in massively multiplayer online games. interactive technology & smart education 2: 89–100. egenfeldt-nielsen, s. 2005. beyond edutainment: exploring the educational potential of computer games. phd diss., it-university copenhagen, denmark. gee, j.p. 2003. what video games have to teach us about learning and literacy. new york: palgrave macmillan. grabinger, s., j. dunlap, and j. duffield. 1997. rich environments for active learning. alt-j 5, no. 2: 5–17. hollins, p., and s. robbins. in press. educational affordances of multi-user virtual environments. in living virtually: researching new worlds, ed. d. heider. new york: peter lang publishing. honebein, p.c. 1996. seven goals for the design of constructivist learning environments. in constructivist learning environments: case studies in instructional design, ed. b.g. wilson, 11–23. englewood cliffs, nj: educational technology publications. hughes, m. 2006. welcome to simple. http://technologies.law.strath.ac.uk/simple2/index.php? option=com_content&task=view&id=17&itemid=38. jenkins, h. 2002. game theory. technology review march. http://www.technologyreview.com/ energy/12784/?a=f kambouri, m., s. thomas, and h. mellar. 2006. playing the literacy game: a case study in adult education. learning, media and technology 31, no. 4: 395–410. knowles, m. 1998. the adult learner. 5th ed. houston, tx: butterworth-heinemann. kolb, d.a. 1984. experiential learning: experience as the source of learning and development. new jersey: prentice hall. land, s.m., and m.j. hannafin. 2000. student-centered learning environments. in theoretical foundations of learning environments, ed. d.h. jonassen and s.m. land, 1–23. mahwah, nj: lawrence erlbaum associates. lave, j., and e. wenger. 1991. situated learning. legitimate peripheral participation. cambridge: university of cambridge. mcconnell, d. 2000. implementing computer supported cooperative learning. 2nd ed. london: kogan page. palloff, r.m., and k. pratt. 2003. the virtual student: a profile and guide to working with online learners. san francisco: jossey-bass. prensky, m. 2001. digital game-based learning. new york: mcgraw hill. rieber, l. 1996. seriously considering play: designing interactive learning environments based on the blending of microworlds, simulations and games. education and training resource & development 44: 42–58. rieber, l.p., l. smith, and d. noah. 1998. the value of serious play. educational technology 38, no. 6: 29–37. rose, f. 2007. secret websites, coded messages: the new world of immersive games. wired. http:// www.wired.com/entertainment/music/magazine/16-01/ff_args. rylands, t. 2007. itc to inspire with myst. http://www.timrylands.com/index.html. sandford, r., m. ulicsak, k. facer, and t. rudd. 2006. teaching with games: using commercial off-the-shelf computer games in formal education. bristol: futurelab. www.futurelab.org.uk/ download/pdfs/research/twg_report.pdf. squire, k.d. 2005. changing the game: what happens when videogames enter the classroom? innovate 1, no. 6. http://innovateonline.info/index.php?view=article&id=82 steinkuehler, c.a. 2004. learning in massively multiplayer online games. in proceedings of the 6th international conference on learning sciences, 22–26 june, in santa monica, california. van ments, m. 1995. the development of simulation and gaming in britain: a sagset view. simulation & gaming 26, no. 2: 224–35. whitton, n. 2007. an investigation into the potential of collaborative computer game-based learning in higher education. phd diss. www.playthinklearn.net. whitton, n., and n. hynes. 2006. evaluating the effectiveness of an online simulation to teach business skills. e-journal of instructional science and technology 9, no. 1. http:// www.usq.edu.au/electpub/e-jist/docs/vol9_no1/papers/current_practice/whitton_hynes.htm alt-j, research in learning technology vol. 16, no. 1, march 2008, 5–14 issn 0968-7769 print/issn 1741-1629 online © 2008 association for learning technology doi: 10.1080/09687760701850166 http://www.informaworld.com research article on campus, but out of class: an investigation into students’ experiences of learning technologies in their self-directed study frances deepwella* and samina malikb acentre for the study of higher education, coventry university, uk; binternational islamic university, islamabad, pakistan taylor and francis ltdcalt_a_285159.sgm10.1080/09687760701850166alt-j, reaserch in learning technology0968-7769 (print)/1741-1629 (online)research article2008taylor & francis161000000march 2008francesdeepwellf.deepwell@coventry.ac.uk this paper presents an investigation into how students studying at university engage actively with learning technology in their self-directed study time. the case study surveyed 250 students studying at undergraduate and postgraduate level from a purposive sample of departments within one institution. the study has also conducted focus groups and a number of in-depth follow-up interviews with respondents to the survey. in this article we explore three emerging aspects of the learning experience, namely student expectations of the technology, their lecturers’ engagement with technology and how the technology might support processes of transition in higher education. one key implication is that more academic guidance is needed on what and how to use the technology effectively for independent learning, even where ict skills levels are high. the study also identifies the significant role that the lecturer plays in facilitating students’ use of technology. the findings of this study will be of interest to those working to incorporate learning technologies more effectively in higher education, in particular for those who are looking to improve the engagement of students in self-directed learning. keywords: student experience; learning technologies; self-directed learning; blended learning; case study introduction this paper reports on an investigation into the experiences of students using learning technologies in their learning while studying on campus-based university courses. particular attention is paid to the experiences of campus-based students outside formal classroom interactions in self-directed study. self-directed learning (sdl) is ‘a form of study in which learners have the primary responsibility for planning, carrying out and evaluating their own learning experiences’ (merriam and caffarella 1991, 41). sdl occurs in both informal and formal learning situations and typical activities include peer discussion and collaboration, accessing library resources, reading, research and development and working through tutorial materials and workbooks. within formal academic programmes in the uk, hours are allocated to self-directed study in addition to hours for lectures and seminar attendance. these hours are quantified in quality assurance documents and typically constitute a substantial proportion of an academic programme. we were, therefore, interested in investigating how students were using technology during this time, which is, at least notionally, at the disposal of the students. there is an expectation on the part of the higher education institution that students use the time for studying in depth and preparing their coursework. for the student this notional time is under threat. the threats can come externally from the demands of income generation to cover fees and living expenses, social pressures and an overloaded curriculum, * corresponding author. email: f.deepwell@coventry.ac.uk 6 f. deepwell and s. malik together with a declining staff:student ratio. however, there are also internal factors, such as learner motivation, preparedness to study and conceptions of their student identity that affect how students approach sdl. in an environment where solutions are sought to deal with the new mass higher education community, e-learning (in all its forms and blends) is seen as having potential. there is now unprecedented access to learning resources and information, data analysis and creative software, communication and collaborative technologies in support of learning in higher education. as e-learning practice develops as a field of inquiry distinct from distance and open learning and beyond ‘projects by innovators’ (salmon, 2005), we are starting to discover more about how to embed technology, how to design and assess learning tasks, how to motivate participation and encourage co-construction of knowledge. until recently, much of the research has been practitioner-led rather than from the perspective of the learners (sharpe et al. 2005). however, the jisc is funding a uk research programme to explore the learner experiences of e-learning which is now producing interesting accounts of the student experience across higher education, further education and adult and community education.1 this research programme is also developing new and robust methodologies for researching this complex field of inquiry. the present study explores a sub-section of the student learning experience, namely how learners make use of technology to further their independent learning within formal programmes of study and the role that academic colleagues and the institution plays in supporting such use. we propose that the use of technology, both for campus-based and distance learners, requires a level of self-direction similar to sdl. this is particularly so in an increasingly blended learning delivery of higher education. the individual has to take the initiative and responsibility for what they select, manage and access in a limited time outside formal contact hours. therefore, some of the research questions we wanted to explore in this study were: what is the reality of sdl for students today; how has learning technology pervaded out-of-class study time; who supports the learner in making sense of the technologies and what needs do they have; what expectations are there of academic colleagues in supporting sdl? the present research attempts to explore these aspects, namely the use of technology that students on higher education courses make use of in their learning outside classroom activities. to understand more about some of these issues we conducted two focus groups, a survey and a number of in-depth follow-up interviews. a purposive sample of students studying at undergraduate and postgraduate level from a cross-section of departments and at different stages in their academic programmes was selected for the collection of survey data. background increasingly over the past decade the learning experience of students in higher education in the uk is one that is mediated, at least in part, by technology. institutions have invested heavily in establishing a robust and integrated technical infrastructure not only to support their own administration functions, but also to enable computing access, e-mail communication, online information systems and virtual learning environments (vles). at the same time academic colleagues are increasingly expected, and in some cases required, to use the institutional technologies in order to deliver their teaching. in the uk by 2005 over 95% of institutions of higher education supported a vle (sharpe et al. 2006) and the term ‘blended learning’ is now commonly used to refer to the mix of face-to-face and e-learning that constitutes the higher education experience of students. there are many reasons for the growth in the use of learning technologies cited in the literature (for example: to manage larger numbers of students on modules; to serve the needs of a more diverse student population; the management imperative to become competitive in the new alt-j, research in learning technology 7 online markets), however, there is still uncertainty over the impact of the introduction of technologies, as bonk, kim and zeng outlined: the promises (and, hopefully, the benefits) of blended learning are extensive. for instance, some promote increased learning, others point to the reduction in the need for brick and mortar, and still others allude to engagement, collaboration, success, ownership, and higher quality learning. further research and innovation in the blended learning arena will help sort out the key contributions, benefits, and impact areas. (bonk, kim, and zeng 2004) across the world there has been an upsurge in the use of learning technology in higher education, although the technologies themselves may vary, from video-conferencing to websites, cd roms and mobile phones. the prevailing technologies in the uk throughout the early 2000s, the vles, are characterized by the control that the lecturers have in shaping the online environment. more recently, however, the dominance of institutional vles has started to give way to more personalized, collaborative and student-centred environments made possible by web 2.0 technologies, where the monitoring and surveillance aspects of vles (land and bayne 2005) give way to the individual control and personalization of technologies. in this changing and complex higher education context learning technology has been the subject of much funded research in the uk (funded by jisc, the eu elearning initiative, the higher education academy, to name but a few). with the ability to track aspects of student usage and the enthusiastic engagement of innovative lecturers it was hoped that some of the more elusive questions around the embedding of technology in learning might be answered. research into the use by lecturers of e-learning has increased our understanding of pedagogy (oliver 2006). in part this has been because the technology makes explicit the teaching, learning and assessment aspects that normally remain implicit, and unarticulated, except within a team delivering a shared module. this research has largely been from the perspective of tutorled online material and activities, including the use of ‘learning activity management systems’ (sharpe et al. 2006). the research literature focuses on the materials which are made available to students and how learners interact with set tasks, such as online discussions or group assignments. there have been many studies of distance learning and of informal learning, but reviews of the literature have identified gaps in studies into students’ experiences of blended learning, i.e. of formal courses where there is a combination of face-to-face teaching and online activities (sharpe et al. 2005). a recent set of studies commissioned by jisc are addressing this gap by focusing exclusively on the learners’ experiences, for example the qualitative study of the learner experience of e-learning.2 in this inquiry the research team derived five higher order categories, namely life, formal learning, technology, people and time. within each category were five influencing factors: control, identity, feelings, relationships and abilities (creanor et al. 2006, 23). these categories and influencing factors are derived from narrative accounts of learner experiences, prompted by mediating artefacts, and represent a rich account of the learner perspective, crossing over the boundaries between formal and informal learning and educational sectors. the present case study specifically explores the part played by selfdirected study for students on campus-based programmes, since this constitutes a substantial proportion of the time learners in higher education are studying and remains largely underresearched. it is hoped that this present case study will complement recent learner-centred studies and provide some further insights into the expectations and behaviour of learners with regard to e-learning outside classroom activities. methodology the case study was carried out in early 2007, after receiving approval from the university ethics committee. the purpose of the research was to gain a greater understanding of patterns, 8 f. deepwell and s. malik motivations and engagement with learning technology amongst students learning on campus on traditionally delivered academic programmes. case study is a research approach that investigates real-life phenomena which can be based on qualitative and quantitative evidence (stake 1995). case study research is looking for particularization rather than generalization. methods employed in this study included focus group research, questionnaires and semi-structured interviews with students. coventry university has a relatively long tradition in using learning technology to support campus-based learning; all academic courses and modules at this institution have been supported by an institutional vle since 1999. usage of the vle and associated technologies is widespread, thereby making this a suitable site to investigate the topic. in scoping the inquiry two focus groups were held in order to understand the issues from a student perspective and thus inform the questionnaire design. most of the survey questions were closed choices, with just three open-ended questions at the end which explored the likes and dislikes of learning technologies. the questionnaire was tested prior to the launch, which was in march/april 2007. around 240 students were taken as a purposive sample for collection of data through the survey. the students were studying at undergraduate and postgraduate levels from a cross-section of departments. table 1 shows the spread of responses, which were distributed and collected in class, with the cooperation of the respective lecturers. the chosen method of distribution increases the proportion of completed questionnaires in each class group, although some students in each group chose not to participate in the research, so of 267 questionnaires, 237 were returned (89%). the student demographic was mixed in terms of gender (55% male, 45% female), ethnicity (50% white, 22% black, 19% asian and 8% chinese), status (55% home students, 45% overseas students). less than 1% (n = 14) declared a disability. the age range was between 18 and 64 years old, with just over half of the students (54%) in the 22–29 years old category. semi-structured follow-up interviews took place on campus with students from these groups who were willing to assist further in the research. an inductive, interpretivist analysis of the transcripts of the interviews was carried out by the two researchers independently, using a relatively simple mark-up procedure to identify themes in the student narratives. the themes were then discussed, clustered and agreed within the research team and subsequently validated through further analysis of the survey results, including the responses to the open-ended questions from the survey. preliminary findings overall, the responses from the survey indicate that the students were in favour of using learning technologies to support them in their studies, with around 80% of the sample agreeing that the table 1. distribution and return rate of the survey. module level distributed return rate management research m 9 100% management m 29 93% journalism 1 24 100% advertising 2 26 92% multimedia technology 3 24 75% financial mathematics 3 72 97% social work 3 60 75% clinical psychology 3 23 87% alt-j, research in learning technology 9 technologies are important to their learning and around 70% agreeing that they do not experience difficulties using them. sixty-four per cent agreed that learning technology has positive effects on their attitudes towards learning independently. in terms of time spent on independent, self-directed learning, 10% of students in the survey reported spending over 21 hours per week and 47% between 11 and 20 hours per week. thirtythree per cent reported spending between 5 and 10 hours and the remaining 10% spent under 4 hours per week. these uk findings closely match those of the us national survey of student engagement, 2004, reported in herrington, reeves, and oliver (2006, 100), which reported that of 163,000 students only 11 per cent of full-time students spend more than 25 hours a week studying outside their classes – the minimum number of hours academic staff expect students to study to be successful. a disappointing 44 percent of students surveyed reported studying less than 10 hours a week. whilst the context and scale of these two studies are different, there is close correspondence in the proportion of time that students spent on their independent learning. most of the use of technology is reported to be to access information (76%) or review course materials (73%), manage course information (65%) and communicate with friends (59%). fiftythree per cent used technology for collaborative tasks and 45% of the students used it for selfassessment exercises. less use appears to have been made of technologies to contact lecturers (39%) or other students (34%) or to revise for examinations (38%). this is shown graphically in figure 1. figure 1. summary of student responses on technology uses for self-directed learning the behaviours reported by students in this study differ in this respect from the recent learner experience study by conole et al. (2006, 156), which found ‘a marked lack of enthusiasm for vles’. in the present study the vle was regarded as the first reference point for e-learning and was in general appreciated by students, although there are some issues arising with regard to the way in which the online environment is being used, which we discuss later in this article. overall, there appears to be little differentiation of response amongst students by gender, age, status or ethnic background. one area where some variation can be found is amongst usage by year of study, for example, only 26% of first year students, compared with 51% of respondents in other years, used the online library on a daily or weekly basis. 76 73 65 59 53 45 39 38 34 0 10 20 30 40 50 60 70 80 accessing information reviewing course materials managing course information communicating with friends collaborative tasks self-assessment exercises contacting lecturers revising for exams contacting other students percentage of responses figure 1. summary of student responses on technology uses for self-directed learning. 10 f. deepwell and s. malik initial themes, derived from the interviews and the survey, include insights into student expectations of what the technology can facilitate and what the lecturers should be doing. another area that has emerged is the students’ transition from a previous learning environment into their current stage of learning, which revealed emotional dimensions to the learning experience with technologies. each of these themes will now be discussed. discussion student expectations of technology for learning ‘it has just made life easier and so much more enjoyable.’ interviews with learners show the prevalence of the belief amongst students from a range of courses that e-learning is helpful. although it might have been expected, this is supported by the survey results, where students cite numerous technologies that they use frequently, including cuonline, the university’s virtual learning environment. seventy-six per cent of respondents used cuonline for their self-directed learning and a slightly smaller majority (65%) used learning technology to communicate with lecturers. students were asked to identify from a selection of activities what types of interaction they would like to have more of in their self-directed learning time. most popular, with 71% selecting the option, was online discussion. fifty-six per cent wanted more collaborative group work, 40% wanted interactive exercises, just 22% games and 17% simulations. in response to a selection of skills they found the technologies most useful in developing, creative thinking was the most popular choice (see figure 2). figure 2. what kind of skills are technologies most useful for developing? although learners welcomed online materials as an enhancement, the survey also indicates that students preferred face-to-face communication; 45% of respondents placed great value on face-to-face learning. this was also backed up by the interview data, for example, the comment by one interviewee: but my issue is: its very good all this online learning but having somebody standing by my side is more beneficial. you get the personal feedback if you’re just stuck or having real difficulty. 0 20 40 60 80 100 120 140 160 180 critical thinking problemsolving rote learning analytical thinking creative thinking figure 2. what kind of skills are technologies most useful for developing? alt-j, research in learning technology 11 this leads on to the next theme emerging from the data, namely what the expectations are of lecturers. student expectations of lecturers for facilitating learning through technology our analysis so far has revealed contradictory expectations by students of the role of the lecturer in facilitating student learning through technology. whilst the students did seem to be applying themselves adequately in terms of time spent on independent study, as reported in the preliminary findings above, and wanting more technology-based activities in their courses, they did not seem to be working on their studies independently. there appears to be a strong reliance on the module tutors for information and guidance on learning at all levels of study. the interviews revealed variations in the amount of guidance offered by lecturers. whilst one student (performing arts, year 1) reported that she and her student peers had to work out for themselves what and how to study, another (engineering management, postgraduate) had been provided with a workbook that offered highly detailed guidance on weekly activities, similar to that expected on a distance learning course. within this range of tutor direction other support mechanisms were also mentioned. for instance, one of the students ‘couldn’t praise the library staff highly enough; couldn’t fault them’ in their level of support for her, and yet she did not see this as providing sufficient support in the module and more face-to-face encounters were wanted with the lecturers. amongst the students interviewed there was a high expectation of almost instant tutor feedback, and frustration when this was not received. the lecturers were the first resort for assistance, and only when they were unavailable did the students report turning to other sources of help with their learning, e.g. their friends, colleagues on the course and others. for example, one student (information technology for engineers, postgraduate) regretted the fact that the lecturer was not online simultaneously with the student when he logged on at midnight. in these circumstances he had to contact his friends for advice. another indication of student reliance on lecturing staff is the declared dependence on the supplied reading. for example, one student stated that the most useful independent learning strategy for him was to read the handouts and books provided by the lecturer. in order to pass the module it was felt sufficient by another student to read the directed reading, although he also acknowledged that to get a good mark more independent research through the library portal was required. further contradictions emerge with respect to contact with lecturers. on the one hand, students felt that the computer was being used as a substitute for contact. as one student put it: ‘cuonline is fine but sometimes it’s nice to talk to a person instead of a computer’. they reported that they wanted feedback from lecturers regularly, yet only a relatively low number of the respondents (23%) contacted their lecturers on a weekly basis. also, the students in the interviews reported that they did get feedback if they asked for it. however, it appears from both the survey data and the interviews that feedback received via e-mail or cuonline does not have as high a value placed on it as face-to-face feedback. the final point with regard to student expectations relates to how much the lecturers engaged with learning technologies. when asked in the survey what improvements they would like to see with regard to learning technologies many of the students wanted lecturers to place more information in cuonline. there was also a call for more direction from the lecturers about what to do in the online space. one of the students recounted how the online systems were presented at the start of his course and ‘looked really good’, but he then found that the lecturers supporting the part-time course he was on did not make use of it and it was ‘just an empty web page’. these accounts of student experiences suggest a need for the development of metacognitive skills and 12 f. deepwell and s. malik management of learner expectations and, at the same time, a need for greater consistency of usage across educational programmes. using technologies to ease transition ‘i’m not afraid of learning if i know the direction.’ there appears to be a further issue around the provision of guidance in the use of learning technology, relating to the process of transition. there was evidence of this in the shift from college/ school into university, as well as from employment onto postgraduate courses. the interview data reveal the wide-ranging expectations and needs of first year students toward the university experience as a whole. students reflected on their prior learning experiences as compared with university: ‘i find it very strange, coming from a very structured kind of life at college, coming to university is very, very strange’ and ‘it’s different from high school you don’t get a lot of help from teachers’. another student had come to study after years working in an industrial environment where he had been coached in time management, research skills and prioritizing assignments. he found that his expertise in these metacognitive skills meant that he was prepared for the course, although he wondered how others coming directly from education were managing, since the part-time postgraduate course did not to him seem to provide development in these skills. overall, the students felt that more support should be provided to ease the transition, although the students were not very clear what this should comprise. the interviews revealed a reliance on the support of friends to help them through identified crisis moments. with regard to learning technology, there appears not to be a skills deficit (e-mail, websearching, chat). the reported skill levels from the survey are encouragingly high, with 95% of respondents rating their skill level in using technology for self-directed learning as intermediate or above, leaving just 5% as beginners. however, there appears to be a lack of orientation in the use of the learning technologies preferred by the university, including confusion over how to use the new cuonline, the library portal for e-journals and an eportfolio system (pebblepad) for personal development planning. conclusions and further work ‘i don’t know what we would do without it … to be honest.’ within the context of this study we have explored the experiences of students in relation to their use of learning technology on campus-based courses. the focus on self-directed learning has been helpful in drawing attention to what students are actually doing when they study, rather than to what they feel they should be doing. this focus enabled us to highlight three themes: student expectations of technology, of their lecturers’ use of technology and the role of technology in easing the transition stages of learning. within each of these themes we have identified implications for the institution, the lecturers and the students. the number of hours spent on self-directed study is reportedly quite good for the majority of students (although self-reporting of estimated time is not a true measure) and the skill levels to work with technology are largely present. however, there seems to be a lack of direction in what they are meant to be doing in this self-directed time and there is frequent mention of the role of the lecturers in determining the activities that are undertaken. this suggests that we are some way from merriam and caffarella’s (1991) notion of self-directed learning, where the learner takes responsibility for their own studies. it also highlights the crucial role that tutors play in the learning process at this university, whether face-to-face or mediated by technology. alt-j, research in learning technology 13 interestingly, this was also the case where a detailed study pack was provided. the key to this appears to be the high importance given to lecturer feedback. feedback tells students how successfully they have tackled their academic work. it also provides an indication of the quality of their work and raises motivation among students towards their learning. we have found, however, that learners appear to have high expectations of the speed at which lecturers can provide feedback to students and the detail included and these expectations are perhaps not managed sufficiently within the blended learning process, which still accounts mainly for faceto-face ‘contact hours’. there is also perhaps a case for developing a greater appreciation of online feedback. the research furthermore indicates that there is a need for more direct guidance on what and how to use technology for learning more independently. the significance here is that this advice is appreciated and acted upon by students if it comes from course tutors (as distinct from librarians, for example). where this plays out most appears to be in the transition stages of learning. students need help in adapting to university life and becoming autonomous learners. pitkethly and prosser’s (2001) research into the ‘first-year experience’ advocated that universities need to understand the specific needs of their learners and develop appropriate courses of action. our research implies that these needs are present at different stages in the learning cycle, not just in the first year. with regard to learning technology usage, our analysis is starting to indicate what might be a useful line of development. in this paper we have presented some of the findings of our study. we were attempting to gain a greater understanding of what the reality of sdl is for students today. in this we have explored students’ expectations of the technology itself, the role of the tutor and a particular aspect of the learning experience in higher education, namely transition. the use of technology is pervasive, but the use of technologies for learning is more problematic and there is a need to clarify how students are using them successfully and creatively in order to give a sense of which direction to take for those who are still finding their way. it is hoped that the findings of this study will suggest some guidelines for students about the use of available learning technologies. another outcome of the study will be guidelines to lecturers in improving the opportunities for students to use learning technology in their self-directed learning. notes 1. jisc learner experiences of e-learning phase 1 (see http://www.jisc.ac.uk/whatwedo/programmes/ elearning_pedagogy/elp_learneroutcomes.aspx). 2. lex: learner experiences of e-learning (http://www.jisc.ac.uk/whatwedo/programmes/elearning_ pedagogy/elp_lex.aspx). references bonk, c.j., k.-j. kim, and t. zeng. 2004. future directions of blended learning in higher education and workplace learning settings. in handbook of blended learning, global perspectives, local designs, ed. c.j. bonk and c.r. graham. san francisco, ca: pfeiffer publishing. conole, g., m. de laat, t. dillon, and j. derby. 2006. an in-depth case study of students’ experiences of e-learning – how is learning changing? in proceedings of the 23rd annual ascilite conference: who’s learning? whose technology? http://www.ascilite.org.au/conferences/sydney06/proceeding/ pdf_papers/p127.pdf (accessed june 3, 2007). creanor, l., k. trinder, d. gowan, and c. howells. 2006. the learner experience of e-learning. final report. jisc. http://www.jisc.ac.uk/uploaded_documents/lex%20final%20report_august06.pdf (accessed november 13, 2007). herrington, i., t. reeves, and r. oliver. 2006. a model of authentic activities for online learning. in interactions in online education, ed. c. juwah, 91–103. london: routledge. 14 f. deepwell and s. malik land, r., and s. bayne. 2005. screen or monitor? issues of surveillance and disciplinary power in online learning environments. in education in cyberspace, ed. r. land and s. bayne, 165–178. london: routledgefalmer. merriam, s.b., and r.s. caffarella. 1991. learning in adulthood. a comprehensive guide. san francisco, ca: jossey-bass. oliver, m. 2006. new pedagogies for e-learning? alt-j 14, no. 2: 133–4. pitkethly, a., and m. prosser. 2001. the first year experience project: a model for university-wide change. higher education academy research and development 20, no. 2: 185–198. salmon, g. 2005. flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions. alt-j 13, no. 3: 201–18. sharpe, r., g. benfield, e. lessner, and e. decicco. 2005. scoping study for the pedagogy strand of the jisc e-learning programme. jisc. http://www.jisc.ac.uk/uploaded_documents/scoping%20study% 20final%20report%20v4.1.doc (accessed june 3, 2007). sharpe, r., g. benfield, g. roberts, and r. francis. 2006. the undergraduate experience of blended elearning: a review of uk literature and practice. higher education academy. http://www.heacademy. ac.uk/assets/york/documents/ourwork/research/literature_reviews/blended_elearning_full_review.pdf (accessed december 4, 2007). stake, r.e. 1995. the art of case study research. london: sage. alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 155–170 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020155–16 © 2007 association for learning technology doi: 10.1080/09687760701472177 mediating between practitioner and developer communities: the learning activity design in education experience isobel falconer* glasgow caledonian university, uk taylor and franciscalt_a_247096.sgm10.1080/09687760701472177alt-j research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis152000000june 2007isobelfalconerisobel.falconer@gcal.ac.uk the slow uptake by teachers in post-compulsory education of new technological tools and technology-enhanced teaching methods may be symptomatic of a general split in the e-learning community between development of tools, services and standards, and research into how teachers can use these most effectively (i.e. between the teaching practitioner and technical developer communities). this paper reflects on the experience of transferring knowledge and understanding between these two communities during the learning activity design in education project funded by the uk joint information systems committee. the discussion is situated within the literature on ‘mediating representations’ and ‘mediating artefacts’, and shows that the practical operation of mediating representations is far more complex than previously acknowledged. the experience suggests that for effective transfer of concepts between communities, the communities need to overlap to the extent that a single representation is comprehensible to both. this representation may be viewed as a boundary object that is used to negotiate understanding. if the communities do not overlap a chain of intermediate representations and communities may be necessary. finally, a tentative distinction is drawn between mediating representations and mediating artefacts, based not in the nature of the resources, but in their mode and context of use. introduction in recent decades a host of technological tools—such as online conferencing, interactive whiteboards, wikis, and so on—have become available, providing new opportunities to enhance learning (conole, 2004; conole & dyke, 2004; collis & moonen, 2005). in particular, one supposed benefit of learning technologies is their potential for enabling teaching practitioners (teachers, lecturers, tutors, hereafter referred to as ‘practitioners’) to implement new teaching methods informed by recent theories of learning (see, for example, conole & oliver, 2007). yet there is * caledonian academy, glasgow caledonian university, cowcaddens road, glasgow g4 0ba, uk. email: isobel.falconer@gcal.ac.uk 156 i. falconer little sign that education has changed in any fundamental way despite substantial institutional investment in information and communications technology (collis & van der wende, 2002). although we now know a great deal about how to teach for effective learning, on the whole didactic modes with a focus on transmission of knowledge still predominate in post-compulsory education—the potential of new technologies to enable new styles of learning is not yet being realised (britain & liber, 2004; littlejohn, 2004). this may be because the benefits of e-learning as perceived by technical and educational developers (those, generally in support departments, whose role is to promote teaching excellence, develop teaching and learning materials, advise faculty on pedagogic issues, etc.) are not sufficiently clear or easy to communicate to practitioners, as suggested by beetham (2004; uk joint information systems committee [jisc], 2006), or because they are not well aligned with existing institutional structures, values and rewards as pointed out by seufert and euler (2004) the solution to these problems is complex and likely to involve institutional change strategically focusing on transformation in learning and teaching practice, support for staff and students engaging in these new practices and implementation of new physical and virtual learning environments (littlejohn & peacock, 2003). a number of these strategic changes are proving difficult for institutions. for example, linking empirically based theory to practice, and recognition of valid outcomes aligned to pedagogical theory, is challenging for practitioners within current organisational constraints. for example, teachers may feel prevented from implementing collaborative teaching methods by the need to align with institutionally imposed learning outcomes based on a didactic approach (falconer et al., 2007). similarly, implementing necessary changes in working practices and reward structures to provide sufficient levels of practical support for development of e-learning practice are proving particularly problematic. but all these developments require some sort of translation that makes what the practitioner does on the ground, and its implications, understandable and meaningful in practical terms to those—technical, administrative and managerial—providing the support, facilities and infrastructure (beetham, 2002). conversely, the implications of new theories of teaching, and the affordances of new technological tools, need to be translated into terms that teachers and lecturers can put into practice. this translation is frequently achieved through some sort of representation of practice or theory; for example, a case study or model. but it is apparent that different types of representation are likely to be understandable to different types of users—a translation of a teaching practice that informs university management will probably be meaningless to a technical developer. thus the nature of the communities involved and the purpose of the communication are important factors when considering how best to mediate and translate between them. mediating representations the term ‘mediating representations’ is widely used in knowledge systems and cognitive science disciplines (for example, boose, 1990). i define it as encompassing any mediating between services and learning activities 157 representation that enhances communication among participants and improves their understanding of a concept or practice. in an empirically based study, beetham (2001) discussed the role and usability of such representations in supporting change in teaching practice. her examples of representations ranged from case studies, guidelines and toolkits to computer software and databases (see table 1 for a fuller list). she implies a distinction between representations of practice and representations of learning, which might equate to a distinction between representations originated by teachers and representations originated by educational theorists or psychologists. but in her analysis she treats these on the same footing, being entirely concerned with the end users; in this case, teaching practitioners, educational developers and learning technology support staff. while beetham concentrates only on very limited types of end user, she makes the valuable point that even a single user might have a number of uses or types of interaction with a representation. for example, a teacher might use a case study as a source of information about a pedagogic model, as a straightforward description of a lesson that they can use in their own practice, or as the inspiration for a new teaching approach. alternatively, the user might require different types of representation to support different sorts of interaction. for example, while a case study might be well suited to providing inspiration for a new teaching approach, it might be difficult to derive an operable description of a lesson from it—a lesson plan might be better suited to the table 1. lists of beetham’s and sharpe et al.’s mediating representations (based on beetham, 2001; sharpe et al., 2004) and conole’s mediating artefacts (from jones & conole, 2006) beetham’s and sharpe et al.’s mediating representations conole’s mediating artefacts review narratives guidelines case studies staff development material peer dialogue case study expert guidance framework/toolkit—models, templates, etc. knowledge building article/report networked communication software—learning lesson plans software—activity shell tips and tricks software—learning object demonstrations information resource answer gardens project/service frequently asked questions databases schema bibliographies toolkits concept maps models patterns roughly speaking, conole characterises resources by their type and the list runs from the concrete and contextualised to the abstract and conceptual, while sharpe et al. and beetham characterize them by their format running from text-based to software-based (littlejohn et al., 2006). this table is not designed to imply a mapping between the two. 158 i. falconer latter purpose. the alignment between the purpose of a representation and its form is, then, crucial to its success. beetham’s interest in representations as supporting change leads to a classification of six types of interaction (inform, adopt, adapt, create, guide and comment) that implies a learning cycle as practitioners embed the new ideas conveyed by the representation into their practice. similar cycles have been suggested by beetham (2002), conole and oliver (2001), and mayes and fowler (fowler & mayes, 1999; mayes, 2001). their work thus emphasises that teachers are in the position of learners when changing practice, and representations need to support the stages of a learning cycle. beetham (2001) further characterises the representations along a spectrum that runs from text, such as articles, reviews and case studies, which are highly informative, to software tools such as assessment tools, which are highly adoptable but less informative. such a counterposition between inform and adopt does not seem immediately obvious, although it parallels the payoff between the acquirability and expressiveness of a representation noted by boose (1990). boose uses language as an example: a language that is sufficiently sophisticated to convey complex ideas may be difficult to acquire, while a more basic language is easier to learn but lacks the power to express complex concepts or fine distinctions between concepts. beetham suggests that these counter-positions are driven by time, or rather lack of time. in a follow-up paper, beetham (2002) adds the nuance that lack of time is a manifestation of lack of suitable institutional and reward structures. she also discusses explicitly the types of representation that might be used by five different communities of users (learner, teacher, resource developer, educational developer, strategic developer), but she has no further empirical evidence to show how these representations do in fact mediate in interpreting practice across community boundaries. this problem was taken up by sharpe et al. (2004), who investigated the transfer of knowledge between the education researcher and practitioner communities. however, as in beetham’s previous survey, the main participants were educational developers and practitioners rather than education researchers. the most effective representations cited appeared to be those around which they could interact with colleagues (i.e. originators and end users were part of the same, or overlapping, communities)—a conclusion that is embedded within our definition of a mediating representation as enhancing communication. however, it leaves unanswered the issue of how representations originating in the academic education research community make their way into the educational developer community in the first place. not explored by beetham, or sharpe et al., but relevant to this last issue, is the idea of ‘boundary objects’. boundary objects are a concept that originated in actornetwork theory (star, 1989) and were adopted by wenger (1998) and tuomi-grohn and engestrom (2003). boundary objects are common to the communities between whom they mediate, but are viewed in different ways by those communities. they act to bring about coordination and negotiate understanding between the communities, and form a learning space in their own right (edwards, 2005). to act as a boundary object in this way, the practices of both communities must encompass the object, although not necessarily from the same perspective (i.e. the communities must have mediating between services and learning activities 159 a common boundary and not be too far removed from each other). if we consider beetham’s or sharpe et al.’s representations as boundary objects, we can see why they are most effective when used for communication between closely overlapping communities. conole (2006; jones & conole, 2006) discusses a very similar list of resources to beetham (see table 1), but refers to them as ‘mediating artefacts’ in an approach that is rooted in activity theory; her work has been followed up by oliver (2006). rather than considering the resources as integral to the practices of a community, conole views them as tools for making decisions. mediating artefacts will help practitioners decide which technological tools and educational theories are appropriate to their situation and aims. providing support for this choice, it is argued, will help bridge the gap between recent educational theory and implementation in practice. despite their different perspectives, however, many of the conclusions of conole’s survey of mediating artefacts, parallel beetham and sharpe et al.’s. for example, she draws a continuum from open theoretical maps at one end to restrictive but easy-to-use software tools at the other end. and she notes that the use of models of educational theory has been criticised because practitioners may not understand how to apply them effectively—she is essentially making the same point as sharpe et al. when they find that the intervention of educational developers is usually necessary between academic theorist and practitioner. like beetham and sharpe et al., conole is essentially concerned with practitioners as the users of mediating artefacts, although she does mention learners and technical developers occasionally. however, if we view the dialogplus toolkit, developed by conole and others, as a representation (conole & fill, 2005), we begin to see how it might work to mediate between the educational developer and practitioner communities. dialogplus is an online tool that presents practitioners with a range of options to help them develop effective learning activities (see figure 1). however, the theory of learning that underpins the process is hidden from the user. figure 1. screenshot of the dialogplus toolkit intended to help practitioners design technology enhanced learning activities (http://www.nettle.soton.ac.uk/toolkit/activities/default.aspx?action=create) in this case, the educational developer has not intervened in person to interpret a model or theory to the practitioner, but has represented their understanding in the toolkit. this highlights the structural difference between viewing such resources as tools and viewing them as representations: the representation view entails communication between at least two participants, not necessarily both from the same community, whereas the tools view hides the role of one of the participants, embedding it within the tool. we note here a confusion of terminology—in that activity theory, upon which conole bases her account, divides mediation by artefacts into ‘sign mediation’ and ‘tool mediation’ (for example, engestrom et al., 1999), with conole’s ‘artefacts’ corresponding to ‘tools’ while sharpe et al.’s and beetham’s use of ‘representations’ corresponds more closely to ‘signs’ with its emphasis on communication. however, other authors who have drawn on activity theory, such as leadbetter (2004), have validly used ‘mediating artefacts’ to signify resources that are a focus for communication. in the terms of the current paper, leadbetter’s use of resources would count as mediating representations. 160 i. falconer other work that uses the language of mediating representations, such as that of goodyear and steeples (1998), similarly concentrates on communication between members of a single community, or closely overlapping communities, of practitioners and educational developers. however, beetham (2004), sharpe et al. (2004), seufert and euler (2004), oliver (2004), and scott (2004) have all highlighted the variety of different stakeholders in learning technology, and the multiplicity of their purposes in using it. these authors have begun to articulate the range of user types, including teachers, learners, technical support staff, technical developers, educational developers, librarians, managers and evaluators. thus there are many more communities between whom communication is necessary than have been discussed hitherto in the context of mediating representations, although goodyear et al. (2004) in their work on patterns begin to address the gap between the more widely divergent communities of practitioners and technical developers. one of the conditions for the success of the learning activity design in education (ladie) project was to bridge this gap. methodology this paper takes a case study approach, which uses the experience of the ladie project (http://www.elframework.org/refmodels/ladie; funded by the jisc) as a test bed for exploring a number of issues of the role of representations in transferring knowledge and understanding between different communities. figure 1. screenshot of the dialogplus toolkit intended to help practitioners design technology enhanced learning activities (http://www.nettle.soton.ac.uk/toolkit/activities/default.aspx?action= create) mediating between services and learning activities 161 the project faced the challenge of translating teaching practice in post-compulsory education across a community boundary into terms that were useful to technical developers. these experiences are situated in recent literature on the role of mediating representations and artefacts, which provides the concepts used to analyse reflections. the ladie project: initial approach the jisc-funded ladie project was a contribution to the development of the jiscdest e-framework for education and research being developed jointly by the jisc and the australian department for education, science and training (dest) (http:/ /www.e-framework.org/). the e-framework supports a service-oriented approach to the development and delivery of education. this approach reflects a move from monolithic, powerful but inflexible virtual learning environments, towards a more flexible linking of individual ‘service’ components (such as chat and assessment tools, etc.) via loosely coupled, standards-based, interfaces. essential to its development is the documenting of user requirements and processes in a coherent way, and to use these to derive a set of interoperable network services that conform to appropriate standards. the requirements are being documented in the form of ‘reference models’ or ‘domain maps’ that will enable members of the community to collaborate in the development of service components that meet their needs (olivier et al., 2005). thus, it is suggested, a ‘reference model’ will provide a mediating representation that enables practitioners and technical developers to communicate about their requirements (figure 2). figure 2. a reference model is a mediating representation that enables users to communicate their requirements for the services system a reference model provides: ● an abstract model of what has to be accomplished to meet the needs addressed. ● the description of the chosen means of implementing this model—defined by existing and developing technologies, specifications and standards (olivier et al., 2005). the aim of the ladie project was to develop a learning activity reference model that was firmly based in practical experience of teaching and learning (http:// www.elframework.org/refmodels/ladie). in this case, the needs were defined as those figure 2. a reference model is a mediating representation that enables users to communicate their requirements for the services system 162 i. falconer of effective reusable learning activities. therefore ladie planned to use a ‘top-down’ and ‘bottom-up’ approach, documenting the requirements of effective learning activities in the form of ‘use cases’ through a series of practitioner workshops, and mapping these onto existing interoperability standards and specifications. use cases are a way of specifying the behaviour requirements of a system, generally in a simple text form (cockburn, 2001). they are mediating representations that serve to enhance communication about the ways in which the user needs a system to behave (figure 3). figure 3. abbreviated example of a use case specifying the requirements for running an online discussion conference derived from the ladie project so we can see, already, in the initial project methodology, recognition that two mediating representations, the use case and the reference model, and an intermediate community of learning technologists, seem necessary to bridge the communication gap between practitioners and technical developers, in developing the web services system. the communication looks more like figure 4 than figure 2. figure 4. communication structure and mediating representations implied by the proposed ladie methodology the ladie project: experience the project eventually documented 16 learning activities, gathered mainly from three one-day practitioner workshops. the 30 participants were a roughly equal mix of practitioners from higher education and further education, educational developers plus a couple of secondary school teachers. they were known to be e-learning enthusiasts and were invited for this reason. the aim was to document activities covering a wide range of possible teaching approaches and subject disciplines. interviews with individual practitioners were used to fill in disciplinary gaps in workshop participation. the initial intention was to get the practitioners themselves to document their activities in the form of use cases, by providing them with templates similar to that of figure 3, and guiding them through the writing process. we tried this in our first workshop, but it proved extremely difficult to implement in the time available. even with the scaffolding provided by the templates, and experts on hand to mediate the process, considerably more experience of writing use cases would be necessary before this community could utilise this type of representation of their practice effectively. the main two difficulties, even for this relatively expert group of participants, were in: ● decontextualising the learning activity and concentrating on the structure of the process; and ● separating the roles and actions of the practitioner in the process from the aims of the learner. as noted in other studies, the practitioner community is accustomed to representing its practice in heavily contextualised forms, generally case studies (sharpe et al., 2004; falconer & littlejohn, 2006). in the ladie project it seemed that this customary practice was too far removed from that of the learning technologists developing a reference model for the gap to be bridged by a single representation type, a use case, without a great deal of further and peripheral communication that would bring the communities closer together. given the time constraints highlighted by beetham (2001), this means that the use case representation is unlikely to be effective in this user context. mediating between services and learning activities 163 use case summary teacher runs an online a discussion conference with students primary actor (and goal) teacher to administer the discussion conference successfully other actors (and goals) students to engage in dialogue and develop concepts to pass the course moderator to keep conference running smoothly, both socially and organisationally preconditions 1 system with conferencing facilities main success scenario 1 teacher sets up small group conferences in system and defines student groups 2 system attaches students to teacher-defined discussion groups 3 students discuss asynchronously in small group conferences 4 teacher monitors small group conferences 5 system saves records of conferences 6 students access saved conferences for future work extensions 1a teacher is overworked and will not have time to monitor conferences 1a1 teacher appoints student moderators for conferences 3a some messages are inappropriate 3a1 moderators edit or remove messages 4a student participation is low 4a1 teacher changes conference structure so new messages are more easily visible (e.g. by doing away with subconferences) figure 3. abbreviated example of a use case specifying the requirements for running an online discussion conference derived from the ladie project 164 i. falconer in subsequent ladie workshops we abandoned this approach, and, instead, asked participants to document their activities as case studies. we provided templates that prompted them to detail the teaching context and learner needs explicitly, alongside a stepwise description of what they, and their students, did during the activity (http:/ /www.elframework.org/refmodels/ladie/guides/larm_pedagogy30-03-06.doc). this approach seemed to work far better from the practitioner’s perspective. we then took these case studies and derived use cases from them, consulting back to the casestudy authors to check our interpretations. the main communication around the developing use cases, however, was among the learning technologists in the ladie team. this demonstrated clearly that we were two different, although overlapping, communities of learning technologists— practitioner focused and technically focused (the project team was deliberately put together in this way to implement the top-down, bottom-up approach)—and that a substantial amount of communication between the two was necessary to develop a representation that was meaningful and fit for purpose for both. this supports sharpe et al.’s suggestion that representations of practice should be ‘enhanced by their participation in collaborative activities’ (sharpe et al., 2004). problems and misunderstandings negotiated through this process included: ● granularity and aggregation—the granularity of a learning activity was the subject of much debate at workshops but most participants settled on something that had a definable beginning, middle and end, generally in the form ‘provide students with figure 4. communication structure and mediating representations implied by the proposed ladie methodology mediating between services and learning activities 165 some information, set a task based on the information, report on the result of the task’. such activities comprised a number of different subactivities (e.g. information gathering, discussion forum, report writing), but these tended to be similar from one activity to another. in writing the use cases it often proved more effective to disaggregate these subactivities and re-aggregate the functionality required of similar ones together into generic use cases such as the discussion shown in figure 3. ● breadth versus depth—the initial conception of the practitioner-focused team members was that the documented learning activities should cover as wide a range of service components as possible. however, negotiation over the form of the use cases demonstrated that this supposition was based on a false assumption of the sophistication of existing interoperability standards. what the technically focused team members needed was to develop a core reference model based on an in-depth analysis of the requirements of a limited number of core functions such as discussions and quizzes. the purpose of the case studies, from their viewpoint, was to identify which these core functions were. thus, collaborating over development of use cases proved a very effective means of bridging this partial gap between the two overlapping communities of learning technologists. thus, the eventual structure and mediating representations that proved necessary in the ladie project to provide meaningful communication between all communities of users looked something like figure 5. figure 5. communication structure and mediating representations used in the ladie project however, once the project was completed and the reference model written, these active communications disappeared. the reference model was written in three figure 5. communication structure and mediating representations used in the ladie project 166 i. falconer parallel parts, aimed at technical developers, learning technologists and practitioners, reflecting recognition that the practices and purposes of these communities are sufficiently different that a single representation would not be meaningful to all of them. once written and stabilised, it was envisaged that the reference model would mediate between users and services as in figure 1. as in the case of conole and fill’s dialogplus toolkit, the role of the originating participants (practitioners and learning technologists) is hidden, embedded within the model, and the model could be viewed as a tool for the user. in this case it might be more appropriate to consider the model as a mediating artefact, in line with conole’s perspective, than as a mediating representation that enhances communication. it is early days yet to tell whether it will be possible to use the reference model unambiguously in this way, or whether as noted by conole (2006) such a formalised representation leaves sufficient gaps of interpretation that communication between the end user and the originator or another ‘expert’ is necessary for effective use. discussion our experience as users of representations to mediate communication between a variety of communities has shown that the process is considerably more complex than is apparent from previous studies in the learning technology literature based on single or closely related communities. in particular, it seems that gaps between some communities, even those united around a common endeavour such as teachers communicating their needs to technical developers, are too wide to be bridged by a single type of representation. in the ladie project it took at least three communication steps—each centred around a different type of mediating representation, and involving two intermediate communities of participants—to migrate a concept of system requirements from practitioner to technical developer. in each of these steps the communities overlapped: practitioners–practitioner-focused learning technologists; practitioner-focused–technicallyfocused learning technologists; technically-focused learning technologists–technical developers. the overlap allowed active reciprocal communication between originators and end users around the representation—that this is necessary is for migrating concepts is suggested by sharpe et al. (2004) and by conole’s (2006) work on effective mediation, and is assumed in our definition of a mediating representation. the necessity we found for overlap between communities supports an interpretation of our representations as boundary objects that enabled a transfer of concepts from one community to another. at each step the practices of both communities encompassed the representation used, although they viewed it from different perspectives. eventually, a chain of such steps allowed us to transfer concepts from the practitioner to technical developer communities. in the ladie project the overlapping communities were sufficiently close in their practices, language and modes of thinking to be able to communicate around the representation within a reasonable time frame. the time-driven constraints on effective representations noted by beetham (2001, 2002) and conole (2006) were most mediating between services and learning activities 167 apparent at the first stage, documenting practitioners’ learning activities. they became less significant at the second and third stages where participants (project team members) were being rewarded explicitly for communicating concepts. in developing the use cases, when relatively free from time constraints, we had representations that seemed both information rich and highly adoptable by learning technologists. however, the same representations proved neither adoptable nor informative to the practitioner community, and this underlines the obvious point that representations need to be tailored to the practice of their participants. when we consider the number of different combinations of communities that may need to communicate or collaborate during e-learning projects, the complexity of finding appropriate representations becomes apparent. finally, i suggest that, as the mediating representations reach a stable form and cease to be the locus of reciprocal communication, it may be possible to view them instead as ‘mediating artefacts’ or tools, in line with conole’s work. thus, as implied by our initial overview, the distinction between ‘mediating representation’ and ‘mediating artefact’ is one of the processes employed and the context of use, rather than of the nature of the resources. conclusions i have used the experience of the ladie project as a case study of the processes involved in communicating ideas between different communities. these processes have been articulated in terms of the concepts of mediating representations, mediating artefacts and boundary objects. our experience supports the following conclusions: ● the process of using representations to communicate between different communities is considerably more complex than hitherto recognised in the learning technology literature. ● for effective communication, communities must overlap, and the representations used should be common to both, forming ‘boundary objects’. ● if communities do not overlap, then a series of communications between intermediate, and overlapping, communities is necessary using representations appropriate to each step. ● a significant number of different representation forms are needed to support communication between the large number of combinations of communities that may need to communicate or collaborate during e-learning projects. ● the counterposition between informative and adoptable, or acquirability and expressiveness, noted in the literature is a consequence of time constraints, as suggested by beetham. for effective communication, time has to be made available. ● in a ‘mediating artefact’ the input of the originating community is embedded and hidden, but is still present: the distinction between ‘mediating representation’ and ‘mediating artefact’ depends on the context of use, rather than on the nature of the resource. 168 i. falconer acknowledgements the author thanks the other members of the ladie project team, especially allison littlejohn, gráinne conole, ann jeffrey and peter douglas, for their support on the project, and thanks the jisc and the university of dundee for financial and institutional support. an earlier version of this paper was given at the networked learning conference, lancaster, uk, april 2006. references beetham, h. (2001) how do representations of practice enable practice to change?, educational developments, 2(4), 19–22. beetham, h. (2002) developing learning technology networks through shared representations of practice, in: c. rust (ed.) improving student learning through learning technologies (oxford, oxford centre for staff and learning development). beetham, h. (2004) review: developing e-learning models for the jisc practitioner communities: a report for the jisc e-pedagogy programme (bristol, jisc). available online at: http:// www.jisc.ac.uk/uploaded_documents/review%20models.doc (accessed 22 june 2007). boose, j. h. (1990). knowledge acquisition tools, methods, and mediating representations, in: h. motoda, r. mizoguchi, j. h. boose & b. r. gaines (eds) proceedings of the first japanese knowledge acquisition for knowledge-based systems workshop: jkaw-90 (tokyo, ohmsha ltd). britain, s. & liber, o. (2004) a framework for pedagogical evaluation of virtual learning environments, jisc e-learning pedagogies programme report (bristol, jisc). available online at: http:// www.jisc.ac.uk/uploaded_documents/jtap-041.doc (accessed 22 june 2007). cockburn, a. (2001) writing effective use cases (boston, ma, addison-wesley). collis, b. & van der wende, m. (2002) models of technology and change in higher education (twente, center for higher education policy studies), 85. collis, b. & moonen, j. (2005) an on-going journey: technology as a learning workbench, faculty of behavioural sciences, university of twente. available online at: http://bettycollisjefmoonen. nl/rb.htm (accessed 22 june 2007). conole, g. (2004). report on the effectiveness of tools for e-learning: report for the jisc commissioned ‘research study on the effectiveness of resources, tools and support services used by practitioners in designing and delivering e-learning activities’ (bristol, jisc). available online at: http:// www.jisc.ac.uk/uploaded_documents/report%20on%20the%20effectiveness%20of%20tools %20v5_martin_oliver.doc (accessed 22 june 2007). conole, g. (2006) mediating artefacts to guide choice in creating and undertaking learning activities, in: s. banks, v. hodgson, c. jones, b. kemp, d. mcconnell & ch. smith (eds) proceedings of the fifth international conference on networked learning 2006 (lancaster, lancaster university). conole, g. & dyke, m. (2004) what are the affordances of information and communication technologies?, alt-j, 12(2), 111–122. conole, g. & fill, k. (2005) a learning design toolkit to create pedagogically effective learning activities, journal of interactive media in education, 8. available online at: www-jime.open. ac.uk/2005/08/conole-2005-08.pdf (accessed 22 june 2007) conole, g. & oliver, m. (2001) decision-making resources for embedding theory into practice, in: t. okamoto et al. (eds) proceedings ieee international conference on advanced learning technology: issues, achievements and challenges, 6–8 august (washington, dc, ieee computer society), 327–330. conole, g. & oliver, m. (eds) (2007) contemporary perspectives in e-learning research: themes, methods and impact on practice (london, routledgefalmer). mediating between services and learning activities 169 edwards, r. (2005) contexts, boundary objects and hybrid spaces: theorising learning in lifelong learning, paper presented at the 35th annual scutrea conference, university of sussex, uk, 5–7 july. available online at: http://www.leeds.ac.uk/educol/documents/142037.htm (accessed 28 january 2007). engeström, y., miettinen, r. & punamäki, r. (1999) perspectives on activity theory (cambridge, cambridge university press). falconer, i. & littlejohn, a. (2006) mod4l report: case studies, exemplars and learning designs. mod4l project report for jis c. available online at: http://mod4l.com/tiki-download _file.php?fileid=2 (accessed 29 january 2007). falconer, i., beetham, h., oliver, r., lockyer, l. & littlejohn, a. (2007) mod4l final report: representing learning designs. report from the jisc mod4l project. available online at: http:// mod4l.com/tiki-download_file.php?fileid=7 (accessed 22 june 2007). fowler, c. j. h. & mayes, t. (1999) learning relationships from theory to design, alt-j, 7(3), 6–16. goodyear, p. & steeples, c. (1998) creating shareable representations of practice, alt-j, 6(3), 16–23. goodyear, p., avgeriou p., et al. (2004) towards a pattern language for networked learning, paper presented at the networked learning conference 2004. available online at: http://www.networkedlearningconference.org.uk/past/nlc2004/proceedings/individual_papers/goodyear_et_al.htm (accessed 22 june 2007). jisc (2006) circular for design for learning funding call. available online at: http://www.jisc.ac.uk/ index.cfm?name=funding_01_06 (accessed 22 june 2007). jones, c. & conole, g. (2006) who will own the new vle? sharing practice, problems and alternative solutions, paper presented at the proceedings of the 23rd annual ascilite conference: who’s learning? whose technology? available online at: http://www.ascilite.org.au/conferences/ sydney06/proceeding/pdf_papers/p95.pdf (accessed 22 june 2007). leadbetter, j. (2004) the role of mediating artefacts in the work of educational psychologists during consultative conversations in schools, educational review, 56(2), 133–145. littlejohn, a. (2004) the effectiveness of resources, tools and support services used by practitioners in designing and delivering e-learning activities: final report (bristol, jisc). available online at: http://www.elearning.ac.uk/ resources/effectivefinal/view (accessed 22 june 2007). littlejohn, a. & peacock, s. (2003) from pioneers to partners: the changing voices of staff developers, in: j. k. seale (ed.) learning technology in transition: from individual enthusiasm to institutional implementation (lisse, swets and zeitlinger). littlejohn, a., falconer, i. & mcgill, l. (2006) characterising effective e-learning resources, computers and education. available online at: http://www.sciencedirect.com/science?_ob= articleurl&_udi=b6vcj-4m21sw7-1&_user=128597&_coverdate=10%2f05%2f2006& _alid=521904214&_rdoc=1&_fmt=full&_orig=search&_cdi=5956&_sort=d&_docanchor=&v iew=c&_acct=c000010621&_version=1&_urlversion=0&_userid=128597&md5=f6fdcb190 e3576b4a7f79a80ade2e108(accessed 5 october 2006). mayes, t. (2001) learning technology and learning relationships, in: j. stephenson (ed.) teaching and learning online (london, kogan paul), 16–26. oliver, m. (2004) assessing the relevance of the review of e-learning theories, frameworks and models and the mapping table to evaluators. jisc e-learning models desk study, (bristol, jisc), 7. available online at: http://www.jisc.ac.uk/uploaded_documents/evaluators.doc (accessed 22 june 2007). oliver, r. (2006) ronline: mediating artefacts to support technology-facilitated learning, in: a. bunker & i. vardi (eds), critical visions: thinking, learning and researching in higher education (milperra, nsw, herdsa), 259–267. olivier, b., roberts, t. & blinco, k. (2005) the e-framework for education and research: an overview. paper prepared on behalf of dest (australia), jisc-cetis (uk). available online at: http://www.e-framework.org/resources/eframeworkrv1.pdf (accessed 22 june 2007). 170 i. falconer scott, j. (2004) assessing the relevance of the review of e-learning theories, frameworks and models and the mapping table to designers. jisc e-learning models desk study (bristol, jisc), 9. available online at: http://www.jisc.ac.uk/uploaded_documents/designers.doc (accessed 22 june 2007). seufert, s. & euler, d. (2004) sustainability of e-learning innovations—findings of expert interviews. scil report 2 (st gallen, scil university of st gallen). sharpe, r., beetham. h. & ravenscroft, a. (2004) active artefacts: representing our knowledge of learning and teaching, educational developments, 5(2), 16–21. star, s. l. (1989) the structure of ill-structured solutions: boundary objects and heterogeneous distributed problem solving, in: l. gasser & m. huhns (eds) distributed artificial intelligence (vol. ii) (london, pitman). tuomi-grohn, t. & engestrom, y. (eds) (2003) between work and school: new perspectives on transfer and boundary-crossing (london, pergamon). wenger, e. (1998) communities of practice: learning, meaning, and identity (cambridge, cambridge university press). reviews edited by philip barker researching into teaching methods in colleges and universities by clinton bennett, lorraine foreman-peck and chris higgins, london: kogan page, 1996. isbn: 0-7494-1768-4, 136 (+ vii) pages, paperback. £14.99. although the range of research undertaken by universities and colleges covers a very wide spectrum, each of the books in this kogan page series is narrowly focused on a particular form of research, that of small-scale 'insider research'. this is defined as research into teaching and learning which is undertaken by tutors with the specific aim of improving their teaching practice. the series is intended to be quick and easy to read, to provide starting points for research into teaching practice, and to give pointers to ways in which outcomes may be recorded and published. it aims to deal with contemporary issues affecting teachers in colleges and universities, and to include a range of research methods with suggestions for how these might be used or adapted in particular circumstances. each book in the series shares a version of the opening and concluding paragraphs. the opening paragraph introduces a concept of insider research based on the notion of the reflective practitioner of education: this model takes the view that teachers should play an active role in constructing and reconstructing knowledge with the aim of improving teaching. 'research' is related mainly to the rating system defined by the higher education funding council for england (hefce). the text is careful not to offer a blueprint for research activity. this would be unlikely to be of practical value, given the very wide range of /contexts and activities in which readers would / be engaged. it seeks instead to provide pointers / to how to begin reflective research activity, and ' to the limitations of small-scale research, and stresses the need for objectivity. the kinds of group which might be suited to study, and their particular perspectives, are listed, as are the differences between the traditions of qualitative and quantitative research methodologies in education (there is also a useful appendix which introduces the use of it in qualitative analysis). there is a synopsis of methods for data collection and analysis. chapters 2 and 3 deal mainly with the changing role of the lecturer, and highlight some contemporary directions for research into teaching. this again is mostly concerned with the situation in the uk, and highlights policy initiatives such as national vocational qualifications (nvqs), the influence of teaching-quality assessments, and problems of semesterization. the move towards a mass education system is related to problems in teaching large groups. aspects of research into student learning include strategies for learning and students' perceptions of learning and of assessment. the extent to which students are able to take charge of their own learning is contrasted with views about the extent of teacher guidance. open, resource-based, and distance learning are detailed, and the relative advantages and disadvantages are discussed. chapter 4 deals with approaches and issues arising from vocationalization of the curriculum, again specifically related to the uk. however, the rich literature on students' learning styles and its relationship to styles of 69 reviews teaching and to personality is not indicated, and the referencing might have been enhanced by the addition of several key texts in this area. the use of computer technology for the delivery of educational materials is discussed, though it is associated only with models of interaction classified as 'surface' or 'deep', and little is said about the evaluation of learning outcomes. the culture of educational organizations, as a focus for research, is a useful contribution in chapter 5. an explanation of the various cultures, borrowed from charles handy's perspective on institutional cultures, is informative. linked with this is discussion of institutional change and the management of change. the concluding chapter is a synopsis of the processes leading to getting research published. this deals with the process from both the author's and publisher's points of view, and offers good advice on the negotiation of contracts. finally, there is an annotated list of publishers. for those new to publishing, this chapter provides an excellent overview. the book is well referenced, and reading lists are helpfully annotated with comments on the scope and suitability of the texts chosen. each chapter also suggests a number of focused research tasks: these are displayed as separate tables. this is a small book of only six comparatively short chapters. however, as an introduction to help tutors to begin researching into aspects of their own work, it is clearly written and informative at an appropriate level. it provides sound advice, and good pointers to the kinds of research which might be undertaken by tutors, though it does focus on the research-assessment culture which is current in the uk. it will perhaps be of particular value to those who are expert in the practice of their subject but not necessarily in its pedagogy, and who are approaching formal, publishable research for the first time. david durling, edge ltd, hartlepool designing electronic performance support tools — improving workplace performance with hypertext, hypermedia and multimedia by g. h. and e. f. stevens, englewood cliffs, nj: educational technology publications, 1995. isbn: 0-87778-283-0. 265 pages, paperback. $39-95. electronic performance-support systems (epsss) are becoming an important mechanism for providing on-the-job, just-in-time training at appropriate points of need within both academic and industrial/commercial organizations. they are also important as a support facility within organizations which undergo dynamic change in order to meet the demands placed upon them. as the benefits of this approach become more widely realized, there is a growing demand for their use. unfortunately, in the past there has been a distinct shortage of appropriate tools and methodologies to enable the rapid creation and application of performance-support techniques. this book is therefore a welcome publication. the book is organized into ten chapters, a glossary, a bibliography and a subject index. the major topics covered include a discussion of human performance in relation to electronic performance support; the important constituent technologies that make up an electronic performance-support environment; the anatomy of an application; planning information structures and file structures; navigation; enduser interface design; the design and use of embedded tools; implementation techniques; hardware and software issues; and a series of case studies that deal with performance support and organizational change. chapter 1 starts off with a discussion of a range of background considerations relating to the need for and use of epss tools within organizations. an important aspect of this chapter is the mapping of generic performancesupport tools onto organizational functions. in chapter 2 the authors provide a description of various approaches to the design and development of epss facilities. they suggest an object-oriented approach involving rapid prototyping, and the chapter provides a useful checklist for selecting a development tool based on the characteristics of the application involved and the environment in which it exists. application characteristics are further discussed in chapter 3, which goes into hypermedia and object-orientation in greater depth, while chapter 4 deals with the essential planning issues that need to be considered as part of hypermedia design. these issues include analysis of opportunities for performance support; planning how information in an application should be organized and labelled; and selecting file structures which will be effective for the application. 70 alt-} volume 4 number 3 chapter 5 considers the important topic of navigation. it outlines some of the common navigational problems associated with applications and design tools. some of the challenges that designers have to meet are discussed; they include end-user orientation; focusing users' attention; and providing contextual understanding of the information in an application. tools available to manage these challenges are described. chapter 6 deals with end-user interface design for performance-support applications primarily for screen displays. it provides a set of guidelines for creating a consistent and effective user-interface. the authors also present a ninestep systematic methodology for interface design, along with checklists for using this approach. embedded tools to improve end-user productivity and performance are important aspects of all epss facilities. this issue is discussed in chapter 7. typical examples of some of the types of tool considered include note-taking utilities; calculation and analysis tools; and auditing tools that can be used to track users' navigation and use of facilities. previous chapters in the book have each presented essential design strategies for all application areas. in chapter 8 the authors elaborate on selected guidelines for specific types of application. the three broad types of application considered are instructional, on-line reference and job task automation. the issues considered within the examples presented include optimizing navigational design; feedback; information structuring; user access; identifying what should be automated; and helping clients to define and communicate their expectations. chapter 9 deals with the hardware and software tools needed to create hypermedia and multimedia applications in epss environments. within the context of selecting hardware and software resources, the authors consider three main perspectives: what is needed to create applications, what is required in order to run applications, and what is the best way to disseminate them. the mpc (multimedia pc) standard is briefly discussed, and there are also some useful sections on the design implications of using digital audio and video. the final chapter is used to present three case studies relating to epss and organizational change. they are rooted in the areas of claims processing in a health/medical insurance company; company reorganization to achieve an optimal information environment based on the use of epss; and fleet management within a large automobile leasing company. these case studies serve to illustrate the positive benefits of epss facilities, and the importance of understanding the interaction between epss tool design and organizational structure. nowadays, the use of epss facilities is starting to challenge conventional approaches to the use of computers in education and training. for those wishing to get involved in this area, this book contains some very useful tips and techniques. philip barker, university of teesside educational telecommunications, 1996: proceedings of ed-medial ed-telecom 96, edited by patricia carlson and fillia makedon, charlottesville va: association for the advancement of computing in education (aace), 1996. isbn: 1-880094-22-3. 395 pages, paperback. it would be impossible, even if the space available here were much greater, properly to describe the contents of this weighty volume, the proceedings of ed-telecom 96, an international conference focusing on educational telecommunications, held in boston massachusetts in june this year. there are 58 full papers and 46 short papers, covering the complete range of telematics in education, everything from evaluating distance-learning programmes to creating a virtual campus. these papers were selected from over 300 presentations at this leading conference. as with all the proceedings i have ever come across (and this, of course, is inevitable) there is good and bad. in these proceedings, however, there is very good and very bad. many of the short papers are simply too short to tell us anything except that someone is working in such and such an area (though this can be useful in certain circumstances, i agree); others manage to put forward innovative ideas about which one would have liked to hear much more. but the real interest in this volume lies in the full papers which make up 337 of the 395 (large-size) pages. and here, the standard is even more variable. at one extreme, there are some exceptionally badly written, badly constructed descriptive-pieces of the type which now makes me tear my hair out. they often open with something like: "the use of the microcomputer to provide an integrated 71 reviews learning environment for teaching a variety of disciplines has received much attention in recent years.' they then continue on the lines of: "we did this, then we did that, and the students liked it [positive feedback is the correct expression], though there were a few students who had some reservations (see figure 29 for a graphical representation).' at the other extreme, however, this volume contains some excellent analyses which reveal the real problems of distance education, and which suggest interesting solutions, as well as identifying and warning against not so obvious traps into which the novice distance teacher might fall. the contents are not grouped by theme, and unfortunately there is no introduction to explain the thinking behind the order in which the papers appear (i could not work it out). this is surprising because the editors mention in their short preface the 22 major areas supposedly covered by the conference. but this criticism is not very significant: a glance at the list of titles (or at the list of authors at the back of the book, if one feels that is important) is all one needs in order to find the material most appropriate to one's area of activity. and anyone working in educational telematics will, for sure, find quite a lot of food for thought here even if they almost choke on some of it. so, if you are working in this field, or even just interested in it, i suggest you get an inter-library loan form as soon as possible. gabriel jacobs, university of wales swansea the adventures of kc in london (five cdroms) by adrian brockett, arabesque multimedia, 9 woodland road, darlington, county durham dl3 7bj. tel. 01325 350700. of all multimedia training applications, language-teaching programs are conspicuously unsuccessful. this is usually for one of two reasons. often, they are designed by programmers without too much input from teachers, with the result that, while the appearance is attractive with a variety of clever activities, the pedagogical content is low. the alternative is that they are designed by teachers with little experience of programming, and the result is that however worthy the content, the programming is dull and fails to take advantage of the opportunities offered by multimedia format. the adventures ofkc in london is quite exceptional in that it appears to be designed by a writer who is neither programmer nor teacher. as a result, this program is unfriendly to use, and the language content (what little there is of it) lacks accuracy, purpose and direction. the program is an efl teaching application designed for use by native arabic speakers who have completed four or five years of language instruction in schools. the material provided for evaluation consisted of a cd-rom with a. tutorial program and one out of five teaching programs or 'adventures'. the tutorial program not only explains the contents of the teaching programs and how they work, but also sets the 'scenario' which is that the crown jewels have been stolen from the tower of london by thieves who are, for no obvious reason, the final six letters of the english alphabet. the jewels have to be rescued by kc, who is a suitcase. the adventure provided for review was number 3 in the series, sport in britain. the other four adventures are arriving in london, shopping in london, the houses of parliament and the tower of london. much of sport in britain is not an adventure at all. instead, the teaching program consists of four short texts in english, under 1,100 words in total, which appear in writing and which can be listened to. there are translation and notepad options. there is also a context icon which, when you highlight a word in the text, repeats the word and the text verbatim quite why i have not discovered yet. there is also a facility to allow users to record any part of the text and compare their speech with that of a native speaker. the native-speaker recordings have a marked background hiss and are not performed by actors. there are questions at the end of each text, three or four questions of a gap-fill, multiple-choice or yes/no type. when you get enough of these right, you can progress to a series of crosswords, listening and gap filling games where kc recovers the jewels if you get the answers correct. the program runs very, very slowly and there is no way of bailing out once it has started. the real problems, however, lie in the teaching material. four short reading texts and a handful of questions is very little indeed for a whole cdrom. the text itself is dull and artificial. this sort of exchange, taken from the text called 'conversation', is typical: nick: oh, i think football is much more exciting than cricket, don't you? rob: yes, i far prefer it. you can't beat a good game of football. 72 alt-j volume 4 number 3 nobody speaks like this in real life. further, the writing and the thought it represents are sloppy. i found myself instinctively reaching for a red pen to correct it. for example: athletics and water sports have traditionally taken place in summer because of the weather. floods may occur in winter because of the rain, but athletics in summer is not an inevitable consequence of heat and sunshine. the texts also include frequent factual inaccuracies. i shall confine myself to pointing out that the winners at wimbledon do not receive large gold plates, the charity shield is not played between the winners of the fa cup and the league cup, and a member of the royal family does not always present the fa cup. the questions on the text and there are incredibly few of them are ill thought out. material for the answers is not always included in the text, and at least two multiple-choice questions have more than one possible correct answer with the result that one is marked wrong. the answers to the games appear to rely on memorization of the texts rather than ability in english. of course, this may appeal to arabic learners who will otherwise find the content and level of this program a great challenge. as one of my colleagues commented on trying the program: 'if they try and learn from this sort of material in the middle east, it's no wonder they're so bad at english.' this program will not teach anybody english, although you might enjoy playing with the games if you happen to know english already. james milton, university of wales swansea being digital by nicholas negroponte, london: coronet books (hodder and stoughton), 1996. isbn: 0-340-64930-5. 249 pages, paperback. £6.99. this is quite an intriguing book from which i have often quoted material. as director of the media lab at mit, its author nicholas negroponte should be quite well-known to anyone who has delved into or who is working in the field of digital multimedia. in this book, negroponte attempts to popularize the digital world of computers by writing an accessible and readable pocket book that can be read as easily on. a beach as it could in a library. the accessibility of the book is reflected in the fact that it has been translated into over thirty different languages. being digital is organized into seven basic parts: an introduction; eighteen chapters arranged into three basic theme areas; an epilogue; an 'after words'; and an acknowledgements section. the introduction is subtitled the paradox of a book'. here negroponte explains why he has committed his thoughts, ideas, comments and arguments to an 'old-fashioned book' when an electronic book (of the type described later in chapter 5) might have been more appropriate. one of the three reasons he gives for publishing on paper rather than electronically is simply that 'there are just not enough digital media in the hands of executives, politicians, parents, and all those who most need to understand this radically new culture'. within the main body of the book, part 1 deals with the fundamental nature of digital information, its importance, how it can be moved from one location to another, and the implications of shifting from an analogue (or 'atom-based') culture to a digital (*bit-based') world of information. part 2 is devoted to enduser interfaces and human-computer interaction. it discusses numerous topics, many of which were extensively researched in the early days of the media lab. some of the more important of these are interactivity, sensory richness and machine intelligence; multimodal interfaces (such as 'put-that-there'); interaction devices; video conferencing; use of gestures; graphical persona; virtual reality; telepresence and remote manipulators; speech input/output; and intelligent interface agents. part 3 deals with 'digital life'; it discusses the various ways in which 'being digital' is likely to influence our lives in the years to come. the major issues considered in this section include the postinformation age; time-shifting and video-ondemand; wearable media; household robots; and email and the growing demand for and use of the internet as a knowledge resource. learning using lego/logo systems is also discussed, as is the importance of personal computers as educational tools. according to negroponte (p. 220), through computers, 'ten years from now, teenagers are likely to enjoy a much richer panorama of options because the pursuit of intellectual achievement will not be tilted so much in favour of the book-worm, but instead cater to a wider range of cognitive styles, learning patterns, and expressive behaviors'. in the epilogue, negroponte expresses some 73 reviews concerns about the negative implications of the digital age unemployment, the misuse of technology, software piracy, and so on but is optimistic about what he believes are the four redeeming qualities of being digital: its ability to achieve decentralization, globalization, harmonization and empowerment. finally, in the acknowledgements section, the author gives a concise person-oriented summary of both the history of the media lab and of the production of being digital. although this book does not say anything exceptional or revolutionary, it does provide a helpful synthesis of many useful and powerful ideas relating to digital media and their use by current and future societies. sadly, there is no bibliography, and not even so much as a url link to a further reading list. however, in his excellent review of the hardback version of the book (isbn: 0-34064525-3), joseph (the computer jour/ia/, volume 39, number 3, 1996, page 252) suggests that being digital should be 'required reading for every computing student and professional'. i concur. philip barker, university ofteesside the java sourcebook — a complete guide to creating java applets for the web by e. arnoff, new york: john wiley and sons, 1996. isbn: 0471-14859-8. 498 pages, paperback. £19.99. despite its popularity and pervasiveness, the world wide web (and many applications based on it) has a number of constraints. two of the most significant of these are the limited interactivity of web pages, and the often low responsiveness of many web sites with respect to handling multimedia information. these limitations are inherent in the way in which client/server applications are implemented using the http approach. in order to overcome these (and some of the other) limitations, sun microsystems introduced in 1995 the java programming language for web applications and the hotjava browser. according to sun, java is a 'simple, object-oriented, distributed, interpreted, robust, secure, architecture-neutral, portable, high performance, multi-threaded and dynamic language'. this book provides a useful introduction and guide to the java language and its use for the creation of web applets. the book is organized into four basic parts. part 1 (the first two chapters) is the shortest; it describes how to get and install the java development kit, run the hotjava browser (and netscape), and visit java-enhanced sites on the web. part 2 is the largest section (comprising chapters 3 to 12) and covers the nuts and bolts of java. part 3 (chapters 13 to 17) deals with building applets using a 'line-by-line' approach. finally, part 4 (three appendices) summarizes the reference material available in the sdk and api packages, the standard libraries for handling i/o and network communication. part 2, then, contains the details of the java language itself. the early chapters in this section provide a basic introduction to java (and object orientation), a description of how to build java applications using an appropriate development platform (unix, mac or pc), and a review of java-language basics. subsequent chapters deal with more detailed treatments of various programming topics such as flow of control; objects and classes; interfaces; packages; managing data; handling exceptions; and threads. having covered the basics, the author uses part 3 to cover the creation of applets that can be run within a java-capable browser such as hotjava or netscape 2.0+. the first chapter in this section provides an applet overview, and the following chapters deal with applet basics, user interfaces, graphics and multimedia. the important issues covered in these five chapters include in-line applets in html (the '<applet>' tag); parameter passing; alignment; the awt display model; layout classes; containers and components; user-interface components provided by awt; event handling; frames, windows and dialogues; drawing methods; use of images; using mediatracker; and sound clips and animation. although this is a technical book, it is extremely readable, and is relatively free of typographical errors and coding mistakes. it contains lots of well-thought out examples to illustrate the points being made. in fact, anyone wanting to learn how to create dynamic, interactive web pages will find it a useful starting point provided they have at least some programming experience. unfortunately, however, despite its usefulness, what this book does not do to any great extent is to point to any of the significant weaknesses of this approach to creating applets. for those wishing to follow up this issue, a useful diving-in point would be the discussion presented by ted lewis in the june 1996 edition 74 alt-} volume 4 number 3 of ieee computer (volume 29, number 6, pages 8-9). worth mentioning too, for those who are familiar with c programming, is that there is a companion book entitled java for c/c++ programmers (isbn: 0-471-15324-9; also published by wiley). philip barker, university of teesside learners on the superhighway by keith yeomans, leicester national institute of adult continuing education (niace), 1996. isbn: 1872941-96-6. 112 pages, paperback. £6.50. this is a report of a study-tour in north america, undertaken in 1994 by keith yeomans, of organizations and individuals involved in educational communications. the aim of the research was to identify areas of good practice and strategy which might be relevant to future developments in the uk and europe. in the opening chapter, yeomans sets out a summary of his conclusions. among the most important of these are that policy makers and researchers in the field should raise awareness of the educational potential of telecommunications, arrange for local and regional demand to be quantified, develop strategies to minimize costs, set up local learning networks, ensure that the communications industry make full use of cross-sectoral partnerships, and so forth. these and related suggestions are expanded in the next five short chapters. it would be hard to find many people who would disagree with the views expressed. who, besides the odd luddite, would not concede that the information superhighway represents a new opportunity for teachers, and for students of all ages and in all locations? yeomans does identify certain interesting differences in attitude between europe and the usa, such as the fact that in europe, with our public-service culture, we find it hard to understand the american commercial attitude towards using the internet for education. and he does make some points which still need to be hammered home even though so many of us have aching arms, such as the danger that telematic education, despite its potential for wide dissemination of learning, may in practice widen the gap between the haves and have-nots. but the overall impression one has after reading this report is that one has been surprised about very little its author has said. although the book is 112 pages long, the body of it ends on page 55. all the rest is given over to two appendices: contributor profiles, and documents. the first is a 40-page descriptive list of north american companies, educational establishments and people contacted by yeomans. the second is a list of books, reports and the like which he collected during the period of his research. such lists can come in useful, but there is a great deal in them which i am sure nobody in the uk will ever need, and, in the case of the documents, would ever be able to obtain if they did need them, other than by contacting yeomans personally. the reader will have gathered that i have not been terribly convinced by the usefulness of this publication for alt members. but it would be unfair and misleading to end my review negatively. for a start, yeoman's pedigree as a specialist in the field is not in doubt indeed, he is experienced, fairly well known and has done some excellent work. certainly, a considerable amount of effort has gone into writing this report. further, despite my reservations, some people may read it with interest, and may find it quite useful if writing about telematics in the usa, and/or to cite in learned articles (for that, all the right words and phrases are there for the taking). finally, £6.50 is hardly expensive when compared with the price of some commercial reports which run to hundreds of pounds, which are written in cliches and platitudes calculated to provoke intense irritation, and which contain no new material at all. gabriel jacobs, university of wales swansea using activities in training and development by leslie rae, london: kogan page, 1996. isbn: 07494-1891-5.214 pages, paperback, £19.99. this book catalogues and discusses the range of activities available for use in learning and training. although it is not therefore about computer-based learning directly, i wondered whether it might act as a source of ideas or a set of guidelines for activities supported or promoted by computer-based learning technology. there are two main sections. the first six chapters deal with factors common across all types of activities, beginning with a brief theoretical justification based on the ideas of kolb, and honey and mumford, of why learning through activity can be effective. this is followed by a discussion of the practical decisions required in the planning, selection and 75 reviews preparation of activities, such as ways of constituting groups and the use of instruction sheets. the rest of the section then concentrates on techniques for observing, recording and reviewing the actions of participants. we are reminded that 'in many cases as much, if not more learning emerges from a review or discussion of the activity as from the activity itself. the section therefore considers strategies and methods for observing interactions between pairs and within groups. there is a chapter on the use of activity analysis for recording the incidence of agreed kinds of behaviour such as giving information, summarizing, and so on. all of this is presented at a firmly practical level that avoids the complexities usually introduced by texts on behaviour analysis and research methods, and is clearly pitched at practitioners rather than theorists. the last chapter in the section considers different ways of providing feedback to participants during debriefing sessions. the author identifies this as the most essential stage of all, and this chapter looks the most useful. section 2 consists of seven chapters describing specific types of activities. most of these are concerned with interpersonal activities such as introductions and energizers, one-to-one activities such as role-play, group questions and discussions, and experiential activities such as t-groups for raising sensitivity and awareness. these chapters have a common structure that includes a description of each activity, membership and timing requirements, planning and design, usage, examples and variations. there is plenty of practical guidance about different ways in which the activities can be organized and about the avoidance of possible problems. much of this might be useful to lecturers in higher education considering alternative ways of running group tutorials, although there is little about dealing with stale and weary students, and the touchy-feely kinds of activities would be of limited use in many disciplines. there is an amusing section on how to run a bad discussion (e.g. start by going on at length with your own opinions and then interrupt and disagree with others' contributions), but then it is one thing to know the right and wrong things to do and another to apply them in practice. those of us interested primarily in computersupported activities would probably turn first to the chapters on structured group activities, and case studies and simulations. rae's only references to computers appear here, with a brief mention that simulation used to be an unwieldy matter before computers were available to present reports, statistics and other information, or to simulate more dynamic virtual worlds, or to act as analytical or communication tools. given that many authorities now recognize that the success and effectiveness of computer-supported activities depends as much on their social and organizational context as on their supporting technology (e.g. laurillard, 1992; kaye 1992, pp. 14-15), rae's practical emphasis on effective preparation and organization is appropriate. it is not a good idea to give students a computer-based activity and just leave them to get on with it. the checklists in the book might help in avoiding otherwise unforeseen difficulties, and the activity descriptions might help in the design of new activities, for example by applying the list of different kinds of documents that might be used in an in-tray activity. clearly the book is written from a great deal of personal experience of working with most of the activities it covers. it provides a gentle introduction to the wide range of activities available to lecturers and teachers. but in attempting to discuss learning activities in a subject-free way, independent of specific disciplines, the book tends towards far too many generalities. at the end is a list of published activity resources. the topic areas they cover, such as customer care, assertiveness, counselling, managing stress, communication and leadership, reinforced my feeling that the book is really orientated towards the development of personal and interpersonal skills for managers, trainers and the like. it is difficult to relate this to learning technology in the computer-based sense. references kaye, a. (1992), collaborative learning through computer conferencing, the najaden papers, berlin: springer verlag. laurillard, d. (1992), 'learning through collaborative computer simulations', british journal of educational technology, 23, 164-71. robert ward, university of huddersfield 76 alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 119–130 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020119–12 © 2007 association for learning technology doi: 10.1080/09687760701482218 barriers and enablers in integrating cognitive apprenticeship methods in a web-based educational technology course for k–12 (primary and secondary) teacher education michele d. dickey* miami university, usa taylor and francis ltdcalt_a_248098.sgm10.1080/09687760701482218alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis152000000june 2007micheledickeydickeymd@muohio.edu the purpose of this study is to investigate the integration of a cognitive apprenticeship model into an educational technology web-based course for pre-service primary through secondary teacher education. specifically, this study presents an overview of methods, tools and media used to foster the integration of a cognitive apprenticeship model, and presents the types of barriers and enablers encountered when attempting to participate in a computer-mediated cognitive apprenticeship. the methodological framework for this investigation is a qualitative case study of an educational technology course for pre-service primary through secondary teacher education. the findings of this study reveal that various tools, methods and media were used to varying degrees of success to foster cognitive apprenticeship methods in a web-based learning environment. the goal of this study was to better understand the pragmatics, suitability, affordances and constraints of integrating cognitive apprenticeship methods in a web-based distance education course for teacher education. introduction with the widespread adoption of the international society for technology in education standards in the united states for integrating technology into primary and secondary education and an increasingly greater emphasis being placed on educational technology in primary through secondary education worldwide, it is important that educational technology programs for pre-service primary through secondary teacher education students find innovative methods and models for the effective use of technology in teaching and learning. however, this poses a challenge * educational psychology, mcguffey hall 100d, oxford, oh 45056, usa. email: dickeymd@ muohio.edu 120 m. d. dickey for pre-service teacher education programs attempting to provide distance education. traditionally, pre-service teacher education students become enculturated into an authentic practice by taking university classes and by participating in field experiences in primary and secondary classrooms. typically these experiences culminate in a student-teaching experience. however, this process changes with the use of web-based courses or programs, which may in many ways seem antithetical to this enculturation process. all too often, web-based instruction is reduced to uninspiring content presented in a linear text-based format (daviault & coelho, 2003). transposed lectures and text-based content do little to model or enculturate preservice teachers into the practice of integrating technology for their future teaching practice. yet, because cognitive apprenticeship methods foster the emergence of practical skills within an authentic setting, finding methods of integrating cognitive apprenticeship methods in web-based environments may provide an effective means to help enculturate pre-service primary through secondary teacher education students into the practice of integrating technology into their future teaching practices. the purpose of this study is to investigate the integration of a cognitive apprenticeship model into an educational technology web-based course for pre-service primary and secondary teacher education students in an american university. specifically, this study presents an overview of methods, tools and media used to foster the integration of a cognitive apprenticeship model, and presents the types of barriers and enablers encountered when attempting to participate in a computer-mediated cognitive apprenticeship. the goal of this study is to better understand the pragmatics, suitability, affordances and constraints of integrating cognitive apprenticeship methods in a web-based distance education course for teacher education. theoretical framework pre-service k–12 (primary and secondary) teacher education students are in the process of being educated within a field of practice. it is important to reinforce that notion, yet difficult to achieve in web-based courses. according to brown et al. (1996), prior to the emergence of formal schooling, the apprenticeship model was the most common means of educating learners in fields of practice. cognitive apprenticeship methods consist of several stages (collins et al., 1989). first the teacher identifies and models the expert-level practices or strategies for learners. next, the teacher designs ‘scaffolds’ that will support learners in their attempt to apply expert practices or strategies. the term scaffold as applied to teaching is used to describe the temporary support framework provided to support learners in the process of extending competencies (bransford et al., 2000). scaffolding may take many forms and serve various functions. hannafin et al. (1999) identified four types of scaffolding to include: conceptual, meta-cognitive, procedural and strategic. conceptual scaffolding provides learners with hints and recommendations. meta-cognitive scaffolding helps students plan, organize, reflect and regulate during an activity (brown, 1998). procedural scaffolding includes support that helps students to perform a task, action cognitive apprenticeship methods in web-based educational technology 121 or process. strategic scaffolding provides supports that help learners in applying knowledge, principles and experiences to various and new situations. scaffolding can be provided in many forms and may include various resources and media. although scaffolding is an important element, coaching is also an important part of cognitive apprentice methods. whereas scaffolding may be related to individual areas, the role of coaching is much broader with the purpose of overseeing the student learning process (collins et al., 1991). the teacher coaches the students in the process of adopting and using the expert practices or strategies by offering guidance during the entire process and by critiquing performance and offering feedback throughout. while students are adopting and learning new practices, they are strongly encouraged to articulate their reasoning in using various strategies and practices. they are encouraged to reflect on their own experiences as well as those of their classmates. during this process the teacher gradually reduces or fades the support (coaching) as learners move towards independently applying what they have learned to new challenges. because cognitive apprenticeship methods foster the emergence of practical skills within an authentic setting, it may provide an effective means to enculturate preservice primary and secondary teacher education students into the practice of integrating technology into their future teaching practices. cognitive apprenticeship methods may be particularly well-suited for a web-based environment for educational technology because students must use educational technology to learn about educational technology. literature review there have been several notable studies on the integration of technology and cognitive apprenticeship methods (järvelä, 1995; liu, 1998; liu & pedersen, 1998; hendricks, 2001; liu & hsaio, 2002), although relatively few explicitly investigate the use of cognitive apprenticeship methods in a web-based environment. research has been conducted about the integration of cognitive apprenticeship methods in the design of internet-delivered online video case studies to support early literacy instruction for pre-service teachers (schrader et al., 2003). although the distribution of the video case studies was online, the course itself was taught in a traditional classroom format. the findings of the study by schrader et al. indicate that video case studies of best practices helped initiate conversation and allowed learners multiple perspectives. in contrast, saarenkunnas et al. (2000) investigated the integration of cognitive apprenticeship methods in the form of case-based conferencing for a finnish and american collaborative course for pre-service teachers. the underlying technology that supported the collaborative course relied upon the use of both web-based asynchronous conferencing tools and video conferencing. although the focus of the study by saarenkunnas et al. (2000) is primarily on the web-based discussions, the findings indicate that the integration of cognitive apprenticeship methods did enhance learning. both of these findings indicate value and potential for integrating cognitive apprenticeship methods in web-based instruction. it is also worthy to note that in both the initiatives by schrader et al. (2003) and saarenkunnas et al. (2000), designers 122 m. d. dickey and researchers relied on the use of existing technologies such as digital video, video conferencing and discussion forums rather than developing new technologies for the integration of cognitive apprenticeship methods for teaching and learning. although there have been numerable studies of the integration of technology and cognitive apprenticeship methods (järvelä, 1995; liu, 1998; liu & pedersen, 1998; hendricks, 2001; liu & hsaio, 2002), few of these studies have focused on the use of cognitive apprenticeship methods in a web-based environment. as an increasing number of programs worldwide offer courses online, it is important to explore how innovative methods such as the cognitive apprenticeship model can be integrated into online learning environments to enhance teaching and learning. the purpose of this study is to investigate the integration of a cognitive apprenticeship model into an educational technology web-based course for pre-service primary and secondary teacher education students in an american university by examining the methods, tools and media used to foster a cognitive apprenticeship model and to examine the types of barriers and enablers encountered during the process. methodology the methodological framework for this investigation is an interpretive case study (merriam, 1998). the reason for choosing an interpretive framework for this study is to provide a better understanding of the technology affordances and constraints of web-based learning environments, and to generate analysis about the impact of technology tools, media and methods for fostering a cognitive apprenticeship model in a web-based learning environment. setting and content this setting for this investigation is a web-based technology integration course entitled integrating technology and education practicum (i-tep). i-tep is a onecredit semester course for teacher education students of junior/senior standing. the focus of i-tep is to model technology integration methods and strategies for preservice k–12 (primary through secondary) teachers. currently i-tep consists of five thematic modules. each module focuses on both instructional design and the integration of a technology tool to enhance learning. modules include learning theories (cognitive apprenticeship model), communication tools (email, electronic mailing lists, web logs, instant messenger, and discussion groups), presentation tools (the web, powerpoint, graphic and audio software, and analogue and digital video), productivity tools (databases, spreadsheets, word processors, and graphical organizers) and the web. in each thematic module, students are assigned readings and exemplars to view. students are then required to respond to the readings and exemplars by using various communication tools (web logs, email, instant messenger tools, etc.). next, students complete assignments in which they design lesson plans integrating a specific type of technology tool to enhance learning. concurrently, students also produce educational media using the tool integrated in the lesson plan. cognitive apprenticeship methods in web-based educational technology 123 data collection the data presented in this study were collected during the summer 2005 semester. observation data were gathered from 42 students (three graduate and 39 undergraduate students) from 11 different teacher education licensure programs. all of the students in this study gave consent for participating in the study, and all participants’ names have been changed to preserve anonymity. additionally, 21 students participated in email interviews. findings modeling: tools, media and methods cognitive apprenticeship methods were manifested and integrated in the web-based environment in several ways. modeling was supported through the use of text-based models and exemplars of lesson plans. additionally, students were presented with models and exemplars of a variety of educational media created using various technology tools. modeling was also provided for fostering both skill and cognitive development through a series of digital videos and flash animations and text-based instructions called over my shoulder (oms). the oms media were not skill-based tutorials, but instead were designed to demonstrate internal processes the instructor went through to create educational media. the oms videos consisted of video captures of the instructor’s computer screen while she created educational media using a particular tool. the sound track consisted of the instructor using a ‘thinkaloud’ protocol to reveal the thought process in which the instructor engaged while creating educational media. the videos and animations were specifically not skill tutorials because they purposefully included errors made by the instructor along with articulated problem-solving strategies the instructor used to overcome problems and errors that occurred. for example, one assignment required students to create a webquest using their choice of web-editing software. students could select from a variety videos or animations in which the instructor used the corresponding web editing software (macromedia dreamweaver, microsoft frontpage, netscape composer, etc.) to create a webquest. the purpose of the oms media was to allow learners look over the instructor’s shoulder and to see and hear the internal processes the instructor when through while creating educational media. the oms flash animations were similar, but instead of a sound track relied on thought balloons to articulate internal processes. the oms videos and animations purposely included some common mistakes and other problems encountered while creating educational media. the instructor modeled and explained how she dealt with problems and issues that occurred while working with technology. by watching, listening and even following along with the videos and animations, students observed the instructor modeling problem-solving and troubleshooting techniques while creating educational media with various technology tools. the purpose of the oms media was to illustrate techniques and skills, but more importantly to model methods and techniques for problem-solving while using technology tools. 124 m. d. dickey the oms media was also designed to provide various levels of scaffolding. the oms videos provided the greatest amount of scaffolding through the use of both video and sound. the flash animations offered less scaffolding. learners could view the process, but instead of a soundtrack thought-balloons were used to articulate internal processes. the text-based instructions provided the least amount of scaffolding. enablers. all of the students enrolled in i-tep had their own computers. the majority of students in this study had broadband internet access either from their homes or from locations in or near the university. the university provides discount pricing for broadband internet access for all students living on-campus and in the local town of the main campus. additionally, the university provides greatly discounted prices for students purchasing microsoft office software as well discount pricing for some macromedia and adobe products. the oms materials were offered in many formats. they were created in three file formats (audio video interleave, quicktime, and macromedia flash file format) to accommodate various web browser plug-ins. students had the availability of downloading the compressed oms materials from the course web site. additionally, students could request a set of cds, a dvd, or vhs tapes of all of the oms videos and animations. barriers. requests for cds and dvds of the oms materials occurred throughout the course. during the first week of the course, only five students requested cds and one student requested a dvd of the oms materials. however, as the course progressed, more students began requesting copies due to the time it took to download the files. all of the videos on the course web site were compressed and none of the students reported problems with expanding files; however, many reported having to a wait one or two hours while material was downloaded. although initially there were more requests for cds than dvds, by the end of the semester there was equal demand for copies in each format. there were, however, no requests for vhs tapes. during the span of the course, there were no reports of the course web site being inaccessible; however, occasionally some of the resource and reference web sites were unavailable. while occasionally a student would note this, none of the students reported this as a barrier to access. instead, those students used the other resources provided. additionally, there were no reports of internet or email access being problematic during the time of the course. there were, however, eight more urgent reports of issues with accessing and/or downloading the videos. eight students complained about the amount of time it took to download the media. it is interesting to note, however, that six of these complaints were expressed through email the evening before assignments were due. when these students were questioned as to whether this issue was a problem of access to technology or of time management, two stated that time management was probably the cause. cognitive apprenticeship methods in web-based educational technology 125 i procrastinate a lot sometimes. (mindy) i should have started sooner. (carrie) the other four students failed to respond. observations of student performances through teacher–student interactions revealed that although students used all of the materials, the videos and animations seemed to have the most impact in modeling techniques and skills. during email interactions, students often referenced a specific oms video or animation. … in that section of the video when you added the picture. (cara) … like the problem you had when you uploaded the webquest (sic) [referencing the video for uploading a webpage]. (scott) i did it the same way you did, but i must have made a mistake [referencing the video for uploading a webpage]. (liz) the oms material was also the topic most discussed in student web-log postings and online interactions. in students’ final reflections, many students seemed to strongly associate the use of cognitive apprenticeship methods with aspects of modeling found on the oms videos and animations. the cognitive apprenticeship was very helpful when it came to these assignments. it was easy for me to follow and guided me along. (sara) i like the idea of using cognitive apprenticeship methods. i know that i learn so much more from watching someone else model something before i attempt it. (todd) i could watch someone perform the tasks to get an idea, but in order for me to realy [sic] learn and remember, i need to work it on my own. (beth) there were no reported problems or barriers with file formats, nor did any students request assistance for downloading plug-ins. while most students worked on the course from home, at various times, several reported using computers on campus because of having a wider range of software available and faster computers. scaffolding and coaching: tools, media and methods scaffolding refers to the temporary support framework provided to learners to support them in the process of extending competencies (bransford et al., 2000). within the itep learning environment, scaffolding was provided in many ways. first, each of the four thematic modules was designed to build upon skills learned in previous modules. the level of instructional support was also designed to scaffold learners in an attempt to foster independence and risk-taking for students. scaffolding was provided in the type of assistance provided to students by the instructor. during the first two modules, students’ email requests for assistance were answered within a time span of 2–10 hours; however, as the course progressed, students were encouraged to seek other means for assistance. as each module progressed, the response time for email requests for assistance became longer. for the final module, students had a 15– 24 hour delay in response. the purpose of imposing this delay was because, as future 126 m. d. dickey teachers, many of the students will need to find resources for problem-solving and trouble-shooting technology problems. extending response times encouraged students, particularly those students who tended to procrastinate on assignments, to search the web to find immediate answers to questions. coaching is part of the teacher’s role within a cognitive apprenticeship environment. the role of the teacher is to coach students in the process of adopting expert practices and strategies by offering feedback, guidance and prompting when needed. within the i-tep web-based environment, coaching was provided using many different strategies. first, email and instant messenger were used to answer initial questions and help learners get acclimated to the learning environment. a web-log also provided a means of guidance and feedback. because the course required students to create educational media, each student had access to their own individual online web folder. the instructor also had full access and privileges to each student’s web folder. students saved their work in this folder. as students created media and encountered problems, the instructor was able to look at look at the work-in-progress and prompt students and offer guidance. enablers. there were no reports of students being unable to access their email. students used email as the primary tool when requesting assistance, although a few students used instant messaging tools. students received feedback via email and there were no reports of students not receiving feedback. there were five students who voiced complaints about timely responses to email on the due day of the first major assignment. however, the requests were sent two to three hours before the due date and time of an assignment. students were informed that email was answered in the order in which it was received. when asked, all five admitted to procrastinating. it is interesting to note that four students appeared to change their work habits and submitted subsequent work well before the final due date and time. all student work was uploaded into student web folders. in one of the initial assignments, students changed the permission settings on their web folder to allow the instructor access. when students encountered problems, they emailed the instructor who then could access the work and help coach the student in the problem-solving process. barriers. while email worked well for coaching, several students provided feedback at the end of the course recommending the use of a chat tool for synchronous coaching. they felt the process might have been more effective if they could get immediate access rather than using email; however, email was an important tool because it forced students to reflect and to at times seek alternative methods for problem-solving. because answers were not provided on-demand, students often had to rely on other methods. in turn, this delay was part of the scaffolding process. most k–12 teachers do not have immediate access to on-demand technical problem-solving. it could be argued that the short delay many students experienced in an asynchronous environment helped foster new avenues for problem-solving. cognitive apprenticeship methods in web-based educational technology 127 exploration, reflection, articulation: tools, methods and media within a cognitive apprenticeship framework, learners should be encouraged to explore new ideas and viewpoints, and to form hypotheses and test them (enkenberg, 2001). exploration in the i-tep learning environment was supported in several ways. learners were presented with a wide variety of models and exemplars of lesson plans and corresponding educational media. they were encouraged to look at resources critically and to reflect and articulate their ideas and thoughts to their web-logs. at the start of each module, learners were required to reflect upon their initial thoughts about the topic of the module and their opinions of the relevance of the tool-type for teaching and learning. after the module activities, learners followed up with a final reflection. the purposes of the pre and post reflections were twofold. because the course content focuses on both instructional design of lesson plans and skill development, the reflections were used to foster students in becoming critical consumers of educational technology as well as to become aware of their learning processes while developing technology skills. enablers. because i-tep is a general education course, materials had to be either very general in nature so they would be relevant for students from a variety of teacher education programs or they needed to be curriculum-centered with enough samples relevant to each of the teacher education programs. in i-tep, students were provided with both. generic samples where used to introduce concepts, then students were provided with curriculum-specific examples to explore concepts related to their field. there were no reports of a lack of relevant exemplars, artifacts and resources to explore from students from any of the teacher education programs. several of the technologies such as group blogs and reflection questionnaires enabled students multiple means of reflection and articulation. additionally, because all student work was accessible via the web, students could view each others’ work, which further enhanced articulation among students. barriers. there were no reports of students not being able to access group blogs, reflective questionnaires or other students’ work; however, there were five reports of students being unable to access resource web sites. this did not appear to cause much difficulty because a variety of supplemental resources for each teacher education program had been provided. summary overall, the combination of tools, method and media outlined seemed to foster cognitive apprenticeship methods within a web-based learning environment. however, it should be noted that this initiative represented a significant investment of time and energy for the instructor. typically, faculty expect to invest time and energy in developing curriculum, materials and resources for teaching; however, a 128 m. d. dickey web-based environment is much more time-intensive. the oms videos, although easily created with camtasia screen-capturing software, took a great deal of time to create. additionally, the flash animations also represent a significant time investment. although the materials, web site, blog, questionnaires and web-based curriculum design were also time-consuming, the greatest investment of time was in teaching the course. coaching and scaffolding along with supporting reflection and articulation can be significant in a traditional classroom; however, typically the raditional classroom is time-locked with a beginning and end to class time. in a webbased environment, there is no time-lock beyond those that a teacher imposes. without time-lock limits, coaching students can rapidly become an ongoing teaching experience, which, although it is a great enabler for students, can rapidly become a barrier for the teacher. discussion one of the main concerns of educators teaching in a distance learning environment is the issue of access. overall, there were few problems of access to most of the course materials. although the various components of cognitive apprenticeship methods were integrated throughout the course in various ways, the oms materials were some of the most apparent elements of modeling. because the video and animation were not step-by-step ‘how to’ instructions, but rather were designed to illustrate and demonstrate thinking process while creating educational media, the oms materials were an integral part of the course. provisions such as options for downloading, a variety of file formats, and the availability of cds and dvds seemed to be an effective method for supporting the technique. it is important to note that the participants in this study are students at a mid-sized liberal arts state university; however, the students at this university are predominantly from families in the upper-middle economic range. most students had broadband internet access. the selection of media in this study was based the technology access provided by the university and based upon prior experience with this student demographic. it is acknowledged that the students in this study may not be representative of the broad spectrum of students in educational licensure programs. the choice to present the course as a web-based environment and the selection of media such as video and animation used in this study to support cognitive apprenticeship methods may be a barrier for many students. observations and students’ reflections revealed that students found the oms materials instrumental in modeling and used them as point of departure when requesting assistance. these findings are somewhat consistent with findings of schrader et al. (2003). whereas schrader et al. found that the use of video case studies of best practices helped initiate conversation and allowed for multiple perspectives, observations and interactions with students in the i-tep web-based environment revealed that the use of video and animation allowed students to experience different methods of problem-solving—and for many students it helped foster a sense of presence in a distributed learning environment. cognitive apprenticeship methods in web-based educational technology 129 this is interesting to note because collins et al. (1989) stress the importance of modeling and coaching as methods of implementing cognitive apprenticeship methods. jonassen (1999) argues that there are two types of modeling: behavioral modeling of overt performance, and cognitive modeling of covert cognitive processes. whereas behavioral modeling focuses on psychomotor skills, cognitive modeling involves an externalizing of internal processes. the use of video and animation provided students with both overt behavioral modeling of skills, but more importantly the covert cognitive modeling of internal processes of decision-making and problemsolving along with rationales for choices and options. conclusion the span of the qualitative case study is by no means comprehensive, but rather is intended to examine both how methods and media can foster the integration of cognitive apprenticeship methods. however, it is not the purpose of this study to prescribe methods for integrating cognitive apprenticeship methods in web-based learning environments, nor to designate how an online course should be structured. rather, the findings of this study revealed that various media and processes could help foster the integration of cognitive apprenticeship methods for a web-based teacher education course. however, it should be noted that the applicability of these findings is context dependent. the students in this study had access to computer and internet access that supported the use of video and animations, email, instant messenger tools and other web-based tools. the premise of this investigation is that the cognitive apprenticeship model holds relevance for both modeling and the effective use of technology for pre-service teacher education students as well as being a viable model for the design of a web-based learning environment. acknowledgement the preparation of the research was supported in part by the research center for educational technology research fellowship. the ideas expressed in this paper do not necessarily reflect the position of the grant agency. references bransford, j., brown, a. & cocking, r. (eds) (2000) how people learn: brain, mind, experience, and school (washington, dc, national academy press). brown, j. s. (1998) internet technology in support of ‘communities-of-practice’: the case of xerox, accounting, management and information technology, 8, 227–236. brown, j. s., collins, a. & duguid, p. (1996) situated cognition and the culture of learning, in: h. mclellan (ed.) situated learning perspectives (englewood cliffs, nj, educational technology publications, inc.), 19–44. collins, a., brown, j. s. & newman, s. e. (1989) cognitive apprenticeship: teaching the craft of reading, writing, and mathematics, in: l. b. resnick (ed.) knowing, learning, and instruction: essays in honor of robert glaser (hillsdale, nj, lawrence erlbaum associates), 453–494. 130 m. d. dickey collins, a., brown, j. s. & holum, a. (1991) cognitive apprenticeship: making thinking visible, american educator, 15(3), 6–11. daviault, c. & coelho, m. (2003) forum: a new approach to the production of educational content, paper presented at siggraph: international conference on computer graphics and interactive techniques educators program from the 30th annual conference on computer graphics and interactive techniques, san diego, ca, 27–31 july. enkenberg, j. (2001) instructional design and emerging models in higher education, computers in human behavior, 17, 495–506. hannafin, m. j., land, s. & oliver, k. (1999) open learning environments: foundations, methods, and models, in: c. m. reigeluth (ed.) instructional-design theories and models: a new paradigm of instructional theory (vol. 2) (hillsdale, nj, lawrence erlbaum associates), 115–140. hendricks, c. c. (2001) teaching causal reasoning through cognitive apprenticeship: what are results from situated learning?, the journal of educational research, 94(5), 302–311. järvelä, s. (1995) the cognitive apprenticeship model in a technology rich learning environment: interpreting the learning interaction, learning and instruction, 5, 237–259. jonassen, d. h. (1999) designing constructivist learning environments, in: c. m. reigeluth (ed.) instructional design theories and models (vol. 2) (mahwah, nj, erlbaum associates), 215–239. liu, m. (1998) a study of engaging high-school students as multimedia designers in a cognitive apprenticeship-style learning environment, computers in human behavior, 14(3), 387–415. liu, m. & y. hsiao. (2002) middle school students as multimedia designers: a project-based learning approach, journal of interactive learning research, 13(4), 311–337. liu, m. & pedersen, s. (1998) hypermedia design and elementary school students, journal of interactive learning research, 9(2), 155–182. merriam, s. b. (1998) qualitative research and case study applications in education (san francisco, ca, jossey-bass). saarenkunnas, m., järvelä, s., häkkinen, p., kuure, l., taalas, p. & kunelius, e. (2000) ninter—networked interaction: theory-based cases in teaching and learning, learning environments research, 3, 35–50. schrader, p. g., leu, d. j., kinzer, c. k., ataya, r., teale, w. h., labbo, l. d. & cammack, d. (2003) using internet delivered video cases, to support pre-service teachers’ understanding of effective early literacy instruction: an exploratory study, instructional science: an international journal of learning and cognition, 31, 317–340. alt-j, research in learning technology vol. 14, no. 1, march 2006, pp. 39–54 issn 0968-7769 (print)/issn 1741-1629 (online)/06/010039–16 © 2006 association for learning technology doi: 10.1080/09687760500479936 using multimedia to enhance the accessibility of the learning environment for disabled students: reflections from the skills for access project david sloana*, john stratfordb and peter gregora auniversity of dundee, uk; buniversity of sheffield, uk taylor and francis ltdcalt_a_147976.sgm10.1080/09687760500479936alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2006taylor & francis141000000march 2006davidsloandsloan@computing.dundee.ac.uk as educators’ awareness of their responsibilities towards ensuring the accessibility of the learning environment to disabled students increases, significant debate surrounds the implications of accessibility requirements on educational multimedia. there would appear to be widespread concern that the fundamental principles of creating accessible web-based materials seem at odds with the creative and innovative use of multimedia to support learning and teaching, as well as concerns over the time and cost of providing accessibility features that can hold back resource development and application. yet, effective use of multimedia offers a way of enhancing the accessibility of the learning environment for many groups of disabled students. using the development of ‘skills for access’, a web resource supporting the dual aims of creating optimally accessible multimedia for learning, as an example, the attitudinal, practical and technical challenges facing the effective use of multimedia as an accessibility aid in a learning environment will be explored. reasons why a holistic approach to accessibility may be the most effective in ensuring that multimedia reaches its full potential in enabling and supporting students in learning, regardless of any disability they may have, will be outlined and discussed. introduction multimedia in e-learning offers significant potential in enhancing the learning environment, through helping to widen access to education, enhancing the learning experience and supporting multiple learning styles (thornhill et al., 2002; littlejohn, 2003). as availability, capability and affordability of technology (hardware and * corresponding author. digital media access group, division of applied computing, university of dundee, dundee dd1 4hn, uk. email: dsloan@computing.dundee.ac.uk 40 d. sloan et al. software) improves, teachers have increased opportunity to create and use multimedia in curriculum design. for example, video and audio recordings and photographs could be used to capture and convey real-world experiences and events, whether from the past or in real time. animated interactive diagrams could be used to demonstrate and simulate concepts in many different disciplines, allowing students to study effects over time or as the result of changing specific parameters. for disabled students, multimedia can offer the potential to significantly improve the accessibility of the learning environment and, indeed, extending the learning environment to otherwise inaccessible places; for example, using video to enhance the accessibility of field-trip environments to students with physical disabilities. at the same time, poorly designed or inappropriate use of multimedia could effectively exclude a disabled student from a learning programme. given the legal imperatives of the uk’s disability discrimination act (dda) in the united kingdom (hmso, 1995), extended in 2001 to cover the provision of post-16 education, and of similar legislation in many countries around the world, educational establishments and their staff have increased responsibility to ensure that unjustified discrimination in not experienced by disabled students. consequently, there is significant activity in the e-learning community towards ensuring that accessibility to disabled people is a key consideration in the development of educational online resources (seale, 2004), particularly those intended for reuse for different purposes in different environments (brewer & treviranus, 2003). guidelines for the creation of accessible e-learning resources (ims global learning consortium, 2002) offer accessibility advice in the context of creating resources for learning and teaching. the ongoing ims accessforall work, involving the accessibility for learner information package specification (ims acclip) and accessforall metadata specification, (both available from the ims website1) together provide a framework for storing a learner’s specific accessibility preferences and a structured way of describing a resource’s specific accessibility characteristics, the degree to which it may be customised to enhance its accessibility while preserving pedagogic aim and any equivalent alternatives. making multimedia optimally accessible the broadcasting sector has for some time been taking steps to address issues relating to accessibility of broadcast television services, to the extent that key accessibility techniques developed in ‘traditional’ broadcasting have been adapted for multimedia web content (clark, 2002; carey, 2005a). in the united kingdom, the independent television commission (itc) produced guidance for the provision of accessibility features for broadcast television, and commissioned research exploring issues relating to the accessibility of interactive digital television provision, with a specific focus on the issues faced by older people (carmichael, 1999). ofcom, the itc’s regulatory successor from 2003, assumed the role of promoting accessibility in uk broadcasting, and publishes online the itc guidance on various aspects of access features for television, as well as setting quotas of accessible programming for broadcasters to multimedia enhancing accessibility of learning for disabled students 41 meet (ofcom, undated). despite this, concerns have been expressed that accessibility has not as yet played a suitably important role in the rollout of digital television services in the united kingdom (carmichael et al., 2005). on the web, content and e-learning developers concerned about the accessibility of their resources to disabled students can look to the web content accessibility guidelines (wcag) (web accessibility initiative [wai], 1999) for advice and support. the wcag, produced by the wai of the world wide web consortium (w3c) are the de facto standard for accessibility of web content, providing a solid foundation on which to build a working knowledge of accessible web design techniques. they also offer a framework on which website evaluation methodologies and automated accessibility checking tools can be based, and a reference for policymakers and benchmarking exercises. version 1.0 of the wcag overwhelmingly concentrates on html-delivered content, and only three of the 65 checkpoints appear to refer directly to multimedia. one such checkpoint includes multimedia within a wider list of webpage elements: provide a text equivalent for every non-text element […] this includes: images […] animations (e.g., animated gifs), applets and programmatic objects […] sounds (played with or without user interaction), stand-alone audio files, audio tracks of video, and video. (checkpoint 1.1) two wcag checkpoints specifically address the use of multimedia: until user agents can automatically read aloud the text equivalent of a visual track, provide an auditory description of the important information of the visual track of a multimedia presentation. (checkpoint 1.3) for any time-based multimedia presentation (e.g., a movie or animation), synchronize equivalent alternatives (e.g., captions or auditory descriptions of the visual track) with the presentation. (checkpoint 1.4) in each checkpoint, the basic premise is that text and/or audio alternatives must be provided for the multimedia content. expanding on these two wcag checkpoints listed, the two key accessibility features required for optimally accessible multimedia are (clark, 2002): 1. synchronised text captions, providing spoken and key non-spoken sound information in text, for people who are deaf or hearing impaired. 2. audio descriptions—audio files synchronised with the main soundtrack of the media clip. these allow people who cannot see the clip to understand its content through the provision of spoken descriptions of key visual events that would not be discernible from the main soundtrack alone. if these features cannot be provided, a more basic requirement is to provide a text transcript of the media clip, in html format, clearly linked from the page containing the media clip in question. the disadvantage of this solution is that, by its nature, it is an alternative to the media clip, rather than an integrated, synchronised solution as with captions and audio descriptions. 42 d. sloan et al. synchronised signed translations of spoken content are also a highly desirable accessibility feature, serving the requirements of people whose primary language is a sign language—and who may have extreme difficulty reading on-screen text. this is currently a particularly rare accessibility solution in practice, given the work required to produce and publish a synchronised signed translation. however, ongoing efforts have attempted to develop an automated animated signed translation facility, through an avatar; for example, the visicast project (elliot et al., 2000). an additional accessibility feature often required is subtitles, a textual translation of the spoken content of the media clip into the language of the viewing population. this accessibility solution caters for a cognitive disability, that of being unable to understand the spoken content despite being able to hear it. yet, in the united kingdom and ireland, the term ‘subtitles’ is widely, but confusingly, used to refer not just to foreign-language translation but also to captions for people unable to hear the soundtrack (see, for example, the list of accessibility solutions in carey [2005a]). other accessibility requirements for multimedia include the need to ensure that operation is keyboard accessible, to allow people who for reasons of visual or mobility impairment have difficulty in using, or are unable to use, a mouse. for video and audio, this includes basic operations of play, pause, stop, fast-forward and rewind, controlling display and audio properties, and enabling or adjusting accessibility features such as captioning and audio description. interactive animated content such as macromedia flash may present similar problems if its operation requires fine mouse control. for animated content to be optimally accessible to people who cannot see the animation, there is also a need to ensure that the experiences and information provided by the animation are available in audio format, either by exposing the content to allow a screen reader to read it out, or through audio soundtrack(s) provided with the animation (regan, 2002). multimedia as an assistive technology for many web users, accessibility problems result not from the absence of an alternative to information requiring a specific sensory capability or level of manual dexterity, but from the inability to process and understand the information presented on screen. overwhelmingly this problem is due to the presentation, style and content of on-screen text (seeman, 2002). people with dyslexia, with concentration difficulties, with reduced short-term memory or with other specific learning difficulties may all struggle with textual information. similarly problems may be encountered for people for whom english is not their first language (vanderheiden, 2000). for all these groups, arguably the most inaccessible way of presenting information is through a long page of on-screen text. it follows that presenting information in alternative ways—pictures, diagrams, animated diagrams, video clips, audio recordings—is far more effective in conveying information and experiences, and hence supporting comprehension and retention of information. multimedia thus becomes an accessibility solution, an assistive technology. multimedia enhancing accessibility of learning for disabled students 43 however, for multimedia development, the wider potential of multimedia as an enabler, as a way of making information more accessible to certain groups of disabled people, is not immediately obvious from an inspection of the wcag. the only indication offered of this potential offered is found in wcag checkpoint 14.2: supplement text with graphic or auditory presentations where they will facilitate comprehension of the page. it would seem reasonable to infer from this checkpoint, however, that the use of multimedia to enhance comprehension of information, experiences or concepts presented as online text is indeed an important step towards accessibility. slatin and rush (2002) make this very point in their book maximum accessibility, noting that: in some cases, the best way to enhanced accessibility may be to use more media … used in the right way, multimedia is a critical resource for accessibility! the key phrase in this quotation is ‘used in the right way’. there is still a need to ensure that every reasonable effort is taken to ensure that the information and, perhaps more significantly, experience provided by the media clip in question is as accessible as possible to as many as possible of the intended audience. challenges to the provision of accessible multimedia despite the arguments promoting multimedia as a positive contributor to enhancing the accessibility of the e-learning environment, there are many challenges to be overcome before multimedia can reach its potential as an enabler for disabled students. attitudinal the relative prominence of accessible web design requirements for ensuring content can be accessed by people with visual impairments has contributed to the inaccurate perception that all principles of web accessibility can be summarised as ‘provide all information in text format’—a position that may be further distorted to imply that graphics and colour, let alone video, audio and interactive animated media, is prohibited on account of accessibility. the argument that accessible web content is incompatible with visually pleasing, engaging online experiences has been confidently met by assertions that highly accessible sites can indeed maintain a high degree of visually stimulating content (zeldman, 2003; petrie et al., 2004). however, the message that multimedia is an important contributor to accessibility for some groups may unfortunately be lost among the more prominent and apparently higher priority demands for multimedia to be provided with accessibility features to overcome sensory impairments. the wai classed the aforementioned checkpoint 14.2 on supplementing text with graphics or audio as a priority three wcag checkpoint, the lowest of the three priority levels allocated to each checkpoint. this arguably recognises that the task of 44 d. sloan et al. ‘supplementing text’ with multimedia is likely to require a reasonable degree of production skill and resources, beyond the capacity of many web authors who would be most comfortable providing text, and possibly simple graphics. at the same time it may be interpreted as playing down the positive impact on accessibility of the judicious use of multimedia. the result may be at best a failure to acknowledge the role multimedia content can play in enhancing the accessibility of the information or experience provided by a resource. for example, in an accessibility review of the bbci website (robertson et al., 2002), no mention was made of the positive accessibility role that the multimedia present on the site could offer. at worst, there could be a rejection or withdrawal of potentially valuable multimedia content that, as well as potentially enhancing the learning environment more generally, offers specific accessibility benefits for other groups, particularly people with specific learning difficulties (seeman, 2002). for multimedia producers and designers, this issue may adversely affect their perceptions of accessibility, given that the strongest message is ‘provide alternatives’ rather than ‘use multimedia’. the authors have had personal experience of producers who, when confronted with the issue of accessibility and potentially discriminatory resources, have experienced a range of emotions: ● defensiveness or hostility, as they perceive that their previously well-regarded work is coming under attack for being discriminatory. ● embarrassment, as they realise that their work may exclude certain students from accessing it. ● fear, and helplessness, as they are told that their work must be ‘made accessible’ or face the consequences of legal action, without necessarily having the skills or support to make the necessary changes within current work schedules and budgets. ● de-motivation, as they perceive that their work is no longer ‘legal’, as it may be discriminatory, and is therefore no longer of any worth. while some of these reactions may seem extreme, the adverse impact of designer attitude to accessibility should not be underestimated. a perception also exists among some developers that, when it comes to multimedia, providing accessibility features may be seen as ‘too expensive’, with no cost benefit. this is consistent with a more general resistance to arguments against inclusive design, despite examples of commercial success in, for example, vehicle or household object design (keates & clarkson, 2004). carey (2005b) notes that the ‘too expensive and not worth it’ argument has historically been the case with regard to analogue broadcast media. this perception remains, as pointed out in criticism of recent accessibility guidance issued by ofcom to uk broadcasters (carmichael et al., 2005), despite on the one hand the nature of digital media offering vastly improved capacity for bundling accessibility features such as captions, subtitles, audio descriptions and signed translations along with the media clip, and on the other hand in the reusability, and hence increased potential audience, of digital media, whether web-based or broadcast media that can be recorded and reused through personal video recorder technology. multimedia enhancing accessibility of learning for disabled students 45 practical there are a number of practical issues relating to effective provision of accessible multimedia. knowledge and awareness of accessibility issues remains relatively low among web and multimedia developers, although the situation has undoubtedly improved in recent years. the time required to implement accessibility features can be significant, and even more so if done retrospectively. at the same time, while general principles of accessible multimedia are widely accepted, there is a lack of solid research-based guidelines in many more specific aspects of accessible multimedia design. tasks such as creating caption files for a pre-existing video clip and synchronising them appropriately with the clip, or creating an accurate transcript of a long audio clip, require practical knowledge of the authoring techniques necessary. the more subjective issue of what to include in a caption or audio description file may also make the task more difficult and lengthy than initially realised. indeed, while guidance such as that provided by ofcom may exist, the lack of examples of captioned online video and its implications for best practice is emphasised by clark (2004), who attempted to define best practice on online captioning, but failed to find any training or quality certification programme dedicated to online captioning. clark notes: nearly everyone doing online captioning today is self-taught. and: we are aware of no surveys whatsoever of viewer preferences in online captioning. given this, clark then suggests that: […] we need to produce a much larger corpus of online captioned video, in order to make solid recommendations for best practices in online captioning. while evidence-based best practice may be lacking, some encouraging developments have taken place to ease the practical task of improving the accessibility of multimedia content. the development of tools such as the media access generator (magpie2) from the us-based national centre for accessible media eases the task of authoring and adding synchronised captions and audio description files to digital video. improvements to macromedia’s flash and shockwave rich media formats, and authoring tools for creating flash and shockwave, have offered greater capacity to include accessibility features such as keyboard accessibility and enhanced compatibility with screen reading technology (regan, 2002). bennett et al. (2003) describe the development of smirk, an authoring tool to facilitate accessible presentation creation with the intention of improving the accessibility of a traditional lecturing scenario. development by the w3c of the synchronised multimedia integration language (smil) specification (w3c, 2005) creates an open standard for associating captions and audio description files and their timing information with a media file. however, on a more abstract scale is the issue of multimedia as an accessibility aid. this concept is difficult to encapsulate in solid guidelines applicable across a range of scenarios, and therefore a subjective analysis of the textual information or concept 46 d. sloan et al. requiring illustration is required, depending on the pedagogic aim of the e-learning resource in question, and its context in the wider learning environment or programme. a desire for rules and checklists is surprisingly prevalent among media developers, illustrated in recent feedback received by the authors after the launch of the skills for access website (discussed later). this is unfortunate, given that the most effective approach to providing an effective accessibility solution is to consider the pedagogic aim of the multimedia object and the resource in which it will appear, and the role of the learning object within a wider context, and using this contextual information to guide the most appropriate approach to optimising accessibility. establishing responsibility for accessibility there is also an issue raised by the situation whereby responsibility for ensuring educational multimedia is optimally accessible may not lie with the agency best placed to provide the accessibility features. while it may be common for teachers to develop their own web-based teaching resources using tools provided by a virtual learning environment or web authoring tool, much multimedia used in e-learning resources is professionally produced, either commissioned through institutions’ own production units or external agencies. the increasing interest in reusable learning objects again highlights the situation where the creator of a resource may not be the direct user. the most appropriate and effective place to author and combine accessibility features is at the point of production, but the provision of accessibility features is not commonly considered a standard part of media production, and may require a specific request at the time of commissioning. some multimedia used in an educational context may be captured or generated live—for example, to facilitate synchronous communication between remote peers or learners and teachers. provision of live accessibility features such as captions may in such situations prove extremely challenging, especially as human and technical resources are likely to be limited. a significant, and increasing, amount of digital video and audio material is also made available for ‘off-the-shelf’ educational use; for example, in the united kingdom, through services such as the educational media online service hosted by edina3 and the british universities film and video council.4 it is not clear how much of this digital video content has been provided with captions and audio descriptions, but an optimistic estimate is likely to be an extremely small percentage. thus, while the production stage is the most appropriate stage for providing accessibility features, much of the material made available for use by e-learning developers and users may not in fact have the necessary accessibility features, and the providers may be unwilling or unable to provide such features, given the amount of material that would need to be addressed. yet the legal position is complex, at least with respect to the uk’s dda (sloan, 2004), when the relationship between the media producer, the organisation providing access to the media, and the educational establishment is considered. it may only be when the material is used by an educational establishment in a way that is discriminatory to a disabled student that legislative requirements come into play. thus, unless otherwise specified by the educational multimedia enhancing accessibility of learning for disabled students 47 institution, the most immediate legal responsibility appears to fall on the e-learning developer or teacher to avoid discriminatory practice, rather than the organisation(s) producing and/or making available the media resource. a scenario may thus emerge whereby e-learning developers around the country are working independently on providing their own captions for the same uncaptioned video clip obtained from an online digital data archive, although an alternative scenario may be the rejection of potentially valuable multimedia content on account of an absence of accessibility features. the ims accessforall work is potentially of great significance here in that it offers a solid framework for specifying accessibility features and limitations of individual resources, and identifying suitable alternatives where access barriers currently exist; for example, where multimedia has been acquired without captions or audio descriptions. technical aside from the practical issues of developing expertise in providing accessibility solutions to multimedia, and the time and resources required to develop the features, technical challenges may make more difficult the task of creating optimally accessible multimedia for educational purposes. the availability to developers of the appropriate design and evaluation tools necessary to create optimally accessible material is a critical issue, requiring organisations to consider accessibility from a procurement perspective. with web-delivered multimedia, there is also the complexity of the accessibility capabilities of multimedia technologies and accessibility support of media players; that is, the software that works with the web browser to play the media file and any accessibility feature provided. the w3c has acknowledged the importance of the media player in ensuring accessibility, along with the web browser, assistive technology and other ‘user agents,’ in developing the user agent accessibility guidelines (wai, 2002). significant improvements in media player and media format accessibility have taken place, motivated by legislation in the united states (the 1998 amendment to section 508 of the rehabilitation act) prohibiting federal organisations from providing or using technology with specific accessibility barriers. unfortunately, at the time of writing, accessibility issues still exist with each of the main media players used for delivering multimedia online. media player accessibility issues include limited keyboard accessibility to key functions of the features of the player, particularly when the media player exists as an ‘embedded’ interface within a webpage, instead of acting as a stand-alone player opened through activating a link on the webpage (webaim, undated). while embedded media integrates the media clip more seamlessly within a webpage, the limitations of keyboard accessibility mean that people who for reasons of mobility impairment or visual impairment are unable to use a mouse may have extreme difficulty accessing the multimedia content. media players also have limitations in terms of their support for the accessibility features described earlier. for example, accessibility features may exist only as 48 d. sloan et al. ‘advanced’ features not available in the free version of the player, the ability to turn on and off captions is not available in the free version of quicktime, while at the time of writing windows media player cannot support audio description files at all. the flash player, from macromedia, is the most common means of delivering rich animated content through websites, and is increasingly used as an alternative way of delivering video content without requiring a dedicated media player. despite the aforementioned improvements, some accessibility issues remain with the flash player, in terms of its support for screen readers and its keyboard accessibility (heins & regan, 2002). the task of providing and making available accessibility features such as captions and audio descriptions is made more difficult through the use of one technology, flash, to play another, the media clip. skills for access: supporting media developers and producers in an effort to support developers wishing to create optimally accessible multimedia content for learning, and aware of the challenges that face developers in achieving this goal, the skills for access project was funded by the higher education funding council for england and department of employment and learning northern ireland to create a web resource devoted to multimedia, accessibility and learning (stone et al., 2003). importantly, the project brought together a professional media and e-learning production unit (the learning development and media unit (ldmu) at the university of sheffield, uk) and accessibility and inclusive design experts (the digital media access group (dmag) at the university of dundee, uk). as media and e-learning producers, ldmu not only produced media assets for the resource but, as representatives of the intended audience, were involved in specifying requirements for the resource, and evaluating its appropriateness. the expert advice offered by dmag was then effectively ‘trialled’ internally within the project team during the resource development, and practical challenges in following accessibility guidelines were identified and documented as part of the information provided by the resource. launched in may 2005, the resulting web site5 was designed with accessibility and web standards in mind, and provides advice and support in a number of ways: ● the skills for access approach to accessibility, learning and multimedia. the dual approach to multimedia accessibility is introduced; that is, teachers and e-learning developers are encouraged to use multimedia wherever possible to enhance the accessibility of the traditional learning environment, but at the same time to make best efforts to ensure the multimedia resource—or more specifically the intended learning outcome of the resource—is as accessible as possible to the target audience. ● how-to guides. these are a series of resources focusing on a specific accessibility objective applied to a particular multimedia technology. each guide outlines the objective, discusses details of techniques for achieving the objective and suggests testing mechanisms for ensuring the objective have been met. importantly, each how-to guide also provides links to other relevant web resources that give more multimedia enhancing accessibility of learning for disabled students 49 information on the issue in hand, or provide examples. topics covered include captioning and audio description of video, and supporting interaction of animated content with alternative input devices. ● articles and case studies. a number of short articles focusing on the potential of multimedia to enhance accessibility of learning, and teaching, were commissioned from experts in the field of multimedia, learning and accessibility. case studies are also provided, some focusing on disabled students and staff sharing their experiences in using e-learning and multimedia. other case studies were supplied by media and e-learning developers; who discuss specific projects relating to creating accessible multimedia. as a demonstration of how multimedia can be used to enhance both the provision of information and the browsing experience, several of the case studies are video based. ● supporting resources. supporting resources include background information, and links to more detailed resources, on disabilities, multimedia technologies and specific access challenges that might be encountered by a disabled student when accessing a learning programme. also provided are details of the skills for access website construction, discussing the features of the site and how it supports accessibility. areas of further development of the site include publishing new case studies, with the aim of building up a catalogue of experiences that can be used to inform the community through sharing success stories and frustrations alike. developing accessible multimedia for learning: lessons from skills for access many lessons were learnt along the way during the project. as knowledge of the issues, including legislative requirements, grew among the production team, attitudes of media producers to accessibility became more favourable. another revelation was the complexity of the current situation of media players and support for accessibility features, and the implications of the work required to implement accessibility solutions. despite the best efforts to provide information that supported users while encouraging them to think more widely about the issues, some initial feedback has indicated that there is still a preference for short checklist-style information, even though this may lead to inappropriate solutions being chosen. those involved in the project also experienced first hand the benefit of considering accessibility from the beginning of the production process, and maintaining this influence throughout. the unavoidable additional time required to implement accessibility solutions, and the need to consider this in scheduling and resourcing projects, was also made clear. this was reflected in ldmu staff feedback and by quotes from two separate case studies published on the skills for access website: it seems to us, therefore, that the pivotal question that runs through the whole implementation of accessible multimedia issue is time. where do you find the time to do it all? (case study: providing captioned video clips for the skills for access website6) 50 d. sloan et al. the major factor in achieving (the aims of the captioning exercise) was that of time. the initial transcription of the 7.5 minute video took around 30 minutes. this time could be dramatically reduced if even a rough transcript had been available—which emphasises the point that accessibility requirements should be taken into account at the planning stages (in this case, before filming) rather than subsequently ‘retro-fitting’ an inaccessible resource. (case study: captioning with quicktime smil, patrick lauke7) it is clear that, for multimedia production, the task of ‘retrofitting’ existing work to be made optimally accessible is particularly challenging. at the same time, it is easy for developers to apply accessibility solutions inappropriately, creating solutions that may appear to be technically ‘accessible’, but to fail to provide disabled people with the equivalent intended learning experience, or solutions that dilute the learning experience for other students and do little to benefit the disabled learner. this underlines the importance, from an accessibility perspective, of advance dialogue with the commissioner of any production work, in order that teacher and producer can both gain an idea of the pedagogic aims of the resource and its intended role in teaching and learning. this approach acknowledges arguments that, given the varying nature of accessibility requirements among individuals with specific impairment or combination of impairments, combined with the unique aims and context of an online resource (such as audience characteristics and usage environment), a guideline-based approach is not sufficient to ensure the most appropriate accessibility solutions are provided (milne et al., 2005). for e-learning accessibility in general, and multimedia in particular, a holistic approach is advised (kelly et al., 2004, 2005), acknowledging the role e-learning plays in enhancing the wider learning environment. the need for a holistic approach to multimedia accessibility in the united kingdom, the legislation of the dda does not specify technical requirements that must be met by digital resources in order to avoid discrimination. accompanying codes of practice have been provided to give practical guidance on how unlawful discrimination may be involved—for example, the code of practice for providers of post 16 education and related services (disability rights commission, 2002), which gives examples of where the dda might apply in an educational establishment, and what might constitute, in the terms of the legislation, ‘reasonable adjustments’ to practice in order to ensure that unjustified discrimination does not occur. yet there is little in the way of prescriptive technical information in terms of elearning design. this may seem at first glance frustrating to e-learning developers, but in fact provides an opportunity to use the most appropriate resources, or combination of resources, to create an optimally accessible learning environment. the use of multimedia may therefore be considered a reasonable adjustment to make to the provision of education in order to reduce exclusion. to approach the issue of accessibility in relation to a specific multimedia learning resource, there is a need to move beyond a technical definition of the level of accessibility to be achieved, to a wider context. it is easy to imagine a scenario whereby a multimedia enhancing accessibility of learning for disabled students 51 developer, given a set of accessibility guidelines, may try to apply inappropriate accessibility solutions. for example, it would be clearly inappropriate to provide captions for a video used in a modern language class to test students’ ability to listen and understand spoken dialogue. the skills for access resource encourages a holistic approach, establishing context by considering the following points: ● what are the aims, the pedagogic goals of this piece of multimedia? how does the resource fit in with the rest of the learning environment? ● will all students be required to use it? is its use compulsory for completion of a course or for assessment? ● where is the resource intended to be used—a controlled environment like a laboratory or classroom, or potentially anywhere? what assumptions can be made about the browsing and access technology available to intended users (and their knowledge of that technology)? ● what are the potential barriers to using the multimedia resource for its intended purpose? what levels of sensory or motor abilities are required? how might specific learning difficulties or other cognitive impairments affect the ability to use the resource? ● what alternatives already exist and what alternatives can be reasonably created? how was the subject or topic previously taught? ● what is the best way that the information or experiences can be presented such that: 1. as many as possible of the intended audience can achieve the intended learning objectives using the multimedia resource? 2. those affected by remaining accessibility barriers can achieve the same objectives in a way best suited to them? the answers to these questions should help e-learning developers and media producers establish a sound basis on which to build a strategy for accessible design, and to ensure that the accessibility solutions chosen are most appropriate for the context in which the resource will be used. this proactive approach is required by the need of the uk dda for educational providers to be anticipatory in the steps they take to accommodate the needs of future disabled learners, and ultimately promotes a culture of accessible design. conclusion the judicious use of multimedia in an educational context offers significant potential in making the learning environment more accessible to disabled students, although this message can be lost in favour of the equally important, but higher-profile, demands that multimedia be provided with additional features to make it accessible. while resources supporting designers improve in quality and quantity, there is still scope for improvement in the support for accessibility of the tools and technologies available to developers and end users to ensure that using online multimedia is as easy as possible regardless of personal access needs. additionally, given that some 52 d. sloan et al. accessibility features do require extra time to implement, production schedules need to consider this from the start, while a more joined-up approach to the provision of accessible multimedia across the wider educational community is also needed. the need for a holistic approach to multimedia e-learning is clear. e-learning resource developers, multimedia producers and teachers must work together to establish the context of a multimedia resource in order to assess: ● how best the resource in question can enhance the accessibility of the learning environment. ● how to ensure the resource—and the learning experience it provides—is itself is as accessible as possible to the target audience. acknowledgements thanks to ldmu staff steve collier, claire allam, graham mcelearney, chris firth, katrin thomson and karen waddicor for their input, and to the reviewers of this paper for their comments. notes 1. http://www.imsglobal.org/accessibility/ 2. http://ncam.wgbh.org/webaccess/magpie/ 3. http://www.edina.ac.uk 4. http://www.bufvc.ac.uk 5. http://www.skillsforaccess.org.uk 6. http://www.skillsforaccess.org.uk/casestudies.php?id=157 7. http://www.skillsforaccess.org.uk/casestudies.php?id=162 references bennett, s., hewitt, j., kraithman, d. & britton, c. (2003) making chalk and talk accessible, in: proceedings of acm conference on universal usability (cuu) 2003 (new york, acm press), 119–125. brewer, j. & treviranus, j. (2003) developing and reusing accessible content and applications, in: a. littlejohn (ed.) reusing online resources—a sustainable approach to e-learning (london, kogan page), 119–128. carey, k. (2005a) accessibility: the current situations and new directions, ariadne, 44. available online at: http://www.ariadne.ac.uk/issue44/carey/ (accessed 28 september 2005). carey, k. (2005b) the case for a european centre for accessible media, paper presented at the nesta seminar, 19 february. available online at: http://www.nesta.org.uk/assets/pdf/casefor-ecam-nesta-seminar.pdf (accessed 28 february 2005). carmichael, a. (1999) style guide for the design of interactive television services for elderly viewers (winchester, independent television commission). available online at: http://www.computing.dundee.ac.uk/staff/acarmichael/carmichael_designstyleguide.pdf (accessed 21 february 2005). carmichael, a., rice, m. & sloan, d. (2005) digital interactive television in the uk: is the opportunity for ‘inclusivity’ being missed?, paper presented at the proceedings of accessible design in the digital world conference (addw 2005), british computer society, electronic workshops multimedia enhancing accessibility of learning for disabled students 53 in computing. available online at: http://ewic.bcs.org/conferences/2005/accessible/session4/ paper1.htm (accessed 28 september 2005). clark, j. (2002) building accessible websites (indianapolis, new riders). clark, j. (2004) best practices in online captioning. available online at: http://joeclark.org/access/ captioning/bpoc/?axx (accessed 28 september 2005). disability rights commission (2002) disability discrimination act 1995—part iv code of practice for providers of post 16 education and related services. available online at: http://www.drcgb.org/uploaded_files/documents/4008_187_post-16%20part%201.pdf (accessed 14 october 2005). elliot, r., glauert, j., kennaway, j. & marshall, i. (2000) the development of language processing support for the visicast project, in: proceedings of assets 2000—the fourth international acm conference on assistive technologies (new york, acm press), 101–108. heins, j. & regan, b. (2002) building standards-conformant accessible learning objects with flash mx, macromedia white paper. available online at: http://download.macromedia.com/pub/ solutions/downloads/accessibility/fl_learning_obj.pdf (accessed 21 february 2005). hmso (1995) disability discrimination act 1995. available online at: http://www.legislation.hmso.gov.uk/acts/acts1995/1995050.htm (accessed 14 november 2005). ims global learning consortium (2002) ims guidelines for developing accessible learning applications, version 1.0. available online at: http://ncam.wgbh.org/salt/guidelines/ (accessed 21 february 2005). keates, s. & clarkson, j. (2004) countering design exclusion—an introduction to inclusive design (london, springer-verlag). kelly, b., phipps, l. & swift, e. (2004) developing a holistic approach for e-learning accessibility, canadian journal of learning and technology, 30(3). available online at: http://www.cjlt.ca/ content/vol30.3/kelly.html (accessed 14 november 2005). kelly, b., sloan, d., phipps, l., petrie, h. & hamilton, f. (2005) forcing standardization or accommodating diversity? a framework for applying the wcag in the real world, in: proceedings of 2005 international cross-disciplinary workshop on web accessibility (w4a) (new york, acm press), 46–54. littlejohn, a. (2003) reusing online resources—a sustainable approach to e-learning (london, kogan page). milne, s., dickinson, a., carmichael, a., sloan, d., eisma, r. & gregor, p. (2005) are guidelines enough? an introduction to designing websites for older people, ibm systems journal, 44(3), 557–572. available online at: http://www.research.ibm.com/journal/sj/443/milne.html (accessed 14th november 2005). ofcom (undated) television access services. available online at: http://www.ofcom.org.uk/tv/ifi/ guidance/tv_access_serv/ (accessed 28 september 2005). petrie, h., hamilton, f. & king, n. (2004) tension, what tension? website accessibility and visual design, in: proceedings of the 2004 international cross-disciplinary workshop on web accessibility (w4a) (new york, acm press), 13–18. regan, b. (2002) accessibility and macromedia flash, in: constructing accessible web sites (berkeley, glasshaus/apress), 267–282. robertson, v., shelat, b., stewart, t., travis, d. & tynan, a. (2002) accessibility study of bbci: problems faced by users with disabilities, bbc and system concepts. available online at: http:// www.bbc.co.uk/commissioning/newmedia/pdf/bbci_accessibility_study_7-10-02.pdf (accessed 28 september 2005). seale, j. (2004) the development of accessibility practices in e-learning: an exploration of communities of practice, alt-j, 12(1), 51–63. seeman, l. (2002) inclusion of cognitive disabilities in the web accessibility movement, in: proceedings of the international www conference (www 11) (hawaii, usa). available online at: http://www.conf.esc.soton.ac.uk/archive/00000201/(accessed 14 november 2005). slatin, j. & rush, s. (2002) maximum accessibility (boston, addison wesley). 54 d. sloan et al. sloan, m. (2004) content service providers and the disability discrimination act 1995, a briefing paper for jisc legal information service. available online at: http://www.jisclegal.ac.uk/ publications/legalimplicationsdda.htm (accessed 28 september 2005). stone, s., sloan, d. & stratford, j. (2003) skills for access: promoting accessible, engaging multimedia in e-learning, assignation, 21(1), 21–22. thornhill, s., asensio, m. & young, c. (2002) click and go: video streaming—a guide for educational development. available online at: http://www2.umist.ac.uk/isd/lwt/clickgo/the_guide/ video_streaming-the_guide128bit.pdf (accessed 29 september 2005). vanderheiden, g. (2000) fundamental principles and priority setting for universal usability, in: proceedings of the 2000 conference on universal usability (cuu 2000) (new york, acm press), 32–37. w3c (2005) synchronised multimedia integration language (smil) 2.0, world wide web consortium. available online at: http://www.w3.org/tr/2005/rec-smil2-20050107/ (accessed 28 september 2005). web accessibility initiative (1999) web content accessibility guidelines 1.0. world wide web consortium. available online at: http://www.w3.org/tr/wcag10/ (accessed 28 september 2005). web accessibility initiative (2002) user agent accessibility guidelines 1.0. world wide web consortium. available online at: http://www.w3.org/tr/uaag10/ (accessed 28 september 2005). webaim (undated) media player accessibility. available online at: http://www.webaim.org/ techniques/captions/mediaplayers/ (accessed 21 february 2005). zeldman, j. (2003) designing with web standards (indianapolis, new riders). calt586677 117..127 using technology to encourage student engagement with feedback: a literature review stuart hepplestone*, graham holden, brian irwin, helen j. parkin and louise thorpe student and learning services, sheffield hallam university, sheffield, uk (received 8 november 2010; final version received 13 april 2011) this article presents a review of the literature over the past 10 years into the use of technological interventions that tutors might use to encourage students to engage with and action the feedback that they receive on their assessment tasks. the authors hypothesise that technology has the potential to enhance student engagement with feedback. during the literature review, a particular emphasis was placed on investigating how students might better use feedback when it is published online. this includes where an adaptive release technique is applied requiring students to submit an action plan based on their feedback to activate the release of their grade, and electronic generation of feedback using statement banks. key journals were identified and a snowball technique was used to select relevant literature. the use of technology to support and enhance student learning and assessment is well documented in the literature, and effective feedback practices are similarly well published. however, in terms of the use of technology to support and enhance feedback processes and practices (i.e. production, publication, delivery and students making use of feedback through technology), we found the literature to be limited. keywords: engagement; feedback; technology introduction feedback on assessments is an integral feature of effective and efficient teaching and learning, and can be one of the most powerful ways in which to enhance and strengthen student learning (black and wiliam 1998; sadler 2010). price et al. (2010, 277) argue that feedback is “the most important part of the assessment process”. feedback enables learning by providing information that can be used to improve and enhance future performance. this paper explores current literature around the appropriate use of technology in the production and delivery of feedback. it was anticipated that the literature would support the hypothesis that technology has the potential to enhance student engagement with feedback.1 models, principles and conditions that support good feedback practice have been proposed (for example, gibbs and simpson 2004; mcdowell et al. 2005; nicol and macfarlane-dick 2006). for feedback to be effective and attended to by students, it needs to be timely (i.e. while it matters to them and useful for future assessments), legible, and aligned with explicit assessment criteria, as emphasised by nicol (2009, *corresponding author. email: s.j.hepplestone@shu.ac.uk research in learning technologyaquatic insects vol. 19, no. 2, july 2011, 117–127 issn 2156-7069 print/issn 2156-7077 online � 2011 association for learning technology doi: 10.1080/21567069.2011.586677 http://www.informaworld.com 337): “feedback should be of sufficient quantity; timely; it should focus on learning not marks; it should be related to assessment criteria and be understandable, attended to and actually used by students to make improvements on their work”. appropriate use of learning technologies can encourage better engagement with feedback. despite the promotion of good feedback, much of the practice reported tends to focus on the ineffectiveness of feedback (bloxham and boyd 2007; hounsell 2008; rowe and wood 2007; rust, o’donovan, and price 2005). pressures in the uk higher education sector (department for education and skills 2003) resulting in modularisation and semesterisation have seen the bunching of assessment tasks toward the end of the module. this not only limits the scope for feedback practices that feed forward into future assessments (higgins, hartley, and skelton 2002; irons 2008; price and o’donovan 2008; yorke 2001), but also into their professional practice (poulos and mahony 2007). these pressures have also seen the writing of feedback under increasingly tight time constraints (chanock 2000). the result has been a negative impact on the student experience of feedback as supported by responses to the national student survey (higher education funding council for england 2007). students have expressed dissatisfaction with the adequacy of the feedback they receive both in terms of timing and usefulness (mutch 2003; price et al. 2010). this is echoed further by large-scale (hounsell and entwistle 2007) and small-scale (crook, gross, and dymott 2006) studies into the student experience of assessment and feedback. staff complain that feedback does not work (weaver 2006) and that students do not act on feedback (mutch 2003). it is claimed that students are only concerned with their grades (wojtas 1998; nesbit and burton 2006), see feedback as a means to justify the grade (price and o’donovan 2008; price et al. 2010) or only read the qualitative comments if the quantitative mark is outside of their expectations (duncan 2007). some authors have argued that student disengagement with feedback is based on sceptical or anecdotal evidence from tutors (carless 2006; higgins, hartley, and skelton 2002; weaver 2006), and burke (2009, 41) wonders whether “students interpret the term ‘feedback’ literally and use it only to look back on work they have completed, and are not aware or able to use tutor comments to ‘feed-forward’ and contribute to their ongoing development”. in addition, maclellan (2001, 316) suggests that use of ‘implicit criteria’ by tutors means that students do not view feedback on their learning as helpful, and bloxham and boyd (2007) and higgins, hartley, and skelton (2002) point to difficulties for students in deciphering handwritten feedback comments as a further reason why students do not read or make use of their tutors’ comments. there is a growing body of research exploring how technology might be used to support effective and efficient feedback practices. nicol (2009) explains the aims of the re-engineering assessment practices in scottish education project and demonstrates the ways in which technology can be used to support the development of self-regulation, the organisation of assessment tasks and the provision of feedback in large (150–900 students) first-year classes, where modularisation and larger student numbers have reduced formative assessment and feedback opportunities. approach this article builds on an earlier literature review undertaken and made available via a wiki by the authors in 2008/09 in preparation for a research project to explore 118 s. hepplestone et al. how students at sheffield hallam university engage with feedback delivered electronically when three specific technological interventions are used. the research project (parkin et al. forthcoming) aimed to find out how students might engage with feedback and formulate actions to improve future learning when: feedback and grades are published online; feedback is published online without the grade and using an adaptive release process whereby students are encouraged to reflect on and submit an action plan based on their feedback to activate the release of the grade; and feedback is produced via electronic comment banks aligned to the original assessment criteria and presented in a table or grid format. in this article, recent studies are considered alongside the previous literature in providing an up-to-date review of the technological interventions that might encourage student engagement with feedback. a literature search was undertaken using the terms ‘technology’, ‘feedback’ and ‘engagement’ across electronic catalogues and search engines. in addition, the contents of the association for learning technology’s journal, research in learning technology, assessment & evaluation in higher education and british journal of educational technology (identified as being three of the key journals with a focus on the use of educational technology and assessment in higher education) were browsed from 2001, as well as relevant conference proceedings including the association for learning technology’s conference and the international computerassisted assessment conference. articles were selected based on both the title of the paper and its abstract, and we used the ‘snowball’ technique to identify additional sources from references within the articles. our focus is on the production, delivery and the process for making use of that feedback rather than the format of the feedback itself. we were particularly interested in research and discussion that explored the publication of feedback and grades online, the automated adaptive release of grades (or how technology can support encouraging student engagement through disengaging the grade from feedback), and the use of technology to automatically generate feedback aligned to assessment criteria through pre-populated statement banks. this literature review aims to address the comment by burke (2009, 42) in which the author claims “it appears that we currently have a blind spot in relation to strategies for students making effective use of feedback” and supports whitelock (2009, 199): e-assessment has the potential to offer new forms of assessment with immediate feedback to students . . . it is therefore becoming increasingly important to construct a pedagogically driven model for e-assessment that can incorporate e-assessment and e-feedback into a holistic dialogue learning framework, which recognises the importance of students reflecting upon and taking control of their own learning. findings our findings have been presented within the technological interventions identified in the approach. two additional sections have also been included regarding the use of technology to encourage student engagement with feedback as there are a wide range of articles focused on the use of computer-assisted assessment with instant research in learning technology 119 feedback and a growing use of peer assessment supported by electronic or online tools. it felt worth capturing this wealth of literature. using technology to publish feedback sending tutors’ comments electronically (bloxham and boyd 2007; crossouard and pryor 2009; denton 2001a, 2001b, 2003; denton et al. 2008; gipps 2005; irons 2008) is an effective and simple means of communicating formative feedback to students, and can enhance the way in which students receive and engage with feedback. this may be feedback on an individual or group piece of work, or feedback generic to an entire student cohort on a module. students can receive their feedback in privacy, enabling them to respond to their feedback in different ways and at different times (price and o’donovan 2008). a number of other studies have reported on the greater impact of electronic or online feedback (van den boom et al. 2004; guardado and shi 2007; tuzi 2004). price and o’donovan (2008) argued that feedback should be incorporated into the learning and teaching process both to improve student engagement with feedback and to enable the effectiveness to be measured. maclellan (2001) argued that students should be monitoring their own performance in order to make effective use of feedback to generate improvement in learning, and this has been supported by carless (2006, 229) who suggested that students should be provided with the “means to distinguish accurately their achievements in different assignments”. fulda (2005) reports on us colleges and universities mandating that grades be entered online. not only does this offer cost and efficiency savings in transcribing grades and reducing the risk of error, the same interface can be made available to students allowing them quicker access to their grades. producing feedback electronically examples of producing feedback electronically include the use of ‘track changes’ and ‘comments’ in microsoft word to alter and annotate the student’s original word-processed work (crossouard and pryor 2009), and comments typed in a separate document or digital ink using a tablet pc (plimmer and mason 2006) providing individual feedback on student work. heinrich et al. (2009) reported on the use of technology in the context of managing and marking assignments for a large student cohort. it was claimed that the real impact of technology is in administration efficiencies (e.g. documents are easily accessible to all involved in the marking process any time and any place). as such this frees up time for focusing on producing quality, more detailed and legible feedback, with the option to provide direct links to further online resources, articles and books. recent practice and studies have focused on the development of systematic methods to produce and return feedback that has been mapped against predefined assessment criteria or learning outcomes (irons 2008; sadler 2010). the use of microsoft office applications, templates and the computer-supported generation of individualised feedback documents from statement or question response banks have been developed to improve quality in response to increased student numbers (denton 2001a, 2001b, 2003; denton et al. 2008; heinrich et al. 2009; hepplestone and mather 2007; irons, 2008). hornby (2005) reports on the use of checklists and statement banks to automate the feedback process using a criterion reference 120 s. hepplestone et al. scheme in which grade related statements were generated and cut-and-pasted into a personalised feedback report for each student and returned electronically. a recent innovation is the use of audio feedback (middleton 2009), although to date studies and practice have remained small-scale despite personal recording devices being readily available and recording software relatively easy to use. in a recent study by rodway-dyer, dunne, and newcombe (2009) of first-year students receiving audio feedback, they found that the main advantage was the greater detail and depth of information offered, as well as removing misunderstandings due to illegible handwriting. a major disadvantage of the approach is that it often separates the assignment and feedback with loss of annotations (ribchester, france, and wheeler 2007), making it difficult for students to identify the point in their work being discussed (rodway-dyer, dunne, and newcombe 2009). rotheram (2009) advocates the use of audio feedback, claiming that students have the advantage of replaying feedback, although rodway-dyer, dunne, and newcombe (2009) found no evidence to suggest that students will no more listen repeatedly to audio feedback than they would re-read written feedback. merry and orsmond (2008) found that all students listened to the feedback with a copy of their submitted work in front of them at least once, with the majority of students altering their work as they listened to the feedback. in a further study of first-year and third-year students by ribchester, france, and wheeler (2007) it was found that the length of the recording may impact on student engagement: the first-year students had a natural listening threshold of around five minutes while the third-year students listened to the whole recording. despite claims by merry and osmond (2008) and rotheram (2009) that audio files are easy to store, and the findings by ribchester, france, and wheeler (2007) that students accessed the feedback off-campus, there was a reluctance by students to store feedback on personal devices (ribchester, france, and wheeler 2007). adaptive release of grades we were interested in how technology might be used to encourage students to engage with their feedback by initially providing feedback without the grades, and only revealing the grade once the student had completed a task to indicate that they had read, considered and sought the opportunity to use the feedback in their future learning. although we were unable to find examples from the literature to support this practice through the use of technology, several authors have advocated that disengaging the grade or withholding the grade from feedback promotes student learning (for example, boud and falchikov 2006; butler 1988; carless 2006; ecclestone and swan 1999; nicol 2007; potts 1992; rust, o’donovan, and price 2005). such practice also reflects the view that feedback can only support learning if it involves the production of evidence or is built into personal development planning (for example, bloxham and boyd 2007; draper 2009; higgins, hartley, and skelton 2002; irons 2008; mutch 2003; rust, o’donovan, and price 2005). computer-assisted assessment although not a focus of the original literature review, it was apparent that one of the most published uses of technology to provide students with formative feedback (or e-feedback) is through computer-based testing, quizzes or assessment using mulresearch in learning technology 121 tiple-choice or similar objective question types (bull and mckenna 2004; denton et al. 2008; handley and cox 2007). the purpose is to promote independent learning and enable students to monitor their progress and development by undertaking assessment tasks at a time and place of their choosing (irons 2008). challis (2005) promotes the opportunity computer-aided testing allows for students to effectively learn and explore areas of perceived weakness in privacy without the fear of revealing mistakes to peers or tutors, while miller (2009) discusses the use of computerassisted assessment to provide feedback to improve student learning. such tests are often made available as open access, essentially allowing students to repeat the assessment an unlimited number of times. however, despite their formative nature, the results of these tests may contribute towards a summative grade, and are therefore initially made available as a single attempt to generate the grade, and subsequently made openly available at the end of the module to aid revision. the use of commercial software products can deliver detailed formative feedback for each individual question, or specifically tailored to a particular response, more efficiently than is possible with traditional assessment (brown, bull, and race 1999; bull and mckenna 2004; gipps 2005), and students favour the immediacy of such feedback as it keeps the activity and result closely connected, meaningful and current (charman 1999; denton et al. 2008). peat and franklin (2002) describe the development of various online computer-based assessment components (including the use of weekly quizzes, mock examinations and self-assessment modules) and how students respond to these changes, while hornby (2005) discusses the efficiencies afforded by the use of automated computer assessment in allowing students to take control of their learning and self-assess. computer-adaptive testing (challis 2005; conole and warburton 2005; irons 2008) can be used in order to match the formative assessment to the individual students’ ability as well as setting benchmarks for students to attain before moving on to the next learning or assessment activity. peer assessment technological innovations and developments have been used to engage students in the process of peer assessment and peer feedback (freeman and mckenzie 2002; li, liu, and steckelberg 2010; loddington et al. 2009). both freeman and mckenzie (2002) and loddington et al. (2009) illustrate that the use of technology can improve the fairness of group assessment by acknowledging, rating and rewarding individual contributions and enhancing students’ learning from team tasks, whereas li, liu, and steckelberg (2010) found a significant relationship between the quality of the peer feedback the students provided for others and the quality of the students’ own final projects. conclusions this literature review found that there are a significant number of publications and articles that promote and suggest good feedback principles, models and practice. similarly, a wide range of literature focuses on the belief that feedback practice in its present form does not work. in some cases the authors seek answers by criticising current structures in higher education (i.e. modularisation, large student numbers and time pressures) for feedback that does not feed forward into future modules or assignments. also some literature seeks to blame students for focusing only on the 122 s. hepplestone et al. grade, not seeing the connections between modules and assessments, or concentrating on feedback comments only when the grade does not meet their expectations for effort put into completing the assessment. a growing number of studies support the hypothesis that technology has the potential to enhance student engagement with feedback, suggesting that changing the process by which feedback is made available to students can enhance student engagement with feedback. feedback published online offers students a level of flexibility in being able to read it at a time convenient to them and concentrate more deeply on the comments in the absence of their peers. as the feedback is stored online, students are offered further convenience in accessing this information whenever and wherever they complete future assessments, and where available via a virtual learning environment, for example, the feedback can be used within the context of the rest of their learning. where grades are aggregated alongside this feedback, students can use this quantitative information to inform their performance in future assessment tasks. there is further evidence to suggest that feedback returned electronically is timelier as time is saved during administrative processes. publishing feedback online can often require a change in the way feedback is produced. the use of typed comments on students’ original work enables students to see legibly the areas requiring focus and attention. the use of electronic statement banks to generate feedback aligned to the original assessment criteria, often used to speed up the generation of feedback, enables students to view their strengths and weaknesses in a structured and cohesive way and use these to set future learning targets. the growing use of audio feedback enables tutors to provide more detailed feedback than traditional written methods, although it is important to consider that students’ attention spans may diminish during listening to lengthy recordings. in using both statement banks and audio feedback, it is essential that any points raised are referred in some way back to the students’ original work, otherwise the feedback may appear separate and unhelpful. there is evidence that providing feedback without the grade encourages students to focus on the actual comments. we were interested in identifying methods used, in particular the use of technology, to encourage students to reflect on and develop action plans as a result of receiving their feedback without the grade, only receiving their grade after acting on their feedback. although we could not find any evidence of the process being supported electronically, there were suggestions that student behaviour on making use of feedback could be changed through linking the process into personal development planning. while reviewing the literature, two further uses of technology to support student engagement with feedback were identified. for some time it has been argued that the use of computer-assisted assessment can engage students by providing feedback instantly and is therefore closely connected with activity. often students repeat the assessment without the fear of publicly revealing any inadequacies in their knowledge to peers and tutors. providing students with access to such tests are increasingly popular through easy-to-use online assessment tools and within virtual learning environments. peer-assessment and peer-feedback activities are increasingly making use of technology. it is claimed that by providing feedback on their peers’ work (often anonymously), students are better equipped to engage with published assessment criteria and often make greater improvements on their own work as a result of giving feedback than from the feedback they receive themselves from their peers. research in learning technology 123 as a result of conducting this review of the literature about the technological interventions that might encourage students to engage with their feedback, we realise that day-to-day effective assessment and feedback practice is rarely reported in the literature. it may be that practitioners do not see the value in what they do due to its proximity to their practice, and we feel that there is a need for day-to-day practice to be reported and celebrated. often examples of good practice can be seen at conferences and small-scale internal events. where it is published, there tends to be a focus on the problematic nature of assessment and feedback, rather than reporting on effective strategies as well as where the use of technology can support the production, delivery and student engagement with feedback. note 1. please note, as a clarifying point this paper will use the term ‘module’ to mean an individual unit or course of study; and where quoted from third-party sources, the term ‘marks’ means ‘grades’. references black, p., and d. wiliam. 1998. assessment and classroom learning. assessment in education 5, no. 1: 7–74. bloxham, s., and p. boyd. 2007. developing effective assessment in higher education. maidenhead: open university press. boud, d., and n. falchikov. 2006. aligning assessment with long-term learning. assessment & evaluation in higher education 31, no. 4: 399–413. brown, s., j. bull, and p. race. 1999. computer-assisted assessment in higher education. london: kogan page. bull, j., and c. mckenna. 2004. blueprint for computer-assisted assessment. london: routledgefalmer. burke, d. 2009. strategies for using feedback students bring to higher education. assessment & evaluation in higher education 34, no. 1: 41–50. butler, r. 1988. enhancing and undermining intrinsic motivation: the effects of task-involving and ego-involving evaluation of interest and performance. british journal of educational psychology 58, no. 4: 1–14. carless, d. 2006. differing perceptions in the feedback process. studies in higher education 21, no. 2: 219–33. challis, d. 2005. committing to quality learning through adaptive online assessment. assessment & evaluation in higher education 30, no. 5: 519–27. chanock, k. 2000. comments on essays: do students understand what tutors write? teaching in higher education 5, no. 1: 95–105. charman, d. 1999. issues and impacts of using computer-based assessments (cbas) for formative assessment. in computer assisted assessment in higher education, ed. s. brown, j. bull, and p. race, 85–93. london: kogan page. conole, g., and b. warburton. 2005. a review of computer-assisted assessment. alt-j research in learning technology 13, no. 1: 17–31. crook, c., h. gross, and r. dymott. 2006. assessment relationships in higher education: the tension of process and practice. british educational research journal 32, no. 1: 95–114. crossouard, b., and j. pryor. 2009. using email for formative assessment with professional doctorate students. assessment & evaluation in higher education 34, no. 4: 377–88. denton, p. 2001a. generating and e-mailing feedback to students using ms office. in proceedings of the 5th international computer assisted assessment conference, july 2–3, ed. m. danson, 157–73. loughborough: learning and teaching development, loughborough university. denton, p. 2001b. generating coursework feedback for large groups of students using ms excel and ms word. university chemistry education 5, no. 1: 1–8. 124 s. hepplestone et al. denton, p. 2003. returning feedback to students via email using electronic feedback 9. learning and teaching in action 2, no. 1. http://www.celt.mmu.ac.uk/ltia/issue4/denton. shtml. denton, p., j. madden, m. roberts, and p. rowe. 2008. students’ response to traditional and computer-assisted formative feedback: a comparative case study. british journal of educational technology 39, no. 3: 486–500. department for education and skills. 2003. the future of higher education. cm 5735. london: the stationery office. draper, s. 2009. what are learners actually regulating when giving feedback? british journal of educational technology 40, no. 2: 306–15. duncan, n. 2007. ‘feed-forward’: improving students’ use of tutors’ comments. assessment & evaluation in higher education 32, no. 3: 271–83. ecclestone, k., and j. swann. 1999. litigation and learning: tensions in improving university lecturers’ assessment practice. assessment in education: principles, policy and practice 6, no. 3: 377–89. freeman, m., and j. mckenzie. 2002. spark, a confidential web-based template for selfand peer-assessment of student teamwork: benefits of evaluating across different subjects. british journal of educational technology 33, no. 5: 551–69. fulda, j.s. 2005. the ethical limitations of online grading systems. british journal of educational technology 36, no. 3: 559–61. gibbs, g., and c. simpson. 2004. conditions under which assessment supports students’ learning. learning and teaching in higher education 1: 3–31. gipps, c.v. 2005. what is the role for ict-based assessment in universities? studies in higher education 30, no. 2: 171–80. guardado, n., and l. shi. 2007. esl students’ experiences of online peer feedback. computers and composition 24, no. 4: 443–61. handley, k., and b. cox. 2007. beyond model answers: learners’ perceptions of self-assessment materials in e-learning applications. alt-j research in learning technology 15, no. 1: 21–36. heinrich, e., j. milne, a. ramsay, and d. morrison. 2009. recommendations for the use of e-tools for improvements around assignment marking quality. assessment & evaluation in higher education 34, no. 4: 469–79. hepplestone, s., and r. mather. 2007. meeting rising student expectations of online assignment submission and online feedback. in proceedings of the 11th international computer-assisted assessment conference, july 10–11, ed. farzana khandia, 269–75. loughborough: learning and teaching development, loughborough university. higgins, r., p. hartley, and a. skelton. 2002. the conscientious consumer: reconsidering the role of assessment feedback in student learning. studies in higher education 27, no. 1: 53–64. higher education funding council for england. 2007. 2007 teaching quality information data. bristol: hefce. http://www.hefce.ac.uk/learning/nss/data/2007/. hornby, w. 2005. dogs, stars, rolls royces and old double-decker buses: efficiency and effectiveness in assessment. in: enhancing practice: reflections on assessment: volume 1, 15–28. gloucester: quality assurance agency for higher education. hounsell, d. 2008. the trouble with feedback: new challenges, emerging strategies. tla interchange 2. http://www.tla.ed.ac.uk/interchange/spring2008/hounsell2.htm. hounsell, d., and n. entwistle. 2007. learning and teaching at university: the influence of subjects and settings. teaching and learning research briefings no. 31. london: esrc teaching and learning research programme. irons, a. 2008. enhancing learning through formative assessment and feedback. abingdon: routledge. li, l., x. liu, and a.l. steckelberg. 2010. assessor or assessee: how student learning improves by giving and receiving peer feedback. british journal of educational technology 41, no. 3: 525–36. loddington, s., k. pond, n. wilkinson, and p. willmot. 2009. a case study of the development of webpa: an online peer moderated marking tool. british journal of educational technology 40, no. 2: 329–41. research in learning technology 125 maclellan, e. 2001. assessment for learning: the differing perceptions of tutors and students. assessment & evaluation in higher education 26, no. 4: 307–18. mcdowell, l., k. sambell, v. bazin, r. penlington, d. wakelin, h. wickes, and j. smailes. 2005. assessment for learning: current practice exemplars from the centre for excellence in learning and teaching. newcastle: centre for excellence in teaching and learning, university of northumbria. merry, s., and p. orsmond. 2008. students’ attitudes to and usage of academic feedback provided via audio files. bioscience education 11. http://www.bioscience.heacademy.ac.uk/ journal/vol11/beej–11–3.aspx. middleton, a. 2009. beyond podcasting: creative approaches to designing educational audio. alt-j research in learning technology 17, no. 2: 143–55. miller, t. 2009. formative computer-based assessment in higher education: the effectiveness of feedback in supporting student, learning. assessment & evaluation in higher education 34, no. 2: 181–92. mutch, a. 2003. exploring the practice of feedback to students. active learning in higher education 4, no. 24: 24–38. nesbit, p.l., and s. burton. 2006. student justice perceptions following assignment feedback. assessment & evaluation in higher education 31, no. 6: 655–70. nicol, d. 2007. improving assessment after the task. re-engineering assessment practices in scottish education (reap) online resources. http://www.reap.ac.uk/nss/nssafter02. html. nicol, d. 2009. assessment for learner self-regulation: enhancing achievement in the first year using learning technologies. assessment & evaluation in higher education 34, no. 3: 335–52. nicol, d.j., and d. macfarlane-dick. 2006. formative assessment and self-regulated learning: a model and seven principles of good feedback practice. studies in higher education 31, no. 2: 199–218. parkin, h.j., s. hepplestone, g. holden, b. irwin, and l. thorpe. forthcoming a role for technology in enhancing student engagement with feedback. assessment & evaluation in higher education. peat, m., and s. franklin. 2002. supporting student learning: the use of computer-based formative assessment modules. british journal of educational technology 33, no. 5: 515– 23. plimmer, b., and p. mason. 2006. a pen-based paperless environment for annotating and marking student assignments. in proceedings of seventh australian user interface conference (auic2006), january 16–19, ed. wayne piekarski, 37–44. darlinghurst, australia: australian computer society inc. potts, d. 1992. case study: you can’t teach those things to rats. a case for neither grading nor failing students. educational and training technology international 29, no. 4: 296– 309. poulous, a., and m.j. mahony. 2007. effectiveness of feedback: the students’ perspective. assessment & evaluation in higher education 33, no. 2: 143–54. price, m., and b. o’donovan. 2008. feedback – all that effort, but what is the effect? paper presented at the earli/northumbria assessment conference, august 27–29, in potsdam, germany. price, m., k. handley, j. millar, and b. o’donovan. 2010. feedback: all that effort, but what is the effect? assessment & evaluation in higher education 35, no. 3: 277–89. ribchester, c., d. france, and a. wheeler. 2007. podcasting: a tool for enhancing feedback? paper presented at the 4th education in a changing environment international conference 2007, september 12–14, in salford, uk. rodway-dyer, s., e. dunne, and m. newcombe, m. 2009. audio and screen visual feedback to support student learning. in proceedings of the 16th alt-c international conference, september 8–10, ed. hugh davies and linda creanor, 61–9. oxford: association for learning technology. rotheram, b. 2009. sounds good: quicker, better assessment using audio feedback. jiscfunded project. http://sites.google.com/site/soundsgooduk/downloads. 126 s. hepplestone et al. rowe, a.d., and l.n. wood. 2007. what feedback do students want? paper presented at the aare 2007 international educational research conference, november 25–29, in fremantle, australia. rust, c., b. o’donovan, and m. price. 2005. a social constructivist assessment process model: how the research literature shows us this could be best practice. assessment & evaluation in higher education 30, no. 3: 231–40. sadler, d.r. 2010. beyond feedback: developing student capability in complex appraisal. assessment & evaluation in higher education 35, no. 5: 535–50. tuzi, f. 2004. the impact of e-feedback on the revisions of l2 writers in an academic writing course. computers and composition 21, no. 2: 217–35. van den boom, g., f. paas, j.j.g. van merriënboer, and t. van gog. 2004. reflection prompts and tutor feedback in a web-based learning environment: effects on students’ self-regulated learning competence. computers in human behavior 20, no. 4: 551–67. weaver, m.r. 2006. do students value feedback? student perceptions of tutors’ written responses. assessment & evaluation in higher education 31, no. 3: 379–94. whitelock, d. 2009. editorial: e-assessment: developing new dialogues for the digital age. british journal of educational technology 40, no. 2: 199–202. wojtas, o. 1998. feedback? no, just give us the answers. times higher education supplement, september 25, 7. yorke, m. 2001. formative assessment and its relevance to retention. higher education research and development 20, no. 2: 115–26. research in learning technology 127 alt-j, research in learning technology vol. 15, no. 1, march 2007, pp. 83–97 issn 0968-7769 (print)/issn 1741-1629 (online)/07/010083–15 © 2007 association for learning technology doi: 10.1080/09687760601130057 the university student experience of face-to-face and online discussions: coherence, reflection and meaning robert a. ellisa*, peter goodyeara, agi o’haraa and michael prosserb auniversity of sydney, australia; bhigher education academy, australia taylor and francis ltdcalt_a_212937.sgm10.1080/09687760601130057alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis151000000march 2007robertellisr.ellis@itl.usyd.edu.au this paper reports on an investigation into learning through discussions by undergraduate social work students. second-year students studying psychology for social work experienced discussions began with face-to-face tutorials, and then continued for some time after online. this study used closed-ended questionnaires to investigate what students thought they were learning through discussions (their concepts), and how they engaged in the discussions face-to-face and online (their approaches). significant associations were found among students’ concepts of discussions, approaches and levels of achievement. the results suggest that students who do not understand how discussions can help them to interrogate, reflect on and revise their ideas tended not to approach either face-to-face or online discussions in ways likely to improve their understanding or their levels of achievement. this type of insight is critical for teacher/designers wishing to create university experiences in which discussion is used to promote learning. introduction e-learning as a key part of the university student learning experience is no longer the preserve of distance teaching universities (hefce, 2004; educause, 2005). it is now common for campus-based universities to invest systematically in e-learning technologies to support the student learning experience. motivations to use information and communications technology in combination with face-to-face teaching activities include creating new opportunities for learning collaborations and partnerships, strengthening and extending learning communities, promoting new ways of communicating and investigating, and providing better access to an increasingly wide range of discipline-specific educational and research-based pedagogic resources. * corresponding author. institute of teaching and learning, carslaw f07, university of sydney, sydney, nsw2006, australia. email: r.ellis@itl.usyd.edu.au 84 r. a. ellis et al. while e-learning activities and materials have a lot to offer, it is reasonable to ask how students experience coherence in their learning activity when it is distributed across face-to-face and online contexts. integrating online learning to complement face-to-face experiences presents students and teachers with some difficulties. in order to embed the online part of the experience, it is sensible and meaningful to use it to help students to attain the intended learning outcomes of the course (biggs, 2005). but what happens when students are expected to follow a learning activity across face-to-face and online contexts in order to achieve their learning outcomes? we should not assume that following the intent of the teacher/ designer throughout a learning activity across face-to-face and online contexts is unproblematic. certainly we need some evidence about how students experience learning across the combined or ‘blended’ learning contexts, otherwise we may put the outcomes of an increasing amount of the university student learning experience at risk. this study is designed to investigate a university student experience of learning through discussions that extend across face-to-face and online contexts. the research setting was a course in psychology taken by second-year social work students. as part of their learning activities, students were expected to engage in structured discussions in face-to-face and online tutorial groups. the discussions were intended to help them reflect on, and understand, theoretical underpinnings and key issues in their course. this research study investigates the strength of associations between key aspects of the face-to-face and online student experiences, and the extent to which the students experienced associations between these contexts and their learning outcomes. the main research questions are as follows: ● how do students experience the combination of face-to-face and online discussions? ● do all students experience them in ways that support their learning? ● what aspects of the student experience of discussions are important for the teacher/designer when considering their systematic use in the student experience? theoretical background the theoretical background for this study includes phenomenographic research on student learning in higher education, and research on reflection as an important aspect of the student learning experience. the former provides the basis for the closed-ended instruments used in the investigation. the latter is used to extend the scope of the research and to offer a way of understanding the motivation of this intervention into the ways the social work students were engaging in discussion. research into student learning in higher education seminal research on student learning in higher education, over the past 30 years, has systematically identified the relational nature of key aspects of the student experience (marton & saljo, 1976a, b; entwistle & ramsden, 1983; ramsden, 2002; marton & student experience of face-to-face and online discussions 85 booth, 1997; prosser & trigwell, 1999; biggs, 2003). these aspects are shown in figure 1. figure 1. the 3p model of learning (after trigwell & prosser, 1997) studies that underpin figure 1 have shown that key aspects of the student learning experience in higher education include: ● students’ conceptions—what they think they are learning: box 5 (crawford et al., 1998; prosser & trigwell, 1999). ● students’ approaches—what they do when they learn: box 4 (prosser & millar, 1989). ● how these aspects are related to their perceptions and the learning context: boxes 3 and 2 (ramsden, 1991)—and to their prior experiences: box 1 (prosser & trigwell, 1999). a significant part of this research tradition has been the development and use of questionnaires consisting mainly of closed-ended questions to identify and probe these aspects of student learning. the revised study process questionnaire (biggs et al., 2001), the course experience questionnaire (ramsden, 1991) and conceptions of learning questionnaires (crawford et al., 1998) are examples of instruments that have been used to investigate approaches, perceptions and conceptions, respectively. development of the questionnaires used in the current study has been informed by knowledge of the development processes and structure of these wellestablished instruments (see method section). in earlier research we have reported on the qualitative analysis of interview data in which students have told us about how they learn through discussions (ellis et al., 2006). the context of that study was similar to the one described in this paper in both structure and content, as it was a pre-requisite for the course investigated in this ��������������� � ��� ��� ��� � � ���� �� ������������ �������� � ����� ����� � ���� ������ �������� ���� �� �������� �� �������� ����������� � ��� ����� �������� �� � � ����� ����� � ��������� ����� � ���� ��� ����� ���� ����� � �������� �� � ���� ����� � ���� �� ����� �� �������� ��� �������� ��� ����� �������� ��� ��� ����� ���� ������ � ���� �� ����� �� �������� ��� �������� !" figure 1. the 3p model of learning (after trigwell & prosser, 1997) 86 r. a. ellis et al. study. in the previous study we identified qualitatively different conceptions of discussions, and approaches to discussions, in face-to-face and online contexts. these are summarized below. categories of conceptions of learning through discussions: ● conception a: discussions as a way of challenging ideas and beliefs in order to arrive at a more complete understanding. ● conception b: discussions as a way of challenging and improving your ideas. ● conception c: discussions as a way of collecting ideas. ● conception d: discussions as a way of checking your ideas are right. the categories of conceptions can be separated into two groups: categories a & b and categories c & d. categories a & b conceived of discussions as a way of challenging the ideas being discussed to promote holistic understanding. these categories of conceptions were meaningfully related to the course goals and were logically associated with the categories of approaches reported below. in contrast, categories c & d conceived of discussions as a much less interactive way of learning, more about hearing and checking the accuracy of ideas rather than developing and reconceptualizing them. the previous study also identified qualitatively different approaches to learning through discussions, both face-to-face and online. categories of approaches to learning through face-to-face discussions: ● approach a: engaging in face-to-face discussions to analyse experiences and opinions through feedback. ● approach b: engaging in face-to-face discussions to analyse experiences and opinions. ● approach c: engaging in face-to-face discussions to hear other experiences and ideas. ● approach d: engaging in face-to-face discussions to fulfil task requirements. categories of approaches to learning through online discussions: ● approach a: engaging in online discussions to evaluate postings to reflect on key ideas. ● approach b: engaging in online discussions to evaluate postings to challenge ideas. ● approach c: engaging in online discussions to use postings to add to ideas. ● approach d: engaging in online discussions to read postings to avoid repetition. the categories of approaches a–d in both face-to-face and online discussions are qualitatively different. it is possible to group each set of categories into two. in both sets of categories, those labelled a–b are approaches with an underlying intention of reflection; in face-to-face discussions, the approaches emphasize the analysis of the experiences of others; and in the online discussions, the emphasis is on the evaluation of postings in relation to the ideas being considered. in contrast, categories c–d are qualitatively different. in the face-to-face context, hearing ideas and completing the task requirements were emphasized. in the online context, the use of postings as a student experience of face-to-face and online discussions 87 way of adding to existing ideas and reading postings to avoiding accidental repetition of others’ postings were emphasized. reflection is an important construct underlying the category system presented above. it was prominent in the in-depth interviews with students in the previous study. the students reported reflection as one of the most valuable affordances that the online postings provided. this was true of the ideas that had been expressed in class and the additional ideas that were being posted online. this aspect of the student experience, opportunity for reflection, was one of the main motivations for continuing with the discussion-based teaching strategy in the course under investigation in this study. research into reflection as a way of learning reflection and reflective practice have been the focus of important research into student learning. deep reflection is recognized as a higher-order learning activity (schön, 1983; biggs, 2003). reflection on discussions as a way of learning can be usefully conceptualized as reflection occurring during learning and reflection occurring on learning that has taken place (schön, 1983; salmon, 2002). despite superficial differences in terminology, research into online discussions, computermediated conferencing and networked learning often focuses on very similar kinds of learning tasks. the results from such studies reveal a degree of convergence: that good online discussions foster effective collaborative learning (dewiyanti et al., 2005); that students can learn through reflecting deeply on the online submissions made by others if they understand the purpose of the activity and approach it in useful ways (ellis et al., 2004); that reflection can be part of an active learning structure for online discussions that facilitate the sharing of different viewpoints and ideas to develop understanding (gilbert & dabbagh, 2005); and that students do not necessarily have to participate directly in a discussion to benefit from reflecting through listening or reading (mckendree et al., 1998). the purpose of the current study is to extend previous research to investigate how key parts of the face-to-face and online learning experiences were related, from a student perspective, so that a better understanding, and evidence, of the nature and extent of the success of the intervention could be obtained. the research site and nature of the educational intervention second-year social work students studied psychology as a core part of their undergraduate degree in a metropolitan australian university. the lecturer embedded structured face-to-face and online discussions into the course, with the aim of helping the students extend their face-to-face discussions into the online environment, thereby providing opportunities for reflection on theoretical ideas and on the practice of social work. the introduction of the blended discussions was accompanied by instruction in class of the value of extending the classroom discussions and their ideas into the online environment. students were encouraged to use their composition 88 r. a. ellis et al. process of writing postings, and the postings of the other students, to reflect on the key ideas of their course. the students were expected to engage in the discussions to help them to come to terms with knowledge and theories in psychology and to understand their relevance and application to the profession of social work. the online discussions were prompted by the issues and questions that had been raised in the face-to-face tutorials, but often went beyond these. the students were therefore able to extend their face-to-face discussions by making at least two postings (of 200 words each) online during the week following their face-to-face tutorial. they had the choice of starting their own discussion thread or, alternatively, they could reply to a posting made by another student. the students were awarded 13% of their final mark for tutorial participation, determined by the quality of their online postings. all postings by students were made using the discussion tool in webct, campus version 4.1. participants in this study were recruited by approaching the students in one of their lectures at the end of the 13-week semester. volunteers were asked to complete three questionnaires. in this cohort of social work students, 48 out of 105 students completed the questionnaires, representing 46% of the cohort. research method questionnaires students who took part in the surveys completed three closed-ended questionnaires: the ‘conceptions of learning through discussions’ questionnaire, the ‘face-to-face approaches to learning’ questionnaire and the ‘online approaches to learning’ questionnaire. the original development of the questionnaires was informed by previous research (biggs, 1987; ramsden, 1991; crawford et al., 1998; prosser et al., 2000) and they had been refined in more recent research (ellis & calvo, 2004). the ‘conceptions of learning through discussions’ questionnaire has two subscales; a cohesive subscale and a fragmented subscale. the cohesive subscale investigates a conception of discussions that is orientated towards probing the underlying meaning of topics, their relationship to the learning outcomes of the course and their usefulness in terms of a holistic understanding of the disciplinary issues being discussed. the fragmented subscale investigates a conception of discussions that separates the purpose of the discussions from the context in which they are held. this type of conception is one that emphasizes elements of discussions such as communication skills, finding correct answers or plugging holes in arguments, rather than anything more directly concerned with meaningful understanding. the fragmented and cohesive subscales of the questionnaire are consistent with the theoretical model shown in figure 1, and their alpha values1 fell within an acceptable range (α = 0.74, α = 0.89) compared with similar previous studies (0.60 < α < 0.89) (ramsden et al., 1989; ellis & calvo, 2004). using discussions as a way of reflecting is a constituent part of the coherent subscale: for student experience of face-to-face and online discussions 89 example, item 11 states ‘discussing in this subject is like a process of reflection that allows us to better understand the things we study’. the two questionnaires investigating face-to-face and online approaches had previously been piloted as one questionnaire. students reported that they preferred to answer items on questionnaires that more clearly separated the two contexts and their answers. this led to the development of a ‘face-to-face approaches to discussions’ questionnaire and an ‘online approaches to discussions’ questionnaire. the ‘face-to-face approaches to discussions’ questionnaire is divided into two subscales: deep approaches and surface approaches. the deep subscale taps approaches orientated towards understanding ideas from the experience of others, to learn about the ideas of others and to evaluate the way they think about a topic. the surface subscale taps approaches that were less meaning-directed in intent, such as using discussions to complete set tasks, or reduce the need to do as much reading by seeing what others have read. the face-to-face approaches subscales are consistent with the theoretical model shown in figure 1 and their alpha values (deep α = 0.81, surface α = 0.66) fell within an acceptable range compared with similar previous studies (0.60 < α < 0.89) (ramsden et al., 1989; ellis & calvo, 2004). reflective evaluation is a constituent part of the face-to-face deep approach subscale. for example, item 11 states ‘i use discussions in class as a way of evaluating the way i am thinking about a topic’. the ‘online approaches to discussions’ questionnaire is similarly divided into two subscales: deep and surface. the deep subscale taps approaches orientated towards integrating key ideas from postings into reconceived conceptions, both reflecting on ideas before making comment and evaluating the ideas of others using some understanding of who is making the comment. the surface subscale taps aspects such as waiting to the last possible moment before making a posting, worrying about appearing ignorant or posting only because postings are assessed. the online approach subscales are consistent with the theoretical model shown in figure 1 and their alpha values (deep α = 0.73, surface α= 0.76 fell within an acceptable range compared with similar previous studies (0.60 < α < 0.89) (ramsden et al., 1989; ellis & calvo, 2004). reflection is a constituent part of a deep approach to online discussions. for example, item five states ‘online discussions allow me time to reflect on what i wish to say before writing my posting’. table 1 presents example items of the subscales and their alpha coefficients. results the conceptions and approaches questionnaires were analysed to investigate the extent of coherence across the face-to-face and online contexts in the student experience of discussions. the statistical treatment included correlation, factor and cluster analyses. correlation analyses investigate relationships between pairs of variables. factor analyses investigate relationships among groups of variables and add to the integrity to the study. cluster analyses are used to see whether there are similar groupings of students in the population. 90 r. a. ellis et al. correlation analyses pearson correlation coefficients were used to investigate the strength of the relationships between pairs of variables. table 2 presents the results of the correlation analyses. table 2 provides some indication of relationships between the face-to-face and online contexts and outcomes. the cohesive conceptions variable shows a large positive association with the deep face-to-face approach variable (r = 0.77, p < 0.00), a negative medium association with the surface face-to-face approach variable (r = − 0.30, p < 0.01) and a medium positive association with the deep online approach table 1. scales of the questionnaires used in this study, defining items and internal consistency reliabilities scale number defining items conceptions of learning through discussions cohesive conception (five items, α = 0.89) 3 by discussing i can develop my understanding of the things i am studying in this subject 11 discussing in this subject is like a process of reflection that allows us to better understand the things we study fragmented conception (three items, α = 0.74) 2 the purpose of discussions in this subject is to learn how to win the argument 9 discussing in this subject is just about finding the right answer approaches to learning through face-to-face discussions face-to-face deep approaches (four items, α = 0.81) 1 i find that at times the discussions in class help me to understand the topic from the experience of others 5 i find that discussions in class help me to understand the topic being discussed more deeply. face-to-face surface approaches (three items, α = 0.66) 6 i discuss issues in class so that i do not have to do as much reading on a topic 8 i engage in discussions in class only in order to finish a task approaches to learning through online discussions online deep approaches (four items, α = 0.73) 3 in looking at postings online, i integrate key ideas into my own understanding of a topic 5 online discussions allow me time to reflect on what i wish to say before writing my posting online surface approaches (three items, α = 0.76) 4 when i look at online postings, my main concern is to avoid posting something that suggests i do not know what i’m talking about 12 i usually make online postings at the last moment possible student experience of face-to-face and online discussions 91 t ab le 2 . c o rr el at io n s b et w ee n p ar ts o f th e st u d en t le ar n in g ex p er ie n ce a n d a ch ie ve m en t v ar ia b le c o h es iv e co n ce p ti o n s f ra gm en te d co n ce p ti o n s d ee p f ac eto fa ce a p p ro ac h s u rf ac e fa ce -t o fa ce a p p ro ac h d ee p o n li n e ap p ro ac h s u rf ac e o n li n e ap p ro ac h c o u rs e m ar k c o n ce p ti o n s c o h es iv e co n ce p ti o n s 1 −0 .2 0 0. 77 ** −−−− 0 .3 0* 0. 41 ** 0 .1 3 0 .1 0 f ra gm en te d c o n ce p ti o n s 1 −0 .0 8 0. 35 * 0 .0 3 0 0 .0 8 9 −−−− 0 .3 1* f ac eto -f ac e ap p ro ac h es d ee p f ac eto -f ac e ap p ro ac h 1 −0 .1 3 0. 41 ** 0 .1 8 −0 .1 0 s u rf ac e fa ce -t o -f ac e ap p ro ac h 1 −0 .1 0 0 .0 6 −0 .0 9 o n li n e ap p ro ac h es d ee p o n li n e ap p ro ac h 1 0 .0 6 0 .0 6 s u rf ac e o n li n e ap p ro ac h 1 −0 .2 1 a ch ie ve m en t c o u rs e m ar k (c m ) 1 n = 4 8 . *p < 0 .0 1 , ** p < 0 .0 0 1 , b o ld = s ta ti st ic al ly s ig n ifi ca n t. 92 r. a. ellis et al. variable (r = 0.41, p < 0.00). the fragmented conceptions variable shows a medium positive association with the surface face-to-face approach variable (r = 0.35, p < 0.00) and a medium negative association with the course mark (r = −0.31, p < 0.00). while the remaining correlation results are not statistically significant, they are generally in the expected directions. factor analyses factor analyses look at the structural relationships among groups of variables. factor analyses group variables together that seem to be related to each other and identify those variables that are not part of each group (see table 3). in table 3, factor one shows that a higher score on the cohesive conception variable (0.89) is positively related to the deep face-to-face approach variable (0.89), and the deep online approach variable (0.64). factor two shows that a higher score on the fragmented conceptions variable (0.76) is positively associated with the surface faceto-face approach variable (0.56), the surface online approach variable (0.49) and negatively related to the course mark (−0.69). the factor analysis suggests that the variable representing a cohesive conception of discussions as a way of learning is strongly associated with a deep face-to-face approach to discussions and a deep online approach to discussions. likewise the variable representing a fragmented conception of discussions is strongly linked to a surface face-to-face approach to discussions and a surface online approach to discussions, and is negatively related to the course mark. the factor analysis suggests that the variable representing a cohesive conception of discussions as a way of learning is strongly associated with a deep face-to-face approach to discussions and a deep online approach to discussions. likewise the table 3. principal components factor analysis of conceptions, approaches and achievement variables variable factor one factor two conceptions cohesive conceptions 0.89 fragmented conceptions 0.76 face-to-face approaches deep face-to-face approach 0.89 surface face-to-face approach 0.56 online approaches deep online approach 0.64 surface online approach 0.47 achievement course mark −0.69 loadings of magnitude less than 0.4 are omitted. varimax rotation, kmo (kaiser-meyer-olkin) = 0.58. eigenvalue 2.3 and 1.6, 55% variance explained, n = 48. student experience of face-to-face and online discussions 93 variable representing a fragmented conception of discussions is strongly linked to a surface face-to-face approach to discussions and a surface online approach to discussions, and is negatively related to the course mark. correlation and factor analyses are used to interrogate data at the level of the variables. to investigate the data at the level of the individual student, a cluster analysis was used. the following methodology draws on a modified version of the approach reported in prosser et al. (2000). hierarchical cluster analysis is a technique that can be used to identify subgroups within a sample on the basis of similarities of the variables being investigated. in this cluster analysis, standardized scores are used for the subscales of conceptions of learning through discussions (fragmented and cohesive), approaches to learning through discussions (deep and surface) and the course mark that all students received. standardized scores have a mean of zero and a standard deviation of one to make comparisons easier. the hierarchical cluster analysis using ward’s technique (seifert, 1995) revealed two clusters of students, based on the increasing value of the squared euclidean distance between the clusters (crawford et al, 1998; prosser et al., 2000). the results are presented in table 4. table 4 shows two groups of students identified in the cluster analysis: one cluster (21 students) scored strongly positive on the fragmented and surface variables, and negatively on the others; and another cluster (27 students) scored strongly positive on the cohesive and deep variables, and negatively on the others. in table 4, one cluster of students who held a poor conception of learning through discussions, judged by a large negative score on the cohesive conception table 4. cluster analysis of parts of the student learning experience and achievement (n = 48) cluster (standardized means) variable cluster 1 (n = 21) cluster 2 (n = 27) statistical significance if p < 0.05 conceptions cohesive conceptions −0.69 0.54 0.001 fragmented conceptions 0.23 −0.18 – face-to-face approaches deep face-to-face approach −0.62 0.48 0.001 surface face-to-face approach 0.65 −0.50 0.001 on-line approaches deep on-line approach −0.51 0.40 0.001 surface on-line approach 0.02 −0.02 – achievement course mark 0.00 −0.01 – 94 r. a. ellis et al. variable (−0.69, p < 0.00), had a strong negative score on the deep face-to-face approach variable (−0.62, p < 0.00), a strong positive score on the surface face-toface approach variable (0.65, p < 0.00) and a strong negative score on the deep online approach variable (−0.51, p < 0.00). in contrast, the other cluster of students who held a good conception of learning through discussions, judged by a large positive score on the cohesive conception variable (0.54, p < 0.00), had a medium positive score on the deep face-to-face approach variable (0.48, p < 0.00), a large negative score on the surface face-to-face approach variable (−0.50, p < 0.00) and a medium positive score on the deep online approach variable (0.40, p < 0.00). discussion and conclusions the results of the study are promising and consistent with both prior research and the theoretical framework in which the study is conducted. the small size of the sample means that any interpretations of the outcomes should be seen as suggestive of trends and tendencies, rather than as conclusive evidence of relationships. bearing that in mind, there are a number of implications for thinking about student learning, teaching and further research. the consistency of the results across the different types of analyses endorses the value of this approach to researching the coherence between face-to-face and online learning contexts. this is an important outcome if we acknowledge that the growth of what is coming to be known as ‘blended learning’ in higher education has resulted in a need to devise new ways of measuring and investigating qualitatively different learning experiences of students when they are expected to follow a learning activity across different learning contexts. in this study, discussion activities were shown to be fragmented from more meaningful learning when the students lacked an awareness of associations between discussions and the learning outcomes of the course or purpose of the task. the evidence suggesting this is the scales scores for fragmented conceptions and surface approaches in the analyses. if the students did not understand how discussions could help them interrogate, reflect on and revise their ideas, they tended not to approach either the face-to-face or online discussions in ways likely to improve their understanding. this would mean that any reflective aspects of the experience, which were part of the initial motivation for the intervention, would be lost on these students. it is clear from the analyses that discussion activities stretching across the contexts were coherent for some students. the responses of slightly more than one-half of the students (as measured by the cluster analyses) suggest that the students experienced the discussions in a more coherent and meaningful way. cohesive conceptions of discussions tended to be strongly associated with deep approaches in face-to-face and online contexts. while there was no strong association with course mark in the cluster analyses, there was association at the level of variables in the factor analyses. this needs closer investigation in further studies. it may be that a larger population sample may find a significant relationship between achievement and the quality of the student experience of face-to-face and online discussions 95 experience. overall, the qualitative variation indicated by the analyses within and across the face-to-face and online contexts has important implications for practice. much is made of the benefits of online discussion for student learning, including its reflective, collaborative and evaluative affordances (salmon, 2001; pilkington & walker, 2003). the results of this study are consistent with a positive view of the value of online discussions, but suggest a few contingencies that probably need to be considered if discussions are to support deep learning. one is that their value needs to be considered in relation to face-to-face discussions when the student experience contains both: in other words, we cannot simply hope for positive outcomes from online discussions if we insert them in a face-to-face context without considering how their design should intersect with the discussions in the face-to-face context. other important contingencies suggested as a way of extracting the most from online discussions is that they are more likely to benefit the student experience of learning if students understand how the discussions are related to the purpose of the task and learning outcomes of their course and if discussions are approached by the students in ways that are likely to help them interrogate, review and reconceive fundamental understandings. these features of good discussions cannot be taken for granted. without understanding why they are discussing, and how to approach the discussions appropriately, in both face-to-face and o-line contexts, students find it difficult to see the connections between the discussions in class and online and what they are supposed to be learning. for the teacher/designer, this insight is critical. it should be one of the main foundations informing the shape and design of such ‘distributed’ discussions. for some time now, the literature has been telling us that it is unsafe to assume that students know how to discuss online (see, for example, kaye, 1992; goodyear, 1996). this study shows that it is unsafe to assume that they will all know how to discuss things effectively face-to-face, despite this activity being around for centuries. when face-to-face and online discussions are both involved, the dangers of some students missing out on learning opportunities are even greater. further research with larger samples and in other discipline areas would build confidence in these findings. but there are already clear implications that students need to be helped to a better understanding of what they can gain through mindful engagement in productive discussions (whether online, face-to-face or both) and that such an understanding needs to be reflected in the design of discursive learning tasks. acknowledgements the authors are pleased to acknowledge the financial assistance of the australian research council (grant dp0559282 awarded to p.g., r.a.e. and m.p.) in funding used to carry out this study. they would also like to thank ana-maria bliuc for assistance with the literature search for this study. note 1. cronbach’s alpha reliability coefficient for internal consistency. 96 r. a. ellis et al. references biggs, j. b. (1987) student approaches to learning and studying (melbourne, australian council for educational research). biggs, j. b. (2003). teaching for quality learning at university (2nd edn) (buckingham, society for research into higher education & open university press). biggs, j. b. (2005) aligning teaching for constructing learning. higher education academy discussion paper. available online at: http://www.heacademy.ac.uk/embedded_object.asp?id =21686&filename=biggs (accessed 14 january 2006). biggs, j. b., kember, d. & leung, d. y. p. (2001) the revised two-factor study process questionnaire: r-spq-2f, british journal of educational psychology, 71, 133–149. crawford, k., gordon, s., nicholas, j. & prosser, m. (1998) university mathematics students’ conception of mathematics, studies in higher education, 23, 87–94. dewiyanti, s., brand-gruwel, s. & jochems, w. (2005) applying reflection and moderation in an asynchronous computer-supported collaborative learning environment in campus-based higher education, british journal of educational technology, 36(4), 673–376. educause (2005) ecar study of students and information technology 2005: convenience, connection, control, and learning on research on american undergraduate students and use of ict (colarado, educause). ellis, r. a. & calvo, r. a. (2004) learning through discussions in blended environments, educational media international, 40(1), 263–274. ellis, r. a., calvo, r. a., levy, d. & tan, k. (2004) learning through discussions, higher education research and development, 23(1), 73–93. ellis, r. a., goodyear, p., prosser, m. & o’hara, a. (2006) how and what university students learn through online and face-to-face discussions: conceptions, intentions and approaches, journal of computer assisted learning, 22, 244–256. entwistle, n. & ramsden, p. (1983) understanding student learning (london, croom helm). gilbert, p. k. & dabbagh, n. (2005) how to structure online discussions for meaningful discourse: a case study, british journal of educational technology, 36(1), 5–18. goodyear, p. (1996) asynchronous peer interaction in distance education: the evolution of goals, practices and technology, training research journal, 1, 71–102. hefce (2004) e-learning strategy (london, hefce). kaye, a. (1992) learning together apart, in: a. kaye (ed.) collaborative learning through computer conferencing: the najaden papers (berlin, springer verlag), 1–24. marton, f. & booth, s. (1997) learning and awareness (nj, lawrence erlbaum and associates publishers). marton, f. & säljö, r. (1976a) on qualitative differences in learning. i. outcome and process, british journal of educational psychology, 46, 4–11. marton, f. & säljö, r. (1976b) on qualitative differences in learning. ii. outcome as a function of the learner’s conception of the task, british journal of educational psychology, 46, 115–127. mckendree, j., stenning, k., mayes, t., lee, j. & cox, r. (1998) why observing a dialogue may benefit learning, journal of computer assisted learning, 14, 110–119. pilkington, r. & walker, s. (2003) facilitating debate in networked learning: reflecting on online synchronous discussion in higher education, instructional science, 31, 41–63. prosser, m. & millar, r. (1989) the ‘how’ and ‘what’ of learning physics, european journal of psychology of education, 4, 513–528. prosser, m. & trigwell, k. (1997) perceptions of the teaching environment and its relationship to approaches to teaching, british journal of educational psychology, 67, 25–35. prosser, m. & trigwell, k. (1999) understanding learning & teaching: the experience in higher education (buckingham, society for research into higher education & open university press). student experience of face-to-face and online discussions 97 prosser, m., hazel, e. & waterhouse, f. (2000) students’ experiences of studying physics concepts: the effects of disintegrated perceptions and approaches, european journal of psychology of education, 15(1), 61–74. ramsden, p. (1991) a performance indicator of teaching quality in higher education: the course experience questionnaire, studies in higher education, 16, 129–150. ramsden, p. (2002) learning to teach in higher education (2nd edn) (london, routledge). ramsden, p., martin, e. & bowden, j. (1989) school environment and sixth form pupils’ approaches to learning, british journal of educational psychology, 59, 129–142. salmon, g. (2001) e-moderating: the key to teaching and learning online (london, kogan page). salmon, g. (2002) mirror, mirror on my screen … exploring online reflections, british journal of educational technology, 33(4), 379–391. schön, d. a. (1983) the reflective practitioner: how professionals think in action (new york, basic books). seifert, t. (1995) characteristics of egoand task-orientated students: a comparison of two methodologies, british journal of educational psychology, 65, 125–138. trigwell, k. & prosser, m. (1997) towards an understanding of individual acts of teaching and learning, higher education research and development, 16(2), 241–252. alt-j, research in learning technology vol. 15, no. 1, march 2007, pp. 51–65 issn 0968-7769 (print)/issn 1741-1629 (online)/07/010051–15 © 2007 association for learning technology doi: 10.1080/09687760601129851 an information and communications technology (ict)-enabled method for collecting and collating information about pre-service teachers’ pedagogical beliefs regarding the integration of ict michael vallance* future university, japan taylor and francis ltdcalt_a_212917.sgm10.1080/09687760601129851alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis151000000march 2007associate professor michaelvallancemichael@fun.ac.jp this paper describes a method that utilized technology to collect and collate quantitative and qualitative data about pre-service teachers’ use of networked technologies during a 12-week undergraduate course, and the impact of this use on their pedagogical beliefs regarding the integration of information and communications technology (ict). the technologies used captured and analysed students’ spoken and written communication while engaging in four synchronous online tasks, and also collected evaluation data from online interviews, surveys and diaries. the richness of data afforded by this ict-enabled method enabled the research to produce a rich narrative of how the students used the technology and provided evidence of a change in pre-service teachers’ pedagogical beliefs during the course. introduction technology use by teaching practitioners has dramatically altered the learning environment, but data that are gathered to determine its effect is often collected and collated using non-technological methods. this paper demonstrates a method for collecting and collating data in research that involves the use of computer-based technologies by the researcher and the research participants. the pragmatic method is illustrated by the acquisition of data acquired in a doctoral research study. this information and communications technology (ict)-enabled method allowed the researcher to capture a variety of qualitative data that, when combined with the * school of systems information science, department of media architecture, future university, 116-2 kamedanakano-cho, hakodate, hokkaido, t 041-8655, japan. email: michael@fun.ac.jp 52 m. vallance quantitative data, produced abundant, organized and accessible information that could be efficiently analysed and interpreted. before proceeding, it must be clear what this paper is not about. it is not about presenting alterations to the tried and tested research practice of qualitative and quantitative data collection (trochim, 2005). this proposed ict-enabled method simply utilizes technology to better serve the researcher by efficiently collecting data and reducing the time spent collating that data. through better organization, the proposed method allows for a richness in a collection of data that may not be afforded otherwise. it is acknowledged that the researcher must still undertake many of the time-consuming tasks such as transcribing communication and inputting survey responses. however, once completed, this ict-enabled method facilitates an association between textual and visual communication, which can lead to a richer interpretation by the researcher. the research study to situate the context for exemplifying the data collection and collation, a summary of the research is first provided. the doctoral research was entitled ‘the impact of synchronous inter-networked teacher training in ict integration’, and set out to facilitate new perspectives and experiences in technology-based learning. to enable this, the study investigated the development of pre-service teachers’ pedagogical beliefs in the integration of ict over a 12-week, final-year undergraduate course (towndrow & vallance, 2004) where collaborative synchronous networked tasks were developed and implemented. the contributions by the pre-service teachers to the process of the iterative task development, post-task discussions and the ongoing commentaries on a bulletin board system (bbs) provided insights into any change in their pedagogical beliefs. the relevance of research in synchronous networking for higher education and business organizations was reinforced in may 2005, when microsoft revealed its vision of the future workspace. in its white paper entitled ‘digital workstyle: the new world of work’ (microsoft, 2005), collaboration was considered a key capability of its future technology and end-users: instant messaging, telephony and conference will converge in an environment that gives information workers easy and flexible access to all of their communication tools, with a secure infrastructure that archives the entire collaborative process in one place. collaborative workspaces will be simple to set up and use with co-workers, partners and customers to create documents, work on projects and find and use relevant business information. a product such as the shared documents developed during this research may have contained a presence that was facilitated by the collaborative process. in other words, the finished document represented an artefact of a synchronous process of work by a number of collaborators. thus, this research of the synchronous networking exemplifies a collaborative workspace in practice and provided pre-service teachers with an authentic experience of collaboration that can be transferred to their in-service teaching. pedagogical beliefs regarding the integration of ict 53 however, today’s technology literate ‘net-geners’ (tapscott, 1997) still require guidance as they develop competencies using synchronous technology, implementing synchronous collaboration, and communicating in a shared space (levy, 2003). the result of the doctoral research revealed, for instance, that the participants (the pre-service teachers) developed academic competencies that were more appropriate and desirable for integrating technology than, say, skills-focused technical workshops. a factor in the development of these competencies was task design. in other words, the data revealed that awareness of good task design led to a better understanding of ict usage to support learning. in addition, the competencies became apparent during the process of collating data. they provided a context for interpretation and discussion of the iterative process of informed ict integration and associated task design. synchronous networked tasks were employed to encourage pre-service teachers to consider their existing practices and facilitate an approach to teaching and learning that supported informed integration of ict. it is acknowledged that it will be difficult to prove what is good design for synchronous networked learning (jones & asensio, 2000). however, this study was an iterative process and, as suggested by jones and asensio, ‘the products of design are not outcomes of the design itself but of a deeply social and situated set of work practices’ (2000, p. 255). supporting this, the participants were therefore involved in designing the synchronous tasks, and throughout the 12-week course developed a community that reflected, shared and explained their evolving personal beliefs and understandings of ict-enabled pedagogical practices. the iterative development and recorded implementation of the tasks, and the ensuing discussions, revealed that the beliefs of the participants emerged and altered through participation in synchronous networked activities. the participants developed a personal understanding of informed ict integration by sharing and articulating ideas, linking these ideas to their own experiences, engaging with tasks and being exposed to different learning approaches. the qualitative evidence categorized these reflections into three broad competencies: generic, epistemic and declarative. consolidation, new ideas and core skills were deemed generic competencies and linked to what goodyear (2001) calls ‘transformative potential’. demonstrating understanding, responding to different points of view and expressing own explanations represented declarative conceptual knowledge and applying such knowledge in problem-solving or arguments. epistemic fluency, where learners are required to be flexible in their use of knowledge (morrison & collins, 1996), was represented in this study by participants’ values and connections to their school environment. throughout the research, the participants began to construct and internalize what they were doing, and consider why they were doing certain activities when implementing a synchronous network-based task. this may be considered an initial stage in a process of participants changing their pedagogical beliefs. subsequent stages of participants’ development in pedagogical beliefs were the sharing of their points of view, expressing what they had internalized through discussion and critiquing the shared information. immersed in these discussions (during the task process, on the 54 m. vallance bbs or during the interviews), the participants refined their understanding of informed ict integration. in summary, at the beginning of this study many of the participants viewed their desirable teaching as a transmission of content. as this study developed, the participants were actively involved in, as well as reflecting upon, designing good learning tasks in a synchronous networking environment. the resultant learning shifted from content to activities. to implement synchronous networking tasks successfully, the participants had to become good task designers and focus upon the activities and anticipated cognitive outcomes that may lead to good learning. goodyear (2001) states that content in these ict-rich environments becomes the context, and the learning is what the students do with the content. this research revealed that the learning shifted from content to activities; that is, to what students actually did with the tasks’ content. this was borne out of a development of academic competencies facilitated by synchronous networked task design and implementation. collecting the data: an overview the methods used to collect the data needed to enable the research aims to be met. firstly, to explore quantitative and qualitative data in a natural setting; and secondly, to obtain an understanding of the process of network-based learning (how students used technology) and the product of network-based learning (the impact of use on pedagogical beliefs) within the context of training teachers to integrate ict (lortie, 1975; zhong & shen, 2002). this resulted in data being collected from a variety of sources. during the research, the pre-service teachers (hereafter referred to as participants) engaged in four synchronous tasks where each computer action and their corresponding communication (both text and spoken) were digitally recorded. immediately after each task, a selected participant subgroup was interviewed to reflect upon the collaborative experience, and discuss the activities and associated cognitive outcomes required to complete the synchronous task. at the end of each lesson, all the participants also completed a survey with 14 ‘yes/no’ activity and cognitive outcome questions. the qualitative and quantitative data were consequently analysed and an interpretation offered to the participants and the researcher’s colleagues. the interpretation was posted online within one week for all participants to clarify, comment, amend, develop and elaborate prior to the iterative design and implementation of another task two weeks hence. despite the researcher’s ‘normal’ teaching and administrative load, the immediacy of an interpretation provided a topic for discussion in the following seminars. the resultant digital method collected data from a variety of sources, and the procedure is represented in figure 1. figure 1. flowchart for data collection technology utilization was a focal point of this research but the challenge was how to collect data that would be representative of activities and cognitive outcomes (knipe & lee, 2002) in the process of the synchronous tasks. the researcher needed to collect data of the participants working on a task, the communication that facilitated the task process and completion, and an interpretation by both participants pedagogical beliefs regarding the integration of ict 55 and researcher of what occurred. the use of digital video would offer greater flexibility (smith, 1981) and allow for a more detailed retrospective analysis (edwards & westgate, 1987) than using techniques that involved live coding (bowman, 1994). a video recording would also provide the contexts for meaning of exponents before, during and after participants’ actions (bowman, 1994). thus, an ict-enabled procedure for data collection that utilized the existing technologies in the learning environment was developed; in this case, a digital classroom with 30 computers equipped with videoconference cameras, arranged on five hexagonal tables called ‘learning islands’. two alternative software solutions considered were hypertranscribe1 and transana.2 hypertranscribe and transana are designed to synchronize recorded video and a typed transcription. these are useful tools for researchers but were not adopted in this study as the researcher needed to view the data of participants working on a synchronous task and simultaneously capture their computer actions. to the best of the researcher’s knowledge, no such software exists—hence the development of this customized ict-enabled method of data collection and collation. the tasks having established that the data were collected quantitatively and qualitatively, it is timely to discuss the tasks that were utilized to facilitate this process. the tasks incorporated the use of synchronous networking technologies adhering to selwyn’s framework of activities, integration, collaboration and shared space (selwyn, 1997). for effective use of ict, the participants were provided with opportunities to cognitively engage with the materials with ‘high levels of learner control’ (steeples et al., 2000, p. 328). throughout the tasks in this research, the participants were figure 1. flowchart for data collection 56 m. vallance provided with a context and an achievable objective, but the design and process of the tasks, and thus associated learning paths, were flexible. in other words, the networked participants determined how each task was implemented. the 12-week undergraduate course that provided the context for this study did not aim to develop technical skills, but focused upon the application of ict (theory, historical perspectives, pedagogy, practice, procedures, etc.) in order to encourage and develop pre-service teachers’ understanding of informed use through an experiential learning process (towndrow & vallance, 2004). all the activities undertaken by the participants were summarized on the class bbs for sharing and asynchronous discussion. as mentioned, throughout the course there were four instances of data collection while the participants were undertaking a task specifically using synchronous networking technology. the participants’ activities were recorded digitally: what they did when using the computer (screen capture) and what they did while working (video and audio capture). immediately after completing a task, a focus group was formed in order to discuss the task implementation with the researcher. each task conformed to selwyn’s framework as illustrated in table 1. it must be noted at this juncture that tasks 2, 3 and 4 were a result of the interpretation of the data and corresponding discussions, and were thus designed in conjunction with the participants. a concern may be the limited number of synchronous tasks; however, within the 12-week undergraduate course a number of additional tasks were undertaken either supporting the synchronous collaboration or more traditional classroom activities. the next section will explain how the different components of the data were collected and collated. capturing the tasks the task implementation consisted of participants working in groups of three around a computer terminal. displayed on their monitor screen was a document that was synchronized with other participants at another location. in other words, a number of participants could access one document and type text, and that text would be immediately viewed by participants located elsewhere (synchronous document collaboration). all participants could also simultaneously text chat in the chat pane while waiting for the one group to complete their document text. this is illustrated in figure 2. the software utilized for synchronous document collaboration was istorm.3 the most technology intensive component of the research data included the collection of the participants’ communication while on task, and simultaneously recording every computer screen action as they worked on that task. figure 2. screen capture actions and communication capturing students’ spoken communication the study was conducted in a digital classroom using computers upon which were attached small, digital, apple isight cameras. one camera was fitted with a wideangle lens (named vitacon m-power) to ensure that all participants’ communication pedagogical beliefs regarding the integration of ict 57 t ab le 1 . s el w yn ’s f ra m ew o rk a n d t h e sy n ch ro n o u s ta sk s c h ar ac te ri st ic e xp la n at io n t as k 1 , re p o rt o u tl in e t as k 2 , ex p o si to ry e ss ay t as k 3 , id io m t as k 4 , n ar ra ti ve a ct iv it ie s t ea ch er s sh o u ld o ff er s tu d en ts s el fd ir ec te d l ea rn in g ac ti vi ti es t h at en co u ra ge s el fex p re ss io n t h e st u d en ts a re g iv en r o le s an d r es p o n d a cc o rd in gl y. s tu d en ts a re p ro vi d ed o p p o rt u n it ie s to e xp re ss t h ei r o p in io n s b u t in t as k 1 t h ei r co n tr ib u ti o n s ar e in fl u en ce d b y th ei r as si gn ed r o le s s tu d en ts a re p ro vi d ed w it h t h e o p p o rt u n it y to ch o o se a n e ss ay t it le fr o m a s el ec te d l is t an d th en a p ar ti cu la r ro le t o ex p re ss v ie w p o in ts . t as k 2 i s m o re f le xi b le t h an ta sk 1 s tu d en ts a re p ro vi d ed w it h a n i d io m a n d c an o ff er a n y co n te xt a n d p er so n al in te rp re ta ti o n . t as k 3 i s co n si d er ed fl ex ib le i n t er m s o f th e ac ti vi ty ’s p ro ce ss a n d p ro d u ct f le xi b le l ea rn in g en co u ra ge d . s tu d en ts a re p ro vi d ed w it h f iv e p ic tu re s an d th en d ec id e a la n gu ag e ex p o n en t fr o m t h e e n gl is h cu rr ic u lu m a n d u se i c t to p ro m o te t h ei r an d t h ei r le ar n er s’ u n iq u en es s in te gr at io n t ea ch er s ar e m o re ef fe ct iv e af te r re ce iv in g ex te n si ve tr ai n in g in t h e in te gr at io n o f te ch n o lo gy w it h th e cu rr ic u lu m in te gr at io n o f te ch n o lo gy th ro u gh f o rm al t ex t d ev el o p m en t, i n fo rm al t ex t d is cu ss io n s an d l in ki n g b o th w it h t h e re p o rt in g st ru ct u re b ei n g a ke y ge n re o f th e e n gl is h s yl la b u s u se o f in fo rm at io n te ch n o lo gy ( it ) to d ev el o p a l es so n t h at in te gr at es i t i n t h e co n st ru ct io n o f an ex p o si to ry l es so n o u tl in e. i n a d d it io n , u se o f o rg an iz at io n s o ft w ar e an d w eb s ea rc h in g st ra te gi es c o m b in e to su p p o rt s tu d en t to st u d en t co ll ab o ra ti o n t as k is c u st o m iz ed f o r st u d en ts t o o ff er in d ep en d en t o p in io n s b as ed u p o n o u to fsc h o o l ex p er ie n ce s (f o r th e co n te xt a n d in te rp re ta ti o n o f th e id io m ). i c t s u p p o rt s th e in te ra ct io n s. i n t h e co n st ru ct io n o f id io m ’s m ea n in g an d u se g ra p h ic i m ag es o ff er st u d en ts a n o p p o rt u n it y to o rd er , in te rp re t an d d ev el o p m ea n in g w it h in th e im ag es , an d a ls o t h ei r li n ka ge . m u lt im o d al it y co m b in ed w it h i c t f o r co m m u n ic at io n f ac il it at es an o p en -e n d ed t as k c o ll ab o ra ti o n s m al l gr o u p co ll ab o ra ti o n o n th e co m p u te r is es p ec ia ll y ef fe ct iv e w h en s tu d en ts h av e re ce iv ed t ra in in g in th e co ll ab o ra ti ve p ro ce ss s tu d en ts w o rk o n t ex t in r ea l ti m e w h il e d ev el o p in g id ea s in c h a t f ra m e in fo rm al ly e ac h s tu d en t is u n iq u e in t h ei r ex p er ie n ce u si n g it f o r te ac h in g. u si n g re al -t im e, r em o te te ch n o lo gy to d ev el o p a n es sa y o u tl in e n o rm al ly u n d er ta ke n in d iv id u al ly , o n p ap er , p ro vi d es a u n iq u e ex p er ie n ce i n co ll ab o ra ti o n s tu d en ts f ro m d if fe re n t sc h o o l s ec to rs a n d e ve n le ve ls ( p ri m ar y an d se co n d ar y) c an w o rk to ge th er . d if fe re n t in fo rm at io n n ee d s to b e co m b in ed fo r s u cc es sf u l co m p le ti o n o f th e ta sk e ac h g ro u p h as a d if fe re n t p ic tu re s o a ll n ee d t o co m b in e an d t h en r ein te rp re t th e im ag es t o fo rm t h e fi n al o u tc o m e 58 m. vallance t ab le 1 . c on ti n u ed c h ar ac te ri st ic e xp la n at io n t as k 1 , re p o rt o u tl in e t as k 2 , ex p o si to ry e ss ay t as k 3 , id io m t as k 4 , n ar ra ti ve s h ar ed s p ac e e xe m p la ry co m p u te ru si n g te ac h er s b en ef it fr o m a s o ci al n et w o rk o f o th er co m p u te ru si n g te ac h er s at t h ei r sc h o o l a ll r ep o rt c o n tr ib u ti o n s fr o m va ry in g ro le s b ei n g d ev el o p ed ca n b e vi ew ed b y an y st u d en t. f o ll o w -u p p ro m o te s sh ar in g o f p ro ce ss a n d o u tc o m e t ea m d ev el o p m en t o f a n es sa y o u tl in e u si n g re al ti m e co m m u n ic at io n . t h e it u se d c an f o rm th e p la tf o rm f ro m w h ic h to i n te gr at e it w it h le ar n er s. l es so n o u tp u t th en s h ar ed o n b b s f o r u se b y al l, p lu s o p p o rt u n it ie s fo r fo ll o w u p p o st -l es so n d is cu ss io n s an d re fi n em en ts p ri m ar y an d s ec o n d ar y st u d en tte ac h er s p ro vi d e in p u t as co n te xt s an d in te rp re ta ti o n s m ay d if fe r. a gr ee d u p o n id io m m ea n in g an d u se to b e u se d i n s ch o o ls . c an r ed ev el o p d u ri n g in -s er vi ce p er io d v ia b b s l at er a ll p ic tu re s ca n b e re o rd er ed i n i s to rm b y an y u se r, w h ic h th u s af fe ct s th e st o ry a n d it s d ev el o p m en t. t h e o u tc o m e ca n b e p o st ed t o a b b s f o r o th er s to o ff er d if fe re n t in te rp re ta ti o n s, p ee r re vi ew a n d u se i n c la ss pedagogical beliefs regarding the integration of ict 59 while in front of the computer monitor was digitally recorded. the microphone built into the camera was also set to its highest sensitivity and maximum quality. to record the digital video and audio of the students’ communication to the computer hard drive, a free application called quicktime broadcaster was utilized. the application was set up to record video at mpeg-4 format, 30 frames per second, with audio set to 16 bit, 44.1 khz stereo for a balance between quality and file size. the application’s ‘broadcast’ button was selected once the participants were assured to be captured within the view of the camera, and then the application was hidden from view by pressing the � (command) and h keys. upon completion of the task, the ‘broadcast’ button was again selected to terminate the recording. the resultant spoken communication movie file was thus automatically saved. capturing students’ computer text communication the next challenge was to record all instances of the participants’ computer actions. this was more problematic but the digital classroom had been equipped with an application called apple remote desktop. this allowed the teacher’s computer to access and view all other computers in the classroom. the participants’ actions could therefore figure 2. screen capture actions and communication 60 m. vallance be viewed from the remote teacher’s computer, but how could the actions be digitally recorded? to enable such recording, an application called snapz pro x4 was utilized. snapz pro x has the function of recording screen actions as a mpeg-4 video. therefore, by using the teacher’s monitor to display one group’s monitor at full screen size, snapz pro x was activated to record all screen actions throughout the duration of the task. the most reliable settings for snapz pro x were 15 frames per second viewed at 100%. a snapshot of the synchronous document collaboration in action is illustrated in figure 3. participants used the primary pane to type their collaborative document and the chat pane on the right for more informal text communication. figure 3. synchronous collaboration collating the communication data the next stage of the method was to combine the two sets of data. this was undertaken by first transcribing the spoken communication and identifying exponents representing communicative functions. for example, when observing the digitally recorded movie file and transcript of one group undertaking task 1, a number of functions were in evidence: making decisions, identifying, familiarization, turn-taking, formulating figure 3. synchronous collaboration pedagogical beliefs regarding the integration of ict 61 answers, associated suggestion and questioning. these functions, and those in tasks 2, 3 and 4, were not pre-determined by the researcher, but resulted through interpretation of the language exponents during the study. this is an important statement as this study had no predetermined expectations of what the participants would discuss nor directed the on-task discourse to focus upon specific functions. the study observed the processes and products that synchronous networking tasks facilitated, and analysed the collated data. therefore, the functions were derived from an analysis of the data rather than being imposed upon it (bowman, 1994). next, the combined video capture of the spoken communication and the transcribed text was saved as one mpeg-4 movie file. this file was then embedded within the full screen capture of the synchronous document process. finally, chapter markers were added to the final movie file in order to access each exponent seamlessly during the research analysis. combining the spoken communication and its text transcription first, each spoken sentence had to be transcribed and then manually tagged with a time stamp. this process was seamless as the spoken communication movie was already in a digital format that could be displayed on the researcher’s computer. next to this movie file was a plain text document, opened in text edit, for typing purposes. as the researcher played the movie file, the time stamp was noted and the transcription inserted. figure 4 illustrates a small portion of transcribed text with time stamps. for example, [00:00:23.00] represented [hour:minute:second:tenths of a second]. figure 4. transcription file figure 4. transcription file 62 m. vallance quicktime pro was then used to open the transcription file, resulting in it automatically being converted to a movie file with the relevant time stamps (see figure 5). figure 5. transcription file in quicktime the transcription and the previously recorded spoken communication movie files had to be the same duration, to the second, in order to combine synchronously. for files that were slightly different in duration, the end of one of the files had to be cropped. once the duration of both movie files were synchronized, the transcription file was selected (using edit—select all), then added to the spoken communication movie (edit—add to movie). the resultant file consisted of a movie-in-a-movie effect as shown in figure 6. figure 6. transcription and communication movie file the penultimate stage in the procedure was to merge this combined transcription and communication movie file to the screen capture of the recorded participants’ actions in the process of generating the collaborative document. the combined transcription and spoken communication movie file was selected (edit—select all), then added to the screen capture movie file (edit—add to movie), as shown in figure 5. transcription file in quicktime figure 6. transcription and communication movie file pedagogical beliefs regarding the integration of ict 63 figure 2. in this case, the transcription was located to one side in order to view paralinguistic features during communication. this was undertaken by opening the quicktime show movie properties window, selecting visual settings and offsetting the text track by adjusting by 320 × 430 pixels. adding chapter markers finally, chapter markers were added to the collated movie so that the researcher could quickly locate an exponent, listen and view a related example. to simplify and automate this process an applescript was made available to the researcher. entitled chapterman (available online at: http://homepage.mac.com/mvallance/filesharing.html), there are four simple commands; ● create chapter track. ● add chapter at playhead. ● rename chapter. ● delete chapter. after opening the collated movie file, a chapter track was created using chapterman. then each exponent was identified in the movie and a chapter added at the desired position. this procedure was repeated until all the exponents were identified. the movie was then saved. this completes the task data collection and collation. as shown in figure 2, at full screen size the movie file can be accessed to reveal the participants’ actions in the document collaboration while simultaneously listening to their spoken communication in the process of the task. this customized approach allowed the researcher to illustrate data in a format considered best for analysing the data. the interviews one group of participants was interviewed immediately after completing each task. again the interview was digitally captured as it took place in the vicinity of the networked computer. the isight camera and quicktime broadcaster software were used to record the interview. the procedure was similar to that described above where quicktime broadcaster was used to record the video and audio to the computer’s hard disk. again the text was transcribed as described above, converted to the mpeg-4 movie format in quicktime, and added to the interview movie file. chapter markers were added again so that the researcher could quickly synchronize examples of related exponents. the surveys at the end of each lesson, all participants completed a survey, which attempted to categorize the activities and cognitive outcomes (knipe & lee, 2002) prevalent in the synchronous document collaboration process. a discussion of the quantitative data is 64 m. vallance outside the scope of this paper but the survey helped focus the interviews on perceived learning facilitated by the process of the collaborative task design and implementation. the data were also utilized by the researcher to seek connections between the captured qualitative data and the quantitative survey data. essentially, this was an attempt to ensure trustworthiness of the collected data (guba & lincoln, 1981). the researcher’s diary the procedure described in the previous section may, at first, appear complicated. however, once the first task had been completed the remaining tasks were collated much more quickly. the most time-consuming, but necessary, procedure was the transcribing. once that had been completed, the collation was straightforward. it is believed that the key to the success of this particular study was to display an interpretation of the data as soon as possible after the task completion, using a weblog. the researcher undertook this within one week (despite having a lecturer’s teaching load). the interpretation provided content for the discussion in seminars as well as opportunities for participants to express opinions, ask questions, seek clarification and ultimately collaborate in a shared experience of what was meant by informed use of ict to support learning. moreover, to facilitate asynchronous communication between the researcher and the participants outside the designated seminars, a class bbs was set up. the postings could be read by all the participants, thereby sharing the information and facilitating a more open discussion of the task actions and the resultant learning (jonassen et al., 1999; vallance, 2005). conclusion promoters of ict provide a lot of rhetoric for the need to change education systems, yet none provide concrete proposals that are pedagogically focused (cuban, 2002) nor do they provide substantive details on how such change can be achieved (towndrow & vallance, 2004). in other words, ict adoption is gaining recognition as a necessity in education systems worldwide but ‘how’ and ‘why’ ict is adopted and utilized in an informed way remains unclear. one way to provide some clarity and analyse the effect and impact of participants using technology is to consider utilizing the tools readily available in, for example, a digital classroom. this paper has thus provided a context to consider an ict-enabled method of data collection and collation that may lead to new insights on technology utilization. notes 1. see http://www.researchware.com/ht/ 2. see http://www.transana.org 3. see http://www.mathgamehouse.com 4. see http://www.ambrosiasw.com pedagogical beliefs regarding the integration of ict 65 references bowman, m. d. (1994) using video in research, scre spotlight no. 45. available online at: http:// www.scre.ac.uk/spotlight/spotlight45.html (accessed 25 july 2006). cuban, l. (2002) oversold & underused. computers in the classroom (cambridge, harvard university press). edwards, a. d. & westgate, d. p. g. (1987) investigating classroom talk (lewes, falmer press). goodyear, p. (2001) psychological foundations for networked learning, in: c. steeples & c. jones (eds) networked learning: perspectives and issues (london, springer verlag). guba, e.g. & lincoln, y.s. (1981). effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches (san francisco, ca, jossey-bass, inc.). jonassen, d. h., peck, k. l. & wilson, b. g. (1999) learning with technology: a constructivist perspective (upper saddle river, nj, prentice-hall). jones, c. & asensio, m. (2000) designs for networked learning in higher education: a phenomenograpic investigation of practitioners’ accounts in design, in: c. steeples & c. jones (eds) networked learning: perspectives and issues (london, springer verlag). knipe, d. & lee, m. (2002) the quality of teaching and learning via videoconferencing, british journal of educational technology, 33(3), 301–311. levy, p. (2003) a methodological framework for practice-based research in networked learning, instructional science, 31, 87–109. lortie, d. c. (1975) schoolteacher (chicago, university of chicago press). microsoft (2005) digital workstyle: the new world of work. available online at: http:// www.microsoft.com/billgates/speeches/2005/05-19ceosummit.asp (accessed 19 july 2005). morrison, d. & collins, a. (1996) epistemic fluency and constructivist learning environments, in: b.g. wilson (ed.) constructivist learning environments: case studies in instructional design (englewood cliffs, nj, educational technology publications), 107–119. selwyn, n. (1997) teaching information technology to the computer shy: a theoretical approach to a practical problem, journal of vocational education and training, 49(3), 395–408. smith, h. w. (1981) strategies of social research (englewood cliffs, nj, prentice hall). steeples, c., jones, c. r. & goodyear, p. (2000) beyond e-learning: a future for networked learning, in: c. steeples & c. jones (eds) networked learning: perspectives and issues (london, springer verlag). tapscott, d. (1997) growing up digital: the rise of the net generation (new york, mcgraw-hill). towndrow, p. a. & vallance, m. (2004) using it in the language classroom: a guide for teachers and students in asia (3rd edn) (singapore, longman). trochim, w. m. (2005) the research methods knowledge base (2nd edn). available online at: http:// www.socialresearchmethods.net/kb/ (acessed 25 july 2006). vallance, m. (2005) the challenge of informed use: a unique it experience for teachers and students from the peoples’ republic of china, the journal of asia tefl, 2(1), 55–73. zhong, y. x. & shen, h. s. (2002). where is the technology induced pedagogy? snapshots from a multimedia efl classroom, british journal of educational technology, 33(1), 39–52. alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 171–172 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020171–02 © 2007 association for learning technology doi: 10.1080/09687760701482275 discussion section the tipping point gilly salmon* university of leicester, uk taylor and franciscalt_a_248104.sgm10.1080/09687760701482275alt-j research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis152000000june 2007gillysalmongilly.salmon@le.ac.uk; gks13@leicester.ac.uk how delightful that pam moule (2007) has taken the trouble to challenge the fivestage model for e-learning! i would hate to think that the model has served to cut off any discourses about e-learning at a time of huge potential for the future of higher and professional education. no one benefits by any model being taken or used uncritically or out of context or culture. the challenges we face in providing exciting, worthwhile and effective learning through whatever medium grow daily. i would be so pleased if others could respond too, so that this model, first developed more than 12 years ago in a different era but since used extensively, could continue to be further examined for relevance and helpfulness. the five-stage model is rooted in social constructivism as dr moule spotted, but more as an underlying philosophy of the world of knowledge than as an instructional or normative theoretical stance. the model was originally researched using the literature of the day and large-scale text-based asynchronous conferencing platforms (cosy and firstclass) in the first few years of the 1990s when virtual learning environments still weakly twinkled in the developers’ eyes. my study deployed cognitive mapping methodology (eden & ackermann, 1998) and resulted in the emergent grounded five-stage model. more than 1000 students and tutors in distance-taught open university business school courses were involved. at the time they were offered free rein, through the software, to interact with each other in whatever ways they chose. i would find it all but impossible, perhaps unethical, to undertake such a study now so i am glad i did so at the time, over two years. i suspect that the grounded nature of the model later resulted in its popularity as a device for understanding, since it felt ‘natural’ and educators and learners reported ‘ah ha!’ moments of recognition of their early online experiences. no prior theories of learning were ‘applied’, although they were later used to interpret and build upon constantly emerging understandings and uses. the model ‘caught on’ when the first edition of e-moderating was published in 2000 (salmon, 2000, 2004). my approach in that book was to describe one major application of it—to explain the competencies and training development of online university teachers and facilitators and all the different ways in which these can be promoted and * beyond distance research alliance, attenborough tower, university of leicester, university road, leicester le1 7rh, uk. email: gks13@leicester.ac.uk 172 g. salmon experienced. that work continues still and more recently has been combined with a wide range of established face-to-face group work (www.learningingroups.com) in the service of learning. there have been many others in different contexts who have used and written about it, not only for e-moderating training but for student and community development. for me, the five-stage model’s popularity arises because it works as a framework for use that has enabled individual academics and course teams (often those working without the benefit of instructional designers) wishing to work online productively, to customise, personalise and adapt it to many purposes, contexts, countries and technologies rather than as a constraint. it’s proved ‘blendable’ in its application to notions of communities of practice (see, for example, our use in a widescale community of expertise project: http://uk.omnium.net.au/elks/base/) and to growing notions of e-literacy among learners of different kinds. we now have the ‘blessing’ of virtual learning environments and of many more learning and knowledge technologies. there is also a range of new issues where the model is still providing a relevant backdrop, typically in conjunction with small-scale low-cost online activities (salmon, 2002). for example, at the university of leicester we are now applying it across disciplines from arts to science in campus-based and distance learning through our higher education academy (hea)-funded pathfinder project ‘adelie’ (www.le.ac.uk/adelie) as well as in web 2.0 research including blogs, wikis and podcasting (www.impala.ac.uk). we have found, for example, that the model is useful is encouraging wiki use among small professional communities and in blending podcasting into virtual learning environments. a key issue that is raised by the suggested ‘ladder’ approach in the moule article is one that we are currently researching at leicester through tony churchill’s study, and one with which i am sure many readers of alt-j are wrestling. to what extent and in what ways can we suggest, enable, encourage, promote (demand that) academic staff design for presenting and facilitating online learning? there is no evidence so far that there is an easy pathway between instructivist and constructivist approaches. many courses report learners and teachers tipping into the abyss as attempts are made to move from information provision to knowledge generation. can it be a choice, a step-by-step approach, or does it need evangelical support or lots of staff development? can we explore this? references eden, c. & ackermann, f. (1998) making strategy (london, sage). moule, p. (2007) challenging the five-stage model for e-learning: a new approach, alt-j, 15(1), 37–50. salmon, g. (2000) e-moderating: the key to teaching and learning online (1st edn) (london, kogan page). salmon, g. (2002) e-tivities: the key to active online learning (london, taylor and francis). salmon, g. (2004) e-moderating: the key to teaching and learning online (2nd edn) (london, taylor and francis). alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 101–102 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020101–02 © 2007 association for learning technology doi: 10.1080/09687760701470874 editorial the problem with representing knowledge with and about technology taylor and francis ltdcalt_a_246966.sgm10.1080/09687760701470874alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)editorial2007taylor & francis152000000june 2007martinoliverm.oliver@ioe.ac.uk the articles contained in this issue raise a series of questions that all, to a greater or lesser degree, call in to doubt our ability to communicate with each other. this is a particularly melodramatic way of expressing it, perhaps, but it serves to highlight a fundamental concern: even if we know something, can we share that knowledge with anyone else? more specifically, for researchers in this field, if we think we know something about technology, how can we communicate that understanding to others? the general problems of communication are illustrated by a series of articles from dickey, wheeler, lee et al. and by bakker et al. there are many things that teachers find hard to communicate to students in the first place, particularly those that relate to tacit practices. dickey’s study explores one way in which such practices can be communicated to students—involving an embodied representation of knowledge, communication through demonstration. however, even if we work out how to represent what we value to our students, they will gain no credit for what they learn unless they can communicate it back to us, especially in a form suitable for assessment. even formatively, it would be very hard to give feedback to a student who could not explain to a tutor what they currently understood, at least in some form or another. laurillard’s (2001) conversational framework serves to illustrate this point clearly; with one-half of the interactions missing, there could be little hope of reciprocal dialogue. so do we support them in acquiring the communicational practices we prefer, for example, by using graphic organisers to improve their academic writing, as lee et al. propose? do we open up our own communicational repertoire, as teachers, to adopt a form of communication in which many students are already fluent, such as instant messenging? (bakker et al.’s work calls into question, however, whether students would like us to do this even if we wanted to.) irrespective of what we do, can any of these approaches offer some educational advantage that makes them better than our default position of face-to-face communication? wheeler’s analysis suggests that, for some people, under certain conditions, perhaps they can. without making sweeping generalisations, such work begins to identify implications for the way we design educational experiences. tompsett calls us to practice reflexivity in our research work, and the point is particularly apt here. these communicational issues do not just exist for our students; 102 editorial they are real issues for researchers in this field too. there is considerable interest, and no small amount of funding, in developing ways to represent and share educational practice. whether these representations are expressed as case studies, instantiated as toolkits, turned into formalised design patterns, modelled as use cases, and so on, a hidden assumption is that some pedagogy, some value, can be communicated from the designer to a teacher, and perhaps further to a learner. falconer’s article raises questions about how and if this can be achieved. however we try to express our understanding, if the experiences of our audience are too remote it will be hard or even impossible to make ourselves understood. falconer argues that even closely related communities can be far enough apart that they struggle to communicate with each other. our discussion section develops this point. previously, we published pam moule’s article outlining a model intended to help tutors engage with the pedagogy of elearning. when explaining this model, she found it helpful to draw a contrast with salmon’s five-step model for e-learning. in this issue, each takes a turn to explain why their model was necessary; each ends by inviting further debate and discussion that will—they propose—allow our understanding of e-learning to develop. tompsett has also written in alt-j about our attempts to represent knowledge about e-learning, and hence we invited him to offer a third perspective on this discussion. like falconer, he suggests that even when we share intentions—even where we appear to share the same terminology—we may end up talking past each other, neither party quite communicating what they intended. this is not, he proposes, a problem that can just be solved. it is, however, a problem we can all engage with. recognising the problem is a good first step towards this. martin oliver institute of education, uk references laurillard, d. (2001) rethinking university teaching: a framework for the effective use of educational technology (london, routledge). flexible learning in a partnership context for beginning teachers brian m. sova and roger g. hacker the graduate school of education, university of western australia a preliminary inventory was carried out for a sample of male and female student teachers who had email accounts. preliminary inventory data revealed that the mean number of times per week that males used email was significantly higher than the mean number for female students. this inventory data indicated that characteristics such as gender affect internet educational technology utilization, and this factor was addressed within the experimental design for the main study. in the main study, student teachers were assigned to four treatment groups for learning lesson-planning skills. the first two groups received a traditional university lecture followed by a seminar. for group i, the seminar was provided at university. for group 2, the seminar was provided as part of the student's school-based experiences. groups 3 and 4 received identical tuition to groups 1 and 2 respectively, except that tuition was provided on a flexible learning basis, being delivered electronically via the internet with tutorial support from the university lecturer by way of asynchronous email. it was found that student achievement was significantly higher when tuition was provided as part of the student's school-based experiences, and also that achievement was significantly higher when tuition was provided on a flexible learning basis involving computermediated communications. a separate an ova was carried out for the entire sample of the four treatment groups in order to address the factor of student gender, which was controlled in the experiment. however, there were no significant differences in achievement associated with student gender. this surprising finding suggests that although male. students may make greater usage of information technology, when required to do so, female students perform on a par with their male counterparts. introduction universities have been slow to adapt to technological change. as the teaching and learning committee of the university of western australia observe in alternative modes of teaching and learning (1996, 2): the new world that is emerging values flexibility over rigidity, and process over content. yet with our complex system of faculties and departments, courses and units, curricula and assessment, we offer students little control over their own learning. our current model is predominantly didactic rather than negotiated, and we prefer to control learning resources, rather than offer them. our school system is already adapting, restructuring and adopting technology to make the curricula more relevant to individual learners' characteristics, to make learning more active, and to empower students to take 50 alt-] volume 6 number i responsibility for their own learning. students will come to expect no less from their tertiary experiences. teacher educators have been slow to realize the potential of technological change. where computer-mediated delivery has been used to educate teachers, it has been mostly in the field of continuing teacher education (cte) and some universities now offer postgraduate teacher-education programmes via flexible delivery involving off-campus, computermediated delivery. in the area of initial teacher education (ite) there has been less experimentation. selinger (1996) describes an 18-month, part-time, distance-taught pgce offered by the open university for potential primary and secondary teachers who were unable to attend a full-time course. however, this programme was delivered mainly via the traditional open university print media packages, with the use of information technology in this programme being restricted to synchronous/asynchronous email for tutorial support and text-based conferences. otherwise, there has been little reported experimentation with flexible delivery in ite programmes, and little or no work has been done which involves comparing the efficacy of flexible delivery compared with traditional university teaching. in recent years, partnership models, which involve students spending a greater proportion of their time in schools, have become fashionable with teacher educators. however, as with computer-mediated delivery, little is known about the efficacy of a partnership model when compared with the traditional alternative. the study reported here aims to explore the efficacy of these two innovations with respect to the delivery of a courseware module for ite. the null hypotheses tested were: (1) that the efficacy of computer-mediated delivery of courseware is not different from that of traditional, university delivery methods, and (2) that the efficacy of learning in partnership with schools is not significantly different from university-based learning. methods preliminary inventory methods. student teachers were randomly approached at the university of western australia and asked to complete inventory forms. the sample of student teachers included male (n=14) as well as female (n=21) students. inventory form questions included one which asked students to indicate their gender and one which asked those students with an email account to indicate the approximate number of times per week they checked them. main study sample. the sample comprised 43 student teachers of science, studying a lesson-planning courseware module as part of their diploma in education programme. science was the major teaching area for 21 students who had studied a science to at least the third year of their undergraduate programme. it was a minor teaching area for 22 of the students who had studied a science to the second year of their undergraduate programme. eighteen of the students were male and 25 female. all of the students studied the same courseware designed to develop lesson planning skills, which comprised a lecture followed by a seminar. the students were assigned to four treatment groups. the assignment was purposive with respect to the level to which science had been studied successfully in the student's undergraduate programme and with respect to student gender, as it was thought that these variables might also influence achievement. 51 brian m. sova and roger g. hacker flexible learning in a partnership context for beginning teachers general objectives: • has the topic which subsumes the lesson plan been clearly identified? • have the particular objectives of this topic which relate to the lesson been identified? • are these objectives in the syllabus? specific objectives • have the specific objectives of this lesson been identified? • are these specific objectives appropriate for the age and ability levels of the class? resource materials: • have audio-visual resource materials needed for the lesson been identified? • have science resource materials required been identified? • are quantities, and other details required by the laboratory technician provided? • are the resource materials t o be used varied and interesting? safety: have all possible hazards associated with practical work been identified? sequence and timing: • has the lesson's place within the topic sequence been indicated? • has the intended overall length of the lesson been shown? lesson content • have the activities in which students will engaged been adequately described? • has an estimate for the time that each activity is likely t o take been provided? • are these time-estimates realistic? instructional strategies: • have the teaching strategies to be employed been identified? • is the selection of strategies appropriate to objectives, content and class characteristics? • have key questions for students been identified? starting the lesson: • does the lesson plan link effectively with previous learning? • has provision been made for reviewing the main points of the previous lesson? • does the introduction clarify for students the objectives and the lesson structure? • are the concepts to be taught located within the conceptual framework of the topic? the main body of the lesson: are the selected activities at the correct level of difficulty for the class? are the selected activities varied? are objectives and procedures t o be clarified at the start of each activity? will it be clear t o pupils that one activity has concluded and that another commenced? are links between one activity and the next made clear in the lesson plan? are activities included which require the active participation of students? is there sufficient material t o keep all pupils working throughout the lesson? have contingency plans been made, should activities take longer than expected? ending the lesson: • is there provision for summarizing the main ideas that emerge during the lesson? • will the main findings of the lesson be related to the objectives of the lesson? • will the degree to which students have attained the lesson objectives be assessed? • has provision been made for homework or recommended follow-up activities? table i: the categories of the science lesson plan rating instrument 52 • alt-) volume 6 number i the first group (n=ll) received both the lecture and seminar in a traditional university setting. the second group (n=12) received the lecture at university supplemented by a school-based courseware package to be completed with the student's supervising teacher. the third group (n=10) and the fourth group (n=10) were given training in using email and an internet browser. for these groups, the courseware used by to the first two groups was delivered via the internet with electronic tutorial support provided by the university lecturer. main study instruments. an instrument called the science lesson plan rating instrument (slpri) was developed for the purpose of assessing lesson planning achievement. the categories of the slpri are shown in table 1. appraisers using the instrument are required to judge the quality of the lesson plan against a set of criteria. using a dichotomous scoring key for each of the 34 criteria, a maximum score of 34 points could be attained. main study procedure. six appraisers were selected for training in using the instrument. these six appraisers were all science-education lecturers involved in initial teacher education. after a day of intensive training in using the instrument, reliability trials were carried out. appraisers were required to assess three sample lesson plans provided; the three appraisers with the closest agreement in assessing these lesson plans were selected to participate in the study. student teachers were pre-tested with respect to their lesson-planning skills by being asked to prepare a lesson on the differences between animal and plant cells. students were then assigned to the four treatment groups and post-tested with respect to their lesson-planning skills by being asked to prepare a further lesson on the differences between animal and plant cells. to avoid experimenter-expectation effects (barber, 1973), the roles of investigator and experimenter were separated and the investigator did not participate in data collection. the lesson plans resulting from the pre-test and post-test were randomly assigned to appraisers to avoid systematic rating errors in the results. results preliminary inventory results. as shown in table 2, preliminary inventory data revealed that the mean number of times per week in which male student teachers used email accounts (5.14) was significantly higher than the mean number for female students (2.21). the inventory data supported the consideration of gender as a factor within the experimental design of the main study. main study results. students' post-test scores on the slpri were converted to percentages and subjected to a two-way analysis of co-variance, with the delivery strategy (traditional or computer-mediated) and the context (university-based or partnership) as factors, and the pre-test scores as a co-variate. the results of this analysis are shown in table 3. discussion both of the null hypotheses were rejected. students' achievement gains were higher when computer-mediated delivery strategies were employed when compared with traditional university delivery methods (see table 3). also, achievement gains were higher when 53 brian m. sovo and roger g. hacker flexible learning in a partnership context for beginning teachers source of variance between subjects within subjects total (male mean = 5.14; female df ss 1 72.043 33 547.000 34 619.042 mean = 2.21) ms 72.042 16.576 f-test 4.346 p value <.05 table 2:an analysis of variance (anova) for the number of times female and male student teachers (n=35) checked their email per week. source of variation sum of squares delivery strategy 711.1 context interactions: (two-way) explained residual total key: ns * * * * * * 977.1 58.67 2309.2 2907.422 5216.7 degrees of freedom 1 1 1 4 38 42 not significant, p>0.05 p< 0.05 p < 0 . 0 l p< 0.001 mean square 71 i.i 977.1 58.67 577.3 76.5 124.2 f ratio 9.29 12.77 0.767 7.545 level of significance ## * * * ns table 3: post-test slpr1 scores by delivery method and context with pre-test scores as co-variate students worked in a partnership arrangement with schools than in a traditional university-based environment. the finding that computer-mediated delivery within a flexible learning context is more effective than traditional university teaching for an ite courseware module is consonant with findings in other fields of study. reasons for this greater efficacy require further study. clark (1994) asserts that media as well as media attributes fail to influence learning, and that delivery media are not directly responsible for motivating learning. schutte (1997) suggests that the greater efficacy of computer-mediated communications relates to the 54 alt-] vb/ume 6 number i enhanced levels of interaction among students. the national council for educational technology (ncet, 1996) suggests that advantages of computer-mediated delivery are enhanced peer-group interaction, opportunities for self-paced work and access to advice from peers, tutors and experts via the internet on demand. mason (1994) suggests that computer-mediated delivery can improve professional development and training by enhancing tutorial support, providing a forum for interaction between teachers and learners, and a medium for mutual support for students, facilitating access to expert debate. the conclusion that achievement gains were higher when students worked in a partnership arrangement with schools, rather than in a traditional university-based environment, is significant in that we were unable to locate other quantitative empirical studies which had addressed this issue. this finding is supportive of policies which involve student teachers spending more time in schools and undertaking more of their educational studies in a school-based setting. achievement gains were 36 per cent higher for the treatment group where computermediated delivery on a flexible learning basis was employed in a partnership context, when compared with traditional delivery methods. this finding supports attwood's (1994, 11) assertion that: it is still early days in the development of formalized partnership models of teacher education, and new initiatives are yet to be fully evaluated. at present it needs special attention, mainly because of constraints on resources and staff expertise. however, all the project reports indicated that where there was adequate staff support, a suitable resource base and a shared understanding of the appropriate use of it in education, supported through course and assessment requirements, it in partnership worked well. separate anovas were carried out for the entire sample on the two factors which were controlled in the experiment, namely the year-level to which science had been studied successfully in the student's undergraduate programme, and the student's gender. the null hypothesis that the year-level to which science had been studied did not influence lessonplanning achievement was rejected (f=5.53 on 1 degree of freedom, p=0.024), and this finding confirms that subject-matter understanding is an important factor in determining lesson-planning achievement. however, there were no significant differences in achievement associated with student gender (f=0.03, p=0.86). this surprising finding suggests that although male students may make greater use of information technology as supported by the preliminary data (see also, for example, davidson, 1995), when required to do so, female students perform on a par with their male counterparts. references attwood, g. (1994), partnerships in initial teacher education: information technology for tomorrow's teachers, coventry: national council for educational technology. barber, t. x. (1973), 'pitfalls in research: nine investigator and experimenter effects' in travers, r.w. (ed), second handbook of research on teaching, new york: rand mcnally. clark, r. e. (1994), 'media will never influence learning', educational technology research and development, 42 (2) 21-9. 55 brian m. sova and roger g. hacker flexible learning in a partnership context for beginning teachers davidson, k. (1995), 'liberté, egalité, interneté', new scientist, 146, 38-42. teaching and learning committee (1996), alternative modes of teaching and learning, perth: university of western australia. mason, r. (1994), using communications media in open and flexible learning, london: kogan page. ncet (1996), teacher training and computer-mediated communications, coventry: national council for educational technology. schutte, j. g. (1997), 'virtual teaching in higher education: the new intellectual superhighway or just another traffic jam?', http:www.csun.edu/sociology/virtexp.htm selinger, m. (1996), 'beginning teachers using information technology: the open university model', journal of information technology for teacher education, 5 (3) 253-70. 56 alt-j, research in learning technology vol. 15, no. 1, march 2007, pp. 21–36 issn 0968-7769 (print)/issn 1741-1629 (online)/07/010021–16 © 2007 association for learning technology doi: 10.1080/09687760601129539 beyond model answers: learners’ perceptions of self-assessment materials in e-learning applications karen handleya* and benita coxb aoxford brookes university, uk; bimperial college london, uk taylor and francis ltdcalt_a_212885.sgm10.1080/09687760601129539alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis151000000march 2007karenhandleykhandley@brookes.ac.uk the importance of feedback as an aid to self-assessment is widely acknowledged. a common form of feedback that is used widely in e-learning is the use of model answers. however, model answers are deficient in many respects. in particular, the notion of a ‘model’ answer implies the existence of a single correct answer applicable across multiple contexts with no scope for permissible variation. this reductive assumption is rarely the case with complex problems that are supposed to test students’ higher-order learning. nevertheless, the challenge remains of how to support students as they assess their own performance using model answers and other forms of non-verificational ‘feedback’. to explore this challenge, the research investigated a management development e-learning application and investigated the effectiveness of model answers that followed problem-based questions. the research was exploratory, using semi-structured interviews with 29 adult learners employed in a global organisation. given interviewees’ generally negative perceptions of the model-answers, they were asked to describe their ideal form of self-assessment materials, and to evaluate nine alternative designs. the results suggest that, as support for higher-order learning, self-assessment materials that merely present an idealised model answer are inadequate. as alternatives, learners preferred materials that helped them understand what behaviours to avoid (and not just ‘do’), how to think through the problem (i.e. critical thinking skills), and the key issues that provide a framework for thinking. these findings have broader relevance within higher education, particularly in postgraduate programmes for business students where the importance of prior business experience is emphasised and the profile of students is similar to that of the participants in this research. introduction formative feedback is a vital part of learners’ efforts to apply and practice the principles that they learn: as laurillard (1993, p. 61) has said, ‘action without feedback is completely unproductive for a learner’. feedback is usually interpreted as material that assesses, verifies and comments on the learner’s response to assignment questions. in * business school, oxford brookes university, wheatley, oxford, ox33 1hx, uk. email: khandley@ brookes.ac.uk 22 k. handley and b. cox some educational settings, however, the possibilities for giving verificational feedback, followed by adaptive comments on errors or areas for improvement, is limited. this is the case both in self-paced e-learning (traditionally known as computer-aided instruction [cai]) and also occasionally in large-scale lectures where lecturers may resort to rhetorical question-and-answer sessions addressed to the entire audience. in e-learning, the desire to provide verification has tended to produce one of two approaches: either a proliferation of multiple choice questions (mcqs) amenable to computer-aided assessment; or the transfer of ‘complex’ forms of assessment to electronic discussion forums where academic staff can respond to individual postings with the benefit of their expertise and understanding. increasingly, these approaches are combined as newer pedagogic technologies become integrated with the old, and approaches are blended to balance the advantages of one with the disadvantages of others. furthermore, recent technologies such as blogging and wikis have expanded the repertoire of communication tools to promote student reflection and knowledge integration (for example, downes, 2004; fountain, 2005). nevertheless, there continues to be a reliance on mcq quizzes (despite pedagogic limitations) and on staff-moderated discussion forums (despite resource constraints). with respect to mcqs and similar types of binary exercises, schank and cleary (1995) have derided the over-use of this method of assessment because it risks ‘dumbing down’ students’ learning experience. the reliability and validity of mcqs have also been questioned (davies, 2002; conole & warburton, 2005). validity is a particular problem when assessing students’ higher-order learning that has traditionally relied on essays or short-answer questions (rowntree, 1987). however, the limitations of artificial intelligence technologies such as natural language processing means that computational assessment of free-text continues to be problematic (gardner, 1987). the second approach, which we have broadly categorised as the use of discussion forums, is not without its own problems. the principal one concerns resourcing: staff (and indeed students themselves; moore, 2002) often feel they do not have the time to respond carefully and individually to each posting because of other demands on their time. this means that students may be left without feedback—a ‘black hole experience’ in cyberspace (suler, 1997). the twin problems of limited staff resources to moderate discussion forums, and the deficiencies of mcqs and artificial intelligence technologies, continue to create challenges for staff engaged in designing materials for students. a conventional response to these challenges has been to offer students ‘model answers’ to enable them to self-assess their own work (for example, laurillard, 1998) rather than give adaptive feedback that provides an external assessment. the proposed benefits of self-assessment are that students are encouraged and empowered to develop their own critical faculties rather than rely on the authoritative judgement of external sources such as academic staff (boud, 1995). however, model answers are deficient in many ways, particular in their presumption of a single correct answer regardless of context, aims, personal characteristics, and so on. this reductive assumption is rarely the case with complex problems designed to test students’ higher-order learning. learners’ perceptions of self-assessment materials 23 nevertheless the challenge remains of how to develop students if the conventional form of verificational, adaptive feedback cannot be computer generated. to address this challenge, the research presented in this article investigates the effectiveness (and deficiencies) of model-answer feedback, and identifies alternative materials that can facilitate students’ self-assessment. the research was exploratory, using semi-structured interviews with 29 adult learners who used a seven-hour cai application to help develop their project management skills. the application included tutorials and a number of activities, which combined open questions with model answers as ‘feedback’. the model answers did not provide verification in the form of ‘correct’ or ‘incorrect’. dissatisfaction with model answers quickly became apparent during early interviews, and so the research re-focused its efforts on two questions: what made feedback ‘useful’ (whether verificational or not), and what alternatives to model-answer feedback could be provided. literature review the importance of formative feedback is widely acknowledged as a critical input to the process of learning (butler & winne, 1995; askew, 2000; laurillard, 2002; black & wiliam, 2003; taras, 2003; roos & hamilton, 2005). however, educational theory differs in its articulation of the function of adaptive feedback and self-assessment materials such as model answers. in her review of contemporary feedback paradigms, askew (2000, pp. 3–15) distinguishes between three paradigms: receptive-transmission, constructivist, and co-constructivist. in the receptive-transmission paradigm, the ‘expert’ gives corrective or explanatory information to students to help them learn ‘the truth’. in the constructivist paradigm, experts engage in an ‘expanded discourse’ to help students gain new understandings, but cannot dictate what those understandings will be since students are influenced by a number of sources. in the co-constructivist paradigm, the dichotomy of expert and novice dissipates and there is recognition that the ‘teacher’ also learns from the ‘student’ through dialogue and participation in shared practices (lave & wenger, 1991). in an older literature on ‘programmed instruction’ inspired by behaviourists such as skinner (1958) and thorndike (1932), feedback was viewed as a reinforcement mechanism: either rewarding desirable behaviour, or ‘punishing’ or withdrawing rewards if the student made errors. emphasis was given to behaviour that could be observed and possibly shaped through a schedule of feedback interventions. the cognitive science revolution of the 1970s and early 1980s firmly swept aside the behaviourist paradigm and refocused attention on the mind and its purported parallels with computational and symbolic processes. inspired by these new ideas, the e-learning literature on intelligent tutoring systems and cai became firmly grounded in the receptive-transmission feedback theories (for example, wenger, 1987). as a consequence, intelligent tutoring systems and cai designers tended—and to some extent still do—to espouse the mcq and its binary equivalents because they could be used to ‘diagnose’ and then ‘correct’ students’ faulty knowledge. this assumption is embedded in seminal feedback models such as that of kulhavy and stock (1989) and 24 k. handley and b. cox bangert-drowns et al. (1991). the problem with ‘model’ answers as a form of selfassessment is that they, too, imply a receptive-transmission pedagogy that is considerably out of favour in current academic debates. in the context of self-paced learning, a more appropriate way to conceptualise elearning feedback and self-assessment materials may therefore be one that acknowledges the role of self-regulated learning, and that then investigates the type of self-assessment material that facilitate that process. in this regard, a classic model of self-regulated learning that can be applied to e-learning is that of butler and winne (1995). they argue that although learners tend to seek out external sources of feedback (e.g. tutor comments), the most effective students also generate internal feedback by monitoring their performance against self-generated or given criteria. it is in this connection that self-assessment becomes particularly important. the pedagogic benefits of self-assessment have been widely discussed. brown and dove (1991, p. 59), for example, cite the advantages of student motivation, exchange of ideas, transferable personal skills and the development of a community of learning. however, self-assessment comes with its own cautions and challenges; for example, it can be demanding and time-consuming, and students may lack the capability to understand and apply assessment criteria (brown & dove, 1991, p. 61). self-assessment ‘in its more rigorous form’ involves learners not only in evaluating their work, but also in identifying and specifying the criteria and standards which should be applied to it (race, 1991, p. 7). here we see the developmental potential of self-assessment methods, which is ‘more transformational, elusive and confronting to conventional teaching than it is normally expedient to recognise’ (boud, 1995, p. 1). this is because the process of identifying criteria—especially if these are debated openly with tutors—acknowledges the socio-constructivist process of scaffolding student learning to broadly fit the norms of the educational community to which they belong (see also vygotsky, 1978). the experience of many students using stand-alone cai, webct quizzes, and so on (albeit as one among many elements of learning and teaching) is that the predominant forms of activity are the closed question plus mcq, or the open question plus model answer. however, given the deficiencies of mcqs and model answers, which have been discussed above, the need for alternative ways to facilitate student selfassessment is particularly important. research study aims and objectives in view of the relative lack of prior empirical research on alternatives to model answers following students’ short-answer responses, this research aimed to investigate learners’ perspectives on what constitutes effective non-verificational e-feedback. to address this aim, three research objectives were identified: 1. obtain learners’ perceptions of examples of model answer e-feedback taken from an e-learning application they have recently completed. learners’ perceptions of self-assessment materials 25 2. solicit their views on what constitutes ‘useful’ e-feedback. 3. solicit their perceptions and evaluations of alternatives to model-answer feedback. research methods the research adopted an exploratory approach relying mainly on semi-structured interviews, asking learners about specific examples of feedback as well as hypothetical examples. the research participants were selected from a global consulting organisation, ‘profco’. profco was selected because it produced high-quality, innovative cdrom applications for its employees. it was not feasible (nor an intention) to make statistical generalisations covering the whole population in profco, because interviews were in depth and were with a smaller (n = 29) number of people than would be the case in large-scale hypotheticodeductive research. nevertheless, the risk of systematic bias was reduced by adopting a time-based strategy (sapsford, 1999) whereby all learners based in the uk and us offices who had completed zentoria within three weeks prior to requesting an interview were selected for interview. the 29 interviewees were then stratified according to their level of expertise, because prior research (chi et al., 1988) indicates that expertise is an important mediator of learner expectations and therefore also of their evaluations of educational materials. level of expertise was attributed to learners using the proxy of their managerial ‘grade’, where one represented ‘new entrant’ and seven represented ‘partner’. new entrants were typically in their early 20s while consultants were often in the early 30s. thus the profile of younger research participants resembles that of postgraduate masters-level programmes (e.g. m.sc. in management), while the profile of consultants resembles that of executive masters and short-course programmes, which are a growing area of higher education. a mixed methodology was used, and methods were selected for their appropriateness to each of the three objectives. given the small sample size, no statistical generalisations have been made; nevertheless, we present some numerical data for informational purposes, recognising that the data are indicative only. obtaining learners’ perceptions of specific examples of model answer e-feedback the first objective was to ascertain learners’ perceptions of ‘real’ model-answer feedback, based on their prior experience of an authentic e-learning application. the application selected for this purpose was zentoria, which was designed specifically for the participating organisation with the aim of developing learners’ project management skills. the storyline running through zentoria was of a project team of management consultants helping their client to develop an environmentally friendly car. the application was created using macromedia director and adobe photoshop by a team of information technology developers and educational designers. the seven-hour course could be taken at the learner’s own pace, and was structured around two elements: tutorials, which provided a grounding in project management principles; and activities (mcqs and open questions), which portrayed problems typical of project 26 k. handley and b. cox management. the application included a ‘scratchpad’—a notebook facility in which learners could type their short-answer responses to open questions. on finishing their responses, learners were presented with model answers that varied in several ways. the 29 interviewees were asked about their experiences of completing three openquestion activities. these activities (scenario > task > type of ‘feedback’) are summarised in table 1. semi-structured interviews were conducted to elicit interviewees’ views on the activities, and particularly on the model-answer e-feedback. interviews were analysed using ‘meaning interpretation’ and ‘meaning categorisation’ approaches (kvale, 1996), as well as open-coding, constant comparative methods, deviant case analysis, tabulations and a modified form of inter-rater cross-checks. analysis was facilitated through the use of microsoft excel, and nvivo.1 the qualitative data analysis package nvivo was used to collate and then analyse comments from interviewees. analysis was facilitated by developing a ‘coding template’ to categorise interviewees’ comments according to theme. some codes were identified from the theoretical literature (e.g. the importance of ‘student engagement’), but most were developed inductively by reading and re-reading the transcripts, or by listening to the audio-tapes. selected extracts were discussed with two colleagues engaged in similar research in order to clarify and improve consistency of interpretations. this strategy provided a balance between seeking full inter-rate reliability (which is difficult if not undesirable in qualitative research because of the ‘tyranny of the lowest common denominator’; kvale, 1996, p. 181), and reliance on one person’s subjective interpretations. microsoft excel was used to analyse numerical data. this was done in the final piece of research where learners were asked to comment on nine self-assessment designs. comments were transformed into a rating scale of −2 to +2 according to the strength and direction (favourable or not towards the design) of those comments (boyatzis, 1998). soliciting learners’ views on what constitutes ‘useful’ e-feedback as will be discussed later, many learners expressed dissatisfaction with the model answer e-feedback in zentoria. from the perspective of this research, a different table 1. summary of activities used to ascertain learners’ perceptions of model answers in zentoria activity reference scenario task feedback zen1 unauthorised extension to project scope open question model answer zen2 loss of confidence by project director in the calibre of the project team open question, accompanied by a finished example of a similar open question model answer zen3 preparation for the first meeting of a new project team open question presentation of some of the issues to consider learners’ perceptions of self-assessment materials 27 interviewing strategy was required if alternative feedback designs were to be identified. therefore the interview schedule was supplemented with new questions, asking interviewees what made feedback ‘useful’, and what they perceived was generally ‘missing’ from the zentoria feedback. these qualitative data were analysed using the methods described above. soliciting learners’ evaluations of alternative non-verificational e-feedback designs the third research objective was developed once it became apparent that learners were dissatisfied with model answers. in addition to asking for their general opinions about ‘useful’ feedback, the remaining interviewees were presented with further material in the form of descriptions of nine hypothetical feedback designs that could follow short-answer open questions (table 2). the designs were presented to interviewees for evaluation and comment. qualitative responses were transformed into a rating scale of −2 to +2 as described above. where no comments were made, a zero ranking was recorded. results learners’ perceptions of specific examples of model answer e-feedback when asked about the model-answer feedback, many learners recalled their disappointment about the lack of assessment and verification. most, however, also recognised that automated verificational feedback on free-text answers was an unrealistic expectation. over and above these generally shared sentiments, learners varied somewhat in the way they responded to the model answers: as a new piece of instruction, as reinforcement for existing knowledge, or as a prompt to ‘tune’ existing mental schemata. table 2. nine hypothetical non-verificational feedback designs evaluated by interviewees short-hand name description given to interviewees ma model answer kd&d key dos and don’ts exmples several real-life examples, each describing a project situation, the intervention and the outcome bp description of the formal, best practice intervention issues explanation of the underlying issues behind the problem exp. thnk proc description of how an expert would think through the problem consequences video showing the consequences of several problem interventions exp. p&c video showing an expert comparing the pros and cons of several problem interventions peer debate video showing peers in an informal setting debating the merits of several problem interventions 28 k. handley and b. cox e-feedback as a new piece of instruction. many learners stopped typing free-text answers when they realised that the e-feedback in zentoria was non-verificational. instead, they read the e-feedback as a new piece of instructional text. e-feedback as reinforcement. for some interviewees, non-verificational feedback was useful in re-enforcing their views. for example: … it helped to crystallise some ideas that weren’t fully formed in my mind. (z8) e-feedback as a prompt for reflection and self-assessment. for a few interviewees, and mainly those already experienced in project management, the model answers acted as a prompt for further reflection and self-assessment. by thinking through the solutions and explanations given in the model answer, learners felt that they ‘tuned’ and improved their existing knowledge schemata (rumelhart & norman, 1978). the feedback helped them to achieve a sense of closure to the questions raised in their minds by doing the task, while at the same time validating and modifying their current ways of thinking about the type of problem represented. however, only a few learners experienced this sense of reflection and closure. an important differentiating factor seemed to be their tolerance of the ambiguity of the feedback. some interviewees were tolerant of feedback that did not verify their own answers. these learners seemed satisfied with feedback that offered ‘food for thought’—new ideas, perspectives and possibilities that allowed them to assess the feasibility and applicability of their own ‘solution’ to the problem task and that added richness to their existing ways of thinking about the problem. although the model answers gave no ‘feed back’ in the true cybernetic sense, they did promote self-generated learning. learners’ views about what constitutes useful feedback having asked learners about three specific instances of model-answer feedback, they were asked two supplementary questions and invited to talk about their generic expectations of e-learning feedback following mcqs or short-answer questions. the first question was ‘what makes feedback useful?’ this was asked in recognition of the vital importance of understanding student expectations of e-learning feedback; indeed, alavi and gallupe (2003) argue that the management of student expectations is a key principle of technology-mediated learning. in response to the question, interviewees commented mainly on the instructional function of feedback—what they wanted it to ‘do’. the responses were analysed and grouped into six categories, which are presented in table 3 using the terminology of instructional design, enriched by interviewees’ own comments. the prevalence of each category of comment is given in the final column. many interviewees cited several functions of e-feedback, and thus the total for comments exceeds the number of interviewees. comments indicate that e-feedback learners’ perceptions of self-assessment materials 29 was more frequently perceived as useful for verifying and reinforcing existing knowledge or explaining concepts, rather than facilitating more radical knowledge (re)construction by offering alternative insights, reflections and perspectives. however, a different picture emerged when interviewees were asked what was ‘missing’ from the model-answer feedback they received in zentoria, and what they would have gained from a (human) tutor. a wide range of responses were given, including ‘what-if scenarios’, ‘behaviours to avoid’, ‘sense of realism’, ‘alternative perspectives’ and ‘opportunity for debate’. an analysis of results that address the first two objectives of the research study suggests that, overall, learners held two key expectations of e-feedback, which at first sight seem contradictory: firstly, the expectation for more verificational feedback (implying a ‘right’ answer); and secondly, the expectation for more discursive and relativistic feedback. the latter expectation was indicated by comments recommending a move away from a theoretically ‘model’ answer, towards feedback that exemplified different ways of applying theory to practice. interviewees’ desire for alternative perspectives and ‘what-ifs’ suggested an expectation for less dualistic, causal-explanatory feedback. by comparison, the primary expectation—for verificational feedback—seemed somewhat of a paradox. this apparent paradox becomes understandable if one considers the mediating variable of ‘experience’; for example, that less experienced students seemed most likely to want to know the behaviours they should avoid. table 3. illustrations of interviewees’ expectations of ‘useful’ feedback instructional function of feedback illustration of interviewees’ expectations cited by interviewees (n) verify ‘… confirms what you in your own mind mentally have come up with’ (z6) 34 reinforce ‘[feedback] is very useful for confirming your ideas, and helping you -, you know if you think of something on your own and then it gets confirmed, it helps you remember that point’ (z7) 26 correct errors, or state correct behaviours ‘… straight answers … like “do this in this situation”, for example. in the real world of course it may not turn out that way but maybe [it’d be nice to have] a classic case like “this is what you would do”. that’d be nice so that we don’t have to figure it out on our own’ (z17) 21 inform ‘… contains all the points. it’s more of a checklist’ (z18) 10 explain ‘… if it’s possible to get the “why” in there. if you’re not right, are you on the right track? or … or “gee that’s a common mistake, but …”’ (z20) 31 promote reflection ‘… not rehash the things that are basic, but really to give a more stretching analysis’ (z19) 14 30 k. handley and b. cox learners’ evaluations of alternative non-verificational feedback designs to probe the apparent contradictions in the results, 26 of the 29 interviewees were asked to comment on nine hypothetical non-verificational e-feedback designs that were developed following analysis of early interviews. interviewees were asked how effective the designs would be as a follow-on from learners’ short-answer responses to open questions. the nine designs are described in table 2. interviewees’ extrapolated ratings are summarised in figure 1, which show that, overall, the highest average rating was given to materials giving ‘several real-life examples’. the lowest average rating was given to ‘model answer’. given the exploratory nature of this part of the research, numerical analysis does not generate statistical generalisations; nevertheless, we provide some quantitative indicators where this may help the reader gauge the range of responses (table 4). we also present a summary of interviewees’ comments for each design. figure 1. averaged and ranked interviewee ratings for hypothetical non-verificational feedback designs ( n = 26) ‘real-life examples’ had the highest overall ranking of the nine designs, with learners preferring ‘real-life examples’ over feedback, which was ‘too textbook-like’ (z22). learners wanted ‘authenticity’ (see also leung, 2003). however, some interviewees (e.g. z26) tempered their positive comments with qualifying remarks. with the real-life examples, they’re kind of a fairy tale and it’s oftentimes hard to give enough background information … to present a complex issue in the time available. (z26) an ‘expert’s process of thinking through the problem’ was ranked highly by most interviewees, but was ranked only sixth by grade-two junior consultants. this result is not surprising if one considers that an emphasis on process and on reasoning skills is exmples several real-life examples exp proc description of how an expert would think through the problem kd&d key do's and don'ts cons video: consequences of several problem interventions issues explanation of the underlying issues behind the problem exp p&c video: expert comparing the pros and cons of several problem interventions bp description of the formal, best practice intervention peers video: peers debating the merits of several problem interventions model answer 0.0 0.5 1.0 1.5 2.0 e xm p ls e x p p ro c k d a d c o n s is s u e s e xp p & c b p p e e rs m a non-verificational feedback designs a v e ra g e r a ti n g ma figure 1. averaged and ranked interviewee ratings for hypothetical non-verificational feedback designs (n = 26) learners’ perceptions of self-assessment materials 31 embedded in the ethos of many management consultancy firms. many firms subscribe to what schein (1987) referred to as ‘process consulting’: they apply standard methodologies to evaluate the effectiveness of processes in client operations, and generally provide solutions that impact at the level of those client processes. findings indicate that experts were admired because they were credited with understanding these processes. however, inexperienced interviewees, perhaps because they had not yet been socialised into the organisation and had not yet adopted a processual approach to problems, ranked the ‘expert-thinking process’ feedback only sixth. ‘key dos and don’ts’ were perceived as useful by experienced as well as inexperienced learners. an advantage of this type of feedback is that it explicitly warns of behaviours to avoid. the key dos and don’ts would be very good because then you know what is wrong, and those types of things stay with you. (z23, emphasis in original speech) ‘consequences’ feedback was highly valued by the more experienced interviewees (grade five, ranked third; grade four, ranked third) and modestly valued by the less experienced (grade three, ranked fifth; grade two, ranked fourth). this type of feedback seemed to fulfil two main functions. the first was to promote social learning (bandura, 1986) through vicarious experience. there was also recognition that, unless well done, consequence feedback could be ‘cheesy’ (z7) (if using video) or table 4. description, average ratings and overall ranking of nine non-verificational feedback designs reference description average rating between –2 and +2 (total sample n = 26) overall ranking exmpls several real-life examples, each describing a project situation, the intervention and the outcome 1.24 first exp proc description of how an expert would think through the problem 1.19 second kd&d key dos and don’ts 1.00 third cons video showing the consequences of several possible interventions 0.88 fourth issues explanation of the underlying issues behind the problem 0.65 fifth exp p&c video showing an expert comparing the pros and cons of several possible interventions 0.62 sixth bp description of the formal, best practice intervention 0.46 seventh peers video showing peers in an informal setting debating the merits of several possible interventions 0.29 eighth ma model answer 0.26 ninth 32 k. handley and b. cox ‘something to laugh about’ (z15). several interviewees suggested that the value of consequence feedback would be enhanced if supplemented by comments from an expert. ‘issues’ underlying the given problem, provided by an expert, was ranked fifth overall. for inexperienced learners it was ranked last, for example, because they saw ‘issues’ as being too dependent on the circumstances of the specific problem. interviewees with more knowledge and experience were more accommodating, and saw issue-based feedback as a useful prompt for reflection and for understanding the rationale behind decisions. it is possible that interviewees who were positive about ‘issues’ feedback recognised that issues tend to be repeated across multiple problems. they become a way of indexing experiences of problems and their solutions, and of recalling them when analogous problems arise. issues-based thinking is one example of a more structured, ‘critical’ method of thinking that students develop during their time at university. ‘experts discussing pros and cons’ was ranked low by most interviewees. this was surprising if compared with the other ‘expert’ feedback, which related to the process of solving problems. this discrepancy was possibly because although experts were credited with using effective processes for thinking through problems, the culmination of their thinking process—the solution—was not credited with being necessarily correct. ‘best practice’ scored a low average rating, for reasons similar to those given for the model answer. for some interviewees, however, best practice feedback was welcomed precisely because it established the organisation’s ‘policy’ against which actual practice could be benchmarked. for some interviewees, ‘peer debate’ was perceived as useful because peers would advocate new ideas and perspectives that could be reflected on. however, several interviewees suggested that peer-debate feedback would create confusion unless ‘controlled or even scripted to be sure to get the appropriate points across’. this suggests that interviewees were uncomfortable having to deal with divergent opinions. ‘model-answer feedback’ was ranked the lowest of the nine designs. interviewees implied that model answers were inappropriate for complex questions or topics: ‘getting the perfect answer is not always great, because it’s not a perfect world’ (z16). discussion analysis shows that this group of learners were generally unsatisfied with model answers as a response to their short-answer responses, and that only the more reflective learners used them for self-assessment. as alternatives to model answers, interviewees gave a number of suggestions about the sort of responses they would prefer in the context of a stand-alone element of a course; for example, learning what to ‘avoid’ and not just what to ‘do’; learning about process (e.g. how to think through a problem) and not just action (e.g. what to do in a specific situation); and learning by hearing of other people’s experiences and not just from theory. while the importance of these situated aspects are recognised in the educational literature, the principles are rarely applied to the design of e-feedback in cai applications. learners’ perceptions of self-assessment materials 33 what this group of learners seemed to desire at the end of each exercise was a sense of ‘closure’. a definitive single, correct answer was not necessarily required (see also kruglanski, 1989) nor deemed appropriate if learners recognised the complex and situated nature of the problem. in those circumstances, a programmed response that addressed the task in terms of relevant issues, alternative perspectives, or the process of resolving the problem were sometimes considered more valuable because they prompted reflection leading to a deeper understanding of the problem. however, the number of learners who adopted this perspective was relatively few. those who did tended to have sufficient expertise and self-confidence to rely on their own judgements about whether their response was ‘right’. they would then reflect on what else they could learn from the model-answer feedback. inexperienced learners, on the other hand, wanted first and foremost to know whether they were ‘right’, because they did not have the judgemental maturity with which to make a self-assessment. here we see parallels with the work of miller and parlett (1974) drawing on the research by perry (1970) on the intellectual development of students. these authors identified student transitions from a ‘cue-deaf’ (dualist) orientation in which they believed there to be a body of ‘right’ knowledge to be accumulated in order to pass examinations, through ‘cue-consciousness’ (multiplicitous/relativist) and leading (for some) to a ‘cue-seeking’ (committed) orientation. zentoria students who were cue-seeking recognised the relativity of knowledge and of the tasks they were given to solve, but were also secure in their own values and self-knowledge, which meant they sometimes disagreed with a solution imposed in a model answer. indeed, one could argue that the advanced students were not only questioning the validity of the model answers, but were also questioning the appropriateness of the questions themselves. in other words, their suggestions implied a rejection of questions of the type ‘what would you do?’ and their replacement by more nuanced questions such as ‘how would you think through the options?’ or ‘what are the key issues?’. going forward, this indicates a possible progression from involving students in self-assessment via criteria development to their involvement via question development. this is an important area for further research. bearing in mind the range of student experiences, there may be some benefit in developing a multi-layered exercise that seeks to promote reflection in those students who are ready for it, while also providing a verificational element for those needing the security of benchmark answers. the proposed exercise is depicted in figure 2 and includes a divergent-reflective cycle as well as a convergent-verificational cycle, leading, ideally, to the students’ sense of closure. figure 2. proposed structure for divergent:convergent activity another alterative would be to exploit the newer technologies that facilitate conversation between students as they collaboratively debate the assignment questions and devise a repertoire of solutions for them. one possibility is the use of wiki technologies that enable the co-production and co-editing of a single text (fountain, 2005). the limitation, however, is that wikis may inhibit inexperienced or shy students who are struggling to develop their own voice and authority (barton, 2004). an alternative, therefore, would be the use of blogging technologies to encourage individuals to engage in conversations with each other while maintaining their sense of identity (for 34 k. handley and b. cox example, downes, 2004). these are important areas for future development and research. conclusion this research has investigated learners’ perceptions of model-answer e-feedback, including their evaluation of alternatives. alternatives to the conventional modelanswer format need to be developed if e-learning is to move away from an over-reliance on mcqs and model answers as the standard method of assessment, particularly where students are expected to meet higher-order learning objectives. using semi-structured interviews, the research found that learners are generally dissatisfied with model-answer e-feedback for a variety of reasons, which vary according to their level of experience and prior knowledge in the subject taught. for complex topics requiring judgement and experience, many of the learners interviewed felt that model answers were inappropriate because they failed to indicate the nuances of the given problem. instead, alternative non-verificational feedback designs were considered more effective. these included ‘examples from real-life’, and ‘issues’ to help students structure their thinking of the problem. the findings of this research have broader relevance within higher education, particularly in postgraduate programmes for business students where the importance of prior business experience is emphasised (mintzberg, 2004), and where the profile of students is similar to that of the participants in this research. these students need to be supported in their efforts to build on existing knowledge and experiences, and to reconsider what they perceive to be the relevant conceptual and practical issues. the topic of project management, which was the basis of the application used for this research, is an ideal vehicle for achieving this because it encourages an integrative approach to business studies that combines theory and practice. task (open question) feedback (non-verificational) learner’s outcome (sense of closure) task (multiple-choice question) feedback (verificational) 3. learner’s sense of inquiry prompting convergent thinking 2. learner’s interaction with feedback 1. learner’s sense of inquiry prompting divergent thinking 4. learner’s interaction with feedback figure 2. proposed structure for divergent:convergent activity learners’ perceptions of self-assessment materials 35 note 1. nvivo is a qualitative data analysis package owned by qsr international (http:// www.qsrinternational.com/). references alavi, m. & gallupe, r. (2003) using information technology in learning: case studies in business and management education programs, academy of management learning and education, 2(2), 139–153. askew, s. (ed.) (2000) feedback for learning (london, routledge). bandura, a. (1986) social foundations of thought and action: a social cognitive theory (englewood cliffs, nj, prentice-hall). bangert-drowns, r. l., kulik, c. l., kulik, j. a. & morgan, m. (1991) the instructional effect of feedback in test-like events, review of educational research, 61(2), 213–237. barton, m. (2004) the future of rational–critical debate in online public spheres. available online at: http://collegewriting.us/barton/shared%20documents/revisedarticle.doc (accessed 17 july 2006) black, p. & wiliam, d. (2003) ‘in praise of educational research’: formative assessment, british educational research journal, 29(5), 623–637 boud, d. (1995) enhancing learning through self assessment (london, kogan page). boyatzis, r. e. (1998) , transforming qualitative data: thematic analysis and code development (thousand oaks, ca, sage publications). brown, s. & dove, p. (1991) opening mouths to change feet: some views on self and peer assessment, in: s. brown & p. dove (eds) self and peer assessment, standing conference on educational development, birmingham, sced paper 63. butler, l. b. & winne, p. h. (1995) feedback and self-regulated learning: a theoretical synthesis, review of educational research, 65(3), 245–281. chi, m. t. h., glaser, r. & farr, m. j. (eds) (1988) the nature of expertise (hillsdale, nj, lawrence erlbaum associates). conole, g. & warburton, b. (2005) a review of computer-assisted assessment, alt-j research in learning technology, 13(1), 17–31. davies, p. (2002) there’s no confidence in multiple-choice testing, in: proceedings of the 6th international caa conference, loughborough university, 4–5 july, pp. 117–130. available online at: http://www.caaconference.com/pastconferences/2002/proceedings/davies_p1.pdf (accessed 11 august 2006). downes, s. (2004) educational blogging, educause review, 39(5), 14–26. fountain, r. (2005) wiki pedagogy. available online at: http://www.profetic.org:16080/dossiers/ rubrique.php3?id_rubrique=110 (accessed 20 july 2006). gardner, h. (1987) epilogue, in: the mind’s new science: a history of the cognitive revolution (new york, basicbooks) (original text published 1985). kruglanski, a. w. (1989) lay epistemics and human knowledge: cognitive and motivational bases (new york, plenum press). kulhavy, r. w. & stock, w. a. (1989) feedback in written instruction—the place of response certitude, educational psychology review, 1(4), 279–308. kvale, s. (1996) interviews: an introduction to qualitative research interviewing (thousand oaks, ca, sage publications). laurillard, d. (1993) rethinking university teaching: a framework for the effective use of educational technology (1st edn) (new york, routledge). laurillard, d. (1998) multimedia and the learner’s experience of narrative, computers & education, 31, 229–242. 36 k. handley and b. cox laurillard, d. (2002) rethinking university teaching: a framework for the effective use of educational technology (2nd edn) (new york, routledge). lave, j. & wenger, e. (1991) situated learning: legitimate peripheral participation (cambridge, cambridge university press). leung, a. c. k. (2003) contextual issues in the construction of computer-based learning programs, journal of computer assisted learning, 19, 501–516. miller, c. m. l. & parlett, m. (1974) up to the mark: a study of the examination game, monograph 21 (guildford, society for research in higher education). mintzberg, h. (2004) managers not mbas: a hard look at the soft practice of managing and management development (london, ft/prentice hall). moore, m. g. (2002) what does research say about the learners using computer-mediated communication in distance learning?, the american journal of distance education, 16(2), 61–64. perry, w. g. (1970) forms of intellectual and ethical development in the college years (new york, holt, rinehart & winston). race, p. (1991) learning through assessment, in: s. brown & p. dove (eds) self and peer assessment, standing conference on educational development, birmingham, sced paper 63. roos, b. & hamilton, d. (2005) formative assessment: a cybernetic viewpoint, assessment in education: principles, policy and practice, 12(1), 7–20. rowntree, d. (1987) assessing students: how shall we know them? (2nd edn) (london, kogan page). rumelhart, d. & norman, d. (1978) accretion, tuning and restructuring: three modes of learning, in: j. w. cotton & r. l. klatzky (eds) semantic factors in cognition (hillsdale, nj, lawrence erlbaum associates). sapsford, r. (1999) survey research (london, sage publications). schank, r. c. & cleary, c. (1995) engines for education (hillsdale, nj, lawrence erlbaum associates). schein, e. h. (1987) process consultation: lessons for managers and consultants (reading, ma, addison-wesley). skinner, b. f. (1958) teaching machines, science, 128, 969–977. suler, j. r. (1997) the black hole of cyberspace. available online at: www.rider.edu/suler/psycyber/ psycyber.html (accessed 28 july 2006). taras, m. (2003) to feedback or not to feedback in student self-assessment, assessment and evaluation in higher education, 28(5), 549–565. thorndike, e. l. (1932) , the fundamentals of learning (new york, teachers college press). vygotsky, l. s. (1978) mind in society: the development of higher mental processes (cambridge, ma, harvard university press). wenger, e. (1987) artificial intelligence and tutoring system (los altos, ca, morgan kaufmann). alt-j, research in learning technology vol. 16, no. 3, september 2008, 197–205 issn 0968-7769 print/issn 1741-1629 online © 2008 association for learning technology doi: 10.1080/09687760802526749 http://www.informaworld.com uncanny spaces for higher education: teaching and learning in virtual worlds siân bayne* university of edinburgh, edinburgh, uk taylor and francis ltdcalt_a_352842.sgm (received 7 april 2008; final version received 8 september 2008) 10.1080/09687760802526749alt-j, research in learning technology1466-4658 (print)/1743-7903 (online)original article2008taylor & francis163000000september 2008dr sianbaynesian.bayne@ed.ac.uk this paper brings together the theory of the uncanny as it emerges in cultural theory, with an understanding of the uncanniness and troublesomeness seen to be inherent in certain understandings of teaching and learning in higher education. drawing on research into students’ experiences of learning in virtual worlds, it explores the sense in which teaching in such spaces materialises and extends the positive aspects of uncertainty, strangeness, disquietude and troublesomeness in online higher education. keywords: virtual world; second life; uncanny introduction avatars are nothing but corpses. so, somebody comes along and will fill those dead corpses with something that is believed to be identity or feelings? for me they are artificially normed identities, perhaps even desires… (eleanor, weblog posting)1 students immersed in second life and other virtual worlds quite regularly describe such environments in terms of their ‘deathliness’, their blurring of the boundary between fantasy and reality, and their foregrounding of issues to do with identity which move beyond the weird ontology of the avatar to the nature of being as a learner across the digital and material domains. such concerns connect directly with the theme of ‘uncanniness’ which this paper will explore. the theory of the uncanny has gained a new significance in the last few decades, with a resurgence of interest in the theme in literary and cultural theory (for example kristeva 1991; castle 1995; arnzen 1997; grenville 2002; tie 2004), robotics (mori 1970; ramey 2005) and technology studies (for example johnson 1999; weight 2006). there are few explicit examples of its application in education, however, with carrington (2005) being an important exception. here, i will draw together insights from students who are actively using virtual worlds as learners, with a theoretical framework drawn largely from freud’s (1919/2003) understanding of the nature of the uncanny. i will then extend this into a discussion of the nature of higher education (he), drawing on barnett’s (2005, 2007) concern with the ‘ontological shift’ in he, and on other constructions of higher education which emphasise the necessity of its troublesomeness (perkins 2006; meyer and land 2005, 2006) and its immersion of learners in productive states of uncertainty (royle 2003). i will end by suggesting that *email: sian.bayne@ed.ac.uk 198 s. bayne second life and other virtual worlds may function as spaces where a positive ‘pedagogy of uncertainty’ (barnett 2007, 137) may be explored and conducted by learners and teachers in higher education. method some of the research data presented here are drawn from a higher education academyfunded project which conducted a virtual ethnography of three formal courses in higher education which were using social networking technologies.2 the virtual ethnographic methodology involved the research associate on the project being immersed in the day-today online and offline interactions of approximately 100 learners in three formal degree programmes spanning undergraduate and postgraduate levels in two large scottish universities. these programmes included full-time undergraduate students in divinity and design engineering and part-time postgraduate distance learners at masters level. data were generated from student weblogs (produced as part of the assessed element of their programme of study), wikis and course discussion boards, and also through online and offline interviews with students and teachers (see hemmi, bayne, and land in press). the interviews took place across a range of environments including online chat, second life, telephone and face-to-face sessions. additional data given here effectively extend this project, being drawn from the textual and thematic analysis of weblogs and discussion board postings of students on the university of edinburgh’s msc in e-learning.3 the students quoted here are all mature postgraduates, and are in general describing experiences as relative newcomers to virtual worlds. scenes of the uncanny what do we mean when we talk about the ‘uncanny’? though the uncanny can create fear, it is something which goes beyond the simply frightening: the uncanny is ghostly. it is concerned with the strange, weird and mysterious, with a flickering sense (but not conviction) of something supernatural. the uncanny involves feelings of uncertainty, in particular regarding the reality of who one is and what is being experienced. suddenly one’s sense of oneself … seems strangely questionable. … it is a crisis of the natural, touching upon everything that one might have thought was ‘part of nature’: one’s own nature, human nature, the nature of reality and the world. but the uncanny is not simply an experience of strangeness or alienation. more specifically, it is a peculiar commingling of the familiar and unfamiliar. … as a crisis of the proper and natural, it disturbs any straightforward sense of what is inside and what is outside. the uncanny has to do with a strangeness of framing and borders, an experience of liminality. (royle 2003, 1) freud’s essay on the theme, originally published in 1919 as ‘das unheimliche’ (literally, the ‘unhomely’), constructs the psychology of the uncanny in terms of the way in which it involves the return of something long repressed, which was once familiar. these may be childhood complexes, or the “remnants of animistic mental activity” (freud 1919/ 2003, 147) – hangovers from an earlier stage of cultural development in which the return of the dead, the supernatural, and the strangeness of other parallel worlds were believed in as real possibilities, in a way which contemporary accounts of the rational have disallowed. freud describes several situations in which the uncanny might emerge, which connect well with royle’s account quoted above: the fear of death and ghosts; the questioning of the alt-j, research in learning technology 199 nature of the human inherent in instances of animation and automation; the questioning of the nature of selfhood which emerges via the theme of the ‘double’ or doppelganger; and – most crucial to the argument presented here – the association of the uncanny with the theme of intellectual uncertainty and the blurring of the boundary between fantasy and reality. it is not hard to see here the link with virtual worlds and the ways of being they embody. in the section which follows, i will expand briefly on each of these themes, connecting them with quotes from students which hint at the ‘uncanniness’ of the learner experience of virtual worlds. death and ghostliness “to many people,” freud tells us, “the acme of the uncanny is represented by anything to do with death, dead bodies, revenants, spirits and ghosts” (148). in the research described here, it was striking how regularly the theme of ‘deathliness’ emerged in student accounts of entering and negotiating virtual worlds (in this case, generally second life). the quote from eleanor with which i began this paper described the ‘corpse’-like nature of the avatar in which identity, and even desire, is artificially normed by the technological environment. here, the avatar is constructed as zombie, as a carcass re-animated in the interplay between user and environment. for other students, the experience of entering the virtual world is itself like one of dying. in the quotes that follow, margaret describes her experience during an induction session in second life run as part of the msc in e-learning at edinburgh, while maisie reflects on identity issues in the virtual world as part of an end-of-course assignment: sometimes trees block my view and i can’t see where i am. it almost felt like a blind person moving around in an unknown territory without the white stick. since our meeting place was so close to the sea i was really worried of drowning because of this time delay. that is another experience that i could not even begin to describe. it truly gives you an out of the body experience with unshakable feelings of getting lost, drowning and even dying. (margaret, weblog posting) virtual worlds’ challenging disassociation of identity from the physical body in exchange for the graphic or text media of the screen can seem at best like going inside a cartoon and in the worst of cases like going through a type of physical death only to awake in the other world with all of one’s senses intact. (maisie, essay extract) for cara, however, ‘ghostliness’ emerges in quite a different way, being more to do with the opportunities for embodied, real-life ‘lurkers’ to be present in second life yet invisible – ‘looking over the shoulder’ of the user at the keyboard: embodied people can’t lurk – they have presence. however, it occurred to me afterwards that there may well be disembodied lurkers in sl, as we show our rl colleagues what is going on on screen. i’ve done this a couple of times. i didn’t do it during our tutorial, but i might have – and now i’m questioning the ethics of this. (cara, weblog posting) in the kind of reversal which is typical in the operations of the uncanny, the ‘ghost’ here becomes, not the avatar as in eleanor’s account, but the real-life person – the living body of the ‘colleague’. the theme of ontological disorientation which pervades these accounts extends across each ‘scene of the uncanny’ being unfolded here, and will be returned to later in terms of its connection to the ‘ontological shift’ in higher education (barnett 2005, 2007). 200 s. bayne animation, automation and insanity an “excellent case” of the uncanny, according to freud, is one highlighted by jentsch (1906) in his early essay on the psychology of the unheimlich. this is to do with animation and the appearance of life in the inanimate – in some senses a return to the zombie theme, which raises “doubt as to whether an apparently animate object really is alive and, conversely, whether a lifeless object might not perhaps be animate”. freud goes on: in this connection [jentsch] refers to the impressions made on us by waxwork figures, ingeniously constructed dolls and automata. to these he adds the uncanny effect produced by epileptic fits and the manifestations of insanity, because these arouse in the onlooker vague notions of automatic – mechanical – processes that may lie hidden behind the familiar image of a living person. (freud 1919/2003, 135) the in-between status of the avatar – a ‘doll’ invested to some extent with the interactional and personality characteristics of the ‘real’ user – encapsulates in its very existence something of this aspect of the uncanny. add to this the ability to automate movement via animation (to enable the avatar to dance, to wave, to kiss, to embrace) and the disorienting blur between life and lifelessness becomes even more foregrounded. the existence of ‘bots’ in-world, avatars controlled by code rather than by a living user, adds a further layer. when the animation and coding ‘goes wrong’ and the avatar is caught in a mechanised, repetitive loop of movement, the effect is without doubt uncanny as the appearance of contingent reality upon which our tolerance and understanding of the virtual world rests is shaken and disrupted. the ‘manifestations of insanity’ associated with this aspect of the uncanny are intriguingly described in an account given by maisie of an encounter – the time in the palace (a 2d virtual world) – with an unknown avatar. maisie describes staying behind in the palace after a tutorial, to continue to chat with another student, fiona. she describes how their chat is interrupted by a strange avatar called ‘happy birthday’: we carried on talking and then it said; “i have loads of goat cheese and i want to give it to you”. i thought, “what on earth is this? is it drugs he is offering or something else? is he just a looney?” we did not say anything. ‘happy birthday’ went on saying even more incoherently: “oh, lord jjjjesus” – with repeated ‘j’s – “i have loads of goat cheese and it’s not evil”, or something like that. at that point i thought, “oh dear, i’d better leave this place now, this is getting a bit much”. ten or fifteen seconds went by – i had not left because i was a little ‘puzzled’ by all that – and then ‘happy birthday’, moving slightly towards us … said: “god, save lord jjjesus, the devil is not here, i have all of this goat cheese that i want to give to you…”. silence and then fiona said “i don’t think so”. i told fiona i would see her later and left. i was afraid! the fact that ‘it’ got closer to me made me really uneasy. so uneasy, that ‘i’ did not even care about leaving ‘fiona’ with ‘it’ alone. after logging out, i found the whole experience strange, unexpected and totally unbelievable! … what i found unbelievable was the level of immersion in which my real ‘i’ was. in spite of having a third person view of the whole incident, the virtual being actually managed to scare the real maisie – to the point that she pulled her virtual representation out of the chess room. … the most shocking thing for me was the impact that its language and its use of silence and space had on me. (maisie, weblog posting) the double the doppelganger is perhaps one of the most pervasive and compelling instances of the uncanny, and the most indicative of the kind of ontological uncertainty we engage with alt-j, research in learning technology 201 when we work within virtual worlds. to enter the virtual world as a user (rather than as one of cara’s ‘ghostly’ onlookers), we are obliged to construct an avatar, to engage with the prospect of our own ‘double’ which, “in all its nuances and manifestations” (freud 1919/ 2003, 141) encapsulates something of the uncanny: this relationship [between ‘self’ and ‘double’] is intensified by the spontaneous transmission of mental processes from one of these persons to the other – what we would call telepathy – so that the one becomes co-owner of the other’s knowledge, emotions and experience. moreover, a person may identify himself with another and so become unsure of his true self; or he may substitute the other’s self for his own. the self may thus be duplicated, divided and interchanged. (freud 1919/2003, 141) it is perhaps in this foregrounding of the nature of selfhood that some of the most powerful pedagogical possibilities of virtual worlds lie. as one tutor commented, “if one wants to explore virtuality … it does not seem appropriate to commit oneself to being one thing” (michael, interview). yet it is here that the most disquieting effects of the virtual also appear – when we work in virtual worlds, we are never entirely able to ‘fix’ the identity of the person with whom we are interacting. in second life tutorials, the identities of tutor and students are largely unconnected to their ‘real life’ equivalents – each member of the class is ‘doubled’ in a disorienting yet exhilarating context of interaction. for some students, this sense of ontological ‘play’ has a profound pedagogical value. margaret describes the experience of being metaphorically embodied in class second life tutorial sessions in the following way: we like to experiment with the appearance of the avatar and through experimentation i think we can gain some understanding of who we really are. philosophically, i think finding ourselves or who we really are, is the ultimate learning experience and all that we do in our lifetime contributes to it. … i think that the feeling of presence counts a lot in sl because otherwise it would be just like a normal chat session. it is what we do with our avatars and how we react to people and other situations that matter most in sl, i think. so i would say that our identities are more real when expressed though an avatar. (margaret, weblog posting) for others, it is deeply disturbing. for instance joe, a co-student of margaret’s, describes the basis of an unease with virtual worlds in terms of cultural specificity and religious belief: the appearance of an avatar may sadly be a ‘put-off’ for those with which it interacts. our perception of attractive and acceptable varies quite a lot and colour mix of ‘clothing’ or type of face chosen by one (avatar) may put the other off and lead to creation of ‘dislikes’. choice of animal heads for avatar faces may send a terribly wrong message in my culture. such an avatar may be considered a sorcerer or witch/wizard. who would want to interact with such a ‘person’? there is also some possibility of offending some people’s religious inclination, e.g. suggesting idolatry. the bible … teaches that we are made in god’s image and if one adopts the image or looks of an animal then s/he is adopting the image of an idol hence s/he is practising idolatry! it is even worse in cases where a person considers the animal face to represent their identity better than their real face. (joe, weblog posting) both margaret and joe comment on the notion of the ‘double’ or the representation gaining a reality-status in the mind of the user which is greater than that of the embodied self. the 202 s. bayne relation between ‘copy’ and ‘original’, ‘self’ and ‘double’ is thoroughly disturbed. for margaret, this is a valued and profound aspect of avatar pedagogy. for joe, it is a deeply problematic instance of the challenge the virtual world might pose to existing religious belief. intellectual uncertainty the final scene of the uncanny which i wish to explore here is the one which is perhaps most crucial to the arguments which follow. this is the notion of the uncanny – proposed by jentsch (1906), questioned by freud (1919/2003), but highlighted as central by later theorists (for example royle 2003; haughton 2003) – as being to do with intellectual uncertainty and the disquietude which occurs when “the boundaries between imagination and reality are erased” (kristeva 1991, 188). the strange, dream-like nature of the virtual world, in which reality and imagination become inseparable, was commented on by several students. the following students all describe their early experiences with second life, prior to formal tutorial sessions on the msc in e-learning: [1:00] dark roly: its really weird, [1:00] you: second life? [1:01] dark roly: yes. its fun but sort of weird, i can see why it becomes addictive. [1:01] you: in what way d’you find it weird? [1:02] dark roly: communicating via an avatar, sort of real and not real all at the same time. (dark, conversation in-world) the wind sounds in the background tend to make it feel like this is a dream or at least all happening in your head. when someone makes a sound then it really stands out like a banshee wailing in the night. (simon, weblog posting) on tuesday evening (night before last) i went on to second life (or should that be *into*). it loaded fine at five o’clock and [my children] changed my clothes for me, so they now no longer believe that this course is hard work and want to know when they can come to play in mummy’s dreamworld as they call it. (gillian, weblog posting) key to understanding this aspect of the uncanny is the idea that in the blurring of the boundary between fantasy and reality the relation between the sign (in our case, the avatar) and what it apparently signifies (the selfhood of the user) shifts and becomes problematic. kristeva (1991), quoting freud (1919/2003, 150), describes this well: what brings together all these symbolic processes … lies in a weakening of the value of signs as such and of their specific logic. the symbol ceases to be a symbol and “takes over the full function of the thing it symbolizes”. in other words, the sign is not experienced as arbitrary but assumes a real importance. as a consequence, the material reality that the sign was commonly supposed to point to crumbles away to the benefit of imagination, which is no more than “the over-accentuation of psychical reality in comparison with material reality”. (kristeva 1991, 186) in other words, the ‘uncanniness’ of the avatar may rest partly in its tendency to be perceived as “taking over the full function” of the real-life user and his or her embodied selfhood. the alt-j, research in learning technology 203 ‘psychical reality’ of the avatar becomes privileged above the ‘material reality’ of the user. this, in turn, relates back to the ontological anxiety foregrounded in many of the student accounts presented here and engages in the notion of intellectual uncertainty surrounding the nature of being in the virtual world. for jentsch, it is this intellectual uncertainty which is “the essential condition for the emergence of a sense of the uncanny” (freud 1919/2003, 125). i wish now to expand on this theme of intellectual uncertainty, by exploring ways in which the strange, the troublesome, the ontological, the uncertain – in fact, the uncanny – can be seen to be integral to the idea of a ‘genuine’ higher education. the ‘uncanniness’ of higher education in exploring the possibility of a theory of ‘haunted teaching’, royle (2003) – drawing on freud and the notion of the uncanny – describes the pure necessity of intellectual uncertainty as the basis of a high quality university teaching: intellectual uncertainty is not necessarily or simply a negative experience, a dead-end sense of not knowing or of indeterminacy. it is just as well an experience of something open, generative, exhilarating (the trembling of what remains undecidable). i wish to suggest that ‘intellectual uncertainty’ is in part what freud’s essay has to teach and, indeed, that this is a crucial dimension of any teaching worthy of the name. (52) the ontological uncertainty foregrounded in the student accounts given here perhaps simply indicates that second life and other virtual worlds materialise this uncertainty in new ways – they defamiliarise our sense of selfhood and our mode of being together within the pedagogical context, and in doing so ask us to reflect on it afresh as teachers and learners. barnett’s (2007) text, subtitled ‘being a student in an age of uncertainty’, highlights the centrality to the contemporary student of the kind of intellectual indeterminacy described by royle. for barnett “the uncertainty principle reaches deeply into the student’s being” (127), and one of the prime goals of a genuine higher education must be to help students become able to accept and live in a positive way with the kind of intellectual uncertainty which is integral to a life informed by academic ways of thinking and practising. such a way of thinking is almost inevitably uncomfortable and anxiety-inducing. indeed, for barnett, “a pedagogical achievement of higher education is that – on the student’s part – of coming to live purposefully with anxiety” (36, original emphasis). a connection might be made here with the notion of ‘troublesome knowledge’ described by perkins (1999, 2006) – the “ritual knowledge, inert knowledge, conceptually difficult knowledge, foreign knowledge, and tacit knowledge” (2006, 37) which may induce in students a sense of anxiety and disorientation, appearing “counter intuitive, alien …, or incoherent” (meyer and land 2006, 9). for meyer and land, such troublesome knowledge is itself implicated in the notion of ‘threshold concepts’, which suggests that there is an ontological dimension to the way in which students come to negotiate – though not necessarily resolve – such areas of difficulty and uncertainty. in negotiating a threshold concept, the perception of the student undergoes a “significant shift … [which] may lead to a transformation of personal identity, a reconstruction of subjectivity” (7). to cross such a threshold is to experience the kind of liminality which royle (2003, 1) sees as central to the experience of the uncanny (meyer and land 2005). the terms emerging, therefore, in recent accounts of the process and purpose of higher education – haunted teaching, troublesome knowledge, risk, anxiety, thresholds, liminality, uncertainty – might incline us towards an understanding of contemporary higher education 204 s. bayne as itself incorporating something of the uncanny. for barnett (2007), in fact, the ontological dimension of studenthood implicitly and explicitly involved in such terms is absolutely central to an understanding of what it means to conduct higher education in the twenty-first century. higher education, he argues, is “in urgent need of an ontological turn” (26), an understanding that “a genuine higher education is none other than a transformation of being” (38). in an age of rapid change, in which all knowledge is conceived as contestable and contingent, such transformation must be in terms of “the taking on of a mode of being for uncertainty” (39). to teach, therefore, is to engage with new ontologies which make students ‘strangers to themselves’ (kristeva 1991), to immerse students in worlds of uncanny strangeness: the student is perforce required to venture into new places, strange places, anxiety-provoking places. this is part of the point of higher education. if there was no anxiety, it is difficult to believe that we could be in the presence of a higher education. (barnett 2007, 147) in conclusion: virtual worlds and the ‘pedagogy of uncertainty’ as this paper has attempted to demonstrate, virtual worlds can, for some students, themselves represent such worlds of strangeness – they can be uncanny spaces, in which new, anxiety-provoking yet rich understandings of the nature of being in a digital age can be confronted. in some senses, the virtual world literalises the metaphorical and cognitive ‘strangeness’ and ‘troublesomeness’ which some of the most interesting accounts of contemporary higher education describe. i do not suggest here that virtual worlds must be the primary location of a new higher education which is concerned with the ontological. my argument is rather that such spaces can be used with some students, in some contexts, as a way of working productively with the ‘strangeness’ and ‘uncanniness’ of contemporary academic – and digital – ways of being. in such alluring, difficult, problematic spaces we might have new opportunities to nurture what barnett refers to as a “pedagogy for uncertain times”: a pedagogy for uncertain times has itself to be uncertain. it is open, it is daring, it is risky, it is, itself, unpredictable … a pedagogy for uncertainty will be ontologically disturbing and enthralling all at once. it will be electric, as one move sparks another and in unpredictable ways … this pedagogy is a form of restrained anarchy; even a disciplined anarchy – with its spaces and its risks. (2007, 137) notes 1. all student names have been changed. 2. thanks to professor ray land and dr akiko hemmi, collaborators on the hea project, for allowing me to use the data here. 3. many thanks to the students on the msc in e-learning at the university of edinburgh, who so generously allowed me to use their words here. references arnzen, m. 1997. introduction. the return of the uncanny. paradoxa: studies in world literary genres,special issue, 3, nos. 3–4. barnett, r. 2005. recapturing the universal in the university. educational philosophy and theory 27, no. 6: 785–97. ———. 2007. a will to learn: being a student in an age of uncertainty. buckingham, uk: society for research in higher education and open university press. alt-j, research in learning technology 205 carrington, v. 2005. the uncanny, digital texts and literacy. language and education 19, no. 6: 467–82. castle, t. 1995. the female thermometer: eighteenth-century culture and the invention of the uncanny. oxford: oxford university press. freud, s. 1919/2003. the uncanny. london: penguin classics. grenville, b. 2002. the uncanny: experiments in cyborg culture. london: arsenal. haughton, h. 2003. ‘introduction’ to sigmund freud. in the uncanny, vii–lx. london: penguin classics. hemmi, a., s. bayne, and r. land. in press. the appropriation and repurposing of social technologies in higher education. journal of computer assisted learning. jentsch, e. 1908. zur psychologie des unheimlichen. psychiatrisch-neurologische wochenschrift 8: 195–205. johnson, c. 1999. ambient technologies, uncanny signs. oxford literary review 21: 117–34. kristeva, j. 1991. strangers to ourselves. london: harvester wheatsheaf. meyer, j.h.f., and r. land. 2005. threshold concepts and troublesome knowledge (2): epistemological considerations and a conceptual framework for teaching and learning. higher education 49, no. 3: 373–88. ———. 2006. threshold concepts: an introduction. in overcoming barriers to student understanding: threshold concepts and troublesome knowledge, ed. j.h.f. meyer, and r. land, 3–18. london: routledgefalmer. mori, m. 1970. the uncanny valley. energy 7, no. 4: 33–5. http://graphics.cs.ucdavis.edu/∼staadt/ ecs280/mori1970otu.pdf perkins, d. 1999. the many faces of constructivism. educational leadership 57, no. 3: 6–11. ———. 2006. constructivism and troublesome knowledge. in overcoming barriers to student understanding: threshold concepts and troublesome knowledge, ed. j.h.f. meyer, and r. land, 33–47. london: routledgefalmer. ramey, c.h. 2005. the uncanny valley of similarities concerning abortion, baldness, heaps of sand, and humanlike robots. in proceedings of views of the uncanny valley workshop: ieee-ras international conference on humanoid robots, tsukuba, japan, 8–13. royle, n. 2003. the uncanny. manchester: manchester university press. tie, w. 2004. the psychic life of governmentality. culture, theory and critique 45, no. 2: 161–76. weight, j. 2006. i, apparatus, you: a technosocial introduction to creative practice. convergence: the international journal of research into new media technologies 12, no. 4: 413–46. alt-j, research in learning technology vol. 17, no. 2, july 2009, 131–141 issn 0968-7769 print/issn 1741-1629 online © 2009 association for learning technology doi: 10.1080/09687760903033074 http://www.informaworld.com guidelines for supporting re-use of existing digital learning materials and methods in higher education judith schoonenbooma*, henk sligtea and eja kliphuisb auniversity of amsterdam – sco-kohnstamm institute, amsterdam, the netherlands; binholland university – icto-okr, haarlem, the netherlands taylor and francis ltdcalt_a_403480.sgm (received 6 may 2008; final version received 16 november2008) 10.1080/09687760903033074alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2009association for learning technology172000000july 2009dr judithschoonenboomj.i.schoonenboom@uva.nl the literature on the re-use of learning materials has largely focused on the development of materials. this paper explores how re-use can be stimulated after learning materials have been developed and made available. we searched for and developed guidelines that support staff and/or management most frequently adopt in cases of (un)successful re-use of existing digital teaching and learning materials and methods by teachers in higher education. in a grounded theory approach, we collected existing guidelines from 11 literature studies, and developed new guidelines from 19 (mainly dutch) case studies. through constant comparison, we developed ‘low-level’ guidelines, which were classified into high-level guidelines. five high-level guidelines turned out to be robust when subjected to repeated comparison with the case studies. further, this paper elaborates existing guidelines by showing specific elaborations of them in our case studies. keywords: educational innovation; higher education; teacher support; re-use; digital learning materials introduction there is now an extensive literature on the re-use of learning objects, i.e. resources “usually digital and web-based, that can be used and re-used to support learning” (wiley 2000). up till now, the literature has focused on the development of materials, with significant attention being paid to the design of learning objects to optimise their re-use. this has led to a debate on the extent to which learning objects should contain such diverse elements as context (quinton 2007), pedagogy (mccormick 2003) and learning objectives (boyle 2003). learning materials have been classified into various hierarchies (cisco systems 2003; wagner 2002; wiley 2000), and learning object design trade-offs have been formulated, e.g. between size and re-usability (south and monson 2000). attention has also been paid to encouraging re-use by making learning materials available through repositories, using metadata (sonntag 2007). this has led to the formulation of principles for good metadata (niso press 2004); the development of metadata standards such as learning object metadata (lom) (institute of electrical and electronics engineers 2002); a discussion on the use of tagging versus metadata (smith 2008); and the formulation of a trade-off between learning material size and costs of metadata (wiley 2002). *corresponding author. email: j.schoonenboom@ond.vu.nl 132 j. schoonenboom et al. however, much less attention has been paid to the question of how re-use can be stimulated after development. in this situation, one or more collections of existing learning materials are available, and the question is how can teachers be stimulated to re-use these existing materials? this question has become increasingly relevant in recent years, as an abundance of learning materials has become available through the internet and through institutional and shared repositories. in this paper we will focus on the re-use of existing learning materials in dutch higher education. re-using existing learning materials in higher education is different from developing re-usable learning materials, in several respects. firstly, it is educational support staff and managers rather than developers who support re-use in this situation. secondly, support staff have fewer possibilities than developers to create an optimal situation for re-use. they will be confronted with a range of existing materials, varying in terms of fit to teachers’ needs, quality of the metadata and appropriateness of their size. thirdly, while developers can stimulate re-use by developing high-quality learning materials, support staff focus must be on the teachers, coaching them towards successful re-use in their teaching. although this post hoc stimulation of re-use has received less attention than the development of learning materials, several guidelines for it can be found in the literature. an example is peer review (campbell 2003; lim 2007; metros, bennett, and diaz 2001), which was shown to resolve teachers’ anxiety about the quality of the learning materials (littlejohn 2003a). the aim of our study is to bring together the most important guidelines from the literature and develop additional guidelines from existing case studies. we develop a theory for supporting the re-use of existing learning materials, which is both ‘grounded’ in the data (charmaz 2006) and ‘humble’ in the sense of cobb et al. (2003, 10), the theory being “concerned with domain-specific [support] processes, … accountable to the activity of design”, and “provid[ing] detailed guidance in organising [support]”. although these guidelines may be matched onto stages of rogers’ (2003) ‘grand theory’ of the innovation process, this is not the focus of our paper. most of the research for our study was conducted in a six-month study for surf, the dutch foundation for ict in higher education (schoonenboom et al. 2007). our main research question is: what guidelines do support staff and/or management most frequently follow in successful and unsuccessful cases of re-use of existing digital teaching and learning materials and methods by teachers in higher education? how can these guidelines be classified? in this research question, following a guideline can be done either explicitly or implicitly. some guidelines were mentioned by participants as guidelines they had followed, others were derived from information on cases of re-use, and participants may not have been aware of them. we looked at successful and unsuccessful cases of re-use of existing learning materials, as we would expect that useful guidelines should contribute to the success of re-use (and ignoring them may lead to unsuccessful reuse). our research aims to discover the guidelines most frequently followed. the more often a guideline is encountered in successful cases, the more likely it is that its success is not a coincidence, and the more useful it will be to educational support staff. guidelines are classified not only to arrive at a more manageable presentation, but also to arrive at more general guidelines to be followed, and to inform practice in several ways. alt-j, research in learning technology 133 our criteria for including learning materials and methods is whether they can be used by individual teachers in their own teaching. examples include materials such as study tasks, courses, pictures or theses, or methods such as working in small virtual groups, concept mapping or the use of webcams. outside the scope of this research lies, for example, the institution-wide deployment of competence-based learning, or the development of a master degree at a distance. methodology design guidelines were derived through two methods. firstly, we searched for guidelines in a sample of relevant literature. these guidelines were not treated as ‘proven important guidelines’, but as input for the formulation of guidelines. secondly, we derived guidelines from a sample of case studies of successful and unsuccessful re-use of existing learning materials. we then applied a grounded theory approach (charmaz 2006), in which we derived general guidelines from the literature and case studies samples together, using constant comparison. finally, the resulting guidelines were compared with the recent literature. sampling from practice we followed a multi-stage purposeful sampling strategy (onwuegbuzie and leech 2007). our first sample contained case studies of re-use of existing digital learning materials and methods in dutch higher education. following the general principles of robert stake (1995, 2006), we selected those cases from which we expected to learn most. we used the following criteria for selecting case studies: (1) we knew the cases; (2) they were recent; (3) descriptions with some detail were available; (4) we knew something interesting was done on re-use of existing materials; (5) something had been learned and/or was assumed by the participants on success or fail factors with reuse; and (6) which together led to a variety of guidelines. on the basis of the first five criteria, we selected a first set of case studies. since it turned out that the guidelines derived from these case studies were too limited in scope, we then used theoretical sampling (glaser and strauss 1967) to increase the variety of the guidelines, to ensure they covered the whole innovation process and to find more evidence and counterevidence for our guidelines-in-development. we added a second sample of case studies from an existing collection, which had been brought together in our earlier study learning objects in practice (schoonenboom et al. 2006). in this study, carried out between 2004 and 2006, we identified and contributed to solutions for important problems for those working with learning objects in dutch higher education. in this study, a set of six dutch case studies had been collected, which met the following criteria: (1) they were related to digital learning materials; (2) they had successfully applied for funding with the dutch national surf organisation or the dutch digital university; (3) they had been finished successfully; and (4) the results were used in practice. this second sample had in the original study already been supplemented by two other case studies of successful practice and two running experiments, in order to increase diversity and to increase the validity of the discovered guidelines. the literature that we used as data for developing guidelines was also taken from our study learning objects in practice (schoonenboom et al. 2006). at the start of this 134 j. schoonenboom et al. project in 2004, four learning objects experts, each with four or more years of experience (both at a national and an international level) conducted a literature review in the problems of working with digital learning materials. a description of the 19 case studies and 11 literature studies used in the analyses can be found in schoonenboom et al. (2007). procedure the procedure consisted of four steps. figure 1 shows the results of each step, which in the text are indicated in bold. figure 1. summary of one practice, showing the elements of the derivation of guidelines: description of success, short descriptions of actions, low-level guidelines and high-level guidelines; numbers of high-level guidelines refer to their number in figure 2. step 1. identification of ‘actions’ in case studies detailed descriptions of the case studies were either collected or prepared by the researchers. each description included a description of the success or lack of success of the practice. within each example of practice, ‘actions’ were identified, which seemed to have yielded or contributed to its success or lack of success. ��� ����� �� ��� ������ � � �� ��������� �� ��� ���� �� � �������� � �������� � ��� �������� ��� ��� ��� �� � ���������� ����� �� ��� � ��� ������ � �������� �� �������� �� ��� ��������� ������ ���������� �������� �� ������ ��������� � � ���� �� ����� ������ ����� ����� �� !�������� ��� ��������� �� � � ��� ��� ����� �� � � ������� � ���� � ������� " ��� ����� ���� ����� #$%% ��������� ���� ���������& � � ��� ������ ��� ��������� ��� ���� �� � ����� '$%% ���������� � � ��� ���(��� ������ ���� � ���� ����� ��� � ���������� �� �� ����� �� �� � ��� �� �)� ��� ���������� �� ���� ���� ���� �� �� ������� ��� ������ � �*� �� ���� �������� �� ������ +����� ���� � � �,� -���� �� �� ��� � ���� � �������� ���� ���� �� �� � ������ ��� ����� �������� ���� � � ����� ��� ������� � ������������� ���� �� ��� ������ � ������� �� ������ �.� -���� �� �� ��� ���� �� �� ��������� ����� ���� �� � ������ � � ��� ����� ������� �� ������ � ������� �� ��� ��� /� ) 0���� �������� �������� � � ������ ������ ������� � ��� �� ����� ���� ����������� �������� �� �������� �� � ����� �����+����� ������� �1 )� /� , �� ��� �2�� � ���� �������� � � ����� �� ��� ������ ���� �� ������ �� ������ ��� ���� � � ��������� � � ������ �� ������ ���� �� ������ ����� � ����� ��� �� ��� ��� �������� ����� ��� �� ����� ������� �����+����� ������� �1 ,� /� . � ���� ��� ������� �� � ������ � �� ������ ����� �� ���� �������� � � ����� �� � � ������� �� ��� 3������ �� ��� �������� � � ������ � �������� �����+����� ������� �1 #� figure 1. summary of one practice, showing the elements of the derivation of guidelines: description of success, short descriptions of actions, low-level guidelines and high-level guidelines; numbers of high-level guidelines refer to their number in figure 2. alt-j, research in learning technology 135 step 2. formulation of low-level guidelines for each action, a short description was created. the literature sample was scanned for existing guidelines, and short descriptions of these guidelines were prepared. from the short descriptions of both the existing guidelines in the literature and the practice actions, low-level guidelines were derived. the derivation of the low-level guidelines was an iterative process, using repeated comparison (charmaz 2006; glaser and strauss 1967). first, a set of guidelines was formulated, and as many short descriptions as possible were grouped under these guidelines. reflection on the remaining descriptions led to the formulation of new guidelines and so on, until each short description was grouped under one low-level guideline. then, the literature and the detailed descriptions of the case studies were analysed again, looking for additional instances of actions that would fit under the low-level guidelines. step 3. formulation of high-level guidelines finally, the low-level guidelines were categorised into high-level guidelines, such as ‘provide easy access to the materials and methods’ in figure 2. the same procedure of repeated comparison was used as in step 2. step 4. testing the validity of the guidelines the resulting high-level guidelines were tested for robustness. we performed a splithalf test, in which we randomly divided the literature and practice cases into two. the split-half test showed that guidelines derived from 10 or more cases were robust in that they could be derived from either half of the cases. these five guidelines were kept, and the other non-robust guidelines were removed. finally, we checked a recent three-volume work on learning objects for remarks on these guidelines (harman and koohang 2007; koohang and harman 2007a, 2007b). validity throughout our study, we had to deal with a variety of validity threats. consequently, we used a variety of strategies to deal with these threats. ● case studies were divided among the researchers, and each researcher analysed those practices with which he or she was most familiar (stake 1995, 2006). most of the case studies we knew very well and over a long period, thereby reducing the risk of misinterpretations (becker 1970). ● we used multiple data sources; we used multiple cases, which were multiple in kind, including both descriptions of case studies and literature studies, and by multiple authors. this reduced the risk of biases, resulting from studying one specific case using a specific type of resource by one specific researcher. ● descriptions of case studies had been validated by participants. this applied to all samples. ● to ensure that the guidelines would be robust enough to capture a variety of situations, we monitored the guidelines at any point in our research project, to see if they provided enough variation. when it turned out that the guidelines of our first sample did not explicitly refer to digital learning materials, we included a second sample of case studies and literature studies. 136 j. schoonenboom et al. ● the descriptions and analyses of the six most demanding, initially selected case studies were either done by two researchers, or by one researcher and checked by another. ● within each individual practice, we devoted one question in our analysis template to a systematic search for additional evidence and for rival explanations of the presumed relation between action and success (or failure). evidence often included signs of growing and sustained use. sometimes, the presumed relation was backed by evidence from the literature. ● with each guideline that we had derived from an individual practice, we checked its occurrence in all other case studies. furthermore, we checked whether guidelines found in one case study were contradicted in other case studies, but we found no examples of this. ● reformulations in the guidelines were thoroughly discussed between researchers. outcomes figure 2 shows the five high-level guidelines that we derived. after each guideline, the number of cases on which it is based is indicated in brackets. figure 2. the five high-level guidelines for supporting the re-use of existing digital learning materials and methods; numbers indicate the number of the 30 case studies and literature studies on which each item is based. guideline 1 is mostly relevant to managers who at the beginning of the process are considering re-use of existing learning materials. guidelines 2–5 become relevant once the decision for re-use has been taken and is mostly relevant to support staff. the sections below each present one high-level guideline, with an explanation of how it was derived from its underlying low-level guidelines. a document is available in dutch that shows for all guidelines the whole derivation process from detailed descriptions to actions to low-level guidelines to high-level guidelines (schoonenboom et al. 2007). 1. develop an appropriate vision on the incipient re-use of the materials and methods within the programme, department or institution from the beginning an appropriate vision of re-use should be developed by the teacher(s), department or institution involved. our study showed that the following elements of this vision, which had been identified in the literature, occur frequently in the case studies: ��� ������� � ������ �� � � �� �� ��� �� � ��� ������ �� ��� � ��� �� � �� � ��� ����� ���� ��� ������ �� ��� ��� �� ������� ��� ���� �� � � ����� �� ��� � ��� �� ����!�� �"� ���� �� � ��� �� �� � � � ���� ��� �� �����# ��� ����� �� � �� �� �� �� ������� ��� ���� �� � ��� �� �� ���� $� � � � �� �� %�� �� ����� �� �������� �� &���� ��� $� � � �� � ��� �� �� ������� ����'�� �(� )� %�� ������ � � �� � ��� �� ��� � ��� �� �� ������� �� ��� �����*� ����+�� figure 2. the five high-level guidelines for supporting the re-use of existing digital learning materials and methods; numbers indicate the number of the 30 case studies and literature studies on which each item is based. alt-j, research in learning technology 137 (1) the materials should have a demonstrable added value for teaching (rogers 2003), which is visible to the teachers and managers involved. examples include materials that can be taken into account in students’ assessment and materials on which use feedback can be provided by either teacher or students. (2) re-use of materials should be considered a gradual process of small changes. teachers are more inclined to apply small innovations than to change the whole curriculum (rogers 2003). (3) re-use of materials should be viewed as a way of improving education, not as a way of reducing costs. in fact, costs may go up (pennings et al. 2005). our case studies show that institutions should consider costs for updating the materials (littlejohn, jung, and broumley 2003). also, materials or methods that might be provided for free during the course of a project might require a license after the project has finished. new in our study is that it shows examples in practice of visions that may stimulate re-use. we found that this vision might be a big or a small idea. in one practice, a repository was established in support of the ‘big’ idea of problem-based learning. existing learning materials were collected that together support the whole cycle of problem-based learning in the department involved. in another practice, the ‘small’ idea of visual learning was prominent, and learning materials and methods were collected that could contribute to this idea. although earlier literature has identified ideas such as problem-based learning as a basis for developing new learning materials (many examples in koohang and harman 2007a; wiley 2003), we found that this is also relevant in working with existing learning materials. 2. provide easy access to the materials and methods if it is too difficult for teachers to get access to the digital learning materials and methods, they will not use them. this point has received much attention in the literature on learning object development. as a first solution to this problem, the importance of good quality metadata has been emphasised, so that users find relevant, and neither too few nor too many learning materials (koper 2003; sonntag 2007). a second solution is the use of ‘tailored wrappers’, information on the local context of use, which is added to existing learning materials (quinton 2007). these two solutions show up in our case studies, but with two elaborations. first, two case studies show that it can be useful not to provide access to all learning materials of an available repository, but instead to make a selection of materials that are deemed relevant to the teachers involved. second, one practice made use of tailored wrappers which contained: information on which colleagues had already used these materials; information on where one could get help; own metadata with familiar local terms which was added on top of the existing metadata. 3. provide materials and methods that are a good fit to teachers’ own needs and way of teaching if materials and methods are a good fit to teachers’ own needs and ways of teaching, then it will be easy for teachers to use them (rogers 2003). on the other hand, learning materials cannot easily be used if, for example, the cultural gap with the teacher’s own 138 j. schoonenboom et al. practice is too wide (littlejohn 2003b), or each piece is too large, containing too much material that is not useful (downes 2002). in line with the literature, our case studies show that if the conditions of use of the target institution are too different from those where the materials were developed, then re-use will not be possible. a good fit also implies that teachers should be able to use the materials selectively, only the elements they deem to be useful. student needs must be considered too; for example, materials that fit the students’ future professional practice can be particularly useful. materials that are a poor fit include those that are too complex for the level at which they are intended to be used, or those that clash significantly with aspects of the pedagogy in use, or if they are built upon an ‘alien’ pedagogy. 4. provide materials of good quality, and enable teachers and students to judge the quality of materials learning materials should be of good quality. at the level of the repository, too much variation between learning materials is considered a problem (downes 2002). teachers and students should be enabled to judge their quality (littlejohn 2003a); peer review is seen as a very useful method (metros, bennett, and diaz 2001). in line with the literature, we found that the collection of materials in one repository should be consistent, and with respect to the domain involved the materials should be as complete as possible, and of equal quality. for students, it is important that multimedia and interactive materials are provided (ravenscroft, boyle, and jones 2004). regarding judgement of quality by teachers and students, we found in the cases that we studied the use of peer review, in which peer teachers tested the usability of materials before use in the institution. in one practice, peer review was seen as especially important for materials on the web because of the unclear status of these materials. the second form of judgement in our case studies was rating and annotating by users to inform their peers on how to use the learning material involved (sonntag 2007). in addition to what is known from the literature, we found in one example of practice a clear limit to differences in the size of learning materials. here, teachers did not appreciate that the results of one search could include materials as diverse as study tasks and pictures. in examining peer review, we found that teachers rely partly on whether they know the person who developed the materials and the status that this person has. 5. enable necessary adaptations of the materials and methods to the context teachers should be able to use the materials and methods in their teaching. an important element of this is that they should be able to adapt the materials and methods to their own teaching context. several factors might hinder use. as a necessary condition, learning materials should ‘run’ in the institution’s virtual learning environment. also, copyright problems concerning the re-use of the materials should be solved (campbell 2003). teachers must be provided with the necessary tools and support. lack of userfriendly tools with which teachers can re-use learning objects is considered a problem (mcnaught 2003). also, time is an important constraining factor (littlejohn, jung, and broumley 2003), and enough time should be provided for teachers to evaluate and test materials. alt-j, research in learning technology 139 in line with these findings in the literature, we found in our research that the userfriendliness of the materials and the occurrence of technical problems were also constraining factors. if adaptation takes too much time, teachers will not use the materials. we also found the need for specific forms of support. support should be provided to users at several levels of expertise, for both students, managers, beginning teachers and experienced teachers. explaining the pedagogical possibilities of the materials helps, as this may not be clear from the materials themselves. collective support should be complemented by individual support. in addition to what was found in the literature, our case studies showed that use of the materials is stimulated by removing two important practical hindrances to re-use by enabling teachers and students to work with the materials from home and by making the necessary software freely available. one additional form of support in the case studies was to provide support to beginners by letting them join more experienced colleagues. conclusions this paper has contributed to the discussion on re-use of learning materials in several ways. it has brought together existing answers and developed new answers to questions that had perhaps been solved for the development of learning materials from scratch, but not for situations in which teachers and their supporters are working with existing learning materials. our case studies have shown that also in this situation, an overarching educational idea or vision is required. furthermore, although the existing materials may not all be optimal, access can be optimised by making a relevant selection, by adding own metadata on top of existing metadata and by bringing down the range of different levels of aggregation of learning materials that teachers can find in one search. our study has grounded existing guidelines and provided an overview by showing which of these are important in practice, in that they occur often in our cases. it has specified existing guidelines, in that it has shown some specific elaborations of these guidelines in practice. finally, our paper has demonstrated a method for deriving guidelines from case studies, testing their robustness and combining these with findings in the literature. references becker, h.s. 1970. sociological work: method and substance. chicago, il: aldine. boyle, tom. 2003. design principles for authoring dynamic, reusable learning objects. australian journal of educational technology 19: 46–58. campbell, l.m. 2003. engaging with the learning object economy. in reusing online resources: a sustainable approach to e-learning, ed. a. littlejohn, 35–45. london: kogan page. charmaz, k. 2006. constructing grounded theory: a practical guide through qualitative analysis. london; thousand oaks, ca: sage publications. cisco systems. 2003. reusable learning object strategy: designing and developing learning objects for multiple learning approaches. version 4.5. http://www.e-novalia.com/materiales/rlow__07_03.pdf cobb, p., j. confrey, a. disessa, r. lehrer, and l. schauble. 2003. design experiments in educational research. educational researcher 32: 9–13. downes, s. 2002. the learning object economy. http://www.downes.ca/files/ learning_object_economy.doc 140 j. schoonenboom et al. glaser, b.g., and a.l. strauss. 1967. the discovery of grounded theory: strategies for qualitative research. new york: aldine de gruyter. harman, k., and a. koohang. 2007. learning objects: applications, implications, & future directions. santa rosa, ca: informing science press. institute of electrical and electronics engineers. 2002. draft standard for learning object metadata. new york: institute of electrical and electronics engineers, inc. http:// ltsc.ieee.org/wg12/20020612-final-lom-draft.html koohang, a., and k. harman. 2007a. learning objects and instructional design. santa rosa, ca: informing science. koohang, a., and k. harman. 2007b. learning objects: theory, praxis, issues, and trends. santa rosa, ca: informing science. koper, r. 2003. combining re-usable learning resources to pedagogical purposeful units of learning. in reusing online resources: a sustainable approach to e-learning, ed. a. littlejohn, 46–59. london: kogan page. lim, g. 2007. instructional design and pedagogical considerations for the ins-and-outs of learning objects. in learning objects and instructional design, ed. a. koohang and k. harman, 89–137. santa rosa, ca: informing science press. littlejohn, a. 2003a. an incremental approach to staff development in the reuse of learning resources. in reusing online resources: a sustainable approach to e-learning, ed. a. littlejohn, 221–33. london: kogan page. littlejohn, a. 2003b. issues in reusing online resources. in reusing online resources: a sustainable approach to e-learning, ed. a. littlejohn, 1–6. london: kogan page. littlejohn, a., i. jung, and l. broumley. 2003. a comparison of issues in reuse of resources in schools and colleges. in reusing online resources: a sustainable approach to e-learning, ed. a. littlejohn, 212–20. london: kogan page. mccormick, r. 2003. keeping the pedagogy out of learning objects. paper presented at the 10th biennial conference of the european association for research on learning and instruction (earli), september 26–30, in padova, italy. http://www.eun.org/eun.org2/ eun/include_to_content/celebrate/file/keepingpedagogyoutoflos3v2.doc mcnaught, c. 2003. identifying the complexity of factors in the sharing and reuse of resources. in reusing online resources: a sustainable approach to e-learning, ed. a. littlejohn, 199–211. london: kogan page. metros, s., k. bennett, and v. diaz. 2001. the promise and pitfalls of learning objects: status of digital resource collections. presentation given at educause 2001, october 28–31, in indianapolis, in. http://net.educause.edu/ir/library/powerpoint/nli0201.pps niso press. 2004. understanding metadata. bethesda, md: niso press. http:// www.niso.org/publications/press/understandingmetadata.pdf onwuegbuzie, a.j., and n.l. leech. 2007. a call for qualitative power analyses. quality and quantity 41: 105–21. pennings, l., m. van staden, s. limonard, and v. frissen. 2005. van bloei naar groei; condities voor ontwikkeling en toepassing van business modellen voor webbased educatieve content. [from bloom to growth: conditions for development and application of business models for web-based educational content]. delft, the netherlands: tno. http:// www.ez.nl/dsresource?objectid=143709&type=pdf quinton, s.r. 2007. contextualisation of learning objects to derive meaning. in learning objects: theory, praxis, issues, and trends, ed. a. koohang and k. harman, 113–79. santa rosa, ca: informing science. ravenscroft, a., t. boyle, and r. jones. 2004. using learning objects to enhance blended learning. in jisc innovative e-learning practice case studies, ed. t. boyle, j. cook, and j. clay. london: london metropolitan university. http://www.jisc.ac.uk/ uploaded_documents/learnobjs_ed.doc rogers, e.m. 2003. diffusion of innovations. 5th ed. new york: free press. schoonenboom, j., h. hermans, s. poortman, f. benneker, b. emans, j. meijer, f. de vries, and p. sloep. 2006. learning objects in practice. 6 vols. utrecht, the netherlands: stichting digitale universiteit. http://www.sco-kohnstamminstituut.uva.nl/zips/learning%20objects%20in%20practice.zip schoonenboom, j., h. sligte, e. kliphuis, and l. van elk. 2007. hergebruik van materialen en methodieken: een op praktijkvoorbeelden gebaseerd beslissingsschema voor het hoger alt-j, research in learning technology 141 onderwijs – digitaal beslissingsschema [re-use of materials and methods: a practicebased decision scheme for higher education – digital decision scheme]. utrecht, the netherlands: stichting surf. smith, g. 2008. tagging: people-powered metadata for the social web. berkeley, ca: new riders. sonntag, m. 2007. syntax and semantics of learning object metadata: the ieee/ims lom and beyond. in learning objects: theory, praxis, issues, and trends, ed. a. koohang and k. harman, 417–505. santa rosa, ca: informing science. south, j.b., and d.w. monson. 2000. a university-wide system for creating, capturing, and delivering learning objects. in the instructional use of learning objects: online version, ed. d. wiley. http://reusability.org/read/chapters/south.doc stake, r.e. 1995. the art of case study research. thousand oaks, ca: sage. stake, r.e. 2006. multiple case study analysis. new york: the guilford press. wagner, e.d. 2002. steps to creating a content strategy for your organization. the elearning developers’ journal, the elearning guild. http://www.elearningguild.com/pdf/2/ 102902mgt-h.pdf wiley, d.a. 2000. connecting learning objects to instructional design theory: a definition, a metaphor, and a taxonomy. in the instructional use of learning objects: online version, ed. d.a. wiley. http://reusability.org/read/chapters/wiley.doc wiley, d.a. 2002. learning objects need instructional design theory. in the astd e-learning handbook, ed. a. rossett, 115–26. new york: mcgraw-hill. http://opencontent.org/docs/ astd.pdf wiley, d.a. 2003. learning objects: difficulties and opportunities. utah state university. http://opencontent.org/docs/lo_do.pdf alt-j, research in learning technology vol. 15, no. 2, june 2007, pp. 131–141 issn 0968-7769 (print)/issn 1741-1629 (online)/07/020131–11 © 2007 association for learning technology doi: 10.1080/09687760701482234 methodological issues in using sequential representations in the teaching of writing chien-ching leea*, jeannette boprya and john hedbergb ananyang technological university, singapore; bmacquarie university, australia taylor and franciscalt_a_248100.sgm10.1080/09687760701482234alt-j research in learning technology0968-7769 (print)/1741-1629 (online)original article2007taylor & francis152000000june 2007chien-chingleeleecc@ntu.edu.sg this study looks at a specific application of ainsworth’s conceptual framework for learning with multiple representations in the context of using multiple sequential graphic organizers that are student-generated for a process-writing task. process writing refers to writing that consists of multiple drafts. it may be a process of re-writing without feedback or re-writing based on feedback where the teacher or peers will provide feedback on the original draft and then the students will revise their writing based on the feedback given. the objective was to explore how knowledge of students’ cognitive processes when using multiple organizers can inform the teaching of writing. the literature review analyzes the interaction of the design, function and task components of the framework; culminating in instructional approaches for using multiple organizers for classes with students of different writing abilities. extended implications for designers of concept mapping tools based on these approaches are provided. introduction graphic organizers are visual representations of ideas in keyword format. they can be categorized according to function (e.g. compare and contrast, problem and solution) or form (e.g. spider map and tree diagram). multiple representations, on the other hand, refer to different types of representations in terms of modality (visual, audio or textual) or number (more than one representation of the same type). the purpose of this paper is to explore how knowledge of students’ cognitive processes when using multiple organizers informs the teaching of writing. this paper will present the writing process of poor writers and how graphic organizers scaffold the process, before proceeding to review the use of multiple representations in learning and the methodological issues in the use of sequential graphic organizers in process writing. * corresponding author. school of materials science and engineering, nanyang technological university, nanyang avenue, singapore 639798, singapore. email: leecc@ntu.edu.sg 132 c.-c. lee et al. current perspectives on the use of organizers in the teaching of writing research on the writing process of poor writers shows that they do not have a sense of priority. thus, they spend little time planning and begin writing although they are still confused about the task (zamel, 1983). they often lack a mental representation of what they are writing as they cannot define their task well (scardamalia & bereiter, 1986a), which leads them to practice knowledge-telling by writing whatever they could recall about the topic without thinking about the structural coherence of their writing or any awareness of audience expectations. they do not have a sense of direction (perl, 1979) when writing; hence, they use the trial-and-error approach to trigger further writing (scardamalia & bereiter, 1986b). they also have very inflexible plans (richards, 1990), and do not possess systematic strategies to help them to generate enough content to write. therefore they are unwilling to discard anything that might ‘fit’ into the topic. finally, they tend to think that their lack of composing competence is due to their limited language resources (richards, 1990), and hence they focus on language issues. graphic organizers have often been used to scaffold students’ writing. table 1 shows how graphic organizers can help poor writers. drawing graphic organizers may help students to plan their writing in the prewriting stage so that they will have a clear sense of direction when they write. graphic organizers thus help students to identify what information they should look for, what information is missing and what information is redundant to their writing via visual representation. in a tree diagram, for example, the order of the ideas can be seen clearly by looking at its location within the hierarchy. recording notes from text and then reviewing them in an organizer reinforces learning as this allows students to access the information or input twice (once when they record the information into the graphic organizers, and the second time when they review the information represented in their graphic organizers to write their essays). this review also encourages students to re-examine their writing goals in the light of their task goal. similarly, the visibility of the graphic organizers allows students to manipulate the ideas in the graphic organizers. students may be prompted to revise an idea that does not fit a cell in a matrix or be encouraged to look for further information due to highlighted gaps in their knowledge. thus, they can get a clear sense of direction before writing begins. flower and hayes (1980) suggest that if students concentrate on language in the initial stages of writing, this may truncate the flow of their thoughts. the use of table 1. graphic organizers to scaffold writing problems in writing benefits of using graphic organizers no sense of priority forced to fill in ideas and link them into an argument lack of mental representation, no sense of direction information is indexed by location, reviewing writing goals while coding information in dual modalities inflexible plans ideas are easily manipulated concerned about language focus on keywords, not sentence structure methodological issues in using sequential representations 133 graphic organizers helps students to concentrate on meaning rather than form (sentences), as only keywords are used in the organizers. students can also learn to write systematically as they translate the ideas from the organizer into their essays. practice in writing with organizers may support students in developing richer procedural knowledge about writing. for example, benton et al. (1993) found that writing notes using a matrix helped students form a coherent overview of the subject, and students who wrote their essays from their notes were more organized and coherent in their writing. they also wrote lengthier essays. multiple representations and learning there are many advantages to using multiple representations. mayer (1997) claims in his multimedia theory that students learn better from pictures and words rather than learning from words alone. two of the assumptions of his multimedia theory explain that this is because working memory is more efficient if information is processed in dual channels as the working memory is limited in its processing capacity. additionally, he proposes, students need to be active participants in the learning process, selecting and organizing information into coherent representations. this is facilitated by students having to recode information from one modality to another. spiro et al.’s (1992) cognitive flexibility theory also supports the use of multiple representations to provide students with situation-sensitive knowledge that would prepare them for problem-solving transfer. additionally, multiple representations prompt students to compare and contrast the various situations covered so that they avoid reductive bias (premature conclusions based on prior knowledge). mayer (2001), however, highlights that there are competitive effects in the use of multiple representations in his instructional design principles for multimedia. he finds that multiple representations distract students from their focus of attention when there is too much redundant or extraneous information. for example, mayer and moreno’s (1998) study found that students performed worse in the animation (visual), narration (audio) and on-screen text (visual) group than the animation (visual) and narration (audio) group. although the information in the visual channel was the same (and thus redundant) but in different forms, students still processed both, which added to their cognitive load. similarly, moreno and mayer (2000) found that students who received the narration, background music and sounds treatment fared the worst. adding environmental sounds did not interfere with students’ understanding of the lightning process being narrated because the stimuli were relevant and coordinated with the narration. however, the background music interfered with students’ comprehension of what was being learnt because it was not relevant to their understanding. mayer’s modality, redundancy and coherence principles are pro-active steps that the teacher needs to take in structuring the task. however, they do not provide enough insights into the use of representations when they are generated by students for sequential processes within a task in a classroom context. ainsworth’s (2006) conceptual framework for learning with multiple representations, however, expands 134 c.-c. lee et al. on mayer’s principles, by recognizing the different functions of representations and the cognitive tasks that students engage in when using them. her framework has three components: design, function and task. the design parameters are number, form, information, sequence and translation. ainsworth (2006) suggests that the number of representations should be sufficient for the task and not overwhelming. this is because the cognitive tasks involved in using the representations are complex. in addition, students have to be familiar with the form of the representations to be able to select and construct them correctly. furthermore, how the representations are sequenced (concurrently or sequentially, and in what order) and the way the information is distributed between the representations may not be clear to the students, thus leaving students confused about how to translate (coordinate and integrate information) between representations. she also recommends that we should first identify the function of the representations to see whether they are complementary, constraining or constructing when providing support for processing. she explains that recognizing the functions of the representations is important as it determines whether the representations have to be co-present when information is processed. if the representations have to be copresent, this would involve a higher cognitive load. the multiple representations complement each other when they differ in terms of the processes each supports or the information contained. the constraining function happens in two ways: when a familiar representation is introduced prior to an unfamiliar one so that students can learn by analogy, and by explicit illustration. finally, multiple representations have the constructing function when learners integrate information from the various representations introduced to construct new knowledge that would be difficult to achieve with only one representation. ainsworth (2006) also highlights the complexity of the cognitive tasks students perform when using multiple representations, and thus advocates that students be trained to understand, select and construct representations appropriate to the task. the next section discusses how ainsworth’s model of design, function and cognitive tasks interplay with each other in the context of using multiple graphic organizers in process writing. methodological issues in using sequential graphic organizers in writing process writing requires students to write multiple drafts based on feedback from their peers or teacher. the writing process could also be divided into three stages: categorizing and extracting, integrating information and drawing conclusions. thus, for each stage, students are to write their drafts, comment on each others’ drafts, and revise their drafts based on their peers’ feedback and, finally, based on their teacher’s feedback. peer feedback helps to provide students with a mental model of readers (berg, 1999), multiple perspectives of what is being learnt (van boxtel et al., 2000) and expand their understanding (witbeck, 1976) so that students are clearer about any incongruity between their ideas and how their ideas are perceived by their audience. in addition, feedback by the teacher helps to provide students with the motivation to methodological issues in using sequential representations 135 revise in a multi-draft setting (jacobs et al., 1998).what are the issues involved if these textual drafts are replaced by ‘organizer’ drafts? number and sequence in process writing, the organizers have a complementary function. thus, the organizers are generated, commented and revised sequentially according to the purposes of each stage of the writing process. this creates the split attention effect. to overcome this problem, pollock et al. (2002) suggest that teachers should first isolate the interacting elements and then introduce them simultaneously in order to lessen students’ cognitive load. this is because as long as students do not have a basic schema in long-term memory, learning will be impeded because of the high cognitive load. in a class with students of mixed abilities in writing, the implication may mean that students should not be allowed to write their essays until they have their ideas represented in the organizers coherently. sweller (2004), however, cautions that this strategy may have the expertise reversal effect on expert writers because if a ‘knowledge-based central executive is already available, an instruction-based version will be redundant and may interfere with additional learning’ (p. 25). seufert’s (2003) results also show that students with high prior knowledge do not seem to benefit from instruction using multiple representations. in fact, instruction interferes with their mental model construction. mayer (1997) also agrees that students with high prior knowledge seem to be able to construct a mental model of the content studied just from text, without the aid of visuals. thus it may be better then to encourage students with high prior knowledge to use organizers with the constructing function, while students with low prior knowledge should be introduced to organizers with constraining and complementary functions to support their learning. an alternative would be for teachers to consider giving students the choice and flexibility in the number of organizers that need to be generated, commented on and revised. this means that teachers have to be selective as to which process they want to make compulsory for students to enjoy the benefits of process writing. it is suggested that support for processing should be focused on the initial stages of writing; namely, the categorizing and extracting information process rather than the later stages. this is in line with ferris’s (1997) finding that students are most likely to revise in the early stages of writing compared with later stages. form schnotz and bannert (2003) find that inappropriate selection of representations may impede students’ mental model construction. thus, training is important to enable students to understand, select and construct the organizers appropriate for the task. winn (1993) states that students’ knowledge of the content of the diagram lets students anticipate what to look for next and where to look for it. 136 c.-c. lee et al. griffin et al. (1995) also agree that training is important in helping students optimize the features of the organizers. their results show that, in the absence of explicit instructions, the performance of the students is similar to rereading the text. this is because students are not able to relate the new information in the text to the existing information in the organizer (kloster & winne, 1989). chmielewski and dansereau (1998) further find that, with training, students are able to recall more macro-level ideas than students not trained in the construction and use of the knowledge map. ainsworth (2006) and cheng et al. (2001) further highlight that the computational offloading, re-representation and constraining properties of the representations should all be considered, since different properties of the representations may have different effects on students’ elaboration. students have to re-represent their ideas from one representation to another for the different stages in writing. in addition, students need to understand the relationship between the constrained and the constraining representation if they are to translate between representations. the different representations also offer different computational offloading properties, thus triggering students to use different strategies when trying to understand them. ainsworth et al. (2002), for example, show that students use perceptual strategies when using pictorial representations but rule-based strategies when using mathematical representations. stark et al. (2002) suggest that elaboration training (ability to draw inferences from worked examples) should be given to help students improve the quality of their example elaboration (in this context, students’ revision). their study shows that students with elaboration training mostly used deep cognitive elaboration, which was also associated with a tolerance for ambiguity, while students without elaboration training used superficial elaboration. information the way information is distributed among the sequential organizers also affects students’ processing. ainsworth et al.’s (2002) study found that students have difficulties translating between representations. students in the pictorial and mathematical condition (mixed) did not improve as compared with those given only one type of representation. when the study was repeated, the mixed group improved as they learnt to focus on the representation that contained all the relevant information and ignore the other representation. this may mean that metacognitive instruction is important to raise students’ awareness about the different purposes of each stage in the writing process in the early stages of instruction. sweller (2004) also agrees that explanations should be given in the early stages. this will enable students to concentrate on generating, commenting and revising the content when using the organizers. translation students have difficulties translating between representations because of the form of the representations and their processing strategies. lowe (2003) reports that students methodological issues in using sequential representations 137 have difficulties identifying conceptually significant information in static graphics as they are not salient. lewalter’s (2003) study further shows that rehearsal strategies (memorizing and recapitulation) are the most frequently used strategies when students are studying static and dynamic visuals. students very rarely use elaboration strategies (building connections between new information and prior knowledge) and control strategies (strategies to plan and regulate learning). this result is similar to paulus’ (1999) finding that students often engage in superficial revisions; namely, paraphrasing or rewording rather than meaning-level revisions. mayer and sims’s (1994) study also found that students with different spatial abilities process information differently. their results show that students with high spatial ability are able to devote more cognitive resources to building referential connections (interacting connections) between visual and verbal representation, while students with low spatial ability devote more cognitive resources to building representation connections between visually presented material and its visual representation. thus, teachers should train students in giving peer review and revising following faigley and witte’s (1981) taxonomy of revision changes in text and stark et al.’s (2002) coding system for elaboration behavior, so that they can see the type of comments on which they have focused (figure 1) and, on the part of the writers, the type of revisions in which they have engaged (figure 2). this is greatly facilitated by organizers that offer computational offloading, re-representation and constraining properties which are hierarchical in nature (have a heading and sub-heading structure) as they help students see the focus of their changes. figure 1. sample showing peer comments coded using faigley and witte’s (1981) and stark et al.’s (2002) categories. note: pc refers to peer comments. categorizations in parentheses refer to type of commentsfigure 2. sample showing a student’s revisions based on peer comments. note: rp refers to revision based on peers’ comments, and tc refers to teacher comments. categorizations in parentheses refer to type of revision made the following explanation presents an adaptation of faigley and witte’s (1981) taxonomy for use in analyzing graphic organizers. surface comments in the organizers figure 1. sample showing peer comments coded using faigley and witte’s (1981) and stark et al.’s (2002) categories. note: pc refers to peer comments. categorizations in parentheses refer to type of comments 138 c.-c. lee et al. consist of formal comments affecting the spelling of words. text-base comments are based on the location of the idea being commented on. if the comment concerns a main idea, it is considered a macro comment. addition macro comments add a main idea into the organizer, while deletion macro comments delete a main idea from the organizer. substitution macro comments refer to comments where the peer deletes the original main idea and substitutes another main idea in the same location. distribution macro comments refer to instances where the peer may want to change a main idea into a sub-idea of another main idea. consolidation macro comments, on the other hand, refer to instances where the peer combines two main ideas to form a new main idea. stark et al.’s (2002) coding system includes goal-operator comments and monitoring elaboration. goal-operator comments occur when students are out-of-topic or when their peers feel that the students’ ideas do not meet the audience’s needs and purpose of the writing task. positive monitoring comments occur when the peers affirm that they understand what the students have written as it is presented well. negative monitoring comments occur when the peers mention that they do not understand what the students are trying to present. this may be because the organizers are too brief. the same categorizations apply to micro-comments except that it is at the sub-idea level. students’ revisions can be similarly analyzed with the addition of one more category; that is, to ignore their peers’ or teacher’s comment at the macro or micro level of ideas. table 2 outlines the proposed instructional approaches suitable for classes with students of different writing abilities in the order they should be introduced in the classroom as gleaned from the discussion above. figure 2. sample showing a student’s revisions based on peer comments. note: rp refers to revision based on peers’ comments, and tc refers to teacher comments. categorizations in parentheses refer to type of revision made methodological issues in using sequential representations 139 implications four main implications arise from the above approaches for designers of concept mapping tools. firstly, visual mapping software should have multiple-entry levels for users with different writing abilities. good writers should have the option to map the ideas for the whole essay in one screen while poor writers should have the option to map their ideas one stage at a time. a further option should be provided where students can choose to view ideas at certain hierarchies (e.g. top level and sub-idea level 1 for all the main points simultaneously to check for flow) or just focus on one main idea with all its sub-ideas for further elaboration. secondly, a small comment box could be added to remind users of the purpose of each stage and how information is distributed among the different representations. this would help writers keep a mental blueprint of what they are doing and not be lost in the details. thirdly, as different organizers offer different computational offloading, re-representation and constraining properties and students do have preferences regarding the organizers used (lee, 2005), students need to be trained to select and construct appropriate organizers. finally, training in peer review and example elaboration also needs to take into account the ‘inflexibility’ and time taken to map ideas using software compared with mapping ideas using pen and paper. in order to offset these disadvantages, it would be advantageous if mapping software could provide a ‘track changes’ view that could be enabled or disabled according to the students’ choice. this view would label table 2. instructional approaches for classes according to students’ writing ability approach class with good writers class with good and poor writers class with poor writers simultaneous introduction of different elements yes no (isolate-interacting approach) no (isolate-interacting approach) awareness of the purpose of each stage of the writing process yes yes yes awareness of how information is distributed among the different representations for each stage yes yes yes number of stages required for feedback less (only the first stage) compulsory for the first stage and optional for the later stages all the stages training in selection and construction of the organizers yes yes yes training in peer review yes yes yes training in example elaboration optional yes yes time to complete the assignment less time more time more time, with less assignments 140 c.-c. lee et al. the changes in terms of micro and macro changes; and also types of changes according to faigley and witte’s (1981) taxonomy of revision changes to motivate students to do meaning-level revisions rather than just paraphrasing ideas. conclusion this paper has considered a specific application of ainsworth’s framework for the use of multiple representations in writing, by looking at how the knowledge of students’ decisions with regards to the usage of graphic organizers generated sequentially inform the teaching of writing. from the literature review, we have proposed instructional approaches for the use of multiple graphic organizers for classes with students of different writing abilities. extended implications for designers of concept mapping tools based on these approaches have also been provided. it is hoped that these insights will provide for further development of ainsworth’s framework for use in other contexts. references ainsworth, s. (2006) deft: a conceptual framework for considering learning with multiple representations, learning and instruction, 16, 183–189. ainsworth, s., bibby, p. & wood, d. (2002) examining the effects of different multiple representational systems in learning primary mathematics, the journal of the learning sciences, 11(1), 25–61. benton, s. l., kiewra, k. a., whitfill, j. m. & dennison, r. (1993) encoding and externalstorage effects on writing processes, journal of educational psychology, 85(2), 267–280. berg, e. c. (1999) the effects of trained peer response on esl students’ revision types and writing quality, journal of second language writing, 8(3), 215–241. cheng, p. c. h., lowe, r. k. & scaife, m. (2001) cognitive science approaches to understanding diagrammatic representations, artificial intelligence review, 15, 79–94. chmielewski, t. l. & dansereau, d. f. (1998) enhancing the recall of text: knowledge mapping training promotes implicit transfer, journal of educational psychology, 90(3), 407–413. faigley, l. & witte, s. (1981) analyzing revision, college composition and communication, 32, 400–414. ferris, d. (1997) the influence of teacher commentary on student revision, tesol quarterly, 31(2), 315–339. flower, l. s. & hayes, j. r. (1980) the dynamics of composing: making plans and juggling constraints, in: l. w. gregg & r. s. erwin (eds) cognitive processes in writing (hillsdale, nj lawrence erlbaum associates). griffin, c. c., malone, l. d. & kammenui, e. j. (1995) effects of graphic organizer instruction on fifth-grade students, journal of educational research, 89(2), 98–107. jacobs, g. m., curtis, a., braine, g. & huang, s. y. (1998) feedback on student writing: taking the middle path, journal of second language writing, 7(3), 307–317. kloster, a. m. & winne, p. h. (1989) the effects of different types of organizers on students’ learning from text, journal of educational psychology, 81(1), 9–15. lee, c. c. (2005) students’ choice of organizers based on their perceptual operations, paper presented at the e-learn conference, vancouver, canada, 24–28 october. lewalter, d. (2003) cognitive strategies for learning from static and dynamic visuals, learning and instruction, 13(2), 177–189. methodological issues in using sequential representations 141 lowe, r. k. (2003) animation and learning: selective processing of information in dynamic graphics, learning and instruction, 13(2), 157–176. mayer, r. e. (1997) multimedia learning: are we asking the right questions?, educational psychologist, 32, 1–9. mayer, r. e. (2001) multimedia learning (cambridge, cambridge university press). mayer, r. e. & moreno, r. (1998) a split-attention effect in multimedia learning: evidence for dual processing systems in working memory, journal of educational psychology, 90, 312–320. mayer, r. e. & sims, v. k. (1994) for whom is a picture worth a thousand words? extensions of a dual-coding theory of multimedia learning, journal of educational psychology, 86, 389–401. moreno, r. & mayer, r. e. (2000) a learner-centered approach to multimedia explanations: deriving instructional design principles from cognitive theories, interactive multimedia electronic journal of computer-enhanced learning, 2(2). available online at: http://imej.wfu. edu/articles/2000/2/05/index.asp (accessed 24 october 2005). paulus, t. m. (1999) the effect of peer and teacher feedback on student writing, journal of second language writing, 8(3), 265–289. perl, s. (1979) the composing processes of unskilled college writers, research in the teaching of english, 13(4), 317–336. pollock, e., chandler, p. & sweller, j. (2002) assimilating complex information, learning and instruction, 12, 61–86. richards, j. c. (1990) the language teaching matrix (cambridge, cambridge university press). scardamalia, m. & bereiter, c. (1986a) writing, in: r. f. dillon & r. j. sternberg (eds) cognition and instruction (orlando, fl, academic press inc.). scardamalia, m. & bereiter, c. (1986b) research on written composition, in: m. c. wittrock (ed) handbook of research on teaching (3rd edn) (new york, american educational research association). schnotz, w. & bannert, m. (2003) construction and interference in learning from multiple representation, learning and instruction, 13(2), 141–156. seufert, t. (2003) supporting coherence formation in learning from multiple representations, learning and instruction, 13, 227–237. spiro, r. j., feltovitch, p. j., jacobson, m. j. & coulson, r. l. (1992) knowledge representation, content specification and the development of skill in situation-specific knowledge assembly: some constructivist issues as they relate to cognitive flexibility theory and hypertext, in: m. p. driscoll (ed.) psychology of learning for instruction (boston, ma, allyn and bacon). stark, r., mandl, h., gruber, h. & renkl, a. (2002) conditions and effects of example elaboration, learning and instruction, 12(1), 39–60. sweller, j. (2004) instructional design consequences of an analogy between evolution by natural selection and human cognitive architecture, instructional science, 32, 9–31. van boxtel, c., linden, j. & kanselaar, g. (2000) collaborative learning tasks and the elaboration of conceptual knowledge, learning and instruction, 10(4), 311–330. winn, w. (1993) an account of how readers search for information in diagrams, contemporary educational psychology, 18, 162–185. witbeck, m. c. (1976) peer correction procedures for intermediate and advanced esl composition lessons, tesol quarterly, 10(3), 321–326. zamel, v. (1983) the composing processes of advanced esl students: six case studies, tesol quarterly, 17(2), 165–187. alt-j, research in learning technology vol. 16, no. 3, september 2008, 231–244 issn 0968-7769 print/issn 1741-1629 online © 2008 association for learning technology doi: 10.1080/09687760802526715 http://www.informaworld.com an investigation of the role of background music in ivws for learning debbie richardsa*, eric fassbendera, ayse bilginb and william forde thompsonc adepartment of computing, macquarie university, sydney, australia; bdepartment of statistics, macquarie university, sydney, australia; cdepartment of psychology, macquarie university, sydney, australia taylor and franciscalt_a_352839.sgm (received 31 march 2008; final version received 8 september 2008) 10.1080/09687760802526715alt-j, research in learning technology0968-7769 (print)/1741-1629 (online)original article2008taylor & francis163000000september 2008debbierichardsrichards@ics.mq.edu.au empirical evidence is needed to corroborate the intuitions of gamers and game developers in understanding the benefits of immersive virtual worlds (ivws) as a learning environment and the role that music plays within these environments. we report an investigation to determine if background music of the genre typically found in computer-based role-playing games has an effect on learning in a computer-animated history lesson about the macquarie lighthouse within an ivw. in experiment 1, music stimuli were created from four different computer game soundtracks. seventy-two undergraduate students watched the presentation and completed a survey including biographical details, questions on the historical material presented and questions relating to their perceived level of immersion. while the tempo and pitch of the music was unrelated to learning, music conditions resulted in a higher number of accurately remembered facts than the no music condition. one soundtrack showed a statistically significant improvement in memorisation of facts over other music conditions. also an interaction between the levels of perceived immersion and ability to accurately remember facts was observed. experiment 2, involving 48 undergraduate students, further investigated the effect of music, sense of immersion and how different display systems affect memory for facts. keywords: immersive virtual worlds; learning; music introduction technology does not offer an automatic solution to the various problems experienced in education. from an economic point of view it is often difficult to find any evidence of the benefits of technology (landauer 1995). however, when there is “a good fit between a particular learning situation and specific technical solution” striking positive results can be found (draper, cargill, and cutts 2002, 16). the benefits of using computer games which allow students (children in these particular cases) to learn as they play are demonstrated in games like ‘kahootz’ (http://www.kahootz.com/), ‘where in the world is carmen san diego?’ (http://www.learningcompany.com/) and ‘zoombinies’ (http://www.learningcompany.com/ ) which require and train problem-solving abilities. going a step beyond such computer games are immersive virtual worlds (ivws) which allow students to engage in experiential learning through role-playing, and has been recognised as a useful means of learning new skills, practicing problem-solving and gaining experience. mason and mitroff state that “stories, drama, role-plays, art, graphics, one-to-one contact and group discussions may be more effective in some information contexts [… over] the language of abstract symbols and ‘hard’ data” *corresponding author. email: richards@ics.mq.edu.au 232 d. richards et al. (1973, 484). ivws offer a means to facilitate distributed learning that is not restricted to a specific physical location or time. further, the costs involved in bringing people together and setting up a convenient and simulated or real environment are reduced. in many cases, such as training fire fighters to put out a house fire (for example, st. julian and shaw 2003), an ivw makes possible something that is prohibitively expensive, dangerous or undesirable in the natural world. to date, the ivw field has been driven by the entertainment industry rather than educationalists. the result has been that while examples of successful and useful systems can be found, there is often little understanding and usually no theory behind whether or why something does or does not assist learning. much of the use of games and ivws in education is based on assumptions such as that interactivity will produce better learning and that immersion will produce greater engagement and lead to better learning. but the evidence for such assumptions is not clear. for example, in our earlier experiments we found that interactivity does not always lead to improved learning (richards 2006). despite the lack of theory and empirical evidence for the value or role that music plays (a discussion of relevant issues can be found in zehnder and lipscomb 2006), we nevertheless find music within most, if not all, game environments. we can conclude that game developers intuitively believe that music is a desirable feature even if its effects are not understood. in this paper we report our investigations of music within an ivw and its effect on learning while considering what others have found in related areas. literature review one of the most well-known studies concerning music and intelligence produced the socalled ‘mozart effect’ (rauscher, shaw, and ky 1993). this study investigated the effect of listening to classical music before subjects performed a spatial reasoning test. they found that 36 college students who listened to 10 minutes of mozart’s sonata for two pianos in d major, k448, scored 8–9 points higher in a subsequent spatial ability iq test, compared to when they listened to a relaxation tape or to no audio stimulus at all. the results have been heavily discussed in the literature and other researchers have tried to replicate the results, some with success (rideout and dougherty 1998) and others not (mckelvie and low 2002). the current view is that such effects of music on spatial-temporal processing are related to temporary changes in mood and arousal levels that arise from music listening (husain, thompson, and schellenberg 2002; thompson, schellenberg, and husain 2001). along these lines, bulgarian psychologist lozanov (1992) recommended several specific musical pieces that make listeners open to suggestion and put the listener into a state of mind that is beneficial for learning foreign languages (lozanov 1988). a somewhat different study was conducted by mcfarland and kennison (1988) who found that subjects who had to solve a tactual maze task with either their right or left hand performed badly when they were listening to music with the ear that was on the same side of the body as the hand they were performing the task with, due to ‘intrahemispheric competition’. typical of many music studies, under some circumstances music improved performance and at other times worsened performance. another study linking music and learning at the psycho-physiological level was conducted by dryden and vos (2001) who found that for learning, the ‘relaxed alertness’ state of mind known as the alpha state was the most beneficial. much of the research on the role of music on learning attributes an indirect link between the two. for example, vester (1975) says that we learn and remember more, the more senses are included in the learning process. we could thus conclude that when we add music to alt-j, research in learning technology 233 visual and/or other aural stimuli, greater learning is possible. davies states that “music’s power to evoke emotions also enhances learning”, and goes on to say that “heightened involvement creates a stronger neural connection, which in turn makes it easier to remember information” (2000, 149). we believe that this is even truer if the visual and the aural senses are combined. in our earlier work (fassbender and heiden 2006), we found evidence for the benefit of the inclusion of the visual sense on learning. our study included ‘the virtual memory palace’, a 3d environment which offers the possibility to freely explore and utilise a virtual three-dimensional architectural representation of an ancient greek (and formerly purely mental) memory technique. two areas of research particularly relevant to our study which we discuss further in the next section, are the work by boltz, schulkind, and kantra (1991) on mood congruence in film music, and the work by thompson and colleagues on the role of pitch and tempo on mood (thompson, schellenberg, and husain 2001; husain, thompson, and schellenberg 2002). boltz, schulkind, and kantra (1991) investigated the effects of background music on the remembering of filmed events and found that participants remembered visual information (i.e. movie scenes) better if such information was accompanied by mood-congruent music. further experiments showed that mood-congruent pairings of video and audio are jointly encoded into the cognitive system (boltz 2004). in their experiments, thompson, schellenberg, and husain (2001) found that participants performed significantly better in an associated paper-folding and cutting task when they listened to a ‘pleasant and energetic’ mozart piece in major key as opposed to a second group who listened to a ‘slow, sad’ albioni piece in minor key. they see that the ‘mozart effect’ is more likely to be a result of mood and arousal, and conclude that: it is possible, then, that the mozart effect has little to do with mozart in particular or with music in general. rather, it may represent an example of enhanced performance caused by manipulation of arousal or mood. (thompson, schellenberg, and husain 2001, 248) research question looking at the literature, there seem to be contradictory opinions and inconclusive evidence about the effect of music on learning. in our investigations we aim to explore whether music increases learning in an ivw. our particular focus is on memorisation of facts. our goal is to contribute insights and rigour to the study and practice of pedagogy via ivws and provide a basis for theory development in the field. to this end, we have designed an ivw which serves a dual function: to provide a learning environment for the history of the macquarie lighthouse as well as an experimental environment within which our investigations can be made. it is commonly observed that players frequently become completely occupied within an ivw and their attention is so intensely focused on the fantasy world in which they are roaming about, that almost nothing can disturb them and get them out of these worlds and back into reality. one explanation for this sense of engagement could be csikszentmihalyi’s ‘flow’ theory (1990) which involves the “feeling of complete and energized focus in an activity, with a high level of enjoyment and fulfilment” (chen 2007, 31). another explanation could be that “video game play is an activity which lies in the domain of intrinsic motivation” (holt 2000, 9). furthermore, music seems to have an effect on motivation. in an unpublished doctoral thesis by weisskoff (1981) “students who received the music condition scored significantly higher with regard to continuing motivation” (cited in eady 234 d. richards et al. and wilson 2004, 243) when reporting about a study of children receiving language lessons – with or without background music. from our experience and observation, and in line with zehnder and lipscomb (2006), we believe that music contributes to the sense of total immersion within an ivw. consequently, the aim of our research is to examine effects of music on memory for events and facts that are learned during game experiences and to explore the significance of music for feelings of immersion in game-like virtual-immersive environments. in our study we consider the potential effects of background music on the perception and memory of events and facts experienced in an ivw. as music is multidimensional, in our initial experiments we have narrowed our focus to the effect of pitch and tempo of music based on their powerful effects on emotion and mood (thompson, schellenberg, and husain 2001; ilie and thompson 2006; boltz 2004; boltz, schulkind, and kantra 1991). consequently, our research question asks how changes in pitch and tempo of soundtracks of computer-based role-playing games affect the memory for the historical facts delivered by an avatar (virtual persona) in the ivw. the macquarie lighthouse history lesson ivw as a precursor to answering the above research question it was necessary to create an ivw. to provide such a learning environment we chose to create a computer-animated history lesson presenting the history of the macquarie lighthouse, the first lighthouse of australia and the southern hemisphere. despite this importance to australian settlement, the history of the macquarie lighthouse is not widely known and is thus an ideal topic for evaluating learning of historical facts. since the effect of interactivity on learning was not part of this study, we created a non-interactive computer-animated history lesson to minimise the number of variables and provide greater control. even though the course was non-interactive, it was created to look and feel like a typical conversation with an avatar in a fully featured computer role-playing game and ivw. during the computer-animated history lesson, numerous facts about the history of the lighthouse, including dates, names of people, ships, equipment and places are presented. twenty-nine questions were asked of participants following the presentation to determine how well they recalled the information given. to create the macquarie lighthouse ivw, a game mod (the technical term used to refer to a modification of the original application) was developed with ‘the elder scrolls construction set’ (tescs) (http://www.elderscrolls.com/downloads/updates_utilities.htm). tescs is an expansion for the game ‘elder scrolls 4 – oblivion’ (bethesda softworks: http:/ /www.elderscrolls.com/home/home.htm) which allows the creation of complete virtual scenarios and supplies the developer with the necessary tools to design a virtual world in his or her own style. in the first step, the landscape surrounding the macquarie lighthouse was modelled in tescs. to achieve a high level of authenticity, the landscape was modelled by means of an accurate land survey supplied by the sydney harbour federation trust. afterwards, a 3d model of the macquarie lighthouse was created with 3d studio max (http:// www.autodesk.com) and then imported into tescs (figure 1). the avatar in the foreground is a fictitious descendant of the first lighthouse keeper, and delivers the history of the lighthouse to the users. figure 1. a 3d model of the macquarie lighthouse together with a virtual avatar who is explaining the history of the lighthouse to the users. six different versions of this history lesson were produced using different background musical stimuli, which will be explained in more detail later. the computer-animated history lesson was displayed by means of a cave-like display system called a ‘cone’ to increase the level of visual immersion. figure 2 shows a pilot-tester who is watching the video narration and listening to the avatar. alt-j, research in learning technology 235 figure 2. a pilot-tester watching and listening to the 3d narration about the history of the macquarie lighthouse. the cone-display system consists of three projectors which display the virtual world onto a semi-cylindrical screen canvas. the user is positioned slightly off centre towards the canvas to allow a 160° field of view (fov) which simulates almost the maximum of 180° of the natural human fov. through this setup the virtual reality occupies most of the user’s visual sense and the user gets the impression that the world displayed on the screen canvas is almost real; he or she feels immersed in this virtual reality. figure 2. a pilot-tester watching and listening to the 3d narration about the history of the macquarie lighthouse. figure 1. a 3d model of the macquarie lighthouse together with a virtual avatar who is explaining the history of the lighthouse to the users. 236 d. richards et al. methodology – stimuli and measures in order to gain a somewhat representative sample of the computer role-playing game genre, the soundtracks from four contemporary computer role-playing games, namely ‘oblivion’ (http://www.elderscrolls.com), ‘baldur’s gate’ (http://www.bioware.com/games/baldurs_ gate/), ‘world of warcraft’ (http://www.worldofwarcraft.com) and ‘icewind dale’ (http:// www.blackisle.com/ or http://www.mobygames.com/game/icewind-dale), were selected for manipulation. from each soundtrack of each game three musical pieces were pre-selected. these three pre-selected pieces were then presented to a group of six pilot-testers who were recruited from the department of computing at macquarie university. all of the pilottesters have extensive experience with computer role-playing games and the associated soundtracks. the pilot-testers were asked to rank the musical pieces on a scale from 1 to 5 as being: (a) not representative (1) or representative (5) for the role-playing game genre; and (b) not suitable (1) or suitable (5) for the 3d animated video presentation. the musical pieces that scored the highest overall value in both categories were then selected for further experiments. tempo and pitch manipulations as discussed earlier, two musical attributes that have powerful effects on emotion and mood (which in consequence might have an effect on learning and immersion) are pitch and tempo (husain, thompson, and schellenberg 2002; thompson, schellenberg, and husain 2001). four musical pieces (oblivion, baldur’s gate, world of warcraft, icewind dale) were selected and manipulated in tempo and pitch, based on work by ilie and thompson (2006). tempo was reduced/increased by 21%/26% and pitch was lowered/increased by two semitones. the manipulations were accomplished using a macbook pro in ableton live 6.0.7. ableton live is a professional daw (digital audio workstation) software, which allows convenient and high quality tempo and pitch manipulations. resulting pieces were then presented to three music experts from the department of contemporary music studies at macquarie university. none of the experts found irritating distortions or could identify the manipulations as such. the three possible tempos (slow, medium/original, fast) and three possible pitches (low, medium/original and high) resulted in nine possible combinations. for reasons of practicality (size of participant pool, available laboratory time, etc.) it was decided to test only five of the nine possibilities: condition 1: s/l – slow tempo/low pitch; condition 3: f/l – fast tempo/low pitch; condition 5: m/m – medium tempo/ medium pitch (tempo and pitch unaltered); condition 7: s/h – slow tempo/high pitch; condition 9: f/h – fast tempo/high pitch. each of the five manipulations was performed on the four different musical pieces, indicating that all musical pieces received the same tempo/pitch manipulation. a control group (condition 10) was assigned to the ‘no music’ condition and during the 3d history course they only listened to the voice of the avatar without any musical background. experiment 1 a total of 72 undergraduate students (45 female, 27 male) from 19 to 56 years old (mean 24.2 years of age) were recruited by advertisement on campus and from introductory statistics and computing classes at macquarie university, sydney. they were allocated into six experimental conditions; therefore each experimental condition had 12 participants. fifteen participants indicated english as their first language while 57 participants answered that english was their ‘second or other’ language. alt-j, research in learning technology 237 the experiment consisted of three stages: ● stage 1 – a short pre-experiment biographical questionnaire (e.g. age, gender, first language, etc.). ● stage 2 – an 11:38 minute video narration of the history course. participants sat in the centre of the display system where they watched and listened to the stimuli wearing high quality noise-cancelling stereo headphones. ● stage 3 – a post-experiment set of questionnaires: ● 29 questions about the history of the macquarie lighthouse as covered in the video narration multiple-choice questionnaire about facts that they just learned in the history lesson (e.g., what was the name of the first lighthouse keeper, in which year was the first lighthouse built, how many people were stationed at the lighthouse, etc.). ● 27 questions about the immersiveness of the video narration (e.g., how well were you concentrating, how much did you lose track of time). ● 6 questions about participants’ prior experience with music and computer/ console games (e.g., how many years have you been playing a musical instrument, how many years have you been playing computer/console games). results for experiment 1 we consider the results regarding the effect of music, levels of concentration and the sense of immersion on the remembering of facts in each of the following subsections. results for music figure 3 shows the mean number of correctly answered questions from the content section of the questionnaire for each of the experimental conditions along with the standard deviations for the participants. figure 3. comparison of the mean number of correctly answered questions in the virtual-immersive 3d narration. histograms show the frequencies of correctly answered questions as bars and the normal distribution as a normal curve. descriptive statistics in figure 3 show the mean scores and standard deviations for each of the five categories of musical manipulations and the control group, as well as for each tempo and pitch condition as aggregate. the mean number of correctly answered questions ranged from a minimum of 14.7 questions by the control group without music (experiment condition 10) to a maximum of 17.6 questions by the group under the slow tempo/low pitch condition (experiment condition 1). the magnitudes of the mean scores of the musical conditions are close to each other and the variations are wide, therefore it is hard to detect any statistically significant difference between the means (only a difference of 1.7/29 facts between the highest and lowest number of correctly answered questions). even though the participants who listened to background music during the computer-animated history lesson of the macquarie lighthouse performed better than their peers in the no-music control group, a one-way analysis of variance (anova) showed that the experimental (music) conditions were not a significant main effect for learning (f(5,71) = 0.63, p = 0.68). in other words, we were unable to detect any statistically significant difference between the mean number of correctly answered questions by participants under the six experimental conditions. we also observe from figure 3 that the slow tempo/low pitch participants are the best performers in terms of learning (mean = 17.6). in figure 4 we can see the mean number of questions answered correctly was higher under the low pitch condition compared to the high pitch condition. furthermore, the slow tempo (upper light grey line) had a better effect on learning than the fast tempo (lower dark 238 d. richards et al. grey line), regardless of the pitch. we investigated the effects of tempo and pitch on learning by using two-way analysis of variance (anova). neither tempo nor pitch as main effects had a statistically significant effect on learning (both f(1,45) = 0.44, p = 0.51). the interaction of these effects was also not statistically significant (f(1,44) = 0.04, p = 0.85). figure 4. the mean number of questions answered correctly by tempo and pitch (original soundtrack condition medium tempo/medium pitch is removed). each study participant was exposed to one piece of game music (oblivion, baldur’s gate, world of warcraft, icewind dale), except the participants in the control group (no music – 12 participants). each game music was listened to by 15 participants. we compared the effects of different game music on learning regardless of which tempo and/or pitch was used. we found a significant difference between the ‘oblivion’ soundtrack and the ‘no music’ condition using dunnett’s two-sided t-test (p = 0.034). those participants who listened to the ‘oblivion’ soundtrack, on average, answered 4.4 (95% confidence intervals 0.3–8.5) questions more correctly than their peers in the ‘no music’ (control group) (figure 5). figure 5. the mean number of questions answered correctly while listening to game music or no music (control). results regarding concentration a one-way anova contrast test established that music made it subjectively easier to concentrate (table 1). participants under the ‘slow tempo/low pitch’ condition found it figure 3. comparison of the mean number of correctly answered questions in the virtual-immersive 3d narration. histograms show the frequencies of correctly answered questions as bars and the normal distribution as a normal curve. alt-j, research in learning technology 239 easiest to concentrate. participants in the control group (no music) found it hardest to concentrate. participants who did not listen to music found it harder to concentrate with the biggest difference between slow tempo/low pitch and no music (figure 6). figure 6. ‘how hard was it to concentrate?’ under the influence of different musical background stimuli. results regarding immersion moreover, to investigate the feeling of immersion we asked participants a number of related questions including the degree to which they lost track of time in the ivw. they answered on a 5 point scale from 0 having ‘not at all’ lost track of time and 4 having ‘completely’ figure 4. the mean number of questions answered correctly by tempo and pitch (original soundtrack condition medium tempo/medium pitch is removed). figure 5. the mean number of questions answered correctly while listening to game music or no music (control). 240 d. richards et al. lost track of time. losing track of time was highly correlated with the total number of questions answered correctly (r = 0.37, p = 0.003). this correlation was statistically significant at 0.01 level. the linear regression showed that 35% of variation in the total number of questions answered correctly was explained with the degree of losing track of time (r2 = 0.347). based on the linear regression, we conclude that on average if participants did not lose track of time, they answered 11.8 questions correctly. for each further level of losing track of time (i.e. from 1 to 4 on the scale) the total number of questions answered correctly increased by 1.9. for example, if participants completely lost track of time, then on average they would have answered 19.4 questions correctly, which is 7.6 more questions than those table 1. one-way anova contrast test – music makes easier to concentrate (subjective selfevaluation of participants). contrast coefficients which game music contrast oblivion baldurs gate wow iwd no music 1 .25 .25 .25 .25 −1 contrast tests contrast value of contrast std. error t df sig. (2-tailed) how hard to concentrate assume equal variances 1 −.72 .292 −2.45 67 .017 does not assume equal variances 1 −.72 .250 −2.87 18.74 .010 figure 6. ‘how hard was it to concentrate?’ under the influence of different musical background stimuli. alt-j, research in learning technology 241 participants who did not lose track of time at all. there was no significant relationship between tempo of the background music and losing track of time. experiment 2 one problem that became obvious during analysis of the data from experiment 1 was that the absolute tempo and pitch of the various soundtracks differed quite widely from each other. other problems were the small number of participants in each of the experimental conditions and the ‘between subjects’ design. these limitations were addressed in experiment 2. further, although results were not statistically significant, we were interested in testing the trend in the data showing that the numbers of correctly answered questions are relatively greater with slower tempo and lower pitch. therefore we decided to narrow our focus by using slow tempo and low pitch experimental conditions. the design of experiment 2 can be seen in figure 7. firstly, the experimental conditions and included variables were reduced to one soundtrack (oblivion) and a silent condition. these two conditions were then presented to participants in a ‘within subjects’ design to allow a more direct analysis as opposed to the ‘between subjects’ design of the first experiment. thirdly, to eliminate the problems of differing absolute tempo and pitch, only one version of the tempo and pitch manipulations has been used (slow tempo, low pitch which showed beneficial tendencies). while we recognise that this experiment design means that the results might not be generalisable for the whole genre of music of computer role-playing games, this design allows for a more concise analysis. also, the overall number of participants needed was reduced while increasing the number of participants for each of the new individual conditions. to provide enough statistical power it was decided to include 24 participants in each group, resulting in 48 participants overall. figure 7. experiment design for follow-up experiments focuses on one soundtrack and investigates the effect on memory of one particular soundtrack in a ‘within subjects’ design. group 1 watched the computer-animated history lesson with the oblivion soundtrack (slow tempo/low pitch) in the first half of the lesson and silence in the second half. group 2 watched the two conditions in reverse order (silence first, then oblivion). these parts go for 5:32 (first half) and 5:23 (second half). the slight difference in length is due to a sectional break which could only be made after a sentence was finished by the narratoravatar. while most of the questions from the questionnaire of the first experiment were reused to allow for cross-analysis, a problem in the original experiment was that the number figure 7. experiment design for follow-up experiments focuses on one soundtrack and investigates the effect on memory of one particular soundtrack in a ‘within subjects’ design. 242 d. richards et al. of facts in the two halves was unbalanced. thus, four questions were omitted from the first half and five questions added to the second half. this way there are 15 questions in each half (30 overall). the complexity and type of questions in both halves are similar. furthermore, some qualitative questions and additional questions regarding immersion and presence were included. in this regard, it was decided to also test the effect of different display systems on feelings of immersion. thus, half the participants watched the computeranimated history lesson in the cone display system and the other half was allocated to a 3monitor display system on a computer desk. preliminary results for experiment 2 the results show that the change from a ‘between subjects’ design to a ‘within subjects’ design could not confirm the benefit of the oblivion soundtrack at slow tempo and low pitch. in other words, we did not observe a direct statistically significant difference between the half where participants listened to background music and the half where they listened only to the narrator without background music. however, the different display systems showed a statistically significant difference (p = .032) in favour of the 3-monitor solution. participants in the cone system remembered 14.96 items on average and 17.96 items in the 3-monitor display system. these results then led to closer investigations which revealed a three-way interaction between the display system, the music/no-music condition and the two halves of the history lesson. interestingly, when the cone and 3-monitor display groups were separated and the music condition was then compared with the no-music condition within these groups, it was discovered that those participants in the 3-monitor solution who listened to music in the first half of the computer-animated history lesson remembered significantly more facts from the second half than their peers who listened to no music in the first half. in contrast, those participants in the cone system performed significantly better if they listened to no music in the first half and music in the second half. these results are statistically significant at .027 (3-monitor) and .017 (cone) level. a possible explanation for this result is that participants in the cone system experienced cognitive overload when faced with the overwhelming experience of an unfamiliar display system together with the music. one observation by the experimenters from this second experiment is that participants seemed to feel much more comfortable with the 3-monitor solution than in the cone system. however, analysis and interpretation of these results are ongoing. general discussion in summary, changes in tempo and pitch did not show a statistically significant difference on the amount of correctly answered questions. nor did ‘music’ versus ‘no music’ if all soundtracks were combined in the same (music) category. however, when we looked at each individual soundtrack and compared it to the ‘no music’ control group in the first experiment, we found a statistically significant difference between the oblivion soundtrack and the control group. the statistically significant result for the oblivion soundtrack indicates that it may be beneficial for encoding information into memory. when we consider what distinguishes oblivion from the other samples, we are not able to pinpoint tempo or pitch. one conjecture is that oblivion could be the most congruent piece for the setting of the lighthouse scene. the importance of congruency between music and accompanying media has been the subject of considerable research. properties of music that are congruent with accompanying media act to highlight certain features of that media over others and hence alt-j, research in learning technology 243 can greatly influence how those media are remembered. in advertisements for cars, for example, accompanying classical music can act to highly convey both the vitality and high cultural associations of the car (cohen 2001). as another example, music that exhibits an unambiguous ending may connote the end of a film (thompson, russo, and sinclair 1994). according to the congruence-associationist model outlined by marshall and cohen (1988), sources of congruency act by directing attention to particular aspects of an accompanying film over others (see also bolivar, cohen, and fentress 1994). these effects on attention, in turn, influence interpretations and memory for that material. finally, we found a statistically significant effect between the number of questions answered correctly and the participant’s feeling of immersion in the virtual-immersive environment. those who said that they were more immersed (by saying that they lost track of time) in the 3d history course remembered more facts than those who said that they did not lose track of time. concentration was also positively affected in the music conditions. conclusions the role of background music in games and ivws is a largely unexplored field. previous studies concerning music and learning or intelligence have had mixed results and as in the case of the mozart effect, it is difficult to repeat findings. in our own studies we did find one soundtrack which showed a statistically significant improvement in memorisation of facts. we conjectured that the reason may be that the piece of music was most congruent to the content being taught, but clearly more research needs to be conducted in this area. we also found a link between levels of perceived immersion and ability to accurately remember facts. again the link may not be direct between music, memory and the sense of immersion, but the link is worth investigating further. further analysis of the data from the second experiment is being conducted to this end. it is our goal to take the field of ivw beyond the intuitions of gamers and game developers and to contribute towards unpacking the appeal and benefits of ivws as a learning environment and the role that music may play within that environment. acknowledgements thanks to alan taylor for his contribution, his general interest in this work and particularly for his data analyses related to the experiments. this project is partly funded by australian research council discovery grant coded dp0558852 and macquarie university research infrastructure grant titled ‘virtual reality engine’. the principal author is funded by imurs (international macquarie university research scholarship). references bolivar, v.j., a.j. cohen, and j.c. fentress. 1994. semantic and formal congruency in music and motion pictures: effects on the interpretation of visual action. psychomusicology 13, no. 1: 28–59. boltz, m.g. 2004. the cognitive processing of film and musical soundtracks. memory & cognition 32, no. 7: 1194–205. boltz, m.g., m. schulkind, and s. kantra. 1991. effects of background music on the remembering of filmed events. memory & cognition 19, no. 6: 593–606. chen, j. 2007. flow in games (and everything else). communications of the acm 50, no. 4: 31–4. cohen, a. 2001. music as a source of emotion in film. in music and emotion: theory and research, ed. p. juslin and j. sloboda, 249–72. oxford: oxford university press. csikszentmihalyi, m. 1990. flow: the psychology of optimal experience. new york: harper & row. 244 d. richards et al. davies, m.a. 2000. learning … the beat goes on. childhood education 76, no. 3: 148. draper, s.w., j. cargill, and q. cutts. 2002. electronically enhanced classroom interaction. australian journal of educational technology 18, no. 1: 13–23. dryden, g., and j. vos. 2001. the learning revolution to change the way the world learns. stafford: network educational press ltd. eady, i., and j.d. wilson. 2004. the influence of music on core learning. education 125, no. 2: 243–8. fassbender, e., and w. heiden. 2006. the virtual memory palace. journal of computational information systems 2, no. 1: 457–64. holt, r. 2000. examining video game immersion as a flow state. b.a. thesis, department of psychology, brock university, st. catharines, ontario, canada. husain, g., w.f. thompson, and e.g. schellenberg. 2002. effects of musical tempo and mode on arousal, mood, and spatial abilities: re-examination of the ‘mozart effect’. music perception 20, no. 20: 151–71. ilie, g., and w.f. thompson. 2006. a comparison of acoustic cues in music and speech for three dimensions of affect. music perception 23, no. 4: 319–29. landauer, d. 1995. the trouble with computers: usefulness, usability, and productivity. cambridge, ma: mit press. lozanov, g. 1988. the foreign language teacher’s suggestopedic manual. new york: gordon and breach science publishers. ———. 1992. suggestology and outlines of suggestopedy. philadelphia: gordon and breach science publishers. marshall, s.k., and a.j. cohen. 1988. effects of musical soundtracks on attitudes toward animated geometric figures. music perception 6: 95–112. mason, r.o., and i.i. mittroff. 1973. a program for research on management information systems. management science 19, no. 5: 475–87. mcfarland, r.a., and r.f. kennison. 1988. asymmetrical effects of music upon spatial-sequential learning. journal of general psychology 115, no. 3: 263. mckelvie, p., and j. low. 2002. listening to mozart does not improve children’s spatial ability: final curtains for the mozart effect. british journal of developmental psychology 20, no. 2: 241. rauscher, f.h., g.l. shaw, and k.n. ky. 1993. music and spatial task performance. nature 365, no. 6447: 611. richards, d. 2006. is interactivity actually important? proceedings of the third australasian conference on interactive entertainment (ie’2006). perth: murdoch university. rideout, b.e., and s. dougherty. 1998. effect of music on spatial performance: a test of generality. perceptual & motor skills 86, no. 2: 512. st. julien, t.u., and c.d. shaw. 2003. firefighter command training virtual environment. in proceedings of the 2003 conference on diversity in computing, atlanta, georgia, usa, october 15-18, 2003, tapia ‘03, 30–33. new york: acm. thompson, w.f., f.a. russo, and d. sinclair. 1994. effects of underscoring on the perception of closure in filmed events. psychomusicology 13: 9–27. thompson, w.f., e.g. schellenberg, and g. husain. 2001. arousal, mood, and the mozart effect. psychological science 12, no. 3: 248–51. vester, f. 1975. denken , lernen, vergessen : was geht in unserem kopf vor, wie lernt das gehirn, und wann lässt es uns im stich? stuttgart: deutsche verlags-anstalt. weisskoff, r.s. 1981. the relationship of pop/rock music to children’s task performance and continuing motivation in language arts instruction. hartford, ct: university of connecticut. zehnder, s.m., and s.d. lipscomb. 2006. the role of music in video games. in playing video games: motives, responses, and consequences, ed. p. vorderer and j. bryant, 241–7. mawwah, nj: lawrence erlbaum associates publishers. calt586674 155..172 the chicken or the egg? investigating the transformational impact of learning technology janet f. buchan* division of learning and teaching services, charles sturt university, australia (received 30 january 2010; final version received 29 april 2011) this study aimed to investigate the transformational impact of introducing significant new learning technology in an australian university over the time period 2007–2009. the exploration of this transformation is grounded in a social–ecological systems approach to the management of technology enhanced learning environments in the face of constant change. the transformational impact is described using the adaptive cycle framework. the single case study had a whole-of-institution systems focus. data collection targeted a variety of stakeholders involved in the use of, support for users of, implementation and maintenance of learning technology. a variety of data collection methods were used including interviews, document collection and a reflective journal. the research shows that learning technology has significant transformational potential at both individual and at institutional levels. however, this study has unearthed a ‘chicken or the egg’ conundrum. although learning technology is an important part of the future of educational institutions, the adaptability of the organisation and capacity to predict, plan for and support ongoing changes in learning technology is an important part of realising the transformational potential and effectiveness of learning technology. keywords: transformation; learning technology; online learning environment; adaptive cycle framework; learning management system; para-analysis introduction this study aimed to investigate the transformational impact of introducing of a new online learning environment (ole) in a regional australian university over the time period 2007–2009. the exploration of this transformation is grounded in the adaptive cycle framework (buchan 2008a, 2008b) and this study aims to develop this framework further. the adaptive cycle framework forms part of a social-ecological systems approach to the management of the learning environment (cumming et al. 2006; walker et al. 2006; buchan 2008b). the social–ecological systems approach has both a temporal and physical basis and requires that one takes a holistic view of the learning environment. this approach informs the basic premise in this research; that is, one must focus at two levels, individual and institutional, when investigating aspects of learning technology such as its transformational potential. *email: jbuchan@csu.edu.au research in learning technologyaquatic insects vol. 19, no. 2, july 2011, 155–172 issn 2156-7069 print/issn 2156-7077 online � 2011 association for learning technology doi: 10.1080/21567069.2011.586674 http://www.informaworld.com background the simplest definition of transform is ‘to change’ (collins concise dictionary), and at an institutional level, transformation has been noted as one kind of change (kezar 2003). it is appropriate that in defining transformation in higher education we place value on the learning that takes place during a transformation. transformational learning (a term used interchangeably with transformative learning) is a topical area of research (mckewen 2009). transformational learning has been described as “the expansion of consciousness through the transformation of basic world view and specific capabilities of self” (elias 1997 quoted in mckewen 2009). while much of the research into transformative learning in higher education is centred on formal, student-centred learning experiences (mezirow and associates 2000), in this study the focus is on collective learning at an institutional level. although an organisation itself cannot ‘learn’, transformational learning by individuals within the organisation expands the collective consciousness and may transform (change) the world view and collective capabilities of the organisation (buchan 2010a). this study forms part of a broader doctoral research project and is part of a series of publications and presentations on this research (buchan 2008a, 2008b, 2010a, 2010b). this particular study explores the transformational impact at a variety of levels within charles sturt university (csu) of the introduction of a new online learning environment (ole) within the institution over the time period 2007–2009. the platform for the ole is the open source (sakai) collaboration and learning environment (cle). csu is the largest provider of distance education in australia with some 33,000 students, of whom over two-thirds are enrolled as distance education students. csu is a regional, multi-campus, mixed mode (distance/on-campus) university and its campuses are distributed over hundreds of kilometres in new south wales. the wide range of students and physical locations provide significant challenges for communication and teaching. from the late 1990s onwards, the university had an ole that consisted primarily of in-house developed systems with an increasing range of tools. the university did not have a learning management systems equivalent and had limited online interactivity or web 2.0 functionality. the sakai cle extends the concepts of similar systems such as learning management systems, content (courseware) management systems and virtual learning environments to use beyond the classroom such as research purposes. sakai is distributed as free and open source software under an educational community licence (berg and korcuska 2009). the social–ecological systems approach has been used to ground the exploration of the transformational potential of learning technology. within this approach, the focus remains on maintaining the best possible learning environment for staff and students amidst the constant change within an organisation, and under the influence of external environmental factors. there has been considerable research into the acceptance, uptake and use of learning technology (rogers 2003; hannan 2005; keller 2005; benson and palaskas 2006). there is also a body of research around the change management and implementation of learning technology (benson and palaskas 2006; uys 2009; conole, white, and oliver 2007). the social–ecological system model uses two dynamics to describe a changing system: the adaptive cycle and panarchy (buchan 2008b, 2010b; walker and salt 2006). the properties of the social–ecological system 156 j.f. buchan approach that determine a system’s capacity to manage change are its resilience, adaptability and transformability. this research, however, steps outside those areas of knowledge and looks at new ways of understanding changing systems and transformation with a view to improved management practices. the adaptive cycle the adaptive cycle is one of the key dynamics of the social-ecological systems approach. it is grounded in ecological studies and effectively describes the dynamics of an ecosystem and how such a system might respond to changes in the environment. at its simplest, the adaptive cycle has two opposing modes; a development loop, the fore loop; and a release and reorganisation loop, the back loop (figure 1). the author has developed the adaptive cycle framework (figure 2) as a systems analysis tool for understanding and managing the dynamics of a changing environment, in particular the technology-enhanced learning environment, when an educational institution moves through a period of transformation brought about by the introduction of new learning technology (buchan 2008b; uys 2009). the adaptive cycle framework will be used here to contextualise the findings of this study and to illustrate the transformation at an institutional level that can be attributed to learning technology. the findings and deep thinking associated with this phase of the research have contributed to the further development of the adaptive cycle framework as it is presented here. the adaptive cycle framework uses descriptors that are being developed as the notable factors that can be used to build up a picture of the transformation process at the institutional systems level. because the framework was adapted from another discipline area, the environmental management field, the original descriptors have been modified for this new applied use (buchan 2008b). potential refers to the capital, resources and energy stored in the system and the overall capacity to carry out the core business of teaching and research. for a university, the resources include the physical infrastructure; structural assets such as buildings and equipment, information technology (it) infrastructure – hardware and software; and the people (i.e. staff and students). connectedness refers to the relationships, interactions and figure 1. simple representation of the adaptive cycle (after walker and salt 2006). research in learning technology 157 dependencies between the individual components of the system. in technologyenhanced learning environments these connections and interactions include those between the academic staff and learning and teaching support staff, connections between staff and students, the connections between it support areas and staff and students, amongst others. stability in this context means staying the same, largely unchanging, balanced, lasting. policy and processes are the day-to-day formal processes and procedures at microand macro-levels that enable an organisation to run efficiently. because it is a ‘cycle’ there is effectively no beginning or end. it is becoming clear through the findings of this research that with respect to the introduction and implementation of learning technology there is also no single ideal place in the adaptive cycle to start transforming and this will vary for each individual situation (uys 2009). this research investigates how different parts of the institutional system move through transformation since different parts of a system may move at different speeds. the adaptive cycle framework is presented as a metaphor to view and understand a changing system. it provides a way to predict what is likely to happen next and to manage for this in order to provide a smooth transformation. similarly, although all individuals will need to ‘weather’ the changes in a system during periods of change, some individuals may be better suited to operating in one phase than another. there are four phases in the adaptive cycle framework: institutionalisation, creative destruction, reorganisation, and rapid growth. the institutionalisation (conservation) phase is usually the longest phase. the potential of capital, resources and energy stored in the system increases over time as a system moves towards the peak of this phase. net growth slows and the system becomes increasingly connected, less flexible and thus more vulnerable to internal and external changes. disturbances and changes lead to the release phase, referred to as the creative destruction phase. in this brief release phase the dynamics are chaotic, but the destruction that ensues has a creative element (walker and salt 2006). stable relationships, processes and institutional structures will be shaken up. this is a period of release of bound-up resources in which some existing structures fall apart. in nature, the transition from conservation to release phase can happen quickly – for example, when a fire or flood goes through the ecosystem – or more slowly – for figure 2. the adaptive cycle framework. 158 j.f. buchan example, climate change. in higher education the change may be gradual – universities responding to economic trends and competition in the sector. sudden change can be initiated – for example, changes driven by new government agendas or external environmental factors such as the global financial crisis. the creative destruction phase is followed by the reorganisation phase in which innovation and renewal can take hold. with learning technology, innovation and experimentation are the order of the day with early adopters likely to play a key role. in order to harness the power of innovation and experimentation for long-term transformation and renewal of the system, an institution needs to move into the rapid growth phase, which begins the ‘mainstreaming’ of new processes. in the case of learning technology this might include new ways of teaching and new modes of support for teaching. the rapid growth phase sees an increase in potential and connectedness between the components of the system-associated resources. rapid growth is characterised by the activity of ‘opportunists’ who capitalise on the existing conditions and opportunities (e.g. the availability of new learning technology, special project funding, professional development, etc.) to help embed the learning technology as part of the mainstream, institutional system. because it is a ‘cycle’ there is effectively no beginning or end. it is becoming clear through the findings of this research that with respect to the introduction and implementation of learning technology there is also no single ideal place in the adaptive cycle to start transforming and this will vary for each individual situation. this research investigates how different parts of the institutional systems move through transformation since different parts of a system may move at different speeds. the adaptive cycle framework is presented as a metaphor to view and understand a changing system. it provides a way to predict what is likely to happen next and to manage for this in order to provide a smooth transformation (uys and buchan 2009). similarly, although all individuals will need to ‘weather’ the changes in a system during periods of change, some individuals may be better suited to operating in one phase than another. methodology the period covered in this study was 2007–2009. data collection extended prior to and beyond this period to provide an important historical background to the changing technology-enhanced learning environment at the university. a case-study approach has been used with a regional, multi-campus australian university as the single case study. this qualitative research study primarily uses an ethnographic approach with the researcher as participant (wiersma 2000; walter 2006). a variety of data collection methods were used to address the many-faceted research study. in order to address the aims of this particular paper, a selection of data was made from the extensive data collection. in 2008, during the first full year of the implementation of the new ole, targeted in-depth structured interviews were done with 16 staff who had been involved in the implementation, support and/or use of learning technology across the university. those interviewed were representative of a cross-section of the university staff involved with learning technology: academics, faculty members in leadership positions, and staff supporting learning and teaching such as learning and teaching services staff, it management staff and student support staff. these interviews raised wide-ranging questions with the primary research in learning technology 159 focus being on gathering information to understand attitude to change and transformation at an individual and at an institutional level. late in 2009, six further interviews were conducted to explore aspects of transformation at the institutional level in more depth. the focus of the interviews relevant to this paper is the two-level focus on the transformational impact of change associated with learning technology at both individual and institutional levels. the interviews also explored aspects of educational management in order to identify issues which staff and students face in the current learning environment and to determine effective strategies that can be used in the successful implementation and support of learning technology. in keeping with the ethnographic approach a journal was kept of personal notes and reflections during and after regular divisional/unit meetings, professional development workshops, during local, national and international conferences and through formal and informal discussions in a variety of groups. these personal notes have been used to develop an understanding of the overall ‘system’ that is involved with the introduction and support of learning technology at the university. aspects of this system include timelines, roles of individuals and groups and inter-personal relationships. these notes and reflections identify the potential impact factors at organisational level and from the broader higher education environment. secondary sources of data included historical documents, memos, university records and a variety of personal communications that were used to build up a picture of how the technology-enhanced learning environment is changing. these records included committee and working group agendas and minutes; working party reports; strategic plans; official memos; policy and regulatory statements; newsletters and media releases; informal personal communications and user surveys around the use of learning technology. over 300 documents, covering 1998–2010, were collected. the documents were a source of data that contributed towards identifying some of the potential impact factors that are important in institutional transformation associated with learning technology. the records were collated in excel and coded according to type of record, date of publication/writing, author, audience and key words. findings the findings are grouped into two main areas in keeping with the research aims. firstly, evidence is presented that contributes towards identifying the transformational impacts of learning technology (table 1 and table 2). secondly, evidence is presented that contributes to demonstrating the efficacy of the adaptive cycle framework as a metaphor for framing transformation at an institutional systems level (table 3). the responses of participants in the interviews, researcher observation and reflection, and documentary evidence have been used to compile the data below. the primary data sources (interviews and observation) have been combined with the secondary data sources (institutional surveys, reports, memos, conference proceedings, etc.). there is no difference in weighting and overall importance given to the primary versus secondary data sources. each serves its own purpose in contributing to the overall picture of the transformational impact of learning technology. as transformation is essentially ‘a change’, both positive and negative impacts have been noted. 160 j.f. buchan a summary of some of the observed transformational impacts of learning technology at an individual level is given in table 1. a variety of data have been collected and synthesised to determine key areas of the contribution of learning technology to the transformation of learning and teaching. from the data, six key areas emerged in the contributions of learning technology to transforming learning and teaching in the university. table 2 summarises the selection of evidence that illustrates some of these key areas. table 3 illustrates transformation at an institutional level. the phases of the adaptive cycle framework have been used to contextualise some of the evidence to demonstrate the events and features related to the introduction of the new online table 1. summary of transformational impacts of learning technology at individual level. academic staff (some of these attributes also apply to other staffa) learning and teaching services support staffa + more adaptable to new technology, prepared to try new thingsa # new skills in a range of learning management systems and web 2.0 tools, and digital media tools (e.g. captivate) + more (constructively) critical of new technologya # new skills in designing learning experiences for online medium, designing digital media – heading towards a base-level technology expertise + awareness that learning needs should drive the technology, interact is only a set of tools + stepping up in providing professional development around use of technology and embedding subjects in the e-learning environment # reflecting on learning design in new mediaa + increased workloads # new skills in designing blended learning experiencesa + new skills, learning management systems and wide range of web 2.0 tools, captivate, dreamweaver, designing learning experiences for onlinea information technology servicesa # enabling the use of new pedagogies and learning approaches with students + internet access and communication systems have transformed work practices and lives + higher expectations of the standard of technologya # increased skills in programming and use of new applications + lack of trust in university it systems (where some new tools have not worked well) # development of new collaborative processes for software development. + increased workloadsa student services support staff a students + internet access and communication systems have transformed work practices and lives + immersion in technology creates digital natives who adapt quickly to using different learning technologies + value the facility to have a social presence + at micro level, technology has solved specific communication issues for distance education students + at micro level, technology has solved specific communication issues for distance education students + increased skills in new university systems note: +, data from interviews; #, data from researcher observation and document analysis. research in learning technology 161 t ab le 2 . c o n tr ib u ti o n o f le ar n in g te ch n o lo g y to tr an sf o rm in g le ar n in g an d te ac h in g . u n d er st an d in g an d aw ar en es s o f le ar n in g te ch n o lo g y in cr ea se d o p p o rt u n it ie s fo r n et w o rk in g ar o u n d le ar n in g an d te ac h in g ac ro ss th e u n iv er si ty in d iv id u al sk il ls in cr ea se d u se o f d ig it al m ed ia a v ai la b il it y o f to o ls fo r te ac h in g o n li n e r ed es ig n o f co u rs es fo r fl ex ib le an d b le n d ed le ar n in g + u se rs n o w u n d er st an d m o re ab o u t te ch n o lo g y an d ar e m o re d is ce rn in g ; m o re d ig it al ly li te ra te # c s u e d c o n fe re n ce s b ri n g to g et h er st af f fr o m al l d iv is io n s, li b ra ry , st u d en t se rv ic es , le ar n in g an d te ac h in g se rv ic es , it , ac ad em ic s fr o m al l fa cu lt ie s an d d if fe re n t d is ci p li n es # o v er al l in cr ea se in it sk il ls am o n g st ac ad em ic , su p p o rt an d ad m in is tr at iv e st af f # in cr ea se in u se o f te ch n o lo g y th ro u g h d ig it al m ed ia + # en h an ce s cr o ss ca m p u s te ac h in g ; m ak es fl ex ib le an d b le n d ed le ar n in g p o ss ib le # b le n d ed an d fl ex ib le le ar n in g ad v an ce s p o ss ib le + u se rs ar e n o w m o re cr it ic al o f te ch n o lo g y an d b et te r ab le to m ak e d ec is io n s ab o u t u si n g n ew to o ls , et c. # ic t -c o p is d is tr ib u te d ac ro ss al l ca m p u se s, fa cu lt ie s an d d iv is io n s o f th e u n iv er si ty . t h e c s u in te ra ct si te “a b o u t ic t in te g ra ti o n ” h as 5 3 7 m em b er s. y am m er u se d as a p ro fe ss io n al n et w o rk in g si te h as 2 0 5 m em b er s # d ev el o p in g co re co m p et en ci es fo r ed u ca ti o n al d es ig n er s # in cr ea se in ab il it y to d es ig n fo r d ig it al m ed ia an d o n li n e le ar n in g + en ab le s st u d en ts an d st af f to h av e a so ci al p re se n ce in su b je ct s; # en ab le s a co ll ab o ra ti v e le ar n in g en v ir o n m en t c ro ss -c am p u s o ff er in g o f ex am p le s b le n d ed le ar n in g co u rs es – ea rl y ch il d h o o d /i n fo rm at io n st u d ie s/ h u m an m o v em en t/ n u rs in g + st u d en ts -i m m er si o n in te ch n o lo g y cr ea te s d ig it al n at iv es w h o ad ap t q u ic k ly to u si n g d if fe re n t le ar n in g te ch n o lo g ie s # a u s ak ai co n fe re n ce s 2 0 0 7 /0 8 /0 9 n et w o rk in g w it h o th er in st it u ti o n s # in cr ea se d it d ev el o p m en t sk il ls fo r p ro g ra m m er s # u se o f c d -r o m s fo r d e st u d en ts h as in cr ea se d : 2 0 0 5 , a to ta l o f 9 0 0 0 c d ro m s se n t o u t to st u d en ts ; 2 0 0 6 , 1 5 ,0 0 0 ; 2 0 0 7 , 3 0 ,0 0 0 ; 2 0 0 9 , 4 0 ,0 0 0 + en ab le s le ct u re rs to co p e w it h h ig h an d co m p le x w o rk lo ad s; # en ab le s co u rs es to ru n w it h st af f sh o rt ag es an d in cr ea se d w o rk lo ad s n o te : + , d at a fr o m in te rv ie w s; # , d at a fr o m re se ar ch er o b se rv at io n an d d o cu m en t an al y si s. 162 j.f. buchan t ab le 3 . t ra n sf o rm at io n at an in st it u ti o n al le v el . f ac u lt ie s/ sc h o o ls l ea rn in g an d te ac h in g su p p o rt in fo rm at io n te ch n o lo g y su p p o rt a c re a ti ve d es tr u ct io n (c h a o s) p h a se 2 0 0 6 u n iv er si ty -w id e fa cu lt y re st ru ct u re 2 0 0 7 (n ew c s u st ra te g y 2 0 0 7 – 2 0 11 ) f ir st -r o u n d st ra te g ic v o lu n ta ry se p ar at io n s in fa cu lt ie s s ta ff se co n d ed to o l e p ro g ra m m e to im p le m en t in te ra ct w o rk in g in m u lt ist ak eh o ld er te am s o n o l e p ro g ra m m e 2 0 0 8 a ll d e an d in te rn al su b je ct s m o v e to th e n ew c s u in te ra ct ; f ac u lt y o f e d u ca ti o n em b ra ce s in te ra ct to su p p o rt w h o le o f fa cu lt y te am ap p ro ac h to cr o ss -c am p u s co u rs es ; fu rt h er is o la te d fa cu lt y re st ru ct u re s s ta ff se co n d ed to o l e p ro g ra m m e to im p le m en t in te ra ct ; e d s u n d er g o in te n se p ro fe ss io n al d ev el o p m en t in te ra ct im p le m en ta ti o n cr ea te s h ig h d em an d o n it s er v ic e d es k ; re sp o n si b il it y sh ar ed w it h sp ec ia li st e d s 2 0 0 9 in tr o d u ct io n o f n ew s u b je ct o u tl in e to o l ac ro ss u n iv er si ty ca u se s m aj o r ch an g e an d d is ru p ti o n to p ro ce ss es ; n ew f ac u lt y te am m o d el o f e d su p p o rt ; in cr ea se d ca su al is at io n o f ac ad em ic st af f in al l fa cu lt ie s n ew d iv is io n st ru ct u re g iv es e d s m o re re sp o n si b il it y fo r ed u ca ti o n al te ch n o lo g y su p p o rt ; in tr o d u ct io n o f n ew s u b je ct o u tl in e to o l ac ro ss u n iv er si ty ca u se s m aj o r ch an g e an d d is ru p ti o n to p ro ce ss es it p ro g ra m m er s/ so lu ti o n s co o rd in at o rs d ed ic at ed to le ar n in g te ch n o lo g y (l t s ) 2 0 1 0 in cr ea se d ca su al is at io n o f st af f c h an g e in ed u ca ti o n al te ch n o lo g is t su p p o rt ; e d s w it h sp ec ia li st sk il ls r eo rg a n is a ti o n p h a se (i n n o va ti o n a n d re n ew a l) 2 0 0 7 in te ra ct p il o ts ru n in ea ch sc h o o l al lo w st af f to tr y n ew w ay s o f te ac h in g ; so m e sc h o o ls u se p il o t ex te n si v el y fo r te ac h in g ; o th er s m o re co n se rv at iv e to st ar t; n ew fa cu lt ie s an d sc h o o ls se tt li n g e d s in it ia te p il o ts in sc h o o ls ; u p sk il l in u si n g n ew te ch n o lo g y ; su p p o rt st af f in p re p ar in g to u se in te ra ct in te ac h in g ; ch an g e o f ro le s, ch an g e o f sc h o o l p ro ce ss es , tr y in g n ew p ro ce ss es in sc h o o ls ; f le x ib le le ar n in g an d in te ra ct t ea m fo rm ed to o v er se e e d re la te d as p ec ts o f in te ra ct im p le m en ta ti o n p ro g ra m m er s le ar n to u se o p en so u rc e , b ec o m e p ar t o f s ak ai co m m u n it y (c o n ti n u ed o n n ex t p a g e) research in learning technology 163 t ab le 3 . (c o n ti n u ed ) f ac u lt ie s/ sc h o o ls l ea rn in g an d te ac h in g su p p o rt in fo rm at io n te ch n o lo g y su p p o rt a 2 0 0 8 f le x ib le l ea rn in g in st it u te es ta b li sh ed ; m id y ea r in te ri m (q u al it at iv e) re p o rt b y e d s o n sc h o o l u sa g e o f in te ra ct in d ic at es m ix ed u se , ra n g in g fr o m m in im al an d b as ic u se o f st an d ar d in te ra ct to o ls to in n o v at iv e u se o f w id e v ar ie ty o f to o ls (c h at , w ik i, et c. ); c s u e d 2 0 0 8 c o n fe re n ce p re se n ta ti o n s d em o n st ra te a w ea lt h o f u se s o f in te ra ct an d o th er te ch n o lo g y ; p re se n ta ti o n s co n fi rm th e v ar ie d le v el s o f te ch n o lo g y ad o p ti o n an d u se e d s w o rk w it h ac ad em ic s to lo o k at al te rn at iv e w ay s o f te ac h in g , re so u rc e p ro v is io n in n ew o n li n e le ar n in g en v ir o n m en t; f le x ib le le ar n in g an d in te ra ct t ea m ac ti v e p ro g ra m m er s le ar n to u se o p en so u rc e, b ec o m e p ar t o f s ak ai co m m u n it y 2 0 0 9 r eg u la r ic t -e n ab le d c o p fo ru m s d em o n st ra te an d en co u ra g e sh ar in g p ra ct ic e an d ex am p le s o f in n o v at io n an d re n ew al in l & t t ri al li n g m ai n st re am p ro d u ct io n o f in te ra ct b as ed le ar n in g re so u rc es ; n ew fa cu lt y e d te am st ru ct u re ; n ew co ll ab o ra ti o n s in te ra ct – s er v ic e d es k re sp o n si b il it y m o v es to l t s sp ec ia li st ; it p ro g ra m m er s b u il d n ew su b je ct o u tl in e to o l 2 0 1 0 r eg u la r ic t -e n ab le d c o p fo ru m s d em o n st ra te an d en co u ra g e ex am p le s o f in n o v at io n an d re n ew al in l & t ; co u rs e m ap p in g se rv ic e tr ia ll ed an d m ak es u se o f n ew o n li n e su b je ct o u tl in e to o l; p il o ts o f an d re le as e o f si n g le to o ls , e. g . g ra d eb o o k , o n li n e m ee ti n g to o l (w im b a) ; d o m s (e q u el la ); p o d ca st in g to o l in in te ra ct ; co u rs e m ap p in g u si n g te ch n o lo g y e d s co n ti n u e to tr y n ew in te ra ct to o ls an d su p p o rt ac ad em ic st af f; e d s ta k e o n n ew ar ea s o f p ro fe ss io n al fo cu s in te ch n o lo g y o n g o in g ro ll o u t o f n ew in te ra ct to o ls , s ak ai 3 d ev el o p m en t; c s u le ad in g co n tr ib u to r to s ak ai 3 co m m u n it y p ro je ct ; l t s fu ll cu st o d ia n s o f th e o l e . s er v ic e le v el ag re em en ts w it h d it r a p id g ro w th p h a se 2 0 0 7 2 0 0 8 u se o f in te ra ct in al l d e su b je ct s; ra p id u p ta k e o f co re in te ra ct to o ls e d s w o rk in g w it h ac ad em ic s to lo o k at al te rn at iv e w ay s o f te ac h in g , re so u rc e p ro v is io n in n ew o n li n e le ar n in g en v ir o n m en t it p ro g ra m m er s m as te ri n g s ak ai co d in g (c o n ti n u ed o n n ex t p a g e) 164 j.f. buchan t ab le 3 . (c o n ti n u ed ) f ac u lt ie s/ sc h o o ls l ea rn in g an d te ac h in g su p p o rt in fo rm at io n te ch n o lo g y su p p o rt a 2 0 0 9 f ac u lt y o f e d u ca ti o n se co n d s e d s an d st af f fo r d ev el o p in g o n li n e co u rs es ; in cr ea se d u se o f in te ra ct in in te rn al su b je ct s; f le x ib le l ea rn in g in st it u te g ro w th ; f le x ib le l ea rn in g in st it u te fe ll o w sc h em e es ta b li sh ed ; – so m e fa cu lt ie s o ff er in n o v at io n p ro je ct fu n d in g e d s se co n d ed fo r sp ec ifi c p ro je ct s re v is in g fu ll co u rs es fo r sc h o o ls in e d u ca ti o n ; (f le x ib le l ea rn in g an d in te ra ct t ea m d is b an d ed ) ed u ca ti o n al te ch n o lo g is ts an d so m e w it h sp ec ia li st ex p er ti se su p p o rt in te ra ct d ev el o p m en t; n ew ic t -e n ab le d le ar n in g co m m it te e it p ro g ra m m er s co n tr ib u ti n g to s ak ai co m m u n it y ; d ed ic at ed s er v ic e d es k h el p fo r in te ra ct 2 0 1 0 e d s em p lo y ed fo r sp ec ifi c p ro je ct s re v is in g fu ll co u rs es fo r o n li n e f ac u lt ie s o f e d u ca ti o n , a rt s an d s ci en ce ; m o re st ab il it y in p ro ce ss es in so m e sc h o o ls in u si n g th e n ew su b je ct o u tl in e to o l e d s em p lo y ed fo r sp ec ifi c p ro je ct s re v is in g fu ll co u rs es fo r o n li n e f ac u lt ie s o f e d u ca ti o n , a rt s an d s ci en ce ; in cr ea se d d em an d fo r m ai n st re am in g p ro d u ct io n o f in te ra ct -b as ed le ar n in g re so u rc es it p ro g ra m m er s co n tr ib u ti n g to s ak ai co m m u n it y in st it u ti o n a li sa ti o n p h a se 2 0 0 6 in cr ea se d u se o f ex is ti n g o l e to o ls ; l ea rn in g an d te ac h in g su p p o rt u n it ; le ar n in g m at er ia ls p ro d u ct io n su p p o rt u n it ; ce n tr al is ed p ro d u ct io n o f o n li n e le ar n in g re so u rc es o n re q u es t, st u d y g u id es , al l su b je ct o u tl in es (l m c ) it su p p o rt fo r ed u ca ti o n al te ch n o lo g y o n a p ro je ct b as e (p ro je ct s er v ic e c en tr e) ; 2 0 0 7 u n iv er si ty -w id e im p le m en ta ti o n p ro g ra m m e ar o u n d in te ra ct w it h p d , et c. c en tr al is ed p ro d u ct io n o f o n li n e le ar n in g re so u rc es o n re q u es t – st u d y g u id es , al l su b je ct o u tl in es it su p p o rt fo r ed u ca ti o n al te ch n o lo g y o n a p ro je ct b as e (p ro je ct s er v ic e c en tr e) 2 0 0 8 a ll d e an d in te rn al su b je ct s m o v e to th e n ew c s u in te ra ct e d s p ro v id e w id es p re ad p ro fe ss io n al d ev el o p m en t an d as si st an ce w it h in te ra ct it se rv ic es m ai n ta in ex is ti n g o l e (c o n ti n u ed o n n ex t p a g e) research in learning technology 165 t ab le 3 . (c o n ti n u ed ) f ac u lt ie s/ sc h o o ls l ea rn in g an d te ac h in g su p p o rt in fo rm at io n te ch n o lo g y su p p o rt a 2 0 0 9 in cr ea si n g u se o f in te ra ct , d is ta n ce ed u ca ti o n an d in te rn al r ep la ce e d an d b le n d ed le ar n in g , ti m es av in g an d re u se o f le ar n in g re so u rc es d iv is io n o f l ea rn in g an d t ea ch in g s er v ic es fo rm ed ; m in im al ce n tr al is ed p ro d u ct io n o f o n li n e re so u rc es , p ro fe ss io n al d ev el o p m en t p ro g ra m m es es ta b li sh ed , e d s p ro v id e at y o u rd es k h el p , o l e p ro g ra m m e en d s. o n g o in g su p p o rt o f o l e , w el ld ev el o p ed co n n ec ti o n s w it h s ak ai co m m u n it y 2 0 1 0 f le x ib le l ea rn in g in st it u te an d f le x ib le l ea rn in g in st it u te fe ll o w s sc h em e p ro d u ct io n se rv ic es o ff er ed to su p p o rt o n li n e su b je ct o u tl in e d ev el o p m en t o n a n ee d s b as is l t s b ec o m e fu ll cu st o d ia n s o f th e o l e . s er v ic ele v el ag re em en ts w it h d it ; n ew ed u ca ti o n al te ch n o lo g y fr am ew o rk , d ed ic at ed s er v ic e d es k h el p fo r in te ra ct n o te : c o p ; d e ; d it ; e d ; ic t , in fo rm at io n an d co m m u n ic at io n s te ch n o lo g ie s; l m c ; l & t ; l t s ; o l e , o p en le ar n in g en v ir o n m en t; p d . a it is n o te d h er e th at th e d at a p re se n te d fo r it su p p o rt ar e n o t ex te n si v e an d ar e li m it ed to th at av ai la b le th ro u g h th e re se ar ch er ’s in v o lv em en t in in te rd iv is io n al in it ia ti v es . 166 j.f. buchan learning environment that characterise each of the phases of the framework. for reasons of space, only a snapshot of the evidence is provided. the events and observations that have been identified in each phase of the transformation process have been used to extract and generalise some features that appear to characterise each phase. these features can be used to identify what phase a part of an institution might be in with respect to an institutional transformation process. it is acknowledged that these features are not yet comprehensive and are only a starting point for discussion and ongoing research. there may also be some limitations in their generalisability to other organisations since this research is based on a single case study. these features are as follows: � creative destruction. loss of normal connections, changes in interactions amongst stakeholders, inefficiencies in operations, loss of dependencies, changes in roles, freeing up of resources/people from old ways of doing things. � reorganisation. innovation, trying new ways of doing things, trial and experimentation in day-to-day operations, sharing of ideas, questioning of status quo, pilots and trials of new technology, teams/communities of practice set up, inefficiencies, leadership emerges. � rapid growth. new processes and procedures developed, sharing of practice, acceptance of technology, improved efficiency, leadership cemented, creating and taking opportunities to make the most use of new technology and available support opportunities, new connections, interactions and dependencies forming, collaboration. � institutionalisation. improved efficiencies in operations, long-lasting relationships develop, ongoing development of processes, ongoing small-scale renewal and review and improvement of processes, building up of resources, centralised services have well-developed processes and procedures. discussion it should be noted that prior to the introduction of the new ole there has always been an extremely active learning and teaching focus at the university with school and faculty-based opportunities for people to share practice, along with regular learning and teaching conferences. since the 1990s, the introduction of new learning technology tools has been accompanied by sound research and a striving for continuous improvement (chan, lee, and mcloughlin 2006; dalgarno 1998; burr and morton-allen 2001). this study was limited to looking at the transformational impact of learning technology over a particular time frame that coincides with the introduction of a new ole and the move to web 2.0 capacity within the university to support university strategic directions. when investing in new technology, institutions often look to a demonstration of the ‘value for money’ and want measureable, tangible benefits. this research suggests that the benefits go beyond new ways of teaching and improved student learning. at the individual level there is evidence of transformation in a number of areas (table 1 and table 2). there has been a transformation in the skills of academic and support staff in using technology and web 2.0 tools specifically and an increased capability in designing a variety of learning experiences to make use of research in learning technology 167 the new tools. for educational designers the core role has changed and has necessitated a significant increase in skills and knowledge. student support services staff noted that the access to technology has transformed the way they can teach and support students. there have been changes in individuals’ attitudes to technology. a pertinent transformation that has been noted is that some academic and support staff are now more discerning and aware of technology, have higher expectations and report less tolerance of systems that do not match up to expectations. for it programmers and developers there has been a major increase in individual skills consistent with the development of new collaborative processes in the sakai community for the open source software development. 1 at the institutional level, the implementation of the new ole, csu interact, across the whole of the institution during 2007/08 affected the entire university in some way, from faculties to student support services and staff administration. this study has focused on a number of key areas and stakeholders directly involved with support of the academic staff and the technology itself. if one was to position the case study university in the adaptive cycle framework with respect to learning technology – more specifically web 2.0 technology – the status quo in 2006, prior to the beginning of the implementation of csu interact, would have been in the late institutionalisation phase. at this time, because of its lack of web 2.0 capacity, the university was extremely vulnerable to changes in the external environment (buchan 2008a). the beginnings of the implementation of csu interact in 2007 can be said to mark the start of the creative destruction phase associated with the move to the new ole and web 2.0 technology. at that time this was not the only phenomenon to cause ‘creative destruction’ for faculty restructures and university-wide staffing rationalisations targeting academic staff made an impact in 2006/07 and for some faculties this continued in 2008/09. each year following the initial implementation of csu interact, new university initiatives have been identified as having caused some degree of disruption (buchan 2010b). some stakeholders such as the academic staff appeared to be affected by more changes to systems, processes and technology than others. there has been constant innovation and renewal in all key areas. for those responsible for implementation of the new ole, it services and learning and teaching services, the innovation and renewal has taken place earlier than the ‘receivers’ of the new technology (i.e. the faculties). there were ongoing new initiatives associated with learning technology and an associated need for constant innovation and renewal in a variety of areas including institutional structures and operations. evidence from the csu education 2005, 2008 and 2009 internal conferences demonstrate that innovation and renewal in learning and teaching with technology are strong in most faculties. information and communications technology-enabled community of practice forums initiated in 2009 have provided the opportunity for staff to share their innovation and use of learning technology. some faculties and schools demonstrated a tendency towards a constant, self-generated state of innovation and renewal, while for others innovation and renewal in learning technology has been the consequence of a more ‘involuntary’ move into the creative destruction phase and the ensuing innovation and renewal as a result of the introduction of the new ole. there is less evidence to illustrate widespread progress through the rapid growth phase. the evidence illustrating rapid growth tends to be made up of the many small initiatives that began ‘mainstreaming’ the use of the new technology and 168 j.f. buchan associated support processes. at less than two years after the introduction of csu interact this may be expected, especially considering the other external factors and technologies that have impacted. the introduction of a new, university-wide, online subject outline system (msi) in 2009 has been a significant impact factor that has caused a major disruption for faculties and learning and teaching support staff, and appears to have affected the ongoing consolidation of the use of the new ole and other technologies. it has also been difficult to clearly distinguish between innovation and renewal, the rapid growth phase and the move towards the early institutionalisation phase. this is an area that requires more exploration in order to refine the model. there is evidence to confirm that parts of the system have moved into a new institutionalisation phase with respect to learning technology. there is the beginning of building-up of resources such as learning and teaching support services and professional development programmes. there are it staff dedicated to learning technology development, flexible learning institute programmes and fellows to promote the scholarly use of learning technology. there has been major structural change at institutional level (learning and teaching services and it services) to support ongoing strategic directions in learning technology (buchan 2010b). the system has, however, not yet reached a stability phase with respect to learning technology. this is to be expected since the institution is still in a rapid growth phase with respect to new learning technology. conclusions and future directions transformation is not always smooth and predictable. when analysed using the adaptive cycle framework, different parts of an institutional system appear to move through the phases of transformation at different speeds. for maximum transformational benefit of major technological change, the change agents responsible for implementing new learning technology and associated support processes (it support staff, learning and teaching services, faculty leaders, etc.) need to be prepared for regular periods of creative destruction; to spend time in the innovation and renewal phase; and to develop the necessary skills and processes to assist their own areas to move quickly through the rapid growth phase in order to reach the relative stability of the institutionalisation phase for those whom they support. in reality, in this study there were many different initiatives and changes taking place figure 3. representation of a dynamic and stable institutional phase. research in learning technology 169 at any one time in the university, the combined impact of which appeared to affect the realisation of the transformational potential of new learning technology. the message for management then, is to take a holistic and long-term approach to identifying the many different impact factors that need to be taken into consideration when managing for an effective implementation of new learning technology initiatives. on the evidence presented, the earlier assumption that all parts of an institutional system will move/need to move through all the phases of the adaptive cycle framework during the introduction of a major new learning technology is now questioned. with the need to constantly upgrade systems and to keep pace with emerging learning technologies, institutional systems need to be responsive and to have sufficient support structures in place at the institutional level to be able to quickly and effectively implement new learning technology. an assumption was made earlier that for institutional transformation to take place the starting point for the institution would be in the institutionalisation phase. it is suggested here that transformation can begin at any phase in the adaptive cycle. the initiation of transformation can come effectively from areas strong in innovation and with a need for constant renewal. however, the effectiveness of that transformation will depend on how well the process is managed and supported in the institution. a time frame of less than three years appears to be too short for a complete transformation to take place in the case-study institution and for the institution to return to something resembling the institutionalisation phase; that is, stability, building up resources and connections, developing policies and new processes following the introduction of the new online learning environment and web 2.0 technology. the original characteristics/properties by which the stable, institutionalisation phase was defined have also been questioned. what has emerged from this study is that the institutionalisation phase can perhaps be better described as ‘a period when normal business operations, i.e. effective teaching and student learning, can efficiently take place while withstanding minor internal and external changes’. innovation and change is a good thing, but can be costly in time and resources. this does not necessarily mean that there should be no change in the institutional processes, but that the processes and the people are adaptable and resilient (i.e. able to absorb change). in terms of the adaptive cycle framework, a possible ‘stable’ but dynamic institutionalisation phase situation could be illustrated by rapid cycling (adapting) within the institutionalisation phase (figure 3) and occasional generation of transformation from within the reorganisation (innovation and renewal) phase. this would facilitate innovation and renewal while retaining productivity and stability. learning technology should never be stable and unchanging. a stable state is perhaps more effectively described as one of dynamic stability. this is an area for further study through the exploration of two more properties of the social–ecological systems approach – resilience and adaptability. this study has unearthed a ‘chicken or the egg’ conundrum. although learning technology is an important part of the future of educational institutions, the adaptability of the organisation and capacity to predict, plan for and support ongoing changes in learning technology is an important part of realising the transformational potential and effectiveness of learning technology. the impact of technology spreads beyond the user level; that is, simply having access to appropriate technology for particular learning and teaching purposes. if an educational organisation truly embraces learning technology as strategically important to its future, then it 170 j.f. buchan will need to do more than choose and implement new systems and applications. it will need to prepare to undergo significant transformation in a variety of areas to support the vision for learning and teaching and the institution’s technology needs of the future. note 1. see sakai website: http://sakaiproject.org. references benson, r., and t. palaskas. 2006. introducing a new learning management system: an institutional case study. australasian journal of educational technology 22, no. 4: 548– 67. berg, a.,and m. korcuska. 2009. sakai courseware management. the official guide. ed. j. lumsden, s. mangarole and a. shanker. birmingham: packt publishing. buchan, j. 2008a. rethinking management strategies for the online learning environment. paper presented at alt-c rethinking the digital divide, 9–11 september 2008, in leeds, uk. buchan, j. 2008b. tools for survival in a changing educational technology environment. paper presented at the where are you now in the landscape of educational technology? ascilite conference, december, in melbourne. buchan, j. 2010a. developing a dynamic and responsive online learning environment: a case study of a large australian university. international journal of open source software and processes 2, no. 1: 32–48. buchan, j.f. 2010b. putting ourselves in the big picture: a sustainable approach to project management for e-learning. the journal of distance education 24, no. 1: 55–76. burr, l., and m. morton-allen, 2001. inventing a better mouth trap: the development and implementation of forum software at charles sturt university. paper presented at the ausweb01 seventh australian world wide web conference, in coffs harbour, southern cross university. chan, a., m. lee, and c. mcloughlin. 2006. everyone’s learning with podcasting: a charles sturt university experience. paper presented at the ascilite conference, december, in sydney. conole, g., s. white, and m. oliver. 2007. the impact of e-learning on organisational roles and structures. in contemporary perspectives in e-learning research: themes methods and impact on practice, eds. g. conole and m. oliver, 69–81. abingdon: routledge. cumming, g., d.h. cumming, m. cumming, and c.l. redman. 2006. scale mismatches in social–ecological systems: causes, consequences, and solutions. in exploring resilience in social–ecological systems. comparative studies and theory development, ed. b. walker, a. anderies, a. kinzig, and p. ryan, 23–40. melbourne: csiro. dalgarno, b. 1998. tools for authoring constructivist computer assisted learning resources: a review. paper presented at the flexibility: the new wave ascilite conference, december, in wollongong. hannan, a. 2005. innovating in higher education: contexts for change in learning technology. british journal of educational technology 36, no. 6: 975–85. keller, c. 2005. virtual learning environments: three implementation perspectives. learning, media and technology 30, no. 3: 299–311. kezar, a. 2003. synthesis of scholarship on change in higher education. higher education policy 16, no. 1: 39–53. mckewen, l. 2009. transformational learning. research briefing, university of gloucestershire, pedagogic research and scholarship institute, gloucester. mezirow, j., and associates. 2000. learning as transformation: critical perspectives on a theory in progress. jossey bass higher and adult education series. san francisco: jossey bass. rogers, e.m. 2003. diffusion of innovation. 5th ed. new york: free press. research in learning technology 171 uys, p. 2009. change and innovation strategies during the implementation of an open source lms: an australian case study. research, reflections and innovations in integrating ict in education. paper presented at the v international conference on multimedia and ict in education (m-icte2009), in lisbon, portugal. walker, b., and d. salt. 2006. resilience thinking. washington, dc: island press. walker, b., j. anderies, a.p. kinzig, and p. ryan, eds. 2006. exploring resilience in social– ecological systems. comparative studies and theory development. melbourne: csiro publishing. walter, m. 2006. social research methods: an australian perspective: oxford university press. wiersma, m. 2000. research methods in education: an introduction. ma: allyn and bacon. 172 j.f. buchan